|
|
Randomization Tests: Contingency Tables (Fisher's Exact Test)
Author(s)
David M. Lane
Prerequisites
Contingency Tables
Learning Objectives
- State the situation when Fisher's exact test can be used
- Calculate Fisher's exact test
- Describe how conservative the Fisher exact test is relative to a Chi Square test
The chapter on Chi Square showed one way to test the relationship between two nominal variables. A special case of this kind of relationship is the difference between proportions. This section shows how to compute a significance test for a difference in proportions using a randomization test. Suppose, in a fictitious experiment, 4 subjects in an Experimental Group and 4 subjects in a Control Group are asked to solve an anagram problem. Three of the 4 subjects in the Experimental Group and none of the subjects in the Control Group solved the problem. Table 1 shows the results in a contingency table.
Table 1. Anagram Problem Data
|
|
Experimental |
Control |
Total |
|
Solved
|
3 |
0 |
3 |
|
Did Not Solve
|
1
|
4 |
5 |
| Total |
4 |
4 |
8 |
The significance test we are going to perform is called the Fisher Exact Test. The basic idea is to take the row totals and column totals as "given" and add the probability of obtaining the pattern of frequencies obtained in the experiment and the probabilities of all other patterns that reflect a greater difference between conditions. The formula for obtaining any given pattern of frequencies is:
where N is the total sample size (8), n is the sample size for the first group (4), r is the number of successes for the first group (3), and R is the total number of successes (3). For this example, the probability is
Since more extreme outcomes do not exist given the row and column totals, the p value is 0.0714. This is a one-tailed probability since it only considers outcomes as extreme or more extreme favoring the Experimental Group. An equally extreme outcome favoring the Control Group is shown in Table 2, which also has a probability of 0.0714. Therefore, the two-tailed probability is 0.1428. Note that in the Fisher Exact Test, the two-tailed probability is not necessarily double the one-tailed probability.
Table 2. Anagram Problem Favoring Control Group
| |
Experimental |
Control |
Total |
| Solved |
0 |
3 |
3 |
| Did not Solve |
4 |
1 |
5 |
| Total |
4 |
4 |
8 |
The Fisher Exact Test is "exact" in the sense that it is not based on a statistic that is approximately distributed as, for example, Chi Square. However, because it assumes that both marginal totals are fixed, it can be considerably less powerful than the Chi Square test. Even though the Chi Square test is an approximate test, the approximation is quite good in most cases and tends to have too low a Type I error rate more often than too high a Type I error rate (see for yourself using this simulation).
Please answer the questions:
|
|