Specific Comparisons (Independent Groups)

Difference Between Two Means (Independent Groups)

Learning Objectives

  1. Define linear combination
  2. Specify a linear combination in terms of coefficients
  3. Do a significance test for a specific comparison

There are many occasions on which the comparisons among means are more complicated than simply comparing one mean with another. This section shows how to test these more complex comparisons. The methods in this section assume that the comparison among means was decided on before looking at the data. Therefore these comparisons are called planned comparisons. A different procedure is necessary for unplanned comparisons.


In case of four means, we can compute the difference between the "success" mean and the "failure" mean by multiplying each "success" mean by 0.5, each failure mean by -0.5 and adding the results. We call the value obtained "L" for "linear combination."
Now, the question is whether the value of L is significantly different from 0. The general formula for L is:

where ci is the ith coefficient and Mi is the ith mean. The formula for testing L for significance is shown below


MSE is the mean of the variances.

The value of n is the number of subjects in each group.

We need to know the degrees for freedom in order to compute the probability value. The degrees of freedom is

df = N - k

where N is the total number of subjects and k is the number of groups.

In a later chapter on Analysis of Variance, you will see that comparisons such as this are testing what is called an interaction. In general, there is an interaction when the effect of one variable differs as a function of the level of another variable.

Multiple Comparisons

The more comparisons you make, the greater your chance of a Type I error. It is useful to distinguish between two error rates: (1) the per-comparison error rate and (2) the familywise error rate. The per-comparison error rate is the probability of a Type I error for a particular comparison. The familywise error rate is the probability of making one or more Type I error in a family or set of comparisons. In the attribution experiment discussed above, we computed two comparisons. If we use the 0.05 level for each comparison, then the per-comparison rate is simply 0.05. The family-wise rate can be complex. Fortunately, there is a simple approximation that is fairly accurate when the number of comparisons is small. Define α as the per-comparison error rate and c as the number of comparisons, the following inequality always holds true for the familywise error rate (FW) can be approximated as:

FW ≤ cα

This inequality is called the Bonferroni inequality.

The Bonferroni inequality can be used to control the familywise error rate as follows: If you want to the familywise error rate to be α, you use α/c as the per-comparison error rate. This correction, called the Bonferroni correction, will generally result in a family wise error rate less than α.

Should the familywise error rate be controlled? Unfortunately, there is no clear-cut answer to this question. The disadvantage of controlling the familywise error rate is that it makes it more difficult to obtain a significant result for any given comparison: The more comparisons you do, the lower the per-comparison rate must be and therefore the harder it is to reach significance. That is, the power is lower when you control the familywise error rate. The advantage is that you have a lower chance of making a Type I error.

One consideration is the definition of a family of comparisons. Let's say you conducted a study in which you were interested in whether there was a difference between male and female babies in the age at which they started crawling. After you finished analyzing the data, a colleague of yours had a totally different research question: Do babies who are born in the winter differ from those born in the summer in the age they start crawling? Should the familywise rate be controlled or should it be allowed to be greater than 0.05? Our view is that there is no reason you should be penalized (by lower power) just because your colleague used the same data to address a different research question. Therefore, the familywise error rate need not be controlled. Consider the two comparisons done on the attribution example at the beginning of this section: These comparisons are testing completely different hypotheses. Therefore, controlling the familywise rate is not necessary.

Orthogonal Comparisons

In the preceding sections, we talked about comparisons being independent. Independent comparisons are often called orthogonal comparisons. There is a simple test to determine whether two comparisons are orthogonal: If the sum of the products of the coefficients is 0, then the comparisons are orthogonal.