|
Specific Comparisons (Independent Groups)
Prerequisites
Difference
Between Two Means (Independent Groups)
Learning Objectives
- Define linear combination
- Specify a linear combination in terms of coefficients
- Do a significance test for a specific comparison
There are many occasions on which the comparisons
among means are more complicated than simply comparing one mean
with another. This section shows how to test these more complex
comparisons. The methods in this section assume that the comparison
among means was decided on before looking at the data. Therefore
these comparisons are called planned comparisons.
A different procedure is necessary for unplanned
comparisons.
In case of four means, we can compute the difference
between the "success" mean and the "failure"
mean by multiplying each "success" mean by 0.5, each
failure mean by -0.5 and adding the results. We call the value
obtained "L" for "linear combination."
Now, the question is whether the value of L is significantly different
from 0. The general formula for L is:
where ci is the ith coefficient
and Mi is the ith mean. The formula for
testing L for significance is shown below
MSE is the mean of the variances.
The value of n is the number of subjects in each
group.
We need to know the degrees for freedom in order
to compute the probability value. The degrees of freedom is
df = N - k
where N is the total number of subjects and k
is the number of groups.
Online
Calculator: t distribution
In a later chapter on Analysis of Variance, you
will see that comparisons such as this are testing what is called
an interaction. In general, there
is an interaction when the effect of one variable differs as
a function of the level of another variable.
Multiple Comparisons
The more comparisons you make, the greater your
chance of a Type I error. It is useful to distinguish between
two error rates: (1) the per-comparison error
rate and (2) the familywise error rate.
The per-comparison error rate is the probability of a Type I error
for a particular comparison. The familywise error rate is the
probability of making one or more Type I error in a family or
set of comparisons. In the attribution experiment discussed above,
we computed two comparisons. If we use the 0.05 level for each
comparison, then the per-comparison rate is simply 0.05. The family-wise
rate can be complex. Fortunately, there is a simple approximation
that is fairly accurate when the number of comparisons is small.
Define α as the per-comparison error rate and c as the number
of comparisons, the following inequality always holds true for
the familywise error rate (FW) can be approximated as:
FW ≤ cα
This inequality is called the Bonferroni
inequality.
The Bonferroni inequality can be used to control
the familywise error rate as follows: If you want to the familywise
error rate to be α, you use α/c as the per-comparison
error rate. This correction, called the Bonferroni
correction, will generally result in a family wise error
rate less than α.
Should the familywise error rate be controlled?
Unfortunately, there is no clear-cut answer to this question.
The disadvantage of controlling the familywise error rate is that
it makes it more difficult to obtain a significant result for
any given comparison: The more comparisons you do, the lower the
per-comparison rate must be and therefore the harder it is to
reach significance. That is, the power is lower when you control
the familywise error rate. The advantage is that you have a lower
chance of making a Type I error.
One consideration is the definition of a family
of comparisons. Let's say you conducted a study in which you
were interested in whether there was a difference between male
and female babies in the age at which they started crawling.
After you finished analyzing the data, a colleague of yours
had a totally different research question: Do babies who are
born in the winter differ from those born in the summer in the
age they start crawling? Should the familywise rate be controlled
or should it be allowed to be greater than 0.05? Our view is
that there is no reason you should be penalized (by lower power)
just because your colleague used the same data to address a
different research question. Therefore, the familywise error
rate need not be controlled. Consider the two comparisons done
on the attribution
example at the beginning of this section: These comparisons
are testing completely different hypotheses. Therefore, controlling
the familywise rate is not necessary.
Orthogonal Comparisons
In the preceding sections, we talked about comparisons
being independent. Independent comparisons are often called orthogonal
comparisons. There is a simple test to determine whether
two comparisons are orthogonal: If the sum of the products of
the coefficients is 0, then the comparisons are orthogonal.
|