Significance Testing and Confidence Intervals
Prerequisites
Confidence
Intervals, Introduction to Hypothesis Testing, Significance
Testing
Learning Objectives
 Determine from a confidence interval whether a test is significant
 Explain why a confidence interval makes clear that one should not accept
the null hypothesis
There is a close relationship between
confidence intervals and significance tests. Specifically, if
a statistic is significantly different from 0 at the 0.05 level
then the 95% confidence interval will not contain 0. All values
in the confidence interval are plausible values for the parameter
whereas values outside the interval are rejected as plausible
values for the parameter. In the Physicians'
Reactions case
study, the 95% confidence interval for the difference
between means extends from 2.00 to 11.26. Therefore, any value
lower than 2.00 or higher than 11.26 is rejected as a plausible
value for the population difference between means. Since zero
is lower than 2.00, it is rejected as a plausible value and a
test of the null hypothesis that there is no difference between
means is significant. It turns out that the p value
is 0.0057. There is a similar relationship between the 99% confidence
interval and Significance at the 0.01 level.
Whenever an effect is significant, all values
in the confidence interval will be on the same side of zero (either
all positive or all negative). Therefore, a significant finding
allows the researcher to specify the direction of the effect.
There are many situations in which it is very unlikely two conditions
will have exactly the same population means. For example, it is
practically impossible that aspirin and acetaminophen provide
exactly the same degree of pain relief. Therefore, even before
an experiment comparing their effectiveness is conducted, the
researcher knows that the null hypothesis of exactly no difference
is false. However, the researcher does not know which drug offers
more relief. If a test of the difference is significant, then
the direction of the difference is established because the values
in the confidence interval are either all positive or all negative.
If the 95% confidence interval contains zero (more
precisely, the parameter value specified in the null hypothesis),
then the effect will not be significant at the 0.05 level. Looking
at nonsignificant effects in terms of confidence intervals makes
clear why the null hypothesis should not be accepted when it is
not rejected: Every value in the confidence interval is a plausible
value of the parameter. Since zero is in the interval, it cannot
be rejected. However, there is an infinite number of values in
the interval (assuming continuous measurement), and none of them
can be rejected either.
