Interpreting Significant Results

Prerequisites
Introduction to Hypothesis Testing, Statistical Significance, Type I and II Errors, One and Two-Tailed Tests

Learning Objectives

  1. Discuss whether rejection of the null hypothesis should be an all-or-none proposition
  2. State the value of a significance test when it is extremely likely that the null hypothesis of no difference is false even before doing the experiment.

When a probability value is below the α level, the effect is statistically significant and the null hypothesis is rejected. However, not all statistically significant effects should be treated the same way. For example, you should have less confidence that the null hypothesis is false if p = 0.049 than p = 0.003. Thus, rejecting the null hypothesis is not an all-or-none proposition.

If the null hypothesis is rejected, then the alternative to the null hypothesis (called the alternative hypothesis) is accepted.

There are many situations in which it is very unlikely two conditions will have exactly the same population means. For example, it is practically impossible that aspirin and acetaminophen provide exactly the same degree of pain relief. Therefore, even before an experiment comparing their effectiveness is conducted, the researcher knows that the null hypothesis of exactly no difference is false. However, the researcher does not know which drug offers more relief. If a test of the difference is significant, then the direction of the difference is established. This point is also made in the section on the relationship between confidence intervals and significance tests.