Misconceptions

Prerequisites
Introduction to Hypothesis Testing, Statistical Significance, Type I and II Errors

Learning Objectives

1. State why the probability value is not the probability the null hypothesis is false
2. Explain why a low probability value does not necessarily mean there is a large effect
3. Explain why a non-significant outcome does not mean the null hypothesis is probably true

Misconceptions about significance testing are common. This section lists three important ones.

1. Misconception: The probability value is the probability that the null hypothesis is false.
   
   Proper interpretation: The probability value is the probability of a result as extreme or more extreme given that the null hypothesis is true. It is the probability of the data given the null hypothesis. It is not the probability that the null hypothesis is false.
   
   
2. Misconception: A low probability value indicates a large effect.
   
   Proper interpretation: A low probability value indicates that the sample outcome (or one more extreme) would be very unlikely if the null hypothesis were true. A low probability value can occur with small effect sizes, particularly if the sample size is large.
   
   
3. Misconception: A non-significant outcome means that the null hypothesis is probably true.
   
   Proper interpretation: A non-significant outcome means that the data do not conclusively demonstrate that the null hypothesis is false.