Interpreting Non-Significant Results

Prerequisites
Introduction to Hypothesis Testing, Significance Testing , Type I and II Errors

Learning Objectives

1. State what it means to accept the null hypothesis
2. Explain why the null hypothesis should not be accepted
3. Describe how a non-significant result can increase confidence that the null hypothesis is false
4. Discuss the problems of affirming a negative conclusion

When a significance test results in a high probability value, it means that the data provide no evidence that the null hypothesis is false. However, the high probability value is not evidence that the null hypothesis is true.

Concluding that the null hypothesis is true is called accepting the null hypothesis. To do so is a serious error.

Binomial Calculator
Further argument for not accepting the null hypothesis

Do not accept the null hypothesis when you do not reject it.

So how should the non-significant result be interpreted? What if I claimed to have been Socrates in an earlier life? Since I have no evidence for this claim, I would have great difficulty convincing anyone that it is true. However, no one would be able to prove definitively that I was not.

Although there is never a statistical basis for concluding that an effect is exactly zero, a statistical analysis can demonstrate that an effect is at most small. This is done by computing a confidence interval. If all effect sizes in the interval are small, then it can be concluded that the effect is small.