All Pairwise Comparisons Among Means

Prerequisites
Difference Between Two Means (Independent Groups)

Learning Objectives

  1. Define pairwise comparison
  2. Describe the problem with doing t tests among all pairs of means
  3. Calculate the Tukey hsd test
  4. Explain why Tukey test should not necessarily be considered a follow-up test

Many experiments are designed to compare more than two conditions. As the number of tests increases, the probability that one or more of the tests will result in a Type I error increases.

The Type I error rate can be controlled using a test called the Tukey Honestly Significant Difference test or Tukey HSD for short. The Tukey HSD is based on a variation of the t distribution that takes into account the number of means being compared. This distribution is called the studentized range distribution.


The Tukey HSD Test is computed as follows:

  1. Compute the means and variances of each group.
  2. Compute MSE which is simply the mean of the variances.

  3. Compute:



    for each pair of means where Mi is one mean, Mj is the other mean, and n is the number of scores in each group.

  4. Compute p for each comparison using the Studentized Range Calculator. The degrees of freedom is equal to the total number of observations minus the number of means.

The assumptions of the Tukey test are essentially the same as for an independent-groups t test: normality, homogeneity of variance, and independent observations. The test is quite robust to violations of normality. Violating homogeneity of variance can be more problematical than in the two-sample case since the MSE is based on data from all groups. The assumption of independence of observations is important and should not be violated.