that at least one pair of samples differs. Let’s say you are interested in treatments A, B and C in an experiment, and will conduct hypothesis tests for differences between A and B, A and C, and B and C, each with a significance level of alpha = 0.05.
Each time you do an independent test, the individual Type I error is 0.05, so the probability of not making an error is 0.95. The probability of not making such an error on three consecutive tests is 0.95 * 0.95 * 0.95 = 0.8574. So the Type I error rate overall, or the family-wise error rate, is 1-0.8574 = 0.1426. This is considerably more than the individual rate, and it increases as the number of comparisons increase.
If a is the probability of individual comparison-wise type I error, then the probability aFW of family-wise type I error is usually calculated as follows:
|
where C is the total number of individual comparisons made. You can see that to keep the overall family-wise error rate at 0.05 you will need to substantially reduce the alpha level for each test.