- Complete pair-wise comparisons for ANOVA
- Calculate a confidence interval and p-value for pair-wise comparisons and explain what it means
Type I Error
Recall that sometimes, due to chance, the result of the hypothesis test does not align with reality. If we reject a correct null hypothesis, we have made a type I error.
type I error
The probability of committing a type I error is equal to the significance level: [latex]P(\text{Type I Error}) = \alpha[/latex].
family-wise error rate
Suppose we perform [latex]m[/latex] independent hypothesis tests.
The probability of making a type I error (at least one false rejection) is: [latex]1-(1-\alpha)^m[/latex].
In our example, we have six comparisons, so the probability of committing a type I error is: [latex]1-(1-0.05)^6 = 0.265 = 26.5\%[/latex]. This is likely too high and definitely not [latex]0.05[/latex]. To avoid this problem, we need a method to maintain an overall level of significance even when several tests are performed.We call this the family-wise error rate.
The family-wise error rate is defined as the probability of rejecting at least one of the true null hypotheses.