Question

correctly explained the general rationale behind a post-hoc analysis. It's used to determine which specific pairings led to the main effect, if one exists. That's one of the issues with an ANOVA - if you have a main effect, you're not sure which comparisons led to the result. Therefore, while an ANOVA is powerful and allows you to compare multiple groups, it doesn't tell you which comparisons are significant. Another issue with ANOVAs is that you often have three groups that are comprised of different subjects. For example, you may have three groups of eight subjects. You wouldn't really know, therefore, if an effect was due to the manipulation or if it was due to the subjects themselves. After all, how do you know that group one didn't consist of people who were significantly different from the others to begin with, regardless of the manipulation? Based on this confound, how might you try to change an experimental set-up so that you wouldn't have to worry about the subjects being different?

Answer 1

To reduce the variation between the subjects, we can use a randomised block design ANOVA which will help reduce the confounding. Blocks can be formed on the known factors which could affect the outcome. We could assign homogenous units to a particular block, for example, people above age 50 will fall in one block and people below 50 will fall in another block. After forming the blocks, treatments could be assigned to the observations randomly. Hence, this type of experimental setup will reduce confounding.