Question

NHST is a dominant method for testing theories using statistics (Field, 2017). Field mentions three common misconceptions about what a statically significant result allows you to conclude. Describe these three misconceptions and discuss a situation in which each of these misconceptions might lead to false results.

Answer 1

First, both H0 and H1 have some usually unknown pre-study or “prior” probabilities, pr(H0) and pr(H1). Nevertheless, these probabilities may be approximated through extensive substantive knowledge. For example, we may know about a single published study claiming to demonstrate H1 by showing a difference between appropriate experimental conditions. However, in conferences we may have also heard about 9 highly powered but failed replication attempts very similar to the original study. In this case we may assume that the odds of H0:H1 are 9:1, that is, pr(H1) is 1/10. Of course, these pre-study odds are usually hard to judge unless we demand to see our colleagues' “null results” hidden in their drawers because of the practice of not publishing negative findings. Current scientific practices appreciate the single published “positive” study more than the 9 unpublished negative ones perhaps because NHST logic only allows for rejecting H0 but does not allow for accepting it and because researchers erroneously often think that the single published positive study has a very small, acceptable error rate of providing false positive statistically significant results which equals α, or the p-value. So, they often spuriously assume that the negative studies somehow lacked the sensitivity to show an effect while the single positive study is perceived as a well-executed sensitive experiment delivering a “conclusive” verdict rather than being a “lucky” false positive (Bakan, 1966). (See a note on pilot studies in Serious Underestimation of the Proportion of False Positive Findings in NHST).

NHST completely neglects the above mentioned pre-study information and exclusively deals with rows 2–4 of Table ​Table1.1. NHST computes the one or two-tailed p-value for a particular data set assuming that H0 is true. Additionally, NHST logic takes long-run error probabilities (α and β) into account conditional on H0 and H1. These long-run probabilities are represented in typical 2 × 2 NHST contingency tables but note that β is usually unknown in real studies.

As we have seen, NHST never computes the probability of H0 and H1 being true or false, all we have is a decision mechanism hoping for the best individual decision in view of long-run Type I and Type II error expectations. Nevertheless, following the repeated testing logic of the NHST framework, for many experiments we can denote the long-run probability of H0 being true given a statistically significant result as False Report Probability (FRP), and the long-run probability of H1 being true given a statistically significant result as True Report Probability (TRP). FRP and TRP are represented in row 5 of Table ​Table11 and it is important to see that they refer to completely different conditional probabilities than the p-value.

Table 1 "pr" stands for probability True null effect True positive (Ho) effect (H1) Pre-experiment Long run of experiments pr(Ho) pr(H1) probability of Ho and Single experiment The conditional p-value probability of having this data or more extreme data given that Ho is true The conditional Long run of Alpha level (a) Power 1-B probability of having experiments Type I error a significant test result True Positive Hit False Positive False Alarm given that Ho or H1 are true The conditional Long run of 1-a- experiments Confidence β-1-power probability of not having a significant TypeII error False Negative level True test result given that Negative Correct Miss Rejection Ho or H1 are true Post-experiment Long run of FRP TRP probability of Ho and experiments pr(Ho significant pr(H1 1significant H1 given a significant result) result) test result