Question

Suppose that 90% of proposed treatments in a particular disease are ineffective and 10% are effective. If you run a set of randomized trials that have 80% power for the effective treatments and 5% Type I error rate for the ineffective ones, what is the probability that a treatment with a statistically significant result is truly effective?

Answer 1

We are given here that:

P( ineffective ) = 0.9 and P( effective ) = 0.1

Power of test is 80%, therefore, we have here:
P( + | effective ) = 0.8 and therefore P( - | effective ) = 0.2

5% Type I error rate for the ineffective ones here is given as:
P( + | ineffective ) = 0.05 and therefore P( - | ineffective ) = 0.95

Using law of total probability, we have here:

P(+) = P( + | effective )P( effective ) + P( + | ineffective )P( ineffective )

P(+) = 0.8*0.1 + 0.05*0.9 = 0.125

Now using bayes theorem, given that the test is positive, the treatment is really positive is computed here as:

P( effective | +) = P(+ | effective)P(effective) / P(+)

P( effective | +) = 0.8*0.1/ 0.125 = 0.64

Therefore 0.64 is the required probability here.