10.8. Crucial Error Rates When the Null Hypothesis Is Likely to Be True

Suppose "Dr. Art Ary" is planning a small trial to obtain some sound human data on a novel biologic called nissenex, which could reduce percent atheroma volume in patients with atherosclerosis. Even Dr. Ary is skeptical about nissenex, however, giving it a 2% chance of being truly effective: γ = 0.02. Using a reasonable characterization of the infinite data set presuming nissenex is really efficacious, the power for the key hypothesis test is judged to be 0.83 at α = 0.05 and N = 120. Accordingly, the crucial error rates are α^* = 0.75 and β^* = 0.004. Thus, three out of four significant tests will be misleading.

Does this high α^* value imply that the study should not be run? No. If this trial yields p ≤ 0.05, it would push this line of research forward to a 1 − 0.75 = 0.25 chance that nissenex is effective, a major shift from the prior γ = 0.02. If p > 0.05, then there is a 1 − 0.004 = 0.996 chance that nissenex has null or near-null efficacy, perhaps solidifying Dr. Ary's initial skepticism. Thus, either outcome will help Dr. Ary decide whether to continue with further studies. He also considers using α = 0.20, which gives 0.95 power and makes α^* = 0.91 and β^* = 0.001.

1 − α^* = 0.09 is considerably weaker than the 0.25 computed for α = 0.05, and there is little practical difference in β^* values (0.004 versus 0.001). Thus, Dr. Ary will use α = 0.05, but he understands that given his current prior skepticism regarding the efficacy of nissenex in treating atherosclerosis, not even a p ≤ 0.05 outcome will sway him to prematurely publicly taut nissenex as effective. It will, of course, encourage him and his sponsors to design and carry out a more confirmatory study. This is prudent scientific practice. If everyone followed it, the scientific literature would not be cluttered with "significant" findings that fail to replicate in further, larger studies and meta-analyses, provided that any such work takes place (Ioannidis, 2005a, b).