Is it possible for a result to be statistically nonsignificant but substantively significant?

It is quite possible, and unfortunately quite common, for a result to be statistically significant and trivial. It is also possible for a result to be statistically nonsignificant and important.

Consider the case of a new drug that researchers hope will cure Alzheimer’s disease (Kirk 1996). They set up a trial study involving two groups each with 6 patients. One group receives the experimental treatment while the other receives a placebo. At the end of the trial they notice a 13 point improvement in the IQ of the treated group and no improvement in the control group. The drug seems to have an effect. However, the t statistic is statistically nonsignificant. The results could be a fluke. What to do?

Which of the following choices makes more sense to you:

(a) abandon the study – the result is statistically nonsignificant so the drug is ineffective
(b) conduct a larger study – a 13 point improvement seems promising

“The primary product of a research inquiry is one or more measures of effect size, not p values.”
~ Jacob Cohen

How to manuals

“Statistical significance is the least interesting thing about the results. You should describe the results in terms of measures of magnitude – not just, does a treatment affect people, but how much does it affect them.”
~ Gene Glass