Clinical Versus Statistical Significance

Partido, B.B.

There are two types of significance used to interpret research studies – statistical significance and clinical significance. They are not the same thing. One answers the question, “Are the statistical results due to random chance?” (statistical significance) and the other answers the question, “So what? Will the results matter to our patients?” (clinical significance).

Statistical Significance

Statistical significance is when events occur or differences exist not due to chance. As long as the research result is less than the alpha level set before the study commenced, researchers proclaim a study finding to be statistically significant.

Statistical significance is a function of sample size.  If you have a large enough sample size, almost anything can be found to be statistically significant!  Research costs time, effort, and money. Researchers are trying to show relationships between or among their variables of interest – but to do that they need an adequate sample of representative subjects. If the sample size is not large enough, random error is increased, and the results may not show significant differences (even if the intervention really works better than the SOC) because there are not enough subjects to show that difference. A small sample size is a major reason for making this Type II error.  You can also have studies that have too much power.   The problem with very large sample sizes is that very small, statistically significant differences between the research groups can be found. These statistically significant results may not necessarily be clinically significant, though.

Points to understand about statistical significance:

• Calculating statistical significance: Researchers enter data into statistical software (i.e. SPSS), run statistical analyses, and determine statistical significance based on the p-value. If p<.05, there is statistical significance and of p>.05, there is no statistical significance.
• Null hypothesis: The null hypothesis is always assumed to be true. In most studies, the researcher is trying to reject or disprove the null hypothesis that their variables of interest have no relationship. By rejecting or disproving the null, they are able to accept the alternative hypothesis and state their variable is statistically significant.
• Whenever you read a study remember that there may always be differences between the groups .  The most common reason for nonsignificant findings is not having a large enough sample size to find a difference if one really exists!

Clinical Significance

Clinical significance is sometimes called clinical importance or practical importance or how meaningful the results would be to the patient? Determining clinical significance is important to healthcare providers who prescribe patient care treatments. Research findings with very large or very small treatment effect sizes are easier to interpret than those in the middle. Most providers would change their practice based on large, statistically significant treatment effects or continue with the status quo for small or trivial treatment effects, even if the effect was statistically significant.

Confidence intervals are starting to be reported over p-values in the nursing and medical literature because they are more helpful for clinical decision making than p-values. The confidence interval signifies a range wherein the true population parameter lies with 90%, 95%, or 99% confidence.  We can use the range of the reported confidence interval to help us determine clinical significance.

Conclusion

Statistical significance tells us how likely a research result is a chance finding based on the researcher’s predetermined significance level.  Many factors impact statistical power.

Very small differences between the groups tested can be found to be statistically significant if you have a very large sample. The researchers can say their intervention made a difference – because they are able to reject the null hypothesis of no difference.

Research findings may not be important enough to fundamentally change a provider’s prescribing practice or treatment choice, even if found to be statistically significant. Clinical significance tells us how effective or meaningful the research finding might be to patients.

The determination of clinical significance is a subjective decision. The decision will depend on which disease process or condition is being studied, how many people are affected by the condition, etc. For some conditions, very small changes could have a large impact on symptoms, disease burden, or quality of life.