Supplementary Written Evidence Submitted by Professor Marcus Munafò

(RRE0096)

There was mention of publication bias and the failure to report null results.

This is an established problem, but the extent to which it is compounded by other biases is perhaps less well appreciated. The figure attached here illustrates this.

These data are from clinical trials of antidepressants, the results of which were registered with the FDA (notably, without that transparency, this analysis would not have been possible).

About half of those trials indicated a benefit of the treatment over the comparator / placebo. However, whilst nearly all of the trials indicating benefit were published, only about half of those indicating no difference were published (publication bias).

What happens after that is most interesting. Of the trials indicating no difference, some were “converted” into trials indicating a benefit through outcome reporting bias (i.e., switching the outcome that is the main focus of the analysis, post hoc).

Then, those trials that are still indicating no difference are “spun” – for example, results where the p-value is close to the (arbitrary) 0.05 threshold (e.g., 0.059) are given a positive gloss (“some evidence of efficacy” and so on).

Finally, there is a strong bias in terms of which findings are cited. Positive findings (those studies that indicate a benefit) are much more likely to be cited than those that indicate no difference.

The net result is that the visible evidence to a casual observer of the scientific literature (including researchers themselves without the time or resources to conduct a systematic review) is radically different to the underlying reality.

Of note, I don’t think this is unique to clinical trials – it is simply (as noted above) that the transparency mandated by regulators makes this analysis possible in the first place.

Taken together, this illustrates (I think) the negative consequences of a focus on discovery and finding something, rather than simply getting the correct answer to your research question, and reporting that straightforwardly.

(December 2021)