Written evidence submitted by Cardiff University (COM0067)

Submitted by the School of Psychology, Cardiff University, based on evidence gathered by a collaborative research project between the School of Psychology and the School of Journalism, Media and Cultural Studies.

We address the following highlighted issue: 'Any further steps needed by the media and broadcasters to improve the quality, accessibility and balance of their science coverage; and science coverage in broadcasters' programme-making.'

Summary

• Mainstream media are still the primary source of health and science information for most adults 1, and the effect of health news stories on behaviour and attitudes is well established, for example in cases such as MMR vaccine uptake 2. Thus exaggerations and simplifications in news can have widespread ramifications for public health 3.
• Much modern health and science news arises in response to press releases issued by leading science and medical journals 4. In two major studies we found a strong association between exaggerations in news and those already present in these journal press releases.
• But surprisingly these exaggerations do not appear to enhance the likelihood or volume of news coverage.
• We also found that caveats (expressions about limitations) - widely considered something to avoid in press releases and currently very rare - did not reduce news coverage and were in fact often reiterated in news. Thus the fears that they would be ignored or make a press release unappealing seem unfounded.
• We conclude that there is scope to reduce exaggerations and introduce appropriate caveats without harming news interest. We are currently testing this with a field trial working with press officers in universities, funders and journals.
• We will then help press officer organisations (STEMPRA, Science Media Centre) update practice guidelines and training for press officers.

1. Brief introduction about us and our reason for submitting evidence

1.1. In 2012 we formed a collaboration between experimental psychologists, health psychologists and journalism academics to study exaggerations and misleading claims in health and science news. We focus on areas relevant for human health – biomedical and social sciences. We have carried the two largest studies ever performed in this field to build a better evidence-base of where things go right and where things go wrong in the chain between published peer-reviewed studies, press releases and news reports. We followed this up with laboratory and online research on how readers understand or misunderstand different phrases. We are now midway through a trial collaborating with press officers to study actual press releases in the real world. see http://sites.cardiff.ac.uk/insciout/

1.2. Although most individual exaggerations might be considered negligible for public health, occasional exaggerations have very large effects (MMR and autism; statins etc). In addition the cummulative effect of small exaggerations could have a large impact on public health (e.g. obesity) given that over 1000 new health stories occur each year, with up to 40 million readers gaining their main health information through mainstream media. Thus it is generally accepted by press officers that news arising from their work could have large public health impact (the evidence for this comes partly from our own research on the similarities between news stories and the press releases they arise from).

2. Key findings relevant for this call.

2.1. In our first study we assessed the news generated by press releases from all Russell Group universities throughout 2011, as well as assessing the press releases themselves and the peer-reviewed papers they were based on or inspired by. We found that most exaggerations in health-related science news were already present in press releases (Sumner, P. et al., 2014,.BMJ, 349 (10.1136/bmj.g7015).  We also found, contrary to popular belief, that exaggerations were not associated with more news coverage. Together these results offered the opportunity that improving press releases would materially improve news coverage of health-related topics. Note on Quality: the BMJ is a field-leading cross disciplinary medical journal. Our paper reporting this work has already received 44 citations in the first year or so, and received widespread attention beyond academia. It was 52nd in the Altmetric top 100 list in 2015 (2nd for BMJ).

2.2. We applied the same approach to press releases from leading journals. We found that exaggeration rates tended to be lower in journal press releases compared to universities, but the association with news exaggerations was just as strong. Moreover, we replicated the result that exaggerations did not appear to enhance news coverage. By combining the two sets of data together, we have a database of over 1000 press releases and associated news coverage. We then used this to assess whether two common assumptions are true: that caveats in press releases will harm news interest, and if there still is news, the caveat will be ignored by journalists anyway. We found neither of these to be true - half the news stories arising from press releases with caveats also reported the caveat, compared to around 5% of spontaneous caveats in news. This evidence implies that it is possible to effectively include caveats in press releases. The work is in submission as:  Sumner et al. (in submission) Exaggerations and caveats in news and press releases about health and science. The manuscript is attached below for reference.

2.3. Importantly, our evidence does not suggest simply blaming press officers. Academics often supply the quotes and material that contain exaggerations, and may exaggerations may be inadvertent. The whole system is under pressure from the competition model between universities to attract prestige, funding and students.

2.4. To assess balance, we analysed quotes in news stories. 89% had quotes; 72% of these stories contained quotes identical to those included in the press release; 44% had alternative or additional quotes from the authors of the associated peer-reviewed journal article; 5% contained quotes identical to text in the journal article; 8% had quotes from other sources (e.g. funders) related to the research; 30% had quotes from independent scientists or 'experts'. This last figure is the key one: 70% of stories had quotes only from sources associated with the research, with no quotes from independent sources. This almost certainly occurs due to time pressures on journalists.

2.5. In parallel with this work, we have been assessing whether current guidance on how best to phrase the most common type of research findings (correlational findings) is optimal. We find that it is not, because people do not interpret the phrases the way the guidance assumes. Rather, readers (whether statistically trained or not) are much more sensitive to words such as 'might' or 'may' than they are to the difference between, for example, 'associated with a decrease' and 'cuts'  (the former only claims a correlation, while the latter claims a cause). Guidelines should therefore focus on the use of phrases that people are sensitive to, rather than splitting hairs between phrases that, while logically distinct, people treat as equivalent anyway. This work is in submission as:  Adams et al. (in submission) Perceived strength of correlational and causal phrases: a guide to wording claims about health-related research. Journal of Experimental Psychology, Applied (a highly respected journal published by the American Psychological Association).

3. Relevant data that will be available in 2017

3.1. Our current project is running a real-life randomised controlled trial (RCT) with collaborating press officers to measure how changes to real press releases affect news (trial registration at http://www.isrctn.com/ISRCTN10492618). This experiment is modelled on a drug trial, where we are inserting specific words and phrases into real press releases from several universities and journals before they are released, and monitoring the content of news arising (comparing 'active' vs. 'placebo' conditions). This will provide direct causal evidence for whether and to what extent such press release content affects news coverage. The trial runs from October 2015 to December 2016, after which our data will available on request

3.2. This work is funded by an ESRC transformative grant (2014-16), and by various small grants and assistance from Cardiff University. We work with the Science Media Centre, STEMPRA, MRC, BMJ, Biomed Central, several universities and an advisory board comprising: Rob Dawson (BBSRC), Fiona Fox (Science Media Centre), Ruth Francis (Biomed Central), James Gallagher (BBC), Helen Jamison (Wellcome Trust), Alok Jha (The Guardian), Carmel Turner (MRC), Bob Ward (Grantham Institute) and Ed Yong (National Geographic).

4. Conclusions and Recommendations

For the data we have gathered so far, our conclusions and recommendations to the committee are:

• Be data driven as far as is possible. Widely held assumptions do not always match the data gathered in projects such as ours (for example the assumptions about having to avoid caveats to be newsworthy)
• One clear path to improving media and broadcaster coverage is to start with their sources: chiefly avoiding hype and including more caveats/ information about strength of evidence in press releases from academic institutions and journals.
• Mainstream newspaper stories are often draw from only one source (probably due to time restrictions). The Science Media Centre can supply independent quotes from scientists, and put journalists in direct contact with other scientists.
• Exaggeration in university press releases is likely driven by competition between universities for publicity, prestige, funding and student applications.  It is not in scope for this committee to influence this, but simply be aware of this unintended outcome of introducing a competition model between universities.

1.              Castell, S. et al. Public Attitudes to Science. Ipsos Mori report for Department for Business Innovation and Skills 1–202 (2015).

2.              Boyce, T. Health, Risk and News: The MMR Vaccine and the Media. (Peter Lang, 2007).

3.              Leveson, B. An Inquiry Into the Culture, Practices and Ethics of the Press. (pp 22, 80, 690-91, 1803, 2012).

4.              Williams, A. & Clifford, S. Mapping the Field: Specialist science news journalism in the UK national media. Science and the Media Expert Group, Department of Business, Innovation and Skills. (2009).

Exaggerations and caveats in press releases and health-related science news

Petroc Sumner1,2, Solveiga Vivian-Griffiths1,2, Jacky Boivin2, Andrew Williams3, Lewis Bott2, Rachel Adams2, Christos A. Venetis4, Leanne Whelan2, Bethan Hughes2, Christopher D. Chambers1,2

1Cardiff University Brain Research Imaging Centre (CUBRIC), School of Psychology, Cardiff University, CF10 3AT, UK.

2School of Psychology, Cardiff University, CF10 3AT, UK.

3School of Journalism, Media & Cultural Studies, Cardiff University, CF10 3NB, UK.

4School of Women’s and Children’s Health, University of New South Wales (UNSW) and Graduate School of Medicine, Faculty of Science, Medicine and Health, University of Wollongong.

Acknowledgments. We thank our professional advisory group for helpful input into this research. This study was supported by grants from the ESRC (ES/M000664/1) and Cardiff University.

April 2016

Abstract

Background. Exaggerated or simplistic news is often blamed for adversely influencing public health. However, recent findings suggested many exaggerations were already present in university press releases, which scientists approve. Surprisingly, these exaggerations were not associated with more news coverage. Here we test whether these two controversial results also arise in press releases from prominent science and medical journals. We then investigate the influence of mitigating caveats in press releases, to test assumptions that caveats harm news interest or are ignored.

Methods and Findings Using quantitative content analysis, we analyzed press releases (N=534) on biomedical and health-related science issued by leading peer-reviewed journals. We similarly analysed the associated peer-reviewed papers (N=534) and news stories (N=582). Main outcome measures were advice to readers and causal statements drawn from correlational research. Exaggerations in press releases predicted exaggerations in news (odds ratios 2.4 and 10.9, 95% CIs 1.3 to 4.5 and 3.9 to 30.1) but were not associated with increased news coverage, consistent with previous findings. Combining datasets from universities and journals (996 press releases, 1250 news), we found that when caveats appeared in press releases there was no reduction in journalistic uptake, but there was a clear increase in caveats in news (odds ratios 9.6 and 9.5 for caveats for advice and causal claims, CIs 4.1 to 24.3 and 6.0 to 15.2). The main study limitation is its retrospective correlational nature.

Conclusions. For health and science news directly inspired by press releases, the main source of both exaggerations and caveats appears to be the press release itself. However we find no evidence that exaggerations increase, or caveats decrease, the likelihood of news coverage. These findings should be encouraging for press officers and scientists who wish to minimise exaggeration and include caveats in their press releases.

Introduction

Press releases have long been used as 'information subsidies' to facilitate science news(1), and have become the dominant link between academia and the media (2,3). As structural changes to the news industry’s revenues and profitability have put pressure on staffing levels, journalists are expected to produce more copy in less time. Due to these economic contexts, journalists in general, and science/medical journalists in particular, routinely use the content of press releases in their news stories, often without sufficient checks and research to independently evaluate the claims (4-9). Since the majority of people, at least in western populations, use news media as their main source for up-to-date science and health information (10), science press releases could have a large cumulative effect on public perceptions of science and on health-related behaviours (8,11-16).

Press releases routinely condense complex scientific findings and theories into digestible packets, providing an efficient means for disseminating new science of interest to publics in a helpful form for journalists (17). Information and quotes in a press release are highly likely to be included in related news (18). However, because of this synergy, any exaggerations, message creep or misinformation in press releases may also be reproduced in the news (19,20). On the other hand, caveats to strong or simplistic claims in press releases are rarely present (20), presumably because they are assumed to hamper a clear news message and therefore to harm news interest and/or be excluded from news anyway.

In a study of health-related news and press releases based on research in 20 major UK universities, we previously found that common types of subtle exaggeration frequently appearing in news were highly associated with what was written in the corresponding press release (20). We analysed advice given to readers, causal claims based on observational data and claims about humans based on non-human research. We found that a third or more press releases contained stronger advice, causal statements or human claims than any found in the peer-reviewed journal article they were based on. Moreover, the existence of these forms of exaggeration in press releases strongly predicted their presence within news (odds ratios between 6.5 and 56). Counter to common assumption, however, exaggerated press releases were not more likely to attract news. In other words, the main source of news exaggerations appeared to be not the journalists themselves, but the text of academic press releases. But the premise upon which academics might be tempted to subtly exaggerate in their press releases - that it would encourage more news uptake - appeared to be false.

While university press releases have a clearly defined role in facilitating the communication of science, they are just one source of science news. Previous research has indicated that press releases distributed by academic journals are potentially more important(7,9). If the associations between news and journal press releases are similar to those with university press releases, this will show the provocative results of our previous study hold in distinct population. If the results show a different pattern from those with university press releases, then any effort to address how exaggerations appear in health and science news will have to differ for the different pathways.

There are several reasons why university press releases and journal press releases might differ with regard to subtle exaggerations and their transmission into news: in universities and academic institutions, press officers routinely involve academic authors in the development of press releases, whereas not all journals do this (partly due to volume and time pressures), and academic authors may have less expectation that they will be heavily involved (the journal normally owns the copyright). Relatedly, journal press offices may have stronger reliance on the text of the peer-reviewed paper, and may also have access to the academic editor for the paper. In the case of non-human studies, a journal may have less reason to worry about drawing attention to animal research facilities as they are unlikely to suffer reputational or material damage from the release of such information in the same way a university might. Other environmental and stylistic differences exist as well, but there is, to date, no directly comparable data to assess rates of exaggeration and news uptake (e.g. (21-23).

The scientific process relies on justifying conclusions, and also considering the limits of each conclusion, and interviews with press officers (17) suggest it should be common to find caveats at the end of press releases, because they are important information for news consumers with illnesses or considering changes to health-related behaviour. But caveats (e.g. 'We still need more research to clarify whether vitamin D directly prevents bowel cancer or if people with higher levels are generally healthier') and justifications (e.g. ' Unlike previous researchers [authors] looked at children from all social backgrounds') appeared in only about 10% of press releases and news in Sumner et al. (2014) (see also (22). It is commonly assumed that caveats would harm news interest or would be excluded by journalists even if included in press releases. In the second part of this study we assess whether these assumptions hold, or whether caveats might in fact be beneficially included in press releases. We similarly analyse the use of justifications in the translation of nuanced scientific findings into clear news stories. These questions require a very large dataset.

In sum, this study first investigates whether exaggerations in science and health news are associated with exaggerations in journal press releases, and whether press releases that were exaggerated attracted more news than those that did not. We analysed the text of press releases from major journals, the text of the news arising from these press releases, and the text in the associated peer-reviewed papers. Secondly, we analyse the use and effects of caveats and justifications by combining new data from journal press releases with previously collected data from university press releases (20).

Methods.

Press releases based on possible relevance to human health, psychology or neuroscience were identified for studies published in the following journals, for the entire year of 2011: Lancet, British Medical Journal (BMJ), Science, Nature, Nature Neuroscience, Nature Immunology, Nature Medicine, and Nature Genetics. The press releases were collected from either publicly available repositories (journal web pages or EurekAlert) or press sites for science journalists (Nature Publishing Group kindly provided us with free access to all their press releases for the purpose of this study). This resulted in 534 relevant press releases with associated peer-reviewed journal articles. News articles (n=582) resulting from these press releases were identified from UK national news media by searching the Nexis database, BBC.co.uk, uk.reuters.com, and by performing a Google search.

The process of data extraction and analysis was identical to that in Sumner et al. (2014), and focussed on advice given, causal claims, and conclusions about humans based on non-human research. Statements were assessed in press releases and news stories, and compared to corresponding measurements in the peer-reviewed journal articles, which were always taken as baseline.

Health advice: Each source was coded for maximum level of advice out of four possible categories: no advice, implicit advice (e.g. 'for adults with a BMI greater than 35 kg/m2...permanent calorie reductions of more than 500 calories per day would be needed...'), explicit advice but not to the reader or general public (e.g. 'clinicians must exercise caution in their use of aflibercept'), or explicit advice to the reader or general public (e.g. 'patients who are at increased risk of cardiovascular events should consider starting statin treatment promptly and continuing it long-term'). The set of journal article, press release and news was included in the analysis if at least one source contained advice that was implicit or explicit; there were 247 such sets, with 411 news stories.

Causal claims: Press releases, news articles and journal articles were coded for the strength of the main statement of findings (according to 7 levels: no statement, statement of no relationship, correlational, ambiguous, conditional causal, can cause, and causal) in either the title and the first two sentences of the news and the press releases, or abstract and discussion in the journal articles. This analysis focused on studies based on correlational cross-sectional and longitudinal designs, and excluded all qualitative, intervention, or simulation designs. In total 164 press releases and associated journal articles and 237 news stories were included in the analysis.

Human inferences from research on non-humans: For each journal article based on a non-human sample (animals, cells, or simulations), the associated press release and news article were coded for whether the findings or conclusions were stated as explicitly non-human, implicitly human or explicitly human. There were 112 journal articles and press releases and 64 news stories. However, we could not do most of the analyses for this type of exaggeration since only one exaggerated press release had associated news. Therefore raw results are in supporting information (S1). 

Caveats and Justifications. Because caveats and justifications tend to rare, in order to analyse them we combined datasets from the journal press releases described above with our previous study of university communications (Sumner et al., 2014), which used identical procedures The presence, or not, of caveats for causal/correlational statements were coded for every press release (N=428); caveats for explicit advice could be coded only where explicit advice appeared (N=188). Presence or absence of justifications for causal statements were coded only if the press release contained an explicit statement of relationship (N=355), while justifications for advice were coded only for those that contained explicit advice (N=188).

Double coding The coding for each journal article, press release, and associated news stories took 3-4 hours per set; the entire dataset required approximately 700 full days of research time before analysis. Within this, we randomly selected 147 press releases (27.5% of all press release sample) and associated 144 news stories (24.7% of all news sample) for independent double-coding (i.e. a different research assistant coded the entire set again), which resulted in concordance rate of 98.3% for the results reported below (κ=0.97). Note that where concordance is high and resource limitations require a direct choice between more double coding or collecting more samples (in this case increasing the N for press releases and news stories), statistically the latter strategy is better for the reliability of results.

Analysis. As in our previous study, exaggeration was defined as claims or advice in press releases or news that were coded at a level above any of the associated statements in the peer-reviewed paper. For example, where the news makes a causal claim but the journal article did not. Thus we are recording what might be termed message creep beyond the journal article text; we are not, in this research, attempting to judge whether the journal article itself contains exaggeration (there would not be universal agreement on what was scientifically accurate; (24) . Therefore our definition of 'exaggeration' means 'exaggeration beyond any that might already exist in the peer-reviewed paper'.

For direct comparison to Sumner et al (2014), we used generalised estimating equations to determine percentages and 95% confidence intervals (CI) for exaggeration rates, while adjusting for the clustering of several news articles per press release (using an exchangeable working correlation). The generalised estimating equations framework was also employed to estimate the association (in odds ratios) between exaggeration in the press release and exaggeration in the news, and the association between caveats or justifications in press releases and news. Note that these analyses included only those journal articles and press releases for which there was at least one news story (and the news could be appropriately coded for the relevant analysis). We compare the characteristics of press releases with and without associated news using bootstrapped 95% CI and standard inferential statistical tests. All coding sheets (N=534 for journal press releases, N=462 for university press releases), full instructions for coding, and data analysis files and programs are available online at https://github.com/SolveigaVG/JournalPROpenData.git and at http://dx.doi.org/10.6084/m9.figshare.903704

Results

1.  Exaggeration rates in press releases

We found that 23% of press releases included more direct or explicit advice than the associated journal article (bootstrapped 95% confidence interval, CI, 17% to 28%). Similarly, 21% of press releases contained exaggerated causal statements from correlational research (CI 15% to 27%). These rates are not negligible, but are all lower than those we previously measured in equivalent analyses for university press releases, which were 40% (CI 33% to 46%) and 33% (CI 26% to 40%), respectively.

2.  Association between press release and news exaggeration

As shown in Figure 1, there was strong association between exaggerations in press releases and news. For advice, 27% of news (CI 22% to 33%) had more direct or explicit advice than the journal article, and the odds of this were 2.4 times higher (odds ratio 2.4, CI 1.3 to 4.5) when the press release also contained exaggerated advice (46%, CI 32% to 60%) than when it did not (25%, CI 19% to 31%). For main statements describing correlational studies, 38% of news stories (CI 27% to 51%) had stronger causal wording than the associated journal article, and the odds of exaggeration were 10.9 times higher (CI 3.9 to 30.1) when press releases contained exaggerated causal claims (80%, CI 63% to 91%) than when they did not (26%, CI 18% to 38%). This pattern is highly similar to that found for university press releases (cf Figure 2 in Sumner et al. 2014)

Table 1. Summary of results for our analyses of advice, primary claims from correlational data ('causal claims') and human inference from non-human studies ('human inference'). The key results are that news uptake does not obviously depend on press release (PR) exaggeration , while  news exaggeration does appear strongly associated with press release exaggeration . Further information, including percentages and 95% CIs, are provided in results sections 1-3 and Figures 3 and 4. 

Figure 1. The proportions of news with exaggerated advice (A), or causal statements from correlational research (B) were higher when the associated press releases (PR) contained such exaggeration (N for Advice, PR=247, news=411; causal claims, PR=164, news=237). Error bars are 95% confidence intervals. See Table 1 for odds ratios.

3.  Effect of exaggeration in press releases on news uptake

There was no evidence that exaggeration in press releases is associated with increased news uptake (Figure 2). For advice, 101/191 (53%) press releases without exaggeration had news articles while only 19/56 (34%) of press releases with exaggeration had news (difference -19%, CI -5% to -34%). For causal claims, 63/129 (49%) of press releases without exaggeration had news compared with 23/35 (66%) of press releases with exaggeration (difference 17%, CI -1% to 35%).

Similarly, of those press releases that had some news, the number of news stories was not greater for press releases with exaggeration. When press releases did not contain exaggerated advice, they were associated with 3.5 news stories per press release, compared to 3.0 news stories per press release that did contain exaggerated advice (CI of difference -1.5 to 0.6). Press releases not containing exaggerated causal claims were associated with 3.1 news stories per press release, compared to 3.0 news stories per press release containing exaggerated causal claims (CI of the difference -1.2 to 1.2). This lack of association between exaggeration and news uptake echos the results for university press releases (cf Figure 3 in Sumner et al., 2014).

A. Advice

B. Statements of relationship

Figure 2. The proportion of PRs that have resulting news articles when the PRs do not contain exaggerations (left bars) compared to PRs that do contain exaggerations (right bars). Error bars are bootstrapped 95% confidence intervals. See Table 1 for odds ratios.

4.  Caveats for advice and causal claims

The use of caveats is of key interest for understanding the difficult translation of nuanced scientific findings into clear news stories. Caveats were rare in university press releases (Sumner et al. 2014), which might reflect the assumption that they hamper a clear message and thus potentially harm news interest, or the belief that they would be ignored by journalists and are therefore a pointless addition to the press release. This rarity has previously limited any systematic analysis. Here, we combined the data for journal press releases with the previously published dataset for university press releases (Sumner et al. 2014), given that the main results reported above were similar across both sources of science news. The overall rate of caveats in press releases were 15% (29/188) for advice and 14% (62/428) for causal statements about correlational research.

5.1. No evidence that caveats reduce news uptake. For advice there was news uptake for 48% (14/29) of press releases with caveats, and similarly for 56% (89/159) of press releases without caveats (CI of the difference: -28% to 12%; see Figure 3A,B). For causal statements there was news uptake for 63% (39/62) of press releases with caveats, compared to 47% (171/366) without caveats (CI of the difference 3% to 29%).  Thus, if anything, caveats for causal statements are associated with greater news uptake, although we cannot say whether this is in any part due to the caveat itself, or additional characteristics of press releases that also include caveats.

5.2. Caveats do get into news. Caveats for advice appeared in 37/264 (14%) of news stories, and caveats for causal statements appeared in 107/607 (18%) of news. This low rate, taken alone, might imply that journalists tend to avoid caveats, but when compared to the caveats in press releases a different picture emerges (Figure 3C,D). The odds for caveated advice were 9.5 times higher (odds ratio 9.5, CI 2.8 to 32) when the press release contained caveats (47%, CI 20% to 74%) than when it did not (8.5%, CI 4.4% to 13%). The odds for caveated causal statements were 7.6 times higher (odds ratio 7.6, CI 4.2 to 14) when the press release contained a caveat (46%, CI 35% to 57%) than when it did not (10%, CI 6.5% to 13%). Thus caveats in press releases do not appear to be routinely ignored by journalists. We also found no association between caveats and exaggeration within press releases or news (Supporting Information S2). In other words, caveats were not systematically used to balance exaggeration, and neither were they systematically associated with an overall cautious (non-exaggerating) approach.

Figure 3. a) News uptake for PRs with and without caveats for explicit advice. b) News uptake for PRs with and without caveats for statements of relationship. c) Association between caveats for explicit advice in the PR and caveats for explicit advice in resulting news articles. d) Association between caveats for statements of relationship in the PR and caveats for statements of relationship in resulting news articles. All error bars are bootstrapped 95% confidence intervals.

5.  Justifications for advice and causal claims

The overall rates of justifications in press releases were 10% (19/188) for advice and 21% (74/355) for causal statements about correlational research.

6.1. Justifications and news uptake. For advice there was news uptake for 74% (14/19) of press releases with justifications, and for 53% (89/169) of press releases without justifications (bootstrapped CI of the difference: -2% to 41%). For causal statements there was news uptake for 43% (32/74) of press releases with justifications, but 54% (151/281) without justifications (CI of the difference -15% to 2%; see Figure 4A,B)).

6.2. Justifications are reported in news. Justifications for advice and causal statements appeared in 41/264 (16%) and 69/538 (13%) of news stories, respectively. The odds for advice were 6.1 times higher (odds ratio 6.1, CI 2.0 to 18) when the press release contained justifications (42%, CI 19% to 66%) than when it did not (11%, CI 5.5% to 16%). The odds for causal statements were 23 times higher (odds ratio 23, 95% CI 11 to 50) when a press release contained justifications (48%, CI 34% to 64%) than when it did not (3.8%, CI 1.8% to 5.8%; Figure 4C,D). Thus, like caveats, justifications appear to be noticed and utilised. We also found that justifications were included more often alongside exaggerated advice, but for causal claims were instead associated with a more cautious approach in which exaggeration is diminished news (Supporting Information S3).

Figure 4. a) News uptake for PRs with and without justifications for explicit advice. b) News uptake for PRs with and without justifications for statements of relationship. c) Association between justifications for explicit advice in the PR and justifications for explicit advice in resulting news articles. d) Association between justifications for statements of relationship in the PR and justifications for statements of relationship in resulting news articles. All error bars are bootstrapped 95% confidence intervals.

Discussion

In summary, we obtained the following key results: 1) Exaggeration rates were lower in our sample of journal press releases compared to the previous study of university press releases; 2) Exaggerations in news were strongly associated with exaggerations in press releases; 3) Exaggerations were not associated with higher likelihood of news or more news stories; 4) Caveats in press releases did not seem to harm news uptake, and strongly raised the likelihood of caveats appearing in associated news pieces; 5) Similarly, justifications in press releases strongly raised the likelihood of similar statements in news, but had no consistent effect on news uptake. Results 2 and 3 generalise (to a new and even larger dataset) the provocative findings of a previous study (Sumner et al. 2014). Result 4 challenges received opinion that explicitly mentioning caveats hampers newsworthiness and/or are likely to be ignored by journalists.

Overall then, we find strong associations between press releases and news for all the types of phrases and sentences analysed (exaggerations, caveats or justifications), but no evidence that such factors systematically influence whether a press release is judged newsworthy. The relationships between press release content and news apply to both the major sources of science news: universities and journals. These results should be encouraging to press release authors (press officers and scientists) who would prefer to take a more nuanced approach but have feared losing news interest.

Note that the types of exaggeration studied here are quite subtle and routine: for example changing a phrase like 'related to' or 'might increase' to a direct causal phrase like 'boosts'. We have no evidence that these small exaggerations (or message creep) represent conscious efforts to hype, and interviews with press officers show they feel a strong obligation to avoid hype (17). Unintended subtle exaggerations may arise for other reasons such as trying to use simpler and more direct language with fewer words per sentence. Moreover, within our data will be cases (especially in some examples of health advice given to readers) where exaggeration as defined here could be considered entirely appropriate rephrasing within the context of the press officers' professional vision (17). In the other direction, our data doubtless include cases where strong statements – statements that would be considered hype by some readers – were not defined as exaggeration because they were already contained in the peer-reviewed journal article, which we employed as the best available baseline. Without a separate objective baseline, or a consensus from experts in each field, it is impossible to assess whether the claims in the journal article were themselves hyped. These inevitable variations and contextual factors in individual cases are why it was important to generate very large dataset – the largest of its kind ever studied – to extract overall patterns.

Although exaggerations in science and health reporting are often subtle, we do believe that the cumulative effect of hundreds per year might be damaging to a reader's understanding of health-related matters, and to their understanding of the relative uncertainty of different conclusions, such as those based on observational research vs randomised controlled trials. Recent data shows that observational studies are just as likely as RCTs to receive news coverage {Wang:2015wc}. It is also possible that trust in medical research may be eroded when, over time, opposing conclusions are both presented with apparent high certainty, or seemingly conflicting advice is offered to readers. Erosion of understanding and trust in scientific research and advice could then hamper public health initiatives such as changing alcohol guidelines, and they may have large knock on effects for high profile controversies, such as have arisen over vaccines or statins. 

One limitation of this work is the number of journals sampled, which was set primarily to achieve a manageable project; we also note that not all journals issue press releases (e.g. the New England Journal of Medicine does not). The major limitation of this work is its observational (correlational) nature. It is always possible that some of the apparently causal links are circumstantial or mediated by factors we did not analyse. For example the rare use of caveats might potentially have co-occurred with unknown other characteristics influencing news uptake in order to disguise any caveat influence. The possibility of unknown correlated factors is generally true for correlational research.

We also cannot be sure that the relationships between press releases and news reported here would continue to hold as strongly if circumstances changed. For example, if the rates of caveats were to increase markedly, it is possible that they would be ignored much more often; their current impact on news might be related to their low frequency. It is only possible to definitively answer such questions by experimentally changing the current rate.

Exaggerations in journal press releases were lower than in the university press releases studied previously (Sumner et al 2014). This is not consistent with the assumption held by many scientists that it is academic authors who reduce exaggeration in press releases; authors tend to be less involved in writing and revising journal press releases (who have no obligation to consult them) than university press releases. Instead, the different rates may reflect the different professional habits, contexts, priorities, and pressures of press officers working in each environment. For instance there are fewer elite scientific/medical journals than universities, and because their publication dates are regular, they are an established feature of journalistic routines (news diaries) in all national and specialist news outlets. Thus perceived competition between journals is likely to be less marked than between universities.

Exaggerations and caveats for human inference drawn from non-human research were especially low in number, which meant we could not meaningfully analyse them further. However the low numbers are interesting in themselves. Firstly, exaggeration rates were much lower than for university press releases (11%, CI 5% to 17%, vs 36%, CI 28% to 46%), which we speculate may reflect journals worrying less than universities about revealing animal research programs and thereby attracting the attention of protestors or suffering reputational damage. It will be interesting to discover whether the Concordat on Openness on Animal Research changes current practice in this regard. Secondly, caveats for non-human research were very low across both datasets (N<5), even compared to the low rate of caveats overall. Our study offers no explanation for this trend.

In conclusion, the data presented here indicate that it might be possible to avoid some common small exaggerations and to increase the use of caveats and justifications in press releases without hampering news uptake. The evidence also suggests that these small changes to press releases are likely to carry through into news. This might be one important part of the challenge to enhance understanding and reduce confusion about lifestyle choices relevant to health.

References

1.              Gandy OH. Beyond agenda setting. Ablex Publishing Corporation; 1982. 1 p.

2.              Nelkin D. Selling Science. W H Freeman & Company; 1995. 1 p.

3.              Williams A. Environmental news journalism, public relations, and news sources. hansen A, Cox R, editors. 2015. 10 p.

4.              Lewis J, Williams A, Franklin B. A compromised fourth estate? Journalism Studies. 2008 Feb;9(1):1–20.

5.              Bauer MW, Bucchi M. Journalism, Science and Society. Routledge; 2008. 1 p.

6.              Kiernan V. Embargoed Science. University of Illinois Press; 2006. 1 p.

7.              Williams A, Clifford S. Mapping the Field: Specialist science news journalism in the UK national media. Science and the Media Expert Group, Department of Business, Innovation and Skills. 2009.

8.              Weitkamp E, Eidsvaag T. Agenda Building In Media Coverage of Food Research. Journalism Practice. Routledge; 2014 Oct 28;8(6):871–86.

9.              Taylor JW, Long M, Ashley E, Denning A, Gout B, Hansen K, et al. When Medical News Comes from Press Releases—A Case Study of Pancreatic Cancer and Processed Meat. Rooman I, editor. PloS one. Public Library of Science; 2015 Jun 17;10(6):e0127848.

10.              Castell S, Charlton A, Clemence M, Pettigrew N, Pope S, Quigley A, et al. Public Attitudes to Science. Ipsos Mori report for Department for Business Innovation and Skills. 2015 Dec pp. 1–202.

11.              Boyce T. Health, Risk and News: The MMR Vaccine and the Media. New York: Peter Lang; 2007.

12.              Lewis J. Constructing Public Opinion. Open University Press; 2001.

13.              Ramsay ME. Measles: the legacy of low vaccine coverage. Archives of Disease in Childhood. 2013 Sep 12;98(10):752–4.

14.              Cooper BEJ, Lee WE, Goldacre BM, Sanders TAB. The quality of the evidence for dietary advice given in UK national newspapers. public underst sci. 2012 Jul 19;21(6):664–73.

15.              Leveson B. An Inquiry Into the Culture, Practices and Ethics of the Press. pp 22, 80, 690-91, 1803; 2012.

16.              Schwitzer G. How Do US Journalists Cover Treatments, Tests, Products, and Procedures? An Evaluation of 500 Stories. PLoS Med. Public Library of Science; 2008 May 27;5(5):e95.

17.              Samuel G, Williams C, Gardner J. UK science press officers, professional vision and the generation of expectations. public underst sci. SAGE Publications; 2015 Aug 11;:0963662515597188.

18.              Schwartz LM, Woloshin S, Andrews A. Influence of medical journal press releases on the quality of associated newspaper coverage: retrospective cohort study. … : British Medical Journal. 2012;344:d8164.

19.              Brechman J, Lee CJ, Cappella JN. Lost in Translation?: A Comparison of Cancer-Genetics Reporting in the Press Release and Its Subsequent Coverage in the Press. Science Communication. 2009 May 8;30(4):453–74.

20.              Sumner P, Vivian-Griffiths S, Boivin J, Williams A, Venetis CA, Davies A, et al. The association between exaggeration in health related science news and academic press releases: retrospective observational study. BMJ. British Medical Journal Publishing Group; 2014 Dec 10;349(dec09 7):g7015–5.

21.              Schwartz LM. The Media Matter: A Call for Straightforward Medical Reporting. Ann Intern Med. American College of Physicians; 2004 Feb 3;140(3):226–8.

22.              Woloshin S, Schwartz LM, Casella SL. Press Releases by Academic Medical Centers: Not So Academic? Ann Intern Med. 2009;150:613–8.

23.              Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, Mantz J, et al. Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study. PLoS Med. Public Library of Science; 2012 Sep 11;9(9):e1001308.

24.              Hansen AS. The changing uses of accuracy in science communication. public underst sci. SAGE Publications; 2016 Mar 8;:0963662516636303.