Written supplementary evidence submitted by DMG Media (OSB0220)
Case study: How Google discriminates against some news publishers in search results
1. This case study is presented in response to a request from Baroness Kidron at the Joint Committee on the Online Safety Bill hearing on October 21, 2021. It contains information extracted from DMG Media’s main submission to the Joint Committee, but focuses only on discrimination in Google search. We would be very happy to answer any further questions the Joint Committee may have.
2. It is a matter of deep concern to us that there is a growing body of evidence that Google, the dominant search engine, sets its algorithms to favour certain news publishers and discriminate against others. As with social media we fear that the Online Safety Bill will legitimise and institutionalise the skewing of search algorithms to further the aims of a commercial monopoly, to the detriment of freedom of expression, a pluralistic media, and open democratic debate.
3. MailOnline is the most visited news website in the UK (excluding broadcasters) and the fifth most visited English-language news website in the world. It would be logical therefore to expect, when a member of the public uses Google to search for a news subject such as ‘Covid’ or ‘Brexit’, that MailOnline stories would generally appear high up in the first page of search results.
4. That is not the case. Data from search analytics companies Sistrix and NewsDashboard UK shows that Google overwhelmingly favours two news websites - the Guardian and BBC - in search results and discriminates heavily against most other major British news websites, including and in some respects particularly MailOnline. Indeed MailOnline’s share of search visibility for many important news search terms is close to zero – for example, for the term ‘Covid’ it was just 0.22pc for the month of July this year. Google’s algorithms have in fact consistently reduced MailOnline’s search visibility since 2013.
5. Search visibility is significant because it measures not the choices made by users, but the choices made by Google’s algorithms when users make requests for particular search terms. The Sistrix search visibility index is the industry standard and measures Google’s ranking across sets of representative keywords.
6. Particularly striking evidence that MailOnline was being discriminated against by Google came in 2019, when an algorithm change in early June cut MailOnline’s search visibility by 50pc, while other news websites’ visibility improved. Three months later, following protests to Google at the highest level, MailOnline’s search visibility was equally suddenly restored. At neither point did we make any changes to the structure or presentation of the site which would explain its rejection or subsequent re-acceptance by Google’s algorithms.
7. At the time we were only able to guess why Google had reduced MailOnline’s search visibility. However when reviewing evidence for the Competition and Markets Authority’s (CMA) market study into online platforms and digital advertising, it became apparent to us that the June algorithm change coincided with the introduction by Google of its new Unified Pricing rules for digital ad markets. These rules had the effect of limiting the use by publishers of header bidding, a means of setting price floors which enabled us to fill more of our ad inventory with better-paying non-Google demand.
8. We have since learned that other major publishers which made use of header bidding, such as the News York Times and Conde Nast magazines, also saw search visibility drop in June 2019.
9. The consequence of Unified Pricing was that by the end of the three-month period June-September 2019, Google had forced MailOnline to sell twice as much ad inventory through Google’s ad exchange, while Google paid half as much for each ad slot.
10. For most of 2020 MailOnline’s overall UK search visibility index, as measured by Sistrix, hovered at around 100 – only a quarter of the best figures recorded in 2012-15, but similar to the level before the dramatic drop in June 2019. However, from January this year we started to see another steady decline in MailOnline’s overall search visibility index, which has now stood for several months at just over 50, around half the level seen through most of 2020, and one-eighth of the 2012-2015 peak. (See Tables 1 and 2).
Table 1. MailOnline UK overall search visibility index (desktop) – last 10 years (source: Sistrix)
Table 2. MailOnline UK overall search visibility index (desktop) – last 12 months (source: Sistrix)
11. When MailOnline’s search visibility is plotted against its UK rivals a disturbing pattern emerges. Pre-2015 Google heavily favoured the BBC, with the Guardian, MailOnline and Telegraph broadly grouped together. From 2015 onwards two distinct groups start emerging. The Guardian and BBC are consistently favoured, with a visibility index currently standing at around 400, while the Mail, Telegraph and Sun are grouped together with consistently poor visibility, currently standing at around 50-75. (See Table 3 - this chart does not include the Mirror and Express, both of which score slightly below MailOnline).
Table 3. UK competitive search visibility last 10 years (source: Sistrix)
12. The implications for freedom of expression, media plurality and democratic debate should be obvious. How Google’s search algorithms work is the company’s most closely-guarded secret. Google tells the public:
‘To give you the most useful information, Search algorithms look at many factors, including the words of your query, relevance and usability of pages, expertise of sources and your location and settings.’
The public place great faith in Google, and imagine that when they search for news on politics, health, business, or any number of other topics, Google’s emphasis on relevance and expertise means the content they are shown has been picked because it gives the most reliable and useful information. Unless they are students of search visibility they have no idea that when they search for news Google’s algorithms invariably steer them towards two particular news sources, the Guardian and BBC.
13. Moreover, there is nothing in the Bill as presently drafted to prevent Google from picking two other preferred news providers in the future, if it should suit its interests – indeed, as Table 3 shows, at times the Guardian’s own search visibility index has risen, then fallen sharply, as determined by Google’s whim.
14. This pattern is repeated across many search terms, as the following charts demonstrate. The most striking is Table 4, which shows shares of UK mobile search visibility for the term ‘Covid’ during July this year, in which the Telegraph and Independent each score barely over 1pc and MailOnline scores only 0.22pc, while the BBC scores 12.65pc – 58 times the share of MailOnline – putting it ahead of the World Health Organisation and NHS websites, and the Guardian on 8.63pc.
Table 4. ‘Covid’ - UK Mobile Overall Share of Search Visibility, July 4 – August 2, 2021. (source: NewsDashboard UK)
15. When it comes to politics, the Guardian is the winner for the term ‘Brexit’, along with other pro-Remain websites. Table 5 shows the Guardian scored 19.45pc, while the FT, not normally a good performer because of its paywall, won 17.83pc of Google search requests, closely followed by the Independent. Despite Brexit being a subject to which MailOnline devoted a great deal of coverage, MailOnline received close to zero – a derisory 0.1pc. This is not to repeat the arguments over Brexit, but to make the point that Google can and does direct search traffic overwhelmingly to one side of an issue only.
Table 5. ‘Brexit’ - UK Mobile Overall Share of Search Visibility, July 4 – August 2, 2021. (source: NewsDashboard UK)
16. In contrast, traffic for Facebook, where results are determined by users’ decisions to share content rather than choices made by the platform’s algorithms, and therefore indicate which news the public rather than the platform prefers, tells a very different story. Despite fluctuations caused by individual stories going viral, MailOnline traffic has remained largely consistent over the last year, as table 7 shows.
Table 7. Facebook MailOnline article views – 12 months to August 1. (source: Adobe)
17. Indeed, global figures for June this year show MailOnline is the second most popular English-language news website across the whole of Facebook, receiving twice as many visits as the BBC and Guardian combined (see Table 8).
Table 8. Top publishers on Facebook, June 2021 (source: Newswhip)
18. What are the reasons for this stark discrepancy? Google never explains how its algorithms work, so we cannot be certain why it discriminates so consistently against some publishers and in favour of others. We believe we now have a convincing case that its two dramatic algorithm changes in June and September 2019 were dictated by its commercial self-interest, and part of its successful campaign to maximise its profits by preventing publishers from using header bidding in digital advertising.
19. For the rest, it certainly appears that Google’s bias against MailOnline is much more pronounced when its algorithms are ranking political stories than for stories of more general interest. Whether this is a deliberate company policy, or simply the result of unconscious bias on the part of the Californian web engineers programming algorithms preferring websites that echo their own left-liberal views, we cannot know. But it is clearly happening.
20. Our concerns about the commercial effects of discrimination in search led us to argue successfully that the Competition and Markets Authority (CMA) should include search in its market study into online platforms and digital advertising. We maintained that it was impossible to plan our business without fair warning and explanation of algorithm changes, and without remedies when those changes cause commercial damage. The CMA’s Final Report found:
‘It is clear that many publishers rely on Google and Facebook for a significant
proportion of their traffic and that changes to key search algorithms by either
of these can have a significant impact on publisher businesses. We would,
therefore, consider it reasonable that publishers have sufficient explanation of
how these algorithms work and sufficient notification of changes to them
where they might notably impact upon their businesses. We consider that
provision to publishers of sufficient explanation about how the key search
algorithms work as well as explanation and notification of changes to these
are areas that would appropriately be covered by the proposed code of
21. Slowly, the government is putting in place regulatory structures to deal with these problems. The Digital Markets Unit recommended by the CMA has now started work, and is drawing up codes of conduct in advance of the forthcoming Digital Competition Bill, which will give it statutory powers. Its original remit was economic – to prevent Google reinforcing its dominant market position by directing traffic to its favoured publishers, and away from those such as MailOnline which seek to protect their revenue by using methods such as header bidding to secure advertising from non-Google sources.
22. However, news publishers are not just businesses, they are also participants in the political process – in particular those which are not broadcasters, and are therefore free to editorialise and campaign on the great issues of the day. Google’s policy in the UK over the last decade has been to direct search to the two publishers which it currently favours – and marginalise other voices such as the Mail, Telegraph, Mirror and the Sun. This is a serious threat to a pluralistic media, and in turn to democracy. Whatever one’s political views, democracy cannot thrive unless all voices can be heard.
23. For many years the only digital regulator in the UK has been the Information Commissioner, which is solely focused on privacy. Google has consistently used privacy regulation as a reason to deny user data to rival companies in the digital advertising market, and move the digital advertising industry into its own walled garden. The damaging effect single-issue regulation has had on other matters of concern in the digital ecosystem – in this case commercial competition – was recognised by the CMA in its digital advertising market study and addressed by the Government when it set up the Digital Markets Unit, in which the ICO and Ofcom participate as well as the CMA.
24. We are very concerned that even before Parliament gives the DMU the promised statutory powers to impose codes of conduct that will require platform algorithms to operate in a way which is fair, consistent, transparent and non-discriminatory, online safety regulation without a clear, cast-iron exemption for news publishers will provide Google with a new lever to legitimise discrimination against those it does not favour.
25. Therefore it is vital not only that news publishers’ content is fully exempted from the duty of care as it applies to search engines, but that the duty of care is also extended so that search engines are prohibited from using it as a lever to operate their algorithms in ways that are arbitrary and/or discriminatory.
10 November 2021
 The Guardian had no visibility pre-2013 because at that point it changed to its current domain name, theguardian.com. The Sun had very low visibility 2013-15 because it was behind a paywall.
 CMA Digital Advertising Market Study, Appendix S, p.10