Written evidence submitted by Minderoo Centre for Technology and Democracy, University of Cambridge (SMH0051)

By Dr Hugo Leal, Dr Stefanie Felsberger, and Professor Gina Neff
 

Harmful by Design: Current Approaches to Misinformation and How to Improve Harm Mitigation

HoC Science, Innovation and Technology Select Committee - Call for evidence on social media, misinformation and harmful algorithms

The Minderoo Centre for Technology and Democracy is an independent team of academic researchers at the University of Cambridge who are radically rethinking the power relationships between digital technologies, society, and the planet. Through our ambitious research agenda, we are enhancing public understanding of digital technologies and delivering positive changes to society’s relationship with these technologies. We are also part of the EU Horizon 2020 funded project AI4TRUST: AI-based-technologies for trustworthy solutions against disinformation, a project of 11 partners across 17 countries building trustworthy, beyond state-of-the-art misinformation detection and mitigation tools to amplify the human efforts to counter disinformation.

Summary of Submission:

The submission identifies critical challenges in addressing online misinformation and harmful content, focusing on three areas of concern:

1. Social media platforms' current business models prioritise user engagement, data extraction, and profit over harm-mitigation. This results in algorithmic recommendation systems that favour engagement over safety, and an approach to content moderation that is insufficient for the systematic and effective mitigation of harmful content.
    
2. The situation is compounded by the opacity which surrounds platform data and algorithms. Knowledge is routinely denied to those outside the companies themselves, making it impossible to fully map harms, repeat scientific experiments, or understand the effectiveness of harm-mitigation strategies. It also hampers the effectiveness of regulation, as regulators must rely on self-reporting from platforms on harms and the implementation of mitigation strategies.
    
3. A focus on content moderation at the individual level renders harm mitigation strategies less effective than approaches that aim to address collective harms and networks of harmful content. Generative AI and LLMs present new risks, but understanding networks of human-to-human and inter-platform connections is vital to tackle the problem of misinformation.

Recommendations:

1. A more holistic approach to online harms needs to evolve. An improved government approach could be informed by a shift to consider networks of harm and whole-network scale approaches to mitigate those harms.
    

2. A better data access framework is needed that builds the online safety regime in the UK.
    
3. Companies are not incentivised to tackle this problem via their business models. As policy develops through the implementation of the Online Safety Act and the expected AI Bill, the Government should consider creative approaches in addition to more stringent regulations. Models such as the Seoul Declaration and Christchurch Call provide frameworks to build upon.

Consultation Questions:

1. To what extent do the business models of social media companies, search engines and others encourage the spread of harmful content, and contribute to wider social harms?

1. The current business model of social media platforms and most other Internet-based companies is driven by a so-called ‘data imperative’ to maximise data extraction and utilise it for profit. Companies profit from this data via its application in targeted advertising, predictive analytics, and its value for developing new AI systems. With companies’ business models tied to monetising the collection of data, their platforms are designed to prioritise maximum user engagement, rather than user safety or public wellbeing.[1]
    
2. This has several consequences. Platform’s recommendation algorithms often prioritise emotionally charged, sensational content over balanced or nuanced content. Platforms are designed to induce loyal behaviours that in their most extreme, yet common form, turn customers into addicted users. Harmful or inaccurate content has been proven to disseminate faster than accurate information or content debunking misinformation or disinformation.[2] The rapid spread of harmful content facilitates complex contagion of these harms, and multiple engagements with disinformation facilitate people adopting such claims.[3]
    
3. This problem is compounded by platforms’ profit-driven approach to content moderation which prioritises automation, the offshoring of content moderation work to global majority countries, and a focus on individual pieces of content rather than a wider focus on the systematic spread of misinformation, disinformation, or harmful content.
    
4. Most moderation takes a  user-focused approach, examining individual posts or content on a case by case basis. By focusing on individual posts and users, commercial content moderation becomes a labour-heavy, socio-technical operation carried out by an army of (often precariously employed) labourers and/or by automated machine learning systems.[4] In this context, “moderating” is synonymous with ‘takedown or leave-up’ decisions, acting on posts (‘content’) according to platforms’ self-regulation principles.
    
5. A second approach surrounds a “harms perspective”, which uses a more maximalist understanding of categories of content that fall under a limited number of legal and illegal harms, ranging from spam to the spread of terrorist content or child sexual abuse imagery. This approach is much more effective than moderating on a case-by-case basis. Both approaches are grounded in the companies’ internal policies and Terms of Service (ToS). This means ToS have an outsized role in their power to ‘regulate a wide range of behaviors’ in the absence of stronger legal regulation.[5]

2. How do social media companies and search engines use algorithms to rank content, how does this reflect their business models, and how does it play into the spread of misinformation, disinformation and harmful content?

6. Scientific research shows that social media algorithms reward virality, which is linked to highly emotional and/or fabricated content, such as the promotion of disinformation for political and/or economic purposes.[6] Critics argue that platforms’ approach of prioritising engagement over community safety is harmful by design.[7] Giving evidence to the Joint Committee on the Online Safety Bill in 2021, whistleblower Frances Haugen said of Facebook, ‘It routinely tries to reduce the discussion to things like, “You can either have transparency or privacy. Which do you want to have?”, or, “If you want safety, you have to have censorship”’. Haugen continued, ‘in reality it has lots of non-content based choices that would sliver off a half percentage point of growth, or a percentage point of growth. Facebook is unwilling to give up those slivers for our safety”.[8] Having algorithms optimised for engagement that in turn feed on virality is not a matter of fate, but a question of choice. The same algorithms could be optimised to reward well-being, happiness, cooperative behaviours, or public interest.
    
7. Companies can present these simplified narratives because social media algorithms are opaque, especially to those outside the platform companies. Academic and civil society researchers are routinely denied access to information that would allow them to scrutinise the dynamics of information diffusion and social media algorithms. Knowledge about the details of the impact of algorithms is unavailable to all except the companies themselves.
    
8. Research from 2020 showed that about 1 in 10 algorithmically recommended YouTube videos had a conspiratorial nature.[9] In response, YouTube began ‘tweaking’ their recommendation system and demoting ‘borderline content’. Recently, its VP of Engineering wrote on the company’s blog that the changes led to a ‘70% drop in watch time on non-subscribed, recommended borderline content in the US’.[10] There are three important takeaways from YouTube’s claim. First, it is possible to contain the dissemination of non-illegal but harmful online content. Second, this drop concerns only “non-subscribed” channels in the US, which accounts for about 10% of the global users. Information on how YouTube’s changes affected the consumption of harmful content outside the US, in the UK and among the other 90% of customers/users, is missing. Third, and most worrying, YouTube’s claims are neither verifiable nor reproducible. Lack of access to data means those tasked with holding platform companies to account and ensuring compliance with legislation must take these companies at their word. This deprives policy makers, governmental institutions, academic researchers, journalists, and civil society of the tools they need for independent assessment of the health of social media platforms.
    
9. Ofcom is currently consulting on researcher access to information from regulated online services under the Online Safety Act.[11] Ofcom’s consultation is a vital opportunity to redress these barriers and ensure that people can fully understand how harmful content spreads and how its harms can be redressed.

2a. What role do generative artificial intelligence (AI) and large language models (LLMs) play in the creation and spread of misinformation, disinformation and harmful content?

10. Generative AI (GenAI) and LLMs have the potential to spread misinformation, disinformation, and harmful content. While the human role in spreading mis/disinformation, as well as low tech strategies such as “cheap fakes” must not be underestimated, GenAI is playing a significant role in the creation and spread of harmful content, such as non-consensual intimate imagery.
     
11. The role that LLMs play in the spread of mis/disinformation and harmful content is twofold. First, LLMs can be used to plan or craft content for disinformation campaigns at scale.  Guardrails that prevent such use can still be circumvented.[12] The second risk is that LLMs often provide information that is not accurate. So-called ‘hallucinations’ occur when LLMs ‘make up’ information that does not exist. These are hard for users to identify when LLMs do not provide sources for their output. This content can be either inaccurate or harmful, but the full extent of this is impossible to assess for researchers or fact-checkers who cannot monitor/verify what information LLMs provide to individual users at scale due to a lack of transparency or scrutiny currently allowed by LLM companies.
     
12. Use of LLMs is still not widespread. A 2024 study on public perception of GenAI in Argentina, Denmark, France, Japan, the UK, and the USA, found that in the UK only 2 percent of respondents use tools like ChatGPT daily. Only 5 percent of respondents in all countries reported using ChatGPT to access recent news, but 11 percent use it to answer factual questions.[13]
     
13. Audio-visual content generated by AI presents unique challenges in countering misinformation. Our AI4TRUST project finds that fact-checkers, journalists and other stakeholders report growing use of visual and image-based representations of mis/disinformation and they report needing improved tools for detecting AI-generated content—especially AI-generated audio content—and for tools for better understanding the dynamics of spread of mis/disinformation.
     
14. GenAI is currently more often used to create humorous content rather than for sophisticated disinformation campaigns. The majority of harmful GenAI content is non-consensual intimate imagery, including deepfake pornography and AI-generated nude photos.[14] People can now engage AI-powered digital avatars, such as Character.AI, which have enabled the creation of avatars based on real people. AI chatbots mimicking Brianna Ghey, a transgender teenager who was murdered, and Molly Russell, a teenager who died by suicide, were detected and taken down. The creation of such chatbots raises serious concerns about online safety and harm for users of the platforms but also those affected by the misuse of this technology. 
     
15. GenAI can and does play a role in the spread of misinformation, disinformation, and harmful content. Yet, the human role in spreading such content must not be underestimated. GenAI has garnered much focus due its capabilities and the current hype around its possible applications and consequences. However, influencers, public figures and other central nodes in online and offline networks have a remarkable impact on the spread of information and research shows that a small number of highly connected users are responsible for sharing a disproportionate amount of false information.[15]
     
16. The biggest challenge to understanding the full scale and role of GenAI in the spread of mis/disinformation and harmful content is the lack of data access for researchers and civil society. Without researcher data access, research can only focus on the user-side which limits understanding the scale of online safety issues, assessing the effectiveness of existing safety issues, and identifying emerging challenges. Access to data for researchers can enable robust and independent analysis of online harms and is crucial for evidence-based policy making, informing effective policies and interventions.

3. What role did social media algorithms play in the riots that took place in the UK in summer 2024?

17. We want to highlight that a focus on algorithms (in the abstract) and how they prioritise incendiary content can overshadow the role of specific messaging platforms. Relying on large self-organised groups, these platforms play a central role in the spread of narratives, while  fostering a sense of ‘community.’ This is yet another case that shows the importance of looking at the convergence of organic (human-driven) and synthetic (algorithmic-driven) dissemination. Messaging platforms, such as Telegram, can work in tandem with algorithms: while algorithms flag specific content to users, priming them to accept certain claims of disinformation campaigns, messaging platforms enable the organic recruitment, assemblage and collective action.
     
18. To understand how social media algorithms have played a role in spurring people to participate in racially motivated riots, we should focus not only on exposure to harmful or false information (i.e., when users see harmful content), but also on the adoption of these positions (i.e., when users internalise harmful narratives and begin reproducing them or acting on them). Much is made of so-called online ‘echo chambers’. Recent research based on cross-national survey data showed that ‘in almost all countries covered, only a minority (approximately 5%) of internet users inhabit echo-chambers, and in every country covered, more internet users consume no online news at all than occupy partisan online echo chambers’.[16] Instead, networks of harmful information expand beyond any specific echo chamber. Information, real or false, circulates in a media ecosystem that comprises both online and offline platforms, and both social media and “legacy media”. Online social media platforms now “seed” narratives that feed traditional outlets. Newsrooms rely on and report trending topics from social media platforms, for example.
     
19. Moreover, the increasing popularity of social media networks for news makes this problem more acute. As all social networks tend towards concentration, “network effects” determine an unequal distribution of influence (e.g. links, centrality) within social networks. This phenomenon is well documented and has been demonstrated multiple times.[17] In the case of social media platforms, it means that a minority of actors will always tend to gain control over a significant portion of the available resources, be that followers, income from generated content, etc. Without intervention, these network effects will occur. When a central set of actors craft and start spreading a narrative in a number of equally central channels, the accounting exercise of individual exposure is accessory to the fact that the narrative will take hold through the unequal and unfettered power of the “network effects”.
     
20. Highly emotional content generates more engagement than more neutral messages, and this is a function of network effects. For instance, Vosoughi et al. showed that false political news on Twitter in particular was more viral than any other type of mis/disinformation.[18]
     
21. A growing body of academic work focuses on the far-right’s use of Telegram in the UK.[19] We know the summer 2024 riots in the UK were driven by far-right actors who connected a highly emotional triggering event with their anti-immigration and islamophobic narratives. While Telegram public channels were undoubtedly used to stoke tension and, ultimately, provoke street contention, without access to data we cannot know the role played by other social media platforms beyond anecdotal accounts. Incidentally, Telegram is one of the few online platforms that maintain an open API for access to public broadcast and discussion channels. Although this does not allow researchers to draw conclusions about the role of social media platforms as a whole in the 2024 riots, it makes a partial analysis possible.
     
22. A lack of external access to platform data creates problems beyond content moderation in areas including community safety and national security. A report published in December by HMICFRS, which independently assesses the effectiveness of police forces in England and Wales, concluded that police leaders were too slow to respond to the riots that took place in summer 2024.[20] A further report will examine misinformation specifically. What is clear is that intelligence assessments did not sufficiently predict the ‘rising tide of disorder'. Proper social media data disclosure frameworks could provide us with early warning systems to prevent such instances in the future. Currently, we must rely on incomplete pictures from independent observers, or on platforms themselves to raise the alarm.

4. How effective is the UK's regulatory and legislative framework on tackling these issues?

4a. How effective will the Online Safety Act be in combatting harmful social media content?

23. The Online Safety Act prioritises the moderation of harmful content at the “individual level” over the regulation of collective and societal harms at the networked levels. Yet, harms on social media platforms play out within human-to-human and inter-platform network structures. A deeper understanding of the nature and dynamics of social networks can assist efforts towards a more effective online safety regime.
     
24. The individual-level approach limits the Act’s approach to countering the ways in which misinformation, disinformation, and harmful content spread online. In general, the online harms classified in the Act reflect this focus. Out of the four general categories identified as online harms, three (illegal harms, content harmful to adults, and content harmful to children) have ‘targeted individuals’ as the frame of reference, and only one (misinformation and disinformation content) address the networked relational dynamics of harmful content. Harms arising from repeated instances of the same contextual attacks against groups, such as systemic racism, sexism, homophobia, should leverage a wide range of counter strategies: from digital literacy and content moderation to a better understanding of how disinformation campaigns spread.
     
25. Third, the effectiveness of the Online Safety Act is hampered by questions around enforcement and independent scrutiny of whether tech companies are in adherence to the Act. While fines can be a useful tool to demonstrate the seriousness of the regulator, for many large companies, fines do not seriously impact their revenue streams. As online safety regimes develop, the Government should consider creative approaches in addition to more stringent regulations. Enforcement is further complicated in the current context, because regulators have to rely on platforms to self-report and implement mitigation measures without independent scrutiny and verification. Without further provisions for data access the impact of the Online Safety Act and its effectiveness cannot be independently evaluated.

4b. What more should be done to combat potentially harmful social media and AI content?

26. We recommend that content and context are considered in combating mis/disinformation. We propose a set of measures focused on context as well as content mediation. In short, an approach to platform governance that builds on the idea of context mediation prioritises interventions at the whole-network scale over individual users and single posts. In this situation, highly central nodes (such as highly influential spreaders of harmful narratives) and highly viral content present a potential risk to the whole-network and deserve closer inspection. Context mediation must balance controls on the ability of a message to go viral and “speech freedoms”. A clear example of this is “synthetic regulation” that audits and controls how algorithms determine the influence of users, the diffusion of content and the spread of harmful messages or content.
     
27. Another approach to content moderation on a network scale is declustering narratives: an intervention targeting the sub-network or cluster level. Declustering aims to slow the dissemination of specific narratives within and across identifiable networked communities. For example, whistleblowers have shown how adolescent girls with eating disorders should not see recommendations for posts that trigger episodes of disordered eating.[21] For declustering to work, we need better agreement on definitions of social harms. Some categories have a nearly unanimous consensus on harm (e.g. child abuse and sexual exploitation material). By connecting communities and content this approach provides a workable way to address the impact of harmful narratives on communities while protecting individual speech freedoms.
     
28. Third, we propose account accountability as an intervention targeted at highly influential users and aimed at addressing power asymmetries in online social networks. Highly central “influencers”, brokers and salient platforms play an outsized role in spreading harmful content. For example, different studies have shown that removing up to twelve accounts would contain the dissemination of socially harmful narratives across the various social media platforms.[22] For proper implementation of account accountability, we need a clear understanding of network dynamics and power distribution in social media platforms.
     
29. The reported practice of VIP whitelisting by Facebook is an illustration of how user engagement and business interests are prioritised over societal well-being. According to leaked documents by whistleblower Frances Haugen, the company runs a program named Xcheck that shields influential accounts from moderation enforcement.[23] Platforms understand that, left unchecked, central accounts foster engagement. Highly influential accounts bring revenue while moderating these accounts could lead to influential users with millions of followers accusing the company of censorship. A tentative implementation of contextual account accountability would be to invert and subvert the rules of the Xcheck programme altogether.
     
30. We have seen some cooperative initiatives involving multiple stakeholders, namely tech companies, governments and civil society organisations, on content whose qualification as threatening, harmful, or downright illegal is largely agreed. These initiatives go from shared spam lists to terrorist content and the filtering of child sexual abuse material. These mostly automated upload filters are deployed prior to any publication of such material with the intention to prevent the content from reaching the platform rather than having to chase down already platformed messages and users (e.g. CSAM and PhotoDNA, hashing). Even among those who believe the individual right to free speech supersedes the freedom from harassment and abuse, we will not find many advocates of the right to spam or diffuse child sexual abuse material. We believe there are more categories of fundamental rights shared across cultures that could also be included in this collective effort to optimise the internet for the common good. These practices can pave the way for a more informed discussion about the possible paths moving from moderating individuals and their content to preventing collective and societal threats.

4c. What role do Ofcom and the National Security Online Information Team play in preventing the spread of harmful and false content online

31. We reiterate that without a robust data access framework for researchers in academia and civil society organisations, explanations about the processes leading to the spread of harmful and false content online are becoming a product of unverifiable educated guessing. In the aftermath of a wave of platforms’ API shutdowns and/or continuous data opacity, researchers are increasingly unable to produce and reproduce the same kind of evidence-based research that made legislation like the Online Safety Act in the UK or the EU’s Digital Services Act a social and democratic imperative.
     
32. We believe that the first step to preventing the spread of mis/disinformation as well as harmful content leading to the creation of harmful contexts, is the principle of ‘platform observability’. This principle ‘seeks to address the conditions, means, and processes of knowledge production about large-scale socio-technical systems’.[24] This means we as a society have the right to know. We must be able to examine the networked content, contexts and the underlying socio-technical conditions for the dissemination of narratives and cultures. Policymakers and the research that informs policymaking must be based on evidence.
     
33. For that purpose, the proposed data access framework for researchers will guarantee the production and reproduction of evidence-based research about both the synthetic and organic drivers of engagement. Under this framework, we propose, platforms should provide data access to independent researchers assuring: i) transparent access to vetted researchers in in Academia and Civil Society Organisations; ii) Interoperable access systems shared across the online platforms (accessible through an API and/or online databases/repositories); data safety and confidentiality requirements in line with existing laws and regulations.

[End]

18 December 2024