Written evidence submitted by Joe Whittaker, Ellie Rogers, Nicholas, Micallef, Sara Correia (SMH0018)

What are the links between social media algorithms, generative AI and the spread of harmful content online?

Cyber Threats Research Centre – Swansea University

The Cyber Threats Research Centre (CYTREC) explores a range of online threats, from terrorism, extremism, cybercrime, to child sexual exploitation online and grooming. Its core aims are to:

• Better understand online threats
• Produce new original empirical research
• Inform policy and practice

CYTREC is an interdisciplinary centre. Its experts have backgrounds in law, criminology, political science, linguistics and psychology. It is also collaborative and engages with non-academic stakeholders at all stages of the research process. CYTREC works with partners to ask the research questions that matter, to share findings and to produce policy recommendations. CYTREC’s partners include RUSI, Tech Against Terrorism and the NSPCC. Its work has been presented around the world, including to the UK Parliament, the UK Home Office, US State Department, Europol and NATO Advanced Training Courses.

Several CYTREC researchers have developed an interest and published work on the amplification of harmful content via social media recommendation algorithms. As such, we believe that our knowledge can be of use in this call for evidence.

Summary

• Social media platforms have developed sophisticated recommendation algorithms, most of which rely on artificial intelligence, which are fine-tuned to show users content that the platforms believe that they will be interested in seeing. This is to keep users on the platform to increase advertising revenue.
• This has led to a substantial concern from policymakers, the media, and researchers, who believe that these algorithms may be promoting harmful, and in some cases, illegal material on their platforms, and amplifying it for beyond its natural reach.
• Looking at the body of research, there is little evidence to suggest that illegal content – such as terrorist, suicide-promoting, or child abuse materials – is being amplified. This is because there is very little research. The optimistic view is that illegal material is often removed from platforms and therefore not available to be recommended. We would urge caution against this conclusion. While platforms have become better at removing illegal content, it is better to view this lack of evidence as a gap in research to be filled in future.
• There is some relatively strong evidence that legal but harmful content – such as mis/disinformation and extremism – may be amplified via recommendation algorithms. Most studies tend to show that if users begin to interact with such material, they are likely to be shown more. There are some dissenting studies that have found that there is little-to-no amplification, and others which paint a more complex picture.
• There are some substantial methodological gaps which limit our confidence in the existing body of research. This includes (i) a heavy emphasis on a single platform (YouTube); (ii) most studies being conducted using English-language content; (iii) a reliance on a methodology that draws the potential videos that could be recommended, rather than looking at content that is actually recommended; and (iv) the use of “black box” studies which feed data into an opaque algorithm and assess what comes out at the other end.
• There are several ways in which platforms seek to address this challenge, including: removing content so it cannot be recommended; “downranking” content to make it more difficult to find; and algorithmically promoting counter-narratives and fact-checking.

1. Introduction

Recent years have seen a substantial concern around the role of social media recommendation algorithms and the potential to promote harmful content. Several stakeholders have suggested that platforms, driven by a desire to keep users on their sites to maximise profits, may be amplifying problematic and inflammatory content, such as extremist material, mis/disinformation, and suicide content. This, in turn, may lead towards offline harms, which has a severe impact on our society, such as engaging in political violence, the erosion of trust in democracy, or self-harm.

In the Online Harms White Paper, the UK Government made this concern clear by stating that such algorithms may promote disinformation by stopping users from seeing content originating from sources that challenge their viewpoint, meaning that they may create a false perception that the story is true and believed by far more people than is the case.[1] Similarly, the former EU Counter Terrorism Commissioner expressed concern, noting that tech platforms were amplifying legal, but harmful content via recommendations which can act as a conduit for violent radicalisation by mainstreaming extreme views.[2] Ofcom, the regulator of the Online Safety Act (OSA), has also highlighted the importance of recommendation systems in the enforcement of the act, outlining several ways in which illegal materials – such as suicide content or hateful speech – may be amplified,[3] as well as legal but harmful content, which should not be accessed by children under the OSA.[4]

This written evidence has three aims: i) to outline the limited available information about the ways that social media recommendation algorithms operate, ii) to present the evidence base of the empirical research literature on the amplification of harmful content, and iii) to discuss and evaluate the methods that are used to counter this amplification.

2. Social Media Recommendation Algorithms

Recommendation algorithms (also referred to as recommendation systems) are designed to show users online content which they may be interested in seeing. To do this, the platform collects user data, such as their demographic attributes, perceived interests, and behaviours,[5] with the aim of finding content that users are most likely to engage with, making them more likely to remain on the platform, and therefore increase advertising revenue.[6] They use a machine learning-driven framework that analyses user data and contextual information to predict user preferences and provide personalised item recommendations.[7] This involves a technique called Collaborative Filtering, which relies on identifying patterns in user interactions by leveraging similarities in behaviour or preferences across users.[8] This is often combined with content-based filtering (which matches user profiles to item attributes), knowledge-based approaches (which rely on explicit domain knowledge), and social network-based techniques (utilising relational data).[9] Together, these methods emulate decision-making processes by incorporating both individual and community-driven insights to enhance user experience in social media platforms.

There is relatively little detail known about how these algorithms operate in practice; social media platforms keep this intellectual property a closely guarded secret.[10] However, in recent years, some of the biggest social media platforms have given a broad overview of how they work. YouTube notes that their recommendation system is responsible for a significant amount of their overall traffic, using different data points such as clicks, time spent watching videos, responses to surveys, likes and dislikes. Their system uses artificial intelligence (AI) to curate and prioritise the pool of videos recommended to users.[11]  The same is true for other large platforms. Meta – the parent company of Facebook and Instagram – states that both platforms use AI to personalise recommendations from content which originates outside of a user’s individual network.[12] This is also the case for TikTok’s “For You” feature, which uses machine learning to select, predict, and rank the content the users see.[13]

These recommendation systems are an integral part of how most social media platforms operate. YouTube, which boasts 2.7 billion monthly active users, has several different ways in which it recommends content to users (from suggestions in the side bar, to the auto-play “up next”, to the feed on YouTube Shorts). These recommendations account for over 70% of the platform’s traffic.[14] Facebook and Instagram both operate AI-driven personalised feeds to deliver content to users, meaning that almost everything that a user sees is driven by its algorithm. In 2023, Mark Zuckerberg noted that more than 20% of content in users’ feeds was now recommended by AI from people, groups, or accounts that the user does not follow.[15] Less is known about the level of traffic on TikTok or X’s personalised feeds (both called “For You”), but it is clear that recommended content is an essential driver of user experience.

3. What does the empirical research tell us?

3.1 Amplification of Illegal Content

A key part of the OSA is to protect all users (adults and children) from illegal content. However, there is a very limited empirical base to understand the scope of the amplification of illegal materials.[16] Two studies, which assess the proliferation of content or accounts from al Qaeda (on Twitter/X) and the so-called Islamic State (on YouTube), find that terrorist-supporting material was amplified.[17] However, it should be noted that these are both older studies for which data collection took place (2013 and 2016 respectively), before platforms began to take a more proactive approach to removing terrorist content, such as developing AI detection models[18] and hash-sharing databases.[19] There are no studies which have sought to assess whether material – such as suicide promotion; child sexual abuse; or extreme pornography – are amplified by recommendation systems. The optimistic view of this is that platforms have automated algorithms that make them relatively well adept at identifying and moderating such extreme content, so it is therefore not available to be recommended. While it is doubtlessly true that the large tech platforms have become better at removing illegal content (particularly terrorist and child abuse materials), the absence of evidence does not mean that there is no amplification, rather that there is a knowledge gap that ought to be filled.

3.2 Amplification of Legal but Harmful Content

While there is limited evidence as to the amplification of illegal material, there is substantially more research which attempts to understand the proliferation of content that is not clearly illegal (or even violative of platforms’ rules) but may potentially be harmful. This is sometimes referred to as “legal but harmful” or “borderline” content. Several studies demonstrate that platforms may be amplifying two types of this content: Mis/disinformation and extremism.

The proliferation of misinformation[20] or disinformation[21] on social media has become a priority policy concern in recent years. Research has shown that social media platforms may amplify this content and in doing so, restrict users from viewing reliable content. Several studies have focused on Covid-19 health disinformation, finding the false narratives about hydroxychloroquine were promoted on YouTube[22] or that individuals on the same platform who watch videos with false information about vaccines are likely to be recommended more of the same.[23] Studies looking at other types of health topics, such as treatments for cancer[24] and the Zika virus[25] also found that misinformation may be amplified. Political conspiracies may also be promoted, such as false claims that the US Presidential Election of 2020 was stolen.[26] The amplification of legal but harmful extremist[27] content is also a concern. Most of this is focused on the far-right with research demonstrating its prevalence within the recommendation systems of YouTube,[28] Twitter,[29] and TikTok.[30] There is also some evidence to suggest that Islamist extremism may be subject to amplification,[31] as well as misogynistic incel extremism.[32] There is very little research that looks into the wider set of harms beyond mis/disinformation and extremism, aside from two studies which show that that age-inappropriate content (due to violence or inuendo) may be recommended to children.[33]

It is important to note that some studies have found no evidence of the amplification of harmful content. One piece of research found that YouTube’s recommendation system actively pushed users away from extremist and misinformation content,[34] while two studies that use a longitudinal design found that there was no evidence to suggest that YouTube’s algorithm was bringing users towards extremist material,[35] or that the platform’s moderation policies were becoming more adept at stopping such recommendations.[36] One study found that recommendations had positive effects on some platforms (YouTube) and null effects on others (Reddit and Gab),[37] and another found that YouTube does present users with far-right recommendations, but this effect was driven by the individuals who (via a pre-experiment survey) scored highly on measures of racial resentment.[38] This finding suggests that the recommendation system may play a role in amplifying content, but the individual users’ decisions are also important.

Looking specifically at the riots that took place in the UK in the summer of 2024, it is clearly too early for rigorous empirical research to have undergone peer review, therefore all findings should be tentative. That being noted, the Institute for Strategic Dialogue outlined several ways in which social media platforms played a role in algorithmic amplification of mis/disinformation which had an impact on the offline violence. They highlight that several pieces of false information about the Southport attacker, such as an incorrect name, being known to the MI6, and asylum status proliferated on both X’s “Trending in the UK” and TikTok’s recommended searches. They also found that several verified users on X (those that pay for their content to be algorithmically amplified and receive a blue tick) propagated false narratives and called to “permanently remove Islam from Great Britain.”[39] This research is anecdotal and does not test the amplification of content, but merely reports that potentially harmful content was being recommended. Finally, it is important to note that since the change of ownership of X in 2022, it has become almost impossible to conduct research on the platform because of restricted researcher access[40] and lawsuits[41] against organisations who have conducted work on the platform. This will directly disincentivise future work from being undertaken in future as the barriers deter the scientific research community from undertaking work that has a positive societal impact.

3.3 Methodological Gaps: What is lacking within the body of research?

Although there has been a substantial increase of research on this topic in recent years, the body of literature has some substantial flaws. The first is that the vast majority of studies are focused on one specific platform – YouTube. In a meta review undertaken in 2022, 60% of the research focused on YouTube, and the proportion has only grown since that point.[42] The high proportion of content on YouTube may lead one to infer that the platform has the biggest problem with amplifying harmful content. While this may be true, we would urge caution against this conclusion. The most parsimonious answer is that YouTube is substantially easier to conduct this sort of research on than other platforms such as Facebook, Twitter, TikTok, or Instagram. As such, it may be a result of convenience rather than following the problem.

The second problem stems from the first. The most popular type of study design is to log in to YouTube’s applications programming interface (API) – the platform infrastructure that allows researchers to generate metadata and study it – to retrieve a set of Related Videos, which are videos that could potentially be recommended to users. While this offers useful information on the supply content available to be recommended, this method cannot account for how the platform uses its AI-generated software to create personalised recommendations. Instead, most studies use what is called a “random walk” algorithm which assesses the probability of coming across problematic content from within the pool of potential recommendations.[43]

The third methodological gap within the body of research is that almost every study uses a “black box” design. Put simply, users do not have access to internal social media data and cannot manipulate the recommendation algorithm, so instead simply feed information into an opaque system and assess what comes out the other end. Only one study focusing on harmful content, which was undertaken in collaboration with Twitter (pre-takeover), has used internal social media data; the authors created a field experiment by giving a small selection of users a chronological timeline as opposed to an algorithmically driven one.[44] It is vital for researchers to have internal access to social media data, otherwise it is impossible to test causal hypotheses on the role of recommendation systems without being subject to confounding variables, sampling biases, and approximations of user behaviour.[45]

Finally, there are wider problems with regard to the content that is under investigation within this corpus. As noted above, there is little investigation which seeks to look specifically at whether illegal content is being amplified and when legal but harmful content is under study, it is almost exclusively either mis/disinformation or extremism. Within this body, most of the research collects data in English, with some instances of both Portuguese (mostly in the Brazilian context) and German. This presents an important gap because it is likely that the methods that are deployed to counter amplification of problematic content are fine-tuned for English-language material, leaving other regions such as the Global South significantly underrepresented. This is a critical oversight, as these regions are also vulnerable to misinformation, with unique socio-political contexts and large language communities, that exacerbate the spread and impact of such content yet remain largely unaddressed by existing countermeasures. For example, when YouTube began its policy of downranking “borderline content”, it tested it on the US market first before rolling it out more broadly.[46] The above-mentioned study’s findings on so-called Islamic State videos on YouTube exemplifies this issue; Arabic-language videos were more likely to be recommended than English-language ones.[47]

4. Responses to Algorithmic Amplification

There are several different content moderation methods that can be used to attempt to counter the algorithmic amplification of harmful content. The first is the most simple – removing the content, so it is not available to be recommended. Platforms are legally obliged to remove a wide range of illegal content once they become aware of it, as set out in the OSA (as well as other pieces of legislation such as the EU’s Terrorism Content Online Regulation). Platforms are also required to assess and mitigate the risk of users’ exposure to illegal content through algorithmic recommendation. However, as discussed above, the evidence base does not suggest that there is widespread amplification of illegal content, but rather of content which falls under the remit of “legal but harmful.” While many platforms have terms of service that do not allow certain types of legal content to be posted, many state that it is national governments’ place to set the standards of what type of speech is permissible. Underlying this issue is over 2000 years of academic debate around freedom of expression; liberal democracies tend to provide political speech (which encapsulates much of the legal but harmful content that proliferates online, such as fringe politics and mis/disinformation) with the highest level of protection.[48] Therefore, simply removing all content which may play a role in causing harm is not an attractive option and, in practice, will cause substantial harms to citizens’ human rights.

Another option is to merely make certain types of content more difficult to access. This is something that the largest social media platforms (e.g. Meta; X; TikTok; Reddit) all do with content that they deem to be borderline.[49] The underlying logic of this approach was articulated in 2018 by Meta CEO Mark Zuckerberg; he identified that content that comes close to the “policy line” where it would be removed tends to create very high levels of engagement. Therefore, Meta would tip its hand on the algorithmic scale to make the content close to the line (i.e. legal but harmful) less easy to find.[50] There are several ways to implement this, for example: removing all problematic content from potential recommendations; simply downranking it so it becomes less likely (but not impossible) to be recommended; or not recommending content to particular categories of users (for example, children).[51] At first glance, this approach has some benefits. Using this approach, borderline content is not removed, which seems to alleviate many issues with freedom of expression, but at the same time it makes problematic content more difficult to find. However, it has also been criticised for failing to adhere to human rights norms. In particular: (i) the concept of “borderline content” is not adequately defined by any social media platform, meaning that users cannot have a clear understanding of how their content will be moderated; (ii) this ambiguity leads to an unclear understanding of how necessary and proportional the response is; and (iii) there is little-to-no transparency on how this concept is used in practice; platforms have to date not detailed its use in their transparency reports.[52]

A third method of countering the amplification of problematic content is to algorithmically promote counter-speech which challenges it. As with downranking, many platforms state that they are doing this already.[53] One example is Moonshot’s “Redirect Method” which identifies keyword searches that relate to extremist ideologies and then uses advertising technology to offer an opportunity for users to click through to a counter speech intervention. This has been conducted with both jihadist[54] and far-right and incel content.[55] This type of approach is most attractive from the perspective of freedom of expression, but is clearly time and labour intensive, which makes it difficult to scale up. In addition, its efficacy is still unknown, for example in their report on Redirection of those viewing far-right content on Facebook, Moonshot show that there were over 57,523 searches of far-right key words, only 2,228 of those individuals clicked on the advert, and of those only 254 remained on the site that they were redirected to, of which only 25 clicked the “Get Help” button (0.04% of those who originally searched for far right content in the first place).[56]

5. Conclusion

This written evidence has sought to shed light on the ways in which social media recommendation algorithms – many of which are powered by artificial intelligence – may amplify harmful content. It has done this in three ways: Firstly, by offering a brief overview of what recommendation systems are, and the little that is known about how they operate. Secondly, by giving an overview of the empirical research in this area, including the several methodological flaws in the field. Thirdly, it has outlined some of the ways which can be adopted to counter the threat, such as content removal, downranking, and the amplification of counter speech. Considering the broad question of the applicability, and possibly efficacy, of the OSA in combatting these potential harms, there are clearly some advantages; it creates a duty for platforms to assess and mitigate the risk of algorithmic amplification of illegal content. On the other hand, as far as adults are concerned, the main focus of the OSA is on illegal content, but the body of research shows that there is not an evidence base to suggest that illegal content is being amplified. Instead, the research points to the proliferation of “legal but harmful” content, which is only covered for child users. This suggests that the scope of the legislation is too narrow to stem the potential harms that can occur as a result of the amplification of this type of content. With the widespread availability of generative AI tools, there is a risk that both state and non-state actors, each of a range of motivations (e.g., financial gain, hacktivism, spreading disinformation or ideology, etc.) will leverage the technology to generate borderline content at scale.

17 December 2024