11              Call for evidence: Social media, misinformation and harmful algorithms

December 2024

Call for evidence: Social media, misinformation and harmful algorithms

Science, Innovation and Technology Select Committee

5Rights Foundation response

December 2024
 

TW: This response includes discussion of specific examples of Child Sexual Abuse Material.

5Rights welcomes the Science, Innovation and Technology Select Committee’s inquiry into the role that social media companies played in the spread of mis/disinformation during the UK 2024 riots.

For over a decade, 5Rights has advocated for an online world which treats children the same way we do offline. This means recognising their rights,[1] including their rights to safety, privacy, participation and information and, crucially, their right to life – ensuring the digital world is built with this in mind.[2] Practically, this means ensuring digital services and products are safe and age-appropriate for them to use and if they cannot be made safe, preventing children from accessing them.

We develop new thinking, innovative frameworks and technical standards to challenge received narratives of tech exceptionalism with a focus on implementable change. While we work exclusively with and for children under the age of 18, our solutions and research are relevant to many other communities.

In 2021, we published Pathways[3] which established the pathways between the design of digital services and products and the risks children face online. It showed services such as Facebook, Instagram and TikTok were allowing children, some as young as 13 years old, to be directly targeted within 24 hours of creating an account with a stream of harmful content. Through interviews with engineers and experts we established that the proliferation of harm online is a consequence of how services are designed, which is driven by the business model. While this research looked at harm to children, its findings on what drives design decisions of digital businesses and products are universal.

Key points

• While the 2024 UK riots were predominantly instigated and driven by adults, children were amongst those who were arrested and or charged in connection with the disorder.[4] [5] For children not involved, many will have come into contact with, and been impacted by, the coverage and conversation about the violence through social media.[6]
• The spread of harmful content online is a consequence of the design choices of digital services and products whose business models are predicated on maximising the time, reach and engagement of users on their services.
• Most digital businesses rely on advertising revenue, which is boosted by collecting as much data as possible to accurately profile would-be customers. This data is used to fuel features such as algorithms and recommender systems which are designed to maintain the attention and engagement of the user.
• Research has found that the more eccentric or ‘outside of the norm’ ideas or content tends to receive the most attention and engagement from users, which means that algorithms – even if not intended – are often responsible for spreading harmful content, by virtue of the way they have been designed.
• Digital businesses prioritise growth over safety. There is currently no incentive or strong enough regulation or enforcement of existing regulation in the UK to instruct these businesses to design algorithms to not spread harmful content and activity which is having a grave impact on wider society.
• These businesses must be held accountable for how their services and products are designed, particularly where there is an impact offline. The 2024 UK riots is merely the most recent and high profile example of how this is materialising.

1. To what extent do the business models of social media companies, search engines and others encourage the spread of harmful content, and contribute to wider social harms?

Business models of digital businesses drive design objectives

Digital services and products (including social media companies, search engines and others) have three core design objectives:[7] 

1. To maximise time spent on the service, capturing as much of a user’s attention as possible.
2. To maximise reach and draw as many people into their product as possible.
3. To maximise activity by encouraging as much content generation and interaction as possible.

These objectives are driven by their business model, which is to extract as much personal information (data) from users as possible, to help them predict, modify and influence user behaviour. For most digital businesses, financial success is dependent on advertising revenue, and advertising revenue is in turn dependent on knowing as much as possible about users[8] and keeping them engaged.[9]

Put simply: “The quantity of customers paying attention to the content on a product equates to the value of the business. The more people paying attention, the more people there are to serve adverts to. The more advertising, the more profit.”[10]

Algorithms and recommender systems are a core part of the business model

These objectives lead engineers and designers to create ‘choice architecture’ which “shapes what content is seen, by whom, and how users behave and interact.”[11]

Some of the most prominent and widely used features and functionalities by social media companies within this architecture are algorithms, infinite scrolling and easily swipe-able reels of content.[12]

Digital businesses who rely on this business model commonly use behavioural science to underpin their choice architectures. Humans take the path of least resistance by default, so services are often designed to remove as much friction as possible to keep them on the platform.[13]

Algorithms are fuelled by extensive data collection based on known and inferred characteristics of users (including physical, cognitive, behavioural and socioeconomic attributes).[14] This helps the system to decide what to show the user, based on their interests and other characteristics.

The spread of harmful content by algorithms is a consequence of the business model

The way algorithms are designed (to increase attention and engagement) often drives more extreme content to be shared, as more eccentric or ‘outside of the norm’ ideas or content tend to receive the most attention and engagement, which can lead individuals to express opinions that deviate from the norm.[15]

Some of the most popular social media services also reward extreme opinions by design, in the absence of an interactable or viewable ‘dislike’ option.[16]

Algorithms contribute to a “feedback loop of negativity” on social media services, with research by the University of Cambridge finding users are 1.91 times more likely to share news articles that are negative rather than positive.[17] Further,” loops” can also be “filter bubbles" that legitimise user engagement with harmful content – including mis/disinformation.

Personalised recommender algorithms “redistributes the information we consume to disproportionately confirm our beliefs.”[18]  These bubbles are increasingly harmful when paired with the fact automatic recommendations decrease the sense of agency from users[19] – in particular where there are limited means to intervene in these recommendations.

Ultimately, algorithms act as “gateways” that determine what users can see and can unintentionally guide users down spirals of harmful content.[20]

2. How do social media companies and search engines use algorithms to rank content, how does this reflect their business models, and how does it play into the spread of misinformation, disinformation and harmful content?

We refer to our response above, however emphasise that, driven by the business model, content ranked by popularity (for example, content as part of a newsfeed) is given pride of place on the screen and algorithms designed to promote already popular content help it travel further while engagement rates increase.[21]

3. What role do generative artificial intelligence (AI) and large language models (LLMs) play in the creation and spread of misinformation, disinformation and harmful content?

Generative artificial intelligence (GenAI) helps to facilitate the creation of persuasive disinformation and illegal and harmful content at scale, speed and low cost.[22] GenAI technologies already have the capability to create synthetic content that is more persuasive than, and indistinguishable from, human-generated content.[23] [24]

The creation and dissemination of harmful content through GenAI is possible as the companies who create and make available the proprietary technology (AI systems) make it easy to create this content and without adequate guardrails. Indeed, the catalyst for an AI model producing or spreading harmful content such as disinformation can be as simple as inputting nothing more than a well-structured question that instructs the system to “Do Anything Now.”[25]

There is a growing bank of evidence that shows GenAI is assisting the creation and spread of harmful content.

Research by the Center for Countering Digital Hate[26] into Google’s GenAI tool Bard (now Gemini) found in 78 of 100 cases it promoted false and potentially harmful narratives without additional context or disclaimers – including on topics such as climate, vaccines, LGBTQ+ hate and sexism.

Digital service providers, including social media companies and search engine operators, are responsible for hosting and spreading harmful AI-generated content. This stems from revenue-driven business models, weak or non-existent safety strategies and poor content moderation.

Case study: Instagram and AI-generated Child Sexual Abuse Material (CSAM)

Between December 2023 and May 2024, 5Rights worked with the Online Child Sexual Exploitation and Abuse (CSEA) Covert Intelligence Team (OCCIT) on an investigation into how AI-generated Child Sexual Abuse Material (CSAM) is being shared on Instagram.[27] [28] It revealed that Meta allows users access to AI-generated CSAM in as little as two clicks on the platform.

The investigation found a number of accounts on the service created with the key purpose of posting AI-generated CSAM of real and/or non-real children. These accounts are followed by hundreds and thousands of users – some with as many as 150,000 followers. The accounts include AI-generated images of children which were recommended to other users through Instagram’s own in-app features.

The investigation found content included:

• Children in sexualised costumes, including AI-generated superhero costumes and ‘fetish’ costumes published on multiple user accounts;
• Images of children in a state of undress; and
• Content of children in a sexual fetish context, with accounts permitted to publish sexualised images of animated characters known to be children.

The investigation found profiles which have shared content of real, underage girls dancing alongside sexual comments acknowledging subjects are underage and/or videos of men explicitly masturbating.

Some accounts also provided external links to where users can create and access AI-generated CSAM for payment. In the most egregious cases, these links lead to websites hosting spaces where groomers were encouraging children to create self-generated CSAM.

4. How effective is the UK's regulatory and legislative framework on tackling these issues?

The UK's current legislative framework on online safety and children’s data protection is one of the most robust in the world.

The Age Appropriate Design Code

In 2018, Parliament introduced the Age Appropriate Design Code[29] (AADC, Children‘s Code or ‘the Code‘) as part of the Data Protection Act, which set out 15 standards digital businesses must abide by when handling children’s data, enforceable by the Information Commissioner’s Office (ICO).

Since it came into force in 2021, companies around the world – including the biggest tech companies in Silicon Valley – have made changes to the design of their platforms in order to make them safer for children.[30] Of particular relevance, the AADC includes the principles that children’s data must not be used to profile them,[31] and only ever processed when it is in their best interest to do so.[32]

The ICO has said that “it is unlikely that the commercial interests of an organisation will outweigh a child's right to privacy”[33] meaning it is unlikely that it would ever be in a child’s best interests to process their data through an algorithm designed to keep them engaged on the platform.

The Online Safety Act 2023

In October 2023, Parliament passed the Online Safety Act. Although many of its provisions are not yet in force, the Act is a positive step towards to creating a safer online world for the UK public.

The strongest powers in Act are given to the protection of children, with additional measures to give adults more agency over the types of content they see on the largest and most high-risk services (Category 1).[34]

The Act also introduces new priority offences, including offences relating to incitement to violence which leads ‘offline‘ harm,[35] as was seen during the riots. 5Rights believes the Act is robust in many respects, particularly for children’s safety online and we support the duty of care provisions and intention for accountability to be places on services.

The Age Appropriate Design Code and Online Safety Act provide a strong legislative framework for children’s safety, particularly as it relates to the design of digital services and products which include the safety and deployment of algorithms. However, while Parliament has set this agenda, we are concerned that the ambition and strength of the legislative framework is not being delivered through regulations and enforcement.

With regard to the Age Appropriate Design Code, we have raised concerns that the Information Commissioner’s Office (ICO) has not been robust in its enforcement of this Code. The ICO has not issued a fine under the Code, despite where we have provided evidence of rank non-compliance.[36] Where the ICO had announced it had sent notices to 11 services for breaches of the Code this year, they chose not to name the services.[37] Companies in breach of the Code being subject to heightened scrutiny is a matter of public interest and is an act of enforcement in itself. The ICO must be more transparent in its enforcement action so tech companies are held accountable for where it is not complying with UK law.

We have also raised concerns that the Online Safety Act codes of practice put forward by Ofcom, do not reflect the intention of the legislation and will leave children exposed to known harm.[38] [39] Ofcom’s proposals do not meet the intention of the Act which is to see that services are made ”safe by design." The majority of Ofcom’s proposals focus on ex-ante measures which cannot claim to meet the definition of safety by design. In addition, the codes function as ’safe harbour’ measures, which means services can simply adopt Ofcom’s proposals to be deemed compliant, even if risks to children are still present on the platform.

• How effective will the Online Safety Act be in combatting harmful social media content?

If the Act had been in force when the riots were taking place, a number of its illegal harms provisions would have been triggered. This includes Schedule 7 (Priority offences) with regards to the incitement to violence online. Indeed, some of the individuals charged in connection with the riots were done so under the Online Safety Act.[40]

However, while the legislation is robust in some respects, there are several key areas where it could, currently, fail to combat harms relevant to this inquiry.

‘Safe harbour’ provisions and algorithms

The Online Safety Act instructs Ofcom to create codes of practice to set out how services must comply with the Act. These codes serve as a ‘safe harbour’ meaning services only need comply with the limited measures[41] Ofcom outlines in the codes. This will mean that even where services have not met the safety duty and people are still coming to harm or facing unacceptable risk, services will still be in compliance with the law.[42]

Specifically on algorithms, Ofcom’s risk register includes considerable evidence of the risk posed by recommender systems, including how algorithms show suicide and self-harm content and eating disorder content to children who have not sought it out.

Ofcom’s own research found 7-in-10 teenage boys had seen content promoting misogynistic views via a recommender system[43] and that children can fall into ‘rabbit holes’ and filter bubbles where their feeds are filled with harmful content and fewer alternative kinds of content are shown.[44]

Despite this, there are limited measures which would fully mitigate or manage the risk from recommender systems. While the Code would require the prominence of harmful content in recommender systems to be limited, this would not address cumulative harm in a robust way. For example, these measures would not address harm caused by concentration and volume of Primary Priority Content (PPC),[45] Priority Content (PC),[46] and 'adjacent' content (e.g. high dosage of dieting, juicing, fitness, weight loss journey videos which could cause harm relating to eating disorders with no counternarrative). There are also no measures which would address ‘filter bubbles’ or ‘rabbit holes’ (i.e. when the way an algorithm is designed pushes users towards increasingly extreme content) as opposed to just removing PPC or PC.

Research has found that children in particular can ‘microdose’ on harmful misogynistic content through algorithms, which normalises these attitudes.[47] This normalisation has found its way into the classroom, with sexual harassment, sexual abuse and safeguarding increasingly being referenced in Ofsted reports.[48]

Misinformation and the risks to children

Another gap is that misinformation is not included in the categories of harmful content for adults or children within the Online Safety Act. This is despite research demonstrating the harm it is having on society and its connection to radicalisation.[49]

Due to their still developing cognitive abilities,[50] including their emotional regulation and moral development, children are uniquely at risk of radicalisation online. According to latest official data, 14% of those arrested for terrorism offences in the year ending 31 March 2023 were 17 and under.[51] Indeed, the UK Government guidance[52] identifies online radicalisation as a growing risk to children.

• What more should be done to combat potentially harmful social media and AI content?

As set out earlier in our response, much of the UK’s existing online safety and data protection legislation, if enforced robustly and fully, could address the issues being investigated within this inquiry. With that said, there are two key areas that must be addressed in our legislative and regulatory framework to further combat harm online:

Amend the “safe harbour” boundary from the Online Safety Act codes of practice

As outlined above, it is Ofcom’s contention is that the wording of Online Safety Act prevents them from enforcing against services that do not fully meet their safety duties. In response to its draft Illegal Harms Code of Practice consultation, Ofcom argued:

“Codes set out the measures we recommend service providers follow to address their risks. We cannot include a generic measure that recommends service providers should remove all risks, as some stakeholders have suggested. The safety duties in the Act only require providers to take proportionate steps and we can only make recommendations we are satisfied are proportionate, having impact assessed them. We cannot assess the impact of a proposal if we do not know what compliance with it would entail.”[53]

Ofcom’s interpretation of the role of the codes of practice actively disincentivises tech companies from finding more effective ways to tackle these harms online. This was not Parliament’s intention and this will not meet public expectation of what this regulation will do.

We acknowledge that there is some value in the regulatory framework that Ofcom has produced to tackle illegal harms and ensure children’s safety, but it currently fails to reflect the purpose of the Act, the intention of Parliament or the needs of children and their parents. 

We recommend reconstituting the ‘safe harbour’ boundary so that services would only be entitled to claim it if they have:

1. Complied with all the measures in the codes; and 
2. Taken steps to address remaining risks on the service. 

This framework would also be adaptable to many different services who will have their own mitigation strategies in place and have very complex design models (see figure below).

Regulate GenAI models and LLMs

The AI tech industry has little to no regulation in the UK, despite the evidence of risk these technologies pose to ,[54] [55] [56] [57] wellbeing[58], and national security[59].  These are not problems of the future; they are the problems of the here and now.

To meet the government's online safety objectives, regulatory authorities must be given the tools and a duty to investigate risky AI technologies such as algorithms, Large Language Models (LLMs) and GenAI models and establish an agreed standard by which to assess them. In some cases, it will be appropriate to create new offences.

Illegal content is still illegal even if its online, but the mechanism by which it can be created is not. For example, the possession and creation of GenAI CSAM is illegal, , the mechanism that enables its creation is not. To address this the Government should introduce a new criminal offence of possessing, sharing, creating or distributing an AI file that is trained on or trained to create CSAM,[60]

Which bodies should be held accountable for the spread of misinformation, disinformation and harmful content as a result of social media and search engines’ use of algorithms and AI?

As set out earlier in this response, the means by which individuals come into contact with harmful content online is not an accident but a consequence of the way tech companies design their systems and products. As Marisa Ressa, the 2021 Nobel peace prize winner explained: “Tech sucked up our personal experiences and data, organized it with artificial intelligence, manipulated us with it, and created behaviour at a scale that brought out the worst in humanity.”[61]

The responsibility for the spread of misinformation, disinformation and harmful content on social media and search engines must sit with those who have built the system and products that allow it to spread unfettered – the tech companies themselves.