Written evidence submitted by the Molly Rose Foundation (SMH0016)

Molly Rose Foundation (MRF) welcomes the opportunity to respond to the Science, Innovation and Technology Committee’s inquiry call for evidence. MRF was established following the death of 14-year-old Molly Russell in November 2017. At Molly’s inquest, the coroner determined that exposure to harmful online content ‘contributed to her death in a more than minimal way.’

MRF focuses on suicide prevention, with a particular emphasis on technology-facilitated harm. In this submission, we set out:

-          our understanding of the ways in which the business models and systematic design choices of social media platforms contribute to preventable online and societal harms;

-          our assessment of the impact of social media algorithms and other user engagement features, including their central role in the dissemination and spread of harmful online content;

-          our concerns that Ofcom’s choices when implementing the Online Safety Act are likely to substantially limit the impact of the U.K.’s regulatory response. In light of Ofcom’s Code of Practice on illegal harms, and its draft Code relating to children, MRF identifies structural weaknesses with the regulatory framework and asserts that a new Act is urgently required.

1. The contribution of social media business models to risk profiles

In MRF’s assessment, the spread of harmful online content is primarily driven by the business models and systemic design choices of social media platforms. Their business model is fundamentally geared towards the maximisation of user engagement on their services - an expression of the so-called ‘attention economy’.

In practice, this means the commercial strategy of tech firms typically emphasises key performance metrics such as time spent on their platform; Daily and Monthly Active user rates; and whether new design features maximise and extend time spent on and user engagement with their products.

In this context, safer and more responsible design choices have been de-emphasised; and we have at best typically seen piecemeal design and safety improvements in response to high-profile tragedies such as Molly’s, and a result of sustained media and political pressure.

In effect, the business models of social media platforms have been geared towards externalising the costs of harms to individuals and society, including in the form of rapidly growing societal and user harms.

Research shows that technology plays a role in around one-quarter (24%) of deaths by suicide among young people aged 10 to 19, equivalent to one young life lost each week.[1] MRF analysis estimates that the social and economic cost of internet-related deaths by suicide among young people is £486 million per year (in 2024 prices.)[2]

Separately, DSIT’s recently impact assessment shows that a 10 per cent reduction in exposure to online harm could result in annualised social and economic benefits of £2.5 billion per year.[3]

This suggests that, if we were to more effectively assert the ‘polluter pays’ principle, and in turn put the onus more directly on tech companies to mitigate harm through a set of ‘safety-by-design’ regulatory requirements, we could reasonably expect to see a substantial reduction in the costs currently borne by individuals and society. This includes costs currently borne law enforcement, the NHS, and in the form of adverse impacts to the safety, wellbeing, and life chances of young people.

There is considerable evidence to demonstrate a significant causal relationship between the commercial models of large social media sites and their resulting risk profile. Most major platforms have typically treated Trust and Safety primarily as a cost centre. Meta warns investors in its annual SEC findings that, ‘if we fail to retain existing users or add new users, or if our users decrease their level of engagement with our products, our revenue, financial results, and business may be significantly harmed.’ [4]

There is evidence that suggests Meta has tolerated safety improvements only where there is no appreciable impact on user engagement rates. For example, legal disclosures show that Instagram opted not to proceed with the rollout of Project Daisy, which would have removed ‘like’ counts from the feeds of under 18s, because it would incur a 1% decline in advertising revenue. This was despite internal research showing that seeing content with algorithmically driven extreme like counts (1 million + likes) resulted in significant adverse effects on youth mental health.[5]

By early 2019, Meta’s leadership team were explicitly warned there was that there was a causal relationship between its product design and exposure to harmful self-harm and suicide content, with an internal report shared with members of its Leadership Team finding there was a ‘palpable risk of similar incidents [to Molly’s death]’ because of how its algorithms recommended large amounts of harmful content.[6]

MRF research has found a direct relationship between the commercial significance of parts of Instagram’s product and the likelihood of being algorithmically recommended harmful content on it. For example, as part of our research into the nature and prevalence of suicide, self-harm and harmful forms of depression content on Instagram, we were served more harmful content (99% of all recommended posts) when watching Reels than on any other part of Instagram’s platform.[7]

Instagram has repeatedly stated it identified Reels as a priority area in its efforts to maximise user engagement rates. In its quarterly earnings calls, Meta consistently highlights the performance of Reels against its preferred user engagement metrics. For example, in the company’s Q4 2023/24 call Mark Zuckerberg stated that Reels now accounts for half of all time spent by users on the platform.[8]

2. The role of algorithms and other engagement features in the spread of harmful content 

There is substantial evidence to suggest that platform algorithms are responsible for promoting large amounts of harmful content to social media users; and that regulated services will continue to make systemic design choices despite the reasonably foreseeable risk these decisions will contribute to individual and societal level harms.

MRF research into the most engaged social media posts, shared using well-known suicide and self-harm hashtags, found that almost half of posts on TikTok (49%) and Instagram (48%) contained material that promoted or glorified suicide and self-harm, referenced suicide ideation, or otherwise contained intense themes of misery, hopelessness and depression.[9]

As a result of algorithmic amplification, many of these posts had recorded extraordinary and deeply disturbing levels of reach. For example, more than half of the posts we identified as harmful on TikTok (54%) had received over 1 million views. Half of these posts (51%) had been liked at least 250,000 times, while one in eight (12%) had been liked over 1 million times.

Our analysis suggests that much of this content reflects the failure of platforms to effectively moderate content according to their terms of service. We found a substantial amount of clearly violative content on Instagram, with its failure to rapidly or effectively remove such content meaning it could then be algorithmically recommended to users at scale.

We also identified a substantial risk of cumulative harm, whereby users may experience significant adverse effects because of being algorithmically exposed to large concentrations of harmful content and/or combinations of such content. As in Molly’s case, this will likely discrete items of content that may not be harmful in isolation, but that could foreseeably contribute to harmful effects when viewed in large amounts.

Examples of harmful combinations of content categories may include posts that reference suicide, self-harm and intense themes of emotional distress, hopelessness and despair.[10]

Recent research suggests that in cases where algorithms reinforce and amplify depression-related search preferences, this can lead to a vicious circle in which children are more likely to interpret content more negatively (as a result of their cognitive biases being reinforced); to experience negative rumination about content and/or negative experiences they’ve had; and in turn to require potentially harmful forms of online reassurance and approval, which may then leave them increasingly susceptible to approaches from a range of malign actors.[11]

We also have emerging concerns about the ways in which young boys are being recommended combinations of mental health, misogynistic and far-right content, in ways that clearly may contribute towards harmful individual and societal level effects. [12]

While personalised recommender systems are the primary driver of young people’s exposure to harmful content, we also wish to draw attention to a broader range of user engagement and content discoverability features that may also contribute towards adverse effects on the safety, well-being and mental health of young people.

For example, in our research into the nature and spread of harmful content on major social media platforms, we identified a broad range of high-risk user engagement mechanisms that can contribute towards so-called ‘rabbit hole’ effects.

These include TikTok’s use of auto-complete suggestions for search terms; hashtag recommendations at the end of videos; and bundled lists of recommended themes and hashtags that are interspersed into search results. On an account used to explore mental health related content, TikTok recommended a range of potentially harmful hashtags and search results to us (‘people also search for ‘quickest way to end it’, ‘others searched for ‘I feel like I’m drowning’ and ‘I don’t feel I’ll be here much longer.’)[13]

We also wish to draw attention to how personalised recommender algorithms can contribute towards secondary mechanisms that may increase the risk profile on many sites.

While platform recommender algorithms primarily operate to recommend personalised content to users, they also have the effect of enabling users to identify and form networks with other accounts with similar interests (so-called ‘assortative relating’ effects).[14] While in some situations this can have protective or beneficial effects, there are also substantial risks that this can be exploited and/or result in unintended consequences. For example, these mechanisms can make it easier for malign actors to identify and make contact with vulnerable users, including for reasons related to sexual, sadistic or self-harm related grooming.[15]

3. The Online Safety Act and the likely effectiveness of the UK’s regulatory framework

MRF has significant concerns about the likely impact of the nascent regulatory regime.

Over the last year, Ofcom has set out how it intends to implement its regulatory scheme, and in December 2024 it published its first Code of Practice covering illegal content. While the regulator’s proposals contain some important and welcome measures, its overall approach is deeply unambitious and are unlikely to be effective in substantially reducing the velocity and virality with which illegal and harmful content is able to spread.

Ofcom’s choices when implementing the Act have ultimately exposed deep systemic weaknesses with the regime. The Act’s design means that regulated platforms are granted a ‘safe harbour’ if they adopt the measures set out in Ofcom’s codes, but the codes themselves are so weak that some large platforms could counterintuitively scale back their existing largely ineffective and highly deficient safety measures.

In our assessment, a new Act is urgently required to address these issues – and to refocus the regime on clear, sustained improvements in harm reduction. We recommend that the Government urgently commits to introducing a new Act that can:

Reassert an overarching Duty of Care:

The distinct structural and evidential barriers that are preventing Ofcom developing suitably ambitious Codes are a major drag on the current regime’s effectiveness.

A second Act can introduce an overarching Duty of Care and would place the primary onus on tech firms to identify and respond to harms caused on their sites, rather than in effect putting responsibility on the regulator to develop inherently reactive and prescriptive codes.

Introduce a new harm reduction duty:

Ofcom should be subject to a new duty to deliver annual, measurable improvements in harm reduction, with this clear and unambiguous duty re-centring the regime and adding much-needed urgency and ambition to Ofcom’s approach.

As it stands, the regulator has declined to offer any assessment of the impact of its regulatory scheme, nor has it published any criteria or outcome measures that it intends to use to determine the impact of its initial Codes;

More robustly tackles the risks posed by personalised recommender systems:

As it stands, Ofcom’s approach to tackling the algorithmic spread of harmful content is entirely insufficient. Under Ofcom’s proposals, platforms likely to be used by children will be required to prevent or restrict the algorithmic recommendation of Primary Priority and certain forms of Priority Content, but this can only be enforced where regulated services had prior knowledge that content was present and likely to be harmful.

In practice, this means platforms can only be held liable where sufficient content moderation arrangements currently exist – but in most cases, platform moderation efforts are unacceptably poor. While no major platform currently publishes their ‘leakage rates’ (the amount of harmful content that is missed), recent MRF analysis has found that there is significant under-moderation of suicide and self-harm content on many major sites.[16]

Furthermore, major platforms have failed to invest in adequate moderation technologies, and this gap is typically most pronounced on the highest-risk and most frequently used parts of their services. For example, in the case of Instagram and TikTok, at best one-fifth of violative suicide and self-harm content is detected on video or image-based content.[17]

In Ofcom’s illegal codes, the regulator will require companies to test the safety implications of its recommender systems, but only if the platform already undertakes on-platform testing. According to the regulator’s risk assessment framework, product safety testing is classified as an enhanced rather than a core output, meaning it will not be required in every case.

Offers a strong and coherent response to the risks posed by AI-generated content, including chatbots:

MRF has significant concerns about the potential impact that generative AI may have on the risk profile of most major social media sites. Generative AI is significantly reducing the cost and technical barriers to reduce harmful content, and we are already seeing AI generated posts that promote suicide and self-injury.

While generative AI content is in the scope of the Online Safety Act, the first iterations of Ofcom’s draft codes of practice do not specify specific measures that platforms should take to address the risks of it being used to produce or share harmful content.

MRF is concerned about the likely impact that downstream regulation can likely have while the Government signals it is unlikely to proceed anything more than light-touch regulation of upstream foundation models.[18] It seems almost inevitable that the commercial imperatives to introduce AI functionality will, in most if not all major regulated companies, take primacy over any assessment of regulatory and/or compliance risk stemming from the current iteration of the OSA.

We are also deeply concerned about the potential risks posed by AI-generated chatbots. Following the discovery of AI chatbots in the persona of Molly and Brianna Ghey on the platform Character.AI,[19] and despite extensive engagement with Ofcom, the regulator has been unable to satisfactorily resolve ambiguity about how the Online Safety Act may apply, and in particular whether AI-generated chatbots could trigger the illegal part of the regime.

The Independent Reviewer of Terrorism Legislation Jonathan Hall has previously questioned whether an AI-generated chatbot could trigger the illegal scheme,[20] largely as a function of the application of the mens rea test built into Ofcom’s Illegal Content Judgement Guidance.   This suggests a potential loophole whereby the illegal content scheme could be triggered if comments were made by a person, but not if the exact same comments were generated by a chatbot.

Given the range of harmful applications that have already been observed,[21] any potential legal ambiguity must be rapidly resolved.

17 December 2024