Written evidence submitted by the NSPCC (SMHOO32)

SIT Select Committee Call for Evidence - Social media, misinformation and harmful algorithms

This response addresses some of the key themes of this call for evidence. Firstly, it lays out some of the harms children face online due to the business models of tech platforms, such as the financial incentives of algorithms and the lack of resource for product safety and moderation. It outlines the effectiveness of the Online Safety Act in addressing some of the harms online and our concerns about areas it will not tackle. It also addresses which bodies should be held accountable for harmful content.

The NSPCC is one of the UK’s leading children’s charities. Ensuring every child is safe online is one of the NSPCC’s organisational impact goals. We are committed to using our expertise, based on a strong research and evidence base and direct work with children, to advocate for an ambitious regulatory framework which tackles the preventable harms children face online.

To what extent do the business models of social media companies, search engines and others encourage the spread of harmful content, and contribute to wider social harms?

There are number of key ways in which social media companies’ business models are failing to keep children safe. Firstly, in order to increase user engagement, companies use algorithms that often expose children to harmful content such as pro-suicide and misogynistic content. There is also often a lack of product safety testing, in which new tools are designed and brought out by companies without robust assessment of what the impact of them on children’s safety could be. This often happens at the expense of innovation in solutions that would increase safety. Finally, companies often do not properly resource moderation teams that would monitor and remove harmful content due to the financial investment required.

Feeding harmful content through algorithms 

Increasing user engagement and exposure to content that might interest them is a central part to the business model of social media companies. By automatically presenting users with content that interests them based on their viewing habits, personal data, and ‘likes’, users spend more time on platforms[1]. This is fundamental to the business model of social media companies who then use data generated by interests to target them with relevant advertisements, a critical part of their revenue stream.[2]

Ofcom is clear that algorithms, or recommender systems, can pose a significant risk to children online.[3] Algorithms have been shown to promote a range of illegal and harmful content to children and young people, including suicide, self-harm, and eating disorder content; sexual content; misogynistic content; and violent content. For example:

• The Centre for Countering Digital Hate found that a 13-year-old child could be recommended suicide content within 2.6 minutes of being on TikTok.[4]
• Research from the Molly Rose Foundation found that algorithms enable dangerous content to have a far reach; over half of the most engaged harmful posts (relating to suicide and self-harm) they surveyed on TikTok had been viewed over 1 million times.[5]
• On X, recommender systems have shared videos of children being sexually assaulted and prompted users to follow content relating to exploited children.[6]

Young people have raised that they often feel they lack control over what they see online, and that algorithms push content that they do not want to see or engage with.

“Twitter has been recommending me posts about a manga cartoon series I’m really into, however some of the posts really unnerve me. Some show the students from the show in sexual scenarios with their teachers or with other students. The students in the show are, like, 15- 16, which makes me uncomfortable because they’re minors.” Girl aged 18, Childline[7]

There is a particular risk for children already struggling with their mental health. Illegal or very harmful content is often viewed alongside less serious but still harmful content which contributes to the cumulative impact. Studies have shown that algorithms have caused harmful content to be automatically targeted to users who self-declared as children. Cumulative active engagement with hazardous content, such as active membership of pro-anorexia or extremist communities online, leads to significant and severe harm.[8]

In recent years, algorithms have had a concerning impact in pushing misogynistic content. Influencers such as Andrew Tate, who has promoted coercive and violent behaviour towards women, have often being suggested to young men and boys. An Observer investigation showed that users would often be recommended Andrew Tate videos on TikTok after viewing non-harmful content generally aimed at men.[9]

In so many cases, evidence clearly shows that children do not seek out harmful content, but rather it is shown to them on their ‘feeds’. In a recent study into teenage children’s experiences of violence on social media, the Youth Endowment Fund found that 70% of respondents had encountered real-world violence online in the last year even though very few actively sought it out. [10]

Whilst this call for evidence is particularly focused on violence, it is important to note that children and young people (and online users more generally) encounter a range of harms that are produced by the same algorithms. Whilst there are periods where specific types of content are particularly prevalent – such as violent content, or misogynistic content – addressing the drivers of algorithmic harm holistically is key to building safer platforms.

Lack of investment in safety

Tech companies have regularly rolled out new features and tools on their platforms without necessary safety precautions, as well as having failed to invest in user safety. The failure of tech platforms to provide sufficient measures, including having sufficient moderation teams in place to monitor harmful content on their platforms, is often a financial decision.

It is vital that features and functionalities are not only assessed in relation to the content that they push, but the behaviours they impact. The young people we have consulted have consistently emphasised that mechanisms which increase their engagement on a platform (such as streaks for regular use and endless scrolling) – a core part of service’s business model – negatively impact them because they are encouraging an overuse of the platform. Ofcom evidence has shown the harms caused by features that increase user engagement.[11]

Children have been found to be more susceptible to metrics that demonstrate their relative popularity, e.g. the number of likes on a post or followers to their accounts.[12] The NSPCC’s latest evidence review identified quantification of social activity and popularity as one of the main features which can increase online risk and harm to children.[13] In the most concerning cases, many of these tools have facilitated unwanted requests for sexual interactions. Social incentives to build networks by adding ‘friends’ on social media platforms has meant some children have connected to unknown adults who may target them for abuse.

Given the scale of harm to children, we would expect action to be taken by platforms to address these issues. However, some of them have failed to engage with evidence of risks on their own services. These have included:

• Whistle-blower Frances Haugen shared internal research by Facebook which had found that Instagram was negatively impacting the mental health of teenagers, including making girls feel worse about their bodies. Rather than acting on this research, Facebook buried it.[14]
• Arturo Béjar, previously an employee at Meta, argued the lack of transparency about the harms teenagers experience on Instagram meant they failed to base decisions in data about user’s experiences and the safety settings on Instagram did not address the root causes of risk on the platform.[15]

Cost-saving measures must not come at the expense of user safety. Meta’s cuts last year to its Trust and Safety team came as part of the company’s “year of efficiency”.[16] X has also reduced its headcount in Trust and Safety by one third over the last two years.[17] AI tools do not yet have the capacity to moderate large social media companies so, in the meantime, human moderation is still required for many categories of illegal and harmful content and to verify decisions made by AI moderation.

It is clear that not enough is being done by platforms to address known harms. Key areas requiring greater investment are product testing and proactive content moderation. Services should utilise their expertise and financial capacity to invest in protecting children against preventable harms. The pro-innovation agenda that many tech companies promote should be applied to developing new solutions to ensure children do not have to face harmful and dangerous content. 

What role do generative artificial intelligence (AI) and large language models (LLMs) play in the creation and spread of misinformation, disinformation and harmful content?

Generative AI (Gen AI) and large language models (LLMs, a subtype of Gen AI) can both be used to harmful content which can be created directly by children as a result of their prompts to Gen AI systems, and by various actors seeking to spread harmful content which impacts children. Gen AI is capable of creating image-based harmful content that is difficult to distinguish from reality, increasing its overall impact.

AI-generated harmful content is primarily distributed on social media platforms and on messaging systems. The growing proliferation of AI-generated content on these channels increases the chance that children will consume harmful content passively without realising it is AI-generated. While harmful content has always existed on these sites, AI means that the scale of the problem is now much larger and is growing rapidly.

“A group of boys at school used deepfake to make a video of me saying I’m gay. They’ve made fake chat screenshots of me saying I want to do sexual things to them as well. I have questioned my sexuality but haven’t come out to anyone, that doesn’t stop the bullies though. I want to tell a teacher but it’s my word against all these other boys.” Call to Childline from a boy, aged 14

“Can I ask questions about ChatGPT? How accurate is it? I was having a conversation with it and asking questions, and it told me I might have anxiety or depression. It’s made me start thinking that I might?” Call to Childline from a girl, aged 12

Both open-source and commercial Gen AI models are liable to create disinformation and misinformation. Open-source models allow disinformation actors to create content without model developers being able to monitor it. As a result, those engaging in disinformation campaigns are less likely to be relying on closed models; individuals can train open-source models to their ends and remove safeguards.[18]

In terms of regulating this harmful content, the scope of current online safety legislation may not capture standalone Gen AI models and instead only those Gen AI features incorporated on social media and search services. This gap leaves children (and indeed, all internet users) vulnerable to this content.

While there are potential solutions to AI-generated harmful content currently available, this risk demands a long-term strategic approach which adapts to the risk as it evolves over time.

How effective is the UK's regulatory and legislative framework on tackling these issues?

We welcome the introduction of the Online Safety Act, which is a vital child protection measure that puts the onus onto tech companies to keep children safe online. We worked closely with the Government and Parliamentarians to shape the Act as it went through Parliament, and it is crucial that Ofcom are robust in their implementation to ensure the right protections are in place so children can safely enjoy the online world.

To date, the (draft) Codes of Practice published by Ofcom contain some important measures that will help tackle some of the issues raised in this inquiry – this includes provisions relating to carrying out robust risk assessments, introducing safety testing for recommender systems, and strengthening content moderation efforts. However, there are a number of areas relevant to this inquiry where Ofcom must go much further and be more ambitious in their approach.

There are also limits to the scope of the Online Safety Act itself. Most notably, it does not cover the creation of AI-generated content on AI services. As a result, for example, the creation of deepfakes is out of scope of the regulation.

This section will explore the effects of the Online Safety Act on key areas relevant to the inquiry and some of our main concerns about what it may fail to tackle.

Innovation

As technology changes, it is important that the bar continues to be raised for companies to invest in innovative solutions that can help keep children safe online. The Online Safety Act states that if a service provider follows all measures in a Code of Practice, it will be deemed as compliant with the regulation – making the Codes a ‘safe harbour’ for services. However, significant limits in Ofcom’s Codes mean that a service could technically follow all measures, and so be compliant overall, but continue to host major risks to users. As a result, there is a significant risk that internal decision-makers may favour rolling out older technologies recommended in the Codes over new, innovative measures, regardless of how impactful they are. This disincentivises innovation and makes it less likely that services will quickly adapt to emerging harms on their sites.

There are a number of steps we suggest that Ofcom take to support innovation that improves safety, including using more outcomes-based measures in the Codes to require services to develop new solutions, sharing best practice, and shining a light on poor performance so that where services take minimal action and fail to address all the harms on their site, there is a reputational risk.

We also recommend the Committee considers the impact that the ‘safe harbour nature of Codes’, as prescribed in the Act, has on innovation by regulated services.

Private messaging

We often hear from children and young people that harmful content does not only come from public forums, but often from less visible private ones. One of our primary concerns with the Online Safety Act is the lack of requirements for private and end-to-end encrypted (E2EE) services. Certain exemptions may mean expectations for these services will be severely limited, and large platforms such as WhatsApp will not need to introduce any substantial changes. As a result, we have serious concerns that some of the harms online will merely be moved from public to private spaces.

Private messaging services have exposed children to a range of harms including bullying and abuse, exposure to self-harm and suicide content, and exposure to sexual content.

“A guy I used to date has been spreading false rumours about me on WhatsApp, saying I’m on drugs and I’ve had sex with loads of boys. There are loads of comments from people I don’t even know calling me a ‘slag’ and a ‘crack head’. I try to just ignore it but then I get so paranoid walking round school, wondering what people are thinking about me. It’s so stressful and I don’t know what to do.” Call to Childline from a girl, aged 16

“My so-called friend added me to this WhatsApp group chat, where people were saying really horrible things about me, like my parents don’t love me and that I should kill myself. At first, I thought it was just some sick joke, but then I realised it wasn’t. I kept thinking, why are they saying these things, it doesn’t make sense?! I’ve now blocked this friend, but I don’t know how I’m meant to get passed this, like mentally. I feel so hurt, angry and empty right now.” Call to Childline from a girl, aged 16

Private messaging is also the frontline of online grooming. Data from ONS shows that 74% of approaches to children by someone they do not know online first take place via private messaging.[19] These platforms are exploited by offenders, who move children that they have met on more public platforms to these sites for exploitation. This is because of the well-known challenges of detection CSEA in E2EE platforms. The NCA have highlighted that the roll out of E2EE on Facebook Messenger and Instagram could mean that the alerts they receive from Meta via NCMEC, which enable them to find perpetrators and safeguard children, could fall by 92%.[20]

Currently, harmful content can spread on private messaging services which platforms have shielded themselves from moderating due to the roll out of end-to-end encryption. Much of the public discourse on harm online focuses on public platforms as they are, by definition, more visible than private messaging services. However, it is clear that many children face risks in private forums where the scale of the harm is hard to measure, and current legislation may struggle to tackle it.

This Committee should consider the harms of private messaging services as well as those on public platforms when making conclusions or recommendations on this topic.

Which bodies should be held accountable for the spread of misinformation, disinformation and harmful content as a result of social media and search engines’ use of algorithms and AI?

The Online Safety Act makes clear that tech companies must be responsible for tackling harm on their platforms, and we welcome this approach. It is now the role of Ofcom to ensure that it effectively implements the Act, and is willing to challenge companies that do not fulfil their obligations. Provisions in the Act, such as the liability of senior management and the potential to fine companies as much as 10% of their global revenue, are important enforcement tools which Ofcom must use swiftly where companies fail to comply.

As well as the role of Ofcom and tech companies themselves, the Government and Parliament will be vital in making sure that Ofcom is doing all it can within the scope of the Act to improve safeguards. The recent Statement of Strategic Priorities set out positive intentions from the Government towards the regulator which should guide Ofcom to be ambitious in its implementation of the Act. Many of the areas that the NSPCC wants to see in the enforcement of the Act, such as safety by design and agile regulation, were included in the statement, helping to empower Ofcom license raise the bar for tech companies to improve safety.

The passage of the Online Safety Act involved cross-party support and Parliament can continue to be an important advocate for the prevention of online harms. We welcome this call for evidence on this subject to better understand the harms that exist and the role of algorithms in spreading them. We would welcome the Science, Innovation and Technology Committee dedicating more time to child safety online, and using sessions to hold Ofcom accountable on their implementation of the Act to better understand their approach and progress.

18 December 2024