OFFICIAL

Written evidence submitted by The Department for Science, Innovation and Technology (DSIT) (SMH0061)

 

Name of Inquiry: Social media, misinformation and harmful algorithms
 

 

Introduction

The Department for Science, Innovation and Technology (DSIT) is submitting evidence to the Committee inquiry on behalf of the government. DSIT was responsible for the development and passage of the Online Safety Act (OSA), which introduces a range of measures to tackle online harms. The department is now working on: the implementation of the new laws; international engagement with countries and institutions around the world to promote a safer online environment; and leading the department’s work on protecting from mis- and disinformation and AI-generated threats.

In the course of carrying out its responsibilities, the department has considered the questions that the Committee welcomes submissions on and is submitting evidence on the requested points raised.

 

Summary

Online platforms provide citizens with services and opportunities which can be of great benefit. However, some online platforms can also be used in ways that can cause harm. The government is clear that we must address online content and behaviour that can lead to harm for users or society.

While the primary responsibility for harmful social media content rests with those individuals and groups who create and post it, social media platforms have a responsibility to ensure that those seeking to spread harm online are not being facilitated by the platform or service. A platform’s commercially-focused design choices, such as their algorithms for promoting content, may result in harm as a by-product. For example, platforms may start amplifying posts because they drive engagement, which may contain harmful content.

The government is committed to implementing the Online Safety Act (OSA) to introduce protections for children and adults online. The OSA puts new duties on social media companies and search services, making them more responsible for their users’ safety on their platforms. Ofcom is responsible for ensuring platforms fulfil their duties under the OSA.

The government has high expectations, on behalf of this country, that companies should ensure that safety is baked in from the start. Companies releasing products into our society should see that as a privilege, not a right. This is why in the government’s draft Statement of Strategic Priorities for online safety, published on 20 November, safety by design is set out as the first of five key priorities.

Under the OSA, providers will have duties to implement systems and processes to reduce risks their services are used for illegal activity, and to take down illegal content when it does appear. Illegal content includes content relating to: racially or religiously aggravated public order offences; inciting violence; selling illegal drugs or weapons; illegal disinformation; illegal foreign interference; and terrorism. The list is longer, but these content types are most relevant to the Committee’s inquiry. Platforms will also be required to prevent children from accessing harmful and age-inappropriate content, and provide parents and children with clear and accessible ways to report problems online when they do arise.

During this summer’s unrest, the government worked with the major platforms to tackle content contributing to that disorder. This included proactively referring content for platforms to assess and act on, in line with their terms of service.  Ofcom have been using the findings from the public disorder in the summer to inform engagement with regulated services ahead of the duties coming into effect in 2025.

Huge opportunities offered by AI come with risks that could threaten global stability and undermine our values. We recognise the concerns around AI models generating large volumes of content that is indistinguishable from human generated pictures, voice recordings or videos.  This includes the challenge of a broader range of actors being able to produce this content at scale at a lower cost than previously. To better understand these challenges, the government has established a Central AI Risk Function (CAIRF), which brings together policymakers and AI experts with a mission to continuously identify, assess and prepare for AI associated risks. The regulatory framework established by the OSA is designed to be future facing and technology-neutral. Regulated services will be required to remove content which is shared on their services if it is illegal or harmful to children, whether it is real, or AI generated. This can include misinformation and disinformation.

Beyond regulation, the government is working to equip both children and adults with the knowledge and skills to navigate the online world. Media literacy can help tackle a wide variety of online safety issues for all internet users, including children. It means understanding that online actions have real-world consequences, critically evaluating online information, and contributing to a respectful online environment. It is a key tool to build people’s resilience to misinformation and disinformation (including AI-generated deepfakes). 

DSIT is developing an evaluation framework for monitoring the implementation of the OSA and the core outcomes. This monitoring and evaluation work will track the effectiveness of the online safety regime over time, as duties come into effect, and feed into a Post Implementation Review (PIR) of the OSA. The PIR will assess the effectiveness of the legislation against its primary objectives, including how the OSA has addressed harmful online content.

Government continues to identify areas where we can build on the OSA.  Nothing is off the table when it comes to keeping citizens safe. It is important policy remains evidence-based, and we very much welcome the Committee’s inquiry to add to the evidence and information which the government has available, when considering future policy.

Questions

Question 1: To what extent do the business models of social media companies, search engines and others encourage the spread of harmful content, and contribute to wider social harms.

 

In many cases, online services’ business models may be based on increasing user engagement. Increased user engagement may allow these services to achieve certain commercial objectives, for example relating to selling information or space related to advertising.

 

Where platforms design their services to increase user engagement, they may fail to pay due regard to user safety. For example, some online platforms may amplify certain posts because they drive engagement, with little thought given to users’ safety. 

 

This government is clear that social media companies must do more to keep people safe on their platforms. Through the OSA, the government will address the spread of illegal and harmful content, by imposing new duties on providers.

 

Providers will have duties to implement systems and processes to reduce risks their services are used for illegal activity, and to take down illegal content when it does appear. Illegal content includes content relating to: racially or religiously aggravated public order offences; inciting violence; selling illegal drugs or weapons; illegal disinformation; illegal foreign interference; and terrorism. The list is longer, but these content types are most relevant to the Committee’s inquiry.

 

Platforms will also be required to prevent children from accessing harmful and age-inappropriate content, and provide parents and children with clear and accessible ways to report problems online when they do arise.

 

In December 2024, Ofcom prepared draft codes of practice that recommend the practical measures that providers can put in place to fulfil their ‘illegal content safety duties’, including their duties to take steps relating to the design and operation of their service to reduce the risk of users encountering illegal content. Ofcom is also required under the OSA to prepare codes of practice that recommend the practical measures that providers can put in place to fulfil their ‘child safety duties’. In its draft codes, Ofcom has proposed steps companies could take, including requiring algorithms to filter out harmful content for child users. We expect the illegal content duties to be in force by Spring 2025, followed by the child safety duties in Summer 2025.

Question 2: How do social media companies and search engines use algorithms to rank content, how does this reflect their business models, and how does it play into the spread of misinformation, disinformation and harmful content?

 

Some social media platforms use algorithms or content recommender systems. Through these automated decision-making procedures, online platform services can select and promote content for many millions of users - tailored to them individually via an automated analysis of their viewing habits and preferences.

This can help platforms drive customer engagement. In turn, increased customer engagement may then help providers achieve certain commercial objectives. For example, these objectives may relate to selling user information and online space to advertisers.

A platform’s commercially-focused design choices, such as their algorithms for promoting content, may result in harm as a by-product. For example, platforms may start amplifying certain posts because they drive engagement. When doing so, these services may not pay due regard to users’ safety.

We have taken this into account under the OSA regulatory framework. The OSA requires service providers to assess how the design of their services is linked to illegal content, and therefore to the spread of illegal content such as illegal abuse and certain kinds of illegal disinformation. They will have to look at how their services use algorithms, allow users to forward or share content, or to express views on other users’ content.

In its codes of practice, Ofcom will set out steps providers can take to tackle illegal content and content which is harmful to children. These steps will include, where appropriate, content-neutral design choices or interventions relating to algorithms.

Ofcom prepared final versions of its illegal content codes of practice in December 2024. These set out that providers should run thorough tests on their recommender systems to make sure they are not serving illegal content. Providers will also need to have effective user-reporting and content moderation systems in place. This means they should take down any illegal content when they become aware of it, and remove it from their recommender systems.

The government acknowledges the impact algorithms can especially have on the risk of harm to children. To reflect this, the OSA will require services that are likely to be accessed by children to specifically consider how its algorithms, features and functionalities may impact children’s exposure to harmful content as part of their children’s risk assessment duties.

Adult users will also have control over certain types of misinformation and disinformation content which fall in scope of the user empowerment duties. This includes misinformation and disinformation which amounts to suicide content, content that promotes self-harm, content that promotes eating disorders, and content that demonstrates hostility on the basis of race, ethnicity, religion, disability, sex, gender assignment or sexual orientation.  

 

They will be able to freely choose whether to access this content (unless it’s already prohibited under a company’s terms of service), choosing to ‘turn on’ the user empowerment tools or not. Platforms will be required to proactively ask their registered adult users whether they wish to use the user empowerment tools at the first possible opportunity and ensure these tools are easy to access. 

 

Question 2a: What role do generative artificial intelligence (AI) and large language models (LLMs) play in the creation and spread of misinformation, disinformation and harmful content?

 

Huge opportunities offered by AI come with risks that could threaten global stability and undermine our values. We recognise the concerns around AI models generating large volumes of content that is indistinguishable from human-generated pictures, voice recordings or videos.  This includes the challenge of a broader range of actors being able to produce this content at scale at a lower cost than previously. To better understand these challenges, the government has established a Central AI Risk Function (CAIRF), which brings together policymakers and AI experts with a mission to continuously identify, assess and prepare for AI associated risks. The remit of the risk function is broad as it focuses on all AI systems, including Large Language Models (LLMs), narrow models and wider impacts of AI adoption and investment. This will include exploring solutions for enabling users, and institutions, to determine what media is real and what is AI-generated as a key part of tackling a wide range of AI risks.  

 

The government recognises the challenges that AI-generated content can pose for the information environment, including its potential impact on democratic processes. The use of Generative AI during the UK 2024 General Election was however not at the scale anticipated. The Alan Turing Institute’s Centre for Emerging Technology and Security have reported they found “no evidence that AI-enabled disinformation or deepfakes meaningfully impacted the UK or European election result”.

 

In preparation for the 2024 General Election, teams worked closely across government to ensure we were ready to respond to threats from AI-enabled election interference. This includes the Election Cell which coordinated a wide range of teams across government to respond to issues, including AI-generated disinformation. We are also working closely with our international partners to tackle this shared challenge. The AI Seoul Summit in May 2024 reinforced the government’s commitment to international collaboration on the risks of frontier AI being used to generate mis/disinformation. 

 

The regulatory framework established by the OSA is designed to be future-facing and technology-neutral. Ofcom will take an iterative approach to its guidance and codes of practice, updating them as necessary to keep pace with new and emerging technologies such as generative AI and LLMs. Regulated services will be required to remove content which is illegal or harmful to children, whether it is real, or AI-generated. This includes misinformation and disinformation content. Additionally, for the largest in-scope services, they will be required to remove harmful content (including AI-generated content) where it contravenes terms of service. 

Question 3: What role did social media algorithms play in the riots that took place in the UK in summer 2024?

 

The primary responsibility for harmful social media content rests with those individuals and groups who create and post it. We saw prosecutions and convictions for illegal online activity during the summer riots, some of the first convictions of this kind.

 

However, social media platforms have a responsibility to ensure that those seeking to spread hate online are not being facilitated by the platform or service, and that they have nowhere to hide. During this summer’s unrest, we worked with the major platforms to tackle content contributing to that disorder. This included proactively referring content for platforms to assess and act on in line with their terms of service. The government regularly engages with major social media platforms to support their efforts and ensure they remain accountable.

 

Under the OSA, in-scope services will be required to take steps to prevent the spread of illegal material and to mitigate the risks to the safety of people in the UK arising from illegal online activity, and content and conduct which harms children. These duties were not in effect during the summer public disorder but Ofcom’s proposed illegal harms codes and risk assessment guidance set clear expectations. User-to-user services should:

 

Ofcom have been using the findings from the public disorder in the summer to inform engagement with regulated services ahead of the duties coming into force in 2025. As part of Ofcom's consultation on additional measures for the codes in spring 2025, they will bring forward a new measure that would describe additional steps for services to take regarding their crisis response protocols. Our message to social media companies has been clear: act now rather than wait for the legislation to take effect. 

 

The OSA also seeks to hold user-to-user services over a designated threshold, which may include social media platforms, to account by ensuring that they clearly and consistently uphold their terms of service. Under the OSA’s terms of service and accountability duties, if certain types of hateful content, as well as mis- and disinformation, are prohibited in the largest platforms’ terms of service, they will have to remove it.

 

Question 4: How effective is the UK's regulatory and legislative framework on tackling these issues?

Question 4a: How effective will the Online Safety Act be in combatting harmful social media content? 

 

Requirements on services

 

In addition to the duties relating to illegal content discussed in answers to previous questions, the OSA will place requirements on user-to-user and search services which are likely to be accessed by children to protect child users from a range of types of harmful content which do not meet a criminal threshold, but nonetheless present a material risk to an appreciable number of children in the UK. These requirements are known as the child safety duties.

 

The child safety duties require service providers to prevent children from encountering ’primary priority content’ on user-to-user services through the use of highly effective age assurance. The OSA defines ‘primary priority content’ as pornography; content that encourages or provides instructions for self-harm; eating disorders; and suicide. 

 

Providers must also take age-appropriate measures to protect children from the harms of ‘priority content’.  There are six categories of priority content, some of which relate to material which might circulate online during civil unrest, such as ‘content which incites hatred against people of a particular race or religion’ or ‘content which encourages an act of serious violence against a person’.

 

Providers will need to protect children from harmful mis- and disinformation where it intersects with these named categories of harmful content: for example, an online challenge which is promoted to children on the basis of mis- or disinformation, or abusive content with a foundation in mis- or disinformation. Content which encourages the ingestion, inhalation or exposure to harmful substances is included in the OSA as a category of priority harmful content, which will protect children from mis- and disinformation content risking children’s health, such as content which suggests, for example “ingesting bleach to have an abortion”.

 

Service providers should also provide age-appropriate protections for ‘non-designated content’ which is content identified by platforms that doesn’t fall into the previous categories but nonetheless presents a material risk of significant harm to an appreciable number of children in the UK. This includes mis- and disinformation that is assessed to meet that risk threshold.

 

Providers of both search services and user-to-user services which are likely to be accessed by children have a duty to conduct risk assessments to determine the risk of children encountering harmful content and then take measures to reduce that risk. Search services will have a duty to use proportionate systems and processes to prevent the risk of children encountering search content that is primary priority content. Search services must also minimise the risk of children encountering other priority content where the children fall into age groups judged to be at risk of harm from that particular type of content. The measures employed by search services to minimise the risks of encountering content can involve the design of functionalities, algorithms and other features relating to the search engine, as well as content prioritisation or user support measures.

 

The OSA also updated Ofcom’s statutory duty to promote and address media literacy in relation to regulated services in several new areas. For example, Ofcom is now required to help the public understand the nature and impact of harmful content and online behaviour, and help them reduce their own and other people’s exposure to it – especially where this content or behaviour disproportionately affects certain groups, such as women and girls. Ofcom is also required to raise awareness of the nature and impact of mis- and disinformation and help the public assess the reliability, accuracy and authenticity of content found on regulated services. To meet these objectives, Ofcom needs to pursue, commission, or encourage other organisations to deliver media literacy activities and initiatives.

 

Finally, the OSA will require Ofcom to set up an advisory committee on disinformation and misinformation. The advisory committee will enable tech companies, civil society organisations and experts to come together in order to build cross-sector understanding and technical knowledge of mis- and disinformation. The role of the committee is to, via cross-sector working, provide non-binding advice to Ofcom on how regulated services should deal with misinformation and disinformation on their platforms, how Ofcom exercise their transparency powers, and how Ofcom carries out its statutory duty to promote media literacy in relation to mis- and disinformation. 

 

Implementation and Evaluation

 

The OSA is still being implemented but parts due to come into effect from next Spring will require platforms to take action on content that is illegal. We expect the child safety duties will come into effect by summer 2025.

 

Separately services over the designated threshold (also known as ‘Category 1 Services’) under the OSA will also have to consistently enforce their own terms of service, including where these prohibit harmful legal content.

 

DSIT is developing an evaluation framework, a plan for monitoring the implementation of the OSA and the core outcomes. The framework draws on new and existing data sources (for example, Ofcom’s Online Experiences Tracker) to develop and track specific metrics, including how often UK users encounter priority and primary priority content. Evaluation work will also assess the implementation and effectiveness of the measures in-scope platforms have taken, including those related to age assurance, transparency reporting and risk assessments.

Following the recommendations of the National Audit Office report (Preparedness for online safety regulation, 2023) DSIT is working closely with Ofcom to ensure that the data Ofcom collects as part of its evaluation activities will support DSIT’s own evaluation of the effectiveness of the regime and the achievement of its policy objectives, and that the evaluation efforts are aligned and comprehensive.

This monitoring and evaluation work will track the effectiveness of the online safety regime over time, as duties come into effect, and feed into a Post Implementation Review (PIR) of the OSA. The PIR will assess the effectiveness of the legislation against its primary objectives, including how the OSA has addressed harmful online content.

Question 4b: What more should be done to combat potentially harmful social media and AI content? 

 

The government is committed to making the internet safer. Our current focus is implementing the OSA, but the government continues to identify areas where we can build on the current framework. Nothing is off the table when it comes to keeping citizens safe. It is important that policy remains evidence-based, and we very much welcome the Committee’s inquiry to add to the evidence and information which the government has available, when considering future policy.

Online safety requires a broad toolkit that goes beyond regulation. The government set out five priorities in the draft Statement of Strategic Priorities published by the DSIT Secretary of State which it believes will support ambitious delivery of the OSA to provide users with the safest online experiences. These are: safety by design, transparency and accountability, agile regulation, inclusivity and resilience and technology and innovation. As the independent regulator, Ofcom will need to have regard to these as it exercises its online regulatory functions and will provide annual reporting to set out what action it has taken in consequence of these priorities.

 

It is equally vital to equip both children and adults with the knowledge and skills to navigate the online world. Media literacy can help tackle a wide variety of online safety issues for all internet users, including children. It means understanding that online actions have real-world consequences, critically evaluating online information, and contributing to a respectful online environment. It is a key tool to build people’s resilience to misinformation and disinformation (including AI-generated deepfakes).  

 

Since 2022, DSIT has provided nearly £3 million in funding for media literacy projects that empower users to make safe and informed choices online. In 2024, this included £0.5 million to scale up two programmes to provide media literacy support to teachers, children aged 11-16, and other professionals working with families and parents/carers. 

 

The government has also established an independent Curriculum and Assessment Review. The review will seek to deliver an excellent foundation in the core subjects of reading, writing and maths, and a broader curriculum that readies young people for life and work and reflects the diversities of our society. This includes the key digital skills needed for future life and the critical thinking skills needed to ensure children are resilient to misinformation and extremist content online. 

 

Furthermore, given the pace of AI technology’s development, it’s important to set clear expectations for the behaviour of frontier AI developers and ensure they are trusted by the public. Our intention is to make sure our statute book is fit for the age of AI and that accountability is assigned appropriately. We will also ensure that our existing expert regulators have the right expertise and resources to make proportionate and informed regulatory decisions about AI in their sectors.

 

We also believe that coordinating with international partners is vital to effectively tackle cross-border challenges posed by AI. We will continue to engage closely with our international partners, including the US and EU, as we further develop our approach to AI governance including our legislative proposals.

 

The government continues to investigate the potential for detecting AI-generated content to support transparency of synthetically generated content and welcomes technical innovation in this space. This includes both assessing evidence on the feasibility of technical solutions that support transparency of content, and any levers the government may have to ensure that these technologies are developed and deployed in a beneficial way. To facilitate innovation in deepfake detection, DSIT, the Home Office, the Alan Turning Institute, and the Accelerated Capability Environment (ACE) came together in March 2024 to deliver the Deepfake Detection Challenge. The Challenge saw government and its partners working with industry and academia to carry out fast paced, phased work to determine existing capabilities and identify gaps requiring innovative approaches and novel solutions to overcome the challenges that deepfakes pose.

 

Question 4c: What role do Ofcom, and the National Security Online Information Team (NSOIT) play in preventing the spread of harmful and false content online?

 

Ofcom is responsible for ensuring platforms fulfil their duties under the OSA including to help stem the spread of illegal content and protect children from content which is harmful to them. To empower them to fulfil this role, the government has provided significant levers to support their work. For example, by setting out clear priorities through the draft Statement of Strategic Priorities which we expect will support Ofcom to deliver ambitious online safety regulation.

 

Once the new safety duties are in effect, platforms will have to show they have systems and processes in place to meet the requirements set out by the OSA. Ofcom will monitor how effective those processes are at protecting internet users from harm and will have powers to take action against companies that do not follow the duties. These include the ability to fine companies up to £18 million or 10% of their qualifying worldwide revenue, whichever is greater.

 

With the agreement of the courts, Ofcom will be able to require payment providers, advertisers and internet service providers to stop working with a site, preventing it from generating money or being accessed from the UK. 

 

Additionally, Ofcom will be required to establish an advisory committee on disinformation and misinformation. The committee will have an important role in bringing together technology companies, civil society organisations, and sector experts to advise Ofcom. Their focus will be on building understanding and technical knowledge of the challenges relating to mis- and disinformation and ultimately helping to determine how to prevent the spread of these online.

 

To accomplish this, they will advise on how regulated services should deal with misinformation and disinformation on their platforms, how Ofcom exercise their transparency powers, and how Ofcom carries out its statutory duty to promote media literacy in relation to mis- and disinformation. Specifically, the updated media literacy duties require Ofcom to raise awareness of the nature and impact of mis- and disinformation, to help the public assess the reliability, accuracy and authenticity of content found on regulated services. The Chair of the committee, Lord Allan of Hallam, was announced on 27 November and the rest of the members will be appointed early in the new year.

 

Separately, NSOIT leads DSIT’s operational response to information risks impacting UK audiences which present a threat to public safety or national security in the UK. The team conducts targeted open-source monitoring and analysis to identify and assess potential narrative threats within its remit. It also engages regularly with platforms to understand the measures which platforms have in place to counter mis- and disinformation, as well as sharing information on disinformation narratives and trends. For example, prior to the general election, there was regular platform engagement with a view to minimizing the amount of mis- and disinformation aimed at the UK electorate.

 

Where content within this remit is identified which is likely to breach platforms’ terms of service, the team may use DSIT’s trusted flagger status to raise this with platforms. This is only done in limited circumstances: the post in question has to fall within NSOIT’s defined remit and has to be capable of causing harm to national security or public safety. Additionally, the post has to be in breach of the platform’s terms of service. Where a post is flagged to a social media platform, the decision of how to deal with that post lies solely with the platform. NSOIT has no influence or say in that decision.

Question 5: Which bodies should be held accountable for the spread of misinformation, disinformation and harmful content as a result of social media and search engines’ use of algorithms and AI?

 

As we noted under Question 3, the primary responsibility for harmful social media content rests with those individuals and groups who create and post it. We saw prosecutions and convictions for illegal online activity during the summer riots, some of the first convictions of this kind. However, social media platforms have a responsibility to keep people safe on their platforms and to ensure that those seeking to spread misleading content online are not being facilitated.

 

The OSA will require platforms to take swift action on illegal content and protect children from content which is harmful, including where it has been AI-generated. It will also require platforms over the designated threshold (Category 1) to enforce their own terms of service, including where these prohibit the spreading of misinformation, disinformation and harmful content. Where this occurs, platforms will be required to remove it and enforce these duties consistently. These measures have been designed to address the most egregious forms of mis- and disinformation while respecting freedom of expression.

 

Ofcom will have responsibility for ensuring platforms are protecting users. Once these new safety duties are in effect, platforms will have to show they have systems and processes in place to meet the requirements set out by the OSA. Ofcom will monitor how effective those processes are at protecting internet users from harm and will have powers to take action against companies that do not follow the duties.  

 

The government also engages with major social media platforms on a regular basis to support their efforts and ensure they remain accountable. This includes proactively referring hateful or manipulated content which breaks platforms’ terms of service.

 

18 December 2024