Written evidence submitted by Ofcom (SMH0078)

Ofcom is the UK’s independent communications regulator. We regulate landline and mobile phone services, broadband, TV, radio, postal services, and radio spectrum.

Under the Online Safety Act 2023 (“the Act”), Ofcom also became the regulator for online safety, with a responsibility to help make online services safer for the people who use them.

Examples of online services falling under our remit include:

• services that host user-generated content (such as social media and messaging services);
• search engines;
• services on which pornographic content is published or displayed; and
• UK-established video-sharing platforms (VSPs).

Services that fall under our remit have to have in place systems and processes to protect users from illegal content and activity online, as well as protecting children from harmful content. This includes content that is AI-generated, such as deepfakes. The Act does not make Ofcom responsible for removing individual examples of illegal or harmful online content. Under the Act, we have powers to take enforcement action where services fail to comply with the relevant duties.

We have identified eight targets for immediate action, based on where harm is greatest and where we know there are clear steps services can take:

• Services prioritise user safety, tackling risks to users, and naming senior accountable people. We want to see services completing effective risk assessments and having internal risk and governance structures that address online safety. This includes identifying senior people and making clear who is accountable and responsible for user safety within services, as appropriate. 
• Children are protected from harmful content and activity, including pornography. We expect services to have safety measures that protect children, including tough age checks to stop them seeing content such as suicide and self-harm material and pornography, and to filter or downrank harmful material in content recommended by algorithms.   
• Offenders can’t share child sexual abuse content and children don’t face unsafe contact. We want the sharing of CSAM to become much more difficult, and children to be protected from contact from adults they don’t already know. Almost a fifth of children experience sexual solicitation from adults they have chatted with online – change is needed to stop this.
• Illegal content – including hate and terror – is taken down quickly. We expect services to have trust and safety measures that can tackle illegal content – including hate speech, terrorist content and content that encourages or assists suicide – and ensure it is taken down quickly once services are aware of it.
• Women and girls face less gendered harm and abuse online. We want women and girls to feel safe online and not have to deal with as much harassment and misogyny, with effective reporting routes available to them where they experience harmful behaviour.
• Online fraud is reduced. We want to see services having systems and processes in place to make it harder for online fraud to take place - including detecting, deterring, and preventing fraudulent activity. The categorisation duties will also require some services to deploy systems and processes designed to prevent fraudulent advertising.
• Users – especially children – are empowered to have control over their online experience. We expect the biggest and most widely used (category 1) services to provide users with the tools to better understand their choices online, and the ability to adjust the content they see and who they interact with online. Our proposals for these services will draw on behavioural insights research and our long-standing Making Sense of Media programme.
• More transparency in how platforms keep users safe, and what Ofcom is doing to deliver positive change. We will use our new transparency powers to help shed light on platforms’ trust and safety measures and highlight good and poor practices. This will help to empower the public with more information. We’ll also look to be transparent about what we are doing as the online safety regulator to push positive change and stop wrongdoing.

We are currently developing and publishing Codes of Practice and guidance for companies falling under the scope of the new legislation. These Codes and Guidance explain how companies can comply with their new duties. We recently published the first edition of our Codes of Practice and guidance on tackling illegal harms, including Illegal Content Codes for user-to-user and search services.

The Illegal Harms Codes came into force on 17 March 2025. Providers will now need to adopt the relevant safety measures set out in the Codes, or use other effective measures to protect users from illegal content and activity. The Codes set out the steps that companies can take to comply, including:

• Service providers should do a thorough risk assessment and put in place appropriate governance arrangements to track and manage risk.
• High risk service providers should use an automated tool called hash matching to detect and remove child sexual abuse material.
• High risk service providers should take steps to make it harder for strangers to contact children online.
• User-to-user services should have a content moderation function to review and assess suspected illegal content. It should also allow for the swift take down of illegal content.
• High risk service providers should appropriately resource and train their content moderation teams.
• Service providers should operate easy to use reporting and complaints systems and should create a special channel/hotline for relevant regulators and law enforcement bodies to report fraud to them.

We expect to publish our Codes on the Protection of Children by the end of April 2025. Subject to parliamentary approval, these will be in force from July.

We welcome the opportunity to respond to this important inquiry. In this response, we explain how harmful content is covered by the Online Safety Act, before moving on to the extent to which misinformation and disinformation are within scope of the Act. We then outline the business models and algorithms that are currently commonly used by online services, and how this can impact the spread of harmful content on their services. We also explain how the advent of Generative AI (GenAI) – and the rise of deepfakes – has changed the online content landscape. Finally, we conclude by outlining Ofcom’s response to the riots last summer.

11 April 2025

How harmful content is addressed by the Online Safety Act

The Act requires online services that host user-generated content and search services to protect their users from illegal content, and in the case of children, from harmful content online. Services must assess and manage safety risks arising from content and conduct on their sites and apps. Ofcom’s role as the UK’s online safety regulator is to ensure that services have the appropriate systems and processes to protect people from harm. These are set out in our Codes of Practice, which cover governance; reporting and complaints; content moderation; safe design measures, such as tools to prevent children being approached by strangers; access restrictions, such as age assurance; and clear policies/terms of service.

The Act recognises that in-scope services are very different, in their size, resources and the risks they pose to people in the UK. Different safety measures will be appropriate for different types of service and our recommendations vary for services depending on their size and degree of risk. The Act is clear: first and foremost, the onus sits with service providers themselves, to properly assess the risks their users may encounter, and decide what specific steps they need to take, in proportion to the size of the risk, and the resources and capabilities available to them.

Illegal content

‘Illegal content’ is a new concept created by the Act, defined as ‘content that amounts to a relevant offence’.[1] The Act sets out the ‘relevant offences’ in scope of the criminal law in the UK for the purposes of identifying ‘illegal content’. Under the Act, the relevant offences comprise: a) A list of priority offences; and b) ‘Non-priority’ (or ‘other’) offences.

The priority offences are set out in Schedules 5 (Terrorism offences), 6 (CSEA offences) and 7 (Priority offences) of the Act. These are the most serious offences covered by the Act, as defined by Parliament, and all providers will need to act to prevent users encountering content amounting to one of these offences. In total there are over 130 priority offences in scope of the Act.[2]

For all priority illegal offences in scope of the Act, we set out in our Statement on Illegal Harms:

• Guidance on how to carry out a suitable and sufficient risk assessment.
• An analysis of the causes and impact of illegal harms, and the factors that give rise to a risk of these harms.
• Specific and cross-cutting measures in our Codes of Practice that services can take to tackle the risk of illegal harms online.[3]
• Guidance to help companies in making judgements about whether content is illegal content.

Content that is harmful to children

When complying with the children’s safety duties, service providers need to consider different types of content that is harmful to children. The Act defines ‘content that is harmful to children’ in three broad categories:

Codes of Practice

The first editions of the Codes of Practice, which come into force this year, will bring about a safer life online for UK users. They will also provide a firm foundation on which we can build in the years to come.

Our assessment, having engaged extensively with stakeholders since the Act passed, is that the new rules, and the measures proposed in the first sets of Codes will drive significant change. They will require much more effective governance and risk assessment by service providers. Other cross-cutting measures, such as clear and effective content moderation and reporting and complaints systems, will increase the detection and removal of illegal content, reducing associated harm. The Codes also include more novel approaches, such as measures to protect children against grooming in the Illegal Harms Codes which have not to our knowledge been proposed by any other regulator around the world. 

Our Codes also materially expand the number of high-risk services that use hash matching (that is, techniques that can detect known illegal or otherwise harmful images and videos) and bring about a step change in the detection and removal of Child Sexual Abuse Material (CSAM). The draft Children’s Codes of Practice go beyond current industry practice with measures including robust age assurance, safer algorithms, and tools and support to help children stay safe.

During these early years of the online safety regime, we will enforce the safety duties considering the measures outlined in the first edition of our Codes, while actively considering how to strengthen them further. We will continue to seek input from the public, civil society, service providers, and other expert bodies as this novel regulatory regime develops. In the coming months we will be publishing an additional consultation to build on the measures in our Illegal Harms Codes. This will include proposals in the following areas:

• Blocking the accounts of those found to have shared CSAM.
• Use of AI to tackle illegal harms, including CSAM.
• Use of hash-matching to prevent the sharing of non-consensual intimate imagery and terrorist content.
• Crisis response protocols for emergency events (such as last summer’s riots).

Transparency

The Online Safety Act requires categorised services,[4] which we expect to include some of the most widely used social media and search services, to produce transparency reports at least annually. Transparency reporting requirements are a key tool for driving effective and meaningful change. We will issue annual transparency notices to all categorised service providers requiring them to publish the information set out in the notice in a public transparency report. Ofcom has powers to tailor transparency notices to individual services. In addition, we will publish our own annual reports based on provider transparency reports: analysing industry trends, identifying good practice and summarising additional information, such as new research and other sources of data, to help contextualise this information for users. We will publish our online safety transparency guidance in the first half of 2025.

Ofcom has powers to require categorised services to make public a broad range of information as set out in Schedule 8 to the Act. This includes information on measures taken or in use by a provider to comply with any duty set out in section 71 or 72 (terms of service), and any other measures taken or in use by a provider which relate to online safety matters.

Enforcing the rules

Our Supervision team is engaging with selected services that pose particular risk, either because of their size or because of the nature of the service. This targeted oversight includes understanding services’ measures in detail, assessing how well they protect users, and pushing for timely improvements where necessary. We have developed our supervisory approach through our existing work to regulate UK-established Video-sharing platforms since 2020.

Our objective is to drive compliance to make UK citizens safer online as quickly as possible and often this is best achieved through working with services to encourage voluntary compliance. But we will, if necessary, launch enforcement action where we determine that a service provider is not complying with its duties, for example where we consider it is not taking appropriate steps to protect users from harm.

Under the Act, where we identify compliance failures, we can impose fines of up to £18m or 10% of the service provider’s qualifying worldwide revenue (whichever is greater). In the most serious cases of non-compliance, we can seek a court order imposing business disruption measures, which may require third parties (such as providers of payment or advertising services, or ISPs) to withdraw, or limit access to, the services in the UK. 

Furthermore, service providers may commit a criminal offence if they fail to comply with an information notice or if they fail without reasonable excuse to take compliance action which is specified in an Ofcom decision finding that the service is in breach of certain duties relating to child sexual exploitation and abuse and child safety. In such cases, directors and other senior managers of the provider may also be criminally liable for the failures.

On 1 March we launched an enforcement programme to monitor if services are meeting their illegal content risk assessment duties and record keeping duties under the Act. One of our first priorities is to scrutinise the compliance of sites and apps that may present particular risks of harm from illegal content due to their size or nature – for example because they have a large number of users in the UK, or because their users may risk encountering some of the most harmful forms of online content and conduct, including child sexual exploitation and abuse, terrorism, hate crimes, content encouraging or assisting suicide, and fraud.

On 17 March we launched an enforcement programme to assess the measures being taken by providers of file-sharing and file-storage services that present particular risks of harm to UK users from image-based CSAM to ensure users do not encounter, and offenders are not able to disseminate, such content on their services. Given the acute harm caused by the spread of online CSAM, assessing providers’ compliance with their safety duties in this area has been identified as one of our early priorities for enforcement.

Under our VSP work, Ofcom has issued several requests for information from in-scope services to promote transparency about the measures services take to protect their users and bring about improvements. In July 2024, Ofcom found that TikTok contravened its duties to provide accurate information in response to one such request and we imposed a penalty of £1,875,000 as a result. 

How misinformation and disinformation are addressed under the Online Safety Act

The Online Safety Act does not explicitly identify disinformation or misinformation as harms that need to be addressed by online services. However, it does name several offences and provisions that are relevant to them. These are outlined below. Services must also have particular regard to the importance of protecting users’ rights to freedom of expression and privacy when implementing safety measures.[5]

Foreign Interference: As a priority offence under the Act, all in-scope companies are required to assess the risk of this occurring on their services and take measures to prevent users from encountering content amounting to a ‘foreign interference offence’ on their service. This is a new criminal offence introduced through the National Security Act 2023. It aims to tackle malign state-linked interference, such as state-sponsored disinformation, targeting UK political processes.

We set out the steps we expect companies to take to tackle foreign interference online in our December 2024 Statement on protecting people from illegal harms online.

False communications offence: This new offence, introduced through the Online Safety Act, makes it an offence for a person to send a message which conveys information that they know to be false if the person intended the message, or the information in it, to cause non-trivial psychological or physical harm to a likely audience. The user must also have no reasonable excuse for sending the message. As it focuses on knowingly false information, misinformation is not captured by this offence. We set out our analysis, codes measures and guidance relating to this offence as a part of our December 2024 Statement on protecting users from illegal harms online.

Terms of Service duty: Category 1 services have a duty to use proportionate systems and processes to ensure that taking down or restricting access to content, and suspending or banning users, is only carried out in accordance with their terms of service. They must consistently enforce any provisions in their terms of service related to these actions. This includes where they have provisions which are relevant to misinformation and disinformation. We will consult on our proposals no later than early 2026.

Media literacy duties in relation to mis and disinformation: Since 2003 we have had responsibilities to promote and research media literacy, to ensure that audiences are equipped with the skills to navigate and understand content including news and critically engage with, for example, the accuracy and partiality of news sources and content.[6]

Through the Online Safety Act, Ofcom’s media literacy duties have been further clarified. Ofcom now has a specific requirement (section 165) to heighten the public’s awareness and understanding of ways in which they can protect themselves and others when using regulated services, in particular by helping them to establish the reliability, accuracy and authenticity of content and understand the nature and impact of disinformation and misinformation, and reduce their and others’ exposure to it. The Act states that we are to perform these duties by (among other ways) pursuing or commissioning activities and initiatives, encouraging others to do so, and through our research.

In October 2024, we published a multi-year media literacy strategy, setting out Ofcom’s media literacy objectives and priorities for the next three years. Within each of the three sections of the strategy – Research, Evidence and Evaluation, Engaging Platforms and People and Partnerships – there is an explicit focus on understanding what works in supporting people to identify and build resilience to misinformation and disinformation. In addition we are exploring the media literacy implications of AI, both opportunities and challenges, including the advent of deepfakes.

Advisory Committee: As required by the Online Safety Act, we are in the process of establishing an advisory committee to advise us about specific areas of our work relevant to disinformation and misinformation, including but not limited to:

• How providers of regulated services should deal with disinformation and misinformation on such services.
• Ofcom’s exercise of its transparency powers (as outlined above), including powers to require information from services about matters relating to disinformation and misinformation.
• Ofcom’s exercise of its duties to promote media literacy in relation to countering disinformation and misinformation on regulated services (as outlined above).
• Ofcom’s development of policy in relation to the programme of work on media we trust and value, one of the four priority areas set out in our Plan of Work.

We published Terms of Reference for the Advisory Committee in November 2024, and have appointed Lord Allan of Hallam, a non-executive member of Ofcom Board, as the Chair of the Committee. This reflects the importance of this Committee and its work, and to ensure senior-level accountability. We are currently in the process of appointing members for this committee, and once this is complete the Committee’s work will begin. As required by the Act, the Committee will publish a report within 18 months of its establishment and periodic ones after that.

How do online services’ business models and algorithms impact harmful content?

As explained above, user-to-user services (including ‘social media services’ such as Facebook, Instagram, TikTok, and YouTube), and search services (including ‘search engines’ such as Google and Bing) are in scope of the Online Safety Act.

The Online Safety Act specifically requires both user-to-user and search services to assess how the design of their service – including their business model – impacts the risk of harm to users in relation to illegal content and content harmful to children. The Act also requires Ofcom to produce full industry-wide risk assessments (the ‘register of risks’) of the causes and impacts of specified online harms. In our illegal harms register of risks, we set out that the potential for users to experience these specified harms on a particular online service can be impacted by the service’s business model.

However, as we explain below, a service’s business model does not necessarily determine its risk of harm. Risks associated with advertising-based business models can be impacted by how a service’s recommender system functions. This means that the resultant risk of harm can be affected by a service’s recommender system design choices as well as by the way that users interact on it. There are various factors impacting why services might make particular design choices, such as advertising-driven financial incentives and reputational risk.

The Online Safety Act requires regulated services to risk assess how their use of algorithms impacts the likelihood of users encountering illegal content or content that is harmful to children. These services are required to manage and mitigate the risks that have been identified and, where proportionate, to implement safety measures regarding the design of their algorithms.

We have summarised below the different business models and algorithms that are commonly used for user-to-user services and search services. This includes algorithms known as recommender systems which present personalised content feeds to users. In each case, we describe the design choices that each type of service might make that can affect the content delivered to users, as well as how such outcomes might also be driven by business incentives and user behaviour.

Business Models

Ofcom’s illegal harms register of risks sets out the most prevalent business models for both user-to-user services and search services, and how these business models may impact the risk of specified harms to users. We have outlined the relevant aspects of this register of risks below. As many of these risks are linked to the design of services’ recommender systems, we then outline below in this evidence the impact of recommender systems as well as how we’re addressing the associated risks through the implementation of the online safety regime.

User-to-user services

There are several revenue models for user-to-user services including subscription fees or charging transaction fees for marketplace purchases. However, these services often generate most of their income through an advertising revenue model – which means that they generate income from advertisers paying to display advertisements for a product or service.

User-to-user services with advertising-based and subscription-based revenue models can be incentivised to design systems and features that personalise and individualise the user experience in a way that maximises time spent on the service and user engagement with content. By lengthening the time spent or level of engagement (e.g. viewing, resharing, commenting, ‘liking’ etc.) with content, users are more likely to encounter more revenue-generating adverts.

A service that prioritises engagement may serve users feeds of content that are most likely to keep them engaged for a prolonged period. Engagement is often measured by users’ interaction with content. Where harmful content exists on a service, recommender systems could amplify the visibility of this content if users are positively engaging with it. Some evidence shows that content that creates outrage or strong reactions can encourage user engagement. [7] The prioritisation of user engagement and a focus on keeping users on a service can therefore increase the risk of users encountering harmful content.

However, the reputational risk of exposing users to harmful content could negatively affect a service’s revenue in the long run, creating some countervailing financial incentives. Some users may disengage with services where they encounter harmful content, while advertisers and the wider industry (such as payment providers and investors) may put commercial pressure on services to clamp down on harmful content.

The engagement-driven risks associated with advertising-based business models also largely depend on the design of services’ recommender systems. While a revenue-driven focus on user engagement can increase the risk that users encounter harmful content, user-to-user services can equally make design decisions – including within their recommender systems – to prioritise high-quality, trustworthy, and authoritative content. This is set out in more detail below.

Search services

Search services are designed to help users find information across the internet. They typically require users to provide a ‘prompt’ or ‘query’ against which results from across the internet are retrieved and organised. The primary function of a search service is to deliver search results that match the user’s query as close as possible.

Search services typically operate an advertising-based business model, which may lead them to prioritise sponsored content at the top of their search results. This might make it more difficult for users to find relevant information beneath these sponsored listings, while users may also not identify sponsored search results as advertising. Ofcom’s research has found that less than half (47%) of search engine users are confident and able to recognise advertising in search engine results[8].

Advertising-based revenue models could risk amplifying harmful content if bad actors seek to target paid-for advertising to appear in sponsored search results. However, as further described below, the design decisions that search services can make are important factors in determining risks to users.

Algorithms and recommender systems

Algorithms are sets of computing instructions present across all online services. They are a basic component of any computer programme or software. In this context, ‘algorithms’ to refer to the type of algorithmic and autonomous systems used by user-to-user services and search services for the purposes of content distribution. Such services often use sophisticated algorithms to learn about user behaviour and perform complex computations about what type of content to recommend to users over time. 

The spread of harmful content can be facilitated by a number of functionalities on online services, including direct messaging, the formation of private groups and communities, and resharing. While algorithms might be one factor in the spread of misinformation and disinformation, there can be other means through which users come in contact with harmful content. We describe how user-to-user services and search services use different types of algorithms to achieve different objectives in relation to the content they serve to users below.

User-to-user services

User-to-user services use algorithms (known as ‘recommender systems’) to personalise users’ content feeds at scale, helping these users to encounter content they are likely to find relevant and engaging with minimal search costs, while also helping creators to reach their audience. Recommender systems use a variety of techniques (such as ‘machine learning’) to observe user behaviour in relation to content, including likes and reactions, comments, watch time, reshares, and saves. Positive user engagement on content can result in further recommendations of similar content.

The prominence of content and the order in which it is presented to the user on user-to-user services – known as ‘content ranking’ – is a complex process. Content ranking can depend on the content discovery ‘surface’ or ‘feed’; examples include a ‘news’ feed, ‘trending’ feed, ‘for you’ feed, groups and pages, or search functionalities. Content ranking can be categorised into two notional types: chronological ranking and algorithmic ranking driven by recommender systems. These are discussed in more detail below.

Ranking in search services

While related to recommender systems, search services employ a different algorithmic system used for the purposes of information retrieval in relation to a search query. The primary objective for the algorithms used by a search service is to ensure that the search results match the search query. Each search result is given a relevancy score based on how well it matches the user’s search query, presented in descending order of relevance, meaning that the most relevant queries will appear closer to the top of the page.

Depending on the nature of the query, the relevance of each link might be adjusted to ensure high-quality, credible, and trustworthy results. There are also contextual factors that might affect the ranking of search results such as time of day, time of week, language, and location.  For example, if the query is about current affairs or news, a service might put in place filters around credibility and authoritativeness to ensure high prominence of trusted news providers. Similarly, if a query is about health, a service might place filters around accuracy to ensure that the user is presented with information from credible sources of health or medical information.

This means that the trustworthiness and credibility of links can be factors that determine how search results are presented, but this depends on the design decisions made by search developers to build and integrate these factors into their search engine architecture.

Addressing algorithms under the Online Safety Act

Under the Online Safety Act, all regulated services are required to risk assess how the design of their service and their use of algorithms impacts the likelihood of users encountering illegal content or content that is harmful to children. Services need to manage and mitigate their identified risks and, where proportionate, implement safety measures regarding the design of their algorithms. Ofcom has set out such measures within both our Codes of Practice on illegal harms and on the protection of children.

Illegal harms: In December 2024, we published our Illegal Harms Codes of Practice, setting out how regulated services should comply with their duties to address illegal harms on their platforms. These Codes of Practice came into force on 17th March and contain recommended safety measures based on governance and risk management, service design, and increased control for users over their online experiences. We have included a specific recommendation that:

• Certain user-to-user services should collect safety metrics when testing their recommender systems. This will help these services to better understand whether adjustments to the design of their recommender systems might contribute to the risk of their users encountering illegal content, therefore promoting a safety by design approach to the operation of algorithmic systems.

Protection of children: In May 2024, we also launched a consultation on how online services should comply with their duties relating to content that is harmful to children. These duties apply to content specifically defined under the Online Safety Act as harmful to children; this includes suicide, self-harm and eating disorder content, violent content, and pornographic content, but does not include misinformation or disinformation.

In our consultation, we identified recommender systems as a key pathway for children to encounter these specific types of harmful content. They also play a part in narrowing down the type of content presented to users, potentially leading to increasingly harmful content recommendations for children, as well as exposing children to cumulative harm over time through repeated exposure to harmful content.

Within our May consultation, we proposed draft Codes of Practice with recommended safety measures specifically targeted at ensuring recommender systems do not operate harm to children. This includes recommended measures that user-to-user services which operate a recommender system, and which are at higher risk of harmful content (as defined by the Act) should:

• Configure their recommender systems to filter out the most harmful content from children’s feeds and reduce the visibility of other harmful content.
• Large services should also enable children to provide negative feedback on content that is recommended to them, giving them more control over the content in their feeds.

How does Generative AI impact the creation and dissemination of harmful content

The increasing availability in recent years of tools powered by Generative AI (GenAI) is changing the online content landscape, particularly in terms of the creation and sharing of ‘deepfake’ content. Last year, Ofcom published a research paper called ‘deepfake defences’, outlining the impact of online deepfakes and analysing what can be done to tackle their proliferation.

Deepfakes created using GenAI tools represent a new means for people to create and disseminate harmful content. However, simpler so-called ‘cheapfake’ techniques – such as reusing old content for new contexts, or slowing down or accelerating video footage – can likewise mislead people. For example, a video of Rishi Sunak in 2023 was cut to make it look as though he did not know how to use a hammer.

Likewise, not all deepfake content – such as that created by GenAI tools – is harmful. GenAI and related tools can be used, for example, to augment TV and film outputs in their post-production, enhance photos and videos with everyday filters, and create entertaining or satirical material.

Our ‘deepfake defences’ paper outlined that deepfakes created with GenAI can also take forms which harm individuals and society in three main ways:

• Deepfakes that demean: these are created to humiliate or abuse a victim-survivor by falsely depicting them in a certain manner or performing a certain act, for example sexual activity. We have seen high-profile figures such as Taylor Swift, Alexandria Ocasio-Cortez, and Giorgia Meloni being targeted in this way.
• Deepfakes that defraud: these are used to deceive a target by misrepresenting someone else’s identity. They are primarily used to assist fraudulent behaviour such as scam communications and false advertising.
• Deepfakes that disinform: these are produced to sow distrust or spread disinformation across a large group. They are generally intended to be spread widely across the internet, their objectives being to influence or shape societal opinion on key political or societal issues. Some disinforming deepfakes, however, are more narrowly targeted at select groups of individuals or a community bound by certain beliefs.

During the 2024 UK general election, there were several deepfakes that purportedly showed political candidates making inflammatory and divisive comments. For example, the BBC reported that a network of X accounts were creating and sharing a fake audio clip of Wes Streeting, in which he appeared to insult fellow candidate Diane Abbott.

For these types of harmful use cases (deepfakes that demean, defraud, or disinform), the advent of GenAI has changed the online deepfake landscape in the following ways:

• Ease-and-cost of use: GenAI tools can make it simpler and cheaper to create convincing, life-like synthetic deepfake content. GenAI tools can be harnessed by anybody with basic technical literacy, with users only needing to enter simple prompts into model interfaces. This lowers the barrier to entry to creating harmful content, allowing new types of actors to participate.
• Capable of creating more convincing content: Although deepfakes created using more traditional technical tools can often be high quality, it is much easier to create convincing content using GenAI models. These GenAI tools continue to improve; the latest video models can be used to create synthetic characters that not only sound lifelike, but which express a range of nuanced facial expressions and body changes.
• Capable of a greater range of content creation and manipulation: GenAI is easily capable of creating entirely new content as well as editing that which already exists. In this way, GenAI allows for more types of content to be created, including long passages of audio, that previous tools struggle with.
• Adaptable to specific use cases: GenAI models are adaptable, especially when they are released on an open-source basis. Bad actors can ‘fine tune’ open-source GenAI models with additional data, meaning they are more likely to create a specific type of harmful content

How are GenAI and deepfakes addressed by the Online Safety Act?

Under the Online Safety Act, regulated online services are required to treat AI-generated content in the same way as human-generated content when it is shared on their platforms. These services need to address AI-generated content such as deepfakes in line with the Online Safety Act duties outlined above, including taking measures to take down illegal content of which they are aware.

The Online Safety Act can also apply directly to GenAI tools depending on whether their functionalities bring them in scope of the Act. In November 2024, we explained in an open letter to industry the types of GenAI services that might be regulated under the Act. All in-scope services will need to comply with the relevant duties set out in the Act.

Ofcom’s Codes of Practice will help regulated services to meet these duties and protect their users from the risk of encountering illegal content and content that is harmful to children, including where risks are posed by AI-generated content such as deepfakes.

Particularly relevant measures in Ofcom’s illegal harms Codes of Practice and draft protection of children Codes of Practice that will contribute to addressing deepfake and GenAI harms include:

• Recommender systems: As already outlined, services should collect safety metrics from recommender system tests to understand whether design changes could increase exposure to illegal content, such as some types of deepfake content. For children specifically, we have proposed measures that services should design their recommender systems so that they filter out and limit harmful content – including in the form of deepfakes – from children’s feeds.
• User verification and labelling schemes: Services should have clear internal policies for operating notable user or paid-for verification schemes (where those exist) and improving transparency for users about what verified status means in practice. This may help users to assess whether a user posting content is authentic or impersonating a high-profile individual or organisation.
• Content moderation: Services should set performance targets for content moderation functions, adequately resourcing content moderation teams, and training staff on how to identify and take down illegal content and take action against content harmful to children, which would likewise include deepfake content.

In the coming months, we will consult on additional proposals to strengthen our illegal harms Codes of Practice, which should contribute to tackling deepfake and GenAI harms. We will propose how services can use automated tools to proactively detect illegal content and the content most harmful to children, including where this content is AI-generated.

How can we further mitigate online deepfake harms?

Addressing deepfakes is likely to require further action from all actors across the technology supply chain – from the developers that create GenAI models and related tools, to the platforms that host this technology, through to the user-facing services that act as spaces for deepfake content to be shared and amplified. The Government and Parliament are currently legislating to criminalise the creation of non-consensual intimate deepfakes and to tackle AI models purposely trained and finetuned to generate CSAM.

Our research paper on ‘deepfake defences’ identified four principal routes for actors to mitigate deepfake harms:

• Prevention involves efforts to limit the creation of harmful deepfakes. This can include adopting prompt filters to prevent models being instructed to create certain types of content (e.g. nude content); removing harmful content from model training datasets; blocking outputs before they are presented to users; and red teaming models
• Embedding involves attaching contextual information to synthetic media through tools including watermarks, provenance metadata and labels
• Detection encompasses efforts to distinguish real from fake content, even where no contextual data has been attached to that content.
• Enforcement involves setting clear rules about the types of synthetic content that can be created and shared on online services. It also involves acting against users that breach those rules, for example through suspending or removing user accounts.

The above techniques all show promise in mitigating the creation or spread of harmful deepfakes. However, they each have limitations and weak spots. Some embedding measures, for example, may be susceptible to removal by bad actors, and their successful employment requires extensive coordination between different stakeholders.

We are continuing our research into mitigations for harms caused by varying forms of deepfakes. We are currently exploring and assessing the merits and limitations of potential safety measures based on embedding techniques and deepfake detection. This research is an important part of developing the available evidence base which will inform future iterations of our Codes of Practice under the online safety regime.

Ofcom’s response to the riots in Summer 2024

As set out above, Ofcom’s role is to regulate the systems and processes of services that are in scope of the Online Safety Act. The Act places new legal duties on platforms. They are required to take steps to prevent the spread of illegal material and mitigate the risks to the safety of people in the UK arising from illegal online activity, and content and conduct which harms children. This includes activity that provokes violence or stirs up hatred, and false communications intended to cause harm.

Ofcom acted swiftly after the violent disorder began last Summer, issuing a public statement on 5 August, followed by an open letter to tech firms on 7 August, emphasising the importance of taking steps to protect users and reminding them that they did not need to wait for the new duties to come into force before doing so.

In parallel, our Supervision teams spoke directly and in detail with many of the largest social media and messaging platforms from early August, to find out their response to the riots. We also met with a range of stakeholders from civil society, Government and law enforcement. To ensure we had a clear understanding of events, information was provided to us confidentially. Given the duties in the Act were not then in force, we did not seek to determine whether they would have been breached by what we saw.

Nonetheless, these events have clearly highlighted questions services will need to address as the duties come into force. While some told us they took action to limit the spread of illegal content, we have seen evidence that it nonetheless proliferated, and appears to have contributed to the significant violent disorder which followed the tragic murders in Southport.

While not all platforms experienced significant levels of illegal and harmful material, others have confirmed to us they were dealing with high volumes, reaching the tens of thousands of posts in some cases. Of the numerous convictions which have followed, some of those convicted have been found guilty of online posts threatening death or serious harm, stirring up racial hatred, or sending false information with intent to cause harm.

Services told us they took a range of actions in response to these events, including:

• The implementation of incident response protocols by some services. This included setting up internal cross-functional groups focused on the crisis, monitoring spikes in relevant types of harmful content, and the blocking of URLs leading to illegal and harmful content on other sites.
• Actions to prioritise the moderation and removal of illegal and harmful material linked to the protests, and to suspend or close down accounts and channels involved in spreading such content.
• Proactive engagement with civil society and/or law enforcement partners to seek guidance. In one instance, a service proactively reached out to a civil society organisation focused on anti-Muslim hatred requesting training to improve their moderation systems and took down potentially illegal content based on a referral from law enforcement agencies.

The events of last summer highlight questions that several platforms would have had to answer, had the duties been in force at the time. Specifically, as set out in our Codes and Guidance, user-to-user services should:

• Include in their terms of service provisions specifying how individuals are to be protected from priority illegal content. In any future incident, we will be concerned if services do not have clear and consistent terms of service prohibiting illegal hateful content. 
• Have systems or processes designed to swiftly take down illegal content once aware of it, by prioritising the moderation of viral content, and having content moderation teams that are adequately resourced and trained to meet increases in demand caused by crises. On some platforms, false information regarding the identity of the attacker continued to spread in the three days it took for his real identity to be made public, even when there was evidence of intent to stir up racial and religious hatred. Now the duties are in force, tech firms will need to act promptly when they become aware they are hosting such material, and we will want to understand their targets for removing it, and whether they are meeting them. 
• Provide effective and accessible mechanisms for users to complain about illegal content, including material that stirred up hatred, provoked violence or constituted violent threats. Given the volume of potentially illegal content which appeared during this period, we will want reassurance that platforms’ reporting systems are easy to find and use, and that they can cope with the number of user reports they receive, especially in crises, and moderate flagged content swiftly. This includes messaging services, especially those hosting groups comprised of large numbers of individuals. 
• Assess whether changes made to recommender systems are likely to increase users’ exposure to illegal material. Given evidence that illegal and harmful content linked to these events was recommended and amplified by some services, we will expect services to be able to show how they test their recommender systems following any changes to ensure they don’t expose users to additional risk. 

In addition to these protections from illegal material for all users, our Protection of Children Codes, once finalised, will recommend that sites and apps likely to be accessed by children take additional steps to ensure that children are protected from material which, while not illegal, is harmful, including violent content and content that is abusive or incites hatred.

We have also used this work to examine whether there are further measures we should add to future iterations of our Codes of Practice. Specifically, as part of the consultation on additional measures for the Codes, we are bringing forward a new measure that would place additional requirements on services regarding their crisis response protocols. We are also considering the evidence base for new measures on livestreaming functionalities and recommender systems.