Clean Up The Internet is a UK-based non-profit, also with a presence in Brussels, concerned about the degradation in online discourse and its implications for society and democracy. We campaign for government and tech industry action, to increase civility and respect online, and to reduce harms such as bullying, trolling, intimidation, fraud, and misinformation.
For the past few years our major focus has been on fake and anonymous social media accounts - researching how they're misused, and developing proposals for proportionate measures to address these harms whilst protecting freedom of expression. We had some success influencing the Online Safety Act (OSA), with the inclusion of the user identity verification duty in Section 64.
This submission draws on a report we produced in the aftermath of the summer riots, and earlier research into the role of fake and anonymous accounts in spreading conspiracy theories, fuelling hate and abuse, and enabling fraud. Its focus is the role of social media accounts in facilitating concealed or deceptive identities (“anonymous or fake accounts”), as this is our area of greatest expertise. However it also considers other features and functionalities, including recommender algorithms and AI, particularly the way in which these features and functionalities interact with the use of fake accounts.
Different functionalities on social media platforms are often used in combination, including when they are used to do harm. Thus bad actors using fake accounts to seed and spread false and inflammatory content may use AI to create a greater volume of higher quality content. And thus inauthentic engagement from fake accounts can influence recommender algorithms, leading to inflammatory content being further amplified.
The committee is right to be focusing on platform features and functionalities, and the business model which drives decisions about how these features and functionalities are deployed. Tackling the design features which enable and encourage harmful online content to spread, and using regulation to create new incentives for platforms to design safer products, is the best way to prevent harm arising, and to prevent UK users falling victim. Whilst AI and recommender algorithms are two important examples of features and functionalities which need to be tackled in this way, they are not the only two. The ability to create fake and anonymous accounts is another such feature. Private groups, channels, or pages, and the ability to livestream videos, would be other examples.
Platforms’ business models clearly often incentivise unsafe design, and do not provide strong enough incentives for them to make their products safe, so regulation is clearly required. Over a year after its adoption, the OSA is only beginning to be implemented, and so it is too early to offer a definitive verdict on its effectiveness. However, we identify concerns about the pace and ambition of Ofcom’s approach to implementation, as well as some limitations in the Act.
All of the major social media platforms rely overwhelmingly on advertising for revenue. A reliance on online advertising does not ensure good outcomes for internet users. This has long been accepted, including within the industry. Indeed, the founders of Google, Brin and Page, wrote a paper in 1998 which included in an appendix titled “Advertising and Mixed Motives” a concise summary of the inevitable conflicts of interest that would arise were their nascent online search service to be driven by advertising metrics.
For social media platforms, an advertising business model means an extremely powerful incentive to maximise “engagement” i.e. the number of users of the platform, and the amount of time each of those users spend on the platform. Any user account is a set of “eyeballs” which can be sold to an advertiser. The greater the amount of time those “eyeballs” are online, the greater the number of adverts that can be sold.
With maximising engagement as the guiding design principle, other considerations such as the safety of users, or accuracy of the information which is promoted, are inevitably compromised. Platform features and functionalities are adopted and optimised by social media platforms on the basis of engagement. Features which do not drive engagement are not prioritised or are even downgraded and abandoned, and features which could introduce “friction” into the recruitment of users, or reduce the number of claim-able users, or the amount of time they spend on the platform, are resisted.
Platforms’ prioritisation of engagement is reflected in platforms’ approach to user account and profile creation. If the ultimate goal is to maximise the number of (ostensible) accounts, and the amount of activity, then it makes little sense to introduce any barriers to accounts being created. Account creation processes are optimised to minimise the time they take and the effort they require. There’s little incentive to prevent deceptive profiles or restrict bad actors from creating multiple fake accounts – on the contrary, those accounts can be included in inflated claims of user numbers, which are quoted to shareholders and advertising customers..
This laissez-faire approach to account creation is clearly being regularly exploited by bad actors seeking to seed and spread divisive content, whether to further a political aim or simply to stoke division. As Ofcom described in its recently-published Illegal Content Register of Risks, “The ability to create fake user profiles can be exploited by perpetrators of foreign interference operations – both to disseminate content and to impersonate authoritative and high-profile sources. The use of coordinated networks on social media accounts can also be used to amplify content and spread narratives across services.”
To offer some recent examples:
In all these examples bad actors were able to create multiple deceptive profiles, to seed and disseminate deceptive content. Where platforms took action, it was only after the accounts had been operating for some time, and in many cases only after independent third parties had detected and publicised them. The advertising business model clearly did not incentivise platforms to take sufficient action to tackle such inauthentic activity. Moreover, inauthentic activity by disinformation operations actively served the business model - these inauthentic accounts will have been included in claims of reach made to advertisers, and adverts will have been served alongside inauthentic, harmful, content.
The advertising business model, and the ruthless prioritisation of engagement which it incentivises, has led to platforms developing systems, processes, features and functionalities which, taken together, pose huge risks to individuals and to society. Several of these design choices were particularly significant in fuelling the summer 2024 disorder in the UK:
In most cases these systems, processes, features and functionalities were not designed to further an extremist agenda, but rather to maximise advertising revenue by boosting engagement. But choices made to promote engagement were then exploited by a range of actors, including:
In section 4 of this submission we explore in a little more detail how these features and functionalities, exploited by this range of bad actors, combined with what happened in Southport on 29 July 2024 to make a major contribution to the disorder which followed.
On almost all the major platforms, recommender algorithms are the default means by which content is served to users. There is a residual notion of a chronological “timeline” of content produced by accounts which a user has proactively chosen to follow, but this is not the default experience on any of the major platforms.
There is a lack of transparency around how algorithms select content - but the overall aim is to further the business model by keeping users engaged as long as possible. Other considerations such as veracity or balance of content, or health and wellbeing of the user, are subordinate to engagement. Two factors in how algorithms select content to promote seems to have been significant to the role they play in the spread of misinformation, disinformation and harmful content:
i) Content which makes users feel angry, outraged, upset has been found to be a potent way to keep them engaged. A 2018 Facebook internal presentation, leaked to the press in 2020, described how “our algorithms exploit the human brain’s attraction to divisiveness” and warned they could offer “more and more divisive content in an effort to gain user attention & increase time on the platform”.
ii) Algorithms boost content on the basis of inauthentic engagement from fake accounts. Algorithms measure and respond to what they see as engagement with content (in the form of clicks, likes, shares, comments, etc.) as a signal to recommend it to more people. Authentic users sometimes seek to game this with exhortations such as “let’s get this trending”. Networks of fake accounts under the operation of bad actors, coordinated to push particular messages or pieces of content, are well adapted to exploit this and they are able to “game” platform algorithms to secure further amplification for scams, disinformation, and inflammatory content. Such “coordinated inauthentic manipulation or automated exploitation” of TikTok’s recommender algorithms, using a network of fake accounts, is now the subject of an EU investigation under the DSA, following the suspension of the Romanian Presidential Election due to concerns of Russian interference.
Generative AI is marketed in many sectors as a productivity booster. For bad actors seeking to create and spread harmful content on social media, the ability to machine-generate copy, images and videos has the potential to boost productivity by increasing both the quality and quantity of their output, and making it harder to detect.
AI can support bad actors in creating convincing-looking networks of fake accounts, by aiding the creation of multiple, plausible, individual profiles, including by generating realistic fake photos which cannot be detected as inauthentic through reverse-image searches, and plausible-sounding, unique biographical copy. Generative AI can support the deployment of those fake accounts to maximum effect, for example by creating multiple differently-worded variations of the same messages, rather than relying on “copy and paste”. It can also enable non-English speakers to produce more authentic-sounding, idiomatic, and grammatically correct copy, including replies to other users’ comments. As well as increasing the potency of a disinformation campaign, the elimination of “copy and paste” and improved language makes it far harder for end users, researchers or platforms’ own moderation systems to detect.
One disinformation operation in September 2023, which paired ChatGPT with a network of fake accounts on X, was detected after an error by its operators led to some of the fake accounts posting the following ChatGPT error message: “I cannot fulfil this request as it goes against OpenAI’s use case policy by promoting hate speech or targeted harassment. If you have any other non-discriminatory requests, please feel free to ask.”
The Institute for Strategic Dialogue took these error messages as the starting point for an investigation which uncovered a network of “at least 64 accounts which appear to be using content generated by OpenAI’s ChatGPT app to engage in a targeted harassment campaign…the goal of the campaign appears to be to undermine support for Navalny [the well known opponent of Putin] and the ACF [his opposition organisation] among pro-Ukraine American and Western audiences.” The report found that the AI-generated content “could easily pass as authentic”, and that “it may be more or less impossible for users to be certain that an account is using AI generated content unless – as in this case – the operators are sloppy enough to post a refusal from the AI.”
In the absence of human errors of the kind which enabled this specific investigation by ISD, as AI becomes more sophisticated AI-generated content is going to become more and more difficult to detect. We would suggest that this will place an even higher premium on measures to restrict the ability of networks of non-verified, inauthentic accounts to disseminate such content, and to make it easier for users to spot and avoid such accounts.
With the exception of a few smaller platforms (and perhaps X, since its acquisition by Elon Musk), social media platforms have not been designed to promote extremist ideas or encourage violent disorder. However, subsequent and serial design choices made in the pursuit of engagement have led platforms to offer features and functionalities which are easily exploited by bad actors including extremists and hostile foreign states. And the same business model, coupled with weak regulation, has provided insufficient incentives to action to prevent misuse. This has meant that regardless of the intention, in practice social media platforms have provided extremists with extremely potent communications infrastructure.
Over the longer term, problems with the design and operation of social media helped create a context in which a section of the UK population has become more receptive to anti-immigrant, racist, and extremist messages. Social media has served to normalise hateful content. Not only have platforms failed to remove such content (even where it violates the law and/or a platform’s T&Cs), but they have also actively promoted it to more users in navigation features such as newsfeeds, trending topics and hashtags, and by recommending groups, pages, and channels where such content is being actively discussed. Bad actors, including both extremist networks and foreign states seeking to sow division, use networks of fake accounts to seed and spread hateful content and conspiracy theories, gaming these recommendation systems through inauthentic likes and shares to generate yet further algorithmic amplification.
The cumulative impact of this is that a worrying number of UK users have been exposed to, and been drawn towards a toxic world-view where false claims about crimes committed by immigrants seem more plausible, and violence seems like a more acceptable response. As Hope Not Hate puts it, “The last decade has seen far-right extremists use social media to amplify their voices and attract audiences unthinkable for most of the postwar period.”
Alongside this dissemination and normalisation of destabilising content and ideas, social media has also provided extremists with the tools to build decentralised networks of sympathisers and activists, who are digitally connected via groups, pages, channels, and hashtags; share a set of ideas, narratives, and conspiracy theories; and take their lead from a common set of influencers/super-spreaders.
Some of this connective tissue and organising capacity is developed on smaller, less well-known platforms such as Gab or Telegram. But the largest platforms also play a crucial role. X has gained the most notoriety, after last November reinstating accounts of dangerous influencers such as Andrew Tate and “Tommy Robinson”. But whilst other mainstream platforms have maintained bans for these particular individuals, they continue to provide access to other accounts which share inflammatory content to large followings. “Turning Point UK” enjoys 190K followers on Facebook, and declared on 9 August that “anyone jailed from the Enough is Enough Riots will be seen as a political prisoner”. 2023 research commissioned by Ofcom found that the largest YouTube channel associated with the UK extreme right had close to 2 million subscribers.
All the mainstream platforms offer features - Facebook’s private groups for example - which also provide important focal points. Political agitators, and foreign-state backed operations, are able to use fake and anonymous accounts to seed and spread content within these groups.
This meant that social media had therefore helped create a context in which false information about Muslims and migrants had a receptive audience, including users who had become drawn into a violent, racist world-view - and where these users were loosely networked, in a way which lent itself to the organising of spontaneous, potentially violent street protests.
In the immediate aftermath of the horrendous killings in Southport, false claims that the suspect was a Muslim and an asylum seeker who had arrived by boat quickly spread across social media. The idea was promoted by known far-right influencer accounts, their followers, and fake accounts - and also promoted by platforms’ recommender algorithms, which pushed it to more users via news feeds, trending topics, and hashtags. The false name of the alleged attacker, “Ali al-Shakati” for example appeared on X as a “trending in the UK” topic, whilst on TikTok, search results for “Southport” recommended “Ali al-Shakati arrested in Southport”. Subsequently other falsehoods which slurred immigrants or Muslims (e.g. a false claim that two men were stabbed by a Muslim assailant at an anti-immigration protest in Stoke) were spread in a similar way.
Alongside the spread of these specific, incendiary falsehoods, social media platforms also enabled the expression of escalatory and inflammatory sentiments related to these falsehoods and the ensuing violence via groups, pages, hashtags (e.g. #enoughisenough, #saveourchildren; local pages/groups/channels; far right aligned pages/groups/channels). These same channels enabled the instigation and promotion of local disorder through the sharing of targets, locations, meeting points and times, followed by live streams from events.
Some of the most direct incitement took place on smaller platforms such as Telegram, but some appeared on the largest platforms - for example the violence targeting a mosque in Southport on 30th July was promoted by an anonymous TikTok channel which published videos including statements such as “No Face, No Case: protect your identity”. Such content was again amplified through a mixture of sharing by genuine users, fake accounts, and platforms’ algorithmic recommendation systems.
Many comments clearly passed a criminal threshold for offences including incitement to racial hatred and disorder - several social media users have subsequently been jailed for incitement. And a significant volume surely also violated platforms’ stated Terms of Service against hateful conduct. Yet, not only were they not moderated in a timely fashion, but in many cases they were amplified by platforms' recommender systems.
An obvious limitation of the Online Safety Act, (though one which we believe could be minimised with vigorous and creative enforcement), is that disinformation could appear not to be fully within its scope. Some disinformation will clearly be caught because it is a priority offence (such as hate speech, or the false communications offence, or foreign interference). However, harmful manipulation of the information environment does not only occur through content which is obviously criminal, and is not always tasked, directed, or arranged by a foreign power (something which in any case can be very hard to prove conclusively).
For example a network of accounts which manipulates discourse by amplifying negative media stories and stereotypes about Muslims may not be caught. Nor may a network spreading false rumours of vote rigging in an election, or false stories of climate change being a “hoax” propagated by a “global elite”. This could place limits on Ofcom’s ability to direct platforms to implement measures which aim to address harmful disinformation in all its forms.
The only provisions in the OSA which could address disinformation more broadly are The Advisory Committee on Disinformation and Misinformation, which is to be set up to advise Ofcom, and changes to Ofcom’s media literacy policy to include social media - necessary but nowhere near sufficient provisions. There are no specific measures to address health misinformation (e.g. regarding vaccines), or election disinformation (unless caught by a foreign interference offence), nor misinformation during ‘information incidents’ such as during terror attacks or during the August 2024 riots. These are legislative gaps which we recommend that the government considers filling. For example the EU’s DSA includes provisions for additional measures which platforms are required to implement during elections to promote “electoral integrity”, whereas the OSA currently makes no such provision.
That said, many of the features and functionalities which fuel the spread of harmful misinformation are the same features and functionalities which enable other forms of harmful behaviour which are already caught by the OSA. For example financially-motivated scammers, hostile foreign governments, and extremist political movements all make use of networks of fake accounts in similar ways. In some cases the same inauthentic networks may even be used for both financially motivated and political purposes at different times. If Ofcom could be encouraged to make use of its existing powers more fully, to tackle the priority offences already designated by the OSA, the changes this would require to platforms’ design would likely have knock-on benefits for other forms of harmful content including disinformation.
Whilst we understand that any regulatory framework takes time to set up, we are concerned about the pace of implementation thus far. We question whether consultation documents truly need to be quite so many hundreds of pages long or why quite so many months need to pass between consultations closing and final codes being issued. The biggest downside of this slow pace is that it means UK users are still waiting for any additional protection from harm. But another downside is that it makes it harder to say for sure how effective the Online Safety Act is because for the most part it has yet to be implemented.
The Illegal Content Codes have finally been published, just two days before the deadline for this inquiry, and stretch to 2,500 pages which has allowed limited time to fully analyse them. But at first read they do not appear to depart that significantly from the drafts published last year (which itself of course raises questions as to how the intervening time has been spent), and the concerns which we expressed during that consultation remain.
A cross-referencing of Ofcom’s Register of Risks, which identifies risk factors associated with illegal harms, and their list of required measures under the Codes Of Practice, reveals many risk factors are left unaddressed. Thus the ability to create fake and anonymous profiles is identified in the Risk Register as a “stand out” risk factor, associated with a range of offences including Terrorism, CSEA, Harassment, Hate, Intimate Image Abuse, Fraud, and Foreign Interference. Yet despite that, which would have seemed to point to making it a priority, no mitigating measures are required in the Codes of Practice.
This is particularly problematic because, under S41(1) of the Act, a provider “is to be treated as complying with a relevant duty if the provider takes or uses the measures described in a code of practice which are recommended for the purpose of compliance with the duty in question”. This “safe harbour” provision, means the effectiveness of the measures which Ofcom recommends is critical. Ofcom needs to be sufficiently demanding to deliver the improvement in online safety which is the purpose of the Act and is the direction it received from Parliament. It is manifestly not the intent of S41(1) for a platform with a continued high prevalence of illegal harm to enjoy a “safe harbour”, simply because they’ve followed an insufficiently stringent set of measures.
In other words, S41(1) requires Ofcom to be confident that its measures are sufficiently stringent to fulfil the Act’s stated purpose of ensuring that companies “identify, mitigate and manage the risks of harm” and that their platforms are “safe by design”. It has to be obvious that the “safe harbour” was only intended by the legislators to be available in circumstances in which the measures and recommendations clearly satisfied the aims of the legislation in terms of reducing harms.
One issue which seems to be behind Ofcom’s timid approach is an approach to “proportionality” which fails to adequately consider the consequences of inaction. While the OSA requires regulated services take a “proportionate” approach to fulfilling their duties, and requires Ofcom to consider companies’ resources to comply, Ofcom is also required to look at the severity of harm. However Ofcom appears to have interpreted the requirement to act proportionately primarily as meaning it should avoid imposing costs on companies - even large platforms with vast resources at their disposal. In other words Ofcom appears to be adopting a definition of proportionality which restricts its ability to mitigate harmful design choices driven by platforms’ business model. This doesn’t make sense, and we would encourage politicians to challenge Ofcom to strike a more appropriate balance.
In our meetings with Ofcom we have repeatedly heard them reference a fear of being judicially reviewed by platforms - a fear of litigation by the platforms appears to be deterring them from recommending measures which could conflict with platforms’ business models. We have during at least two public conferences now told Ofcom officials that it is almost inevitable that platforms will seek to challenge regulation in the courts at some point – but that it is not a sensible strategy for avoiding this to duck challenging some of platforms’ most harmful design choices.
More than that, we have also told Ofcom officials, in a constructive spirit, that if they do not fulfil the obligations imposed on them by the OSA, then civil society will also have the option of suing them. The point is that there is so much at stake that it is inevitable that Ofcom will find itself in court. It has a choice whether it wants to find itself justifying decisions to protect the profits of the online platforms, or to be defending the enforcement of the OSA as it was intended by Parliament. If Ofcom do the latter, and do it well, it will see off any attempted judicial review and we will intervene in support in court.
The case of fake and anonymous accounts is an important illustration of Ofcom’s slow and timid approach to implementation. The ability to create fake and anonymous user profiles has been acknowledged by Ofcom in their risk assessment to be a “stand out” risk factor, associated with a very long list of “priority offences” including terrorism, harassment, hate offences, sexual exploitation and abuse, fraud, and foreign interference. Yet the Codes of Practice fail to propose significant measures to address this, limiting themselves to a rather modest measure aiming to address impersonation of notable figures or brands under “paid verification” schemes.
Ofcom’s justification for failing to even explore a measure requiring platforms to offer user verification in its Illegal Content Codes of Practice is that the OSA does require that measure for Category One platforms, which they would come to in “phase 3” of implementation. In other words they are leaving until last in the queue a risk factor which they themselves have acknowledged "stands out" as likely to cause most harm. Initially, Ofcom’s Roadmap to regulation” set out that it was planning to consult on phase 3 in 2025, with the measures coming into force at the earliest in 2026. Last month, Ofcom announced changes to the roadmap which included a year’s delay to phase 3, meaning consultation “up to a year later than originally planned” in 2026 and measures not in force before 2027.
We argued in our formal response to Ofcom’s consultation, that it made no sense not to consider a measure in the illegal content codes with the potential to address a “stand out risk factor” for priority offences, on the grounds that they’d be getting to it at a later date for a small subset of platforms. Delaying the measure would mean the harms enabled by fake and anonymous accounts - including priority illegal offences - would continue to impact UK users. Limiting the measure only to Category One platforms, rather than any platform where fake and anonymous accounts were a relevant risk factor in illegal behaviour, would limit the protection that UK users would ever experience. This was already a bad outcome for UK social media users under the previous timetable. A further 12 month delay makes it even worse. Despite our best efforts we have never had a convincing explanation as to why Ofcom would have decided to delay such an important provision.
We suggest the following additional measures could be taken by Ofcom and the UK government to combat potentially harmful social media and AI content. There is nothing to stop platforms acting voluntarily in the absence of government action. Indeed the largest social media platforms surely already have access to the resources and data to act. However, platforms have known about these problems for many years and chosen not to act, so we focus here on steps which can be taken by the regulator and the government to push them in the right direction.
Ofcom could do more to combat harmful content by:
The UK government could do more to combat harmful content by setting a timetable for reviewing the OSA and considering revisions to strengthen it, which could include considering whether changes are required to:
Finally, if we could make one suggestion for your Committee’s approach to this inquiry. Our feeling is that one of the problems with Ofcom’s approach is that it has allowed itself to be persuaded by the platforms that keeping UK citizens safe will be too expensive. We have appreciated our meetings with Ofcom, where the officials always say we raise valid points, but clearly they also are also in frequent meetings with the platforms and have accepted the platforms’ explanations about “proportionality”. We have urged many times now that the platforms articulate why implementing reasonable safety measures, such as user verification, would disproportionately damage their profits - in a forum where we can hear what they are saying and there is an opportunity challenge them. Perhaps this is a line of inquiry your committee could pursue in its evidence sessions?
17 December 2024