Written evidence submitted by Dr Giulia Gentile and Professor Lorna Woods (SMH0038)

(Both Essex Law School). The views expressed do not represent those of our employers or institutions of affiliation.

1)     Social media and search engines’ business model and the spreading of social harms

Social media and search engine’s business models rely on two features: attention from users,[1] and personalised targeted content.[2] This business models aims to enhance users’ engagement with personalised content, which, in turn, gives rise to a circle of data extraction and iterative tracking of choices, preferences and interests. As part of this model, some services share revenue with content creators; there are incentives then for creators to create and share content that generates engagement. The demand of increasing engagement impacts both how content is spread and created.

As demonstrated by extensive research,[3] the attention- and personalised-content-based business model may lead to polarisation and radicalisation precisely thanks to the spreading of harmful information, such as hate speech or ‘fake news'.[4]

The attention- and personalisation-centred business model can facilitate the spreading of harmful content in several ways.

First, harmful content can be circulated through advertising. Meta[5] and other advertising platforms show sponsored content that reflects users’ taste and interests – and this can include harmful and even illegal content. For example, a recent study has revealed that adverts for games which did not disclose the presence of loot boxes, in breach of the applicable rules, were watched on TikTok around 300 million times.[6] Meta has had difficulty enforcing its own rules on adverts for guns.[7]

Second, emotionally engaging yet harmful content may spread because of its popular nature. For instance, Meta’s algorithm boosts content with a high number of likes, shares, and views as it signals higher engagement[8] -- which can also include harmful content.

Third, social media’s business model can encourage the spread of harmful content by amplifying the engagement with content similar to what users watched in the past. In other words, if harmful content was previously watched, the social media’s algorithm may show similar harmful content to the same user in the future. The similarity criterion is applied by the Instagram Reels’ algorithm.[9]

Fourth, harmful content may be diffused by social media platforms that seek to increase engagement with recently shared content. An example on point is Twitter’s (X) algorithm,[10] which boosts content recently shared.

Fifth, harmful content may spread when social media target with harmful content profiled users based on specific features, such as their device type, language preferences, country settings, and other selected categories. This is the case for TikTok’s algorithm,[11] which targets users belonging to the same categories with the same content. In practice, in case user X belonging to category A and B has watched a certain type of content, the latter content can be shown by the algorithm also to users Y and Z if they belong to the same categories.

Regardless of the business model of social media, the main challenge around the fight against harmful content (excluding ‘illegal content’ within the meaning of the Online Safety Act) relates the very notion of ‘harmful content’.

Harm is a multi-layered, context-dependant concept.[12] Such complexity and relativity are reflected in the notion of ‘harmful content’, which is equally relative and contextual in nature: content that may be harmful for an individual may not be as such for another individual. Despite this, the nature of rules is that they have to apply generally. Accordingly, a completely individualistic approach is not practicable; moreover, such an approach is likely unworkable at scale.

At the same time, some content, such as misinformation and fake news, is harmful for the community in itself because it is liable to negatively impact the passive dimension of freedom of expression, (ie the ability to access accurate and reliable information), and, ultimately, the fundamental value of effective and informed democratic participation. The protection of effective and informed democratic participation should be considered in the context of the duties imposed on States to protect media pluralism and the right of the public to be ‘properly informed’.[13] We note, however, that the definition of harm[14] in the Online Safety Act is limited to individual harm, with societal impacts caught, insofar as they are caught at all, through considering harms to groups of individuals or indirect harms. It is far from clear that harms to the information environment or infringements of the passive aspect of freedom of expression would be caught by this framing.

An approach to definition of harmful content which tries to reflect these different considerations would combine more ‘objective’, collective standards such as the features of the content – eg whether content is accurate, reliable or not etc – with the subjective, individual aspects of the impact (and thus harm) of the content as understood across representative population groups (eg taking into account the likely developmental stage of different ages of children in the light of existing evidence).

The objective standards, such as accuracy and reliability among others, may be drawn from existing rules on freedom of expression developed by the European Court of Human Rights.[15] A similar approach has been acknowledged in the Ofcom’s note ‘Addressing harmful online content’.[16]

Another objective factor is the breadth of the reached audience. One question that has not been addressed by the current regime is whether those users who have a significant impact on the formation of public opinion (eg because of number of followers) should be subject to more specific rules around the quality of the content they produce.

2)     Online platforms’ ranking algorithms and the spreading of harmful content

The ranking algorithms used by social media platforms and search are forms of AI. The role of AI may increase as search engines and other services move to provide generative-AI-generated summaries. 

Social media and search engines use different methods in ranking their content. Because they are driven by data, ranking algorithms tend to be historical and reinforce previous preferences. 

For instance, Facebook’s algorithm uses parameters such as the content watched by Facebook friends and followers, the level of engagement, the content type based on preferences of users, the quality of content measured in the form of authenticity, informativity, and content accuracy to rank content.[17]

The Youtube algorithm relies on the watch history, the video performance (ie whether the content is popular), the context (ie preference to videos that viewers often watch together) and the watch hour (ie if any of the videos keep the viewers engaged and have a higher average watch time).[18]

As explained above, Instagram’s algorithm relies on similar factors but also the information linked to a specific post.[19]

The approach to content ranking adopted by these platforms does not seem to give particular weight to the ‘objective’ factors, including reliability among others, noted earlier; nor do ranking algorithms routinely distinguish between (ethical) journalism outputs – understood here as journalism following some of the norms subscribed to in media regulatory models (whether self-regulation or otherwise) -- and reproduction of gossip and rumours. While the traditional media has long also relied on some of the attention-grabbing techniques common in the field of disinformation and carry stories that do not fit into the category of hard news, they have an orientation towards trying to report a story accurately. They are required to do so under professional standards applicable to the sector.[20] 

We note that at least those services in Category 1 will have to have some regard to the need to protect news publisher content.[21]  A 'recognised news publisher' for the purposes of the Act is either a UK-licensed broadcaster or is an entity the principal purpose of which is the publication of news-related material subject to its own editorial control and which is subject to a standards code.[22] While the news publisher content duties are aimed mainly at imposing safeguards around take-down decisions, it could be that the satisfaction of the recognised news publisher text is another factor that should be given weight in recommender systems.

3)     The role of Large Language Models and generative AI in spreading harmful content

Large Language Models (LLMs) and generative AI (genAI) have historically contributed to the spreading of harmful information as the content they create can contain ‘hallucinations’.[23] The integration of these tools into social media platform raises the risk that there will be an increase in synthetic content. As well as the inaccurate information in a particular post, the existence of AI hallucinations leads to a general uncertainty about the accuracy of anything in the digital environment where AI could have been used.[24] This uncertainty may give rise to what has been termed the liar's dividend,[25] as well as checking costs. In the context of racist speech, many AI models – because they use historic data – replicate biases in society and may produce racist, ableist or sexist content. Moreover, some concerns have been raised about AI content polluting the online environment which feeds into the base data for the models themselves.

To name but some examples of hallucination spread by LLMs and genAI, in May 2024 BBC reported that Google AI suggested that cheese should be glued to pizzas and that geologists recommended humans eat one rock per day.[26] These suggestions went viral. Previously, in 2023 a US lawyer used ChatGPT to draft written pleadings, and it later transpired that the judicial authorities included by the tool did not exist. The lawyer faced proceedings for violation of ethical and professional standards due to the use of ChatGPT.[27] The list of examples could continue and includes several cases where the information produced by AI tools is not so obviously false.

4)     Effectiveness of the UK regulatory framework

1. Additional measures to combat potentially harmful social media and AI content

The diffusion of AI-generated information is likely to have a twofold impact on the information environment: first, the speed of text generation through AI models could lead to an exponential overload, resulting in higher challenges in carrying effective content moderation and therefore scrutiny on online platforms; second, the reduced scope for scrutiny could lower the quality of publicly available information and thus the right to access information and to be properly informed of the public.[28]

A periodic revision of the state of enforcement of the Act, coupled with feedback and consultations with stakeholder parties (including platforms, civil society and users) appears a helpful measure, with particular reference to the challenges thrown up by new technologies. While the current concern relates to AI and genAI in particular, there will continue to be new developments which will change the information environment.

A Digital Services Act-style provision such as the Crisis response mechanism (Article 36 thereof) could provide a useful model for Ofcom and the National Security Online Information Team in managing situations where the spread of harmful information entails civil unrest. Crucially, the Article 36 model, while it envisages regulatory oversight, leaves the responsibility for reviewing the risks and identifying appropriate responses to the regulated services. It does not provide a mechanism where the Government may direct that specific content is dealt with in a particular way. (Note, while the DSA does also envisage a take-down regime whereby particular content items are identified, safeguards and oversight mechanisms have been included).

2. The role of the National Security Online Information Team (NSOIT) in preventing the spread of harmful and false online content

According to the previous Government's response to a written question, ‘the NSOIT identifies content which is within one of the areas of focus ministers have agreed, is assessed to pose a risk to national security or public safety and which is assessed to breach the terms and conditions of the relevant platform it may share that content with the platform. No action is mandated by the Government, it is entirely up to the platform to determine whether or not to take any action in line with their terms of service.’[29]

It may be that the NSOIT does work in such a way as to insulate the platforms from Government pressure. Nonetheless, the work of the NSOIT is not transparent; it would be better to put this body on a statutory footing with formal oversight mechanisms.

18 December 2024