Written evidence submitted by Dr Kimberley Hardcastle (SMH0026)
Newcastle Business School, Northumbria University
Personal background and expertise
Dr Kimberley Hardcastle is an Assistant Professor in Business and Marketing at Newcastle Business School, Northumbria University. Her expertise lies in the societal implications of emerging technologies, including AI, algorithms, digital platforms and consumer behaviour. She is a member of the steering group for The Northumbria Centre for Responsible AI, which is dedicated to ensuring the responsible development of AI. Her current research includes working with key partners in the North East to adopt a citizen well-being approach, safeguarding communities, upskilling the region and addressing the challenges posed by emerging technologies. With a strong research focus on the business models of digital platforms and their role in shaping societal outcomes, she is well-placed to provide critical evidence on how algorithms, social media and search engines contribute to the spread of misinformation, disinformation and wider social harms.
I make this submission as part of the Science, Innovation and Technology Committee's inquiry into the links between algorithms used by social media and search engines to rank content, generative AI and the spread of harmful or false content online. The submission focuses on three questions from the call for written evidence:
1. To what extent do the business models of social media companies, search engines and others encourage the spread of harmful content and contribute to wider social harms?
Profit-Driven Attention Economy
1.1 Social media companies and search engines' core business models are rooted in the attention economy (1). Herbert A. Simon, a renowned economist, psychologist and Nobel laureate first coined the term "attention economy" in the late 1960s, recognising the economic nature of information overload (2). The concept has gained prominence in the evolving digital age and refers to commodifying human attention as a scarce resource. Human attention is limited; therefore, attention is highly valued by businesses and organisations (3). The attention economy has significant financial implications, with major tech companies generating substantial revenues. Businesses, particularly social media and search engines, compete to capture and sustain user engagement, often leveraging data-driven algorithms to surface content that maximises time spent on platforms (4).
(1) Myllylahti, M. (2018). An attention economy trap? An empirical investigation into four news companies’ Facebook traffic and social media revenue. Journal of Media Business Studies, 15(4), 237-253.
(2) https://www.un.org/sites/un2.un.org/files/attention_economy_feb.pdf
(3) https://www.humanetech.com/youth/the-attention-economy
(4) Brady, W. J., Wills, J. A., Jost, J. T., Tucker, J. A., & Van Bavel, J. J. (2017). Emotion shapes the diffusion of moralized content in social networks. Proceedings of the National Academy of Sciences, 114(28), 7313-7318.
1.2 Social media platforms and search engines generate revenue primarily through targeted advertising, which incentivises maximising user engagement. This reliance on ad generated revenue forms the core of their business model (5)(6). By collecting and analysing vast amounts of user data, these platforms create detailed profiles to deliver personalised ads that are more likely to engage users (7). This business model directly incentivises maximising user engagement, as longer time spent on platforms generates more advertising opportunities (8). Algorithms are thus optimised to surface content that keeps users active, often favouring emotionally charged, sensational or polarising material, which is shown to drive higher interaction rates and prolonged attention (8).
(5) https://www.statista.com/statistics/271258/facebooks-advertising-revenue-worldwide/
(6) https://www.marketingweek.com/social-media-spend-200bn/
(7) Araujo, T., Copulsky, J. R., Hayes, J. L., Kim, S. J., & Srivastava, J. (2020). From purchasing exposure to fostering engagement: Brand–consumer experiences in the emerging computational advertising landscape. Journal of Advertising, 49(4), 428-445.
(8) Facebook Papers – Profit Motive - https://facebookpapers.com/?s=&cat=&tags=44&post_type=post
1.3 Influential theories in psychology suggest that human moral senses are shaped by the social world, where emotionally charged events and narratives are central to moral evaluation and judgment (9). On social media, content that provokes strong moral or emotional reactions, such as outrage, naturally aligns with these psychological processes, resulting in higher levels of interaction and engagement (10). Platforms capitalise on this by using algorithms that prioritise such content, reinforcing the connection between emotionally charged material and user attention (11). Research shows that emotionally charged content, particularly material that provokes outrage or fear, garners more interactions and prolonged user engagement which exploits cognitive vulnerabilities such as the human tendency to prioritise emotionally salient information over rational deliberation (12).
(9) Haidt, J. (2001). The emotional dog and its rational tail: a social intuitionist approach to moral judgment. Psychological review, 108(4), 814.
(10) Fox, J., & Moreland, J. J. (2015). The dark side of social networking sites: An exploration of the relational and psychological stressors associated with Facebook use and affordances. Computers in human behavior, 45, 168-176.
(11) https://news.mit.edu/2018/study-twitter-false-news-travels-faster-true-stories-0308
(12) Martel, C., Pennycook, G., & Rand, D. G. (2020). Reliance on emotion promotes belief in fake news. Cognitive research: principles and implications, 5, 1-20.
1.4 Social media companies and search engines exploit our cognitive vulnerabilities by leveraging innate human biases (13) (14). Algorithms are designed to amplify emotionally charged or novel content, capitalising on the human proclivity for fear, outrage and validation to sustain attention and maximise engagement. This creates feedback loops that prioritise immediate emotional responses over critical and well-considered thinking, drawing users into cycles of compulsive interaction. As a result, algorithms designed to optimise engagement often amplify divisive or harmful content, including misinformation and disinformation. These vulnerabilities are key to the platforms’ ability to shape behaviour and sustain their profit-driven business models.
(13) Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good. Review of general psychology, 5(4), 323-370.
(14) Rouault, M., & Fleming, S. M. (2020). Formation of global self-beliefs in the human brain. Proceedings of the National Academy of Sciences, 117(44), 27268-27276.
1.5 Advertising has always been about persuading people to buy, but social media advertising is uniquely powerful because it combines AI data-driven predictions. The goal of these technologies within online platforms is to maximise user engagement, exert 24/7 influence over billions of users, redefine social norms and perceptions and use hyper-personalised targeting based on detailed behavioural profiles, creating an unprecedented scale of behavioural control unmatched by any previous advertising business model.
2. How do social media companies and search engines use algorithms to rank content, how does this reflect their business models and how does it play into the spread of misinformation, disinformation, and harmful content?
Network Effects and Content Amplification
2.1 The concept of message noise in social media environments highlights how the overwhelming volume of content disrupts meaningful communication and encourages diverse interpretations. This noise, which includes misinformation and irrelevant material, diminishes user focus and facilitates the spread of harmful narratives. Such dynamics highlight the dual challenges posed by content oversaturation and algorithmic prioritisation, illustrating the complex relationship between engagement-driven platform designs and their broader societal impacts (15).
(15) Hardcastle, K., Edirisingha, P., & Cook, P. (2024). Identifying sources of noise within the networked interplay of marketing messages in social media communication. International Journal of Internet Marketing and Advertising, 20(2), 164-187.
2.2 Social media platforms also utilise network effects to scale user engagement, they do this by designing systems where the value of the platform increases as more users join and interact (16). As users engage with content, algorithms identify and prioritise trending posts, creating viral feedback loops that amplify visibility and engagement (17). Viral content, regardless of its veracity, benefits from these effects as many social media algorithms prioritise content with high initial engagement. This creates feedback loops where harmful or false content is repeatedly surfaced to maximise clicks and advertising revenue. Features like likes, shares and comments encourage user participation, while recommendation algorithms connect similar users, reinforcing the cycle.
(16) Kietzmann, J. H., Hermkens, K., McCarthy, I. P., & Silvestre, B. S. (2011). Social media? Get serious! Understanding the functional building blocks of social media. Business horizons, 54(3), 241-251.
(17) https://digitalmarketinginstitute.com/blog/how-do-social-media-algorithms-work
2.3 Recommendation algorithms, developed by platforms like Facebook and YouTube to assist user decisions, create feedback loops where user choices often shaped by algorithmic outputs are fed back as new data (18). This cycle reinforces similar recommendations, amplifying already popular content and narrowing user exposure. Over time, this process forms filter bubbles or echo chambers, isolating users from diverse perspectives and suppressing differing viewpoints.
2.4 Echo chambers form not just due to shared beliefs but because they actively isolate users from opposing viewpoints. Echo chambers are reinforced by trust dynamics, where members rely solely on in-group information while dismissing external sources (19). When linked to algorithms, these dynamics become amplified, as algorithms prioritise engagement and surface content that aligns with users' established beliefs. Algorithmic amplification occurs when certain online content gains prominence at the expense of alternative viewpoints, driven by data from user interactions such as clicks, likes, comments and shares (20). This data fuels algorithms to prioritise similar content, shaping what users see and further amplifying dominant narratives while marginalising others (21). This creates a reinforcing loop where individuals see only content that strengthens their preexisting views, deepening the isolation from alternative perspectives. This has further implications for the sharing of misinformation, as studies have shown that people are more concerned with sharing information rather than the authenticity of the story (22).
(19) Diaz Ruiz, C., & Nilsson, T. (2023). Disinformation and echo chambers: how disinformation circulates on social media through identity-driven controversies. Journal of public policy & marketing, 42(1), 18-35.
(20) https://time.com/6111310/facebook-papers-disturbing-content/
(21) Facebook Papers – Algorithmic Harm – https://facebookpapers.com/?s=&cat=&tags=37&post_type=post & Facebook Papers – Misinformation - https://facebookpapers.com/?s=&cat=&tags=39&post_type=post
2.5 Algorithms are inherently persuasive technologies, designed to shape user experiences on social media platforms. Platforms like Facebook, Twitter, Instagram, Snapchat and TikTok leverage these technologies to influence users’ opinions, attitudes and behaviours, aligning their design with the core objective of maximising engagement and driving profit.
2.6 The Center for Humane Technology accentuates that persuasive technologies are deliberately designed to exploit human cognitive vulnerabilities to maximise user engagement (23). Social media platforms and search engines utilise techniques rooted in behavioural psychology to capture attention and sustain it for as long as possible (24) (25). Features such as infinite scroll, autoplay videos and personalised notifications tap into human tendencies like the difficulty of stopping incomplete tasks and the lure of variable rewards, creating compulsive usage patterns (26). These technologies are not neutral; they are crafted to hijack attention and manipulate user behaviour, often at the expense of well-being.
2.7 For example, infinite scroll leverages the human tendency to avoid decision-making by removing natural stopping points, encouraging endless engagement. Similarly, autoplay capitalises on the path of least resistance, keeping users watching content without active input. Personalised notifications exploit the fear of missing out, compelling users to return to the platform. These mechanisms align with the platforms' profit-driven models, which prioritise time spent and advertising views over user autonomy or mental health. By exploiting cognitive vulnerabilities, these persuasive technologies create a feedback loop that fosters overuse and amplifies the spread of emotionally charged or sensational content.
(23) https://www.humanetech.com/youth/social-media-and-the-brain
(24) Menon, V., & Uddin, L. Q. (2010). Saliency, switching, attention and control: a network model of insula function. Brain structure and function, 214, 655-667.
(25) Berridge, K. C., & Kringelbach, M. L. (2015). Pleasure systems in the brain. Neuron, 86(3), 646-664.
(26) https://www.humanetech.com/youth/persuasive-technology
2.8 The social media business model is designed to maximise engagement, as higher user interaction leads to more lucrative advertising placements therefore, algorithms favour ads and sponsored content, generating direct financial gains for platforms. However, this prioritisation of engagement and monetisation can result in users being exposed to sensationalised or biased information, further entrenching their perspectives.
2.9 The amplification of misinformation and harmful content is a direct consequence of how these algorithms function. Because they prioritise engagement, content that provokes strong reactions, regardless of its veracity or ethical implications, is often ranked higher. This engagement bias contributes to the formation of echo chambers, where users are predominantly exposed to information that aligns with their pre-existing views, amplifying confirmation bias and facilitating the spread of misinformation and disinformation. This is exacerbated by the opaque nature of algorithmic processes; the lack of transparency surrounding exactly how algorithms make such decisions hinders accountability and makes it challenging to address systemic biases. This can have serious consequences, such as public unrest, the erosion of trust in public institutions, negative psychological impacts from exposure to harmful content and increased political polarisation due to the amplification of extreme viewpoints.
3. What role do generative artificial intelligence (AI) and large language models (LLMs) play in the creation and spread of misinformation, disinformation, and harmful content?
3.1 Generative AI tools, such as OpenAI's GPT models, play a significant role in the creation and spread of misinformation, disinformation and harmful content by enabling the mass production of convincing fake news, deepfake videos and synthetic social media posts (27). These models can impersonate credible sources by mimicking human writing styles, making it easier for ‘bad actors’ to create content that appears authentic and trustworthy. Once generated, this content is amplified by social media algorithms optimised for engagement, further spreading false or harmful narratives across platforms.
3.2 The challenge in detecting AI-generated content arises from its increasing sophistication, which makes it harder to differentiate from authentic material. Existing detection mechanisms often lag behind the rapid advancements in generative AI, exacerbating the spread of misinformation and complicating efforts to address its impact (28).
3.3 A study by MIT researchers found that AI-generated content can be difficult to distinguish from human-written text, models like GPT-3 can generate incorrect text that appears convincing, which could be used to generate false narratives quickly and cheaply for conspiracy theorists and disinformation campaigns (29). The sophistication of generative AI has made it increasingly difficult for traditional detection tools to identify fake content. Researchers at the University of California, Berkeley and other institutions have highlighted the challenges posed by AI-generated text and deepfakes, noting that current detection algorithms are often outpaced by advancements in AI (30).
(30) https://www.ischool.berkeley.edu/news/2024/new-research-combats-burgeoning-threat-deepfake-audio
Summary
4.1 The business models of social media companies and search engines rely on engagement-based advertising, where revenue depends on capturing and maintaining user attention. Algorithms amplify emotionally charged, sensational, or polarising content to maximise interactions, creating feedback loops that fuel misinformation, disinformation and social harms like polarisation and echo chambers. By exploiting cognitive vulnerabilities through persuasive technologies, such as infinite scroll, autoplay and personalised notifications, platforms prioritise profit over user well-being. Transparency around how algorithms operate remains limited, leaving users and regulators unable to fully scrutinise how content ranking decisions contribute to these harmful dynamics.
4.2 Generative AI and large language models (LLMs) present an additional layer of risk by facilitating the rapid and scalable creation of misinformation and disinformation. These tools can generate hyper-realistic text, images and videos that are increasingly difficult to distinguish from authentic content, accelerating the spread of false narratives online. Changing incentives is critical to mitigating these harms; the focus must shift away from profit-maximising engagement models toward safeguarding information integrity and user well-being. Regulatory interventions should mandate transparency around algorithmic processes. Currently, the algorithms are designed to optimise for what captures attention or drives consumption. Regulators should require companies to disclose the data they use and how their algorithms work, building a framework that shifts from a consumer protection approach that treats AI like a traditional static product to one that treats AI as a dynamic, complex system requiring full transparency and ongoing oversight ensuring that platforms prioritise societal interests over short-term financial gains.
17 December 2024