About the contributors
This contribution is made by The Alan Turing Institute’s Hate Speech: Measures & Counter-measures project, which forms part of the Institute’s Public Policy Programme. We are happy to work more closely with the House of Lords Select Committee and to provide any further information or guidance as needed. Please contact Dr. Bertie Vidgen, email@example.com
Dr. Bertie Vidgen is a post-doctoral researcher at The Alan Turing Institute, Research Associate at the Oxford Internet Institute and Visiting Researcher at the Open University. His research focuses on online abuse, hate speech, misinformation and the far right. He is a co-Investigator on The Alan Turing Institute’s project, ‘Methods for abusive content detection’, lead researcher on The Alan Turing Institute’s project ‘Hate speech: measures and counter-measures’ as well as an organiser of the 4th workshop on abusive language online (ACL). He received his PhD from the University of Oxford. In his work he combines advanced computational tools and social science theory, with the aim of contributing to both academic knowledge and the work of policymakers.
Professor Helen Margetts is Director of the Public Policy Programme at The Alan Turing Institute, and Professor of Society and the Internet at the University of Oxford and Professorial Fellow of Mansfield College. From 2011 to 2018, she was Director of the Oxford Internet Institute, a multi-disciplinary department of the University of Oxford dedicated to understanding the relationship between the Internet and society, before which she was UCL's first professor of Political Science and Director of the School of Public Policy (1999-2004). She sits on the UK government’s Digital Economy Council, the Home Office Scientific Advisory Council, the WEF Global Agenda Council on Agile Government and the Ada Lovelace Institute for Data Ethics. She was a member of the UK government’s Digital Advisory Board (2011-16). She is the founding Editor of the journal Policy and Internet, published by Wiley. In 2018 she was awarded the Friedrich Schiedel Prize by the Technical University of Munich for research and research leadership in technology and politics. In the 2019 New Years Honours List she was awarded an OBE for services to social and political science. In 2019 she was elected as a Fellow of the British Academy and also took up a visiting appointment as the John F Kluge Senior Chair in Technology and Society at the Library of Congress.
Question 8: To what extent does social media negatively shape public debate, either through encouraging polarisation or through abuse deterring individuals from engaging in public life?
Social media has transformed contemporary politics, from making huge audiences instantaneously accessible to sidestepping the role played by ‘gatekeepers’, such as the traditional broadcast media. Social media are where people find, consume and share political information and news, and participate, communicate and organise politically. In this way, social media enable users to engage in new ‘tiny acts’ of participation (through liking, following, viewing, sharing and so on) which have the positive effect of drawing people into politics (including groups where participation has traditionally been low) and sometimes this scan scale up into large mobilizations and campaigns for policy change. Although most such mobilizations fail, the speed and scale of the ones that succeed have had dramatic effects on political life across the world, with almost every country in the world seeing a rise in collective action organised through social media, and offering people with no more resources than a mobile phone the opportunity to fight injustice and campaign for policy change. Social media have had a number of positive effects on public life and political discourse, such as enabling like-minded people who are separated by time, space, social networks and culture, to connect with each other, and in those cases fostering respectable and constructive dialogues. Support networks and activist groups which would not previously be possible are now commonplace and it has never been easier to learn about competing perspectives, find out new information and to engage with people who have different views.
However, social media have also given new voice to otherwise marginalised actors across the political spectrum, such as extremist far right protestors and niche left-wing environmentalists. Some argue that the affordances of social media have fundamentally changed how people interact with each other. Anonymity and not being face-to-face with interlocutors mean that people are far more impolite and angrier. This line of reasoning suggests that although people troll, dox and ‘cancel’ opponents online, if they met face-to-face in the offline world then they would be civil and reasonable. As one study of comments below online news articles put it: ‘Anyone can become a troll’. For social science researchers, disentangling the effects – both positive and negative – of social media is a considerable challenge: its ubiquity and the volatile nature of contemporary politics means that we cannot be sure exactly which political developments are due to social media and which are due to other causes. And, much as we are often pressed to ‘take a side’ when discussing whether social media has had a positive or negative impact, the truth is inevitably more complex: depending on how it is used, and by whom, social media can have both a positive and negative impact.
This consultation response focuses on one particular aspect of online debate which has raised concern in recent years – abusive language. Our research in Hate Speech: Measures and counter-measures shows the myriad problems it creates for society: it can inflict harm on any victims who are targeted, create a sense of fear and exclusion amongst their communities, toxify public discourse and, even, motivate other forms of extremist and hateful behavior through a cycle of “cumulative extremism”. Online abuse is a deeply harmful part of online discourse which, whilst it is often defended on the basis of free speech, we should take every effort to better understand, challenge and eradicate.
Prevalence of online abuse
Surprisingly, relatively little is known about the prevalence of online abuse, which seriously constrains our ability to understand and tackle it. Through an ongoing review, we have identified several key insights about contemporary online abuse in the UK:
Overall, these results suggest that whilst the prevalence of online abuse is low, especially in terms of content which is illegal or contravenes platforms’ guidelines, many people are still exposed to it. This somewhat counterintuitive finding (that online abuse is rare but still seen by many) is crucial for understanding the contemporary landscape of online abuse. It is a key reason for why online abuse is so hard to tackle: even just a few bits of abusive content online can have huge impact which, given the need to balance taking down abuse with protecting individuals’ freedom of speech and right to privacy, creates a ‘wicked problem’ for policymakers.
How abuse manifests
Abuse impacts on different people and groups very unevenly. For instance, MPs and public figures often receive far more online abuse than ordinary citizens. Our research on how MPs used Twitter during the 2017 general election campaign, also shows that how they receive abuse is very different to other online actors. We find that whilst there is always a background level of online abuse, infrequent but intense ‘events’ occur too: brief periods of time in which MPs are targeted by huge amounts of abuse from a wide array of Twitter users. Figure 1 shows the number of abusive tweets received by the Conservative MP Nadine Dorries. The turquoise sections show the occurrence of hate ‘events’ and the red sections show the background level of abuse she constantly experiences. Most of the abuse she receives happens during the short sharp event periods.
Figure 1, Abuse received by Nadine Dorries during the 2017 general election
Intense hate events are likely to have considerable emotional and mental health impact on MPs and may create the perception that everyone is opposed to them – even though, in many cases, it is just a small but vocal minority who are attacking them for a brief period. We need to develop ways of providing more support to figures who are in the public eye, and not just accept online abuse as part of the rough and tumble of politics. Otherwise, we risk excluding certain types of people from public spaces, in particular those who are either more vulnerable or have less support around them. This is not only unfair but will also reduce the diversity and quality of public discourse, negatively impacting all citizens and potentially alienating some groups from civic engagement.
Our research also shows that experiences of online abuse are also highly uneven in other ways, such as by peoples’ demographics. This is based on original analysis of previously unpublished data from the Oxford Internet Institute’s 2019 Oxford Internet Survey (OxIS). We are grateful to the authors for giving us permission to use this data, and to Dr. Grant Blank for making it available. Our results show that younger people, non-Whites and more regular Internet users are more likely to experience online abuse. This unevenness in how online abuse manifests means that when we talk its impact on online democracy, participation and discourse we should not adopt a ‘broad brush’ and act as though all users’ experiences are the same: experiences of online abuse vary systematically by background and activity.
Online abuse: OxIS 2019 survey results
Two questions from the 2019 OxIS survey relate to online abuse
(1) Have you seen cruel or hateful comments or images posted online?
(2) Have you received obscene or abusive emails?
Figure 2, Experiences of online abuse, split by ethnicity (OxIS 2019 data)
Figure 3, Experiences of online abuse, split by age (OxIS 2019 data)
Figure 4, Experiences of online abuse, split by Internet use (OxIS 2019 data)
Finally, our results show that gender played a far smaller role in experiences of online abuse. For both variables, a similar proportion of Males as Females experienced online abuse. However, we note that other research indicates that gender is a crucial factor in experiences of online abuse, and so advise caution when interpreting this result.
Figure 5, Experiences of online abuse, split by Gender (OxIS 2019 ddata)
Question 11: How could the moderation processes of large technology companies be improved to better tackle abuse and misinformation, as well as helping public debate flourish?
Content moderation is crucial for ensuring a safe and fair digital ecosystem in which all users are free to reap the benefits of communication technologies. There is no such thing as a non-moderated online space: all platforms are ‘designed’ and users’ experiences curated. Platform design effects everything that users experience, including what features are available to them, what content they are exposed to, and how they can connect/engage with other people. Thus, the question facing society is not whether we want content to be moderated but how we want it to be moderated.
In this evidence submission, four key aspects of content moderation are discussed: (1) Scrutiny, transparency and collaboration, (2) Identifying harmful content: the role of technology, (3) How should platforms intervene? and (4) Time sensitive interventions in content moderation.
Evaluating the moderation processes of large (and small) tech companies is challenging for one overarching reason: they provide little information and only rarely open up their processes to outside scrutiny. For instance, according to Pew Research, the most popular social media platforms in the USA in 2019 are Facebook, Twitter, YouTube, Snapchat, Instagram, WhatsApp, LinkedIn, Pinterest and Reddit (Pew Research, 2019). Of these 9 platforms, only 4 (Facebook, Twitter, YouTube, Reddit) provide statistics on the amount of hate speech and bullying content they remove globally. At present, transparency reporting is entirely optional and there are no clear frameworks for what should be reported on. Platforms each report different statistics at different levels of granularity and use different categories. This makes it very difficult to compare their processes and to build an evidence base about what works and what needs to be improved.
Part of the reason for the lack of transparency, and inconsistency in how platforms report on content moderation, is that it is a commercially sensitive area. Nonetheless, the impact of ineffective content moderation is felt by all in society, particularly by marginalised and vulnerable groups. As such, whilst respecting their right to protect their commercial interests, platforms should aim to share more information with researchers, civil society and policymakers through cross-sector collaborations. We make three proposals for improving collaboration:
Detecting harmful content at scale in a timely, robust and fair manner is remarkably difficult. Despite the increasing sophistication of available computational tools, most companies still rely on humans to review individual posts – an approach which is both expensive and necessarily reactive (human moderators do not proactively find content, they only review content reported to them). To improve content moderation, industry practitioners need to approach it as not only an engineering task (requiring ever more computational power, big datasets and machine learning) but also a social challenge. This is because content moderation is fundamentally a question of fairness: who is given protection from unsolicited harmful content? Who is allowed/enabled to spread such content? Which groups systematically have their content over-moderated? Which groups are left feeling unsafe and, in some cases, feel they need to withdraw from online spaces? If we keep approaching content moderation as only an engineering problem, or a financial imposition, we will end up with coarse and ineffective moderation strategies which do not challenge discrimination and injustice but perpetuate them.
Based on our research at The Alan Turing Institute, we identify five challenges that content detection methodologies must address. These are based on our review of hate speech detection technologies but are also relevant for other types of harm. Given existing technologies it is highly unlikely that these challenges would be solved solely with automated computational methods:
Once harmful content has been detected, platforms have to decide how it should be handled. This is an area in which platforms could both provide considerably more information about their existing processes and do far more to strength public debate and ensure users’ safety online. Most public discourse around content moderation focuses on bans, such as suspending and blocking users or removing their content – but this is simply the most overt end of a wide spectrum of interventions, which include:
There are clear cases when suspending individual social media accounts may be viewed as entirely appropriate – such as with violent threats, paedophilic content and revenge porn. But, for many other types of content, platforms should consider drawing from the far wider array of options available to them. Unless we open up this space of debate, the ethical, political and social implications of these different options will not be fully explored and more nuanced and appropriate responses for dealing with harmful content will not be developed. Developing a wide range of moderation interventions will also expand the range and types of harms that can be moderated; if bans are your only option then you can only tackle the most extreme forms of content. If you have a wider range of interventions then you can deal with a wider range of problems.
Different interventions may work better for tackling certain types of online harm: some forms of misinformation are shared in order to drive traffic to certain websites and then let the hosts derive financial benefit from advertising (in which case demonetising could work well). In contrast, political figures might share false, inflammatory or hateful content to gain public attention and increase their notoriety (in which case constraining the number of shares could work) and some porn stars share shocking and sensitive content to drive attention, but are only interested in appealing to certain target markets (in which case a sensitivity warning could be effective). However, it is important to draw attention here to the fact that there is little evidence about what interventions work to tackle harmful online content. To remedy this, the Government should utilise its unique position to evaluate different approaches. More broadly, it should adopt a joint-up approach which considers how different types of harmful content intersect and considers the impact of content moderation on the entire digital ecosystem rather than a single platform. It is possible that when mainstream platforms aggressively moderate content they push some users to migrate to niche alternative platforms, which could, in turn, lead to greater radicalization – we need to build an evidence base that addresses these complex issues and not only think about platforms in isolation.
The 2019 Christchurch terrorist attack showed the need for platforms to develop time sensitive responses to harmful online content: in just a few hours the New Zealand terrorist’s homemade and livestreamed video of the attack had spread across social media. The challenge here is that during these highly volatile, unpredictable periods the amount of harmful online content sharply accelerates: tools, process and systems which are sufficient in normal periods start to break. To address this challenge, we propose four strategies for short intense periods where there is a spike in the amount of harmful content:
 For a full discussion, see: Margetts et al. (2015), Political Turbulence: how social media shape collective action, Oxford: Princeton University Press.
 Cheng et al., ‘Anyone can become a troll: causes of trolling behaviour in online discussions’. Available at: https://dl.acm.org/citation.cfm?id=2998213
 As discussed. In the recent Online Harms White Paper from DCMS and the Home Office, available at: https://www.gov.uk/government/consultations/online-harms-white-paper
 A policy briefing report summarising the available evidence on online abuse is forthcoming from The Alan Turing Institute’s Hate Speech: Measures and counter-measures team.
 Reported by Facebook here: https://www.facebook.com/FacebookSingapore/posts/did-you-know-that-475-billion-pieces-of-content-are-shared-daily/563468333703369/
 See Gorrell et al. (2018), ‘Twits, Twats and Twaddle’, available at https://www.aaai.org/ocs/index.php/ICWSM/ICWSM18/paper/viewPaper/17861
 Publication is forthcoming.
 Please see OxIS’s website for more information about the survey: https://oxis.oii.ox.ac.uk and Dr. Grant Blank’s online profile: https://www.oii.ox.ac.uk/people/grant-blank/
OxIS is the longest-running academic survey of Internet use in Britain and uses a multi-stage national probability sample of 2,000 people. Current internet users, non-users and ex-users are included, which allows the data to be used for representative estimates of Internet use, with a low margin of error.
 Figures from the USA are used as comparable survey data for the UK is not currently available.
 See: https://www.aclweb.org/anthology/W17-1101, https://arxiv.org/pdf/1705.09899 & https://www.aclweb.org/anthology/P19-1163
 Vidgen et al., ‘Frontiers and Challenges in abusive content detection’, Proceedings of the 3rd workshop on abusive language online, Available at: https://www.aclweb.org/anthology/W19-3509
 See: https://www.aclweb.org/anthology/W18-5112
 For more information about regulation of online harms, see the Turing’s Public Policy Programme’s response to the ‘Online Harms’ white paper: https://www.turing.ac.uk/research/research-programmes/public-policy/programme-articles/turings-public-policy-programme-responds-online-harms-white-paper
 Based on a previous blog post by this submission’s author for The Conversation: https://theconversation.com/four-ways-social-media-platforms-could-stop-the-spread-of-hateful-content-in-aftermath-of-terror-attacks-113785
 See: https://conferences.sigcomm.org/imc/2017/papers/imc17-final145.pdf