Mr Ben Nimmo – written evidence (DAD0089)

Disinformation: Detecting Operations, Defining Norms, Deterring Threats

Ben Nimmo, Director of Investigations, Graphika

Deception online and avenues of response

Online deception and influence campaigns come in many forms and are conducted for many reasons. Their variety and the speed at which they evolve mean that the responses to such campaigns have to be flexible and collaborative, with different actors – social-media platforms, operational researchers and official bodies – each contributing in different but complementary ways.

The current situation is radically different from that in 2016, and has improved significantly, but further improvements are needed in three core areas: detecting threat actors at an early stage, defining social and legal norms to guide cross-platform enforcement, and deterring future malicious behaviour. Political leadership can play an important role in defining norms and deterring future threats. 

The scope of the challenge

Deceptive online campaigns vary widely.  They can include companies or groups running large numbers of fake accounts that they rent out as followers to social-media users who want to look important,[1] or using fake accounts to post positive reviews of their own products.[2] They can include scammers who post political content to make their readers click on pay-per-click advertising sites[3] or pornography sites,[4] or who try to defraud readers with fake personas, subscription offers or hard-luck stories.[5]

They can also include media outlets that create networks of fake accounts to promote their content,[6] and “black PR” firms that create similar networks to promote political or business clients.[7] They can include opaque, pro-state groups who run fake accounts to drive users towards propaganda websites,[8] or who post propaganda on the social-media platforms themselves.[9]

On a more individual level, they can include operations that set up fake personas, often posing as journalists or activists, to promote their own narratives or discredit their opponents.[10] They can even include smear campaigns from state-run media,[11] attacking critics with false or distorted claims that are then amplified by genuine users, some of whom may not know that they are boosting state-funded messaging.[12]

Making the challenge even more complex, sophisticated influence operations can combine different elements. For example, clickbait websites which are only aimed at making money can use political content to attract attention.[13] Political operators can use apparently commercial “bots”[14] for amplification[15] or intimidation.[16] Commercial companies can rent out entire campaigns aimed at political influence.[17] Thus, deceptive operations online do not fall into discrete sections, but form a continuum, with different operations characterised by different combinations of tactics, and often working across many different platforms.[18]

The main connecting feature between such operations is falsehood, but even that falsehood can play out in different ways. Many operations use fake assets, ranging from individual social-media accounts to entire websites that purport to be legitimate news sources.[19] Some promote false information or doctored leaks[20] on real channels.[21] Some use false personas to leak genuine documents aimed at discrediting their targets.[22] Some operations avoid falsehood entirely and focus on promoting one-sided coverage of key events, omitting key voices rather than falsifying the facts.[23]

The complexity of this situation is, as yet, poorly understood by the general public. The last three years have seen a significant increase in the volume of reporting on issues such as bots, troll farms, and profile pictures generated by artificial intelligence, but some of this reporting has been overly dramatized and tended to create confusion rather than clarity. One key long-term need is for greater media and public understanding of the different scale, significance and likely impact of different online operations.

Detecting false behaviour patterns

When trying to expose influence operations, the first challenge is to detect them. The social-media platforms carry the primary responsibility for detecting malicious activity on their platforms, but independent researchers and appropriate governmental agencies also have a part to play. This is not an area where it would be appropriate for elected officials to play a direct role, but legislation can help to shape the environment in which investigators work, and can hold the social-media platforms to account, making sure that they have the processes in place to learn and implement the lessons of recent years.

The year 2016 was the nadir in terms of detection. The Russian operation that targeted U.S. politics that year successfully registered a Twitter account posing as a Tennessee Republican to a Russian mobile-phone number[24] and paid for political ads on Facebook with Russian roubles,[25] and still managed to operate into late 2017. Since 2018, however, Twitter, Facebook and Google have significantly ramped up their detection and disruption efforts, announcing takedowns of information operations from areas diverse as Central America,[26] Europe,[27] the Middle East,[28] China,[29] the United States and Vietnam.[30] At the same time, they have instituted changes designed to make information operations harder to run and easier to spot: for example, Twitter has cracked down on mass automation of posts, while Facebook now routinely shows which country pages with a significant following are managed from.

Platforms are working increasingly closely with external investigators, including the author of this paper, to expose multi-platform operations. In May 2019, Facebook detected a Russian operation on its platform and shared its findings with the Atlantic Council’s Digital Forensic Research Lab (DFRLab).[31] The DFRLab investigation found that the operation spread across over 30 other platforms, including a number of German and Austrian sites.[32] In November, Reuters and Graphika demonstrated that a separate leak of UK-US trade documents had originally been posted on some of the same sites, using the same techniques to evade detection.[33] Platforms including Medium and Reddit provided further insights, and finally Reddit attributed the amplification of the trade leaks back to the Russian operation,[34] and exposed almost 40 more accounts that the Russian operation had also run.[35] The Russian operation ran across many platforms: it took insights from many different investigators to expose its full scope.

Many of the recent takedowns were the result of internal investigations by the platforms, but some were triggered by external researchers: for example, a large-scale Iranian operation across multiple platforms and websites was first exposed by investigators at cyber-security firm FireEye.[36] Just before the mid-term elections in the United States in November 2018, the FBI tipped off the platforms that it had found a live Russian operation on their services, triggering an instant investigation and takedown.[37]

When external researchers find online operations before the platforms do, it is often seen as proof that the platforms need to improve their detection capabilities. This is, in fact, a standing challenge: platforms will always need to do more and do it better, because the more persistent malicious actors respond to takedowns and platform changes by searching for new ways to stay hidden.[38]

This operational detection and exposure of information operations is not an area where parliamentary or government intervention can or should play a direct role. However, regulation could play a supporting role. Based on the White Paper on Online Harms, one role for an independent regulator could be to ensure that all platforms above a certain size have processes in place to quickly investigate potential influence operations on their properties, and processes to learn the lessons from previous operations so that they become better at spotting repetitive tactics. This would be especially important as social-media platforms become more diverse, with major platforms headquartered in countries other than the U.S.

Regulation could also help to ensure that the legal framework promotes and enables cooperation between platforms and external investigators,[39] including how to share information and data.[40]

The framework would need to work along three vectors: ensuring that platforms provide appropriate information and data to researchers; ensuring that researchers adhere to professional standards and do not, for example, sell on data or violate innocent users’ privacy; and ensuring that the legal framework is sufficiently flexible to allow platforms and researchers to share information about accounts that are suspected of conducting information operations, without the fear of violating regulations such as GDPR. These are complex issues, as they seek to reconcile the need for transparency with the need for privacy, but the debate is necessary to provide a foundation for cooperation between multiple different platforms and research groups.

Defining the terms

Parliamentary and governmental leadership have more potential when it comes to the question of establishing norms for acceptable behaviour. At present, each platform has its own terms of service and its own interpretation of them, and behaviour which would be considered a violation on one platform can be considered acceptable on another. Given the nature of the different platforms and the vital importance of avoiding any erosion of freedom of speech, this patchwork will continue, but political leadership could help promote the creation of a core of shared norms and define the scope of acceptable behaviour, especially in political campaigning.

This will require political leadership by deed as well as by word. As a bare minimum, political parties and candidates should clearly articulate their own standards of online transparency – for example, stating that they will not use fake accounts or inauthentic amplification, and will not tolerate the use of such activity on their behalf, including by political consultancies. The ability of political leaders to set the tone of the debate, and to act as models of good behaviour, or of justification for bad behaviour, should not be underestimated.

Political leadership could also promote research into, and discussion of, the question of how norms of behaviour are created - that is, how to arrive at a set of informal but broadly agreed standards of online behaviour that do not rise to the level of legislation. Social media could be seen as representing, in effect, the emergence of new societies in which citizens interact in new ways; as such, the ongoing debate over standards of online behaviour could be seen as a search for new social norms within those societies. Research into the creation of such norms in other social settings could shed light on how abusive and threatening behaviour can be minimised by means other than the purely legislative.

Finally, regulators could also play a role in providing clear guidelines over how terms such as abuse, harassment and incitement to harassment are to be interpreted as they apply to online speech. Reports of public figures, and some private ones, leaving social media because of harassment remain a regular feature of internet coverage, and abuse against investigators, journalists and human-rights defenders remains a grave concern.[41] Freedom of expression is a vital component of democracy and must be protected, but hate campaigns against public figures are themselves aimed at stifling free speech. Without clear guidelines in place, enforcement is always likely to remain ad hoc and patchy. Providing a clear framework would potentially enable the platforms to take action more quickly and more consistently.

Deterrence

A final role for regulation could be in the realm of deterrence. In general, the strongest sanction a social-media company can take against malicious actors in the information space (as opposed to actual criminal activity) is to shut down all their accounts and ban them from the platform. This disrupts the operation, but it does not prevent the operators from trying again: Russia’s Internet Research Agency, for example, had at least five successive sets of assets taken down between September 2017 and October 2019.[42]

Platforms cannot impose direct financial or organisational costs on groups which conduct information operations or sell fake engagement online: as such, the platforms have a strong disruptive power, but very little deterrent power. This is an area where official action could conceivably play a deterrent role.

In November 2019, the US Federal Trade Commission for the first time fined a US-based company for selling fake followers and likes on social media, arguing that this empowered the company’s clients to deceive potential investors, partners and employees, and thus constituted an illegal deception.[43] It is too early to say what effect this ruling will have on the trade in fake accounts in the US; the company itself went bankrupt after the New York Times exposed it in 2018.[44] However, legislation that imposes an increased cost on fake-account operators, as selling a non-existent product (engagement from “people” who do not exist), would have the potential to deter some operators, at least those based domestically, narrow down information operators’ access to false amplification, and signal the political will to address the problem.

Political will could also reinforce the UK’s ability to deter covert foreign influence attempts, such as the Russian amplification of the US-UK trade leaks during the 2019 election. At present, information and interference operations are a low-cost and deniable activity for foreign states, and they are often perceived as having a huge potential impact. (This perception is exaggerated, but persists.) As such, they are likely to remain attractive.

UK government policy could aim at reinforcing the national ability to detect and attribute foreign information operations, and to increase the costs of such operations via a range of political, diplomatic and economic sanctions. These should look beyond targeting the operators and their immediate sponsors, and consider ways to increase the cost for the ultimate beneficiary of the operation, if this is a foreign government. This is an area where political will and diplomatic determination will be crucial.

Ultimately, the UK’s ability to deter future interference will require action across multiple policy areas, including diplomacy and law enforcement, to address not just the platforms where information operations are conducted, but the operators behind them.

2