Dr Tom Watts – Written Evidence (AIW0014)

Introduction

• This submission is authored by Dr. Tom F.A. Watts who is a Leverhulme Early Career Researcher based at Royal Holloway, University of London.

• My current project titled "Great Power Competition and Remote Warfare: Change or Continuity in Practice?" (ECF-2022-135) examines the relationship between the strategic practices associated with the concept of remote warfare, the dynamics of change and continuity in American foreign policy, and the use of autonomy in weapon systems.

• Between September 2020-September 2022, I worked as an Associate Researcher on the European Research Council (ERC) funded AutoNorms Project based at the Centre for War Studies, University of Southern Denmark. Alongside Dr Ingvild Bode, I co-authored a report examining the integration of automated and autonomous functions into air defence systems.[1] We are currently preparing to publish a new report examining the use of automated and autonomous functions as part of the global development, testing, and fielding of loitering munitions.[2]

• The views expressed in this submission are my own. They do not necessarily reflect those of my funder, institution, or research collaborators.

Recommendations

• The UK should adopt an operative definition of AWS aligned with the definitions used by key international partners such as the United States.

• The UK should acknowledge that weapon systems integrated with autonomous and automated functions are already in widespread global use. The lessons which can be learnt from existing state practices of operating systems which can select, detect, and engage targets with limited human intervention should therefore be examined.

• The UK should continue to recognise the importance of transparency as a mechanism for limiting popular misunderstandings of its policies regarding AWS. Where possible, more information should be released regarding the character of human-machine interaction involved with the operation of weapon systems with latent autonomous functions.
  

What do you understand by the term autonomous weapons system (AWS)? Should the UK adopt an operative definition of AWS?

• Since the start of the United Nations Group of Governmental Experts (GGE) debates on the emerging technologies in the area of lethal autonomous weapons systems, the task of defining an Autonomous Weapon System (AWS) has become increasingly politicised. There is no universally agreed definition of an AWS, and this is unlikely to change.

• Technical definitions of key concepts such as autonomy, automation, and artificial intelligence have similarly meant different things, to different stakeholder groups, at different times. This lack of consensus reflects the interest different stakeholders have in defining these concepts in a way which reflects their perceived interests.[3]

• An AWS can be broadly defined as “any type of weapon system which utilizes machine autonomy to select and apply force without immediate human control or intervention”.[4] An AWS is thus any weapon system which, once activated, can identify, track, and strike prespecified target profiles within the parameters established by human agents.

• The Ministry of Defence’s (MoD) now withdrawn Joint Doctrine Publication 0-30.2 Unmanned Aircraft Systems defined “fully autonomous weapon systems” as “machines with the ability to understand higher-level intent, being capable of deciding a course of action without depending on human oversight and control”.[5]

• The reference to “higher level intent and direction” rightly drew two sets of criticism and should be excluded from any future operative definition of AWS:

1. It set a very high (and possibly unattainable) technical bar for autonomy;

2. It diverged from the definitions used by many other states, impeding the UK’s ability to assume a leadership role in the international debates on AWS.[6]

• To help meet its ambition of working with its partners to “shape the international environment”,[7] the UK should adopt an operative definition of AWS aligned with those commonly used in the international debates regarding these technologies.

• The UK Government’s recently published Integrated Review Refresh 2023 describes the United States as being the UK’s “most important ally and partner”.[8] The Defence Artificial Intelligence Strategy similarly notes that the United States is the “world’s leading AI nation and our most important international partner”.[9]

• The UK should therefore consider aligning its operative definition of an AWS with the definition used by the United States.

• The United States does not have a government wide definition of AWS. These technologies are, however, defined in the Department of Defence’s (DoD) Directive 3000.09 Autonomy in Weapon Systems. First published in November 2012, a revised version of this directive was issued in January 2023 and will be valid for up to ten years.

• Directive 3000.09 sets out the DoD’s policies for the development, fielding, and review requirements for weapons installed with autonomous functions. It defines an AWS as being “[a] weapon system that, once activated, can select and engage targets without further intervention by an operator”.[10]

• Directive 3000.09 also outlines a definition for “semi-autonomous” weapons systems such as homing munitions which are used to attack prespecified target categories: “[a] weapon that, once activated, is intended to only engage individual targets or specific target groups”.[11]

• To be clear: DoD Directive 3000.09 does not provide an exact template for formulating the UK’s national policy on AWS. The many procedural and organisational differences between the DoD and the MoD make its wholescale adoption infeasible.

• Moreover, the requirement outlined in Directive 3000.09 that AWS be designed to “allow commanders and operators to exercise appropriate levels of human judgment over the use of force”[12] presents an ambiguous and possibly elastic standard of human-machine interaction.[13]

• Whilst recognising these points, aligning the UK’s operative definition of AWS with that outlined in DoD Directive 3000.09 could help amplify the UK’s contributions to the multilateral policy debates on these systems. Furthermore, it could help strengthen bilateral discussions with a key partner and developer of AI associated technologies.

What are the possible challenges, risks, benefits and ethical concerns of AWS? How would AWS change the makeup of defence forces and the nature of combat?

• This inquiry frames AWS as a novel set of technologies. This is reflected in this question’s call to examine “the possible challenges, risks, benefits and ethical concerns of AWS” and discuss “[h]ow would AWS change the makeup of defence forces and the nature of combat?”.

• As noted by the United States’ delegation to the UN GGE however, “[t]here is substantial State practice in using autonomous functions and features in weapon systems for decades”.[14]

• This includes the use of “defensive autonomous weapon systems”: namely, air defence systems which are described as utilizing “autonomous functions that assist in targeting incoming missiles or other projectiles”.[15]

• Air defence systems have been operated within certain control parameters to defend UK forces from time-sensitive attacks for decades. The Raytheon Block 1B model Phalanx Close-In Weapon Systems rotated between Royal Navy and Royal Fleet Auxiliary vessels, for example, provide short-range defence against anti-ship missiles. Described as an “automated” weapon system in the MoD’s now withdrawn Joint Doctrine Publication 0-30.2, Unmanned Aircraft Systems,[16] the Phalanx can reportedly track up to six targets simultaneously. Once activated in an automatic mode, it can detect, track, and engage incoming missiles without immediate human supervision.[17]

• Although often overlooked in the international debates on AWS, it is important to study existing categories of weapons installed with autonomous and automated functions because research suggests that their operation has already shaped what is considered to be appropriate levels of human control over the use of force.[18]

• In certain situations, those tasked with operating air defence systems appear to have found it difficult to exercise deliberative control over these systems because of the speed at which they operate, the complexity of the tasks they perform, and the demands human agents are placed under when supervising their activities. As the human operator’s role in air defence systems has changed from active controllers to passive supervisors, human operators have on certain occasions appeared to have lost situational awareness and a functional understanding of how targeting decisions are made.[19]

• This inquiry should therefore examine the trend toward increasing autonomy in weapon systems rather than what the MoD’s policy paper on Ambitious, Safe, Responsible AI suggests is concerns regarding “introducing elements of autonomy to the use of weapons systems”.[20] In short: any analysis of the challenges, risks, benefits, and ethical concerns associated with AWS should include a focus on existing practices of developing, testing, and fielding systems with autonomous and automated functions.

• In addition to air defence systems, this focus invites scrutiny of the global testing and development of loitering munitions - a distinct type of expendable uncrewed aircraft which can utilise sensor-based targeting to conduct attacks against a range of objects.[21]

• The first loitering munitions such as the IAI (Israel Aerospace Industries) Harpy were developed during the 1980s. These systems were principally designed to locate and attack enemy radar systems as part of Suppression of Enemy Air Defence (SEAD) missions. Thereafter, a new generation of more portable loitering munitions have been developed to provide tactical battlefield support to infantry forces involved in the full spectrum of conflict from irregular warfare operations through interstate warfighting.

• An United Nations Panel of Experts on Libya report published in March 2021 suggested that Kargu-2 loitering munitions manufactured by the Turkish weapons manufacturer STM had been “programmed to attack targets without requiring data connectivity between the operator and the munition: in effect, a true ‘fire, forget and find’ capability”.[22]

• Whilst this claim is disputed by the system’s manufacturer, it underscores the importance of examining existing state practices of operating loitering munitions for the debates on the challenges, risks, and ethical concerns associated with AWS.

• This matters because, as with other states including the United States, Israel, Turkey, and Poland, the UK has manufactured loitering munitions for its armed forces.

• The Fire Shadow loitering munition was developed as part of the British Army’s Indirect Fire Precision Attack programme which specified an operational need for an “all-weather, 24-hour Indirect Fire System for the precision attack of targets at extended range”.[23] In 2007, a consortium headed by MBDA UK announced that its Fire Shadow loitering munition had been awarded the contract for the associated Loitering Munition Capability Demonstration Assessment.[24]

• An unspecified number of Fire Shadow systems were manufactured and underwent demonstration trials at sites in both the UK and Sweden between 2008 and 2011. A planned deployment of this system to Afghanistan in 2012 was cancelled due to concerns about its technical reliability.[25] The Fire Shadow would not ultimately enter service with the British Army. Its development was formally cancelled as part of the MoD’s Annual Report and Accounts 2017-18 at a reported loss of £95 million.[26]

What are your views on the Government's AI Defence Strategy and the policy statement ‘Ambitious, safe, responsible: our approach to the delivery of AI-enabled capability in Defence’? Are these sufficient in guiding the development and application of AWS? How does UK policy compare to that of other countries?

• The development of increasingly autonomous weapon systems is a crucial domain in the geostrategic competition which has been intensifying between the United States and China since the mid-2000s.[27]

• The unclassified summary of the 2018 National Defence Strategy published by the United States DoD underscored the importance of investing in the “military application of autonomy, artificial intelligence, and machine learning, including rapid application of commercial breakthroughs, to gain competitive military advantages”.[28] Similar themes are expressed in the 2021 National Security Commission on Artificial Intelligence and the 2022 National Defence Strategy.[29]

• A regular subtext of the United States written statements at the GGE has been that the integration of autonomy into weapon systems should not be stigmatized. The importance of considering the operational context in which AWS are deployed in any assessment of these technologies’ ethics and legality is emphasised. This principle underpins the position that autonomy could be used to reduce the risk of civilian harm in combat operations and help “effectuate” the intent of military commanders.[30]

• The UK’s position on AWS has evolved in line with the policy positions taken in Washington. This is reflected in the general themes of the Government's AI Defence Strategy and policy statement on ‘Ambitious, safe, responsible’ AI, as well as the recent joint proposal (alongside other partners) for a set of “principles and good practices”[31] regarding these technologies submitted at the GGE.

• When approached on a thematic level, recent UK and US policy documents share a commitment to designing AWS associated technologies through a set of “accountable”, “ethical” and “responsible” practices which are compliant with existing international humanitarian law.

• In both cases, the development of increasingly autonomous weapon systems is explained in functional and geostrategic terms: AWS are not only needed to improve the efficacy and warfighting capabilities of both state’s armed forces, but to preserve a competitive military advantage over potential adversaries.

• Moving forward, the UK should remain vigilant of the need to carefully balance the geostrategic pull of developing increasingly autonomous weapon systems with its stated commitment to “oppos[ing] the creation and use of systems that would operate without meaningful and context-appropriate human involvement throughout their lifecycle”.[32]

• UK policy on AWS associated technologies appears to be predicated on the stance that if “competitors” and “adversaries” elect to develop increasingly autonomous weapon systems, then the UK will be compelled to respond in kind.

• This perception, which underpins a wider set of concerns about a global AI “arms race”, risks generating a self-reinforcing cycle which could make the development of new types of autonomous weapon systems appear increasingly desirable.

• There is no easy solution to firewalling the UK’s commitment to the “responsible” development of AWS associated technologies. As the MoD’s ethical principles for AI in Defence rightly notes, however: “[w]hat our systems do, how we intend to use them, and our processes for ensuring beneficial outcomes result from their use should be as transparent as possible, within the necessary constraints of the national security context”.[33]

• Transparency is an important mechanism for building public trust in the UK’s approach to the development and fielding of AWS. It can help facilitate constructive dialogue with, and accountability from academia and civil society organisations. For this reason, measures such as the Ministry of Defence AI Ethics Advisory Panel appointed to the 2nd Permanent Secretary for Defence should be retained and, where appropriate, expanded in focus.

• Furthermore, whenever possible, more information should be released regarding the character of human-machine interaction involved with the operation of weapon systems with latent autonomous functions.

• As part of research co-authored with Dr. Ingvild Bode, I have created data catalogues detailing the autonomous and automated functions of a global sample of air defence systems and loitering munitions. This was achieved using a range of open-source material including: (1) press releases and marketing material from weapons manufacturers; (2) press releases and factsheets published by defence ministries; (3) technical and policy reports authored by non-governmental organisations; and (4) news reports from reputable international media outlets.[34]

• This type of open-source analysis allows researchers to develop a general understanding of how autonomy and automation could be used in these systems.

• However, it does not allow for making firm determinations of whether (and if so how) autonomous functions have, in practice, been integrated into any particular system. It is generally not possible to determine whether a system can be classified as an AWS because the operator’s role in authorising a strike cannot be verified.

• This ambiguity risks producing popular misunderstandings regarding the UK’s approach to the development of autonomy and associated technologies.

• For example, the UK should provide additional clarification and guidance on the claim made in the MoD’s Joint Doctrine Publication 0-30 UK Air Power, last updated in November 2022, that “[t]he UK does not possess armed autonomous aircraft systems, and has no intention of developing them”.[35]

• This is because the UK military has ordered an unspecified number of Switchblade-300 loitering munitions manufactured by the American defence company AeroVironment – an overview of which has been provided in the textbox below.

• It is important to discuss this specific system because AeroVironment officials have claimed that “the technology to achieve a fully autonomous mission with Switchblade pretty much exists today”.[36] It is therefore not inconceivable that, in the future, some commentators may speculate that the Switchblade 300 will be upgraded to identify, track, and strike prespecified target profiles with without immediate human control or intervention.

• Providing a potential template for releasing information regarding the character of human-machine interaction involved with the operation of this system, the United States has previously presented an overview of its Counter-Rocket, Artillery, and Mortar (C-RAM) System to the GGE.

• This presentation explained how autonomous and automated functions had been integrated into the C-RAM system to defend US military installations from various forms of aerial attack, as well as detailing the character of human-machine interaction involved with its operation.[39]

10 April 2023