Peter Burt, Researcher, Drone Wars UK – Written Evidence (AIW0008)

               Drone Wars UK is a UK based non-government organisation which undertakes critical research into the use of armed unmanned systems, commonly known as drones, and emerging military technologies such as artificial intelligence (AI).  We scrutinise the increasing use of technology for security purposes, undertaking research, education, and campaigning on these issues, and advocate a human security approach (see www.dronewars.net for more information).

               Our recent work has included a study investigating the limitations of human control over highly automated weapon systems,[1] a review of the military applications of AI,[2] and an assessment of Ministry of Defence (MoD) policy on the military use of A1.[3]  We are a founding member of the Global Campaign to Stop Killer Robots and the UK Campaign to Stop Killer Robots.

Question 1

Autonomy in weapons systems is a cause for concern as it threatens human rights, raises issues over  compliance with the laws of armed conflict, and is based on complex and often opaque technologies which may not be clearly understood by the operator.

Drone Wars UK adopts the approach proposed by Article 36 for understanding autonomy in weapon systems, and understands the term 'autonomous weapon system' to apply to a broad range of technologies which apply armed force based on the computer processing of sensor inputs.[4]

Within this category, weapon systems which specifically target people, such as anti-personnel sentry robots, are particularly troubling and should be banned.  Also problematic are systems which cannot be effectively controlled by humans, either because their functioning cannot be explained, or because they are able to set their own goals, or because their operation is too fast and complex for a human to be able to control in any meaningful way.  This would include systems where target profiles are constructed through machine learning or where target profiles may change without human approval.  These systems cannot be considered to be under human control and for this reason should also be banned.  Other autonomous military systems, such as automated air defence systems, pose hazards to civilians and so there needs to be positive obligations on states, manufacturers, and military commanders to ensure that they remain under meaningful control and in compliance with the laws of armed conflict.

Automated air defence systems are already in common use, and while not completely autonomous, with a human operator ostensibly in command, have complex computer-controlled and increasingly autonomous features.  There have been a number of high-profile failures of such systems including the shooting down of Iran Air Flight 655 (1988), Malaysian Airlines MH 17 (2014), Ukrainian Airlines PS752 (2020), and two instances where friendly aircraft were attacked by Patriot air defence systems during the Second Gulf War (2003).  There are three main causes for this: (1) the speed at which these systems operate; (2) the complexity of the tasks they perform; and (3) the demands their use places on human operators. As more and more tasks have been delegated to computer systems, the human operators of air defence systems have changed from active controllers to passive supervisors.  This has made human control over specific use of force decisions increasingly meaningless, resulting in dangerous mistakes with tragic consequences.[5]

Autonomous features are already being integrated into the critical functions of widely used weapon systems as they evolve.  Although important, meaningful human control over such systems is hard to implement.  There is therefore a need for international controls on autonomy in current weapon systems as well as a ban on the most harmful types of autonomous systems.

The UK should adopt a definition of an autonomous weapon system based on the conceptualisation proposed by Article 36, but it should be noted that an international definition of an autonomous weapon system should be agreed through a process of dialogue and discussion on a treaty for controlling such systems.

Question 2

Technical risks associated with artificial intelligence, resulting from the opaque nature of its operation and unpredictability of its results, mean that autonomous weapon systems pose a high risk of undermining the laws of war and eroding human rights protections put in place to safeguard both civilians and combatants.  For the vast majority of citizens the benefits will be negligible and will be dwarfed by the risks.

The emergence of drones as a war-fighting technology over the last thirty years has had a dramatic impact on the shape of war, and has resulted in a number of threats to global security including the increased use of drones for targeted killing, even away from the battlefield.[6]  The evolution of autonomous weapon systems can be expected to pose similar risks.

Autonomous weapon systems raise profound moral and societal questions about the role and responsibility of humans in the use of force and the taking of human life.  With new advances constantly emerging, technological advances are outpacing ethical deliberation.  A fundamental ethical hazard relates to the delegation of a decision to take a human life to the calculations of a machine which is not accountable for its actions.  This reduces humans to mere objects, not even worthy of the recognition that they are living beings deserving of dignity and rights.  Autonomous weapon systems would have no moral sense on which to base decisions, no ability to deal with ambiguity, no empathy or compassion, and no capacity to imagine or take responsibility for the consequences of their actions.  This has concerning implications, too, for any society that is prepared to tolerate killing in this way.  There will be no winners in the race for military autonomy: the game is pointless and corrupting and even the 'prize' is pernicious.  Life and death decisions on the battlefield, and indeed any decision of great consequence for any human, must always as a matter of principle be made by humans.[7]

Question 3:

We believe that the development and deployment of AI-enabled autonomous weapons would give rise to a number of grave risks, primarily the loss of human values on the battlefield.  We do not consider that an autonomous weapons system could ever be 'safe, reliable, and accountable', or that it could ever be ethically acceptable to allow a machine to take a human life.  As a result it will be necessary to control autonomous weapons systems – and indeed, other computer-based systems which make decisions which have significant consequences for humans[8] – at the international and national level, and ban systems which have the greatest potential to cause harm.

Binding international legislation is needed to control autonomous weapon systems in the form of a new international treaty.  This treaty would ban autonomous systems which target humans or which cannot be effectively controlled by humans, with further controls on other types of automated weapon systems.  A key feature of this treaty would be the principle that a weapon must always be under meaningful human control – it must be operated by a human exercising oversight over the system who has an understanding of how the weapon functions and awareness of the consequences of its use in a given situation. 

Experience from recent conflict in Ukraine, Nagorno-Karabakh, and the Middle East shows we are beginning to see a new generation of weapon systems being deployed which are showing a trend towards decreasing levels of human control.  At present a human operator is able to approve an attack using these weapons, but the requirement for human approval could be removed with technical upgrades to the system.  Other military systems which do not have the 'critical function' of targeting or firing a weapon routinely operate in a fully autonomous mode without the need for human control.  This poses risks, gradually normalising the development of autonomous military systems and AI-based weapons and bringing the development of weapon systems that operate outside human control a step closer.  There is only a limited amount of time available to call a halt to this slide, and action to introduce an appropriate international legal instrument is needed now. 

The UK should play a constructive role in the negotiation of this treaty and should also uphold the Guiding Principles affirmed by the international Group of Governmental Experts on Emerging Technologies in the Area of Lethal Autonomous Weapons Systems, with the aim of establishing them as norms in the conduct of armed combat.  There is also a need to educate engineers and researchers about their ethical responsibilities in developing technologies which may give rise to human rights abuses and which has dual use concerns.  Work has  been done in this respect in the biosciences,[9] but to date there has been little professional engagement with this theme in the engineering and computing sectors.

               Question 4:

Although the UK government and governments of certain other arms-exporting nations assert that existing IHL is adequate to control the use of autonomous weapon systems, we disagree.  Autonomous weapon systems would be likely to be used in unexpected ways during warfare creating new situations and problems which are beyond the scope of existing IHL, and which could never have been envisaged when current legal instruments were adopted. 

Unfortunately, we have seen how states can erode the laws of war and the protections they provide.  International law is itself a political instrument, and governments have an interest in influencing and manipulating the rules that govern the use of armed force to suit their own purposes.  Colonel Daniel Reisner, for example, former head of the Israeli Defence Force's International Law Division told Haaretz newspaper in January 2009 that in relation to targeted killing, we are currently seeing “a revision of international law.  If you do something for long enough, the world will accept it. The whole of international law is now based on the notion that an act that is forbidden today becomes permissible if executed by enough countries … International law progresses through violations.”[10] 

Over recent years the US, Israeli, and Russian militaries, followed at times by the UK, have instituted aggressive policies intended to deliberately stretch and change international law in directions which suit their purpose.  They have attempted to create new legal space by conducting operations which violate conventional interpretations of the laws of armed conflict, and then creating legal justifications for such acts.[11]  Efforts to create international acceptance for targeted assassinations undertaken by drones are an example of such an approach.  A state which is determined to gain a military advantage by deploying autonomous weapon systems could be expected to adopt similar tactics to gain acceptance for them.

A binding international legal instrument is needed to control autonomous weapon systems and outlaw the most harmful of these.  This would help to reaffirm and reinforce existing IHL, clarify 'grey areas', create norms against the use of weapon systems without meaningful human control, and, by defining the use of weapons not under human control as a war crime, would raise the stakes for those contemplating doing so.

Question 5: 

The government's AI Defence Strategy explains to some extent how the government intends to take forward the development of military AI systems but does not explain in any convincing way why such systems are necessary or how harm resulting from the use of such systems can be eliminated.[12]  Government aspirations for the potential military use of AI are served with a heavy dose of optimism.  Despite decades of research, AI systems are still too fragile and error-prone to be relied upon in safety-critical applications such as driverless cars, let alone applications relating to the use of military force.

The Defence Secretary's foreword to the strategy summarises the government's justification for the military use of AI.  The foreword asserts that “AI-enabled systems do indeed pose a threat to our security, in the hands of our adversaries, and it is imperative that we do not cede them a vital advantage.”  This is simply another way of saying ‘if we don't do this, then others will’, therefore we must develop these systems.  Any new technology poses a potential threat to security, as does any adversary.  Such reasoning can be (and invariably is) used to justify any advanced new weapon.  No alternatives - such as steps that the UK might take to reduce international tensions, or work with other nations to control the development of harmful technologies or prevent their proliferation – appear to have been seriously considered.

The justification for the UK's use of AI for military purposes is based on a narrative of maintaining Western dominance cloaked in the guise of maintaining the UK's values because, as the Defence AI Strategy says, “we know that adversaries will use technology in ways that we would consider unethical and unsafe”.  In fact, we know no such thing. While we may ascribe all sort of motives and potential risks to adversaries,  little information about the military AI doctrine of the UK's main adversaries, Russia and China, is available and there is no reason to think that Western development of AI weapon systems will be less harmful than other nations.

Currently a majority of nations (87) support negotiation of a legally binding instrument on autonomous weapons systems based on statements made at the United Nations General Assembly and at Convention on Certain Conventional Weapons meetings.[13]  This includes 'adversaries' such as China and Iran.  Eleven nations, mainly US-oriented or arms exporting nations and Russia, do not support such an instrument and 36 nations have not yet declared their position on the matter.

The Defence AI Strategy acknowledges the “extreme and even existential risks” which AI may pose to humanity, but measures for controlling these risks are poorly defined.  The aspiration for an “ambitious delivery of capability”, intended to “enable – rather than constrain – the adoption of AI-enabled solutions and capabilities” and avoid “self-imposed limitations which would risk being arbitrary, constraining, and habitually out-dated”, is a clear indication that MoD will be focusing on developing AI capabilities first and foremost, with safeguarding taking second place.  Throughout the document, commitments to address ethical concerns and questions of trust are hedged with caveats that they must not impede AI development or collaboration on AI.  We conclude that MoD seems to think it has now 'ticked a box' on its path towards implementing AI technologies: it can now say that it has an ethical approach to AI, and use this as a fig-leaf to encourage support from industry and academia without showing real commitment to measures that would control the harmful use of AI systems.

Regulation of AI systems will be intrinsically different to anything that MoD regulates at present, and will raise a host of new issues.  The strategy states that the MoD's approach to AI risk management will be based on the ALARP (As Low As Reasonably Practical) principle that is commonly used for safety-critical and safety-involved systems.[14]  Given the novel features of AI technology, a more rigorous, precautionary approach will be necessary, particularly with high-risk AI applications which are involved in warfighting and are in the early stages of development, requiring MoD to look beyond the 'reasonably practical' safety standard.  If appropriate, this could be relaxed when greater confidence in new systems has been earned.

Scrutiny of the MoD's AI strategy documents raises significant questions about its approach to the governance of AI and digital technologies.  It also points to a deep conflict between the government’s stated democratic values rooted in human rights on the one hand, and a technocratic impulse to 'retain an advantage' by racing forward with AI at all costs on the other.

Question 6: 

Existing legal provisions and regulations are inadequate to govern autonomous weapons systems, and the use of AI in the broader civil sector.  Action will be needed to protect the public if AI is to become the technology which shapes our future.  Measures which the UK could take might include the following:

1. Introduce an Artificial Intelligence Act to regulate the technology and ban the most harmful applications, as the European Union is currently doing.[15]  There should be no exemptions for military applications – potentially the most harmful and high-risk applications of all.
2. Establish an Artificial Intelligence regulatory agency at arms-length from government.  This should have real teeth and should cover all sectors, including the military.  MoD's current internal regulators operate to lower standards than their civilian cousins, are less transparent, and are not independent.
3. Clearly state that the UK will ensure that its weapon systems will always be under meaningful human control and that the UK supports a binding international treaty to incorporate such a requirement into international humanitarian law.
4. Update export control laws to regulate and prohibit the transfer of military and dual use technologies and ban the sale of weapons to regimes with poor human rights records.
5. Establish a 'digital Hippocratic oath', to be administered by professional institutes and universities, committing computer scientists to uphold ethical principles.

None of these control measures will find favour with the government, tech sector, and research interests which are advocating for the widespread use of AI, and Parliamentarians and civil society will have to fight hard to take them forward.  The barriers to taking the necessary action are the same as those which invariably block progressive reform: obstruction by powerful states (including the UK); shortcomings in mechanisms for international governance; pressure from corporate interests; and the political reluctance to move beyond a 'business as usual' approach.

Peter Burt

Drone Wars UK

April 2023