Professor Toby Walsh – Written Evidence (AIW0026)

I am grateful for the opportunity to prepare this submission to the Inquiry into the use of AI in weapon systems.

For background, I am Chief Scientist of UNSW’s new AI Institute. I have spent the last forty years as a researcher into Artificial Intelligence. As AI has started to leave the laboratory, my research has increasingly considered the benefits, risks, and limitations of AI. Most relevant to your inquiry, I have been an active contributor to the debate around lethal autonomous weapon systems. For example, I organised debates on the use of AI in weapon systems at the two leading AI conferences, AAAI and IJCAI in 2013 and 2014. In 2015, I helped release an open letter  to the UN on lethal autonomous weapons that attracted the signature of thousands of AI researchers, as well as organised the 2017 open letter  to the UN from founders of AI and Robotics companies, and the 2018 boycott  of KAIST due to their involvement in the development of such weapons. I have spoken half a dozen times at meetings in the UN on this topic, as well as to heads of state, to parliamentary bodies and numerous members of parliaments, as well as frequently in the media and many public fora. I was an expert advisor to the drafting of the MOD’s Defence AI Strategy in 2022. In May 2022, I was banned indefinitely from Russia for my public criticism of the use of AI by the Russian military.

Professor Toby Walsh FAA FACM FAAAS

Question 1: What do you understand by the term autonomous weapons system (AWS)? Should the UK adopt an operative definition of AWS?

1.0. Autonomous weapons are those that identify, track and target without human oversight. Autonomy is a capability that can be added to any weapon system.

1.1. I strongly believe that worrying about the definition of AWS is a distraction from making progress on regulating this space.

1.2. Let me give an analogy. Nuclear weapons are never defined in any UN treaty regarding the proliferation or prohibition of nuclear weapons. This is a good thing since these treaties will cover nuclear weapon systems still to be invented. I suspect it is likely too difficult to define autonomy precisely, or to define such terms in diplomatic discussion today like “meaningful human control”, especially if we wish these definitions to be future proof.

1.3. As with nuclear weapons, an implicit and informal definition of autonomous weapon systems will arise out of international consensus. This will likely permit the current generation of weapon systems like the RAF’s Brimstone fire-and-forget missile, but it will draw the line at more sophisticated types of autonomy, especially where the weapon system has autonomy over timescales of minutes, hours or days. We will be able to put most weapon systems clearly on one side or other of this line. This will be enough for any regulation to be effective.

Question 2: What are the possible challenges, risks, benefits and ethical concerns of AWS? How would AWS change the makeup of defence forces and the nature of combat?

2.0. AWS will redefine how we fight war. Large, expensive, manned weapon platforms will be replaced by small, cheap and increasingly capable unmanned systems.

2.1. Without controls in place, such weapons will become widely available to a wide range of state and non-state actors. Indeed, even before such weapon systems are sufficiently accurate to be deployed by responsible parties like the MOD, terrorist organisations and rogue states are likely to use them. A weapon that kills 50% of the wrong people is arguably even more terrorful than one that only kills the intended people.

2.2. Most of the benefits of AWS can be had without giving up meaningful human control. For instance, consider the ability of autonomous systems to reduce collateral damage by targeting more accurately than humans can.  Full autonomy is not required to achieve this. It is also achieved by a unmanned weapon system which is fired by a human operator and whose only autonomy is to disengage when the target is identified not to be a legitimate military target. We can achieve a reduction in collateral damage without giving up meaningful human control of the initial target selection.

2.3. Autonomous weapon systems increase the risk of flash conflicts. We know what happens when we put complex computer systems against each other in an uncertain and competitive environment. It’s called the stock market. And the only way to stop dangerous feedback loops and undesirable outcomes is to use “circuit breakers”. On the stock market, we can simply unwind transactions when such a situation occurs. But we cannot unwind the start of WW3 when the autonomous weapon systems facing each other in the DMZ have a “flash crash”.

2.4. Autonomous weapon systems decrease accountability and increase global insecurity. It will be much harder to determine who has attacked you. We already have drone attacks on Russia’s Hmeimim airbase where it is not certain who was behind the attack.

2.5. Autonomous weapon systems will be used in swarms that overwhelm conventional forces. Autonomous weapons can fight 24/7. They will never tire, and never rest. They will work at speeds and accuracies that are super-human. Defence will only be possible with other autonomous systems.

2.6. As UN Secretary-General António Guterres has stated, autonomous weapons are politically unacceptable and morally repugnant. They clearly violate the Martens clause. Repeated surveys in multiple countries including the UK have demonstrated that autonomous weapons do not comply with the dictates of the public conscience.

Question 3: What safeguards (technological, legal, procedural or otherwise) would be needed to ensure safe, reliable and accountable AWS?

3.0. AI software systems are by far the most complex systems built by humankind. GPT3 has, for example, over 175 billion parameters. The overall complexity of machine learning models dwarfs even the largest computer software coded by hand.

3.1. We have little idea of how such complex systems work, their capabilities and their limitations. Our recent collective experience with ChatGPT should be a valuable caution. The idea that we can safely and reliably give such complex systems responsibility to decide on who lives or who dies is absurd.

3.1. The fundamental legal and technical challenge of autonomy is accountability. Only humans can ultimately be held account. Machines cannot. Therefore (and unsurprisingly) the debate around AWS has rightly focused on maintaining meaningful human control. This is only way to ensure accountability legally and technically.

Question 4: Is existing International Humanitarian Law (IHL) sufficient to ¬¬ensure any AWS act safely and appropriately? What oversight or accountability measures are necessary to ensure compliance with IHL? If IHL is insufficient, what other mechanisms should be introduced to regulate AWS?

4.0. No, IHL is inadequate to ensure any AWS acts safely and appropriately. Some of the risks identified in Q3 are not those which concern IHL. Issues such as proliferation, global insecurity, accountability and terrorism go beyond the narrower humanitarian concerns of IHL

4.1. Of course, some of the risks around AWS are covered by IHL. For instance, principles such as distinction and proportionality address some of the potential harms of (poorly implemented) autonomous weapon systems.

4.2. I reject, however, the implicit premise of the question that we do not need to consider issues already covered by IHL. The international community has frequently introduced new regulations on top of IHL to ensure particular weapon technologies are used appropriately.

4.3. As an example, chemical weapons clearly violate international humanitarian law, in particular the 1925 Geneva Protocol. However, the 1993 Chemical Weapons Convention came into force to regulate them more strongly. The Convention was signed and ratified by the United Kingdom.

4.4. In a similar way, we need new regulation around autonomous weapons systems, both to clarify how such weapons violate IHL, and also to address other concerns such as accountability, proliferation and global security.

Question 5: What are your views on the Government's AI Defence Strategy and the policy statement ‘Ambitious, safe, responsible: our approach to the delivery of AI-enabled capability in Defence’? Are these sufficient in guiding the development and application of AWS? How does UK policy compare to that of other countries?

5.0. The UK’s strategy and policy statement are a good start. AI will help the MOD deliver greater security to the UK and to its partners. From smarter logistics to minefield clearing robots, there are many uncontroversial ways in which AI can improve the UK’s defence.

5.1. However, it is not the MOD’s use of autonomy that is my primary concern. I am confident that the UK’s military will, as with previous technologies, deploy AI responsibly irrespective of the international rules and norms.

5.1. I am very concerned, however, that other state and non-state actors will not act as responsibly as the UK. It is for this reason that the UK needs to take a more active position in the international negotiations.

5.2. The UK is one of the world’s major arms exporters. It therefore has a moral responsibility to define the international norms in this space (not just those for use by the MOD). It also has an important role to play in these negotiations to improve global security.

Question 6: Are existing legal provisions and regulations which seek to regulate AI and weapons systems sufficient to govern the use of AWS? If not, what reforms are needed nationally and internationally; and what are the barriers to making those reforms?

6.0. No, existing legal provisions and regulations are inadequate. As the UN Secretary-General António Guterres has stated, autonomous weapons systems should be prohibited by new international law.

6.1. Various states are resisting the call by dozens of nations to begin the process of drawing up regulation in this space. These nations are seeking to gain a technical and military lead on other countries. Letting this happen will further decrease global security.

6.2. States like Russia are exploiting the consensus based mechanisms of fora like the UN’s CCW to stall progress. The UK needs to use its diplomatic weight as a member of the Security Council, and as a major nuclear power, to push back strongly against this.

6.3. Progress might be best made by the UK pushing for regulation in domains where consensus could be easier to obtain (such as with autonomous anti-personnel weapons, and autonomous nuclear weapons).

Professor Toby Walsh

April 2023