Dr Elke Schwarz – Written Evidence (AIW0009)

Dr Elke Schwarz is Reader in Political Theory at Queen Mary University London (QMUL). She has worked in the space of emerging military technologies and their ethical implications for over a decade and has published on the specific issue of military AI and AWS for the past five years. She is also a member of the International Committee for Robot Arms Control (ICRAC). A selection of published work that may be of interest to the Select Committee’s tasks is included in the footnotes.

In my evidence, I will respond to questions 1, 2 and 5 only as these are within my specific realm of expertise. It is the bigger picture view on these risks and challenges to which I shall direct the focus of my evidence submission.

What do you understand by the term autonomous weapons system (AWS)? Should the UK adopt an operative definition of AWS?

1. Autonomous weapon systems are systems which feature in their functionality some level of autonomous processing. This means that the system can processes data and ‘act’ based on this data without human interference. This is a minimum function of autonomous systems, which can then range from autonomous navigation to autonomous lethal action. In less advanced systems, some form of algorithmic processing enables autonomous action, this could be quite narrow IF/THEN reasoning. In more advanced AI-enabled systems, machine learning gives the systems the capacity to learn from environmental input and adjust actions autonomously in a more dynamic feedback loop. So there is a fair spectrum of autonomy in function and there is a fair spectrum of autonomy in use.

2. The debate on AWS typically centres around the most divisive aspect of AWS – namely the system’s potential capability to make lethal decisions (against humans) autonomously. Here the ICRC definition is, in my view, a good and useful working definition: which describes (L)AWS as “any weapon system with autonomy in its critical functions – that is, a weapon system that can select (search for, identify, track or select) and attack (use force against, neutralize, damage or destroy) targets without human intervention.”

3. The UK had in the past adopted a far too broad and elastic definition of autonomy – this seems unhelpful in understanding the specific challenges of lethal autonomous systems. The more precise and differentiated a definition, the better so that regulation can be enacted appropriately. This is, naturally, a political challenge, but one where international consensus should be sought in the interest of mitigating the not insubstantial risks of (L)AWS. At present, there is too much inaccurary and confusion over what AWS are, what they can do and what they cannot do. It is important to have precision in language.

What are the possible challenges, risks, benefits and ethical concerns of AWS?

4. The possible challenges, risks, benefits and ethical concerns are manifold, and a good decade’s worth of debate on the topic of lethal autonomous weapons systems provides a comprehensive basis from which to depart. Much of the debate on AI-enabled weapons systems is dealing in hypotheticals, which makes it difficult to consider concrete data on the efficacy and challenges of AI-enabled autonomous weapons systems. Moreover, the AI arms race is underwritten by an almost irrational believe in Artificial Intelligence as the inevitable future for all things, I would strongly caution against this attitude in the development and use of AI-enabled autonomous weapon systems.

5. One of the core challenges, then, is not to succumb to an undue sense of urgency in developing and employing AI-enabled autonomous weapon systems. A cautious approach which takes seriously that a responsible approach to AI is knowing where and when NOT to use it, is warranted.

6. The faith in AI as a panacea overlooks the fact that AI has not yet proven to be particularly robust or consistently infallible, and that there is – to date – little evidence that these significant limitations can be overcome in some near or distant future. Increased computational processing speeds and the theoretical availability of vast amounts of data facilitate the illusion that the world can be rendered in all its facets as data, quickly, so that it can be acted on. It creates the illusion, also, that war is an engineering problem, or a predominantly technological problem and increasingly brackets the social and political aspects of war. This can lead to a dangerous tunnel vision in which AI seems a good solution but re-directs attention away from other modes of addressing conflict more productively and perhaps ethically.

7. It is worth briefly highlighting the well-known limitations of AI enabled autonomous type technologies, based on what we know about how AI / ML works for real world environment. The bottom line is, AI is not very intelligent when it comes to complex, open environments. In fact, most technology experts would argue that AI-enabled autonomous systems are nowhere near being able to operate reliably in highly dynamic and uncertain contexts, let alone be able to distinguish between friend or foe in the messy reality of the battlefield.

8. Systems that employ AI for the full kill-chain are likely to be marred by incomplete, low-quality, incorrect or discrepant data. This, in turn, will lead to highly brittle systems and biased, harmful outcomes that will likely yield counterproductive outcomes. Autonomous systems tend to be build and tested on rather limited samples of data, sometimes synthetic data, sometimes inappropriate data – it is simply not possible to model the complexities of a battlefield accurately. It is impossible to appropriately train on the data needed for such a system to work accurately.

9. Add to this that research in machine learning for AI is by far not as scientifically robust as it might appear from media accounts – we know this, for example, from the healthcare industry. AI is good only for quite clearly delineated, narrow tasks to be executed in closed systemic environments. Applications elsewhere are highly likely to lead to faulty decisions, to misapplications of force and other harmful errors.

10. There are other factors, such as the inherent unpredictability of system outcomes and the probabilistic nature of AI reasoning which has a logic of error and accident implicit as a feature, not a bug, which pose significant challenges to any hybrid command and control structure in which a human should retain oversight over actions.

11. There are two key challenges implicit with prioritising military AI in an attempt to not fall behind in the AI arms race. Given what we know about the efficacy of AI this seems a dangerous prioritisation for two reasons:

1. Immature or faulty technologies: Where states are racing to produce an emerging technology first, the chances are there is not enough or not rigorous enough oversight in development and testing, of the system before use. “Move fast and break things”, the Silicon Valley ethos, is inappropriate for potentially lethal systems. There is also the crucial issue of how to test these systems – in order to get them battle tested one needs a battle to test the system on. But the logic of AI is such that it relies on frequent updates and iterations in order to functioning well across changing environments, contexts and time. How might one, then tackle the verification and validation process needed for the safe use of AI weapon systems if such a validation and testing would be a frequent, perhaps daily, requirement to stay up to date and relevant? Can processes be put in place that would warrant that any operator or commander would have sufficient knowledge and understanding of the systems to retain viable responsibility over the use of the system in all instances?

2. Proliferation and escalation: As we can see from over a decade of drone warfare, emerging systems extensively rely on components that have dual use, meaning they are produced and used for civilian contexts as much as military operations, which makes them a lot tougher to regulate and a lot easier to acquire without much oversight or control. As the technology and its systemic components becomes cheaper and more easily to acquire. Again, drones are a case in point – proliferation has soared by state and non-state actors.

These systems are attractive for their cost efficiency and their ability to transfer risks away from operators and troops to targeted populations. This might significantly alter the willingness to engage lethal force (lower the threshold to use violence). Importantly, this potential for system fog on account of uncertain machine outcomes holds an enormous potential for escalation at speed. By bracketing the human increasingly from this accelerated action chain, the check points, or nodes, for de-escalation shrink considerably.

12. The second core challenge is moral responsibility. It is clear to any observer and scholar of warfare that ethics in and for war remains crucial across global contexts of conflict. This must be safeguarded to the greatest possible extent, not sidelined. This will erode restraint in the use of force and in turn weaken global peace and security.

13. Crucially, when the human is bracketed in favour of machine logics, we lose an important ethical dimension in ways that cannot be substituted by AI-logics. We risk that the human is morally removed from the action and by extension, we lose moral competencies. As tenuous as morality in warfare is, diminished moral competencies for lethal actions are never a good prospect and the question of who – or what – can be held responsible for errors, mistakes, accidents, or atrocities committed with the use of AI-enabled targeting systems is extremely fuzzy. Here we speak of a so-called responsibility gap.[1] A disregard of moral concerns and human moral competencies a pathway to post-conflict peace.

14. A crunch point in the debates on lethal autonomous weapons systems, and indeed AI enabled systems where more and more scope for decision-making is done by the machine is the question of control: to what extent can the operator, and indeed the commander exert meaningful moral control over an AI system. I have written about the challenges that arise to any sense of moral control with AWS, but just three I want to highlight here:

1. Cognition: We have cognitive limitations with ultimately make us accept the machine decision much more uncritically. In fact we have severe limitations in overriding a machine decision in time-critical moments. This is well documented as automation bias.

2. Epistemic decision-basis: Knowledge about a situation in which one should intervene or use human judgment deteriorates. Situational awareness becomes routed through the system, which means a reduced capacity to develop an appropriate mental model necessary to overcome possible system failure or error and make a morally relevant. Those operating or working with the system tend to not have sufficient information on the backend – meaning: how was the system trained, on what data, what biases might be implicit and so on in order to make an informed and conscientious decision of ethical concern.

3. Temporal horizons: Finally, this is the most obvious: where accelerate action is paramount, the action time horizon shrinks. Given that accelerated action is the desired outcome for the use of AI-enabled weapon systems, there is an inherent tension here.

15. All three eradicate the human ability to make appropriate moral decisions and take responsibility for actions conducted with AWS. [2] And I very strongly push back against any notion that ethics could be programmed into a system. This is a red herring – ethics as such is not programmable, neither can the process of moral decision making be outsourced to an AI-system. It is a contradiction in terms.[3] Ethics is dynamic, it is an ongoing process in which the human ability to take responsibility for a morally significant action is crucial.

How would AWS change the makeup of defence forces and the nature of combat?

16. With AWS, one cannot readily assume that AI can be used in a tool-like capacity to simply support commanders and operators. Rather, the use of digital technology shapes and changes human practices, outlooks, aims and so on.[4] It directs actions toward the logic of AI.[5] AI warfare is not human warfare – it prioritises the technical elements of war and brackets the human dimension of war. But without the latter, wars can neither be won, nor can they end, nor can they pave the way for a peaceful post-war context. We can expect AI warfare to prioritise speed and efficiency (and thus lethality), but that can serve as a strategy for conflict only to an extent. As the Undersecretary for Multilateral Affairs and International Economic Relations for the Philippines, Carlos J. Sorreta, remarked at the recent REAIM Summit: “AI in the military domain is ultimately about speed in waging war. Speed might be good for waging war, but perhaps not so much for peace. Delays in armed conflict are critical breathing spaces for diplomacy to work, for peace to be given a chance”. This is on point. In fact, a crucial question to ask is this: what kinds of war will AWS produce? Can accelerated wars conducted with AWS be won by anyone?

17. A further political point should be taken into account here too: Those driving the speed of military AI development tend to sit in the driver seat of the directions for the use of the technology as well. Here I would urge caution too not to hitch one’s wagon too comprehensively to any one global actor. Specifically, it stands to reason, that power differentials as to how the technology is rolled out, used and / or regulated will emerge between those that have, for whichever reason, an advantage in developing AI. Here, I want to point toward the US/UK collaboration on Project Convergence, for example. Allied cooperation is, of course, desirable, but there should be limits as to the interoperability between two military organisations’ technologies. With AI-enabled systems, the drawing of these delineations becomes somewhat of a challenge and may de-stabilise equal relations between allied nations.

What are your views on the Government's AI Defence Strategy and the policy statement ‘Ambitious, safe, responsible: our approach to the delivery of AI-enabled capability in Defence’? Are these sufficient in guiding the development and application of AWS? How does UK policy compare to that of other countries?

18. It is laudable that the UK has a military AI strategy and it is clear that a lot of conscientious thought has gone into producing the strategy. I am encouraged in particular by the list of key challenges to Defence AI Adoption and the fact that the document seeks to take the human / people dimension into account. The Ethical Principles for AI in Defence are a good start to think some of the more thorny issues of military AI ethics through.

19. However, it is crucial that the existence of the ethical principles is not mistaken for the realisation of ethical military AI. I was fortunate to be able to join a workshop on the launch of the ethical principles at the Royal Airforce in 2022 and the realities of military operations, and the people involved, have let crystallise that the five principles are really rather core challenges that in many cases will be unlikely to be satisfied, ethically, than principles. Ethical principles are not recipes, they are, again, dynamic. They require realisation in plural, morally challenging context and for that a dynamic, human, approach is required.

20. We should accept, that there is a perhaps irresolvable tension in the very idea of human-centric military AI ethics. There are clear limits of human control over AI, and there are clear limits to moral decision making with AI.[6]

21. We should also understand, that at this stage AWS that are employed in highly dynamic battlegrounds, or that are employed to target humans, cannot be responsibly employed, unless one chooses to define responsibility in a very narrow and technical sense and not in the moral sense. However, to do so would go against the principles outlined in the document. The aforementioned responsibility gap issue is a case in point.

22. We should acknowledge that with the fast-paced development of machine learning techniques, AI system decision-making might be unintelligible to the human, thus evading the principle of understanding. The issue of frequent updates, iterations and reassessment may make understanding the system in anything other than a very  narrow context with easily controllable variables too challenging.

23. It is, finally, also crucial to accept that the issue of bias is currently extremely complex and there are no clear pathways to eliminate bias from AI systems. Alignment is a problem that is far from being solved, and any AI system trained on a specific set of data, used according to specific parameters is likely to have a bias or produce unexpected by biased outcomes.

24. In short, the principles need to remain a guide for restricting the roll-out of military AI especially in weapon systems and should be frequently revisited for appropriateness.

Dr Elke Schwarz

April 2023