Ousman Noor, Government Relations Manager, Stop Killer Robots Submission – Written Evidence (AIW0018)

Stop Killer Robots is a global coalition of 200+ civil society organisations across 67 countries advocating for a new international treaty on Autonomous Weapons Systems (AWS). Established in 2013, the campaign has engaged with all multilateral forums on the issue, including at the Convention on Conventional Weapons (CCW), Human Rights Council (HRC) and United Nations General Assembly (UNGA) and other international conferences.

This submission is prepared for the United Kingdom’s Artificial Intelligence in Weapon Systems Committee’s (the ‘Committee’) Call for Evidence on autonomous weapons systems. A response is provided to each of the six questions set out by the Committee.

1.     What do you understand by the term autonomous weapons system (AWS)? Should the UK adopt an operative definition of AWS?

Autonomous weapons systems are systems that select and apply force to targets based on the processing of data from sensor inputs. In these systems, upon activation, there is a period of time where the weapon system can apply force to a target without additional human approval. The specific object to be attacked, and the exact time and place of the attack, are determined by sensor processing, instead of the human operator. This means the operator does not determine specifically where, when or against what force is applied. AWS therefore change the relationship between people and technology, by disconnecting and dislocating humans in the use of force.

The characterisation of AWS relates to the process by which force is applied, rather than to a specific type of weapon system. The processing of sensor data to select and engage targets with force can be incorporated into a broad range of weapons systems, and the use of AI is not determinative of whether a system is an AWS. Within the broad scope of weapons systems that would be considered AWS, we recommend that certain systems should be prohibited, and others subject to regulations.

Stop Killer Robots recommends that the UK adopt this characterisation of AWS, focussing on the process by which force is applied, rather than adopting a definition that focuses on a specific type of weapon or technology.

2.     What are the possible challenges, risks, benefits and ethical concerns of AWS? How would AWS change the makeup of defence forces and the nature of combat?

Autonomous weapons systems pose a range of ethical, security, environmental and legal risks including issues of compliance with international human rights law (IHRL) and the international humanitarian law principles (IHL) of distinction, proportionality, precaution and the prohibition of indiscriminate attacks.

The challenges presented are serious and have potential to erode our commitment to human dignity, to erode respect for existing legal rules and fundamental rights, to further weaken accountability for harms in conflict and to negatively shape our relationship with automated decision making across all areas of society. These issues have implications beyond the immediate context of controlling behavior in conflict and demand a strong international normative and operational framework that is legally binding.

Legal Risks

By diminishing the control of the human operator, AWS undermine accountability and

responsibility in conflict and raise serious concerns over compliance with the international

humanitarian law (IHL) principles of distinction, proportionality, precaution and the prohibition of indiscriminate attacks. These rules require a process of determining the specific target, location, timing, duration and extent of the force applied, in order to ensure that the use of force is lawful. Through relying on the processing of sensor data to select and engage a target with force, an AWS would undermine these processes required to ensure compliance with IHL.

The use of AWS beyond armed conflict, for example in border control and policing, would

undermine International Human Rights Law (IHRL) including the right to freedom of assembly, the right to privacy and the right to life and dignity. Apart from the requirements for relevant court proceedings with due process, the use of force requires the principles of necessity and proportionality to be met. AWS would automate the application of force based on the processing of sensor data, absent the required legal processes or judgements to be fulfilled.

The use of AWS means that the human operator does not have direct control over when, where and against who, force is to be applied. This means that responsibility and liability for the effects of the weapon cannot be straightforwardly assigned to the human operator. In the context of armed conflict, where the fog of war already makes it difficult to hold individuals to account, AWS would further undermine accountability upon perpetrators of violence and would make providing retributive justice to victims even more difficult. [1]

Ethical Risks

AWS rely on the processing of data from sensors to apply force. This process can lead to systematic prejudice due to racial, gender, ability and other algorithmic bias that permeates artificial intelligence systems.[2] Evidence from the use of artificial intelligence in the processing of data in policing, criminal sentencing, access to education, healthcare, housing and employment demonstrates that racialized and historically marginalized populations are disproportionately harmed by the automation of decision making processes.[3] AWS can be developed to target and use force against people of a particular demographic and augment existing racial, gender and ability inequalities, or inadvertently do so as the unintended consequence of algorithmic bias.[4]

Allowing machines to take human life would dehumanize people, harming them based on processing of sensor inputs and treating humans as data, sensed and sorted by a machine. By killing or wounding people in this way, these systems would undermine human dignity, and is the ultimate form of digital dehumanization. 

Delegating the facility to take a human life to machines should be recognized as a

challenge to the principles of humanity because machines cannot respect human dignity.[5] The profound ethical concerns expressed by faith leaders, scientists, States, private sector tech companies, military veterans, the public and the UNSG prove that these emerging weapons systems are problematic under the dictates of public conscience.[6] In global surveys, the most cited reason for opposition to autonomous weapons systems is that they would “cross a moral line as machines should not be allowed to kill” (Ipsos, February 2021). UNESCO has also recommended that “life and death decisions should not be ceded to AI systems”.15

Security Risks

Through investing in the development and use of AWS, absent clear legal boundaries, there is a danger that technology will become available for unauthorized use and proliferate globally.[7] Rapid and unchecked investments in AWS would risk generating an arms race between competing military powers and fuel global insecurity.[8] Apart from use by militaries in conflict, AWS might also be used for criminality and undermining social and national security.[9] The establishment of clear legal limits to the development and use of AWS now, is required to prevent the deterioration of global insecurity in the future.

Environmental Risks

Weapon systems have a damaging impact on the environment and biodiversity. The use of landmines, cluster munitions and explosive, chemical, biological and nuclear weapons have all had hazardous consequences for the natural world. Through automating the use of violence, AWS would undermine human supervision over the consequences of using weapons to the wider environment. Militaries are major contributors to carbon emissions yet are often excluded from emissions inventories and global emissions targets[10] and while the potential energy cost of training and operating autonomous weapons systems is yet to be known, studies on the carbon footprint of information and communications technology (ICT) and artificial intelligence (AI) in general indicate that AI is expected to substantially contribute to climate change.[11] AWS have the potential to undermine efforts to combat climate change and protect biodiversity by eroding human responsibility for the consequences of using weapon systems to the degradation of the planet.

3.     What safeguards (technological, legal, procedural or otherwise) would be needed to ensure safe, reliable and accountable AWS?

An international legally binding instrument with a combination of both prohibitions and regulations is required to safeguard against the wide range of risks posed by autonomous weapons systems.

Prohibitions are required on AWS that either (I) target people or (II) cannot be used with Meaningful Human Control (MHC)

I. Autonomous weapons systems that target people: Sensor-based weapons systems that apply force due to the presence or proximity of a person would use proxy indicators (such as weight, heat-signal shape, ‘object recognition,’ movement or biometrics) as a basis for encoding patterns of sensor data (target profiles) intended to represent humans.

We consider these systems unacceptable because they reduce people to objects, and so are dehumanizing to civilian and military victims alike. Such systems would pose fundamental challenges to IHL, including the principles of distinction and the prohibition of indiscriminate attacks and the protection of the civil population against the effects of hostilities. Claiming that technology can distinguish between people that can or cannot be legally targeted abandons the case-sensitive human judgment upon which the law is based and would open the door to biased data sets and prejudicial algorithms. Claiming systems will only be used in areas where there are no civilians, pushes the burden of avoiding harm onto civilian populations and weakens the normative rejection available to us if we act now. Furthermore, assumptions that AWS could accurately detect civilians at all are unfounded.[12] Such systems should be rejected on legal and ethical grounds and as a matter of societal precaution.

II. Autonomous weapons systems that cannot be used with meaningful human control: Certain autonomous weapons systems will necessarily be incapable of MHC by a human commander and should be subject to prohibition. Examples would include systems where the location and duration of their functioning cannot be appropriately limited, and systems where the external conditions and circumstances that will trigger a specific application of force cannot be appropriately predicted or understood.[13]

MHC should be understood to capture the degree of control required to mitigate operational and ethical hazards and to ensure that legal rules and ethical principles can be substantively applied if systems are used. MHC should ensure the user can effectively direct and predict the effects of an attack, as well as ensure those effects are in line with the user’s intent. In this way the user can fulfill their obligations under IHRL and IHL, as well as ensure moral responsibility and accountability. MHC will be context specific, and obligations to ensure that control need to be formulated in broad terms such that they can apply to diverse systems and cater to technological developments in the future. [14]

Autonomous weapons systems that are not prohibited should be subject to regulations and positive obligations to ensure MHC. There are systems that use sensors to select and engage targets that we do consider unacceptable per se, but which would pose grave problems if they were to be used without meaningful human control.

Obligations will need to highlight key components that, taken together, are necessary to ensure meaningful human control. They include:
 

I. Decision-making components: The human operator should have an understanding of how the system functions, an understanding of the operational environment, and sufficient time for deliberation.

II. Technological components: The weapon system’s embedded features should enable the operator to enhance meaningful human control, including through ensuring predictability and reliability. Mechanisms for ‘human-machine interaction’, including where necessary the ability for the operator to intervene, should work to avoid unintended harm.

III. Operational components: The degree of mobility and duration of operation that an autonomous weapons system is capable of will be important factors in determining whether a human operator is able to exercise meaningful human control. The location and duration of a system’s operation need to be appropriately limited for an operator to control and predict the effects of the system and to apply existing legal rules. Such limits will also depend on the type of environmental context in which the weapon is operating (e.g. populated areas, at sea, etc) - with some systems likely being inappropriate in certain contexts.

4.     Is existing International Humanitarian Law (IHL) sufficient to ­­ensure any AWS act safely and appropriately? What oversight or accountability measures are necessary to ensure compliance with IHL? If IHL is insufficient, what other mechanisms should be introduced to regulate AWS?

No, existing legal rules are not sufficient to address the serious risks and challenges posed by autonomous weapons systems. An international legally binding instrument on AWS is required with a combination of both prohibitions and regulations, as set out above.  A legally binding instrument would provide a durable framework offering the benefit of legal certainty and stability for the development and use of AWS now and in the future.

In addition, new legally binding rules would strengthen existing requirements for weapon reviews under Article 36 of Additional Protocol I of the Geneva Conventions. These reviews provide an obligation to review weapons, means and methods of warfare against States’ legal obligations. The establishment of international legal obligations relating to the development and use of AWS would promote standardization of States’ review processes around the world and help ensure that they are conducted transparently and consistently.

A non-binding instrument would risk promoting the widespread development and use of AWS, absent clear legal and ethical boundaries of acceptability. Without universalization of clear legal standards, there would be significant discrepancies in how States apply any non-binding practices, resulting in AWS being considered acceptable in one State, but unacceptable in another.

5.     What are your views on the Government's AI Defence Strategy[1] and the policy statement ‘Ambitious, safe, responsible: our approach to the delivery of AI-enabled capability in Defence’[2]? Are these sufficient in guiding the development and application of AWS? How does UK policy compare to that of other countries?

The UK Government’s AI Defence Strategy and the policy ‘Ambitious, safe, responsible: our approach to the delivery of AI-enabled capability in Defence’, provide starting points for the development of clear legal rules on the development and use of autonomous weapons systems.

At the core of the policy is a recognition that the use of systems that cannot comply with the IHL principles of distinction, necessity, humanity and proportionality would constitute a breach of international law. The policy also recognises that in order to comply with IHL, some form of context appropriate human involvement, supervision, or control that is needed over weapons systems that autonomously identify, select and engage targets. This provides a firm foundation from which it is possible to establish a combination of both prohibitions and regulations to ensure that systems can be used in compliance with IHL.

However, while the policy recognises that human involvement, supervision, or control is needed, there is absence of clarity over what accountability mechanisms "context appropriate" will be evaluated or delivered, what systems must be explicitly prohibited, and what specific regulations, including positive obligations and limits, are needed to comply with IHL. Further, the policy fails to fully engage with IHRL, ethical, humanitarian and security concerns, as set out above. In order to properly safeguard against these risks, clear prohibitions are required on systems that cannot be used with meaningful human control, and on systems that target humans, and regulations to ensure all other systems are used with MHC. 

In addition, the UK policy suggests that best practice and codes of conduct are sufficient to deal with the risks posed by AWS. Again, for the reasons set out above, clear international legally binding rules are urgently needed, and non-binding principles are a wholly insufficient response.

6.     Are existing legal provisions and regulations which seek to regulate AI and weapons systems sufficient to govern the use of AWS? If not, what reforms are needed nationally and internationally; and what are the barriers to making those reforms?

Existing IHL is insufficient to deal with the risks posed by AWS. Stop Killer Robots advocates for the establishment of a new international legally binding treaty with clear rules relating to the development and use of AWS.

A total of 90 States support an international legally binding instrument on AWS.[15] The number of States confirming support has continued to increase each year, demonstrating clear political momentum recognising the urgent need to launch negotiations. The call is also supported by the International Committee of the Red Cross (ICRC),[16] experts in technology and artificial intelligence,[17] faith leaders,[18] military veterans,[19] and civil society organizations around the world.

Apart from discussions within the United Nations (UN), States are taking initiative to agree upon the urgent need to launch negotiations for a legally binding instrument within international conferences. These include a Communique agreed by Latin American and Caribbean States at Belen, Costa Rica in February 2023 and a Special Statement made by 22 Heads of State at the Ibero-American Summit adopted at the Declaration of the Dominican Republic in March 2023.

While international momentum towards negotiating a new treaty gathers pace, a primary barrier remains the reluctance of certain states, including the UK, that have so far opposed the establishment of an international legal instrument on AWS. However, with additional international conferences planned throughout the year, increasing numbers of States are converging on the need to launch negotiations on an urgent basis. Stop Killer Robots recommends that the UK join others in supporting the establishment of a new treaty that will reshape humanity’s relationship to technology, achieve new standards in the use of force and help promote a more peaceful world for our generations and those to come.

Conclusion

Stop Killer Robots welcomes the initiative taken by the Committee and appreciates the multi stakeholder invitation to submit evidence relating to the development and use of AWS. As set out in response to these questions, Stop Killer Robots recommend that the UK take urgent action to launch negotiations on a legally binding treaty on AWS to both prohibit and regulate the development and use of AWS.

With ongoing instabilities in international security and mounting evidence of autonomous weapons systems being used in contemporary conflicts, political leadership is required to avoid the devastating humanitarian consequences of allowing the development and use of autonomous weapons systems to proliferate.

Further to this submission, Stop Killer Robots remains available to provide any additional clarification on any aspect of the recommendations set out herein. For any enquiries, please contact Ousman Noor, Government Relations Manager, Stop Killer Robots on ousman@stopkillerrobots.org. Alternatively, the UK’s national campaign can be contacted on robotsuk@una.org.uk.

Thank you

Ousman Noor

Stop Killer Robots

April 2023