Written evidence from the UK Campaign to Stop Killer Robots (TFP0028)



1. This submission is made on behalf of the UK Campaign to Stop Killer Robots (UK CSKR). We are a network of UK-based NGOs, tech experts and academics who are concerned with the risks associated with growing autonomy in weapons systems. The UK Steering Committee includes Amnesty International UK, Article 36, Drone Wars UK, United Nations Association UK and the Women’s International League for Peace and Freedom UK. Each member of the UK CSKR may not necessarily endorse or take a position on all points made in this submission.  Our submission focusses on questions; 1,2,4 and 5 according to our field of interest and expertise.

As a coalition of UK-based NGOs we are calling for a legally binding instrument to pre-emptively ban Lethal Autonomous Weapons Systems (LAWS) by mandating meaningful human control of critical functions in weapons systems; specifically weapons that surveil, track and target humans and infrastructure. Lack of human control raises fundamental ethical, moral, technological, legal and security concerns which we hope the committee will consider as part of its inquiry.



Question 1.    What technologies are shifting power? What is the FCDO’s understanding of new technologies and their effect on the UK’s influence?


      The submission focuses on the technologies underpinning the development of LAWS, ie Artificial Intelligence (AI), which use machine learning/deep learning (ML and  DL), and autonomous robotic technologies.  Lethal autonomous weapon systems are weapons which combine sensor, AI, and robotic technology and can independently search for and attack targets without human intervention - “killer robots”. 


      Weapon systems with high degrees of autonomy already exist, including air defence systems such as the Patriot and Aegis missile systems, sentry robots, loitering munitions and 'fire and forget' missiles such as the MBDA Brimstone.  Although offensive LAWS have not yet, as far as is known, entered service, the building-block technologies which underpin their use are easy for industrialised nations to acquire and have reached sufficient maturity for their construction to be feasible.  LAWS raise grave concerns about maintaining human dignity and ensuring compliance with humanitarian law during conflict.


      Furthermore, significant technological asymmetry in warfare may lead to disadvantaged states attempting to compensate, for example by proliferating existing weapons, cyber or financial warfare with devastating effects on national infrastructure.


      The FCDO has an important role to play in advocating the government's position on LAWS and new technologies internationally, but other government departments  have similar duties.  The Ministry of Defence (MoD) also has a key role to play in controlling the development of, and formulating policy on, emerging military technologies.  It is essential that MoD engages in public debate over policy on new military technologies, including AI and autonomous systems, and in particular articulates a robust set of ethical principles to govern their use and development.  The US Department of Defence has shown considerable interest in the development of autonomous weaponry but, following a study by the Defence Innovation Board, has published a set of ethical principles to determine how the US military will use AI[1]Although the UK MoD has shown similar interest, setting up a number of programmes to develop and test military autonomous technologies and AI which could form the components for LAWS[2], to date the department has remained largely silent on their governing ethical framework.  In our view such a framework is essential and must be developed in an open, public consultative process.  It will be crucial to be able to demonstrate to allies and adversaries, as well as to its citizenry, that the UK intends to take a principled approach to the use of military technology.  Ethical guidelines serve to define limits within which industry and contractors working for the MoD should remain when developing new weaponry products.  The development of the upcoming military AI strategy, announced in the March 2021 Defence Command Paper, presents an important opportunity to consult the public and produce such a framework.


      To date the UK has taken a cautious approach to policy on LAWS in international statements. For example, during discussions on the topic under the auspices of the 1980 Convention on Certain Conventional Weapons (CCCW), a more decisive position is needed, with more articulation of the risks and ethical issues arising from autonomous military technologies.  The UK must unequivocally, loudly state that its military will always operate its weapons under meaningful human control, and that LAWS which are not under human control should be the subject of an international ban.  Ultimately, the government must be willing to accept and promote robust controls over harmful technology of this nature, and needs to remove the qualifications and hedges which weaken its current position.


      As a starting point, the government should express an absolute commitment to upholding human rights and freedoms in its actions, including the actions of its military, and state that technology should be used to enhance rather than deny these rights.  It is important too that the UK 'walks the talk' in its commitment to human rights, and behaves consistently with its international human rights obligations.


      Within ML lie evident and ubiquitous issues which embed certain potential harms and known unknowns.  Algorithmic bias, problematic training data[3], false positives and unpredictability have made already commercially used ML-based AI prone to failure or discriminatory offences (recidivism, gender-based violence, disablism[4] and so on)[5] which have led to fines,[6] class actions[7] and increasing calls for regulation.  The UK must learn from current trends towards regulation and mitigation against litigation and become a global leader in ethical standards as well as commercial regulation.  It has an opportunity to lead the world in global business standards protecting human rights values as well as commercial interests of British industry in the global supply chain. 


      The government must stigmatise opacity within the dual-use sectors from unintended uses and bad actors.  Commercial investors indicate a preference in future-proofing and divestment from toxic portfolios. The move to UN Global Compact commitments and the Business Human Rights Forum standards requires increased accountability and culpability, from board level down.  Johan Anders, head of the ethics panel of the Norwegian Sovereign Pension Fund has made clear that:


              “If you think about developing technology for recognising cancer, that is fine. But if you are               adapting it to track down a certain type of individual in a certain environment, and               cooperating with others to make an autonomous weapon out of it, don’t be surprised if we               take a look at you.”[8]



Question 2.    How can the FCDO engage with private technology companies to influence and promote the responsible development and use of data and new technologies?


      LAWS are viewed with considerable concern by many leaders in the tech field.  Tesla’s Elon Musk and Alphabet’s Mustafa Suleyman led a group of more than 100 leading robotics experts in a 2017 call to the international community to protect humanity from these weapons before it is too late[9]:


              Lethal autonomous weapons threaten to become the third revolution in warfare. Once               developed, they will permit armed conflict to be fought at a scale greater than ever, and at               timescales faster than humans can comprehend. These can be weapons of terror, weapons               that despots and terrorists use against innocent populations, and weapons hacked to behave               in undesirable ways. We do not have long to act. Once this Pandora’s box is opened, it will               be hard to close.”


      The direct development of autonomous military technologies and military AI is being led principally by arms companies.  As these companies manufacture and sell highly harmful high-technology products, there is a need for government to keep their activities under scrutiny and monitor and regulate their export.  Despite existing export controls on military and dual-use technology, there has been a concerningly consistent pattern over many years of the supply of British military equipment to abusive regimes.  We consider that the system of export controls needs to be tightened significantly to prevent sophisticated military technologies from falling into the wrong hands.  We also consider that funding to arms companies for the development of new military equipment should be conditional on their products and activities remaining within ethical limits.


  Universities play an important role in the development of new technologies, increasingly in partnership with industry.  It is important that they too are subject to regulation, and that there are controls on the use of intellectual property developed by universities, the nature of their research activities, funding relationships and the involvement of overseas partners and students in research programmes relating to technologies with potential for harm.  The government should encourage efforts to build an ethical research culture, currently missing from many universities.


  While military-specific technology may be developed by weapons companies, many vital technologies are increasingly 'dual use' – they have the potential to be applied for both beneficial or harmful purposes, which is particularly concerning.  There should be a responsibility on researchers and developers of new technologies to identify both the beneficial and malicious uses of technology they develop and introduce meaningful measures to mitigate against the risk of its malicious use.  A legal obligation should be introduced to protect technology from reverse engineering and restrict it to its intended use. We see from the US Department of Defence's Project Maven[10] (a controversial AI programme facilitated by Google), and others, that there is great concern among engineers[11] about issues such as the lack of regulatory framework, transparency around final use or consistency with corporate ethical codes of conduct. Crucially, there is little assurance that technology or code produced within UK companies may be ring-fenced from military purposes.  This could create an apprehensive workforce[12]. To attract global talent and become a world leader in AI, the UK needs to build a reputation as a champion for ethical and socially conscious innovation.


  The nature of multinational companies working on dual-use technologies in the global supply chain means that technology, as well as expertise (brain drain), crosses national borders and can be sold by industry to any potentially hostile actor or for unforeseen maluse, with profit and harms dislocated.


  The government should take action to ensure that UK-based companies are responsible for any liabilities arising from the use of technologies they have developed, both at home and overseas. This includes ensuring that they have adequate reserves and insurance to cover liabilities. Regulation to prevent and control harms arising from the development of emerging technologies should specifically cover harms overseas.  Incentivising business to develop along ethical guidelines could create safer and more profitable outcomes. 


  Under the terms of Article 36 of the 1977 Additional Protocol to the 1949 Geneva Conventions, states have an obligation to determine whether, “in the study, development, acquisition or adoption of a new weapon, means or method of warfare”, its use would “in some or all circumstances be prohibited by international law”.  Article 36 legal reviews have an important role to play in ensuring that new technologies are used responsibly in the development of military products.  The UK should encourage states not currently undertaking Article 36 legal reviews to do so, and be more transparent about its own review process.


  To build better relations with a UK industry eager for their technology to be used as a force for good the UK should lead by example and work for a legally binding treaty on LAWS.  They should also lead an inclusive process to create their new military AI strategy and an ethical code of conduct for emerging technology and weapons systems.



Question 4.    How can the FCDO use its alliances to shape the development of, and promote compliance with, international rules and regulations relating to new and emerging technologies? Is the UK taking sufficient advantage of the G7 Presidency to achieve this?


  Firstly, the UK government should take opportunities to reaffirm its commitment to and compliance with human rights law and international humanitarian law, both in words and actions.  Secondly, it should support existing agreements which exist to control advanced military technologies, such as the Convention on Certain Conventional Weapons and the Missile Technology Control Regime.  It can do this by leading by example on compliance with the terms of such agreements, resisting attempts to weaken any of existing terms, seek to engage allies in complying with such agreements and act against persistent offenders with soft law mechanisms and unilateral sanctions.


  The UK should also support the introduction of a new treaty to ban LAWS, ensuring that weapons remain under meaningful human control at all times.  The treaty should apply to all systems that apply force based on processing sensor inputs and should outlaw systems which target people and those which cannot be meaningfully controlled by a human, including opaque technologies which are too complex to be understood.  Remaining sensor-based systems should be subject to positive obligations, for example on the location and duration of use and their target specifications, to protect existing laws of war from erosion.[13]  The International Committee for the Red Cross has recommended that states adopt new legally binding rules along these lines[14], and the United Nations Secretary General has also called for LAWS to be prohibited under international law.[15]


  It is important to introduce controls on potentially harmful new technologies into UK law. The European Union has recently published a draft regulation on AI to guarantee the safety and rights of people and businesses whilst allowing uptake of AI technology.[16]   The draft regulations follow a risk-based approach.  Systems with an unacceptable risk, which are considered a clear threat to the safety, livelihoods and rights of people will be banned. This would include LAWS.  High-risk AI systems, including all remote biometric identification systems, will be subject to strict obligations before they can be put on the market and would be recorded on a database maintained by the European Commission.  Systems with limited risks will have specific transparency obligations to ensure that users understand that they are interacting with a machine.  Although the UK is no longer a member of the EU, it shares the same democratic and rights-based values as the EU and is affected by its regulations, the UK should thus adopt legislation providing at least the same levels of protection as the EU's regulations.


  Addressing the UN in September 2019, the Prime Minister committed to organising a summit in London to help guide the norms and standards for the development of emerging technology.  This would be useful in shaping an international conversation on the control of emerging technologies.  It is disappointing that the government has, as yet, taken no action to set up this summit.



Question 5.    Should the Government’s approach to meeting the challenges of technology nationalism and digital fragmentation be based on self-sufficiency, joining with allies or like-minded nations or supporting a coherent global framework?


  With regard to military technology, the government's approach should be based firmly supporting, and where appropriate extending, multilateral arms control regimes while considering other human rights obligations such as the Convention on the Elimination of All Forms of Discrimination Against Women and UN Security Council Resolution 1325 on Women and Peace and Security, which these particularly discriminatory technologies infringe upon.


  The UK should learn from other mid-level and regional powers who can be more effective in making themselves central to the regulation and reform agenda. Austria, for example, has been strategically influential in disarmament affairs.  Significantly, Europe with the UK has led the world, (including a now following US), in financial regulation technology and the creation and implementation of legislation and technology associated with the Markets in Financial Instruments (MIFID II) and Market Abuse Regulation (MAR) Directives.  This has resulted in the establishment of a wealth of extremely valuable regulatory technology (Reg Tech) companies based in Europe and the UK, rather than the US, which had hitherto been the tech-company norm.[17]  This suggests there is an extremely positive future for regulation of sensitive industries (based around algorithmic based abuse) which could be applied to various associated fields, including those related to software for autonomous weapons. The technological and commercial opportunities of leading in reg tech and associated infrastructure would give a market advantage as well as ethical and legal control over national and global standards. While some regions may not subscribe domestically to regulation (eg the US), the enduring and increasing nature of global, cross-border standardisation together with associated fines and obstacles, incentivises all companies to comply for ease of trade, commercial advantage and to mitigate litigation.  This contributes to the creation of norms and often law.


  The UK, therefore, has a unique opportunity to be a global leader in mitigating systemic algorithmic risks.  While the EU's attempt at regulation[18] of digital technology[19] is underway, it exhibits contradictory aims.  It recognises bias and claims a core focus on rights protections but, simultaneously, the implementation of problematic technology is underway.  Meanwhile the rush toward incomplete and inadequate regulatory frameworks could exacerbate concerns of bias and discrimination.   The UK can learn from this mistake by taking a detailed and open approach in all its technology-related policies, developing a thoughtful framework through ongoing consultative processes responsive to changes, while listening to civil society experts addressing the particular issues of AI bias[20], data processing and surveillance[21] technology which could be components used for LAWS.  Including heterodox analyses, such as an intersectional feminist foreign policy lens, would be useful.[22]



  In recent years we have seen a renewed interest in “minilateralism”: smaller sub-UN coalitions of like-minded states. Such coalitions can be powerful tools for increasing UK influence for example at the UN and can shoulder some of the work of convening and caucusing for a reform agenda. However, minilateralism cannot take the place of multilateralism, or of the UN, and attempts to do so are dangerous for reasons of both perception and substance, particularly if they are based around the use of armed force.


  In terms of substance, minilateral initiatives (such as the G7) lack the universality of the United Nations, thus investing in them represents an investment in the inequality within our global system, furthering the risk of “two speed” global governance whereby some states and communities are better served by, and therefore more invested in, our global system than others who, if alienated, are more likely to turn spoiler. Furthermore, if the UK only caucuses with like-minded states, it loses the ability to influence the states where pressure most needs to be brought to bear. Moreover, many issues cross traditional minilateral lines. The UK's allies on issues like human rights are not the same allies in all areas.


  Minilateralism among the like-minded can be perceived as neocolonialism; a small group of invariably rich "white" countries attempting to dictate terms to the rest of the world. This perception in turn weakens the UK’s diplomatic capabilities across all agendas.







Submitted by the UK branch of the Campaign to Stop Killer Robots, members of the global coalition working to ban fully autonomous weapons: www.stopkillerrobots.org/uk_campaign/







June 2021

[1]'DOD Adopts Ethical Principles for Artificial Intelligence'.  US Department of Defense, 24 February 2020. https://www.defense.gov/Newsroom/Releases/Release/Article/2091996/dod-adopts-ethical-principles-for-artificial-intelligence/

              'AI Principles: Recommendations on the Ethical Use of Artificial Intelligence by the Department of Defense'.  Defense Innovation Board, 31 October 2019. https://media.defense.gov/2019/Oct/31/2002204458/-1/-1/0/DIB_AI_PRINCIPLES_PRIMARY_DOCUMENT.PDF

[2]'Off The Leash: The Development of Autonomous Military Drones in the uK'.  Drone Wars UK, 10 November 2018.  https://dronewars.net/wp-content/uploads/2018/11/dw-leash-web.pdf

[3]Sharkey. N,  A Feminist Future Begins By Banning Killer Robots,  Mar 2020

[4]AI Now, "Disability, Bias and AI."  Nov 2019

[5]p7-8 Acherson. R,  A WILPF Guide to Killer Robots.  Jan 2020

[6]Violation Tracker Individual Record, United Health Group


[7]Good Jobs First Violation Tracker; Class action lawsuits alleging discrimination against customers

[8]Joachim Dagenborg, Gwladys Fouche: 'EXCLUSIVE-Norway wealth fund's ethics watchdog warns firms not to make killer robots'.  Reuters, 11 March 2016.

[9]Future of Life Institute: 'Lethal Autonomous Weapons Pledge'.  https://futureoflife.org/lethal-autonomous-weapons-pledge/

[10]Cheryl Pellerin: 'Project Maven to Deploy Computer Algorithms to War Zone by Year’s End'.  US Department of Defense, 21 July 2017.  https://www.defense.gov/Explore/News/Article/Article/1254719/project-maven-to-deploy-computer-algorithms-to-war-zone-by-years-end/ 

[11]Google employees ask tech giant to pull out of Pentagon AI project    https://globalnews.ca/news/4124514/google-  project-maven-open-letter-pentagon/

[12]Williams, O.  Workers in the AI sector are quitting over ethical concerns, New Statesman,  May 2019

[13]'Regulating Autonomy in Weapons Systems'.  Article 36, October 2020.  https://article36.org/wp-content/uploads/2020/10/Regulating-autonomy-leaflet.pdf

[14] 'ICRC position on autonomous weapon systems'.  International Committee of the Red Cross, 12 May 2021.  https://www.icrc.org/en/document/icrc-position-autonomous-weapon-systems

[15]'Autonomous weapons that kill must be banned, insists UN chief'.  United Nations, 25 March 2019.  https://news.un.org/en/story/2019/03/1035381

[16] European Commission: 'Europe fit for the Digital Age: Commission proposes new rules and actions for excellence and trust in Artificial Intelligence'. 21 April 2021. https://ec.europa.eu/commission/presscorner/detail/en/ip_21_1682

[17] European Commision: Markets in Financial Instruments (MiFID II) - Directive 2014/65/EU   https://ec.europa.eu/info/law/markets-financial-instruments-mifid-ii-directive-2014-65-eu_en

[18]Tech services could face EU bans if they breach rules, Thierry Breton says amid criticism over Google ruling


[20]Frank. J - Vice President for UN Affairs, "The stakes are too high to let facial recognition regulation slide," Sep 9, 2019

[21]Lindsey. N, Facial Recognition Surveillance Now at a Privacy Tipping Point, Feb 2019

[22]'What is a Feminist Foreign Policy?' Centre For Feminist Foreign Policy