19
Examination of witnesses
Dr Jurriaan van Diggelen, Professor Dame Muffy Calder and Professor Gopal Ramchurn.
Q81 The Chair: Good day and welcome to the House of Lords AI in Weapons Systems Committee. Our evidence today focuses on questions of accountability, capability and transparency.
I am very glad to welcome our witnesses to this public session. In the room we have Professor Dame Muffy Calder from the University of Glasgow, which we have already heard quite a lot about in our work. We are also delighted to welcome, albeit remotely, Professor Gopal Ramchurn from the University of Southampton and Dr Jurriaan van Diggelen from the Netherlands Organisation for Applied Scientific Research. Thank you very much for being with us today. May I remind you, as I remind all of us, that this session is being broadcast and will be transcribed?
My name is Christopher Cocksworth, a Lord Spiritual and a member of the Select Committee. I am standing in today for our distinguished Chair, Lord Lisvane, who very much regrets that he is not able to be with us today.
Following our Lord Chair’s example, I will begin with the first question. It is a big question concerning the area we have been grappling with all along, which is agency in general and what is meant by meaningful human control of machines.
I have a sharper version of that question, which is along these lines. We are conscious of the evidence that, over time, people can become over-reliant on machine-generated recommendations. I must admit, when I am in my car and I have the satnav on, I sometimes find myself over-relying upon it and thinking, “I’m not quite sure that is right, but I’ll do it anyway”. Sometimes it is even the opposite: “No, you’re always wrong. I’ll do my own thing”. That is not helpful either.
Is that a big concern in AI weapons systems? What might be done about that bias to believe? This is a broad topic and we will need to keep relatively focused. Professor Calder, you wanted to address that general question of meaningful human control by giving us a little bit of background that may help us conceptually.
Professor Dame Muffy Calder: To answer this question, I wanted to say a few words about the kinds of machines or systems we are talking about here. To my mind, these are cyber-physical systems. They consist of physical devices called sensors and actuators that are in the physical world. They sense data and they effect actions.
In the software world, there is a notion of control, which includes situational awareness—what situation is happening?—and decision-making. Those decisions will affect the sensors, perhaps where they are or how often they are communicated with, and control the actuators.
Humans may be involved in that decision-making. We have a range from purely automated to degrees of autonomy. Two standard phrases or definitions used are “human in the loop”, where the human has the ability to stop or start these actions, or “human on the loop”, where the human has oversight of the whole system but not explicit events. In either case, there are serious issues, which we will come on to later, about the human involvement, boredom and challenge involved in this decision-making.
To complete the picture of a cyber-physical system, a third component is networking communications, so communications within the system and from the outside world to control the system. I mention that because we may lose that communication; there may be issues of timing and lag; there may be issues of bandwidth and power, particularly if the system is remote.
Finally, just to draw your attention to the probabilistic components, there is a lot of uncertainty throughout these systems. For example, even with sensing, sensors can degrade or be upgraded; they can fail or downright lie. When we are dealing with rare events, it can be very difficult to distinguish between an error and where something really unusual has just happened.
That is the context of those machines. This raises two questions in my mind. First, what is special about a weapons system as a cyber-physical system? Secondly, what is special about AI as a probabilistic component? By the way, my other colleagues should not worry. I will give you more time to answer questions in a minute.
Just going back to what is special about a weapons system as a CPS, every cyber-physical system is designed for a purpose and will operate in a legal framework. In this case, we have the potential for destruction and harm to humans. It may be offensive; it may be defensive. What is special about it is that decisions may have to be made very quickly and there may be problems with timing and timeline. To my mind, the key actuation or act will be weapons firing, but that is for others to determine.
To the second question, what is special about the AI component as a probabilistic component? There are a couple of things. As soon as you have probabilistic behaviour in a system, it becomes very hard to test and reason about it. An AI component makes that even more interesting because the AI component itself needs to be tested. We are going to be talking about that. The role of data in that is really important. It can also be very tricky to monitor the behaviour of the AI component within that bigger, broader software system.
There are two key questions for me. Where is the human agency in the system and where is the AI in the system? If the AI component is controlling air conditioning, is that such a concern? If it is controlling weapons firing, I imagine that it is more of a concern. I do not mean to be flippant, but that puts it in context. You have to think about where the AI component is in that whole system.
The Chair: Thank you very much. I wonder whether our other two witnesses might want to add to that general comment. We also ought to begin to focus on the specific question of whether there is a bias to believe and trust the system built into the human-machine dynamic.
Professor Gopal Ramchurn: Thank you for the opportunity to be here today. For me, there are three elements to the issue of meaningful human control. The first is reaction time; the second is complacency; and the third is the notion of learning from experience and whether humans are already guided by machines without realising it.
In terms of reaction time, it is irresponsible to put people in situations where they have to make decisions alongside machines that can make decisions at a sub-second level. They have to work at machine speed, and it is difficult to ask for meaningful human control in that situation.
The second thing is around complacency. If we drive a car using automated tools for a long time, for example, we tend to become over-reliant on those systems. This is particularly the case for pilots who fly drones, for example. They get tired and they tend to rely on the machine to make decisions in low-risk situations. Suddenly, a high-risk situation arises, and they are not prepared for it. They can become complacent about what is happening in the world. Asking for meaningful human control does not really account for the fact that humans get tired and become over-reliant on automation.
Thirdly, as you will know, every day you get offered suggestions on your favourite e-commerce or movie streaming platform. These machines are shaping your preferences for various products and services. There is a real question here as to whether we are in control of our choices, whether these AI-based systems already shape our decisions. That is where these three dimensions of meaningful human control raise the issue of responsible design.
Are we thinking holistically about these systems of humans and machines; how they form partnerships; and how humans and machines influence each other’s decisions in situations where one can react more quickly than the other? This poses real problems for us, for our lawyers, for our military commanders and for civil society.
The Chair: Dr van Diggelen, I wonder whether you might come in. Can I add in another dimension, that of time? We are very conscious of the speed at which all of this is operating and the adaptability of these weapons systems to changing circumstances. How do we retain any meaningful human control at speed?
Dr Jurriaan van Diggelen: That is an important problem for meaningful human control, especially in the military. I believe that real-time control is one way of establishing control, where the human steps in and in real time makes decisions that are executed by the machine. We can also move the moment of control earlier in time, for example in the planning phase, which is what we call prior control.
The human specifies a contract with the machine, thinks out all the possible events that could happen and specifies the directives the machine must follow in case certain things happen. Then the machine goes off autonomously. When these situations arise, it will act according to the instructions given by the human earlier.
That is a form of control that is relatively new in human-machine interaction, but it is quite common in human interaction. When human teams interact with each other, they do that all the time. You should also realise that these kinds of behaviours and interactions will develop over time.
At the moment these instructions are given, it is really important that the human can anticipate how the scenario might unfold. The human must also be aware of how the robot will act in those circumstances and which directives to give. In that way, you can shift the moment of control to one in which there is more time.
I would like to make anther point on the reliance issue. Over‑reliance is indeed an issue, but we can design for that. We can make sure that, in the interaction, the machine communicates to the human when the machine is uncertain or when the machine sees novel situations it has not encountered in its training set. If we add that to the human-machine interaction, the human and the machine will gradually, over time, develop calibrated trust. The human will know what the machine can and cannot do, and when to step in.
Q82 Lord Hamilton of Epsom: It strikes me that there is a contradiction in terms here because we are talking about human control and autonomous weapons systems. Surely, by the nature of an autonomous weapons system, there is no human control. Should we not be talking about responsibility for the whole system, probably in the hands of some military commander, who would then pull the whole system if it were not operating properly?
Professor Dame Muffy Calder: Usually when we say “autonomous system”, we think about degrees of autonomy. There is a spectrum: having some actions automated, having some sets of functions automated or having the complete system automated. When I hear the word “autonomous” I do not assume it is fully autonomous.
That is a really key point in these systems. To what degree do we have full autonomy? This occurs in many paradigms from driverless cars to medical devices, prosthetics, water networks, et cetera.
Q83 Lord Browne of Ladyton: Dr van Diggelen referenced the idea of anticipating events where you may need to change the behaviour of the machine and programming it to do this in advance. This is exactly the space I want to dig into.
*************************************************
The algorithms that instruct these machines do not always operate in a deterministic fashion, we are regularly told. We are told that an algorithm can behave markedly differently in what we perceive to be identical sets of circumstances at different times. That will cause the committee a great deal of difficulty as to what recommendations we make, if we are dealing with non-deterministic machines. How do we set thresholds for its behaviour? How do we put it into a mindset? I kind of know how you do this with children as they grow up because I have experienced that, although I was never very successful.
I just wonder whether this is possible with this type of technology. We do not know enough about this technology, how the technology behaves and how it works, to know whether we can apply these sorts of human controls to it. When people talk to me about these machines, my sense is that these machines will do what is in their best interests, not necessarily in our best interests.
Can we, with confidence, train the machine in the same way as we can train a soldier, to act instinctively in a particular way when he or it is in a particular set of circumstances? Honestly, the evidence we have suggests to me that we probably will not be able to.
Professor Gopal Ramchurn: These are really good points. The research shows that there are different degrees of autonomy or, as we would like to call it, flexible autonomy levels that would apply in different situations. As you rightly say, these systems are probabilistic. They may not even correctly recognise the situation they are in in order to choose the right course of action. Even if they recognise the right situation, they might choose a random course of action that is unexpected. In those situations, what matters is the level of risk you incur.
If it is a low-risk situation, you might be okay with a machine being automated and maybe making some mistakes that do not have a huge impact. In high-stakes and high-risk situations, you may want to have more control. That is why we talk about five or nine different levels of autonomy. You can code the machine to ask for permission to take action; you can code the machine to act and then tell you what has happened; or you can ask the machine to do everything without telling you, depending on the level of risk it perceives.
The level of risk is something that a human may be better able to judge or that could be trained into the machine. You could learn this over multiple experiences. The challenge arises when you face a previously unseen situation. That is where it is important to ensure that whoever is making the decision is accountable for it. That means tracking where data is coming from and where decisions are being made by the human, by the machine or jointly.
The Chair: I knew the challenge today would be to move on from question one. We could spend our whole time on it. We do have three bids for supplementaries on this question. If you allow me, I will take those together so we can get them on the table, and then I will invite brief responses to them.
Baroness Hodgson of Abinger: You were mentioning prior control and pre-planning. I do understand that. You may plan something and, once it has happened, you may want to have further decision-making in the planning. What worries me particularly is that I see how efficient a machine can be for systematic decision-making, but there is a lack of empathy. I do not see how you could ever put empathy into its decision-making. That causes me great worry. In matters of conflict, some form of empathy would come into the human decision-making. I do not see how this is included in machine decision-making.
Lord Grocott: I have a pretty short question. I want to get in our minds, or in my mind at any rate, what is distinctive about a weapons system that includes AI or, indeed, a fully autonomous weapons system. We are basically talking about whether it works. It seems to me that a lot of this discussion is, “Does it do what it is intended to do? Can it be relied on to work in all circumstances or not?” That to me seems like it is different in degree maybe, but it is not different as a principle from any other weapon.
I do not know. Does a machine gun work better at certain temperatures than others? Is it disabled because of whatever factor you can imagine? It is helpful in our thinking, or at least in my thinking, if I know that the problem we are wrestling with here is not a brand-new and unthought-of problem when it comes to whether the weapon works or not, which seems to me a very crucial part of the agonising we are doing about AI and AWS.
Lord Triesman: I do not have any interests to declare; I know I have to start with that. This is not supposed to be a dystopian question at all. I am a mathematician by training.
I have been trying to draw on my past experience of running a trading floor in a large investment bank and my observations of the traders, many of them a good deal younger than I and almost all of them mathematicians. When trading in derivatives, while they came to know about the mathematical bases of the operations they were doing, it seemed to me very unclear that they had an instant in‑depth understanding of the algorithms. None the less, when presented with various sorts of information, they gravitated very rapidly to particular solutions in a very fast timeframe. This is nothing like as advanced as the AI and weapons systems we are talking about.
This is not just a matter of meaningful human control, but does the way in which machines work, to any extent, train humans to respond in a particular way? We have just heard descriptions about using our satnavs when they are obviously telling us ridiculous things. People begin to behave almost automatically, not just in the questions they ask but in the instructions they give, because that is how they have been guided to behave. Is there a risk in that?
Q84 The Chair: We have a clutch of questions there on determinism, empathy, distinctiveness and a sort of reverse training. May I ask you to be really disciplined in your answers? You will not be able to address them all.
Dr Jurriaan van Diggelen: There is a lot to unpack there. With respect to the empathy question, that really hits the main point. The question is how much power we are willing to hand over to the machine. There is a line to be drawn there. For people interested in this debate over meaningful human control, the line is that the machine should not be capable of making moral decisions.
What does it mean to make a moral decision? That is the machine, by itself, making a decision that has not been pre-planned by the human, that affects human values, and that might have legal consequences. Those decisions should be left to the human because, indeed, the machine does not have empathy and it does not have consciousness. We are uncomfortable when the machine makes these decisions by itself. We should make sure the human in these cases uses prior control.
The difference between AI systems and traditional systems is that traditional systems act in more of a deterministic way and operate in predictable environments. The machine can cope with unforeseen circumstances. It can interpret the situation and come up with a solution itself. We should be very careful to distinguish between those decisions that a machine can perform autonomously, and those that are morally laden and must be made by the human either in real time or using prior control.
The last issue that came up is the relation between human and machine. The proper metaphor would be a team partner. The human and the machine should collaborate together, and the machine should be aware that it should involve the human in certain circumstances. You should also develop your team over time. One of the things that will develop is that the machine will know when to involve or not involve the human.
The Chair: Thank you very much. That was impressively succinct.
Professor Dame Muffy Calder: I have three quick points. First of all, the big question is about why we automate at all. Typically, we automate when we have repetition or an unsafe environment. Those are the reasons why we want to automate tasks. We need to think about that in this context. Why are we automating?
Secondly, whenever we are talking about AI components, it all depends on how the component was trained, how the training data relates to the testing data and how that relates to the data it is going to meet in real life. It is extremely important to consider the alignment of those. That is really going to affect the overall behaviour of that component. I keep coming back to that question. Where is the AI component in the system?
The last point is about this phrase “understandable AI”. We want the outputs of any AI system to be contextualised to support human agency. It is back to this. All three of us are talking about the same thing. It is called human autonomy teaming. How do we work together with machines? How do we define those boundaries?
The Chair: Thank you very much. We are learning a lot.
Professor Gopal Ramchurn: I can talk about the notion of feedback loops, where machines are now guiding humans based on the data they have used to train and to classify images, for example. We are at risk of letting humans be guided in ways that are not suitable in certain situations.
On the notion of empathy and whether machines work or not, for me the key point is about who is responsible for failures in the system. When the machine has done something unacceptable or unlawful, who is responsible?
We are struggling a bit with that question. It is not a matter just for this group or for ethicists and philosophers to decide. It is matter for civil society to decide. At the end of the day, these autonomous weapons will be used against enemies who are looking to win a war. We should be mindful of those situations that are high stakes and are likely to affect us in unpredictable ways.
Q85 Lord Hamilton of Epsom: What methods are available to ensure that AI systems will perform robustly in settings that might be different from those in which they have been trained according to their original requirements? What testing systems will be needed to evaluate the performance of AI systems?
I would like to add something to that. Do our witnesses agree that, in all probability, we will come to a conclusion as to whether these systems are working as intended only by going to war?
Professor Dame Muffy Calder: Going back to the testing question, to my mind there are two aspects to testing. There is testing of the AI component, which is a fairly well-defined concept. After we have trained a model, we then test it. We have notions of ground truth so we can understand how it performs, and then we deploy it on the real data. That component is part of a bigger system, which may be a totally deterministic software framework. We have to test that.
It is the coming together of two disciplines, which are called software engineering and data science. It is about testing in those two different contexts.
Professor Gopal Ramchurn: The element of systems testing and verification is already in place, but, from the research we are doing in the UKRI trustworthy autonomous systems programme, we are finding that these testing frameworks are not suitable for AI systems that work alongside humans.
In some cases, the humans tend to cause most of the problems, as we saw with the Watchkeeper programme. Most of the drones that were built as part of this programme crashed because of human error. That is because the design, the testing and the verification framework cannot account for human behaviours and human unpredictability. This is the key research challenge we are trying to address.
Dr Jurriaan van Diggelen: This is an interesting point. A machine learning model can make predictions only based on the patterns it has seen in the training data. It cannot generate entirely new knowledge when it has not seen anything before. During development, we should make sure we have a diverse training set; try to anticipate in which kinds of situations this system will be used; and include examples from those situations in the training set.
During testing, we can use techniques like red teaming, where people outside the development team try to come up with examples that could occur in practice but that the developers did not think of and include in their training set.
Lord Hamilton of Epsom: My question has not really been answered. When it comes to conventional weapons, when we go to war we discover that some fail in terms of what we expected of them and some exceed what we expected of them. Surely you have to test these systems in war to know what they are really capable of and whether they are as errant as some people believe they might be.
Professor Dame Muffy Calder: In general, for any system, until we deploy it in what we call the wild, in the real world, we just do not know. We can test, reason and analyse, but the proof of the pudding is in the eating. It is in the actual deployment, whatever the purpose of the system. I keep going back to the purpose of and the role intended for the system.
Dr Jurriaan van Diggelen: The problem in warfare is that the enemy will try to be unpredictable. They will try to act in a way the system has not been trained upon.
Professor Gopal Ramchurn: The question we are getting at is about whether it really gives you any military advantage. It may be working perfectly, but, if you are spending a lot of time fixing its issues and trying to understand why it is making a decision because you are accountable for what it does, you are wasting time, which could put you at risk.
There are a number of military personnel who have been shot down because they were spending too much time looking at drone control systems on their tablets, for example. Testing these systems in the wild, at war, is definitely essential before you can say they confer any kind of military advantage.
Q86 Lord Browne of Ladyton: This carries on from the question I asked earlier. There was something very significant in the last sentence of Lord Hamilton’s question. Conflict is chaos; conflicted environments are chaotic. We seek to train human beings to operate instinctively in those environments so they are not distracted from their job by what is going on around them. We train them to relate to each other, which is why we celebrate the fact that people will go to war not for their country or their king, but for their mates. They will behave in a particular way.
These are chaotic environments, in which we anticipate that things will change, and the enemy gets a vote. They are constantly trying to disturb those environments. In relation to electronic equipment, they are constantly trying to beat security and interfere with it.
Similarly to the question I asked earlier, this is not what we need to be told because we know this. We need to be told whether these machines are capable of being hardened to operate in a way in which we want them to or predict they will in that set of circumstances when there are built into them inherently unpredictable behaviours. Is it wise to put these machines into these environments when, in the cool of a laboratory, they are not guaranteed to behave the same way twice?
Dr Jurriaan van Diggelen: It is a very complex and interrelated environment, but we could try to identify well-defined contexts within these tasks and test our systems for these tasks, and then we can reliably deploy them. The behaviour of the AI should not be very unpredictable to the human. If it is, we should not use it.
The Chair: Do you agree, Professor Calder?
Professor Dame Muffy Calder: Yes, absolutely. I keep going back to my point: where is the AI in that overall system? I would stress that we should pick apart the offensive and the defensive roles for these systems. Perhaps we have different overall agreements about what we want when we are talking about a defensive weapon rather than an offensive weapon. As pointed out, the enemy per se is also trying to confuse us and our systems.
Q87 Lord Fairfax of Cameron: I hope this question is not too general. It is about the advent of quantum computing. Do you see any relevance of quantum computing to these questions or is that a red herring?
Professor Dame Muffy Calder: My quick answer is that it is a red herring. There are other aspects of quantum technology, such as quantum imaging and quantum sensing. They might increase our abilities to, for example, sense and actuate in terms of those systems. That could make a bigger difference than quantum computation per se.
Professor Gopal Ramchurn: It is a red herring for the moment, but longer term we will look at building machine learning systems that work at a quantum level and therefore will bring in new capabilities when it comes to autonomous weapons systems. It is not for now.
Q88 Lord Sarfraz: I was wondering what mechanisms we can put in place to reduce the risk of adversaries trying to tamper with training data right from the outset. Are there ways to mitigate or prevent this?
Dr Jurriaan van Diggelen: The selection of the training data should be done by the development team. You should not just let your AI go out in the wild and collect any training data it encounters. The development team should carefully check that the data has not been tampered with—so no online learning.
The enemy can tamper with reality and in that way create a mismatch between what the system has learned and what it sees in reality. That is a real risk. The enemy will try to be unpredictable and camouflage systems in a strange way so that the AI will not recognise it.
Professor Gopal Ramchurn: The real risk here is where we have a machine learning system for which the training data has been tampered with, which then results in this machine learning system misclassifying certain things. For example, a child holding a toy gun could be mistaken by the system for a terrorist. That is the potential risk that you have with training data that has been tampered with. Data is at the core of most machine learning systems specifically.
The way to solve these issues is to try to assure and protect that data pipeline. That was what Project Maven was trying to do in the US a few years back, so securing the data pipeline by securing the data centres in which these datasets are stored.
As my colleague just talked about, in terms of fooling the systems, there is a risk. If you are using a black box machine learning system, and you do not understand how it works and how it has learned from this training data, it could be subject to certain kinds of attacks. You could use some of the features that it has learned from this training data, even though it is unadulterated data, to fool the system. We have seen lots of examples of this in the literature. There is a risk with black box machine learning systems in particular.
The Chair: Professor Calder, would you be able to come in on that last point? Is there anything more you would like to add on the black box dilemma?
Professor Dame Muffy Calder: You have said it very well. The classic problem of AI is about that training data. In a sense, if you open up the black box and try to summarise what it does, you have kind of given the game away to the adversary, who can then go and poison the data. If you have controlled all the data yourself, that is fine, but that is becoming increasingly difficult and unrealistic. We are having off-the-shelf components here or going to open source out on the web, et cetera. It is very hard to control those data sources.
Q89 Baroness Hodgson of Abinger: That led me into thinking about the security of the systems in general. There is the whole sci-fi concept that somebody tampers with the machine and turns it back on the people who are trying to use it. How could one prevent other people accessing these systems?
Professor Dame Muffy Calder: There are technical answers here, but this is also part of the bigger question of unintended uses of technology. Who thought a jet would become a bomb? We can turn any system and use it in ways that were not intended. I am not talking about the cybersecurity, but the system of getting into the machine and the network is a different issue here from trying to tamper with the data used in the training steps. We would expect in these systems that you would be retraining them at regular intervals. How often they are retrained is a big question.
Q90 Lord Clement-Jones: It is a bit of a change of subject here, given that we have been talking about, in a sense, the nature of the weapons. How can the MoD translate high-level principles, such as reliability and responsibility, to operational reality? What procedures are needed to ensure AI-enabled weapons systems will be responsibly designed and lawfully used? It is all about compliance. I am thinking of the MoD policy statement, ambitious, safe and responsible, and its setting out of the various principles that it thinks should be applied, human centricity, responsibility, understanding, and bias and harm mitigation. As well as the question about compliance mechanisms, you might also comment on the realism of trying to have these ethical principles.
Professor Dame Muffy Calder: Every organisation that develops software has to face these issues. They already face them even without AI. What are the processes of developing software? How do they test? How do they verify? Are they open about the methods they use?
It becomes even more acute in this context. A lot of it simply comes down to culture in the organisation. How professional are the people involved in developing this software? What is the culture about the transparency of the development, the deployment, the discussion of fairness and the ability to challenge? I do not mean challenge within the software system they are building but within the whole software process.
Lord Clement-Jones: What is your view about the realism of embedding that culture in a ministry, a department devoted to defence or, indeed, an armed service itself?
Professor Dame Muffy Calder: I do not think that I am able to comment on the cultural aspects of that particular department. It comes back to the professional software developers, how they have been educated and the culture they are working in together. I think anyone would say that it is not just about the software development; we need other specialists within those teams as well. It is always, again, about diversity and those teams’ ability to challenge each other.
Professor Gopal Ramchurn: I would like to comment on the notion of adding new principles or ways of working to the development of these systems. The worry there is that these principles may interfere with what is already in place in a way that undermines that military advantage we might have. These principles have to be implemented in a way that supports those teams on the ground.
Working with the teams on the ground, we can build these systems very quickly, as we have seen in Ukraine. Many machine learning systems or drone-based systems were built by personnel on the ground within two weeks and deployed with quite a bit of success. One thing is to make sure that these principles are implemented additively to what is already in place.
That needs collaboration with people on the ground. It has to be fair to the humans involved in the systems. The notion of accountability is a foundational element of all these AI-based systems, to give assurance or comfort to the humans in that organisation that, when they make a decision, someone will be able to prove that they are responsible for it or some machine has decided on their behalf within certain regulatory or legal boundaries. There need to be a number of ground-up and top-down approaches to implementing those principles, taking into account the reality of the situation that people face in war.
Lord Clement-Jones: You have explained the issues. We have talked about the culture and so on. How realistic do you think it is to embed those in an organisation in these circumstances?
Professor Gopal Ramchurn: That is where we need to involve not just engineers and scientists in building these systems and implementing these principles. People from different disciplinary backgrounds should be involved, social scientists, psychologists and others, who will help make sure that systems adhere to these principles in ways that are fair to humans and key stakeholders.
Dr Jurriaan van Diggelen: It is also important that a principle such as meaningful human control is not necessarily in conflict with military effectiveness. The military will want to keep control over its AI-based systems, but we should show it how to do that. For that, you need these requirements to be much more precise and to spell out how they translate to training requirements over the entire lifecycle of the system, in the way you develop your user interfaces and monitor over time, et cetera.
It will be difficult to do software audits. It will also be very difficult to prove that the enemy has used its AI-based systems in a fully autonomous mode. Unlike chemical weapons, there will be no spill of chemicals or signs that they have been used autonomously.
Lord Clement-Jones: You talk about the training. How do you make sure, once everybody is trained, that the principles are being put into effect?
Dr Jurriaan van Diggelen: I agree with the previous witness. That is also a part of culture. We did that in safety research, from which we understand that the safety culture of an organisation, with everybody logging incidents, being aware of these problems and trying to improve it, is important for having safe systems. The same holds for responsible AI.
Lord Grocott: We have been talking a lot about principles, but Professor Ramchurn mentioned legal boundaries. The law is more our territory really, as part of a legislature. To what extent is there enough in current legal frameworks or boundaries that exist, particularly international humanitarian law, to cover these weapons, however defined? We still do not have a definition of the weapons, which is a bit of a handicap. Do we need new laws? That is a question. I realise that any regulation would presumably involve some international agreements, but, at some point, the dear old legislature would come into this. Do you think that something new is needed?
The Chair: May I hold your response to that? That is an important theme, but I would like to move on to it in a moment. I wanted to give Lord Browne a chance to come in on the specifics.
Q91 Lord Browne of Ladyton: I am interested in, specifically, how we do these things that we aspire to do. We know that there has been no public consultation in the United Kingdom of our country’s appetite for the deployment of weapons systems that can operate autonomously or semi‑autonomously. We have had none.
In our recent past history, we have examples of technological advances in which we built a parallel system of studying their ethics. In our country and in the United States—I cannot speak for other countries, but I suspect that most developed countries certainly did this—as bioengineering improved and the concerns that we had about its misuse or effects developed, we built a parallel research system. It was not funded quite as well as the research into the bioengineering, but it looked at the bioethics. We had this parallel system of bioethics in which there was this constant conversation going on.
For most of these countries, this conversation has developed into a national regulator. In the United States there are federal regulators and state regulators in the bioengineering field that have come out of this bioethics discussion. At the G7, our Prime Minister announced that he was going to lead global regulation of AI. Yesterday he started and met with three people who are key figures in the AI industry in the United Kingdom for a couple of hours, so the job is done.
Should we not do this? We are having these complicated conversations, week after week after week, with people who are drawing out these things that we are learning the vocabulary for as we go along but have responsibility for the public policy. Should we not just take from the bioengineering example and establish an AI ethics frame of research that we put sufficient money in to start to test these things?
Professor Gopal Ramchurn: I completely agree. That was actually my main recommendation to this committee, to fund a body with the authority to have these conversations, bring in the key stakeholders and work alongside the Ministry of Defence to advise and inform the public and various stakeholders about the implications of these systems. Currently, the conversation is owned by the media and there is a lot of scaremongering going around. This is led by people who are not experts in the field or who have formed opinions about autonomous weapons systems based on what they have read in a book.
You need to have a wider conversation because it is not just about the technical elements of these systems but the values that we hold and the ethics we have in our society. This varies across different publics. We cannot decide what autonomous systems we would allow and how we would allow the deployment of these systems just with a small group of people. Going to the tech industry and the Silicon Valley titans, as we call them, for any opinion is clearly not the right way to go.
There have already been quite a few investments in the UK to look at responsible AI and trustworthy autonomous systems. We are part of these initiatives and we are looking to grow some of these groups that bring researchers from different disciplines to address these key issues. Something specifically focused on defence and autonomous weapons systems is urgently needed to avoid this conversation being taken over by the media and non-experts in the field.
The Chair: Dr van Diggelen, do you agree?
Dr Jurriaan van Diggelen: I agree. AI is developing constantly. We have recently had this next generation AI with the big language models and generative AI. I would also advise to not only focus on kinetic weapons systems. Cognitive warfare is also a weapons system. This new generation of AI turns out to be a very good means of influencing public opinion. We should look at a broad scope.
Professor Dame Muffy Calder: It is difficult to have meaningful conversations if they are too abstract. I would be focusing on the actuation. Remember, these are cyber-physical systems. We can look at the classification of the physical actions, for example weapons firing. As I said, AI in an air conditioning system does not particularly bother me. Weapons firing bothers me and we have to make sure that the conversations include those actuation events; otherwise it just feels a bit like navel gazing.
Q92 The Chair: Let us now move on to Lord Grocott’s question about international law, international humanitarian law, the translation of our ethics into operational principles, and then the holding of us and other nations to account. Are our legalities sufficient thus far? Who would like to come in on this to begin with? Then Lord Mitchell will sharpen the question up a little in a moment.
Professor Gopal Ramchurn: There is huge debate in the legal community around whether we should allow autonomous weapons systems. The challenge is getting everyone to agree, when they all have different ethics and values. For example, in Japan people tend to have very close relationships with machines. They see them as equivalent to humans in some cases; they hold machines as companions. In other countries, the UK’s adversaries, they might have different views on what machines are allowed to do to humans and, therefore, not want to restrict the use of these machines during warfare.
It is a difficult challenge to try to get everyone to agree on a set of rules around restricting the use of autonomous systems in specific war settings. I think that we are going to struggle to get that agreement. We need to be mindful of the capabilities that others are developing and come up with meaningful solutions that resonate with our publics in the UK.
Lord Grocott: We understand that there are lots of different views about this. Could I ask you for your expert guidance as to whether you think new rules, as yet not quite defined because of the difficulties you described, are needed?
Professor Gopal Ramchurn: The difficulty there is that, if you impose rules, you need to be able to police them. AI is no different from any other software. If you say that only certain people are allowed to build software, you are not going to be able to police it. It is difficult to impose those rules.
In particular, it is difficult to impose the rules when the adversaries are going to use these weapons in ways that you cannot control. It is the same thing with nuclear weapons. You can come to an agreement on how and when people should be able to use them in warfare, but anyone can build an AI weapon, potentially, these days. That is difficult to regulate.
Q93 Lord Mitchell: My question leads on from what Lord Grocott was saying. What international standards of practice on development, testing and use of AI-enabled and autonomous weapons systems are needed, if any? When I sit on this committee and listen to these very profound questions that we are asking, which have major implications, I wonder whether the same questions are being asked in Moscow and Beijing. This is an issue like nuclear, where it takes two to tango and we cannot come to any agreements unless the same things are being done elsewhere. I have no feel for it.
Professor Dame Muffy Calder: Are they being asked in other countries? I could only assume yes. Also, are they being asked in other sectors? To my mind, this is not just about regulation of AI as a technology, but about its use. We have to ask the same questions in medical devices and driverless cars. Are you going to buy a driverless car from Moscow? Are you going to buy a medical device from China? We are facing these same issues across many sectors.
The question of regulation is quite hard in the software sector, because we do not have these physical residues. Unlike nuclear or chemical weapons, it is hard to find the traces. We have to go back to the culture and the people who develop the software. A lot of that is about education, particularly higher education, and the people participating in those activities. These are international and cross-sector questions.
Dr Jurriaan van Diggelen: Regulation can definitely be effective. We have the AI Act developed by the European Union. It is very important to have things such as that in place. We should be aware that it has to develop and keep up with new sorts of AI. For example, with the generative AI there should maybe be regulations such as the human having the right to know whether he or she is talking to a machine or a human. These are very concrete and enforceable rules. We should see which of these kinds of rules are feasible and useful to translate to the military domain. There is definitely a role for legislation there.
Professor Gopal Ramchurn: Currently, to my knowledge, there are no well-defined standards for AI-based systems in autonomous weapons development. There is an urgent need to develop testing frameworks and standards that would help the defence industry build and deploy these weapons responsibly.
The research community has responded to that need in some ways. There are initiatives in the AI community that look to educate researchers or use the voice they have in various sectors to raise those issues with decision-makers. That said, most of the research in AI is potentially funded by defence. There is always this conflict when doing research with defence companies into tools that do not look like they will be used in autonomous weapons but have a dual purpose. It is a tricky thing to control. Developing standards, principles and education is key to ensuring that these weapons are built, deployed and regulated responsibly.
Q94 Lord Hamilton of Epsom: I would very much like to return to the whole question of international humanitarian law. I think that you have indicated that it would be very difficult to get any agreement to expand the spectrum of international humanitarian law, not least because the Russians refuse to even entertain the idea of signing up to any of this and will not even talk about it.
Surely there is a reverse side of the same coin, which is that the existing signatories to international humanitarian law have signed up because it is largely meaningless. By that I mean that the only charge it seems anybody can stand up on this law is war crimes. As they know that the chance of ever proving or finding anybody guilty of war crimes is a very small percentage, they are prepared to sign up to it. They know that nothing will ever happen under IHL.
Professor Gopal Ramchurn: I am not a law expert, so I would defer to expert colleagues on my team. When it comes to proving who is at fault, that is where AI-based systems can potentially be equipped with data‑tracking and provenance-tracking toolkits that would help us prove guilt in situations where war crimes might be committed. Establishing standards around that may help a bit with the enforcement of these laws. I would defer to my expert law colleagues to provide a view on this.
The Chair: Dr van Diggelen, are you a bit more optimistic than Lord Hamilton?
Dr Jurriaan van Diggelen: An important result of this effort and these discussions that we are having is to become more aware of unforeseen risks of this technology. We do not know very well how this will play out if we deploy AI in all sorts of domains. Becoming aware of that will benefit every country and definitely humanity.
Lord Browne of Ladyton: This is a global challenge. We cannot insulate ourselves from this. When I was the Secretary of State for Defence, people were telling me that in five years’ time soldiers would go to war with iPads, and they were right. It is all very well for us to talk about what others are not doing. At least in this Parliament we can decide what we should be doing.
I do not have my iPad with me, but I have my iPhone with me. If you want to see just how internationally connected we are, look at the back of your iPhone. It says, “designed in California” and “assembled in China”. I do not know where the bits were manufactured, but it was assembled in China. I guarantee you that it says the same on the back of an iPad, so we are giving our soldiers this. We are giving them something that the Chinese have assembled. We are not likely to get anybody else to assemble them, because we can do it more cheaply there than anywhere else.
We are interconnected. We have to realise the degree to which we are interconnected and work on this. We should take people such as Eric Schmidt seriously when they say that AI, not just in autonomous weapons systems or driverless cars, poses potential and existential risks to the human race. These may be uncomfortable, worrying and difficult things for us that hurt our heads. They do, but we need to engage with them. We should learn from how we have done it with other technologies.
The Chair: Bearing that point in mind, I would like to draw to close with a question we have been putting to all witnesses. In your answer, please feel free to respond to Lord Browne’s point.
Q95 Lord Fairfax of Cameron: If each of you could make one recommendation to the UK Government in this area, what would it be?
Professor Dame Muffy Calder: First, we need consensus about human agency in these systems. Then we design capability around supporting that. It is not about the technology per se, but about designing to support some human agency, for example weapons firing.
Secondly, do not underestimate the power of the international science community. It is extremely important that we stay connected.
Lastly, there is a bit of hype at the moment. I do not think that we have just fallen off the cliff edge. This is just a continuum of software development and automation.
Professor Gopal Ramchurn: I will go back to my initial point. We need to set up a council that would look at bringing in different stakeholders to have these conversations around meaningful human control and all these issues that we have just talked about. We should try to achieve consensus, but I do not think that we will achieve it in all cases.
There is the point about getting away from the hype and trying to take that narrative away from the media. As a professor of AI, I do not see AI as posing an existential risk. Climate change is probably more likely to pose that risk. We need to now take charge of that conversation and acknowledge that it is a sensitive issue and the public should be involved in there.
Ethicists, philosophers, engineers, computer scientists and law makers are all great people to involve in that conversation but, at the end of the day, the public will have to put our politicians in Parliament and will decide on who is most effective at defending the nation. To engage the public in a national conversation would be my advice to the Government.
Dr Jurriaan van Diggelen: My advice would be to look more broadly than the weapons systems and at all ways in which AI will be deployed in the military. It will likely be part of the systems where many interacting components communicate. Maybe the AI will be used as a decision support system upon which humans will base their further decisions.
Eventually, this could very well lead to the use of weapons, but then it is a little more disguised than discussion around autonomous weapons systems. AI as an information processing entity is a very common application and I think that we will see the first applications of AI in defence like that. We can lose control of these systems as well.
The Chair: Dr Jurriaan van Diggelen, Professor Dame Muffy Calder and Professor Gopal Ramchurn, we are very grateful for your time and the thought you have put into this session. It has been hugely stimulating and you have helped us with our thinking. We wish you well in your work. Thank you very much.
