Examination of witnesses
Professor Alan Winfield, Dr Ing Konstantinos Karachalios.
Q18 The Chairman: Can I very warmly welcome Professor Alan Winfield, who is professor of robot ethics at the University of the West of England, Bristol, and Dr Ing Konstantinos Karachalios, managing director of the IEEE Standards Association? We are delighted to see you, and thank you very much for coming today.
I have a little rubric that I need to go through every time we have a witness session which I will go through now. This session is open to the public. A webcast of the session goes out live and is subsequently accessible via the parliamentary website. A verbatim transcript will be taken of your evidence. This will be put on the parliamentary website. A few days after this evidence session, you will be sent a copy of the transcript to check for accuracy. We would be grateful if you could advise us of any corrections as soon as possible. If, after the session, you wish to clarify or amplify any points made during your evidence, or have any additional points to make, you are welcome to submit supplementary written evidence to us.
First, perhaps you would like to introduce yourselves for the record, and then we will start with the questions, starting with Professor Winfield.
Professor Alan Winfield: I am an old-fashioned chartered engineer. I worked on safety-critical systems in industry before moving back into academia 25 years ago. I co-founded what is now the Bristol Robotics Laboratory, and, as you said, I am a professor of robot ethics in the University of the West of England, Bristol. I am probably the only professor of robot ethics in the world, which is kind of cool, I think.
The Chairman: I suppose you wrote your own job description.
Professor Alan Winfield: Something like that, yes. I am also a visiting professor at the University of York and associate fellow at the Leverhulme Centre for the Future of Intelligence in Cambridge.
To give you a brief summary of what I have been doing in robot ethics for the last 10 years or so, first, I co-organised a joint EPSRC/AHRC working group that drafted the ethical principles of robotics. Those were published in 2011[1]. That led me to be become a member of the British Standards Institution working group that drafted BS 8611, a guide to the ethical design of robots and robotic systems. We believe it is the world’s first published ethical standard in robots. To come right up to date, I am involved with Konstantinos in the IEEE Standards Association’s global initiative for ethical considerations in artificial intelligence and autonomous systems, and I am a member of its executive. I also co-chair its general principles committee and, for my sins, I am chair of the Working Group P7001, which is developing a new standard on transparency in autonomous systems[2].
The Chairman: Thank you very much indeed. Professor Karachalios.
Dr Ing Konstantinos Karachalios: First, thank you for pronouncing my name correctly; it is not so easy. Secondly, I am also an engineer. I studied in Germany and have a doctorate in nuclear reactor safety, so I understand a bit about safety issues. I worked for the public interest at the great European organisation, the European Patent Office, in many functions. My path led me to my current position as managing director of a global organisation that makes standards in every aspect of life you can imagine. You may know the wi-fi family of standards, for instance; it is produced through my organisation. Every time you connect to the internet you go through the protocols that we created. This is quite influential. That is my management role. There is another part of my personality, and Alan mentioned it: I launched the initiative on ethical aspects. There is a reason for this and I hope this will come out in this discussion. It is the reason why I am here in front of you today.
Q19 The Chairman: Thank you very much indeed. I am going to start with a broad question; you have seen the way they develop. Does the development and use of artificial intelligence give rise to new and distinctive ethical issues, or are they variations or pre-existing ethical questions? The question is: will established ethical principles suffice, or do we need new ethical principles? Obviously, we may go on to talk about the kinds of mechanisms that may be needed as well, but perhaps you would start by answering that broad opening question.
Dr Ing Konstantinos Karachalios: This is a very big question. I watched the previous sessions you have had here and they have tried to answer part of it. I was very intrigued by the session with the journalists, I must say. They posed some very critical and interesting questions. I would like to start from there and not repeat them. There is a series of issues which is part of the work we are doing in the initiative. We have produced a book. The second version is about to come out and I would be pleased to send this to you because it has chapters with all possible questions, such as on transparency, accountability and legal issues, where the best people in the world have come together, framed them in a very succinct way and proposed recommendations and general principles to address them. I would be pleased to send to you the second version, which will come out in two months. I am sure that Alan will go into more detail if there are specific issues.
I would like to use my presence here to address first the positive side of these technologies that I see coming and that can accelerate, to put it very briefly, the promise of technology to satisfy the material needs of humanity. This is a big promise. It is the promise of Buckminster Fuller in the 1970s—in the last century—that war will be obsolete as long as we satisfy our basic needs. The question is: why is it not happening? We must put a lot of energy in to make it happen, because the potential is there already. This is very positive, and all these technologies can accelerate this. There is nothing new per se, but they can massively accelerate it.
Secondly, there are critical aspects, three of which are beyond technical expertise. The first is that code is above law. It has already become a reality. Where decades ago Lawrence Lessig said that code is law, code is now above law, and I do not believe that the policymakers should accept this. You should reclaim the territory that you are losing.
There are examples that I can give you if you would like, but I will go to the second point, which is that the possibilities that are given by these technologies, together with sensoring ubiquitous data-gathering and so on, may lead to an erosion of democracy. This is very serious, and it is already happening. It means that if our private life cannot be distinguished from public life, nobody can be an autonomous political actor. This is serious and happening already. We must not kill democracy in order to protect it. These technologies play a massive role in this context.
The third metaproblem is what I would call the Stanislav Petrov theorem. I do not know if you know this guy. He died impoverished a few months ago in Moscow. On the night of 25 to 26 September 1983, he was in a bunker surveying the computer systems and saw that the computers were telling him that the US had attacked the Soviet Union. He ignored them. He said two sentences, and this is the theorem: “We know better. We made you”, and he ignored them. This is a very important action. If we lose it, we have no future. We must always be able to say, like Petrov, “We know you. We are better than you. We made you”. If not, there is not much future for us. Those are the three levels of risk that I see.
The Chairman: Thank you. I may come back to you, because clearly you are talking about new impacts, so in a sense you are arguing that we need new ethical concepts.
Dr Ing Konstantinos Karachalios: Yes.
The Chairman: We need to unpack that a bit further. Professor Winfield, what is your take on that?
Professor Alan Winfield: Like Konstantinos, I think it is a super-interesting question. It is a difficult question, because one of the problems with artificial intelligence is that we do not know what natural intelligence is. There is a fundamental definitional problem. We are trying to build something to emulate natural intelligence, yet we do not have a scientific theory of natural intelligence. Let me be a little more specific.
The first thing I would say as an engineer is that AIs—let us call them AIs, because those are the words that we have in front of us—are just engineered systems like washing machines and word processors. They should therefore be subject to the same engineering best practices, and if those AIs are safety critical, such as driverless car autopilots or medical-diagnosis AIs, they should be held to the same high standards of provable safety and reliability that we would expect from aircraft autopilots or medical equipment.
To come to your question, at the same time, all social AIs, such as chatbots, personal assistant AIs, care robots, robot pets and robot toys, have the potential to deceive. In other words, the human might come to believe that the AI cares about them, and in turn the human forms an attachment with it. An AI is an engineered system and, of course, it cannot care about someone any more than a washing machine can. This potential for deception is a special property of AIs and something that requires special ethical consideration. It is not the only example, but it is the one that I would like to present to you as evidence.
The Chairman: The way you talk about it is very interesting. It is almost like stripping away the mystique, really.
Professor Alan Winfield: Exactly.
The Chairman: Just treat them as machines, basically.
Professor Alan Winfield: That is exactly right, yes.
The Chairman: I am going to bring in Lord Swinfen in a minute, but I want to ask you about the mechanisms. You have both accepted the need for new principles, in a sense, although you have not been specific. But, let us face it, through partnership on AI, from what you and the IEEE have put together there are principles that can be adopted. How do you put those into practice? What mechanisms are required to ensure that any principles you adopt are going to be applied?
Dr Ing Konstantinos Karachalios: There are several elements. I think every institution should do its best. No one player can face it and tackle it all. It is your responsibility to regulate; it is our responsibility to self-regulate. I can tell you what we can do and how we can contribute. We, the technical community, have a duty. We have a duty to stop obfuscating the dialogue around these things and put them under the spotlight, which is what Alan is doing right here, and to do a better job from the beginning. These are engineering systems, and we cannot let out software systems that have an impact on the physical environment without them being thoroughly tested, as beta versions or whatever.
We must force the software engineers to do a job like the ones the civil engineers do. You build a bridge and you are sure it will work, or almost sure. We must take into account our safety, dignity and privacy in the design and not as an afterthought. This is what we are trying to do at the IEEE; we try to design and develop systems that will help the engineers and architects to do a better job from the beginning, otherwise we can only say, “Okay, we did what we did and others will deal with the consequences and find solutions for the problems we have created”. You must stop accepting this from us and we must do a better job.
Lord Swinfen: Do we need a definition of natural intelligence so that we can define artificial intelligence?
Professor Alan Winfield: Probably, yes, but it is very hard to give one. Here is a very simple definition: natural intelligence means doing the right thing at the right time. However, that is not a scientific definition or a measurable definition. The truth is that there is no adequate scientific definition of natural intelligence. In fact, there is no general theory of intelligence.
Dr Ing Konstantinos Karachalios: May I add something? We can assume that our brains are computer machines, Turing machines—and there are people who believe this—or we can say that our intelligence has a source that we cannot define; it is deeper. People call it the soul, or the spirit, or whatever, but there can be different sources feeding our intelligence. I would not exclude it. To believe that we are machines is a very reductionist view of human beings that has not only social but political implications. This is an interesting discussion and worth getting out. Unfortunately, it does not take place within the scientific community.
The Chairman: I am going to bring in Lord Levene, but, really, we cannot rely on the Turing test as being anything useful?
Dr Ing Konstantinos Karachalios: We must leave it open that our intelligence has other sources deeper than this.
Q20 Lord Levene of Portsoken: Professor Winfield, one comment you made rang an immediate bell with me when you were talking about washing machines. We have a mews house. The young couple who were living in the mews house went out on a Sunday morning and left the washing machine running. The next thing was that it caught fire and virtually gutted the house. Of course, you get the insurance company involved, which is also going to have a big interest in this. That is a minuscule problem. The major one, if what we have read is correct, is that the disaster at Grenfell Tower was apparently caused by a fridge. What, if anything, at the moment is being done to try to bring into a clearer light who is responsible for it?
Professor Alan Winfield: It is certainly true that AI systems complicate the question of responsibility, but in many ways they should not. It is the designers, manufacturers, owners and operators who should be held responsible, in exactly the same way that we attribute responsibility for failure of a motor car, for instance. If there turns out to be a serious problem, generally speaking the responsibility is the manufacturers’. That would typically be the case, but going back to the point we have just made about not granting AI some special status, we need to treat AI as an engineered system that is held to very high standards of provable safety. If that AI happens to be in a washing machine, we need standards for that AI.
Lord Levene of Portsoken: I have one quick supplementary question. You say the manufacturer, but which one? Is it the manufacturer of the washing machine or the one that made the printed circuit board in it that failed?
Professor Alan Winfield: Typically, if it is an aircraft, we would regard the final manufacturer—Airbus, Boeing or whoever—as responsible, irrespective of the fact that there is a huge supply chain. Typically—forgive me if I am wrong; you probably know better than I do—it is the final assembler of all the subsystems that assumes responsibility for the safety and reliability of those subsystems.
The Chairman: I am sure we will come on to that with our next evidence session on legal accountability and so on, but in principle you are right.
Lord Giddens: The issue of what intelligence is is so central to this whole thing, I find the elusive and vague way in which people use it amazing. I would ask you to develop your point a bit more. I taught psychology in Cambridge for quite a while, and you always had to teach the evolution of the debates on intelligence. There is a huge literature. There are disagreements about whether Spearman’s g exists. Nevertheless, it seems to me there is one really distinctive thing and it is that to be a human being you have to know semantics, and you have to know the meaning of things. You cannot know the meaning of things unless you act in the world, it seems to me, and you are the possessor of a society and a culture. That would never, as far as I can see, apply to machines.
Therefore, I want to know what implications one draws from that. I do not know if you know Searle’s famous philosophical attack on the idea of AI: that it could never rival human intelligence precisely because, no matter how clever you are, as a computer speaking Chinese you would not know the meaning of the words you are saying. That is what Searle argues.
Professor Alan Winfield: The Chinese room argument.
Lord Giddens: That is a really substantial point, because what intelligence is is the core of it all, and I have not found many people before, listening to you today, who have made that connection and explored that relationship. It seems to be so crucial to me.
The Chairman: We will come back to that, if we may.
Viscount Ridley: As a postscript to Lord Giddens, with whom I nearly always agree on everything, I am not sure I agree with him on this.
Lord Giddens: I agree that we agree on everything.
Viscount Ridley: Was it not Justice Potter Stewart of the US Supreme Court who said about pornography, “You can’t define it but you know it when you see it”? Is that not good enough for intelligence?
The Chairman: Can we come back to that, maybe as an adjunct to answering Baroness Bakewell’s question? Otherwise, we will run out of time, and we certainly do not want to do that.
Q21 Baroness Bakewell: We are getting in really deep here. Where is the responsibility to lie for ethical developments even if we cannot define natural intelligence? Someone has to hold responsibility. Should it be within organisations? You speak about your institution being about bringing about consensus and why it takes so long. Can there be a consensus about something that is so amorphous? We use words such as “duty” and “obligation”, but all these are personally governed by a whole background of information, so where are we to lodge our trust in the decisions to be made?
Dr Ing Konstantinos Karachalios: We are not going to solve all the problems, but I think we can get better. Although it is vast and difficult, we can get more concrete. This is what we are trying to do. Also, morals and ethical values are not the same everywhere; they are very contextualised. What we are trying to do with the first standardisation project we have started, is to enable the software architects and engineers to take at all ethical considerations into account—they are not doing that now—to think and to be more self-reflective. This is the problem. It is not a problem for which we can find a perfect solution. You need to reflect upon what you are doing, and if you are not doing that you are a danger to everybody else; you are just a war machine. The engineering community must stop being a war machine against society. We must assume our responsibility and be self-reflective. This is what it is all about: it is an education of ourselves. We owe this to society.
Baroness Bakewell: But I must ask: who are “we”?
Dr Ing Konstantinos Karachalios: In our case it is the technical and scientific communities that have a duty. It is your duty to support us or to encourage us or to pressure us to do this, because nobody can do our job. We have to do our own job.
The Chairman: Do we need an identifiable cadre, therefore, of AI engineers? Is that what we are arguing for?
Professor Alan Winfield: I would say that more generally we need every AI engineer to be trained in ethics, and in responsible research and innovation.
Baroness Bakewell: Should we also have specialists as we do in hospitals? We have ethics committees, and the people sitting on those ethics committee have specialist knowledge.
Professor Alan Winfield: That is absolutely right, yes.
Baroness Bakewell: We need specialist ethical engineers.
Professor Alan Winfield: I would certainly advocate—and I have said this for several years—that we have reached the point when, for instance, AI research projects should be subject to ethical approval as medical research has been for many years, for the reasons that we are discussing.
Baroness Bakewell: Where are you to find a consensus on who they should be?
Professor Alan Winfield: That is a great question. There are two sides of the coin, one side is education, which we are talking about here, but the other side is standards, because it is in standards that you effectively express, articulate and formalise ethical principles, and these are precisely the routes that we are taking here in the (IEEE) initiative.
The Chairman: We have to have people like you setting those standards.
Dr Ing Konstantinos Karachalios: I found what Lord Giddens said very interesting, and I would like to say a few words on this; I have been thinking a lot about what Lord Giddens has mentioned. The conclusion I came to is that human intelligence is not the capacity to choose between options that you are given; it is the capacity to pose a dilemma that is there but which people do not want to see. It is there in the conditions, and it is the courage of your heart to pose a dilemma, not to solve it. The dilemma is very painful and you pay a price. Only humans have this capacity. A chicken has the capacity to choose different options—which seed to pick up—but a human has the capacity to pose a dilemma. This is something no computer can do.
The Chairman: Offline there may well be further discussion, but you make a very interesting point.
Q22 Viscount Ridley: In relation to the last conversation, I remember having a conversation with Ian Wilmut shortly after he cloned Dolly the sheep. In a similar session to this, he was asked to describe the science and then asked to leave the room while the grander people discussed the ethics. He was very cross about that and said, “No, we biotechnologists have views on ethics, too”. It is worth remembering that point as one goes forward, and I am sure you do.
My question is about transparency. We cannot be far from the point when artificial intelligence diagnoses a disease or offers a legal opinion without being able to explain how it reached that conclusion. That is already true of human beings in some sense, but is it a particular problem with AI, with robots, and how transparent should artificial intelligence systems be?
Professor Alan Winfield: I take a very hard line on this. I think it should always—always—be possible to find out why an AI made a particular decision. That is very easy to say, of course, but we know that in practice it can be very difficult. It seems to me absolutely unacceptable that one might accept the decision of a medical-diagnosis AI or a mortgage application recommender system without understanding why it made that decision. Many members of the AI community will get very cross with me—I am sure they are right now if they are watching—because, of course, what I am effectively saying is that we should not be using systems that are not transparent, such as deep learning systems. I would simply apply an engineering approach whereby we need to be able to understand why the system makes the decisions it does; otherwise, if a system goes wrong and causes harm, we simply cannot find out what went wrong.
I would like to offer the analogy of aircraft autopilots. We all understand very well the standards that we set for the engineering of those systems, and an AI autopilot and driverless cars, for instance, should be held to no less a standard of safety, and explainability or transparency.
Viscount Ridley: Is not AlphaGo already failing that threshold?
Professor Alan Winfield: It is, yes.
The Chairman: By which you mean deep learning?
Viscount Ridley: By which I mean move 37 in game two, which we were told about and nobody knows why it did it.
Professor Alan Winfield: My challenge to the AI community—and they are smart guys, smart men and women—is to invent AI systems that are explainable. I do not believe that it is technically impossible.
The Chairman: Has the horse not already bolted, though?
Professor Alan Winfield: It may well have done, except that we can still regulate—and I believe that we should—and say that it is simply not acceptable to have, for instance, a medical-diagnosis AI that cannot be explained.
Viscount Ridley: I have a supplementary question on that that I was supposed to ask. Does the degree of acceptable transparency differ depending on the situation? In other words, some of our evidence has suggested that it will be fine in some circumstances but not in others, so in a game it might be all right and in a medical diagnosis it might not be. Would you accept that?
Professor Alan Winfield: I would indeed. I am focused primarily, as you can probably tell, on safety-critical systems. That is where if the AI goes wrong, physical, financial or psychological harm could result—in other words, where harms result from a failing AI.
The Chairman: Dr Karachalios, do you agree with Professor Winfield’s hard line?
Dr Ing Konstantinos Karachalios: Yes, and I believe that there are different aspects of it. Transparency is not explainability. It is different. A system that can explain what it does is different from a fully transparent system. There are efforts to make decisions more transparent. There are elements of the European Commission’s GDPR that are forcing this transparency. This is the right way to go—and it can be done—to give direction to the researchers and scientists.
In addition, it is very difficult to understand why dataset A has been transformed to dataset B, because we do not remember the way computers remember. We do not have this capacity. Even if we see it, we do not understand how it came about. This is a problem that we need to understand, because if you do not get a job or you are refused medical treatment, nobody can tell you why. We need systems that can explain this, I agree with you.
Baroness Bakewell: Would your view that there should be knowledge of how these things work, governed by your ethical concerns, be shared beyond western culture and western thinking? Would they be shared with other more elusive cultures that may have a different concept of natural intelligence, conscience, duty and so on?
Professor Alan Winfield: The question of interoperability applies, for instance. If a manufacturer somewhere in the world wants to sell us a driverless car, that driverless car should be fitted with the equivalent of an aircraft flight data recorder, and just like an aircraft flight data recorder the data contained in that should be available by law to an accident investigator, regardless of the culture from where the car comes. The point is if that culture wants to sell us driverless cars, we should insist that the logged data should be publicly accessible.
Baroness Bakewell: Can you imagine down the decades, with the development of different and divergent cultures, these values not being universal?
Professor Alan Winfield: I can, but in a global economy you are not going to help yourself economically.
Baroness Bakewell: So the driver would be economic, would it?
Professor Alan Winfield: I guess, yes.
The Chairman: You seem to be making a distinction between transparency and explainability. Would either of you like to unpack that?
Dr Ing Konstantinos Karachalios: Full transparency is having the source code to see how it is written. It is fully transparent but it does not explain anything. You do not understand it.
Professor Alan Winfield: An example might help to clarify. If you have a driverless car and it is involved in an accident, the accident investigators need to understand what happened to cause the accident. That requires transparency, not necessarily explainability. Explainability for the user of a care robot—say it is an elderly person with a care robot in their home—means that that elderly person should be able to ask the robot in some fashion, “Why did you just do that?”
Lord Giddens: Following up what Lord Ridley said, this would imply that you need analogous structures to those that exist in the professions, would it not? In medicine, you have organisations that oversee the decisions that are made, and obviously you cannot investigate all the decisions that are made, but in all these cases, including airlines, you have supervisory institutions, and the argument would seem to me you must have those also wherever IT is deployed.
Professor Alan Winfield: I think that is right, yes.
Lord Giddens: In the case of driverless cars, it is possible that it will never take off at all because you will not be able to solve the question of responsibility in law.
Professor Alan Winfield: Exactly.
The Chairman: Do you want to roll that into your question, Lord Giddens?
Q23 Lord Giddens: What system of accountability should there be for artificial intelligence systems? How far should they be held accountable compared to other institutions? In a way, I tried to suggest how I would answer that.
Professor Alan Winfield: The first thing to say, and I am sure that I speak for Konstantinos as well, is that an AI can never be responsible, so it is humans and not robots or AIs who are the responsible entity here. The question then, of course, is figuring out who exactly is responsible in a given situation, and that is where the transparency is so important. If we do not have transparency, we cannot properly have accountability. That is probably not a very satisfactory answer.
Lord Giddens: Yes, except the most important thing, it seems to me, is to be able to enforce transparency where it is needed.
Professor Alan Winfield: Exactly.
Lord Giddens: You could never have a system in human medicine where every decision is transparent. You simply have a system whereby decisions can be questioned and a code of professional ethics.
Professor Alan Winfield: I absolutely agree. In January, the Alan Turing Institute published a report in which they recommended what they call an IA watchdog. I am not clear in my own mind whether you need a single body or a body that is specific to particular domains. Care robots might be one body, driverless cars might be another. I think we absolutely need to have those kind of watchdogs.
Lord Giddens: If I might interject, your example is good, because in airline systems you already have that.
Professor Alan Winfield: Exactly, and I think that is a great model of ethical governance.
The Chairman: You are not saying that the use of AI should be totally risk free, because there is bound to be an element of risk in autonomous vehicles or whatever it may be.
Professor Alan Winfield: It cannot be.
The Chairman: When we talk about accountability, it is accountability in so far as you do not accept risk yourself?
Professor Alan Winfield: I am sorry, I am not sure I understand the question.
The Chairman: There is an element of risk that you accept.
Professor Alan Winfield: That is absolutely true.
The Chairman: You cannot expect AI to be totally risk free, so you are not 100% accountable for the consequences in every case for the use of AI.
Professor Alan Winfield: Indeed, yes, and there is no question that there will be accidents for which there is not.
The Chairman: Otherwise we would never adopt AI in any particular respect.
Professor Alan Winfield: That is exactly right. Sorry, I misunderstood.
The Chairman: I am just testing the proposition, basically.
Professor Alan Winfield: Yes, I agree.
The Chairman: You have a very robust approach to the adoption of AI, but the question is: does it go all the way to saying that you are legally 100% liable for every consequence that occurs?
Professor Alan Winfield: It cannot, I agree with you.
Q24 Baroness Grender: I am going to ask about the issue of bias. It was interesting that you used the phrase “potential to deceive” earlier on. We all believe we lack bias, but all research suggests that we are wrong. One of the questions that we want to ask is: how can artificial intelligence systems be developed so that they are not discriminatory or there is no bias in the algorithms built in by the individuals who have written it?
Dr Ing Konstantinos Karachalios: Of course, if we replace "bias" with our "human preferences", this is what makes us human. You can never be neutral; it is us. This is projected in what we do. It is projected in our engineering systems and algorithms and the data that we are producing. The question is how these preferences can become explicit, because if it can become explicit it is accountable and you can deal with it. If it is presented as a fact, it is dangerous; it is a bias and it is hidden under the table and you do not see it. It is the difficulty of making implicit things explicit. This is one of the most difficult things in life, and in politics.
We have a specific project that we started precisely on how we can make our algorithms, data and the interface explicit, to make clear the preferences and the bias. This is the best we can do. We cannot have bias-free anything, but at least it can be explicit. Our project, P7003, aims to define precisely what an unacceptable bias would be. The words must be explicit: how you can treat data, the collection of data and the quality of the data, and so on. This is very important, because otherwise people will be faced with decisions that will affect their lives and they will not know why they have affected their lives in a negative way and so on. It is time to do it, and we have started this.
Professor Alan Winfield: I would say, again as an engineer, that a biased AI is a badly designed AI. It is an AI that has been designed with uncurated data. More than that, as Konstantinos said, AIs reflect the unconscious biases of their human designers. It is critical, therefore, that design teams reflect the gender, age and ethnic mix of the societies that they hope to serve; otherwise, there will be inevitable unconscious biases.
Baroness Grender: Children cannot be part of the design team, for instance, and the Children’s Commissioner for England noted that the risk of children being excluded from the development of AI is where bias can creep in. Yet they are potentially the most exploited group, because they are so susceptible to everything. Let us talk very specifically about children and the bias issue. Just as an example, what do you do to overcome it?
Professor Alan Winfield: There is a very troubling example of one of these conversational AIs—a loudspeaker that you can speak to. In the US it was discovered that very young children were using this and were becoming bad mannered. The problem is that an AI system does not require a child to say, “Please can you tell me what’s on television?”, and, “Thank you”, afterwards. To me that suggests that the design team did not even have parents let alone child psychologists or teachers on the team. It would have been ever so easy technically to build into the chatbot a requirement for politeness—“please” and “thank you”, essentially. You are quite right about young children. These were pre-school children, and they are too young to be part of that design team. Surely adult proxies should and must be part of the design team.
Baroness Grender: How do you bring that about? What do you need to do to ensure that happens?
Professor Alan Winfield: It is part of well-established frameworks, and responsible research and innovation would require this, such as the 2014 Rome declaration[3] and the EPSRC AREA framework[4].
The Chairman: You used the interesting expression “uncurated data”. We are going to look at the strength of data protection and so on in the context of AI applications. It is interesting that in the current Bill and the GDPR there is that point about automated decision-taking, but it is only automated decision-taking; it is not partly automated decision-taking. Do you think that makes any difference in the current context?
Dr Ing Konstantinos Karachalios: Of course. Fully automated decision-taking is very dangerous. There should always be supervision by humans as to the final outcome, if it is of critical importance to other humans. It is not only about the safety systems that guarantee the safety of physical systems. I do not know if you have heard what happened after the Las Vegas massacre, but there were news feeds on Google and Facebook with terrible news that was fake. They were feeding the public with fake news, because there was no journalist there to supervise them.
Professor Alan Winfield: Machine-generated news.
Dr Ing Konstantinos Karachalios: It can cause a lot of harm to society and to people. Just because there is a computer in the loop, they let it be. We should not accept it any more. We should say, “Stop it. You are unqualified for this”. This is very evident to me. We should not have any lenience. Just because the computers are in the loop, they do not deserve any asylum for this.
Q25 Lord Hollick: We come to the ownership and exploitation of data in the public domain. When the Royal Free NHS Foundation Trust entered into an arrangement with DeepMind to hand over 1.6 million patient records, it obviously sparked controversy about consent and privacy, but it also sparked a lively discussion about how that data should be exploited and who should get the benefit of that exploitation and over what period. Can you both please tell us the ideal structure of a deal to protect the public interest, and to ensure that the public interest is suitably rewarded in terms of data, information, apps and financially and in a way that would still encourage the digital companies to enter into those arrangements? What is the ideal deal?
Dr Ing Konstantinos Karachalios: This is a huge question and probably one of the most important ones. I would say, though, before we go there, that we must reverse the current situation. The current situation is we have no agency over our personal data any more. This is a global problem. In the way we connect with the networks, we have lost any agency over digital identity and our data. This is a huge failure and a danger to democracy. This is not about money; it is very political. What can be done? There are practical solutions in how we structure our access to the networks so that we keep agency and we reveal and conceal what we want about ourselves and get some control over it. This is the first thing.
The second concerns the data that we have generated anyway. A lot of it is in the public interest and we should not privatise it. Data generated in the public domain can be useful for many purposes, such as helping clinicians to understand the behaviour of diseases. This can be of benefit to humanity and it should not be privatised. Of course, it must be done in a way that does not expose personal information.
This goes back to the first question I mentioned. This is an extremely complex aspect and has different levels of consequences. The highest level is the danger to democracy, because with no control of your identity you are a slave. You may be in a golden cage with a lot of high-tech gadgets and so on, but you are a slave. This is not the future; we are there, in my opinion.
Professor Alan Winfield: The problem is that we have all made a Faustian pact with the big AI companies. They give us really cool systems such as search engines, machine translation or social networks for free in exchange for our personal data. It transpires that data is extraordinarily valuable. What worries me, in addition to the question of what to do about it, is that many people simply do not understand the extent to which their own data is being used in this way.
Lord Hollick: Let us stick to medical records, because nobody who goes to see a doctor or to hospital expects their data to be made available. They do not give consent, but they may do when they use a search engine, so your argument does not apply there. How are the interests of the public to be protected?
Professor Alan Winfield: By having very strong privacy protections on personal medical data.
Lord Hollick: How do the Royal Free NHS Foundation Trust and similar bodies in the public domain get access to the skills in order to argue their side of the case to protect the public interest?
The Chairman: Are data trusts a solution?
Professor Alan Winfield: They may well be. To be perfectly honest, this is outside my area as an engineer, but I am on the ethics advisory board for the big EU Human Brain Project, and we take extraordinarily seriously the privacy of brain data from brain scans. I understand from colleagues that there are really great technologies for anonymising that data so that science can still get the benefit from that data without it being possible, as it were, to trace it back to an individual who was the source of that data. I think we need to have data trusts, as the Lord Chairman suggested.
Lord Levene of Portsoken: Clearly, people give up the data, but have they given permission for it to be used? We are all faced with using something online and you have an eight-page document that you are supposed to read and click “yes” at the bottom of if you want it to happen. I think the number of people who read those is tiny, yet by doing that you are giving permission for you know not what. Has any thought been given as to how that can better be regularised?
Dr Ing Konstantinos Karachalios: I call it “broken consent”, and I alluded to it in my previous explanation here. We have started a whole series of standardisation projects in this effort that we are making, and it is no coincidence that more than half of them are about data—children’s data, student data, data gathered in the workplace and general privacy considerations. There are no easy answers to this, but I can tell you that hundreds of people right now are working on this. They do not have an agenda and they are not paid by any multinational or whatever. We are trying to reclaim this lost territory and do whatever we can. I think it is extremely important and I am ensuring the entire organisation is putting a lot of emphasis to try to address these issues.
Lord Hollick: There is a stark contrast between the public sector and the private sector, because in the private sector there are battles royal being fought about who owns the data and you are not allowed to use that data. I fail to see why the horse has bolted in the public sector but in the private sector this remains a very live issue of keeping the horse in the stable.
The Chairman: I am sure we are going return to the whole data issue later in our inquiry, but I would like to come on to Lord Swinfen now.
Q26 Lord Swinfen: Many respondents to our call for evidence believed that the growing use of artificial intelligence was likely to exacerbate existing economic disparities. Do we need to address the potential economic disparities that could be caused by widespread use of this new system, and, if so, how?
Professor Alan Winfield: I absolutely believe that we do, yes. It is vital that not only the benefits of AI should be shared by all in society but the wealth created by AI. That is really important. I often tell people not to forget, of course, that the basic technologies were all funded by taxpayers, so in a sense there should be a premium back to the taxpayer. It would not have happened without that initial investment in military research, if it is the internet, or civilian research. The world wide web was funded by European taxpayers and basic AI technologies were all developed in universities, funded by taxpayers. It seems to me perfectly right and proper that we should all share the wealth. How do you do that? First, of course, the big AI companies should pay their taxes, but we also need to have innovative wealth distribution systems. We should be looking at things such as a universal basic income or a universal conditional income or a negative income tax. I am not an economist, but one thing is clear to me: something has gone very wrong in the advanced economies when, for some of decades, we have had increased productivity but wage stagnation. This can only be because of automation.
Dr Ing Konstantinos Karachalios: I agree. I would start with the famous curve where productivity has grown and middle incomes have stagnated since the 1970s, since the onset of ICT technologies. They are not fulfilling their promise for more equality or more progress in society. More and more wealth is accumulated by fewer and fewer people, and not only wealth but power. These technologies that we are talking about may even accelerate the pace, and this is not a place we want to be. A British artist, James Bridle, made a nice exposition called “car trap”. He says that this technological progress may take us to places we do not want to go to as a society. We may lose skills, jobs and personal autonomy. We may lose—this is also very important and goes back to our intelligence and cognisance—the capacity to make sense of the world. We do not want to go there, which means that we cannot just leave the technology to rage. We must put a framework around this. It is not the regulation of technology production itself but the outcome and the distribution that needs to be regulated, and I think this should be done.
The Chairman: You have certainly stiffened our backbone generally in the course of this afternoon.
Q27 Lord St John of Bletso: We have read the recent report by Professor Dame Wendy Hall and Jérôme Pesenti on growing the artificial intelligence industry in the UK and the general recommendation that there is no need for regulation. Dr Karachalios, you spoke about IEEE P7000 and IEEE P7001 and a range of international standards on ethical developments. My question to you both is: do you believe that the ethical development and use of AI requires regulation? If so, what type of regulation?
Professor Alan Winfield: First, we need standards, and I think we need standards more than we need regulation. Standards are the hidden infrastructure of the modern world. Standards are as important as roads and telephones. They are part of the infrastructure of modern life. There is no doubt in my mind that some standards will need teeth, particularly if, and I am sorry to keep going back to the phrase I used, they are safety critical. If the AI is part of a safety-critical system that has the potential to cause serious harm, those standards need to be mandated, so there needs to be regulation. I am not saying that we need to regulate everything, but we need to be selectively regulating particularly the safety-critical systems that we have already mentioned.
The Chairman: But not the ethical principles?
Professor Alan Winfield: The ethical principles underpin the standards, some of which are mandated. I see that there is a flow. In fact, I submitted written evidence to the Commons Select Committee last year and I made exactly this point that ethics underpin standards, some of which become mandated in regulation[5]. I think that is the way that ethics flows through to regulation.
Dr Ing Konstantinos Karachalios: I would make three very brief points. The first is that we should mandate human oversight of critical systems and of systems that influence public opinion and so on. I gave an example before. These should be mandated. We cannot say that just because there is a computer in the loop you can do whatever you want. This is what happens now and it is apparent that this does not work.
The second point is self-regulation. This is what we are doing. This should happen at the origin of system design and construction. The architects of the systems must start taking the contextual aspects of technology into account and not just speed to the market and functionality. The technology must protect our dignity by design, not as an afterthought. I think the time of innocence is over. We should say “enough is enough”.
The third point is that if you combine this, regulation would promote rather than hinder innovation, because a lot of these technologies would not take off because we would not trust them, and we would have reason not to trust them. If you regulate in such a way that you infuse trust and make it trustworthy, the technology will take off. It is to the benefit of everybody to have a kind of regulation that will help technology take off because people will trust it.
Lord St John of Bletso: I am a bit concerned that I might run short of time. I just looked at the ethics and legal and data capital and there were two approaches: the soft approach and the hard approach. Obviously we have the hard approach with the GDPR, and I was going to go on to the soft approach.
Dr Ing Konstantinos Karachalios: We are making the soft approach before the hard approach comes.
Lord Swinfen: Who is to draw up these standards, bearing in mind that the Civil Service is probably not in the forefront of artificial intelligence?
Professor Alan Winfield: I think it is primarily the role of professional bodies, of which the IEEE is one, and there are other standards bodies, such as the International Organization for Standardization and of course the British Standards Institution in the UK. These are the bodies that should be responsible for developing these standards, but, essentially, it is within the community, as Konstantinos said.
Dr Ing Konstantinos Karachalios: Everybody should do a job at a different level. You have a responsibility to mandate something. We have a responsibility to do a better job at the beginning. These are different types of standards and we should not confuse them. Ours are bottom up and voluntary, but they can be very powerful. Wi-Fi is voluntary, but everybody uses it because it is useful. Our ambition is to produce standards that will be used by the technical community, because they make sense and this will make your life much easier.
Viscount Ridley: Dr Karachalios, you said that good regulation can stimulate intervention, but you would surely concede that bad regulation can do the opposite. There is a moral hazard and an opportunity cost if you get it wrong and it could end up stifling innovation and preventing good things from happening.
Dr Ing Konstantinos Karachalios: Of course.
Q28 The Chairman: One final sentence from each of you, and this is probably a very unfair question: if there was one recommendation you would like to see the Committee make at the end of this inquiry, what would it be?
Dr Ing Konstantinos Karachalios: I speak in the country of Lord Byron and Winston Churchill and so on and coming here is very inspiring. I would say: give the kids digital literacy but do not confuse it with literacy per se. Give your kids literacy and give them the means to make sense of the world. Without this, digital literacy means nothing.
Professor Alan Winfield: I would go back to my plea for ethical governance. We need a body, a kind of AI watchdog that essentially is responsible for finding that balance, as you said, which I agree is not easy, but we need a body that is responsible for making those recommendations to government.
The Chairman: Do you mean to build on Dame Wendy Hall’s AI council or something of that sort?
Professor Alan Winfield: Indeed, yes.
The Chairman: Thank you very much indeed. We have had a very stimulating session. Clearly the IEEE is putting itself front and centre in this debate and I think that is extremely welcome. Thank you very much indeed.
Dr Ing Konstantinos Karachalios: Either as a Committee or personally, if you want to continue this discussion—and there are many things that we have opened up but we have not been able to explore in depth—we are available.
The Chairman: Thank you very much.
