23
Examination of witnesses
Dr Simon Beard, Dr Stephen Cave, Caroline Gorski and John Thornhill.
Q42 The Chair: Welcome to this evidence session of the Lords Select Committee on Risk Assessment and Risk Planning. In this session, we will be discussing the risks associated with technology. We have four witnesses: Dr Simon Beard, the academic programme manager and senior research adviser at the Centre for the Study of Existential Risk at Cambridge; Dr Stephen Cave, the executive director at Leverhulme Centre for the Future of Intelligence at Cambridge; Caroline Gorski, the director of R2 Data Labs at Rolls-Royce; and John Thornhill, the innovation editor at the Financial Times.
We will be taking a transcript of the meeting and publishing that on the Committee website. You will all have the opportunity to make corrections to that transcript where necessary. We expect to end the meeting at about 11.45 am, so please keep an eye on the time. To all of our witnesses, please do not feel it is necessary for each of you to answer every question. Some of you will wish to chip in on some of them and not on others, and that is fine.
I will begin with Mr Thornhill. You wrote an article last month in which you discussed the issue of our dependence on technology. What do you see as the risks of our growing dependence on technology?
John Thornhill: Good morning. It is a great pleasure to speak to this very distinguished Committee. Lord Rees, in his book On the Future, has outlined all of the main technological worries that we ought to be concerned about. That is the starting point for any discussion. The issue that I would focus on is the interconnectedness between them all and the sheer difficulty of knowing what the greatest technological risks are and how they can interact with each other. They can combine in ways that we cannot foresee or anticipate, and the nature of crises is that they expose risks that you had never seen before. We have seen during this pandemic that the offline and online worlds have, effectively, merged and that digitisation has accelerated.
To ask you a question, how could you function if you were cut off from your electronic devices or from the internet, and how could our societies function? We saw, during the financial crisis, when things blew up, that there were interconnections that no one had imagined. Financial models broke down because of connections that no one foresaw. We have seen the same during the pandemic. We have seen the same, to some extent, with Brexit. In that sense, I wonder whether science-fiction writers might be better guides than policy analysts for imagining some of the interconnections.
I will just throw out three. Imagine some of the low technologies: drone technologies, for example, combined with facial recognition technology. You could imagine swarms of drones with facial recognition technologies being unleashed on cities. The Iranians have clearly done that in Saudi Arabia already. We worry about quantum computing and how it is going to affect cryptography in the financial services industry. What happens if it was applied to a more generalised cyberattack on port infrastructure? We all know about the reliance that we have on Dover. What would happen if Dover was hacked? A more general issue would be some geopolitical events, and the one that I would question is China and Taiwan. What would happen if there were hostilities across the Taiwan Strait and, effectively, the supply of silicon chips around the world stopped? I have lots of questions and no answers, but I am delighted that you are all focusing on this.
The Chair: It is going to be our task, so far as we can, to try to come up with some answers as well. On the risks from our dependence on technology, does that dependence arise mostly out of risks of cyberattacks or other risks as well? For example, every bit of the technology that we rely on depends on electricity.
John Thornhill: All of these things are interconnected. The issue that I would most focus on is our vulnerability to cyberattack, because that can degrade so many other functions of our societies. I would initially focus on cyberattacks and the way that they could disrupt the interconnections between other sectors.
The Chair: Are the British Government adequately prepared to deal with the risk of cyberattacks?
John Thornhill: The honest answer is that I have absolutely no idea. Our defence capabilities in GCHQ have an extremely high reputation. A number of people have highlighted sometimes the randomness of attacks. This was in the documentary “The Perfect Weapon”. When you look at the WannaCry and NotPetya attacks, they took down, temporarily at least, Rosneft, the oil company, DLA Piper, an international law firm, Cadbury’s chocolate factory in Tasmania and various hospitals in America. It is almost the randomness of some of the connections that are made that makes them incredibly hard to mitigate.
Dr Simon Beard: I am going to sound a bit like a doom-monger, but you did ask an existential risk expert along. We are probably going to be talking mostly about computer technologies in this session, and there are good reasons for that. However, It is really important to note that we are not only becoming increasingly dependent on computer-based technologies. Other kinds of technological risk that we need to think about are, for instance, our growing dependence on agricultural technologies to guarantee the world’s food supply. One of the things that a lot of us have been very grateful for in this current crisis is that there has not been any disruption to the food supply or the food distribution network, but these are very heavily invested in technologies; they are sometimes computer based, sometimes biotech based and sometimes environmental technologies based. There are all sorts of risks in that system as well. These technologies have insulated against some risks, but we have also created some new risks in this way.
Another point to make is that one of the biggest technological risks that we currently face is a risk of the first Industrial Revolution: that all of these things use electricity, and our electricity generation is driving climate change. Technology is not just a new thing and is not just a computer-based thing.
Our cyber systems are not only at risk of attack either. We can have cyber vulnerabilities due to naturally occurring disasters, normal accidents and bugs in the system. It is a really good question and something that we definitely need to be concerned about, but it is really important not to narrow our vision on this one thing that sounds really scary.
Dr Stephen Cave: I completely agree with the points made by the other witnesses. John’s point about interconnectedness is such an important one, to which we will perhaps be able to return. I agree with Simon that it is important to take a big-picture view because technological dependence is as old as our species. All of the massive increases in population over the last century have been highly dependent on particular agricultural technologies. Of course, we are today dependent upon electricity.
The new and emerging digital technologies are creating new forms of dependency on the internet, even specific applications like Google, or on infrastructure like the cell phone network. There is one thing in particular that I want to highlight that has not been mentioned yet, which is the way in which AI and related technologies might exacerbate existing vulnerabilities. As we use AI to automate processes in the healthcare system, the energy grid and other critical systems, we might be making those systems more complex, more opaque and difficult to oversee.
The Chair: We have specific questions about AI in due course, and we will be able to come on to that when we ask those. Caroline, is there anything you want to say on this?
Caroline Gorski: To be perfectly honest, given the strategic importance of the work that Rolls-Royce does and the nature of the clients we work with, we are, of course, acutely aware of our responsibility to protect our products, our data and our operations specifically in regard to cybersecurity, but given the public nature of this conversation, I am afraid I am not able to comment further at this point.
Q43 Lord O'Shaughnessy: Welcome to our guests. You have started to set out what some of the biggest risks that we face are. I am sure we could spend a very interesting if not slightly doom-laden hour going through those, but the specific purpose of the Committee is to look at risk assessment and planning. My question draws down into that process. This is a question to all and any of you: what do you consider to be the most significant or urgent technological risks that are not properly accounted for in the UK’s current national risk assessment strategy?
Secondly, how might our approach to national risk planning, in the light of what you highlight as the significant or urgent challenges, change to make sure we are better prepared to mitigate these risks rather than simply responding to them if and when they occur? The reason for that theme coming out is that it was highlighted as a difference between perhaps how we do national security planning, which is much more mitigation focused, and other forms of planning, which are often mopping up once things have already gone wrong.
Dr Stephen Cave: Currently, the risk assessment focuses on two particular digital technologies—cyberattacks and disinformation—presumably because these are relatively distinct, discrete and novel, but the approach of highlighting a few specific uses of digital technology that are relatively distinct underestimates the very many ways in which these technologies will impact across the board and in which they will interrelate.
AI, for example, is not one single, distinct risk; it is a very general technology, more akin to steam or electricity. If we imagine, 200 years ago, a committee like this considering the risks of the steam engine, if it focused specifically on the technology itself, then policy might be about preventing engines from exploding, for example, or people getting their hands caught in the gears, but then it would miss the industrialisation of warfare, rapid urbanisation, the creation of a new working class, the rise of communism and fascism and so on.
These general multipurpose technologies can be transformational and disrupt many aspects of our lives, and therefore impact on many different risks. The current risk assessment framework fails to account for this impact of AI and related technologies on everything by trying to focus on a couple of specific, discrete areas.
To try to identify these discrete risks that can be put in the chart alongside, say, flooding is the wrong approach. I would suggest the question should be about which undesirable outcomes, like the collapse of critical infrastructure, civil unrest or failure of the democratic process, become more likely because of the development of technologies like AI and the transformations it will bring. An approach like that would help to address the interrelatedness issue that has already been mentioned. The focus currently on specific risks like cyberattacks fails to capture this interplay.
If we take, for example, the terrible events last week in the US with the storming of Capitol Hill, when we look at Trumpism we see how it has been fuelled by online disinformation. It is bad enough as it is but imagine what Trumpism might be like if a few million truck drivers had been replaced by driverless lorries, and they and all the attendant jobs in diners and so on had been made redundant. It is those kinds of interrelatedness between the impacts of digital technology that are not being captured by the current approach.
To move on to the second part of your question about how to address that, if we look at the current risk framework, it lumps things together that are really very different categories of things. Some of them are more like bad outcomes, such as the country being flooded or civil unrest. Some things are more like causes of those outcomes, such as cyberattacks or attacks on transport. Attacks on transport may or may not cause disruption and may or may not cause loss of life. This conflation of cause and effect and trying to identify very specific examples of each is failing to grasp the overall impact of these technologies.
An alternative approach might be to focus on a smaller number of specific undesirable outcomes, such as civil unrest or the collapse of critical infrastructure, and then think in a more nuanced way about the different factors that might contribute to them, which might be cyberattack and disinformation and the impact of automation-induced unemployment or increasing opacity and so on. This would allow us to consider technology across the board and all of the different effects that it might have.
Lord O'Shaughnessy: That is a really helpful answer. I love the analogy to the steam engine as well.
John Thornhill: If I could just build on that, this focus on disinformation is the one that I would also highlight. Donald Trump was compared with a distributed denial-of-service attack on democracy. The 57,000 tweets that he managed to tweet out during his presidency just absorbed so much attention and distracted everybody from every other issue.
That is really almost a beginning of the process that we are seeing with a lot of the AI technologies that Stephen has been talking about and focusing on. This is going to lead to a great multiplication of this. I had fun last year playing around with GPT-3, the language generation system produced by OpenAI, which is quite phenomenal in its ability to generate very plausible text. The amplification of disinformation is one really interesting and quite disturbing factor that we need to focus on.
In terms of the mitigation, I was fascinated to read about Sweden’s approach to this. I am sure that Lord Robertson knows a lot more about this than I do. They have a concept called total defence, which is something they developed in the 1940s and have now revived. They conduct national resilience exercises and include 15 national agencies, the parliament, local government, companies and the central bank. They try to model what would happen in extreme circumstances.
They conducted one last year and, from what I understand, the conclusions of this were threefold. One is that they were very much concerned by the whole issue of disinformation and the extreme difficulty of communicating with the population in a clear way in a time of a national emergency. I believe they are creating a psychological defence agency specifically to focus on this issue of trying to have clear channels of communication with people.
The second was that, in peacetime, you normally have the military supporting civil society. In times of emergency, that is clearly switched round, and civil society has to support the military. How do you ensure that that switchover is as rapid and seamless as possible?
The third part focuses on the unlikely partnerships that need to develop in a time of emergency. One that they focused on was the retail logistics chain and the military logistics chain, and how you ensure that your supply of food is maintained in a time of national emergency, when critical infrastructure goes down. To what extent does the military logistics structure have to kick in to support that?
The final point that they made, which was very important, was that it absolutely has to be the top-level leadership in all of these organisations who participate in these exercises testing national resilience—it is no good sending the third deputy responsible for these exercises—so that the top leadership are absolutely aware of what the issues are and that they are the ones who are prioritising the response.
Caroline Gorski: I wanted to build on the comments that John made. I was particularly taken by that description of those unlikely partnerships that need to potentially emerge in those circumstances. We saw that in the work that Rolls-Royce was a contributor towards in regard to the ventilator challenge back in the early part of the pandemic. We can see that from a corporate perspective, but it expands out beyond the corporate domain.
One of the things that I am genuinely interested in here is the renewed social contract that we, collectively, may need to consider as a society, particularly in regard to how we help our children, the next generation, to be more resilient to some of the threats that are being articulated here. That might imply that we should be thinking about educational elements as part of our risk planning strategy, all the way into curricula, which would be a fascinating place to have more focus and more attention, potentially.
The second point I wanted to make—here I am specifically constraining my comments to artificial intelligence risk, given that is my area of expertise—is that, in terms of developing optimal approaches for addressing AI risk or emerging technology risks more broadly, these need significant international co-operation. They emerge from interconnected, geographically distributed networks that are completely location-agnostic. They cannot be solved on a nation state-by-nation state basis.
Dr Simon Beard: It is worth pointing out that there are a wide variety of risk assessment and risk management institutions in the UK, and a lot of risk assessment goes on at the local government level. However, the UK policy context is certainly very heavily influenced by this national security risk assessment and the shorter, published national risk register. A really important word there is “security”; it is a security risk assessment. When you look into the methodology, while it has many advantages, it is heavily influenced by a security background and a security approach to risk.
If I could just briefly point out a couple of features of this.
It considers risks only in the immediate future. It is really looking at the things that we are expecting to happen now. Technological risks are risks that we need to act on now, but they are not going to reach their full extent for years to come.
It thinks about risks as events and as these exogenous, external things that might happen. If you look at them, they are all events. They are not really what I would consider a risk.[1] They are just bad things that might happen.
It disaggregates across many different things. One of the really important things about the national security risk assessment is that it gives a lot of attention to attacks, a reasonable amount of attention to accidents and very little attention to systemic risks. A lot more attention needs to be given to systemic risks within this context.
The final thing that really reflects its security background is, of course, the fact that it is a classified document, which means that it is not open to the same amount of scrutiny or peer review that I would expect my risk assessment work to receive. The UK could benefit a lot from opening this up and having less of a security mindset to risk, because these are not just national risks. The risks that we really need to worry about are global risks in technology and in many other things as well.
Q44 Lord Robertson of Port Ellen: A lot of our questions, inevitably, will be crossing different territories. We are going to cover stuff, inevitably, on the same basis. To what extent can we in the UK determine our risk mitigation policy? John Thornhill has mentioned the Swedish example. Sweden has done that specifically, but given the range of risks that are now involved, how much can we do to mitigate it in the UK? How much of it is going to depend on a degree of international action that, so far, has not manifested itself?
Dr Stephen Cave: First, there is a huge amount that can be done domestically, for example in preparing our workforce for potential economic disruption, for making our public services resilient and so on. The bulk of what needs to be done can be done domestically. At the same time, as others have pointed out, many of the risks do, of course, cross borders: risks arising from climate change or major disruption of war elsewhere in the world, and risks arising from digital technologies too, such as taming big tech. Coming up with internationally agreed standards for what we mean by “responsible AI” would be enormously beneficial and make our risk management in this country much easier, as well as facilitating trade, innovation and so on. The international dimension is extremely important but, at the same time, it should not stop us from getting on with the things that can be done nationally.
Dr Simon Beard: It is really easy to see this as, “The UK is not in a position to manage our own risks. We need to reach out internationally.” It looks like a very negative thing. There is definitely another way of looking at this. The UK is also in a position to deal with risks that everyone is facing.
That can feel like a burden: “Why should we do this?” However, certainly, one of the things that we have noticed at the Centre for the Study of Existential Risk is that the level of global interest in global risks is huge amongst all sorts of people. If the UK is a leader in tackling global risk, that is one of the ways that we can define a place for ourselves on the global stage. That attracts talent and political and cultural capital. It is a good thing to do in ethical terms, but in terms of where the UK currently is and where we want to get to, thinking about global risk management and about what we can do to contribute to that is a very sensible thing for the UK to be thinking about and investing in, for our own benefit as well as for everyone else’s. There are opportunities here as well as costs.
Caroline Gorski: I would chime in here to say that there is a very powerful contribution and part of the conversation that can be offered by those global commercial entities like Rolls-Royce, for example, but not only in my sector. We in the UK have extraordinary representation from global financial services. Those organisations are operating in a global risk context, so the power of the conversation that could support the ambition that Simon has just articulated could really benefit from including those voices and experiences in the conversation.
John Thornhill: I completely agree with my colleagues on the panel that international co-operation might be desirable and might be needed, but whenever I write in the FT that we need new international institutions, I get a lot of reader comments like, “Have you not noticed that multilateralism is dead and that this is not going to happen?” That absolutely should not stop us from doing whatever we can in the UK to address these issues, and something like the UK online harms Bill is a very interesting attempt to do exactly what our own Government can do to address some of these problems, even if they would be more easily solved in a global agreement on the subjects.
Lord Robertson of Port Ellen: Is Brexit going to affect any of these thoughts?
Dr Simon Beard: It depends on what we do with it. Brexit has opportunities and costs and so on. For instance, looking at technological regulation, the EU has been a standard setter, and that has been a really powerful thing for its influence around the world. People have to build up to EU standards. If we take Brexit and say, “We are just going to completely ignore what the EU is doing. We are going to go our own way and try to outcompete by lowering our standards”, that is not going to benefit us or anyone else very much because people are going to want to move within EU standards anyway.
If we try to do even better than they have by introducing legislation that they are going to want to copy and make use of because we can now make decisions more quickly than they can and draw upon some of the really great expertise that the UK has in relevant fields, then Brexit gives us opportunities to shape global regulation. EU companies are going to want to work to UK standards, as well as the other way around, because there is a lot of cross-border trade. We could use this as a real opportunity to build the UK’s influence in the world. I am not convinced that is what is going to happen, so I am not going to say that Brexit will be a good or a bad thing, but the opportunity is there.
Lord Robertson of Port Ellen: I was going to ask how likely that was.
The Chair: Before this becomes another conversation about Brexit, we will move on to Lord Clement-Jones.
Q45 Lord Clement-Jones: John Thornhill, how effective are current UK security governance and regulatory structures at managing technological advances and their associated risks? To some extent, we have covered this ground, but you made a comment in your article: “Our governance structures remain stuck in the analogue age. We either need to reimagine their scope or invent new ones.” How could the risk assessment process be used to inform the governance of technological innovation?
John Thornhill: We should imagine that the online and offline worlds have effectively merged and all of our institutions are still overwhelmingly focused on the offline world, so there is a lot of scope to reimagine existing institutions and to figure out how they can respond to particular threats.
To give you one very small but human example, I was talking to a former Labour MP who had been the target of anti-Semitic abuse. She had received a whole load of death threats, both through the post and online. The police were very good at responding to the letters that had been shoved through her post box, because they would come in and fingerprint the letters and try to track down the people who sent them. There were mechanisms and mentalities that were designed to respond to that, but she said that they had absolutely no idea about how to respond to an online threat, which she took more seriously. We need to build expertise into our institutions to figure out what the specific online threats are, how they merge with offline threats and how they can jointly respond.
It is really a question of reimagining our existing institutions and perhaps figuring out whether we need specific institutions to address some of the issues that we have been talking about. One of the ones that there has been quite a lot of debate about in the US has been whether they should create the equivalent of a Food and Drug Administration for pre-authorising algorithms that are used in very sensitive areas such as healthcare or judicial sentencing. It is something that you have focused on in your reports in the House of Lords on AI. There are some specific institutions like that that might be worth considering, but the overwhelming focus should be on reimagining the existing institutions that we have.
Lord Clement-Jones: Is the online harms approach, with a regulator coming down the track, the first swallow, in a sense, on this?
John Thornhill: Yes. It contains some very interesting ideas for how you can focus on specific online harms and take them as seriously as some of the offline harms that we have, but it clearly needs to be joined up in how it interacts with other institutions.
Lord Clement-Jones: Of course, one of the things is that the Government are potentially creating risks in their own use of technology.
John Thornhill: Yes. Who guards the guardians in that sense? There is a very interesting book that has come out, which I have been reading, by Ron Deibert, who runs the Citizen Lab at the Munk School in the University of Toronto. He argues that there has been an unholy alliance between big government and big tech, as it were. After 9/11, the National Security Agency thought it was very helpful to have surveillance capitalism operating to get an amazing insight into the behaviour of individuals. We need more civil society constraints and scrutiny of what government and private companies are doing with data. In that sense, we need new institutions to scrutinise government in this area as well.
Dr Stephen Cave: My main worry is about the siloisation of technological risks and, as I say, the difficulty the current system has in accounting for interrelationships and general widespread effects. At the moment, as you know, what gets counted as a risk has to meet a certain threshold. It has to be something that is going to impact on the scale of coastal flooding and, as I say, it is a bit of a mishmash of causes and effects; it is not just apples and oranges but more like apples and tractors and malicious cyberattacks.
Pulling apart the outcomes we want to avoid and then thinking more subtly about the ways in which new technologies might contribute to them would enable us much better to grapple with the impact of these technologies. At the moment, once a risk has been identified, very sensibly it is assigned to a government department to take responsibility for. That accountability and responsibility is crucial, but can, of course, easily contribute to siloisation.
If, instead, there are certain undesirable end states like collapse of critical infrastructure or civil unrest, all government departments might be able to contribute or advise on ways in which those end states might come about, drawing on their particular expertise. We need government departments that are expert in understanding the impacts of digital technology, but they should then be liaising with all other government departments on their respective risks, for example. That requires a somewhat more nuanced matrix-like approach to risk assessment.
Lord Clement-Jones: Does it also require a more centralised approach to compliance and so on?
Dr Stephen Cave: Perhaps it does. The Cabinet Office already has oversight. I would not have thought that would need to be strengthened. It is largely a co-ordination problem in identifying the specific bad outcomes that should be the focus of attention, but the conversation could be much broader about what kind of factors might contribute to bringing those outcomes about and the ways in which they interrelate. It might require more co-ordination but the structure for that already exists.
Dr Simon Beard: I will just offer two quick and quite specific instances where risk governance has failed recently or where more needs to be done. We recently did an analysis of the MoD’s approach to risk assessment for new technologies and found that new technologies are being risk-assessed at the procurement stage, which is very welcome, but are only being risk-assessed under current conditions. There is no account being made of how these technologies will interface with technologies in 10, 20 or 30 years’ time, during the life course of whatever is being bought. There really is a need for more foresight and more critical engagement with this, because that is where the risk is going to come. We need to be thinking in the future.
This need for joined-up thinking is so crucial. One really clear example of this was the national biosecurity strategy, which was a centralised document that looked across all the different biosecurity threats that we faced—many different kinds of natural pathogens but also biotechnology and artificial pathogens—and they really recommended a common vulnerabilities approach of building up our general ability to respond to biosecurity threats by stockpiling personal protective equipment, investing in our national laboratory infrastructure and so on.
However, government policy was really led by the national security risk assessment framework, which, as I said, is very disaggregated and chose to select influenza pandemics as public enemy number one and focused very much on those, to the exclusion of other kinds of biological risk. I am sure that I do not need to tell everyone how that worked out. It is really important to do this well across the board and not to let any one agenda or one idea of what these risks are take hold.
Caroline Gorski: Unlike my distinguished colleagues on the panel, I am representing a slightly different part of the experience here, in that I am speaking really to represent the safety-critical industrial sector. Rolls-Royce operates in highly regulated markets and, as a safety-critical industrial player, we believe regulation is essential to help to ensure and assure product and operational safety. In that relationship, we understand our role to be as a constructive partner in that conversation. We work with our regulators to demonstrate our compliance to the regulatory regime, but we also work to support them in developing new regulatory instruments in response to technologies as they develop and as markets change.
Part of where we need to get to with this conversation is a greater degree of maturity in the relationship between the regulators and the regulated sectors in some of these more emerging technology spaces, because until those organisations are also able to understand the positive bilateral relationships that they have with their regulatory regimes, we are going to continue to find that this is a troublesome question.
Lord Clement-Jones: Of course, sandboxing is now becoming quite a useful instrument, is it not?
Q46 Lord Browne of Ladyton: Thank you to our witnesses for their fascinating contributions so far. I draw attention to my recorded interest: I am a visiting researcher at the Centre for the Study of Existential Risk. I have not been visiting there very much recently; none of us has.
I want to ask a very specific question about the apparent ineffectiveness of our current security structures in the context of this specific question about the effectiveness of structures, and the implications of that ineffectiveness for governance and regulatory structures. I am sorry to do this again to you, Mr Thornhill, but we have to start with you, because your recent article influenced some of these questions. In that recent article, you referred to the events at FireEye. There are a number of different names given in what has been written about it: the Sunburst hack or the SolarWinds Orion hack. My understanding is that that was targeted at organisations that had world-class cybersecurity teams. FireEye itself is reputed to have had cybersecurity that was on a par with the world’s top defence and financial businesses.
Secondly, the conclusion was that this was almost certainly the work of a nation, so that feeds into this whole international ability to regulate. This generates significant implications for what we could do with regulation or governance in the face of this level of ineffectiveness against a hack of this nature.
Mr Thornhill, you might want to respond to that first, and others may also have something to say.
John Thornhill: I would make several points. One is that, in this world that we are moving into, when it comes to conflict, there is clearly no on/off switch; it is almost a permanent state. Whether we are at war with another country is a moot point, in the sense that we are all jostling with everyone else for economic, strategic advantage. What is so disturbing about the attacks on FireEye—and we have seen it in the U—is that these are rolling attacks; it is a permanent state. We should bear in mind that it is a new world and we have to deal with it permanently.
The second point is the asymmetry involved in a lot of these attacks. Some of the most effective attacks have been conducted by North Korea or Iran, which are some of the least digitised economies. I remember that, in the film “The Perfect Weapon”, when the US was considering retaliation against North Korea for the attacks that were launched against Sony, it was pointed out that North Korea had only 28 websites in operation. That is the other dimension of this. Our understanding of hybrid war and constant struggle needs to be factored into our thinking on how we deal with all of these issues.
Q47 Viscount Thurso: Can I come first, please, to Caroline Gorski, to talk about the Aletheia Framework? What is Rolls-Royce attempting to achieve with the Aletheia Framework and where does it have its optimum application?
Caroline Gorski: Before I answer your question, I will briefly describe what it is, in case members of the panel are not familiar. In December 2020, Rolls-Royce made its AI ethics and trustworthiness toolkit, which we call the Aletheia Framework, freely available under Creative Commons licence. It has been two years in development. It was built based on more than a decade of experience of operating artificial intelligences in our engine health monitoring capability. That ensures that safety-critical products—the engines that are in operation on aircraft around the world—are able to be maintained effectively by using predictive self-learning algorithms to continuously monitor and respond to near-real-time events.
We have been working in this space of advanced data analytics for more than 30 years, and, as I say, using those artificial intelligences in that real-time engine health monitoring since about 1999. We built the Aletheia Framework around an internal assurance challenge. Our AIs that operate in engine health monitoring fall within the regulatory regime of EASA, the European aerospace safety authority.
When we started thinking about developing artificial intelligences for further processes—in this particular instance, we were looking at the robot inspection of components as they come out of our manufacturing or servicing environments—we recognised that these were critical components; they play a role in a safety-critical industrial context. We needed to make sure that we could trust the outputs of those artificial intelligence robot inspectors to at least the same degree of verification as we could trust human inspectors, whose work was being augmented and supported by those robot inspectors.
That was the basis for coming up with the framework. Having developed it internally, we took it out for a peer review process. We took it to other safety-critical industries, to technology companies and to academics. We did that in the spirit of saying, “We have made this. We are sure you have something else that would be just as good, if not better. We will share ours with you, you share yours with us, and we will all learn together.” Rather to our surprise—I was genuinely quite startled—what happened when we went through that peer review phase was almost everybody said, “This is the first time we have ever seen something like this”, which proceduralises the ability to develop assurance and trustworthiness in artificial intelligences as they are deployed in safety-critical contexts.
Because we had discovered that this appeared to be something that had not been done before, we came back to Rolls-Royce, and my chairman and CEO very rightly supported the decision that we should make the framework available to everybody. For a start, our own industry needs to move forward in this regard, but it has application outside our own space.
By publishing that work openly and freely for use, we are hoping to demonstrate how other organisations that operate in safety-critical settings might be able to develop their own AI ethics and trustworthiness frameworks or, indeed, adopt the Aletheia Framework, if they find it helpful. It also starts a conversation about how these frameworks more generally can be used to create artificial intelligences that people can trust.
Viscount Thurso: You used the words “trust” and “trustworthy” multiple times throughout that very full answer. Indeed, at the launch of it, that was one of the key things that you said: that you have had to challenge yourselves to ensure that it is the right thing to do and that it is trustworthy. A recent book I read about the 2016 American election highlighted a group of humans who were doing the censorship role on fake news on Facebook. They were accused of leaning too far against the right. Mark Zuckerberg replaced it with an algorithm, whereupon it promptly went in the opposite direction and flooded us with fake news from the other side. How do you, therefore, ensure trustworthiness and balance?
Caroline Gorski: Thank you for the question. That is a really important distinction. There is a tripartite set of issues that we need to talk about when we talk about AI ethics. So far, we talk mostly about only one of them.
The first, which we talk about a lot, is fairness—the question of how we can ensure that the differential outcomes that come from algorithms that are presenting options to human beings are not reinforcing existing prejudicial or biased positions. We talk an awful lot about fairness in AI ethics, but we do not necessarily talk about the other two things, the first of which is trustworthiness and the second of which is safety.
From our perspective, trustworthiness speaks to the ability of the developers of an AI system to offer assurances that their algorithm behaves as they expect it to behave across its lifetime. You do that by managing the inputs to the algorithm, understanding what the expected outputs are and continuously monitoring the algorithm to make sure that it is not suffering from what we would call algorithmic drift. This is where it adopts, during its self-learning processes, degrees of mutation, which mean that its outputs are not as would be expected.
It is really important to note that assuring trustworthiness—knowing that your algorithm behaves as you would expect it to—does not necessarily assure fairness. You can have a trustworthy but reliably nefarious AI, just as you can have a trustworthy, reliably benign AI. You can have an AI that does appalling things in a way that is exactly as it is programmed to do, so you must have both fairness and trustworthiness.
The third element of that tripartite position is safety. I would hark back here to the point that John made earlier around harms. We are not yet at the level of maturity in the general conversation around artificial intelligence to have proper questions about safety, and I think we need to get there fast. A conversation around safety in AI asks ethical questions about how the outcomes of any algorithm might negatively impact on the physical reality of people, or, indeed, the physical reality of the planet. That should extend beyond just physical questions into questions of psychological harm, political harm and democratic and economic harm. Therefore, as a result, the question of AI safety should be as applicable to YouTube as it is to Tesla.
Viscount Thurso: That is a massive amount to digest. Thank you very much.
Q48 Baroness McGregor-Smith: Caroline, I read through the framework yesterday. I am particularly interested in knowing whether you feel business can mitigate all the risks in the framework itself or what government intervention or support it needs. Secondly, what are the learnings within the framework for government as it looks to deal with technology risks?
Caroline Gorski: On support, in Rolls-Royce we published the Aletheia Framework to trigger a conversation. We wished to take the peer review conversations that we have had over the last 12 months, which were in camera, and create the opportunity for those to happen in the public domain. We are already in conversation with a number of governmental bodies and entities, such as the Office for AI, the Centre for Data Ethics and Innovation, the Alan Turing Institute and Digital Catapult. I could go on and on; there are many.
We are hopeful that, over the first part of this year, that conversation will really start to gather some traction. It will not be one that we drive. We want to be a member of it, but it is not Rolls-Royce’s job to necessarily do anything more than simply be the catalyst for that conversation happening. It probably needs to happen on a much bigger scale. It also probably needs to happen on a more multinational scale and not simply as a UK conversation. To that end, we are also in discussions with UNESCO, which is developing a global AI risk framework, which I believe it will be publishing towards the end of 2021.
On what government can take from the experience, as I said before, Rolls-Royce is an organisation that operates in highly regulated markets. I would encourage the Government in the UK to do more to ensure that organisations like ours, which work in sectors where safety criticality is fundamental to the well-being of our customers, are more active, visible and audible in the conversation. Thus far, the industrial sector has perhaps not had that degree of participation in the discussion around artificial intelligence, which is often dominated either by consumer AI questions, by big technology or, indeed, by medtech. I would encourage the Government to lean into their safety-critical industrial sector and call on it to participate more actively in those conversations.
Baroness McGregor-Smith: What can the public learn from your framework? What can an average individual in the UK take from it? I went through it; it is long and complicated. What would your message be to them?
Caroline Gorski: That is a really good question, and you are right to call out that this absolutely was not a framework written with members of the public in mind; it is to manage industrial artificial intelligence developments and would need significant work to make it something that the public could easily engage with and would really get benefit from.
Interestingly, though, when we start talking about it, we tend to use consumer examples to explain how it works. One of the examples we give is to think about how you would go about selecting a new energy supplier. What would you do? You would start by asking, “How much did I pay for my energy over the last year? How much is that a month? If I consume the same amount of energy, would my tariff be any better or any worse with this energy supplier?” What you are doing there is an independent check of the algorithm. You are saying, “My experience in the real world is this. The algorithm is telling me that. Does that match up? Is that the same?”
The five checks that sit in the middle of the framework, which are currently expressed in very technical language, could be rendered in a way which would help members of the public to think, “How do I check that this algorithm is trustworthy and safe and is going to deliver the experience that I expect or want?” More work is needed to get to the point where it could be used in that way. I completely acknowledge that.
Q49 Lord Triesman: Good morning, everybody. It has been fascinating so far. Some of the questions that I want to ask follow on closely from the last question that Baroness McGregor-Smith asked. It has always seemed to me that one of the most significant risks we run is ignorance, if we just do not know what is being talked about. Dr Cave, what level of understanding of artificial intelligence do we think we have in this country and in government? What needs to be done to better educate people so that they can ask the appropriate questions?
Let me just lead on from that to say that, although I quite frequently hear people saying that we need to know a lot more about all sorts of technologies and bunches of technologies such as artificial intelligence, it is much harder to find out where and how we could do that education, and who might be responsible. I would be very grateful for your thoughts on that as well.
Dr Stephen Cave: You will not be surprised to hear that understanding of AI is very mixed across society. We conducted a survey, together with the BBC, in which we asked people how they would describe AI to a friend. About 40% gave an answer that we might consider fairly accurate, inasmuch as it mentioned computers and some kind of replication of aspects of human cognition, such as computers making decisions, computers learning and computers thinking. About 25% said robots or something related, which is of course not the same thing. About 12% of a sample size of around 1,000 gave very emotive responses, along the lines of, “Computers doing things instead of people. I hate it”, or just, “Scary robots” and so on.
We might take some satisfaction from the 40% who gave a fairly accurate answer, but giving a conceptually accurate answer along the lines of “computers thinking” does not mean that people have an understanding of how contemporary technologies that we now call AI work and what their appropriate uses and limitations are.
This level of confusion is entirely understandable. For most of my life, AI was an entirely fictional construct and a very popular one. I grew up watching “Star Wars” and “Battlestar Galactica”. That is what AI meant. Now the term AI has quite suddenly moved from the silver screen and into our lives, but the real-world technology that we call AI is very different from the highly anthropomorphic visions of metal men and women that we grew up with. Contemporary AI is really much closer to what, until recently, we called “big data” than to “The Terminator”, but, unfortunately, it is the latter that people tend to think about. Of course, the underlying technology and the advances in hardware and software that have enabled these breakthroughs are extremely complex and are the preserve of experts.
We have a real mismatch between three things: the popular conception of AI that we have all grown up with; the deceptively simple apps that we all engage with every day; and the very complex and rapidly changing underlying technology.
As you said, Lord Triesman, ignorance brings with it its own risks, and these misconceptions feed very directly into discussions about ethical AI and risks. Generally speaking, if people are too trusting of AI—if they over-trust—they will be exposed to risks such as manipulation, privacy violation and loss of autonomy at an individual level and, at a societal level, the kinds of impacts of widespread disinformation that we have seen recently.
On the other hand, if people are too fearful, that also comes with real risks, because many of these systems will be immensely beneficial, revolutionising medical diagnosis and so on. Think of the tracing apps used in Covid. They would not necessarily deserve the term “AI”, but thinking of data-driven technologies broadly speaking, their success depends upon uptake, which depends upon trust, a theme that we mentioned earlier.
Some of the work in my centre in Cambridge has focused on looking at ways in which the misrepresentation of AI exacerbates existing injustices and social division, such as excluding people of colour, women and so on. It is a major ethical issue.
It is not possible to put any kind of easy number on how many people trust or do not trust AI, because the answers they give to that kind of question are very context-dependent. It varies enormously as to what people think AI is, which impacts on the meaningfulness of these answers. We can generalise and say that understanding of contemporary AI technologies is shallow and trust in them is very fragile. People might begin by being well disposed to trying a new technology that promises to improve this or that, but they are quick to blame it when it goes wrong. We saw that with the A-level algorithm used last year, which people were then very quick to blame when it produced results perceived to be unfair.
Addressing public understanding is enormously complex. The trouble with the public is that there are so many of them. Reaching all 60 million-plus people is extremely difficult and requires many different approaches. It is great that digital literacy has been mainstreamed in schools. It would be great to see it also throughout all professional and further education, through to people who are no longer in the workforce and retirees. Digital literacy has so many benefits in helping people to understand the opportunities and limitations of these technologies, but also to adapt to the kinds of impacts that these technologies are likely to have, and in giving us the kind of workforce that will help us to lead in developing these technologies. The importance of digital literacy across the board cannot be overstated.
A final point is that the onus is not just on individuals to try to understand these technologies but on the creators of these technologies to make them understandable. The onus should also be on those developing and deploying these technologies, whether in the private or public sector, to make as much information about how they are working, what their underlying technologies are and what is being done with data as accessible as possible to ordinary people.
Lord Triesman: I can see how, in a variety of courses at the higher education level, you could introduce people to the kinds of concepts that Stephen was just describing. But journalists are also educators, whether they would choose to be or not. The FT is possibly one of the more educational newspapers. In what ways can we try to make sure that people do not fall through the gap, which might mean that they do not understand the benefits or the risks and just see it all as a kind of Frankenstein endeavour that they will never understand and probably should never be troubled with?
Dr Stephen Cave: It is very hard to stop people falling through the gaps. The approach has to be so multipronged, including supporting education in the workforce. But not everyone is in the workforce, so it has to support education also for those who are currently unemployed, as well as for retirees and so on. Different parts of the system will deliver this education in these different contexts.
On working with the media, I am glad that you mentioned journalists. John does fantastic work at the FT, which has resisted the temptation to use pictures of “The Terminator”, unlike almost every other mainstream media source. We have done a lot of work with the BBC in just spreading awareness about how much these kinds of images matter and how much they can distort public understanding, with grievous consequences.
We are just now concluding a study that is not yet published. We are getting the results from of a study on representations not of the algorithms themselves but of scientists and engineers. As you know, there is an enormous deficit of women in the AI sector. It is currently about 20% women and 80% men. Representations of AI scientists in mainstream media run at about 10% women and 90% men, and it is well established that these kinds of perceptions influence the career choices of young people. Just making media organisations aware of these kinds of impacts will, I hope, help to instigate change.
John Thornhill: First of all, I want to amplify one of Stephen’s points: there is a very big risk of not using these technologies. There is a danger that politicians end up blaming AI, as they used to blame Brussels, for bad decisions that they do not like. Politicians still have agency in the way that they still had sovereignty when we were members of the EU, and so it is very easy to say, “The algorithm was wrong”, and then you get crowds on the streets. It was not just the A-level results in this country; it was also at Stanford, where an algorithm was blamed for how members of hospital staff were prioritised for their vaccinations against Covid.
Secondly, when it comes to the media, a lot of people would argue that it is obvious that AI expertise is unanimous on a lot of these important issues, but it really is not. There is a very broad spectrum of opinion on AI. A number of very interesting books on this subject have come out. If you looked at the existential risks posed by AI, you would have Stuart Russell from Berkeley university at one end of that spectrum, and Rodney Brooks at the other. They take very different views about that as an issue, for example. As a journalist trying to report this, there is not one definitive answer but a spectrum of views, which rather gets lost in the debate.
The final point I would like to make is on public education. I am a board director at the Ada Lovelace Institute, which is a relatively new institution focusing on data and AI. It has been conducting some fascinating biometrics councils on facial recognition technology, where it invites, say, 60 members of the public to debate the use, application and ethics of facial recognition technology. It has been amazing to watch the evolution of a debate when people are presented with the use of technology in context.
Most people will be instinctively opposed to the use of facial recognition technology. When it is applied in general as an indiscriminate dragnet on a Cardiff street, they are right to oppose it. When they are then presented with other scenarios in which it is used in very specific circumstances to identify criminals in a crowd who have murdered someone, they accept that there are legitimate uses. The uses of technology very much depend on the context in which they are used and the ways in which we communicate and explain the ways in which they are being used. There is an enormous job to do, by all public authorities that use AI systems, to explain the ways they are being used and, to get to Caroline’s point, to ensure that they are seen as trustworthy institutions deploying this technology responsibly.
Q50 Lord Rees of Ludlow: Looking somewhat ahead in AI, is there going to be an arms race between the cybersecurity people and the cyberattackers, both aided by AI? Is the widespread expertise in AI going to aggravate the problem of cyberattacks?
Dr Simon Beard: There has been an arms race between what we often call black-hat and white-hat hackers—we use very combative language to describe these people—going on for 30 or 40 years. They will use whatever the current technology is to expose, exploit or protect from many different kinds of threat. We are already seeing many different kinds of AI used in this.
One of the big things that we are seeing now, which is really important, is that it is often said that the weakest point in our cybersecurity, which is a very vulnerable system to begin with, is the humans. One of the things that AI is increasingly able to do is to take advantage of that human element by deceiving people or by finding out information about them that they think no one has. In all sorts of different ways, the AI-human interface is becoming more and more the wild west of cybersecurity, and that will carry on being something that is fought over.
This means that cybersecurity, which came out of something that was just so hopelessly geeky that no one was ever going to understand it, is becoming something that people are having to engage with in their day-to-day lives, because attempts are being made to take their identity or to deceive them in various ways. It is a very scary thing for most people.
Can I also just say one really quick thing about AI education? There is one point that has not been made yet. The question assumes that there is a lot of information and expertise and that it is just a matter of getting it out there. We need to understand educating people about AI. It is like we are saying, “Let us educate everyone about quantum mechanics”, but it is 1930, and we do not even really understand the quantum mechanics yet. Some scientists have some really good ideas and we are making good progress, but it is a very hard subject and so much is not known. We need to educate and to get past some of these really difficult archetypes that dominate the debate, because it is very scary. But we also need to recognise that the science is nowhere near settled, and we need to have a good dose of humility in how we discuss and talk about AI at the highest, most educated levels, because it is very hard.
Q51 Lord Willetts: One lesson from this fascinating session is that we clearly need rigorous horizon-scanning. If we are to understand the systemic risks from emerging technologies, rigorous scanning of what these technologies are and what they might be capable of is important. Some of this is done in other parts of government but should more of it be done, and how can the civil industrial side of scanning of technologies be linked in to the risk assessment side? Perhaps Dr Beard might like to comment on that first.
Dr Simon Beard: This is really important. We talk to a variety of civil servants and people across government, and it is really important to note that there is a lot of appetite for doing more of this. The argument is being heard but it does seem like practices are not keeping up with that and the need for futures thinking. We will send in some written evidence to give more technical guidance on the methodologies for horizon-scanning that we think are most suitable for risks, so I will not cover those in too much detail.
I want to say a little bit about what horizon-scanning, foresight and all of these things are, because it is easy to get mislaid by some of the hype. There is no magic here. There is evidence that some people who are trained in these things—what we call superforecasters—can make very good predictions going forward about 12 months. There is no real evidence that anyone is better than anyone else on their own beyond that. When we talk about horizon-scanning and foresight in broader terms, we are really talking about methods for how to bring together diverse opinions and views, and how to find the people who have seen the future already, perhaps because they are already living it or they have made the connection between things that have happened before and what is happening now and can see how this is all going to work out.
At the heart of horizon-scanning is getting together a diverse group of people with very different epistemic positions and knowledge bases, getting them to share their knowledge and experience in a way that makes sure that everyone can see what everyone else can see, and then using a structured process to bring them together into common judgments that can be used to guide policy. For instance, for us, a real concern is trying to pick the priorities—the most important concerns out of all of these—that respect that diversity.
There are many ways that foresight can fail. If everyone feels that there are some experts and they need to go along with them, this is not going to work. If people think that they cannot make any change in the system and that this is all a pointless exercise, it will not work. People need to be emboldened and protected.That is going to make it hard for government to do this well, so please make strong recommendations about horizon-scanning and foresight. We need that, but we need to think a lot about how it is done. My number one recommendation for making sure that it is done right is to make sure that it is done in a way that is transparent; make sure that people can see what that process was, how it was applied and how that influenced the decisions that were taken, so that, if there were people who really should have been in that room but were not included, they can recognise that and say so, and be included next time. The strength of this process is the diversity of perception that you are able to access when you apply this stuff well. It is not about expertise, personal brilliance or newfound methods. You should definitely emphasise that when discussing this; it is really important.
Lord Willetts: I am sure that diversity and transparency are very important. I just wondered if John Thornhill wanted to comment on the issue of how you can do foresight exercises well, both on technology and risk?
John Thornhill: I am a member of a technology council for a consultancy. It brings together experts from lots of different fields, whether it is robotics, quantum computing, AI or nanotechnology. The challenge is to try to forecast which of these technologies is going to have the biggest impact and in what timeframe. Very rapidly, these discussions break down because what is fascinating is that, when people start talking about the convergence or the combination of these two technologies, all of those calculations and forecasts are scrambled.
As Stephen was saying earlier, AI in particular is a general-purpose technology. It is an empowering, amplifying technology that is going to amplify the power of all these individual technologies, so it is extraordinarily difficult to forecast in a narrow sense. I absolutely echo what Simon is saying: it is the people who can think laterally about the connections that can be made between different technologies who are really interesting. In the same way that Uber and Airbnb can create huge new businesses off the basis of very simple technological innovations that anyone could have applied, we need to find people who can think about the connections that are really quite obvious but that people who are at the forefront of this technology are not necessarily thinking about.
Lord Willetts: You began this session by warning us about how important it was to spot the interconnectedness of things as a driver of risk. As we get to the end, you are warning us that that is really rather hard to do.
The Chair: You also began the session, Mr Thornhill, by saying that possibly the challenge was a lack of imagination and that we ought to be questioning science-fiction writers. We are going to be doing that, and it would help us if you could write to us in due course with some suggested questions that we should ask of science-fiction writers.
Q52 Lord Mair: This has been a very interesting session. Can I ask each of you, very briefly, to suggest one single policy recommendation that our committee should make to the Government?
Caroline Gorski: Very briefly, I would suggest broadening the scope of the conversation. I am chiming there not only with the point that I made earlier but also the points that other members of the panel have made. We need multiple voices talking specifically about the ethics and the risk of artificial intelligence. At the moment, that conversation is rather too narrow and it needs to be broader.
Dr Simon Beard: My recommendation is also one about breadth. It is about the breadth of how you see this committee. I work with all sorts of amazing people who do ‘ethical AI’ or ‘responsible innovation’ and one of my questions is why they call it that. My father, Andrew Beard, was a bridge-builder. Many of you will have been across his bridges. He never said, “I build non-collapsing bridges”. He was just a bridge-builder.
We need engineers to see ethics, safety and responsibility as being as innate in the engineering of biotechnology and AI as they are in the ethics of building bridges, for instance. That is going to require some really broad thinking by you, as the committee, and a willingness to say some things that might sound quite political and quite odd to talk about in a risk context. If you focus only on these narrow questions of risk assessment and risk management policy, you are going to miss a huge amount of where the technological risks we face are coming from and what the UK Government can do about that.
Lord Mair: As an engineer, I say amen to that.
Dr Stephen Cave: My point builds on that. The ethical AI and responsible technology agenda is extremely well aligned with risk management. They both have the same desired outcome of a flourishing society. I want to reiterate the point I made at the start: incorporating that into risk assessment would be easier if the conflation of different kinds of category that we see at present—these causes and effects, and many things in between—is stopped. If they are pulled apart, and we look specifically at undesirable outcomes and particular states that we want to avoid, we can think more imaginatively about how technology can contribute to all of them, potentially, rather than trying to focus on specific technological applications as intrinsic risks. If we did that, some of what the other panellists have mentioned, such as looking at interconnectedness, bringing in a more diverse set of voices and so on, would be much easier.
John Thornhill: I am fascinated, as you can tell, by the Swedish example of these national resilience exercises. However, I would go a bit beyond what is done there and look at it not just from the top down—from what the agencies, the companies and the banks are doing—but also from the bottom up, to engage ordinary citizens in different fields on how they would cope in some of these emergencies. It would be more interesting, in a way, to understand how people on Mumsnet or in the farming community, or teachers and healthcare workers, would respond in these circumstances, thereby fusing in national resilience exercises at a strategic level the very practical impacts that they could have on day-to-day life.
The Chair: The Swedish example, where every member of the citizenry is involved, is one that we will need to consider in some detail, so thank you for that.
I must say that this evidence session has been fascinating, not least because it has taken us in directions that we were not really expecting. It has set us challenges, as well as setting our witnesses challenges. Thank you all very much indeed for that. It has been a very interesting session.
