Written evidence submitted by Simon Beard (ROB0045)

Evaluating extreme risks of artificial intelligence – a response to the Science and Technology Select Committee enquiry into Robotics and Artificial Intelligence by Simon Beard, Centre for the Study of Existential Risk, University of Cambridge

This submission is made in a personal capacity by Simon Beard, research associate on the project ‘Evaluating Extreme Technological Risks’ at the Centre for the Study of Existential Risk, University of Cambridge. This response deals with the social, legal and ethical issues relating to the evaluative uncertainty surrounding the long-term risks of artificial intelligence. It is being coordinated with the response of Professor Huw Price (Bertrand Russell Professor of Philosophy, Cambridge), which deals with the desirability of early attention to the likely long-term impacts of AI.

Executive Summary

In this submission I highlight some of the uncertainty surrounding the long term risks of developing artificial intelligence. I argue that whilst these risks demand our attention, the uncertainty that surrounds them is being greatly reduced by contemporary research and that work to reduce these risks is already under way. I also highlight some of the key ethical dimensions of evaluating these risks and point out that the distribution of the benefits from the long term development of artificial intelligence may be as morally significant as the size of its potential risks. Finally I highlight some key policy initiatives that it would be wise to take in terms of promoting world leading research into AI safety in the UK and developing stronger connections between policy makers, researchers in the field of AI safety and industry.

Artificial Intelligence and Existential Risk

1. Some of the most important social, legal and ethical issues raised by the development of artificial intelligence relate to high impact risks that could be posed by AI in the long term. These include existential risks, threats of premature extinction of earth originating intelligence or of the permanent and drastic destruction of its potential for desirable future development[1], as well as risks involving less severe global catastrophes.
2. Although the probability of existential and catastrophic risks is highly uncertain, and may be very low, a number of recent reports[2] [3], scientific editorials[4] and policy discussions[5] have highlighted that emerging technologies may pose such risks; due to their rapidly developing nature, such risks are likely to be understudied, and may not be adequately evaluated in global policy and technology governance analyses. In his 2014 Annual Report the Government Chief Scientific advisor argued that such risks should not lead us to abandon the development of new technologies, but that “we must constantly scan the horizon and do our best to prevent and mitigate adverse consequences of new technologies”[6].
3. Experts have highlighted that artificial general intelligence could pose an existential or catastrophic risk if it achieved a level of general intelligence greater than that of human beings. One plausible and well-explored scenario would be if an artificial intelligence possessed the following properties:

• Goal non-alignment: the artificial intelligence’s goals were not sufficiently well-specified to avoid the possibility of catastrophic consequences  and
• Decisive advantage: the artificial intelligence was sufficiently capable and unconstrained in its operations that anticipating, preventing or modifying its activities was difficult or impossible. An artificial intelligences might gain such an advantage by achieving a qualitatively higher level of problem-solving ability than a human, operating at significantly faster speeds than a human or being significantly better at coordinating its activity on a global scale than human beings could[7].

4. Other circumstances under which artificial intelligence could present an existential or catastrophic risk were if it gave a decisive advantage to a human or group of humans who misused it, or if its creation precipitated a catastrophic event such as a global war.
5. Artificial general intelligence is characterized by its ability to learn across any domain of knowledge or activity, to adapt to its environment and to exploit developments in one domain to make progress in another domain. Whilst artificial intelligences are currently able to perform at a level greater than or equivalent to that of human beings in some domains, such as chess, correctly answering trivia questions and stock market trading, this performance is domain-specific. Artificial general intelligence is therefore likely to be qualitatively very different to current artificial intelligence[8].
6. The consensus amongst experts is that advances in general artificial intelligence are most likely to have significant benefits to humanity, but that there is some risk of producing ‘extremely bad’ or ‘catastrophic’ outcomes[9]. Furthermore, whilst many researchers are confident that general artificial intelligence at a level greater than or equal to that of human beings can be developed in theory, most believe that it is still some way away. In a recent survey of experts across a number of fields related to the development of artificial intelligence, median estimates were that there is only a 10% chance of having developed this level of artificial intelligence by 2022, but a 90% chance of having developed it by 2075[10]. However, these results cover a very wide range of opinions, with some experts arguing that general artificial intelligence, if it is even possible, will take centuries to develop.
7. A growing community of researchers are working to understand the long-term risks posed by artificial intelligence, with UK researchers playing a leading role. As well as the Centre for the Study of Existential Risk, significant research projects are being pursued at the Leverhulme Centre for the Future of Intelligence (at the University of Cambridge), the Future of Humanity Institute (at the University of Oxford), Imperial College London and the Strategic Research Centre for AI Policy. Other prominent centers of research worldwide include the Machine Intelligence Research Institute at the University of California Berkley, the 100 Year Study on Artificial Intelligence at Stanford University, the Future of Life Institute and the Global Catastrophic Risk Institute.
8. In the rest of this submission I briefly summarize some of the reasons why there is so much uncertainty about the risks associated with artificial general intelligence and how researchers are working to reduce this (paragraphs 9 – 16). I then set out some of the social, ethical and legal issues raised by this uncertainty and how we should respond to it (paragraphs 17 – 25).

Uncertainties about the long-development of artificial intelligence

9. One key barrier to evaluating the risks of developing artificial general intelligence is forecasting how it is likely to be developed and what capacities it will have. Historically, developments in artificial intelligence have proven very hard to predict[11].
10. However, analysis of the most promising techniques in artificial intelligence are providing useful indicators of future progress in AI, and may identify key bottlenecks to be overcome. For instance, deep learning has proven applicable to an exceptionally broad range of tasks, and researchers expect continuing progress on the underlying techniques. Similarly, approaches such as reinforcement learning (used by Google DeepMind as part of AlphaGo and DQN) show promise for AI systems that can learn and adapt to their environments[12].
11. Advances in fields such as neuroscience and cognitive science are also likely to continue providing valuable insights; for example, such advances may identify specific breakthroughs needed to achieve cognitive capabilities that are currently unique to humans and animals.[13].
12. Together these developments are helping forecasters of AI to consider possible pathways towards the development of artificial general intelligence in much more detail, and to calibrate their estimates against past performance. They are also being harnessed to new, more structured methods of horizon scanning to combine diverse expert perspectives into more reliable roadmaps for how progress towards artificial general intelligence is developing[14].
13. A roadmap for the development of whole brain emulations as an approach to general artificial intelligence was produced in 2008.[15] Work is ongoing to develop further roadmaps that reflect recent advances and future prospects in a range of machine learning and AI techniques, as well as the current state of the art in neuroscience and cognitive science.
14. The long term safety of AI development does not only depend upon how AI is developed. We also need to consider other developments in computational hardware and related technologies that may interact with AI systems. Furthermore, it is necessary to consider the social, political and environmental contexts in which the critical breakthroughs occur[16]. For instance, research suggests that whether artificial general intelligence ultimately emerges out of a competitive race or a cooperative endeavor could greatly affect probability of a catastrophic outcome[17].
15. Over the next two years the Strategic Research Center for AI Policy aims to survey work on forecasting these developments and their potential impact, and analyze the interactions between these developments and the emergence of artificial intelligence at a level greater than or equal to that of humanity.

The control problem

16. Depending on how AI develops and the context in which it emerges, it is possible to envisage a number of pathways in which the development of artificial intelligence could pose an existential or catastrophic risk[18]. Determining how to avoid such catastrophic outcomes is sometimes known as the control problem. Research in this area focuses on strategies to ensure that the capabilities of artificial general intelligence systems can be reliably understood and contained, that a system’s behaviour will be predictable in all relevant environments, and that goals and values for such a system will be designed such that they will be pursued without negative consequences, intentional or otherwise.
17. Long-term work on the control problem is likely to dovetail with near-term technical work on AI safety; including methods to make the internal processes of AI systems more transparent, and methods to predict circumstances in which AI systems will produce unwanted or unpredictable behaviours[19]. Work on the control problem also draws on advances in other relevant fields such as cybersecurity for near-term AI, and is guided by work on the development of policy and ethical guidelines for near-term AI.
18. Approaches to solving the control problem are a very new area of study, however progress is being made by a number of leading researchers and the importance of this challenge is increasingly being recognized by developers of artificial intelligence, such as the OpenAI project, alongside academics[20].
19. Key problems that need to be tackled as a priority are 1) the extent to which the long term safety of artificial intelligence depends upon the kind of development that takes place and 2) how to steer the development of artificial intelligence along the safe pathways and what safeguards should be put in place before any artificial intelligence is created, rather than during or after its creation. Whilst the creation of artificial general intelligence at or above the level of humanity seems unlikely to happen for a number of decades, it is imperative that necessary research on the control problem takes place, and that its findings are implemented, well before such AI is developed.
20. Furthermore, it is not at present clear how difficult the control problem will be to solve. In part this is due to the current uncertainty in forecasting what artificial intelligence will be like and when and how it will be developed. As forecasting improves, the nature of the control problem and its solutions will become easier for researchers to study.

Moral difficulties in evaluating the long-term development of AI

21. There are also considerable challenges in understanding the moral importance of the existential risk posed by artificial intelligence because of a poor understanding of the kind of ethical judgments that are necessary to evaluate catastrophic and existential risks in general. Risks of this scale involve significant amounts of uncertainty and costs and benefits that affect the far future.
22. Key challenges to evaluating these outcomes include the fact that existential risks may only involve the loss of potential future lives[21], the role of counterfactuals in our assessment of an outcome’s desirability[22] and the difficulty of decision making when there is uncertainty about the size of the values under consideration[23]. Progress is currently being made on ethical decision making under these conditions in the contexts of climate change and biosecurity and the Center for the Study of Existential Risk is drawing on this work to develop alternatives to conventional Cost Benefit Analysis for evaluating extreme risks associated with emerging technologies, such as artificial intelligence.
23. From a moral perspective however, this uncertainty may be less important than the question of who is likely to benefit from the long-term development of artificial intelligence. Because the probabilities of any existential or catastrophic risk from artificial intelligence are likely to be low and the potential benefits from artificial intelligence are likely to be very great it is almost certain that artificial intelligence represents a net benefit to humanity. Such a benefit may not be justified however if it imposed a risk on all of humanity, but primarily benefited only a small percentage of people. The moral justification for long-term development of artificial intelligence may therefore depend as much on ensuring that its benefits are widely shared as on ensuring that its associated risks are minimized.

Regulation and policy

24. The challenges of avoiding existential risk from the development of artificial intelligence are complex and given the likely time-frame it would be unwise to reject the benefits of artificial intelligence because of them. However, it is important that existential risk forms part of the discussion about the social, legal and ethical issues raised by the long-term development of artificial intelligence.
25. At present, research into long-term policy to avoid existential risks from artificial general intelligence is focusing on two key areas. Firstly, how to effectively combine efficient regulation of long term and short-term risks from AI, so that as many people can enjoy the benefits of artificial intelligence as possible. Secondly, what lessons can be learned from regulating other dual use technologies that have long term benefits but pose potentially catastrophic risks, for instance nuclear power and biotechnology. For instance, as with these industries, it may prove challenging to monitor and enforce regulations relating to artificial general intelligence, particularly if the necessary computing hardware is widely available or easily acquired, and if the access to the software and necessary knowledge is difficult to control.
26. Due to the need to improve forecasting of AI development in order to understand the risks it poses, it is important that research into AI safety is conducted in partnership with the development of artificial intelligence. Such partnerships are already being developed with a range of industry bodies and we are enthusiastic about the potential to develop strong industry norms around AI safety.
27. It is also important that the development of AI does not become too competitive, either between different research groups or different countries, particularly after certain thresholds in capability have been achieved. While we are still a considerable distance from risky thresholds in AI development, processes that encourage collaboration ahead of time between leading research groups on long term safety relevant issues should be encouraged. This might be achieved via an arrangement similar to the ‘precompetitive sharing’ of information in the aerospace and automotive industries.

Key recommendations

28. That policy discussions about the social, legal and ethical issues relating to the long-term development of artificial intelligence should include a consideration of the existential risks that it poses that are based on the best available evidence and include all relevant parties, such as policy makers, industry and academia.
29. That it is nevertheless advisable for such discussion to be differentiated from other concerns about the shorter term and less dramatic social, legal, ethical and economic issues raised by developments in artificial intelligence.
30. That the government should continue to support world-leading research into AI safety in the UK.
31. That, as far as possible, AI safety research should be closely connected to research into the development of advanced AI and machine learning and how this might be used to benefit all of humanity.

April 2016