Written Evidence Submitted by Professor Chris Reed
(GAI0059)
I am responding in my capacity as Professor of Electronic Commerce Law at the Centre for Commercial Law Studies, Queen Mary University of London. I am an academic, specialising in technology law for over 40 years. I began investigating AI in the 1980s, and most recently have been working on the regulation of AI including its governance. I am a Fellow and former Chair of the Society for Computers and Law.
My relevant publications include:
Reed, Grieman and Early, ‘Non-Asimov explanations – regulating AI through transparency’, in ‘Law in the Era of Artificial Intelligence: Nordic Yearbook of Law & Informatics 2020-2021’, Ch 14, https://irilaw.org/2022/02/16/new-publication-nordic-yearbook-of-law-and-informatics-2020-2021/).
Reed, ‘Data Trusts for Lawful AI Data Sharing’ (Ch 3 in Gary Chan Kok Yew & Man Yip (eds), AI, Data and Private Law: Translating Theory Into Practice, Hart 2021)
Reed, ‘Cyberspace Institutions, Community and Legitimate Authority’ in Orkun Aksell and John Linarelli (eds) The Future of Commercial Law (Oxford: Hart Publishing 2020) Ch 6.
Reed & Irene Ng, Data Trusts as an AI Governance Mechanism (February 14, 2019), https://ssrn.com/abstract=3334527or http://dx.doi.org/10.2139/ssrn.3334527
Reed, “How should we regulate artificial intelligence?” Phil. Trans. R. Soc. A 376 : 20170360.
I am currently engaged on a research project to investigate how we might impose regulatory requirements for AI to act fairly and reasonably.
Each organisation which develops and/or uses AI devises its own governance system, and so no overarching answer is possible. In broad terms, some organisations are developing sophisticated governance systems, most are just beginning to understand that some governance is needed, and some have no governance systems at all.
The main aims of any system for the governance of AI use should be:
(a) to ensure that the AI is being used for a purpose which is within its capabilities (eg it should be clear that a probation advice system is not designed to decide criminal sentencing),
(b) that the potentially adverse consequences of using the AI have been anticipated and mitigated, particularly in edge cases and
(c) that the AI’s performance is monitored to identify and correct problems arising from its use.
The main aims of a governance system for the development of AI should be rather different:
(a) to be explicit about what the AI is, and is not, designed for,
(b) to ensure that the training data is representative and full enough,
(c) to ensure that testing captures a sufficient range of likely uses to identify all the main issues needing resolution, including edge cases where its decision-quality is uncertain, and
(d) to oversee post-development monitoring to identify and correct out-of-line decisions by the AI.
The main weakness of any governance system is if it fails to ask the correct questions about the technology. This is unsurprising, because the correct questions are only just emerging from the research community, and are continually changing as AI technology takes new directions. This is an area where broad and regularly updated guidance for AI governance bodies might be very useful.
The main governance system in which I have participated directly is through my former membership of Queen Mary’s Research and Ethics Committee. The strength of academic research ethics committees is that they take the time to think deeply about the proposed activity and identify potential ethical issues. It helps that they include expertise from a wide range of disciplines. The amount of time spent examining complex cases is extensive, and non-academic organisations are unlikely to be able to justify such expense, particularly in terms of the time of leading experts within the organisation.
From a lawmaking perspective it might seem obvious that the decision-making of an AI should require some kind of explanation (as, for example, provided in Articles 13(2)(f) and 14(2)(g) of the GDPR). The difficulty with creating such a provision in law is that:
As an example, imagine an AI which is used by a company to review job applications. A rejected applicant might want an explanation which assures them that other applicants were more suited for the position. The general public might want to know that the company is operating fair hiring processes. If a rejected applicant brought a legal action against the company, the judge would need an explanation of the factors which led to the rejection decision, compared to decisions made about other applicants, focused entirely on those matters which were legally relevant to the judge’s decision. An expert witness in that case might want explanations about the training and testing of the AI, how its developers determined which factors in an application should be identified and labelled for the training of the AI, and how factor weightings were determined.
It should be apparent that all these explanations are radically different, and all but one of them is unique to the facts underlying the particular decision in question, and would not necessarily apply to any other demand for an explanation. Most explanations will need to be contextual.
If we imagine a different AI application, such as a medical diagnosis AI or an AI which drives a vehicle, we can see that different kinds and methods of explanation which would be needed to be meaningful.
Finally, the use of machine learning to develop an AI can mean that its developers do not themselves ‘know’ how it works, in the sense in which a human decision-maker describes how they would make an equivalent decision. The developers will know how they trained and tested it, and can use eXplainable AI (XAI) tools to investigate individual decisions of the AI and gain some understanding of its workings, but translating this information into an explanation which is meaningful to others is a complex and inexact task. And it still focuses on individual decisions, rather than producing a generalised narrative about how the AI works (see my recent research publication ‘Non-Asimov explanations – regulating AI through transparency’, cited at question 1 above)
Finally, mandating transparency might be overly demanding for many types of AI. As an example, an AI which recommends music I might like to listen to, based on my past music listening, seems hardly to demand much transparency. If I don’t like its recommendations, I can simply stop using it.
First, not all decisions involving AI need to be reviewed and scrutinised. The music recommendation AI is an obvious example.
Second, I see no conceptual difference between the public and private sectors here – whether review and scrutiny is desirable depends on what the AI is doing and its potential to cause harm.
AI is such a complex and fast-moving field that it is far too early to give a definitive answer to this question. As an interim approach, I take the view that a suitably cautious approach might be as follows:
(a) For sectors which already have a regulator, such as medicine, transport and finance, this question should be left to the sectoral regulator. Those sectors are regulated because their activities have the potential to cause substantial harm to others, and the sectoral regulator is best placed to decide what levels of review and scrutiny are desirable to achieve the overall regulatory objectives.
(b) For AI applications outside the regulated sectors, such as recruitment AIs, legal and regulatory obligations (including review and scrutiny) should be based on the degree of risk which arises from using that AI. It would seem entirely appropriate to demand that those responsible for high risk AI implementations should establish internal review and scrutiny regimes. Guidelines from an authoritative source would be helpful in ensuring those regimes were effective. Whether external review is necessary is a political decision, based on lawmakers’ understanding of the public good. However, enacting external review for all high risk AI is likely to create a highly expensive regulator because so many AI applications are potentially high risk.
(c) For low risk AI applications, I currently see no need for external review and scrutiny. Authoritative guidelines about the need for, and shape of, internal review and scrutiny would be helpful here.
I should make clear that although I use the terms high and low risk, I am not recommending the definitions in the EU draft AI Act. These seem to me not clearly thought through, and driven to some extent by the demands of the EU as an institution rather than the needs of the public. The UK should develop a better understanding of the distinction, considering of course the approaches adopted elsewhere in classifying risk but not placing excessive weight on them.
This is far too unfocused a question. AI is penetrating all aspects of human life, and we would not dream of asking ‘How should human life be regulated?’.
This suggests that the regulatory focus should be gradualist and incremental, starting with obviously risky uses of AI and only extending to other uses when their risks (if any) become apparent. Building an overarching regulatory system for AI is an exercise in predicting the future, and no lawmaker in the world has a strong track record of accurate prediction in the technology field.
Based on my researches to date, my recommendation would be to begin by focusing on ensuring proper professional due diligence in the production and implementation of high risk AI.
Regulation would focus on three aspects for development:
(a) Training – was the AI trained on appropriate data which adequately represents the field of use. The aim would be to ensure, for example, that a medical diagnosis AI was not trained exclusively on data from medical students, who are not representative of the population to whose treatment the AI would be applied. Much clinical trial data is flawed because women have been excluded from trials for what seemed like good medical reasons, and this kind of gap in training data needs to be avoided.
(b) Testing – AI performance needs to be evaluated against test data which adequately represents a full range of the likely uses to which the AI is to be put. The 2016 accident in which a Tesla vehicle turned in front of an oncoming vehicle was in a situation for which the technology had not been tested. In the Uber 2018 collision which killed a pedestrian crossing the road at night, either the testing failed to consider cases where a potential collision risk could not be identified, or an inadequate response was programmed.
(c) Monitoring in use and retraining. Unexpected actions by an AI will inevitably occur in use, and these need to be detected and the AI retrained to reduce the risk of their recurrence. The Tesla crash might be an example here.
Producing a perfect AI is even less likely than writing a perfect computer program, and so obligations of reasonable care/due diligence on the part of developers is all that society can reasonably demand from them. Our current legal understanding of this human standard is flexible enough here – the greater the risk to others, the more care is required.
It is currently too early to demand that AI decisions should be fair, or reasonable, or have similar characteristics. My current researches indicate that we cannot yet define these terms in ways which can be used by AI developers, though I hope to shed light on ways in which this might be achieved.
Implementing an AI developed by a third party would require the implementer to take reasonable care/due diligence to ensure that the AI was an appropriate tool for use in that domain (a modified music recommendation system would not make a good recruitment AI), to ensure that it is sufficiently accurate for that purpose, to monitor the AI in use and report unexpected outcomes its developer, and, if necessary, take the AI out of use or constrain its uses. For example, if a recruitment AI appeared to be excluding some candidates on the ground of their race, the user should report this to the developer and cease to use the AI to process applications from candidates of that race.
As part of such a regulatory system, high risk AI developers and implementers might reasonably be required to provide information to their regulator. It is not immediately apparent what that information should be, and I would suggest this is left for the regulator to decide, based on the regulatory objective of the regulator. The regulator of an AI controlled robotic surgeon needs to know different things from the regulator of an autonomous vehicle.
The best regulators for already regulated sectors are likely to be the existing sectoral regulators. The AI needs to work as part of the wider regulated activity, and a separate AI Regulator would lack expertise in these areas. However, sectoral regulators would need to coordinate to deal with AIs which have uses in multiple sectors.
Whether any currently unregulated activities involving AI create such a high risk that they need regulating is, as yet, unclear to me. Our current regulatory systems will probably continue usable for some time – the user of an AI recruitment system which discriminates on grounds of race will be in breach of the current non-discrimination legislation, for example. It would make sense for lawmakers to begin considering how such a general regulator might work, and how its sphere of authority would be reduced from all of human life, so as to be ready if it transpires that general regulation is needed.
The regulatory system is currently completely unfit, because it is based on human decision-making processes. As explained in my answer to question 3, AIs rarely (probably never) make decisions in the same way that humans do.
However, the development and use of AI is at too early a stage for us to understand what a legal framework which was fit for purpose would look like. Fortunately there are stopgap solutions which should be adequate until we have, collectively, achieved a clear understanding.
One area where it is already clear that problems exist is in claims for compensation under the tort of negligence. There is no simplistic solution to this problem (and the draft EU AI Liability Act misses the point almost completely). In very simple terms there are two areas where the law is inadequate:
(a) The allocation of responsibility between developer and user if an AI is used for decision making. An example which is easy to understand is an accident caused by a self-driving vehicle. Normally we would ask if the human driver was negligent, but that is no longer a sensible question as there is no human driver. The law will therefore ask whether the developer took reasonable care in developing the AI, and/or whether the user of the vehicle (the operator, not the occupant) took reasonable care in selecting and operating it, in both cases such that the accident ought not to have occurred. Each defendant is likely to blame the other, and some legal presumption of primary responsibility would simplify litigation. An even simpler stopgap is to impose no-fault liability on some appropriate person, as the UK already does in this situation for claims by the injured person (section 2(1), Automated Vehicles Act 2018, imposing liability on the insurer). Note, though, that this stopgap is only workable because compulsory insurance already exists.
It is important to note that there may be no one-size-fits-all solution. For example, imposing strict liability on the users of medical AI technology would have complex knock-on effects, and would need thorough consideration. The same is likely to be true for other AI uses where negligence claims are likely.
(b) Proof of the claim. The burden of proof is on the claimant in negligence, and under the UK rules of discovery the claimant is entitled to see all relevant documents in the possession of the defendant. Thus in theory, the records relating to the AI’s training, testing, monitoring, etc could be analysed to make an argument that the developer or user was negligent. In practice, these records would be meaningful only to an expert in the field, and perhaps not even to such a person without assistance from the defendant (which would not be forthcoming in adversarial litigation). Clearly the cost of litigation, and its duration, would both increase greatly.
The obvious stopgap here is to reverse the burden of proof, requiring the defendant (who has both the technical records and the necessary expertise) to prove that reasonable care was taken. But this is likely to be appropriate only for some kinds of high risk AI – even producing a list of likely candidates would require complex research. We can be reasonably confident that strict liability for AI-driven vehicles (or a reversal of the burden of proof) would result in overall lower insurance costs because of the reduction in accidents. But strict (or reverse proof) liability for medical AI might even increase costs if so many more patients are diagnosed and treated that claims increase in number.
Singapore’s Model Artificial Intelligence Governance Framework is probably the most integrated governance system so far. It covers the main issues discussed in this response, and proposes a workable roadmap towards (eventual) legislation. However, Singapore has substantial differences from the UK, particularly in the close relationship between regulators and businesses, and simply transplanting its Framework would not be appropriate.
(November 2022)