Written Evidence Submitted by Sense about Science
(GAI0078)
Sense about Science is an independent charity that promotes the public interest in sound science and evidence. We work with communities all over the UK to make sense of evidence, and with researchers and policy makers to raise the standard of evidence in public life. We have 20 years of experience working with experts and the public to create our making sense guides that provide clear and balanced evidence behind important and often misunderstood areas of science.
Five years ago, we became concerned by the growing gulf between the increasingly complex applications of data science and AI to analysis and decision making, and the ability of people (including politicians, journalists, service commissioners) to ask significant questions about them.
We produced the first public guide to data science in 2019 (Data science: a guide for society). This was followed by the Using Artificial Intelligence to Support Healthcare Decisions guide.
We are currently leading a project with the European Commission Competence Centre on Modelling (CC-MOD) to produce Commission guidance on using and communicating model-based evidence, to be launched with the Commission and EU Parliament early 2023.
The public’s ability to question evidence and understand the reasoning behind policy decisions affects how well received or how accepting the public will be of decisions.[1] Data science and AI are fast becoming ‘black boxes’ where the basis of a decision is unclear, not only to the people it affects but to the people applying the decision. We have been contacted by people turned down for credit and also by people in very senior roles in banks who do not know how their decisions are made. When people have asked for more information about how decisions are made, they have been told that this is ‘proprietary’. This is a response not just from banks, but from government departments (on risk assessments), and from councils (on housing decisions). The government has an opportunity here to set an example, providing accessible evidence and explanations about how an AI- based decision has been reached.
The Committee’s previous inquiry into algorithms in decision-making (proposed by Sense about Science as a My Science Inquiry) has already identified the need for government to continue to make public sector data sets available and produce a list where algorithms are used within central government. Since then, however, there have been significant developments including a legal challenge to the Department of Work and Pensions’ decision not to disclose its benefit fraud algorithm, and a 2019 UN report into the UK’s “digital welfare state”, which found that algorithms are “highly likely” to repeat biases reflected in existing data and make them even worse.
To be meaningful, it is essential that not only the application of AI but also the explicit workings – the elements that are often referred to as proprietary - are transparent, especially for systems whose outcomes significantly impact people’s lives. We hope that the Committee will be able to address this tension.
AI is a powerful tool for making decisions more efficiently. The Committee will be aware that it is being used in an increasing range of assessments and decisions. This is only true efficiency, however, if the outcomes are sufficiently reliable for the application and include a transparent account of the predicted accuracy, and if this account is reviewed and updated as AI based decisions are implemented – rather like phase 4 of a clinical trial, in which medicines are monitored post licensing.
Our work with the public, decision makers and service commissioners has focused on identifying the key questions to ask. We have distilled these into the following three areas:
The data used to develop the AI should be appropriate for answering the question that is being addressed. The type of relationship should be explained, for example, correlation does not always mean causation and the failure to identify those differences can lead to incorrect interpretation or misuse of data. It is important to consider whether the data is representative of the thing we are interested in. It is also important to review whether the data from which the machine will learn might be different or limited in practice to the data used to build the tool.
The right thing should be measured for the specified purpose. It is important that any missing variables are verified that they do not affect the data. It should also be ensured that no bias or human prejudice is playing a factor in the interpretation of the data. How true these assumptions are can make a massive impact on how true the results of the analysis turn out to be.
The model should be tested to see how well it performs by testing accuracy, sensitivity or specificity. It should be determined whether the model reproducible for an entirely different data set. This level of precision should be high enough to reflect the importance and weight being put onto the evidence. It is also important to consider whether it would make valuable real-world recommendations, and if it’s really better than what we’ve already got.
Measures to govern AI aim to avoid the adoption or automation of biases, discrimination and fallacious argument. It is not feasible to set specific standards on all possible applications, which range from predicting transport needs to identifying fraud risks, cancer risks and military targets. Mandating meaningful transparency and accountability is likely to be more practical and useful. Measures should be based on principles rather than rigid rules. For example, using AI to make decisions about healthcare will have different thresholds and ways of scrutinising the evidence for making decisions depending on the type of condition, the options for treating and the effectiveness of current practices.
Transparency about where and how AI is used seems patchy at best, transparency of where and how AI is used, with no clear process for people to question or seek to better understand decisions that are based on it. In our experience, it is likely that the public will be more personally concerned with AI decisions that use their personal data, such as in healthcare settings, to inform decisions that are specific to them and this is where people have least recourse. Results from the 2021 World Risk Poll also indicate that concern over the use of AI is higher amongst those who have experienced greater discrimination. But measures to improve transparency and accountability cannot not be limited to this because non-personal AI-based decisions have far-reaching effects.
As well as restating the need for transparency and accountability, and elaborating options for improving it, we hope that the Committee will look at possibilities for creating an accessible ombudsman role, open to people both directly and indirectly affected by decisions. We emphasise accessible, which includes being widely publicised, actively equipping people with an understanding of what they have a right to expect and being sufficiently resourced to respond.
(November 2022)
[1] Our 2022 What Counts? inquiry into how well government’s evidence for covid decisions served society identified a public need for greater transparency of evidence and reasoning behind policy decisions that is applicable well beyond the pandemic.