Science and Technology Committee
Oral evidence: Algorithms in decision-making, HC 351
Tuesday 23 January 2018
Ordered by the House of Commons to be published on 23 January 2018.
Members present: Norman Lamb (Chair); Vicky Ford; Bill Grant; Darren Jones; Stephen Metcalfe; Neil O’Brien; Graham Stringer.
Questions 294 - 393
Witnesses
I: Elizabeth Denham, Information Commissioner.
II: Margot James MP, Minister for Digital and the Creative Industries, Department for Digital, Culture, Media and Sport; Oliver Buckley, Deputy Director, Digital Charter and Data Ethics, Department for Digital, Culture, Media and Sport; and Andrew Elliot, Data Protection Bill Manager, Department for Digital, Culture, Media and Sport.
Written evidence from witnesses:
– Information Commissioner’s Office
– Department for Digital, Culture, Media and Sport
Witness: Elizabeth Denham.
Q294 Chair: Welcome. It is very nice to see you. Thank you for coming along. We have a number of questions for you, and I will start.
During our inquiry we have been focusing on issues relating to discrimination and bias. How significant do you think that issue is in the emerging and expanding use of algorithms across a range of different applications? Are the rules sufficient to address any concerns that there may be?
Elizabeth Denham: Data protection has dealt with issues such as transparency, fairness and bias for more than 20 years. It is an increasing problem that impacts on people when we have more and more AI systems that are not transparent and decisions that are made by AI systems in organisations that are not clear and that are not scrutinised by regulators and individuals.
The advantage is that the GDPR brings in new rules for organisations and new rights for individuals that will provide the tools to do just that.
Q295 Chair: It is sufficient, is it, to meet the challenge we face?
Elizabeth Denham: The legal tools are there in the GDPR and in the Data Protection Bill that is making its way through Parliament, but the test will be in the implementation and use of those tools. As a regulator, we think the law will make us fit for purpose in helping individuals to assert their rights, and in working proactively with organisations to ensure that they can innovate, protect fairness and build their algorithms in such a way that they are not biased or discriminatory. We have new powers to look at the way systems are built, and there is a new responsibility for organisations to bake in fairness and transparency at the outset.
Q296 Chair: Are there areas where you have particular or the greatest concerns—the criminal justice system, for example?
Elizabeth Denham: The more significant decision is on somebody’s life in the criminal justice system—that is a great example—but health, employment and housing are areas of significance, and those would be areas of focus for our office in working with organisations in those sectors to get the public policy right and to ensure that the technology is built in a way that allows transparency, scrutiny and rights.
There may be some red lines—in criminal justice, for example, where AI decisions tools could be part of a decision—but there certainly needs to be human intervention at the end of the day around sentencing or determining parole. That is really important.
Q297 Chair: Your standard remit revolves, in particular, around the use of data, but is it wide enough and clear enough to cover the application of data in algorithms and the outputs from those algorithms?
Elizabeth Denham: If you look at data protection and the wider field of data ethics, it is a bit like a Venn diagram. There is part of our remit—but it is not going to go far enough to answer all questions that need to be answered in society. I will give you an example. Although we can look at decision tools and AI that involve personal data and make a determination on whether that is fair for one individual or group of individuals, when there are questions around the use of health or NHS data to be used more broadly to solve medical diagnoses or to identify new pathways for clinical diagnoses and care, a data ethics group or centre such as the one that the Government are proposing could play a really good role in leading a public dialogue about broader uses and applications of new technology to some of our very valuable NHS data.
Q298 Chair: Do you think we cover the ground sufficiently with a combination of your role plus the regulation within individual sectors? You have mentioned health. Is that the way to do it, do you think?
Elizabeth Denham: I do not think that we need an AI regulator. That is going to confuse the landscape. Our remit covers personal data, which extends across all the sectors. You could think about the ICO as perhaps a first among equals, but all the other sector regulators—the FCA, Ofcom, Ofgem and the National Data Guardian—all those oversight organisations and regulators have a role in looking at AI and how it is applied in the context of each sector.
The other thing about GDPR is that it provides for codes of conduct and certification. For the first time, an industry sector, for example, could come up with a code of conduct that works within the financial services sector, and it can be endorsed and reviewed by the FCA as well as the ICO. Then we could turn to third parties to check and certify that the AI is implemented in a fair way.
Q299 Chair: So, standards that are developed by industry but are accredited by the relevant regulator. Is that the model?
Elizabeth Denham: I think that is the model. Overall, if we had a set of principles that were developed by the regulators with input from Government and a data ethics centre, those principles could be interpreted and developed as standards within the sectors. In that way, you have cohesion and harmonisation.
Q300 Chair: So, there are overarching principles, but they need to be interpreted for each sector.
Elizabeth Denham: Yes, for each sector. But the regulators need to be brought together. Our office is looking at convening the regulators to talk about AI systems. The ICO touches all those sectors if the AI systems involve the use of personal data.
Q301 Chair: Presumably there is an awful lot of work to be done with each of the sector regulators in upskilling to understand the potential risks and the ethics involved in this technology. Is that right?
Elizabeth Denham: That is exactly right. There is upskilling that needs to happen within the Information Commissioner’s Office, too, because we are going to be involved in algorithmic auditing. We have not conducted such an audit before, so we need partnerships with expert agencies and help from academics and technology companies. We are looking at secondments and creative new ways to upskill in this market.
Q302 Chair: The autumn statement announced the creation of a centre for data ethics. The remit has not been announced yet. What do you think it should be doing? How urgent is it that it gets established and applies its guidance to the system?
Elizabeth Denham: The UK Government is ahead of many other Governments in looking to establish a data ethics centre for the protection of data and for the encouragement of innovation. The ICO welcomes the centre for data ethics and innovation. It can fill in some of the gaps. It can play a leadership role, especially in looking at the landscape, looking at what is ahead, doing the environmental scanning, writing papers and promoting public discourse on new technologies.
We would work with the centre. We have been commenting on the terms of reference. It is important that it is not a regulator. It should be a deliberative body, not an adjudicative body, because that confuses the landscape. You do not want two sets of permissions and oversight.
The centre for data ethics could help promote the UK as a place of innovation and as a leading country in AI technologies, and also as a leading country in the protection of data. Getting that balance right would make us a leader in this field, ahead of the US and certainly ahead of many countries in Europe. I welcome partnership with the data ethics centre.
Q303 Chair: Bias can happen, I understand, both because of the data that are used within the algorithm and because of a lack of data. For people applying for bank accounts, if there are insufficient data from the communities from where the person comes, for example, that can in itself introduce a bias—from the lack of data. Is there anything that could be done to address that? I suppose that the auditing of outcomes from such algorithms is one option, but how do you see your role in addressing issues of that sort?
Elizabeth Denham: The GDPR provides some tools for that. The first one is a requirement for organisations to conduct a data protection impact assessment when they are building AI or other technological systems that could have an impact on individuals. That forces the organisation to think through carefully what data are going into an AI system, how decisions are going to be made and what the output is.
The expectation is that, when we go to a company to scrutinise or audit its AI systems, the first thing we will do is ask for its data protection impact assessment. How did you think the training data that went into the development of the AI systems were fair and representative of the population? Those kinds of questions, and that tool, are new.
Q304 Chair: Is there an obligation to publish the impact assessment?
Elizabeth Denham: There is not. I am not promoting the need to publish it, because I think there would be commercial sensitivities in the development of new systems and new technologies, but it is there for us as a regulator to see. That is a different type of scrutiny than public scrutiny.
Q305 Chair: But it requires someone to come to you with a complaint, presumably, before you can then investigate it.
Elizabeth Denham: No. In fact, if companies or organisations are building systems that use high-risk or sensitive data, they have to come to us with their data protection impact assessment for us to make comments—so, that is proactive.
Chair: So, it is proactive. Right. Good.
Q306 Stephen Metcalfe: I want to pick up on the point about auditing AI systems and going in and asking for the impact assessment. Presumably, each company chooses its own criteria for that impact assessment and for the standards that it applies. Do you think it would be useful if there was some sort of industry-wide standard about what should be made available to you when you go in—an accreditation, a kitemark or a brand that says that something conforms to a set of standards that has been pre-established?
Elizabeth Denham: I think that certification, codes of conduct and seal programmes are the way to go. Because this area is complicated and it is all about context, these codes of conduct, seal programmes and kitemarks should be sector specific, but if they involve personal data, they must, under the GDPR, be approved by the ICO.
That is a really useful avenue to ensure that standards are consistent and harmonised across the sector, but I do not think you could have a certification scheme that works across autonomous fields such as health, financial services and transport. There is just too much.
Chair: It has to be sector specific.
Q307 Neil O'Brien: This question follows from both of those. During the course of this inquiry, some people have said to us that, to have full transparency, the nature of machine learning means that you have to publish the training data that it was grown on. What is your view on that?
Elizabeth Denham: I think we need a layered approach to transparency. In some cases, where a decision has been made by a machine that has significant impact on an individual, the GDPR requires that they have the right to challenge the decision and a right to have it explained to them. Explainability or transparency to an individual is different from the kind of scrutiny that a regulator or expert is going to bring to the table. If we push for full transparency, that does not lead to explainability and understandability on the part of the public.
Who wants to see code? Who needs to see code? That is not part of explainability; that is part of scrutiny. We may need, as a regulator, to look under the hood or behind the curtain to see what data were used, what training data were used, what factors were programmed into the system and what question the AI system was trained to answer. I do not think it is too hard—I just think we have to think about transparency in different ways.
Q308 Neil O'Brien: So, your view, if I am summarising it right, is that there should not be a general requirement to publish, but it should be accessible to your office, in the event of a complaint. Should it be available to anybody else? Should it be available to the complainant if they want to go away and find their own experts to dig through it?
Elizabeth Denham: It depends on the context, and it depends on the level of commercial sensitivity. That is a standard that could be determined sector by sector. When you are dealing with public sector data and the use of AI systems, there may be more of an argument for the publication of the data—but, again, as long as it is not personal data, because you are then contravening the Data Protection Act. It is a complicated question.
Q309 Neil O'Brien: It definitely is. How would you make that kind of judgment? Let us say I am in the private sector and I have used a machine-learning algorithm to make a decision. Someone is very unhappy about the decision that has been made, and they complain to you. You start looking at the source code, or rather the training data. Let us say it is Google, although it could be any other big tech firm. They say, “Can I see their source data?” What sorts of factors would you think about?
You said there was a trade-off between commercial sensitivity and the interests of the individual. Beyond the individual case, are you able to flesh out how you would think about that?
Elizabeth Denham: In our explanation to the complainant, as the regulator, we would have to give sufficient information for them to understand how we came to our conclusion, because the decision could be subject to judicial review. We have to explain whether or not we thought that there was bias or discrimination. That does not mean that we have to encourage all the training data to be published. I guess my answer to that question is that it depends on the context.
Q310 Neil O'Brien: Fair enough.
In your submission you say that, for transparency, people should be given “some basic information” about the use of their personal data. What might that kind of “basic information” entail?
Elizabeth Denham: That is a general requirement across the piece. Companies and public bodies have to give individuals some basic information about how their data are being used and who they are shared with. That is not just specific to AI—that is across the piece. People have to understand how their data are being used.
Q311 Neil O'Brien: So, that is broadly the same as everything else.
We have touched upon this a little bit already, but there seems to be some uncertainty about whether the GDPR imposes a right to explanation. What is your take on this? Would you like to see it as part of the regulatory environment in the UK?
Elizabeth Denham: It is part of the regulatory environment in the UK once the GDPR has direct effect in May.
Q312 Neil O'Brien: So, you think it does impose that.
Elizabeth Denham: Explainability in certain cases. I have talked to you about the broad requirement to explain to individuals how their data are used and shared, but in the context of AI and the explainability of a decision made by a machine, it is contained in the law. There is also a right to ask for an explanation of the details of the decision and the complaint that can be taken to our office.
Q313 Neil O'Brien: How are your office’s transparency powers going to be strengthened by the GDPR and the Data Protection Bill? Is there any way in which you would like the proposed legislation to go further?
Elizabeth Denham: Is your question directed at our powers in relation to AI and explainability, or across the whole—
Neil O'Brien: I would be particularly interested in that but, if there is something else across the piece that you think is missing, I would be interested in that, too.
Elizabeth Denham: The most important change in the law, in the GDPR, is about accountability. Accountability was best practice in the past, but accountability in the GDPR means that organisations have to identify the risks that they are creating for others and take steps to mitigate those risks. That includes being able to demonstrate to a regulator that they have done that work. Accountability requires someone to be responsible. It requires data protection impact assessments. It requires notification to individuals on how data are being used. It really means that privacy by design has to be baked into all of a business’s practices and processes. It is not an add-on or bolt-on any more; it really requires putting the citizen or the consumer at the centre. I think that is the biggest change, and it is a good change.
If your question is, do I think I have the suite of powers that I need to be able to regulate in this big-data world, I think it is a vast improvement on what we have right now, but the proof is in the pudding as regards our actually having the resources and expertise to get out there and do the work.
Q314 Neil O'Brien: So, it is less about a power and more a resource question from your point of view.
Elizabeth Denham: Yes.
Q315 Chair: What are the consequences or sanctions for breaching that accountability duty under the GDPR?
Elizabeth Denham: Breaching the accountability provisions is subject to all kinds of sanctions. First, we can order an organisation to cease processing the data. We can also impose an administrative monetary penalty of £17 million or 4% of global turnover. That is a significant step up from the fining power that we have now.
Q316 Vicky Ford: Given that many of the organisations that you might be looking at will be international, global organisations, how important is it that you can co-ordinate with your counterparts in other countries? Do you have those powers? What more do we need to do to ensure that you have that co-ordination power?
Elizabeth Denham: It is enormously important, when we have global data flows and big companies with monopolies over the collection of personal data, that the ICO can continue to co-ordinate our enforcement activities not only with our European counterparts but with our colleagues around the world. We obviously do that now within Europe. We also co-ordinate some enforcement activities with the US, Canada, Australia and other countries, so we have the ability to work with our colleagues. That is really important.
With the UK leaving the EU, it is really important that we have the ability to continue to collaborate and co-operate with our European counterparts, because the EU will have the largest block of data protection supervisors on the planet, and they will have a lot of power over making determinations on what the big companies, in particular, are doing with data. We want to be at the table to protect, speak up for and act for UK citizens regarding complaints, breaches and investigations.
Q317 Chair: You say that you want to see trading data behind algorithms to ascertain fairness in different circumstances. Given that many machine-learning algorithms are not fully understood even by the companies deploying them, do you or will you have the skills necessary to undertake that task?
Elizabeth Denham: I hear from many companies and organisations that we might as well just throw up our hands and give up—that it will be too hard to explain machine-learning super-AI systems. We might be able to do it for narrow AI systems, but we cannot do it for super-AI systems—when you start getting into neural networks and so on.
I have heard from other experts, however, that it is an engineering problem that can be solved. We need to follow that lead and look for ways for us at least to be able to explain the basic decision-making and the programme that has gone into the algorithm in the first place. We are working with the Alan Turing Institute to help build a framework on the explainability of super-AI and how we can do that.
We should not give up. AI and big data are not a new game that plays by different rules. The bottom line is that, when we are talking about sensitive personal data and the use of AI to make decisions that affect people’s lives, opaque algorithms are not going to cut it—they are not going to comply with data protection. That might be a question for Parliament to figure out, but we have got to crack that. I am not convinced that there is no way to do it—I think there will be a way. There are many willing experts in this country to find that way.
Q318 Chair: But it is a significant challenge to get there.
Elizabeth Denham: It is a significant challenge.
Q319 Bill Grant: The Committee often hears that accountability is important in algorithms. Bearing in mind that you have already mentioned codes of conduct, ethics boards and certification and seals, what should effective accountability look like, in your view—bearing in mind that you have mentioned three elements? What do you feel effective accountability should look like?
Elizabeth Denham: I think I will know it when I see it. If I went into an organisation to do an audit, I might say, “I want to see your commitment to data protection.” That is really what accountability is. It is not about a tick-box exercise of compliance. I would see an organisation where the board is involved in risk assessments and making decisions about data protection—that is a boardroom issue. There would be somebody senior and independent designated as the data protection officer. I would look at the policies and the training, and at how breach reporting and data protection impacts were part of the everyday business of the organisation. I would also expect that, when asked, the development staff, tech staff and engineers could explain what they are doing and what question it is they are trying to answer.
Q320 Bill Grant: So, you are very clear in your mind that accountability should be embedded in the organisation.
Elizabeth Denham: Embedded.
Q321 Bill Grant: It is a top-dollar one: it is an influential individual on the board or in that range of influence.
Elizabeth Denham: If I rang up an organisation and said, “Who is your data protection officer and they said, “What’s that?” I would know right off that we had a problem. If I talked to the data protection officer and asked about access to the board, board decisions and reports to the board and there was none, that would be an alarm bell. I would also look at the training. If they said, “Oh, data protection training? We did that in 2004,” I would know that it was not an evergreen, embedded value.
Q322 Bill Grant: So, it has to become embedded, like health and safety has in a journey over 20 or 30 years.
Elizabeth Denham: And social responsibility. I would look at the code of ethics for the organisation, too. It is connected to that.
Q323 Vicky Ford: Given that, if they breach it, they could be fined up to 4% of global turnover, there are probably not very many organisations that are acting in that way at the moment. Would you agree?
Elizabeth Denham: I would agree, but I think that more and more organisations are paying attention to their new responsibilities.
On the GDPR, we were at a tipping point in the power imbalance between individuals, companies and the state. The forthcoming legislation is about resetting that balance.
It is going to take time. I have written a blog piece headed “GDPR is not Y2K”. It is not a one-time case of “Let’s fix this for 25 May 2018.” It is an evergreen, evolutionary process to put people at the centre.
Q324 Bill Grant: As I understand it, the ICO is currently looking at ways in which certification seals for algorithms can be applied. How would that work, and how viable would it be? Is that confined to certain areas, or could it be extended over a range of activities with algorithms?
Elizabeth Denham: I think that industries that make wide use of AI decision-making that has impact on individuals are the ones that should be focused on developing a code. That goes with the sector regulator, the industry association and the industry around that. They can do the development. We can be involved—and we have to be involved—in approving the code. Then, either the ICO or third-party certification bodies would provide the seal.
We have never done this before. We are at the beginning stages of figuring out how certification can work. It perhaps makes sense to start with the financial services sector, which has an interest in standards and moving forward with that.
Q325 Chair: So, within financial services, there would potentially be a system whereby any algorithm used in that sector must get that certification in order to be permitted for use.
Elizabeth Denham: Should get certification—and that leads to the trust that is needed. Individuals think that somebody has their back. Individuals think that somebody is watching and taking care of this. The best providers or companies—those that are regulated—would seek certification and a seal.
Q326 Bill Grant: In noting the points about certification, seals and accountability, does an algorithm have a shelf life? Should it be the subject of a review date—or indeed the code that is applied to that algorithm or range of algorithms? Should there be points on the journey where it may become life expired? I do not know how long an algorithm lives for. How often, in accountability, should we review the effectiveness of the codes or indeed the effectiveness of the algorithm itself?
Elizabeth Denham: I am not a computer scientist, but I understand that there needs to be an evergreen review of algorithms, especially machine-learning algorithms, because they are evolving and changing, and they are even creating their own data. They need to be reviewed. How often they need to be reviewed depends on the context and on the design of the algorithm.
Q327 Bill Grant: In your submission, Elizabeth, you suggested that auditability should be built into algorithms. What does “auditability” really mean for an algorithm system?
Elizabeth Denham: When I have spoken to experts in the design of these systems, they say that audit needs to be built in by design. It goes to your previous question—how often do algorithms need to be reviewed, and are the data still the right data, or is a new bias being created in the development of the algorithm?
My understanding is that the audit capability—the ability for somebody to take a look—needs to be programmed in. There are experts, such as those working at the Alan Turing Institute and elsewhere, who could advise you more on that.
Q328 Chair: In the criminal justice system, given the developments in the States, where there is an increasing application on all sorts of decisions that could impact on someone’s liberty, do you think that there has to be a process, in effect, of requiring certification in some way for applications to be used to guide a police force, for example, in how it applies its powers?
Elizabeth Denham: I would hope that a police force that was using AI in making decisions would, first, have human intervention built in. That is a basic requirement—it is a requirement in the law. I would also expect that oversight of the development and use of those tools related to either recidivism or sentencing.
Whatever the decision is, because of the great impact of those decisions, I would like to see certification or at least, in the criminal justice system, those agencies working really closely with our office, because that is where I think public trust is paramount.
Q329 Darren Jones: I wish to turn the conversation to the perspective of the consumer, or data subject, in their experience of using algorithms. My first question concerns the legal bases for processing data subjects’ personal data. I have three points that I hope we might be able to discuss on that.
The first is on whether there are derogations for research purposes in the Data Protection Bill, for example. We had a really interesting conversation with our health panel last week, when we talked about permissions fusing data for research purposes, and how that might then be used for application in clinical scenarios in treating patients. Do you think there is a problem in that extension of the use from research and development into the treatment of patients, potentially?
My second question—before we get on to consent—is about the idea of legitimate interest. Do you see that as a valid legal basis for algorithmic use, and certainly the use of these smart algorithms that learn and process the data in new ways?
Elizabeth Denham: Your first question was around whether the derogation in the Data Protection Bill relating to medical research was sufficient to allow the kind of activity that we probably have going on right now—which is using data at the testing stage and then moving them into the clinical application. That is the issue that we had in Royal Free/DeepMind. What is the legal basis of acquiring medical data—identifiable clinical data—to be used in a trial? There is a problem in our law, because the consent that is needed is at least an opt-out consent by patients for data to be used for the trial or research phase. It is not for the clinical stage. How much data do you need at that testing or research stage before you move into the testing? I think we have a challenge in UK law. Perhaps that is an issue that the future centre for data ethics can examine.
You can obviously use anonymous data. You could use data where the patient has at least given their opt-out consent—just talking about research. However, when it comes to treatment and care, it is really clear what the legal basis is. It is fine there—but I think your question is around what you do at the testing phase. You could use pseudononymous data or anonymous data. You could get consent in some form, but it is not a free-for-all in terms of research.
Q330 Darren Jones: On the question of consent, I think that experts on our panel shared the view that the current consent model is perhaps not quite fit for purpose for some of these scenarios. They have talked about things like broad consent—the idea that you might say to your GP, “It’s fine to use my data for my health purposes,” and data ethics boards or others could make decisions on their behalf as to what those purposes might be. Do you share that view? Do you think that would be a sensible development in what we understand consent to be?
Elizabeth Denham: I think we have a lot of stresses and strains in our current model. The NHS is such a valuable resource. When we are talking about health data, people expect and want data to be used to find new solutions—to save babies and find new pathways for treatment. That is a decision that needs to be made by Parliament. We probably need—we do need—a deep public conversation: a meaningful public consultation about the use of health data and the application of machine learning and AI.
The public need to be behind whatever we decide to do, and I think it would need a change in the law to open up those datasets for more broad research and the use of AI.
Q331 Darren Jones: The other scenario that consumers often interact with are terms and conditions, privacy policies and the various documents that explain what is happening to their information. Do you think there is something different that we should be doing in how we use terms and conditions or privacy policies in explaining how algorithms use personal data?
As a supplementary to that, when consumers have the right to explainability, my understanding from the GDPR is that it is when there is a legal or significant effect. I would be interested in your views on what you think a “significant effect” might be and on how you might then explain that in a privacy policy or set of terms and conditions.
Elizabeth Denham: Starting with my view on terms and conditions and privacy policies, I think the system is broken. Most people do not read terms and conditions. Even my expert staff just say, “Oh, for goodness’ sake—I agree.” Nobody really wants to get to the end of terms and conditions or a privacy policy.
Some companies have moved to a layered notice. You get basic information at the beginning. “This app is going to use your location. Do you agree or do you not agree?” That is a layered notice. The consumer who wants more information can go behind it, and it gets more and more sophisticated. When it comes to children, we really need to do a better job—companies need to do a better job of communicating with children on how apps and websites work.
We need a new system. The GDPR brings in a requirement for plain language that is simple to understand and for clear notices in a way that we have not had. GDPR is a driver for better notices.
When it comes to privacy notices for the use of AI that may have significant effects on the individual, that is the work that we are starting to do with the Alan Turing Institute, which is, “What is a clear notice? What does that look like? How are people going to understand it?” There is a lot of work to be done in that field, and I am really excited about coming up with a framework of notice for AI and significant effects.
You asked me for my view on significant effects. There is guidance on this, which the Article 29 working party—the group of European regulators—has issued on profiling and notification. There is a whole list of significant effects, but it is basically decisions that impact on people’s lives: in the criminal justice system, to obtain financial services, loans, housing and healthcare. Those are some examples, but there is detailed guidance that I can send you.
Q332 Darren Jones: My last question is on transfers of data. As we know, many of these algorithms, or certainly companies that use software provided by suppliers who might use algorithms by others, involve a lot of movement of data. You have talked about global data flows. The Government will obviously seek to get an equivalence decision as part of its Brexit negotiations.
You have mentioned the GDPR quite a lot, and the article 29 working party. Is it your view that, once we have equivalence, which everyone agrees we need to have, in order to maintain that, we will need to track and copy EU jurisprudence on GDPR application? If that is the case—I think this is an extension of Vicky’s question—do you feel able, as a third-country data supervisor, to influence those conversations at an EU level?
Elizabeth Denham: I have the experience of once being a third-country regulator in Canada, because Canada is an adequate jurisdiction to the EU. I can tell you that Canada is not horrendously influential in what happens in the EU. That said, the ICO has a strong influence around the table with our European Union counterparts, because we are a really large authority. We are probably the largest data protection authority in Europe and perhaps globally.
However, if we were a third country, and even if we had an equivalent law, if we are not a decision-maker in trans-border cases and we are not making decisions, then we are not going to be as influential as we are now. The EU is a large block of data protection supervisors.
We will still be influential, because we can participate through bilateral agreements that allow us to do joint enforcement. The bilateral agreement with France allows us to jointly investigate a breach that affects UK citizens. Again, that will not be the priority of the European data protection board, which is going to be an adjudicative board, not an advisory board.
Q333 Darren Jones: Just to build on the first part of that question, the European data protection board will obviously give guidance as this new area of law develops. From your perspective, to maintain equivalence, will we just need to comply with that, or will we be able to deviate from it?
Elizabeth Denham: If we are a third country completely outside the EU, and if we have an adequacy finding, if we want to maintain that finding, we will have to have at least equivalent—essentially equivalent—law. Does it have to be cut and paste and exact? No, but the adequacy findings will have to be reviewed on an ongoing basis. If we significantly change UK law and if the European Commission believes that that makes us fall below the essential equivalency standard, that could put our adequacy finding at risk.
Q334 Vicky Ford: I wanted to come in when you were talking about medical ethics, and especially medical data. Is that the area where you see the new centre for data ethics having a speciality? I am trying to get what that centre specialises in. How does it work with the Alan Turing Institute and with yourselves? How does it work with other organisations for medical ethics? How do you see the jigsaw fitting together?
Elizabeth Denham: I very much see the new data ethics body as being a place to ensure that there is meaningful public dialogue. Perhaps the law needs to change. That is not something that I would do. I would not create a public dialogue about how the law needs to change. I am a creature of statute. I am a regulator.
There needs to be a discussion in this country about the use of data for medical research. That is my view. This could be one of the priority areas for the centre for data ethics—it could take a look at that.
I also think it could advise regulators. It could convene a group of regulators to talk about a specific issue. It could do horizon scanning and expert reports.
We at the ICO and many other regulators are concerned, however: if it becomes a decision-making body, that will create complexities for organisations, as they do not know whose decisions they are supposed to be following. You do not want to crowd the space, but you want to add to the space, bring some co-ordination and focus on accountability and data governance.
Q335 Graham Stringer: How is your investigation into the use of data analytics in political campaigns going?
Elizabeth Denham: It is probably the most complicated investigation that the ICO has undertaken. We are looking at the relationship between political campaigns, data analytics companies, social media platforms and others that are collecting and sharing personal data to micro-target individual voters or individuals. The purpose of the investigation, from a data protection perspective, is to see what is actually happening now and to be able to make that transparent to the public. We need to figure out whether the public reasonably expect their data to be used in the way they are used.
What has happened in the past few years is that, with big data, cloud computing and analytics, the old-fashioned data collection and analysis that campaigns and political parties have always done has shifted significantly, perhaps without taking the voters with them. I do not know—we will see—but our report is more about transparency and being able to demonstrate to the public who is involved and how their data are being used.
At the end of the day, the question may really be for Parliament, or it is an ethical question. The whole behavioural advertising model, according to which we all understand how we get shoes and soap, and “If you like this movie, then you are going to like that movie”—maybe we are okay with that behavioural advertising model, but if that same model is applied to our democratic institutions and our elections, in the political context, what do we think about that? Is that okay, or does there need to be more transparency and choice? We hope to reveal what is happening in the use of data. We are not investigating fake news, bots or campaign financing; we are just looking at data protection issues in micro-targeting.
Graham Stringer: You say you are not investigating bots.
Elizabeth Denham: Not directly.
Q336 Graham Stringer: Are you concerned that 5% of Twitter accounts, or whatever it is, appear to be bots? Is that part of your concern?
Elizabeth Denham: I think it is generally concerning. I think that it points to more care, work and action needing to be taken by some of these big companies to ensure that there are not fake accounts out there. It is only related in tangent to our investigation. We are really looking at how data are crunched and shared, and how they may be manipulated to target people and create, in lookalike environments, something almost like ‘transactional politics’. What is the impact of transactional politics? That is a legal but also an ethical question.
Q337 Graham Stringer: When will you report, or when do you expect to report?
Elizabeth Denham: We expect that the report will be done some time in the spring.
Q338 Graham Stringer: So, fairly soon.
Have you had particular difficulties getting information out of the bodies involved in campaigning on the European referendum?
Elizabeth Denham: Yes. We have had to serve an information notice on some parties, which means that we are trying to compel information that they will not share with us.
Q339 Graham Stringer: When you say “parties”, do you mean political parties or parties to the referendum?
Elizabeth Denham: Sorry—campaigns involved in the referendum. It is in the public domain that we have served an information notice on UKIP, because we need to be provided with the information. We are now before the tribunal.
Q340 Chair: Is this only UKIP, or have you served other notices?
Elizabeth Denham: In terms of political parties, that is the only notice that we have served.
Q341 Chair: What about other organisations?
Elizabeth Denham: There is another one that is in train.[1]
Q342 Chair: Are you able to say who that is?
Elizabeth Denham: No, but it will be published soon.
Q343 Graham Stringer: I do not know exactly the extent of the inquiry, but will it cover whether you think the current electoral regulatory bodies such as the Electoral Commission are equipped and competent to deal with elections and campaigns?
Elizabeth Denham: I think that is a question for the Electoral Commission.
Q344 Graham Stringer: But will you be looking at how the Electoral Commission itself performs and deals with data?
Elizabeth Denham: No. We are dealing with the use of personal data in campaigns and elections. We have a memorandum of understanding with the Electoral Commission, and we are working together. It has an inquiry looking into the financing of the referendum. Again, that is a separate investigation, with a separate focus.
Q345 Graham Stringer: Before you appeared, I had to look up what a “filter bubble” is. Are you looking at how filter bubbles are involved in political campaigning?
Elizabeth Denham: When you say a “filter bubble”, are you talking about an echo chamber?
Graham Stringer: I have no idea whether I am talking about an echo chamber.
Elizabeth Denham: Definitions.
Q346 Graham Stringer: What I am talking about is when search engines know who you are when you ask a question and they direct you a particular way.
Elizabeth Denham: Yes, we are. That is part of looking at how data are used, to be able to target an individual with a message. The other way of expressing that filter bubble is a lookalike audience. Data are provided through a social media company to say, “These people look like you. They could be like you.” That is part of behavioural advertising or micro-targeting. We are looking at that.
Q347 Graham Stringer: Are you able to distinguish, or will you distinguish, between what is market research and what are political campaigns?
Elizabeth Denham: Both market research and political campaigns are subject to data protection legislation, but they run by slightly different rules. You probably know those rules as well as I do, but market research is about obtaining information by a political party or a company that is truly doing research, as opposed to delivering a marketing message. Once it is a delivery of a message, by a political campaign, a political party or a company, saying “Hey, use our services,” it is no longer market research, and there are different rules. I can write to you to distinguish all the rules, if that is helpful.
Graham Stringer: Thank you.
Q348 Chair: You have written a blog on this, as you mentioned. Are the rules for the use of personal data in political campaigns clear enough? Is that one area that you will be commenting on?
Elizabeth Denham: It is interesting. We have met all the political parties in the context of this investigation. I think the political parties agree on one thing: that more and more data are available to them, but there is uncertainty on the use of those data. Perhaps this investigation will also recommend a code of conduct on which political parties can agree. You want a level playing field; you do not want one political party to be using data in a way that slightly crosses lines, while the other is not.
I think the political parties know that the technology has changed so quickly, and the use of third parties, the availability of analytics and the power of the social media platforms have taken over. We need to take a step back and ask what people will accept.
Q349 Chair: Would that code of conduct in effect be a voluntary agreement between parties, or does it need to have some force behind it?
Elizabeth Denham: It could perhaps start with a voluntary agreement, but it should end up being a code of conduct that falls under the GDPR codes I was talking to earlier.
The political parties have said that they are open to saying, “Okay, where’s the line?” I do not think there is a huge smoking gun in our investigation in any way, shape or form, but I think it will reveal that practices have changed really quickly. It is time to ask, “Are we okay with that?”
Q350 Chair: What is right and what is wrong—or what is allowed and what is not.
Elizabeth Denham: What is allowed and what should not be allowed. There is so much value in political campaigns and political parties talking to the electorate. They have to. They have to get people out to vote. They have to encourage people to be involved in our democratic system in public debate. We know that data have to be shared and used to deliver messages, but how far is too far, and how much data do political campaigns and political parties need to have to be able effectively to deliver their message?
Q351 Vicky Ford: I could listen to you all day—I think we are really lucky to have you in the UK—thank you.
What I have heard you say in this session is that you have the legal tools, or the GDPR is a good step in giving you the legal tools, and that we need to ensure that you continue to have international influence. I just wanted to check that there are no other resources or tools that you think you need to do this job that we should be aware of.
Elizabeth Denham: I really think it is important that Government believe that they have to push for as close a relationship as we could possibly have with Europe when it comes to data, because data underpins everything that we do. We are talking a lot about commerce, but there is also law enforcement co-operation. When it comes to criminal justice and national security, we need to be able to share data. It is really important that the Government have said, “We are bringing in the GDPR and we are transposing it to UK law. Plus, in the Data Protection Bill, we have gone further in bringing in the law enforcement directive and rules for national security agencies from convention 108.” That is really good, but we do not want those barriers to data flowing. We also want the UK regulator to continue to be influential in Europe and beyond. I look forward to that.
Chair: Thank you very much indeed. It has been a fascinating session, and we really appreciate your time.
Witnesses: Margot James MP, Oliver Buckley and Andrew Elliot.
Q352 Chair: It is very good to see you. Thanks for coming along this afternoon. Will you all introduce yourselves before we start the questions?
Margot James: Yes. Thank you very much, Chair. I am Margot James, Minister for Digital and the Creative Industries.
Oliver Buckley: I am Oliver Buckley and I am deputy director for the digital charter and data ethics in DCMS.
Andrew Elliot: I am Andrew Elliot, and I am the deputy director responsible for the Data Protection Bill in DCMS.
Q353 Chair: Thank you. I will start the questions. The growing use of algorithms of increasing sophistication poses new challenges for identifying and preventing bias. We have had a lot of discussion around bias and discrimination. What are the Government doing to resolve these challenges?
Margot James: The Government take these challenges extremely seriously. We regard the emergence of this technology as offering a great opportunity in many areas of policy and public life. At the same time, we are acutely aware of the challenges that the emergent use of these algorithms, in association with other forms of artificial intelligence, presents.
There is quite a wide range of measures that the Government are taking to protect the individual, the individual’s privacy and their rights to appeal in the way that these decisions are starting to be made using algorithms. As you have probably been discussing this morning with the Information Commissioner, we have the centre for data ethics and innovation in process—it is about to be established.
Q354 Chair: What is the timescale for that?
Margot James: I hesitate to put it as a number of months, but it will definitely come into existence in its interim form within the next few months. We are in the process of consulting on its remit and, as you know, we have been given funding, at least for a three-year period, of £9 million. Initially, as an interim measure, it is going to be an official committee within my Department. Over the next two years, we aim to put it on a statutory footing.
Q355 Chair: So, you imagine it will be established by the summer, for example by the summer recess.
Margot James: Yes. I would be very much hoping that it will be established—
Q356 Chair: And then it would be a permanent feature of the landscape through statute in due course.
Margot James: Yes, it will be, indeed. We recognise that its functions will be very important.
Q357 Chair: Could you or perhaps Andrew say something about how the Data Protection Bill prevents bias in the application of algorithms?
Margot James: Yes. I might invite Andrew to talk about the specific issue of the prevention of bias. I recognise how important that is. We have evidence from elsewhere in the world of algorithms that have introduced bias that none of us would want to see. We very much recognise that we explicitly need to tackle bias.
Andrew, would you like to give any further detail on that?
Andrew Elliot: There are a few things that are relevant. The Data Protection Bill sits alongside the GDPR. The two of them are interlocked. Within the GDPR there are requirements that data processing is fair and decision-making using automated processing is fair. Therefore, there is a requirement in there to address issues such as bias.
Additionally, if a data controller was going to do automated decision-making and there was a risk that the decisions that arose from that process would have a significant impact on the individual, there is a requirement that they conduct an impact assessment. Within that impact assessment, they must address that sort of issue. If a particularly high risk arises from that, there is also a duty within the GDPR, at article 36, to give notice before processing takes place to the Information Commissioner to show that they have considered all these factors.
Additionally, if an individual has a decision taken about them through an automated process, there is a requirement that they are notified that a decision has been taken, that they have the decision explained to them and that they have the ability to ask for that decision to be reconsidered with human intervention. Overall, there are a number of mechanisms that speak to the bias issue.
Q358 Chair: We heard about the notification process from the Information Commissioner. This will presumably be quite an additional workload for the Information Commissioner. Has there been an assessment of what impact it will have on the organisation?
Andrew Elliot: The Information Commissioner needs to be fully resourced to operate the new regulatory regime, and the Government remain committed to that. We are in regular dialogue with her to ensure that she is adequately resourced. Recently we have had some extensive discussions with her and with the Treasury to agree how we can help further with that. An announcement was made two weeks ago concerning the pay flexibility arrangements that we have given to the Information Commissioner to allow her to attract the right staff. Her organisation and her support are funded through a fee structure, which the Data Protection Bill provides. There will be a tiered fee arrangement, dependent on the sort of data that are being processed and the size of the data controller.
Chair: The fee paid by the organisation that is using the algorithm.
Andrew Elliot: Yes. That is correct. There will be three tiers of fee: the lowest tier for small processors, up to a top tier for large organisations.
Q359 Chair: Back on the centre for data ethics, Margot, you said that there is a consultation under way with regard to its remit. Do you have a view about what its remit is likely to be regarding algorithms?
Margot James: Yes: it will have a specific remit in respect of algorithms, how they are used and wider forms of artificial intelligence. I would expect it to have that to the fore within its remit. Its remit will also encompass the basics of data management and processing and, in respect of algorithms, how those algorithms are applied to a wider context of decision-making.
Oliver Buckley: If I could just clarify, the consultations that have been happening to date are of an informal nature at this stage. We expect to be launching a formal consultation in future.
Q360 Chair: When will that happen?
Oliver Buckley: We do not have a precise date for that yet.
Q361 Chair: Is it imminent?
Oliver Buckley: We would expect it to be over the course of the next few months.
Chair: That is suitably vague.
Margot James: I would like to add that I see this as an extremely important priority. If we are going to get it launched in its interim state halfway through the year, we need to get on with the formal consultation. There have of course been informal consultations with all stakeholders—but, yes, we do need to get on with it. We get that message from the Committee, and I agree with you.
Q362 Chair: What is the extent of your Department’s responsibility, particularly the overlap with other Departments, such as health and the use of algorithms within health, justice, the criminal justice system and so forth? How far do you reach into other Departments’ areas of responsibility?
Margot James: I will invite Ollie to give his answer to this, but clearly we have to interface with all other Government Departments where there is the potential for algorithms to affect decision-making in their areas. Their views will be encouraged. Indeed, we have been consulting other Government Departments.
The use of artificial intelligence in healthcare has been exempt from previous regulations in this area, so I think that the Department of Health has its own specific role and responsibility for the area, but we will of course be in dialogue with it.
Oliver Buckley: We expect the centre to be cross-sectoral in its remit and to provide advice to policy makers and regulators in all relevant—
Q363 Chair: And your Department will be responsible for that centre.
Oliver Buckley: Correct.
Q364 Chair: But, on issues such as the application of algorithms in the criminal justice system, where there are obviously big human rights issues at stake, how much responsibility will you and your Department have for issues of that sort, or will it be very much down to the Ministry of Justice to get the regulatory framework right?
Oliver Buckley: It would be down to the relevant Department to account for the advice that is given to it. It is worth noting that there do exist, within Government, communities run by the Government Digital Service that bring together data leaders from across Whitehall to discuss pertinent issues and share best practice. In 2016 the Government Digital Service published a data science ethical framework, which is designed to ensure that, where data science projects are being developed in Departments, due consideration is given to potential risks in the approaches they take, and that they consider the appropriate mitigations.
Chair: Good.
Margot James: It is important to bear in mind that some of these bodies and policies are still in an embryonic state, but, once they come to full fruition and become independent, all Government Departments will have to defer to those bodies. The benefit of them is their independence. Although at the moment perhaps Government Departments do not have a free run but certainly more independence, once these bodies establish themselves as independent entities, all Government Departments, including our own, will have to defer to their rulings.
Q365 Graham Stringer: This weekend I got a new phone. I signed the consent form, and I asked the young man who was selling me the phone whether anybody ever asked to look at the detail of what they were signing. The answer was that he had never ever had one person ask to look at it, and he did not think his colleagues had. Do you think we need to change to a different model of consent, if it is now so complicated that people just want their phone, so they do not look at it?
Margot James: You raise a very good point. The way that this issue of consent is evolving is to acknowledge that, particularly in respect of algorithm-based decision-making, citizens should be required to give more active consent than they have perhaps had to hitherto. That is a strain that has come out of various reports.
The Chair mentioned the health sector. The Caldicott review has obviously led the way in establishing new forms of consent. Indeed, the opt-out that she has recommended comes into force in May this year. I see that as having relevance for other areas, too.
Q366 Graham Stringer: Particularly within the healthcare sector, the concept of implied consent has been developed. Do you think that could be applied elsewhere?
Margot James: I am more aware of active consent and the right of patients to opt out of having their own data used for any purpose other than their own treatment than I am in terms of implied consent. I think that things are moving away from implied consent towards a more active form of consent.
Do you want to clarify that, Andrew?
Andrew Elliot: Under the GDPR and the Bill, health data is a special category of data. Therefore, it has an extra consent requirement associated with it. Normal personal data under article 6 of the GDPR has a higher level of consent than we currently have under the Data Protection Act, but under article 9 of the GDPR health data require explicit consent.
With your phone example, that may not have contained health data, but the difference with GDPR and article 6 is that you have to have an affirmative action. Previously you could just have a default tick box. That is a change in respect of regular personal data. Then, with health data, it is explicit consent. That does not mean ticking a box or pressing a button—you have to do something more than that to demonstrate that you have collected the data subject’s consent and sought their understanding as to what they are consenting to.
Q367 Chair: Consent for what use in healthcare? The implied consent relates to direct care, where there is a sort of acceptance about clinicians sharing information involved in your direct care, but the implication is that you will agree with that, because it is in your interest that the relevant clinicians share the information. Are you saying that that is changing?
Andrew Elliot: No—that continues to exist. The Bill provides for circumstances where consent is not required, where it is in the substantial public interest, which is provision permitted by the GDPR. The GDPR says that a member state may set out what it considers to be in the substantial public interest. The Bill makes provision for that sort of clinical provision that you have just described. Consent is not required in that circumstance.
Q368 Graham Stringer: I understand that, where the data are used for research, marketing or whatever else, it is getting tougher to use that information in that way, but if it is in the person’s interest, as it is in the patient’s interest to have the information shared with other clinicians, do you see any cases where that could be extended outside the health world?
Margot James: I see no reason why not. Just as there are special considerations in the use of health data, there will no doubt be special considerations in the use of other forms of data as well, but given that this issue has always been so important in the healthcare area there is no reason why other sectors cannot benefit from that body of knowledge.
Q369 Graham Stringer: When data are shifted between companies or there is a merger between companies, does that invalidate the previous consent?
Margot James: Is your question pertaining to healthcare or in general?
Graham Stringer: Generally.
Margot James: That is a very interesting question, which I would like to reflect on further. I do not know whether you have already reflected on this.
Andrew Elliot: It depends on the specific facts of the case. If you had two entities and one was to acquire the other, if the data were to move as a consequence of that acquisition from one entity to the other, the data have moved from one data controller to another. The GDPR and the Bill require the data subject to be given notice of that transfer. However, it is possible that the merger does not actually involve the moving of that data, in which case there is no specific requirement. Obviously, the new owner would still have to comply with the law.
Q370 Graham Stringer: When the Information Commissioner fined TalkTalk for its breach in 2015, there was a concern that the hackers obtained data that TalkTalk had acquired from another company. What are the implications of that?
Margot James: Looking back to that case, I think that the fine pertained to the inadequate controls that TalkTalk had to safeguard that data, rather than the fact that the data had moved from one company to another. As Andrew said, those controls are going to get more stringent with the passage of GDPR.
Q371 Graham Stringer: The Data Protection Act 1998 allows individuals to request information from an organisation. It can tell them to stop processing their data. If individuals do not know about what the companies are doing in processing their data, does that not mean that the current legislation does not go far enough? Do you think that the new legislation should take it further in informing individuals?
Margot James: Yes. I think that is one of the purposes behind the updated legislation that is going through Parliament at the moment. The updated Data Protection Act will require the active consent of individuals that their data be collected. People must be informed if decisions are going to be made by algorithms rather than human management. Companies must make them aware of that. As Andrew said earlier, they will then have the right of appeal should they disagree with any decision made by an algorithm. Yes—we recognise that the existing legislation does not go far enough in today’s world, which is why we are updating it.
Q372 Stephen Metcalfe: We have spoken a little bit about bias, the ability to challenge bias and the right to appeal. What lies at the heart of this is transparency. There is a lot of talk around transparency and algorithms, but there is not necessarily clarity about what that means. Does that mean the ability to examine the code itself, so that you can unpack the black box? But there is concern that, the larger the AI or algorithm, the more incomprehensible it is, so that is a complete and utter waste of time. Or there is the other side, which is that you could demand to know how that decision was arrived at. Do the Government have a view yet on those models—if either of them—or on what transparency actually means in terms of algorithmic decision-making?
Margot James: I do not see those two modes as necessarily being mutually exclusive. You rightly point out that some of these decisions will be based on algorithms that, regardless of how transparent they might be, could be almost incomprehensible. The Information Commissioner’s role is very important in holding companies to account, not just for having somebody in place as an accountable individual who is responsible for information. It is not enough for companies to make people aware that decisions are being made by an algorithm or an automated decision-making process. Companies will have to make people aware, as much as possible, of how those algorithms have been arrived at and what the basis is, in everyday English, that lies behind the decision-making mechanisms.
Q373 Stephen Metcalfe: Should that individual who is responsible within a particular organisation—I think you said for the data, but responsible for the writing or the controlling of an algorithm and how it functions, I assume—be accredited in any way? Should they be applying standards that have been adopted, if not across the whole algorithm industry, at least across their sector, so that you know that the algorithm is being developed in line with certain guidelines, which can then be challenged by the Information Commissioner?
Margot James: That sounds to me like a very natural course of development, but I think that, in the first instance, it would be for the centre for data ethics and innovation to do some work on that. It may well be that, early on in the centre’s life, it will look at that. These individuals are going to be quite exposed, and I think they will need much of what you have just outlined. However, I think it would be for the centre to advise on that.
Q374 Stephen Metcalfe: Perhaps to work with the British Standards Institution to develop a set of guidelines.
Are we doing this quickly enough? There is an explosion of algorithms and automated decision-making, yet we are now talking about setting up things to look at perhaps introducing standards at some later date. Are the Government and the development of the industry out of step with each other?
Margot James: I will invite comment from Ollie on this, because he has been involved in it for a great deal longer than I have been.
I would say that there is now a great urgency. I would also say it is important that we get this right. Sometimes, rushing is the enemy of getting something right. As ever in Government, it can be frustrating that developments take their course and gather momentum, and it sometimes feels like Government are playing catch-up.
I do not know whether you want to add anything to that, Ollie.
Oliver Buckley: Just to say that, as far as we are aware, the new centre that we are creating is the first institution of this type anywhere in the world. As the Minister said, we are acutely conscious of the need to move quickly with that.
In this country we are also in a very strong position because we have a depth of expertise in industry here, as a world-leading centre for the advancement of these technologies, with companies such as DeepMind, based in London, that are giving active consideration to these sorts of ethical issues. Our universities are also some of the leading centres of thinking. One of the things that the centre can do, which will not be starting with a completely clean sheet of paper, will be to help us to harness this existing expertise.
Margot James: That is a very good point.
Q375 Stephen Metcalfe: I have a final question regarding the Cabinet Office and the publishing of the ethics framework. Are you aware of anything being developed in the private sector on its own, or will it need a push from Government?
Margot James: I am obviously aware of the ethical framework and the ongoing consultation on that framework. Is your question about whether there is a private sector organisation that is also engaged in establishing these principles?
Stephen Metcalfe: Yes.
Margot James: I do not know the answer to that. Do you?
Oliver Buckley: There are a number of industry-led initiatives. This is at a global scale, that we are aware of. There is the currently quite nascent Partnership on AI—some of the big tech firms, civil society organisations and others have come together to look into issues particularly around the ethical dimensions of this new technology. There is the IEEE process. There is also a set of principles—the Asilomar principles—which have been industry developed and led, to which we are giving consideration as we develop our own thinking.
As to whether we might hope for industry here to coalesce around one set of standards, that is the kind of issue that we hope the centre will be able to advise on.
Q376 Chair: Before I bring in Darren, I will quickly return to the question of bias. We have talked about how an individual who believes that there has been bias in the decision that has emerged from the application of an algorithm can challenge it and ask for an explanation. Where the situation revolves around an allegation of indirect bias—perhaps based on postcode or where they are from, rather than a claim of direct ethnic discrimination, say—how effective will the system be in getting an explanation that gives a proper, understandable answer to the individual? I am happy for Andrew or Ollie to answer that if necessary.
Andrew Elliot: I am not sure I can answer that terribly effectively. The requirement will be that the data controller who operates this automated decision-making kit must provide meaningful information. The question is really what satisfies that requirement for meaningful information. We would expect that people would be given sufficient information to be able to have a reasonable understanding of why the decision fell the way it fell and therefore to be equipped with sufficient background information to be able to challenge it. I do not know whether that would necessarily be sufficient to identify bias. The difficulty is that it may be that the controller themselves have not spotted that there is a bias in the dataset that they are operating.
Q377 Chair: The thing is, bias can be very real, even if it is indirect. We want to ensure that there is an effective mechanism to get to the bottom of concerns of that sort.
Oliver Buckley: On a forward-looking thought, we have not mentioned the potential that AI and data-driven technologies have to help expose bias where it was not clear or evident before and to provide tools for mitigating it. We would certainly anticipate the centre taking an interest in the development of those tools, working with partners outside.
Another point is simply that these technologies are evolving, and so, too, are the questions. There will be increasingly novel questions in this regard. Again, this is where we hope the centre will be able to provide that kind of forward-looking advice to regulators and policy makers.
Q378 Vicky Ford: We have just met the Information Commissioner, and she was very clear that she saw the centre for data ethics as a deliberative organisation, not an adjudicative one—one that will suggest the framework and the way forward and make recommendations, but that will not itself be the regulator. This is how we would put Britain at the forefront of ethical thinking on AI. Do you share that same vision, seeing it as a deliberator and thinker itself, not the regulator?
Margot James: Yes, I do share that vision. That is not to say that regulation in this area might not be required at some point.
Vicky Ford: Exactly.
Margot James: In fact, the Data Protection Bill provides for secondary legislation at a later date, should the need arise for greater regulation. I would not rule that out but, at present, I would share that vision, yes.
Q379 Vicky Ford: The regulators are here, and there is a thinker and adviser there. Therefore, everybody knows their roles.
Andrew Elliot: We are very careful to ensure that the Information Commissioner’s regulatory role is protected. Largely, her functions are derived from the GDPR. Ensuring we have full and correct implementation of GDPR is essential to the future free flow of data. Therefore, as we develop the role of the new ethics centre, we have to ensure that we do not step on her toes and create a problem of that sort.
Q380 Darren Jones: I will ask a few questions about the value of personal data as an asset. We have heard in previous hearings that, certainly when you think about the amount of data that the national health service holds, probably uniquely in the world, when we provide access to that data, that has significant value to it. I just wonder, Minister, whether your Department has that on the agenda to try to understand what the value is of data that we hold as the state, and whether we ought to be doing something with it to realise that value.
Margot James: That is very much a live issue that you raise. It has many facets. There is the fact that the Government, across the board, and not just in health, have access to a vast amount of data, as does the private sector. It undoubtedly has a value but, in its raw, unanalysed state, its value is of course quite limited. When focus is applied to it, it of course has a huge increase in value.
There has been some movement in the budget about the Ordnance Survey service and the value of the data that exists within our mapping, which the Ordnance Survey organisation has quietly accumulated over many decades. That has considerable value. Does it mean that Government should exact a price for the benefit of the taxpayer by releasing that information, or, conversely, could you argue that the value will be realised by the commercial exploitation of that asset—under strict regulations, of course? It is a difficult question to answer in terms of how you should realise that value and whether you should put a price upon it.
You mentioned health in your question. I was struck by the arrangement between a private company and the Oxford hospitals trust for the exploration of how algorithms can help patient care in that trust. That is going to be a commercial relationship. Oxford has negotiated an equity stake, I believe, for the use of the data that have been accumulated through patients over the years. I think that, in the end, we will have to have a balance between ensuring that, where the data have a huge value, that value is realised to the benefit of the organisation financially, and the greater good.
Q381 Darren Jones: Do you think that the Oxford example, which sounds encouraging—
Margot James: Yes.
Darren Jones: Has it learned lessons from the DeepMind example, where, for the treatment of kidney disease, access to all that health data was provided free of charge?
Margot James: Quite possibly. I could not be specific about that, but that is quite possible. One other thing we should bear in mind: for years we have been aware of the value of NHS data. It is a single huge insurance system, probably the biggest in the world, and the richest source of data in the world, but we have not mined it as effectively as we might have done, and there is no doubt that companies such as the one that you have mentioned have some hugely important skills and expertise to bring to enable us to realise that value for the benefit of patients.
Q382 Darren Jones: One thing that DeepMind said, which I think is really important, was that the data it was given were not useable. It had to invest a huge amount of money and resource into cleaning them, tagging them and making them machine readable so that it could do the work. Do you think there is a role for Government in doing that? Will you be bidding to the Treasury for investment so that the public sector can do that work in making its data useable for algorithms?
Margot James: I think you would have to ask a Department of Health Minister, if that question particularly pertains to health. I have certainly considered that. On balance, this expertise and this capability are evolving globally at such a pace that I fear that, if one were to try and establish such a source of expertise within the public sector, we might be holding ourselves back. That would be my response.
Andrew Elliot: The Government have released 40,000 datasets under our open data programme. We have had to change the way we hold some of those datasets to achieve that. There have been benefits. Although the data are given away without charge, they are given away in a manner that has benefits for the UK, for citizens, who can access data that they previously could not access and in innovative ways.
There has been a revolution in the way people use public transport in recent years, largely driven through access to this sort of data. There are citizen benefits, but also economic benefits while we develop skills in this country to process data, and the tech skills that are required, which attracts the sorts of investment and businesses that we need to see.
Oliver Buckley: One of the recommendations from the AI review of last autumn was that we develop the concept of data trusts. The idea behind data trusts is that they facilitate sharing between multiple organisations, but do so in a way that ensures that the proper privacy protections and other relevant protections are in place, that there is a governance of the data, which ensures that the voices of interested parties are represented in that governance, and that there is a fair sharing of the value that can be derived from those data. That was a recommendation in the autumn, and we are beginning work now to develop that further, with an aim of piloting data trusts in future.
Q383 Darren Jones: Excellent. I would encourage you to do that cross-departmentally. It concerns me slightly, Minister, when you say that we will have to ask the Department of Health. Likewise with the conversation earlier about whether the data ethics centre should sit within your Department or, as I think is the case with the Government Digital Service, in the Cabinet Office, this cross-departmental strategy is really important, especially when it comes to funding decisions on how we get the most value out of the data that we own. So, do power through on that cross-departmental influence.
Margot James: I take what you say to heart. That is absolutely right. I felt that the health question that you raised was, to a certain extent, a matter for the Department of Health, in that I felt that it would probably have considered this in more depth, as it is responsible for the NHS. As to whether it would be appropriate for the NHS to try to develop this expertise on its own, I certainly do not mean it in the wider context of our responsibility in cross-departmental working on the wider picture.
Q384 Darren Jones: I have one last supplemental. I will be quick—I am conscious of time. It is an extension to the earlier question about the Data Protection Bill. As a member state, we will be able to define what is in the national public interest for derogations from consent—for example, around the collection and use of health data. The example that I am imagining is that, when you go to the GP, you obviously agree to give the information to the GP. One of the issues that we have been grappling with in this inquiry is how you take consent from patients when that routine health data is shared for research purposes, but where that comes back to making clinical decisions for the treatment of the patient. Does the Bill deal with that issue? If it does not, do you think we need to think about new forms of consent for that flow from personal data to research and back again?
Andrew Elliot: The way the Bill operates in respect of health data for research has been the subject of quite a lot of debate in the House of Lords during Committee stage and Report stage. Our concern has been to ensure that the medical research community, largely championed through the Wellcome Trust, which made significant representations on this, are able to continue their world-leading research.
We have agreed—we tabled amendments—to change the way we derogate from article 89, which means that consent is not always required where there is an appropriate oversight body that has agreed that the medical research meets certain standards. Universities already have processes in place to ensure that their ethics bodies have agreed this sort of thing.
There are now means within the Bill to allow medical research to be conducted with the right safeguards, but without consent getting in the way, where it is appropriate. The question is, having established that, how do those data get transferred into day-to-day clinical use? I think it depends on what the output is. If you have now created an artificial intelligence system based on that data, the first question is, are those data that the system is now operating on, which it has now managed to develop from the research base, still identifiable, or have you managed to anonymise them? If you anonymise them, you clear the way, effectively because you have protected the individuals and you are no longer dealing with what amounts to personal data any more. You have used personal data as the raw material to build a system that is no longer built on continuing personal data to be able to be applied in a clinical situation.
Q385 Darren Jones: But if you can reidentify them.
Andrew Elliot: If you can reidentify them, we are back to the question of whether or not consents are required, but the Bill provides further protections against unlawful reidentification.
Q386 Chair: Before I bring in Vicky, I will quickly return to the question of where value accrues from data. Andrew, you have described how enormous value to the wider economy has flowed from the release of big data from Government, for example in travel, and all of the innovation that has emerged from that.
On the other side of the coin, big concerns are expressed about this enormous value of data within the NHS, and that the value could just accrue to some big conglomerate in the US, for example, with no direct benefit to British patients or to the wider British economy. How do you work out your philosophical position on that dilemma? How do you work out what the approach should be in healthcare to ensure that the UK benefits from this extraordinary, unique resource that sits in the NHS, unmatched anywhere in the world?
Margot James: I would go back to the case that I outlined just now, between the private company and the Oxford University NHS trust. I think that is a good model to follow, whereby there is the patient good that will come of these data being used effectively, subject to all the controls that we have been discussing this morning, and there is a commercial gain to the company that provides most of the expertise. I think that there should be a way built in to ensure that the trust, in that scenario, shares in that commercial gain.
Q387 Chair: So, collaborations can deliver benefit to patients in a particular trust, for example. Also, the algorithm that might have developed out of the collaboration could be exploited, in the best possible sense of the word, all over the world. Should the NHS be ensuring that it gets benefit and financial payback from the enormous opportunity that might have emerged from the use of those data?
Margot James: Yes. You make a very good point—that the commercial benefit is as yet to be determined.
Q388 Chair: Are you getting inspiration?
Margot James: I do not mind fresh thinking. I will read this out and then return to what I was saying. It is not just about financial benefits in terms of revenues for private companies, but also money saved by Government. Yes, I agree with that. Most importantly—I have already mentioned this—there are the social and health benefits for patients. I have mentioned that; so, that is true.
Your question was about the commercial benefits that could flow, not just from realising the opportunities that the data provide in the UK but the modelling that might go elsewhere in the world. I can safely say that we must bear that in mind. Indeed we must: it is not just a commercial return within this country. The potential is enormous for global—
Q389 Chair: So we must ensure that we do not overlook the enormous value that the NHS sits on in terms of data.
Margot James: Yes, indeed. We have to strike a balance, have we not? We cannot be so focused on sharing commercial return that we deter the technology company whose expertise we clearly need.
Q390 Vicky Ford: I have a group of questions that pick up on what other people have said in this investigation on algorithms. The first is on social media and its use in elections, which the Information Commissioner is looking at. She has told us that there is also an important role for the Electoral Commission. She has also suggested that this may be an area where a code of conduct agreed between political parties could be helpful, because it is such an evolving situation. Could you possibly take that away and think about a code of conduct developing in the UK for the use of algorithms in social media when it comes to electioneering?
Margot James: Yes—I would certainly be raising this issue with the Information Commissioner at my next meeting with her. It is very much a live issue. As you say, she has been researching it, and she is going to be coming to us with her recommendations. A code of practice between the parties sounds a good idea and something that we will consider.
Q391 Vicky Ford: The next bit picks up on some of the stuff we have had from health. At our previous hearing, when we met the Medicines and Healthcare products Regulatory Agency, the UK agency here, we were told that, where a computer algorithm is already embedded in a medical device such as a heart monitor, that device is approved through the CE marking it gets from current standard setting. He said it is very important that those devices continue to have that CE marking post Brexit. Do you think that the MHRA should also be considering the regulatory oversight of other health-related algorithms and developing standards in those areas, too? Could you take away that sort of—
Margot James: Yes. You can argue that an algorithm is a device, and therefore, if the MHRA should not have oversight, certainly it should be consulted on the use of algorithms in so far as you can say they are devices. I can see that linkage, yes.
Q392 Vicky Ford: Coming to the third point, I am sure your team will look at what the Information Commissioner said earlier, but, picking that up, she is going to have not only the GDPR powers of regulation but ongoing powers to consider how data are processed and algorithms used. In her session with us she made it very clear that this was a helpful legal framework, that the GDPR will continue to evolve, and that it is important that we not only implement the regulation but continue to be able to have access to data flows, for both business and security, and can influence the ongoing global dialogue on data ethics, data privacy and so on.
I guess my question to you is, will you take away from here the point that it is not just the GDPR but that it is also in your role to keep focused on the international influence that we need to have on this international discussion on algorithms, data ethics and so on?
Margot James: Yes, I think that is very important. As Ollie said earlier, we have a leading position already in this area. We want to continue to be in the lead in the area. That is not to say, however, that we cannot learn from other countries. We will maintain our dialogue not just within the EU but obviously with other countries that are also concerned in this area—for sure.
Vicky Ford: I think the rest of section 11 has sort of been covered already.
Q393 Chair: I think so.
Thank you very much indeed. We have actually finished two minutes before our allotted time—so you have got away with it.
Margot James: That is the benefit of the slot just before lunch. Thank you very much, Chair. I wish to thank everyone on the Committee for their questions and for your continued hugely important work in this area. We much appreciate it.
Chair: Thank you.