15              shedding light on ai – a framework for algorithmic oversight

Written evidence submitted by 5Rights Foundation (OSB0206)

Shedding light on AI

A duty to regulate automated decision-making systems that impact children

Table of Contents

Introduction

Definitions

Four step model for regulating AI

Regulatory duty to investigate

Proposed amendments for the Draft Online Safety Bill

Conclusion

Introduction

The Draft Online Safety Bill exemplifies the UK government’s commitment to putting children front and centre of making the UK the safest place in the world to be online. It builds on the increasing global consensus that digital services that impact on children must be designed for their use. Childhood is a time of experimentation and personal growth, and while no environment is entirely risk free, digital services “likely to be accessed by children” must be designed to be private, safe and rights respecting - by default.

Central to the digital world is artificial intelligence, commonly referred to as AI. AI is not a standalone or fixed technology but plays a part in automated decision-making (ADM) systems and many other data-driven features common across digital services. Automated systems shape the experiences of children and young people in the digital world, both as a result of their direct engagement, for example receiving friend/follower or content recommendations, and from systems that they may not interact with directly, for example automated decision-making used to allocate welfare funding.

Much emphasis is put on the challenges of regulating new and emerging technologies, but AI is not new. The term ‘AI’ was coined in the 1950s to describe the science and engineering of machines making automated choices against specific criteria based on available information. In many ways, the word 'intelligence' is used to give humans confidence in the efficacy and authority of machine-made choices. Since then, huge advancements in the application of AI and greater availability of data have led to more sophisticated, data-driven decision making. Systems that use AI are still human-made with specific objectives, design goals, chosen inputs, a set of rules by which information is given importance or weight, and a combination of outcomes and outputs. At each of these stages, automated decisions are made that are often imperceptible to those they impact, particularly if they are a child.

Automated decision-making sits behind features that are ubiquitous across services likely to be accessed by children. It can support children to navigate the online world and the mass of content available, and help them to identify activities and outcomes that are useful or beneficial to them. But there are also many situations when automated decision-making systems undermine their rights or put them at risk. For example;

• Friend recommendations are made by 75% of the most popular social networking sites.[1] These connect users based on the data profiles the platform has built about them, irrespective of their age, which has been found to enable predators to contact children.[2]
• Misinformation is spread and amplified by automated systems that frequently promote content that is most likely to engage users, irrespective of its quality or impact. In 2020, vaccine misinformation alone was worth up to $1bn for the largest services.[3]
• Nudges are optimised to encourage children to make in-app purchases or engage with gambling-style features. In 2019, British children collectively spent £270 million on loot boxes and other in-app purchases.[4]
• Harmful material that violates the community rules of services is recommended to children, including material promoting self-harm or suicide behaviours, disordered eating and pornography.[5]

The online safety duties set out in the draft Online Safety Bill offer a vision of what a responsible digital world looks like. However, to meet the safety objectives of the Bill, Ofcom must not only have the tools but a duty to investigate algorithms on behalf of children, and an agreed standard by which to assess them. A duty of this kind with the four-step process described in this paper would ensure the risks to children created by algorithms are identified, eliminated, mitigated or effectively managed.

This four-step process is platform neutral and can be applied across different sectors, including but not limited to social media, entertainment, health and education. It can also be applied to different parts or features of a service, including advertising, content recommendation, moderation and reporting.

Such a regime would give clarity to businesses in fulfilling their safety duty to children and power to the regulator to inquire, analyse and assess whether a system is conforming to requisite standards. When new risks and harm are revealed, they can act as an early warning, especially when that harm was an unintentional by-product of an automated decision-making process optimised for another purpose.

This short paper builds on the work of many in the international community, notably the Centre for Data Ethics and Innovation[6], UNICEF[7], IEEE[8], the Ada Lovelace Institute[9] and the Council of Europe[10]. We are grateful for their expertise and recognise that this practical application of their work could not have been done without their thoughtful and detailed insights.

Thanks are due also to Dr Rebekah Tromble, Associate Professor in the School of Media and Public Affairs and Director of the Institute for Data, Democracy, and Politics at George Washington University. Dr Tromble developed the four-step model articulated in this report.

5Rights is committed to building the digital world young people deserve. That world is one in which they share the benefits of digital engagement as participants, citizens and consumers, and in which businesses respect and uphold their existing rights and respond to their needs and evolving capacities - automatically.

Definitions

• Artificial intelligence (AI) describes when machines are able to mimic the problem-solving and decision-making capabilities of the human mind.
• Machine learning is a branch of artificial intelligence (AI) that employs computational algorithms to detect patterns in—and learn iteratively from—data, generating output with minimal human intervention and improving over time.
• An algorithm is a sequence of instructions or set of rules designed to complete a task or solve a problem.
• Automated decision-making is the process of making a decision by automated means, without any human involvement. These decisions can be based on factual data, as well as on digitally created profiles or inferred data.[11]
• Algorithmic bias is commonly used to describe an automated system that produces results that discriminate against or disadvantages groups of people (for example, based on age, disability, gender, or race).
• Algorithmic fairness is an automated system that produces results that do not discriminate against nor systematically disadvantage groups of people: it also seeks to ensure that automated systems do not violate rights, exploit vulnerability, manipulate, nor withhold information in a way that impairs one’s ability to make informed choices.
• A child is a person under the age of 18.[12]
• Document analysis is a research method involving the review and interpretation of written materials such as emails, legal records, and meeting notes, designed to gather evidence on the topic being studied and answer specific questions.
• Code analysis is a way of assessing how an algorithm is structured and how it might function in practice without actually executing the program, allowing errors or vulnerabilities to be detected.
• Variables are individual items in a dataset being analysed, for example age, gender and location.

Four step model for regulating AI 

The four-step model set out below offers a mixed method approach to algorithmic oversight. It describes how regulators can evaluate each element of an automated decision-making process, from the goals, inputs, implementation and outcomes, to ensure that applications of AI meet the established rights and needs of children, as set out in:

• The United Nations Convention on the Rights of the Child to which the UK is a signatory. General Comment 25 on children’s rights in relation to the digital environment is the authoritative document which sets out the relevance of the convention to the digital world.[13]
• General Data Protection Regulation (GDPR)[14] and the Age Appropriate Design Code,[15] and the associated guidance from the Information Commissioners Office, which set out the standards providers of digital products and services must meet in relation to children’s data.
• Existing laws and protections that pertain to children, such as the Equality Act 2010, Children Act 1989, and some consumer laws, as well as government guidance, which may also be relevant when considering the impact of AI systems on children.[16]

Any assessment of automated decision-making systems must have the flexibility to uncover harms that are currently unknown or not anticipated. It must also allow for potential improvements or benefits to be identified so that they might be shared and used to guide best practice.

1. Understand the design goals

Aim:

Algorithms are formulated with a purpose and intended outcomes. In assessing the fairness and appropriateness of algorithms, it is important for the regulator to understand the original intent and goals of its creators and how those goals evolved over time, by asking the following questions:

1. What was/were the problem(s) or challenge(s) those designing the algorithm set out to address?
2. What was/were the intended outcome(s)?
3. Why was this product, feature or process considered necessary?
4. Who was involved in defining the problem(s) and desired outcome(s)—including internal and external stakeholders? What was their role in shaping the understanding of the problem(s) and desired outcome(s)?
5. How and why did any of these things change over time?

Method:

• Undertake interviews with stakeholders
• Analyse information - product development documents and internal communications such as emails and meeting notes in which the algorithmic product was discussed.             

2. Consider the data inputs

Aim:

Every algorithm contains a series of inputs — data points and variables that can be thought of as the “ingredients” of the algorithm. Unfair, discriminatory or biased outcomes are often the result of problematic data (“garbage in, garbage out”). It is therefore essential that any framework intended to examine algorithmic fairness assess the quality and appropriateness of the data used to build and train the algorithm, by asking the following:

1. What features (variables) did the algorithm’s designers want to include as inputs and why?
2. Were they able to include those features? Did they have to settle for proxies and/or exclude some features altogether and why?
3. What dataset(s) was/were used as input(s) for building, training, and testing the algorithm?
4. Were other datasets considered? For training/testing? For final implementation? If not, why not?
5. If so, what were the perceived advantages and disadvantages, strengths and weaknesses of this/these datasets compared to other options?
6. Were multiple datasets and/or features tested? If so, how were they evaluated? And why were the final datasets/features selected?
7. Who had input into these decisions, and what was their role in the process?

Methods:

• Undertake interviews with stakeholders
• Analyse information - product development documents and internal communications
• Code analysis
• Data sample analysis.

3. Assess the model selection and execution

Aim:

If data inputs are the “ingredients” of an algorithm, the mathematical model and its parameters offer the instructions for how to put the algorithmic recipe together. They lay out how the inputs should be combined, at what point and in what amount, as well as the ways in which those inputs might be altered or transformed. Careful scrutiny of the model and the assumptions it is built upon is needed to assess its appropriateness. Note that such scrutiny is possible even with machine learning algorithms. The questions to consider as part of this scrutiny include:

1. What is the mathematical formula/model applied?
2. Why was this model selected?
3. What assumptions are built into this model?
4. Did those designing or implementing the algorithm deviate from any of the assumptions built into the model? If so, how and why?
5. Within the model, what is being optimised? How is this optimisation carried out (e.g., how are the various features weighted)?
6. How and when was the model tested and changed/updated?
7. When changes were made, what were the reasons for making those changes?

Method:

• Undertake interviews with stakeholders
• Analyse information - product development documents and internal communications
• Code analysis
• Implementation experiments (e.g., running independent tests on real or synthetic data, including on platform).

4. Identify outputs and outcomes

Aim:

After an algorithm is launched, it will generate certain outputs. It is important to examine these outputs to reveal whether the model performs as intended. However, at this stage, it is also important to look at the actual outcomes — the real world impacts generated by the algorithm(s) and its uses.

The previous three steps help to determine why and how something went wrong, what elements of the design and implementation result in discrimination, disadvantage, exploitation, manipulation, or rights violations. However, the output (step four) is likely to be the first place that harm is identified and the stage at which it is shown that the four step process is necessary.

Many observers note that algorithms are not autonomous, neutral entities. They are designed by people, with all the biases, blind spots, and other foibles associated with being human. It is therefore crucial to examine the interplay of technical features on the one hand, business decisions and human interactions on the other. The regulator will need researchers and investigators with training in the social sciences as well as computer scientists to conduct such assessments.

Below we lay out the three lenses through which to examine algorithmic outputs and outcomes. First, we describe assessments of the ways in which relevant companies interpret outputs and outcomes, as well as their techniques for mitigating perceived harms. Second, we outline a broad approach for considering how users interact with and are impacted by algorithms. Finally, we discuss broad approaches to uncovering impacts on society as a whole.

1. Companies

Aim:

To examine how either the company that designed the algorithm or companies that make use of those algorithms evaluate outputs and outcomes.

Questions:

1. What model outputs (variables) does a company use internally? (I.e., What outputs matter to them and why?) In what ways do they use these outputs?
2. What is the internal process for evaluating the performance of an algorithm? What standards are applied? What metrics are applied? By whom?
3. What is the internal process for determining whether an algorithm should be changed? Who is involved in this process? Who makes final decisions and how?
4. What, if anything, is the company doing to assess larger impacts on users and society?
5. If such assessments occur, are they ad hoc or systematic?
6. What techniques and methodologies are used for such an assessment? What standards and metrics are applied? Who is involved in this process and how?
7. Are changes ever made to algorithms on the basis of such assessments? What is the process for doing so? Who is involved in this process? Who makes final decisions and how?

Method:

• Interviews
• Document analysis
• Code analysis.

2. Users

Aim:

To assess whether users’ reasonable expectations for how they interact with and what they expect from an algorithm align with the actual outcomes, and whether any harms (either perceived by the user or not) accrue.

Questions:

1. What, if anything, do users understand the algorithm to be doing? Are they even aware that an algorithm is involved? If they are, do they perceive specific advantages and disadvantages to the algorithm?
2. What do users expect from the algorithm? Are outcomes aligned with those expectations?
3. Is the algorithm creating disparities between users and non-users and/or between different types of users?
4. Is the algorithm limiting user choice(s)? If so, in what ways? And what are the consequences (positive or negative) of those limitations?
5. Does the algorithm directly or indirectly exploit user vulnerabilities?
6. Does it directly or indirectly manipulate users?
7. Does it violate users’ rights or contribute in any way to the violation of those rights?

Method:

• User surveys and interviews
• (Controlled) experimental user studies.

3. Societal impacts

Aim:

To understand the social, financial, environmental and human impacts of automated decision-making systems.

Questions:

1. Is the algorithm contributing directly or indirectly to social harms? If so, in what ways? And to whom? Is the harm caused by certain features of the algorithm? Can these harms be mitigated by changes to the algorithm? Can these harms be mitigated without causing harm to others?
2. Is the algorithm benefitting certain members of society? If so, are those benefits accrued fairly and equitably?
3. Is the algorithm benefitting society as a whole? If so, in what ways? Can those benefits be amplified or expanded?
4. Are there “best practice” lessons to be learned from the design and implementation of this algorithm?

Method:

• A variety of social scientific and humanistic research designs.

Regulatory duty to investigate

Children cannot be expected to understand or take action against automated decision-making or algorithmic unfairness, it is unlikely that they have the developmental capacity, the knowledge or the resource to understand the subtle, cumulative or even acute nudges and impacts those automated systems have on their online experience. In fact – many children do not understand that an algorithm could be responsible for introducing them to a 'suggested friend’ nor do they have the tools to prevent an onslaught of automated harmful material. The Online Safety Bill must give Ofcom not only the powers to interrogate automated systems but create the expectation that they will be actively analysing automated decision-making systems and algorithms of services that impact on children – a duty to investigate.

The proposed duty is similar to that of the Financial Conduct Authority, which has a duty to investigate when it appears that a regulated person or investment scheme has failed to protect consumers or might have a significant adverse effect on the financial system or on competition. Similar duties are proposed for a new pro-competition regime that would give the Digital Markets Unit (part of the Competition and Markets Authority) a duty to ‘monitor’ markets and the activities of firms to identify breaches of the statutory code of conduct.

In order to fulfil this duty, Ofcom must have the expertise, resource, and processes in place to scrutinise the design goals, data inputs, model selection and outputs and outcomes of algorithms. Where there is evidence to show such systems are discriminating against or systematically disadvantaging children or violating their rights, Ofcom should set out a mandatory course of action for compliance.

While transparency is a key component of the four-step process set out above, decades of research show transparency alone can result in layers of obfuscation and does not always result in better systems or more positive outcomes. The value of transparency lies not in the availability of information itself, but in the way it allows for scrutiny and accountability. A duty for Ofcom to undertake the four steps on automated decision-making systems that impact on children would deliver that accountability.

Companies often use commercial sensitivity as a defence to usurp transparency reporting requirements. On the whole, this should be resisted, and where there are legitimate commercial sensitivities, the regulator must have the power to maintain private oversight.

Proposed amendments for the Draft Online Safety Bill

Part 2, Chapter 2 – Providers of user-to-user services

In addition to the existing record keeping and review duties of clause 16, insert

(Before 4) A duty to keep a written record relating to the design and operation of algorithms and automated decision-making systems, and any decisions relating to the operation of such systems as may be needed by the regulator.

Part 2, Chapter 3 – Providers of search services

In addition to the existing record keeping and review duties of clause 25, insert:

(Before 4) A duty to keep a written record relating to the design and operation of algorithms and automated decision-making systems, and any decisions relating to the operation of such systems as may be needed by the regulator.

Part 4, Chapter 1 – OFCOM’s General Duties

NEW Clause 59: Duty to investigate and audit the design and operation of algorithms and automated decision-making systems that impact children in relation to the findings of OFCOM’S risk assessment as set out clause 61 (2)(c).

(1) A duty:

(a) To provide statutory guidance in relation to children’s rights and applicable law as it relates to automated systems to meet the safety objectives (clause 30)

b) Where there is evidence from OFCOM’s risk assessments that an automated decision-making system is discriminating against or systematically disadvantaging groups of children or violating their rights, exploiting vulnerabilities, manipulating, or withholding information, OFCOM will investigate and audit such systems.

Part 4, Chapter 3 – OFCOM’s Risk Assessments

Clause 61 (1) OFCOM must carry out a risk assessment to identify, assess and understand the risks of harm to individuals presented by regulated services.

(2) The risk assessment must, amongst other things—

(a) assess the levels of risk of harm presented by regulated services of different kinds, including by giving separate consideration to—

(i) the risk of harm to individuals in the United Kingdom presented by illegal content,

(ii) the risk of harm to children in the United Kingdom, in different age groups, presented by content that is harmful to children, and

(iii) the risk of harm to adults in the United Kingdom presented by content that is harmful to adults; (

b) identify characteristics of different kinds of regulated services that are relevant to such risks of harm, and assess the impact of those kinds of characteristics on such risks.

NEW Before (3) to identify the risk of harm to children from the design and operation of algorithms and automated decision-making

(3) OFCOM must develop risk profiles for different kinds of regulated services, categorising the services as OFCOM consider appropriate, taking into account

(a) the characteristics of the services, and

(b) the risk levels and other matters identified in the risk assessment.

(4) As soon as reasonably practicable after completing a risk assessment, OFCOM must publish a report on the findings.

NEW Before (5) Where findings of the risk assessment indicate that algorithms or automated systems are found to be contributing to or likely to harm children OFCOM Must commence an investigation and audit of the regulated service provider(s) to discharge its clause 59 duty.

(5) OFCOM must ensure that the risk assessment is kept up to date.

(6) In this section the “characteristics” of a service include the functionalities of the service, its user base, business model, governance and other systems and processes. (7) In this section— “content that is harmful to adults” has the meaning given by section 46; “content that is harmful to children” has the meaning given by section 45; “illegal content” has the meaning given by section 41.

Part 4, Chapter 5 – Ofcom’s Information powers

Clause 70: Power to require information

(4) The information that may be required by OFCOM under subsection (1) includes, in particular, information that they require for any one or more of the following purposes—

(a) the purpose of assessing compliance by a provider of a regulated service

…

(m) the purpose of investigating algorithms and automated decision-making systems, in relation to OFCOM’s duty under section 59             

Conclusion

A child must not be asked to police the automated decisions of the tech sector. The industry is worth over $5 trillion to the world economy[17] and is central to children’s lives and life outcomes. Algorithmic oversight is critical if the next generation of digital technologies, products and services are to offer children safety and respect for their rights, by design.

The four-step model of algorithmic oversight will allow meaningful oversight of the goals, inputs, implementation and outcomes of algorithms and automated decision-making systems. This transparency will drive a change in corporate behaviour that meets the expectations of parents and children and fulfil the promises government has made to them. By giving the regulator a duty to interrogate automated decision-making systems on behalf of children, and service providers a clear process by which it will be done, the risks to children from automated decision-making systems can be reduced – by default and design.

There is no silver bullet to fix all the ills of the digital world or to guarantee children will be safe from harm, either through regulation or technological development. But the argument that regulation and accountability stifle innovation or impose limits on a child’s freedom in the digital world is simply untrue. Each wave of regulation has been met with creative and practical solutions. As lawmakers across the globe look to the UK to continue its singular place as leader in online safety for children algorithmic oversight would continue that march.

It is in the interests of all parties to have a more equitable and trustworthy system of oversight that allows growth and innovation but which reduces negative outcomes for children.

To do nothing is no longer an option.

8 October 2021