WRITTEN EVIDENCE SUBMITTED BY AMNESTY INTERNATIONAL UK

(RAI0051)

Submission to the Inquiry on Human Rights and the Regulation of AI by the Joint Committee on Human Rights

Amnesty International UK (AIUK) is a national section of a global movement of over ten million people. Collectively, our vision is of a world in which every person enjoys all human rights enshrined in the Universal Declaration of Human Rights and other international human rights instruments. Our mission is to undertake research and action focused on preventing and ending grave abuses of these rights. We are independent of any government, political ideology, economic interest or religion.

1.                           Summary

2.                             We are deeply concerned that the Government’s approach to AI is unduly optimistic about the positive impacts of the technology and does not give due consideration to human rights.
3.                             Amnesty International advocates a precautionary principle to the implementation of digitisation, automation and artificial intelligence as “solutions” to complex systemic social and economic problems. We have identified that many uncritical implementations of AI and digital technologies have exacerbated existing inequalities and rights violations.
4.                             The Government should critically assess whether automation and deployment of AI is the correct and appropriate approach to reaching public policy or other stated aims without a risk of human rights violations.
5.                             This requires providing clarity and certainty to technology innovators by drawing red lines on technologies incompatible with human rights and levelling up with the public to identify underlying systemic problems that require a broader approach, acknowledging the limits of proposed technological solutions.
6.                             In our submission we focus on the use of AI by UK public authorities, specifically in the areas of law enforcement and social security, as we have carried out direct research on these topics that we believe will be useful for the Committee.

Human rights violations in predictive policing

7.                             AIUK’s research into the use by UK police forces of predictive policing systems has identified violations of the right to a fair trial, presumption of innocence, the right to freedom of assembly and association and the right to equality and non-discrimination. More automation and AI will make this worse and the problems are intractable, so such systems should be banned.
8.                             Predictive policing systems are in tension with data protection principles in UK law, included but not limited to purpose limitation, lawfulness, fairness, and transparency, accuracy and accountability. The identified lack of transparency restricts any meaningful way of challenging processing or enforcing other rights.
9.                             Data sharing from AI profiling across public sector bodies that becomes input for other AI systems can be difficult or impossible to trace, remove or correct. This could result in breaches of data protection principles of accuracy and fairness.
10.                         Bias in data sources used in predictive policing tools, and risk assessment tools, leads to racial discrimination and discriminatory treatment in breach of the UK’s international human rights obligations. Police forces could be in breach of the Public Sector Equality Duty (PSED) under s. 149 of the Equality Act 2010.
11.                         Mechanisms to remove bias by increasing the diversity of data may not be a viable option in policing without longer term reductions of racist policing that eventually create a data and feedback loop free from racial bias.
12.                         The combined human rights impact of various predictive AI tools used in conjunction, such as geographic hotspots, profiling and facial recognition, is not well understood but can impact freedom of association and assembly, among other rights.

Automation and social security

13.                         AIUK has identified that increasing automation in social security systems, including include eligibility assessments, risk profiling and fraud detection, and decisions on claims and payments, often leads to discrimination, further marginalisation and surveillance of impacted people and communities. The operations of these systems are almost entirely hidden from public scrutiny and AIUK has raised concerns about the lack of transparency
14.                         Automation and AI are amplifying, rather than resolving, systemic flaws with the current social security regime that undermine the right to social protection outlined in Article 9 of the International Covenant on Economic, Social, and Cultural Rights (ICESCR), ratified by the UK in 1976. In addition to the direct harms to individuals and families involved, this could damage social cohesion.
15.                         The automated systems for fraud and error detection often target individuals for further investigation based on some aspect of their profile being considered suspicious, not on any concrete evidence of wrongdoing.  Our view is that these predictive systems are a form of social scoring system of the kind that at face value are banned in jurisdictions like the EU (Art 5 EU AI Act).
16.                         The replacement of human interaction within the UK social security system is leading to impersonal and rigid decision-making processes that reduce the opportunities for claimants to explain complex situations or receive exceptional support.
17.                         Artificial Intelligence systems with anthropomorphic interfaces, from chat boxes to realistic video avatars, are not the solution to the problems around empathy, complex needs and vulnerability in social security. Research shows that interactive AI tools and personas can combine cognitive flaws and hallucinations with problematic capacities for deception and manipulation. 

AI Regulation

18.                         To protect and promote human rights, the UK must establish a rights-based, binding and enforceable AI regulation, developed through a transparent, accountable, and participatory process, which centres the concerns and priorities of people and communities at most risk of human rights harms.
19.                         In addition to providing detailed mechanisms for ensuring that AI technologies uphold human rights, we need clear red lines with explicit prohibitions on the development, production, sale, use, and export of AI technologies which are incompatible with human rights.

Ban on predictive policing and social scoring

20.                         AIUK believes that the use of AI and data-based predictive, profiling and risk assessment systems by police, law enforcement and criminal justice authorities in the UK to predict, profile or assess the risk or likelihood of offending, re-offending or other criminalised behaviour, or the occurrence or re-occurrence of an actual or potential criminal offence(s), of individuals, groups or locations, should be prohibited.
21.                         Digital tools that can serve a clear necessity and provide adequate safeguards for human rights could bring the benefits of automation under a better regulatory framework.

Safeguards, transparency, accountability and redress in data and AI systems

22.                         As a starting point, legislation must embed human rights principles directly into the design and deployment of AI and digital systems used by public bodies.
23.                         Generic equalities protections are not sufficient to prevent discrimination in AI systems. Laws must include explicit safeguards against algorithmic bias and indirect discrimination.
24.                         Relevant public authorities should have a duty to publish details of all AI predictive, profiling and risk prediction systems they are developing or using on a publicly available and accessible register.
25.                         Legislation should guarantee the right to meaningful human oversight of automated decisions and create stronger complementary protections to address the current gaps in data protection around profiling and decision systems that are not based solely on automated processing.

Council of Europe AI Framework Convention

26.                         The Convention is a positive example of a rights-based approach to AI regulation and could be beneficial in the UK context if the gaps we have identified are addressed at national level.
27.                         The limited scope of the Convention, which only applies to the public sector and some private providers could lower its impact. Commercial confidentiality and corporate concentration of the AI and tech industry are major challenges to regulating the sector.

Limitations of the EU AI Act

28.                         The EU’s AI Regulation (the AI Act) fell short of setting a gold-standard for rights-respecting AI regulation[1] and failed to meet many of the safeguards called for by civil society to ensure the protection and promotion of fundamental human rights[2]. 
29.                         Overall, the AI Act includes many provisions that appear to provide strong protections, but these are coupled with an extensive regime of exceptions and loopholes. These include bypassing restrictions on public biometrics, or the need for risk assessments in high-risk systems.  The scope of the Act excludes some of the most vulnerable individuals, such as migrants, altogether.

30.                    Question 1-How can Artificial Intelligence (AI) affect individual human rights for good or ill, in particular in the areas of: privacy and data usage, discrimination and bias, effective remedies for violations of human rights?

31.                         In our submission we will focus on the use of AI by UK public authorities, specifically in the areas of law enforcement and social security, as we have carried out direct research on these topics that we believe will be useful for the Committee.
32.                         More broadly, Amnesty International’s position is to adopt a precautionary principle to the implementation of digitisation, automation and artificial intelligence as “solutions” to complex systemic social and economic problems. In an Annex, we provide links to dozens of examples where uncritical implementation of digital technologies has exacerbated existing inequalities, rights violations, and environmental harms. Many of our recommendations also have a broader application beyond the risks and harms we have documented in policing and social security.

33.                         Artificial Intelligence and existing digital technologies

34.                         Discussions of the human rights impacts of AI often struggle with problems of definitions and scope. We take a flexible approach to these and believe that while the current wave of AI technologies presents some novel risks, there is also a continuity of problems from existing digital systems.
35.                         Drawing a precise line separating big data and sophisticated algorithms, or deep learning and neural nets from generative AI may be less relevant for human rights impacts than understanding how risks and problems are carried through new technologies. These include well know issues that start with the modelling of complex social problems.[3]
36.                         In the following paragraphs we illustrate the human rights impacts of existing digital systems and point at how these will likely be made worse if modern AI technology is applied in a similar manner.
37.                         Our recent research is consistent with reports and investigations from other organisations going back years[4], which shows that it is unlikely that the relevant public bodies involved will self-correct the identified negative human rights impacts in their future adoption of more sophisticated AI tools.

38.                       Predictive policing systems

39.                         AIUK’s research into the use by UK police forces of predictive policing systems – ‘Automated Racism – How police data and algorithms code discrimination into policing’ – has identified violations of the right to a fair trial, presumption of innocence, the right to freedom of assembly and association and the right to equality and non-discrimination[5].
40.                         This is not a hypothetical or future risk. Almost three-quarters of UK police forces use data-based and data-driven systems to attempt to predict, profile, and assess the risk of crime or criminalised behaviour occurring in the future, including but not limited to reoffending.
41.                         These predictive systems can be categorised broadly into two groups: those focusing on individuals and those that analyse geographical areas, raising different human rights issues.

42.                         Main types of predictive policing

43.                           Individual focused predictive and profiling tools:

44.                         Individual profiling systems are used to produce a ‘risk score’ for an individual, which will subsequently influence how a person is monitored by the police. One example is the Violence Harm Assessment (VHA), which has been used by the Metropolitan Police Service (MPS) since 2020 to assess the risk of a person being involved in violence by scoring them based on existing data and intelligence stored on that individual.
45.                         An individual does not need to have been convicted of a crime to be included – resulting in people being stopped because they are on a ‘database’, which can cause psychological harm to those profiled. AIUK spoke to David*, who has been described as a ‘standard risk’ by the police about the impact of individual profile systems on his life. David stated that he has been ‘stopped and searched’ over ’50 times’ by the Avon and Somerset police, with the impact of policing and profiling on him leading to a diagnosis of anxiety and post-traumatic stress disorder.

46.                           Geographic Predictive Policing Tools:

47.                         AIUK found that at least 32 police forces across the UK – including Essex Police, West Midlands Police and Merseyside Police – have used geographic predictive policing systems. These tools seek to predict “hotspots” where crime will likely occur and point areas that police will patrol and monitor closely for potential operations, such as stop and search.
48.                         There is limited evidence of the effectiveness of these geographic tools. In 2021, the Home Office, gave almost £5 million to 20 police forces for hotspot policing programme ‘Operation Grip’. Both the pilot and the wider Grip programme concluded that there was no significant reduction of crime as a result of its roll out.

49.                         Human Rights Impacts of predictive policing tools

50.                           Privacy and Data Usage

51.                         Predictive policing systems are in tension with data protection principles in UK law, included but not limited to purpose limitation, lawfulness, fairness, and transparency, accuracy and accountability.
52.                         Police use of predictive, profiling and risk assessment systems use personal data about large numbers of individuals, including sensitive data. These predictions, profiles and risk assessments can lead to interventions by the state, including questioning, searches and even arrest.
53.                         Profiling and decisions of such nature require a solid regulatory and operational framework under GDPR and related legislation, which includes mandatory safeguards and restrictions on the implementation of full automation. As autonomy and adaptability are core aspects of AI systems, their implementation in the context of predictive policing may be severely limited, if allowed at all.
54.                         It is particularly concerning that those profiled and risk assessed lack any meaningful way of challenging the accuracy of the interference that is made. There is a lack of transparency, as people are not informed if they feature on the VHA or other systems, and accessing this information via data protection subject access requests (SARs) is approved on a ‘case-by-case basis’. The VHA in particular is reminiscent of the MPS’ Gangs Matrix, which was found in breach of data protection laws in 2018.
55.                         We have found concerning evidence that some of these individual profiles are being shared with other state agencies, including the Crown Prosecution Service, the Department for Work and Pensions, probation services, local authorities and other third-party organisations. The stigma of suspicion or guilt can follow individuals throughout their interactions with local services, including employment, housing and education.
56.                         As AI systems are implemented throughout the public sector, data from police profiling can end up as input into other systems making this problem worse. Removing or correcting inaccurate data from AI models is very difficult and costly.[6] This could result in breaches of data protection principles of accuracy and fairness.
57.                         In addition, as data is shared across systems, the necessity or justification for the processing may vary but should remain lawful. The inherently adaptive nature of AI systems opens the possibility of other uses, which creates further challenges to the principle of purpose limitation and even the lawfulness of processing. The involvement of law enforcement and other purposes makes the legal analysis of the data sharing even more complicated.

58.                           Discrimination and Bias

59.                         Use of predictive policing tools, and risk assessment tools, leads to racial discrimination and discriminatory treatment in breach of the UK’s international human rights obligations. In particular, the use of these systems results in the disproportionate targeting of Black and racialised people and people from a lower socio-economic background. 
60.                         Our research shows that from May and August 2024 Black people made up the highest proportion of individuals included in the VHA, at 66 per cent. The MPS has itself noted issues with racial disproportionality and the VHA, including that it promotes the adultification of Black children.
61.                         Similarly, the use of geographic-focused crime prediction tools leads to the same areas and communities repeatedly being targeted and profiled by police intervention and enforcement. This often results in areas with high population levels of Black and racialised residents, as well as high levels of deprivation, being targeted. We saw this happening in Southend, for example.
62.                         The root cause of these systems’ reinforcement of and contribution to racial profiling and racist policing is the use of biased and discriminatory data – such as from stop and search – which already reflects institutional racism and discrimination inherent in current policing, as highlighted in the recent Casey Review.[7] 
63.                         Bias in data sources creating a feedback loop that leads to discriminatory outcomes is one of the best-known issues with AI, but bears repeating and stressing, as it keeps happening and may be difficult to fix. In other AI fields, such as facial recognition or recruitment, the approach to remove bias typically involves increasing the diversity of data – e.g. more Black images or women’s CVs. Here, this may not be a viable option without longer term reductions of racist policing that eventually create a data and feedback loop free from racial bias.
64.                         Nevertheless, such difficulties do not remove the need and duty to on public bodies to avoid discrimination. Police forces using predictive profiling and risk assessment systems have openly admitted to failing to carry out Equality Impact Assessments, which could be in breach of the Public Sector Equality Duty (PSED) under s. 149 of the Equality Act 2010.

65.                           Freedoms of association and assembly

66.                         The use of predictive and profiling systems to target geographic areas can lead to a ‘chilling effect’ in people’s ability and willingness to exercise their right to freedom of association and assembly. The Home Office has acknowledged that hotspot policing ‘may have the effect of displacing crime by time or place’, or at best, result in ‘modifying the crime in the surrounding area’.
67.                         These geographic systems can operate in conjunction with other AI tools to identify individuals within a geographical area, such as ANPR and biometrics particularly facial recognition. The combined human rights impacts have not been fully explored.

68.                       Automation in Social Security Systems

69.                         Our report “Too much technology, not enough empathy”[8] highlights the negative impacts of automation technology in the British social security system. The Department for Work and Pensions (DWP) increasingly uses digital automation to drive decision making and processing of social security with decreasing levels of human involvement. These administrative processes include eligibility assessments, risk profiling and fraud detection, and decisions on claims and payments. DWP’s decisions have life-changing implications for individuals facing destitution, or even criminal proceedings.
70.                         Increasing automation is part of a wider drive towards digitisation that has negative human rights impacts that will only be amplified by AI.

71.                         Human Rights Impacts of Social Security Automation

•                                                                                             Right to Social Security: Claimants face barriers to accessing benefits due to digital systems, undermining their right to social protection.
•                                                                                             Adequate Standard of Living: Delays, errors, and inaccessible systems contribute to poverty and insecurity, violating the right to an adequate standard of living.
•                                                                                             Due Process and Accountability: Automated decisions often lack transparency and avenues for appeal, compromising procedural fairness.

72.                         Discrimination and Bias

•                   Digital Exclusion: Groups such as older people, disabled individuals, and those with limited digital literacy are disproportionately affected.
•                   Indirect Discrimination: Automated systems may unintentionally embed biases, leading to unequal treatment based on socioeconomic status, ethnicity, or disability.
•                   Algorithmic Bias: There is concern that data-driven decisions may reflect and reinforce existing inequalities.

73.                         Deepening breaches of the right to social security

74.                         The right to social security is outlined in Article 9 of the International Covenant on Economic, Social, and Cultural Rights (ICESCR), ratified by the UK in 1976. It is also recognised in the Universal Declaration of Human Rights (1948) and other treaties, including the Convention on the Rights of the Child (1989) and the International Labour Organisation (ILO) Convention No. 102 (1952).
75.                         Our report “Social Insecurity” documented how the UK’s social security system does not legally guarantee essential security payments that ensure access to basic needs such as housing, food, and education.[9] Despite increased social security spending, poverty rates remain unacceptably high, with claimants reporting severe hardships, including reliance on food banks and struggles to afford basic needs like heating and rent.
76.                         Automation is amplifying, rather than resolving, systemic flaws with the current social security regime that, in addition to the direct harms to individuals and families involved, could damage social cohesion.

77.                           Lack of transparency and accountability

78.                         Despite these impacts, the DWP has committed millions of pounds for projects that use AI, including for advanced analytics in fraud and error detection and generative AI tools. The operations of these systems are almost entirely hidden from public scrutiny. AIUK has raised concerns about the lack of transparency from within the department which makes it difficult to ascertain how technology is used, what business processes and criteria sit behind the automated decisions driven by the systems and how they have been assessed for bias, fairness and accuracy.  Claimants themselves do not know who (or what) made a decision and therefore how to challenge it.

79.                           Automating fraud and error is a form of social scoring

80.                         In our report on the use of technology in the UK social security regime we were particularly concerned by the use of AI in detecting fraud, errors and debt, including overpayments (FED). Risk scoring is used to target individuals for further investigation based on some aspect of their profile being considered suspicious, not on any concrete evidence of wrongdoing.
81.                         These systems rely on opaque algorithms that may further rely on spurious associations rather than actual actions or behaviours, introducing bias and discrimination into decision making. Quite often, these characteristics or traits are tied directly or by proxy to an individual’s immigration status, race, disability or gender, making certain individuals and communities more likely to be targeted than others. The lack of transparency over these technologies compounds this problem.
82.                         Our view is that these predictive systems are a form of social scoring system of the kind that at face value are banned in jurisdictions like the EU (Art 5 EU AI Act). The use of data collected in a different context and detrimental treatment linked to inferred characteristics from third parties, rather than behaviour, are problematic. Risk assessments of criminal behaviour, such as fraud, on the basis of profiling and not actual evidence is also banned in the EU in principle.

83.                           Lack of empathy and trust will not be solved by avatars

84.                         Our research identified the lack of empathy as a central problem within the UK social security system as automation replaces human interaction, leading to impersonal and rigid decision-making processes. Online structured interactions reduce the opportunities for claimants to explain complex situations or receive exceptional support.
85.                         AI systems with anthropomorphic interfaces, from chat boxes to realistic video avatars, may appear as a solution to provide a personalised and empathetic service to individuals, and indeed there are myriad offers of such services. Various thinktanks have calculated savings of billions of pounds in the automation of calls to DWP and HMRC.[10]
86.                         While current AI systems can mimic much human behaviour, it is far from clear that they may be able to solve the issues we have identified around empathy, complex needs and vulnerability.
87.                         Research has shown that language models that were able to achieve near perfect scores on medical examinations, struggled in interactions with real users and should not be deployed.[11] AI systems that shine on mathematical Olympiads struggle with basic logic.[12] Researchers have found that “LLMs express fragility in conducting simple mathematical reasoning, with word-based tasks that involve counting to ten often posing a significant challenge.”[13]
88.                         There are broader issues of trust in putting advanced AI systems on the frontline of public services. In addition to the cognitive limitations we just described and the well-known problem of “hallucinations”, there is extensive evidence of their potential for deception and manipulation.[14] There is also evidence that people’s reactions to AI personas, including fear, are highly variable depending on the expectations of the role.[15] Manipulating AI systems to meet psychological needs – e.g. to appear as friendly or authoritative – raises ethical or even legal considerations.

89.                         A wider problem beyond the UK and DWP

90.                         Our findings in relation to the UK and DWP are consistent with increasing evidence produced by Amnesty International and civil society organisations in other countries on how social security automation often leads to discrimination, further marginalization and surveillance of impacted people and communities.
91.                         Prominent cases in European countries similar to the UK include the Dutch child benefits risk-profiling system scandal that led to the collapse of the Government, mass surveillance in fraud detection algorithms used by the Danish welfare agency, and similar systems used in Sweden and France, with the Swedish case being currently being investigated by the Swedish Privacy Protection Authority (IMY) on discrimination grounds. We provide links to these and other reports in the annex.

92.                    Question 3. To what extent is the Government’s policy approach to deploying AI, expressed in its “AI Opportunities Action Plan”, sufficiently robust in respect of safeguarding human rights?

93.                         We are deeply concerned that the Government’s approach to AI is unduly optimistic about the positive impacts of the technology and does not give due consideration to human rights.
94.                         Our view is that the Government’s approach should be to critically assess whether automation and deployment of AI is the correct and appropriate approach to reaching public policy or other stated aims, particularly making sure that AI deployment does not exacerbate or pose risk of human rights violations.
95.                         This requires providing clarity and certainty to technology innovators by drawing red lines on technologies incompatible with human rights, and levelling up with the public to identify underlying systemic problems that require a broader approach, acknowledging the limits of proposed technological solutions.
96.                         Assessing whether the deployment of a particular technology is appropriate should also include an assessment of the financial resources allocated to developing and deploying these systems. Investing in automation should never involve divesting from other critical state functions.

97.                    Question 4- What would be needed in any future UK legislation to protect human rights?

98.                         Our research has showed that existing AI and algorithmic systems in areas such as policing and social security are likely not in compliance with existing legal and regulatory frameworks such as equalities and data protection, and human rights commitments more broadly. We do not see this solely as an enforcement problem, though, and think that the current regime is insufficient to combat the human rights impacts of the use of these technologies.

99.                       Principles for AI Regulation

100.                     To protect and promote human rights, the UK must establish a rights-based, binding and enforceable AI regulation, developed through a transparent, accountable, and participatory process, which centres the concerns and priorities of people and communities at most risk of human rights harms. Such regulation should at minimum:
1.                          Ensure explicit prohibitions on the development, production, sale, use, and export of AI technologies which are incompatible with human rights, including systems used for public facial recognition, social scoring, predictive policing, biometric categorization, emotion recognition, risk assessment and profiling tools that violate rights of migrants, refugees and asylum seekers.
2.                          Reject loopholes and exemptions which risk violation of human rights, such as blanket and disproportionate exemptions based on, public interest, law enforcement, national security, or for military technologies.
3.                          Ensure human rights due diligence throughout the AI lifecycle and at any stage of the supply chain, including an obligation for developers and deployers of AI technologies to conduct and publish thorough human rights, equality, and data protection assessments.
4.                          Ensure public accountability and transparency measures when developing and deploying AI technologies, including through the creation and maintenance of a publicly accessible database where developers and deployers are obliged to disclose essential information related to the development and use of AI technologies.
5.                          Empower people and communities impacted by AI, including through effective redress and remedy mechanisms, and the right to information and explanation of AI-supported decision-making for impacted people.

101.                 Effective Remedies for Violations of Human Rights in the use of AI and Algorithms in Policing and Social security

102.                     AIUK believes that the use of data-based predictive, profiling and risk assessment systems by police, law enforcement and criminal justice authorities in the UK to predict, profile or assess the risk or likelihood of offending, re-offending or other criminalised behaviour, or the occurrence or re-occurrence of an actual or potential criminal offence(s), of individuals, groups or locations, should be prohibited.
103.                     The above general principles for an AI Regulation would prohibit the use of problematic predictive policing and social security social scoring we identified in our research. Digital tools that can serve a clear necessity and provide adequate safeguards for human rights could bring the benefits of automation under a better regulatory framework.

Our specific recommendations for enabling transparency, accountability and redress in artificial intelligence and digital systems in police, social security and legal authorities:

Embedding human rights principles and explicit safeguards

104.                     As a starting point, legislation must embed human rights principles directly into the design and deployment of AI and digital systems used by public bodies. These requirements must go beyond privacy and include the need to uphold the rights to social security, non-discrimination, and an adequate standard of living. These rights should not be treated as secondary to efficiency or cost-saving goals.
105.                     Generic equalities protections are not sufficient to prevent discrimination in AI systems. Laws must include explicit safeguards against algorithmic bias and indirect discrimination. This is particularly important given our findings that certain groups—such as ethnic communities, disabled people, older adults, and migrants—are often disproportionately affected by digital systems.

Transparency

106.                     There must be clear transparency requirements on all data-based predictive, profiling and risk assessment systems used by police and legal system authorities. At a minimum, there should be a statutory obligation on UK police forces and other law enforcement authorities across England and Wales, Scotland and Northern Ireland, including criminal legal system authorities (such as the Ministry of Justice and prison and probation services), to register and publish details of all AI predictive, profiling and risk prediction systems they are developing or using on a publicly available and accessible register. This publicly accessible register must include:

• What the intended purpose of the system is;
• How the system is operated in practice, including a standard operating procedure;
• All data types that the system uses, including the sources of that data;
• What decisions or outcomes the system influences;
• Any internal reviews or evaluations.

107.                     Future law should require mandatory human rights and equality impact assessments before digital systems are introduced. These assessments must be publicly available and include algorithmic fairness evaluations to ensure that automated decision-making does not reinforce or exacerbate existing inequalities.

Accountability and redress

108.                     In the UK there are not clear or sufficient mechanisms for the collective enforcement of rights in response to data-driven systems. AI UK has found that this contravenes international standards.
109.                      Legislation should guarantee the right to meaningful human oversight of automated decisions. Individuals must be able to have their cases reviewed by a person, especially when decisions impact their access to essential benefits.
110.                     Legislation should bring statutory obligations on public bodies on UK police forces and other law enforcement and criminal legal system authorities using data analytics systems to provide clear, accessible routes for individuals to challenge decisions and seek redress in all circumstances. The data protection regime offers little protection against prediction and profiling systems that are not based solely on automated processing. Therefore, stronger complementary protections are needed for hybrid systems with human involvement.
111.                      Independent and robust oversight mechanisms should also be established or strengthened to monitor compliance and enforce human rights standards in the use of public sector technology.

112.             Question 8. How much difference will the Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law make to the protection of human rights in the UK?  

113.                     Our view is that the Convention is a positive example of a rights-based approach to AI regulation and could be beneficial in the UK context if the gaps we have identified before the adoption of the Convention are addressed at national level.
114.                     The Convention is a binding instrument, but it gives great latitude to the national governments to adopt and maintain the necessary measures to comply with the general principles. This requires vigilance and commitment from national legislators and other stakeholders.
115.                     The limited scope of the Convention, which only applies to the public sector and some private providers could lower its impact. Commercial confidentiality and corporate concentration of the AI and tech industry are major challenges to regulating the sector.

116.             Question 9. What lessons can be drawn from regulation of the impact of AI on human rights in other jurisdictions, such as the European Union?

117.                     Amnesty International have noted that the EU’s AI Regulation (the AI Act) fell short of setting a gold-standard for rights-respecting AI regulation[16] and failed to meet many of the safeguards called for by civil society to ensure the protection and promotion of fundamental human rights[17]. 
118.                     Overall, the AI Act includes many provisions that appear to provide strong protections, but these are coupled with an extensive regime of exceptions and loopholes. These include bypassing restrictions on public biometrics, or the need for risk assessments in high-risk systems.  The scope of the Act excludes some of the most vulnerable individuals, such as migrants, altogether.

(Sep 2025)

119.             ANNEX: Amnesty International Reports for further reference:

120.                        Social protection, access to essential services

• The Netherlands: Dutch childcare benefit scandal an urgent wake-up call to ban racist algorithms - Amnesty International [EN]
• Serbia: World Bank-funded digital welfare system exacerbating poverty, especially for Roma and people with disabilities [EN]
• India/Global: New technologies in automated social protection systems can threaten human rights [EN]
• Denmark: AI-powered welfare system fuels mass surveillance and risks discriminating against marginalized groups [EN]
• Denmark: Easy-to-read version: Coded Injustice: Surveillance and Discrimination in Denmark’s automated welfare state [EN, DA, AR]
• France: Discriminatory algorithm used by the social security agency must be stopped - Amnesty International – jointly with national civil society partners [EN]
• Sweden: Authorities must discontinue discriminatory AI systems used by welfare agency - Amnesty International – in support of investigation done by Lighthouse Reports and Svenska Dagbladet [EN]
• Netherlands: Profiled without protection: Students in the Netherlands hit by discriminatory fraud detection system [EN]
• Global: Governments’ adoption of unchecked technologies in social protection systems undermines rights [EN]
• United Kingdom: Government’s unchecked use of tech and AI systems leading to exclusion of people with disabilities and other marginalized groups [EN]

121.                        Policing, national security and military

• The Netherlands: End dangerous mass surveillance policing experiments [EN]
• EU companies selling surveillance tools to China’s human rights abusers [EN]
• Ban the Scan campaign website [EN]
• Israel/OPT: Israeli authorities using facial recognition to entrench apartheid [EN]
• United Kingdom: Met Police using 'racially discriminatory' Gangs Matrix database [EN]
• The Netherlands: Recording dissent: Camera surveillance at peaceful protests in the Netherlands [EN, NL]

122.                        Regulation, advocacy, and policy setting

• European Union: A civil society coalition statement: An EU Artificial Intelligence Act for Fundamental Rights [EN]
• Council of Europe: Amnesty International's Recommendations on the Draft Framework Convention on Artificial Intelligence, Human Rights, Democracy and the Rule of Law [EN]

15