NCC Group — Written evidence (NTL0005)

Introduction

1.1. NCC Group is delighted to have the opportunity to engage with the Lords Justice and Home Affairs Committee’s important inquiry and offer our expertise as a global cyber security and software resilience business. Indeed, through our work and research we are acutely aware of the rapidly evolving use of new technologies such as machine learning (ML), artificial intelligence (AI) and algorithmic tools to aid decision-making, and we believe that it is crucially important that security implications are considered from the outset. We are keen to ensure that security and safety considerations are not seen as a blocker or a cost but as an enabler of future-proof systems that, by their design, avoid mistakes that are expensive and otherwise costly to fix later. It is through this lens that we offer our insights into the use of new technologies in the application of the law.

1.2. NCC Group’s mission is to make the world safer and more secure. We are trusted by more than 14,000 customers worldwide – including a wide variety of legal and public sector institutions – to help protect their operations from ever-changing cyber threats. In support of our mission, NCC Group continually invests in research and innovation as an intrinsic part of our business model to match the rapidly evolving and complex digital environment. This includes researching AI and ML to understand the risks and opportunities these technologies present. Indeed, we have an internal working group of technical experts who are researching adversarial AI and are due to publish a White Paper shortly on the types of attacks we are seeing, and are likely to see, on ML systems to better inform security research in this space.

1.3. We believe that much confusion has arisen around the terms AI, ML and others. So as to aid clarity, we define these terms below:

1.3.1. Artificial Intelligence (AI) is an overarching term for systems that employ computer intelligence. This includes, for example, systems that can play games against humans, or systems that automate creative processes such as legal writing.

1.3.2 Machine Learning (ML), for us, is a subfield of AI and computer science that provides computers with the ability to learn, without being explicitly programmed, when exposed to new data. This is done through the study and construction of algorithms that produce models from training data which are then used to make predictions on further data. In that context, supervised learning entails an algorithm being trained with labelled data, such as using natural language processing (NLP) to extract relevant textual data from legal documents; unsupervised learning entails the algorithm making its own decisions and inferences, such as arbitration or contract-negotiation mobile phone apps; and reinforcement learning entails data being presented as a dynamic environment, such as in driverless cars or a legal advocacy application, which consumes opposing arguments and learns the best possible defence to those arguments.

1.3.3. Adversarial ML describes an attack whereby ML models are manipulated – usually by manipulating data inputs – with the objective of causing the model to make incorrect assessments. Most ML-based products in use today are ‘black box’ appliances that are placed onto networks and configured to consume data, process it and output decisions without humans having much knowledge of what’s happening, giving adversaries a myriad of vectors available to attempt the manipulation of data that might ultimately affect operations. In addition, a growing number of online resources are available to support adversarial ML tasks.

1.4. Generally, NCC Group considers AI and ML as complementary tools to aid human decision-making with the aspiration of autonomous operation; however, there remain significant questions as to how these technologies can and should be rolled out securely, safely and free from bias.

1.5. ML algorithms are, by design, susceptible to influence and change based on their inputs and lifetime. This presents the opportunity for significant security risks, particularly from adversarial ML attacks. We also believe that it is inevitable that attackers will start using AI and ML for offensive operations to aid their own efficacy. Research and tools to support both of these scenarios are becoming more accessible, datasets are becoming larger and skills are becoming more widespread, and once criminals or maligned state actors decide that it is economically rational or valuable to use AI and ML in their attacks, they will. While no particular sector will be more vulnerable than another, it’s easy to see how adversarial ML could significantly undermine the use of AI and ML-based systems in the application of the law. For example, it could be used to infer how an ML model would make a specific legal decision, and thus what inputs could be crafted to manipulate a desired legal outcome. 

1.6. The democratisation of technology and its widespread availability risk inadvertent consequences too. As a result of ever-growing openly accessible AI/ML frameworks becoming available to software developers that abstract data science and algorithmic details, developers will deploy ML and AI systems without necessarily understanding their underlying mathematics and associated operations, leading to potentially poor outputs.

1.7. In addition, as technologies like AI become increasingly ubiquitous across the legal system and society as a whole, the potential for bias exists. For example, there have been many examples of AI-based facial recognition tools repeatedly falsely identifying minority groups and genders – with individuals with multiple minority characteristics particularly at risk. Further, predictive AI and ML-based technologies can fall into the trap of equating correlation with causation. For example, recidivism scores – used to assess whether an individual convicted of a crime is likely to reoffend – can be based statistical correlations, such as low income, rather than causations. This could result in people from low-income households being automatically assigned a high recidivism score, and as a result more likely to be sentenced to prison. To build a safer and more secure future for all, removing or reducing inherent existing biases and taking steps to ensure that social issues are not exacerbated will be crucial. Led by our Race and Ethnicity Steering Committee, we’ve been exploring what can be done to reduce bias in AI and ML and maximise social outcomes.

1.8. Informed by this work and our technical expertise and research, we believe that the following interventions could help to ensure that AI and ML applications are deployed in a safe and secure manner. Most of these interventions will be relevant to all sectors – not just in the application of the law. It’s therefore important that policymaking is joined-up to avoid the development of silos and isolated initiatives across sectors.

1.8.1. AI and ML should be seen as useful tools to aid human decision-making, rather than replacing human decisions altogether in the short term, outside constrained use cases. AI and ML-based implementations can and do go wrong. In addition, some things – for example, establishing the human story or the extenuating circumstances behind a crime – are hard to codify algorithmically. We therefore should ensure we can augment and question the outcomes produced by these technologies.

1.8.2. Developers and trainers must ensure that datasets are representative and reporting processes established that effectively identify and act on potential bias in a system.

1.8.3. Organisations using AI and ML-based applications should promote good data acquisition and management practices and establish clear processes and mechanisms through which applications and their supporting datasets can be carefully vetted and verified on a continuous basis. Independent, third-party product validation and oversight may help organisations to do this. 

1.8.4. As part of its work to develop a National AI Strategy, the Government should consider what regulatory interventions may be needed to ensure the responsible, secure and resilient use of AI and ML-based applications across all sectors. In doing so, we believe a regulator should be appointed, with the expertise to effectively scrutinise the training, operation and use of these technologies and encourage cross-industry learning about best practice.

1.8.5. In the meantime, a set of principles should be produced by the UK’s National Cyber Security Centre (NCSC) detailing good practice steps that should be taken to maximise the cyber security of AI and ML-based technologies. In addition, the Information Commissioner’s Office (ICO) should be charged with developing principles on the legal and ethical acquisition of both training data sets and resultant ML/AI models and algorithms.

1.8.6. Independent security research is playing an ever more important role as the digital environment – and subsequently the threat landscape – evolves. The UK must ensure that its legislative frameworks – including the Computer Misuse Act 1990 - remain fit for purpose to reflect new and emerging technologies and ensure that important security research can continue and evolve. 

1.8.7. Public and private investment in research and innovation and skills development should continue.

NCC Group is passionate about sharing our expertise and insights with policy-makers and parliamentarians who are tackling crucially important questions about the use of emerging technologies. We would be delighted to give oral evidence to the Committee’s inquiry to help explore the proposals we raise in our submission in more detail.

Questions

2.1. Do you know of technologies being used in the application of the law? Where? By whom? For what purpose?

2.1.1. There are already hundreds of AI-based companies in the legal sphere, ranging from small start-ups to established large enterprises. The common themes of applications and services offered by these enterprises non-exhaustively include:

2.1.1.1. Due diligence – AI tools used to process and assess large volumes of legal and contract documentation

2.1.1.2. Electronic billing – intelligent and automatic billing of lawyers’ time

2.1.1.3. Predictions – AI software used to forecast litigation outcomes

2.1.1.4. Analytics – AI detection of trends from previous case law, win/loss rates and profiling of individual judges and their judgement tendencies

2.1.1.5. Cost reduction – AI arbitration in small claims cases which reduces costs all round, rendering legal proceedings more accessible and affordable (notwithstanding the risks involved)

2.2. What should new technologies used for the application of the law aim to achieve? In what instances is it acceptable for them to be used? Do these technologies work for their intended purposes, and are these purposes sufficiently understood?

2.2.1. There are many opportunities for AI and ML to help professionals and public servants in the legal and justice sectors to do their jobs more efficiently and accurately. For example, a key part of the legal profession involves reading and analysing significant amounts of evidential text. ML-based systems such as NLP have the potential to streamline this task, allowing professionals to process additional information, more quickly, potentially resulting in better-informed decisions.

2.2.2. However, given the security and bias risks presented by these new technologies, and the widely reported instances where AI and ML have failed to deliver intended outcomes, we would caution against “algorithmic authority” being granted to AI and ML-based systems, where machines are making authoritative, yet critical decisions. These technologies should be viewed as tools to augment human-decision making, rather than outright replacements. Further, the wider environment in which the technologies are deployed, and the likely behaviours of users need to be assessed, potentially with controls implemented to prevent over-reliance on the data as appropriate.

2.3. Do new technologies used in the application of the law produce reliable outputs, and consistently so? How far do those who interact with these technologies (such as police officers, members of the judiciary, lawyers, and members of the public) understand how they work and how they should be used?

2.3.1. There are two core components to AI and ML-based applications: (1) the algorithms themselves and (2) the data they use. In the end, any application is only as good and successful as the quality of data used to train it. To ensure AI and ML-based applications can consistently produce reliable outputs, it is essential that steps are taken to ensure the data is up to date, secure and, as far as possible, free from bias. These steps should include:

2.3.1.1. Establishing clear processes and mechanisms through which applications can be carefully vetted and their respective data supply chains sanitised, particularly where data originates from untrusted sources, such as the Internet and end-users.

2.3.1.2. Introducing independent, third party product validation and additional research into required product updates for ML-based systems to ensure their continued security. In our experience, current products often lack third party validation. Many claims made by ML product vendors, predominantly about products’ effectiveness in detecting threats, are often unproven, or not verified by independent third parties.

2.3.1.3. Regularly updating and retraining applications with the latest available data. A system based on analysing case law, for example, could become quickly outdated, and therefore inaccurate, if not kept up to date.

2.3.1.4. Analysing datasets to ensure they are representative and appropriate for the jurisdiction in which they are used. This should take into account the diversity of the development team responsible for sourcing the datasets, as this may result in unconscious biases. Creating synthetic, representative data could be a future solution to this.

2.3.1.5. As recommended by the Centre for Data Ethics and Innovation’s (CDEI) recent review into bias in algorithmic decision-making[1], a multidisciplinary approach to reviewing systems and algorithms - which offers a legal, policy and operational perspective in additional to a technical review – should be taken to reduce bias wherever possible.

2.3.1.6. Responsibility for issues of bias shouldn’t end when products or systems have been released. There should be a clear reporting process that allows organisations to receive and act on information about potential biases in a system. Lessons can be learnt from the security industry where there are established protocols for disclosing vulnerabilities in a system. Indeed, NCC Group is currently looking at how we might be able to use existing systems of disclosure to draw attention to privacy issues and bias.

2.3.2. As the Committee points to, we believe that those who are utilising these systems should have a basic understanding of how they work and what they can – and perhaps more importantly what they can’t – achieve. To this end, we’d welcome a public information campaign. 

2.3.3. NCC Group would also draw attention to the focused skills investment required to ensure genuine UK leadership in AI and ML which we would define as producing core AI frameworks, as opposed to using AI frameworks developed by others. Indeed, AI/ML are closely linked to data science and are very mathematical subjects. While there are many AI frameworks available for use that abstract away from the low-level minutiae/mathematics of AI, there is likely a major skills shortage of people with deep technical understanding of AI and its algorithms. There is therefore a danger that as a nation, the UK will be using AI frameworks developed by other nations, reliant on the assurances that they provide in the security of those frameworks. We strongly believe that this is a much less desirable outcome to being in a position where the UK is the producer of the core AI frameworks (that others might then use).

2.4. How do technologies impact upon the rule of law and trust in the rule of law and its application? Your answer could refer, for example, to issues of equality. How could any negative impacts be mitigated?

2.4.1. As we have highlighted, AI and ML-based applications can be easily undermined if they are not secure and/or free from bias, as far as is possible. Headlines about security breaches or applications leading to false or biased outcomes could quickly erode the public’s trust in the legal and justice systems’ use of these new technologies.

2.4.2. In addition to the steps that can be implemented to ensure the safety and security of the datasets algorithms rely on (as outlined in detail under question 3), organisations should take steps to ensure their software is resilient against outages and/or attacks. As we become more reliant on these technologies, the potential impact of a system being compromised or becoming unavailable for a period of time will increase significantly. Software resilience, such as the use of escrow agreements, should therefore be an important consideration for organisations using AI and ML-based applications.

2.4.3. Industry also has a tendency not to learn from the lessons of others. There are many publicised examples of where AI has gone wrong, such as AI bots learning bad behaviours and becoming abusive and racist in their interactions, and AI-based recruitment processes that have been shown to be biased in their CV analyses and sifting processes. NCC Group consistently sees the same in cyber security, where despite daily publicised data breaches, many other organisations continue to make the same mistakes which eventually results in their own data breach or cyber incident, which could otherwise have been avoided by following industry best practices and learning from the mistakes of others.

2.4.4. Policymakers may want to consider what regulatory levers – such as industry codes of practice or Government-backed accreditation schemes – could be used to ensure organisations in the legal and justice sectors and providers of AI/ML-based technologies are taking the right steps and learning from the mistakes of the past. In doing so, it will be important to appoint a responsible regulator, backed up with the resource and expertise to effectively scrutinise practices in this area.

2.5. With regards to the use of these technologies, what costs could arise? Do the benefits outweigh these costs? Are safeguards needed to ensure that technologies cannot be used to serve purposes incompatible with a democratic society?

2.5.1. Many AI systems require high performance computing power, which is often presented within cloud computing environments. Incessant, high performance computing can be expensive so the cost of operating legal AI systems could be significant, as an additional cost to existing people-based costs such as solicitor and barrister fees. The heavy reliance on cloud technologies also means there are potential costs associated with any lack of availability. Cloud infrastructures do sometimes go down, rendering all applications reliant upon them inoperable until normal service is resumed. High dependence on such technologies could mean that a future outage incurs significant cost to legal proceedings through delays. Good software resilience, including the use of escrow agreements, will help to mitigate against this risk.

2.5.2. The black box nature of AI and ML-based systems makes them hard to audit. This means understanding why an AI reached a certain decision can be very difficult. We note that a key tenet of the justice system in a democratic society is the ability to challenge and appeal decisions. However, if a decision is partly reached using an AI/ML-based system then this may be difficult. One area of evolving research and development that could resolve this issue is ‘explainable AI’. Explainable AI tools help technical experts to understand how and why an AI reached an outcome. We believe this could have an important role to play in the justice system, particularly in facilitating the appeals process.

2.5.3. We also see potential problems with the use of predictive AI and ML in application of the law where correlation may be confused with causation (e.g. as with the example of recidivism scores). ‘Causal AI’ – which can help identify the precise relationships of cause and effect – could have a greater role to play, alongside explainable AI, in deepening developers’ and users’ understanding of the root causes of outcomes and ensuring correlation is not mistaken for causation.

2.6. What mechanisms should be introduced to monitor the deployment of new technologies? How can their performance be evaluated prior to deployment and while in use? Who should be accountable for the use of new technologies, and what accountability arrangements should be in place? What governance and oversight mechanisms should be in place?

2.6.1. As the Committee indicates, not only is it essential that clear processes are established to vet technologies before they are deployed, but there must also be mechanisms in place to ensure that their performance is continuously evaluated. The datasets which underpin AI and ML-based applications need to be constantly updated, maintained and retrained to correct any false outcomes and keep pace with new data and new objectives (e.g. as case law and societal norms change). In addition, security testing and assurance activities, backed up by investment in research, need to be undertaken on a continuous basis to ensure vulnerabilities are addressed and the latest threat landscape is understood and acted upon.

2.6.2. We believe that both the provider and the end-user should be accountable for the use of AI/ML-based technologies, with responsibilities clearly defined. Technology providers need to ensure that their products are secure and should take steps to minimise bias. Meanwhile, end-users should understand the limits of what can be achieved through the use of these technologies, and ensure they are being deployed to supplement human decision-making rather than replace it. To ensure consistency, these principles should learn from and align with the UK Government’s ‘Secure by Design’ approach to smart devices. Under this approach, manufacturers are responsible for only making available those solutions that are secure by design, limiting the user's choice to make 'wrong decisions'. At the same time, users retain a level of responsibility to familiarise themselves with the solutions (particularly where they are deploying technology with an impact on others).

2.6.3. As we have highlighted, we believe that a regulator should be appointed, with responsibility for overseeing the application of AI and ML-based technologies and promoting their secure use. This could be a cross-sector regulator which would then oversee the use of these technologies across all sectors, or could be a regulator specific to the legal and justice sectors which is supported by a central body such as the CDEI. If the latter approach is taken, we echo the Committee on Standards in Public Life’s recommendation[2] that the CDEI is established on a statutory footing and given the resource to effectively support sectoral regulators.

2.7. How far does the existing legal framework around new technologies used in the application of the law support their ethical and effective use, now and in the future? What (if any) new legislation is required? How appropriate are current legal frameworks?

2.7.1. Currently, there is very little legislation and regulation overseeing the safe and secure rollout of AI and ML-based technologies. We would support further reviews into what regulatory and legal levers might be used to ensure organisations developing and using AI/ML are taking the appropriate steps.

2.8. How can transparency be ensured when it comes to the use of these technologies, including regarding how they are purchased, how their results are interpreted, and in what ways they are used?

2.8.1. Explainable AI and causal AI could help to increase the transparency of AI and ML-based systems, by enabling technical experts to understand how and why a system reached a certain conclusion. We believe these tools could be particularly critical in the justice system, where the ability to appeal a decision is a core tenet.

2.8.2. There are also questions about whether the supply chain of an AI system needs to be understood and vetted. We note, and welcome, the Department for Digital, Culture, Media and Sport’s (DCMS) ongoing review of supply chain cyber security. In our response to DCMS’s recent call for evidence we highlighted that this is a complex challenge, but there are workable, practical and realistic steps which can be taken, including: standardising the way in which third party risk is managed; regulating to mandate minimum security standards and exploring means of supplier resilience accreditation; using third party / aggregated security or information exchanges to provide transparency within, and across industry sectors; and, putting specific focus on the resilience of organisations' software supply chains.

2.9. Are there relevant examples of good practices and lessons learnt from other fields or jurisdictions which should be considered?

2.9.1. There are a number of institutes across academia and industry that are developing and promoting good practices for secure, safe and ethical AI. Just one example is the Oxford Commission on AI & Good Governance (OxCAIGG). Despite great resources being freely and readily available, without any mandate to use or follow such resources and guidelines, it is unfortunately the case that they are often overlooked by system developers. This may be down to the fact that the plethora of available resources, studies and research makes for a fragmented landscape. It is therefore difficult for organisations to understand what authoritative sources of guidance and information they should follow. Appropriate regulation that mandates a minimum approach, coupled with education, may help to overcome this issue and guide developers and users in the right direction. 

2.10. This Committee aims to establish some guiding principles for the use of technologies in the application of the law. What principles would you recommend?

2.10.1. There are a number of broad principles we would recommend to ensure the secure roll-out of AI and ML-based technologies. These include:

2.10.1.1. Outside of constrained use cases, AI and ML should only ever act as tools to aid human decision-making rather than replace human decisions altogether.

2.10.1.2. Explainable AI should be the norm – so that outcomes can be reviewed, questions and reversed if necessary.

2.10.1.3. AI and ML tools should be audited, assured and validated before and during their deployment, covering algorithms, datasets and use of technology in practice. For the purposes of auditing AI and ML-based systems, NCC Group has developed ten questions that should be asked which pick up on the discussion points raised above. The checklist is intentionally framed as questions rather than specific instructions as the approach – for example, determining the appropriate frequency of retraining a system – will depend on the system, what it is used for and other changing circumstances. The purpose of the questions is to establish a framework that allows for critical consideration of the technology in question[3]:

1. What datasets were used for training the system and how appropriate/current were these for the target operating environment? This should include understanding how representative the datasets are.
2. How often does the system retrain, or how often should it be retrained?
3. What level of testing/evaluation has been to understand the rate at which the system leads to false positives and false negatives?
4. What (if any) adversarial security testing has been performed against the system?
5. What (if any) defence mechanisms exist?
6. What level of ‘algorithmic authority’ will the system present, or what level of trust will be placed in the system’s decision making?
7. What level of human supervision and authority override will be required during system operation?
8. Are there any ethical issues presented by the system?
9. What level of transparency and auditing exists?
10. What assurances exist in the data supply chain?

2.10.1.4. There should be recognised processes for reporting AI/ML vulnerabilities, biases and remediation methods to allow for cross-sector learning and avoiding the repetition of mistakes, with consequences in place where avoidable errors have occurred.

27 August 2021

11