Dr Kyriakos N. Kotsoglou, Senior Lecturer in Law, Northumbria University/Research Fellow, University of Lausanne — Written evidence (NTL0006)
Law & Technology
Should Legal Officials be The Long Arm of the Algorithm?
Summary
Decision support systems are necessary and valuable. There are however legitimate concerns, both empirical and normative in nature as regards their impact on the application of the law. The hereby submitted evidence will shed light on some questionable assumptions underpinning discussions of ‘Law & Technology’ and explain that legal operations (notably: a judge’s activity) cannot be described in logico-mechanical terms. The abovementioned assumptions are nothing but misunderstandings, misrepresentations and misinterpretations of basic concepts underpinning normative systems. What is more, these misunderstandings about the very possibility of automated decision-making a) threaten to usurp the role of the legitimate decision-maker, and b) necessitate the development of a framework safeguarding the legal officials’ decision-making prerogative. In view of the procedural architecture of Western legal orders epistemic considerations assisted by decision support systems need to be filtered, and validated through a network of constitutional rights, legal and evidential principles and values.
The author of this text is Senior Lecturer at Northumbria Law School (Northumbria University) and Research Fellow at Ecole des sciences criminelles – University of Lausanne. His main area of expertise lies in criminal evidence and expert evidence. Kyri has published widely on the issue of statistical evidence and theory of evidence, and the (failed) efforts to digitise legal processes. He has also acted as senior consultant for a multinational auditing company (PwC, 2018) sharing his expertise on the limitations of algorithmic processes in the legal system. The reason for submitting evidence are the author’s concerns regarding the intelligibility of the discussion on Law & Technology, and the risk of turning legal officials to the “long arm of the algorithm”.
A brave new world? [Q1, Q3]
1. Hardly a day goes by without AI evangelists trumpeting their game-changing achievements: ‘self-driving cars’, ‘brain-like computers’ that can ‘deep-learn’ from experience, ‘hive-mind controlled drones’ and last but not least algorithms that promise to ‘automate law’s application’ are for a considerable number of IT-experts and futurologists clear evidence that we are slowly but inexorably entering a brave new world of ‘intelligent machines’ and ‘automated decisions’ in contexts ranging from the mundane (dictation software) over the political (China’s Social Credit System) to the existential (medical diagnosis).
2. What is more, technological progress needs to be viewed in the light of resourcing pressures across the public and private sectors. The need for increased efficiency and smooth information management, in addition to traditional overarching goals of modern legal orders (fairness, respect of the individual’s autonomy, and factual accuracy) drive, indeed elicit, the demand for the development of automated decision-making processes based on statistical data and inferences.
3. However, this world is neither new nor brave –or intelligent. A.I., both the concept and the field, is all the rage in academic circles for quite a while (von Neumann 1958) alas the initial euphoria was very soon replaced by an ‘appreciation of the profound difficulty of the problem’ (Bansal, 2013, p. 1). To put it in simple words: ‘A.I.’ is a misnomer, for it cannot emulate human-level reasoning and competence (Pearl 2019).
4. The hereby submitted evidence will shed light on some questionable assumptions underpinning discussions of ‘Law & Technology’, and explain that legal operations (notably: a judge’s activity) cannot be described in logico-mechanical terms. The abovementioned assumptions are nothing but misunderstandings, misrepresentations and misinterpretations of basic concepts underpinning normative systems. What is more, these misunderstandings about the very possibility of automated decision-making threaten to usurp the role of the legitimate decision-maker, and necessitate the development of a framework safeguarding the legal officials’ decision-making prerogative.
Areas of Application [Q1, Q3]
5. One of the main areas of application for automated decision-making processes are legal orders, especially the criminal justice system with algorithmic tools being increasingly deployed in areas such as policing (e.g. facial recognition, see Kotsoglou/Oswald 2020), assessment of evidence, sentencing and rehabilitation. Upon careful examination, the use of algorithmic tools impinges on the core aspects of decision-making in the criminal process too: In England and Wales for example, a DNA profile with a likelihood ratio of 1 billion (something which is automatically calculated) in favor of the prosecution’s proposition will trigger a case to answer for the defendant, and can, ultimately, lead (combined with the defendant’s silence or inability to provide a convincing alternative explanation consistent with innocence) to a guilty verdict (R v Tsekiri [2017] EWCA Crim 40; see Kotsoglou/McCartney 2021).
6. The examples of reliance on statistical/automated decision-making processes in the criminal justice system are legion. In the U.S.A. a growing number of states and counties are replacing the bail system by a risk-based system to inform decisions on pretrial release (Starr 2014) while at the same time the First Step Act 2018 assigns a central role to algorithmic risk assessment in rehabilitative programs. The so-called risk-and-needs-assessment (RNA) actuarial tools are championed by Governments, the judiciary, and AI-evangelists (National Center for State Courts 2007).
7. These tools are widely considered to be capable of standardising decision-making, increasing accuracy, eliminating the subjective element from justice administration and, ultimately, arbitrariness. COMPAS is one of several risk-assessment algorithms being used in the U.S. to predict hot spots of violent crime, determine the appropriate type of supervision, or provide information that might be useful in sentencing. Probation, and prison services across the U.K. use a system called the Offender Assessment System (OASys) to (re-)assess offenders at various points during their sentence.
Algorithms and Language [Q1, Q3]
8. The main question to be answered is whether computers understand (any) language let alone the legal language. The answer is unequivocally ‘no’. The reason for that is that computer programs are defined by their syntactical structure; understanding human language is a radically different cultural process as it involves semantics and grammar. The meaning of a word is its rule-governed use, which involves two-way abilities. The ability to follow a rule of meaning –not simply: the ability to act according to a rule– is closely related with intentionality and the ability not to follow a rule (Baker/Hacker 2009). Computers lack the ability of any mental representation of the world, let alone of intentionality.
9. IT-researchers fail to understand that although a super-computer could within seconds ‘find’ in the cloud every word ever written, the individual case to be adjudicated is not already a member of any reference class. An algorithm is not able to identify (or not identify) the new case as an instance of a general rule which we supposedly could/would apply mechanically. Each new decision is a step into the void, as the fact-finder must decide whether he/she can establish a similarity relation between the already adjudicated cases and the new one in order to bring a case under the scope of a legal rule. What is more, the complexity of the individual case poses even for a super-computer an analytically intractable problem, no matter how ‘sophisticated’ or ‘intelligent’ its programmers claim it to be (Cherniak 1984).
Computers and Complexity [Q2, Q3]
10. IT-researchers and A.I.-visionaries fail to apprehend the historic lesson that legal scholars learned at least since the collapse of the Prussian Civil Code (1794) with its more than 20,000 paragraphs trying to anticipate every imaginable combination of facts. In a constantly evolving world characterised by a radically unpredictable future, every codification attempting to be completely comprehensive would only moments after its enactment be in need of radical revision in order to catch the multitude of situations that (can) occur in real life. In law we are constantly facing a problem of (unstable) semantics –not a structural one– due to the indeterminacy of language. Computers do not understand natural languages in any meaningful sense.
11. The underlying assumption in A.I.-literature (especially in ‘machine-learning’ oriented literature) is that by tapping into unprecedented technical capabilities to mine vast amounts of data and create an all-encompassing library of situations and combinatorial possibilities against which to compare the data input, computers should be able of overcoming all the above-mentioned practical hurdles. In other words, the aspiration of automated decision-making hinges on the assumption that we can write a computer program which can anticipate every imaginable contingency and thus release legal officials from the burden of decision-making and the ‘tyranny of the particular’, i.e. the need for discretion. This assumption is flawed, indeed unworkable.
12. Modern legal orders have –at least to some extent– internalised the message that it is a fool’s errand to try to anticipate all possible combinations of facts in the future, which is a condition for automated decision-making. On the contrary, legal orders rely heavily on contextual principles such as the term ‘reasonable’. (Legal) meaning is not an empirical phenomenon to be verified. It is a normative ascription embedded in a form of life. Its complexity outperforms any supercomputer.
The decision-making prerogative [Q3, Q4]
13. The very idea of automated decision-making processes violates the decision-making prerogative of the respective legal official who is authorised to resolve via a logical jump an analytically intractable practical problem.
14. The same processes disregard the normative architecture and values underpinning the legal order incl. its decision-making process. E.g. the recent British case Bridges showed that manufacturers of Automated Facial Recognition (AFR) technology can set the default-threshold value for false alarm rates and false reject rates (Blackstone-ratio) ‘at will or the end user is by design allowed to “change the threshold value to whatever they choose”’ (R v Bridges v SSWP and SSHD, at [24]).
15. Technology companies not only disregard the Blackstone-ratio, i.e. the acceptable rate of factual errors, operating in a respective legal order, but also allow the end-user to set the threshold value for the rate of errors at will, say, at 0,1%, 0,01%, or 0,001%. This can have dramatic implications on the frequency of the so-called ‘matching’, i.e. the algorithmic output of the device, which at best will merely shape, and at worst will pre-empt the decision of the legal official.
Technology and the individual case [Q3, Q7]
16. Another important question is whether statistical information –accurate, as it could ever be– manages to standardise, or even replace decision-making, and if yes (or no), why? The question at its kernel is whether a statistical inference from a taxonomic categorisation of population aggregates to the individual could ever be warranted. The short answer to this question is: No. A scientifically warranted method which could inform us about the individual case is antithetical to sciences’ basic feature: generality. A reference class with a single member would be a contradiction in terms (Sober 2008). Scientific propositions including algorithmic output which is necessarily statistical in nature refer to a target system (reference class) and not directly to an individual case or event. It is the latter that is crucial to the law’s administration. The question is thus who will individualise the statistical propositions resulting from a statistical/algorithmic process, i.e. who has the decision-making prerogative.
17. Legal orders and in particular the criminal justice system of England and Wales have their own established routines for validating procedural claims such as criminal charges. One of those routines is the decision-making process in which a trier of fact decides how to resolve a factual conflict. This normatively structured decision-making process under uncertainty is not a proto-scientific-apparatus, but an advanced institutional tool designed for particular kinds of work in the social arena. Therefore, legal adjudication is not a half-baked routine for folk-validation due to be replaced by automated methods devised by big-tech companies, but a complex adaptive system. The idea that some algorithmically validated (therefore: general) proposition guarantees the (factual and normative) rectitude of a legal verdict (individual norm) commits the fallacy of taking the extra inferential step based on assumptions that go far beyond what can be logically warranted by the underlying procedure (Biedermann et al. 2008).
18. Decision support systems are necessary and valuable. There are however legitimate concerns, both empirical and normative in nature as regards their impact on the application of the law. One of those concerns is nothing less than the fear of a creeping dehumanised justice in which either the authorised legal officials will have to formally cede their decision-making prerogative by deferring judgment to the algorithmic output, or they will feel compelled to accept it. In the words of a senior police officer, human decision makers may lack the confidence and knowledge to question or override automatically generated recommendations (The Law Society 2019, p. 20). They will thus end up being ‘the long arm of the algorithm’ (Kotsoglou/Oswald 2020).
Transparency [Q2, Q8]
19. Transparency is a core value of Western legal orders in general and of English law in particular. Even the English criminal process with its holistic judgments and unreasoned verdicts has quite successfully and despite all its institutional adaptations immunised itself against criticism as a legitimate model of fact-finding. After all, the Grand Chamber of the European Court of Human Rights recognised that the practice of providing no reasons for decisions does not infringe the fundamental rights of the accused. The Strasburg court added however that the ability to understand the verdict is ‘a vital safeguard against arbitrariness’ (Taxquet v Belgium [GC], para 90). It cannot be overstressed that giving reasons for decisions is one thing. Having reasons for a decision is quite another. Opaque algorithms let alone machine-learning on which court decisions may be based, fail on the latter requirement. For cryptic decisions based on intransparent algorithmic processes are a paradigmatic case of arbitrary decision-making. An oracular, sphinx-like algorithmic output is antithetical to core values of Western democracies.
20. Secrecy and democracy have an antithetical nature. Accountability can be regarded as a basic tenet, indeed as a bedrock political principle in liberal societies since Plato, whose concept thereof (‘λόγον διδόναι’) meant that a public decision maker has the duty to give a systematic account, which is explanatory and open to scrutiny. Openness enables us to see which values and which style of argumentation people bring into their decision making. Deploying algorithms with protected proprietary rights or machine-learning tools is not only inconsequential for the individual case but undemocratic too.
21. What is more, the overconcentration of decisional powers on the programmers of the algorithm and their employers who are often powerful multinational companies, ends up being a problem of democratic legitimacy. We should all be concerned.
Human Dignity [Q2, Q3, Q10]
22. Another common feature of modern Western legal orders especially in the wake of the 2nd World War is the emphasis put on human dignity. Although a full-blown analysis of that notion would go far beyond the scope of this text, we do identify an infringement of human dignity when individuals are treated not on the basis of their own actions or omissions but on the basis of the aggregate behaviour of the reference class they belong to. This type of wholesale approach would be antithetical to fundamental (constitutional) axioms of modern legal orders including the principle of individual responsibility.
23. The Equality Act 2010 in the U.K. (and similar legal or ethical frameworks around the world) are a special instantiation of the abovementioned normative requirement. E.g. according to the Equality Act 2010, direct discrimination is when a person is treated less favorably than another because of a protected characteristic. In other words, it is unlawful to treat someone in a way solely on the basis of their membership to a reference class, when the salient feature is a protected characteristic (for example age, race, sex, disability).
24. In view of the set of principles of humane treatment, personal autonomy, and, in effect, individualised evidence, any type of one-size-fits-all approach to decision-making whether in the context of policing, adjudication or probation, raises serious questions about both legitimacy and lawfulness of the respective decision/verdict. In Western legal orders, the individual and not the group reigns supreme whereas an automated approach to law’s administration based on ‘head-counting’ calculations would mean than among other things an individual can be arrested or punished merely for belonging to some reference class, not for his or her actions or omissions.
25. E.g. the act of identifying a person or stop-and-searching an individual solely based on an algorithmically generated similarity score, or even convicting a defendant in a criminal court because of a sufficiently high statistical (match) probability, does not fail or succeed from the point of view of logicality. Insurance companies, banks and casinos consider, and act upon (non-individualised) statistical scores – with great success. In view of the procedural architecture of Western legal orders, however, epistemic considerations assisted by decision support systems need to be filtered, and validated through a network of constitutional rights, legal and evidential principles and values. The latter are (again: in the case of modern Western legal orders) anthropocentric, not group-mediated. The requirement of specific/individualised evidence is not yet another component in the algorithmic set-up of some company’s software. It is the bulwark against automated decision-making processes which can have legal or other significant effects on individuals.
26. Human dignity, prohibition of degrading treatment, reasonableness etc. are normative features; not (falsifiable) empirical claims.
[Note to the reader: Text in square brackets refers to the questions on your Call for Evidence.]
27 August 2021
Literature
Baker G.P. / Hacker P.M.S. (2009). Wittgenstein: Rules, Grammar and Necessity - Volume 2 of an analytical commentary on the Philosophical Investigations – Essays and Exegesis of §§185—242, 2. Wiley Blackwell
Bansal, S. K. (2013). Artificial intelligence. New Delhi: APH Publ.
Biedermann A. et al (2008). Decision theoretic properties of forensic identification: underlying logic and argumentative implications. In: 177 Forensic Science International (2008), pp. 120–132.
Cherniak C (1984). Computational complexity and the universal acceptance of logic. 81 J Philos, pp. 739–758
Kotsoglou, K.N. / Oswald M. (2020). The Long Arm of the Algorithm? Automated Facial Recognition as Evidence and Trigger for Police Intervention, in: 2 Forensic Science International: Synergy 2, pp. 86–89.
Kotsoglou K.N. / McCartney C. (2021). To the exclusion of all others? DNA profile and transfer mechanics—R v Jones (William Francis) [2020] EWCA Crim 1021 (03 Aug 2020). 25 The International Journal of Evidence & Proof, pp. 135-140.
The Law Society Commission on the Use of Algorithms in the Justice System & The Law Society of England and Wales, Algorithms in the Criminal Justice System, 2019, online available: https://www.lawsociety.org.uk/en/topics/research/algorithm-use-in-the-criminal-justice-system-report
National Center for State Courts (2007). Conference of Chief Justices and Conference of State Court Administrators, Resolution 12 In Support of Sentencing Practices that Promote Public Safety and Reduce Recidivism, available at https://ccj.ncsc.org/__data/assets/pdf_file/0019/23536/08012007-support-sentencing-public-safety-reduce-recidivism.pdf.
von Neumann, John (1958). The Computer and the Brain. New Haven/London: Yale Univ. Press.
Pearl, Judea (2019). The Limitations of Opaque Learning Machines. In: John Brockman (ed.), Possible Minds: 25 Ways of Looking at AI. Penguin Press.
Sober E. (2008), Evidence and Evolution, The Logic Behind the Science. Cambridge: University Press, 2008.
Starr, S.B. (2014). Evidence-Based Sentencing and the Scientific Rationalization of Discrimination, in: 66 Stanford Law Review (2014)