20
Examination of witnesses
Dr Kate Haworth, Dr James Tompkinson and Professor Helen Fraser.
Q1 The Chair: Welcome to this hearing of the Public Services Committee. This is our first hearing into police transcription. I am going to ask our witnesses to introduce themselves, and then we will go straight to the first question.
Dr Kate Haworth: I am an academic researcher. My main area of research has been on investigative interviewing for the last 20 years. Since 2018 I have been leading a project called For the Record, specifically looking into issues around the production and use of police interview transcripts, particularly as evidence. I want to mention for the record that I was a practising barrister but that is some time ago now, so I draw on that experience but I am not here in that capacity at all. I am here as an academic and a linguist.
Dr James Tompkinson: I am a lecturer in sociolinguistics at the University of York. I work on forensic linguistics, forensic speech science—my background is sort of in both of those areas. I was a research associate on the For the Record project, working with Kate at Aston before I moved to York. I work across a range of areas involving spoken and written language in the forensic domain.
Professor Helen Fraser: I am the director of a research hub for language and forensic evidence at the University of Melbourne. For probably well over 20 years—maybe getting up to 30 years already, I hate to admit—I have been working on forensic transcription, specifically police transcripts of indistinct forensic audio.
Q2 Lord Blencathra: I have a number of questions for the panel, but first I have a brief question for Dr Haworth. All my questions are about ROTIs and ROVIs, not forensic recordings. I stumbled across your 2018 report called “The Routine Contamination of Police Evidence” in the International Journal of Evidence and Proof—not my usual bedtime reading—and was shocked by the distortions you uncovered, so I am grateful to my colleagues on this committee for the fact that we can question you and your colleagues today. Is that 2018 report still relevant? Have things stayed the same or got better or worse?
Dr Kate Haworth: Thank you for finding and reading it, to start with. Essentially things have not changed at all in that time. That 2018 report was done on a very small data sample and from a smaller range of experience, which was enough to make me think that this needed further investigation because I had the same questions as you: "Surely it cannot still be like that?” That is why we instigated the For the Record project. I know we will talk in more detail about what our research has shown, but I do not think anything we found has changed or contradicted anything that was in the 2018 report.
Lord Blencathra: That is rather worrying. For all the panel, I was the Home Office Minister for Law and Order between 1993 and 1997. I introduced the regulations on twin-deck tape recording and we all assumed it would be played in court in all circumstances, so I was horrified to discover that that did not happen. Even video recordings are transcribed. Can the panel tell us why transcripts are acted out? What is the driver to not using the pure evidence of the recording? Who is demanding transcripts rather than the real thing? What is going on?
Dr Kate Haworth: The simple answer is that I wish I knew. I have spent some time trying to find out why this process of reading out loud in the court happens. I shall explain what happens: the ROTI, the record of taped interview, is produced by the police as a document for the CPS case file, which is then taken to court, so they are presented in court as part of every single criminal trial. What tends to happen is that it is the transcript that is there to start with, not the recording. Then, as you say, that transcript is read out loud by the prosecutor, taking the part of—I might get this the wrong way round—the interviewee, with a police interviewer reading out the part of the interviewer. They are not necessarily the same interviewer; they might be a different gender, or they might have had nothing to do with the case but walked past the door at the wrong moment. They then perform this by reading it out like a play script.
I had that in my time at the Bar and thought, “This is a bit strange”, and I am still trying to find out why it is done. The best answers that I have had have been that it is possibly partly to do with the principle of orality, that you have to introduce evidence orally in court—I see your scepticism there—or it is down to practicality, because obviously we all know about acoustics in old buildings and old courtrooms, where playing a cassette tape was probably somewhat challenging. However, I described it in my report as feeling like a legacy solution to a practical problem that does not really exist anymore. We have digitisation—we no longer have cassette tapes—and we have ABE evidence, and I think all courts now are well set up for having video evidence presented. My suspicion is that, as with many things in the legal system, it just got stuck and nobody has changed it because there has not really been a push to do so, and things have just carried on, but I cannot find a good reason why it still happens.
Dr James Tompkinson: To add to what Kate said, and speaking slightly more generally, I have found when working in this area that often people who do not understand linguistics and maybe even the law tend to think that speech and writing are more interchangeable than they actually are. So I can go to a conference and talk to linguists, and I can come up with the conclusion that speech and writing are different and they look at you and say, “Right, yeah”, but, when you speak to police officers and people who do not have that working knowledge, you find that the differences between the two are not perhaps as obvious as they are to me as a linguist. I wonder if that could potentially be a driver in the general lack of separation of the two modalities and a lack of understanding more widely about what those modalities are and what they do.
Lord Blencathra: Professor Fraser, is there any difference in Australia?
Professor Helen Fraser: I agree with and endorse what both my colleagues have said. I think we can take it a little further back. I have just been doing some historical research, going back to the time before we had mandatory electronic recording of police interviews. I suspect that it goes back to the days when there was just the written record of interview, and it would have made it come alive more in the court to read it out in that way. I agree with what James has just said: there has been a lack of recognition that these are really different modalities. I also think that, when the recordings were brought in in those days, playing audio in court was not easy at all, and then they probably just kept the same practice going. To my knowledge, that is not done in Australia. It might be, and perhaps I just do not know, but I have not heard anybody speaking about it. I was very surprised when I first came across Kate’s work on that topic.
Baroness Cass: Can I add a further point of clarification? It seems like an even greater own goal to let the prosecutor read the defendant’s transcript. They can put any emphasis on it.
Dr Kate Haworth: Yes, absolutely. In terms of simple problematic things that are built into the process, the transcript is part of the prosecution case being presented, and it is only prosecution representatives doing the performance. I should emphasise that generally it is only an extract—we are not talking about an entire play being performed from the full transcript—but it is performed by prosecution representatives. There are also other aspects where unintentional bias can creep in; things like accents are more difficult and can be harder to represent. A lot of what we are looking at is about interference with the evidence and introducing change where there does not need to be any. Inevitably, having someone read the transcript out is a change, and you are putting it only in the hands of one side. Again, I must emphasise that I am not for a minute suggesting that anyone is deliberately doing anything wrong here, but if you are given something to read out then you have to do it somehow.
The Chair: You have created quite a list of people wanting to ask you questions, but we will go back to Lord Blencathra for now.
Q3 Lord Blencathra: The answers to my final question on this might elaborate on some of the problems of reading out transcripts. What might be the impact of the problems in transcripts? How likely or prevalent are such problems? Dr Haworth’s 2018 report identified four stages where things change. First, you have the original spoken interaction. That is changed slightly when it is recorded, possibly changed considerably when it is transcribed and then changed again when it is read or acted in court. The report identified problems with audibility, transcription, editing and, lastly, the serious problem of acting out in court. Can you please elaborate on those possible problems?
Dr James Tompkinson: To try to paint a more general picture, a transcript is only really as good as the purpose that it is intended for. By that, I mean you can have a transcript that may be used as an aid to understanding, with the audio recording being played. That has a very defined purpose, and that helps you to narrow in on what that transcript should look like and how it should be presented. However, when you have transcripts that are multifunctional and multipurpose, and they are being used interchangeably—seen as a direct copy of an audio recording—that is potentially a big problem. My general position on this is that generally, if it is evidence, you want the most original form that you can get. Now, once somebody is out of the interview room then it is already a bit decontextualised, but I would say that an audio recording was a more original form than a transcript, so my recommendation would just be to use the audio recording where possible.
Professor Helen Fraser: I will briefly touch on the previous question about the idea of it being prosecution evidence. In going back to the judges’ rules in the earlier days, I have noticed that now, due to many things that happened in the 1980s and 1990s, we are a little more sceptical about police and prosecutors. Back in the days when these rules were developed, there was not that level of distrust of the police at all; they were very highly trusted. So that is something that has maybe shifted slightly. Of course we should still trust police, but we do it in a different way now.
As for the kind of problems that can come up with transcripts, I am particularly interested in the use of transcripts of indistinct covert recordings—I think that is one of the forms of forensic audio that you have been discussing. In that case, the transcripts are not just used as a convenient record that enables us, instead of having to play through the whole audio, to skim through the page, which is already a useful aspect of transcripts. With indistinct audio, the transcript is intended to assist the court in understanding the words that are said. Unfortunately, if there are any errors in the transcript, people do not tend to pick up those errors nearly as much as we expect them to, and we have had many miscarriages of justice through misunderstanding of this kind of crucial evidence.
Lord Blencathra: What about you, Dr Haworth? I know we could do 30 minutes on the problems with ROTIs and ROVIs.
Dr Kate Haworth: I will be very brief. I go back to the legislative intent of PACE, which was to bring in recording in order to reduce the possibility of interfering with evidence and of people being misrepresented through the records that were being produced by the police. Building on Helen’s point, the police now, particularly in the UK, are acutely aware of the importance of getting this right. Unfortunately, the process of converting to a transcript, and particularly of reading it out, seems to undermine that intent.
In the PACE codes, there is no actual requirement to transcribe, which I was quite surprised about—it is simply practice. However, one thing that the codes do mention is not misrepresenting the demeanour of the person, and that is the key thing that goes missing. It is the demeanour that you lose. It is not what people say but how they say it that can be so important. In terms of why that matters, we are putting this in front of a jury as part of the evidence, and the point of criminal evidence in a court is to invite the jury and the judge to assess and literally judge the person that they are listening to, in which case a partial representation or something that has been changed in some way is always a problem.
As James has said, this is criminal evidence, so the basic principle ought to be the same as for any other type of criminal evidence, like crime scenes. We all know now that we do not expect the police to blunder through a crime scene and touch things. That is very well known; I have noticed that in TV dramas now it is a shortcut to show the audience that we have gone back in time. You see a police scene and the police are walking through it, and everyone now thinks, “Oh, gosh, we wouldn’t do that now”. What we are talking about here feels like the same thing, but we just have not quite caught up with linguistic evidence.
I feel that asking how often these things go wrong is the wrong question. The point is that there is a risk every time. I think James had the numbers on how many of these transcripts are produced, and it is thousands every year, and every single time you are relying on a transcript you are creating a risk. How often that risk comes to fruition is probably not the most important question. The point is: if it is a known risk that we can avoid, why not just avoid it?
The Chair: Lord Bradley, because none of us has yet declared our interests the first time we spoke, could you start us off by doing so?
Q4 Lord Bradley: I have no specific interest to declare; I just have a general interest in the topic. I was interested in Dr Haworth’s reply that video equipment is now pretty good. Admittedly, individual experiences do not necessarily replicate through, but my experience as a juror in a criminal court in Manchester where there was video evidence from a child was that it was appalling and none of us could hear it. So they adjourned, transcribed the evidence and handed it out individually to us as jurors to read and come to our conclusions on the evidence on that basis. That did not seem to me to suggest that their video equipment was good. I have been to a number of courts where there has been investment in that area to improve it. I would not want to think that my experience was replicated more generally.
Dr Kate Haworth: Yes, I apologise if I implied that it is always good. I do not mean that; I mean that it is a lot better. The technology that is available is certainly better now than when this practice developed in the 1990s. I am sure there are many cases where it really is not very good at all but, given the equipment and what could be done, it certainly could be. I think James can talk more about the practical side of that than I can.
Dr James Tompkinson: There is definitely a point in what you are saying. There is something to raise that could be true of any type of audio recording where the quality might be less than ideal. If you have had somebody transcribe that under one set of conditions, when they had really good-quality headphones and were able to go back and relisten to certain sections and play it over and over again, and that audio is then played in a courtroom in very different acoustic circumstances, you may end up in a position where there are things in the transcript that the jurors or the triers of fact cannot hear, just because of technical factors. There is a cautionary tale there. Wherever possible, if you are going to play audio, particularly indistinct audio of the kind that Helen was talking about, it should definitely be a consideration to try and give the triers of fact something that reasonably resembles the listening conditions in which the transcript was produced. Otherwise, exactly as you said, you run the risk of having a document that says something but the triers of fact say, “Well, I can’t hear that”, and it is not because of the recording, it is because of the playback facilities.
Q5 Baroness Wyld: I have no relevant interests to declare. I want, if you can bear it, to go back to the point about acting out. You set out how it works and how the prosecution reads out the transcript, and you set out what you see as the potential risk. Can you set out for us the role of the judge and the defence team within that context?
Dr Kate Haworth: Essentially, they do not really have one. it is simply that when the prosecution are presenting their evidence, this is how they choose to present it, or this is how it is routinely presented.
Baroness Wyld: So if our worry came to fruition and somebody overdramatised, for example, although you were careful to say you are not suggesting that happens often or even at all—
Dr Kate Haworth: Not on purpose.
Baroness Wyld: But can the judge intervene? Can the defence challenge?
Dr Kate Haworth: I would assume so, yes. As I understand it, anyone would have the discretion to challenge it, but first you have to recognise the issue, as James said, that sometimes people tend to assume that there has not been much of a change and it does not really matter, despite the fact that we all fundamentally know how things can go wrong from text to speech. That is why emojis exist. We have probably all had the experience of sending a text message where you intend one meaning but the recipient gets a completely different meaning, because the message is just the words. I think we all know that there are differences and problems, so yes, it could be challenged.
Baroness Wyld: What I am getting at, speaking as a lay person—this will already be blindingly obvious to you—is that within a trial everything is disputed and there is a sort of ping-pong going on. Is that not the case for this evidence as well, where there is an argument over how something is presented and the jury has to assess that argument? Is it that different from many other pieces of evidence?
Dr Kate Haworth: It certainly could be. There is an element of people just being used to it, so that is just how it is done. Essentially, if you are the defence lawyer, you are not necessarily challenging what the individuals are doing; you are trying to challenge the entire process. That feels like quite a big ask in the middle of a criminal trial on a Monday morning in some Crown Court, but you could—but then what exactly would you be disputing? Still, I encourage defence lawyers everywhere to do this. It is a valid question. Why not? Could it be challenged more? I think it is more about the process. The system is set up that way, and it is quite hard as an individual lawyer to be challenging the system. If there were particularly egregious examples, I am sure they would be challenged.
Baroness Wyld: They could get up and say, “You have overemphasised that point” or “You have presented it in this way but isn’t it plausible that it could be different?” They could say to the defendant, “Is that how you said it?”
Dr Kate Haworth: They could, but I do not know of that happening. As I said, I think people tend to just go along with it because it has always been done that way, but what you describe is theoretically possible.
The Chair: And the accused could do the same. They could say to their lawyer, “That is not how I said it”, and then they could ask to listen to that part of the transcript, could they not?
Dr Kate Haworth: All of which would be incredibly time-consuming. The point that we really want to emphasise to the committee is that you can avoid that risk completely by simply using something closer to the original evidence. Even the transcripts, I argue, would be better than reading aloud.
Q6 Lord Carter of Coles: Good morning and good evening to Australia. I have no interest to declare in this inquiry. My question is about how far transcripts effectively reflect interviews or forensic audit. We have had some of the answers to that already, but it might be interesting to see how we can make transcripts better—or did I hear from Dr Haworth that in fact the risk is so great that we should move to audio?
Professor Helen Fraser: You are right to say that the risk is so great that we should always play the audio, even if we have the transcript. A transcript is still useful; in the case that we heard about before with the child who could not be heard, a reliable transcript there could well have been useful. It is similar to seeing subtitles on the television; as long as the transcript is reliable, it is as good as having subtitles. It can help you to hear what is said. If the words are in the transcript, then you can hear in the audio the intonation and other kinds of information that, as Kate has said, is reflected with emojis and other things in text messages. We want to avoid having a transcript that does not enable that.
It is often thought—I think this is one of the misconceptions that James was referring to earlier—that if only the transcript had all the information in it, then we could just use the transcript and not have to use the audio. Unfortunately, it is just not like that. The analogy that I like to use for that is a map. A map is a representation of a landscape. It has a function, a purpose and an end user, somebody that you know is going to use it. If you tried to give an Ordnance Survey map to somebody when you were just trying to show them how to get to such-and-such shop, that would be worse than having something with less information. If you went the whole hog and tried to put everything into a map, you would be recreating the landscape, and that would be completely unwieldy. The point of the representation in a transcript is to assist the end user in whatever purpose they have with that transcript.
Lord Carter of Coles: Thank you, that is helpful.
Dr James Tompkinson: I will just make a couple of points to build on what Helen said. A challenge that we encounter a lot in our research is that you are trying to make recommendations and help a system, but of course every speaker, every person, is different and so is every interview, and you are trying to find a way through all that.
We talk about “effectively reflecting” in terms of accuracy, but it is difficult to put a number on how accurate a transcript is. One suggestion is something called word error rate, which is where you calculate how many words are correct and how many are mistranscribed, but that treats all words as being the same when they are not. The difference between “a” and “the”, two words that could have lots of evidential salience, is huge, but that metric would treat those things as being the same.
Another point regarding police interview transcripts is that, in the ones that I have encountered, sometimes there is a mix of verbatim information that has been transcribed and information that is summarised by the transcriber. Now, if a transcriber summarises then they have to bring their voice into that transcript. If you are going to summarise what I am saying now, you need to use the words “James said” as a minimum. That means the transcriber is left in a position where they have to decide what to leave in, what to take out, what to summarise and how to summarise it. At the very least, that is going to bring in some level of inconsistency. It might not be accurate. You might have two summaries that sort of reflected the audio, but they would be quite different from one another. In little experiments that I have done with students, I give them a piece of audio and say, “I want each of you to produce a summary of this”, and you get as many different summaries as there are people. These things are worth thinking about when we talk about accuracy. You can say it is an accurate transcript but actually it is a bit more complicated than that, and when we make recommendations, we are trying to think about ways to deal with that in a systematic way.
Dr Kate Haworth: Helen and James have answered most of this for you. The main point is about summarising, which I spoke about more in my report. A key problem is that these are called transcripts but the transcribers do not write down all the words. Consistency, the word that James mentioned, is incredibly important, because every police force has different practices on this. Some summarise, some do not, so you can go across a border and you might get different treatment, which is problematic. Then the summarising is done by transcribers, who are employed essentially as typists. I emphasise that the transcribers I met were incredibly dedicated and aware of the seriousness of what they were doing, and they are doing the best job they can, but they are being asked to summarise a legal document that is criminal evidence. When I spoke to them, they were talking really quite articulately about things like duress, consent and points to prove, which they have learned through listening to lots of these interviews, but that is surely not the right way to do it. They have no input and no training, but they are the people tasked with picking out what is most important evidentially in a police interview, and then—I emphasise this to the committee—it is that version which is presented to the court as evidence. So summarising is another key area that we want to bring to your attention.
The Chair: We will move on to training in the next question, so we will be able to pick up some of those points then.
Q7 Lord Shipley: I have no particular interest to declare. I have a specific question for Professor Fraser. A few minutes ago, you talked about there being “many” examples of miscarriage of justice. In your written evidence you said there were numerous cases of actual or potential “transcript injustice”, and at the end of your written submission you said that miscarriage of justice, acknowledged or not, is inevitable. I am trying to get at whether there are any numbers. How many cases are we actually talking about? What examples can be given of damage to individuals caused by this? I note the reference to Dr Haworth’s 2018 report as well. This is really about getting to the extent of the problem that we could be dealing with. Do we know?
Professor Helen Fraser: What an excellent question. That is something we would all very much like to know. We in our hub have made several attempts to try to quantify that in some way, but it is extremely difficult to do. One reason is that you cannot just go to a database, look up how many cases there were with indistinct audio, get a sample of, let us say, 50 cases—something that we have done with other types of language evidence—and look through to see how many of those 50 have different types of issues, and quantify things in that way. We do not really have the sort of accounting that enables us easily to do that kind of quantification, but we are just starting an important new project aimed at doing it. At the moment, we mainly have quite anecdotal evidence, and there is some value in that, but we do not have the numbers that would really quantify it in the way that you are talking about.
Still, something that can help us to get an impression of whether it is a fairly large problem is, as one of the others mentioned earlier, the large number of interviews that are conducted and recorded, and of course a fairly big percentage of them will be turning up in court one way or another. The same goes for forensic audio. Covert recording surveillance is done for a lot of serious crimes these days, and they collect a lot of that kind of evidence, which is used in court quite a lot. So, while we can be fairly sure that it is a biggish problem, we do not know exactly how big, and we would definitely like to. For me, an important recommendation is to try to enable the accounting, maybe with key words that could be used in cases that would enable us to track what exactly is going on.
It is not all doom and gloom. Some cases are done very well. We would like to know that and be able to differentiate what makes a case go well versus one that does not. Studies like that have been done in the social sciences for many other types of evidence, and they are extremely insightful and have been beneficial for the whole social justice area. I for one, and I think the others would agree, would be supportive of enabling more research.
Dr James Tompkinson: I cannot tell you how many miscarriages of justice there are, but when I was working on Kate’s project at Aston, we sent FOI requests out to all 43 police forces in England and Wales to ask them how many police interview transcripts their units were producing. I can give you some figures: one force with 14 full-time transcribers produced 80 to 90 a week, so 4,000-ish a year, while another force with seven full-time transcribers produced 29 a week, so about 1,500 a year. Not all forces reported it in exactly the same way; some said, “It’s this many hours per month”, and that kind of thing, but you are talking about quite a lot of transcripts or records of interviews being produced. We cannot quantify or qualify many of those, but the scale of the number alone is worth just mentioning in response to your question.
The Chair: It might be useful for you, if possible, to let us have that data. That would be great.
Q8 Lord Prentis of Leeds: The moment may have passed for this question. Still, I must declare an interest to begin with: I have represented police support staff outside London for over 20 years. I agree with the comment made by Dr Haworth that police authorities vary in the way that they handle work. However, when this matter is discussed in the papers, they talk about the police doing the transcribing, but my impression is that it is not the police who do transcribing, as Dr Haworth mentioned. It could be the secretary, a community support officer or just someone who was in the office at the time. What is the standard procedure for transcription?
Dr Kate Haworth: I repeat that I am very supportive of the police support staff who are tasked with this; it is very much about the process, not the people. For most police forces, there is a dedicated pool of typists who are specifically employed for this task. In fact, in one force they are called audio summarisers, which tells you the importance that is placed on the summarising. So, they are employed for this task but essentially they are just typists, and those are the people who are doing the task. In terms of standards—
The Chair: I am going to ask you to hold back on that until we get to the next question, but it would be really useful for you to clarify who does this. Is the answer that anyone who is asked to can do it?
Dr Kate Haworth: No, they are specifically employed for the task. They are ROTI typists who are employed by the police force. Occasionally, some interviewers said they would do it themselves, but mainly there is a dedicated pool of people who are civilian employees of the police, so it is a police task that is done within the police, but it is civilian employees within the police force who transcribe the ROTIs. They will occasionally do other similar audio, such as 999 calls, but it is their job specifically to do ROTIs.
Q9 Lord Laming: I confirm that I have no relative interests to declare. I thought the papers that each of you provided for the committee were extremely helpful, so thank you for that. My question, which is simple and straightforward, follows on from what we have just been talking about. One of the things that struck me in reading the papers—I hope I did not misunderstand this—is that, while there are national rules about how the police investigate people of all kinds and they have to follow the rules about how they conduct their investigation, there are no national rules about how the transcription is made. What struck me in the papers is that different police forces might have a specially dedicated team or, as one of the papers referred to, hand the transcript to the junior typist in the force. Because transcription is so important, for the reasons that you have spoken about, should there not be national procedures about the way in which transcription is handled?
The Chair: If Baroness Cass asks her question now, we can roll the answers into both, if that is okay.
Baroness Cass: I have no relative interests to declare. We have got very clearly from the papers that it is a postcode lottery in how this is done. When it comes to this group of people who are transcribing, there are various things. You have the input, if you like, of their qualifications when they arrive, and then the support and training they get once they are in. One of the papers emphasised the importance of experience and, on quality assurance, James made some useful and important recommendations about peer review and so on. Could you say a bit about the relative importance of each of those aspects, although we understand that it varies enormously from force to force?
The Chair: If you could respond to both those questions, that would be really helpful.
Dr Kate Haworth: Hopefully this will be a shorter contribution, because the short answer is no. There are no national standards and no guidance, and everybody does it differently. That was literally an expression used to me by two separate groups of transcribers, years apart in different forces: everybody does it differently. And that is the problem. They are trained in the practical systems, but again we go back to what James said earlier: there is a lack of recognition that going from speech to text is a real skill, and that is the bit that is not trained. They learn through a kind of peer guidance because the lack of training and guidance creates a vacuum.
I gave you the example of question marks. An officer had once told one transcriber, “Don’t put question marks in”. She never put a question mark in again and, because they work as a group, nor had anyone who came into her group. When I brought the groups together—this was within one force but they had never really got together before—two other groups were just horrified. They had never seen that, and had no idea that other transcribers in the same force were doing things differently. Others would put a question mark at the end of everything a police officer said, but of course they are not always questions, so that is also misleading.
Consistency is the key. Whichever way we choose to do it, it should all be the same. Everybody should be able to look at a transcript and know, for example, what “dot dot dot” means, which we currently do not.
Baroness Cass: You mentioned peer review. Is that happening anywhere? Is it realistic?
Dr James Tompkinson: If I go back a step, I can definitely answer that. The short answer to your question is yes, you want to try to level the playing field if you can. How you do that is of course the next step, but it is certainly a good idea.
To draw another point of comparison, in the UK there are a small number of phonetically-trained experts who work in independent labs and produce transcripts of indistinct audio that can be admitted as expert evidence in court. What struck me when I joined this project was the disparity between these two worlds. You have one world that treats transcription evidence as something that is done by experts, albeit a small number of them, and then the other where the transcripts are done by police civilian staff, as Kate said.
So, implementing peer review and checking is a good thing. Of course, you cannot guarantee that you will catch every error, but I cannot think of a good reason not to do that. We could borrow from the world of forensic linguistics and forensic speech science, where checking findings is recommended as good practice by certain codes of practice documents—it is in the Forensic Science Regulator document. So, there are some areas where simple practical things could be usefully borrowed.
It is easy for an academic to say, “You need to do this, this and this”, but what then needs to be worked out is how to ensure that those recommendations are practically implementable so that they work. There are ways to do that, but they will involve people like us working with police officers and other people to get it right. I do not want to say, “This is exactly how you should be doing it”, but my opinion is that having checking and peer review is a good idea because you might catch some errors, and I do not think that is a bad thing.
Baroness Cass: Professor Fraser, feel free to enlarge on any of that. I do not know whether you have an international perspective on whether we can learn from other jurisdictions.
Professor Helen Fraser: I would like to make two points. I am supportive of everything that has already been said. In Australia we have dedicated transcription agencies that do police interviews. I have not investigated them to anything remotely like the extent that Kate and James have there, but on the other hand, I have not heard many complaints. Even among people who work with transcripts, it is not something that has come up as a major problem. It is professionalised to some extent here, not as much as court transcription is professionalised but much more so than what I understand is done there.
My second point is that it is extremely important to have standards, protocols and consistency across different police forces. In general, they are useful to have for training and all the other reasons that have been said. The only caveat that I would like to bring in, and this might be what James was getting at at the end there, is that it is important to test that the recommended standards are going to give the outcomes that you want. I am a great believer in evidence-based practice, and I think that is congenial to police these days as well; there is a lot about evidence-based policing. Studies are done to see what works and what does not, which we were talking about before, so that well-informed recommendations can be made. Should the question marks be there or not? Should they only be there when there are questions? Who knows? We probably all have opinions on those things, but there is probably a good way of determining what is the most useful in terms of helping people to understand the material appropriately.
Baroness Cass: Anything else internationally? Are there places where they have got rid of transcripts and just use the audio?
Dr Kate Haworth: International comparisons are slightly awkward because the legal position of the transcripts is different. I do not know how widespread it is that the transcript is introduced in evidence; it is certainly different in inquisitorial systems, for example. Another key difference here is that we have the qualified right to silence, so that courts can draw inferences if you say something different or fail to mention it. We all know from the caution that if you have not mentioned something sooner, then the court can “draw inferences”, which would inevitably be a negative inference. However, that only works if the point of comparison is reliable. If you have a complete right to silence, as many other countries do, then the position is different. So those are particular reasons why it is difficult: the fact that a transcript is introduced as a prosecution exhibit every single time and the fact that we have the qualified right to silence mean that it is being used directly for comparison. Maybe you failed to mention something, but maybe it was summarised out. Those are particular features of this jurisdiction.
The details of the checking are one thing but, more generally, while it seems glaringly obvious that transcripts should be checked, we found that that is not necessarily happening, and a particular factor in that was who is exhibiting the record. In one force that I went into, the transcribers formally exhibit the record. In most other places, and this was the previous practice in that force too, the interviewer exhibits the record, but I found in that force that the transcribers were still doing what they used to, which was passing the transcript to the officer for checking with a note saying, “Please check”. The officers were thinking, “No, it’s your exhibit. You’re responsible. You’re signing a statement saying that it is accurate, therefore I can’t touch it”. So, they were not checking them and there was a real possibility in that force that nobody was checking at all. So, the minutiae of how we do this is one thing, but the simplest thing that we recommend is that they ought to be checked. There is no system for checking them at all that we found at the moment.
Baroness Cass: I want to ask a tiny question of fact. Someone said in the written evidence that the defence is able to check the MG15, but presumably they do not have access to their own recording, and sometimes you yourself do not remember what you said.
Dr Kate Haworth: Yes. These are disclosed to the defence, at which point the defence have the opportunity to listen and check through. Things will have changed now, but when I was in practice, the only time I ever heard or had the opportunity to look at a recording, I had literally just walked into chambers. I had not eaten lunch. I was handed a file to go and do a trial 30 minutes later—it was a 15-minute walk away—and I had a choice of either reading the file or somehow trying to listen to a 45-minute interview. That was literally the only time I ever saw the recording. Now, obviously that is the Bar and there are solicitors before that stage but, again, it is about the simple awareness of what has been handed to you.
I am sure that most lawyers—certainly I would have put myself in this category—assume that someone has checked it. Going back to what James said about the uses to which they are put, often reliance is placed on them because simply nobody has the time. There is an assumption that surely this process has been designed by somebody, but it has not. We have ended up in a situation where there is an assumption of checking and of all sorts of things that simply have not happened. That has emerged as a process.
The Chair: That phrase you used—was it “present the exhibit”? Would you say exactly what that means and entails, please?
Dr Kate Haworth: Yes, although I will say that I am not entirely sure. My personal experience goes back 20 years and I am sure it has changed, so I am going off what I have been told by my participants in the focus groups. The transcript will be produced and passed to the CPS, then you, the person who has produced it, writes a statement saying, “I have created this record and I certify that this is an accurate reflection of what was said”, and then that document will be another CPS file type. That is my understanding.
The Chair: So it is like ownership. It is who it links back to.
Dr Kate Haworth: That is the problem: unfortunately, nobody was clear about whether that passed ownership or not. You are right: ownership is a key problem. The exhibiting was seen by the police interviewers as giving ownership to the transcribers, but the transcribers, who are there as typists, did not think they were being given that responsibility, so it was falling between the two. That is what I mean by exhibiting.
The Chair: That is really helpful.
Q10 Baroness Pidgeon: I should say that I have no interests to declare in this investigation. AI is the obvious thing to throw in here, because that levels it all out, does it not? There are different AI systems. I have seen some evidence in our briefing from Dr Lauren Harrington, which shows how very different systems are, and you touched on the 71% word error rate earlier. How far do you think AI could help in this space, even if not now but in the future?
Dr James Tompkinson: I will just say that I work with Lauren Harrington at the University of York. Her PhD work is the place where I would go to talk about this, but I can speak a bit more generally. In one sense, it is a difficult question because the developments in AI and speech technology are moving so fast that we do not know what is going to happen in future, but I have some concerns now—some cautionary notes, I suppose—to put with this. I can see how it could be seen as a kind of fast track to standardisation, because a system is going to give you a transcript that is notionally the same. However, I would be wary of any claims like, “Our transcription system is 90-whatever per cent accurate”, which you see quite a lot, for the reasons that I explained earlier: it depends on what speakers you are using, what the system classes as an error and the quality of the recordings that you are going to put into that system. Generally, the poorer quality the recording, the worse job it is going to do.
There is also the risk that an AI system could produce something that was phonetically plausible and then prime you into thinking that you were hearing something that was not there. We have all had the experience of listening to a song or seeing some lyrics written down and thinking we know what those things are, then you see the actual lyrics and you think, “Oh, that’s something different”, but how you parsed that speech was phonetically plausible.
Another issue goes back to what we were saying earlier about the element of responsibility and accountability. If an AI system produces a transcript, who is responsible for that transcript? At least with a human transcriber, you can go and explain it; you can explain to a police officer, a jury or whoever, what you have done.
I have a third concern, as someone who frequently has Zoom meetings for work and finds that Zoom struggles with my voice at times, about the element of a non-standard speaker, someone who does not speak with a standard southern British English accent. That might be someone from the north-east or who has English as a second language or something like that. If you are going to recommend that a system should be used at policy level, you want to make sure that that system is treating speakers fairly, and you do not want a system that will work really well for some speakers but not for others. If you have that, you have essentially swapped one problem for another, and I do not think you want that.
Those would be my cautionary tales about the use of AI as it stands right now. I also do not think it should be used for indistinct audio, because the poorer the quality of the recording, the less chance you have of it being accurate. That is what I would say about AI.
Dr Kate Haworth: I am going to defer completely to the others on this.
Professor Helen Fraser: I completely agree with everything that has been said. A funny experience that we have here a lot, when we go and talk to lawyers and others about indistinct forensic audio and all the problems with that, is that the first question, inevitably, is: “Why don’t you get AI to do it? It’s so much more objective”. We are starting to have a little more experience with AI. I use AI transcripts a lot every day because they are so convenient but, oh my gosh, they make an awful lot of mistakes.
I will add that the reason we know how many mistakes they make is that we know what we call ground truth—we know what the actual thing was that was recorded. However, especially with forensic audio and to some extent even with interviews, we do not have that reliable reference to decide whether it is right or wrong. Some things are clearly wrong if they have been transcribed in a garbled way, but at other times, what it says could be correct and unless you have a good reason to question it, you will probably accept it. So, I would not recommend AI for this type of thing at all, at least for the foreseeable future.
The Chair: Does anyone else want to come in on that? I will come in, then: it is going to happen, though, is it not? It is going to be very difficult for your bit of the world to say, “We’re not going to use AI”. I take what you say, that you cannot just trust it and it needs checking, but is there perhaps a need to try to jump ahead of all this and say, “It is bound to happen. This is how it needs to happen to make it as good as good as possible, et cetera”? Otherwise, you are on one side saying, “Don’t do it” but meanwhile the others are doing it without the benefit of your expertise.
Dr James Tompkinson: That is a good question. In one sense, I do not want to be caught trying to predict the future but I also agree with what you said. This is a problem, and I can give you one practical example of where I have a concern about it, maybe not at an accuracy level but at a process level. A piece of technology that is not primarily used for transcription—something like body-worn video—might have an added system that will produce a transcript. My concern, and maybe this addresses your question, is that if we end up in that space, then we are taking the production of a transcript outside of a process that we can manage.
We do not know what is going to happen with AI. If I was going to make a recommendation, I would say that it goes back to what I said earlier: you want to make sure that it is doing the job it is intended for, and that it is fairly represented in all types of speakers as best as possible. If we end up in a place in the future where that happens, then we could have a look at it, but until that time, the cautionary tales that I have given will still apply.
The Chair: Lord Blencathra, has the particular question that you wanted raised been covered?
Lord Blencathra: Not really.
The Chair: Then before I ask the final question, which is on recommendations, do you want to ask that question, so that it is covered?
Q11 Lord Blencathra: Dr Tompkinson said in answer to the last question in the survey—what would you change if a police suspect interview needs to be presented to a jury?—that it should be the audio or the video. The guiding principle, he said, should be presenting evidence in its most original form to the court. Dr Haworth, in answer to the same question, said, “No written version can adequately stand as a complete replacement for a recording when it comes to being used as evidence in court … I suggest that the default position should be using the recording. Perhaps the single most important recommendation I can make to this inquiry is to bring an end to this practice”. That seems fairly clear, but could you briefly elaborate on why you feel so strongly about this and any possible impact it may have on miscarriages of justice?
Dr Kate Haworth: The basic point that we have been trying to impart to the committee is that it is about reducing interference and reducing the risk of contamination of evidence. Everything comes back to that. It is like the fact that we know we are not supposed to eat ultra-processed food; it feels as if we know that every process takes it further away from where we started. All the things that we are looking at are additional processes, so the simplest thing to do is to reduce the number of processes that the evidence undergoes.
There is something that we have not really covered, and it goes back to your question about how serious a problem this is. James and our previous colleague Felicity Deamer did experimental work. James can talk about this better than I do, but I will give you the headline. We gave people the same data. We gave one group the audio and another group the transcripts, and we got measurable differences in terms of how those potential jurors perceived the person speaking. Most significantly, on the question, “Do you think the interviewee is telling the truth or not?”, we got a statistically significant difference between those groups depending on whether somebody had listened to the audio or read a transcript of exactly the same data. That should not happen—it should not matter which version you see—but our experimental work shows that it does. We cannot get inside the heads of jurors and say in an individual case, “That’s how the jury decided”, because we do not mess with juries and that is right. The closest we can get is our experimental work, which I think had 250 participants in the end. James can talk more about that, if that is appropriate, because that is the closest we can get to answering it.
Baroness Cass: In one direction. That is quite important.
Dr James Tompkinson: In follow-up work that we have done, I have said that it will depend on the interviewee, and indeed on lots of factors, but even the fact that there is the potential for there to be a difference in any direction is a problem, and it is good enough to raise that sort of caution. If you have an audio recording then you should play it. In the modern world, there is the ability to do so. We talked earlier about extracts, and you can extract bits from digital recordings quite easily now. Once, it would have meant somebody rewinding a tape to try to get to the right place, but that is not the case now. There are simple technological things that can address some of the cautions about that that we might have had before.
Q12 The Chair: We will move on to the last question, and I have no interests to declare either. We will make some recommendations to the Government and we wanted to ask you what recommendations you would recommend we make. However, these are only recommendations to the Government; these committees do not make recommendations elsewhere. Do you perhaps have one or two things that you might like us to consider?
Professor Helen Fraser: The key recommendations are clear from everything that has come so far. The first and strongest one would be to end the acting out of interview transcripts, and in fact even minimising reading from transcripts. It is much better to play the audio so that the court has both the audio and the transcript, because they work together as long as you have a reliable transcript. I would add that we should make sure that lawyers are aware of the risks involved in these practices. It came up earlier that lawyers might not object enough when this is done, possibly just because they are not really aware of the risks. The next point more or less follows from that: all audio used as evidence in court should be played in court with a reliable transcript.
Following from that, let us take appropriate steps, out of all the things that we have said so far, to improve the reliability and usefulness of transcripts of both interview and forensic audio, noting that they have different requirements and recalling what came up just then, which is an example of what I call evidence-based practice. It would be a natural assumption that it would be similar to play the audio or look at the transcript, but we now know from the experiment that James has just recounted that that is not the case. So let us do some research to find out which ways work and which ones do not, and use the ones that work.
Dr James Tompkinson: To make some additional points to what I have already said about not playing audio, the recommendations that I made in my report speak to what we talked about earlier and what I have tried to describe regarding police suspect interview transcripts. They could level the playing field a bit to minimise the existing amount of variation. That can be standardised in standardising summarising processes, so if we are going to use them then let us think about how, but if we are not then let us make that a policy. We could have a standardised layout that could be used by all police forces, and we could introduce a standardised checking process.
The other thing I want to say goes back to what we said right at the start. If we are going to start investing money in this then a good place to start might be to make sure that the transcribers doing this job have good-quality equipment, such as decent headphones and playback software, so they can go back and listen to sections of audio repeatedly. I do not know what the position is or what they are using, but in my opinion that would be a better investment than a piece of AI tech that said, “Oh, I can do this, that or the other”. Everything I have said in my report is about trying to get a more standardised process. Once we have that, we can start looking at the fine grain in it.
Dr Kate Haworth: I endorse the things that Helen and James have said. It is about standardisation and consistency and reducing interference as far as possible. However, I am also aware that it is quite hard to know where to target this, which I appreciate is a problem for you in terms of writing a report to the Government. Where do you go for this? There is a general lack of ownership and responsibility for it. I thought it would be easy to find someone and say, “This is a bad thing”, but there is no one person, no obvious target. The only legislation about this is the Criminal Justice Act 2003—Sections 133 and 134(1), I believe, although I will check—which says that the transcript is admissible as a copy. That is the only clear target I can see for saying, “These aren’t copies”. That is wrong; they are not. You cannot say that a transcript is the same version as the audio.
Other than that, it is quite difficult to know what to target to identify because we are talking about lawyers’ practice. Again, there is nothing in the PACE codes. There is nothing written down for lawyers about how you are supposed to do it; these things just happen. I very much hope that the committee is able to do something that I certainly have not been able to: get people to hear those messages. Get them to the right people who actually have any power to change any of this. It is remarkably hard to find out who that is, and that tells us something, right? There is no ownership or responsibility for this process. Once we find that, these are the key messages: it is about standardisation, consistency and accuracy, but also neutrality, which is the other key point in terms of making sure that nobody is disadvantaged.
Lord Bradley: Very quickly on that point, we will be having witnesses from the police and the court system later on, but have you had any engagement with, for example, the College of Policing or the Association of Police and Crime Commissioners to raise the issues and attempt to get standardisation across the 43 police forces?
Dr Kate Haworth: I have tried. I know people in both those organisations. The trouble is that they do not have any responsibility for this. The transcripts are produced within the police just as an administrative task, and I have not found who would actually be able to instruct a police force to do it differently. The requests seem to come from the CPS, but again I do not know who would be in a position to actually make the changes, particularly about the process in court, because there is nobody who sets out rules; it is tradition and practice. So, I am very keen to hear the next evidence session that you have on this.
Lord Blencathra: Surely, we are going into a blind alley if we worry about getting consistency between police forces, checking the transcript and looking at AI to get a better version of it. If we present the real thing, we do not need all this palaver about making sure the transcript is better.
Dr Kate Haworth: You are quite right. Our first position, and possibly we have not made this clear enough, is that we want the audio, the original version—well, it is not the original version, but as close as we can get to it. That should be the starting position: start with the version with the least interference.
What we have not covered is that transcripts are just a practical necessity. People need something to skim-read, and you cannot skim-hear, so they serve a purpose. That is why we are not saying, idealistically, that everyone should always listen to the audio, because practical reality comes in. Transcripts are going to exist and are going to be used. That is why our recommendations come in. Let us make transcripts as good as we can but, importantly, make sure that people understand that they are not a substitute but an assistance. They are not a complete substitute for an audio or video recording.
The Chair: Thanks very much. That is helpful and has given us some ideas to put in our report. I thank Dr Howard, Dr Tompkinson and Professor Fraser for an informative and enjoyable session. I have certainly learned a lot and enjoyed the discussion. We are grateful to you for making time to come today and, particularly, to call in from Melbourne. We wish you well in your continuing work. I will close the meeting.
