Written evidence submitted by the Home Office

Letter from Mike Wells CBE, Chief Operating Officer, UK Visas and Immigration
to the Chair of the Committee, 17 August 2016

English Language Testing

Thank you for your letter of 27 July to the Permanent Secretary asking several further questions concerning ETS.

As the lead Home Office Official with responsibility for ETS, your letter was forwarded to me for a response. Consequently, please find answers to your questions below.

I apologise for the delay in providing these replies.

82.            The Committee has heard reports that at some test centres the microphones were not connected to the servers and an applicant’s test was not recorded but substituted with another. Is this possible and to what extent has this part of the process been subject to investigation?

The Home Office would welcome sight of any evidence that the Committee has seen on this which would be fully investigated.

The Home Office Immigration Enforcement criminal investigations into the fraud uncovered evidence of impersonation i.e. the use of proxy test takers, the type of fraud shown in the Panorama documentary. Whilst there have been suggestions of other methods of fraud, no evidence has been uncovered during the course of the criminal investigation to support these scenarios.

The expert evidence report of Kroll Ontrack found no evidence in the information made available to them to suggest the substitution of recordings in this way.

83.        What checks have been done to determine that the time stamps of the recordings match the time that applicant’s sat the test? Has any other metadata been analysed?

ETS informed the Home Office that the metadata does not contain any information regarding when they were recorded. The issue of the timings of a speaking test has, to our knowledge, only been raised in one case before the Upper Tribunal and on that occasion, ETS was able to provide precise timings for the end of the test and uploading of the data. ETS has also provided considerable evidence to the Home Office and in support of matters before both the First-Tier and Upper Tribunals as to the processes for ensuring that voices match individual records.

84.              Where were the recordings stored, where are they held now? Were they all submitted in real time to ETS?

Our understanding is that once recordings are transmitted back to ETS in America, they are stored securely on servers in its data centres. These servers are accessible only to authorised ETS personnel and the files are typically stored for 999 days. The recordings were submitted back to ETS at the end of the speaking tests.

85.        What checks have been made of the CCTV footage at the test centres under investigation? Where are the CCTV recordings held?

No CCTV footage was recovered by the Home Office Crime and Financial Investigation teams from the test centres under investigation.

Enquiries indicate that CCTV had been installed at some of the colleges which were subject of the criminal investigations. However, at the time of the arrest and search phase, the colleges had ceased to operate and none was recoverable.

Other colleges did have CCTV installed but it did not provide coverage of the relevant areas and/or periods of criminal activity.

86.        What has Kroll Ontrack been asked to investigate, when were they appointed and at what cost?

The Home Office instructed Kroll Ontrack in early April 2016 to provide an independent expert view on whether it is possible that:

• A recording of a genuine test taker’s voice might be re-used by a fraudulent test centre and submitted to ETS as the TOEIC speaking test of other test- takers; and
• A genuine test-taker could take the ETS TOEIC test honestly and yet it could transpire, without that test-takers knowledge, that ETS’s recordings of that person’s test have someone else’s voice.

The key finding from this report is that while there are potentially a variety of frauds which might theoretically have been perpetrated, the only actual evidence is of impersonation and the use of proxy test-takers. This is line with what the Home Office has found through criminal investigations and the type of fraud seen on the Panorama programme.

The Kroll Ontrack report supports the fact that the test centre staff and test candidates were unable to replace audio recordings within the ETS TOEIC user interface and found no evidence of a genuine candidates recording being swapped outside of the TOIEC user interface.

The costs to date of instructing Kroll (including VAT) are £16,920.

87.              Can the Home Office forward a copy of the Kroll Ontrack report and contact details of the person at Kroll Ontrack leading on this work?

A copy of the Kroll Ontrack report is attached at Annex A.

The report has been produced by Mr Richard Heighway. 

His contact details are:

Kroll Ontrack Legal Technologies Ltd

25 Farringdon Street

London EC4A 4AB

88.        a) Have any of the individuals arrested and/or interviewed in the criminal investigation been asked whether genuine applicants could have been caught up in the fraud without being aware of it?

b)  Who is leading the investigation?

c)  How many people are working on it? and

d)  How many are former police officers?

Several test takers have admitted involvement in criminal abuse of the TOEIC process. Many others have made no comment following arrest/ interview. However, during the course of such interviews; it has not been alleged/suggested that genuine applicants could have been caught up in the fraud without being aware of it.

The Senior Responsible Officer is Mr Steve Dann, Director of Immigration Enforcement Criminal & Financial Investigations (CFI). Assistant Director Chris Foster is the Senior Investigating Officer (SIO) on the overall Inquiry (named Project FACADE) concerning TOEIC abuse. Each of the 14 ‘live’ Project FACADE investigations has a nominated SIO from within regional HO CFI offices.

Presently, there are 23 officers currently employed across 8 regional CFI offices. The investigations are supported by numerous other officers during surges of operational activity.

There are four serving police officers (on secondment to CFI), and, seven former police officers engaged on the criminal investigations.

89.        Can the Home Office provide a breakdown of all trials that have taken place, are scheduled to take place or are pending a decision from the CPS? The information should include whether any of the trials are subject of reporting restrictions and if so, a justification for this.

This is detailed at Annex B - Project Facade Investigations & Trials.

90.        Can the Home Office provide a witness statement from ETS?

Should the Home Affairs Select Committee require a witness statement from ETS they should contact Jones Day Solicitors who are the legal representatives for ETS and ETS Global BV.

It should be noted that ETS has already provided witness statements from their staff for the criminal proceedings which were also produced at a recent Upper Tribunal hearing in response to an order from the court. We understand that an ETS employee has been directed to attend a forthcoming criminal trial in August.

91.        Have any other components of the test (reading, writing, listening) been investigated for deception?

ETS’ analysis and UKVI’s response have focused on the speaking elements of testing because of the evidence of malpractice in the testing process. The investigations undertaken by the Home Office Crime and Financial Investigation team have examined all aspects of the testing process.

92.        According to q 61 in my letter addressed to James Brokenshire on 6th June, the Home Office identified malpractice or anomalous patterns of results in 5 test centres and their licenses were terminated. Can the Home Office name the 5 test centres that have been closed down?

The five centres that were terminated as test centres by the licensee due to identified malpractice or anomalous patterns of results were:

• Flexilearning Ltd
• Impel College of London Ltd
• London Business College
• Quality Education Centre Ltd
• Regal International College

93.        What was the minimum number of invalid/questionable tests that were taken into consideration before a particular Test centre was shut down

The Home Office does not have the authority to shut down test centres but, where malpractice was identified, the Home Office could stop accepting certificates from that centre for immigration purposes

There was no defined minimum of invalid or questionable tests identified before action was taken; each identified breach or concern was considered on its individual merits with all abuse taken seriously.

94.        a) Can the Home Office provide the names and job descriptions (including relevant qualifications) of all those who were on the delegation that visited ETS in the USA?

b)  Did the delegation include an expert in voice recognition?

c)  How many voice recordings did the delegation listen to?

The delegation which visited ETS was led by Paul Morrison, the then Director of In- Country Migration Casework in the Home Office.

The delegation also included Peter Millington, Tier 4 Sponsorship Compliance Manager, Andrew Williams, a lawyer from the Treasury Solicitors Department, and Kate Gillibrand, an analyst from Immigration Enforcement.

It is for English language test examination bodies to decide whether test certificates should be awarded or withdrawn, not the Home Office. The purpose of the visit was to understand the processes operated by ETS and why ETS were satisfied that they were able to identify to the correct standard where fraud had occurred. An expert in voice recognition was not in the delegation as this was not the purpose.

During the course of the visit representatives of ETS explained and demonstrated the background to the approach/process they used to identify invalid tests including the opportunity to listen to voice recordings.

The delegation listened to multiple randomly selected voice recordings and not just examples prepared by ETS for the purpose of the visit (ETS’ analysts randomly opened and played matched files at Paul Morrison’s request, rather than just playing ones prepared ahead).

The exact number of recordings listened to was not recorded.

As the Committee knows since this visit the Home Office commissioned Professor J P French Chairman of JP French Associates Forensic Speech & Acoustics Laboratory and Prof of Forensic Speech Science at York University to carry out a further independent expert review.  He has confirmed that the ETS analysis process was robust and suitably cautious being much more likely to produce false negatives than false positives.

95.        What is the level of training and expertise of the people who are responsible for verifying the voice recordings at ETS?

Verification was undertaken by two analysts working entirely separately, one of whom would always be an Office of Testing Integrity (OTI) analyst experienced in listening and comparing voice samples. Each member of staff received mandatory training in voice recognition analysis and was paired initially, for mentoring purposes, with an experienced OTI analyst.

The two analysts only became involved when ETS’ Automated Speech Recognition (ASR) software had identified a match between the voice on two or more tests taken in different names. When that happened it was therefore a three stage process and a test result was declared invalid only when all three stages concurred on a match.

96.        We have learnt that, as a matter of routine, if an allegation of impropriety is made at an ETS run test centre, ETS moves to invalidate all tests taken at that centre for a period around the allegation and invites all those involved to re-sit the test. Is that correct?

Practice undertaken more generally by ETS across the wide variety of different types of tests it sets is a matter for ETS. ETS has not supplied testing in the UK for the Home Office since early 2014.

The test analysis results which ETS supplied to the Home Office distinguished between invalid and questionable. We have explained in answer to Question 99 the distinction between the two.

In the analysis supplied to the Home Office, ETS distinguished between invalid and questionable results. Results were declared invalid only when all three stages of analysis concurred that there was a voice match between two or more tests.

We understand that in normal practice where ETS assess a result as questionable they would offer a re-test. As explained in Question 104 nobody about whom we were notified of a questionable result was refused immigration leave on that basis without first being given the opportunity to resit a test with another provider and attend an interview with a Home Office official.

97.        a) When signing the contract with ETS what was the Home office stipulation about dealing with fraud?

b)  Can the Home Office provide the relevant extract from the License Agreement?

In 2011 ETS tendered for a license to provide a recognised qualification for UK visa applicants. There was no contract signed with any of the test providers as a result of the 2011 exercise. Instead, ETS was granted a license by the Home Office to provide the TOEIC and TOEFL tests to be used for visa purposes.

In the requirements for the tendering exercise there were specification requirements that outlined what was expected.

The Licence Agreement outlined the following:

6.1.11  during the Term, the Licensee shall continue to meet all the criteria that it was assessed against to obtain its Licence.

6.1.12  the Licensee shall continue to meet the security requirements set out in the Specification throughout the Term.

A copy of the licence agreement was annexed with the response letter of 6 May to the committee from the then Immigration Minister.

An extract of the specification setting out the security requirements is included at Annex C.

98.        Can the Home Office confirm ETS policy in dealing with allegations of fraud and whether ETS has invalidated test results on the basis set out above?

As stated in the Home Office response of June 2016, the Home Office asked ETS America to review all test results with a view to identifying those which had been obtained by cheating following the suspension of their provision of testing in the UK in February 2014.

ETS in America were able in stages to review test results taken dating back to 2011 and over a number of months they provided batches of spreadsheets that had details of individual tests taken. Each spreadsheet covered a number of test centres and tests taken over a period of time. The data was split across several tabs generally labelled ‘cancelled,’ ‘would offer retest’ and ‘released.’ These became known as Invalid, Questionable, and Released respectively.

ETS in America have analysed the results of the TOEIC tests taken at UK test centres. It was and remains the case that English language test providers are responsible for determining whether a candidate has properly passed their test.

A detailed explanation of the process followed by ETS in declaring a test as invalid is set out in the response to Question 99.

99.        Can the Home Office confirm that each and every one of the 56,000 invalid and questionable tests been subject to the full analysis process of voice recognition software and two human analysts?

Our understanding from the information provided by ETS is that all tests were subject to the automatic speech recognition (ASR) analysis to identify where a proxy test taker was used - in other words to identify whether a voice was unique or whether it was matched to another test. Tests were analysed by test centre in batches of 300- 400 tests from multiple days around the same date depending on the size of the test centre. The audio files were processed by the voice biometric engine comparing each test to all other tests in that batch and creating a flag of any results that were a match. Each flagged match was additionally subjected to a further human verification process. Only where a test was matched to one or more tests by the ASR was the result reviewed by the two independent analysts. A test could not be declared as invalid without the ASR system identifying it as a match and then, crucially, both of the human analysts independently reaching the same conclusion that the voices matched.  In effect, three systems were required to be in agreement.

If no match was made by the ASR, then it would be possible for a test to be declared questionable because of irregularities at a test centre without undergoing the full analysis set out above

As Prof French says in his report the ETS analysis process was robust and suitably cautious being much more likely to produce false negatives than false positives.

100.   What quality assurance has the Home Office taken outside of the single day’s visit to ETS?

The Home Office has consistently ensured that appropriate steps were taken to understand the processes operated by ETS and why ETS were satisfied that they were able to identify to the correct standard where fraud had occurred.

As the Committee knows this has been supported by the findings of the independent report by Professor French, Chairman of JP French Associates and Professor of Forensic Speech Science at the University of York. Moreover the Courts have to date consistently found that the Home Office evidence is sufficient to establish a presumption of fraud.

As we have previously informed the Committee, the Home Office has also conducted extensive criminal investigations. These have resulted in 115 organisers (college directors, test centre administrators, agents and proxy test takers) having been arrested and or interviewed. Cash in the region of £100,000 and suspected to be the proceeds of the associated criminality has been detained.

ETS provided the Home Office with three key witness statements and hundreds of voice recordings to support these criminal investigations. Nine people have already been convicted and a further 17 defendants are to appear in several criminal trials this year.

101. Professor French has concluded that the number of false positives emanating from the ETS analysis would be very small. What allowance has the Home Office made for the very small number of false positives?

Professor French did indeed conclude that the number of false positives would be very small. In fact his conclusions went further in stating that false negatives were much more likely than false positives. The Home Office believes this appropriately gave the balance of any doubt to the test applicant. Moreover Professor French concluded that using the 2% error rate which ETS established for the automatic speaker recognition system when they conducted a pilot of their process in similar TOEFL tests if it were to apply to the TOEIC tests, he would estimate the rate of false positives to be substantially less than 1% after the process of assessment by trained listeners had been applied. Put another way this means that substantially more than 99% of positives are such.

It is worth noting that nobody with a questionable test certificate had action taken against them for that reason without first being given the chance to re-sit a test and attend an interview. Where we have either refused or revoked somebody’s leave because of an invalid test certificate it is open to them to seek recourse through either administrative review (where applicable), a statutory appeal where this applies or through judicial review.

As the Committee is aware the Courts have to date consistently found that our standard evidence is sufficient to satisfy the presumption of fraud. The most recent key ruling in this regard was from the Court of Appeal on 29 June in the case of Shezhad & Chowdhury. It is then up to the appellant to establish an innocent explanation for their involvement.

102.   How many certificates in total were granted by ETS in each of the years that they were licensed?

And

103.   a) How many tests were declared valid in 2011?

b)  How many tests were declared valid in 2012?

c)  How many tests were declared valid in 2013?

d)  How many tests were declared valid in 2014?

This table is a summary of the data we received from ETS. It has been classified by year into invalid, questionable and released certificates for both tests taken in the UK and those overseas.

104.   How many individuals whose test results were neither “invalid” or “questionable” have been considered illegitimate by the Home Office on the sole ground that they had taken their test at a centre where large numbers of invalid and questionable results had been diagnosed?

None. No action was taken against an individual on the basis of deception of English language requirement unless we had been notified by ETS that their test result was invalid. Nobody about whom we were notified of a questionable test certificate had action taken against them without first being given the opportunity to re-sit a test with another provider and then attend an interview with a Home Office official.

105.   The Home Office relies on a spreadsheet to support its case against individuals:

a)  Who prepared that spreadsheet?

b)  Who has verified the spreadsheet?

c)  Can the Home Office provide the Committee with the spreadsheet?

ETS provided data on the results of their analysis. This was provided in the form of Excel spreadsheets. There were several spreadsheets received over a period of time as the analysis progressed. By means of the Home Office’s data matching systems, the entries on the ETS spreadsheet were matched against immigration records. The Home Office’s Performance Reporting and Analysis Unit (PRAU), then collated the individual spreadsheets with the matched Home Office records into a master spreadsheet. This spreadsheet was used as a reference by caseworkers when considering applications where an individual had been linked to ETS. No action was taken until the case worker was satisfied that an individual had relied upon an invalid TOEIC test in support of an application to the Home Office for leave.

The spreadsheets themselves contain personal information and so it is not possible to share them with the committee.

106.   How many of the 56,000 individuals have been given their voice recordings in order for them to verify that it is their voice on the recordings?

Responsibility for validating test results rests with the test providers. If an individual wishes to obtain their voice recording then they should approach ETS. Our understanding is that wherever the tribunal has asked for the recording to be disclosed (which we believe to be on six occasions), ETS have complied with this request.

As set out previously, however, the Courts have consistently found that our standard evidence is sufficient to satisfy the presumption of fraud and therefore that it is not necessary to routinely obtain individual voice recordings to establish this. Indeed it is also worth noting that when voice recordings have been provided and it has then been accepted that the voice is not that of the alleged test taker this has not settled matters but led to other claims about the evidence. This is what the Kroll Ontrak report seeks to address.

107.   What allowance has been made for twins/close relatives?

The ETS analysis tests batches of audio records from a single centre for a period of a few days at a time. ETS has not informed us of any specific provision made in their analysis for tests taken at the same test centre at or about the same time by twins or close relatives. However, the Home Office is satisfied that the ETS process of establishing cheating is suitably cautious. As stated in response to Question 98, ETS set “conservative” thresholds for their automated process of testing voice samples.

Any errors were thereby more likely to be false negatives than false positives.

108. What evidence did the Home Office provide to sponsors when it requested that sponsorship be withdrawn from students named on the ETS list?

Sponsors were informed that ETS had withdrawn a significant, high volume of Secure English Language Test (SELT) results from students who had taken the Test of English for International Communication (TOEIC) tests at some of their approved centres. Sponsors were further advised that some of the certificates had been obtained fraudulently and that students had received certificates for tests that they did not take.

Communications to sponsors provided the number and names of their students who had TOEIC certificates withdrawn by ETS.

A key principle of sponsorship is that those who benefit from immigration play their part in tackling abuse. All sponsors sign up to this obligation and therefore must fulfil their own responsibilities to maintain their licence. Sponsors were therefore requested to undertake a comprehensive review of any sponsored students identified via the ETS analysis process elsewhere described.

The Home Office took action directly itself to curtail the leave of those who had obtained leave using a test certificate subsequently notified by ETS as invalid. The Home Office used information on sponsors of significant numbers of those so notified by ETS to target compliance inspections of sponsors.

The Independent Chief Inspector of Borders and Immigration reviewed this in a report published in July 2015 of his inspection into the effectiveness of the monitoring of Tier 4 sponsor licences. This inspection examined how effectively the Home Office monitored Tier 4 sponsors’ compliance with their sponsor duties, and also looked at the Home Office’s handling of its investigations into sponsors linked to ETS (known as Operation Firewall).

His report found that the letters the Home Office sent to suspended sponsors “clearly explained the reasons for these actions”, he went on to find that the Home Office operation “was a success in terms of the way UKVI swiftly mobilised to deal with the problems with ETS identified [and that]…the outcomes… improved the effectiveness of the Tier 4 compliance regime”.

As well as achieving the removal of more than 50 non-compliant sponsors from the register, the operation had led to the creation of a dedicated Tier 4 compliance network to deal with the more complex cases; and also to improvements in the scrutiny of new licence applications and of secure English Language Test (SELT) providers.

109. Can the Home Office provide a breakdown of the Nationalities of the 56,000 and what test category they fall under?

This can be found at Annex D – Tests by Nationality.

110. We have received evidence from individuals who have successfully appealed against an allegation of deception but found a subsequent application for a visa to be rejected because of that allegation. If an individual successfully appeals an ETS allegation why is the allegation not removed from their file?

Each case is considered on its own merits. There may be scenarios where an appeal against an allegation of deception has been allowed by the First Tier Judge.

However, in instances where the Home Office decided to appeal that decision to the Upper Tribunal or above, it will continue to consider the TOEIC certificate as invalid pending the outcome of the appeal.

An application for a visa may also be refused where the individual fails to meet any other relevant entry clearance requirements.

111.   a) How much is the Home Office spending in legal costs for cases that are in court relating to English language testing?

b) Has a date been set for the appeal in the Qadir case and the Upper Tribunal hearing in the Shehzad/Chowdhury case?

As we have previously explained to the Committee, a Gold Command was established in 2014 shortly after the discovery of the fraud to manage the operational response to the ETS situation. This Gold Command existed until May 2015 when it was disbanded and its functions were subsumed into business as usual processes. It is not possible to differentiate ETS specific litigation costs from other operational costs because Home Office data systems are unable to disaggregate the exact reasons why people about whom we were notified of an invalid or questionable ETS certificate may be involved in litigation. It is quite possible that such litigation may be unrelated to the matter of English language testing.

The Court of Appeal is expected to hear the Home Office appeal against the determination of Qadir and SM On either the 26th or 27th October 2016.

In respect of Mr Shehzad, because both the First Tier Tribunal and the Upper Tribunal had erred in their conclusions on jurisdiction, the appeal of Mr Shehzad is at an end as we are not aware that Mr Shehzad has taken any further action in respect of this matter.