Written evidence submitted by the Institute of Physics and Engineering in Medicine (IPEM)

(GAI0051)

About IPEM 

• IPEM is a professional association and Learned Society with around 4,700 members working in hospitals, academia and industry, who are medical physicists, clinical and biomedical engineers and technologists working with applications of physics and engineering applied to medicine. 
• Our mission is to constantly improve human health by the application of physics and engineering to the prevention, diagnosis and treatment of disease through research, innovation, education and clinical practice. 
• As a charity, IPEM’s aim is to promote for the public benefit the advancement of physics and engineering applied to medicine and to advance public education in the field. We do so by supporting and publishing research and supporting the dissemination of knowledge and innovation through project funding and scientific meetings; and by setting standards for education, training and continuing professional development for healthcare scientists and clinical engineers. 
• In compiling this response, IPEM consulted with members of its Clinical and Scientific Computing Special Interest Group and the Science, Technology, Engineering, Research and Innovation Council.

IPEM’s response 

• How effective is current governance of AI in the UK?
• What are the current strengths and weaknesses of current arrangements, including for research?

1. In UK healthcare, governance related to AI is largely achieved via pre-existing regulatory frameworks (for example via medical device legislation and associated standards). We believe that the current lack of AI-specific standards has caused uncertainty over the appropriate steps required to ensure safe, effective development and clinical use of AI. For example, clinical and scientific computing staff may be called upon to evaluate commercial AI auto-contouring software, but there is little guidance on how this should be done.

2. It is noted, however, that the MHRA is embarking on the Software and AI as a Medical Device Change Programme, which will help to provide additional clarity to developers and consumers. New standards such as BS30440 and BS AAMI 34971 are welcome additions to the available literature.

3. We suggest that existing governance arrangements for research are generally adequate for AI-focused studies. However, there remains problems with access to clinical data and inconsistency of interpretation of information governance guidance between trusts.

• What measures could make the use of AI more transparent and explainable to the public?

4. We suggest that healthcare workers are best placed to make the use of AI more transparent and explainable to the public. Improving their understanding of AI should therefore be a priority. If a clinician does not understand an AI tool and what it is and isn’t capable of, then trust (from both a patient and healthcare worker perspective) is likely to be low. Alternatively, if a clinician can use an AI tool with confidence, then the software becomes an asset and can be better explained to the patient and public.

• How should decisions involving AI be reviewed and scrutinised in both public and private sectors?

5. We believe that in healthcare, decisions involving the deployment and routine use of AI should be made jointly with healthcare staff (clinicians, nurses, clinical scientists, engineers and others). The use of AI should be audited, with discrepancies between AI / clinician being analysed and subject to a second review if necessary. Longer term post-market surveillance should be used to monitor how AI performs, not just in terms of raw software performance but also how it is used and how it continues to affect the patient pathway. Such an auditable approach could also be used to plot trends and catch any drifts in key indicators.

• Are current options for challenging the use of AI adequate and, if not, how can they be improved?

6. Under existing arrangements a patient can refuse any form of treatment, which could include AI-informed care. However, it may not be obvious to a patient that AI was involved in their treatment, and there is no obligation for a NHS organisation to advertise this. Being open and transparent about decision making, with an emphasis on clinicians taking overall responsibility for patient care, is likely to be the best way to reassure patients and the public. If the education of front-line clinicians can be improved then they will have the confidence to discuss any potential issues that the patient has.

• How should the use of AI be regulated, and which body or bodies should provide regulatory oversight?

7. We believe that in healthcare, AI regulation is likely best achieved under the MHRA and CQC, bodies which already regulate many aspects of AI development and deployment under pre-existing frameworks. In a similar way to other healthcare technologies, AI relies on healthcare staff and interconnected services to perform adequately. Therefore, regulation should ideally take a systems approach to verify that the wider workflow involving AI is safe, inclusive, effective and robust. To ensure reliability and validity it should ideally be a requirement that assessments of AI performance are based on data that is representative of the wider clinical population, including common as well as uncommon patient and disease characteristics.

• To what extent is the legal framework for the use of AI, especially in making decisions, fit for purpose?

8. We believe that the legal framework around healthcare decisions involving AI is not clear.  It is largely untested. For instance, if a clinician makes a decision that disagrees with AI it is not clear how this would impact a liability case. We suggest that it would be beneficial for relevant professional bodies to produce guidance on how AI could and should be used to inform decisions on patient care.

• Is more legislation or better guidance required?

9. We suggest that better guidance, for example in the form of AI-specific development or deployment standards, would be more helpful than more legislation.

• What lessons, if any, can the UK learn from other countries on AI governance?

10. Other healthcare regulatory agencies around the world, such as the US Food and Drug Administration (FDA), are addressing similar concerns related to AI governance. The MHRA are clearly aware of such international work (for example, collaborating on the good machine learning practice document). However, there are other international bodies not focused on healthcare that have created informative documents. For example, the US government recently produced the Blueprint for an AI Bill of Rights document. This lists five principles and associated practices, all of which are relevant to health and whose requirements extend beyond existing legal obligations for medical devices. The document mandates, for instance, that “Designers, developers, and deployers of automated systems should take proactive and continuous measures to protect individuals and communities from algorithmic discrimination and to use and design systems in an equitable way.”

(November 2022)