Black box no more: a scoping review of AI governance frameworks to guide procurement and adoption of AI in medical imaging and radiotherapy in the UK

Technological advancements in computer science have started to bring artificial intelligence (AI) from the bench closer to the bedside. While there is still lots to do and improve, AI models in medical imaging and radiotherapy are rapidly being developed and increasingly deployed in clinical practice. At the same time, AI governance frameworks are still under development. Clinical practitioners involved with procuring, deploying, and adopting AI tools in the UK should be well-informed about these AI governance frameworks. This scoping review aimed to map out available literature on AI governance in the UK, focusing on medical imaging and radiotherapy. Searches were performed on Google Scholar, Pubmed, and the Cochrane Library, between June and July 2022. Of 4225 initially identified sources, 35 were finally included in this review. A comprehensive conceptual AI governance framework was proposed, guided by the need for rigorous AI validation and evaluation procedures, the accreditation rules and standards, and the fundamental ethical principles of AI. Fairness, transparency, trustworthiness, and explainability should be drivers of all AI models deployed in clinical practice. Appropriate staff education is also mandatory to ensure AI’s safe and responsible use. Multidisciplinary teams under robust leadership will facilitate AI adoption, and it is crucial to involve patients, the public, and practitioners in decision-making. Collaborative research should be encouraged to enhance and promote innovation, while caution should be paid to the ongoing auditing of AI tools to ensure safety and clinical effectiveness.


INTRODUCTION
Artificial intelligence (AI) is increasingly employed in healthcare. 1 Recent technology and neuroscience breakthroughs brought AI from the lab to the clinic. 2[6][7][8] While AI-driven applications for use in clinical practice are increasing, there is, in parallel exponential interest and discussion on the need for rigorous AI governance frameworks. 9I governance may entail processes related to the ethical use and deployment of AI tools, regulation and accreditation of AI models, liability, accountability, data protection processes, and education, among others.
In medical imaging, the implementation of robust AI governance frameworks is required for the safe adoption of AI in clinical practice.
Although AI is a ubiquitous term, AI governance remains loosely defined and largely underdeveloped, with no consensus on what AI governance might entail. 10Recent research has shown important variation among organisations and countries regarding governance.A lack of standardisation could impede AI adoption, create market disparities, and compromise safety. 11rocesses like procurement, validation and evaluation, monitoring and decommissioning, as part of the AI product lifecycle, are all impacted by the lack of standardised AI governance frameworks.Hence, there is an urgent need to propose a robust, unified governance framework to enhance the trustworthiness and transparency of AI systems and mitigate any potential risks associated with the implementation of AI-enabled solutions. 12,13 medical imaging departments, healthcare professionals, including radiographers, radiologists, and medical physicists, are often responsible for procuring medical imaging equipment.Multidisciplinary teams also ensure the safe running of the equipment by implementing robust quality assurance programmes and escalating concerns, as required.[20][21][22] Scoping reviews are ideal for exploring emerging literature on fast-developing topics and identifying knowledge gaps. 23This scoping review, part of a more comprehensive research project exploring the notion of AI governance in medical imaging and radiotherapy, aims to map out all currently available literature (peer-reviewed and grey literature) in these fields and propose a comprehensive AI governance framework.
The research question that guides this scoping review is "What might be relevant to an AI governance framework in medical imaging and radiotherapy in the UK?".This question was generated in line with the Population, Concept, Context framework for scoping reviews. 24

Review protocol
This article is structured in line with the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) checklist. 25 explicit review protocol was followed for inclusion and exclusion criteria, data extraction methods and objectives. 24Ethical approval was obtained from City, University of London School of Health and Psychological Sciences Research Ethics Committee (ref: ETH2122-1015).

Eligibility criteria
Table 1 below demonstrates the eligibility criteria applied to this scoping review.

Information sources
The following databases were searched: a) Google Scholar, b) PubMed, and c) The Cochrane Library.Searches were initiated on 30 May 2022, and the last search was performed on 17 June 2022.A new search was also performed on 15 October 2022, to ensure we captured any latest papers.

Search
A consistent methodology was applied for all database searches to enhance the reproducibility of this scoping review. 26All searches were performed using explicit, pre-defined keywords related to the topic under exploration.Appropriate search strings were developed using Boolean operators '' AND'' and ''OR'' to narrow down the results.A completely worked example of the search strategy for one database is provided as Supplementary Material.The pearl growing search technique was also applied to identify relevant sources of evidence in the reference lists of already obtained studies.

Selection of sources of evidence
A researcher (NS) initially screened the articles at the level of titles and abstracts to identify relevant/non-relevant results.Non-relevant articles were excluded at this stage, based on the content provided in titles/abstracts, and only relevant articles were assessed for eligibility based on inclusion and exclusion criteria (full-text evaluation).All results were saved on the Zotero reference manager, v. 6.0.13 (Corporation for Digital Scholarship, Virginia), for further evaluation, and duplicates were automatically removed.All relevant studies were read thoroughly and evaluated against the eligibility criteria.A senior researcher (CM) then advised on the final study selection, and that list was reviewed by the research team and finalised based on consensus reading amongst the researchers.

Data charting process
To extract all meaningful data from the eligible studies, a datacharting form was used to allow a visual map of the included studies and enable correlations and convergence of ideas and topics. 27This charting table (Table 2) allowed for a standardised method of data charting, and any disagreements were resolved until a consensus was reached.

Data items
The team extracted essential study characteristics related to author(s), year of publication, country of origin, study population, and sample size.At the same time, any critical data relevant to the scoping review question was also extracted.Studies published only in the English language.
Full-text articles only.
Both peer-reviewed studies and grey literature (white papers, guidelines, guidance, standards, and regulations) will be included.
All study designs were eligible for this review.• Fairness.
• Respect for autonomy laws (UK and EU).

Data analysis and synthesis of results
All included studies were coded and then grouped according to their content (e.g.ethics, regulation, validation etc.).Content analysis was performed to identify concepts, categories, and themes 28 of AI governance included in the eligible papers.[31]

Selection of sources of evidence
In total, 35 articles were included in this scoping review.The following diagram (Figure 1) demonstrates the details related to the search process, screening of articles, and final selection.
Out of 35 sources of evidence included in this scoping review, 28 articles were identified as reviews, while 7 related to 'grey literature' (government guidance and guidelines issued by regulatory, advisory, or professional bodies).In terms of geographical distribution, the obtained studies were from the UK (n = 8), USA (n = 6), EU (n = 5), Canada (n = 1), Australia (n = 1), India (n = 1), while a further 13 of them were identified as binational/multinational.With regard to year of publication, many of them were published in 2021 (n = 15), followed by those in 2020 (n = 8), 2019 (n = 6), 2018 (n = 3) and 2022 (n = 3).Figure 2 provides information on the types of the obtained sources of evidence.
The main outcome of this study was to construct an AI governance framework, based on published evidence.

Suggested AI governance framework
The main findings were classified into concepts, then grouped into categories and further synthesised into themes in line with a content analysis approach. 32The results of this scoping review enabled the researchers to propose a conceptual AI governance framework based on the most widely discussed topics and principles of AI governance.The following figure (Figure 3) demonstrates the seven pillars of AI governance, as identified in the literature and synthesised here into themes.
The above AI governance framework includes some fundamental principles of AI governance, such as validation, evaluation, and auditing of AI systems, while also highlighting the importance of research and innovation, appropriate staff education, and effective leadership to ensure the safe and successful deployment of AI in clinical practice.Some of the most important categories, allowing finer detail of the suggested AI governance framework, as identified by this scoping review, are summarised in the following figure (Figure 4) and in the discussion.

Validation and evaluation
7][38] Evidence suggests the need for internal and external validation using  unseen data.Validation has also been used in other contexts, e.g. to assess the generalisability/interoperability of the model. 39,40andardised, rigorous validation protocols and curated databases have been developed to facilitate this process. 41Assessment of the clinical effectiveness of an AI model examines both its ability to generate the intended output effectively and to have a meaningful clinical impact. 34,42Hence, a 3-phase validation process has been proposed, consisting of validation against the test set of the data set (internal validation), validation against unseen data (external validation), and validation against diverse datasets from multiple centres. 35ile this might sound optimal for diagnostic accuracy and clinical efficiency, in reality, data privacy and security when it comes to training and testing AI tools often make large, diverse, multisite data sets inaccessible to developers. 43 model interoperability enables seamless data flow across different imaging centres, systems, or geographical areas. 42,44nteroperability in hardware, software, and data use can be achieved if large, representative training data sets are employed. 45 common interoperability software framework has been suggested in clinical imaging. 41timation of projected additional costs compared to standard practice 46 and their impact on the use of available resources 39 are also central to a comprehensive evaluation process.

Regulations, legislation, and ethics
The papers reinforce the need for rigorous regulatory frameworks to safely use any AI-enabled medical devices; however, stringent standards may come at the cost of limiting innovation. 44Regulations must be applied to data protection, safety, and ethical use of AI models. 47However, regulatory frameworks have not yet been standardised many countries 48 due to technological advancements. 49 the EU, all medical devices must be classified according to their risk and conform to the required CE regulations. 39In the UK, all medical devices should be registered with the MHRA and undergo a UK Conformity Assessment (UKCA) from July 2024 due to the UK's exit from the EU. 50CE-marked devices will be accepted in the UK until that date.Devices CE-marked under the Medical Devices Directive (before 2021) will have a further 3 years to gain a UKCA certification, and devices CE-marked under the Medical Devices Regulation will have a further 5 years.In addition, all medical devices should also comply with the General Data Protection Regulation and, in the UK, the NHS Digital Technologies Assessment Criteria framework. 20,42oreover, all AI models should be classified according to their intended purpose, and this classification should be based on their potential risk to the system and service users. 46[53][54][55] Informed consent is another vital element of AI governance relating to the right of every patient/data owner to be informed about when their data will be used and for what purpose. 56onsent reinforces the right to human autonomy. 53,57Also, informed consent from data owners must be a dynamic process, and data owners should have the right to access their data and decide how it can be accessed by third parties. 51 addition, all organisations must take appropriate steps to ensure data safety against potential adversarial attacks since data collection and usage raises concerns regarding cybersecurity. 51afety might also be compromised by the 'algorithm update problem' or 'concept drift' , which can impact the algorithm's performance over time as new data comes in. 58Algorithms could be locked to remain static, limiting usability; a solution to this challenge could be to assess the algorithms for any changes periodically. 44,58Ongoing data safety can be challenging and costly to maintain but necessary.
Serious concerns have been raised regarding potential patient harm from poor use or insufficient validation of AI models. 59iability of health practitioners is associated mainly with medical malpractice and negligence, while developers' liability would most likely fall under product design liability. 59Clinical practitioners may be liable for implementing inappropriate or nonvalidated AI tools in clinical practice or for failing to substantiate their recommendations.
All AI models must address the principle of fairness 36,39,51 and avoidance of harm by minimising bias. 45For AI models to be fair, it is essential to ensure equity of benefits and costs and BJR A review to guide AI adoption in medical imaging/radiotherapy eliminate discrimination, stigmatisation, and unfair bias. 36Algorithmic biases are often due to non-diverse training data sets or testing only on specific population groups. 44,60AI models are prone to discrimination biases, which have been confirmed to be an important ethical issue. 53 addition, AI models should be transparent to be fair, inclusive, and easy to evaluate.Transparency requires AI models to be always available to interrogation. 47,61Also, transparent AI solutions will help build trust between patients and healthcare providers and between developers and clinical practitioners as end users. 52Another way to increase the transparency of AI models is to establish good traceability.This principle refers to the standardised documentation of all development processes, including data collection, labelling, devices used, and annotation tools. 44rthermore, AI models must be explainable so that the patients and the trained staff can understand the reasoning behind their decision-making process. 39This could enhance trust between end users and AI technologies.Explainability is closely associated with transparency. 38,58 contrast, interpretability, often confused with explainability, refers to the ability of a model to make correct associations between cause and effect.Interpretability increases when AI

BJR
Stogiannos et al models are explainable, although these models may exhibit reduced performance due to becoming more prone to external manipulation. 48 auditing and quality assurance All organisations must develop ongoing procedures to test the AI model's performance throughout its life cycle.20 Real-world performance monitoring, in the form of pre-and post-market surveillance, has been suggested to assess any deviation in the model's performance over time.40 These procedures should focus on load tests, safety, bias testing 20 and clinical and technical performance over time.Regular audits have been recommended to test the clinical safety of these models, particularly after model updates.Reporting of these audits should include accuracy, model biases, and clinical outcomes.48 All vendors should outline appropriate plans to assess their model's performance drifts, automatically install any necessary updates and mitigate any risks from these updates.39

AI research and innovation
Research is fundamental for improving clinical practice, patient outcomes, staff well-being and optimising workflows. 56However, there is still a lot to be done to ensure prospective studies are prioritised and that links between industry and academia are strengthened. 62With more academic-industry partnerships, it is vital to ensure the impartiality of researchers; researcher and clinician internships in AI startups will be central to supporting ethical, person-centred research. 62ospective research studies are essential to assess and document the real added value of AI in healthcare. 46There is a need to evaluate AI tools after implementation, and there are already some checklists to assess research quality and risk of bias. 468 Appropriate staff training on AI technologies enables them to build confidence in effectively and safely using these AI tools 20 for improved workflows, better patient outcomes and higher job satisfaction.In addition, it was found that appropriate training/education provided to healthcare professionals will also increase trustworthiness. 45blic, patient, and practitioner involvement (PPI) Many AI developers, unfortunately, seek user feedback retrospectively.Still, the cost of the afterthought can be huge both for the service-user and the organisation and could render AI adoption impractical.Prospective user, patient and public involvement should be included at all stages, from design to product roll-out and throughout its life cycle. 50Clinical usability is vital when deploying AI tools since user-friendly interfaces, and accessible, inclusive applications are central to effective AI adoption. 42,66y stakeholders of an AI solution must be actively involved at all stages of the AI integration process. 39Staff involvement will ensure that the AI tool will meet their ergonomic, workflow and performance needs, as well as the needs of their patients.Also, participation in decisions affecting people's lives is a core human right, requiring access to information and freedom of expression. 55Moreover, patients and the community must also be involved in AI product design and delivering of solutions to clinical problems.They are experts through lived experience and have unique insights into the challenges of workflows and usability of clinical tools. 39Public and patient representatives will be more effective if they are invited to act as research co-producers 48 rather than as reviewers of research outcomes at later stages.Key stakeholders may include clinicians, patients, operational and administrative leaders, hospital administrators, and regulatory agencies. 33AI adoption relies on a diverse, highly engaged, well-trained AI ecosystem.

Leadership and staff management
Effective leadership is vital for supporting any new venture in healthcare and beyond, which is true for any successful AI adoption initiative.A well-informed, agile senior leadership should identify and support AI champions for change of culture and knowledge transfer in key practice areas. 20Furthermore, it is essential to enable diverse, multidisciplinary teams to carry the work forward and not rely on only some professionals. 41portance of AI governance frameworks This review underlines that AI governance is multiparametric, and all elements must be finely tuned to safely and effectively deploy AI models in clinical practice.The need for rigorous governance frameworks has been suggested in healthcare and other contexts, such as public administration, 67 finance, 68 and academia. 69AI governance frameworks will also play a fundamental role in mitigating the risks associated with using AI models while also allowing to maximise their benefits, 70 enable safe implementation, 71 and build trust between humans and AI. 72For the seamless adoption of AI, training is essential. 16hilst radiologists have been leading on designing and delivering AI-related educational initiatives, 73,74 radiographers have just started investing in AI education initiatives in medical imaging, 20,21,75 as central to adoption. 14,15,17JR A scoping review to guide AI adoption in medical imaging/radiotherapy Pre-and post-market considerations The main challenges associated with adopting AI solutions can be classified into pre-and post-market considerations.Before procurement of an AI model, it is crucial to know its purpose and clearly define the clinical problem. 76This will ensure that this model fits the needs of the organisation and the end-users.In addition, another crucial pre-market consideration is ensuring that the AI model has been thoroughly validated and that appropriate checks have been performed onsite to assess the model's suitability. 77Furthermore, end-users should ensure that the AI solution does not discriminate against vulnerable groups and that appropriate measures have been taken to mitigate algorithmic bias. 78Finally, all AI models should be assessed based on regulatory standards, and caution should be paid to the required regulatory aspects (e.g.CE marks) to be in place. 71st-market assessment of the model's clinical safety and effectiveness is paramount to ensure that the model will continue to perform as per the initial design and that no harm will impact end-users.Ongoing monitoring of AI tools is essential due to the dynamic nature of these environments; hence, evaluation of these systems should be thoroughly performed throughout their life cycle. 79This monitoring should be standardised. 80An important post-market consideration is the potential risk of algorithmic biases over time.Third parties should be assigned for clinical audits, as they have proved valuable in detecting the weaknesses of AI models. 81All healthcare professionals that use AI models should be able to timely recognise and escalate performance failures resulting from data shifts. 82AI governance principles should therefore be applied to AI models throughout their life cycle, from model development to decommissioning. 83

Financial considerations
Another essential part of AI solutions is the financial implications of clinically deploying these models.Healthcare systems are struggling after COVID-19, decimated by poor staffing and access to material resources. 84Before procuring AI models, a detailed cost/benefit analysis 85 should aim to reduce costs and improve services to benefit the patients and healthcare staff. 86ifferent financial models in healthcare delivery mean different reimbursement models exist.So financial considerations will vary in other countries.

Limitations
The nature of a scoping review means that the findings may be broad due to the general nature of the research question in scoping reviews. 87The framework is intentionally generic in nature, so it can be adjusted and contextualised to individual and local policies and circumstances.However, as evidence is still being developed around AI governance, this was the most appropriate methodology to gather the necessary evidence.
While Health Economics and Outcomes Research is central to AI adoption, 88 it was not included in this review.This was partly because of the complexity of addressing this topic for different organisations.The little evidence starting to emerge, mainly from North America, shows that reimbursement analysis will be central to AI adoption. 89,90However, it remains to be seen how relevant this would be in the UK context.
Finally, since AI is a rapidly changing field, and the number of AI-related publications is growing exponentially, this review will inevitably miss some relevant published work nearer the time of acceptance and publication.
Given the paucity of a unified AI governance framework in medical imaging and radiotherapy in the UK, our results could be used as robust groundwork to develop locally comprehensive AI governance schemes.It is also opportune that a new British Standards Institution specification, currently in public consultation, will be released in the UK in the spring of 2023 to help clarify the finer details of AI adoption. 91

CONCLUSION
This scoping review identified the key elements of AI governance in the UK, focusing on medical imaging and radiotherapy.The proposed conceptual governance framework encompasses rigorous validation and evaluation procedures of AI tools, ongoing monitoring of these models' safety and clinical effectiveness, and compliance with the appropriate accreditation bodies and regulatory standards.The fundamental ethical principles associated with the safe use of AI tools should also be followed, and all AI models should be fair, transparent, trustworthy, and explainable.Staff should be confident about using AI tools in clinical practice.Appropriate staff training is necessary to build trust between AI and humans and ensure the acceptability of new technology.Effective leadership and staff management will further enhance the safe adoption of AI, while research also plays a fundamental role in driving sustainable innovation and growth.

FUNDING
This study received funding from the College of Radiographers Research Award (50028HY-209) and by the City University Radiography Research Fund account.
BJRA scoping review to guide AI adoption in medical imaging/radiotherapy BJRA scoping review to guide AI adoption in medical imaging/radiotherapy of the ethical challenges, and some of the measures we may take to protect against misuse of AI.
and staff the confidence that the digital tools they use meet NHS' clinical safety, data protection, technical security, interoperability and usability and accessibility standards.

Figure 1 .
Figure 1.PRISMA flow diagram.PRISMA, Preferred Reporting Items for Systematic reviews and Meta-Analyses.

Figure 2 .
Figure 2. Types of the obtained sources of evidence.

Figure 4 .
Figure 4. Important concepts under each pillar of the suggested AI governance framework.

Table 2 .
Data extraction form Title