Evaluating the safety and patient impacts of an artificial intelligence command centre in acute hospital care: a mixed-methods protocol.
Ontology highlight
ABSTRACT: Introduction
This paper presents a mixed-methods study protocol that will be used to evaluate a recent implementation of a real-time, centralised hospital command centre in the UK. The command centre represents a complex intervention within a complex adaptive system. It could support better operational decision-making and facilitate identification and mitigation of threats to patient safety. There is, however, limited research on the impact of such complex health information technology on patient safety, reliability and operational efficiency of healthcare delivery and this study aims to help address that gap.Methods and analysis
We will conduct a longitudinal mixed-method evaluation that will be informed by public-and-patient involvement and engagement. Interviews and ethnographic observations will inform iterations with quantitative analysis that will sensitise further qualitative work. Quantitative work will take an iterative approach to identify relevant outcome measures from both the literature and pragmatically from datasets of routinely collected electronic health records.Ethics and dissemination
This protocol has been approved by the University of Leeds Engineering and Physical Sciences Research Ethics Committee (#MEEC 20-016) and the National Health Service Health Research Authority (IRAS No.: 285933). Our results will be communicated through peer-reviewed publications in international journals and conferences. We will provide ongoing feedback as part of our engagement work with local trust stakeholders.
SUBMITTER: McInerney C
PROVIDER: S-EPMC8889317 | biostudies-literature | 2022 Mar
REPOSITORIES: biostudies-literature
Publications
Evaluating the safety and patient impacts of an artificial intelligence command centre in acute hospital care: a mixed-methods protocol.
McInerney Ciarán C McCrorie Carolyn C Benn Jonathan J Habli Ibrahim I Lawton Tom T Mebrahtu Teumzghi F TF Randell Rebecca R Sheikh Naeem N Johnson Owen O
BMJ open 20220301 3
<h4>Introduction</h4>This paper presents a mixed-methods study protocol that will be used to evaluate a recent implementation of a real-time, centralised hospital command centre in the UK. The command centre represents a complex intervention within a complex adaptive system. It could support better operational decision-making and facilitate identification and mitigation of threats to patient safety. There is, however, limited research on the impact of such complex health information technology o ...[more]
Similar Datasets
Project description:Digital technologies in health care, including artificial intelligence (AI) and robotics, constantly increase. The aim of this study was to explore attitudes of 2020 medical students' generation towards various aspects of eHealth technologies with the focus on AI using an exploratory sequential mixed-method analysis. Data from semi-structured interviews with 28 students from five medical faculties were used to construct an online survey send to about 80,000 medical students in Germany. Most students expressed positive attitudes towards digital applications in medicine. Students with a problem-based curriculum (PBC) in contrast to those with a science-based curriculum (SBC) and male undergraduate students think that AI solutions result in better diagnosis than those from physicians (p < 0.001). Male undergraduate students had the most positive view of AI (p < 0.002). Around 38% of the students felt ill-prepared and could not answer AI-related questions because digitization in medicine and AI are not a formal part of the medical curriculum. AI rating regarding the usefulness in diagnostics differed significantly between groups. Higher emphasis in medical curriculum of digital solutions in patient care is postulated.
| S-EPMC9027704 | biostudies-literature
Project description:IntroductionThe use and value of artificial intelligence (AI)-driven tools and techniques are under investigation in detecting coronary artery disease (CAD). EchoGo Pro is a patented AI-driven stress echocardiography analysis system produced by Ultromics Ltd. (henceforth Ultromics) to support clinicians in detecting cardiac ischaemia and potential CAD. This manuscript presents the research protocol for a field study to independently evaluate the accuracy, acceptability, implementation barriers, users' experience and willingness to pay, cost-effectiveness and value of EchoGo Pro.Methods and analysisThe 'Evaluating AI-driven stress echocardiography analysis system' (EASE) study is a mixed-method evaluation, which will be conducted in five work packages (WPs). In WP1, we will examine the diagnostic accuracy by comparing test reports generated by EchoGo Pro and three manual raters. In WP2, we will focus on interviewing clinicians, innovation/transformation staff, and patients within the National Health Service (NHS), and staff within Ultromics, to assess the acceptability of this technology. In this WP, we will determine convergence and divergence between EchoGo Pro recommendations and cardiologists' interpretations and will assess what profile of cases is linked with convergence and divergence between EchoGo Pro recommendations and cardiologists' interpretations and how these link to outcomes. In WP4, we will conduct a quantitative cross-sectional survey of trust in AI tools applied to cardiac care settings among clinicians, healthcare commissioners and the general public. Lastly, in WP5, we will estimate the cost of deploying the EchoGo Pro technology, cost-effectiveness and willingness to pay cardiologists, healthcare commissioners and the general public. The results of this evaluation will support evidence-informed decision-making around the widespread adoption of EchoGo Pro and similar technologies in the NHS and other health systems.Ethics approval and disseminationThis research has been approved by the NHS Health Research Authority (IRAS No: 315284) and the London South Bank University Ethics Panel (ETH2223-0164). Alongside journal publications, we will disseminate study methods and findings in conferences, seminars and social media. We will produce additional outputs in appropriate forms, for example, research summaries and policy briefs, for diverse audiences in NHS.
| S-EPMC11448110 | biostudies-literature
Project description:BackgroundArtificial intelligence (AI)-powered technologies are being increasingly used in almost all fields, including medicine. However, to successfully implement medical AI applications, ensuring trust and acceptance toward such technologies is crucial for their successful spread and timely adoption worldwide. Although AI applications in medicine provide advantages to the current health care system, there are also various associated challenges regarding, for instance, data privacy, accountability, and equity and fairness, which could hinder medical AI application implementation.ObjectiveThe aim of this study was to identify factors related to trust in and acceptance of novel AI-powered medical technologies and to assess the relevance of those factors among relevant stakeholders.MethodsThis study used a mixed methods design. First, a rapid review of the existing literature was conducted, aiming to identify various factors related to trust in and acceptance of novel AI applications in medicine. Next, an electronic survey including the rapid review-derived factors was disseminated among key stakeholder groups. Participants (N=22) were asked to assess on a 5-point Likert scale (1=irrelevant to 5=relevant) to what extent they thought the various factors (N=19) were relevant to trust in and acceptance of novel AI applications in medicine.ResultsThe rapid review (N=32 papers) yielded 110 factors related to trust and 77 factors related to acceptance toward AI technology in medicine. Closely related factors were assigned to 1 of the 19 overarching umbrella factors, which were further grouped into 4 categories: human-related (ie, the type of institution AI professionals originate from), technology-related (ie, the explainability and transparency of AI application processes and outcomes), ethical and legal (ie, data use transparency), and additional factors (ie, AI applications being environment friendly). The categorized 19 umbrella factors were presented as survey statements, which were evaluated by relevant stakeholders. Survey participants (N=22) represented researchers (n=18, 82%), technology providers (n=5, 23%), hospital staff (n=3, 14%), and policy makers (n=3, 14%). Of the 19 factors, 16 (84%) human-related, technology-related, ethical and legal, and additional factors were considered to be of high relevance to trust in and acceptance of novel AI applications in medicine. The patient's gender, age, and education level were found to be of low relevance (3/19, 16%).ConclusionsThe results of this study could help the implementers of medical AI applications to understand what drives trust and acceptance toward AI-powered technologies among key stakeholders in medicine. Consequently, this would allow the implementers to identify strategies that facilitate trust in and acceptance of medical AI applications among key stakeholders and potential users.
| S-EPMC10831593 | biostudies-literature
Project description:BackgroundClinical decision support systems are designed to utilize medical data, knowledge, and analysis engines and to generate patient-specific assessments or recommendations to health professionals in order to assist decision making. Artificial intelligence-enabled clinical decision support systems aid the decision-making process through an intelligent component. Well-defined evaluation methods are essential to ensure the seamless integration and contribution of these systems to clinical practice.ObjectiveThe purpose of this study was to develop and validate a measurement instrument and test the interrelationships of evaluation variables for an artificial intelligence-enabled clinical decision support system evaluation framework.MethodsAn artificial intelligence-enabled clinical decision support system evaluation framework consisting of 6 variables was developed. A Delphi process was conducted to develop the measurement instrument items. Cognitive interviews and pretesting were performed to refine the questions. Web-based survey response data were analyzed to remove irrelevant questions from the measurement instrument, to test dimensional structure, and to assess reliability and validity. The interrelationships of relevant variables were tested and verified using path analysis, and a 28-item measurement instrument was developed. Measurement instrument survey responses were collected from 156 respondents.ResultsThe Cronbach α of the measurement instrument was 0.963, and its content validity was 0.943. Values of average variance extracted ranged from 0.582 to 0.756, and values of the heterotrait-monotrait ratio ranged from 0.376 to 0.896. The final model had a good fit (χ262=36.984; P=.08; comparative fit index 0.991; goodness-of-fit index 0.957; root mean square error of approximation 0.052; standardized root mean square residual 0.028). Variables in the final model accounted for 89% of the variance in the user acceptance dimension.ConclusionsUser acceptance is the central dimension of artificial intelligence-enabled clinical decision support system success. Acceptance was directly influenced by perceived ease of use, information quality, service quality, and perceived benefit. Acceptance was also indirectly influenced by system quality and information quality through perceived ease of use. User acceptance and perceived benefit were interrelated.
| S-EPMC8209524 | biostudies-literature
Project description:ObjectiveMedical students, as clinicians and healthcare leaders of the future, are key stakeholders in the clinical roll-out of artificial intelligence-driven technologies. The authors aim to provide the first report on the state of artificial intelligence in medical education globally by exploring the perspectives of medical students.MethodsThe authors carried out a mixed-methods study of focus groups and surveys with 128 medical students from 48 countries. The study explored knowledge around artificial intelligence as well as what students wished to learn about artificial intelligence and how they wished to learn this. A combined qualitative and quantitative analysis was used.ResultsSupport for incorporating teaching on artificial intelligence into core curricula was ubiquitous across the globe, but few students had received teaching on artificial intelligence. Students showed knowledge on the applications of artificial intelligence in clinical medicine as well as on artificial intelligence ethics. They were interested in learning about clinical applications, algorithm development, coding and algorithm appraisal. Hackathon-style projects and multidisciplinary education involving computer science students were suggested for incorporation into the curriculum.ConclusionsMedical students from all countries should be provided teaching on artificial intelligence as part of their curriculum to develop skills and knowledge around artificial intelligence to ensure a patient-centred digital future in medicine. This teaching should focus on the applications of artificial intelligence in clinical medicine. Students should also be given the opportunity to be involved in algorithm development. Students in low- and middle-income countries require the foundational technology as well as robust teaching on artificial intelligence to ensure that they can drive innovation in their healthcare settings.
| S-EPMC9067043 | biostudies-literature
Project description:BackgroundPredicting the likelihood of success of weight loss interventions using machine learning (ML) models may enhance intervention effectiveness by enabling timely and dynamic modification of intervention components for nonresponders to treatment. However, a lack of understanding and trust in these ML models impacts adoption among weight management experts. Recent advances in the field of explainable artificial intelligence enable the interpretation of ML models, yet it is unknown whether they enhance model understanding, trust, and adoption among weight management experts.ObjectiveThis study aimed to build and evaluate an ML model that can predict 6-month weight loss success (ie, ≥7% weight loss) from 5 engagement and diet-related features collected over the initial 2 weeks of an intervention, to assess whether providing ML-based explanations increases weight management experts' agreement with ML model predictions, and to inform factors that influence the understanding and trust of ML models to advance explainability in early prediction of weight loss among weight management experts.MethodsWe trained an ML model using the random forest (RF) algorithm and data from a 6-month weight loss intervention (N=419). We leveraged findings from existing explainability metrics to develop Prime Implicant Maintenance of Outcome (PRIMO), an interactive tool to understand predictions made by the RF model. We asked 14 weight management experts to predict hypothetical participants' weight loss success before and after using PRIMO. We compared PRIMO with 2 other explainability methods, one based on feature ranking and the other based on conditional probability. We used generalized linear mixed-effects models to evaluate participants' agreement with ML predictions and conducted likelihood ratio tests to examine the relationship between explainability methods and outcomes for nested models. We conducted guided interviews and thematic analysis to study the impact of our tool on experts' understanding and trust in the model.ResultsOur RF model had 81% accuracy in the early prediction of weight loss success. Weight management experts were significantly more likely to agree with the model when using PRIMO (χ2=7.9; P=.02) compared with the other 2 methods with odds ratios of 2.52 (95% CI 0.91-7.69) and 3.95 (95% CI 1.50-11.76). From our study, we inferred that our software not only influenced experts' understanding and trust but also impacted decision-making. Several themes were identified through interviews: preference for multiple explanation types, need to visualize uncertainty in explanations provided by PRIMO, and need for model performance metrics on similar participant test instances.ConclusionsOur results show the potential for weight management experts to agree with the ML-based early prediction of success in weight loss treatment programs, enabling timely and dynamic modification of intervention components to enhance intervention effectiveness. Our findings provide methods for advancing the understandability and trust of ML models among weight management experts.
| S-EPMC10512114 | biostudies-literature
Project description:BackgroundGiven the rapidity with which artificial intelligence is gaining momentum in clinical medicine, current physician leaders have called for more incorporation of artificial intelligence topics into undergraduate medical education. This is to prepare future physicians to better work together with artificial intelligence technology. However, the first step in curriculum development is to survey the needs of end users. There has not been a study to determine which media and which topics are most preferred by US medical students to learn about the topic of artificial intelligence in medicine.ObjectiveWe aimed to survey US medical students on the need to incorporate artificial intelligence in undergraduate medical education and their preferred means to do so to assist with future education initiatives.MethodsA mixed methods survey comprising both specific questions and a write-in response section was sent through Qualtrics to US medical students in May 2021. Likert scale questions were used to first assess various perceptions of artificial intelligence in medicine. Specific questions were posed regarding learning format and topics in artificial intelligence.ResultsWe surveyed 390 US medical students with an average age of 26 (SD 3) years from 17 different medical programs (the estimated response rate was 3.5%). A majority (355/388, 91.5%) of respondents agreed that training in artificial intelligence concepts during medical school would be useful for their future. While 79.4% (308/388) were excited to use artificial intelligence technologies, 91.2% (353/387) either reported that their medical schools did not offer resources or were unsure if they did so. Short lectures (264/378, 69.8%), formal electives (180/378, 47.6%), and Q and A panels (167/378, 44.2%) were identified as preferred formats, while fundamental concepts of artificial intelligence (247/379, 65.2%), when to use artificial intelligence in medicine (227/379, 59.9%), and pros and cons of using artificial intelligence (224/379, 59.1%) were the most preferred topics for enhancing their training.ConclusionsThe results of this study indicate that current US medical students recognize the importance of artificial intelligence in medicine and acknowledge that current formal education and resources to study artificial intelligence-related topics are limited in most US medical schools. Respondents also indicated that a hybrid formal/flexible format would be most appropriate for incorporating artificial intelligence as a topic in US medical schools. Based on these data, we conclude that there is a definitive knowledge gap in artificial intelligence education within current medical education in the US. Further, the results suggest there is a disparity in opinions on the specific format and topics to be introduced.
| S-EPMC9636531 | biostudies-literature
Project description:BackgroundArtificial intelligence (AI) methods and AI-enabled metrics hold tremendous potential to advance surgical education. Our objective was to generate consensus guidance on specific needs for AI methods and AI-enabled metrics for surgical education.Study designThe study included a systematic literature search, a virtual conference, and a 3-round Delphi survey of 40 representative multidisciplinary stakeholders with domain expertise selected through purposeful sampling. The accelerated Delphi process was completed within 10 days. The survey covered overall utility, anticipated future (10-year time horizon), and applications for surgical training, assessment, and feedback. Consensus was agreement among 80% or more respondents. We coded survey questions into 11 themes and descriptively analyzed the responses.ResultsThe respondents included surgeons (40%), engineers (15%), affiliates of industry (27.5%), professional societies (7.5%), regulatory agencies (7.5%), and a lawyer (2.5%). The survey included 155 questions; consensus was achieved on 136 (87.7%). The panel listed 6 deliverables each for AI-enhanced learning curve analytics and surgical skill assessment. For feedback, the panel identified 10 priority deliverables spanning 2-year (n = 2), 5-year (n = 4), and 10-year (n = 4) timeframes. Within 2 years, the panel expects development of methods to recognize anatomy in images of the surgical field and to provide surgeons with performance feedback immediately after an operation. The panel also identified 5 essential that should be included in operative performance reports for surgeons.ConclusionsThe Delphi panel consensus provides a specific, bold, and forward-looking roadmap for AI methods and AI-enabled metrics for surgical education.
| S-EPMC10634198 | biostudies-literature
Project description:BACKGROUND:Clinicians' time with patients has become increasingly limited due to regulatory burden, documentation and billing, administrative responsibilities, and market forces. These factors limit clinicians' time to deliver thorough explanations to patients. OpenNotes began as a research initiative exploring the ability of sharing medical notes with patients to help patients understand their health care. Providing patients access to their medical notes has been shown to have many benefits, including improved patient satisfaction and clinical outcomes. OpenNotes has since evolved into a national movement that helps clinicians share notes with patients. However, a significant barrier to the widespread adoption of OpenNotes has been clinicians' concerns that OpenNotes may cost additional time to correct patient confusion over medical language. Recent advances in artificial intelligence (AI) technology may help resolve this concern by converting medical notes to plain language with minimal time required of clinicians. OBJECTIVE:This pilot study assesses patient comprehension and perceived benefits, concerns, and insights regarding an AI-simplified note through comprehension questions and guided interview. METHODS:Synthea, a synthetic patient generator, was used to generate a standardized medical-language patient note which was then simplified using AI software. A multiple-choice comprehension assessment questionnaire was drafted with physician input. Study participants were recruited from inpatients at the University of Colorado Hospital. Participants were randomly assigned to be tested for their comprehension of the standardized medical-language version or AI-generated plain-language version of the patient note. Following this, participants reviewed the opposite version of the note and participated in a guided interview. A Student t test was performed to assess for differences in comprehension assessment scores between plain-language and medical-language note groups. Multivariate modeling was performed to assess the impact of demographic variables on comprehension. Interview responses were thematically analyzed. RESULTS:Twenty patients agreed to participate. The mean number of comprehension assessment questions answered correctly was found to be higher in the plain-language group compared with the medical-language group; however, the Student t test was found to be underpowered to determine if this was significant. Age, ethnicity, and health literacy were found to have a significant impact on comprehension scores by multivariate modeling. Thematic analysis of guided interviews highlighted patients' perceived benefits, concerns, and suggestions regarding such notes. Major themes of benefits were that simplified plain-language notes may (1) be more useable than unsimplified medical-language notes, (2) improve the patient-clinician relationship, and (3) empower patients through an enhanced understanding of their health care. CONCLUSIONS:AI software may translate medical notes into plain-language notes that are perceived as beneficial by patients. Limitations included sample size, inpatient-only setting, and possible confounding factors. Larger studies are needed to assess comprehension. Insight from patient responses to guided interviews can guide the future study and development of this technology.
| S-EPMC7305564 | biostudies-literature
Project description: Not available
| S-EPMC10248742 | biostudies-literature