Dataset Information

Accuracy and safety of an autonomous artificial intelligence clinical assistant conducting telemedicine follow-up assessment for cataract surgery.

ABSTRACT:

Background

Artificial intelligence deployed to triage patients post-cataract surgery could help to identify and prioritise individuals who need clinical input and to expand clinical capacity. This study investigated the accuracy and safety of an autonomous telemedicine call (Dora, version R1) in detecting cataract surgery patients who need further management and compared its performance against ophthalmic specialists.

Methods

225 participants were recruited from two UK public teaching hospitals after routine cataract surgery between 17 September 2021 and 31 January 2022. Eligible patients received a call from Dora R1 to conduct a follow-up assessment approximately 3 weeks post cataract surgery, which was supervised in real-time by an ophthalmologist. The primary analysis compared decisions made independently by Dora R1 and the supervising ophthalmologist about the clinical significance of five symptoms and whether the patient required further review. Secondary analyses used mixed methods to examine Dora R1's usability and acceptability and to assess cost impact compared to standard care. This study is registered with ClinicalTrials.gov (NCT05213390) and ISRCTN (16038063).

Findings

202 patients were included in the analysis, with data collection completed on 23 March 2022. Dora R1 demonstrated an overall outcome sensitivity of 94% and specificity of 86% and showed moderate to strong agreement (kappa: 0.758-0.970) with clinicians in all parameters. Safety was validated by assessing subsequent outcomes: 11 of the 117 patients (9%) recommended for discharge by Dora R1 had unexpected management changes, but all were also recommended for discharge by the supervising clinician. Four patients were recommended for discharge by Dora R1 but not the clinician; none required further review on callback. Acceptability, from interviews with 20 participants, was generally good in routine circumstances but patients were concerned about the lack of a 'human element' in cases with complications. Feasibility was demonstrated by the high proportion of calls completed autonomously (195/202, 96.5%). Staff cost benefits for Dora R1 compared to standard care were £35.18 per patient.

Interpretation

The composite of mixed methods analysis provides preliminary evidence for the safety, acceptability, feasibility, and cost benefits for clinical adoption of an artificial intelligence conversational agent, Dora R1, to conduct follow-up assessment post-cataract surgery. Further evaluation in real-world implementation should be conducted to provide additional evidence around safety and effectiveness in a larger sample from a more diverse set of Trusts.

Funding

This manuscript is independent research funded by the National Institute for Health Research and NHSX (Artificial Intelligence in Health and Care Award, AI_AWARD01852).

SUBMITTER: Meinert E

PROVIDER: S-EPMC11266473 | biostudies-literature | 2024 Jul

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Accuracy and safety of an autonomous artificial intelligence clinical assistant conducting telemedicine follow-up assessment for cataract surgery.

Meinert Edward E Milne-Ives Madison M Lim Ernest E Higham Aisling A Boege Selina S de Pennington Nick N Bajre Mamta M Mole Guy G Normando Eduardo E Xue Kanmin K

EClinicalMedicine 20240703

<h4>Background</h4>Artificial intelligence deployed to triage patients post-cataract surgery could help to identify and prioritise individuals who need clinical input and to expand clinical capacity. This study investigated the accuracy and safety of an autonomous telemedicine call (Dora, version R1) in detecting cataract surgery patients who need further management and compared its performance against ophthalmic specialists.<h4>Methods</h4>225 participants were recruited from two UK public teac ...[more]

PMID: 39050586

Similar Datasets

Project description:BackgroundDue to an aging population, the demand for many services is exceeding the capacity of the clinical workforce. As a result, staff are facing a crisis of burnout from being pressured to deliver high-volume workloads, driving increasing costs for providers. Artificial intelligence (AI), in the form of conversational agents, presents a possible opportunity to enable efficiency in the delivery of care.ObjectiveThis study aims to evaluate the effectiveness, usability, and acceptability of Dora agent: Ufonia's autonomous voice conversational agent, an AI-enabled autonomous telemedicine call for the detection of postoperative cataract surgery patients who require further assessment. The objectives of this study are to establish Dora's efficacy in comparison with an expert clinician, determine baseline sensitivity and specificity for the detection of true complications, evaluate patient acceptability, collect evidence for cost-effectiveness, and capture data to support further development and evaluation.MethodsUsing an implementation science construct, the interdisciplinary study will be a mixed methods phase 1 pilot establishing interobserver reliability of the system, usability, and acceptability. This will be done using the following scales and frameworks: the system usability scale; assessment of Health Information Technology Interventions in Evidence-Based Medicine Evaluation Framework; the telehealth usability questionnaire; and the Non-Adoption, Abandonment, and Challenges to the Scale-up, Spread and Suitability framework.ResultsThe evaluation is expected to show that conversational technology can be used to conduct an accurate assessment and that it is acceptable to different populations with different backgrounds. In addition, the results will demonstrate how successfully the system can be delivered in organizations with different clinical pathways and how it can be integrated with their existing platforms.ConclusionsThe project's key contributions will be evidence of the effectiveness of AI voice conversational agents and their associated usability and acceptability.International registered report identifier (irrid)PRR1-10.2196/27227.

Project description:ObjectiveT cell immunity plays an important role in anti-tumor effects and immunosuppression often leads to the development and relapse of cancer. This study aimed to investigate the effect of T cell numbers on the long-term prognosis of patients with hepatocellular carcinoma (HCC) and construct an artificial neural network (ANN) model to evaluate its prognostic value.MethodsWe enrolled 3,427 patients with HCC at Beijing Ditan Hospital, Capital Medical University, and randomly divided them into two groups of 1,861 and 809 patients as the training and validation sets, respectively. Cox regression analysis was used to screen for independent risk factors of survival in patients with HCC. These factors were used to build an ANN model using Python. Concordance index, calibration curve, and decision curve analysis were used to evaluate the model performance.ResultsThe 1-year, 3-year, 5-year, and 10-year cumulative overall survival (OS) rates were 66.9%, 45.7%, 34.9%, and 22.6%, respectively. Cox multivariate regression analysis showed that age, white blood cell count, creatinine, total bilirubin, γ-GGT, LDH, tumor size ≥ 5 cm, tumor number ≥ 2, portal vein tumor thrombus, and AFP ≥ 400 ng/ml were independent risk factors for long-term survival in HCC. Antiviral therapy, albumin, T cell, and CD8 T cell counts were independent protective factors. An ANN model was developed for long-term survival. The areas under the receiver operating characteristic (ROC) curve of 1-year, 3-year, and 5-year OS rates by ANNs were 0.838, 0.833, and 0.843, respectively, which were higher than those of the Barcelona Clinic Liver Cancer (BCLC), tumor node metastasis (TNM), Okuda, Chinese University Prognostic Index (CUPI), Cancer of the Liver Italian Program (CLIP), Japan Integrated Staging (JIS), and albumin-bilirubin (ALBI) models (P < 0.0001). According to the ANN model scores, all patients were divided into high-, middle-, and low-risk groups. Compared with low-risk patients, the hazard ratios of 5-year OS of the high-risk group were 8.11 (95% CI: 7.0-9.4) and 6.13 (95% CI: 4.28-8.79) (P<0.0001) in the training and validation sets, respectively.ConclusionHigh levels of circulating T cells and CD8 + T cells in peripheral blood may benefit the long-term survival of patients with HCC. The ANN model has a good individual prediction performance, which can be used to assess the prognosis of HCC and lay the foundation for the implementation of precision treatment in the future.

Project description:BackgroundConsumer-oriented mobile self-diagnosis apps have been developed using undisclosed algorithms, presumably based on machine learning and other artificial intelligence (AI) technologies. The US Food and Drug Administration now discerns apps with learning AI algorithms from those with stable ones and treats the former as medical devices. To the author's knowledge, no self-diagnosis app testing has been performed in the field of ophthalmology so far.ObjectiveThe objective of this study was to test apps that were previously mentioned in the scientific literature on a set of diagnoses in a deliberate time interval, comparing the results and looking for differences that hint at "nonlocked" learning algorithms.MethodsFour apps from the literature were chosen (Ada, Babylon, Buoy, and Your.MD). A set of three ophthalmology diagnoses (glaucoma, retinal tear, dry eye syndrome) representing three levels of urgency was used to simultaneously test the apps' diagnostic efficiency and treatment recommendations in this specialty. Two years was the chosen time interval between the tests (2018 and 2020). Scores were awarded by one evaluating physician using a defined scheme.ResultsTwo apps (Ada and Your.MD) received significantly higher scores than the other two. All apps either worsened in their results between 2018 and 2020 or remained unchanged at a low level. The variation in the results over time indicates "nonlocked" learning algorithms using AI technologies. None of the apps provided correct diagnoses and treatment recommendations for all three diagnoses in 2020. Two apps (Babylon and Your.MD) asked significantly fewer questions than the other two (P<.001).Conclusions"Nonlocked" algorithms are used by self-diagnosis apps. The diagnostic efficiency of the tested apps seems to worsen over time, with some apps being more capable than others. Systematic studies on a wider scale are necessary for health care providers and patients to correctly assess the safety and efficacy of such apps and for correct classification by health care regulating authorities.

Project description:Some research has already reported the diagnostic value of artificial intelligence (AI) in different endoscopy outcomes. However, the evidence is confusing and of varying quality. This review aimed to comprehensively evaluate the credibility of the evidence of AI's diagnostic accuracy in endoscopy. Before the study began, the protocol was registered on PROSPERO (CRD42023483073). First, 2 researchers searched PubMed, Web of Science, Embase, and Cochrane Library using comprehensive search terms. Then, researchers screened the articles and extracted information. We used A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR2) to evaluate the quality of the articles. When there were multiple studies aiming at the same result, we chose the study with higher-quality evaluations for further analysis. To ensure the reliability of the conclusions, we recalculated each outcome. Finally, the Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) was used to evaluate the credibility of the outcomes. A total of 21 studies were included for analysis. Through AMSTAR2, it was found that 8 research methodologies were of moderate quality, while other studies were regarded as having low or critically low quality. The sensitivity and specificity of 17 different outcomes were analyzed. There were 4 studies on esophagus, 4 studies on stomach, and 4 studies on colorectal regions. Two studies were associated with capsule endoscopy, two were related to laryngoscopy, and one was related to ultrasonic endoscopy. In terms of sensitivity, gastroesophageal reflux disease had the highest accuracy rate, reaching 97%, while the invasion depth of colon neoplasia, with 71%, had the lowest accuracy rate. On the other hand, the specificity of colorectal cancer was the highest, reaching 98%, while the gastrointestinal stromal tumor, with only 80%, had the lowest specificity. The GRADE evaluation suggested that the reliability of most outcomes was low or very low. AI proved valuabe in endoscopic diagnoses, especially in esophageal and colorectal diseases. These findings provide a theoretical basis for developing and evaluating AI-assisted systems, which are aimed at assisting endoscopists in carrying out examinations, leading to improved patient health outcomes. However, further high-quality research is needed in the future to fully validate AI's effectiveness.

Dataset Information

Accuracy and safety of an autonomous artificial intelligence clinical assistant conducting telemedicine follow-up assessment for cataract surgery.

Background

Methods

Findings

Interpretation

Funding

Publications

Accuracy and safety of an autonomous artificial intelligence clinical assistant conducting telemedicine follow-up assessment for cataract surgery.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets