Video-based formative and summative assessment of surgical tasks using deep learning.
Ontology highlight
ABSTRACT:
SUBMITTER: Yanik E
PROVIDER: S-EPMC9852463 | biostudies-literature | 2023 Jan
REPOSITORIES: biostudies-literature
Publications
Video-based formative and summative assessment of surgical tasks using deep learning.
Yanik Erim E Kruger Uwe U Intes Xavier X Rahul Rahul R De Suvranu S
Scientific reports 20230119 1
To ensure satisfactory clinical outcomes, surgical skill assessment must be objective, time-efficient, and preferentially automated-none of which is currently achievable. Video-based assessment (VBA) is being deployed in intraoperative and simulation settings to evaluate technical skill execution. However, VBA is manual, time-intensive, and prone to subjective interpretation and poor inter-rater reliability. Herein, we propose a deep learning (DL) model that can automatically and objectively pro ...[more]
Similar Datasets
Project description:ObjectivesThis study aimed to develop an automated skills assessment tool for surgical trainees using deep learning.BackgroundOptimal surgical performance in robot-assisted surgery (RAS) is essential for ensuring good surgical outcomes. This requires effective training of new surgeons, which currently relies on supervision and skill assessment by experienced surgeons. Artificial Intelligence (AI) presents an opportunity to augment existing human-based assessments.MethodsWe used a network architecture consisting of a convolutional neural network combined with a long short-term memory (LSTM) layer to create two networks for the extraction and analysis of spatial and temporal features from video recordings of surgical procedures, facilitating action recognition and skill assessment.Results21 participants (16 novices and 5 experienced) performed 16 different intra-abdominal robot-assisted surgical procedures on porcine models. The action recognition network achieved an accuracy of 96.0% in identifying surgical actions. A GradCAM filter was used to enhance the model interpretability. The skill assessment network had an accuracy of 81.3% in classifying novices and experiences. Procedure plots were created to visualize the skill assessment.ConclusionOur study demonstrated that AI can be used to automate surgical action recognition and skill assessment. The use of a porcine model enables effective data collection at different levels of surgical performance, which is normally not available in the clinical setting. Future studies need to test how well AI developed within a porcine setting can be used to detect errors and provide feedback and actionable skills assessment in the clinical setting.
| S-EPMC11870904 | biostudies-literature
Project description:BACKGROUND:Pre-recorded videotapes have become the standard approach when teaching clinical communication skills (CCS). Furthermore, video-based feedback (VF) has proven to be beneficial in formative assessments. However, VF in CCS with the use of pre-recorded videos from real-life settings is less commonly studied than the use of simulated patients. To explore: 1) perceptions about the potential benefits and challenges in this kind of VF; 2) differences in the CCC scores in first-year medical residents in primary care, before and after a communication program using VF in a curricular formative assessment. METHOD:We conducted a pre/post study with a control group. The intervention consisted of VF sessions regarding CCS, performed in a small group with peers and a facilitator. They reviewed clinical consultations pre-recorded in a primary care setting with real patients. Before and after the intervention, 54 medical residents performed two clinical examinations with simulated patients (SP), answered quantitative scales (Perception of Patient-Centeredness and Jefferson Empathy Scale), and semi-structured qualitative questionnaires. The performances were scored by SP (Perception of Patient-Centeredness and CARE scale) and by two blind raters (SPIKES protocol-based and CCOG-based scale). The quantitative data analysis employed repeated-measures ANOVA. The qualitative analysis used the Braun and Clarke framework for thematic analysis. RESULTS:The quantitative analyses did not reveal any significant differences in the sum scores of the questionnaires, except for the Jefferson Empathy Scale. In the qualitative questionnaires, the main potential benefits that emerged from the thematic analysis of the VF method were self-perception, peer-feedback, patient-centered approach, and incorporation of reflective practices. A challenging aspect that emerged from facilitators was the struggle to relate the VF with theoretical references and the resident's initial stress to record and watch oneself on video. CONCLUSION:VF taken from real-life settings seems to be associated with a significant increase in self-perceived empathy. The study of other quantitative outcomes related to this VF intervention needs larger sample sizes. VF with clinical patients from real healthcare settings appears to be an opportunity for a deeper level of self-assessment, peer-feedback, and reflective practices.
| S-EPMC7041283 | biostudies-literature
Project description:BackgroundStudy motivation and knowledge retention benefit from regular student self-assessments. Inclusion of certainty-based learning (CBL) in computer-assisted formative tests may further enhance this by enabling students to identify whether they are uninformed or misinformed regarding the topics tested, which may trigger future study actions including instructor consultation.MethodsUsing a cross-over study design involving two out of thirteen computer-assisted formative assessments (CAFAs) of a first-year cell biology course, we compared student-instructor interactions, student learning experiences and final exam scores between two (bio)medical science student cohorts who worked with different CBL-containing CAFAs.ResultsA total of 389 students participated in the study. After completion 159 (41%) filled in a questionnaire on their experience with CBL during supervised CAFAs. In the control group the median duration of student-instructor interactions was 90 s (range 60-140 s), and this increased with 20 s to 110 s (range 60-150 s) in the group working with a CBL-based CAFA. The number of interactions was similar in both groups (0.22 per student per hour, regardless of CBL inclusion). Forty percent of the students expected that CBL would positively influence their study behavior, and 23% also anticipated a positive effect on examination scores. Student examination scores, however, were not affected by CBL. Almost half of the students (43%) were in favor of CBL inclusion in future computer-assisted learning modules, whereas 33% did not see merit in including CBL in CAFAs.ConclusionsIncorporation of CBL in a single formative assessment led to a slight increase in student-instructor interaction times, but had effect neither on the number of student-instructor interactions nor on exam scores. CBL inclusion positively influenced student's appreciation of the coursework, presumably by helping students to evaluate their mastery level and identify misconceptions. A more extensive enrollment of CBL beyond an individual formative assessment, throughout a course or a curriculum, may possibly reveal positive effects on study efficacy.
| S-EPMC6544949 | biostudies-literature
Project description:Abstract Background Purse-string suture in transanal total mesorectal excision is a key procedural step. The aims of this study were to develop an automatic skill assessment system for purse-string suture in transanal total mesorectal excision using deep learning and to evaluate the reliability of the score output from the proposed system. Methods Purse-string suturing extracted from consecutive transanal total mesorectal excision videos was manually scored using a performance rubric scale and computed into a deep learning model as training data. Deep learning-based image regression analysis was performed, and the purse-string suture skill scores predicted by the trained deep learning model (artificial intelligence score) were output as continuous variables. The outcomes of interest were the correlation, assessed using Spearman’s rank correlation coefficient, between the artificial intelligence score and the manual score, purse-string suture time, and surgeon’s experience. Results Forty-five videos obtained from five surgeons were evaluated. The mean(s.d.) total manual score was 9.2(2.7) points, the mean(s.d.) total artificial intelligence score was 10.2(3.9) points, and the mean(s.d.) absolute error between the artificial intelligence and manual scores was 0.42(0.39). Further, the artificial intelligence score significantly correlated with the purse-string suture time (correlation coefficient = −0.728) and surgeon’s experience (P< 0.001). Conclusion An automatic purse-string suture skill assessment system using deep learning-based video analysis was shown to be feasible, and the results indicated that the artificial intelligence score was reliable. This application could be expanded to other endoscopic surgeries and procedures. The aims of this study were to develop an automatic skill assessment system for purse-string suture in transanal total mesorectal excision (TaTME) using artificial intelligence (AI) and to evaluate the reliability of the score output. The score output by AI exhibited statistically significant correlation with the manual score, purse-string suture time, and surgeon’s experience. To the best of our knowledge, this is the first report on automatic skill assessment for purse-string suture in TaTME.
| S-EPMC9991500 | biostudies-literature
Project description:PurposeSurgeons' skill in the operating room is a major determinant of patient outcomes. Assessment of surgeons' skill is necessary to improve patient outcomes and quality of care through surgical training and coaching. Methods for video-based assessment of surgical skill can provide objective and efficient tools for surgeons. Our work introduces a new method based on attention mechanisms and provides a comprehensive comparative analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room.MethodsUsing a dataset of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluated image feature-based methods and two deep learning methods to assess skill using RGB videos. In the first method, we predict instrument tips as keypoints and predict surgical skill using temporal convolutional neural networks. In the second method, we propose a frame-wise encoder (2D convolutional neural network) followed by a temporal model (recurrent neural network), both of which are augmented by visual attention mechanisms. We computed the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and predictive values through fivefold cross-validation.ResultsTo classify a binary skill label (expert vs. novice), the range of AUC estimates was 0.49 (95% confidence interval; CI = 0.37 to 0.60) to 0.76 (95% CI = 0.66 to 0.85) for image feature-based methods. The sensitivity and specificity were consistently high for none of the methods. For the deep learning methods, the AUC was 0.79 (95% CI = 0.70 to 0.88) using keypoints alone, 0.78 (95% CI = 0.69 to 0.88) and 0.75 (95% CI = 0.65 to 0.85) with and without attention mechanisms, respectively.ConclusionDeep learning methods are necessary for video-based assessment of surgical skill in the operating room. Attention mechanisms improved discrimination ability of the network. Our findings should be evaluated for external validity in other datasets.
| S-EPMC10323985 | biostudies-literature
Project description:Despite progressive improvements over the decades, the rich temporally resolved data in an echocardiogram remain underutilized. Human assessments reduce the complex patterns of cardiac wall motion, to a small list of measurements of heart function. All modern echocardiography artificial intelligence (AI) systems are similarly limited by design - automating measurements of the same reductionist metrics rather than utilizing the embedded wealth of data. This underutilization is most evident where clinical decision making is guided by subjective assessments of disease acuity. Predicting the likelihood of developing post-operative right ventricular failure (RV failure) in the setting of mechanical circulatory support is one such example. Here we describe a video AI system trained to predict post-operative RV failure using the full spatiotemporal density of information in pre-operative echocardiography. We achieve an AUC of 0.729, and show that this ML system significantly outperforms a team of human experts at the same task on independent evaluation.
| S-EPMC8408163 | biostudies-literature
Project description:Background/objectivesIn the field of surgical medicine, the planning and execution of liver resection procedures present formidable challenges, primarily attributable to the intricate and highly individualized nature of liver vascular anatomy. In the current surgical milieu, intraoperative ultrasonography (IOUS) has become indispensable; however, traditional 2D ultrasound imaging's interpretability is hindered by noise and speckle artifacts. Accurate identification of critical structures for preservation during hepatectomy requires advanced surgical skills.MethodsAn AI-based model that can help detect and recognize vessels including the inferior vena cava (IVC); the right (RHV), middle (MHV), and left (LVH) hepatic veins; the portal vein (PV) and its major first and second order branches the left portal vein (LPV), right portal vein (RPV), and right anterior (RAPV) and posterior (RPPV) portal veins, for real-time IOUS navigation can be of immense value in liver surgery. This research aims to advance the capabilities of IOUS-guided interventions by applying an innovative AI-based approach named the "2D-weigthed U-Net model" for the segmentation of multiple blood vessels in real-time IOUS video frames.ResultsOur proposed deep learning (DL) model achieved a mean Dice score of 0.92 for IVC, 0.90 for RHV, 0.89 for MHV, 0.86 for LHV, 0.95 for PV, 0.93 for LPV, 0.84 for RPV, 0.85 for RAPV, and 0.96 for RPPV.ConclusionIn the future, this research will be extended for real-time multi-label segmentation of extended vasculature in the liver, followed by the translation of our model into the surgical suite.
| S-EPMC11545685 | biostudies-literature
Project description:BackgroundThere is an urgent need for more diverse methods for student evaluation, given the sudden shift to online learning necessitated by the coronavirus disease 2019 (COVID-19) pandemic. Innovative assessment tools will need to cover the required competencies and should be used to drive self-learning. Self-assessments and peer assessments may be added to the traditional classroom-based evaluations to identify individual insecurities or overconfidence. Identification of these factors is essential to medical education and is a focus of current research.MethodsA modified operational assessment was introduced for the evaluation of third-year medical students. This intervention has facilitated sustained education and has promoted interactive student learning. Members of the entering class of 2017 participated in an integrated team and a competency-based online project that involved innovative item creation and case presentation methods.ResultsThe new assessment process has been implemented successfully with positive feedback from all the participants; a usable product has been generated.ConclusionsWe created new assessment tools in response to the COVID-19 pandemic that have been used successfully at our institution. These tools have provided a framework for integrated and interactive evaluations that can be used to facilitate the modification of traditional assessment methods.
| S-EPMC8346809 | biostudies-literature
Project description:Assessing nontechnical skills in operating rooms (ORs) is crucial for enhancing surgical performance and patient safety. However, automated and real-time evaluation of these skills remains challenging. To explore the feasibility of using motion features extracted from surgical video recordings to automatically assess nontechnical skills during cardiac surgical procedures. This cross-sectional study used video recordings of cardiac surgical procedures at a tertiary academic US hospital collected from January 2021 through May 2022. The OpenPose library was used to analyze videos to extract body pose estimations of team members and compute various team motion features. The Non-Technical Skills for Surgeons (NOTSS) assessment tool was employed for rating the OR team's nontechnical skills by 3 expert raters. NOTSS overall score, with motion features extracted from surgical videos as measures. A total of 30 complete cardiac surgery procedures were included: 26 (86.6%) were on-pump coronary artery bypass graft procedures and 4 (13.4%) were aortic valve replacement or repair procedures. All patients were male, and the mean (SD) age was 72 (6.3) years. All surgical teams were composed of 4 key roles (attending surgeon, attending anesthesiologist, primary perfusionist, and scrub nurse) with additional supporting roles. NOTSS scores correlated significantly with trajectory (r = 0.51, P = .005), acceleration (r = 0.48, P = .008), and entropy (r = -0.52, P = .004) of team displacement. Multiple linear regression, adjusted for patient factors, showed average team trajectory (adjusted R2 = 0.335; coefficient, 10.51 [95% CI, 8.81-12.21]; P = .004) and team displacement entropy (adjusted R2 = 0.304; coefficient, -12.64 [95% CI, -20.54 to -4.74]; P = .003) were associated with NOTSS scores. This study suggests a significant link between OR team movements and nontechnical skills ratings by NOTSS during cardiac surgical procedures, suggesting automated surgical video analysis could enhance nontechnical skills assessment. Further investigation across different hospitals and specialties is necessary to validate these findings.
| S-EPMC11292454 | biostudies-literature
Project description:BackgroundGay, bisexual, and other men who have sex with men (GBMSM) continue to be overrepresented in human immunodeficiency virus (HIV) infection in the United States. HIV prevention and care interventions that are tailored to an individual's serostatus have the potential to lower the rate of new infections among GBMSM. Mobile technology is a critical tool for disseminating targeted messaging and increasing uptake of basic prevention services including HIV testing, sexually transmitted infection (STI) testing, and pre-exposure prophylaxis (PrEP). Mobile Messaging for Men (M-Cubed) is a mobile health HIV prevention intervention designed to deliver video- and text-based prevention messages, provide STI and HIV information, and link GBMSM to prevention and healthcare resources. The current report describes an iterative process of identifying and selecting publicly available videos to be used as part of the M-Cubed intervention. We also conducted interviews with GBMSM to assess the acceptability, comprehension, and potential audience reach of the selected video messages.MethodsThe selection of videos included balancing of specific criteria [e.g., accuracy of scientific information, video length, prevention domains: HIV/STI testing, antiretroviral therapy (ART), PrEP, engagement in care, and condom use] to ensure that they were intended for our GBMSM audiences: HIV-negative men who engage in condomless anal sex, HIV-negative men who do not engage in condomless anal sex, and men living with HIV. This formative study included in-person interviews with 26 GBMSM from three U.S. cities heavily impacted by the HIV epidemic-New York City, Detroit, and Atlanta.ResultsFollowing a qualitative content analysis, the study team identified five themes across the interviews: participant reactions to the video messages, message comprehension, PrEP concerns, targeting of video messaging, and prompted action.ConclusionsStudy results informed a final selection of 12 video messages for inclusion in a randomized controlled trial of M-Cubed. Findings may serve as a guide for researchers who plan to develop HIV prevention interventions that utilize publicly available videos to promote behavioral change. Further, the findings presented here suggest the importance of developing videos with broad age and gender diversity for use in interventions such as M-Cubed, and in other health promotion settings.
| S-EPMC9902231 | biostudies-literature