Dataset Information

Subjective vs Computerized Assessment of Surgeon Skill Level During Mastoidectomy.

ABSTRACT: This pilot study examines the use of surgical instrument tracking and motion analysis in objectively measuring surgical performance. Accuracy of objective measures in distinguishing between surgeons of different levels was compared to that of subjective assessments. Twenty-four intraoperative video clips of mastoidectomies performed by junior residents (n = 12), senior residents (n = 8), and faculty (n = 4) were sent to otolaryngology programs via survey, yielding 708 subjective ratings of surgical experience level. Tracking software captured the total distance traveled by the drill, suction irrigator, and patient's head. Measurements were used to predict surgeon level of training, and accuracy was estimated via area under the curve (AUC) of receiver operating characteristic curves. Key objective metrics proved more accurate than subjective evaluations in determining both faculty vs resident level and senior vs junior resident level. The findings of this study suggest that objective analysis using computer software has the potential to improve the accuracy of surgical skill assessment.

SUBMITTER: Close MF

PROVIDER: S-EPMC8136835 | biostudies-literature | 2020 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Subjective vs Computerized Assessment of Surgeon Skill Level During Mastoidectomy.

Close Michaela F MF Mehta Charmee H CH Liu Yuan Y Isaac Mitchell J MJ Costello Mark S MS Kulbarsh Kyle D KD Meyer Ted A TA

Otolaryngology--head and neck surgery : official journal of American Academy of Otolaryngology-Head and Neck Surgery 20200630 6

This pilot study examines the use of surgical instrument tracking and motion analysis in objectively measuring surgical performance. Accuracy of objective measures in distinguishing between surgeons of different levels was compared to that of subjective assessments. Twenty-four intraoperative video clips of mastoidectomies performed by junior residents (n = 12), senior residents (n = 8), and faculty (n = 4) were sent to otolaryngology programs via survey, yielding 708 subjective ratings of surgi ...[more]

PMID: 32600121

Similar Datasets

Project description:ImportanceTechnical skill in complex surgical procedures may affect clinical outcomes, and there is growing interest in understanding the clinical implications of surgeon proficiency levels.ObjectivesTo determine whether surgeon scores representing technical skills of robot-assisted kidney surgery are associated with patient outcomes.Design, setting, and participantsThis quality improvement study included 10 urological surgeons participating in a surgical collaborative in Michigan from July 2021 to September 2022. Each surgeon submitted up to 7 videos of themselves performing robot-assisted partial nephrectomy. Videos were segmented into 6 key steps, yielding 127 video clips for analysis. Each video clip was deidentified and distributed to at least 3 of the 24 blinded peer surgeons from the collaborative who also perform robot-assisted partial nephrectomy. Reviewers rated technical skill and provided written feedback. Statistical analysis was performed from May 2023 to January 2024.Main outcomes and measuresReviewers scored each video clip using a validated instrument to assess technical skill for partial nephrectomy on a scale of 1 to 5 (higher scores indicating greater skill). For all submitting surgeons, outcomes from a clinical registry were assessed for length of stay (LOS) greater than 3 days, estimated blood loss (EBL) greater than 500 mL, warm ischemia time (WIT) greater than 30 minutes, positive surgical margin (PSM), 30-day emergency department (ED) visits, and 30-day readmission.ResultsAmong the 27 unique surgeons who participated in this study as reviewers and/or individuals performing the procedures, 3 (11%) were female, and the median age was 47 (IQR, 39-52) years. Risk-adjusted outcomes were associated with scores representing surgeon skills. The overall performance score ranged from 3.5 to 4.7 points with a mean (SD) of 4.1 (0.4) points. Greater skill was correlated with significantly lower rates of LOS greater than 3 days (-6.8% [95% CI, -8.3% to -5.2%]), EBL greater than 500 mL (-2.6% [95% CI, -3.0% to -2.1%]), PSM (-8.2% [95% CI, -9.2% to -7.2%]), ED visits (-3.9% [95% CI, -5.0% to -2.8%]), and readmissions (-5.7% [95% CI, -6.9% to -4.6%]) (P < .001 for all). Higher overall score was also associated with higher partial nephrectomy volume (β coefficient, 11.4 [95% CI, 10.0-12.7]; P < .001).Conclusions and relevanceIn this quality improvement study on video-based evaluation of robot-assisted partial nephrectomy, higher technical skill was associated with lower rates of adverse clinical outcomes. These findings suggest that video-based evaluation plays a role in assessing surgical skill and can be used in quality improvement initiatives to improve patient care.

Project description:Measuring cognitive load is important for surgical education and patient safety. Traditional approaches of measuring cognitive load of surgeons utilise behavioural metrics to measure performance and surveys and questionnaires to collect reports of subjective experience. These have disadvantages such as sporadic data, occasionally intrusive methodologies, subjective or misleading self-reporting. In addition, traditional approaches use subjective metrics that cannot distinguish between skill levels. Functional neuroimaging data was collected using a high density, wireless NIRS device from sixteen surgeons (11 attending surgeons and 5 surgery resident) and 17 students while they performed two laparoscopic tasks (Peg transfer and String pass). Participant's subjective mental load was assessed using the NASA-TLX survey. Machine learning approaches were used for predicting the subjective experience and skill levels. The Prefrontal cortex (PFC) activations were greater in students who reported higher-than-median task load, as measured by the NASA-TLX survey. However in the case of attending surgeons the opposite tendency was observed, namely higher activations in the lower v higher task loaded subjects. We found that response was greater in the left PFC of students particularly near the dorso- and ventrolateral areas. We quantified the ability of PFC activation to predict the differences in skill and task load using machine learning while focussing on the effects of NIRS channel separation distance on the results. Our results showed that the classification of skill level and subjective task load could be predicted based on PFC activation with an accuracy of nearly 90%. Our finding shows that there is sufficient information available in the optical signals to make accurate predictions about the surgeons' subjective experiences and skill levels. The high accuracy of results is encouraging and suggest the integration of the strategy developed in this study as a promising approach to design automated, more accurate and objective evaluation methods.

Project description:Introduction: The objective of this study was to determine if use of an automated irrigation pump (AIP) during ureteroscopy (URS) and percutaneous nephrolithotomy (PCNL) affects circulating nurse labor, irrigation-related issues, and surgeon and nurse satisfaction when compared to manual hand pump (HP) irrigation. Methods: Eighty consecutive adult patients undergoing unilateral URS or PCNL were prospectively randomized to irrigation with the HP or AIP. Preoperative pump setup time, intraoperative pump maintenance time, total pump time (setup+maintenance), and the number of irrigation-related concerns verbalized by the surgeon intraoperatively were recorded; postoperatively, surgeons and nurses rated their satisfaction with the irrigation system (1 = highly dissatisfied to 10 = highly satisfied). Results: Eighty patients were enrolled (39 AIP and 41 HP); 51 patients underwent URS and 29 patients underwent PCNL. On univariate analysis, the AIP resulted in a significantly reduced total pump time for URS (2.9 vs 5.9 minutes) and PCNL (4.6 vs 33.9 minutes; p < 0.001). The number of irrigation-related concerns was significantly lower in the AIP group during URS (1.2 vs 2.8, p < 0.001), but not during PCNL (1.9 vs 4.0, p = 0.07). The AIP was associated with significantly higher nurse satisfaction during URS (9.2/10 vs 6.5/10, p < 0.001) and PCNL (9.4/10 vs 4.4/10, p = 0.001). There was no significant association between pump type and surgeon satisfaction. On multivariate analysis of URS cases controlling for body mass index and number of stones, use of the AIP was a predictor of total pump time <5 minutes (odds ratio 25.8, 95% confidence interval [CI] 4.0-165.4; p < 0.001) and favorable (8-10/10) nurse satisfaction rating (odds ratio 25.4, 95% CI 4.1-164.0; p < 0.001). Operative time, stone-free rate, and liters of irrigant used with the HP and AIP were similar. Conclusions: During URS and PCNL, the AIP was associated with a significant reduction in irrigation pump time and higher nurse satisfaction.

Project description:PurposeSurgeons' skill in the operating room is a major determinant of patient outcomes. Assessment of surgeons' skill is necessary to improve patient outcomes and quality of care through surgical training and coaching. Methods for video-based assessment of surgical skill can provide objective and efficient tools for surgeons. Our work introduces a new method based on attention mechanisms and provides a comprehensive comparative analysis of state-of-the-art methods for video-based assessment of surgical skill in the operating room.MethodsUsing a dataset of 99 videos of capsulorhexis, a critical step in cataract surgery, we evaluated image feature-based methods and two deep learning methods to assess skill using RGB videos. In the first method, we predict instrument tips as keypoints and predict surgical skill using temporal convolutional neural networks. In the second method, we propose a frame-wise encoder (2D convolutional neural network) followed by a temporal model (recurrent neural network), both of which are augmented by visual attention mechanisms. We computed the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, and predictive values through fivefold cross-validation.ResultsTo classify a binary skill label (expert vs. novice), the range of AUC estimates was 0.49 (95% confidence interval; CI = 0.37 to 0.60) to 0.76 (95% CI = 0.66 to 0.85) for image feature-based methods. The sensitivity and specificity were consistently high for none of the methods. For the deep learning methods, the AUC was 0.79 (95% CI = 0.70 to 0.88) using keypoints alone, 0.78 (95% CI = 0.69 to 0.88) and 0.75 (95% CI = 0.65 to 0.85) with and without attention mechanisms, respectively.ConclusionDeep learning methods are necessary for video-based assessment of surgical skill in the operating room. Attention mechanisms improved discrimination ability of the network. Our findings should be evaluated for external validity in other datasets.

Project description:Need to assess the skill of ecosystem modelsAccelerated changes to global ecosystems call for holistic and integrated analyses of past, present and future states under various pressures to adequately understand current and projected future system states. Ecosystem models can inform management of human activities in a complex and changing environment, but are these models reliable? Ensuring that models are reliable for addressing management questions requires evaluating their skill in representing real-world processes and dynamics. Skill has been evaluated for just a limited set of some biophysical models. A range of skill assessment methods have been reviewed but skill assessment of full marine ecosystem models has not yet been attempted.Northeast us atlantis marine ecosystem modelWe assessed the skill of the Northeast U.S. (NEUS) Atlantis marine ecosystem model by comparing 10-year model forecasts with observed data. Model forecast performance was compared to that obtained from a 40-year hindcast. Multiple metrics (average absolute error, root mean squared error, modeling efficiency, and Spearman rank correlation), and a suite of time-series (species biomass, fisheries landings, and ecosystem indicators) were used to adequately measure model skill. Overall, the NEUS model performed above average and thus better than expected for the key species that had been the focus of the model tuning. Model forecast skill was comparable to the hindcast skill, showing that model performance does not degenerate in a 10-year forecast mode, an important characteristic for an end-to-end ecosystem model to be useful for strategic management purposes.Skill assessment is both possible and advisableWe identify best-practice approaches for end-to-end ecosystem model skill assessment that would improve both operational use of other ecosystem models and future model development. We show that it is possible to not only assess the skill of a complicated marine ecosystem model, but that it is necessary do so to instill confidence in model results and encourage their use for strategic management. Our methods are applicable to any type of predictive model, and should be considered for use in fields outside ecology (e.g. economics, climate change, and risk assessment).

Dataset Information

Subjective vs Computerized Assessment of Surgeon Skill Level During Mastoidectomy.

Publications

Subjective vs Computerized Assessment of Surgeon Skill Level During Mastoidectomy.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets