Dataset Information

Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery.

ABSTRACT:

Background

Simulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation.

Methods

Eight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of trainers. Trainee performance was scored using an equally weighted Objective Structured Assessment of Technical Skills (OSATS) tool and a modified Procedure-Based Assessment (PBA) tool. Validity evidence was analyzed in accordance with Messick's validity framework. Both 'junior' (ST2-ST4) and 'senior' trainees (ST 5-ST8) were included to allow for comparative analysis.

Results

Thirteen trainees were assessed by ten assessors across eight stations. Inter-station reliability was high (α = 0.81), and inter-rater reliability was acceptable (inter-class correlation coefficient 0.77). A significant difference in mean station score was observed between junior and senior trainees (44.82 vs 58.18, p = .004), while overall mean scores were moderately correlated with increasing training year (rs = .74, p = .004, Kendall's tau-b .57, p = 0.009). A pass-fail score generated using borderline regression methodology resulted in all 'senior' trainees passing and 4/6 of junior trainees failing the assessment.

Conclusion

This study reports validity evidence for a novel simulation-based assessment, designed to assess the operative competence of higher specialist trainees in general surgery.

SUBMITTER: Toale C

PROVIDER: S-EPMC11362445 | biostudies-literature | 2024 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery.

Toale Conor C Morris Marie M Roche Adam A Voborsky Miroslav M Traynor Oscar O Kavanagh Dara D

Surgical endoscopy 20240717 9

<h4>Background</h4>Simulation is increasingly being explored as an assessment modality. This study sought to develop and collate validity evidence for a novel simulation-based assessment of operative competence. We describe the approach to assessment design, development, pilot testing, and validity investigation.<h4>Methods</h4>Eight procedural stations were generated using both virtual reality and bio-hybrid models. Content was identified from a previously conducted Delphi consensus study of tr ...[more]

PMID: 39020120

Similar Datasets

Project description:ImportanceThe need for trainee sex equality within surgical training has resulted in an appraisal of the training experience in the New Zealand general surgery training program.ObjectiveTo investigate the association between trainee sex and surgical autonomy in the operating room in the New Zealand general surgery training program.Design, setting, and participantsRetrospective cohort study conducted from December 10, 2012, to December 10, 2017, examining all endoscopic, major, and minor procedures performed by all New Zealand general surgery trainees in every training hospital in New Zealand.Main outcomes and measuresThe primary outcome was the level of meaningful autonomy by each New Zealand general surgery trainee (ie, trainee as primary operator without the surgeon mentor scrubbed for the case). Outcomes were compared using multivariable analysis.ResultsThis study included 120 New Zealand general surgery trainees (42 women [35%] and 78 men [65%]) who were analyzed over 279.5 trainee-years (88.5 trainee-years for women and 191.0 trainee-years for men). Included were 119 380 general surgery procedures (17 465 endoscopic, 56 964 major, and 44 951 minor) in 18 hospitals. By the end of the 5-year training program, female trainees had a lower cumulative mean autonomous caseload than male trainees for endoscopic (284.0 [95% CI, 207.0-361.0] vs 352.2 [95% CI, 282.9-421.6], P = .03), major (139.9 [95% CI, 76.7-203.2] vs 198.1 [95% CI, 142.3-254.0], P = .02), and minor (456.3 [95% CI, 394.8-517.9] vs 519.9 [95% CI, 465.6-574.2], P = .007) procedures.Conclusions and relevanceAfter accounting for differences among trainees, hospital type, number of female and male surgeon mentors at each hospital, and trainee seniority, female trainees performed fewer cases with meaningful autonomy compared with male trainees. These findings support the need for pragmatic solutions to address this bias and further investigations on mechanisms contributing to discrepancies.

Project description:ObjectiveTo aid the shortage of general practitioners (GPs) in Germany, since 2011 medical specialists from other fields may switch specialisation by undergoing a slim-lined training programme (lateral entry) into general practice (GP). Available published qualitative and quantitative data on lateral entrants (LEs) is scarce. Aim of the study was an explorative analysis of LEs in the competence centre for postgraduate medical education Baden-Wuerttemberg (KWBW).MethodsIn 2016, a multicentric cohort study was initiated including all GP trainees entering the KWBW. Data from 2016 to 2022 was included (T0). A follow-up with graduates from the KWBW was performed once in 2023 (T1). Surveys at T0 and T1 were self-developed, piloted, and included questions on social demography, education, qualification, current training, working environment and professional plans. Dataset was analysed exploratively.ResultsAt T0, 884 GP trainees participated (response 95.2%). 23.8% of participants were LEs (N=210). Most LEs were specialists from anaesthesiology (34.8%), internal medicine (28.5%) and surgery (20.6%). LEs had been working in their previous specialty for a median of 3 years before starting GP-training. At T0, LEs were seven years older than their colleagues (p<0,001). The proportion of men among LEs was higher (34.3% vs 25.1%, p=0,009). LEs were more decisive to open their own GP practice (p=0,009). At T1, 48 LEs participated of which 92% were practising GPs (N=36). 64,5% considered themselves rural GPs and work in communities <20.000 people (N=36). LEs mainly choose GP because of its specific way of working, job dissatisfaction and personal motives such as opportunity to start a practice. Most LEs strongly agreed that they would switch to GP again.DiscussionLEs are a relevant party among GP trainees. Many LEs become self-employed and contribute essentially to providing primary care. Lateral entry attracts specialists, also from rural areas, seeking for professional satisfaction. This is why LEs should follow their new path within a regional competence centre providing GP specific courses, mentoring and a professional network.ConclusionsLEs graduating from KWBW have become an important pillar of primary care in Baden-Wuerttemberg.

Project description:ImportanceUnderstanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education.ObjectiveTo evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees.Design, setting, and participantsThis case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US. The study included ratings for 2605 trainees from 1884 attending surgeon raters. Analyses were conducted between September 2021 and December 2021 using bayesian generalized linear mixed-effects models and marginal predicted probabilities.ExposuresLongitudinal SIMPL ratings.Main outcomes and measuresPerformance expectations for 193 unique general surgery procedures based on an individual trainee's prior successful ratings for a procedure, clinical year of training, and month of the academic year.ResultsUsing 63 248 SIMPL ratings, the association between prior and future performance was positive (β, 0.13; 95% credible interval [CrI], 0.12-0.15). The largest source of variation was postgraduate year (α, 3.15; 95% CrI, 1.66-6.03), with rater (α, 1.69; 95% CrI, 1.60-1.78), procedure (α, 1.35; 95% CrI, 1.22-1.51), case complexity (α, 1.30; 95% CrI, 0.42-3.66), and trainee (α, 0.99; 95% CrI, 0.94-1.04) accounting for significant variation in practice ready ratings. After marginalizing overcomplexity and trainee and holding rater constant, mean predicted probabilities had strong overall discrimination (area under the receiver operating characteristic curve, 0.81) and were well calibrated.Conclusions and relevanceIn this study, prior performance was associated with future performance. This association, combined with an overall modeling strategy that accounted for various facets of an assessment task, may offer a strategy for quantifying competence as performance expectations.

Dataset Information

Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery.

Background

Methods

Results

Conclusion

Publications

Development and validation of a simulation-based assessment of operative competence for higher specialist trainees in general surgery.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets