Project description:IntroductionSince the advent of medical education systems, managing high-stakes exams has been a top priority and challenge for all policymakers. However, considering machine learning (ML) techniques as a replacement for medical licensing examinations, particularly during crises such as the COVID-19 outbreak, could be an effective solution. This study uses ML models to develop a framework for predicting medical students' performance on high-stakes exams, such as the Comprehensive Medical Basic Sciences Examination (CMBSE).Material and methodsPrediction of students' status and score on high-stakes examinations faces several challenges, including an imbalanced number of failing and passing students, a large number of heterogeneous and complex features, and the need to identify at-risk and top-performing students. In this study, two major categories of ML approaches are compared: first, classic models (logistic regression (LR), support vector machine (SVM), and k-nearest neighbors (KNN)), and second, ensemble models (voting, bagging (BG), random forests (RF), adaptive boosting (ADA), extreme gradient boosting (XGB), and stacking).ResultsTo evaluate the models' discrimination ability, they are assessed using a real dataset containing information on medical students over a five-year period (n = 1005). The findings indicate that ensemble ML models demonstrate optimal performance in predicting CMBSE status (RF and stacking). Similarly, among the classic regressors, LR exhibited the highest root-mean-square deviation (RMSD) (0.134) and coefficient of determination (R2) (0.62), whereas the RF model had the highest RMSD (0.077) and R2 (0.80) overall. Furthermore, Anatomical Sciences, Biochemistry, Parasitology, and Entomology grade point average (GPA) and grades demonstrated the strongest positive correlation with the outcomes.ConclusionComparing classic and ensemble ML models revealed that ensemble models are superior to classic models. Therefore, the presented framework could be considered a suitable alternative for the CMBSE and other comparable medical licensing examinations.

Project description:BackgroundFailure rates in postgraduate examinations are often high and many candidates therefore retake examinations on several or even many times. Little, however, is known about how candidates perform across those multiple attempts. A key theoretical question to be resolved is whether candidates pass at a resit because they have got better, having acquired more knowledge or skills, or whether they have got lucky, chance helping them to get over the pass mark. In the UK, the issue of resits has become of particular interest since the General Medical Council issued a consultation and is considering limiting the number of attempts candidates may make at examinations.MethodsSince 1999 the examination for Membership of the Royal Colleges of Physicians of the United Kingdom (MRCP(UK)) has imposed no limit on the number of attempts candidates can make at its Part 1, Part 2 or PACES (Clinical) examination. The present study examined the performance of candidates on the examinations from 2002/2003 to 2010, during which time the examination structure has been stable. Data were available for 70,856 attempts at Part 1 by 39,335 candidates, 37,654 attempts at Part 2 by 23,637 candidates and 40,303 attempts at PACES by 21,270 candidates, with the maximum number of attempts being 26, 21 and 14, respectively. The results were analyzed using multilevel modelling, fitting negative exponential growth curves to individual candidate performance.ResultsThe number of candidates taking the assessment falls exponentially at each attempt. Performance improves across attempts, with evidence in the Part 1 examination that candidates are still improving up to the tenth attempt, with a similar improvement up to the fourth attempt in Part 2 and the sixth attempt at PACES. Random effects modelling shows that candidates begin at a starting level, with performance increasing by a smaller amount at each attempt, with evidence of a maximum, asymptotic level for candidates, and candidates showing variation in starting level, rate of improvement and maximum level. Modelling longitudinal performance across the three diets (sittings) shows that the starting level at Part 1 predicts starting level at both Part 2 and PACES, and the rate of improvement at Part 1 also predicts the starting level at Part 2 and PACES.ConclusionCandidates continue to show evidence of true improvement in performance up to at least the tenth attempt at MRCP(UK) Part 1, although there are individual differences in the starting level, the rate of improvement and the maximum level that can be achieved. Such findings provide little support for arguments that candidates should only be allowed a fixed number of attempts at an examination. However, unlimited numbers of attempts are also difficult to justify because of the inevitable and ever increasing role that luck must play with increasing numbers of resits, so that the issue of multiple attempts might be better addressed by tackling the difficult question of how a pass mark should increase with each attempt at an exam.

Project description:BackgroundIncorporating emerging knowledge into Emergency Medical Service (EMS) competency assessments is critical to reflect current evidence-based out-of-hospital care. However, a standardized approach is needed to incorporate new evidence into EMS competency assessments because of the rapid pace of knowledge generation.ObjectiveThe objective was to develop a framework to evaluate and integrate new source material into EMS competency assessments.MethodsThe National Registry of Emergency Medical Technicians (National Registry) and the Prehospital Guidelines Consortium (PGC) convened a panel of experts. A Delphi method, consisting of virtual meetings and electronic surveys, was used to develop a Table of Evidence matrix that defines sources of EMS evidence. In Round One, participants listed all potential sources of evidence available to inform EMS education. In Round Two, participants categorized these sources into: (a) levels of evidence quality; and (b) type of source material. In Round Three, the panel revised a proposed Table of Evidence. Finally, in Round Four, participants provided recommendations on how each source should be incorporated into competency assessments depending on type and quality. Descriptive statistics were calculated with qualitative analyses conducted by two independent reviewers and a third arbitrator.ResultsIn Round One, 24 sources of evidence were identified. In Round Two, these were classified into high- (n = 4), medium- (n = 15), and low-quality (n = 5) of evidence, followed by categorization by purpose into providing recommendations (n = 10), primary research (n = 7), and educational content (n = 7). In Round Three, the Table of Evidence was revised based on participant feedback. In Round Four, the panel developed a tiered system of evidence integration from immediate incorporation of high-quality sources to more stringent requirements for lower-quality sources.ConclusionThe Table of Evidence provides a framework for the rapid and standardized incorporation of new source material into EMS competency assessments. Future goals are to evaluate the application of the Table of Evidence framework in initial and continued competency assessments.

Dataset Information

Using technology to prevent fraud in high stakes national school examinations: Evidence from Indonesia☆☆☆

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets