Dataset Information

Predictive validity of A-level grades and teacher-predicted grades in UK medical school applicants: a retrospective analysis of administrative data in a time of COVID-19.

ABSTRACT:

Objectives

To compare in UK medical students the predictive validity of attained A-level grades and teacher-predicted A levels for undergraduate and postgraduate outcomes. Teacher-predicted A-level grades are a plausible proxy for the teacher-estimated grades that replaced UK examinations in 2020 as a result of the COVID-19 pandemic. The study also models the likely future consequences for UK medical schools of replacing public A-level examination grades with teacher-predicted grades.

Design

Longitudinal observational study using UK Medical Education Database data.

Setting

UK medical education and training.

Participants

Dataset 1: 81 202 medical school applicants in 2010-2018 with predicted and attained A-level grades. Dataset 2: 22 150 18-year-old medical school applicants in 2010-2014 with predicted and attained A-level grades, of whom 12 600 had medical school assessment outcomes and 1340 had postgraduate outcomes available.

Outcome measures

Undergraduate and postgraduate medical examination results in relation to attained and teacher-predicted A-level results.

Results

Dataset 1: teacher-predicted grades were accurate for 48.8% of A levels, overpredicted in 44.7% of cases and underpredicted in 6.5% of cases. Dataset 2: undergraduate and postgraduate outcomes correlated significantly better with attained than with teacher-predicted A-level grades. Modelling suggests that using teacher-estimated grades instead of attained grades will mean that 2020 entrants are more likely to underattain compared with previous years, 13% more gaining the equivalent of the lowest performance decile and 16% fewer reaching the equivalent of the current top decile, with knock-on effects for postgraduate training.

Conclusions

The replacement of attained A-level examination grades with teacher-estimated grades as a result of the COVID-19 pandemic may result in 2020 medical school entrants having somewhat lower academic performance compared with previous years. Medical schools may need to consider additional teaching for entrants who are struggling or who might need extra support for missed aspects of A-level teaching.

SUBMITTER: McManus IC

PROVIDER: S-EPMC8678544 | biostudies-literature | 2021 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Predictive validity of A-level grades and teacher-predicted grades in UK medical school applicants: a retrospective analysis of administrative data in a time of COVID-19.

McManus I C IC Woolf Katherine K Harrison David D Tiffin Paul A PA Paton Lewis W LW Cheung Kevin Yet Fong KYF Smith Daniel T DT

BMJ open 20211216 12

<h4>Objectives</h4>To compare in UK medical students the predictive validity of attained A-level grades and teacher-predicted A levels for undergraduate and postgraduate outcomes. Teacher-predicted A-level grades are a plausible proxy for the teacher-estimated grades that replaced UK examinations in 2020 as a result of the COVID-19 pandemic. The study also models the likely future consequences for UK medical schools of replacing public A-level examination grades with teacher-predicted grades.<h4 ...[more]

PMID: 34916308

Similar Datasets

Project description:BackgroundAlthough used widely, there is limited evidence of the BioMedical Admissions Test's (BMAT) predictive validity and incremental validity over prior educational attainment (PEA). We investigated BMAT's predictive and incremental validity for performance in two undergraduate medical schools, Imperial College School of Medicine (ICSM), UK, and Lee Kong Chian School of Medicine (LKCMedicine), Singapore. Our secondary goal was to compare the evidence collected with published evidence relating to comparable tools.MethodsThis was a retrospective cohort study of four ICSM (1188 students, entering 2010-2013) and three LKCMedicine cohorts (222 students, 2013-2015). We investigated associations between BMAT Section 1 ('Thinking Skills'), Section 2 ('Scientific Knowledge and Applications') and Section 3a ('Writing Task') scores, with written and clinical assessment performance across all programme years. Incremental validity was investigated over PEA (A-levels) in a subset of ICSM students.ResultsWhen BMAT sections were investigated independently, Section 2 scores predicted performance on all written assessments in both institutions with mainly small effect sizes (standardised coefficient ranges: ICSM: 0.08-0.19; LKCMedicine: 0.22-0.36). Section 1 scores predicted Years 5 and 6 written assessment performance at ICSM (0.09-0.14) but nothing at LKCMedicine. Section 3a scores only predicted Year 5 clinical assessment performance at ICSM with a coefficient <0.1. There were no positive associations with standardised coefficients >0.1 between BMAT performance and clinical assessment performance. Multivariable regressions confirmed that Section 2 scores were the most predictive. We found no clear evidence of incremental validity for any BMAT section scores over A-level grades.DiscussionSchools who wish to assess scientific knowledge independently of A-levels may find BMAT Section 2 useful. Comparison with previous studies indicates that, overall, BMAT seems less useful than comparable tools. Larger scale studies are needed. Broader questions regarding why institutions adopt certain admissions tests, including those with little evidence, need consideration.

Project description:BackgroundStructural and interpersonal anti-Indigenous racism is prevalent in Canadian healthcare. The Truth and Reconciliation Commission calls on medical schools to address anti-Indigenous bias in students. We measured the prevalence of interpersonal anti-Indigenous bias among medical school applicants to understand how the medical school selection process selects for or against students with high levels of bias.MethodsAll applicants to a single university in the 2020-2021 admissions cycle were invited to participate. Explicit anti-Indigenous bias was measured using two sliding scale thermometers. The first asked how participants felt about Indigenous people (from 0, indicating 'cold/unfavourable' to 100, indicating 'warm/favourable') and the second asked whether participants preferred white (scored 100) or Indigenous people (scored 0). Participants then completed an implicit association test examining preferences for European or Indigenous faces (negative time latencies suggest preference for European faces). Explicit and implicit anti-Indigenous biases were compared by applicant demographics (including gender and racial identity), application status (offered an interview, offered admission, accepted a position), and compared to undergraduate medical and mathematics students.ResultsThere were 595 applicant respondents (32.4% response rate, 64.2% cisgender women, 55.3% white). Applicants felt warmly toward Indigenous people (median 96 (IQR 80-100)), had no explicit preference for white or Indigenous people (median 50 (IQR 37-55), and had mild implicit preference for European faces (- 0.22 ms (IQR -0.54, 0.08 ms)). There were demographic differences associated with measures of explicit and implicit bias. Applicants who were offered admission had warmer feelings toward Indigenous people and greater preference for Indigenous people compared to those were not successful.ConclusionsMedical school applicants did not have strong interpersonal explicit and implicit anti-Indigenous biases. Outlier participants with strong biases were not offered interviews or admission to medical school.

Project description:BACKGROUND:The association between perfectionism and depression in the medical profession can ultimately influence physicians' performance negatively. In medical students, especially maladaptive perfectionism is connected with distress and lower academic performance. The expression of perfectionism and symptoms of depression at the time of medical school application is not known. Therefore, we explored perfectionism and symptoms of depression in participants of multiple mini-interviews for medical school admission and investigated possible differences between applicants who were eventually admitted or rejected. METHODS:After the multiple mini-interviews admission procedure at Hamburg Medical School in August 2018, 146 applicants filled out a questionnaire including sociodemographic data and the following validated instruments: Multidimensional Perfectionism Scale by Hewitt and Flett (MPS-H), Multidimensional Perfectionism Scale by Frost (MPS-F), Patient Health Questionnaire (PHQ-9), Generalized Anxiety Disorder Scale (GAD-7), and a 10-item version of the Big Five Inventory (BFI-10). The two groups of admitted and rejected applicants were compared and the correlation between symptoms of depression and perfectionism further explored. RESULTS:The admitted applicants were significantly more extrovert and had lower depression scores compared to the rejected applicants. In both groups, the composite scales of Adaptive Perfectionism (r = .21, p = .011) and Maladaptive Perfectionism (r = .43, p < .001) as well as their components correlated significantly with the PHQ-9 results. Maladaptive Perfectionism accounted for about 18% of variance in the PHQ-9 score. CONCLUSIONS:Rejected medical school applicants who participated in a multiple mini-interviews admission procedure showed higher levels of depression symptoms than admitted applicants. The degree of depressive symptoms can be partly explained by Maladaptive Perfectionism scores. Since coping in medical school and in postgraduate medical education require robust mental health, perfectionism questionnaires could be an additional tool in medical school selection processes.

Project description:ObjectiveDescribe the experiences and views of medical applicants from diverse social backgrounds following the closure of schools and universities and the cancellation of public examinations in the UK due to COVID-19.DesignCross-sectional questionnaire study, part of the longitudinal UK Medical Applicant Cohort Study (UKMACS).SettingUK medical school admissions in 2020.Participants2887 participants completed an online questionnaire from 8 April to 22 April 2020. Eligible participants had registered to take the University Clinical Admissions Test in 2019 and agreed to be invited to take part, or had completed a previous UKMACS questionnaire, had been seriously considering applying to medicine in the UK for entry in 2020, and were UK residents.Main outcome measuresViews on calculated grades, views on medical school admissions and teaching in 2020 and 2021, reported experiences of education during the national lockdown.ResultsRespondents were concerned about the calculated grades that replaced A-level examinations: female and Black Asian and Minority Ethnic applicants felt teachers would find it difficult to grade and rank students accurately, and applicants from non-selective state schools and living in deprived areas had concerns about the standardisation process. Calculated grades were generally not considered fair enough to use in selection, but were considered fair enough to use in combination with other measures including interview and aptitude test scores. Respondents from non-selective state (public) schools reported less access to educational resources compared with private/selective school pupils, less online teaching in real time and less time studying during lockdown.ConclusionsThe COVID-19 pandemic has and will have significant and long-term impacts on the selection, education and performance of our medical workforce. It is important that the views and experiences of applicants from diverse backgrounds are considered in decisions affecting their future and the future of the profession.

Project description:BackgroundThe medical specialities chosen by doctors for their careers play an important part in the workforce planning of health-care services. However, there is little theoretical understanding of how different medical specialities are perceived or how choices are made, despite there being much work in general on this topic in occupational psychology, which is influenced by Holland's RIASEC (Realistic-Investigative-Artistic-Social-Enterprising-Conventional) typology of careers, and Gottfredson's model of circumscription and compromise. In this study, we use three large-scale cohorts of medical students to produce maps of medical careers.MethodsInformation on between 24 and 28 specialities was collected in three UK cohorts of medical students (1981, 1986 and 1991 entry), in applicants (1981 and 1986 cohorts, N = 1135 and 2032) or entrants (1991 cohort, N = 2973) and in final-year students (N = 330, 376, and 1437). Mapping used Individual Differences Scaling (INDSCAL) on sub-groups broken down by age and sex. The method was validated in a population sample using a full range of careers, and demonstrating that the RIASEC structure could be extracted.ResultsMedical specialities in each cohort, at application and in the final-year, were well represented by a two-dimensional space. The representations showed a close similarity to Holland's RIASEC typology, with the main orthogonal dimensions appearing similar to Prediger's derived orthogonal dimensions of 'Things-People' and 'Data-Ideas'.ConclusionsThere are close parallels between Holland's general typology of careers, and the structure we have found in medical careers. Medical specialities typical of Holland's six RIASEC categories are Surgery (Realistic), Hospital Medicine (Investigative), Psychiatry (Artistic), Public Health (Social), Administrative Medicine (Enterprising), and Laboratory Medicine (Conventional). The homology between medical careers and RIASEC may mean that the map can be used as the basis for understanding career choice, and for providing career counselling.

Project description:BackgroundKnowledge in natural sciences generally predicts study performance in the first two years of the medical curriculum. In order to reduce delay and dropout in the preclinical years, Hamburg Medical School decided to develop a natural science test (HAM-Nat) for student selection. In the present study, two different approaches to scale construction are presented: a unidimensional scale and a scale composed of three subject specific dimensions. Their psychometric properties and relations to academic success are compared.Methods334 first year medical students of the 2006 cohort responded to 52 multiple choice items from biology, physics, and chemistry. For the construction of scales we generated two random subsamples, one for development and one for validation. In the development sample, unidimensional item sets were extracted from the item pool by means of weighted least squares (WLS) factor analysis, and subsequently fitted to the Rasch model. In the validation sample, the scales were subjected to confirmatory factor analysis and, again, Rasch modelling. The outcome measure was academic success after two years.ResultsAlthough the correlational structure within the item set is weak, a unidimensional scale could be fitted to the Rasch model. However, psychometric properties of this scale deteriorated in the validation sample. A model with three highly correlated subject specific factors performed better. All summary scales predicted academic success with an odds ratio of about 2.0. Prediction was independent of high school grades and there was a slight tendency for prediction to be better in females than in males.ConclusionsA model separating biology, physics, and chemistry into different Rasch scales seems to be more suitable for item bank development than a unidimensional model, even when these scales are highly correlated and enter into a global score. When such a combination scale is used to select the upper quartile of applicants, the proportion of successful completion of the curriculum after two years is expected to rise substantially.

Dataset Information

Predictive validity of A-level grades and teacher-predicted grades in UK medical school applicants: a retrospective analysis of administrative data in a time of COVID-19.

Objectives

Design

Setting

Participants

Outcome measures

Results

Conclusions

Publications

Predictive validity of A-level grades and teacher-predicted grades in UK medical school applicants: a retrospective analysis of administrative data in a time of COVID-19.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets