Dataset Information

A Retrospective Cohort Analysis Comparing Analytic and Holistic Marking Rubrics in Medical Research Education.

ABSTRACT:

Objectives

The use of analytic rubrics in assessing and grading students' performance has become more prominent among instructors due to its reliability and validity in ensuring consistency in student evaluation. However, there is limited evidence demonstrating the consistency of examiner judgments between analytic marking rubrics and holistic marking rubrics.

Methods

Therefore, we aimed to compare the consistency of marks given using holistic marking methods and analytic rubrics at an Australian university by analyzing the mean mark differences and number of adjudications between two rubric types as well as the inter-rater reliability between two assessors. We analyzed all scores for project manuscripts between 2016 and 2021 for Honours medical students. We compared the mean mark differences graded using the Kruskal-Wallis test and Welch t-test. We used chi-squared tests to compare the frequency of adjudications for each rubric type. We assessed interrater reliability by comparing the marks between the two examiners utilizing Pearson correlation.

Results

We found that analytic rubrics have lower mean mark differences and fewer adjudicators are required. We showed a strong positive association between the consistency of marks given and the use of analytic rubrics when compared to holistic marking. Pearson correlation showed a low but stronger correlation between marks awarded by the two assessors when analytic rubrics were used (r = 0.36), compared to holistic marking rubrics (r = 0.24).

Conclusions

Our findings suggest that the use of analytic rubrics may increase the consistency and reliability between two independent examiners in marking medical students' work.

SUBMITTER: Yeo SW

PROVIDER: S-EPMC11359436 | biostudies-literature | 2024 Jan-Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A Retrospective Cohort Analysis Comparing Analytic and Holistic Marking Rubrics in Medical Research Education.

Yeo Siew Wan SW Signorelli Christina C Vo Khanh K Smith Greg G

Journal of medical education and curricular development 20240101

<h4>Objectives</h4>The use of analytic rubrics in assessing and grading students' performance has become more prominent among instructors due to its reliability and validity in ensuring consistency in student evaluation. However, there is limited evidence demonstrating the consistency of examiner judgments between analytic marking rubrics and holistic marking rubrics.<h4>Methods</h4>Therefore, we aimed to compare the consistency of marks given using holistic marking methods and analytic rubrics ...[more]

PMID: 39211294

Similar Datasets

Project description:BackgroundResearch abstracts are submitted for presentation at scientific conferences; however, criteria for judging abstracts are variable. We sought to develop two rigorous abstract scoring rubrics for education research submissions reporting (1) quantitative data and (2) qualitative data and then to collect validity evidence to support score interpretation.MethodsWe used a modified Delphi method to achieve expert consensus for scoring rubric items to optimize content validity. Eight education research experts participated in two separate modified Delphi processes, one to generate quantitative research items and one for qualitative. Modifications were made between rounds based on item scores and expert feedback. Homogeneity of ratings in the Delphi process was calculated using Cronbach's alpha, with increasing homogeneity considered an indication of consensus. Rubrics were piloted by scoring abstracts from 22 quantitative publications from AEM Education and Training "Critical Appraisal of Emergency Medicine Education Research" (11 highlighted for excellent methodology and 11 that were not) and 10 qualitative publications (five highlighted for excellent methodology and five that were not). Intraclass correlation coefficient (ICC) estimates of reliability were calculated.ResultsEach rubric required three rounds of a modified Delphi process. The resulting quantitative rubric contained nine items: quality of objectives, appropriateness of methods, outcomes, data analysis, generalizability, importance to medical education, innovation, quality of writing, and strength of conclusions (Cronbach's α for the third round = 0.922, ICC for total scores during piloting = 0.893). The resulting qualitative rubric contained seven items: quality of study aims, general methods, data collection, sampling, data analysis, writing quality, and strength of conclusions (Cronbach's α for the third round = 0.913, ICC for the total scores during piloting = 0.788).ConclusionWe developed scoring rubrics to assess quality in quantitative and qualitative medical education research abstracts to aid in selection for presentation at scientific meetings. Our tools demonstrated high reliability.

Project description:BackgroundGenome cohort studies are used to analyze interactions between genetic and environmental factors, providing valuable information for personalized healthcare. Large-scale and long-term cohort studies require a number of specially trained personnel, of whom those involved in obtaining informed consent play a vital role, especially during the initial phase of such studies. The Japanese Society of Human Genetics (JSHG) previously established a certification system for genome medical research coordinators (GMRCs) responsible for obtaining written consent via face-to-face explanation. Meanwhile, in the Tohoku Medical Megabank Organization (ToMMo), GMRCs are expected to play important roles not only in obtaining informed consent and conducting various assessments, but also in communicating with participants throughout the long-term follow-up. Based on the JSHG program, we therefore developed a specific education and training program for ToMMo GMRCs consisting of 17 lectures, one practical training session on the informed consent procedure, and written and interview examinations. Re-education workshops aimed at self-improvement are also carried out following certification. In this study, we evaluated the education and training program in terms of overall understanding, usefulness, and satisfaction using an anonymous questionnaire.MethodsAn anonymous questionnaire addressing each aspect of the education and training program (understanding, usefulness, and satisfaction) was distributed among 152 qualified ToMMo GMRCs. Responses were received from 94 participants (61.8%).ResultsThere was a significant association between the level of overall understanding of lectures and medical qualification (nurse or clinical laboratory technologist), but not with age or educational background. The level of understanding and overall usefulness were lower in sessions related to genetics and epidemiology than those dealing with ToMMo practices. In the re-education workshops, GMRCs showed a preference for and hoped to learn more about both background knowledge and research progress in the ToMMo.ConclusionsThe results of our questionnaire suggest that not all ToMMo GMRCs are able to understand everything during the initial education and training program, especially in terms of genomic medicine. Continuous re-education is therefore vital in improving knowledge, skills and motivation, and preparing GMRCs for a specialist role in community-based personalized healthcare.

Dataset Information

A Retrospective Cohort Analysis Comparing Analytic and Holistic Marking Rubrics in Medical Research Education.

Objectives

Methods

Results

Conclusions

Publications

A Retrospective Cohort Analysis Comparing Analytic and Holistic Marking Rubrics in Medical Research Education.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets