Dataset Information

Grading reflective essays: the reliability of a newly developed tool- GRE-9.

ABSTRACT:

Background

The main objective of this study is the development of a short reliable easy-to-use assessment tool in the aim of providing feedback to the reflective writings of medical students and residents.

Methods

This study took place in a major tertiary academic medical center in Beirut, Lebanon. Seventy-seven reflective essays written by 18 residents in the department of Family Medicine at the American University of Beirut Medical Center (AUBMC) were graded by 3 raters using the newly developed scale to assess the scale reliability. Following a comprehensive search and analysis of the literature, and based on their experience in reflective grading, the authors developed a concise 9-item scale to grade reflective essays through repeated cycles of development and analysis as well as the determination of the inter-rater reliability (IRR) using intra-class correlation coefficients (ICC) and Krippendorff's Alpha.

Results

The inter-rater reliability of the new scale ranges from moderate to substantial with ICC of 0.78, 95% CI 0.64-0.86, p < 0.01 and Krippendorff's Alpha was 0.49.

Conclusions

The newly developed scale, GRE-9, is a short, concise, easy-to-use reliable grading tool for reflective essays that has demonstrated moderate to substantial inter-rater reliability. This will enable raters to objectively grade reflective essays and provide informed feedback to residents and students.

SUBMITTER: Makarem NN

PROVIDER: S-EPMC7520967 | biostudies-literature | 2020 Sep

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Grading reflective essays: the reliability of a newly developed tool- GRE-9.

Makarem Nisrine N NN Saab Basem R BR Maalouf Grace G Musharafieh Umayya U Naji Fadila F Rahme Diana D Brome Dayana D

BMC medical education 20200925 1

<h4>Background</h4>The main objective of this study is the development of a short reliable easy-to-use assessment tool in the aim of providing feedback to the reflective writings of medical students and residents.<h4>Methods</h4>This study took place in a major tertiary academic medical center in Beirut, Lebanon. Seventy-seven reflective essays written by 18 residents in the department of Family Medicine at the American University of Beirut Medical Center (AUBMC) were graded by 3 raters using th ...[more]

PMID: 32988408

Similar Datasets

Project description:BackgroundClinical examination of trachoma is used to justify intervention in trachoma-endemic regions. Currently, field graders are certified by determining their concordance with experienced graders using the kappa statistic. Unfortunately, trachoma grading can be highly variable and there are cases where even expert graders disagree (borderline/marginal cases). Prior work has shown that inclusion of borderline cases tends to reduce apparent agreement, as measured by kappa. Here, we confirm those results and assess performance of trainees on these borderline cases by calculating their reliability error, a measure derived from the decomposition of the Brier score.Methods and findingsWe trained 18 field graders using 200 conjunctival photographs from a community-randomized trial in Niger and assessed inter-grader agreement using kappa as well as reliability error. Three experienced graders scored each case for the presence or absence of trachomatous inflammation-follicular (TF) and trachomatous inflammation-intense (TI). A consensus grade for each case was defined as the one given by a majority of experienced graders. We classified cases into a unanimous subset if all 3 experienced graders gave the same grade. For both TF and TI grades, the mean kappa for trainees was higher on the unanimous subset; inclusion of borderline cases reduced apparent agreement by 15.7% for TF and 12.4% for TI. When we assessed the breakdown of the reliability error, we found that our trainees tended to over-call TF grades and under-call TI grades, especially in borderline cases.ConclusionsThe kappa statistic is widely used for certifying trachoma field graders. Exclusion of borderline cases, which even experienced graders disagree on, increases apparent agreement with the kappa statistic. Graders may agree less when exposed to the full spectrum of disease. Reliability error allows for the assessment of these borderline cases and can be used to refine an individual trainee's grading.

Project description:Reactive agility (RAG) and change of direction speed (CODS) are important determinants of success in football (soccer), but there is an evident lack of information on reliable and valid football-specific testing procedures which will be applicable in defining sport-specific RAG and CODS in youth players. This study evaluated reliability and construct validity of newly developed tests of football-specific RAG (FS_RAG) and CODS (FS_CODS), which involved the ball kicking football technique. Additionally, factors associated with FS_RAG and FS_CODS were evaluated. The participants were youth football players (n = 59; age: 13.40 ± 1.25 years) divided according to their age into U13 (11-12 years of age; n = 29), and U15 (13-14 years of age; n = 30) categories. Additionally, performance levels (starters [first-team] vs. non-starters [substitutes]) were observed in each age category. The dependent variables were newly developed FS_RAG and FS_CODS tests. The independent variables were sprinting capacities over 10 and 20 meters (S10M, S20M), countermovement jump (CMJ), the reactive strength index (RSI), and a generic CODS test of 20 yards (20Y). The newly developed FS_CODS and FS_RAG were observed as dependent variables. Results showed appropriate intra-testing and inter-testing reliability of the FS_RAG and FS_CODS, with somewhat better reliability of the FS_CODS (ICC=0.82 and 0.79, respectively). Additionally, better reliability was evidenced in U15 than in U13 (ICC: 0.82-0.85, and 0.78-0.80 for U15 and U13, respectively). Independent samples t-test indicated significant differences between U13 and U15 in S10 (t-test: 3.57, p < 0.001), S20M (t-test: 3.13, p < 0.001), 20Y (t-test: 4.89, p < 0.001), FS_RAG (t-test: 3.96, p < 0.001), and FS_CODS (t-test: 6.42, p < 0.001), with better performance in U15. Starters outperformed non-starters in most capacities among U13, but only in FS_RAG among U15 (t-test: 1.56, p < 0.05). Multiple regression calculations indicated nonsignificant association between independent and dependent variables in U13 (FS_CODS: 19%, FS_RAG: 21% of the explained variance, both p > 0.05), but independent variables explained significant proportion of both dependent variables in U15 (FS_CODS: 35%, FS_RAG: 33% explained variance, both p < 0.05). The study confirmed the applicability of newly developed tests in distinguishing studied age categories of players. Results indicate that superiority in all studied fitness capacities is translated into performance level in U13. Meanwhile, FS_RAG seems to be important determinant of quality in U15.

Project description:BackgroundFor the diagnosis of seasonal influenza, clinicians rely on point-of-care testing (POCT) using commercially available kits developed against seasonal influenza viruses. However, POCT has not yet been established for the diagnosis of pandemic influenza A virus (H1N1pdm) infection due to the low sensitivity of the existing kits for H1N1pdm.Methodology/principal findingsAn immunochromatography (IC) test kit was developed based on a monoclonal antibody against H1N1pdm, which does not cross-react with seasonal influenza A or B viruses. The efficacy of this kit (PDM-IC kit) for the diagnosis of H1N1pdm infection was compared with that of an existing kit for the detection of seasonal influenza viruses (SEA-IC kit). Nasal swabs (n = 542) were obtained from patients with flu-like syndrome at 13 clinics in Osaka, Japan during the winter of 2010/2011. Among the 542 samples, randomly selected 332 were further evaluated for viral presence by reverse transcriptase polymerase chain reaction (RT-PCR). The PDM-IC kit versus the SEA-IC kit showed higher sensitivity to and specificity for H1N1pdm, despite several inconsistencies between the two kits or between the kits and RT-PCR. Consequently, greater numbers of false-negative and false-positive cases were documented when the SEA-IC kit was employed. Significant correlation coefficients for sensitivity, specificity, and negative prediction values between the two kits were observed at individual clinics, indicating that the results could be affected by clinic-related techniques for sampling and kit handling. Importantly, many patients (especially influenza-negative cases) were prescribed anti-influenza drugs that were incongruous with their condition, largely due to physician preference for patient responses to questionnaires and patient symptomology, as opposed to actual viral presence.Conclusions/significanceConcomitant use of SEA-IC and PDM-IC kits increased the likelihood of correct influenza diagnosis. Increasing the credibility of POCT is anticipated to decrease the inappropriate dispensing of anti-influenza drugs, thereby minimizing the emergence of drug-resistant H1N1pdm strains.

Project description:BackgroundThere is a need for a survey instrument to measure arthralgia (joint pain) that has been psychometrically validated in the context of existing reference instruments. We developed the 16-item Patient-Reported Arthralgia Inventory (PRAI) to measure arthralgia severity in 16 joints, in the context of a longitudinal cohort study to assess aromatase inhibitor-associated arthralgia in breast cancer survivors and arthralgia in postmenopausal women without breast cancer. We sought to evaluate the reliability and validity of the PRAI instrument in these populations, as well as to examine the relationship of patient-reported morning stiffness and arthralgia.MethodsWe administered the PRAI on paper in 294 women (94 initiating aromatase inhibitor therapy and 200 postmenopausal women without breast cancer) at weeks 0, 2, 4, 6, 8, 12, 16, and 52, as well as once in 36 women who had taken but were no longer taking aromatase inhibitor therapy.ResultsCronbach's alpha was 0.9 for internal consistency of the PRAI. Intraclass correlation coefficients of test-retest reliability were in the range of 0.87-0.96 over repeated PRAI administrations; arthralgia severity was higher in the non-cancer group at baseline than at subsequent assessments. Women with joint comorbidities tended to have higher PRAI scores than those without (estimated difference in mean scores: -0.3, 95% confidence interval [CI] -0.5, -0.2; P<0.001). The PRAI was highly correlated with the Functional Assessment of Cancer Therapy-Endocrine Subscale item "I have pain in my joints" (reference instrument; Spearman r range: 0.76-0.82). Greater arthralgia severity on the PRAI was also related to decreased physical function (r=-0.47, 95% CI -0.55, -0.37; P<0.001), higher pain interference (r=0.65, 95% CI 0.57-0.72; P<0.001), less active performance status (estimated difference in location (-0.6, 95% CI -0.9, -0.4; P<0.001), and increased morning stiffness duration (r=0.62, 95% CI 0.54-0.69; P<0.0001).ConclusionWe conclude that the psychometric properties of the PRAI are satisfactory for measuring arthralgia severity.

Project description:BackgroundDespite the established need to prioritize professionalism-training in developing future physicians, very few medical programs in the Gulf Region embed in their curricula discrete contextualized courses aimed at developing the corresponding competencies, while fostering self-directed learning. This study aims at exploring the perception of undergraduate medical students in a multi-cultural, multi-ethnic setting regarding their understanding of, and personal experience with professionalism through their engagement with the content of an innovative curriculum-based professionalism course, offered at a Medical School in Dubai, United Arab Emirates.MethodsThe study used a qualitative phenomenological research design. Out of 33 students, 29 students had submitted reflective essays. The content of these essays was inductively analyzed following a six-step framework for conducting thematic analysis. The framework's steps include familiarizing oneself with the data, generating initial codes, searching for themes, reviewing themes, defining and naming themes, and producing the report.FindingsThe inductive qualitative analysis generated the Professionalism Learning Journey model. This conceptual model includes four interconnected themes: Awareness, Acknowledgement, Realization, and Application. The generated model depicts the trajectory that the learners appear to experience while they are engaging with the content of the course.ConclusionIntegrating a professionalism-training course into an undergraduate medical curriculum is likely to be positively appraised by the learners. It raises their awareness, enables them to value the subject matter and the sophistication of its application, and empowers them to put into practice the taught principles, on an individual basis and collectively. This is especially true when the course is entrenched in constructivism experiential learning theory and designed to foster self-directed learning. The introduced conceptual model, in conjunction with the innovative professionalism-training course curriculum, can serve as a template for other competencies and other schools.

Project description:BackgroundThe global burden of cancer continues to rise and complementary and alternative medicine (CAM) is attracting a lot of interest. However, quality of online information on CAM, particularly on platforms like YouTube, remains questionable. This study aimed to create a comprehensive assessment tool to assess the quality of CAM-related YouTube videos, crucial for informed decision-making in oncology.MethodsThe assessment tool was developed by adapting existing criteria for website content analysis to video rating. A YouTube search was conducted using German-language terms related to CAM ("complementary medicine (CM) for cancer" and "alternative medicine (AM) for cancer"). In total 25 videos were assessed based on the defined criteria and assigned to five different types of providers (journalism, healthcare organization, hospital or health insurance, independent person, non-medical organization). Statistical analysis was conducted using IBM SPSS Statistics 27.ResultsInterrater reliability analysis showed an Intraclass Correlation Coefficient (ICC) of .91, indicating good to excellent agreement. The average video result was of poor quality, with none of the videos meeting all criteria. The videos achieved a mean rating of 38.2 points (SD: 6.5 points; possible range: 20-60 points). Journalism-based videos showed the most views per days, particularly surpassing those from hospitals or health insurance providers (Kruskal-Wallis-Test: z = 3.14, P = .02). However, there was no statistically significant correlation between video quality and the type of provider or interaction indices. Videos retrieved under the search term "CM" generally scored higher in quality than those under "AM" (Mann-Whitney U test: U = 39.5, P = .04). Nonetheless, "CM" videos were less frequently viewed (Mann-Whitney U test: U = 31.0, P = .01).ConclusionThis study, the first of its kind focusing on CAM in cancer care emphasized the challenges in identifying credible sources on social media platforms such as YouTube. The developed assessment tool offers a more detailed evaluation method for health-related videos but requires further refinement and testing. Collaboration between healthcare and media entities is suggested to improve the dissemination of reliable information on platforms like YouTube.

Dataset Information

Grading reflective essays: the reliability of a newly developed tool- GRE-9.

Background

Methods

Results

Conclusions

Publications

Grading reflective essays: the reliability of a newly developed tool- GRE-9.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets