Dataset Information

What steps are necessary to create written or web-based selected-response assessments?

ABSTRACT: Before we work out what constitutes an assessment's value for a given cost in medical education, we must first outline the steps necessary to create an assessment, and then assign a cost to each step. In this study we undertook the first phase of this process: we sought to work out all the steps necessary to create written selected-response assessments. First, the lead author created an initial list of potential steps for developing written assessments. This was then distributed to the other three authors. These authors independently added further steps to the list. The lead author incorporated the contributions of these others and created a second draft. This process was repeated until consensus was achieved amongst the study's authors. Next, the list was shared by means of an online questionnaire with 100 healthcare professionals with experience in medical education. The results of the authors' and healthcare professionals' thoughts and feedback on the steps, needed to create written assessment, are outlined below in full. We outlined the steps that are necessary to create written or web-based selected-response assessments.

SUBMITTER: Morgan M

PROVIDER: S-EPMC4309946 | biostudies-literature | 2014

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

What steps are necessary to create written or web-based selected-response assessments?

Morgan Matt M Dory Valérie V Lubarsky Stuart S Walsh Kieran K

Journal of educational evaluation for health professions 20141108

Before we work out what constitutes an assessment's value for a given cost in medical education, we must first outline the steps necessary to create an assessment, and then assign a cost to each step. In this study we undertook the first phase of this process: we sought to work out all the steps necessary to create written selected-response assessments. First, the lead author created an initial list of potential steps for developing written assessments. This was then distributed to the other thr ...[more]

PMID: 25417862

Similar Datasets

Project description:BackgroundUncertainty is integral to evidence-informed decision making and is of particular importance for preference-sensitive decisions. Communicating uncertainty to patients and the public has long been identified as a goal in the informed and shared decision-making movement. Despite this, there is little quantitative research on how uncertainty in health information is perceived by readers.ObjectiveThe aim of this study was to examine the impact of different uncertainty descriptions regarding the evidence for a treatment effect in a written research summary for the public.MethodsWe developed 8 versions of a research summary on a fictitious drug for tinnitus with varying degrees (Q1), sources (Q2), and magnitudes of uncertainty (Q3). We recruited 2099 members of the German public from a web-based research panel. Of these, 1727 fulfilled the inclusion criteria and were randomly presented with one of these research summaries. Randomization was conducted by using a centralized computer with a random number generator. Web-based recruitment and data collection were fully automated. Participants were not aware of the purpose of the study and alternative presentations. We measured the following outcomes: perception of the treatment effectiveness (primary), certainty in the judgement of treatment effectiveness, perception of the body of evidence, text quality, and intended decision. The outcomes were self-assessed.ResultsFor the primary outcome, we did not find a global effect for Q1 and Q2 (P=.25 and P=.73), but we found a global effect for Q3 (P=.048). Pairwise comparisons showed a weaker perception of treatment effectiveness for the research summary with 3 sources of uncertainty compared to the version with 2 sources of uncertainty (P=.04). Specifically, the proportion of the participants in the group with 3 sources of uncertainty that perceived the drug as possibly beneficial was 9% lower than that of the participants in the group with 2 sources of uncertainty (92/195, 47.2% vs 111/197, 56.3%, respectively). The proportion of the participants in the group with 3 sources of uncertainty that considered the drug to be of unclear benefit was 8% higher than that of the participants in the group with 2 sources of uncertainty (72/195, 36.9% vs 57/197, 28.9%, respectively). However, there was no significant difference compared to the version with 1 source of uncertainty (P=.31). We did not find any meaningful differences between the research summaries for the secondary outcomes.ConclusionsCommunicating even a large magnitude of uncertainty for a treatment effect had little impact on the perceived effectiveness. Efforts to improve public understanding of research are needed to improve the understanding of evidence-based health information.Trial registrationGerman Clinical Trials Register DRKS00015911, https://www.drks.de/drks_web/navigate.do?navigationId=trial.HTML&TRIAL_ID=DRKS00015911.International registered report identifier (irrid)RR2-10.2196/13425.

Project description:Background The COVID-19 pandemic in parallel with concerns about bias in grading resulted in many medical schools adopting pass/fail clinical grading and relying solely on narrative assessments. However, narratives often contain bias and lack specificity. The purpose of this project was to develop asynchronous faculty development to rapidly educate/re-educate > 2000 clinical faculty spread across geographic sites and clinical disciplines on components of a well-written narrative and methods to minimize bias in the assessment of students. Methods We describe creation, implementation, and pilot data outcomes for an asynchronous faculty development curriculum created by a committee of volunteer learners and faculty. After reviewing the literature on the presence and impact of bias in clinical rotations and ways to mitigate bias in written narrative assessments, the committee developed a web-based curriculum using multimedia learning theory and principles of adult learning. Just-in-time supplemental materials accompanied the curriculum. The Dean added completion of the module by 90% of clinical faculty to the department chairperson’s annual education metric. Module completion was tracked in a learning management system, including time spent in the module and the answer to a single text entry question about intended changes in behavior. Thematic analysis of the text entry question with grounded theory and inductive processing was used to define themes of how faculty anticipate future teaching and assessment as a result of this curricula. Outcomes Between January 1, 2021, and December 1, 2021, 2166 individuals completed the online module; 1820 spent between 5 and 90 min on the module, with a median time of 17 min and an average time of 20.2 min. 15/16 clinical departments achieved completion by 90% or more faculty. Major themes included: changing the wording of future narratives, changing content in future narratives, and focusing on efforts to change how faculty teach and lead teams, including efforts to minimize bias. Conclusions We developed a faculty development curriculum on mitigating bias in written narratives with high rates of faculty participation. Inclusion of this module as part of the chair’s education performance metric likely impacted participation. Nevertheless, time spent in the module suggests that faculty engaged with the material. Other institutions could easily adapt this curriculum with provided materials. Supplementary Information The online version contains supplementary material available at 10.1186/s12909-023-04237-w.

Project description:Background and aimsDue to the COVID-19 pandemic, tele-health has gained popularity for both providing services and delivering assessments to children with disabilities. In this manuscript, we discuss the process of collecting standardized oral language, reading, and writing tele-assessment data with early elementary children with autism spectrum disorder (ASD) and offer preliminary findings related to child and parent engagement and technology issues.MethodsThe data presented are from pretest assessments during an efficacy study examining the electronic delivery of a listening comprehension intervention for children with ASD. Pretest sessions included a battery of standardized language, reading, and writing assessments, conducted over Zoom. The authors operationalized and developed a behavioral codebook of three overarching behavioral categories (parent involvement, child disengagement, and technology issues). Researchers coded videos offline to record frequencies of indicated behaviors across participants and assessment subtests.ResultsInvolvement from parents accounted for the highest number of codes. Children showed some disengagement during assessment sessions. Technology issues were minimal. Behavioral categories appeared overall limited but varied across participants and assessments.ConclusionsParent involvement behaviors made up approximately two-thirds of the coded behaviors. Child disengagement behaviors made up approximately one-fourth of the coded behaviors, and these behaviors occurred more frequently across many different participants (with lower frequencies but greater coverage across children). Technology problems specific to responding to assessment items were relatively uncommon.ImplicationsClear guidelines including assessment preparation, modification of directions, and guidelines for parents who remain present are among the implications discussed. We also provide practical implications for continued successful adapted tele-assessments for children with ASD.

Project description:BackgroundHigh-frequency patient-reported outcome (PRO) assessments are used to measure patients' symptoms after surgery for surgical research; however, the quality of those longitudinal PRO data has seldom been discussed.ObjectiveThe aim of this study was to determine data quality-influencing factors and to profile error trajectories of data longitudinally collected via paper-and-pencil (P&P) or web-based assessment (electronic PRO [ePRO]) after thoracic surgery.MethodsWe extracted longitudinal PRO data with 678 patients scheduled for lung surgery from an observational study (n=512) and a randomized clinical trial (n=166) on the evaluation of different perioperative care strategies. PROs were assessed by the MD Anderson Symptom Inventory Lung Cancer Module and single-item Quality of Life Scale before surgery and then daily after surgery until discharge or up to 14 days of hospitalization. Patient compliance and data error were identified and compared between P&P and ePRO. Generalized estimating equations model and 2-piecewise model were used to describe trajectories of error incidence over time and to identify the risk factors.ResultsAmong 678 patients, 629 with at least 2 PRO assessments, 440 completed 3347 P&P assessments and 189 completed 1291 ePRO assessments. In total, 49.4% of patients had at least one error, including (1) missing items (64.69%, 1070/1654), (2) modifications without signatures (27.99%, 463/1654), (3) selection of multiple options (3.02%, 50/1654), (4) missing patient signatures (2.54%, 42/1654), (5) missing researcher signatures (1.45%, 24/1654), and (6) missing completion dates (0.30%, 5/1654). Patients who completed ePRO had fewer errors than those who completed P&P assessments (ePRO: 30.2% [57/189] vs. P&P: 57.7% [254/440]; P<.001). Compared with ePRO patients, those using P&P were older, less educated, and sicker. Common risk factors of having errors were a lower education level (P&P: odds ratio [OR] 1.39, 95% CI 1.20-1.62; P<.001; ePRO: OR 1.82, 95% CI 1.22-2.72; P=.003), treated in a provincial hospital (P&P: OR 3.34, 95% CI 2.10-5.33; P<.001; ePRO: OR 4.73, 95% CI 2.18-10.25; P<.001), and with severe disease (P&P: OR 1.63, 95% CI 1.33-1.99; P<.001; ePRO: OR 2.70, 95% CI 1.53-4.75; P<.001). Errors peaked on postoperative day (POD) 1 for P&P, and on POD 2 for ePRO.ConclusionsIt is possible to improve data quality of longitudinally collected PRO through ePRO, compared with P&P. However, ePRO-related sampling bias needs to be considered when designing clinical research using longitudinal PROs as major outcomes.

Dataset Information

What steps are necessary to create written or web-based selected-response assessments?

Publications

What steps are necessary to create written or web-based selected-response assessments?

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets