Dataset Information

The new normal? Redaction bias in biomedical science.

ABSTRACT: A concerning amount of biomedical research is not reproducible. Unreliable results impede empirical progress in medical science, ultimately putting patients at risk. Many proximal causes of this irreproducibility have been identified, a major one being inappropriate statistical methods and analytical choices by investigators. Within this, we formally quantify the impact of inappropriate redaction beyond a threshold value in biomedical science. This is effectively truncation of a dataset by removing extreme data points, and we elucidate its potential to accidentally or deliberately engineer a spurious result in significance testing. We demonstrate that the removal of a surprisingly small number of data points can be used to dramatically alter a result. It is unknown how often redaction bias occurs in the broader literature, but given the risk of distortion to the literature involved, we suggest that it must be studiously avoided, and mitigated with approaches to counteract any potential malign effects to the research quality of medical science.

SUBMITTER: Grimes DR

PROVIDER: S-EPMC8633797 | biostudies-literature | 2021 Dec

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

The new normal? Redaction bias in biomedical science.

Grimes David Robert DR Heathers James J

Royal Society open science 20211201 12

A concerning amount of biomedical research is not reproducible. Unreliable results impede empirical progress in medical science, ultimately putting patients at risk. Many proximal causes of this irreproducibility have been identified, a major one being inappropriate statistical methods and analytical choices by investigators. Within this, we formally quantify the impact of inappropriate redaction beyond a threshold value in biomedical science. This is effectively truncation of a dataset by remov ...[more]

PMID: 34966555

Similar Datasets

Project description:ImportanceDiversity in the ophthalmology profession is important when providing care for an increasingly diverse patient population. However, implicit bias may inadvertently disadvantage underrepresented applicants during resident recruitment and selection.ObjectiveTo evaluate the association of the redaction of applicant identifiers with the review scores on ophthalmology residency applications as an intervention to address implicit bias.Design, setting, and participantsIn this quality improvement study, 46 faculty members reviewed randomized sets of 462 redacted and unredacted applications from a single academic institution during the 2019-2020 ophthalmology residency application cycle.InterventionsApplications electronically redacted for applicant identifiers, including name, sex or gender, race and ethnicity, and related terms.Main outcomes and measuresThe main outcome was the distribution of scores on redacted and unredacted applications, stratified by applicant's sex, underrepresentation in medicine (URiM; traditionally comprising American Indian or Alaskan Native, Black, and Hispanic individuals) status, and international medical graduate (IMG) status; the application score β coefficients for redaction and the applicant and reviewer characteristics were calculated. Applications were scored on a scale of 1 to 9, where 1 was the best score and 9 was the worst score. Scores were evaluated for a significant difference based on redaction among female, URiM, and IMG applicants. Linear regression was used to evaluate the adjusted association of redaction, self-reported applicant characteristics, and reviewer characteristics with scores on ophthalmology residency applications.ResultsIn this study, 277 applicants (60.0%) were male and 71 (15.4%) had URiM status; 32 faculty reviewers (69.6%) were male and 2 (0.4%) had URiM status. The distribution of scores was similar for redacted vs unredacted applications, with no difference based on sex, URiM status, or IMG status. Applicant's sex, URiM status, and IMG status had no association with scores in multivariable analysis (sex, β = -0.08; 95% CI, -0.32 to 0.15; P = .26; URiM status, β = -0.03; (95% CI, -0.36 to 0.30; P = .94; and IMG status, β = 0.39; 95% CI, -0.24 to 1.02; P = .35). In adjusted regression, redaction was not associated with differences in scores (β = -0.06 points on a 1-9 scale; 95% CI, -0.22 to 0.10 points; P = .48). Factors most associated with better scores were attending a top 20 medical school (β = -1.06; 95% CI, -1.37 to -0.76; P < .001), holding an additional advanced degree (β = -0.86; 95% CI, -1.22 to -0.50; P < .001), and having a higher United States Medical Licensing Examination Step 1 score (β = -0.35 per 10-point increase; 95% CI, -0.45 to -0.26; P < .001).Conclusions and relevanceThis quality improvement study did not detect an association between the redaction of applicant characteristics on ophthalmology residency applications and the application review scores among underrepresented candidates at this institution. Although the study may not have been powered adequately to find a difference, these findings suggest that the association of redaction with application review scores may be preempted by additional approaches to enhance diversity, including pipeline programs, implicit bias training, diversity-centered culture and priorities, and targeted applicant outreach. Programs may adapt this study design to probe their own application screening biases and track over time before-and-after bias-related interventions.

Dataset Information

The new normal? Redaction bias in biomedical science.

Publications

The new normal? Redaction bias in biomedical science.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets