Browse
Submit Data
Databases
API
Help

Dataset Information

0 Views

0 Connections

0 Citations

0 Reanalyses

0 Downloads

Omics score: 0

Risk scores, label bias, everything but the kitchen sink.

ABSTRACT: In designing risk assessment algorithms, many scholars promote a "kitchen sink" approach, reasoning that more information yields more accurate predictions. We show, however, that this rationale often fails when algorithms are trained to predict a proxy of the true outcome, for instance, predicting arrest as a proxy for criminal behavior. With this "label bias," one should exclude a feature if its correlation with the proxy and its correlation with the true outcome have opposite signs, conditional on the other model features. This criterion is often satisfied when a feature is weakly correlated with the true outcome, and, additionally, that feature and the true outcome are both direct causes of the proxy outcome. For example, criminal behavior and geography may be weakly correlated and, due to patterns of police deployment, direct causes of one's arrest record-suggesting that excluding geography in criminal risk assessment will weaken an algorithm's performance in predicting arrest but will improve its capacity to predict actual crime.

SUBMITTER: Zanger-Tishler M

PROVIDER: S-EPMC10980258 | biostudies-literature | 2024 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Json Xml

Publications

Risk scores, label bias, and everything but the kitchen sink.

Zanger-Tishler Michael M Nyarko Julian J Goel Sharad S

Science advances 20240329 13

In designing risk assessment algorithms, many scholars promote a "kitchen sink" approach, reasoning that more information yields more accurate predictions. We show, however, that this rationale often fails when algorithms are trained to predict a proxy of the true outcome, for instance, predicting arrest as a proxy for criminal behavior. With this "label bias," one should exclude a feature if its correlation with the proxy and its correlation with the true outcome have opposite signs, conditiona ...[more]

PMID: 38552013

Similar Datasets

Everything but the Kitchen Sink: An Analysis of Bacterial and Chemical Contaminants Found in Syringe Residue From People Who Inject Drugs.

Project description:BackgroundPeople who inject drugs (PWID) are at high risk of severe wounds, invasive infections, and overdoses. To date, there are few data on the bacterial and chemical contaminants PWID are exposed to when using illicitly manufactured fentanyls and stimulants.MethodsPreviously used injection drug use equipment was recovered in St Louis, Missouri, by harm reduction organizations over a 12-month period. Syringe residue was analyzed for bacterial contaminants by routine culturing followed by whole genome sequencing of single bacterial isolates. Chemical adulterants in syringe residue were identified by liquid chromatography-mass spectrometry.ResultsBacteria were cultured from 58.75% of 160 syringes analyzed. Polymicrobial growth was common and was observed in 23.75% of samples. Bacillus cereus was the most common pathogen present and was observed in 20.6% of syringe residues, followed closely by Staphylococcus aureus at 18.8%. One hundred syringes underwent mass spectrometry, which demonstrated that chemical adulterants were common and included caffeine, diphenhydramine, lidocaine, quinine, and xylazine.ConclusionsAnalysis of syringe residue from discarded drug use equipment demonstrates both chemical and biological contaminants, including medically important pathogens and adulterants.

| S-EPMC10766411 | biostudies-literature

Minimizing the "Kitchen Sink" Approach: De-implementation of Unnecessary Care for Patients with Anaphylaxis in a Pediatric Emergency Department.

Project description:Epinephrine is the only medication that prevents morbidity and mortality in anaphylaxis. Systemic corticosteroids and H2 receptor antagonists (H2RA) may benefit select cases but are not universally indicated. This study aims to de-implement the universal use of steroids and H2RAs and emphasize epinephrine-focused care for children with anaphylaxis during acute care visits. The study aims to reduce steroid and H2RA use from 81% and 60%, respectively, to 30% by December 2019.MethodsThe primary outcome measures were the percent of patients receiving steroids and H2RAs in the emergency department (ED) or urgent care (UC). Process measure was the frequency of intravenous (IV) line placement. Balancing measures were ED/UC length of stay, admission rate, and ED/UC return visit rate. In addition, a multidisciplinary team designed the following interventions: (1) anaphylaxis clinical pathway to emphasize epinephrine-focused care, outline criteria for second-line therapies and a provider guideline for ED/UC observation; (2) standardize unit-based anaphylaxis medication kits; (3) optimize electronic medical record tools, including order sets and discharge instructions to be concordant with guideline recommendations.ResultsThe study included 870 patients. There was special cause variation in the use of steroids (81%-33%) and H2RAs (60%-11%), ED/UC Length of stay decreased (6.2-5.0 hours). There was no special cause variation in admission rates or ED/UC return visit rates.ConclusionUniversal use of systemic steroids and H2RAs can be safely de-implemented in pediatric patients with anaphylaxis using quality improvement methods.

| S-EPMC8970089 | biostudies-literature

Inflation of polygenic risk scores caused by sample overlap and relatedness: Examples of a major risk of bias.

Project description:Polygenic risk scores (PRSs) are an important tool for understanding the role of common genetic variants in human disease. Standard best practices recommend that PRSs be analyzed in cohorts that are independent of the genome-wide association study (GWAS) used to derive the scores without sample overlap or relatedness between the two cohorts. However, identifying sample overlap and relatedness can be challenging in an era of GWASs performed by large biobanks and international research consortia. Although most genomics researchers are aware of best practices and theoretical concerns about sample overlap and relatedness between GWAS and PRS cohorts, the prevailing assumption is that the risk of bias is small for very large GWASs. Here, we present two real-world examples demonstrating that sample overlap and relatedness is not a minor or theoretical concern but an important potential source of bias in PRS studies. Using a recently developed statistical adjustment tool, we found that excluding overlapping and related samples was equal to or more powerful than adjusting for overlap bias. Our goal is to make genomics researchers aware of the magnitude of risk of bias from sample overlap and relatedness and to highlight the need for mitigation tools, including independent validation cohorts in PRS studies, continued development of statistical adjustment methods, and tools for researchers to test their cohorts for overlap and relatedness with GWAS cohorts without sharing individual-level data.

| S-EPMC11393675 | biostudies-literature

The stability and reliability of attentional bias measures in the dot-probe task: Evidence from both traditional mean bias scores and trial-level bias scores.

Project description:A new method of calculating attentional bias from the dot-probe task measures fluctuations in bias towards and away from emotional stimuli over time using trial level bias score metrics. We assessed the stability and reliability of traditional attentional bias scores and trial level bias score measures of attentional bias across time in two five-block dot-probe task experiments in non-clinical samples. In experiments 1 and 2, both traditional attentional bias scores and trial level bias score measures of attentional bias did not habituate/decrease across time. In general, trial level bias score metrics (i.e., attention bias variability as well as the mean biases toward and away from threat) were more reliable than the traditional attention bias measure. This pattern was observed across both experiments. The traditional bias score, however, did improve in reliability in the later blocks of the fearful face dot-probe task. Although trial level bias score measures did not habituate and were more reliable across blocks, these measures did not correlate with state or trait anxiety. On the other hand, trial level bias score measures were strongly correlated with general reaction time variability-and after controlling for this effect no longer superior in reliability in comparison to the traditional attention bias measure. We conclude that general response variability should be removed from trial level bias score measures to ensure that they truly reflect attention bias variability.

| S-EPMC7864391 | biostudies-literature

A Systematic Review of Predictor Composition, Outcomes, Risk of Bias, and Validation of COVID-19 Prognostic Scores.

Project description:BackgroundNumerous prognostic scores have been published to support risk stratification for patients with coronavirus disease 2019 (COVID-19).MethodsWe performed a systematic review to identify the scores for confirmed or clinically assumed COVID-19 cases. An in-depth assessment and risk of bias (ROB) analysis (Prediction model Risk Of Bias ASsessment Tool [PROBAST]) was conducted for scores fulfilling predefined criteria ([I] area under the curve [AUC)] ≥ 0.75; [II] a separate validation cohort present; [III] training data from a multicenter setting [≥2 centers]; [IV] point-scale scoring system).ResultsOut of 1522 studies extracted from MEDLINE/Web of Science (20/02/2023), we identified 242 scores for COVID-19 outcome prognosis (mortality 109, severity 116, hospitalization 14, long-term sequelae 3). Most scores were developed using retrospective (75.2%) or single-center (57.1%) cohorts. Predictor analysis revealed the primary use of laboratory data and sociodemographic information in mortality and severity scores. Forty-nine scores were included in the in-depth analysis. The results indicated heterogeneous quality and predictor selection, with only five scores featuring low ROB. Among those, based on the number and heterogeneity of validation studies, only the 4C Mortality Score can be recommended for clinical application so far.ConclusionsThe application and translation of most existing COVID scores appear unreliable. Guided development and predictor selection would have improved the generalizability of the scores and may enhance pandemic preparedness in the future.

| S-EPMC11006104 | biostudies-literature

Genetic Risk Scores.

Project description:Genome-wide variation data with millions of genetic markers have become commonplace. However, the potential for interpretation and application of these data for clinical assessment of outcomes of interest, and prediction of disease risk, is currently not fully realized. Many common complex diseases now have numerous, well-established risk loci and likely harbor many genetic determinants with effects too small to be detected at genome-wide levels of statistical significance. A simple and intuitive approach for converting genetic data to a predictive measure of disease susceptibility is to aggregate the effects of these loci into a single measure, the genetic risk score. Here, we describe some common methods and software packages for calculating genetic risk scores and polygenic risk scores, with focus on studies of common complex diseases. We review the basic information needed, as well as important considerations for constructing genetic risk scores, including specific requirements for phenotypic and genetic data, and limitations in their application. © 2019 by John Wiley & Sons, Inc.

| S-EPMC6941594 | biostudies-literature

Gene-environment dependencies lead to collider bias in models with polygenic scores.

Project description:The application of polygenic scores has transformed our ability to investigate whether and how genetic and environmental factors jointly contribute to the variation of complex traits. Modelling the complex interplay between genes and environment, however, raises serious methodological challenges. Here we illustrate the largely unrecognised impact of gene-environment dependencies on the identification of the effects of genes and their variation across environments. We show that controlling for heritable covariates in regression models that include polygenic scores as independent variables introduces endogenous selection bias when one or more of these covariates depends on unmeasured factors that also affect the outcome. This results in the problem of conditioning on a collider, which in turn leads to spurious associations and effect sizes. Using graphical and simulation methods we demonstrate that the degree of bias depends on the strength of the gene-covariate correlation and of hidden heterogeneity linking covariates with outcomes, regardless of whether the main analytic focus is mediation, confounding, or gene × covariate (commonly gene × environment) interactions. We offer potential solutions, highlighting the importance of causal inference. We also urge further caution when fitting and interpreting models with polygenic scores and non-exogenous environments or phenotypes and demonstrate how spurious associations are likely to arise, advancing our understanding of such results.

| S-EPMC8097011 | biostudies-literature

Polygenic transcriptome risk scores (PTRS) can improve portability of polygenic risk scores across ancestries.

Project description:BackgroundPolygenic risk scores (PRS) are valuable to translate the results of genome-wide association studies (GWAS) into clinical practice. To date, most GWAS have been based on individuals of European-ancestry leading to poor performance in populations of non-European ancestry.ResultsWe introduce the polygenic transcriptome risk score (PTRS), which is based on predicted transcript levels (rather than SNPs), and explore the portability of PTRS across populations using UK Biobank data.ConclusionsWe show that PTRS has a significantly higher portability (Wilcoxon p=0.013) in the African-descent samples where the loss of performance is most acute with better performance than PRS when used in combination.

| S-EPMC8759285 | biostudies-literature

Kitchen sponge

Project description:Kitchen sponge microbiome

| PRJEB18578 | ENA

Bias Reduction Methods for Propensity Scores Estimated from Error-Prone EHR-Derived Covariates.

Project description:As the use of electronic health records (EHR) to estimate treatment effects has become widespread, concern about bias introduced by error in EHR-derived covariates has also grown. While methods exist to address measurement error in individual covariates, little prior research has investigated the implications of using propensity scores for confounder control when the propensity scores are constructed from a combination of accurate and error-prone covariates. We reviewed approaches to account for error in propensity scores and used simulation studies to compare their performance. These comparisons were conducted across a range of scenarios featuring variation in outcome type, validation sample size, main sample size, strength of confounding, and structure of the error in the mismeasured covariate. We then applied these approaches to a real-world EHR-based comparative effectiveness study of alternative treatments for metastatic bladder cancer. This head-to-head comparison of measurement error correction methods in the context of a propensity score-adjusted analysis demonstrated that multiple imputation for propensity scores performs best when the outcome is continuous and regression calibration-based methods perform best when the outcome is binary.

| S-EPMC8210692 | biostudies-literature

OmicsDI is part of the ELIXIR infrastructure

OmicsDI is an Elixir interoperability service. Learn more ›

Tweets

OmicsDI Databases

PRIDE
PeptideAtlas
MassIVE
JPOST Repository
Physiome Model Repository

EGA
EVA
ENA
LINCS
PAXDB
Cell Collective

MetaboLights
Metabolomics Workbench
MetabolomeExpress
GNPS
BioModels
FAIRDOMHub

ArrayExpress
dbGaP
ExpressionAtlas
GEO
NODE

Information

Databases
Help
API
Contact us
Code on GitHub
Terms of use
Submit Data