Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.
Ontology highlight
ABSTRACT:
SUBMITTER: Vickers AJ
PROVIDER: S-EPMC4724785 | biostudies-literature | 2016 Jan
REPOSITORIES: biostudies-literature
Publications
Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests.
Vickers Andrew J AJ Van Calster Ben B Steyerberg Ewout W EW
BMJ (Clinical research ed.) 20160125
Similar Datasets
Project description:Comparing diagnostic tests on accuracy alone can be inconclusive. For example, a test may have better sensitivity than another test yet worse specificity. Comparing tests on benefit risk may be more conclusive because clinical consequences of diagnostic error are considered. For benefit-risk evaluation, we propose diagnostic yield, the expected distribution of subjects with true positive, false positive, true negative, and false negative test results in a hypothetical population. We construct a table of diagnostic yield that includes the number of false positive subjects experiencing adverse consequences from unnecessary work-up. We then develop a decision theory for evaluating tests. The theory provides additional interpretation to quantities in the diagnostic yield table. It also indicates that the expected utility of a test relative to a perfect test is a weighted accuracy measure, the average of sensitivity and specificity weighted for prevalence and relative importance of false positive and false negative testing errors, also interpretable as the cost-benefit ratio of treating non-diseased and diseased subjects. We propose plots of diagnostic yield, weighted accuracy, and relative net benefit of tests as functions of prevalence or cost-benefit ratio. Concepts are illustrated with hypothetical screening tests for colorectal cancer with test positive subjects being referred to colonoscopy.
| S-EPMC5471848 | biostudies-other
Project description:Rapid diagnostic tests (RDTs) for bloodstream infections have the potential to reduce time to appropriate antimicrobial therapy and improve patient outcomes. Previously, an in-house, lipid-based, matrix-assisted laser desorption/ionization-time of flight mass spectrometry (MALDI-TOF MS) method, Fast Lipid Analysis Technique (FLAT MS), has shown promise as a rapid pathogen identification method. In this study, FLAT MS for direct from blood culture identification was evaluated and compared to FDA-cleared identification methods using the Benefit-risk Evaluation Framework (BED-FRAME) analysis. FLAT MS was evaluated and compared to Bruker Sepsityper and bioMérieux BioFire FilmArray BCID2 using results from a previous study. For this study, 301 positive blood cultures were collected from the University of Maryland Medical Center. The RDTs were compared by their sensitivities, time-to-results, hands-on time, and BED-FRAME analysis. The overall sensitivity of all platforms compared to culture results from monomicrobial-positive blood cultures was 88.3%. However, the three RDTs differed in their accuracy for identifying Gram-positive bacteria, Gram-negative bacteria, and yeast. Time-to-results for FLAT MS, Sepsityper, and BioFire BCID2 were all approximately one hour. Hands-on times for FLAT MS, Sepsityper, and BioFire BCID2 were 10 (±1.3), 40 (±2.8), and 5 (±0.25) minutes, respectively. BED-FRAME demonstrated that each RDT had utility at different pathogen prevalence and relative importance. BED-FRAME is a useful tool that can used to determine which RDT is best for a healthcare center.
| S-EPMC10793330 | biostudies-literature
Project description:BackgroundThis meta-analysis aimed to assess the value of the C2HEST score to facilitate population screening and detection of AF risk in millions of populations and validate risk scores and their composition and discriminatory power for identifying people at high or low risk of AF. We searched major indexing databases, including Pubmed/Medline, ISI web of science, Scopus, Embase, and Cochrane central, using ("C2HEST" OR "risk scoring system" OR "risk score") AND ("atrial fibrillation (AF)" OR "atrial flutter" OR "tachycardia, supraventricular" OR "heart atrium flutter") without any language, study region or study type restrictions between 1990 and 2021 years. Analyses were done using Meta-DiSc. The title and abstract screening were conducted by two independent investigators.ResultsTotally 679 records were found through the initial search, of which ultimately, nine articles were included in the qualitative and quantitative analyses. The risk of AF accompanied every one-point increase of C2HEST score (OR 1.03, 95% CI 1.01-1.05, p < 0.00001), with a high heterogeneity across studies (I2 = 100%). The SROC for C2HEST score in the prediction of AF showed that the overall area under the curve (AUC) was 0.91 (95% CI 0.85-0.96), AUC in Asian population was 0.87 (95% CI: 0.78-0.95) versus non-Asian 0.95 (95% CI 0.91-0.99), and in general population was 0.92 (95% CI 0.85-0.99) versus those with chronic conditions 0.83 (95% CI 0.71-0.95), respectively.ConclusionsThe results of this research support the idea that this quick score has the opportunity for use as a risk assessment in patients' AF screening strategies.
| S-EPMC8643379 | biostudies-literature
Project description:Microbiome data are becoming increasingly available in large health cohorts, yet metabolomics data are still scant. While many studies generate microbiome data, they lack matched metabolomics data or have considerable missing proportions of metabolites. Since metabolomics is key to understanding microbial and general biological activities, the possibility of imputing individual metabolites or inferring metabolomics pathways from microbial taxonomy or metagenomics is intriguing. Importantly, current metabolomics profiling methods such as the HMP Unified Metabolic Analysis Network (HUMAnN) have unknown accuracy and are limited in their ability to predict individual metabolites. To address this gap, we developed a novel metabolite prediction method, and we present its application and evaluation in an oral microbiome study. The new method for predicting metabolites using microbiome data (ENVIM) is based on the elastic net model (ENM). ENVIM introduces an extra step to ENM to consider variable importance (VI) scores, and thus, achieves better prediction power. We investigate the metabolite prediction performance of ENVIM using metagenomic and metatranscriptomic data in a supragingival biofilm multi-omics dataset of 289 children ages 3-5 who were participants of a community-based study of early childhood oral health (ZOE 2.0) in North Carolina, United States. We further validate ENVIM in two additional publicly available multi-omics datasets generated from studies of gut health. We select gene family sets based on variable importance scores and modify the existing ENM strategy used in the MelonnPan prediction software to accommodate the unique features of microbiome and metabolome data. We evaluate metagenomic and metatranscriptomic predictors and compare the prediction performance of ENVIM to the standard ENM employed in MelonnPan. The newly developed ENVIM method showed superior metabolite predictive accuracy than MelonnPan when trained with metatranscriptomics data only, metagenomics data only, or both. Better metabolite prediction is achieved in the gut microbiome compared with the oral microbiome setting. We report the best-predictable compounds in all these three datasets from two different body sites. For example, the metabolites trehalose, maltose, stachyose, and ribose are all well predicted by the supragingival microbiome.
| S-EPMC8573316 | biostudies-literature
Project description:ObjectiveThe objective of this study is to provide a method to calculate model performance measures in the presence of resource constraints, with a focus on net benefit (NB).Materials and methodsTo quantify a model's clinical utility, the Equator Network's TRIPOD guidelines recommend the calculation of the NB, which reflects whether the benefits conferred by intervening on true positives outweigh the harms conferred by intervening on false positives. We refer to the NB achievable in the presence of resource constraints as the realized net benefit (RNB), and provide formulae for calculating the RNB.ResultsUsing 4 case studies, we demonstrate the degree to which an absolute constraint (eg, only 3 available intensive care unit [ICU] beds) diminishes the RNB of a hypothetical ICU admission model. We show how the introduction of a relative constraint (eg, surgical beds that can be converted to ICU beds for very high-risk patients) allows us to recoup some of the RNB but with a higher penalty for false positives.DiscussionRNB can be calculated in silico before the model's output is used to guide care. Accounting for the constraint changes the optimal strategy for ICU bed allocation.ConclusionsThis study provides a method to account for resource constraints when planning model-based interventions, either to avoid implementations where constraints are expected to play a larger role or to design more creative solutions (eg, converted ICU beds) to overcome absolute constraints when possible.
| S-EPMC10018264 | biostudies-literature
Project description:The gene-expression ratio technique was used to design a molecular signature to diagnose MPM from among other potentially confounding diagnoses and differentiate the epithelioid from the sarcomatoid histological subtype of MPM.
2013-05-01 | GSE42977 | GEO
Project description:BackgroundMachine learning (ML) can be an effective tool to extract information from attribute-rich molecular datasets for the generation of molecular diagnostic tests. However, the way in which the resulting scores or classifications are produced from the input data may not be transparent. Algorithmic explainability or interpretability has become a focus of ML research. Shapley values, first introduced in game theory, can provide explanations of the result generated from a specific set of input data by a complex ML algorithm.MethodsFor a multivariate molecular diagnostic test in clinical use (the VeriStrat® test), we calculate and discuss the interpretation of exact Shapley values. We also employ some standard approximation techniques for Shapley value computation (local interpretable model-agnostic explanation (LIME) and Shapley Additive Explanations (SHAP) based methods) and compare the results with exact Shapley values.ResultsExact Shapley values calculated for data collected from a cohort of 256 patients showed that the relative importance of attributes for test classification varied by sample. While all eight features used in the VeriStrat® test contributed equally to classification for some samples, other samples showed more complex patterns of attribute importance for classification generation. Exact Shapley values and Shapley-based interaction metrics were able to provide interpretable classification explanations at the sample or patient level, while patient subgroups could be defined by comparing Shapley value profiles between patients. LIME and SHAP approximation approaches, even those seeking to include correlations between attributes, produced results that were quantitatively and, in some cases qualitatively, different from the exact Shapley values.ConclusionsShapley values can be used to determine the relative importance of input attributes to the result generated by a multivariate molecular diagnostic test for an individual sample or patient. Patient subgroups defined by Shapley value profiles may motivate translational research. However, correlations inherent in molecular data and the typically small ML training sets available for molecular diagnostic test development may cause some approximation methods to produce approximate Shapley values that differ both qualitatively and quantitatively from exact Shapley values. Hence, caution is advised when using approximate methods to evaluate Shapley explanations of the results of molecular diagnostic tests.
| S-EPMC8265031 | biostudies-literature
Project description:Multiple sclerosis (MS) is an inflammatory autoimmune disease that causes demyelination of nerve cell axons. This paper is devoted to the study of relapsing-remitting multiple sclerosis (RRMS) biomarkers using an LC-MS/MS-based targeted metabolomics approach and the assessment of changes in the profile of 13 amino acids and 29 acylcarnitines in plasma during the relapse of the disease. A significant increase (p < 0.05) in the concentration of glutamate in plasma in patients with RRMS was detected, while the sum of leucine and isoleucine was reduced. A decrease in the concentration of decenoylcarnitine (C10:1, p < 0.05) was observed among acylcarnitines, and this metabolite was detected as a biomarker for the disease for the first time. Several models based on a single marker or multiple pre-selected markers and multivariate analysis with a dimension reduction technique were compared in their effectiveness for the classification of RRMS and healthy controls. The best results for cross-validation showed models of general linear regression (GLM, AUC = 0.783) and random forest model (RF, AUC = 0.769) based on pre-selected biomarkers. Validation of the models on the test set showed that the RF model based on selected metabolites was the most effective (AUC = 0.72). The results obtained are promising for further development of the system of clinical decision support for the diagnosis of RRMS based on metabolic data.
| S-EPMC6849630 | biostudies-literature
Project description:In order to search for circRNA differentially expressed in colorectal cancer tissues and adjacent normal tissues, the cancer tissues and adjacent normal tissues of 3 patients with colorectal cancer(TNM:ⅡB) were selected for circRNA microarray detection. Differentially expressed circRNAs were screened.Then qRT-PCR was used to verify.
2022-04-27 | GSE197991 | GEO
Project description:BackgroundThe World Health Organization (WHO) has targeted a reduction in viral hepatitis-related mortality by 65% and incidence by 90% by 2030, necessitating enhanced hepatitis B treatment and prevention programmes in low- and middle-income countries. Hepatitis B e antigen (HBeAg) status is used in the assessment of eligibility for antiviral treatment and for prevention of mother-to-child transmission (PMTCT). Accordingly, the WHO has classified HBeAg rapid diagnostic tests (RDTs) as essential medical devices.MethodsWe assessed the performance characteristics of three commercially available HBeAg RDTs (SD Bioline, Alere, South Africa; Creative Diagnostics, USA; and Biopanda Reagents, UK) in two hepatitis B surface antigen-positive cohorts in Blantyre, Malawi: participants of a community study (n = 100) and hospitalised patients with cirrhosis or hepatocellular carcinoma (n = 94). Two investigators, blinded to the reference test result, independently assessed each assay. We used an enzyme-linked immunoassay (Monolisa HBeAg, Bio-Rad, France) as a reference test and quantified HBeAg concentration using dilutions of the WHO HBeAg standard. We related the findings to HBV DNA levels, and evaluated treatment eligibility using the TREAT-B score.ResultsAmong 194 HBsAg positive patients, median age was 37 years, 42% were femaleand 26% were HIV co-infected. HBeAg prevalence was 47/194 (24%). The three RDTs showed diagnostic sensitivity of 28% (95% CI 16-43), 53% (38-68) and 72% (57-84) and specificity of 96-100% for detection of HBeAg. Overall inter-rater agreement κ statistic was high at 0.9-1.0. Sensitivity for identifying patients at the threshold where antiviral treatment is recommended for PMTCT, with HBV DNA > 200,000 IU/ml (39/194; 20%), was 22, 49 and 54% respectively. Using the RDTs in place of the reference HBeAg assay resulted in 3/43 (9%), 5/43 (12%) and 8/43 (19%) of patients meeting the TREAT-B treatment criteria being misclassified as ineligible for treatment. A relationship between HBeAg concentration and HBeAg detection by RDT was observed. A minimum HBeAg concentration of 2.2-3.1 log10IU/ml was required to yield a reactive RDT.ConclusionsCommercially available HBeAg RDTs lack sufficient sensitivity to accurately classify hepatitis B patients in Malawi. This has implications for hepatitis B public health programs in sub-Saharan Africa. Alternative diagnostic assays are recommended.
| S-EPMC8157469 | biostudies-literature