Applicability Area: A novel utility-based approach for evaluating predictive models, beyond discrimination.
Ontology highlight
ABSTRACT:
SUBMITTER: Liu S
PROVIDER: S-EPMC10785877 | biostudies-literature | 2023
REPOSITORIES: biostudies-literature
Publications
Applicability Area: A novel utility-based approach for evaluating predictive models, beyond discrimination.
Liu Star S Wei Shixiong S Lehmann Harold P HP
AMIA ... Annual Symposium proceedings. AMIA Symposium 20230101
Translating prediction models into practice and supporting clinicians' decision-making demand demonstration of clinical value. Existing approaches to evaluating machine learning models emphasize discriminatory power, which is only a part of the medical decision problem. We propose the Applicability Area (ApAr), a decision-analytic utility-based approach to evaluating predictive models that communicate the range of prior probability and test cutoffs for which the model has positive utility; large ...[more]
Similar Datasets
Project description:The primate visual system has an exquisite ability to discriminate partially occluded shapes. Recent electrophysiological recordings suggest that response dynamics in intermediate visual cortical area V4, shaped by feedback from prefrontal cortex (PFC), may play a key role. To probe the algorithms that may underlie these findings, we build and test a model of V4 and PFC interactions based on a hierarchical predictive coding framework. We propose that probabilistic inference occurs in two steps. Initially, V4 responses are driven solely by bottom-up sensory input and are thus strongly influenced by the level of occlusion. After a delay, V4 responses combine both feedforward input and feedback signals from the PFC; the latter reflect predictions made by PFC about the visual stimulus underlying V4 activity. We find that this model captures key features of V4 and PFC dynamics observed in experiments. Specifically, PFC responses are strongest for occluded stimuli and delayed responses in V4 are less sensitive to occlusion, supporting our hypothesis that the feedback signals from PFC underlie robust discrimination of occluded shapes. Thus, our study proposes that area V4 and PFC participate in hierarchical inference, with feedback signals encoding top-down predictions about occluded shapes.
| S-EPMC5930045 | biostudies-literature
Project description:Many wheat (Triticum aestivum L.) production regions are threatened annually by drought stress. Carbon isotope discrimination (Δ) has been identified as a potentially useful trait in breeding for improved drought tolerance in certain environments. Broad use of Δ as a selection criterion is limited, however, mainly due to an inconsistent relationship observed between grain yield and Δ and, to a lesser extent, because of the high resource demand associated with phenotyping. The efficiency of selection may be improved by the identification and verification of molecular markers for use in marker-assisted selection (MAS), and a reliable relationship to grain yield may be established based on a location's total amount and distribution of precipitation over the growing season. Given the environmental variability in precipitation dynamics, it is necessary to evaluate this relationship in target breeding environments. In this study, grain Δ was collected on a panel of 480 advanced soft white winter wheat varieties grown in five Pacific Northwest environments. A genome-wide association study approach was used to evaluate the amenability of grain Δ to MAS. The genetic architecture of grain Δ was determined to be characterized by multiple, small effect marker-trait associations with limited repeatability across environments, suggesting that MAS will be ineffective at improving Δ selection efficiency. Further, the relationship between grain yield and Δ ranged from neutral (r = -0.01) to moderately positive (r = 0.44) in the target environments. Such moderate correlations, coupled with variability in this relationship, indicate that direct selection for Δ may not be beneficial.
| S-EPMC7706333 | biostudies-literature
Project description:Accurate risk prediction has been the central goal in many studies of survival outcomes. In the presence of multiple risk factors, a censored regression model can be employed to estimate a risk prediction rule. Before the prediction tool can be popularized for practical use, it is crucial to rigorously assess its prediction performance. In our motivating example, researchers are interested in developing and validating a risk prediction tool to identify future lung cancer cases by integrating demographic information, disease characteristics and smoking-related data. Considering the long latency period of cancer, it is desirable for a prediction tool to achieve discriminative performance that does not weaken over time. We propose estimation and inferential procedures to comprehensively assess both the overall predictive discrimination and the temporal pattern of an estimated prediction rule. The proposed methods readily accommodate commonly used censored regression models, including the Cox proportional hazards model and the accelerated failure time model. The estimators are consistent and asymptotically normal, and reliable variance estimators are also developed. The proposed methods offer an informative tool for inferring time-dependent predictive discrimination, as well as for comparing the discrimination performance between candidate models. Applications of the proposed methods demonstrate enduring performance of the risk prediction tool in the PLCO study and detected decaying performance in a study of liver disease.
| S-EPMC10084512 | biostudies-literature
Project description:Evaluation of cost-utility is critical in assessing the medical utility of predictive or prognostic biomarkers. Current methods involve complex state-transition models, requiring comprehensive data inputs. We propose a simplified decision-analytic tool to explore the relative effect of factors contributing to the cost-utility of a biomarker. We derived a cost-utility metric, the "test incremental cost-effectiveness ratio" (TICER) for biomarker-guided treatment compared to no biomarker use. This method uses data inputs readily accessible through clinical literature. We compared our results with traditional cost-effectiveness analysis of predictive biomarkers for established (HER2-guided trastuzumab, ALK-guided crizotinib, OncotypeDX-guided adjuvant chemotherapy) and emerging (ROS1-guided crizotinib) targeted treatments. We conducted sensitivity analysis to determine which factors had the greatest impact on TICER estimates. Base case TICER for HER2 was $149,600/quality-adjusted life year (QALY), for ALK was $22,200/QALY, and for OncotypeDX was $11,600/QALY, consistent with literature-reported estimates ($180,000/QALY, $202,800/QALY, $8900/QALY, respectively). Base case TICER for ROS1-guided crizotinib was $205,900/QALY. Generally, when treatment cost is considerably greater than biomarker testing costs, TICER is driven by clinical outcomes and health-related quality of life, while biomarker prevalence and treatment cost have a lesser effect. Our simplified decision-analytic approach produces values consistent with existing cost-effectiveness analyses. Our results suggest that biomarker value is mostly driven by the clinical efficacy of the targeted agent. A user-friendly web tool for complete TICER analysis has been made available for open use at http://medicine.yale.edu/lab/pusztai/ticer/ .
| S-EPMC5990969 | biostudies-literature
Project description:ObjectivesTo summarise multivariable predictive models for 30-day unplanned hospital readmissions (UHRs) in paediatrics, describe their performance and completeness in reporting, and determine their potential for application in practice.DesignSystematic review.Data sourceCINAHL, Embase and PubMed up to 7 October 2021.Eligibility criteriaEnglish or German language studies aiming to develop or validate a multivariable predictive model for 30-day paediatric UHRs related to all-cause, surgical conditions or general medical conditions were included.Data extraction and synthesisStudy characteristics, risk factors significant for predicting readmissions and information about performance measures (eg, c-statistic) were extracted. Reporting quality was addressed by the 'Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis' (TRIPOD) adherence form. The study quality was assessed by applying six domains of potential biases. Due to expected heterogeneity among the studies, the data were qualitatively synthesised.ResultsBased on 28 studies, 37 predictive models were identified, which could potentially be used for determining individual 30-day UHR risk in paediatrics. The number of study participants ranged from 190 children to 1.4 million encounters. The two most common significant risk factors were comorbidity and (postoperative) length of stay. 23 models showed a c-statistic above 0.7 and are primarily applicable at discharge. The median TRIPOD adherence of the models was 59% (P25-P75, 55%-69%), ranging from a minimum of 33% to a maximum of 81%. Overall, the quality of many studies was moderate to low in all six domains.ConclusionPredictive models may be useful in identifying paediatric patients at increased risk of readmission. To support the application of predictive models, more attention should be placed on completeness in reporting, particularly for those items that may be relevant for implementation in practice.
| S-EPMC8968996 | biostudies-literature
Project description:ImportanceMachine learning for augmented screening of perinatal mood and anxiety disorders (PMADs) requires thorough consideration of clinical biases embedded in electronic health records (EHRs) and rigorous evaluations of model performance.ObjectiveTo mitigate bias in predictive models of PMADs trained on commonly available EHRs.Design, setting, and participantsThis diagnostic study collected data as part of a quality improvement initiative from 2020 to 2023 at Cedars-Sinai Medical Center in Los Angeles, California. The study inclusion criteria were birthing patients aged 14 to 59 years with live birth records and admission to the postpartum unit or the maternal-fetal care unit after delivery.ExposurePatient-reported race and ethnicity (7 levels) obtained through EHRs.Main outcomes and measuresLogistic regression, random forest, and extreme gradient boosting models were trained to predict 2 binary outcomes: moderate to high-risk (positive) screen assessed using the 9-item Patient Health Questionnaire (PHQ-9), and the Edinburgh Postnatal Depression Scale (EPDS). Each model was fitted with or without reweighing data during preprocessing and evaluated through repeated K-fold cross validation. In every iteration, each model was evaluated on its area under the receiver operating curve (AUROC) and on 2 fairness metrics: demographic parity (DP), and difference in false negatives between races and ethnicities (relative to non-Hispanic White patients).ResultsAmong 19 430 patients in this study, 1402 (7%) identified as African American or Black, 2371 (12%) as Asian American and Pacific Islander; 1842 (10%) as Hispanic White, 10 942 (56.3%) as non-Hispanic White, 606 (3%) as multiple races, 2146 (11%) as other (not further specified), and 121 (<1%) did not provide this information. The mean (SD) age was 34.1 (4.9) years, and all patients identified as female. Racial and ethnic minority patients were significantly more likely than non-Hispanic White patients to screen positive on both the PHQ-9 (odds ratio, 1.47 [95% CI, 1.23-1.77]) and the EPDS (odds ratio, 1.38 [95% CI, 1.20-1.57]). Mean AUROCs ranged from 0.610 to 0.635 without reweighing (baseline), and from 0.602 to 0.622 with reweighing. Baseline models predicted significantly greater prevalence of postpartum depression for patients who were not non-Hispanic White relative to those who were (mean DP, 0.238 [95% CI, 0.231-0.244]; P < .001) and displayed significantly lower false-negative rates (mean difference, -0.184 [95% CI, -0.195 to -0.174]; P < .001). Reweighing significantly reduced differences in DP (mean DP with reweighing, 0.022 [95% CI, 0.017-0.026]; P < .001) and false-negative rates (mean difference with reweighing, 0.018 [95% CI, 0.008-0.028]; P < .001) between racial and ethnic groups.Conclusions and relevanceIn this diagnostic study of predictive models of postpartum depression, clinical prediction models trained to predict psychometric screening results from commonly available EHRs achieved modest performance and were less likely to widen existing health disparities in PMAD diagnosis and potentially treatment. These findings suggest that is critical for researchers and physicians to consider their model design (eg, desired target and predictor variables) and evaluate model bias to minimize health disparities.
| S-EPMC11615713 | biostudies-literature
Project description:Molecular clock models are commonly used to estimate evolutionary rates and timescales from nucleotide sequences. The goal of these models is to account for rate variation among lineages, such that they are assumed to be adequate descriptions of the processes that generated the data. A common approach for selecting a clock model for a data set of interest is to examine a set of candidates and to select the model that provides the best statistical fit. However, this can lead to unreliable estimates if all the candidate models are actually inadequate. For this reason, a method of evaluating absolute model performance is critical. We describe a method that uses posterior predictive simulations to assess the adequacy of clock models. We test the power of this approach using simulated data and find that the method is sensitive to bias in the estimates of branch lengths, which tends to occur when using underparameterized clock models. We also compare the performance of the multinomial test statistic, originally developed to assess the adequacy of substitution models, but find that it has low power in identifying the adequacy of clock models. We illustrate the performance of our method using empirical data sets from coronaviruses, simian immunodeficiency virus, killer whales, and marine turtles. Our results indicate that methods of investigating model adequacy, including the one proposed here, should be routinely used in combination with traditional model selection in evolutionary studies. This will reveal whether a broader range of clock models to be considered in phylogenetic analysis.
| S-EPMC7107558 | biostudies-literature
Project description:Glaucoma is a progressive optic neuropathy that can lead to irreversible blindness. Its main risk factor is elevated intraocular pressure (IOP). The trabecular meshwork (TM) acts as a filter between the anterior chamber of the eye and the aqueous humor collecting ducts, and dysfunction of this meshwork is responsible for the increased IOP in primary open-angle glaucoma (POAG). Considering that the culture conditions of human TM cells (HTMC) influence gene expression, we used human TM explants (HTMEx), which most closely mimic physiological conditions, to study the transcriptome of HTMC. The transforming growth factor-beta 2 (TGF-β2) signaling pathway has been implicated in the pathophysiology of POAG. To better characterize the role of TGF-β2 in this pathophysiology, we used bulk RNA sequencing and immunohistological analyses to establish gene signatures of TGF-β2-exposed HTMEx and correlate them with morphological alterations. We identified differentially upregulated genes primarily involved in ECM regulation, as well as profibrotic TGF-β signaling pathways, confirmed using confocal microscopy to highlight changes in trabecular architecture, TGFβ2-induced F-actin rearrangements, and extracellular matrix (ECM) deposition. Enrichment analysis also revealed modulations of gene expression related to cytoskeletal organization, as well as activation of the bone morphogenic protein (BMP) and Wnt signaling pathways in response to TGF-β2.
2023-07-03 | GSE236302 | GEO
Project description:ObjectiveThe clinical symptoms of schizophrenia are associated with serious social, quality of life and functioning alterations. Typically, data on health utilities are not available in clinical studies in schizophrenia. This makes the economic evaluation of schizophrenia treatments challenging. The purpose of this article was to provide a mapping function to predict unobserved utility values in patients with schizophrenia from the available clinical and socio-demographic information.MethodsThe analysis was performed using data from EuroSC, a 2-year, multi-centre, cohort study conducted in France (N = 288), Germany (N = 618), and the UK (N = 302), totalling 1208 patients. Utility was calculated based on the EQ-5D questionnaire. The relationships between the utility values and the patients' socio-demographic and clinical characteristics (Positive and Negative Syndrome Scale--PANSS, Calgary Depression Scale for Schizophrenia--CDSS, Global Assessment of Functioning--GAF, extra-pyramidal symptoms measured by Barnes Akathisia Scale-BAS, age, sex, country, antipsychotic type) were modelled using a random and a fixed individual effects panel linear model.ResultsThe analysis demonstrated the prediction ability of the used parameters for estimating utility measures in patients with schizophrenia. Although there are small variations between countries, the same variables appear to be the key predictors. From a clinical perspective, age, gender, psychopathology, and depression were the most important predictors associated with the EQ-5D.ConclusionThis paper proposed a reliable, robust and easy-to-apply mapping method to estimate EQ-5D utilities based on demographic and clinical measures in schizophrenia.
| S-EPMC4830865 | biostudies-literature
Project description:Accurate estimation of carrier probabilities of cancer susceptibility gene mutations is an important part of pre-test genetic counselling. Many predictive models are available but their applicability in the Asian population is uncertain. We evaluated the performance of five BRCA mutation risk predictive models in a Chinese cohort of 647 women, who underwent germline DNA sequencing of a cancer susceptibility gene panel. Using areas under the curve (AUCs) on receiver operating characteristics (ROC) curves as performance measures, the models did comparably well as in western cohorts (BOADICEA 0.75, BRCAPRO 0.73, Penn II 0.69, Myriad 0.68). For unaffected women with family history of breast or ovarian cancer (n = 144), BOADICEA, BRCAPRO, and Tyrer-Cuzick models had excellent performance (AUC 0.93, 0.92, and 0.92, respectively). For women with both personal and family history of breast or ovarian cancer (n = 241), all models performed fairly well (BOADICEA 0.79, BRCAPRO 0.79, Penn II 0.75, Myriad 0.70). For women with personal history of breast or ovarian cancer but no family history (n = 262), most models did poorly. Between the two well-performed models, BOADICEA underestimated mutation risks while BRCAPRO overestimated mutation risks (expected/observed ratio 0.67 and 2.34, respectively). Among 424 women with personal history of breast cancer and available tumor ER/PR/HER2 data, the predictive models performed better for women with triple negative breast cancer (AUC 0.74 to 0.80) than for women with luminal or HER2 overexpressed breast cancer (AUC 0.63 to 0.69). However, incorporating ER/PR/HER2 status into the BOADICEA model calculation did not improve its predictive accuracy.
| S-EPMC6629692 | biostudies-literature