Effects of machine learning errors on human decision-making: manipulations of model accuracy, error types, and error importance
Ontology highlight
ABSTRACT:
SUBMITTER: Matzen L
PROVIDER: S-EPMC11345344 | biostudies-literature | 2024 Aug
REPOSITORIES: biostudies-literature
Similar Datasets
Project description:Machine learning (ML) has the potential to transform patient care and outcomes. However, there are important differences between measuring the performance of ML models in silico and usefulness at the point of care. One lens to use to evaluate models during early development is actionability, which is currently undervalued. We propose a metric for actionability intended to be used before the evaluation of calibration and ultimately decision curve analysis and calculation of net benefit. Our metric should be viewed as part of an overarching effort to increase the number of pragmatic tools that identify a model's possible clinical impacts.
| S-EPMC9871014 | biostudies-literature
Project description:Humans can learn under a wide variety of feedback conditions. Reinforcement learning (RL), where a series of rewarded decisions must be made, is a particularly important type of learning. Computational and behavioral studies of RL have focused mainly on Markovian decision processes, where the next state depends on only the current state and action. Little is known about non-Markovian decision making, where the next state depends on more than the current state and action. Learning is non-Markovian, for example, when there is no unique mapping between actions and feedback. We have produced a model based on spiking neurons that can handle these non-Markovian conditions by performing policy gradient descent [1]. Here, we examine the model's performance and compare it with human learning and a Bayes optimal reference, which provides an upper-bound on performance. We find that in all cases, our population of spiking neurons model well-describes human performance.
| S-EPMC4405578 | biostudies-literature
Project description:Machine learning and artificial intelligence have entered biomedical decision-making for diagnostics, prognostics, or therapy recommendations. However, these methods need to be interpreted with care because of the severe consequences for patients. In contrast to human decision-making, computational models typically make a decision also with low confidence. Machine learning with abstention better reflects human decision-making by introducing a reject option for samples with low confidence. The abstention intervals are typically symmetric intervals around the decision boundary. In the current study, we use asymmetric abstention intervals, which we demonstrate to be better suited for biomedical data that is typically highly imbalanced. We evaluate symmetric and asymmetric abstention on three real-world biomedical datasets and show that both approaches can significantly improve classification performance. However, asymmetric abstention rejects as many or fewer samples compared to symmetric abstention and thus, should be used in imbalanced data.
| S-EPMC8549182 | biostudies-literature
Project description: Not available
| S-EPMC11322186 | biostudies-literature
Project description: Not available
| S-EPMC11322447 | biostudies-literature
Project description:Background:The number of cases from the coronavirus disease 2019 (COVID-19) global pandemic has overwhelmed existing medical facilities and forced clinicians, patients, and families to make pivotal decisions with limited time and information. Main body:While machine learning (ML) methods have been previously used to augment clinical decisions, there is now a demand for "Emergency ML." Throughout the patient care pathway, there are opportunities for ML-supported decisions based on collected vitals, laboratory results, medication orders, and comorbidities. With rapidly growing datasets, there also remain important considerations when developing and validating ML models. Conclusion:This perspective highlights the utility of evidence-based prediction tools in a number of clinical settings, and how similar models can be deployed during the COVID-19 pandemic to guide hospital frontlines and healthcare administrators to make informed decisions about patient care and managing hospital volume.
| S-EPMC7347420 | biostudies-literature
Project description:The deployment of machine learning for tasks relevant to complementing standard of care and advancing tools for precision health has gained much attention in the clinical community, thus meriting further investigations into its broader use. In an introduction to predictive modelling using machine learning, we conducted a review of the recent literature that explains standard taxonomies, terminology and central concepts to a broad clinical readership. Articles aimed at readers with little or no prior experience of commonly used methods or typical workflows were summarised and key references are highlighted. Continual interdisciplinary developments in data science, biostatistics and epidemiology also motivated us to further discuss emerging topics in predictive and data-driven (hypothesis-less) analytics with machine learning. Through two methodological deep dives using examples from precision psychiatry and outcome prediction after lymphoma, we highlight how the use of, for example, natural language processing can outperform established clinical risk scores and aid dynamic prediction and adaptive care strategies. Such realistic and detailed examples allow for critical analysis of the importance of new technological advances in artificial intelligence for clinical decision-making. New clinical decision support systems can assist in prevention and care by leveraging precision medicine.
| S-EPMC9544754 | biostudies-literature
Project description:Estimating the relationships between individuals is one of the fundamental challenges in many fields. In particular, relationship estimation could provide valuable information for missing persons cases. The recently developed investigative genetic genealogy approach uses high-density single nucleotide polymorphisms (SNPs) to determine close and more distant relationships, in which hundreds of thousands to tens of millions of SNPs are generated either by microarray genotyping or whole-genome sequencing. The current studies usually assume the SNP profiles were generated with minimum errors. However, in the missing person cases, the DNA samples can be highly degraded, and the SNP profiles generated from these samples usually contain lots of errors. In this study, a robust machine learning approach was developed for estimating the relationships with high error SNP profiles. In this approach, a hierarchical classification strategy was employed first to classify the relationships by degree and then the relationship types within each degree separately. As for each classification, feature selection was implemented to gain better performance. Both simulated and real data sets with various genotyping error rates were utilized in evaluating this approach, and the accuracies of this approach were higher than individual measures; namely, this approach was more accurate and robust than the individual measures for SNP profiles with genotyping errors. In addition, the highest accuracy could be obtained by providing the same genotyping error rates in train and test sets, and thus estimating genotyping errors of the SNP profiles is critical to obtaining high accuracy of relationship estimation.
2022-07-27 | GSE209804 | GEO
Project description:Antimicrobial resistance (AMR) and healthcare associated infections pose a significant threat globally. One key prevention strategy is to follow antimicrobial stewardship practices, in particular, to maximise targeted oral therapy and reduce the use of indwelling vascular devices for intravenous (IV) administration. Appreciating when an individual patient can switch from IV to oral antibiotic treatment is often non-trivial and not standardised. To tackle this problem we created a machine learning model to predict when a patient could switch based on routinely collected clinical parameters. 10,362 unique intensive care unit stays were extracted and two informative feature sets identified. Our best model achieved a mean AUROC of 0.80 (SD 0.01) on the hold-out set while not being biased to individuals protected characteristics. Interpretability methodologies were employed to create clinically useful visual explanations. In summary, our model provides individualised, fair, and interpretable predictions for when a patient could switch from IV-to-oral antibiotic treatment. Prospectively evaluation of safety and efficacy is needed before such technology can be applied clinically.
| S-EPMC10787786 | biostudies-literature
Project description:Aging is a natural process with varying effects. As we grow older, our bodies become more susceptible to aging-associated diseases. These diseases, individually or collectively, lead to the formation of distinct aging phenotypes. Identifying these aging phenotypes and understanding the complex interplay between coexistent diseases would facilitate more personalized patient management, a better prognosis, and a prolonged lifespan. Many studies distinguish between successful aging and frailty. However, this simple distinction fails to reflect the diversity of underlying causes. In this study, we sought to establish the underlying causes of frailty and determine the patterns in which these causes converge to form aging phenotypes. We conducted a comprehensive geriatric examination, cognitive assessment, and survival analysis of 2,688 long-living adults (median age = 92 years). The obtained data were clustered and used as input data for the Aging Phenotype Calculator, a multiclass classification model validated on an independent dataset of 96 older adults. The accuracy of the model was assessed using the receiver operating characteristic curve and the area under the curve. Additionally, we analyzed socioeconomic factors that could contribute to specific aging patterns. We identified five aging phenotypes: non-frailty, multimorbid frailty, metabolic frailty, cognitive frailty, and functional frailty. For each phenotype, we determined the underlying diseases and conditions and assessed the survival rate. Additionally, we provided management recommendations for each of the five phenotypes based on their distinct features and associated challenges. The identified aging phenotypes may facilitate better-informed decision-making. The Aging Phenotype Calculator (ROC AUC = 92%) may greatly assist geriatricians in patient management.
| S-EPMC11745442 | biostudies-literature