Bayesian modeling of human-AI complementarity.
Ontology highlight
ABSTRACT:
SUBMITTER: Steyvers M
PROVIDER: S-EPMC8931210 | biostudies-literature | 2022 Mar
REPOSITORIES: biostudies-literature
Publications
Bayesian modeling of human-AI complementarity.
Steyvers Mark M Tejeda Heliodoro H Kerrigan Gavin G Smyth Padhraic P
Proceedings of the National Academy of Sciences of the United States of America 20220311 11
SignificanceWith the increase in artificial intelligence in real-world applications, there is interest in building hybrid systems that take both human and machine predictions into account. Previous work has shown the benefits of separately combining the predictions of diverse machine classifiers or groups of people. Using a Bayesian modeling framework, we extend these results by systematically investigating the factors that influence the performance of hybrid combinations of human and machine cl ...[more]
Similar Datasets
Project description:MotivationThe recent emergence of cloud laboratories-collections of automated wet-lab instruments that are accessed remotely, presents new opportunities to apply Artificial Intelligence and Machine Learning in scientific research. Among these is the challenge of automating the process of optimizing experimental protocols to maximize data quality.ResultsWe introduce a new deterministic algorithm, called PaRallel OptimizaTiOn for ClOud Laboratories (PROTOCOL), that improves experimental protocols via asynchronous, parallel Bayesian optimization. The algorithm achieves exponential convergence with respect to simple regret. We demonstrate PROTOCOL in both simulated and real-world cloud labs. In the simulated lab, it outperforms alternative approaches to Bayesian optimization in terms of its ability to find optimal configurations, and the number of experiments required to find the optimum. In the real-world lab, the algorithm makes progress toward the optimal setting.Data availability and implementationPROTOCOL is available as both a stand-alone Python library, and as part of a R Shiny application at https://github.com/clangmead/PROTOCOL. Data are available at the same repository.Supplementary informationSupplementary data are available at Bioinformatics online.
| S-EPMC8275326 | biostudies-literature
Project description:The characterization of intrinsically disordered proteins is challenging because accurate models of these systems require a description of both their thermally accessible conformers and the associated relative stabilities or weights. These structures and weights are typically chosen such that calculated ensemble averages agree with some set of prespecified experimental measurements; however, the large number of degrees of freedom in these systems typically leads to multiple conformational ensembles that are degenerate with respect to any given set of experimental observables. In this work we demonstrate that estimates of the relative stabilities of conformers within an ensemble are often incorrect when one does not account for the underlying uncertainty in the estimates themselves. Therefore, we present a method for modeling the conformational properties of disordered proteins that estimates the uncertainty in the weights of each conformer. The Bayesian weighting (BW) formalism incorporates information from both experimental data and theoretical predictions to calculate a probability density over all possible ways of weighting the conformers in the ensemble. This probability density is then used to estimate the values of the weights. A unique and powerful feature of the approach is that it provides a built-in error measure that allows one to assess the accuracy of the ensemble. We validate the approach using reference ensembles constructed from the five-residue peptide met-enkephalin and then apply the BW method to construct an ensemble of the K18 isoform of the tau protein. Using this ensemble, we indentify a specific pattern of long-range contacts in K18 that correlates with the known aggregation properties of the sequence.
| S-EPMC2956375 | biostudies-other
Project description:Human settlement of the Caribbean represents the only example in the Americas of peoples colonizing islands that were not visible from surrounding mainland areas or other islands. Unfortunately, many interpretive models have relied on radiocarbon determinations that do not meet standard criteria for reporting because they lack critical information or sufficient provenience, often leading to specious interpretations. We have collated 2484 radiocarbon determinations, assigned them to classes based on chronometric hygiene criteria, and constructed Bayesian colonization models of the acceptable determinations to examine patterns of initial settlement. Colonization estimates for 26 islands indicate that (i) the region was settled in two major population dispersals that likely originated from South America; (ii) colonists reached islands in the northern Antilles before the southern islands; and (iii) the results support the southward route hypothesis and refute the "stepping-stone model."
| S-EPMC6957329 | biostudies-literature
Project description:To make decisions in a social context, humans have to predict the behavior of others, an ability that is thought to rely on having a model of other minds known as "theory of mind." Such a model becomes especially complex when the number of people one simultaneously interacts with is large and actions are anonymous. Here, we present results from a group decision-making task known as the volunteer's dilemma and demonstrate that a Bayesian model based on partially observable Markov decision processes outperforms existing models in quantitatively predicting human behavior and outcomes of group interactions. Our results suggest that in decision-making tasks involving large groups with anonymous members, humans use Bayesian inference to model the "mind of the group," making predictions of others' decisions while also simulating the effects of their own actions on the group's dynamics in the future.
| S-EPMC6881156 | biostudies-literature
Project description:The rise of machine learning and deep learning technologies have allowed researchers to automate image classification. We describe a method that incorporates automated image classification and principal component analysis to evaluate computational models of biological structures. We use a computational model of the kinetochore to demonstrate our artificial-intelligence (AI)-assisted modeling method. The kinetochore is a large protein complex that connects chromosomes to the mitotic spindle to facilitate proper cell division. The kinetochore can be divided into two regions: the inner kinetochore, including proteins that interact with DNA; and the outer kinetochore, comprised of microtubule-binding proteins. These two kinetochore regions have been shown to have different distributions during metaphase in live budding yeast and therefore act as a test case for our forward modeling technique. We find that a simple convolutional neural net (CNN) can correctly classify fluorescent images of inner and outer kinetochore proteins and show a CNN trained on simulated, fluorescent images can detect difference in experimental images. A polymer model of the ribosomal DNA locus serves as a second test for the method. The nucleolus surrounds the ribosomal DNA locus and appears amorphous in live-cell, fluorescent microscopy experiments in budding yeast, making detection of morphological changes challenging. We show a simple CNN can detect subtle differences in simulated images of the ribosomal DNA locus, demonstrating our CNN-based classification technique can be used on a variety of biological structures.
| S-EPMC6868055 | biostudies-literature
Project description:We aim at modeling the appearance of distinct tags in a sequence of labeled objects. Common examples of this type of data include words in a corpus or distinct species in a sample. These sequential discoveries are often summarized via accumulation curves, which count the number of distinct entities observed in an increasingly large set of objects. We propose a novel Bayesian method for species sampling modeling by directly specifying the probability of a new discovery, therefore, allowing for flexible specifications. The asymptotic behavior and finite sample properties of such an approach are extensively studied. Interestingly, our enlarged class of sequential processes includes highly tractable special cases. We present a subclass of models characterized by appealing theoretical and computational properties, including one that shares the same discovery probability with the Dirichlet process. Moreover, due to strong connections with logistic regression models, the latter subclass can naturally account for covariates. We finally test our proposal on both synthetic and real data, with special emphasis on a large fungal biodiversity study in Finland. Supplementary materials for this article are available online.
| S-EPMC10947068 | biostudies-literature
Project description:MeDIP-seq is an enrichment-based DNA methylation profiling technique that measures abundance of methylated DNA. While this technique offers efficiency advantages over direct methylation profiling, it does not provide absolute quantification of DNAm necessary for cell type deconvolution. We introduce decemedip, a Bayesian hierarchical model for cell type deconvolution of methylated sequencing data that leverages reference atlases derived from direct methylation profiling. We demonstrate its accuracy and robustness through simulation studies and validation on cross-platform measurements, and highlight its utility in identifying tissue-specific and cancer-associated methylation signatures using MeDIP-seq profiling of patient-derived xenografts and cell-free DNA. decemedip is available at \url{https://github.com/nshen7/decemedip}.
2025-03-31 | GSE290990 | GEO
Project description:Proteochemometrics (PCM) is an approach for bioactivity predictive modeling which models the relationship between protein and chemical information. Gaussian Processes (GP), based on Bayesian inference, provide the most objective estimation of the uncertainty of the predictions, thus permitting the evaluation of the applicability domain (AD) of the model. Furthermore, the experimental error on bioactivity measurements can be used as input for this probabilistic model. In this study, we apply GP implemented with a panel of kernels on three various (and multispecies) PCM datasets. The first dataset consisted of information from 8 human and rat adenosine receptors with 10,999 small molecule ligands and their binding affinity. The second consisted of the catalytic activity of four dengue virus NS3 proteases on 56 small peptides. Finally, we have gathered bioactivity information of small molecule ligands on 91 aminergic GPCRs from 9 different species, leading to a dataset of 24,593 datapoints with a matrix completeness of only 2.43%. GP models trained on these datasets are statistically sound, at the same level of statistical significance as Support Vector Machines (SVM), with [Formula: see text] values on the external dataset ranging from 0.68 to 0.92, and RMSEP values close to the experimental error. Furthermore, the best GP models obtained with the normalized polynomial and radial kernels provide intervals of confidence for the predictions in agreement with the cumulative Gaussian distribution. GP models were also interpreted on the basis of individual targets and of ligand descriptors. In the dengue dataset, the model interpretation in terms of the amino-acid positions in the tetra-peptide ligands gave biologically meaningful results.
| S-EPMC4083135 | biostudies-literature
Project description:Artificial intelligence (AI) provides considerable opportunities to assist human work. However, one crucial challenge of human-AI collaboration is that many AI algorithms operate in a black-box manner where the way how the AI makes predictions remains opaque. This makes it difficult for humans to validate a prediction made by AI against their own domain knowledge. For this reason, we hypothesize that augmenting humans with explainable AI improves task performance in human-AI collaboration. To test this hypothesis, we implement explainable AI in the form of visual heatmaps in inspection tasks conducted by domain experts. Visual heatmaps have the advantage that they are easy to understand and help to localize relevant parts of an image. We then compare participants that were either supported by (a) black-box AI or (b) explainable AI, where the latter supports them to follow AI predictions when the AI is accurate or overrule the AI when the AI predictions are wrong. We conducted two preregistered experiments with representative, real-world visual inspection tasks from manufacturing and medicine. The first experiment was conducted with factory workers from an electronics factory, who performed [Formula: see text] assessments of whether electronic products have defects. The second experiment was conducted with radiologists, who performed [Formula: see text] assessments of chest X-ray images to identify lung lesions. The results of our experiments with domain experts performing real-world tasks show that task performance improves when participants are supported by explainable AI with heatmaps instead of black-box AI. We find that explainable AI as a decision aid improved the task performance by 7.7 percentage points (95% confidence interval [CI]: 3.3% to 12.0%, [Formula: see text]) in the manufacturing experiment and by 4.7 percentage points (95% CI: 1.1% to 8.3%, [Formula: see text]) in the medical experiment compared to black-box AI. These gains represent a significant improvement in task performance.
| S-EPMC11681242 | biostudies-literature
Project description:Artificial Intelligence is making rapid and remarkable progress in the development of more sophisticated and powerful systems. However, the acknowledgement of several problems with modern machine learning approaches has prompted a shift in AI benchmarking away from task-oriented testing (such as Chess and Go) towards ability-oriented testing, in which AI systems are tested on their capacity to solve certain kinds of novel problems. The Animal-AI Environment is one such benchmark which aims to apply the ability-oriented testing used in comparative psychology to AI systems. Here, we present the first direct human-AI comparison in the Animal-AI Environment, using children aged 6-10 (n = 52). We found that children of all ages were significantly better than a sample of 30 AIs across most of the tests we examined, as well as performing significantly better than the two top-scoring AIs, "ironbar" and "Trrrrr," from the Animal-AI Olympics Competition 2019. While children and AIs performed similarly on basic navigational tasks, AIs performed significantly worse in more complex cognitive tests, including detour tasks, spatial elimination tasks, and object permanence tasks, indicating that AIs lack several cognitive abilities that children aged 6-10 possess. Both children and AIs performed poorly on tool-use tasks, suggesting that these tests are challenging for both biological and non-biological machines.
| S-EPMC9172850 | biostudies-literature