Dataset Information

A general framework for inference on algorithm-agnostic variable importance.

ABSTRACT: In many applications, it is of interest to assess the relative contribution of features (or subsets of features) toward the goal of predicting a response - in other words, to gauge the variable importance of features. Most recent work on variable importance assessment has focused on describing the importance of features within the confines of a given prediction algorithm. However, such assessment does not necessarily characterize the prediction potential of features, and may provide a misleading reflection of the intrinsic value of these features. To address this limitation, we propose a general framework for nonparametric inference on interpretable algorithm-agnostic variable importance. We define variable importance as a population-level contrast between the oracle predictiveness of all available features versus all features except those under consideration. We propose a nonparametric efficient estimation procedure that allows the construction of valid confidence intervals, even when machine learning techniques are used. We also outline a valid strategy for testing the null importance hypothesis. Through simulations, we show that our proposal has good operating characteristics, and we illustrate its use with data from a study of an antibody against HIV-1 infection.

SUBMITTER: Williamson BD

PROVIDER: S-EPMC10652709 | biostudies-literature | 2023

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

A general framework for inference on algorithm-agnostic variable importance.

Williamson Brian D BD Gilbert Peter B PB Simon Noah R NR Carone Marco M

Journal of the American Statistical Association 20220105 543

In many applications, it is of interest to assess the relative contribution of features (or subsets of features) toward the goal of predicting a response - in other words, to gauge the variable importance of features. Most recent work on variable importance assessment has focused on describing the importance of features within the confines of a given prediction algorithm. However, such assessment does not necessarily characterize the prediction potential of features, and may provide a misleading ...[more]

PMID: 37982008

Similar Datasets

Project description:MotivationLocal ancestry inference (LAI) is the high resolution prediction of ancestry labels along a DNA sequence. LAI is important in the study of human history and migrations, and it is beginning to play a role in precision medicine applications including ancestry-adjusted genome-wide association studies (GWASs) and polygenic risk scores (PRSs). Existing LAI models do not generalize well between species, chromosomes or even ancestry groups, requiring re-training for each different setting. Furthermore, such methods can lack interpretability, which is an important element in each of these applications.ResultsWe present SALAI-Net, a portable statistical LAI method that can be applied on any set of species and ancestries (species-agnostic), requiring only haplotype data and no other biological parameters. Inspired by identity by descent methods, SALAI-Net estimates population labels for each segment of DNA by performing a reference matching approach, which leads to an interpretable and fast technique. We benchmark our models on whole-genome data of humans and we test these models' ability to generalize to dog breeds when trained on human data. SALAI-Net outperforms previous methods in terms of balanced accuracy, while generalizing between different settings, species and datasets. Moreover, it is up to two orders of magnitude faster and uses considerably less RAM memory than competing methods.Availability and implementationWe provide an open source implementation and links to publicly available data at github.com/AI-sandbox/SALAI-Net. Data is publicly available as follows: https://www.internationalgenome.org (1000 Genomes), https://www.simonsfoundation.org/simons-genome-diversity-project (Simons Genome Diversity Project), https://www.sanger.ac.uk/resources/downloads/human/hapmap3.html (HapMap), ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516 (Human Genome Diversity Project) and https://www.ncbi.nlm.nih.gov/bioproject/PRJNA448733 (Canid genomes).Supplementary informationSupplementary data are available from Bioinformatics online.

Project description:Machine learning has been successfully applied in recent years to screen materials for a variety of applications. However, despite recent advances, most screening-based machine learning approaches are limited in generality and transferability, requiring new models to be created from scratch for each new application. This is particularly apparent in catalysis, where there are many possible intermediates and transition states of interest in addition to a large number of potential catalytic materials. In this work, we developed a new machine learning framework that is built on chemical principles and allows the creation of general, interpretable, reusable models. Our new architecture uses latent variables to create a set of submodels that each take on a relatively simple learning task, leading to higher data efficiency and promoting transfer learning. This architecture infuses fundamental chemical principles, such as the existence of elements as discrete entities. We show that this architecture allows for the creation of models that can be reused for many different applications, providing significant improvements in efficiency and convenience. For example, our architecture allows simultaneous prediction of adsorption energies for many adsorbates on a broad array of alloy surfaces with mean absolute errors (MAEs) around 0.20-0.25 eV. The integration of latent variables provides physical interpretability, as predictions can be explained in terms of the learned chemical environment as represented by the latent space. Further, these latent variables also serve as new feature representations, allowing efficient transfer learning. For example, new models with useful levels of accuracy can be created with less than 10 data points, including transfer learning to an experimental data set with an MAE less than 0.15 eV. Lastly, we show that our new machine learning architecture is general and robust enough to handle heterogeneous and multifidelity data sets, allowing researchers to leverage existing data sets to speed up screening using their own computational setup.

Dataset Information

A general framework for inference on algorithm-agnostic variable importance.

Publications

A general framework for inference on algorithm-agnostic variable importance.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets