An illustration of model agnostic explainability methods applied to environmental data.
Ontology highlight
ABSTRACT:
SUBMITTER: Wikle CK
PROVIDER: S-EPMC10187774 | biostudies-literature | 2023 Feb
REPOSITORIES: biostudies-literature
Publications
An illustration of model agnostic explainability methods applied to environmental data.
Wikle Christopher K CK Datta Abhirup A Hari Bhava Vyasa BV Boone Edward L EL Sahoo Indranil I Kavila Indulekha I Castruccio Stefano S Simmons Susan J SJ Burr Wesley S WS Chang Won W
Environmetrics 20221025 1
Historically, two primary criticisms statisticians have of machine learning and deep neural models is their lack of uncertainty quantification and the inability to do inference (i.e., to explain what inputs are important). Explainable AI has developed in the last few years as a sub-discipline of computer science and machine learning to mitigate these concerns (as well as concerns of fairness and transparency in deep modeling). In this article, our focus is on explaining which inputs are importan ...[more]
Similar Datasets
Project description:Understanding the inferences of data-driven, machine-learned models can be seen as a process that discloses the relationships between their input and output. These relationships consist and can be represented as a set of inference rules. However, the models usually do not explicit these rules to their end-users who, subsequently, perceive them as black-boxes and might not trust their predictions. Therefore, scholars have proposed several methods for extracting rules from data-driven machine-learned models to explain their logic. However, limited work exists on the evaluation and comparison of these methods. This study proposes a novel comparative approach to evaluate and compare the rulesets produced by five model-agnostic, post-hoc rule extractors by employing eight quantitative metrics. Eventually, the Friedman test was employed to check whether a method consistently performed better than the others, in terms of the selected metrics, and could be considered superior. Findings demonstrate that these metrics do not provide sufficient evidence to identify superior methods over the others. However, when used together, these metrics form a tool, applicable to every rule-extraction method and machine-learned models, that is, suitable to highlight the strengths and weaknesses of the rule-extractors in various applications in an objective and straightforward manner, without any human interventions. Thus, they are capable of successfully modelling distinctively aspects of explainability, providing to researchers and practitioners vital insights on what a model has learned during its training process and how it makes its predictions.
| S-EPMC8596373 | biostudies-literature
Project description:BackgroundFollowing the identification of a late mortality signal, the Food and Drug Administration (FDA) convened an advisory panel that concluded that additional clinical study data are needed to comprehensively evaluate the late mortality signal observed with the use of drug-coated balloons (DCB) and drug-eluting stent (DES). The objective of this review is to (1) identify and summarize the existing clinical and cohort studies assessing paclitaxel-coated DCBs and DESs, (2) describe and determine the quality of the available data sources for the evaluation of these devices, and (3) present methodologies that can be leveraged for proper signal discernment within available data sources.MethodsStudies and data sources were identified through comprehensive searches. original research studies, clinical trials, comparative studies, multicenter studies, and observational cohort studies written in the English language and published from January 2007 to November 2021, with a follow-up longer than 36 months, were included in the review. Data quality of available data sources identified was assessed in three groupings. Moreover, accepted data-driven methodologies that may help circumvent the limitations of the extracted studies and data sources were extracted and described.ResultsThere were 39 studies and data sources identified. This included 19 randomized clinical trials, nine single-arm studies, eight registries, three administrative claims, and electronic health records. Methodologies focusing on the use of existing premarket clinical data, the incorporation of all contributed patient time, the use of aggregated data, approaches for individual-level data, machine learning and artificial intelligence approaches, Bayesian approaches, and the combination of various datasets were summarized.ConclusionDespite the multitude of available studies over the course of eleven years following the first clinical trial, the FDA-convened advisory panel found them insufficient for comprehensively assessing the late-mortality signal. High-quality data sources with the capabilities of employing advanced statistical methodologies are needed to detect potential safety signals in a timely manner and allow regulatory bodies to act quickly when a safety signal is detected.
| S-EPMC10920218 | biostudies-literature
Project description:BackgroundGene set analysis is considered to be a way of improving our biological interpretation of the observed expression patterns. This paper describes different methods applied to analyse expression data from a chicken DNA microarray dataset.ResultsApplying different gene set analyses to the chicken expression data led to different ranking of the Gene Ontology terms tested. A method for prediction of possible annotations was applied.ConclusionBiological interpretation based on gene set analyses dependent on the statistical method used. Methods for predicting the possible annotations for genes with unknown function from the expression data at hand could be useful, but our results indicate that careful validation of the predictions is needed.
| S-EPMC2712751 | biostudies-literature
Project description:IntroductionSpiking neural networks (SNNs) are a model of computation that mimics the behavior of biological neurons. SNNs process event data (spikes) and operate more sparsely than artificial neural networks (ANNs), resulting in ultra-low latency and small power consumption. This paper aims to adapt and evaluate gradient-based explainability methods for SNNs, which were originally developed for conventional ANNs.MethodsThe adapted methods aim to create input feature attribution maps for SNNs trained through backpropagation that process either event-based spiking data or real-valued data. The methods address the limitations of existing work on explainability methods for SNNs, such as poor scalability, limited to convolutional layers, requiring the training of another model, and providing maps of activation values instead of true attribution scores. The adapted methods are evaluated on classification tasks for both real-valued and spiking data, and the accuracy of the proposed methods is confirmed through perturbation experiments at the pixel and spike levels.Results and discussionThe results reveal that gradient-based SNN attribution methods successfully identify highly contributing pixels and spikes with significantly less computation time than model-agnostic methods. Additionally, we observe that the chosen coding technique has a noticeable effect on the input features that will be most significant. These findings demonstrate the potential of gradient-based explainability methods for SNNs in improving our understanding of how these networks process information and contribute to the development of more efficient and accurate SNNs.
| S-EPMC10565802 | biostudies-literature
Project description:Complex traits like type 1 diabetes mellitus (T1DM) are generally taken to be under the influence of multiple genes interacting with each other to confer disease susceptibility and/or protection. Although novel methods are being developed, analyses of whole-genome scans are most often performed with multipoint methods that work under the assumption that multiple trait loci are unrelated to each other; that is, most models specify the effect of only one locus at a time. We have applied a novel approach, which includes decision-tree construction and artificial neural networks, to the analysis of T1DM genome-scan data. We demonstrate that this approach (1) allows identification of all major susceptibility loci identified by nonparametric linkage analysis, (2) identifies a number of novel regions as well as combinations of markers with predictive value for T1DM, and (3) may be useful in characterizing markers in linkage disequilibrium with protective-gene variants. Furthermore, the approach outlined here permits combined analyses of genetic-marker data and information on environmental and clinical covariates.
| S-EPMC1181942 | biostudies-literature
Project description:PurposeLimited investigation is available on the correlation between environmental phenols' exposure and estimated glomerular filtration rate (eGFR). Our target is established a robust and explainable machine learning (ML) model that associates environmental phenols' exposure with eGFR.MethodsOur datasets for constructing the associations between environmental phenols' and eGFR were collected from the National Health and Nutrition Examination Survey (NHANES, 2013-2016). Five ML models were contained and fine-tuned to eGFR regression by phenols' exposure. Regression evaluation metrics were used to extract the limitation of the models. The most effective model was then utilized for regression, with interpretation of its features carried out using shapley additive explanations (SHAP) and the game theory python package to represent the model's regression capacity.ResultsThe study identified the top-performing random forest (RF) regressor with a mean absolute error of 0.621 and a coefficient of determination of 0.998 among 3,371 participants. Six environmental phenols with eGFR in linear regression models revealed that the concentrations of triclosan (TCS) and bisphenol S (BPS) in urine were positively correlated with eGFR, and the correlation coefficients were β = 0.010 (p = 0.026) and β = 0.007 (p = 0.004) respectively. SHAP values indicate that BPS (1.38), bisphenol F (BPF) (0.97), 2,5-dichlorophenol (0.87), TCS (0.78), BP3 (0.60), bisphenol A (BPA) (0.59) and 2,4-dichlorophenol (0.47) in urinary contributed to the model.ConclusionThe RF model was efficient in identifying a correlation between phenols' exposure and eGFR among United States NHANES 2013-2016 participants. The findings indicate that BPA, BPF, and BPS are inversely associated with eGFR.
| S-EPMC11324456 | biostudies-literature
Project description:BackgroundDNA microarray technology provides a powerful tool for characterizing gene expression on a genome scale. While the technology has been widely used in discovery-based medical and basic biological research, its direct application in clinical practice and regulatory decision-making has been questioned. A few key issues, including the reproducibility, reliability, compatibility and standardization of microarray analysis and results, must be critically addressed before any routine usage of microarrays in clinical laboratory and regulated areas can occur. In this study we investigate some of these issues for the Applied Biosystems Human Genome Survey Microarrays.ResultsWe analyzed the gene expression profiles of two samples: brain and universal human reference (UHR), a mixture of RNAs from 10 cancer cell lines, using the Applied Biosystems Human Genome Survey Microarrays. Five technical replicates in three different sites were performed on the same total RNA samples according to manufacturer's standard protocols. Five different methods, quantile, median, scale, VSN and cyclic loess were used to normalize AB microarray data within each site. 1,000 genes spanning a wide dynamic range in gene expression levels were selected for real-time PCR validation. Using the TaqMan assays data set as the reference set, the performance of the five normalization methods was evaluated focusing on the following criteria: (1) Sensitivity and reproducibility in detection of expression; (2) Fold change correlation with real-time PCR data; (3) Sensitivity and specificity in detection of differential expression; (4) Reproducibility of differentially expressed gene lists.ConclusionOur results showed a high level of concordance between these normalization methods. This is true, regardless of whether signal, detection, variation, fold change measurements and reproducibility were interrogated. Furthermore, we used TaqMan assays as a reference, to generate TPR and FDR plots for the various normalization methods across the assay range. Little impact is observed on the TP and FP rates in detection of differentially expressed genes. Additionally, little effect was observed by the various normalization methods on the statistical approaches analyzed which indicates a certain robustness of the analysis methods currently in use in the field, particularly when used in conjunction with the Applied Biosystems Gene Expression System.
| S-EPMC1764432 | biostudies-literature
Project description:BackgroundHarmonization techniques make different gene expression profiles and their sets compatible and ready for comparisons. Here we present a new bioinformatic tool termed Shambhala for harmonization of multiple human gene expression datasets obtained using different experimental methods and platforms of microarray hybridization and RNA sequencing.ResultsUnlike previously published methods enabling good quality data harmonization for only two datasets, Shambhala allows conversion of multiple datasets into the universal form suitable for further comparisons. Shambhala harmonization is based on the calibration of gene expression profiles using the auxiliary standardization dataset. Each profile is transformed to make it similar to the output of microarray hybridization platform Affymetrix Human Gene. This platform was chosen because it has the biggest number of human gene expression profiles deposited in public databases. We evaluated Shambhala ability to retain biologically important features after harmonization. The same four biological samples taken in multiple replicates were profiled independently using three and four different experimental platforms, respectively, then Shambhala-harmonized and investigated by hierarchical clustering.ConclusionOur results showed that unlike other frequently used methods: quantile normalization and DESeq/DESeq2 normalization, Shambhala harmonization was the only method supporting sample-specific and platform-independent biologically meaningful clustering for the data obtained from multiple experimental platforms.
| S-EPMC6366102 | biostudies-literature
Project description:BackgroundStatistical methods for ranking differentially expressed genes (DEGs) from gene expression data should be evaluated with regard to high sensitivity, specificity, and reproducibility. In our previous studies, we evaluated eight gene ranking methods applied to only Affymetrix GeneChip data. A more general evaluation that also includes other microarray platforms, such as the Agilent or Illumina systems, is desirable for determining which methods are suitable for each platform and which method has better inter-platform reproducibility.ResultsWe compared the eight gene ranking methods using the MicroArray Quality Control (MAQC) datasets produced by five manufacturers: Affymetrix, Applied Biosystems, Agilent, GE Healthcare, and Illumina. The area under the curve (AUC) was used as a measure for both sensitivity and specificity. Although the highest AUC values can vary with the definition of "true" DEGs, the best methods were, in most cases, either the weighted average difference (WAD), rank products (RP), or intensity-based moderated t statistic (ibmT). The percentages of overlapping genes (POGs) across different test sites were mainly evaluated as a measure for both intra- and inter-platform reproducibility. The POG values for WAD were the highest overall, irrespective of the choice of microarray platform. The high intra- and inter-platform reproducibility of WAD was also observed at a higher biological function level.ConclusionThese results for the five microarray platforms were consistent with our previous ones based on 36 real experimental datasets measured using the Affymetrix platform. Thus, recommendations made using the MAQC benchmark data might be universally applicable.
| S-EPMC3128035 | biostudies-literature
Project description:Vehicular ad hoc networks (VANETs) are intelligent transport subsystems; vehicles can communicate through a wireless medium in this system. There are many applications of VANETs such as traffic safety and preventing the accident of vehicles. Many attacks affect VANETs communication such as denial of service (DoS) and distributed denial of service (DDoS). In the past few years the number of DoS (denial of service) attacks are increasing, so network security and protection of the communication systems are challenging topics; intrusion detection systems need to be improved to identify these attacks effectively and efficiently. Many researchers are currently interested in enhancing the security of VANETs. Based on intrusion detection systems (IDS), machine learning (ML) techniques were employed to develop high-security capabilities. A massive dataset containing application layer network traffic is deployed for this purpose. Interpretability technique Local interpretable model-agnostic explanations (LIME) technique for better interpretation model functionality and accuracy. Experimental results demonstrate that utilizing a random forest (RF) classifier achieves 100% accuracy, demonstrating its capability to identify intrusion-based threats in a VANET setting. In addition, LIME is applied to the RF machine learning model to explain and interpret the classification, and the performance of machine learning models is evaluated in terms of accuracy, recall, and F1 score.
| S-EPMC10319271 | biostudies-literature