Dataset Information

Quantitative interpretation explains machine learning models for chemical reaction prediction and uncovers bias.

ABSTRACT: Organic synthesis remains a major challenge in drug discovery. Although a plethora of machine learning models have been proposed as solutions in the literature, they suffer from being opaque black-boxes. It is neither clear if the models are making correct predictions because they inferred the salient chemistry, nor is it clear which training data they are relying on to reach a prediction. This opaqueness hinders both model developers and users. In this paper, we quantitatively interpret the Molecular Transformer, the state-of-the-art model for reaction prediction. We develop a framework to attribute predicted reaction outcomes both to specific parts of reactants, and to reactions in the training set. Furthermore, we demonstrate how to retrieve evidence for predicted reaction outcomes, and understand counterintuitive predictions by scrutinising the data. Additionally, we identify Clever Hans predictions where the correct prediction is reached for the wrong reason due to dataset bias. We present a new debiased dataset that provides a more realistic assessment of model performance, which we propose as the new standard benchmark for comparing reaction prediction models.

SUBMITTER: Kovacs DP

PROVIDER: S-EPMC7966799 | biostudies-literature | 2021 Mar

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Quantitative interpretation explains machine learning models for chemical reaction prediction and uncovers bias.

Kovács Dávid Péter DP McCorkindale William W Lee Alpha A AA

Nature communications 20210316 1

Organic synthesis remains a major challenge in drug discovery. Although a plethora of machine learning models have been proposed as solutions in the literature, they suffer from being opaque black-boxes. It is neither clear if the models are making correct predictions because they inferred the salient chemistry, nor is it clear which training data they are relying on to reach a prediction. This opaqueness hinders both model developers and users. In this paper, we quantitatively interpret the Mol ...[more]

PMID: 33727552

Similar Datasets

Project description:ObjectiveTo assess the methodological quality of studies on prediction models developed using machine learning techniques across all medical specialties.DesignSystematic review.Data sourcesPubMed from 1 January 2018 to 31 December 2019.Eligibility criteriaArticles reporting on the development, with or without external validation, of a multivariable prediction model (diagnostic or prognostic) developed using supervised machine learning for individualised predictions. No restrictions applied for study design, data source, or predicted patient related health outcomes.Review methodsMethodological quality of the studies was determined and risk of bias evaluated using the prediction risk of bias assessment tool (PROBAST). This tool contains 21 signalling questions tailored to identify potential biases in four domains. Risk of bias was measured for each domain (participants, predictors, outcome, and analysis) and each study (overall).Results152 studies were included: 58 (38%) included a diagnostic prediction model and 94 (62%) a prognostic prediction model. PROBAST was applied to 152 developed models and 19 external validations. Of these 171 analyses, 148 (87%, 95% confidence interval 81% to 91%) were rated at high risk of bias. The analysis domain was most frequently rated at high risk of bias. Of the 152 models, 85 (56%, 48% to 64%) were developed with an inadequate number of events per candidate predictor, 62 handled missing data inadequately (41%, 33% to 49%), and 59 assessed overfitting improperly (39%, 31% to 47%). Most models used appropriate data sources to develop (73%, 66% to 79%) and externally validate the machine learning based prediction models (74%, 51% to 88%). Information about blinding of outcome and blinding of predictors was, however, absent in 60 (40%, 32% to 47%) and 79 (52%, 44% to 60%) of the developed models, respectively.ConclusionMost studies on machine learning based prediction models show poor methodological quality and are at high risk of bias. Factors contributing to risk of bias include small study size, poor handling of missing data, and failure to deal with overfitting. Efforts to improve the design, conduct, reporting, and validation of such studies are necessary to boost the application of machine learning based prediction models in clinical practice.Systematic review registrationPROSPERO CRD42019161764.

Project description:Soil pollution with cadmium (Cd) poses serious health and environmental consequences. The study investigated the incubation of several soil samples and conducted quantitative soil characterization to assess the influence of biochar (BC) on Cd adsorption. The aim was to develop predictive models for Cd concentrations using statistical and modeling approaches dependent on soil characteristics. The potential risk linked to the transformation and immobilization of Cd adsorption by BC in the soil could be conservatively assessed by pH, clay, cation exchange capacity, organic carbon, and electrical conductivity. In this study, Long Short-Term Memory (LSTM), Bidirectional Gated Recurrent Unit (BiGRU), and 5-layer CNN Convolutional Neural Networks (CNNs) were applied for risk assessments to establish a framework for evaluating Cd risk in BC amended soils to predict Cd transformation. In the case of control soils (CK), the BiGRU model showed commendable performance, with an R2 value of 0.85, indicating an approximate 85.37% variance in the actual Cd. The LSTM model, which incorporates sequence data, produced less accurate results (R2=0.84), while the 5-layer CNN model had an R2 value of 0.91, indicating that the CNN model could account for over 91% of the variation in actual Cd levels. In the case of BC-applied soils, the BiGRU model demonstrated a strong correlation between predicted and actual values with R2 (0.93), indicating that the model explained 93.21% of the variance in Cd concentrations. Similarly, the LSTM model showed a notable increase in performance with BC-treated soil data. The R2 value for this model stands at a robust R2 (0.94), reflecting its enhanced ability to predict Cd levels with BC incorporation. Outperforming both recurrent models, the 5-layer CNN model attained the highest precision with an R2 value of 0.95, suggesting that 95.58% of the variance in the actual Cd data can be explained by the CNN model's predictions in BC-amended soils. Consequently, this study suggests developing ecological soil remediation strategies that can effectively manage heavy metal pollution in soils for environmental sustainability.

Project description:IntroductionThere was limited evidence on the quality of reporting and methodological quality of prediction models using machine learning methods in preterm birth. This systematic review aimed to assess the reporting quality and risk of bias of a machine learning-based prediction model in preterm birth.Material and methodsWe conducted a systematic review, searching the PubMed, Embase, the Cochrane Library, China National Knowledge Infrastructure, China Biology Medicine disk, VIP Database, and WanFang Data from inception to September 27, 2021. Studies that developed (validated) a prediction model using machine learning methods in preterm birth were included. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement and Prediction model Risk of Bias Assessment Tool (PROBAST) to evaluate the reporting quality and the risk of bias of included studies, respectively. Findings were summarized using descriptive statistics and visual plots. The protocol was registered in PROSPERO (no. CRD 42022301623).ResultsTwenty-nine studies met the inclusion criteria, with 24 development-only studies and 5 development-with-validation studies. Overall, TRIPOD adherence per study ranged from 17% to 79%, with a median adherence of 49%. The reporting of title, abstract, blinding of predictors, sample size justification, explanation of model, and model performance were mostly poor, with TRIPOD adherence ranging from 4% to 17%. For all included studies, 79% had a high overall risk of bias, and 21% had an unclear overall risk of bias. The analysis domain was most commonly rated as high risk of bias in included studies, mainly as a result of small effective sample size, selection of predictors based on univariable analysis, and lack of calibration evaluation.ConclusionsReporting and methodological quality of machine learning-based prediction models in preterm birth were poor. It is urgent to improve the design, conduct, and reporting of such studies to boost the application of machine learning-based prediction models in preterm birth in clinical practice.

Dataset Information

Quantitative interpretation explains machine learning models for chemical reaction prediction and uncovers bias.

Publications

Quantitative interpretation explains machine learning models for chemical reaction prediction and uncovers bias.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets