Dataset Information

Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression.

ABSTRACT:

Background

The debate of whether machine learning models offer advantages over standard statistical methods when making predictions is ongoing. We discuss the use of a meta-learner model combining both approaches as an alternative.

Methods

To illustrate the development of a meta-learner, we used a dataset of 187,757 people with depression. Using 31 variables, we aimed to predict two outcomes measured 60 days after initiation of antidepressant treatment: severity of depressive symptoms (continuous) and all-cause dropouts (binary). We fitted a ridge regression and a multi-layer perceptron (MLP) deep neural network as two separate prediction models ("base-learners"). We then developed two "meta-learners", combining predictions from the two base-learners. To compare the performance across the different methods, we calculated mean absolute error (MAE, for continuous outcome) and the area under the receiver operating characteristic curve (AUC, for binary outcome) using bootstrapping.

Results

Compared to the best performing base-learner (MLP base-learner, MAE at 4.63, AUC at 0.59), the best performing meta-learner showed a 2.49% decrease in MAE at 4.52 for the continuous outcome and a 6.47% increase in AUC at 0.60 for the binary outcome.

Conclusions

A meta-learner approach may effectively combine multiple prediction models. Choosing between statistical and machine learning models may not be necessary in practice.

SUBMITTER: Liu Q

PROVIDER: S-EPMC9112573 | biostudies-literature | 2022 May

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression.

Liu Qiang Q Salanti Georgia G De Crescenzo Franco F Ostinelli Edoardo Giuseppe EG Li Zhenpeng Z Tomlinson Anneka A Cipriani Andrea A Efthimiou Orestis O

BMC psychiatry 20220516 1

<h4>Background</h4>The debate of whether machine learning models offer advantages over standard statistical methods when making predictions is ongoing. We discuss the use of a meta-learner model combining both approaches as an alternative.<h4>Methods</h4>To illustrate the development of a meta-learner, we used a dataset of 187,757 people with depression. Using 31 variables, we aimed to predict two outcomes measured 60 days after initiation of antidepressant treatment: severity of depressive symp ...[more]

PMID: 35578254

Similar Datasets

Project description:BackgroundIn health research, several chronic diseases are susceptible to competing risks (CRs). Initially, statistical models (SM) were developed to estimate the cumulative incidence of an event in the presence of CRs. As recently there is a growing interest in applying machine learning (ML) for clinical prediction, these techniques have also been extended to model CRs but literature is limited. Here, our aim is to investigate the potential role of ML versus SM for CRs within non-complex data (small/medium sample size, low dimensional setting).MethodsA dataset with 3826 retrospectively collected patients with extremity soft-tissue sarcoma (eSTS) and nine predictors is used to evaluate model-predictive performance in terms of discrimination and calibration. Two SM (cause-specific Cox, Fine-Gray) and three ML techniques are compared for CRs in a simple clinical setting. ML models include an original partial logistic artificial neural network for CRs (PLANNCR original), a PLANNCR with novel specifications in terms of architecture (PLANNCR extended), and a random survival forest for CRs (RSFCR). The clinical endpoint is the time in years between surgery and disease progression (event of interest) or death (competing event). Time points of interest are 2, 5, and 10 years.ResultsBased on the original eSTS data, 100 bootstrapped training datasets are drawn. Performance of the final models is assessed on validation data (left out samples) by employing as measures the Brier score and the Area Under the Curve (AUC) with CRs. Miscalibration (absolute accuracy error) is also estimated. Results show that the ML models are able to reach a comparable performance versus the SM at 2, 5, and 10 years regarding both Brier score and AUC (95% confidence intervals overlapped). However, the SM are frequently better calibrated.ConclusionsOverall, ML techniques are less practical as they require substantial implementation time (data preprocessing, hyperparameter tuning, computational intensity), whereas regression methods can perform well without the additional workload of model training. As such, for non-complex real life survival data, these techniques should only be applied complementary to SM as exploratory tools of model's performance. More attention to model calibration is urgently needed.

Project description:BackgroundPostoperative pneumonia, a prevalent form of hospital-acquired pneumonia, poses significant risks to patients' prognosis and even their lives. This study aimed to develop and validate a predictive model for postoperative pneumonia in surgical patients using nine machine learning methods.ObjectiveOur study aims to develop and validate a predictive model for POP in surgical patients using nine machine learning algorithms. By evaluating the performance differences among these machine learning models, this study aims to assist clinicians in early prediction and diagnosis of POP, providing optimal interventions and treatments.MethodsRetrospective data from electronic medical records was collected for 264 patients diagnosed with postoperative pneumonia and 264 healthy control surgical patients. Through correlation screening, chi-square tests, and feature importance ranking, 47 variables were narrowed down to 5 potential predictive factors based on the main cohort of 528 patients. Nine machine learning models, including k-nearest neighbors, support vector machine, random forest, decision tree, gradient boosting machine, adaptive boosting, naive bayes, general linear model, and linear discriminant analysis, were developed and validated to predict postoperative pneumonia. Model performance was evaluated using the area under the receiver operating curve, sensitivity, specificity, accuracy, precision, recall, and F1 score. A distribution plot of feature importance and feature interaction was obtained to interpret the machine learning models.ResultsAmong 17,190 surgical patients, 264 (1.54%) experienced postoperative pneumonia, which resulted in adverse outcomes such as prolonged hospital stay, increased ICU admission rates, and mortality. We successfully established nine machine learning models for predicting postoperative pneumonia in surgical patients, with the general linear model demonstrating the best overall performance. The AUC of the general linear model on the testing set was 0.877, with an accuracy of 0.82, specificity of 0.89, sensitivity of 0.74, precision of 0.88, and F1 score of 0.80. Our study revealed that the duration of bed rest, unplanned re-operation, end-tidal CO2, postoperative albumin, and chest X-ray film were significant predictors of postoperative pneumonia.ConclusionOur study firstly demonstrated that the general linear model based on 5 common variables might predict postoperative pneumonia in the general surgical population.

Dataset Information

Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression.

Background

Methods

Results

Conclusions

Publications

Development and validation of a meta-learner for combining statistical and machine learning prediction models in individuals with depression.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets