Dataset Information

Sample size considerations and predictive performance of multinomial logistic prediction models.

ABSTRACT: Multinomial Logistic Regression (MLR) has been advocated for developing clinical prediction models that distinguish between three or more unordered outcomes. We present a full-factorial simulation study to examine the predictive performance of MLR models in relation to the relative size of outcome categories, number of predictors and the number of events per variable. It is shown that MLR estimated by Maximum Likelihood yields overfitted prediction models in small to medium sized data. In most cases, the calibration and overall predictive performance of the multinomial prediction model is improved by using penalized MLR. Our simulation study also highlights the importance of events per variable in the multinomial context as well as the total sample size. As expected, our study demonstrates the need for optimism correction of the predictive performance measures when developing the multinomial logistic prediction model. We recommend the use of penalized MLR when prediction models are developed in small data sets or in medium sized data sets with a small total sample size (ie, when the sizes of the outcome categories are balanced). Finally, we present a case study in which we illustrate the development and validation of penalized and unpenalized multinomial prediction models for predicting malignancy of ovarian cancer.

SUBMITTER: de Jong VMT

PROVIDER: S-EPMC6590172 | biostudies-literature | 2019 Apr

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Sample size considerations and predictive performance of multinomial logistic prediction models.

de Jong Valentijn M T VMT Eijkemans Marinus J C MJC van Calster Ben B Timmerman Dirk D Moons Karel G M KGM Steyerberg Ewout W EW van Smeden Maarten M

Statistics in medicine 20190106 9

Multinomial Logistic Regression (MLR) has been advocated for developing clinical prediction models that distinguish between three or more unordered outcomes. We present a full-factorial simulation study to examine the predictive performance of MLR models in relation to the relative size of outcome categories, number of predictors and the number of events per variable. It is shown that MLR estimated by Maximum Likelihood yields overfitted prediction models in small to medium sized data. In most c ...[more]

PMID: 30614028

Similar Datasets

Project description:BACKGROUND/AIMS:Clinical trials for Alzheimer's disease have been aimed primarily at persons who have cognitive symptoms at enrollment. However, researchers are now recognizing that the pathophysiological process of Alzheimer's disease begins years, if not decades, prior to the onset of clinical symptoms. Successful intervention may require intervening early in the disease process. Critical issues arise in designing clinical trials for primary and secondary prevention of Alzheimer's disease including determination of sample sizes and follow-up duration. We address a number of these issues through application of a unifying multistate model for the preclinical course of Alzheimer's disease. A multistate model allows us to specify at which points during the long disease process the intervention exerts its effects. METHODS:We used a nonhomogeneous Markov multistate model for the progression of Alzheimer's disease through preclinical disease states defined by biomarkers, mild cognitive impairment and Alzheimer's disease dementia. We used transition probabilities based on several published cohort studies. Sample size methods were developed that account for factors including the initial preclinical disease state of trial participants, the primary endpoint, age-dependent transition and mortality rates and specifications of which transition rates are the targets of the intervention. RESULTS:We find that Alzheimer's disease prevention trials with a clinical primary endpoint of mild cognitive impairment or Alzheimer's disease dementia will require sample sizes of the order many thousands of individuals with at least 5 years of follow-up, which is larger than most Alzheimer's disease therapeutic trials conducted to date. The reasons for the large trial sizes include the long and variable preclinical period that spans decades, high rates of attrition among elderly populations due to mortality and losses to follow-up and potential selection effects, whereby healthier subjects enroll in prevention trials. A web application is available to perform sample size calculations using the methods reported here. CONCLUSION:Sample sizes based on multistate models can account for the points in the disease process when interventions exert their effects and may lead to more accurate sample size determinations. We will need innovative strategies to help design Alzheimer's disease prevention trials with feasible sample size requirements and durations of follow-up.

Project description:BackgroundWe suggest an adaptive sample size calculation method for developing clinical prediction models, in which model performance is monitored sequentially as new data comes in.MethodsWe illustrate the approach using data for the diagnosis of ovarian cancer (n = 5914, 33% event fraction) and obstructive coronary artery disease (CAD; n = 4888, 44% event fraction). We used logistic regression to develop a prediction model consisting only of a priori selected predictors and assumed linear relations for continuous predictors. We mimicked prospective patient recruitment by developing the model on 100 randomly selected patients, and we used bootstrapping to internally validate the model. We sequentially added 50 random new patients until we reached a sample size of 3000 and re-estimated model performance at each step. We examined the required sample size for satisfying the following stopping rule: obtaining a calibration slope ≥ 0.9 and optimism in the c-statistic (or AUC) < = 0.02 at two consecutive sample sizes. This procedure was repeated 500 times. We also investigated the impact of alternative modeling strategies: modeling nonlinear relations for continuous predictors and correcting for bias on the model estimates (Firth's correction).ResultsBetter discrimination was achieved in the ovarian cancer data (c-statistic 0.9 with 7 predictors) than in the CAD data (c-statistic 0.7 with 11 predictors). Adequate calibration and limited optimism in discrimination was achieved after a median of 450 patients (interquartile range 450-500) for the ovarian cancer data (22 events per parameter (EPP), 20-24) and 850 patients (750-900) for the CAD data (33 EPP, 30-35). A stricter criterion, requiring AUC optimism < = 0.01, was met with a median of 500 (23 EPP) and 1500 (59 EPP) patients, respectively. These sample sizes were much higher than the well-known 10 EPP rule of thumb and slightly higher than a recently published fixed sample size calculation method by Riley et al. Higher sample sizes were required when nonlinear relationships were modeled, and lower sample sizes when Firth's correction was used.ConclusionsAdaptive sample size determination can be a useful supplement to fixed a priori sample size calculations, because it allows to tailor the sample size to the specific prediction modeling context in a dynamic fashion.

Project description:Background and Objectives: Pulmonary complications are a leading cause of morbidity after cardiac surgery. The aim of this study was to develop models to predict postoperative lung dysfunction and mortality. Materials and Methods: This was a single-center, observational, retrospective study. We retrospectively analyzed the data of 11,285 adult patients who underwent all types of cardiac surgery from 2003 to 2015. We developed logistic predictive models for in-hospital mortality, postoperative pulmonary complications occurring in the intensive care unit, and postoperative non-invasive mechanical ventilation when clinically indicated. Results: In the "preoperative model" predictors for mortality were advanced age (p < 0.001), New York Heart Association (NYHA) class (p < 0.001) and emergent surgery (p = 0.036); predictors for non-invasive mechanical ventilation were advanced age (p < 0.001), low ejection fraction (p = 0.023), higher body mass index (p < 0.001) and preoperative renal failure (p = 0.043); predictors for postoperative pulmonary complications were preoperative chronic obstructive pulmonary disease (p = 0.007), preoperative kidney injury (p < 0.001) and NYHA class (p = 0.033). In the "surgery model" predictors for mortality were intraoperative inotropes (p = 0.003) and intraoperative intra-aortic balloon pump (p < 0.001), which also predicted the incidence of postoperative pulmonary complications. There were no specific variables in the surgery model predicting the use of non-invasive mechanical ventilation. In the "intensive care unit model", predictors for mortality were postoperative kidney injury (p < 0.001), tracheostomy (p < 0.001), inotropes (p = 0.029) and PaO2/FiO2 ratio at discharge (p = 0.028); predictors for non-invasive mechanical ventilation were kidney injury (p < 0.001), inotropes (p < 0.001), blood transfusions (p < 0.001) and PaO2/FiO2 ratio at the discharge (p < 0.001). Conclusions: In this retrospective study, we identified the preoperative, intraoperative and postoperative characteristics associated with mortality and complications following cardiac surgery.

Dataset Information

Sample size considerations and predictive performance of multinomial logistic prediction models.

Publications

Sample size considerations and predictive performance of multinomial logistic prediction models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets