Dataset Information

Better null models for assessing predictive accuracy of disease models.

ABSTRACT: Null models provide a critical baseline for the evaluation of predictive disease models. Many studies consider only the grand mean null model (i.e. R2) when evaluating the predictive ability of a model, which is insufficient to convey the predictive power of a model. We evaluated ten null models for human cases of West Nile virus (WNV), a zoonotic mosquito-borne disease introduced to the United States in 1999. The Negative Binomial, Historical (i.e. using previous cases to predict future cases) and Always Absent null models were the strongest overall, and the majority of null models significantly outperformed the grand mean. The length of the training timeseries increased the performance of most null models in US counties where WNV cases were frequent, but improvements were similar for most null models, so relative scores remained unchanged. We argue that a combination of null models is needed to evaluate the forecasting performance of predictive models for infectious diseases and the grand mean is the lowest bar.

SUBMITTER: Keyel AC

PROVIDER: S-EPMC10162537 | biostudies-literature | 2023

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Better null models for assessing predictive accuracy of disease models.

Keyel Alexander C AC Kilpatrick A Marm AM

PloS one 20230505 5

Null models provide a critical baseline for the evaluation of predictive disease models. Many studies consider only the grand mean null model (i.e. R2) when evaluating the predictive ability of a model, which is insufficient to convey the predictive power of a model. We evaluated ten null models for human cases of West Nile virus (WNV), a zoonotic mosquito-borne disease introduced to the United States in 1999. The Negative Binomial, Historical (i.e. using previous cases to predict future cases) ...[more]

PMID: 37146010

Similar Datasets

Project description:BACKGROUND:Wearable sleep monitors are of high interest to consumers and researchers because of their ability to provide estimation of sleep patterns in free-living conditions in a cost-efficient way. OBJECTIVE:We conducted a systematic review of publications reporting on the performance of wristband Fitbit models in assessing sleep parameters and stages. METHODS:In adherence with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, we comprehensively searched the Cumulative Index to Nursing and Allied Health Literature (CINAHL), Cochrane, Embase, MEDLINE, PubMed, PsycINFO, and Web of Science databases using the keyword Fitbit to identify relevant publications meeting predefined inclusion and exclusion criteria. RESULTS:The search yielded 3085 candidate articles. After eliminating duplicates and in compliance with inclusion and exclusion criteria, 22 articles qualified for systematic review, with 8 providing quantitative data for meta-analysis. In reference to polysomnography (PSG), nonsleep-staging Fitbit models tended to overestimate total sleep time (TST; range from approximately 7 to 67 mins; effect size=-0.51, P<.001; heterogenicity: I2=8.8%, P=.36) and sleep efficiency (SE; range from approximately 2% to 15%; effect size=-0.74, P<.001; heterogenicity: I2=24.0%, P=.25), and underestimate wake after sleep onset (WASO; range from approximately 6 to 44 mins; effect size=0.60, P<.001; heterogenicity: I2=0%, P=.92) and there was no significant difference in sleep onset latency (SOL; P=.37; heterogenicity: I2=0%, P=.92). In reference to PSG, nonsleep-staging Fitbit models correctly identified sleep epochs with accuracy values between 0.81 and 0.91, sensitivity values between 0.87 and 0.99, and specificity values between 0.10 and 0.52. Recent-generation Fitbit models that collectively utilize heart rate variability and body movement to assess sleep stages performed better than early-generation nonsleep-staging ones that utilize only body movement. Sleep-staging Fitbit models, in comparison to PSG, showed no significant difference in measured values of WASO (P=.25; heterogenicity: I2=0%, P=.92), TST (P=.29; heterogenicity: I2=0%, P=.98), and SE (P=.19) but they underestimated SOL (P=.03; heterogenicity: I2=0%, P=.66). Sleep-staging Fitbit models showed higher sensitivity (0.95-0.96) and specificity (0.58-0.69) values in detecting sleep epochs than nonsleep-staging models and those reported in the literature for regular wrist actigraphy. CONCLUSIONS:Sleep-staging Fitbit models showed promising performance, especially in differentiating wake from sleep. However, although these models are a convenient and economical means for consumers to obtain gross estimates of sleep parameters and time spent in sleep stages, they are of limited specificity and are not a substitute for PSG.

Project description:Background and purposeStroke is associated with an increased risk of dementia. To assist in the early identification of individuals at high risk of future dementia, numerous prediction models have been developed for use in the general population. However, it is not known whether such models also provide accurate predictions among stroke patients. Therefore, the aim of this study was to determine whether existing dementia risk prediction models that were developed for use in the general population can also be applied to individuals with a history of stroke to predict poststroke dementia with equivalent predictive validity.MethodsData were harmonized from 4 stroke studies (follow-up range, ≈12-18 months poststroke) from Hong Kong, the United States, the Netherlands, and France. Regression analysis was used to test 3 risk prediction models: the Cardiovascular Risk Factors, Aging and Dementia score, the Australian National University Alzheimer Disease Risk Index, and the Brief Dementia Screening Indicator. Model performance or discrimination accuracy was assessed using the C statistic or area under the curve. Calibration was tested using the Grønnesby and Borgan and the goodness-of-fit tests.ResultsThe predictive accuracy of the models varied but was generally low compared with the original development cohorts, with the Australian National University Alzheimer Disease Risk Index (C-statistic, 0.66) and the Brief Dementia Screening Indicator (C-statistic, 0.61) both performing better than the Cardiovascular Risk Factors, Aging and Dementia score (area under the curve, 0.53).ConclusionsDementia risk prediction models developed for the general population do not perform well in individuals with stroke. Their poor performance could have been due to the need for additional or different predictors related to stroke and vascular risk factors or methodological differences across studies (eg, length of follow-up, age distribution). Future work is needed to develop simple and cost-effective risk prediction models specific to poststroke dementia.

Project description:Stroke is a leading cause of disability and mortality, resulting in substantial socio-economic burden for healthcare systems. With advances in artificial intelligence, visual image information can be processed into numerous quantitative features in an objective, repeatable and high-throughput fashion, in a process known as radiomics analysis (RA). Recently, investigators have attempted to apply RA to stroke neuroimaging in the hope of promoting personalized precision medicine. This review aimed to evaluate the role of RA as an adjuvant tool in the prognosis of disability after stroke. We conducted a systematic review following the PRISMA guidelines, searching PubMed and Embase using the keywords: 'magnetic resonance imaging (MRI)', 'radiomics', and 'stroke'. The PROBAST tool was used to assess the risk of bias. Radiomics quality score (RQS) was also applied to evaluate the methodological quality of radiomics studies. Of the 150 abstracts returned by electronic literature research, 6 studies fulfilled the inclusion criteria. Five studies evaluated predictive value for different predictive models (PMs). In all studies, the combined PMs consisting of clinical and radiomics features have achieved the best predictive performance compared to PMs based only on clinical or radiomics features, the results varying from an area under the ROC curve (AUC) of 0.80 (95% CI, 0.75-0.86) to an AUC of 0.92 (95% CI, 0.87-0.97). The median RQS of the included studies was 15, reflecting a moderate methodological quality. Assessing the risk of bias using PROBAST, potential high risk of bias in participants selection was identified. Our findings suggest that combined models integrating both clinical and advanced imaging variables seem to better predict the patients' disability outcome group (favorable outcome: modified Rankin scale (mRS) ≤ 2 and unfavorable outcome: mRS > 2) at three and six months after stroke. Although radiomics studies' findings are significant in research field, these results should be validated in multiple clinical settings in order to help clinicians to provide individual patients with optimal tailor-made treatment.

Dataset Information

Better null models for assessing predictive accuracy of disease models.

Publications

Better null models for assessing predictive accuracy of disease models.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets