Dataset Information

Facilitating validation of prediction models: a comparison of manual and semi-automated validation using registry-based data of breast cancer patients in the Netherlands.

ABSTRACT:

Background

Clinical prediction models are not routinely validated. To facilitate validation procedures, the online Evidencio platform ( https://www.evidencio.com ) has developed a tool partly automating this process. This study aims to determine whether semi-automated validation can reliably substitute manual validation.

Methods

Four different models used in breast cancer care were selected: CancerMath, INFLUENCE, Predicted Probability of Axillary Metastasis, and PREDICT v.2.0. Data were obtained from the Netherlands Cancer Registry according to the inclusion criteria of the original development population. Calibration (intercepts and slopes) and discrimination (area under the curve (AUC)) were compared between semi-automated and manual validation.

Results

Differences between intercepts and slopes of all models using semi-automated validation ranged from 0 to 0.03 from manual validation, which was not clinically relevant. AUCs were identical for both validation methods.

Conclusions

This easy to use semi-automated validation option is a good substitute for manual validation and might increase the number of validations of prediction models used in clinical practice. In addition, the validation tool was considered to be user-friendly and to save a lot of time compared to manual validation. Semi-automated validation will contribute to more accurate outcome predictions and treatment recommendations in the target population.

SUBMITTER: van Steenbeek CD

PROVIDER: S-EPMC6556016 | biostudies-literature | 2019 Jun

REPOSITORIES: biostudies-literature

ACCESS DATA

Publications

Facilitating validation of prediction models: a comparison of manual and semi-automated validation using registry-based data of breast cancer patients in the Netherlands.

van Steenbeek Cornelia D CD van Maaren Marissa C MC Siesling Sabine S Witteveen Annemieke A Verbeek Xander A A M XAAM Koffijberg Hendrik H

BMC medical research methodology 20190608 1

<h4>Background</h4>Clinical prediction models are not routinely validated. To facilitate validation procedures, the online Evidencio platform ( https://www.evidencio.com ) has developed a tool partly automating this process. This study aims to determine whether semi-automated validation can reliably substitute manual validation.<h4>Methods</h4>Four different models used in breast cancer care were selected: CancerMath, INFLUENCE, Predicted Probability of Axillary Metastasis, and PREDICT v.2.0. Da ...[more]

PMID: 31176362

Similar Datasets

Project description:Background: Electrocardiogram (ECG) contamination is present in diaphragm electromyography (EMGdi) recordings. Obtaining EMGdi without ECG contamination is crucial for EMG amplitude analysis. Manually selecting EMGdi in between QRS complexes has been most commonly applied in recent years (manual method). We developed a semi-automated analysis method based on Least Mean Square Adaptive Filtering combined with a synchronously recorded separate ECG channel to remove ECG artifacts from the EMGdi signals. We hypothesized that this approach would shorten analysis duration and might minimize the potential for inter-rater disagreement. Aims: We aimed to evaluate agreement between the semi-automated method and the manual method and inter-rater reliability of the manual method. Methods: Electromyography signals of seven patients with COPD were recorded using an esophageal catheter during an exercise test on a cycle ergometer. Four patients subsequently participated in an inspiratory muscle training (IMT) program for 8 weeks. After IMT, the tests were repeated. EMGdi/EMGdiMax as obtained either manually by the two assessors or retrieved from the semi-automated method were compared. Results: Semi-automated EMGdi/EMGdiMax agreed well with values obtained by one of the two manual assessors (assessor 1) both at pre-intervention measurements (mean difference -0.5%, 95% CI: -19.6 to 18.6%) and for the pre/post IMT differences (mean difference 1.2%, 95% CI: -16.8 to 19.2%). Intra-class correlation coefficients between methods were 0.96 (95% CI: 0.94-0.97) at pre-intervention measurements and 0.78 (95% CI: 0.58-0.89) for pre/post IMT differences (both p < 0.001). EMGdi/EMGdiMax from assessor 2 was systematically lower than from assessor 1 and agreed less well with the semi-automated method both at pre-intervention measurements (mean difference: 9.3%, 95% CI: -11.4 to 29.9%) and for pre/post IMT differences (mean difference 7.0%, 95% CI: -20.4 to 34.4%). Analysis duration of the semi-automated method was significantly shorter (29 ± 9 min) than the manual method (82 ± 20 min, p < 0.001). Conclusion: The developed semi-automated method is more time efficient and will be less prone to inter-rater variability that was observed when applying the manual analysis method. It is, therefore, proposed as a new standard for objective EMGdi amplitude analyses in future studies.

Dataset Information

Facilitating validation of prediction models: a comparison of manual and semi-automated validation using registry-based data of breast cancer patients in the Netherlands.

Background

Methods

Results

Conclusions

Publications

Facilitating validation of prediction models: a comparison of manual and semi-automated validation using registry-based data of breast cancer patients in the Netherlands.

Similar Datasets

OmicsDI is part of the ELIXIR infrastructure

Tweets