Distributional bias compromises leave-one-out cross-validation.

ABSTRACT: Cross-validation is a common method for estimating the predictive performance of machine learning models. In a data-scarce regime, where one typically wishes to maximize the number of instances used for training the model, an approach called 'leave-one-out cross-validation' is often used. In this design, a separate model is built for predicting each data instance after training on all other instances. Since this results in a single test data point available per model trained, predictions are aggregated across the entire dataset to calculate common rank-based performance metrics such as the area under the receiver operating characteristic or precision-recall curves. In this work, we demonstrate that this approach creates a negative correlation between the average label of each training fold

SUBMITTER: Austin GI 

PROVIDER: S-EPMC11177965 | biostudies-literature | 2024 Jun

REPOSITORIES: biostudies-literature