Distributional bias compromises leave-one-out cross-validation.

ABSTRACT: Cross-validation is a common method for evaluating machine learning models. "Leave-one-out cross-validation," in which each data instance is used to test a model trained on all other instances, is often used in data-scarce regimes. As common metrics such as the R² score cannot be calculated for a single prediction, predictions are commonly aggregated across folds for performance evaluation. Here, we prove that this creates "distributional bias": a negative correlation between the average label of each training fold and the label of its corresponding test instance. As machine learning models tend to regress to the mean of their training data, this bias tends to negatively affect performance evaluation and hyperparameter optimization. We demonstrate that distributional bias

SUBMITTER: Austin GI 

PROVIDER: S-EPMC12662204 | biostudies-literature | 2025 Nov

REPOSITORIES: biostudies-literature