Improved small-sample estimation of nonlinear cross-validated prediction metrics.

ABSTRACT: When predicting an outcome is the scientific goal, one must decide on a metric by which to evaluate the quality of predictions. We consider the problem of measuring the performance of a prediction algorithm with the same data that were used to train the algorithm. Typical approaches involve bootstrapping or cross-validation. However, we demonstrate that bootstrap-based approaches often fail and standard cross-validation estimators may perform poorly. We provide a general study of cross-validation-based estimators that highlights the source of this poor performance, and propose an alternative framework for estimation using techniques from the efficiency theory literature. We provide a theorem establishing the weak convergence of our estimators. The general theorem is applied in detail to tw

SUBMITTER: Benkeser D 

PROVIDER: S-EPMC7954141 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature