Learning Curves for Noisy Heterogeneous Feature-Subsampled Ridge Ensembles.

ABSTRACT: Feature bagging is a well-established ensembling method which aims to reduce prediction variance by combining predictions of many estimators trained on subsets or projections of features. Here, we develop a theory of feature-bagging in noisy least-squares ridge ensembles and simplify the resulting learning curves in the special case of equicorrelated data. Using analytical learning curves, we demonstrate that subsampling shifts the double-descent peak of a linear predictor. This leads us to introduce heterogeneous feature ensembling, with estimators built on varying numbers of feature dimensions, as a computationally efficient method to mitigate double-descent. Then, we compare the performance of a feature-subsampling ensemble to a single linear predictor, describing a trade-off between no

SUBMITTER: Ruben BS 

PROVIDER: S-EPMC10350086 | biostudies-literature | 2024 Jan

REPOSITORIES: biostudies-literature