Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks.

ABSTRACT: An increasingly popular machine learning paradigm is to pretrain a neural network (NN) on many tasks offline, then adapt it to downstream tasks, often by re-training only the last linear layer of the network. This approach yields strong downstream performance in a variety of contexts, demonstrating that multitask pretraining leads to effective feature learning. Although several recent theoretical studies have shown that shallow NNs learn meaningful features when either (i) they are trained on a single task or (ii) they are linear, very little is known about the closer-to-practice case of nonlinear NNs trained on multiple tasks. In this work, we present the first results proving that feature learning occurs during training with a nonlinear model on multiple tasks

SUBMITTER: Collins L 

PROVIDER: S-EPMC11486479 | biostudies-literature | 2024 Jul

REPOSITORIES: biostudies-literature