The inverse variance-flatness relation in stochastic gradient descent is critical for finding flat minima.

ABSTRACT: Despite tremendous success of the stochastic gradient descent (SGD) algorithm in deep learning, little is known about how SGD finds generalizable solutions at flat minima of the loss function in high-dimensional weight space. Here, we investigate the connection between SGD learning dynamics and the loss function landscape. A principal component analysis (PCA) shows that SGD dynamics follow a low-dimensional drift-diffusion motion in the weight space. Around a solution found by SGD, the loss function landscape can be characterized by its flatness in each PCA direction. Remarkably, our study reveals a robust inverse relation between the weight variance and the landscape flatness in all PCA directions, which is the opposite to the fluctuation-response relation (aka Einstein relation) in equil

SUBMITTER: Feng Y 

PROVIDER: S-EPMC7936325 | biostudies-literature | 2021 Mar

REPOSITORIES: biostudies-literature