Accelerated optimization in deep learning with a proportional-integral-derivative controller.

ABSTRACT: High-performance optimization algorithms are essential in deep learning. However, understanding the behavior of optimization (i.e., learning process) remains challenging due to the instability and weak interpretability of the algorithms. Since gradient-based optimizations can be interpreted as continuous-time dynamical systems, applying feedback control to the dynamical systems that model the optimizers may provide another perspective for exploring more robust, accurate and explainable optimization algorithms. In this study, we present a framework for optimization called controlled heavy-ball optimizer. By employing the proportional-integral-derivative (PID) controller in the optimizer, we develop a deterministic continuous-time optimizer called Proportional-Integral-Derivative Accelerated

SUBMITTER: Chen S 

PROVIDER: S-EPMC11599775 | biostudies-literature | 2024 Nov

REPOSITORIES: biostudies-literature