Adaptive Checkpoint Adjoint Method for Gradient Estimation in Neural ODE.

ABSTRACT: Neural ordinary differential equations (NODEs) have recently attracted increasing attention; however, their empirical performance on benchmark tasks (e.g. image classification) are significantly inferior to discrete-layer models. We demonstrate an explanation for their poorer performance is the inaccuracy of existing gradient estimation methods: the adjoint method has numerical errors in reverse-mode integration; the naive method directly back-propagates through ODE solvers, but suffers from a redundantly deep computation graph when searching for the optimal stepsize. We propose the Adaptive Checkpoint Adjoint (ACA) method: in automatic differentiation, ACA applies a trajectory checkpoint strategy which records the forward-mode trajectory as the reverse-mode tra

SUBMITTER: Zhuang J 

PROVIDER: S-EPMC8299461 | biostudies-literature | 2020

REPOSITORIES: biostudies-literature