Gradient Decomposition Methods for Training Neural Networks With Non-ideal Synaptic Devices.

ABSTRACT: While promising for high-capacity machine learning accelerators, memristor devices have non-idealities that prevent software-equivalent accuracies when used for online training. This work uses a combination of Mini-Batch Gradient Descent (MBGD) to average gradients, stochastic rounding to avoid vanishing weight updates, and decomposition methods to keep the memory overhead low during mini-batch training. Since the weight update has to be transferred to the memristor matrices efficiently, we also investigate the impact of reconstructing the gradient matrixes both internally (rank-seq) and externally (rank-sum) to the memristor array. Our results show that streaming batch principal component analysis (streaming batch PCA) and non-negative matrix factorization (NMF) decompositio

SUBMITTER: Zhao J 

PROVIDER: S-EPMC8645649 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature