On Consensus-Optimality Trade-offs in Collaborative Deep Learning.

ABSTRACT: In distributed machine learning, where agents collaboratively learn from diverse private data sets, there is a fundamental tension between consensus and optimality. In this paper, we build on recent algorithmic progresses in distributed deep learning to explore various consensus-optimality trade-offs over a fixed communication topology. First, we propose the incremental consensus-based distributed stochastic gradient descent (i-CDSGD) algorithm, which involves multiple consensus steps (where each agent communicates information with its neighbors) within each SGD iteration. Second, we propose the generalized consensus-based distributed SGD (g-CDSGD) algorithm that enables us to navigate the full spectrum from complete consensus (all agents agree) to complete disa

SUBMITTER: Jiang Z 

PROVIDER: S-EPMC8478077 | biostudies-literature | 2021

REPOSITORIES: biostudies-literature