Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning.

ABSTRACT: Federated learning is an emerging research paradigm enabling collaborative training of machine learning models among different organizations while keeping data private at each institution. Despite recent progress, there remain fundamental challenges such as the lack of convergence and the potential for catastrophic forgetting across real-world heterogeneous devices. In this paper, we demonstrate that self-attention-based architectures (e.g., Transformers) are more robust to distribution shifts and hence improve federated learning over heterogeneous data. Concretely, we conduct the first rigorous empirical investigation of different neural architectures across a range of federated algorithms, real-world benchmarks, and heterogeneous data splits. Our experiments show that simply replacing co

SUBMITTER: Qu L 

PROVIDER: S-EPMC9826695 | biostudies-literature | 2022 Jun

REPOSITORIES: biostudies-literature