Journal Article
Machine Learning

Robust and Communication-Efficient Federated Learning From Non-i.i.d. Data

Felix Sattler(Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute), Simon Wiedemann(Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute), Klaus‐Robert Müller(Technische Universität Berlin), Wojciech Samek(Fraunhofer Institute for Telecommunications, Heinrich Hertz Institute)
November 1, 2019IEEE Transactions on Neural Networks and Learning Systems1,720 citations
PDFDOI

1.7k

Citations

0

Influential Citations

IEEE Transactions on Neural Networks and Learning Systems

Venue

2019

Year

Abstract

Federated learning allows multiple parties to jointly train a deep learning model on their combined data, without any of the participants having to reveal their local data to a centralized server. This form of privacy-preserving collaborative learning, however, comes at the cost of a significant communication overhead during training. To address this problem, several compression methods have been proposed in the distributed training literature that can reduce the amount of required communication by up to three orders of magnitude. These existing methods, however, are only of limited utility in the federated learning setting, as they either only compress the upstream communication from the clients to the server (leaving the downstream communication uncompressed) or only perform well under idealized conditions, such as i.i.d. distribution of the client data, which typically cannot be found in federated learning. In this article, we propose sparse ternary compression (STC), a new compression framework that is specifically designed to meet the requirements of the federated learning environment. STC extends the existing compression technique of top- k gradient sparsification with a novel mechanism to enable downstream compression as well as ternarization and optimal Golomb encoding of the weight updates. Our experiments on four different learning tasks demonstrate that STC distinctively outperforms federated averaging in common federated learning scenarios. These results advocate for a paradigm shift in federated optimization toward high-frequency low-bitwidth communication, in particular in the bandwidth-constrained learning environments.

Analysis

Why This Paper Matters

Federated learning enables collaborative model training without sharing raw data, but communication overhead is a major bottleneck. Existing compression methods either only compress upstream communication or assume i.i.d. data, which is unrealistic in federated settings. This paper introduces sparse ternary compression (STC), a framework specifically designed for the federated learning environment that compresses both upstream and downstream communication and performs well under non-i.i.d. data.

By addressing both directions of communication and relaxing the i.i.d. assumption, STC makes federated learning more practical for real-world scenarios where clients have heterogeneous data and bandwidth is limited. The paper's advocacy for high-frequency, low-bitwidth communication represents a shift in federated optimization research.

Technical Contributions

  • Bidirectional compression: STC extends top-k gradient sparsification with a novel mechanism to compress downstream communication from server to clients, unlike prior methods that only compress upstream updates.
  • Ternarization and Golomb encoding: Weight updates are ternarized (to -1, 0, +1) and optimally Golomb-encoded, reducing bitwidth while preserving essential information.
  • Robustness to non-i.i.d. data: The framework is designed to handle heterogeneous client data distributions, a common challenge in federated learning.
  • Empirical validation: Experiments on four learning tasks show STC outperforms federated averaging (FedAvg) in common federated scenarios.

Results

The paper reports that STC distinctively outperforms federated averaging across four different learning tasks under non-i.i.d. data distributions. While specific metrics are not detailed in the abstract, the results demonstrate that STC achieves better model performance than FedAvg when communication is compressed, particularly in bandwidth-constrained environments. The experiments likely measure accuracy or convergence speed under varying compression rates.

Significance

STC addresses a critical barrier to deploying federated learning at scale: communication efficiency. By enabling effective compression in both directions and under realistic non-i.i.d. conditions, it paves the way for federated learning in bandwidth-limited settings such as mobile and IoT networks. The paper's paradigm shift toward high-frequency, low-bitwidth communication has influenced subsequent research on federated optimization and compression. With over 1700 citations, it is a foundational work in communication-efficient federated learning.

federated-learningcommunication-efficiencymodel-compressionnon-iid-datagradient-sparsificationternarizationdistributed-training

Related Papers