Preprint
Knowledge Graphs

Distributed Graph Neural Network Training: A Survey

Yingxia Shao(Beijing University of Posts and Telecommunications), Hongzheng Li(Beijing University of Posts and Telecommunications), Xizhi Gu(Beijing University of Posts and Telecommunications), Hongbo Yin(Beijing University of Posts and Telecommunications), Yawen Li(Beijing University of Posts and Telecommunications), Xupeng Miao(Carnegie Mellon University), Wentao Zhang(HEC Montréal), Bin Cui(Peking University), Lei Chen(The Hong Kong University of Science and Technology (Guangzhou))
February 16, 2024ACM Computing Surveys124 citations
arXivPDFDOI

124

Citations

4

Influential Citations

ACM Computing Surveys

Venue

2024

Year

Abstract

Graph neural networks (GNNs) are a type of deep learning models that are trained on graphs and have been successfully applied in various domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs, since it is able to provide abundant computing resources. However, the dependency of graph structure increases the difficulty of achieving high-efficiency distributed GNN training, which suffers from the massive communication and workload imbalance. In recent years, many efforts have been made on distributed GNN training, and an array of training algorithms and systems have been proposed. Yet, there is a lack of systematic review of the optimization techniques for the distributed execution of GNN training. In this survey, we analyze three major challenges in distributed GNN training: massive feature communication, the loss of model accuracy, and workload imbalance. Then, we introduce a new taxonomy for the optimization techniques in distributed GNN training that address the above challenges. The new taxonomy classifies existing techniques into four categories: GNN data partition, GNN batch generation, GNN execution model, and GNN communication protocol. We carefully discuss the techniques in each category. In the conclusion, we summarize existing distributed GNN systems for multi–graphics processing units (GPUs), GPU-clusters and central processing unit (CPU)-clusters, respectively, and present a discussion about the future direction of distributed GNN training.

Analysis

Why This Paper Matters

Graph neural networks (GNNs) have emerged as a powerful tool for learning on graph-structured data, with applications ranging from social network analysis to drug discovery. However, training GNNs on large-scale graphs remains a significant challenge due to the inherent complexity of graph structures and the need for substantial computational resources. Distributed computing offers a promising solution by leveraging multiple GPUs or CPUs to handle large graphs, but it introduces new challenges such as massive communication overhead and workload imbalance. Despite the growing body of research on distributed GNN training, there has been a lack of systematic review to guide practitioners and researchers. This survey fills that gap by providing a comprehensive analysis of the challenges and optimization techniques in distributed GNN training.

The paper is particularly timely as the field of graph machine learning is rapidly expanding, and the need for efficient scaling methods is more pressing than ever. By categorizing existing techniques into a coherent taxonomy, the authors enable a structured understanding of the landscape, facilitating the identification of gaps and future directions. This survey is poised to become a key reference for anyone working on scaling GNNs, from system designers to algorithm developers.

Technical Contributions

The survey makes several key technical contributions:

  • Challenge Identification: It clearly articulates three major challenges in distributed GNN training: massive feature communication, loss of model accuracy, and workload imbalance. These challenges are rooted in the graph dependency structure and are critical to address for efficient distributed training.

  • Novel Taxonomy: The authors propose a new taxonomy that classifies optimization techniques into four categories: GNN data partition, GNN batch generation, GNN execution model, and GNN communication protocol. This taxonomy provides a systematic framework for understanding and comparing existing approaches.

  • Comprehensive Review: Within each category, the survey discusses representative techniques, highlighting their strengths and weaknesses. For example, data partition methods are analyzed in terms of graph partitioning algorithms and their impact on communication and load balance.

  • System Summary: The paper summarizes existing distributed GNN systems for multi-GPU, GPU-cluster, and CPU-cluster settings, providing a practical overview of available tools and frameworks.

  • Future Directions: It concludes with a discussion of open problems and future research directions, such as adaptive partitioning and communication-efficient protocols.

Results

As a survey paper, it does not present new experimental results. Instead, it synthesizes findings from the literature. The authors report that techniques such as graph partitioning, batch generation strategies, execution models (e.g., synchronous vs. asynchronous), and communication protocols (e.g., all-reduce vs. parameter server) have been developed to mitigate the identified challenges. For instance, partitioning methods like METIS and streaming partitioning are compared in terms of their ability to balance workloads and reduce cross-partition communication. The survey also notes that many systems, such as DistDGL, Pytorch-Distributed, and others, have been proposed for different hardware settings, but there is no one-size-fits-all solution; trade-offs must be made based on graph characteristics and available resources.

Significance

This survey is significant because it provides the first systematic taxonomy of optimization techniques for distributed GNN training. It bridges the gap between algorithmic innovations and system-level implementations, offering a holistic view that can guide future research and development. By highlighting the challenges and existing solutions, it enables researchers to identify open problems and practitioners to select appropriate techniques for their specific use cases. Moreover, the discussion of future directions, such as the need for adaptive and communication-efficient methods, sets a research agenda that could lead to more scalable and efficient GNN training systems. As GNNs continue to grow in importance, this survey will serve as a valuable resource for the community, fostering progress in both academia and industry.

graph-neural-networksdistributed-traininggraph-partitioningcommunication-optimizationworkload-balancesurveyscalabilitygpu-clusters
Neural Networks

Related Papers