Journal Article
Large Language Models

Understanding the Potential of FPGA-based Spatial Acceleration for Large Language Model Inference

Hongzheng Chen(Cornell University), Jiahao Zhang(Tsinghua University), Yixiao Du(Cornell University), Shaojie Xiang(Cornell University), Zichao Yue(Cornell University), Niansong Zhang(Cornell University), Yaohui Cai(Cornell University), Zhiru Zhang(Cornell University)
April 4, 2024ACM Transactions on Reconfigurable Technology and Systems96 citations
arXivPDFDOI

96

Citations

8

Influential Citations

ACM Transactions on Reconfigurable Technology and Systems

Venue

2024

Year

Abstract

Recent advancements in large language models (LLMs) boasting billions of parameters have generated a significant demand for efficient deployment in inference workloads. While hardware accelerators for Transformer-based models have been extensively studied, the majority of existing approaches rely on temporal architectures that reuse hardware units for different network layers and operators. However, these methods often encounter challenges in achieving low latency due to considerable memory access overhead. This article investigates the feasibility and potential of model-specific spatial acceleration for LLM inference on field-programmable gate arrays (FPGAs). Our approach involves the specialization of distinct hardware units for specific operators or layers, facilitating direct communication between them through a dataflow architecture while minimizing off-chip memory accesses. We introduce a comprehensive analytical model for estimating the performance of a spatial LLM accelerator, taking into account the on-chip compute and memory resources available on an FPGA. This model can be extended to multi-FPGA settings for distributed inference. Through our analysis, we can identify the most effective parallelization and buffering schemes for the accelerator and, crucially, determine the scenarios in which FPGA-based spatial acceleration can outperform its GPU-based counterpart. To enable more productive implementations of an LLM model on FPGAs, we further provide a library of high-level synthesis (HLS) kernels that are composable and reusable. This library will be made available as open-source. To validate the effectiveness of both our analytical model and HLS library, we have implemented Bidirectional Encoder Representations from Transformers (BERT) and Generative Pre-trained Transformers (GPT2) on an AMD Xilinx Alveo U280 FPGA device. Experimental results demonstrate our approach can achieve up to 13.4× speedup when compared to previous FPGA-based accelerators for the BERT model. For GPT generative inference, we attain a 2.2× speedup compared to Design for Excellence, an FPGA overlay, in the prefill stage, while achieving a 1.9× speedup and a 5.7× improvement in energy efficiency compared to the NVIDIA A100 GPU in the decode stage.

Analysis

Why This Paper Matters

Large language models (LLMs) with billions of parameters have created immense demand for efficient inference hardware. While GPUs dominate the landscape, their temporal architecture—reusing the same hardware units across layers—incurs significant memory access overhead that limits latency. This paper systematically explores an alternative: spatial acceleration on FPGAs, where distinct hardware units are specialized for each operator or layer and communicate directly via a dataflow architecture. This approach dramatically reduces off-chip memory traffic, a primary bottleneck in LLM inference.

The significance lies in the paper's comprehensive analytical model that can predict performance across single- and multi-FPGA settings, enabling designers to identify optimal parallelization and buffering schemes without exhaustive hardware implementation. Combined with an open-source library of composable HLS kernels, this work lowers the barrier for FPGA-based LLM acceleration, potentially making it a viable alternative to GPUs for latency-sensitive or energy-constrained deployments.

Technical Contributions

  • Spatial acceleration architecture: Specialized hardware units for each operator/layer with direct dataflow communication, minimizing off-chip memory accesses.
  • Analytical performance model: Estimates throughput and resource utilization for spatial LLM accelerators on FPGAs, extendable to multi-FPGA distributed inference.
  • Open-source HLS kernel library: Composable, reusable high-level synthesis kernels for productive FPGA implementation of LLMs.
  • Parallelization and buffering optimization: The analytical model identifies the most effective schemes for given FPGA resources.

Results

  • BERT: Up to 13.4× speedup compared to previous FPGA-based accelerators.
  • GPT prefill stage: 2.2× speedup over Design for Excellence (an FPGA overlay).
  • GPT decode stage: 1.9× speedup and 5.7× improvement in energy efficiency compared to NVIDIA A100 GPU.
  • All implementations validated on AMD Xilinx Alveo U280 FPGA device.

Significance

This paper provides strong evidence that FPGA-based spatial acceleration can outperform GPUs in energy efficiency for LLM inference, particularly in the decode stage where memory-bound operations dominate. The analytical model and open-source HLS library offer practical tools for researchers and engineers to explore this alternative. As LLMs continue to grow, energy-efficient inference becomes critical for both data center and edge deployments. This work opens a path toward more sustainable AI infrastructure by demonstrating that FPGAs can be a competitive platform for LLM inference, especially when latency and power are primary concerns.

large-language-modelsfpgaspatial-accelerationinferencedataflow-architecturehigh-level-synthesistransformer
Large Language ModelsTransformers

Related Papers