Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency

Analysis

Why This Paper Matters

Deploying Large Language Models (LLMs) on edge devices is a growing necessity as AI moves toward on-device processing for privacy, latency, and connectivity reasons. However, the computational and memory demands of LLMs pose significant challenges, particularly in resource-constrained environments. This paper addresses a critical gap by systematically evaluating the energy efficiency, accuracy, and latency of quantized LLMs on a real edge device—the Raspberry Pi 4. While prior work has focused on model compression or performance benchmarks, few studies integrate hardware-level energy profiling with LLM benchmarking to provide a holistic view of sustainability.

The significance of this work lies in its practical focus. By using the Ollama library, which applies default Post-Training Quantization (PTQ) and weight-only quantization, the study reflects a realistic deployment scenario for many developers. The inclusion of five diverse datasets (CommonsenseQA, BIG-Bench Hard, TruthfulQA, GSM8K, and HumanEval) ensures that the evaluation covers reasoning, knowledge, math, and code generation tasks, making the findings broadly applicable. This paper is a valuable resource for AI practitioners seeking to deploy LLMs on edge devices while balancing performance and energy consumption.

Technical Contributions

The paper makes several key technical contributions:

• Comprehensive Benchmarking: Evaluates 28 quantized LLMs, covering a wide range of model sizes and quantization levels, providing a large-scale comparison.
• Hardware-Level Energy Measurement: Uses a high-resolution, hardware-based tool to capture real-world power consumption, which is more accurate than software-based estimates.
• Multi-Dataset Evaluation: Benchmarks on five standardized datasets, enabling task-specific insights into quantization trade-offs.
• Trade-off Analysis: Systematically analyzes the interplay between energy efficiency, inference speed, and output accuracy, identifying Pareto-optimal configurations.
• Actionable Insights: Provides practical recommendations for selecting quantization levels and models for resource-constrained edge deployments.

Results

The study reveals clear trade-offs between energy efficiency, inference speed, and accuracy. Lower quantization levels (e.g., 4-bit) generally reduce energy consumption and latency but may degrade accuracy, especially on complex reasoning tasks like GSM8K and HumanEval. Conversely, higher quantization levels (e.g., 8-bit) preserve accuracy but increase energy and latency. The optimal configuration depends on the task: for simple knowledge-based tasks, aggressive quantization is acceptable, while for code generation or math reasoning, higher precision is necessary. The hardware-based energy measurements provide concrete data on power consumption, enabling developers to estimate battery life and operational costs.

Significance

This paper contributes to the growing field of sustainable AI by providing empirical evidence on how to deploy LLMs efficiently on edge devices. The findings can guide developers in selecting the right quantization settings for their specific use cases, reducing the carbon footprint of AI inference. Moreover, the methodology—combining hardware energy profiling with LLM benchmarking—sets a precedent for future studies. As edge AI becomes more prevalent, this work will help bridge the gap between AI capability and environmental sustainability, making it a valuable reference for both researchers and practitioners.