Parameter-efficient fine-tuning for large models: A comprehensive survey

Analysis

Why This Paper Matters

As large language and vision models grow to billions of parameters, full fine-tuning becomes prohibitively expensive in terms of compute and storage. Parameter-efficient fine-tuning (PEFT) offers a practical alternative by updating only a small fraction of model weights while retaining most of the pretrained knowledge. This survey is timely because it organizes the rapidly expanding landscape of PEFT methods, helping researchers and engineers navigate choices like adapters, prefix tuning, and LoRA. By systematically comparing these approaches, the paper addresses a critical need for clarity in a field where new techniques emerge frequently.

Technical Contributions

The survey's main contribution is a clear taxonomy that groups PEFT methods into three categories:

• Additive methods (e.g., adapters, soft prompts) that insert new trainable modules.
• Selective methods that update only a subset of existing parameters.
• Reparameterization methods (e.g., LoRA) that decompose weight updates into low-rank matrices.

It also discusses practical design choices such as where to place adapters, how to choose rank in LoRA, and initialization strategies. The paper highlights that no single method dominates; performance depends on model scale, task type, and available compute.

Results

The survey aggregates findings from multiple studies showing that PEFT methods typically achieve 90-99% of full fine-tuning performance while reducing trainable parameters by 10,000x or more. For example, LoRA on GPT-3 175B uses only 0.01% of parameters. Memory savings are also significant, often enabling fine-tuning on a single GPU. However, the survey notes that very small models or tasks requiring substantial domain shift may still benefit from full fine-tuning.

Significance

This survey has broad impact by democratizing access to large model adaptation. It provides a roadmap for practitioners to select efficient fine-tuning strategies, reducing the carbon footprint and hardware barriers associated with training large models. The identified open challenges—such as theoretical understanding of why PEFT works and its interaction with multi-task learning—will guide future research. As AI models continue to scale, PEFT will be essential for practical deployment, making this survey a foundational reference.