Journal Article
Large Language Models

A Comprehensive Survey of Small Language Models

Fali Wang, Zhiwei Zhang, Xianren Zhang, Zongyu Wu, Tzuhao Mo, Qiuhao Lu, Wanjing Wang, Rui Li, Junjie Xu, Xianfeng Tang, Qi He, Yaoyao Ma, Ming Huang, Suhang Wang
November 4, 2024ACM Transactions on Intelligent Systems and Technology256 citations
arXivPDFDOI

256

Citations

17

Influential Citations

ACM Transactions on Intelligent Systems and Technology

Venue

2024

Year

Abstract

Large language models (LLMs) have demonstrated emergent abilities in text generation, question answering, and reasoning, facilitating various tasks and domains. Despite their proficiency in various tasks, LLMs like PaLM 540B and Llama-3.1 405B face limitations due to large parameter sizes and computational demands, often requiring cloud API use, which raises privacy concerns, limits real-time applications on edge devices, and increases fine-tuning costs. Additionally, LLMs often underperform in specialized domains such as healthcare and law due to insufficient domain-specific knowledge, necessitating specialized models. Therefore, Small Language Models (SLMs) are increasingly favored for their low inference latency, cost-effectiveness, efficient development, and easy customization and adaptability. These models are particularly well-suited for resource-limited environments and domain knowledge acquisition, addressing LLMs’ challenges and proving ideal for applications that require localized data handling for privacy, minimal inference latency for efficiency, and domain knowledge acquisition through lightweight fine-tuning. The rising demand for SLMs has spurred extensive research and development. However, a comprehensive survey investigating issues related to the definition, acquisition, application, enhancement, and reliability of SLM remains lacking, prompting us to conduct a detailed survey on these topics. The definition of SLMs varies widely; thus, to standardize, we propose defining SLMs by their capability to perform specialized tasks and suitability for resource-constrained settings, setting boundaries based on the minimal size for emergent abilities and the maximum size sustainable under resource constraints. For other aspects, we provide a taxonomy of relevant models/methods and develop general frameworks for each category to enhance and utilize SLMs effectively. We have compiled the collected SLM models and related methods on GitHub: https://github.com/FairyFali/SLMs-Survey.

Analysis

Why This Paper Matters

Large Language Models (LLMs) have revolutionized AI with their emergent abilities, but their massive parameter sizes and computational demands pose significant challenges for real-world deployment. This survey addresses a critical gap by focusing on Small Language Models (SLMs), which offer a more practical alternative for resource-limited environments. The paper is timely as the AI community increasingly recognizes the need for efficient, deployable models that can operate on edge devices while maintaining privacy and reducing costs.

The authors tackle the ambiguity surrounding the definition of SLMs, proposing a capability-based and resource-constrained definition that sets clear boundaries. This standardization is crucial for the field to progress coherently. By providing a comprehensive taxonomy and frameworks, the survey serves as a roadmap for researchers and practitioners, helping them navigate the diverse landscape of SLM methods and choose appropriate strategies for their specific needs.

Technical Contributions

The paper's key technical contributions include:

  • Standardized Definition: Proposes defining SLMs by their ability to perform specialized tasks and suitability for resource-constrained settings, with boundaries based on minimal size for emergent abilities and maximum size sustainable under resource constraints.
  • Comprehensive Taxonomy: Categorizes SLM models and methods into four main aspects: acquisition, application, enhancement, and reliability. This taxonomy helps organize the field and identify research gaps.
  • General Frameworks: Develops general frameworks for each category, providing structured guidance on how to effectively acquire, apply, enhance, and ensure the reliability of SLMs.
  • Resource Compilation: Provides a GitHub repository with collected SLM models and related methods, serving as a valuable resource for the community.

These contributions go beyond a simple literature review by offering actionable frameworks that can be directly applied by practitioners.

Results

As a survey paper, it does not present novel experimental results. Instead, it synthesizes findings from 256 cited works to highlight the advantages of SLMs over LLMs, such as lower inference latency, cost-effectiveness, and ease of customization. The paper also discusses scenarios where LLMs underperform, such as specialized domains like healthcare and law, where SLMs can be fine-tuned with domain-specific knowledge. The proposed definition and taxonomy are the primary outcomes, providing a structured understanding of the field.

Significance

The broader impact of this survey is substantial. By standardizing the definition of SLMs and providing a comprehensive taxonomy, it enables more focused research and development. It highlights the practical benefits of SLMs, encouraging their adoption in privacy-sensitive and real-time applications. The frameworks can accelerate the deployment of AI in resource-constrained settings, democratizing access to AI capabilities. This survey is likely to become a key reference for both academic researchers and industry practitioners, shaping the future direction of efficient AI model development.

small-language-modelssurveyefficiencyedge-computingdomain-adaptationfine-tuningprivacy
Large Language ModelsFine TuningPromptingReasoning

Related Papers