Preprint
Large Language Models

Text Data Augmentation for Deep Learning

Connor Shorten(Florida Atlantic University), Taghi M. Khoshgoftaar(Florida Atlantic University), Borko Furht(Florida Atlantic University)
July 19, 2021Journal Of Big Data1,685 citations
PDFDOI

1.7k

Citations

12

Influential Citations

Journal Of Big Data

Venue

2021

Year

Abstract

Natural Language Processing (NLP) is one of the most captivating applications of Deep Learning. In this survey, we consider how the Data Augmentation training strategy can aid in its development. We begin with the major motifs of Data Augmentation summarized into strengthening local decision boundaries, brute force training, causality and counterfactual examples, and the distinction between meaning and form. We follow these motifs with a concrete list of augmentation frameworks that have been developed for text data. Deep Learning generally struggles with the measurement of generalization and characterization of overfitting. We highlight studies that cover how augmentations can construct test sets for generalization. NLP is at an early stage in applying Data Augmentation compared to Computer Vision. We highlight the key differences and promising ideas that have yet to be tested in NLP. For the sake of practical implementation, we describe tools that facilitate Data Augmentation such as the use of consistency regularization, controllers, and offline and online augmentation pipelines, to preview a few. Finally, we discuss interesting topics around Data Augmentation in NLP such as task-specific augmentations, the use of prior knowledge in self-supervised learning versus Data Augmentation, intersections with transfer and multi-task learning, and ideas for AI-GAs (AI-Generating Algorithms). We hope this paper inspires further research interest in Text Data Augmentation.

Analysis

Why This Paper Matters

This survey is significant because it systematically organizes the rapidly growing field of text data augmentation for deep learning, a topic that has become critical as NLP models scale and require more robust generalization. By categorizing augmentation motifs and frameworks, the paper provides a clear roadmap for practitioners and researchers. It also highlights the gap between NLP and computer vision in augmentation maturity, pointing to underexplored opportunities such as counterfactual examples and consistency regularization.

The paper's emphasis on generalization and overfitting is timely, as deep learning models often struggle with these issues. By discussing how augmentations can construct test sets, the authors offer a practical lens for evaluating model robustness. The inclusion of tools like offline/online pipelines and controllers makes the survey actionable for engineers deploying NLP systems.

Technical Contributions

  • Motif categorization: The paper distills augmentation strategies into four motifs: strengthening local decision boundaries, brute force training, causality and counterfactual examples, and the meaning-form distinction. This provides a conceptual foundation for understanding why augmentations work.
  • Framework enumeration: A concrete list of text augmentation frameworks is provided, covering methods like back-translation, synonym replacement, and generative models.
  • Generalization measurement: The paper highlights how augmentations can be used to construct test sets that measure generalization, addressing a key challenge in deep learning.
  • Practical tools: Describes consistency regularization, controllers, and offline/online augmentation pipelines, offering implementation guidance.
  • Cross-domain comparison: Identifies key differences between NLP and computer vision augmentation, such as the discrete nature of text versus continuous images, and suggests promising ideas like adversarial augmentation for NLP.

Results

The survey does not present new experimental results but synthesizes findings from 1685 cited works. It notes that NLP augmentation is less mature than computer vision, with many techniques still in early stages. The paper does not provide quantitative metrics or benchmarks, but its value lies in the comprehensive taxonomy and identification of research gaps.

Significance

This paper serves as a foundational reference for researchers and practitioners working on text data augmentation. By clarifying the theoretical underpinnings and practical tools, it can accelerate progress in NLP robustness and generalization. The discussion of AI-GAs (AI-Generating Algorithms) and intersections with transfer and multi-task learning points to future directions that could shape the field. The high citation count (1685) reflects its impact as a go-to survey for the community.

data-augmentationnatural-language-processingdeep-learningtext-augmentationgeneralizationoverfittingself-supervised-learning
Computer Vision

Related Papers