Automatic differentiation in machine learning: a survey

Analysis

Why This Paper Matters

This survey is a foundational reference for understanding the role of automatic differentiation (AD) in machine learning. Published in 2017, it arrived at a pivotal moment when deep learning was exploding in popularity, and the need for efficient gradient computation was paramount. The paper clarifies the distinctions between AD, symbolic differentiation, and numerical differentiation, and argues that AD is the most suitable technique for ML. It has been cited over 3700 times, indicating its significant influence on both research and practice.

The paper also addresses a critical gap: despite AD's success in other fields, ML practitioners were slow to adopt it. By dispelling misconceptions and highlighting successful applications, the authors helped bridge this gap. This work is often credited with contributing to the widespread adoption of AD frameworks like TensorFlow and PyTorch, which have become the standard tools for modern deep learning.

Technical Contributions

• Comprehensive taxonomy of AD techniques: The paper categorizes AD into forward and reverse modes, explaining their computational complexities and suitability for different problem sizes.
• Clarification of misconceptions: It addresses common misunderstandings, such as the confusion between AD and symbolic differentiation, and the belief that AD is inefficient for high-dimensional problems.
• Survey of applications: It covers a wide range of ML applications where AD is beneficial, including gradient-based optimization, hyperparameter tuning, sensitivity analysis, and solving differential equations.
• Discussion of software tools: The paper reviews existing AD tools and frameworks, highlighting their features and limitations, and discusses the integration of AD into ML libraries.

Results

As a survey, the paper does not present new experimental results. Instead, it synthesizes existing knowledge and provides a qualitative assessment of AD's advantages. It emphasizes that AD can compute derivatives with machine precision and at a cost proportional to the original function evaluation, which is a key advantage over numerical differentiation. The paper's impact is measured by its citation count (3701) and its role in shaping the adoption of AD in ML.

Significance

The broader impact of this paper is immense. It has helped establish AD as a core technology in machine learning, enabling the development of deep learning frameworks that rely on automatic gradient computation. The concepts discussed, such as reverse-mode AD (backpropagation), are now fundamental to training neural networks. The paper also influenced the design of differentiable programming languages and the growth of areas like neural ODEs and physics-informed neural networks. By providing a clear and accessible overview, it has educated a generation of ML practitioners and researchers, making AD an indispensable tool in the AI field.