Multi-Task Learning for Dense Prediction Tasks: A Survey

Analysis

Why This Paper Matters

Multi-task learning (MTL) has emerged as a promising paradigm in computer vision, enabling simultaneous learning of multiple dense prediction tasks such as semantic segmentation, depth estimation, and surface normal prediction. By sharing representations across tasks, MTL can improve performance, reduce computational cost, and decrease memory footprint compared to training separate models for each task. This survey, published in IEEE TPAMI, provides a timely and comprehensive overview of the state-of-the-art in MTL for dense prediction tasks, which are crucial for applications like autonomous driving, robotics, and augmented reality.

The paper's significance lies in its holistic treatment of both architectural and optimization aspects of MTL. It not only categorizes existing methods but also offers an experimental evaluation across multiple benchmarks, providing practical insights into the strengths and weaknesses of different approaches. With over 1000 citations, it has become a key reference for researchers entering the field or seeking to advance MTL techniques.

Technical Contributions

• Architecture Overview: The survey systematically reviews MTL network architectures, including hard parameter sharing, soft parameter sharing, and modular designs. It discusses the trade-offs between shared and task-specific parameters, and how different architectures handle the balance between tasks.
• Optimization Methods: It examines various optimization strategies for joint learning, such as loss weighting schemes, gradient manipulation, and task scheduling. The paper highlights the challenges of conflicting gradients and negative transfer, and summarizes techniques to mitigate them.
• Experimental Evaluation: The authors conduct extensive experiments on dense prediction benchmarks (e.g., NYUD-v2, PASCAL-Context, Cityscapes) to compare different MTL methods. They evaluate both architectural and optimization-based strategies, providing quantitative results and qualitative analyses.
• Taxonomy and Guidelines: The paper proposes a taxonomy of MTL approaches and offers guidelines for selecting appropriate methods based on task relatedness, computational constraints, and performance goals.

Results

The survey reports experimental results across several benchmarks, showing that MTL can outperform single-task learning in many cases, but also that negative transfer can occur if tasks are not properly balanced. For example, on NYUD-v2, certain MTL architectures achieve improvements in semantic segmentation and depth estimation while reducing inference time. The paper also highlights that optimization methods like uncertainty weighting and GradNorm can lead to better performance than simple loss summation. However, the results vary depending on the task combination and dataset, underscoring the need for careful design.

Significance

This survey has had a substantial impact on the computer vision community, as evidenced by its high citation count. It provides a foundational understanding of MTL for dense prediction, guiding future research directions such as dynamic architecture search, task grouping, and efficient training. By consolidating knowledge and offering practical evaluations, it accelerates progress toward more integrated and efficient vision systems. The insights are also relevant to broader AI areas where multi-task learning is applied, including NLP and speech recognition.