ICML2022-Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models
FreeTechniques and systems to train and serve bigger models
FreeFree tier
About ICML2022-Welcome to the "Big Model" Era: Techniques and Systems to Train and Serve Bigger Models
This is the ICML 2022 tutorial 'Welcome to the Big Model Era: Techniques and Systems to Train and Serve Bigger Models' presented by Hao Zhang, Lianmin Zheng, Zhuohan Li, and Ion Stoica. It covers the motivation and challenges of scaling up models, introduces new algorithmic techniques and system architectures for model-parallel training and inference, and provides guidance on using open-source toolsets to train and serve big models like GPT-3, PaLM, and vision transformers. The tutorial includes sessions on intra- and inter-operator parallelism, auto parallelization, and hands-on tool usage.
Key Features
Introduction to the big model era and trends driving model scaling
Intra-operator and inter-operator parallelism strategies
Auto parallelization of ML computation
Coverage of systems for training and serving GPT-3, PaLM, and vision transformers
Tutorial on using open-source toolsets for big model training and serving
Presented by leading researchers from UC Berkeley
Pros & Cons
Pros
- Free and accessible tutorial recording
- Covers both algorithmic and system-level perspectives
- Includes practical sessions with open-source tools
- Presented by experts in the field
Cons
- Tutorial is from 2022 and may not reflect the latest advances
- Not a standalone tool; requires supplementary materials and tools to apply
- Limited to introductory and intermediate depth; advanced users may need more detail
Best For
Training large-scale models with billions of parametersServing large models efficiently for inferenceUnderstanding model parallelism and distributed training techniquesLowering the technical barrier for adopting big models in ML researchLearning about open-source systems for large model deployment
FAQ
What is this tutorial about?
It is an ICML 2022 tutorial titled 'Welcome to the Big Model Era: Techniques and Systems to Train and Serve Bigger Models' that explains techniques and systems for training and serving large models such as GPT-3, PaLM, and vision transformers.
Who are the presenters?
The presenters are Hao Zhang, Lianmin Zheng, Zhuohan Li, and Ion Stoica.
What topics are covered?
Topics include trends driving big models, intra- and inter-operator parallelism, auto parallelization of ML computation, and tools for big model training and serving.
Is the tutorial free?
Yes, the tutorial is freely available as part of ICML 2022.
Where can I find the tutorial materials?
The tutorial includes a project page (linked on the site) and video recordings of each session.