Graduate Database Systems (Spring 2020)
* **When**: *Tuesday/Thursday 9:30-12:00*
layout: default
Graduate Database Systems (Spring 2020)
- When: Tuesday/Thursday 9:30-12:00
- Where: Soda 320
- Instructor: Joe Hellerstein
- Office Hours: TBD.
- Announcements: Piazza
- Sign-up to Present: Google Spreadsheet Later in the semester we will be doing student presentations which we will fill in here.
- If you have reading suggestions please send a pull request to this course website on Github by modifying the index.md file.
Course Description
The Database Systems field has been exploring issues in data storage, management, processing and analysis for over 50 years. In our data-rich 21st century, the lessons of the field apply broadly across Computer Science and beyond.
In this course we will study a range of papers from the distant and recent past. The goal is to provide a strong foundation for applied computer scientists of many stripes, whether or not they plan to use the techniques in the context of traditional database systems. Given the breadth and history of the field, it is not possible to cover it comprehensively. Our focus will largely be on core database systems topics, and while the emphasis will be on systems work we will not shy away from theory. The readings will mix classic papers with emerging results, and revisit topics that may be ripe for revival given current technology trends.
New Course Format
The course will have two phases. For the first few weeks, we will read papers from some of the foundational systems in the field. Classes will be in lecture format for this phase, with class discussion. This will give an overview of recurring themes in the field at a relatively high level, and provide a common basis for subsequent papers.
In the second phase, we will follow a weekly pattern used in other Berkeley graduate courses in recent years. The Thursday lecture will be presented by Prof. Hellerstein and will cover the context of the topic as well as a high-level overview of the reading for the week. The Tuesday lecture will be organized around a mini program committee meeting for the weeks readings. Students will be required to submit detailed reviews for a subset of the papers and lead the paper review discussions. The goal of this format is to both build a mastery of the material and develop your ability to evaluate and review research--including your own.
Course Syllabus
<!-- This is the dates for all the lectures -->{% capture dates %} 1/21/20 1/23/20 1/28/20 1/30/20 2/4/20 2/6/20 2/11/20 2/13/20 2/18/20 2/20/20 2/25/20 2/27/20 3/3/20 3/5/20 3/10/20 3/12/20 3/17/20 3/19/20 3/24/20 3/26/20 3/31/20 4/2/20 4/7/20 4/9/20 4/14/20 4/16/20 4/21/20 4/23/20 4/30/20 5/5/20 5/7/20 {% endcapture %} {% assign dates = dates | split: " " %}
This is a tentative schedule. Specific readings are subject to change as new material is published.
<a href="#today"> Jump to Today </a>
<table class="table table-striped syllabus"> <thead> <tr> <th style="width: 5%"> Week </th> <th style="width: 10%"> Date (Lec.) </th> <th style="width: 85%"> Topic </th> </tr> </thead> <tbody>{% include syllabus_entry %}
Introduction and Course Overview
This lecture will be an overview of the class, requirements, and an introduction to the history of machine learning and systems research.
<div class="reading"> <div class="optional_reading" markdown="1">- How to read a paper provides some pretty good advice on how to read papers effectively.
- Timothy Roscoe's writing reviews for systems conferences will also help you in the reviewing process.
{% include syllabus_entry %}
Holiday (Labor Day)
There will be no class but please sign-up for the weekly discussion slots.
{% include syllabus_entry %}
Big Ideas and How to Evaluate ML Systems Research
- Submit your review before 1:00PM.
- Lecture slides: [pdf, pptx]
- SysML: The New Frontier of Machine Learning Systems
- Read Chapter 1 of Principles of Computer System Design. You will need to be on campus or use the Library VPN to obtain a free PDF.
- A Few Useful Things to Know About Machine Learning
- A Berkeley View of Systems Challenges for AI
Additional Machine Learning Reading
- Kevin Murphy's Textbook Introduction to Machine Learning. This provides a very high-level overview of machine learning. You should probably know all of this.
- Stanford CS231n Tutorial on Neural Networks. I recommend reading Module 1 for a quick crash course in machine learning and some of the techniques used in this class.
Additional Systems Reading
Open Debate about the Field
- Rich Sutton's Post on Compute in ML and the corresponding Shimon Whiteson twitter debate
{% include syllabus_entry %}
Machine Learning Life-cycle
This lecture will discuss the machine learning life-cycle, spanning model development, training, and serving. It will outline some of the technical machine learning and systems challenges at each stage and how these challenges interact.
- Lecture slides: [pdf, pptx]
- Template Slide Format for PC Meeting [Google Drive]
{% include syllabus_entry %}
Discussion of Papers on Machine Learning Life-cycle
- Submit your review before 1:00PM.
- Slides and scribe notes from the PC Meeting. (These are only accessible to students enrolled in the class.)
- Hidden Technical Debt in Machine Learning Systems
- TFX: A TensorFlow-Based Production-Scale Machine Learning Platform
- Towards Unified Data and Lifecycle Management for Deep Learning
Software:
</div> </div>{% include syllabus_entry %}
Database Systems and Machine Learning
In the previous lecture we saw that data and feature engineering is often the dominant hurtle in model development. Database systems are often the source of data and the platform in which feature engineering takes place. This lecture will cover some of the big ideas is database systems and how they relate to work on machine learning in databases.
- Lecture slides: [pdf, pptx]
- Project Proposal Sign-up doc. You must be enrolled in the class or on the waitlist to access this document. Please add any projects you are thinking about starting and list yourself as interested in anyone else's projects.
{% include syllabus_entry %}
Discussion of Database Systems and Machine Learning
- Submit your review before 1:00PM.
- Slides for PC Meeting posted. (These slides will only be accessible to students enrolled in the class.)
- Towards a Unified Architecture for in-RDBMS Analytics
- Materialization Optimizations for Feature Selection Workloads
- Learning Generalized Linear Models Over Normalized Data
- Learning Linear Regression Models over Factorized Joins
- MauveDB: Supporting Model-based User Views in Database Systems
- The MADlib Analytics Library or MAD Skills, the SQL
{% include syllabus_entry %}
Machine Learning Frameworks and Automatic Differentiation
This week we will discuss recent development in model development and training frameworks. While there is a long history of machine learning frameworks we will focus on frameworks for deep learning and automatic differentiation. In class we will review some of the big trends in machine learning framework design and basic ideas in forward and backward automatic differentiation.
Project proposals are due next Monday
{% include syllabus_entry %}
Machine Learning Frameworks and Automatic Differentiation
Update: Two of the readings were changed to reflect a focus on deep learning frameworks. The previous readings on SystemML and KeystoneML have been moved to optional reading.
- Submit your review before 1:00PM.
- Slides for PC Meeting ] These slides will only be accessible to students enrolled in the class.
- Automatic differentiation in ML: Where we are and where we should be going
- TensorFlow: A System for Large-Scale Machine Learning
- JANUS: Fast and Flexible Deep Learning via Symbolic Graph Execution of Imperative Programs
Pipeline Training Frameworks (Classical)
- KeystoneML: Optimizing Pipelines for Large-Scale Advanced Analytics
- SystemML: Declarative Machine Learning on Spark
Automatic Differentiation and Differentiable Programming
- Automatic Differentiation in Machine Learning: a Survey
- Roger Grosse's Lecture Notes on Automatic Differentiation
- A Differentiable Programming System to Bridge Machine Learning and Scientific Computing
Deep Learning Frameworks with Automatic Differentiation
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Theano: A Python Framework for Fast Computation of Mathematical Expressions and Theano: A CPU and GPU Math Compiler in Python
- Automatic differentiation in PyTorch
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Deep Learning Primitives
</div> </div>{% include syllabus_entry %}
Distributed Model Training
This week we will discuss developments in distributed training. We will quickly review the statistical query model pushed by early map-reduce machine learning frameworks and then discuss advances in parameter servers and distributed neural network training.
Project Proposals Due!
- One Page Project description due at 11:59 PM. Check out the suggested projects. Submit a link to your one page Google document containing your project descriptions to this google form. You only need one submission per team but please list all the team member's email addresses. You can also update your submission if needed.
{% include syllabus_entry %}
Discussion on Distributed Model Training
- Submit your review before 1:00PM.
- Slides for PC Meeting (These slides will only be accessible to students enrolled in the class.)
- Scaling Distributed Machine Learning with the Parameter Server
- PipeDream: Generalized Pipeline Parallelism for DNN Training
- Adaptive Communication Strategies to Achieve the Best Error-Runtime Trade-off in Local-Update SGD
- Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
- Integrated Model, Batch, and Domain Parallelism in Training Neural Networks
- Effect of batch size on training dynamics
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Hogwild!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent[pdf]
- Large Scale Distributed Deep Networks
- Scaling Distributed Machine Learning with In-Network Aggregation
ImageNet in X Minutes
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Now anyone can train Imagenet in 18 minutes
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
All-Reduce
- Baidu Ring All-Reduce Blog Post
- The original Ring All-Reduce Paper "Bandwidth Optimal All-reduce Algorithms for Clusters of Workstations"
- Visual intuition on ring-Allreduce for distributed Deep Learning
- Double Binary Trees
{% include syllabus_entry %}
Prediction Serving
Until recently, much of the focus on systems research was aimed at model training. However, recently there has been a growing interest in addressing the challenges of prediction serving. This lecture will frame the challenges of prediction serving and cover some of the recent advances.
{% include syllabus_entry %}
Power Outage Related Holiday
Unfortunately, class was canceled and so the PC Meeting has been moved to Monday. Note that early project presentations are also due next Friday.
{% include syllabus_entry %}
Discussion on Prediction Serving
- Submit your review before 1:00PM.
- Slides for PC Meeting (These slides will only be accessible to students enrolled in the class.)
- Pretzel: Opening the Black Box of Machine Learning Prediction Serving Systems
- InferLine: ML Inference Pipeline Composition Framework (pre-print)
- Focus: Querying Large Video Datasets with Low Latency and Low Cost
The Prediction-Serving Systems: What happens when we wish to actually deploy a machine learning model to production? ACM Queue article provides a nice overview.
Systems Reading:
- Live Video Analytics at Scale with Approximation and Delay-Tolerance
- LASER: A Scalable Response Prediction Platform For Online Advertising
- TensorFlow-Serving: Flexible, High-Performance ML Serving
- Clipper: A Low-Latency Online Prediction Serving System
- Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
- The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
- The Case for Predictive Database Systems: Opportunities and Challenges.
More Efficient Models:
- Paul Viola and Michael Jones Rapid Object Detection using a Boosted Cascade of Simple Features CVPR 2001.
Performance Breakdown of various models
</div> </div>{% include syllabus_entry %}
Project Presentations
{% include syllabus_entry %}
Finish Project Presentations and Start Model Compilation
This week we will explore the process of compiling/optimizing deep neural network computation graphs. This reading will span both graph level optimization as well as the compilation and optimization of individual tensor operations.
{% include syllabus_entry %}
Discussion of Model Compilation
- Submit your review before 1:00PM.
- Slides for PC Meeting (These slides will only be accessible to students enrolled in the class.)
- Optimizing DNN Computation with Relaxed Graph Substitutions
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- Learning to Optimize Halide with Tree Search and Random Programs
- Learning to Optimize Tensor Programs: The TVM story is two fold. There's a System for ML story (above paper) and this paper is their the ML for System story.
- Exploring Hidden Dimensions in Parallelizing Convolutional Neural Networks
- TensorComprehensions
- Supporting Very Large Models using Automatic Dataflow Graph Partitioning
{% include syllabus_entry %}
PG&E and Fire Related Cancellation
Unfortunately, due to the power outage, lecture is canceled today. To make up for lost lecture(s) and accommodate our guest speakers, we will skip the overview lecture this week and start with the PC meeting on Machine Learning Applied to Systems. However, this will put a little extra pressure on the neutral presenters to provide additional context. We will then cover the discussion on machine learning hardware the following Monday.
{% include syllabus_entry %}
Discussion of Machine Learning Applied to Systems Day 1
- Submit your review before 1:00PM.
- Slides for PC Meeting (These slides will only be accessible to students enrolled in the class.)
- Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms
- Device Placement Optimization with Reinforcement Learning
- The Case for Learned Index Structures
{% include syllabus_entry %}
Hardware Acceleration for Machine Learning
This lecture will be presented by Kurt Keutzer and Suresh Krishna who are experts in processor design as well as network and architecture co-design.
{% include syllabus_entry %}
Discussion Hardware Acceleration for Machine Learning
- Submit your review before 1:00PM.
- Slides for PC Meeting (These slides will only be accessible to students enrolled in the class.)
- A Configurable Cloud-Scale DNN Processor for Real-Time AI
- In-Datacenter Performance Analysis of a Tensor Processing Unit
- Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks
- Efficient Processing of Deep Neural Networks: A Tutorial and Survey
- A great spreadsheet analysis of the power and performance characteristics of all the publicly available hardware accelerators for deep learning (GPUs, CPU, TPUs).
- Nvidia post comparing different GPUs across a wide range of networks.
{% include syllabus_entry %}
(11/11) Administrative Holiday
{% include syllabus_entry %}
Discussion of Machine Learning Applied to Systems Day 2
- Submit your review before 1:00PM.
- Slides for PC Meeting coming soon. (These slides will only be accessible to students enrolled in the class.)
- AuTO: Scaling Deep Reinforcement Learning to Enable Datacenter-Scale Automatic Traffic Optimization
- Neural Adaptive Video Streaming with Pensieve
- Neural Adaptive Content-aware Internet Video Delivery
{% include syllabus_entry %}
Learning with Adversaries
This week we will discuss machine learning in adversarial settings. This includes secure federated learning, differential privacy, and adversarial examples.
{% include syllabus_entry %}
Discussion on Learning with Adversaries
- Submit your review before 1:00PM.
- Slides for PC Meeting coming soon. (These slides will only be accessible to students enrolled in the class.)
- Communication-Efficient Learning of Deep Networks from Decentralized Data
- Privacy Accounting and Quality Control in the Sage Differentially Private ML Platform
- Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
- Helen: Maliciously Secure Coopetitive Learning for Linear Models
- Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference
- Rendered Insecure: GPU Side Channel Attacks are Practical
- The Algorithmic Foundations of Differential Privacy
- Federated Learning: Collaborative Machine Learning without Centralized Training Data
- Federated Learning at Google ... A comic strip?
- SecureML: A System for Scalable Privacy-Preserving Machine Learning
- More reading coming soon ...
{% include syllabus_entry %}
Autonomous Driving
Autonomous vehicles will likely transform society in the next decade and are fundamentally AI enabled systems. In this lecture we will discuss the AI-Systems challenges around autonomous driving.
{% include syllabus_entry %}
(11/29) Holiday (Thanksgiving)
{% include syllabus_entry %}
Discussion on Autonomous Driving
Everyone must do one of the readings (you pick).
- Submit your review before 1:00PM.
- Slides for PC Meeting coming soon. (These slides will only be accessible to students enrolled in the class.)
- Self-Driving Cars: A Survey. This is a slightly longer survey so focus more on the overview and framing first few pages of the autonomous driving problem and common solutions.
- The Architectural Implications of Autonomous Driving: Constraints and Acceleration
- ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst
- An Open Approach to Autonomous Vehicles
- End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners
DARPA Grand Challenges
- Software Infrastructure for an Autonomous Ground Vehicle
- Stanley: The Robot that Won the DARPA Grand Challenge
- Tartan Racing: A Multi-Modal Approach to the DARPA Urban Challenge
- Towards a Viable Autonomous Driving Research Platform
- Engineering Autonomous Driving Software
{% include syllabus_entry %}
Conclusion!
{% include syllabus_entry %}
(12/6) RRR Week
{% include syllabus_entry %}
(12/9) RRR Week
{% include syllabus_entry %}
(12/16) Poster Presentations
{% include syllabus_entry %}
(12/20) No Class
Don't forget to submit your final reports. As noted on Piazza, the final report should be 6-pages plus references (2-column, 10pt font, unlimited appendix). Please submit your report using this form:
<center> <h3> <a href="https://forms.gle/tLcLeEzRzueFGUcG6">Submit Your Report Here</a> </h3> </center>You only need to submit the project once per team. The write-up should discuss the problem formulation, related work, your approach, and your results.
</td> </tr> </tbody> </table>Projects
Detailed candidate project descriptions will be posted shortly. However, students are encourage to find projects that relate to their ongoing research.
Grading
Grades will be largely based on class participation and projects. In addition, we will require weekly paper summaries submitted before class.
- Projects: 60%
- Weekly Summaries: 20%
- Class Participation: 20%
Related Documents
Community AI Agent Skills Discovery Sources
**Research Date:** 2026-03-26
GPU Selection Guide for Large Language Models (LLMs)
This guide helps you choose the right GPU for running Large Language Models, whether you're using them for inference, fine-tuning, or training.
ReleaseKit - Technical Requirements Document
ReleaseKit provides **two interfaces** to the same underlying functionality:
api_llm Specification
Provide direct, transparent HTTP API bindings for major LLM providers without abstraction layers or automatic behaviors.