# 15-779 | Schedule https://www.cs.cmu.edu/~zhihaoj2/15-779/schedule.html ## [15-779: Advanced Topics in Machine Learning Systems (LLM Edition) ](https://www.cs.cmu.edu/~zhihaoj2/15-779/) * [ Home ](https://www.cs.cmu.edu/~zhihaoj2/15-779/) * [ Logistics ](https://www.cs.cmu.edu/~zhihaoj2/15-779/logistics.html) * [ Schedule ](https://www.cs.cmu.edu/~zhihaoj2/15-779/schedule.html) * [ Materials ](https://www.cs.cmu.edu/~zhihaoj2/15-779/materials.html) ## Schedule * When: Wednesday/Friday 11:00 - 12:20pm * Where: GHC 4303 | Date | Plan | Readings | | --- | --- | --- | | 08/29 Fri | Week 1: **Course Introduction** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/01-course-introduction.pdf) ] | * (Required) [ How to Read a Paper](https://cs.cmu.edu/~zhihaoj2/papers/HowtoReadPaper.pdf) | | | 09/03 Wed | Week 2: **Introduction to ML Systems** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/02-MLSys-basics.pdf) ] | * (Required) [ TensorFlow: A System for Large-Scale Machine Learning ](https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf) * (Required) [ PyTorch: An Imperative Style, High-Performance Deep Learning Library ](https://arxiv.org/abs/1912.01703) | | | 09/05 Fri | Week 2: **GPU Architecture and CUDA Programming (1)** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/03-CUDA-programming.pdf) ] | | | | 09/10 Wed | Week 3: **GPU Architecture and CUDA Programming (2)** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/03-CUDA-programming.pdf) ] | | | | 09/12 Fri | Week 3: **Case Study: Transformers, Attention, FlashAttention** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/05-transformers-attention.pdf) ] | * (Required) [Transformer: Attention is All you Need ](https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html) * (Required) [Fast and Memory-Efficient Exact Attention with IO-Awareness ](https://arxiv.org/abs/2205.14135) * (Optional) [ Language Models are Unsupervised Multitask Learners ](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) | | | 09/17 Wed | Week 4: **Advanced CUDA Programming: Warp Specialization** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/06-warp-specialization.pdf) ] | | | | 09/19 Fri | Week 4: **Advanced CUDA Programming: Mega Kernel** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/07-mega-kernel.pdf) ] | * (Required) [ Designing a Low-Latency Megakernel for Llama-1B ](https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles) * (Required) [ Compiling LLMs into a MegaKernel ](https://zhihaojia.medium.com/compiling-llms-into-a-megakernel-a-path-to-low-latency-inference-cf7840913c17) | | | 09/24 Wed | Week 5: **ML Compilers Part 1: Tile-based DSL** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/08-ML-compilers-part-1.pdf) ] | * (Required) [ Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations ](https://www.eecs.harvard.edu/~htk/publication/2019-mapl-tillet-kung-cox.pdf) * (Required) [TVM: An Automated End-to-End Optimizing Compiler for Deep Learning](https://www.usenix.org/conference/osdi18/presentation/chen) * (Optional) [ MLIR: A Compiler Infrastructure for the End of Moore's Law ](https://arxiv.org/abs/2002.11054) | | | 09/26 Fri | Week 5: **ML Compilation: Kernel Auto-tuning** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/09-ML-compilers-part-2.pdf) ] | * (Required) [Learning to Optimize Tensor Programs](https://arxiv.org/abs/1805.08166) * (Required) [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://arxiv.org/abs/2006.06762) * (Optional) [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://arxiv.org/abs/1802.04730) | | | 10/01 Wed | Week 6: **Advanced ML Compilation: Graph-Level Optimizations** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/10-graph-level-optimizations.pdf) ] | * (Required) [ TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions](https://cs.stanford.edu/~zhihao/papers/sosp19.pdf) * (Required) [ PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections](https://www.usenix.org/conference/osdi21/presentation/wang) * (Optional) [ Equality Saturation for Tensor Graph Superoptimization ](https://arxiv.org/pdf/2101.01332.pdf) | | | 10/03 Fri | Week 6: **Advanced ML Compilation: ML Superoptimization** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/11-kernel-superoptimization.pdf) ] | * (Required) [ Mirage: A Multi-Level Superoptimizer for Tensor Programs ](https://arxiv.org/abs/2405.05751) | | | 10/08 Wed | Week 7: **ML Parallelization: Data Parallelism** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/12-ML-parallelization-part1.pdf) ] | * (Required) [ ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054) * (Required) [ ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://arxiv.org/abs/2104.07857) * (Optional) [ ZeRO-Offload: Democratizing Billion-Scale Model Training](https://arxiv.org/abs/2101.06840) | | | 10/10 Fri | Week 7: **ML Parallelization: Zero Redundancy and PyTorch Fully Sharded Data Parallelism** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/12-ML-parallelization-part1.pdf) ] | * (Required) [ ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://arxiv.org/abs/1910.02054) * (Required) [ ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://arxiv.org/abs/2104.07857) * (Optional) [ ZeRO-Offload: Democratizing Billion-Scale Model Training](https://arxiv.org/abs/2101.06840) | | | 10/15 Wed | Week 8: **Fall Break; No Class** [ slides ] | | | | 10/17 Fri | Week 8: **Fall Break; No Class** [ slides ] | | | | 10/22 Wed | Week 9: **ML Parallelization: Model and Pipeline Parallelism** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/13-ML-parallelization-part2.pdf) ] | * (Required) [ PipeDream: Generalized Pipeline Parallelism for DNN Training ](https://www.microsoft.com/en-us/research/uploads/prod/2019/08/fiddle_pipedream_sosp19.pdf) * (Required) [ Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://arxiv.org/abs/1909.08053) * (Optional) [ Memory-Efficient Pipeline-Parallel DNN Training ](http://proceedings.mlr.press/v139/narayanan21a/narayanan21a.pdf) | | | 10/24 Fri | Week 9: **Advanced ML Parallelization: Automated ML Parallelization** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/14-auto-parallelization.pdf) ] | * (Required) [ Beyond Data and Model Parallelism for Deep Neural Networks ](https://proceedings.mlsys.org/paper/2019/hash/c74d97b01eae257e44aa9d5bade97baf-Abstract.html) * (Required) [Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning](https://www.usenix.org/system/files/osdi22-zheng-lianmin.pdf) | | | 10/29 Wed | Week 10: **LLMs: Serving (Batching, PagedAttention, RadixAttention)** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/15-LLM-serving-part1.pdf) ] | * (Required) [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://arxiv.org/pdf/2309.06180) * (Required) [SGLang: Efficient Execution of Structured Language Model Programs](https://arxiv.org/pdf/2312.07104) | | | 10/31 Fri | Week 10: **LLMs: Serving (Speculative Decoding)** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/16-LLM-serving-part2.pdf) ] | * (Required) [Fast Inference from Transformers via Speculative Decoding](https://proceedings.mlr.press/v202/leviathan23a/leviathan23a.pdf) * (Required) [Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://arxiv.org/abs/2305.09781) | | | 11/05 Wed | Week 11: **LLMs: Training (Parameter-Efficient Fine-Tuning)** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/17-LLM-finetuning.pdf) ] | * (Required) [ LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685) * (Required) [ QLoRA: Efficient Finetuning of Quantized LLMs](https://arxiv.org/abs/2305.14314) | | | 11/07 Fri | Week 11: **Meetings to discuss project ideas** [ slides ] | | | | 11/12 Wed | Week 12: **Guest Lecture: New Problems in LLM Post-training Systems** [ slides ] | | Banghua Zhu (UW/NVIDIA) | | 11/14 Fri | Week 12: **Guest Lecture: Advancing the Pareto Frontier of Training Open Language Models** [ slides ] | | Mengzhou Xia (OpenAI/CMU) | | 11/19 Wed | Week 13: **LLMs: Mixture-of-Experts (Architectures, Kernels, Parallelism)** [ [slides](https://www.cs.cmu.edu/~zhihaoj2/15-779/slides/18-mixture-of-experts.pdf) ] | | | | 11/21 Fri | Week 13: **Working on course project** [ slides ] | | | | 11/26 Wed | Week 14: **Thanksgiving Break; No Class** [ slides ] | | | | 11/28 Fri | Week 14: **Thanksgiving Break; No Class** [ slides ] | | | | 12/03 Wed | Week 15: [ slides ] | | | | 12/05 Fri | Week 15: | [ ![SCS Logo](https://www.cs.cmu.edu/~zhihaoj2/15-779/img/logo/cmuscs.png) ](https://cs.cmu.edu) [ ![Catalyst Logo](https://www.cs.cmu.edu/~zhihaoj2/15-779/img/logo/catalyst.svg) ](https://catalyst.cs.cmu.edu)