# Schedule - Large Language Models: Methods and Applications / Fall 2024

https://2024.cmu-llms.org/schedule/

| ![](https://2024.cmu-llms.org/_images/logo.png)  |  [Carnegie Mellon University](https://www.lti.cs.cmu.edu/)   
[**Large Language Models: Methods and Applications**](https://2024.cmu-llms.org/ "Large Language Models: Methods and Applications / Fall 2024 - Carnegie Mellon University")   
Fall 2024  |  
| --- | --- |  
[ ](https://2024.cmu-llms.org/schedule/)
# Main Navigation
  * [ Home ](https://2024.cmu-llms.org/)
  * [ Syllabus ](https://2024.cmu-llms.org/syllabus/)
  * [ Schedule ](https://2024.cmu-llms.org/schedule/)
  * [ Assignments ](https://2024.cmu-llms.org/assignments/)
  * [ Useful Links ](https://2024.cmu-llms.org/materials/)


# Schedule
  * Event
Date
Description
Course Material
  * Lecture
08/27/2024   
Tuesday 
**Building blocks of modern LLMs**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-08-26-course-intro-and-lm-basics.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [Jurasky and Martin, Chapter 10](https://web.stanford.edu/~jurafsky/slp3/10.pdf)
    * [Illustrated Transformer](http://jalammar.github.io/illustrated-transformer/)
  * Lecture
08/29/2024   
Thursday 
**Transformer architecture and pre-training learning objectives**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-08-29-lm-architectures-training-objectives.pdf%20%0A%20%20%20%20 "Download slides")] [[quiz](https://storage.googleapis.com/cmu-llms/2024/quizzes/08-29-2024.pdf%20%0A%20%20%20%20 "Download quiz")] 
Readings:
    * [What Language Model Architecture and Pretraining Objective Work Best](https://proceedings.mlr.press/v162/wang22u/wang22u.pdf)
  * Assignment
09/01/2024   
Sunday 
Assignment #1 released 
[[Preparing and Exploring Pre-Training Data](https://2024.cmu-llms.org/assignments/01)] 
  * Lecture
09/03/2024   
Tuesday 
**Pre-training data curation and tokenization**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-09-03-learning-objectives-and-pretraining-data.pdf%20%0A%20%20%20%20 "Download slides")] [[quiz](https://storage.googleapis.com/cmu-llms/2024/quizzes/09-03-2024.pdf%20%0A%20%20%20%20 "Download quiz")] 
Readings:
    * [Documenting C4](https://arxiv.org/pdf/2104.08758)
    * [Decline of AI Data Commons](https://arxiv.org/pdf/2407.14933)
    * [ClueWeb22](https://arxiv.org/pdf/2211.15848)
  * Lecture
09/05/2024   
Thursday 
**Architecture Advancement on Transformers**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-09-05-architecture-advancement-on-transformers.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [Activation Functions](https://arxiv.org/abs/1710.05941)
    * [Layer norms](https://proceedings.mlr.press/v119/xiong20b/xiong20b.pdf)
    * [Grouped-Query Attention](https://arxiv.org/pdf/2305.13245)
    * [RoPE embeddings](https://arxiv.org/pdf/2104.09864)
  * Lecture
09/10/2024   
Tuesday 
**Automatic evaluation of LLMs**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-09-10-automatic-evaluation.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [GEM Benchmark](https://arxiv.org/pdf/2102.01672.pdf)
    * [BIG-Bench, chapters 1 and 2](https://arxiv.org/pdf/2206.04615)
    * [Holistic Evaluation of Language Models](https://arxiv.org/pdf/2211.09110)
    * [Evaluation Challenges](https://aclanthology.org/D17-1238.pdf)
  * Due
09/10/2024 14:00   
Tuesday 
Assignment #1 due 
  * Lecture
09/12/2024   
Thursday 
**In-context learning, task-oriented finetuning, and parameter-efficient tuning methods**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-09-12-finetuning.pdf%20%0A%20%20%20%20 "Download slides")] [[quiz](https://storage.googleapis.com/cmu-llms/2024/quizzes/09-12-2024.pdf%20%0A%20%20%20%20 "Download quiz")] 
Readings:
    * [Prompt Tuning](https://arxiv.org/abs/2104.08691)
    * [Blog post on param-efficient tuning](https://www.leewayhertz.com/parameter-efficient-fine-tuning/)
    * [Really detailed survey paper](https://arxiv.org/pdf/2203.06904)
  * Assignment
09/14/2024   
Saturday 
Assignment #2 released 
[[Implementing a Transformer from scratch and doing inference with the HuggingFace API](https://2024.cmu-llms.org/assignments/02)] 
  * Lecture
09/17/2024   
Tuesday 
**Retrieval-augmented generation - guest lecture from Akari Asai**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-09-17-akari_cmullm_ralm.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [RAG](https://arxiv.org/abs/2005.11401)
    * [In-context RAG](https://arxiv.org/abs/2302.00083)
    * [REALM](https://arxiv.org/abs/2002.08909)
  * Lecture
09/19/2024   
Thursday 
**Industry guest lecture - Deep Ganguli from Anthropic - Instruction tuning and alignment**
[[slides_instruction_tuning](https://storage.googleapis.com/cmu-llms/2024/2024-09-19-instruction-tuning.pdf%20%0A%20%20%20%20 "Download slides_instruction_tuning")] [[slides_alignment](https://storage.googleapis.com/cmu-llms/2024/cmu_lecture_deep_ganguli_09_19_24.pdf%20%0A%20%20%20%20 "Download slides_alignment")] [[quiz](https://storage.googleapis.com/cmu-llms/2024/quizzes/09-19-2024.pdf%20%0A%20%20%20%20 "Download quiz")] 
Readings:
    * [TO](https://arxiv.org/abs/2110.08207)
    * [FLAN](https://arxiv.org/pdf/2109.01652)
    * [Constitutional AI](https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback)
  * Lecture
09/24/2024   
Tuesday 
**Surprising Behaviours of In-Context Learning and Model Alignment**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-09-24-weird-behaviours.v2.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [Chain-of-thought prompting](https://arxiv.org/pdf/2201.11903)
    * [What makes in-context learning work?](https://aclanthology.org/2022.emnlp-main.759.pdf)
    * [Are aligned models adversarially aligned?](https://proceedings.neurips.cc/paper_files/paper/2023/file/c1f0b856a35986348ab3414177266f75-Paper-Conference.pdf)
  * Lecture
09/26/2024   
Thursday 
**Embedding Learning and Tool Use**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-09-26-embedding-learning-and-tool-use.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [Dense Retrieval Training](https://arxiv.org/pdf/2007.00808)
    * [What are tools anyway](https://arxiv.org/pdf/2403.15452)
    * [Toolformer](https://openreview.net/pdf?id=Yacmpz84TH)
  * Lecture
10/01/2024   
Tuesday 
**Industry Guest lecture - Sara Hooker from Cohere- "What does scale give us"**
[[](https://2024.cmu-llms.org/schedule/coming%20soon%20%0A%20%20%20%20 "Download ")] [[slides](https://2024.cmu-llms.org/schedule/ "Download slides")] [[quiz](https://storage.googleapis.com/cmu-llms/2024/quizzes/10-01-2024.pdf%20%0A%20%20%20%20 "Download quiz")] 
  * Assignment
10/01/2024   
Tuesday 
Assignment #3 released 
[[Retrieval-augmented generation and tool-use](https://2024.cmu-llms.org/assignments/03)] 
  * Due
10/01/2024 14:00   
Tuesday 
Assignment #2 due 
  * Lecture
10/03/2024   
Thursday 
**Scaling Up LLMs - Scaling laws**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-10-03-scaling-up-scaling-laws.pdf%20%0A%20%20%20%20 "Download slides")] 
Suggested Readings:
    * [Scaling laws](https://arxiv.org/abs/2001.08361)
    * [Chinchilla](https://arxiv.org/abs/2203.15556)
  * Lecture
10/08/2024   
Tuesday 
**Scaling up LLMs - Optimization and Parallel Training**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-10-08-Scaling-Up-Parallel-Training.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [GPipe](https://arxiv.org/abs/1811.06965)
    * [DeepSpeed](https://arxiv.org/abs/1910.02054)
    * [Adam](https://arxiv.org/abs/1412.6980)
  * Lecture
10/10/2024   
Thursday 
**Interpretability methods**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-10-10-interpretability.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [Edge Probing](https://arxiv.org/abs/1905.06316)
    * [Loss Landscape](https://arxiv.org/abs/1712.09913)
  * 10/15/2024 00:00   
Tuesday 
Fall break - no class
  * 10/17/2024 00:00   
Thursday 
Fall break - no class
  * Due
10/22/2024 14:00   
Tuesday 
Assignment #3 due 
  * Exam
10/22/2024 14:00   
Tuesday 
Midterm exam 
  * Lecture
10/24/2024   
Thursday 
**RLHF Methods - guest lecture from Yiming Zhang**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-10-24-rlfh-methods.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [RLHF](https://arxiv.org/abs/2203.02155)
    * [DPO](https://arxiv.org/abs/2305.18290)
  * Assignment
10/25/2024   
Friday 
Assignment #4 released 
[[Comparing models and mitigating bias](https://2024.cmu-llms.org/assignments/04)] 
  * Lecture
10/29/2024   
Tuesday 
**Attacking LLM systems + benchmark leakage**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-10-29-attacking-llms.pdf%20%0A%20%20%20%20 "Download slides")] 
Suggested Readings:
    * [How does LLM safety training fail](https://openreview.net/pdf?id=jA235JGM09)
    * [Extracting training data from production models](https://arxiv.org/abs/2311.17035)
  * Lecture
10/31/2024   
Thursday 
**Bias and ethical issues**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-10-31-ethics.pdf%20%0A%20%20%20%20 "Download slides")] 
**Suggested Readings:**
    * [Redteaming LLMs](https://arxiv.org/abs/2209.07858)
  * Lecture
11/05/2024   
Tuesday 
**Democracy day - no class**
[[](https://2024.cmu-llms.org/schedule/coming%20soon%20%0A%20%20%20%20 "Download ")] [[slides](https://2024.cmu-llms.org/schedule/ "Download slides")] 
  * Assignment
11/05/2024   
Tuesday 
Assignment #5 released 
[[Improving training efficiency](https://2024.cmu-llms.org/assignments/05)] 
  * Due
11/05/2024 14:00   
Tuesday 
Assignment #4 due 
  * Lecture
11/07/2024   
Thursday 
**Efficient Pretraining with Sparse Models**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-11-07-efficient-pretraining-with-sparse-models.pdf%20%0A%20%20%20%20 "Download slides")] 
Suggested Readings:
    * [Review of MoE](https://arxiv.org/abs/2209.01667)
    * [Switch Transformer](https://arxiv.org/abs/2101.03961)
    * [MatFormer](https://arxiv.org/pdf/2310.07707)
  * Assignment
11/11/2024   
Monday 
Assignment #6 released 
[[Mini-project](https://2024.cmu-llms.org/assignments/06)] 
  * Lecture
11/12/2024   
Tuesday 
**Efficient Inference Methods**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-11-12-efficient-inference-methods.pdf%20%0A%20%20%20%20 "Download slides")] 
Suggested Readings:
    * [Speculative Decoding](https://arxiv.org/abs/2211.17192)
    * [FlashAttention](https://arxiv.org/abs/2205.14135)
    * [PagedAttention](https://arxiv.org/abs/2309.06180)
  * Lecture
11/14/2024   
Thursday 
**Chatbots and LLM Agents**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-11-14-chatbots-agents.pdf%20%0A%20%20%20%20 "Download slides")] 
– **Readings:**
    * [Chatbot Arena](https://arxiv.org/abs/2403.04132)
    * [Evaluating human evaluation](https://arxiv.org/abs/2107.00061)
  * Due
11/14/2024 14:00   
Thursday 
Assignment #5 due 
  * Lecture
11/19/2024   
Tuesday 
**Legal considerations - guest lecture from A Feder Cooper**
[[slides_ai_authorship](https://storage.googleapis.com/cmu-llms/2024/2024-11-19-ai-authorship.pdf%20%0A%20%20%20%20 "Download slides_ai_authorship")] [[slides_guest_lecture](https://storage.googleapis.com/cmu-llms/2024/2024-11-19-cooper-guest-lecture.pdf%20%0A%20%20%20%20 "Download slides_guest_lecture")] 
Readings:
    * [Generative AI Supply Chain](https://arxiv.org/abs/2309.08133)
  * Lecture
11/21/2024   
Thursday 
**Long-context models**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-11-21-long-context-llms.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [Lost in the middle](https://arxiv.org/abs/2307.03172)
    * [ProLong](https://arxiv.org/abs/2410.02660)
  * Lecture
11/26/2024   
Tuesday 
**Training with Synthetic Data**
[[slides](https://storage.googleapis.com/cmu-llms/2024/2024-11-26-training-with-synthetic-data.pdf%20%0A%20%20%20%20 "Download slides")] 
Readings:
    * [MATES](https://arxiv.org/abs/2406.06046)
    * [Montessori-Instruct](https://arxiv.org/abs/2410.14208)
  * 11/28/2024 00:00   
Thursday 
Thanksgiving - no class
  * Lecture
12/03/2024   
Tuesday 
**Mini-Project Presentation**
[[slides](https://docs.google.com/presentation/d/1X7XvOchAItAKVGAGNC9YSn7BlwAL0_7pA9vE7uqqQVI/edit?usp=sharing%20%0A%20%20%20%20 "Download slides")] 
  * Lecture
12/05/2024   
Thursday 
**Mini-Project Presentation**
[[slides](https://docs.google.com/presentation/d/1P0OUPGl9OLd7a6oXqI9Bx0TP205OCP-A9G6KM-L3_hI/edit?usp=sharing%20%0A%20%20%20%20 "Download slides")] 
  * Due
12/10/2024 23:59   
Tuesday 
Assignment #6 due 


5000 Forbes Ave  
Pittsburgh, PA 15213 
  * [ lti.cs.cmu.edu/ ](https://www.lti.cs.cmu.edu/)