# CS 185/285

https://rail.eecs.berkeley.edu/deeprlcourse/

# [CS 185/285](https://rail.eecs.berkeley.edu/deeprlcourse/)
  * [Calendar](https://rail.eecs.berkeley.edu/deeprlcourse/calendar)
  * [Resources](https://rail.eecs.berkeley.edu/deeprlcourse/resources)
  * [Syllabus](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus)
  * [Staff](https://rail.eecs.berkeley.edu/deeprlcourse/staff)
  * Menu 


  * [Calendar](https://rail.eecs.berkeley.edu/deeprlcourse/calendar)
  * [Resources](https://rail.eecs.berkeley.edu/deeprlcourse/resources)
  * Menu 


  * [Resources](https://rail.eecs.berkeley.edu/deeprlcourse/resources)
  * Menu 


### [Calendar](https://rail.eecs.berkeley.edu/deeprlcourse/calendar)
### [Resources](https://rail.eecs.berkeley.edu/deeprlcourse/resources)
  * [Previous Offerings](https://rail.eecs.berkeley.edu/deeprlcourse/resources#prevoffs)
  * [Courses](https://rail.eecs.berkeley.edu/deeprlcourse/resources#courses)
  * [Relevant Textbooks](https://rail.eecs.berkeley.edu/deeprlcourse/resources#textbooks)
  * [Misc Links](https://rail.eecs.berkeley.edu/deeprlcourse/resources#misclinks)


### [Syllabus](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus)
  * [Prerequisites](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#prereqs)
  * [Technology](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#technology)
  * [Materials](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#materials)
  * [Collaboration](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#collaboration)
  * [Late Policy](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#late)
  * [Grading](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#grading)


### Meta
  * [Staff](https://rail.eecs.berkeley.edu/deeprlcourse/staff)


CS 185/285 at UC Berkeley
# Deep Reinforcement Learning
Lectures: 9 - 10 am on Wednesdays and 8 - 10 am on Fridays, both in [Hearst Annex A1](https://rtl.berkeley.edu/classroom-database/hearst-annex-a1)
  

**Announcement** : The default final project options are now available: [Offline-to-Online RL Default Final Project](https://rail.eecs.berkeley.edu/deeprlcourse/static/misc/offline_to_online_rl_default_final_project.pdf) and [LLM RL Default Final Project](https://rail.eecs.berkeley.edu/deeprlcourse/static/misc/llm_rl_default_final_project.pdf). 
**Announcement** : The [final project outline](https://rail.eecs.berkeley.edu/deeprlcourse/static/misc/final_project_outline.pdf) has been released. 
  

**Looking for deep RL course materials from past years?**
Recordings of lectures from Fall 2023 are [here](https://www.youtube.com/playlist?list=PL_iWQOsE6TfVYGEGiAOMaOzzv41Jfm_Ps), and materials from previous offerings are [here](https://rail.eecs.berkeley.edu/deeprlcourse/resources#prev-off).   
  
**Email all staff (preferred):** cs285-staff-sp2026@lists.eecs.berkeley.edu 
  * ## Instructor [Sergey Levine](https://people.eecs.berkeley.edu/~svlevine/)
svlevine@eecs.berkeley.edu
Office Hours: Wednesdays 8 - 9 AM in Hearst Annex A1
  * ## Head GSI [Seohong Park](https://seohong.me/)
seohong@berkeley.edu
Office Hours: Fri 1:15p-2:15p in Berkeley Way West 1204
  * ## GSI [Vivek Myers](https://people.eecs.berkeley.edu/~vmyers/)
vmyers@berkeley.edu
Office Hours: Tue 4p-5p in Berkeley Way West 1204
  * ## GSI [Kevin Black](https://kevin.black/)
kvablack@berkeley.edu
Office Hours: Tue 9a-10a in Berkeley Way West 1212
  * ## GSI [Pranav Atreya](https://pranavatreya.github.io/)
pranavatreya@berkeley.edu
Office Hours: Mon 5p-6p in Berkeley Way West 1216
  * ## GSI [Mitsuhiko Nakamoto](https://nakamotoo.github.io/)
nakamoto@eecs.berkeley.edu
Office Hours: Thursday 4p-5p Berkeley Way West 1204
  * ## GSI [Catherine Glossop](https://catglossop.github.io/)
catherine_glossop@berkeley.edu
Office Hours: Thu 10a-11a in Berkeley Way West 1211


### Week 1 Overview
## Course Intro & Imitation Learning
Monday, January 19 – Friday, January 23
  * [Homework 1: Imitation Learning](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw1.pdf)


  * [Lecture 1: Introduction](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-1.pdf)
  * [Lecture 2: Behavioral Cloning](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-2.pdf)


### Week 2 Overview
## Imitation Learning & RL Basics
Monday, January 26 – Friday, January 30
  * [Homework 1: Imitation Learning](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw1.pdf)


  * [Section 1: PyTorch Tutorial](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-1.pdf)
  * [Lecture 3: Behavioral Cloning Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-3.pdf)
  * [Lecture 4: RL Basics](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf)


### Week 3 Overview
## Policy Gradients & Actor Critic
Monday, February 2 – Friday, February 6
  * [Homework 1: Imitation Learning](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw1.pdf)
  * [Homework 2: Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf)


  * [Section 2 Part 1: Probability Review](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-2-1.pdf)
  * [Section 2 Part 2: BC Distributional Shift](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-2-2.pdf)
  * [Lecture 5: Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-5.pdf)
  * [Lecture 6: Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-6.pdf)


### Week 4 Overview
## Value-Based RL
Monday, February 9 – Friday, February 13
  * [Homework 1: Imitation Learning](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw1.pdf)
  * [Homework 2: Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf)


  * [Section 3: Policy Gradients and Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-3.pdf)
  * [Lecture 7: Value-Based RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-7.pdf)
  * [Lecture 8: Q-learning in Practice](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-8.pdf)


### Week 5 Overview
## Advanced Policy Gradients
Monday, February 16 – Friday, February 20
  * [Homework 2: Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf)
  * [Homework 3: Q-Learning and Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw3.pdf)


  * [Section 4: DQN and SAC](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-4.pdf)
  * [Lecture 9: Advanced Policy Gradients Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-9.pdf)
  * [Lecture 10: Advanced Policy Gradients Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-10.pdf)


### Week 6 Overview
## Variational Inference
Monday, February 23 – Friday, February 27
  * [Homework 2: Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf)
  * [Homework 3: Q-Learning and Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw3.pdf)


  * [Section 5: Advanced Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-5.pdf)
  * [Lecture 11: Variational Inference](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-11.pdf)
  * [Lecture 12: VI in RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-12.pdf)


### Week 7 Overview
## Finishing VI & LLM RL
Monday, March 2 – Friday, March 6
  * [Homework 3: Q-Learning and Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw3.pdf)
  * [Homework 4: LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw4.pdf)


  * [Section 6: Variational Inference](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-6.pdf)
  * [Lecture 13: Control as Inference](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-13.pdf)
  * [Lecture 14: LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-14.pdf)


### Week 8 Overview
## Model-Based RL
Monday, March 9 – Friday, March 13
  * [Homework 3: Q-Learning and Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw3.pdf)
  * [Homework 4: LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw4.pdf)


  * [Section 7: IRL and LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-7.pdf)
  * [Lecture 15: Model-Based RL Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-15.pdf)
  * [Lecture 16: Model-Based RL Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-16.pdf)


### Week 9 Overview
## Offline Reinforcement Learning
Monday, March 16 – Friday, March 20
  * [Homework 4: LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw4.pdf)
  * [Homework 5: Offline RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw5.pdf)


  * [Section 8: Model-Based RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-8.pdf)
  * [Lecture 17: Offline RL Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-17.pdf)
  * [Lecture 18: Offline RL Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-18.pdf)


### Week 10 Overview
## Exploration & RL Theory
Monday, March 23 – Friday, March 27
  * [Homework 4: LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw4.pdf)
  * [Homework 5: Offline RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw5.pdf)


  * [Section 9: Offline RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-9.pdf)
  * [Lecture 19: Exploration](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-19.pdf)
  * [Lecture 20: RL Theory](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-20.pdf)


### Week 11 Overview
## Midterm Review
Monday, April 6 – Friday, April 10
  * [Homework 5: Offline RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw5.pdf)


  * [Lecture 21: Midterm Review Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-21.pdf)
  * [Lecture 22: Midterm Review Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-22.pdf)


### Week 12 Overview
## Exploration & Multi-task RL
Monday, April 13 – Friday, April 17
  * [Homework 5: Offline RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw5.pdf)


  * [Lecture 23: Advanced Exploration](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-23.pdf)
  * [Lecture 24: Multi-task RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-24.pdf)


### Week 13 Overview
## Open Problems & Guest Lecture
Monday, April 20 – Friday, April 24
  * [Lecture 25: Challenges and Open Problems](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-25.pdf)


### Week 14 Overview
## Final Guest Lecture
Monday, April 27 – Friday, May 1
## Final Project Information
### Default Project Options
  * [Offline-to-Online RL Default Final Project](https://rail.eecs.berkeley.edu/deeprlcourse/static/misc/offline_to_online_rl_default_final_project.pdf)
  * [LLM RL Default Final Project](https://rail.eecs.berkeley.edu/deeprlcourse/static/misc/llm_rl_default_final_project.pdf)


### Project Outline
  * [Final Project Outline](https://rail.eecs.berkeley.edu/deeprlcourse/static/misc/final_project_outline.pdf)


## Homeworks
See [Syllabus](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#materials) for more information.
![Modal logo](https://rail.eecs.berkeley.edu/deeprlcourse/static/images/modal-logo-icon.png)We are grateful for generous compute support from [Modal](https://modal.com) for students enrolled in the course.
  * [Homework 1: Imitation Learning](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw1.pdf)
  * [Homework 2: Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw2.pdf)
  * [Homework 3: Q-Learning and Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw3.pdf)
  * [Homework 4: LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw4.pdf)
  * [Homework 5: Offline RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/homeworks/hw5.pdf)


## Lecture Slides
See [Syllabus](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#materials) for more information.
  * [Lecture 1: Introduction](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-1.pdf)
  * [Lecture 2: Behavioral Cloning](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-2.pdf)
  * [Lecture 3: Behavioral Cloning Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-3.pdf)
  * [Lecture 4: RL Basics](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-4.pdf)
  * [Lecture 5: Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-5.pdf)
  * [Lecture 6: Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-6.pdf)
  * [Lecture 7: Value-Based RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-7.pdf)
  * [Lecture 8: Q-learning in Practice](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-8.pdf)
  * [Lecture 9: Advanced Policy Gradients Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-9.pdf)
  * [Lecture 10: Advanced Policy Gradients Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-10.pdf)
  * [Lecture 11: Variational Inference](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-11.pdf)
  * [Lecture 12: VI in RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-12.pdf)
  * [Lecture 13: Control as Inference](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-13.pdf)
  * [Lecture 14: LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-14.pdf)
  * [Lecture 15: Model-Based RL Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-15.pdf)
  * [Lecture 16: Model-Based RL Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-16.pdf)
  * [Lecture 17: Offline RL Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-17.pdf)
  * [Lecture 18: Offline RL Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-18.pdf)
  * [Lecture 19: Exploration](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-19.pdf)
  * [Lecture 20: RL Theory](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-20.pdf)
  * [Lecture 21: Midterm Review Part 1](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-21.pdf)
  * [Lecture 22: Midterm Review Part 2](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-22.pdf)
  * [Lecture 23: Advanced Exploration](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-23.pdf)
  * [Lecture 24: Multi-task RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-24.pdf)
  * [Lecture 25: Challenges and Open Problems](https://rail.eecs.berkeley.edu/deeprlcourse/static/slides/lec-25.pdf)


## Discussion Section Slides
See [Syllabus](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#materials) for more information.
  * [Section 1: PyTorch Tutorial](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-1.pdf)
  * [Section 2 Part 1: Probability Review](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-2-1.pdf)
  * [Section 2 Part 2: BC Distributional Shift](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-2-2.pdf)
  * [Section 3: Policy Gradients and Actor Critic](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-3.pdf)
  * [Section 4: DQN and SAC](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-4.pdf)
  * [Section 5: Advanced Policy Gradients](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-5.pdf)
  * [Section 6: Variational Inference](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-6.pdf)
  * [Section 7: IRL and LLM RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-7.pdf)
  * [Section 8: Model-Based RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-8.pdf)
  * [Section 9: Offline RL](https://rail.eecs.berkeley.edu/deeprlcourse/static/sections/section-9.pdf)


### [Calendar](https://rail.eecs.berkeley.edu/deeprlcourse/calendar)
### [Resources](https://rail.eecs.berkeley.edu/deeprlcourse/resources)
  * [Previous Offerings](https://rail.eecs.berkeley.edu/deeprlcourse/resources#prevoffs)
  * [Courses](https://rail.eecs.berkeley.edu/deeprlcourse/resources#courses)
  * [Relevant Textbooks](https://rail.eecs.berkeley.edu/deeprlcourse/resources#textbooks)
  * [Misc Links](https://rail.eecs.berkeley.edu/deeprlcourse/resources#misclinks)


### [Syllabus](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus)
  * [Prerequisites](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#prereqs)
  * [Technology](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#technology)
  * [Materials](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#materials)
  * [Collaboration](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#collaboration)
  * [Late Policy](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#late)
  * [Grading](https://rail.eecs.berkeley.edu/deeprlcourse/syllabus#grading)


### Meta
  * [Staff](https://rail.eecs.berkeley.edu/deeprlcourse/staff)