# Readings | Deep Learning | Electrical Engineering and Computer Science | MIT OpenCourseWare

https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/readings/

###  Browse Course Material  ![](https://ocw.mit.edu/static_shared/images/close_small.b0a2684ccc0da6407a56.svg)
  * [ Syllabus ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/syllabus/)
* * *
  * [ Readings ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/readings/)
* * *
  * [ Lecture Notes ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/lists/lecture-notes/)
* * *
  * [ Lecture Videos ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/video_galleries/lecture-videos/)
* * *
  * [ Homework ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/lists/homework/)
* * *
  * [ Final Project ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/final-project/)
* * *
    * [ Final Project Ideas ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/project-ideas/)
    * [ Final Project Grading Rubric ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/final-project-grading-rubric/)


![](https://ocw.mit.edu/static_shared/images/close_small.b0a2684ccc0da6407a56.svg)
##  Course Info 
###  Instructors 
  * [Prof. Phillip Isola](https://ocw.mit.edu/search/?q=Prof.+Phillip+Isola)
  * [Prof. Sara Beery](https://ocw.mit.edu/search/?q=Prof.+Sara+Beery)
  * [Dr. Jeremy Bernstein](https://ocw.mit.edu/search/?q=Dr.+Jeremy+Bernstein)


###  Departments 
  * [Electrical Engineering and Computer Science](https://ocw.mit.edu/search/?d=Electrical+Engineering+and+Computer+Science)


###  As Taught In 
Fall 2024 
###  Level 
[Undergraduate](https://ocw.mit.edu/search/?l=Undergraduate)  
[Graduate](https://ocw.mit.edu/search/?l=Graduate)  

###  Topics 
  * [Engineering](https://ocw.mit.edu/search/?t=Engineering)
    * [Computer Science](https://ocw.mit.edu/search/?t=Computer+Science)
      * [Artificial Intelligence](https://ocw.mit.edu/search/?t=Artificial+Intelligence)
      * [Graphics and Visualization](https://ocw.mit.edu/search/?t=Graphics+and+Visualization)


###  Learning Resource Types 
_notes_ Lecture Notes
_theaters_ Lecture Videos
_assignment_ Problem Sets
_grading_ Projects with Examples
_auto_stories_ Readings
* * *
[ Download Course ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/download)
_menu_
[ ![MIT OpenCourseWare](https://ocw.mit.edu/static_shared/images/ocw_logo_white.cabdc9a745b03db3dad4.svg) ](https://ocw.mit.edu/)
  * [ _search_ ](https://ocw.mit.edu/search/)
  * [Give Now](https://giving.mit.edu/give/to/ocw/?utm_source=ocw&utm_medium=homepage_banner&utm_campaign=nextgen_home)
  * [About OCW](https://ocw.mit.edu/about)
  * [Help & Faqs](https://mitocw.zendesk.com/hc/en-us)
  * [Contact Us](https://mitocw.zendesk.com/hc/en-us/requests/new)


[ ![MIT OpenCourseWare](https://ocw.mit.edu/static_shared/images/ocw_logo_white.cabdc9a745b03db3dad4.svg) ](https://ocw.mit.edu/)
[ _search_ ](https://ocw.mit.edu/search/) [ GIVE NOW ![](https://ocw.mit.edu/static_shared/images/heart_burgundy.e7c8635e8cc6538b89fa.svg) ](https://giving.mit.edu/give/to/ocw/?utm_source=ocw&utm_medium=homepage_banner&utm_campaign=nextgen_home) [about ocw](https://ocw.mit.edu/about) [help & faqs](https://mitocw.zendesk.com/hc/en-us) [contact us](https://mitocw.zendesk.com/hc/en-us/requests/new)
6.7960 | Fall 2024 | Undergraduate, Graduate   
  

#  [Deep Learning](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/)
![](https://ocw.mit.edu/static_shared/images/expand.a062b1c47b121e3c7e03.svg) Menu
More Info 
  * [ Syllabus ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/syllabus/)
* * *
  * [ Readings ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/readings/)
* * *
  * [ Lecture Notes ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/lists/lecture-notes/)
* * *
  * [ Lecture Videos ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/video_galleries/lecture-videos/)
* * *
  * [ Homework ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/lists/homework/)
* * *
  * [ Final Project ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/final-project/)
* * *
    * [ Final Project Ideas ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/project-ideas/)
    * [ Final Project Grading Rubric ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/final-project-grading-rubric/)


## Readings
Readings labeled [Vision] are from [_Foundations of Computer Vision_](https://visionbook.mit.edu/) by Antonio Torralba, Phillip Isola, and William T. Freeman. (MIT Press, 2024. ISBN: 9780262048972.) The book is available [online](https://visionbook.mit.edu/) under a CC BY-NC-ND license.
### Session 1: Introduction to Deep Learning
**Required readings:**
  * [Notation for this course](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/resources/mit6_7950_f24_notation_pdf/)
  * [Vision] [Chapter 12: Neural Networks.](https://visionbook.mit.edu/neural_nets.html)


**Optional readings:**
  * [Vision] [Chapter 13: Neural Networks as Distribution Transformers](https://visionbook.mit.edu/neural_nets_as_distribution_transformers.html)


### Session 2: How to Train a Neural Net
**Required readings:**
  * [Vision] [Chapter 10: Gradient-Based Learning Algorithms](https://visionbook.mit.edu/gradient_descent.html)
  * [Vision] [Chapter 14: Backpropagation](https://visionbook.mit.edu/backpropagation.html)


### Session 3: Approximation Theory
No required readings.
**Optional readings:**
  * [Deep learning theory lecture notes](https://mjt.cs.illinois.edu/dlt/) sections 2 and 5


### Session 4: Architectures: Grids
**Required readings:**
  * [Vision] [Chapter 24: Convolutional Neural Nets](https://visionbook.mit.edu/convolutional_neural_nets.html)


### Session 5: Architectures: Graphs
**Required readings:**
  * Hamilton, William. [_Graph Representation Learning_](https://www.cs.mcgill.ca/~wlh/grl_book/), chapter 5 (mainly focus on the content in section 5.1) 


**Optional readings:**
  * Xu, Keyulu, Weihua Hu, et al. “[How Powerful Are Graph Neural Networks?](https://arxiv.org/abs/1810.00826)” arXiv preprint arXiv:1810.00826 (2018).
  * Sanchez-Lengeling, Benjamin, Emily Reif, et al. “[A Gentle Introduction to Graph Neural Networks](https://distill.pub/2021/gnn-intro/).” Distill, 2021.


### Session 6: Generalization Theory
No required readings.
**Optional readings:**
  * Zhang, Chiyuan, Samy Bengio, et al. “[Understanding Deep Learning Requires Rethinking Generalization](https://arxiv.org/abs/1611.03530).” _arXiv preprint arXiv:1611.03530_ (2016).
  * Belkin, Mikhail, Daniel Hsu, et al. “[Reconciling Modern Machine-Learning Practice and the Classical Bias–Variance Trade-Off](https://www.pnas.org/doi/full/10.1073/pnas.1903070116).” _Proceedings of the National Academy of Sciences_ 116, no. 32 (2019): 15849–15854.


### Session 7: Scaling Rules for Optimization
**Required readings:**
  * [13.7 General Steepest Descent](https://kenndanielso.github.io/mlrefined/blog_posts/13_Multilayer_perceptrons/13_7_General_steepest_descent.html)


### Session 8: Architectures: Transformers
**Required readings:**
  * [Vision] [Chapter 26: Transformers](https://visionbook.mit.edu/transformers.html) (Note that this reading focuses on examples from vision, but you can apply the same architecture to any kind of data.)


### Session 9: Hacker’s Guide to Deep Learning
**Optional readings:**
  * [A Recipe for Training Neural Networks](https://karpathy.github.io/2019/04/25/recipe/)
  * [Rules of Machine Learning: Best Practices for ML Engineering (PDF)](https://martin.zinkevich.org/rules_of_ml/rules_of_ml.pdf)


### Session 10: Architectures: Memory
**Required readings:**
  * [Vision] [Chapter 25: Recurrent Neural Networks](https://visionbook.mit.edu/recurrent_neural_nets.html)


**Optional readings:**
  * [Deep Learning Recurrent Networks: Stability Analysis and LSTMs (PDF)](https://deeplearning.cs.cmu.edu/S22/document/slides/lec14.recurrent.pdf)


### Session 11: Representation Learning: Reconstruction-Based
**Required readings:**
  * [Vision] [Chapter 30: Representation Learning](https://visionbook.mit.edu/representation_learning.html)


**Optional readings:**
  * Bengio, Yoshua, Aaron Courville, and Pascal Vincent. “[Representation Learning: A Review and New Perspectives](https://arxiv.org/abs/1206.5538).” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 35, no. 8 (2013): 1798–1828.


### Session 12: Representation Learning: Similarity-Based
**Required readings:**
  * Continue with session 11 readings


**Optional readings:**
  * Wang, Tongzhou, and Phillip Isola. “[Understanding Contrastive Representation Learning Through Alignment and Uniformity on the Hypersphere](https://arxiv.org/abs/2005.10242).” In _International Conference on Machine Learning_ , pp. 9929–9939. PMLR, 2020.


### Session 13: Representation Learning: Theory
No required readings.
**Optional readings:**
  * Cho, Youngmin, and Lawrence Saul. “[Kernel Methods for Deep Learning](https://papers.nips.cc/paper_files/paper/2009/hash/5751ec3e9a4feab575962e78e006250d-Abstract.html).” _Advances in Neural Information Processing Systems_ 22 (2009).
  * Lee, Jaehoon, Yasaman Bahri, et al. “[Deep Neural Networks as Gaussian Processes](https://arxiv.org/abs/1711.00165).” _arXiv preprint arXiv:1711.00165_ (2017).


### Session 14: Generative Models: Basics
**Required readings:**
  * [Vision] [Chapter 32: Generative Models](https://visionbook.mit.edu/generative_models.html)


### Session 15: Generative Models: Representation Learning Meets Generative Modeling
**Required readings:**
  * [Vision] [Chapter 33: Generative Modeling Meets Representation Learning](https://visionbook.mit.edu/generative_modeling_and_rep_learning.html)


**Optional readings:**
  * Kingma, Diederik P., and Max Welling. “[Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114).” _arXiv preprint arXiv:1312.6114_ (2013).


### Session 16: Generative Models: Conditional Models
No required readings.
**Optional readings:**
  * [Vision] [Chapter 34: Conditional Generative Models](https://visionbook.mit.edu/conditional_generative_models.html)


### Session 17: Generalization: Out-of-Distribution (OOD)
**Required readings:**
  * Mądry, Aleksander, and Ludwig Schmidt. “[A Brief Introduction to Adversarial Examples](https://gradientscience.org/intro_adversarial/).” gradient science, July 6, 2018. 
  * Mądry, Aleksander, Ludwig Schmidt, and Dimitris Tsipras. “[Training Robust Classifiers (Part 1)](https://gradientscience.org/robust_opt_pt1/).” gradient science, July 11, 2018.


**Optional readings:**
  * Koh, Pang Wei, Shiori Sagawa, et al. “[Wilds: A Benchmark of In-the-Wild Distribution Shifts](https://arxiv.org/abs/2012.07421).” In _International Conference on Machine Learning_ , pp. 5637–5664. PMLR, 2021.
  * Geirhos, Robert, Jörn-Henrik Jacobsen, et al. “[Shortcut Learning in Deep Neural Networks](https://arxiv.org/abs/2004.07780).” _Nature Machine Intelligence_ 2, no. 11 (2020): 665–673.
  * Tsipras, Dimitris, Shibani Santurkar, et al. “[From Imagenet to Image Classification: Contextualizing Progress on Benchmarks](https://arxiv.org/abs/2005.11295).” In _International Conference on Machine Learning_ , pp. 9625–9635. PMLR, 2020.
  * Xiao, Kai, Logan Engstrom, et al. “[Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://arxiv.org/abs/2006.09994).” _arXiv preprint arXiv:2006.09994_ (2020).
  * Xu, Keyulu, Mozhi Zhang, et al. “[How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://arxiv.org/abs/2009.11848).” _arXiv preprint arXiv:2009.11848_ (2020).


### Session 18: Transfer Learning: Models
**Required readings:**
  * [Vision] [Chapter 37: Transfer Learning and Adaptation](https://visionbook.mit.edu/transfer_learning.html)


**Optional readings:**
  * Farahani, Abolfazl, Sahar Voghoei, et al. “[A Brief Review of Domain Adaptation](https://arxiv.org/abs/2010.03978).” _Advances in Data Science and Information Engineering: Proceedings from ICDATA 2020 and IKE 2020_ (2021): 877–894.
  * Kay, Justin, Timm Haucke, et al. “[Align and Distill: Unifying and Improving Domain Adaptive Object Detection](https://arxiv.org/abs/2403.12029).” _arXiv preprint arXiv:2403.12029_ (2024).


### Session 19: Transfer Learning: Data
**Required readings:**
  * Continue with session 19 readings.


### Session 20: Scaling Laws
**Required readings:**
  * Kaplan, Jared, Sam McCandlish, et al. “[Scaling Laws for Neural Language Models](https://arxiv.org/abs/2001.08361).” _arXiv preprint arXiv:2001.08361_ (2020).


**Optional readings:**
  * Hoffmann, Jordan, Sebastian Borgeaud, et al. “[Training Compute-Optimal Large Language Models](https://arxiv.org/abs/2203.15556).” _arXiv preprint arXiv:2203.15556_ (2022).
  * Sharma, Utkarsh, and Jared Kaplan. “[A Neural Scaling Law from the Dimension of the Data Manifold](https://arxiv.org/abs/2004.10802).” _arXiv preprint arXiv:2004.10802_ (2020).
  * Sorscher, Ben, Robert Geirhos, et al. “[Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning](https://arxiv.org/abs/2206.14486).” _Advances in Neural Information Processing Systems_ 35 (2022): 19523–19536.
  * McCandlish, Sam, Jared Kaplan, et al. “[An Empirical Model of Large-Batch Training](https://arxiv.org/abs/1812.06162).” _arXiv preprint arXiv:1812.06162_ (2018).


### Session 21: Large Language Models
No required readings.
**Optional readings:**
  * Kojima, Takeshi, Shixiang Shane Gu, et al. “[Large Language Models Are Zero-Shot Reasoners](https://arxiv.org/abs/2205.11916).” _Advances in Neural Information Processing Systems_ 35 (2022): 22199–22213.


### Session 22: AI for Musical Creativity
No required readings.
### Session 23: Metrized Deep Learning
No required readings.
**Optional readings:**
  * Bernstein, Jeremy, and Laker Newhouse. “[Modular Duality in Deep Learning](https://arxiv.org/abs/2410.21265).” _arXiv preprint arXiv:2410.21265_ (2024).
  * Flynn, Thomas. “[The Duality Structure Gradient Descent Algorithm: Analysis and Applications to Neural Networks](https://arxiv.org/abs/1708.00523).” _arXiv preprint arXiv:1708.00523_ (2017).
  * Large, Tim, Yang Liu, et al. “[Scalable Optimization in the Modular Norm](https://arxiv.org/abs/2405.14813).” _Advances in Neural Information Processing Systems_ 37 (2024): 73501–73548.


### Session 24: Inference Methods for Deep Learning
No required readings.
**Optional readings:**
  * Sun, Yu, Xiaolong Wang, et al. “[Test-Time Training with Self-Supervision for Generalization Under Distribution Shifts](https://arxiv.org/abs/1909.13231).” In _International Conference on Machine Learning_ , pp. 9229–9248. PMLR, 2020.
  * Zelikman, Eric, Yuhuai Wu, et al. “[Star: Bootstrapping Reasoning with Reasoning](https://arxiv.org/abs/2203.14465).” _Advances in Neural Information Processing Systems_ 35 (2022): 15476–15488.


### Session 25: Efficient Policy Optimization Techniques for LLMs
No required readings.
![](https://ocw.mit.edu/static_shared/images/left_arrow.3c482a1f6223bff3f0e6.svg) ![](https://ocw.mit.edu/static_shared/images/close_small.b0a2684ccc0da6407a56.svg)
##  Course Info 
###  Instructors 
  * [Prof. Phillip Isola](https://ocw.mit.edu/search/?q=Prof.+Phillip+Isola)
  * [Prof. Sara Beery](https://ocw.mit.edu/search/?q=Prof.+Sara+Beery)
  * [Dr. Jeremy Bernstein](https://ocw.mit.edu/search/?q=Dr.+Jeremy+Bernstein)


###  Departments 
  * [Electrical Engineering and Computer Science](https://ocw.mit.edu/search/?d=Electrical+Engineering+and+Computer+Science)


###  As Taught In 
Fall 2024 
###  Level 
[Undergraduate](https://ocw.mit.edu/search/?l=Undergraduate)  
[Graduate](https://ocw.mit.edu/search/?l=Graduate)  

###  Topics 
  * [Engineering](https://ocw.mit.edu/search/?t=Engineering)
    * [Computer Science](https://ocw.mit.edu/search/?t=Computer+Science)
      * [Artificial Intelligence](https://ocw.mit.edu/search/?t=Artificial+Intelligence)
      * [Graphics and Visualization](https://ocw.mit.edu/search/?t=Graphics+and+Visualization)


###  Learning Resource Types 
_notes_ Lecture Notes
_theaters_ Lecture Videos
_assignment_ Problem Sets
_grading_ Projects with Examples
_auto_stories_ Readings
* * *
[ Download Course ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/download)
[ ![MIT Open Learning](https://ocw.mit.edu/static_shared/images/mit_ol.4165342f87abb1da46fd.svg) ](https://openlearning.mit.edu/)
Over 2,500 courses & materials 
Freely sharing knowledge with learners and educators around the world. [Learn more](https://ocw.mit.edu/about)
[ ![facebook](https://ocw.mit.edu/static_shared/images/Facebook.d1f5caeb73d7d12505a2.png) ](https://www.facebook.com/MITOCW)
[ ![instagram](https://ocw.mit.edu/static_shared/images/Instagram.4df41828ffee4ff33d8a.png) ](https://www.instagram.com/mitocw)
[ ![x \(formerly twitter\)](https://ocw.mit.edu/static_shared/images/x-formerly-twitter-black.f8c75ad9f42902726d25.png) ](https://twitter.com/MITOCW)
[ ![youtube](https://ocw.mit.edu/static_shared/images/Youtube.7c9f62c4f1dc9515ebb4.png) ](https://www.youtube.com/mitocw)
[ ![linkedin](https://ocw.mit.edu/static_shared/images/linkedin-black.2f7f8a6a3899f5d1e1d6.png) ](https://www.linkedin.com/company/mit-opencourseware/)
[ ![bluesky](https://ocw.mit.edu/static_shared/images/bluesky-black.9f4523fcefa9b6f25be7.png) ](https://bsky.app/profile/mitocw.bsky.social)
[ ![mastodon](https://ocw.mit.edu/static_shared/images/mastodon-black.9e2de31a28415c123800.png) ](https://mastodon.social/@mitocw)
© 2001–2026 Massachusetts Institute of Technology 
[Accessibility](https://accessibility.mit.edu)
[Creative Commons License](https://creativecommons.org/licenses/by-nc-sa/4.0/)
[Terms and Conditions](https://ocw.mit.edu/pages/privacy-and-terms-of-use/)
Proud member of: [ ![Open Education Global](https://ocw.mit.edu/static_shared/images/oeg_logo.8a31f7b87f30df2d0169.png) ](https://www.oeglobal.org/)
[ ![facebook](https://ocw.mit.edu/static_shared/images/Facebook.d1f5caeb73d7d12505a2.png) ](https://www.facebook.com/MITOCW)
[ ![instagram](https://ocw.mit.edu/static_shared/images/Instagram.4df41828ffee4ff33d8a.png) ](https://www.instagram.com/mitocw)
[ ![x \(formerly twitter\)](https://ocw.mit.edu/static_shared/images/x-formerly-twitter-black.f8c75ad9f42902726d25.png) ](https://twitter.com/MITOCW)
[ ![youtube](https://ocw.mit.edu/static_shared/images/Youtube.7c9f62c4f1dc9515ebb4.png) ](https://www.youtube.com/mitocw)
[ ![linkedin](https://ocw.mit.edu/static_shared/images/linkedin-black.2f7f8a6a3899f5d1e1d6.png) ](https://www.linkedin.com/company/mit-opencourseware/)
[ ![bluesky](https://ocw.mit.edu/static_shared/images/bluesky-black.9f4523fcefa9b6f25be7.png) ](https://bsky.app/profile/mitocw.bsky.social)
[ ![mastodon](https://ocw.mit.edu/static_shared/images/mastodon-black.9e2de31a28415c123800.png) ](https://mastodon.social/@mitocw)
© 2001–2026 Massachusetts Institute of Technology 
![](https://ocw.mit.edu/static_shared/images/external_link.58bbdd86c00a2c146c36.svg)
#  You are leaving MIT OpenCourseWare 
close
Please be advised that external sites may have terms and conditions, including license rights, that differ from ours. MIT OCW is not responsible for any content on third party sites, nor does a link suggest an endorsement of those sites and/or their content. 
Stay Here  [ Continue ](https://ocw.mit.edu/courses/6-7960-deep-learning-fall-2024/pages/readings/)