# CS224U: Natural Language Understanding - Spring 2023

https://cs224u.stanford.edu/

# [CS224U: Natural Language Understanding](http://web.stanford.edu/class/cs224u/index.html)
  * [Home](http://web.stanford.edu/class/cs224u/index.html)
  * [Projects](http://web.stanford.edu/class/cs224u/projects.html)
  * [Policies and requirements](http://web.stanford.edu/class/cs224u/requirements.html)
  * [Background materials](http://web.stanford.edu/class/cs224u/background.html)
  * [Podcast](http://web.stanford.edu/class/cs224u/podcast/index.html)


* * *
### [Course info](http://web.stanford.edu/class/cs224u/#info)
**CS224u can be taken entirely online and asynchronously. Our class meetings will be recorded, and the core content will also be delivered via slides, videos, and Python notebooks.**
MW 3:00–4:20 pm, [Gates B1](http://campus-map.stanford.edu/?srch=Gates+B1)
[Discussion](https://edstem.org/us/courses/38101/discussion/)
[Canvas](https://canvas.stanford.edu/courses/170806)
[Gradescope](https://www.gradescope.com/courses/524125)
[ Github](https://github.com/cgpotts/cs224u/)
Staff email:  
cs224u-spr2223-staff@lists.stanford.edu
Cloud credits and computing support for student projects generously provided by [AWS Educate](https://aws.amazon.com/education/awseducate/)
* * *
### [Teaching team](http://web.stanford.edu/class/cs224u/)
[![Chris](http://web.stanford.edu/class/cs224u/images/ChrisPotts.jpg)](http://web.stanford.edu/~cgpotts/)
#### [Christopher Potts](http://web.stanford.edu/~cgpotts/)
[![Kawin](https://kawine.github.io/assets/new_profile.jpeg)](https://kawine.github.io/)
#### [Kawin Ethayarajh](https://kawine.github.io/)
[![Sidd](https://www.siddkaramcheti.com/assets/img/sidd.jpg)](https://www.siddkaramcheti.com/)
#### [Sidd Karamcheti](https://www.siddkaramcheti.com/)
[![Mina](https://brushbrushbrushyourteeth.files.wordpress.com/2022/07/md_25-2.jpg)](https://brushbrushbrushyourteeth.files.wordpress.com/)
#### [Mina Lee](https://brushbrushbrushyourteeth.files.wordpress.com/)
[![Siyan](https://siyan-sylvia-li.com/img/avatar.jpg)](https://siyan-sylvia-li.com/)
#### [Siyan Li](https://siyan-sylvia-li.com/)
[![Lisa](https://xiangli1999.github.io/img/lisa.jpg)](https://xiangli1999.github.io/)
#### [Xiang (Lisa) Li](https://xiangli1999.github.io/)
[![Tolúlope](https://tolulope.carrd.co/assets/images/image01.jpg)](https://tolulope.carrd.co/)
#### [Tolúlope Ògúnremí](https://tolulope.carrd.co/)
[![Tianyi](https://tiiiger.github.io/images/headshot_new.jpeg)](https://tiiiger.github.io/)
#### [Tianyi Zhang](https://tiiiger.github.io/)
* * *
### [Previous years](http://web.stanford.edu/class/cs224u/#previous)
  * [2022](http://web.stanford.edu/class/cs224u/2022/)
  * [2021](http://web.stanford.edu/class/cs224u/2021/)
  * [2020](http://web.stanford.edu/class/cs224u/2020/)
  * [2019](http://web.stanford.edu/class/cs224u/2019/)
  * [2018](http://web.stanford.edu/class/cs224u/2018/)
  * [2016](http://web.stanford.edu/class/cs224u/2016/)
  * [2015](http://web.stanford.edu/class/cs224u/2015/)
  * [2014](http://web.stanford.edu/class/cs224u/2014/)
  * [2013](http://web.stanford.edu/class/cs224u/2013/)
  * [2012](http://web.stanford.edu/class/cs224u/2012/)


## [Schedule](http://web.stanford.edu/class/cs224u/)  
|   | Materials  | Resources  | Assignments  |  
| --- | --- | --- | --- |  
| Apr 3  | 
  1. [Course introduction and overview](http://web.stanford.edu/class/cs224u/slides/cs224u-intro-2023-handout.pdf)
  2. Notebook: [Course set-up](https://github.com/cgpotts/cs224u/blob/main/setup.ipynb)
  3. [Optional background materials](http://web.stanford.edu/class/cs224u/background.html)

 | 
  1. [Levesque 2013](http://www.cs.toronto.edu/~hector/Papers/ijcai-13-paper.pdf)
  2. [Manning 2015](https://www.mitpressjournals.org/doi/pdf/10.1162/COLI_a_00239)
  3. [Philosophy of understanding](https://crfm.stanford.edu/assets/report.pdf#philosophy) (§2.6 of the "Foundation Models" report)
  4. [Artificial Intelligence: Last Week Tonight with John Oliver](https://youtu.be/Sqa8Zo2XWc4)

 |   |  
|   | Domain adaptation for supervised sentiment  |  
| Apr 5  | 
  1. Overview of Assign/bakeoff 1
  2. [Contextual word representations](http://web.stanford.edu/class/cs224u/slides/cs224u-contextualreps-2023-handout.pdf)
  3. [Diffusion objectives for text](http://web.stanford.edu/class/cs224u/slides/lisa-224u-diffusion.pdf) [Lisa]
  4. [Fantastic language models and how to build them](http://web.stanford.edu/class/cs224u/slides/sidd-fantastic-lms-cs224u.pdf) [Sidd]

 | 
  1. SST: [Socher et al. 2013](https://aclanthology.org/D13-1170/)
  2. DynaSent: [Potts, Wu et al. 2020](https://aclanthology.org/2021.acl-long.186/)
  3. [Smith 2020](https://arxiv.org/abs/1902.06006)
  4. The Pile: [Gao et al. 2020](https://arxiv.org/abs/2101.00027)
  5. Transformer: [Vaswani et al. 2017](https://papers.nips.cc/paper/7181-attention-is-all-you-need)
  6. [The Annotated Transformer](http://nlp.seas.harvard.edu/annotated-transformer/)
  7. Relative positional encoding: [Shaw et al. 2018](https://arxiv.org/abs/1803.02155)
  8. GPT: [Radford et al. 2018](https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf)
  9. BERT: [Devlin et al. 2018](https://www.aclweb.org/anthology/N19-1423/)
  10. RoBERTa: [Liu et al. 2019](https://arxiv.org/abs/1907.11692)
  11. ELECTRA: [Clark et al. 2019](https://openreview.net/pdf?id=r1xMH1BtvB)
  12. T5: [Raffel et al. 2019](https://arxiv.org/abs/1910.10683)
  13. [BART: Lewis et al. 2020](https://aclanthology.org/2020.acl-main.703/)
  14. DistilBERT: [Sanh et al. 2019](https://arxiv.org/abs/1910.01108)
  15. Diffusion-LM: [Li et al. 2022](https://proceedings.neurips.cc/paper_files/paper/2022/hash/1be5bc25d50895ee656b8c2d9eb89d6a-Abstract-Conference.html)

 | 
  1. [Assign/bakeoff 1](https://github.com/cgpotts/cs224u/blob/main/hw_sentiment.ipynb): due Apr 17, 3:00 pm Pacific
  2. [Quiz 0](https://canvas.stanford.edu/courses/170806/quizzes/133191): due Apr 17, 3:00 pm Pacific
  3. [Quiz 1](https://canvas.stanford.edu/courses/170806/quizzes/133506): due Apr 17, 3:00 pm Pacific

 |  
| Apr 10  |  
| Apr 12  |  
|   | Retrieval augmented in-context learning  |  
| Apr 17  | 
  1. [Overview of Assign/bakeoff 2](http://web.stanford.edu/class/cs224u/slides/cs224u-hw2-overview-2023.pdf)
  2. [Information retrieval](http://web.stanford.edu/class/cs224u/slides/cs224u-neuralir-2023-handout.pdf)
  3. [In-context learning](http://web.stanford.edu/class/cs224u/slides/cs224u-incontextlearning-2023-handout.pdf)
  4. [Prompters before prompts and promptees](https://drive.google.com/file/d/1RIOAOTOOPyVLezFiIfGnYJSE8ofKuR4L/view) [Mina]

 | 
  1. [Khattab et al. 2021](https://ai.stanford.edu/blog/retrieval-based-NLP/)
  2. ColBERT: [Khattab and Zaharia 2020](https://arxiv.org/abs/2004.12832)
  3. DPR: [Karpukhin et al. 2020](https://aclanthology.org/2020.emnlp-main.550/)
  4. SPLADE: [Formal et al. 2021](https://dl.acm.org/doi/10.1145/3404835.3463098)
  5. RAG: [Lewis et al. 2020](https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html)
  6. GPT-3: [Brown et al. 2020](https://arxiv.org/abs/2005.14165)
  7. RLHF for LLMs: [Ouyang et al. 2022](https://arxiv.org/abs/2203.02155)
  8. CoT: [Wei et al. 2022](https://arxiv.org/abs/2201.11903)
  9. Retrive-then-read: [Lazaridou et al. 2022](https://arxiv.org/abs/2203.05115)
  10. DSP: [Khattab et al. 2022](https://arxiv.org/abs/2212.14024)

 | 
  1. [Assign/bakeoff 2](https://github.com/cgpotts/cs224u/blob/main/hw_openqa.ipynb): due Apr 26, 3:00 pm Pacific
  2. [Quiz 2](https://canvas.stanford.edu/courses/170806/quizzes/134114): due Apr 26, 3:00 pm Pacific

 |  
| Apr 19  |  
| Apr 24  |  
|   | Advanced behavioral evaluation  |  
| Apr 26  | 
  1. [Overview of Assign/bakeoff 3](http://web.stanford.edu/class/cs224u/slides/cs224u-hw3-overview-2023.pdf)
  2. [Advanced behavioral evaluation of NLU models](http://web.stanford.edu/class/cs224u/slides/cs224u-behavioraleval-2023-handout.pdf)

 | 
  1. [Jia and Liang 2017](http://aclweb.org/anthology/D17-1215)
  2. [Glockner et al. 2018](https://www.aclweb.org/anthology/P18-2103)
  3. [Liu et al. 2019](https://www.aclweb.org/anthology/N19-1225)
  4. [Naik et al. 2019](https://www.aclweb.org/anthology/C18-1198)
  5. ANLI: [Nie et al. 2020](https://aclanthology.org/2020.acl-main.441/)
  6. Dynabench: [Kiela et al. 2021](https://aclanthology.org/2021.naacl-main.324/)
  7. COGS: [Kim and Linzen 2020](https://aclanthology.org/2020.emnlp-main.731/)
  8. ReCOGS: [Wu et al. 2023](https://arxiv.org/abs/2303.13716)

 | 
  1. [Assign/bakeoff 3](https://github.com/cgpotts/cs224u/blob/main/hw_recogs.ipynb) due May 8, 3:00 pm Pacific
  2. [Quiz 3](https://canvas.stanford.edu/courses/170806/quizzes/134298): due May 8, 3:00 pm Pacific

 |  
| May 1  |  
| May 3  |  
|   | Analysis methods  |  
| May 8  | 
  1. [Lit review overview](http://web.stanford.edu/class/cs224u/slides/cs224u-litreview-overview-2023.pdf)
  2. [Siyan: What I did for 224u final project](http://web.stanford.edu/class/cs224u/slides/siyan-projects-cs224u.pdf)
  3. [Course review in the form of a Jeopardy! game](https://jeopardylabs.com/play/nlu-jeopardy-4)
  4. [Analysis methods for NLU](http://web.stanford.edu/class/cs224u/slides/cs224u-analysis-2023-handout.pdf)

 | 
  1. LIME: [Ribeiro et al. 2016](https://dl.acm.org/doi/10.1145/2939672.2939778)
  2. Probing: [Tenney et al. 2018](https://aclanthology.org/P19-1452/)
  3. IG: [Sundararajan et al. 2017](http://proceedings.mlr.press/v70/sundararajan17a.html)
  4. Causal abstraction: [Geiger et al. 2022](https://ai.stanford.edu/blog/causal-abstraction/)
  5. IIT: [Geiger, Wu, et al. 2022](https://proceedings.mlr.press/v162/geiger22a.html)
  6. DAS: [Geiger, Wu, et al. 2023](https://arxiv.org/abs/2303.02536)
  7. Circuits: [Cammarata et al. 2020](https://distill.pub/2020/circuits/)

 | 
  1. [Lit review](http://web.stanford.edu/class/cs224u/projects.html#litreview): due May 17, 3:00 pm Pacific
  2. [Quiz 4](https://canvas.stanford.edu/courses/170806/quizzes/134582): due May 29, 3:00 pm Pacific

 |  
| May 10  |  
| May 15  |  
|   | NLP methods  |  
| May 17  | 
  1. [Experiment protocol overview](http://web.stanford.edu/class/cs224u/slides/cs224u-protocol-overview-2023.pdf)
  2. [NLP methods and metrics](http://web.stanford.edu/class/cs224u/slides/cs224u-methods-2023-handout.pdf)
  3. Real-world NLP assessments [Kawin]

 | 
  1. [Resnik and Lin 2010](http://www.cs.colorado.edu/~jbg/teaching/CMSC_773_2012/reading/evaluation.pdf)
  2. [Smith 2011, Appendix B](http://www.cs.cmu.edu/~nasmith/LSP/)
  3. Dynascores: [Ma et al. 2021](https://papers.nips.cc/paper/2021/hash/55b1927fdafef39c48e5b73b5d61ea60-Abstract.html)
  4. [Santhanam et al. 2022](https://arxiv.org/abs/2212.01340)

 | 
  1. [Experimental protocol](http://web.stanford.edu/class/cs224u/projects.html#protocol): due May 29, 3:00 pm Pacific

 |  
| May 22  |  
| May 24  |  
|   | Your projects  |  
| May 29  | 
  1. Memorial Day (no class)

 |  
| May 31  | 
  1. [Presenting your research](http://web.stanford.edu/class/cs224u/slides/cs224u-presenting-2023-handout.pdf)

 | 
  1. [Jason Eisner's Advice for Research Students](https://www.cs.jhu.edu/~jason/advice/)
  2. [Stuart Shieber on reporting research results](http://web.stanford.edu/class/cs224u/readings/shieber-writing.pdf)
  3. [David Goss on math style](http://www.math.osu.edu/~goss.3/hint.pdf)
  4. [Novelist Cormac McCarthy’s tips on how to write a great science paper](https://www.nature.com/articles/d41586-019-02918-5)
  5. [Geoff Pullum's Five Golden Rules (well, actually six) for giving academic presentations](http://www.lel.ed.ac.uk/~gpullum/goldenrules.html)
  6. [Patrick Blackburn: How to give a good talk](http://web.stanford.edu/class/cs224u/readings/blackburn2001.pdf)
  7. Datasheets: [Gebru et al 2018](https://arxiv.org/abs/1803.09010)
  8. Model Cards: [Mitchell et al. 2019](https://doi.org/10.1145/3287560.3287596)

 | 
  1. [Final paper](http://web.stanford.edu/class/cs224u/projects.html#paper): due June 10, 11:30 am Pacific (end of our scheduled exam time, which we will not use)

 |  
| Jun 5  |  
| Jun 7  |  
* * *
  * Maintained by [Christopher Potts](http://web.stanford.edu/~cgpotts/). ©Stanford University. [Terms of Use](http://stanford.edu/site/terms.html) | [Copyright Complaints](http://stanford.edu/site/copyright.html).
  * Design: [HTML5 UP](http://html5up.net)