# 11-777 MMML

https://cmu-multicomp-lab.github.io/mmml-course/fall2026/

11-777 MMML
[Fall2026](https://multicomp.cs.cmu.edu/mmml-course/fall2026/) [Fall2023](https://multicomp.cs.cmu.edu/mmml-course/fall2023/) [Fall2022](https://multicomp.cs.cmu.edu/mmml-course/fall2022/) [Fall2020](https://multicomp.cs.cmu.edu/mmml-course/fall2020/)
[home](https://multicomp.cs.cmu.edu/mmml-course/fall2026/) [schedule](https://multicomp.cs.cmu.edu/mmml-course/fall2026/schedule/) [readings](https://multicomp.cs.cmu.edu/mmml-course/fall2026/readings/) [syllabus](https://multicomp.cs.cmu.edu/mmml-course/assets/pdf/MultimodalML-Fall2026-Syllabus.pdf) [projects](https://multicomp.cs.cmu.edu/mmml-course/fall2026/projects/)
![](https://multicomp.cs.cmu.edu/mmml-course/assets/img/cmu-logo.png)
#  **M** ulti**M** odal **M** achine **L** earning 
##  11-777 • Fall 2026 • Carnegie Mellon University 
Multimodal machine learning (MMML) is a vibrant interdisciplinary research area that advances core goals of artificial intelligence by integrating and modeling multiple communicative modalities, including language, speech, vision, and other sensory signals. Originating from early work in audio-visual speech recognition and evolving through modern language-and-vision systems such as image and video captioning, MMML addresses unique challenges arising from heterogeneous data sources and complex interdependencies between modalities. 
This course introduces the fundamental mathematical and algorithmic concepts in machine learning and deep learning that underpin the six central challenges of multimodal machine learning: 
  1. Representation
  2. Alignment
  3. Reasoning
  4. Generation
  5. Transference
  6. Quantification


Students will study both foundational and modern approaches to these challenges, including multimodal autoencoders, deep canonical correlation analysis, multimodal tensor fusion, multi-kernel learning, attention mechanisms, multimodal recurrent models, multimodal transformers, neuro-symbolic methods, mutual information–based objectives, and multimodal graph neural networks. 
The course also examines state-of-the-art probabilistic models and computational algorithms through readings of recent research literature, with attention to open problems and emerging directions in the field. Applications discussed throughout the course include multimodal affect recognition, language grounding, language-vision navigation, image and video captioning, and cross-modal multimedia retrieval. 
* * *
  * **Time:** Tuesday/Thursday 9:30 AM-10:50 AM 
  * **Location:** [ POS :: A35 ](https://www.google.com/maps?q=40.441166,-79.942242)
  * **Discussion:** [ Piazza ](https://piazza.com/class/mss0w3c7g6q10l)
  * **HW submission:** [ Canvas ](https://canvas.cmu.edu/courses/55519)
  * **Online lectures:** The lectures will be live-streamed through [ Panopto ](https://canvas.cmu.edu/courses/55519/external_tools/269), recorded, and made available on [ YouTube ](https://www.youtube.com/playlist?list=PLoZgVqqHOumTY2CAQHL45tQp6kmDnDcqn). 
  * **Contact:** Students should ask all course-related questions on [ Piazza ](https://piazza.com/class/mss0w3c7g6q10l), where you will also find announcements. For external enquiries, personal matters, or emergencies, create a private post in [ Piazza ](https://piazza.com/class/mss0w3c7g6q10l). 


* * *
[ ![](https://multicomp.cs.cmu.edu/mmml-course/assets/img/lp.jpg) ](https://www.cs.cmu.edu/~morency/)
  * Instructor [ Louis-Philippe Morency ](https://www.cs.cmu.edu/~morency/)
  * Email: morency@cs.cmu.edu 


![Kashish Gandhi](https://multicomp.cs.cmu.edu/mmml-course/assets/img/kgandhi.jpg)
  * TA Kashish Gandhi
  * Email: kgandhi2@andrew.cmu.edu


![Manu Gaur](https://multicomp.cs.cmu.edu/mmml-course/assets/img/manu.jpg)
  * TA Manu Gaur
  * Email: mgaur@andrew.cmu.edu


![Shruti Jain](https://multicomp.cs.cmu.edu/mmml-course/assets/img/shruti.jpg)
  * TA Shruti Jain
  * Email: spjain@andrew.cmu.edu


![Yousha Mahamuni](https://multicomp.cs.cmu.edu/mmml-course/assets/img/yousha.jpeg)
  * TA Yousha Mahamuni
  * Email: ymahamun@andrew.cmu.edu


![Nikita Malik](https://multicomp.cs.cmu.edu/mmml-course/assets/img/nikita.jpg)
  * TA Nikita Malik
  * Email: nikitama@andrew.cmu.edu


![Aaditya Vikram](https://multicomp.cs.cmu.edu/mmml-course/assets/img/aaditya.jpeg)
  * TA Aaditya Vikram
  * Email: aadityas@andrew.cmu.edu


![Koushik Viswanadha](https://multicomp.cs.cmu.edu/mmml-course/assets/img/koushik.jpeg)
  * TA Koushik Viswanadha
  * Email: koushikv@andrew.cmu.edu


![Danqing Wang](https://multicomp.cs.cmu.edu/mmml-course/assets/img/danqing.jpg)
  * TA Danqing Wang
  * Email: danqingw@andrew.cmu.edu


![Yuteng Zhang](https://multicomp.cs.cmu.edu/mmml-course/assets/img/yuteng.jpeg)
  * TA Yuteng Zhang
  * Email: yutengz@andrew.cmu.edu


## Announcements
## Multimodal Machine Learning
  * CMU MultiComp Lab


  * [ CMU-MultiComp-Lab](https://github.com/CMU-MultiComp-Lab)
  * [ YouTube](https://www.youtube.com/channel/UCqlHIJTGYhiwQpNuPU5e2gg)


© Copyright 2026 Carnegie Mellon University.   
Powered by [Jekyll](http://jekyllrb.com/) with [al-folio](https://github.com/alshedivat/al-folio) theme.