# CS194-26/294-26 Image Manipulation, Computer Vision and Computational Photography https://www-inst.eecs.berkeley.edu/~cs194-26/sp20 | | | --- | | ![Description: \[SCS dragon logo\]](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/ucbseal_139_540.png) | **CS194-26/294-26: Image Manipulation, Computer Vision and Computational Photography** [Computer Science Division](http://www.cs.berkeley.edu/) [ University of California Berkeley](http://www.berkeley.edu/) | **INSTRUCTOR:** [Alexei (Alyosha) Efros](http://www.eecs.berkeley.edu/~efros/) (Office hours: after lecture) **GSI:**[Ashish Kumar](https://ashishkumar1993.github.io/) (Office hours: 5-6pm Wed at Soda Alcove-341B), Violet Fu (Office hours: 5pm-6pm Fri at Soda Alcove-341B), and Shivam Parikh (Office hours: 11am-1pm Mon at Cory 531). **UNIVERSITY UNITS:** 4 **SEMESTER:** Spring 2020 **WEB PAGE:** [http://inst.eecs.berkeley.edu/~cs194-26/fa20/](http://inst.eecs.berkeley.edu/~cs194-26/sp20/) **Q &A:** [Piazza Course Website](http://piazza.com/berkeley/spring2020/cs19426) **LOCATION:** Hearst Field Annex A1 **TIME** : TueThu 5:00 PM-6:30 PM **MIDTERM:** April 16th, Thurs, during the class. **PREREQUISITES:** This is a heavily project-oriented class, therefore good programming proficiency (at least **CS61B**) is absolutely essential. Moreover, familiarity with linear algebra (**MATH 54** or **EE16A/B** or Gilbert Strang's online [class](https://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/video-lectures/)) and calculus are vital. Experience with neural networks (e.g. **CS189**) is a plus. For these taking **CS294-26** , consent of instructor is required to register (please sign up on the waitlist first). **COURSE DESCRIPTION:** The aim of this advanced undergraduate course is to introduce students to computing with visual data (images and video). We will cover acquisition, representation, and manipulation of visual information from digital photographs (image processing), image analysis and visual understanding (computer vision), and image synthesis (computational photography). Key algorithms will be presented, ranging from classical (e.g. Gaussian and Laplacian Pyramids) to contemporary (e.g. ConvNets, GANs), with an emphasis on using these techniques to build practical systems. This hands-on emphasis will be reflected in the programming assignments, in which students will have the opportunity to acquire their own images and develop, largely from scratch, the image analysis and synthesis tools for solving applications. **PROGRAMMING ASSIGNMENTS:** | [Project 1: Images of the Russian Empire -- colorizing the Prokudin-Gorskii photo collection](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/hw/proj1)[](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/hw/proj1) ![Description: http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/pub/www/images/3-8086-left.jpg](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image004.jpg) See student submissions [here](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj1/?C=M;O=D) ![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/trophy.png) **Class Choice Awards:** [Scott Shao](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj1/cs194-26-aes/) [Project 2: Fun with Filters and Frequencies](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/hw/proj2/index.html) ![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/hybrid.png) ![orple](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/orple.jpg) See student submissions [here](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj2/?C=M;O=D) ![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/trophy.png) **Class Choice Awards:** [Scott Shao](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj2/cs194-26-aes/) [ Project 3: Face Morphing and Modelling a Photo Collection ](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/hw/proj3/index.html) ![morph](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/morph.png) See student submissions [here](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj3/?C=M;O=D) ![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/trophy.png) **Class Choice Awards:** [Zhimin Cai](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj3/cs194-26-ace/) Also, see [the class morph video](https://www.youtube.com/watch?v=O2p-U615Rzs&feature=youtu.be). [ Project 4: Classification and Segmentation ](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/hw/proj4/index.html) ![morph](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/hw/proj4/out.jpg) See student submissions [here](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj4/?C=M;O=D) ![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/trophy.png) **Class Choice Awards:** [Andrew Lee](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj4/cs194-26-adg/) [ Project 5: (Auto)stitching and photo mosaics ](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/hw/proj5/index.html) ![stitching](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image028.jpg) See student submissions [partA](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj5A/?C=M;O=D) [partB](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/proj5B/?C=M;O=D) [Final Project](https://inst.eecs.berkeley.edu/~cs194-26/sp20/hw/final-project/index.html) ![multifredo](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/multifredo.jpg) See student submissions [pre-canned](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/projFinalAssigned/?C=M;O=D) [own proposed](https://inst.eecs.berkeley.edu/~cs194-26/sp20/upload/files/projFinalProposed/?C=M;O=D) | | --- | **TEXT:** There is a textbook that covers most (if not all) of the topics related to Computational Photography. This will be the primary reference for the course: [Computer Vision: Algorithms and Applications, RichardSzeliski, 2010](http://szeliski.org/Book/) There is a number of other fine texts that you can use for general reference: Computer Vision: A Modern Approach (2nd edition), Forsyth and Ponce _(classic computer vision text)_ Vision Science: Photons to Phenomenology, Stephen Palmer _(great book on human visual perception) _Digital Image Processing, 2nd edition, Gonzalez and Woods _(a good general image processing text) _Linear Algebra and its Applications, Gilbert Strang _(a truly wonderful book on linear algebra)_ **CLASS NOTES** The instructor is extremely grateful to a large number of researchers for making their slides available for use in this course.[Steve Seitz](http://www.cs.washington.edu/homes/seitz/) and [Rick Szeliski](http://research.microsoft.com/%7Eszeliski/) have been particularly kind in letting me use their wonderful lecture notes.In addition, I would like to thank [Paul Debevec](http://www.debevec.org/), [Stephen Palmer](http://socrates.berkeley.edu/%7Eplab/), [Paul Heckbert](http://www.cs.cmu.edu/%7Eph/), [David Forsyth](http://luthuli.cs.uiuc.edu/%7Edaf/), [Steve Marschner](http://www.cs.cornell.edu/%7Esrm/) and others, as noted in the slides.The instructor gladly gives permission to use and modify any of the slides for academic and research purposes. However, please do also acknowledge the original sources where appropriate. **TENTATIVE CLASS SCHEDULE:** | **CLASS DATE** | **TOPICS** | **Material** | | --- | --- | --- | | Tues Jan 21 | **Introduction ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image006.jpg)** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Introduction.pdf) [pptx](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Introduction.pptx) * [How Photography Became an Art Form](https://medium.com/@aaronhertzmann/how-photography-became-an-art-form-7b74da777c63) by [Aaron Hertzmann](https://www.dgp.toronto.edu/~hertzman/) | | Thurs Jan 23 | **Capturing Light... in man and machine ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image011.png)** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/CapturingLight.pdf) [pptx](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/CapturingLight.pptx) | | Tues Jan 28 | **The Camera ****![Pinhole Camera](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image019.jpg) ** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/camera.pdf) [pptx](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/camera.ppt) * Szeliski Ch. 2 | | Tues/Thurs Jan 30 | **Sampling and Reconstruction ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image010.gif) ** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/ImageProcessingFiltering.pdf) [pptx](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/ImageProcessingFiltering.ppt) * Start Szeliski Ch 3 * Reinhard et al., [Color Transfer Between Images](http://dl.acm.org/citation.cfm?id=618848), IEEE Computer Graphics and Applications, 2001 | | Tues Feb 4 | **Sampling and Reconstruction ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image098.gif)** | * Point Processing Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/PointProcessing.pdf) [pptx](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/PointProcessing.ppt) * Continue Szeliski Ch 3 | | Thurs Feb 6 | **Derivative and Template Filters ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/lena.png)** | * Derivative and Template Filters Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/DerivativeTemplateFilters.pdf) [pptx](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/DerivativeTemplateFilters.ppt) | | Tues Feb 11 | **Image Blending and Compositing ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image033.gif)** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/FreqDomainPyramidsBlending.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/FreqDomainPyramidsBlending.ppt) * Continue Szeliski Ch 3 + Sec 9.3.4 * Additional Reading: * Burt and Adelson, [A multiresolution spline with application to image mosaics](http://persci.mit.edu/pub_pdfs/spline83.pdf), ACM ToG, 1983 * McCann & Pollard, [Real-Time Gradient-Domain Painting](http://graphics.cs.cmu.edu/projects/gradient-paint/), SIGGRAPH 2008 * Perez et al., [Poisson Image Editing](http://www.cs.jhu.edu/~misha/Fall07/Papers/Perez03.pdf), SIGGRAPH 2003 * Bhat et al., [GradientShop: A Gradient-Domain Optimization Framework for Image and Video Filtering](http://grail.cs.washington.edu/projects/gradientshop/), SIGGRAPH 2010 * Avidan and Shamir, [Seam Carving for Content-Aware Image Resizing, ](http://www.win.tue.nl/~wstahw/edu/2IV05/seamcarving.pdf)SIGGRAPH 2007 * Kwatra et al., [Graphcut Textures: Image and Video Synthesis Using Graph Cuts](http://www.cc.gatech.edu/cpl/projects/graphcuttextures/), SIGGRAPH 2003 * Agarwala et al., [Interactive Digital Photomontage](http://grail.cs.washington.edu/projects/photomontage/), SIGGRAPH 2004 | | Thurs Feb 13 | **Compression and Gradient Domain ****** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/CompressionGradientDomain.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/CompressionGradientDomain.ppt) | | Tues Feb 18 | **Edge Detection and Image Warping ****** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/warping.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/warping.ppt) * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Edges.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Edges.pptx) | | Thurs Feb 20 | **Image Morphing ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/morph.gif) ** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/morphing.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/morphing.ppt) * Continue Szeliski Ch 3 | | Feb 25 | **Data-driven Methods: Faces ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image037.jpg) ** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/faces.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/faces.ppt) * Galton, ["Composite portraits made by combining those of many different persons into a single figure."](http://galton.org/essays/1870-1879/galton-1878-nature-composite.pdf), Nature, 1878 * Rowland and Ferrett, ["Manipulating Facial Appearance through Shape and Color"](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Papers/faces.pdf), CG&A, 1995 * Blanz and Vetter, ["A Morphable Model for the Synthesis of 3D Faces"](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Papers/BlanzVetter99.pdf), SIGGRAPH 1999 * Cootes, Edwards, and Taylor, ["Active Appearance Models"](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Papers/eccv98_aam.pdf), ECCV 1998 | | Feb 27, Mar 3 | **Data-driven Methods: Video Textures ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image039.jpg) ** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Video.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Video.pptx) * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Texture.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/Texture.pptx) * Schodl et al., [ Video Textures](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Papers/videotex.pdf) SIGGRAPH' 00 * Efros and Leung, [ Texture Synthesis by Non-parametric Sampling](http://graphics.cs.cmu.edu/people/efros/research/EfrosLeung.html) ICCV'99 * Efros and Freeman, [ Image Quilting for Texture Synthesis and Transfer, ](http://graphics.cs.cmu.edu/people/efros/research/quilting.html) SIGGRAPH'01 * Hertzmann et al. [Image Analogies](http://www.mrl.nyu.edu/projects/image-analogies/), SIGGRAPH 2001. | | Thurs Mar 5 | **Feature Learning with Neural Networks ****** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/NeuralNets.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/NeuralNets.ppt) | | Tues, Thurs Mar 10, Mar 12 | **Convolutional Neural Networks ****** | * Part 1 Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/ConvNets.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/ConvNets.pptx) * Part 2 Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/ConvNetsII.pdf) | | Mar 17, 19 | **Modeling Light ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image067.jpg) ** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/ModelingLight.pdf), [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/ModelingLight.ppt) * [Light of Other Days](https://www.physics.utoronto.ca/~jharlow/slowglass.htm) by Bob Shaw * [ Lightfield ](http://graphics.stanford.edu/projects/lightfield/) | | Mar 31, Apr 7 | **Homographies and Mosaics ****![](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image058.gif) ** | * Szeliski Ch 9 * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/mosaic.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/mosaic.ppt) * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/MoreMosaics.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/MoreMosaics.ppt) | | Apr 9 | **Automatic Alignment ****![auto](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/463_files/image045.gif)** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/feature-alignment.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/feature-alignment.ppt) * Brown et al., [“Multi-Image Matching using Multi-Scale Oriented Patches”](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Papers/MOPS.pdf), CVPR 2005 * M. Brown and D. G. Lowe, [“Recognising Panoramas”](http://matthewalunbrown.com/papers/iccv2003.pdf), ICCV 2003 * [RANSAC](http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/FISHER/RANSAC/) | | Tues, Thurs Apr 14, 16 | **Scene Modeling for a Single View ****** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/SingleView.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/SingleView.ppt) * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/SingleViewReconstruction.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/SingleViewReconstruction.ppt) | | Tues, Thurs Apr 21, 23 | **Multiview Geometry: Stereo & Structure from Motion ****** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/multiview.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/multiview.pptx) | | Tues, Thurs Apr 28, 30 | **Class chosen special topics ****** | * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/postmodern.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/postmodern.pptx) * Slides: [pdf](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/self-supervision.pdf) [ppt](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/Lectures/self-supervision.ppt) | **CAMERAS: Although it is not required, students are highly encouraged to obtain a digital camera for use in the course.** **METHOD OF EVALUATION: Grading will be based on a set of programming and written assignments (60%), a midterm exam + several in-class pop quizzes (20%) and a final project (20%).For the programming assignments, students will be allowed a total of 5 (five) late days per semester; each additional late day will incur a 10% penalty.** Students taking CS294-26 will also be required to submit a conference-style paper describing their final project. **PROGRAMMING RESOURCES : Students will be encouraged to use either MATLAB (with the Image Processing Toolkit) or Python (with either scikit-image or opencv) as their primary computing platform.Specific libraries in both languages offer tons of build-in image processing functions.Here is a link to some [useful MATLAB and Python resources](https://www-inst.eecs.berkeley.edu/~cs194-26/sp20/programming.html) compiled for this class.** **PREVIOUS OFFERINGS OF THIS COURSE : Previous offerings of this course can be found [_here_](http://inst.eecs.berkeley.edu/~cs194-26/fa18/).** **SIMILAR COURSES IN OTHER UNIVERSITIES:** * [_Computational Photography_](http://courses.engr.illinois.edu/cs498dh3/) (Hoiem, UIUC) * [_Computational Photography_](http://www.cs.brown.edu/courses/csci1290/) (Hays, Brown) * [_Digital and Computational Photography_](http://stellar.mit.edu/S/course/6/sp12/6.815/) (Durand, MIT) * [_Computer Vision_](http://www.cs.washington.edu/education/courses/cse576/05sp/)(Seitz & Szeliski, UWashington) Page design courtesy of [_Doug James_](http://www.cs.cmu.edu/~djames/)