CS 6662 - Computational Imaging
Fall 2024

Course description

Computational imaging is the holistic design of imaging systems together with algorithms, blending ideas from computer vision, optics, imaging, and machine learning to overcome the limits of traditional cameras and imaging systems. Computational imaging enabled the capture of the first image of the black hole, the imaging of invisible objects, and is used in most smart-phone cameras for HDR imaging and super-resolution. This course will provide an overview of the state of the art in computational imaging. We will learn how to mathematically model different aspects of imaging systems, such as noise, aberrations, and light propagation. In addition, we will learn how to formulate and solve imaging inverse problems using both classical and modern deep-learning-based approaches. Throughout the course, we will discuss exciting active research topics such as lensless imaging, compressive imaging, phase microscopy, time-of-flight imaging, and tomography. The class includes project-based homework assignments where you'll get practical experience trying out the concepts presented in class on real camera hardware. The class will culminate in an open-ended final project where you will implement a computational imaging system both in hardware and software.

This class is targeted at graduate students and will provide a foundation for doing research in computational imaging. You should take this class if you're:

• a computer vision/graphics/robotics/HCI researcher and want to learn more about camera hardware and the physics of light formation.
• an applied physicist and want to learn some practical skills for improving imaging systems
• an engineer/physicist working on optical systems and want to learn more about imaging algorithms.
• using imaging systems (microscope, telescope, cameras) and want to learn how they work and how to push them past their limits
• excited about light field cameras or capturing black holes with massive earth-sized telescopes and you want to learn more!
• anyone else! Come learn about imaging systems with us!

By the end of the course, you will be able to:

• Mathematically model different aspects of imaging systems, including noise, aberrations, and wavelength dependence.
• Formulate and solve imaging inverse problems for several imaging systems (e.g. deconvolution, denoising, tomography, phase imaging) using several different methods.
• Differentiate and distinguish different inverse problem algorithms, from classic to deep methods.
• Implement and evaluate a computational imaging system in both hardware and software through a final project, with your choice of imaging system.

Prerequisites

Knowledge of linear algebra and working knowledge of Python is required. Knowledge of convex optimization, computer vision, and machine learning are recommended but not required.

Course Materials

Camera kit. You will need access to a mirrorless or DSLR camera with aperture and exposure control and the ability to shoot in RAW mode. If you do not have a camera, you can loan out a mirrorless camera to use in your homework assignments and final projects. We have 30 camera kits available. See our camera tutorial for instructions on how to set up and use your camera.

There is no required textbook. Each lecture will include references and papers that you will need to consult to successfully complete the course. The following textbook can be useful references in general. All of them are available online for free, or through the Cornell libraries:

• Computer Vision: Algorithms and Applications, by Richard Szeliski.
• Computational Imaging Book, by Ayush Bansai, Achuta Kadambi, and Ramesh Raskar.
• Foundation of Computer Vision, by Antonio Torralba, Phillip Isola, William T. Freeman.

Course Units, learning outcomes, and assessments

Unit 1: From bits to images — the image processing pipeline

By the end of this unit, you will be able to:

• Describe the common elements of imaging systems and their role
• Describe and mathematically model the image processing pipeline (demosaicing, white balancing, gamma correction, etc.)
• Describe the effects of aperture, exposure, ISO, and focal length on image capture.

Assessments: Exploring Pinhole cameras, camera controls, and the image processing pipeline. [HW1]

Unit 2: Imaging forward models

By the end of this unit, you will be able to:

• Mathematically model noise in CMOS sensors.
• Describe and analyze the effect of common lens aberrations
• Mathematically model common imaging forward models (e.g. blur)
• Describe shift-invariant and shift-varying models.

Assessments: Building and modeling computational cameras. [HW2]

Unit 3: Imaging inverse problems.

By the end of this unit, you will be able to:

• Formulate and solve imaging inverse problems using a convex optimization approach (e.g. FISTA, ADMM)
• Distinguish and identify whether an inverse problem is underdetermined, overdetermined, and ill-conditioned.
• Describe and analyze the impact of priors on imaging inverse problems.
• Formulate and solve imaging inverse problems using a deep-learning-based approach.

Assessments: Using inverse problems to recover images from a lensless camera. [HW3]

Unit 4: Novel cameras and applications.

By the end of this unit, you will be able to:

• Identify and describe several current research topics in computational imaging.
• Mathematically model a novel computational imaging system.
• Implement and evaluate a computational imaging system in hardware and software.

Assessments: Final project: implement and analyze a computational imaging system of your choice.

Evaluation

Your final grade will be made up of:

• Class participation (10%).
• Three homework assignments (40%).
• Final project (50%).

Class participation: You are expected to regularly attend class, do any pre-class reading assignments, and actively participate in classroom discussions.

Homework assignments: Homework assignments will give you the chance to implement and explore lecture materials both in hardware (on your mirrorless camera) and in sofware (in Python). We will provide some Jupyter Notebook starter code for assignments. The homework assignments are long and thorough, so please start early so you have ample time to iterate, work together, and get help (stop by office hours if you get stuck!). Homework assignments will include bonus material and several optional photography contests!

Late days: It's important to stay on track with homework assignments so you don't get behind. However, unexpected circumstances sometimes arise. For this reason, you can take a total of five free late days during the semester. We believe that prompt feedback is important for learning, so we plan to release the homework solutions soon after the due date. Due to this, we cannot accept any submissions that are more than five days late.

Final project: This class will culminate in a final project in which you will build, implement, and analyze a computational imaging system of your choice. You'll have chances to get early feedback on the project through a project proposal and check-ins. The project will conclude with a final presentation during lecture as well as a project report. A panel of judges will select the top two projects, which will receive a prize! More details on the final project can be found here.

Grading system. We will adhere to Cornell's official grading system when assigning grades. See details here.

Course expectations & policies

Class participation: Discussion, collaboration, and teamwork are essential to participate in the broader scientific community. In this class, we will practice thinking critically about new concepts, discussing ideas, and brainstorming together as scientific teams. We will do this through active learning in the classroom. You are expected to actively participate in class discussions, answer questions, and engage in any group activities and discussions. We value all perspectives and ideas, and are here to learn and grow together. Due to the interactive nature of lectures, all students are expected to attend lectures.

Collaboration policy: We encourage students to work in groups on homework assignments, but you must submit your own work. You must write your own code, build your own imaging setups, take your own photographs, and produce your own write-up. If you work together on a homework assignment, please include the names of your collaborators in your writeup. You must not supply code or write-ups from this course with students in future instances of this course, or make any material available online. Any online repositories for this class must be kept private.

Mental Health and Wellbeing: Your mental health and wellbeing are important to me. If you or a friend are struggling emotionally or feeling stressed, fatigued, or burned out, there are campus resources available to you: here. Help is also available any time day or night through Cornell’s 24/7 phone consultation (607-255-5155). You can also reach out to me, your college student services office, or Cornell Health for support.

Students with Disabilities: Your access in this course is important to me. Please contact Student Disability Services (SDS) sds.cornell.edu to register and request an accommodation letter as early as possible. More information can be found here. If you have, or think you might have, a disability, please contact Student Disability Services for a confidential discussion: sds_cu@cornell.edu or visit sds.cornell.edu to learn more.

Please note this syllabus is subject to change. Any changes will be announced in class or on the course website.