Project 5: Diffusion Models | Introduction to Computer Vision

Hole Filling

"Make it Real"

A Lithograph of a Waterfall

A Lithograph of a Skull

An Oil Painting of an Old Man

An Oil Painting of People Around a Fire

Key Information

Assigned	Thursday, April 17, 2025 (fork the starter code from Part A: GitHub Classroom, and Part B: GitHub Classroom)
Due	Part A: Friday, April 25, 2025 (early due date): submit to your Part A GitHub repo and CMSX by 8:00 PM Part B: Tuesday, May 6, 2025 (Extended): submit to your Part B GitHub repo and CMSX by 11:59 PM
Code Submission	For each part, submit the following to your GitHub Classroom repository: - Completed notebook file (`.ipynb`) with figures and results - Python backup file (`.py`) with your final implementation Also save your Colab notebook with figures and results as a PDF and upload it to the corresponding CMSX assignment for Part A and Part B.
Individual Project	This project must be completed individually (no group work).

Synopsis

This project introduces you to diffusion models for image generation. You'll implement and explore these models across two parts, each with its own Colab notebook and due date (see Key Information for details).

In this project, you will learn:

What diffusion models are and how they work
How to implement diffusion sampling loops to generate images
How to use diffusion models for creative tasks like image inpainting and optical illusions
How to train your own diffusion model on a small dataset (MNIST)

Tip: Start early! This may be the most challenging project of the semester, and training or sampling can take time.

Using Google Colab

Google Colab is a cloud-based platform for running Python code, similar to Jupyter Notebooks. It requires no installation and saves your work directly to your Google Drive, just like Docs or Sheets.

We recommend using a Colab notebook with GPU enabled to complete this project efficiently.

Note: Colab has a free usage limit. If you're running long training sessions—especially close to the deadline—you may encounter resource restrictions. Some students work around this by:

Using a different Google account (within Google’s usage policies)
Upgrading to Colab Pro (a paid option) for extended runtime and better resources

Colab Setup (Same as NeRF Project)

Check out the code base from your Github Classroom repo
Upload .ipynb notebook to your Google drive
Double clicking on the notebook on Google drive should give you an option for opening it in Colab
Alternatively, you can directly open Colab and upload notebook following this:
```
File -> Upload notebook...
```
If you haven’t used Colab or Jupyter Notebooks before, first read the Colaboratory welcome guide
As already stated, Colab requires almost no setup, so there is no need to install PyTorch locally

Detailed Instructions

The project is divided into two parts. Each part has its own notebook, submission requirements, and implementation details. Please refer to the corresponding instruction pages:

Part A: View instructions for Part A — includes sampling loop implementation and creative image tasks
Part B: View instructions for Part B — includes training your own diffusion model on MNIST

Acknowledgements

This project was a joint effort by Daniel Geng, Ryan Tabrizi, and Hang Gao, advised by Liyue Shen, Andrew Owens, and Alexei Efros.

Last updated 16 April 2025