PA5 was adapted from an assignment for the Stanford course taught by Andrej Karpathy and Fei-Fei Li.
- Assigned: Wednesday, April 22, 2015
- Code Due: Monday, May 4, 2015 (by 9:00am)
- Artifact: No artifact
- This assignment should be done in teams of 2 students.
In this assignment you will practice writing backpropagation code, and training Neural Networks and Convolutional Neural Networks. The goals of this assignment are as follows:
- understand Neural Networks and how they are arranged in layered architectures
- understand and be able to implement (vectorized) backpropagation
- implement the core parameter update loop of mini-batch gradient descent
- effectively cross-validate and find the best hyperparameters for Neural Network architecture
- understand the architecture of Convolutional Neural Networks and train gain experience with training these models on data
Get the code:
This will create the directory skeleton. To get updates to the code you can then simply run
git clone http://www.cs.cornell.edu/courses/cs4670/2015sp/projects/pa5/skeleton.git
git pull. Note: before diving into the code, we suggest reading the notebooks below to familiarize yourself with what you will be doing. The actual assignment is inside the notebooks; the notebooks will call code inside the
cs4670folder (which you will write).
Getting set up on the virtual machine: For the VirtualBox virtual machine, there are only a few missing dependencies to install:
# Install missing dependency sudo apt-get install libncurses-dev # There's no need for a virtualenv on the virtual machine since there's nothing but the assignments there. (expect this to take ~20min) sudo pip install -r requirements.txt
Getting set up on Mac OSX: You're also welcome to use the virtual machine, but here are steps to getting this to work natively on OSX:
# Set up homebrew (if you don't already have it) ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)" # Install python and dependencies brew install wget gcc pkg-config python libjpg libpng freetype # fix a link problem ln -s /usr/local/opt/freetype/include/freetype2 /usr/local/include/freetype # ** IF NOT USING VIRTUALENV (see below) ** # (expect this to take ~20min) pip install -r requirements.txt
Getting set up on Windows: If someone gets it working on Windows and posts a walkthrough, we can include it here. Otherwise, we recommend using the VirtualBox virtual machine.
[Optional] Python virtual environment: Once you have unzipped the starter code, you might want to create a virtual environment for the project. If you choose not to use a virtual environment, it is up to you to make sure that all dependencies for the code are installed on your machine. To set up a virtual environment, run the following:
cd pa5 sudo pip install virtualenv # This may already be installed virtualenv .env # Create a virtual environment source .env/bin/activate # Activate the virtual environment pip install -r requirements.txt # Install dependencies (will take 20min) # Work on the assignment for a while ... deactivate # Exit the virtual environment
If not using a virtual environment, run
sudo pip install -r requirements.txt.
Once you have the starter code, you will need to download the CIFAR-10 dataset.
Run the following from the
cd cs4670/datasets bash get_datasets.sh
Compile the Cython extension: Convolutional Neural Networks require a very efficient implementation. We have implemented of the functionality using Cython; you will need to compile the Cython extension before you can run the code. From the
cs4670 directory, run the following command:
python setup.py build_ext --inplace
After you have the CIFAR-10 data, you should start the IPython notebook server from the
pa5 directory. If you are unfamiliar with IPython, you should read our
Note that the command
ipython notebook is the actual text you
notebook is not the name of the notebook.
Submitting your work:
Once you are done working run the
this will produce a file called
Q1: Two-layer Neural Network
The IPython Notebook
two_layer_net.ipynb will walk you through implementing a two-layer neural network on CIFAR-10. You will write a hard-coded 2-layer Neural Network, implement its backprop pass, and tune its hyperparameters.
Q2: Modular Neural Network
The IPython Notebook
layers.ipynb will walk you through a modular Neural Network implementation. You will implement the forward and backward passes of many different layer types, including convolution and pooling layers.
conv_backward_naivefunction is extra credit.
Q3: ConvNet on CIFAR-10
The IPython Notebook
convnet.ipynb will walk you through the process of training a (shallow) convolutional neural network on CIFAR-10. It wil then be up to you to train the best network that you can.
Q4: Do something extra!
In the process of training your network, you should feel free to implement anything that you want to get better performance. You can modify the solver, implement additional layers, use different types of regularization, use an ensemble of models, or anything else that comes to mind. If you implement these or other ideas not covered in the assignment then you will be awarded some bonus points.