Term | Spring 2022 | Instructor | Christopher De Sa |

Course website | www.cs.cornell.edu/courses/cs4780/2022sp/ | [email hidden] | |

Schedule | MW 7:30-8:45PM | Office hours | Wednesdays 2PM |

Room | Statler 185 | Office | Gates 450 |

[Course Texts] [Course Calendar] [Canvas] [Discussion] [Vocareum]

Description: CS4/5780 provides an introduction to machine learning, focusing on supervised learning and its theoretical foundations. Topics include regularized linear models, boosting, kernels, deep networks, generative models, online learning, and ethical questions arising in ML applications.

Prerequisites: probability theory (e.g. BTRY 3080, ECON 3130, MATH 4710, ENGRD 2700), linear algebra (e.g. MATH 2940), calculus (e.g. MATH 1920), and programming proficiency (e.g. CS 2110). You must also pass the prelim exam.

Format: In accordance with university guidelines, the first two weeks of the course will be taught entirely online. Both lectures and office hours will be conducted over zoom. In-person lectures are expected to resume starting on February 7.

Material: The course is based on books, papers, and other texts in machine learning, scalable optimization, and systems. Texts will be provided ahead of time on the website on a per-lecture basis. You aren't expected to necessarily read the texts, but they will provide useful background for the material we are discussing.

Logistics: For enrolled students the companion Canvas page serves as a hub for access to the lecture zoom links, TA office hour zoom links, the TA office hour schedule, Ed Discussions (the course forum), Vocareum (for course projects), Gradescope (for HWs), and quizzes (for the placement exam and paper comprehension quizzes). If you are enrolled in the course you should automatically have access to the site. Please let us know if you are unable to access it.

Late Days: All assignments have two (2) slip days associated with them. This means that you can submit everything (except exams) up to two days after the posted deadline without penalty. Further extensions warranted by extreme circumstances will be handled as needed on a case-by-case basis (inquire on Ed).

Grading: For CS4780, your grade in this course is comprised of three components: homework, exams, and projects. CS5780 also has an additional paper comprehension component. Please also read through the given references in concert with the lectures.

- Homework: There will be a number of homework assignments throughout the course, typically made available roughly one to two weeks before the due date. The homework primarily focuses on theoretical aspects of the material and is intended to provide preparation for the exams. Homework may be completed in groups of up to two. The assignments themselves will be made available via Gradescope (through Canvas).
- Project: To provide hands on learning with the methods we will discuss in class there are a number of programming projects throughout the course. The projects may be completed solo or in a group of two. They are accessed through assignments in Canvas that will launch Vocareum.
- Exams: There will be two exams for this class, an evening prelim and a (comprehensive) final exam.
- Paper comprehension: Students enrolled in this course at the graduate level (i.e., enrolled in 5780) are required to read assigned research papers and complete the associated online quiz. Papers will be assigned roughly once every one to two weeks.

CS 4780 grade breakdown. For CS 4780, paper comprehension is not a required component, but you may choose to complete them if you want. Your final grade consists of the maximum of

60% | Exams | or | 50% | Exams | or | 55% | Exams | or | 45% | Exams |

10% | Problem Sets | 10% | Problem Sets | |||||||

40% | Projects | 40% | Projects | 35% | Projects | 35% | Projects | |||

10% | Paper Comprehension | 10% | Paper Comprehension |

CS 5780 grade breakdown. For CS 5780, the paper comprehension is a required component, and your final grade consists of the maximum of either

55% | Exams | or | 45% | Exams |

10% | Problem Sets | |||

35% | Projects | 35% | Projects | |

10% | Paper Comprehension | 10% | Paper Comprehension |

Regardless of which grading scheme you are targeting (or ends up being the maximizer), homework must be completed. Homework will be graded for correctness and these scores will be used to compute your overall homework grade. Provided you make a good faith effort (as specified in class) on the homework they do not factor into your final grade under the first scheme above. However, failure to provide a good faith effort for any homework assignments will result in a 5% penalty per missing assignment.

Inclusiveness: You should expect and demand to be treated by your classmates and the course staff with respect. You belong here, and we are here to help you learn—and enjoy—this course. If any incident occurs that challenges this commitment to a supportive and inclusive environment, please let the instructor know so that we can address the issue. We are personally committed to this, and subscribe to the Computer Science Department's Values of Inclusion.

COVID-19 considerations. We understand that the ongoing global health pandemic impacts all of you in varied and profound ways. Therefore, flexibility is important as we continue to navigate the current state of affairs. While many aspects of this course are built with flexibility in mind, if situations arise that may require additional accommodations please reach out to the instructors to discuss potential arrangements.

Mental health resources. Cornell University provides a comprehensive set of mental health resources and the student group Body Positive Cornell has put together a flyer outlined the resources available.

Participation. You are encouraged to actively participate in class. This can take the form of asking questions in class, responding to questions to the class, and actively asking/answering questions on the online discussion board.

Collaboration policy. Students are free to share code and ideas within their stated project/homework group for a given assignment, but should not discuss details about an assignment with individuals outside their group. The midterm and final exam are individual assignments and must be completed by yourself.

Academic integrity. The Cornell Code of Academic Integrity applies to this course.

Accommodations. In compliance with the Cornell University policy and equal access laws, we are available to discuss appropriate academic accommodations that may be required for student with disabilities. Requests for academic accommodations are to be made during the first three weeks of the semester, except for unusual circumstances, so arrangements can be made. Students are encouraged to register with Student Disability Services to verify their eligibility for appropriate accommodations.

Main texts. While this course does not explicitly follow a specific textbook, there are several that are very useful references to supplement the course.

**Machine Learning A Probabilistic Perspective by Murphy**

We will provide section numbers to this text alongside many of the lectures (abbreviated as MLaPP in the schedule). This text is available digitally through the Cornell University Library. [Cornell library]**The Elements of Statistical Learning by Hastie, Tibshirani, and Friedman**

This text provides a comprehensive introduction to statistical learning and provides in-depth discussion of many of the topics in this course (abbreviated as ESL in the schedule). The book is available directly from the authors. [Book website]

Additional references.

**An Introduction to Statistical Learning by James, Witten, Hastie, and Tibshirani**

This book provides a good overview of some methods in statistical learning, some of which we will discuss. The book is available online through the books website and via the Cornell Library. [Book website]**Patterns, Predictions, and Actions by Hardt and Recht**

A very nice new book that covers many of the topics we do in this class (abbreviated as PPA in the schedule). The book is available directly from the authors. [Book website]**Fairness and Machine Learning by Barocas, Hardt, and Narayanan**

While a work in progress, this text provides insight into fairness as a central tenet of machine learning. In particular, it highlights ethical challenges that arise in the practice of machine learning. The current version of this book is available directly from the authors. [Book website]

Background references.

**Linear Algebra by Kahn Academy**

Relive the basics of linear Algebra. Everybody loves Kahn Academy. [Linear Algebra (Kahn Academy)]**Linear algebra course by Strang**

Portions of this course will utilize your knowledge of linear algebra. If you feel you need additional preparation, or would like to revisit the topic, you may find Gilbert Strangs linear algebra course quite useful. [MIT Open Courseware]**Matrix Methods in Data Analysis, Signal Processing, and Machine Learning by Strang**

A subsequent course to the above by Strang covers some of the same topics we will (particularly for the linear algebra part of the course) and you may find the videos a useful additional resource. [MIT Open Courseware]

Software. Software we will use in CS4/5780 includes Python, NumPy, and PyTorch.

Course calendar may be subject to change.

Monday, January 24 Jan 23Jan 24Jan 25Jan 26Jan 27Jan 28Jan 29 |
Lecture 1. Introduction. [Slides]
Prelim Exam Assigned. Programming Assignment -1 assigned. Reading material: MLAPP: 1.1, 1.2, and 1.3; ESL: Ch. 1; and PPA: Ch. 1. |

Wednesday, January 26 Jan 23Jan 24Jan 25Jan 26Jan 27Jan 28Jan 29 |
Lecture 2. ML Basics. [Slides] [Notes] [Handwritten Notes]
Reading material: ESL: 2.1 and 2.2.. |

Monday, January 31 Jan 30Jan 31Feb 1Feb 2Feb 3Feb 4Feb 5 |
Lecture 3. k nearest neighbors and the curse of dimensionality. [Slides] [Notebook] [Notes]
Homework 1 assigned. Programming Assignment 0 assigned. Reading material: MLAPP: 1.1, 1.2, 1.4.2, 1.4.3, and 1.4.9. |

Wednesday, February 2 Jan 30Jan 31Feb 1Feb 2Feb 3Feb 4Feb 5 |
Lecture 4. K-means clustering. [Light Slides] [Dark Slides] [Notes]
Programming Assignment 1 assigned. Prelim Exam Due. Reading material: ESL: 14.3.6 and 14.3.7, and MLAPP: 11.4.1 and 11.4.2. |

Monday, February 7 Feb 6Feb 7Feb 8Feb 9Feb 10Feb 11Feb 12 |
Lecture 5. Principal component analysis. [Notes] [Printable Notes PDF] [Document Camera Notes]
Reading material: ESL: 14.5.1 and 14.5.2. |

Wednesday, February 9 Feb 6Feb 7Feb 8Feb 9Feb 10Feb 11Feb 12 |
Lecture 6. The Perceptron. [Notes] [Printable Notes PDF] [Document Camera Notes]
Paper Reading 1 assigned: Cover and Hart 1967. Programming Assignment 0 due. Reading material: Perceptron Wiki. |

Monday, February 14 Feb 13Feb 14Feb 15Feb 16Feb 17Feb 18Feb 19 |
Lecture 7. MLE and MAP.
[Notes] [Printable Notes PDF]
Homework 1 due. Homework 2 assigned. Reading material: Nice Youtube video for MLE and MAP. Ben Taskar's lecture notes. ESL: 8.2.2-8.3. |

Wednesday, February 16 Feb 13Feb 14Feb 15Feb 16Feb 17Feb 18Feb 19 |
Lecture 8. Naive Bayes.
[Notes] [Printable Notes PDF] [Document Camera Notes]
Programming Assignment 1 due. Programming Assignment 2 assigned. Reading material: ESL: 6.6.3; Tom Mitchell’s book chapter.. |

Monday, February 21 Feb 20Feb 21Feb 22Feb 23Feb 24Feb 25Feb 26 |
Lecture 9. Naive Bayes II.
[Notes Same As Naive Bayes I] [Printable Notes PDF] [Document Camera Notes (with Lecture 10)]
Reading material: ESL: 6.6.3; Tom Mitchell’s book chapter. |

Wednesday, February 23 Feb 20Feb 21Feb 22Feb 23Feb 24Feb 25Feb 26 |
Lecture 10. Logistic regression.
[Notes] [Printable Notes PDF] [Document Camera Notes (with Lecture 9)]
Homework 2 due. Paper Reading 1 due. Homework 3 assigned. Reading material: MLaPP: 8.2, 8.3, and 8.4; ESL 4.4. |

Monday, February 28 Feb 27Feb 28Mar 1Mar 2Mar 3Mar 4Mar 5 |
February Break. No Classes. No lecture. |

Wednesday, March 2 Feb 27Feb 28Mar 1Mar 2Mar 3Mar 4Mar 5 |
Lecture 11. Gradient descent and Newton's method.
[Notes] [Printable Notes PDF] [Document Camera Notes]
Programming Assignment 2 due. Programming Assignment 3 assigned. Reading material: MLaPP: 8.2, 8.3, and 8.4. |

Monday, March 7 Mar 6Mar 7Mar 8Mar 9Mar 10Mar 11Mar 12 |
Lecture 12. Linear Regression.
[Notes] [Printable Notes PDF]
Reading material: MLaPP 8, 8.1, 8.2, 8.3.1, 8.3.2, 8.3.4, 8.6; Tom Mitchell’s book chapter on Naive Bayes and Linear Regression; ISL: 3.1; and ESL 3.2. |

Wednesday, March 9 Mar 6Mar 7Mar 8Mar 9Mar 10Mar 11Mar 12 |
Lecture 13. Support vector machines (SVM).
[Notes] [Printable Notes PDF]
Homework 3 due. Homework 4 assigned. |

Monday, March 14 Mar 13Mar 14Mar 15Mar 16Mar 17Mar 18Mar 19 |
Lecture 14. SVM + Empirical risk minimization (ERM).
[Notes] [Printable Notes PDF]
Programming Assignment 3 due. Programming Assignment 4 released. Reading material: MLaPP 6.5-6.5.3.2. |

Wednesday, March 16 Mar 13Mar 14Mar 15Mar 16Mar 17Mar 18Mar 19 |
Midterm Review. |

Thursday, March 17 Mar 13Mar 14Mar 15Mar 16Mar 17Mar 18Mar 19 |
Prelim Exam. 7:30PM. BKL200, BKL219, BKL335. |

Monday, March 21 Mar 20Mar 21Mar 22Mar 23Mar 24Mar 25Mar 26 |
Lecture 15. Kernels.
[Notes] [Printable Notes PDF]
Homework 4 due. Homework 5 assigned. Reading material: MLaPP Ch. 14. |

Wednesday, March 23 Mar 20Mar 21Mar 22Mar 23Mar 24Mar 25Mar 26 |
Lecture 16. Kernels II.
[Notes] [Printable Notes PDF]
[All Classifiers Demo]
Programming Assignment 4 due. Programming Assignment 5 assigned. Reading material: MLaPP Ch. 14. |

Monday, March 28 Mar 27Mar 28Mar 29Mar 30Mar 31Apr 1Apr 2 |
Lecture 17. Bias-variance tradeoff. [Notes] [Printable Notes PDF] [Under/Overfitting Demo] |

Wednesday, March 30 Mar 27Mar 28Mar 29Mar 30Mar 31Apr 1Apr 2 |
Lecture 18. ERM Returns + Model Selection.
[Notes] [Printable Notes PDF] [Gradient Descent Under/Overfitting Demo]
Homework 5 due. |

Monday, April 4 Apr 3Apr 4Apr 5Apr 6Apr 7Apr 8Apr 9 |
Spring Break. No Classes. No lecture. |

Wednesday, April 6 Apr 3Apr 4Apr 5Apr 6Apr 7Apr 8Apr 9 |
Spring Break. No Classes. No lecture. |

Monday, April 11 Apr 10Apr 11Apr 12Apr 13Apr 14Apr 15Apr 16 |
Lecture 19. Classification and regression trees.
[Notes] [Printable Notes PDF] [KD-Tree Demo]
Programming Assignment 6 assigned. |

Wednesday, April 13 Apr 10Apr 11Apr 12Apr 13Apr 14Apr 15Apr 16 |
Lecture 20. Classification and regression trees II. [Notes] [Printable Notes PDF] Programming Assignment 5 due April 14. Homework 6 assigned. |

Monday, April 18 Apr 17Apr 18Apr 19Apr 20Apr 21Apr 22Apr 23 |
Lecture 21. Bagging. [Notes] [Printable Notes PDF] [Document Camera Notes]
Reading material: ISL 8.2. |

Wednesday, April 20 Apr 17Apr 18Apr 19Apr 20Apr 21Apr 22Apr 23 |
Lecture 22. Boosting. [Notes] [Printable Notes PDF] [Multi-Lecture Document Camera Notes]
Programming Assignment 6 due. Programming Assignment 7 assigned. Reading material: ISL 8.2. |

Monday, April 25 Apr 24Apr 25Apr 26Apr 27Apr 28Apr 29Apr 30 |
Lecture 23. Neural Networks. [Notes] [Printable Notes PDF] [Multi-Lecture Document Camera Notes] Homework 7 assigned. Reading material: ESL 11.3, 11.4, 11.5. For more in-depth coverage of the modern practice of neural networks, see the Deep Learning Book Part II. |

Wednesday, April 27 Apr 24Apr 25Apr 26Apr 27Apr 28Apr 29Apr 30 |
Lecture 24. Neural Networks II. [Notes (same as previous)] [Multi-Lecture Document Camera Notes] Homework 6 due. Programming Assignment 8 assigned. |

Monday, May 2 May 1May 2May 3May 4May 5May 6May 7 |
Lecture 25. Neural Networks III. [Notes (same as previous)] [Multi-Lecture Document Camera Notes] |

Wednesday, May 4 May 1May 2May 3May 4May 5May 6May 7 |
Lecture 26. Neural Networks IV. [Notes (same as previous)] [Document Camera Notes] Programming Assignment 7 due. |

Monday, May 9 May 8May 9May 10May 11May 12May 13May 14 |
Lecture 27.Final Exam Review. Homework 7 due. Programming Assignment 8 due May 10. |

Wednesday, May 18 May 15May 16May 17May 18May 19May 20May 21 |
Final Exam. 2:00PM. Barton Hall 100 West. |