Syllabus for CS6787

Advanced Machine Learning Systems — Fall 2020

[Piazza site] [Lecture zoom link] [Office hours zoom link] [Canvas lecture videos link]

Course Modality Info. CS6787 will be offered both hybrid-in-person and online, subject to the following policies.

• While most of the lectures will be given in person (with online component via zoom), some of the lectures will be given online-only. The modality of each lecture will be listed on the course calendar. If a class is listed as being given online-only, do not come to the classroom. Note that lectures may be converted from in-person to online-only on an ad hoc basis as developments occur &emdash; you will be sent an email if this happens.
• The classroom has a limited number of seats. You will be assigned a seat number sometime before the second lecture (after giving students a chance to add/drop after the first lecture's course overview). If you have not been assigned a seat, please do not come to the classroom (attend remotely instead).
• If you are taking the class online, you will generally still be expected to join the zoom call for the lectures synchronously. While lectures will be recorded, in-class discussion is central to the learning goals of CS6787. Note, though, that you will not be graded on attendance, and students enrolled in the in-person class are not required to attend the lectures in person.
• For the time being, students on the waitlist will be given full access to lectures and course materials, to enable course shopping as smoothly as possible.
• Office hours will be given remotely via zoom only.

So you've taken a machine learning class. You know the models people use to solve their problems. You know the algorithms they use for learning. You know how to evaluate the quality of their solutions.

But when we look at a large-scale machine learning application that is deployed in practice, it's not always exactly what you learned in class. Sure, the basic models, the basic algorithms are all there. But they're modified a bit, in a bunch of different ways, to run faster and more efficiently. And these modifications are really important—they often are what make the system tractable to run on the data it needs to process.

CS6787 is a graduate-level introduction to these system-focused aspects of machine learning, covering guiding principles and commonly used techniques for scaling up learning to large data sets. Informally, we will cover the techniques that lie between a standard machine learning course and an efficient systems implementation: both statistical/optimization techniques based on improving the convergence rate of learning algorithms and techniques that improve performance by leveraging the capabilities of the underlying hardware. Topics will include stochastic gradient descent, acceleration, variance reduction, methods for choosing hyperparameters, parallelization within a chip and across a cluster, popular ML frameworks, and innovations in hardware architectures. An open-ended project in which students apply these techniques is a major part of the course.

Prerequisites: Knowledge of machine learning at the level of CS4780. If you are an undergraduate, you should have taken CS4780, since it is a prerequisite. Optionally, knowledge of computer systems and hardware on the level of CS 3410 would be useful, but this is not a prerequisite.

Format: About half of the classes will involve traditionally formatted lectures. For the other half of the classes, we will read and discuss two seminal papers relevant to the course topic. These classes will involve presentations by groups of students of the paper contents (each student will sign up in a group to present one paper for 15-20 minutes) followed by breakout discussions about the material. Historically, the lectures have occurred on Mondays and the discussions have occurred on Wednesdays, but due to the non-standard timeline this semester, these course elements will be scheduled irregularly (see schedule below).

Grading: Students will be evaluated on the following basis.

Paper review parameters: Paper reviews should be about one page (single-spaced) in length. The review guidelines should mirror what an actual conference review would look like (although you needn't assign scores or anything like that). In particular you should at least: (1) summarize the paper, (2) discuss the paper's strengths and weaknesses, and (3) discuss the paper's impact. For reference, you can read the ICML reviewer guidelines. Of course, your review will not be precisely like a real review, in large part because we already know the impact of these papers. You can submit any review up to two days late with no penalty. Students who presented a paper do not have to submit a review of that paper (although you can if you want).

Final project parameters (subject to change): The final project can be done in groups of up to three (although more work will be expected from groups with more people). The subject of the project is open-ended, but it must include:

• the implementation of a machine learning system for some task,
• exploring one or more of the techniques discussed in the course (or similar techniques subject to instructor approval),
• to empirically evaluate the performance and compare it with some baseline method, in two ways:
• statistical performance (e.g. iterations to converge to some accuracy threshold), and
• hardware performance (e.g. throughput or wall-clock time).

• The main body should be about one page in length.
• It should describe the project you intend to do.
• It should contain at least one citation of a relevant paper that we did not cover in class (but preferably more).
• It should include some preliminary or exploratory work you've already done, that helps to support the idea that your project is feasible (this preliminary work can be very minimal, but should indicate that you've got started—or at least have a clear idea how to do so).
• In addition to the one-page text proposal, it should contain one short experiment plan per person, which should consist of:
  • a hypothesis
  • a proxy statement which describes what metric you are going to use to measure the variables you care about
  • a short protocol statement describing what you are going to do
  • the results you expect to get
  The experiment plan should not be longer than half a page, and may be much shorter.

Course Calendar

Course calendar may be subject to change as events unfold.