Term  Spring 2024  Instructor  Christopher De Sa 
Room  Phillips Hall 101  [email hidden]  
Schedule  MW 7:30pm – 8:45pm  Office hours  W 2:30pm – 3:30pm 
Forum  Ed Discussion  Office  Gates 426 
So you've taken a machine learning class. You know the models people use to solve their problems. You know the algorithms they use for learning. You know how to evaluate the quality of their solutions.
But when we look at a largescale machine learning application that is deployed in practice, it's not always exactly what you learned in class. Sure, the basic models, the basic algorithms are all there. But they're modified a bit, in a bunch of different ways, to run faster and more efficiently. And these modifications are really important—they often are what make the system tractable to run on the data it needs to process.
CS6787 is a graduatelevel introduction to these systemfocused aspects of machine learning, covering guiding principles and commonly used techniques for scaling up learning to large data sets. Informally, we will cover the techniques that lie between a standard machine learning course and an efficient systems implementation: both statistical/optimization techniques based on improving the convergence rate of learning algorithms and techniques that improve performance by leveraging the capabilities of the underlying hardware. Topics will include stochastic gradient descent, acceleration, variance reduction, methods for choosing hyperparameters, parallelization within a chip and across a cluster, popular ML frameworks, and innovations in hardware architectures. An openended project in which students apply these techniques is a major part of the course.
Prerequisites: Knowledge of machine learning at the level of CS4780. If you are an undergraduate, you should have taken CS4780 or an equivalent course, since it is a prerequisite. Knowledge of computer systems and hardware on the level of CS 3410 is recommended, but this is not a prerequisite.
Format: About half of the classes will involve traditionally formatted lectures. For the other half of the classes, we will read and discuss two seminal papers relevant to the course topic. These classes will involve presentations by groups of students of the paper contents (each student will sign up in a group to present one paper for 1520 minutes) followed by breakout discussions about the material. Historically, the lectures have occurred on Mondays and the discussions have occurred on Wednesdays, but due to the nonstandard timeline this semester, these course elements will be scheduled irregularly (see schedule below).
Grading: Students will be evaluated on the following basis.
20%  Paper presentation 
10%  Discussion participation 
20%  Paper reviews 
10%  Programming assignments 
40%  Final project 
Paper review parameters: Paper reviews should be about one page (singlespaced) in length. The review guidelines should mirror what an actual conference review would look like (although you needn't assign scores or anything like that). In particular you should at least: (1) summarize the paper, (2) discuss the paper's strengths and weaknesses, and (3) discuss the paper's impact. For reference, you can read the ICML reviewer guidelines. Of course, your review will not be precisely like a real review, in large part because we already know the impact of these papers. You can submit any review up to two days late with no penalty. Students who presented a paper do not have to submit a review of that paper (although you can if you want).
Final project parameters (subject to change): The final project can be done in groups of up to three (although more work will be expected from groups with more people). The subject of the project is openended, but it must include:
Monday, January 22 In Person Jan 21Jan 22Jan 23Jan 24Jan 25Jan 26Jan 27  Lecture #1: Overview. [Slides] [Demo Notebook]

Wednesday, January 24 In Person Jan 21Jan 22Jan 23Jan 24Jan 25Jan 26Jan 27  Lecture #2: Backpropagation & ML Frameworks. [Slides] [Demo Notebook]

Monday, January 29 In Person Jan 28Jan 29Jan 30Jan 31Feb 1Feb 2Feb 3  Lecture #3: Hyperparameters and Tradeoffs. [Slides] [Demo Notebook]

Wednesday, January 31 In Person Jan 28Jan 29Jan 30Jan 31Feb 1Feb 2Feb 3  Paper Discussion 1a. Attention is all you need Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. In Advances in neural information processing systems (NeurIPS), 2017. Paper Discussion 1b. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Sergey Ioffe, Christian Szegedy. Proceedings of the International Conference on Machine Learning (ICML), 2015. 
Monday, February 5 In Person Feb 4Feb 5Feb 6Feb 7Feb 8Feb 9Feb 10  Lecture #4: Kernels and Dimensionality Reduction. [Slides] [Demo Notebook]

Wednesday, February 7 In Person Feb 4Feb 5Feb 6Feb 7Feb 8Feb 9Feb 10  Paper Discussion 2a. Palm: Scaling language modeling with pathways. Aakanksha Chowdhery Journal of Machine Learning Research (JMLR), 2023. Paper Discussion 2b. Language models are fewshot learners. Tom Brown In Advances in neural information processing systems (NeurIPS), 2020. Due: Review of paper 1a or 1b. 
Monday, February 12 In Person Feb 11Feb 12Feb 13Feb 14Feb 15Feb 16Feb 17  Lecture #5: Adaptive Methods & NonConvex Optimization. [Slides] [Demo Notebook]

Wednesday, February 14 In Person Feb 11Feb 12Feb 13Feb 14Feb 15Feb 16Feb 17  Paper Discussion 3a. Random features for largescale kernel machines. Ali Rahimi and Benjamin Recht. In Advances in Neural Information Processing Systems (NeurIPS), 2007. Paper Discussion 3b. Feature Hashing for Large Scale Multitask Learning. Kilian Weinberger, Anirban Dasgupta, Josh Attenberg, John Langford and Alex Smola. Proceedings of the International Conference on Machine Learning (ICML), 2009. Released: Programming Assignment 2. 
Monday, February 19 Online Only Feb 18Feb 19Feb 20Feb 21Feb 22Feb 23Feb 24  Lecture #6: Hyperparameter Optimization. [Slides] [Demo Notebook]

Wednesday, February 21 In Person Feb 18Feb 19Feb 20Feb 21Feb 22Feb 23Feb 24  Paper Discussion 4a. Random shuffling beats sgd after finite epochs. Jeff Haochen and Suvrit Sra. Proceedings of the International Conference on Machine Learning (ICML), 2019. Paper Discussion 4b. Adam: A method for stochastic optimization. Diederik Kingma and Jimmy Ba. Proceedings of the International Conference on Learning Representations (ICLR), 2015. Due: Review of paper 3a or 3b. 
Monday, February 26  February Break: No classes. 
Wednesday, February 28 In Person Feb 25Feb 26Feb 27Feb 28Feb 29Mar 1Mar 2  Paper Discussion 5a. Random search for hyperparameter optimization. James Bergstra and Yoshua Bengio. Journal of Machine Learning Research (JMLR), 2012. Paper Discussion 5b. Scaling laws for neural language models. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. arXiv preprint arXiv:2001.08361, 2020. 
Monday, March 4 In Person Mar 3Mar 4Mar 5Mar 6Mar 7Mar 8Mar 9  Lecture #7: Parallelism. [Slides] [Demo Notebook]

Wednesday, March 6 In Person Mar 3Mar 4Mar 5Mar 6Mar 7Mar 8Mar 9  Paper Discussion 6a. Mapreduce for machine learning on multicore. ChengTao Chu, Sang K Kim, YiAn Lin, YuanYuan Yu, Gary Bradski, Andrew Y. Ng, and Kunle Olukotun In Advances in Neural Information Processing Systems (NeurIPS), 2007. Paper Discussion 6b. Hogwild: A lockfree approach to parallelizing stochastic gradient descent. Feng Niu, Benjamin Recht, Christopher Re, and Stephen Wright. In Advances in Neural Information Processing Systems (NeurIPS), 2011. 
Monday, March 11 In Person Mar 10Mar 11Mar 12Mar 13Mar 14Mar 15Mar 16  Lecture #8: Distributed Learning. [Slides]
Due: Review of paper 5a or 5b. 
Wednesday, March 13 In Person Mar 10Mar 11Mar 12Mar 13Mar 14Mar 15Mar 16  Paper Discussion 7a. Flashattention: Fast and memoryefficient exact attention with ioawareness. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. In Advances in Neural Information Processing Systems (NeurIPS), 2022. Paper Discussion 7b. A System for Massively Parallel Hyperparameter Tuning. Liam Li Proceedings of the 2nd Conference on Machine Learning and Systems (MLSys), 2020. 
Monday, March 18 In Person Mar 17Mar 18Mar 19Mar 20Mar 21Mar 22Mar 23  Lecture #9: LowPrecision Arithmetic. [Slides]
Due: Review of paper 6a or 6b. Inclass project feedback activity. 
Wednesday, March 20 In Person Mar 17Mar 18Mar 19Mar 20Mar 21Mar 22Mar 23  Paper Discussion 8a. Large scale distributed deep networks. Jeff Dean In Advances in Neural Information Processing Systems (NeurIPS), 2012. Paper Discussion 8b. Towards federated learning at scale: System design. Keith Bonawitz In Proceedings of the 2nd MLSys Conference (MLSys), 2019. 
Monday, March 25 In Person Mar 24Mar 25Mar 26Mar 27Mar 28Mar 29Mar 30  Lecture #10: Inference and Compression. [Demo Notebook]
Due: Review of paper 7a or 7b. Due: Final project proposals. 
Wednesday, March 27 In Person Mar 24Mar 25Mar 26Mar 27Mar 28Mar 29Mar 30  Paper Discussion 9a. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, and Yonghui Wu. In Advances in Neural Information Processing Systems (NeurIPS), 2019. Paper Discussion 9b. Efficiently scaling transformer inference. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. In Proceedings of Machine Learning and Systems (MLSys), 2023. 
Monday, April 1  Spring Break: No classes. 
Wednesday, April 3  Spring Break: No classes. 
Monday, April 8 In Person Apr 7Apr 8Apr 9Apr 10Apr 11Apr 12Apr 13  Lecture #11: Machine Learning Frameworks II.
Due: Review of paper 8a or 8b. 
Wednesday, April 10 In Person Apr 7Apr 8Apr 9Apr 10Apr 11Apr 12Apr 13  Paper Discussion 10a. Deep learning with limited numerical precision. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Proceedings of the International Conference on Machine Learning (ICML), 2015. Paper Discussion 10b. LoRA: LowRank Adaptation of Large Language Models. Edward J. Hu, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Proceedings of the International Conference on Learning Representations (ICLR), 2021. 
Monday, April 15 In Person Apr 14Apr 15Apr 16Apr 17Apr 18Apr 19Apr 20  Lecture #12: Hardware for Machine Learning.
Due: Review of paper 9a or 9b. 
Wednesday, April 17 In Person Apr 14Apr 15Apr 16Apr 17Apr 18Apr 19Apr 20  Paper Discussion 11a. Deep compression: Compressing deep neural networks with pruning, trained quantization and Huffman coding. Song Han, Huizi Mao, and William J Dally. Proceedings of the International Conference on Learning Representations (ICLR), 2016. Paper Discussion 11b. GPTQ: Accurate posttraining quantization for generative pretrained transformers. Frantar, Elias, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Proceedings of the International Conference on Learning Representations (ICLR), 2023. 
Monday, April 22 In Person Apr 21Apr 22Apr 23Apr 24Apr 25Apr 26Apr 27  Lecture #13: Modern Generative AI.
Due: Review of paper 10a or 10b. 
Wednesday, April 24 Online Only Apr 21Apr 22Apr 23Apr 24Apr 25Apr 26Apr 27  Paper Discussion 12a. Indatacenter performance analysis of a tensor processing unit. Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In Proceedings of the 44th Annual International Symposium on Computer Architecture (ISCA), 2017. Paper Discussion 12b. A Configurable CloudScale DNN Processor for RealTime AI. Jeremy Fowers, Kalin Ovtcharov, Michael Papamichael, Todd Massengills, et al. In Proceedings of the 45th Annual International Symposium on Computer Architecture (ISCA), 2018. 
Monday, April 29 In Person Apr 28Apr 29Apr 30May 1May 2May 3May 4  Lecture #14: Large Scale ML on the Cloud. [Slides]
Due: Review of paper 11a or 11b. Due: Final project abstract draft. Can be submitted late until Wednesday afternooon; will discuss in class on Wednesday. 
Wednesday, May 1 In Person Apr 28Apr 29Apr 30May 1May 2May 3May 4  Lecture #15: Final Project Disussion. 
Monday, May 6 In Person May 5May 6May 7May 8May 9May 10May 11  Lecture #16: Final Project Disussion. 