CS 4220/Math 4260: Numerical Analysis: Linear and Nonlinear Problems

Cornell University, Spring 2021.
Instruction mode: online, synchronous lectures.
Lectures: 3:45PM–4:35PM (ET) M/W/F, virtual on zoom.

Instructor: Austin Benson, Assistant Professor, Computer Science (arb@cs.cornell.edu).
Office hours: 2:00PM–3:00PM (ET) Wednesdays or by appointment, virtual on zoom.

TA: Michael Xu, PhD student, Center for Applied Mathematics (xx294@cornell.edu).
Office hours: 4:00PM–5:00PM (ET) Thursdays, virtual on zoom.

Course support staff: Jordan Staiti, Computer Science (jas2262@cornell.edu).


Course topics: direct and iterative algorithms for solving linear systems; least squares; eigenvalue problems; matrix factorizations such as QR, LU, and the SVD; numerical optimization basics such as gradient descent and (quasi-)Newton.

Feedback: We welcome feedback on the class during the semester. You should always feel free to email the professor and TA. If you want to provide anonymous (to Prof. Benson and TA Michael) feedback, please send it via email to Jordan, and the information will be communicated.

You should expect and demand to be treated by your classmates and the course staff with respect. You belong here, and we are here to help you learn and enjoy this course. If any incident occurs that challenges this commitment to a supportive and inclusive environment, please let the instructors know so that the issue can be addressed. We are personally committed to this, and subscribe to the Computer Science Department’s Values of Inclusion.
(This statement is based on a similar one from Dan Grossman at Washington.)

Course administration: Recommended reading material:
  • [B] Prof. David Bindel's CS4220 course notes from Spring 2020.
  • [AG] A First Course in Numerical Methods. U. Ascher and C. Greif. Available online.
  • [TB] Numerical Linear Algebra. L. N. Trefethen and D. Bau III. Available online (no direct link, need to search).
  • [NW] Numerical Optimization. J. Nocedal and S. J. Wright. Available online.
Other useful material:
Note: this schedule is tentative and subject to change.

Week 1
Lecture 1 (M 02/08): course intro and basic linear algebra review
Readings: B Jan 22, Jan 24, Jan 27. [notes] [video]

Lecture 2 (W 02/10): multiplying matrices and vectors on computers; structured matrices
Readings: B Jan 22, Jan 24, Jan 27. [notes] [video] [demo]

Lecture 3 (F 02/12): norms
Readings: B Jan 22, Jan 24, Jan 27. [notes] [video]

Week 2
Lecture 4 (M 02/15): the SVD
Readings: TB 4–5. [notes] [video] [demo]

Lecture 5 (W 02/17): sensitivity and conditioning
Readings: B Jan 29. [notes] [video]

Lecture 6 (F 02/19): floating point and backward stability
Readings: B Jan 31. [notes] [video] [demo]

Week 3
Lecture 7 (M 02/22): Gaussian elimination and LU
Readings: B Feb 5. [notes] [video]

Lecture 8 (W 02/24): pivoted LU
Readings: B Feb 10, 12; TB 21. [notes] [video]

Lecture 9 (F 02/26): LU error analysis (Michael Xu lecturing)
Readings: B Feb 10, 12. [notes] [video]

Week 4
Lecture 10 (M 03/01): SPD and sparse LU
Readings: B Feb 14. [notes] [video]

Lecture 11 (W 03/03): introduction to least squares
Readings: B Feb 17. [notes] [video]

Lecture 12 (F 03/05): computing QR factorizations
Readings: B Feb 19. [notes] [video] [demo]

Week 5
Lecture 13 (M 03/08): computing QR factorizations and stability; least squares sensitivity and conditioning
Readings: B Feb 17, 19, 21. [notes] [video]

No lecture W 03/10 (wellness day)

No lecture F 03/12 (instructor "travel")

Week 6
Lecture 14 (M 03/15): least squares and regularization
Readings: B Feb 21. [notes] [video]

Lecture 15 (W 03/17): introduction to eigenvalue problems
Readings: B Feb 28. [notes] [video]

Lecture 16 (F 03/19): power iteration and subspace iteration
Readings: B Mar 3. [notes] [video]

Week 7
Lecture 17 (M 03/22): QR iteration
Readings: B Mar 4. [notes] [video] [demo]

Lecture 18 (W 03/24): practicalities and other considerations
Readings: B Mar 4. [notes] [video]

Lecture 19 (F 03/26): introduction to numerical optimization in one dimension
Readings: B Mar 9, Apr 6. [notes] [video]

Week 8
Lecture 20 (M 03/29): 1-D optimization and root finding
Readings: B Mar 9, 11. [notes] [video] [demo]

Lecture 21 (W 03/31): introduction to multivariate numerical optimization
Readings: B Apr 13. [notes] [video]

Lecture 22 (F 04/02): Newton's method in many dimensions
Readings: B Apr 15. [notes]

Week 8
Lecture 23 (M 04/05): quasi-Newton methods
Readings: B Apr 20. [notes] [video]

Lecture 24 (W 04/07): gradient methods
Readings: B Apr 22. [notes] [video]

Lecture 25 (F 04/09): line search
Readings: B Apr 24. [notes] [video] [demo]

Coursework will be managed through, and assignments submitted on, CMS. The required coursework consists of four components:
  • Homeworks (48%).
    There will be six homeworks that will have theoretical and/or coding components. Each is worth 8% of your grade. Submissions need to be typeset with latex.
  • Midterm exam (16%).
    There will be a take-home midterm, released on March 29 and due April 5 before class starts. The format is just like the homework, except that you have to work on the problems on your own.
  • Final exam (32%).
    There will be a take-home final, with a format similar to previous offerings of 4220.
  • Participation (4%).
    Showing up to virtual lectures, asking questions, coming to office hours, providing feedback, Ed Discussion involvement, and other engagement with the course will count towards participation.