Optimization for Machine Learning (Fall 2026)

Lecture weeks are approximate. Slides will be posted after each lecture.

Date	Content	Slides
Week 1 Aug 17, 19, 21	Course overview and objectives; optimization terminology; review of calculus, linear algebra, and probability.	Introduction Math Review
Week 2 Aug 24, 26, 28	Unconstrained optimization; necessary and sufficient optimality conditions; convex functions and sets; convex optimization.	Optimality Convexity Convex Optimization
Week 3 Aug 31, Sep 2, 4	Descent directions, step sizes, and Armijo backtracking; convergence analysis, stationarity, and conditioning.	Descent Methods Convergence Analysis
Week 4 Sep 9, 11	Gradient descent for strongly convex and convex objectives; convergence rates, iteration complexity, and first-order oracle complexity.	Gradient Descent for Strongly Convex Objectives Gradient Descent for Convex Objectives
Week 5 Sep 16, 18No class Sep 14 (conference)	Nesterov acceleration; convergence and complexity analysis; restarting for strongly convex objectives.	Nesterov Acceleration
Week 6 Sep 21, 23, 25	Constrained optimization: necessary optimality conditions, Lagrangian duality, and KKT conditions.	Constrained Optimization Lagrangian Duality KKT Conditions
Week 7 Sep 28, 30, Oct 2	Slater's condition and sensitivity; dual method and convergence analysis; primal-dual method and convergence analysis.	Slater's Condition and Sensitivity Dual Method and Convergence Primal-Dual Method and Convergence (optional)
Week 8 Oct 7, 9	Stochastic optimization: stochastic gradient descent; mini-batching, variance reduction, and adaptive methods; in-class midterm.
Week 9 Oct 12, 14, 16	Nonsmooth optimization: proximal, projected, and mirror methods.
Week 10 Oct 19, 21, 23	Nonconvex optimization: stationarity, PL conditions, and escaping saddle points.
Week 11 Oct 26, 28, 30	Neural network optimization.
Week 12 Nov 2, 4, 6	No classes (conference).
Week 13 Nov 9, 11, 13	Optimization for diffusion/flow models.
Week 14 Nov 16, 18, 20	Optimization for large language models.
Week 15 Nov 23	Policy optimization in reinforcement learning.
Week 16 Nov 30	Student presentations.