Optimization for Machine Learning (Fall 2026)

Lecture weeks are approximate. Slides will be posted after each lecture.

Date	Content	Slides
Week 1 Aug 17, 19, 21	Course overview and objectives; optimization terminology; review of calculus, linear algebra, and probability.	Introduction Math Review
Week 2 Aug 24, 26, 28	Unconstrained optimization; necessary and sufficient optimality conditions; convex functions and sets; convex optimization.	Optimality Convexity Convex Optimization
Week 3 Aug 31, Sep 2, 4	Descent directions, step sizes, and Armijo backtracking; convergence analysis, stationarity, and conditioning.	Descent Methods Convergence Analysis
Week 4 Sep 9, 11	Gradient descent for strongly convex and convex objectives; convergence rates, iteration complexity, and first-order oracle complexity.	Gradient Descent for Strongly Convex Objectives Gradient Descent for Convex Objectives
Week 5 Sep 16, 18No class Sep 14 (conference)	Nesterov acceleration; convergence and complexity analysis; restarting for strongly convex objectives.	Nesterov Acceleration
Week 6 Sep 21, 23, 25	Constrained optimization: necessary optimality conditions, Lagrangian duality, and KKT conditions.	Constrained Optimization Lagrangian Duality KKT Conditions
Week 7 Sep 28, 30, Oct 2	Slater's condition and sensitivity; dual method and convergence analysis; primal-dual method and convergence analysis.	Slater's Condition and Sensitivity Dual Method and Convergence Primal-Dual Method and Convergence (optional)
Week 8 Oct 7, 9	Empirical risk minimization; generalization bounds, regularization, and cross-validation; in-class midterm.	Empirical Risk Minimization Generalization
Week 9 Oct 12, 14, 16	Stochastic optimization and sample average approximation; stochastic gradient descent, step sizes, and convergence analysis.	Stochastic Optimization Stochastic Gradient
Week 10 Oct 19, 21, 23	Robust stochastic approximation; stochastic variance reduced gradient (SVRG); constrained empirical risk minimization, empirical duality, and dual ascent.	Robust Stochastic Gradient Stochastic Variance Reduced Gradient (SVRG) Constrained Empirical Risk Minimization
Week 11 Oct 26, 28, 30	Neural network optimization.
Week 12 Nov 2, 4, 6	No classes (conference).
Week 13 Nov 9, 11, 13	Optimization for diffusion/flow models.
Week 14 Nov 16, 18, 20	Optimization for large language models.
Week 15 Nov 23	Policy optimization in reinforcement learning.
Week 16 Nov 30	Student presentations.