Optimization for Machine Learning (Fall 2026)

Lecture weeks are approximate. Slides will be posted after each lecture.

DateContentSlides
Week 1
Aug 17, 19, 21
Course overview and objectives; optimization terminology; review of calculus, linear algebra, and probability.
Week 2
Aug 24, 26, 28
Unconstrained optimization; necessary and sufficient optimality conditions; convex functions and sets; convex optimization.
Week 3
Aug 31, Sep 2, 4
Descent directions, step sizes, and Armijo backtracking; convergence analysis, stationarity, and conditioning.
Week 4
Sep 9, 11
Gradient descent for strongly convex and convex objectives; convergence rates, iteration complexity, and first-order oracle complexity.
Week 5
Sep 16, 18No class Sep 14 (conference)
Nesterov acceleration; convergence and complexity analysis; restarting for strongly convex objectives.
Week 6
Sep 21, 23, 25
Constrained optimization: necessary optimality conditions, Lagrangian duality, and KKT conditions.
Week 7
Sep 28, 30, Oct 2
Slater's condition and sensitivity; dual method and convergence analysis; primal-dual method and convergence analysis.
Week 8
Oct 7, 9
Empirical risk minimization; generalization bounds, regularization, and cross-validation; in-class midterm.
Week 9
Oct 12, 14, 16
Stochastic optimization and sample average approximation; stochastic gradient descent, step sizes, and convergence analysis.
Week 10
Oct 19, 21, 23
Robust stochastic approximation; stochastic variance reduced gradient (SVRG); constrained empirical risk minimization, empirical duality, and dual ascent.
Week 11
Oct 26, 28, 30
Nonconvex optimization; Polyak–Łojasiewicz (PL) condition.
Week 12
Nov 6No class Nov 2, 4 (conference)
Backpropagation; vanishing and exploding gradients; gradient clipping and adaptive gradient methods (AdaGrad, RMSprop, and Adam).
Week 13
Nov 9, 11, 13
Variational diffusion models; evidence lower bound (ELBO) and DDPM training; score matching, denoising score matching, and Langevin sampling.
Week 14
Nov 16, 18, 20
Optimization for large language models.
Week 15
Nov 23
Student presentations.
Week 16
Nov 30
Student presentations.