COSC 594/650
Overview
Schedule
Homework
Project
Optimization for Machine Learning (Fall 2026)
Lecture weeks are approximate. Slides will be posted after each lecture.
Date
Content
Slides
Week 1
Aug 17, 19, 21
Course overview and objectives; optimization terminology; review of calculus, linear algebra, and probability.
Introduction
Math Review
Week 2
Aug 24, 26, 28
Unconstrained optimization; necessary and sufficient optimality conditions; convex functions and sets; convex optimization.
Optimality
Convexity
Convex Optimization
Week 3
Aug 31, Sep 2, 4
Descent directions, step sizes, and Armijo backtracking; convergence analysis, stationarity, and conditioning.
Descent Methods
Convergence Analysis
Week 4
Sep 9, 11
Gradient descent for strongly convex and convex objectives; convergence rates, iteration complexity, and first-order oracle complexity.
Gradient Descent for Strongly Convex Objectives
Gradient Descent for Convex Objectives
Week 5
Sep 16, 18
No class Sep 14 (conference)
Nesterov acceleration; convergence and complexity analysis; restarting for strongly convex objectives.
Nesterov Acceleration
Week 6
Sep 21, 23, 25
Constrained optimization: necessary optimality conditions, Lagrangian duality, and KKT conditions.
Constrained Optimization
Lagrangian Duality
KKT Conditions
Week 7
Sep 28, 30, Oct 2
Slater's condition and sensitivity; dual method and convergence analysis; primal-dual method and convergence analysis.
Slater's Condition and Sensitivity
Dual Method and Convergence
Primal-Dual Method and Convergence
(optional)
Week 8
Oct 7, 9
Empirical risk minimization; generalization bounds, regularization, and cross-validation; in-class midterm.
Empirical Risk Minimization
Generalization
Week 9
Oct 12, 14, 16
Stochastic optimization and sample average approximation; stochastic gradient descent, step sizes, and convergence analysis.
Stochastic Optimization
Stochastic Gradient
Week 10
Oct 19, 21, 23
Robust stochastic approximation; stochastic variance reduced gradient (SVRG); constrained empirical risk minimization, empirical duality, and dual ascent.
Robust Stochastic Gradient
Stochastic Variance Reduced Gradient (SVRG)
Constrained Empirical Risk Minimization
Week 11
Oct 26, 28, 30
Nonconvex optimization; Polyak–Łojasiewicz (PL) condition.
Nonconvex Optimization
Polyak–Łojasiewicz (PL) Condition
Week 12
Nov 6
No class Nov 2, 4 (conference)
Backpropagation; vanishing and exploding gradients; gradient clipping and adaptive gradient methods (AdaGrad, RMSprop, and Adam).
Backpropagation
Clipping and Adaptive Gradient Methods
Week 13
Nov 9, 11, 13
Variational diffusion models; evidence lower bound (ELBO) and DDPM training; score matching, denoising score matching, and Langevin sampling.
Variational Diffusion Models
Score-Based Optimization
Week 14
Nov 16, 18, 20
Optimization for large language models.
Week 15
Nov 23
Student presentations.
Week 16
Nov 30
Student presentations.