Optimization for Machine Learning (Fall 2026)

Course Description

This course introduces the optimization foundations for modern machine learning (or machine decision-making/artificial intelligence). Topics include first-order, stochastic, constrained, non-smooth, minimax, and policy optimization, with applications to generative modeling, and reinforcement learning. The course emphasizes how operational, safety, robustness, and alignment requirements are translated into optimization problems. Students will read, reproduce, and critique recent optimization research for machine learning, culminating in a conference-format research paper and reproducible software artifact.

Learning Outcomes

By the end, students should be able to:

• Formulate machine decision-making problems using mathematical optimization language.

• Derive optimality conditions and use convexity, smoothness, duality, and constraint qualifications correctly.

• Analyze convergence and complexity of representative optimization algorithms.

• Implement and diagnose optimization algorithms using objective, stationarity, and constraint-violation metrics.

• Read research papers critically, reproduce central results, and identify defensible research gaps.

• Develop theoretical or empirical research that validates falsifiable claims.

• Produce conference-format manuscripts, reproducible repositories, peer reviews, and rebuttal.

• Use AI tools transparently while independently verifying mathematical claims, code, experiments, and citations.

Prerequisites

Students are expected to have a foundation in linear algebra, probability, and calculus, as well as basic knowledge of Python. Measure-theoretic probability is not required.

Required Materials

There is no required textbook. Lecture notes and problem sets are available on this site and on Canvas. The following are recommended references:

• Stephen Boyd and Lieven Vandenberghe, Convex Optimization, Cambridge University Press, 2004.

• Sebastien Bubeck, Convex Optimization: Algorithms and Complexity, Foundations and Trends in Machine Learning, 2015.

• Shai Shalev-Shwartz and Shai Ben-David, Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, 2014.

Requirements

• 5% of the grade: Class participation, including questions, discussions, and feedback.

• 10% of the grade: Lecture scribing. Notes are due one week after the assigned lecture and are posted on Canvas.

• 20% of the grade: Five homework assignments, each worth 4%.

• 20% of the grade: In-class midterm.

• 45% of the grade: Project, including proposal and research question (10%), technical checkpoint (5%), presentation (10%), peer review and rebuttal (5%), and final report (15%).

Teaching Assistant

Anik Saha (asaha15@vols.utk.edu). Office: Min Kao Building 349. Office hours: Thursday, 12:00 pm - 1:00 pm.

Communications

Use Canvas for homework submissions and course-related discussions. For help with course material, email the instructor or teaching assistant, or attend their office hours.

Use of Generative AI

AI tools may assist with learning and research, but students must independently understand and verify all submitted work. Submissions must include an AI Use Statement documenting any assistance, or state that no AI tools were used. AI tools are prohibited during the in-class midterm unless explicitly authorized in writing. See the syllabus for the full policy and assignment-specific requirements.