Interplay Between Reinforcement Learning and Generative Artificial Intelligence (Fall 2025)

Lecture weeks are approximate. Lecture slides will be posted after each lecture.

Week	Content	Lecture slides and recommended reading
Week 1 Aug 18–22	Introduction Course overview; reinforcement learning and generative AI	Slides Interplay between RL and GenAI
Weeks 1–2 Aug 18–29	Policy optimization Policy gradients, natural policy gradients, convergence, and constrained RL	Slides Policy Optimization (I) Policy Optimization (II) Policy Optimization (III) Policy Optimization (IV) Recommended reading On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift Convergence and Sample Complexity of Natural Policy Gradient Primal-Dual Methods for Constrained MDPs
Week 3 Sep 1–5No class Sep 1	Variational inference Variational autoencoders and diffusion models	Slides Variational Inference (I) Variational Inference (II) Variational Inference (III) Recommended reading Understanding Diffusion Models: A Unified Perspective
Weeks 4–5 Sep 8–19	Language model alignment and composition Preference optimization, human feedback, and composition of language models	Slides LLM Alignment (I) LLM Alignment (II) LLM Alignment (III) LLM Alignment (IV) LLM Composition Recommended reading MaxMin-RLHF: Alignment with Diverse Human Preferences Direct Preference Optimization: Your Language Model is Secretly a Reward Model Self-Play Preference Optimization for Language Model Alignment SALSA: Soup-based Alignment Learning for Stronger Adaptation in RLHF Alignment of Large Language Models with Constrained Learning One-Shot Safety Alignment for Large Language Models via Optimal Dualization
Week 6 Sep 22–26	Diffusion model alignment Fine-tuning, preference optimization, and inference-time alignment	Slides DM Alignment (I) DM Alignment (II) DM Alignment (III) Recommended reading Diffusion Model Alignment Using Direct Preference Optimization Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding Composition and Alignment of Diffusion Models using Constrained Learning
Week 7 Sep 29–Oct 3	Diffusion model composition Compositional generation, model merging, and sampling	Slides DM Composition (I) DM Composition (II) DM Composition (III) Recommended reading Diffusion Soup: Model Merging for Text-to-Image Diffusion Models Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC Composition and Alignment of Diffusion Models using Constrained Learning
Week 8 Oct 6–10Fall break Oct 6–7	Unlearning Unlearning in language and diffusion models	Slides Unlearning: DM Unlearning: LLM Recommended reading Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning Erasing Concepts from Diffusion Models Erasing Undesirable Influence in Diffusion Models Large Language Model Unlearning Unlearning in Diffusion Models: A Unified Framework with KL Divergence and Likelihood Constraints
Week 9 Oct 13–17	Diffusion policies Diffusion models for offline and online reinforcement learning	Slides DM for Offline RL DM for Online RL Recommended reading Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning Efficient Diffusion Policies for Offline Reinforcement Learning Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems Efficient Online Reinforcement Learning for Diffusion Policy
Week 10 Oct 20–24	Diffusion models for robot planning Motion planning, implicit priors, and Langevin dynamics	Slides DM for Robotic Planning (I) DM for Robotic Planning (II) DM for Robotic Planning (III) Recommended reading Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem Learning Implicit Priors for Motion Optimization Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models Multi-Robot Motion Planning with Diffusion Models
Week 11 Oct 27–31No class Oct 31	Language models for decision-making World models, reasoning, and language-based reward design	Slides LLMs for RL Recommended reading Language to Rewards for Robotic Skill Synthesis Reasoning with Language Model is Planning with World Model
Weeks 12–13 Nov 3–14	Student paper presentations	Paper list and presentation guidelines
Week 14 Nov 17–21	Course summary Review of the connections between RL and GenAI	Slides Review