Reinforcement Learning
Lecture Notes: Math and Applications
Self-contained lecture notes on reinforcement learning for economists, developed from Ben Moll’s four lectures. The organising idea is that reinforcement learning is Monte-Carlo dynamic programming: the same Bellman recursions, with expectations estimated from sampled experience rather than computed from a known model. Covers dynamic programming and the link to RL, Monte Carlo methods and stochastic approximation, temporal difference learning with Sarsa and Q-learning, and policy gradient and actor-critic methods, with the proofs the slides defer worked out in full.
Preface
\[ \newcommand{\E}{\mathbb{E}} \newcommand{\R}{\mathbb{R}} \newcommand{\Prob}{\mathbb{P}} \newcommand{\BR}{\operatorname{BR}} \newcommand{\eps}{\varepsilon} \newcommand{\given}{\,\vert\,} \newcommand{\argmax}{\operatorname*{arg\,max}} \newcommand{\argmin}{\operatorname*{arg\,min}} \newcommand{\sm}{\setminus} \newcommand{\defeq}{\equiv} \]
This note is based on Ben’s latest lectures on RL, mainly focused on the math behind it w/ also some applications.
Written in Obsidian with LaTeX Suite for the math, rendered as a Quarto book for the web version.
These notes are a study reconstruction, not official course materials. Any errors introduced in the rebuilding are mine.
Sources
One chapter per lecture, following each lecture’s scope and ordering:
| Chapter | Lecture |
|---|---|
| 1. The Basic Idea of RL, and Its Link to Dynamic Programming | Lecture 1 |
| 2. Monte Carlo Methods and Stochastic Approximation | Lecture 2 |
| 3. Temporal Difference Learning, Sarsa, and Q-learning | Lecture 3 |
| 4. Policy Gradient and Actor-Critic Methods | Lecture 4 |
Where the slides state a result and hand its proof off to the references, the proof is worked out and written up here rather than cited. Those arguments follow Shiyu Zhao’s Mathematical Foundations of Reinforcement Learning, which is also the book Ben Moll’s slides are themselves adapted from. Full details, and the rest of what the lectures cite, are in the References appendix. The Notation appendix collects the symbols, including the one place where the book deliberately changes convention.
Every algorithm is given twice: once as a statement, and once as the handful of lines of Python that produced this book’s figures, folded away under the algorithm it implements.
License & attribution
The exposition and all figures here are my own work, released under CC BY-NC-SA 4.0: share and adapt for non-commercial purposes with attribution and share-alike. The underlying material is not mine. These notes reconstruct Ben Moll’s reinforcement learning lectures, which are in turn adapted from Shiyu Zhao’s Mathematical Foundations of Reinforcement Learning. Everything here is rewritten in my own words with original figures: no text, figures, or numbering from the lecture slides or from Zhao’s book is reproduced, and all rights in those original materials remain with their authors. This is a non-commercial educational study aid, not endorsed by or affiliated with Ben Moll, Shiyu Zhao, LSE, or Westlake University. If you are a rights holder with any concern, contact me and I will address it promptly.