References

What these notes reconstruct

Ben Moll, Reinforcement Learning for Economists, lecture slides, London School of Economics. https://benjaminmoll.com/lectures/

: The primary source. Four lectures, one per chapter here.

Shiyu Zhao, Mathematical Foundations of Reinforcement Learning, Springer, 2024. https://github.com/MathFoundationRL/Book-Mathematical-Foundation-of-Reinforcement-Learning

: Moll’s slides state twice that they are “mostly a shortened/adapted version” of Zhao’s, so this book sits behind the lectures throughout. It is also where the slides send the reader whenever a proof is deferred, and therefore the source consulted for every argument this book fills in: Dvoretzky’s theorem and its multi-variable form (Chapter 6 there), the convergence of TD, Sarsa, and Q-learning and the action-value Bellman equations (Chapter 7), and the policy gradient theorem (Chapter 9). The full book and individual chapter PDFs are free on the repository above, which also carries the grid-world code the lectures point at.

Cited along the way

Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book.html

: The standard reference. Cited here for the constant-\(\alpha\) Monte Carlo comparison in Chapter 1, for the incremental-versus-episodic distinction in Chapter 3 (where this book departs from its “online/offline” vocabulary, since “offline RL” now means something else), and for the REINFORCE pseudocode in Chapter 4.

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller, “Deterministic Policy Gradient Algorithms”, ICML, 2014.

The deterministic policy gradient theorem of Chapter 4.

Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour, “Policy Gradient Methods for Reinforcement Learning with Function Approximation”, NeurIPS, 2000.

The stochastic policy gradient theorem of Chapter 4.

Kevin P. Murphy, Reinforcement Learning: An Overview, 2025. https://arxiv.org/abs/2412.05265

: A broad and current survey. Less accessible than Zhao as a first pass, and cited in Chapter 4 for the framing of actor-critic methods.

Further reading the lectures recommend

Lilian Weng, “Policy Gradient Algorithms”, 2018. https://lilianweng.github.io/posts/2018-04-08-policy-gradient/

: A clear map of the policy gradient family beyond what Chapter 4 covers, including the algorithms this book stops short of.

Andrej Karpathy, “Deep Dive into LLMs like ChatGPT”, 2025. https://www.youtube.com/watch?v=7xTGNNLPyMI

: For where Chapter 4 actually gets used: RLHF, RLVR, and reasoning models.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, “Proximal Policy Optimization Algorithms”, 2017. https://arxiv.org/abs/1707.06347

: The standard practical policy gradient algorithm, and the natural next step from the on-policy constraint that closes Chapter 4.

Everything this book stops short of

The reconstruction follows the lectures and does not go past them. Readers wanting the topics the four lectures skip should go to Zhao: \(n\)-step Sarsa and TD(\(\lambda\)), off-policy learning with importance sampling, Deep Q-learning with experience replay and target networks, the convergence theory of function approximation, and the wider actor-critic family (A2C, DPG, TRPO, PPO).