Advanced Reinforcement Learning

Aims

The aim of this module is to present state-of-the-art reinforcement learning (RL) methods, incentivise students to understand RL theory and develop skills for coding deep RL methods, and encourage students to pair up and push beyond state-of-the-art in a challenging mini-project.

RL has seen unprecedented success in recent years. However, the majority of RL methods still require intricate skills and insights for successful applications. The goal of this module is to communicate the promising aspects of RL, but also ensure that students understand the limitations of the current RL methods.

Objectives and Learning Goals

Students will learn the following technical concepts:

- fundamental RL terminology and mathematical formalism; a brief history of RL and its connection to neuroscience and biological systems

- RL methods for discrete action spaces, e.g. deep Q-learning and large-scale Monte Carlo Tree Search

- methods for exploration, modelling uncertainty, and partial observability for RL

- modern policy gradient and actor-critic methods

- concepts needed to construct model-based RL and Model Predictive Control methods

- approaches to make RL data-efficient and ways to enable simulation-to-reality transfer

- examples of fine-tuning foundation models and large language models (LLMs) with human feedback; safe RL concepts; examples of using RL for safety validation

- examples of using RL for scientific discovery

Students will also gain practical experience with coding and analysing RL methods to uncover their strengths and shortcomings, as well as proposing novel extensions to improve the performance of existing RL methods in a short mini-project.

Syllabus

Topic 1: Introduction & RL in Discrete Action Spaces

- A brief overview of RL: foundational ideas, history, and books

- Q-learning, function approximation and deep Q-learning; nonstationarity in RL and its implications for deep learning; example applications (video games; initial example: Atari)

Reading: S&B Chapters 3, 4, 6, 9.1-9.5

Topic 2: Mathematical Fundamentals

- Markov decision processes, Bellman equations

- Policy and value iteration, temporal difference learning, derivation of Q learning

Reading: CsSz Chapters 1, 2, 3, 4.3, A1, A2

Topic 3: Policy Gradient and Actor-critic Methods for Continuous Action Spaces

Due: Coding exercise 1 due at the start of the class meeting (take-home, not marked, serves as a checkpoint to ensure students are preparing for the test).

Participation: students can earn participation points by explaining their solutions during an active in-class discussion.

- Policy gradient theorem, actor-critic methods

- Proximal policy optimisation; example applications

Reading: Chapter 13: Policy Gradient Methods (skip 13.6); CS Chapter 4.4.; Topic 3 papers.

Topic 4: Exploration, Uncertainty, Data-efficient RL & Sim-to-real Transfer

Due: Coding exercise 2 due at the start of the class meeting (take-home, not marked, instead serves as a checkpoint to ensure students are preparing for the test).

Participation: students can earn participation points by explaining their solutions during an active in-class discussion.

- Multi-armed bandits, Bayesian optimisation, regret analysis

- Data-efficient learning from real data (e.g. policy search in robotics), real-to-sim inference and differentiable simulation, data-efficient simulation-to-reality transfer

- RL for physical systems (successful examples in locomotion, open problems in contact-rich robot manipulation)

Reading: S&B Chapter 2; CS Chapter 4.2; Topic 4 papers.

Topic 5: Connections to Neuroscience; Reproducibility & hyperparameter optimization; Partial Observability & Sequence Modelling (on another topic for an invited lecture)

Test: Theory & coding test at the class meeting start (50 minutes, in-class, closed-book)

- Connection to RL in neuroscience and biological systems

- Reproducibility & hyperparameter optimization

- Partial Observability, Memory, and Sequence Modelling

Reading: Topic 5 papers.

Topic 6: Model-based RL and Model Predictive Control

- Monte Carlo Tree Search; example applications (AlphaGo)

- Learning dynamics models; planning with learned models

- Model Predictive Control

Reading: S&B Chapter 8.1-8.3; Topic 6 papers.

Topic 7: RL with Human Feedback; RL for Scientific Discovery

- Fine-tuning large language models (LLMs) and other foundation models with human feedback (TRLX, RL4LMs, a light-weight overview of RLHF)

- Examples of RL for molecular design and drug discovery, active learning for synthesising new materials, RL for theorem proving, and RL for nuclear fusion experiments

Reading: Topic 7 papers.

Topic 8: Student Presentations

- Students will give mini-project presentations: a figure for an algorithm overview, and at least two plots with main results as they will appear in the mini-project report. (The report is due several days after the last class meeting, but the figures should be the same as in this in-class presentation).

Assessment

The module will have a test of RL theory & coding skills for RL methods (30%), a mini-project (60%), and a presentation/participation (10%) assessment. Students will also be required to complete two brief practical exercises in preparation for the test. These will not be marked, instead, they will serve as checkpoints to ensure students are preparing for the test. We will discuss the unmarked coding exercises during class, with opportunities for students to earn participation points.

Test (30%): students will be required to take a 50-minute, in-class, closed-book test that will test the understanding of RL fundamentals and ability to comprehend implementation of the foundational and state-of-the-art RL methods.

Mini-project (60%): This mini-project will focus on improving RL under resource constraints. It will build on the take-home exercises, but for the project the students will be challenged to push beyond the state of the art: achieve high reward performance while reducing compute resources & time to achieve high rewards. Students can apply innovations they learn from lectures and assigned readings, seek further insights from other sources, and then show research potential by adding novel aspects to the resulting RL algorithms.

Students will work in pairs, but when proposing the plan for their project, they will also need to clarify individual responsibilities, so that their work can be assessed individually.

Students will need to submit a short report that describes the proposed method and shows key plots that present the results. Each pair should submit one report (maximum 2000 words total; maximum 1000 words per student), at least one algorithm summary figure, and at least two figures with main results. Each paragraph and plot should be marked with the name of the student who contributed, so that the contributions of each student can be clearly separated for marking.

Participation (10%): Students will be expected to attend class sessions in person, contribute at least one substantial explanation/comment to the in-class discussions about unmarked take-home coding exercises, and briefly present the results of their mini-project. This component will help to ensure that we have lively and active class meetings, where students can practice articulating their knowledge.

Recommended Reading

Books for Required Reading   

[S&B] Reinforcement Learning: An Introduction (second print edition). Richard S. Sutton, Andrew G. Barto. [Available from the book’s website as a free PDF updated in 2022]

[CZ] Algorithms for Reinforcement Learning. Csaba Szepesvari. [Available from the book’s website as a free PDF updated in 2019]. Note that the 2022 reprint has a different chapter numbering (but same content as the 1st edition).

Papers for Required Reading:

Topic 1:

Human-level control through deep reinforcement learning. Mnih, et al. Nature, v 518, pp 529-533, 2015.

Topic 3:

Policy Gradient Methods for Reinforcement Learning with Function Approximation. Sutton, et al. Advances in Neural Information Processing Systems (NeurIPS), 2000.

Proximal Policy Optimization Algorithms. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, Oleg Klimov. arXiv:1707.06347, 2017.

Topic 4:

A Survey on Policy Search Algorithms for Learning Robot Controllers in a Handful of Trials. Konstantinos Chatzilygeroudis, Vassilis Vassiliades, Freek Stulp, Sylvain Calinon, Jean-Baptiste Mouret. IEEE Transactions on Robotics, 2020.

Learning Dexterous In-hand Manipulation. OpenAI: Andrychowicz, et al. The International Journal of Robotics Research, 39(1):3-20, 2020.

Optional:

Recent Advances in Bayesian Optimization. Xilu Wang, Yaochu Jin, Sebastian Schmitt, Markus Olhofer. ACM Computing Surveys, v 55-13s-287, pp 1-36, 2023.

Topics 5:

Dopamine neurons report an error in the temporal prediction of reward during learning. Jeffrey R. Hollerman & Wolfram Schultz. Nature Neuroscience, v 1, pp 304–309, 1998.

States versus Rewards: Dissociable Neural Prediction Error Signals Underlying Model-Based and Model-Free Reinforcement Learning. Jan Gläscher, Nathaniel Daw, Peter Dayan, John P. O'Doherty. Neuron 27;66(4):585-95, 2010.

Deep Reinforcement Learning that Matters. Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. AAAI Conference on Artificial Intelligence, 2018.

Topics 6:

Mastering the Game of Go with Deep Neural Networks and Tree Search. Silver et al. Nature, v 529, pp 484-489, 2016.

Optional:

Selective Dyna-style Planning Under Limited Model Capacity. Zaheer Abbas, Samuel Sokota, Erin J. Talvitie, Martha White. International Conference on Machine Learning (ICML), 2020.

Topics 7:

Attention is All you Need. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, Illia Polosukhin. Advances in Neural Information Processing Systems (NeurIPS), 2017.

Illustrating Reinforcement Learning from Human Feedback (RLHF). Lambert et al., 2022.

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. Shao et al., arXiv:2402.03300, 2024.

Optional:

Scientific Discovery in the Age of Artificial Intelligence. Wang et al. Nature 620, 47–60, 2023.

Books for Additional Optional Reading

[MK] Algorithms for Decision Making. Mykel J. Kochenderfer Tim A. Wheeler
Kyle H. Wray. [Available from the book’s website as a free PDF updated in 2023]

[DB] Reinforcement Learning and Optimal Control. Dimitri Bertsekas. [Available from the book’s website as a free PDF updated in 2023]