We consider the Robbins-Monro update
and argue that this can be approximated by the o.d.e.:
Lyapunov Functions
Lyapunov functions are an extremely convenient device for proving that a dynamical system converges. We cover:
- The Lyapunov argument
- La Salle’s Invariance Principle
- An Alternative argument for Convex Functions
- Exponential Convergence Rates
Optimal Stopping
Here are the slides from Lectures
Please read Section 1.6 from the notes:
Please attempt Ex53, Ex54, Ex56, Ex57.
Algorithms for MDPs
Here are the slides from Lectures
Please read Section 1.5 from the notes:
Please attempt Ex39, 40 & 41 [if you can code], 42 and 43.
Infinite Time Horizon MDP
Here are the slides from Lectures
Please read Section 1.4 from the notes:
Please attempt Ex35, Ex36, Ex37.
Markov Decision Processes
Here are the slides from Lectures
3_Markov Decision Processes [pdf]
Please read Section 1.3 from the notes:
Please attempt exercises Ex22, Ex23, Ex24, Ex25.
Markov Chains
Here are the slides from Lectures
Please read Section 1.2 from the notes:
Dynamic Programming
Slides from Lectures are here:
Please read Section 1.1 of the notes:
Stochastic Control Notes [pdf]
Please attempt exercises Ex3, Ex5 and Ex6 from the notes.
Stochastic Control 2020
Another year of MATH69122! — aka Stochastic Control.
This year, I will try to keep updating PDFs with slides and notes for each lecture. I’ll keep notes for the course in the “PDF” tab above. These are also here:
Here is a rough plan for each week of lectures:
Kalman Filter
Kalman filtering (and filtering in general) considers the following setting: we have a sequence of states 
