Applied Probability Notes – Page 2

Lyapunov Functions

Lyapunov functions are an extremely convenient device for proving that a dynamical system converges. We cover:

The Lyapunov argument
La Salle’s Invariance Principle
An Alternative argument for Convex Functions
Exponential Convergence Rates

Continue reading “Lyapunov Functions”

Optimal Stopping

Here are the slides from Lectures

6_Optimal Stopping

Please read Section 1.6 from the notes:

Stochastic_Control_2020 [pdf]

Please attempt Ex53, Ex54, Ex56, Ex57.

Algorithms for MDPs

Here are the slides from Lectures

5_Algorithms for MDPs

Please read Section 1.5 from the notes:

Stochastic_Control_2020 [pdf]

Please attempt Ex39, 40 & 41 [if you can code], 42 and 43.

Infinite Time Horizon MDP

Here are the slides from Lectures

4_Infinite Time Horizon

Please read Section 1.4 from the notes:

Stochastic_Control_2020 [pdf]

Please attempt Ex35, Ex36, Ex37.

Markov Decision Processes

Here are the slides from Lectures

3_Markov Decision Processes [pdf]

Please read Section 1.3 from the notes:

Stochastic_Control_2020 [pdf]

Please attempt exercises Ex22, Ex23, Ex24, Ex25.

Markov Chains

Here are the slides from Lectures

2_Markov_Chains

Please read Section 1.2 from the notes:

Stochastic_Control_2020

Dynamic Programming

Slides from Lectures are here:

1_Dynamic_Programming

Please read Section 1.1 of the notes:

Stochastic Control Notes [pdf]

Please attempt exercises Ex3, Ex5 and Ex6 from the notes.

Stochastic Control 2020

Another year of MATH69122! — aka Stochastic Control.

This year, I will try to keep updating PDFs with slides and notes for each lecture. I’ll keep notes for the course in the “PDF” tab above. These are also here:

Stochastic Control 2020 [pdf]

Here is a rough plan for each week of lectures:

Continue reading “Stochastic Control 2020”

Kalman Filter

Kalman filtering (and filtering in general) considers the following setting: we have a sequence of states $x_t$ , which evolves under random perturbations over time. Unfortunately we cannot observe $x_t$ , we can only observe some noisy function of $x_t$ , namely, $y_t$ . Our task is to find the best estimate of $x_t$ given our observations of $y_t$ . Continue reading “Kalman Filter”

Temporal Difference Learning – Linear Function Approximation

For a Markov chain $\hat{x} = (\hat x_t : t\in\mathbb Z_+)$ , consider the reward function

$\label{TD:Reward} R(x) := \mathbb E_{x} \left[ \sum_{t=0}^\infty r(\hat x_t) \right]$

associated with rewards given by $r = (r(x) : x\in\mathcal X)$ . We approximate the reward function $R(x)$ with a linear approximation,

$R(x;\bm w) = \bm w^\top \bm \phi (x) = \sum_{j\in \mathcal J} w_j \phi_j(x) .$

Continue reading “Temporal Difference Learning – Linear Function Approximation”