Kalman filtering (and filtering in general) considers the following setting: we have a sequence of states , which evolves under random perturbations over time. Unfortunately we cannot observe , we can only observe some noisy function of , namely, . Our task is to find the best estimate of given our observations of . Continue reading “Kalman Filter”

# Category: Control

## Temporal Difference Learning – Linear Function Approximation

For a Markov chain , consider the reward function

associated with rewards given by . We approximate the reward function with a linear approximation,

Continue reading “Temporal Difference Learning – Linear Function Approximation”

## Stochastic Linear Regression

We consider the following formulation of Lai, Robbins and Wei (1979), and Lai and Wei (1982). Consider the following regression problem,

for where are unobservable random errors and are unknown parameters.

Typically for a regression problem, it is assumed that inputs are given and errors are IID random variables. However, we now want to consider a setting where we sequentially choose inputs and then get observations , and errors are a martingale difference sequence with respect to the filtration generated by .

## Foster-Lyapunov

Foster’s Lemma provides a natural condition to prove the positive recurrence of a Markov chain.

## Experts and Bandits (non-stochastic)

- Weighted majority algorithm its variant for Bandit Problems.

## More on Merton Portfolio Optimization

## Diffusion Control Problems

- The Hamilton-Jacobi-Bellman Equation.
- Heuristic derivation of the HJB equation.
- Davis-Varaiya Martingale Prinicple for Optimality