- The Hamilton-Jacobi-Bellman Equation.
- Heuristic derivation of the HJB equation.
- Davis-Varaiya Martingale Prinicple for Optimality

# Category: Probability

## Stochastic Integration – a Heuristic view

Heuristic derivation of

- the Stochastic Integral
- Stochastic Differential Equations
- Ito’s Formula

Continue reading “Stochastic Integration – a Heuristic view”

## Infinite Time Horizon, MDP

- Positive Programming, Negative Programming & Discounted Programming.
- Optimality Conditions.

## Markov Chains

- A short introduction to Markov chains for dynamic programming
- Definition, Markov Property, some Potential Theory.

## Talagrand’s Concentration Inequality

We prove a powerful inequality which provides very tight gaussian tail bounds “” for probabilities on product state spaces . Talagrand’s Inequality has found lots of applications in probability and combinatorial optimization and, if one can apply it, it generally outperforms inequalities like Azzuma-Hoeffding.

## Markov Chains: a functional view

- Laplacian; Adjoints; Harmonic fn; Green’s fn; Forward Eqn; Backward Eqn.
- Markov Chains and Martingales; Green’s Functions and occupancy; Potential functions; time-reversal and adjoints.

## Spitzer’s Lyapunov Ergodicity

We show that relative entropy decreases for continuous time Markov chains.

## A Mean Field Limit

We consider a system consisting of interacting objects. As we let the number of objects increase, we can characterize the limiting behaviour of the system.

## Cross Entropy Method

In the *Cross Entropy Method*, we wish to estimate the likelihood

Here is a random variable whose distribution is known and belongs to a parametrized family of densities . Further is often a solution to an optimization problem.

## Sanov’s Theorem

Sanov’s asks how *likely* is it that the empirical distribution some IIDRV’s is *far* from the distribution. And shows that the relative entropy determines the likelihood of being far.