What follows is a heuristic derivation of the Stochastic Integral, Stochastic Differential Equations and Itô’s Formula.

# Author: appliedprobability

## Continuous Time Dynamic Programming

Discrete time Dynamic Programming was given in the post Dynamic Programming. We now consider the continuous time analogue.

## Optimal Stopping

An Optimal Stopping Problem is an Markov Decision Process where there are two actions: meaning to stop, and meaning to continue. Here there are two types of costs

This defines a *stopping problem*.

## Algorithms for MDPs

For infinite time MDPs, we cannot apply to induction on Bellman’s equation from some initial state – like we could for finite time MDP. So we need some algorithms to solve MDPs.

## Markov Chains: A Quick Review

This section is intended as a brief introductory recap of Markov chains. A much fuller explanation and introduction is provided in standard texts e.g. Norris, Bremaud, or Levin & Peres (see references below).

## Infinite Time Horizon

Thus far we have considered finite time Markov decision processes. We now want to solve MDPs of the form

## Markov Decision Processes

Markov decision processes are essentially the randomized equivalent of a dynamic program.