- HJB equation for Merton Problem; CRRA utility solution; Proof of Optimality.
- Multiple Assets; Dual Value function Approach.

# Author: appliedprobability

## Diffusion Control Problems

We consider a continuous time analogue of Markov Decision Processes.

## Stochastic Integration: A Quick Summary

What follows is a heuristic derivation of the Stochastic Integral, Stochastic Differential Equations and Itô’s Formula.

## Continuous Time Dynamic Programming

Discrete time Dynamic Programming was given in the post Dynamic Programming. We now consider the continuous time analogue.

## Optimal Stopping

An Optimal Stopping Problem is an Markov Decision Process where there are two actions: meaning to stop, and meaning to continue. Here there are two types of costs

This defines a *stopping problem*.

## Algorithms for MDPs

For infinite time MDPs, we cannot apply to induction on Bellman’s equation from some initial state – like we could for finite time MDP. So we need some algorithms to solve MDPs.

## Markov Chains: A Quick Review

This section is intended as a brief introductory recap of Markov chains. A much fuller explanation and introduction is provided in standard texts e.g. Norris, Bremaud, or Levin & Peres (see references below).