Stationary Distributions — Recap – Applied Probability Notes

We have characterized recurrence in terms of return times. However, if a Markov chain is positive recurrent, what proportion of time does it spend in each state?

Theorem. For an irreducible positive recurrent Markov chain, there exists a probability distribution $\pi = (\pi(x) : x \in \mathcal{X})$ such that for all $x \in \mathcal{X}$

$\displaystyle \frac{1}{T} \sum_{t=1}^T \mathbb{I}[X_t = x] \to \pi(x) \quad \text{as } T \to \infty$

Moreover, $\pi$ is the unique solution to the full balance equations

$\displaystyle \sum_{x \in \mathcal{X}} \pi(x) = 1, \quad \pi P = \pi$

Conversely, if the full balance equations have a solution, then the Markov chain is positive recurrent.

The distribution $\pi$ is stationary in the sense that

$\displaystyle \mathbb{P}(X_0 = x) = \pi(x) \ \forall x \in \mathcal{X} \quad \Rightarrow \quad \mathbb{P}(X_t = x) = \pi(x) \ \forall x \in \mathcal{X}, \ \forall t$

When the above convergence holds, the chain is called ergodic.
The distribution $\pi(x)$ is called the stationary (or equilibrium) distribution.
The full balance equations can also be written as

$\displaystyle \sum_{y \neq x} \pi(y) P_{yx} = \sum_{y \neq x} \pi(x) P_{xy}$

This can be interpreted as: the proportion of time entering state $x$ equals the proportion of time leaving $x$ .

Figure: flow of probability into and out of a state.

Reversibility

Instead of full balance, we may require that transitions between each pair of states balance. This is called detailed balance.

Definition. A distribution $(\pi(x) : x \in \mathcal{X})$ satisfies detailed balance if

$\displaystyle \pi(x) P_{xy} = \pi(y) P_{yx} \quad \forall x,y \in \mathcal{X}$

Detailed balance implies full balance, and hence gives a stationary distribution.
It is often easier to verify detailed balance than full balance.
Many Markov chains (especially in Monte Carlo methods) are designed to be reversible.

Definition (Time reversal and reversibility). For a Markov chain $X = (X_t : t \in \mathbb{Z})$ , define the reversed process