Lecture 017

Discrete Time Markov Chain

Descrite Time Markov Chain (DTMC): a stochastic process $\{X_0, X_1, X_2, ...\}$ where $X_n$ is the state at discrete timestep $n$ , such that

$\begin{align*} &Pr\{X_{n + 1} = j | X_n = i, X_{n - 1} = i_{n - 1}, X_{n - 2} = i_{n - 2}, ..., X_0 = i_0\}\\ =& Pr\{X_{n + 1} = j | X_n = i\} \tag{Markovian Property: hostorical state doesn't affect conditional distribution of future states}\\ =& Pr_{ij} \tag{Stationary Property: specific value of time step $n$ in current state doesn't affect conditional distribution of future states}\\ \end{align*}$

Discrete: discrete time step. Stochastic: a sequence of random variable (sequence of states at each time step). $\mathbb{P}$ can have infinite number of state.

Limiting Distribution: Umbrella Problem

Mor travels from "Gate" to "Home" and back every day. She puts one umbrella in "Gate" and one in "Home" at beginning. Every time it rains with probability $p$ , Mor brings takes an umbrella (if exist) at her place and walk to "Gate" or "Home". What is the probability that Mor get wet.

Tansition Probability Matrix ( $\mathbb{P}$ ): a description of the Markov Chain where $\mathbb{P}_{ij}$ represent transition probability from state $i$ to state $j$ in next step.

$\mathbb{P} = \begin{bmatrix} 0 & 0 & 1\\ 0 & 1-p & p\\ 1-p & p & 0\\ \end{bmatrix}$

Here is the Markov Chain we designed where each state's transition probability to other states is characterized by the rows. We have $m = 3$ states represent how many umbrella Mor has at her current place. The change we get wet is the $Pr\{\text{having no umbrella currently}\} \cdot Pr\{\text{it rains}\} = \pi_0 \cdot p$ . Observe each row sum to $1$ .

This DTMC simplifies problem by having as few states as possible. Other times, it might help to expand the number of states than it originally has in the problem.

Limiting Probability Matrix: To calculate the probability of each state that Mor will end up with, we calculate the limiting probability by giving probability transition matrix a big power:

$\lim_{n \to \infty} \mathbb{P}^n = \begin{bmatrix} 0.23 & 0.385 & 0.385\\ 0.23 & 0.385 & 0.385\\ 0.23 & 0.385 & 0.385\\ \end{bmatrix}$

Notice $(\mathbb{P}^2)_{ij} = \sum_k p_{ik} \cdot p_{kj}$ is the probability going the path $(\forall k)(i \to k \to j)$ , meaning the probability a state will transition from $i$ to $k$ in two steps.

Also notice in this specific example, $\pi_2 = \pi_1$ . This is because number of visits to state $2$ and state $1$ is equal because state $0$ will eventually go to state $1$ . If we collapse state $0$ , the chance of staying at state $2$ is equal to state $1$ .

Limiting Probability: the limiting probability (independent from starting state $i$ ) for state $j$ is:

$\pi_j = \lim_{n \to \infty} (\mathbb{P}^n)_{ij}$

Ergodicity Issue: for now we do not ask whether the limiting probability actually exists. (it may not exists due to limit does not exists)

Limiting distribution: where $m$ is the number of states and $\sum_{i = 0}^{m - 1} = 1$ .

$\vec{\pi} = \left(\pi_0, \pi_1, ..., \pi_{m - 1}\right)$

Notice every row of limiting probability is a limiting distribution.

Stationary Distribution: Umbrella Problem

Stationary Distribution: a distribution $\vec{\pi} = \left(\pi_0, \pi_1, ..., \pi_{m - 1}\right)$ if it satisfy Stationary Equations:

$\vec{\pi}P = \vec{\pi}$ : $(\forall k)(\sum_k \pi_k \cdot p_{kj} = \pi_j)$ (it is stationary)
$\sum_{i = 0}^{m - 1} \pi_i = 1$ (it is a distribution)

$\begin{align*} \pi_0p_{00} + \pi_1p_{10} + \pi_2p_{20} = \pi_0\\ \pi_0p_{01} + \pi_1p_{11} + \pi_2p_{21} = \pi_1\\ \pi_0p_{02} + \pi_1p_{12} + \pi_2p_{22} = \pi_2\\ \end{align*}$

Looking at a graph, it is often nice to think the stationary equation as $\pi_{\text{node to}} = \sum_{i = \text{node from}} \pi_{i} Pr\{\text{from}\to\text{to}\}$

Stationary Distribution Equals Limiting Distribution

Theory: given limiting distribution $\vec{\pi}$ exists, then

$\vec{\pi}$ is also a stationary distribution and
no other stationary distribution exists

Visual Proof: once the power is very big ( $n \to \infty$ ), each $\pi_i$ will become "sticky" because multiply by another $P$ doesn't really change the value of $\pi_i$ .

Proof: given $\vec{\pi}$ limiting distribution exists, then $\vec{\pi}$ is stationary distribution. We want to show for all $j \in \{0, 1, ..., m - 1\}$ -th element, $\pi_j = \sum_{k = 0}^{m - 1} \pi_k \cdot P_{kj}$

$\begin{align*} \pi_{j} =& \lim_{n \to \infty}(P^{n + 1})_{ij}\\ =& \lim_{n \to \infty} \sum_{k = 0}^{m - 1} (P^n)_{ik} \cdot P_{kj}\\ =& \sum_{k = 0}^{m - 1} \lim_{n \to \infty} (P^n)_{ik} \cdot P_{kj} \tag{by discrete}\\ =& \sum_{k = 0}^{m - 1} \pi_k \cdot P_{kj}\\ \end{align*}$

Note in above we use $\pi$ to denote limiting distribution and use $\pi'$ to denote stationary distribution. This proof above is likely in the exam.

Proof: given $\vec{\pi}'$ the stationary distribution and $\vec{\pi}$ the limiting distribution, we show for all $j \in \{0, 1, ..., m - 1\}$ -th element, $\vec{\pi}'_j = \vec{\pi}_j$ .

$\begin{align*} \vec{\pi}' =& \vec{\pi}' \cdot \mathbb{P}^n\\ \vec{\pi}'_j =& \sum_{k = 0}^{m - 1} \vec{\pi}'_k \cdot \mathbb{P}^n_{kj} \tag{by looking only at $j$-th entry}\\ \lim_{n \to \infty} \vec{\pi}'_j =& \lim_{n \to \infty} \sum_{k = 0}^{m - 1} \vec{\pi}'_k \cdot \mathbb{P}^n_{kj}\\ \vec{\pi}'_j =& \sum_{k = 0}^{m - 1} \vec{\pi}'_k \cdot \lim_{n \to \infty} \mathbb{P}^n_{kj} \tag{by $m$ is finite}\\ =& \sum_{k = 0}^{m - 1} \vec{\pi}'_k \cdot \pi_j\\ =& \pi_j \sum_{k = 0}^{m - 1} \vec{\pi}'_k\\ =& \pi_j\\ \end{align*}$

Stationary Markov Chain: a Markov chain (where stationary probabilities exist) is "stationary" or "in steady state" if the initial state is chosen according to stationary probabilities.

Solving Stationary Equations

Given limiting distribution exists, we can obtain it by solving stationary equations. We solve $\vec{\pi}$ that satisfy:

$\vec{\pi}P = \vec{\pi}$ ( $m - 1$ equations)
$\sum_{i = 0}^{m - 1} \pi_i = 1$ ( $1$ equation)

For $\mathbb{P}^{m \times m}$ , only $m - 1$ many equations will be linearly independent. We solve a system of $(m - 1) + 1$ many equations to find $\vec{\pi}$ .

Doubly Stochastic Matrix: entries in each row sum up to $1$ , and entry in each column sum up to $1$ .

If $\mathbb{P}$ is a transition matrix that is doubly stochastic with $m$ states in a finite DTMC with existing limiting distribution, then the stationary distribution is uniform.

Table of Content