Lecture 006
Z-Transforms
Reason for study z-transform:
computing higher moments
solving recurrence relation, especially in Markov chains
Generating Functions: these are equivalence
z-Transforms
Laplace transform
characteristic functions
Fourier transform
Steps:
create onion
differentiate
plug in
Definition
z-transform of function: for a discrete function p(i) where i = 0, 1, 2, ... , the z-transform of p(i) is
G_p(z) = \sum_{i = 0}^\infty p(i)z^i
z-transform of random variable: where we assume typically that \|z\| \leq 1 and think of z as constant.
\widehat{X}(z) = G_p(z) = E[z^X]
the convergence is guaranteed when X is non-negative and \|z\| \leq 1 .
Note that \widehat{X} = 1 for all X .
Convergence proof: assume - 1 \leq z \leq 1 , we have -1 \leq \widehat{X}(z) \leq 1 for all z .
\begin{align*}
&-1 \leq z^i \leq 1\\
=& -Pr\{x = i\} \leq z^i Pr\{X = i\} \leq Pr\{X = i\}\\
=& - \sum_i Pr\{x = i\} \leq \sum_i z^i Pr\{X = i\} \leq \sum_i Pr\{X = i\}\\
=& -1 \leq \widehat{X}(z) \leq 1\\
\end{align*}
Building the Onion
Bernoulli
For X \sim \text{Bernoulli}
\widehat{X}(z) = z^0(1 - p) + z^1p = 1 - p + pz
Binomial
For X \sim \text{Binomial}
\begin{align*}
\widehat{X}(z) = E[z^X] =& \sum_{t = 0}^n z^i {n \choose i} p^i (1 - p)^{n - i}\\
=& \sum_{t = 0}^n {n \choose i} (pz)^i (1 - p)^{n - i}\\
=& (zp + (1 - p))^n\\
\end{align*}
Or, using linearity of z-transform (of Bernoulli):
\widehat{X_{Binomial}}(z) = \left(\widehat{X_{Bernoulli}}(z)\right)^n = (1 - p + pz)^n
z-transform of sum of two independent binomial: W = X + Y where X \sim \text{Binomial}(n, p), Y \sim \text{Binomial}(m, p)
We have \widehat{W}(z) = \widehat{X}(z) + \widehat{Y}(z) = (zp + (1 - p))^{m + n}
This implies that W \sim \text{Binomial}(m + n, p) , which make sense.
Geometric
For X \sim \text{Geometric}
\begin{align*}
\widehat{X}(z) = E[z^X] =& \sum_{i = 1}^\infty z^i p(1 - p)^{i - 1}\\
=& zp\sum_{i = 1}^\infty (z(1 - p))^{i - 1}\\
=& zp\sum_{i = 0}^\infty (z(1 - p))^i\\
=& \frac{zp}{1 - z(1 - p)} \tag{by assume $\left\|z(1 - p)\right\| \leq 1$ for convergence}\\
\end{align*}
Poisson
For X \sim \text{Poisson}
\begin{align*}
\widehat{X}(z) = E[z^X] =& \sum_{i = 0}^\infty z^i \frac{e^\lambda \lambda^i}{i!}\\
=& e^{-\lambda}\sum_{i = 0}^\infty \frac{(\lambda z)^i}{i!}\\
=& e^{-\lambda}e^{\lambda z}\\
=& e^{-\lambda(1 - z)}\\
\end{align*}
Onion Peeling
You have a onion that does converge. You can hold it in your hand. Building onion is easy, but what do you do when you try to peel an onion? You cry.
Onion Peeling: for X discrete, integer-valued, non-negative random variable with Pr\{X = i\} for i = 0, 1, 2, ... :
\begin{align*}
\widehat{X}(z)|_{z = 1} =& E[1^X] = 1\\
\widehat{X}'(z)|_{z = 1} =& E[X]\\
\widehat{X}''(z)|_{z = 1} =& E[X(X - 1)]\\
\widehat{X}'''(z)|_{z = 1} =& E[X(X - 1)(X - 2)]\\
\widehat{X}''''(z)|_{z = 1} =& E[X(X - 1)(X - 2)(X - 3)]\\
\end{align*}
Note: if above moments are not defined at z = 1 , one can instead consider the \lim_{z \rightarrow 1} where evaluating the limit may require using L'Hospital's rule.
Proof:
\begin{align*}
\widehat{X}(z) =& p_X(0)z^0 + p_X(1)z^1 + p_X(2)z^2 + p_X(3)z^3 + p_X(4)z^4 + ...\\
\widehat{X}'(z) =& p_X(1) + 2p_X(2)z^1 + 3p_X(3)z^2 + 4p_X(4)z^3 + ...\\
\widehat{X}'(z)|_{z = 1} =& p_X(1) + 2p_X(2) + 3p_X(3) + 4p_X(4) + ...\\
\widehat{X}'(z)|_{z = 1} =& E[X]\\
\widehat{X}''(z) =& 2p_X(2) + 3\cdot 2p_X(3)z + 4\cdot 3p_X(4)z^2 + ...\\
\widehat{X}''(z)|_{z = 1} =& 2p_X(2) + 3\cdot 2p_X(3) + 4\cdot 3p_X(4) + ...\\
\widehat{X}''(z)|_{z = 1} =& E[X(X - 1)]\\
...\\
\end{align*}
Example: Variance of Geometric
\begin{align*}
\widehat{X}(z) =& \frac{zp}{1 - z(1 - p)} \tag{build onion}\\
E[X] =& \frac{d}{dz} \left(\frac{zp}{1 - z(1 - p)}\right)|_{z = 1}\\
=& \frac{p}{(1 - z(1 - p))^2}|_{z = 1}\\
=& \frac{1}{p} \tag{peeling onion 1st layer}\\
E[X^2] =& \widehat{X}''(z)|_{z = 1} + E[X]\\
=& \frac{2p(1 - p)}{(1 - z(1 - p))^3}|_{z = 1} + \frac{1}{p}\\
=& \frac{2(1 - p)}{p^2} + \frac{1}{p}\\
=& \frac{2 - p}{p^2} \tag{peeling onion 2nd layer}\\
Var(X) =& E[X^2] - (E[X])^2\\
=& \frac{1 - p}{p^2}\\
\end{align*}
An onion holds all the information about the distribution. There is an injective mapping from the set of discrete non-negative random variable to the set of z-transforms. z-transform uniquely determines the distribution. // TODO: practice question
Linearity of z-Transforms
Given X, Y discrete random variables. Then
X \perp Y \implies \widehat{X + Y}(z) = \widehat{X}(z) \cdot \widehat{Y}(z)
proof: E[z^{X + Y}] = E[z^X \cdot z^Y] = E[z^X] \cdot E[z^Y] = \widehat{X}(z) \cdot \widehat{Y}(z)
Conditioning on z-Transforms
Let X, A, B be discrete random variable with
X = \begin{cases}
A & \text{with probability } p\\
B & \text{with probability } (1 - p)\\
\end{cases}
Then we have
\widehat{X}(z) = \widehat{A}(z)p + \widehat{B}(z)(1 - p)
which can be easily proved by conditioning on expectation.
Summing a Random Number of i.i.d. Random Variable
Let X_1, X_2, ... be i.i.d. discrete random variables, where X_i \sim X . Let N be a positive discrete random variable where (\forall i)(N \perp X_i) . Then:
\begin{align*}
S =& \sum_{i = 1}^N X_i \implies \widehat{S}(z) = \widehat{N}\left(\widehat{X}(z)\right)\\
=& E[\widehat{X}(z)^N] = E[(E[z^X])^N] = E[z^{XN}]\\
\end{align*}
Using Z-Transform to Solve Recurrence
General Formula for Febonaci Number: f_n = \frac{1}{\sqrt{5}}(\phi^n - (-\phi)^{-n}) where \phi = \frac{1 + \sqrt{5}}{2} .
The proof is too hard. Refer to the book when you can. // TODO: practice question