Lecture 005

Summing a Random Number of i.i.d. Random Variable

So you have $X_1, X_2, ...$ random variables with the same distribution. And $N$ discrete, positive, integer random variable with $X_i \perp N$

Let us compute $S = \sum_{i = 1}^N X_i$

Theorem: Let $X_1, X_2, ...$ be i.i.d. (Independent and identically distributed) random variables, where $X_i \sim X$ . Let $S = \sum_{i = 1}^N X_i$ where $N \perp X_i$ , then:

$\begin{align} E[S] =& E[N]E[X]\\ E[S^2] =& E[N]Var(X) + E[N^2]E[X]^2\\ Var(S) =& E[N]Var(X) + E[X]^2Var(N)\\ \end{align}$

Expectation

$\begin{align*} E[S] =& E[\sum_{i = 1}^N X_i]\\ =& \sum_{n = 1}^\infty E[\sum_{i = 1}^N X_i | N = n] \cdot Pr\{N = n\}\\ =& \sum_{n = 1}^\infty E[\sum_{i = 1}^n X_i | N = n] \cdot Pr\{N = n\}\\ =& \sum_{n = 1}^\infty E[\sum_{i = 1}^n X_i] \cdot Pr\{N = n\} \tag{by $X_i, N$ independent}\\ =& \sum_{n = 1}^\infty nE[X_i] \cdot Pr\{N = n\}\\ =& E[X_i]\sum_{n = 1}^\infty n \cdot Pr\{N = n\}\\ =& E[X_i]E[N]\\ \end{align*}$

Variance

You want to compute $Var(S) = E[S^2] - E[S]^2$ . We now compute $E[S^2]$

$\begin{align*} &E[S^2]\\ =& E[(\sum_{i = 1}^N X_i)^2]\\ =& \sum_n E[(\sum_{i = 1}^N X_i)^2 | N = n] Pr\{N = n\}\\ =& \sum_n E[(\sum_{i = 1}^n X_i)^2] Pr\{N = n\}\\ =& \sum_n E[(X_1 + X_2 + ... + X_n)^2] Pr\{N = n\}\\ =& \sum_n nE[X_1^2] + (n^2 - n)E[X_1X_2] Pr\{N = n\}\tag{Note that $X_1 \not\perp X_1$, $X_1 \perp X_2$}\\ =& \sum_n nE[X^2] Pr\{N = n\} + \sum_n (n^2 - n)E[X]^2 Pr\{N = n\}\\ =& E[N]E[X^2] + E[N^2]E[X]^2 - E[N]E[X]^2\\ =& E[N](E[X^2] - E[X]^2) + E[N^2]E[X]^2\\ =& E[N]Var(X) + E[N^2]E[X]^2\\ \end{align*}$

With above $E[S^2] = E[N]Var(X) + E[N^2]E[X]^2$ :

$\begin{align*} Var(S) =& E[S^2] - E[X]^2\\ =& E[N]Var(X) + E[N^2]E[X]^2 - E[X]^2\\ =& E[N]Var(X) + E[N^2]E[X]^2 - E[X]^2E[N]^2 \tag{by $E[S] = E[X]E[N]$ above}\\ =& E[N]Var(X) + E[X]^2(E[N^2] - E[N]^2)\\ =& E[N]Var(X) + E[X]^2Var(N)\\ \end{align*}$

Example: Tree Growing

We have time $t = 0, 1, 2, 3...$ , and start from a node. On each step, with probability $\frac{1}{2}$ , a leaf will stay inert, with probability $\frac{1}{2}$ , a leaf will split to 2.

Define $X_t = \text{number of leaves in time} = t$ , then we see:

Define $Y_i = \begin{cases} 1 & \text{with half probability}\\ 2 & \text{with half probability}\\ \end{cases}$

$\begin{align*} X_t =& \sum_{i = 1}^{X_{t - 1}} Y_i\\ =& \sum_{i = 1}^{X_{t - 1}} 1 \cdot Pr\{\text{this leaf stay inert}\} + 2 \cdot Pr\{\text{this leaf split to 2}\}\\ \end{align*}$

We can easily calculate:

$E[Y] = \frac{3}{2}$
$E[Y^2] = 1^2 \frac{1}{2} + 2^2 \frac{1}{2} = \frac{5}{2}$
$Var(Y) = \frac{10}{4} - \frac{9}{4} = \frac{1}{4}$

Now using the formula we calculate:

$\begin{align*} &E[X_t]\\ =& E[X_{t - 1}] \cdot E[Y]\\ =& E[X_{t - 1}](\frac{3}{2})\\ =& (\frac{3}{2})^{y - 1}(\frac{3}{2}) \tag{by unfolding $E[X_{t - 1}]$}\\ =& (\frac{3}{2})^y\\ \end{align*}$

$\begin{align*} &Var(X_t)\\ =& E[X_{t - 1}]Var(Y) + E[Y]^2Var(X_{t - 1})\\ =& (\frac{3}{2})^{t - 1} \cdot \frac{1}{4} + (\frac{3}{2})^2 Var(X_{t - 1})\\ \end{align*}$

Stochasticall Domination

Stochastically Dominate: $X$ stochastically dominates $Y$ (write as $X \geq_{st} Y$ ) if $(\forall i)(Pr\{X > i\} \geq Pr\{Y > i\})$

The Tail Distribution for Gaussian is Q-function. When two gaussian is compared on the same graph, we will see one of their Q-function is always above the other.

Jensen's Inequality

$E[X^2] \geq E[X]^2$

$E[X^s] \geq E[X]^s (\text{for integer } x \geq 2)$

Definition of convex function: (all point on convex function sit above a line that pass through it) A real-valued function $g(x)$ on interval $S \subseteq \mathbb{R}$ is convex on $S$ if:

$g(t) = at + b$
$(\forall x \in S)(g(x) \geq y(x) \equiv ax + b)$ where the line $y(x) = ax + b$ is called a supporting line for the function $g$ at $t$ .

Theorem: Let $X$ be a random variable that takes on values in an interval $S$ and let $g : S \rightarrow \mathbb{R}$ be a convex function on $S$ , then $E[g(X)] \geq g(E[X])$

Visual Proof of Jensen's Inequality. For detail, visit https://en.wikipedia.org/wiki/Jensen%27s_inequality

Table of Content