Lecture 002

Discrete Random Variable: countable number of values

Distribution of Ramdom Variables

Probability Mass Function (p.m.f.): $P_x(i) = Pr\{X = i\}$
Accumulative Distribution Function (a.d.f.): $F_x(i) = Pr\{X = i\}$
Tail: $\bar{F}_x(i) = Pr\{X > i\}$

Bernoulli

$X \sim \text{Bernoulli}(p)$ :

Experiment: flip p-coin ( $P_x(1) = p$ ) once
Random Variable $X = \begin{cases} 1 &\text{if Head}\\ 0 &\text{if tail}\\ \end{cases}$
p.m.f.: $P_x(i) = \begin{cases} p &\text{if } i = 1\\ 1-p &\text{if } i = 0\\ \end{cases}$
Summation: $p+(1-p) = 1$

Binomial

$X \sim \text{Binomial}(n, p)$ :

Experiment: flip p-coin ( $P_x(1) = p$ ) n times
Random Variable $X = \text{number of heads}$
p.m.f.: $P_x(i) = {n \choose i} p^i(1-p)^{n-i}$
Summation: Binomial Series $\sum_{i = 0}^n {n \choose i}p^i(1 - p)^{n - i} = (p + (1 - p))^n = 1$

Geometric

$X \sim \text{Geometric}(p)$ :

Experiment: flip p-coin ( $P_x(1) = p$ ) until get a head
Random Variable $X = \text{number of flips until getting a head (including the head time)}$
p.m.f.: $P_x(i) = (1-p)^{i-1}p$ where $i = 1, 2, 3, ...$
Tail: $\bar{F}_x(i) = (1-p)^i$
Summation: Geometric Series $\sum_{i = 1}^{\infty} (1 - p)^{i - 1} \cdot p = \sum_{i = 0}^{\infty} (1 - p)^i \cdot p = \sum_{i = 0}^{\infty} (1 - p)^i \cdot p = \frac{1}{1 - (1 - p)} = 1$

Poisson

$X \sim \text{Poisson}(\lambda)$ :

Experiment: mix non-negative independent distribution, where $\lambda$ is the peak. Models mixture of a very large number of independent source, each with a very small individual probability.
Distribution: $P_x(i) = \frac{e^{-\lambda}\lambda^i}{i!}$ for $i = 0, 1, 2, ...$
Summation: Taylor Series $e^{-\lambda} \sum_{i = 0}^{\infty} \frac{\lambda^i}{i!} = e^{-\lambda}e^{\lambda} = 1$

When $n$ is large and $p$ is small, $\text{Binomial}(n, p) \simeq \text{Poisson}(np)$

Joint Probability

Joint Probability Mass Function:

$P_{X, Y}(x, y) = Pr\{X = x \cap Y = y\} \text{ where }\sum_x \sum_y Pr\{X = x \cap Y = y\} = 1$

Marginal Probability: probability obtained by summing along another dimension

$P_X(x) = \sum_y P_{X, Y}(x, y)$

Independent of Joint Probability:

$X \perp Y \iff (\forall x \in X), (\forall y \in Y)(Pr\{X = x \cap Y = y\} = Pr\{X = x\} \cdot Pr\{Y = y\})$

Example: Each day, event $A$ happens at probability $p_1$ , event $B$ happens at probability $p_2$ , what is the probability that $p_1$ happens before $p_2$ ?

Brute Force: Let $X_1, X_2$ denotes the first day event $A, B$ happens.

$\begin{align*} Pr\{X_1 < X_2\} &= \sum_{k_1 = 1}^{\infty} \sum_{k_2 = k_1 + 1}^{\infty} (1-p_1)^{k_1 - 1}p_1(1-p_2)^{k_2 - 1}p_2\\ &= p_1p_2 \sum_{k_1 = 1}^{\infty} (1-p_1)^{k_1 - 1} (\sum_{k_2 = k_1 + 1}^{\infty} (1-p_2)^{k_2 - 1})\\ &= p_1p_2 \sum_{k_1 = 1}^{\infty} (1-p_1)^{k_1 - 1} ((1-p_2)^{k_1} \sum_{k_2 = 1}^{\infty} (1-p_2)^{k_2 - 1})\\ &= p_1p_2 \sum_{k_1 = 1}^{\infty} (1-p_1)^{k_1 - 1} ((1-p_2)^{k_1} \frac{1}{1-(1-p_2)})\\ &= p_1p_2 \sum_{k_1 = 1}^{\infty} (1-p_1)^{k_1 - 1} ((1-p_2)^{k_1} \frac{1}{p_2})\\ &= p_1 \sum_{k_1 = 1}^{\infty} (1-p_1)^{k_1 - 1} ((1-p_2)^{k_1})\\ &= p_1 (1-p_2) \sum_{k_1 = 1}^{\infty} (1-p_1)^{k_1 - 1} ((1-p_2)^{k-1})\\ &= p_1 (1-p_2) \sum_{k_1 = 1}^{\infty} ((1-p_1)(1-p_2))^{k_1 - 1}\\ &= \frac{p_1 (1-p_2)}{1 - (1-p_1)(1-p_2)}\\ \end{align*}$

Another ways of thinking: $Pr\{p_1\text{ before }p_2\} = Pr\{p_1 \cup \bar{p_2} | \lnot (p_1 \cap p_2)\}$ (on the day of not the same, what is the probability that it is in this specific configuration?)

Law of Total Probability for Discrete Random Variables: for an event $E$ and a discrete random variable $Y$ :

$Pr\{E\} = \sum_{y}Pr\{E \cap Y = y\} = \sum_y Pr\{E | Y = y\} \cdot Pr\{Y = y\}$

Conditioning:

$\begin{align*} Pr\{X_1 < X_2\} = &\sum_{k = 1}^\infty Pr\{X_1 < X_2 | X_1 = k\} \cdot Pr\{X_1 = k\}\\ = &\sum_{k = 1}^\infty Pr\{k < X_2 | X_1 = k\} \cdot Pr\{X_1 = k\}\\ = &\sum_{k = 1}^\infty Pr\{k < X_2\} \cdot Pr\{X_1 = k\} \tag{by $X_1, X_2$ independence}\\ = &\sum_{k = 1}^\infty (1 - p_2)^k \cdot (1-p_1)^{k-1}p_1\\ = &p_1(1- p_2)\sum_{k = 1}^\infty ((1-p_2)(1-p_1))^{k - 1}\\ = &p_1(1- p_2)\frac{1}{1 - (1 - p_2)(1 - p_1)}\\ \end{align*}$

Table of Content