Since we don't know the tail (Pr\{X \geq k\}) for some distribution (e.g. Binomial, Poisson), we want to
Tail Bounds: give an upper bound to Pr\{X \geq k\}
Concentration Bound (Inequality): give an upper bound to Pr\{|X - E[X]| \geq k\}
// TODO: example
Markov Bound
For non-negative random variable X, for all a > 0:
Pr\{X \geq a\} \leq \frac{E[X]}{a}
Visual Proof: Fix a non-negative integer random variable X. Plot Pr\{X \geq x\} as a function of x. Area under curve is \sum_{x = 0}^\infty Pr\{X \geq x\} = \sum_{x = 0}^\infty x \cdot Pr\{X = x\} = E[X]. Thus x \cdot Pr\{X \geq x\} \leq E[X] Note each area of vertical bar represent may contain multiple x values.
Proof:
\begin{align*}
E[X] =& \int_0^\infty x f_X(x) dx\\
\geq \int_a^\infty x f_X(x) dx\\
\geq \int_a^\infty a f_X(x) dx\\
= a \int_a^\infty f_X(x) dx\\
= a Pr\{X \geq a\}\\
\end{align*}
Note: Markov Bound is extremely weak bound because it only cares about E[X]. But it helps us to derive other bounds.
Inverse Markov: Let Y be non-negative random variable and Y \leq b and 0 < a < b, then
Pr\{Y \leq a\} \leq \frac{E[b - Y]}{b - a}
Chebyshev's Inequality
For random variable X with finite E[X], Var(X), for all a > 0:
Pr\{|X - E[X]| \geq a\} \leq \frac{Var(X)}{a^2}
When you are using bounds that require absolute value but you don't have absolute value in your equation, you can addict manually because Pr\{X \leq a\} \leq Pr\{X \leq a \cup -X \leq a\} = Pr\{|X| \leq a\}
Note that this bound is exponentially decreasing with high \epsilon. The bound is particularly strong with high \epsilon.
// TODO: proof
Compare two Chernoff Bound for Binomial:
for p_i = p = \frac{1}{2}: Pretty Chernoff Bound for Binomial is stronger
In Ugly: the value of \epsilon is only 0.5 which result weaker bound.
for p_i = p = \frac{1}{n}: Ugly Chernoff Bound for Binomial is stronger with n \rightarrow \infty
In Pretty: \delta does not increase with n (it is only strong when \delta \in \Theta(n)). Pretty is only great when \delta \in \Theta(n) (not even \Theta(\ln n))
Hoeffding Bound
Let X_1, X_2, ..., X_n be independent (not necessarily iid) random variables with a_i \leq X_i \leq b_i for all i where a_1, b_i are reals. Let X = \sum_{i = 1}^n X_i. Then: