Lecture

Decision Rules

Decision Rules: a mapping from data to hypothesis

Type I Error: When H_0 negative result is true, we delcare H_1 positive result. (false positive: you tested positive, but is false - false alarm)

\text{significance of a test} = Pr\{\text{declare } H_1 | H_0 \} = Pr\{\cup \text{some events that let you declare } H_1 | H_0 \}

Type II Error: When H_1 positive result is true, we delcare H_0 negative result. (false negative: you tested negative, but is false - miss)

(1 - \text{power}) = Pr\{\text{declare } H_0 | H_1 \} = Pr\{\cup \text{some events that let you declare } H_0 | H_1 \}

If you decrease one type of error, it usually increase another type of error.

We view H_0 or H_1 as latent variable.

Maximim Likelihood (ML) Decision Rule and Likelihood Ratio Tests (LRT)

Likelihood Ratio Test: if \land(k) \geq \tau, we declear H_0, otherwise H_1.

H_0 \iff \land(k) = \frac{Pr\{X = k | H_0\}}{Pr\{X = k | H_1\}} > \tau

Example: X = x are COVID-19 detected level (the higher level is, usually the more likely you have COVID-19), H_0 is you actually don't have COVID-19.

Pr\{X=k \| H_*\} X=0 X=1 X=2 X=3
H_0 0.6 0.3 0.1 0.0
H_1 0.1 0.2 0.3 0.4

Maximim Likelihood (ML): special case of Likelihood Ratio Tests (LRT) when \tau = 1.

MAP Decision Rule

If Pr\{H_0 | X = k\} \geq^? Pr\{H_1 | X = k\}, then declear H_0, otherwise H_1. This is essentially equal to:

Pr\{X = k | H_0\} Pr\{H_0\} \geq^? Pr\{X = k | H_1\}Pr\{H_1\}

Example: given Pr\{H_0\} = 0.8, Pr\{H_1\} = 0.2, we transform the table to joint distribution.

Pr\{X=k\cap H_*\} X=0 X=1 X=2 X=3
H_0 0.48 0.24 0.08 0
H_1 0.02 0.04 0.06 0.08

MAP \to LRT: This is equivalent to set \tau = \frac{Pr\{H_1\}}{Pr\{H_0\}}. Also you see MAP is skewed by the prior probability, which is not good. (People in non-COVID cluster might have lower chance to be detected.)

Error Probability

Average Error Probability (AEP)

Pr\{Error\} = Pr\{\text{Declare } H_1 | H_0\} \cdot Pr\{H_0\} + Pr\{\text{Declare } H_0 | H_1\} \cdot Pr\{H_1\}

It is essentially the sum of two joint probability in case we are wrong.

Theorem: MAP minimizes AEP.

Proof: AFSOC (\exists \sigma)(\sigma \text{ is not MAP } \land \sigma \text{ achieves lowest AEP}). Since \sigma is not MAP, then there exists X = k' such that it is either:

  1. \sigma \to H_0, but Pr\{\{X = k'\}\} < Pr\{\{X = k'\} \cap H_1\}
  2. \sigma \to H_1, but Pr\{\{X = k'\}\} > Pr\{\{X = k'\} \cap H_1\}

WLOG, we are in case 1.

\text{AEP}_\sigma = \sum_{k \in \{\sigma \to H_0\}} Pr\{\{X = k\} \cap H_1\} + \sum_{k \in \{\sigma \to H_1\}} Pr\{\{X = k\} \cap H_0\}

But then there exists another policy \sigma' that has AEP of following

\text{AEP}_{\sigma'} = \sum_{k \in \{\sigma \to H_0\} - \{k'\}} Pr\{\{X = k\} \cap H_1\} + \sum_{k \in \{\sigma \to H_1 \cap \{k'\}\}} Pr\{\{X = k\} \cap H_0\} < \text{AEP}_{\sigma}

Cost Model

E[\text{Cost}] = E[\text{Cost} | \text{Declare}H_1 \land H_0]Pr\{\text{Declare}H_1 | H_0\}Pr\{H_0\} + E[\text{Cost} | \text{Declare }H_0 \land H_1]Pr\{\text{Declare}H_0 | H_1\}Pr\{H_1\}

This is equivalent to set

\tau = \frac{Pr\{H_1\} E[\text{Cost} | \text{Declare }H_1 \land H_0]}{Pr\{H_0\} E[\text{Cost} | \text{Declare }H_0 \land H_1]} = \frac{Pr\{H_1\} E[\text{T1 Cost}]}{Pr\{H_0\} E[\text{T2 Cost}]}

Continuous Random Variable

We just change likelihood: f_{X | H_0}(x), f_{X | H_1}(x). However, the prior is still discrete because they are prior for hypothesis.

Differential Privacy via Laplace Noise

Table of Content