Probability Foundations

🎲 Probability Foundations: Mastering Uncertainty

Probability is the mathematical framework for quantifying uncertainty. In software engineering, it is used for everything from A/B testing and risk analysis to machine learning and system reliability.

🟢 Level 1: The Axioms of Probability

1. Kolmogorov’s Axioms

Probability is defined over a sample space ( $S$ ) and an event space ( $E \subseteq S$ ). For any event $A$ :

Non-negativity: $P(A) \ge 0$ .
Normalization: $P(S) = 1$ .
Additivity: For disjoint events $A_1, A_2, \dots$ : $P(\bigcup_{i} A_i) = \sum_{i} P(A_i)$ .

2. Basic Properties

Complement: $P(A^c) = 1 - P(A)$ .
General Addition Rule: $P(A \cup B) = P(A) + P(B) - P(A \cap B)$ .

🟡 Level 2: Conditional Probability and Independence

3. Conditional Probability

The probability of event $A$ occurring given that event $B$ has already occurred: $P(A|B) = \frac{P(A \cap B)}{P(B)}$

4. Independence

Two events $A$ and $B$ are independent if the occurrence of one does not affect the probability of the other: $P(A|B) = P(A) \iff P(A \cap B) = P(A)P(B)$

import random

# A simple simulation to show independence
n_trials = 10000
coin_flips = [(random.choice(['H', 'T']), random.choice(['H', 'T'])) for _ in range(n_trials)]

# P(Second flip is H)
p_h2 = sum(1 for f in coin_flips if f[1] == 'H') / n_trials
# P(Second flip is H | First flip is H)
p_h2_given_h1 = sum(1 for f in coin_flips if f[0] == 'H' and f[1] == 'H') / sum(1 for f in coin_flips if f[0] == 'H')

print(f"P(H2): {p_h2:.4f}, P(H2 | H1): {p_h2_given_h1:.4f}")

🔴 Level 3: Bayes’ Theorem

5. The Power of Bayesian Inference

Bayes’ theorem allows us to update the probability of a hypothesis ( $H$ ) based on new evidence ( $E$ ): $P(H|E) = \frac{P(E|H)P(H)}{P(E)}$

$P(H|E)$ : Posterior (Updated belief).
$P(E|H)$ : Likelihood (Evidence given hypothesis).
$P(H)$ : Prior (Initial belief).
$P(E)$ : Evidence (Normalization factor).

6. Law of Total Probability

To find the denominator $P(E)$ , we sum over all possible hypotheses: $P(E) = \sum_i P(E|H_i)P(H_i)$

# Bayesian Spam Filter Concept
# P(Spam | 'Winner') = P('Winner' | Spam) * P(Spam) / P('Winner')

p_spam = 0.2
p_winner_given_spam = 0.9
p_winner_given_not_spam = 0.1

p_winner = (p_winner_given_spam * p_spam) + (p_winner_given_not_spam * (1 - p_spam))
p_spam_given_winner = (p_winner_given_spam * p_spam) / p_winner

print(f"Probability of spam given the word 'Winner': {p_spam_given_winner:.4f}")