Category · Mathematics for Software
Probability
The math of what you don't know for certain - and the toolkit behind randomized algorithms, A/B tests, spam filters, and every model that learns from data. This page is a deep, practical walkthrough: why it matters, how it shapes the way you write code, how it drives model training, and everything worth knowing in between.
Live demo — Law of Large Numbers
Flip a simulated fair coin. Watch the empirical probability of heads wobble at first, then settle toward 0.5 as the number of flips grows.
01 · Why it's needed
The world your code runs in is uncertain
Deterministic logic assumes clean inputs and known outcomes. Real systems don't work that way — networks drop packets, users behave unpredictably, sensors have noise, and data is never a perfect sample of the world. Probability is the formal language for reasoning correctly under that uncertainty, instead of guessing.
Decisions under uncertainty
Should you retry a request, cache a result, or roll back a deploy? Every one of these is a bet, and probability tells you the odds.
Measuring what's real
A/B tests, error rates, and latency percentiles are only meaningful once you know how much natural variation to expect.
Modeling noisy signals
Sensors, user behavior, and network conditions are inherently random — probability lets you build systems that are robust to that noise instead of broken by it.
Foundation of machine learning
Every model that 'learns from data' is, under the hood, estimating a probability distribution and updating its beliefs as evidence arrives.
02 · How it shapes coding & logic
It changes how you write and reason about code
Probability isn't just theory that sits beside your code — it directly shapes data structures, algorithms, and how you test and reason about correctness.
Randomized algorithms
Quicksort's random pivot, skip lists, and randomized load balancing all use randomness to guarantee good average-case behavior, trading worst-case determinism for expected performance.
Probabilistic data structures
Bloom filters, HyperLogLog, and count-min sketches trade a small, mathematically bounded error probability for massive memory savings — you can't reason about them without probability.
Hashing & collisions
Understanding hash collision probability (a direct application of the birthday paradox) explains why hash table sizing and hash function quality matter.
Testing & fuzzing
Property-based testing and fuzzers generate random inputs and rely on probability to argue that enough trials will surface edge cases.
Caching & eviction
Some cache eviction and sampling strategies use randomization (e.g. random replacement, reservoir sampling) that only make sense once you can reason about expected hit rates.
Concurrency & reliability
Retry policies, exponential backoff, timeouts, and circuit breakers are all designed around the probability of transient failure.
Security & cryptography
Key generation, nonces, and salts depend on strong randomness — probability theory is what lets you reason about how hard they are to guess.
03 · How it drives model training
Machine learning is applied probability
Strip away the frameworks and a training loop is: assume a probability distribution shape, measure how well it fits the data, and nudge its parameters to fit better. Here's where probability shows up directly in the pipeline.
Loss functions are likelihoods
Cross-entropy, the default classification loss, is the negative log-likelihood of your labels under the model's predicted distribution. Minimizing loss = Maximum Likelihood Estimation.
Weight initialization
Neural network weights are typically drawn from Normal or Uniform distributions with carefully chosen variance (Xavier/He init) so gradients neither explode nor vanish.
Regularization as priors
L2 regularization is equivalent to assuming a Gaussian prior over weights in a Bayesian framing — 'shrink weights toward zero' is a probabilistic belief, not just a penalty term.
Dropout is a Bernoulli mask
Each unit is independently zeroed with probability p during training — a direct, literal application of the Bernoulli distribution as a regularizer.
Generative models
GANs, VAEs, and diffusion models are explicitly trying to learn and sample from a probability distribution that matches real data.
Uncertainty & confidence
Softmax outputs, Bayesian neural nets, and Monte Carlo dropout all let a model say 'how sure am I', which is essential for safety-critical predictions.
Naive Bayes & probabilistic classifiers
Some classifiers apply Bayes' theorem directly, assuming feature independence to make classification tractable and fast.
Evaluation & significance
Comparing two models' accuracy fairly requires statistical significance testing — otherwise you can't tell a real improvement from random noise.
04 · Topics to master
The full curriculum, with deep-dive notes
Tap a topic to expand it. Each one includes the core formula and why it actually matters for coding and machine learning — not just the theory.
- ›Every probability problem starts by naming the sample space — the full set of things that could happen — before asking about any one event inside it.
- ›The three Kolmogorov axioms (non-negativity, total probability 1, additivity over disjoint events) are the rules every distribution you'll ever code against must obey.
- ›In code, this is why you validate that your softmax output, your histogram, or your random.choice weights sum to 1 — you're enforcing P(S) = 1 by hand.
05 · Practice
Problems worth working through
Try each one on paper before revealing the solution — probability intuition is built by getting a few of these wrong first.
A fair six-sided die is rolled twice. What is the probability that the sum of the two rolls equals 8?
Hint: List the (die1, die2) pairs that sum to 8 out of all 36 equally likely pairs.
A spam filter flags 1% of all real emails and 90% of all spam. If 5% of incoming email is spam, what is P(spam | flagged)?
Hint: This is a direct Bayes' theorem setup — build P(flagged) from both the spam and not-spam paths first.
You call a flaky API with a 20% chance of failing on each independent attempt, and you retry up to 3 times. What is the probability all 3 attempts fail?
Hint: Independent failures multiply: this is a Binomial/Bernoulli chain, not an 'OR' of probabilities.
A discrete random variable X takes values 0, 1, 2 with probabilities 0.5, 0.3, 0.2. Compute E[X] and Var(X).
Hint: E[X] = Σx·P(x). For variance, compute E[X²] first, then subtract E[X]².
A server receives requests at an average rate of 4 per minute (Poisson). What is the probability it receives exactly 2 requests in a given minute?
Hint: Plug λ = 4 and k = 2 into the Poisson PMF.
Two binary classifiers, A and B, each independently predict correctly with probability 0.8. If you use 'majority of 1' (either one being right counts, since there are only 2), what's the probability at least one is correct?
Hint: It's often easier to compute the complement: both wrong.
06 · Quiz
Check your understanding
Six questions connecting probability directly to coding and ML practice. Pick an answer for each, then submit to see your score.
1. Which rule must every valid probability distribution satisfy?
2. In machine learning, minimizing cross-entropy loss is equivalent to:
3. What does the Central Limit Theorem tell you?
4. A Bloom filter's false-positive behavior is best modeled using ideas from:
5. Dropout during neural network training can be modeled as:
6. If P(A) and P(B) are independent, which is true?
Answer all 6 questions to submit.
07 · Cheat sheet
Quick reference
Every formula above, condensed. Keep this open while you code or train models.
Core rules
- Complement
- P(Aᶜ) = 1 − P(A)
- Union (general)
- P(A ∪ B) = P(A) + P(B) − P(A ∩ B)
- Conditional
- P(A|B) = P(A ∩ B) / P(B)
- Independence
- P(A ∩ B) = P(A)·P(B)
- Bayes' theorem
- P(A|B) = P(B|A)P(A) / P(B)
Expectation & spread
- Expectation
- E[X] = Σ x·P(x) (or ∫ x f(x) dx)
- Variance
- Var(X) = E[X²] − E[X]²
- Std deviation
- σ = √Var(X)
- Covariance
- Cov(X,Y) = E[XY] − E[X]E[Y]
- Linearity of E
- E[aX + bY] = aE[X] + bE[Y] (always true)
Named distributions
- Bernoulli(p)
- E = p, Var = p(1−p)
- Binomial(n,p)
- E = np, Var = np(1−p)
- Poisson(λ)
- E = λ, Var = λ
- Uniform(a,b)
- E = (a+b)/2, Var = (b−a)²/12
- Normal(μ,σ²)
- E = μ, Var = σ²
- Exponential(λ)
- E = 1/λ, Var = 1/λ²
ML-relevant
- Entropy
- H(p) = −Σ p(x) log p(x)
- Cross-entropy loss
- H(p,q) = −Σ p(x) log q(x)
- KL divergence
- KL(p‖q) = Σ p(x) log(p(x)/q(x))
- Log-likelihood
- ℓ(θ) = Σ log P(xᵢ | θ)
- Softmax
- P(class k) = eᶻᵏ / Σⱼ eᶻʲ