Machine learning is applied mathematics. Every model, every training algorithm, every evaluation metric rests on a small number of mathematical disciplines. You can use ML frameworks without understanding the math — many practitioners do — but you cannot debug a failing model, design a new architecture, or reason about why a technique works or fails without it.
This article covers the essential mathematical concepts and, critically, shows how each one connects to specific ML techniques. The goal is not to replace a textbook but to give you a map: which math matters, where it appears, and why it is worth understanding.
ML operates on data, and data is represented as vectors and matrices. Linear algebra is the language in which virtually every ML computation is expressed.
A single data point — an image, a sentence, a user profile — is represented as a vector. A dataset is a matrix where each row is a data point and each column is a feature. A batch of images is a tensor (a higher-dimensional generalization of matrices).
Why this matters: when a framework documentation says a layer expects input of shape (batch_size, seq_len, embed_dim), it is describing a 3D tensor. Understanding shapes is understanding linear algebra.
The core operation in neural networks is matrix multiplication. A fully connected layer computes y = Wx + b, where W is a weight matrix, x is the input vector, and b is a bias vector. This single operation — repeated billions of times with different weights — is what neural networks do.
| ML Technique | How Matrix Multiplication Appears |
|---|---|
| Fully connected layers | Direct: output = input × weights |
| Convolutional layers | Convolution is equivalent to matrix multiplication with a structured (Toeplitz) matrix |
| Attention mechanism | Query × Key^T computes attention scores; scores × Value produces the output |
| PCA | Eigendecomposition of the covariance matrix identifies principal components |
| Recommendation systems | User-item interaction as matrix factorization |
GPU acceleration exists because matrix multiplication is massively parallelizable. The entire hardware ecosystem of ML — NVIDIA GPUs, Google TPUs, Apple Neural Engines — is fundamentally optimized for this one operation.
An eigenvector of a matrix A is a vector v that, when multiplied by A, only changes in scale: Av = λv, where λ is the eigenvalue. This decomposition reveals the fundamental structure of a linear transformation.
In ML practice:
SVD factorizes any matrix M into M = UΣV^T, where U and V are orthogonal matrices and Σ is diagonal with non-negative entries (the singular values). SVD generalizes eigendecomposition to non-square matrices.
In ML practice:
ML models learn to map inputs into vector spaces where geometric relationships encode semantic relationships. Word embeddings (Word2Vec, GloVe) place semantically similar words near each other in vector space. The famous result that king - man + woman ≈ queen is a statement about vector arithmetic in embedding space.
Modern transformer models learn contextual embeddings — the same word gets different vectors depending on context. These embedding spaces are the internal representations where all of the model's "understanding" lives.
If linear algebra describes what a model computes, calculus describes how it learns. Training a neural network is an optimization problem solved by calculus. See CalculusRefreshForCS for a targeted engineer-focused guide.
The derivative of a function tells you how the output changes as the input changes. For a function of many variables (like a loss function with millions of parameters), the gradient is the vector of all partial derivatives — it points in the direction of steepest increase.
Gradient descent — the workhorse of ML training — updates parameters by moving in the opposite direction of the gradient:
θ_new = θ_old - α × ∇L(θ)
where θ is the parameter vector, α is the learning rate, and ∇L(θ) is the gradient of the loss function.
A neural network is a composition of functions: layer 1 feeds into layer 2 feeds into layer 3, and so on. The chain rule from calculus tells us how to compute derivatives of composed functions:
d/dx (f(g(x))) = f'(g(x)) × g'(x)
Backpropagation is simply the chain rule applied systematically through the layers of a neural network. Starting from the loss at the output, gradients flow backward through each layer, and each layer's weights are updated based on their contribution to the error.
| Concept | Mathematical Basis | ML Application |
|---|---|---|
| Gradient descent | First-order derivatives | Training any differentiable model |
| Backpropagation | Chain rule for composed functions | Computing gradients in neural networks |
| Learning rate schedules | Function behavior near optima | Cosine annealing, warmup, step decay |
| Second-order methods | Hessian (matrix of second derivatives) | Adam optimizer approximates curvature information |
| Vanishing gradients | Repeated multiplication of small derivatives | Why deep networks were hard to train before residual connections |
When gradients flow backward through many layers, they are repeatedly multiplied by weight matrices and activation derivatives. If these factors are consistently less than 1, gradients shrink exponentially (vanishing). If greater than 1, they grow exponentially (exploding).
This is why deep networks were difficult to train before key innovations:
ML models operate in a world of uncertainty — noisy data, incomplete information, stochastic training. Probability theory provides the framework for reasoning rigorously about this uncertainty.
A probability distribution describes the likelihood of different outcomes. Key distributions in ML:
P(A|B) = P(B|A) × P(A) / P(B)
Bayes' theorem relates the probability of a hypothesis given evidence to the probability of the evidence given the hypothesis. It is the foundation of:
Given observed data, MLE finds the parameters that make the observed data most probable. Training a neural network with cross-entropy loss is maximum likelihood estimation: you are finding the weights that maximize the probability of the correct labels given the inputs.
Concrete connection: The cross-entropy loss function used in classification is the negative log-likelihood of the categorical distribution defined by the model's softmax outputs. Minimizing cross-entropy = maximizing likelihood.
The expected value (mean) of a random variable is its long-run average. Variance measures how spread out the values are. These concepts appear everywhere:
The central limit theorem says that the average of many independent random variables approaches a Gaussian distribution regardless of the individual distributions. This is why stochastic gradient descent works: even though individual gradient estimates (from single examples) are noisy, the average gradient over a mini-batch is a reliable estimate of the true gradient. Larger batches give more reliable gradients (lower variance) but provide less regularization and use more memory.
Training a model means finding parameters that minimize a loss function. This is an optimization problem, and the theory of optimization underlies every training algorithm. Detailed implementations can be found in OptimizationAlgorithms.
A convex function has a single global minimum — any local minimum is the global minimum. Linear regression and logistic regression have convex loss landscapes, guaranteeing that gradient descent finds the optimal solution.
Neural networks have non-convex loss landscapes with potentially trillions of local minima, saddle points, and flat regions. Yet gradient descent works remarkably well in practice. Current understanding suggests that in high-dimensional spaces, most local minima have loss values close to the global minimum, and saddle points (not local minima) are the primary obstacle.
| Algorithm | Key Idea | When to Use |
|---|---|---|
| SGD | Update using gradient from a single example or mini-batch | Baseline; still competitive with tuning |
| SGD + Momentum | Accumulate a running average of gradients to dampen oscillations | Standard for computer vision (ResNets, etc.) |
| Adam | Adaptive per-parameter learning rates using first and second moment estimates | Default choice; works well out of the box |
| AdamW | Adam with decoupled weight decay (proper L2 regularization) | Standard for transformer training |
| LBFGS | Quasi-Newton method using curvature information | Small models where exact line search is feasible |
Adam's adaptive learning rates are derived from estimates of the gradient's first moment (mean) and second moment (uncentered variance). This is probability and optimization working together.
The learning rate α controls the step size in gradient descent. Too large: training diverges. Too small: training is painfully slow or gets stuck. Modern practice uses learning rate schedules:
The mathematical intuition: near the beginning of training, the loss landscape is poorly conditioned and large steps cause instability. As training progresses, the model approaches a minimum where smaller steps are needed for precision.
Regularization prevents overfitting by adding constraints to the optimization:
Information theory, founded by Claude Shannon, provides tools for quantifying information, uncertainty, and the quality of probabilistic predictions. See InformationTheory for a foundational survey.
Entropy measures the uncertainty in a random variable:
H(X) = -Σ p(x) log p(x)
A fair coin has entropy 1 bit. A biased coin (90% heads) has entropy 0.47 bits — it is more predictable, hence carries less information per flip.
In ML: The entropy of a classifier's output distribution indicates its confidence. A uniform distribution over 10 classes (entropy = log₂(10) ≈ 3.32 bits) means total uncertainty. A distribution concentrated on one class (entropy near 0) means high confidence.
Cross-entropy measures the difference between two probability distributions:
H(p, q) = -Σ p(x) log q(x)
where p is the true distribution and q is the model's predicted distribution.
This is the standard classification loss function. When you train a neural network classifier, you are minimizing the cross-entropy between the true labels (one-hot vectors) and the model's softmax predictions. Cross-entropy equals entropy plus KL divergence: H(p, q) = H(p) + D_KL(p || q). Since the true label entropy H(p) is constant, minimizing cross-entropy is equivalent to minimizing KL divergence — making the model's predictions as close as possible to the true distribution.
Kullback-Leibler divergence measures how one probability distribution differs from another:
D_KL(p || q) = Σ p(x) log(p(x) / q(x))
In ML practice:
Modern ML research is powered by high-performance open-source libraries that implement these mathematical foundations.
grad), vectorize (vmap), and JIT-compile (jit) to GPU/TPU.Consider training a neural network to classify images. Every mathematical concept above plays a role:
These are not separate concerns — they are different lenses on the same system. Understanding them together is what separates practitioners who can build ML systems from those who can only run them.
For practitioners wanting to build mathematical fluency:
You do not need a math degree. You need fluency with the specific concepts that appear in ML — and this article maps out which those are.