Computer Vision is the scientific and engineering discipline dedicated to enabling algorithmic systems to extract high-level semantic understanding, spatial geometries, and structural inferences from digital images, video streams, and multi-modal sensory inputs.
This article provides the mathematical, structural, and algorithmic foundations of modern computer vision: from fundamental spatial filtering and classical convolutional neural networks (CNNs) to modern Vision Transformers (ViTs), object detection pipelines, and 3D neural radiance fields (NeRFs).
A digital grayscale image is represented mathematically as a discrete 2D spatial function I(x, y) \in [0, 255]^{H \times W}, where x \in \{1, \dots, W\} and y \in \{1, \dots, H\}. For RGB color images, I(x, y, c) \in [0, 255]^{H \times W \times 3}.
Spatial filtering transforms an input image I by sliding a localized kernel matrix K \in \mathbb{R}^{(2k+1) \times (2k+1)} across spatial coordinates:
2D Convolution Operation:
Input Feature Map (5x5) Kernel (3x3) Output Map (3x3)
[ 1 2 3 0 1 ] [ . . . ]
[ 0 1 2 3 1 ] * [ 1 0 -1 ] = [ . 12 . ]
[ 1 0 1 2 0 ] [ 2 0 -2 ] [ . . . ]
[ 2 1 0 1 2 ] [ 1 0 -1 ] (Sobel-X Filter)
[ 0 1 2 1 0 ]
Image gradients identify sharp intensity transitions (edges):
Gradient Magnitude: \|\nabla I\| = \sqrt{(\partial_x I)^2 + (\partial_y I)^2}; Gradient Orientation: \theta = \arctan\left(\frac{\partial_y I}{\partial_x I}\right).
Convolutional Neural Networks encode translation equivariance (f(g(x)) = g(f(x))) and local spatial parameter sharing.
+-------------------------------------------------------------------------------+
| CLASSICAL CNN ARCHITECTURAL PROGRESSION |
+-------------------------------------------------------------------------------+
| LeNet-5 (1998) | AlexNet (2012) | VGG-16 (2014) | ResNet-50 (2015) |
| - 5x5 Conv layers | - ReLU activation | - 3x3 Conv stacks | - Residual Skip |
| - Average Pooling | - Max Pooling | - Deep receptive | Connections |
| - 60K Parameters | - Dropout, GPU | fields | - Solves Vanishing|
| | - 60M Parameters | - 138M Parameters | Gradients (100+L|
+-------------------+-------------------+-------------------+-------------------+
In deep feedforward networks, standard stacked non-linear layers suffer from vanishing and exploding gradients. The Residual Network (ResNet) introduces identity shortcut connections:
where \mathbf{x} and \mathbf{y} are input and output vectors, and \mathcal{F} is the residual mapping to be learned.
Residual Bottleneck Block:
Input x
|
+-----+-----+
| | (Identity Shortcut)
v |
[ 1x1 Conv, BN, ReLU ]
| |
[ 3x3 Conv, BN, ReLU ]
| |
[ 1x1 Conv, BN ] |
| |
v |
( + ) <--------+
|
[ReLU]
|
Output y
During backpropagation, the gradient with respect to input \mathbf{x} preserves an additive identity term:
guaranteeing that gradient signals propagate unattenuated through hundreds of layers.
Vision Transformers (ViT) discard spatial convolutions in favor of global multi-head self-attention applied directly to sequences of non-overlapping image patches.
Vision Transformer (ViT) Processing Pipeline:
Image (224x224x3)
|
[ Patch Extraction (16x16) ] ---> N = (224/16)² = 196 Patches
|
[ Linear Projection to D-dim ] -> Matrix X ∈ ℝ^{196 x D}
|
[ Prepend [CLS] Token + Add Positional Embeddings ] -> Z₀ ∈ ℝ^{197 x D}
|
[ L × Transformer Encoder Blocks (LayerNorm, Multi-Head Self-Attention, MLP) ]
|
[ Extract [CLS] Head Output ] ---> MLP Classification Head
For an input sequence matrix Z \in \mathbb{R}^{N \times D}, Query (Q), Key (K), and Value (V) projections are computed via learned weight matrices W_Q, W_K, W_V \in \mathbb{R}^{D \times d_k}:
While standard ViT has \mathcal{O}(N^2) computational complexity with respect to image resolution, hierarchical architectures like the Swin Transformer compute self-attention within local shifted windows, restoring linear \mathcal{O}(N) complexity.
Object detection requires predicting both categorical classification labels and spatial 2D bounding boxes (x_{\text{center}}, y_{\text{center}}, w, h).
+-------------------------------------------------------------------------------+
| OBJECT DETECTION PARADIGMS |
+-------------------------------------------------------------------------------+
| Paradigm | Representative Models | Key Mechanism |
+---------------------+-----------------------+---------------------------------+
| Two-Stage Detectors | Faster R-CNN, Mask R-CNN| Region Proposal Network (RPN) + |
| | | RoIAlign feature extraction |
| One-Stage Detectors | YOLOv8/v9, RetinaNet, | Dense single-pass anchor/anchor-|
| | SSD | free grid predictions |
| Transformer-Based | DETR, Deformable DETR | Bipartite Hungarian matching + |
| | | learned object queries |
+---------------------+-----------------------+---------------------------------+
One-stage dense detectors suffer from extreme class imbalance between foreground objects and millions of easy background bounding box candidates. Focal Loss dynamically down-weights easy examples:
where p_t is model estimated probability for the ground truth class, and \gamma \ge 0 is the tunable focusing parameter (typically \gamma = 2.0).
Recovering 3D geometry from 2D images relies on projective geometry and neural volume rendering.
Pinhole Camera Projective Geometry:
World Coordinates (X_w, Y_w, Z_w)
|
[ Extrinsic Matrix: [R | t] ] (Rotation & Translation from World to Camera Frame)
v
Camera Coordinates (X_c, Y_c, Z_c)
|
[ Intrinsic Matrix: K ] (Focal lengths f_x, f_y, principal point c_x, c_y)
v
Pixel Coordinates (u, v, 1)ᵀ = (1/Z_c) · K · [R | t] · (X_w, Y_w, Z_w, 1)ᵀ
A Neural Radiance Field represents a continuous 3D scene as a multi-layer perceptron (MLP):
mapping spatial 3D location \mathbf{x} = (x, y, z) and 2D viewing direction \mathbf{d} = (\theta, \phi) to emitted color \mathbf{c} = (r, g, b) and volume density \sigma.
Volume Rendering Along Ray r(t) = o + t·d:
Ray from Camera Origin (o)
\
\----[t₁]------[t₂]------[t₃]------[tₙ]----->
σ₁, c₁ σ₂, c₂ σ₃, c₃ σₙ, cₙ
The expected color C(\mathbf{r}) of camera ray \mathbf{r}(t) = \mathbf{o} + t \mathbf{d} between near and far bounds t_n, t_f is computed via numerical quadrature:
where Transmittance T(t) = \exp\left(-\int_{t_n}^t \sigma(\mathbf{r}(s)) ds\right) denotes the probability that the ray travels from t_n to t without hitting any intervening particles.
Modern advancements (such as 3D Gaussian Splatting) replace implicit neural ray-marching with explicit 3D anisotropic Gaussians rasterized via tile-based CUDA kernels, achieving real-time rendering at >100\,\text{FPS}.
+-------------------+-------------------+--------------------+------------------------+
| Task Domain | Canonical Loss | Leading Models | Key Evaluation Metric |
+-------------------+-------------------+--------------------+------------------------+
| Image Classif. | Cross-Entropy | ConvNeXt, ViT-H/14 | Top-1 / Top-5 Accuracy |
| Object Detection | GIoU / CIoU + CE | YOLOv9, DINO-DETR | mAP@[0.5:0.95] |
| Semantic Segm. | Cross-Entropy+Dice| SegFormer, Mask2Former| Mean IoU (mIoU) |
| Instance Segm. | Mask BCE + Box L1 | Mask R-CNN, SAM | Mask mAP |
| Novel View Synth. | Photometric L2/L1 | Instant-NGP, 3DGS | PSNR, SSIM, LPIPS |
+-------------------+-------------------+--------------------+------------------------+