Machine Learning for Investing: A Survey

By 2026, the application of Machine Learning (ML) to investing has definitively shifted from "phenomenological" pattern matching to Scientific Causal Discovery. For decades, quantitative researchers sought algorithms capable of predicting price trajectories by sifting through massive alternative datasets. This methodology often led to the creation of models that were highly performant in-sample but dangerously fragile out-of-sample. Today, the focus is no longer simply "What will the price be?" but rather, "What is the causal mechanism driving the return, and under what conditions does this mechanism fundamentally break down?"

This paradigm shift was necessitated by a series of high-profile quantitative failures throughout the early 2020s, where classical machine learning algorithms memorized historical correlations that abruptly dissolved during acute macroeconomic shocks. In this comprehensive survey, we explore the state-of-the-art architectures and mathematical methodologies redefining modern asset management. We will delve deeply into Causal Factor Investing, Hierarchical Risk Parity, Bayesian Neural Networks, and Graph Neural Networks for systemic risk modeling, outlining both their theoretical foundations and their real-world implementation gotchas.

1. Causal Factor Investing (The 2025 Paradigm)

The "Factor Zoo"—the academic and practitioner explosion of thousands of reported market factors—has been largely debunked as a "Factor Mirage". It is now widely accepted that traditional factor discovery suffered extensively from p-hacking, selection bias, and a lack of rigorous out-of-sample robustness testing. Researchers were repeatedly discovering statistical artifacts rather than genuine, persistent economic premiums.

A. The Causal ML Workflow and Protocol

Following the seminal research of Marco Lopez de Prado (2025) and integrating Judea Pearl’s structural causal models, leading quantitative shops have adopted a strict Causal ML workflow to prevent spurious discoveries. This methodology forces the researcher to hypothesize a structural economic mechanism before even looking at the historical data.

The core of this workflow relies on three foundational pillars:

Causal Graphing (DAGs): Mapping variable interdependencies using Directed Acyclic Graphs. Before calculating any correlations, researchers construct a DAG to explicitly state their assumptions about how macroeconomic and firm-specific variables influence one another.
Do-Calculus: Using graph surgery to select appropriate control variables and avoid "collider bias." If an algorithm controls for a variable that is actually influenced by both the treatment and the outcome, it can accidentally create a mathematical correlation where none exists in reality. This was a fatal flaw in many older multi-factor regression models.
Double Machine Learning (DML): Using advanced ML techniques to estimate treatment effects while strictly controlling for high-dimensional confounders. By employing cross-fitting techniques, DML allows researchers to use non-parametric machine learning models (like Random Forests or Gradient Boosted Trees) to isolate the true causal effect of a financial factor from the noise of a highly complex market environment. Crucially, DML relies on the property of Neyman orthogonality, which ensures that small, inevitable errors in estimating the nuisance parameters (the confounding variables) do not systematically bias the estimate of the primary causal parameter.

By moving to this causal framework, funds have successfully eliminated historically misleading signals, saving billions in misallocated capital. For example, understanding true causality prevents automated systems from buying during a severe systemic liquidity drain simply because historical data superficially suggested a "buy the dip" mean-reversion pattern.

2. Advanced Portfolio Optimization: Hierarchical Risk Parity (HRP)

The most significant theoretical breakthrough in recent portfolio theory was the formal validation of Hierarchical Risk Parity (HRP). While Harry Markowitz's Mean-Variance Optimization (MVO) dominated the 20th century, its extreme sensitivity to small changes in expected returns and covariance matrices earned it the notorious moniker of the "error maximizer."

A. Overcoming Markowitz's Instability

In Risk Magazine (Jan 2025), Lopez de Prado and his colleagues provided the first analytical proof that HRP is significantly less noisy and more stable than classical MVO.

In MVO, calculating optimal portfolio weights requires the inversion of a covariance matrix. When financial assets are highly correlated—which happens frequently during market sell-offs—the covariance matrix becomes mathematically ill-conditioned. Its inversion subsequently magnifies tiny estimation errors. HRP ingeniously bypasses matrix inversion entirely by employing graph theory and unsupervised machine learning clustering techniques.

The mathematical formulation for the distance matrix in HRP relies on transforming the empirical correlation matrix into a true metric space:

D(i, j) = \sqrt{\frac{1}{2} (1 - \rho_{i,j})}

Where $D(i, j)$ represents the calculated distance between asset $i$ and asset $j$ , and $\rho_{i,j}$ is their estimated Pearson or Spearman correlation. Using this distance metric, a hierarchical clustering algorithm groups similar assets into a tree structure known as a dendrogram.

Capital is then distributed top-down through this tree structure. At each branch, the algorithm allocates capital inversely proportional to the cluster's variance:

\alpha = \frac{V_2}{V_1 + V_2}

Where $V_1$ and $V_2$ are the variances of the two sub-clusters, and the allocation to the first sub-cluster is $\alpha$ , and the second is $1 - \alpha$ .

The Analytical Update: We now possess derived analytical values for the "noise" of allocation weights coming from estimated covariance matrices. The proofs demonstrate that the expected variance of the portfolio weights under HRP is bounded tightly, whereas MVO weights can theoretically oscillate towards infinity as the condition number of the covariance matrix worsens.
Finding: HRP's tree-based clustering approach effectively regularizes the covariance matrix, preventing the "Optimizer's Curse." Markowitz optimization essentially treats estimated returns as exact, leading the optimizer to aggressively exploit the largest estimation errors. Because HRP ignores expected returns (which are notoriously difficult to forecast) and relies solely on the topological structure of market risk, it is structurally robust. When a portfolio manager allocates a massive $500M to a strategy, they can be mathematically assured that a minor 1% error in covariance estimation will not trigger a $250M rebalancing requirement, saving heavily on transaction costs and slippage.

3. Position Sizing: Bayesian Neural Networks (BNNs)

Traditional ML architectures provide singular point estimates (e.g., "The predicted return is exactly 5.2%"). However, standard neural networks suffer from severe overconfidence. A standard ReLU network will linearly extrapolate its predictions into unfamiliar territory, often outputting a softmax probability of 99% for entirely out-of-distribution inputs. A 5% return prediction derived from dense, clean historical data is fundamentally different from a 5% prediction extrapolated into an unobserved, highly volatile macroeconomic regime.

Advanced 2026 models mitigate this by utilizing Bayesian Neural Networks (BNNs). BNNs provide a probability distribution over the model weights themselves, yielding an explicit, mathematically sound measure of model confidence.

Epistemic vs. Aleatoric Uncertainty

BNNs update their internal weights as full probability distributions rather than single scalar values using Bayes' theorem:

P(W | D) = \frac{P(D | W) P(W)}{P(D)}

Where $W$ represents the weights of the neural network and $D$ represents the observed market data.

This Bayesian formulation allows quantitative trading systems to decompose uncertainty into two distinct types:

Aleatoric Uncertainty: The inherent, irreducible noise in the financial markets (e.g., the random walk component of daily micro-structure movements).
Epistemic Uncertainty: Model ignorance. This uncertainty spikes when the model encounters data unlike anything it observed during its training phase.

Uncertainty-Aware Position Sizing: Quants now utilize Epistemic Uncertainty to dynamically and explicitly size positions. If a model predicts a high positive return but exhibits high variance in its internal weight distributions, the position size is automatically penalized and downsized. For instance, if a traditional model signals a $50K per contract trade based on a momentum breakout, but the BNN outputs a wide epistemic confidence interval due to unprecedented VIX levels, the execution engine might throttle the trade down to a mere $5K exposure. This effectively prevents the model from confidently marching off a cliff during unprecedented macroeconomic regimes.

4. Supply Chain Risk: Graph Neural Networks (GNNs)

By 2025, advanced quantitative research had successfully applied Graph Neural Networks (GNNs) to map and model systemic financial risk and fundamental vulnerabilities across global equities. While traditional tabular machine learning models look at a company in isolation, GNNs model the global economy accurately as a complex, interconnected web of dependencies.

Adjacency Matrices: Companies are represented as individual nodes in a massive graph. Edges represent supply chain dependencies, co-ownership structures, joint ventures, or overlapping board intersections.
Message Passing Framework: Critical information (e.g., financial distress, sudden revenue growth, localized geographic shocks) is passed along these edges.

The update rule for a node's representation in a basic Graph Convolutional Network (GCN) architecture can be expressed mathematically as:

H^{(l+1)} = \sigma \left( \tilde{D}^{-\frac{1}{2}} \tilde{A} \tilde{D}^{-\frac{1}{2}} H^{(l)} W^{(l)} \right)

Where $\tilde{A}$ is the adjacency matrix with added self-connections, $\tilde{D}$ is the degree matrix, $H^{(l)}$ are the node features at layer $l$ , $W^{(l)}$ is the trainable weight matrix for that specific layer, and $\sigma$ is a non-linear activation function.

Risk Propagation and Real-World Limitations: This mathematical structure allows the neural network to "learn" how economic shocks propagate through the physical and financial world. If a tier-2 semiconductor supplier in Taiwan faces a localized disruption (a "Hormuz Shock" precursor), the GNN propagates this hidden risk through the graph. It can identify a vulnerable consumer electronics firm in the US—whose direct tier-1 suppliers appear perfectly fine but are fundamentally dependent on the disrupted node—months before the earnings impact is publicly reported. Funds utilizing GNNs were able to successfully short companies with hidden $1.2B exposures to disrupted maritime routes long before traditional equity analysts updated their discounted cash flow models. However, practitioners must note that GNNs require high-quality, high-fidelity relational data. Hidden supply chain links are often deeply proprietary, requiring advanced imputation methods to approximate missing edges.

5. Execution and Tactical Allocation (RL)

Reinforcement Learning (RL) has moved well beyond simulated toy environments to dominate Execution and Tactical Allocation in live, adversarial markets.

SAPPO (Sentiment-Augmented Proximal Policy Optimization): Kirtac & Germano (July 2025) demonstrated that RL agents using Large Language Model (LLM)-derived sentiment arrays in their advantage function drastically outperform baseline models. They reported achieving out-of-sample Sharpe Ratios of 1.90 versus 1.55 for traditional Deep RL implementations. By actively penalizing trades executed during periods of opaque or highly volatile news sentiment, the RL agent naturally learns to step back and preserve capital when the market environment becomes hostile.
Hierarchical RL and the Sim-to-Real Gap: Modern execution engines use a hierarchical, multi-agent architecture to combat the infamous "sim-to-real" gap. Agents trained purely on historical limit order book data often exploit phantom liquidity that wouldn't actually exist in live trading, because their own large orders would alter the state of the book. To solve this, a "Master" agent makes low-frequency decisions regarding asset allocation and target inventory holding (Medium-Term). Simultaneously, specialized high-frequency "Sub-agents" operate at the microsecond level. These Sub-agents are tasked solely with acquiring the Master's target inventory while actively minimizing slippage, market impact, and adverse selection. This strict separation of concerns prevents high-frequency microstructure noise from confusing the macro-level strategic planner.

6. Case Study: The 2026 Iran War Shock

The theoretical superiority of these modern ML architectures was empirically proven during the unexpected February 2026 kinetic escalation in the Middle East.

When the geopolitical conflict abruptly intensified, traditional Markowitz-style optimizers failed catastrophically due to a violent "Correlation Breakdown." In a sheer panic, correlations between traditionally uncorrelated asset classes (like equities and certain fixed-income instruments) surged rapidly toward 1.0; almost all risk assets dropped in tandem. Portfolios built heavily on historical covariance assumptions suffered massive, outsized drawdowns because the mathematical foundation of their perceived diversification vanished literally overnight.

The ML Alternative: Portfolios managed using a robust combination of Hierarchical Risk Parity (HRP) and Minimum Sentiment Connectedness (MSC) maintained their structural integrity. By continuously analyzing global alternative data streams, the causal models identified a "Systemic Chokepoint" sentiment signature approximately 72 hours before the Hormuz escalation became fully priced into the broader equity markets.

Because the Bayesian Neural Networks detected surging epistemic uncertainty in the technology sector (refusing to extrapolate past bull-market behaviors into the new regime), and the GNNs flagged extreme, cascading vulnerability to the global energy supply chain, these models automatically and dynamically shifted capital allocations. They systematically reduced high-beta technology exposure and scaled smoothly into defensive commodities, sovereign bonds, and localized domestic energy producers. While traditional benchmarks suffered a rapid -15% index drop in a single week, the modernized Causal ML portfolios mitigated the losses to a highly manageable -3%, demonstrating unequivocally that the era of Scientific Causal Discovery in finance had fully arrived and was critical for capital preservation.

See Also: