Data Validation with Pandera and Great Expectations

Code gets tests; data mostly gets hope. Yet the majority of production ML incidents are data incidents — an upstream schema change, a unit switch, a silent null flood — that flow through typed, green-tested code and corrupt results downstream. Data validation frameworks make expectations about data executable: Pandera for lightweight in-code dataframe contracts, Great Expectations (GX) for suite-based validation with reporting. This page covers both, how to choose, and — the part frameworks can't decide for you — where the checks belong.

Pandera: dataframe schemas as code

Pandera expresses a schema as a Python class (or object) and validates pandas/Polars frames against it — types, ranges, nullability, uniqueness, and cross-column logic:

import pandera.pandas as pa
from pandera.typing import Series

class Orders(pa.DataFrameModel):
    order_id: Series[str] = pa.Field(unique=True)
    amount: Series[float] = pa.Field(gt=0, le=100_000)
    country: Series[str] = pa.Field(isin=["DE", "FR", "US"])
    created_at: Series["datetime64[ns]"]

    @pa.dataframe_check
    def refund_not_before_order(cls, df):
        return (df["refunded_at"].isna()) | (df["refunded_at"] >= df["created_at"])

validated = Orders.validate(df, lazy=True)   # lazy=True collects ALL failures

Key practices: lazy=True so one run reports every violation rather than dying on the first; @pa.check_types on function signatures turns schemas into typed contracts at function boundaries; schema.example() generates property-test data via Hypothesis integration. Pandera's sweet spot is developer-owned validation — contracts living next to the transformation code they guard, versioned in the same repo, running in unit tests and pipeline steps alike.

Great Expectations: suites, checkpoints, and data docs

GX organizes validation as expectation suites (declarative JSON/Python collections like expect_column_values_to_not_be_null, expect_column_mean_to_be_between) executed by checkpoints against batches of data, producing Data Docs — browsable HTML reports of what passed and failed. Two capabilities distinguish it from Pandera:

Profiling-assisted authoring — generate a candidate suite from a known-good batch, then prune; this bootstraps coverage over wide tables quickly.
Stakeholder-facing results — Data Docs give analysts and data owners a shared artifact; validation stops being something only engineers can read.

The cost is machinery: contexts, datasources, and stores mean a real adoption curve, and suites live as configuration that must be governed. GX fits organizational validation — many tables, mixed audiences, warehouse-level checks — where Pandera fits code-level contracts.

Choosing between them (and the neighbors)

Pandera — Python-native pipelines, contracts owned by the same engineers who own the code, fast unit-test-style feedback.
Great Expectations — cross-team data platforms, reporting requirements, non-engineer stakeholders.
dbt tests — if the transformation layer is dbt, its built-in and package tests (unique, not_null, accepted_values, dbt-expectations) cover warehouse tables with the least new machinery.
Soda / data observability platforms — continuous monitoring with anomaly detection rather than hand-written assertions; complementary, not a replacement (see Data Observability).

Mixing is normal: Pandera at function boundaries, dbt tests in the warehouse, an observability layer watching trends.

Where validation belongs in a pipeline

Checks have a natural placement hierarchy, in descending order of value per check:

Ingest boundary — validate third-party and upstream data the moment it enters your system; this is where the least-controlled data arrives and where quarantine is cheapest.
Contract boundaries between teams/steps — producer validates outputs (shift-left, per Shift Left Data Engineering); consumers validate assumptions they depend on.
Model input — the final gate before training or inference: feature ranges, null rates, category sets. Training-time validation doubles as drift documentation for serving time.

Validating only at the end ("check the dashboard numbers") localizes nothing; a failure there starts an archaeology project instead of pointing at the offending step.

Handling failures: severity tiers, quarantine, and alerts

A validation failure needs a policy, not just a red X. The pattern that scales is severity-tiered:

Hard failures (schema break, corrupt keys, impossible values) → stop the pipeline; downstream running on garbage costs more than lateness.
Soft failures (elevated null rate, distribution drift) → proceed, alert, and record; block only on trend confirmation.
Row-level quarantine — route failing rows to a side table with the failure reason and process the clean remainder; this keeps one bad partner file from stopping the world, while the quarantine table's growth rate becomes itself a monitored metric.

Whatever the tier, failures must land in an owned channel with the failing check, batch identity, and sample rows — an unactionable validation alert trains people to ignore validation.