Workflow Approval and Human-in-the-Loop Pattern

The Workflow Approval and Human-in-the-Loop (HITL) Pattern models long-running operational processes that must pause durably at explicit decision gates until one or more human reviewers—or multi-stage policy boards—approve, reject, request modifications, or time out via SLA escalation. Enterprise applications rely on this pattern everywhere high-risk mutations meet organizational governance: financial disbursements, access-privilege grants, contract redlines, compliance exceptions, and irreversible actions proposed by autonomous AI agents.

Unlike an ordinary synchronous CRUD update or a short automated microservice Saga Pattern, human approvals span hours to weeks, cross server restarts and software deployments, require Separation-of-Duties (SoD) and quorum rules, and must preserve the exact snapshot of what was approved so payload tampering between submission and review is impossible. For broader orchestration and domain patterns, consult Design Patterns Hub, Enterprise Integration Hub, Order Orchestration and Sagas, and Agentic Workflow Design.


1. Durable State Machine, Quorum Rules, and Snapshot Pinning

Never track multi-step workflows with fragile boolean flags (is_approved, manager_reviewed, finance_ok). Model the approval workflow explicitly as a persisted finite-state machine tied to an immutable Payload Snapshot Hash and ordered Approval Steps.

stateDiagram-v2
    [*] --> Draft
    Draft --> PendingReview : submit(snapshotHash, idempotencyKey)
    PendingReview --> PendingReview : stepApproved(quorumRemaining > 0)
    PendingReview --> Escalated : slaTimeoutReached
    Escalated --> PendingReview : reassignedToBackupApprover
    Escalated --> Expired : hardDeadlineExceeded
    PendingReview --> ChangesRequested : requestChanges(reason)
    ChangesRequested --> Draft : revisePayload(newVersion)
    PendingReview --> Approved : finalStepApproved(quorum == 0)
    PendingReview --> Rejected : anyStepRejected(reason)
    Approved --> Executing : dispatchSideEffects(outbox)
    Executing --> Completed : downstreamConfirmed
    Executing --> CompensationRequired : irreversibleDispatchFailed

Three architectural invariants separate production-grade HITL workflows from naive status columns:

  1. Snapshot Digest Pinning (anti-TOCTOU guarantee): A requester submits a purchase order for \$4,500, the manager reviews it, and right before clicking Approve, another tab updates the line items to \$45,000. To prevent Time-of-Check to Time-of-Use (TOCTOU) approval hijacking, compute a canonical SHA-256 digest of the submitted payload (payload_digest) when entering PendingReview. Either freeze edits completely while PendingReview (requiring explicit transition back to Draft / ChangesRequested which voids prior approvals), or include expectedDigest on every approve() API call so stale reviews fail immediately with 409 DIGEST_MISMATCH.
  2. Separation of Duties (SoD) and Non-Delegable Quorum: Enforce server-side via Fine-Grained Authorization Pattern that the submitter cannot approve their own step, and in M-of-N quorum approvals (e.g., 2-of-3 security leads), ensure distinct actor_id entries rather than counting one person approving twice.
  3. Separated Business Decision vs. Side-Effect Execution: Transitioning from PendingReview to Approved commits atomically in the database alongside an Outbox Pattern execution command, moving asynchronously into Executing -> Completed. Never call an external payment wire or provisioning API synchronously inside the HTTP handler that records the human's click!
Workflow Pattern VariantTypical LatencyBest Use CasePrimary Failure Guard Required
Single-Approver GateMinutes to 2 daysManager time-off, minor expense claimsAnti-self-approval SoD + stale-digest optimistic concurrency
Sequential Multi-Stage Chain1 to 10 business daysLegal -> Finance -> VP sign-off on vendor contractsDynamic step recalculation if amount/terms change mid-flight
M-of-N Parallel QuorumHours to 3 daysProduction DB break-glass, wire disbursements \ge \$50,000Distinct reviewer constraint + quorum race lock (SELECT ... FOR UPDATE)
Agentic Tool-Call InterruptSeconds to hoursAI agent requesting destructive DB DDL, customer refund, or external emailPersisted checkpoint state, parameter diff view, and expiry TTL before resume

2. Quantifying SLA Escalation Latency and Quorum Throughput

Enterprise approval queues often fail silently when an assigned approver goes on vacation or steps are configured sequentially without timeouts. Modeling queue delay mathematically helps size SLA escalation timers and backup approver pools cleanly.

Suppose an organization processes \lambda = 120 multi-stage approvals/day across S = 3 sequential stages (Team Lead, Finance, Compliance), where each stage currently waits until a hard timeout T_{\text{escalate}} before routing to a backup pool. Let p_{\text{missed}} = 0.12 be the probability an individual primary approver misses the initial notification and triggers escalation, and let T_{\text{active}} be the median active review time (3.5\text{ hours} per stage). Expected end-to-end completion time is:

E[T_{\text{e2e}}] = \sum_{i=1}^{S} \left( (1 - p_{\text{missed},i}) \cdot T_{\text{active},i} + p_{\text{missed},i} \cdot (T_{\text{escalate},i} + T_{\text{active,backup}}) \right)

Compare T_{\text{escalate}} = 48\text{ h} versus aggressive tiered escalation (T_{\text{escalate}} = 8\text{ h} with a shared on-call reviewer group):

E[T_{\text{e2e,slow}}] = 3 \times \left( 0.88(3.5) + 0.12(48 + 3.5) \right) = 3 \times (3.08 + 6.18) = 27.78\text{ hours}
E[T_{\text{e2e,fast}}] = 2 \times \left( 0.88(3.5) + 0.12(8 + 2.0) \right) = 2 \times (3.08 + 1.20) = 8.56\text{ hours}

Parallelizing independent review stages plus reducing T_escalate from 48 h to 8 h cuts expected approval cycle time by 69.2% while preserving 100% policy coverage.


3. Database Schema and Idempotent Decision Engine

A durable approval engine separates the immutable Approval Instance + Frozen Snapshot from individual Approval Step Decisions and Escalation Deadlines, guaranteeing concurrency safety under simultaneous clicks (SELECT ... FOR UPDATE) and duplicate network retries (Idempotency Patterns).

CREATE TABLE workflow_approvals (
    tenant_id        uuid NOT NULL,
    approval_id      uuid NOT NULL DEFAULT gen_random_uuid(),
    subject_type     text NOT NULL,       -- e.g. 'PURCHASE_ORDER', 'AGENT_TOOL_CALL'
    subject_id       text NOT NULL,
    requester_id     uuid NOT NULL,
    status           text NOT NULL CHECK (status IN (
        'PENDING_REVIEW', 'ESCALATED', 'CHANGES_REQUESTED',
        'APPROVED', 'REJECTED', 'EXPIRED', 'EXECUTING', 'COMPLETED'
    )),
    payload_snapshot jsonb NOT NULL,      -- exact reviewed state rendered in UI
    payload_digest   text NOT NULL,       -- SHA-256 hex digest preventing TOCTOU edits
    required_quorum  int NOT NULL CHECK (required_quorum >= 1),
    current_step     int NOT NULL DEFAULT 1,
    escalate_at      timestamptz NOT NULL,
    expires_at       timestamptz NOT NULL,
    version          int NOT NULL DEFAULT 1,
    created_at       timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, approval_id)
);

CREATE TABLE workflow_approval_votes (
    tenant_id        uuid NOT NULL,
    vote_id          uuid NOT NULL DEFAULT gen_random_uuid(),
    approval_id      uuid NOT NULL,
    step_number      int NOT NULL,
    actor_id         uuid NOT NULL,
    decision         text NOT NULL CHECK (decision IN ('APPROVE', 'REJECT', 'REQUEST_CHANGES')),
    comment          text NOT NULL DEFAULT '',
    idempotency_key  text NOT NULL,
    decided_at       timestamptz NOT NULL DEFAULT now(),
    PRIMARY KEY (tenant_id, vote_id),
    FOREIGN KEY (tenant_id, approval_id) REFERENCES workflow_approvals (tenant_id, approval_id),
    -- Guarantees one reviewer cannot double-vote inside an M-of-N quorum step
    UNIQUE (tenant_id, approval_id, step_number, actor_id),
    UNIQUE (tenant_id, idempotency_key)
);

CREATE INDEX idx_approvals_pending_escalate
    ON workflow_approvals (escalate_at)
    WHERE status = 'PENDING_REVIEW';

Safe Decision Transition Handler (with Quorum & Outbox Resumption)

import type { PoolClient } from "pg";
import { enqueueOutboxEvent } from "../outbox/outbox";

export interface CastApprovalVoteInput {
  readonly tenantId: string;
  readonly approvalId: string;
  readonly actorId: string;
  readonly decision: "APPROVE" | "REJECT" | "REQUEST_CHANGES";
  readonly expectedDigest: string;
  readonly comment: string;
  readonly idempotencyKey: string;
}

export async function recordApprovalDecision(
  tx: PoolClient,
  input: CastApprovalVoteInput
): Promise<{ status: string; votesRecorded: number; requiredQuorum: number }> {
  // 1. Idempotency check inside the same transaction
  const dup = await tx.query<{ decision: string }>(
    `SELECT decision FROM workflow_approval_votes WHERE idempotency_key = $1`,
    [input.idempotencyKey]
  );

  // 2. Lock workflow row to serialize concurrent quorum approvals
  const { rows } = await tx.query<{
    status: string;
    requester_id: string;
    payload_digest: string;
    required_quorum: number;
    current_step: number;
    expires_at: Date;
    subject_type: string;
    subject_id: string;
  }>(
    `SELECT status, requester_id, payload_digest, required_quorum,
            current_step, expires_at, subject_type, subject_id
       FROM workflow_approvals
      WHERE approval_id = $1
        FOR UPDATE`,
    [input.approvalId]
  );

  const wf = rows[0];
  if (!wf) {
    throw Object.assign(new Error("Approval record not found"), { statusCode: 404, code: "NOT_FOUND" });
  }
  if (dup.rows.length > 0) {
    const countRes = await tx.query<{ cnt: string }>(
      `SELECT count(*)::text AS cnt FROM workflow_approval_votes
        WHERE approval_id = $1 AND step_number = $2 AND decision = 'APPROVE'`,
      [input.approvalId, wf.current_step]
    );
    return { status: wf.status, votesRecorded: Number(countRes.rows[0].cnt), requiredQuorum: wf.required_quorum };
  }

  // 3. Enforce invariant guards: state, expiry, SoD, and anti-TOCTOU snapshot hash
  if (wf.status !== "PENDING_REVIEW" && wf.status !== "ESCALATED") {
    throw Object.assign(new Error(`Cannot vote in state ${wf.status}`), { statusCode: 409, code: "INVALID_STATE" });
  }
  if (wf.expires_at.getTime() <= Date.now()) {
    throw Object.assign(new Error("Approval request has expired"), { statusCode: 409, code: "APPROVAL_EXPIRED" });
  }
  if (wf.requester_id === input.actorId) {
    throw Object.assign(new Error("Requester cannot approve or review own submission"), {
      statusCode: 403,
      code: "SOD_SELF_REVIEW_FORBIDDEN",
    });
  }
  if (wf.payload_digest !== input.expectedDigest) {
    throw Object.assign(new Error("Payload changed since review screen loaded"), {
      statusCode: 409,
      code: "DIGEST_MISMATCH",
    });
  }

  // 4. Insert vote (UNIQUE constraint prevents double voting by same actor)
  await tx.query(
    `INSERT INTO workflow_approval_votes
       (tenant_id, approval_id, step_number, actor_id, decision, comment, idempotency_key)
     VALUES ($1, $2, $3, $4, $5, $6, $7)`,
    [input.tenantId, input.approvalId, wf.current_step, input.actorId, input.decision, input.comment, input.idempotencyKey]
  );

  let nextStatus = wf.status;
  let votesRecorded = 0;

  if (input.decision === "REJECT") {
    nextStatus = "REJECTED";
  } else if (input.decision === "REQUEST_CHANGES") {
    nextStatus = "CHANGES_REQUESTED";
  } else {
    const cntRes = await tx.query<{ cnt: string }>(
      `SELECT count(*)::text AS cnt FROM workflow_approval_votes
        WHERE approval_id = $1 AND step_number = $2 AND decision = 'APPROVE'`,
      [input.approvalId, wf.current_step]
    );
    votesRecorded = Number(cntRes.rows[0].cnt);
    if (votesRecorded >= wf.required_quorum) {
      nextStatus = "APPROVED";
    }
  }

  if (nextStatus !== wf.status) {
    await tx.query(
      `UPDATE workflow_approvals SET status = $1, version = version + 1 WHERE approval_id = $2`,
      [nextStatus, input.approvalId]
    );
    await enqueueOutboxEvent(tx, {
      tenantId: input.tenantId,
      aggregateType: "WorkflowApproval",
      aggregateId: input.approvalId,
      eventType: `WorkflowApproval.${nextStatus}.v1`,
      payload: {
        approvalId: input.approvalId,
        subjectType: wf.subject_type,
        subjectId: wf.subject_id,
        decidedBy: input.actorId,
      },
    });
  }

  return { status: nextStatus, votesRecorded, requiredQuorum: wf.required_quorum };
}

4. Production Failure Modes and Human-Loop Anti-Patterns

  1. Holding HTTP connections or in-memory coroutines open for human decisions: Blocking an application worker thread, Temporal/BullMQ active lease, or browser tab while waiting for a human to respond guarantees timeouts and lost tasks on deploy. Always persist state (PENDING_REVIEW), commit cleanly, release workers, and resume asynchronously via outbox webhook/event.
  2. Double-click or simultaneous quorum race conditions: Without SELECT ... FOR UPDATE and UNIQUE (approval_id, step_number, actor_id), two approvers clicking within 20 ms—or one reviewer double-clicking—can fire duplicate downstream disbursement sagas (WorkflowApproval.APPROVED.v1). Serialize transitions with row-level locks and unique vote constraints.
  3. TOCTOU mutability during active review: Allowing PATCH /orders/:id while status === "PENDING_REVIEW" lets an attacker submit a \$100 order and bump it to \$100,000 before approval completes. Lock pending records or require explicit expectedDigest matching in both the review UI and the vote endpoint.
  4. Orphaned workflows when approvers leave the company: Hardcoding assigned_approver_user_id = 'alice' without role-group resolution (FINANCE_APPROVER_POOL), escalation timers, and admin reassignment endpoints traps business records permanently in PENDING_REVIEW. Always run a periodic SLA sweep worker (Background Job Processing) over idx_approvals_pending_escalate and pair every workflow mutation with Audit Trail and Domain History Pattern.

5. Agent Implementation & Output Contract

When an automated coding agent implements human approvals, compliance sign-offs, or agentic human-in-the-loop tool confirmation gates, enforce this structural contract:

Required Architecture & Data Elements

  1. Explicit state enum & transition guard: Define an explicit state machine (PENDING_REVIEW, ESCALATED, CHANGES_REQUESTED, APPROVED, REJECTED, EXPIRED, EXECUTING, COMPLETED) with zero implicit boolean flags.
  2. Immutable reviewed snapshot + SHA-256 digest: Store payload_snapshot and payload_digest at submission time, return payload_digest in the reviewer GET endpoint, and require expectedDigest on every POST /approve or /reject call.
  3. Dual-control & SLA sweep: Enforce requester !== actorId in database/policy checks, bind UNIQUE (approval_id, step_number, actor_id) for quorums, and emit an outbox event on terminal state entry so downstream effects execute idempotently via background workers.

Mandatory Test Scenarios to Generate

  1. Anti-TOCTOU tamper test: Submit snapshot A (digest_A), simulate stale payload digest (digest_B) on approve -> verify 409 DIGEST_MISMATCH.
  2. Quorum & SoD concurrency test: Submitter attempting approval fails 403 SOD_SELF_REVIEW_FORBIDDEN; same approver voting twice fails cleanly or returns idempotent count 1/2; two distinct eligible approvers reach 2/2 -> APPROVED and emit exactly one outbox resumption event.
  3. Expiry & escalation sweep test: Advance mock clock past escalate_at and expires_at -> verify escalation transitions and 409 APPROVAL_EXPIRED rejection for late votes.