Roadmap: Composable Deterministic Reranking

Status (2026-10-13): Phase 1 is built behind flags (v2.3.6, July 2026) and nothing is on by default. SectionReranker and SectionRerankChain live in com.wikantik.knowledge.bundle inside wikantik-main (not as a RerankStage contract in wikantik-api, as sketched below). wikantik.bundle.rerank.chain takes an ordered list of stage names (mmr, metadata-boost, llm); it is blank by default, which is the identity chain and leaves bundle output unchanged. The built stages differ from the sketch below: mmr is a lexical MMR (MmrSectionReranker; wikantik.bundle.rerank.mmr.lambda, default 0.7), and metadata-boost is a bounded rank tie-breaker on page confidence, authoritative up and stale down (wikantik.bundle.rerank.metadata_boost.positions 1.5, .window 24), not a multiplicative score boost. The knee cutoff is a separate flag rather than a chain stage (wikantik.bundle.knee.enabled, default false; retain_ratio 0.5). The knee was measured and rejected on 2026-07-11: live recall@12 fell from 0.704 to 0.426 at retain_ratio=0.90, because contextual embeddings pack the section cosines into a band with no usable elbow. The measurement tables for mmr and metadata-boost in eval/bundle-corpus/baseline-notes.md are still blank, so neither has been accepted for default use. The scheduled evaluation that would gate them exists (see RoadmapRetrievalEvaluationObservability).

Gap statement: one fixed fusion step where competitors ship a configurable ranking pipeline

Wikantik's bundle assembly fuses BM25 and dense candidates with Reciprocal Rank Fusion and stops. There is no diversity stage, no metadata/business-rule boosting, and no principled result cutoff. The project has correctly measured and rejected LLM-based reranking — this brief is explicitly about deterministic stages, which are cheap, debuggable, and absent.

Market grounding: Vectara's chain reranker sets the composability bar

Vectara ships a chain reranker: multiple rerankers applied sequentially in one query pipeline (up to 50 stages), each with its own score cutoff and result limit. Chainable types include a multilingual neural reranker, Maximal Marginal Relevance (MMR) for diversity, a User Defined Function reranker for custom business logic over metadata, and a knee reranker that cuts the tail where scores fall off (Vectara chain reranker docs). Positioned for exactly the cases Wikantik hits: crowded topic clusters where near-duplicate sections crowd out coverage, and ranking that should weigh metadata (freshness, verification) alongside relevance.
Independent support for the LLM-rerank rejection: adversarial testing of the leading LLM-judge evaluation frameworks found none could distinguish factually wrong from factually correct retrieved contexts (Atlan, LLM evaluation frameworks compared) — consistent with Wikantik's own shuffled-input recall-collapse finding. Deterministic stages sidestep that judge problem entirely.

Current Wikantik state

Bundle pipeline: package com.wikantik.knowledge.bundle (inside the wikantik-main module — see the naming trap in RoadmapTargetArchitecture) — DefaultBundleAssemblyService, DenseChunkSectionSource (global top-K dense chunks, wikantik.bundle.dense.top_k=300), BM25 chunk hybrid via LuceneBm25ChunkIndex with its own RRF fusion config (wikantik.bundle.bm25.*), sections_per_page=20, top-12 output.
Measured dead levers, do not revisit: LLM listwise reranker (wikantik.bundle.reranker.enabled, default off — shuffled-input recall collapses), HyDE, doc2query, page-level KG rerank (zero lift even with a Claude-quality KG; the code was deleted in 2026-07).
Rich, machine-readable ranking signals already on every page: verified_at, confidence (authoritative/provisional/stale), type, cluster, date, citation staleness — only confidence is consumed by ranking, and only by the optional metadata-boost tie-breaker.
Evaluation harness to measure any change: eval/bundle-corpus/ with baseline notes; section recall@12 currently ~0.74.

Proposed direction

Post-fusion rerank stage SPI in bundle assembly: an ordered list of deterministic RerankStage implementations, config-driven (wikantik.bundle.rerank.chain=mmr,metadata-boost,knee), each with per-stage cutoff/limit semantics.
Stage 1 — MMR diversity: penalize near-duplicate sections (cosine similarity over already-computed chunk embeddings) so the top-12 covers more distinct ground. Directly addresses the sibling-page crowding failure mode seen in retrieval verification.
Stage 2 — metadata boosts: bounded multiplicative boosts from frontmatter — verified/authoritative up, stale-cited down, optional recency. Fixed signal set first; a user-defined expression language only if a real tenant asks.
Stage 3 — knee cutoff: replace the fixed top-12 with a score-elbow cut, letting thin queries return fewer, denser sections (also sharpens the coverage signal).

Phase 1 scope and acceptance criteria

First session scope: the RerankStage SPI (contract in wikantik-api, wiring at the BundleServiceWiring seam) + the MMR stage behind its flag + one harness run. Metadata-boost and knee stages are separate sessions, each gated the same way.

A stage is accepted only when all four hold:

Flag-off is byte-identical: with the stage disabled, bundle output is unchanged (regression suite green) — the SPI itself must be a provable no-op.
No recall regression: overall section recall@12 stays ≥ 0.74 on eval/bundle-corpus with the stage on.
The stage's own metric moves: for MMR, a diversity measure (distinct-page count or intra-bundle redundancy in the top-12) improves; for metadata-boost, verified/authoritative sections rank above equally-scored stale ones on a constructed test set; for knee, thin queries return fewer sections without losing gold ones.
Latency stays negligible: p95 bundle-assembly delta under ~20 ms at k=300 candidates.

A stage that fails gate 2 or 3 is rejected and recorded in the dead-levers list — the 0.74→0.68 enrichment regression is the standing proof that plausible changes hurt.

Verify before designing on these

Vectara's specifics (50 chainable stages, four reranker types) are competitive framing, not requirements — the target is three well-measured stages, not chain length.
"Sibling-page crowding suppresses coverage" is a real observed failure mode here, but its magnitude is unmeasured — the MMR diversity metric in the acceptance gate is what turns it from anecdote into a number.

Non-goals

No LLM rerank stages, ever — measured dead (shuffled-input recall collapse), independently corroborated. The SPI is deterministic by contract.
No user-defined boost expression language until a real tenant asks — fixed signal set only.
No re-attempt of the page-level KG rerank (measured zero lift; the code is gone).

Investigation starting points

[MEASURE] MMR lambda sweep; and check whether sections_per_page de-dup in the assembler already provides implicit partial MMR — measure the marginal effect, not the textbook effect.
[MEASURE] Metadata boosts at fusion (adjusting RRF inputs) vs post-fusion (adjusting fused scores) — post-fusion is more debuggable; let the harness arbitrate if both are plausible.
[AGENT] Knee detection input: RRF scores are rank-derived and quantized — likely need the true dense cosines the section candidates already carry (coverage-signal work); design the stage interface so stages can request raw scores.
[MEASURE] Latency at k=300 against the 480 RPS envelope (the O(k²) worst case is the thing to bound).