Roadmap: Composable Deterministic Reranking

Gap statement: one fixed fusion step where competitors ship a configurable ranking pipeline

Wikantik's bundle assembly fuses BM25 and dense candidates with Reciprocal Rank Fusion and stops. There is no diversity stage, no metadata/business-rule boosting, and no principled result cutoff. The project has correctly measured and rejected LLM-based reranking — this brief is explicitly about deterministic stages, which are cheap, debuggable, and absent.

Market grounding: Vectara's chain reranker sets the composability bar

Vectara ships a chain reranker: multiple rerankers applied sequentially in one query pipeline (up to 50 stages), each with its own score cutoff and result limit. Chainable types include a multilingual neural reranker, Maximal Marginal Relevance (MMR) for diversity, a User Defined Function reranker for custom business logic over metadata, and a knee reranker that cuts the tail where scores fall off (Vectara chain reranker docs). Positioned for exactly the cases Wikantik hits: crowded topic clusters where near-duplicate sections crowd out coverage, and ranking that should weigh metadata (freshness, verification) alongside relevance.
Independent support for the LLM-rerank rejection: adversarial testing of the leading LLM-judge evaluation frameworks found none could distinguish factually wrong from factually correct retrieved contexts (Atlan, LLM evaluation frameworks compared) — consistent with Wikantik's own shuffled-input recall-collapse finding. Deterministic stages sidestep that judge problem entirely.

Current Wikantik state

Bundle pipeline: package com.wikantik.knowledge.bundle (inside the wikantik-main module — see the naming trap in RoadmapTargetArchitecture) — DefaultBundleAssemblyService, DenseChunkSectionSource (global top-K dense chunks, wikantik.bundle.dense.top_k=300), BM25 chunk hybrid via LuceneBm25ChunkIndex with its own RRF fusion config (wikantik.bundle.bm25.*), sections_per_page=20, top-12 output.
Measured dead levers, do not revisit: LLM listwise reranker (wikantik.bundle.reranker.enabled, default off — shuffled-input recall collapses), HyDE, doc2query, page-level KG rerank (zero lift even with a Claude-quality KG; dormant at boost=0).
Rich, machine-readable ranking signals already on every page: verified_at, confidence (authoritative/provisional/stale), type, cluster, date, citation staleness — unused by ranking today.
Evaluation harness to measure any change: eval/bundle-corpus/ with baseline notes; section recall@12 currently ~0.74.

Proposed direction

Post-fusion rerank stage SPI in bundle assembly: an ordered list of deterministic RerankStage implementations, config-driven (wikantik.bundle.rerank.chain=mmr,metadata-boost,knee), each with per-stage cutoff/limit semantics.
Stage 1 — MMR diversity: penalize near-duplicate sections (cosine similarity over already-computed chunk embeddings) so the top-12 covers more distinct ground. Directly addresses the sibling-page crowding failure mode seen in retrieval verification.
Stage 2 — metadata boosts: bounded multiplicative boosts from frontmatter — verified/authoritative up, stale-cited down, optional recency. Fixed signal set first; a user-defined expression language only if a real tenant asks.
Stage 3 — knee cutoff: replace the fixed top-12 with a score-elbow cut, letting thin queries return fewer, denser sections (also sharpens the coverage signal).

Phase 1 scope and acceptance criteria

First session scope: the RerankStage SPI (contract in wikantik-api, wiring at the BundleServiceWiring seam) + the MMR stage behind its flag + one harness run. Metadata-boost and knee stages are separate sessions, each gated the same way.

A stage is accepted only when all four hold:

Flag-off is byte-identical: with the stage disabled, bundle output is unchanged (regression suite green) — the SPI itself must be a provable no-op.
No recall regression: overall section recall@12 stays ≥ 0.74 on eval/bundle-corpus with the stage on.
The stage's own metric moves: for MMR, a diversity measure (distinct-page count or intra-bundle redundancy in the top-12) improves; for metadata-boost, verified/authoritative sections rank above equally-scored stale ones on a constructed test set; for knee, thin queries return fewer sections without losing gold ones.
Latency stays negligible: p95 bundle-assembly delta under ~20 ms at k=300 candidates.

A stage that fails gate 2 or 3 is rejected and recorded in the dead-levers list — the 0.74→0.68 enrichment regression is the standing proof that plausible changes hurt.

Verify before designing on these

Vectara's specifics (50 chainable stages, four reranker types) are competitive framing, not requirements — the target is three well-measured stages, not chain length.
"Sibling-page crowding suppresses coverage" is a real observed failure mode here, but its magnitude is unmeasured — the MMR diversity metric in the acceptance gate is what turns it from anecdote into a number.

Non-goals

No LLM rerank stages, ever — measured dead (shuffled-input recall collapse), independently corroborated. The SPI is deterministic by contract.
No user-defined boost expression language until a real tenant asks — fixed signal set only.
No re-attempt of the page-level KG rerank (measured zero lift; stays dormant at boost=0).

Investigation starting points

[MEASURE] MMR lambda sweep; and check whether sections_per_page de-dup in the assembler already provides implicit partial MMR — measure the marginal effect, not the textbook effect.
[MEASURE] Metadata boosts at fusion (adjusting RRF inputs) vs post-fusion (adjusting fused scores) — post-fusion is more debuggable; let the harness arbitrate if both are plausible.
[AGENT] Knee detection input: RRF scores are rank-derived and quantized — likely need the true dense cosines the section candidates already carry (coverage-signal work); design the stage interface so stages can request raw scores.
[MEASURE] Latency at k=300 against the 480 RPS envelope (the O(k²) worst case is the thing to bound).