Wikantik's retrieval-quality harness runs a curated query set against the live search stack, on demand and every night, so a retrieval change can be judged with numbers instead of a vibe check. The scheduled runner, the database-backed query-set store (retrieval_query_sets / retrieval_queries / retrieval_runs), the Prometheus gauges and the admin dashboard are Phase 5 of AgentGradeContentDesign, and all of them have shipped.
Before merging a retrieval-touching change (chunker, embeddings, fusion weights, query parsing), when investigating an anecdotal quality regression, or when you want to read the nightly quality gauges.
DefaultRetrievalQualityRunner (wikantik-main, package com.wikantik.knowledge.eval) is the harness. For one (query set, mode) pair it:
retrieval_queries (tables retrieval_query_sets, retrieval_queries, retrieval_runs come from migration V016);canonical_ids and scores them against the query's expected_ids with binary relevance — nDCG@5, nDCG@10, Recall@20 and MRR;retrieval_runs row, and updates the wikantik_retrieval_* Prometheus gauges.The seeded set is core-agent-queries (V017): 16 hand-curated questions, one per agent-cookbook runbook plus a cross-cluster query. Queries with empty expected_ids are skipped; a query whose retriever throws is skipped and marks the run degraded.
Modes (com.wikantik.api.eval.RetrievalMode):
| Mode | What it runs |
|---|---|
bm25 | Lucene BM25 only |
hybrid | BM25 + dense, RRF-fused — the production default |
hybrid_graph, hybrid_graph_weighted | Retired. The Knowledge Graph rerank was deleted in 2026-07; the names survive only so historical retrieval_runs.mode rows still parse, and a fresh run degrades to hybrid |
/admin/retrieval-quality): one row per (query set, mode) with the latest value and a sparkline per metric.POST /admin/retrieval-quality/run with {"query_set_id":"core-agent-queries","mode":"hybrid"} (admin session required). The call is synchronous and returns ndcg_at_5, ndcg_at_10, recall_at_20, mrr, queries_evaluated, queries_skipped and degraded.GET /admin/retrieval-quality?query_set_id=core-agent-queries&mode=hybrid&limit=30 returns recent runs. An absolute drop of more than 5 points in nDCG@5 against the previous run of the same set and mode is the design's alert threshold; the same figure is the wikantik_retrieval_ndcg_at_5{set,mode} gauge.degraded and queries_skipped — before trusting the number.bin/search-eval, which scores eval/retrieval-queries.csv through GET /api/search (see WikantikSearchRefinement).The same runner fires nightly at wikantik.retrieval.cron.hour_utc (default 03:00 UTC) for every non-retired mode when wikantik.retrieval.cron.enabled is true (the default); each failed (set, mode) pair is logged at WARN and counted in wikantik_retrieval_run_failed_total. Operator detail lives in docs/admin/RetrievalQuality.md.
Two sibling harnesses exist. The context bundle has a section-recall corpus (eval/bundle-corpus/, bin/eval/run-baseline.py, floors in thresholds.properties) that a scheduler can run when wikantik.bundle.eval.interval.hours is above 0 (default 0 = disabled). The offline embedding-model and fusion experiments are described on RetrievalExperimentHarness.
degraded: true run as clean. The aggregate is computed over the queries that did not throw, so it can look better than reality.expected_ids. They are canonical_ids; deleting or replacing a page makes queries_skipped climb and produces phantom regressions. Refresh the retrieval_queries rows.(set, mode) over time.bm25 results to judge a hybrid change. The modes are independent measurements.bm25 row. If the embedding service is down the dense side falls back to BM25 and the row quietly measures BM25 — see HandlingEmbeddingServiceOutages.Cluster Hub: Agent Cookbook (agent-cookbook)