Roadmap: Retrieval Evaluation and Observability as a Product Surface

Status (2026-10-13): Phase 1 is built (v2.3.6, July 2026); the dashboard, per-stage attribution and OpenTelemetry phases are not. BundleEvalScheduler runs eval/bundle-corpus/queries.csv through the live bundle path every wikantik.bundle.eval.interval.hours (default 0 = disabled; a real run needs a live embedding index, and under Tomcat or a container wikantik.bundle.eval.corpus must be an absolute path), writes one bundle_eval_run row per run (migration V045: overall and per-category recall@12 plus a regression flag), and logs a BUNDLE-EVAL REGRESSION WARN when a recall falls below the floors in eval/bundle-corpus/thresholds.properties. Runbook: docs/agents/bundle-eval-runbook.md. A separate, older subsystem also exists: nightly retrieval_runs for BM25 and hybrid (nDCG, recall, MRR), Prometheus gauges and an admin dashboard at /admin/retrieval-quality with a Run-now button (wikantik.retrieval.cron.enabled / .hour_utc; see docs/admin/RetrievalQuality.md). Still open: a recall-trend dashboard over bundle_eval_run (read it with SQL today), per-stage recall attribution, OpenTelemetry (no OTel dependency in any pom), the published methodology page, and the [OWNER] query-log privacy decision.

Gap statement: strong internal plumbing, no customer-visible surface

Wikantik has arguably more honest evaluation machinery than the LLM-judge frameworks the market standardized on — a retrieval experiment harness, retrieval-quality CI, a per-bundle coverage signal computed from true dense cosines, and a real-traffic query log. But all of it is internal plumbing. Competitors ship dashboards, traces, and built-in metrics that buyers can see. The gap is productization, not science — which makes it one of the cheapest credibility wins on the roadmap.

Market grounding

Metric baseline: RAGAS established the reference-free metric set (faithfulness, answer relevancy, context precision, context recall); DeepEval adds 50+ metrics with native Pytest/CI integration and regression gating; TruLens pairs its RAG Triad with OpenTelemetry-based tracing (Atlan framework comparison, Maxim, top RAG evaluation tools).
The judge caveat, in Wikantik's favor: independent adversarial testing found none of the leading LLM-judge frameworks could distinguish factually wrong from factually correct retrieved contexts, and they all assume the index itself is trustworthy — leaving content quality unmeasured. Wikantik's citation-staleness and verification machinery addresses exactly that unmeasured layer; and with no answer synthesis (ADR-0001) there is nothing to score for faithfulness/hallucination — a positioning point to state explicitly.
Observability bar: Bedrock's AgentCore ships built-in retrieval traces, agentic traces, and per-knowledge-base metrics (Bedrock KB docs); production monitoring of retrieval (not just offline eval) is positioned as what most open-source eval tools lack; stage-wise per-component metrics are called mandatory — end-to-end scores alone cannot localize a regression.

Current Wikantik state

The retrieval experiment harness — implemented, not scheduled. Precision note for implementers: there is no production class named RetrievalExperimentHarness; the actual code is com.wikantik.search.embedding.experiment.ExperimentHarness (test scope) driven by bin/run-embedding-experiment.sh, with the design documented in the RetrievalExperimentHarness wiki page. eval/bundle-corpus/ holds the eval corpus and baseline notes (section recall@12 ~0.74); retrieval-quality CI came from the agent-grade content work.
Coverage block on every bundle (strong/partial/weak from true dense cosine) — shipped to prod.
list_retrieval_queries (admin MCP) — deduped real-traffic query log with max_avg_results filtering for under-served queries; briefing_log telemetry for the briefing surface.
wikantik-observability — health checks, Prometheus metrics, request correlation; dashboards live in the separate jakemon stack. OpenTelemetry is greenfield: no OTel dependency exists anywhere in the codebase today.

Proposed direction

Schedule the harness: nightly/weekly runs against the eval corpus with results persisted; alert on recall regression beyond a threshold. This is configuration plus a results table, not new code of substance.
Retrieval-quality dashboard (admin UI): recall trend from harness runs, under-served-query burn-down from the query log, coverage-confidence distribution over live traffic, index freshness (embed-queue depth, last re-index).
OpenTelemetry traces on the retrieval path: spans for candidate retrieval → fusion → rerank chain → bundle assembly, propagated through /api/bundle and both MCP servers — stage-wise timing and result counts per query, exportable to any OTel backend.
Publish the methodology: a public page documenting the metric definitions, the harness, and the measured-dead-levers list — honesty as marketing, and a genuine differentiator against LLM-judge theater.

Phase 1 scope and acceptance criteria

First session scope: scheduled harness runs + a persisted results table (numbered migration) + a regression alert. No dashboard, no OTel.

Phase 1 is done when:

A scheduled (nightly or weekly) run executes the harness against eval/bundle-corpus and persists per-run recall@12 (overall, and per retrieval stage where the harness supports it) with timestamp and build/config identity.
An induced regression — temporarily enabling a known-bad configuration in a test environment — fires the alert; a normal run does not.
A short runbook page documents how to read a result row and respond to an alert.
The results table migration re-applies as a no-op.

Dashboard (step 2) and OTel (step 3) are separate phases with their own gates; do not bundle them into Phase 1.

Verify before designing on these

"Production retrieval monitoring is what most open-source eval tools lack" comes from a vendor's product blog — true enough to motivate the work, not a competitive analysis to cite.
The RAGAS/DeepEval/TruLens metric taxonomies are accurate as of the research sweep, but Wikantik should adopt retrieval-stage metrics (recall@k, MRR, NDCG per stage), not the generation-side metrics (faithfulness, answer relevancy) that presuppose synthesis this platform refuses by design.

Non-goals

No faithfulness/hallucination scoring — structurally inapplicable under ADR-0001 (state this in the published methodology, don't apologize for it).
No LLM-as-judge metrics anywhere in the acceptance path (the measured judge weakness is the reason).
No public/customer-facing dashboard in Phase 1–2 — admin UI first; external exposure waits on the query-log privacy decision below.

Investigation starting points

[AGENT] Metric inventory: what does ExperimentHarness already compute vs what needs adding for per-stage attribution (lexical, dense, fused, post-rerank)? Stage-wise attribution is the point.
[AGENT] OTel wiring in a servlet/Tomcat stack alongside the existing request-correlation filter — reuse the correlation ID as trace ID?
[OWNER] Query-log privacy: retention, redaction, and per-tenant scoping — real queries can contain sensitive text, and this must be decided before a dashboard makes them visible (ties to RoadmapCompliancePosture).
[AGENT] Dashboard placement: extend the admin SPA (default — it is the product answer) vs Grafana panels in the jakemon stack for the single-prod deployment.
[OWNER] Per-tenant eval corpora vs a shared benchmark corpus per release — deferred until the tenancy posture is ratified.