Roadmap Target Architecture: Incremental Evolution to RAG-as-a-Service

This page is the architectural spine for the RagServiceRoadmapHub capability briefs. It defines an end state that is reachable incrementally from the current architecture — every step ships value on its own, no big-bang rewrite — and shows which seam each roadmap capability plugs into. Read WikantikArchitecture first for the current state.

Design stance: evolve the modular monolith, do not replatform

Wikantik is a modular Java monolith — one WAR, a WikiEngine orchestrator, manager interfaces with default implementations, provider-pattern storage, a WikiEvent bus, PostgreSQL + pgvector as the single datastore, and in-process Lucene indexes. The roadmap does not require microservices, a message broker, a workflow engine, or a second database. Every capability lands as a new module, a new SPI seam in an existing pipeline, or new tables behind an existing chokepoint. The single deliberate exception is the fleet control plane (tenancy), which is outside the monolith by design because the monolith itself is the unit it manages.

Two standing pressures shape everything below: wikantik-main is already a god module (the critique page's first-listed debt), and the architecture is governed by freeze/ratchet gates (DecompositionArchTest, the PMD complexity gate). The rule that reconciles growth with that debt:

New capabilities land in new modules or existing satellite modules — never as new packages in wikantik-main. Contracts go in wikantik-api; wikantik-main may gain only thin wiring at its established seams, and growth of its existing packages (e.g. new stages beside the bundle pipeline) is tolerated where the code they extend already lives there.

Current-state anchors the evolution builds on

These existing mechanisms are load-bearing for the roadmap — extend them, do not duplicate them:

Provider pattern (PageProvider, AttachmentProvider, SearchProvider) — the extension idiom the connector SPI copies.
Derived pages (com.wikantik.derived.*, ADR-0004: machine-owned regenerable bodies, reflow preserving curation) — the content model every connector writes into.
Async-by-design indexing (save → chunk → embed → dense-index decoupled from the write path, breaker → BM25 fallback) — the template for every new background pipeline (connector sync, ACL sync, vision captioning).
Event-incremental projection (OntologyEventListener re-projects on save/rename, nightly scheduler reconciles) — the listener+reconciler pattern connector and ACL sync reuse.
Post-startup wiring seams (KnowledgeSubsystemFactory, WikiEngine.patchContextRetrievalService, SearchWiringHelper) — where new retrieval stages attach without touching getManager (DecompositionArchTest R-2 keeps the allow-list closed).
Single enforcement chokepoint (checkPermission as the audited call, view-ACL-gated candidate retrieval, the ontology's materialization-time public/restricted split) — authorization pushdown extends this; it must never grow a parallel path.

Target module map

Naming trap (verified against the repo): the wikantik-knowledge Maven module contains only the knowledge MCP server (com.wikantik.knowledge.mcp, com.wikantik.knowledge.retrieval). The bundle pipeline's com.wikantik.knowledge.bundle and com.wikantik.knowledge.chunking packages live physically inside the wikantik-main module. Do not scaffold retrieval-pipeline code in the wikantik-knowledge module because the package name suggests it.

New modules (all depending on wikantik-api, none on wikantik-main internals):

wikantik-connectors — the SourceConnector SPI, sync scheduler, cursor/tombstone state. Individual connectors as sub-modules or plugin JARs so the WAR does not bloat with every source's SDK (RoadmapConnectorFramework).
wikantik-authz-sync — external principal mapping and ACL-sidecar resolution (RoadmapExternalAclInheritance). Kept separate from wikantik-connectors because it is the natural open-core seam: the module boundary is the license boundary, enforceable by ArchUnit and buildable as a separate artifact (RoadmapPackagingAndPricing).
wikantik-sdk-python / wikantik-sdk-ts — out-of-reactor client repos generated against the frozen API contracts (RoadmapClientSdkDeveloperExperience).
Fleet control plane — a separate small repo/service (inventory, provisioning, rolling upgrades, backup scheduling), managing tenant-in-a-box compose stacks over SSH/API. Explicitly not a WAR module (RoadmapMultiTenancyPosture).

Existing modules that grow, bounded: wikantik-ingest gains OCR and image-caption extraction (keeping the heavy-deps isolation rationale it exists for); wikantik-rest gains admin resources for connector/eval/audit surfaces; wikantik-observability gains OTel span export; wikantik-main hosts the retrieval pipeline seams below in its existing com.wikantik.knowledge.bundle package (see the naming trap above). The already-proposed wikantik-renderer extraction stays on the debt list — the roadmap neither needs it nor excuses deferring it forever.

Retrieval pipeline seams: QueryPlanner in front, RerankChain behind

The bundle pipeline (package com.wikantik.knowledge.bundle, physically inside the wikantik-main module — see the naming trap above) gains two SPI seams, contracts in wikantik-api:

query → [QueryPlanner] → candidate sources (BM25 + dense) → RRF fusion → [RerankChain] → bundle + coverage

QueryPlanner (default: identity/pass-through). The coverage-conditional sub-query decomposer (RoadmapAgenticRetrieval) is its first non-trivial implementation. Fail-closed: any planner failure degrades to pass-through, mirroring the Ollama→BM25 breaker.
RerankChain — an ordered, config-driven list of deterministic RerankStages (MMR, metadata boost, knee cutoff — RoadmapComposableReranking), each with per-stage cutoff/limit.
Telemetry is part of the SPI contract: every stage self-reports duration, in/out counts, and score distribution into the request's OTel span tree — stage-wise evaluation (RoadmapRetrievalEvaluationObservability) falls out of the interface instead of being bolted on.
Both seams attach at the existing post-startup wiring (patchContextRetrievalService / BundleServiceWiring pattern), keeping them off the getManager allow-list. Contracts (QueryPlanner, RerankStage) go in wikantik-api; default implementations live beside the pipeline in the existing com.wikantik.knowledge.bundle package — growing an existing package, consistent with the design-stance rule.

Every stage lands flag-gated and is accepted only on eval/bundle-corpus movement — the pipeline SPI makes experiments cheap; the harness decides what survives.

Authorization pushdown: extend the one chokepoint

External ACL inheritance does not add an authorization layer; it widens the existing one downward:

New tables (numbered migrations): source_principals (external identity → wiki principal, keyed by what SCIM/SSO already provision) and per-derived-page ACL sidecars written at sync time.
A single RetrievalAuthorizationPredicate component renders the caller's effective permission set into index-native filters: a SQL predicate joined into pgvector queries, a Lucene filter clause for BM25/HNSW. Pre-filtering in the index, never post-filtering in application code — the in-database pattern the market research recommends and Wikantik's PostgreSQL-first stack is uniquely placed to do.
The predicate is consumed inside the existing candidate-retrieval path, so /api/bundle, MCP tools, and search all inherit it — no per-surface reimplementation, no drift between surfaces.
Fail-closed semantics: an unresolvable external principal or stale sidecar denies, and permission-sync lag is surfaced by the same drift machinery that tracks citation staleness.

Data layer: PostgreSQL-first, no second datastore

All new state is tables in the one database, behind numbered idempotent migrations: connector sync state (cursors, content hashes, tombstones), principal mappings and ACL sidecars, eval-run results and retrieval-audit events, caption chunks (which are just chunks — the vision-captioning output re-enters the existing chunk→embed→index pipeline untouched). This keeps backup (3-2-1 NAS pull), migrations, and the silo-per-tenant story single-artifact. The only new persistent artifact class is per-connector credentials, which need encryption-at-rest — a deliberate design decision in the connector brief, not an afterthought (note: the existing CryptoUtil is one-way password hashing only; reversible credential encryption is greenfield).

Tenancy: the monolith is the tenant unit

The silo-per-tenant posture (RoadmapMultiTenancyPosture) is what makes this whole page incremental: no tenant_id columns, no row-level-security retrofit, no shared-store schema surgery anywhere. The application stays single-tenant; isolation is the deployment boundary (one compose stack per tenant); the fleet control plane and a shared Ollama inference pool live outside. If a future B2C-scale demand forces shared-store tenancy, that is a new roadmap, not a hidden assumption in this one.

API stability contract for SDKs

Publishing SDKs freezes surfaces. Before the first SDK release: bundle/briefing/changes response DTOs are declared stable in wikantik-api (com.wikantik.api.bundle already exists as the contract home), versioning policy stated (additive-only within a major), and an OpenAPI spec covering /api/bundle|briefing|changes is generated and CI-diffed so an accidental breaking change fails the build — the same ratchet philosophy as the complexity gate, applied to the wire.

Invariants that must not break

Any design that violates one of these is wrong for this codebase, whatever the market says:

ADR-0001: the platform assembles cited evidence; it never synthesizes answers — the QueryPlanner decomposes queries, it does not generate content.
Fail-closed retrieval: no LLM on the query path may make retrieval unavailable; every intelligent stage degrades to the deterministic pipeline.
Async indexing: the save path stays fast; all enrichment (embedding, captioning, sync, projection) is queued background work.
One authorization chokepoint: every retrieval surface flows through the same permission predicate; no surface-local ACL logic.
PostgreSQL-first: new state means new tables and migrations, not new infrastructure.
wikantik-main does not grow: contracts to wikantik-api, implementations to satellite modules or existing packages at established seams; no new packages in wikantik-main.
Measure-first: retrieval-affecting changes ship flag-gated and live or die by the eval harness — the rejected-levers list (LLM rerank, HyDE, doc2query, page-level KG rerank) stays rejected.
Apache 2.0 core stays complete: the open-core seam is a module boundary; the core must remain a fully functional product without the enterprise modules.

Capability-to-seam map

Capability brief	Architectural seam	New module(s)	Schema impact
RoadmapConnectorFramework	Provider-pattern SPI + derived pages + async queue	`wikantik-connectors`	sync state, credentials
RoadmapExternalAclInheritance	Authorization predicate pushdown at the existing chokepoint	`wikantik-authz-sync`	principal map, ACL sidecars
RoadmapMultiTenancyPosture	Deployment boundary; control plane external	fleet repo (non-WAR)	none
RoadmapMultimodalIngestion	`wikantik-ingest` extraction + existing chunk pipeline	— (grows `wikantik-ingest`)	caption chunks reuse chunk tables
RoadmapComposableReranking	`RerankChain` SPI post-fusion	— (in `wikantik-main`, existing `com.wikantik.knowledge.bundle` pkg)	none
RoadmapAgenticRetrieval	`QueryPlanner` SPI pre-retrieval, coverage-conditional	— (in `wikantik-main`, existing `com.wikantik.knowledge.bundle` pkg)	none
RoadmapRetrievalEvaluationObservability	Telemetry in the stage SPI contract + OTel export	— (grows `wikantik-observability`)	eval results, audit events
RoadmapCompliancePosture	Existing audit log widened to retrieval events	—	retrieval audit table
RoadmapPackagingAndPricing	Module boundary = license boundary	enterprise artifact split	none
RoadmapClientSdkDeveloperExperience	Frozen DTO contracts + spec-diff CI gate	SDK repos (out-of-reactor)	none