Roadmap: Multimodal Ingestion — Diagrams, Images, Scanned Documents

Status (2026-10-13): Not started; the gap statement below still holds. Text extraction is the only ingestion path: Tika-based document upload (POST /api/ingest, createPages-gated, derived pages) and the connector framework (RoadmapConnectorFramework) both produce derived pages from extracted text. There is no vision captioning, no OCR (no Tesseract wiring anywhere in the repository), no caption chunks and no diagram eval set in eval/bundle-corpus. Attachments are still stored and served but never indexed. The Tika version in the root pom is now 3.3.2. Operator guide for the existing path: docs/admin/DerivedPagesAndIngest.md.

Gap statement: visual content is invisible to a text-only retrieval stack

Wikantik's ingestion is text/markdown-centric: Tika extracts text from office formats, and images/diagrams pass through as opaque attachments that no index ever sees. The sharpest formulation from the research: lexical and dense text retrieval simply cannot retrieve diagrams and figures — the content class where architecture decisions, network topologies, and process flows actually live. For a knowledge base aimed at engineering teams, that is a systematic blind spot, and it compounds once connectors (RoadmapConnectorFramework) start pulling in slide decks and scanned PDFs.

Market grounding

Amazon Bedrock Knowledge Bases: Smart Parsing selects a per-document parsing strategy covering PDFs, PPTX, DOCX, documents with embedded visuals, audio, video, and scanned documents; ingests multimodal content via automatic extraction and captioning; supports multimodal retrieval — image extraction, image-as-query, combined text+image queries via multimodal embeddings (Bedrock KB docs, product page).
Retrieval-side rationale: lexical search performs well for code and prose but cannot retrieve visual content, so competitive retrieval systems need vision-capable ingestion or retrieval (LightOn, retrieval in the age of agents).

Current Wikantik state

wikantik-ingest: Tika 3.3 (tika-core + tika-parsers-standard-package) + flexmark-html2md; deliberately isolated so the heavy PDFBox/POI dependencies stay out of wikantik-main. OCR is not wired (Tesseract not configured — zero OCR references in the codebase today).
Attachments: stored and served via AttachmentManager/AttachmentProvider, never indexed.
Local inference: Ollama already in the (offline) pipeline for extraction and embeddings — the natural host for a vision-captioning model; the extraction queue and breaker patterns are reusable.
Chunker: ContentChunker emits heading-anchored chunks with heading_path — the anchor a caption chunk would attach to.

Proposed direction (phased, cheapest useful step first)

Phase 1 — vision captioning at ingest: run page images and embedded document figures through a local vision model (Ollama); store the caption as an indexed chunk anchored to the image's position (heading_path of the surrounding section) and as alt-text on the rendered page (an accessibility win in the same stroke). Retrieval stays text-only; the content becomes text.
Phase 2 — OCR for scanned PDFs: wire Tesseract into the Tika pipeline behind a per-document heuristic (no text layer → OCR); scanned-document ingestion is table stakes for legal/ops corpora.
Phase 3 (later) — multimodal retrieval: multimodal embeddings and image-as-query. Only justified if Phases 1–2 show demand; it forks the dense-index schema and should not be attempted before the eval harness can measure it.

Phase 1 scope and acceptance criteria

First session scope: extend eval/bundle-corpus with 10–15 queries whose gold answer is a diagram/figure, then a batch captioner (extractor-CLI pattern) over the attachments of one test page set. No save-pipeline hook yet.

Phase 1 is done when:

The diagram-query eval set exists with gold sections agreed before any captioning code runs (the eval comes first — without it Phase 1 cannot be honestly accepted or rejected).
Captions are indexed as chunks anchored to the correct heading_path, and rendered pages show them as alt-text.
Diagram-set recall@12 ≥ 0.5 (baseline is ~zero — visual content is currently unretrievable), and overall corpus recall@12 shows no regression below the 0.74 baseline (caption chunks must not crowd out text sections).
The captioner is idempotent: re-running over already-captioned attachments produces no duplicate chunks.

Verify before designing on these

Bedrock's multimodal breadth (audio, video, image-as-query) is their feature list, not this roadmap's requirement — Phases 1–2 deliberately stop at captions and OCR.
"A vision model produces captions good enough to win retrieval" is the load-bearing assumption — it is resolved by the Phase-1 measurement, not by model marketing.

Non-goals

No multimodal embeddings or image-as-query until Phases 1–2 are measured and demand is shown (Phase 3 forks the dense-index schema — do not start it casually).
No audio or video ingestion.
No cloud vision APIs — captioning stays on the local Ollama host (cost governance + self-host posture).

Investigation starting points

[MEASURE] Vision model selection on the existing GPU budget: caption quality vs latency for diagram-heavy content — does a gemma-class vision model produce captions with enough discriminating vocabulary to win retrieval (the contextual-embedding lesson: generic text hurts, specific text wins)? The diagram eval set decides.
[MEASURE] Caption chunk placement: own pseudo-section vs merged into the surrounding section's chunk — run both against the diagram eval set.
[AGENT] Ingest-time vs backfill: batch captioner over existing attachments vs save-pipeline hook for new ones; both likely needed — sequence them.
[AGENT] Tika+Tesseract packaging for Phase 2: image size, native deps in the Docker image, per-page OCR cost; keep it in wikantik-ingest to preserve the dependency isolation.