Roadmap: Multimodal Ingestion — Diagrams, Images, Scanned Documents

Gap statement: visual content is invisible to a text-only retrieval stack

Wikantik's ingestion is text/markdown-centric: Tika extracts text from office formats, and images/diagrams pass through as opaque attachments that no index ever sees. The sharpest formulation from the research: lexical and dense text retrieval simply cannot retrieve diagrams and figures — the content class where architecture decisions, network topologies, and process flows actually live. For a knowledge base aimed at engineering teams, that is a systematic blind spot, and it compounds once connectors (RoadmapConnectorFramework) start pulling in slide decks and scanned PDFs.

Market grounding

Amazon Bedrock Knowledge Bases: Smart Parsing selects a per-document parsing strategy covering PDFs, PPTX, DOCX, documents with embedded visuals, audio, video, and scanned documents; ingests multimodal content via automatic extraction and captioning; supports multimodal retrieval — image extraction, image-as-query, combined text+image queries via multimodal embeddings (Bedrock KB docs, product page).
Retrieval-side rationale: lexical search performs well for code and prose but cannot retrieve visual content, so competitive retrieval systems need vision-capable ingestion or retrieval (LightOn, retrieval in the age of agents).

Current Wikantik state

wikantik-ingest: Tika 3.3 (tika-core + tika-parsers-standard-package) + flexmark-html2md; deliberately isolated so the heavy PDFBox/POI dependencies stay out of wikantik-main. OCR is not wired (Tesseract not configured — zero OCR references in the codebase today).
Attachments: stored and served via AttachmentManager/AttachmentProvider, never indexed.
Local inference: Ollama already in the (offline) pipeline for extraction and embeddings — the natural host for a vision-captioning model; the extraction queue and breaker patterns are reusable.
Chunker: ContentChunker emits heading-anchored chunks with heading_path — the anchor a caption chunk would attach to.

Proposed direction (phased, cheapest useful step first)

Phase 1 — vision captioning at ingest: run page images and embedded document figures through a local vision model (Ollama); store the caption as an indexed chunk anchored to the image's position (heading_path of the surrounding section) and as alt-text on the rendered page (an accessibility win in the same stroke). Retrieval stays text-only; the content becomes text.
Phase 2 — OCR for scanned PDFs: wire Tesseract into the Tika pipeline behind a per-document heuristic (no text layer → OCR); scanned-document ingestion is table stakes for legal/ops corpora.
Phase 3 (later) — multimodal retrieval: multimodal embeddings and image-as-query. Only justified if Phases 1–2 show demand; it forks the dense-index schema and should not be attempted before the eval harness can measure it.

Phase 1 scope and acceptance criteria

First session scope: extend eval/bundle-corpus with 10–15 queries whose gold answer is a diagram/figure, then a batch captioner (extractor-CLI pattern) over the attachments of one test page set. No save-pipeline hook yet.

Phase 1 is done when:

The diagram-query eval set exists with gold sections agreed before any captioning code runs (the eval comes first — without it Phase 1 cannot be honestly accepted or rejected).
Captions are indexed as chunks anchored to the correct heading_path, and rendered pages show them as alt-text.
Diagram-set recall@12 ≥ 0.5 (baseline is ~zero — visual content is currently unretrievable), and overall corpus recall@12 shows no regression below the 0.74 baseline (caption chunks must not crowd out text sections).
The captioner is idempotent: re-running over already-captioned attachments produces no duplicate chunks.

Verify before designing on these

Bedrock's multimodal breadth (audio, video, image-as-query) is their feature list, not this roadmap's requirement — Phases 1–2 deliberately stop at captions and OCR.
"A vision model produces captions good enough to win retrieval" is the load-bearing assumption — it is resolved by the Phase-1 measurement, not by model marketing.

Non-goals

No multimodal embeddings or image-as-query until Phases 1–2 are measured and demand is shown (Phase 3 forks the dense-index schema — do not start it casually).
No audio or video ingestion.
No cloud vision APIs — captioning stays on the local Ollama host (cost governance + self-host posture).

Investigation starting points

[MEASURE] Vision model selection on the existing GPU budget: caption quality vs latency for diagram-heavy content — does a gemma-class vision model produce captions with enough discriminating vocabulary to win retrieval (the contextual-embedding lesson: generic text hurts, specific text wins)? The diagram eval set decides.
[MEASURE] Caption chunk placement: own pseudo-section vs merged into the surrounding section's chunk — run both against the diagram eval set.
[AGENT] Ingest-time vs backfill: batch captioner over existing attachments vs save-pipeline hook for new ones; both likely needed — sequence them.
[AGENT] Tika+Tesseract packaging for Phase 2: image size, native deps in the Docker image, per-page OCR cost; keep it in wikantik-ingest to preserve the dependency isolation.