Roadmap: External Source Connector Framework

Status (2026-10-13): The framework this brief proposed has been built, so the gap statement below describes the July 2026 starting point, not today. The wikantik-connectors module (v2.3.6, July 2026) implements the SourceConnector SPI (com.wikantik.api.connectors, in wikantik-api) and a SyncOrchestrator with hash-dedup, cursor resume and tombstones, and ships seven connector types: filesystem, web crawler, sitemap, RSS/Atom feed, Google Drive (OAuth2), GitHub and Confluence. Six are admin-creatable (ConnectorConfigCodec.UI_TYPES: webcrawler, sitemap, feed, gdrive, github, confluence); filesystem stays properties-only because it reads arbitrary server paths. Since v2.3.7, definitions live in the connector_configs table (V048) and are hot-applied without a restart; sync state is in connector_sync_state and connector_synced_item (V046), per-run history in connector_sync_run (V049), and secrets in the AES-GCM encrypted connector_credentials store (V047), bound to the origin they were stored for. Admin UI: /admin/connectors with a guided add-connector wizard; wikantik.connectors.enabled is the kill switch (default true). Connector-synced pages are derived pages (ADR-0004). Still open: S3, SharePoint and Slack connectors, webhook/push receivers (sync is scheduled pull with cursors), and ACL enforcement. Source ACL references are carried in connector_synced_item.acl_refs but nothing reads them at retrieval time (see RoadmapExternalAclInheritance). Operator guide: docs/admin/Connectors.md in the repository.

Gap statement: zero external connectors in a market defined by them

Wikantik ingests wiki-native pages, Tika-extracted document uploads (POST /api/ingest), and MCP writes. It has no connectors to external systems, no continuous synchronization, and no incremental change detection. Every surveyed source names connector breadth and production-grade sync the primary differentiator among enterprise RAG platforms. This is the widest gap in the portfolio and the one that most limits the "RAG-as-a-service" claim.

Market grounding: the connector benchmarks

Glean: 100+ enterprise connectors (Slack, Teams, Jira, GitHub, Confluence, Google Drive) across native/partner/custom tiers, with three freshness models per connector — full indexed crawl, query-time live retrieval, hybrid — using incremental crawls, webhooks, and push APIs (Glean connector docs).
Onyx (open-source, closest analog): 40+ connectors with continuous sync, plus a push API and file upload; document-level permission sync is gated to the paid Enterprise Edition (Onyx connector docs).
Amazon Bedrock Knowledge Bases: managed connectors for S3, SharePoint, Confluence, Google Drive, OneDrive, and a web crawler; event-driven ingestion (EventBridge/SQS) with metadata sidecars (Bedrock KB docs).
Unstructured: 71 source/destination connectors; argues connectors must support OAuth 2.0, API keys, service accounts, and token auth, and must preserve source metadata (authorship, update history, origin) through the pipeline (Unstructured on connectors).
Economics of incremental sync: for a 100k-document corpus at 2% daily churn, changed-only processing cuts monthly cost by more than 90% versus full reprocessing (same source). Incremental sync is not a nicety; it is the cost model.

Current Wikantik state: the substrate already exists

The derived-pages machinery is the right foundation — what is missing is the source side:

wikantik-ingest (com.wikantik.ingest.{SourceExtractor, TikaSourceExtractor, ExtractionResult}) — pure Tika 3.3 extraction, isolated from wikantik-main.
com.wikantik.derived.* — a page is derived iff frontmatter has derived_from; body is machine-owned and regenerable (ADR-0004); reflow re-extracts while preserving body-independent curation. Surfaces: POST /api/ingest, /admin/derived/{reflow,status}.
IngestDocumentsCli (wikantik-extract-cli) — batch folder-walking HTTP client, effectively a proto-connector.
The provider pattern (PageProvider, AttachmentProvider) is the established extension idiom to model a connector SPI on.
Design doc: docs/superpowers/specs/2026-06-14-derived-pages-design.md.

Proposed direction

Connector SPI + sync scheduler in a new wikantik-connectors module: SourceConnector interface (enumerate, fetch, cursor/checkpoint, delete-detection), per-connector sync state persisted in PostgreSQL, scheduled incremental runs modeled on OntologyRebuildScheduler.
Source-item → derived-page mapping: each external item becomes a derived page with derived_from pointing at a connector-scoped URI, source metadata mapped into frontmatter, and reflow semantics reused for re-sync.
v1 connector set (pick by user demand, not breadth): Confluence, Google Drive, GitHub (repos + issues), S3, web crawler, Slack.
Delete propagation from source to derived page (required for GDPR right-to-be-forgotten; see RoadmapCompliancePosture).
Webhook/push receivers as a later phase; scheduled pull with cursors first.

Phase 1 scope and acceptance criteria

First session scope: the SourceConnector SPI in wikantik-connectors + the sync-state migration + ONE read-only connector against a local fixture (filesystem tree or static HTTP fixture). No OAuth, no scheduler, no webhooks, no real external service.

Phase 1 is done when these integration tests are green (TDD: write them first):

A fixture corpus syncs into derived pages — derived_from carries the connector-scoped URI, source metadata lands in frontmatter, and the pages pass the frontmatter schema validator.
A sync killed mid-run and restarted resumes from its cursor without duplicating or re-processing already-synced items.
A tombstoned (deleted-at-source) item removes or archives its derived page on the next sync.
The sync-state migration re-applies as a no-op (project migration convention), and the item schema carries source ACL references (unenforced — the RoadmapExternalAclInheritance co-design constraint).

Out of Phase 1's DoD: scheduler wiring, credential encryption, any real-service connector. Those are Phase 2, gated on the Phase-1 SPI surviving review.

Verify before designing on these

The ">90% cost reduction from incremental sync" figure is a vendor illustration. The design premise (sync must be incremental) holds regardless, but do not size scheduler economics from that number — measure against a realistic corpus churn profile.
Connector counts (Glean 100+, Onyx 40+, Unstructured 71) are market framing for why this matters, not requirements. The v1 target is a working SPI plus a handful of well-tested connectors, not breadth.
Before porting ideas from Onyx's connector code, check the license boundary: read only the open-source core, never its ee/ (Enterprise Edition) directories.

Non-goals

No bidirectional sync (wiki → source writes).
No real-time webhook receivers in v1 — scheduled pull with cursors first.
No ACL enforcement in this brief — the schema carries ACL references; enforcement is RoadmapExternalAclInheritance.
No connector marketplace or third-party plugin distribution story.

Investigation starting points

[AGENT] Read Onyx's open-source connector interface as the reference design (core only, not ee/ — see above) — what does its checkpoint/state model look like, and what did they get wrong?
[AGENT] Schema design for sync state (cursors, content hashes, tombstones) — must be a numbered migration under bin/db/migrations/.
[OWNER] Identity and auth per connector: OAuth 2.0 app registrations vs service accounts, where refresh tokens live, and the encryption-at-rest approach (note: CryptoUtil is hash-only today; reversible credential encryption is greenfield, and the key-management choice is a security-posture/spend decision).
[AGENT] How does connector-sourced frontmatter interact with StructuralSpinePageFilter's save-time canonical_id enforcement and the frontmatter schema validator?
[AGENT] Rate-limit and backoff behavior per source API; resumability of a half-failed sync.
[AGENT] Co-design constraint: the item schema must carry source ACL references from day one, even if enforcement lands later — the ACL semantics are decided in RoadmapExternalAclInheritance (where the security-posture questions are [OWNER]).