OmniVault

What It Is

OmniVault is what you get if you take Paperless-NGX's premise — your documents, searchable and organized, on your own hardware — and refuse to accept the "documents" limitation. Photos, video, audio, code, archives, CAD files: everything gets full-text extraction where text exists, metadata extraction where it doesn't, immutable version history, and AI enrichment (descriptions, summaries, transcriptions, embeddings) layered on top. Search runs three ways in parallel: Elasticsearch for full text, PostgreSQL for structured and spatial queries, pgvector for semantic similarity.

The backend is a 65-module Gradle monolith in Java 21 that can decompose into satellite microservices, with a Python worker handling the ML-heavy jobs — OCR, Whisper transcription, speaker diarization, CLIP embeddings — over a Redis queue.

A Pipeline With No Pipeline

Uploading one file kicks off a lot of derived work: a HEIC photo needs format conversion, thumbnails, two kinds of metadata extraction, an AI description, an embedding, and a search index entry — and half of those depend on outputs of the others. The obvious design is a central pipeline definition that knows the whole graph. OmniVault doesn't have one. Instead there are three small interfaces — upload triggers, completion triggers, and post-processing triggers — and 38 Spring beans that each answer a single question: given what just happened, should my job run? Component scanning discovers them, a single dispatcher routes events through them, and the pipeline "definition" is whatever graph they collectively imply. At boot the dispatcher logs the discovered graph — the closest thing to a pipeline file the system has, and it's output rather than input.

Fan-In

The interesting hard part is jobs with multiple prerequisites. Search indexing needs the extracted text and the extracted metadata — for the same file version, since versions are immutable and a new upload can arrive mid-pipeline. A HEIC description needs the converted pixels and the exiftool GPS data. A fan-in tracker accumulates completions keyed by file version and trigger, and fires each trigger exactly once when its set is complete — with a five-minute timeout that fires with whatever showed up, so one failed upstream job degrades the result instead of wedging the pipeline.

Queue Families & Kill Switches

Every job type declares a queue family — in-process Java, Python worker, printer daemon, cloud sync — and the enum constructor requires it, so a new job type without a routing decision is a compile error rather than a runtime surprise. On top of that, triggers check runtime gates before firing: is AI enabled for this vault, is OCR enabled, is the Python worker reachable, is the PDF converter up. The graph you actually get is the declared graph intersected with the health of the system — which is exactly what the demo below lets you play with.

What Drawing It Found

Building the explorer turned into an audit. Reading every trigger to transcribe its gating conditions surfaced work the pipeline does that nothing consumes, and work it skips without saying so: entity extraction whose output is stored where no query can reach it, HEIC and SVG files deferred to a coordinator class that was deleted in a refactor — so they get no perceptual hash and no CLIP embedding — markdown that never reaches the search index at all, and indexing that runs exactly once, leaving transcripts and OCR text semantically searchable but lexically invisible.

Nodes carrying a warning marker in the demo below are annotated with what they're quietly failing to do. Each annotation is transcribed from the source, same as the edges.