The Pipeline That Assembles Itself

OmniVault is my self-hosted file manager — think Paperless-NGX, but refusing to accept that "documents" is a file type. Upload a HEIC photo to it and count what has to happen: the format gets converted to something browsers can render, a thumbnail gets generated from the converted pixels, metadata gets extracted twice (once through Tika, once through exiftool, because they're good at different things), a vision model writes a searchable description, the description gets embedded into pgvector, and the whole bundle lands in the search index. That's one file type. A PDF takes a different path through OCR and layout analysis; a video splits into frames and an audio track, and the audio track goes on to transcription and speaker diarization. Some of this work is independent, a lot of it depends on the outputs of other pieces, and about half of it is optional depending on what services are up and what the vault's owner has enabled.

The reflexive design for this is a pipeline definition: a YAML file, or an orchestrator class, or — in the fancier shops — a workflow engine, something that holds the whole graph in one place where you can point at it. I've written several of these over the years and I can report that the file where the whole graph lives is where systems like this go to die. Every new job type edits the same overgrown structure. Every conditional — only for images, only if OCR is on, only when the converter service is reachable — nests a little deeper into a config format that was never meant to express logic. Eventually someone adds a templating layer, and then you have two problems and a YAML preprocessor.

OmniVault has no pipeline definition. It has three small interfaces — an upload trigger, a completion trigger, and a post-processing trigger — and thirty-eight Spring beans that each implement exactly one of them. A bean answers a single question about a single job: should OCR run for this upload? Its entire worldview is one method that looks at the context and returns either job parameters or an empty list. The completion flavor adds one more method, declaring which upstream job types it wants to hear about:

public interface CompletionJobTrigger {
    Set<JobType> requiredJobTypes();
    List<? extends JobParams> evaluate(CompletionContext ctx);
}

That's the whole contract. There is no registry to update and no graph file to edit, because Spring's component scan is the registry: a dispatcher collects every trigger bean on the classpath, runs the upload triggers in order when a file lands, and indexes the completion triggers by the job types they subscribe to. The AI module contributes its ten triggers just by being on the classpath — compile the monolith without it and those edges simply don't exist. At startup the dispatcher logs the graph it discovered, which is the closest thing to a pipeline file the system has, and it's output rather than input. I find that reversal genuinely pleasing: the documentation can't drift from the implementation when it's generated by the implementation.

The part that earns its keep is fan-in. Search indexing needs the extracted text and the extracted metadata — both of them, and both for the same file version, because versions in OmniVault are immutable and a new version can arrive while the old one is still mid-pipeline. The HEIC description needs the converted pixels (the vision model can't read HEIC) and the exiftool metadata (so the description can mention where the photo was taken). A completion trigger declares multiple required job types, and a fan-in tracker accumulates completions keyed by file version and trigger class, firing exactly once when the set is complete. There's a five-minute timeout: if some prerequisite never completes — a job failed, a service was down — the trigger fires anyway with whatever arrived. One dead upstream job degrades the result instead of wedging the pipeline forever, which is the correct trade for a system whose operator is me, on a weekend, not watching dashboards.

The subtlety is that "multiple prerequisites" turns out to mean two different things. Indexing is a true AND: it genuinely wants both inputs, and waiting is the whole point. But the summarization trigger also declares five types — text extraction, transcription, OCR, Pandoc, and a second extraction engine — and it doesn't want all five. It wants whichever text producer actually ran, because no file is simultaneously a PDF, a video, and a Word document. Those are alternatives, not prerequisites. So a trigger declares which of the two it means, and the dispatcher routes accordingly: wait for the set, or evaluate each time one of them lands. Getting that distinction wrong is invisible in the code and expensive at runtime — an alternatives-trigger treated as a fan-in waits for a set that can never complete.

The other thing that keeps the graph honest at runtime is that triggers check the world before firing. Every AI trigger checks a per-vault kill switch, a per-feature configuration flag, and whether the AI service is actually configured; OCR checks its own vault setting; anything destined for the Python worker checks that the Redis transport is up; document conversion checks that Gotenberg is reachable. The graph you get in practice is the declared graph intersected with the current health of the system, which means an outage shrinks the pipeline instead of filling a dead-letter queue. And each job type declares which queue family executes it — in-process Java, Python worker, printer daemon, cloud sync — as a constructor argument on the enum, so a new job type without a routing decision isn't a runtime surprise, it's a compile error. The mapping is the contract between producer and consumers, and the compiler enforces it for free.

Drawing the graph is also how I found what it doesn't do. Entity recognition runs spaCy over every document that produces text, and writes the results to a rendition marked not-indexable that no query, no API, and no screen ever reads — the extraction is real and the output is unreachable. HEIC photos are skipped by the image-analysis and embedding triggers on the grounds that a coordinator will pick them up after conversion, and that coordinator does not exist; it was deleted in a refactor and the comment outlived it, so HEIC and SVG files quietly get no perceptual hash and no CLIP embedding. Markdown files are never full-text indexed at all, because skipping the extraction step for text that is already text leaves the indexing fan-in with no prerequisites to wait for, so it never fires. And indexing only happens once, which means the transcript of an audio file — produced minutes after the index was written — is semantically searchable and lexically invisible.

Every one of those is invisible in the code and obvious in the picture. That is not an argument against scattering the pipeline across thirty-eight small files; it's an argument that if you do, you owe yourself a way to see the whole thing at once. The boot-time log was supposed to be that, and it isn't, because a line reading TextSummarization <- [TEXT_EXTRACTION, AUDIO_TRANSCRIPTION, OCR, PANDOC_EXPORT, KREUZBERG] tells you what a trigger subscribes to and nothing about whether anyone reads what it produces.

Would I recommend this to a team of forty? Probably not without much better tooling on the graph — a pipeline you can't see is only charming when it's small enough to hold in your head, and mine demonstrably wasn't quite. But the pipeline definition I set out to avoid didn't actually disappear; it's just been scattered across thirty-eight files, each one too small and too boring to rot. I didn't eliminate the God object, I diced it finely enough that no single piece is worth fighting over — and then discovered that a God object at least has the decency to be wrong in one place.

If you'd rather see the graph than read about it, the interactive explorer is on the project page — pick a file type, watch its subgraph light up, flip the runtime gates to see how the pipeline degrades when services go down, and click through the nodes carrying a warning marker to read what each of them is quietly failing to do.