Your RAG pipeline can't read a wiring diagram. This one can.

This article walks through vision-doc-rag, a workable example from the open-source SIE repo on GitHub: a document RAG pipeline that retrieves and answers questions over page images instead of extracted text. SIE is Superlinked’s self-hosted inference engine, one cluster that serves embedding, reranking, OCR, vision, entity-extraction, and generation models behind three primitives (extract, encode, score). The example is a complete project with a bundled corpus, CLI, and web UI. Built by Superlinked.

Take a scanned NASA rocket nozzle drawing from the 1960s. No text layer. No OCR output worth keeping. Now ask a standard RAG pipeline a question about it.

You’ll get nothing, because standard RAG never saw the page. The PDF-to-text step threw the drawing away before retrieval even started. Every pipeline built on extract-then-embed has this hole: pinout diagrams, circuit schematics, engineering drawings, scanned forms. The information is right there on the page, and the pipeline is structurally blind to it.

The vision-doc-rag example in the SIE repo closes the hole by refusing to leave the image domain. Retrieve by image, answer by image. OCR appears nowhere in the scoring path.

Three models, no text extraction

The pipeline renders every PDF page to a PNG and works with pixels from then on.

ColQwen2.5 handles retrieval. It encodes each page image (and each query) as a set of 128-dimensional multivectors, and ranking uses late interaction: MaxSim scoring between query vectors and page vectors. This is the ColPali family of techniques, and the practical consequence is that a page ranks well because its diagrams and layout match the query, not because some OCR pass happened to produce the right words.

Qwen3.5-4B answers. It’s a vision LLM, so it receives the top-ranked page as an image and reads it directly: the table, the diagram, the handwritten annotation, whatever is there.

Qwen3-VL-Reranker-2B is the optional second stage, reordering candidates while staying in the visual modality. Even the reranking never degrades to text.

All three run as CUDA-backed endpoints on one SIE deployment, and the corpus queries return the kind of answers that make the approach click: ask about the default PostgreSQL port and get 5432 from a rendered docs page; ask about GPIO voltage on a Raspberry Pi and get 3.3V read off the pinout diagram; ask about the NASA nozzle and get an answer from a scanned drawing that contains no machine-readable text at all.

Multi-tenancy is a tag, not an afterthought

Every encoded page carries a client tag, and queries scope to a tenant. The demo ships with three: embedded-lab holds microcontroller datasheets, ops-eng holds PostgreSQL and Kubernetes documentation, and aerospace holds the NASA reports. An ops-eng query cannot surface an aerospace page. If you’re building document search for multiple customers, this is the isolation model you need, demonstrated in miniature.

Running it

You need Python 3.12 and a GPU-backed SIE deployment. The flow is five steps: spin up SIE with CUDA, fetch the PDFs, render pages to PNGs, encode them with ColQwen2.5, then query through either the CLI or the bundled web UI.

The corpus is deliberately small so ingestion finishes fast on local hardware while keeping the visually hard cases in. MaxSim runs in plain Python at this scale; the README points to multivector stores like LanceDB and Vespa when you grow past a few thousand pages, so the demo architecture has a stated path to production rather than a dead end.

One operational note the README is honest about: these models are several gigabytes each and load slowly. For managed deployments, keep minReplicas: 1 on the generation bundle so you’re not paying the cold-start tax on every question.

What late interaction buys you

The multivector detail is worth unpacking, because it’s why this retrieval works on pages a single embedding would blur. A conventional bi-encoder compresses an entire page into one vector; a dense schematic and its caption and its table all average into a single point, and the specifics that would match your query get smoothed away. ColQwen2.5 instead keeps a set of 128-dimensional vectors per page, roughly one per visual patch.

At query time, MaxSim lets each query token find its best-matching patch independently, then sums those maxima. A question about one pin on a 40-pin header can match the exact region of the pinout diagram where that pin is drawn, even though the rest of the page is about something else. That per-region matching is the property text chunking tries to approximate and never quite reaches, because chunk boundaries are guesses and patch boundaries come free with the image.

The argument underneath

There’s a quiet claim embedded in this example: OCR is a lossy compression step, and RAG pipelines have been paying for that loss without noticing. Text extraction works fine on clean digital PDFs and quietly destroys everything else. Vision-first retrieval doesn’t ask “what text is on this page”; it asks “does this page look like the answer”, which is a strictly more general question.

If your document corpus includes anything scanned, drawn, or diagram-heavy (and honestly, whose doesn’t), clone the example and feed it your worst pages. Ask it something a text pipeline gets wrong. The comparison takes an afternoon and settles the argument with your own documents.

Try it on GitHub: superlinked/sie/examples/vision-doc-rag