PDF-to-Markdown is easy to demo and hard to trust. Here's how to test it.

This article walks through document-to-markdown, a workable example from the open-source SIE repo on GitHub: a PDF-to-Markdown pipeline with a deterministic evaluation harness attached. SIE is Superlinked’s self-hosted inference engine, one cluster that serves embedding, reranking, OCR, vision, entity-extraction, and generation models behind three primitives (extract, encode, score). The example is a complete project; four commands take you from fetching the PDFs to a graded eval. Built by Superlinked.

Every PDF conversion library has the same demo: feed in a clean page, get out clean Markdown, ship it. Then a real document arrives. A financial report with nested tables. A two-column investor deck. A government form where the labels and the fields are only related by position on the page. The conversion still runs, still produces Markdown, and somewhere in the middle a table has silently lost a column.

Silent is the problem. Nobody reviews conversion output line by line, so the corruption flows straight into your RAG index or your data pipeline, and you find out weeks later when an answer cites a number that doesn’t exist in the source.

The document-to-markdown example in the SIE repo treats conversion as something you verify, not something you trust. It converts four genuinely difficult public PDFs through SIE’s docling model and then runs 25 deterministic checks against the output: exact facts that must appear, section order where linear order applies, and how many Markdown tables each document must still contain.

The test corpus is the point

The four documents were picked to break naive converters in four different ways.

NVIDIA’s Q4 FY2025 CFO commentary is dense financial prose packed with figures where a single transposed number changes the meaning. The SiriusPoint investor presentation is a designed slide deck, which means layout-driven reading order that plain text extraction scrambles. The Docling research paper is academic two-column formatting with references and figures. And the FEMA form is the hardest category of all: form labels and values whose relationship is purely spatial.

Results from the run we committed, against SIE Cloud at api.superlinked.com: NVIDIA passed 6/6 checks in 9.2 seconds, SiriusPoint 6/6 in 12.2s, the Docling paper 6/6 in 6.7s, and the FEMA form 7/7 in 1.3s, giving 25 of 25. We ran it three times; all three passed and all three produced byte-identical Markdown, so this is reproducible rather than merely repeated. The committed run is the only one recorded by the current code, and it is also the fastest of the three; the other two are in the repository with their timings, 46.3 and 30.1 seconds against its 29.3. Read any of them as single measurements on shared Cloud hardware we do not control, not as a benchmark; a session’s first call also pays provisioning, about 15 seconds on the first document. uv run verify-run runs/cloud-20260916 rechecks the whole bundle offline, without a key.

And 25 of 25 does not mean the output is clean: not one of the fifteen tables in these four documents survives structurally. Docling expands a spanning header across every cell it covers, welds stacked tables into one, and space-joins cells that spanned several rows, so the Docling paper’s own benchmark table comes back with 4 16 and 177 s 167 s in single cells. The checks pass anyway because they test exact facts, section order and that tables are present, not that a table is the right shape.

What the pipeline looks like

The conversion itself is one call:

result = client.extract(
    "docling",
    Item(document=Path("data/pdfs/nvidia-q4-fy2025-cfo-commentary.pdf")),
)
markdown = result["data"]["markdown"]

That’s the whole integration surface. docling runs behind SIE’s extract endpoint, so the same client call works against a local container or the SIE Cloud API; you point it at whichever with environment variables.

The full workflow is four commands:

cd examples/document-to-markdown
uv sync
uv run fetch-documents
uv run convert-documents --run-id local
uv run eval-documents runs/local

Fetch pulls the four public PDFs, convert produces the Markdown, eval runs the 25 checks and reports pass/fail per document. Because the checks are deterministic (string presence, order assertions, table counts), a regression that moves a fact, an ordering or a table count shows up as a red check rather than as a vibe. A regression that only changes a table’s shape does not, which is why all 25 pass over the broken tables above.

Why deterministic checks beat eyeballing

The alternative to this harness is what most teams actually do: convert a few documents, scroll through the output, and declare it fine. That review catches formatting weirdness and misses the failures that matter. A transposed revenue figure still looks like a revenue figure. A section that moved still reads fluently. Your eyes verify plausibility; they don’t verify facts.

Deterministic checks invert the economics. Writing contains:$39,331, the assertion this repo makes about NVIDIA’s fourth-quarter revenue, takes thirty seconds and catches a transposed figure forever, on every future run, at zero marginal cost. The 25 checks in this repo took someone maybe an hour to write, and they re-run over four documents on every model upgrade, dependency bump, and infrastructure change from here on. Manual review guards exactly one conversion, once.

What they do not catch is worth being equally blunt about. A table that gained a duplicate column still looks like a table, and SiriusPoint’s did gain one: Q1'24 comes back twice, and partway down the two copies stop agreeing, one reading $91 where the other reads Underlying 5 $108. All six checks on that document passed. Eyes and assertions miss different things, and the shape of a table is in the gap.

There’s a second benefit that only shows up over time: the checks document what your pipeline actually depends on. Six months from now, when someone asks whether it’s safe to swap extraction models, the answer is a command, and the harness re-grades the converted output without redoing the conversion.

Honest about the edges

The README states its own boundary: the example measures whether the converted structure keeps the facts and the order an application needs, and claims no universal PDF benchmark score. Encrypted files, handwriting, and pages dominated by diagrams need separate tests and may need an OCR or vision model instead. That candor is useful. It tells you exactly when to stay on the docling path and when to reach for a different extract model, which in SIE is an identifier change rather than a new integration.

Steal the harness, not just the converter

Every fact these checks look for survived the conversion; the evaluation pattern is the transferable asset. Swap in your own PDFs, write ten checks for the facts your pipeline actually depends on (the revenue figure, the section ordering, the table that feeds your ETL), and you have a regression suite for document extraction. Run it whenever you upgrade a model or change infrastructure. Document conversion stops being a leap of faith and becomes a tested dependency, which is what it should have been all along.

Four difficult PDFs, 25 checks, 29.3 seconds of sequential SIE conversion time on the recorded run before file and eval overhead. Clone it and run the eval before you believe any converter’s demo, including this one.

Try it on GitHub: superlinked/sie/examples/document-to-markdown