---
title: Test your reranker on documents where wrong answers have consequences
description: "Walk through the SIE rerank example: a hash-verified reranker evaluation on verbatim passages from official documents."
canonical_url: https://superlinked.com/blog/rerank
last_updated: 2026-08-10
---

*This article walks through [rerank](https://github.com/superlinked/sie/tree/main/examples/rerank), a workable example from the open-source [SIE repo](https://github.com/superlinked/sie) on GitHub: a hash-verified reranker evaluation on verbatim passages from official documents. SIE is Superlinked's self-hosted inference engine, one cluster that serves embedding, reranking, OCR, vision, entity-extraction, and generation models behind three primitives (extract, encode, score). The example is a complete project; two commands run the whole evaluation locally or against a hosted cluster. Built by Superlinked.*

Most reranker benchmarks are built from synthetic queries over Wikipedia-adjacent text. Your production system ranks passages from contracts, regulations, filings, and case law. The gap between those two worlds is where retrieval quality quietly dies, and no leaderboard number will warn you.

The rerank example in the SIE repo closes that gap with a small, sharp evaluation: can Qwen3-Reranker-4B put the correct primary-source passage first when every candidate is a verbatim excerpt from a real authoritative document?

Four document types, chosen because they're the ones where a retrieval mistake actually costs something. An SEC filing. A CMS healthcare claim example. An NTSB accident report. A Supreme Court opinion. No paraphrases, no synthetic distractors; the candidates are exact text from the originals, so the reranker faces the same ambiguity a production system would.

## Verbatim or it doesn't count

The design choice that makes this example trustworthy is provenance. Every passage candidate in `data/cases.json` traces back to its source document through a URL, a locator, and a SHA-256 hash, with sources catalogued in `data/sources.json`. Before any ranking happens, the harness verifies the hashes. If a passage has drifted even one character from the published original, the run fails with a nonzero exit code.

This sounds pedantic until you've debugged an eval where someone "cleaned up" a test passage and accidentally made it easier. Hash-pinned inputs mean the benchmark you run today is byte-identical to the one that produced the committed results, and anyone reviewing your numbers can verify that claim independently.

## What a run looks like

The mechanics are deliberately plain. Evaluation prompts go to the reranker endpoint (Qwen/Qwen3-Reranker-4B, served by SIE locally or on a hosted cluster), each request and response is captured as an audit envelope recording the arguments and the provenance data, and results are validated against expected rankings.

The pass condition is strict: the expected passage must rank first. Not top-three, not "high enough". First. The run also fails if hashes diverge or candidates are incomplete, so a green run means the full chain held: right inputs, complete candidates, correct ranking.

```bash
uv sync
uv run python run.py
```

That runs against a local SIE server. Point `SIE_BASE_URL` and `SIE_API_KEY` at a remote cluster and the same script evaluates your hosted deployment instead. Either way, artifacts are checksummed into `verified-run/manifest.json`, with raw model responses preserved separately from processed results so you can audit what the model actually returned rather than what the harness made of it.

## What makes primary sources hard

Verbatim official text punishes rerankers in ways synthetic benchmarks don't. Regulatory and legal documents are internally repetitive by design: the same statute gets cited in four sections, the same dollar figure appears in the summary and the detail table, the same party names run through every paragraph. Surface-level similarity is everywhere, so a reranker that keys on vocabulary overlap sees a field of near-identical candidates.

The passage that actually answers the question is distinguished by something subtler: it's where the document commits, where the holding is stated rather than summarized, where the figure is established rather than referenced. Ranking that passage first requires the model to read for function, and function is exactly what paraphrased benchmark data trains away. Testing on the untouched text is the only way to know.

## Why the audit trail matters more than the score

A reranker eval that just prints a number answers one question. This one answers the follow-ups too. Which passage did the model prefer instead? What were the exact inputs? Is the committed result reproducible on my hardware? Because envelopes, hashes, and manifests are all part of the run output, the answer to "can I verify this" is always yes.

That property matters most in exactly the domains this example draws from. If you're building retrieval over regulatory or legal text, someone will eventually ask you to prove the system surfaces the controlling passage and not a plausible neighbor. "Our reranker scores well on BEIR" is not an answer to that question. "Here is a hash-verified run on verbatim excerpts from the source corpus, and here is the audit envelope for every request" is.

## Make it yours

The four bundled cases are a template. The real move is swapping in your own documents: pull the exact passages your system must distinguish, hash them, define the expected winner, and run. You'll know within minutes whether Qwen3 Reranker handles your domain's hard cases, and you'll have a regression harness that fails loudly the day a model upgrade changes the answer.

Small example, strict standards, everything verifiable. Clone it and point it at the passages you actually care about.

**Try it on GitHub:** [superlinked/sie/examples/rerank](https://github.com/superlinked/sie/tree/main/examples/rerank)