---
title: Quality Evaluation
description: Measure retrieval quality against an SIE endpoint with NDCG@10, using the shipped retrieval-ablation benchmark as the template.
canonical_url: https://superlinked.com/docs/evals/quality
last_updated: 2026-08-25
---

Quality evaluation answers one question: given your queries, does the model surface the right documents? You need a corpus, queries, and relevance judgments. Rank metrics such as NDCG@10 do the rest. The SIE repository ships a complete example of exactly this workflow.

## The Shipped Benchmark

Source: [examples/retrieval-ablation/benchmark_ablation.py](https://github.com/superlinked/sie/blob/main/examples/retrieval-ablation/benchmark_ablation.py)

[examples/retrieval-ablation](https://github.com/superlinked/sie/tree/main/examples/retrieval-ablation) benchmarks page-level document search on six bank 10-K filings from SEC EDGAR. The script does four things:

1. Encodes 2,942 pages with dense and multi-vector models through `sie.encode`.
2. Encodes 1,854 real queries the same way.
3. Runs six ablation conditions: BM25, dense vector search, RRF fusion, cross-encoder reranking, multi-vector reranking, and multi-vector direct search.
4. Scores every ranked list with NDCG@10, MRR@10, and Recall@10 against 8,766 relevance judgments.

Eight models across the sweep (a dense encoder, three cross-encoder rerankers, and five multi-vector models, with `BAAI/bge-m3` pulling double duty as encoder and multi-vector model), one SIE endpoint, no model serving to manage. The [docs write-up](/docs/examples/benchmark/) covers the methodology and results in full.

## Run It Against Your Endpoint

Create a `.env` in the example directory first. The example also uses Turbopuffer for its BM25 and vector index:

```
# Use https for any remote endpoint; reserve http:// for local,
# unauthenticated development, since SIE_API_KEY travels as a bearer token.
SIE_BASE_URL=https://your-sie-endpoint
# Optional: only needed for managed/auth-enabled SIE clusters.
SIE_API_KEY=
TURBOPUFFER_API_KEY=tpuf_...
```

Then:

```bash
uv sync

# Validate config (no GPU needed)
uv run python benchmark_ablation.py --dry-run

# Full run: every condition and model sweep, all 1,854 queries
uv run python benchmark_ablation.py --gpu l4-spot

# Skip the baselines (BM25, dense, RRF); still sweeps every reranker
# and multi-vector model in conditions 4-6
uv run python benchmark_ablation.py --gpu l4-spot --skip-conditions 1,2,3
```

Encoding and search results cache to `cache/ablation/`, so re-runs skip completed steps. Cross-encoder reranking checkpoints every 100 queries.

## What It Found

Source: [examples/retrieval-ablation/RESULTS.md](https://github.com/superlinked/sie/blob/main/examples/retrieval-ablation/RESULTS.md)

All figures below come from the example's [RESULTS.md](https://github.com/superlinked/sie/blob/main/examples/retrieval-ablation/RESULTS.md):

| Strategy | NDCG@10 | Recall@10 |
| --- | --- | --- |
| Dual multi-vector pool, then `mxbai-rerank-large-v2` | **0.621** | **0.665** |
| Cross-encoder rerank over a hybrid BM25+vector pool | 0.600 | 0.640 |
| `bge-m3` multi-vector direct | 0.435 | 0.482 |
| `bge-m3` dense vector | 0.396 | 0.438 |
| BM25 | 0.185 | 0.239 |

Reranking dominated. RRF fusion actually scored below plain vector search on this dataset (0.358 vs 0.396): BM25 diluted a strong vector signal. That is the point of running evals on your own data; the result was not obvious in advance.

## Running Your Own MTEB-Style Eval

Any MTEB or BEIR retrieval task decomposes into the same three parts: a corpus, queries, and qrels. The loop is short:

1. Encode the corpus with `client.encode` (batch the calls).
2. Encode queries with `is_query=True`.
3. Rank by similarity, and optionally rerank the top candidates with `client.score`.
4. Compute NDCG@10 against the qrels. The shipped benchmark implements `ndcg_at_k`, `mrr_at_k`, and `recall_at_k` in plain Python; nothing heavier is required.

[Custom Evals](/docs/evals/custom/) walks through this pattern with runnable code.

## What's Next

- [Custom Evals](/docs/evals/custom/) - the worked pattern on your own labeled data
- [Performance Evaluation](/docs/evals/performance/) - latency and throughput measurement
- [Reranking models](/docs/score/models/) - choosing a cross-encoder to test