Skip to content
Most AI tasks no longer need a frontier model Read launch post

Quality Evaluation

Quality evaluation answers one question: given your queries, does the model surface the right documents? You need a corpus, queries, and relevance judgments. Rank metrics such as NDCG@10 do the rest. The SIE repository ships a complete example of exactly this workflow.

examples/retrieval-ablation benchmarks page-level document search on six bank 10-K filings from SEC EDGAR. The script does four things:

  1. Encodes 2,942 pages with dense and multi-vector models through sie.encode.
  2. Encodes 1,854 real queries the same way.
  3. Runs six ablation conditions: BM25, dense vector search, RRF fusion, cross-encoder reranking, multi-vector reranking, and multi-vector direct search.
  4. Scores every ranked list with NDCG@10, MRR@10, and Recall@10 against 8,766 relevance judgments.

Eight models across the sweep (a dense encoder, three cross-encoder rerankers, and five multi-vector models, with BAAI/bge-m3 pulling double duty as encoder and multi-vector model), one SIE endpoint, no model serving to manage. The docs write-up covers the methodology and results in full.

Create a .env in the example directory first. The example also uses Turbopuffer for its BM25 and vector index:

# Use https for any remote endpoint; reserve http:// for local,
# unauthenticated development, since SIE_API_KEY travels as a bearer token.
SIE_BASE_URL=https://your-sie-endpoint
# Optional: only needed for managed/auth-enabled SIE clusters.
SIE_API_KEY=
TURBOPUFFER_API_KEY=tpuf_...

Then:

uv sync
# Validate config (no GPU needed)
uv run python benchmark_ablation.py --dry-run
# Full run: every condition and model sweep, all 1,854 queries
uv run python benchmark_ablation.py --gpu l4-spot
# Skip the baselines (BM25, dense, RRF); still sweeps every reranker
# and multi-vector model in conditions 4-6
uv run python benchmark_ablation.py --gpu l4-spot --skip-conditions 1,2,3

Encoding and search results cache to cache/ablation/, so re-runs skip completed steps. Cross-encoder reranking checkpoints every 100 queries.

All figures below come from the example’s RESULTS.md:

StrategyNDCG@10Recall@10
Dual multi-vector pool, then mxbai-rerank-large-v20.6210.665
Cross-encoder rerank over a hybrid BM25+vector pool0.6000.640
bge-m3 multi-vector direct0.4350.482
bge-m3 dense vector0.3960.438
BM250.1850.239

Reranking dominated. RRF fusion actually scored below plain vector search on this dataset (0.358 vs 0.396): BM25 diluted a strong vector signal. That is the point of running evals on your own data; the result was not obvious in advance.

Any MTEB or BEIR retrieval task decomposes into the same three parts: a corpus, queries, and qrels. The loop is short:

  1. Encode the corpus with client.encode (batch the calls).
  2. Encode queries with is_query=True.
  3. Rank by similarity, and optionally rerank the top candidates with client.score.
  4. Compute NDCG@10 against the qrels. The shipped benchmark implements ndcg_at_k, mrr_at_k, and recall_at_k in plain Python; nothing heavier is required. The published Cloud quality numbers follow trec_eval conventions with graded relevance; Measurement Methodology states the scoring rule.

Custom Evals walks through this pattern with runnable code.

Talk to us

Tell us about your use case and we'll get back to you shortly.