---
title: Custom Evals
description: "Evaluate models on your own labeled data: encode and rerank with the SDK, then score the rankings with NDCG and recall."
canonical_url: https://superlinked.com/docs/evals/custom
last_updated: 2026-08-25
---

A few hundred labeled queries from your own domain settle model choice faster than any leaderboard. The pattern below is the same one the shipped [retrieval-ablation benchmark](/docs/examples/benchmark/) runs at larger scale: encode, rank, optionally rerank, score.

## Structure the Labeled Set

Three pieces: a corpus, queries, and relevance judgments (qrels).

```python
corpus = [
    {"id": "doc1", "text": "Machine learning uses algorithms to learn from data."},
    {"id": "doc2", "text": "The weather forecast predicts rain tomorrow."},
]
queries = [
    {"id": "q1", "text": "What is machine learning?"},
]
qrels = {
    "q1": {"doc1": 2},  # higher grade = more relevant
}
```

Any graded scale works; the retrieval-ablation dataset uses 1 for relevant and 2 for highly relevant. Pull queries from real logs where you can, and have someone who knows the domain judge relevance. Unjudged documents count as not relevant.

## Encode and Rank

#### Python

```python
import numpy as np
from sie_sdk import SIEClient

client = SIEClient("http://your-sie-endpoint:8080")
MODEL = "BAAI/bge-m3"
CANDIDATES = 50  # rerank pool size; keep this larger than the final cutoff

doc_vecs = np.stack([r["dense"] for r in client.encode(MODEL, corpus)])
doc_vecs /= np.linalg.norm(doc_vecs, axis=1, keepdims=True)

rankings = {}
for query in queries:
    q = client.encode(MODEL, {"text": query["text"]}, is_query=True)
    q_vec = q["dense"] / np.linalg.norm(q["dense"])
    order = np.argsort(-(doc_vecs @ q_vec))
    rankings[query["id"]] = [corpus[i]["id"] for i in order[:CANDIDATES]]
```

#### TypeScript

```typescript
import { SIEClient } from "@superlinked/sie-sdk";

const corpus = [
  { id: "doc1", text: "Machine learning uses algorithms to learn from data." },
  { id: "doc2", text: "The weather forecast predicts rain tomorrow." },
];
const queries = [{ id: "q1", text: "What is machine learning?" }];

const client = new SIEClient("http://your-sie-endpoint:8080");
const MODEL = "BAAI/bge-m3";
const CANDIDATES = 50;  // rerank pool size; larger than the final cutoff

function normalize(v: Float32Array): Float32Array {
  let n = 0;
  for (const x of v) n += x * x;
  n = Math.sqrt(n);
  return v.map((x) => x / n);
}

function dot(a: Float32Array, b: Float32Array): number {
  let s = 0;
  for (let i = 0; i < a.length; i++) s += a[i] * b[i];
  return s;
}

const docResults = await client.encode(MODEL, corpus);
const docVecs = docResults.map((r) => normalize(r.dense!));

const rankings: Record<string, string[]> = {};
for (const query of queries) {
  const q = await client.encode(MODEL, { text: query.text }, { isQuery: true });
  const qVec = normalize(q.dense!);
  rankings[query.id] = docVecs
    .map((v, i) => ({ id: corpus[i].id, sim: dot(v, qVec) }))
    .sort((a, b) => b.sim - a.sim)
    .slice(0, CANDIDATES)
    .map((e) => e.id);
}
```

`encode` accepts a list and returns results in the same order; chunk large corpora into batches. Query-side encoding (`is_query=True` in Python, `isQuery: true` in TypeScript) matters for models with asymmetric encoding such as BGE and E5.

## Rerank the Top Candidates

Cross-encoder reranking won every condition in the shipped benchmark, and reranker size was the single biggest quality lever (+14.5% NDCG@10 for the large reranker over the base). Pool recall was the bottleneck: the same reranker scored 0.600 NDCG@10 on an 89-candidate pool and 0.621 on a 151-candidate pool (per its [RESULTS.md](https://github.com/superlinked/sie/blob/main/examples/retrieval-ablation/RESULTS.md)). So retrieve more candidates than you intend to keep, then let the reranker order them:

#### Python

```python
K = 10  # final cutoff for the metrics below

by_id = {c["id"]: c for c in corpus}
for query in queries:
    pool = [by_id[doc_id] for doc_id in rankings[query["id"]]]
    result = client.score(
        "mixedbread-ai/mxbai-rerank-large-v2",
        query={"text": query["text"]},
        items=pool,
    )
    rankings[query["id"]] = [s["item_id"] for s in result["scores"]][:K]
```

#### TypeScript

```typescript
const K = 10;  // final cutoff for the metrics below

const byId = new Map(corpus.map((c) => [c.id, c]));
for (const query of queries) {
  const pool = rankings[query.id].map((docId) => byId.get(docId)!);
  const result = await client.score(
    "mixedbread-ai/mxbai-rerank-large-v2",
    { text: query.text },
    pool,
  );
  rankings[query.id] = result.scores.map((s) => s.itemId).slice(0, K);
}
```

Scores come back sorted by relevance, most relevant first, with each item's id echoed.

## Score the Ranking

NDCG and recall are a few lines of plain Python. These match the metric functions the retrieval-ablation benchmark ships:

```python
import math

def ndcg_at_k(ranked_ids, rels, k=10):
    dcg = sum(rels.get(d, 0) / math.log2(i + 2) for i, d in enumerate(ranked_ids[:k]))
    ideal = sorted(rels.values(), reverse=True)[:k]
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0

def recall_at_k(ranked_ids, rels, k=10):
    relevant = {d for d, r in rels.items() if r > 0}
    return len(relevant & set(ranked_ids[:k])) / len(relevant) if relevant else 0.0

ndcg = sum(ndcg_at_k(rankings[qid], qrels[qid]) for qid in qrels) / len(qrels)
```

To compare models, swap `MODEL` and rerun. Keep the corpus, queries, and qrels frozen while you iterate; a moving eval set cannot tell you whether the model improved.

For a full multi-strategy comparison template (BM25, dense, hybrid fusion, multi-vector, cross-encoder rerank), start from [examples/retrieval-ablation](https://github.com/superlinked/sie/tree/main/examples/retrieval-ablation) and swap in your own dataset.

## What's Next

- [Quality Evaluation](/docs/evals/quality/) - the shipped benchmark and MTEB-style workflows
- [Performance Evaluation](/docs/evals/performance/) - latency and throughput measurement
- [SDK Reference](/docs/reference/sdk/) - full `encode` and `score` signatures