---
title: Measurement Methodology
description: "How the published Cloud benchmark numbers are measured: the quality reference and its 2% band, the fixed-rate performance method, how to repeat a published number with the public example, the spread that repeat runs showed, how to time a task page's per-request number on your own GPU, and which numbers on this site are measured internally."
canonical_url: https://superlinked.com/docs/evals/methodology
last_updated: 2026-11-09
---

The published Cloud benchmark numbers for the six models below are SIE Cloud numbers, measured through the public API from a separate client machine in the same cloud region as the API. This page states the method behind those numbers so you can repeat it: what a record holds, where the quality reference comes from, how load is applied, which setup mistakes produced wrong numbers in our own runs, and what three repeat runs of the public example looked like.

## What We Measure

Two things, kept apart.

Quality is how good the model's output is on a public benchmark, compared with the model's official number. The score is the benchmark's own metric against its ground truth; for NFCorpus and SciFact that is NDCG@10, for AskUbuntuDupQuestions map@1000.

Performance is latency and throughput at a fixed request rate against a warm model: what one request costs in time at a steady load. Not the peak capacity of the client or of the service.

The rest of the site's numbers were made another way. The per-request times on the task pages come from public studies that ran SIE on a GPU server, with the client inside the server's container sending one request at a time; [Check a Time on a Task Page](#check-a-time-on-a-task-page) repeats ten of them on your own GPU. Most accuracy figures on the [model pages](/models/) come from our internal measurement pipeline, which runs SIE against the named public datasets. You can repeat many of those too. [Measured Internally](#measured-internally) lists the numbers you cannot.

## Where the Published Numbers Come From

Each of those numbers comes from one record. There are twelve: six model and output pairs, each with one quality record and one performance record. All of them were measured on 2026-09-05 through the public API.

| Model | Output | Dataset (MTEB) | Performance load |
| --- | --- | --- | --- |
| `BAAI/bge-m3` | multivector | SciFact | fixed 1 rps |
| `BAAI/bge-m3` | sparse | NFCorpus | fixed 10 rps |
| `lightonai/GTE-ModernColBERT-v1` | multivector | SciFact | fixed 1 rps |
| `prithivida/Splade_PP_en_v2` | sparse | NFCorpus | fixed 10 rps |
| `Qwen/Qwen3-Reranker-0.6B` | scores (rerank) | AskUbuntuDupQuestions | fixed 0.25 rps |
| `Qwen/Qwen3-Reranker-4B` | scores (rerank) | AskUbuntuDupQuestions | fixed 0.25 rps |

A record carries what a repeat needs: the model name and its pinned revision; the dataset name, revision, subset, limit and a content digest; where the client ran, with its worker and process count; the request operation, items per request, input limit and output type; the load level with its start and end time; each metric's name, unit, aggregation and value; and SHA-256 hashes of the raw evidence and the run manifest.

A run becomes a website number after two checks: a validator marks it valid, then a person approves it. Each of the twelve numbers landed through its own reviewed change.

## Quality

### Where the Reference Comes From

The reference is chosen in a fixed order:

1. The official number, when it was reported for the same task and the same main metric.
2. Otherwise, the model's official implementation run by us on one GPU class in fp32: sentence-transformers for dense and sparse models, PyLate for ColBERT models, Transformers for the Qwen rerankers.

SIE's own output is never the reference, and a reference is never lowered to match SIE.

### Scoring

The website numbers use the MTEB retrieval tasks NFCorpus, SciFact and AskUbuntuDupQuestions, scored with the conventions of [mteb](https://github.com/embeddings-benchmark/mteb) 2.7.5: [pytrec_eval](https://github.com/cvangysel/pytrec_eval) ndcg, map, recall and precision at cuts 1, 3, 5, 10, 20, 100 and 1000 with graded relevance, plus mrr and accuracy. The main metric for retrieval is NDCG@10. AskUbuntuDupQuestions reports map@1000 as its main score in MTEB, and so do we.

The task, the model profile, the dataset limit and any other setting are part of a benchmark's identity. A run on a subset is never compared with a full-dataset number.

### The 2% Band

A result is read against its reference with a 2% band:

| Verdict | When |
| --- | --- |
| Pass | Within 2% of the reference, in either direction |
| Fail | More than 2% worse. A run with no output or a missing dependency also fails, and so does a gated model whose reference file is missing |
| Warn | More than 2% better. Treated as suspicious and checked |
| Measurement only | No reference has been established for that model and task yet, or the reference came from different hardware or a different model revision |

One model on one task may get a 5% band, with a written reason. Anything beyond that is listed as a known failure.

The band is the policy maintainers apply when they read a delta and decide whether a published number still holds. The public example applies the same band to a quality run and prints the verdict beside the delta. The example is not a gate: a comparison verdict never changes the exit status, which is 1 only when a run wrote no predictions.

### The Published Quality Numbers

Run cold, from the same client as the performance runs:

| Model | Dataset | Metric | Score |
| --- | --- | --- | ---: |
| `lightonai/GTE-ModernColBERT-v1` | SciFact | NDCG@10 | 0.760 |
| `BAAI/bge-m3` (multivector) | SciFact | NDCG@10 | 0.696 |
| `Qwen/Qwen3-Reranker-4B` | AskUbuntuDupQuestions | main score | 0.645 |
| `Qwen/Qwen3-Reranker-0.6B` | AskUbuntuDupQuestions | main score | 0.595 |
| `prithivida/Splade_PP_en_v2` | NFCorpus | NDCG@10 | 0.331 |
| `BAAI/bge-m3` (sparse) | NFCorpus | NDCG@10 | 0.284 |

The Qwen3-Reranker-0.6B number predates the server change that scores this reranker in float32 so its top candidates stop tying. A full run today scores about 3% higher and reads `WARN (suspicious-improvement)`.

## Performance

### Client and Load

The client is a separate machine in the same cloud region as the API, reaching it over the public internet: two workers of two processes each for encode, one worker of two processes for rerank.

Requests go out at a fixed rate with deterministic arrivals; request `i` is due at `i / rate` seconds. Adding processes does not raise the rate. Each run has three levels of 60 seconds, and the website shows only the lowest:

| Request type | Levels (rps) | In-flight cap per process |
| --- | --- | --- |
| Sparse encode | 10, 20, 25 | 64 |
| Multivector encode | 1, 3, 5 | 64 |
| Rerank | 0.25, 0.5, 1 | 128 |

### Requests

An encode request carries one document, drawn from the first 256 non-empty corpus documents, ordered by document id compared as a string, and sent as text. A rerank request carries one query with its 20 candidates, drawn from the first 64 queries. Input admission caps 512 units for sparse, 8192 for multivector, and 1000 tokens per rerank document.

One attempt per request, no retries; a run with any SDK retry is invalid. The timeout is 30 s for encode and 300 s for rerank.

### Run Order

1. One readiness request, with up to 300 s to scale from zero.
2. 30 s of warmup, unrecorded, then 10 s of settle.
3. The three measured levels, 60 s each.
4. 10 s of drain.

Warmup requests are excluded from every metric.

### Metrics

| Metric | Definition |
| --- | --- |
| Latency p50, p95, p99 | From first send to final response, over successful requests; nearest-rank percentiles; processes merged, never averaged |
| Achieved rate | Successful requests completed inside the window, divided by the window length |
| Success rates | Overall, inside the window, and without retries |
| Scheduling lag p99 | How late the client sent its requests. A run is invalid above 20 ms |

### The Published Performance Numbers

| Model | Load | p50 (ms) | p95 (ms) | p99 (ms) | Achieved rate (rps) |
| --- | --- | ---: | ---: | ---: | ---: |
| `prithivida/Splade_PP_en_v2` (sparse) | 10 rps | 144.4 | 241.6 | 491.5 | 9.98 |
| `BAAI/bge-m3` (sparse) | 10 rps | 172.7 | 217.8 | 387.0 | 9.98 |
| `BAAI/bge-m3` (multivector) | 1 rps | 353.7 | 1164.3 | 1407.2 | 1.0 |
| `lightonai/GTE-ModernColBERT-v1` | 1 rps | 178.8 | 198.8 | 285.0 | 1.0 |
| `Qwen/Qwen3-Reranker-0.6B` | 0.25 rps | 210.5 | 230.2 | 230.2 | 0.25 |
| `Qwen/Qwen3-Reranker-4B` | 0.25 rps | 489.5 | 579.4 | 579.4 | 0.25 |

All six runs had a success rate of 1.0. At 0.25 rps for 60 s there are 15 requests, so p95 and p99 are the same request, the slowest one.

## Known Traps

Most wrong numbers we found came from the setup, not the model. Each row below was found in a real run.

| Trap | What happens | What it did |
| --- | --- | --- |
| Lower precision | fp16 or bf16 inference while the official number used fp32 | 2 to 7% lower on one code-retrieval task, up to 8.5% on one model |
| Wrong prompt | A task instruction is added to a model that was not trained with one | One sparse model fell from 0.154 to 0.061 on one task |
| Missing prompt | A model that needs a task instruction does not get it | One model fell from 0.267 to 0.250 |
| Different GPU class | The same model scores differently on another GPU type | 4 to 5% between two GPU types on one task; one late-interaction setup scored 0 on a new GPU generation |
| Reference under another protocol | The official number used a recipe nobody can reproduce, or the scoring changed after the reference was set | A scoring switch gave -13.5% on one task and a false +37% on another |
| Subset vs full dataset | A limited run is compared with a full-dataset number | An OCR model scored 0.674 on 50 pages and 0.777 on the full set |
| Library or tokenizer version | A new library version changes how text reaches the tokenizer | About 3% left on one model after the precision fix |
| Unpinned model revision | The model changes upstream and the old reference no longer fits | One model stopped scoring between two runs |
| Model missing from the plan | The model is skipped silently | No number at all, instead of a failure |
| Missing optional dependency | A missing package looks like a quality failure | The benchmark never ran |
| Self-measured reference | The reference was taken from SIE's own output | The check passes forever, whatever SIE returns |
| Load too high for the client | The client, not the server, becomes the limit | A 10/20/25 rps multivector run failed validity and was rerun at 1/3/5 |
| Client location | Latency includes the network path to the API | The client's location is recorded, and a run without it is rejected |

## Repeat a Published Number

Source: [examples/evaluations/published-benchmarks/README.md](https://github.com/superlinked/sie/blob/main/examples/evaluations/published-benchmarks/README.md)

[examples/evaluations/published-benchmarks](https://github.com/superlinked/sie/tree/main/examples/evaluations/published-benchmarks) in the SIE repository repeats the method above from one client process on your machine. `references/` holds twelve files, one projection of each published record: model and profile, task key, pinned dataset revision, request shape, timing constants and the published metrics. `run.py quality` sends the same requests through the SDK, ranks in the client, scores the ranking with pytrec_eval and writes a result with the delta against every published metric. `run.py performance` sends a record's requests at its published rate and aggregates their timing the way the record did. `score.py` recomputes every metric from the files a run wrote, with no key and no inference call; quality rescoring also loads the relevance judgments from the pinned dataset (the Hugging Face cache or the network), performance rescoring needs only the request log. `validation/` holds the example's own observed runs, the ones the [Expected Spread](#expected-spread) section describes.

Every result records `comparable`, `comparable_reasons`, `comparison_basis` and `validity`, so a result says for itself whether it was made the way the record was: the same model, profile, task key, output contract, request shape and endpoint, from one process, with one execution revision throughout the run. A self-hosted server reached directly reports no execution revision, so `execution_revisions` stays empty there and that check cannot fire. It does not compare the served model's revision with the record's, so a result made after the served model changed can still read `comparable: true`.

| Model | Quality reference | Performance reference |
| --- | --- | --- |
| `prithivida/Splade_PP_en_v2` | `prod-web-fw-20260905-splade-nfcorpus-quality-02` | `prod-web-fw-20260905-splade-nfcorpus-perf-01` |
| `BAAI/bge-m3` (sparse) | `prod-web-fw-20260905-bge-m3-sparse-nfcorpus-quality-04` | `prod-web-fw-20260905-bge-m3-sparse-nfcorpus-perf-03` |
| `lightonai/GTE-ModernColBERT-v1` | `prod-web-fw-20260905-gte-moderncolbert-scifact-quality-06` | `prod-web-fw-20260905-gte-moderncolbert-scifact-perf-v2-r1` |
| `BAAI/bge-m3` (multivector) | `prod-web-fw-20260905-bge-m3-multivector-scifact-quality-08` | `prod-web-fw-20260905-bge-m3-multivector-scifact-perf-v2-r1` |
| `Qwen/Qwen3-Reranker-0.6B` | `prod-web-fw-20260905-qwen3-reranker-0-6b-askubuntu-quality-10` | `prod-web-fw-20260905-qwen3-reranker-0-6b-askubuntu-perf-09` |
| `Qwen/Qwen3-Reranker-4B` | `prod-web-fw-20260905-qwen3-reranker-4b-askubuntu-quality-12` | `prod-web-fw-20260905-qwen3-reranker-4b-askubuntu-perf-11` |

From `examples/evaluations/published-benchmarks`, install the locked dependencies. The default endpoint, `https://api.superlinked.com`, needs your API key in the environment:

```sh
uv sync --frozen
export SIE_API_KEY=your-key
```

Against the default endpoint, every run goes through your key as normal API traffic. A smoke run keeps the whole SciFact corpus and the first 50 queries that have a relevance judgment, so it still encodes every document:

```sh
uv run python run.py quality \
  --reference references/prod-web-fw-20260905-gte-moderncolbert-scifact-quality-06.json \
  --task "mteb/SciFact?limit=50"
```

The full run repeats the published quality record. Drop `--task` and the reference's own task key applies:

```sh
uv run python run.py quality \
  --reference references/prod-web-fw-20260905-gte-moderncolbert-scifact-quality-06.json
```

A performance run takes about two minutes: one readiness request, 30 s of warmup, 10 s of settle, 60 s of scheduled requests and up to 10 s of drain. It runs one level, the record's published one:

```sh
uv run python run.py performance \
  --reference references/prod-web-fw-20260905-gte-moderncolbert-scifact-perf-v2-r1.json
```

Each run writes its result under `results/`. To recompute every metric from what a run wrote, with no key and no inference call:

```sh
uv run python score.py results/<id>-<utc>.json
```

The example prints each metric beside the published value and the delta. A quality run also prints one `verdict:` line, which reads the run against the record's `main_score` with the 2% band; a `FAIL` adds `expected:`, `got:` and `reproduce:`. The smoke run above reads `MEASUREMENT`, because a task key with `limit` gets no pass or fail. A performance run has no verdict. `score.py` recomputes the verdict from the result's baseline and the rescored metrics.

### Run It Against Your Own Server

The example also runs against the open-source server. Start `sie-server serve` with the reference's model and pass `--endpoint http://localhost:8080` to either command. Leave `SIE_API_KEY` unset. A key in your shell is scoped to the endpoint `SIE_BASE_URL` names, or to the default endpoint when it is unset, so it is never sent to another server. Only a server in token mode asks for one: it prints where it saved the token it generated; export that value in `SIE_API_KEY` together with `SIE_BASE_URL=http://localhost:8080`.

The result reads `comparable: false`, with the reason `endpoint http://localhost:8080 != https://api.superlinked.com`. Quality from the same pinned model revision should land in the same band as the record. Performance describes your hardware and your network path. And `execution_revisions` stays empty, because a server reached directly reports no execution revision.

The first request loads the model. The readiness request waits up to 300 s for it, so a cold server still gives a valid run; a first download and load that takes longer fails that request, the run says so, and a second run once the model is loaded is valid. The README's "Run it against your own server" section has the full commands.

## Test a New Model

Source: [examples/evaluations/published-benchmarks/README.md](https://github.com/superlinked/sie/blob/main/examples/evaluations/published-benchmarks/README.md)

The same example tests a model that has no record, so you can check it against its official number before you open a pull request. Add the model's config to `packages/sie_server/models/` in a checkout, as [Adding Models](/docs/engine/adding-models/) describes, and from the root of that checkout run `mise run serve -- --host 127.0.0.1 -m my-org/my-model`. Then, from [examples/evaluations/published-benchmarks](https://github.com/superlinked/sie/tree/main/examples/evaluations/published-benchmarks), run it:

```sh
uv run python run.py quality \
  --endpoint http://localhost:8080 \
  --model my-org/my-model \
  --task mteb/SciFact \
  --output dense \
  --items-per-request 32
```

Before any inference request, the example looks the official number up in the public MTEB results repository, [embeddings-benchmark/results](https://github.com/embeddings-benchmark/results), and prints it with a link to the file. The lookup sends the model id and the task name to GitHub, so pass the number yourself for a model id you want to keep private. The example ships no official numbers of its own.

When the repository has no number for that model and task, the run reads `MEASUREMENT` and the output says why. Then take the number from a model card, paper or leaderboard, on the same benchmark and the same metric as the run, and pass it with `--baseline ndcg_at_10=<official-value> --baseline-source "<URL>"`. The result records its source. A number you pass wins over the lookup. `--baseline` takes `main_score`, `accuracy` or a metric with a cut such as `ndcg_at_10` or `map_at_1000`, with a value above 0 and at most 1, and a bad value is refused before any request. The result reads `comparable: false`, because there is no record to compare it with.

The `verdict:` line applies [the 2% band](#the-2-band): `PASS` from baseline × 0.98 to baseline ÷ 0.98, `FAIL (regression)` below that, `WARN (suspicious-improvement)` above it. A run with no number to compare with reads `MEASUREMENT`, and so does a task key with `limit`, such as `mteb/SciFact?limit=50`: a limited run gets no pass or fail against a full-dataset number. A run with no predictions or a metric of exactly 0 reads `FAIL (no-output)`, limited or not. As in a repeat, the verdict gates nothing: the exit status stays 0 unless the run wrote no predictions.

`--output dense` works on encode tasks, in quality and performance runs, and a quality run scores it with cosine over L2-normalised vectors. A quality run refuses dense on a profile named `muvera`, because those vectors are not always meant for cosine. `sparse` and `multivector` work too. The score tasks, `mteb/AskUbuntuDupQuestions`, `mteb/MMarcoReranking`, `mteb/T2Reranking`, `mteb/CMedQAv1-reranking` and `mteb/CMedQAv2-reranking`, take neither `--output` nor `--items-per-request`.

The example sends no instruction unless you pass `--instruction <text>`, and then only with queries, on encode tasks. It cannot tell whether your model needs one, because the public models endpoint does not expose a query template, so check the model card. A wrong instruction and a missing one have both cost real score; [Known Traps](#known-traps) has the numbers.

## Check a Number on a Model Page

The same example runs most text retrieval and rerank rows on the [model pages](/models/). Take the task, model, profile and output from the row, and pass the row's metric and value as `--baseline`, such as `--baseline ndcg_at_10=0.31411`, to read your run against it. `<output>` is `dense`, `sparse` or `multivector`, whichever the model or profile returns.

| Row | Flags |
| --- | --- |
| A plain row | `--model <model> --task <task> --output <output> --items-per-request 32` |
| A profile row | `--model <model>:<profile> --task <task> --output <output> --items-per-request 32` |
| A reranker row | `--model <reranker> --task <task>` |
| A row that says "reranking the top 50 from `<model>`" | `--model <reranker> --task "<task>?candidates_model=<model>&candidates_k=50"` |

It does not run image retrieval, or tasks outside MTEB such as extraction, classification, OCR, detection, speech and generation.

## Check a Time on a Task Page

Source: [examples/evaluations/published-benchmarks/README.md](https://github.com/superlinked/sie/blob/main/examples/evaluations/published-benchmarks/README.md)

Ten per-request times on this site come from studies whose inputs are public in the [superlinked/sie-task-evidence](https://huggingface.co/datasets/superlinked/sie-task-evidence) dataset. `run.py latency`, in the same example, times them on your own server and prints your numbers beside the published ones.

| `--page` | Where the site shows it | Model | Image tag | Timed on |
| --- | --- | --- | --- | --- |
| `classify` | `/classify`: 13.1 ms | `fastino/GLiNER2.5-Decide` | `v0.11.0-cuda12-transformers5` | RTX PRO 6000 |
| `search` | home page and `/about`: 15.7 ms | `Qwen/Qwen3-Embedding-4B` | `v0.11.0-cuda12-sglang-embedding` | RTX PRO 6000 |
| `named-entities` | industry pages, extract entities: 24 ms | `numind/NuNER_Zero-span` | `v0.11.0-cuda12-default` | RTX PRO 6000 |
| `redact` | `/redact`: 61 ms | `superlinked/pii-redact` | `v0.11.0-cuda12-default` | RTX PRO 6000 |
| `typed-decisions` | `/typed-decisions`: 13.8 ms | `MoritzLaurer/deberta-v3-large-zeroshot-v2.0` | `v0.11.0-cuda12-default` | RTX PRO 6000 |
| `rerank` | `/rerank`: 55 ms | `Qwen/Qwen3-Reranker-4B:fast` | `v0.11.0-cuda12-default` | H100 |
| `request-routing` | `/request-routing`: 8.5 ms | `Qwen/Qwen3-Reranker-4B:fast` | `v0.11.0-cuda12-default` | H100 |
| `ocr` | industry pages, OCR a page: 287 ms | `zai-org/GLM-OCR` | `v0.11.0-cuda12-sglang-vision-extract` | H100 |
| `chat` | `/chat`: 0.74 s | `Qwen/Qwen3.8-27B-FP8` | `v0.11.0-cuda13-sglang-cu130` | H100 |
| `structured-output` | industry pages, structured output: 938 ms | `Qwen/Qwen3.8-27B-FP8` | `v0.11.0-cuda13-sglang-cu130` | H100 |

On a Linux machine with an NVIDIA GPU, start the page's image with its model preloaded. For `classify`:

```sh
docker run --gpus all -p 127.0.0.1:8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  -e SIE_PRELOAD_MODELS=fastino/GLiNER2.5-Decide \
  ghcr.io/superlinked/sie-server:v0.11.0-cuda12-transformers5
```

Then, from `examples/evaluations/published-benchmarks` on the same machine:

```sh
uv sync --frozen
uv run python run.py latency --page classify --endpoint http://localhost:8080
```

`chat` and `structured-output` need an 80 GB GPU, and their CUDA 13 image needs an NVIDIA driver from the 580 series or newer.

The run rebuilds the page's request list from the study's inputs at pinned revisions and stops before the first request if its SHA-256 differs from the one the study published. The `redact` and `typed-decisions` studies published none, so for those two the check uses a hash the example pins. It sends the study's warm-up requests without timing them. Then every request goes out once, one in flight. The output shows your n, p50, p95 and mean beside the published values, and `results/` gets the time of every request.

Two differences apply to every page. The study's client sent each request over a local socket inside the server's container; yours goes over HTTP through the SDK. And your server runs on your GPU, not the study's. Some pages add more, such as a published median over fresh servers, and the output names them all.

## Measured Internally

The public example cannot repeat these numbers yet:

- Model-page rows outside text retrieval and reranking: image retrieval, extraction, classification, OCR, detection, speech and generation. They come from the internal measurement pipeline.
- The `/detect` time, 82.2 ms. Its 300 photos are not in the evidence dataset.

The task pages' accuracy and cost charts state their own method. The dataset, scoring and prices sit beside each chart, and "Run this eval" opens the example or the recorded evidence.

## What Differs When You Repeat It

Same method, different client. Four things change. The first is named in the result file's `comparable_reasons`; the other three are not.

**One process instead of four or two.** The encode records ran two workers of two processes, the reranker records one worker of two. The example runs one process. The schedule is the union of the record's per-process partitions, so the offered load is the same, but one process carries a 4× or 2× higher per-process rate, and the in-flight cap and the 20 ms lag gate bind sooner. Every performance run from the example is therefore `comparable: false`, with the reason listed. A full quality run with no override, and one execution revision throughout, is `comparable: true`; the smoke key or a revision change during the run sets it to false.

**Your client's location.** The records' client sat in the same cloud region as the API. From anywhere else, every latency includes the network path and the response transfer, and a multivector response is hundreds of kilobytes. Read the rate and success metrics first, and read the latency deltas in the light of where the run was made.

**A verdict that gates nothing.** A quality run prints a verdict from the 2% band above, even when the result reads `comparable: false`. That band is also how maintainers read a result, and their reading decides whether a number holds; the example's verdict decides nothing. It leaves the exit status alone, and nothing the example writes reaches a record or this site.

**Service state the client cannot pin.** Server-side truncation and any default instruction are part of the service's configuration. The SDK exposes a model's maximum sequence length and weights revision, and not those two settings, so a run cannot verify them on its own.

## Expected Spread

The example's `validation/` directory holds three consecutive runs of the Splade NFCorpus quality reference and three of the GTE-ModernColBERT SciFact performance reference at 1 rps, made from one client outside the API's region. These are observed values from one client; no value here is a pass threshold or quality floor.

### Quality, Splade NFCorpus

All three runs were `comparable: true` and `valid: true`.

| Metrics | Across the three runs |
| --- | --- |
| Main score (0.33056) and every ndcg, map, recall and precision cut at 1 to 20 | Equal to the published value in every run |
| The 100 and 1000 cuts, and mrr at 3 and beyond | Moved by up to 0.00062 between runs |
| ndcg@1000 and recall@1000 | About 0.0002 and 0.0011 below the published value in every run |

The sparse weights the service returns differ slightly from run to run: most document scores are identical between runs, the rest differ by about 0.001, at most 0.03, so near-tied documents swap places. That moves the 100 and 1000 cuts and the mrr cuts from 3 up. The ndcg@1000 and recall@1000 offsets sit at the cut where the order of equal scores is unspecified.

### Performance, GTE-ModernColBERT SciFact at 1 rps

All three runs were `valid: true`: 60 requests, no failure, no timeout, no SDK retry, an in-flight peak of 1 or 2. The rate and success metrics equaled the published values in every run. The latency deltas are consistent with the client's location. Milliseconds as the example reports them:

| Metric | Published | Observed, three runs | Delta range | Spread between runs |
| --- | ---: | ---: | ---: | ---: |
| Scheduling lag p99 (ms) | 5.886 | 9.626 to 14.316 | 3.740 to 8.430 | 4.690 |
| Latency p50 (ms) | 178.819 | 444.978 to 464.829 | 266.159 to 286.010 | 19.850 |
| Latency p95 (ms) | 198.758 | 637.308 to 654.308 | 438.550 to 455.550 | 17.000 |
| Latency p99 (ms) | 285.029 | 890.516 to 1121.944 | 605.487 to 836.915 | 231.428 |

The p50 and p95 moved by about 20 ms or less between runs. The p99 is one request in 60; it moved by 231 ms between consecutive runs. The lag p99 of 9.6 to 14.3 ms stayed under the 20 ms tolerance in every run.

A transport probe from the same client split three warm encode requests of one document into 268 to 284 ms to the first response header, of which one network round trip, the TLS handshake, took 163 ms, which leaves an estimated 105 to 121 ms for the service's own answer, followed by 485 to 561 ms of response transfer for the document's multivector. The probe shows the transport share of one request. It does not decompose or bound the runs' percentiles.

## What's Next

- [Quality Evaluation](/docs/evals/quality/): the shipped retrieval benchmark and MTEB-style workflows.
- [Performance Evaluation](/docs/evals/performance/): timing metadata, pools and load patterns for your own runs.
- [Custom Evals](/docs/evals/custom/): the worked pattern on your own labeled data.
- [examples/evaluations/published-benchmarks](https://github.com/superlinked/sie/tree/main/examples/evaluations/published-benchmarks): the public example on GitHub.