Measurement Methodology

The published Cloud benchmark numbers for the six models below are SIE Cloud numbers, measured through the public API from a separate client machine in the same cloud region as the API. This page states the method behind those numbers so you can repeat it: what a record holds, where the quality reference comes from, how load is applied, which setup mistakes produced wrong numbers in our own runs, and what three repeat runs of the public example looked like.

What We Measure

Two things, kept apart.

Quality is how good the model’s output is on a public benchmark, compared with the model’s official number. The score is the benchmark’s own metric against its ground truth; for NFCorpus and SciFact that is NDCG@10, for AskUbuntuDupQuestions map@1000.

Performance is latency and throughput at a fixed request rate against a warm model: what one request costs in time at a steady load. Not the peak capacity of the client or of the service.

The rest of the site’s numbers were made another way. The per-request times on the task pages come from public studies that ran SIE on a GPU server, with the client inside the server’s container sending one request at a time; Check a Time on a Task Page repeats ten of them on your own GPU. Most accuracy figures on the model pages come from our internal measurement pipeline, which runs SIE against the named public datasets. You can repeat many of those too. Measured Internally lists the numbers you cannot.

Where the Published Numbers Come From

Each of those numbers comes from one record. There are twelve: six model and output pairs, each with one quality record and one performance record. All of them were measured on 2026-09-05 through the public API.

Model	Output	Dataset (MTEB)	Performance load
BAAI/bge-m3	multivector	SciFact	fixed 1 rps
BAAI/bge-m3	sparse	NFCorpus	fixed 10 rps
lightonai/GTE-ModernColBERT-v1	multivector	SciFact	fixed 1 rps
prithivida/Splade_PP_en_v2	sparse	NFCorpus	fixed 10 rps
Qwen/Qwen3-Reranker-0.6B	scores (rerank)	AskUbuntuDupQuestions	fixed 0.25 rps
Qwen/Qwen3-Reranker-4B	scores (rerank)	AskUbuntuDupQuestions	fixed 0.25 rps

A record carries what a repeat needs: the model name and its pinned revision; the dataset name, revision, subset, limit and a content digest; where the client ran, with its worker and process count; the request operation, items per request, input limit and output type; the load level with its start and end time; each metric’s name, unit, aggregation and value; and SHA-256 hashes of the raw evidence and the run manifest.

A run becomes a website number after two checks: a validator marks it valid, then a person approves it. Each of the twelve numbers landed through its own reviewed change.

Quality

Where the Reference Comes From

The reference is chosen in a fixed order:

1. The official number, when it was reported for the same task and the same main metric.
2. Otherwise, the model’s official implementation run by us on one GPU class in fp32: sentence-transformers for dense and sparse models, PyLate for ColBERT models, Transformers for the Qwen rerankers.

SIE’s own output is never the reference, and a reference is never lowered to match SIE.

Scoring

The website numbers use the MTEB retrieval tasks NFCorpus, SciFact and AskUbuntuDupQuestions, scored with the conventions of mteb 2.7.5: pytrec_eval ndcg, map, recall and precision at cuts 1, 3, 5, 10, 20, 100 and 1000 with graded relevance, plus mrr and accuracy. The main metric for retrieval is NDCG@10. AskUbuntuDupQuestions reports map@1000 as its main score in MTEB, and so do we.

The task, the model profile, the dataset limit and any other setting are part of a benchmark’s identity. A run on a subset is never compared with a full-dataset number.

The 2% Band

A result is read against its reference with a 2% band:

Verdict	When
Pass	Within 2% of the reference, in either direction
Fail	More than 2% worse. A run with no output or a missing dependency also fails, and so does a gated model whose reference file is missing
Warn	More than 2% better. Treated as suspicious and checked
Measurement only	No reference has been established for that model and task yet, or the reference came from different hardware or a different model revision

One model on one task may get a 5% band, with a written reason. Anything beyond that is listed as a known failure.

The band is the policy maintainers apply when they read a delta and decide whether a published number still holds. The public example applies the same band to a quality run and prints the verdict beside the delta. The example is not a gate: a comparison verdict never changes the exit status, which is 1 only when a run wrote no predictions.

The Published Quality Numbers

Run cold, from the same client as the performance runs:

Model	Dataset	Metric	Score
lightonai/GTE-ModernColBERT-v1	SciFact	NDCG@10	0.760
BAAI/bge-m3 (multivector)	SciFact	NDCG@10	0.696
Qwen/Qwen3-Reranker-4B	AskUbuntuDupQuestions	main score	0.645
Qwen/Qwen3-Reranker-0.6B	AskUbuntuDupQuestions	main score	0.595
prithivida/Splade_PP_en_v2	NFCorpus	NDCG@10	0.331
BAAI/bge-m3 (sparse)	NFCorpus	NDCG@10	0.284

The Qwen3-Reranker-0.6B number predates the server change that scores this reranker in float32 so its top candidates stop tying. A full run today scores about 3% higher and reads WARN (suspicious-improvement).

Performance

Client and Load

The client is a separate machine in the same cloud region as the API, reaching it over the public internet: two workers of two processes each for encode, one worker of two processes for rerank.

Requests go out at a fixed rate with deterministic arrivals; request i is due at i / rate seconds. Adding processes does not raise the rate. Each run has three levels of 60 seconds, and the website shows only the lowest:

Request type	Levels (rps)	In-flight cap per process
Sparse encode	10, 20, 25	64
Multivector encode	1, 3, 5	64
Rerank	0.25, 0.5, 1	128

Requests

An encode request carries one document, drawn from the first 256 non-empty corpus documents, ordered by document id compared as a string, and sent as text. A rerank request carries one query with its 20 candidates, drawn from the first 64 queries. Input admission caps 512 units for sparse, 8192 for multivector, and 1000 tokens per rerank document.

One attempt per request, no retries; a run with any SDK retry is invalid. The timeout is 30 s for encode and 300 s for rerank.

Run Order

1. One readiness request, with up to 300 s to scale from zero.
2. 30 s of warmup, unrecorded, then 10 s of settle.
3. The three measured levels, 60 s each.
4. 10 s of drain.

Warmup requests are excluded from every metric.

Metrics

Metric	Definition
Latency p50, p95, p99	From first send to final response, over successful requests; nearest-rank percentiles; processes merged, never averaged
Achieved rate	Successful requests completed inside the window, divided by the window length
Success rates	Overall, inside the window, and without retries
Scheduling lag p99	How late the client sent its requests. A run is invalid above 20 ms

The Published Performance Numbers

Model	Load	p50 (ms)	p95 (ms)	p99 (ms)	Achieved rate (rps)
prithivida/Splade_PP_en_v2 (sparse)	10 rps	144.4	241.6	491.5	9.98
BAAI/bge-m3 (sparse)	10 rps	172.7	217.8	387.0	9.98
BAAI/bge-m3 (multivector)	1 rps	353.7	1164.3	1407.2	1.0
lightonai/GTE-ModernColBERT-v1	1 rps	178.8	198.8	285.0	1.0
Qwen/Qwen3-Reranker-0.6B	0.25 rps	210.5	230.2	230.2	0.25
Qwen/Qwen3-Reranker-4B	0.25 rps	489.5	579.4	579.4	0.25

All six runs had a success rate of 1.0. At 0.25 rps for 60 s there are 15 requests, so p95 and p99 are the same request, the slowest one.

Known Traps

Most wrong numbers we found came from the setup, not the model. Each row below was found in a real run.

Trap	What happens	What it did
Lower precision	fp16 or bf16 inference while the official number used fp32	2 to 7% lower on one code-retrieval task, up to 8.5% on one model
Wrong prompt	A task instruction is added to a model that was not trained with one	One sparse model fell from 0.154 to 0.061 on one task
Missing prompt	A model that needs a task instruction does not get it	One model fell from 0.267 to 0.250
Different GPU class	The same model scores differently on another GPU type	4 to 5% between two GPU types on one task; one late-interaction setup scored 0 on a new GPU generation
Reference under another protocol	The official number used a recipe nobody can reproduce, or the scoring changed after the reference was set	A scoring switch gave -13.5% on one task and a false +37% on another
Subset vs full dataset	A limited run is compared with a full-dataset number	An OCR model scored 0.674 on 50 pages and 0.777 on the full set
Library or tokenizer version	A new library version changes how text reaches the tokenizer	About 3% left on one model after the precision fix
Unpinned model revision	The model changes upstream and the old reference no longer fits	One model stopped scoring between two runs
Model missing from the plan	The model is skipped silently	No number at all, instead of a failure
Missing optional dependency	A missing package looks like a quality failure	The benchmark never ran
Self-measured reference	The reference was taken from SIE’s own output	The check passes forever, whatever SIE returns
Load too high for the client	The client, not the server, becomes the limit	A 10/20/25 rps multivector run failed validity and was rerun at 1/3/5
Client location	Latency includes the network path to the API	The client’s location is recorded, and a run without it is rejected

Repeat a Published Number

examples/evaluations/published-benchmarks in the SIE repository repeats the method above from one client process on your machine. references/ holds twelve files, one projection of each published record: model and profile, task key, pinned dataset revision, request shape, timing constants and the published metrics. run.py quality sends the same requests through the SDK, ranks in the client, scores the ranking with pytrec_eval and writes a result with the delta against every published metric. run.py performance sends a record’s requests at its published rate and aggregates their timing the way the record did. score.py recomputes every metric from the files a run wrote, with no key and no inference call; quality rescoring also loads the relevance judgments from the pinned dataset (the Hugging Face cache or the network), performance rescoring needs only the request log. validation/ holds the example’s own observed runs, the ones the Expected Spread section describes.

Every result records comparable, comparable_reasons, comparison_basis and validity, so a result says for itself whether it was made the way the record was: the same model, profile, task key, output contract, request shape and endpoint, from one process, with one execution revision throughout the run. A self-hosted server reached directly reports no execution revision, so execution_revisions stays empty there and that check cannot fire. It does not compare the served model’s revision with the record’s, so a result made after the served model changed can still read comparable: true.

Model	Quality reference	Performance reference
prithivida/Splade_PP_en_v2	prod-web-fw-20260905-splade-nfcorpus-quality-02	prod-web-fw-20260905-splade-nfcorpus-perf-01
BAAI/bge-m3 (sparse)	prod-web-fw-20260905-bge-m3-sparse-nfcorpus-quality-04	prod-web-fw-20260905-bge-m3-sparse-nfcorpus-perf-03
lightonai/GTE-ModernColBERT-v1	prod-web-fw-20260905-gte-moderncolbert-scifact-quality-06	prod-web-fw-20260905-gte-moderncolbert-scifact-perf-v2-r1
BAAI/bge-m3 (multivector)	prod-web-fw-20260905-bge-m3-multivector-scifact-quality-08	prod-web-fw-20260905-bge-m3-multivector-scifact-perf-v2-r1
Qwen/Qwen3-Reranker-0.6B	prod-web-fw-20260905-qwen3-reranker-0-6b-askubuntu-quality-10	prod-web-fw-20260905-qwen3-reranker-0-6b-askubuntu-perf-09
Qwen/Qwen3-Reranker-4B	prod-web-fw-20260905-qwen3-reranker-4b-askubuntu-quality-12	prod-web-fw-20260905-qwen3-reranker-4b-askubuntu-perf-11

From examples/evaluations/published-benchmarks, install the locked dependencies. The default endpoint, https://api.superlinked.com, needs your API key in the environment:

uv sync --frozen
export SIE_API_KEY=your-key

Against the default endpoint, every run goes through your key as normal API traffic. A smoke run keeps the whole SciFact corpus and the first 50 queries that have a relevance judgment, so it still encodes every document:

uv run python run.py quality \
  --reference references/prod-web-fw-20260905-gte-moderncolbert-scifact-quality-06.json \
  --task "mteb/SciFact?limit=50"

The full run repeats the published quality record. Drop --task and the reference’s own task key applies:

uv run python run.py quality \
  --reference references/prod-web-fw-20260905-gte-moderncolbert-scifact-quality-06.json

A performance run takes about two minutes: one readiness request, 30 s of warmup, 10 s of settle, 60 s of scheduled requests and up to 10 s of drain. It runs one level, the record’s published one:

uv run python run.py performance \
  --reference references/prod-web-fw-20260905-gte-moderncolbert-scifact-perf-v2-r1.json

Each run writes its result under results/. To recompute every metric from what a run wrote, with no key and no inference call:

uv run python score.py results/<id>-<utc>.json

The example prints each metric beside the published value and the delta. A quality run also prints one verdict: line, which reads the run against the record’s main_score with the 2% band; a FAIL adds expected:, got: and reproduce:. The smoke run above reads MEASUREMENT, because a task key with limit gets no pass or fail. A performance run has no verdict. score.py recomputes the verdict from the result’s baseline and the rescored metrics.

Run It Against Your Own Server

The example also runs against the open-source server. Start sie-server serve with the reference’s model and pass --endpoint http://localhost:8080 to either command. Leave SIE_API_KEY unset. A key in your shell is scoped to the endpoint SIE_BASE_URL names, or to the default endpoint when it is unset, so it is never sent to another server. Only a server in token mode asks for one: it prints where it saved the token it generated; export that value in SIE_API_KEY together with SIE_BASE_URL=http://localhost:8080.

The result reads comparable: false, with the reason endpoint http://localhost:8080 != https://api.superlinked.com. Quality from the same pinned model revision should land in the same band as the record. Performance describes your hardware and your network path. And execution_revisions stays empty, because a server reached directly reports no execution revision.

The first request loads the model. The readiness request waits up to 300 s for it, so a cold server still gives a valid run; a first download and load that takes longer fails that request, the run says so, and a second run once the model is loaded is valid. The README’s “Run it against your own server” section has the full commands.

Test a New Model

The same example tests a model that has no record, so you can check it against its official number before you open a pull request. Add the model’s config to packages/sie_server/models/ in a checkout, as Adding Models describes, and from the root of that checkout run mise run serve -- --host 127.0.0.1 -m my-org/my-model. Then, from examples/evaluations/published-benchmarks, run it:

uv run python run.py quality \
  --endpoint http://localhost:8080 \
  --model my-org/my-model \
  --task mteb/SciFact \
  --output dense \
  --items-per-request 32

Before any inference request, the example looks the official number up in the public MTEB results repository, embeddings-benchmark/results, and prints it with a link to the file. The lookup sends the model id and the task name to GitHub, so pass the number yourself for a model id you want to keep private. The example ships no official numbers of its own.

When the repository has no number for that model and task, the run reads MEASUREMENT and the output says why. Then take the number from a model card, paper or leaderboard, on the same benchmark and the same metric as the run, and pass it with --baseline ndcg_at_10=<official-value> --baseline-source "<URL>". The result records its source. A number you pass wins over the lookup. --baseline takes main_score, accuracy or a metric with a cut such as ndcg_at_10 or map_at_1000, with a value above 0 and at most 1, and a bad value is refused before any request. The result reads comparable: false, because there is no record to compare it with.

The verdict: line applies the 2% band: PASS from baseline × 0.98 to baseline ÷ 0.98, FAIL (regression) below that, WARN (suspicious-improvement) above it. A run with no number to compare with reads MEASUREMENT, and so does a task key with limit, such as mteb/SciFact?limit=50: a limited run gets no pass or fail against a full-dataset number. A run with no predictions or a metric of exactly 0 reads FAIL (no-output), limited or not. As in a repeat, the verdict gates nothing: the exit status stays 0 unless the run wrote no predictions.

--output dense works on encode tasks, in quality and performance runs, and a quality run scores it with cosine over L2-normalised vectors. A quality run refuses dense on a profile named muvera, because those vectors are not always meant for cosine. sparse and multivector work too. The score tasks, mteb/AskUbuntuDupQuestions, mteb/MMarcoReranking, mteb/T2Reranking, mteb/CMedQAv1-reranking and mteb/CMedQAv2-reranking, take neither --output nor --items-per-request.

The example sends no instruction unless you pass --instruction <text>, and then only with queries, on encode tasks. It cannot tell whether your model needs one, because the public models endpoint does not expose a query template, so check the model card. A wrong instruction and a missing one have both cost real score; Known Traps has the numbers.

Check a Number on a Model Page

The same example runs most text retrieval and rerank rows on the model pages. Take the task, model, profile and output from the row, and pass the row’s metric and value as --baseline, such as --baseline ndcg_at_10=0.31411, to read your run against it. <output> is dense, sparse or multivector, whichever the model or profile returns.

Row	Flags
A plain row	--model <model> --task <task> --output <output> --items-per-request 32
A profile row	--model <model>:<profile> --task <task> --output <output> --items-per-request 32
A reranker row	--model <reranker> --task <task>
A row that says “reranking the top 50 from <model>”	--model <reranker> --task "<task>?candidates_model=<model>&candidates_k=50"

It does not run image retrieval, or tasks outside MTEB such as extraction, classification, OCR, detection, speech and generation.

Check a Time on a Task Page

Ten per-request times on this site come from studies whose inputs are public in the superlinked/sie-task-evidence dataset. run.py latency, in the same example, times them on your own server and prints your numbers beside the published ones.

--page	Where the site shows it	Model	Image tag	Timed on
classify	/classify: 13.1 ms	fastino/GLiNER2.5-Decide	v0.11.0-cuda12-transformers5	RTX PRO 6000
search	home page and /about: 15.7 ms	Qwen/Qwen3-Embedding-4B	v0.11.0-cuda12-sglang-embedding	RTX PRO 6000
named-entities	industry pages, extract entities: 24 ms	numind/NuNER_Zero-span	v0.11.0-cuda12-default	RTX PRO 6000
redact	/redact: 61 ms	superlinked/pii-redact	v0.11.0-cuda12-default	RTX PRO 6000
typed-decisions	/typed-decisions: 13.8 ms	MoritzLaurer/deberta-v3-large-zeroshot-v2.0	v0.11.0-cuda12-default	RTX PRO 6000
rerank	/rerank: 55 ms	Qwen/Qwen3-Reranker-4B:fast	v0.11.0-cuda12-default	H100
request-routing	/request-routing: 8.5 ms	Qwen/Qwen3-Reranker-4B:fast	v0.11.0-cuda12-default	H100
ocr	industry pages, OCR a page: 287 ms	zai-org/GLM-OCR	v0.11.0-cuda12-sglang-vision-extract	H100
chat	/chat: 0.74 s	Qwen/Qwen3.8-27B-FP8	v0.11.0-cuda13-sglang-cu130	H100
structured-output	industry pages, structured output: 938 ms	Qwen/Qwen3.8-27B-FP8	v0.11.0-cuda13-sglang-cu130	H100

On a Linux machine with an NVIDIA GPU, start the page’s image with its model preloaded. For classify:

docker run --gpus all -p 127.0.0.1:8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  -e SIE_PRELOAD_MODELS=fastino/GLiNER2.5-Decide \
  ghcr.io/superlinked/sie-server:v0.11.0-cuda12-transformers5

Then, from examples/evaluations/published-benchmarks on the same machine:

uv sync --frozen
uv run python run.py latency --page classify --endpoint http://localhost:8080

chat and structured-output need an 80 GB GPU, and their CUDA 13 image needs an NVIDIA driver from the 580 series or newer.

The run rebuilds the page’s request list from the study’s inputs at pinned revisions and stops before the first request if its SHA-256 differs from the one the study published. The redact and typed-decisions studies published none, so for those two the check uses a hash the example pins. It sends the study’s warm-up requests without timing them. Then every request goes out once, one in flight. The output shows your n, p50, p95 and mean beside the published values, and results/ gets the time of every request.

Two differences apply to every page. The study’s client sent each request over a local socket inside the server’s container; yours goes over HTTP through the SDK. And your server runs on your GPU, not the study’s. Some pages add more, such as a published median over fresh servers, and the output names them all.

Measured Internally

The public example cannot repeat these numbers yet:

• Model-page rows outside text retrieval and reranking: image retrieval, extraction, classification, OCR, detection, speech and generation. They come from the internal measurement pipeline.
• The /detect time, 82.2 ms. Its 300 photos are not in the evidence dataset.

The task pages’ accuracy and cost charts state their own method. The dataset, scoring and prices sit beside each chart, and “Run this eval” opens the example or the recorded evidence.

What Differs When You Repeat It

Same method, different client. Four things change. The first is named in the result file’s comparable_reasons; the other three are not.

One process instead of four or two. The encode records ran two workers of two processes, the reranker records one worker of two. The example runs one process. The schedule is the union of the record’s per-process partitions, so the offered load is the same, but one process carries a 4× or 2× higher per-process rate, and the in-flight cap and the 20 ms lag gate bind sooner. Every performance run from the example is therefore comparable: false, with the reason listed. A full quality run with no override, and one execution revision throughout, is comparable: true; the smoke key or a revision change during the run sets it to false.

Your client’s location. The records’ client sat in the same cloud region as the API. From anywhere else, every latency includes the network path and the response transfer, and a multivector response is hundreds of kilobytes. Read the rate and success metrics first, and read the latency deltas in the light of where the run was made.

A verdict that gates nothing. A quality run prints a verdict from the 2% band above, even when the result reads comparable: false. That band is also how maintainers read a result, and their reading decides whether a number holds; the example’s verdict decides nothing. It leaves the exit status alone, and nothing the example writes reaches a record or this site.

Service state the client cannot pin. Server-side truncation and any default instruction are part of the service’s configuration. The SDK exposes a model’s maximum sequence length and weights revision, and not those two settings, so a run cannot verify them on its own.

Expected Spread

The example’s validation/ directory holds three consecutive runs of the Splade NFCorpus quality reference and three of the GTE-ModernColBERT SciFact performance reference at 1 rps, made from one client outside the API’s region. These are observed values from one client; no value here is a pass threshold or quality floor.

Quality, Splade NFCorpus

All three runs were comparable: true and valid: true.

Metrics	Across the three runs
Main score (0.33056) and every ndcg, map, recall and precision cut at 1 to 20	Equal to the published value in every run
The 100 and 1000 cuts, and mrr at 3 and beyond	Moved by up to 0.00062 between runs
ndcg@1000 and recall@1000	About 0.0002 and 0.0011 below the published value in every run

The sparse weights the service returns differ slightly from run to run: most document scores are identical between runs, the rest differ by about 0.001, at most 0.03, so near-tied documents swap places. That moves the 100 and 1000 cuts and the mrr cuts from 3 up. The ndcg@1000 and recall@1000 offsets sit at the cut where the order of equal scores is unspecified.

Performance, GTE-ModernColBERT SciFact at 1 rps

All three runs were valid: true: 60 requests, no failure, no timeout, no SDK retry, an in-flight peak of 1 or 2. The rate and success metrics equaled the published values in every run. The latency deltas are consistent with the client’s location. Milliseconds as the example reports them:

Metric	Published	Observed, three runs	Delta range	Spread between runs
Scheduling lag p99 (ms)	5.886	9.626 to 14.316	3.740 to 8.430	4.690
Latency p50 (ms)	178.819	444.978 to 464.829	266.159 to 286.010	19.850
Latency p95 (ms)	198.758	637.308 to 654.308	438.550 to 455.550	17.000
Latency p99 (ms)	285.029	890.516 to 1121.944	605.487 to 836.915	231.428

The p50 and p95 moved by about 20 ms or less between runs. The p99 is one request in 60; it moved by 231 ms between consecutive runs. The lag p99 of 9.6 to 14.3 ms stayed under the 20 ms tolerance in every run.

A transport probe from the same client split three warm encode requests of one document into 268 to 284 ms to the first response header, of which one network round trip, the TLS handshake, took 163 ms, which leaves an estimated 105 to 121 ms for the service’s own answer, followed by 485 to 561 ms of response transfer for the document’s multivector. The probe shows the transport share of one request. It does not decompose or bound the runs’ percentiles.

What’s Next

• Quality Evaluation: the shipped retrieval benchmark and MTEB-style workflows.
• Performance Evaluation: timing metadata, pools and load patterns for your own runs.
• Custom Evals: the worked pattern on your own labeled data.
• examples/evaluations/published-benchmarks: the public example on GitHub.