# Search page sources

The /search page compares SIE's Qwen3 Embedding 4B with OpenAI
text-embedding-3-large, the most widely deployed embedding API, and with Voyage
4 Large, the embeddings Anthropic's documentation points to. This file gives the
provenance of everything the page shows and the complete results of the study
behind it, every model included.

## What the page shows

**The headline and the proof.** The study's eight public retrieval benchmarks
(below): 6,200 questions, every model on the same questions. The share of
questions whose first result is relevant, averaged over the eight benchmarks:
Voyage 4 Large 54.7%, SIE Qwen3 Embedding 4B 54.6%, OpenAI
text-embedding-3-large 50.8%, Voyage 4 Lite 50.5%, OpenAI
text-embedding-3-small 47.5%. The headline, "Better search than OpenAI
text-embedding-3-large, for 53% less", names one rival. "Better": Qwen3
Embedding 4B minus text-embedding-3-large is +3.8 points, with a
task-stratified bootstrap 95% interval of +2.7 to +4.9. The proof heading adds
the second rival, "level with Voyage 4 Large": Qwen3 Embedding 4B minus Voyage 4
Large is −0.2 points, interval −1.4 to +1.0, inside the 1.5-point band the page
treats as level; on nDCG@10 Voyage 4 Large is 0.7 points ahead (interval −1.3
to −0.1). Per-question results and the statistics are in
[`lookalike-search/mteb`](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/b939e410/lookalike-search/mteb)
(`stats.json`, `per_query.json.gz`).

**Price.** Real-time list prices per 1M input tokens: SIE Qwen3 Embedding 4B
$0.06 ([SIE price list](https://superlinked.com/pricing/sie-cloud-prices.json));
OpenAI text-embedding-3-large $0.13 and text-embedding-3-small $0.02
([OpenAI pricing](https://developers.openai.com/api/docs/pricing), captured in
the site's price table on 9 July 2026); Voyage 4 Large $0.12 and Voyage 4 Lite
$0.02 ([Voyage pricing](https://docs.voyageai.com/docs/pricing), captured on
9 August 2026). Every rival price was re-read unchanged on 11 October 2026, the
Voyage prices on [MongoDB's Voyage AI billing page](https://www.mongodb.com/docs/voyageai/management/billing/),
where Voyage's pricing now lives. "For 53% less": 1 − 0.06 / 0.13 is 53.8%,
rounded down to 53%. Against Voyage 4 Large the saving is 1 − 0.06 / 0.12, 50%.
Voyage 4 Lite and text-embedding-3-small cost less than SIE and rank a relevant
passage first less often; the two bar charts show both. Free allowances and batch
discounts are excluded on every side.

**The hero and the pain cases.** Questions from the look-alike test below,
chosen by one rule from the recorded rankings: SIE Qwen3 Embedding 4B and
Qwen3 Embedding 8B rank the answer first, and OpenAI text-embedding-3-large
ranks the question's named look-alike first. Twelve of the 440 questions meet
it. The hero is `q340` ("Once my boarding pass has been scanned at the gate, can
the airline still bump me off an overbooked flight?"): SIE ranked 14 CFR
250.7(a) first; text-embedding-3-large ranked 14 CFR 250.11(a) first and the
answer second. The pain cases are `q146` (PostgreSQL `multixact_member_buffers`,
default 32, against the offsets cache's 16; PostgreSQL License, source
`config.sgml` at postgres 168fa9dd) and `q353` (45 CFR 164.412(a), a written
law-enforcement statement, against (b), an oral one). Passages are verbatim and
were re-read at their sources on 8 October 2026. Every ranking is in
`lookalike-search/rankings.json` of
[superlinked/sie-task-evidence@b939e410](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/b939e410/lookalike-search).
The pain shows OpenAI only: on the look-alike test Voyage 4 Large and Voyage 4
Lite rank more answers first than SIE (table below), so a Voyage miss would not
be a fair selection.

**Consequence evidence.**
[plastic-labs/honcho #967](https://github.com/plastic-labs/honcho/issues/967)
(opened 3 August 2026, open, read on 11 October 2026): the project's memory
store is searched by embedding similarity alone, and exact numbers, names and
brands are smoothed away ("考了72分" embeds almost like "考了70分"); exact-token
questions passed 3% to 37% of the time. Magesh et al., "Hallucination-Free?
Assessing the Reliability of Leading AI Legal Research Tools" (Stanford
RegLab, [arXiv:2405.20362](https://arxiv.org/abs/2405.20362), 30 May 2024):
retrieval-based legal research tools (Lexis+ AI, Westlaw AI-Assisted Research,
Ask Practical Law AI) gave hallucinated answers, incorrect or cited to a source
that does not support them, to between 17% and 33% of questions.

**The API pane.** The two encode calls recorded on `https://api.superlinked.com`
on 30 September 2026 for question `q120` and three HHS breach-notification
passages (45 CFR 164.408(b), 164.408(c), 164.406(a)): the question as a query,
then the passages as documents. The ranking is the caller's cosine over the
returned vectors. The recorded query vector matches the study's `is_query`
vector for `q120` at cosine 0.99986. Run opens the console on the same
question, passages and model.

**Switching.** The switch sends the passages as documents and the question as a
query with the SIE SDK's `is_query=True`, as the study did. Sent through an
OpenAI-compatible route without the query instruction, a question is encoded
as a document; in the study that cost 5.2 points on the eight public
benchmarks.

**Run locally.** Qwen3 Embedding 4B runs on SIE's `sglang-embedding` bundle,
which the released `cuda12-sglang` server image carries; `SIE_BUNDLE` selects
it. It needs an NVIDIA GPU; there is no Apple Silicon path for this model yet.

**Speed.** The page states no timing: neither its headline nor its proof
heading claims speed, so its proof is two bar charts (right result first, and
price per 1M tokens) with three parameters beside them: what was tested, how
an answer is scored, and whose prices and when. The home page and the
Serverless comparison show two medians per question for this task: SIE 16 ms
and OpenAI text-embedding-3-large 238 ms. Both come from the task-page latency
programme
([`latency-fastest-config/2026-10-11/results/search.json`](https://huggingface.co/datasets/superlinked/sie-task-evidence/blob/2f74a5c09c3cefa0f5f754f92949600fcd4dc1aa/latency-fastest-config/2026-10-11/results/search.json)
at revision `2f74a5c09c3cefa0f5f754f92949600fcd4dc1aa`), which timed the
questions of the look-alike test (below), one question embedded per request.

- **SIE:** `Qwen/Qwen3-Embedding-4B` on `sie-server` (the `sglang-embedding`
  bundle, server code of public `superlinked/sie` at `2b3f7d2e`) behind the
  Rust sidecar, on one NVIDIA RTX PRO 6000 on Modal (8 CPU, 32 GiB), the client
  in the same container on the sidecar's socket. All 440 questions, one
  request at a time, after 20 warm-up requests: median 15.64 ms, 95th
  percentile 17.7 ms, mean 15.81 ms. The two surfaces round the median up to
  16 ms. The same pass on an L4 has a median of 50.91 ms and on an H100
  18.97 ms; they show the fastest of the three. With eight requests in flight on the
  RTX PRO 6000 the median is 45.73 ms.
- **OpenAI text-embedding-3-large:** `https://api.openai.com/v1/embeddings`
  from one laptop on a consumer connection in San Francisco, on 12 October
  2026. 300 of the 440 questions (sampled with seed 20260930), one request at
  a time, after 5 warm-ups: median 238.64 ms, 95th percentile 464.19 ms, mean
  254.25 ms. The two surfaces round the median down to 238 ms. The median of OpenAI's
  own `openai-processing-ms` header is 126.5 ms, so about 112 ms of the median
  is network and TLS. SIE's median on the same 300 questions is 15.64 ms.
- **The method, in one sentence:** "SIE timed on a GPU server with the client
  alongside; OpenAI text-embedding-3-large over its public API from San
  Francisco." It is stated here only: the home page and the Serverless
  comparison show the two medians and no method. The GPU, the question counts
  and the shape of each request are in the two entries above. The two sides do
  not share a network path: SIE's figure has no network hop and OpenAI's is a
  public API call.
- **What is claimed.** /search claims no speed: its headline and its proof
  heading claim quality and price. The home page and the Serverless comparison
  show the two medians. At the median the ratio is 15.26, 15 rounded down; the
  home page states that multiple.

**An earlier timing, not on the page.** On 30 September 2026, 300
single-question encode calls from one client measured a median of 400 ms for
SIE Qwen3 Embedding 4B on SIE's hosted API, 203 ms for text-embedding-3-large
and 164 ms for text-embedding-3-small. That run called all three over the
network from the same client. The SIE figure the home page and the Serverless
comparison show is the model on a GPU server with no network hop, so the two
runs measure different things.

## The evaluation

The study ran on 30 September 2026. The tables below keep every model we ran.

**The look-alike test.** 5,658 passages from 40 public sources: eCFR parts
with parallel rules (OSHA, FMCSA, DOT, CFPB, HIPAA, FMLA, FLSA, FTC), IRS
Publications 15 and 15-A, NIST SP 800-63B-4, twelve Kubernetes documentation
pages (CC BY 4.0) and four PostgreSQL documentation chapters (PostgreSQL
License). 440 questions, each with one answer passage and one named look-alike.
Claude Sonnet 5 drafted each question from its answer and the five passages
nearest to it by BM25; GPT-6 Sol kept a draft only if it picked the answer over
the look-alike blind; every kept question was read by eye and 86 were dropped;
the set was frozen before any tested model saw it. The score is the share of
questions whose first result is the answer. Every failed or missing reply
counts as a miss.

**Single embedding models**, answer ranked first, of 440 (30 September 2026):

| Model | First | Share |
| --- | --- | --- |
| Voyage 4 Large | 349 | 79.3% |
| Voyage 4 Lite | 336 | 76.4% |
| SIE Qwen3 Embedding 8B | 316 | 71.8% |
| Cohere Embed v4 | 309 | 70.2% |
| SIE Qwen3 Embedding 4B | 306 | 69.5% |
| OpenAI text-embedding-3-large | 295 | 67.0% |
| OpenAI text-embedding-3-small | 266 | 60.5% |

Head to head against OpenAI text-embedding-3-large, on the same 440 questions
(questions only SIE ranked first / only OpenAI ranked first; exact two-sided
McNemar; difference in points with a 95% interval clustered by the 40 sources):

| SIE model | SIE only / OpenAI only | McNemar p | Difference, 95% interval |
| --- | --- | --- | --- |
| Qwen3 Embedding 8B | 54 / 33 | 0.031 | +4.8 (-0.3 to +9.8) |
| Qwen3 Embedding 4B | 47 / 36 | 0.27 | +2.5 (-2.1 to +7.1) |

Against text-embedding-3-small: Qwen3 Embedding 8B 71 / 21 and Qwen3 Embedding
4B 71 / 31 (both p < 0.001). The pre-registered primary comparison was Qwen3
Embedding 4B against text-embedding-3-large; Qwen3 Embedding 8B was added by
amendment. The 8B row ran without its query instruction: the open-source
server's `qwen2_flash` adapter did not apply the configured default instruction
(fixed in sie #621 and #623), so its questions were encoded as
`Instruct: \nQuery:`. A rerun on 8 October 2026 that sent the query instruction
explicitly
([sie-task-evidence@dc6ca439](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/dc6ca439838f8d93392567884ff6d3be89f54bc5/search-instructed-qwen8b-440/2026-10-08))
also ranked 316 of 440 first. Serverless offers Qwen3 Embedding 4B for search,
so the page shows the 4B. Voyage 4 Lite ranked the answer first on 59 questions SIE
Qwen3 Embedding 4B missed; the reverse happened on 29 (p = 0.002).

**Eight public MTEB retrieval benchmarks** (CQADupstackPhysics, CosQA, FiQA2018,
LegalBenchConsumerContractsQA, NFCorpus, SCIDOCS, SciFact, StackOverflowQA;
6,200 queries), share of queries whose first result is relevant, averaged over
tasks: Voyage 4 Large 54.7%, SIE Qwen3 Embedding 8B 54.8%, SIE Qwen3 Embedding
4B 54.6%, Cohere Embed v4 52.3%, OpenAI text-embedding-3-large 50.8%, Voyage 4
Lite 50.5%, OpenAI text-embedding-3-small 47.5%. Qwen3 Embedding 4B minus
text-embedding-3-large is +3.8 points (task-stratified bootstrap 95% interval
+2.7 to +4.9); in nDCG@10, 0.599 against 0.565. Qwen3 Embedding 8B's nDCG@10 is
0.608. Public tasks may overlap an open model's training data.

**Embedding plus reranking**, on a frozen 100-question sample stratified by
source (seed 20261007), answer ranked first, of 100:

| Pipeline | First |
| --- | --- |
| Voyage 4 Lite top 10, reranked by Voyage rerank-3 | 90 |
| SIE Qwen3 Embedding 8B (query instruction sent) top 10, reranked by SIE Qwen3 Reranker 4B | 82 |
| SIE Qwen3 Embedding 4B top 10, reranked by ZeroEntropy zerank-2 on SIE | 81 |
| Octen Embedding 8B top 10, reranked by zerank-2 on SIE | 81 |
| Voyage 4 Lite top 10, reranked by zerank-2 on SIE | 81 |
| SIE Qwen3 Embedding 4B top 10, reranked by SIE Qwen3 Reranker 4B | 80 |
| Octen Embedding 8B top 10, reranked by SIE Qwen3 Reranker 4B | 80 |
| Voyage 4 Lite top 10, reranked by SIE Qwen3 Reranker 4B | 79 |
| Octen Embedding 8B alone | 72 |

On the same sample, single models scored Voyage 4 Large 81, Voyage 4 Lite 76,
Cohere Embed v4 73, SIE Qwen3 Embedding 8B 72 and SIE Qwen3 Embedding 4B 65.
The Qwen3 Reranker rows use float32 scoring; an earlier run of SIE's 4B
pipeline on the released server, which normalised scores in bfloat16 and tied
many candidates, scored 75. A further screen of two open rerankers, Qwen3
Reranker 8B and mxbai-rerank-large-v2, also stayed below 88. No pipeline was
confirmed on fresh questions.

Evidence:
[the pilot and its gates](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/d971b789e2701e91a68dfe65525f77c035c3e234/evaluations/2026-10-07/lookalike-search-pilot-results-v1),
[the float32 reranker follow-up](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/168088b2582ef2b25d780b02978e5d89e93f3aca/evaluations/2026-10-08/lookalike-search-f32-rerank-followup-v1),
and the frozen inputs, split, gold and scorer at
[superlinked/sie-task-evidence@f960e26f](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/f960e26fa427b86286694891d449e3158c0d4b84/lookalike-search-pilot/2026-10-07).

## Limits

One gold passage per question; about 15 of the 440 questions probably have a
second valid answer, and they count against every model alike. The questions
were written and filtered by Anthropic and OpenAI models; a vendor-family effect
was not measured. The 100-question sample is a screen: its intervals are wide,
and the 340 remaining questions were never used.