vLLM vs SGLang vs SIE for Production Search Pipelines

Short version: in a production search pipeline these three do not sit in the same place. A single search request passes through four hops, and only the last one is generative. vLLM and SGLang are genuine alternatives to each other at that final hop; SIE (Superlinked Inference Engine) serves the three hops before it. The comparison only looks like a three-way choice if you collapse the pipeline into one box.

So rather than rank them, trace a request.

SIE is open source and free under Apache 2.0. Stop reading and start cutting per-token costs whenever you like: the engine runs on your own hardware, from a laptop to a Kubernetes cluster.

★ Star the repo on GitHub Run the 2-minute quickstart →

A production search request has four hops

Ask a real question of a real corpus and here is what has to happen.

Retrieve. The query gets encoded and matched against an index. In anything past a prototype this is not one encoder: dense embeddings catch meaning, sparse embeddings catch exact terms the dense vector blurs away, and multi-vector late interaction catches token-level detail a single vector compresses out. Each is a separate model producing a separate representation.

Rerank. The shortlist from retrieval goes to a cross-encoder, which scores query and document jointly rather than independently. This is the biggest precision win in most pipelines, and a different model class again.

Extract. Before the answer is assembled, the retrieved documents often need structure pulled out of them: entities, fields, a schema-valid object, text from a scanned page. More models, none of them generative.

Generate. The answer gets written. One generative model, streaming tokens to a user.

Four hops, three of them non-generative, and the number of distinct models involved is somewhere between four and ten before you have added anything clever.

Hop by hop, which layer serves what

The pattern is not subtle. vLLM and SGLang own one row of that table; it is the row you may not even need if your generative model is small.

vLLM and SGLang are competing with each other, not with the rest of the pipeline

Both are LLM serving engines built to spread one large generative model across GPUs for token throughput. Both do continuous batching. They are legitimate alternatives to each other, and choosing between them is a real decision about your generation hop: model family support, long-context behavior, structured output handling, and how each performs on the specific weights you plan to serve. That decision should be made by benchmarking your model on your hardware, not from anyone’s comparison table, and this article is not going to invent numbers for it.

What matters for pipeline design is that whichever one you pick, it is doing the same job in the same slot, and neither one addresses the first three hops.

One wrinkle here. SIE selects SGLang as one of its compute backends automatically, alongside its own PyTorch and Candle in Rust backends, sweeping parameters per model to pick the best serving path. So SGLang can end up in your stack twice: once as your generation engine if you choose it, and once underneath SIE for models where it is the right runtime. That is not a conflict, and it means the “vs SGLang” framing is partly a category error.

The three hops before generation are one workload

This is the part that decides your architecture. Retrieve, rerank and extract look like three problems and are actually one: many small models, short requests, and the expensive thing is switching between them without leaving GPUs idle.

Deployed the obvious way, each of those models gets its own server and its own GPU. Under bursty traffic a per-model-per-GPU layout averages roughly 21% utilization, which means the majority of the bill is idle hardware. Serving dense, sparse, multi-vector, and rerank from one deployment instead of four is the specific case we wrote up separately.

Two mechanics do the work. A stateless gateway publishes to one shared queue and a sidecar in each worker pod pulls from the pool and forms a full batch before the GPU runs, rather than each worker batching only its own local slice. With many small 50 to 100 millisecond requests, local queues pack badly and a half-full batch costs nearly as much wall-clock as a full one. Reported effect on real small-model workloads was a throughput lift of more than 50%. Separately, an LRU residency scheduler packs models into shared VRAM, keeping hot models resident and evicting cold ones only under pressure. In the published example five models share two GPUs at a 71% warm-start hit rate.

Neither mechanic has anything to do with prefill, decode, or KV cache. That is why the generation hop is a separate concern.

Per-hop model choice is a real tradeoff

Once the hops are separated you can pick per hop, which is where the quality and cost numbers get interesting. From the published per-task comparisons on the managed tier, at $/1B tokens, nDCG@10 and p50 latency:

Two things stand out. On dense retrieval, bge-m3 matches OpenAI’s text-embedding-3-large on quality at 0.63 while costing $13.58 against $65 and returning in 23 ms against 120 ms. On reranking, bge-reranker-v2-m3 at 0.68 beats a frontier small model on quality and returns in 45 ms where a large model takes 400 to 900 ms, which matters because reranking sits in the request path.

Those are managed list comparisons. If you self-host, the economics are different again and measured separately: sustained ingestion on one GPU puts bge-m3 at 235,430 tokens per second and $0.0036 per million tokens, with full methodology here. Do not mix the two sets of figures, they answer different questions.

A reference pipeline

Sketch of the four hops (vector DB and prompt assembly left as placeholders):

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")
query = "late delivery penalty"

# Hop 1: retrieve. Dense and sparse from one endpoint, named per request.
dense = client.encode("BAAI/bge-m3", Item(text=query))
sparse = client.encode("prithivida/Splade_PP_en_v2", Item(text=query), output_types=["sparse"])
# candidates come back from your vector DB with text retained by id
candidates = vector_db.hybrid_search(dense, sparse, limit=100)
candidate_text = {c["id"]: c["text"] for c in candidates}

# Hop 2: rerank the shortlist with a cross-encoder.
ranked = client.score(
    "BAAI/bge-reranker-v2-m3",
    Item(text=query),
    [Item(id=c["id"], text=c["text"]) for c in candidates],
)
top_ids = [entry["item_id"] for entry in ranked["scores"][:10]]

# Hop 3: extract structure from what came back.
fields = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text=candidate_text[top_ids[0]]),
    labels=["organisation", "date", "amount"],
)

# Hop 4: generate. Same client if the model is small, your LLM engine if it is not.
top_passages = [candidate_text[i] for i in top_ids]
answer = client.generate(
    "Qwen/Qwen3-4B-Instruct-2507",
    build_prompt(top_passages, fields),
    max_new_tokens=512,
)

Three hops, one client, one endpoint, models named per request. The fourth hop is the only place a separate engine becomes a question.

Where pipelines actually break

In practice the failure is rarely that the generation engine was the wrong choice. It is that retrieval, reranking and extraction were built as four separate deployments, each pinned to its own GPU, each scaled independently, each with its own client and its own monitoring. The cost shows up as idle GPUs and the velocity cost shows up every time someone wants to A/B test a second reranker and has to provision hardware to do it.

Pick your generation engine on its merits. Then treat everything before it as one workload rather than four.

FAQ

Should I use vLLM or SGLang for my generation hop? Benchmark both on your model and your hardware. They are close peers and the answer depends on model family and context length rather than on anything generalizable.

Does SIE replace vLLM or SGLang in a search pipeline? No. It serves the retrieval, reranking and extraction hops. If your generative model is small it can serve that too, through generate. If it is large, keep vLLM or SGLang for that hop.

Why not run my encoders on vLLM to keep one system? You get a GPU per model with no eviction, and you do not get score or extract as operations, so reranking and extraction end up as separate services anyway.

Do I need all three retrieval modes? No, and most pipelines start with dense only. The argument for serving them from one deployment is that adding sparse or multi-vector later becomes a config change rather than a new deployment.

Can I keep my vector database? Yes. SIE produces the vectors and your database indexes them, with documented integrations for Chroma, Qdrant, Weaviate and LanceDB.

Star the repo on GitHub · Read the encode docs · Browse the model catalog