---
title: vLLM vs SGLang vs SIE for Production Search Pipelines
description: Trace one search request through retrieve, rerank, extract and generate. vLLM and SGLang compete for the last hop. SIE serves the three before it. Almost nobody needs to choose.
canonical_url: https://superlinked.com/blog/vllm-vs-sglang-vs-sie-search-pipelines
last_updated: 2026-07-28
---

Short version: in a production search pipeline these three do not sit in the same place. A single search request passes through four hops, and only the last one is generative. vLLM and SGLang are genuine alternatives to each other at that final hop; SIE (Superlinked Inference Engine) serves the three hops before it. The comparison only looks like a three-way choice if you collapse the pipeline into one box.

So rather than rank them, trace a request.

> **SIE is open source and free under Apache 2.0.** Stop reading and start cutting per-token costs whenever you like: the engine runs on your own hardware, from a laptop to a Kubernetes cluster.
>
> [★ Star the repo on GitHub](https://github.com/superlinked/sie) [Run the 2-minute quickstart →](https://superlinked.com/docs/quickstart)

## A production search request has four hops

Ask a real question of a real corpus and here is what has to happen.

**Retrieve.** The query gets encoded and matched against an index. In anything past a prototype this is not one encoder: dense embeddings catch meaning, sparse embeddings catch exact terms the dense vector blurs away, and multi-vector late interaction catches token-level detail a single vector compresses out. Each is a separate model producing a separate representation.

**Rerank.** The shortlist from retrieval goes to a cross-encoder, which scores query and document jointly rather than independently. This is the biggest precision win in most pipelines, and a different model class again.

**Extract.** Before the answer is assembled, the retrieved documents often need structure pulled out of them: entities, fields, a schema-valid object, text from a scanned page. More models, none of them generative.

**Generate.** The answer gets written. One generative model, streaming tokens to a user.

Four hops, three of them non-generative, and the number of distinct models involved is somewhere between four and ten before you have added anything clever.

## Hop by hop, which layer serves what

| Hop | Model class | Operation | Layer |
| --- | --- | --- | --- |
| Retrieve, dense | Bi-encoder | `encode` | SIE |
| Retrieve, sparse | Learned sparse | `encode` | SIE |
| Retrieve, multi-vector | Late interaction | `encode` | SIE |
| Rerank | Cross-encoder | `score` | SIE |
| Extract | NER, extraction, OCR | `extract` | SIE |
| Generate, small model | Small generative (~4B) | `generate` | SIE |
| Generate, large model | Frontier-scale generative | Completion | vLLM or SGLang |

The pattern is not subtle. vLLM and SGLang own one row of that table; it is the row you may not even need if your generative model is small.

## vLLM and SGLang are competing with each other, not with the rest of the pipeline

Both are LLM serving engines built to spread one large generative model across GPUs for token throughput. Both do continuous batching. They are legitimate alternatives to each other, and choosing between them is a real decision about your generation hop: model family support, long-context behavior, structured output handling, and how each performs on the specific weights you plan to serve. That decision should be made by benchmarking your model on your hardware, not from anyone's comparison table, and this article is not going to invent numbers for it.

What matters for pipeline design is that whichever one you pick, it is doing the same job in the same slot, and neither one addresses the first three hops.

One wrinkle here. SIE selects SGLang as one of its compute backends automatically, alongside its own PyTorch and Candle in Rust backends, sweeping parameters per model to pick the best serving path. So SGLang can end up in your stack twice: once as your generation engine if you choose it, and once underneath SIE for models where it is the right runtime. That is not a conflict, and it means the "vs SGLang" framing is partly a category error.

## The three hops before generation are one workload

This is the part that decides your architecture. Retrieve, rerank and extract look like three problems and are actually one: many small models, short requests, and the expensive thing is switching between them without leaving GPUs idle.

Deployed the obvious way, each of those models gets its own server and its own GPU. Under bursty traffic a per-model-per-GPU layout averages roughly [21% utilization](https://superlinked.com/blog/should-you-self-host-inference), which means the majority of the bill is idle hardware. Serving dense, sparse, multi-vector, and rerank from one deployment instead of four is the [specific case we wrote up separately](https://superlinked.com/blog/one-gpu-four-retrieval-modes-multi-model-search).

Two mechanics do the work. A stateless gateway publishes to one shared queue and a sidecar in each worker pod pulls from the pool and forms a full batch before the GPU runs, rather than each worker batching only its own local slice. With many small 50 to 100 millisecond requests, local queues pack badly and a half-full batch costs nearly as much wall-clock as a full one. Reported effect on real small-model workloads was a throughput lift of more than 50%. Separately, an LRU residency scheduler packs models into shared VRAM, keeping hot models resident and evicting cold ones only under pressure. In the [published example](https://superlinked.com/blog/should-you-self-host-inference) five models share two GPUs at a 71% warm-start hit rate.

Neither mechanic has anything to do with prefill, decode, or KV cache. That is why the generation hop is a separate concern.

## Per-hop model choice is a real tradeoff

Once the hops are separated you can pick per hop, which is where the quality and cost numbers get interesting. From the published per-task comparisons on the managed tier, at $/1B tokens, nDCG@10 and p50 latency:

| Hop | Model | Price | Quality | p50 |
| --- | --- | --- | --- | --- |
| Retrieve, dense | `intfloat/e5-base-v2` | `$6` | `0.52` | `12 ms` |
| Retrieve, dense | `BAAI/bge-m3` | `$13.58` | `0.63` | `23 ms` |
| Retrieve, sparse | `prithivida/Splade_PP_en_v2` | `$8` | `0.32` | `25 ms` |
| Retrieve, multi-vector | `lightonai/GTE-ModernColBERT-v1` | `$12` | `0.73` | `40 ms` |
| Rerank | `BAAI/bge-reranker-v2-m3` | `$10` | `0.68` | `45 ms` |

Two things stand out. On dense retrieval, `bge-m3` matches OpenAI's `text-embedding-3-large` on quality at `0.63` while costing `$13.58` against `$65` and returning in `23 ms` against `120 ms`. On reranking, `bge-reranker-v2-m3` at `0.68` beats a frontier small model on quality and returns in `45 ms` where a large model takes `400` to `900 ms`, which matters because reranking sits in the request path.

Those are managed list comparisons. If you self-host, the economics are different again and measured separately: sustained ingestion on one GPU puts `bge-m3` at `235,430` tokens per second and `$0.0036` per million tokens, with [full methodology here](https://superlinked.com/blog/sie-vs-hosted-embedding-apis). Do not mix the two sets of figures, they answer different questions.

## A reference pipeline

Sketch of the four hops (vector DB and prompt assembly left as placeholders):

```python
from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")
query = "late delivery penalty"

# Hop 1: retrieve. Dense and sparse from one endpoint, named per request.
dense = client.encode("BAAI/bge-m3", Item(text=query))
sparse = client.encode("prithivida/Splade_PP_en_v2", Item(text=query), output_types=["sparse"])
# candidates come back from your vector DB with text retained by id
candidates = vector_db.hybrid_search(dense, sparse, limit=100)
candidate_text = {c["id"]: c["text"] for c in candidates}

# Hop 2: rerank the shortlist with a cross-encoder.
ranked = client.score(
    "BAAI/bge-reranker-v2-m3",
    Item(text=query),
    [Item(id=c["id"], text=c["text"]) for c in candidates],
)
top_ids = [entry["item_id"] for entry in ranked["scores"][:10]]

# Hop 3: extract structure from what came back.
fields = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text=candidate_text[top_ids[0]]),
    labels=["organisation", "date", "amount"],
)

# Hop 4: generate. Same client if the model is small, your LLM engine if it is not.
top_passages = [candidate_text[i] for i in top_ids]
answer = client.generate(
    "Qwen/Qwen3-4B-Instruct-2507",
    build_prompt(top_passages, fields),
    max_new_tokens=512,
)
```

Three hops, one client, one endpoint, models named per request. The fourth hop is the only place a separate engine becomes a question.

## Where pipelines actually break

In practice the failure is rarely that the generation engine was the wrong choice. It is that retrieval, reranking and extraction were built as four separate deployments, each pinned to its own GPU, each scaled independently, each with its own client and its own monitoring. The cost shows up as idle GPUs and the velocity cost shows up every time someone wants to A/B test a second reranker and has to provision hardware to do it.

Pick your generation engine on its merits. Then treat everything before it as one workload rather than four.

## FAQ

**Should I use vLLM or SGLang for my generation hop?** Benchmark both on your model and your hardware. They are close peers and the answer depends on model family and context length rather than on anything generalizable.

**Does SIE replace vLLM or SGLang in a search pipeline?** No. It serves the retrieval, reranking and extraction hops. If your generative model is small it can serve that too, through `generate`. If it is large, keep vLLM or SGLang for that hop.

**Why not run my encoders on vLLM to keep one system?** You get a GPU per model with no eviction, and you do not get `score` or `extract` as operations, so reranking and extraction end up as separate services anyway.

**Do I need all three retrieval modes?** No, and most pipelines start with dense only. The argument for serving them from one deployment is that adding sparse or multi-vector later becomes a config change rather than a new deployment.

**Can I keep my vector database?** Yes. SIE produces the vectors and your database indexes them, with documented integrations for Chroma, Qdrant, Weaviate and LanceDB.

[Star the repo on GitHub](https://github.com/superlinked/sie) · [Read the encode docs](https://superlinked.com/docs/encode) · [Browse the model catalog](https://superlinked.com/models)