llm-d vs Dynamo vs SIE: Multi-Model Serving Compared

Short version: all three of these sit above the inference runtime, not inside it, so they are genuinely comparable. What separates them is workload shape. llm-d and NVIDIA Dynamo make one large model faster across many GPUs; SIE (Superlinked Inference Engine) makes many small models cheaper on a few. The deciding question is not model size, it is how many distinct models a single request touches.

SIE is open source and free under Apache 2.0. Stop reading and start cutting per-token costs whenever you like: the engine runs on your own hardware, from a laptop to a Kubernetes cluster.

★ Star the repo on GitHub Run the 2-minute quickstart →

All three are cluster layers, not runtimes

A lot of inference comparisons put things in the same list that do not belong there. vLLM, SGLang and TensorRT-LLM are runtimes: they execute a model on hardware. llm-d, Dynamo and SIE are the layer above, the one that decides which worker gets which request, how work is batched, when capacity scales, and what happens when a model is not resident.

Dynamo is explicit about this in its own README: it sits above engines like vLLM, SGLang and TensorRT-LLM and turns a cluster of GPUs into a coordinated system, and if you are running a single model on a single GPU your engine alone is probably enough. llm-d is built on vLLM and wires into Kubernetes through the Gateway API Inference Extension. SIE runs its own backends written in PyTorch and Candle in Rust, and selects SGLang for some models.

So the comparison is fair. All three answer the same question: what does the control plane do. They just answer it for different traffic.

llm-d and Dynamo optimize the large-model path

Both projects are organized around the same core insight, that a generative request has two phases with different hardware appetites. Prefill is compute-bound, decode is memory-bandwidth-bound, and running them on the same worker means each phase spends part of its life starving the other. Split them onto separate GPU pools and each pool saturates the resource it actually needs.

Everything else follows from that split. Both use NVIDIA’s NIXL transfer library to move KV cache between prefill and decode workers. Both route with cache awareness rather than round-robin: llm-d’s inference scheduler scores decode pods by prefix match length, and Dynamo’s KV-aware router weighs worker queue depth against KV cache overlap and makes a probabilistic decision from the combination. Dynamo adds a Planner that profiles a workload before deployment and then forecasts traffic at runtime with time-series models, adjusting prefill and decode worker counts against a latency target, plus a KV Block Manager for tiered cache offload. llm-d adds SLO-aware autoscaling and flow control for multi-tenant serving.

The published gains are real and, importantly, they are gains on large-model traffic. From llm-d’s own figures, drawn from partner engineering blogs: up to 70% higher tokens per second from prefill and decode disaggregation against standard vLLM on GPT-OSS on B200 hardware, 10 to 30% higher throughput on the same infrastructure for GPT-OSS-120B and Llama 3.3 70B on MI300X, and around 40% lower time to first token with predicted-latency scheduling.

The projects are candid about where that stops paying. llm-d’s own disaggregation guide recommends the pattern for large models rather than small ones, for long input sequences rather than short ones, and for sparse mixture-of-experts architectures. Dynamo keeps aggregated topology as a first-class mode precisely because a lot of traffic does not need the split.

Read that list against an agent workload and the mismatch is obvious. An encoder call is a few hundred tokens in, one vector out. No decode phase to disaggregate. No KV cache worth routing around.

SIE optimizes the many-small-models path

Agent inference around the LLM is the inverse problem. Instead of one 120B model spread across many GPUs, you have dozens of models in the 100M to 4B range, each request is short, and the expensive thing is switching between them.

That changes what the control plane has to be good at, and SIE is built around two mechanics rather than prefill and decode.

The first is a shared queue. In the conventional pattern a central scheduler commits each request to a worker up front, and each worker batches only its own local slice. With many small, fast requests those local queues run uneven and pack badly, and a half-full batch costs almost as much wall-clock as a full one. SIE inverts it: a stateless gateway publishes work to one shared queue on NATS JetStream, and a sidecar inside each worker pod pulls from the pool and forms a full batch before the GPU runs. Traffic never fragments into starved local queues. In the Infer() Summit writeup, Daniel Svonava reported this lifting throughput by more than 50% on real small-model workloads, with GPU efficiency in the illustrative model rising from 51% to 92%.

The second is model packing. Pinning one model per GPU means five models need five always-on GPUs, and with bursty traffic they average roughly 21% utilization. An LRU residency scheduler loads multiple models into the VRAM of one GPU, keeps hot models resident, and evicts cold ones only when something else needs the slot. In that same worked example, five models share two GPUs at a 71% warm-start hit rate. Lazy loading with eviction, pinning and capacity reservation is what lets one cluster absorb traffic that touches dozens of models.

Neither of those mechanics helps a single 120B model. Both of them are the whole game when your traffic touches forty small ones.

At a glance

The deciding question is model count per request, not model size

If you want a single test, count the distinct models one user-facing request touches.

Touch one model, and the question is how to make that model fast across a fleet. That is llm-d and Dynamo territory, and the disaggregation gains above are the reason to go there.

Touch a dozen, and the question is how to stop paying for idle GPUs while models sit resident that nobody is calling. That is what shared queueing and LRU packing address, and neither llm-d nor Dynamo is trying to solve it.

The numbers make the second case concrete. A contract review agent built on nine small models, running one orchestrator across six tools, is a single agent. An organization runs many of them, so cluster traffic spans forty models at once. Per-model GPU pinning is then not a tuning problem; it is the entire bill.

A mixed fleet runs more than one of these

The honest answer for most production stacks is not a single choice.

If you serve a large generative model at high concurrency and also run the retrieval, reranking, extraction and safety models around it, those are two different workloads with two different optimal control planes. Running Dynamo or llm-d for the generation tier and SIE for the small-model tier is a reasonable architecture, not a hedge. The tiers talk over HTTP and neither needs to know how the other schedules.

The boundary is not perfectly clean, though. Dynamo’s router includes a device-aware weighted mode built for heterogeneous fleets, with CPU embedding encoders sharing an endpoint alongside GPU ones; there is genuine overlap at the edges. The difference is emphasis. For Dynamo that is one routing mode among many. For SIE the heterogeneous small-model fleet is the design center.

Try it on your own traffic

The cheapest way to settle this is to measure your own request mix. Count the distinct models per request, then look at your average GPU utilization. If utilization is low and model count is high, the packing and queueing mechanics are where your money is.

SIE runs locally with one command, so you can profile against your real model mix before committing to any cluster:

pip install "sie-server[local]" && sie-server serve

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")
client.encode("BAAI/bge-m3", Item(text="profile this against your own model mix"))

FAQ

Is SIE a replacement for llm-d or Dynamo? Not for the workload they target. If your problem is serving one large generative model across a fleet, they solve it and SIE does not. If your problem is dozens of small models sharing a few GPUs, the reverse holds.

Can I run SIE alongside Dynamo or llm-d? Yes. They are separate control planes over separate GPU pools, reached over HTTP. A common split is generation on Dynamo or llm-d and everything around it on SIE.

Do llm-d or Dynamo handle embeddings and reranking? They can serve models, but they are optimized for generative traffic and its prefill and decode structure. Encoder and reranker calls have no decode phase, so the disaggregation and KV routing machinery has nothing to work with.

Which is the safest choice if my workload is still changing? Count models per request now and estimate where it goes. Agent workloads tend to add models rather than grow one, because each new capability is usually a new specialist model rather than a bigger generalist.

Are these numbers comparable head to head? No, and nobody should present them that way. The llm-d and Dynamo figures cited here are published by those projects and their partners on large-model benchmarks. The SIE figures are measured on small-model workloads. They describe different workloads, which is the point of the comparison.

Star the repo on GitHub · Read the engine docs · Browse the model catalog