---
title: The Open-Source Inference Landscape, and Where SIE Fits
description: Serving models on Kubernetes with open-source tools now spans 79 entries in 13 groups. What problem each group solves, which projects solve it, and where SIE sits among them.
canonical_url: https://superlinked.com/blog/open-source-inference-landscape
last_updated: 2026-11-13
---

You want to serve models on Kubernetes, you go looking for the right tools, and you come back with a list of seventy-odd project names, half of which sound like they do the same thing. KServe or llm-d? Kueue or KAI or HAMi? The tools aren't the hard part. It's the fragmentation and fog you have to cut through to see which problem each one solves.

Coming back from KubeCon NA 2026 in Salt Lake City, where inference took over most of the agenda, I put every open-source project I could find on one map. This post walks through it. The map covers open source only. Managed services and commercial products solve many of the same problems, and they're left out on purpose. A few entries are open source with strings attached: TensorRT-LLM, the GPU Operator and the DCGM exporter are Apache-2.0 but depend on NVIDIA's proprietary drivers and runtime, and Grafana is the one AGPL project here. I'll start with how the map is laid out, then go through its 13 groups one at a time: the problem each group solves, and what each project in it does. At the end, I'll say where SIE, the inference engine I work on at Superlinked, fits on the map, and where it doesn't. If you want my notes on the talks themselves, they're in [what I learned at KubeCon NA 2026](/blog/kubecon-na-2026-inference).

<BlogSieCta />

## How to Read the Map

The map follows a single inference request, left to right and top to bottom. The first row is where a request lands: the agent that sends it, the gateway that routes it, the queue that holds it, and the autoscaler that watches that queue to decide how many workers to run. The second row is who answers it: the serving layer (one for big LLMs, one for small models), the engine that runs the math, and whatever decides what stays in GPU memory. The third row is what all of that runs on: the GPUs and how they're shared, how model files get to a node, and Kubernetes itself. Down the right side is the work that cuts across every row: security, observability and benchmarking.

Several boxes look alike from a distance (a gateway and a queue both hand out requests; autoscaling and GPU scheduling both decide capacity), so each section below ends its problem statement with a short note on how that box differs from its neighbors.

Full disclosure: the orange box that says "SIE by Superlinked" is ours, and I drew the map, so weigh where I put it accordingly.

## Where a Request Lands

### 1. Agents and Protocols

**The problem.** Agents don't behave like the web services Kubernetes was built for. An agent holds state across many turns, runs for minutes or hours, sits idle and then bursts, calls tools, and sometimes runs code. It needs a standard way to call tools and other agents, a runtime that can park it while it waits, and limits on what it's allowed to touch.

**How it's different.** Everything else on the map answers requests. This box is what sends them. Agent Substrate and kagent manage the agent's own state and lifecycle, not any model's.

**The projects.**
- **MCP** (Model Context Protocol) is the standard way for an agent to call a tool, and **A2A** is the standard way for one agent to call another.
- **Agent Substrate**, new in the CNCF Sandbox, maps many stateful agents onto a smaller pool of pods, one actor per session, and uses gVisor checkpoints to suspend an agent and resume it later.
- **kagent** runs agents as Kubernetes resources (its 1.0 release runs on Agent Substrate), and **Agent Sandbox**, from Kubernetes SIG Apps, manages the lifecycle of a sandboxed agent.
- **OpenShell**, an NVIDIA-led project new in the Sandbox, sets policy for what an agent's runtime can reach: files, network, credentials and model endpoints.
- **Cadence** (from Uber) runs durable workflows, so a multi-step agent task survives a restart, and **Drasi** runs continuous queries that let an agent react to changes in data.
- **SAFE-MCP** is a community-built threat catalog for MCP: attack techniques such as tool poisoning, with mitigations and detection rules for each.

### 2. Gateways and Routing

**The problem.** A normal load balancer spreads requests evenly, which is wrong for LLMs. One request might be a 20-token question and the next a 30,000-token document, and a server that already holds the KV cache for a prompt's opening tokens can answer it much faster than one that doesn't. The gateway has to know which model a request wants, how busy each server is, and what's in each server's cache.

**How it's different.** A gateway decides where a request goes the moment it arrives, and pushes it there. A queue (box 3) doesn't decide at all: it holds the request until a worker takes it. (The inference extension has added flow control that can hold requests while every server is busy, so gateways queue a little too, but they still pick the server.) And a gateway only picks the server; running and coordinating the model is box 5's job.

**The projects.**
- The **Gateway API Inference Extension**, a Kubernetes SIG Network subproject, adds two pieces to the standard Gateway API: an InferencePool (a group of model servers) and an endpoint picker that chooses a server by model, load and KV cache. Most of the gateways below implement it.
- **Envoy AI Gateway**, generally available since its 1.0 release, adds model routing and token-based rate limits on top of Envoy Gateway. Envoy itself has added MCP filters and a pluggable queueing policy.
- **kgateway** and **agentgateway** (from Solo.io) implement the Gateway API and the inference extension; agentgateway also handles MCP and agent traffic.
- **Kuadrant** adds authentication and rate limits to AI routes, and **NGINX Gateway Fabric** is NGINX's Gateway API implementation.

### 3. Queues and Messaging

**The problem.** A router pushes each request to a server it picks, based on a guess about how busy every server is. A queue turns that around: the gateway drops each request on the queue, and each worker pulls as much work as it has room for. Nobody has to guess, a request stays on the queue until a worker finishes it (so a crashed worker doesn't lose it), and the length of the queue is the best signal you have for when to add or remove workers. That's why autoscalers like KEDA sit on top of the queue.

**How it's different.** Where a gateway picks a server and pushes, a queue lets the workers pull. It adds a hop, but it separates when work arrives from when it gets done, and that separation is what lets you scale workers on demand and all the way to zero. Don't let the name fool you about Kueue, either: it queues whole workloads (jobs, or pods asking for GPUs) until there's quota for them, not individual inference requests, so it lives with GPU scheduling in box 8.

**The projects.**
- **NATS JetStream** is the persistent, acknowledged queue in NATS, which graduated in the CNCF this week. It's the queue SIE uses between its gateway and its workers.
- **Apache Kafka** is the log-based workhorse for event streams, and **RabbitMQ** is the classic message broker. KEDA has scalers for both, so either can carry inference work the same way.

Managed queues such as Amazon SQS and Google Cloud Pub/Sub work the same way, and KEDA can scale on them too. They're off the map because they're services, not open-source projects.

### 4. Autoscaling

**The problem.** GPU utilization is a poor signal for when to add capacity, because a GPU can look busy while requests pile up, or look idle while it waits on memory. And GPU nodes are slow to start. Inference needs autoscalers that watch the right signals (queue depth, KV cache use, time to first token and request saturation) and that can scale to zero when nobody's calling.

**How it's different.** Autoscaling decides how many copies of a model server run. GPU scheduling (box 8) decides where each copy goes and how much of a GPU it gets. One answers "how many", the other "where".

**The projects.**
- **KEDA**, a graduated CNCF project, scales pods on events and queues, down to zero.
- **HPA** and **VPA** scale pods out and up; for LLMs, HPA works best on queue depth, KV cache use and time to first token.
- **Cluster Autoscaler** and **Karpenter** add and remove nodes; Karpenter can provision nodes for the devices a DRA claim asks for.
- **llm-d WVA** (the workload variant autoscaler) is llm-d's own autoscaler for LLM serving.

## Who Answers It

### 5. Model Serving and Coordination

**The problem.** An engine knows how to run one copy of one model. Something has to turn that into a service: start the right copies on the right GPUs, keep them healthy, hand them requests and use the hardware well. What that takes depends on the shape of your models, so this box has two halves.

**How it's different.** An engine (box 6) runs one copy of a model on the GPUs it's given. This layer decides how many copies of which models run where, and keeps them working together. The two halves handle different shapes of the same problem: multi-GPU serving runs many replicas of an LLM, some of them spanning several GPUs, and fractional-GPU serving packs many small models onto one GPU.

#### LLM Serving

One copy of an LLM can't keep up with the traffic, and the largest models don't fit on one GPU. So LLM serving is a multi-GPU problem: running many replicas, sending each request to the replica that already holds its prompt prefix in cache, and often splitting the prefill phase (reading the prompt) and the decode phase (writing the answer) onto separate GPUs and moving the KV cache between them. Only the biggest models need one copy spread across several nodes.

- **llm-d**, a CNCF Sandbox project that shipped 1.0 in October, is the reference for distributed LLM serving on Kubernetes: an endpoint-picker router, prefill and decode on separate GPUs, a KV cache indexer and its own autoscaler.
- **KServe**, a CNCF Incubating project, gives you custom resources for serving a model (InferenceService, and LLMInferenceService for LLMs), a model cache, and multi-node serving.
- **NVIDIA Dynamo** is a distributed serving framework with a KV-aware router and a CPU memory tier for the KV cache.
- **LeaderWorkerSet** starts a group of pods as one unit, which is what a model spread across several nodes needs, and **KubeRay**, donated to the CNCF this week, runs Ray clusters on Kubernetes.
- **KAITO** (the Kubernetes AI Toolchain Operator, a CNCF project that started at Microsoft) deploys models on Kubernetes, and according to a KubeCon session this week, its MultiRoleInference resource sets up llm-d's separate prefill and decode pools for you.

#### Task-Specific / Small Models

A single agent step can call an embedder, a reranker, an entity extractor and a guard model before it calls the LLM once. Each of those models needs only a fraction of a GPU. Give each one its own deployment and its own GPU, and you're paying for a dozen cards that sit mostly idle; put them together, and something has to decide which models are loaded, batch their requests and scale them. That also makes this half a memory manager for model weights, which is where it overlaps with box 7.

- **SIE**, Superlinked's open-source inference engine, serves many small and mid-size models on shared GPUs behind one API. More on it below.
- **Dynamo-Triton** (NVIDIA's Triton Inference Server, renamed in 2025) hosts several models in one server, with dynamic batching and explicit model loading and unloading.
- **KServe ModelMesh** loads and unloads models across a pool of serving pods based on how they're used, built for many models that each see little traffic.
- **Ray Serve** can multiplex several models on one replica and give each replica a fraction of a GPU.
- **Hugging Face TEI** (Text Embeddings Inference) serves one embedding or reranking model per server, very fast.

### 6. Inference Engines

**The problem.** Something has to run the model's math fast on the GPU you have. That means batching requests together, managing GPU memory, and using the right kernels for the card.

**How it's different.** An engine knows nothing about other copies of its model, other models or the cluster. Its job is narrow: make one model fast on the hardware it gets. Everything in box 5 is built on top of engines.

**The projects.**
- **vLLM** (under the PyTorch Foundation) is the most common LLM engine: continuous batching, with the KV cache managed in small pages (PagedAttention) inside a pool it reserves when it starts.
- **SGLang** is an LLM engine that reuses the work it already did for shared prompt prefixes (RadixAttention). **TensorRT-LLM** is NVIDIA's engine, tuned for its own GPUs.
- **llama.cpp** runs models on CPUs, laptops and Apple Silicon.
- **PyTorch** (the framework), **FlashAttention** (a fast attention kernel) and **ONNX Runtime** are what most encoders, rerankers and classifiers run on.
- **wasmCloud** runs WebAssembly applications, and with wasi:webgpu those applications can run models on whatever GPU the node has.

### 7. KV Cache and Model Memory

**The problem.** GPU memory is the scarcest thing in the cluster. For an LLM, what fills it is the KV cache: the stored keys and values for every token of every conversation in flight, which grows with every token. For small models, what fills it is the model weights. Either way, something has to decide what stays in GPU memory, what moves to cheaper storage and what gets thrown away.

**How it's different.** GPU scheduling (box 8) decides which GPU a pod gets. This box decides what lives in that GPU's memory once the pod is running, and the scheduler never sees inside it.

**The projects.**
- The **llm-d KV indexer** keeps track of which server holds the cache for which prompt prefix, so the router can send a request where its cache already is.
- **LMCache** stores and reuses KV cache across requests and across engine instances.
- The **Dynamo KV tier** moves KV cache out to CPU memory when the GPU runs short.
- **NIXL** (NVIDIA's Inference Xfer Library) moves KV cache blocks straight from one GPU's memory to another over RDMA, which is what makes splitting prefill from decode practical, and it can also move the cache out to networked storage.

## What It Runs On

### 8. GPU Scheduling and Sharing

**The problem.** Kubernetes started out handing out GPUs as whole devices, one or more per pod. That's fine for a 70B model and wasteful for anything smaller. The scheduler also has to know which GPUs sit next to which network cards, which pods have to start together, and whose quota a GPU counts against.

**How it's different.** Scheduling works at the level of pods and devices: whether a pod gets a whole GPU, a fractional GPU or a multi-GPU allocation, and which ones. It doesn't decide how many pods run (that's box 4), and it doesn't see what a process does with the memory it was given (that's box 7).

**The projects.**
- **Dynamic Resource Allocation (DRA)**, generally available since Kubernetes 1.34, lets a pod ask for a device by its properties instead of a count: a slice of a partitionable GPU (that part is still in beta), or a GPU and a network card on the same PCIe root.
- **Kueue** handles quotas and queues for batch and AI jobs, and can count quota in GPU memory instead of whole GPUs. **Workload-aware scheduling**, not yet generally available, teaches the default scheduler to start a group of pods together and place them by topology.
- **HAMi**, a CNCF Incubating project, virtualizes GPUs so each pod gets a cap on GPU memory and compute, enforced in software by intercepting CUDA calls (its docs call it a soft limit and point to MIG for hard isolation); it's moving to a lighter mode built on DRA. The **KAI Scheduler**, a CNCF Sandbox project that started at NVIDIA (Run:ai), gives pods fractions of a GPU, works with HAMi, and adds hierarchical queues.
- **Volcano** is a batch scheduler, and the **NVIDIA GPU Operator** installs and manages the GPU drivers on every node.

### 9. Packaging and Distribution

**The problem.** Model weights run to gigabytes, and container images for AI often do too. Moving those bytes into a node is a big share of a cold start, and an air-gapped cluster can't fetch them from the internet at all.

**How it's different.** This box is about the bytes: getting model weights and images onto a node quickly. Once they're there, the platform (box 10) runs them and the serving boxes use them.

**The projects.**
- **KitOps** packages models, prompts and MCP servers as OCI artifacts (ModelKits), so they can live in the same registry as your container images.
- **ORAS** pushes and pulls any OCI artifact, **Harbor** is the registry, and **Dragonfly** spreads large downloads peer to peer across nodes.
- **KServe LocalModelCache** keeps model weights on the nodes, so a new pod doesn't download them again.

### 10. Platform Foundation

**The problem.** Everything above runs on the same base, and that base is changing for AI too.

**How it's different.** These aren't inference projects. They're the base the inference projects run on, and two of them (Kubeflow and OpenRL) are about training rather than serving.

**The projects.** **Kubernetes** itself, and **containerd**, which now unpacks image layers in parallel to speed up multi-gigabyte AI images. **Kubeflow**, newly graduated, runs training and ML pipelines. **OpenRL**, new in the Sandbox, orchestrates reinforcement learning post-training on Kubernetes. **Helm**, a graduated CNCF project, packages a Kubernetes application as a chart, so a whole inference stack (SIE included) installs with one command.

## The Cross-Cutting Work

### 11. Security and Policy

**The problem.** Agents run code, call tools and hold credentials. Model endpoints carry sensitive data. You need to know which workload is which, control what each one can do, and isolate the ones you don't fully trust.

**How it's different.** A gateway's authentication (box 2) decides who may call a model. This box decides what the workloads themselves may do inside the cluster: what they can reach, which identity they carry, and how isolated they are.

**The projects.** **Kyverno** and **OPA** enforce policy on what gets deployed and what it may do. **SPIFFE / SPIRE** give every workload a verifiable identity. **Confidential Containers** run workloads in encrypted memory, and **gVisor**, donated to the CNCF this week, gives each container its own sandboxed kernel.

### 12. Observability

**The problem.** A standard dashboard can't tell you which pod is using which GPU, how long the first token took, or why the KV cache is full. Inference needs telemetry that understands GPUs and models.

**How it's different.** Observability watches real traffic in production and tells you what happened. Benchmarking (box 13) makes up its own traffic before you ship and tells you what will happen.

**The projects.** **OpenTelemetry**, newly graduated, has semantic conventions for generative AI traces. **Prometheus** collects the metrics and **Fluent Bit** the logs. The **DCGM exporter** reports GPU metrics (how busy the compute units and memory are), and in Kubernetes mode maps each GPU to the pod using it; under time-slicing, it can no longer tell the pods apart. **Kepler** measures energy use per workload, and **Grafana** turns all of it into dashboards.

### 13. Benchmarking and Simulation

**The problem.** GPUs are expensive enough that you want to know how a setup will perform before you buy it, and to catch regressions after every upgrade.

**How it's different.** You generate the load and measure the result, before real users show up. Simulators like inference-sim go a step further and leave out the GPUs entirely.

**The projects.** **inference-perf**, from the Kubernetes Serving working group, **GuideLLM** and NVIDIA's **AIPerf** load-test inference servers. **llm-d inference-sim** simulates a model server, so you can test routing and autoscaling without any GPUs.

## Where SIE Fits

[SIE](https://github.com/superlinked/sie) is open source, and it lives in the task-specific / small models half of box 5, model serving. It also does box 7's job for model weights. It serves embedders, rerankers, entity extractors, classifiers, guard models, OCR and mid-size LLMs from one deployment, behind one HTTP API (there's an MCP server too, which puts it in box 1 as a tool provider). Under the hood it uses the engines from box 6: PyTorch and FlashAttention for the small models, and SGLang for the larger ones.

SIE packs several models onto one GPU. It loads each model the first time it's called, unloads the least recently used one when GPU memory fills up, and lets you pin the models that must stay loaded. Embedding, scoring and extraction requests go from a stateless gateway onto a NATS JetStream queue (box 3), and the workers pull from it and build their own batches. For LLM generation, the gateway sends each request to the worker most likely to already hold its prompt prefix in cache. You install it with one Helm chart.

It works with the boxes around it:

- **Gateways (box 2).** SIE sits behind any Gateway API implementation as an ordinary backend. For the small models it doesn't need the endpoint picker, because its workers pull work from the queue; for generation, its own gateway does the cache-aware routing.
- **GPU scheduling (box 8).** HAMi, KAI and DRA give pods fractional GPUs. SIE does fractional-GPU serving one level down: it shares a GPU between models inside one worker pod. The two stack: if tenants need hard memory isolation, give each tenant its own MIG instance with its own SIE workers on it, and let SIE pack that tenant's models inside. Models that share one worker share its memory. HAMi's software caps are a softer option. A DRA claim on its own doesn't cap how much memory a process uses; the partitioning underneath it has to.
- **Queues and autoscaling (boxes 3 and 4).** The NATS JetStream queue is how SIE hands out work, and KEDA scales SIE's worker pools on queue metrics, down to zero.
- **Observability (box 12).** SIE ships Grafana dashboards. DCGM sees the GPU, not the models sharing it, so per-model numbers have to come from SIE itself.

For small models, SIE makes different choices from the LLM side of the map. Its workers **pull** from a queue, because encoders keep no KV cache to route on and full batches are what make them cheap. Its memory pressure comes from **model weights**, so it loads and unloads models instead of managing a cache. And it shares a GPU **inside one worker pod**, so moving capacity from one model to another doesn't reschedule anything.

## Where SIE Doesn't Fit

SIE isn't for serving frontier-scale LLMs across many nodes. It can split a generation model that doesn't fit on one GPU across up to eight GPUs on one server with tensor parallelism (since 0.8.0), but multi-node serving and splitting prefill from decode are llm-d, KServe and Dynamo's job (the LLM half of box 5), and I'd keep your big LLMs there and put SIE beside them, behind the same gateway. SIE also doesn't enforce memory isolation between tenants on its own; for that, give each tenant its own MIG instance and its own SIE workers, with HAMi's software caps as a softer option. By default it doesn't place models across the cluster from one central plan: any worker in a pool can load a model, so a popular one can end up loaded on several workers (there's an opt-in placement planner if you want one). And it pulls weights from the Hugging Face Hub by default, with an optional cache in S3, GCS or Azure storage, instead of from OCI ModelKits (box 9), which matters if your cluster is air-gapped.

## Conclusion

Thirteen boxes, 79 open-source entries, and most of them built around one big LLM: route requests by KV cache, split prefill from decode, share GPUs between pods, scale on LLM signals. That work is good, and you should use it for your LLM. The small models your agents call on every step need a different shape: many models packed onto one GPU, a queue the workers pull from, and scaling down to zero between bursts. That's the task-specific / small models half of box 5, and that's where SIE fits.

If you think a project belongs on the map and isn't there, or sits in the wrong box, tell me. I'll update it. I'm sure the map will be out of date by KubeCon EU anyway.

Until next time.