What I Learned in the Inference Tracks at KubeCon NA 2026

KubeCon + CloudNativeCon is the Cloud Native Computing Foundation’s (CNCF) big conference for the people who build and run Kubernetes, and this year’s North American edition, in Salt Lake City from November 9 to 12, added a new AI Inference + Agentic track. For four days, the people running some of the largest model deployments in the world stood on stage and talked about what breaks when you serve models on Kubernetes. I went to learn, and I learned a lot.

My notes from the week come down to three things:

1. Serving large models across many GPUs. The frontier-scale teams split each request into “prefill” and “decode” on separate GPUs, and most of their hard problems come back to moving and sharing the KV cache.
2. Scaling the agents that call those models. Agents hold state, their traffic is idle-then-burst, and health probes fail on agents that are just busy reasoning, so they don’t scale like the web apps Kubernetes grew up serving. Agent Substrate is the best example of a project built for that.
3. What the LLM hype overlooked: task-specific open-source models. The embedders, rerankers, extractors and guard models that agents call on every step barely came up, and neither did packing them onto fractional GPUs. The platform engineers I spoke with in the hallways have exactly those problems, and they solve them by hand.

I’ll take them in that order, and then say where SIE, the open-source inference engine I work on at Superlinked, fits into the third one.

What KubeCon Is, and Why Inference Took Over This Year

If you haven’t been, KubeCon is where the Kubernetes community meets: maintainers, platform teams, cloud providers and a very large expo hall. The CNCF runs it, and a lot of what gets announced there turns into the default way people run infrastructure a year or two later.

This year, AI inference was the main story. Monday hosted a co-located event, Cloud Native AI + Inference Day, opened by Yuan Tang, Rajas Kakodkar and Yuzhui Liu, next to Agentics Day (MCP + Agents) and Observability Day. The main conference, Tuesday to Thursday, had the AI Inference + Agentic track.

The project news pointed the same way. In its November 10 announcement, the CNCF added Agent Substrate, OpenShell and OpenRL to the Sandbox; graduated Kubeflow, OpenTelemetry and NATS (along with Buildpacks, Karmada and KubeVirt); said its Kubernetes AI Conformance Program had reached 40 certified platforms (a running total, not 40 platforms on the current release); and announced a Certified Kubernetes Inference Engineer exam planned for 2027. Two milestones from earlier in the year set the scene too: llm-d, in the CNCF Sandbox since March, shipped v1.0 on October 20 at PyTorch Conference North America, and HAMi became a CNCF Incubating project in July.

Serving Large Models Across Many GPUs

The talks I enjoyed most were from teams serving very large models at very large scale, because they work on hard problems I don’t get to see every day. Most of those problems come back to the KV cache.

A quick refresher. When an LLM answers a request, it does two different kinds of work:

• Prefill. The model reads your whole prompt in one pass and, for every token, computes a “key” and a “value” at every layer and keeps them. That stored set is the KV cache. Prefill is limited by how much compute the GPU has.
• Decode. The model then writes the answer one token at a time, and each new token looks back at the KV cache of everything before it. Decode is limited by memory bandwidth, and the KV cache grows with every token.

Think of prefill as reading the whole case file before a meeting and decode as answering questions with your notes open. If one GPU does both, a long prompt arriving mid-conversation makes everyone else’s answers stutter. So the big teams split them: one pool of GPUs does prefill, another does decode, and each is sized for its own job. The catch is that the KV cache, which can be gigabytes per request, now has to travel from one GPU to the other, quickly.

I sat in on three of the talks on this, and read the descriptions of a few more. The short version:

• OpenAI on rolling out inference engines. Meng Dong and Zihan Zheng opened Monday with “Rolling Out LLM Inference Engines on Kubernetes”: “Updating an LLM inference engine is not the same as rolling a stateless Kubernetes service.” A rollout has to keep the prefill-to-decode ratio balanced and respect GPU topology, and at OpenAI a preflight pipeline can keep GPU-backed pods pinned for hours.
• KAITO on splitting prefill and decode. Andy Zhang and Linbo He’s “Prefill Here, Decode There” showed KAITO (the Kubernetes AI Toolchain Operator, which started at Microsoft) setting up llm-d’s prefill and decode pools through a MultiRoleInference custom resource. The abstract puts the gain at 30 to 60% on prefill-heavy workloads, and admits the setup is complicated.
• Moving the KV cache with NIXL. NVIDIA’s NIXL (Inference Xfer Library) copies KV blocks from one GPU’s memory to another’s over RDMA, without stopping in CPU memory. In “Your KV Cache Is Bigger Than Your GPU”, Kfir Toledo and Nishtha Jain (IBM Research and Google Cloud) offloaded the cache through NIXL to networked storage: “The bottleneck isn’t compute. It’s that there’s nowhere to keep the cache.”
• Sharing the cache across replicas. From the session description, not my notes: “A Shared KV Cache for vLLM” (Mert Toslali and Dipanwita Guhathakurta, from work by Mengmei Ye and Chen Wang at IBM Research) replaces each vLLM replica’s private cache with a shared pool, so two replicas stop computing the same prompt prefix twice, and a surviving replica can pick up a dead one’s cache.
• Why the cache matters even more for agents. Two October posts on the vLLM blog by Sunita Nadampalli at AWS fill in the rest. In multi-turn agent conversations, the decode GPU sends its cache back to the prefill side instead of recomputing the history: time to first token on GLM-5-FP8 with 20k-token conversations dropped from 2,334 ms to 702 ms (against a baseline that recomputed everything, so it’s the best case). The second, written with Nicolò Lucchesi of Mistral, pushes the cache to the decode side as soon as it’s ready: up to about 3x lower time to first token on Qwen3-32B with 16k-token prompts at 8 queries per second.

Pro-tip: If you’re tuning LLM serving, look at the signals the llm-d team recommended in “Best Practices for LLM Autoscaling” (Jiazhou Gao, Abhishek Malvankar): queue depth, KV cache utilization, time to first token and request saturation. Raw GPU utilization tells you very little about an LLM.

Scaling the Agents That Call the Models

The second thing I learned was about the other end of the request: the agents calling these models, and how you run thousands of them on a cluster. Agents behave nothing like the web apps Kubernetes grew up serving, so the usual way of scaling pods doesn’t fit them.

Shaghayegh Moradirad, drawing on the design of OpenAI’s Codex Python SDK, put it well in the description of her lightning talk, “An Agent Is Not a Function Call”: “Most AI APIs are shaped like function calls: send input, receive output.” Coding agents hold state across turns, stream their work as they go, and can be steered or interrupted in the middle. Henrique Santana and Lanay Marques’s “Agents Are Not Microservices” lists what that does to a cluster: agent pods hold state, traffic is idle-then-burst, and health probes fail on pods that are perfectly healthy and just busy reasoning. (Kubernetes sees an agent thinking hard and decides it must have died, which is a little unfair to the agent.)

Agent Substrate, which joined the CNCF Sandbox this week, is the best example I saw of scaling agents properly. It maps thousands of stateful agents onto a much smaller pool of worker pods, and uses gVisor checkpoints to suspend and resume each agent’s memory and files in under a second. The project claims more than 30x oversubscription. In their demo, the team said they had battle-tested it with 10,000 agents, and they’re aiming for a billion agents running in the same cluster. Jaana Dogan, Dmitry Berkovich, Tim Hockin, Eitan Yarmush and Luiz Scheidegger presented it in Tuesday’s keynote, Tim Hockin and Dmitry Berkovich followed with “Kubernetes Solutions for Agent-Shaped Problems”, and kagent 1.0 now runs on it.

The other challenge everybody was talking about on the agent side was sandboxes: where an agent’s code runs, the policy it runs under, and the ingress and egress controls on what it can reach. None of that is specific to inference, but it came up in talk after talk. Two projects on the map are about exactly this. Agent Sandbox, from Kubernetes SIG Apps, manages the lifecycle of a sandboxed agent. OpenShell, an NVIDIA-led project that just joined the CNCF Sandbox (the CNCF’s name for its earliest project stage, which doesn’t make this paragraph any easier to read), sets policy for what an agent’s runtime can reach: files, network, credentials and model endpoints.

Two lightning talks were about what all those calls cost. Erica Hughberg’s “Cents and Sensibility” proposed cost-based limits, because “tool-calling agents can quickly spend dollars through iterations and context growth.” Ahmed Ibrahim’s “Agents Need a REPL” pointed out that “most agents call one tool per model turn” and showed agents calling tools from code instead.

Scale agents up like that and you also get the shape of the inference traffic they send: long, growing contexts (hence the multi-turn KV cache work above), many small calls per task, and bursts with nothing in between. Keep the last two in mind, because they come back below.

Everything Else on the Schedule

The rest of the week filled in the stack around those two threads. Gateways are learning to send a request to the model server that already holds its prompt prefix in cache. Schedulers hand out a fraction of a GPU instead of a whole one. Autoscalers watch queue depth and time to first token instead of CPU. And a good chunk of the talks were about getting multi-gigabyte models into a cluster faster and telling which pod is burning which GPU.

The GPU piece I’d single out is Dynamic Resource Allocation (DRA). The classic way to ask for a GPU is a count (nvidia.com/gpu: 1), and sharing a card meant setting up time-slicing or MIG partitions in the device plugin ahead of time. DRA, generally available since Kubernetes 1.34, lets a pod ask for a device by its properties instead: a slice of a partitionable GPU (that part is still in beta), or a GPU and a network card on the same PCIe root, which is what you want when a KV cache has to move between GPUs over RDMA. The rest of the stack is building on it. HAMi, which caps each pod’s GPU memory and compute in software, is moving to a lighter mode built on DRA, and Karpenter can provision nodes for the devices a DRA claim asks for. For small models, DRA makes a fraction of a GPU something a pod can ask for by what it needs. It still hands that slice to one pod, though, so someone still has to decide which models run inside it.

One talk outside inference really stood out for me: Decheng Dai’s “Anthropic’s Kubernetes Scheduler for Frontier AI: What Worked, What Broke”, on Thursday afternoon. It’s about the scheduler Anthropic built for training models. The starting point, from the abstract, is that the default Kubernetes scheduler scores pods one at a time and has no way to say “all 256 of these, on contiguous hardware, or none”. So Anthropic wrote its own, and then had to keep it fast as its clusters grew toward GKE’s limit of 65,000 nodes. It was fantastic how in-depth they went.

There’s too much for one post, so I put every piece on one map, with the problem each project solves, in the Open-Source Inference Map.

What the LLM Hype Overlooked: Task-Specific Open-Source Models

Remember what an agent does all day: search, rank what it found, pull fields out of a document, check whether a prompt is safe, and only then ask the big model to reason. Most of those steps are small, task-specific models (embedders, rerankers, entity extractors, classifiers, guard models, OCR), and a single agent step can call several of them before it calls the LLM once.

I went through about 60 inference sessions on the schedule, and none I found was about serving those models. That’s not because the problems don’t exist. They got overshadowed by the LLM challenges, and naturally so: if you’re going to the trouble of serving your own LLM, it’s a large reasoning model, and that’s where the hard, unsolved problems on stage were. Small models came up mostly as a way to save money. A telecom running cache-aware llm-d sent easy requests to a small model, and the OpenRL keynote talked about pairing “massive foundation LLMs for broad reasoning with lightweight, task-tuned models for targeted execution”. The closest talks to the serving problem were “Frameworkless ML Serving”, which scaled thousands of models from a queue (one model per pod), and a lightning talk, “One GPU Pool to Serve Them All”. (I couldn’t see every slot, so if I missed yours, tell me and I’ll happily stand corrected.)

The hallway was a different story. With the platform engineers I spoke with, a pattern kept repeating:

• There’s a list of approved open models. Most of their companies restricted which open models they could use.
• Each approved model gets hand-rolled YAML. Once a model was approved, someone wrote the manifests and got it served.
• The hard part is the person in the loop. Putting a small model on a fractional GPU wasn’t what slowed them down. What did was that somebody had to manage the catalog by hand: “this model isn’t being used yet, let’s take it out”, then “sorry, here’s a new model, we need to add it.” That’s ongoing operational overhead, and at one company, a whole inference team takes care of it.
• Every model is its own island. Each model was served separately, with no common API, so teams ended up building an API gateway in front of them, almost every time.
• Scaling on the queue isn’t the default yet. A lot of the talks said to scale on queue size rather than GPU utilization, but in practice that wasn’t what teams were doing.

Why Small Models Need Their Own Serving Stack

Most of the multi-GPU work I described above doesn’t apply to an encoder such as an embedding model or a reranker, which reads its input in one pass and keeps no KV cache at all. Run a dozen small models with the tools on the schedule, and you get the hallway list above, plus a few more problems:

• One deployment per model. Picture forty models: forty Helm releases, forty autoscalers and a fleet of GPUs sitting mostly idle, because each model holds its own hardware whether traffic is hitting it or not.
• The LLM engines want the whole card. The GPU-sharing work at KubeCon (fractional GPUs, memory caps, DRA) gives each pod an isolated slice, which helps, and it still leaves open which models should stay loaded in that slice. vLLM and SGLang reserve most of the GPU’s memory when they start (vLLM’s default is 90%, and both can be turned down by hand), so a second engine on the same card has little left.
• Routing many short requests wastes the GPU. A router picks a worker for each request from a snapshot of their queues, and with lots of short requests that snapshot is stale by the time the request lands. In our own experiments, that kind of routing struggled to push GPU utilization past 20 to 30% under constant load (for the workload we tested; yours will differ).
• Agent traffic comes in bursts. You either keep the GPUs on and pay for the quiet hours, or scale to zero and make the first request wait for a model to load.
• Workers die halfway through a batch. If the request lived only in the dead worker’s memory, it’s gone, and the agent waiting on it hangs.

A lot of existing tools already solve parts of this, and they’re good at it. But most of them work one model at a time. A KServe InferenceService, a Helm chart or a Hugging Face TEI container each serves one model, so forty models means forty of them, each written, reviewed and rolled out on its own. The multi-model servers go further. NVIDIA Triton (now Dynamo-Triton) hosts several models in one server with dynamic batching and explicit loading and unloading, KServe ModelMesh loads and unloads models across a pool of pods based on use, and Ray Serve can multiplex models on one replica and hand replicas a fraction of a GPU. Even with those, each model is usually still described and rolled out on its own, and someone on the platform team still owns the catalog.

Where SIE Is Different

I work at Superlinked, so read this part knowing where I sit. SIE, our open-source inference engine, orchestrates many models together, automatically, instead of one model at a time. You give it the catalog of models you’ve approved, and it decides which ones are loaded on which GPUs, packs several onto one card and scales the workers on demand. It runs in your own cluster, and you install it with one Helm chart, with overlays for EKS, GKE and AKS. Taking the hallway list one at a time:

• The catalog is config, not YAML. One SIE deployment serves 100+ models. Adding a model is a config change, with no new release to ship and no Helm file per model. (Hundreds of models, one deployment has the details.)
• Nobody moves models in and out by hand. SIE loads a model the first time it’s called and unloads the least recently used one when GPU memory fills up. You pin the few models on your critical path so they never get unloaded, and SIE handles the rest. For SGLang models without a memory setting of their own, it works out how much memory each one needs and tells the engine to take exactly that, so several fit on one card.
• One API for every model. Your application, or your agent, gets one HTTP endpoint for embedding, scoring, extraction and generation, with the same request shape whichever model is behind it, so nobody has to build another gateway. There’s an MCP server too, so an agent can call the models as tools.
• One queue for every model. This is the part I find most interesting, and it only works because SIE is multi-model. For embedding, scoring and extraction, a stateless gateway puts each request on a NATS JetStream queue (NATS graduated in the CNCF this week, which I’ll take as a sign), and any worker in a pool can pick up a request for any model. Each worker pulls what it can handle right now and builds its own batches, grouped by model, operation and LoRA adapter, instead of a router guessing which worker has room. When traffic moves from one model to another, a worker unloads one and loads the other inside the same pod, without rescheduling anything. In our experiments, this roughly doubled the cluster’s throughput for the workload we tested. (LLM generation works differently: the gateway sends each request to the worker most likely to already hold its prompt prefix in cache.) (How to serve a fleet of small open-source models and routing, batching, model configs and LoRA in one cluster go deeper.)
• Scaling on the queue is the default. KEDA scales each worker pool on queue metrics, down to zero when nobody’s calling. A request that arrives at zero gets a 503 with a PROVISIONING error code and a Retry-After header while a worker starts and loads the model (the SDK retries for you with wait_for_capacity=True), so for the critical-path models you keep a minimum number of warm workers.
• A dead worker doesn’t lose the request. A request stays on the queue until a worker finishes it. If the worker dies first, JetStream hands it to another one after a 30-second timeout (five minutes for generation), up to 20 tries.

Queues on their own aren’t new at KubeCon. “Frameworkless ML Serving” scaled thousands of models from a queue with KEDA, and a tutorial packed several LLMs onto one GPU with a solver choosing where each goes. But both still ran one model per pod, so every model got its own pods and its own scaling. I didn’t see anyone else running one shared queue across many small models.

It also sits next to the rest of the stack. Keep your LLMs on llm-d or KServe, and put SIE beside them for the small models, behind the same Gateway API front door. If tenants need hard memory isolation, give each tenant its own MIG instance with its own SIE workers on it, and let SIE pack that tenant’s models inside. HAMi’s caps are a lighter option: it enforces them in software, by intercepting CUDA calls, so treat them as a soft limit. A limit worth knowing: by default, SIE doesn’t place models from one central plan, so a popular model can end up loaded on several workers. There’s an opt-in placement planner for teams that want one.

Conclusion

KubeCon NA 2026 left me with three things. The Kubernetes community has converged on how to serve big LLMs across many GPUs: an inference-aware gateway, llm-d or KServe, prefill and decode on separate GPUs, and a KV cache that moves over NIXL. Agents became the main customer for all of it, and projects like Agent Substrate are working out how to run thousands of them on a small pool of pods. And the task-specific open-source models those agents call on every step got overshadowed, which leaves platform teams hand-rolling YAML, managing a model catalog by hand and building yet another gateway. That’s the part SIE does.

If you were in Salt Lake City and saw a talk on serving small models that I missed, or you’re the person moving models in and out of your cluster right now, I’d love to talk. The recordings should show up on the CNCF YouTube channel over the next few weeks, so start with the KV cache talks and work your way out.

Until next time.