--- title: "Sharing GPUs between models: SIE vs HAMi, MIG and Run:ai" description: "MIG, HAMi and Run:ai decide how much of a GPU each pod gets. SIE decides which models are loaded on it right now. What each layer controls, what happens on an out-of-memory error, and when to use which." canonical_url: https://superlinked.com/blog/sharing-gpus-between-models last_updated: 2026-10-16 --- Short version: these tools answer different questions, so most comparisons of them talk past each other. MIG, HAMi and NVIDIA Run:ai decide how much of a GPU each pod gets. That share is set when the pod is scheduled; Run:ai's dynamic fractions also let a pod borrow memory above it until the owner takes it back. SIE (Superlinked Inference Engine) works one level up: it knows which model each request needs, loads that model the first time a request arrives and unloads the least recently used one when memory runs short. Use MIG when tenants need a hardware boundary between them. Use SIE when many models should share GPUs according to their traffic. ## Two questions that get mixed up "How much of this GPU does a pod get?" is a scheduling question. It is answered when the pod is scheduled, in gigabytes and streaming multiprocessors, before the first request arrives. "Which models should be on this GPU right now?" is a serving question. It changes request by request, because an agent's traffic does not spread evenly over its models: the embedding model runs on every query, the OCR model runs when a scanned document shows up, and the guard model runs twice per conversation. Partitioning tools answer the first question well and cannot see the second. That is why a slice sized for an OCR model sits idle at 3 a.m. while the embedding model next to it queues. ## What each layer controls **NVIDIA MIG splits the hardware.** A supported data-center GPU (A100, H100, H200 and B200 among them; not the L4) divides into as many as seven instances. Each instance has its own memory, L2 cache slice, memory controllers and streaming multiprocessors, and a fault in one does not reach the others ([MIG user guide](https://docs.nvidia.com/datacenter/tesla/mig-user-guide/), updated 13 October 2026). The profiles are fixed shapes. Changing them needs an idle GPU and a reset. **HAMi caps what a pod may allocate.** HAMi intercepts CUDA calls inside the container. A pod requests `nvidia.com/gpumem` and `gpucores`; when it allocates past its memory quota, `cuMemAlloc` returns `CUDA_ERROR_OUT_OF_MEMORY` to that pod, and compute is throttled to its share ([HAMi](https://github.com/Project-HAMi/HAMi), v2.10.0). This is software enforcement, and HAMi's own [virtualization docs](https://project-hami.io/docs/core-concepts/gpu-virtualization) point to MIG when you need true isolation. **Run:ai schedules fractions.** A GPU fraction gives a pod a hard memory cap with time-sliced compute. Dynamic fractions add a request and a limit: a pod can borrow memory above its request, and the docs warn that memory is "susceptible to an 'OOM Kill'" when its owner reclaims it. GPU memory swap moves idle workloads to CPU RAM so more containers fit on a GPU, at the granularity of a whole container ([Run:ai docs](https://run-ai-docs.nvidia.com/)). NVIDIA released Run:ai as open source under Apache 2.0 on [13 October 2026](https://developer.nvidia.com/blog/nvidia-runai-is-now-open-source-as-part-of-nvidia-dsx-ai-factory-platform/). **SIE manages models inside worker pools.** You assign each model to a pool, and each pool runs on its own workers and GPUs. Inside a pool, a worker loads a model on its first request, keeps hot models resident and evicts the least recently used model when GPU memory passes 95%. Pinned models are never evicted. Every model reserves memory sized to its weights and its KV budget, so small models share a GPU: Qwen3Guard 4B and Granite Guardian 2B sit on one card together. And SIE refuses any configuration that puts an LLM in the same pool as embedding, scoring or extraction models. ## Memory and KV cache Under a static partition, each model's memory share is decided before traffic arrives. Run two vLLM servers on one GPU and you split it yourself: `--gpu-memory-utilization` defaults to 0.92 per instance, and the [vLLM docs](https://docs.vllm.ai/en/latest/configuration/engine_args.html) suggest 0.5 each for two. An idle model keeps its half. A busy one cannot borrow it. SIE moves that decision to request time. The LLM sits in its own pool, where no encoder can land, and you pin it so nothing evicts it; its KV cache is not shared with anything you did not put there. The guard models share a second pool. Embedding and extraction models share a third and load as traffic reaches them. One limit worth stating plainly: SIE does not throttle a long-context burst by default. The token-budget admission check is opt-in. If one pool must absorb bursts without slowing its neighbours inside that pool, give the bursty model its own pool or turn the budget on. NVIDIA measured what static packing costs in its own [many-model serving guide for Dynamo](https://docs.nvidia.com/dynamo/dev/many-model-serving/interpreting-gpu-sharing-results): two Qwen3-4B models packed onto one A100-40GB kept 60 to 88% of dedicated-GPU throughput with GPU fractions, and 45 to 73% with HAMi. Those are NVIDIA's numbers on its workload, not a comparison with SIE. They show the price of fixing each model's share in advance. ### Two guard models shared 22.77 GiB on an H100 We sent six alternating SDK requests to Qwen3Guard-Gen-4B and Granite Guardian 3.0 2B on one H100 80 GB. The first request loaded Qwen; the second loaded Granite alongside it. All six succeeded, and both models stayed resident in the same processes, using **22.77 GiB** of device memory together. Their KV budgets were 32,768 tokens for Qwen and 8,192 for Granite. Before it loads a model, SIE checks that model's memory reservation against free memory; the two reservations came to 24.8 GiB. Then we made the next model not fit. With Granite unloaded, we filled GPU memory until less than its reservation was free and requested Granite again. SIE evicted the unpinned model and loaded Granite. The [measurement record](/benchmarks/native-inference-2026-10-16/results.json) has the model versions, settings and observations. ## When a model runs out of memory or crashes **MIG.** A fault stays inside its instance. That is the point of the hardware. **HAMi.** The pod that exceeds its quota gets an out-of-memory error; its neighbours keep their quotas. HAMi's docs do not describe what happens to other pods when a process faults the device itself. **Run:ai.** Memory borrowed above a dynamic fraction's request can be OOM-killed when the owner takes it back. Plan for that if you use borrowing. **SIE.** When a batch runs out of memory, the worker clears caches, evicts the least recently used model on that GPU (never a pinned one) and splits the batch, up to 16 smaller batches. Only then does the caller get a `503 RESOURCE_EXHAUSTED` with `Retry-After`, which the SDK retries. Each SGLang-served model runs in its own process; if one crashes, SIE unloads it and reloads it on the next request after a short cooldown, and backs off if it keeps crashing. Pools run on separate workers, so a failure in one pool does not stop another. Inside a pool, recovery can unload another model to save the batch in hand. That model reloads on its next request. If a model must never pay that reload, pin it or give it its own pool. ### The healthy model kept serving through a crash On one H100 80 GB, we pinned Qwen3Guard-Gen-4B and killed Granite Guardian's SGLang process. Qwen kept its process and answered **96 of 96 requests**, with a two-second pause between requests, before, during and after the recovery. | Qwen requests | Successful / attempted | | --- | --- | | Before the crash | 25 / 25 | | Around recovery | 46 / 46 | | After recovery | 25 / 25 | | Total | **96 / 96** | SIE detected the dead process in **1.08 seconds**. After a configured 30-second cooldown, the next Granite request started the reload, and Granite was serving again 91 seconds after the kill. The [measurement record](/benchmarks/native-inference-2026-10-16/results.json) has the configuration and timings. ## When to use which - **MIG** when tenants must not affect each other at the hardware level, or a compliance review asks for a physical boundary, and your GPUs support it. - **HAMi** when teams share a Kubernetes cluster and you want per-pod memory caps without MIG hardware. - **Run:ai** when the problem is scheduling across teams and projects: quotas, priorities and fair share over a GPU fleet. - **SIE** when one application or one platform serves many models with uneven traffic and you want them behind one API, sharing GPUs by demand. They are not mutually exclusive. Partitioning decides what a pod may use; SIE decides what a worker loads. A cluster can run MIG or HAMi for the workloads that need a hard cap and SIE pools for the models that should share by traffic. ## At a glance | | SIE | MIG | HAMi | Run:ai fractions | | --- | --- | --- | --- | --- | | Unit of sharing | A model in a worker pool | A hardware instance | A pod's memory and compute quota | A pod's fraction of a GPU | | Knows which model a request needs | Yes | No | No | No | | Idle model's memory | Evicted under pressure; pinned models stay | Reserved | Reserved | Reserved; swap moves idle containers to CPU RAM | | How limits are enforced | Engine residency per pool | Hardware partition | CUDA call interception | Scheduler and fraction limits | | Fault boundary | Between pools, on separate workers; per process for SGLang-served models | Per instance, in hardware | Allocation errors stay with the pod; no hardware isolation | No hardware isolation between fractions | | Change without a restart | Models load and unload with traffic | No: reconfiguring needs an idle GPU | No: quota is set per pod | Dynamic fractions borrow up to a limit | ## FAQ **Do I need MIG if I run SIE?** Only if tenants need a hardware fault boundary between them. SIE's pools separate workloads by process and GPU; MIG separates them inside one GPU, in hardware. **Will a long-context burst on the LLM push out my embedding model?** No. They are in different pools by design, on different workers. A burst competes only with what shares the LLM's pool, and you choose what that is. **Can two small LLMs share one GPU in SIE?** Yes. Each model reserves memory for its own weights and KV budget, so guard-sized models such as Qwen3Guard 4B and Granite Guardian 2B load side by side. A large LLM that should own its GPU can say so in its profile. **What does Run:ai memory swap give me that SIE eviction does not?** Swap keeps a whole container's GPU state in CPU RAM, so it resumes faster than a cold load. SIE frees memory one model at a time and reloads weights when the model is needed again; the [autoscaling docs](https://superlinked.com/docs/deployment/autoscaling) put model loading at 10 to 120 seconds. Pin the models that cannot wait. **Can I run HAMi and SIE on the same cluster?** They act at different layers, so there is no conflict in design: HAMi caps what a pod may allocate and SIE decides what a worker loads. Validate the combination on your own deployment before relying on it. [Star the repo on GitHub](https://github.com/superlinked/sie) · [Read the engine docs](https://superlinked.com/docs/engine) · [SIE vs vLLM](https://superlinked.com/blog/sie-vs-vllm-multi-model-workloads) · [llm-d vs Dynamo vs SIE](https://superlinked.com/blog/llm-d-vs-dynamo-vs-sie-multi-model-serving)