--- title: Multi-GPU Serving description: Split a generation model that does not fit on one GPU across several GPUs with tensor parallelism, on a single server or a Kubernetes worker pool. canonical_url: https://superlinked.com/docs/deployment/multi-gpu last_updated: 2026-09-23 --- A model too large for one GPU can run on several. SIE starts one SGLang engine that splits the model across a block of GPUs with tensor parallelism, and the model keeps that block until it unloads. > **Note — Available in SIE 0.8.0:** > > Multi-GPU serving ships with SIE 0.8.0. That release adds the `tensor_parallel_size` load-time option and the Helm chart's `gpu.deviceGroup` and `shmSize` settings. ## When to split a model Split a model across GPUs only for a generation model that does not fit on one GPU. If the model fits on one GPU, start with separate copies on the available GPUs. On Kubernetes, a pool with `gpu.count` above 1 and no `gpu.deviceGroup` already runs one worker per GPU. Splitting adds communication between GPUs, so benchmark your workload before expecting lower latency or higher throughput, especially on GPUs without a fast interconnect. ## Declare the width Source: [packages/sie_server/src/sie_server/config/device_groups.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/config/device_groups.py) Set `adapter_options.loadtime.tensor_parallel_size` on the model's `default` profile. The SGLang generation and embedding adapters accept it, as do the adapters built on the SGLang generation adapter; any other adapter refuses the option when the model loads. ```yaml sie_id: my-org/my-large-model hf_id: my-org/my-large-model tasks: generate: context_length: 32768 max_output_tokens: 4096 profiles: default: max_batch_tokens: 16384 compute_precision: bfloat16 kv_budget_tokens: 32768 adapter_path: sie_server.adapters.sglang.generation:SGLangGenerationAdapter adapter_options: loadtime: tensor_parallel_size: 4 request_read_timeout_s: 600 startup_timeout_s: 900 ``` ### Load-time options Source: [packages/sie_server/src/sie_server/adapters/sglang/generation.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/adapters/sglang/generation.py) Source: [packages/sie_server/src/sie_server/adapters/sglang/embedding.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/adapters/sglang/embedding.py) All six options go under `adapter_options.loadtime`, and both SGLang adapters accept each of them. | Option | Default | Rules | |--------|---------|-------| | `tensor_parallel_size` | `1` | Integer from 1 to 8, and no more than the number of GPUs the server has | | `request_read_timeout_s` | Generation: `SIE_SGLANG_GENERATE_READ_TIMEOUT_S` if set, otherwise none. Embedding: `60` | Seconds to wait for the next bytes from the engine before the request fails. A positive number. Required for generation above width 1 when that variable is unset | | `startup_timeout_s` | Environment startup budget, otherwise `900` | Seconds to wait for engine readiness. A finite number above zero. Generation above width 1 must declare its own value, even when an environment budget is set | | `watchdog_timeout_s` | The engine's default | Seconds one forward batch may run before the engine crashes itself instead of hanging. A positive number | | `disable_piecewise_cuda_graph` | `true` above width 1, `false` at width 1. On the `sglang-cu130` bundle, `true` at every width | Turns off only SGLang's prefill CUDA graph capture (called piecewise capture before SGLang 0.5.20), not CUDA graphs as a whole. No effect when `disable_cuda_graph` is set | | `nccl_port` | A free port from 30400 to 30499 | Rendezvous port the group's GPU processes use to find each other. Integer from 1024 to 65535. Only allowed above width 1 | A stalled group of GPUs sends no bytes and raises no error, which is why generation needs a read cap above width 1. The cap fails the request and does not restart the engine. SGLang's prefill CUDA graph capture can hang or run out of memory above width 1, so SIE turns it off unless the profile sets `disable_piecewise_cuda_graph: false`. The `sglang-cu130` bundle runs SGLang 0.5.20, which captures a prefill graph for multimodal models that earlier versions never captured, so on that bundle SIE turns the capture off at every width. Choose `startup_timeout_s` from the profile's measured startup time, including per-rank graph compilation and capture. The example's `900` seconds is a starting point. On Kubernetes, keep it within `workers.common.modelReadyTimeoutSec` and below the worker's liveness probe budget; an invalid declared value fails the load instead of falling back to a default. ### The width must fit the model Source: [packages/sie_server/src/sie_server/adapters/sglang/_server.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/adapters/sglang/_server.py) Tensor parallelism divides attention heads between GPUs, so the width must divide `num_attention_heads` evenly. It must also be a multiple or a divisor of `num_key_value_heads`: the engine splits key/value heads when there are at least as many as the width and replicates them when there are fewer. SIE reads both counts from the model's `config.json` (from `text_config` for multimodal models) and refuses a width that breaks either rule before the engine starts. When the config is in neither a local model directory nor the Hugging Face cache, SIE skips this check and the engine applies the same rule after it loads the weights. ## What SIE refuses Source: [packages/sie_server/src/sie_server/config/model.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/config/model.py) Source: [packages/sie_server/src/sie_server/core/loader.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/core/loader.py) SIE reserves GPUs from the declared width alone. It refuses: - **Engine placement flags in `extra_launch_args`.** The refused flags are `--tensor-parallel-size`, `--tp-size`, `--data-parallel-size`, `--dp-size`, `--expert-parallel-size`, `--ep-size`, `--pipeline-parallel-size`, `--pp-size`, `--nnodes`, `--node-rank`, `--dist-init-addr`, `--base-gpu-id` and `--gpu-id-step`. SGLang accepts any unambiguous prefix of a flag, so abbreviations such as `--tp` or `--tensor-parallel` are refused too, as is the `--flag=value` form. - **Engine listener flags in `extra_launch_args`.** `--host`, `--port` and `--nccl-port`, including abbreviations and `--flag=value`, are reserved by SIE. Set `loadtime.nccl_port` to choose a rendezvous port. SIE owns the engine's HTTP host and port; these are separate from the worker server's own CLI arguments. - **Device-visibility variables in `extra_env`.** `CUDA_VISIBLE_DEVICES`, `NVIDIA_VISIBLE_DEVICES` and `ROCR_VISIBLE_DEVICES` are refused in any letter case. The launcher writes the device mask itself, after the profile's environment. - **Load-time options the adapter does not accept.** A misspelled `tensor_parallel_size` fails the load instead of serving on one GPU, and the error names the closest accepted spelling. This check applies to every model. - **A width larger than the number of GPUs the server has.** A server with two GPUs refuses width 4 outright. Reserved flags, device-visibility variables, and a width that is not an integer from 1 to 8 are refused when SIE reads a model config; the [Config API](/docs/engine/config-api/) applies the same checks when a config is written. Unknown load-time options and a width above the server's GPU count are refused when the model loads. > **Caution — Check custom configs before upgrading:** > > These checks also apply to existing models. Previously ignored load-time options now fail the load. Replace a placement override such as `extra_launch_args: ["--tp", "4"]` with `loadtime.tensor_parallel_size: 4`, remove reserved flags and device masks from adapter overrides, and declare the generation startup and read budgets above. Check each profile's resolved options, including inherited values, before deploying the upgrade. ## How the server places a model Source: [packages/sie_server/src/sie_server/core/registry.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/core/registry.py) The server's GPUs are the devices listed in `SIE_DEVICES`, for example `cuda:0,cuda:1,cuda:2,cuda:3`. Without it the server has one device, so it refuses any width above 1. The Helm chart sets it for you (see [Kubernetes worker pools](#kubernetes-worker-pools)). A model with width N claims a contiguous block of N GPUs, such as `cuda:2` and `cuda:3` for width 2. The claim is exclusive and lasts as long as the model stays loaded; no other model loads onto those GPUs. When no block is empty, the server evicts unpinned models to free one: - It skips any block that holds a pinned model. Pinned models are never evicted to make room for a group. - It picks the block that needs the fewest evictions. - On a tie, it picks the block whose most recently used model has been idle longest. - If no block can be freed, the load fails with a placement error, and a later request can retry it after other models unload. A single-GPU model can displace a group too: when every GPU is claimed, it evicts the least recently used unpinned group. The claim ends when the model unloads, whether it is evicted or unloaded on request. A load that fails releases its GPUs right away. ### Each profile loads separately Source: [packages/sie_server/src/sie_server/core/loader.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/core/loader.py) Source: [packages/sie_gateway/src/state/model_registry.rs](https://github.com/superlinked/sie/blob/main/packages/sie_gateway/src/state/model_registry.rs) The server runs every non-default profile as a separate model, `:`, with its own engine; a request loads one when it names that model, or when the gateway routes a grammar-constrained request to the profile named in the model's `grammar_profile`. A profile that extends `default` without a nonempty `loadtime` block of its own inherits the width (one with a nonempty block must repeat it) and claims a separate block, so a width-2 model with such a profile loaded beside it needs 4 GPUs, and on 2 GPUs loading one evicts the other. ## Kubernetes worker pools Source: [deploy/helm/sie-cluster/templates/worker-statefulset.yaml](https://github.com/superlinked/sie/blob/main/deploy/helm/sie-cluster/templates/worker-statefulset.yaml) Source: [deploy/helm/sie-cluster/values.yaml](https://github.com/superlinked/sie/blob/main/deploy/helm/sie-cluster/values.yaml) A worker pool with `gpu.count` above 1 normally runs one worker per GPU. Add `gpu.deviceGroup: true` and the pod runs a single worker that owns every GPU in the pod instead: the chart requests all `gpu.count` GPUs for that one container and lists them in its `SIE_DEVICES`, `cuda:0,cuda:1,cuda:2,cuda:3` for four. `gpu.deviceGroup` requires a `gpu.count` of at least 2, and the chart refuses to render otherwise. This pool runs one worker across four GPUs with the `sglang` bundle, which carries the SGLang generation adapter. The node selector assumes your L4 nodes have the label `nvidia.com/gpu.product: NVIDIA-L4`; replace it with the actual GPU-model label on your nodes. `gpuType` and `machineProfile` identify the serving pool, and do not constrain the physical GPU type. Its CPU and memory are four times the chart's single-GPU `l4` pool, and its `shmSize` is four times the `8Gi` default; treat them as starting points and size them for the model and the node. ```yaml workers: pools: l4-4x: enabled: true machineProfile: l4-4x gpuType: l4 nodeSelector: nvidia.com/gpu.product: NVIDIA-L4 gpu: count: 4 deviceGroup: true shmSize: 32Gi resources: requests: cpu: "16" memory: "64Gi" limits: cpu: "32" memory: "128Gi" tolerations: - key: nvidia.com/gpu operator: Exists effect: NoSchedule bundles: sglang: minReplicas: 1 maxReplicas: 2 ``` The pod schedules only on a node with four free GPUs. In the model config, declare a `tensor_parallel_size` no larger than `gpu.count`; setting them equal gives the model the whole pod. Send requests to the pool with `gpu="l4-4x"`. The pod reports one serving slot, however many GPUs it holds. `SIE_GPU_COUNT` on the sidecar and `gpu_count` in cluster health both count serving slots, not GPUs. Every worker pod mounts `/dev/shm` as a memory-backed volume. Its size limit is the pool's `shmSize`, or `workers.common.shmSize` (default `8Gi`) when the pool sets none. A tensor-parallel engine runs one process per GPU, and those processes can exchange data through shared memory, so a device-group pool may need a larger `shmSize`. What the worker writes there counts against its container's memory limit, so raise `resources.limits.memory` along with it. ## What's Next - [Model Profiles](/docs/engine/profiles/) - profile fields and adapter options - [Adding Models](/docs/engine/adding-models/) - the full model config format - [Hardware & Capacity](/docs/deployment/resources/) - GPU selection and memory planning - [Scale-from-Zero & Autoscaling](/docs/deployment/autoscaling/) - how worker pods scale