Multi-GPU Serving

A model too large for one GPU can run on several. SIE starts one SGLang engine that splits the model across a block of GPUs with tensor parallelism, and the model keeps that block until it unloads.

When to split a model

Split a model across GPUs only for a generation model that does not fit on one GPU.

If the model fits on one GPU, start with separate copies on the available GPUs. On Kubernetes, a pool with gpu.count above 1 and no gpu.deviceGroup already runs one worker per GPU. Splitting adds communication between GPUs, so benchmark your workload before expecting lower latency or higher throughput, especially on GPUs without a fast interconnect.

Declare the width

Set adapter_options.loadtime.tensor_parallel_size on the model’s default profile. The SGLang generation and embedding adapters accept it, as do the adapters built on the SGLang generation adapter; any other adapter refuses the option when the model loads.

sie_id: my-org/my-large-model
hf_id: my-org/my-large-model
tasks:
  generate:
    context_length: 32768
    max_output_tokens: 4096
profiles:
  default:
    max_batch_tokens: 16384
    compute_precision: bfloat16
    kv_budget_tokens: 32768
    adapter_path: sie_server.adapters.sglang.generation:SGLangGenerationAdapter
    adapter_options:
      loadtime:
        tensor_parallel_size: 4
        request_read_timeout_s: 600
        startup_timeout_s: 900

Load-time options

All six options go under adapter_options.loadtime, and both SGLang adapters accept each of them.

Option	Default	Rules
`tensor_parallel_size`	`1`	Integer from 1 to 8, and no more than the number of GPUs the server has
`request_read_timeout_s`	Generation: `SIE_SGLANG_GENERATE_READ_TIMEOUT_S` if set, otherwise none. Embedding: `60`	Seconds to wait for the next bytes from the engine before the request fails. A positive number. Required for generation above width 1 when that variable is unset
`startup_timeout_s`	Environment startup budget, otherwise `900`	Seconds to wait for engine readiness. A finite number above zero. Generation above width 1 must declare its own value, even when an environment budget is set
`watchdog_timeout_s`	The engine’s default	Seconds one forward batch may run before the engine crashes itself instead of hanging. A positive number
`disable_piecewise_cuda_graph`	`true` above width 1, `false` at width 1. On the `sglang-cu130` bundle, `true` at every width	Turns off only SGLang’s prefill CUDA graph capture (called piecewise capture before SGLang 0.5.20), not CUDA graphs as a whole. No effect when `disable_cuda_graph` is set
`nccl_port`	A free port from 30400 to 30499	Rendezvous port the group’s GPU processes use to find each other. Integer from 1024 to 65535. Only allowed above width 1

A stalled group of GPUs sends no bytes and raises no error, which is why generation needs a read cap above width 1. The cap fails the request and does not restart the engine. SGLang’s prefill CUDA graph capture can hang or run out of memory above width 1, so SIE turns it off unless the profile sets disable_piecewise_cuda_graph: false. The sglang-cu130 bundle runs SGLang 0.5.20, which captures a prefill graph for multimodal models that earlier versions never captured, so on that bundle SIE turns the capture off at every width.

Choose startup_timeout_s from the profile’s measured startup time, including per-rank graph compilation and capture. The example’s 900 seconds is a starting point. On Kubernetes, keep it within workers.common.modelReadyTimeoutSec and below the worker’s liveness probe budget; an invalid declared value fails the load instead of falling back to a default.

The width must fit the model

Tensor parallelism divides attention heads between GPUs, so the width must divide num_attention_heads evenly. It must also be a multiple or a divisor of num_key_value_heads: the engine splits key/value heads when there are at least as many as the width and replicates them when there are fewer.

SIE reads both counts from the model’s config.json (from text_config for multimodal models) and refuses a width that breaks either rule before the engine starts. When the config is in neither a local model directory nor the Hugging Face cache, SIE skips this check and the engine applies the same rule after it loads the weights.

What SIE refuses

SIE reserves GPUs from the declared width alone. It refuses:

Engine placement flags in extra_launch_args. The refused flags are --tensor-parallel-size, --tp-size, --data-parallel-size, --dp-size, --expert-parallel-size, --ep-size, --pipeline-parallel-size, --pp-size, --nnodes, --node-rank, --dist-init-addr, --base-gpu-id and --gpu-id-step. SGLang accepts any unambiguous prefix of a flag, so abbreviations such as --tp or --tensor-parallel are refused too, as is the --flag=value form.
Engine listener flags in extra_launch_args. --host, --port and --nccl-port, including abbreviations and --flag=value, are reserved by SIE. Set loadtime.nccl_port to choose a rendezvous port. SIE owns the engine’s HTTP host and port; these are separate from the worker server’s own CLI arguments.
Device-visibility variables in extra_env. CUDA_VISIBLE_DEVICES, NVIDIA_VISIBLE_DEVICES and ROCR_VISIBLE_DEVICES are refused in any letter case. The launcher writes the device mask itself, after the profile’s environment.
Load-time options the adapter does not accept. A misspelled tensor_parallel_size fails the load instead of serving on one GPU, and the error names the closest accepted spelling. This check applies to every model.
A width larger than the number of GPUs the server has. A server with two GPUs refuses width 4 outright.

Reserved flags, device-visibility variables, and a width that is not an integer from 1 to 8 are refused when SIE reads a model config; the Config API applies the same checks when a config is written. Unknown load-time options and a width above the server’s GPU count are refused when the model loads.

How the server places a model

The server’s GPUs are the devices listed in SIE_DEVICES, for example cuda:0,cuda:1,cuda:2,cuda:3. Without it the server has one device, so it refuses any width above 1. The Helm chart sets it for you (see Kubernetes worker pools).

A model with width N claims a contiguous block of N GPUs, such as cuda:2 and cuda:3 for width 2. The claim is exclusive and lasts as long as the model stays loaded; no other model loads onto those GPUs.

When no block is empty, the server evicts unpinned models to free one:

It skips any block that holds a pinned model. Pinned models are never evicted to make room for a group.
It picks the block that needs the fewest evictions.
On a tie, it picks the block whose most recently used model has been idle longest.
If no block can be freed, the load fails with a placement error, and a later request can retry it after other models unload.

A single-GPU model can displace a group too: when every GPU is claimed, it evicts the least recently used unpinned group.

The claim ends when the model unloads, whether it is evicted or unloaded on request. A load that fails releases its GPUs right away.

Each profile loads separately

The server runs every non-default profile as a separate model, <model>:<profile>, with its own engine; a request loads one when it names that model, or when the gateway routes a grammar-constrained request to the profile named in the model’s grammar_profile. A profile that extends default without a nonempty loadtime block of its own inherits the width (one with a nonempty block must repeat it) and claims a separate block, so a width-2 model with such a profile loaded beside it needs 4 GPUs, and on 2 GPUs loading one evicts the other.

Kubernetes worker pools

A worker pool with gpu.count above 1 normally runs one worker per GPU. Add gpu.deviceGroup: true and the pod runs a single worker that owns every GPU in the pod instead: the chart requests all gpu.count GPUs for that one container and lists them in its SIE_DEVICES, cuda:0,cuda:1,cuda:2,cuda:3 for four. gpu.deviceGroup requires a gpu.count of at least 2, and the chart refuses to render otherwise.

This pool runs one worker across four GPUs with the sglang bundle, which carries the SGLang generation adapter. The node selector assumes your L4 nodes have the label nvidia.com/gpu.product: NVIDIA-L4; replace it with the actual GPU-model label on your nodes. gpuType and machineProfile identify the serving pool, and do not constrain the physical GPU type. Its CPU and memory are four times the chart’s single-GPU l4 pool, and its shmSize is four times the 8Gi default; treat them as starting points and size them for the model and the node.

workers:
  pools:
    l4-4x:
      enabled: true
      machineProfile: l4-4x
      gpuType: l4
      nodeSelector:
        nvidia.com/gpu.product: NVIDIA-L4
      gpu:
        count: 4
        deviceGroup: true
      shmSize: 32Gi
      resources:
        requests:
          cpu: "16"
          memory: "64Gi"
        limits:
          cpu: "32"
          memory: "128Gi"
      tolerations:
        - key: nvidia.com/gpu
          operator: Exists
          effect: NoSchedule
      bundles:
        sglang:
          minReplicas: 1
          maxReplicas: 2

The pod schedules only on a node with four free GPUs. In the model config, declare a tensor_parallel_size no larger than gpu.count; setting them equal gives the model the whole pod. Send requests to the pool with gpu="l4-4x".

The pod reports one serving slot, however many GPUs it holds. SIE_GPU_COUNT on the sidecar and gpu_count in cluster health both count serving slots, not GPUs.

Every worker pod mounts /dev/shm as a memory-backed volume. Its size limit is the pool’s shmSize, or workers.common.shmSize (default 8Gi) when the pool sets none. A tensor-parallel engine runs one process per GPU, and those processes can exchange data through shared memory, so a device-group pool may need a larger shmSize. What the worker writes there counts against its container’s memory limit, so raise resources.limits.memory along with it.

What’s Next

Model Profiles - profile fields and adapter options
Adding Models - the full model config format
Hardware & Capacity - GPU selection and memory planning
Scale-from-Zero & Autoscaling - how worker pods scale