Cold starts and SIE Relay

SIE Cloud runs models on dedicated GPU capacity, and that capacity can scale to zero when a model has no traffic. Starting it again takes time. While it starts, SIE Relay can answer the request through a vetted inference partner, so your application gets an answer instead of waiting for the cold start.

A relayed request is validated the same way and returns the same response format, although latency and output can differ slightly from dedicated capacity. It costs the same as a request served by dedicated capacity, and the response headers say which side served it.

When SIE Relay serves a request

The platform decides which requests to relay. When a model’s dedicated capacity is not ready, because it has scaled to zero or is still starting, a request for that model can be relayed. Dedicated capacity starts when sustained demand warrants it, so a model with little traffic can be served by SIE Relay for longer. Once dedicated capacity is ready, it serves the model’s requests again.

A request is relayed only before dedicated capacity accepts it, so it never runs twice.

Some models and profiles are served only through SIE Relay. The console labels them Served by SIE Relay.

For a model with dedicated capacity, a request is never relayed when:

it is an embedding or reranking request. SIE Relay serves only generation and extraction
it carries the X-SIE-Remote: forbid header, which the SDK remote option sets
your organization turned SIE Relay off
it names a profile that runs on dedicated capacity, or asks for grammar-constrained structured output
no SIE Relay partner serves the model in the region of your API key

Such a request starts dedicated capacity if it is not running, then waits for it, as described in Retry while capacity starts. Batch jobs are never relayed either. For a model with dedicated capacity, a cold start delays them. The batch and jobs APIs refuse a model or profile served only through SIE Relay with 400 INVALID_REQUEST, so send those requests to the realtime endpoints.

See who served a request

Response headers

Header	Meaning
`X-SIE-Served-By`	`local` for dedicated capacity, `remote` for anything else
`X-SIE-Upstream`	Who served a `remote` request, `sie-relay` for SIE Relay
`X-SIE-Fallback-Reason`	Why the request was relayed, for example `provisioning`
`X-SIE-Fallback-Error`	Error code of a failed relay attempt

provisioning means dedicated capacity was not ready, because it had scaled to zero or was still starting. A model or profile served only through SIE Relay sends no X-SIE-Fallback-Reason, because nothing else serves it. When a relay attempt fails, the response is the refusal from dedicated capacity, with X-SIE-Served-By: local and the code of the failed attempt in X-SIE-Fallback-Error.

The headers appear on successful responses and on server errors (5xx). Client errors (4xx) carry none of them. A relayed answer carries these headers:

HTTP/1.1 200 OK
X-SIE-Served-By: remote
X-SIE-Upstream: sie-relay
X-SIE-Fallback-Reason: provisioning

In the SDKs

The Python and TypeScript SDKs read these headers into the request metadata of each result: result["request"] in Python and result.request in TypeScript. A field is absent when the response did not carry its header. These fields and the remote client option need SDK version 0.10.0 or later.

Header	Python	TypeScript
`X-SIE-Served-By`	`served_by`	`servedBy`
`X-SIE-Upstream`	`upstream`	`upstream`
`X-SIE-Fallback-Reason`	`fallback_reason`	`fallbackReason`
`X-SIE-Fallback-Error`	`fallback_error`	`fallbackError`

In Python, RequestError and ServerError carry the same metadata in .request, read from the error response. In current SDK releases, ProvisioningError carries no request metadata, so read X-SIE-Fallback-Error from the raw response to see why a relay attempt failed. Streamed responses send the headers too, but the SDK streaming methods return chunks without them.

Python
TypeScript

import os

from sie_sdk import SIEClient

with SIEClient(
    "https://api.superlinked.com",
    api_key=os.environ["SIE_API_KEY"],
) as client:
    result = client.generate(
        "Qwen/Qwen3.5-4B",
        "Explain vector search in one sentence.",
        max_new_tokens=64,
    )

request = result.get("request", {})
print(request.get("served_by"))        # "local" or "remote"
print(request.get("upstream"))         # "sie-relay" when SIE Relay served it
print(request.get("fallback_reason"))  # "provisioning" when capacity was not ready

import { SIEClient } from "@superlinked/sie-sdk";

const client = new SIEClient("https://api.superlinked.com", {
  apiKey: process.env.SIE_API_KEY,
});
try {
  const result = await client.generate(
    "Qwen/Qwen3.5-4B",
    "Explain vector search in one sentence.",
    { maxNewTokens: 64 },
  );

  const { servedBy, upstream, fallbackReason } = result.request ?? {};
  console.log(servedBy); // "local" or "remote"
  console.log(upstream); // "sie-relay" when SIE Relay served it
  console.log(fallbackReason); // "provisioning" when capacity was not ready
} finally {
  await client.close();
}

Opt out of SIE Relay

A header opts out a single request or client. The organization setting opts out every request your organization makes.

Keep requests on dedicated capacity

Send the X-SIE-Remote: forbid header. Dedicated capacity then answers the request, including the 503 refusal while capacity starts. The header takes exactly one value. Any value other than forbid, or a second X-SIE-Remote header, is refused with 400. A model or profile served only through SIE Relay has no dedicated capacity, so a forbid request for it is refused with 400 as well. On /v1/encode, /v1/score and /v1/extract the error code is INVALID_INPUT. /v1/generate and the OpenAI-compatible endpoints answer with an OpenAI error object instead, whose code is invalid_request.

In the SDKs, the remote client option sends the header with every request the client makes. There is no per-call option, so create a second client for the requests that must stay on dedicated capacity. Both SDKs reject any value other than "forbid" when the client is created.

Python
TypeScript

import os

from sie_sdk import SIEAsyncClient, SIEClient

# Every request from these clients stays on dedicated capacity.
client = SIEClient(
    "https://api.superlinked.com",
    api_key=os.environ["SIE_API_KEY"],
    remote="forbid",
)
async_client = SIEAsyncClient(
    "https://api.superlinked.com",
    api_key=os.environ["SIE_API_KEY"],
    remote="forbid",
)

import { SIEClient } from "@superlinked/sie-sdk";

// Every request from this client stays on dedicated capacity.
const client = new SIEClient("https://api.superlinked.com", {
  apiKey: process.env.SIE_API_KEY,
  remote: "forbid",
});

Over plain HTTP, send the header yourself. -i prints the response headers:

curl -i \
  https://api.superlinked.com/v1/chat/completions \
  -H "Authorization: Bearer $SIE_API_KEY" \
  -H "Content-Type: application/json" \
  -H "X-SIE-Remote: forbid" \
  -d '{"model":"Qwen/Qwen3.5-4B","messages":[{"role":"user","content":"hello"}],"max_completion_tokens":16}'

Turn SIE Relay off for your organization

SIE Relay is on by default. Owners and admins can turn it off for the whole organization in the console, under Settings → SIE Relay. With SIE Relay off:

requests wait for dedicated capacity during a cold start, as described in Retry while capacity starts
a model or profile served only through SIE Relay is refused with 403 THIRD_PARTY_SERVING_DISABLED, although it still appears in the model list

Retry while capacity starts

When a request is not relayed and dedicated capacity is still starting, the gateway answers 503 with the error code PROVISIONING and a Retry-After header, in seconds. Wait that long and send the request again. Capacity keeps starting in the meantime. Keep retrying while the answer is 503 PROVISIONING.

On the native endpoints (/v1/encode, /v1/score, /v1/extract and /v1/generate), Retry-After estimates how long capacity still needs to start, from 5 to 300 seconds. It is 60 seconds when there is no estimate yet. On the OpenAI-compatible endpoints, such as /v1/chat/completions and /v1/embeddings, and on /v1/rerank, it is at most 60 seconds, so a long start can take several retries.

This refusal reaches requests that are never relayed, and requests whose relay attempt failed. After a failed attempt, the 503 also carries X-SIE-Fallback-Error.

Both SDKs retry for you by default. With capacity waiting on (wait_for_capacity=True in Python, waitForCapacity: true in TypeScript), the client honors Retry-After and retries until the provisioning budget runs out: provision_timeout_s in Python, 900 seconds by default, or provisionTimeout in TypeScript, 900000 milliseconds by default. With waiting off, or once the budget runs out, the client raises ProvisioningError. Its retry_after in Python (seconds) or retryAfter in TypeScript (milliseconds) holds the server’s hint.

Error codes

Code	HTTP status	Meaning
`PROVISIONING`	503	Dedicated capacity is starting and the request was not relayed. Retry after the `Retry-After` delay
`THIRD_PARTY_SERVING_DISABLED`	403	Your organization turned SIE Relay off, and the model or profile is served only through SIE Relay
`THIRD_PARTY_RUNTIME_UNLISTED`	403	The request needs a model or profile that SIE Relay does not offer
`THIRD_PARTY_REGION_UNAVAILABLE`	403	The model is not available through SIE Relay in the region of your API key
`INVALID_REQUEST`	400	A batch or job names a model or profile served only through SIE Relay
`INVALID_INPUT`	400	On `/v1/encode`, `/v1/score` or `/v1/extract`, `X-SIE-Remote` holds a value other than `forbid` or appears twice, or a `forbid` request asked for a model or profile served only through SIE Relay
`invalid_request`	400	The same `X-SIE-Remote` refusal from `/v1/generate` or an OpenAI-compatible endpoint, in an OpenAI error object

The SDKs raise the 403 and 400 refusals as RequestError, with the code in code (invalid_request for a 400 from /v1/generate or an OpenAI-compatible endpoint), and do not retry them.

Price

A request costs the same whether dedicated capacity or SIE Relay serves it. Both are billed at the model’s listed price. The usage export marks each row dedicated or relay in its served_by column.

Usage export

A relay row covers requests that SIE Relay answered, the ones whose responses carried X-SIE-Upstream: sie-relay. Everything else is dedicated, including requests served by remote capacity that SIE Cloud operates itself. The CSV from Usage & billing → Export CSV in the console includes the column.

Data handling

SIE Relay partners state that they keep no copy of the requests SIE Relay sends them. SIE Relay serves a model for an API key only in the regions where SIE Cloud offers that model through SIE Relay. Otherwise a request for a model with dedicated capacity waits for that capacity, and a request for a model served only through SIE Relay is refused with 403 THIRD_PARTY_REGION_UNAVAILABLE. An API key without a pinned region is checked against the region of the SIE Cloud deployment that receives the request.