Retries over HTTP
The Python and TypeScript SDKs handle everything on this page for you: they wait for a cold model and retry rate-limited calls by default. Read on if you call SIE Serverless with plain HTTP, an OpenAI client library, or from an agent’s tool.
What a cold model returns
Section titled “What a cold model returns”A model with no traffic can scale to zero. The first request after that starts it, and until it is ready you get one of two 503 Service Unavailable answers. Both carry a Retry-After header in seconds.
PROVISIONING: the model’s capacity is starting. On the native endpoints:
HTTP/1.1 503 Service UnavailableRetry-After: 60X-SIE-Error-Code: PROVISIONINGContent-Type: application/json
{"error": {"code": "PROVISIONING", "message": "Capacity for model 'BAAI/bge-m3' is starting. Retry after the number of seconds in Retry-After."}}On the OpenAI-compatible endpoints the same refusal is an OpenAI error object with "type": "server_error" and "code": "provisioning". The X-SIE-Error-Code: PROVISIONING header is there on both. Cold starts and SIE Relay lists the Retry-After range per endpoint and which requests SIE Relay can answer instead.
MODEL_LOADING: capacity is up and the model is loading onto it. This one comes with a short Retry-After, usually a few seconds:
{"detail": {"code": "MODEL_LOADING", "message": "Local model is loading"}}While capacity starts, the gateway can hold a request for a while before it answers. A cold request can therefore take much longer than a warm one and then either succeed or return the 503. Give each attempt a generous timeout; the SDKs wait up to 150 seconds for a response.
What a cold start looks like
Section titled “What a cold start looks like”A cold start looks like an outage to a client that gives up early. Over plain HTTP, a request to a model that is starting goes like this:
- The first attempts answer
503withX-SIE-Error-Code: PROVISIONINGand aRetry-Afterheader. Each answer can take a while to arrive, because the gateway can hold the request first. - You wait the number of seconds in
Retry-Afterand send the same request again. Several503answers in a row are normal. - One attempt is held until the model is ready and then answers
200 OK. That attempt takes much longer than a warm call.
So set each attempt’s timeout to at least 150 seconds. A client with a 30 or 60 second timeout gives up on the attempt that would have succeeded, and starts over. And keep retrying for several minutes, not several seconds.
Which responses to retry
Section titled “Which responses to retry”Retry when the status is 429 or 503 and the response has a Retry-After header. Wait that many seconds, then send the same request again. Keep going until you get another status or your own deadline passes.
| Status | Code | Meaning |
|---|---|---|
503 | PROVISIONING | The model’s capacity is starting |
503 | MODEL_LOADING | The model is loading onto running capacity |
503 | KEY_STATE_UNAVAILABLE | The gateway has not confirmed the API key yet, for example seconds after you created it |
429 | RATE_LIMIT | Over a rate limit |
Without an X-SIE-Fallback-Error header, these responses mean the request did not run: SIE refused it before any model worked on it, so a retry never runs it twice and is safe for generation too.
A 503 that does carry X-SIE-Fallback-Error is different. It comes after a failed SIE Relay attempt, and that attempt may have run the request upstream before it failed. Retrying encode, score or extract calls is still fine, since running them again gives the same result. For a POST that generates output, resend it only when a duplicate run is acceptable.
Do not retry other 4xx errors, and do not retry 502 MODEL_LOAD_FAILED: the model cannot load and the answer will not change.
A cold start can take minutes. The SDKs give up after 900 seconds; a deadline of 10 to 15 minutes is a reasonable default for a batch script, and much shorter is fine for an interactive app that can show “warming up”.
A retry loop without the SDK
Section titled “A retry loop without the SDK”url="https://api.superlinked.com/v1/encode/BAAI/bge-m3"body='{"items": [{"text": "hello"}]}'deadline=$(( $(date +%s) + 900 ))
while :; do code=$(curl -sS -o response.json -D headers.txt -w '%{http_code}' --max-time 150 \ -X POST "$url" \ -H "Authorization: Bearer $SIE_API_KEY" \ -H "Content-Type: application/json" \ -d "$body") wait=$(grep -i '^retry-after:' headers.txt | tr -dc '0-9') [[ ( $code == 429 || $code == 503 ) && -n $wait ]] || break (( $(date +%s) + wait < deadline )) || break echo "HTTP $code, retrying in ${wait}s" >&2 sleep "$wait" # Check again before the next attempt, in case the sleep ran long. (( $(date +%s) < deadline )) || breakdoneecho "HTTP $code"cat response.jsonimport osimport time
import requests
RETRYABLE = {429, 503}
def post_with_retry(url, payload, deadline_s=900): deadline = time.monotonic() + deadline_s while True: resp = requests.post( url, json=payload, headers={"Authorization": f"Bearer {os.environ['SIE_API_KEY']}"}, timeout=(10, 150), ) retry_after = resp.headers.get("Retry-After", "") if resp.status_code not in RETRYABLE or not retry_after.isdigit(): break wait = int(retry_after) if time.monotonic() + wait >= deadline: break print(f"HTTP {resp.status_code} {resp.headers.get('X-SIE-Error-Code')}, retrying in {wait}s") time.sleep(wait) # Check again before the next attempt, in case the sleep ran long. if time.monotonic() >= deadline: break resp.raise_for_status() return resp.json()
result = post_with_retry( "https://api.superlinked.com/v1/encode/BAAI/bge-m3", {"items": [{"text": "hello"}]},)The deadline limits when a new attempt can start, not the total run time. Both loops check it before every attempt, including after a sleep, and send nothing once it has passed. An attempt that starts just before the deadline still runs until it finishes or times out. In curl, --max-time 150 caps the whole attempt at 150 seconds. In Python, timeout=(10, 150) caps the connect at 10 seconds and each wait for data at 150 seconds, so it does not cap the attempt’s total time.
OpenAI client libraries already retry 429 and 503 and read Retry-After, but only a couple of times by default. Raise their retry count (max_retries in the OpenAI Python SDK) or wrap the call in a loop like the one above when a cold start is likely.
A retry loop for chat completions
Section titled “A retry loop for chat completions”These loops call /v1/chat/completions. They retry 429 and 503 with Retry-After like the loop above, print X-SIE-Error-Code on every retry, and stop on any other error. The status and headers decide the retry, so a response body that fails to read does not stop it, and the deadline is checked again after each wait. Two differences from the loop above: they wait 5 seconds when Retry-After is not a whole number of seconds, and they stop at a 503 that carries X-SIE-Fallback-Error, since that generation may already have run. Pass resend_after_relay_failure=True in Python or true as the second argument in TypeScript if a duplicate run is acceptable.
The Python version uses only the standard library; its timeout caps each wait for data, as in the requests loop. The TypeScript version needs fetch and AbortSignal.timeout (Node 18 or later, Bun or Deno), and its timeout caps the whole attempt. It uses top-level await, so save it as an ES module such as chat.mts and run it with bun chat.mts, npx tsx chat.mts, or node chat.mts on a Node version that runs TypeScript directly.
import jsonimport osimport timeimport urllib.errorimport urllib.request
URL = "https://api.superlinked.com/v1/chat/completions"DEADLINE_S = 900 # stop starting new attempts after 15 minutesTIMEOUT_S = 150 # per attempt: a request to a starting model can be held this longDEFAULT_WAIT_S = 5 # if Retry-After is not a whole number of seconds
def chat(payload, resend_after_relay_failure=False): body = json.dumps(payload).encode() deadline = time.monotonic() + DEADLINE_S while True: request = urllib.request.Request( URL, data=body, method="POST", headers={ "Authorization": f"Bearer {os.environ['SIE_API_KEY']}", "Content-Type": "application/json", }, ) try: with urllib.request.urlopen(request, timeout=TIMEOUT_S) as resp: return json.load(resp) except urllib.error.HTTPError as err: status, headers = err.code, err.headers try: detail = err.read().decode() except Exception: # the status and headers still decide the retry detail = ""
code = headers.get("X-SIE-Error-Code") retry_after = headers.get("Retry-After") retryable = status in (429, 503) and retry_after is not None # After a failed SIE Relay attempt the request may already have run. if headers.get("X-SIE-Fallback-Error") and not resend_after_relay_failure: retryable = False if not retryable: raise RuntimeError(f"HTTP {status} {code}: {detail}")
digits = retry_after.isascii() and retry_after.isdigit() wait = int(retry_after) if digits else DEFAULT_WAIT_S if time.monotonic() + wait >= deadline: raise RuntimeError(f"Gave up at the deadline: HTTP {status} {code}") print(f"HTTP {status} {code}, retrying in {wait}s") time.sleep(wait) if time.monotonic() >= deadline: # the sleep can run long raise RuntimeError(f"Gave up at the deadline: HTTP {status} {code}")
reply = chat({ "model": "Qwen/Qwen3.5-4B", "messages": [{"role": "user", "content": "Say hello in five words."}], "max_tokens": 64,})print(reply["choices"][0]["message"]["content"])const URL = "https://api.superlinked.com/v1/chat/completions";const DEADLINE_MS = 15 * 60_000; // stop starting new attempts after 15 minutesconst TIMEOUT_MS = 150_000; // per attempt: a request to a starting model can be held this longconst DEFAULT_WAIT_S = 5; // if Retry-After is not a whole number of seconds
async function chat(payload: object, resendAfterRelayFailure = false): Promise<any> { const body = JSON.stringify(payload); const deadline = performance.now() + DEADLINE_MS; while (true) { const resp = await fetch(URL, { method: "POST", headers: { Authorization: `Bearer ${process.env.SIE_API_KEY}`, "Content-Type": "application/json", }, body, signal: AbortSignal.timeout(TIMEOUT_MS), }); if (resp.ok) return resp.json();
const code = resp.headers.get("X-SIE-Error-Code"); const retryAfter = resp.headers.get("Retry-After"); // null when absent let retryable = (resp.status === 429 || resp.status === 503) && retryAfter !== null; // After a failed SIE Relay attempt the request may already have run. if (resp.headers.get("X-SIE-Fallback-Error") && !resendAfterRelayFailure) { retryable = false; } // The status and headers still decide the retry if the body fails to read. const detail = await resp.text().catch(() => ""); if (!retryable) throw new Error(`HTTP ${resp.status} ${code}: ${detail}`);
const wait = /^\d+$/.test(retryAfter ?? "") ? Number(retryAfter) : DEFAULT_WAIT_S; if (performance.now() + wait * 1000 >= deadline) { throw new Error(`Gave up at the deadline: HTTP ${resp.status} ${code}`); } console.log(`HTTP ${resp.status} ${code}, retrying in ${wait}s`); await new Promise((resolve) => setTimeout(resolve, wait * 1000)); if (performance.now() >= deadline) { // the sleep can run long throw new Error(`Gave up at the deadline: HTTP ${resp.status} ${code}`); } }}
const reply = await chat({ model: "Qwen/Qwen3.5-4B", messages: [{ role: "user", content: "Say hello in five words." }], max_tokens: 64,});console.log(reply.choices[0].message.content);For agent tools
Section titled “For agent tools”Agents that call SIE Serverless through a generic HTTP tool often treat the first 503 as a failure. Paste this into the tool’s description or the system prompt:
If SIE returns HTTP 503 or 429 with a
Retry-Afterheader, the model is starting or you sent too many requests. Unless the response also has anX-SIE-Fallback-Errorheader, the request did not run. Wait the number of seconds inRetry-After, then send the identical request again. IfX-SIE-Fallback-Erroris present and the request generates text, the first attempt may have run, so resend it only when a second run is acceptable. Repeat until you get a different status, for up to 15 minutes. Do not change the model or the request, and do not report a failure while these retries are still going. Send several items in one call instead of one call per item.
If your agent framework can run Python, give it the SDK instead; it does all of this by default.
Using a coding agent?
Section titled “Using a coding agent?”When a coding agent writes the client code for you, paste these lines into its instructions (AGENTS.md, CLAUDE.md or your tool’s equivalent) so the code it writes waits out a cold start:
SIE Serverless returns HTTP 503 with a Retry-After header (in seconds) while a model starts.Wait that many seconds and send the same request again. Keep retrying for up to 15 minutes;a cold start can take minutes, so several 503s in a row are expected.Set the HTTP client timeout to at least 150 seconds, because a request to a starting modelcan be held that long before it answers.HTTP 429 means too many requests: slow down, wait Retry-After seconds, then retry.Retry a 503 or 429 only when it has a Retry-After header; without one, treat it as an error.If Retry-After is present but is not a whole number of seconds, wait 5 seconds.Do not retry other 4xx errors. If a 503 also has an X-SIE-Fallback-Error header, do notresend a generation request unless a duplicate run is acceptable.Retry code and details: https://superlinked.com/docs/cloud/retries/