Rate limits
SIE Serverless limits how fast each API key can send requests and how many it can have open at once. Requests over a limit get 429 Too Many Requests with a Retry-After header. Nothing runs for a refused request and nothing is billed, so retrying it is safe, generation included.
Limits
Section titled “Limits”Each API key has its own limits. Your organization has a second, higher set shared by all its keys, so spreading traffic over more keys does not raise the total past the organization column.
| Endpoints | Per key | Per organization |
|---|---|---|
Realtime: /v1/encode, /v1/score, /v1/extract, /v1/embeddings, /v1/audio/transcriptions | 50 requests/s, 32 in flight | 200 requests/s, 128 in flight |
Generation: /v1/generate, /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages | 5 requests/s, 4 in flight | 25 requests/s, 20 in flight |
Batch jobs: /v1/jobs, /v1/batches, /v1/files | 2 requests/s, 2 in flight | 10 requests/s, 8 in flight |
“In flight” counts requests that have been accepted and have not finished. A streamed generation stays in flight until its last event arrives. Short bursts above the per-second rate are absorbed; a sustained rate above it is not.
The limits are enforced across several gateway instances, each holding a share, so treat them as approximate. You can occasionally see a 429 slightly below a figure, or get a little above it. Model lists and other metadata reads (/v1/models for example) are not counted.
Generation also has a shared capacity limit. When it is full, a generation request gets the same 429 even if your key is under its own limits.
The 429 response
Section titled “The 429 response”Every 429 carries Retry-After in whole seconds (at least 1) and X-SIE-Error-Code: RATE_LIMIT. Concurrency refusals use Retry-After: 1; rate refusals use the time until the next request would be admitted.
On the native endpoints the body is:
HTTP/1.1 429 Too Many RequestsRetry-After: 1X-SIE-Error-Code: RATE_LIMITContent-Type: application/json
{"detail": {"code": "RATE_LIMIT", "message": "API key concurrent request limit exceeded."}}/v1/messages uses the same body. On the OpenAI-compatible endpoints it is an OpenAI error object, which OpenAI client libraries already treat as a rate limit:
{ "error": { "message": "API key request rate limit exceeded.", "type": "rate_limit_error", "param": null, "code": "rate_limit_exceeded" }}The message says which limit was hit: the key or the organization, the request rate or the in-flight count. Branch on the status and code, not the message text.
What the SDKs do
Section titled “What the SDKs do”The Python and TypeScript SDKs retry a 429 for you. They wait the Retry-After delay and send the request again until the provisioning budget runs out (provision_timeout_s in Python, 900 seconds by default; provisionTimeout in TypeScript, 900000 milliseconds by default). Then they raise RateLimitError, which holds the last Retry-After in retry_after (Python, seconds) or retryAfter (TypeScript, milliseconds).
So an SDK caller over the limit sees slower calls, not errors. But a loop that keeps 200 requests open against a 32-request limit spends most of its time waiting.
Fan-out workloads
Section titled “Fan-out workloads”Most 429s come from sending one item per request in a wide parallel loop. Two changes fix it.
Put many items in one call. /v1/encode, /v1/score and /v1/extract take a list of items, and one call counts as one request however many items it carries. 1,000 texts in 10 calls of 100 stay well under every limit; 1,000 parallel calls of one text each hit the 32 in-flight limit at once.
Cap your own concurrency below the in-flight limit. In Python, SIEAsyncClient takes max_concurrency, which holds extra calls in the client until a slot frees up. In TypeScript, or with your own HTTP client, use a semaphore or a worker pool of the same size.
import asyncioimport os
from sie_sdk import SIEAsyncClient
texts = [f"document {i}" for i in range(10_000)]chunks = [texts[i : i + 100] for i in range(0, len(texts), 100)]
async def main(): # 16 calls in flight, half the per-key realtime limit. async with SIEAsyncClient( "https://api.superlinked.com", api_key=os.environ["SIE_API_KEY"], max_concurrency=16, ) as client: results = await asyncio.gather( *(client.encode("BAAI/bge-m3", [{"text": t} for t in chunk]) for chunk in chunks) ) return [item for batch in results for item in batch]
embeddings = asyncio.run(main())import { SIEClient } from "@superlinked/sie-sdk";
const client = new SIEClient("https://api.superlinked.com", { apiKey: process.env.SIE_API_KEY,});
const texts = Array.from({ length: 10_000 }, (_, i) => `document ${i}`);const chunks: string[][] = [];for (let i = 0; i < texts.length; i += 100) chunks.push(texts.slice(i, i + 100));
// 16 workers, each sending one chunk at a time.const results: unknown[][] = new Array(chunks.length);let next = 0;async function worker() { while (next < chunks.length) { const i = next++; results[i] = await client.encode( "BAAI/bge-m3", chunks[i].map((text) => ({ text })), ); }}try { await Promise.all(Array.from({ length: 16 }, worker));} finally { await client.close();}Generation is tighter: 4 in flight per key. An agent that runs tool calls in parallel should send generation requests one or two at a time per key, and batch the embedding or scoring calls around them.
Without the SDK, handle 429 the same way as a cold model’s 503: wait Retry-After seconds and retry. Retries over HTTP has a loop you can copy.