Skip to content
Most AI tasks no longer need a frontier model Read launch post

Rate limits

SIE Serverless limits how fast each API key can send requests and how many it can have open at once. Requests over a limit get 429 Too Many Requests with a Retry-After header. Nothing runs for a refused request and nothing is billed, so retrying it is safe, generation included.

Each API key has its own limits. Your organization has a second, higher set shared by all its keys, so spreading traffic over more keys does not raise the total past the organization column.

EndpointsPer keyPer organization
Realtime: /v1/encode, /v1/score, /v1/extract, /v1/embeddings, /v1/audio/transcriptions50 requests/s, 32 in flight200 requests/s, 128 in flight
Generation: /v1/generate, /v1/chat/completions, /v1/completions, /v1/responses, /v1/messages5 requests/s, 4 in flight25 requests/s, 20 in flight
Batch jobs: /v1/jobs, /v1/batches, /v1/files2 requests/s, 2 in flight10 requests/s, 8 in flight

“In flight” counts requests that have been accepted and have not finished. A streamed generation stays in flight until its last event arrives. Short bursts above the per-second rate are absorbed; a sustained rate above it is not.

The limits are enforced across several gateway instances, each holding a share, so treat them as approximate. You can occasionally see a 429 slightly below a figure, or get a little above it. Model lists and other metadata reads (/v1/models for example) are not counted.

Generation also has a shared capacity limit. When it is full, a generation request gets the same 429 even if your key is under its own limits.

Every 429 carries Retry-After in whole seconds (at least 1) and X-SIE-Error-Code: RATE_LIMIT. Concurrency refusals use Retry-After: 1; rate refusals use the time until the next request would be admitted.

On the native endpoints the body is:

HTTP/1.1 429 Too Many Requests
Retry-After: 1
X-SIE-Error-Code: RATE_LIMIT
Content-Type: application/json
{"detail": {"code": "RATE_LIMIT", "message": "API key concurrent request limit exceeded."}}

/v1/messages uses the same body. On the OpenAI-compatible endpoints it is an OpenAI error object, which OpenAI client libraries already treat as a rate limit:

{
"error": {
"message": "API key request rate limit exceeded.",
"type": "rate_limit_error",
"param": null,
"code": "rate_limit_exceeded"
}
}

The message says which limit was hit: the key or the organization, the request rate or the in-flight count. Branch on the status and code, not the message text.

The Python and TypeScript SDKs retry a 429 for you. They wait the Retry-After delay and send the request again until the provisioning budget runs out (provision_timeout_s in Python, 900 seconds by default; provisionTimeout in TypeScript, 900000 milliseconds by default). Then they raise RateLimitError, which holds the last Retry-After in retry_after (Python, seconds) or retryAfter (TypeScript, milliseconds).

So an SDK caller over the limit sees slower calls, not errors. But a loop that keeps 200 requests open against a 32-request limit spends most of its time waiting.

Most 429s come from sending one item per request in a wide parallel loop. Two changes fix it.

Put many items in one call. /v1/encode, /v1/score and /v1/extract take a list of items, and one call counts as one request however many items it carries. 1,000 texts in 10 calls of 100 stay well under every limit; 1,000 parallel calls of one text each hit the 32 in-flight limit at once.

Cap your own concurrency below the in-flight limit. In Python, SIEAsyncClient takes max_concurrency, which holds extra calls in the client until a slot frees up. In TypeScript, or with your own HTTP client, use a semaphore or a worker pool of the same size.

import asyncio
import os
from sie_sdk import SIEAsyncClient
texts = [f"document {i}" for i in range(10_000)]
chunks = [texts[i : i + 100] for i in range(0, len(texts), 100)]
async def main():
# 16 calls in flight, half the per-key realtime limit.
async with SIEAsyncClient(
"https://api.superlinked.com",
api_key=os.environ["SIE_API_KEY"],
max_concurrency=16,
) as client:
results = await asyncio.gather(
*(client.encode("BAAI/bge-m3", [{"text": t} for t in chunk]) for chunk in chunks)
)
return [item for batch in results for item in batch]
embeddings = asyncio.run(main())

Generation is tighter: 4 in flight per key. An agent that runs tool calls in parallel should send generation requests one or two at a time per key, and batch the embedding or scoring calls around them.

Without the SDK, handle 429 the same way as a cold model’s 503: wait Retry-After seconds and retry. Retries over HTTP has a loop you can copy.

Talk to us

Tell us about your use case and we'll get back to you shortly.