---
title: Rate limits
description: Per-key and per-organization request limits on SIE Serverless, the 429 response above them, and how to fan out work without hitting them.
canonical_url: https://superlinked.com/docs/cloud/rate-limits
last_updated: 2026-11-17
---

SIE Serverless limits how fast each API key can send requests and how many it can have open at once. Requests over a limit get `429 Too Many Requests` with a `Retry-After` header. Nothing runs for a refused request and nothing is billed, so retrying it is safe, generation included.

## Limits

Each API key has its own limits. Your organization has a second, higher set shared by all its keys, so spreading traffic over more keys does not raise the total past the organization column.

| Endpoints | Per key | Per organization |
| --- | --- | --- |
| Realtime: `/v1/encode`, `/v1/score`, `/v1/extract`, `/v1/embeddings`, `/v1/audio/transcriptions` | 50 requests/s, 32 in flight | 200 requests/s, 128 in flight |
| Generation: `/v1/generate`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`, `/v1/messages` | 5 requests/s, 4 in flight | 25 requests/s, 20 in flight |
| Batch jobs: `/v1/jobs`, `/v1/batches`, `/v1/files` | 2 requests/s, 2 in flight | 10 requests/s, 8 in flight |

"In flight" counts requests that have been accepted and have not finished. A streamed generation stays in flight until its last event arrives. Short bursts above the per-second rate are absorbed; a sustained rate above it is not.

The limits are enforced across several gateway instances, each holding a share, so treat them as approximate. You can occasionally see a `429` slightly below a figure, or get a little above it. Model lists and other metadata reads (`/v1/models` for example) are not counted.

Generation also has a shared capacity limit. When it is full, a generation request gets the same `429` even if your key is under its own limits.

## The 429 response

Every `429` carries `Retry-After` in whole seconds (at least 1) and `X-SIE-Error-Code: RATE_LIMIT`. Concurrency refusals use `Retry-After: 1`; rate refusals use the time until the next request would be admitted.

On the native endpoints the body is:

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 1
X-SIE-Error-Code: RATE_LIMIT
Content-Type: application/json

{"detail": {"code": "RATE_LIMIT", "message": "API key concurrent request limit exceeded."}}
```

`/v1/messages` uses the same body. On the OpenAI-compatible endpoints it is an OpenAI error object, which OpenAI client libraries already treat as a rate limit:

```json
{
  "error": {
    "message": "API key request rate limit exceeded.",
    "type": "rate_limit_error",
    "param": null,
    "code": "rate_limit_exceeded"
  }
}
```

The `message` says which limit was hit: the key or the organization, the request rate or the in-flight count. Branch on the status and code, not the message text.

## What the SDKs do

The Python and TypeScript SDKs retry a `429` for you. They wait the `Retry-After` delay and send the request again until the provisioning budget runs out (`provision_timeout_s` in Python, 900 seconds by default; `provisionTimeout` in TypeScript, 900000 milliseconds by default). Then they raise `RateLimitError`, which holds the last `Retry-After` in `retry_after` (Python, seconds) or `retryAfter` (TypeScript, milliseconds).

So an SDK caller over the limit sees slower calls, not errors. But a loop that keeps 200 requests open against a 32-request limit spends most of its time waiting.

## Fan-out workloads

Most `429`s come from sending one item per request in a wide parallel loop. Two changes fix it.

**Put many items in one call.** `/v1/encode`, `/v1/score` and `/v1/extract` take a list of items, and one call counts as one request however many items it carries. 1,000 texts in 10 calls of 100 stay well under every limit; 1,000 parallel calls of one text each hit the 32 in-flight limit at once.

**Cap your own concurrency below the in-flight limit.** In Python, `SIEAsyncClient` takes `max_concurrency`, which holds extra calls in the client until a slot frees up. In TypeScript, or with your own HTTP client, use a semaphore or a worker pool of the same size.

#### Python

```python
import asyncio
import os

from sie_sdk import SIEAsyncClient

texts = [f"document {i}" for i in range(10_000)]
chunks = [texts[i : i + 100] for i in range(0, len(texts), 100)]

async def main():
    # 16 calls in flight, half the per-key realtime limit.
    async with SIEAsyncClient(
        "https://api.superlinked.com",
        api_key=os.environ["SIE_API_KEY"],
        max_concurrency=16,
    ) as client:
        results = await asyncio.gather(
            *(client.encode("BAAI/bge-m3", [{"text": t} for t in chunk]) for chunk in chunks)
        )
    return [item for batch in results for item in batch]

embeddings = asyncio.run(main())
```

#### TypeScript

```typescript
import { SIEClient } from "@superlinked/sie-sdk";

const client = new SIEClient("https://api.superlinked.com", {
  apiKey: process.env.SIE_API_KEY,
});

const texts = Array.from({ length: 10_000 }, (_, i) => `document ${i}`);
const chunks: string[][] = [];
for (let i = 0; i < texts.length; i += 100) chunks.push(texts.slice(i, i + 100));

// 16 workers, each sending one chunk at a time.
const results: unknown[][] = new Array(chunks.length);
let next = 0;
async function worker() {
  while (next < chunks.length) {
    const i = next++;
    results[i] = await client.encode(
      "BAAI/bge-m3",
      chunks[i].map((text) => ({ text })),
    );
  }
}
try {
  await Promise.all(Array.from({ length: 16 }, worker));
} finally {
  await client.close();
}
```

Generation is tighter: 4 in flight per key. An agent that runs tool calls in parallel should send generation requests one or two at a time per key, and batch the embedding or scoring calls around them.

Without the SDK, handle `429` the same way as a cold model's `503`: wait `Retry-After` seconds and retry. [Retries over HTTP](/docs/cloud/retries/) has a loop you can copy.
