---
title: Performance Evaluation
description: Measure latency and throughput against an SIE endpoint with response timing metadata, isolated resource pools, and the Grafana dashboards shipped in the Helm chart.
canonical_url: https://superlinked.com/docs/evals/performance
last_updated: 2026-08-25
---

SIE gives you two vantage points on performance: per-request timing that arrives with every encode response, and cluster-wide dashboards fed by OpenTelemetry. Between them you can tell whether time went to queueing, tokenization, or the GPU, and whether the cluster keeps up as load grows.

## Request Timing Metadata

Source: [packages/sie_server/src/sie_server/types/responses.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/types/responses.py)

Encode responses carry a server-side timing breakdown when the server tracked one, and the SDK attaches it to each result:

#### Python

```python
from sie_sdk import SIEClient

client = SIEClient("http://your-sie-endpoint:8080")
result = client.encode("BAAI/bge-m3", {"text": "Hello world"})
print(result["timing"])
```

#### TypeScript

```typescript
import { SIEClient } from "@superlinked/sie-sdk";

const client = new SIEClient("http://your-sie-endpoint:8080");
const result = await client.encode("BAAI/bge-m3", { text: "Hello world" });
console.log(result.timing);
```

| Field | Meaning |
| --- | --- |
| `total_ms` | Total end-to-end time |
| `queue_ms` | Time waiting in queue |
| `tokenization_ms` | Time tokenizing input |
| `inference_ms` | GPU forward pass time |
| `postprocessing_ms` | Postprocessor transforms time |

The table shows the wire field names. The Python SDK's typed `TimingInfo` covers `total_ms`, `queue_ms`, `tokenization_ms`, and `inference_ms`; `postprocessing_ms` is omitted from the wire when it is 0 and has no typed field in the SDK. The TypeScript SDK exposes a four-field camelCase subset: `totalMs`, `queueMs`, `tokenizationMs`, `inferenceMs`.

Score responses return scores and token usage without a timing block, so time reranking calls client-side.

## Server Time vs Wall Time

Client-side wall time adds network transfer and any SDK retries on top of `total_ms`. The SDK retries transparently while capacity provisions or a model cold-loads, so a single slow call is often a scale-up event rather than slow inference. In the Python SDK, check `client.last_retry_count` after a call to see how many retries it performed, and keep cold-start runs out of your steady-state numbers.

## Isolating a Benchmark With Pools

Source: [packages/sie_sdk/src/sie_sdk/client/sync.py](https://github.com/superlinked/sie/blob/main/packages/sie_sdk/src/sie_sdk/client/sync.py)

Latency measured while production traffic shares the GPUs is a measurement of contention, not of the model. Resource pools reserve exclusive capacity for the duration of a benchmark:

#### Python

```python
client.create_pool("eval", {"l4": 2})
result = client.encode("BAAI/bge-m3", {"text": "Hello"}, gpu="eval/l4")
client.delete_pool("eval")
```

#### TypeScript

```typescript
await client.createPool("eval", { l4: 2 });
const result = await client.encode("BAAI/bge-m3", { text: "Hello" }, { gpu: "eval/l4" });
await client.deletePool("eval");
```

## Load Patterns

The public repo does not ship a load-testing harness. Drive load with an asyncio script against the SDK, or any HTTP load generator you already use. Two patterns cover most questions:

- **Constant rate.** Hold a fixed request rate for several minutes and record latency percentiles. This is your steady-state number for capacity planning.
- **Ramp.** Increase the rate stepwise until p99 degrades. The point where `queue_ms` starts climbing while `inference_ms` stays flat is saturation; the fix at that point is more replicas, not a faster model.

Run each measurement long enough to outlast autoscaling, and separate results taken during scale-up from results at stable worker counts.

## Cluster Dashboards

The Helm chart ships seven Grafana dashboards in [`deploy/helm/sie-cluster/files/dashboards/`](https://github.com/superlinked/sie/tree/main/deploy/helm/sie-cluster/files/dashboards), provisioned automatically when the bundled monitoring stack is enabled:

- SIE Cluster Overview
- SIE Model Performance
- SIE Perf Tuning (L4)
- SIE Queue Routing
- SIE Worker Health
- SIE Tracing
- SIE Generation

For benchmark runs, SIE Model Performance is the one to watch: per-model QPS, p95 latency, average batch size, and a latency breakdown by operation. Setup and example PromQL queries are in [Monitoring](/docs/deployment/monitoring/).

## What's Next

- [Quality Evaluation](/docs/evals/quality/) - measure retrieval accuracy with NDCG@10
- [Monitoring & Observability](/docs/deployment/monitoring/) - the full metrics and dashboard stack
- [SDK Reference](/docs/reference/sdk/) - client options for timeouts and GPU routing