---
title: Monitoring & Observability
description: Health checks, Prometheus metrics, real-time TUI, and observability for SIE servers.
canonical_url: https://superlinked.com/docs/deployment/monitoring
last_updated: 2026-05-20
---

SIE provides multiple monitoring interfaces. Use health endpoints for container orchestration. Expose Prometheus metrics for alerting. Stream real-time status via WebSocket. Monitor interactively with the TUI.

## Health Endpoints

Source: [packages/sie_server/src/sie_server/api/health.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/api/health.py)

SIE exposes Kubernetes-compatible health probes for liveness and readiness checks.

### Liveness Probe

```bash
curl http://localhost:8080/healthz
# Returns: ok
```

The `/healthz` endpoint returns `200 OK` if the server process is alive. Use this for Kubernetes liveness probes. A failed check triggers container restart.

### Readiness Probe

```bash
curl http://localhost:8080/readyz
# Returns: ok
```

The `/readyz` endpoint returns `200 OK` if the server is ready to accept traffic. Use this for Kubernetes readiness probes. A failed check removes the pod from service endpoints.

**Kubernetes configuration:**

```yaml
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
```

## Prometheus Metrics

Source: [packages/sie_server/src/sie_server/observability/metrics.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/observability/metrics.py)

SIE exposes Prometheus-format metrics at `/metrics`. All metrics use the `sie_` prefix.

### Metrics Reference

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `sie_requests_total` | Counter | `model`, `endpoint`, `status` | Total requests processed |
| `sie_request_duration_seconds` | Histogram | `model`, `endpoint`, `phase` | Request duration breakdown |
| `sie_batch_size` | Histogram | `model` | Items per batch |
| `sie_tokens_processed_total` | Counter | `model` | Total tokens processed |
| `sie_queue_depth` | Gauge | `model` | Current pending items in queue |
| `sie_model_loaded` | Gauge | `model`, `device` | Model load state (1=loaded, 0=not) |
| `sie_model_memory_bytes` | Gauge | `model`, `device` | GPU memory usage per model |

### Duration Phases

The `sie_request_duration_seconds` histogram tracks latency by phase:

| Phase | Description |
|-------|-------------|
| `total` | End-to-end request latency |
| `queue` | Time spent waiting in the request queue |
| `tokenize` | Tokenization and preprocessing time |
| `inference` | GPU inference time |

### Histogram Buckets

**Duration buckets (seconds):** 0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0

**Batch size buckets:** 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024

### Scrape Configuration

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'sie'
    static_configs:
      - targets: ['localhost:8080']
    metrics_path: /metrics
    scrape_interval: 15s
```

## sie-top TUI

> **Caution — Work in progress:**
>
> `sie-admin` and `sie-top` are not yet released. The CLI and package name may change before general availability.

Source: [packages/sie_admin/src/sie_admin/top/app.py](https://github.com/superlinked/sie/blob/main/packages/sie_admin/src/sie_admin/top/app.py)

The `sie-top` command provides a real-time terminal interface for monitoring SIE servers.

### Installation

```bash
pip install 'sie-admin[top]'
```

### Usage

```bash
# Monitor local server (auto-detects mode)
sie-top

# Monitor specific server
sie-top localhost:8080

# Force worker mode (connect to individual worker)
sie-top --worker worker-0.sie.svc:8080

# Force cluster mode (connect to gateway)
sie-top --cluster gateway.example.com:8080
```

Mode is auto-detected by probing the gateway `/health` endpoint. Use `--worker` or `--cluster` to force a specific mode.

### Features

The TUI displays:

- **Server info:** Version, uptime, user, PID
- **GPU table:** Device name, memory usage, compute utilization, trend sparkline
- **Model table:** Name, state, device, memory, queue depth, QPS sparkline
- **Detail panel:** Selected GPU or model with 60-second history charts

**Keyboard shortcuts:**

| Key | Action |
|-----|--------|
| `j` / `Down` | Move selection down |
| `k` / `Up` | Move selection up |
| `?` | Show help |
| `q` | Quit |

## WebSocket Status

Source: [packages/sie_server/src/sie_server/api/ws.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/api/ws.py)

SIE streams real-time status over WebSocket at `/ws/status`. Updates push every 200ms.

### Connection

```python
import asyncio
import websockets
import json

async def monitor():
    async with websockets.connect("ws://localhost:8080/ws/status") as ws:
        async for message in ws:
            status = json.loads(message)
            print(f"Loaded models: {status['loaded_models']}")
            print(f"GPU type: {status['gpu']}")
```

### Status Message Format

```json
{
  "timestamp": 1703001234.567,
  "gpu": "l4",
  "loaded_models": ["bge-m3", "e5-base-v2"],
  "server": {
    "version": "0.1.0",
    "uptime_seconds": 3600,
    "user": "sie",
    "working_dir": "/app",
    "pid": 1
  },
  "gpus": [
    {
      "device": "cuda:0",
      "name": "NVIDIA L4",
      "gpu_type": "l4",
      "utilization_pct": 45,
      "memory_used_bytes": 8589934592,
      "memory_total_bytes": 23622320128,
      "memory_threshold_pct": 85
    }
  ],
  "models": [
    {
      "name": "bge-m3",
      "state": "loaded",
      "device": "cuda:0",
      "memory_bytes": 2147483648,
      "queue_depth": 0,
      "queue_pending_items": 0,
      "config": {
        "hf_id": "BAAI/bge-m3",
        "adapter": "bge_m3",
        "inputs": ["text"],
        "outputs": ["dense", "sparse"]
      }
    }
  ],
  "counters": {},
  "histograms": {}
}
```

### Model States

| State | Description |
|-------|-------------|
| `available` | Config loaded, weights not in memory |
| `loading` | Weights currently loading to GPU |
| `loaded` | Ready for inference |
| `unloading` | Weights being evicted from GPU |

## Grafana Dashboards

SIE includes pre-built Grafana dashboards in the Helm chart at [`deploy/helm/sie-cluster/files/dashboards/`](https://github.com/superlinked/sie/tree/main/deploy/helm/sie-cluster/files/dashboards). These are automatically provisioned when deploying with Grafana's sidecar.

Example queries for common panels:

### Request Rate

```txt
sum(rate(sie_requests_total{status="success"}[5m])) by (model)
```

### P99 Latency

```txt
histogram_quantile(0.99,
  sum(rate(sie_request_duration_seconds_bucket{phase="total"}[5m])) by (le, model)
)
```

### GPU Memory Usage

```txt
sum(sie_model_memory_bytes) by (device)
```

### Queue Depth

```txt
sum(sie_queue_depth) by (model)
```

### Batch Efficiency

```txt
histogram_quantile(0.5,
  sum(rate(sie_batch_size_bucket[5m])) by (le, model)
)
```

## Alert Rules

Source: [deploy/helm/sie-cluster/files/alerts/sie-rules.yaml](https://github.com/superlinked/sie/blob/main/deploy/helm/sie-cluster/files/alerts/sie-rules.yaml)

The `sie-cluster` chart can render pre-configured Prometheus alert rules:

| Alert | Severity | Condition | Description |
|-------|----------|-----------|-------------|
| `SIEWorkerDown` | critical | Worker scrape target down for 2 min | A worker pod is unreachable |
| `SIENoHealthyWorkers` | critical | No worker scrape targets healthy for 1 min | All workers are down or unreachable |
| `SIEWorkerHighQueueDepth` | warning | Queue depth > 50 for 5 min | Workers are overwhelmed, consider scaling up |
| `SIEGPUMemoryHigh` | warning | GPU memory > 90% for 5 min | Risk of OOM, LRU eviction may be insufficient |
| `SIEGPUTemperatureHigh` | warning | GPU temp > 80°C for 5 min | GPU throttling likely, check cooling |
| `SIEGPUECCErrors` | critical | Double-bit ECC errors increase over 1h | Hardware issue likely |
| `SIEGatewayDown` | critical | Gateway scrape target down for 1 min | Traffic cannot be routed |
| `SIEHighErrorRate` | warning | Gateway 5xx rate > 5% for 5 min | Server or model errors spiking |
| `SIEHighLatency` | warning | p95 latency > 5s for 5 min | Request latency is elevated |
| `SIEConfigDown` | critical | Config scrape target down for 2 min | Config writes are blocked; gateways serve cached state |
| `SIEProvisioningStuck` | warning | Pod Pending for 10 min | Check scheduling events and GPU capacity |
| `SIEScaleUpFailed` | warning | FailedScheduling event in 10 min | Likely insufficient GPU capacity |

### Installing Alert Rules

Alert rules are included in the `sie-cluster` chart when kube-prometheus-stack is installed or `alertRules.enabled` is true:

```bash
helm upgrade --install sie oci://ghcr.io/superlinked/charts/sie-cluster \
  --version 0.1.10 \
  -n sie \
  -f helm-values.yaml \
  --set alertRules.enabled=true
```

### Custom Alerts

Add custom alerts to your Prometheus configuration:

```yaml
# Alert when P99 latency exceeds 5 seconds
- alert: SIEHighLatency
  expr: |
    histogram_quantile(0.99,
      sum(rate(sie_request_duration_seconds_bucket{phase="total"}[5m])) by (le, model)
    ) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High P99 latency for model {{ $labels.model }}"
```

---

## Logging

Source: [packages/sie_server/src/sie_server/core/logging.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/core/logging.py)

SIE supports both human-readable and structured JSON logging.

### Log Levels

Enable verbose logging with `--verbose` or `-v`:

```bash
sie-server serve --verbose
```

### JSON Logging

Enable JSON format for Loki and log aggregation systems:

```bash
sie-server serve --json-logs
```

Or via environment variable:

```bash
export SIE_LOG_JSON=true
sie-server serve
```

### JSON Log Format

```json
{
  "timestamp": "2025-12-18T10:30:00.123Z",
  "level": "INFO",
  "logger": "sie_server.api.encode",
  "message": "Inference completed",
  "model": "bge-m3",
  "request_id": "abc123",
  "trace_id": "def456",
  "latency_ms": 45.2,
  "batch_size": 16,
  "gpu_type": "l4"
}
```

### Structured Fields

JSON logs include optional fields when available:

| Field | Description |
|-------|-------------|
| `model` | Model name for the request |
| `request_id` | Unique request identifier |
| `trace_id` | OpenTelemetry trace ID |
| `latency_ms` | Request latency in milliseconds |
| `batch_size` | Number of items in the batch |
| `gpu_type` | Detected GPU type |

## What's Next

- [Scale-from-Zero](/docs/deployment/autoscaling/) - autoscaling lifecycle and troubleshooting
- [Troubleshooting](/docs/reference/troubleshooting/) - common issues and solutions
- [CLI Reference](/docs/reference/cli/) for all server options
- [API Reference](/docs/reference/api/) for endpoint documentation