---
title: Monitoring & Observability
description: Health checks, OpenTelemetry metrics, collector-backed dashboards, and observability for SIE servers.
canonical_url: https://superlinked.com/docs/deployment/monitoring
last_updated: 2026-08-25
---

SIE exposes monitoring across the control plane and worker pods. Inside each Kubernetes worker pod, the SIE server sidecar owns queue health; the Python `sie-server` adapter owns model execution. Use health endpoints for orchestration. Application metrics leave each process once over OTLP, then the bundled OpenTelemetry Collector routes them to Prometheus and Grafana, with optional fan-out to a remote observability backend. WebSocket streams provide interactive status.

> **Note — Available in SIE 0.6.21:**
>
> The collector-backed metrics contract ships with SIE 0.6.21.

## Health Endpoints

Source: [packages/sie_gateway/src/handlers/health.rs](https://github.com/superlinked/sie/blob/main/packages/sie_gateway/src/handlers/health.rs)

Source: [packages/sie_server_sidecar/src/readiness.rs](https://github.com/superlinked/sie/blob/main/packages/sie_server_sidecar/src/readiness.rs)

Source: [packages/sie_server/src/sie_server/api/health.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/api/health.py)

Source: [packages/sie_config/src/sie_config/health.py](https://github.com/superlinked/sie/blob/main/packages/sie_config/src/sie_config/health.py)

SIE exposes Kubernetes-compatible health probes for liveness and readiness checks. In Docker, the Python `sie-server` process owns these endpoints. In Kubernetes, the gateway, config service, and both containers inside each worker pod have their own health contract.

| Component | `/healthz` | `/readyz` |
|-----------|------------|-----------|
| `sie-gateway` | Process liveness, returns `ok` | Process readiness. It does not wait for SIE server sidecar health or `sie-config` |
| SIE server sidecar (`worker-sidecar` container) | Process liveness | Fresh IPC `Ping` to the in-pod Python process and no active drain |
| `sie-server` | Python process liveness | Adapter process ready to receive work |
| `sie-config` | Config process liveness | Registry initialized and able to serve config endpoints |

### Liveness

```bash
curl http://localhost:8080/healthz
# Returns: ok
```

Use `/healthz` for Kubernetes liveness probes. A failed check triggers container restart.

### Readiness

```bash
curl http://localhost:8080/readyz
# Returns: ok
```

Use `/readyz` for Kubernetes readiness probes. On the gateway, readiness means the process can accept traffic and return `503 PROVISIONING` for cold-start capacity; worker-pod availability is exposed through `/health`, inference responses, and metrics.

**Kubernetes configuration:**

```yaml
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5
```

## OpenTelemetry Metrics

Source: [telemetry/contract.yaml](https://github.com/superlinked/sie/blob/main/telemetry/contract.yaml)

Source: [deploy/helm/sie-cluster/templates/_otel-collector-config.tpl](https://github.com/superlinked/sie/blob/main/deploy/helm/sie-cluster/templates/_otel-collector-config.tpl)

Source: [deploy/helm/sie-cluster/templates/servicemonitor.yaml](https://github.com/superlinked/sie/blob/main/deploy/helm/sie-cluster/templates/servicemonitor.yaml)

SIE services emit canonical OpenTelemetry metrics once over OTLP. Application containers do not expose a Prometheus `/metrics` endpoint. The bundled OpenTelemetry Collector receives OTLP and exposes Prometheus-compatible metrics on port `9464`. It can also fan out the same observations to a remote OTLP backend.

Canonical OpenTelemetry names use dots. The collector owns the underscore-form Prometheus compatibility names used by PromQL, Grafana, alerts, and KEDA. The checked-in telemetry contract defines the full inventory, attributes, histogram boundaries, and export eligibility.

### Metrics Reference

| Canonical OpenTelemetry name | Prometheus wire name | Type | Description |
|------------------------------|----------------------|------|-------------|
| `sie.gateway.requests` | `sie_gateway_requests_total` | Counter | Gateway inference responses by operation, outcome, HTTP status, and machine profile |
| `sie.gateway.request.duration` | `sie_gateway_request_duration_seconds` | Histogram | End-to-end gateway request latency |
| `sie.gateway.lane.queue.depth` | `sie_gateway_lane_queue_depth` | Gauge | Exact JetStream pending and delivered-but-unacknowledged work by physical lane |
| `sie.gateway.lane.queue.snapshot.timestamp` | `sie_gateway_lane_queue_snapshot_timestamp_seconds` | Gauge | Freshness companion for exact-lane queue depth |
| `sie.worker.requests` | `sie_worker_requests_total` | Counter | Completed worker items by operation, outcome, backend, lane, model, and profile |
| `sie.worker.request.duration` | `sie_worker_request_duration_seconds` | Histogram | Per-item worker latency |
| `sie.worker.batch.size` | `sie_worker_batch_size` | Histogram | Items in each formed batch |
| `sie.worker.queue.depth` | `sie_worker_queue_depth` | Gauge | Current worker queue depth |
| `sie.worker.model.loaded` | `sie_worker_model_loaded` | Gauge | Current model residency |
| `sie.worker.model.memory` | `sie_worker_model_memory_bytes` | Gauge | Model memory reported by the active backend |

### Enable Prometheus and Grafana

The bundled `kube-prometheus-stack` installs Prometheus, Grafana, Alertmanager, and the Prometheus Operator. Enabling it also creates the collector, its `ServiceMonitor`, the SIE alert rules, and dashboard ConfigMaps:

```bash
helm upgrade --install sie oci://ghcr.io/superlinked/charts/sie-cluster \
  --version 0.6.26 \
  --namespace sie \
  --create-namespace \
  --set kube-prometheus-stack.install=true
```

With an existing Prometheus Operator, set `serviceMonitor.enabled=true` instead. If Prometheus runs outside the SIE namespace, add its namespace to `observability.otel.collector.prometheus.networkPolicy.scrapeNamespaceNames`.

### Manual Prometheus Scrape

Scrape the collector, not the gateway or workers. For a release named `sie` in namespace `sie`, the static equivalent of the chart-managed `ServiceMonitor` is:

```yaml
# prometheus.yml
scrape_configs:
  - job_name: 'sie-otel-collector'
    static_configs:
      - targets: ['sie-sie-cluster-otel-collector.sie.svc:9464']
    metrics_path: /metrics
    scrape_interval: 5s
```

## Live Status

Source: [packages/sie_server/src/sie_server/api/ws.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/api/ws.py)

Source: [packages/sie_gateway/src/handlers/health.rs](https://github.com/superlinked/sie/blob/main/packages/sie_gateway/src/handlers/health.rs)

For live status without the Grafana stack, SIE has two built-in surfaces:

- **WebSocket stream:** the Python `sie-server` process streams real-time server, GPU, and model status over `/ws/status` (see [WebSocket Status](#websocket-status) below).
- **Gateway health polling:** poll the gateway `/health` endpoint for aggregate cluster status (worker count, GPUs, loaded models).

## WebSocket Status

Source: [packages/sie_server/src/sie_server/api/ws.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/api/ws.py)

The Python `sie-server` process streams real-time status over WebSocket at `/ws/status`. Updates push every 200ms. In Kubernetes, the gateway also exposes `/ws/cluster-status` for aggregate cluster status, while routing health comes from SIE server sidecar NATS heartbeats.

### Connection

```python
import asyncio
import websockets
import json

async def monitor():
    async with websockets.connect("ws://localhost:8080/ws/status") as ws:
        async for message in ws:
            status = json.loads(message)
            print(f"Loaded models: {status['loaded_models']}")
            print(f"GPU type: {status['gpu']}")
```

### Status Message Format

```json
{
  "timestamp": 1703001234.567,
  "gpu": "l4",
  "loaded_models": ["bge-m3", "e5-base-v2"],
  "server": {
    "version": "0.1.0",
    "uptime_seconds": 3600,
    "user": "sie",
    "working_dir": "/app",
    "pid": 1
  },
  "gpus": [
    {
      "device": "cuda:0",
      "name": "NVIDIA L4",
      "gpu_type": "l4",
      "utilization_pct": 45,
      "memory_used_bytes": 8589934592,
      "memory_total_bytes": 23622320128,
      "memory_threshold_pct": 95
    }
  ],
  "models": [
    {
      "name": "bge-m3",
      "state": "loaded",
      "device": "cuda:0",
      "memory_bytes": 2147483648,
      "queue_depth": 0,
      "queue_pending_items": 0,
      "config": {
        "hf_id": "BAAI/bge-m3",
        "adapter": "bge_m3",
        "inputs": ["text"],
        "outputs": ["dense", "sparse"]
      }
    }
  ],
  "counters": {},
  "histograms": {}
}
```

### Model States

| State | Description |
|-------|-------------|
| `available` | Config loaded, weights not in memory |
| `loading` | Weights currently loading to GPU |
| `loaded` | Ready for inference |
| `unloading` | Weights being evicted from GPU |
| `failed` | Last load attempt failed; config still present |

## Collector-backed Grafana Dashboards

SIE includes pre-built Grafana dashboards in the Helm chart at [`deploy/helm/sie-cluster/files/dashboards/`](https://github.com/superlinked/sie/tree/v0.6.26/deploy/helm/sie-cluster/files/dashboards). Grafana's sidecar provisions them automatically. The collector can route the same OTLP observations to a remote backend without changing application instrumentation.

These example PromQL queries use the collector's compatibility names and pin a release named `sie` in namespace `sie`. Change `namespace` and collector `service` for your release; keep `endpoint="prometheus"`.

### Request Rate

```txt
sum by (operation) (
  rate(sie_gateway_requests_total{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service="sie-gateway",
    outcome="success"
  }[5m])
)
```

### P99 Latency

```txt
histogram_quantile(0.99,
  sum by (le, operation) (
    rate(sie_gateway_request_duration_seconds_bucket{
      namespace="sie",
      service="sie-sie-cluster-otel-collector",
      endpoint="prometheus",
      producer_service="sie-gateway"
    }[5m])
  )
)
```

### GPU Memory Usage

```txt
max by (model, profile, backend, lane) (
  sie_worker_model_memory_bytes{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service="sie-worker"
  }
)
```

### Queue Depth

```txt
max by (pool, machine_profile, bundle) (
  sie_gateway_lane_queue_depth{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service="sie-gateway"
  }
  and on (producer_instance, collector_generation, pool, machine_profile, bundle)
  (
    abs(time() - sie_gateway_lane_queue_snapshot_timestamp_seconds{
      namespace="sie",
      service="sie-sie-cluster-otel-collector",
      endpoint="prometheus",
      producer_service="sie-gateway"
    }) < 20
  )
)
```

### Batch Efficiency

```txt
avg by (model, profile) (
  sie_worker_batch_fill_ratio{
    namespace="sie",
    service="sie-sie-cluster-otel-collector",
    endpoint="prometheus",
    producer_service=~"sie-worker|sie-worker-sidecar"
  }
)
```

## Alert Rules

Source: [deploy/helm/sie-cluster/files/alerts/sie-rules.yaml](https://github.com/superlinked/sie/blob/main/deploy/helm/sie-cluster/files/alerts/sie-rules.yaml)

The `sie-cluster` chart can render pre-configured Prometheus alert rules:

| Alert | Severity | Condition | Description |
|-------|----------|-----------|-------------|
| `SIEWorkerDown` | critical | SIE server sidecar container not ready for 2 min | A worker pod is unavailable |
| `SIENoHealthyWorkers` | critical | No ready SIE server sidecar containers for 1 min | All worker pods are unavailable |
| `SIEWorkerHighQueueDepth` | warning | Fresh lane queue depth > 50 for 5 min | The physical lane may need more capacity |
| `SIEGPUMemoryHigh` | warning | GPU memory > 90% for 5 min | Risk of OOM, LRU eviction may be insufficient |
| `SIEGPUTemperatureHigh` | warning | GPU temp > 80°C for 5 min | GPU throttling likely, check cooling |
| `SIEGPUECCErrors` | critical | Double-bit ECC errors increase over 1h | Hardware issue likely |
| `SIEGatewayDown` | critical | No ready gateway containers for 1 min | Traffic cannot be routed |
| `SIEHighErrorRate` | warning | Gateway 5xx rate > 5% for 5 min | Server or model errors spiking |
| `SIEHighLatency` | warning | p95 latency > 5s for 5 min | Request latency is above target |
| `SIEGenerationServerErrorSpike` | warning | Generation 5xx rate > 0.02 req/s for 5 min | Generation failures are increasing on a machine profile |
| `SIEModelLoadingRetrySpike` | info | Model-load errors or timeouts > 0.05/s for 10 min | Workers are repeatedly failing to load a model |
| `SIEResourceExhaustedSpike` | info | Terminal OOM recovery > 0.1/s for 10 min | Worker capacity is constrained |
| `SIEConfigDown` | critical | Config container not ready for 2 min | Config writes are blocked; gateways serve cached state |
| `SIEProvisioningStuck` | warning | Pod Pending for 10 min | Check scheduling events and GPU capacity |
| `SIEScaleUpFailed` | warning | FailedScheduling event in 10 min | Likely insufficient GPU capacity |

### Installing Alert Rules

The bundled `kube-prometheus-stack` command above installs the alert rules automatically. With an existing Prometheus Operator, enable them explicitly:

```bash
helm upgrade --install sie oci://ghcr.io/superlinked/charts/sie-cluster \
  --version 0.6.26 \
  --namespace sie \
  -f helm-values.yaml \
  --set alertRules.enabled=true
```

### Custom Alerts

Add custom alerts to your Prometheus configuration:

```yaml
# Alert when P99 latency exceeds 5 seconds
- alert: SIEHighLatencyP99
  expr: |
    histogram_quantile(0.99,
      sum by (le, operation) (
        rate(sie_gateway_request_duration_seconds_bucket{
          namespace="sie",
          service="sie-sie-cluster-otel-collector",
          endpoint="prometheus",
          producer_service="sie-gateway"
        }[5m])
      )
    ) > 5
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "High P99 latency for {{ $labels.operation }}"
```

---

## Logging

Source: [packages/sie_server/src/sie_server/core/logging.py](https://github.com/superlinked/sie/blob/main/packages/sie_server/src/sie_server/core/logging.py)

SIE supports both human-readable and structured JSON logging.

### Log Levels

Enable verbose logging with `--verbose` or `-v`:

```bash
sie-server serve --verbose
```

### JSON Logging

Enable JSON format for Loki and log aggregation systems:

```bash
sie-server serve --json-logs
```

Or via environment variable:

```bash
export SIE_LOG_JSON=true
sie-server serve
```

### JSON Log Format

```json
{
  "timestamp": "2025-12-18T10:30:00.123Z",
  "level": "INFO",
  "logger": "sie_server.api.encode",
  "message": "Inference completed",
  "model": "bge-m3",
  "request_id": "abc123",
  "trace_id": "def456",
  "latency_ms": 45.2,
  "batch_size": 16,
  "gpu_type": "l4"
}
```

### Structured Fields

JSON logs include optional fields when available:

| Field | Description |
|-------|-------------|
| `model` | Model name for the request |
| `request_id` | Unique request identifier |
| `trace_id` | OpenTelemetry trace ID |
| `latency_ms` | Request latency in milliseconds |
| `batch_size` | Number of items in the batch |
| `gpu_type` | Detected GPU type |

## What's Next

- [Scale-from-Zero](/docs/deployment/autoscaling/) - autoscaling lifecycle and troubleshooting
- [Troubleshooting](/docs/reference/troubleshooting/) - common issues and solutions
- [CLI Reference](/docs/reference/cli/) for all server options
- [API Reference](/docs/reference/api/) for endpoint documentation