SIE vs TEI (Text Embedding Inference): Benchmarks, Cost, Setup

Short version: TEI (Text Embeddings Inference, from Hugging Face) is a lightweight single-model server. SIE (Superlinked Inference Engine) is a multi-model cluster. For exactly one embedding model at low volume, TEI is a reasonable choice and the setup is shorter. The economics separate at the second model, because TEI’s unit of deployment is the instance and SIE’s is the GPU.

SIE is open source and free under Apache 2.0. Stop reading and start cutting per-token costs whenever you like: the engine runs on your own hardware, from a laptop to a Kubernetes cluster.

★ Star the repo on GitHub Run the 2-minute quickstart →

What each one is built to do

TEI serves text embedding and reranking models, one model per server, behind an OpenAI-compatible HTTP API. It is a single Docker container with a small footprint, it is battle-tested, and it integrates cleanly with the rest of the Hugging Face ecosystem. Apache 2.0.

SIE serves 100+ models from one deployment across four operations: encode, score, extract and generate. Models load on demand and share GPU memory through least-recently-used eviction. LoRA adapters hot-load without a restart. The deployment ships with a load-balancing Rust gateway, KEDA autoscaling with scale to zero, Grafana dashboards, and Terraform modules for EKS and GKE. Also Apache 2.0.

The difference in one line: TEI answers “how do I serve this encoder,” and SIE answers “how do I serve forty of them well.”

Benchmarks: what is measured, and what is not

Being straight about the evidence here. The published SIE embedding benchmark is measured against hosted APIs (Voyage, OpenAI, Cohere), not against TEI. There is no head-to-head SIE versus TEI throughput run published yet, and inventing one would be worse than useless.

What can be said with numbers is the SIE side of the comparison, measured on Modal GPUs at L4 ($0.80/hr) and RTX PRO 6000 ($3.03/hr), on sustained batch ingestion:

Two things in that SIE table matter when you later compare it with TEI. Tail latency is tight, with p99 staying within a few milliseconds of p50, which is what you want when embedding sits inside a request path. And throughput scales linearly by adding workers at constant unit cost, because nodes are ceil(load / knee) and cost per token is GPU$/hr / (3600 × knee), which is independent of node count. Full methodology is in the hosted embedding API benchmark.

For single-model, single-request serving, treat TEI and SIE as comparable and pick on operational fit rather than throughput. Both are thin wrappers over the same model weights on the same GPU. Where SIE’s architecture is designed to pull ahead is concurrency, not raw single-request speed:

• Concurrent requests, where full-batch formation before the GPU runs beats per-worker local batching
• Multiple model types in one workload, where models share GPU memory instead of each holding a dedicated instance
• Large corpus indexing, where asynchronous batching cuts wall-clock time

Those are SIE design claims, not matched TEI measurements. If you need a defensible head-to-head number for a specific model on your own hardware, run both containers on the same GPU with the same corpus. That is a one-afternoon test and it beats any vendor’s table, including ours.

Cost: the multiplier is instance count, not per-token price

Neither of these charges per token, so the cost comparison is not about pricing at all. It is arithmetic on how many GPUs your model count forces you to keep running.

TEI’s unit is the instance: one model per server. Four models means four instances; each instance holds its GPU whether or not traffic is arriving. SIE’s unit is the GPU: models load on demand, hot ones stay resident, cold ones get evicted when something else needs the slot.

Worked as a model rather than a measurement, at the L4 rate above and a 720-hour month:

The published packing example runs five models on two GPUs at a 71% warm-start hit rate, against a per-model-per-GPU layout that averages roughly 21% utilization under bursty traffic. That utilization figure is the real finding. Under a one-model-per-instance layout you are not mostly paying for inference; you are mostly paying for idle.

Three honest caveats. Packing depends on VRAM headroom, so large models or long-context configurations pack less well. Warm-start hit rate depends on your traffic distribution, and a genuinely uniform spread across forty models will thrash. And if you run exactly one model at steady load, there is nothing to pack and the two costs converge.

Setup: one container versus one cluster

TEI is shorter to start, and there is no point pretending otherwise:

docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-embeddings-inference \
  --model-id BAAI/bge-m3

SIE has three entry points depending on how far you are. Locally it is also one command:

pip install "sie-server[local]" && sie-server serve

On Kubernetes, Terraform provisions the GPU cluster and Helm deploys the engine:

module "sie" {
  source = "superlinked/sie/aws"
  region = "us-east-1"
  gpus   = ["a100-40gb", "l4-spot"]
}

terraform apply
helm install sie-cluster oci://ghcr.io/superlinked/charts/sie-cluster

And there is SIE Cloud, where Superlinked runs the cluster, which is the right comparison point if what you actually dislike about TEI is not the container but owning the GPUs at all. Same engine, same model IDs, same SDK, so moving between managed and self-hosted is a base URL change rather than a migration.

For a single model the TEI command wins on brevity. For a cluster serving many models, the Terraform and Helm path replaces work you would otherwise do by hand for each instance: provisioning, networking, autoscaling and monitoring, once instead of per model.

At a glance

When TEI is the right answer

It is worth being clear about this, because a comparison that concludes “always us” is not useful to anyone.

Pick TEI when you serve one embedding model and have no near-term plan to add another, when you want the smallest possible footprint, when your team is deep in Hugging Face infrastructure already, or when you need something running in the next ten minutes and the model is the only variable you care about.

The moment you add a reranker, an extractor, or a second encoder to A/B test, the one-model-per-server cost is the thing you start paying, and that is the cost SIE removes.

Migrating from TEI

SIE exposes a compatible REST API and uses the same Hugging Face model ID format, so migration is usually an endpoint change plus installing the SDK. There is a TEI migration guide with the specifics.

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")
client.encode("BAAI/bge-m3", Item(text="same model id, different server"))

FAQ

Is SIE faster than TEI? For one model handling one request at a time, treat them as comparable. The gap opens under concurrency, multiple model types, and large batch indexing jobs. Run both on your hardware if the number matters to your decision.

Is SIE overkill for one model? For exactly one model and no plans to add more, yes, TEI is reasonable. The design assumption behind SIE is that you will end up with more than one.

Does SIE support every model TEI supports? It supports 100+ models including the major embedding and reranker families. If something you need is missing it can be added, since the engine is open source.

Is setup harder? For a single model, yes, slightly. For a multi-model production deployment, the Terraform and Helm path is less work than managing one instance per model.

Can I keep TEI and add SIE? Yes. They are both plain HTTP services and nothing stops you pointing different parts of your pipeline at each.

Star the repo on GitHub · Read the encode docs · Browse the model catalog