---
title: Should You Self-Host Inference?
description: "When self-hosting inference beats a hosted API: the ~10M-requests-a-month break-even, whether sub-40B open models are good enough, the hardware you need, and how to serve many small models on one cluster."
canonical_url: https://superlinked.com/blog/should-you-self-host-inference
last_updated: 2026-07-10
---
Self-host the steady, high-volume, latency-sensitive parts of your workload, and rent the API for spiky traffic and the hardest frontier reasoning. Past roughly 10 million requests a month, one saturated GPU running a sub-40B open model usually beats the per-token price, gives you control over pricing and model behavior, and keeps your prompts and context inside your own cloud. Below that, or when you truly need the absolute frontier model, a hosted API is still the right call.
That is the short answer. The longer one is a talk Daniel Svonava (CEO of Superlinked) gave at Infer() Summit 2026, where he works through the economics, the model quality question, the serving architecture, and a real agent built entirely on self-hosted small models.
Watch the full talk, then read on for the break-even math and the serving pattern:
## Self-hosting pays off once a GPU stays busy
Self-hosting trades a per-token bill for a fixed GPU cost. A hosted API meters every token you send. A GPU is a fixed hourly cost, so once you can keep it busy, heavy and steady workloads get much cheaper.
The break-even depends on volume. In the talk's worked example, a dense 32B model on one GPU at 50% utilization is compared with a \$0.50 per million token managed API. At about 500 tokens per request, that crossover lands around 10 million requests a month (roughly 5B tokens/month of API spend versus one busy GPU-hour budget). Mixture-of-experts and 4B-class models cross far sooner, and an idle GPU never crosses at all. Past that point, a single busy GPU serving a 30B model works out to roughly \$0.06 to \$0.85 per million tokens on the June 2026 rates shown in the talk.
Rent the API when your volume is low or spiky (under about 5 million requests a month, where a dedicated GPU sits idle), you need the absolute frontier model for the hardest reasoning, or your whole inference bill is still under about \$2K to \$3K a month so GPU operations are not worth it yet.
Self-host when a task runs steady past that 10 million request line, data control or compliance means prompts and context cannot leave your cloud, or you ship inference inside a product that runs across many customer clouds. On the cost point, Intercom reported cutting around \$250K a month by replacing a hosted GPT call with a fine-tuned 14B Qwen model on one Fin AI pipeline task (Fergal Reid, Intercom Chief AI Officer, Chain of Thought, 2026).
Control is the second argument. When you rent inference, the vendor owns both the price and the behavior. They can reprice the model or quietly retune how it answers, with no say from you. After June 1, 2026, GitHub raised [model multipliers for annual Copilot Pro and Pro+ subscribers who stayed on legacy request-based billing](https://docs.github.com/en/copilot/reference/copilot-billing/request-based-billing-legacy/model-multipliers-for-annual-plans); those multipliers do not apply to the new usage-based model. That kind of mid-plan change is hard to plan around when the model lives on someone else's account.
## Small open models cover many enterprise tasks
For most enterprise work, yes. By "small" the talk means models under 40 billion parameters, the ones that fit comfortably on a single GPU two to four generations behind the latest hardware.
Plot the [Artificial Analysis Intelligence Index](https://artificialanalysis.ai/models/comparisons/qwen3-6-27b-vs-gpt-5-1) over time and two things stand out. The frontier is not climbing as steeply as it used to, and the small open-weight models are still climbing fast. The gap is closing. In the talk's snapshot, GPT-5.1 and Qwen3.6-27B scored roughly 48 versus 46 on that index (about 96% of the GPT-5.1 score), released about five months apart. Scores on Artificial Analysis move by reasoning mode and by date, so treat those numbers as the talk's cited snapshot rather than a permanent ranking, and re-check the live comparison before you decide. On this index, the two were close enough that waiting a few months often buys frontier-adjacent quality in a form factor that runs on one GPU.
Small models do not need to win the hardest reasoning benchmarks. For a support flow, an enterprise workflow, or a document pipeline, they are plenty smart.
## Domain adaptation can beat a generalist model
Open weights are not fixed. You can adapt them, and it is cheap.
Take [`Snowflake/snowflake-arctic-embed-l-v2.0`](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-v2.0), a 568M-parameter multilingual embedding model, and ask what it takes to improve German legal retrieval. In the Infer() Summit 2026 talk, Daniel walks through a Superlinked LoRA run: a single adapter (r128, alpha 256, all-linear, about 56.9M trainable parameters, roughly 10% of the model) trained on one L4 24GB GPU in 41 minutes for about \$0.80, producing a 114MB adapter with these results:
On GerDaLIRSmall (German legal, in-domain, nDCG@10), the base model scores 0.2613. The legal LoRA lifts it to 0.3129, a 19.7% gain. On LegalQuAD (German legal, transfer, nDCG@10), the base 0.5097 rises to 0.5660, up 11.0%. The trade-off is a small regression on general German tasks, so a 2:1 legal-to-general training mix holds most of the legal gain (+18.1% and +8.0% on the two legal sets) while keeping general German within about 2% (GermanDPR down 2.1%, GermanQuAD down 0.6%).
That is a domain-specific embedder pushing past a generalist frontier model on the problem you care about, for the price of a coffee. The same recipe adapts embedder, reranker, and decoder families, and modern serving engines can hold multiple adapters in memory and swap between them, so one deployment can serve different languages or domains. See also SIE's [LoRA docs](/docs/engine/lora).
## A specialized model exists for most tasks
Almost always. As of July 2026, [Hugging Face hosts on the order of 3 million models](https://huggingface.co/models). Most are not state-of-the-art, but hundreds each month are the best available option in some narrow niche: OCR for receipts in a specific language, entity extraction, reranking, speech-to-text, forecasting, and more.
The two real barriers are discovering what exists and actually running it. The models are there. Which is why the interesting problem shifts from "is there a model" to "how do I serve dozens of them well."
## One rented GPU covers many sub-40B workloads
Less than most people assume. A sub-40B model at FP8 or similar quantization, with a practical context window and KV-cache budget, typically fits one GPU you can rent today. Full-precision or very long-context serving can need more VRAM.
Entry tier is an L4 or A10 (24GB) at roughly \$0.39 to \$1.00 an hour, which runs 4B-class models and embeddings and provisions in minutes almost anywhere. The workhorse tier is an A100 80GB or RTX PRO 6000 96GB at about \$0.90 to \$1.80 an hour, enough for 30B-class models at FP8 or mixture-of-experts, available on every neocloud on demand. Headroom is an H100 80GB at roughly \$1.80 to \$4 an hour, and only large H100 clusters tend to be reservation-gated. Those ranges are the cheapest credible on-demand rates surveyed in the talk for June 2026 across [RunPod](https://www.runpod.io/pricing), [Vast.ai](https://vast.ai/pricing), Hyperstack, and DeepInfra (product and region vary; spot is often much lower). Re-check live pricing before you budget.
## Embeddings are a low-risk self-hosting entry point
Embeddings are the easiest entry point to self-hosting: the models are small, easy to serve, and one of the first things teams pull in-house. On sustained batch ingestion on a single RTX PRO 6000, SIE embedding workers land here, scaling linearly at constant unit cost per worker (full methodology in [SIE vs hosted embedding APIs](/blog/sie-vs-hosted-embedding-apis)):
| Embedder | Throughput (tok/s, 1 GPU) | Query P50 | \$ / 1M tokens | Embed 1B |
| --- | --- | --- | --- | --- |
| SIE bge-m3 | 235,430 | 15 ms | \$0.0036 | ~\$3.60 |
| SIE stella-1.5B | 82,382 | 27 ms | \$0.0102 | ~\$10 |
| SIE Qwen3-Emb-4B | 48,675 | 27 ms | \$0.0173 | ~\$17 |
| Voyage-4-large | 150,000 (tier cap) | 180 ms | \$0.12 | \$120 |
| OpenAI text-embedding-3-large | 166,667 (tier cap) | 157 ms | \$0.13 | \$130 |
| Cohere embed-v4.0 | 17,067 | 84 ms | \$0.12 | \$120 |
The provider rows are usage-tier ceilings, while the SIE rows are single-GPU and scale out linearly. In that benchmark, stella and Qwen3 land within a few hundredths of ndcg of the hosted frontier (winning 2 of 8 MTEB tasks in the comparison set), and bge-m3 is the throughput and cost pick. Embedding a billion-token corpus on one GPU takes about an hour, at latencies well below any managed provider and roughly a tenth of the cost.
## Shared queues and model packing raise GPU utilization
Because a runtime is not a cluster. Runtimes like [vLLM](https://docs.vllm.ai/en/latest/serving/parallelism_scaling/), [SGLang](https://docs.sglang.ai/references/multi_node_deployment/multi_node.html), and [TensorRT-LLM](https://nvidia.github.io/TensorRT-LLM/latest/features/parallel-strategy.html) can already distribute a model across GPUs and nodes. TEI from Hugging Face covers embeddings and reranking. What they still leave to you is the surrounding control plane: routing, shared queues, autoscaling, and multi-model lifecycle. Open-source cluster options for that layer include NVIDIA Dynamo, llm-d, KServe, and the [Superlinked Inference Engine (SIE)](https://github.com/superlinked/sie).
Most of those target large models. SIE is built for the sub-40B world: many small models on one cluster, one-click install into different clouds, with LLM, OCR, vision, embedding, and reranking coverage under a single gateway. Serving many small models well comes down to two utilization tricks, because on a single-GPU-per-model layout, utilization is where the money leaks.
The first trick is a shared queue. In the traditional route-then-batch pattern, a central scheduler commits each request to a worker up front, and each worker batches only its own local slice. With many small, fast requests (50 to 100 milliseconds each), those local queues run uneven and pack poorly, and a half-full batch takes almost as long as a full one. SIE inverts this: a stateless gateway publishes work to one shared queue (built on NATS JetStream, deliberately low-overhead, no Kafka), and a sidecar inside each worker pod pulls from the pool and forms a full batch before the GPU runs. Traffic never fragments into starved local queues. Daniel reported this lifted throughput by more than 50% on real small-model workloads; in the illustrative model on the slide, GPU efficiency rises from 51% to 92%, roughly 1.8x the useful throughput per GPU at the same latency.
The second trick is packing models onto shared GPUs. Pinning one model per GPU means five models need five always-on GPUs, and with bursty traffic they average around 21% utilization. An LRU residency scheduler instead loads multiple models into the VRAM of the same GPU, keeps hot models resident, and evicts cold ones only when something else needs the slot ([engine docs](/docs/engine)). In the example, five models share two GPUs at a 71% warm-start hit rate. Lazy loading with eviction, pinning, and capacity reservation lets one heterogeneous cluster absorb traffic that touches dozens of different models.
## A small-model cluster separates gateway, queues, and workers
A self-hosted small-model cluster has a few moving pieces. A stateless gateway resolves the model, profile, and pool, speaks an OpenAI-compatible API over HTTP and MsgPack, and returns a 202 while capacity spins up. A NATS queue holds pool-separated work streams in JetStream and lets worker pods scale from zero. A worker pool keeps GPUs fed, with a server sidecar batcher in front of backends like Candle, PyTorch, and SGLang, and larger models routed to larger machine profiles while idle models evict under pressure. SIE runs its own backends written in PyTorch and Candle in Rust alongside other open-source ones, sweeping parameters to pick the best serving path per model.
The design goal underneath all of this is that AI engineers and infrastructure engineers should not have to talk to each other to get their jobs done. AI engineers call the cluster API to add models and profiles, create and size pools, pin hot models, and run inference. Infrastructure engineers own the cloud account, set machine profiles and capacity, configure KEDA autoscaling from zero, and watch GPU, queue, and eviction health. Treat the AI engineers as internal customers and the two worlds stay decoupled.
## Small open models can run a real contract agent
Yes, and the interesting part is that you use many small models rather than one big one. Daniel's example is a contract review agent built on the OpenAI Agents SDK, with every model call served by one self-hosted cluster. It extracts the facts (parties, dates, amounts, governing law, including from a scanned signature page), flags risky clauses with a severity and a one-line suggested redline, confirms whether the contract is actually signed, and surfaces upcoming obligations and deadlines.
One orchestrator drives six tools across nine small models. A granite-guardian model screens input. A Qwen3-4B investigator calls tools in any order: Qwen3-0.6B to classify the document, GLiNER to extract entities, LightOnOCR plus Qwen3.5-4B to read the signature page, bge-m3 plus a Qwen3 reranker to search the clauses, a Qwen3-4B reasoning sub-agent to analyze clause risk, and sqlcoder for text-to-SQL over the obligations database. A final Qwen3-4B synthesizer, with no tools, emits a schema-valid review. Because one agent alone touches nine models, and a real organization runs many such agents, cluster traffic ends up spanning 20, 30, 40 models at once, which is exactly the case the pooling and packing tricks are built for.
Run against CUAD, a set of 500-plus real commercial contracts filed with the SEC, the agent earned its keep. It caught a contradiction where a late-delivery penalty was written as both 30% and 5% in the same contract, flagged a force-majeure clause missing its required certificate, and stayed honest: when a signature scan was unreadable, it marked execution uncertain rather than inventing a signature. The code is in the [SIE repo](https://github.com/superlinked/sie).
## Self-host when cost or control justifies it
Self-host when cost, control, or your product demand it. On Superlinked's published comparisons: gte-multilingual all-in on AWS EKS (L4) comes out about [50x cheaper than managed embedding APIs](/) for high-utilization workloads (see also [SIE vs hosted embedding APIs](/blog/sie-vs-hosted-embedding-apis)); bge-m3 is about 2.7x faster than Cohere rerank-3.5 on MTEB AskUbuntu in Superlinked's evals; Qwen3.6-27B held about 96% of GPT-5.1 on the talk's [Artificial Analysis](https://artificialanalysis.ai/models/comparisons/qwen3-6-27b-vs-gpt-5-1) snapshot; and the stack can run fully in your cloud or [air-gapped](/docs/deployment/offline).
Rent the API when your volume is low or spiky, your bill is still small, or you genuinely need the frontier model for the hardest reasoning. Everywhere else, the small open models are smart enough, the hardware is a single GPU you can rent in two minutes, and the deployment is yours, which means no lost access and far less platform risk.
SIE is Apache 2.0 and open source at [github.com/superlinked/sie](https://github.com/superlinked/sie). Watch the full talk above.