How to Serve a Fleet of Small Open-Source Models
Small open-source models are increasingly capable of handling individual tasks inside AI agents, from embeddings and reranking to OCR, extraction, classification and structured generation. But replacing one general-purpose API with a fleet of specialised models changes the infrastructure problem completely: instead of figuring out how to serve one enormous model, you need to keep many smaller models busy, available and easy to swap.
Why use multiple small models instead of one large LLM?
Because many AI workflows are collections of narrower tasks.
Consider a contract-review agent.
One part of the workflow might need OCR. Another extracts entities. Another compares clauses. Another performs retrieval. Another generates structured output. Another applies a safety or policy check.
Those jobs do not necessarily require the same model.
Instead of sending every operation to one general-purpose frontier model, you can select a model specifically suited to each task.
This creates a different architecture:
one agent, many models.
The talk uses a nine-model contract-review agent as an example of what this can look like in practice.
A company running several agents may therefore end up operating dozens of model-task combinations rather than one central LLM endpoint.
The question becomes less about whether suitable open models exist and more about how to serve all of them without creating an infrastructure mess.
Are small open-source models actually good enough?
For specific tasks, they increasingly can be.
The important distinction is between general capability and task-specific capability.
A smaller model does not need to match a frontier model at every possible task to be useful. It needs to perform well enough on the particular task assigned to it.
That might be:
- embedding text
- reranking search results
- OCR
- named entity recognition
- image classification
- text-to-SQL
- document extraction
- guardrails
- structured output
- code review
- tool calling
Open-model ecosystems such as Hugging Face contain models trained and fine-tuned specifically for many of these jobs.
The practical workflow is therefore:
- break the application into individual inference tasks
- identify candidate models for each task
- evaluate them on your own data
- adapt or fine-tune where useful
- deploy the models that meet the required quality, latency and cost targets
This is fundamentally different from selecting one model and prompting it to do everything.
Browse the model catalog for task-specific open models that already ship with SIE.
Why is serving small models different from serving one large LLM?
Large-model inference infrastructure is generally designed around a scarce, expensive model that spans one or more GPUs.
The infrastructure needs to decide where each incoming request should go while accounting for things such as worker availability, memory and KV-cache state.
That naturally leads to a top-down architecture:
request -> router -> selected worker
Small-model workloads behave differently.
Requests can be much faster and more numerous. There may also be many different models sharing the same infrastructure.
If the router must make a perfect placement decision for every short request, routing itself can become part of the bottleneck.
Meanwhile, individual GPU workers can end up with poorly balanced local queues.
The result is an odd situation where plenty of GPU capacity exists but a significant amount of it remains idle.
Why can top-down routing struggle with small-model inference?
Because the state used to make the routing decision is already changing by the time the decision is made.
Imagine several GPU workers, each processing many short inference requests.
A central router sees a snapshot of their queue states and chooses a worker.
But those workers are processing requests quickly. By the time another scheduling decision is made, the snapshot may already be outdated.
This is particularly problematic when the requests themselves are small.
A slightly inefficient scheduling decision matters much more when you make thousands of them.
In the experiments discussed in the talk, conventional routing setups using small-model traffic made it difficult to push GPU utilisation beyond roughly 20 to 30 percent under constant load.
That result is specific to the experiments described in the talk, not a universal benchmark for vLLM, SGLang or every routing configuration.
The architectural lesson is broader: scheduling designed around large-model inference is not automatically optimal for fleets of small models.
What is shared-queue inference?
Shared-queue inference changes who decides what a GPU should process next.
Instead of a router assigning each request directly to an individual worker, incoming requests enter a shared queue.
The architecture becomes roughly:
client -> gateway -> shared queue -> workers
The workers then pull work from the queue themselves.
This matters because the worker has the best information about its own current state.
It knows:
- what it is already processing
- how much capacity remains
- what model it currently has available
- how large a batch it can handle
Rather than asking a central router to continually predict the state of every GPU, each worker participates directly in scheduling its own work.
Why should inference workers form their own batches?
Because batch size has a major effect on GPU throughput.
Too small a batch and the GPU is underutilised.
Too large a batch and latency or memory consumption can increase.
For small-model traffic, the ideal batch can change rapidly because a worker may process many short requests in a small amount of time.
Allowing the worker to pull from a shared queue means it can construct a batch based on its own current capacity.
That removes some of the guesswork from the central router.
In the Superlinked experiments described in the talk, moving to centralised queueing and worker-side batching produced roughly double the cluster throughput for the tested workload.
Treat that as a Superlinked benchmark for that particular setup, not a guarantee that every cluster will double its throughput.
What does the gateway do?
The gateway should do as little work as possible.
Every request passes through it, so expensive operations at the gateway can turn it into the next bottleneck.
In the architecture described in the talk, the gateway:
- receives the inference request
- inspects enough of it to understand its shape
- attaches routing and model metadata
- places the request into the shared queue
- leaves most of the inference work to the workers
Large binary inputs such as images or video need additional care because repeatedly serialising and copying them can become expensive.
The broader principle is simple:
keep the central request path thin.
Anything that can safely happen farther down the distributed system should not unnecessarily block the gateway.
Why use more than one inference runtime?
Because there is no single runtime that is optimal for every open model.
Different model architectures behave differently.
A runtime that performs extremely well for generative transformers may not be the best way to run an encoder, OCR system, reranker or specialised extraction model.
SIE uses multiple compute engines rather than forcing every model through the same execution environment. Today that includes PyTorch, Flash Attention and SGLang, with the server selecting an engine per model. Published image bundles (default, sglang and transformers5) package the adapters each family needs; see bundles for which models land where.
The important architectural idea is that the API layer should abstract these runtime differences from the application.
The application asks for inference.
The infrastructure decides how that particular model should run.
Why does Superlinked use a worker abstraction?
Supporting many models creates dependency problems as well as scheduling problems.
Different model families may require:
- different Python packages
- different library versions
- different inference runtimes
- different batching behaviour
- different GPU memory profiles
- different preprocessing
Trying to force every model into one giant runtime environment quickly becomes difficult to maintain.
In Kubernetes, SIE places a thin server sidecar beside a Python sie-server adapter process in each GPU worker pod. The gateway publishes work onto a JetStream stream; the sidecar pulls and batches it, then calls the adapter over IPC. The adapter loads the model on first use and runs inference. That split keeps scheduling and queue intake separate from model-family-specific Python and CUDA dependencies. See How SIE processes inference requests.
The goal is to allow the cluster to support very different model architectures without making application code care about the details.
Can multiple AI models run on the same GPU?
Yes, and with small models it can make sense to do so.
The traditional deployment model often looks like this:
one model -> dedicated worker pool -> dedicated GPUs
That assumption comes partly from enormous models that consume most or all of the memory available on their GPUs.
Small models change the equation.
If several models fit into GPU memory, dedicating an entire GPU to each one can waste capacity.
Instead, inference infrastructure can combine several strategies:
- pin heavily used models in memory
- load less frequently used models on demand
- evict models under memory pressure
- pack multiple models onto the same GPU
This is especially useful for agent workloads where an individual task-specific model might only be called occasionally.
What is model packing?
Model packing means using the memory capacity of one GPU to serve multiple smaller models rather than reserving that GPU for a single model.
Suppose an agent needs:
- an embedding model
- a reranker
- an extraction model
- a guardrail model
- a small generative model
If each model is relatively small, reserving five separate GPUs may make little economic sense.
Some models can coexist.
Others can be loaded and unloaded as demand changes.
The inference system therefore starts to resemble a cache-management problem as much as a traditional model-serving problem.
Frequently used models stay hot.
Less frequently used models can be loaded lazily and evicted when memory is needed elsewhere.
SIE loads models on demand and uses LRU eviction when GPU memory fills up, so the catalog stays addressable without pre-loading everything.
Why are LoRAs useful with small models?
Small models become considerably more useful when they can be adapted cheaply to a particular task.
LoRA, or Low-Rank Adaptation, lets teams adapt a pretrained model without retraining every parameter.
That introduces another infrastructure challenge.
AI engineers may create new LoRAs or fine-tuned checkpoints frequently. If deploying every new variant requires a ticket, Docker-image rebuild and manual infrastructure change, experimentation slows down.
The ideal workflow separates responsibilities:
AI engineers choose, evaluate and adapt models.
Infrastructure engineers operate the cluster.
Deploying a new supported model or adapter should not require those two teams to coordinate manually every time.
The talk describes this operational handoff as one of the less obvious bottlenecks in adopting small models.
SIE supports LoRA adapters and config-driven model adds (including GitOps flows that avoid image rebuilds when the adapter already exists in a deployed bundle).
How cheap can adapting a small model be?
In one proof of concept discussed in the talk, Superlinked trained a LoRA for approximately $0.80 and reported an 18 percent improvement in retrieval quality on the German Legal STS dataset used for that experiment.
That should be presented exactly as what it is:
a proof of concept on one dataset and task.
It does not mean an $0.80 LoRA will improve arbitrary models or workloads by 18 percent.
What it demonstrates is that model adaptation can become cheap enough to be part of normal experimentation rather than a major training project.
What workloads are easiest to move to self-hosted small models?
Embeddings are one of the clearest starting points.
Embedding models are:
- relatively small
- highly batchable
- predictable
- easy to evaluate
- called frequently in search and RAG systems
The talk presents Superlinked measurements showing hundreds of thousands of embedding tokens per second on a single GPU for some model and hardware combinations.
Do not generalise that benchmark to every embedding model or GPU.
The broader point is that embedding inference can often be run efficiently on comparatively modest hardware, and self-hosted inference is often the right home once volume is steady.
Other promising workloads include:
- reranking
- named entity recognition
- classification
- OCR
- synthetic-data generation
- annotation generation
- structured extraction
- task-specific generation
These are particularly attractive when the workload is high-volume and the quality can be evaluated systematically.
How should small models be benchmarked?
Maximum throughput on its own is not enough.
A useful concept discussed in the talk is the knee of the throughput-versus-latency curve.
As request volume increases, throughput initially rises with it.
Eventually, the server reaches saturation.
Beyond that point, asking it to handle more requests no longer produces much extra throughput, but latency starts increasing quickly.
That turning point is the knee.
It is often a much more useful operating target than simply asking how many requests a server can process before it falls over.
When comparing model-serving configurations, measure at least:
- throughput
- latency
- GPU utilisation
- batch size
- memory consumption
- concurrency
- model load time
And evaluate those metrics at the workload shape you actually expect in production.
Why does pre-tuning models matter?
Open-source inference software gives engineers enormous flexibility.
That flexibility also creates work.
Batch sizes, runtime parameters, memory settings and execution strategies may need tuning for:
- the model
- the GPU
- request size
- concurrency
- latency requirements
If every new model creates another parameter-sweep project, supporting a large model catalog becomes expensive.
The approach described in the talk is to benchmark and tune model configurations ahead of time so that supported models ship with configurations intended for the hardware and runtime on which they will execute.
The goal is to move model-serving optimisation out of each individual user’s deployment process and into the inference platform itself.
What is automated inference optimisation?
Once a system supports many models, manually finding an optimal configuration for every model and GPU combination does not scale.
The talk describes an automated research loop for this process.
Conceptually, the system:
- selects a model and hardware configuration
- runs benchmarks
- changes inference parameters
- measures throughput and latency
- compares the result
- repeats the process
- packages the resulting configuration
This turns inference optimisation into a repeatable search problem.
Instead of asking every infrastructure team to rediscover a good setup independently, those results can be encoded into the model-serving system.
How does Superlinked Inference Engine serve small models?
Superlinked Inference Engine, or SIE, is an open-source inference server and production cluster designed to serve the different models used inside AI applications and agents.
SIE currently:
- ships a catalog of 100+ supported models for embeddings, reranking, extraction, OCR and document processing, vision, guardrails and generation
- loads models on demand with LRU eviction so multiple models can share a GPU
- wraps PyTorch, Flash Attention and SGLang behind one API surface
- exposes OpenAI-compatible endpoints such as
/v1/embeddingsand/v1/chat/completions - deploys the same image from a laptop to Kubernetes with a load-balancing gateway, KEDA autoscaling and cloud paths for AWS, GCP and Azure
Start with the SIE quickstart, browse the model catalog, or try SIE Cloud if you want an API without standing up the cluster yourself. For production topology, see deployment.
The bigger lesson: small models change the infrastructure problem
Moving from one large model to many specialised models is not simply an exercise in replacing API calls.
It changes the shape of inference.
Instead of:
one model + many requests
you increasingly have:
many models + many small requests
That affects routing.
It affects batching.
It affects GPU allocation.
It affects model loading.
It affects runtime selection.
It affects the relationship between AI engineers and infrastructure teams.
Small models can reduce the amount of compute required for individual tasks, but only if the infrastructure is capable of sharing that compute efficiently.
The challenge is therefore not just finding good small open models.
It is making a fleet of them behave like one coherent production system.
Frequently asked questions
What is a small language model?
There is no universal parameter threshold. In this context, a small model means a model compact enough to run on relatively accessible GPU hardware, often fitting on a single GPU, and typically selected for a specific task rather than maximum general-purpose capability.
Why use several small models instead of one large LLM?
Different models can specialise in tasks such as embeddings, reranking, OCR, extraction, classification and generation. Using task-specific models can provide better control over cost, latency and infrastructure, provided they meet the required quality threshold.
Can multiple AI models run on one GPU?
Yes. If the models fit within available GPU memory, several small models can share one GPU. Frequently used models can remain loaded while others are loaded and evicted dynamically.
What is shared-queue inference?
Shared-queue inference places incoming work into a common queue that GPU workers pull from. This differs from a central router assigning every request to a specific worker in advance.
Why is batching important for GPU inference?
GPUs generally process batches more efficiently than individual requests. Good batching increases hardware utilisation, but overly large batches can increase latency or memory use.
What is LoRA?
LoRA, or Low-Rank Adaptation, is a parameter-efficient technique for adapting a pretrained model by training a relatively small set of additional parameters rather than retraining the entire model.
What is the best first workload to self-host?
There is no universal answer, but embedding workloads are often attractive because the models are relatively compact, requests are easy to batch and output quality is straightforward to evaluate.
What is Superlinked Inference Engine?
Superlinked Inference Engine is an open-source inference server and production cluster for serving the models used by AI agents and applications through one infrastructure layer.