---
title: A Practical Guide for Choosing Models for Your AI Agents
description: "Key considerations and trade-offs for picking the embedding, reranking, extraction and generation models that fit your agent's quality, latency and cost targets."
canonical_url: https://superlinked.com/blog/choosing-models-for-agents
last_updated: 2026-07-10
---

Choosing models for an AI agent means matching each job your agent performs to a model that hits your quality and latency targets on hardware you can actually run. A model that wins a benchmark on paper is the wrong choice if it triples your p99 latency or does not fit on the GPU you have.

An agent is never a single model call. It is a pipeline of jobs: it searches and retrieves context, turns documents into Markdown, produces structured output, guards content, and drives the reasoning loop. Each of those jobs maps to one of four primitives that SIE serves (`encode`, `score`, `extract`, and `generate`), and each primitive has its own set of candidate models with very different quality, latency, and cost profiles. The [model catalog](/models) lets you rank every model by a benchmark and see how latency, throughput, and cost move as you switch hardware. This guide explains how to read those numbers and turn them into a decision.

<BlogSieCta />

## Eight dimensions that decide your model pick

| Dimension | Key Considerations | Trade-Offs / Recommendations |
| --- | --- | --- |
| **Job to Primitive** | Which stage of the agent loop needs a model: search, document to Markdown, structured output, guard content, drive the loop | Map the job to a primitive first: `encode` and `score` for search, `extract` for documents and structured output, `generate` for guarding and reasoning. Pick the job before you pick the model. |
| **Quality vs Cost and Latency** | Benchmark score against price per 1M tokens and per-request latency | Bigger models score higher but cost more and run slower. Pick the smallest model that clears your quality bar, not the top of the table. |
| **Benchmark Fit** | Whether the headline benchmark resembles your data | A model that leads on NFCorpus may trail on legal or finance text. Rank by the benchmark closest to your domain. |
| **Model Size vs Hardware** | Parameter count against the GPU you can provision | Sub-500M models run comfortably on an L4. Multi-billion parameter models need bigger accelerators like RTX-PRO-6000. Size drives both cost and the hardware floor. |
| **Representation** | Dense, sparse, or multi-vector output for retrieval | Dense is the default. Sparse helps when exact terms must match. Multi-vector raises recall at higher storage and compute cost. |
| **Modality and Language** | Multimodal, multilingual, and long-context needs | Text-only models are cheaper and faster. Add multimodal or multilingual only where the workload requires it. |
| **Generalist vs Specialist** | One broad model against a purpose-built one | Specialist extractors and rerankers often beat larger general models on their task, at a fraction of the size. |
| **One Model vs a Fleet** | How many models the agent needs live at once | A single cluster that loads models on demand and hot-swaps them keeps GPU utilization high and avoids one process per model. |

<BlogCatalogCta />

## Start with the job, not the model

Pick the primitive before you pick the model, because the primitive is decided by what the agent is trying to do at that step. The five jobs an agent runs map cleanly onto four primitives, and each primitive is its own table in the catalog with its own quality metric.

- **Search and retrieve** uses `encode` to turn queries and documents into vectors, then `score` to rerank the best candidates. Quality is measured in ndcg@10.
- **Document to Markdown** uses `extract` with OCR and document models that read a page and return clean, structured text.
- **Structured output** also uses `extract`, this time with entity, relation, and bounding-box models that pull named fields, PII, or regions out of content. Quality is measured in F1.
- **Guard content** uses `generate` with lightweight guard models that classify or filter input and output before it reaches a user or a tool.
- **Run the agent loop** uses `generate` with instruction-tuned models that plan, call tools, and produce the final response. Quality is measured in accuracy.

Deciding the primitive first collapses the field from hundreds of models to a shortlist you can actually compare, because you are now looking at one table with one metric instead of the whole catalog.

## Find your quality floor, then optimize cost and latency

Quality and cost move together on the catalog. Latency and throughput follow from the hardware control. The benchmark control sets the quality column; switching GPU type updates the cost and speed columns across every table. Read those controls as one trade-off.

The pattern is consistent across primitives. On the [model catalog](/models?task=encode) as of July 2026 (L4 hardware, NFCorpus ranking), the strongest embedding scores sit near 0.43 ndcg@10 on 7B-parameter models that cost roughly \$0.06 to \$0.16 per 1M tokens and answer in hundreds of milliseconds, while a 305M model like [`Alibaba-NLP/gte-multilingual-base`](/models/alibaba-nlp-gte-multilingual-base) scores about 0.37 ndcg@10 at around \$0.0040 per 1M tokens and tens of milliseconds of latency. That is a large cost and latency saving for a modest quality difference, and for many retrieval pipelines the smaller model is the correct choice.

The rule of thumb is to find your quality floor first, then take the cheapest, fastest model that clears it. Chasing the top of the table is only worth it when the last few points of the benchmark translate into real end-to-end wins for your agent, which is rarer than it looks once a reranker is in the pipeline.

Cost figures on the catalog are approximate, computed from list GPU prices (see the catalog cost caveat), so your real number depends on the provider you deploy SIE with. Treat them as a way to compare models against each other, not as a quote. Re-check the live table before you commit; scores and derived costs move as benchmarks and hardware options update.

## Rank by the benchmark that matches your domain

Rank models by the benchmark that resembles your data, not by the headline average, because retrieval quality is domain-specific. A model that leads on one corpus can fall behind on another, and the catalog lets you re-rank the entire table by any single benchmark for exactly this reason.

The available benchmarks are chosen to cover distinct domains, so use them as proxies for your own:

- **Finance:** FiQA2018 and NanoFiQA2018Retrieval
- **Scientific and academic:** SciFact and SCIDOCS
- **Legal:** LegalBenchConsumerContractsQA
- **Code and developer content:** StackOverflowQA and CosQA
- **Visual and multimodal:** Flickr30kI2TRetrieval and Vidore3HrRetrieval
- **General open-domain:** NFCorpus and CQADupstackPhysicsRetrieval

If your agent searches contracts, rank by the legal benchmark and ignore the general-domain ordering. The best general model and the best legal model are frequently different models, and picking on the wrong benchmark is one of the most common and most expensive retrieval mistakes.

## Size sets your hardware floor

Model size sets the hardware floor, so choose a size band that fits the accelerator you can provision before you compare quality. The catalog groups models into size buckets (under 100M, 100M to 500M, 500M to 1B, and over 1B) and lets you switch the benchmark hardware between an L4 and an RTX-PRO-6000 to see how the same model behaves on each.

Smaller models cost less to run and widen your deployment options. A sub-500M embedding or reranker model runs comfortably on a modest L4 and can serve high throughput at single-digit-millisecond to low-hundreds-of-milliseconds latency. Multi-billion parameter models deliver the top benchmark scores but need larger accelerators and cost more per request. When latency and throughput matter more than the last point of recall, a small model on cheap hardware usually wins on total cost of ownership.

Because SIE serves every model on the same cluster, you can start on smaller hardware for prototyping and move the exact same model onto larger GPUs for production without changing your retrieval code.

## Pick dense first; add sparse or multi-vector when the workload needs it

For the search job, the representation your embedding model produces matters as much as its benchmark score, because it changes what kinds of queries succeed. The catalog tags every model as dense, sparse, or multi-vector, alongside multimodal, multilingual, and long-context tags.

- **Dense** is the default. One vector per item captures semantic meaning and handles paraphrase and synonymy well. It is the right starting point for most agents.
- **Sparse** models keep term-level signal, which matters when exact tokens, codes, or identifiers must match. They pair naturally with dense vectors in hybrid search.
- **Multi-vector** models represent an item as several vectors, raising recall on hard queries and multimodal content at the cost of more storage and heavier scoring.

Layer the other tags on top of this choice. Add multimodal only if the agent searches over images or document pages, multilingual only if your corpus spans languages, and long context only if your documents genuinely exceed a standard window. Each of those capabilities narrows the field and often raises cost, so add them deliberately rather than by default.

## Retrieve wide with encode; rerank narrow with score

Use a cheap embedding model to generate candidates and a reranker to sort them. A two-stage pipeline beats a single-stage one on both quality and cost. On the [score table](/models?task=score) as of July 2026, reranker models measured with `score` reach materially higher ndcg@10 than bi-encoder embedding models measured with `encode`, because a reranker reads the query and document together instead of comparing precomputed vectors. The strongest rerankers sit near 0.69 ndcg@10, well above the roughly 0.43 ceiling of the [encode table](/models?task=encode).

Running a high-accuracy reranker over your entire corpus would be prohibitively slow and expensive. The standard pattern is to let a fast, cheap embedding model retrieve a large candidate set, then rerank only that shortlist with `score`. You get most of the reranker's quality on a small fraction of its cost, and you keep the heavy compute off your vector database.

The flow with SIE looks like this: `encode` the query, retrieve candidates from any vector database, then `score` the shortlist with a cross-encoder.

```python
from sie_sdk import SIEClient
from sie_sdk.types import Item
from qdrant_client import QdrantClient

sie = SIEClient("http://localhost:8080")
qdrant = QdrantClient(url="http://localhost:6333")

query = "How do I choose a reranker for my agent?"
q_vec = sie.encode("BAAI/bge-m3", Item(text=query), output_types=["dense"])

hits = qdrant.search(collection_name="docs", query_vector=q_vec.dense, limit=100)

items = [Item(id=str(hit.id), text=hit.payload["text"]) for hit in hits]
result = sie.score(
    "Qwen/Qwen3-Reranker-4B",
    query=Item(text=query),
    items=items,
)

hit_by_id = {str(h.id): h for h in hits}
for entry in result["scores"][:10]:
    hit = hit_by_id[entry["item_id"]]
    print(f"{entry['score']:.3f}\t{hit.payload.get('title', '')}\t{hit.payload.get('text')[:180]}")
```

This split lets you tune each stage independently: pick the embedding model for throughput and the reranker for accuracy, and change either one without touching the other.

<BlogCatalogCta />

## Reach for specialist extract and generate models first

For document reading, structured output, and guarding, a purpose-built model usually beats a larger general one. Reach for the specialist first. The `extract` and `generate` tables in the catalog are full of models that are small for their job precisely because they are focused.

For `extract`, the choice depends on what you are pulling out. Document-to-Markdown work uses OCR and document models that read a full page and return clean text. Structured-output work uses entity, relation, and bounding-box models: named-entity extractors, PII detectors, and region detectors that return typed fields rather than prose. On the catalog these are ranked by F1, and compact models around 110M to 450M parameters score competitively, which means field extraction rarely needs a large language model at all.

For `generate`, separate the two jobs the primitive covers. Guarding content is a classification-style task that a small, dedicated guard model handles cheaply and predictably, so you do not spend a large reasoning model on a safety check. Driving the agent loop is where an instruction-tuned model earns its cost, since it plans, calls tools, and writes the final answer. Even here the trade-off holds: on the catalog a 4B instruct model can match or edge out a 27B model on accuracy while running several times cheaper per token, so measure before assuming bigger is better.

## Serve the fleet from one cluster, not one GPU per model

An agent needs several models live at once, so the operational question is how you serve a fleet without a GPU per model or a redeploy every time you swap one. Model choice and the serving layer have to line up; that gap is often what separates a demo from a system you can run.

Expensive patterns to avoid in the hot path include running a high-accuracy reranker over large candidate sets, hosting many multi-billion parameter models on dedicated GPUs, keeping idle models resident when traffic is bursty, and frequent model churn that forces repeated restarts. Cheaper patterns to lean on include fast ANN candidate generation followed by batched reranking and batched `encode` calls at ingest for hybrid indexing, plus lazy loading so a model only occupies GPU memory when it is actually serving traffic.

SIE is built for the fleet case. Per the [engine docs](/docs/engine), it loads models on demand and shares one GPU across many models with LRU eviction under memory pressure, so you can host the embedding, reranker, extractor, guard, and instruct models your agent needs on constrained GPU capacity. That turns the model catalog into a mix-and-match set as your agent's needs change.

## Work through the per-job checklist

Selecting the models behind an agent is a set of per-job decisions, not one global pick. For each stage of the loop, work through the same short checklist:

- Which primitive the job needs: `encode`, `score`, `extract`, or `generate`
- Your quality floor on the benchmark closest to your domain
- Your latency and throughput targets on the hardware you can provision
- The representation, modality, and language the job actually requires
- Whether a small specialist clears the bar before you reach for a larger model
- How the model fits alongside the others your agent runs at the same time

The [model catalog](/models) is designed for exactly this workflow: filter by task and primitive, rank by your benchmark, switch hardware to see the cost and latency you would really pay, and use the [compare view](/models/compare) to put your shortlist side by side. There is no single best model for an agent. The best model for each job is the smallest, cheapest, fastest one that clears your quality bar on hardware you can run, and the best set of models is the one your inference layer can serve together without falling over.

[Get in touch](#contact) to talk through matching models to your agent's pipeline, or explore [SIE on GitHub](https://github.com/superlinked/sie).