Small Models Decide What Reaches Your Search LLM
Not every search query needs an expensive LLM. Yet many AI search and retrieval-augmented generation (RAG) systems send every request through the same costly pipeline, even when a cached answer, a simple classification or a smaller model could handle the work. By introducing specialised models for screening, semantic caching, query routing, retrieval and verification, developers can build more efficient AI search systems without sacrificing the quality that users expect.
Watch the discussion: Daniel Svonava (Superlinked) and Doug Turnbull explore how small, task-specific models can improve search relevance, reduce unnecessary LLM calls and make AI search infrastructure more efficient.
Explore the full presentation slides →
Why traditional RAG pipelines waste LLM compute
Most RAG applications follow a familiar process: receive a question, retrieve relevant documents, send those documents to a large language model and generate an answer.
The problem is that not every query requires the same amount of work.
Some questions have already been answered. Others need only a lookup or classification. Some require a straightforward answer from one document, while others need reasoning across several conflicting sources.
Sending every request to the same large model can create unnecessary costs, latency and operational complexity.
The alternative is to let smaller models make decisions before the search LLM is invoked.
Rather than replacing the LLM entirely, these models act as specialised components, determining which requests need generation, which evidence should be retrieved and how much reasoning is actually necessary.
How do small models improve an AI search pipeline?
A small-model search gateway can divide a request into six stages. Each stage performs a specific task, allowing the application to avoid unnecessary work and reserve expensive generation for requests that genuinely need it.
1. Screen: Check safety and sensitive information before generation
Before retrieval or generation begins, smaller models can classify incoming requests and identify potentially unsafe content.
Models such as Qwen3Guard can evaluate prompts against safety categories, while specialised PII detection and redaction models can identify and mask personal information.
This allows applications to apply safety policies and privacy controls before passing requests to external models or storing information in caches.
Screening is not infallible. Applications still need explicit policies, testing and appropriate safeguards around sensitive data.
2. Cache: Answer repeated questions without calling an LLM
Semantic caching uses embeddings to identify previously answered questions with similar meanings, even when their wording differs.
For example:
- “What is the default size of the multixact member cache?”
- “How big is the multixact member cache by default?”
Both questions are asking for the same information.
Rather than retrieving documents and generating a new answer each time, the system can use an embedding model to identify a suitable cached response.
However, semantic similarity alone is not sufficient to establish that an answer can safely be reused. Cache decisions should account for access permissions, document versions and the precise meaning of the question.
When implemented correctly, semantic caching can eliminate entire generation calls for repeat queries.
3. Route: Choose the right workflow and model for each request
Not every user question belongs in the same workflow.
A documentation question might need search and retrieval. An account request may require an application action. A complex analytical question might need a stronger reasoning model.
Specialised classification and decision models, including GLiNER2.5 Decide, can score requests against possible intents or tasks.
The application uses those scores to select the appropriate workflow.
A second decision can determine whether a cheaper model is capable of completing the task or whether a stronger model is required.
This is where small-model routing can produce measurable cost savings.
In Superlinked’s model-routing benchmark, a Qwen3 Reranker 4B routed 53.4% of requests to a cheaper model, reducing model costs by 36% while retaining 95.9% of the stronger model’s quality score.
The evaluation used 728 held-out requests covering science, mathematics, coding and general chat. Routing costs were included.
These results demonstrate the potential of task-specific model routing, but they are not a benchmark of a complete production search pipeline.
4. Retrieve and rerank: Give the LLM better evidence
The quality of an AI search answer depends heavily on the information retrieved before generation.
Embedding models identify documents that are semantically relevant to a query. Reranking models then evaluate candidate passages more closely to determine which best satisfy the user’s actual request.
Consider a legal research question asking for the final version of a regulation rather than an earlier proposal.
Both documents might contain very similar language. An embedding model can retrieve both, but a reranker can use the query’s specific instruction to prioritise the correct document.
The results can be significant.
In a Superlinked evaluation across eight retrieval benchmarks and 6,200 questions, Qwen3 Embedding 4B ranked the relevant passage first in 54.6% of cases, compared with 54.7% for Voyage 4 Large.
At the published API prices used in the comparison, Qwen3 Embedding 4B cost 0.12 for Voyage 4 Large.
That is comparable top-1 retrieval performance at half the listed token price, although other retrieval metrics and workloads can produce different rankings.
Reranking provides a further opportunity to apply the application’s own definition of relevance before sending evidence to the generation model.
5. Generate: Use the LLM only when it adds value
Once the request has passed screening, caching, routing and retrieval, the application has a clearer understanding of what needs to be generated.
A simple factual question may be suitable for a smaller language model. A more complicated request involving several sources, competing claims or multi-step reasoning may justify a more capable model.
The goal is not to eliminate large language models.
It is to stop using them for work that smaller, specialised models can perform effectively.
By reducing unnecessary generation and selecting models according to task complexity, applications can direct compute towards the requests where it creates the most value.
6. Verify: Check whether the answer is supported by the evidence
Generation is not necessarily the final step.
A response may sound plausible while containing claims that are not supported by the retrieved documents.
Groundedness models such as Granite Guardian can be used to evaluate whether a generated response follows from the supplied evidence.
If the response contains unsupported claims, the application can retrieve additional information, regenerate the response or decline to answer.
This is an important architectural pattern for production RAG applications, particularly in environments where factual accuracy and traceability matter.
The verification stage shown in the presentation is an application design pattern, not a claim that the complete workflow has already been benchmarked end to end.
What does this mean for AI search costs and performance?
The biggest opportunity is not necessarily finding one model that is dramatically cheaper than an LLM.
It is reducing the amount of work required to answer each request.
Semantic caching can eliminate unnecessary generation. Classification can prevent requests from entering the wrong workflow. Model routing can reserve expensive reasoning for harder questions. Embeddings and rerankers can improve the evidence presented to the generation model.
Together, these techniques provide a more flexible approach to building AI search and RAG systems.
However, individual benchmark savings should not be added together to predict the cost reduction of a complete application.
The real outcome depends on production traffic, cache hit rates, routing accuracy, retrieval quality, latency, serving costs and the number of requests requiring fallback or verification.
The right measure is the performance of the complete search request, not the performance of one model in isolation.
Serving specialised search models with Superlinked SIE
Using several task-specific models introduces a new infrastructure problem: how do you deploy, manage and serve those models efficiently?
A production search application might use different models for safety, embeddings, classification, reranking, generation and groundedness checks.
Each model may have different computational requirements and traffic patterns.
Superlinked’s Small-model Inference Engine (SIE) provides a unified inference layer for serving these specialised models.
With SIE, developers can:
- Serve multiple model types through one inference interface, including embedding models, rerankers, classifiers and supported language models.
- Use workload-aware batching and worker pools to manage inference across different tasks.
- Separate interactive search from bulk indexing, allowing each workload to use appropriate batching and capacity policies.
- Self-host inference inside their own cloud infrastructure for greater control over deployment and data processing.
- Use hosted inference when they want to call models without managing GPU infrastructure themselves.
SIE handles model serving, while your application remains responsible for the actual search workflow, including caching, document retrieval, access controls, routing thresholds and verification policies.
This separation allows engineering teams to focus on improving search quality and application behaviour rather than maintaining a collection of independent model-serving systems.
Deploy SIE in your own infrastructure →
How to introduce small models into an existing RAG application
You do not need to rebuild your entire search pipeline to benefit from specialised models.
Start by identifying one expensive or repetitive decision that your application currently delegates to an LLM.
That might be classifying the intent of a query, selecting between two generation models, reranking retrieved passages or identifying repeated questions.
Replace that operation with an appropriate specialised model and test the result against representative requests that were not used to tune the model.
Measure answer quality, end-to-end latency, cost per request and fallback rates.
If the new approach improves the complete application, keep it. Then evaluate the next opportunity.
This incremental approach makes it easier to understand where smaller models create genuine value and where a more powerful model is still necessary.
Frequently asked questions
Can small language models replace LLMs in RAG?
Small models can replace certain tasks within a RAG pipeline, including classification, routing and some generation workloads. They do not necessarily replace the need for a larger reasoning model. The objective is to use the most appropriate model for each task.
What is the difference between embeddings and reranking?
Embedding models represent queries and documents as vectors, allowing a search system to retrieve semantically similar content. Rerankers examine candidate results more closely and score their relevance to a specific query or instruction. Using both can improve the evidence selected for generation.
How does semantic caching reduce LLM costs?
Semantic caching identifies previously answered questions that are equivalent in meaning. When an answer is safe and appropriate to reuse, the application can return it without making another generation call. The effectiveness depends on repeat traffic, cache validity and similarity thresholds.
What is small-model routing?
Small-model routing uses a specialised classifier or scoring model to choose how an incoming request should be processed. It can select workflows or decide whether a request should go to a cheaper or more powerful LLM.
Does Superlinked SIE automatically orchestrate a RAG pipeline?
No. SIE serves the models used by an application. Developers retain control over retrieval, caching, routing logic and output policies. SIE provides the infrastructure needed to execute the model calls behind those decisions.
Build search systems that spend compute where it matters
The next improvement in AI search does not have to come from a larger generation model.
It can come from making better decisions about when generation is needed, which documents should be retrieved and which model is appropriate for the task.
Small, specialised models make those decisions possible.
Watch the full discussion with Daniel Svonava and Doug Turnbull, then explore the accompanying slides for architecture diagrams, model comparisons and detailed benchmark results.