--- title: Should You Use an LLM for Every Agent Decision? description: Small encoder models can route requests, detect sensitive data, rank evidence, and verify outputs faster and more predictably than a generative LLM. canonical_url: https://superlinked.com/blog/decision-models-for-agent-routing last_updated: 2026-10-22 --- An AI agent makes a surprising number of decisions before it writes a sentence. It has to identify the user's intent, decide which tool to call, find sensitive data, rank evidence, and judge whether the final answer can be returned safely. Many teams send every one of those jobs to the same generative model. That works, but it makes the system slower, more expensive, and harder to control. A large model may spend hundreds of generated tokens explaining a decision that the application needs as one label. For bounded decisions, an [encoder model](/decision-models) is usually the better tool. ## A decision is a scoring problem A generative model produces an answer token by token. An encoder reads the input in one forward pass and returns a label, span, score, or vector. There is no chain of thought to generate and no verbose response to parse. That difference matters inside an agent. A routing step does not need prose. It needs a score for labels such as `research`, `support`, or `transaction`. A redaction step needs the character spans containing names and account IDs. A retrieval step needs relevance scores for passages. Models such as [GLiClass and GLiNER2](/classify) are built for this kind of work. The application receives structured outputs that control its next action. It can also set a threshold and escalate uncertain cases to a stronger model or a human reviewer. This creates a clean division of labour. Encoders handle frequent, bounded decisions. A generative model handles synthesis, open-ended analysis, and writing. ## Small models improve when the task is specific The strongest case for a small model appears when a team has a stable decision and a representative dataset. In one unfair-contract-clause test from [the presentation](/smarter-cheaper-agents-with-slms), a fine-tuned 150 million parameter ModernBERT model reached 62.2% recall at a 95% precision floor. A LoRA-tuned Qwen3 4B model reached 62.7%. A prompted GPT-6 Luna model, given the rulebook and 16 labelled examples, reached 0.7% recall under the same precision constraint. The result does not prove that a 150 million parameter encoder beats a general model on legal work. It proves something narrower and more useful: training on the dataset's annotation rules can beat prompting when the output is a precise, repeated decision. The dataset contained 5,532 training clauses. The held-out test contained 3,882 clauses, including 402 unfair clauses. Those details matter because deployment thresholds must be calibrated on independent data. A benchmark result copied from another team cannot set your production cutoff. ## Calibration turns scores into policy A classifier returns probabilities or scores. The application still needs to decide what happens next. Suppose a model checks whether a contract clause needs review. A low threshold catches more risky clauses, but sends more harmless clauses to reviewers. A high threshold reduces reviewer load, but can miss more risk. The right cutoff depends on the cost of each error. Set the threshold with held-out cases that reflect production. Track precision and recall at the operating point you plan to use. Then define an uncertainty band. Scores above the upper threshold go to review. Scores below the lower threshold can take the safe default. Everything in between can move to a larger model or a reviewer. That policy should be versioned with the model. Changing a threshold can alter the product as much as changing the weights. ## Start with one branch in the agent Do not rebuild the whole agent around small models at once. Pick one high-volume branch with a clear output. Intent routing, PII detection, and evidence ranking are good candidates. Create a held-out evaluation from real traffic. Define acceptable false positives and false negatives. Compare a prompted general model with an off-the-shelf encoder, then fine-tune only if neither passes. The smallest model that passes that evaluation has earned the job. Keep the general model for work that still benefits from generation. ## Questions teams ask about decision models ### What is a decision model in an AI agent? A decision model reads an input and returns a bounded output such as a label, span, score, or vector. Agents can use these outputs to route requests, redact sensitive data, rank evidence, and decide whether to escalate an answer. ### Why use an encoder instead of an LLM for routing? An encoder scores the available routes in one forward pass. It avoids per-token generation, returns a predictable structure, and can be calibrated against the cost of routing errors. ### How do you calibrate a classifier for production? Run the classifier on held-out production-like cases, sweep the decision threshold, and select the precision and recall trade-off your product can accept. Keep uncertain cases in an escalation band. ### Can a small classifier outperform a frontier model? Yes, on a narrow task with stable labels and good training data. The comparison must use the same held-out cases and the same error constraint. It does not imply that the classifier performs better on open-ended work.