Improve the Agent System Before You Fine-Tune the Model

When an agent fails, the model gets blamed first. Teams swap providers, raise the reasoning setting, or start a fine-tune. The failure often sits elsewhere.

The tool description may be vague. The retrieved evidence may omit the answer. Conversation state may have been truncated. A serving profile may apply the wrong context limit. None of those faults disappear when the team buys a larger model.

The surrounding agent system deserves the same experimental discipline as the model.

The system defines the job

An agent system includes the system instructions, tool schemas, retrieval process, state management, output checks, and the serving profile used for the model call. Together, these parts determine what the model sees and which actions it can take.

A tiny model with a clear tool schema can beat a stronger model forced to guess at ambiguous fields. Good retrieval can remove the need for long reasoning. A verifier can catch unsupported claims after generation and send only doubtful answers to a stronger path.

This is why system work can produce a larger gain than fine-tuning. It changes the problem presented to the model.

Start with the smallest complete loop

Build a minimal agent that can complete the task from end to end. Give it one or two tools with plain descriptions. Keep state explicit. Return raw errors instead of hiding them behind retries.

Then run a small set of models through the same loop. Include at least one inexpensive open model and the stronger model you already trust. Hold prompts, tools, and test cases constant.

If every model fails the same cases, inspect the surrounding system. If only the smaller models fail, the task may require a stronger model or a targeted fine-tune. That distinction saves weeks of random changes.

Complexity needs to earn its place. Add planning, memory, retries, or subagents only when the evaluation shows a repeatable gain. Each extra mechanism creates more states to debug and more ways for costs to grow.

Version the whole evaluated release

A model name cannot describe an agent release. Version the model revision, prompt, tool schema, retrieval configuration, state rules, and serving profile together.

Change one variable at a time. Replay the same independent cases. Record task success, end-to-end latency, and cost per successful task. A lower token bill means little if retries erase the saving.

Raw traces matter. An aggregate score can tell you that quality fell from 84% to 79%. It cannot tell you whether the agent chose the wrong tool, received empty search results, or lost a required field during truncation.

Give the team access to the input, selected tool, tool result, generated answer, verification outcome, timing, and model profile for each case. Research agents can inspect those traces and suggest experiments, but their suggestions should still face the same held-out evaluation.

Sensitive data changes where the loop can run

Automated agent research becomes more useful when it can see realistic traces. In regulated or private systems, those traces may contain client data, document text, or internal identifiers.

Self-hosted models let the research loop run next to that data. The team can generate synthetic labels, inspect failures, and test alternative prompts without sending the raw material to a third-party model API.

That does not remove governance work. Access rules, retention, redaction, and audit logs still apply. It does make the boundary clearer: the models and traces stay inside infrastructure the organisation controls.

Fine-tune after the system stops moving

Fine-tuning works best when the task interface has settled. If tool schemas and prompts change every week, training examples encode a moving target.

First remove system failures. Then collect the remaining model failures into a stable dataset. Fine-tune against that dataset and compare the adapter with the unchanged baseline agent.

The model should receive credit only for gains it produced. That sounds obvious. Agent projects lose this discipline quickly because several parts of the system change at once.

What teams ask before changing the model

What surrounds an agent model?

The surrounding system gives a model instructions, tools, retrieved evidence, conversation state, output checks, and runtime settings. It determines how the model performs a task.

Why improve the agent system before fine-tuning?

Many failures come from ambiguous tools, missing evidence, lost state, or incorrect context limits. Fine-tuning the model will not fix those faults. A system-level evaluation shows which component actually failed.

How should teams evaluate an agent system?

Version the model, prompt, tool schema, retrieval settings, state rules, and serving profile together. Replay held-out cases, change one variable per experiment, and inspect raw traces alongside task success, latency, and cost.

When should an agent model be fine-tuned?

Fine-tune after the task interface has stabilised and the remaining failures consistently point to model behaviour. Train on those failure categories, then compare the adapter with the same unchanged agent system.