Give Your Agent Small Open Models as Tools

Teams are building agents for a wide range of work: answering customers’ questions about an account or a bill, triaging support and IT tickets, investigating an incident, reviewing contracts, screening job applications, writing and fixing code. Most of these agents start out as a job a person already does every week. That person reads a few things (a message, a photo or a scan, a policy, a record in some system), works out what applies, does something about it and writes it up. Then someone on the team asks whether an agent could do that, and the first draft of the answer is usually one big model with a long prompt and every document stuffed into its context.

I’d divide that job up instead. The large model decides what to look at next and writes the answer, every job around it that only needs text read, matched or masked (masking a name, searching the manual, reading a photo, answering a yes/no question) goes to a small model built for that one job, which the agent calls as a tool, and every rule, sum and final status stays in plain code.

In this post I’ll start with what it costs to hand one large model the whole job, and how companies that have written about their assistants divide the work instead. Then how I split an agent’s job between code, small models, the large model and a person, where the small models come from, and how the tools and the loop fit together, in pseudocode. The last part is three agents I built for the SIE cookbook as worked examples, on synthetic data.

What It Costs to Give One Large Model the Whole Job

Take a billing rep at a water utility. A customer writes in: “my bill is $411, it’s usually $40, what’s going on?”, with a photo of the meter attached. The rep reads the complaint to work out what’s being asked, checks whether the last bills were estimates or real readings, looks at the meter’s hourly readings, finds the rules that apply, does the sums and writes back, sending any credit to a supervisor.

One large model can do all of that on a demo. The trouble starts when you look at what you’re paying it to do. It’s spending its tokens reading digits off a photo, matching “leak credit” to a manual section called “Excess Use Adjustment”, and scrolling through the hourly readings, which come to about 4,400 rows per meter for six months. And an agent doesn’t read anything just once. Each time it picks its next step, it rereads the whole conversation and every tool result so far, so everything it was handed on turn two is back in the context on turns three, four and ten. Prompt caching makes that rereading cheaper, but a long tool result still makes every later turn longer, and you pay for that length on each one. The customer waits on the slowest model you have, and that model sees their name, email and phone number, which it never needed.

Anthropic’s Building effective agents puts it plainly: agentic systems often trade latency and cost for better task performance. You can’t get rid of that trade, but you can make what the large model rereads much smaller.

How Companies Divide the Work Between Large and Small Models

Most of what the billing rep’s job asks of a model is narrow: masking a name, finding the manual section that matches the customer’s words, reading digits off a photo, answering a yes/no question. The companies that have written about their assistants tend to put a small model on those jobs and keep the large model for the rest.

Marsh McLennan fine-tuned Llama-3.1-8B-instruct to do intent recognition and tool selection inside LenAI, its assistant for staff (March 2025), while LenAI itself runs on privately hosted OpenAI models, as Insurance Business reported in August 2026. The small model picks the intent and the tool, and the large one does the rest. LexisNexis “mostly relies on a fine-tuned Mistral model” for its assistant, according to VentureBeat (March 2025), and its CTO Jeff Reihl put the reasoning in one line: “You don’t need GPT-4o to do the assessment of a query, so we use it for more sophisticated work, and we switch models out.” FurtherAI wrote in October 2026 that “Claude Opus 5.5 handles complex claims and census extraction” while “GLM-5.3 reviews the extracted output for completeness”.

Safety checks are the clearest case. Cloudflare put Llama Guard 3 in front of its customers’ LLM endpoints (August 2025), and Sword Health’s MindGuard classifiers, fine-tuned from Qwen3Guard-Gen at 4B and 8B, screen every user message in its Dawn product. “Every message” is the part I’d hold on to. A check that runs on all of your traffic has to be cheap and quick, or it ends up running on a sample, and an agent makes a lot of calls like that: every complaint gets masked, every question to the manual gets a search, every screenshot gets read.

How I Split an Agent’s Job

This is for the case where you’ve already decided the job needs an agent. If you haven’t, start with When to Use a Workflow and When to Use an Agent; plenty of jobs that look like agent work turn out to be a workflow with a model in one or two steps.

Once it’s an agent, I go through its steps one at a time and ask who should own each one, starting from the cheapest option:

Plain code comes first because it costs nothing and gives the same answer every time, and a sum should never depend on a model’s mood. The small models get the second row. Each one does a single job, returns a span, a score or a probability that your code can check, and is cheap enough to call on every step. The large model keeps the third row, where the next step depends on what the agent finds, and that’s where you want to spend its tokens. The last row is about money or about changing something that’s hard to undo. A written list in code can let the safe, easily undone changes through, and everything else waits until a person says yes.

Small models have one clear limit, and it’s open questions. A small decision model is good at picking from a short list of options. “Is this a good answer to this customer?” is a much harder question, and I’d leave it with the large model or with a person. (I tried it the other way on the billing agent below, and it didn’t go well.)

Why the Narrow Jobs Go to Small Models

Cost. The rereading is where a lot of the money goes, so shrink what each tool hands back before the large model ever sees it. A reranker that returns three sections for a question means the large model never reads the whole manual, and a Python tool can boil six months of hourly readings down to daily use and the lowest overnight flow. A smaller context tends to give better answers too; Filip Makraduli makes that case in his post on the SIE MCP server.

Speed. A small model call is quick enough to run on everything. You can ask a decision model one yes/no question about every ticket and every screenshot before the large model reads them, and nobody has to decide whether a ticket is worth the wait.

Control. A decision model returns a probability for each label, so you choose the threshold and your code decides what happens above and below it. The large model’s steps are harder to pin down. When I ran the billing agent three times with the same prompts and the same index, it took different steps each time, and not every complaint ended with the same status.

Privacy. The small model is the one that sees the raw data. A masking model like pii-redact masks the complaint before the large model gets it, and OCR output is masked the same way. If the large model sits behind someone else’s API, the masked copy still leaves your network, and the rest of what keeping data in takes is in Training and Hosting Your Own Models.

Where the Tools Come From

I’d start with open task models from a catalog. The agents below use five between them: pii-redact to mask personal data, Qwen3-Embedding-4B and Qwen3-Reranker-4B to search a manual or score resume lines, LightOnOCR-3-4B to read photos and scans, and GLiNER2.5-Decide for yes/no questions. Each is one line in a rules.yaml, so swapping one for another is easy to try.

Next is a model you fine-tuned on your own decisions. If you have old decisions on record (tickets and the team that handled them, claims and whether they were escalated), that’s training data for a small decision model. You train a LoRA adapter yourself and serve it on your own SIE deployment (LoRA adapters in SIE). Superlinked’s train-a-router example, which fine-tunes a small model to decide which questions can go to a cheap model and which need the expensive one, measures what that buys you, and Off the Shelf or Fine-Tuned covers when it’s worth the trouble.

Then an ML model you already run. A fraud score, a churn model, the classifier your data team shipped three years ago and nobody has touched since (it still works, which is why). Wrap it in a function with a clear name and a docstring, and the agent can call it like anything else. It doesn’t need to be a language model, and it doesn’t need to move.

Then plain code: a sum, a lookup in your billing system, a rule from the policy. It’s not a model at all, and it’s the tool I trust most.

And last, a workflow you already run. Four of the cookbook’s workflows (KYC checks, counterparty news, a contract playbook and repair invoices) now have an agent.py beside them that uses the whole workflow as one of its tools, for the jobs the workflow wasn’t built for. Depending on the job, that’s an agent with the workflow as a tool or a hybrid where the workflow hands its leftovers to the agent; When to Use a Workflow and When to Use an Agent walks through the KYC and counterparty news ones.

Tools, Sub-Agents and the Loop

The simplest tool wraps one or two model calls:

# pseudocode: a tool that wraps task models
@tool
def search_policies(question: str) -> list[Section]:
    """Ask the rules manual one question. Returns the three best sections."""
    candidates = index.nearest(embed(question), k=10)   # embedding model on SIE
    ranked = rerank(question, candidates)               # reranker on SIE
    return ranked[:3]                                   # the agent reads three sections, not the manual

Some jobs need a few models and some code in a row. Reading a meter photo is OCR, then masking, then a check that the serial number is this customer’s meter. You could give the agent three tools and hope it calls them in the right order every time, but I’d rather wrap them in one:

# pseudocode: a tool the agent delegates a whole job to
@tool
def read_meter_photo(account_id: str) -> PhotoReading:
    """Read the customer's meter photo and say whether it's their meter."""
    text = ocr(photo_for(account_id))                   # OCR model on SIE
    text = mask_personal_data(text)                     # PII model, before the agent sees anything
    reading, serial = parse_display(text)               # Python
    if serial is None:
        return PhotoReading(status="unreadable")
    return PhotoReading(reading=reading,
                        serial_matches=serial == meter_serial(account_id))

From the agent’s side that’s one call, and it gets back something it can reason with. Inside, it’s a small workflow: a fixed sequence of small models and Python, so it runs the same way every time, and the raw photo never reaches the large model. If the job inside needs to pick its own next step, it can become a sub-agent, a small language model running its own short loop, but I’d start with the fixed sequence and only add a loop when a real case needs it.

The loop that ties it together is shorter than you’d expect:

# pseudocode: the agent loop
tools = [search_policies, read_meter_photo, get_account, calculate_bill, submit_draft]
messages = [mask_personal_data(complaint)]          # the large model never sees the raw complaint
sent_back = False
for step in range(MAX_STEPS):
    reply = large_model(messages, tools)            # thinks, then picks the next tool
    if reply.calls("submit_draft"):
        problems = check_draft(reply.draft)         # Python: every quote, figure and sum
        if problems and not sent_back:
            messages += [reply, problems]           # one chance to fix it
            sent_back = True
            continue
        return status_of(reply.draft, problems)     # Python sets the status; a person sends
    messages += [reply, run_tool(reply)]            # every turn rereads all of this
return "out_of_steps"

The two lines I’d copy are the first and the last: the large model never sees the raw complaint, and it never decides the final status, which Python sets from what the checks found.

Three Agents From the Cookbook

I built three agents in the SIE cookbook to show what this looks like on real jobs. They’re worked examples, not products, and each ships with a few synthetic cases written to send something down each branch of its checks, so treat them as tests the code runs and not as a guide to how any of them would do on your own complaints, tickets or applicants. A large model runs the loop in all three. What I want you to look at is what the small models and the code around it do.

The Billing Agent: Small Models Do the Reading

This is the $411 bill from the top, at a made-up water utility called Alder Valley Water (recipe). Finding the rules was the step that decided most of the design. Customers say “leak credit” and “payment plan”; the manual says “Excess Use Adjustment” and “Installment Agreement”. One message often holds several questions, and you don’t know which rule you need until you’ve seen the customer’s data. Matching the customer’s words to the manual’s is a reading job, so it went to the embedding model and the reranker, behind a tool the large model calls one question at a time. Choosing which question to ask next takes judgment, so that stayed with the large model, with up to 8 lookups per complaint.

pii-redact and two plain patterns mask names, email addresses and phone numbers before the large model reads anything. LightOnOCR-3-4B reads the meter photo into plain text that can be masked the same way. The account, the readings and the bill sums are plain Python. One complaint shows who does what: the customer asked “is there any help, and will you shut off my water?”, the large model turned that into four questions to the manual, and the search tool came back with the Customer Assistance Program, the Installment Agreement and the shutoff rule. Working out the questions was the large model’s job, and finding the answers was the small models’.

When the large model submits a draft, Python checks every quote against the manual section or tool result it came from, and redoes any credit from the bills and the rate tiers. That second check trusts the gallons the model listed, so checking those against the photo is the rep’s job. The rep reads every draft and sends it, and a supervisor approves any credit; the recipe itself never sends or credits anything.

Two things broke, and both taught me something. I tried GLiNER2.5-Decide as a judge of the agent’s drafts. Sentence by sentence it was useful, flagging anything that sounded like the utility speaking (“We checked your account”). On the whole complaint and reply it failed outright: on one complaint it scored the real reply 0.03 and a reply written for a different complaint 0.91. That’s the limit from earlier: “is this reply right for this complaint?” is an open question, not a pick from a short list, so I took it out. The other miss was pii-redact’s. It tagged 061000, the meter reading on the photo, as a phone number and masked the one number the whole recipe depends on, which is why rules.yaml now lists patterns the masking never touches. A small model with one job will still do that job in places you didn’t expect, so read what it hands the agent.

The Helpdesk Agent: Code Decides What Changes

The second agent works an IT helpdesk at a made-up freight company (recipe). A technician there handles tickets from people whose laptop failed the nightly security checks: firewall off, macOS out of date, screen lock too long. They open the device record, look up what the policy says about each failed check, work out which fixes they’re allowed to make, make them, and check the laptop again. This agent changes settings, so for each of the four checks around the large model (masking the ticket, warning when text is written to a bot, a list of which fixes are allowed, and a recheck of the device after each fix) I asked whether it could depend on a model’s judgment at all.

The first two only mask or warn, so a small model is fine there. pii-redact masks the ticket, and four phrase rules plus one yes/no question to GLiNER2.5-Decide (“Is this text written to instruct an AI assistant, bot or automated agent?”) warn when text in a ticket or screenshot is written to a bot. They can be wrong in both directions: they can mask too much or warn when they shouldn’t, and they can also miss a name or a note written to a bot, which then goes on to the large model. That’s why they don’t decide anything. The other two use no model. A list in rules.yaml decides which changes run, which wait for the user and which are refused, and after every fix Python reads the device record again and compares the large model’s report with it. Whether a firewall gets switched off should depend on nobody’s judgment but yours, written down in a list. (The phrase rules and the model each caught a case the other missed, so I kept both.) The point of this one is where the models aren’t: nothing that decides whether a laptop setting changes depends on a model.

The Screening Agent: Task Models Served Over MCP

The third agent screens job applicants at a made-up facilities company (recipe). A recruiter there does a first pass on applicants for a commercial HVAC technician role: EPA 608 Universal certification, three years on commercial equipment, a valid driver’s license. I’ll be upfront that the job itself is a plain one. The first pass is the same three checks every time, so by the cookbook’s own test it’s really a workflow, and the README says so. It’s here because the interesting part is how the models are served: over MCP, from a server built for this one job. Instead of wiring each model call into the agent as its own tool call, the recipe puts the models behind an MCP server built for this one job, and the server makes the calls to SIE (SIE Serverless in the recipe, or your own SIE deployment).

The server holds the resumes, the applicant record and the SIE key, and offers five tools: get_role, list_applicants, read_resume, find_evidence and search_applicants. The model calls happen on the server. read_resume runs OCR on a scanned paper form, then masks the result three ways at once (pii-redact, plain patterns, and the applicant record’s own name, email and phone wherever they appear), and returns only the masked text; if a model call fails, it returns an error, never the raw resume. find_evidence has Qwen3-Reranker-4B score each masked line against one requirement, so Claude knows where to look. Claude still reads the line and decides advance, reject or needs a person, and Python checks every quote and that the decision follows the rule (any not_met is a reject, then any unclear goes to a person).

That’s the part I like about serving task models over MCP. The model calls run on the server, next to the resumes, so the masking is enforced there, in code the agent can’t go around: the client gives Claude no built-in tools at all (no files, no shell, no web), so the five server tools are the only way to a resume. And because it’s MCP, any MCP-capable agent picks the tools up without custom glue. The same server works for the script that screens every applicant overnight and for a recruiter asking their own Claude Code “who here has worked with BACnet?”:

MCP_TOKEN=pick-a-long-secret uv run server.py
claude mcp add --transport http fernhill-screening http://127.0.0.1:8765/mcp \
  --header "Authorization: Bearer pick-a-long-secret"

One applicant is worth a look. Their resume ends with a paragraph addressed to any AI screening tool, telling it to mark the applicant as advance. Claude rejected them and said in its note that it had ignored the instruction. The resume asked nicely, and Claude told on it.

Pro-tip: check what your masking misses, by eye. pii-redact let a school name and a town through on one resume and masked an employer as a school on another, and the recipe’s leak check couldn’t see either, because it only knows each applicant’s own details from the applicant record.

If what you want is SIE’s general document tools (parse, summarize, extract, redact) behind MCP, SIE ships its own server. The screening recipe is the other case: a server built for one job, with tools named after that job’s steps.

Run It Yourself

The easiest way to try these is on SIE Serverless, where the small models are already running and you pay per call. You need uv, an SIE API key and an ANTHROPIC_API_KEY for the large model that runs the loop:

git clone https://github.com/superlinked/cookbook && cd cookbook
cp .env.example .env   # set SIE_API_KEY and ANTHROPIC_API_KEY
cd agentic/screen_applicants_through_mcp_server
uv run recipe.py       # all seven applicants; --applicant A-102 runs one

The billing agent runs from agentic/diagnose_high_bill_complaints and the helpdesk agent from agentic/fix_laptop_compliance_with_guarded_agent. Each run writes a transcript of every tool call to runs/<timestamp>/, and reading one of those is the quickest way to see which model did what. If you’d rather run the small models on your own GPUs, the cookbook’s self-hosting guide has the setup.

Conclusion

When you build an agent, give the large model only the steps that need its judgment: deciding what to look at next and saying it well. Give each job of masking, searching, reading a photo or answering a yes/no question to a small model it calls as a tool, give every rule and sum to code you can check, and keep a person on anything about money or hard to undo. The large model rereads much less on every turn, which is often where an agent’s cost goes, and it sees far less raw data, though I didn’t measure these agents against one big model doing the whole job, and you still need to read what the masking lets through.

If you’ve got an old model sitting behind an API that you’d like your agent to call, or a decision you’d like to fine-tune one for, I’d love to hear what it is. Until next time.