Building Legal AI Agents With Small Models: Lessons From an AI-Native Law Firm

Can small AI models perform real legal work?

For many repetitive, well-defined tasks, the answer is yes. Document classification, entity extraction, contract search and sensitive-data detection don’t necessarily require a frontier LLM.

The challenge is knowing which tasks to automate, how to evaluate their accuracy and how to combine specialised models into a reliable production workflow.

We discussed these challenges with Milo and Marianna Brezinova, the CEO and CTO of Fellow, an AI-native law firm serving startups and investors. They joined Daniel Svonava, CEO of Superlinked, to discuss how they’re automating their firm, why general-purpose AI tools have limitations and where smaller, self-hosted models fit into their plans.

How an AI-native law firm automated 200 hours of work per month

Fellow offers an interesting example of what happens when a law firm treats automation as part of its infrastructure rather than another software subscription.

With just three lawyers and a CTO, the firm has grown to nearly 300 clients in approximately 18 months.

One of its biggest achievements is document filing. Across more than 200 clients, Fellow processes over 1,000 files each month through an automated system, saving an estimated 200 hours of manual work.

The firm has also automated much of its invoicing and time tracking.

But getting there required moving beyond conventional workflow automation tools.

Fellow initially used n8n to connect services and automate processes. As its filing system became more complex, maintaining interconnected workflows became increasingly difficult. The team eventually transitioned towards a custom Python-based architecture.

Now, as Fellow explores automating more substantive legal work, it faces a different set of technical challenges:

• How can AI process documents without introducing unacceptable errors?
• How can the firm avoid becoming dependent on a single model provider?
• How can it control inference costs as automation scales?
• Can specialised open-source models perform individual legal tasks reliably?
• How can confidential client information remain under the firm’s control?

These are also the problems that Superlinked’s inference infrastructure for legal AI is designed to help engineering teams address.

Why legal AI needs more than a general-purpose LLM

“We are an AI law firm that doesn’t like to use AI.”

That statement from Fellow’s CEO captures one of the biggest challenges in legal AI development.

Legal work requires exceptional precision. Two clauses that appear semantically similar can have materially different legal consequences.

A single word can change the allocation of liability or the rights of a contracting party.

And when an AI model generates a convincing but incorrect interpretation, checking its work can take longer than performing the review manually.

Fellow’s approach is to make as much of the workflow deterministic as possible, introducing AI only where it adds value.

For legaltech engineers, this suggests a useful architecture:

Deterministic software handles rules, calculations and validation. Specialised AI models handle bounded tasks involving unstructured information. Larger reasoning models and lawyers handle complex interpretation and judgment.

Instead of asking a single model to review an entire contract, the application can break the process into smaller operations.

These include reading the document, extracting entities, identifying relevant clauses, retrieving precedents, comparing terms and producing a structured summary.

The advantage is that each operation can use a model selected and evaluated specifically for that task.

And many of those models can be substantially smaller than the frontier LLMs typically used for legal AI.

Four legal AI workflows you can build with small models

1. Contract review with specialised AI models

The problem: Reviewing commercial contracts requires multiple operations, but using a general-purpose LLM for every step can introduce unnecessary inference costs, latency and complexity.

A contract review system might need to identify liability caps, termination rights, governing law, payment obligations and unusual contractual provisions.

Rather than asking one large model to perform the entire review, developers can build a pipeline around specialised model tasks.

For example:

1. Read the agreement: Use PDF-to-Markdown models to convert scanned agreements into machine-readable text while preserving document structure.
2. Extract contract information: Use document field extraction and named entity extraction to identify parties, dates and key information.
3. Find relevant clauses: Use semantic search to retrieve contractual provisions and precedents.
4. Prioritise evidence: Use reranking models to identify the most relevant clauses before passing them to a reasoning model.
5. Generate structured findings: Use structured output to produce review results in a format the application can validate.
6. Review exceptions: Escalate ambiguous or consequential findings to a qualified lawyer.

The key is that the reasoning model doesn’t need to perform every preceding task itself.

A working example: Superlinked’s contract review agent

Superlinked has published a self-hosted contract review agent demonstrating how this approach works.

The application brings together ten model-driven jobs, including document triage, OCR, vision, embeddings, reranking, entity extraction, text-to-SQL, reasoning, orchestration and safety checks.

Rather than integrating a different hosted inference provider for every operation, the example runs its models through a single Superlinked Inference Engine (SIE) cluster.

The architecture separates investigation from synthesis:

• An Investigator agent gathers evidence through a restricted set of tools.
• A Synthesizer agent produces a structured review from those findings.

The system also includes bounded database access and an execution ledger that records model operations.

It is a runnable engineering example using real SEC-filed commercial contracts, not a claim that contract review can be delegated to AI without oversight.

For legaltech CTOs, it demonstrates how a complex legal agent can combine multiple specialised models without requiring a separate inference service for every task.

Explore the contract review agent and its architecture

2. Automated document classification and filing

The problem: Law firms manage thousands of incoming documents, but filing them correctly requires identifying the relevant client, matter, document type and status.

This is particularly interesting because Fellow already has a functioning document filing system.

During the interview, the team discussed whether its existing workflow could be a suitable candidate for evaluating self-hosted open-source models.

Why start with document filing?

Unlike an open-ended legal reasoning task, classification produces a relatively constrained result.

An incoming document might need to be categorised as an employment agreement, NDA, financing document, board resolution or client correspondence.

That makes it a natural fit for small text classification models.

A potential architecture could work as follows:

1. An incoming document is processed and converted into machine-readable text.
2. A classification model identifies the document type.
3. An entity extraction model identifies relevant names and organisations.
4. Deterministic application logic checks the proposed client or matter assignment.
5. The document is filed automatically when the required checks pass.
6. Ambiguous cases are sent for human review.

Why it’s a good first project

Fellow already has historical filing decisions that could be used to create an evaluation dataset.

Developers could compare the classification decisions of a smaller model against previously approved filing results.

This makes it possible to measure accuracy, identify recurring errors and determine whether the new model offers sufficient quality at a lower operating cost.

It also provides a practical opportunity to test zero-shot classification before investing in domain-specific fine-tuning.

Fellow is exploring this direction, rather than reporting an existing production deployment of SIE.

3. Automated venture financing document preparation

The problem: Preparing financing documents involves repetitive drafting, information extraction and comparison against established contractual templates.

During the interview, Fellow identified venture financing as a promising area for future automation.

Many US venture financing transactions use standardised National Venture Capital Association (NVCA) documents, with substantial similarities across deals.

This creates an opportunity to separate routine information processing from actual legal judgment.

A document preparation agent could:

• Use document field extraction to capture financing terms, investor details and company information.
• Use named entity extraction to identify parties and other legally relevant entities.
• Use long-document retrieval to find specific provisions across existing agreements and firm precedents.
• Use structured output to prepare information for a template-based drafting system.
• Escalate disputed terms, unusual provisions and legal decisions to a lawyer.

The result would be a system that automates much of the information-gathering and document preparation process while preserving human control over the negotiated legal outcome.

Where smaller models fit

A specialist extraction model doesn’t need to understand the entire legal strategy of a financing round to identify the amount being invested or the names of the investors.

Similarly, a retrieval model doesn’t need to draft a new clause to identify similar provisions from previous transactions.

Using the appropriate model for each operation keeps the broader workflow easier to evaluate.

The challenge is ensuring that extracted information, selected precedents and generated provisions are legally appropriate for the particular transaction.

4. Private legal research and precedent retrieval

The problem: Legal research agents need to retrieve the right sources before they can produce reliable answers.

An agent researching a legal question might need to search internal precedents, contracts, regulatory guidance and matter documents.

Even a highly capable LLM can produce an incorrect answer if it receives incomplete, outdated or irrelevant context.

This makes retrieval quality particularly important.

A legal research system could combine:

• Semantic search to find passages by meaning.
• Hybrid keyword search to preserve exact matches for legal terms, references and names.
• Multi-vector retrieval to improve matching across long documents.
• Reranking to prioritise the evidence most relevant to the question.
• A reasoning model to synthesise an answer from the retrieved material, with citations.

In this architecture, smaller models handle the frequent retrieval and ranking operations. Larger models are reserved for questions that genuinely require synthesis and reasoning.

A working example: Evaluating retrieval strategies with SIE

Superlinked has published a retrieval benchmark example comparing different retrieval approaches on 1,854 questions about SEC 10-K filings.

The experiment evaluates combinations of embedding, multi-vector and reranking models.

The strongest tested pipeline combined two complementary multi-vector retrieval models with a cross-encoder reranker, outperforming the single dense-model baseline on the benchmark.

Although the example uses financial filings rather than legal precedents, the same evaluation methodology is relevant to legal research.

A law firm could construct an equivalent benchmark using its own documents and expert-reviewed relevance judgments.

A second example: Private compliance RAG

Superlinked also provides a private, fine-tuned compliance RAG example.

This pipeline combines a domain-adapted embedding model with a custom reranker that also prunes irrelevant context.

The example demonstrates how developers can serve domain-specific model adaptations through one SIE cluster.

For legaltech teams building research applications around their own private knowledge bases, this is a useful reference architecture for specialised retrieval and controlled deployment.

What makes a legal AI task suitable for a small model?

Not every legal task should be moved to a smaller model.

The best candidates generally share three characteristics.

High repetition: The same task occurs frequently across documents or matters. Smaller models can have a meaningful cost advantage when deployed at sufficient volume.

Constrained outputs: The result is straightforward to define, such as a document classification, extracted field, ranked list or structured decision.

Measurable accuracy: The firm has historical records, approved outputs or expert-reviewed examples that can be used to evaluate model performance.

For example, document classification and contract metadata extraction are natural candidates for small models.

Complex questions involving commercial negotiation, conflicting case law or legal strategy may still need stronger reasoning models and experienced lawyers.

The goal is not to use small models everywhere.

It’s to identify where a smaller model can meet the required quality standard and avoid paying for unnecessary general-purpose reasoning.

How to evaluate small models before deploying them

Model selection is particularly important in legal applications, where errors can have significant consequences.

A small model that makes more mistakes may ultimately cost more than a frontier model once human review and remediation are included.

Before substituting a model, teams should define a representative test set using real documents and approved outcomes.

For example:

Workflow	Useful evaluation metric
Document filing	Classification accuracy and incorrect matter assignment rate
Contract entity extraction	Precision, recall and exact field accuracy
Clause retrieval	Recall@k, nDCG and source relevance
Contract risk identification	False-negative rate and false-positive rate
Document processing	Field accuracy, table preservation and reading order

Compare smaller models against the model currently in production, then evaluate the complete workflow.

This includes inference cost, latency, retries, human escalation rates and the time reviewers spend validating outputs.

Superlinked provides custom evaluation documentation and a worked retrieval evaluation example that developers can adapt to their own datasets.

Where an existing open-source model doesn’t achieve the required quality, domain-specific fine-tuning or post-training may be worth exploring.

But fine-tuning should follow evaluation, not precede it.

Why self-hosted AI matters for law firms

As Fellow expands its AI operations, the team is increasingly concerned about dependency on model providers, pricing changes and confidential client information.

These considerations become more important when AI moves from an occasional productivity tool into the operational infrastructure of a law firm.

Relying entirely on third-party model APIs can introduce several dependencies:

• Provider pricing and usage limits
• Model retirement or unexpected behaviour changes
• Data processing arrangements and residency requirements
• Availability and reliability of external services
• Restrictions on model customisation

Self-hosting open-source models gives engineering teams more direct control over model versions, infrastructure and deployment environments.

For organisations handling confidential legal information, that can be especially valuable.

However, self-hosting isn’t automatically more secure or less expensive. Access controls, client isolation, logging, infrastructure costs and operational responsibilities still need to be addressed.

The important advantage is architectural flexibility.

A firm might use managed inference for low-risk experiments while keeping sensitive production workloads inside its own infrastructure.

Explore Superlinked’s legal AI deployment options

How Superlinked helps legaltech teams run specialised AI models

The Superlinked Inference Engine is an open-source inference server that enables developers to run multiple specialised models through a common interface.

Instead of managing separate inference services for embeddings, reranking, extraction and generation, teams can serve these capabilities through SIE.

For a legaltech application, this means the infrastructure behind a contract review or legal research agent can support:

• Document processing and OCR
• Classification and entity extraction
• Semantic and multi-vector retrieval
• Reranking and evidence selection
• Structured generation and agent workflows
• Model evaluation and self-hosted deployment

Teams can begin experimenting with the hosted SIE API and deploy the open-source engine inside their own infrastructure when appropriate.

Superlinked’s Native Inference offering also supports teams with model evaluation, post-training and deployment.

The objective isn’t to replace every frontier model.

It’s to stop using expensive general-purpose models for tasks that smaller, specialised models can handle reliably.

A practical starting point for legaltech CTOs

If you’re building AI infrastructure for a law firm or legal software product, the first project shouldn’t necessarily be a fully autonomous legal agent.

Start with a workflow you already understand.

Choose a frequent, expensive or operationally frustrating task. Gather representative examples, define expected outcomes and establish a baseline against your existing implementation.

Then evaluate where a specialised model could substitute for a larger one.

Document classification, contract metadata extraction and reranking of legal precedents are sensible starting points.

Once the smaller model meets your accuracy requirements, evaluate its latency, infrastructure costs and behaviour under production conditions.

From there, the same architecture can be extended into more sophisticated legal agents.

Frequently asked questions

Can small language models perform legal document review?

Small models can handle individual stages of legal document review, including classification, extraction, retrieval and some narrowly defined assessments. More complex legal reasoning may still require stronger models and qualified human review.

What legal AI tasks are best suited to small models?

Document classification, entity extraction, clause retrieval, reranking, PII detection and structured information extraction are strong candidates. Each should be evaluated against the firm’s actual documents and quality requirements.

Can law firms run AI models without sending client documents to third-party inference APIs?

Yes. Self-hosted open-source models can run inside infrastructure controlled by the firm or its clients. The broader application must also be configured to keep sensitive data within the intended environment.

Do smaller models need fine-tuning for legal applications?

Not always. Teams should first benchmark existing open-source models against representative legal tasks. Fine-tuning becomes useful when a model falls short on a clearly defined task and sufficient high-quality training data is available.

How can legaltech companies reduce inference costs?

Break AI workflows into smaller operations, use specialised models where they meet accuracy requirements, run deterministic code where possible and reserve frontier models for complex reasoning. Measure cost per successful workflow rather than just cost per model call.

Build legal AI around the task, not the model

Fellow’s experience illustrates an important shift in legal AI.

The biggest opportunity may not be building one agent capable of doing everything a lawyer does.

It may be identifying the hundreds of smaller operations that make legal work possible, automating them reliably and preserving human judgment where it matters most.

For legaltech engineering teams, smaller open-source models offer a way to build that infrastructure with greater flexibility over model selection, inference costs and deployment.

Want to explore this approach?

Start with Superlinked’s self-hosted contract review agent, browse the available inference tasks, or explore SIE for legal AI teams.