Should Production Agents Use Non-Transformer Models?

Transformer alternatives attract attention for good reasons. State-space models promise different scaling behaviour. Stateless designs can reduce dependence on a growing key-value cache. Research papers report attractive throughput and context-length results.

Production teams should still start with proven small models.

The useful question is not which architecture looks best on paper. It is which complete system meets the task’s quality, latency, cost, and operational requirements.

Architecture is one part of the result

A model can have an efficient core and still be expensive to ship. The serving engine needs kernels for the architecture. Quantized weights need to work on the available hardware. Tool-calling templates must match the agent. Batching, observability, and autoscaling all need to behave under real traffic.

Transformers benefit from years of work across that stack. Mixture-of-experts models reduce the number of active parameters per token. Better key-value cache exchange makes distributed generation more practical. Smaller transformer models now cover many classification, retrieval, extraction, and generation jobs.

That installed base matters. A theoretical efficiency gain can disappear if the team has to maintain custom kernels, patch an inference server, or give up the batching behaviour it already relies on.

Split the workload before replacing the architecture

An agent contains several different computational problems. Routing needs classification. Retrieval needs embeddings and relevance scores. Sensitive-data handling needs span extraction. The final response may need generation.

Running one large transformer for every step creates waste. Replacing that model with one large state-space model may preserve the same design mistake.

Split the workload into bounded tasks first. Use off-the-shelf encoders for labels, spans, and scores. Use a small generative model for structured drafts. Escalate only the requests that need stronger reasoning.

This change usually offers a clearer path to lower cost than an architecture migration. It also lets each model improve independently.

Retrieval shows how ideas can cross architecture boundaries. Late-interaction and multivector methods apply selective comparison to search while keeping a familiar model and serving path. A team can gain some of the efficiency benefits without replacing its whole stack.

Demand an end-to-end benchmark

Compare candidate architectures on the product task. Include the input lengths, concurrency, batch sizes, quantization, and hardware planned for production.

Measure task success and end-to-end latency. Add cost per successful task. Record time to first token for generative work and throughput for short encoder jobs. Include model load time and memory pressure if the system serves several models on the same GPUs.

Long context makes this especially important. A model may support a large configured window while delivering an unusable response time at the maximum length. In one profile cited in the presentation, a quantized GLM-5.3 Flash deployment used two H200 GPUs for a 929,000-token prompt. Time to first token reached 97 seconds. The weights fit, but that fact alone did not make the workload interactive.

Treat published model numbers as a shortlist, then reproduce the test that matches your application.

Choose the boring path when it passes

Research teams should keep testing new architectures. Product teams need a stronger reason to own them.

If a non-transformer model produces a measured gain that survives integration, use it. If a small transformer already meets the quality target on available infrastructure, ship it and work on the next bottleneck.

Model architecture should disappear behind a stable task API. The application asks to encode, score, extract, or generate. A serving profile selects the model, runtime, context limit, and quantization. That makes future experiments cheaper because the agent does not depend on one model family’s quirks.

The winning system is the one the team can evaluate, operate, and replace.

Questions to answer before changing architecture

What is a non-transformer language model?

A non-transformer language model uses an architecture other than transformer attention, such as a state-space model. These designs can change how memory and computation scale with sequence length.

Are state-space models cheaper than transformers?

They can be cheaper for some workloads, but architecture alone does not determine production cost. Runtime support, quantization, batching, hardware, and task quality all affect the result.

Should an agent use one model for every step?

No. Routing, extraction, retrieval, and generation have different output shapes. Small task-specific models can handle the bounded steps while a generative model handles open-ended work.

How should teams compare model architectures?

Run an end-to-end evaluation on production-like inputs and hardware. Measure task success, latency, throughput, memory use, and cost per successful task. Include the integration and maintenance work required by the serving stack.