Skip to content
Most AI tasks no longer need a frontier model Read launch post

LoRA Adapters

LoRA adds trained low-rank weights to a base model. SIE can keep several compatible adapters alongside that base and select one for a request. Support depends on the model adapter and its profile; adding a request field does not give an unsupported model LoRA support.

A request may select only an adapter declared by its model profile. It cannot supply an arbitrary repository or filesystem path for the server to load.

For an existing Qwen/Qwen3-Embedding-4B model configuration using SGLangEmbeddingAdapter, add a profile like this. Replace the example repository with your compatible, trained LoRA:

profiles:
lora:
extends: default
adapter_options:
loadtime:
lora_paths:
legal: org/qwen3-legal-lora
max_loras_per_batch: 2
preload_loras: []

Here legal is the served name; org/qwen3-legal-lora is the source. A dictionary entry can also pin its source with id and a full 40-character commit SHA in revision. The existing base-model configuration and its revision remain in place.

For SGLang, an omitted preload_loras preloads all declarations. An empty list defers them; a list of declared names preloads that subset. Only declared adapters can load later.

For native encode, extract and score calls, put lora_adapter in the runtime options. These examples select the lora profile above and its legal adapter.

from sie_sdk import SIEClient
from sie_sdk.types import Item
with SIEClient("http://localhost:8080") as client:
result = client.encode(
"Qwen/Qwen3-Embedding-4B",
Item(text="breach of fiduciary duty"),
options={"profile": "lora", "lora_adapter": "legal"},
)

The equivalent native HTTP request is:

curl -X POST http://localhost:8080/v1/encode/Qwen/Qwen3-Embedding-4B \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
-d '{"items":[{"text":"legal query"}],"params":{"options":{"profile":"lora","lora_adapter":"legal"}}}'

The field location follows the endpoint’s existing request envelope:

APISelector location
Native encode or extractparams.options.lora_adapter
Native scoreoptions.lora_adapter
Generate or chat completionsTop-level lora_adapter
System One or Decisions compatibility APITop-level lora_adapter, a SIE extension

The compatibility field still requires a model profile that supports LoRA. With an OpenAI-compatible client, a SIE extension can be supplied through extra_body without changing the upstream request schema.

For generation, use the Python SDK’s lora_adapter argument or the TypeScript SDK’s loraAdapter option. Both send the canonical top-level wire field. Generation retains its canonical-only interface.

Preserve defaults and reject conflicting selections

Section titled “Preserve defaults and reject conflicting selections”

Omitting the native selector preserves the selected profile’s runtime default. Explicit null selects the base model, including when the profile has a LoRA default:

with SIEClient("http://localhost:8080") as client:
base_result = client.encode(
"Qwen/Qwen3-Embedding-4B",
Item(text="general query"),
options={"profile": "lora", "lora_adapter": None},
)

TypeScript uses null in adapterOptions; leaving the property undefined omits it. Generation’s omitted and null selectors both select its base model.

Existing native and decision clients can keep using the legacy request aliases lora and lora_id. lora_adapter and lora accept a declared served name or source ID. The legacy lora_id remains restricted to declared source IDs, so legal in the example belongs in lora_adapter, while org/qwen3-legal-lora is a source ID.

If more than one selector is supplied, their values must be identical. A name and null conflict, even if one spelling is legacy. A served name and its source ID are also different values; send one selector. Conflicts and invalid types are rejected before loading or dispatch.

These request aliases do not rename model-configuration fields. Existing adapter_options.runtime.lora_id defaults remain supported where the model profile permits them.

Loading and capacity depend on the model adapter

Section titled “Loading and capacity depend on the model adapter”

The registry can load declared LoRAs on demand for adapters that expose runtime loading. This includes supported PEFT adapters and SGLang generation and embedding adapters. SGLang is not limited to startup-only loading.

The explicit reranker and entity-extraction profiles support runtime selection:

  • Standard, unquantized BERT and XLM-R rerankers use sie_server.adapters.cross_encoder.runtime_lora:CrossEncoderLoraAdapter as their adapter_path.
  • GLiNER2 entity extraction uses sie_server.adapters.gliner2.entity_lora:GLiNER2EntityLoraAdapter. This profile is entity-only; it does not enable classification, relations or typed decisions.

Set the explicit adapter_path and declare compatible lora_paths in the profile.

A first request can encounter LORA_LOADING while its adapter loads. Python encode calls retry the documented loading response within a bounded budget; load failures and exhausted capacity are still errors. Fetching and attaching are separate operations, and attaching may require waiting for active inference. Loading is not a latency guarantee.

Resident limits, preload pins and available memory determine whether another adapter can load. The registry can evict an eligible least-recently-used adapter; pinned startup adapters are not eviction candidates. An evicted declaration can load again on a later request. Memory use depends on the trained adapter and backend, so measure it for your deployment.

Decision models use separate, explicit LoRA profile classes. Choose the class for the native model and a bundle that includes it:

Model familyProfile adapter_pathBundle
Layasie_server.adapters.laya.runtime_lora:LayaLoraAdapterdefault
GLiClass uni-encodersie_server.adapters.gliclass.runtime_lora:GLiClassLoraAdapterdefault
GLiNER2.5 Decidesie_server.adapters.gliner2.decide_lora:GLiNER2DecideLoraAdaptertransformers5
Perplexity native decisionssie_server.adapters.pplx_decider.runtime_lora:PerplexityDecisionLoraAdaptertransformers5

For example, add this profile to the existing convaiinnovations/laya configuration. Replace the source with a compatible LoRA trained for that base model:

profiles:
lora:
extends: default
adapter_path: sie_server.adapters.laya.runtime_lora:LayaLoraAdapter
adapter_options:
loadtime:
lora_paths:
legal: org/laya-legal-lora

Set SIE_MAX_LORAS_PER_MODEL=2 in the server environment to keep at most two LoRAs loaded per model (the default is 10). The serving engine applies this limit to every profile and overrides a profile’s max_loaded_loras load option.

Select it on a native extract call with options={"profile": "lora", "lora_adapter": "legal"}. The pinned {id, revision} declaration form works here too. These profiles load declared adapters on demand; they do not use SGLang’s preload_loras option.

Laya, GLiNER2.5 Decide and Perplexity retain their native typed-decision schemas, including when called through the System One or Decisions compatibility API. GLiClass retains its labels and label_groups classification interface; it does not accept a typed output_schema.

Use ordinary PEFT LoRA exports with task_type: null, bias: none and explicit target-module names. These profiles require unquantized base models and adapter_model.safetensors. They reject replacement heads (modules_to_save), embedding deltas, DoRA, RS-LoRA and per-layer rank or alpha patterns. Targets must match the native model’s supported linear layers; a LoRA trained for a different architecture is not interchangeable.

Ordinary profiles keep their existing execution path. In a LoRA-enabled profile, an adapted request runs eagerly and selecting the base restores its normal execution policy, including its configured CUDA graphs. The LoRA-enabled profile still performs selection and synchronization. Loading and attaching can also delay requests; hot loading does not promise zero overhead or watch a repository for new revisions.

SGLang LoRA loaders, supported PEFT embedding loaders, and the explicit cross-encoder, GLiNER2 entity and decision profiles above use SIE’s configured cache policy. Set SIE_LOCAL_CACHE to the local Hugging Face-style cache; SIE_CLUSTER_CACHE can provide a shared copy. Set SIE_HF_FALLBACK=false to disable their Hub fallback. Loading fails if the required adapter is unavailable.

Stage both adapter_config.json and adapter_model.safetensors for the declared revision. A partial snapshot is not a cache hit, and an explicit commit pin must match exactly. Without a revision pin, the loader follows the cached refs/main commit rather than choosing another available snapshot.

Hub-backed base checkpoints for Laya, GLiNER2.5 Decide and Perplexity also use SIE_LOCAL_CACHE. For these native loaders, either SIE_HF_FALLBACK=false or HF_HUB_OFFLINE=1 disables Hub access during checkpoint resolution.

An offline deployment also needs its required base-model and tokenizer artifacts staged, along with installed runtime dependencies. Disabling Hub fallback does not supply missing files or dependencies.

For more configuration details, see Profiles and the Model Catalog.

Talk to us

Tell us about your use case and we'll get back to you shortly.