Inference

How Do You Deploy an Embedding Model on AWS?

Deploying an embedding model on AWS with SIE requires three steps: configure your GPU cluster using the SIE Terraform module, deploy the inference server with Helm, and connect using the SIE SDK. The process takes under 30 minutes and produces a production-grade inference cluster in your own AWS account with automatic scaling and model management.

Why deploy on AWS rather than using a managed API?

Managed embedding APIs (OpenAI, Cohere, Voyage) charge per token. At production scale (encoding millions of documents or serving real-time queries), per-token costs become prohibitive. Self-hosting on your own AWS GPUs reduces inference costs by up to 50×, keeps data within your AWS account (critical for compliance), and gives you full control over model selection and configuration.

Prerequisites

Before deploying SIE on AWS, you need:

An AWS account with permissions to create EC2 instances, VPCs, and IAM roles
Terraform installed locally (>= 1.3)
Helm installed locally (>= 3.0)
kubectl configured to connect to your cluster
The sie-sdk Python package (pip install sie-sdk)

Step 1: Configure the cluster with Terraform

SIE provides an official AWS Terraform module that provisions the GPU infrastructure:

# main.tf
module "sie" {
  source  = "superlinked/sie/aws"
  version = "~> 1.0"

  aws_region        = "us-east-1"
  project_name      = "sie"
  gpu_instance_type = "g6.2xlarge"   # one NVIDIA L4
  gpu_capacity_type = "SPOT"
  gpu_min_size      = 0
  gpu_max_size      = 4
}

output "kubectl_config_command" {
  value = module.sie.kubectl_config_command
}

output "sie_irsa_role_arn" {
  value = module.sie.sie_irsa_role_arn
}

terraform init
terraform plan    # review what will be created
terraform apply   # provision the cluster (~15-20 minutes)

This creates an EKS cluster with an L4 GPU node group and the IAM role the SIE pods run under. For several GPU types, use the module’s gpu_node_groups list instead; the AWS deployment guide covers it.

Step 2: Deploy the inference server with Helm

Once the cluster is provisioned, deploy the SIE inference server:

# Point kubectl at the new cluster
$(terraform output -raw kubectl_config_command)

# Install the chart from GHCR. Worker pools default to off, so enable the L4 pool.
helm upgrade --install sie oci://ghcr.io/superlinked/charts/sie-cluster \
  -n sie --create-namespace \
  --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=$(terraform output -raw sie_irsa_role_arn)" \
  --set workers.pools.l4.enabled=true \
  --set workers.pools.l4.bundles.default.minReplicas=1

The chart deploys the SIE gateway and one L4 worker. To scale workers with demand, add --set keda.install=true --set autoscaling.enabled=true; the AWS deployment guide covers scale-from-zero and gated models.

Step 3: Verify the deployment

# Check pods are running
kubectl get pods -n sie

# Reach the gateway from your machine
kubectl -n sie port-forward svc/sie-sie-cluster-gateway 8080:8080 &

# Check the endpoint is healthy
curl http://localhost:8080/health
# → JSON with the gateway status, its workers and the models they serve

Step 4: Start encoding

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")  # the port-forward from step 3

# Load a model on first use (cached on GPU after first request)
results = client.encode(
    "BAAI/bge-m3",
    [
        Item(text="self-hosted inference on AWS"),
        Item(text="reduce embedding API costs with your own GPU"),
    ],
)

print(f"Encoded {len(results)} documents, dim={len(results[0]['dense'])}")

Models are downloaded from Hugging Face on first use and cached on the GPU node. Subsequent requests are served from the cache with full GPU throughput.

GPU selection guide for AWS

GPU	Instance type	Best for
A100 40GB	p4d.24xlarge	Largest models, highest throughput
A100 80GB	p4de.24xlarge	Very large models (7B+)
L4	g6.xlarge (spot)	Cost-efficient, medium throughput
T4	g4dn.xlarge	Low-cost dev/staging

For most embedding workloads (BGE-M3, E5-large), L4 spot instances offer the best cost-performance ratio. Use A100s for 7B+ instruction-following models.

Cost optimisation tips

Use spot instances: L4 spot instances reduce GPU costs by 60-70% vs on-demand. SIE handles spot interruption gracefully.

Enable autoscaling: scale down to 1 node during off-peak hours, scale up for batch indexing jobs.

Batch encode: send large batches per request to maximise GPU utilisation:

# Encode 1000 documents in one call — much more efficient than 1000 individual calls
from sie_sdk.types import Item

results = client.encode(
    "BAAI/bge-m3",
    [Item(text=d) for d in document_batch_of_1000],
)
vectors = [r["dense"] for r in results]

Share the cluster across models: SIE’s multi-model support means one cluster serves all your embedding, reranking, and extraction needs.

Frequently asked questions

How long does deployment take? Terraform provisioning takes ~15-20 minutes. Helm deployment takes ~3 minutes. Total time to first encode request: under 30 minutes.

Is my data sent outside my AWS account? No. SIE runs entirely within your AWS account. Encoding requests go from your application to your SIE cluster, never to Superlinked’s servers.

What AWS regions are supported? SIE supports all AWS regions with GPU instance availability: us-east-1, us-west-2, eu-west-1, ap-southeast-1, and others. Specify aws_region in the Terraform module.

Do I need Kubernetes experience? Basic familiarity with kubectl and Helm is helpful. SIE’s Terraform module abstracts the EKS cluster setup, so you don’t need deep Kubernetes expertise.