"A red leather handbag": one query, six images built to fool it

This article walks through multimodal-search, a workable example from the open-source SIE repo on GitHub: text-to-image search over licensed photos, evaluated against deliberate hard negatives. SIE is Superlinked’s self-hosted inference engine, one cluster that serves embedding, reranking, OCR, vision, entity-extraction, and generation models behind three primitives (extract, encode, score). The example is a complete project; two commands reproduce the ranking, and the test suite re-derives it offline. Built by Superlinked.

Text-to-image search demos usually cheat by accident. Search “dog” against a folder containing one dog photo and five landscapes, and any halfway-decent embedding model looks brilliant. The demo proves nothing because the distractors never had a chance.

The multimodal-search example in the SIE repo does the opposite. The query is “a red leather handbag”, and the six candidate images are picked to be hard negatives: some match the color but not the object, some match the object but not the color, some are unrelated entirely. The model has to hold color and object type in the same representation and rank the one true match first. That’s the actual production problem, miniaturized. Anyone who has shipped search over a licensed image catalog knows that the painful queries are exactly these: attribute plus object, where near-misses outnumber matches.

Why hard negatives are the whole game

Consider what “a red leather handbag” actually requires from an embedding space. The model must place a red dress far from the query even though the color matches. It must place a brown leather satchel far away even though the object and material match. It must do this while keeping the one correct image closest of all, and it must do it from a single joint representation of text and pixels, with no category metadata to lean on.

Production image search fails on precisely these boundaries. Users don’t complain when “handbag” returns handbags; they complain when “red handbag” returns a red coat, because the red coat scored higher on color than the actual handbag scored on both attributes combined. A benchmark without hard negatives never measures that boundary, which is why models look great in demos and mediocre in production. Six carefully chosen images test more than six thousand random ones.

What’s running underneath

The model is google/siglip-so400m-patch14-384, served through SIE’s encode endpoint. SigLIP maps text and images into a shared 1,152-dimensional space, so similarity between a query string and a photograph is just cosine similarity between two vectors.

The pipeline embeds the one text query and the six images, computes cosine similarity locally, and validates that the correct handbag ranks first. Full vectors come back from the endpoint (no server-side ranking black box), which enables the part of this example that deserves the most attention.

The receipts

This is an evaluation harness dressed as a demo, and the audit story is thorough in four specific ways.

Image bytes are cryptographically verified before any request is sent, so the inputs are provably the catalogued files. Every request produces an envelope recording inputs, hashes, and byte lengths. A manifest pins the exact model version and the hardware it ran on (an NVIDIA L4). And the test suite recomputes similarity scores offline from the saved vectors, meaning the committed ranking can be re-derived on any laptop without touching a GPU or an endpoint.

That last property is the one to internalize. The claim “our model ranks the red leather handbag first” isn’t something you take on faith from a README; it falls out of the saved embeddings and a few lines of arithmetic anyone can run. Even the image metadata in data/sources.json carries proper attribution, which matters if you work with licensed content and your legal team asks where the test images came from.

Running it

Two commands against a local SigLIP deployment:

uv sync
uv run python run.py --output run-output/results.json

Or against hosted infrastructure:

SIE_BASE_URL=https://your-cluster.example SIE_API_KEY=your-key uv run python run.py

Same script either way. The results file contains the full ranking with scores, ready to diff against the committed run.

The choice of SigLIP over CLIP is quietly practical too. SigLIP’s sigmoid training objective handles the many-negatives regime better than CLIP’s softmax, and the so400m checkpoint sits at a sweet spot: strong retrieval quality at a size one L4 serves comfortably. For a catalog-search workload where you’re embedding millions of images once and queries forever after, that serving cost is the number that ends up on the invoice.

Six images today, your catalog tomorrow

The scale is deliberately tiny; the structure is not. Swap the six photos for a sample of your own catalog, write queries that mix attributes the way your users do (“navy wool overcoat”, “matte black ceramic vase”), and define which image must win each query. Now you have a hard-negative benchmark for your domain, with cryptographic verification that nobody has quietly swapped a test image, and offline re-scoring that turns “trust me” into “check the math”.

Run it once to see SigLIP separate a red leather handbag from a red dress and a brown leather satchel. Then make the negatives harder and find out where it breaks. That second run is the one that tells you something.

Try it on GitHub: superlinked/sie/examples/multimodal-search