Which price label belongs to the empty shelf gap? Geometry answers, OCR proves it.

This article walks through retail-shelf-audit, a workable example from the open-source SIE repo on GitHub: a two-model pipeline that detects an empty shelf facing in a real supermarket photo and links it to its price label by geometry alone. SIE is Superlinked’s self-hosted inference engine, one cluster that serves embedding, reranking, OCR, vision, entity-extraction, and generation models behind three primitives (extract, encode, score). The example is a complete project on a CC0-licensed image, with committed records you can verify without a GPU. Built by Superlinked.

A shelf photo shows a gap where a product should be. Somewhere on that shelf edge is the price label for the missing item, surrounded by labels for its neighbors. A human matches gap to label without thinking. Software has a harder time, and the naive approach (run OCR over the whole frame, grep for product names) fails in a specific way: it can tell you every label in the photo, and nothing about which one sits under the hole.

Spatial relationships are the actual problem in shelf auditing. The retail-shelf-audit example in the SIE repo solves it with two models and four fixed geometric rules, on one real photo: a 4032×3024 image from case 042 of the Humans in the Loop Supermarket Shelves Dataset (CC0 licensed), showing an empty space where Panadol should be.

Detection first, then geometry, then OCR

The pipeline runs two models with a deliberate division of labor. IDEA-Research/grounding-dino-base handles object detection, finding empty shelf gaps and price tags as bounding boxes. lightonai/LightOnOCR-2-1B reads text, but only from small crops the geometry has already selected.

Between detection and reading sit four fixed rules. Take the highest-confidence empty gap after filtering. Find the price tags that overlap the gap vertically. Deduplicate overlapping boxes and select the strongest aligned pair. Then cut crops expanded 20% horizontally and 25% vertically, upsample them 3x, and hand only those crops to OCR.

On case 042, DINO located the empty facing at [2043.8, 2137.0, 623.6, 402.4] with a confidence of 0.274, the geometry linked it to the adjacent notice and shelf label, and OCR on the selected crops returned the product: “Panadol Child, 5-12Yrs Elixir 100ml”, alongside the out-of-stock notice. The OCR step took 5270ms on an NVIDIA L4.

That 0.274 confidence is worth pausing on. It’s low, and the pipeline still works, because the rules operate on relative confidence after filtering rather than demanding a high absolute score. Real shelf photos are cluttered and badly lit; a pipeline that needs 0.9 confidence to proceed would sit idle in a real store.

No product knowledge allowed

The design decision that gives this example its title: the runtime selector contains no hardcoded product names and no known prices. Evidence linkage rests purely on geometry.

That constraint is what makes the audit auditable. If the pipeline knew to look for “Panadol”, a skeptical reviewer could never distinguish detection from confirmation bias; the system would find what it was told to find. Because the linkage is geometric, anyone can review the output without domain knowledge: here is the detected gap, here are the boxes that overlap it vertically, here is the crop, here is the text OCR read from that crop. Every step is checkable from the saved artifacts, which is exactly what “auditable” has to mean if the number feeds a compliance report or a vendor dispute.

Running it

Verify the committed records first, no GPU needed:

uv sync --frozen && uv run verify-retail-records

Then run the audit against your own SIE endpoint:

export SIE_BASE_URL="https://your-sie-endpoint.example"
uv run audit-retail-shelf --run-id cloud-042

Detection and OCR both go through the same SIE deployment, so the two-model pipeline is one endpoint and one client rather than two vendor integrations.

Crop-then-OCR beats OCR-then-search

The ordering of the pipeline is a lesson on its own. Running OCR first over the full 4032×3024 frame produces a soup of every readable string in the photo, and then you’re stuck doing spatial reasoning over text coordinates after the fact, with OCR errors already baked in. Detection-first flips it: find the geometry, choose the crops, and only then spend OCR effort on regions that matter.

The crops also get preprocessing the full frame can’t: 20% horizontal and 25% vertical expansion so context survives the cut, then 3x upsampling so shelf-label text (tiny at native resolution) arrives at the OCR model legible. That’s why a 1B-parameter OCR model reads “Panadol Child, 5-12Yrs Elixir 100ml” cleanly off a shelf edge. The 5270ms OCR call happens twice per gap instead of dozens of times per frame, which changes the cost profile of auditing a whole store.

One facing today, a planogram tomorrow

Stockouts cost retailers real revenue, and most shelf-audit tooling answers with a percentage: “shelf availability 94%”. The number is only as good as the evidence behind it, and evidence is what this example produces: a specific gap, a specific label, specific OCR text, linked by rules you can print on an index card.

Scaling from one facing to a full aisle is iteration; the per-gap logic is what has to be right. It’s here, on a CC0 image, with the coordinates committed. Clone it, run the verification, then point DINO at a shelf photo of your own and see what the geometry finds.

Try it on GitHub: superlinked/sie/examples/retail-shelf-audit