# Sparse embeddings page sources

## Recorded model output

- Endpoint: `https://api.superlinked.com` (`POST /v1/encode/<model>`,
  `output_types: ["sparse"]`, JSON request and response)
- Run date: 2026-09-15
- Inputs: 13 texts, each encoded once by both models (26 recorded calls, all
  HTTP 200)
- Runner: `apps/site/tests/fixtures/reference/sparse/run.py` (Python standard
  library only). The requests, raw responses, decoded records and manifest sit
  beside it.

| Model | Hugging Face revision | License | Vocabulary file | SHA-256 |
| --- | --- | --- | --- | --- |
| [`prithivida/Splade_PP_en_v2`](https://huggingface.co/prithivida/Splade_PP_en_v2) | `f0d4aa214dcb60c274052a52c0497535e3aec64c` | Apache-2.0 | `vocab.txt` (30,522 entries) | `07eced375cec144d27c900241f3e339478dec958f92fddbc551f295c992038a3` |
| [`BAAI/bge-m3`](https://huggingface.co/BAAI/bge-m3), SIE profile `sparse` | `5617a9f61b028005a4858fdac845db406aefb181` | MIT | `tokenizer.json` (250,002 entries) | `21106b6d7dab2952c1d496fb21d5dc9db75c28ed361a05f5020bbba27810dd08` |

The revisions are the `hf_revision` values in the SIE model configuration for
each served model.

## Display transformation

The API returns vocabulary indices and float32 weights. The generated snippet
prints exactly those ids and weights. The page shows each id beside its token,
and the token is a display-side decode, like offsets resolved against source
text elsewhere on the site. The steps are these and nothing else:

1. Each index is mapped to its entry in the pinned vocabulary file above.
2. Terms are sorted by weight. Each card shows the 12 strongest terms, with no
   term removed or reordered.
3. A SPLADE term counts as in the text when it occurs in the BERT uncased
   WordPiece tokenization of the input (lowercase, accent stripping,
   punctuation split, greedy longest match). Every other returned term counts
   as added by the model.
4. A bge-m3 sparse term counts as in the text when its surface, without the
   SentencePiece word marker, occurs in the input. All 13 bge-m3 responses
   contain only such terms.
5. The hero match score is the dot product of the query and listing SPLADE
   vectors: 13.998. Terms present in both texts' WordPiece tokens (`led`,
   `color`) contribute 5.342; terms SPLADE added to at least one of the two
   texts contribute the remaining 8.656. This is arithmetic over the recorded
   vectors, not a separate model call.

Weights are rounded for display only: three decimals in the hero and two on
the cards. Site CI compares every displayed index, token and weight with the
saved responses.

The strongest terms include model misses as well as useful expansions. For the
ATM message, SPLADE reads "pounds" as weight and adds `weigh` and `weight`; for
the card delivery message it adds `years` next to `weeks`. The page keeps them.

## Primary sources

### BANKING77 customer messages

- PolyAI, BANKING77 test split, CC BY 4.0:
  <https://github.com/PolyAI-LDN/task-specific-datasets/blob/master/banking_data/test.csv>
- Paper: <https://arxiv.org/abs/2003.04807>
- Messages used, verbatim, with their dataset intent:
  - `card_arrival`: "My card still hasn't arrived after 2 weeks. Is it lost?"
  - `transfer_not_received_by_recipient`: "I transferred money but the
    recipient says it has not arrived. Why would this be?"
  - `compromised_card`: "Can you freeze my card because someone used it while
    I was out of town. I did not make these purchases."
  - `wrong_amount_of_cash_received`: "I'm at an ATM and withdrew 30 pounds and
    was only given 10. What should I do?"
  - `cash_withdrawal_charge` (playground): "Do ATM cash withdrawals carry a
    charge now? They've been free in the past, but all of a sudden I have to
    pay to make ATM withdrawals?"
  - `pending_top_up` (recorded, not displayed): "My card was topped this
    morning but I can't see the funds. Why didn't it complete?"

### Amazon Shopping Queries Dataset

- Amazon Science, Shopping Queries Dataset (ESCI), Apache-2.0:
  <https://github.com/amazon-science/esci-data>
- Test split rows read through the Hugging Face mirror
  `tasksource/esci`.
- Hero pair: example 564555, query `color switching led lights`, product
  `B07F3X9Y98`, judged Exact.
- Product search card: example 530782, query `chrome notebook`.
- Also recorded: product `B00M9K7L8S` (Samsung Chromebook listing) for the
  `chrome notebook` pair.

### CPSC recall notices

- U.S. Consumer Product Safety Commission, Recall API
  (`saferproducts.gov/RestWebServices/Recall`). U.S. federal government work,
  public domain.
- Displayed: recall 25437, ESR HaloLock wireless power banks, Hazards field:
  <https://www.cpsc.gov/Recalls/2025/ESR-HaloLock-Wireless-Power-Banks-Recalled-Due-to-Fire-and-Burn-Hazards-Distributed-by-Waymeet>
- Also recorded: recall 25431 (Werner ladders) and recall 25432 (Wolfgang Puck
  tea kettles).