# Classify page sources The page evidence is a pre-registered run on SNIPS user requests. Two earlier evaluations are kept below with their fixtures for disclosure: CLINC150 support messages routed to domain queues (two label configurations), and CFPB complaint narratives whose reference labels were chosen by the consumers themselves (two label configurations). None of those runs appears on the page. ## SNIPS pre-registration Everything in this section was committed and pushed before any model call on this set, and the set is run once. - Dataset: SNIPS natural language understanding benchmark, custom intent engines. Coucke A. et al., "Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces", 2018, - Repository: at commit `be5d12b65a7c50fe484186d81cf918eb160b0b53`, folder `2017-06-custom-intent-engines`, one `validate_.json` file per intent - License: CC0 1.0 Universal (repository `LICENSE`). Crowd workers wrote more than 2,000 queries for each intent (repository README). - File SHA-256: - `AddToPlaylist`: `14cc9ee49ae477b1452632a6e835498cf4c32da397b16967c82abb3e50403e83` - `BookRestaurant`: `4f7258ccc768b777b2eb8916532db9aa50dbd058ddf1ffb0dc59218963e7442a` - `GetWeather`: `60c095e193655d6ca92be220fed3926ca25e18cd70c85664145efe25d80993af` - `PlayMusic`: `922212775e26259d5bffeaf79b6048d070ca017d229df00b2cc9800f71c75d20` - `RateBook`: `c1f8b4114da0bb5e8afd78a8062766e18b2eabb0e717539598db01b09d04619b` - `SearchCreativeWork`: `8a5e830eddb185d3e4a9d27ded140a145d04766bdce0e58834676ede32d13e77` - `SearchScreeningEvent`: `81e1445713e03c7c3e615353c5f3fee05f369fe9834100b9b9735900a96b9498` - Model: `knowledgator/gliclass-large-v3.0`, single-label, server threshold 0.0 - Model revision: the public SIE model config pins Hugging Face commit `e065d1844f913a9aa611cf33623a9538b8aa8841`; SIE Cloud returned `x-sie-model-revision: 10333b84de80b402376b626eb25366fb081d3faeb893eb4b01cf32e8c27e4aff` on every response in all five recorded runs. The service does not document how that header maps to a Hugging Face commit, so no mapping is claimed. - Case builder: `apps/site/tests/fixtures/reference/classify/snips/build_cases.py` - Runner: `python3 apps/site/tests/fixtures/reference/classify/run.py --set snips` ### Label rule All seven intents in alphabetical order. Each label is the CamelCase intent name split into lowercase words. The 7 labels, exactly as sent: `add to playlist` · `book restaurant` · `get weather` · `play music` · `rate book` · `search creative work` · `search screening event` ### Message rule Validation rows only. A row's text is its segments joined, with runs of whitespace collapsed to one space. A row is eligible when the text has at least 4 words, contains only ASCII characters and is the first occurrence of that exact text. Eligible rows are ordered by `sha256("classify-snips-v1:" + intent + ":" + row index)`, and the first 2 rows of each intent are taken. No row was chosen or dropped by hand. The reference label is the intent the row was collected for; the model never sees it. ### Build bar The page is built only if the reference label ranks first for at least 10 of the 14 messages. ### Selected messages | Row | Message | Reference label | |---|---|---| | AddToPlaylist 91 | Add a song to my playlist madden nfl 16 | add to playlist | | AddToPlaylist 61 | add Highway Patrolman in my playlist this is al green | add to playlist | | BookRestaurant 49 | book a table at a Fried chicken restaurant | book restaurant | | BookRestaurant 9 | I want to book a brasserie for 3 people in Netherlands Antilles. | book restaurant | | GetWeather 52 | what is the overcast forecast for the current position starting on jul. 19, 2030 | get weather | | GetWeather 8 | What kind of weather will be in Ukraine one minute from now? | get weather | | PlayMusic 9 | Play Subconscious Lobotomy from Jennifer Paull. | play music | | PlayMusic 48 | Play the top twenty Hisham Abbas on Youtube | play music | | RateBook 24 | Give Small Screen, Big Picture a 0 out of 6 rating. | rate book | | RateBook 50 | give The Zenith Angle one out of 6 points | rate book | | SearchCreativeWork 77 | Looking for the picture with the name of Who Made Stevie Crye? | search creative work | | SearchCreativeWork 45 | Can you look up the Molecular Oncology saga? | search creative work | | SearchScreeningEvent 27 | What is the nearest movie house with Window Connection playing at lunch | search screening event | | SearchScreeningEvent 82 | what are some close by animated movies showing | search screening event | ### Recorded result The run went from `22:20:44Z` to `22:20:50Z` on 2026-09-15 against SIE Cloud, server version `0.7.3`, and all 14 requests returned HTTP 200 on the first attempt. The label for the intent each request was written for ranked first for 14 of 14 messages, above the pre-registered bar of 10. Twelve responses gave the winning label a score of 1.000. The two closest calls were both `play music` requests: `Play the top twenty Hisham Abbas on Youtube` scored `play music` 0.981 against `add to playlist` 0.016, and `Play Subconscious Lobotomy from Jennifer Paull.` scored 0.997 against `search creative work` 0.003. The page uses the closest correct call as its hero and playground example. Single-label scores sum to 1 across the 7 labels, so they rank the labels for one request; they are not calibrated confidence. Requests, raw responses and the manifest are in `apps/site/tests/fixtures/reference/classify/snips/`, and the unit test checks their SHA-256 values and every displayed score against them. # Earlier evaluation: CLINC150 support messages ## CLINC150 pre-registration Everything in this section was committed and pushed before any model call on this set, and the set is run once. - Dataset: CLINC150, Larson et al., "An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction", EMNLP-IJCNLP 2019, - Repository: at commit `828f8093932c8fe6ca7936c3d2e52903b1c523de` - `data/data_full.json` SHA-256: `36923c3705a59e08fe9c3883d8bc2dd966ef93e22cb78ac41171782a698d56e0` - `data/domains.json` SHA-256: `b947b579d3b8e74b06f93b01083d8efaff2888b43a3e362533bd88a6e1211b3a` - License: Creative Commons Attribution 3.0 Unported (repository `LICENSE`). Crowd workers wrote the queries by paraphrasing seed phrases or answering scenarios (repository README). - Model: `knowledgator/gliclass-large-v3.0`, single-label, server threshold 0.0 - Case builder: `apps/site/tests/fixtures/reference/classify/clinc150/build_cases.py` - Runner: `python3 apps/site/tests/fixtures/reference/classify/run.py --set clinc150` ### Queue rule Every CLINC150 domain in `domains.json` order except `small_talk` and `meta`, which hold assistant chit-chat and assistant controls rather than work a support queue owns. Each label is the domain key with underscores replaced by spaces. The 8 labels, exactly as sent: `banking` · `credit cards` · `kitchen and dining` · `home` · `auto and commute` · `travel` · `utility` · `work` ### Message rule Only the in-scope `test` split. A row is eligible when its intent belongs to one of the 8 queue domains, its text has at least 4 words, and it is the first occurrence of that exact text; 3,546 rows qualify. Eligible rows are ordered by `sha256("classify-clinc150-v1:" + test row index)` and taken in that order, skipping a row whose domain already has 2 messages, until 12 are selected. No row was chosen or dropped by hand. The reference answer is the domain of the row's human-assigned intent; the model never sees it. ### Selected messages | Test row | Message | Intent | Reference queue | |---|---|---|---| | 2332 | when do i have to pay my internet | bill_due | banking | | 748 | reservations for two need to be made using michael's name for 7:00pm at olive garden | restaurant_reservation | kitchen and dining | | 1775 | what day are we in | date | utility | | 3281 | tell me how much my state taxes amount to | taxes | work | | 723 | will you reserve a table at olive garden for 3 people at 7 | restaurant_reservation | kitchen and dining | | 1467 | what are my benefits on the insurance | insurance | work | | 3133 | i need to know how many pounds are in 10 kilos | measurement_conversion | utility | | 2387 | i need to replace a card that my daughter spilled juice on | damaged_card | credit cards | | 2963 | on the speaker do you know what song is on | what_song | home | | 981 | which timezone is denver located | timezone | travel | | 3185 | i need a hotel in phoenix near chase field from january 4th to the 8th | book_hotel | travel | | 1627 | what is my apr rate | apr | credit cards | The rule selected no `auto and commute` message; that queue still competes in every request. ### Config 2: described queues Config 1 (the 8 domain names above) ran first. After its result, this second label configuration was fixed, committed and pushed before any config 2 request, and runs once on the same 12 messages with the same model and threshold. Rule: each label is the domain key with underscores replaced by spaces, then `: `, then that domain's intent names from `domains.json` in file order, with underscores replaced by spaces, joined with `, `. No hand edits. The worst-case input is 389 of 512 tokens, so every queue keeps all 15 intents and nothing is truncated. Because the descriptions list CLINC150's own intent names, each message's human-assigned intent name appears inside its reference queue's label (for example `book hotel` in `travel`). Config 2 therefore measures how far naming the request types a queue owns moves routing, with that overlap disclosed. The exact labels are in `clinc150/cases.json` under `definition_labels`. ### Recorded result Both configs ran once on 2026-09-15 against SIE Cloud, server version `0.7.3`, and all 24 requests returned HTTP 200. - Config 1, domain names (`21:33:09Z` to `21:39:51Z`; the first request included a 397-second cold start): the reference queue ranked first for 3 of 12 messages. `home` was the top queue for 6 of the 12. - Config 2, described queues (`21:50:45Z` to `21:50:50Z`): the reference queue ranked first for 5 of 12. It gained the internet bill (`banking`) and the damaged card (`credit cards`) and lost no config 1 match. `travel` was the top queue for 9 of the 12, and the reference queue's rank fell for 5 messages, rose for 2 and stayed the same for 5. Neither config put the reference queue first for half of the messages, so this evaluation did not produce a page claim. Requests, raw responses and both manifests are kept in `apps/site/tests/fixtures/reference/classify/clinc150/`. # Earlier evaluation: CFPB complaint narratives This evaluation ran first. Its reference labels are the products consumers selected on the CFPB form, which made the result hard to interpret, so the page moved to CLINC150. Its fixtures now live in `apps/site/tests/fixtures/reference/classify/cfpb/`. ## Recorded model output - Model: `knowledgator/gliclass-large-v3.0` - Hugging Face revision pinned in the public SIE model config: `e065d1844f913a9aa611cf33623a9538b8aa8841` (SIE Cloud does not document how its served revision header below maps to a Hugging Face commit, so no mapping is claimed) - Model card: - Paper: - Endpoint: SIE Cloud, `https://api.superlinked.com/v1/extract/knowledgator%2Fgliclass-large-v3.0` - Run window: 2026-09-15, from `20:20:21Z` to `20:20:27Z` - Runtime: SIE Cloud, server version `0.7.3` (`x-sie-server-version` response header) - Model revision header returned by SIE: `10333b84de80b402376b626eb25366fb081d3faeb893eb4b01cf32e8c27e4aff` - Runner: `apps/site/tests/fixtures/reference/classify/run.py` - Run manifest: `apps/site/tests/fixtures/reference/classify/manifest.json` - Requests: 12, all HTTP 200 on the first attempt, each between 0.41 and 0.79 seconds end to end from the client. These timings are not a benchmark. ## Recorded result The queue the consumer chose ranked first for 3 of the 12 complaints. Of the nine misses, the model ranked Credit reporting first for five, Credit card for three and Debt collection for one. In single-label mode the scores for one complaint sum to 1, so a consumer queue in second or third place often scores near 0. Every complaint's scores are kept exactly as returned. Each request sends one complaint narrative and the same 10 queue labels, with no other options. The SIE configuration runs GLiClass in single-label mode with a 0.0 threshold, so the response carries a score for every label, sorted from highest to lowest. The API returns no token or phrase attribution, so the page highlights no words as a reason for a score. The website does not serve the raw model files. Site CI verifies the non-served fixtures in `apps/site/tests/fixtures/reference/classify/cfpb/`. The `requests/` and `responses/` files hold the bytes exactly as sent and received, and the unit test checks their SHA-256 against `manifest.json`. An earlier run the same day, from `19:59:36Z`, returned identical scores for all 12 complaints but stored re-serialized responses, so this run replaced it. ## Primary source - Publisher: Consumer Financial Protection Bureau - Collection: CFPB Consumer Complaint Database Narratives Archive, FOIA Electronic Reading Room, - File: `CCDB_Export_20_July_2026.zip`, - Retrieved: 2026-09-15 - ZIP SHA-256: `aafebbaaf0962c3d29532411ac2133c01bae830b7ea6ed1931a6e42278437d18` - CSV inside: `CCDB_Export_20_July_2026.csv`, SHA-256 `beb05a0a61e1ccfe186ad1f7c6ab062bc7a089f7bda28adfe7e6c4c9ccefa57c` - Rights basis: the CFPB website legal notice states that information created by the CFPB is in the public domain. On 2026-08-14 the CFPB stopped publishing narratives in the live database and moved previously published narratives to its FOIA Reading Room as public records. Consumers opted in to publication, and the CFPB removed personal information before release (redactions appear as `XXXX`). Please cite the CFPB as the source. The July 2026 export is the newest archive file with narratives. The August 2026 export has narrative text on 1 of 667,140 rows because publication stopped before those complaints would have been published. ## Labels and reference answers The 10 queue labels map one-to-one to CFPB product values: | Queue label | CFPB product | |---|---| | Credit reporting | Credit reporting or other personal consumer reports | | Debt collection | Debt collection | | Credit card | Credit card | | Checking or savings account | Checking or savings account | | Money transfer or virtual currency | Money transfer, virtual currency, or money service | | Mortgage | Mortgage | | Vehicle loan or lease | Vehicle loan or lease | | Personal or payday loan | Payday loan, title loan, personal loan, or advance loan | | Student loan | Student loan | | Prepaid card | Prepaid card | ### Definition labels (second run config) A second run sends the same 12 complaints to the same model with the same server threshold, but each queue label carries the CFPB's own scope for that product. The rule was fixed and committed before that run and was not changed afterwards: - take every `Sub-product` value the CFPB publishes for the mapped product in the same July 2026 export, excluding `I do not know`; - sort the values alphabetically and join them with `; `; - prefix the queue label and a colon, for example `Vehicle loan or lease: Lease; Loan`. The CFPB's [Consumer complaint form product and issue options](https://files.consumerfinance.gov/f/documents/cfpb_consumer_complaint_form_product_issue_options_August_2023_FINAL.pdf) (August 2023, SHA-256 `6827dc88463217507e103fb0ec0eb7583dc203eb608ff6efd152bebc18655535`) lists the same sub-products on the form consumers use. CFPB's Ask CFPB pages define only five of the ten products in prose, so this list is the one CFPB source that covers every queue the same way. The full labels are in `cases.json` under `definition_labels`. Result: the run went from `20:32:12Z` to `20:32:18Z` on 2026-09-15, and all 12 requests returned HTTP 200. The consumer's queue ranked first for 3 of 12 complaints, the same count as plain queue names, but not the same complaints: Money transfer or virtual currency moved to first place, and Credit card fell to eighth. Across the 12 complaints the consumer's queue ranked higher for 3, lower for 6 and unchanged for 3. Longer labels did not help. The requests, responses and `manifest.definitions.json` stay in the fixtures so this result can be checked. The reference answer for each complaint is the queue for the product the consumer selected on the CFPB form. Consumers choose it themselves, so it is real but noisy: a complaint about a collector reporting to a credit bureau can reasonably sit under either product. The model never sees the product field. The CFPB product "Debt or credit management" is not a queue. ## Selection All selection happened before any model call. 1. Automated filter over the July export (`tmp/classify/shortlist_cfpb.py`, not committed): product in the 10 queues; narrative 250 to 1,300 characters; at most 4 redaction runs; no profanity; the complained-about company's name does not appear in the narrative; the narrative contains a keyword belonging to a different queue, such as "credit report" in a debt collection complaint. 580 narratives qualified, listed in a fixed hash order. 2. Editorial pass in that order: narratives naming a brand, statute-citation template letters, and narratives with health, family or crime details were skipped. One or two complaints per queue were kept. Every kept complaint was run, and every result is kept in the fixtures. ## Complaints Model input is the `Consumer complaint narrative` column, verbatim, with outer whitespace stripped. Per-complaint text and row checksums are in `apps/site/tests/fixtures/reference/classify/cases.json`. | Complaint ID | Received | Consumer chose | CFPB issue | |---|---|---|---| | 23876229 | 2026-07-05 | Credit reporting | Improper use of your report | | 24120783 | 2026-07-12 | Debt collection | Attempts to collect debt not owed | | 24182087 | 2026-07-14 | Debt collection | Communication tactics | | 24198294 | 2026-07-14 | Credit card | Other features, terms, or problems | | 24638689 | 2026-07-27 | Checking or savings account | Problem with a lender or other company charging your account | | 23772737 | 2026-07-01 | Checking or savings account | Managing an account | | 24177261 | 2026-07-14 | Money transfer or virtual currency | Trouble accessing funds in your mobile or digital wallet | | 24191603 | 2026-07-14 | Mortgage | Trouble during payment process | | 24123820 | 2026-07-13 | Vehicle loan or lease | Repossession | | 24280923 | 2026-07-16 | Personal or payday loan | Getting the loan | | 24193226 | 2026-07-14 | Student loan | Problem with a company's investigation into an existing problem | | 23886378 | 2026-07-06 | Prepaid card | Problem with a purchase or transfer |