# Sources for the grounded-answer run Reviewed on 2026-09-21. Every input is a passage and a question from the development set of SQuAD 2.0, a public reading-comprehension benchmark. The article "Stop an LLM answering what your documents never said" reports this run: what was asked, what the model replied, and how each reply scored. This run was recorded for a /chat task page that has since been rebuilt around a multi-turn corpus. The run itself is unchanged; only what publishes it moved. ## License and attribution SQuAD 2.0 is released under the Creative Commons Attribution-ShareAlike 4.0 International license (). Its passages are paragraphs from English Wikipedia, whose authors are the contributors listed in each article's history. Neither the SQuAD authors nor the Wikimedia Foundation endorses this page or Superlinked. - Dataset file: - SHA-256 of that file: `80a5225e94905956a6446d296ca1093975c4d3b3260f1d6c8f68bc2ab77182d8` - Paper: *Know What You Don't Know: Unanswerable Questions for SQuAD*, Changes: passages are used verbatim. Questions are used verbatim. Nothing was reworded, trimmed or corrected, including the occasional crowdworker typo. ## Selection, fixed before the first call Twelve development-set articles on distinct, non-political subjects, one passage each, so no single field frames the page. For each article: one paragraph between 70 and 190 words carrying both an answerable and an unanswerable question, then one question of each kind that is well formed and whose reference answer is a short unambiguous string. That selection, the prompt and the four checks were written into `apps/site/tests/fixtures/blog/grounded-answers/inputs/cases.json` and `run.py` and committed before any request was sent. `run.py verify` re-downloads the dataset, checks it against the digest above, and confirms every pinned passage, question, answerability flag and reference answer is the upstream one. | Page slug | Subject | Wikipedia article | Answerable question id | Unanswerable question id | | --- | --- | --- | --- | --- | | `steam-engine` | Mechanical engineering | Steam_engine | `571127a5a58dae1900cd6cc5` | `5ad3cd43604f3c001a3ff187` | | `pharmacy` | Pharmacy practice | Pharmacy | `5726da89dd62a815002e92b5` | `5a6cb7ce4eec6b001a80a63a` | | `construction` | Construction industry | Construction | `57273e50dd62a815002e9a04` | `5a23548782b03a001a4aa787` | | `immune-system` | Immunology | Immune_system | `5728f2e26aef051400154897` | `5ad49faaba00c4001a268e28` | | `oxygen` | Chemistry | Oxygen | `571a4b0f10f8ca1400304fd7` | `5ad22a95d7d075001a4285db` | | `geology` | Earth science | Geology | `57265f605951b619008f70de` | `5a58db973e1742001a15ce50` | | `prime-number` | Mathematics | Prime_number | `572970c11d04691400779467` | `5a8913713b2508001a72a4a2` | | `complexity-theory` | Computer science | Computational_complexity_theory | `56e16839cd28a01900c6788a` | `5ad532575b96ef001a10ab7e` | | `amazon-rainforest` | Ecology | Amazon_rainforest | `5725b81b271a42140099d098` | `5a0c6698f5590b0018dab3e5` | | `rhine` | Geography | Rhine | `572f567cb2c2fd1400568040` | `5ad2649ad7d075001a429160` | | `force` | Physics | Force | `57373f80c3c5551400e51e8f` | `5ad26103d7d075001a428fef` | | `ctenophora` | Marine biology | Ctenophora | `5725c337271a42140099d163` | `5a834af4e60761001a2eb51d` | A thirteenth passage, paragraph 1 of Steam_engine, carries one further call that the old page used for its playground. It is not scored and is counted in no total the article publishes. It is still present in `evaluation.json`, which holds 25 result rows for 24 scored questions, so any count taken over those rows without excluding it comes out one too high. ## The four checks Written before the run. Each answer is scored against all four. | Check | Passes when | | --- | --- | | `format` | the reply is exactly two lines, the first starting `Answer: ` and the second `Quote: ` | | `quote` | the quoted text appears in the passage character for character, or is exactly `none` when the answer is the abstention line | | `reference` | the answer contains one of SQuAD's reference answers, or, for a question the passage cannot answer, is exactly `Not stated in the passage.` | | `length` | the answer is 25 words or fewer | `reference` compares with SQuAD's own normalization: lowercase, drop articles and punctuation, collapse whitespace. Two documented additions, applied to both sides: Unicode dashes fold to `-` and Unicode quotes to `'`, so two spellings that differ only in a code point a reader cannot see compare equal. ## Recorded model run - Model: `Qwen/Qwen3.8-27B-FP8` (Apache-2.0), - Served execution revision (`x-sie-model-revision` response header): `8bd714204e67a1c6c81f84b0dc486b6a6e96e943c42ff488f6b3cbf936e07955` - SIE server version (`x-sie-server-version`): `0.7.3` - Endpoint: `POST https://api.superlinked.com/v1/chat/completions` - Request: `model`, a system message with the instruction, a user message with the passage and the question, and `max_completion_tokens: 256`. No sampling fields were sent. - Run date: 2026-09-21 UTC, taken from the `requested_at` values in `calls.json`, which span 19:00:23Z to 19:01:48Z. One attempt per question, no retries. - Result: HTTP 200 on 24 of 24 questions, all 24 answered. - Scores: `format` 24 of 24, `quote` 24 of 24, `length` 24 of 24, `reference` 20 of 24. All four checks passed on 20 of 24. - Latency: 1.4 to 7.1 seconds per question, median 3.5 seconds, measured as client wall clock from sending the request to receiving the full response. - Usage: 220 to 377 prompt tokens and 13 to 85 completion tokens per question. - One further call on the thirteenth passage passed all four checks and is not counted in the 24. ## The four answers that missed the reference All four are quoted in full here, because the page describes one of them as a model error and three as wording differences, and that is a person's reading rather than something a check decides. **The model error.** `complexity-theory__unanswerable`, the case the article quotes in full. Question: *What are three basic primary resources used to gauge complexity?* The passage names two. The model answered "Time, storage, and communication are three basic primary resources used to gauge complexity." and quoted "quantifying the amount of resources needed to solve them, such as time and storage". The quote is real and passes the `quote` check; it names two resources and does not carry the answer above it. **Wording differences.** These three carry the reference answer's meaning in other words: - `geology__unanswerable`, *Where in the ocean are volcano arcs located?* -> "The passage does not specify the exact ocean locations where volcano arcs are found." It declines, in its own phrasing rather than the fixed line. - `amazon-rainforest__unanswerable`, *What covers most of the Amazon basin of Central America?* -> "The passage states the Amazon rainforest covers most of the Amazon basin in South America, not Central America." It rejects the question's premise rather than using the fixed line. - `ctenophora__answerable`, *How many species of Ctenophora have been validated?* -> "Only 100 to 150 species of ctenophores have been validated according to the passage." The reference is `100–150`; the answer writes the same range in words. ## The verify-and-revise section The article's closing section reports a second experiment, not this one. It adds a checking stage over the multi-turn park conversations that /chat publishes, in three attempts, none adopted. Its corpus, its pre-registered adoption criteria and all 142 recorded calls are documented at `/reference/chat/SOURCES.md` and live in `apps/site/tests/fixtures/reference/chat/`. ## What this run does not establish - **Not that the answers are true.** `reference` compares against SQuAD's reference answers, which are spans other people annotated. An answer that contains the reference span and surrounds it with a wrong claim still passes. - **Not a hallucination rate.** One run of 24 questions on one day. Sampling defaults were used and no question was repeated, so a rerun can return different wording and a different score. - **Not a benchmark result.** Twelve passages selected by the rule above are not the SQuAD 2.0 development set, and the numbers here are not comparable with published SQuAD scores. - **Not a claim about your documents.** All twelve passages are encyclopedia prose of a similar register. A corpus of tickets, manuals or filings may score differently. ## Runnable example [`examples/chat`](https://github.com/superlinked/sie/tree/5cc5580f110092eeffd67ce5b6bfe8db12311c60/examples/chat) in `superlinked/sie` downloads these recorded calls from the public Hugging Face dataset `superlinked/sie-task-evidence`, pinned to revision `1b6707ad110aaf2c8091c8585400e7b2a7153fa6`, and rescores them with no API key and no inference spend. It reproduces the figures above by a different route from the checks below: different code, different files, and no contact with this repository's fixtures. ## Where the run files live The website does not serve the raw run files. They live in `apps/site/tests/fixtures/blog/grounded-answers/`, and site CI compares the article's published figures with them: `calls.json` holds one entry per recorded call with its request body, response, status, headers, latency and model revision, and `inputs/cases.json` holds the pinned passages, questions and reference answers. `manifest.json`, `evaluation.json` and `run.py` sit beside them. `calls.json` uses the schema `superlinked/sie` uses for `examples/document-to-markdown/runs/*/calls.json`. Each entry carries `entry_sha256`, the RFC 8785 canonical digest of the entry. `src/data/blog-grounded-answers.test.ts` recomputes every figure the article publishes from `evaluation.json` and `inputs/cases.json`, over the twelve scored passages only, and asserts the article's own sentences against the result. It also checks that each cited sentence is a substring of its passage, and that the two replies quoted from the multi-turn run match that run's recording. CI does not check the article's prose beyond those sentences, and it does not check that the twelve passages are representative of documents generally.