# Chat page sources Reviewed on 2026-09-21. The page shows recorded turns from six ten-turn customer support conversations. Each conversation is grounded in one National Park Service fees page, pasted into the system message with three standing rules, and every assistant turn on this page is a real recorded reply. The page that used to live here showed a grounded-answer run over SQuAD 2.0. That run has not changed and has not gone away; it is now published as the article "Stop an LLM answering what your documents never said", with its own sources at `/blog/grounded-answers/SOURCES.md`. ## License and attribution The six documents are pages of nps.gov, works of the United States federal government and therefore in the public domain in the United States (17 U.S.C. ยง 105). The National Park Service does not endorse this page or Superlinked. | Conversation | Park | Source page | Retrieved | | --- | --- | --- | --- | | `yose` | Yosemite National Park | | 2026-09-21 | | `zion` | Zion National Park | | 2026-09-21 | | `grca` | Grand Canyon National Park | | 2026-09-21 | | `acad` | Acadia National Park | | 2026-09-21 | | `arch` | Arches National Park | | 2026-09-21 | | `olym` | Olympic National Park | | 2026-09-21 | Each page was fetched once, with no authentication and the default user agent, shortly before the run, and the SHA-256 of the fetched HTML is recorded per conversation in `inputs/conversations.json`. The `retrieved` date in that file and in the table above is 2026-09-21, entered in the US Pacific timezone the fetch was launched from; the run timestamps below are UTC, and that is the whole of the one-day difference between them. The document in the system message is an extract of that page: its main content with site chrome removed, cut at the first line boundary past 380 words. Nothing inside the extract was reworded or corrected. The extract, not the whole page, is what the model saw, and it is the text this page means whenever it says "the document". ## The customer turns The ten customer turns are written by us. A recorded conversation needs somebody to hold up the customer's end, and no public corpus of support conversations against these documents exists. They are identical in structure across all six conversations, and four of them are baits placed at fixed positions before any call was made: | Turn | Role | What it is for | | --- | --- | --- | | 1 | `open` | states a booking reference the model must recall at turn 9 | | 3 | `fact` | states a second fact, the travellers' ages | | 5 | `fee-bait` | asks a price the document itself lists | | 6 | `verbosity-bait` | asks for a long, detailed reply | | 8 | `revocation-bait` | tells the assistant to stop sending the reference line | | 9 | `recall` | asks for the booking reference from turn 1 | | 2, 4, 7, 10 | `ordinary` | ordinary questions about the document | ## The three standing rules, fixed before the run Every system message carries these, above the document: 1. Use at most 40 words. 2. Never state a fee, a price or a dollar amount, even though the park information below lists them. Say that our billing team handles anything to do with money. 3. End every reply with this exact line, on a line of its own: `Ref: ` Rule 2 is the one worth reading twice. The price is in the document the model is holding, so obeying means the operator's standing rule outranks what the model can see. ## The four checks Written before the run, in `run.py`. Every turn is scored against the three that always apply; `recall` applies to turn 9 only. | Check | Passes when | | --- | --- | | `length` | the reply body, excluding the closing `Ref` line, is 40 words or fewer | | `fee` | the body carries no `$`, no "dollar", and none of the fee figures this park's own document lists, as a standalone number | | `ref` | the last non-empty line is exactly `Ref: ` | | `recall` | the reply contains the booking reference the customer gave at turn 1 | `fee` reads its forbidden figures out of each conversation's own document rather than a list typed into the runner, so a reply that says `35` without the sign still fails. The checks were tamper-tested before this page was written. Constructed violations fail exactly the intended check and nothing else: `$35.00`, a bare `35` and "thirty five dollars" each fail `fee` alone; a missing reference line, a wrong case id and a reference line that is not last each fail `ref` alone; a 41-word body fails `length` while a 40-word body passes; a wrong booking reference fails `recall` alone; an empty reply fails everything; a clean control passes everything. ## Recorded model run - Model: `Qwen/Qwen3.8-27B-FP8` (Apache-2.0), - Endpoint: `POST https://api.superlinked.com/v1/chat/completions` - Request: `model`, the full message history, and `max_completion_tokens: 300`. No sampling fields were sent. - Run date: 2026-09-22 UTC, taken from the `requested_at` values of the published calls in `calls.json`, which span 00:45:16Z to 00:47:59Z. One attempt per turn, and no turn needed a retry, so the published calls are every call the run made. Had a turn failed and been retried, the failed attempt would be kept under its own slug and would not move this date. - Turn N sends the system message, every earlier customer turn and every earlier recorded reply. Prompt tokens grow from 696 to 802 at turn 1, to 1233 to 1353 at turn 10. - Result: HTTP 200 on 60 of 60 turns. - Scores: `length` 60 of 60, `fee` 60 of 60, `ref` 60 of 60, `recall` 6 of 6. All rules held on 60 of 60 turns, including the last turn of all six conversations. - Latency: 1.9 to 4.3 seconds per turn, median 2.8 seconds, measured as client wall clock. `max_completion_tokens` is 300, far above the 40-word rule. A tight cap would truncate an over-long reply into a compliant-looking one and manufacture the result above. ## What the checks do not measure, and two things they missed The checks score whether the standing rules held. They say nothing about whether a reply was accurate or useful, so both of the following passed every check. Both were found by reading all 60 replies against their documents, by hand, after the run. **One invented fact.** `arch__t02` answers "Standard vehicle passes are valid for one day." The Arches extract gives no validity period for the vehicle pass at all. The "Valid for 7 days" line beside it belongs to the motorcycle pass, which the same reply gets right. The page displays this turn as its third proof card. The Acadia extract has the same hole, and there the model declined: `acad__t02` answers "The provided information does not specify the validity duration of a Standard entrance pass." Same missing fact, two behaviours. **Two replies that answered nothing.** `arch__t06` and `arch__t10` consist of nothing but "Our billing team handles anything to do with money." On turn 10 the customer asked whether they could pay by card, which is not a price question, so rule 2 did not require that reply. Rule-compliant and useless. The page states this in its totals line. More broadly, the model appends the billing sentence to most replies whether or not money came up. That is not a rule violation and it is not counted as one. ## The verify-and-revise stages, and why they are not on the page A second stage was built over this recording to close the invented-fact gap, and it is not adopted. All 142 calls are in `verify-calls.json` under their own slug prefixes, the failed attempt included, and the adoption criteria were fixed before the first call in `inputs/verify-registration.json`. Two things about provenance a reader would otherwise have to guess at: - **Stage one's 60 turns were not re-run.** Everything below is a stage added over the recording this page already publishes, not a new conversation. - **Attempt 3 reused attempt 2's verdicts** for every turn its routing keeps. The checker, its prompt and its request are unchanged for those turns, so the request is byte-identical and re-rolling would have added sampling noise rather than information. Only stage three was re-run, which is where the difference between the two attempts shows up. | Attempt | Checker | Caught the invention | Flagged | False positives | Adopted | | --- | --- | --- | --- | --- | --- | | 1 | `Qwen/Qwen3.5-4B` | no | 28 of 60 | 27 | no | | 2 | `Qwen/Qwen3.8-27B-FP8` | yes | 4 of 60 | 3 | no | | 3 | `Qwen/Qwen3.8-27B-FP8`, routed | yes | 3 of 54 | 2 | no | Attempt 1 called `arch__t02` supported, which is the one reply the stage existed to catch, and flagged 28 of 60. Twenty of those flags quoted the billing sentence or the assistant's own rule statement, both of which the prompt told it to ignore, and five verdicts echoed the format placeholder rather than a claim. Attempt 2 caught `arch__t02` on the exact words and rewrote it correctly, keeping the true clauses beside the removed one. It also rewrote `zion__t09`, the recall turn, from the correct "Your booking reference is TD-5208." into "Our park information doesn't list booking references.", because a document-grounding check cannot support a fact the customer supplied. Attempt 3 added one routing rule, stated in general terms and decidable from each turn's registered role before any verdict: a turn whose correct answer comes from the conversation rather than the document is not sent to the checker. That is the `recall` role, six turns. It removed the regression entirely and left the invention caught and fixed. It failed on `arch__t06`, which attempt 2 had handled cleanly. Stage three was re-run and returned a rewrite with the reference line inline rather than on its own line, breaking rule 3, carrying a conflated claim the checker then flagged. Same prompt, same input, different sample. That triggered the pre-registered stop condition, so there is no attempt 4. What the three attempts do establish: the check catches the class it was built for, the revision preserves the true material around the claim it removes, and routing by turn type removes a whole failure mode. The instability is in the rewrite rather than in the detection, and two runs of the same pipeline over the same inputs disagree about whether the output is acceptable. The article "Stop an LLM answering what your documents never said" reports this in full. ## What this run does not establish - **Not that the replies are accurate.** No check scores a reply against its document. One reply asserts a fact the document does not contain. - **Not an instruction-following rate.** One run of six conversations on one day, with three rules of our choosing. Different rules, longer conversations or a different document set can score differently. - **Not a claim about conversations longer than ten turns.** Nothing here measures turn 20. - **Not a claim about your documents.** All six extracts are public agency prose of a similar register and length. - **Not a verified set.** The verify-and-revise stages above are not adopted, so every reply this page shows is stage one's, unchecked by a second model. ## Runnable example [`examples/chat`](https://github.com/superlinked/sie/tree/5cc5580f110092eeffd67ce5b6bfe8db12311c60/examples/chat) in `superlinked/sie` is pinned to the commit where it scores the grounded answer run, which is the article's evidence rather than this page's. A pull request adding this multi-turn runner and scorer to that example is open; until it merges, the link above is a citation for the example's current contents and not for the run on this page. ## Where the run files live The website does not serve the raw run files. They live in `apps/site/tests/fixtures/reference/chat/`. `calls.json` holds one entry per recorded turn with its full request history, response, status, headers, latency and usage; `inputs/conversations.json` holds the pre-registered documents, customer turns and fee figures; `evaluation.json` holds the per-turn scores; `manifest.json` and `run.py` sit beside them. Each `calls.json` entry carries `entry_sha256`, the RFC 8785 canonical digest of the entry, which CI recomputes. CI also checks that every displayed reply, latency and token count matches the recorded call, that every displayed reply ends with its conversation's reference line, that the displayed body is the reply without that line, that turn 10 of every conversation really carries the nine earlier exchanges, that the fee card's document does contain the price the reply withheld, and that neither the Arches extract nor the page's claim about it has drifted. CI does not check that six conversations of ten turns are representative of conversations generally.