# Caption / VQA page sources

## Recorded model output

- Model: `Qwen/Qwen3.8-27B-FP8`, native generate with one image
  (`POST /v1/generate/Qwen__Qwen3.8-27B-FP8`)
- Served model revision header (`x-sie-model-revision`):
  `8bd714204e67a1c6c81f84b0dc486b6a6e96e943c42ff488f6b3cbf936e07955`
- Endpoint: SIE Cloud, `https://api.superlinked.com`, server version `0.7.3`
- Run: 2026-09-15, one complete run of all 12 cases, per-call metadata in
  `apps/site/tests/fixtures/reference/caption-vqa/manifest.json`
- Runner: `apps/site/tests/fixtures/reference/caption-vqa/run.py`. The manifest's
  `runner_sha256` pins the script as it executed; later edits to the runner show
  up in git history rather than changing the recorded run.
- Request per case: the question as `prompt`, `max_new_tokens: 256`, and the
  committed JPEG base64-encoded in `images`. This is the request the shared
  playground snippet builds for this task.

Every answer on the page is the first line of the text SIE returned for the
exact request stored beside it. Stored requests replace the base64 image with
the file path and its SHA-256; the file itself is committed under
`apps/site/public/reference/caption-vqa/`.

### Pre-registration and scoring

Questions, expected answers and the scoring patterns were committed before the
first recorded call, in commit `542208c9` at `22:38:21Z`. The probe ran at
`22:43:25Z` and the full run at `22:46:08Z`, so the registration precedes both,
and the commit timestamps are the binding proof.

One caveat, recorded rather than corrected: `inputs.json` carries a
hand-written `registered_utc` of `2026-09-15T23:40:00Z`, an hour later than the
commit and therefore later than the runs it governs. The value was typed from a
local clock. The file is left exactly as registered because
`manifest.inputs_sha256` pins its bytes and the page test compares the two, so
editing the field now would break the tamper evidence it exists to provide.

Scoring is deterministic regex matching over the answer line; it is a check, not
a model call.

One probe ran before the set, with the bare question as the prompt. The model
answered with visible step-by-step reasoning and concluded `420 psi` on the fire
hose gauge, where the published caption says 400 psi, so the probe is a miss.
That probe also showed that a full-text regex could match a scale label inside
the reasoning rather than the answer. Before any other case ran, two things
changed and were committed in `285516b0` at `22:46:03Z`, five seconds before the
full run began:

1. every prompt ends with `Start your reply with the answer in one sentence.`;
2. the pre-registered patterns score the answer line only, with full-text
   matches recorded alongside for transparency.

Questions, expected answers and patterns were not otherwise changed. The probe
request, response and manifest are kept in
`apps/site/tests/fixtures/reference/caption-vqa/probe/`.

### Results

Ten of the twelve recorded answers match the published caption, the drawing text
or the display. Both misses are analog needle gauges:

- `fire-hose-gauge`: returned `approximately 350 psi`; the caption says 400 psi.
- `mir-pressure-gauge`: returned `approximately **130 mm Hg**`, reading the
  inner sub-dial rather than the outer millimetre scale the red range window
  selects.

Every other case returned the printed or drawn value exactly: the placard number
and hazard class, the flow meter counter, the breaker count and rating, the two
B6 lighting circuits with their load, the relay and firing current, the drain
tank capacity and relief setting, the lockout state, the tire pressure, the
sight glass function and the `USE NO OIL` warning.

### Audit of the displayed misses

Both misses were re-read against the image and the question shown beside them,
so the page never prints a failure next to an answer a reader can see is right:

- `fire-hose-gauge`: the needle's tapered tip sits at the 400 psi mark, beside
  the printed 400, with the blunt tail at lower right. `approximately 350 psi`
  is a misreading of about one major division, not a rounding or wording
  difference.
- `mir-pressure-gauge`: the needle crosses the outer scale just past 760 while
  the range window reads `0-460-960` with its marker on the 960 side, selecting
  that band. `approximately **130 mm Hg**`, which the reply justifies as a mark
  between 120 and 140 on the inner scale, is the wrong scale and the wrong
  position.

Both failed on the value, not on formatting. The patterns accept `psi` and
`pounds` wording and a bare millimetre figure, so a symbol, a space or a dropped
article cannot manufacture a miss. The ten matches were re-read the same way,
including `boiler-backhead`, where `water level gauge` is accepted as a
paraphrase of the caption's `sight glass`. The audit moved nothing: 12 recorded,
10 matched.

**Known limit of the scoring.** Each pattern compares a numeric value in the
units its question asks for, so a correct answer given in other units,
kilopascals in place of psi for instance, would score as a miss. No recorded
answer used other units, so no case is affected. The limit is written down
because it did not bite, not because it cannot.

The page shows seven of the twelve recorded cases: the padlocked switch in the
hero, five in the proof section and the tire pressure display in the playground.
Both misses are among them, and the proof section states the totals over all
twelve. Every recorded case, displayed or not, is listed above and kept in the
fixtures.

Five proof cards rather than three, deliberately. Each card holds one cropped
image, one question, one answer and one check line, which is the short scannable
end of the density rule rather than a transcript or a multi-field table. Three
cards would also distort the rate: two of the twelve recorded answers are
misses and both stay on the page, so a three-card grid would either drop a miss,
which is cherry-picking, or make two thirds of the displayed set failures
against two twelfths of the run. Five is the smallest selection that keeps both
misses, and it reduces the distortion without removing it: two misses among five
cards is 40 percent of the displayed set against 16.7 percent of the twelve
recorded cases, where three cards would have been 67 percent. The line under the
grid carries the full-run figures, so a reader sees both rates.

### Display transformation

Cards show the question and the returned answer line. Every prompt also ended
with the answer instruction quoted above; the playground shows the full prompt
as sent. Answer text keeps the returned wording, with the whitespace
around that first line trimmed, including the Markdown asterisks in the Mir
answer. The untrimmed reply is in the fixtures, and the playground prints the
whole text. Card images are cropped and zoomed for display
only; the model received the full committed file.

The website does not serve the raw run records. Site CI compares the page data
with the non-served fixtures in
`apps/site/tests/fixtures/reference/caption-vqa/`.

## Source images

Every image is a public-domain or CC-licensed file from Wikimedia Commons,
resized for the web from the original. `omre-drain-tank-pid.jpg` is the only
crop: a 2000 by 1400 pixel region of the original drawing, taken so the drain
tank lettering survives the resize. Checksums below are of the committed files.

| Local file | Source | Author | License | Published SHA-256 |
|---|---|---|---|---|
| `fire-hose-gauge.jpg` | [Firefighters check a kilometer of firehoses](https://commons.wikimedia.org/wiki/File:Firefighters_check_a_kilometer_of_firehoses_(6871522).jpg), [DVIDS 6871522](https://www.dvidshub.net/image/6871522) | U.S. Army photo by Pierre-Etienne Courtejoie | Public domain (U.S. Army work) | `51b083dfcab71e5f8bdd84fff6ea6d4b9a874556f473f7e328aeef114b93dffc` |
| `fuel-flow-meter.jpg` | [KC-135 Stratotanker hot-pit refuel at RAF Fairford](https://commons.wikimedia.org/wiki/File:KC-135_Stratotanker_%22hot-pit_refuel%22_at_RAF_Fairford_(8632066).jpg), [DVIDS 8632066](https://www.dvidshub.net/image/8632066) | U.S. Air Force photo by Airman Adam Enbal | Public domain (U.S. Air Force work) | `4b04fe7db07e9adaf0cfe743a7cd08c53dd8ca1dfb0d3dfb835481e2ddd52091` |
| `mir-pressure-gauge.jpg` | [Space station MIR absolute pressure gauge](https://commons.wikimedia.org/wiki/File:Space_station_MIR_absolute_pressure_gauge.jpg) | Steve Jurvetson | CC BY 2.0 | `29ec2123a43cb3d464d3480b85458f1179e089f11d80c7cb209d7d691bc730a8` |
| `diesel-truck-placard.jpg` | [Diesel truck](https://commons.wikimedia.org/wiki/File:Diesel_truck.jpg) | Ben Ostrowsky | CC BY 2.0 | `17d51ad9c1537080358be6209dc5673e1fd618f1bc2a20c681463ed6e9331f21` |
| `breaker-row.jpg` | [ABB 230V 16A fuses in fuse box in German shop](https://commons.wikimedia.org/wiki/File:ABB_230V_16A_fuses_in_fuse_box_in_German_shop,_2024.jpg) | Pittigrilli | CC0 1.0 | `aa311c79477807536987252fee0a13175721d4292fcc56c4010c91779fd2218f` |
| `distribution-board-diagram.jpg` | [Wiring diagram of distribution board](https://commons.wikimedia.org/wiki/File:Wiring_diagram_of_distribution_board.jpg) | Dmitry G | Public domain (released by the author) | `ea0b093ff86ce2169509c45959dd4aee4b53baeafaad9e30a4e804282209e407` |
| `main-switch-lockout.jpg` | [Glavno stikalo, zavarovano pred neželenim vklopom](https://commons.wikimedia.org/wiki/File:Glavno_stikalo,_zavarovano_pred_ne%C5%BEelenim_vklopom.JPG) | Svetina D | Public domain (released by the author) | `c01b3688e0e661f8af5fb68b4e50348fec4459c9bbb8a0ca1cd28127dd04be6d` |
| `tire-pressure-display.jpg` | [Chevy Cruze TPMS display](https://commons.wikimedia.org/wiki/File:Chevy_Cruze_TPMS_display.jpg) | WillisMoon | CC BY 4.0 | `bd196f56f8aa97af11814baabfd047a52b8fb1edbd6c8f4874d1f705acb720a3` |
| `boiler-backhead.jpg` | [A boiler of steamboat Ticonderoga (1906)](https://commons.wikimedia.org/wiki/File:A_boiler_of_steamboat_Ticonderoga_(1906).jpg), Historic American Engineering Record | National Park Service | Public domain (U.S. Government work) | `eab86df2c07c52d3c75ce82720c63211d05efe15d09d0ddcc87ffcadaf213a75` |
| `mariner-shutoff-schematic.jpg` | [Schematic diagram of nitrogen-pressure shutoff system](https://commons.wikimedia.org/wiki/File:Schematic_diagram_of_nitrogen-pressure_shutoff_system.jpg), from [NASA SP-59, Mariner-Venus 1962 Final Project Report](http://history.nasa.gov/SP-59.pdf) | NASA | Public domain (NASA work) | `2f2ef963abe9a4ce6f9a78dfcabf28feadb6be8455a48cf90b6c562a41555f7f` |
| `omre-drain-tank-pid.jpg` | [Organic Moderated Reactor Experiment piping and instrumentation diagram](https://commons.wikimedia.org/wiki/File:Organic_Moderated_Reactor_Experiment_Piping_and_Instrumentation_Diagram.jpg), [OSTI 4307822](https://www.osti.gov/biblio/4307822) | U.S. Atomic Energy Commission | Public domain (U.S. Government work) | `857f392bf29db2cdaf321a24f3487c39613fc385fde9b1160a9d8085f5d006da` |
| `oxygen-gauge-dial.jpg` | [Psidial](https://commons.wikimedia.org/wiki/File:Psidial.jpg) | leapingllamas | CC BY 2.0 | `497f4d7aff8964e4562153b6e54a663f31e034c7985863d6d6783d486022769c` |

## Where each expected answer comes from

Six expected answers are quoted from the file's published caption and three are
read off the drawing itself. One comes from the display text with the caption
confirming the layout, one combines the caption with the panel's own label
strip, and one is the page author's reading. All three are marked below.

| Case | Expected answer | Basis |
|---|---|---|
| `fire-hose-gauge` | 400 psi | Caption: "A manometer indicates 400 psi" |
| `fuel-flow-meter` | 4.37 cubic metres | Caption: "A flow meter reads 04.37 CUBIC METERS" |
| `mir-pressure-gauge` | About 761 mm Hg | **Page author's reading** of the full-resolution photo. The caption states only that the gauge is still accurate and that 1 atm = 760 mm Hg, and that the red window selects the 460 to 960 scale. |
| `diesel-truck-placard` | UN 1993, class 3 | Caption: the truck carries diesel fuel under placard 1993; the flame placard is visible |
| `breaker-row` | Six breakers, B16 | Caption names the ABB 16 A breakers; the F1 to F6 strip counts six |
| `distribution-board-diagram` | Stairs I and Stairs II lighting on B6 breakers, 120 W each | Drawing text. The pre-registered patterns stay frozen as recorded and do not name the breaker type, so a page test additionally requires the displayed answer to say B6 and not B16 |
| `main-switch-lockout` | No, it is padlocked | Caption (Slovenian): secured with a lock and tagged "NE VKLAPLJAJ" (do not switch on) |
| `tire-pressure-display` | Front right, 30 psi | Display text, with the caption confirming the pictograph layout |
| `boiler-backhead` | The water level in the boiler | HAER caption: "sight glass (indicates water level in boiler)" |
| `mariner-shutoff-schematic` | K10; 3 amp | Drawing text |
| `omre-drain-tank-pid` | 6000 gallons; 50 psig | Drawing text |
| `oxygen-gauge-dial` | USE NO OIL, oxygen service | Caption: "manometer for oxygen (use no oil)" |