# Structured output page sources

The /structured-output page reports two runs on the same inputs. On 30
September 2026 a pre-registered head-to-head sent the same documents, the same
instructions and the same JSON Schema to SIE Qwen3.8 27B and to the OpenAI and
Anthropic models of the time, each at the lowest reasoning setting its API
accepts and each through its vendor's own strict structured-output API, and
scored how many of the returned values exactly matched the answer key. On 11
October 2026 a second pre-registered run sent the identical requests to the
current models, GPT-6.1 Sol, GPT-6 Luna, Claude Haiku 5.5 and Claude Sonnet
5.5, each at its default setting, and paired them against the first run's
saved SIE answers. Study designs, sample sizes and decision rules were
written down before any request, and every change after that was recorded as
a dated amendment before the affected requests. Every figure on the page is
computed from the recorded responses, which are public (see Runnable
example), and the site's build fails if a figure on the page stops matching
them.

The page shows SIE and the four current models, each model at its
best-scoring measured setting. Two of the four, Claude Sonnet 5.5 and GPT-6
Luna, were measured in both runs, so each has two settings on record, and
Claude Haiku 5.5 ran at two settings in the second. The first run's other
arms (Claude Sonnet 5, Claude Haiku 4.5 and GPT-6 Sol, all since superseded,
and Claude Opus 5.5 at effort `low`) are reported below and stay in the
public recordings; none of them is on the page.

The page's claim is exactness. Its headings say "exactly right" and never
"right" alone or "beats": the study's metric counts a value only when it
matches the answer key character for character, and read leniently, and on
the fresh complaints, the models are level (see "Exact values, and what
counts as right" and "Fresh records").

## The benchmark

The main test is the public Structured Output Benchmark (SOB,
[arXiv 2604.25359](https://arxiv.org/abs/2604.25359); data
[interfaze-ai/sob](https://huggingface.co/datasets/interfaze-ai/sob) at
revision `c118e38abdef6a8e1beba183405c70b28ff7d5a8`; code
[JigsawStack/sob](https://github.com/JigsawStack/sob) at `da785a8`). Each of its
5,000 text records is a set of Wikipedia passages, a question, a JSON Schema
written for that question, and a checked answer. We drew 500 records with a
fixed seed, stratified by the benchmark's difficulty and schema-complexity
labels.

**Why SOB.** Every vendor now has a strict mode, so a valid record is the floor,
not the test: SOB's own leaderboard has models 86% to 99.9% valid but only 69%
to 83% right on values. SOB scores the values. It is the standard benchmark for
this, with a public dataset, a public scorer and published results for 35
models.

**Prompt and schema.** SOB's own system prompt and user message, unmodified.
The schema sent is SOB's strict form of each record's schema (every property
required, no extra properties), the form OpenAI's strict mode requires, and the
same schema goes to every model.

**Metric.** SOB's `evaluate_record`, unmodified: the share of the answer's
values that exactly match the checked answer, and zero for an answer that does
not parse or validate. The page's figure is the mean over 500 records, as a
percentage.

**Sample size.** SOB's published per-record results put the spread of the
per-record difference between neighbouring models at 0.18 to 0.27. At 0.24, a
3-point non-inferiority margin (one-sided alpha 0.025, power 0.8) needs 502
records; a 4-point gap needs 283. We ran 500.

## Models and settings

The setting each model is charted at, and the name the page gives it.

| Model | API | Setting | Run | Named on the page |
| --- | --- | --- | --- | --- |
| SIE Qwen3.8 27B | Serverless, `/v1/chat/completions`, `response_format` json_schema, strict | temperature 0, reasoning off | 30 September 2026 | |
| Claude Sonnet 5.5 | Anthropic Messages, `output_config.format` json_schema | thinking `between_tools`, the lowest it accepts | 30 September 2026 | thinking off |
| GPT-6.1 Sol | OpenAI Chat Completions, `response_format` json_schema, strict | reasoning medium, the default (`none` is not offered) | 11 October 2026 | medium reasoning |
| GPT-6 Luna | OpenAI Chat Completions, `response_format` json_schema, strict | reasoning none | 30 September 2026 | no reasoning |
| Claude Haiku 5.5 | Anthropic Messages, `output_config.format` json_schema | thinking off | 11 October 2026 | thinking off |

**The rule.** A bar shows each model at its best-scoring measured setting:
the setting with the highest share of values exactly right, to the tenth of a
point the bar prints, and the cheaper one where two settings print the same
score. A price figure in a heading is computed against the rival's cheapest
measured setting. For every model on this chart the two are the same
setting, so each model has one bar and every heading figure comes out of it.

- Claude Sonnet 5.5 prints 77.2% at both of its settings (77.19% with
  thinking `between_tools`, 77.19% with adaptive thinking at its default
  effort), and costs $7,680 at the first against $7,749 at the second. The
  page charts the first.
- GPT-6 Luna scores 76.6% with no reasoning for $227 and 76.3% at medium
  reasoning for $255. The page charts the first.
- Claude Haiku 5.5 scores 75.9% with thinking off for $385 and 75.4% with
  adaptive thinking for $407. The page charts its thinking-off run because it
  scored higher and costs less there.
- GPT-6.1 Sol was run on all 500 records at one setting, reasoning medium.
  Its `xhigh` replies cover 221 records, drawn for the pain search below, and
  are not a measurement of the model.

The current models reject non-default sampling, so no temperature was sent
to them in the second run. Every arm in both runs had the same output
allowance, 2,048 tokens, as SOB uses; reasoning and thinking tokens count
inside it, as each API bills them. GPT-6.1 Sol reached the allowance on 3
records by writing runs of whitespace inside its JSON; those score zero, as
registered. No Gemini key was available, so no Gemini model was run.

## Results

Value accuracy on the 500 records; SIE minus each model, per-record paired
differences with a 10,000-sample bootstrap 95% interval, in points; mean
tokens per record in and out, each on its own tokenizer, reasoning included;
and dollars for 1,000,000 records at real-time list prices (see Prices).
These are the five bars.

| Model | Setting | Values exactly right | SIE minus model | Tokens in / out | $ per 1M documents |
| --- | --- | --- | --- | --- | --- |
| SIE Qwen3.8 27B | reasoning off | 83.5% | ours | 1,948 / 98 | $727 |
| Claude Sonnet 5.5 | thinking `between_tools` | 77.2% | +6.3 (+4.4 to +8.2) | 3,287 / 111 | $7,680 |
| GPT-6 Luna | reasoning none | 76.6% | +6.9 (+5.0 to +8.9) | 1,947 / 65 | $227 |
| Claude Haiku 5.5 | thinking off | 75.9% | +7.6 (+5.5 to +9.8) | 3,287 / 113 | $385 |
| GPT-6.1 Sol | reasoning medium (default) | 73.7% | +9.8 (+7.7 to +11.8) | 1,947 / 84 | $4,735 |

The other measured settings of those models, which the bars leave out:

| Model | Setting | Values exactly right | SIE minus model | Tokens in / out | $ per 1M documents |
| --- | --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 | adaptive thinking, default effort (high) | 77.2% | +6.3 (+4.4 to +8.2) | 3,288 / 117 | $7,749 |
| GPT-6 Luna | reasoning medium (default) | 76.3% | +7.2 (+5.3 to +9.2) | 1,947 / 120 | $255 |
| Claude Haiku 5.5 | adaptive thinking, default effort (medium) | 75.4% | +8.1 (+5.9 to +10.2) | 3,288 / 156 | $407 |

The Claude models rejected 2 of the 500 schemas (a `minItems` above 1),
which score zero as registered; every other record got a valid answer.

**The reading.** The second run pre-states that a model is behind SIE only
when the lower bound of SIE minus that model is above zero. It is for every
setting in both tables: +4.4 against Claude Sonnet 5.5, the page's headline
comparison, at either of its settings, and +5.0 to +7.7 against the others.
The first run's registered bars were parity with its headline rival (lower
bound above −3 points) and a lead over the cheap models (above zero); every
first-run rival is behind SIE too, as the table below shows. The lead is on
exact values, the registered metric. Read leniently, and on the fresh
complaints below, the models are level.

**The first run (30 September 2026).** Same 500 records and the same SIE
answers; rivals at their lowest reasoning settings. Its Claude Sonnet 5.5 and
GPT-6 Luna rows are the two the page charts.

| Model | Setting | Values exactly right | Lenient | SIE minus model | $ per 1M documents |
| --- | --- | --- | --- | --- | --- |
| SIE Qwen3.8 27B | reasoning off | 83.5% | 93.0% | ours | $727 |
| Claude Haiku 4.5 | temperature 0 | 80.4% | 91.5% | +3.1 (+1.1 to +5.0) | $2,850 |
| Claude Sonnet 5 | thinking disabled | 79.3% | 92.8% | +4.2 (+2.3 to +6.2) | $7,677 |
| GPT-6 Sol | reasoning none | 77.7% | 92.3% | +5.8 (+4.1 to +7.5) | $4,538 |
| Claude Sonnet 5.5 | thinking `between_tools` | 77.2% | 93.1% | +6.3 (+4.4 to +8.2) | $7,680 |
| Claude Opus 5.5 | effort `low` | 76.9% | 92.5% | +6.6 (+4.7 to +8.6) | $15,417 |
| GPT-6 Luna | reasoning none | 76.6% | 92.2% | +6.9 (+5.0 to +8.9) | $227 |

## Was the benchmark's answer right?

Before the run we registered a blind audit of 100 records on which the models
disagreed: if the benchmark's answer turned out wrong on more than 10% of them,
the page would use audited answers. The 100 records were drawn with a fixed
seed from the 339 on which SIE Qwen3.8 27B, Claude Sonnet 5, GPT-6 Luna and
GPT-6 Sol did not all agree, and each was read against its own passages with
the models' answers shown unlabelled.

- **The answer was wrong on 1 record of 100** (1%), under the 10% bar, so the
  page scores against the benchmark as published. In that record the question
  names Claudie Haigneré first and the answer puts Mark Shuttleworth in the
  first slot.
- **24 records had a value where a second answer is also defensible**, for
  example a full name where the schema asks for a full name and the answer
  gives the short one, or a list order the schema does not fix. SOB scores
  those strictly for every model alike.

## Exact values, and what counts as right

SOB counts a value right only when it matches the checked answer exactly. That
is the right test for extraction feeding a database, where "PA" and
"Pennsylvania" are different keys, and it is the metric the study registered.
It also counts some answers wrong that a person would accept: a value
rephrased ("Spring 1859" for "spring of 1859"), expanded ("Pennsylvania" for
"PA", "EUR" for "€"), or padded into a sentence.

Read leniently (equal after SOB's own normalisation of case, punctuation and
articles, or one value containing the other), every model of the first run
lands between 91.5% and 93.1%: SIE 93.0%, Claude Sonnet 5.5 (thinking off)
93.1%, GPT-6 Luna (reasoning none) 92.2%. So most of the gap on the page is
SIE returning the document's own words as the value where the other models
rephrase or expand them, which is what a database key needs. The page charts
the registered, exact metric.

## Fresh records: NHTSA complaints

A model may have seen SOB's Wikipedia passages in training. So the study also
ran 150 NHTSA vehicle-safety complaints filed between 1 and 30 September 2026,
after every model's training data, through the page's own complaint schema
(the one in the hero). They were drawn with a fixed seed from the 3,215
complaints filed that month on twelve makes' 2018 to 2026 models, half from
complaints the agency coded with a crash, a fire or an injury. The answer key
is the agency's own coding: crash, fire and number injured, plus the make,
model and year where the narrative names them (64 of 150).

| Model | Values right | SIE minus model |
| --- | --- | --- |
| GPT-6.1 Sol | 92.7% | −0.9 (−2.0 to +0.1) |
| Claude Sonnet 5.5 | 92.3% | −0.6 (−1.2 to −0.1) |
| Claude Haiku 5.5 | 91.9% | −0.1 (−1.0 to +0.8) |
| SIE Qwen3.8 27B | 91.8% | ours |
| GPT-6 Luna | 91.4% | +0.3 (−0.4 to +1.1) |

Each model is at the setting the page charts it at. The fresh records put
every model within 1.3 points of the others, and SIE within a point of each
rival. SIE is 0.1 to 0.9 points behind three of the four current models and
0.3 ahead of GPT-6 Luna; only the gap to Claude Sonnet 5.5 (−0.6, −1.2 to
−0.1) has an interval that excludes zero, so on this set Claude Sonnet 5.5
scores higher than SIE. The registered bar for the set was SIE minus the
headline rival within −5 points, a non-inferiority bar, which SIE meets. The
page's lead rests on the 500 benchmark records above. The fresh set is not on
the page: it is level, SIE's 91.8% beside the rivals' range (91.4% to 92.7%),
and Claude Sonnet 5.5 is 0.6 points ahead there, with an interval clear of
zero.

Every Claude model refused the complaint schema itself, all 150 times: the
Anthropic API does not accept `minimum` on an integer (`injured_people` must
be zero or more). The Claude figures above come from the same schema passed
through the Anthropic SDK's own `transform_schema`, which is what
`messages.parse` sends from a Pydantic model: it moves the constraint into the
field's description. Sent as written, the schema returns a 400 on every call.

## The hero

The complaint in the hero is NHTSA complaint 11764705, filed 16 September 2026,
one of the 150 fresh records. A continuous source passage appears beside values
from SIE Qwen3.8 27B's saved reply, at temperature 0. The narrative describes
a Tahoe High Country with 18,000 miles, two occupants, flames in the engine
compartment and no reported physical injuries. The displayed fields retain
the reported fire and distinguish occupants from injured people. The displayed
passage keeps the fire spread, risk to both occupants and explicit absence of
physical injuries. Restrained highlights appear within the continuous prose
beside one JSON record. The complete source and reply remain in the API viewer
and public recordings.

The agency-coded model is TAHOE; the saved response includes the source-stated
trim, Tahoe High Country. The highlighted hero fields are fire and injuries;
they do not assert an exact answer-key match for that longer model name.
Mileage is not an agency-scored field. The saved reply also says `crash: false`, consistent with
the agency code; the narrative mentions a crash risk without an explicit
crash negation, so the hero does not highlight that value. Comparative
accuracy comes from the full study.

The shared input/output viewer uses this saved document, instruction, schema
and reply. Its copied requests preserve the source bytes, the 2,048-token
output cap, temperature 0 and JSON Schema name `answer`. They use a normal
non-streaming call; the recorded evaluation runner streamed to capture timing
and usage. The visible JSON remains the original saved reply.

## The pain section

The pain section is the bill, for the two strong API models on the chart. It
shows two cases, one per provider, each at the setting the bars chart, which
the card's header names, and nothing of SIE's.

| Case | Tokens a document, in / out | List price per 1M tokens | Bill for 1M documents | Values exactly right | Values not exact |
| --- | --- | --- | ---: | ---: | ---: |
| Claude Sonnet 5.5 (thinking off) | 3,287.4 / 110.5 | $2.00 in, $10.00 out | $7,680 | 77.2% | 22% |
| GPT-6.1 Sol (medium reasoning) | 1,947.0 / 84.1 | $2.00 in, $10.00 out | $4,735 | 73.7% | 26% |

- The tokens are each model's own mean recorded tokens per document on the
  500 benchmark records, reasoning included, and the bill is those tokens at
  the list price: the figure the cost bar charts. The card prints the tokens
  to a tenth, and its own figures give its bill: 3,287.4 × $2.00 + 110.5 ×
  $10.00 = $7,679.80, and 1,947.0 × $2.00 + 84.1 × $10.00 = $4,735.00.
- Claude Sonnet 5.5 counts 3,287 input tokens for the documents GPT-6.1 Sol
  counts as 1,947: the two tokenizers differ, and Anthropic's structured
  outputs add a system prompt that is billed as input
  ([Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs),
  "Prompt modification and token costs").
- The heading, "Claude Sonnet 5.5 bills $7,680 per 1M documents", states the
  headline rival's bill at the cheapest setting it was measured at.
- "Not exact", in each card's header, is 100% minus the model's value
  accuracy, rounded down: 22.8% for Claude Sonnet 5.5 and 26.3% for GPT-6.1
  Sol. It is not in the heading, because every model has such a share: SIE
  Qwen3.8 27B's is 16.5%.
- The page says "exact" and "not exact", never "wrong". The study's metric
  counts a value right only when it matches the answer key character for
  character, and "Exact values, and what counts as right" above shows that,
  read leniently, the models are level. The search below found the same
  thing record by record.

### The rule for an example document, and its outcome

The section was to show, for each of the two rivals, one document from the
study where the rival returned a wrong value, the passage states the right
one and SIE returned it. The rule was written on 13 October 2026 before any
candidate was listed, and applied to the recorded replies only; no model was
called. It found no such document, so the section shows none.

The search ran when the page charted Claude Sonnet 5.5 at adaptive thinking,
so "charted setting" below means that reply and "first-run reply" means the
one with thinking `between_tools`, which the page charts now. It read both
replies of every candidate, and its outcome does not depend on which of the
two is charted: a value Claude Sonnet 5.5 misses with thinking off and has
with adaptive thinking is a weak-setting miss, which the page does not show.

**The rule as first written.** Two cases, one per provider, each at the
charted setting: Claude Sonnet 5.5, then GPT-6.1 Sol. Walk the 500 records in
the order of the registered sample. A record is a candidate when: (1) the
rival's reply is a complete record that parses and validates; (2) one
top-level field whose checked answer is a number or a string of at most 40
characters holds a different, non-empty value, and the difference is one of
fact: two different numbers, or two strings with no word in common after the
benchmark's own normalisation; (3) SIE's saved reply has the checked answer in
that field and its whole record scores 1.0; (4) every other recorded reply of
the same model is wrong in the same field (Claude Sonnet 5.5's first-run
reply; GPT-6.1 Sol's reply at `xhigh`, which must be on file); (5) the
checked answer stands verbatim in the passage inside one or two sentences of
at most 320 characters; (6) read against the whole passage and the question,
the passage states the checked answer with no second defensible reading and
nothing supports the rival's value. Take the first record that passes, never
the worst.

**What it found.** Nothing as written. Three amendments followed, each
written before the records it admitted were read:

1. Nested fields were admitted (not list items, whose order the benchmark
   does not fix). Five candidates for each rival, four shared. None passes the
   reading: "United Kingdom" for "British", "Democratic" for "Democrat",
   "United States" for "American", "1" for "single movement" and a role
   written out as a phrase are the same fact in other words; and for a
   debut year (2002 for 2003, record `d996d225`) the passage says the series
   "aired on Discovery Channel from 2003-2010" and, in another paragraph, that
   it "debuted in September 2002".
2. "No word in common" shut out a wrong date or name, so the string test
   became the lenient reading above: the strings differ after normalisation
   and neither contains the other. Ten candidates for Claude Sonnet 5.5 and 16
   for GPT-6.1 Sol. None passes: each is a longer or shorter form of the same
   name or phrase ("Wayne Philip Colin Sleep" for "Wayne Sleep", "Mario
   Gianluigi Puzo" for "Mario Puzo", "Eastern Slovakia" for "east Slovakia",
   "Devi Bhagavata Purana" for "Devi Bhagavata Mahapurana", a value padded
   into a sentence).
3. List items and booleans were admitted, and SIE's record needed only to
   score at least as high as the rival's. 28 candidates for Claude Sonnet 5.5
   and 51 for GPT-6.1 Sol. None passes: more wording ("Spring 1859",
   "Pennsylvania" for "PA", "feet" for "ft", "One Day International" for
   "ODI"); lists shifted by one item; the one record the study's own audit
   found wrong in the answer key (`decb5253`), where the rivals follow the
   question; and an award category the passage does not give (`80d61e5f`).

Then every remaining field where a rival's charted reply differs in a number,
a boolean or a string, and SIE has the checked answer, was read whatever
conditions 4 and 5 said:

- The count of titles with a doubles partner (`feef451c`): 14 from Claude
  Sonnet 5.5 at both settings, 1,666 and 0 from GPT-6.1 Sol, against a checked
  answer of 15. The passage says the 1996 US Open was "the 5th US Open, 15th
  Grand Slam and 66th overall title for Fernández": her own 15th, not 15 with
  that partner. The passage does not state the count the field asks for, and
  GPT-6.1 Sol answered 15 at `max` effort.
- Where GPT-6.1 Sol's default reply misses and SIE's does not (a film title
  with a letter dropped, a coach's other name, full legal names), its `xhigh`
  reply has the checked answer. A weak-setting miss is not shown.
- Condition 4 was then narrowed to its purpose: only a reply at a *stronger*
  setting than the charted one has to be wrong too. That admits Claude Sonnet
  5.5's misses at its charted, strongest setting (33 candidates). Those that
  are not plain wording are four names the passage itself gives: "Anett
  Futatabi" for "Annet Futatabi" (`f06d9947`), where the passage's own
  paragraph heading is spelled "Anett Futatabi"; "Systemax" for "Global
  Equipment Company" (`49d99c95`), where the passage says the company took
  that name in 1999; "Chip Esten" for "Charles Esten", "known professionally
  as Chip Esten and Charles Esten"; and a singer's birth name for her stage
  name.
- GPT-6 Luna, OpenAI's other charted model, was read the same way at its
  charted setting (47 candidates): wording, a list shifted by one, a birth
  name the passage gives, a release year the passage gives both ways.

So the run holds no record, for either provider, where a rival's value is
wrong in fact, SIE's is right and the passage states the right one. What it
holds, for Claude Sonnet 5.5 at adaptive thinking, is 240 fields where SIE's
value matches the answer key and Claude Sonnet 5.5's does not: 83 differ in capitals,
punctuation or an article, 93 are the answer with words added, 16 are the
answer with words dropped, 43 are the other string differences read above,
and 5 are the numbers and booleans above. In 151 of the 240 the answer key's value stands
in the passage word for word and Claude Sonnet 5.5's does not. That is the
gap the page's quality bars measure: exact values, the document's own words.
The page does not present any of these as a wrong fact.

## Recorded and not on the page

An earlier version of the pain section showed two things the run recorded.
Neither is on the page. Both are kept here.

### A schema keyword Claude does not support

NHTSA complaint 11761393, filed 1 September 2026, one of the 150 fresh
complaints, was sent with the page's complaint schema exactly as written. Its
narrative reports that the passenger sustained stomach bruises; the agency
coded one injury, and SIE Qwen3.8 27B returned `injured_people: 1`.

Claude Sonnet 5.5, at its default setting, returned no record. The API
answered with HTTP 400, `invalid_request_error`, and the message
`output_config.format.schema: For 'integer' type, property 'minimum' is not
supported`, recorded on 11 October 2026. The same refusal is recorded for
every one of the 150 complaints from Claude Sonnet 5.5 and Claude Haiku 5.5
(at both settings) in the current run, and from Claude Sonnet 5.5, Claude
Opus 5.5, Claude Haiku 4.5 and Claude Sonnet 5 in the first. Anthropic's
structured-outputs documentation lists numerical constraints such as
`minimum`, `maximum` and `multipleOf`, and string length constraints, as
unsupported
([Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs)).
Its SDK helpers strip such constraints from the schema Claude receives and
check them in your code afterwards; a raw request gets the 400. The recorded
refusals are in the public dataset's `calls/nhtsa/` folders (see Runnable
example), with request identifiers removed.

This is a JSON Schema keyword Claude's structured outputs do not support, a
missing feature and not a measure of answer quality. That every one of the
150 complaints got the same answer shows the behaviour is deterministic. The
page's API section sends the same schema, `"minimum": 0` included, to SIE
Qwen3.8 27B and shows the recorded reply; it says nothing about Claude.

### An unfinished reply from GPT-6.1 Sol

SOB record `80d61e5f3cb8…` (from HotpotQA; the passage is from Wikipedia's
article on Rami Malek). The question is "Who plays Elliot Alderson and was
born on May 12, 1981?", and the schema asks for each award won for the role
with its name and category. The passage says he "won a Critics' Choice Award
and the Primetime Emmy Award for Outstanding Lead Actor in a Drama Series".

GPT-6.1 Sol at its default setting, reasoning medium, the setting the chart
shows, was asked four times: its reply in the run and three repeats with the
identical request. Twice, including the run's own reply, it stopped right
after the Critics' Choice entry's `"award_category":` key and wrote only
whitespace until the 2,048-token allowance ran out, so the JSON object never
closed and there was no record to use. The other two replies closed. The
run's reply ends in 11,741 trailing whitespace characters. Across the 500
benchmark records the same arm stopped this way on 3 different documents,
this one included; each scores zero. SIE Qwen3.8 27B returned a complete,
schema-valid record for all 500.

This is what was observed: two of four saved replies to this request, and 3
of the 500 benchmark documents in the run. It supports no wider claim about
GPT-6.1 Sol's reliability.

The other two GPT-6.1 Sol replies left the Critics' Choice category as an
empty string. The passage attaches the category to the Emmy and gives none
for the Critics' Choice Award, so an empty value is a defensible reading. SIE
returned "Outstanding Lead Actor in a Drama Series" for both awards, which
matches the answer key, but the passage supports that only for the Emmy. At
effort `xhigh`, GPT-6.1 Sol returned the key text `"award_name       "` once
as the category and "Best Actor in a Drama Series", a category the passage
does not state, three times. Under SOB's scoring both SIE and GPT-6.1 Sol
score zero on this record: Sol's for the unfinished reply, SIE's because it
split "Egyptian-American" into two nationalities, which takes the record below
SOB's 95% path-match gate.

## The study shown

[The Structured Output Benchmark](https://arxiv.org/abs/2604.25359)
(arXiv:2604.25359, first version 28 April 2026; Singh, Khurdula, Khemlani and
Agarwal) is the benchmark this study used. It was read at the source on 13
October 2026. Its abstract reports 21 models evaluated and that "models
achieve near-perfect schema compliance, yet the best Value Accuracy, measured
by exact leaf-value match, reaches only 83.0% on text, 67.2% on images, and
23.7% on audio". The page's headline for it is "schema compliance is
near-perfect, value accuracy is not", and it prints none of those figures:
the 83.0% is the best model on the benchmark's full text set, and beside this
page's bars, drawn from 500 of its records, it would read as a score to
compare with them.

Also read that day, and not on the page: OpenAI's guide
[Structured model outputs](https://developers.openai.com/api/docs/guides/structured-outputs)
says, under "Handling mistakes", "Structured Outputs can still contain
mistakes. If you see mistakes, try adjusting your instructions, providing
examples in the system instructions, or splitting tasks into simpler
subtasks." The page's card for a documentation page cuts so long an address
short at phone width, so the page shows the study alone.

## Prices

Every cost on the page is dollars for 1,000,000 documents, from each model's
own mean recorded tokens per document on the 500 benchmark records, reasoning
included, so a model that writes or thinks longer, or a tokenizer that counts
more tokens for the same text, costs what it would cost you.

- **Every rival at its real-time list price** per 1M tokens, checked on 11
  October 2026 from [OpenAI's pricing page](https://developers.openai.com/api/docs/pricing)
  and [Anthropic's](https://platform.claude.com/docs/en/about-claude/pricing)
  and stated in the run's pre-registration: GPT-6.1 Sol $2.00 in / $10.00 out;
  GPT-6 Luna $0.10 / $0.50; Claude Haiku 5.5 $0.10 / $0.50; Claude Sonnet 5.5
  $2.00 / $10.00. No rival reported cached input tokens in this run. Both
  vendors also sell batch processing at half price for jobs that can wait up
  to a day; SIE has no batch rate, so the page compares real-time with
  real-time. At batch prices Claude Sonnet 5.5 costs $3,840, and ours would
  still be 81% less.
- **SIE's default-offer price** for `Qwen/Qwen3.8-27B-FP8`, read on 13 October
  2026 from the site's rate book `2026-10-13-production-bootstrap-v36`: $0.25
  per 1M input tokens, $2.75 per 1M output tokens and $0.025 per 1M cached
  input tokens, on SIE's recorded tokens from the 30 September run. The exact rate rows are
  `generate:Qwen/Qwen3.8-27B-FP8:default:input_tokens:us`,
  `generate:Qwen/Qwen3.8-27B-FP8:default:output_tokens:us` and
  `generate:Qwen/Qwen3.8-27B-FP8:default:cached_input_tokens:us`. Other serving
  profiles do not contribute to the page's cost calculation.

Against Claude Sonnet 5.5 that is $727 against $7,680 for 1,000,000
documents: 90% less (1 − 727 / 7,680 = 90.5%, rounded down), stated on the
page as that exact figure and in those words, "90% less", in the hero, the
proof heading, the adoption heading and the description. $7,680 is Claude
Sonnet 5.5's cheapest measured setting; against its other one ($7,749) the
saving is 90.6%, the same 90%. GPT-6 Luna ($227) and Claude Haiku 5.5 ($385)
cost less than SIE, and their values were significantly less often exactly
right; the page's two bar charts show both. Because two models cost less,
every statement of the saving names the rival it is measured against, as the
adoption heading does: "Extract your records for 90% less than Claude Sonnet
5.5".

The page's proof is two sorted bar charts, field values exactly right and cost
per 1M documents, one bar a model at the setting in Models and settings.
Its heading is "Most values exactly right, 90% less than Claude Sonnet 5.5":
SIE's bar is the longest of the five, and the saving is the hero's. Beside
the charts the page states three parameters of the study: what was tested
(500 documents from the public Structured Output Benchmark), how a value is
scored (exact match to the answer key, per field) and the prices (each
model's own recorded tokens at real-time list prices, 11 October 2026). Each
rival's setting, its other measured settings and the fresh records are in
this file only.

## Repeatability and latency

In the first run the first 100 records were sent to every model a second time,
one at a time from one client.

| Model | Same answer both times | Median time | 90th percentile |
| --- | --- | --- | --- |
| SIE Qwen3.8 27B | 94% | 7.1 s | 12.9 s |
| Claude Haiku 4.5 | 92% | 1.4 s | 1.9 s |
| GPT-6 Sol | 87% | 1.9 s | 2.7 s |
| GPT-6 Luna | 84% | 1.5 s | 2.1 s |
| Claude Opus 5.5 | 81% | 3.3 s | 5.3 s |
| Claude Sonnet 5.5 | 78% | 1.8 s | 2.8 s |
| Claude Sonnet 5 | 71% | 2.3 s | 3.1 s |

The current run timed each provider at 8 requests in flight from one laptop:
median 1.6 to 2.7 seconds per request. SIE's structured output is slower, and
the page makes no speed claim. These response times describe the recorded
profile and workload, independently of later serving configurations.

## Runnable example

The page's reproduction links open
<https://github.com/superlinked/sie/tree/96b7f9b63d9035853851cc901fc2e146e2ad62ba/examples/structured-output-accuracy>.
It holds the runner, the scorer (SOB's own, fetched at a pinned commit) and a
fetcher for the recorded responses of every model in the first run, which are
in the public dataset [superlinked/sie-task-evidence](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/0e1fb325cf25bd403ff678dd6031f649981eb434/structured-output-accuracy),
folder `structured-output-accuracy/` at revision
`0e1fb325cf25bd403ff678dd6031f649981eb434`. `python3 fetch.py` then
`uv run score.py` re-derives every first-run figure with no API key.

The current run is in the same dataset, folder
[`structured-output-current-arms-2026-10-11/`](https://huggingface.co/datasets/superlinked/sie-task-evidence/tree/0e3ec95fa2d796e4cce5770338c180f38d7e57a1/structured-output-current-arms-2026-10-11)
at revision `0e3ec95fa2d796e4cce5770338c180f38d7e57a1`: its pre-registration
(first published at revision `6d17541e3ff3d462d220994463d2898408734cc9`, with
the same bytes), runner, scorer, every raw reply, the reports and the pain
search, including the four GPT-6.1 Sol replies above (`calls/e1/gpt-6.1-sol.jsonl`
and `pain/repro.jsonl`). Its scorer reproduces every first-run arm to 12
decimal places.

## The switch

Each tab of "Switch to SIE in 5 minutes" starts from the call the study made
to that vendor and ends on the call the study made to SIE, the one the API
section shows. Every side sends the reader's own `schema`, `instruction` and
`document`, and the same system message: the instruction, then "Required JSON
schema:" and the schema, beside the schema the API itself is held to.

- Anthropic: `client.messages.create` with `output_config` holding the JSON
  Schema, and `max_tokens=2048`.
- OpenAI: `client.chat.completions.create` with a strict `response_format`,
  and `max_completion_tokens=2048`.
- SIE: `SIEClient` and `chat_completions` with the same strict
  `response_format`, `max_completion_tokens=2048` and temperature 0. From the
  call's first line to its closing parenthesis it is the API section's code.

The study's runner streamed each reply to capture timing; the page's calls
are the non-streaming form of the same requests. The reasoning or thinking
setting each rival ran at is in Models and settings and is not in the
snippet. SIE also serves an OpenAI-compatible endpoint, which the page does
not show.

## Limits

- SOB's records are general-knowledge passages, not business documents; the
  NHTSA complaints are the page's check on the reader's own kind of document.
- The recorded default Qwen3.8 27B run used an 8,192-token total-context
  profile (prompt, schema and answer together); every recorded case fit.
  These quality outputs and default-offer workload costs are not measurements
  or prices for a different context profile.
- One run per setting. The second run's arms were recorded eleven days after
  SIE's answers; the inputs, prompt, schema, allowance and scorer are
  identical.