Read app screenshots for 69% less than GPT-6.1 Sol, at 97% of its field accuracy
Give your agent typed records from dashboards and admin tools, without building an integration for each app. SIE serves an open vision model with your schema.
Gemma 4 31B
Typed app record
- title
- Approve API access matrix string
- status
- open string
- priority
- 1 number
- category
- Documentation string
Ask for the complete record in one schema
{
"assignee": {"type":["string","null"]},
"due_date": {"type":["string","null"]},
"subtasks": {"type":"array","items":{"type":"object","properties":{"title":{"type":["string","null"]},"assignee":{"type":["string","null"]},"time_tracking":{"type":["string","null"]}},"required":["title","assignee","time_tracking"],"additionalProperties":false}}
}Read the complete strict schema| Model | Answered |
|---|---|
| Gemma 4 31B | 100/100 |
| Claude Sonnet 5.5 | 78/100 |
| Claude Haiku 5.5 | 78/100 |
Claude Sonnet 5.5 and Claude Haiku 5.5 each returned answers for 78 of the 100 requests. Separately, 22 of the original schemas exceed Claude’s documented strict-mode union limit of 16.
97% of Sol’s field accuracy for 69% less
| Model | Mean field agreement | USD per 1,000 answered screenshots |
|---|---|---|
| Gemma 4 31B (100/100) | 94.147% | $1.8732 |
| GPT-6.1 Sol, medium (100/100) | 96.197% | $6.0702 |
| Claude Sonnet 5.5 (78/100) | 95.671% | $7.5357 |
| Claude Haiku 5.5 (78/100) | 95.392% | $0.3817 |
| GPT-6 Luna (15/100) | 84.602% | $0.7499 |
20 screens per app. Mean per-screen normalized field agreement on answered requests. Each model’s accuracy and cost cover the requests it answered.
- Fully matching
- 63/100 Gemma · 70/100 Sol
- Prices
- Gemma 4 31B at its Serverless rate; each API at list price on its recorded tokens
- Recorded
- October 12, 2026
Gemma 4 31B on Serverless: $1.8732 per 1,000 screenshots on the 100-screen study. $0.60 input / $2.00 output per million tokens.
Send the screenshot and the complete schema
The original PNG, unchanged instruction and complete strict schema.
generate = {
"messages": [
{"role": "system", "content": original_instruction},
{"role": "user", "content": "", "images": [
{"data": original_png_base64, "format": "png"},
]},
],
"grammar": {"kind": "json_schema", "value": complete_schema,
"label": "fixture_kb01", "strict": True},
"max_new_tokens": 32768,
"temperature": 0, "presence_penalty": 0,
}Download the complete recorded recipeView complete request
The complete download includes the original PNG bytes. This preview abbreviates only those bytes.
{
"messages": [
{
"role": "system",
"content": "Read only the supplied full screenshot and return one JSON object conforming to the schema. Do not use external knowledge or invent hidden content. Copy strings as visibly displayed, preserving spelling, case, diacritics, signs, currency symbols and punctuation. Values that are absent, blank, only placeholders or not fully readable are null. Checkbox states are booleans; counts are integers; money and dates are strings. Include every complete requested row in visible order. Ignore navigation and fields not requested. Do not output Markdown, commentary or reasoning.\n\nExtract the displayed task reference, title, labelled summary fields, dates and description. Copy the main task values only; exclude action links such as Assign to me. Use null for a requested time-spent value when that line is absent. Include every complete subtask row in its top-to-bottom order, with its displayed time-tracking string.\n"
},
{
"role": "user",
"content": "",
"images": [
{
"data": "[original PNG base64, complete bytes in download]",
"format": "png"
}
]
}
],
"max_new_tokens": 32768,
"temperature": 0,
"presence_penalty": 0,
"grammar": {
"kind": "json_schema",
"value": {
"type": "object",
"properties": {
"task_reference": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
},
"status": {
"type": [
"string",
"null"
]
},
"priority": {
"type": [
"integer",
"null"
]
},
"complexity": {
"type": [
"integer",
"null"
]
},
"category": {
"type": [
"string",
"null"
]
},
"swimlane": {
"type": [
"string",
"null"
]
},
"column": {
"type": [
"string",
"null"
]
},
"position": {
"type": [
"integer",
"null"
]
},
"assignee": {
"type": [
"string",
"null"
]
},
"creator": {
"type": [
"string",
"null"
]
},
"time_estimated": {
"type": [
"string",
"null"
]
},
"time_spent": {
"type": [
"string",
"null"
]
},
"due_date": {
"type": [
"string",
"null"
]
},
"created": {
"type": [
"string",
"null"
]
},
"modified": {
"type": [
"string",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
},
"subtasks": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {
"type": [
"string",
"null"
]
},
"assignee": {
"type": [
"string",
"null"
]
},
"time_tracking": {
"type": [
"string",
"null"
]
}
},
"required": [
"title",
"assignee",
"time_tracking"
],
"additionalProperties": false
}
}
},
"required": [
"task_reference",
"title",
"status",
"priority",
"complexity",
"category",
"swimlane",
"column",
"position",
"assignee",
"creator",
"time_estimated",
"time_spent",
"due_date",
"created",
"modified",
"description",
"subtasks"
],
"additionalProperties": false
},
"label": "fixture_kb01",
"strict": true
}
}- task_reference
- Task #1
- title
- Approve API access matrix
- status
- open
- priority
- 1
- complexity
- 5
- category
- Documentation
- swimlane
- Default swimlane
- column
- Backlog
- position
- 1
- assignee
- Inés Ramos
- creator
- Fixture Administrator
- time_estimated
- 7.5 hours
- time_spent
- 1.5 hours
- due_date
- 2026-11-04 00:00
- created
- 2026-10-10 09:00
- modified
- 2026-10-10 09:00
- description
- Check role matrix AX-09 against request CB-207. Approval ceiling: €2,750.00.
| Title | Assignee | Time tracking |
|---|---|---|
| Collect source checklist | Inés Ramos | 1.50h estimated |
| Review literal identifiers | Liam Chen | 0.50h spent / 2.50h estimated |
| Record final sign-off | Zofia Nowak | 1.00h spent / 3.50h estimated |
Full JSON output
{"task_reference": "Task #1", "title": "Approve API access matrix", "status": "open", "priority": 1, "complexity": 5, "category": "Documentation", "swimlane": "Default swimlane", "column": "Backlog", "position": 1, "assignee": "Inés Ramos", "creator": "Fixture Administrator", "time_estimated": "7.5 hours", "time_spent": "1.5 hours", "due_date": "2026-11-04 00:00", "created": "2026-10-10 09:00", "modified": "2026-10-10 09:00", "description": "Check role matrix AX-09 against request CB-207. Approval ceiling: \u20ac2,750.00.", "subtasks": [{"title": "Collect source checklist", "assignee": "In\u00e9s Ramos", "time_tracking": "1.50h estimated"}, {"title": "Review literal identifiers", "assignee": "Liam Chen", "time_tracking": "0.50h spent / 2.50h estimated"}, {"title": "Record final sign-off", "assignee": "Zofia Nowak", "time_tracking": "1.00h spent / 3.50h estimated"}]}SDK example. Use the complete schema with your own screenshot. Save the original PNG as image.png for the SDK code.
result = client.chat_completions(
"google/gemma-4-31B-it",
messages,
max_completion_tokens=32768,
extra_body={"temperature":0,"presence_penalty":0},
response_format={
"type": "json_schema",
"json_schema": {"name": "fixture_kb01", "strict": True, "schema": schema},
},
)Complete SDK examples
from pathlib import Pathimport base64import jsonfrom sie_sdk import SIEClientclient = SIEClient(api_key="API keysk-sie-…")schema = json.loads("""schema{ "type": "object", "properties": {···}, ··· "additionalProperties": false}""")instruction = "instruction···Read only the supplied full screenshot and return one JSON object conforming to the schema.···"prompt = f"{instruction}\n\nJSON schema:\n{json.dumps(schema, indent=2, ensure_ascii=False)}"image = base64.b64encode(Path(imageimage.pngbrowse).read_bytes()).decode("ascii")messages = [ {"role": "system", "content": prompt}, { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}}, ], },]result = client.chat_completions( "modelgoogle/gemma-4-31B-it", messages, max_completion_tokens=32768, extra_body={"temperature":0,"presence_penalty":0}, response_format={ "type": "json_schema", "json_schema": {"name": "fixture_kb01", "strict": True, "schema": schema}, },)choices = result["choices"]content = choices[0]["message"]["content"] if choices else Noneif not isinstance(content, str): raise ValueError("The model returned no JSON content.")print(json.loads(content))import { readFile } from 'node:fs/promises';
import { SIEClient } from '@superlinked/sie-sdk';
const client = new SIEClient(undefined, { apiKey: 'sk-sie-…' });
const schema = {
"type": "object",
"properties": {···},
···
"additionalProperties": false
};
const instruction = "···Read only the supplied full screenshot and return one JSON object conforming to the schema.···";
const prompt = `${instruction}\n\nJSON schema:\n${JSON.stringify(schema, null, 2)}`;
const image = await readFile("image.png");
const result = await client.chatCompletions({
model: 'google/gemma-4-31B-it',
messages: [
{ role: 'system', content: prompt },
{
role: 'user',
content: [
{ type: 'image_url', image_url: { url: `data:image/png;base64,${Buffer.from(image).toString('base64')}` } },
],
},
],
max_completion_tokens: 32768,
temperature: 0,
presence_penalty: 0,
response_format: {
type: 'json_schema',
json_schema: { name: 'fixture_kb01', strict: true, schema },
},
});
const content = result.choices[0]?.message.content;
if (typeof content !== 'string') throw new Error('The model returned no JSON content.');
console.log(JSON.parse(content));document_image=$(base64 < 'image.png' | tr -d '\n')
curl https://api.superlinked.com/v1/chat/completions \
-H "Authorization: Bearer sk-sie-…" \
-H "Content-Type: application/json" \
-d "{\"model\":\"google/gemma-4-31B-it\",\"messages\":[{\"role\":\"system\",\"content\":\"···Read only the supplied full screenshot and return one JSON object conforming to the schema.···\\n\\nJSON schema:\\n{\\n \\\"type\\\": \\\"object\\\",\\n \\\"properties\\\": {···},\\n ···\\n \\\"additionalProperties\\\": false\\n}\"},{\"role\":\"user\",\"content\":[{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,$document_image\"}}]}],\"max_completion_tokens\":32768,\"temperature\":0,\"presence_penalty\":0,\"response_format\":{\"type\":\"json_schema\",\"json_schema\":{\"name\":\"fixture_kb01\",\"strict\":true,\"schema\":{\"type\":\"object\",\"properties\":{\"task_reference\":{\"type\":[\"string\",\"null\"]},\"title\":{\"type\":[\"string\",\"null\"]},\"status\":{\"type\":[\"string\",\"null\"]},\"priority\":{\"type\":[\"integer\",\"null\"]},\"complexity\":{\"type\":[\"integer\",\"null\"]},\"category\":{\"type\":[\"string\",\"null\"]},\"swimlane\":{\"type\":[\"string\",\"null\"]},\"column\":{\"type\":[\"string\",\"null\"]},\"position\":{\"type\":[\"integer\",\"null\"]},\"assignee\":{\"type\":[\"string\",\"null\"]},\"creator\":{\"type\":[\"string\",\"null\"]},\"time_estimated\":{\"type\":[\"string\",\"null\"]},\"time_spent\":{\"type\":[\"string\",\"null\"]},\"due_date\":{\"type\":[\"string\",\"null\"]},\"created\":{\"type\":[\"string\",\"null\"]},\"modified\":{\"type\":[\"string\",\"null\"]},\"description\":{\"type\":[\"string\",\"null\"]},\"subtasks\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"title\":{\"type\":[\"string\",\"null\"]},\"assignee\":{\"type\":[\"string\",\"null\"]},\"time_tracking\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"title\",\"assignee\",\"time_tracking\"],\"additionalProperties\":false}}},\"required\":[\"task_reference\",\"title\",\"status\",\"priority\",\"complexity\",\"category\",\"swimlane\",\"column\",\"position\",\"assignee\",\"creator\",\"time_estimated\",\"time_spent\",\"due_date\",\"created\",\"modified\",\"description\",\"subtasks\"],\"additionalProperties\":false}}}}"Build the "Screenshot mining" capability into my app using the Superlinked Inference Engine (SIE).
Context
- SIE is an OpenAI-style inference API. Python SDK: `from sie_sdk import SIEClient`; TypeScript: `@superlinked/sie-sdk`.
- Base URL: https://api.superlinked.com (or my regional endpoint). Auth: Bearer key from env `SIE_API_KEY` (never hard-code it).
- Model: google/gemma-4-31B-it (OpenAI-compatible endpoint: /v1/chat/completions). Keep the model id configurable.
Task
- Input: an uploaded app screenshot plus the fields to extract.
- Behaviour: return every field in the submitted schema as validated JSON
- Send one POST /v1/chat/completions request (SDK: chat_completions / chatCompletions) with a system message and a user message. The system message is "Read only the supplied full screenshot and return one JSON object conforming to the schema. Do not use external knowledge or invent hidden content. Copy strings as visibly displayed, preserving spelling, case, diacritics, signs, currency symbols and punctuation. Values that are absent, blank, only placeholders or not fully readable are null. Checkbox states are booleans; counts are integers; money and dates are strings. Include every complete requested row in visible order. Ignore navigation and fields not requested. Do not output Markdown, commentary or reasoning.\n\nExtract the displayed task reference, title, labelled summary fields, dates and description. Copy the main task values only; exclude action links such as Assign to me. Use null for a requested time-spent value when that line is absent. Include every complete subtask row in its top-to-bottom order, with its displayed time-tracking string.\n", a blank line, "JSON schema:" and the submitted schema as JSON indented by two spaces; response_format only constrains decoding, so the model reads the field descriptions from this text. The user message holds only an image_url part. Base64-encode the original local image bytes as a canonical data:image/png, data:image/jpeg or data:image/webp URI (normalize jpg to image/jpeg). Use max_completion_tokens 32768, temperature 0 and response_format {type: "json_schema", json_schema: {name: "fixture_kb01", strict: true, schema}} with the submitted schema unchanged. Parse choices[0].message.content as JSON and fail loudly when it is missing or invalid. Submitted schema: {
"type": "object",
"properties": {
"task_reference": {
"type": [
"string",
"null"
]
},
"title": {
"type": [
"string",
"null"
]
},
"status": {
"type": [
"string",
"null"
]
},
"priority": {
"type": [
"integer",
"null"
]
},
"complexity": {
"type": [
"integer",
"null"
]
},
"category": {
"type": [
"string",
"null"
]
},
"swimlane": {
"type": [
"string",
"null"
]
},
"column": {
"type": [
"string",
"null"
]
},
"position": {
"type": [
"integer",
"null"
]
},
"assignee": {
"type": [
"string",
"null"
]
},
"creator": {
"type": [
"string",
"null"
]
},
"time_estimated": {
"type": [
"string",
"null"
]
},
"time_spent": {
"type": [
"string",
"null"
]
},
"due_date": {
"type": [
"string",
"null"
]
},
"created": {
"type": [
"string",
"null"
]
},
"modified": {
"type": [
"string",
"null"
]
},
"description": {
"type": [
"string",
"null"
]
},
"subtasks": {
"type": "array",
"items": {
"type": "object",
"properties": {
"title": {
"type": [
"string",
"null"
]
},
"assignee": {
"type": [
"string",
"null"
]
},
"time_tracking": {
"type": [
"string",
"null"
]
}
},
"required": [
"title",
"assignee",
"time_tracking"
],
"additionalProperties": false
}
}
},
"required": [
"task_reference",
"title",
"status",
"priority",
"complexity",
"category",
"swimlane",
"column",
"position",
"assignee",
"creator",
"time_estimated",
"time_spent",
"due_date",
"created",
"modified",
"description",
"subtasks"
],
"additionalProperties": false
}. Preserve top_k when supplied, presence_penalty 0 and the submitted whitespace setting.
Deliverables
- A typed client wrapper, an application-level function for this task, error handling for timeouts/empty input, and unit tests with a stubbed client.
- Wire it into my existing stack (ask me which framework if unclear) and add a short usage example.- task_reference
- string | null
- title
- string | null
- status
- string | null
- priority
- integer | null
- complexity
- integer | null
- category
- string | null
- swimlane
- string | null
- column
- string | null
- position
- integer | null
- assignee
- string | null
- creator
- string | null
- time_estimated
- string | null
- time_spent
- string | null
- due_date
- string | null
- created
- string | null
- modified
- string | null
- description
- string | null
- subtasks
- array
Schema illustration. The saved output above belongs to the recorded request.
Read the screens your agent already uses
Serverless API
We host the models; you call one API.
- No idle GPUs, pay for what you use
- SOC 2 Type 2, US or EU data residency
- $50 of credit to start, no card required
Self-hosted (Kubernetes)
SIE in your clusters, backed by our engineers.
- Data stays in your own cloud account
- Same models and API as Serverless
- Terraform to your cloud in minutes
Run locally
The open-source server on your own machine.
docker run --gpus all -p 8080:8080 ghcr.io/superlinked/sie-server:latest-cuda12-default