Why did we open-source our inference engine? Read the post

Read app screenshots for 69% less than GPT-6.1 Sol, at 97% of its field accuracy

Give your agent typed records from dashboards and admin tools, without building an integration for each app. SIE serves an open vision model with your schema.

Gemma 4 31B

Typed app record
Kanboard task detail: Approve API access matrix, status open, priority 1 and category Documentation
Kanboard · task detail
Recorded JSON fields
title
Approve API access matrix string
status
open string
priority
1 number
category
Documentation string

Ask for the complete record in one schema

Original nullable schema · excerpt
{
  "assignee": {"type":["string","null"]},
  "due_date": {"type":["string","null"]},
  "subtasks": {"type":"array","items":{"type":"object","properties":{"title":{"type":["string","null"]},"assignee":{"type":["string","null"]},"time_tracking":{"type":["string","null"]}},"required":["title","assignee","time_tracking"],"additionalProperties":false}}
}
Read the complete strict schema
ModelAnswered
Gemma 4 31B100/100
Claude Sonnet 5.578/100
Claude Haiku 5.578/100

Claude Sonnet 5.5 and Claude Haiku 5.5 each returned answers for 78 of the 100 requests. Separately, 22 of the original schemas exceed Claude’s documented strict-mode union limit of 16.

97% of Sol’s field accuracy for 69% less

Field accuracy and cost
Answered requests
Mean field agreement ↑ USD per 1,000 answered screenshots →
Mean per-screen field agreement against cost per 1,000 answered screenshots, with the number of requests each model answered
ModelMean field agreementUSD per 1,000 answered screenshots
Gemma 4 31B (100/100)94.147%$1.8732
GPT-6.1 Sol, medium (100/100)96.197%$6.0702
Claude Sonnet 5.5 (78/100)95.671%$7.5357
Claude Haiku 5.5 (78/100)95.392%$0.3817
GPT-6 Luna (15/100)84.602%$0.7499
100 fictional screens from five apps

20 screens per app. Mean per-screen normalized field agreement on answered requests. Each model’s accuracy and cost cover the requests it answered.

Fully matching
63/100 Gemma · 70/100 Sol
Prices
Gemma 4 31B at its Serverless rate; each API at list price on its recorded tokens
Recorded
October 12, 2026

Gemma 4 31B on Serverless: $1.8732 per 1,000 screenshots on the 100-screen study. $0.60 input / $2.00 output per million tokens.

Send the screenshot and the complete schema

Recorded request · excerpt

The original PNG, unchanged instruction and complete strict schema.

generate = {
    "messages": [
        {"role": "system", "content": original_instruction},
        {"role": "user", "content": "", "images": [
            {"data": original_png_base64, "format": "png"},
        ]},
    ],
    "grammar": {"kind": "json_schema", "value": complete_schema,
        "label": "fixture_kb01", "strict": True},
    "max_new_tokens": 32768,
    "temperature": 0, "presence_penalty": 0,
}
Download the complete recorded recipe
View complete request

The complete download includes the original PNG bytes. This preview abbreviates only those bytes.

{
  "messages": [
    {
      "role": "system",
      "content": "Read only the supplied full screenshot and return one JSON object conforming to the schema. Do not use external knowledge or invent hidden content. Copy strings as visibly displayed, preserving spelling, case, diacritics, signs, currency symbols and punctuation. Values that are absent, blank, only placeholders or not fully readable are null. Checkbox states are booleans; counts are integers; money and dates are strings. Include every complete requested row in visible order. Ignore navigation and fields not requested. Do not output Markdown, commentary or reasoning.\n\nExtract the displayed task reference, title, labelled summary fields, dates and description. Copy the main task values only; exclude action links such as Assign to me. Use null for a requested time-spent value when that line is absent. Include every complete subtask row in its top-to-bottom order, with its displayed time-tracking string.\n"
    },
    {
      "role": "user",
      "content": "",
      "images": [
        {
          "data": "[original PNG base64, complete bytes in download]",
          "format": "png"
        }
      ]
    }
  ],
  "max_new_tokens": 32768,
  "temperature": 0,
  "presence_penalty": 0,
  "grammar": {
    "kind": "json_schema",
    "value": {
      "type": "object",
      "properties": {
        "task_reference": {
          "type": [
            "string",
            "null"
          ]
        },
        "title": {
          "type": [
            "string",
            "null"
          ]
        },
        "status": {
          "type": [
            "string",
            "null"
          ]
        },
        "priority": {
          "type": [
            "integer",
            "null"
          ]
        },
        "complexity": {
          "type": [
            "integer",
            "null"
          ]
        },
        "category": {
          "type": [
            "string",
            "null"
          ]
        },
        "swimlane": {
          "type": [
            "string",
            "null"
          ]
        },
        "column": {
          "type": [
            "string",
            "null"
          ]
        },
        "position": {
          "type": [
            "integer",
            "null"
          ]
        },
        "assignee": {
          "type": [
            "string",
            "null"
          ]
        },
        "creator": {
          "type": [
            "string",
            "null"
          ]
        },
        "time_estimated": {
          "type": [
            "string",
            "null"
          ]
        },
        "time_spent": {
          "type": [
            "string",
            "null"
          ]
        },
        "due_date": {
          "type": [
            "string",
            "null"
          ]
        },
        "created": {
          "type": [
            "string",
            "null"
          ]
        },
        "modified": {
          "type": [
            "string",
            "null"
          ]
        },
        "description": {
          "type": [
            "string",
            "null"
          ]
        },
        "subtasks": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "title": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "assignee": {
                "type": [
                  "string",
                  "null"
                ]
              },
              "time_tracking": {
                "type": [
                  "string",
                  "null"
                ]
              }
            },
            "required": [
              "title",
              "assignee",
              "time_tracking"
            ],
            "additionalProperties": false
          }
        }
      },
      "required": [
        "task_reference",
        "title",
        "status",
        "priority",
        "complexity",
        "category",
        "swimlane",
        "column",
        "position",
        "assignee",
        "creator",
        "time_estimated",
        "time_spent",
        "due_date",
        "created",
        "modified",
        "description",
        "subtasks"
      ],
      "additionalProperties": false
    },
    "label": "fixture_kb01",
    "strict": true
  }
}
Recorded Gemma output
task_reference
Task #1
title
Approve API access matrix
status
open
priority
1
complexity
5
category
Documentation
swimlane
Default swimlane
column
Backlog
position
1
assignee
Inés Ramos
creator
Fixture Administrator
time_estimated
7.5 hours
time_spent
1.5 hours
due_date
2026-11-04 00:00
created
2026-10-10 09:00
modified
2026-10-10 09:00
description
Check role matrix AX-09 against request CB-207. Approval ceiling: €2,750.00.
subtasks · ordered records
TitleAssigneeTime tracking
Collect source checklistInés Ramos1.50h estimated
Review literal identifiersLiam Chen0.50h spent / 2.50h estimated
Record final sign-offZofia Nowak1.00h spent / 3.50h estimated
Full JSON output
{"task_reference": "Task #1", "title": "Approve API access matrix", "status": "open", "priority": 1, "complexity": 5, "category": "Documentation", "swimlane": "Default swimlane", "column": "Backlog", "position": 1, "assignee": "Inés Ramos", "creator": "Fixture Administrator", "time_estimated": "7.5 hours", "time_spent": "1.5 hours", "due_date": "2026-11-04 00:00", "created": "2026-10-10 09:00", "modified": "2026-10-10 09:00", "description": "Check role matrix AX-09 against request CB-207. Approval ceiling: \u20ac2,750.00.", "subtasks": [{"title": "Collect source checklist", "assignee": "In\u00e9s Ramos", "time_tracking": "1.50h estimated"}, {"title": "Review literal identifiers", "assignee": "Liam Chen", "time_tracking": "0.50h spent / 2.50h estimated"}, {"title": "Record final sign-off", "assignee": "Zofia Nowak", "time_tracking": "1.00h spent / 3.50h estimated"}]}

SDK example. Use the complete schema with your own screenshot. Save the original PNG as image.png for the SDK code.

SDK call · example excerpt
result = client.chat_completions(
    "google/gemma-4-31B-it",
    messages,
    max_completion_tokens=32768,
    extra_body={"temperature":0,"presence_penalty":0},
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "fixture_kb01", "strict": True, "schema": schema},
    },
)
Complete SDK examples
from pathlib import Path
import base64
import json
from sie_sdk import SIEClient
​
client = SIEClient(api_key="API keysk-sie-…")
​
schema = json.loads("""
schema{
"type": "object",
"properties": {···},
···
"additionalProperties": false
}
""")
instruction = "instruction···Read only the supplied full screenshot and return one JSON object conforming to the schema.···"
prompt = f"{instruction}\n\nJSON schema:\n{json.dumps(schema, indent=2, ensure_ascii=False)}"
image = base64.b64encode(Path(imageimage.pngbrowse).read_bytes()).decode("ascii")
messages = [
{"role": "system", "content": prompt},
{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
],
},
]
result = client.chat_completions(
"modelgoogle/gemma-4-31B-it",
messages,
max_completion_tokens=32768,
extra_body={"temperature":0,"presence_penalty":0},
response_format={
"type": "json_schema",
"json_schema": {"name": "fixture_kb01", "strict": True, "schema": schema},
},
)
choices = result["choices"]
content = choices[0]["message"]["content"] if choices else None
if not isinstance(content, str):
raise ValueError("The model returned no JSON content.")
print(json.loads(content))
import { readFile } from 'node:fs/promises';
import { SIEClient } from '@superlinked/sie-sdk';

const client = new SIEClient(undefined, { apiKey: 'sk-sie-…' });

const schema = {
  "type": "object",
  "properties": {···},
  ···
  "additionalProperties": false
};
const instruction = "···Read only the supplied full screenshot and return one JSON object conforming to the schema.···";
const prompt = `${instruction}\n\nJSON schema:\n${JSON.stringify(schema, null, 2)}`;
const image = await readFile("image.png");
const result = await client.chatCompletions({
  model: 'google/gemma-4-31B-it',
  messages: [
    { role: 'system', content: prompt },
    {
      role: 'user',
      content: [
        { type: 'image_url', image_url: { url: `data:image/png;base64,${Buffer.from(image).toString('base64')}` } },
      ],
    },
  ],
  max_completion_tokens: 32768,
  temperature: 0,
  presence_penalty: 0,
  response_format: {
    type: 'json_schema',
    json_schema: { name: 'fixture_kb01', strict: true, schema },
  },
});
const content = result.choices[0]?.message.content;
if (typeof content !== 'string') throw new Error('The model returned no JSON content.');
console.log(JSON.parse(content));
document_image=$(base64 < 'image.png' | tr -d '\n')
curl https://api.superlinked.com/v1/chat/completions \
  -H "Authorization: Bearer sk-sie-…" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"google/gemma-4-31B-it\",\"messages\":[{\"role\":\"system\",\"content\":\"···Read only the supplied full screenshot and return one JSON object conforming to the schema.···\\n\\nJSON schema:\\n{\\n  \\\"type\\\": \\\"object\\\",\\n  \\\"properties\\\": {···},\\n  ···\\n  \\\"additionalProperties\\\": false\\n}\"},{\"role\":\"user\",\"content\":[{\"type\":\"image_url\",\"image_url\":{\"url\":\"data:image/png;base64,$document_image\"}}]}],\"max_completion_tokens\":32768,\"temperature\":0,\"presence_penalty\":0,\"response_format\":{\"type\":\"json_schema\",\"json_schema\":{\"name\":\"fixture_kb01\",\"strict\":true,\"schema\":{\"type\":\"object\",\"properties\":{\"task_reference\":{\"type\":[\"string\",\"null\"]},\"title\":{\"type\":[\"string\",\"null\"]},\"status\":{\"type\":[\"string\",\"null\"]},\"priority\":{\"type\":[\"integer\",\"null\"]},\"complexity\":{\"type\":[\"integer\",\"null\"]},\"category\":{\"type\":[\"string\",\"null\"]},\"swimlane\":{\"type\":[\"string\",\"null\"]},\"column\":{\"type\":[\"string\",\"null\"]},\"position\":{\"type\":[\"integer\",\"null\"]},\"assignee\":{\"type\":[\"string\",\"null\"]},\"creator\":{\"type\":[\"string\",\"null\"]},\"time_estimated\":{\"type\":[\"string\",\"null\"]},\"time_spent\":{\"type\":[\"string\",\"null\"]},\"due_date\":{\"type\":[\"string\",\"null\"]},\"created\":{\"type\":[\"string\",\"null\"]},\"modified\":{\"type\":[\"string\",\"null\"]},\"description\":{\"type\":[\"string\",\"null\"]},\"subtasks\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"title\":{\"type\":[\"string\",\"null\"]},\"assignee\":{\"type\":[\"string\",\"null\"]},\"time_tracking\":{\"type\":[\"string\",\"null\"]}},\"required\":[\"title\",\"assignee\",\"time_tracking\"],\"additionalProperties\":false}}},\"required\":[\"task_reference\",\"title\",\"status\",\"priority\",\"complexity\",\"category\",\"swimlane\",\"column\",\"position\",\"assignee\",\"creator\",\"time_estimated\",\"time_spent\",\"due_date\",\"created\",\"modified\",\"description\",\"subtasks\"],\"additionalProperties\":false}}}}"
Build the "Screenshot mining" capability into my app using the Superlinked Inference Engine (SIE).

Context
- SIE is an OpenAI-style inference API. Python SDK: `from sie_sdk import SIEClient`; TypeScript: `@superlinked/sie-sdk`.
- Base URL: https://api.superlinked.com (or my regional endpoint). Auth: Bearer key from env `SIE_API_KEY` (never hard-code it).
- Model: google/gemma-4-31B-it (OpenAI-compatible endpoint: /v1/chat/completions). Keep the model id configurable.

Task
- Input: an uploaded app screenshot plus the fields to extract.
- Behaviour: return every field in the submitted schema as validated JSON
- Send one POST /v1/chat/completions request (SDK: chat_completions / chatCompletions) with a system message and a user message. The system message is "Read only the supplied full screenshot and return one JSON object conforming to the schema. Do not use external knowledge or invent hidden content. Copy strings as visibly displayed, preserving spelling, case, diacritics, signs, currency symbols and punctuation. Values that are absent, blank, only placeholders or not fully readable are null. Checkbox states are booleans; counts are integers; money and dates are strings. Include every complete requested row in visible order. Ignore navigation and fields not requested. Do not output Markdown, commentary or reasoning.\n\nExtract the displayed task reference, title, labelled summary fields, dates and description. Copy the main task values only; exclude action links such as Assign to me. Use null for a requested time-spent value when that line is absent. Include every complete subtask row in its top-to-bottom order, with its displayed time-tracking string.\n", a blank line, "JSON schema:" and the submitted schema as JSON indented by two spaces; response_format only constrains decoding, so the model reads the field descriptions from this text. The user message holds only an image_url part. Base64-encode the original local image bytes as a canonical data:image/png, data:image/jpeg or data:image/webp URI (normalize jpg to image/jpeg). Use max_completion_tokens 32768, temperature 0 and response_format {type: "json_schema", json_schema: {name: "fixture_kb01", strict: true, schema}} with the submitted schema unchanged. Parse choices[0].message.content as JSON and fail loudly when it is missing or invalid. Submitted schema: {
  "type": "object",
  "properties": {
    "task_reference": {
      "type": [
        "string",
        "null"
      ]
    },
    "title": {
      "type": [
        "string",
        "null"
      ]
    },
    "status": {
      "type": [
        "string",
        "null"
      ]
    },
    "priority": {
      "type": [
        "integer",
        "null"
      ]
    },
    "complexity": {
      "type": [
        "integer",
        "null"
      ]
    },
    "category": {
      "type": [
        "string",
        "null"
      ]
    },
    "swimlane": {
      "type": [
        "string",
        "null"
      ]
    },
    "column": {
      "type": [
        "string",
        "null"
      ]
    },
    "position": {
      "type": [
        "integer",
        "null"
      ]
    },
    "assignee": {
      "type": [
        "string",
        "null"
      ]
    },
    "creator": {
      "type": [
        "string",
        "null"
      ]
    },
    "time_estimated": {
      "type": [
        "string",
        "null"
      ]
    },
    "time_spent": {
      "type": [
        "string",
        "null"
      ]
    },
    "due_date": {
      "type": [
        "string",
        "null"
      ]
    },
    "created": {
      "type": [
        "string",
        "null"
      ]
    },
    "modified": {
      "type": [
        "string",
        "null"
      ]
    },
    "description": {
      "type": [
        "string",
        "null"
      ]
    },
    "subtasks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "title": {
            "type": [
              "string",
              "null"
            ]
          },
          "assignee": {
            "type": [
              "string",
              "null"
            ]
          },
          "time_tracking": {
            "type": [
              "string",
              "null"
            ]
          }
        },
        "required": [
          "title",
          "assignee",
          "time_tracking"
        ],
        "additionalProperties": false
      }
    }
  },
  "required": [
    "task_reference",
    "title",
    "status",
    "priority",
    "complexity",
    "category",
    "swimlane",
    "column",
    "position",
    "assignee",
    "creator",
    "time_estimated",
    "time_spent",
    "due_date",
    "created",
    "modified",
    "description",
    "subtasks"
  ],
  "additionalProperties": false
}. Preserve top_k when supplied, presence_penalty 0 and the submitted whitespace setting.

Deliverables
- A typed client wrapper, an application-level function for this task, error handling for timeouts/empty input, and unit tests with a stubbed client.
- Wire it into my existing stack (ask me which framework if unclear) and add a short usage example.
Schema map
Requested record shape
task_reference
string | null
title
string | null
status
string | null
priority
integer | null
complexity
integer | null
category
string | null
swimlane
string | null
column
string | null
position
integer | null
assignee
string | null
creator
string | null
time_estimated
string | null
time_spent
string | null
due_date
string | null
created
string | null
modified
string | null
description
string | null
subtasks
array

Schema illustration. The saved output above belongs to the recorded request.

Read the screens your agent already uses

Self-hosted (Kubernetes)

SIE in your clusters, backed by our engineers.

  • Data stays in your own cloud account
  • Same models and API as Serverless
  • Terraform to your cloud in minutes

Run locally

The open-source server on your own machine.

docker run --gpus all -p 8080:8080 ghcr.io/superlinked/sie-server:latest-cuda12-default

Switch to SIE in 5 minutes

import anthropic

client = anthropic.Anthropic()
reply = client.messages.create(
    model="claude-sonnet-5-5", messages=messages,
    max_tokens=32768,
    output_config={"format": {"type": "json_schema", "schema": schema}},
)
from openai import OpenAI

client = OpenAI()
reply = client.chat.completions.create(
    model="gpt-6.1-sol", messages=messages,
    reasoning_effort="medium",
    max_completion_tokens=32768,
    response_format={"type": "json_schema", "json_schema": {
        "name": "screenshot_record", "strict": True, "schema": schema,
    }},
)
SIE
from sie_sdk import SIEClient

client = SIEClient()
reply = client.chat_completions(
    "google/gemma-4-31B-it", messages,
    max_completion_tokens=32768,
    response_format={"type": "json_schema", "json_schema": {
        "name": "screenshot_record", "strict": True, "schema": schema,
    }},
)
from sie_sdk import SIEClient

client = SIEClient()
reply = client.chat_completions(
    "google/gemma-4-31B-it", messages,
    max_completion_tokens=32768,
    response_format={"type": "json_schema", "json_schema": {
        "name": "screenshot_record", "strict": True, "schema": schema,
    }},
)

Contact us

Tell us about your use case and we'll get back to you shortly.

Apply for an inference grant

Free capacity on our hosted cluster for selected projects. Tell us what you run and we reply by email.