Ask AI about the screen
Asks a model a question about what is on the screen or in a region of it.
Ask AI question about the screen
Details #
Reads the text on the screen (or the region you give) with on-device text recognition, then asks the model your question about that text. The model never sees a picture, only the recognised text, so it cannot answer questions about images or colours. It needs the Screen Recording permission, and fails if no text is found.
Parameters #
| Parameter | Type | Required | Values | What it does |
|---|---|---|---|---|
question | text | Yes | Required. What to ask about the screen. | |
dryRunUsesModel | yes / no | true lets a dry run call a cloud model. | ||
hud | yes / no | false turns off the "Thinking…" HUD with the AI badge that shows while the model works. On by default; a dry run never shows it. | ||
into | text | The variable for the answer. | ||
model | text | haiku or sonnet, for the cloud; or a model name when provider is a provider id. | ||
preset | text | An AI preset from Settings › AI (its id), or default for the default preset. Pick one in the Inspector's Model menu; renaming a preset changes nothing here, and a deleted one runs on the default preset with a warning. Without it the step behaves as before. | ||
provider | text | auto, device or cloud; or a provider id (anthropic, openai, gemini, or an OpenAI-compatible server's id) together with model. | ||
region | any value | What to read: "window" (the front window, the default when left out), "screen" (the main display) or a rectangle {"x": 0, "y": 0, "w": 800, "h": 400} in screen points from the top left. | ||
sample | any value | What a dry run answers when no model runs. |
Every step also accepts label, onFailure, timeout and dryRunSafe.
Examples #
- Ask AI about the screen: “What error is shown?”
Gotchas #
Where it runs.
providerisauto(the default),deviceorcloud.autouses Apple Intelligence on this Mac when it is on and the request fits its small window (about 4,000 tokens), otherwise your cloud model (your own Anthropic key first, then your Switchboard plan).devicenever leaves the Mac and fails if it is off or the text is too long.cloudneeds a model set up in Settings > AI. model (haiku, the default, orsonnet) only matters for the cloud.Output. The answer goes into
into. The step's own output also carriesvalue,provider(device,own-keyorplan),model,ms,tokensIn,tokensOutandcents, plusrepaired: truewhen the first answer was rejected.A bad answer. If the answer does not fit, the model gets one second try with the errors. If that fails too, the step fails and follows its
onFailure.Time limit. These steps get 60 s by default, not 10 s.
Dry run. The screen is still read in a dry run, so Screen Recording must be allowed and some text must be on screen. These steps are safe in a dry run. The on-device model runs for real. A cloud model does not (it costs money): the step answers with
sample, or an empty string, unless you setdryRunUsesModeltotrue. If no model is set up, a dry run usessampletoo.Text you send to a cloud model leaves your Mac. Do not send secrets or private data to it.