Browse the docs

Read text on screen

Reads the words, QR codes, barcodes or a table in part of the screen, and stores the text.

Read the text in region

Details #

Takes a picture of part of the screen and reads it on your Mac: text with on-device recognition (the accurate recogniser, with language correction), and QR codes and barcodes. Nothing leaves your Mac. It reads anything you can see and cannot select: a paused video, a screenshot in a chat, a scanned PDF, an error dialog. It needs the Screen Recording permission.

What into holds. The text. When a QR code or barcode is in the area, the code's contents come first and are the text; the recognised words are still in screenRead.ocr. If nothing was found, the text is empty and the step still succeeds, so the macro can say "No text found".

The screenRead variable. After every read, screenRead holds the details:

  • text: the same text as into.

  • kind: text, barcode or none.

  • lines: how many lines the text has (or how many codes).

  • barcode and barcodes: the first code's contents, and every code's, top to bottom.

  • link: true when the code (or the only text) is one web address.

  • table and hasTable: when three or more rows have words that line up in two or more columns, table is those rows with the columns separated by tabs, ready to paste into Numbers or Excel as cells. Otherwise it is empty.

  • ocr: the recognised words even when a code took the text slot.

Lines. lines decides what happens to the line breaks in the picture:

  • keep: one line per line of text, as drawn.

  • join: one paragraph. A word split by a hyphen at the end of a line is put back together.

  • auto (the default): joins lines that end mid-sentence (no closing punctuation, and the next line starts in lower case) and keeps the other breaks.

Where to look. region is window (the front window, the default), screen (the main display), all, or an area {"x": 0, "y": 0, "w": 800, "h": 200} in screen points from the top left. A Ask for input of kind pickRegion answers with exactly that shape, so "region": "{{region}}" reads what you dragged over.

Parameters #

ParameterTypeRequiredValuesWhat it does
actionsyes / notrue to open Actions for This over what you read, when all of it is one thing: a tracking number, an email address, a link or a type you taught. screenRead.thing and thing, thing.type and thing.label then describe it. Prose and anything else that is plain text opens nothing. Default off; the text is read, and copied by your own steps, exactly as before.
detectany valueA list of what to look for: text, barcodes, or both. Default both.
intotextThe variable that receives the text.
languageslist of textLanguage codes to read, such as ["en-US", "fr-FR"]. Left out, Switchboard detects the language.
linestextkeep, join or auto. Default auto.
regionany valueWhat to read: window, screen, all or an area {x, y, w, h} (width and height work too). Default window.

Every step also accepts label, onFailure, timeout and dryRunSafe.

Examples #

Copy the text in a box you drag
Do
  1. Drag over the screen to pick an area
  2. Read the text in the picked area into words
  3. Copy “{{words}}” to the clipboard
  4. Show “Copied {{screenRead.lines}} lines”
Open in Switchboard to try it
Open a tracking number's page
Do
  1. Drag over the screen to pick an area
  2. Read the text in the picked area into number
  3. Open https://www.google.com/search?q={{number}}
Open in Switchboard to try it
Capture a table and paste it as rows
Do
  1. Drag over the screen to pick an area
  2. Read the text in the picked area into words
  3. Copy “{{ screenRead.hasTable ? screenRead.table : wor…” to the clipboard
Open in Switchboard to try it
Capture, translate and show
Do
  1. Drag over the screen to pick an area
  2. Read the text in the picked area into words
  3. Ask AI to rewrite: “Translate to English: {{words}}”
  4. Show “{{english}}” for 6 s
Open in Switchboard to try it

Gotchas #

  • Screen Recording. Without the permission the step fails with a message that says so. Allow Switchboard Engine under System Settings > Privacy & Security > Screen & System Audio Recording.

  • Small or blurry text. Recognition works best on text at least about 10 points tall. Zoom in first, then read.

  • Tables. A table needs three or more rows and two or more columns with a clear gap between them. Rows of ordinary prose are never turned into a table.

  • Dry run. The step is simulated: it reads nothing and sets only into. screenRead is not set, so give it a starting value if later steps use it.

  • Time limit. Reading a whole screen can take a second or two.

See also #

Ask for input, Pick a screen point or rectangle, Work with a picture, Ask AI about the screen