Finding things on screen

Some steps act on another app's window: Click, Set field value, Read field value and Wait. They need to know which thing to act on, and a locator is how you tell them. It is not a screen position. It records what the element is, so the macro keeps working when the window moves or the app changes a little.

What a locator holds #

A locator describes one element in an app's accessibility tree, the structure macOS gives to screen readers and automation. It can record:

• role: the kind of element, such as AXButton or AXTextField. Always present.

• identifier: the developer's own name for the element. The strongest clue when the app sets one.

• title and description: the visible label and the accessibility description.

• path: the chain of parents from the app down to the element.

• frame: where the element sat inside its window.

• inWindow: the window title, so a second window with a similar button does not confuse it.

• alternatives: other locators for the same element, tried in order.

• fallbackPosition: a last-resort click position.

You rarely write one by hand. The editor builds it when you Record or press Re-record on a step and click the element, and Switchboard keeps it up to date as it runs.

How a match is decided #

Switchboard reads the app's tree and scores every element against the locator. Each attribute the locator recorded is worth points:

Only the attributes the locator actually recorded count. A locator with no identifier is scored out of the points it has, so an element can still reach 100%. The score is the points earned divided by the points possible.

• Title and description earn full points for an exact match. If they differ only in capitals, or one contains the other, they earn about half.

• Path earns points for each step that matches at the same depth (same role, and the same position among siblings or the same title).

• Frame is measured inside the window, so moving the window costs nothing. The closer the element's centre, the more it earns, and a quarter of the window's diagonal or more earns nothing.

• Value is never scored, because it changes.

The best element decides the outcome:

If several elements tie, Switchboard uses the first and says so in the trace as "ambiguous".

Alternatives and self-repair #

If the primary locator scores under 70%, each locator in alternatives is tried in turn, and the best one that reaches 70% wins. The trace says "matched through alternative 0".

When a different locator wins, Switchboard re-records: the winner becomes the primary, the old primary moves into the alternatives (up to five are kept), and the step's output carries the updated locator along with the confidence. That is how a macro survives an app update that renames a button.

Waiting before acting #

Apps take a moment to draw their windows, so a Click or Set field value does not fail the moment the element is missing. It waits, for up to the step's timeout, for the element to appear and be enabled. It wakes as soon as the app reports a change, and otherwise checks every quarter of a second.

Use Wait when you want to wait on purpose before doing something else. It can wait for an element, for a window by title, or for an app to launch, and with gone set it waits for the thing to disappear.

Steps that read another app are simulated in a dry run unless you mark them "Safe in a dry run" in the inspector. See Debugging.

To check without waiting, use the Element exists condition in an Only if list or an If step.

When nothing matches #

If the primary and every alternative stay under 70%, a click tries two more things before it fails.

1. Text on screen. If the app exposes almost no elements (see below) and the locator has a title or description, Switchboard takes a screenshot of the window, reads its text, and clicks the text it finds. This needs the Screen Recording permission.

2. The recorded position. If the locator has a fallbackPosition, it is used, but only when the window is within 10% of the size it had when the locator was recorded. A resized window means the position cannot be trusted, and the step fails instead of clicking the wrong place.

If both are unavailable the step fails with a failure block that says what it looked for, what it found, the nearest candidate and a suggestion. Debugging explains how to read it.

Apps with no accessibility tree #

Some apps do not describe their interface: games, canvas-based tools, and some Electron or custom-drawn apps. Switchboard notices when a window holds only a handful of elements and then reports that the app "doesn't describe its elements to Accessibility".

For those apps you have two options:

• Text recognition. Described above. The click step uses it on its own when a locator has a title or description. Matching is case-insensitive. An exact line wins, otherwise the shortest line containing your text.

• Pictures. Look for an image instead of an element, below.

Some apps expose their tree only when asked. Switchboard switches that on for the app when it can.

Finding a picture #

An image locator looks for a small PNG on the screen or in the front window. It is a locator whose role is image.

• image is a path relative to the Library folder, conventionally assets/<macro id>/<name>.png.

• threshold is how closely the picture must match, from 0.3 to 0.99. The default is 0.85.

• scope is screen (all displays, the default) or window (the front window only).

• pick chooses among several matches: best, top, bottom, left or right.

Matching works on the picture's brightness pattern, so colour themes matter less than shape. It copes with Retina scaling. If the picture is missing the step says so; if it is not found, the failure block reports the best score it saw and the score it needed.

The Image found condition asks the same question without clicking.

In a dry run imageFound does not look at the screen. It answers with sample, which defaults to true.

Image locators need the Screen Recording permission. See Troubleshooting permissions.

Recording #

The editor's Record command (⇧⌘R) turns what you do into steps: each click becomes a click with a full locator, typed text becomes a type step, and key chords become key presses. It pauses while secure input is on, so passwords are never captured. Press ⎋ in the app you are recording to finish; the steps are inserted as one edit you can undo.

The Recorder also writes menu choices, controls, app switches and copy and paste as the steps they mean: see Recording a macro.

To replace the locator of one step, select it and press Re-record in the inspector, then click the element (⎋ cancels). Convert to Element (Macro menu) turns a click by position into a click on the element under it. Both need the Accessibility permission.

Generalizing a recording #

A recording is literal: it clicks exactly what you clicked, three times if you did it three times. Generalize… (Macro menu, or the button beside Make a macro when you finish recording) turns it into a macro that says what you meant. Nothing changes until you have read the proposal and pressed Apply, which is one Undo.

Switchboard first does the safe part with plain rules, then asks the model only for names, labels and a description:

• Repeats. Two or more runs of the same steps that differ only in the text typed become one Repeat over a list. If the runs are identical, it is a plain count. A short run of identical clicks is left alone.

• Waits. A click that follows a change of app or window, after a pause of at least 1.5 seconds in the recording, gets a Wait for its element first, up to 5 seconds.

• Inputs. Text you typed that appears nowhere else becomes an input with your text as its default, so the macro still does the same thing until you change it. At most three.

• Selection. A recording that starts by copying with ⌘C keeps the copy in a variable called selection.

• Tokens. Text you typed that is exactly what you just copied becomes {{copied}}, and today's date becomes a date() token, so the macro types what is true when it runs.

The model may name the macro, label steps and rename the variables above. It may not change a locator, a position, a key or a script, or add a step that acts on your Mac. If it tries, the proposal is refused and you are told which step it touched. A generalized macro arrives turned off, like any macro made with AI.

Tips #

• Prefer elements that have an identifier. They survive relabelling.

• Keep inWindow set when an app has several windows with the same controls.

• Add an alternative that uses only role and title. It survives a restructured window.

• If a click works but the trace says "low confidence", re-record the step soon.