- menu_click walks the real macOS menu bar in one step (three screenshot
rounds per level become one call + one picture)
- fill_form writes every named field with one AXValue write each, in one step
- wait_for polls the control tree (no screenshots) until text appears/gone —
replaces the screenshot-polling loops that filled whole turns with identical
pictures
- batch steps take an optional expect: the batch stops and shows the screen at
the step whose promised text never showed, instead of running a blind plan
on sand
Menu/fill are macOS accessibility; other systems keep click+typing per field
(the tool descriptions say which is which).
The bench's precision failures (quick model clicking the wrong number, Retina
scaling, covered windows) all come from acting on pixels. A plain left click
now asks the frontmost app's accessibility tree first — by name, or by the
numbered spot — and typing writes AXValue into the focused field once,
keyboard events only when accessibility can't reach. Any app with a thin tree
behaves exactly as before: every AX answer that isn't a clear yes falls
through to CGEvent.
macOS code type-checked with --target x86_64-apple-darwin; gate: build on the
macos-arm64 runner + bench tasks T2/T5 (clicking by name) on a Mac.
Linux/Windows: stubs, pixel path unchanged.
AnyBot Dev