A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Acting by an app's number (read_app's e7) is now a real click at that control's spot, the
same path as screenshot coordinates. agent-desktop only reads; its own actions (a second set
with its own key names and failures: no typing in terminals, "/" and "`" failing off the main
thread) are gone, with computer_use_select, background clicks and press_key's app.
- Every action comes back with the screen after it (or, with no picture allowed, the front
app's numbered controls), so the bot no longer spends a step on a screenshot to check.
- Coordinates map without remembering a screenshot: the picture's size depends only on the
screen. The phone's taps go straight to the screen as the person's, in the pixels of the
picture it was sent; they used to borrow the bot's last screenshot scale and landed 1.25x off
on the bots' computer, or failed with "screenshot first".
- Mac typing pastes (⌘V, clipboard put back): with 注音 on, text sent as key events vanished
while return arrived, so commands typed into a terminal ran empty. Bare printing keys
("slash", "r") are typed the same way; keys like "slash" and "grave" have names.
- The bots' Docker computer: one docker exec per step (action, settle, picture), never
`mousemove --sync` (it waits forever when the pointer is already there: a second click on
the same spot hung two minutes), open_app (chromium = the browser browser_* drive, xterm).
Tests: box_screen_is_exact (real Docker: bot and phone coordinates exact, same spot twice in
under 2 s, ASCII + Chinese + named key typed), Calculator by numbers and Terminal typing under
注音 through the signed app, remote_e2e 28/28, Linux build + 151 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
王性驊