- Typed into the middle of Chromium's 「about:blank」, an address came out as
chrome://blankchrome//settings/search, and every step after it went wrong. Typing into a
field by name or number now selects a one-line field's text first (ctrl+a / cmd+a) so the
text replaces it; append=true keeps it. Text areas are left alone.
- A model sending name plus x: 0, y: 0 had its click land in the screen's corner: a name (or
number) now always wins over coordinates.
- After the VM went down hard, the box's first start failed in gVisor ("mount source already
has a filestore file … repeated submounts are not supported") and the second worked: waking
or restarting the box tries once more on exactly that error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
A quick model looking numbers up pressed 1, 2, 3, 1 for 1234, and 1234 × 5678 = didn't fit in
an 8-action batch (it split it and lost a key). The click and batch descriptions now lead with
the control's name (name "4"), and a batch takes up to 16 actions.
The benchmark gets four tasks on things none of the others use (Mousepad and its save dialog,
a Chromium bookmark, ImageMagick, a page of nested web components), run only when asked for
(--tasks N1,…): a check that the changes carry over to what they weren't tuned on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
`anybot --ask` with ANYBOT_ASK_HERE=1 offers this computer too (headless runs leave it out), so
screen and menu-bar steps can be tried on a Mac without the window. Tried on a Mac mini: the
menu bar's icons listed in one step, a status icon clicked by its name, AppleScript reading the
front app, Tailscale's own CLI answering — all without a screenshot.
An App Store app's CLI joined to `> file` stays in the sandbox and dies; the note now says so
(a bot blamed --json for it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
One model call per step now: a replay step needs none; the quick model's step falls back to the
bot's model when it can't be used (but a too-long chat is folded and asked again, whichever model
asked); each attempt is in the work log. A zoom is a look (safe to do again after a crash).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
bench/computer-use/run.py runs each task through `anybot --ask` with a fresh chat, checks the
result inside the box, and records success, steps, time split between model and tools, route
switches and a per-step transcript. It refuses to run against the real box (needs its own
ANYBOT_BOX_NAME / ANYBOT_BOX_VOLUME and a scratch data folder). --keep-skills carries learned
skills into the next run to measure replay. README has the tasks, how to run, and the
2026-11-09 numbers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Two copies on one folder each keep their own idea of the chats and overwrite each other's saves,
and a second copy starting up would take the first one's open turns in the work log for a crash
and close them. The window, `--serve` and `--ask` now hold an OS file lock on anybot.lock in the
data folder (flock; LockFileEx on Windows), and `anybot setup` holds it while it writes the
settings, after stopping the service and before starting it again.
The lock goes with the process however it ends, so a copy that crashed or was killed never blocks
the next one. A new copy waits up to 5 s for one that is closing (a relaunch, a service restart).
Past that, a second window shows a small notice naming the holder's process and the folder, and
`--serve` / `--ask` say it on stderr and exit 1 (launchd's and systemd's restart then retry). Each
ANYBOT_DATA_DIR has its own lock; a screenshot run doesn't take it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
agent-desktop knows an app by its program's name, but the X11 window says its class: a Snap
Firefox is "firefox_firefox", so the front app was never found and the bot got a bare picture
(「看不出最前面是哪個 App（WINDOW_NOT_FOUND）」). The window's process now gives the name.
When there are no controls at all and the desktop's accessibility switch (org.a11y.Status
IsEnabled) is off — most apps then show none — AnyBot turns it on, as a screen reader would, and
tells the bot that apps already open show their numbers after a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Firefox dropped Chrome's DevTools protocol, so bidi.rs speaks WebDriver BiDi and does each
DevTools command the browser tools use the BiDi way (real mouse and keys, typing, screenshots,
PDF, dialogs, uploads, downloads, tabs). The 18 tools themselves are unchanged and pass the same
dashboard test on Firefox as on Chrome.
- A computer with only Firefox runs AnyBot's own profiles in it (under the Snap/Flatpak corner
of the home folder when Firefox is packaged that way).
- The person's own Firefox profiles are choices too, by the names Firefox shows (profiles.ini
and the newer profile switcher's SQLite list): 「阿金（Firefox）」, or just 「阿金」. Firefox
can't be taken over once open, so a profile open the everyday way is refused with "close it
first"; AnyBot then opens it itself, logins intact.
- Settings lists them without a remove button; the messages name Firefox as an option.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
setup.sh offers 「區網 IP」 (alone, or next to Tailscale): Caddy serves https://<LAN IP> with its
own small certificate authority (tls internal, default_sni for clients that send no name) and
hands out the root at http://<LAN IP>/anybot-ca.crt for each device to trust once.
docker-compose.lan.yml binds Caddy to that address only, so it sits beside tailscale serve.
The app now accepts, for the account server, the certificates this computer trusts as well as
the built-in public ones (HTTP and the WebSocket alike), so it connects once the root is
trusted; before that it says what to install instead of "the certificate is wrong".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Every warning `cargo clippy --all-targets` gave (50 on macOS, 8 more only on
Linux, 1 in the relay), fixed without changing what anything does:
- long types get a name (Peer, Members, Imported, Joining, Mention, Stream…);
- the two big enum variants are boxed (the vault's login plan, joining an
account);
- a test now waits for the copy it relaunched (it left a zombie behind);
- a hint's format had a dead `if` whose two branches were both "";
- the rest is what clippy suggests: sort_by_key, as_chunks, is_none_or,
is_multiple_of, while let, from_ref, next_back, struct init, no casts to
the same type.
Checked: cargo test (177 macOS, 181 Linux), relay tests (11), remote_e2e
28/28, and the live Linux test (X11; Wayland clicks, keys, Chinese typing,
scroll and screenshots).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
A new mobile/index.html in the desktop's (and Grok Bot's) shape: the sidebar
with its sections and folders, the chat with steps, thinking, reactions, quotes
and the approval card, 對話詳細資訊, Bot 設定, routines, pages, computers and
設定. Each action goes to the desktop's own code on the app's thread
(api/act/<what> -> App::phone_act): appctl, set_pinned, restart_bot,
toggle_reaction, box_task, channels_restart and the rest. No new rules.
Keys, channel tokens and the vault never reach the phone; a model key can only
be written. Admins keep what the phone always let them do; making, changing or
deleting bots, routines, memories, pages, folders and settings stays the
owner's. The state the phone polls now carries the sidebar's section order.
Checked by using it on a preview brain: routines, memories, templates, folders,
keys, signing another phone out, approvals and a chat with a fake model.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
The phone showed a bare 「刪除技能」 when 新 Bot deleted the test skill; save/use/delete skill,
remember and forget now carry the name or the fact.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
The tool only said it put the file in; on a page that shows nothing about it the bot spent five
more steps checking (shell, read, screenshot, two scripts). Now the result names the file the
field holds (and its size) and the page's new text, and an empty field is an error. Real run in
the gVisor box with Grok: the sign-up form went from 10 steps / 35 s to 6 / 20 s; the second
time it followed the skill it had learned, by names only, 6 steps / 21 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
- box_numbers_only_on_what_is_seen is ignored again (a test inserted before it in cbada6b took
its mark: every cargo test drove the real box), and the path test runs
- gvisor-vm.sh restarts the VM's Docker only when runsc was just registered (a restart stopped
Bot 的電腦), and `${var}` before full-width text (bash read 「）」's first byte as part of
the name and stopped)
- a scheduled box update and a PDF's name use the person's time zone (a server keeps UTC)
- 容器隔離 lists 標準 before 強隔離
- screenshots of real data can open 設定 › 電腦 (ANYBOT_SHOT_VIEW=computers) and scroll
(ANYBOT_SHOT_SCROLL)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
容器隔離 in 設定 › 電腦: standard (runc, the container is the only wall, Chromium runs
--no-sandbox) or hardened (runsc: a virtualized kernel, all capabilities dropped but the few
the desktop needs, ANYBOT_BOX_SANDBOX=1 turns Chromium's own sandbox back on). Box image v7.
deploy/mac/gvisor-vm.sh makes a Lima VM with Docker + gVisor for a Mac (started at login),
gvisor-migrate.sh moves the box's files and logins there.
Checked and finished before committing:
- the hardened choice is offered only on a Docker host that has runsc; a box set to it on a
host without it opens the standard way and says so (it used to not start at all)
- the settings row shows what the box really runs on, and when 更新 is needed
- no swap only under gVisor (a standard box keeps Docker's default)
- the migrate script sees the running Mac app ("AnyBot", not "anybot")
- the VM script starts an existing VM by name, checks runsc and the kernel from the Mac
- the version test says 7
Real run in the Lima VM: desktop, VNC, numbered controls, xdotool and screenshots work; no
Chromium process has --no-sandbox, renderers run with seccomp and their own namespaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Every browser_open is a new tab and nothing closed them. When a job ends (answered, stopped
or failed) each bot browser it used keeps only its front page; a browser that isn't open isn't
started for this, and one another job is still using is left alone. The six-tab limit stays
for long jobs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
- A job of five or more real steps is looked back on after the answer: the short way that
worked becomes a skill (or improves the one it followed); the reply says so. The prompt has
the bot check its skills before starting.
- Upload, download and PDF on Bot 的電腦's browser look for and save files in the box, not on
the brain (a picture in a Threads post took 24 steps).
- The bot's browser keeps the six tabs used last (19 Threads tabs had filled the box's 2 GB).
- Words name the control, not the box around it: a form's 「Customer name:」, 「Large」 and
「Submit order」 used to land on the <p> holding them and do nothing.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Both cloud brains had 「Etc/UTC」 as the time zone (the server's, set when they were made), so pages
said 「今天 01:03」 for 09:03 and a routine for 9 in the morning would run at 5 in the afternoon. The
phone now says where the person is; a brain still on a server's zone takes it (a zone the person
chose stays). Page times use the person's zone, not the computer's.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
The phone page was bare: one row per bot, a chat listing every step, a menu with four lines.
- Home: the main bot large at the top with its model and last words, the other bots with times,
then its computers (own box and the account's, online or not, 看畫面), the newest pages and the
routines with their next run.
- Chat: the bot's name opens its settings — model (chat models only), computer, past conversations
(restore), regenerate, new chat. A turn's steps fold into 「已完成 N 個步驟」 when done (the newest
three stay while it works); copy and regenerate under replies; four ready starts in a new chat;
a jump back to the newest message.
- Menu: 外觀 (follow the phone, light, dark).
- API: api/state carries pages and models; each bot its model, computers, routines, past
conversations; POST api/bots/{id}/set | regenerate | restore. The home's last line skips errors.
Checked on a local copy of the brain's data (cut off from the account and Docker) at iPhone size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
From looking at the real phone page after the fixes: a long command squeezed its step's name into
one character per line (執/行/指/令), and sending a file showed as 「工具」. The name keeps its width,
the detail is cut with an ellipsis, and send_file, react and view_demo_step have names; the window
says 「正在傳檔案給你」 instead of 「正在記下來」 while a file is sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
From the second round of real runs:
- Direct mode no longer asks the model to check "always ask" rules before steps that only look
(screenshot, showing the screen, reading a page or file): none of those rules can be about them,
and the check cost each look at the person's Mac 10-20 s. Clicks and typing are still checked.
- An error inside the model's answer stream ("The model is currently at capacity…") is retried after
8 s and said in Chinese; it used to become the bot's reply, in English, never retried.
- send_file says it sends from any computer, the person's included (the bot copied a file to its own
computer first, which only works for text).
- A shown screen's file name carries no time (a server's clock is UTC).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
agent-desktop comes from GitHub, and only the crates registry and target were cached, so every
server update fetched it again; a slow GitHub made nobledesktop's update fail (cargo exit 101, then
a fetch that hung for over ten minutes).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
- A turn may take up to 100 steps (was 25, which cut off real work; Grok Bot sets no limit).
Every 25 steps the bot is asked whether it is still getting anywhere.
- The system prompt asks for independent tools in one step and combined commands.
- Browser steps for the box always go to the box's Chromium. They used to depend on whether
the brain itself had a Chrome, so a cloud brain sent them to "the person's browser".
- The box's DevTools forward passes bytes with a plain read/write loop. std::io::copy splices
on Linux and held back short requests, so the box browser never opened from a Linux brain.
- A gateway's HTML error page shows as plain words.
- Real test a_brain_without_chrome_browses_on_its_box (ignored): passes in a Linux container
with no Chrome against a real box, and fails without the routing fix.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
box_real_usage_drill (ignored) drives the whole path on a page that reports what really landed:
one batch fills and sends a form (six steps, Chinese typing, 1.6 s), a bare-x/y click on a canvas
lands 1 px off its target, a drag ends where it was pointed, scrolling reveals a field that is
then clicked and typed by number, and the same spot clicked twice says so. Every step is timed:
look 477 ms, steps 650-880 ms.
Caught: the a11y tree is exposed after the screen settles (measured 0.2-0.35 s behind a scroll),
so a list read in parallel with the picture can miss a control the picture already shows.
box_look now starts the capture in the background and gives the a11y read 0.18 s before running;
the parallel capture pays for it, and a scrolled-in field is in the list every run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- computer_use_batch: up to 8 planned actions run in one docker exec (steps chained with && and a
done-marker each), the first failure stops the rest, one picture comes back at the end. Typed
text rides inside the script quoted (one stdin for the chain). The prompt tells the model when
to use it (a form whose fields are all visible) and when not to (step 2 needs step 1's screen).
- The box waits for the screen to stop changing, not the clock: two matching small probes (~60 ms
each) mean settled — ~160 ms on a still screen where every step paid a fixed 500 ms, and a page
mid-load is waited for (up to ~0.9 s) instead of captured mid-change. The picture is JPEG, and
the a11y read runs in parallel with the capture (look ~307 ms; two steps with pictures 1102 ms).
(SETTLE_SH is a group command: dash needs ';' after it — found by box_screen_is_exact.)
- The home volume (every bot's logins) is never one command from loss: snapshots keep generations
(last 7 days, then one per week; the old single snapshot migrates by file date), Reset and
Restore save the running box first and refuse to wipe if saving fails, a box built on an empty
volume re-fills itself from the newest snapshot, and a sleeping holder container keeps the
volume attached so `docker volume prune` can't take it. make doctor shows volume and snapshot
state; the 電腦 card shows the box's average step times and says resets save first.
- box_batch_in_one_round_trip (ignored): three typed steps, ASCII and Chinese, land in one trip.
On the real box after upgrading v3→v5: box_update_keeps_files (files survive, snapshot 2 代),
box_numbers_are_exact, box_screen_is_exact, box_end_to_end, box_traffic_via_mac pass; 155 unit tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Found on the server's box after the update to 4:
- Chromium called its whole window "not showing" (it took itself for minimized) while its page
was on the screen; the reader (like XBot's) skipped anything not showing, so a page had no
numbers. Now: visible with a place on the screen counts, containers are walked through, and
only a page that isn't showing (a background tab) is left out.
- Two browser profiles: Chromium's own (~/.config/chromium, where a person logged in by hand)
and AnyBot's (~/.config/anybot-browser); the launcher used AnyBot's and Threads showed as
logged out. One profile now (anybot-profile → Chromium's own; a box with only AnyBot's keeps
that); both folders stay as they were.
- The launcher's lock fd went to Chromium, which then held it for its whole life: the next start
waited 30 s and failed. And its flags line had been lost in an edit.
box_numbers_are_exact passes twice on a box from this image; the launcher answers in 0.09 s when
the browser runs; box_screen_is_exact passes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unused (found by taking away the dead-code allowances; nothing else used them, tests aside):
computer::specs, quiet_button, the never-made Poke::Spin, ten icons and their SVGs, the
theme's chrome/bubble_agent colours, ImportRow.line, voice's multipart/transcribe/speak
wrappers, and the old keychain writers vault::save/remove/index_add with their live test
(the vault store replaced them; reading old keychain entries for the import stays).
One copy each: percent-encoding (httpd::url_encode, was four), tool arguments (tools::args_of,
was three), the WebSocket handshake (httpd::accept_websocket, was three), the read pacing
(wire::Pace, was two: relay and app), docker with input and a time limit (dk_with_input, was
two). Docs: FEATURES §看畫面 (VNC), protocol (v* frames, /screen, tickets), architecture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The core: the bot works the computer; a person watches it like a colleague, and takes over
when it needs them. VNC is made for that (only what changed, every mouse move and key, touch on
phones), so the pictures sent one by one go, and with them their own input path.
- vnc.rs: a VNC proxy to the box's x11vnc through one docker exec (a Python bridge that ends
with either side). It signs in to the box with the box's own password (its VNC_PASSWORD) and
offers the viewer no password, so the password never leaves the brain. The person's clicks
and keys make the bot's steps wait until 8 s after they stop; text typed in the viewer's box
(any language) is pasted in the box.
- Account server: /screen (the viewer page) and /novnc.js (noVNC 1.6.0, MPL-2.0, bundled);
/relay/vnc/<host> takes the browser's WebSocket (account session, or a 10-minute ticket the
app asks for with POST /relay/screen-ticket; owners and admins) and carries it over that
AnyBot's lasting connection (vopen / vdata / vtext / vclose). Both ends of that connection
wait 2 ms instead of 40 ms to read while things move: a request and its answer took up to
160 ms more than needed.
- Phone page: 看 Bot 的電腦 and 接管 open the viewer page (also on the local network: /api/vnc on
the AnyBot's own phone server). This Mac's own screen moved to the ⋯ menu.
- Mac app: 看畫面 opens the viewer in the browser (a screen of a brain: the brain's computer by
ticket; else this computer's box, noVNC straight from it). The in-app picture viewer, the
live JPEG stream, the person's click/move/scroll/key endpoints and the input shell are gone.
- Opening a Mac app uses LaunchServices (`open -a`): agent-desktop's launch waited on the window
through accessibility and failed without 輔助使用 now and then.
Measured on this Mac (made-up account, local relay + headless brain + the real box): viewer
signed in in 0.4–0.7 s, a request and its answer through relay + brain + box 25–58 ms median;
headless Chrome shows the desktop live in /screen. remote_e2e 28/28 ×2, 149 unit tests, relay 11.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- One lasting docker exec takes the desktop as a JPEG inside the box and sends a picture only
when it changed (quicker while it changes, slower while still). Viewers ask for the picture
after the one they have and get it at once. It was one exec per picture (a 1 MB PNG decoded
and encoded again on the brain) and a fixed 0.7 s / 1 s pause: about one picture a second.
Measured on the Mac's box: first picture 45 ms, ~90 KB (was ~1.1 MB), 9–19 pictures a second
while it changes; viewers take at most ~8 a second over the network.
- The loop in the box ends the one before it, ends itself when a write fails ("0" lines while
still) and is ended when nobody watched for 10 s (ending docker exec on this side left it
running in the box).
- The person's clicks and keys go through one lasting shell in the box (text as base64), not a
docker exec each.
- Answers over the lasting connection keep their headers (the relay dropped them): a picture's
number reaches the app. JPEG/PNG/PDF/zip bodies are no longer deflated again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Acting by an app's number (read_app's e7) is now a real click at that control's spot, the
same path as screenshot coordinates. agent-desktop only reads; its own actions (a second set
with its own key names and failures: no typing in terminals, "/" and "`" failing off the main
thread) are gone, with computer_use_select, background clicks and press_key's app.
- Every action comes back with the screen after it (or, with no picture allowed, the front
app's numbered controls), so the bot no longer spends a step on a screenshot to check.
- Coordinates map without remembering a screenshot: the picture's size depends only on the
screen. The phone's taps go straight to the screen as the person's, in the pixels of the
picture it was sent; they used to borrow the bot's last screenshot scale and landed 1.25x off
on the bots' computer, or failed with "screenshot first".
- Mac typing pastes (⌘V, clipboard put back): with 注音 on, text sent as key events vanished
while return arrived, so commands typed into a terminal ran empty. Bare printing keys
("slash", "r") are typed the same way; keys like "slash" and "grave" have names.
- The bots' Docker computer: one docker exec per step (action, settle, picture), never
`mousemove --sync` (it waits forever when the pointer is already there: a second click on
the same spot hung two minutes), open_app (chromium = the browser browser_* drive, xterm).
Tests: box_screen_is_exact (real Docker: bot and phone coordinates exact, same spot twice in
under 2 s, ASCII + Chinese + named key typed), Calculator by numbers and Terminal typing under
注音 through the signed app, remote_e2e 28/28, Linux build + 151 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
remote_e2e.py: with E2E_SCREENSHOT=1 a screenshot of the hands comes back to the model as a
JPEG; with E2E_TYPE=1 also a click, typing (Chinese too) and Enter in a terminal there. The fake
model no longer takes a screenshot's picture message for the person speaking. E2E_APP picks the
app to open. deploy/test/wayland: the same test on a headless sway in Docker whose
XDG_RUNTIME_DIR is full (31/31).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The screenshot buffer was a sparse file in XDG_RUNTIME_DIR. With that folder full
(a 1.6 GB Hyprland log), Hyprland wrote the pixels into pages that couldn't exist
and died with SIGBUS in CScreenshareFrame::copyShm. Now the buffer is a memfd with
every page reserved up front; with no room the screenshot fails in AnyBot instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
王性驊