The quick model learned what its last click did only by re-reading two long
numbered lists. A multiset diff of (kind, name) between consecutive looks adds
two lines and answers it directly — including the case that matters most, a
dialog that appeared entirely.
The bench's precision failures (quick model clicking the wrong number, Retina
scaling, covered windows) all come from acting on pixels. A plain left click
now asks the frontmost app's accessibility tree first — by name, or by the
numbered spot — and typing writes AXValue into the focused field once,
keyboard events only when accessibility can't reach. Any app with a thin tree
behaves exactly as before: every AX answer that isn't a clear yes falls
through to CGEvent.
macOS code type-checked with --target x86_64-apple-darwin; gate: build on the
macos-arm64 runner + bench tasks T2/T5 (clicking by name) on a Mac.
Linux/Windows: stubs, pixel path unchanged.
The model picks action=screenshot/click/typing/… on a single `computer` tool
(the shape Anthropic's and OpenAI's computer-use models are trained on):
choosing among twelve near-identical names was its own error source, and the
twelve schemas were the biggest stable part of every request. Calls expand to
the internal per-action tools at the model-output boundary, so approvals,
repeat guards, skill replay, the remote limb protocol and the work log keep
their existing names — old limbs need no update.
A computer-use turn sends the same system + tool schemas every step, and the
history is append-only between steps: three breakpoints (system, last tool,
last content block) let steps 2..n of a turn read the cache the first step
wrote. CACHE_STATS counts input/cache_read/cache_creation tokens for make
doctor and the bench (target: cache_read > 70% of input on Claude routes).
Follow-ups in the plan (same item): freeze facts into the request at turn
start, keep the quick-model tool list constant per step (stable prefixes).
The person wants the local box and Lima VM off whenever this AnyBot is set
to show another brain, not stopped after two idle minutes.
- sleep_on_a_screen has no idle wait and also runs right at start, before
the loop's first 20-second pause; a resume of an interrupted rebuild is
done on brains only (it would start the box on a screen).
- gvisor-vm.sh no longer registers the VM to start with the Mac: AnyBot
starts it itself when it is the brain and wants its computer.
Verified on this Mac: VM started by hand while AnyBot was closed; the new
AnyBot (a screen of the cloud brain) stopped it 4 seconds after launch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
The Mac mini shows the cloud brain, whose bots work on the brain's own
computer and borrow only this Mac's screen, browser and apps. Its local
box still ran, and the Lima VM under it had grown to its whole 6 GB: the
Mac never gets back memory a VM once used.
- On a screen the box stops after 2 idle minutes, and the Lima VM too when
nothing else runs in it. probe() no longer starts the VM there, and the
volume holder and scheduled updates run on brains only. Becoming a brain
again starts both (probe/ensure_running already wake the VM).
- Screen mode is decided from the settings before the box loop's first
look, which used to start the VM at launch.
- gvisor-vm.sh: 4 GB instead of 6 (box capped at 2 GB), and the VM's
ModemManager, udisks2 and multipathd are turned off.
Verified on this Mac: box and VM stopped by themselves 2 minutes after the
app started; the VM process (4.4 GB plus compressed pages) is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- quick_tools indexed serde_json Values mutably, which inserted
actions.items.properties = null into click/typing/scroll; xAI refused
the whole tool list, so since ba1fce5 every step fell back to the main
model. It now uses get_mut and the test checks nothing but ref changes.
- The front app's menu bar titles and any open menu (submenus, right-click
menus) are listed with numbers and their shortcuts; an option-key
alternate in the same place is noted on the visible item.
- Disclosure triangles, colour wells and date fields get numbers; an
icon-only control without a name goes by its description.
- When the front window can't be read the list says so (with the reason)
instead of 「」 above the menu bar alone.
- Computer facts add each system's conventions (Mac: cmd+shift+G, save
panel, default format, /bin/sh echo -n; Linux: GTK ctrl+L).
- `anybot --help` and unknown -- flags print the usage and exit instead of
opening the window on the real data folder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- Keys and typing into no field go to whatever is in front: a cmd+w meant for TextEdit reached
the person's terminal after the TextEdit window had closed (its "close this window?" came up
and was cancelled). They are now refused, with why, when the app in front isn't the one of the
latest look.
- The quick model's screen tools have no `ref`: looking numbers up is where it pressed the wrong
keys; it names controls (two of a name come back as trouble for the bot's own model).
- Instructions: macOS permission and security prompts are for the person to answer (a bot
pressed 「不允許」 on one), and check what's in front before a shortcut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- Two identical clicks in one step opened the Mac's Wi‑Fi menu and closed it again: the second is
skipped with a word on why (count: 2 for a double click).
- macOS Calculator had no focused window for a moment after 「乘」 and the list came back with
only the menu bar: an empty or failed read is tried once more 0.3 s later.
- Benchmark notes: three more unseen apps on box v9 (16/16, GTK 2 apps 4-10x faster with gail)
and the Mac mini run with Screen Recording (zoom, Wi‑Fi menu, Calculator checked by eye).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- Box v9: GTK_MODULES=gail:atk-bridge. lxappearance and PCManFM (GTK 2) offered AT-SPI no
controls at all (0 numbers); with gail they offer 37 and 17.
- The quick model ending a turn without acting goes to the bot's own model every time, not only
after changes: one stopped after a single look to ask 「要我繼續嗎？」.
- Instructions: finish what was asked; put an unexpected screen right (close what's in the way,
open the app) instead of stopping to ask, except for what the person must do or confirm.
- Benchmark: three more held-out tasks (lxappearance theme, PCManFM folder, Mousepad
find-and-replace), checked by their results; every task starts with those apps closed too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- The one-line-field check got the step's name as 「#14（輸入框…）」 on the box and never
matched, so the select-and-paste never ran there and addresses still came out mangled. It
reads both forms now, with a test on the script the box runs.
- Trouble twice in a turn and the rest of the turn is the bot's own model's (a quick model
guessed its way through Mousepad's save dialog for 79 steps).
- A name that isn't on the screen comes back with the names close to it ([12]「New Window」…),
so the step is put right at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Typed key by key into Chromium's address bar right after it opened, the page load and the
autocomplete took turns with the keys and the address came out mangled (chrome://blankchrome//…,
colons gone) — even after selecting the old text first. A one-line field (one typing replaces)
now gets its text pasted in one go, with a short pause before return; terminals and text areas
are still typed key by key. Reproduced and checked on the box's Chromium.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- Typed into the middle of Chromium's 「about:blank」, an address came out as
chrome://blankchrome//settings/search, and every step after it went wrong. Typing into a
field by name or number now selects a one-line field's text first (ctrl+a / cmd+a) so the
text replaces it; append=true keeps it. Text areas are left alone.
- A model sending name plus x: 0, y: 0 had its click land in the screen's corner: a name (or
number) now always wins over coordinates.
- After the VM went down hard, the box's first start failed in gVisor ("mount source already
has a filestore file … repeated submounts are not supported") and the second worked: waking
or restarting the box tries once more on exactly that error.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
A quick model looking numbers up pressed 1, 2, 3, 1 for 1234, and 1234 × 5678 = didn't fit in
an 8-action batch (it split it and lost a key). The click and batch descriptions now lead with
the control's name (name "4"), and a batch takes up to 16 actions.
The benchmark gets four tasks on things none of the others use (Mousepad and its save dialog,
a Chromium bookmark, ImageMagick, a page of nested web components), run only when asked for
(--tasks N1,…): a check that the changes carry over to what they weren't tuned on.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
`anybot --ask` with ANYBOT_ASK_HERE=1 offers this computer too (headless runs leave it out), so
screen and menu-bar steps can be tried on a Mac without the window. Tried on a Mac mini: the
menu bar's icons listed in one step, a status icon clicked by its name, AppleScript reading the
front app, Tailscale's own CLI answering — all without a screenshot.
An App Store app's CLI joined to `> file` stays in the sandbox and dies; the note now says so
(a bot blamed --json for it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
One model call per step now: a replay step needs none; the quick model's step falls back to the
bot's model when it can't be used (but a too-long chat is folded and asked again, whichever model
asked); each attempt is in the work log. A zoom is a look (safe to do again after a crash).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
bench/computer-use/run.py runs each task through `anybot --ask` with a fresh chat, checks the
result inside the box, and records success, steps, time split between model and tools, route
switches and a per-step transcript. It refuses to run against the real box (needs its own
ANYBOT_BOX_NAME / ANYBOT_BOX_VOLUME and a scratch data folder). --keep-skills carries learned
skills into the next run to measure replay. README has the tasks, how to run, and the
2026-11-09 numbers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Two copies on one folder each keep their own idea of the chats and overwrite each other's saves,
and a second copy starting up would take the first one's open turns in the work log for a crash
and close them. The window, `--serve` and `--ask` now hold an OS file lock on anybot.lock in the
data folder (flock; LockFileEx on Windows), and `anybot setup` holds it while it writes the
settings, after stopping the service and before starting it again.
The lock goes with the process however it ends, so a copy that crashed or was killed never blocks
the next one. A new copy waits up to 5 s for one that is closing (a relaunch, a service restart).
Past that, a second window shows a small notice naming the holder's process and the folder, and
`--serve` / `--ask` say it on stderr and exit 1 (launchd's and systemd's restart then retry). Each
ANYBOT_DATA_DIR has its own lock; a screenshot run doesn't take it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
agent-desktop knows an app by its program's name, but the X11 window says its class: a Snap
Firefox is "firefox_firefox", so the front app was never found and the bot got a bare picture
(「看不出最前面是哪個 App（WINDOW_NOT_FOUND）」). The window's process now gives the name.
When there are no controls at all and the desktop's accessibility switch (org.a11y.Status
IsEnabled) is off — most apps then show none — AnyBot turns it on, as a screen reader would, and
tells the bot that apps already open show their numbers after a restart.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Firefox dropped Chrome's DevTools protocol, so bidi.rs speaks WebDriver BiDi and does each
DevTools command the browser tools use the BiDi way (real mouse and keys, typing, screenshots,
PDF, dialogs, uploads, downloads, tabs). The 18 tools themselves are unchanged and pass the same
dashboard test on Firefox as on Chrome.
- A computer with only Firefox runs AnyBot's own profiles in it (under the Snap/Flatpak corner
of the home folder when Firefox is packaged that way).
- The person's own Firefox profiles are choices too, by the names Firefox shows (profiles.ini
and the newer profile switcher's SQLite list): 「阿金（Firefox）」, or just 「阿金」. Firefox
can't be taken over once open, so a profile open the everyday way is refused with "close it
first"; AnyBot then opens it itself, logins intact.
- Settings lists them without a remove button; the messages name Firefox as an option.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
setup.sh offers 「區網 IP」 (alone, or next to Tailscale): Caddy serves https://<LAN IP> with its
own small certificate authority (tls internal, default_sni for clients that send no name) and
hands out the root at http://<LAN IP>/anybot-ca.crt for each device to trust once.
docker-compose.lan.yml binds Caddy to that address only, so it sits beside tailscale serve.
The app now accepts, for the account server, the certificates this computer trusts as well as
the built-in public ones (HTTP and the WebSocket alike), so it connects once the root is
trusted; before that it says what to install instead of "the certificate is wrong".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Every warning `cargo clippy --all-targets` gave (50 on macOS, 8 more only on
Linux, 1 in the relay), fixed without changing what anything does:
- long types get a name (Peer, Members, Imported, Joining, Mention, Stream…);
- the two big enum variants are boxed (the vault's login plan, joining an
account);
- a test now waits for the copy it relaunched (it left a zombie behind);
- a hint's format had a dead `if` whose two branches were both "";
- the rest is what clippy suggests: sort_by_key, as_chunks, is_none_or,
is_multiple_of, while let, from_ref, next_back, struct init, no casts to
the same type.
Checked: cargo test (177 macOS, 181 Linux), relay tests (11), remote_e2e
28/28, and the live Linux test (X11; Wayland clicks, keys, Chinese typing,
scroll and screenshots).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
A new mobile/index.html in the desktop's (and Grok Bot's) shape: the sidebar
with its sections and folders, the chat with steps, thinking, reactions, quotes
and the approval card, 對話詳細資訊, Bot 設定, routines, pages, computers and
設定. Each action goes to the desktop's own code on the app's thread
(api/act/<what> -> App::phone_act): appctl, set_pinned, restart_bot,
toggle_reaction, box_task, channels_restart and the rest. No new rules.
Keys, channel tokens and the vault never reach the phone; a model key can only
be written. Admins keep what the phone always let them do; making, changing or
deleting bots, routines, memories, pages, folders and settings stays the
owner's. The state the phone polls now carries the sidebar's section order.
Checked by using it on a preview brain: routines, memories, templates, folders,
keys, signing another phone out, approvals and a chat with a fake model.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
The phone showed a bare 「刪除技能」 when 新 Bot deleted the test skill; save/use/delete skill,
remember and forget now carry the name or the fact.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
daniel wang
AnyBot Dev