`anybot --ask` with ANYBOT_ASK_HERE=1 offers this computer too (headless runs leave it out), so
screen and menu-bar steps can be tried on a Mac without the window. Tried on a Mac mini: the
menu bar's icons listed in one step, a status icon clicked by its name, AppleScript reading the
front app, Tailscale's own CLI answering — all without a screenshot.
An App Store app's CLI joined to `> file` stays in the sandbox and dies; the note now says so
(a bot blamed --json for it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
Every warning `cargo clippy --all-targets` gave (50 on macOS, 8 more only on
Linux, 1 in the relay), fixed without changing what anything does:
- long types get a name (Peer, Members, Imported, Joining, Mention, Stream…);
- the two big enum variants are boxed (the vault's login plan, joining an
account);
- a test now waits for the copy it relaunched (it left a zombie behind);
- a hint's format had a dead `if` whose two branches were both "";
- the rest is what clippy suggests: sort_by_key, as_chunks, is_none_or,
is_multiple_of, while let, from_ref, next_back, struct init, no casts to
the same type.
Checked: cargo test (177 macOS, 181 Linux), relay tests (11), remote_e2e
28/28, and the live Linux test (X11; Wayland clicks, keys, Chinese typing,
scroll and screenshots).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
From looking at the real phone page after the fixes: a long command squeezed its step's name into
one character per line (執/行/指/令), and sending a file showed as 「工具」. The name keeps its width,
the detail is cut with an ellipsis, and send_file, react and view_demo_step have names; the window
says 「正在傳檔案給你」 instead of 「正在記下來」 while a file is sent.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
Unused (found by taking away the dead-code allowances; nothing else used them, tests aside):
computer::specs, quiet_button, the never-made Poke::Spin, ten icons and their SVGs, the
theme's chrome/bubble_agent colours, ImportRow.line, voice's multipart/transcribe/speak
wrappers, and the old keychain writers vault::save/remove/index_add with their live test
(the vault store replaced them; reading old keychain entries for the import stays).
One copy each: percent-encoding (httpd::url_encode, was four), tool arguments (tools::args_of,
was three), the WebSocket handshake (httpd::accept_websocket, was three), the read pacing
(wire::Pace, was two: relay and app), docker with input and a time limit (dk_with_input, was
two). Docs: FEATURES §看畫面 (VNC), protocol (v* frames, /screen, tickets), architecture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- hand_over_computer (the original's box hand-off card): instruction, live screen with 接管,
跳過 / 我完成了 / 接管; 我完成了 also in the viewer; a typed message answers it too. Header,
sidebar and notifications say 等你處理 / 「{name}：需要你」; the phone shows it with 接管.
- Mac: the content reaches the top (no grey title bar); the top strip drags the window.
- Panel 電腦 card: each line stops before the picture; the brain's name on its own line.
- Viewer: buttons first, the title ends in … when narrow; the picture is centred.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A bot's own computer is 「<bot> 的電腦」; this one and the account's others go by their names.
The server brain picked 你的電腦 by sorted host id, so a desktop left on the server won.
- 「我的電腦」 is the computer the person typed on (the relay names it); unknown means ask.
- A step moved back to the chosen computer tells the bot so.
- Composer chip and bot settings choose the computer by name; a screen asks the brain.
- A screen's 電腦 card and viewer show the brain's computer, not this one's box.
- A screen's chat copy that falls short asks for everything again instead of appending;
清除聊天記錄 on a screen clears the brain's chat.
- Model requests to a private address go through the person's computer first.
- Phone typing on the box sends the text the tool reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- 看圖小幫手: when a bot's model can't take pictures, a model of the
person's that can (Grok, Claude, GPT, Gemini… whichever has a key)
describes each picture the person sent, text and numbers included, and
the bot gets that description. Remembered per picture; helpers that
fail are skipped for ten minutes.
- On the Mac, commands still can't read AnyBot's data folder (the keys),
but may read the chat's attachments in it (not write them).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
王性驊