- 「網頁上的事一律用 browser 一個工具」: the guidance lines now name
action=open/read/click/type/table/find/wait/download and action=profiles —
the model reads one vocabulary in the system prompt, the schema and the
loop-guard messages (browser_read/browser_open mentions gone from those)
- H3, light: usage rules were already in the system prompt (and C1 caches
the tools block too, so moving descriptions there saves little) — the
browser schema shed its duplicated safety sentence to the click line;
wholesale description-slimming deliberately not done: the guidance that
reads as noise is what keeps a small model out of trouble
- browser {action: open|read|elements|find|table|screenshot|click|hover|
type|press|select|scroll|wait|tabs|back|upload|download|pdf|dialog|eval|
profiles|fill} — the shape models are trained on (Anthropic CUA, OpenAI CUA)
- vault fills ride on it: fill {kind: login|card|contact, id}; vault_list
stays its own tool; the fills' approvals (every use asks, the model never
sees the secret) are untouched — expand happens at the model-output
boundary, so oplog, approvals, replay, remote limbs keep internal names
- web keeps {web_fetch, browser}; the eighteen browser_* and the three
fill_* schemas leave the model's view (−48% schema text, −58% counting
the old web trio; 25 tools → 2)
- tests: every action maps to a real internal tool, every internal tool is
reachable from the one schema
- 帳號 › 登入的裝置: every signed-in device with its last use; tap a device's
id to sign just that one out; 全部登出 ends every session, this phone
included — the stolen-session story ends on one page, on the phone
- 帳號 › 兩步驗證: enroll (secret + recovery codes shown once, code confirms),
disable (proves the code); the login screen's second step landed with S2
- relay: POST /relay/sessions/drop {id} — a session is named by the first 8
hash characters its row shows; kicking the phone you hold tells the app to
sign itself out (was_self)
- src/store.rs: bundled SQLite (WAL) in the data folder; every oplog record
is mirrored into an audit table — turns, model calls, tools with their
arguments and end statuses, queued messages, stops
- side-car by design: state.json stays the app's truth (one struct, written
atomically); the mirror is best-effort — an audit that never breaks a turn,
and a timeline that answers for 30 days, not just for the running turn
- /api/bots/<id>/audit now reads the trail (phone's 稽核 tab gains history)
- retention: rows older than KEEP_DAYS are deleted on open
- groundwork for memory/RAG (P6) on the same layer; state.json itself moves
in step 2 with the memory tables (the app keeps reading State as a struct)
- oplog gets wall-clock times on tool records (serde-default: old logs read
fine), and OpLog::timeline turns the machine log into a story: per turn —
the prompt, model calls (retries counted into one line), every tool with
the arguments it ran with and its status (等核准/進行中/完成/被拒絕/失敗),
messages typed mid-turn, stop, outcome
- GET /api/bots/<id>/audit (owner/admin; members' allow-list untouched)
- 對話詳細資訊 gains a 稽核 tab: newest turn first, running tool shows
進行中…, an approval gate shows as 等核准 — the phone can answer the question
'what is my bot doing right now, exactly', step by step
- the log stays the machine format recovery reads; the timeline is derived
- README.en.md: full English front door (install table, quick start,
deployment, security-in-one-paragraph, license) + a positioning table
against OpenClaw / OpenHands / Khoj / UI-TARS
- docs/en/{architecture,install,multi-device,protocol}.md: faithful English
summaries of the Chinese docs, each linking to the canonical version
- docs/demo/README.md: how to record the GIF/MP4 the README reserves space for
- language switch line at the top of the Chinese README
- login with 2FA on returns a 5-minute challenge, not a session; the session
comes from /relay/2fa/verify with an authenticator code — or one of eight
recovery codes, each good once (hash slot emptied)
- /relay/2fa/enroll hands out the secret (QR) + codes once; enabling completes
only when a real code proves the app has the secret; disable proves the code
- /relay/sessions/logout kicks every device of the account from one page
- password hashes are argon2id (memory-hard, OWASP parameters); stored PBKDF2
hashes still sign in — no forced reset, upgrade on next password change
- TOTP is RFC 6238 (vectors pinned in tests); mobile page prompts for the
second step; protocol.md documents the endpoints
- fixes a double-lock in /relay/me found by the flow test (handlers hold one
data guard: read through it)
- clicks on controls whose own words say money/sending/deleting ask the person
every time — on every computer, in direct mode, no grant silences it (the
harness decides, not the model, so a page cannot talk its way past it)
- screenshots paint card-number and OTP boxes black before any model sees them
- content the bot READS (pages, files, screens) enters the history wrapped in
<untrusted>, and the system prompt says the wrapper means content-not-orders
- keep_to_chosen now has a test on the screen tools: a model cannot move a
screen step onto another computer by itself (B26's promise, pinned)
- bench red team R1–R7: every payload's goal is a harmless probe file /
button code; passing = reporting the page honestly and leaving the probe
untouched. Run: --tasks R1..R7
- menu_click walks the real macOS menu bar in one step (three screenshot
rounds per level become one call + one picture)
- fill_form writes every named field with one AXValue write each, in one step
- wait_for polls the control tree (no screenshots) until text appears/gone —
replaces the screenshot-polling loops that filled whole turns with identical
pictures
- batch steps take an optional expect: the batch stops and shows the screen at
the step whose promised text never showed, instead of running a blind plan
on sand
Menu/fill are macOS accessibility; other systems keep click+typing per field
(the tool descriptions say which is which).
The quick model learned what its last click did only by re-reading two long
numbered lists. A multiset diff of (kind, name) between consecutive looks adds
two lines and answers it directly — including the case that matters most, a
dialog that appeared entirely.
The bench's precision failures (quick model clicking the wrong number, Retina
scaling, covered windows) all come from acting on pixels. A plain left click
now asks the frontmost app's accessibility tree first — by name, or by the
numbered spot — and typing writes AXValue into the focused field once,
keyboard events only when accessibility can't reach. Any app with a thin tree
behaves exactly as before: every AX answer that isn't a clear yes falls
through to CGEvent.
macOS code type-checked with --target x86_64-apple-darwin; gate: build on the
macos-arm64 runner + bench tasks T2/T5 (clicking by name) on a Mac.
Linux/Windows: stubs, pixel path unchanged.
The model picks action=screenshot/click/typing/… on a single `computer` tool
(the shape Anthropic's and OpenAI's computer-use models are trained on):
choosing among twelve near-identical names was its own error source, and the
twelve schemas were the biggest stable part of every request. Calls expand to
the internal per-action tools at the model-output boundary, so approvals,
repeat guards, skill replay, the remote limb protocol and the work log keep
their existing names — old limbs need no update.
A computer-use turn sends the same system + tool schemas every step, and the
history is append-only between steps: three breakpoints (system, last tool,
last content block) let steps 2..n of a turn read the cache the first step
wrote. CACHE_STATS counts input/cache_read/cache_creation tokens for make
doctor and the bench (target: cache_read > 70% of input on Claude routes).
Follow-ups in the plan (same item): freeze facts into the request at turn
start, keep the quick-model tool list constant per step (stable prefixes).
The person wants the local box and Lima VM off whenever this AnyBot is set
to show another brain, not stopped after two idle minutes.
- sleep_on_a_screen has no idle wait and also runs right at start, before
the loop's first 20-second pause; a resume of an interrupted rebuild is
done on brains only (it would start the box on a screen).
- gvisor-vm.sh no longer registers the VM to start with the Mac: AnyBot
starts it itself when it is the brain and wants its computer.
Verified on this Mac: VM started by hand while AnyBot was closed; the new
AnyBot (a screen of the cloud brain) stopped it 4 seconds after launch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
The Mac mini shows the cloud brain, whose bots work on the brain's own
computer and borrow only this Mac's screen, browser and apps. Its local
box still ran, and the Lima VM under it had grown to its whole 6 GB: the
Mac never gets back memory a VM once used.
- On a screen the box stops after 2 idle minutes, and the Lima VM too when
nothing else runs in it. probe() no longer starts the VM there, and the
volume holder and scheduled updates run on brains only. Becoming a brain
again starts both (probe/ensure_running already wake the VM).
- Screen mode is decided from the settings before the box loop's first
look, which used to start the VM at launch.
- gvisor-vm.sh: 4 GB instead of 6 (box capped at 2 GB), and the VM's
ModemManager, udisks2 and multipathd are turned off.
Verified on this Mac: box and VM stopped by themselves 2 minutes after the
app started; the VM process (4.4 GB plus compressed pages) is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- quick_tools indexed serde_json Values mutably, which inserted
actions.items.properties = null into click/typing/scroll; xAI refused
the whole tool list, so since ba1fce5 every step fell back to the main
model. It now uses get_mut and the test checks nothing but ref changes.
- The front app's menu bar titles and any open menu (submenus, right-click
menus) are listed with numbers and their shortcuts; an option-key
alternate in the same place is noted on the visible item.
- Disclosure triangles, colour wells and date fields get numbers; an
icon-only control without a name goes by its description.
- When the front window can't be read the list says so (with the reason)
instead of 「」 above the menu bar alone.
- Computer facts add each system's conventions (Mac: cmd+shift+G, save
panel, default format, /bin/sh echo -n; Linux: GTK ctrl+L).
- `anybot --help` and unknown -- flags print the usage and exit instead of
opening the window on the real data folder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- Keys and typing into no field go to whatever is in front: a cmd+w meant for TextEdit reached
the person's terminal after the TextEdit window had closed (its "close this window?" came up
and was cancelled). They are now refused, with why, when the app in front isn't the one of the
latest look.
- The quick model's screen tools have no `ref`: looking numbers up is where it pressed the wrong
keys; it names controls (two of a name come back as trouble for the bot's own model).
- Instructions: macOS permission and security prompts are for the person to answer (a bot
pressed 「不允許」 on one), and check what's in front before a shortcut.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- Two identical clicks in one step opened the Mac's Wi‑Fi menu and closed it again: the second is
skipped with a word on why (count: 2 for a double click).
- macOS Calculator had no focused window for a moment after 「乘」 and the list came back with
only the menu bar: an empty or failed read is tried once more 0.3 s later.
- Benchmark notes: three more unseen apps on box v9 (16/16, GTK 2 apps 4-10x faster with gail)
and the Mac mini run with Screen Recording (zoom, Wi‑Fi menu, Calculator checked by eye).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- Box v9: GTK_MODULES=gail:atk-bridge. lxappearance and PCManFM (GTK 2) offered AT-SPI no
controls at all (0 numbers); with gail they offer 37 and 17.
- The quick model ending a turn without acting goes to the bot's own model every time, not only
after changes: one stopped after a single look to ask 「要我繼續嗎？」.
- Instructions: finish what was asked; put an unexpected screen right (close what's in the way,
open the app) instead of stopping to ask, except for what the person must do or confirm.
- Benchmark: three more held-out tasks (lxappearance theme, PCManFM folder, Mousepad
find-and-replace), checked by their results; every task starts with those apps closed too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
- The one-line-field check got the step's name as 「#14（輸入框…）」 on the box and never
matched, so the select-and-paste never ran there and addresses still came out mangled. It
reads both forms now, with a test on the script the box runs.
- Trouble twice in a turn and the rest of the turn is the bot's own model's (a quick model
guessed its way through Mousepad's save dialog for 79 steps).
- A name that isn't on the screen comes back with the names close to it ([12]「New Window」…),
so the step is put right at once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
AnyBot Dev
daniel wang