AnyBot/docs/en/architecture.md

Architecture (English summary)

The canonical, up-to-date document is docs/architecture.md (Chinese). This is a faithful, condensed English version for contributors who don't read Chinese.

AnyBot is one Rust binary (anybot, egui/eframe UI) with three modes, plus a small account server (anybot-relay):

Mode	Command	Purpose
Window	anybot	desktop app
Headless	anybot --serve	servers, always-on machines, "hands"
Setup	anybot setup …	terminal pairing wizard (src/setup.rs)

Layers

UI (src/app.rs, src/app/*)  — chat, sidebar, settings, computers, file preview, brain mirror
   │                                   │
Turn loop (agent.rs)          Remote (remote.rs, tunnel.rs, wire.rs)
 call model, run tools,                phone API, device pairing,
 approvals, summarising                brain sync, long WebSocket to relay
   │         │
Models      Tools (tools.rs, cloudbox.rs, botbrowser.rs, computer.rs,
(providers) axread.rs/axwrite.rs, macapps.rs, vault*.rs, mcp.rs, memory.rs,
            routines.rs, triggers.rs, channels.rs)

Key modules: agent.rs (one turn: system prompt, streaming, tool loop, approval gates, per-step model routing StepPlan), providers.rs (OpenAI-compatible/Anthropic/OpenCode… + streaming + retries + a vision helper for models that can't see images), policy.rs (allow/ask/deny per action), computer.rs (one computer tool: screenshot, click/type, plus semantic actions menu_click/fill_form/wait_for; macOS reads the real accessibility tree, screenshots are annotated with numbered marks), cloudbox.rs+box/ (the bot's own Docker computer with desktop + browser), vnc.rs (view/take over the bot's computer through the relay), httpd.rs (tiny shared HTTP server), mobile/index.html (phone UI, single embedded file).

Storage

All in the data dir (macOS ~/Library/Application Support/AnyBot/, Linux ~/.local/share/AnyBot/, %APPDATA%\AnyBot\; ANYBOT_DATA_DIR overrides): state.json (bots, chats, settings, keys — written via temp file + rename, never half-written), attachments/, browser/ (bot browser profile), oplog/bot-<id>.jsonl (append-only work log: every step written to disk before it runs; used to repair chats and offer a "continue" card after a crash), image-notes.json, anybot.lock (OS file lock: one AnyBot per data dir).

Security design

• Approval gates (policy.rs): actions run directly, ask first, or are denied. "Direct mode" skips asks — except for the gates below, which have no grant.
• Sensitive clicks (labels containing send/pay/transfer/delete/publish…): asked every time, direct mode included — a harness policy the model cannot negotiate past.
• Command sandbox (macOS): bot commands run under sandbox-exec; AnyBot's data dir, ~/.gnupg, Keychain, ~/.aws are unreadable/unwritable; only attachments/ and .ssh are readable, nothing outside is writable. Model API keys are stripped from the environment. Documented exceptions: App Store app CLIs, ps/top, and a first-time-asked, pattern-filtered single osascript -e.
• The bot's own computer: a Docker container; with gVisor installed on the Docker host, isolation becomes two walls (virtual kernel + Chromium sandbox). On macOS a Lima VM running Docker+gVisor is provided (deploy/mac/gvisor-vm.sh).
• Vault: passwords, card numbers, CVCs never enter the model; AnyBot fills them into web forms directly.
• Screenshots are painted before the model sees them: card numbers and OTP boxes are masked black (marks::mask_secrets).
• Everything read from outside (screens, web pages, files) arrives wrapped in <untrusted source="…">; the system prompt makes it data, never orders. The real wall is the harness gates, not the wording.
• Relay: stores only hashes (argon2id; legacy PBKDF2 still works, upgraded on next password change); optional 2FA (TOTP + one-time recovery codes); one-page logout for all devices; login lockout per IP+account (8/5 min).
• Egress: --serve binds loopback + detected Docker bridge gateways only, never 0.0.0.0 (src/egress.rs).
• docker.sock is not mounted into the bot's container: a whitelisting docker-socket-proxy sidecar is used instead; rootless Docker works via DOCKER_SOCKET.

The computer-use control chain

Model gets a screenshot + numbered marks built from the accessibility tree (macOS) → clicks a number, or names it (press_named) → harness translates to an AX press (preferred) or a pixel click (fallback) → the diff of the control tree between looks (marks::diff) is appended so the model sees only what changed. bench/computer-use/ measures tasks end-to-end (model inference is ~90% of wall time) and includes the R1–R7 red-team suite for the gates above.