AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
// The work loop: ask the model, run the tools it asks for (asking the person first when the
// rules say so), give it the results, and repeat until it answers in plain words.
// Long chats are folded into a summary before they outgrow the model's context.
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
use base64 ::Engine ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
use std ::sync ::atomic ::{ AtomicBool , Ordering } ;
use std ::sync ::mpsc ::{ channel , Sender } ;
use std ::sync ::{ Arc , Mutex } ;
use std ::time ::Duration ;
use crate ::appctl ;
use crate ::computer ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
use crate ::model ::{ Message , Role , ToolCall , ToolStatus } ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
use crate ::providers ::{ self , AppCall , Approval , ApprovalAnswer , ApprovalKind , ChatRequest , StreamEvent } ;
use crate ::tools ::{ self , Decision } ;
use crate ::web ;
/// Context size assumed when the person hasn't set one (estimated).
pub const DEFAULT_CONTEXT_TOKENS : u64 = 128_000 ;
/// Summarize once the chat fills this share of the context:
2026-11-09 08:37:33 +00:00
/// `client_speculative_summarization_config.tokenUsageThresholdPercentage:70`.
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
pub const COMPACT_THRESHOLD_PERCENT : u64 = 70 ;
/// Extra room allowed in the middle of a task before summarizing again:
/// same config, `tolerancePercentage:5`.
pub const COMPACT_TOLERANCE_PERCENT : u64 = 5 ;
2026-11-09 08:37:33 +00:00
/// Most model calls in one turn: a guard against a runaway turn, not a budget. the reference assistant leaves
/// its turn window open `cloud_agent_stream_conversation_turn_window`
2026-11-06 08:51:25 +00:00
/// `fallbackValues:{maxTurns:null}`) and browser-use stops at 100; 25 cut off real work (fixing
/// a server's Docker took 50+ commands). Going in circles is stopped long before this by step_key.
pub const MAX_STEPS : usize = 100 ;
/// Every this many steps the bot is asked whether it is getting anywhere (estimated).
pub const CHECK_IN_STEPS : usize = 25 ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
/// Newest messages always kept word for word (estimated).
pub const KEEP_RECENT_MESSAGES : usize = 6 ;
/// Characters of each old message given to the summarizer (estimated).
const SUMMARY_SNIPPET_CHARS : usize = 2_000 ;
pub struct Job {
pub req : ChatRequest ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// The quick model that carries out planned steps (same provider), or None: every step uses
/// `req.model`. See `step_plan`.
pub step_model : Option < String > ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
/// Computers this bot may work on (empty when computer tools are off).
pub computers : Vec < tools ::Target > ,
pub default_computer : String ,
2026-11-05 08:47:28 +00:00
/// Which computer is the bot's own, which one the person is using, and whether this is a server.
pub roles : tools ::Roles ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
/// 在這台電腦上執行 is on.
pub local_execution : bool ,
/// This bot's computer tools are on (when off, using them asks the person first).
pub tools_enabled : bool ,
/// 直接操作 mode; the window can switch it on mid-turn ("全部允許").
pub direct : Arc < AtomicBool > ,
/// 在這台電腦上執行 is locked off by an admin policy: refuse, don't ask.
pub local_locked : bool ,
pub grants : Vec < String > ,
pub context_tokens : u64 ,
pub via : String ,
/// The bot's name, for the pop-up text.
pub via_name : String ,
/// Messages the person sends while the turn runs (they steer it; no new turn).
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
pub steer : Arc < Mutex < Vec < Queued > > > ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
/// 自動審核 rules, when on.
pub review : Option < Review > ,
/// 示範教學 this bot has learned (for view_demo_step).
pub demos : Vec < crate ::teach ::Demo > ,
2026-11-06 15:12:53 +00:00
/// The skills there are (a longer job is looked back on against them; see learn).
pub skills : Vec < crate ::memory ::Skill > ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// This turn's id: its lines in the bot's work log carry it.
pub op : String ,
/// The bot's work log (see oplog.rs); None where nothing is kept (tests, the demo).
pub log : Option < crate ::oplog ::OpLog > ,
}
impl Job {
/// Write a line to the work log (on disk before this returns).
fn note ( & self , r : crate ::oplog ::Record ) {
if let Some ( l ) = & self . log {
l . append ( & r ) ;
}
}
}
/// A message the person typed while the bot works: its words, when it was typed, and the id its
/// work-log lines carry (so one typed just before a crash isn't lost).
#[ derive(Clone, Debug) ]
pub struct Queued {
pub id : String ,
pub text : String ,
pub at : i64 ,
}
impl Queued {
pub fn new ( text : impl Into < String > ) -> Self {
Queued { id : crate ::oplog ::new_id ( ) , text : text . into ( ) , at : crate ::model ::now_ms ( ) }
}
}
/// A turn's last line in the work log, written once: explicitly (before Done / Error reach the
/// window), else when the turn's thread ends any other way — stopped (aborted), or a panic
/// (failed).
struct Closing {
log : Option < crate ::oplog ::OpLog > ,
op : String ,
cancel : Arc < AtomicBool > ,
written : bool ,
}
impl Closing {
fn close ( & mut self , outcome : crate ::oplog ::Outcome ) {
if ! self . written {
self . written = true ;
if let Some ( l ) = & self . log {
l . finish ( & self . op , outcome ) ;
}
}
}
}
impl Drop for Closing {
fn drop ( & mut self ) {
let o = if self . cancel . load ( Ordering ::Relaxed ) { crate ::oplog ::Outcome ::Aborted } else { crate ::oplog ::Outcome ::Failed } ;
self . close ( o ) ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
#[ derive(Clone, Default) ]
pub struct Review {
/// 「無需詢問直接允許」規則
pub allow : Vec < String > ,
/// 「一律詢問」規則 (wins when both match)
pub ask : Vec < String > ,
}
pub enum ReviewVerdict {
Allow ( String ) ,
Ask ( String ) ,
NoMatch ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// The reviewer didn't answer, or answered something that isn't a verdict: nobody checked the
/// rules, so this never counts as "no rule matched" (a check that breaks, blocks).
Failed ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// Said in the question when the reviewer couldn't check a step.
pub const REVIEW_FAILED_NOTE : & str = " (自動審核沒有回應,所以先問你) " ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
/// Ask the model whether an action matches the person's rules. "Ask first" wins on conflict
/// (NFRGrQ: 規則衝突時,「先詢問」優先); built-in safety checks have already run before this.
fn review_action ( job : & Job , review : & Review , action : & str , cancel : & AtomicBool ) -> ReviewVerdict {
if review . allow . is_empty ( ) & & review . ask . is_empty ( ) {
return ReviewVerdict ::NoMatch ;
}
let list = | v : & [ String ] | if v . is_empty ( ) { " (無) " . to_string ( ) } else { v . iter ( ) . enumerate ( ) . map ( | ( i , r ) | format! ( " {} . {r} " , i + 1 ) ) . collect ::< Vec < _ > > ( ) . join ( " \n " ) } ;
let req = ChatRequest {
system : " 你是動作審核員。只根據使用者寫的規則判斷,不要自己加規則。只回一行 JSON: { \" verdict \" : \" ask \" | \" allow \" | \" none \" , \" rule \" : \" 符合的規則原文 \" }。如果符合任何「一律詢問」規則就回 ask( 優先) ; 否則符合「無需詢問直接允許」規則就回 allow; 都不符合回 none。 " . into ( ) ,
history : vec ! [ Message ::user ( & format! ( " 要審核的動作: {action} \n \n 「一律詢問」規則: \n {} \n \n 「無需詢問直接允許」規則: \n {} " , list ( & review . ask ) , list ( & review . allow ) ) ) ] ,
tools : vec ! [ ] ,
.. job . req . clone ( )
} ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let Ok ( out ) = providers ::run_step ( & req , cancel , & | _ | { } ) else { return ReviewVerdict ::Failed } ;
parse_verdict ( & out . text )
}
/// The reviewer's one line of JSON. Anything but a clear ask / allow / none is a failed review.
fn parse_verdict ( text : & str ) -> ReviewVerdict {
let text = text . trim ( ) ;
2026-11-05 07:48:20 +00:00
// A truncated answer can put the last "}" before the first "{" — slicing then panics.
let json_part = text . find ( '{' ) . and_then ( | a | text . rfind ( '}' ) . filter ( | b | * b > = a ) . map ( | b | & text [ a ..= b ] ) ) . unwrap_or ( text ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let v : serde_json ::Value = serde_json ::from_str ( json_part ) . unwrap_or_default ( ) ;
let rule = v [ " rule " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) ;
match v [ " verdict " ] . as_str ( ) {
Some ( " ask " ) = > ReviewVerdict ::Ask ( rule ) ,
Some ( " allow " ) = > ReviewVerdict ::Allow ( rule ) ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
Some ( " none " ) = > ReviewVerdict ::NoMatch ,
_ = > ReviewVerdict ::Failed ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
}
2026-11-06 15:12:53 +00:00
/// 自己進化: after a longer job, the model looks back on its steps and keeps the way that worked
/// as a skill (or improves the one it followed), so the next time takes the short path. It runs
/// after Done: the person isn't kept waiting.
fn learn ( job : & Job , history : & [ Message ] , cancel : & AtomicBool , send : & dyn Fn ( StreamEvent ) ) {
let Some ( start ) = history . iter ( ) . rposition ( | m | m . role = = Role ::User & & ! m . steer ) else { return } ;
let turn = & history [ start .. ] ;
if ! crate ::memory ::worth_learning ( turn ) {
return ;
}
let req = ChatRequest {
system : crate ::memory ::LEARN_SYSTEM . into ( ) ,
history : vec ! [ Message ::user ( & crate ::memory ::learn_brief ( turn , & job . skills ) ) ] ,
tools : vec ! [ ] ,
.. job . req . clone ( )
} ;
let out = match providers ::run_step ( & req , cancel , & | _ | { } ) {
Ok ( o ) = > o ,
Err ( e ) = > {
eprintln! ( " AnyBot: 整理做法時模型沒有回應: {e} " ) ;
return ;
}
} ;
match ( crate ::memory ::learned_skill ( & out . text ) , turn . last ( ) ) {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
( Some ( mut k ) , Some ( reply ) ) = > {
k . steps = crate ::memory ::replay_steps ( turn , & crate ::memory ::learned_replay ( & out . text ) ) ;
2026-11-06 15:12:53 +00:00
eprintln! ( " AnyBot: 學到做法「 {} 」 " , k . name ) ;
send ( StreamEvent ::Learned ( k , reply . at ) ) ;
}
_ = > eprintln! ( " AnyBot: 這次的做法沒有存: {} " , out . text . split_whitespace ( ) . collect ::< Vec < _ > > ( ) . join ( " " ) . chars ( ) . take ( 200 ) . collect ::< String > ( ) ) ,
}
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// Take what the person typed mid-turn and add it to the chat as steering. (The window marks it
/// delivered in the work log once the chat holding it is saved; the message keeps the time it was
/// typed, which is how a restart tells it is already there.)
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
fn drain_steer ( job : & Job , history : & mut Vec < Message > , send : & dyn Fn ( StreamEvent ) ) -> bool {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let queued : Vec < Queued > = std ::mem ::take ( & mut * job . steer . lock ( ) . unwrap ( ) ) ;
for q in & queued {
let m = Message { steer : true , at : q . at , .. Message ::user ( & q . text ) } ;
send ( StreamEvent ::Steered ( m . clone ( ) , q . id . clone ( ) ) ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
history . push ( m ) ;
}
! queued . is_empty ( )
}
/// Which computer a call asks for (permission to use it is checked separately).
fn target_for ( job : & Job , args : & str ) -> Result < tools ::Target , String > {
let v = tools ::args_of ( args ) ;
let want = v . get ( " computer " ) . and_then ( | c | c . as_str ( ) ) . filter ( | c | ! c . is_empty ( ) ) . unwrap_or ( & job . default_computer ) . to_string ( ) ;
Remote brain: a computer that just came on is known at once; an offline computer is said, not swapped; snapshots survive an open browser; end-to-end test
- The brain looked the account's computers up once a minute: for up to a minute after the Mac
came on, 「我的電腦」 was unknown and steps went elsewhere. Now every 15 s, and at once when a
message comes from a computer it doesn't know yet.
- A step naming a computer that is off says so in words instead of running on the chosen one
with a note that the person hadn't asked.
- Snapshots: tar's 'file changed' (the open browser) is not a failure; browser caches are left
out; ten minutes for a large home folder.
- box_end_to_end clicks the terminal where it opened; restore only with ANYBOT_TEST_RESTORE=1.
- deploy/test/remote_e2e.py: account server + headless brain + hands + scripted model, 27 checks
(computers, redirect, agent-desktop on the hands, hand-off 3 ways, phone view, sync copy,
edits, hands offline/back, brain restart, clear). anybot --sample-state for such tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:14:23 +00:00
job . computers . iter ( ) . find ( | t | t . name = = want ) . cloned ( ) . ok_or_else ( | | not_here ( job , & want ) )
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
2026-11-06 15:21:04 +00:00
/// One of the bot's own browsers (where, and which profile).
#[ derive(Clone, PartialEq, Eq, Hash, Debug) ]
enum BotBrowser {
Here ( String ) ,
Box ( String ) ,
Device ( String , String ) ,
}
/// The bot's browsers a job used. When the job ends — answered, stopped or failed — each closes
/// the tabs the job left open and keeps the page it ended on (every browser_open is a new tab;
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
/// nothing used to close them, and the box ran out of memory with 19). Only the job's own tabs:
/// another job's, and the person's, stay (see botbrowser's "each job its own tabs").
2026-11-06 15:21:04 +00:00
struct UsedBrowsers {
list : Vec < BotBrowser > ,
via_name : String ,
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
/// This job's name in the browsers (its tabs are kept apart by it).
job : String ,
2026-11-06 15:21:04 +00:00
}
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
/// Names the jobs of this run (a bot's job on another computer's browser carries it along).
static JOB_SEQ : std ::sync ::atomic ::AtomicU64 = std ::sync ::atomic ::AtomicU64 ::new ( 1 ) ;
2026-11-06 15:21:04 +00:00
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
/// A call's arguments with the job's name added: the browser keeps each job's tabs apart by it
/// (also on another computer's browser, which gets the arguments as they are).
fn with_job ( args : & str , job : & str ) -> String {
let mut v : serde_json ::Value = serde_json ::from_str ( if args . trim ( ) . is_empty ( ) { " {} " } else { args } ) . unwrap_or_else ( | _ | serde_json ::json! ( { } ) ) ;
if let Some ( o ) = v . as_object_mut ( ) {
o . insert ( " _job " . into ( ) , serde_json ::json! ( job ) ) ;
2026-11-06 15:21:04 +00:00
}
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
v . to_string ( )
2026-11-06 15:21:04 +00:00
}
impl UsedBrowsers {
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
fn new ( via_name : & str , bot_id : u64 ) -> Self {
let n = JOB_SEQ . fetch_add ( 1 , Ordering ::Relaxed ) ;
UsedBrowsers { list : vec ! [ ] , via_name : via_name . to_string ( ) , job : format ! ( " {bot_id}-{}-{n} " , crate ::model ::now_ms ( ) ) }
}
2026-11-06 15:21:04 +00:00
fn add ( & mut self , b : BotBrowser ) {
if ! self . list . contains ( & b ) {
self . list . push ( b ) ;
}
}
fn close_tabs ( & mut self ) {
for b in std ::mem ::take ( & mut self . list ) {
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
let args = | p : & str | with_job ( & if p . is_empty ( ) { r # "{"action":"tidy"}"# . to_string ( ) } else { serde_json ::json! ( { " action " : " tidy " , " profile " : p } ) . to_string ( ) } , & self . job ) ;
2026-11-06 15:21:04 +00:00
let never = AtomicBool ::new ( false ) ;
match & b {
BotBrowser ::Here ( p ) = > {
let _ = web ::run_full ( " browser_tabs " , & args ( p ) , & never ) ;
}
BotBrowser ::Box ( p ) = > {
let _ = crate ::botbrowser ::in_box ( | | web ::run_full ( " browser_tabs " , & args ( p ) , & never ) ) ;
}
BotBrowser ::Device ( host , p ) = > {
let _ = crate ::remote ::run_on_device ( host , " browser_tabs " , & args ( p ) , & self . via_name ) ;
}
}
}
}
}
impl Drop for UsedBrowsers {
fn drop ( & mut self ) {
self . close_tabs ( ) ;
}
}
2026-11-05 07:00:31 +00:00
enum BrowserAt {
Here ,
Box ,
Device ( String ) ,
}
2026-11-05 08:47:28 +00:00
/// Whose browser a browser step uses: the computer it names, else the bot's chosen computer; a
/// computer without a browser (a server, an SSH one) hands it to the person's computer.
fn browser_place ( job : & Job , name : & str , args : & str ) -> ( BrowserAt , String ) {
2026-11-05 07:00:31 +00:00
if name = = " web_fetch " {
2026-11-05 08:47:28 +00:00
return ( BrowserAt ::Here , String ::new ( ) ) ;
2026-11-05 07:00:31 +00:00
}
let v = tools ::args_of ( args ) ;
let want = v . get ( " computer " ) . and_then ( | c | c . as_str ( ) ) . map ( str ::trim ) . filter ( | c | ! c . is_empty ( ) ) . unwrap_or ( & job . default_computer ) . to_string ( ) ;
2026-11-05 08:47:28 +00:00
let at = | t : & tools ::Target | -> Option < BrowserAt > {
match & t . place {
2026-11-05 13:28:32 +00:00
tools ::Place ::Device { host , .. } = > Some ( BrowserAt ::Device ( host . clone ( ) ) ) ,
2026-11-05 08:47:28 +00:00
tools ::Place ::Box { .. } if crate ::cloudbox ::available ( ) = > Some ( BrowserAt ::Box ) ,
tools ::Place ::Local { .. } if crate ::botbrowser ::available ( ) = > Some ( BrowserAt ::Here ) ,
_ = > None ,
}
} ;
if let Some ( ( b , n ) ) = job . computers . iter ( ) . find ( | t | t . name = = want ) . and_then ( | t | at ( t ) . map ( | b | ( b , t . name . clone ( ) ) ) ) {
return ( b , n ) ;
}
2026-11-05 11:41:49 +00:00
let person = roles_now ( job ) . person . and_then ( | p | job . computers . iter ( ) . find ( | t | t . name = = p ) ) ;
2026-11-05 08:47:28 +00:00
match person . and_then ( | t | at ( t ) . map ( | b | ( b , t . name . clone ( ) ) ) ) . or_else ( | | job . computers . iter ( ) . find_map ( | t | at ( t ) . map ( | b | ( b , t . name . clone ( ) ) ) ) ) {
Some ( x ) = > x ,
None = > ( BrowserAt ::Here , String ::new ( ) ) ,
2026-11-05 07:00:31 +00:00
}
}
2026-11-06 08:51:25 +00:00
/// Is this browser step for the bot's own browser (no permission, no person's computer)? The
/// box's Chromium always is — whether *this* computer has a Chrome doesn't matter there. (Asking
/// this computer made a cloud brain, which has no Chrome, send every browser_open/read meant for
/// the box to the person's browser on the server: 「打不開瀏覽器」, 「Safari 沒有回應」.)
fn bots_browser ( place : & BrowserAt , name : & str , args : & str ) -> bool {
matches! ( place , BrowserAt ::Box ) | | ! web ::uses_this_computer ( name , args )
}
2026-11-05 07:48:20 +00:00
/// Substring search with word boundaries on the ASCII parts: "mac" must not light up inside
/// "macOS", and a device named "Ann" must not light up on "anniversary". Loose matching let
/// unrelated words — including words from page text that arrives inside a message — unlock a
/// computer the person never named, defeating the guard below.
fn finds ( hay : & str , needle : & str ) -> bool {
let hb : Vec < char > = hay . chars ( ) . collect ( ) ;
let nb : Vec < char > = needle . chars ( ) . collect ( ) ;
if nb . is_empty ( ) | | nb . len ( ) > hb . len ( ) {
return false ;
}
let alnum = | c : char | c . is_ascii_alphanumeric ( ) ;
' start : for i in 0 ..= hb . len ( ) - nb . len ( ) {
if hb [ i ] ! = nb [ 0 ] {
continue ;
}
for j in 1 .. nb . len ( ) {
if hb [ i + j ] ! = nb [ j ] {
continue 'start ;
}
}
if alnum ( nb [ 0 ] ) & & i > 0 & & alnum ( hb [ i - 1 ] ) {
continue ;
}
if alnum ( * nb . last ( ) . unwrap ( ) ) & & i + nb . len ( ) < hb . len ( ) & & alnum ( hb [ i + nb . len ( ) ] ) {
continue ;
}
return true ;
}
false
}
2026-11-05 08:47:28 +00:00
/// Words that mean a computer in the person's message: its name (or the part after 「的」, so
/// 「Mac mini」 names 「王小明的Mac mini」), and for the bot's own computer and the one the person
/// is using, the everyday words for them.
fn mentions ( text : & str , t : & tools ::Target , roles : & tools ::Roles ) -> bool {
2026-11-05 07:00:31 +00:00
let low = text . to_lowercase ( ) ;
2026-11-05 08:47:28 +00:00
let name = t . name . to_lowercase ( ) ;
if finds ( & low , & name ) {
return true ;
}
if let Some ( ( _ , tail ) ) = name . rsplit_once ( '的' ) {
let tail = tail . trim ( ) ;
if tail . chars ( ) . count ( ) > = 3 & & tail ! = " 電腦 " & & finds ( & low , tail ) {
return true ;
}
}
if finds ( & name , " mac " ) & & finds ( & low , " mac " ) {
2026-11-05 07:00:31 +00:00
return true ;
}
2026-11-05 08:47:28 +00:00
let words : & [ & str ] = if t . name = = roles . own {
& [ " bot 的電腦 " , " bot的電腦 " , " 你的電腦 " , " 你自己的電腦 " , " 你的虛擬 " , " 虛擬電腦 " , " 雲端電腦 " ]
} else if roles . person . as_deref ( ) = = Some ( t . name . as_str ( ) ) {
& [ " 我的電腦 " , " 我電腦 " , " 我這台 " , " 這台電腦 " , " 本機 " , " 我的筆電 " , " my computer " ]
} else {
& [ ]
2026-11-05 07:00:31 +00:00
} ;
2026-11-05 08:47:28 +00:00
if t . name = = roles . own & & finds ( & low , & name . replace ( ' ' , " " ) ) {
return true ;
}
2026-11-05 07:48:20 +00:00
words . iter ( ) . any ( | w | finds ( & low , w ) )
2026-11-05 07:00:31 +00:00
}
/// A step naming a computer other than the chosen one, which the person didn't ask for, goes
2026-11-05 08:47:28 +00:00
/// to the chosen one — and the bot is told so (quietly moving it made a bot "ask your Mac" six
/// times while every step landed on its own computer).
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
fn keep_to_chosen ( job : & Job , history : & [ Message ] , name : & str , args : & str ) -> ( String , Option < String > ) {
2026-11-05 07:00:31 +00:00
let mut v = tools ::args_of ( args ) ;
2026-11-05 08:47:28 +00:00
let Some ( named ) = v . get ( " computer " ) . and_then ( | c | c . as_str ( ) ) . map ( | c | c . trim ( ) . to_string ( ) ) . filter ( | c | ! c . is_empty ( ) ) else { return ( args . to_string ( ) , None ) } ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// The chosen computer can't do this kind of step (no screen, no browser): moving it there
// would only fail, so the step stays where the bot sent it.
match job . computers . iter ( ) . find ( | t | t . name = = job . default_computer ) {
Some ( chosen ) if named ! = job . default_computer & & can_do ( chosen , name ) = > { }
_ = > return ( args . to_string ( ) , None ) ,
2026-11-05 07:00:31 +00:00
}
let said = history . iter ( ) . rev ( ) . filter ( | m | m . role = = Role ::User & & ! m . summary ) . take ( 3 ) . map ( | m | m . text . as_str ( ) ) . collect ::< Vec < _ > > ( ) . join ( " \n " ) ;
match job . computers . iter ( ) . find ( | t | t . name = = named ) {
2026-11-05 11:41:49 +00:00
Some ( t ) if mentions ( & said , t , & roles_now ( job ) ) = > ( args . to_string ( ) , None ) ,
Remote brain: a computer that just came on is known at once; an offline computer is said, not swapped; snapshots survive an open browser; end-to-end test
- The brain looked the account's computers up once a minute: for up to a minute after the Mac
came on, 「我的電腦」 was unknown and steps went elsewhere. Now every 15 s, and at once when a
message comes from a computer it doesn't know yet.
- A step naming a computer that is off says so in words instead of running on the chosen one
with a note that the person hadn't asked.
- Snapshots: tar's 'file changed' (the open browser) is not a failure; browser caches are left
out; ten minutes for a large home folder.
- box_end_to_end clicks the terminal where it opened; restore only with ANYBOT_TEST_RESTORE=1.
- deploy/test/remote_e2e.py: account server + headless brain + hands + scripted model, 27 checks
(computers, redirect, agent-desktop on the hands, hand-off 3 ways, phone view, sync copy,
edits, hands offline/back, brain restart, clear). anybot --sample-state for such tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:14:23 +00:00
// A computer that isn't on the list now (switched off, its AnyBot closed): the step says
// so instead of quietly running somewhere else.
None = > ( args . to_string ( ) , None ) ,
2026-11-05 07:00:31 +00:00
_ = > {
v [ " computer " ] = serde_json ::Value ::String ( job . default_computer . clone ( ) ) ;
2026-11-05 08:47:28 +00:00
let note = format! ( " (這一步你指定了「 {named} 」,但使用者選的電腦是「 {} 」,而且他沒有說要換,所以這一步在「 {} 」上做。真的需要別台,先問使用者。) \n " , job . default_computer , job . default_computer ) ;
( v . to_string ( ) , Some ( note ) )
2026-11-05 07:00:31 +00:00
}
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// Can this computer do this kind of step: a screen step needs a screen there, a browser step a
/// browser; anything else runs anywhere.
fn can_do ( t : & tools ::Target , tool : & str ) -> bool {
let screen = computer ::is_computer_tool ( tool ) ;
let browser = web ::is_web_tool ( tool ) & & tool ! = " web_fetch " ;
if ! screen & & ! browser {
return true ;
}
match & t . place {
tools ::Place ::Device { .. } = > true ,
tools ::Place ::Box { .. } = > crate ::cloudbox ::available ( ) ,
tools ::Place ::Local { .. } if screen = > computer ::screen_supported ( ) ,
tools ::Place ::Local { .. } = > crate ::botbrowser ::available ( ) ,
_ = > false ,
}
}
2026-11-05 10:25:46 +00:00
/// What element `e7` is, from the latest app list in the chat (「按鈕「7」」).
fn element_named ( history : & [ Message ] , r : & str ) -> Option < String > {
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let tag = format! ( " [ {} ] " , crate ::marks ::number_of ( r ) ? ) ;
2026-11-05 10:25:46 +00:00
history . iter ( ) . rev ( ) . filter ( | m | m . role = = Role ::Tool ) . take ( 12 ) . find_map ( | m | {
m . text . lines ( ) . find_map ( | l | l . trim_start ( ) . strip_prefix ( & tag ) . map ( | rest | rest . trim ( ) . to_string ( ) ) ) . filter ( | e | ! e . is_empty ( ) )
} )
}
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
/// What makes two screen or browser steps "the same" (None for other tools): the tool, the
/// computer and the target, a click's spot on a 16-pixel grid (an inch beside an empty strip is
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// the same miss). Looking, waiting, and moving through a page (scrolling, keys, back) aren't
/// counted: doing those again and again is how a long page or a list is read.
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
fn step_key ( name : & str , args : & str ) -> Option < String > {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
if ! ( computer ::is_computer_tool ( name ) | | web ::is_web_tool ( name ) )
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
| | matches! ( name , " computer_use_screenshot " | " computer_use_show " | " browser_read " | " browser_screenshot " | " browser_wait " | " computer_use_scroll " | " browser_scroll " | " computer_use_press_key " | " browser_press " | " browser_back " )
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
{
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
return None ;
}
let mut v = tools ::args_of ( args ) ;
if let Some ( o ) = v . as_object_mut ( ) {
for k in [ " x " , " y " ] {
if let Some ( n ) = o . get ( k ) . and_then ( serde_json ::Value ::as_f64 ) {
o . insert ( k . into ( ) , serde_json ::Value ::from ( ( n / 16.0 ) . round ( ) as i64 ) ) ;
}
}
}
Some ( format! ( " {name} {v} " ) )
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// The bot's own computer can come or go mid-turn (Docker started or stopped): its place follows
/// what is true now, and the bot is told when it changed. (Fixed at the start of the turn, a box
/// that came up mid-turn stayed "no screen" and every step there ran in the wrong place.)
fn refresh_own ( job : & mut Job , bot_id : u64 ) -> Option < String > {
let own = job . roles . own . clone ( ) ;
let t = job . computers . iter_mut ( ) . find ( | t | t . name = = own ) ? ;
match ( & t . place , crate ::cloudbox ::available ( ) ) {
( tools ::Place ::Sealed { .. } , true ) = > {
t . place = tools ::Place ::Box { dir : crate ::cloudbox ::bot_dir ( bot_id ) } ;
Some ( format! ( " (「 {own} 」現在可以用了:它是你自己的 Linux 桌面,可以執行指令、看畫面、用瀏覽器。) " ) )
}
( tools ::Place ::Box { .. } , false ) = > {
t . place = tools ::Place ::Sealed { root : crate ::model ::bot_computer_dir ( bot_id ) } ;
Some ( format! ( " (「 {own} 」現在連不到 Docker: 暫時不能執行指令、看畫面。) " ) )
}
_ = > None ,
}
}
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
/// How many times in a row this very call (same tool, same arguments) already came back with the
/// very same answer among the latest steps. Waiting tools are left out: there the same answer is
/// the point.
fn same_answers ( history : & [ Message ] , name : & str , args : & str ) -> usize {
if matches! ( name , " read_terminal " | " browser_wait " | " write_terminal " ) {
return 0 ;
}
let want = tools ::args_of ( args ) ;
let calls : std ::collections ::HashMap < & str , ( & str , & str ) > = history . iter ( ) . rev ( ) . take ( 80 ) . flat_map ( | m | m . tool_calls . iter ( ) ) . map ( | c | ( c . id . as_str ( ) , ( c . name . as_str ( ) , c . args . as_str ( ) ) ) ) . collect ( ) ;
// A warning put in front of an answer doesn't make it a different answer.
let bare = | t : & str | -> String { t . lines ( ) . skip_while ( | l | l . starts_with ( " ⚠️ " ) ) . collect ::< Vec < _ > > ( ) . join ( " \n " ) } ;
let mut first : Option < String > = None ;
let mut n = 0 ;
for m in history . iter ( ) . rev ( ) . filter ( | m | m . role = = Role ::Tool ) . take ( 12 ) {
let Some ( ( cn , ca ) ) = calls . get ( m . tool_call_id . as_str ( ) ) else { continue } ;
if * cn ! = name | | tools ::args_of ( ca ) ! = want {
continue ;
}
let text = bare ( & m . text ) ;
match & first {
None = > {
first = Some ( text ) ;
n = 1 ;
}
Some ( f ) if * f = = text = > n + = 1 ,
_ = > break ,
}
}
n
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// "Always allow" for this turn: the grant is kept (once).
fn keep_grant ( grants : & mut Vec < String > , g : String ) {
if ! g . is_empty ( ) & & ! grants . contains ( & g ) {
grants . push ( g ) ;
}
}
2026-11-06 08:51:25 +00:00
/// Every CHECK_IN_STEPS steps, the newest result carries a question for the bot: still getting
/// somewhere, or stuck? Only the model sees it (the chat keeps the result as it was).
fn check_in ( history : & mut [ Message ] , step : usize ) {
if step = = 0 | | ! step . is_multiple_of ( CHECK_IN_STEPS ) {
return ;
}
if let Some ( m ) = history . last_mut ( ) . filter ( | m | m . role = = Role ::Tool ) {
m . text . push_str ( & format! ( " \n \n (這一回合已經做了 {step} 步,最多 {MAX_STEPS} 步。方向對、有進展就繼續做完;如果最近幾步都沒有進展,停下來用一兩句話告訴使用者卡在哪、需要他做什麼。) " ) ) ;
}
}
2026-11-05 08:47:28 +00:00
/// A step's result, with a word first when the step was moved back to the chosen computer.
fn noted ( moved : & std ::collections ::HashMap < String , String > , id : & str , text : & str ) -> String {
match moved . get ( id ) {
Some ( n ) = > format! ( " {n} {text} " ) ,
None = > text . to_string ( ) ,
}
}
Remote brain: a computer that just came on is known at once; an offline computer is said, not swapped; snapshots survive an open browser; end-to-end test
- The brain looked the account's computers up once a minute: for up to a minute after the Mac
came on, 「我的電腦」 was unknown and steps went elsewhere. Now every 15 s, and at once when a
message comes from a computer it doesn't know yet.
- A step naming a computer that is off says so in words instead of running on the chosen one
with a note that the person hadn't asked.
- Snapshots: tar's 'file changed' (the open browser) is not a failure; browser caches are left
out; ten minutes for a large home folder.
- box_end_to_end clicks the terminal where it opened; restore only with ANYBOT_TEST_RESTORE=1.
- deploy/test/remote_e2e.py: account server + headless brain + hands + scripted model, 27 checks
(computers, redirect, agent-desktop on the hands, hand-off 3 ways, phone view, sync copy,
edits, hands offline/back, brain restart, clear). anybot --sample-state for such tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:14:23 +00:00
/// A computer the step names that isn't among this bot's computers right now, in plain words.
fn not_here ( job : & Job , name : & str ) -> String {
format! ( " 「 {name} 」現在連不到:那台電腦可能沒開,或上面的 AnyBot 沒開、沒登入。請告訴使用者,等它連上再做(或問他要不要改用別台)。現在可以用: {} " , job . computers . iter ( ) . map ( | t | t . name . as_str ( ) ) . collect ::< Vec < _ > > ( ) . join ( " 、 " ) )
}
2026-11-05 11:41:49 +00:00
/// Which computer is the person's right now: a message typed on one of the account's computers
/// while the turn runs (a steer, or the answer to a hand-off) moves it there — the turn's own
/// idea was fixed when it started, maybe from the phone, which isn't a computer.
fn roles_now ( job : & Job ) -> tools ::Roles {
let mut r = job . roles . clone ( ) ;
let host = crate ::remote ::person_host ( ) ;
if ! host . is_empty ( ) {
2026-11-05 13:28:32 +00:00
if let Some ( t ) = job . computers . iter ( ) . find ( | t | matches! ( & t . place , tools ::Place ::Device { host : h , .. } if * h = = host ) ) {
2026-11-05 11:41:49 +00:00
r . person = Some ( t . name . clone ( ) ) ;
}
}
r
}
2026-11-05 06:46:42 +00:00
/// Does the call name a computer?
fn names_a_computer ( args : & str ) -> bool {
tools ::args_of ( args ) . get ( " computer " ) . and_then ( | c | c . as_str ( ) ) . map ( | c | ! c . trim ( ) . is_empty ( ) ) . unwrap_or ( false )
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
/// Rough token count: about 4 Latin characters or 1 Chinese character per token (estimated).
/// Which screen a computer_use call is for: the one named, else the bot's default when it has a
/// screen, else the bot's own desktop when there is one, else this Mac.
fn screen_target ( job : & Job , args : & str ) -> Result < tools ::Target , String > {
let v = tools ::args_of ( args ) ;
let has_screen = | t : & tools ::Target | matches! ( t . place , tools ::Place ::Box { .. } | tools ::Place ::Local { .. } | tools ::Place ::Device { .. } ) ;
let named = v . get ( " computer " ) . and_then ( | c | c . as_str ( ) ) . map ( str ::trim ) . filter ( | c | ! c . is_empty ( ) ) ;
if let Some ( n ) = named {
Remote brain: a computer that just came on is known at once; an offline computer is said, not swapped; snapshots survive an open browser; end-to-end test
- The brain looked the account's computers up once a minute: for up to a minute after the Mac
came on, 「我的電腦」 was unknown and steps went elsewhere. Now every 15 s, and at once when a
message comes from a computer it doesn't know yet.
- A step naming a computer that is off says so in words instead of running on the chosen one
with a note that the person hadn't asked.
- Snapshots: tar's 'file changed' (the open browser) is not a failure; browser caches are left
out; ten minutes for a large home folder.
- box_end_to_end clicks the terminal where it opened; restore only with ANYBOT_TEST_RESTORE=1.
- deploy/test/remote_e2e.py: account server + headless brain + hands + scripted model, 27 checks
(computers, redirect, agent-desktop on the hands, hand-off 3 ways, phone view, sync copy,
edits, hands offline/back, brain restart, clear). anybot --sample-state for such tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:14:23 +00:00
let t = job . computers . iter ( ) . find ( | t | t . name = = n ) . ok_or_else ( | | not_here ( job , n ) ) ? ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
if ! has_screen ( t ) {
2026-11-05 08:47:28 +00:00
return Err ( if n = = job . roles . own {
format! ( " {n} 現在沒有螢幕: AnyBot 所在的電腦沒有 Docker( {} )。要用別台請使用者指名。 " , if cfg! ( target_os = " macos " ) { " 安裝 OrbStack 或 Docker Desktop 就會有 " } else { " 安裝 Docker Engine 就會有 " } )
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} else {
format! ( " 「 {n} 」不能遠端操作螢幕,只能用檔案和指令工具。 " )
} ) ;
}
return Ok ( t . clone ( ) ) ;
}
if let Some ( t ) = job . computers . iter ( ) . find ( | t | t . name = = job . default_computer & & has_screen ( t ) ) {
return Ok ( t . clone ( ) ) ;
}
2026-11-05 00:05:29 +00:00
// Not named: a screen that is really there — the bot's desktop, this computer's, then one of
// the account's other computers (a brain on a server has no screen of its own).
let real = | t : & & tools ::Target | match t . place {
tools ::Place ::Box { .. } = > crate ::cloudbox ::available ( ) ,
tools ::Place ::Local { .. } = > computer ::screen_supported ( ) ,
_ = > false ,
} ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
job . computers
. iter ( )
2026-11-05 00:05:29 +00:00
. find ( | t | matches! ( t . place , tools ::Place ::Box { .. } ) & & real ( t ) )
. or_else ( | | job . computers . iter ( ) . find ( | t | matches! ( t . place , tools ::Place ::Local { .. } ) & & real ( t ) ) )
. or_else ( | | job . computers . iter ( ) . find ( | t | matches! ( t . place , tools ::Place ::Device { .. } ) ) )
. or_else ( | | job . computers . iter ( ) . find ( | t | matches! ( t . place , tools ::Place ::Box { .. } ) ) )
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
. or_else ( | | job . computers . iter ( ) . find ( | t | matches! ( t . place , tools ::Place ::Local { .. } ) ) )
. cloned ( )
. ok_or_else ( | | " 找不到可以操作螢幕的電腦 " . to_string ( ) )
}
pub fn estimate_tokens ( s : & str ) -> u64 {
let ( mut ascii , mut other ) = ( 0 u64 , 0 u64 ) ;
for c in s . chars ( ) {
if c . is_ascii ( ) { ascii + = 1 } else { other + = 1 }
}
ascii / 4 + other
}
fn history_tokens ( system : & str , h : & [ Message ] ) -> u64 {
estimate_tokens ( system ) + h . iter ( ) . map ( | m | estimate_tokens ( & m . text ) + m . tool_calls . iter ( ) . map ( | c | estimate_tokens ( & c . args ) ) . sum ::< u64 > ( ) + 4 ) . sum ::< u64 > ( )
}
/// Where to cut so the newest messages stay and no tool call is split from its result:
/// the summary covers everything before a user message.
pub fn compaction_cut ( h : & [ Message ] ) -> Option < usize > {
if h . len ( ) < = KEEP_RECENT_MESSAGES + 1 {
return None ;
}
( 1 ..= h . len ( ) - KEEP_RECENT_MESSAGES ) . rev ( ) . find ( | & i | h [ i ] . role = = Role ::User & & ! h [ .. i ] . iter ( ) . all ( | m | m . summary ) )
}
fn flatten ( h : & [ Message ] ) -> String {
let clip = | s : & str | -> String {
let mut t : String = s . chars ( ) . take ( SUMMARY_SNIPPET_CHARS ) . collect ( ) ;
if s . chars ( ) . count ( ) > SUMMARY_SNIPPET_CHARS {
t . push ( '…' ) ;
}
t
} ;
h . iter ( )
. map ( | m | {
if m . summary {
return format! ( " 【先前的摘要】 \n {} " , m . text ) ;
}
match m . role {
Role ::User = > format! ( " 使用者: {} " , clip ( & m . text ) ) ,
Role ::Assistant = > {
let calls : Vec < String > = m . tool_calls . iter ( ) . map ( | c | format! ( " [使用工具 {} : {} ] " , c . name , tools ::describe ( & c . name , & c . args ) ) ) . collect ( ) ;
format! ( " 助手: {} {} " , clip ( & m . text ) , calls . join ( " " ) )
}
Role ::Tool = > format! ( " 工具結果( {} ) : {} " , m . tool_name , clip ( & m . text ) ) ,
}
} )
. collect ::< Vec < _ > > ( )
. join ( " \n \n " )
}
/// Fold older messages into one summary when the chat is over `percent` of the context.
/// Returns how many messages were replaced.
fn maybe_compact ( job : & Job , history : & mut Vec < Message > , percent : u64 , cancel : & AtomicBool , send : & dyn Fn ( StreamEvent ) ) -> Option < usize > {
let used = history_tokens ( & job . req . system , history ) ;
if used * 100 < job . context_tokens * percent {
return None ;
}
let cut = compaction_cut ( history ) ? ;
send ( StreamEvent ::Retrying ( " 對話很長了,正在把較早的內容整理成摘要… " . into ( ) ) ) ;
let req = ChatRequest {
system : " 你負責把一段對話整理成摘要,讓之後的對話能接著進行。請用條列寫下:使用者的目標與偏好、已經做出的決定、重要的事實與數字、提到的檔案路徑與指令結果、還沒完成的事。不要加入對話裡沒有的內容。 " . into ( ) ,
history : vec ! [ Message ::user ( & format! ( " 請整理下面這段對話: \n \n {} " , flatten ( & history [ .. cut ] ) ) ) ] ,
tools : vec ! [ ] ,
.. job . req . clone ( )
} ;
let quiet = | ev : StreamEvent | {
if matches! ( ev , StreamEvent ::Alive | StreamEvent ::Retrying ( _ ) ) {
send ( StreamEvent ::Alive ) ;
}
} ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
job . note ( crate ::oplog ::Record ::StepAttempt { op : job . op . clone ( ) , step : " compaction " . into ( ) , attempt : 1 } ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let out = providers ::run_step ( & req , cancel , & quiet ) . ok ( ) ? ;
if out . text . trim ( ) . is_empty ( ) | | cancel . load ( Ordering ::Relaxed ) {
return None ;
}
let summary = Message { role : Role ::Assistant , text : out . text . trim ( ) . to_string ( ) , summary : true , via : job . via . clone ( ) , at : crate ::model ::now_ms ( ) , .. Default ::default ( ) } ;
2026-11-05 07:48:20 +00:00
// The window shows the chat unfiltered (group chats drop some messages on the way here);
// the timestamp of the first kept message is the landmark it can find the cut by.
let boundary = history [ cut ] . at ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
history . splice ( .. cut , [ summary . clone ( ) ] ) ;
2026-11-05 07:48:20 +00:00
send ( StreamEvent ::Compacted ( cut , summary , boundary ) ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Some ( cut )
}
/// Wait for the person's answer; stopping the reply counts as a no.
fn wait_answer ( rx : std ::sync ::mpsc ::Receiver < ApprovalAnswer > , cancel : & AtomicBool ) -> ApprovalAnswer {
loop {
if cancel . load ( Ordering ::Relaxed ) {
return ApprovalAnswer ::Reject ;
}
match rx . recv_timeout ( Duration ::from_millis ( 200 ) ) {
Ok ( a ) = > return a ,
Err ( std ::sync ::mpsc ::RecvTimeoutError ::Timeout ) = > continue ,
Err ( _ ) = > return ApprovalAnswer ::Reject ,
}
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// The tool name for doing a skill's steps again.
pub const REPLAY_TOOL : & str = " replay_skill " ;
/// A control's kind and name from its line in a list, without its value or state, which may
/// differ the next time (「勾選框「自動更新」: 1( 已勾) 」 → 「勾選框「自動更新」」).
fn control_key ( desc : & str ) -> & str {
match desc . find ( '」' ) {
Some ( i ) = > & desc [ .. i + '」' . len_utf8 ( ) ] ,
None = > desc . split ( [ ': ' , '( ' ] ) . next ( ) . unwrap_or ( desc ) . trim ( ) ,
}
}
/// A replay step's arguments for now: each control it acted on found again by what it is, in the
/// newest lists (screen steps get today's number; browser steps find it by its words). Err: the
/// control that isn't there.
fn resolve_replay ( step : & crate ::memory ::ReplayStep , history : & [ Message ] ) -> Result < String , String > {
let mut v : serde_json ::Value = serde_json ::from_str ( & step . args ) . unwrap_or_default ( ) ;
let find = | desc : & str | -> Option < String > {
let want = control_key ( desc ) ;
history . iter ( ) . rev ( ) . filter ( | m | m . role = = Role ::Tool ) . take ( 6 ) . find_map ( | m | {
m . text . lines ( ) . find_map ( | l | {
let ( num , rest ) = l . trim_start ( ) . strip_prefix ( '[' ) ? . split_once ( " ] " ) ? ;
( control_key ( rest . trim ( ) ) = = want & & num . chars ( ) . all ( | c | c . is_ascii_digit ( ) ) ) . then ( | | num . to_string ( ) )
} )
} )
} ;
let words = | desc : & str | desc . split_once ( '「' ) . and_then ( | ( _ , r ) | r . rsplit_once ( '」' ) ) . map ( | ( t , _ ) | t . to_string ( ) ) . unwrap_or_else ( | | desc . to_string ( ) ) ;
if step . tool = = " computer_use_batch " {
if let Some ( actions ) = v [ " actions " ] . as_array_mut ( ) {
for ( i , a ) in actions . iter_mut ( ) . enumerate ( ) {
if let Some ( desc ) = step . targets . get ( i ) . filter ( | d | ! d . is_empty ( ) ) {
a [ " ref " ] = serde_json ::Value ::String ( find ( desc ) . ok_or_else ( | | desc . clone ( ) ) ? ) ;
}
}
}
} else if let Some ( desc ) = step . targets . first ( ) . filter ( | d | ! d . is_empty ( ) ) {
if step . tool . starts_with ( " browser_ " ) {
if let Some ( o ) = v . as_object_mut ( ) {
o . remove ( " ref " ) ;
}
v [ " text " ] = serde_json ::Value ::String ( words ( desc ) ) ;
} else {
v [ " ref " ] = serde_json ::Value ::String ( find ( desc ) . ok_or_else ( | | desc . clone ( ) ) ? ) ;
}
}
Ok ( v . to_string ( ) )
}
/// A skill being done again: what is left, which step this is, and how many there are.
struct Replay {
name : String ,
left : std ::collections ::VecDeque < crate ::memory ::ReplayStep > ,
total : usize ,
/// The step that already had a fresh look taken for it (one look per step, then hand back).
looked : Option < usize > ,
}
/// Quick steps in a row before the bot's own model looks at the work again (estimated).
pub const QUICK_RUN_MAX : usize = 4 ;
2026-11-09 03:25:17 +00:00
/// Trouble this many times in a turn and the rest of it is the bot's own model's: the quick model
/// guessing its way through an unfamiliar dialog (names that aren't there, unnamed items) only
/// made it longer each time it got the step back.
pub const QUICK_TROUBLE_MAX : usize = 2 ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// Steps that only look (nothing on any computer changes): a turn of only these needs no check
/// at the end.
fn only_looks ( name : & str ) -> bool {
matches! (
name ,
" computer_use_screenshot " | " computer_use_zoom " | " computer_use_show " | " computer_use_check_permissions " | " browser_read " | " browser_screenshot " | " browser_elements " | " browser_find " | " browser_table " | " browser_tabs " | " browser_wait " | " web_fetch " | " read_file " | " list_dir " | " search " | " read_terminal " | " list_computers " | " list_bots " | " list_routines " | " read_page " | " use_skill " | " react " | " show_card "
)
}
/// 想的跟做的分開: which model takes a step. The bot's own (thinking) model takes the first step
/// (it plans and starts), the step after anything went wrong, one in every QUICK_RUN_MAX, and the
/// end of a turn that changed something (it checks the work and answers). A quick model carries
/// out the planned steps in between: most of a turn's time was the thinking model thinking
/// before each click.
pub struct StepPlan {
quick : Option < String > ,
main_next : bool ,
quick_run : usize ,
/// The way the last acting step went (commands, the screen, a web page).
last_route : Option < & 'static str > ,
2026-11-09 03:25:17 +00:00
/// Steps that went wrong this turn.
troubles : usize ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// Something on a computer may have changed this turn (not only looking).
pub changed : bool ,
}
impl StepPlan {
pub fn new ( quick : Option < String > ) -> Self {
2026-11-09 03:25:17 +00:00
StepPlan { quick , main_next : true , quick_run : 0 , last_route : None , troubles : 0 , changed : false }
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
}
/// The quick model for this step, or None for the bot's own.
pub fn pick ( & mut self ) -> Option < String > {
if self . quick . is_none ( ) | | self . main_next | | self . quick_run > = QUICK_RUN_MAX {
self . quick_run = 0 ;
self . main_next = false ;
return None ;
}
self . quick_run + = 1 ;
self . quick . clone ( )
}
/// After a step: trouble (a step failed, was refused, or repeated) sends the next one back
/// to the bot's own model.
pub fn after ( & mut self , trouble : bool ) {
if trouble {
self . main_next = true ;
2026-11-09 03:25:17 +00:00
self . troubles + = 1 ;
if self . troubles > = QUICK_TROUBLE_MAX & & self . quick . take ( ) . is_some ( ) {
eprintln! ( " AnyBot: 這一回合出了 {} 次問題,剩下的步驟改用 Bot 自己的模型 " , self . troubles ) ;
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
}
}
2026-11-09 04:53:06 +00:00
/// The quick model answered without acting: the bot's own model looks at where things stand
/// and answers instead (or carries on). Returns whether to set that answer aside. Not only
/// after changes: a quick model also stopped half way to ask 「要我繼續嗎?」 after one look.
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
pub fn recheck_answer ( & mut self , was_quick : bool ) -> bool {
2026-11-09 04:53:06 +00:00
let recheck = was_quick ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if recheck {
self . main_next = true ;
}
recheck
}
/// The quick model couldn't be used (not on this account, an error): the rest of the turn
/// goes without it.
pub fn quick_failed ( & mut self ) {
self . quick = None ;
}
pub fn quick_is_off ( & self ) -> bool {
self . quick . is_none ( )
}
/// The way a step's acting calls went. The quick model switching ways on its own (commands
/// to clicking the screen, say) is the bot's own model's call: true sends the next step there.
pub fn changed_route ( & mut self , calls : & [ crate ::model ::ToolCall ] , was_quick : bool ) -> bool {
let route = calls . iter ( ) . filter ( | c | ! only_looks ( & c . name ) ) . map ( | c | route_of ( & c . name ) ) . next ( ) ;
let Some ( route ) = route else { return false } ;
let switched = was_quick & & self . last_route . is_some_and ( | r | r ! = route ) ;
self . last_route = Some ( route ) ;
switched
}
}
2026-11-09 06:03:40 +00:00
/// The quick model's screen tools name their controls only: looking a number up in the list
/// is where it went wrong (1, 2, 3, then 「1」's number again for 4). Without `ref` it says which
/// button the way a person would; two of the same name come back as trouble for the bot's own
/// model.
fn quick_tools ( tools : & [ serde_json ::Value ] ) -> Vec < serde_json ::Value > {
tools
. iter ( )
. map ( | t | {
let mut t = t . clone ( ) ;
2026-11-09 08:50:18 +00:00
let screen = t [ " name " ] . as_str ( ) . is_some_and ( | n | matches! ( n , " computer " | " computer_use_click " | " computer_use_typing " | " computer_use_scroll " | " computer_use_mouse_move " | " computer_use_batch " ) ) ;
2026-11-09 06:03:40 +00:00
if screen {
2026-11-09 06:58:40 +00:00
// `get_mut`, not `t[..]`: indexing a Value mutably inserts every missing key, which
// once gave click and typing an `actions` with `properties: null` — the provider then
// refused the whole tool list and the quick model never ran.
let props = t . get_mut ( " parameters " ) . and_then ( | p | p . get_mut ( " properties " ) ) . and_then ( | p | p . as_object_mut ( ) ) ;
if let Some ( p ) = props {
2026-11-09 06:03:40 +00:00
p . remove ( " ref " ) ;
2026-11-09 06:58:40 +00:00
let items = p . get_mut ( " actions " ) . and_then ( | a | a . get_mut ( " items " ) ) . and_then ( | i | i . get_mut ( " properties " ) ) . and_then ( | p | p . as_object_mut ( ) ) ;
if let Some ( items ) = items {
items . remove ( " ref " ) ;
}
2026-11-09 06:03:40 +00:00
}
if let Some ( d ) = t [ " description " ] . as_str ( ) {
t [ " description " ] = serde_json ::Value ::String ( format! ( " {d} Name the control exactly as the latest list shows it (name); there are no numbers here. " ) ) ;
}
}
t
} )
. collect ( )
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// Which way a tool goes: commands and files, the screen, or a web page.
fn route_of ( name : & str ) -> & 'static str {
if computer ::is_computer_tool ( name ) | | crate ::macapps ::is_mac_tool ( name ) {
" screen "
} else if web ::is_web_tool ( name ) {
" web "
} else {
" command "
}
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
pub fn spawn ( job : Job , bot_id : u64 , tx : Sender < ( u64 , StreamEvent ) > , cancel : Arc < AtomicBool > , ctx : eframe ::egui ::Context ) {
std ::thread ::spawn ( move | | {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let mut job = job ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let send = | ev : StreamEvent | {
let _ = tx . send ( ( bot_id , ev ) ) ;
ctx . request_repaint ( ) ;
} ;
let mut history = job . req . history . clone ( ) ;
let mut grants = job . grants . clone ( ) ;
// Turned on by an "always" answer mid-turn (the window saves the setting too).
let mut tools_on = job . tools_enabled ;
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
// The screen and browser steps of this turn (to stop the same step being tried over and over).
let mut tried : Vec < String > = Vec ::new ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let mut local_on = job . local_execution ;
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
let mut browsers = UsedBrowsers ::new ( & job . via_name , bot_id ) ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// 電腦現況: what someone at each computer already knows (system, sudo, network, apps…).
2026-11-09 08:45:56 +00:00
// Added ONCE here, before the step loop: the turn's system prefix (instructions + facts)
// and its tool list (and the deterministic `quick_tools` map of it) never change between
// steps — that is what makes the prompt-cache breakpoints in providers::stream_anthropic
// hit from step 2 on. Keep it that way: anything that varies per step belongs in the
// history tail, never in the system prompt or the tools.
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let may_use_persons = job . tools_enabled | | job . direct . load ( Ordering ::Relaxed ) ;
job . req . system . push_str ( & crate ::facts ::for_turn ( & job . computers , may_use_persons , & job . via_name ) ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// The turn's first line in the work log, and (whatever happens) its last.
job . note ( crate ::oplog ::Record ::OperationStarted {
op : job . op . clone ( ) ,
at : crate ::model ::now_ms ( ) ,
source_leaf : history . last ( ) . map ( | m | m . at ) . unwrap_or ( 0 ) ,
prompt : history . last ( ) . filter ( | m | m . role = = Role ::User ) . map ( | m | crate ::oplog ::Prompt { at : m . at , text : m . text . clone ( ) } ) ,
} ) ;
let mut closing = Closing { log : job . log . clone ( ) , op : job . op . clone ( ) , cancel : cancel . clone ( ) , written : false } ;
// Folding the chat because the model said it was too long: once per thing the person
// said (a second time means folding didn't help).
let mut overflow_folded = false ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
maybe_compact ( & job , & mut history , COMPACT_THRESHOLD_PERCENT , & cancel , & send ) ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let mut plan = StepPlan ::new ( job . step_model . clone ( ) ) ;
// A quick step's answer was set aside: its (empty) message in the chat is reused.
let mut reuse_slot = false ;
// 重播: a skill's steps done again as they were, with no model in between.
let mut replay : Option < Replay > = None ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
for step in 0 .. MAX_STEPS {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// The next replay step, its controls found again; a control that isn't there gets one
// fresh look, then the bot takes over (told where and why).
let mut scripted : Option < crate ::model ::ToolCall > = None ;
if let Some ( r ) = replay . as_mut ( ) {
if let Some ( next ) = r . left . front ( ) . cloned ( ) {
let k = r . total - r . left . len ( ) + 1 ;
match resolve_replay ( & next , & history ) {
Ok ( args ) = > {
r . left . pop_front ( ) ;
scripted = Some ( crate ::model ::ToolCall { id : format ! ( " replay-{k}-{} " , crate ::model ::now_ms ( ) ) , name : next . tool . clone ( ) , args , status : ToolStatus ::Running } ) ;
}
Err ( _ ) if r . looked ! = Some ( k ) & & ! next . tool . starts_with ( " browser_ " ) = > {
r . looked = Some ( k ) ;
let computer = tools ::args_of ( & next . args ) [ " computer " ] . as_str ( ) . map ( | c | serde_json ::json! ( { " computer " : c } ) ) . unwrap_or ( serde_json ::json! ( { } ) ) ;
scripted = Some ( crate ::model ::ToolCall { id : format ! ( " replay-look-{k}-{} " , crate ::model ::now_ms ( ) ) , name : " computer_use_screenshot " . into ( ) , args : computer . to_string ( ) , status : ToolStatus ::Running } ) ;
}
Err ( missing ) = > {
if let Some ( m ) = history . last_mut ( ) . filter ( | m | m . role = = Role ::Tool ) {
m . text . push_str ( & format! ( " \n \n (重播「 {} 」停在第 {k} 步:畫面上找不到「 {missing} 」。前面 {} 步已經照做了;接下來由你看畫面決定怎麼做完。) " , r . name , k - 1 ) ) ;
}
replay = None ;
plan . after ( true ) ;
}
}
}
}
let quick = if scripted . is_some ( ) { None } else { plan . pick ( ) } ;
let via = match ( & quick , & scripted ) {
( _ , Some ( _ ) ) = > format! ( " {} (重播) " , job . via ) ,
( Some ( q ) , None ) = > job . via . replacen ( & job . req . model , q , 1 ) ,
( None , None ) = > job . via . clone ( ) ,
} ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
if step > 0 {
maybe_compact ( & job , & mut history , COMPACT_THRESHOLD_PERCENT + COMPACT_TOLERANCE_PERCENT , & cancel , & send ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
if drain_steer ( & job , & mut history , & send ) {
overflow_folded = false ;
}
2026-11-06 08:51:25 +00:00
check_in ( & mut history , step ) ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
if let Some ( note ) = refresh_own ( & mut job , bot_id ) {
if let Some ( m ) = history . last_mut ( ) . filter ( | m | m . role = = Role ::Tool ) {
m . text . push_str ( & format! ( " \n \n {note} " ) ) ;
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if ! reuse_slot {
send ( StreamEvent ::NextStep ( via . clone ( ) ) ) ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
reuse_slot = false ;
// Quick steps don't stream their words: an answer may be set aside for the check.
let quiet = | ev : StreamEvent | {
if ! matches! ( ev , StreamEvent ::Text ( _ ) | StreamEvent ::Thinking ( _ ) ) {
send ( ev )
}
} ;
let tried_quick = quick . clone ( ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let mut attempt = 0 ;
2026-11-09 08:50:18 +00:00
let mut out = loop {
2026-11-09 01:02:46 +00:00
// A replay step: no model.
if let Some ( c ) = scripted . clone ( ) {
break providers ::StepOutput { text : String ::new ( ) , thinking : String ::new ( ) , tool_calls : vec ! [ c ] , blocks : vec ! [ ] } ;
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
attempt + = 1 ;
job . note ( crate ::oplog ::Record ::StepAttempt { op : job . op . clone ( ) , step : " assistant " . into ( ) , attempt } ) ;
2026-11-09 01:02:46 +00:00
let main_req = ChatRequest { history : history . clone ( ) , .. job . req . clone ( ) } ;
let quick_now = if plan . quick_is_off ( ) { None } else { quick . clone ( ) } ;
let got = match quick_now {
2026-11-09 06:03:40 +00:00
Some ( q ) = > match providers ::run_step ( & ChatRequest { model : q . clone ( ) , tools : quick_tools ( & main_req . tools ) , .. main_req . clone ( ) } , & cancel , & quiet ) {
2026-11-09 01:02:46 +00:00
Ok ( o ) = > Ok ( o ) ,
// Too long is the chat's problem, not the quick model's: folded below.
Err ( e ) if providers ::is_context_overflow ( & e ) = > Err ( e ) ,
Err ( e ) = > {
eprintln! ( " AnyBot: 快速模型 {q} 不能用( {e} ),這一回合改用 {} " , job . req . model ) ;
plan . quick_failed ( ) ;
providers ::run_step ( & main_req , & cancel , & send )
}
} ,
None = > providers ::run_step ( & main_req , & cancel , & send ) ,
} ;
match got {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
Ok ( o ) = > break o ,
// Too long for the model: that answer is thrown away, older messages are
// folded into a summary (however full the chat looks by our estimate), and
// the same step is asked again.
Err ( e ) if providers ::is_context_overflow ( & e ) & & ! overflow_folded & & ! cancel . load ( Ordering ::Relaxed ) = > {
overflow_folded = true ;
if maybe_compact ( & job , & mut history , 0 , & cancel , & send ) . is_none ( ) {
closing . close ( crate ::oplog ::Outcome ::Failed ) ;
send ( StreamEvent ::Error ( e ) ) ;
return ;
}
}
Err ( e ) = > {
let e = if providers ::is_context_overflow ( & e ) & & overflow_folded { format! ( " {e} \n (已經把較早的對話整理成摘要再試過一次,還是太長。) " ) } else { e } ;
closing . close ( crate ::oplog ::Outcome ::Failed ) ;
send ( StreamEvent ::Error ( e ) ) ;
return ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
} ;
2026-11-09 08:50:18 +00:00
// The model drives the screen with one tool: expand `computer` + action into the
// internal per-action tool right here, so every guard below (repeat detection,
// approvals, replay, the remote link, the work log) keeps its stable per-action name.
for c in out . tool_calls . iter_mut ( ) {
computer ::expand_call ( c ) ;
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let quick = if plan . quick_is_off ( ) { None } else { tried_quick } ;
let via = if quick . is_none ( ) & & scripted . is_none ( ) { job . via . clone ( ) } else { via } ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// Steps moved back to the chosen computer: what the bot is told with each result, and
// the arguments they run with. The chat keeps the call as the model wrote it.
2026-11-05 08:47:28 +00:00
let mut moved : std ::collections ::HashMap < String , String > = std ::collections ::HashMap ::new ( ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let mut runs_with : std ::collections ::HashMap < String , String > = std ::collections ::HashMap ::new ( ) ;
for c in out . tool_calls . iter ( ) {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let ( args , note ) = keep_to_chosen ( & job , & history , & c . name , & c . args ) ;
2026-11-05 08:47:28 +00:00
if let Some ( n ) = note {
moved . insert ( c . id . clone ( ) , n ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
runs_with . insert ( c . id . clone ( ) , args ) ;
2026-11-05 08:47:28 +00:00
}
2026-11-05 07:00:31 +00:00
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let msg = Message {
role : Role ::Assistant ,
2026-11-05 06:46:42 +00:00
text : crate ::i18n ::reply_text ( & out . text ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
thinking : out . thinking ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
via ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
tool_calls : out . tool_calls ,
blocks : out . blocks ,
at : crate ::model ::now_ms ( ) ,
.. Default ::default ( )
} ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// The quick model says it's done after changing things: the bot's own model looks
// at the results and answers (the quick answer is set aside, never shown).
if msg . tool_calls . is_empty ( ) & & job . steer . lock ( ) . unwrap ( ) . is_empty ( ) & & plan . recheck_answer ( quick . is_some ( ) ) {
reuse_slot = true ;
continue ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
send ( StreamEvent ::StepDone ( msg . clone ( ) ) ) ;
let calls = msg . tool_calls . clone ( ) ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
plan . changed | = calls . iter ( ) . any ( | c | ! only_looks ( & c . name ) ) ;
let switched = plan . changed_route ( & calls , quick . is_some ( ) ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
history . push ( msg ) ;
if calls . is_empty ( ) {
// Something typed while it answered keeps the same turn going.
if job . steer . lock ( ) . unwrap ( ) . is_empty ( ) {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
closing . close ( crate ::oplog ::Outcome ::Completed ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
send ( StreamEvent ::Done ) ;
2026-11-06 15:21:04 +00:00
browsers . close_tabs ( ) ;
2026-11-06 15:12:53 +00:00
learn ( & job , & history , & cancel , & send ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
return ;
}
continue ;
}
2026-11-05 07:00:31 +00:00
// (Each call's computer was already kept to the chosen one, see keep_to_chosen.)
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// Whether any step this round failed, was refused or repeated (see StepPlan::after).
let trouble = std ::cell ::Cell ::new ( false ) ;
2026-11-09 05:31:25 +00:00
// The previous call of this step, to catch the same click sent twice in a row.
let mut previous : Option < ( String , String ) > = None ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
for asked in calls {
2026-11-09 05:31:25 +00:00
// The same click twice in one step opens a menu and closes it again (a Wi‑ Fi
// menu did exactly that): the second one isn't done.
let twice = matches! ( asked . name . as_str ( ) , " computer_use_click " | " browser_click " ) & & previous . as_ref ( ) = = Some ( & ( asked . name . clone ( ) , asked . args . clone ( ) ) ) ;
previous = Some ( ( asked . name . clone ( ) , asked . args . clone ( ) ) ) ;
if twice {
send ( StreamEvent ::ToolStatus ( asked . id . clone ( ) , ToolStatus ::Failed ) ) ;
let result = Message ::tool_result ( & asked . id , & asked . name , " 沒有執行:和這一步上一個點擊一模一樣。連點同一個地方兩次會把剛打開的選單又關掉;要點兩下請用 count: 2, 看完結果再決定下一步。 " ) ;
send ( StreamEvent ::ToolResult ( result . clone ( ) ) ) ;
history . push ( result ) ;
continue ;
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// What runs: the call as asked, on the chosen computer (see keep_to_chosen).
let call = ToolCall { args : runs_with . get ( & asked . id ) . cloned ( ) . unwrap_or_else ( | | asked . args . clone ( ) ) , .. asked . clone ( ) } ;
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
// 鬼打牆: the same screen or browser step again and again (a click on the same
// empty spot ten times). The third time it runs with a warning; after that it isn't
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// run at all, and the bot is told to change course. (Counted as the model asked,
// which is how the chat keeps them.)
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let mut repeat_note = String ::new ( ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
if let Some ( key ) = step_key ( & asked . name , & asked . args ) {
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let times = tried . iter ( ) . rev ( ) . take ( 10 ) . filter ( | k | * * k = = key ) . count ( ) + 1 ;
tried . push ( key ) ;
if times > = 4 {
let text = format! ( " 沒有執行:這一步你已經試了 {times} 次,畫面都沒有變成你要的樣子。不要再試同一步。改用別的方法:照截圖上的編號點、網頁改用 browser_read 拿連結直接 browser_open, 或用 hand_over_computer 請使用者幫忙。 " ) ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
trouble . set ( true ) ;
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
send ( StreamEvent ::ToolStatus ( call . id . clone ( ) , ToolStatus ::Failed ) ) ;
let result = Message ::tool_result ( & call . id , & call . name , & text ) ;
send ( StreamEvent ::ToolResult ( result . clone ( ) ) ) ;
history . push ( result ) ;
continue ;
}
if times = = 3 {
repeat_note = " ⚠️ 這一步你已經做了 3 次。如果畫面還不是你要的,下一步一定要換方法,不要再做同一步。 \n " . into ( ) ;
}
}
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
// The same look again and again with the same answer (a blank tab read eleven
// times in a row): the third time with a warning, after that not at all.
if repeat_note . is_empty ( ) {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let same = same_answers ( & history , & asked . name , & asked . args ) ;
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
if same > = 3 {
let text = format! ( " 沒有執行:同一個工具、同樣的參數你已經做了 {same} 次,每次結果都一模一樣,再做也不會不同。換一個做法(例如用 browser_open 打開要看的網址),或直接告訴使用者你看到了什麼。 " ) ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
trouble . set ( true ) ;
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
send ( StreamEvent ::ToolStatus ( call . id . clone ( ) , ToolStatus ::Failed ) ) ;
let result = Message ::tool_result ( & call . id , & call . name , & text ) ;
send ( StreamEvent ::ToolResult ( result . clone ( ) ) ) ;
history . push ( result ) ;
continue ;
}
if same = = 2 {
repeat_note = " ⚠️ 同一步你已經做了 2 次,結果都一模一樣。如果這次還一樣,就換一個做法。 \n " . into ( ) ;
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// Every step ends the same way: its status, then its result (with the notes) and
// any picture, shown and kept.
let finish = | history : & mut Vec < Message > , output : & str , status : ToolStatus , image : Option < String > | {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// A click that changed nothing on the screen is trouble too, though it "worked".
2026-11-09 09:10:25 +00:00
if matches! ( status , ToolStatus ::Failed | ToolStatus ::Denied ) | | ! repeat_note . is_empty ( ) | | output . contains ( crate ::computer ::NOOP_TAG ) {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
trouble . set ( true ) ;
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
job . note ( crate ::oplog ::Record ::ToolFinished { op : job . op . clone ( ) , call_id : call . id . clone ( ) , status : format ! ( " {status:?} " ) . to_lowercase ( ) } ) ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
send ( StreamEvent ::ToolStatus ( call . id . clone ( ) , status ) ) ;
let mut result = Message ::tool_result ( & call . id , & call . name , & format! ( " {repeat_note} {} " , noted ( & moved , & call . id , output ) ) ) ;
result . images . extend ( image ) ;
send ( StreamEvent ::ToolResult ( result . clone ( ) ) ) ;
history . push ( result ) ;
} ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// About to run, every check passed: the work log hears it first, with the
// arguments it runs with and whether it may run again after a crash.
let started = | | {
job . note ( crate ::oplog ::Record ::ToolStarted { op : job . op . clone ( ) , call_id : call . id . clone ( ) , name : call . name . clone ( ) , args : call . args . clone ( ) , replay : crate ::oplog ::replay_of ( & call . name ) } ) ;
send ( StreamEvent ::ToolStatus ( call . id . clone ( ) , ToolStatus ::Running ) ) ;
} ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if call . name = = REPLAY_TOOL {
let v = tools ::args_of ( & call . args ) ;
let name = v [ " name " ] . as_str ( ) . unwrap_or ( " " ) . trim ( ) . to_string ( ) ;
let skill = job . skills . iter ( ) . find ( | k | k . name = = name ) . cloned ( ) ;
match skill {
Some ( k ) if ! k . steps . is_empty ( ) = > {
// Another computer named for the replay: every step goes there.
let mut steps : std ::collections ::VecDeque < _ > = k . steps . clone ( ) . into ( ) ;
if let Some ( c ) = v [ " computer " ] . as_str ( ) . filter ( | c | ! c . trim ( ) . is_empty ( ) ) {
for st in steps . iter_mut ( ) {
let mut a = tools ::args_of ( & st . args ) ;
if a . get ( " computer " ) . is_some ( ) | | computer ::is_computer_tool ( & st . tool ) | | web ::is_web_tool ( & st . tool ) | | st . tool = = " shell " {
a [ " computer " ] = serde_json ::Value ::String ( c . to_string ( ) ) ;
st . args = a . to_string ( ) ;
}
}
}
let total = steps . len ( ) ;
finish ( & mut history , & format! ( " 開始重播「 {name} 」,共 {total} 步: AnyBot 會一步一步照做,每一步先在畫面上找到同一個東西,對不上會停下來交給你。 " ) , ToolStatus ::Done , None ) ;
replay = Some ( Replay { name , left : steps , total , looked : None } ) ;
}
Some ( _ ) = > finish ( & mut history , & format! ( " 「 {name} 」沒有存可以重播的步驟:用 use_skill 讀出做法,照著自己做。 " ) , ToolStatus ::Failed , None ) ,
None = > finish ( & mut history , & format! ( " 沒有叫「 {name} 」的技能。 " ) , ToolStatus ::Failed , None ) ,
}
continue ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
if appctl ::is_app_tool ( & call . name ) {
// App actions run on the window's side, which also asks for deletions.
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let ( rtx , rrx ) = channel ( ) ;
send ( StreamEvent ::AppAction ( AppCall { call_id : call . id . clone ( ) , name : call . name . clone ( ) , args : call . args . clone ( ) , reply : rtx } ) ) ;
let ( output , ok ) = loop {
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
match rrx . recv_timeout ( Duration ::from_millis ( 200 ) ) {
Ok ( r ) = > break r ,
Err ( std ::sync ::mpsc ::RecvTimeoutError ::Timeout ) = > continue ,
Err ( _ ) = > break ( " App 沒有回應 " . to_string ( ) , false ) ,
}
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// The person's own recording, shown back to the bot: nothing to ask.
if call . name = = " view_demo_step " {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let ( output , ok , img ) = crate ::teach ::view_step ( & job . demos , & call . args ) ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } , img ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// Public web pages need no computer and no permission.
2026-11-09 08:37:33 +00:00
// …and neither does AnyBot's own browser (like the reference assistant's own browser).
2026-11-06 08:51:25 +00:00
let at = web ::is_web_tool ( & call . name ) . then ( | | browser_place ( & job , & call . name , & call . args ) ) ;
if let Some ( ( place , on ) ) = at . filter ( | ( place , _ ) | call . name = = " web_fetch " | | bots_browser ( place , & call . name , & call . args ) ) {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
2026-11-06 15:21:04 +00:00
if call . name ! = " web_fetch " {
let profile = tools ::args_of ( & call . args ) . get ( " profile " ) . and_then ( | p | p . as_str ( ) ) . unwrap_or ( " " ) . trim ( ) . to_string ( ) ;
browsers . add ( match & place {
BrowserAt ::Here = > BotBrowser ::Here ( profile ) ,
BrowserAt ::Box = > BotBrowser ::Box ( profile ) ,
BrowserAt ::Device ( host ) = > BotBrowser ::Device ( host . clone ( ) , profile ) ,
} ) ;
}
2026-11-04 09:46:43 +00:00
// 裝置連結: the AnyBot browser of another computer (its own logins).
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
// (On this job's own tabs: another bot may be using the same browser.)
let args = with_job ( & call . args , & browsers . job ) ;
2026-11-05 08:47:28 +00:00
let ( output , ok , img ) = match place {
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
BrowserAt ::Device ( host ) = > match crate ::remote ::run_on_device ( & host , & call . name , & args , & job . via_name ) {
2026-11-04 09:46:43 +00:00
Ok ( v ) = > ( v [ " text " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) , v [ " ok " ] . as_bool ( ) . unwrap_or ( false ) , v [ " image " ] . as_str ( ) . map ( String ::from ) ) ,
Err ( e ) = > ( e , false , None ) ,
} ,
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
BrowserAt ::Box = > crate ::botbrowser ::in_box ( | | web ::run_full ( & call . name , & args , & cancel ) ) ,
BrowserAt ::Here = > web ::run_full ( & call . name , & args , & cancel ) ,
2026-11-04 09:46:43 +00:00
} ;
2026-11-05 08:47:28 +00:00
// Which computer's browser it was, like every other step.
let output = if on . is_empty ( ) { output } else { format! ( " [ {on} ] {output} " ) } ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } , img ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
let is_web = web ::is_web_tool ( & call . name ) ;
let is_screen = computer ::is_computer_tool ( & call . name ) ;
let is_mac = crate ::macapps ::is_mac_tool ( & call . name ) ;
let what = if is_mac {
format! ( " {} {} " , crate ::macapps ::label ( & call . name ) , crate ::macapps ::describe ( & call . name , & call . args ) )
} else if is_screen {
2026-11-05 10:25:46 +00:00
// A control by its number: name it from the app's list the bot last read.
let named = tools ::args_of ( & call . args ) [ " ref " ] . as_str ( ) . map ( str ::trim ) . filter ( | r | ! r . is_empty ( ) ) . and_then ( | r | element_named ( & history , r ) ) . map ( | e | format! ( " ( {e} ) " ) ) . unwrap_or_default ( ) ;
format! ( " {} {} {named} " , computer ::label ( & call . name ) , computer ::describe ( & call . name , & call . args ) )
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} else if is_web {
web ::describe ( & call . name , & call . args )
} else {
tools ::describe ( & call . name , & call . args )
} ;
// Ask the person right away (a pop-up), and wait for the answer.
let ask_person = | kind : ApprovalKind , reason : String , grant : String | -> ApprovalAnswer {
let ( atx , arx ) = channel ( ) ;
send ( StreamEvent ::ToolStatus ( call . id . clone ( ) , ToolStatus ::Waiting ) ) ;
send ( StreamEvent ::NeedApproval ( Approval { call_id : call . id . clone ( ) , tool : call . name . clone ( ) , reason , grant , reply : atx , kind } ) ) ;
wait_answer ( arx , & cancel )
} ;
2026-11-05 09:29:55 +00:00
// 需要處理 (the original's box hand-off): a step only the person can do — signing
// in, a captcha, a code from their phone. A card with the computer's live screen
// appears in the chat; they take over, do it, and say 「我完成了」 or 「跳過」. A
// message typed meanwhile answers it too.
if call . name = = tools ::HANDOFF_TOOL {
let instruction = tools ::args_of ( & call . args ) [ " instruction " ] . as_str ( ) . unwrap_or ( " " ) . trim ( ) . to_string ( ) ;
let ( output , status ) = match screen_target ( & job , & call . args ) {
Err ( e ) = > ( e , ToolStatus ::Failed ) ,
Ok ( _ ) if instruction . is_empty ( ) = > ( " 請在 instruction 寫清楚要使用者做什麼。 " . to_string ( ) , ToolStatus ::Failed ) ,
Ok ( t ) = > {
let ( atx , arx ) = channel ( ) ;
send ( StreamEvent ::ToolStatus ( call . id . clone ( ) , ToolStatus ::Waiting ) ) ;
send ( StreamEvent ::NeedApproval ( Approval { call_id : call . id . clone ( ) , tool : call . name . clone ( ) , reason : instruction . clone ( ) , grant : t . name . clone ( ) , reply : atx , kind : ApprovalKind ::Handoff } ) ) ;
let answer = loop {
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
match arx . recv_timeout ( Duration ::from_millis ( 200 ) ) {
Ok ( a ) = > break Some ( a ) ,
Err ( std ::sync ::mpsc ::RecvTimeoutError ::Timeout ) = > {
if ! job . steer . lock ( ) . unwrap ( ) . is_empty ( ) {
break None ;
}
}
Err ( _ ) = > break Some ( ApprovalAnswer ::Reject ) ,
}
} ;
match answer {
Some ( ApprovalAnswer ::Reject ) = > ( " 使用者選擇跳過這一步。不要再要求同一件事:沒有它就繼續其餘的工作,做不到的照實說。 " . to_string ( ) , ToolStatus ::Denied ) ,
Some ( _ ) = > ( format! ( " [ {} ] 使用者按了「我完成了」。先截圖確認畫面,再接著做。 " , t . name ) , ToolStatus ::Done ) ,
None = > ( " 使用者沒有按按鈕,而是傳了訊息(就在下面)。照他說的做。 " . to_string ( ) , ToolStatus ::Done ) ,
}
}
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , status , None ) ;
2026-11-05 09:29:55 +00:00
continue ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
// 生活卡片: drawn by the window from the call itself.
if call . name = = " show_card " {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , " 卡片已顯示給使用者。 " , ToolStatus ::Done , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// 表情回應: an emoji on the person's latest message (no words needed).
if call . name = = " react " {
let emoji : String = tools ::args_of ( & call . args ) [ " emoji " ] . as_str ( ) . unwrap_or ( " 👍 " ) . chars ( ) . take ( 4 ) . collect ( ) ;
send ( StreamEvent ::React ( emoji . clone ( ) ) ) ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & format! ( " 已在使用者的訊息上按了 {emoji} 。 " ) , ToolStatus ::Done , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// 傳檔案: put a file the bot made into the chat as a card the person can open.
if call . name = = " send_file " {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let path = tools ::args_of ( & call . args ) [ " path " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) ;
let local = match target_for ( & job , & call . args ) {
2026-11-05 07:48:20 +00:00
// Files ride to the chat only from this bot's own folder on the box.
Ok ( tools ::Target { place : tools ::Place ::Box { dir } , .. } ) = > match tools ::box_path ( std ::path ::Path ::new ( & dir ) , & path ) {
Ok ( p ) = > crate ::cloudbox ::copy_out ( & p ) ,
Err ( e ) = > Err ( e ) ,
} ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Ok ( tools ::Target { place : tools ::Place ::Sealed { root } , .. } ) = > Ok ( tools ::resolve ( & root , & path ) ) ,
Ok ( tools ::Target { place : tools ::Place ::Local { workdir } , .. } ) = > Ok ( tools ::resolve ( & workdir , & path ) ) ,
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
// Another of the person's computers: it sends the file over, then it is
// put in the chat like any other.
Ok ( tools ::Target { place : tools ::Place ::Device { host , dir } , .. } ) = > {
let args = serde_json ::json! ( { " path " : path , " workdir " : dir } ) . to_string ( ) ;
crate ::remote ::run_on_device ( & host , crate ::remote ::FETCH_FILE , & args , & job . via_name ) . and_then ( | v | {
let bytes = v [ " data " ] . as_str ( ) . and_then ( | d | base64 ::engine ::general_purpose ::STANDARD . decode ( d ) . ok ( ) ) . ok_or_else ( | | v [ " text " ] . as_str ( ) . unwrap_or ( " 那台電腦沒有把檔案傳過來 " ) . to_string ( ) ) ? ;
let dst = std ::env ::temp_dir ( ) . join ( format! ( " anybot-fetch- {} " , std ::process ::id ( ) ) ) . join ( v [ " name " ] . as_str ( ) . unwrap_or ( " 檔案 " ) ) ;
let _ = std ::fs ::create_dir_all ( dst . parent ( ) . unwrap_or ( std ::path ::Path ::new ( " . " ) ) ) ;
std ::fs ::write ( & dst , & bytes ) . map_err ( | e | e . to_string ( ) ) ? ;
Ok ( dst )
} )
}
2026-11-05 08:47:28 +00:00
Ok ( _ ) = > Err ( format! ( " 這台電腦上的檔案不能直接傳,請先複製到「 {} 」。 " , job . roles . own ) ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Err ( e ) = > Err ( e ) ,
} ;
2026-11-05 07:48:20 +00:00
// Secrets (SSH keys, the app's own data folder, .env…) never ride into the chat,
// same rule as read_file/edit_file.
let local = local . and_then ( | p | if tools ::is_ignored ( & p ) | | tools ::is_ignored ( std ::path ::Path ::new ( & path ) ) { Err ( " 這個檔案是機密的,不能傳到聊天。 " . to_string ( ) ) } else { Ok ( p ) } ) ;
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
let stored = local . and_then ( | p | crate ::attach ::store ( bot_id , & p ) ) ;
// (A file fetched from another computer was only parked here on its way.)
let _ = std ::fs ::remove_dir_all ( std ::env ::temp_dir ( ) . join ( format! ( " anybot-fetch- {} " , std ::process ::id ( ) ) ) ) ;
let ( output , ok ) = match stored {
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Ok ( a ) = > {
let out = format! ( " 已把「 {} 」傳到聊天裡( {} ),使用者點一下就能預覽。 " , a . name , crate ::attach ::human ( a . bytes ) ) ;
send ( StreamEvent ::Attach ( a ) ) ;
( out , true )
}
Err ( e ) = > ( e , false ) ,
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// 檔案預覽 / Bot 做的頁面: open what the bot made on the person's screen.
if call . name = = " open_for_person " {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let v = tools ::args_of ( & call . args ) ;
let path = v [ " path " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) ;
let on_box = match target_for ( & job , & call . args ) {
Ok ( t ) = > match t . place {
tools ::Place ::Box { dir } = > Some ( if path . starts_with ( '/' ) { path . clone ( ) } else { format! ( " {dir} / {path} " ) } ) ,
_ = > None ,
} ,
Err ( _ ) = > None ,
} ;
2026-11-05 08:47:28 +00:00
// On a server nobody sits at its screen: a link opens in the browser of the
// computer the person is using; a file is sent into the chat instead.
2026-11-05 11:41:49 +00:00
let person_now = roles_now ( & job ) . person ;
let person_host = person_now . as_ref ( ) . and_then ( | n | job . computers . iter ( ) . find ( | t | & t . name = = n ) ) . and_then ( | t | match & t . place {
2026-11-05 13:28:32 +00:00
tools ::Place ::Device { host , .. } = > Some ( host . clone ( ) ) ,
2026-11-05 08:47:28 +00:00
_ = > None ,
} ) ;
if job . roles . server {
let web = path . starts_with ( " http:// " ) | | path . starts_with ( " https:// " ) ;
let ( output , ok ) = match ( & person_host , web ) {
( Some ( host ) , true ) = > match crate ::remote ::run_on_device ( host , " browser_open " , & serde_json ::json! ( { " url " : path } ) . to_string ( ) , & job . via_name ) {
2026-11-05 11:41:49 +00:00
Ok ( v ) = > ( format! ( " [ {} ] {} " , person_now . clone ( ) . unwrap_or_default ( ) , v [ " text " ] . as_str ( ) . unwrap_or ( " " ) ) , v [ " ok " ] . as_bool ( ) . unwrap_or ( false ) ) ,
2026-11-05 08:47:28 +00:00
Err ( e ) = > ( e , false ) ,
} ,
( None , true ) = > ( " 不知道使用者現在在用哪台電腦,沒辦法打開給他看。把網址直接寫在回覆裡。 " . to_string ( ) , false ) ,
( _ , false ) = > ( " AnyBot 在伺服器上,檔案沒辦法直接在使用者的螢幕打開。改用 send_file 把檔案傳到聊天裡,使用者點一下就能看。 " . to_string ( ) , false ) ,
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } , None ) ;
2026-11-05 08:47:28 +00:00
continue ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let local = match ( & on_box , target_for ( & job , & call . args ) ) {
( Some ( p ) , _ ) = > crate ::cloudbox ::copy_out ( p ) ,
( None , Ok ( tools ::Target { place : tools ::Place ::Sealed { root } , .. } ) ) = > Ok ( tools ::resolve ( & root , & path ) ) ,
( None , Ok ( tools ::Target { place : tools ::Place ::Local { workdir } , .. } ) ) = > Ok ( tools ::resolve ( & workdir , & path ) ) ,
( None , _ ) if path . starts_with ( " http:// " ) | | path . starts_with ( " https:// " ) = > Ok ( std ::path ::PathBuf ::from ( & path ) ) ,
2026-11-05 08:47:28 +00:00
_ = > Err ( format! ( " 這台電腦上的檔案不能直接打開給使用者看,請先複製到「 {} 」或這台電腦。 " , job . roles . own ) ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} ;
2026-11-05 07:48:20 +00:00
// Secrets never open on the person's screen either.
let local = local . and_then ( | p | if tools ::is_ignored ( & p ) | | tools ::is_ignored ( std ::path ::Path ::new ( & path ) ) { Err ( " 這個檔案是機密的,不能打開。 " . to_string ( ) ) } else { Ok ( p ) } ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let ( output , ok ) = match local {
Ok ( p ) if p . exists ( ) | | path . starts_with ( " http " ) = > match crate ::preview ::system_open ( & p ) . status ( ) {
Ok ( st ) if st . success ( ) = > ( format! ( " 已在使用者的螢幕上打開 {} 。 " , p . display ( ) ) , true ) ,
_ = > ( format! ( " 打不開 {} " , p . display ( ) ) , false ) ,
} ,
Ok ( p ) = > ( format! ( " 找不到 {} " , p . display ( ) ) , false ) ,
Err ( e ) = > ( e , false ) ,
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// 打電話、傳簡訊、在 X 發文: speaks for the person, so asked every single time
// (like the original's send tools; no "always").
if crate ::phone ::is_phone_tool ( & call . name ) {
let what = crate ::phone ::describe ( & call . name , & call . args ) ;
let answer = ask_person ( ApprovalKind ::Action , what , String ::new ( ) ) ;
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
let ( output , status ) = if answer = = ApprovalAnswer ::Reject {
( " 使用者沒有允許。請不要再試,改問使用者。 " . to_string ( ) , ToolStatus ::Denied )
} else {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
match crate ::phone ::run ( & call . name , & call . args ) {
( t , true ) = > ( t , ToolStatus ::Done ) ,
( t , false ) = > ( t , ToolStatus ::Failed ) ,
}
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , status , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// 發布網站: put a page or folder the bot made on the web (asked once per site name).
if call . name = = " publish_site " {
let v = tools ::args_of ( & call . args ) ;
let path = v [ " path " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) ;
let name = v [ " name " ] . as_str ( ) . unwrap_or ( " " ) . trim ( ) . to_lowercase ( ) ;
let grant = format! ( " publish: {name} " ) ;
let mut allowed = grants . contains ( & grant ) ;
if ! allowed {
match ask_person ( ApprovalKind ::Action , format! ( " 把 {path} 發布成公開網站「 {name} 」(任何知道網址的人都看得到) " ) , grant . clone ( ) ) {
ApprovalAnswer ::Once = > allowed = true ,
ApprovalAnswer ::Always ( g ) = > {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant ( & mut grants , g ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
allowed = true ;
}
ApprovalAnswer ::Reject = > { }
}
}
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
let ( output , status ) = if ! allowed {
( " 使用者沒有允許發布。 " . to_string ( ) , ToolStatus ::Denied )
} else {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let local = match target_for ( & job , & call . args ) {
Ok ( tools ::Target { place : tools ::Place ::Box { dir } , .. } ) = > crate ::cloudbox ::copy_out ( & if path . starts_with ( '/' ) { path . clone ( ) } else { format! ( " {dir} / {path} " ) } ) ,
Ok ( tools ::Target { place : tools ::Place ::Sealed { root } , .. } ) = > Ok ( tools ::resolve ( & root , & path ) ) ,
Ok ( tools ::Target { place : tools ::Place ::Local { workdir } , .. } ) = > Ok ( tools ::resolve ( & workdir , & path ) ) ,
2026-11-05 08:47:28 +00:00
Ok ( _ ) = > Err ( format! ( " 這台電腦上的檔案不能直接發布,請先複製到「 {} 」。 " , job . roles . own ) ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Err ( e ) = > Err ( e ) ,
} ;
match local . and_then ( | p | crate ::remote ::publish_site ( & name , & p ) ) {
Ok ( url ) = > ( format! ( " 已發布: {url} " ) , ToolStatus ::Done ) ,
Err ( e ) = > ( e , ToolStatus ::Failed ) ,
}
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , status , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// 保險庫: the bot sees only metadata; filling asks the person first. A login on a
// site it wasn't saved for warns about look-alike sites and can't be "always";
// a card is asked every time (like the original's 允許一次 / 拒絕).
if call . name = = crate ::vault ::LIST_TOOL {
let output = crate ::vault ::list_for_bot ( & call . args ) ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , ToolStatus ::Done , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
if call . name = = crate ::vault ::TOOL | | call . name = = crate ::vault ::CARD_TOOL | | call . name = = crate ::vault ::CONTACT_TOOL {
enum Plan {
Clippy is clean on macOS and Linux, for the app and the relay
Every warning `cargo clippy --all-targets` gave (50 on macOS, 8 more only on
Linux, 1 in the relay), fixed without changing what anything does:
- long types get a name (Peer, Members, Imported, Joining, Mention, Stream…);
- the two big enum variants are boxed (the vault's login plan, joining an
account);
- a test now waits for the copy it relaunched (it left a zombie behind);
- a hint's format had a dead `if` whose two branches were both "";
- the rest is what clippy suggests: sort_by_key, as_chunks, is_none_or,
is_multiple_of, while let, from_ref, next_back, struct init, no casts to
the same type.
Checked: cargo test (177 macOS, 181 Linux), relay tests (11), remote_e2e
28/28, and the live Linux test (X11; Wayland clicks, keys, Chinese typing,
scroll and screenshots).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-07 02:10:28 +00:00
Login ( Box < crate ::vault ::LoginPlan > ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Card ,
Contact ,
}
let planned : Result < ( Plan , String , String ) , String > = if call . name = = crate ::vault ::TOOL {
crate ::vault ::plan_login ( & call . args ) . map ( | pl | {
let q = pl . question ( ) ;
let grant = if pl . matched { format! ( " login: {} " , pl . host ) } else { String ::new ( ) } ;
Clippy is clean on macOS and Linux, for the app and the relay
Every warning `cargo clippy --all-targets` gave (50 on macOS, 8 more only on
Linux, 1 in the relay), fixed without changing what anything does:
- long types get a name (Peer, Members, Imported, Joining, Mention, Stream…);
- the two big enum variants are boxed (the vault's login plan, joining an
account);
- a test now waits for the copy it relaunched (it left a zombie behind);
- a hint's format had a dead `if` whose two branches were both "";
- the rest is what clippy suggests: sort_by_key, as_chunks, is_none_or,
is_multiple_of, while let, from_ref, next_back, struct init, no casts to
the same type.
Checked: cargo test (177 macOS, 181 Linux), relay tests (11), remote_e2e
28/28, and the live Linux test (X11; Wayland clicks, keys, Chinese typing,
scroll and screenshots).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-07 02:10:28 +00:00
( Plan ::Login ( Box ::new ( pl ) ) , q , grant )
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} )
} else if call . name = = crate ::vault ::CARD_TOOL {
crate ::vault ::plan_card ( & call . args ) . map ( | ( host , c ) | ( Plan ::Card , format! ( " 用保險庫的卡片( {} •••• {} )在 {host} 填入付款資料( Bot 看不到卡號和安全碼) " , c . display_title ( ) , c . last4 ) , String ::new ( ) ) )
} else {
crate ::vault ::plan_contact ( & call . args ) . map ( | ( host , c ) | ( Plan ::Contact , format! ( " 把保險庫的聯絡資訊「 {} 」填進 {host} 的表單 " , c . display_title ( ) ) , format! ( " contact: {host} " ) ) )
} ;
let ( output , status ) = match planned {
Err ( e ) = > ( e , ToolStatus ::Failed ) ,
Ok ( ( plan , question , grant ) ) = > {
let mut allowed = ! grant . is_empty ( ) & & grants . contains ( & grant ) ;
if ! allowed {
match ask_person ( ApprovalKind ::Action , question , grant . clone ( ) ) {
ApprovalAnswer ::Once = > allowed = true ,
ApprovalAnswer ::Always ( g ) = > {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant ( & mut grants , g ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
allowed = true ;
}
ApprovalAnswer ::Reject = > { }
}
}
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
if ! allowed {
( " 使用者沒有允許使用保險庫的這個項目。請改問使用者。 " . to_string ( ) , ToolStatus ::Denied )
} else {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let r = match plan {
Plan ::Login ( pl ) = > crate ::vault ::fill_login ( & pl , tools ::args_of ( & call . args ) [ " submit " ] . as_bool ( ) . unwrap_or ( false ) ) ,
Plan ::Card = > crate ::vault ::fill_card ( & call . args ) ,
Plan ::Contact = > crate ::vault ::fill_contact ( & call . args ) ,
} ;
match r {
Ok ( t ) = > ( t , ToolStatus ::Done ) ,
Err ( e ) = > ( e , ToolStatus ::Failed ) ,
}
}
}
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , status , None ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
// MCP connectors: asked once per server (or always allowed), like the original's
// MCP allow-list; direct mode doesn't ask.
if crate ::mcp ::is_mcp_tool ( & call . name ) {
let ( server , tool ) = crate ::mcp ::resolve ( & call . name ) . unwrap_or_default ( ) ;
let grant = format! ( " mcp: {server} " ) ;
let args_short : String = call . args . chars ( ) . take ( 160 ) . collect ( ) ;
let mut allowed = job . direct . load ( Ordering ::Relaxed ) | | grants . contains ( & grant ) ;
if ! allowed {
match ask_person ( ApprovalKind ::Action , format! ( " 使用連接器「 {server} 」的工具: {tool} {args_short} " ) , grant . clone ( ) ) {
ApprovalAnswer ::Once = > allowed = true ,
ApprovalAnswer ::Always ( g ) = > {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant ( & mut grants , g ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
allowed = true ;
}
ApprovalAnswer ::Reject = > { }
}
}
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
let ( output , status , img ) = if allowed {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let ( o , ok , img ) = crate ::mcp ::call ( & call . name , & call . args ) ;
( o , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } , img )
} else {
( " 使用者拒絕了這個操作。請不要再試同樣的操作,改問使用者想怎麼做。 " . to_string ( ) , ToolStatus ::Denied , None )
} ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , status , img ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
continue ;
}
2026-11-05 08:47:28 +00:00
// The person's own apps and everyday browser are on the computer they are using:
// the one they name, else the one they typed on (else this one, when it is theirs).
let person_target = | | -> Result < tools ::Target , String > {
if names_a_computer ( & call . args ) {
return target_for ( & job , & call . args ) ;
}
2026-11-05 11:41:49 +00:00
roles_now ( & job )
2026-11-05 08:47:28 +00:00
. person
2026-11-05 11:41:49 +00:00
. and_then ( | n | job . computers . iter ( ) . find ( | t | t . name = = n ) )
2026-11-05 08:47:28 +00:00
. cloned ( )
. ok_or_else ( | | format! ( " 不知道使用者現在在用哪台電腦。請問他要在哪台做(可以用: {} )。 " , job . computers . iter ( ) . map ( | t | t . name . as_str ( ) ) . collect ::< Vec < _ > > ( ) . join ( " 、 " ) ) )
} ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let target = if is_screen {
screen_target ( & job , & call . args )
2026-11-05 08:47:28 +00:00
} else if is_mac | | is_web {
// (Other browser steps went their own way above; this is the person's own Chrome.)
person_target ( )
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} else {
target_for ( & job , & call . args )
} ;
let on_box = matches! ( target . as_ref ( ) . map ( | t | & t . place ) , Ok ( tools ::Place ::Box { .. } ) ) ;
let on_device = match target . as_ref ( ) . map ( | t | & t . place ) {
2026-11-05 13:28:32 +00:00
Ok ( tools ::Place ::Device { host , .. } ) = > Some ( host . clone ( ) ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
_ = > None ,
} ;
// Permission to use computers at all, and this Mac in particular, is asked for
// when it's needed instead of the step failing quietly.
let mut gate : Result < ( ) , String > = Ok ( ( ) ) ;
2026-11-04 23:56:19 +00:00
// The person just said yes to this very step in a switch's pop-up (it shows the
// step): no second pop-up for it.
let mut said_yes = false ;
2026-11-09 08:37:33 +00:00
// 直接操作: like the reference assistant after its one-time setup ("讓 the reference assistant 在你的電腦上開啟檔案和
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
// 執行任務。自動審核仍會先檢查所有操作", ENHi8c), nothing is asked except what the
// safety checks and the person's "一律詢問" rules stop.
let direct = job . direct . load ( Ordering ::Relaxed ) ;
if direct {
tools_on = true ;
local_on = true ;
}
if let Ok ( t ) = & target {
if ! tools_on {
let reason = format! ( " 讓 {} 使用電腦工具: {} \n (在 {} 上) " , job . via_name , what , t . name ) ;
match ask_person ( ApprovalKind ::EnableTools , reason , " tools:on " . into ( ) ) {
2026-11-04 23:56:19 +00:00
ApprovalAnswer ::Once = > said_yes = true ,
ApprovalAnswer ::Always ( _ ) = > {
tools_on = true ;
said_yes = true ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
ApprovalAnswer ::Reject = > gate = Err ( " 使用者沒有允許使用電腦工具。請改用文字回答,或問使用者想怎麼做。 " . into ( ) ) ,
}
}
// An admin lock (policy) refuses outright instead of asking.
if gate . is_ok ( ) & & matches! ( t . place , tools ::Place ::Local { .. } ) & & job . local_locked {
2026-11-05 08:47:28 +00:00
gate = Err ( format! ( " 「在這台電腦上執行」設成了永不允許(或被管理員關閉),不能在「 {} 」上執行。請改用「 {} 」。 " , t . name , job . roles . own ) ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
if gate . is_ok ( ) & & matches! ( t . place , tools ::Place ::Local { .. } ) & & ! local_on {
2026-11-05 08:47:28 +00:00
let reason = format! ( " 在「 {} 」上執行任務,而不只是在它自己的電腦上: {} {what} " , t . name , if is_web { format! ( " {} " , web ::label ( & call . name ) ) } else { String ::new ( ) } ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
match ask_person ( ApprovalKind ::EnableThisComputer , reason , " local:on " . into ( ) ) {
2026-11-04 23:56:19 +00:00
ApprovalAnswer ::Once = > said_yes = true ,
ApprovalAnswer ::Always ( _ ) = > {
local_on = true ;
said_yes = true ;
}
2026-11-05 08:47:28 +00:00
ApprovalAnswer ::Reject = > gate = Err ( format! ( " 使用者沒有允許在「 {} 」上執行。可以改用「 {} 」,或問使用者。 " , t . name , job . roles . own ) ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
}
}
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
let decision = match ( & target , & gate ) {
// Looking at the screen and using the mouse/keyboard each ask once (or always).
// The Mac's own apps: reading asks once per app, writing once per app (or each
// time), sending every single time (the original never keeps a grant for it).
( Ok ( _ ) , Ok ( ( ) ) ) if is_mac = > {
let app = crate ::macapps ::app_of ( & call . name ) ;
if crate ::macapps ::is_send ( & call . name ) & & call . name = = crate ::macapps ::SEND_MESSAGE & & grants . iter ( ) . any ( | g | g = = " messages:no-ask " ) {
Decision ::Allow
} else if crate ::macapps ::is_send ( & call . name ) {
Decision ::Ask { reason : format ! ( " 用你的「{app}」{what} " ) , grant : String ::new ( ) }
} else if crate ::macapps ::is_write ( & call . name ) {
let grant = format! ( " macapp-write: {app} " ) ;
if grants . contains ( & grant ) { Decision ::Allow } else { Decision ::Ask { reason : format ! ( " 讓 {} 修改你的「{app}」:{what} " , job . via_name ) , grant } }
} else {
let grant = format! ( " macapp: {app} " ) ;
if grants . contains ( & grant ) | | grants . contains ( & format! ( " macapp-write: {app} " ) ) { Decision ::Allow } else { Decision ::Ask { reason : format ! ( " 讓 {} 讀取你的「{app}」:{what} " , job . via_name ) , grant } }
}
}
// Its own desktop: nothing to ask, like the original's box.
( Ok ( _ ) , Ok ( ( ) ) ) if is_screen & & on_box = > Decision ::Allow ,
( Ok ( _ ) , Ok ( ( ) ) ) if is_screen & & call . name = = " computer_use_check_permissions " = > Decision ::Allow ,
// 裝置連結: another of the person's computers; seeing and operating it ask once each.
( Ok ( t ) , Ok ( ( ) ) ) if is_screen & & on_device . is_some ( ) = > {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let looks = matches! ( call . name . as_str ( ) , " computer_use_screenshot " | " computer_use_zoom " | " computer_use_show " ) ;
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
let grant = if looks { format! ( " device-see: {} " , t . name ) } else { format! ( " device-control: {} " , t . name ) } ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
if grants . contains ( & grant ) {
Decision ::Allow
} else {
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
Decision ::Ask { reason : format ! ( " {}「{}」的螢幕:{what} " , if looks { " 看 " } else { " 操作 " } , t . name ) , grant }
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
}
( Ok ( _ ) , Ok ( ( ) ) ) if is_screen = > {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if matches! ( call . name . as_str ( ) , " computer_use_screenshot " | " computer_use_zoom " | " computer_use_show " ) {
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
if grants . iter ( ) . any ( | g | g = = " computer:see " ) {
Decision ::Allow
} else {
Decision ::Ask { reason : format ! ( " 看你的螢幕畫面:{what} " ) , grant : " computer:see " . into ( ) }
}
} else {
2026-11-09 08:37:33 +00:00
// Per app, like the reference assistant: 「the reference assistant 可以看到 {app},並在其中點擊和輸入」.
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let app = computer ::target_app ( & call . name , & call . args ) . unwrap_or_else ( | | " 目前的 App " . into ( ) ) ;
let grant = format! ( " app: {app} " ) ;
if grants . iter ( ) . any ( | g | g = = " computer:control " | | * g = = grant ) {
Decision ::Allow
} else {
Decision ::Ask { reason : format ! ( " 讓 {} 看到「{app}」,並在其中點擊和輸入:{what} " , job . via_name ) , grant }
}
}
}
// Opening a link is visible and harmless; reading a logged-in tab asks first.
( Ok ( _ ) , Ok ( ( ) ) ) if is_web = > {
if call . name = = " browser_read " & & web ::read_needs_permission ( & call . args ) & & ! grants . iter ( ) . any ( | g | g = = " browser:read " ) {
Decision ::Ask { reason : " 讀取你瀏覽器目前分頁的內容(包含已登入的頁面):目前的分頁 " . into ( ) , grant : " browser:read " . into ( ) }
} else {
Decision ::Allow
}
}
( Ok ( t ) , Ok ( ( ) ) ) = > tools ::decide_on ( & call . name , & call . args , t , & grants ) ,
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// Can't reach that computer: reported as a failed step below, not a safety block.
( Err ( _ ) , _ ) = > Decision ::Allow ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
// The person said no to the pop-up: nothing more to ask.
( _ , Err ( _ ) ) = > Decision ::Allow ,
} ;
// 自動審核: when the rules say yes, continue without asking (JOecNC);
// when an "always ask" rule matches, stop and ask (Pca/iV).
let mut review_note = String ::new ( ) ;
let direct = job . direct . load ( Ordering ::Relaxed ) ;
let decision = match ( decision , & job . review ) {
// Direct mode: go ahead, unless an "always ask" rule matches.
( Decision ::Ask { reason , grant } , review ) if direct = > {
2026-11-06 11:47:43 +00:00
// Only looking (a screenshot, a page or file read) changes nothing, so no
// "always ask" rule (deleting, paying, posting…) can be about it: no model
// call to check (it cost every look at the person's Mac 10– 20 s).
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let looks = matches! ( call . name . as_str ( ) , " computer_use_screenshot " | " computer_use_zoom " | " computer_use_show " | " browser_read " | " read_file " | " list_dir " | " search " ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let verdict = match review {
2026-11-06 11:47:43 +00:00
Some ( rv ) if ! rv . ask . is_empty ( ) & & ! looks = > review_action ( & job , & Review { allow : vec ! [ ] , ask : rv . ask . clone ( ) } , & reason , & cancel ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
_ = > ReviewVerdict ::NoMatch ,
} ;
match verdict {
ReviewVerdict ::Ask ( rule ) = > Decision ::Ask { reason : format ! ( " {reason}(符合你的「一律詢問」規則:{rule}) " ) , grant : String ::new ( ) } ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// Nobody could check the person's "always ask" rules: ask, never go ahead.
ReviewVerdict ::Failed = > Decision ::Ask { reason : format ! ( " {reason}{REVIEW_FAILED_NOTE} " ) , grant : String ::new ( ) } ,
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// (No note on the result: the model doesn't need it, and it was on every step.)
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
_ = > {
let _ = grant ;
Decision ::Allow
}
}
}
( Decision ::Ask { reason , grant } , Some ( rv ) ) = > match review_action ( & job , rv , & reason , & cancel ) {
ReviewVerdict ::Allow ( rule ) = > {
review_note = format! ( " (自動審核:符合「 {rule} 」,直接允許) \n " ) ;
Decision ::Allow
}
ReviewVerdict ::Ask ( rule ) = > Decision ::Ask { reason : format ! ( " {reason}(符合你的「一律詢問」規則:{rule}) " ) , grant : String ::new ( ) } ,
ReviewVerdict ::NoMatch = > Decision ::Ask { reason , grant } ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
ReviewVerdict ::Failed = > Decision ::Ask { reason : format ! ( " {reason}{REVIEW_FAILED_NOTE} " ) , grant } ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} ,
( d , _ ) = > d ,
} ;
// Sending mail or messages is approved one by one, even in direct mode.
let decision = if is_mac & & crate ::macapps ::is_send ( & call . name ) & & gate . is_ok ( ) & & target . is_ok ( ) {
review_note . clear ( ) ;
Decision ::Ask { reason : format ! ( " 用你的「{}」{what}(每一封都會先問你) " , crate ::macapps ::app_of ( & call . name ) ) , grant : String ::new ( ) }
} else {
decision
} ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// AppleScript that runs outside the sandbox can drive the person's apps: asked the
// first time even in direct mode, until they choose 「一律允許」.
let on_persons = matches! ( target . as_ref ( ) . map ( | t | & t . place ) , Ok ( tools ::Place ::Local { .. } | tools ::Place ::Device { .. } ) ) ;
let script = tools ::args_of ( & call . args ) [ " command " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) ;
let decision = if call . name = = " shell " & & on_persons & & gate . is_ok ( ) & & ! matches! ( decision , Decision ::Deny ( _ ) ) & & tools ::osascript_outside ( & script ) . is_ok ( ) & & ! grants . iter ( ) . any ( | g | g = = tools ::OSASCRIPT_GRANT ) {
review_note . clear ( ) ;
let short : String = script . chars ( ) . take ( 200 ) . collect ( ) ;
Decision ::Ask { reason : format ! ( " 讓 {} 在沙盒外執行 AppleScript( 可以指揮你電腦上的 App) : {short} " , job . via_name ) , grant : tools ::OSASCRIPT_GRANT . into ( ) }
} else {
decision
} ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let approved = match decision {
_ if gate . is_err ( ) = > Err ( ( gate . clone ( ) . unwrap_err ( ) , ToolStatus ::Denied ) ) ,
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
_ if target . is_err ( ) = > Err ( ( target . clone ( ) . err ( ) . unwrap_or_default ( ) , ToolStatus ::Failed ) ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Decision ::Allow = > Ok ( ( ) ) ,
2026-11-04 23:56:19 +00:00
// Sending (Messages, mail…) still asks every time, even right after a switch.
Decision ::Ask { .. } if said_yes & & ! ( is_mac & & crate ::macapps ::is_send ( & call . name ) ) = > Ok ( ( ) ) ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Decision ::Deny ( r ) = > Err ( ( format! ( " 被安全規則擋下: {r} " ) , ToolStatus ::Denied ) ) ,
Decision ::Ask { reason , grant } = > {
let machine = target . as_ref ( ) . map ( | t | t . name . clone ( ) ) . unwrap_or_default ( ) ;
let reason = format! ( " {reason} \n (在 {machine} 上) " ) ;
match ask_person ( ApprovalKind ::Action , reason , grant ) {
ApprovalAnswer ::Once = > Ok ( ( ) ) ,
ApprovalAnswer ::Always ( g ) = > {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant ( & mut grants , g ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
Ok ( ( ) )
}
ApprovalAnswer ::Reject = > Err ( ( " 使用者拒絕了這個操作。請不要再試同樣的操作,改問使用者想怎麼做。 " . into ( ) , ToolStatus ::Denied ) ) ,
}
}
} ;
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
let mut image = None ;
let ( output , status ) = match ( approved , & target ) {
( Ok ( ( ) ) , Ok ( t ) ) = > {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started ( ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let ids = tools ::RunIds { conversation : bot_id . to_string ( ) , request : call . id . clone ( ) } ;
2026-11-05 08:47:28 +00:00
let ( out , ok ) = if is_mac & & on_device . is_none ( ) {
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
crate ::macapps ::run ( & call . name , & call . args )
2026-11-05 08:47:28 +00:00
} else if let ( true , Some ( host ) ) = ( is_screen | | is_web | | is_mac , on_device . as_ref ( ) ) {
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
match crate ::remote ::run_on_device ( host , & call . name , & call . args , & job . via_name ) {
Ok ( v ) = > {
image = v [ " image " ] . as_str ( ) . map ( String ::from ) ;
( v [ " text " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) , v [ " ok " ] . as_bool ( ) . unwrap_or ( false ) )
}
Err ( e ) = > ( e , false ) ,
}
} else if is_screen & & on_box {
match crate ::cloudbox ::computer_run ( & call . name , & tools ::args_of ( & call . args ) ) {
Ok ( ( out , img ) ) = > {
image = img ;
( out , true )
}
Err ( e ) = > ( e , false ) ,
}
} else if is_screen {
let ( out , ok , img ) = computer ::run_by ( & call . name , & call . args , & job . via_name ) ;
image = img ;
( out , ok )
} else if is_web {
web ::run ( & call . name , & call . args , & cancel )
} else {
tools ::run_on ( & call . name , & call . args , t , & cancel , & ids )
} ;
( format! ( " {review_note} [ {} ] {out} " , t . name ) , if ok { ToolStatus ::Done } else { ToolStatus ::Failed } )
}
( Ok ( ( ) ) , Err ( e ) ) = > ( e . clone ( ) , ToolStatus ::Failed ) ,
( Err ( x ) , _ ) = > x ,
} ;
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
// 給使用者看畫面: the picture goes into the chat for the person (the bot still sees it).
if call . name = = " computer_use_show " {
if let Some ( bytes ) = image . as_deref ( ) . and_then ( | b | base64 ::engine ::general_purpose ::STANDARD . decode ( b ) . ok ( ) ) {
2026-11-06 11:47:43 +00:00
// (No time in the name: a brain on a server keeps UTC, and the chat shows when.)
let name = format! ( " {} 的畫面.jpg " , target . as_ref ( ) . map ( | t | t . name . as_str ( ) ) . unwrap_or ( " 電腦 " ) ) ;
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
if let Ok ( a ) = crate ::attach ::store_bytes ( bot_id , & name , & bytes ) {
send ( StreamEvent ::Attach ( a ) ) ;
}
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish ( & mut history , & output , status , image ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
if cancel . load ( Ordering ::Relaxed ) {
return ;
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
plan . after ( trouble . get ( ) | | switched ) ;
if scripted . is_some ( ) {
if let Some ( r ) = replay . as_ref ( ) {
let k = r . total - r . left . len ( ) ;
let note = if trouble . get ( ) {
Some ( format! ( " \n \n (重播「 {} 」在第 {k} 步出了問題,後面的步驟沒有做。接下來由你判斷怎麼做完。) " , r . name ) )
} else if r . left . is_empty ( ) {
Some ( format! ( " \n \n (重播「 {} 」做完了,共 {} 步。看一下結果,確認真的做到了再回答使用者。) " , r . name , r . total ) )
} else {
None
} ;
if let Some ( n ) = note {
if let Some ( m ) = history . last_mut ( ) . filter ( | m | m . role = = Role ::Tool ) {
m . text . push_str ( & n ) ;
}
replay = None ;
plan . after ( true ) ;
}
}
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}
send ( StreamEvent ::NextStep ( job . via . clone ( ) ) ) ;
send ( StreamEvent ::StepDone ( Message ::assistant ( & format! ( " (這次已經用了 {MAX_STEPS} 個步驟,先停在這裡。要繼續的話,回覆「繼續」。) " ) , & job . via ) ) ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
closing . close ( crate ::oplog ::Outcome ::Completed ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
send ( StreamEvent ::Done ) ;
} ) ;
}
#[ cfg(test) ]
mod tests {
use super ::* ;
use crate ::model ::ToolCall ;
use crate ::providers ::ProviderKind ;
2026-11-05 08:47:28 +00:00
/// A job with nothing in it, for tests that only need a few fields.
fn test_job ( ) -> Job {
Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : String ::new ( ) , api_key : String ::new ( ) , model : String ::new ( ) , system : String ::new ( ) , history : vec ! [ ] , tools : vec ! [ ] , session : String ::new ( ) } ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model : None ,
2026-11-05 08:47:28 +00:00
computers : vec ! [ ] ,
default_computer : String ::new ( ) ,
roles : tools ::Roles ::default ( ) ,
local_execution : false ,
grants : vec ! [ ] ,
context_tokens : DEFAULT_CONTEXT_TOKENS ,
via : String ::new ( ) ,
via_name : String ::new ( ) ,
tools_enabled : true ,
direct : Arc ::new ( AtomicBool ::new ( false ) ) ,
local_locked : false ,
steer : Arc ::new ( Mutex ::new ( vec! [ ] ) ) ,
review : None ,
demos : vec ! [ ] ,
2026-11-06 15:12:53 +00:00
skills : vec ! [ ] ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op : String ::new ( ) ,
log : None ,
2026-11-05 08:47:28 +00:00
}
}
2026-11-05 07:48:20 +00:00
#[ test ]
fn computer_words_need_word_boundaries ( ) {
// "mac" as its own word names this computer; inside longer words (macOS, a device
// named Ann inside "anniversary") it must not — that is what made the guard fire on
// text the person never aimed at any computer.
assert! ( finds ( " 用我的 mac 執行 " , " mac " ) ) ;
assert! ( finds ( " mac 上開檔案 " , " mac " ) ) ;
assert! ( ! finds ( " 在 macOS 上開檔案 " , " mac " ) ) ;
assert! ( ! finds ( " telegram 傳給我 " , " mac " ) ) ;
assert! ( finds ( " 我的電腦好慢 " , " 我的電腦 " ) ) ;
assert! ( ! finds ( " 你的虛擬電腦很吵 " , " bot 的電腦 " ) ) ;
assert! ( finds ( " 在 bot 的電腦跑 " , " bot 的電腦 " ) ) ;
2026-11-05 08:47:28 +00:00
let roles = tools ::Roles { own : " Master 的電腦 " . into ( ) , person : Some ( " 王小明的Mac mini " . into ( ) ) , server : true } ;
2026-11-05 13:28:32 +00:00
let mac = tools ::Target { name : " 王小明的Mac mini " . into ( ) , place : tools ::Place ::Device { host : " m " . into ( ) , dir : String ::new ( ) } } ;
2026-11-05 08:47:28 +00:00
assert! ( mentions ( " 幫我在 Mac 截圖 " , & mac , & roles ) ) ;
assert! ( ! mentions ( " 在 macOS 的登入機制下,幫 Bot 的電腦截圖 " , & mac , & roles ) ) ;
}
Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 11:37:45 +00:00
#[ test ]
fn the_same_answer_again_and_again_is_counted ( ) {
use crate ::model ::ToolCall ;
let call = | id : & str , args : & str | Message { role : Role ::Assistant , tool_calls : vec ! [ ToolCall { id : id . into ( ) , name : " browser_read " . into ( ) , args : args . into ( ) , status : ToolStatus ::Done } ] , .. Default ::default ( ) } ;
let mut h = vec! [ Message ::user ( " 看得到我的 Threads 嗎? " ) ] ;
for i in 0 .. 3 {
h . push ( call ( & format! ( " c {i} " ) , r # "{"computer":"Bot 的電腦"}"# ) ) ;
let warn = if i = = 2 { " ⚠️ 同一步你已經做了 2 次,結果都一模一樣。 \n " } else { " " } ;
h . push ( Message ::tool_result ( & format! ( " c {i} " ) , " browser_read " , & format! ( " {warn} 網址: about:blank " ) ) ) ;
}
assert_eq! ( same_answers ( & h , " browser_read " , r # "{ "computer": "Bot 的電腦" }"# ) , 3 ) ;
// Another page in between: counting starts again.
h . push ( call ( " c9 " , r # "{"computer":"Bot 的電腦"}"# ) ) ;
h . push ( Message ::tool_result ( " c9 " , " browser_read " , " 網址: https://www.threads.com/ " ) ) ;
assert_eq! ( same_answers ( & h , " browser_read " , r # "{"computer":"Bot 的電腦"}"# ) , 1 ) ;
// Waiting on a terminal is allowed to look the same.
assert_eq! ( same_answers ( & h , " read_terminal " , " {} " ) , 0 ) ;
}
2026-11-06 08:51:25 +00:00
#[ test ]
fn long_turns_check_in_instead_of_stopping ( ) {
let mut h = vec! [ Message ::user ( " 修好它 " ) , Message ::tool_result ( " t1 " , " shell " , " ok " ) ] ;
check_in ( & mut h , CHECK_IN_STEPS - 1 ) ;
assert_eq! ( h [ 1 ] . text , " ok " ) ;
check_in ( & mut h , CHECK_IN_STEPS ) ;
assert! ( h [ 1 ] . text . starts_with ( " ok " ) & & h [ 1 ] . text . contains ( & format! ( " 已經做了 {CHECK_IN_STEPS} 步 " ) ) ) ;
// Only a tool result carries it (a steer the person typed is left as they wrote it).
let mut h = vec! [ Message ::user ( " 改用別台 " ) ] ;
check_in ( & mut h , CHECK_IN_STEPS * 2 ) ;
assert_eq! ( h [ 0 ] . text , " 改用別台 " ) ;
}
#[ test ]
fn the_boxs_browser_is_always_the_bots_own ( ) {
// A cloud brain has no Chrome of its own; browser steps for the box still go to the box.
for args in [ r # "{}"# , r # "{"url":"https://www.threads.com"}"# , r # "{"profile":"你的瀏覽器"}"# ] {
assert! ( bots_browser ( & BrowserAt ::Box , " browser_open " , args ) ) ;
assert! ( bots_browser ( & BrowserAt ::Box , " browser_read " , args ) ) ;
assert! ( crate ::botbrowser ::in_box ( | | ! web ::uses_person_browser ( args ) ) ) ;
}
}
/// A made-up model on a local port that answers each request with the next reply: a tool
/// call (name, JSON arguments) or, for an empty name, words.
fn fake_model ( replies : Vec < ( & 'static str , serde_json ::Value ) > ) -> u16 {
use std ::io ::{ Read , Write } ;
let l = std ::net ::TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
std ::thread ::spawn ( move | | {
for ( i , ( name , v ) ) in replies . into_iter ( ) . enumerate ( ) {
let delta = if name . is_empty ( ) {
serde_json ::json! ( { " content " : v . as_str ( ) . unwrap_or ( " " ) } )
} else {
serde_json ::json! ( { " tool_calls " : [ { " index " : 0 , " id " : format ! ( " c{i} " ) , " function " : { " name " : name , " arguments " : v . to_string ( ) } } ] } )
} ;
let body = format! ( " data: {} \n \n data: [DONE] \n \n " , serde_json ::json! ( { " choices " : [ { " delta " : delta } ] } ) ) ;
let ( mut s , _ ) = l . accept ( ) . unwrap ( ) ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 8192 ] ;
loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break ;
}
}
}
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( body . as_bytes ( ) ) ;
}
} ) ;
port
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// A model server answering `replies` (each a stream delta) in order, recording the model
/// each request asked for.
fn recording_model ( replies : Vec < serde_json ::Value > ) -> ( u16 , Arc < Mutex < Vec < String > > > ) {
use std ::io ::{ Read , Write } ;
let l = std ::net ::TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
let asked = Arc ::new ( Mutex ::new ( Vec ::< String > ::new ( ) ) ) ;
let seen = asked . clone ( ) ;
std ::thread ::spawn ( move | | {
for delta in replies {
let Ok ( ( mut s , _ ) ) = l . accept ( ) else { return } ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 16384 ] ;
let body = loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break String ::new ( ) ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break t [ end + 4 .. ] . to_string ( ) ;
}
}
} ;
let v : serde_json ::Value = serde_json ::from_str ( & body ) . unwrap_or_default ( ) ;
seen . lock ( ) . unwrap ( ) . push ( v [ " model " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) ) ;
let out = format! ( " data: {} \n \n data: [DONE] \n \n " , serde_json ::json! ( { " choices " : [ { " delta " : delta } ] } ) ) ;
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( out . as_bytes ( ) ) ;
}
} ) ;
( port , asked )
}
fn tool_delta ( id : & str , name : & str , args : serde_json ::Value ) -> serde_json ::Value {
serde_json ::json! ( { " tool_calls " : [ { " index " : 0 , " id " : id , " function " : { " name " : name , " arguments " : args . to_string ( ) } } ] } )
}
2026-11-09 08:50:18 +00:00
/// The quick model's `computer` tool names controls only: looking a number up in the list
/// is where it went wrong. The bot's own model keeps `ref`; expand_call maps the one tool
/// back to the internal per-action tools everything downstream runs on.
2026-11-09 06:03:40 +00:00
#[ test ]
2026-11-09 08:50:18 +00:00
fn the_quick_models_screen_tool_names_controls_only ( ) {
2026-11-09 06:03:40 +00:00
let specs = crate ::computer ::specs_for ( & [ " C " . to_string ( ) ] , " C " , true ) ;
2026-11-09 08:50:18 +00:00
assert_eq! ( specs . len ( ) , 1 ) ;
assert_eq! ( specs [ 0 ] [ " name " ] , " computer " ) ;
2026-11-09 06:03:40 +00:00
let quick = quick_tools ( & specs ) ;
2026-11-09 08:50:18 +00:00
assert! ( quick [ 0 ] [ " parameters " ] [ " properties " ] . get ( " ref " ) . is_none ( ) ) ;
assert! ( quick [ 0 ] [ " parameters " ] [ " properties " ] . get ( " name " ) . is_some ( ) ) ;
assert! ( quick [ 0 ] [ " parameters " ] [ " properties " ] [ " actions " ] [ " items " ] [ " properties " ] . get ( " ref " ) . is_none ( ) ) ;
2026-11-09 06:58:40 +00:00
// Nothing else changes shape: only `ref` is gone (a stray key here makes the provider refuse
// the whole request).
2026-11-09 08:50:18 +00:00
let keys = | t : & serde_json ::Value | t [ " parameters " ] [ " properties " ] . as_object ( ) . map ( | p | p . keys ( ) . filter ( | k | * k ! = " ref " ) . cloned ( ) . collect ::< Vec < _ > > ( ) ) ;
assert_eq! ( keys ( & specs [ 0 ] ) , keys ( & quick [ 0 ] ) ) ;
2026-11-09 06:58:40 +00:00
assert! ( ! serde_json ::to_string ( & quick ) . unwrap ( ) . contains ( " :null " ) , " a key with no value " ) ;
2026-11-09 08:50:18 +00:00
// The bot's own model keeps ref.
assert! ( specs [ 0 ] [ " parameters " ] [ " properties " ] . get ( " ref " ) . is_some ( ) ) ;
}
#[ test ]
fn the_one_screen_tool_expands_to_the_internal_ones ( ) {
let mut c = crate ::model ::ToolCall { id : " x " . into ( ) , name : " computer " . into ( ) , args : r #" { " action " : " click " , " ref " : " 4 " , " on " : " My Mac " } " #. into ( ) , status : crate ::model ::ToolStatus ::Running } ;
crate ::computer ::expand_call ( & mut c ) ;
assert_eq! ( c . name , " computer_use_click " ) ;
let v : serde_json ::Value = serde_json ::from_str ( & c . args ) . unwrap ( ) ;
assert_eq! ( v [ " ref " ] , " 4 " ) ;
assert_eq! ( v [ " computer " ] , " My Mac " ) ;
assert! ( v . get ( " action " ) . is_none ( ) & & v . get ( " on " ) . is_none ( ) ) ;
// Screenshots keep their name; unknown actions and other tools pass through.
let mut s = crate ::model ::ToolCall { id : " x " . into ( ) , name : " computer " . into ( ) , args : r #" { " action " : " screenshot " } " #. into ( ) , status : crate ::model ::ToolStatus ::Running } ;
crate ::computer ::expand_call ( & mut s ) ;
assert_eq! ( s . name , " computer_use_screenshot " ) ;
let mut bad = crate ::model ::ToolCall { id : " x " . into ( ) , name : " computer " . into ( ) , args : r #" { " action " : " fly " } " #. into ( ) , status : crate ::model ::ToolStatus ::Running } ;
crate ::computer ::expand_call ( & mut bad ) ;
assert_eq! ( bad . name , " computer " ) ;
// describe follows the expansion: the work log says 「點一下 #4」.
assert_eq! ( crate ::computer ::describe ( " computer " , r ## "{"action":"click","ref":"#4"}"## ) , " 點 #4 " ) ;
assert_eq! ( crate ::computer ::describe ( " computer " , r # "{"action":"screenshot"}"# ) , " 螢幕截圖 " ) ;
2026-11-09 06:03:40 +00:00
}
2026-11-09 05:31:25 +00:00
#[ test ]
fn the_same_click_twice_in_a_step_is_done_once ( ) {
let click = serde_json ::json! ( { " name " : " computer_use_click " , " arguments " : serde_json ::json! ( { " computer " : " 沒有這台 " , " name " : " Wi‑ Fi " } ) . to_string ( ) } ) ;
let two = serde_json ::json! ( { " tool_calls " : [
{ " index " : 0 , " id " : " a " , " function " : click . clone ( ) } ,
{ " index " : 1 , " id " : " b " , " function " : click } ,
] } ) ;
let ( port , _ ) = recording_model ( vec! [ two , serde_json ::json! ( { " content " : " 好了 " } ) ] ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 打開 Wi‑ Fi 選單 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut results = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 20 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::ToolResult ( m ) ) = > results . push ( m . text ) ,
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
let _ = a . reply . send ( ApprovalAnswer ::Reject ) ;
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert_eq! ( results . len ( ) , 2 , " {results:?} " ) ;
assert! ( ! results [ 0 ] . contains ( " 一模一樣 " ) , " {results:?} " ) ;
assert! ( results [ 1 ] . contains ( " 和這一步上一個點擊一模一樣 " ) , " {results:?} " ) ;
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
#[ test ]
fn a_control_is_found_again_by_what_it_is ( ) {
assert_eq! ( control_key ( " 勾選框「自動更新」: 1( 已勾) " ) , " 勾選框「自動更新」 " ) ;
assert_eq! ( control_key ( " 按鈕「Close」 " ) , " 按鈕「Close」 " ) ;
let list = Message ::tool_result ( " x " , " computer_use_screenshot " , " 螢幕截圖。 \n [3] 按鈕「Close」 \n [9] 勾選框「自動更新」: 0 " ) ;
let step = crate ::memory ::ReplayStep { tool : " computer_use_click " . into ( ) , args : r #" { " ref " : " 5 " , " computer " : " C " } " #. into ( ) , targets : vec ! [ " 勾選框「自動更新」: 1( 已勾) " . into ( ) ] } ;
let args = resolve_replay ( & step , std ::slice ::from_ref ( & list ) ) . unwrap ( ) ;
assert_eq! ( tools ::args_of ( & args ) [ " ref " ] , " 9 " ) ;
assert_eq! ( tools ::args_of ( & args ) [ " computer " ] , " C " ) ;
// A browser step finds its control by its words.
let web = crate ::memory ::ReplayStep { tool : " browser_click " . into ( ) , args : r #" { " ref " : " 4 " } " #. into ( ) , targets : vec ! [ " 按鈕「開始」 " . into ( ) ] } ;
let a = tools ::args_of ( & resolve_replay ( & web , & [ ] ) . unwrap ( ) ) ;
assert! ( a . get ( " ref " ) . is_none ( ) & & a [ " text " ] = = " 開始 " ) ;
// Not on the screen: says which.
let gone = crate ::memory ::ReplayStep { tool : " computer_use_click " . into ( ) , args : r #" { " ref " : " 5 " } " #. into ( ) , targets : vec ! [ " 按鈕「Save」 " . into ( ) ] } ;
assert_eq! ( resolve_replay ( & gone , std ::slice ::from_ref ( & list ) ) . unwrap_err ( ) , " 按鈕「Save」 " ) ;
}
/// A skill with saved steps is done again by AnyBot: no model between the steps, then the
/// bot's own model checks the result.
#[ test ]
fn a_skill_is_replayed_without_the_model_in_between ( ) {
let ( port , asked ) = recording_model ( vec! [ tool_delta ( " r " , REPLAY_TOOL , serde_json ::json! ( { " name " : " 兩步 " } ) ) , serde_json ::json! ( { " content " : " 確認做完了 " } ) ] ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-replay- {} " , std ::process ::id ( ) ) ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
let skill = crate ::memory ::Skill {
name : " 兩步 " . into ( ) ,
description : " 做兩件事 " . into ( ) ,
instructions : " … " . into ( ) ,
steps : vec ! [
crate ::memory ::ReplayStep { tool : " shell " . into ( ) , args : r #" { " command " : " echo first > r1 . txt " } " #. into ( ) , targets : vec ! [ ] } ,
crate ::memory ::ReplayStep { tool : " shell " . into ( ) , args : r #" { " command " : " echo second > r2 . txt " } " #. into ( ) , targets : vec ! [ ] } ,
] ,
} ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " main " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 照上次那樣做 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
step_model : Some ( " quick " . into ( ) ) ,
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w . clone ( ) } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
local_execution : true ,
tools_enabled : true ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
skills : vec ! [ skill ] ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut results = vec! [ ] ;
let mut last = String ::new ( ) ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 20 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::ToolResult ( m ) ) = > results . push ( m . text ) ,
( _ , StreamEvent ::StepDone ( m ) ) = > last = m . text ,
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
let _ = a . reply . send ( ApprovalAnswer ::Once ) ;
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
// Two model calls: the one that chose to replay, and the bot's own check at the end.
assert_eq! ( * asked . lock ( ) . unwrap ( ) , vec! [ " main " , " main " ] ) ;
assert_eq! ( std ::fs ::read_to_string ( w . join ( " r1.txt " ) ) . unwrap ( ) . trim ( ) , " first " ) ;
assert_eq! ( std ::fs ::read_to_string ( w . join ( " r2.txt " ) ) . unwrap ( ) . trim ( ) , " second " ) ;
assert! ( results [ 0 ] . contains ( " 開始重播「兩步」,共 2 步 " ) , " {results:?} " ) ;
// (「重播做完了」 is added for the model only, like the check-in line: not in the chat.)
assert_eq! ( results . len ( ) , 3 , " {results:?} " ) ;
assert_eq! ( last , " 確認做完了 " ) ;
}
#[ test ]
fn the_quick_model_carries_out_steps_between_the_bots_own ( ) {
let mut p = StepPlan ::new ( Some ( " q " . into ( ) ) ) ;
assert_eq! ( p . pick ( ) , None , " the first step plans: the bot's own model " ) ;
assert_eq! ( p . pick ( ) . as_deref ( ) , Some ( " q " ) ) ;
p . after ( true ) ;
assert_eq! ( p . pick ( ) , None , " after trouble: the bot's own model " ) ;
2026-11-09 03:25:17 +00:00
// Trouble twice: the rest of the turn is the bot's own model's.
let mut twice = StepPlan ::new ( Some ( " q " . into ( ) ) ) ;
twice . pick ( ) ;
twice . after ( true ) ;
twice . pick ( ) ;
twice . after ( true ) ;
assert! ( twice . quick_is_off ( ) & & twice . pick ( ) . is_none ( ) & & twice . pick ( ) . is_none ( ) ) ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
for _ in 0 .. QUICK_RUN_MAX {
assert_eq! ( p . pick ( ) . as_deref ( ) , Some ( " q " ) ) ;
}
assert_eq! ( p . pick ( ) , None , " a look from the bot's own model every QUICK_RUN_MAX steps " ) ;
2026-11-09 04:53:06 +00:00
assert! ( p . recheck_answer ( true ) & & p . pick ( ) . is_none ( ) , " the quick model's answer goes to the bot's own model " ) ;
assert! ( ! p . recheck_answer ( false ) , " the bot's own answer stands " ) ;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
p . quick_failed ( ) ;
assert_eq! ( p . pick ( ) , None ) ;
assert_eq! ( StepPlan ::new ( None ) . pick ( ) , None ) ;
// The quick model leaving commands for the screen on its own: the bot's own model decides.
let tc = | n : & str | crate ::model ::ToolCall { id : " x " . into ( ) , name : n . into ( ) , args : " {} " . into ( ) , status : ToolStatus ::Done } ;
let mut r = StepPlan ::new ( Some ( " q " . into ( ) ) ) ;
assert! ( ! r . changed_route ( & [ tc ( " shell " ) ] , false ) ) ;
assert! ( ! r . changed_route ( & [ tc ( " computer_use_screenshot " ) ] , true ) , " looking isn't switching " ) ;
assert! ( ! r . changed_route ( & [ tc ( " shell " ) ] , true ) ) ;
assert! ( r . changed_route ( & [ tc ( " computer_use_click " ) ] , true ) ) ;
assert! ( ! r . changed_route ( & [ tc ( " browser_click " ) ] , false ) , " the bot's own model may switch " ) ;
assert_eq! ( ProviderKind ::Xai . quick_model ( " grok-4.7 " ) , Some ( " grok-4.20-0309-non-reasoning " ) ) ;
assert_eq! ( ProviderKind ::Xai . quick_model ( " grok-4.20-0309-non-reasoning " ) , None ) ;
assert_eq! ( ProviderKind ::Ollama . quick_model ( " llama3.2 " ) , None ) ;
}
/// End to end with a model server that records which model each request asked for: plan with
/// the bot's model, act with the quick one, back to the bot's after a failed step, and the
/// quick model's 「done」 after changing things replaced by the bot's own check.
#[ test ]
fn a_turn_switches_models_as_planned ( ) {
use std ::io ::{ Read , Write } ;
let l = std ::net ::TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
let asked = Arc ::new ( Mutex ::new ( Vec ::< String > ::new ( ) ) ) ;
let seen = asked . clone ( ) ;
std ::thread ::spawn ( move | | {
let tool = | id : & str , cmd : & str | serde_json ::json! ( { " tool_calls " : [ { " index " : 0 , " id " : id , " function " : { " name " : " shell " , " arguments " : serde_json ::json! ( { " command " : cmd } ) . to_string ( ) } } ] } ) ;
let replies = [
tool ( " a " , " echo plan-and-start " ) ,
tool ( " b " , " exit 3 " ) ,
tool ( " c " , " echo fixed " ) ,
serde_json ::json! ( { " content " : " 做好了(快速模型) " } ) ,
serde_json ::json! ( { " content " : " 確認做好了 " } ) ,
] ;
for delta in replies {
let ( mut s , _ ) = l . accept ( ) . unwrap ( ) ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 16384 ] ;
let body = loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break String ::new ( ) ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break t [ end + 4 .. ] . to_string ( ) ;
}
}
} ;
let v : serde_json ::Value = serde_json ::from_str ( & body ) . unwrap_or_default ( ) ;
seen . lock ( ) . unwrap ( ) . push ( v [ " model " ] . as_str ( ) . unwrap_or ( " " ) . to_string ( ) ) ;
let out = format! ( " data: {} \n \n data: [DONE] \n \n " , serde_json ::json! ( { " choices " : [ { " delta " : delta } ] } ) ) ;
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( out . as_bytes ( ) ) ;
}
} ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-steps- {} " , std ::process ::id ( ) ) ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " main " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 做一件事 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
step_model : Some ( " quick " . into ( ) ) ,
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w . clone ( ) } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
local_execution : true ,
tools_enabled : true ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
via : " Custom · main " . into ( ) ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut shown = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 20 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::StepDone ( m ) ) = > shown . push ( ( m . via . clone ( ) , m . text . clone ( ) ) ) ,
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
let _ = a . reply . send ( ApprovalAnswer ::Once ) ;
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert_eq! ( * asked . lock ( ) . unwrap ( ) , vec! [ " main " , " quick " , " main " , " quick " , " main " ] ) ;
assert_eq! ( shown . iter ( ) . map ( | ( v , _ ) | v . as_str ( ) ) . collect ::< Vec < _ > > ( ) , vec! [ " Custom · main " , " Custom · quick " , " Custom · main " , " Custom · main " ] ) ;
assert_eq! ( shown . last ( ) . unwrap ( ) . 1 , " 確認做好了 " ) ;
assert! ( ! shown . iter ( ) . any ( | ( _ , t ) | t . contains ( " 快速模型 " ) ) , " the set-aside answer was never shown: {shown:?} " ) ;
}
/// AppleScript run outside the sandbox drives the person's apps: asked the first time even
/// in direct mode, and not again once they chose 「一律允許」.
#[ test ]
fn applescript_outside_the_sandbox_is_asked_once_even_in_direct_mode ( ) {
for ( granted , expect_ask ) in [ ( false , true ) , ( true , false ) ] {
let port = fake_model ( vec! [
( " shell " , serde_json ::json! ( { " command " : " osascript -e 'return 1+1' " , " computer " : crate ::model ::THIS_COMPUTER_NAME } ) ) ,
( " " , serde_json ::json! ( " 好了 " ) ) ,
] ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-osa- {} - {granted} " , std ::process ::id ( ) ) ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 算一下 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w . clone ( ) } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
local_execution : true ,
tools_enabled : true ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
grants : if granted { vec! [ tools ::OSASCRIPT_GRANT . into ( ) ] } else { vec! [ ] } ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut asked = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 20 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
asked . push ( a . grant . clone ( ) ) ;
let _ = a . reply . send ( ApprovalAnswer ::Reject ) ;
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
if expect_ask {
assert_eq! ( asked , vec! [ tools ::OSASCRIPT_GRANT . to_string ( ) ] ) ;
} else {
assert! ( asked . is_empty ( ) , " {asked:?} " ) ;
}
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// A reply with nothing in it (a dropped stream) is asked again, not taken as the answer.
#[ test ]
fn an_empty_reply_is_asked_again ( ) {
let port = fake_model ( vec! [ ( " " , serde_json ::json! ( " " ) ) , ( " " , serde_json ::json! ( " 好了 " ) ) ] ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 在嗎 " ) ] , tools : vec ! [ ] , session : String ::new ( ) } ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut texts = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 20 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::StepDone ( m ) ) = > texts . push ( m . text ) ,
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert_eq! ( texts , vec! [ " 好了 " . to_string ( ) ] ) ;
}
2026-11-06 08:51:25 +00:00
/// A brain with no Chrome of its own — a cloud brain — browses on its own box: the bot's
/// browser_open and browser_read for its computer reach the box's Chromium, not the
/// person's browser on the server. Run where no Chrome-family browser is installed, against
/// a box: `ANYBOT_DATA_DIR=<scratch> cargo test --release -- --ignored
/// a_brain_without_chrome_browses_on_its_box --nocapture`.
#[ test ]
#[ ignore ]
fn a_brain_without_chrome_browses_on_its_box ( ) {
let own = crate ::model ::BOT_COMPUTER_NAME ;
let page = " mkdir -p /tmp/anybot-page && printf '<meta charset=utf-8><title>box page</title><h1>盒子裡的測試頁 42</h1>' > /tmp/anybot-page/index.html && (cd /tmp/anybot-page && setsid python3 -m http.server 8766 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served " ;
let port = fake_model ( vec! [
( " shell " , serde_json ::json! ( { " command " : page } ) ) ,
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8766/ " } ) ) ,
( " browser_read " , serde_json ::json! ( { } ) ) ,
( " " , serde_json ::json! ( " 看到了 " ) ) ,
] ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 打開盒子裡的測試頁 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : own . into ( ) , place : tools ::Place ::Box { dir : crate ::cloudbox ::bot_dir ( 1 ) } } ] ,
default_computer : own . into ( ) ,
roles : tools ::Roles { own : own . into ( ) , person : None , server : true } ,
local_execution : false ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut results = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 120 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::ToolResult ( m ) ) = > {
println! ( " {} : {} " , m . tool_name , m . text ) ;
results . push ( m . text )
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
println! ( " this computer has a Chrome-family browser: {} " , crate ::botbrowser ::engine ( ) . is_some ( ) ) ;
assert! ( results [ 0 ] . contains ( " served " ) , " {results:?} " ) ;
assert! ( results [ 1 ] . starts_with ( & format! ( " [ {own} ] " ) ) & & results [ 1 ] . contains ( " 它桌面上的 Chromium " ) , " {} " , results [ 1 ] ) ;
assert! ( results [ 2 ] . contains ( " 盒子裡的測試頁 42 " ) , " {} " , results [ 2 ] ) ;
}
2026-11-06 15:12:53 +00:00
/// The box's browser works with the box's own disk (the brain runs elsewhere): an upload
/// finds the file there, a download and a PDF land there, and old tabs close. Needs Docker
/// and the box: `cargo test the_box_browser_uses_the_box_disk -- --ignored --nocapture`.
#[ test ]
#[ ignore ]
fn the_box_browser_uses_the_box_disk ( ) {
let own = crate ::model ::BOT_COMPUTER_NAME ;
let setup = r #" mkdir -p /tmp/anybot-files /tmp/anybot-up && cd /tmp/anybot-files && cat > index.html <<'EOF'
< meta charset = utf - 8 > < title > files < / title > < h1 > 檔 案 測 試 < / h1 >
< input type = file id = f onchange = " document.getElementById('o').textContent='已選:'+this.files[0].name+' '+this.files[0].size " >
< p id = o > 還 沒 選 < / p > < a href = " report.csv " download > 下 載 報 表 < / a >
EOF
printf ' a , b \ n1 , 2 \ n ' > report . csv & & printf ' hello box ' > ' / tmp / anybot - up / 上 傳 測 試 . txt ' & & ( setsid python3 - m http . server 8768 - - bind 127. 0. 0.1 > / dev / null 2 > & 1 & ) ; sleep 1 ; echo served " #;
let check = " ls /home/bot/Downloads/report.csv /home/bot/Downloads/files-*.pdf && head -c 4 /home/bot/Downloads/files-*.pdf && echo " ;
let pages = r # "curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys; print("pages:", len([t for t in json.load(sys.stdin) if t["type"] == "page"]))'"# ;
let clean = r # "pkill -f 'http[.]server 8768'; rm -rf /tmp/anybot-files /tmp/anybot-up /home/bot/Downloads/report.csv /home/bot/Downloads/files-*.pdf; curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys,urllib.request; [urllib.request.urlopen("http://127.0.0.1:9222/json/close/" + t["id"]) for t in json.load(sys.stdin) if ":8768" in t["url"]]'; echo cleaned"# ;
let mut replies : Vec < ( & 'static str , serde_json ::Value ) > = vec! [
( " shell " , serde_json ::json! ( { " command " : setup } ) ) ,
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8768/ " } ) ) ,
( " browser_upload " , serde_json ::json! ( { " path " : " /tmp/anybot-up/上傳測試.txt " } ) ) ,
( " browser_read " , serde_json ::json! ( { } ) ) ,
( " browser_upload " , serde_json ::json! ( { " path " : " /tmp/anybot-up/沒有這個.txt " } ) ) ,
( " browser_download " , serde_json ::json! ( { " text " : " 下載報表 " } ) ) ,
( " browser_pdf " , serde_json ::json! ( { } ) ) ,
( " shell " , serde_json ::json! ( { " command " : check } ) ) ,
] ;
for n in 1 ..= 8 {
replies . push ( ( " browser_open " , serde_json ::json! ( { " url " : format ! ( " http://127.0.0.1:8768/?n={n} " ) } ) ) ) ;
}
replies . push ( ( " shell " , serde_json ::json! ( { " command " : pages } ) ) ) ;
replies . push ( ( " shell " , serde_json ::json! ( { " command " : clean } ) ) ) ;
replies . push ( ( " " , serde_json ::json! ( " 好了 " ) ) ) ;
let port = fake_model ( replies ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 測試盒子裡的檔案 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : own . into ( ) , place : tools ::Place ::Box { dir : crate ::cloudbox ::bot_dir ( 1 ) } } ] ,
default_computer : own . into ( ) ,
roles : tools ::Roles { own : own . into ( ) , person : None , server : true } ,
local_execution : false ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut results = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 180 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::ToolResult ( m ) ) = > {
println! ( " {} : {} \n " , m . tool_name , m . text . chars ( ) . take ( 300 ) . collect ::< String > ( ) ) ;
results . push ( m . text )
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert! ( results [ 0 ] . contains ( " served " ) , " {} " , results [ 0 ] ) ;
assert! ( results [ 1 ] . contains ( " 檔案測試 " ) , " {} " , results [ 1 ] ) ;
2026-11-06 16:28:35 +00:00
assert! ( results [ 2 ] . contains ( " 已把 /tmp/anybot-up/上傳測試.txt 放進上傳欄位,欄位裡現在是:上傳測試.txt( 9 位元組) " ) & & results [ 2 ] . contains ( " 已選:上傳測試.txt 9 " ) , " {} " , results [ 2 ] ) ;
2026-11-06 15:12:53 +00:00
assert! ( results [ 3 ] . contains ( " 已選:上傳測試.txt 9 " ) , " {} " , results [ 3 ] ) ;
assert! ( results [ 4 ] . contains ( " 這台電腦上沒有「/tmp/anybot-up/沒有這個.txt」 " ) , " {} " , results [ 4 ] ) ;
assert! ( results [ 5 ] . contains ( " 下載好了:/home/bot/Downloads/report.csv " ) , " {} " , results [ 5 ] ) ;
assert! ( results [ 6 ] . contains ( " 已存成 PDF: /home/bot/Downloads/files- " ) , " {} " , results [ 6 ] ) ;
assert! ( results [ 7 ] . contains ( " %PDF " ) & & ! results [ 7 ] . contains ( " No such file " ) , " {} " , results [ 7 ] ) ;
let count = results [ 16 ] . split ( " pages: " ) . nth ( 1 ) . and_then ( | n | n . trim ( ) . parse ::< usize > ( ) . ok ( ) ) . unwrap_or ( 99 ) ;
assert! ( count < = 6 , " {} " , results [ 16 ] ) ;
assert! ( results [ 17 ] . contains ( " cleaned " ) , " {} " , results [ 17 ] ) ;
}
2026-11-06 15:21:04 +00:00
/// When a job ends, the tabs it left open in the box's browser close; the page it ended on
/// stays. Needs the box: `cargo test a_finished_job_closes_its_tabs -- --ignored --nocapture`.
#[ test ]
#[ ignore ]
fn a_finished_job_closes_its_tabs ( ) {
let own = crate ::model ::BOT_COMPUTER_NAME ;
let setup = " mkdir -p /tmp/anybot-tabs && cd /tmp/anybot-tabs && echo '<meta charset=utf-8><title>tab</title>分頁' > index.html && (setsid python3 -m http.server 8770 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served " ;
let port = fake_model ( vec! [
( " shell " , serde_json ::json! ( { " command " : setup } ) ) ,
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8770/?n=1 " } ) ) ,
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8770/?n=2 " } ) ) ,
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8770/?n=3 " } ) ) ,
( " " , serde_json ::json! ( " 好了 " ) ) ,
] ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 開三頁 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : own . into ( ) , place : tools ::Place ::Box { dir : crate ::cloudbox ::bot_dir ( 1 ) } } ] ,
default_computer : own . into ( ) ,
roles : tools ::Roles { own : own . into ( ) , person : None , server : true } ,
local_execution : false ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 120 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
std ::thread ::sleep ( Duration ::from_secs ( 2 ) ) ;
let box_sh = | script : & str | String ::from_utf8_lossy ( & std ::process ::Command ::new ( " docker " ) . args ( [ " exec " , crate ::cloudbox ::container ( ) , " sh " , " -c " , script ] ) . output ( ) . unwrap ( ) . stdout ) . to_string ( ) ;
let left = box_sh ( r # "curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys; print("\n".join(t["url"] for t in json.load(sys.stdin) if t["type"] == "page"))'"# ) ;
println! ( " left open: \n {left} " ) ;
box_sh ( r # "pkill -f 'http[.]server 8770'; rm -rf /tmp/anybot-tabs"# ) ;
assert_eq! ( left . trim ( ) , " http://127.0.0.1:8770/?n=3 " , " {left} " ) ;
}
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
/// Two bots' jobs at once on one browser (the box's): each reads its own page although the
/// other's is in front, and neither closes the other's tab when it ends; a popup a click
/// opens is the clicking job's. Needs the box:
/// `cargo test two_jobs_keep_to_their_own_tabs -- --ignored --nocapture`.
#[ test ]
#[ ignore ]
fn two_jobs_keep_to_their_own_tabs ( ) {
let own = crate ::model ::BOT_COMPUTER_NAME ;
let box_sh = | script : & str | String ::from_utf8_lossy ( & std ::process ::Command ::new ( " docker " ) . args ( [ " exec " , crate ::cloudbox ::container ( ) , " sh " , " -c " , script ] ) . output ( ) . unwrap ( ) . stdout ) . to_string ( ) ;
let urls = r # "curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys; print(" ".join(t["url"] for t in json.load(sys.stdin) if t["type"] == "page"))'"# ;
let clean = r # "pkill -f 'http[.]server 8774'; rm -rf /tmp/anybot-two /home/bot/bot-991 /home/bot/bot-992 /home/bot/bot-993; curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys,urllib.request; [urllib.request.urlopen("http://127.0.0.1:9222/json/close/" + t["id"]) for t in json.load(sys.stdin) if ":8774" in t["url"]]' 2>/dev/null; true"# ;
// (What an earlier run that failed half-way left behind.)
box_sh ( clean ) ;
let setup = r # "mkdir -p /tmp/anybot-two && cd /tmp/anybot-two && echo '<meta charset=utf-8><title>A</title>頁面甲 <a href="b.html" target="_blank">開新視窗</a>' > a.html && echo '<meta charset=utf-8><title>B</title>頁面乙' > b.html && (setsid python3 -m http.server 8774 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served"# ;
let run = | replies : Vec < ( & 'static str , serde_json ::Value ) > , bot : u64 | {
let port = fake_model ( replies ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 開頁面 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : own . into ( ) , place : tools ::Place ::Box { dir : crate ::cloudbox ::bot_dir ( bot ) } } ] ,
default_computer : own . into ( ) ,
roles : tools ::Roles { own : own . into ( ) , person : None , server : true } ,
local_execution : false ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , bot , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
std ::thread ::spawn ( move | | {
let mut out = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 120 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::ToolResult ( m ) ) = > out . push ( format! ( " {} : {} " , m . tool_name , m . text ) ) ,
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
out
} )
} ;
let a = run ( vec! [
( " shell " , serde_json ::json! ( { " command " : setup } ) ) ,
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8774/a.html " } ) ) ,
( " shell " , serde_json ::json! ( { " command " : " sleep 4; echo waited " } ) ) ,
( " browser_read " , serde_json ::json! ( { } ) ) ,
( " " , serde_json ::json! ( " 好了 " ) ) ,
] , 991 ) ;
std ::thread ::sleep ( Duration ::from_millis ( 2500 ) ) ;
let b = run ( vec! [
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8774/b.html " } ) ) ,
( " browser_read " , serde_json ::json! ( { } ) ) ,
( " shell " , serde_json ::json! ( { " command " : " sleep 4; echo waited " } ) ) ,
( " " , serde_json ::json! ( " 好了 " ) ) ,
] , 992 ) ;
let ( a , b ) = ( a . join ( ) . unwrap ( ) , b . join ( ) . unwrap ( ) ) ;
for l in a . iter ( ) . chain ( b . iter ( ) ) {
println! ( " {} \n " , l . chars ( ) . take ( 160 ) . collect ::< String > ( ) ) ;
}
assert! ( a [ 3 ] . starts_with ( " browser_read " ) & & a [ 3 ] . contains ( " 頁面甲 " ) & & ! a [ 3 ] . contains ( " 頁面乙 " ) , " A read another job's page: {} " , a [ 3 ] ) ;
assert! ( b [ 1 ] . starts_with ( " browser_read " ) & & b [ 1 ] . contains ( " 頁面乙 " ) , " {} " , b [ 1 ] ) ;
let left = box_sh ( urls ) ;
println! ( " left after both: {left} " ) ;
assert! ( left . contains ( " 8774/a.html " ) & & left . contains ( " 8774/b.html " ) , " a job closed the other's tab: {left} " ) ;
// A popup a click opens belongs to the job that clicked: it reads it, then keeps only it.
let c = run ( vec! [
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8774/a.html?c " } ) ) ,
( " browser_click " , serde_json ::json! ( { " text " : " 開新視窗 " } ) ) ,
( " browser_read " , serde_json ::json! ( { } ) ) ,
( " " , serde_json ::json! ( " 好了 " ) ) ,
] , 993 )
. join ( )
. unwrap ( ) ;
for l in & c {
println! ( " {} \n " , l . chars ( ) . take ( 160 ) . collect ::< String > ( ) ) ;
}
assert! ( c [ 2 ] . starts_with ( " browser_read " ) & & c [ 2 ] . contains ( " 頁面乙 " ) , " {} " , c [ 2 ] ) ;
// (The tabs close right after the job's Done.)
std ::thread ::sleep ( Duration ::from_secs ( 2 ) ) ;
let left = box_sh ( urls ) ;
println! ( " left after the popup job: {left} " ) ;
assert! ( ! left . contains ( " a.html?c " ) , " the clicking job's first tab stayed: {left} " ) ;
box_sh ( clean ) ;
}
2026-11-06 15:12:53 +00:00
/// Words reach the control they name on a form laid out like httpbin's (each label and its
/// input inside a <p>): typing into 「Customer name:」, ticking 「Large」, pressing 「Submit
/// order」. They used to land on the <p> and do nothing. Needs the box:
/// `cargo test the_words_on_a_form_reach_its_controls -- --ignored --nocapture`.
#[ test ]
#[ ignore ]
fn the_words_on_a_form_reach_its_controls ( ) {
let own = crate ::model ::BOT_COMPUTER_NAME ;
let setup = r #" mkdir -p /tmp/anybot-form && cd /tmp/anybot-form && cat > index.html <<'EOF'
< meta charset = utf - 8 > < title > form < / title >
< form action = " done.html " method = " get " >
< p > < label > Customer name : < input name = " custname " > < / label > < / p >
< p > < label > Telephone : < input type = tel name = " custtel " > < / label > < / p >
< fieldset > < legend > Pizza Size < / legend >
< p > < label > < input type = radio name = size value = " small " > Small < / label > < / p >
< p > < label > < input type = radio name = size value = " large " > Large < / label > < / p > < / fieldset >
< fieldset > < legend > Pizza Toppings < / legend >
< p > < label > < input type = checkbox name = " topping " value = " cheese " > Extra Cheese < / label > < / p >
< p > < label > < input type = checkbox name = " topping " value = " onion " > Onion < / label > < / p > < / fieldset >
< p > < button > Submit order < / button > < / p >
< / form >
EOF
echo ' < meta charset = utf - 8 > < title > done < / title > 送 出 了 ' > done . html ; ( setsid python3 - m http . server 8769 - - bind 127. 0. 0.1 > / dev / null 2 > & 1 & ) ; sleep 1 ; echo served " #;
let clean = r # "pkill -f 'http[.]server 8769'; rm -rf /tmp/anybot-form; curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys,urllib.request; [urllib.request.urlopen("http://127.0.0.1:9222/json/close/" + t["id"]) for t in json.load(sys.stdin) if ":8769" in t["url"]]'; echo cleaned"# ;
let port = fake_model ( vec! [
( " shell " , serde_json ::json! ( { " command " : setup } ) ) ,
( " browser_open " , serde_json ::json! ( { " url " : " http://127.0.0.1:8769/ " } ) ) ,
( " browser_type " , serde_json ::json! ( { " text " : " Customer name: " , " value " : " 李大華 " } ) ) ,
( " browser_type " , serde_json ::json! ( { " text " : " Telephone " , " value " : " 0922000111 " } ) ) ,
( " browser_click " , serde_json ::json! ( { " text " : " Large " } ) ) ,
( " browser_click " , serde_json ::json! ( { " text " : " Extra Cheese " } ) ) ,
( " browser_click " , serde_json ::json! ( { " text " : " Onion " } ) ) ,
( " browser_click " , serde_json ::json! ( { " text " : " Submit order " } ) ) ,
( " shell " , serde_json ::json! ( { " command " : clean } ) ) ,
( " " , serde_json ::json! ( " 好了 " ) ) ,
] ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 填表 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : own . into ( ) , place : tools ::Place ::Box { dir : crate ::cloudbox ::bot_dir ( 1 ) } } ] ,
default_computer : own . into ( ) ,
roles : tools ::Roles { own : own . into ( ) , person : None , server : true } ,
local_execution : false ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut results = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 180 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::ToolResult ( m ) ) = > {
println! ( " {} : {} \n " , m . tool_name , m . text . chars ( ) . take ( 240 ) . collect ::< String > ( ) ) ;
results . push ( m . text )
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert! ( results [ 2 ] . contains ( " 欄位現在是「李大華」 " ) , " {} " , results [ 2 ] ) ;
assert! ( results [ 3 ] . contains ( " 欄位現在是「0922000111」 " ) , " {} " , results [ 3 ] ) ;
let sent = & results [ 7 ] ;
assert! ( sent . contains ( " done.html?custname=%E6%9D%8E%E5%A4%A7%E8%8F%AF&custtel=0922000111&size=large&topping=cheese&topping=onion " ) , " {sent} " ) ;
assert! ( results [ 8 ] . contains ( " cleaned " ) , " {} " , results [ 8 ] ) ;
}
2026-11-06 15:21:04 +00:00
#[ test ]
Each job works on its own tabs; a Lima VM the box lives in is started when it's down
- Bots share a browser (the box's Chromium has one profile). Every step acted on the tab in
front, so two jobs at once read and clicked on each other's pages and one finishing closed
the other's. A job (named, and the name rides along to another computer's browser) owns the
tabs it opens or lands on, popups it clicks open included; its steps use its latest one, and
another job's tabs are never used, listed or closed. When it ends it closes only its own.
Real run on the box: two jobs at once each read their own page with the other's in front,
both tabs survived both endings; a popup was read by the job that opened it.
- A Docker host that is a Lima VM's socket and doesn't answer (the Mac restarted): AnyBot
starts the VM (limactl start, at most once in 5 minutes), and a step that needs the box
waits for it. No login item needed. Real run: the VM stopped, AnyBot had it back in 10 s;
the box then woke on use, still gVisor with Chromium's sandbox.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:50:29 +00:00
fn a_browser_step_carries_its_job ( ) {
let a = with_job ( r # "{"url":"https://example.com","profile":"工作"}"# , " 6-1-1 " ) ;
let v : serde_json ::Value = serde_json ::from_str ( & a ) . unwrap ( ) ;
assert_eq! ( ( v [ " _job " ] . as_str ( ) , v [ " url " ] . as_str ( ) , v [ " profile " ] . as_str ( ) ) , ( Some ( " 6-1-1 " ) , Some ( " https://example.com " ) , Some ( " 工作 " ) ) ) ;
assert_eq! ( serde_json ::from_str ::< serde_json ::Value > ( & with_job ( " " , " j " ) ) . unwrap ( ) [ " _job " ] , " j " ) ;
// Two jobs never share a name.
assert_ne! ( UsedBrowsers ::new ( " b " , 6 ) . job , UsedBrowsers ::new ( " b " , 6 ) . job ) ;
2026-11-06 15:21:04 +00:00
}
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
#[ test ]
fn the_same_miss_counts_as_the_same_step ( ) {
// Clicks a few pixels apart on the same strip are one step; another computer or target isn't.
assert_eq! ( step_key ( " computer_use_click " , r # "{"x":70,"y":355,"computer":"M"}"# ) , step_key ( " computer_use_click " , r # "{"x":66,"y":358,"computer":"M"}"# ) ) ;
assert_ne! ( step_key ( " computer_use_click " , r # "{"x":70,"y":355}"# ) , step_key ( " computer_use_click " , r # "{"x":70,"y":290}"# ) ) ;
assert_ne! ( step_key ( " computer_use_click " , r # "{"ref":"3"}"# ) , step_key ( " computer_use_click " , r # "{"ref":"4"}"# ) ) ;
// Looking is never a repeat; other tools aren't counted.
assert_eq! ( step_key ( " computer_use_screenshot " , " {} " ) , None ) ;
assert_eq! ( step_key ( " shell " , r # "{"command":"ls"}"# ) , None ) ;
assert! ( step_key ( " browser_click " , r # "{"ref":"5"}"# ) . is_some ( ) ) ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// Reading down a page is not going in circles.
assert_eq! ( step_key ( " browser_scroll " , r # "{"direction":"down"}"# ) , None ) ;
assert_eq! ( step_key ( " computer_use_press_key " , r # "{"key":"Down"}"# ) , None ) ;
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
}
2026-11-05 10:25:46 +00:00
#[ test ]
fn a_click_by_number_is_named_in_the_question ( ) {
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let list = Message ::tool_result ( " c1 " , " computer_use_screenshot " , " [王小明的Mac mini] 螢幕截圖( 1024× 576) 。 \n 「計算機」上可以操作的東西(圖上同樣的編號,用 ref 點): \n 文字「0」 \n [4] 按鈕「全部清除」 \n [7] 按鈕「7」 " ) ;
Clippy is clean on macOS and Linux, for the app and the relay
Every warning `cargo clippy --all-targets` gave (50 on macOS, 8 more only on
Linux, 1 in the relay), fixed without changing what anything does:
- long types get a name (Peer, Members, Imported, Joining, Mention, Stream…);
- the two big enum variants are boxed (the vault's login plan, joining an
account);
- a test now waits for the copy it relaunched (it left a zombie behind);
- a hint's format had a dead `if` whose two branches were both "";
- the rest is what clippy suggests: sort_by_key, as_chunks, is_none_or,
is_multiple_of, while let, from_ref, next_back, struct init, no casts to
the same type.
Checked: cargo test (177 macOS, 181 Linux), relay tests (11), remote_e2e
28/28, and the live Linux test (X11; Wayland clicks, keys, Chinese typing,
scroll and screenshots).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-07 02:10:28 +00:00
assert_eq! ( element_named ( std ::slice ::from_ref ( & list ) , " 7 " ) . as_deref ( ) , Some ( " 按鈕「7」 " ) ) ;
assert_eq! ( element_named ( std ::slice ::from_ref ( & list ) , " e7 " ) . as_deref ( ) , Some ( " 按鈕「7」 " ) ) ;
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
assert_eq! ( element_named ( & [ list ] , " 9 " ) , None ) ;
assert_eq! ( computer ::describe ( " computer_use_click " , r # "{"ref":"7"}"# ) , " 點 #7 " ) ;
2026-11-05 10:25:46 +00:00
assert_eq! ( computer ::describe ( " computer_use_click " , r # "{"x":10,"y":20}"# ) , " 點 (10, 20) " ) ;
}
2026-11-05 08:47:28 +00:00
#[ test ]
fn every_computer_answers_to_its_own_name ( ) {
// One name per computer: the bot's own goes by the bot's name, the others by theirs, and
// the everyday words point at the right one (「我的電腦」 = where the person typed).
let roles = tools ::Roles { own : " Master 的電腦 " . into ( ) , person : Some ( " 王小明的Mac mini " . into ( ) ) , server : true } ;
let own = tools ::Target { name : " Master 的電腦 " . into ( ) , place : tools ::Place ::Box { dir : String ::new ( ) } } ;
2026-11-05 13:28:32 +00:00
let mac = tools ::Target { name : " 王小明的Mac mini " . into ( ) , place : tools ::Place ::Device { host : " m " . into ( ) , dir : String ::new ( ) } } ;
let desk = tools ::Target { name : " ai " . into ( ) , place : tools ::Place ::Device { host : " a " . into ( ) , dir : String ::new ( ) } } ;
2026-11-05 08:47:28 +00:00
assert! ( mentions ( " 在 Master 的電腦打開 Threads " , & own , & roles ) ) ;
assert! ( mentions ( " 用master的電腦 " , & own , & roles ) ) ;
assert! ( mentions ( " 你都用 Bot的電腦 " , & own , & roles ) ) ;
assert! ( mentions ( " 在 Mac mini 上開 " , & mac , & roles ) ) ;
assert! ( mentions ( " 截圖我的電腦 " , & mac , & roles ) ) ;
assert! ( ! mentions ( " 截圖我的電腦 " , & desk , & roles ) ) ;
assert! ( ! mentions ( " 截圖我的電腦 " , & own , & roles ) ) ;
assert! ( mentions ( " 在 ai 上跑 " , & desk , & roles ) ) ;
assert! ( ! mentions ( " 用 AI 幫我想標題 " , & mac , & roles ) ) ;
}
#[ test ]
fn a_step_moved_back_to_the_chosen_computer_says_so ( ) {
let roles = tools ::Roles { own : " Master 的電腦 " . into ( ) , person : Some ( " 王小明的Mac mini " . into ( ) ) , server : true } ;
let job = Job {
computers : vec ! [
tools ::Target { name : " Master 的電腦 " . into ( ) , place : tools ::Place ::Box { dir : String ::new ( ) } } ,
2026-11-05 13:28:32 +00:00
tools ::Target { name : " 王小明的Mac mini " . into ( ) , place : tools ::Place ::Device { host : " m " . into ( ) , dir : String ::new ( ) } } ,
2026-11-05 08:47:28 +00:00
] ,
default_computer : " Master 的電腦 " . into ( ) ,
roles ,
.. test_job ( )
} ;
// Not asked for: back to the chosen one, with a word for the bot.
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let ( args , note ) = keep_to_chosen ( & job , & [ Message ::user ( " 幫我截圖 " ) ] , " shell " , r # "{"computer":"王小明的Mac mini"}"# ) ;
2026-11-05 08:47:28 +00:00
assert! ( args . contains ( " Master 的電腦 " ) & & note . unwrap ( ) . contains ( " 王小明的Mac mini " ) ) ;
// Named by the person: it stays.
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let ( args , note ) = keep_to_chosen ( & job , & [ Message ::user ( " 看一下我的電腦畫面 " ) ] , " shell " , r # "{"computer":"王小明的Mac mini"}"# ) ;
2026-11-05 08:47:28 +00:00
assert! ( args . contains ( " 王小明的Mac mini " ) & & note . is_none ( ) ) ;
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.
The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).
The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.
The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
impossible server branches removed; duplicated prompt instructions merged.
Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// The chosen computer has no screen (no Docker): a screen step isn't moved there to fail.
let sealed = Job { computers : vec ! [ tools ::Target { name : " Master 的電腦 " . into ( ) , place : tools ::Place ::Sealed { root : std ::env ::temp_dir ( ) } } , job . computers [ 1 ] . clone ( ) ] , .. job } ;
let ( args , note ) = keep_to_chosen ( & sealed , & [ Message ::user ( " 幫我截圖 " ) ] , " computer_use_screenshot " , r # "{"computer":"王小明的Mac mini"}"# ) ;
assert! ( args . contains ( " 王小明的Mac mini " ) & & note . is_none ( ) , " {args} " ) ;
2026-11-05 07:48:20 +00:00
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
#[ test ]
fn cut_keeps_tool_pairs_and_recent_messages ( ) {
let mut h = vec! [ ] ;
for i in 0 .. 6 {
h . push ( Message ::user ( & format! ( " q {i} " ) ) ) ;
let mut a = Message ::assistant ( " " , " " ) ;
a . tool_calls = vec! [ ToolCall { id : format ! ( " c{i} " ) , name : " list_dir " . into ( ) , args : " {} " . into ( ) , status : ToolStatus ::Done } ] ;
h . push ( a ) ;
h . push ( Message ::tool_result ( & format! ( " c {i} " ) , " list_dir " , " x " ) ) ;
}
let cut = compaction_cut ( & h ) . unwrap ( ) ;
assert_eq! ( h [ cut ] . role , Role ::User ) ;
assert! ( h . len ( ) - cut > = KEEP_RECENT_MESSAGES ) ;
assert! ( compaction_cut ( & h [ .. 5 ] ) . is_none ( ) ) ;
}
#[ test ]
fn tokens_count_chinese_heavier ( ) {
assert_eq! ( estimate_tokens ( " abcdefgh " ) , 2 ) ;
assert_eq! ( estimate_tokens ( " 你好 " ) , 2 ) ;
}
/// Full loop against a fake provider: the model asks to list a folder, gets the result,
/// then answers in words.
#[ test ]
fn loop_runs_tool_then_answers ( ) {
use std ::io ::{ Read , Write } ;
use std ::net ::TcpListener ;
let l = TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
std ::thread ::spawn ( move | | {
let replies = [
" data: { \" choices \" :[{ \" delta \" :{ \" tool_calls \" :[{ \" index \" :0, \" id \" : \" c1 \" , \" function \" :{ \" name \" : \" list_dir \" , \" arguments \" : \" {} \" }}]}}]} \n \n data: [DONE] \n \n " ,
" data: { \" choices \" :[{ \" delta \" :{ \" content \" : \" 裡面有 a.txt \" }}]} \n \n data: [DONE] \n \n " ,
] ;
for body in replies {
let ( mut s , _ ) = l . accept ( ) . unwrap ( ) ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 8192 ] ;
loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break ;
}
}
}
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( body . as_bytes ( ) ) ;
}
} ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-loop- {} " , std ::process ::id ( ) ) ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
std ::fs ::write ( w . join ( " a.txt " ) , " hi " ) . unwrap ( ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 有什麼檔案? " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
2026-11-05 08:47:28 +00:00
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
local_execution : true ,
grants : vec ! [ ] ,
context_tokens : DEFAULT_CONTEXT_TOKENS ,
via : " test " . into ( ) ,
via_name : " Bot " . into ( ) ,
tools_enabled : true ,
direct : Arc ::new ( AtomicBool ::new ( false ) ) ,
local_locked : false ,
steer : Arc ::new ( Mutex ::new ( vec! [ ] ) ) ,
review : None ,
demos : vec ! [ ] ,
2026-11-06 15:12:53 +00:00
skills : vec ! [ ] ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op : String ::new ( ) ,
log : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} ;
let ( tx , rx ) = channel ( ) ;
let cancel = Arc ::new ( AtomicBool ::new ( false ) ) ;
// Run the loop body on this thread's terms: spawn needs an egui context, so drive it directly.
let ctx = eframe ::egui ::Context ::default ( ) ;
spawn ( job , 1 , tx , cancel , ctx ) ;
let mut texts = vec! [ ] ;
let mut results = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 10 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::ToolResult ( m ) ) = > results . push ( m . text ) ,
( _ , StreamEvent ::StepDone ( m ) ) = > texts . push ( m . text ) ,
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert! ( results [ 0 ] . contains ( " a.txt " ) , " {results:?} " ) ;
assert_eq! ( texts . last ( ) . unwrap ( ) , " 裡面有 a.txt " ) ;
}
/// A message sent while the bot works steers the same turn: the model sees it at the
/// next step and the turn only ends (one Done) after that.
#[ test ]
fn mid_turn_message_steers_the_same_turn ( ) {
use std ::io ::{ Read , Write } ;
use std ::net ::TcpListener ;
let l = TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
let seen = Arc ::new ( Mutex ::new ( String ::new ( ) ) ) ;
let seen2 = seen . clone ( ) ;
std ::thread ::spawn ( move | | {
for body in [ " data: { \" choices \" :[{ \" delta \" :{ \" content \" : \" 好 \" }}]} \n \n data: [DONE] \n \n " , " data: { \" choices \" :[{ \" delta \" :{ \" content \" : \" 收到補充 \" }}]} \n \n data: [DONE] \n \n " ] {
let ( mut s , _ ) = l . accept ( ) . unwrap ( ) ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 8192 ] ;
loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break ;
}
}
}
* seen2 . lock ( ) . unwrap ( ) = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( body . as_bytes ( ) ) ;
}
} ) ;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let steer = Arc ::new ( Mutex ::new ( vec! [ Queued ::new ( " 順便用英文 " ) ] ) ) ;
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 打招呼 " ) ] , tools : vec ! [ ] , session : String ::new ( ) } ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
computers : vec ! [ ] ,
default_computer : crate ::model ::BOT_COMPUTER_NAME . into ( ) ,
2026-11-05 08:47:28 +00:00
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
local_execution : false ,
grants : vec ! [ ] ,
context_tokens : DEFAULT_CONTEXT_TOKENS ,
via : " test " . into ( ) ,
via_name : " Bot " . into ( ) ,
tools_enabled : true ,
direct : Arc ::new ( AtomicBool ::new ( false ) ) ,
local_locked : false ,
steer ,
review : None ,
demos : vec ! [ ] ,
2026-11-06 15:12:53 +00:00
skills : vec ! [ ] ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op : String ::new ( ) ,
log : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let ( mut steered , mut done , mut texts ) = ( 0 , 0 , vec! [ ] ) ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 10 ) ) . expect ( " loop stalled " ) {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
( _ , StreamEvent ::Steered ( m , _ ) ) = > {
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
assert! ( m . steer ) ;
steered + = 1 ;
}
( _ , StreamEvent ::StepDone ( m ) ) = > texts . push ( m . text ) ,
( _ , StreamEvent ::Done ) = > {
done + = 1 ;
break ;
}
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert_eq! ( ( steered , done ) , ( 1 , 1 ) ) ;
assert_eq! ( texts , vec! [ " 好 " , " 收到補充 " ] ) ;
assert! ( seen . lock ( ) . unwrap ( ) . contains ( " 順便用英文 " ) ) ;
}
/// Tools off and this Mac not allowed: the step pops up two requests (use tools, use this
/// computer) instead of failing, and runs once both are allowed.
#[ test ]
fn missing_permission_pops_up_and_continues ( ) {
use std ::io ::{ Read , Write } ;
use std ::net ::TcpListener ;
let l = TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
std ::thread ::spawn ( move | | {
let replies = [
" data: { \" choices \" :[{ \" delta \" :{ \" tool_calls \" :[{ \" index \" :0, \" id \" : \" c1 \" , \" function \" :{ \" name \" : \" list_dir \" , \" arguments \" : \" { \\ \" computer \\ \" : \\ \" 你的電腦 \\ \" } \" }}]}}]} \n \n data: [DONE] \n \n " ,
" data: { \" choices \" :[{ \" delta \" :{ \" content \" : \" 看完了 \" }}]} \n \n data: [DONE] \n \n " ,
] ;
for body in replies {
let ( mut s , _ ) = l . accept ( ) . unwrap ( ) ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 8192 ] ;
loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break ;
}
}
}
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( body . as_bytes ( ) ) ;
}
} ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-perm- {} " , std ::process ::id ( ) ) ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
std ::fs ::write ( w . join ( " note.txt " ) , " hi " ) . unwrap ( ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 看看我的電腦 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
2026-11-05 08:47:28 +00:00
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
local_execution : false ,
tools_enabled : false ,
direct : Arc ::new ( AtomicBool ::new ( false ) ) ,
local_locked : false ,
grants : vec ! [ ] ,
context_tokens : DEFAULT_CONTEXT_TOKENS ,
via : " test " . into ( ) ,
via_name : " Bot " . into ( ) ,
steer : Arc ::new ( Mutex ::new ( vec! [ ] ) ) ,
review : None ,
demos : vec ! [ ] ,
2026-11-06 15:12:53 +00:00
skills : vec ! [ ] ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op : String ::new ( ) ,
log : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut kinds = vec! [ ] ;
let mut result = String ::new ( ) ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 10 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
kinds . push ( a . kind ) ;
let _ = a . reply . send ( ApprovalAnswer ::Once ) ;
}
( _ , StreamEvent ::ToolResult ( m ) ) = > result = m . text ,
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert_eq! ( kinds , vec! [ ApprovalKind ::EnableTools , ApprovalKind ::EnableThisComputer ] ) ;
assert! ( result . contains ( " note.txt " ) , " {result} " ) ;
}
2026-11-05 09:29:55 +00:00
/// 需要處理: the model hands a step to the person; the card's request carries the instruction
/// and the computer, 「我完成了」 tells the bot to carry on, 「跳過」 that the step is skipped.
#[ test ]
fn a_hand_off_waits_for_the_person_and_says_what_they_chose ( ) {
use std ::io ::{ Read , Write } ;
use std ::net ::TcpListener ;
for ( answer , expect , status ) in [ ( ApprovalAnswer ::Once , " 我完成了 " , ToolStatus ::Done ) , ( ApprovalAnswer ::Reject , " 跳過 " , ToolStatus ::Denied ) ] {
let l = TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
std ::thread ::spawn ( move | | {
let replies = [
" data: { \" choices \" :[{ \" delta \" :{ \" tool_calls \" :[{ \" index \" :0, \" id \" : \" h1 \" , \" function \" :{ \" name \" : \" hand_over_computer \" , \" arguments \" : \" { \\ \" instruction \\ \" : \\ \" 請登入 Threads \\ \" } \" }}]}}]} \n \n data: [DONE] \n \n " ,
" data: { \" choices \" :[{ \" delta \" :{ \" content \" : \" 好 \" }}]} \n \n data: [DONE] \n \n " ,
] ;
for body in replies {
let ( mut s , _ ) = l . accept ( ) . unwrap ( ) ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 8192 ] ;
loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break ;
}
}
}
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( body . as_bytes ( ) ) ;
}
} ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 幫我登入 Threads " ) ] , tools : vec ! [ ] , session : String ::new ( ) } ,
2026-11-05 13:28:32 +00:00
computers : vec ! [ tools ::Target { name : " 王小明的Mac mini " . into ( ) , place : tools ::Place ::Device { host : " m " . into ( ) , dir : String ::new ( ) } } ] ,
2026-11-05 09:29:55 +00:00
default_computer : " 王小明的Mac mini " . into ( ) ,
roles : tools ::Roles { own : " Master 的電腦 " . into ( ) , person : Some ( " 王小明的Mac mini " . into ( ) ) , server : true } ,
.. test_job ( )
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let ( mut asked , mut result , mut last ) = ( None , String ::new ( ) , None ) ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 10 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
asked = Some ( ( a . kind , a . reason . clone ( ) , a . grant . clone ( ) ) ) ;
let _ = a . reply . send ( answer . clone ( ) ) ;
}
( _ , StreamEvent ::ToolStatus ( _ , st ) ) = > last = Some ( st ) ,
( _ , StreamEvent ::ToolResult ( m ) ) = > result = m . text ,
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert_eq! ( asked , Some ( ( ApprovalKind ::Handoff , " 請登入 Threads " . to_string ( ) , " 王小明的Mac mini " . to_string ( ) ) ) ) ;
assert! ( result . contains ( expect ) , " {result} " ) ;
assert_eq! ( last , Some ( status ) ) ;
}
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
/// 直接操作: tools off, this Mac off, a command that changes files — and still no question.
/// Dangerous commands stay blocked.
#[ test ]
fn direct_mode_asks_nothing_but_still_blocks_danger ( ) {
use std ::io ::{ Read , Write } ;
use std ::net ::TcpListener ;
let l = TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
std ::thread ::spawn ( move | | {
let replies = [
" data: { \" choices \" :[{ \" delta \" :{ \" tool_calls \" :[{ \" index \" :0, \" id \" : \" c1 \" , \" function \" :{ \" name \" : \" shell \" , \" arguments \" : \" { \\ \" command \\ \" : \\ \" echo hi > made.txt \\ \" , \\ \" computer \\ \" : \\ \" 你的電腦 \\ \" } \" }},{ \" index \" :1, \" id \" : \" c2 \" , \" function \" :{ \" name \" : \" shell \" , \" arguments \" : \" { \\ \" command \\ \" : \\ \" sudo ls \\ \" , \\ \" computer \\ \" : \\ \" 你的電腦 \\ \" } \" }}]}}]} \n \n data: [DONE] \n \n " ,
" data: { \" choices \" :[{ \" delta \" :{ \" content \" : \" 好了 \" }}]} \n \n data: [DONE] \n \n " ,
] ;
for body in replies {
let ( mut s , _ ) = l . accept ( ) . unwrap ( ) ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 8192 ] ;
loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break ;
}
}
}
let _ = s . write_all ( b " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " ) ;
let _ = s . write_all ( body . as_bytes ( ) ) ;
}
} ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-direct- {} " , std ::process ::id ( ) ) ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 做一個檔案 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.
Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
list (it was refused outright, while the instructions said it could sudo), and commands with
sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
sandbox broke says so, instead of looking like a broken program.
Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
step's second click landed elsewhere); the box's front window is listed first; a control can
be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
without it came up with no network).
Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
repeated, nothing changed on screen, the quick model switching routes), one in four, and the
final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
them again with no model in between, finding each control again, handing back on a mismatch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w . clone ( ) } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
2026-11-05 08:47:28 +00:00
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
local_execution : false ,
tools_enabled : false ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
local_locked : false ,
grants : vec ! [ ] ,
context_tokens : DEFAULT_CONTEXT_TOKENS ,
via : " test " . into ( ) ,
via_name : " Bot " . into ( ) ,
steer : Arc ::new ( Mutex ::new ( vec! [ ] ) ) ,
review : None ,
demos : vec ! [ ] ,
2026-11-06 15:12:53 +00:00
skills : vec ! [ ] ,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op : String ::new ( ) ,
log : None ,
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
} ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut asked = 0 ;
let mut results = vec! [ ] ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 10 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
asked + = 1 ;
let _ = a . reply . send ( ApprovalAnswer ::Reject ) ;
}
( _ , StreamEvent ::ToolResult ( m ) ) = > results . push ( m . text ) ,
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
assert_eq! ( asked , 0 , " {results:?} " ) ;
assert! ( w . join ( " made.txt " ) . exists ( ) ) ;
assert! ( results [ 1 ] . contains ( " 一律不允許 " ) , " {results:?} " ) ;
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice
Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down):
before something happens a line says it is about to — the turn, each model call, each step once
its checks passed (with the arguments it runs with and whether it may run again), each message
typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still
waiting, usually nothing.
On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything
runs: the message the turn answered comes back if the chat lost it; a step that was running and
reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the
person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows
as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is
saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what
it was doing, what may have run, the messages put back); it goes on only when they press it — in
the window, on a screen of the brain, or on the phone — from where its steps stand, never running
a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays
stopped.
Also in the turn loop:
- 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in
直接操作 mode it used to let the step through unchecked.
- "Too long for this model" folds older messages into a summary and asks the same step again,
once per thing the person said, instead of ending the turn.
- Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from
the newest summary on (it used to delete what it folded).
- A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote
it; the result says where it ran.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// A made-up model that answers each request with the next (HTTP status, body) and keeps
/// what each request said.
fn fake_http ( replies : Vec < ( u16 , String ) > ) -> ( u16 , Arc < Mutex < Vec < String > > > ) {
use std ::io ::{ Read , Write } ;
let l = std ::net ::TcpListener ::bind ( " 127.0.0.1:0 " ) . unwrap ( ) ;
let port = l . local_addr ( ) . unwrap ( ) . port ( ) ;
let seen = Arc ::new ( Mutex ::new ( vec! [ ] ) ) ;
let seen2 = seen . clone ( ) ;
std ::thread ::spawn ( move | | {
for ( status , body ) in replies {
let Ok ( ( mut s , _ ) ) = l . accept ( ) else { return } ;
let mut req = Vec ::new ( ) ;
let mut buf = [ 0 u8 ; 8192 ] ;
loop {
let n = s . read ( & mut buf ) . unwrap_or ( 0 ) ;
if n = = 0 {
break ;
}
req . extend_from_slice ( & buf [ .. n ] ) ;
let t = String ::from_utf8_lossy ( & req ) . to_string ( ) ;
if let Some ( end ) = t . find ( " \r \n \r \n " ) {
let len = t [ .. end ] . lines ( ) . find_map ( | l | l . to_ascii_lowercase ( ) . strip_prefix ( " content-length: " ) . map ( | v | v . trim ( ) . parse ::< usize > ( ) . unwrap_or ( 0 ) ) ) . unwrap_or ( 0 ) ;
if req . len ( ) > = end + 4 + len {
break ;
}
}
}
seen2 . lock ( ) . unwrap ( ) . push ( String ::from_utf8_lossy ( & req ) . to_string ( ) ) ;
let head = if status = = 200 { " HTTP/1.1 200 OK \r \n Content-Type: text/event-stream \r \n Connection: close \r \n \r \n " . to_string ( ) } else { format! ( " HTTP/1.1 {status} Error \r \n Content-Type: application/json \r \n Content-Length: {} \r \n Connection: close \r \n \r \n " , body . len ( ) ) } ;
let _ = s . write_all ( head . as_bytes ( ) ) ;
let _ = s . write_all ( body . as_bytes ( ) ) ;
}
} ) ;
( port , seen )
}
fn sse_text ( t : & str ) -> ( u16 , String ) {
( 200 , format! ( " data: {} \n \n data: [DONE] \n \n " , serde_json ::json! ( { " choices " : [ { " delta " : { " content " : t } } ] } ) ) )
}
fn sse_call ( id : & str , name : & str , args : serde_json ::Value ) -> ( u16 , String ) {
( 200 , format! ( " data: {} \n \n data: [DONE] \n \n " , serde_json ::json! ( { " choices " : [ { " delta " : { " tool_calls " : [ { " index " : 0 , " id " : id , " function " : { " name " : name , " arguments " : args . to_string ( ) } } ] } } ] } ) ) )
}
const OVERFLOW_BODY : & str = r # "{"error":{"message":"This model's maximum context length is 8192 tokens. However, your messages resulted in 9000 tokens."}}"# ;
/// A long chat: enough older messages for a fold to have something to fold.
fn long_chat ( ) -> Vec < Message > {
let mut h = vec! [ ] ;
for i in 0 .. 6 {
h . push ( Message { at : i * 2 + 1 , .. Message ::user ( & format! ( " 第 {i} 個問題 " ) ) } ) ;
h . push ( Message { at : i * 2 + 2 , .. Message ::assistant ( & format! ( " 第 {i} 個回答 " ) , " test " ) } ) ;
}
h . push ( Message { at : 100 , .. Message ::user ( " 現在呢? " ) } ) ;
h
}
/// Run a job to its end, collecting what came back. Approvals get `answer`.
fn drive ( job : Job , answer : ApprovalAnswer ) -> ( Vec < StreamEvent > , Vec < String > ) {
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , Arc ::new ( AtomicBool ::new ( false ) ) , eframe ::egui ::Context ::default ( ) ) ;
let mut evs = vec! [ ] ;
let mut asked = vec! [ ] ;
loop {
let ( _ , ev ) = rx . recv_timeout ( Duration ::from_secs ( 15 ) ) . expect ( " loop stalled " ) ;
match ev {
StreamEvent ::NeedApproval ( a ) = > {
asked . push ( a . reason . clone ( ) ) ;
let _ = a . reply . send ( answer . clone ( ) ) ;
}
StreamEvent ::Done | StreamEvent ::Error ( _ ) = > {
evs . push ( ev ) ;
break ;
}
ev = > evs . push ( ev ) ,
}
}
( evs , asked )
}
#[ test ]
fn a_verdict_is_clear_or_the_review_failed ( ) {
assert! ( matches! ( parse_verdict ( r # "{"verdict":"ask","rule":"刪除前先問"}"# ) , ReviewVerdict ::Ask ( r ) if r = = " 刪除前先問 " ) ) ;
assert! ( matches! ( parse_verdict ( r # "好的 {"verdict":"allow","rule":"讀檔"}"# ) , ReviewVerdict ::Allow ( _ ) ) ) ;
assert! ( matches! ( parse_verdict ( r # "{"verdict":"none","rule":""}"# ) , ReviewVerdict ::NoMatch ) ) ;
for broken in [ " " , " 嗯…讓我想想 " , r # "{"verdict":"maybe"}"# , r # "} {"# ] {
assert! ( matches! ( parse_verdict ( broken ) , ReviewVerdict ::Failed ) , " {broken} " ) ;
}
}
/// 直接操作 with an 「一律詢問」 rule: when the reviewer doesn't give a verdict, the step is
/// asked about, never run unchecked.
#[ test ]
fn a_review_that_fails_asks_instead_of_allowing ( ) {
let ( port , _ ) = fake_http ( vec! [
sse_call ( " c1 " , " shell " , serde_json ::json! ( { " command " : " echo hi > made.txt " , " computer " : crate ::model ::THIS_COMPUTER_NAME } ) ) ,
sse_text ( " 嗯…讓我想想 " ) ,
sse_text ( " 好,我不做了 " ) ,
] ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-review- {} " , std ::process ::id ( ) ) ) ;
let _ = std ::fs ::remove_dir_all ( & w ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 做一個檔案 " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w . clone ( ) } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
direct : Arc ::new ( AtomicBool ::new ( true ) ) ,
review : Some ( Review { allow : vec ! [ ] , ask : vec ! [ " 刪除或修改任何檔案前先問我 " . into ( ) ] } ) ,
.. test_job ( )
} ;
let ( evs , asked ) = drive ( job , ApprovalAnswer ::Reject ) ;
assert_eq! ( asked . len ( ) , 1 , " {asked:?} " ) ;
assert! ( asked [ 0 ] . contains ( REVIEW_FAILED_NOTE ) , " {} " , asked [ 0 ] ) ;
assert! ( ! w . join ( " made.txt " ) . exists ( ) ) ;
assert! ( matches! ( evs . last ( ) , Some ( StreamEvent ::Done ) ) ) ;
}
/// The model says the chat is too long: older messages are folded into a summary and the same
/// step is asked again (with the summary), instead of the turn failing.
#[ test ]
fn an_overflow_folds_the_chat_and_asks_again ( ) {
let ( port , seen ) = fake_http ( vec! [ ( 400 , OVERFLOW_BODY . into ( ) ) , sse_text ( " 摘要:問過六個問題 " ) , sse_text ( " 好了 " ) ] ) ;
let job = Job { req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : long_chat ( ) , tools : vec ! [ ] , session : String ::new ( ) } , .. test_job ( ) } ;
let ( evs , _ ) = drive ( job , ApprovalAnswer ::Reject ) ;
assert! ( matches! ( evs . last ( ) , Some ( StreamEvent ::Done ) ) , " {:?} " , evs . iter ( ) . filter_map ( | e | if let StreamEvent ::Error ( e ) = e { Some ( e . clone ( ) ) } else { None } ) . collect ::< Vec < _ > > ( ) ) ;
assert! ( evs . iter ( ) . any ( | e | matches! ( e , StreamEvent ::Compacted ( .. ) ) ) ) ;
let texts : Vec < String > = evs . iter ( ) . filter_map ( | e | if let StreamEvent ::StepDone ( m ) = e { Some ( m . text . clone ( ) ) } else { None } ) . collect ( ) ;
assert_eq! ( texts , vec! [ " 好了 " ] ) ;
let seen = seen . lock ( ) . unwrap ( ) ;
assert_eq! ( seen . len ( ) , 3 ) ;
assert! ( seen [ 2 ] . contains ( " 摘要:問過六個問題 " ) & & ! seen [ 2 ] . contains ( " 第 0 個問題 " ) , " {} " , seen [ 2 ] ) ;
}
/// Still too long after folding once: the turn ends with a clear error (no loop of folds).
#[ test ]
fn a_second_overflow_ends_the_turn ( ) {
let ( port , seen ) = fake_http ( vec! [ ( 400 , OVERFLOW_BODY . into ( ) ) , sse_text ( " 摘要 " ) , ( 400 , OVERFLOW_BODY . into ( ) ) , sse_text ( " 不該問到這裡 " ) ] ) ;
let job = Job { req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : long_chat ( ) , tools : vec ! [ ] , session : String ::new ( ) } , .. test_job ( ) } ;
let ( evs , _ ) = drive ( job , ApprovalAnswer ::Reject ) ;
match evs . last ( ) {
Some ( StreamEvent ::Error ( e ) ) = > assert! ( providers ::is_context_overflow ( e ) & & e . contains ( " 還是太長 " ) , " {e} " ) ,
_ = > panic! ( " expected an error " ) ,
}
assert_eq! ( seen . lock ( ) . unwrap ( ) . len ( ) , 3 ) ;
}
/// The work log while a turn runs: the turn and its model call are written first; a step
/// waiting for the person isn't "started"; once allowed it is (with the arguments it runs
/// with), then finished; a finished turn leaves no file. A stopped turn ends as stopped.
#[ test ]
fn the_work_log_follows_the_turn ( ) {
use crate ::oplog ::{ reduce , OpLog } ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-oplog-turn- {} " , std ::process ::id ( ) ) ) ;
let _ = std ::fs ::remove_dir_all ( & w ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
std ::fs ::write ( w . join ( " a.txt " ) , " hi " ) . unwrap ( ) ;
let path = w . join ( " log.jsonl " ) ;
for stop in [ false , true ] {
let ( port , _ ) = fake_http ( vec! [ sse_call ( " c1 " , " shell " , serde_json ::json! ( { " command " : " echo hi > b.txt " } ) ) , sse_text ( " 做好了 " ) ] ) ;
let cancel = Arc ::new ( AtomicBool ::new ( false ) ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message { at : 42 , .. Message ::user ( " 做 b.txt " ) } ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [ tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w . clone ( ) } } ] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
local_execution : true ,
op : " turn-test " . into ( ) ,
log : Some ( OpLog ::at ( path . clone ( ) ) ) ,
.. test_job ( )
} ;
let read = | | reduce ( & OpLog ::at ( path . clone ( ) ) . read ( ) ) ;
let raw = | | std ::fs ::read_to_string ( & path ) . unwrap_or_default ( ) ;
let ( tx , rx ) = channel ( ) ;
spawn ( job , 1 , tx , cancel . clone ( ) , eframe ::egui ::Context ::default ( ) ) ;
loop {
match rx . recv_timeout ( Duration ::from_secs ( 15 ) ) . expect ( " loop stalled " ) {
( _ , StreamEvent ::NeedApproval ( a ) ) = > {
let open = read ( ) . open . expect ( " the turn is open " ) ;
assert_eq! ( ( open . op . as_str ( ) , open . prompt . map ( | p | p . at ) ) , ( " turn-test " , Some ( 42 ) ) ) ;
assert! ( open . started . is_empty ( ) , " waiting for the person is not started " ) ;
assert! ( raw ( ) . contains ( " \" step_attempt \" " ) ) ;
if stop {
cancel . store ( true , Ordering ::Relaxed ) ;
drop ( a ) ;
break ;
}
let _ = a . reply . send ( ApprovalAnswer ::Once ) ;
}
( _ , StreamEvent ::ToolResult ( _ ) ) = > {
let open = read ( ) . open . expect ( " the turn is still open " ) ;
assert_eq! ( open . started . len ( ) , 1 ) ;
assert! ( open . started [ 0 ] . finished & & open . started [ 0 ] . replay = = crate ::oplog ::Replay ::Never ) ;
assert! ( raw ( ) . contains ( " echo hi > b.txt " ) ) ;
}
( _ , StreamEvent ::Done ) = > break ,
( _ , StreamEvent ::Error ( e ) ) = > panic! ( " {e} " ) ,
_ = > { }
}
}
if stop {
// The thread winds down on its own; its last line says stopped.
let until = std ::time ::Instant ::now ( ) + Duration ::from_secs ( 5 ) ;
while path . exists ( ) & & std ::time ::Instant ::now ( ) < until {
std ::thread ::sleep ( Duration ::from_millis ( 50 ) ) ;
}
}
assert! ( ! path . exists ( ) , " a closed turn leaves nothing behind: {} " , raw ( ) ) ;
}
}
/// A step moved to the chosen computer runs there, but the chat keeps the call as the model
/// wrote it; the result says where it ran.
#[ test ]
fn a_moved_step_is_kept_as_the_model_asked ( ) {
let ( port , _ ) = fake_http ( vec! [ sse_call ( " c1 " , " list_dir " , serde_json ::json! ( { " computer " : " 王小明的Mac mini " } ) ) , sse_text ( " 好 " ) ] ) ;
let w = std ::env ::temp_dir ( ) . join ( format! ( " anybot-moved- {} " , std ::process ::id ( ) ) ) ;
std ::fs ::create_dir_all ( & w ) . unwrap ( ) ;
std ::fs ::write ( w . join ( " here.txt " ) , " hi " ) . unwrap ( ) ;
let job = Job {
req : ChatRequest { kind : ProviderKind ::Custom , base_url : format ! ( " http://127.0.0.1:{port}/v1 " ) , api_key : String ::new ( ) , model : " m " . into ( ) , system : String ::new ( ) , history : vec ! [ Message ::user ( " 有什麼檔案? " ) ] , tools : tools ::specs ( ) , session : String ::new ( ) } ,
computers : vec ! [
tools ::Target { name : crate ::model ::THIS_COMPUTER_NAME . into ( ) , place : tools ::Place ::Local { workdir : w . clone ( ) } } ,
tools ::Target { name : " 王小明的Mac mini " . into ( ) , place : tools ::Place ::Device { host : " m " . into ( ) , dir : String ::new ( ) } } ,
] ,
default_computer : crate ::model ::THIS_COMPUTER_NAME . into ( ) ,
roles : tools ::Roles { own : crate ::model ::BOT_COMPUTER_NAME . into ( ) , person : Some ( crate ::model ::THIS_COMPUTER_NAME . into ( ) ) , server : false } ,
local_execution : true ,
.. test_job ( )
} ;
let ( evs , _ ) = drive ( job , ApprovalAnswer ::Once ) ;
let asked = evs . iter ( ) . find_map ( | e | if let StreamEvent ::StepDone ( m ) = e { m . tool_calls . first ( ) . cloned ( ) } else { None } ) . unwrap ( ) ;
assert! ( asked . args . contains ( " 王小明的Mac mini " ) , " {} " , asked . args ) ;
let result = evs . iter ( ) . find_map ( | e | if let StreamEvent ::ToolResult ( m ) = e { Some ( m . text . clone ( ) ) } else { None } ) . unwrap ( ) ;
assert! ( result . contains ( " 這一步你指定了「王小明的Mac mini」 " ) & & result . contains ( " here.txt " ) , " {result} " ) ;
}
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
}