AnyBot/src/agent.rs

3349 lines
192 KiB
Rust
Raw Normal View History

// The work loop: ask the model, run the tools it asks for (asking the person first when the
// rules say so), give it the results, and repeat until it answers in plain words.
// Long chats are folded into a summary before they outgrow the model's context.
use base64::Engine;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc::{channel, Sender};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use crate::appctl;
use crate::computer;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
use crate::model::{Message, Role, ToolCall, ToolStatus};
use crate::providers::{self, AppCall, Approval, ApprovalAnswer, ApprovalKind, ChatRequest, StreamEvent};
use crate::tools::{self, Decision};
use crate::web;
/// Context size assumed when the person hasn't set one (estimated).
pub const DEFAULT_CONTEXT_TOKENS: u64 = 128_000;
/// Summarize once the chat fills this share of the context:
/// `client_speculative_summarization_config.tokenUsageThresholdPercentage:70`.
pub const COMPACT_THRESHOLD_PERCENT: u64 = 70;
/// Extra room allowed in the middle of a task before summarizing again:
/// same config, `tolerancePercentage:5`.
pub const COMPACT_TOLERANCE_PERCENT: u64 = 5;
/// Most model calls in one turn: a guard against a runaway turn, not a budget. the reference assistant leaves
/// its turn window open `cloud_agent_stream_conversation_turn_window`
/// `fallbackValues:{maxTurns:null}`) and browser-use stops at 100; 25 cut off real work (fixing
/// a server's Docker took 50+ commands). Going in circles is stopped long before this by step_key.
pub const MAX_STEPS: usize = 100;
/// Every this many steps the bot is asked whether it is getting anywhere (estimated).
pub const CHECK_IN_STEPS: usize = 25;
/// Newest messages always kept word for word (estimated).
pub const KEEP_RECENT_MESSAGES: usize = 6;
/// Characters of each old message given to the summarizer (estimated).
const SUMMARY_SNIPPET_CHARS: usize = 2_000;
pub struct Job {
pub req: ChatRequest,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// The quick model that carries out planned steps (same provider), or None: every step uses
/// `req.model`. See `step_plan`.
pub step_model: Option<String>,
/// Computers this bot may work on (empty when computer tools are off).
pub computers: Vec<tools::Target>,
pub default_computer: String,
/// Which computer is the bot's own, which one the person is using, and whether this is a server.
pub roles: tools::Roles,
/// 在這台電腦上執行 is on.
pub local_execution: bool,
/// This bot's computer tools are on (when off, using them asks the person first).
pub tools_enabled: bool,
/// 直接操作 mode; the window can switch it on mid-turn ("全部允許").
pub direct: Arc<AtomicBool>,
/// 在這台電腦上執行 is locked off by an admin policy: refuse, don't ask.
pub local_locked: bool,
pub grants: Vec<String>,
pub context_tokens: u64,
pub via: String,
/// The bot's name, for the pop-up text.
pub via_name: String,
/// Messages the person sends while the turn runs (they steer it; no new turn).
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
pub steer: Arc<Mutex<Vec<Queued>>>,
/// 自動審核 rules, when on.
pub review: Option<Review>,
/// 示範教學 this bot has learned (for view_demo_step).
pub demos: Vec<crate::teach::Demo>,
/// The skills there are (a longer job is looked back on against them; see learn).
pub skills: Vec<crate::memory::Skill>,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// This turn's id: its lines in the bot's work log carry it.
pub op: String,
/// The bot's work log (see oplog.rs); None where nothing is kept (tests, the demo).
pub log: Option<crate::oplog::OpLog>,
}
impl Job {
/// Write a line to the work log (on disk before this returns).
fn note(&self, r: crate::oplog::Record) {
if let Some(l) = &self.log {
l.append(&r);
}
}
}
/// A message the person typed while the bot works: its words, when it was typed, and the id its
/// work-log lines carry (so one typed just before a crash isn't lost).
#[derive(Clone, Debug)]
pub struct Queued {
pub id: String,
pub text: String,
pub at: i64,
}
impl Queued {
pub fn new(text: impl Into<String>) -> Self {
Queued { id: crate::oplog::new_id(), text: text.into(), at: crate::model::now_ms() }
}
}
/// A turn's last line in the work log, written once: explicitly (before Done / Error reach the
/// window), else when the turn's thread ends any other way — stopped (aborted), or a panic
/// (failed).
struct Closing {
log: Option<crate::oplog::OpLog>,
op: String,
cancel: Arc<AtomicBool>,
written: bool,
}
impl Closing {
fn close(&mut self, outcome: crate::oplog::Outcome) {
if !self.written {
self.written = true;
if let Some(l) = &self.log {
l.finish(&self.op, outcome);
}
}
}
}
impl Drop for Closing {
fn drop(&mut self) {
let o = if self.cancel.load(Ordering::Relaxed) { crate::oplog::Outcome::Aborted } else { crate::oplog::Outcome::Failed };
self.close(o);
}
}
#[derive(Clone, Default)]
pub struct Review {
/// 「無需詢問直接允許」規則
pub allow: Vec<String>,
/// 「一律詢問」規則 (wins when both match)
pub ask: Vec<String>,
}
pub enum ReviewVerdict {
Allow(String),
Ask(String),
NoMatch,
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// The reviewer didn't answer, or answered something that isn't a verdict: nobody checked the
/// rules, so this never counts as "no rule matched" (a check that breaks, blocks).
Failed,
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// Said in the question when the reviewer couldn't check a step.
pub const REVIEW_FAILED_NOTE: &str = "(自動審核沒有回應,所以先問你)";
/// Ask the model whether an action matches the person's rules. "Ask first" wins on conflict
/// (NFRGrQ: 規則衝突時,「先詢問」優先); built-in safety checks have already run before this.
fn review_action(job: &Job, review: &Review, action: &str, cancel: &AtomicBool) -> ReviewVerdict {
if review.allow.is_empty() && review.ask.is_empty() {
return ReviewVerdict::NoMatch;
}
let list = |v: &[String]| if v.is_empty() { "(無)".to_string() } else { v.iter().enumerate().map(|(i, r)| format!("{}. {r}", i + 1)).collect::<Vec<_>>().join("\n") };
let req = ChatRequest {
system: "你是動作審核員。只根據使用者寫的規則判斷,不要自己加規則。只回一行 JSON:{\"verdict\":\"ask\"|\"allow\"|\"none\",\"rule\":\"符合的規則原文\"}。如果符合任何「一律詢問」規則就回 ask(優先);否則符合「無需詢問直接允許」規則就回 allow;都不符合回 none。".into(),
history: vec![Message::user(&format!("要審核的動作:{action}\n\n「一律詢問」規則:\n{}\n\n「無需詢問直接允許」規則:\n{}", list(&review.ask), list(&review.allow)))],
tools: vec![],
..job.req.clone()
};
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let Ok(out) = providers::run_step(&req, cancel, &|_| {}) else { return ReviewVerdict::Failed };
parse_verdict(&out.text)
}
/// The reviewer's one line of JSON. Anything but a clear ask / allow / none is a failed review.
fn parse_verdict(text: &str) -> ReviewVerdict {
let text = text.trim();
// A truncated answer can put the last "}" before the first "{" — slicing then panics.
let json_part = text.find('{').and_then(|a| text.rfind('}').filter(|b| *b >= a).map(|b| &text[a..=b])).unwrap_or(text);
let v: serde_json::Value = serde_json::from_str(json_part).unwrap_or_default();
let rule = v["rule"].as_str().unwrap_or("").to_string();
match v["verdict"].as_str() {
Some("ask") => ReviewVerdict::Ask(rule),
Some("allow") => ReviewVerdict::Allow(rule),
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
Some("none") => ReviewVerdict::NoMatch,
_ => ReviewVerdict::Failed,
}
}
/// 自己進化: after a longer job, the model looks back on its steps and keeps the way that worked
/// as a skill (or improves the one it followed), so the next time takes the short path. It runs
/// after Done: the person isn't kept waiting.
fn learn(job: &Job, history: &[Message], cancel: &AtomicBool, send: &dyn Fn(StreamEvent)) {
let Some(start) = history.iter().rposition(|m| m.role == Role::User && !m.steer) else { return };
let turn = &history[start..];
if !crate::memory::worth_learning(turn) {
return;
}
let req = ChatRequest {
system: crate::memory::LEARN_SYSTEM.into(),
history: vec![Message::user(&crate::memory::learn_brief(turn, &job.skills))],
tools: vec![],
..job.req.clone()
};
let out = match providers::run_step(&req, cancel, &|_| {}) {
Ok(o) => o,
Err(e) => {
eprintln!("AnyBot:整理做法時模型沒有回應:{e}");
return;
}
};
match (crate::memory::learned_skill(&out.text), turn.last()) {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
(Some(mut k), Some(reply)) => {
k.steps = crate::memory::replay_steps(turn, &crate::memory::learned_replay(&out.text));
eprintln!("AnyBot:學到做法「{}」", k.name);
send(StreamEvent::Learned(k, reply.at));
}
_ => eprintln!("AnyBot:這次的做法沒有存:{}", out.text.split_whitespace().collect::<Vec<_>>().join(" ").chars().take(200).collect::<String>()),
}
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// Take what the person typed mid-turn and add it to the chat as steering. (The window marks it
/// delivered in the work log once the chat holding it is saved; the message keeps the time it was
/// typed, which is how a restart tells it is already there.)
fn drain_steer(job: &Job, history: &mut Vec<Message>, send: &dyn Fn(StreamEvent)) -> bool {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let queued: Vec<Queued> = std::mem::take(&mut *job.steer.lock().unwrap());
for q in &queued {
let m = Message { steer: true, at: q.at, ..Message::user(&q.text) };
send(StreamEvent::Steered(m.clone(), q.id.clone()));
history.push(m);
}
!queued.is_empty()
}
/// Which computer a call asks for (permission to use it is checked separately).
fn target_for(job: &Job, args: &str) -> Result<tools::Target, String> {
let v = tools::args_of(args);
let want = v.get("computer").and_then(|c| c.as_str()).filter(|c| !c.is_empty()).unwrap_or(&job.default_computer).to_string();
job.computers.iter().find(|t| t.name == want).cloned().ok_or_else(|| not_here(job, &want))
}
/// One of the bot's own browsers (where, and which profile).
#[derive(Clone, PartialEq, Eq, Hash, Debug)]
enum BotBrowser {
Here(String),
Box(String),
Device(String, String),
}
/// The bot's browsers a job used. When the job ends — answered, stopped or failed — each closes
/// the tabs the job left open and keeps the page it ended on (every browser_open is a new tab;
/// nothing used to close them, and the box ran out of memory with 19). Only the job's own tabs:
/// another job's, and the person's, stay (see botbrowser's "each job its own tabs").
struct UsedBrowsers {
list: Vec<BotBrowser>,
via_name: String,
/// This job's name in the browsers (its tabs are kept apart by it).
job: String,
}
/// Names the jobs of this run (a bot's job on another computer's browser carries it along).
static JOB_SEQ: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(1);
/// A call's arguments with the job's name added: the browser keeps each job's tabs apart by it
/// (also on another computer's browser, which gets the arguments as they are).
fn with_job(args: &str, job: &str) -> String {
let mut v: serde_json::Value = serde_json::from_str(if args.trim().is_empty() { "{}" } else { args }).unwrap_or_else(|_| serde_json::json!({}));
if let Some(o) = v.as_object_mut() {
o.insert("_job".into(), serde_json::json!(job));
}
v.to_string()
}
impl UsedBrowsers {
fn new(via_name: &str, bot_id: u64) -> Self {
let n = JOB_SEQ.fetch_add(1, Ordering::Relaxed);
UsedBrowsers { list: vec![], via_name: via_name.to_string(), job: format!("{bot_id}-{}-{n}", crate::model::now_ms()) }
}
fn add(&mut self, b: BotBrowser) {
if !self.list.contains(&b) {
self.list.push(b);
}
}
fn close_tabs(&mut self) {
for b in std::mem::take(&mut self.list) {
let args = |p: &str| with_job(&if p.is_empty() { r#"{"action":"tidy"}"#.to_string() } else { serde_json::json!({"action": "tidy", "profile": p}).to_string() }, &self.job);
let never = AtomicBool::new(false);
match &b {
BotBrowser::Here(p) => {
let _ = web::run_full("browser_tabs", &args(p), &never);
}
BotBrowser::Box(p) => {
let _ = crate::botbrowser::in_box(|| web::run_full("browser_tabs", &args(p), &never));
}
BotBrowser::Device(host, p) => {
let _ = crate::remote::run_on_device(host, "browser_tabs", &args(p), &self.via_name);
}
}
}
}
}
impl Drop for UsedBrowsers {
fn drop(&mut self) {
self.close_tabs();
}
}
enum BrowserAt {
Here,
Box,
Device(String),
}
/// Whose browser a browser step uses: the computer it names, else the bot's chosen computer; a
/// computer without a browser (a server, an SSH one) hands it to the person's computer.
fn browser_place(job: &Job, name: &str, args: &str) -> (BrowserAt, String) {
if name == "web_fetch" {
return (BrowserAt::Here, String::new());
}
let v = tools::args_of(args);
let want = v.get("computer").and_then(|c| c.as_str()).map(str::trim).filter(|c| !c.is_empty()).unwrap_or(&job.default_computer).to_string();
let at = |t: &tools::Target| -> Option<BrowserAt> {
match &t.place {
tools::Place::Device { host, .. } => Some(BrowserAt::Device(host.clone())),
tools::Place::Box { .. } if crate::cloudbox::available() => Some(BrowserAt::Box),
tools::Place::Local { .. } if crate::botbrowser::available() => Some(BrowserAt::Here),
_ => None,
}
};
if let Some((b, n)) = job.computers.iter().find(|t| t.name == want).and_then(|t| at(t).map(|b| (b, t.name.clone()))) {
return (b, n);
}
let person = roles_now(job).person.and_then(|p| job.computers.iter().find(|t| t.name == p));
match person.and_then(|t| at(t).map(|b| (b, t.name.clone()))).or_else(|| job.computers.iter().find_map(|t| at(t).map(|b| (b, t.name.clone())))) {
Some(x) => x,
None => (BrowserAt::Here, String::new()),
}
}
/// Is this browser step for the bot's own browser (no permission, no person's computer)? The
/// box's Chromium always is — whether *this* computer has a Chrome doesn't matter there. (Asking
/// this computer made a cloud brain, which has no Chrome, send every browser_open/read meant for
/// the box to the person's browser on the server: 「打不開瀏覽器」, 「Safari 沒有回應」.)
fn bots_browser(place: &BrowserAt, name: &str, args: &str) -> bool {
matches!(place, BrowserAt::Box) || !web::uses_this_computer(name, args)
}
/// Substring search with word boundaries on the ASCII parts: "mac" must not light up inside
/// "macOS", and a device named "Ann" must not light up on "anniversary". Loose matching let
/// unrelated words — including words from page text that arrives inside a message — unlock a
/// computer the person never named, defeating the guard below.
fn finds(hay: &str, needle: &str) -> bool {
let hb: Vec<char> = hay.chars().collect();
let nb: Vec<char> = needle.chars().collect();
if nb.is_empty() || nb.len() > hb.len() {
return false;
}
let alnum = |c: char| c.is_ascii_alphanumeric();
'start: for i in 0..=hb.len() - nb.len() {
if hb[i] != nb[0] {
continue;
}
for j in 1..nb.len() {
if hb[i + j] != nb[j] {
continue 'start;
}
}
if alnum(nb[0]) && i > 0 && alnum(hb[i - 1]) {
continue;
}
if alnum(*nb.last().unwrap()) && i + nb.len() < hb.len() && alnum(hb[i + nb.len()]) {
continue;
}
return true;
}
false
}
/// Words that mean a computer in the person's message: its name (or the part after 「的」, so
/// 「Mac mini」 names 「王小明的Mac mini」), and for the bot's own computer and the one the person
/// is using, the everyday words for them.
fn mentions(text: &str, t: &tools::Target, roles: &tools::Roles) -> bool {
let low = text.to_lowercase();
let name = t.name.to_lowercase();
if finds(&low, &name) {
return true;
}
if let Some((_, tail)) = name.rsplit_once('的') {
let tail = tail.trim();
if tail.chars().count() >= 3 && tail != "電腦" && finds(&low, tail) {
return true;
}
}
if finds(&name, "mac") && finds(&low, "mac") {
return true;
}
let words: &[&str] = if t.name == roles.own {
&["bot 的電腦", "bot的電腦", "你的電腦", "你自己的電腦", "你的虛擬", "虛擬電腦", "雲端電腦"]
} else if roles.person.as_deref() == Some(t.name.as_str()) {
&["我的電腦", "我電腦", "我這台", "這台電腦", "本機", "我的筆電", "my computer"]
} else {
&[]
};
if t.name == roles.own && finds(&low, &name.replace(' ', "")) {
return true;
}
words.iter().any(|w| finds(&low, w))
}
/// A step naming a computer other than the chosen one, which the person didn't ask for, goes
/// to the chosen one — and the bot is told so (quietly moving it made a bot "ask your Mac" six
/// times while every step landed on its own computer).
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
fn keep_to_chosen(job: &Job, history: &[Message], name: &str, args: &str) -> (String, Option<String>) {
let mut v = tools::args_of(args);
let Some(named) = v.get("computer").and_then(|c| c.as_str()).map(|c| c.trim().to_string()).filter(|c| !c.is_empty()) else { return (args.to_string(), None) };
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// The chosen computer can't do this kind of step (no screen, no browser): moving it there
// would only fail, so the step stays where the bot sent it.
match job.computers.iter().find(|t| t.name == job.default_computer) {
Some(chosen) if named != job.default_computer && can_do(chosen, name) => {}
_ => return (args.to_string(), None),
}
let said = history.iter().rev().filter(|m| m.role == Role::User && !m.summary).take(3).map(|m| m.text.as_str()).collect::<Vec<_>>().join("\n");
match job.computers.iter().find(|t| t.name == named) {
Some(t) if mentions(&said, t, &roles_now(job)) => (args.to_string(), None),
// A computer that isn't on the list now (switched off, its AnyBot closed): the step says
// so instead of quietly running somewhere else.
None => (args.to_string(), None),
_ => {
v["computer"] = serde_json::Value::String(job.default_computer.clone());
let note = format!("(這一步你指定了「{named}」,但使用者選的電腦是「{}」,而且他沒有說要換,所以這一步在「{}」上做。真的需要別台,先問使用者。)\n", job.default_computer, job.default_computer);
(v.to_string(), Some(note))
}
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// Can this computer do this kind of step: a screen step needs a screen there, a browser step a
/// browser; anything else runs anywhere.
fn can_do(t: &tools::Target, tool: &str) -> bool {
let screen = computer::is_computer_tool(tool);
let browser = web::is_web_tool(tool) && tool != "web_fetch";
if !screen && !browser {
return true;
}
match &t.place {
tools::Place::Device { .. } => true,
tools::Place::Box { .. } => crate::cloudbox::available(),
tools::Place::Local { .. } if screen => computer::screen_supported(),
tools::Place::Local { .. } => crate::botbrowser::available(),
_ => false,
}
}
/// What element `e7` is, from the latest app list in the chat (「按鈕「7」」).
fn element_named(history: &[Message], r: &str) -> Option<String> {
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let tag = format!("[{}] ", crate::marks::number_of(r)?);
history.iter().rev().filter(|m| m.role == Role::Tool).take(12).find_map(|m| {
m.text.lines().find_map(|l| l.trim_start().strip_prefix(&tag).map(|rest| rest.trim().to_string())).filter(|e| !e.is_empty())
})
}
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
/// What makes two screen or browser steps "the same" (None for other tools): the tool, the
/// computer and the target, a click's spot on a 16-pixel grid (an inch beside an empty strip is
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// the same miss). Looking, waiting, and moving through a page (scrolling, keys, back) aren't
/// counted: doing those again and again is how a long page or a list is read.
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
fn step_key(name: &str, args: &str) -> Option<String> {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
if !(computer::is_computer_tool(name) || web::is_web_tool(name))
|| matches!(name, "computer_use_screenshot" | "computer_use_show" | "browser_read" | "browser_screenshot" | "browser_wait" | "computer_use_scroll" | "browser_scroll" | "computer_use_press_key" | "browser_press" | "browser_back")
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
{
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
return None;
}
let mut v = tools::args_of(args);
if let Some(o) = v.as_object_mut() {
for k in ["x", "y"] {
if let Some(n) = o.get(k).and_then(serde_json::Value::as_f64) {
o.insert(k.into(), serde_json::Value::from((n / 16.0).round() as i64));
}
}
}
Some(format!("{name} {v}"))
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// The bot's own computer can come or go mid-turn (Docker started or stopped): its place follows
/// what is true now, and the bot is told when it changed. (Fixed at the start of the turn, a box
/// that came up mid-turn stayed "no screen" and every step there ran in the wrong place.)
fn refresh_own(job: &mut Job, bot_id: u64) -> Option<String> {
let own = job.roles.own.clone();
let t = job.computers.iter_mut().find(|t| t.name == own)?;
match (&t.place, crate::cloudbox::available()) {
(tools::Place::Sealed { .. }, true) => {
t.place = tools::Place::Box { dir: crate::cloudbox::bot_dir(bot_id) };
Some(format!("(「{own}」現在可以用了:它是你自己的 Linux 桌面,可以執行指令、看畫面、用瀏覽器。)"))
}
(tools::Place::Box { .. }, false) => {
t.place = tools::Place::Sealed { root: crate::model::bot_computer_dir(bot_id) };
Some(format!("(「{own}」現在連不到 Docker:暫時不能執行指令、看畫面。)"))
}
_ => None,
}
}
/// How many times in a row this very call (same tool, same arguments) already came back with the
/// very same answer among the latest steps. Waiting tools are left out: there the same answer is
/// the point.
fn same_answers(history: &[Message], name: &str, args: &str) -> usize {
if matches!(name, "read_terminal" | "browser_wait" | "write_terminal") {
return 0;
}
let want = tools::args_of(args);
let calls: std::collections::HashMap<&str, (&str, &str)> = history.iter().rev().take(80).flat_map(|m| m.tool_calls.iter()).map(|c| (c.id.as_str(), (c.name.as_str(), c.args.as_str()))).collect();
// A warning put in front of an answer doesn't make it a different answer.
let bare = |t: &str| -> String { t.lines().skip_while(|l| l.starts_with("⚠️")).collect::<Vec<_>>().join("\n") };
let mut first: Option<String> = None;
let mut n = 0;
for m in history.iter().rev().filter(|m| m.role == Role::Tool).take(12) {
let Some((cn, ca)) = calls.get(m.tool_call_id.as_str()) else { continue };
if *cn != name || tools::args_of(ca) != want {
continue;
}
let text = bare(&m.text);
match &first {
None => {
first = Some(text);
n = 1;
}
Some(f) if *f == text => n += 1,
_ => break,
}
}
n
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// "Always allow" for this turn: the grant is kept (once).
fn keep_grant(grants: &mut Vec<String>, g: String) {
if !g.is_empty() && !grants.contains(&g) {
grants.push(g);
}
}
/// Every CHECK_IN_STEPS steps, the newest result carries a question for the bot: still getting
/// somewhere, or stuck? Only the model sees it (the chat keeps the result as it was).
fn check_in(history: &mut [Message], step: usize) {
if step == 0 || !step.is_multiple_of(CHECK_IN_STEPS) {
return;
}
if let Some(m) = history.last_mut().filter(|m| m.role == Role::Tool) {
m.text.push_str(&format!("\n\n(這一回合已經做了 {step} 步,最多 {MAX_STEPS} 步。方向對、有進展就繼續做完;如果最近幾步都沒有進展,停下來用一兩句話告訴使用者卡在哪、需要他做什麼。)"));
}
}
/// A step's result, with a word first when the step was moved back to the chosen computer.
fn noted(moved: &std::collections::HashMap<String, String>, id: &str, text: &str) -> String {
match moved.get(id) {
Some(n) => format!("{n}{text}"),
None => text.to_string(),
}
}
/// A computer the step names that isn't among this bot's computers right now, in plain words.
fn not_here(job: &Job, name: &str) -> String {
format!("「{name}」現在連不到:那台電腦可能沒開,或上面的 AnyBot 沒開、沒登入。請告訴使用者,等它連上再做(或問他要不要改用別台)。現在可以用:{}", job.computers.iter().map(|t| t.name.as_str()).collect::<Vec<_>>().join("、"))
}
/// Which computer is the person's right now: a message typed on one of the account's computers
/// while the turn runs (a steer, or the answer to a hand-off) moves it there — the turn's own
/// idea was fixed when it started, maybe from the phone, which isn't a computer.
fn roles_now(job: &Job) -> tools::Roles {
let mut r = job.roles.clone();
let host = crate::remote::person_host();
if !host.is_empty() {
if let Some(t) = job.computers.iter().find(|t| matches!(&t.place, tools::Place::Device { host: h, .. } if *h == host)) {
r.person = Some(t.name.clone());
}
}
r
}
/// Does the call name a computer?
fn names_a_computer(args: &str) -> bool {
tools::args_of(args).get("computer").and_then(|c| c.as_str()).map(|c| !c.trim().is_empty()).unwrap_or(false)
}
/// Rough token count: about 4 Latin characters or 1 Chinese character per token (estimated).
/// Which screen a computer_use call is for: the one named, else the bot's default when it has a
/// screen, else the bot's own desktop when there is one, else this Mac.
fn screen_target(job: &Job, args: &str) -> Result<tools::Target, String> {
let v = tools::args_of(args);
let has_screen = |t: &tools::Target| matches!(t.place, tools::Place::Box { .. } | tools::Place::Local { .. } | tools::Place::Device { .. });
let named = v.get("computer").and_then(|c| c.as_str()).map(str::trim).filter(|c| !c.is_empty());
if let Some(n) = named {
let t = job.computers.iter().find(|t| t.name == n).ok_or_else(|| not_here(job, n))?;
if !has_screen(t) {
return Err(if n == job.roles.own {
format!("{n}現在沒有螢幕:AnyBot 所在的電腦沒有 Docker({})。要用別台請使用者指名。", if cfg!(target_os = "macos") { "安裝 OrbStack 或 Docker Desktop 就會有" } else { "安裝 Docker Engine 就會有" })
} else {
format!("「{n}」不能遠端操作螢幕,只能用檔案和指令工具。")
});
}
return Ok(t.clone());
}
if let Some(t) = job.computers.iter().find(|t| t.name == job.default_computer && has_screen(t)) {
return Ok(t.clone());
}
// Not named: a screen that is really there — the bot's desktop, this computer's, then one of
// the account's other computers (a brain on a server has no screen of its own).
let real = |t: &&tools::Target| match t.place {
tools::Place::Box { .. } => crate::cloudbox::available(),
tools::Place::Local { .. } => computer::screen_supported(),
_ => false,
};
job.computers
.iter()
.find(|t| matches!(t.place, tools::Place::Box { .. }) && real(t))
.or_else(|| job.computers.iter().find(|t| matches!(t.place, tools::Place::Local { .. }) && real(t)))
.or_else(|| job.computers.iter().find(|t| matches!(t.place, tools::Place::Device { .. })))
.or_else(|| job.computers.iter().find(|t| matches!(t.place, tools::Place::Box { .. })))
.or_else(|| job.computers.iter().find(|t| matches!(t.place, tools::Place::Local { .. })))
.cloned()
.ok_or_else(|| "找不到可以操作螢幕的電腦".to_string())
}
pub fn estimate_tokens(s: &str) -> u64 {
let (mut ascii, mut other) = (0u64, 0u64);
for c in s.chars() {
if c.is_ascii() { ascii += 1 } else { other += 1 }
}
ascii / 4 + other
}
fn history_tokens(system: &str, h: &[Message]) -> u64 {
estimate_tokens(system) + h.iter().map(|m| estimate_tokens(&m.text) + m.tool_calls.iter().map(|c| estimate_tokens(&c.args)).sum::<u64>() + 4).sum::<u64>()
}
/// Where to cut so the newest messages stay and no tool call is split from its result:
/// the summary covers everything before a user message.
pub fn compaction_cut(h: &[Message]) -> Option<usize> {
if h.len() <= KEEP_RECENT_MESSAGES + 1 {
return None;
}
(1..=h.len() - KEEP_RECENT_MESSAGES).rev().find(|&i| h[i].role == Role::User && !h[..i].iter().all(|m| m.summary))
}
fn flatten(h: &[Message]) -> String {
let clip = |s: &str| -> String {
let mut t: String = s.chars().take(SUMMARY_SNIPPET_CHARS).collect();
if s.chars().count() > SUMMARY_SNIPPET_CHARS {
t.push('…');
}
t
};
h.iter()
.map(|m| {
if m.summary {
return format!("【先前的摘要】\n{}", m.text);
}
match m.role {
Role::User => format!("使用者:{}", clip(&m.text)),
Role::Assistant => {
let calls: Vec<String> = m.tool_calls.iter().map(|c| format!("[使用工具 {}:{}]", c.name, tools::describe(&c.name, &c.args))).collect();
format!("助手:{} {}", clip(&m.text), calls.join(" "))
}
Role::Tool => format!("工具結果({}):{}", m.tool_name, clip(&m.text)),
}
})
.collect::<Vec<_>>()
.join("\n\n")
}
/// Fold older messages into one summary when the chat is over `percent` of the context.
/// Returns how many messages were replaced.
fn maybe_compact(job: &Job, history: &mut Vec<Message>, percent: u64, cancel: &AtomicBool, send: &dyn Fn(StreamEvent)) -> Option<usize> {
let used = history_tokens(&job.req.system, history);
if used * 100 < job.context_tokens * percent {
return None;
}
let cut = compaction_cut(history)?;
send(StreamEvent::Retrying("對話很長了,正在把較早的內容整理成摘要…".into()));
let req = ChatRequest {
system: "你負責把一段對話整理成摘要,讓之後的對話能接著進行。請用條列寫下:使用者的目標與偏好、已經做出的決定、重要的事實與數字、提到的檔案路徑與指令結果、還沒完成的事。不要加入對話裡沒有的內容。".into(),
history: vec![Message::user(&format!("請整理下面這段對話:\n\n{}", flatten(&history[..cut])))],
tools: vec![],
..job.req.clone()
};
let quiet = |ev: StreamEvent| {
if matches!(ev, StreamEvent::Alive | StreamEvent::Retrying(_)) {
send(StreamEvent::Alive);
}
};
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
job.note(crate::oplog::Record::StepAttempt { op: job.op.clone(), step: "compaction".into(), attempt: 1 });
let out = providers::run_step(&req, cancel, &quiet).ok()?;
if out.text.trim().is_empty() || cancel.load(Ordering::Relaxed) {
return None;
}
let summary = Message { role: Role::Assistant, text: out.text.trim().to_string(), summary: true, via: job.via.clone(), at: crate::model::now_ms(), ..Default::default() };
// The window shows the chat unfiltered (group chats drop some messages on the way here);
// the timestamp of the first kept message is the landmark it can find the cut by.
let boundary = history[cut].at;
history.splice(..cut, [summary.clone()]);
send(StreamEvent::Compacted(cut, summary, boundary));
Some(cut)
}
/// Wait for the person's answer; stopping the reply counts as a no.
fn wait_answer(rx: std::sync::mpsc::Receiver<ApprovalAnswer>, cancel: &AtomicBool) -> ApprovalAnswer {
loop {
if cancel.load(Ordering::Relaxed) {
return ApprovalAnswer::Reject;
}
match rx.recv_timeout(Duration::from_millis(200)) {
Ok(a) => return a,
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => continue,
Err(_) => return ApprovalAnswer::Reject,
}
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// The tool name for doing a skill's steps again.
pub const REPLAY_TOOL: &str = "replay_skill";
/// A control's kind and name from its line in a list, without its value or state, which may
/// differ the next time (「勾選框「自動更新」:1(已勾)」 → 「勾選框「自動更新」」).
fn control_key(desc: &str) -> &str {
match desc.find('」') {
Some(i) => &desc[..i + '」'.len_utf8()],
None => desc.split([':', '(']).next().unwrap_or(desc).trim(),
}
}
/// A replay step's arguments for now: each control it acted on found again by what it is, in the
/// newest lists (screen steps get today's number; browser steps find it by its words). Err: the
/// control that isn't there.
fn resolve_replay(step: &crate::memory::ReplayStep, history: &[Message]) -> Result<String, String> {
let mut v: serde_json::Value = serde_json::from_str(&step.args).unwrap_or_default();
let find = |desc: &str| -> Option<String> {
let want = control_key(desc);
history.iter().rev().filter(|m| m.role == Role::Tool).take(6).find_map(|m| {
m.text.lines().find_map(|l| {
let (num, rest) = l.trim_start().strip_prefix('[')?.split_once("] ")?;
(control_key(rest.trim()) == want && num.chars().all(|c| c.is_ascii_digit())).then(|| num.to_string())
})
})
};
let words = |desc: &str| desc.split_once('「').and_then(|(_, r)| r.rsplit_once('」')).map(|(t, _)| t.to_string()).unwrap_or_else(|| desc.to_string());
if step.tool == "computer_use_batch" {
if let Some(actions) = v["actions"].as_array_mut() {
for (i, a) in actions.iter_mut().enumerate() {
if let Some(desc) = step.targets.get(i).filter(|d| !d.is_empty()) {
a["ref"] = serde_json::Value::String(find(desc).ok_or_else(|| desc.clone())?);
}
}
}
} else if let Some(desc) = step.targets.first().filter(|d| !d.is_empty()) {
if step.tool.starts_with("browser_") {
if let Some(o) = v.as_object_mut() {
o.remove("ref");
}
v["text"] = serde_json::Value::String(words(desc));
} else {
v["ref"] = serde_json::Value::String(find(desc).ok_or_else(|| desc.clone())?);
}
}
Ok(v.to_string())
}
/// A skill being done again: what is left, which step this is, and how many there are.
struct Replay {
name: String,
left: std::collections::VecDeque<crate::memory::ReplayStep>,
total: usize,
/// The step that already had a fresh look taken for it (one look per step, then hand back).
looked: Option<usize>,
}
/// Quick steps in a row before the bot's own model looks at the work again (estimated).
pub const QUICK_RUN_MAX: usize = 4;
/// Trouble this many times in a turn and the rest of it is the bot's own model's: the quick model
/// guessing its way through an unfamiliar dialog (names that aren't there, unnamed items) only
/// made it longer each time it got the step back.
pub const QUICK_TROUBLE_MAX: usize = 2;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// Steps that only look (nothing on any computer changes): a turn of only these needs no check
/// at the end.
fn only_looks(name: &str) -> bool {
matches!(
name,
"computer_use_screenshot" | "computer_use_zoom" | "computer_use_show" | "computer_use_check_permissions" | "browser_read" | "browser_screenshot" | "browser_elements" | "browser_find" | "browser_table" | "browser_tabs" | "browser_wait" | "web_fetch" | "read_file" | "list_dir" | "search" | "read_terminal" | "list_computers" | "list_bots" | "list_routines" | "read_page" | "use_skill" | "react" | "show_card"
)
}
/// 想的跟做的分開: which model takes a step. The bot's own (thinking) model takes the first step
/// (it plans and starts), the step after anything went wrong, one in every QUICK_RUN_MAX, and the
/// end of a turn that changed something (it checks the work and answers). A quick model carries
/// out the planned steps in between: most of a turn's time was the thinking model thinking
/// before each click.
pub struct StepPlan {
quick: Option<String>,
main_next: bool,
quick_run: usize,
/// The way the last acting step went (commands, the screen, a web page).
last_route: Option<&'static str>,
/// Steps that went wrong this turn.
troubles: usize,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// Something on a computer may have changed this turn (not only looking).
pub changed: bool,
}
impl StepPlan {
pub fn new(quick: Option<String>) -> Self {
StepPlan { quick, main_next: true, quick_run: 0, last_route: None, troubles: 0, changed: false }
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
}
/// The quick model for this step, or None for the bot's own.
pub fn pick(&mut self) -> Option<String> {
if self.quick.is_none() || self.main_next || self.quick_run >= QUICK_RUN_MAX {
self.quick_run = 0;
self.main_next = false;
return None;
}
self.quick_run += 1;
self.quick.clone()
}
/// After a step: trouble (a step failed, was refused, or repeated) sends the next one back
/// to the bot's own model.
pub fn after(&mut self, trouble: bool) {
if trouble {
self.main_next = true;
self.troubles += 1;
if self.troubles >= QUICK_TROUBLE_MAX && self.quick.take().is_some() {
eprintln!("AnyBot:這一回合出了 {} 次問題,剩下的步驟改用 Bot 自己的模型", self.troubles);
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
}
}
/// The quick model answered without acting: the bot's own model looks at where things stand
/// and answers instead (or carries on). Returns whether to set that answer aside. Not only
/// after changes: a quick model also stopped half way to ask 「要我繼續嗎?」 after one look.
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
pub fn recheck_answer(&mut self, was_quick: bool) -> bool {
let recheck = was_quick;
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if recheck {
self.main_next = true;
}
recheck
}
/// The quick model couldn't be used (not on this account, an error): the rest of the turn
/// goes without it.
pub fn quick_failed(&mut self) {
self.quick = None;
}
pub fn quick_is_off(&self) -> bool {
self.quick.is_none()
}
/// The way a step's acting calls went. The quick model switching ways on its own (commands
/// to clicking the screen, say) is the bot's own model's call: true sends the next step there.
pub fn changed_route(&mut self, calls: &[crate::model::ToolCall], was_quick: bool) -> bool {
let route = calls.iter().filter(|c| !only_looks(&c.name)).map(|c| route_of(&c.name)).next();
let Some(route) = route else { return false };
let switched = was_quick && self.last_route.is_some_and(|r| r != route);
self.last_route = Some(route);
switched
}
}
/// The quick model's screen tools name their controls only: looking a number up in the list
/// is where it went wrong (1, 2, 3, then 「1」's number again for 4). Without `ref` it says which
/// button the way a person would; two of the same name come back as trouble for the bot's own
/// model.
fn quick_tools(tools: &[serde_json::Value]) -> Vec<serde_json::Value> {
tools
.iter()
.map(|t| {
let mut t = t.clone();
let screen = t["name"].as_str().is_some_and(|n| matches!(n, "computer_use_click" | "computer_use_typing" | "computer_use_scroll" | "computer_use_mouse_move" | "computer_use_batch"));
if screen {
// `get_mut`, not `t[..]`: indexing a Value mutably inserts every missing key, which
// once gave click and typing an `actions` with `properties: null` — the provider then
// refused the whole tool list and the quick model never ran.
let props = t.get_mut("parameters").and_then(|p| p.get_mut("properties")).and_then(|p| p.as_object_mut());
if let Some(p) = props {
p.remove("ref");
let items = p.get_mut("actions").and_then(|a| a.get_mut("items")).and_then(|i| i.get_mut("properties")).and_then(|p| p.as_object_mut());
if let Some(items) = items {
items.remove("ref");
}
}
if let Some(d) = t["description"].as_str() {
t["description"] = serde_json::Value::String(format!("{d} Name the control exactly as the latest list shows it (name); there are no numbers here."));
}
}
t
})
.collect()
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// Which way a tool goes: commands and files, the screen, or a web page.
fn route_of(name: &str) -> &'static str {
if computer::is_computer_tool(name) || crate::macapps::is_mac_tool(name) {
"screen"
} else if web::is_web_tool(name) {
"web"
} else {
"command"
}
}
pub fn spawn(job: Job, bot_id: u64, tx: Sender<(u64, StreamEvent)>, cancel: Arc<AtomicBool>, ctx: eframe::egui::Context) {
std::thread::spawn(move || {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let mut job = job;
let send = |ev: StreamEvent| {
let _ = tx.send((bot_id, ev));
ctx.request_repaint();
};
let mut history = job.req.history.clone();
let mut grants = job.grants.clone();
// Turned on by an "always" answer mid-turn (the window saves the setting too).
let mut tools_on = job.tools_enabled;
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
// The screen and browser steps of this turn (to stop the same step being tried over and over).
let mut tried: Vec<String> = Vec::new();
let mut local_on = job.local_execution;
let mut browsers = UsedBrowsers::new(&job.via_name, bot_id);
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// 電腦現況: what someone at each computer already knows (system, sudo, network, apps…).
let may_use_persons = job.tools_enabled || job.direct.load(Ordering::Relaxed);
job.req.system.push_str(&crate::facts::for_turn(&job.computers, may_use_persons, &job.via_name));
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// The turn's first line in the work log, and (whatever happens) its last.
job.note(crate::oplog::Record::OperationStarted {
op: job.op.clone(),
at: crate::model::now_ms(),
source_leaf: history.last().map(|m| m.at).unwrap_or(0),
prompt: history.last().filter(|m| m.role == Role::User).map(|m| crate::oplog::Prompt { at: m.at, text: m.text.clone() }),
});
let mut closing = Closing { log: job.log.clone(), op: job.op.clone(), cancel: cancel.clone(), written: false };
// Folding the chat because the model said it was too long: once per thing the person
// said (a second time means folding didn't help).
let mut overflow_folded = false;
maybe_compact(&job, &mut history, COMPACT_THRESHOLD_PERCENT, &cancel, &send);
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let mut plan = StepPlan::new(job.step_model.clone());
// A quick step's answer was set aside: its (empty) message in the chat is reused.
let mut reuse_slot = false;
// 重播: a skill's steps done again as they were, with no model in between.
let mut replay: Option<Replay> = None;
for step in 0..MAX_STEPS {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// The next replay step, its controls found again; a control that isn't there gets one
// fresh look, then the bot takes over (told where and why).
let mut scripted: Option<crate::model::ToolCall> = None;
if let Some(r) = replay.as_mut() {
if let Some(next) = r.left.front().cloned() {
let k = r.total - r.left.len() + 1;
match resolve_replay(&next, &history) {
Ok(args) => {
r.left.pop_front();
scripted = Some(crate::model::ToolCall { id: format!("replay-{k}-{}", crate::model::now_ms()), name: next.tool.clone(), args, status: ToolStatus::Running });
}
Err(_) if r.looked != Some(k) && !next.tool.starts_with("browser_") => {
r.looked = Some(k);
let computer = tools::args_of(&next.args)["computer"].as_str().map(|c| serde_json::json!({"computer": c})).unwrap_or(serde_json::json!({}));
scripted = Some(crate::model::ToolCall { id: format!("replay-look-{k}-{}", crate::model::now_ms()), name: "computer_use_screenshot".into(), args: computer.to_string(), status: ToolStatus::Running });
}
Err(missing) => {
if let Some(m) = history.last_mut().filter(|m| m.role == Role::Tool) {
m.text.push_str(&format!("\n\n(重播「{}」停在第 {k} 步:畫面上找不到「{missing}」。前面 {} 步已經照做了;接下來由你看畫面決定怎麼做完。)", r.name, k - 1));
}
replay = None;
plan.after(true);
}
}
}
}
let quick = if scripted.is_some() { None } else { plan.pick() };
let via = match (&quick, &scripted) {
(_, Some(_)) => format!("{}(重播)", job.via),
(Some(q), None) => job.via.replacen(&job.req.model, q, 1),
(None, None) => job.via.clone(),
};
if step > 0 {
maybe_compact(&job, &mut history, COMPACT_THRESHOLD_PERCENT + COMPACT_TOLERANCE_PERCENT, &cancel, &send);
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
if drain_steer(&job, &mut history, &send) {
overflow_folded = false;
}
check_in(&mut history, step);
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
if let Some(note) = refresh_own(&mut job, bot_id) {
if let Some(m) = history.last_mut().filter(|m| m.role == Role::Tool) {
m.text.push_str(&format!("\n\n{note}"));
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if !reuse_slot {
send(StreamEvent::NextStep(via.clone()));
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
reuse_slot = false;
// Quick steps don't stream their words: an answer may be set aside for the check.
let quiet = |ev: StreamEvent| {
if !matches!(ev, StreamEvent::Text(_) | StreamEvent::Thinking(_)) {
send(ev)
}
};
let tried_quick = quick.clone();
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let mut attempt = 0;
let out = loop {
// A replay step: no model.
if let Some(c) = scripted.clone() {
break providers::StepOutput { text: String::new(), thinking: String::new(), tool_calls: vec![c], blocks: vec![] };
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
attempt += 1;
job.note(crate::oplog::Record::StepAttempt { op: job.op.clone(), step: "assistant".into(), attempt });
let main_req = ChatRequest { history: history.clone(), ..job.req.clone() };
let quick_now = if plan.quick_is_off() { None } else { quick.clone() };
let got = match quick_now {
Some(q) => match providers::run_step(&ChatRequest { model: q.clone(), tools: quick_tools(&main_req.tools), ..main_req.clone() }, &cancel, &quiet) {
Ok(o) => Ok(o),
// Too long is the chat's problem, not the quick model's: folded below.
Err(e) if providers::is_context_overflow(&e) => Err(e),
Err(e) => {
eprintln!("AnyBot:快速模型 {q} 不能用({e}),這一回合改用 {}", job.req.model);
plan.quick_failed();
providers::run_step(&main_req, &cancel, &send)
}
},
None => providers::run_step(&main_req, &cancel, &send),
};
match got {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
Ok(o) => break o,
// Too long for the model: that answer is thrown away, older messages are
// folded into a summary (however full the chat looks by our estimate), and
// the same step is asked again.
Err(e) if providers::is_context_overflow(&e) && !overflow_folded && !cancel.load(Ordering::Relaxed) => {
overflow_folded = true;
if maybe_compact(&job, &mut history, 0, &cancel, &send).is_none() {
closing.close(crate::oplog::Outcome::Failed);
send(StreamEvent::Error(e));
return;
}
}
Err(e) => {
let e = if providers::is_context_overflow(&e) && overflow_folded { format!("{e}\n(已經把較早的對話整理成摘要再試過一次,還是太長。)") } else { e };
closing.close(crate::oplog::Outcome::Failed);
send(StreamEvent::Error(e));
return;
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
};
let quick = if plan.quick_is_off() { None } else { tried_quick };
let via = if quick.is_none() && scripted.is_none() { job.via.clone() } else { via };
if cancel.load(Ordering::Relaxed) {
return;
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// Steps moved back to the chosen computer: what the bot is told with each result, and
// the arguments they run with. The chat keeps the call as the model wrote it.
let mut moved: std::collections::HashMap<String, String> = std::collections::HashMap::new();
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let mut runs_with: std::collections::HashMap<String, String> = std::collections::HashMap::new();
for c in out.tool_calls.iter() {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let (args, note) = keep_to_chosen(&job, &history, &c.name, &c.args);
if let Some(n) = note {
moved.insert(c.id.clone(), n);
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
runs_with.insert(c.id.clone(), args);
}
}
let msg = Message {
role: Role::Assistant,
text: crate::i18n::reply_text(&out.text),
thinking: out.thinking,
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
via,
tool_calls: out.tool_calls,
blocks: out.blocks,
at: crate::model::now_ms(),
..Default::default()
};
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// The quick model says it's done after changing things: the bot's own model looks
// at the results and answers (the quick answer is set aside, never shown).
if msg.tool_calls.is_empty() && job.steer.lock().unwrap().is_empty() && plan.recheck_answer(quick.is_some()) {
reuse_slot = true;
continue;
}
send(StreamEvent::StepDone(msg.clone()));
let calls = msg.tool_calls.clone();
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
plan.changed |= calls.iter().any(|c| !only_looks(&c.name));
let switched = plan.changed_route(&calls, quick.is_some());
history.push(msg);
if calls.is_empty() {
// Something typed while it answered keeps the same turn going.
if job.steer.lock().unwrap().is_empty() {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
closing.close(crate::oplog::Outcome::Completed);
send(StreamEvent::Done);
browsers.close_tabs();
learn(&job, &history, &cancel, &send);
return;
}
continue;
}
// (Each call's computer was already kept to the chosen one, see keep_to_chosen.)
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// Whether any step this round failed, was refused or repeated (see StepPlan::after).
let trouble = std::cell::Cell::new(false);
// The previous call of this step, to catch the same click sent twice in a row.
let mut previous: Option<(String, String)> = None;
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
for asked in calls {
// The same click twice in one step opens a menu and closes it again (a Wi‑Fi
// menu did exactly that): the second one isn't done.
let twice = matches!(asked.name.as_str(), "computer_use_click" | "browser_click") && previous.as_ref() == Some(&(asked.name.clone(), asked.args.clone()));
previous = Some((asked.name.clone(), asked.args.clone()));
if twice {
send(StreamEvent::ToolStatus(asked.id.clone(), ToolStatus::Failed));
let result = Message::tool_result(&asked.id, &asked.name, "沒有執行:和這一步上一個點擊一模一樣。連點同一個地方兩次會把剛打開的選單又關掉;要點兩下請用 count: 2,看完結果再決定下一步。");
send(StreamEvent::ToolResult(result.clone()));
history.push(result);
continue;
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// What runs: the call as asked, on the chosen computer (see keep_to_chosen).
let call = ToolCall { args: runs_with.get(&asked.id).cloned().unwrap_or_else(|| asked.args.clone()), ..asked.clone() };
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
// 鬼打牆: the same screen or browser step again and again (a click on the same
// empty spot ten times). The third time it runs with a warning; after that it isn't
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// run at all, and the bot is told to change course. (Counted as the model asked,
// which is how the chat keeps them.)
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let mut repeat_note = String::new();
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
if let Some(key) = step_key(&asked.name, &asked.args) {
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let times = tried.iter().rev().take(10).filter(|k| **k == key).count() + 1;
tried.push(key);
if times >= 4 {
let text = format!("沒有執行:這一步你已經試了 {times} 次,畫面都沒有變成你要的樣子。不要再試同一步。改用別的方法:照截圖上的編號點、網頁改用 browser_read 拿連結直接 browser_open,或用 hand_over_computer 請使用者幫忙。");
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
trouble.set(true);
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
send(StreamEvent::ToolStatus(call.id.clone(), ToolStatus::Failed));
let result = Message::tool_result(&call.id, &call.name, &text);
send(StreamEvent::ToolResult(result.clone()));
history.push(result);
continue;
}
if times == 3 {
repeat_note = "⚠️ 這一步你已經做了 3 次。如果畫面還不是你要的,下一步一定要換方法,不要再做同一步。\n".into();
}
}
// The same look again and again with the same answer (a blank tab read eleven
// times in a row): the third time with a warning, after that not at all.
if repeat_note.is_empty() {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let same = same_answers(&history, &asked.name, &asked.args);
if same >= 3 {
let text = format!("沒有執行:同一個工具、同樣的參數你已經做了 {same} 次,每次結果都一模一樣,再做也不會不同。換一個做法(例如用 browser_open 打開要看的網址),或直接告訴使用者你看到了什麼。");
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
trouble.set(true);
send(StreamEvent::ToolStatus(call.id.clone(), ToolStatus::Failed));
let result = Message::tool_result(&call.id, &call.name, &text);
send(StreamEvent::ToolResult(result.clone()));
history.push(result);
continue;
}
if same == 2 {
repeat_note = "⚠️ 同一步你已經做了 2 次,結果都一模一樣。如果這次還一樣,就換一個做法。\n".into();
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// Every step ends the same way: its status, then its result (with the notes) and
// any picture, shown and kept.
let finish = |history: &mut Vec<Message>, output: &str, status: ToolStatus, image: Option<String>| {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// A click that changed nothing on the screen is trouble too, though it "worked".
if matches!(status, ToolStatus::Failed | ToolStatus::Denied) || !repeat_note.is_empty() || output.contains("畫面和上一步完全一樣") {
trouble.set(true);
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
job.note(crate::oplog::Record::ToolFinished { op: job.op.clone(), call_id: call.id.clone(), status: format!("{status:?}").to_lowercase() });
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
send(StreamEvent::ToolStatus(call.id.clone(), status));
let mut result = Message::tool_result(&call.id, &call.name, &format!("{repeat_note}{}", noted(&moved, &call.id, output)));
result.images.extend(image);
send(StreamEvent::ToolResult(result.clone()));
history.push(result);
};
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// About to run, every check passed: the work log hears it first, with the
// arguments it runs with and whether it may run again after a crash.
let started = || {
job.note(crate::oplog::Record::ToolStarted { op: job.op.clone(), call_id: call.id.clone(), name: call.name.clone(), args: call.args.clone(), replay: crate::oplog::replay_of(&call.name) });
send(StreamEvent::ToolStatus(call.id.clone(), ToolStatus::Running));
};
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if call.name == REPLAY_TOOL {
let v = tools::args_of(&call.args);
let name = v["name"].as_str().unwrap_or("").trim().to_string();
let skill = job.skills.iter().find(|k| k.name == name).cloned();
match skill {
Some(k) if !k.steps.is_empty() => {
// Another computer named for the replay: every step goes there.
let mut steps: std::collections::VecDeque<_> = k.steps.clone().into();
if let Some(c) = v["computer"].as_str().filter(|c| !c.trim().is_empty()) {
for st in steps.iter_mut() {
let mut a = tools::args_of(&st.args);
if a.get("computer").is_some() || computer::is_computer_tool(&st.tool) || web::is_web_tool(&st.tool) || st.tool == "shell" {
a["computer"] = serde_json::Value::String(c.to_string());
st.args = a.to_string();
}
}
}
let total = steps.len();
finish(&mut history, &format!("開始重播「{name}」,共 {total} 步:AnyBot 會一步一步照做,每一步先在畫面上找到同一個東西,對不上會停下來交給你。"), ToolStatus::Done, None);
replay = Some(Replay { name, left: steps, total, looked: None });
}
Some(_) => finish(&mut history, &format!("「{name}」沒有存可以重播的步驟:用 use_skill 讀出做法,照著自己做。"), ToolStatus::Failed, None),
None => finish(&mut history, &format!("沒有叫「{name}」的技能。"), ToolStatus::Failed, None),
}
continue;
}
if appctl::is_app_tool(&call.name) {
// App actions run on the window's side, which also asks for deletions.
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let (rtx, rrx) = channel();
send(StreamEvent::AppAction(AppCall { call_id: call.id.clone(), name: call.name.clone(), args: call.args.clone(), reply: rtx }));
let (output, ok) = loop {
if cancel.load(Ordering::Relaxed) {
return;
}
match rrx.recv_timeout(Duration::from_millis(200)) {
Ok(r) => break r,
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => continue,
Err(_) => break ("App 沒有回應".to_string(), false),
}
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, if ok { ToolStatus::Done } else { ToolStatus::Failed }, None);
continue;
}
// The person's own recording, shown back to the bot: nothing to ask.
if call.name == "view_demo_step" {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let (output, ok, img) = crate::teach::view_step(&job.demos, &call.args);
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, if ok { ToolStatus::Done } else { ToolStatus::Failed }, img);
continue;
}
// Public web pages need no computer and no permission.
// …and neither does AnyBot's own browser (like the reference assistant's own browser).
let at = web::is_web_tool(&call.name).then(|| browser_place(&job, &call.name, &call.args));
if let Some((place, on)) = at.filter(|(place, _)| call.name == "web_fetch" || bots_browser(place, &call.name, &call.args)) {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
if call.name != "web_fetch" {
let profile = tools::args_of(&call.args).get("profile").and_then(|p| p.as_str()).unwrap_or("").trim().to_string();
browsers.add(match &place {
BrowserAt::Here => BotBrowser::Here(profile),
BrowserAt::Box => BotBrowser::Box(profile),
BrowserAt::Device(host) => BotBrowser::Device(host.clone(), profile),
});
}
// 裝置連結: the AnyBot browser of another computer (its own logins).
// (On this job's own tabs: another bot may be using the same browser.)
let args = with_job(&call.args, &browsers.job);
let (output, ok, img) = match place {
BrowserAt::Device(host) => match crate::remote::run_on_device(&host, &call.name, &args, &job.via_name) {
Ok(v) => (v["text"].as_str().unwrap_or("").to_string(), v["ok"].as_bool().unwrap_or(false), v["image"].as_str().map(String::from)),
Err(e) => (e, false, None),
},
BrowserAt::Box => crate::botbrowser::in_box(|| web::run_full(&call.name, &args, &cancel)),
BrowserAt::Here => web::run_full(&call.name, &args, &cancel),
};
// Which computer's browser it was, like every other step.
let output = if on.is_empty() { output } else { format!("[{on}] {output}") };
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, if ok { ToolStatus::Done } else { ToolStatus::Failed }, img);
continue;
}
let is_web = web::is_web_tool(&call.name);
let is_screen = computer::is_computer_tool(&call.name);
let is_mac = crate::macapps::is_mac_tool(&call.name);
let what = if is_mac {
format!("{} {}", crate::macapps::label(&call.name), crate::macapps::describe(&call.name, &call.args))
} else if is_screen {
// A control by its number: name it from the app's list the bot last read.
let named = tools::args_of(&call.args)["ref"].as_str().map(str::trim).filter(|r| !r.is_empty()).and_then(|r| element_named(&history, r)).map(|e| format!("({e})")).unwrap_or_default();
format!("{} {}{named}", computer::label(&call.name), computer::describe(&call.name, &call.args))
} else if is_web {
web::describe(&call.name, &call.args)
} else {
tools::describe(&call.name, &call.args)
};
// Ask the person right away (a pop-up), and wait for the answer.
let ask_person = |kind: ApprovalKind, reason: String, grant: String| -> ApprovalAnswer {
let (atx, arx) = channel();
send(StreamEvent::ToolStatus(call.id.clone(), ToolStatus::Waiting));
send(StreamEvent::NeedApproval(Approval { call_id: call.id.clone(), tool: call.name.clone(), reason, grant, reply: atx, kind }));
wait_answer(arx, &cancel)
};
// 需要處理 (the original's box hand-off): a step only the person can do — signing
// in, a captcha, a code from their phone. A card with the computer's live screen
// appears in the chat; they take over, do it, and say 「我完成了」 or 「跳過」. A
// message typed meanwhile answers it too.
if call.name == tools::HANDOFF_TOOL {
let instruction = tools::args_of(&call.args)["instruction"].as_str().unwrap_or("").trim().to_string();
let (output, status) = match screen_target(&job, &call.args) {
Err(e) => (e, ToolStatus::Failed),
Ok(_) if instruction.is_empty() => ("請在 instruction 寫清楚要使用者做什麼。".to_string(), ToolStatus::Failed),
Ok(t) => {
let (atx, arx) = channel();
send(StreamEvent::ToolStatus(call.id.clone(), ToolStatus::Waiting));
send(StreamEvent::NeedApproval(Approval { call_id: call.id.clone(), tool: call.name.clone(), reason: instruction.clone(), grant: t.name.clone(), reply: atx, kind: ApprovalKind::Handoff }));
let answer = loop {
if cancel.load(Ordering::Relaxed) {
return;
}
match arx.recv_timeout(Duration::from_millis(200)) {
Ok(a) => break Some(a),
Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {
if !job.steer.lock().unwrap().is_empty() {
break None;
}
}
Err(_) => break Some(ApprovalAnswer::Reject),
}
};
match answer {
Some(ApprovalAnswer::Reject) => ("使用者選擇跳過這一步。不要再要求同一件事:沒有它就繼續其餘的工作,做不到的照實說。".to_string(), ToolStatus::Denied),
Some(_) => (format!("[{}] 使用者按了「我完成了」。先截圖確認畫面,再接著做。", t.name), ToolStatus::Done),
None => ("使用者沒有按按鈕,而是傳了訊息(就在下面)。照他說的做。".to_string(), ToolStatus::Done),
}
}
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, status, None);
continue;
}
// 生活卡片: drawn by the window from the call itself.
if call.name == "show_card" {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, "卡片已顯示給使用者。", ToolStatus::Done, None);
continue;
}
// 表情回應: an emoji on the person's latest message (no words needed).
if call.name == "react" {
let emoji: String = tools::args_of(&call.args)["emoji"].as_str().unwrap_or("👍").chars().take(4).collect();
send(StreamEvent::React(emoji.clone()));
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &format!("已在使用者的訊息上按了 {emoji}。"), ToolStatus::Done, None);
continue;
}
// 傳檔案: put a file the bot made into the chat as a card the person can open.
if call.name == "send_file" {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let path = tools::args_of(&call.args)["path"].as_str().unwrap_or("").to_string();
let local = match target_for(&job, &call.args) {
// Files ride to the chat only from this bot's own folder on the box.
Ok(tools::Target { place: tools::Place::Box { dir }, .. }) => match tools::box_path(std::path::Path::new(&dir), &path) {
Ok(p) => crate::cloudbox::copy_out(&p),
Err(e) => Err(e),
},
Ok(tools::Target { place: tools::Place::Sealed { root }, .. }) => Ok(tools::resolve(&root, &path)),
Ok(tools::Target { place: tools::Place::Local { workdir }, .. }) => Ok(tools::resolve(&workdir, &path)),
// Another of the person's computers: it sends the file over, then it is
// put in the chat like any other.
Ok(tools::Target { place: tools::Place::Device { host, dir }, .. }) => {
let args = serde_json::json!({"path": path, "workdir": dir}).to_string();
crate::remote::run_on_device(&host, crate::remote::FETCH_FILE, &args, &job.via_name).and_then(|v| {
let bytes = v["data"].as_str().and_then(|d| base64::engine::general_purpose::STANDARD.decode(d).ok()).ok_or_else(|| v["text"].as_str().unwrap_or("那台電腦沒有把檔案傳過來").to_string())?;
let dst = std::env::temp_dir().join(format!("anybot-fetch-{}", std::process::id())).join(v["name"].as_str().unwrap_or("檔案"));
let _ = std::fs::create_dir_all(dst.parent().unwrap_or(std::path::Path::new(".")));
std::fs::write(&dst, &bytes).map_err(|e| e.to_string())?;
Ok(dst)
})
}
Ok(_) => Err(format!("這台電腦上的檔案不能直接傳,請先複製到「{}」。", job.roles.own)),
Err(e) => Err(e),
};
// Secrets (SSH keys, the app's own data folder, .env…) never ride into the chat,
// same rule as read_file/edit_file.
let local = local.and_then(|p| if tools::is_ignored(&p) || tools::is_ignored(std::path::Path::new(&path)) { Err("這個檔案是機密的,不能傳到聊天。".to_string()) } else { Ok(p) });
let stored = local.and_then(|p| crate::attach::store(bot_id, &p));
// (A file fetched from another computer was only parked here on its way.)
let _ = std::fs::remove_dir_all(std::env::temp_dir().join(format!("anybot-fetch-{}", std::process::id())));
let (output, ok) = match stored {
Ok(a) => {
let out = format!("已把「{}」傳到聊天裡({}),使用者點一下就能預覽。", a.name, crate::attach::human(a.bytes));
send(StreamEvent::Attach(a));
(out, true)
}
Err(e) => (e, false),
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, if ok { ToolStatus::Done } else { ToolStatus::Failed }, None);
continue;
}
// 檔案預覽 / Bot 做的頁面: open what the bot made on the person's screen.
if call.name == "open_for_person" {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let v = tools::args_of(&call.args);
let path = v["path"].as_str().unwrap_or("").to_string();
let on_box = match target_for(&job, &call.args) {
Ok(t) => match t.place {
tools::Place::Box { dir } => Some(if path.starts_with('/') { path.clone() } else { format!("{dir}/{path}") }),
_ => None,
},
Err(_) => None,
};
// On a server nobody sits at its screen: a link opens in the browser of the
// computer the person is using; a file is sent into the chat instead.
let person_now = roles_now(&job).person;
let person_host = person_now.as_ref().and_then(|n| job.computers.iter().find(|t| &t.name == n)).and_then(|t| match &t.place {
tools::Place::Device { host, .. } => Some(host.clone()),
_ => None,
});
if job.roles.server {
let web = path.starts_with("http://") || path.starts_with("https://");
let (output, ok) = match (&person_host, web) {
(Some(host), true) => match crate::remote::run_on_device(host, "browser_open", &serde_json::json!({"url": path}).to_string(), &job.via_name) {
Ok(v) => (format!("[{}] {}", person_now.clone().unwrap_or_default(), v["text"].as_str().unwrap_or("")), v["ok"].as_bool().unwrap_or(false)),
Err(e) => (e, false),
},
(None, true) => ("不知道使用者現在在用哪台電腦,沒辦法打開給他看。把網址直接寫在回覆裡。".to_string(), false),
(_, false) => ("AnyBot 在伺服器上,檔案沒辦法直接在使用者的螢幕打開。改用 send_file 把檔案傳到聊天裡,使用者點一下就能看。".to_string(), false),
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, if ok { ToolStatus::Done } else { ToolStatus::Failed }, None);
continue;
}
let local = match (&on_box, target_for(&job, &call.args)) {
(Some(p), _) => crate::cloudbox::copy_out(p),
(None, Ok(tools::Target { place: tools::Place::Sealed { root }, .. })) => Ok(tools::resolve(&root, &path)),
(None, Ok(tools::Target { place: tools::Place::Local { workdir }, .. })) => Ok(tools::resolve(&workdir, &path)),
(None, _) if path.starts_with("http://") || path.starts_with("https://") => Ok(std::path::PathBuf::from(&path)),
_ => Err(format!("這台電腦上的檔案不能直接打開給使用者看,請先複製到「{}」或這台電腦。", job.roles.own)),
};
// Secrets never open on the person's screen either.
let local = local.and_then(|p| if tools::is_ignored(&p) || tools::is_ignored(std::path::Path::new(&path)) { Err("這個檔案是機密的,不能打開。".to_string()) } else { Ok(p) });
let (output, ok) = match local {
Ok(p) if p.exists() || path.starts_with("http") => match crate::preview::system_open(&p).status() {
Ok(st) if st.success() => (format!("已在使用者的螢幕上打開 {}。", p.display()), true),
_ => (format!("打不開 {}", p.display()), false),
},
Ok(p) => (format!("找不到 {}", p.display()), false),
Err(e) => (e, false),
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, if ok { ToolStatus::Done } else { ToolStatus::Failed }, None);
continue;
}
// 打電話、傳簡訊、在 X 發文: speaks for the person, so asked every single time
// (like the original's send tools; no "always").
if crate::phone::is_phone_tool(&call.name) {
let what = crate::phone::describe(&call.name, &call.args);
let answer = ask_person(ApprovalKind::Action, what, String::new());
if cancel.load(Ordering::Relaxed) {
return;
}
let (output, status) = if answer == ApprovalAnswer::Reject {
("使用者沒有允許。請不要再試,改問使用者。".to_string(), ToolStatus::Denied)
} else {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
match crate::phone::run(&call.name, &call.args) {
(t, true) => (t, ToolStatus::Done),
(t, false) => (t, ToolStatus::Failed),
}
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, status, None);
continue;
}
// 發布網站: put a page or folder the bot made on the web (asked once per site name).
if call.name == "publish_site" {
let v = tools::args_of(&call.args);
let path = v["path"].as_str().unwrap_or("").to_string();
let name = v["name"].as_str().unwrap_or("").trim().to_lowercase();
let grant = format!("publish:{name}");
let mut allowed = grants.contains(&grant);
if !allowed {
match ask_person(ApprovalKind::Action, format!("把 {path} 發布成公開網站「{name}」(任何知道網址的人都看得到)"), grant.clone()) {
ApprovalAnswer::Once => allowed = true,
ApprovalAnswer::Always(g) => {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant(&mut grants, g);
allowed = true;
}
ApprovalAnswer::Reject => {}
}
}
if cancel.load(Ordering::Relaxed) {
return;
}
let (output, status) = if !allowed {
("使用者沒有允許發布。".to_string(), ToolStatus::Denied)
} else {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let local = match target_for(&job, &call.args) {
Ok(tools::Target { place: tools::Place::Box { dir }, .. }) => crate::cloudbox::copy_out(&if path.starts_with('/') { path.clone() } else { format!("{dir}/{path}") }),
Ok(tools::Target { place: tools::Place::Sealed { root }, .. }) => Ok(tools::resolve(&root, &path)),
Ok(tools::Target { place: tools::Place::Local { workdir }, .. }) => Ok(tools::resolve(&workdir, &path)),
Ok(_) => Err(format!("這台電腦上的檔案不能直接發布,請先複製到「{}」。", job.roles.own)),
Err(e) => Err(e),
};
match local.and_then(|p| crate::remote::publish_site(&name, &p)) {
Ok(url) => (format!("已發布:{url}"), ToolStatus::Done),
Err(e) => (e, ToolStatus::Failed),
}
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, status, None);
continue;
}
// 保險庫: the bot sees only metadata; filling asks the person first. A login on a
// site it wasn't saved for warns about look-alike sites and can't be "always";
// a card is asked every time (like the original's 允許一次 / 拒絕).
if call.name == crate::vault::LIST_TOOL {
let output = crate::vault::list_for_bot(&call.args);
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, ToolStatus::Done, None);
continue;
}
if call.name == crate::vault::TOOL || call.name == crate::vault::CARD_TOOL || call.name == crate::vault::CONTACT_TOOL {
enum Plan {
Login(Box<crate::vault::LoginPlan>),
Card,
Contact,
}
let planned: Result<(Plan, String, String), String> = if call.name == crate::vault::TOOL {
crate::vault::plan_login(&call.args).map(|pl| {
let q = pl.question();
let grant = if pl.matched { format!("login:{}", pl.host) } else { String::new() };
(Plan::Login(Box::new(pl)), q, grant)
})
} else if call.name == crate::vault::CARD_TOOL {
crate::vault::plan_card(&call.args).map(|(host, c)| (Plan::Card, format!("用保險庫的卡片({} •••• {})在 {host} 填入付款資料(Bot 看不到卡號和安全碼)", c.display_title(), c.last4), String::new()))
} else {
crate::vault::plan_contact(&call.args).map(|(host, c)| (Plan::Contact, format!("把保險庫的聯絡資訊「{}」填進 {host} 的表單", c.display_title()), format!("contact:{host}")))
};
let (output, status) = match planned {
Err(e) => (e, ToolStatus::Failed),
Ok((plan, question, grant)) => {
let mut allowed = !grant.is_empty() && grants.contains(&grant);
if !allowed {
match ask_person(ApprovalKind::Action, question, grant.clone()) {
ApprovalAnswer::Once => allowed = true,
ApprovalAnswer::Always(g) => {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant(&mut grants, g);
allowed = true;
}
ApprovalAnswer::Reject => {}
}
}
if cancel.load(Ordering::Relaxed) {
return;
}
if !allowed {
("使用者沒有允許使用保險庫的這個項目。請改問使用者。".to_string(), ToolStatus::Denied)
} else {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let r = match plan {
Plan::Login(pl) => crate::vault::fill_login(&pl, tools::args_of(&call.args)["submit"].as_bool().unwrap_or(false)),
Plan::Card => crate::vault::fill_card(&call.args),
Plan::Contact => crate::vault::fill_contact(&call.args),
};
match r {
Ok(t) => (t, ToolStatus::Done),
Err(e) => (e, ToolStatus::Failed),
}
}
}
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, status, None);
continue;
}
// MCP connectors: asked once per server (or always allowed), like the original's
// MCP allow-list; direct mode doesn't ask.
if crate::mcp::is_mcp_tool(&call.name) {
let (server, tool) = crate::mcp::resolve(&call.name).unwrap_or_default();
let grant = format!("mcp:{server}");
let args_short: String = call.args.chars().take(160).collect();
let mut allowed = job.direct.load(Ordering::Relaxed) || grants.contains(&grant);
if !allowed {
match ask_person(ApprovalKind::Action, format!("使用連接器「{server}」的工具:{tool} {args_short}"), grant.clone()) {
ApprovalAnswer::Once => allowed = true,
ApprovalAnswer::Always(g) => {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant(&mut grants, g);
allowed = true;
}
ApprovalAnswer::Reject => {}
}
}
if cancel.load(Ordering::Relaxed) {
return;
}
let (output, status, img) = if allowed {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let (o, ok, img) = crate::mcp::call(&call.name, &call.args);
(o, if ok { ToolStatus::Done } else { ToolStatus::Failed }, img)
} else {
("使用者拒絕了這個操作。請不要再試同樣的操作,改問使用者想怎麼做。".to_string(), ToolStatus::Denied, None)
};
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, status, img);
continue;
}
// The person's own apps and everyday browser are on the computer they are using:
// the one they name, else the one they typed on (else this one, when it is theirs).
let person_target = || -> Result<tools::Target, String> {
if names_a_computer(&call.args) {
return target_for(&job, &call.args);
}
roles_now(&job)
.person
.and_then(|n| job.computers.iter().find(|t| t.name == n))
.cloned()
.ok_or_else(|| format!("不知道使用者現在在用哪台電腦。請問他要在哪台做(可以用:{})。", job.computers.iter().map(|t| t.name.as_str()).collect::<Vec<_>>().join("、")))
};
let target = if is_screen {
screen_target(&job, &call.args)
} else if is_mac || is_web {
// (Other browser steps went their own way above; this is the person's own Chrome.)
person_target()
} else {
target_for(&job, &call.args)
};
let on_box = matches!(target.as_ref().map(|t| &t.place), Ok(tools::Place::Box { .. }));
let on_device = match target.as_ref().map(|t| &t.place) {
Ok(tools::Place::Device { host, .. }) => Some(host.clone()),
_ => None,
};
// Permission to use computers at all, and this Mac in particular, is asked for
// when it's needed instead of the step failing quietly.
let mut gate: Result<(), String> = Ok(());
// The person just said yes to this very step in a switch's pop-up (it shows the
// step): no second pop-up for it.
let mut said_yes = false;
// 直接操作: like the reference assistant after its one-time setup ("讓 the reference assistant 在你的電腦上開啟檔案和
// 執行任務。自動審核仍會先檢查所有操作", ENHi8c), nothing is asked except what the
// safety checks and the person's "一律詢問" rules stop.
let direct = job.direct.load(Ordering::Relaxed);
if direct {
tools_on = true;
local_on = true;
}
if let Ok(t) = &target {
if !tools_on {
let reason = format!("讓 {} 使用電腦工具:{}\n(在 {} 上)", job.via_name, what, t.name);
match ask_person(ApprovalKind::EnableTools, reason, "tools:on".into()) {
ApprovalAnswer::Once => said_yes = true,
ApprovalAnswer::Always(_) => {
tools_on = true;
said_yes = true;
}
ApprovalAnswer::Reject => gate = Err("使用者沒有允許使用電腦工具。請改用文字回答,或問使用者想怎麼做。".into()),
}
}
// An admin lock (policy) refuses outright instead of asking.
if gate.is_ok() && matches!(t.place, tools::Place::Local { .. }) && job.local_locked {
gate = Err(format!("「在這台電腦上執行」設成了永不允許(或被管理員關閉),不能在「{}」上執行。請改用「{}」。", t.name, job.roles.own));
}
if gate.is_ok() && matches!(t.place, tools::Place::Local { .. }) && !local_on {
let reason = format!("在「{}」上執行任務,而不只是在它自己的電腦上:{}{what}", t.name, if is_web { format!("{} ", web::label(&call.name)) } else { String::new() });
match ask_person(ApprovalKind::EnableThisComputer, reason, "local:on".into()) {
ApprovalAnswer::Once => said_yes = true,
ApprovalAnswer::Always(_) => {
local_on = true;
said_yes = true;
}
ApprovalAnswer::Reject => gate = Err(format!("使用者沒有允許在「{}」上執行。可以改用「{}」,或問使用者。", t.name, job.roles.own)),
}
}
}
if cancel.load(Ordering::Relaxed) {
return;
}
let decision = match (&target, &gate) {
// Looking at the screen and using the mouse/keyboard each ask once (or always).
// The Mac's own apps: reading asks once per app, writing once per app (or each
// time), sending every single time (the original never keeps a grant for it).
(Ok(_), Ok(())) if is_mac => {
let app = crate::macapps::app_of(&call.name);
if crate::macapps::is_send(&call.name) && call.name == crate::macapps::SEND_MESSAGE && grants.iter().any(|g| g == "messages:no-ask") {
Decision::Allow
} else if crate::macapps::is_send(&call.name) {
Decision::Ask { reason: format!("用你的「{app}」{what}"), grant: String::new() }
} else if crate::macapps::is_write(&call.name) {
let grant = format!("macapp-write:{app}");
if grants.contains(&grant) { Decision::Allow } else { Decision::Ask { reason: format!("讓 {} 修改你的「{app}」:{what}", job.via_name), grant } }
} else {
let grant = format!("macapp:{app}");
if grants.contains(&grant) || grants.contains(&format!("macapp-write:{app}")) { Decision::Allow } else { Decision::Ask { reason: format!("讓 {} 讀取你的「{app}」:{what}", job.via_name), grant } }
}
}
// Its own desktop: nothing to ask, like the original's box.
(Ok(_), Ok(())) if is_screen && on_box => Decision::Allow,
(Ok(_), Ok(())) if is_screen && call.name == "computer_use_check_permissions" => Decision::Allow,
// 裝置連結: another of the person's computers; seeing and operating it ask once each.
(Ok(t), Ok(())) if is_screen && on_device.is_some() => {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let looks = matches!(call.name.as_str(), "computer_use_screenshot" | "computer_use_zoom" | "computer_use_show");
let grant = if looks { format!("device-see:{}", t.name) } else { format!("device-control:{}", t.name) };
if grants.contains(&grant) {
Decision::Allow
} else {
Decision::Ask { reason: format!("{}「{}」的螢幕:{what}", if looks { "看" } else { "操作" }, t.name), grant }
}
}
(Ok(_), Ok(())) if is_screen => {
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
if matches!(call.name.as_str(), "computer_use_screenshot" | "computer_use_zoom" | "computer_use_show") {
if grants.iter().any(|g| g == "computer:see") {
Decision::Allow
} else {
Decision::Ask { reason: format!("看你的螢幕畫面:{what}"), grant: "computer:see".into() }
}
} else {
// Per app, like the reference assistant: 「the reference assistant 可以看到 {app},並在其中點擊和輸入」.
let app = computer::target_app(&call.name, &call.args).unwrap_or_else(|| "目前的 App".into());
let grant = format!("app:{app}");
if grants.iter().any(|g| g == "computer:control" || *g == grant) {
Decision::Allow
} else {
Decision::Ask { reason: format!("讓 {} 看到「{app}」,並在其中點擊和輸入:{what}", job.via_name), grant }
}
}
}
// Opening a link is visible and harmless; reading a logged-in tab asks first.
(Ok(_), Ok(())) if is_web => {
if call.name == "browser_read" && web::read_needs_permission(&call.args) && !grants.iter().any(|g| g == "browser:read") {
Decision::Ask { reason: "讀取你瀏覽器目前分頁的內容(包含已登入的頁面):目前的分頁".into(), grant: "browser:read".into() }
} else {
Decision::Allow
}
}
(Ok(t), Ok(())) => tools::decide_on(&call.name, &call.args, t, &grants),
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// Can't reach that computer: reported as a failed step below, not a safety block.
(Err(_), _) => Decision::Allow,
// The person said no to the pop-up: nothing more to ask.
(_, Err(_)) => Decision::Allow,
};
// 自動審核: when the rules say yes, continue without asking (JOecNC);
// when an "always ask" rule matches, stop and ask (Pca/iV).
let mut review_note = String::new();
let direct = job.direct.load(Ordering::Relaxed);
let decision = match (decision, &job.review) {
// Direct mode: go ahead, unless an "always ask" rule matches.
(Decision::Ask { reason, grant }, review) if direct => {
// Only looking (a screenshot, a page or file read) changes nothing, so no
// "always ask" rule (deleting, paying, posting…) can be about it: no model
// call to check (it cost every look at the person's Mac 10–20 s).
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
let looks = matches!(call.name.as_str(), "computer_use_screenshot" | "computer_use_zoom" | "computer_use_show" | "browser_read" | "read_file" | "list_dir" | "search");
let verdict = match review {
Some(rv) if !rv.ask.is_empty() && !looks => review_action(&job, &Review { allow: vec![], ask: rv.ask.clone() }, &reason, &cancel),
_ => ReviewVerdict::NoMatch,
};
match verdict {
ReviewVerdict::Ask(rule) => Decision::Ask { reason: format!("{reason}(符合你的「一律詢問」規則:{rule})"), grant: String::new() },
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
// Nobody could check the person's "always ask" rules: ask, never go ahead.
ReviewVerdict::Failed => Decision::Ask { reason: format!("{reason}{REVIEW_FAILED_NOTE}"), grant: String::new() },
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// (No note on the result: the model doesn't need it, and it was on every step.)
_ => {
let _ = grant;
Decision::Allow
}
}
}
(Decision::Ask { reason, grant }, Some(rv)) => match review_action(&job, rv, &reason, &cancel) {
ReviewVerdict::Allow(rule) => {
review_note = format!("(自動審核:符合「{rule}」,直接允許)\n");
Decision::Allow
}
ReviewVerdict::Ask(rule) => Decision::Ask { reason: format!("{reason}(符合你的「一律詢問」規則:{rule})"), grant: String::new() },
ReviewVerdict::NoMatch => Decision::Ask { reason, grant },
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
ReviewVerdict::Failed => Decision::Ask { reason: format!("{reason}{REVIEW_FAILED_NOTE}"), grant },
},
(d, _) => d,
};
// Sending mail or messages is approved one by one, even in direct mode.
let decision = if is_mac && crate::macapps::is_send(&call.name) && gate.is_ok() && target.is_ok() {
review_note.clear();
Decision::Ask { reason: format!("用你的「{}」{what}(每一封都會先問你)", crate::macapps::app_of(&call.name)), grant: String::new() }
} else {
decision
};
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
// AppleScript that runs outside the sandbox can drive the person's apps: asked the
// first time even in direct mode, until they choose 「一律允許」.
let on_persons = matches!(target.as_ref().map(|t| &t.place), Ok(tools::Place::Local { .. } | tools::Place::Device { .. }));
let script = tools::args_of(&call.args)["command"].as_str().unwrap_or("").to_string();
let decision = if call.name == "shell" && on_persons && gate.is_ok() && !matches!(decision, Decision::Deny(_)) && tools::osascript_outside(&script).is_ok() && !grants.iter().any(|g| g == tools::OSASCRIPT_GRANT) {
review_note.clear();
let short: String = script.chars().take(200).collect();
Decision::Ask { reason: format!("讓 {} 在沙盒外執行 AppleScript(可以指揮你電腦上的 App):{short}", job.via_name), grant: tools::OSASCRIPT_GRANT.into() }
} else {
decision
};
let approved = match decision {
_ if gate.is_err() => Err((gate.clone().unwrap_err(), ToolStatus::Denied)),
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
_ if target.is_err() => Err((target.clone().err().unwrap_or_default(), ToolStatus::Failed)),
Decision::Allow => Ok(()),
// Sending (Messages, mail…) still asks every time, even right after a switch.
Decision::Ask { .. } if said_yes && !(is_mac && crate::macapps::is_send(&call.name)) => Ok(()),
Decision::Deny(r) => Err((format!("被安全規則擋下:{r}"), ToolStatus::Denied)),
Decision::Ask { reason, grant } => {
let machine = target.as_ref().map(|t| t.name.clone()).unwrap_or_default();
let reason = format!("{reason}\n(在 {machine} 上)");
match ask_person(ApprovalKind::Action, reason, grant) {
ApprovalAnswer::Once => Ok(()),
ApprovalAnswer::Always(g) => {
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
keep_grant(&mut grants, g);
Ok(())
}
ApprovalAnswer::Reject => Err(("使用者拒絕了這個操作。請不要再試同樣的操作,改問使用者想怎麼做。".into(), ToolStatus::Denied)),
}
}
};
if cancel.load(Ordering::Relaxed) {
return;
}
let mut image = None;
let (output, status) = match (approved, &target) {
(Ok(()), Ok(t)) => {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
started();
let ids = tools::RunIds { conversation: bot_id.to_string(), request: call.id.clone() };
let (out, ok) = if is_mac && on_device.is_none() {
crate::macapps::run(&call.name, &call.args)
} else if let (true, Some(host)) = (is_screen || is_web || is_mac, on_device.as_ref()) {
match crate::remote::run_on_device(host, &call.name, &call.args, &job.via_name) {
Ok(v) => {
image = v["image"].as_str().map(String::from);
(v["text"].as_str().unwrap_or("").to_string(), v["ok"].as_bool().unwrap_or(false))
}
Err(e) => (e, false),
}
} else if is_screen && on_box {
match crate::cloudbox::computer_run(&call.name, &tools::args_of(&call.args)) {
Ok((out, img)) => {
image = img;
(out, true)
}
Err(e) => (e, false),
}
} else if is_screen {
let (out, ok, img) = computer::run_by(&call.name, &call.args, &job.via_name);
image = img;
(out, ok)
} else if is_web {
web::run(&call.name, &call.args, &cancel)
} else {
tools::run_on(&call.name, &call.args, t, &cancel, &ids)
};
(format!("{review_note}[{}] {out}", t.name), if ok { ToolStatus::Done } else { ToolStatus::Failed })
}
(Ok(()), Err(e)) => (e.clone(), ToolStatus::Failed),
(Err(x), _) => x,
};
// 給使用者看畫面: the picture goes into the chat for the person (the bot still sees it).
if call.name == "computer_use_show" {
if let Some(bytes) = image.as_deref().and_then(|b| base64::engine::general_purpose::STANDARD.decode(b).ok()) {
// (No time in the name: a brain on a server keeps UTC, and the chat shows when.)
let name = format!("{} 的畫面.jpg", target.as_ref().map(|t| t.name.as_str()).unwrap_or("電腦"));
if let Ok(a) = crate::attach::store_bytes(bot_id, &name, &bytes) {
send(StreamEvent::Attach(a));
}
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
finish(&mut history, &output, status, image);
if cancel.load(Ordering::Relaxed) {
return;
}
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
plan.after(trouble.get() || switched);
if scripted.is_some() {
if let Some(r) = replay.as_ref() {
let k = r.total - r.left.len();
let note = if trouble.get() {
Some(format!("\n\n(重播「{}」在第 {k} 步出了問題,後面的步驟沒有做。接下來由你判斷怎麼做完。)", r.name))
} else if r.left.is_empty() {
Some(format!("\n\n(重播「{}」做完了,共 {} 步。看一下結果,確認真的做到了再回答使用者。)", r.name, r.total))
} else {
None
};
if let Some(n) = note {
if let Some(m) = history.last_mut().filter(|m| m.role == Role::Tool) {
m.text.push_str(&n);
}
replay = None;
plan.after(true);
}
}
}
}
send(StreamEvent::NextStep(job.via.clone()));
send(StreamEvent::StepDone(Message::assistant(&format!("(這次已經用了 {MAX_STEPS} 個步驟,先停在這裡。要繼續的話,回覆「繼續」。)"), &job.via)));
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
closing.close(crate::oplog::Outcome::Completed);
send(StreamEvent::Done);
});
}
#[cfg(test)]
mod tests {
use super::*;
use crate::model::ToolCall;
use crate::providers::ProviderKind;
/// A job with nothing in it, for tests that only need a few fields.
fn test_job() -> Job {
Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: String::new(), api_key: String::new(), model: String::new(), system: String::new(), history: vec![], tools: vec![], session: String::new() },
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model: None,
computers: vec![],
default_computer: String::new(),
roles: tools::Roles::default(),
local_execution: false,
grants: vec![],
context_tokens: DEFAULT_CONTEXT_TOKENS,
via: String::new(),
via_name: String::new(),
tools_enabled: true,
direct: Arc::new(AtomicBool::new(false)),
local_locked: false,
steer: Arc::new(Mutex::new(vec![])),
review: None,
demos: vec![],
skills: vec![],
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op: String::new(),
log: None,
}
}
#[test]
fn computer_words_need_word_boundaries() {
// "mac" as its own word names this computer; inside longer words (macOS, a device
// named Ann inside "anniversary") it must not — that is what made the guard fire on
// text the person never aimed at any computer.
assert!(finds("用我的 mac 執行", "mac"));
assert!(finds("mac 上開檔案", "mac"));
assert!(!finds("在 macOS 上開檔案", "mac"));
assert!(!finds("telegram 傳給我", "mac"));
assert!(finds("我的電腦好慢", "我的電腦"));
assert!(!finds("你的虛擬電腦很吵", "bot 的電腦"));
assert!(finds("在 bot 的電腦跑", "bot 的電腦"));
let roles = tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true };
let mac = tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } };
assert!(mentions("幫我在 Mac 截圖", &mac, &roles));
assert!(!mentions("在 macOS 的登入機制下,幫 Bot 的電腦截圖", &mac, &roles));
}
#[test]
fn the_same_answer_again_and_again_is_counted() {
use crate::model::ToolCall;
let call = |id: &str, args: &str| Message { role: Role::Assistant, tool_calls: vec![ToolCall { id: id.into(), name: "browser_read".into(), args: args.into(), status: ToolStatus::Done }], ..Default::default() };
let mut h = vec![Message::user("看得到我的 Threads 嗎?")];
for i in 0..3 {
h.push(call(&format!("c{i}"), r#"{"computer":"Bot 的電腦"}"#));
let warn = if i == 2 { "⚠️ 同一步你已經做了 2 次,結果都一模一樣。\n" } else { "" };
h.push(Message::tool_result(&format!("c{i}"), "browser_read", &format!("{warn}網址:about:blank")));
}
assert_eq!(same_answers(&h, "browser_read", r#"{ "computer": "Bot 的電腦" }"#), 3);
// Another page in between: counting starts again.
h.push(call("c9", r#"{"computer":"Bot 的電腦"}"#));
h.push(Message::tool_result("c9", "browser_read", "網址:https://www.threads.com/"));
assert_eq!(same_answers(&h, "browser_read", r#"{"computer":"Bot 的電腦"}"#), 1);
// Waiting on a terminal is allowed to look the same.
assert_eq!(same_answers(&h, "read_terminal", "{}"), 0);
}
#[test]
fn long_turns_check_in_instead_of_stopping() {
let mut h = vec![Message::user("修好它"), Message::tool_result("t1", "shell", "ok")];
check_in(&mut h, CHECK_IN_STEPS - 1);
assert_eq!(h[1].text, "ok");
check_in(&mut h, CHECK_IN_STEPS);
assert!(h[1].text.starts_with("ok") && h[1].text.contains(&format!("已經做了 {CHECK_IN_STEPS} 步")));
// Only a tool result carries it (a steer the person typed is left as they wrote it).
let mut h = vec![Message::user("改用別台")];
check_in(&mut h, CHECK_IN_STEPS * 2);
assert_eq!(h[0].text, "改用別台");
}
#[test]
fn the_boxs_browser_is_always_the_bots_own() {
// A cloud brain has no Chrome of its own; browser steps for the box still go to the box.
for args in [r#"{}"#, r#"{"url":"https://www.threads.com"}"#, r#"{"profile":"你的瀏覽器"}"#] {
assert!(bots_browser(&BrowserAt::Box, "browser_open", args));
assert!(bots_browser(&BrowserAt::Box, "browser_read", args));
assert!(crate::botbrowser::in_box(|| !web::uses_person_browser(args)));
}
}
/// A made-up model on a local port that answers each request with the next reply: a tool
/// call (name, JSON arguments) or, for an empty name, words.
fn fake_model(replies: Vec<(&'static str, serde_json::Value)>) -> u16 {
use std::io::{Read, Write};
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
std::thread::spawn(move || {
for (i, (name, v)) in replies.into_iter().enumerate() {
let delta = if name.is_empty() {
serde_json::json!({"content": v.as_str().unwrap_or("")})
} else {
serde_json::json!({"tool_calls": [{"index": 0, "id": format!("c{i}"), "function": {"name": name, "arguments": v.to_string()}}]})
};
let body = format!("data: {}\n\ndata: [DONE]\n\n", serde_json::json!({"choices": [{"delta": delta}]}));
let (mut s, _) = l.accept().unwrap();
let mut req = Vec::new();
let mut buf = [0u8; 8192];
loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break;
}
}
}
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(body.as_bytes());
}
});
port
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
/// A model server answering `replies` (each a stream delta) in order, recording the model
/// each request asked for.
fn recording_model(replies: Vec<serde_json::Value>) -> (u16, Arc<Mutex<Vec<String>>>) {
use std::io::{Read, Write};
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
let asked = Arc::new(Mutex::new(Vec::<String>::new()));
let seen = asked.clone();
std::thread::spawn(move || {
for delta in replies {
let Ok((mut s, _)) = l.accept() else { return };
let mut req = Vec::new();
let mut buf = [0u8; 16384];
let body = loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break String::new();
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break t[end + 4..].to_string();
}
}
};
let v: serde_json::Value = serde_json::from_str(&body).unwrap_or_default();
seen.lock().unwrap().push(v["model"].as_str().unwrap_or("").to_string());
let out = format!("data: {}\n\ndata: [DONE]\n\n", serde_json::json!({"choices": [{"delta": delta}]}));
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(out.as_bytes());
}
});
(port, asked)
}
fn tool_delta(id: &str, name: &str, args: serde_json::Value) -> serde_json::Value {
serde_json::json!({"tool_calls": [{"index": 0, "id": id, "function": {"name": name, "arguments": args.to_string()}}]})
}
/// The same click twice in one step: the second isn't done (it closed a menu the first had
/// just opened). The clicks name a computer that isn't there, so nothing is clicked here.
#[test]
fn the_quick_models_screen_tools_name_controls_only() {
let specs = crate::computer::specs_for(&["C".to_string()], "C", true);
let quick = quick_tools(&specs);
for t in &quick {
let n = t["name"].as_str().unwrap();
if matches!(n, "computer_use_click" | "computer_use_typing" | "computer_use_scroll" | "computer_use_mouse_move") {
assert!(t["parameters"]["properties"].get("ref").is_none() && t["parameters"]["properties"].get("name").is_some(), "{n}");
}
if n == "computer_use_batch" {
assert!(t["parameters"]["properties"]["actions"]["items"]["properties"].get("ref").is_none());
}
}
// Nothing else changes shape: only `ref` is gone (a stray key here makes the provider refuse
// the whole request).
for (a, b) in specs.iter().zip(&quick) {
let keys = |t: &serde_json::Value| t["parameters"]["properties"].as_object().map(|p| p.keys().filter(|k| *k != "ref").cloned().collect::<Vec<_>>());
assert_eq!(keys(a), keys(b), "{}", a["name"]);
}
assert!(!serde_json::to_string(&quick).unwrap().contains(":null"), "a key with no value");
// The bot's own model keeps both.
assert!(specs.iter().any(|t| t["name"] == "computer_use_click" && t["parameters"]["properties"].get("ref").is_some()));
}
#[test]
fn the_same_click_twice_in_a_step_is_done_once() {
let click = serde_json::json!({"name": "computer_use_click", "arguments": serde_json::json!({"computer": "沒有這台", "name": "Wi‑Fi"}).to_string()});
let two = serde_json::json!({"tool_calls": [
{"index": 0, "id": "a", "function": click.clone()},
{"index": 1, "id": "b", "function": click},
]});
let (port, _) = recording_model(vec![two, serde_json::json!({"content": "好了"})]);
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("打開 Wi‑Fi 選單")], tools: tools::specs(), session: String::new() },
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut results = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(20)).expect("loop stalled") {
(_, StreamEvent::ToolResult(m)) => results.push(m.text),
(_, StreamEvent::NeedApproval(a)) => {
let _ = a.reply.send(ApprovalAnswer::Reject);
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert_eq!(results.len(), 2, "{results:?}");
assert!(!results[0].contains("一模一樣"), "{results:?}");
assert!(results[1].contains("和這一步上一個點擊一模一樣"), "{results:?}");
}
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
#[test]
fn a_control_is_found_again_by_what_it_is() {
assert_eq!(control_key("勾選框「自動更新」:1(已勾)"), "勾選框「自動更新」");
assert_eq!(control_key("按鈕「Close」"), "按鈕「Close」");
let list = Message::tool_result("x", "computer_use_screenshot", "螢幕截圖。\n[3] 按鈕「Close」\n[9] 勾選框「自動更新」:0");
let step = crate::memory::ReplayStep { tool: "computer_use_click".into(), args: r#"{"ref":"5","computer":"C"}"#.into(), targets: vec!["勾選框「自動更新」:1(已勾)".into()] };
let args = resolve_replay(&step, std::slice::from_ref(&list)).unwrap();
assert_eq!(tools::args_of(&args)["ref"], "9");
assert_eq!(tools::args_of(&args)["computer"], "C");
// A browser step finds its control by its words.
let web = crate::memory::ReplayStep { tool: "browser_click".into(), args: r#"{"ref":"4"}"#.into(), targets: vec!["按鈕「開始」".into()] };
let a = tools::args_of(&resolve_replay(&web, &[]).unwrap());
assert!(a.get("ref").is_none() && a["text"] == "開始");
// Not on the screen: says which.
let gone = crate::memory::ReplayStep { tool: "computer_use_click".into(), args: r#"{"ref":"5"}"#.into(), targets: vec!["按鈕「Save」".into()] };
assert_eq!(resolve_replay(&gone, std::slice::from_ref(&list)).unwrap_err(), "按鈕「Save」");
}
/// A skill with saved steps is done again by AnyBot: no model between the steps, then the
/// bot's own model checks the result.
#[test]
fn a_skill_is_replayed_without_the_model_in_between() {
let (port, asked) = recording_model(vec![tool_delta("r", REPLAY_TOOL, serde_json::json!({"name": "兩步"})), serde_json::json!({"content": "確認做完了"})]);
let w = std::env::temp_dir().join(format!("anybot-replay-{}", std::process::id()));
std::fs::create_dir_all(&w).unwrap();
let skill = crate::memory::Skill {
name: "兩步".into(),
description: "做兩件事".into(),
instructions: "…".into(),
steps: vec![
crate::memory::ReplayStep { tool: "shell".into(), args: r#"{"command":"echo first > r1.txt"}"#.into(), targets: vec![] },
crate::memory::ReplayStep { tool: "shell".into(), args: r#"{"command":"echo second > r2.txt"}"#.into(), targets: vec![] },
],
};
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "main".into(), system: String::new(), history: vec![Message::user("照上次那樣做")], tools: tools::specs(), session: String::new() },
step_model: Some("quick".into()),
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w.clone() } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: true,
tools_enabled: true,
direct: Arc::new(AtomicBool::new(true)),
skills: vec![skill],
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut results = vec![];
let mut last = String::new();
loop {
match rx.recv_timeout(Duration::from_secs(20)).expect("loop stalled") {
(_, StreamEvent::ToolResult(m)) => results.push(m.text),
(_, StreamEvent::StepDone(m)) => last = m.text,
(_, StreamEvent::NeedApproval(a)) => {
let _ = a.reply.send(ApprovalAnswer::Once);
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
// Two model calls: the one that chose to replay, and the bot's own check at the end.
assert_eq!(*asked.lock().unwrap(), vec!["main", "main"]);
assert_eq!(std::fs::read_to_string(w.join("r1.txt")).unwrap().trim(), "first");
assert_eq!(std::fs::read_to_string(w.join("r2.txt")).unwrap().trim(), "second");
assert!(results[0].contains("開始重播「兩步」,共 2 步"), "{results:?}");
// (「重播做完了」 is added for the model only, like the check-in line: not in the chat.)
assert_eq!(results.len(), 3, "{results:?}");
assert_eq!(last, "確認做完了");
}
#[test]
fn the_quick_model_carries_out_steps_between_the_bots_own() {
let mut p = StepPlan::new(Some("q".into()));
assert_eq!(p.pick(), None, "the first step plans: the bot's own model");
assert_eq!(p.pick().as_deref(), Some("q"));
p.after(true);
assert_eq!(p.pick(), None, "after trouble: the bot's own model");
// Trouble twice: the rest of the turn is the bot's own model's.
let mut twice = StepPlan::new(Some("q".into()));
twice.pick();
twice.after(true);
twice.pick();
twice.after(true);
assert!(twice.quick_is_off() && twice.pick().is_none() && twice.pick().is_none());
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
for _ in 0..QUICK_RUN_MAX {
assert_eq!(p.pick().as_deref(), Some("q"));
}
assert_eq!(p.pick(), None, "a look from the bot's own model every QUICK_RUN_MAX steps");
assert!(p.recheck_answer(true) && p.pick().is_none(), "the quick model's answer goes to the bot's own model");
assert!(!p.recheck_answer(false), "the bot's own answer stands");
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
p.quick_failed();
assert_eq!(p.pick(), None);
assert_eq!(StepPlan::new(None).pick(), None);
// The quick model leaving commands for the screen on its own: the bot's own model decides.
let tc = |n: &str| crate::model::ToolCall { id: "x".into(), name: n.into(), args: "{}".into(), status: ToolStatus::Done };
let mut r = StepPlan::new(Some("q".into()));
assert!(!r.changed_route(&[tc("shell")], false));
assert!(!r.changed_route(&[tc("computer_use_screenshot")], true), "looking isn't switching");
assert!(!r.changed_route(&[tc("shell")], true));
assert!(r.changed_route(&[tc("computer_use_click")], true));
assert!(!r.changed_route(&[tc("browser_click")], false), "the bot's own model may switch");
assert_eq!(ProviderKind::Xai.quick_model("grok-4.7"), Some("grok-4.20-0309-non-reasoning"));
assert_eq!(ProviderKind::Xai.quick_model("grok-4.20-0309-non-reasoning"), None);
assert_eq!(ProviderKind::Ollama.quick_model("llama3.2"), None);
}
/// End to end with a model server that records which model each request asked for: plan with
/// the bot's model, act with the quick one, back to the bot's after a failed step, and the
/// quick model's 「done」 after changing things replaced by the bot's own check.
#[test]
fn a_turn_switches_models_as_planned() {
use std::io::{Read, Write};
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
let asked = Arc::new(Mutex::new(Vec::<String>::new()));
let seen = asked.clone();
std::thread::spawn(move || {
let tool = |id: &str, cmd: &str| serde_json::json!({"tool_calls": [{"index": 0, "id": id, "function": {"name": "shell", "arguments": serde_json::json!({"command": cmd}).to_string()}}]});
let replies = [
tool("a", "echo plan-and-start"),
tool("b", "exit 3"),
tool("c", "echo fixed"),
serde_json::json!({"content": "做好了(快速模型)"}),
serde_json::json!({"content": "確認做好了"}),
];
for delta in replies {
let (mut s, _) = l.accept().unwrap();
let mut req = Vec::new();
let mut buf = [0u8; 16384];
let body = loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break String::new();
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break t[end + 4..].to_string();
}
}
};
let v: serde_json::Value = serde_json::from_str(&body).unwrap_or_default();
seen.lock().unwrap().push(v["model"].as_str().unwrap_or("").to_string());
let out = format!("data: {}\n\ndata: [DONE]\n\n", serde_json::json!({"choices": [{"delta": delta}]}));
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(out.as_bytes());
}
});
let w = std::env::temp_dir().join(format!("anybot-steps-{}", std::process::id()));
std::fs::create_dir_all(&w).unwrap();
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "main".into(), system: String::new(), history: vec![Message::user("做一件事")], tools: tools::specs(), session: String::new() },
step_model: Some("quick".into()),
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w.clone() } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: true,
tools_enabled: true,
direct: Arc::new(AtomicBool::new(true)),
via: "Custom · main".into(),
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut shown = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(20)).expect("loop stalled") {
(_, StreamEvent::StepDone(m)) => shown.push((m.via.clone(), m.text.clone())),
(_, StreamEvent::NeedApproval(a)) => {
let _ = a.reply.send(ApprovalAnswer::Once);
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert_eq!(*asked.lock().unwrap(), vec!["main", "quick", "main", "quick", "main"]);
assert_eq!(shown.iter().map(|(v, _)| v.as_str()).collect::<Vec<_>>(), vec!["Custom · main", "Custom · quick", "Custom · main", "Custom · main"]);
assert_eq!(shown.last().unwrap().1, "確認做好了");
assert!(!shown.iter().any(|(_, t)| t.contains("快速模型")), "the set-aside answer was never shown: {shown:?}");
}
/// AppleScript run outside the sandbox drives the person's apps: asked the first time even
/// in direct mode, and not again once they chose 「一律允許」.
#[test]
fn applescript_outside_the_sandbox_is_asked_once_even_in_direct_mode() {
for (granted, expect_ask) in [(false, true), (true, false)] {
let port = fake_model(vec![
("shell", serde_json::json!({"command": "osascript -e 'return 1+1'", "computer": crate::model::THIS_COMPUTER_NAME})),
("", serde_json::json!("好了")),
]);
let w = std::env::temp_dir().join(format!("anybot-osa-{}-{granted}", std::process::id()));
std::fs::create_dir_all(&w).unwrap();
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("算一下")], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w.clone() } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: true,
tools_enabled: true,
direct: Arc::new(AtomicBool::new(true)),
grants: if granted { vec![tools::OSASCRIPT_GRANT.into()] } else { vec![] },
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut asked = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(20)).expect("loop stalled") {
(_, StreamEvent::NeedApproval(a)) => {
asked.push(a.grant.clone());
let _ = a.reply.send(ApprovalAnswer::Reject);
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
if expect_ask {
assert_eq!(asked, vec![tools::OSASCRIPT_GRANT.to_string()]);
} else {
assert!(asked.is_empty(), "{asked:?}");
}
}
}
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
/// A reply with nothing in it (a dropped stream) is asked again, not taken as the answer.
#[test]
fn an_empty_reply_is_asked_again() {
let port = fake_model(vec![("", serde_json::json!("")), ("", serde_json::json!("好了"))]);
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("在嗎")], tools: vec![], session: String::new() },
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut texts = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(20)).expect("loop stalled") {
(_, StreamEvent::StepDone(m)) => texts.push(m.text),
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert_eq!(texts, vec!["好了".to_string()]);
}
/// A brain with no Chrome of its own — a cloud brain — browses on its own box: the bot's
/// browser_open and browser_read for its computer reach the box's Chromium, not the
/// person's browser on the server. Run where no Chrome-family browser is installed, against
/// a box: `ANYBOT_DATA_DIR=<scratch> cargo test --release -- --ignored
/// a_brain_without_chrome_browses_on_its_box --nocapture`.
#[test]
#[ignore]
fn a_brain_without_chrome_browses_on_its_box() {
let own = crate::model::BOT_COMPUTER_NAME;
let page = "mkdir -p /tmp/anybot-page && printf '<meta charset=utf-8><title>box page</title><h1>盒子裡的測試頁 42</h1>' > /tmp/anybot-page/index.html && (cd /tmp/anybot-page && setsid python3 -m http.server 8766 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served";
let port = fake_model(vec![
("shell", serde_json::json!({"command": page})),
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8766/"})),
("browser_read", serde_json::json!({})),
("", serde_json::json!("看到了")),
]);
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("打開盒子裡的測試頁")], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: own.into(), place: tools::Place::Box { dir: crate::cloudbox::bot_dir(1) } }],
default_computer: own.into(),
roles: tools::Roles { own: own.into(), person: None, server: true },
local_execution: false,
direct: Arc::new(AtomicBool::new(true)),
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut results = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(120)).expect("loop stalled") {
(_, StreamEvent::ToolResult(m)) => {
println!("{}: {}", m.tool_name, m.text);
results.push(m.text)
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
println!("this computer has a Chrome-family browser: {}", crate::botbrowser::engine().is_some());
assert!(results[0].contains("served"), "{results:?}");
assert!(results[1].starts_with(&format!("[{own}]")) && results[1].contains("它桌面上的 Chromium"), "{}", results[1]);
assert!(results[2].contains("盒子裡的測試頁 42"), "{}", results[2]);
}
/// The box's browser works with the box's own disk (the brain runs elsewhere): an upload
/// finds the file there, a download and a PDF land there, and old tabs close. Needs Docker
/// and the box: `cargo test the_box_browser_uses_the_box_disk -- --ignored --nocapture`.
#[test]
#[ignore]
fn the_box_browser_uses_the_box_disk() {
let own = crate::model::BOT_COMPUTER_NAME;
let setup = r#"mkdir -p /tmp/anybot-files /tmp/anybot-up && cd /tmp/anybot-files && cat > index.html <<'EOF'
<meta charset=utf-8><title>files</title><h1>檔案測試</h1>
<input type=file id=f onchange="document.getElementById('o').textContent='已選:'+this.files[0].name+' '+this.files[0].size">
<p id=o>還沒選</p><a href="report.csv" download>下載報表</a>
EOF
printf 'a,b\n1,2\n' > report.csv && printf 'hello box' > '/tmp/anybot-up/上傳測試.txt' && (setsid python3 -m http.server 8768 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served"#;
let check = "ls /home/bot/Downloads/report.csv /home/bot/Downloads/files-*.pdf && head -c 4 /home/bot/Downloads/files-*.pdf && echo";
let pages = r#"curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys; print("pages:", len([t for t in json.load(sys.stdin) if t["type"] == "page"]))'"#;
let clean = r#"pkill -f 'http[.]server 8768'; rm -rf /tmp/anybot-files /tmp/anybot-up /home/bot/Downloads/report.csv /home/bot/Downloads/files-*.pdf; curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys,urllib.request; [urllib.request.urlopen("http://127.0.0.1:9222/json/close/" + t["id"]) for t in json.load(sys.stdin) if ":8768" in t["url"]]'; echo cleaned"#;
let mut replies: Vec<(&'static str, serde_json::Value)> = vec![
("shell", serde_json::json!({"command": setup})),
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8768/"})),
("browser_upload", serde_json::json!({"path": "/tmp/anybot-up/上傳測試.txt"})),
("browser_read", serde_json::json!({})),
("browser_upload", serde_json::json!({"path": "/tmp/anybot-up/沒有這個.txt"})),
("browser_download", serde_json::json!({"text": "下載報表"})),
("browser_pdf", serde_json::json!({})),
("shell", serde_json::json!({"command": check})),
];
for n in 1..=8 {
replies.push(("browser_open", serde_json::json!({"url": format!("http://127.0.0.1:8768/?n={n}")})));
}
replies.push(("shell", serde_json::json!({"command": pages})));
replies.push(("shell", serde_json::json!({"command": clean})));
replies.push(("", serde_json::json!("好了")));
let port = fake_model(replies);
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("測試盒子裡的檔案")], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: own.into(), place: tools::Place::Box { dir: crate::cloudbox::bot_dir(1) } }],
default_computer: own.into(),
roles: tools::Roles { own: own.into(), person: None, server: true },
local_execution: false,
direct: Arc::new(AtomicBool::new(true)),
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut results = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(180)).expect("loop stalled") {
(_, StreamEvent::ToolResult(m)) => {
println!("{}: {}\n", m.tool_name, m.text.chars().take(300).collect::<String>());
results.push(m.text)
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert!(results[0].contains("served"), "{}", results[0]);
assert!(results[1].contains("檔案測試"), "{}", results[1]);
assert!(results[2].contains("已把 /tmp/anybot-up/上傳測試.txt 放進上傳欄位,欄位裡現在是:上傳測試.txt(9 位元組)") && results[2].contains("已選:上傳測試.txt 9"), "{}", results[2]);
assert!(results[3].contains("已選:上傳測試.txt 9"), "{}", results[3]);
assert!(results[4].contains("這台電腦上沒有「/tmp/anybot-up/沒有這個.txt」"), "{}", results[4]);
assert!(results[5].contains("下載好了:/home/bot/Downloads/report.csv"), "{}", results[5]);
assert!(results[6].contains("已存成 PDF:/home/bot/Downloads/files-"), "{}", results[6]);
assert!(results[7].contains("%PDF") && !results[7].contains("No such file"), "{}", results[7]);
let count = results[16].split("pages:").nth(1).and_then(|n| n.trim().parse::<usize>().ok()).unwrap_or(99);
assert!(count <= 6, "{}", results[16]);
assert!(results[17].contains("cleaned"), "{}", results[17]);
}
/// When a job ends, the tabs it left open in the box's browser close; the page it ended on
/// stays. Needs the box: `cargo test a_finished_job_closes_its_tabs -- --ignored --nocapture`.
#[test]
#[ignore]
fn a_finished_job_closes_its_tabs() {
let own = crate::model::BOT_COMPUTER_NAME;
let setup = "mkdir -p /tmp/anybot-tabs && cd /tmp/anybot-tabs && echo '<meta charset=utf-8><title>tab</title>分頁' > index.html && (setsid python3 -m http.server 8770 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served";
let port = fake_model(vec![
("shell", serde_json::json!({"command": setup})),
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8770/?n=1"})),
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8770/?n=2"})),
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8770/?n=3"})),
("", serde_json::json!("好了")),
]);
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("開三頁")], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: own.into(), place: tools::Place::Box { dir: crate::cloudbox::bot_dir(1) } }],
default_computer: own.into(),
roles: tools::Roles { own: own.into(), person: None, server: true },
local_execution: false,
direct: Arc::new(AtomicBool::new(true)),
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
loop {
match rx.recv_timeout(Duration::from_secs(120)).expect("loop stalled") {
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
std::thread::sleep(Duration::from_secs(2));
let box_sh = |script: &str| String::from_utf8_lossy(&std::process::Command::new("docker").args(["exec", crate::cloudbox::container(), "sh", "-c", script]).output().unwrap().stdout).to_string();
let left = box_sh(r#"curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys; print("\n".join(t["url"] for t in json.load(sys.stdin) if t["type"] == "page"))'"#);
println!("left open:\n{left}");
box_sh(r#"pkill -f 'http[.]server 8770'; rm -rf /tmp/anybot-tabs"#);
assert_eq!(left.trim(), "http://127.0.0.1:8770/?n=3", "{left}");
}
/// Two bots' jobs at once on one browser (the box's): each reads its own page although the
/// other's is in front, and neither closes the other's tab when it ends; a popup a click
/// opens is the clicking job's. Needs the box:
/// `cargo test two_jobs_keep_to_their_own_tabs -- --ignored --nocapture`.
#[test]
#[ignore]
fn two_jobs_keep_to_their_own_tabs() {
let own = crate::model::BOT_COMPUTER_NAME;
let box_sh = |script: &str| String::from_utf8_lossy(&std::process::Command::new("docker").args(["exec", crate::cloudbox::container(), "sh", "-c", script]).output().unwrap().stdout).to_string();
let urls = r#"curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys; print(" ".join(t["url"] for t in json.load(sys.stdin) if t["type"] == "page"))'"#;
let clean = r#"pkill -f 'http[.]server 8774'; rm -rf /tmp/anybot-two /home/bot/bot-991 /home/bot/bot-992 /home/bot/bot-993; curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys,urllib.request; [urllib.request.urlopen("http://127.0.0.1:9222/json/close/" + t["id"]) for t in json.load(sys.stdin) if ":8774" in t["url"]]' 2>/dev/null; true"#;
// (What an earlier run that failed half-way left behind.)
box_sh(clean);
let setup = r#"mkdir -p /tmp/anybot-two && cd /tmp/anybot-two && echo '<meta charset=utf-8><title>A</title>頁面甲 <a href="b.html" target="_blank">開新視窗</a>' > a.html && echo '<meta charset=utf-8><title>B</title>頁面乙' > b.html && (setsid python3 -m http.server 8774 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served"#;
let run = |replies: Vec<(&'static str, serde_json::Value)>, bot: u64| {
let port = fake_model(replies);
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("開頁面")], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: own.into(), place: tools::Place::Box { dir: crate::cloudbox::bot_dir(bot) } }],
default_computer: own.into(),
roles: tools::Roles { own: own.into(), person: None, server: true },
local_execution: false,
direct: Arc::new(AtomicBool::new(true)),
..test_job()
};
let (tx, rx) = channel();
spawn(job, bot, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
std::thread::spawn(move || {
let mut out = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(120)).expect("loop stalled") {
(_, StreamEvent::ToolResult(m)) => out.push(format!("{}: {}", m.tool_name, m.text)),
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
out
})
};
let a = run(vec![
("shell", serde_json::json!({"command": setup})),
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8774/a.html"})),
("shell", serde_json::json!({"command": "sleep 4; echo waited"})),
("browser_read", serde_json::json!({})),
("", serde_json::json!("好了")),
], 991);
std::thread::sleep(Duration::from_millis(2500));
let b = run(vec![
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8774/b.html"})),
("browser_read", serde_json::json!({})),
("shell", serde_json::json!({"command": "sleep 4; echo waited"})),
("", serde_json::json!("好了")),
], 992);
let (a, b) = (a.join().unwrap(), b.join().unwrap());
for l in a.iter().chain(b.iter()) {
println!("{}\n", l.chars().take(160).collect::<String>());
}
assert!(a[3].starts_with("browser_read") && a[3].contains("頁面甲") && !a[3].contains("頁面乙"), "A read another job's page: {}", a[3]);
assert!(b[1].starts_with("browser_read") && b[1].contains("頁面乙"), "{}", b[1]);
let left = box_sh(urls);
println!("left after both: {left}");
assert!(left.contains("8774/a.html") && left.contains("8774/b.html"), "a job closed the other's tab: {left}");
// A popup a click opens belongs to the job that clicked: it reads it, then keeps only it.
let c = run(vec![
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8774/a.html?c"})),
("browser_click", serde_json::json!({"text": "開新視窗"})),
("browser_read", serde_json::json!({})),
("", serde_json::json!("好了")),
], 993)
.join()
.unwrap();
for l in &c {
println!("{}\n", l.chars().take(160).collect::<String>());
}
assert!(c[2].starts_with("browser_read") && c[2].contains("頁面乙"), "{}", c[2]);
// (The tabs close right after the job's Done.)
std::thread::sleep(Duration::from_secs(2));
let left = box_sh(urls);
println!("left after the popup job: {left}");
assert!(!left.contains("a.html?c"), "the clicking job's first tab stayed: {left}");
box_sh(clean);
}
/// Words reach the control they name on a form laid out like httpbin's (each label and its
/// input inside a <p>): typing into 「Customer name:」, ticking 「Large」, pressing 「Submit
/// order」. They used to land on the <p> and do nothing. Needs the box:
/// `cargo test the_words_on_a_form_reach_its_controls -- --ignored --nocapture`.
#[test]
#[ignore]
fn the_words_on_a_form_reach_its_controls() {
let own = crate::model::BOT_COMPUTER_NAME;
let setup = r#"mkdir -p /tmp/anybot-form && cd /tmp/anybot-form && cat > index.html <<'EOF'
<meta charset=utf-8><title>form</title>
<form action="done.html" method="get">
<p><label>Customer name: <input name="custname"></label></p>
<p><label>Telephone: <input type=tel name="custtel"></label></p>
<fieldset><legend> Pizza Size </legend>
<p><label> <input type=radio name=size value="small"> Small </label></p>
<p><label> <input type=radio name=size value="large"> Large </label></p></fieldset>
<fieldset><legend> Pizza Toppings </legend>
<p><label> <input type=checkbox name="topping" value="cheese"> Extra Cheese </label></p>
<p><label> <input type=checkbox name="topping" value="onion"> Onion </label></p></fieldset>
<p><button>Submit order</button></p>
</form>
EOF
echo '<meta charset=utf-8><title>done</title>送出了' > done.html; (setsid python3 -m http.server 8769 --bind 127.0.0.1 >/dev/null 2>&1 &) ; sleep 1; echo served"#;
let clean = r#"pkill -f 'http[.]server 8769'; rm -rf /tmp/anybot-form; curl -s 127.0.0.1:9222/json/list | python3 -c 'import json,sys,urllib.request; [urllib.request.urlopen("http://127.0.0.1:9222/json/close/" + t["id"]) for t in json.load(sys.stdin) if ":8769" in t["url"]]'; echo cleaned"#;
let port = fake_model(vec![
("shell", serde_json::json!({"command": setup})),
("browser_open", serde_json::json!({"url": "http://127.0.0.1:8769/"})),
("browser_type", serde_json::json!({"text": "Customer name:", "value": "李大華"})),
("browser_type", serde_json::json!({"text": "Telephone", "value": "0922000111"})),
("browser_click", serde_json::json!({"text": "Large"})),
("browser_click", serde_json::json!({"text": "Extra Cheese"})),
("browser_click", serde_json::json!({"text": "Onion"})),
("browser_click", serde_json::json!({"text": "Submit order"})),
("shell", serde_json::json!({"command": clean})),
("", serde_json::json!("好了")),
]);
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("填表")], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: own.into(), place: tools::Place::Box { dir: crate::cloudbox::bot_dir(1) } }],
default_computer: own.into(),
roles: tools::Roles { own: own.into(), person: None, server: true },
local_execution: false,
direct: Arc::new(AtomicBool::new(true)),
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut results = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(180)).expect("loop stalled") {
(_, StreamEvent::ToolResult(m)) => {
println!("{}: {}\n", m.tool_name, m.text.chars().take(240).collect::<String>());
results.push(m.text)
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert!(results[2].contains("欄位現在是「李大華」"), "{}", results[2]);
assert!(results[3].contains("欄位現在是「0922000111」"), "{}", results[3]);
let sent = &results[7];
assert!(sent.contains("done.html?custname=%E6%9D%8E%E5%A4%A7%E8%8F%AF&custtel=0922000111&size=large&topping=cheese&topping=onion"), "{sent}");
assert!(results[8].contains("cleaned"), "{}", results[8]);
}
#[test]
fn a_browser_step_carries_its_job() {
let a = with_job(r#"{"url":"https://example.com","profile":"工作"}"#, "6-1-1");
let v: serde_json::Value = serde_json::from_str(&a).unwrap();
assert_eq!((v["_job"].as_str(), v["url"].as_str(), v["profile"].as_str()), (Some("6-1-1"), Some("https://example.com"), Some("工作")));
assert_eq!(serde_json::from_str::<serde_json::Value>(&with_job("", "j")).unwrap()["_job"], "j");
// Two jobs never share a name.
assert_ne!(UsedBrowsers::new("b", 6).job, UsedBrowsers::new("b", 6).job);
}
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
#[test]
fn the_same_miss_counts_as_the_same_step() {
// Clicks a few pixels apart on the same strip are one step; another computer or target isn't.
assert_eq!(step_key("computer_use_click", r#"{"x":70,"y":355,"computer":"M"}"#), step_key("computer_use_click", r#"{"x":66,"y":358,"computer":"M"}"#));
assert_ne!(step_key("computer_use_click", r#"{"x":70,"y":355}"#), step_key("computer_use_click", r#"{"x":70,"y":290}"#));
assert_ne!(step_key("computer_use_click", r#"{"ref":"3"}"#), step_key("computer_use_click", r#"{"ref":"4"}"#));
// Looking is never a repeat; other tools aren't counted.
assert_eq!(step_key("computer_use_screenshot", "{}"), None);
assert_eq!(step_key("shell", r#"{"command":"ls"}"#), None);
assert!(step_key("browser_click", r#"{"ref":"5"}"#).is_some());
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// Reading down a page is not going in circles.
assert_eq!(step_key("browser_scroll", r#"{"direction":"down"}"#), None);
assert_eq!(step_key("computer_use_press_key", r#"{"key":"Down"}"#), None);
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
}
#[test]
fn a_click_by_number_is_named_in_the_question() {
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
let list = Message::tool_result("c1", "computer_use_screenshot", "[王小明的Mac mini] 螢幕截圖(1024×576)。\n「計算機」上可以操作的東西(圖上同樣的編號,用 ref 點):\n 文字「0」\n[4] 按鈕「全部清除」\n[7] 按鈕「7」");
assert_eq!(element_named(std::slice::from_ref(&list), "7").as_deref(), Some("按鈕「7」"));
assert_eq!(element_named(std::slice::from_ref(&list), "e7").as_deref(), Some("按鈕「7」"));
Bots click by number: every control on the screen is boxed and numbered, on every computer A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates from a picture, and the web tools' description told it they didn't work on its own computer. - marks.rs: the picture a bot gets has every button, link, field and menu item boxed and numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click / typing / scroll / mouse_move take the number (ref) and land on the control's centre. - Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets no number (topmost window at its centre must be the app's; the Dock's invisible full-screen window doesn't count). open_app waits until the app is really in front. - No going round in circles: a step that changed nothing says so; the same step (clicks within 16 px) a third time in a turn comes with a warning, from the fourth time it isn't run. - The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP, renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a minute (files and logins stay); a new box is never made from an older image. - Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt says web work goes through browser_* by number, and never to read browser cookie or history files. Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28, 153 unit tests; Linux build + 157 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
assert_eq!(element_named(&[list], "9"), None);
assert_eq!(computer::describe("computer_use_click", r#"{"ref":"7"}"#), "點 #7");
assert_eq!(computer::describe("computer_use_click", r#"{"x":10,"y":20}"#), "點 (10, 20)");
}
#[test]
fn every_computer_answers_to_its_own_name() {
// One name per computer: the bot's own goes by the bot's name, the others by theirs, and
// the everyday words point at the right one (「我的電腦」 = where the person typed).
let roles = tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true };
let own = tools::Target { name: "Master 的電腦".into(), place: tools::Place::Box { dir: String::new() } };
let mac = tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } };
let desk = tools::Target { name: "ai".into(), place: tools::Place::Device { host: "a".into(), dir: String::new() } };
assert!(mentions("在 Master 的電腦打開 Threads", &own, &roles));
assert!(mentions("用master的電腦", &own, &roles));
assert!(mentions("你都用 Bot的電腦", &own, &roles));
assert!(mentions("在 Mac mini 上開", &mac, &roles));
assert!(mentions("截圖我的電腦", &mac, &roles));
assert!(!mentions("截圖我的電腦", &desk, &roles));
assert!(!mentions("截圖我的電腦", &own, &roles));
assert!(mentions("在 ai 上跑", &desk, &roles));
assert!(!mentions("用 AI 幫我想標題", &mac, &roles));
}
#[test]
fn a_step_moved_back_to_the_chosen_computer_says_so() {
let roles = tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true };
let job = Job {
computers: vec![
tools::Target { name: "Master 的電腦".into(), place: tools::Place::Box { dir: String::new() } },
tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } },
],
default_computer: "Master 的電腦".into(),
roles,
..test_job()
};
// Not asked for: back to the chosen one, with a word for the bot.
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let (args, note) = keep_to_chosen(&job, &[Message::user("幫我截圖")], "shell", r#"{"computer":"王小明的Mac mini"}"#);
assert!(args.contains("Master 的電腦") && note.unwrap().contains("王小明的Mac mini"));
// Named by the person: it stays.
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
let (args, note) = keep_to_chosen(&job, &[Message::user("看一下我的電腦畫面")], "shell", r#"{"computer":"王小明的Mac mini"}"#);
assert!(args.contains("王小明的Mac mini") && note.is_none());
Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice Web - web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus. - browser_open waits for the page to load and returns it like browser_read (one step fewer per page); actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab. The bot's computer (box v6) - Numbers only on what can be seen: clipped to the window and the page's visible area, none under another app's window (a link scrolled under the toolbar was numbered on Back). - press_key with a typed character no longer swallows the keys after it (xdotool type --args 1). - The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys). - "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs. - One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone). The person's computers - Linux: the window system's front app when it has one, and the reason when there is none (was 「」); hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist. The loop - Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows): it was a plain shell on the brain itself, keys in reach, asked nothing. - The bot's own computer is re-checked each step; unreachable computers are failures, not safety blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys aren't counted as going in circles; steps aren't moved to a computer that can't do them. - One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the impossible server branches removed; duplicated prompt instructions merged. Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses on its box; remote E2E 28/28. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 09:44:59 +00:00
// The chosen computer has no screen (no Docker): a screen step isn't moved there to fail.
let sealed = Job { computers: vec![tools::Target { name: "Master 的電腦".into(), place: tools::Place::Sealed { root: std::env::temp_dir() } }, job.computers[1].clone()], ..job };
let (args, note) = keep_to_chosen(&sealed, &[Message::user("幫我截圖")], "computer_use_screenshot", r#"{"computer":"王小明的Mac mini"}"#);
assert!(args.contains("王小明的Mac mini") && note.is_none(), "{args}");
}
#[test]
fn cut_keeps_tool_pairs_and_recent_messages() {
let mut h = vec![];
for i in 0..6 {
h.push(Message::user(&format!("q{i}")));
let mut a = Message::assistant("", "");
a.tool_calls = vec![ToolCall { id: format!("c{i}"), name: "list_dir".into(), args: "{}".into(), status: ToolStatus::Done }];
h.push(a);
h.push(Message::tool_result(&format!("c{i}"), "list_dir", "x"));
}
let cut = compaction_cut(&h).unwrap();
assert_eq!(h[cut].role, Role::User);
assert!(h.len() - cut >= KEEP_RECENT_MESSAGES);
assert!(compaction_cut(&h[..5]).is_none());
}
#[test]
fn tokens_count_chinese_heavier() {
assert_eq!(estimate_tokens("abcdefgh"), 2);
assert_eq!(estimate_tokens("你好"), 2);
}
/// Full loop against a fake provider: the model asks to list a folder, gets the result,
/// then answers in words.
#[test]
fn loop_runs_tool_then_answers() {
use std::io::{Read, Write};
use std::net::TcpListener;
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
std::thread::spawn(move || {
let replies = [
"data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"c1\",\"function\":{\"name\":\"list_dir\",\"arguments\":\"{}\"}}]}}]}\n\ndata: [DONE]\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"裡面有 a.txt\"}}]}\n\ndata: [DONE]\n\n",
];
for body in replies {
let (mut s, _) = l.accept().unwrap();
let mut req = Vec::new();
let mut buf = [0u8; 8192];
loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break;
}
}
}
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(body.as_bytes());
}
});
let w = std::env::temp_dir().join(format!("anybot-loop-{}", std::process::id()));
std::fs::create_dir_all(&w).unwrap();
std::fs::write(w.join("a.txt"), "hi").unwrap();
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("有什麼檔案?")], tools: tools::specs(), session: String::new() },
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model: None,
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: true,
grants: vec![],
context_tokens: DEFAULT_CONTEXT_TOKENS,
via: "test".into(),
via_name: "Bot".into(),
tools_enabled: true,
direct: Arc::new(AtomicBool::new(false)),
local_locked: false,
steer: Arc::new(Mutex::new(vec![])),
review: None,
demos: vec![],
skills: vec![],
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op: String::new(),
log: None,
};
let (tx, rx) = channel();
let cancel = Arc::new(AtomicBool::new(false));
// Run the loop body on this thread's terms: spawn needs an egui context, so drive it directly.
let ctx = eframe::egui::Context::default();
spawn(job, 1, tx, cancel, ctx);
let mut texts = vec![];
let mut results = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(10)).expect("loop stalled") {
(_, StreamEvent::ToolResult(m)) => results.push(m.text),
(_, StreamEvent::StepDone(m)) => texts.push(m.text),
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert!(results[0].contains("a.txt"), "{results:?}");
assert_eq!(texts.last().unwrap(), "裡面有 a.txt");
}
/// A message sent while the bot works steers the same turn: the model sees it at the
/// next step and the turn only ends (one Done) after that.
#[test]
fn mid_turn_message_steers_the_same_turn() {
use std::io::{Read, Write};
use std::net::TcpListener;
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
let seen = Arc::new(Mutex::new(String::new()));
let seen2 = seen.clone();
std::thread::spawn(move || {
for body in ["data: {\"choices\":[{\"delta\":{\"content\":\"好\"}}]}\n\ndata: [DONE]\n\n", "data: {\"choices\":[{\"delta\":{\"content\":\"收到補充\"}}]}\n\ndata: [DONE]\n\n"] {
let (mut s, _) = l.accept().unwrap();
let mut req = Vec::new();
let mut buf = [0u8; 8192];
loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break;
}
}
}
*seen2.lock().unwrap() = String::from_utf8_lossy(&req).to_string();
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(body.as_bytes());
}
});
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
let steer = Arc::new(Mutex::new(vec![Queued::new("順便用英文")]));
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("打招呼")], tools: vec![], session: String::new() },
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model: None,
computers: vec![],
default_computer: crate::model::BOT_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: false,
grants: vec![],
context_tokens: DEFAULT_CONTEXT_TOKENS,
via: "test".into(),
via_name: "Bot".into(),
tools_enabled: true,
direct: Arc::new(AtomicBool::new(false)),
local_locked: false,
steer,
review: None,
demos: vec![],
skills: vec![],
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op: String::new(),
log: None,
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let (mut steered, mut done, mut texts) = (0, 0, vec![]);
loop {
match rx.recv_timeout(Duration::from_secs(10)).expect("loop stalled") {
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
(_, StreamEvent::Steered(m, _)) => {
assert!(m.steer);
steered += 1;
}
(_, StreamEvent::StepDone(m)) => texts.push(m.text),
(_, StreamEvent::Done) => {
done += 1;
break;
}
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert_eq!((steered, done), (1, 1));
assert_eq!(texts, vec!["好", "收到補充"]);
assert!(seen.lock().unwrap().contains("順便用英文"));
}
/// Tools off and this Mac not allowed: the step pops up two requests (use tools, use this
/// computer) instead of failing, and runs once both are allowed.
#[test]
fn missing_permission_pops_up_and_continues() {
use std::io::{Read, Write};
use std::net::TcpListener;
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
std::thread::spawn(move || {
let replies = [
"data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"c1\",\"function\":{\"name\":\"list_dir\",\"arguments\":\"{\\\"computer\\\":\\\"你的電腦\\\"}\"}}]}}]}\n\ndata: [DONE]\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"看完了\"}}]}\n\ndata: [DONE]\n\n",
];
for body in replies {
let (mut s, _) = l.accept().unwrap();
let mut req = Vec::new();
let mut buf = [0u8; 8192];
loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break;
}
}
}
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(body.as_bytes());
}
});
let w = std::env::temp_dir().join(format!("anybot-perm-{}", std::process::id()));
std::fs::create_dir_all(&w).unwrap();
std::fs::write(w.join("note.txt"), "hi").unwrap();
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("看看我的電腦")], tools: tools::specs(), session: String::new() },
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model: None,
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: false,
tools_enabled: false,
direct: Arc::new(AtomicBool::new(false)),
local_locked: false,
grants: vec![],
context_tokens: DEFAULT_CONTEXT_TOKENS,
via: "test".into(),
via_name: "Bot".into(),
steer: Arc::new(Mutex::new(vec![])),
review: None,
demos: vec![],
skills: vec![],
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op: String::new(),
log: None,
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut kinds = vec![];
let mut result = String::new();
loop {
match rx.recv_timeout(Duration::from_secs(10)).expect("loop stalled") {
(_, StreamEvent::NeedApproval(a)) => {
kinds.push(a.kind);
let _ = a.reply.send(ApprovalAnswer::Once);
}
(_, StreamEvent::ToolResult(m)) => result = m.text,
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert_eq!(kinds, vec![ApprovalKind::EnableTools, ApprovalKind::EnableThisComputer]);
assert!(result.contains("note.txt"), "{result}");
}
/// 需要處理: the model hands a step to the person; the card's request carries the instruction
/// and the computer, 「我完成了」 tells the bot to carry on, 「跳過」 that the step is skipped.
#[test]
fn a_hand_off_waits_for_the_person_and_says_what_they_chose() {
use std::io::{Read, Write};
use std::net::TcpListener;
for (answer, expect, status) in [(ApprovalAnswer::Once, "我完成了", ToolStatus::Done), (ApprovalAnswer::Reject, "跳過", ToolStatus::Denied)] {
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
std::thread::spawn(move || {
let replies = [
"data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"h1\",\"function\":{\"name\":\"hand_over_computer\",\"arguments\":\"{\\\"instruction\\\":\\\"請登入 Threads\\\"}\"}}]}}]}\n\ndata: [DONE]\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"好\"}}]}\n\ndata: [DONE]\n\n",
];
for body in replies {
let (mut s, _) = l.accept().unwrap();
let mut req = Vec::new();
let mut buf = [0u8; 8192];
loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break;
}
}
}
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(body.as_bytes());
}
});
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("幫我登入 Threads")], tools: vec![], session: String::new() },
computers: vec![tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } }],
default_computer: "王小明的Mac mini".into(),
roles: tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true },
..test_job()
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let (mut asked, mut result, mut last) = (None, String::new(), None);
loop {
match rx.recv_timeout(Duration::from_secs(10)).expect("loop stalled") {
(_, StreamEvent::NeedApproval(a)) => {
asked = Some((a.kind, a.reason.clone(), a.grant.clone()));
let _ = a.reply.send(answer.clone());
}
(_, StreamEvent::ToolStatus(_, st)) => last = Some(st),
(_, StreamEvent::ToolResult(m)) => result = m.text,
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert_eq!(asked, Some((ApprovalKind::Handoff, "請登入 Threads".to_string(), "王小明的Mac mini".to_string())));
assert!(result.contains(expect), "{result}");
assert_eq!(last, Some(status));
}
}
/// 直接操作: tools off, this Mac off, a command that changes files — and still no question.
/// Dangerous commands stay blocked.
#[test]
fn direct_mode_asks_nothing_but_still_blocks_danger() {
use std::io::{Read, Write};
use std::net::TcpListener;
let l = TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
std::thread::spawn(move || {
let replies = [
"data: {\"choices\":[{\"delta\":{\"tool_calls\":[{\"index\":0,\"id\":\"c1\",\"function\":{\"name\":\"shell\",\"arguments\":\"{\\\"command\\\":\\\"echo hi > made.txt\\\",\\\"computer\\\":\\\"你的電腦\\\"}\"}},{\"index\":1,\"id\":\"c2\",\"function\":{\"name\":\"shell\",\"arguments\":\"{\\\"command\\\":\\\"sudo ls\\\",\\\"computer\\\":\\\"你的電腦\\\"}\"}}]}}]}\n\ndata: [DONE]\n\n",
"data: {\"choices\":[{\"delta\":{\"content\":\"好了\"}}]}\n\ndata: [DONE]\n\n",
];
for body in replies {
let (mut s, _) = l.accept().unwrap();
let mut req = Vec::new();
let mut buf = [0u8; 8192];
loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break;
}
}
}
let _ = s.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n");
let _ = s.write_all(body.as_bytes());
}
});
let w = std::env::temp_dir().join(format!("anybot-direct-{}", std::process::id()));
std::fs::create_dir_all(&w).unwrap();
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("做一個檔案")], tools: tools::specs(), session: String::new() },
Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before; 16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s. Knowing the machine - facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager, tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the instructions at the start of a turn. - The box's sudo is the bot's: only what follows sudo is checked against the always-denied list (it was refused outright, while the instructions said it could sudo), and commands with sudo run as the box's root (setuid does nothing under gVisor). - Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE / ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one. - Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the sandbox broke says so, instead of looking like a broken program. Reaching what it sees - The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside for the system's own), computer_use_zoom looks closer at full sharpness. - Numbers stay with their controls from look to look (they were reassigned every look, so a step's second click landed elsewhere); the box's front window is listed first; a control can be named instead of numbered (name "4"). - Web page numbering walks open shadow DOM (Chromium settings, web components). - A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry without it came up with no network). Working like a person - Instructions: pick the route, know what done looks like, one route, check once, say what's stuck instead of faking it with a substitute, ask before cutting the connection. - StepPlan: the bot's model takes the first step, the step after trouble (failed, refused, repeated, nothing changed on screen, the quick model switching routes), one in four, and the final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does the planned steps in between, falling back to the bot's model if it can't be used. - Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does them again with no model in between, finding each control again, handing back on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 01:01:03 +00:00
step_model: None,
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w.clone() } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: false,
tools_enabled: false,
direct: Arc::new(AtomicBool::new(true)),
local_locked: false,
grants: vec![],
context_tokens: DEFAULT_CONTEXT_TOKENS,
via: "test".into(),
via_name: "Bot".into(),
steer: Arc::new(Mutex::new(vec![])),
review: None,
demos: vec![],
skills: vec![],
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
op: String::new(),
log: None,
};
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut asked = 0;
let mut results = vec![];
loop {
match rx.recv_timeout(Duration::from_secs(10)).expect("loop stalled") {
(_, StreamEvent::NeedApproval(a)) => {
asked += 1;
let _ = a.reply.send(ApprovalAnswer::Reject);
}
(_, StreamEvent::ToolResult(m)) => results.push(m.text),
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
assert_eq!(asked, 0, "{results:?}");
assert!(w.join("made.txt").exists());
assert!(results[1].contains("一律不允許"), "{results:?}");
}
A turn cut off by a crash leaves a 「繼續」 card, and nothing it may have done runs twice Each bot keeps a work log (oplog/bot-<id>.jsonl, after pi's durable-harness design, cut down): before something happens a line says it is about to — the turn, each model call, each step once its checks passed (with the arguments it runs with and whether it may run again), each message typed mid-turn — synced to disk first. When the turn ends the file shrinks to the messages still waiting, usually nothing. On the next start after AnyBot was killed mid-turn, the log puts the chat right before anything runs: the message the turn answered comes back if the chat lost it; a step that was running and reaches outside (a text, a click, a file, a published site) shows as 「可能已經執行了」, both to the person and in the result the bot reads — no longer as "not allowed"; a step that only looked shows as cut off; messages typed mid-turn that never reached the bot go back into the chat. The chat is saved, then the log closes the turn. A turn the person didn't stop ends with a 「繼續」 card (what it was doing, what may have run, the messages put back); it goes on only when they press it — in the window, on a screen of the brain, or on the phone — from where its steps stand, never running a step that may have run again. Pressing stop is on disk at once, and a stopped turn stays stopped. Also in the turn loop: - 自動審核 that doesn't answer (or answers something that isn't a verdict) asks the person; in 直接操作 mode it used to let the step through unchecked. - "Too long for this model" folds older messages into a summary and asks the same step again, once per thing the person said, instead of ending the turn. - Folding keeps the person's chat whole: the summary goes in at the fold and the model reads from the newest summary on (it used to delete what it folded). - A step moved to the chosen computer runs there, but the chat keeps the call as the model wrote it; the result says where it ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-08 16:49:12 +00:00
/// A made-up model that answers each request with the next (HTTP status, body) and keeps
/// what each request said.
fn fake_http(replies: Vec<(u16, String)>) -> (u16, Arc<Mutex<Vec<String>>>) {
use std::io::{Read, Write};
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let port = l.local_addr().unwrap().port();
let seen = Arc::new(Mutex::new(vec![]));
let seen2 = seen.clone();
std::thread::spawn(move || {
for (status, body) in replies {
let Ok((mut s, _)) = l.accept() else { return };
let mut req = Vec::new();
let mut buf = [0u8; 8192];
loop {
let n = s.read(&mut buf).unwrap_or(0);
if n == 0 {
break;
}
req.extend_from_slice(&buf[..n]);
let t = String::from_utf8_lossy(&req).to_string();
if let Some(end) = t.find("\r\n\r\n") {
let len = t[..end].lines().find_map(|l| l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))).unwrap_or(0);
if req.len() >= end + 4 + len {
break;
}
}
}
seen2.lock().unwrap().push(String::from_utf8_lossy(&req).to_string());
let head = if status == 200 { "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nConnection: close\r\n\r\n".to_string() } else { format!("HTTP/1.1 {status} Error\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", body.len()) };
let _ = s.write_all(head.as_bytes());
let _ = s.write_all(body.as_bytes());
}
});
(port, seen)
}
fn sse_text(t: &str) -> (u16, String) {
(200, format!("data: {}\n\ndata: [DONE]\n\n", serde_json::json!({"choices": [{"delta": {"content": t}}]})))
}
fn sse_call(id: &str, name: &str, args: serde_json::Value) -> (u16, String) {
(200, format!("data: {}\n\ndata: [DONE]\n\n", serde_json::json!({"choices": [{"delta": {"tool_calls": [{"index": 0, "id": id, "function": {"name": name, "arguments": args.to_string()}}]}}]})))
}
const OVERFLOW_BODY: &str = r#"{"error":{"message":"This model's maximum context length is 8192 tokens. However, your messages resulted in 9000 tokens."}}"#;
/// A long chat: enough older messages for a fold to have something to fold.
fn long_chat() -> Vec<Message> {
let mut h = vec![];
for i in 0..6 {
h.push(Message { at: i * 2 + 1, ..Message::user(&format!("第 {i} 個問題")) });
h.push(Message { at: i * 2 + 2, ..Message::assistant(&format!("第 {i} 個回答"), "test") });
}
h.push(Message { at: 100, ..Message::user("現在呢?") });
h
}
/// Run a job to its end, collecting what came back. Approvals get `answer`.
fn drive(job: Job, answer: ApprovalAnswer) -> (Vec<StreamEvent>, Vec<String>) {
let (tx, rx) = channel();
spawn(job, 1, tx, Arc::new(AtomicBool::new(false)), eframe::egui::Context::default());
let mut evs = vec![];
let mut asked = vec![];
loop {
let (_, ev) = rx.recv_timeout(Duration::from_secs(15)).expect("loop stalled");
match ev {
StreamEvent::NeedApproval(a) => {
asked.push(a.reason.clone());
let _ = a.reply.send(answer.clone());
}
StreamEvent::Done | StreamEvent::Error(_) => {
evs.push(ev);
break;
}
ev => evs.push(ev),
}
}
(evs, asked)
}
#[test]
fn a_verdict_is_clear_or_the_review_failed() {
assert!(matches!(parse_verdict(r#"{"verdict":"ask","rule":"刪除前先問"}"#), ReviewVerdict::Ask(r) if r == "刪除前先問"));
assert!(matches!(parse_verdict(r#"好的 {"verdict":"allow","rule":"讀檔"}"#), ReviewVerdict::Allow(_)));
assert!(matches!(parse_verdict(r#"{"verdict":"none","rule":""}"#), ReviewVerdict::NoMatch));
for broken in ["", "嗯…讓我想想", r#"{"verdict":"maybe"}"#, r#"} {"#] {
assert!(matches!(parse_verdict(broken), ReviewVerdict::Failed), "{broken}");
}
}
/// 直接操作 with an 「一律詢問」 rule: when the reviewer doesn't give a verdict, the step is
/// asked about, never run unchecked.
#[test]
fn a_review_that_fails_asks_instead_of_allowing() {
let (port, _) = fake_http(vec![
sse_call("c1", "shell", serde_json::json!({"command": "echo hi > made.txt", "computer": crate::model::THIS_COMPUTER_NAME})),
sse_text("嗯…讓我想想"),
sse_text("好,我不做了"),
]);
let w = std::env::temp_dir().join(format!("anybot-review-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&w);
std::fs::create_dir_all(&w).unwrap();
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("做一個檔案")], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w.clone() } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
direct: Arc::new(AtomicBool::new(true)),
review: Some(Review { allow: vec![], ask: vec!["刪除或修改任何檔案前先問我".into()] }),
..test_job()
};
let (evs, asked) = drive(job, ApprovalAnswer::Reject);
assert_eq!(asked.len(), 1, "{asked:?}");
assert!(asked[0].contains(REVIEW_FAILED_NOTE), "{}", asked[0]);
assert!(!w.join("made.txt").exists());
assert!(matches!(evs.last(), Some(StreamEvent::Done)));
}
/// The model says the chat is too long: older messages are folded into a summary and the same
/// step is asked again (with the summary), instead of the turn failing.
#[test]
fn an_overflow_folds_the_chat_and_asks_again() {
let (port, seen) = fake_http(vec![(400, OVERFLOW_BODY.into()), sse_text("摘要:問過六個問題"), sse_text("好了")]);
let job = Job { req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: long_chat(), tools: vec![], session: String::new() }, ..test_job() };
let (evs, _) = drive(job, ApprovalAnswer::Reject);
assert!(matches!(evs.last(), Some(StreamEvent::Done)), "{:?}", evs.iter().filter_map(|e| if let StreamEvent::Error(e) = e { Some(e.clone()) } else { None }).collect::<Vec<_>>());
assert!(evs.iter().any(|e| matches!(e, StreamEvent::Compacted(..))));
let texts: Vec<String> = evs.iter().filter_map(|e| if let StreamEvent::StepDone(m) = e { Some(m.text.clone()) } else { None }).collect();
assert_eq!(texts, vec!["好了"]);
let seen = seen.lock().unwrap();
assert_eq!(seen.len(), 3);
assert!(seen[2].contains("摘要:問過六個問題") && !seen[2].contains("第 0 個問題"), "{}", seen[2]);
}
/// Still too long after folding once: the turn ends with a clear error (no loop of folds).
#[test]
fn a_second_overflow_ends_the_turn() {
let (port, seen) = fake_http(vec![(400, OVERFLOW_BODY.into()), sse_text("摘要"), (400, OVERFLOW_BODY.into()), sse_text("不該問到這裡")]);
let job = Job { req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: long_chat(), tools: vec![], session: String::new() }, ..test_job() };
let (evs, _) = drive(job, ApprovalAnswer::Reject);
match evs.last() {
Some(StreamEvent::Error(e)) => assert!(providers::is_context_overflow(e) && e.contains("還是太長"), "{e}"),
_ => panic!("expected an error"),
}
assert_eq!(seen.lock().unwrap().len(), 3);
}
/// The work log while a turn runs: the turn and its model call are written first; a step
/// waiting for the person isn't "started"; once allowed it is (with the arguments it runs
/// with), then finished; a finished turn leaves no file. A stopped turn ends as stopped.
#[test]
fn the_work_log_follows_the_turn() {
use crate::oplog::{reduce, OpLog};
let w = std::env::temp_dir().join(format!("anybot-oplog-turn-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&w);
std::fs::create_dir_all(&w).unwrap();
std::fs::write(w.join("a.txt"), "hi").unwrap();
let path = w.join("log.jsonl");
for stop in [false, true] {
let (port, _) = fake_http(vec![sse_call("c1", "shell", serde_json::json!({"command": "echo hi > b.txt"})), sse_text("做好了")]);
let cancel = Arc::new(AtomicBool::new(false));
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message { at: 42, ..Message::user("做 b.txt") }], tools: tools::specs(), session: String::new() },
computers: vec![tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w.clone() } }],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: true,
op: "turn-test".into(),
log: Some(OpLog::at(path.clone())),
..test_job()
};
let read = || reduce(&OpLog::at(path.clone()).read());
let raw = || std::fs::read_to_string(&path).unwrap_or_default();
let (tx, rx) = channel();
spawn(job, 1, tx, cancel.clone(), eframe::egui::Context::default());
loop {
match rx.recv_timeout(Duration::from_secs(15)).expect("loop stalled") {
(_, StreamEvent::NeedApproval(a)) => {
let open = read().open.expect("the turn is open");
assert_eq!((open.op.as_str(), open.prompt.map(|p| p.at)), ("turn-test", Some(42)));
assert!(open.started.is_empty(), "waiting for the person is not started");
assert!(raw().contains("\"step_attempt\""));
if stop {
cancel.store(true, Ordering::Relaxed);
drop(a);
break;
}
let _ = a.reply.send(ApprovalAnswer::Once);
}
(_, StreamEvent::ToolResult(_)) => {
let open = read().open.expect("the turn is still open");
assert_eq!(open.started.len(), 1);
assert!(open.started[0].finished && open.started[0].replay == crate::oplog::Replay::Never);
assert!(raw().contains("echo hi > b.txt"));
}
(_, StreamEvent::Done) => break,
(_, StreamEvent::Error(e)) => panic!("{e}"),
_ => {}
}
}
if stop {
// The thread winds down on its own; its last line says stopped.
let until = std::time::Instant::now() + Duration::from_secs(5);
while path.exists() && std::time::Instant::now() < until {
std::thread::sleep(Duration::from_millis(50));
}
}
assert!(!path.exists(), "a closed turn leaves nothing behind: {}", raw());
}
}
/// A step moved to the chosen computer runs there, but the chat keeps the call as the model
/// wrote it; the result says where it ran.
#[test]
fn a_moved_step_is_kept_as_the_model_asked() {
let (port, _) = fake_http(vec![sse_call("c1", "list_dir", serde_json::json!({"computer": "王小明的Mac mini"})), sse_text("好")]);
let w = std::env::temp_dir().join(format!("anybot-moved-{}", std::process::id()));
std::fs::create_dir_all(&w).unwrap();
std::fs::write(w.join("here.txt"), "hi").unwrap();
let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("有什麼檔案?")], tools: tools::specs(), session: String::new() },
computers: vec![
tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: tools::Place::Local { workdir: w.clone() } },
tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } },
],
default_computer: crate::model::THIS_COMPUTER_NAME.into(),
roles: tools::Roles { own: crate::model::BOT_COMPUTER_NAME.into(), person: Some(crate::model::THIS_COMPUTER_NAME.into()), server: false },
local_execution: true,
..test_job()
};
let (evs, _) = drive(job, ApprovalAnswer::Once);
let asked = evs.iter().find_map(|e| if let StreamEvent::StepDone(m) = e { m.tool_calls.first().cloned() } else { None }).unwrap();
assert!(asked.args.contains("王小明的Mac mini"), "{}", asked.args);
let result = evs.iter().find_map(|e| if let StreamEvent::ToolResult(m) = e { Some(m.text.clone()) } else { None }).unwrap();
assert!(result.contains("這一步你指定了「王小明的Mac mini」") && result.contains("here.txt"), "{result}");
}
}