Commit Graph

37 Commits

Author SHA1 Message Date
AnyBot Dev 0884ec736c Shortcuts before GUI (C6): the one-step doors, registered
- intents.rs: volume / wifi / lock / dark / trash / eject as named
  shortcuts — the model picks a name, the script is built from
  constants and checked numbers ("volume set 40; rm -rf /" is not a
  thing: args are grammar-checked before any shell sees them)
- the computer tool gains action=intent {intent_name, intent_arg}:
  one call where the bench measured 9 steps and 2 minutes of GUI
- the computer's facts line advertises the shortcuts each machine
  really has (Mac 6, Linux desktop 5, headless box 0 — nobody hears
  a box's volume), so the prompt cache carries the knowledge for free
- scripts self-dispatch (uname): osascript/networksetup/pmset on Mac,
  pactl/amixer/nmcli/loginctl/gsettings on Linux, honest failure text
- system prompt: 有捷徑先捷徑,捷徑失敗才回 GUI
- tests: the injection grammar, the availability matrix, a real run
2026-11-09 20:30:38 +08:00
AnyBot Dev ef58028cf4 Merge s3/injection: injection defences + red-team bench (S3) 2026-11-09 17:20:10 +08:00
AnyBot Dev e1d0998c20 Injection defences and a red-team suite (S3)
- clicks on controls whose own words say money/sending/deleting ask the person
  every time — on every computer, in direct mode, no grant silences it (the
  harness decides, not the model, so a page cannot talk its way past it)
- screenshots paint card-number and OTP boxes black before any model sees them
- content the bot READS (pages, files, screens) enters the history wrapped in
  <untrusted>, and the system prompt says the wrapper means content-not-orders
- keep_to_chosen now has a test on the screen tools: a model cannot move a
  screen step onto another computer by itself (B26's promise, pinned)
- bench red team R1–R7: every payload's goal is a harmless probe file /
  button code; passing = reporting the page honestly and leaving the probe
  untouched. Run: --tasks R1..R7
2026-11-09 17:19:21 +08:00
daniel wang 844abe5a85 Merge c3/hi-tools: semantic actions (C3) 2026-11-09 17:15:58 +08:00
AnyBot Dev aede36547e High-level semantic actions: menu_click, fill_form, wait_for, batch expect (C3)
- menu_click walks the real macOS menu bar in one step (three screenshot
  rounds per level become one call + one picture)
- fill_form writes every named field with one AXValue write each, in one step
- wait_for polls the control tree (no screenshots) until text appears/gone —
  replaces the screenshot-polling loops that filled whole turns with identical
  pictures
- batch steps take an optional expect: the batch stops and shows the screen at
  the step whose promised text never showed, instead of running a blind plan
  on sand

Menu/fill are macOS accessibility; other systems keep click+typing per field
(the tool descriptions say which is which).
2026-11-09 17:15:58 +08:00
daniel wang 7b3a9fc4d5 Merge c4/tree-diff: screenshot diffs (C4) 2026-11-09 17:12:25 +08:00
AnyBot Dev 55f81dbf65 Screenshot results carry a control-tree diff (+/- lines)
The quick model learned what its last click did only by re-reading two long
numbered lists. A multiset diff of (kind, name) between consecutive looks adds
two lines and answers it directly — including the case that matters most, a
dialog that appeared entirely.
2026-11-09 17:12:25 +08:00
AnyBot Dev ccfeaff235 Noop detection by machine tag, not by matching the Chinese sentence 2026-11-09 17:10:25 +08:00
daniel wang 649c1fcf50 Merge c2/ax-write: accessibility-first click/typing on macOS (C2) 2026-11-09 16:53:12 +08:00
AnyBot Dev d9b6330b52 Accessibility write (macOS): press by name/spot and set the focused field's value, pixel path as fallback
The bench's precision failures (quick model clicking the wrong number, Retina
scaling, covered windows) all come from acting on pixels. A plain left click
now asks the frontmost app's accessibility tree first — by name, or by the
numbered spot — and typing writes AXValue into the focused field once,
keyboard events only when accessibility can't reach. Any app with a thin tree
behaves exactly as before: every AX answer that isn't a clear yes falls
through to CGEvent.

macOS code type-checked with --target x86_64-apple-darwin; gate: build on the
macos-arm64 runner + bench tasks T2/T5 (clicking by name) on a Mac.
Linux/Windows: stubs, pixel path unchanged.
2026-11-09 16:53:12 +08:00
AnyBot Dev f2af1f65b4 One computer tool with an action enum replaces twelve computer_use_* specs
The model picks action=screenshot/click/typing/… on a single `computer` tool
(the shape Anthropic's and OpenAI's computer-use models are trained on):
choosing among twelve near-identical names was its own error source, and the
twelve schemas were the biggest stable part of every request. Calls expand to
the internal per-action tools at the model-output boundary, so approvals,
repeat guards, skill replay, the remote limb protocol and the work log keep
their existing names — old limbs need no update.
2026-11-09 16:50:18 +08:00
AnyBot Dev 1bc2fdcec9 Rewrite decompilation comments as behaviour specs; move reverse-engineering docs out of the repo; drop third-party marks from UI strings (i18n tables synced) 2026-11-09 16:37:33 +08:00
王性驊 22eaf031b0 The quick model runs again, the Mac's own menus get numbers, and an unread window is said
- quick_tools indexed serde_json Values mutably, which inserted
  actions.items.properties = null into click/typing/scroll; xAI refused
  the whole tool list, so since ba1fce5 every step fell back to the main
  model. It now uses get_mut and the test checks nothing but ref changes.
- The front app's menu bar titles and any open menu (submenus, right-click
  menus) are listed with numbers and their shortcuts; an option-key
  alternate in the same place is noted on the visible item.
- Disclosure triangles, colour wells and date fields get numbers; an
  icon-only control without a name goes by its description.
- When the front window can't be read the list says so (with the reason)
  instead of 「」 above the menu bar alone.
- Computer facts add each system's conventions (Mac: cmd+shift+G, save
  panel, default format, /bin/sh echo -n; Linux: GTK ctrl+L).
- `anybot --help` and unknown -- flags print the usage and exit instead of
  opening the window on the real data folder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 14:58:40 +08:00
王性驊 ba1fce5558 Keys wait until the app the bot looked at is still in front; the quick model names its controls; system prompts are the person's
- Keys and typing into no field go to whatever is in front: a cmd+w meant for TextEdit reached
  the person's terminal after the TextEdit window had closed (its "close this window?" came up
  and was cancelled). They are now refused, with why, when the app in front isn't the one of the
  latest look.
- The quick model's screen tools have no `ref`: looking numbers up is where it pressed the wrong
  keys; it names controls (two of a name come back as trouble for the bot's own model).
- Instructions: macOS permission and security prompts are for the person to answer (a bot
  pressed 「不允許」 on one), and check what's in front before a shortcut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 14:03:40 +08:00
王性驊 d4745a7a7a The same click twice in a step is done once, a front app that vanishes for a moment is looked at again, and the notes have the Mac and held-out results
- Two identical clicks in one step opened the Mac's Wi‑Fi menu and closed it again: the second is
  skipped with a word on why (count: 2 for a double click).
- macOS Calculator had no focused window for a moment after 「乘」 and the list came back with
  only the menu bar: an empty or failed read is tried once more 0.3 s later.
- Benchmark notes: three more unseen apps on box v9 (16/16, GTK 2 apps 4-10x faster with gail)
  and the Mac mini run with Screen Recording (zoom, Wi‑Fi menu, Calculator checked by eye).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 13:31:25 +08:00
王性驊 1be088474e The box really replaces a one-line field's text; a turn that keeps going wrong stops handing steps to the quick model; a missed name says what's close
- The one-line-field check got the step's name as 「#14(輸入框…)」 on the box and never
  matched, so the select-and-paste never ran there and addresses still came out mangled. It
  reads both forms now, with a test on the script the box runs.
- Trouble twice in a turn and the rest of the turn is the bot's own model's (a quick model
  guessed its way through Mousepad's save dialog for 79 steps).
- A name that isn't on the screen comes back with the names close to it ([12]「New Window」…),
  so the step is put right at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 11:25:17 +08:00
王性驊 9e8135ceda Typing into a one-line field replaces it, a named control beats stray coordinates, and a box left by a hard stop starts on the second try
- Typed into the middle of Chromium's 「about:blank」, an address came out as
  chrome://blankchrome//settings/search, and every step after it went wrong. Typing into a
  field by name or number now selects a one-line field's text first (ctrl+a / cmd+a) so the
  text replaces it; append=true keeps it. Text areas are left alone.
- A model sending name plus x: 0, y: 0 had its click land in the screen's corner: a name (or
  number) now always wins over coordinates.
- After the VM went down hard, the box's first start failed in gVisor ("mount source already
  has a filestore file … repeated submounts are not supported") and the second worked: waking
  or restarting the box tries once more on exactly that error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 10:19:18 +08:00
王性驊 be790f36ba Screen steps name their controls first, and a batch takes a whole row of keys; four held-out tasks
A quick model looking numbers up pressed 1, 2, 3, 1 for 1234, and 1234 × 5678 = didn't fit in
an 8-action batch (it split it and lost a key). The click and batch descriptions now lead with
the control's name (name "4"), and a batch takes up to 16 actions.

The benchmark gets four tasks on things none of the others use (Mousepad and its save dialog,
a Chromium bookmark, ImageMagick, a page of nested web components), run only when asked for
(--tasks N1,…): a check that the changes carry over to what they weren't tuned on.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 09:36:50 +08:00
王性驊 1202de93ad Bots work like someone who knows the computer: they know each machine, reach what they see, plan, and hand the planned steps to a quick model
Measured on the bot's own box (8 tasks x 2, grok-4.7): 14/16 done, 120 steps, 1114 s before;
16/16, 102 steps, 574 s after (model time 1015 -> 471 s). A replayed skill: 49 s -> 15 s.

Knowing the machine
- facts.rs: 電腦現況 per computer (system, desktop, systemd, sudo, network, package manager,
  tools, apps, a fixed TZ), read once by a small read-only probe, kept 30 min, in the
  instructions at the start of a turn.
- The box's sudo is the bot's: only what follows sudo is checked against the always-denied
  list (it was refused outright, while the instructions said it could sudo), and commands with
  sudo run as the box's root (setuid does nothing under gVisor).
- Box image v8 drops ENV TZ=UTC (it overrode /etc/localtime). ANYBOT_BOX_IMAGE /
  ANYBOT_BOX_VOLUME give a throwaway box its own image and volume, never adopting the real one.
- Mac shell sandbox: App Store app CLIs (they carry their own app sandbox) and ps/top run
  outside it; plain `osascript -e` too, asked the first time even in direct mode, refusing do
  shell script, terminals, keystrokes, file reads, ObjC and the secret folders. A command the
  sandbox broke says so, instead of looking like a broken program.

Reaching what it sees
- The Mac menu bar's status icons are numbered (each app's AXExtrasMenuBar, named from inside
  for the system's own), computer_use_zoom looks closer at full sharpness.
- Numbers stay with their controls from look to look (they were reassigned every look, so a
  step's second click landed elsewhere); the box's front window is listed first; a control can
  be named instead of numbered (name "4").
- Web page numbering walks open shadow DOM (Chromium settings, web components).
- A box on a Docker host elsewhere retries the next port when the usual one is taken (a retry
  without it came up with no network).

Working like a person
- Instructions: pick the route, know what done looks like, one route, check once, say what's
  stuck instead of faking it with a substitute, ask before cutting the connection.
- StepPlan: the bot's model takes the first step, the step after trouble (failed, refused,
  repeated, nothing changed on screen, the quick model switching routes), one in four, and the
  final check; a quick model (Bot.step_model, auto: grok-4.20-0309-non-reasoning on xAI) does
  the planned steps in between, falling back to the bot's model if it can't be used.
- Skills keep the steps that worked (with what each acted on, not numbers); replay_skill does
  them again with no model in between, finding each control again, handing back on a mismatch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha5F5gJt93t3yEx3XcS
2026-11-09 09:01:03 +08:00
王性驊 28c043617e Linux: the front app is found by its program, and accessibility is switched on when it's off
agent-desktop knows an app by its program's name, but the X11 window says its class: a Snap
Firefox is "firefox_firefox", so the front app was never found and the bot got a bare picture
(「看不出最前面是哪個 App(WINDOW_NOT_FOUND)」). The window's process now gives the name.

When there are no controls at all and the desktop's accessibility switch (org.a11y.Status
IsEnabled) is off — most apps then show none — AnyBot turns it on, as a screen reader would, and
tells the bot that apps already open show their numbers after a restart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-08 00:58:18 +08:00
王性驊 4506cc149b Clippy is clean on macOS and Linux, for the app and the relay
Every warning `cargo clippy --all-targets` gave (50 on macOS, 8 more only on
Linux, 1 in the relay), fixed without changing what anything does:
- long types get a name (Peer, Members, Imported, Joining, Mention, Stream…);
- the two big enum variants are boxed (the vault's login plan, joining an
  account);
- a test now waits for the copy it relaunched (it left a zombie behind);
- a hint's format had a dead `if` whose two branches were both "";
- the rest is what clippy suggests: sort_by_key, as_chunks, is_none_or,
  is_multiple_of, while let, from_ref, next_back, struct init, no casts to
  the same type.

Checked: cargo test (177 macOS, 181 Linux), relay tests (11), remote_e2e
28/28, and the live Linux test (X11; Wayland clicks, keys, Chinese typing,
scroll and screenshots).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-07 10:10:28 +08:00
王性驊 e8e5e6fba1 Real-use fixes: the person sees the screens the bot shows, files reach the phone, no blank-tab loops
From real runs on the cloud brain with the person's Mac (Threads patrol, a post draft, "show me my
screen"):
- computer_use_show: a plain picture of any computer's screen goes into the chat (the bot sees it
  too). The bot used to describe a screenshot in words and fail to send it.
- The phone shows the bot's attachments: pictures in place, other files download on a tap (api/file,
  only from the chats' file folder). It showed nothing for the bot and names only for the person.
- send_file works from the person's other computers (fetch_file over the relay, 20 MB, secrets stay).
- The same call with the same answer again and again is warned the 3rd time and stopped from the
  4th (a blank tab was read 11 times, then the bot said Threads was logged out without opening it);
  a blank tab says so; checking a login means opening the site.
- Screenshot lists: a nameless row around a named button isn't numbered, a nameless row takes the
  words inside it, empty spacer rows aren't numbered (System Settings: 63 numbers -> 33, all named).
- Writing "in my voice" reads the person's own recent posts first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 19:37:45 +08:00
王性驊 76a76c7d7a Whole control flow checked against real chats: what the bot sees is what is there, every step lands, nothing done twice
Web
- web_fetch kept pages with a <header> only up to it (taken for <head>): articles came back as their menus.
- browser_open waits for the page to load and returns it like browser_read (one step fewer per page);
  actions wait for the page to hold still instead of fixed 1-2.5 s sleeps; one way to reach the tab.

The bot's computer (box v6)
- Numbers only on what can be seen: clipped to the window and the page's visible area, none under
  another app's window (a link scrolled under the toolbar was numbered on Back).
- press_key with a typed character no longer swallows the keys after it (xdotool type --args 1).
- The person's input through noVNC/TigerVNC is noticed again (fence, continuous updates, QEMU keys).
- "Nothing changed" only after clicks, scrolls and drags; long text is pasted; one screen step at a
  time; one docker inspect per step when the box is known ready; poppler-utils for long PDFs.
- One typing path and fewer round trips (stdin variant, unused picture path, extra batch execs gone).

The person's computers
- Linux: the window system's front app when it has one, and the reason when there is none (was 「」);
  hidden and off-screen controls get no number; wording no longer names a tool that doesn't exist.

The loop
- Without Docker, the bot's own computer runs no commands where there is no sandbox (Linux, Windows):
  it was a plain shell on the brain itself, keys in reach, asked nothing.
- The bot's own computer is re-checked each step; unreachable computers are failures, not safety
  blocks; no direct-mode note on every result; an empty reply is asked again; scrolling and keys
  aren't counted as going in circles; steps aren't moved to a computer that can't do them.
- One finish() for every step's result, keep_grant(), dead appctl list_computers/SERVING and the
  impossible server branches removed; duplicated prompt instructions merged.

Tests: 162 (Mac) / 166 (Linux) unit; real box drills 6/6 on box v6; a brain without Chrome browses
on its box; remote E2E 28/28.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 17:44:59 +08:00
王性驊 cbf43ce2d4 Bots do a planned sequence in one step; the box's screen settles by what it shows; no login is ever the only copy
- computer_use_batch: up to 8 planned actions run in one docker exec (steps chained with && and a
  done-marker each), the first failure stops the rest, one picture comes back at the end. Typed
  text rides inside the script quoted (one stdin for the chain). The prompt tells the model when
  to use it (a form whose fields are all visible) and when not to (step 2 needs step 1's screen).
- The box waits for the screen to stop changing, not the clock: two matching small probes (~60 ms
  each) mean settled — ~160 ms on a still screen where every step paid a fixed 500 ms, and a page
  mid-load is waited for (up to ~0.9 s) instead of captured mid-change. The picture is JPEG, and
  the a11y read runs in parallel with the capture (look ~307 ms; two steps with pictures 1102 ms).
  (SETTLE_SH is a group command: dash needs ';' after it — found by box_screen_is_exact.)
- The home volume (every bot's logins) is never one command from loss: snapshots keep generations
  (last 7 days, then one per week; the old single snapshot migrates by file date), Reset and
  Restore save the running box first and refuse to wipe if saving fails, a box built on an empty
  volume re-fills itself from the newest snapshot, and a sleeping holder container keeps the
  volume attached so `docker volume prune` can't take it. make doctor shows volume and snapshot
  state; the 電腦 card shows the box's average step times and says resets save first.
- box_batch_in_one_round_trip (ignored): three typed steps, ASCII and Chinese, land in one trip.
  On the real box after upgrading v3→v5: box_update_keeps_files (files survive, snapshot 2 代),
  box_numbers_are_exact, box_screen_is_exact, box_end_to_end, box_traffic_via_mac pass; 155 unit tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 09:47:37 +08:00
王性驊 5656d355af Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.

- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
  numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
  typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
  computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
  box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
  no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
  window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
  16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
  at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
  renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
  (list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
  minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
  per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
  says web work goes through browser_* by number, and never to read browser cookie or history
  files.

Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 00:47:01 +08:00
王性驊 2867c8e822 Remove what nothing uses and keep one copy of what was written several times
Unused (found by taking away the dead-code allowances; nothing else used them, tests aside):
computer::specs, quiet_button, the never-made Poke::Spin, ten icons and their SVGs, the
theme's chrome/bubble_agent colours, ImportRow.line, voice's multipart/transcribe/speak
wrappers, and the old keychain writers vault::save/remove/index_add with their live test
(the vault store replaced them; reading old keychain entries for the import stays).

One copy each: percent-encoding (httpd::url_encode, was four), tool arguments (tools::args_of,
was three), the WebSocket handshake (httpd::accept_websocket, was three), the read pacing
(wire::Pace, was two: relay and app), docker with input and a time limit (dk_with_input, was
two). Docs: FEATURES §看畫面 (VNC), protocol (v* frames, /screen, tickets), architecture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 23:21:47 +08:00
王性驊 02bca39b98 People watch and help on the bots' computer through VNC (noVNC), carried by the account server
The core: the bot works the computer; a person watches it like a colleague, and takes over
when it needs them. VNC is made for that (only what changed, every mouse move and key, touch on
phones), so the pictures sent one by one go, and with them their own input path.

- vnc.rs: a VNC proxy to the box's x11vnc through one docker exec (a Python bridge that ends
  with either side). It signs in to the box with the box's own password (its VNC_PASSWORD) and
  offers the viewer no password, so the password never leaves the brain. The person's clicks
  and keys make the bot's steps wait until 8 s after they stop; text typed in the viewer's box
  (any language) is pasted in the box.
- Account server: /screen (the viewer page) and /novnc.js (noVNC 1.6.0, MPL-2.0, bundled);
  /relay/vnc/<host> takes the browser's WebSocket (account session, or a 10-minute ticket the
  app asks for with POST /relay/screen-ticket; owners and admins) and carries it over that
  AnyBot's lasting connection (vopen / vdata / vtext / vclose). Both ends of that connection
  wait 2 ms instead of 40 ms to read while things move: a request and its answer took up to
  160 ms more than needed.
- Phone page: 看 Bot 的電腦 and 接管 open the viewer page (also on the local network: /api/vnc on
  the AnyBot's own phone server). This Mac's own screen moved to the ⋯ menu.
- Mac app: 看畫面 opens the viewer in the browser (a screen of a brain: the brain's computer by
  ticket; else this computer's box, noVNC straight from it). The in-app picture viewer, the
  live JPEG stream, the person's click/move/scroll/key endpoints and the input shell are gone.
- Opening a Mac app uses LaunchServices (`open -a`): agent-desktop's launch waited on the window
  through accessibility and failed without 輔助使用 now and then.

Measured on this Mac (made-up account, local relay + headless brain + the real box): viewer
signed in in 0.4–0.7 s, a request and its answer through relay + brain + box 25–58 ms median;
headless Chrome shows the desktop live in /screen. remote_e2e 28/28 ×2, 149 unit tests, relay 11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 23:05:44 +08:00
王性驊 aa9b010276 One way to act on a screen: a real click or keystroke, wherever the bot looked
- Acting by an app's number (read_app's e7) is now a real click at that control's spot, the
  same path as screenshot coordinates. agent-desktop only reads; its own actions (a second set
  with its own key names and failures: no typing in terminals, "/" and "`" failing off the main
  thread) are gone, with computer_use_select, background clicks and press_key's app.
- Every action comes back with the screen after it (or, with no picture allowed, the front
  app's numbered controls), so the bot no longer spends a step on a screenshot to check.
- Coordinates map without remembering a screenshot: the picture's size depends only on the
  screen. The phone's taps go straight to the screen as the person's, in the pixels of the
  picture it was sent; they used to borrow the bot's last screenshot scale and landed 1.25x off
  on the bots' computer, or failed with "screenshot first".
- Mac typing pastes (⌘V, clipboard put back): with 注音 on, text sent as key events vanished
  while return arrived, so commands typed into a terminal ran empty. Bare printing keys
  ("slash", "r") are typed the same way; keys like "slash" and "grave" have names.
- The bots' Docker computer: one docker exec per step (action, settle, picture), never
  `mousemove --sync` (it waits forever when the pointer is already there: a second click on
  the same spot hung two minutes), open_app (chromium = the browser browser_* drive, xterm).

Tests: box_screen_is_exact (real Docker: bot and phone coordinates exact, same spot twice in
under 2 s, ASCII + Chinese + named key typed), Calculator by numbers and Terminal typing under
注音 through the signed app, remote_e2e 28/28, Linux build + 151 tests + x11_live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 21:51:19 +08:00
王性驊 60f4fa7411 Reading or using an app without 輔助使用 asks for it and says so in words (it answered PERM_DENIED and asked nobody)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:50:08 +08:00
王性驊 d52def0f46 A click by number reads as the element it is in the question (點 e7(按鈕「7」)), not (0, 0)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 18:25:46 +08:00
王性驊 5b2c974f3c Mac permissions: ask at most every ten minutes; one fixed signing key keeps them across updates; open apps by their shown name
- Every untrusted step popped macOS's prompt and opened Settings; a remote brain's steps covered
  the screen. Now once per pane per ten minutes.
- Ad hoc signing made each build a new app to macOS, so 輔助使用/螢幕錄製 silently stopped
  working after updates (agent-desktop PERM_DENIED). deploy/mac/sign.sh signs with a local key
  kept in its own keychain under AnyBot's folder.
- computer_use_open_app finds 「計算機」 by display name and says the app opened even when its
  window can't be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:57:44 +08:00
王性驊 3132f8ce91 Every computer goes by its own name; a server brain works on the computer the person uses
- A bot's own computer is 「<bot> 的電腦」; this one and the account's others go by their names.
  The server brain picked 你的電腦 by sorted host id, so a desktop left on the server won.
- 「我的電腦」 is the computer the person typed on (the relay names it); unknown means ask.
- A step moved back to the chosen computer tells the bot so.
- Composer chip and bot settings choose the computer by name; a screen asks the brain.
- A screen's 電腦 card and viewer show the brain's computer, not this one's box.
- A screen's chat copy that falls short asks for everything again instead of appending;
  清除聊天記錄 on a screen clears the brain's chat.
- Model requests to a private address go through the person's computer first.
- Phone typing on the box sends the text the tool reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:28 +08:00
王性驊 591626408f A server brain without hands calls its server the brain server, and tells the model which computers really exist 2026-11-05 16:03:34 +08:00
王性驊 a8265409a1 Docker calls time out; the box has a VNC key, limits and sandboxed files; X11/Wayland eyes and hands stop leaking 2026-11-05 15:48:22 +08:00
王性驊 8615881634 Screens: a brain without a screen of its own uses its hands' screen
computer_use_* name the computer whenever the screen is another computer's, and with no
computer named the step goes to a screen that is really there (the bot's desktop, this
computer's, then a 手腳 computer) instead of a brain's missing one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-05 08:05:29 +08:00
王性驊 c248387f00 Add agent-desktop text mode for computer use
Read apps as numbered element lists (computer_use_read_app), act by ref
(click/typing/select/scroll/press_key/open_app) via agent-desktop v1.1.0.
Verified on macOS (Calculator, Finder) and headless Linux (Xvfb+openbox+AT-SPI).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 17:19:59 +08:00
王性驊 d7ee3f5db9 AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 16:51:51 +08:00