Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
|
|
|
#!/bin/bash
|
|
|
|
|
# The one way to start the box's browser (adapted from XBot's box/box-chrome): the same profile
|
|
|
|
|
# and DevTools port every time, so AnyBot's browser tools and the person in the viewer use the
|
|
|
|
|
# same window and logins; web pages exposed to accessibility, so their links and buttons get
|
|
|
|
|
# numbers on the bot's picture. A second start only hands its address to the running one.
|
|
|
|
|
set -euo pipefail
|
|
|
|
|
export DISPLAY="${DISPLAY:-:1}" HOME=/home/bot
|
|
|
|
|
export GTK_MODULES=atk-bridge GTK_A11Y=atspi GNOME_ACCESSIBILITY=1 NO_AT_BRIDGE=0
|
|
|
|
|
if [[ -z "${DBUS_SESSION_BUS_ADDRESS:-}" && -r /tmp/anybot-dbus.env ]]; then
|
|
|
|
|
# shellcheck disable=SC1091
|
|
|
|
|
source /tmp/anybot-dbus.env
|
|
|
|
|
export DBUS_SESSION_BUS_ADDRESS
|
|
|
|
|
fi
|
2026-11-05 17:03:59 +00:00
|
|
|
# One profile. The box used to have two: Chromium's own (~/.config/chromium, what its menu entry
|
|
|
|
|
# opened, where a person logged in by hand) and AnyBot's (~/.config/anybot-browser, what the
|
|
|
|
|
# browser tools opened), so a site could be logged in in one and not the other. Chromium's own
|
|
|
|
|
# is the one now (the other stays as it was); a box that only has AnyBot's keeps that.
|
|
|
|
|
PROFILE=/home/bot/.config/anybot-profile
|
|
|
|
|
if [[ ! -e "$PROFILE" ]]; then
|
|
|
|
|
if [[ -d /home/bot/.config/chromium/Default || ! -d /home/bot/.config/anybot-browser/Default ]]; then
|
|
|
|
|
mkdir -p /home/bot/.config/chromium && ln -s /home/bot/.config/chromium "$PROFILE"
|
|
|
|
|
else
|
|
|
|
|
ln -s /home/bot/.config/anybot-browser "$PROFILE"
|
|
|
|
|
fi
|
|
|
|
|
fi
|
Bot 的電腦 can run under gVisor: its own kernel, Chromium's sandbox back on
容器隔離 in 設定 › 電腦: standard (runc, the container is the only wall, Chromium runs
--no-sandbox) or hardened (runsc: a virtualized kernel, all capabilities dropped but the few
the desktop needs, ANYBOT_BOX_SANDBOX=1 turns Chromium's own sandbox back on). Box image v7.
deploy/mac/gvisor-vm.sh makes a Lima VM with Docker + gVisor for a Mac (started at login),
gvisor-migrate.sh moves the box's files and logins there.
Checked and finished before committing:
- the hardened choice is offered only on a Docker host that has runsc; a box set to it on a
host without it opens the standard way and says so (it used to not start at all)
- the settings row shows what the box really runs on, and when 更新 is needed
- no swap only under gVisor (a standard box keeps Docker's default)
- the migrate script sees the running Mac app ("AnyBot", not "anybot")
- the VM script starts an existing VM by name, checks runsc and the kernel from the Mac
- the version test says 7
Real run in the Lima VM: desktop, VNC, numbered controls, xdotool and screenshots work; no
Chromium process has --no-sandbox, renderers run with seccomp and their own namespaces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-06 16:03:25 +00:00
|
|
|
# The box (a container) is the sandbox by default, so Chromium runs --no-sandbox. Under gVisor
|
|
|
|
|
# (容器隔離=runsc) the box has a virtualized kernel of its own and AnyBot starts it with
|
|
|
|
|
# ANYBOT_BOX_SANDBOX=1 — Chromium's own sandbox works there, so keep it as a second wall.
|
|
|
|
|
SANDBOX=(--no-sandbox --test-type)
|
|
|
|
|
[ "${ANYBOT_BOX_SANDBOX:-0}" = 1 ] && SANDBOX=()
|
|
|
|
|
FLAGS=("${SANDBOX[@]}" --user-data-dir="$PROFILE" --remote-debugging-address=127.0.0.1 --remote-debugging-port=9222
|
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
|
|
|
--remote-allow-origins=* --force-renderer-accessibility --disable-blink-features=AutomationControlled
|
|
|
|
|
--disable-dev-shm-usage --disable-gpu --use-angle=swiftshader --enable-unsafe-swiftshader --password-store=basic
|
|
|
|
|
--no-first-run --no-default-browser-check --disable-session-crashed-bubble --hide-crash-restore-bubble
|
|
|
|
|
--disable-features=TranslateUI --start-maximized)
|
|
|
|
|
# Two starts in the same second (the bot and the person) would both find no browser and fight
|
|
|
|
|
# over the profile ("in use"): one at a time until the first is listening.
|
|
|
|
|
exec 9>/tmp/.anybot-browser.lock
|
|
|
|
|
flock -w 30 9 || true
|
2026-11-05 17:03:59 +00:00
|
|
|
if pgrep -u bot -f -- "--user-data-dir=$PROFILE" >/dev/null || pgrep -u bot -f -- "--user-data-dir=$(readlink -f "$PROFILE")" >/dev/null; then
|
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
|
|
|
flock -u 9
|
2026-11-05 17:03:59 +00:00
|
|
|
exec chromium "${FLAGS[@]}" "$@" 9>&-
|
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
|
|
|
fi
|
|
|
|
|
# A profile from before this box (a new container) still names the old one in its lock.
|
2026-11-05 17:03:59 +00:00
|
|
|
rm -f "$PROFILE"/SingletonLock "$PROFILE"/SingletonSocket "$PROFILE"/SingletonCookie
|
|
|
|
|
# (The lock's fd is not handed to the browser: it would hold the lock for its whole life, and the
|
|
|
|
|
# next start would wait on it.)
|
|
|
|
|
setsid chromium "${FLAGS[@]}" "${@:-about:blank}" >/tmp/anybot-browser.log 2>&1 < /dev/null 9>&- &
|
|
|
|
|
for _ in $(seq 1 100); do
|
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
|
|
|
python3 -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:9222/json/version', timeout=1).close()" 2>/dev/null && break
|
|
|
|
|
sleep 0.25
|
|
|
|
|
done
|