daniel.w
/
AnyBot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
AnyBot/Cargo.toml

91 lines
4.3 KiB
TOML
Raw Normal View History

AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
[package]
name = "anybot"
version = "0.1.0"
Apache-2.0: license file, manifest fields, README section
2026-11-09 08:36:36 +00:00
license = "Apache-2.0"
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
edition = "2021"
One-step install: install.sh, fixed Linux and server builds, clearer README - install.sh (curl | sh): app, hands, server and deps modes; installs system packages (right names per release, e.g. libasound2t64 on Ubuntu 24.04), Rust via rustup (1.95+), Docker for the server; limits parallel compiles on small machines - Server Docker image: add the ALSA build/runtime packages (the build failed), cache the cargo registry and target between rebuilds, JOBS build arg - 加一台電腦 now gives a line that installs AnyBot on a blank computer before joining it - Makefile: make deps installs Rust too, make doctor, clear message when Rust is missing/old - README rewritten: one line per use, getting started, update/remove, troubleshooting, deployment options; docs/install.md and multi-device.md follow it - Tested on fresh ubuntu:24.04 (app) and debian:12 (curl | sh hands) containers Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 16:52:10 +00:00
# Older Rust stops with a clear message instead of odd errors deep in a dependency.
rust-version = "1.95"
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
[[bin]]
name = "anybot"
path = "src/main.rs"
[dependencies]
Replies in Traditional Chinese when that is the setting; a cloud brain's browser is the person's - 語言 繁體中文: a reply's Simplified characters are turned into Traditional with Taiwan wording (zhconv), since some models (Qwen) slip despite the instruction. - A brain on a server: browser steps go to the person's computer (its hands) unless the server's own browser is named; 你的電腦 is listed first. - update_bot says the default computer it set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 06:46:42 +00:00
zhconv = "0.4"
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
eframe = "0.37"
ureq = { version = "3", features = ["json"] }
Bug sweep: gates, relay 2FA, Windows input, snapshots, encodings Permissions and reach - a file leaving a computer passes the same questions as reading it; "looking" shell commands that do more than look ask first - a symlink out of the work/sealed folder counts as outside it (real paths, not just lexical ones); the search walk skips links - web_fetch and skill_install read the public web only: localhost, the LAN and cloud metadata are refused, at every redirect - preview shows a file without running it; browser_upload may ask - vault: logins no longer match across shared suffixes (pages.dev, com.tw, ...); registrable_domain knows public suffixes - axwrite: found elements are retained (use-after-free) Relay - replacing or turning off 2FA needs the current code (rate-limited); new recovery codes only count once enrollment is verified - a removed computer loses its socket and its `call` access - a site name taken by another account answers 409 - RELAY_TRUST_PROXY: the login lock counts the caller's address, not the proxy's (compose files and README) - httpd: half a body is not a request Desktop - sign-in, join and restore work with 2FA on (code field; --serve reads ANYBOT_ACCOUNT_CODE; the setup wizard points to a pairing code) - Windows: text was typed twice; modifiers are held through click, drag and scroll; horizontal scroll; screen control is no longer refused - a recording that hits the limit is transcribed, not thrown away - bot's computer: a new home volume restores the snapshot; a snapshot is read through before the home is wiped; a failed volume removal stops reset/restore - events of a stopped turn no longer land in the next one; each channel message gets a turn of its own - saving a bot's settings keeps what it got meanwhile (three-way merge); a save file that cannot be read is kept - pages in Big5/GBK/Shift_JIS are read in their own encoding (encoding_rs); osascript output past 64 KB no longer stalls - pick_tab keeps a job's own non-http tab; pictures larger than the texture limit are fitted; the auto-update time field takes typing - skills live in <data>/skills; memories are unnumbered in the prompt; the phone may skillify and skill_install CI - the test step fails when the tests fail (it piped into tail) Tests: anybot 268 passed, 27 ignored; relay 16 passed; clippy unchanged. Not run: Windows on a real machine (win.rs type-checked alone), the Docker paths of the bot's computer, the live accessibility writes. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-11-10 03:46:17 +00:00
# Web pages that are not UTF-8 (Big5, GBK, Shift_JIS…), read as what they say they are.
encoding_rs = "0.8"
Lasting connection to the account server, per-item sync, files and models via the brain - Each AnyBot keeps one WebSocket to the account server (pinged, reopened within a second when cut). Requests come down it at once; calls to other AnyBots of the account (brain sync, hands) go up it. Frames carry raw bytes, compressed when that saves space. HTTP long-poll stays as fallback. - The account server knows at once when an AnyBot goes away. - Edits to lists and maps go item by item, so two computers editing one list keep both changes. - Screens fetch chat files from the brain, get model lists through the brain's keys, and show the brain's computer as the brain's. - Moving this computer's bots skips starter bots the brain has and bots already moved. - The phone goes to the brain instead of a computer that shows it. - deploy/linux/hands.sh: a Linux computer (with or without a screen) as hands, kept running by systemd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 11:30:32 +00:00
# 長連線 to the account server (WebSocket over the same TLS ureq uses), and compressing what goes over it.
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12", "logging"] }
HTTPS on the local network's own IP, with a certificate the server signs itself setup.sh offers 「區網 IP」 (alone, or next to Tailscale): Caddy serves https://<LAN IP> with its own small certificate authority (tls internal, default_sni for clients that send no name) and hands out the root at http://<LAN IP>/anybot-ca.crt for each device to trust once. docker-compose.lan.yml binds Caddy to that address only, so it sits beside tailscale serve. The app now accepts, for the account server, the certificates this computer trusts as well as the built-in public ones (HTTP and the WebSocket alike), so it connects once the root is trusted; before that it says what to install instead of "the certificate is wrong". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-07 13:27:07 +00:00
# Which certificates the account server may show: the public ones (built in), plus the ones this
# computer trusts itself, so a server on the local network that signs its own (https://10.0.0.25)
# works once this computer trusts it, the same as for the browser.
webpki-root-certs = "1"
rustls-native-certs = "0.8"
AnyBot's browser drives Firefox too, the person's own Firefox profiles included Firefox dropped Chrome's DevTools protocol, so bidi.rs speaks WebDriver BiDi and does each DevTools command the browser tools use the BiDi way (real mouse and keys, typing, screenshots, PDF, dialogs, uploads, downloads, tabs). The 18 tools themselves are unchanged and pass the same dashboard test on Firefox as on Chrome. - A computer with only Firefox runs AnyBot's own profiles in it (under the Snap/Flatpak corner of the home folder when Firefox is packaged that way). - The person's own Firefox profiles are choices too, by the names Firefox shows (profiles.ini and the newer profile switcher's SQLite list): 「阿金（Firefox）」, or just 「阿金」. Firefox can't be taken over once open, so a profile open the everyday way is refused with "close it first"; AnyBot then opens it itself, logins intact. - Settings lists them without a remove button; the messages name Firefox as an option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha2RVfSYmrhqNnqx9Z4
2026-11-07 16:55:17 +00:00
# Firefox keeps the names of its newer profiles (「女生」「阿金」…) only in a small SQLite file
# (Profile Groups/*.sqlite); bundled, so no system SQLite is needed.
rusqlite = { version = "0.41", features = ["bundled"] }
Lasting connection to the account server, per-item sync, files and models via the brain - Each AnyBot keeps one WebSocket to the account server (pinged, reopened within a second when cut). Requests come down it at once; calls to other AnyBots of the account (brain sync, hands) go up it. Frames carry raw bytes, compressed when that saves space. HTTP long-poll stays as fallback. - The account server knows at once when an AnyBot goes away. - Edits to lists and maps go item by item, so two computers editing one list keep both changes. - Screens fetch chat files from the brain, get model lists through the brain's keys, and show the brain's computer as the brain's. - Moving this computer's bots skips starter bots the brain has and bots already moved. - The phone goes to the brain instead of a computer that shows it. - deploy/linux/hands.sh: a Linux computer (with or without a screen) as hands, kept running by systemd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 11:30:32 +00:00
flate2 = "1"
anybot setup and the server setup: guided setup with a brain picker - anybot setup: a full-screen terminal guide (role, name, account server, account, which brain) that saves the settings, installs a systemd user service (with a virtual screen when there is none), and checks that the account server and the brain see the computer. Flags for scripts. - Hands can be limited to one brain; the account server says which AnyBot asks. Choosing a brain in the app does the same. - deploy/server/setup.sh: domain, account, sign-ups and keys step by step; writes .env and Caddyfile, starts everything and checks HTTPS. The server's brain creates the account on first start. - deploy/linux/hands.sh installs what's needed and opens anybot setup (no password kept in a file any more). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 12:38:45 +00:00
# `anybot setup`: the guided setup in a terminal (servers and Linux computers without a window).
crossterm = "0.29"
unicode-width = "0.2"
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
serde = { version = "1", features = ["derive"] }
serde_json = "1"
dirs = "7"
image = { version = "0.25", default-features = false, features = ["png", "jpeg", "gif", "webp", "bmp", "tiff", "ico"] }
base64 = "0.22"
People watch and help on the bots' computer through VNC (noVNC), carried by the account server The core: the bot works the computer; a person watches it like a colleague, and takes over when it needs them. VNC is made for that (only what changed, every mouse move and key, touch on phones), so the pictures sent one by one go, and with them their own input path. - vnc.rs: a VNC proxy to the box's x11vnc through one docker exec (a Python bridge that ends with either side). It signs in to the box with the box's own password (its VNC_PASSWORD) and offers the viewer no password, so the password never leaves the brain. The person's clicks and keys make the bot's steps wait until 8 s after they stop; text typed in the viewer's box (any language) is pasted in the box. - Account server: /screen (the viewer page) and /novnc.js (noVNC 1.6.0, MPL-2.0, bundled); /relay/vnc/<host> takes the browser's WebSocket (account session, or a 10-minute ticket the app asks for with POST /relay/screen-ticket; owners and admins) and carries it over that AnyBot's lasting connection (vopen / vdata / vtext / vclose). Both ends of that connection wait 2 ms instead of 40 ms to read while things move: a request and its answer took up to 160 ms more than needed. - Phone page: 看 Bot 的電腦 and 接管 open the viewer page (also on the local network: /api/vnc on the AnyBot's own phone server). This Mac's own screen moved to the ⋯ menu. - Mac app: 看畫面 opens the viewer in the browser (a screen of a brain: the brain's computer by ticket; else this computer's box, noVNC straight from it). The in-app picture viewer, the live JPEG stream, the person's click/move/scroll/key endpoints and the input shell are gone. - Opening a Mac app uses LaunchServices (`open -a`): agent-desktop's launch waited on the window through accessibility and failed without 輔助使用 now and then. Measured on this Mac (made-up account, local relay + headless brain + the real box): viewer signed in in 0.4–0.7 s, a request and its answer through relay + brain + box 25–58 ms median; headless Chrome shows the desktop live in /screen. remote_e2e 28/28 ×2, 149 unit tests, relay 11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 15:05:44 +00:00
# VNC sign-in (the 16-byte challenge) for people watching the bots' computer.
des = "0.8"
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
chrono = "0.4"
chrono-tz = "0.10"
iana-time-zone = "0.1"
ring = "0.17"
tungstenite = { version = "0.30", default-features = false, features = ["handshake"] }
calamine = "0.38.0"
resvg = "0.49.0"
hayro = "0.9"
zip = { version = "9", default-features = false, features = ["deflate"] }
rodio = { version = "0.23", default-features = false, features = ["playback", "flac", "mp3", "mp4", "vorbis", "wav", "symphonia-aiff", "symphonia-alac", "symphonia-caf"] }
pdf-extract = "0.12"
unicode-normalization = "0.1"
cpal = "0.18"
mermaid-rs-renderer = { version = "0.3", default-features = false }
pulldown-cmark = { version = "0.13", default-features = false }
Add agent-desktop text mode for computer use Read apps as numbered element lists (computer_use_read_app), act by ref (click/typing/select/scroll/press_key/open_app) via agent-desktop v1.1.0. Verified on macOS (Calculator, Finder) and headless Linux (Xvfb+openbox+AT-SPI). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 09:19:59 +00:00
# Seeing and using other apps through the OS accessibility APIs (AX on macOS, AT-SPI on Linux,
# UI Automation on Windows): numbered UI elements a bot can act on without screenshots.
# vercel-labs/agent-desktop, Apache-2.0, pinned to a release.
agent-desktop-core = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" }
Relay 2FA: TOTP + recovery codes, argon2id, one-page logout (S2) - login with 2FA on returns a 5-minute challenge, not a session; the session comes from /relay/2fa/verify with an authenticator code — or one of eight recovery codes, each good once (hash slot emptied) - /relay/2fa/enroll hands out the secret (QR) + codes once; enabling completes only when a real code proves the app has the secret; disable proves the code - /relay/sessions/logout kicks every device of the account from one page - password hashes are argon2id (memory-hard, OWASP parameters); stored PBKDF2 hashes still sign in — no forced reset, upgrade on next password change - TOTP is RFC 6238 (vectors pinned in tests); mobile page prompts for the second step; protocol.md documents the endpoints - fixes a double-lock in /relay/me found by the flow test (handlers hold one data guard: read through it)
2026-11-09 09:52:42 +00:00
argon2 = "0.5"
Add agent-desktop text mode for computer use Read apps as numbered element lists (computer_use_read_app), act by ref (click/typing/select/scroll/press_key/open_app) via agent-desktop v1.1.0. Verified on macOS (Calculator, Finder) and headless Linux (Xvfb+openbox+AT-SPI). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 09:19:59 +00:00
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
[profile.release]
opt-level = 2
[target.'cfg(not(target_os = "macos"))'.dependencies]
[target.'cfg(target_os = "linux")'.dependencies]
Add agent-desktop text mode for computer use Read apps as numbered element lists (computer_use_read_app), act by ref (click/typing/select/scroll/press_key/open_app) via agent-desktop v1.1.0. Verified on macOS (Calculator, Finder) and headless Linux (Xvfb+openbox+AT-SPI). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 09:19:59 +00:00
agent-desktop-linux = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" }
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
x11rb = { version = "0.13", features = ["xtest", "screensaver", "record"] }
# Wayland: wlroots desktops (sway, Hyprland…) through their own protocols; GNOME and KDE through
# the desktop portals (one "allow" dialog, remembered).
wayland-client = "0.31"
Docker calls time out; the box has a VNC key, limits and sandboxed files; X11/Wayland eyes and hands stop leaking
2026-11-05 07:48:22 +00:00
# (wl.rs: SO_RCVTIMEO on the compositor socket, so a frozen compositor can't freeze the app)
libc = "0.2"
AnyBot: first commit of the working desktop app Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama, custom), tools and approvals, the bot's Docker computer, computer use on macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based toolset, the encrypted vault, pinned grid and sidebar sections, Markdown with Mermaid diagrams, phone access through the relay, and the Linux build and test scripts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
wayland-protocols = { version = "0.32", features = ["client", "staging", "unstable"] }
wayland-protocols-wlr = { version = "0.3", features = ["client"] }
wayland-protocols-misc = { version = "0.3", features = ["client"] }
memmap2 = "0.9"
ashpd = { version = "0.13", default-features = false, features = ["async-io", "remote_desktop", "screencast", "screenshot"] }
zbus = { version = "5", default-features = false, features = ["async-io", "blocking-api"] }
futures-lite = "2"
async-io = "2"
Add agent-desktop text mode for computer use Read apps as numbered element lists (computer_use_read_app), act by ref (click/typing/select/scroll/press_key/open_app) via agent-desktop v1.1.0. Verified on macOS (Calculator, Finder) and headless Linux (Xvfb+openbox+AT-SPI). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 09:19:59 +00:00
[target.'cfg(target_os = "macos")'.dependencies]
agent-desktop-macos = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" }
One way to act on a screen: a real click or keystroke, wherever the bot looked - Acting by an app's number (read_app's e7) is now a real click at that control's spot, the same path as screenshot coordinates. agent-desktop only reads; its own actions (a second set with its own key names and failures: no typing in terminals, "/" and "`" failing off the main thread) are gone, with computer_use_select, background clicks and press_key's app. - Every action comes back with the screen after it (or, with no picture allowed, the front app's numbered controls), so the bot no longer spends a step on a screenshot to check. - Coordinates map without remembering a screenshot: the picture's size depends only on the screen. The phone's taps go straight to the screen as the person's, in the pixels of the picture it was sent; they used to borrow the bot's last screenshot scale and landed 1.25x off on the bots' computer, or failed with "screenshot first". - Mac typing pastes (⌘V, clipboard put back): with 注音 on, text sent as key events vanished while return arrived, so commands typed into a terminal ran empty. Bare printing keys ("slash", "r") are typed the same way; keys like "slash" and "grave" have names. - The bots' Docker computer: one docker exec per step (action, settle, picture), never `mousemove --sync` (it waits forever when the pointer is already there: a second click on the same spot hung two minutes), open_app (chromium = the browser browser_* drive, xterm). Tests: box_screen_is_exact (real Docker: bot and phone coordinates exact, same spot twice in under 2 s, ASCII + Chinese + named key typed), Calculator by numbers and Terminal typing under 注音 through the signed app, remote_e2e 28/28, Linux build + 151 tests + x11_live. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 13:51:19 +00:00
# Typing by pasting (any input method, any language); already in the build through egui.
arboard = { version = "3.7", default-features = false }
Add agent-desktop text mode for computer use Read apps as numbered element lists (computer_use_read_app), act by ref (click/typing/select/scroll/press_key/open_app) via agent-desktop v1.1.0. Verified on macOS (Calculator, Finder) and headless Linux (Xvfb+openbox+AT-SPI). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 09:19:59 +00:00
[target.'cfg(target_os = "windows")'.dependencies]
agent-desktop-windows = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" }
Windows has hands now (C9, engine) - src/win.rs: the same mac:: API the harness drives, on Win32 — SendInput mouse (absolute, multi-click, drag, wheel), keyboard (macOS key codes mapped to VK, unicode text, modifiers), GDI BitBlt screenshot, WindowFromPoint/GetForegroundWindow for window_at and front_app; compiles clean against the windows crate for x86_64-pc-windows-msvc (API-checked, no hardware here) - the control tree was already speaking agent-desktop-windows - intents answer honestly on Windows ("捷徑目前是 macOS/Linux 的") and don't advertise themselves from the facts line - CI: a report-only Windows compile job for wherever a Windows runner lives - needs a real machine: multi-monitor coordinates, high-DPI, UIA-number clicks, Chinese input
2026-11-09 12:50:44 +00:00
windows = { version = "0.61", features = ["Win32_Foundation", "Win32_Graphics_Gdi", "Win32_UI_Input_KeyboardAndMouse", "Win32_UI_WindowsAndMessaging", "Win32_System_Threading"] }