AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
mod agent;
|
|
|
|
|
mod app;
|
|
|
|
|
mod appctl;
|
|
|
|
|
mod attach;
|
|
|
|
|
mod axread;
|
|
|
|
|
mod avatar;
|
|
|
|
|
mod botbrowser;
|
|
|
|
|
mod browser;
|
|
|
|
|
mod cloudbox;
|
|
|
|
|
mod channels;
|
|
|
|
|
mod computer;
|
2026-11-04 09:19:59 +00:00
|
|
|
mod desktop;
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
mod diagram;
|
|
|
|
|
mod platform;
|
|
|
|
|
mod mic;
|
頁面: pages a bot makes and keeps up to date (like the original's tracking pages)
- write_page / read_page / list_pages: Markdown pages with versions (last 10 kept), per bot;
the reply gets a card (title, fading preview, page icon, 頁面 · 更新時間, 最近更新), and
[標題](page:ID) links open the page
- The page opens large: summary, what changed, the version picker, 複製 / 存成檔案 / 刪除;
listed under 資料庫 in the right panel
- Page blocks for richer pages, in the chat's Markdown too: ```stats tiles, ```steps progress,
> [!NOTE]/[!TIP]/[!WARNING]/[!IMPORTANT] call-outs (with tables and Mermaid charts)
- Phone: the card in chats, page links, a page view with the same blocks; charts drawn by the
AnyBot as SVG (GET /api/pages/<id>[/chart/<n>]); members may read pages
- Bots are told to make a page for things to track and to update it from a routine only when
there is real news
- Screenshot views: ANYBOT_SHOT_VIEW=pagecard / page
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 17:41:41 +00:00
|
|
|
mod pages;
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
#[cfg(target_os = "linux")]
|
|
|
|
|
mod x11;
|
|
|
|
|
#[cfg(target_os = "linux")]
|
|
|
|
|
mod wl;
|
|
|
|
|
#[cfg(target_os = "linux")]
|
|
|
|
|
mod portal;
|
|
|
|
|
#[cfg(target_os = "linux")]
|
|
|
|
|
mod desk;
|
|
|
|
|
mod egress;
|
|
|
|
|
mod httpd;
|
|
|
|
|
mod i18n;
|
|
|
|
|
mod i18n_table;
|
|
|
|
|
mod icons;
|
|
|
|
|
mod macapps;
|
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
|
|
|
mod marks;
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
mod mcp;
|
|
|
|
|
mod memory;
|
|
|
|
|
mod model;
|
|
|
|
|
mod notify;
|
|
|
|
|
mod phone;
|
|
|
|
|
mod policy;
|
|
|
|
|
mod preview;
|
|
|
|
|
mod providers;
|
|
|
|
|
mod remote;
|
|
|
|
|
mod routines;
|
2026-11-04 12:38:45 +00:00
|
|
|
mod setup;
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
mod teach;
|
|
|
|
|
mod share;
|
|
|
|
|
mod teams;
|
|
|
|
|
mod theme;
|
|
|
|
|
mod tools;
|
|
|
|
|
mod triggers;
|
|
|
|
|
mod vault;
|
|
|
|
|
mod vault_store;
|
People watch and help on the bots' computer through VNC (noVNC), carried by the account server
The core: the bot works the computer; a person watches it like a colleague, and takes over
when it needs them. VNC is made for that (only what changed, every mouse move and key, touch on
phones), so the pictures sent one by one go, and with them their own input path.
- vnc.rs: a VNC proxy to the box's x11vnc through one docker exec (a Python bridge that ends
with either side). It signs in to the box with the box's own password (its VNC_PASSWORD) and
offers the viewer no password, so the password never leaves the brain. The person's clicks
and keys make the bot's steps wait until 8 s after they stop; text typed in the viewer's box
(any language) is pasted in the box.
- Account server: /screen (the viewer page) and /novnc.js (noVNC 1.6.0, MPL-2.0, bundled);
/relay/vnc/<host> takes the browser's WebSocket (account session, or a 10-minute ticket the
app asks for with POST /relay/screen-ticket; owners and admins) and carries it over that
AnyBot's lasting connection (vopen / vdata / vtext / vclose). Both ends of that connection
wait 2 ms instead of 40 ms to read while things move: a request and its answer took up to
160 ms more than needed.
- Phone page: 看 Bot 的電腦 and 接管 open the viewer page (also on the local network: /api/vnc on
the AnyBot's own phone server). This Mac's own screen moved to the ⋯ menu.
- Mac app: 看畫面 opens the viewer in the browser (a screen of a brain: the brain's computer by
ticket; else this computer's box, noVNC straight from it). The in-app picture viewer, the
live JPEG stream, the person's click/move/scroll/key endpoints and the input shell are gone.
- Opening a Mac app uses LaunchServices (`open -a`): agent-desktop's launch waited on the window
through accessibility and failed without 輔助使用 now and then.
Measured on this Mac (made-up account, local relay + headless brain + the real box): viewer
signed in in 0.4–0.7 s, a request and its answer through relay + brain + box 25–58 ms median;
headless Chrome shows the desktop live in /screen. remote_e2e 28/28 ×2, 149 unit tests, relay 11.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 15:05:44 +00:00
|
|
|
mod vnc;
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
mod voice;
|
2026-11-04 11:30:32 +00:00
|
|
|
mod tunnel;
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
mod web;
|
2026-11-04 11:30:32 +00:00
|
|
|
mod wire;
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
|
|
|
|
|
use eframe::egui;
|
|
|
|
|
use std::path::PathBuf;
|
|
|
|
|
|
|
|
|
|
/// PingFang ships as a downloadable system asset on macOS; its folder name varies per Mac.
|
|
|
|
|
fn pingfang_path() -> Option<PathBuf> {
|
|
|
|
|
let root = std::path::Path::new("/System/Library/AssetsV2");
|
|
|
|
|
for set in std::fs::read_dir(root).ok()?.flatten() {
|
|
|
|
|
if !set.file_name().to_string_lossy().starts_with("com_apple_MobileAsset_Font") {
|
|
|
|
|
continue;
|
|
|
|
|
}
|
|
|
|
|
for asset in std::fs::read_dir(set.path()).into_iter().flatten().flatten() {
|
|
|
|
|
let p = asset.path().join("AssetData/PingFang.ttc");
|
|
|
|
|
if p.exists() {
|
|
|
|
|
return Some(p);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
None
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// PingFang TC Regular inside PingFang.ttc.
|
|
|
|
|
const PINGFANG_TC_INDEX: u32 = 2;
|
|
|
|
|
|
|
|
|
|
/// The window's first size (screenshot mode can ask for another, e.g. ANYBOT_SHOT_SIZE=760x560).
|
|
|
|
|
fn shot_size() -> [f32; 2] {
|
|
|
|
|
std::env::var("ANYBOT_SHOT_SIZE").ok().and_then(|v| {
|
|
|
|
|
let (w, h) = v.split_once('x')?;
|
|
|
|
|
Some([w.parse().ok()?, h.parse().ok()?])
|
|
|
|
|
}).unwrap_or([1180.0, 780.0])
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// 粉圓體, bundled.
|
|
|
|
|
pub const HUNINN: &[u8] = include_bytes!("../fonts/jf-openhuninn-2.1.ttf");
|
|
|
|
|
|
|
|
|
|
/// "huninn" (粉圓體, the default) or "system".
|
|
|
|
|
pub fn font_choice() -> String {
|
|
|
|
|
model::State::path().parent().and_then(|d| std::fs::read_to_string(d.join("font.txt")).ok()).map(|s| s.trim().to_string()).filter(|s| !s.is_empty()).unwrap_or_else(|| "huninn".into())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pub fn load_fonts(ctx: &egui::Context) {
|
|
|
|
|
let mut fonts = egui::FontDefinitions::default();
|
|
|
|
|
// One font for Chinese *and* English: mixing egui's built-in Latin font with a Chinese
|
|
|
|
|
// fallback put the two on different baselines and sizes ("Bot 設定" looked broken).
|
|
|
|
|
// So a CJK font that has good Latin letters goes first; egui's own fonts stay behind it
|
|
|
|
|
// only for symbols and emoji it lacks.
|
|
|
|
|
let mut candidates: Vec<(PathBuf, u32)> = pingfang_path().map(|p| vec![(p, PINGFANG_TC_INDEX)]).unwrap_or_default();
|
|
|
|
|
candidates.extend(
|
|
|
|
|
[
|
|
|
|
|
("/System/Library/Fonts/Hiragino Sans GB.ttc", 0),
|
|
|
|
|
("/System/Library/Fonts/STHeiti Medium.ttc", 0),
|
|
|
|
|
("/System/Library/Fonts/Supplemental/Arial Unicode.ttf", 0),
|
|
|
|
|
("C:\\Windows\\Fonts\\msjh.ttc", 0),
|
|
|
|
|
("C:\\Windows\\Fonts\\msyh.ttc", 0),
|
|
|
|
|
// Linux: Noto Sans CJK (face 3 = Traditional Chinese) on Debian/Ubuntu, Arch, Fedora; then WenQuanYi.
|
|
|
|
|
("/usr/share/fonts/opentype/noto/NotoSansCJK-Regular.ttc", 3),
|
|
|
|
|
("/usr/share/fonts/noto-cjk/NotoSansCJK-Regular.ttc", 3),
|
|
|
|
|
("/usr/share/fonts/google-noto-cjk/NotoSansCJK-Regular.ttc", 3),
|
|
|
|
|
("/usr/share/fonts/truetype/wqy/wqy-microhei.ttc", 0),
|
|
|
|
|
]
|
|
|
|
|
.into_iter()
|
|
|
|
|
.map(|(p, i)| (PathBuf::from(p), i)),
|
|
|
|
|
);
|
|
|
|
|
// Code: the original's `--cursor-font-family-mono` list is ui-monospace, SFMono-Regular, Menlo…
|
|
|
|
|
for path in ["/System/Library/Fonts/SFNSMono.ttf", "/System/Library/Fonts/Menlo.ttc"] {
|
|
|
|
|
if let Ok(bytes) = std::fs::read(path) {
|
|
|
|
|
fonts.font_data.insert("mono".into(), egui::FontData::from_owned(bytes).into());
|
|
|
|
|
fonts.families.entry(egui::FontFamily::Monospace).or_default().insert(0, "mono".into());
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
for (path, index) in candidates {
|
|
|
|
|
if let Ok(bytes) = std::fs::read(&path) {
|
|
|
|
|
let mut data = egui::FontData::from_owned(bytes);
|
|
|
|
|
data.index = index;
|
|
|
|
|
fonts.font_data.insert("cjk".into(), data.into());
|
|
|
|
|
fonts.families.entry(egui::FontFamily::Proportional).or_default().insert(0, "cjk".into());
|
|
|
|
|
// Code stays in the monospace font; Chinese inside code falls back to the same font.
|
|
|
|
|
fonts.families.entry(egui::FontFamily::Monospace).or_default().push("cjk".into());
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
// 粉圓體 (jf open 粉圓, SIL OFL 1.1, fonts/OFL.txt) goes first everywhere it has the letter;
|
|
|
|
|
// the system font above fills in what it lacks (some simplified characters, symbols).
|
|
|
|
|
if font_choice() == "huninn" {
|
|
|
|
|
fonts.font_data.insert("huninn".into(), egui::FontData::from_static(HUNINN).into());
|
|
|
|
|
fonts.families.entry(egui::FontFamily::Proportional).or_default().insert(0, "huninn".into());
|
|
|
|
|
let mono = fonts.families.entry(egui::FontFamily::Monospace).or_default();
|
|
|
|
|
let at = mono.iter().position(|f| f == "cjk").unwrap_or(mono.len());
|
|
|
|
|
mono.insert(at, "huninn".into());
|
|
|
|
|
}
|
|
|
|
|
ctx.set_fonts(fonts);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// The window icon (Linux taskbars and window switchers; macOS uses the app bundle's).
|
|
|
|
|
fn app_icon() -> egui::IconData {
|
2026-11-05 11:20:29 +00:00
|
|
|
svg_icon(include_str!("../icons/anybot.svg"), 256)
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// An SVG drawn to a square RGBA picture of `px` pixels.
|
|
|
|
|
fn svg_icon(svg: &str, px: u32) -> egui::IconData {
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
let tree = resvg::usvg::Tree::from_str(svg, &resvg::usvg::Options::default()).expect("icon svg");
|
|
|
|
|
let mut pm = resvg::tiny_skia::Pixmap::new(px, px).expect("icon pixmap");
|
|
|
|
|
let s = px as f32 / tree.size().width();
|
|
|
|
|
resvg::render(&tree, resvg::tiny_skia::Transform::from_scale(s, s), &mut pm.as_mut());
|
|
|
|
|
// tiny-skia is premultiplied; IconData wants straight alpha.
|
|
|
|
|
let mut rgba = pm.take();
|
|
|
|
|
for c in rgba.chunks_exact_mut(4) {
|
|
|
|
|
let a = c[3] as u32;
|
|
|
|
|
if a > 0 && a < 255 {
|
|
|
|
|
for v in &mut c[..3] {
|
|
|
|
|
*v = ((*v as u32 * 255 + a / 2) / a).min(255) as u8;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
egui::IconData { rgba, width: px, height: px }
|
|
|
|
|
}
|
|
|
|
|
|
2026-11-05 11:20:29 +00:00
|
|
|
/// `anybot --write-icon out.png [size] [file.svg]`: the app icon (or that SVG) as a PNG, drawn at
|
|
|
|
|
/// that size (for the Mac bundle, Linux packages and the web icons).
|
|
|
|
|
fn write_icon(out: &str, px: u32, svg: Option<&str>) -> Result<(), String> {
|
|
|
|
|
let icon = match svg {
|
|
|
|
|
Some(path) => svg_icon(&std::fs::read_to_string(path).map_err(|e| format!("{path}: {e}"))?, px),
|
|
|
|
|
None => svg_icon(include_str!("../icons/anybot.svg"), px),
|
|
|
|
|
};
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
let img = image::RgbaImage::from_raw(icon.width, icon.height, icon.rgba).ok_or("icon")?;
|
|
|
|
|
img.save(out).map_err(|e| e.to_string())
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
fn main() -> eframe::Result<()> {
|
|
|
|
|
let args: Vec<String> = std::env::args().collect();
|
|
|
|
|
let screenshot = args.iter().position(|a| a == "--screenshot").and_then(|i| args.get(i + 1)).map(PathBuf::from);
|
Remote brain: a computer that just came on is known at once; an offline computer is said, not swapped; snapshots survive an open browser; end-to-end test
- The brain looked the account's computers up once a minute: for up to a minute after the Mac
came on, 「我的電腦」 was unknown and steps went elsewhere. Now every 15 s, and at once when a
message comes from a computer it doesn't know yet.
- A step naming a computer that is off says so in words instead of running on the chosen one
with a note that the person hadn't asked.
- Snapshots: tar's 'file changed' (the open browser) is not a failure; browser caches are left
out; ten minutes for a large home folder.
- box_end_to_end clicks the terminal where it opened; restore only with ANYBOT_TEST_RESTORE=1.
- deploy/test/remote_e2e.py: account server + headless brain + hands + scripted model, 27 checks
(computers, redirect, agent-desktop on the hands, hand-off 3 ways, phone view, sync copy,
edits, hands offline/back, brain restart, clear). anybot --sample-state for such tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 10:14:23 +00:00
|
|
|
// For tests that start AnyBots of their own: a whole settings file (the sample bots) to edit.
|
|
|
|
|
if args.get(1).map(String::as_str) == Some("--sample-state") {
|
|
|
|
|
let out = args.get(2).cloned().unwrap_or_else(|| "state.json".into());
|
|
|
|
|
let mut st = model::State::samples();
|
|
|
|
|
st.remote = Default::default();
|
|
|
|
|
if let Err(e) = std::fs::write(&out, serde_json::to_vec_pretty(&st).unwrap_or_default()) {
|
|
|
|
|
eprintln!("{e}");
|
|
|
|
|
std::process::exit(1);
|
|
|
|
|
}
|
|
|
|
|
return Ok(());
|
|
|
|
|
}
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
if args.get(1).map(String::as_str) == Some("--write-icon") {
|
|
|
|
|
let out = args.get(2).cloned().unwrap_or_else(|| "anybot.png".into());
|
|
|
|
|
let px = args.get(3).and_then(|v| v.parse().ok()).unwrap_or(256);
|
2026-11-05 11:20:29 +00:00
|
|
|
if let Err(e) = write_icon(&out, px, args.get(4).map(String::as_str)) {
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
eprintln!("{e}");
|
|
|
|
|
std::process::exit(1);
|
|
|
|
|
}
|
|
|
|
|
return Ok(());
|
|
|
|
|
}
|
Bots click by number: every control on the screen is boxed and numbered, on every computer
A bot asked to open a Threads profile clicked an empty strip ten times: it guessed coordinates
from a picture, and the web tools' description told it they didn't work on its own computer.
- marks.rs: the picture a bot gets has every button, link, field and menu item boxed and
numbered (a tiny built-in digit font, pure Rust), with the same numbers in a list. click /
typing / scroll / mouse_move take the number (ref) and land on the control's centre.
- Where the controls come from: macOS AX and Linux AT-SPI through agent-desktop on a person's
computer (desktop::elements); AT-SPI in the bots' own computer (box/a11y, after XBot's
box-a11y), web pages included. On a Mac a control whose spot another app's window covers gets
no number (topmost window at its centre must be the app's; the Dock's invisible full-screen
window doesn't count). open_app waits until the app is really in front.
- No going round in circles: a step that changed nothing says so; the same step (clicks within
16 px) a third time in a turn comes with a warning, from the fourth time it isn't run.
- The box (version 4): one TigerVNC server instead of Xvfb + x11vnc, the AT-SPI bus and registry
at boot, one browser launcher (box/browser, after XBot's box-chrome: same profile and CDP,
renderer accessibility, a launch lock, stale profile locks cleared). One docker exec per look
(list + picture, ~0.4 s). A brain updates an older box by itself when it has been idle a
minute (files and logins stay); a new box is never made from an older image.
- Gone: computer_use_read_app (the list is in every picture), computer_use_mouse_button, the
per-call picture for the phone (it gets a plain one), the stale browser_read note; the prompt
says web work goes through browser_* by number, and never to read browser cookie or history
files.
Tests: marks (numbering, drawing, fingerprints), box_numbers_are_exact on a box from the new
image (a Threads-like page: the icon links get numbers, Profile clicked by number, Chinese typed
into the numbered field, a click on nothing called out), box_screen_is_exact, VNC e2e 11/11 on
TigerVNC, Calculator 7×8=56 by numbers through the signed app on this Mac, remote_e2e 28/28,
153 unit tests; Linux build + 157 tests + x11_live.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 16:47:01 +00:00
|
|
|
// For testing on a new system: `anybot --tool computer_use_screenshot '{}' computer_use_click
|
|
|
|
|
// '{"ref":"7"}' …` runs computer tools in order (one process, so a picture's numbers carry
|
|
|
|
|
// over), prints what the bot would hear, and saves the numbered pictures that come back as
|
|
|
|
|
// anybot-tool-N.jpg in the temp folder.
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
if args.get(1).map(String::as_str) == Some("--tool") {
|
|
|
|
|
let mut all_ok = true;
|
|
|
|
|
for (i, pair) in args[2..].chunks(2).enumerate() {
|
|
|
|
|
let json = pair.get(1).cloned().unwrap_or_else(|| "{}".into());
|
|
|
|
|
let (text, ok, img) = computer::run(&pair[0], &json);
|
|
|
|
|
println!("{} {}: {text}", pair[0], if ok { "ok" } else { "failed" });
|
|
|
|
|
if let Some(b64) = img {
|
|
|
|
|
use base64::Engine;
|
|
|
|
|
let out = std::env::temp_dir().join(format!("anybot-tool-{i}.jpg"));
|
|
|
|
|
let _ = std::fs::write(&out, base64::engine::general_purpose::STANDARD.decode(b64).unwrap_or_default());
|
|
|
|
|
println!(" saved {}", out.display());
|
|
|
|
|
}
|
|
|
|
|
all_ok &= ok;
|
|
|
|
|
std::thread::sleep(std::time::Duration::from_millis(300));
|
|
|
|
|
}
|
|
|
|
|
std::process::exit(if all_ok { 0 } else { 1 });
|
|
|
|
|
}
|
2026-11-04 09:19:59 +00:00
|
|
|
// For trying agent-desktop on a new system: `anybot --desktop snapshot '{"app":"Finder"}'`.
|
|
|
|
|
if args.get(1).map(String::as_str) == Some("--desktop") {
|
|
|
|
|
let cmd = args.get(2).cloned().unwrap_or_else(|| "apps".into());
|
|
|
|
|
let v: serde_json::Value = serde_json::from_str(args.get(3).map(String::as_str).unwrap_or("{}")).unwrap_or_default();
|
|
|
|
|
match desktop::run(&cmd, &v) {
|
|
|
|
|
Ok(out) => println!("{}", serde_json::to_string_pretty(&out).unwrap_or_default()),
|
|
|
|
|
Err(e) => {
|
|
|
|
|
eprintln!("{e}");
|
|
|
|
|
std::process::exit(1);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
std::process::exit(0);
|
|
|
|
|
}
|
2026-11-04 12:38:45 +00:00
|
|
|
// `anybot setup`: make this computer a brain, or hands for a brain (in a terminal).
|
|
|
|
|
if args.get(1).map(String::as_str) == Some("setup") {
|
|
|
|
|
std::process::exit(setup::run(&args[2..]));
|
|
|
|
|
}
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
if args.get(1).map(String::as_str) == Some("--ask") {
|
|
|
|
|
let text = args[2..].join(" ");
|
|
|
|
|
let ctx = egui::Context::default();
|
|
|
|
|
app::App::ask_cli(&ctx, &text);
|
|
|
|
|
}
|
|
|
|
|
// No window: keep the bots working and answer the phone (always-on Mac or server).
|
|
|
|
|
if args.iter().any(|a| a == "--serve") {
|
|
|
|
|
let ctx = egui::Context::default();
|
|
|
|
|
app::App::serve(&ctx);
|
|
|
|
|
}
|
|
|
|
|
|
2026-11-05 09:29:55 +00:00
|
|
|
let mut viewport = egui::ViewportBuilder::default()
|
|
|
|
|
.with_title("AnyBot")
|
|
|
|
|
.with_inner_size(shot_size())
|
|
|
|
|
.with_min_inner_size([app::SIDEBAR_MIN_WIDTH + app::CHAT_MIN_WIDTH, 480.0])
|
|
|
|
|
.with_app_id("anybot")
|
|
|
|
|
.with_icon(app_icon());
|
|
|
|
|
// On a Mac the window is one surface to the very top (the grey system title bar didn't match
|
|
|
|
|
// a dark window); the traffic lights sit over the sidebar's empty corner.
|
|
|
|
|
if cfg!(target_os = "macos") {
|
|
|
|
|
viewport = viewport.with_fullsize_content_view(true).with_titlebar_shown(false).with_title_shown(false);
|
|
|
|
|
}
|
|
|
|
|
let options = eframe::NativeOptions { viewport, ..Default::default() };
|
AnyBot: first commit of the working desktop app
Rust/egui rebuild of Grok Bot as of 2026-11-04: chat with streaming
providers (xAI, OpenAI, Claude, Gemini, OpenRouter, OpenCode Go, Ollama,
custom), tools and approvals, the bot's Docker computer, computer use on
macOS / Linux X11 / Wayland, AnyBot's browser with a full DevTools-based
toolset, the encrypted vault, pinned grid and sidebar sections, Markdown
with Mermaid diagrams, phone access through the relay, and the Linux
build and test scripts.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uirha3oEtWeHh58zA7ErHa
2026-11-04 08:51:51 +00:00
|
|
|
eframe::run_native(
|
|
|
|
|
"AnyBot",
|
|
|
|
|
options,
|
|
|
|
|
Box::new(move |cc| {
|
|
|
|
|
load_fonts(&cc.egui_ctx);
|
|
|
|
|
Ok(Box::new(app::App::new(&cc.egui_ctx, screenshot)))
|
|
|
|
|
}),
|
|
|
|
|
)
|
|
|
|
|
}
|