daniel.w
/
AnyBot
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'one-screen-path': one way to act on a screen, the result comes back with each step, phone taps exact, typing under any input method, Docker steps in one exec 

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
王性驊 2026-11-05 22:04:30 +08:00
parent db028b0b5b aa9b010276
commit 237fd9f756
15 changed files with 1102 additions and 652 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Cargo.lock generated
View File

@ -336,6 +336,7 @@ dependencies = [
"agent-desktop-linux", "agent-desktop-linux",
"agent-desktop-macos", "agent-desktop-macos",
"agent-desktop-windows", "agent-desktop-windows",
"arboard",
"ashpd", "ashpd",
"async-io", "async-io",
"base64 0.22.1", "base64 0.22.1",

2
Cargo.toml
View File

@ -69,6 +69,8 @@ async-io = "2"
[target.'cfg(target_os = "macos")'.dependencies] [target.'cfg(target_os = "macos")'.dependencies]
agent-desktop-macos = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" } agent-desktop-macos = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" }
# Typing by pasting (any input method, any language); already in the build through egui.
arboard = { version = "3.7", default-features = false }
[target.'cfg(target_os = "windows")'.dependencies] [target.'cfg(target_os = "windows")'.dependencies]
agent-desktop-windows = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" } agent-desktop-windows = { git = "https://github.com/vercel-labs/agent-desktop", tag = "v1.1.0" }

27
FEATURES.md
View File

@ -1042,18 +1042,21 @@ AnyBot 是一個桌面聊天 App，介面和運作方式照 Grok Bot 重做。
## 24. 操作螢幕（滑鼠、鍵盤、截圖） ## 24. 操作螢幕（滑鼠、鍵盤、截圖）
- 兩台電腦有螢幕：「Bot 的電腦」（Linux 桌面）和「你的電腦」（這台 Mac）。每個螢幕工具都有 `computer` 參數；沒指定時用 Bot 的預設電腦，預設電腦沒有螢幕時用 Bot 的電腦。 - 有螢幕的電腦：Bot 自己的電腦（Linux 桌面，Docker）、這台電腦、帳號裡其他開了「裝置連結」的電腦。每個螢幕工具都有 `computer` 參數；沒指定時用 Bot 的預設電腦。
- 在 Bot 的電腦上：用 xdotool 操作（跟原版雲端電腦一樣），不用問你；英文一個鍵一個鍵打，中文和表情符號用剪貼簿貼上（終端機用 Shift+Insert，其他程式用 Ctrl+V），所以任何程式都打得進去；⌘ 當成 Ctrl。 - **動手只有一種方式：真的滑鼠和鍵盤。** Mac 用 CoreGraphics，Linux 用 X11／Wayland（desk.rs），Bot 的電腦用 xdotool。看畫面有兩種：截圖，或用「讀 App 畫面」拿到按鈕清單（e 編號）。用編號點或打字時，AnyBot 會換成那個按鈕在螢幕上的位置，用同一套滑鼠鍵盤去點（以前編號走 agent-desktop 自己的一套動作，按鍵名稱和失敗方式都不一樣，在終端機打不了字、`/` 和 `` ` `` 會出錯，已拆掉）。
- 在你的 Mac 上：只在 macOS。工具名稱和參數跟原版一樣。 - **每個動作做完自動附上新畫面**（點、打字、按鍵、捲動、拖曳、移動滑鼠、打開 App；先等 0.5 秒讓畫面穩定），Bot 不用再花一步截圖確認。拍不到畫面時（Mac 還沒開「螢幕與系統錄音」），改附最前面 App 的按鈕清單。按住滑鼠鍵（拖到一半）不附。
- 需要看得懂圖片的模型。 - 在 Bot 的電腦上：一個動作只開一次 `docker exec`（動作和之後的畫面在同一個指令裡），不用問你；英文一個鍵一個鍵打，中文和表情符號用剪貼簿貼上（終端機用 Shift+Insert，其他程式用 Ctrl+V），所以任何程式都打得進去；⌘ 當成 Ctrl。不用 `xdotool mousemove --sync`：滑鼠已經在那個位置時它會一直等（同一點點兩次會卡兩分鐘）。
- 需要看得懂圖片的模型；看不懂圖片時用按鈕清單。
### 原版的九個工具 ### 原版的九個工具
computer_use_screenshot、click、mouse_move、mouse_button（按住／放開）、drag、scroll、typing、press_key、check_permissions。 computer_use_screenshot、click、mouse_move、mouse_button（按住／放開）、drag、scroll、typing、press_key、check_permissions。另外兩個：
- **「讀 App 畫面」（computer_use_read_app）**：App 視窗的按鈕、欄位、選單、清單和文字，每個控制項一個編號（e1、e2…），同時記下它在螢幕上的位置。很長的內容（終端機的整段捲動紀錄）留開頭 160 字和結尾 400 字，提示字和最新輸出在結尾。Bot 的電腦（Linux 桌面）沒有這個，會說請用截圖。
- **「打開 App」（computer_use_open_app）**：打開或叫到最前面，顯示的名字也可以（「計算機」→ Calculator）。Bot 的電腦上：chromium（瀏覽器，就是 browser_* 在用的那個，不會開第二個）、xterm（終端機）或其他程式名稱。
- **「按住或放開滑鼠」（computer_use_mouse_button）**：action 是 press 或 release，可以先移到 x、y。用來長按或自己控制的拖曳。結果「已按住滑鼠鍵（記得之後放開）。」「已放開滑鼠鍵。」 - **「按住或放開滑鼠」（computer_use_mouse_button）**：action 是 press 或 release，可以先移到 x、y。用來長按或自己控制的拖曳。結果「已按住滑鼠鍵（記得之後放開）。」「已放開滑鼠鍵。」
- **「檢查螢幕權限」（computer_use_check_permissions）**：回報「螢幕錄製：已允許／還沒允許」「輔助使用（滑鼠和鍵盤）：…」，不會跳出系統視窗。Bot 的電腦上一律可以。 - **「檢查螢幕權限」（computer_use_check_permissions）**：回報「螢幕錄製：已允許／還沒允許」「輔助使用（滑鼠和鍵盤）：…」，不會跳出系統視窗。Bot 的電腦上一律可以。
- **背景點擊**：click 加上 `background: true`，在你的 Mac 上把點擊直接送給那個視窗的 App，不移動游標、不把視窗拉到前面（原版用私有 API「skylight-no-raise」做這件事）。結果「已在背景點「App」的 (x, y)，沒有把它拉到前面。」
### 逐個 App 授權（原版：「Grok Bot 可以看到 {app}，並在其中點擊和輸入」） ### 逐個 App 授權（原版：「Grok Bot 可以看到 {app}，並在其中點擊和輸入」）
@ -1099,9 +1102,10 @@ computer_use_screenshot、click、mouse_move、mouse_button（按住／放開）
### 座標 ### 座標
- 用最新截圖上的像素位置。AnyBot 依螢幕大小和截圖大小的比例換算。 - 用截圖上的像素位置。截圖的大小只看螢幕大小（長邊 1,024），所以換算不用記「上一張截圖」：還沒截圖也能點，每個 Bot、手機看到的都是同一個尺寸。
- 例如寬 1,470 點的螢幕送出寬 1,024 的圖，圖上 (512, 333) 就是螢幕上的 (735, 478)。 - 例如寬 1,470 點的螢幕送出寬 1,024 的圖，圖上 (512, 333) 就是螢幕上的 (735, 478)。
- 還沒截圖就要點：「請先用 computer_use_screenshot 看一下畫面，座標要用截圖上的位置。」 - 或用「讀 App 畫面」的編號（ref）：點那個控制項的中心。那個 App 不在最前面時先把它叫到前面，免得點到蓋在上面的視窗。
- 手機網頁看螢幕時：Bot 的電腦送原尺寸畫面，這台電腦送和 Bot 一樣大小的截圖；手機點的位置照它拿到的那張畫面換算，是「你」在操作（不經過 Bot 的工具，你取得控制權時也照樣能點）。以前手機的點擊借用 Bot 上一張截圖的比例，點 Bot 的電腦會偏到右下（多 1.25 倍）或要求先截圖。
- 你自己的滑鼠和鍵盤不會被凍住。 - 你自己的滑鼠和鍵盤不會被凍住。
### 「看你的螢幕」（computer_use_screenshot） ### 「看你的螢幕」（computer_use_screenshot）
@ -1116,11 +1120,11 @@ computer_use_screenshot、click、mouse_move、mouse_button（按住／放開）
- 參數：x、y、左／右／中鍵、點幾下（1–3）、修飾鍵。 - 參數：x、y、左／右／中鍵、點幾下（1–3）、修飾鍵。
- 先移過去，按下後停 0.03 秒再放開；連點時每下隔 0.08 秒（取自原版）。 - 先移過去，按下後停 0.03 秒再放開；連點時每下隔 0.08 秒（取自原版）。
- 結果：「已點 (x, y)。要確認結果可以再截圖。」 - 結果：「已點 (x, y)。」或「已點 e7（按鈕「7」）。」，加上新畫面。
### 「打字」（computer_use_typing） ### 「打字」（computer_use_typing）
- 任何語言都可以，包括中文。 - 任何語言都可以，包括中文。可以給 ref（先點那個欄位）；submit=true 打完再按 return。
- 每段最多 20 個字元單位，不會把一個字切斷。每段按下後停 0.035 秒再放開，段與段之間隔 0.02 秒（取自原版）。 - 每段最多 20 個字元單位，不會把一個字切斷。每段按下後停 0.035 秒再放開，段與段之間隔 0.02 秒（取自原版）。
- 結果：「已打字（N 個字）。」 - 結果：「已打字（N 個字）。」
@ -1131,7 +1135,8 @@ computer_use_screenshot、click、mouse_move、mouse_button（按住／放開）
- 每個鍵按住 0.035 秒，鍵與鍵之間隔 0.02 秒（取自原版）。可以指定按多久，最多 5 秒。 - 每個鍵按住 0.035 秒，鍵與鍵之間隔 0.02 秒（取自原版）。可以指定按多久，最多 5 秒。
- 可用的鍵： - 可用的鍵：
- return／enter、tab、space、delete／backspace、escape、forwarddelete、home、end、pageup、pagedown、方向鍵、F1–F12。 - return／enter、tab、space、delete／backspace、escape、forwarddelete、home、end、pageup、pagedown、方向鍵、F1–F12。
- a–z、0–9 和常見符號。 - a–z、0–9 和常見符號，也可以寫名字：slash、comma、period、semicolon、quote、minus、equal、grave、bracketleft／bracketright、backslash。
- 不是按鍵的單一字元（「@」「？」）直接打出來。
- 修飾鍵：cmd、shift、option、control（也聽得懂 command、alt、ctrl 等寫法）。 - 修飾鍵：cmd、shift、option、control（也聽得懂 command、alt、ctrl 等寫法）。
- 錯誤：「不認得「X」這個鍵」「不認得「X」這個修飾鍵」。 - 錯誤：「不認得「X」這個鍵」「不認得「X」這個修飾鍵」。

4
docs/architecture.md
View File

@ -71,7 +71,7 @@ AnyBot 是一個 Rust 程式（`anybot`），用 egui/eframe 畫介面。同一
## 安全 ## 安全
- **核准關卡**：工具動作照規則直接做、先問或拒絕（`policy.rs`，FEATURES.md 第 16、19 節）。「直接操作模式」可以跳過詢問。 - **核准關卡**：工具動作照規則直接做、先問或拒絕（`policy.rs`，FEATURES.md 第 16、19 節）。「直接操作模式」可以跳過詢問。
- **指令的沙盒（macOS）**：Bot 執行的指令在 `sandbox-exec` 裡跑，讀不到也寫不到 AnyBot 的資料資料夾和 `.ssh`、`.gnupg`、鑰匙圈、`.aws` 等；只有 `attachments/` 可以讀（不能寫）。環境變數裡的模型金鑰會先拿掉。 - **指令的沙盒（macOS）**：Bot 執行的指令在 `sandbox-exec` 裡跑，讀不到也寫不到 AnyBot 的資料資料夾、`.gnupg`、鑰匙圈、`.aws` 等；只有 `attachments/` 和 `.ssh` 可以讀（都不能寫，`.ssh` 可讀讓 git 和 ssh 連得動）。只有沙盒管得到的才是祕密防線，核准層不再逐字掃路徑。環境變數裡的模型金鑰會先拿掉，PATH 會補齊 Homebrew 等位置（從 Finder 或 launchd 啟動也要找得到指令）。
- **Bot 的電腦**：放在容器裡，和你的電腦分開。 - **Bot 的電腦**：放在容器裡，和你的電腦分開。
- **保險庫**：密碼、卡號、CVC 不會交給模型，由 AnyBot 直接填進網頁。 - **保險庫**：密碼、卡號、CVC 不會交給模型，由 AnyBot 直接填進網頁。
- **多台電腦**：金鑰只留在大腦（下面說明）；手腳只接受帳號裡的 AnyBot，用配對碼加入的只接受那一個大腦。 - **多台電腦**：金鑰只留在大腦（下面說明）；手腳只接受帳號裡的 AnyBot，用配對碼加入的只接受那一個大腦。
@ -115,7 +115,7 @@ AnyBot 是一個 Rust 程式（`anybot`），用 egui/eframe 畫介面。同一
### 手腳 ### 手腳
大腦的 Bot 呼叫工具時可以指定 `computer`。如果是帳號裡的另一台，就透過長連線送 `/api/device/run` 給它，由它在自己身上執行（螢幕、檔案、指令、它的 AnyBot 瀏覽器），再把文字和截圖傳回來。那台要開「裝置連結」才接受；若設了只給某個大腦用（`hands_for`），帳號伺服器會附上是哪一台在問，不是那台就拒絕。 大腦的 Bot 呼叫工具時可以指定 `computer`。如果是帳號裡的另一台，就透過長連線送 `/api/device/run` 給它，由它在自己身上執行（螢幕、檔案、指令、它的 AnyBot 瀏覽器），再把文字和截圖傳回來。那台要開「裝置連結」才接受；若設了只給某個大腦用（`hands_for`），帳號伺服器會附上是哪一台在問，不是那台就拒絕。指令在那台的工作資料夾（Bot 的 `workdir`）跑；連線經過帳號伺服器只有 35 秒，所以手腳上的指令超過 25 秒就移到背景，用同一台的 `read_terminal` 追蹤（computer 填那台）。
### 配對碼 ### 配對碼

18
src/agent.rs
View File

@ -131,7 +131,7 @@ fn browser_place(job: &Job, name: &str, args: &str) -> (BrowserAt, String) {
let want = v.get("computer").and_then(|c| c.as_str()).map(str::trim).filter(|c| !c.is_empty()).unwrap_or(&job.default_computer).to_string(); let want = v.get("computer").and_then(|c| c.as_str()).map(str::trim).filter(|c| !c.is_empty()).unwrap_or(&job.default_computer).to_string();
let at = |t: &tools::Target| -> Option<BrowserAt> { let at = |t: &tools::Target| -> Option<BrowserAt> {
match &t.place { match &t.place {
tools::Place::Device { host } => Some(BrowserAt::Device(host.clone())), tools::Place::Device { host, .. } => Some(BrowserAt::Device(host.clone())),
tools::Place::Box { .. } if crate::cloudbox::available() => Some(BrowserAt::Box), tools::Place::Box { .. } if crate::cloudbox::available() => Some(BrowserAt::Box),
tools::Place::Local { .. } if crate::botbrowser::available() => Some(BrowserAt::Here), tools::Place::Local { .. } if crate::botbrowser::available() => Some(BrowserAt::Here),
_ => None, _ => None,
@ -262,7 +262,7 @@ fn roles_now(job: &Job) -> tools::Roles {
let mut r = job.roles.clone(); let mut r = job.roles.clone();
let host = crate::remote::person_host(); let host = crate::remote::person_host();
if !host.is_empty() { if !host.is_empty() {
if let Some(t) = job.computers.iter().find(|t| matches!(&t.place, tools::Place::Device { host: h } if *h == host)) { if let Some(t) = job.computers.iter().find(|t| matches!(&t.place, tools::Place::Device { host: h, .. } if *h == host)) {
r.person = Some(t.name.clone()); r.person = Some(t.name.clone());
} }
} }
@ -651,7 +651,7 @@ pub fn spawn(job: Job, bot_id: u64, tx: Sender<(u64, StreamEvent)>, cancel: Arc<
// computer the person is using; a file is sent into the chat instead. // computer the person is using; a file is sent into the chat instead.
let person_now = roles_now(&job).person; let person_now = roles_now(&job).person;
let person_host = person_now.as_ref().and_then(|n| job.computers.iter().find(|t| &t.name == n)).and_then(|t| match &t.place { let person_host = person_now.as_ref().and_then(|n| job.computers.iter().find(|t| &t.name == n)).and_then(|t| match &t.place {
tools::Place::Device { host } => Some(host.clone()), tools::Place::Device { host, .. } => Some(host.clone()),
_ => None, _ => None,
}); });
if job.roles.server { if job.roles.server {
@ -887,7 +887,7 @@ pub fn spawn(job: Job, bot_id: u64, tx: Sender<(u64, StreamEvent)>, cancel: Arc<
}; };
let on_box = matches!(target.as_ref().map(|t| &t.place), Ok(tools::Place::Box { .. })); let on_box = matches!(target.as_ref().map(|t| &t.place), Ok(tools::Place::Box { .. }));
let on_device = match target.as_ref().map(|t| &t.place) { let on_device = match target.as_ref().map(|t| &t.place) {
Ok(tools::Place::Device { host }) => Some(host.clone()), Ok(tools::Place::Device { host, .. }) => Some(host.clone()),
_ => None, _ => None,
}; };
// Permission to use computers at all, and this Mac in particular, is asked for // Permission to use computers at all, and this Mac in particular, is asked for
@ -1151,7 +1151,7 @@ mod tests {
assert!(!finds("你的虛擬電腦很吵", "bot 的電腦")); assert!(!finds("你的虛擬電腦很吵", "bot 的電腦"));
assert!(finds("在 bot 的電腦跑", "bot 的電腦")); assert!(finds("在 bot 的電腦跑", "bot 的電腦"));
let roles = tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true }; let roles = tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true };
let mac = tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into() } }; let mac = tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } };
assert!(mentions("幫我在 Mac 截圖", &mac, &roles)); assert!(mentions("幫我在 Mac 截圖", &mac, &roles));
assert!(!mentions("在 macOS 的登入機制下，幫 Bot 的電腦截圖", &mac, &roles)); assert!(!mentions("在 macOS 的登入機制下，幫 Bot 的電腦截圖", &mac, &roles));
} }
@ -1171,8 +1171,8 @@ mod tests {
// the everyday words point at the right one (「我的電腦」 = where the person typed). // the everyday words point at the right one (「我的電腦」 = where the person typed).
let roles = tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true }; let roles = tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true };
let own = tools::Target { name: "Master 的電腦".into(), place: tools::Place::Box { dir: String::new() } }; let own = tools::Target { name: "Master 的電腦".into(), place: tools::Place::Box { dir: String::new() } };
let mac = tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into() } }; let mac = tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } };
let desk = tools::Target { name: "ai".into(), place: tools::Place::Device { host: "a".into() } }; let desk = tools::Target { name: "ai".into(), place: tools::Place::Device { host: "a".into(), dir: String::new() } };
let server = tools::Target { name: "雲端大腦".into(), place: tools::Place::Local { workdir: Default::default() } }; let server = tools::Target { name: "雲端大腦".into(), place: tools::Place::Local { workdir: Default::default() } };
assert!(mentions("在 Master 的電腦打開 Threads", &own, &roles)); assert!(mentions("在 Master 的電腦打開 Threads", &own, &roles));
assert!(mentions("用master的電腦", &own, &roles)); assert!(mentions("用master的電腦", &own, &roles));
@ -1192,7 +1192,7 @@ mod tests {
let job = Job { let job = Job {
computers: vec![ computers: vec![
tools::Target { name: "Master 的電腦".into(), place: tools::Place::Box { dir: String::new() } }, tools::Target { name: "Master 的電腦".into(), place: tools::Place::Box { dir: String::new() } },
tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into() } }, tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } },
], ],
default_computer: "Master 的電腦".into(), default_computer: "Master 的電腦".into(),
roles, roles,
@ -1491,7 +1491,7 @@ mod tests {
}); });
let job = Job { let job = Job {
req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("幫我登入 Threads")], tools: vec![], session: String::new() }, req: ChatRequest { kind: ProviderKind::Custom, base_url: format!("http://127.0.0.1:{port}/v1"), api_key: String::new(), model: "m".into(), system: String::new(), history: vec![Message::user("幫我登入 Threads")], tools: vec![], session: String::new() },
computers: vec![tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into() } }], computers: vec![tools::Target { name: "王小明的Mac mini".into(), place: tools::Place::Device { host: "m".into(), dir: String::new() } }],
default_computer: "王小明的Mac mini".into(), default_computer: "王小明的Mac mini".into(),
roles: tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true }, roles: tools::Roles { own: "Master 的電腦".into(), person: Some("王小明的Mac mini".into()), server: true },
..test_job() ..test_job()

10
src/app.rs
View File

@ -870,8 +870,12 @@ impl App {
for c in &self.state.remote_computers { for c in &self.state.remote_computers {
add(&mut v, tools::Target { name: c.name.clone(), place: tools::Place::Ssh { target: c.ssh_target.clone(), dir: c.dir.clone() } }); add(&mut v, tools::Target { name: c.name.clone(), place: tools::Place::Ssh { target: c.ssh_target.clone(), dir: c.dir.clone() } });
} }
// 裝置連結: other AnyBots of this account that allow it. // 裝置連結: other AnyBots of this account that allow it, working in the bot's work folder.
for d in self.linked_devices() { for d in self.linked_devices() {
let mut d = d;
if let tools::Place::Device { dir, .. } = &mut d.place {
dir.clone_from(&b.workdir);
}
add(&mut v, d); add(&mut v, d);
} }
v v
@ -883,7 +887,7 @@ impl App {
pub(super) fn roles_for(&self, b: &Bot, targets: &[tools::Target]) -> tools::Roles { pub(super) fn roles_for(&self, b: &Bot, targets: &[tools::Target]) -> tools::Roles {
let server = self.headless; let server = self.headless;
let local = targets.iter().find(|t| matches!(t.place, tools::Place::Local { .. })).map(|t| t.name.clone()); let local = targets.iter().find(|t| matches!(t.place, tools::Place::Local { .. })).map(|t| t.name.clone());
let device = |h: &str| targets.iter().find(|t| matches!(&t.place, tools::Place::Device { host } if host == h)).map(|t| t.name.clone()); let device = |h: &str| targets.iter().find(|t| matches!(&t.place, tools::Place::Device { host, .. } if host == h)).map(|t| t.name.clone());
let person = match self.typed_on.as_deref() { let person = match self.typed_on.as_deref() {
Some(h) if !h.is_empty() => device(h).or_else(|| if server { None } else { local.clone() }), Some(h) if !h.is_empty() => device(h).or_else(|| if server { None } else { local.clone() }),
_ if !server => local, _ if !server => local,
@ -4041,7 +4045,7 @@ fn system_prompt(b: &Bot, st: &State, computers: &[tools::Target], default_compu
- 需要使用者親手做的步驟（登入、輸入密碼、驗證碼、拼圖、簡訊或 email 驗證、確認付款）：先在電腦上把那一頁打開、點到要輸入的地方，再呼叫 hand_over_computer，instruction 寫清楚要他做什麼。聊天裡會出現那台電腦的即時畫面，他點「接管」做完後按「我完成了」，你再截圖確認、接著做。不要只用文字叫他自己去找視窗；也絕對不要替他打密碼。 - 需要使用者親手做的步驟（登入、輸入密碼、驗證碼、拼圖、簡訊或 email 驗證、確認付款）：先在電腦上把那一頁打開、點到要輸入的地方，再呼叫 hand_over_computer，instruction 寫清楚要他做什麼。聊天裡會出現那台電腦的即時畫面，他點「接管」做完後按「我完成了」，你再截圖確認、接著做。不要只用文字叫他自己去找視窗；也絕對不要替他打密碼。
- 如果讀到的是登入頁：把登入頁開在那台電腦上，用 hand_over_computer 請使用者登入一次（之後會記住），他按「我完成了」再讀。絕對不要叫使用者去改瀏覽器的設定或選單（例如「允許來自 Apple 事件的 JavaScript」），那不需要。也不要說要從 Chrome 複製登入或 Cookie，沒有這個功能。\n\ - 如果讀到的是登入頁：把登入頁開在那台電腦上，用 hand_over_computer 請使用者登入一次（之後會記住），他按「我完成了」再讀。絕對不要叫使用者去改瀏覽器的設定或選單（例如「允許來自 Apple 事件的 JavaScript」），那不需要。也不要說要從 Chrome 複製登入或 Cookie，沒有這個功能。\n\
- AnyBot 瀏覽器有設定檔（profile），每個有自己的登入。使用者說要換 profile／設定檔／另一個帳號時，用 browser_profiles：list 看有哪些，use 切換，add 新增（新的要登入一次）。也可以在 browser_open / browser_read 帶 profile 參數。使用者想用自己平常的 Chrome（自己的 Chrome 設定檔、已經登入的帳號）時，用 profile「你的瀏覽器」：請使用者在 Chrome 切到想要的設定檔，再 browser_read。\n\ - AnyBot 瀏覽器有設定檔（profile），每個有自己的登入。使用者說要換 profile／設定檔／另一個帳號時，用 browser_profiles：list 看有哪些，use 切換，add 新增（新的要登入一次）。也可以在 browser_open / browser_read 帶 profile 參數。使用者想用自己平常的 Chrome（自己的 Chrome 設定檔、已經登入的帳號）時，用 profile「你的瀏覽器」：請使用者在 Chrome 切到想要的設定檔，再 browser_read。\n\
- 你也能像人一樣操作螢幕（{screens}）。computer_use_screenshot 看畫面，再用 computer_use_click / computer_use_typing / computer_use_press_key / computer_use_scroll / computer_use_drag 操作；座標用最新截圖上的像素。每做一步就再截圖確認。用在檔案和網頁工具做不到的事（例如操作其他 App、在 Chrome 切換設定檔）。\n\ - 你也能像人一樣操作螢幕（{screens}），用在檔案、指令和網頁工具做不到的事（例如操作其他 App）。動手只有一種方式：真的滑鼠和鍵盤。開始時用 computer_use_screenshot 看一次；之後每個動作（click、typing、press_key、scroll、drag、open_app）做完都會直接附上新的畫面，不要再另外截圖確認。位置用最新畫面上的像素 x、y；或用 computer_use_read_app 讀出 App 的按鈕清單，用 e 編號（ref）點和打字，拍不到畫面時也能用。要在某個 App 裡打字：先 computer_use_open_app 把它放到最前面，再 computer_use_typing（submit=true 會順便按 return）。網頁上的事用 browser_* 工具，比點螢幕準。\n\
{}\ {}\
- 不要說「我沒有瀏覽器或工具」：先試這些工具。需要權限時 App 會自己跳出視窗問使用者；工具回報要開什麼設定時，照實轉告。", - 不要說「我沒有瀏覽器或工具」：先試這些工具。需要權限時 App 會自己跳出視窗問使用者；工具回報要開什麼設定時，照實轉告。",
now.format("%Y-%m-%d %H:%M"), now.format("%Y-%m-%d %H:%M"),

2
src/app/remote_ui.rs
View File

@ -776,7 +776,7 @@ impl App {
let uses = remote::peer_uses(id); let uses = remote::peer_uses(id);
uses.is_empty() || uses == self.state.remote.host_id uses.is_empty() || uses == self.state.remote.host_id
}) })
.map(|(id, name, _)| tools::Target { name: name.clone(), place: tools::Place::Device { host: id.clone() } }) .map(|(id, name, _)| tools::Target { name: name.clone(), place: tools::Place::Device { host: id.clone(), dir: String::new() } })
.collect() .collect()
} }
} }

602
src/cloudbox.rs
View File

@ -174,15 +174,15 @@ struct Shared {
last_used: Instant, last_used: Instant,
rebuilding: bool, rebuilding: bool,
log: Vec<String>, log: Vec<String>,
/// Screenshot pixel → desktop pixel (for the bot's coordinates). /// The desktop's size in pixels (the bot's pictures are this, fitted; see box_scale).
scale: Option<(f64, f64)>, size: Option<(u32, u32)>,
tunnel: Option<std::process::Child>, tunnel: Option<std::process::Child>,
host_port: u16, host_port: u16,
} }
fn shared() -> &'static Mutex<Shared> { fn shared() -> &'static Mutex<Shared> {
static S: OnceLock<Mutex<Shared>> = OnceLock::new(); static S: OnceLock<Mutex<Shared>> = OnceLock::new();
S.get_or_init(|| Mutex::new(Shared { run: Run::Absent, last_used: Instant::now(), rebuilding: false, log: vec![], scale: None, tunnel: None, host_port: HOST_PORT })) S.get_or_init(|| Mutex::new(Shared { run: Run::Absent, last_used: Instant::now(), rebuilding: false, log: vec![], size: None, tunnel: None, host_port: HOST_PORT }))
} }
fn config() -> &'static Mutex<Config> { fn config() -> &'static Mutex<Config> {
@ -567,6 +567,7 @@ fn free_port(start: u16) -> u16 {
} }
fn create_container() -> Result<(), String> { fn create_container() -> Result<(), String> {
shared().lock().unwrap_or_else(|e| e.into_inner()).size = None;
if !image_exists() { if !image_exists() {
build_image(false)?; build_image(false)?;
} }
@ -1160,17 +1161,17 @@ pub fn copy_out(path_in_box: &str) -> Result<PathBuf, String> {
} }
// ---------- the box's screen (xdotool) ---------- // ---------- the box's screen (xdotool) ----------
// One `docker exec` per step: the xdotool commands, then (for the bot) a short wait and the
fn xdo(args: &[&str]) -> Result<String, String> { // picture of the result, in one shell script. Each exec costs 50–200 ms locally and more on a
let mut a = vec!["exec", "-u", "bot", "-e", "DISPLAY=:1", container(), "xdotool"]; // Docker host elsewhere; a click with a modifier used to be four of them, plus one more for the
a.extend_from_slice(args); // screenshot the bot then had to ask for.
dk(&a).map_err(|e| format!("xdotool：{e}"))
}
/// The desktop as an image (full size). /// The desktop as an image (full size).
pub fn grab() -> Result<image::RgbImage, String> { pub fn grab() -> Result<image::RgbImage, String> {
let png = dk_timeout(&["exec", container(), "import", "-window", "root", "-display", DISPLAY, "png:-"], 10_000)?; let png = dk_timeout(&["exec", container(), "import", "-window", "root", "-display", DISPLAY, "png:-"], 10_000)?;
image::load_from_memory(&png).map(|i| i.to_rgb8()).map_err(|e| format!("截圖讀不出來：{e}")) let img = image::load_from_memory(&png).map(|i| i.to_rgb8()).map_err(|e| format!("截圖讀不出來：{e}"))?;
shared().lock().unwrap_or_else(|e| e.into_inner()).size = Some(img.dimensions());
Ok(img)
} }
/// The latest picture of the box's screen, small (for the 電腦 card). While the box runs it is /// The latest picture of the box's screen, small (for the 電腦 card). While the box runs it is
@ -1196,6 +1197,324 @@ pub fn frame() -> Option<image::RgbaImage> {
image::load_from_memory(&png).ok().map(|i| i.to_rgba8()) image::load_from_memory(&png).ok().map(|i| i.to_rgba8())
} }
/// The desktop's size in pixels (asked once per box).
fn box_size() -> Result<(u32, u32), String> {
if let Some(s) = shared().lock().unwrap_or_else(|e| e.into_inner()).size {
return Ok(s);
}
let out = dk(&["exec", "-u", "bot", "-e", "DISPLAY=:1", container(), "xdotool", "getdisplaygeometry"])?;
let mut it = out.split_whitespace().filter_map(|n| n.parse::<u32>().ok());
let size = match (it.next(), it.next()) {
(Some(w), Some(h)) if w > 0 && h > 0 => (w, h),
_ => return Err("讀不到這台電腦的螢幕大小".to_string()),
};
shared().lock().unwrap_or_else(|e| e.into_inner()).size = Some(size);
Ok(size)
}
/// Desktop pixels per picture pixel: the bot's pictures are the desktop fitted to 1024 wide,
/// always the same size, so its coordinates map without remembering a screenshot.
fn box_scale() -> Result<(f64, f64), String> {
let (w, h) = box_size()?;
let (fw, fh, _) = crate::computer::fit(w, h);
Ok((w as f64 / fw.max(1) as f64, h as f64 / fh.max(1) as f64))
}
/// A step on the box's screen as a shell script (for `sh -c` inside the box, as the bot user),
/// the text it reads on stdin (typing), and what to tell the bot. Coordinates are picture pixels
/// times `scale` (desktop pixels per picture pixel; 1 for a person's viewer, which shows the
/// desktop at full size).
pub fn box_script(name: &str, v: &Value, scale: (f64, f64)) -> Result<(String, Option<String>, String), String> {
let pt = |p: &Value| -> Result<(i64, i64), String> {
let (Some(x), Some(y)) = (p["x"].as_f64(), p["y"].as_f64()) else { return Err("請給 x 和 y（截圖上的位置）".into()) };
Ok(((x * scale.0).round() as i64, (y * scale.1).round() as i64))
};
let at = |p: &Value| format!("({}, {})", p["x"].as_f64().unwrap_or(0.0), p["y"].as_f64().unwrap_or(0.0));
if v["ref"].as_str().map(|r| !r.trim().is_empty()).unwrap_or(false) {
return Err("這台是 Linux 桌面，沒有 App 的編號（ref）：請用截圖上的 x、y。".into());
}
let mods = mods_of(v);
let down: String = mods.iter().map(|m| format!(" keydown {m}")).collect();
let up: String = mods.iter().rev().map(|m| format!(" keyup {m}")).collect();
// Never `mousemove --sync`: it waits forever when the pointer is already there (a second
// click on the same button hung the box for two minutes).
let mv = |(x, y): (i64, i64)| format!("mousemove {x} {y} sleep 0.03");
let wrap = |body: String| -> String {
if mods.is_empty() {
format!("xdotool {body}")
} else {
// Keys held for the step are let go even when the step fails (a stuck ctrl spoils
// every later click and keystroke in the box).
format!("xdotool{down} {body}; rc=$?; xdotool{up}; [ $rc = 0 ]")
}
};
let button = button_no(v["button"].as_str().unwrap_or("left"));
Ok(match name {
"computer_use_click" => {
let n = v["count"].as_u64().unwrap_or(1).clamp(1, 3);
(wrap(format!("{} click --repeat {n} --delay {CLICK_REPEAT_MS} {button}", mv(pt(v)?))), None, format!("已{}{} {}。", if button == "3" { "右鍵" } else { "" }, if n >= 2 { format!("連點 {n} 下") } else { "點".into() }, at(v)))
}
"computer_use_mouse_move" => (format!("xdotool {}", mv(pt(v)?)), None, format!("已把滑鼠移到 {}。", at(v))),
"computer_use_mouse_button" => {
let go = if v["x"].is_null() { String::new() } else { format!("{} ", mv(pt(v)?)) };
let release = v["action"].as_str() == Some("release");
(format!("xdotool {go}{} {button}", if release { "mouseup" } else { "mousedown" }), None, (if release { "已放開滑鼠鍵。" } else { "已按住滑鼠鍵（記得之後放開）。" }).to_string())
}
"computer_use_scroll" => {
let amount = crate::computer::scroll_amount(v["amount"].as_i64());
let dir = v["direction"].as_str().unwrap_or("").trim().to_lowercase();
let wheel = match dir.as_str() {
"up" => "4",
"down" => "5",
"left" => "6",
"right" => "7",
_ => return Err(format!("方向「{dir}」要是 up、down、left 或 right")),
};
let go = if v["x"].is_null() { String::new() } else { format!("{} ", mv(pt(v)?)) };
(wrap(format!("{go}click --repeat {amount} --delay 20 {wheel}")), None, format!("已捲動 {dir} ×{amount}。"))
}
"computer_use_drag" => {
// A point that fails to parse is an error, not a silently dropped waypoint.
let raw = v["path"].as_array().ok_or("拖曳要給 path：一個點陣，每個點有數字 x 和 y（用截圖上的座標）")?;
let path: Vec<(i64, i64)> = raw.iter().map(pt).collect::<Result<_, _>>()?;
if path.len() < 2 {
return Err("拖曳至少要兩個點（每個點要有數字 x 和 y，用截圖上的座標）".into());
}
let mut body = format!("{} mousedown {button} sleep 0.04", mv(path[0]));
let steps: Vec<(f64, f64)> = crate::computer::drag_points(&path.iter().map(|(x, y)| (*x as f64, *y as f64)).collect::<Vec<_>>());
for (x, y) in steps {
body.push_str(&format!(" mousemove {} {} sleep 0.008", x.round() as i64, y.round() as i64));
}
body.push_str(&format!(" mouseup {button}"));
(wrap(body), None, format!("已從 {} 拖曳到 {}。", at(&raw[0]), at(&raw[raw.len() - 1])))
}
"computer_use_typing" => {
let text = v["value"].as_str().unwrap_or("").to_string();
let submit = v["submit"].as_bool().unwrap_or(false);
if text.is_empty() && !submit {
return Err("請給要打的字".into());
}
let mut script = String::new();
if !text.is_empty() {
script.push_str(&type_script(&text));
}
if submit {
script.push_str(" && sleep 0.06 && xdotool key --clearmodifiers Return");
}
let script = script.trim_start_matches(" && ").to_string();
(script, Some(text.clone()).filter(|t| !t.is_empty()), format!("已打字（{} 個字）{}。", text.chars().count(), if submit { "，再按了 return" } else { "" }))
}
"computer_use_press_key" => {
let keys = v["key"].as_str().unwrap_or("");
let hold = v["hold_duration_ms"].as_u64().unwrap_or(0).min(5_000);
let mut parts: Vec<String> = vec![];
for k in keys.split_whitespace() {
match combo_to_x(k) {
Ok(c) if hold > 0 => parts.push(format!("keydown {c} sleep {:.3} keyup {c}", hold as f64 / 1000.0)),
Ok(c) => parts.push(format!("key --clearmodifiers {c}")),
// 「@」「？」: a character, typed.
Err(_) if k.chars().count() == 1 => parts.push(format!("type --clearmodifiers -- {}", crate::tools::sh_quote(k))),
Err(e) => return Err(e),
}
}
if parts.is_empty() {
return Err("請給要按的鍵".into());
}
(format!("xdotool {}", parts.join(" sleep 0.02 ")), None, format!("已按 {keys}。"))
}
other => return Err(format!("沒有「{other}」這個工具")),
})
}
/// Typing, as a script reading the text on stdin. Plain ASCII goes key by key; anything else
/// (Chinese, emoji) is pasted, because X keyboards can't type characters they have no key for in
/// every program: both selections hold it (xclip keeps serving after the exec ends, setsid), and
/// terminals paste with shift+Insert, everything else with ctrl+v.
fn type_script(text: &str) -> String {
if text.is_ascii() {
return format!("xdotool type --clearmodifiers --delay {TYPE_DELAY_MS} --file -");
}
"cat > /tmp/anybot-paste.txt && for s in clipboard primary; do setsid xclip -selection $s -i /tmp/anybot-paste.txt </dev/null >/dev/null 2>&1 & done; sleep 0.15; \
case \"$(xprop -id $(xdotool getactivewindow) WM_CLASS 2>/dev/null | tr A-Z a-z)\" in *term*) k=shift+Insert;; *) k=ctrl+v;; esac; xdotool key --clearmodifiers $k"
.to_string()
}
/// Run a step's script in the box; with `picture`, wait for the screen to settle and return it.
fn box_exec(script: &str, stdin: Option<&str>, picture: bool) -> Result<Option<image::RgbImage>, String> {
use std::io::{Read, Write};
let full = if picture { format!("{{ {script}; }} || exit $?; sleep {:.2}; import -window root png:- 2>/dev/null || true", crate::computer::SETTLE_MS as f64 / 1000.0) } else { script.to_string() };
let mut child = docker()
.args(["exec", "-i", "-u", "bot", "-e", "DISPLAY=:1", container(), "sh", "-c", &full])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.map_err(|e| format!("找不到 docker：{e}"))?;
{
let mut si = child.stdin.take().ok_or("寫不進 Bot 的電腦")?;
if let Some(t) = stdin {
si.write_all(t.as_bytes()).map_err(|e| e.to_string())?;
}
}
let mut so = child.stdout.take().ok_or("沒有輸出")?;
let mut se = child.stderr.take().ok_or("沒有輸出")?;
let out_t = std::thread::spawn(move || {
let mut v = vec![];
let _ = so.read_to_end(&mut v);
v
});
let err_t = std::thread::spawn(move || {
let mut v = vec![];
let _ = se.read_to_end(&mut v);
v
});
let start = Instant::now();
let status = loop {
if let Ok(Some(st)) = child.try_wait() {
break st;
}
if start.elapsed() > Duration::from_secs(30) {
let _ = child.kill();
let _ = child.wait();
return Err("這台電腦太久沒有回應".into());
}
std::thread::sleep(Duration::from_millis(5));
};
let out = out_t.join().unwrap_or_default();
let err = err_t.join().unwrap_or_default();
if !status.success() {
let e = String::from_utf8_lossy(&err).trim().to_string();
return Err(if e.is_empty() { "這一步沒有成功".into() } else { format!("xdotool：{e}") });
}
if !picture || out.is_empty() {
return Ok(None);
}
let img = image::load_from_memory(&out).map(|i| i.to_rgb8()).ok();
if let Some(i) = &img {
shared().lock().unwrap_or_else(|e| e.into_inner()).size = Some(i.dimensions());
}
Ok(img)
}
/// The bot's picture of the box: fitted to 1024 wide, JPEG, base64.
fn picture_of(img: image::RgbImage) -> Result<(String, String), String> {
let (w, h) = img.dimensions();
let (fw, fh, resize) = crate::computer::fit(w, h);
let img = if resize { image::imageops::resize(&img, fw, fh, image::imageops::FilterType::Triangle) } else { img };
let jpeg = crate::computer::encode_jpeg(&img).ok_or("截圖太大了，沒辦法送出")?;
Ok((format!("螢幕截圖（{fw}×{fh}，Linux 桌面）。座標用這張圖上的像素。"), base64::engine::general_purpose::STANDARD.encode(jpeg)))
}
/// Programs people name, as the box knows them.
fn box_program(app: &str) -> Option<&'static str> {
let a = app.trim().to_lowercase();
if ["chrome", "chromium", "google chrome", "browser", "瀏覽器", "網頁瀏覽器"].iter().any(|n| a == *n) {
return Some("chromium");
}
if ["terminal", "終端機", "xterm", "shell", "命令列"].iter().any(|n| a == *n) {
return Some("xterm");
}
None
}
/// Open a program on the box, or bring its window to the front. The browser is AnyBot's
/// browser there (the one browser_* drive), never a second copy fighting over its profile.
fn open_on_box(app: &str) -> Result<String, String> {
let name = app.trim();
if name.is_empty() {
return Err("請給要打開的程式名稱".into());
}
let prog = match box_program(name) {
Some(p) => p.to_string(),
None if name.chars().all(|c| c.is_ascii_alphanumeric() || "._-".contains(c)) => name.to_lowercase(),
None => return Err(format!("這台是 Linux 桌面，沒有「{name}」。可以打開的例如 chromium（瀏覽器）、xterm（終端機），或用指令工具安裝。")),
};
if prog == "chromium" {
browser_port()?;
}
let q = crate::tools::sh_quote(&prog);
// Front if it has a window; else start it and wait (up to 8 s) for one.
let script = format!(
"w=$(xdotool search --onlyvisible --classname {q} 2>/dev/null | tail -1); \
if [ -z \"$w\" ]; then command -v {q} >/dev/null || {{ echo \"沒有 {prog} 這個程式\" >&2; exit 3; }}; (setsid {q} >/dev/null 2>&1 &); \
for i in $(seq 1 40); do sleep 0.2; w=$(xdotool search --onlyvisible --classname {q} 2>/dev/null | tail -1); [ -n \"$w\" ] && break; done; fi; \
[ -n \"$w\" ] && xdotool windowactivate $w 2>/dev/null; true"
);
box_exec(&script, None, false)?;
Ok(format!("已打開「{prog}」，放在最前面。"))
}
/// Run a computer_use tool on the box. Returns (text, screenshot as base64 JPEG).
pub fn computer_run(name: &str, v: &Value) -> Result<(String, Option<String>), String> {
ensure_running()?;
let _busy = Busy::new(true);
match name {
"computer_use_check_permissions" => return Ok(("這台電腦不需要另外的權限：可以截圖，也可以操作滑鼠和鍵盤。".into(), None)),
"computer_use_screenshot" => return picture_of(grab()?).map(|(t, img)| (t, Some(img))),
"computer_use_read_app" => {
return Err("這台是 Linux 桌面，沒有讀 App 文字的方式：用 computer_use_screenshot 看畫面、用座標操作。網頁請用 browser_* 工具（有編號，比座標準）。".into());
}
_ => {}
}
if HELD_BY_PERSON.load(Ordering::SeqCst) {
return Err("使用者正在自己操作這台電腦（已取得控制權）。請等他交還控制權後再操作，或先截圖看看畫面。".into());
}
let (note, picture) = if name == "computer_use_open_app" {
let note = open_on_box(v["app"].as_str().unwrap_or(""))?;
std::thread::sleep(Duration::from_millis(crate::computer::SETTLE_MS));
(note, grab().ok())
} else {
let (script, stdin, note) = box_script(name, v, box_scale()?)?;
let picture = box_exec(&script, stdin.as_deref(), name != "computer_use_mouse_button")?;
(note, picture)
};
match picture.map(picture_of).transpose()? {
Some((t, img)) => Ok((format!("{note}\n{t}"), Some(img))),
None => Ok((note, None)),
}
}
// ---------- the viewer's own input (when the person takes control) ----------
// Same scripts, at the desktop's own pixels (the viewer shows it full size), no picture back.
fn viewer(name: &str, v: Value) {
if let Ok((script, stdin, _)) = box_script(name, &v, (1.0, 1.0)) {
let _ = box_exec(&script, stdin.as_deref(), false);
}
touch();
}
pub fn viewer_click(x: i64, y: i64, button: &str, count: u32) {
viewer("computer_use_click", serde_json::json!({"x": x, "y": y, "button": button, "count": count.clamp(1, 3)}));
}
pub fn viewer_move(x: i64, y: i64) {
viewer("computer_use_mouse_move", serde_json::json!({"x": x, "y": y}));
}
pub fn viewer_scroll(x: i64, y: i64, down: bool, steps: u32) {
viewer("computer_use_scroll", serde_json::json!({"x": x, "y": y, "direction": if down { "down" } else { "up" }, "amount": steps.clamp(1, 20)}));
}
/// A key from a viewer: an X key name (「BackSpace」「Return」, ctrl+c…) or the bot's names.
pub fn viewer_key(combo: &str) {
let c = combo.trim();
if c.is_empty() || !c.chars().all(|ch| ch.is_ascii_alphanumeric() || "+_-".contains(ch)) {
return;
}
// X key names pass straight through (they are what the viewers send).
let x = combo_to_x(c).unwrap_or_else(|_| c.to_string());
let _ = box_exec(&format!("xdotool key --clearmodifiers {x}"), None, false);
touch();
}
pub fn viewer_text(text: &str) {
if !text.is_empty() {
let _ = box_exec(&type_script(text), Some(text), false);
}
touch();
}
fn key_to_x(name: &str) -> Option<String> { fn key_to_x(name: &str) -> Option<String> {
let n = name.trim().to_lowercase(); let n = name.trim().to_lowercase();
let s = match n.as_str() { let s = match n.as_str() {
@ -1266,192 +1585,6 @@ fn button_no(b: &str) -> &'static str {
} }
} }
/// Run a computer_use tool on the box. Returns (text, screenshot as base64 JPEG).
pub fn computer_run(name: &str, v: &Value) -> Result<(String, Option<String>), String> {
ensure_running()?;
let _busy = Busy::new(true);
if HELD_BY_PERSON.load(Ordering::SeqCst) && name != "computer_use_screenshot" && name != "computer_use_check_permissions" {
return Err("使用者正在自己操作這台電腦（已取得控制權）。請等他交還控制權後再操作，或先截圖看看畫面。".into());
}
let scale = || shared().lock().unwrap().scale.ok_or_else(|| "請先用 computer_use_screenshot 看一下畫面，座標要用截圖上的位置。".to_string());
let pt = |v: &Value, s: (f64, f64)| -> Result<(String, String), String> {
let (Some(x), Some(y)) = (v["x"].as_f64(), v["y"].as_f64()) else { return Err("請給 x 和 y".into()) };
Ok((format!("{}", (x * s.0).round() as i64), format!("{}", (y * s.1).round() as i64)))
};
let with_mods = |mods: &[String], f: &dyn Fn() -> Result<String, String>| -> Result<String, String> {
// Track what went down: a failed keydown must release the keys already held, or a
// stuck modifier poisons every later click and keystroke in the box.
let mut pressed: Vec<&String> = vec![];
for m in mods {
if let Err(e) = xdo(&["keydown", m]) {
for p in pressed.iter().rev() {
let _ = xdo(&["keyup", p]);
}
return Err(e);
}
pressed.push(m);
}
let r = f();
for m in pressed.iter().rev() {
let _ = xdo(&["keyup", m]);
}
r
};
match name {
"computer_use_check_permissions" => Ok(("這台電腦不需要另外的權限：可以截圖，也可以操作滑鼠和鍵盤。".into(), None)),
"computer_use_screenshot" => {
let img = grab()?;
let (w, h) = img.dimensions();
let (fw, fh, resize) = crate::computer::fit(w, h);
let img = if resize { image::imageops::resize(&img, fw, fh, image::imageops::FilterType::Triangle) } else { img };
let jpeg = crate::computer::encode_jpeg(&img).ok_or("截圖太大了，沒辦法送出")?;
shared().lock().unwrap().scale = Some((w as f64 / fw.max(1) as f64, h as f64 / fh.max(1) as f64));
Ok((format!("螢幕截圖（{fw}×{fh}，Linux 桌面）。之後的座標請用這張圖上的像素位置。"), Some(base64::engine::general_purpose::STANDARD.encode(jpeg))))
}
"computer_use_click" => {
let (x, y) = pt(v, scale()?)?;
let n = v["count"].as_u64().unwrap_or(1).clamp(1, 3).to_string();
let b = button_no(v["button"].as_str().unwrap_or("left"));
let delay = CLICK_REPEAT_MS.to_string();
with_mods(&mods_of(v), &|| xdo(&["mousemove", "--sync", &x, &y, "click", "--repeat", &n, "--delay", &delay, b]))?;
Ok((format!("已{}。要確認結果可以再截圖。", crate::computer::describe(name, &v.to_string())), None))
}
"computer_use_mouse_move" => {
let (x, y) = pt(v, scale()?)?;
xdo(&["mousemove", "--sync", &x, &y])?;
Ok(("已移動滑鼠。".into(), None))
}
"computer_use_mouse_button" => {
let b = button_no(v["button"].as_str().unwrap_or("left"));
let action = v["action"].as_str().unwrap_or("press");
if !v["x"].is_null() {
let (x, y) = pt(v, scale()?)?;
xdo(&["mousemove", "--sync", &x, &y])?;
}
xdo(&[if action == "release" { "mouseup" } else { "mousedown" }, b])?;
Ok((if action == "release" { "已放開滑鼠鍵。" } else { "已按住滑鼠鍵。" }.into(), None))
}
"computer_use_scroll" => {
let amount = crate::computer::scroll_amount(v["amount"].as_i64());
let dir = v["direction"].as_str().unwrap_or("").trim().to_lowercase();
let b = match dir.as_str() {
"up" => "4",
"down" => "5",
"left" => "6",
"right" => "7",
_ => return Err(format!("方向「{dir}」要是 up、down、left 或 right")),
};
if !v["x"].is_null() {
let (x, y) = pt(v, scale()?)?;
xdo(&["mousemove", "--sync", &x, &y])?;
}
let n = amount.to_string();
with_mods(&mods_of(v), &|| xdo(&["click", "--repeat", &n, "--delay", "20", b]))?;
Ok((format!("已捲動 {dir} ×{amount}。"), None))
}
"computer_use_drag" => {
let s = scale()?;
// A point that fails to parse is an error, not a silently dropped waypoint.
let raw = v["path"].as_array().ok_or("拖曳要給 path：一個點陣，每個點有數字 x 和 y（用截圖上的座標）")?;
let path: Vec<(String, String)> = raw.iter().map(|p| pt(p, s)).collect::<Result<_, _>>()?;
if path.len() < 2 {
return Err("拖曳至少要兩個點（每個點要有數字 x 和 y，用截圖上的座標）".into());
}
let b = button_no(v["button"].as_str().unwrap_or("left"));
with_mods(&mods_of(v), &|| {
xdo(&["mousemove", "--sync", &path[0].0, &path[0].1])?;
xdo(&["mousedown", b])?;
std::thread::sleep(Duration::from_millis(40));
for (x, y) in &path[1..] {
xdo(&["mousemove", "--sync", x, y])?;
std::thread::sleep(Duration::from_millis(8));
}
xdo(&["mouseup", b])
})?;
Ok((format!("已拖曳（{} 個點）。", path.len()), None))
}
"computer_use_typing" => {
let text = v["value"].as_str().unwrap_or("");
if text.is_empty() {
return Err("請給要打的字".into());
}
type_text(text)?;
Ok((format!("已打字（{} 個字）。", text.chars().count()), None))
}
"computer_use_press_key" => {
let keys = v["key"].as_str().unwrap_or("");
let combos: Vec<String> = keys.split_whitespace().map(combo_to_x).collect::<Result<_, _>>()?;
if combos.is_empty() {
return Err("請給要按的鍵".into());
}
let hold = v["hold_duration_ms"].as_u64().unwrap_or(0).min(5_000);
for c in &combos {
if hold > 0 {
xdo(&["keydown", c])?;
std::thread::sleep(Duration::from_millis(hold));
xdo(&["keyup", c])?;
} else {
xdo(&["key", "--clearmodifiers", c])?;
}
std::thread::sleep(Duration::from_millis(20));
}
Ok((format!("已按 {keys}。"), None))
}
other => Err(format!("沒有「{other}」這個工具")),
}
}
/// Type text. Plain ASCII goes key by key; anything else (Chinese, emoji) is pasted, because
/// X keyboards can't type characters they have no key for in every program.
pub fn type_text(text: &str) -> Result<(), String> {
if text.is_ascii() {
let d = TYPE_DELAY_MS.to_string();
xdo(&["type", "--clearmodifiers", "--delay", &d, "--", text])?;
return Ok(());
}
use std::io::Write;
// Both selections hold the text; xclip keeps serving it after docker exec returns (setsid).
let mut child = docker()
.args(["exec", "-i", "-u", "bot", "-e", "DISPLAY=:1", container(), "sh", "-c", "cat > /tmp/anybot-paste.txt && for s in clipboard primary; do setsid xclip -selection $s -i /tmp/anybot-paste.txt </dev/null >/dev/null 2>&1 & done; sleep 0.15"])
.stdin(Stdio::piped())
.stdout(Stdio::null())
.stderr(Stdio::null())
.spawn()
.map_err(|e| e.to_string())?;
child.stdin.take().ok_or("寫不進剪貼簿")?.write_all(text.as_bytes()).map_err(|e| e.to_string())?;
let _ = child.wait();
let class = dk(&["exec", "-u", "bot", "-e", "DISPLAY=:1", container(), "sh", "-c", "xprop -id $(xdotool getactivewindow) WM_CLASS 2>/dev/null"]).unwrap_or_default().to_lowercase();
let paste = if class.contains("term") { "shift+Insert" } else { "ctrl+v" };
xdo(&["key", "--clearmodifiers", paste])?;
Ok(())
}
// ---------- the viewer's own input (when the person takes control) ----------
pub fn viewer_click(x: i64, y: i64, button: &str, count: u32) {
let (x, y, n, b) = (x.to_string(), y.to_string(), count.clamp(1, 3).to_string(), button_no(button));
let _ = xdo(&["mousemove", &x, &y, "click", "--repeat", &n, "--delay", "80", b]);
touch();
}
pub fn viewer_move(x: i64, y: i64) {
let _ = xdo(&["mousemove", &x.to_string(), &y.to_string()]);
}
pub fn viewer_scroll(x: i64, y: i64, down: bool, steps: u32) {
let _ = xdo(&["mousemove", &x.to_string(), &y.to_string(), "click", "--repeat", &steps.clamp(1, 10).to_string(), if down { "5" } else { "4" }]);
}
pub fn viewer_key(combo: &str) {
let _ = xdo(&["key", "--clearmodifiers", combo]);
touch();
}
pub fn viewer_text(text: &str) {
let _ = type_text(text);
touch();
}
// ---------- the browser viewer (noVNC) ---------- // ---------- the browser viewer (noVNC) ----------
fn published_port() -> u16 { fn published_port() -> u16 {
@ -1523,6 +1656,83 @@ mod tests {
assert!(combo_to_x("hyper+x").is_err()); assert!(combo_to_x("hyper+x").is_err());
} }
#[test]
fn a_step_is_one_script() {
let s = (1.25, 1.25);
// Picture pixels → desktop pixels; never `mousemove --sync` (it hangs when the pointer is
// already there).
let (sc, input, note) = box_script("computer_use_click", &serde_json::json!({"x": 512, "y": 320}), s).unwrap();
assert_eq!(sc, "xdotool mousemove 640 400 sleep 0.03 click --repeat 1 --delay 80 1");
assert!(input.is_none() && note.contains("(512, 320)"));
// Keys held for a step are let go even when it fails.
let (sc, _, _) = box_script("computer_use_click", &serde_json::json!({"x": 8, "y": 8, "modifier_keys": ["cmd", "shift"], "count": 2}), s).unwrap();
assert!(sc.starts_with("xdotool keydown ctrl keydown shift mousemove 10 10") && sc.ends_with("rc=$?; xdotool keyup shift keyup ctrl; [ $rc = 0 ]"), "{sc}");
assert!(!sc.contains("--sync"));
// Text goes in on stdin (no quoting to get wrong); return after it when asked.
let (sc, input, _) = box_script("computer_use_typing", &serde_json::json!({"value": "echo 'hi' && ls", "submit": true}), s).unwrap();
assert_eq!(input.as_deref(), Some("echo 'hi' && ls"));
assert!(sc.starts_with("xdotool type --clearmodifiers --delay 12 --file -") && sc.ends_with("xdotool key --clearmodifiers Return"), "{sc}");
let (sc, input, _) = box_script("computer_use_typing", &serde_json::json!({"value": "中文"}), s).unwrap();
assert!(sc.contains("xclip") && input.as_deref() == Some("中文"));
// Keys by name, a character typed, several in order.
let (sc, _, _) = box_script("computer_use_press_key", &serde_json::json!({"key": "slash @ cmd+a"}), s).unwrap();
assert_eq!(sc, "xdotool key --clearmodifiers slash sleep 0.02 type --clearmodifiers -- '@' sleep 0.02 key --clearmodifiers ctrl+a");
// A drag walks its path in small steps and lets go at the end.
let (sc, _, _) = box_script("computer_use_drag", &serde_json::json!({"path": [{"x": 0, "y": 0}, {"x": 48, "y": 0}]}), s).unwrap();
assert!(sc.starts_with("xdotool mousemove 0 0 sleep 0.03 mousedown 1") && sc.contains("mousemove 60 0") && sc.ends_with("mouseup 1"), "{sc}");
// No app numbers on this desktop: said, not guessed.
assert!(box_script("computer_use_click", &serde_json::json!({"ref": "e3"}), s).is_err());
assert!(box_script("computer_use_scroll", &serde_json::json!({"direction": "sideways"}), s).is_err());
}
/// Uses the real Docker: `cargo test --release -- --ignored box_screen_is_exact --nocapture`.
/// The bot's clicks land where its picture says, a phone's tap where the phone's picture says,
/// the same spot twice doesn't hang, typing (ASCII, Chinese, a key by name) reaches the app,
/// and every step comes back with the screen after it.
#[test]
#[ignore]
fn box_screen_is_exact() {
ensure_running().expect("box starts");
let pointer = || {
let out = dk(&["exec", "-u", "bot", "-e", "DISPLAY=:1", container(), "xdotool", "getmouselocation"]).unwrap();
let num = |k: &str| out.split_whitespace().find_map(|w| w.strip_prefix(&format!("{k}:")).and_then(|v| v.parse::<i64>().ok())).unwrap();
(num("x"), num("y"))
};
let (w, h) = box_size().unwrap();
let (fw, fh, _) = crate::computer::fit(w, h);
println!("desktop {w}×{h}, picture {fw}×{fh}");
// A move, in the bot's picture pixels, with no screenshot taken first.
shared().lock().unwrap().size = None;
let (txt, img) = computer_run("computer_use_mouse_move", &serde_json::json!({"x": fw / 2, "y": fh / 2})).unwrap();
assert!(img.is_some(), "a picture comes back: {txt}");
assert_eq!(pointer(), ((w / 2) as i64, (h / 2) as i64));
// The same spot twice: quick, not a two-minute hang.
let t = Instant::now();
computer_run("computer_use_click", &serde_json::json!({"x": 40, "y": 40})).unwrap();
computer_run("computer_use_click", &serde_json::json!({"x": 40, "y": 40})).unwrap();
println!("two clicks with pictures: {} ms", t.elapsed().as_millis());
assert!(t.elapsed() < Duration::from_secs(6));
assert_eq!(pointer(), ((40.0 * w as f64 / fw as f64).round() as i64, (40.0 * h as f64 / fh as f64).round() as i64));
// A phone's tap is in the full-size frame it was sent.
viewer_click(321, 123, "left", 1);
assert_eq!(pointer(), (321, 123));
// Typing reaches the app in front: a fresh terminal, ASCII then Chinese, a key by name, return.
let _ = dk(&["exec", "-u", "bot", container(), "rm", "-f", "/home/bot/exact.txt"]);
let (txt, img) = computer_run("computer_use_open_app", &serde_json::json!({"app": "終端機"})).unwrap();
println!("{txt}");
assert!(img.is_some());
computer_run("computer_use_typing", &serde_json::json!({"value": "echo 'a-b' "})).unwrap();
computer_run("computer_use_typing", &serde_json::json!({"value": "中文"})).unwrap();
computer_run("computer_use_press_key", &serde_json::json!({"key": "space slash"})).unwrap();
let (txt, img) = computer_run("computer_use_typing", &serde_json::json!({"value": " > /home/bot/exact.txt", "submit": true})).unwrap();
assert!(img.is_some() && txt.contains("return"));
std::thread::sleep(Duration::from_millis(500));
let out = dk(&["exec", "-u", "bot", container(), "cat", "/home/bot/exact.txt"]).unwrap_or_default();
println!("typed: {out}");
assert_eq!(out.trim(), "a-b 中文 /");
let _ = box_exec("xdotool key --clearmodifiers ctrl+d", None, false);
}
/// Uses the real Docker: `cargo test --release -- --ignored box_end_to_end --nocapture`. /// Uses the real Docker: `cargo test --release -- --ignored box_end_to_end --nocapture`.
#[test] #[test]
#[ignore] #[ignore]

635
src/computer.rs
View File

@ -12,7 +12,10 @@ use base64::Engine;
use serde_json::{json, Value}; use serde_json::{json, Value};
use std::sync::Mutex; use std::sync::Mutex;
pub const NAMES: [&str; 12] = ["computer_use_screenshot", "computer_use_click", "computer_use_typing", "computer_use_press_key", "computer_use_scroll", "computer_use_mouse_move", "computer_use_drag", "computer_use_mouse_button", "computer_use_check_permissions", "computer_use_read_app", "computer_use_select", "computer_use_open_app"]; pub const NAMES: [&str; 11] = ["computer_use_screenshot", "computer_use_click", "computer_use_typing", "computer_use_press_key", "computer_use_scroll", "computer_use_mouse_move", "computer_use_drag", "computer_use_mouse_button", "computer_use_check_permissions", "computer_use_read_app", "computer_use_open_app"];
/// Wait after an action before taking the picture that comes back with it (menus open, pages
/// start to change; estimated).
pub const SETTLE_MS: u64 = 500;
/// The person took control back from the bot (the floating bar): screen actions wait. /// The person took control back from the bot (the floating bar): screen actions wait.
pub static HELD_BY_PERSON: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false); pub static HELD_BY_PERSON: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
@ -121,17 +124,75 @@ pub fn utf16_chunks(text: &str) -> Vec<Vec<u16>> {
out out
} }
/// One key of a press: a key with its modifiers, or a character typed (「@」「？」 have no key of
/// their own on every keyboard; refusing them sent the bot looking for workarounds).
#[derive(Debug, PartialEq)]
pub enum Key {
Combo(u16, u64),
Char(String),
}
/// "cmd+a delete" → each key of one press, in order (handlePressKey presses a list of keys). /// "cmd+a delete" → each key of one press, in order (handlePressKey presses a list of keys).
pub fn parse_keys(keys: &str) -> Result<Vec<(u16, u64)>, String> { pub fn key_plan(keys: &str) -> Result<Vec<Key>, String> {
let list: Vec<&str> = keys.split_whitespace().collect(); let list: Vec<&str> = keys.split_whitespace().collect();
if list.is_empty() { if list.is_empty() {
return Err("請給要按的鍵".into()); return Err("請給要按的鍵".into());
} }
list.into_iter().map(parse_combo).collect() list.into_iter()
.map(|k| match parse_combo(k) {
// A letter, digit or symbol without a modifier is a character: typed, so an input
// method (注音) can't turn it into a syllable. Named symbol keys mean their character.
Ok((c, 0)) if printable_of(c).is_some() => Ok(Key::Char(printable_of(c).unwrap_or_default())),
Ok((c, f)) => Ok(Key::Combo(c, f)),
Err(_) if k.chars().count() == 1 => Ok(Key::Char(k.to_string())),
Err(e) => Err(e),
})
.collect()
} }
/// Screen points per screenshot pixel, from the last screenshot (x, y). /// The character a printing key gives without shift (US layout), or None for keys like return.
static SCALE: Mutex<Option<(f64, f64)>> = Mutex::new(None); fn printable_of(code: u16) -> Option<String> {
const LETTERS: [u16; 26] = [0, 11, 8, 2, 14, 3, 5, 4, 34, 38, 40, 37, 46, 45, 31, 35, 12, 15, 1, 17, 32, 9, 13, 7, 16, 6];
const DIGITS: [u16; 10] = [29, 18, 19, 20, 21, 23, 22, 26, 28, 25];
if let Some(i) = LETTERS.iter().position(|c| *c == code) {
return Some(((b'a' + i as u8) as char).to_string());
}
if let Some(i) = DIGITS.iter().position(|c| *c == code) {
return Some(((b'0' + i as u8) as char).to_string());
}
let c = match code {
27 => '-',
24 => '=',
33 => '[',
30 => ']',
41 => ';',
39 => '\'',
43 => ',',
47 => '.',
44 => '/',
42 => '\\',
50 => '`',
_ => return None,
};
Some(c.to_string())
}
/// The picture the bot and the phone get of this screen: its size, always the same for the same
/// screen. Coordinates on it map to the screen without remembering any earlier screenshot (that
/// hidden state made clicks fail with "screenshot first" and land wrong for a viewer whose
/// picture was a different size).
pub fn shot_size() -> (u32, u32) {
let (px, py) = mac::screen_pixels();
let (w, h, _) = fit(px.max(1), py.max(1));
(w, h)
}
/// Screen points (the mouse's units) per picture pixel.
pub fn scale_here() -> (f64, f64) {
let (pw, ph) = mac::screen_points();
let (w, h) = shot_size();
(pw / w.max(1) as f64, ph / h.max(1) as f64)
}
pub fn is_computer_tool(name: &str) -> bool { pub fn is_computer_tool(name: &str) -> bool {
NAMES.contains(&name) NAMES.contains(&name)
@ -149,7 +210,6 @@ pub fn label(name: &str) -> &'static str {
"computer_use_mouse_button" => "按住或放開滑鼠", "computer_use_mouse_button" => "按住或放開滑鼠",
"computer_use_check_permissions" => "檢查螢幕權限", "computer_use_check_permissions" => "檢查螢幕權限",
"computer_use_read_app" => "讀 App 畫面", "computer_use_read_app" => "讀 App 畫面",
"computer_use_select" => "選擇",
"computer_use_open_app" => "打開 App", "computer_use_open_app" => "打開 App",
_ => "操作畫面", _ => "操作畫面",
} }
@ -178,7 +238,7 @@ pub fn describe(name: &str, args: &str) -> String {
let t = v["value"].as_str().unwrap_or(""); let t = v["value"].as_str().unwrap_or("");
let short: String = t.chars().take(60).collect(); let short: String = t.chars().take(60).collect();
let at = v["ref"].as_str().filter(|r| !r.trim().is_empty()).map(|r| format!("在 {r} ")).unwrap_or_default(); let at = v["ref"].as_str().filter(|r| !r.trim().is_empty()).map(|r| format!("在 {r} ")).unwrap_or_default();
format!("{at}「{short}{}」", if t.chars().count() > 60 { "…" } else { "" }) format!("{at}「{short}{}」{}", if t.chars().count() > 60 { "…" } else { "" }, if v["submit"].as_bool().unwrap_or(false) { " 再按 return" } else { "" })
} }
"computer_use_press_key" => v["key"].as_str().unwrap_or("").to_string(), "computer_use_press_key" => v["key"].as_str().unwrap_or("").to_string(),
"computer_use_scroll" => format!("{} {} 格，在 {}", v["direction"].as_str().unwrap_or("down"), v["amount"].as_i64().unwrap_or(3), xy()), "computer_use_scroll" => format!("{} {} 格，在 {}", v["direction"].as_str().unwrap_or("down"), v["amount"].as_i64().unwrap_or(3), xy()),
@ -187,7 +247,6 @@ pub fn describe(name: &str, args: &str) -> String {
"computer_use_mouse_button" => format!("{}{}", if v["action"].as_str() == Some("release") { "放開" } else { "按住" }, if v["x"].is_null() { String::new() } else { format!(" {}", xy()) }), "computer_use_mouse_button" => format!("{}{}", if v["action"].as_str() == Some("release") { "放開" } else { "按住" }, if v["x"].is_null() { String::new() } else { format!(" {}", xy()) }),
"computer_use_check_permissions" => "螢幕錄製、輔助使用".into(), "computer_use_check_permissions" => "螢幕錄製、輔助使用".into(),
"computer_use_read_app" | "computer_use_open_app" => v["app"].as_str().filter(|a| !a.is_empty()).unwrap_or("最前面的 App").to_string(), "computer_use_read_app" | "computer_use_open_app" => v["app"].as_str().filter(|a| !a.is_empty()).unwrap_or("最前面的 App").to_string(),
"computer_use_select" => format!("{} → {}", v["ref"].as_str().unwrap_or(""), v["value"].as_str().unwrap_or("")),
_ => String::new(), _ => String::new(),
} }
} }
@ -197,29 +256,29 @@ pub fn specs() -> Vec<Value> {
specs_for(&[], "") specs_for(&[], "")
} }
/// The screen tools, with a `computer` choice when the bot has its own desktop too. /// The screen tools, each with a `computer` choice. One way to act (a real click or keystroke),
/// two ways to see: the picture, or an app's controls as numbered text.
pub fn specs_for(computers: &[String], default: &str) -> Vec<Value> { pub fn specs_for(computers: &[String], default: &str) -> Vec<Value> {
let mods = json!({"type": "array", "items": {"type": "string", "enum": ["cmd", "shift", "option", "control"]}}); let mods = json!({"type": "array", "items": {"type": "string", "enum": ["cmd", "shift", "option", "control"]}});
let rf = json!({"type": "string", "description": "A control's number from computer_use_read_app (e.g. e12), instead of x, y"});
let mut v = vec![ let mut v = vec![
json!({"name": "computer_use_read_app", "description": "Read an app's window as text — its buttons, fields, menus, list items and visible text, each control numbered (e1, e2…). No screenshot or image ability needed, and acting by number happens in the background without moving the person's mouse. Prefer this over screenshots for normal apps (Finder, Mail, LINE, Excel, Settings…); use screenshots for pictures, charts, games or apps that show nothing here. app: the app's name (as in the Dock, e.g. Calculator / 計算機 / Finder); leave out for the app in front.", json!({"name": "computer_use_screenshot", "description": "See a computer's screen. x, y in the other computer_use tools are pixels of this picture. Every action below already returns a fresh picture of the result, so call this only to start or to look again.",
"parameters": {"type": "object", "properties": {"app": {"type": "string"}}}}),
json!({"name": "computer_use_select", "description": "Choose an option in a popup menu, dropdown or list by the control's number from computer_use_read_app.",
"parameters": {"type": "object", "properties": {"ref": {"type": "string"}, "value": {"type": "string", "description": "The option's text"}}, "required": ["ref", "value"]}}),
json!({"name": "computer_use_open_app", "description": "Open an app (or bring it forward if it's running), then read it with computer_use_read_app.",
"parameters": {"type": "object", "properties": {"app": {"type": "string"}}, "required": ["app"]}}),
json!({"name": "computer_use_screenshot", "description": "Take a screenshot of a computer's screen to see what's on it. Coordinates for the other computer_use tools are pixels of the latest screenshot of that computer. Take one before clicking, and again to check the result. Needs a model that can see images.",
"parameters": {"type": "object", "properties": {}}}), "parameters": {"type": "object", "properties": {}}}),
json!({"name": "computer_use_click", "description": "Click a control by its number from computer_use_read_app (ref, e.g. e12), or on the screen at x, y (pixels of the latest screenshot). background=true clicks a window on the person's Mac without bringing it to the front.", json!({"name": "computer_use_click", "description": "Click at x, y (pixels of the latest picture), or on a control by its number (ref) from computer_use_read_app. Returns the screen after the click.",
"parameters": {"type": "object", "properties": {"ref": {"type": "string"}, "x": {"type": "number"}, "y": {"type": "number"}, "button": {"type": "string", "enum": ["left", "right", "middle"]}, "count": {"type": "integer", "description": "2 for a double click"}, "modifier_keys": mods, "background": {"type": "boolean"}}}}), "parameters": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}, "ref": rf, "button": {"type": "string", "enum": ["left", "right", "middle"]}, "count": {"type": "integer", "description": "2 for a double click"}, "modifier_keys": mods}}}),
json!({"name": "computer_use_typing", "description": "Type text (any language) into a field by its number from computer_use_read_app (ref), or where the cursor is.", json!({"name": "computer_use_typing", "description": "Type text (any language) where the cursor is, or into a field by its number (ref; it is clicked first). submit=true presses return after the text (send a message, run a command). Returns the screen after.",
"parameters": {"type": "object", "properties": {"value": {"type": "string"}, "ref": {"type": "string"}}, "required": ["value"]}}), "parameters": {"type": "object", "properties": {"value": {"type": "string"}, "ref": rf, "submit": {"type": "boolean"}}, "required": ["value"]}}),
json!({"name": "computer_use_press_key", "description": "Press a key or shortcut, e.g. return, tab, escape, delete, up, cmd+c, cmd+shift+t, cmd+space. Several keys in order: separate with spaces, e.g. \"cmd+a delete\". On the bot's Linux computer cmd means ctrl. app: send it to that app in the background instead of the app in front.", json!({"name": "computer_use_press_key", "description": "Press a key or shortcut in the app in front: return, tab, escape, delete, up, slash, cmd+c, cmd+shift+t, cmd+space. Several in order: separate with spaces (\"cmd+a delete\"). On a Linux computer cmd means ctrl. Returns the screen after.",
"parameters": {"type": "object", "properties": {"key": {"type": "string"}, "hold_duration_ms": {"type": "integer"}, "app": {"type": "string"}}, "required": ["key"]}}), "parameters": {"type": "object", "properties": {"key": {"type": "string"}, "hold_duration_ms": {"type": "integer"}}, "required": ["key"]}}),
json!({"name": "computer_use_scroll", "description": "Scroll a list or area by its number from computer_use_read_app (ref), or at x, y (give both or neither; neither = where the pointer is).", json!({"name": "computer_use_scroll", "description": "Scroll at x, y or on a control by number (ref); neither = where the pointer is. Returns the screen after.",
"parameters": {"type": "object", "properties": {"ref": {"type": "string"}, "x": {"type": "number"}, "y": {"type": "number"}, "direction": {"type": "string", "enum": ["up", "down", "left", "right"]}, "amount": {"type": "integer", "description": "Scroll steps of 16 pixels, 1-20, default 3"}, "modifier_keys": mods}, "required": ["direction"]}}), "parameters": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}, "ref": rf, "direction": {"type": "string", "enum": ["up", "down", "left", "right"]}, "amount": {"type": "integer", "description": "Scroll steps of 16 pixels, 1-20, default 3"}, "modifier_keys": mods}, "required": ["direction"]}}),
json!({"name": "computer_use_mouse_move", "description": "Move the mouse to x, y (e.g. to show a hover menu).", json!({"name": "computer_use_open_app", "description": "Open an app, or bring it to the front when it is already open (its name as shown, e.g. Ghostty, Calculator, 計算機, Chromium, xterm). Returns the screen after.",
"parameters": {"type": "object", "properties": {"app": {"type": "string"}}, "required": ["app"]}}),
json!({"name": "computer_use_read_app", "description": "Read an app's window as text: buttons, fields, menus, list items and visible text, each control numbered (e1, e2…) for click/typing/scroll's ref. For when a picture can't be taken or isn't enough (exact text, long lists). app: its name; leave out for the app in front. Only on computers that offer it (not the bots' own Linux computer).",
"parameters": {"type": "object", "properties": {"app": {"type": "string"}}}}),
json!({"name": "computer_use_mouse_move", "description": "Move the mouse to x, y (e.g. to show a hover menu). Returns the screen after.",
"parameters": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}}, "required": ["x", "y"]}}), "parameters": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}}, "required": ["x", "y"]}}),
json!({"name": "computer_use_drag", "description": "Drag with the mouse along a path of points (first = press, last = release).", json!({"name": "computer_use_drag", "description": "Drag with the mouse along a path of points (first = press, last = release). Returns the screen after.",
"parameters": {"type": "object", "properties": {"path": {"type": "array", "items": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}}, "required": ["x", "y"]}}, "button": {"type": "string", "enum": ["left", "right"]}, "modifier_keys": mods}, "required": ["path"]}}), "parameters": {"type": "object", "properties": {"path": {"type": "array", "items": {"type": "object", "properties": {"x": {"type": "number"}, "y": {"type": "number"}}, "required": ["x", "y"]}}, "button": {"type": "string", "enum": ["left", "right"]}, "modifier_keys": mods}, "required": ["path"]}}),
json!({"name": "computer_use_mouse_button", "description": "Press and hold, or release, a mouse button (optionally moving to x, y first). For long presses and custom drags.", json!({"name": "computer_use_mouse_button", "description": "Press and hold, or release, a mouse button (optionally moving to x, y first). For long presses and custom drags.",
"parameters": {"type": "object", "properties": {"action": {"type": "string", "enum": ["press", "release"]}, "button": {"type": "string", "enum": ["left", "right", "middle"]}, "x": {"type": "number"}, "y": {"type": "number"}}, "required": ["action"]}}), "parameters": {"type": "object", "properties": {"action": {"type": "string", "enum": ["press", "release"]}, "button": {"type": "string", "enum": ["left", "right", "middle"]}, "x": {"type": "number"}, "y": {"type": "number"}}, "required": ["action"]}}),
@ -315,6 +374,18 @@ pub fn key_code(name: &str) -> Option<u16> {
"f10" => 109, "f10" => 109,
"f11" => 103, "f11" => 103,
"f12" => 111, "f12" => 111,
// Symbol keys by name (a model writes "slash" as often as "/").
"slash" => 44,
"backslash" => 42,
"comma" => 43,
"period" | "dot" => 47,
"semicolon" => 41,
"quote" | "apostrophe" => 39,
"minus" | "hyphen" => 27,
"equal" | "equals" => 24,
"grave" | "backtick" | "backquote" => 50,
"bracketleft" | "leftbracket" => 33,
"bracketright" | "rightbracket" => 30,
_ => u16::MAX, _ => u16::MAX,
}; };
if named != u16::MAX { if named != u16::MAX {
@ -381,23 +452,18 @@ pub fn run(name: &str, args: &str) -> (String, bool, Option<String>) {
/// Same, remembering which bot acted (for the 「正在操作你的電腦」 bar). /// Same, remembering which bot acted (for the 「正在操作你的電腦」 bar).
pub fn run_by(name: &str, args: &str, who: &str) -> (String, bool, Option<String>) { pub fn run_by(name: &str, args: &str, who: &str) -> (String, bool, Option<String>) {
if name != "computer_use_screenshot" && name != "computer_use_check_permissions" { if moves_things(name) {
*LAST_ACTION.lock().unwrap_or_else(|e| e.into_inner()) = Some((std::time::Instant::now(), who.to_string())); *LAST_ACTION.lock().unwrap_or_else(|e| e.into_inner()) = Some((std::time::Instant::now(), who.to_string()));
} }
let r = run_inner(name, &args_of(args)); match run_inner(name, &args_of(args)) {
match r {
Ok((t, img)) => (t, true, img), Ok((t, img)) => (t, true, img),
Err(e) => (e, false, None), Err(e) => (e, false, None),
} }
} }
fn run_text_mode_name(name: &str, v: &Value) -> Option<()> { /// Steps that use the mouse or keyboard (and come back with the screen after them).
let has_ref = v["ref"].as_str().map(|r| !r.trim().is_empty()).unwrap_or(false); pub fn moves_things(name: &str) -> bool {
match name { matches!(name, "computer_use_click" | "computer_use_typing" | "computer_use_press_key" | "computer_use_scroll" | "computer_use_mouse_move" | "computer_use_drag" | "computer_use_mouse_button" | "computer_use_open_app")
"computer_use_open_app" | "computer_use_select" => Some(()),
"computer_use_click" | "computer_use_typing" | "computer_use_scroll" if has_ref => Some(()),
_ => None,
}
} }
/// Ask macOS for a permission (its own prompt, then 系統設定 at that pane) at most once every ten /// Ask macOS for a permission (its own prompt, then 系統設定 at that pane) at most once every ten
@ -430,18 +496,20 @@ pub fn ask_permission(pane: &str) {
let _ = spawn_reaped(&mut std::process::Command::new("open").arg(format!("x-apple.systempreferences:com.apple.preference.security?{pane}"))); let _ = spawn_reaped(&mut std::process::Command::new("open").arg(format!("x-apple.systempreferences:com.apple.preference.security?{pane}")));
} }
const NO_ACCESSIBILITY: &str = "macOS 還沒讓 AnyBot 使用「輔助使用」，所以讀不到、也按不了這台電腦上的 App。這台電腦上已經跳出詢問並打開「系統設定 → 隱私權與安全性 → 輔助使用」：請使用者把 AnyBot 打開（只要一次），再叫我試一次。"; const NO_ACCESSIBILITY: &str = "macOS 還沒讓 AnyBot 使用「輔助使用」，所以不能讀 App，也不能動滑鼠和鍵盤。這台電腦上已經跳出詢問並打開「系統設定 → 隱私權與安全性 → 輔助使用」：請使用者把 AnyBot 打開（只要一次），再叫我試一次。";
const NO_SCREEN_RECORDING: &str = "macOS 還沒讓 AnyBot 錄製螢幕，所以看不到畫面。這台電腦上已經打開「系統設定 → 隱私權與安全性 → 螢幕與系統錄音」：請使用者把 AnyBot 打開（只要一次）。在那之前可以用 computer_use_read_app 讀 App 的文字。";
/// agent-desktop's PERM_DENIED (no 輔助使用), as words the bot can pass on — and the person asked. /// Reading apps and using the mouse and keyboard need 輔助使用 on macOS (other systems: always).
fn plain_permission_words(t: String) -> String { fn controls_allowed() -> bool {
if !t.contains("PERM_DENIED") { !cfg!(target_os = "macos") || crate::axread::trusted(false)
return t; }
fn need_controls() -> Result<(), String> {
if controls_allowed() {
return Ok(());
} }
ask_permission("Privacy_Accessibility"); ask_permission("Privacy_Accessibility");
match t.split_once("，但讀不到它的畫面") { Err(NO_ACCESSIBILITY.into())
Some((opened, _)) => format!("{opened}，但讀不到它的畫面：{NO_ACCESSIBILITY}"),
None => NO_ACCESSIBILITY.to_string(),
}
} }
/// The app a shown name stands for (「計算機」 → Calculator), from the apps' display names. /// The app a shown name stands for (「計算機」 → Calculator), from the apps' display names.
@ -461,73 +529,179 @@ fn app_named(_name: &str) -> Option<String> {
None None
} }
/// The accessibility (text) way: agent-desktop, by element number, in the background. /// Open an app or bring it to the front. A name as the person sees it (「計算機」 on a Chinese
fn run_text_mode(name: &str, v: &Value) -> Option<Result<(String, Option<String>), String>> { /// Mac) isn't one the system launches by; find the app it is and launch that.
let rf = v["ref"].as_str().unwrap_or("").trim().to_string(); fn open_app(app: &str) -> Result<String, String> {
let r = match name { let app = app.trim();
"computer_use_read_app" => crate::desktop::read_app(v["app"].as_str().unwrap_or("")).map(|(app, lines)| { if app.is_empty() {
if lines.is_empty() { return Err("請給要打開的 App 名稱".into());
format!("「{app}」的畫面讀不到內容（有些 App 不提供輔助使用資訊）。可以改用 computer_use_screenshot 看。")
} else {
format!("「{app}」的畫面（e 編號可以用 computer_use_click / typing / select / scroll 的 ref）：\n{}", lines.join("\n"))
} }
}), crate::desktop::launch(app).or_else(|e| match app_named(app) {
"computer_use_open_app" => { Some(real) if real != app => crate::desktop::launch(&real),
let app = v["app"].as_str().unwrap_or("").trim().to_string();
// A name as the person sees it (「計算機」 on a Chinese Mac) isn't one the system
// launches by; find the app it is and launch that.
let launched = crate::desktop::run("launch", &json!({"app": app})).or_else(|e| match app_named(&app) {
Some(real) if real != app => crate::desktop::run("launch", &json!({"app": real})),
_ => Err(e), _ => Err(e),
}); })?;
launched.map(|_| { Ok(format!("已打開「{app}」，放在最前面。"))
std::thread::sleep(std::time::Duration::from_millis(800)); }
// Open is open, even when its window can't be read (that's a separate problem,
// and the bot said "couldn't open" while the app sat there on the screen). /// Bring the app a control belongs to in front, so a click at its spot lands in it (and not in
match crate::desktop::read_app(&app) { /// a window lying over it). Nothing to do when it is in front already.
Ok((a, lines)) => format!("已打開「{a}」。\n{}", lines.join("\n")), fn bring_forward(app: &str) {
Err(e) => format!("已打開「{app}」，但讀不到它的畫面：{e}"), // macOS only: there "launch" brings a running app forward; on Linux it can start a second
// copy, and the names a window and its accessibility give differ.
if !cfg!(target_os = "macos") {
return;
} }
}) let front = mac::front_app().map(|f| f.0).unwrap_or_default();
if app.is_empty() || front.eq_ignore_ascii_case(app) {
return;
} }
"computer_use_select" => crate::desktop::act("select", &rf, &json!({"value": v["value"]})), if crate::desktop::launch(app).is_ok() {
"computer_use_click" if !rf.is_empty() => { std::thread::sleep(std::time::Duration::from_millis(300));
let cmd = match (v["button"].as_str(), v["count"].as_u64()) { }
(Some("right"), _) => "right_click", }
(_, Some(2..)) => "double_click",
_ => "click", /// Where a step acts: a control's number (from computer_use_read_app), or x, y on the picture.
/// Returns the screen point and how to say it (「e7（按鈕「7」）」, 「(512, 300)」).
fn place_of(v: &Value, scale: (f64, f64)) -> Result<Option<(f64, f64, String)>, String> {
if let Some(r) = v["ref"].as_str().map(str::trim).filter(|r| !r.is_empty()) {
let s = crate::desktop::spot(r)?;
bring_forward(&s.app);
return Ok(Some((s.x, s.y, format!("{}（{}）", crate::desktop::short_ref(r)?, s.what))));
}
match (v["x"].as_f64(), v["y"].as_f64()) {
(Some(x), Some(y)) => {
let (sx, sy) = to_screen(x, y, scale);
Ok(Some((sx, sy, format!("({x}, {y})"))))
}
(None, None) if v["x"].is_null() && v["y"].is_null() => Ok(None),
_ => Err("x 和 y 要一起給，而且要是數字（截圖上的位置）".into()),
}
}
/// Press each key of "cmd+a delete" in order.
fn press_keys(keys: &str, hold: f64) -> Result<(), String> {
for (i, k) in key_plan(keys)?.iter().enumerate() {
if i > 0 {
std::thread::sleep(std::time::Duration::from_secs_f64(KEY_GAP_SECS));
}
match k {
Key::Combo(code, flags) => mac::press(*code, *flags, hold),
Key::Char(ch) => mac::type_text(ch),
}
}
Ok(())
}
/// Do one mouse or keyboard step at the screen's own coordinates (`scale`: screen points per
/// picture pixel). The bot's tools and a person tapping on the phone's picture both come here:
/// one way of acting on a screen.
pub fn act(name: &str, v: &Value, scale: (f64, f64)) -> Result<String, String> {
let button = v["button"].as_str().unwrap_or("left");
match name {
"computer_use_click" => {
let (x, y, at) = place_of(v, scale)?.ok_or("請給 x、y（截圖上的位置）或 ref（computer_use_read_app 的編號）")?;
let count = v["count"].as_u64().unwrap_or(1).clamp(1, 3) as i64;
mac::click(x, y, button, count, flags_of(v));
let how = format!("{}{}", if button == "right" { "右鍵" } else { "" }, if count >= 2 { format!("連點 {count} 下") } else { "點".into() });
Ok(format!("已{how} {at}。"))
}
"computer_use_mouse_move" => {
let (x, y, at) = place_of(v, scale)?.ok_or("請給 x 和 y")?;
mac::move_to(x, y);
Ok(format!("已把滑鼠移到 {at}。"))
}
"computer_use_scroll" => {
// CURemoteToolService.handleScroll @0x10009e200 / closure @0x10009e998.
let at = place_of(v, scale)?;
if !v["amount"].is_null() && v["amount"].as_i64().is_none() {
return Err("amount 要是整數".into());
}
let amount = scroll_amount(v["amount"].as_i64());
let dir = v["direction"].as_str().unwrap_or("").trim().to_lowercase();
let (dy, dx) = scroll_pixels(&dir, amount).ok_or_else(|| format!("方向「{dir}」要是 up、down、left 或 right"))?;
mac::scroll(at.as_ref().map(|(x, y, _)| (*x, *y)), dy, dx, flags_of(v));
Ok(format!("已捲動 {dir} ×{amount}{}。", at.map(|(_, _, a)| format!("，在 {a}")).unwrap_or_default()))
}
"computer_use_drag" => {
// Every point must be valid: points that failed to parse used to be dropped in
// silence, so the drag ran on fewer points and reported success at the wrong place.
let raw = v["path"].as_array().ok_or("拖曳要給 path：一個點陣，每個點有數字 x 和 y（用截圖上的座標）")?;
let pts: Vec<(f64, f64)> = raw.iter().map(|p| match (p["x"].as_f64(), p["y"].as_f64()) {
(Some(x), Some(y)) => Ok((x, y)),
_ => Err("拖曳的每個點都要有數字 x 和 y（用截圖上的座標）".to_string()),
}).collect::<Result<_, _>>()?;
if pts.len() < 2 {
return Err("拖曳至少要兩個點（每個點要有數字 x 和 y，用截圖上的座標）".into());
}
let path: Vec<(f64, f64)> = pts.iter().map(|(x, y)| to_screen(*x, *y, scale)).collect();
mac::drag(&path, button, flags_of(v));
// Like the original's "Dragged from (x, y) to (x, y) with left button."
let side = match button {
"right" => "右鍵",
"middle" => "中鍵",
_ => "左鍵",
}; };
crate::desktop::act(cmd, &rf, &json!({})) let (a, b) = (pts[0], pts[pts.len() - 1]);
Ok(format!("已從 ({:.0}, {:.0}) 拖曳到 ({:.0}, {:.0})（{side}）。", a.0, a.1, b.0, b.1))
} }
"computer_use_typing" if !rf.is_empty() => crate::desktop::act("type", &rf, &json!({"text": v["value"]})), "computer_use_mouse_button" => {
"computer_use_scroll" if !rf.is_empty() => crate::desktop::act("scroll", &rf, &json!({"direction": v["direction"], "amount": v["amount"]})), let at = place_of(v, scale)?.map(|(x, y, _)| (x, y));
"computer_use_press_key" if v["app"].as_str().map(|a| !a.trim().is_empty()).unwrap_or(false) => { let release = v["action"].as_str() == Some("release");
// agent-desktop's key names: cmd+c, return, escape… mac::mouse_button(at, button, !release);
let combo = v["key"].as_str().unwrap_or("").replace("command", "cmd").replace("option", "alt").replace("control", "ctrl"); Ok((if release { "已放開滑鼠鍵。" } else { "已按住滑鼠鍵（記得之後放開）。" }).into())
crate::desktop::run("press", &json!({"combo": combo, "app": v["app"]})).map(|_| format!("已在「{}」按 {}。", v["app"].as_str().unwrap_or(""), v["key"].as_str().unwrap_or("")))
} }
_ => return None, "computer_use_typing" => {
let text = v["value"].as_str().unwrap_or("");
let submit = v["submit"].as_bool().unwrap_or(false);
if text.is_empty() && !submit {
return Err("請給要打的字".into());
}
// Into a field by its number: click it first, like a person would.
let at = place_of(&json!({"ref": v["ref"]}), scale)?;
if let Some((x, y, _)) = &at {
mac::click(*x, *y, "left", 1, 0);
std::thread::sleep(std::time::Duration::from_millis(150));
}
if !text.is_empty() {
mac::type_text(text);
}
if submit {
std::thread::sleep(std::time::Duration::from_millis(60));
mac::press(36, 0, KEY_HOLD_SECS);
}
Ok(format!("已打字（{} 個字）{}{}。", text.chars().count(), at.map(|(_, _, a)| format!("在 {a}")).unwrap_or_default(), if submit { "，再按了 return" } else { "" }))
}
"computer_use_press_key" => {
let hold = v["hold_duration_ms"].as_u64().map(|ms| ms.min(MAX_HOLD_MS) as f64 / 1000.0).filter(|s| *s > 0.0).unwrap_or(KEY_HOLD_SECS);
press_keys(v["key"].as_str().unwrap_or(""), hold)?;
Ok(format!("已按 {}。", v["key"].as_str().unwrap_or("")))
}
"computer_use_open_app" => open_app(v["app"].as_str().unwrap_or("")),
other => Err(format!("沒有「{other}」這個工具")),
}
}
/// What the screen looks like now, after a step: the picture when this computer can take one,
/// else the app in front as numbered text. The bot sees the result of each step in the same
/// answer instead of spending another step (another trip to the model) on a screenshot.
fn observe(note: String) -> (String, Option<String>) {
std::thread::sleep(std::time::Duration::from_millis(SETTLE_MS));
if mac::screen_capture_preflight() {
return match screenshot() {
Ok((t, img)) => (format!("{note}\n{t}"), Some(img)),
Err(e) => (format!("{note}\n（看不到結果：{e}）"), None),
}; };
Some(r.map(|t| (t, None))) }
if controls_allowed() {
if let Ok((app, lines)) = crate::desktop::read_app("") {
let shown: Vec<&String> = lines.iter().take(120).collect();
return (format!("{note}\n（這台現在拍不到畫面，改讀最前面的「{app}」；e 編號可以用在 ref）\n{}", shown.iter().map(|l| l.as_str()).collect::<Vec<_>>().join("\n")), None);
}
}
(format!("{note}（這台現在看不到畫面，無法確認結果）"), None)
} }
fn run_inner(name: &str, v: &Value) -> Result<(String, Option<String>), String> { fn run_inner(name: &str, v: &Value) -> Result<(String, Option<String>), String> {
if HELD_BY_PERSON.load(std::sync::atomic::Ordering::SeqCst) && name != "computer_use_read_app" {
if let Some(_) = run_text_mode_name(name, v) {
return Err("使用者正在自己操作這台電腦（已取得控制權）。請等他交還控制權後再操作。".into());
}
}
// Reading and using apps by their controls needs 輔助使用: without it macOS answers
// PERM_DENIED and nobody was asked. Ask (prompt + the Settings pane, now and then) and say it
// plainly; opening an app still opens it.
let by_controls = name == "computer_use_read_app" || run_text_mode_name(name, v).is_some();
if cfg!(target_os = "macos") && by_controls && name != "computer_use_open_app" && !crate::axread::trusted(false) {
ask_permission("Privacy_Accessibility");
return Err(NO_ACCESSIBILITY.into());
}
if let Some(r) = run_text_mode(name, v) {
return r.map(|(t, img)| (plain_permission_words(t), img)).map_err(plain_permission_words);
}
if !cfg!(any(target_os = "macos", target_os = "linux")) { if !cfg!(any(target_os = "macos", target_os = "linux")) {
return Err("操作畫面只支援 macOS 和 Linux".into()); return Err("操作畫面只支援 macOS 和 Linux".into());
} }
@ -535,129 +709,81 @@ fn run_inner(name: &str, v: &Value) -> Result<(String, Option<String>), String>
if !crate::desk::available() { if !crate::desk::available() {
return Err(crate::desk::describe().into()); return Err(crate::desk::describe().into());
} }
if name == "computer_use_screenshot" { match name {
return screenshot().map(|(t, img)| (t, Some(img))); "computer_use_screenshot" => return screenshot().map(|(t, img)| (t, Some(img))),
} "computer_use_check_permissions" => {
#[cfg(target_os = "linux")] #[cfg(target_os = "linux")]
if name == "computer_use_check_permissions" {
return Ok((crate::desk::describe().into(), None)); return Ok((crate::desk::describe().into(), None));
} #[allow(unreachable_code)]
if name == "computer_use_check_permissions" { {
let see = mac::screen_capture_preflight(); let see = mac::screen_capture_preflight();
let control = crate::axread::trusted(false); let control = controls_allowed();
return Ok((format!("螢幕錄製：{}\n輔助使用（滑鼠和鍵盤）：{}{}", if see { "已允許" } else { "還沒允許" }, if control { "已允許" } else { "還沒允許" }, if see && control { "" } else { "\n請使用者到「系統設定 → 隱私權與安全性」打開 AnyBot（或「終端機」）。" }), None)); return Ok((format!("螢幕錄製（看畫面）：{}\n輔助使用（滑鼠、鍵盤、讀 App）：{}{}", if see { "已允許" } else { "還沒允許" }, if control { "已允許" } else { "還沒允許" }, if see && control { "" } else { "\n請使用者到「系統設定 → 隱私權與安全性」打開 AnyBot。" }), None));
}
}
"computer_use_read_app" => {
need_controls()?;
let (app, lines) = crate::desktop::read_app(v["app"].as_str().unwrap_or("")).map_err(plain_words)?;
return Ok((if lines.is_empty() {
format!("「{app}」的畫面讀不到內容（有些 App 不提供輔助使用資訊）。可以改用 computer_use_screenshot 看。")
} else {
format!("「{app}」的畫面（e 編號可以用在 click / typing / scroll 的 ref）：\n{}", lines.join("\n"))
}, None));
}
_ => {}
}
if !moves_things(name) {
return Err(format!("沒有「{name}」這個工具"));
} }
if HELD_BY_PERSON.load(std::sync::atomic::Ordering::SeqCst) { if HELD_BY_PERSON.load(std::sync::atomic::Ordering::SeqCst) {
return Err("使用者正在自己操作這台電腦（已取得控制權）。請等他交還控制權後再操作；可以先截圖看看。".into()); return Err("使用者正在自己操作這台電腦（已取得控制權）。請等他交還控制權後再操作；可以先截圖看看。".into());
} }
if cfg!(target_os = "macos") && !crate::axread::trusted(false) { if name != "computer_use_open_app" {
need_controls()?;
}
let note = act(name, v, scale_here()).map_err(plain_words)?;
// Holding a button down is the middle of something (a long press, a drag): no picture yet.
if name == "computer_use_mouse_button" {
return Ok((note, None));
}
Ok(observe(note))
}
/// agent-desktop's error codes, as words the bot can pass on.
fn plain_words(t: String) -> String {
if t.contains("PERM_DENIED") {
ask_permission("Privacy_Accessibility"); ask_permission("Privacy_Accessibility");
return Err("macOS 還沒讓 AnyBot 使用「輔助使用」，所以不能動滑鼠和鍵盤。我已經打開「系統設定 → 隱私權與安全性 → 輔助使用」：把 AnyBot（或「終端機」）打開，再叫我試一次。只要開一次。".into()); return NO_ACCESSIBILITY.to_string();
} }
let scale = || SCALE.lock().unwrap_or_else(|e| e.into_inner()).ok_or_else(|| "請先用 computer_use_screenshot 看一下畫面，座標要用截圖上的位置。".to_string()); if t.contains("APP_NOT_FOUND") {
let pt = |v: &Value, s: (f64, f64)| -> Result<(f64, f64), String> { return format!("找不到這個 App（沒有開著，或名字不對）：先用 computer_use_open_app 打開它。（{t}）");
let (Some(x), Some(y)) = (v["x"].as_f64(), v["y"].as_f64()) else { return Err("請給 x 和 y".into()) };
Ok(to_screen(x, y, s))
};
match name {
"computer_use_click" => {
let (x, y) = pt(v, scale()?)?;
let button = v["button"].as_str().unwrap_or("left");
let count = v["count"].as_u64().unwrap_or(1).clamp(1, 3) as i64;
if v["background"].as_bool().unwrap_or(false) {
let (app, pid) = mac::window_at(x, y).ok_or("那個位置沒有視窗")?;
mac::click_in_background(pid, x, y, button, count, flags_of(v));
return Ok((format!("已在背景點「{app}」的 ({}, {})，沒有把它拉到前面。要確認結果可以再截圖。", v["x"], v["y"]), None));
}
mac::click(x, y, button, count, flags_of(v));
Ok((format!("已{}。要確認結果可以再截圖。", describe(name, &v.to_string())), None))
}
"computer_use_mouse_move" => {
let (x, y) = pt(v, scale()?)?;
mac::move_to(x, y);
Ok(("已移動滑鼠。".into(), None))
}
"computer_use_scroll" => {
// CURemoteToolService.handleScroll @0x10009e200 / closure @0x10009e998.
let (has_x, has_y) = (!v["x"].is_null(), !v["y"].is_null());
if has_x != has_y {
return Err("x 和 y 要一起給".into());
}
let at = if has_x { Some(pt(v, scale()?)?) } else { None };
if !v["amount"].is_null() && v["amount"].as_i64().is_none() {
return Err("amount 要是整數".into());
}
let amount = scroll_amount(v["amount"].as_i64());
let dir = v["direction"].as_str().unwrap_or("").trim().to_lowercase();
let (dy, dx) = scroll_pixels(&dir, amount).ok_or_else(|| format!("方向「{dir}」要是 up、down、left 或 right"))?;
mac::scroll(at, dy, dx, flags_of(v));
Ok((format!("已捲動 {dir} ×{amount}。"), None))
}
"computer_use_drag" => {
let s = scale()?;
// Every point must be valid: points that failed to parse used to be dropped in
// silence, so the drag ran on fewer points and reported success at the wrong place.
let raw = v["path"].as_array().ok_or("拖曳要給 path：一個點陣，每個點有數字 x 和 y（用截圖上的座標）")?;
let path: Vec<(f64, f64)> = raw.iter().map(|p| pt(p, s)).collect::<Result<_, _>>()?;
if path.len() < 2 {
return Err("拖曳至少要兩個點（每個點要有數字 x 和 y，用截圖上的座標）".into());
}
let button = v["button"].as_str().unwrap_or("left");
mac::drag(&path, button, flags_of(v));
// Like the original's "Dragged from (x, y) to (x, y) with left button."
let pts = v["path"].as_array().cloned().unwrap_or_default();
let at = |p: &Value| format!("({:.0}, {:.0})", p["x"].as_f64().unwrap_or(0.0), p["y"].as_f64().unwrap_or(0.0));
let side = match button {
"right" => "右鍵",
"middle" => "中鍵",
_ => "左鍵",
};
Ok((format!("已從 {} 拖曳到 {}（{side}）。", pts.first().map(at).unwrap_or_default(), pts.last().map(at).unwrap_or_default()), None))
}
"computer_use_mouse_button" => {
let button = v["button"].as_str().unwrap_or("left");
let at = if v["x"].is_null() { None } else { Some(pt(v, scale()?)?) };
let release = v["action"].as_str() == Some("release");
mac::mouse_button(at, button, !release);
Ok((if release { "已放開滑鼠鍵。" } else { "已按住滑鼠鍵（記得之後放開）。" }.into(), None))
}
"computer_use_typing" => {
let text = v["value"].as_str().unwrap_or("");
if text.is_empty() {
return Err("請給要打的字".into());
}
mac::type_text(text);
Ok((format!("已打字（{} 個字）。", text.chars().count()), None))
}
"computer_use_press_key" => {
let keys = parse_keys(v["key"].as_str().unwrap_or(""))?;
let hold = v["hold_duration_ms"].as_u64().map(|ms| ms.min(MAX_HOLD_MS) as f64 / 1000.0).filter(|s| *s > 0.0).unwrap_or(KEY_HOLD_SECS);
for (i, (code, flags)) in keys.iter().enumerate() {
if i > 0 {
std::thread::sleep(std::time::Duration::from_secs_f64(KEY_GAP_SECS));
}
mac::press(*code, *flags, hold);
}
Ok((format!("已按 {}。", v["key"].as_str().unwrap_or("")), None))
}
other => Err(format!("沒有「{other}」這個工具")),
} }
t
}
/// The phone or another viewer acting on this screen as the person: their tap is on the picture
/// they were sent (`shot_size()`), so it maps exactly like the bot's coordinates. Not refused
/// while the person holds the screen (it is the person), and no picture back (the viewer
/// refreshes its own).
pub fn person_act(name: &str, v: &Value) -> Result<(), String> {
need_controls()?;
act(name, v, scale_here()).map(|_| ())
} }
/// The app a screen action will land in: the window under the point for clicks, else the /// The app a screen action will land in: the window under the point for clicks, else the
/// frontmost app. Used for the per-app permission (「AnyBot 可以看到 {app}，並在其中點擊和輸入」). /// frontmost app. Used for the per-app permission (「AnyBot 可以看到 {app}，並在其中點擊和輸入」).
pub fn target_app(name: &str, args: &str) -> Option<String> { pub fn target_app(name: &str, args: &str) -> Option<String> {
let v = args_of(args); let v = args_of(args);
if let Some(app) = v["app"].as_str().filter(|a| !a.trim().is_empty() && matches!(name, "computer_use_read_app" | "computer_use_open_app" | "computer_use_press_key")) { if let Some(app) = v["app"].as_str().filter(|a| !a.trim().is_empty() && matches!(name, "computer_use_read_app" | "computer_use_open_app")) {
return Some(app.trim().to_string()); return Some(app.trim().to_string());
} }
if name == "computer_use_read_app" { if name == "computer_use_read_app" {
return frontmost_app(); return frontmost_app();
} }
if run_text_mode_name(name, &v).is_some() { if let Some(r) = v["ref"].as_str().filter(|r| !r.trim().is_empty()) {
return crate::desktop::last_app(); return crate::desktop::spot(r).ok().map(|s| s.app).or_else(crate::desktop::last_app);
} }
let s = (*SCALE.lock().unwrap_or_else(|e| e.into_inner()))?; let s = scale_here();
let point = match name { let point = match name {
"computer_use_click" | "computer_use_mouse_move" | "computer_use_scroll" | "computer_use_mouse_button" if !v["x"].is_null() => Some(to_screen(v["x"].as_f64()?, v["y"].as_f64()?, s)), "computer_use_click" | "computer_use_mouse_move" | "computer_use_scroll" | "computer_use_mouse_button" if !v["x"].is_null() => Some(to_screen(v["x"].as_f64()?, v["y"].as_f64()?, s)),
"computer_use_drag" => v["path"].as_array().and_then(|a| a.first()).and_then(|p| Some(to_screen(p["x"].as_f64()?, p["y"].as_f64()?, s))), "computer_use_drag" => v["path"].as_array().and_then(|a| a.first()).and_then(|p| Some(to_screen(p["x"].as_f64()?, p["y"].as_f64()?, s))),
@ -719,18 +845,15 @@ pub fn app_at_point(x: f64, y: f64) -> Option<String> {
fn screenshot() -> Result<(String, String), String> { fn screenshot() -> Result<(String, String), String> {
if !mac::screen_capture_preflight() { if !mac::screen_capture_preflight() {
ask_permission("Privacy_ScreenCapture"); ask_permission("Privacy_ScreenCapture");
return Err("macOS 還沒讓 AnyBot 錄製螢幕，所以看不到畫面。我已經打開「系統設定 → 隱私權與安全性 → 螢幕與系統錄音」：把 AnyBot（或「終端機」）打開，再叫我試一次。只要開一次。".into()); return Err(NO_SCREEN_RECORDING.into());
} }
let img = capture_screen()?; let img = capture_screen()?;
let (w, h) = img.dimensions(); // Always the same size for this screen (shot_size), so coordinates map without state.
let (fw, fh, resize) = fit(w, h); let (fw, fh) = shot_size();
let img = if resize { image::imageops::resize(&img, fw, fh, image::imageops::FilterType::Triangle) } else { img }; let img = if img.dimensions() != (fw, fh) { image::imageops::resize(&img, fw, fh, image::imageops::FilterType::Triangle) } else { img };
let jpeg = encode_jpeg(&img).ok_or("截圖太大了，沒辦法送出")?; let jpeg = encode_jpeg(&img).ok_or("截圖太大了，沒辦法送出")?;
let (pw, ph) = mac::screen_points();
let scale = (if fw > 0 { pw / fw as f64 } else { 1.0 }, if fh > 0 { ph / fh as f64 } else { 1.0 });
*SCALE.lock().unwrap_or_else(|e| e.into_inner()) = Some(scale);
let b64 = base64::engine::general_purpose::STANDARD.encode(&jpeg); let b64 = base64::engine::general_purpose::STANDARD.encode(&jpeg);
Ok((format!("你的螢幕截圖（{fw}×{fh}）。之後的座標請用這張圖上的像素位置。"), b64)) Ok((format!("螢幕截圖（{fw}×{fh}）。座標用這張圖上的像素。"), b64))
} }
/// The whole screen as RGB (backing pixels). /// The whole screen as RGB (backing pixels).
@ -798,7 +921,6 @@ mod mac {
fn CGEventSetIntegerValueField(e: Ev, field: u32, v: i64); fn CGEventSetIntegerValueField(e: Ev, field: u32, v: i64);
fn CGEventPost(tap: u32, e: Ev); fn CGEventPost(tap: u32, e: Ev);
fn CGEventCreateScrollWheelEvent(src: Src, units: u32, count: u32, wheel1: i32, ...) -> Ev; fn CGEventCreateScrollWheelEvent(src: Src, units: u32, count: u32, wheel1: i32, ...) -> Ev;
fn CGEventPostToPid(pid: i32, e: Ev);
fn CGDisplayPixelsWide(d: u32) -> usize; fn CGDisplayPixelsWide(d: u32) -> usize;
fn CGDisplayPixelsHigh(d: u32) -> usize; fn CGDisplayPixelsHigh(d: u32) -> usize;
fn CGWindowListCopyWindowInfo(option: u32, relative_to: u32) -> *const c_void; fn CGWindowListCopyWindowInfo(option: u32, relative_to: u32) -> *const c_void;
@ -894,33 +1016,6 @@ mod mac {
}) })
} }
/// A click delivered straight to an app without moving the pointer or raising its window
/// (the original's "skylight-no-raise" path does this with a private API).
pub fn click_in_background(pid: i32, x: f64, y: f64, button: &str, count: i64, flags: u64) {
let (down, up, _, b) = kinds(button);
let src = Source::new();
for n in 1..=count {
if n > 1 {
sleep_secs(CLICK_REPEAT_GAP_SECS);
}
for (i, kind) in [down, up].into_iter().enumerate() {
if i == 1 {
sleep_secs(CLICK_HOLD_SECS);
}
let e = unsafe { CGEventCreateMouseEvent(src.0, kind, CGPoint { x, y }, b) };
if e.is_null() {
continue;
}
unsafe {
CGEventSetIntegerValueField(e, CLICK_STATE, n);
CGEventSetFlags(e, flags);
CGEventPostToPid(pid, e);
CFRelease(e);
}
}
}
}
/// Press or release a mouse button where the pointer is (or at a point). /// Press or release a mouse button where the pointer is (or at a point).
pub fn mouse_button(at: Option<(f64, f64)>, button: &str, press: bool) { pub fn mouse_button(at: Option<(f64, f64)>, button: &str, press: bool) {
let (down, up, _, b) = kinds(button); let (down, up, _, b) = kinds(button);
@ -1091,8 +1186,49 @@ mod mac {
post(unsafe { CGEventCreateScrollWheelEvent(src.0, SCROLL_PIXEL, 2, dy, dx) }, flags); post(unsafe { CGEventCreateScrollWheelEvent(src.0, SCROLL_PIXEL, 2, dy, dx) }, flags);
} }
/// CUTypedCharacters.postChunks: each chunk on key 128, down, 0.035 s, up; 0.02 s between chunks. /// Type text so that any app takes it, whatever the keyboard layout or input method: paste
/// it (⌘V) and put the clipboard back. Text sent as key events (the original's
/// CUTypedCharacters, key 128 carrying the characters) went into the input method instead
/// when one was on — with 注音 every letter vanished while return still arrived, so a
/// command typed into a terminal ran empty. Falls back to those key events without a
/// clipboard.
pub fn type_text(text: &str) { pub fn type_text(text: &str) {
if !paste(text) {
type_as_keys(text);
}
}
/// Wait for the app to read the clipboard before it is put back (Electron apps read it a
/// moment after ⌘V; estimated).
const PASTE_READ_MS: u64 = 350;
fn paste(text: &str) -> bool {
let Ok(mut cb) = arboard::Clipboard::new() else { return false };
// What the person had there comes back afterwards (text or a picture).
let saved_text = cb.get_text().ok();
let saved_image = if saved_text.is_none() { cb.get_image().ok() } else { None };
if cb.set_text(text.to_string()).is_err() {
return false;
}
std::thread::sleep(Duration::from_millis(40));
press(9, FLAG_CMD, KEY_HOLD_SECS);
std::thread::sleep(Duration::from_millis(PASTE_READ_MS));
match (saved_text, saved_image) {
(Some(t), _) => {
let _ = cb.set_text(t);
}
(None, Some(i)) => {
let _ = cb.set_image(i);
}
_ => {
let _ = cb.clear();
}
}
true
}
/// CUTypedCharacters.postChunks: each chunk on key 128, down, 0.035 s, up; 0.02 s between chunks.
fn type_as_keys(text: &str) {
let src = Source::new(); let src = Source::new();
let chunks = utf16_chunks(text); let chunks = utf16_chunks(text);
for (i, chunk) in chunks.iter().enumerate() { for (i, chunk) in chunks.iter().enumerate() {
@ -1148,7 +1284,6 @@ mod mac {
pub fn screen_pixels() -> (u32, u32) { pub fn screen_pixels() -> (u32, u32) {
(0, 0) (0, 0)
} }
pub fn click_in_background(_: i32, _: f64, _: f64, _: &str, _: i64, _: u64) {}
pub fn mouse_button(_: Option<(f64, f64)>, _: &str, _: bool) {} pub fn mouse_button(_: Option<(f64, f64)>, _: &str, _: bool) {}
} }
@ -1215,8 +1350,24 @@ mod tests {
#[test] #[test]
fn a_press_can_be_several_keys() { fn a_press_can_be_several_keys() {
assert_eq!(parse_keys("cmd+a delete").unwrap(), vec![(0, FLAG_CMD), (51, 0)]); assert_eq!(key_plan("cmd+a delete").unwrap(), vec![Key::Combo(0, FLAG_CMD), Key::Combo(51, 0)]);
assert!(parse_keys(" ").is_err()); assert!(key_plan(" ").is_err());
// Keys by the names models use, and characters without a key typed instead.
assert_eq!(key_plan("cmd+grave").unwrap(), vec![Key::Combo(50, FLAG_CMD)]);
// A printing key alone is its character (typed, so 注音 can't swallow it).
assert_eq!(key_plan("slash").unwrap(), vec![Key::Char("/".into())]);
assert_eq!(key_plan("/ @ r return").unwrap(), vec![Key::Char("/".into()), Key::Char("@".into()), Key::Char("r".into()), Key::Combo(36, 0)]);
assert!(key_plan("hyper+x").is_err());
}
#[test]
fn pictures_and_clicks_share_one_size() {
// The picture's size depends only on the screen, so a tap or a click maps the same
// whoever saw which screenshot last (a 2940×1912 Retina screen of 1470×956 points).
let (w, h, _) = fit(2940, 1912);
let s = (1470.0 / w as f64, 956.0 / h as f64);
let (x, y) = to_screen(512.0, 333.0, s);
assert!((x - 735.0).abs() < 0.01 && (y - 478.0).abs() < 1.0);
} }
#[test] #[test]

9
src/desk.rs
View File

@ -86,10 +86,6 @@ pub fn screen_pixels() -> (u32, u32) {
} }
} }
pub fn screen_capture_allowed() -> bool {
available()
}
pub fn screen_capture_preflight() -> bool { pub fn screen_capture_preflight() -> bool {
available() available()
} }
@ -184,11 +180,6 @@ pub fn click(px: f64, py: f64, which: &str, count: i64, flags: u64) {
}); });
} }
/// Linux has no "click without raising": a normal click.
pub fn click_in_background(_pid: i32, px: f64, py: f64, which: &str, count: i64, flags: u64) {
click(px, py, which, count, flags);
}
pub fn mouse_button(at: Option<(f64, f64)>, which: &str, press: bool) { pub fn mouse_button(at: Option<(f64, f64)>, which: &str, press: bool) {
if kind() == Kind::X11 { if kind() == Kind::X11 {
return x11::mouse_button(at, which, press); return x11::mouse_button(at, which, press);

284
src/desktop.rs
View File

@ -1,7 +1,10 @@
// Seeing and using other apps through the OS accessibility APIs, with vercel-labs/agent-desktop // Reading other apps through the OS accessibility APIs, with vercel-labs/agent-desktop compiled
// compiled in (AX on macOS, AT-SPI on Linux, UI Automation on Windows). A snapshot gives every // in (AX on macOS, AT-SPI on Linux, UI Automation on Windows): an app's controls and text as
// control of an app a ref; actions take a ref. No screenshots needed, and by default no pointer // numbered lines (e1, e2…), each number remembered with where it is on the screen. That is all
// movement or focus change (the person can keep working). // it does. Acting on a number is a real click or keystroke at that spot (computer.rs), the same
// way as acting on screenshot coordinates: one way to act on a screen, whatever the bot read it
// with. (agent-desktop's own "semantic" actions were a second way with its own key names and
// failures: typing into a terminal, "/" and "`" from a background thread, focus rules.)
use agent_desktop_core as ad; use agent_desktop_core as ad;
use serde_json::Value; use serde_json::Value;
@ -37,14 +40,12 @@ fn opt(v: &Value, k: &str) -> Option<String> {
Some(st(v, k)).filter(|s| !s.is_empty()) Some(st(v, k)).filter(|s| !s.is_empty())
} }
/// One agent-desktop command (the CLI's names): apps, windows, snapshot, click, double_click, /// One agent-desktop command (the CLI's names): apps, windows, snapshot, launch, permissions.
/// right_click, type, set_value, select, press, scroll, focus_window, launch, screenshot.
pub fn run(cmd: &str, a: &Value) -> Result<Value, String> { pub fn run(cmd: &str, a: &Value) -> Result<Value, String> {
use ad::commands as c; use ad::commands as c;
let ad = adapter(); let ad = adapter();
let ad = ad.as_ref(); let ad = ad.as_ref();
let ctx = ad::CommandContext::default(); let ctx = ad::CommandContext::default();
let r = |rf: &Value| c::helpers::RefArgs { ref_id: st(rf, "ref"), timeout_ms: None };
let out = match cmd { let out = match cmd {
"apps" => c::list_apps::execute(c::list_apps::ListAppsArgs { app: opt(a, "app"), timeout_ms: None }, ad), "apps" => c::list_apps::execute(c::list_apps::ListAppsArgs { app: opt(a, "app"), timeout_ms: None }, ad),
"windows" => c::list_windows::execute(c::list_windows::ListWindowsArgs { app: opt(a, "app"), timeout_ms: None }, ad), "windows" => c::list_windows::execute(c::list_windows::ListWindowsArgs { app: opt(a, "app"), timeout_ms: None }, ad),
@ -53,7 +54,7 @@ pub fn run(cmd: &str, a: &Value) -> Result<Value, String> {
app: opt(a, "app"), app: opt(a, "app"),
window_id: opt(a, "window_id"), window_id: opt(a, "window_id"),
max_depth: a["max_depth"].as_u64().unwrap_or(30).min(60) as u8, max_depth: a["max_depth"].as_u64().unwrap_or(30).min(60) as u8,
include_bounds: false, include_bounds: a["bounds"].as_bool().unwrap_or(true),
interactive_only: a["interactive_only"].as_bool().unwrap_or(true), interactive_only: a["interactive_only"].as_bool().unwrap_or(true),
compact: true, compact: true,
surface: ad::SnapshotSurface::default(), surface: ad::SnapshotSurface::default(),
@ -65,23 +66,6 @@ pub fn run(cmd: &str, a: &Value) -> Result<Value, String> {
ad, ad,
&ctx, &ctx,
), ),
"click" => c::click::execute(r(a), ad, &ctx),
"double_click" => c::double_click::execute(r(a), ad, &ctx),
"right_click" => c::right_click::execute(r(a), ad, &ctx),
"type" => c::type_text::execute(c::type_text::TypeArgs { ref_id: st(a, "ref"), text: st(a, "text"), timeout_ms: None, interval_ms: None }, ad, &ctx),
"set_value" => c::set_value::execute(c::set_value::SetValueArgs { ref_id: st(a, "ref"), value: st(a, "value"), timeout_ms: None }, ad, &ctx),
"select" => c::select::execute(c::select::SelectArgs { ref_id: st(a, "ref"), value: st(a, "value"), timeout_ms: None }, ad, &ctx),
"press" => c::press::execute(c::press::PressArgs { combo: st(a, "combo"), app: opt(a, "app"), window_id: opt(a, "window_id"), force: false }, ad, &ctx),
"scroll" => {
let direction = match st(a, "direction").as_str() {
"up" => ad::Direction::Up,
"left" => ad::Direction::Left,
"right" => ad::Direction::Right,
_ => ad::Direction::Down,
};
c::scroll::execute(c::scroll::ScrollArgs { ref_id: st(a, "ref"), direction, amount: a["amount"].as_u64().unwrap_or(3) as u32, timeout_ms: None }, ad, &ctx)
}
"focus_window" => c::focus_window::execute(c::focus_window::FocusWindowArgs { window_id: opt(a, "window_id"), app: opt(a, "app"), title: opt(a, "title") }, ad),
"launch" => c::launch::execute(c::launch::LaunchArgs { app: st(a, "app"), options: ad::system::launch_options::LaunchOptions { activate: true, ..Default::default() } }, ad), "launch" => c::launch::execute(c::launch::LaunchArgs { app: st(a, "app"), options: ad::system::launch_options::LaunchOptions { activate: true, ..Default::default() } }, ad),
"permissions" => { "permissions" => {
let d = ad::Deadline::standard().map_err(|e| err(e.into()))?; let d = ad::Deadline::standard().map_err(|e| err(e.into()))?;
@ -94,26 +78,50 @@ pub fn run(cmd: &str, a: &Value) -> Result<Value, String> {
// ---------- what the bot uses ---------- // ---------- what the bot uses ----------
/// The latest snapshot (id, app) so the bot can say "e6" instead of the full ref. /// Where a numbered control is: its centre in screen coordinates (the space the mouse uses),
static LAST: std::sync::Mutex<Option<(String, String)>> = std::sync::Mutex::new(None); /// and what it is (「按鈕「7」」), for the step's description.
#[derive(Clone, Debug, PartialEq)]
pub struct Spot {
pub app: String,
pub x: f64,
pub y: f64,
pub what: String,
}
/// "e6", "[e6]", "6" or a full "@snap:e6" → the full ref of the latest snapshot. /// Each number of a reading and its spot (None when the app didn't say where).
pub fn full_ref(r: &str) -> Result<String, String> { pub type Spots = std::collections::HashMap<String, Option<Spot>>;
/// The latest reading: the app and its numbers.
static LAST: std::sync::Mutex<Option<(String, Spots)>> = std::sync::Mutex::new(None);
/// "e6", "[e6]", "6" or a full "@snap:e6" → "e6".
pub fn short_ref(r: &str) -> Result<String, String> {
let r = r.trim().trim_matches(['[', ']']); let r = r.trim().trim_matches(['[', ']']);
if r.starts_with('@') { let r = r.rsplit(':').next().unwrap_or(r);
return Ok(r.to_string());
}
let n = r.trim_start_matches('e'); let n = r.trim_start_matches('e');
if n.is_empty() || !n.chars().all(|c| c.is_ascii_digit()) { if n.is_empty() || !n.chars().all(|c| c.is_ascii_digit()) {
return Err(format!("「{r}」不是元素編號，請用 computer_use_read_app 列出的 e 編號")); return Err(format!("「{r}」不是元素編號，請用 computer_use_read_app 列出的 e 編號"));
} }
let last = LAST.lock().unwrap().clone().ok_or("請先用 computer_use_read_app 看一下 App 的畫面")?; Ok(format!("e{n}"))
Ok(format!("@{}:e{n}", last.0))
} }
/// The app the latest snapshot was of (for the per-app permission). /// Where to click for a number from the latest reading.
pub fn spot(r: &str) -> Result<Spot, String> {
let r = short_ref(r)?;
let last = LAST.lock().unwrap_or_else(|e| e.into_inner());
let (_, spots) = last.as_ref().ok_or("請先用 computer_use_read_app 讀一下那個 App 的畫面")?;
match spots.get(&r) {
Some(Some(s)) => Ok(s.clone()),
// Wayland doesn't tell apps where other windows' controls are: none has a place.
Some(None) if spots.values().all(Option::is_none) => Err(format!("這台電腦讀得到 {r}，但系統不告訴 App 它在螢幕上的位置：請用 computer_use_screenshot 看，用截圖上的 x、y 點。")),
Some(None) => Err(format!("{r} 現在不在畫面上（被捲走或藏起來了）。先捲動或改用截圖上的座標。")),
None => Err(format!("最近一次讀到的畫面裡沒有 {r}：先用 computer_use_read_app 再讀一次。")),
}
}
/// The app the latest reading was of (for the per-app permission).
pub fn last_app() -> Option<String> { pub fn last_app() -> Option<String> {
LAST.lock().unwrap().as_ref().map(|l| l.1.clone()) LAST.lock().unwrap_or_else(|e| e.into_inner()).as_ref().map(|l| l.0.clone())
} }
fn role_name(role: &str) -> &str { fn role_name(role: &str) -> &str {
@ -141,13 +149,87 @@ fn role_name(role: &str) -> &str {
} }
} }
/// An app's controls (numbered) and visible text, as short lines. `app` empty = the app in front. /// Longest value shown whole; longer ones (a terminal's scrollback, a document) keep their
pub fn read_app(app: &str) -> Result<(String, Vec<String>), String> { /// start and their end, where a terminal's prompt and newest output are.
read_app_as(app, true) const VALUE_HEAD: usize = 160;
const VALUE_TAIL: usize = 400;
fn shorten(v: &str) -> String {
let n = v.chars().count();
if n <= VALUE_HEAD + VALUE_TAIL + 20 {
return v.to_string();
}
let head: String = v.chars().take(VALUE_HEAD).collect();
let tail: String = v.chars().skip(n - VALUE_TAIL).collect();
format!("{head} …（中間省略 {} 字）… {tail}", n - VALUE_HEAD - VALUE_TAIL)
} }
fn read_app_as(app: &str, remember: bool) -> Result<(String, Vec<String>), String> { /// The centre of a node's box, when it has a real one on the screen.
let mut args = serde_json::json!({"interactive_only": false, "max_depth": 40}); fn centre(n: &Value) -> Option<(f64, f64)> {
let b = n.get("bounds")?;
let (x, y, w, h) = (b["x"].as_f64()?, b["y"].as_f64()?, b["width"].as_f64()?, b["height"].as_f64()?);
if w < 1.0 || h < 1.0 || x + w <= 0.0 || y + h <= 0.0 {
return None;
}
Some((x + w / 2.0, y + h / 2.0))
}
/// A snapshot as short lines, and each number's spot.
pub fn lines_of(tree: &Value, app: &str) -> (Vec<String>, Spots) {
let mut lines = vec![];
let mut spots = std::collections::HashMap::new();
fn walk(n: &Value, depth: usize, lines: &mut Vec<String>, spots: &mut Spots, parent: &str, app: &str) {
if lines.len() >= 400 {
return;
}
let name = n["name"].as_str().unwrap_or("").trim();
let value = n["value"].as_str().map(str::to_string).or_else(|| n["value"].as_f64().map(|f| f.to_string())).unwrap_or_default();
let value = shorten(value.trim());
let role = n["role"].as_str().unwrap_or("");
let text = if !name.is_empty() && !value.is_empty() && value != name { format!("「{name}」：{value}") } else if !name.is_empty() { format!("「{name}」") } else if !value.is_empty() { format!("：{value}") } else { String::new() };
let mut flags = vec![];
let is_field = matches!(role, "textfield" | "textarea" | "searchfield" | "combobox" | "securetextfield");
for (k, word) in [("checked", "已勾"), ("selected", "已選"), ("disabled", "停用"), ("focused", "游標在這")] {
// Only text fields report focus usefully (Finder marks every sidebar cell focused).
if k == "focused" && !is_field {
continue;
}
if n["states"].as_array().map(|a| a.iter().any(|x| x == k)).unwrap_or(false) || n[k] == true {
flags.push(word);
}
}
let flags = if flags.is_empty() { String::new() } else { format!("（{}）", flags.join("、")) };
let indent = " ".repeat(depth.min(6));
// A cell or label that only repeats its parent's name adds nothing.
let repeats = !name.is_empty() && name == parent;
if repeats && n["ref_id"].is_null() {
// plain text under a control with the same name
} else if repeats && lines.last().map(|l| l.contains(&format!("「{name}」"))).unwrap_or(false) {
// e.g. Finder's row → cell with the same name: keep the row only
} else if let Some(r) = n["ref_id"].as_str() {
let short = r.rsplit(':').next().unwrap_or(r).to_string();
if !text.is_empty() || matches!(role, "button" | "textfield" | "textarea" | "checkbox" | "searchfield" | "popupbutton") {
let what = format!("{}{}", role_name(role), if name.is_empty() { String::new() } else { format!("「{}」", name.chars().take(30).collect::<String>()) });
spots.insert(short.clone(), centre(n).map(|(x, y)| Spot { app: app.to_string(), x, y, what }));
lines.push(format!("{indent}{short} {}{text}{flags}", role_name(role)));
}
} else if !text.is_empty() && matches!(role, "statictext" | "text" | "heading" | "cell" | "value" | "") {
lines.push(format!("{indent}{}{text}", role_name(role)));
}
let next_parent = if name.is_empty() { parent } else { name };
for c in n["children"].as_array().into_iter().flatten() {
walk(c, depth + 1, lines, spots, next_parent, app);
}
}
walk(tree, 0, &mut lines, &mut spots, "", app);
let pad = lines.iter().map(|l| l.len() - l.trim_start().len()).min().unwrap_or(0);
(lines.into_iter().map(|l| l[pad..].to_string()).collect(), spots)
}
/// An app's controls (numbered) and visible text, as short lines. `app` empty = the app in front.
/// The numbers and where they are on the screen are remembered for the next click or typing.
pub fn read_app(app: &str) -> Result<(String, Vec<String>), String> {
let mut args = serde_json::json!({"interactive_only": false, "max_depth": 40, "bounds": true});
if !app.trim().is_empty() { if !app.trim().is_empty() {
args["app"] = Value::String(app.trim().to_string()); args["app"] = Value::String(app.trim().to_string());
} }
@ -171,96 +253,40 @@ fn read_app_as(app: &str, remember: bool) -> Result<(String, Vec<String>), Strin
} }
r => r?, r => r?,
}; };
let snap = v["snapshot_id"].as_str().unwrap_or("").to_string();
let app_name = v["app"].as_str().unwrap_or(app).to_string(); let app_name = v["app"].as_str().unwrap_or(app).to_string();
if remember { let (lines, spots) = lines_of(&v["tree"], &app_name);
*LAST.lock().unwrap() = Some((snap.clone(), app_name.clone())); *LAST.lock().unwrap_or_else(|e| e.into_inner()) = Some((app_name.clone(), spots));
}
let mut lines = vec![];
fn walk(n: &Value, depth: usize, lines: &mut Vec<String>, parent: &str) {
if lines.len() >= 400 {
return;
}
let name = n["name"].as_str().unwrap_or("").trim();
let value = n["value"].as_str().map(str::to_string).or_else(|| n["value"].as_f64().map(|f| f.to_string())).unwrap_or_default();
let role = n["role"].as_str().unwrap_or("");
let text = if !name.is_empty() && !value.is_empty() && value != name { format!("「{name}」：{value}") } else if !name.is_empty() { format!("「{name}」") } else if !value.is_empty() { format!("：{value}") } else { String::new() };
let mut flags = vec![];
let is_field = matches!(role, "textfield" | "textarea" | "searchfield" | "combobox" | "securetextfield");
for (k, word) in [("checked", "已勾"), ("selected", "已選"), ("disabled", "停用"), ("focused", "游標在這")] {
// Only text fields report focus usefully (Finder marks every sidebar cell focused).
if k == "focused" && !is_field {
continue;
}
if n["states"].as_array().map(|a| a.iter().any(|x| x == k)).unwrap_or(false) || n[k] == true {
flags.push(word);
}
}
let flags = if flags.is_empty() { String::new() } else { format!("（{}）", flags.join("、")) };
let indent = " ".repeat(depth.min(6));
// A cell or label that only repeats its parent's name adds nothing.
let repeats = !name.is_empty() && name == parent;
if repeats && n["ref_id"].is_null() {
// plain text under a control with the same name
} else if repeats && lines.last().map(|l| l.contains(&format!("「{name}」"))).unwrap_or(false) {
// e.g. Finder's row → cell with the same name: keep the row only
} else if let Some(r) = n["ref_id"].as_str() {
let short = r.rsplit(':').next().unwrap_or(r);
if !text.is_empty() || matches!(role, "button" | "textfield" | "textarea" | "checkbox" | "searchfield" | "popupbutton") {
lines.push(format!("{indent}{short} {}{text}{flags}", role_name(role)));
}
} else if !text.is_empty() && matches!(role, "statictext" | "text" | "heading" | "cell" | "value" | "") {
lines.push(format!("{indent}{}{text}", role_name(role)));
}
let next_parent = if name.is_empty() { parent } else { name };
for c in n["children"].as_array().into_iter().flatten() {
walk(c, depth + 1, lines, next_parent);
}
}
walk(&v["tree"], 0, &mut lines, "");
let pad = lines.iter().map(|l| l.len() - l.trim_start().len()).min().unwrap_or(0);
let lines = lines.into_iter().map(|l| l[pad..].to_string()).collect();
Ok((app_name, lines)) Ok((app_name, lines))
} }
/// Act on an element by its number: click, double_click, right_click, type, set_value, select, /// Open an app, or bring it to the front when it runs.
/// scroll. Returns what the app shows now (new text since the action). pub fn launch(app: &str) -> Result<(), String> {
pub fn act(cmd: &str, r: &str, extra: &Value) -> Result<String, String> { run("launch", &serde_json::json!({"app": app})).map(|_| ())
let full = full_ref(r)?; }
let app = last_app().unwrap_or_default();
let before = read_app_as(&app, false).map(|x| x.1).unwrap_or_default(); #[cfg(test)]
let mut a = extra.clone(); mod tests {
if !a.is_object() { use super::*;
a = serde_json::json!({});
} #[test]
a["ref"] = Value::String(full.clone()); fn numbers_keep_where_they_are() {
let res = run(cmd, &a); let tree = serde_json::json!({"role": "window", "name": "計算機", "children": [
// Typing into some fields fails as keystrokes; setting the value works there. {"role": "statictext", "value": "54"},
let res = match (cmd, res) { {"ref_id": "@s1:e7", "role": "button", "name": "7", "bounds": {"x": 100.0, "y": 200.0, "width": 40.0, "height": 30.0}},
("type", Err(_)) => run("set_value", &serde_json::json!({"ref": full, "value": a["text"]})), {"ref_id": "@s1:e8", "role": "button", "name": "8", "bounds": {"x": 0.0, "y": 0.0, "width": 0.0, "height": 0.0}},
(_, r) => r, {"ref_id": "@s1:e9", "role": "textarea", "value": format!("{}END", "x".repeat(2000))}
}; ]});
res?; let (lines, spots) = lines_of(&tree, "Calculator");
std::thread::sleep(std::time::Duration::from_millis(400)); assert!(lines.iter().any(|l| l == "e7 按鈕「7」"));
let (_, after) = read_app_as(&app, false).unwrap_or_default(); assert_eq!(spots["e7"], Some(Spot { app: "Calculator".into(), x: 120.0, y: 215.0, what: "按鈕「7」".into() }));
let old: std::collections::HashSet<String> = before.iter().map(|l| l.split_whitespace().skip(1).collect::<Vec<_>>().join(" ")).collect(); // No real box (hidden, scrolled away): known, but not clickable.
let new: Vec<&String> = after.iter().filter(|l| !old.contains(&l.split_whitespace().skip(1).collect::<Vec<_>>().join(" "))).take(30).collect(); assert_eq!(spots["e8"], None);
// Show what changed without numbers (they belong to a different reading); if controls came // A terminal's whole scrollback: the end (prompt, newest output) stays.
// or went, the bot's numbers may now point elsewhere. let e9 = lines.iter().find(|l| l.starts_with("e9")).unwrap();
let strip = |l: &str| -> String { assert!(e9.ends_with("END") && e9.contains("中間省略") && e9.chars().count() < 700);
let t = l.trim(); assert_eq!(short_ref("[e7]").unwrap(), "e7");
match t.split_once(' ') { assert_eq!(short_ref("@s1:e7").unwrap(), "e7");
Some((first, rest)) if first.starts_with('e') && first[1..].chars().all(|c| c.is_ascii_digit()) => rest.to_string(), assert_eq!(short_ref("12").unwrap(), "e12");
_ => t.to_string(), assert!(short_ref("ok").is_err());
} }
};
let controls_changed = new.iter().any(|l| l.trim().starts_with('e')) || before.iter().filter(|l| l.trim().starts_with('e')).count() != after.iter().filter(|l| l.trim().starts_with('e')).count();
let mut out = format!("好了（在「{app}」）。");
if !new.is_empty() {
out.push_str(&format!("\n現在畫面上新出現：\n{}", new.iter().map(|l| strip(l)).collect::<Vec<_>>().join("\n")));
}
if controls_changed {
out.push_str("\n（按鈕或欄位有變動，編號可能已經不同：下一步前先用 computer_use_read_app 看新的清單。）");
}
Ok(out)
} }

8
src/main.rs
View File

@ -203,10 +203,10 @@ fn main() -> eframe::Result<()> {
} }
return Ok(()); return Ok(());
} }
// For testing on a new system: `anybot --tool computer_use_screenshot '{}' computer_use_click // For testing on a new system: `anybot --tool computer_use_read_app '{"app":"Calculator"}'
// '{"x":10,"y":20}' …` runs computer tools in order (one process, so the screenshot's scale // computer_use_click '{"ref":"e7"}' …` runs computer tools in order (one process, so a
// carries over), prints what the bot would hear, and saves screenshots as anybot-tool-N.jpg // reading's numbers carry over), prints what the bot would hear, and saves the pictures that
// in the temp folder. // come back as anybot-tool-N.jpg in the temp folder.
if args.get(1).map(String::as_str) == Some("--tool") { if args.get(1).map(String::as_str) == Some("--tool") {
let mut all_ok = true; let mut all_ok = true;
for (i, pair) in args[2..].chunks(2).enumerate() { for (i, pair) in args[2..].chunks(2).enumerate() {

56
src/remote.rs
View File

@ -638,7 +638,11 @@ fn screen(which: &str, s: &Settings) -> Resp {
} }
} }
/// The phone taps, types or scrolls on a screen (coordinates in the frame's pixels). /// The phone taps, types or scrolls on a screen, as the person, in the pixels of the picture it
/// was sent: the box's frame is its desktop at full size, this computer's is the bot-sized
/// picture (shot_size). It used to go through the bot's tools, which mapped the tap with the
/// bot's last screenshot (a different size: taps landed a quarter off, or failed with "take a
/// screenshot first") and refused it while the person held the screen.
fn screen_input(which: &str, v: &Value, s: &Settings) -> Resp { fn screen_input(which: &str, v: &Value, s: &Settings) -> Resp {
let kind = v["kind"].as_str().unwrap_or(""); let kind = v["kind"].as_str().unwrap_or("");
if which == "box" && kind == "wake" { if which == "box" && kind == "wake" {
@ -647,27 +651,37 @@ fn screen_input(which: &str, v: &Value, s: &Settings) -> Resp {
Err(e) => Resp::error(500, &e), Err(e) => Resp::error(500, &e),
}; };
} }
let (name, args) = match kind { let (x, y) = (v["x"].as_f64().unwrap_or(0.0), v["y"].as_f64().unwrap_or(0.0));
"click" => ("computer_use_click", json!({"x": v["x"], "y": v["y"], "button": v["button"].as_str().unwrap_or("left"), "count": v["count"].as_u64().unwrap_or(1)})), let down = v["direction"].as_str().unwrap_or("down") != "up";
"type" => ("computer_use_typing", json!({"value": v["text"]})), let amount = v["amount"].as_i64().unwrap_or(3).clamp(1, 20);
"key" => ("computer_use_press_key", json!({"key": v["key"]})),
"scroll" => ("computer_use_scroll", json!({"x": v["x"], "y": v["y"], "direction": v["direction"].as_str().unwrap_or("down"), "amount": v["amount"].as_i64().unwrap_or(3)})),
_ => return Resp::error(400, "不認得的操作"),
};
match which { match which {
"box" => { "box" => {
crate::cloudbox::touch(); if !matches!(kind, "click" | "type" | "key" | "scroll") {
match crate::cloudbox::computer_run(name, &args) { return Resp::error(400, "不認得的操作");
Ok(_) => Resp::json(&json!({"ok": true})),
Err(e) => Resp::error(500, &e),
} }
if let Err(e) = crate::cloudbox::ensure_running() {
return Resp::error(500, &e);
}
let (xi, yi) = (x.round() as i64, y.round() as i64);
match kind {
"click" => crate::cloudbox::viewer_click(xi, yi, if v["button"].as_str() == Some("right") { "right" } else { "left" }, v["count"].as_u64().unwrap_or(1) as u32),
"type" => crate::cloudbox::viewer_text(v["text"].as_str().unwrap_or("")),
"key" => crate::cloudbox::viewer_key(v["key"].as_str().unwrap_or("")),
_ => crate::cloudbox::viewer_scroll(xi, yi, down, amount as u32),
}
Resp::json(&json!({"ok": true}))
} }
"mac" if s.mac_from_phone => { "mac" if s.mac_from_phone => {
let (t, ok, _) = crate::computer::run_by(name, &args.to_string(), "手機"); let (name, args) = match kind {
if ok { "click" => ("computer_use_click", json!({"x": x, "y": y, "button": v["button"].as_str().unwrap_or("left"), "count": v["count"].as_u64().unwrap_or(1)})),
Resp::json(&json!({"ok": true})) "type" => ("computer_use_typing", json!({"value": v["text"]})),
} else { "key" => ("computer_use_press_key", json!({"key": v["key"]})),
Resp::error(500, &t) "scroll" => ("computer_use_scroll", json!({"x": x, "y": y, "direction": if down { "down" } else { "up" }, "amount": amount})),
_ => return Resp::error(400, "不認得的操作"),
};
match crate::computer::person_act(name, &args) {
Ok(()) => Resp::json(&json!({"ok": true})),
Err(e) => Resp::error(500, &e),
} }
} }
_ => Resp::error(403, "這台 Mac 沒有開放從手機操作"), _ => Resp::error(403, "這台 Mac 沒有開放從手機操作"),
@ -708,7 +722,13 @@ fn device_run(v: &Value, s: &Settings, via_relay: bool, from: &str) -> Resp {
return Resp::json(&json!({"text": text, "ok": ok})); return Resp::json(&json!({"text": text, "ok": ok}));
} }
if matches!(tool, "shell" | "read_file" | "edit_file" | "list_dir" | "search" | "read_terminal" | "write_terminal") { if matches!(tool, "shell" | "read_file" | "edit_file" | "list_dir" | "search" | "read_terminal" | "write_terminal") {
let t = crate::tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: crate::tools::Place::Local { workdir: crate::tools::expand_home(v["workdir"].as_str().unwrap_or("~")) } }; // The brain passes the bot's work folder inside the tool's args (empty = this home).
let dir = crate::tools::args_of(args).get("workdir").and_then(Value::as_str).unwrap_or("").trim().to_string();
let mut workdir = crate::tools::expand_home(if dir.is_empty() { "~" } else { &dir });
if !workdir.is_absolute() {
workdir = dirs::home_dir().unwrap_or(workdir);
}
let t = crate::tools::Target { name: crate::model::THIS_COMPUTER_NAME.into(), place: crate::tools::Place::Local { workdir } };
let ids = crate::tools::RunIds { conversation: "device".into(), request: httpd::random_hex(6) }; let ids = crate::tools::RunIds { conversation: "device".into(), request: httpd::random_hex(6) };
let (text, ok) = crate::tools::run_on(tool, args, &t, &std::sync::atomic::AtomicBool::new(false), &ids); let (text, ok) = crate::tools::run_on(tool, args, &t, &std::sync::atomic::AtomicBool::new(false), &ids);
return Resp::json(&json!({"text": text, "ok": ok})); return Resp::json(&json!({"text": text, "ok": ok}));

2
src/teach.rs
View File

@ -348,7 +348,7 @@ pub fn prompt_section(demos: &[Demo]) -> String {
if demos.is_empty() { if demos.is_empty() {
return String::new(); return String::new();
} }
let mut s = String::from("\n\n你學過的示範（使用者在他的 Mac 上做給你看的步驟）。使用者要你做同樣的事時，照著在「你的電腦」上做：每一步先截圖確認，座標是錄製時截圖上的位置，畫面不同就找對應的按鈕；輸入的內容可以依新的需求調整。想看某一步當時的畫面，用 view_demo_step。"); let mut s = String::from("\n\n你學過的示範（使用者在他的 Mac 上做給你看的步驟）。使用者要你做同樣的事時，照著在「你的電腦」上做：開始先截圖一次，之後每一步做完都會附上新畫面，座標是錄製時截圖上的位置，畫面不同就找對應的按鈕；輸入的內容可以依新的需求調整。想看某一步當時的畫面，用 view_demo_step。");
for d in demos { for d in demos {
s.push_str(&format!("\n### {}（{}）\n", d.name, d.created)); s.push_str(&format!("\n### {}（{}）\n", d.name, d.created));
for (i, st) in d.steps.iter().enumerate() { for (i, st) in d.steps.iter().enumerate() {

82
src/tools.rs
View File

@ -26,6 +26,9 @@ pub const READ_DEFAULT_LINES: usize = 2_000;
/// Foreground time before a command is moved to the background (estimated; the original's comes per call). /// Foreground time before a command is moved to the background (estimated; the original's comes per call).
pub const SHELL_DEFAULT_TIMEOUT_MS: u64 = 30_000; pub const SHELL_DEFAULT_TIMEOUT_MS: u64 = 30_000;
pub const SHELL_MAX_TIMEOUT_MS: u64 = 600_000; pub const SHELL_MAX_TIMEOUT_MS: u64 = 600_000;
/// A command on a relayed computer must answer before the relay's forward timeout (35 s):
/// on the device it moves to the background by this deadline, and read_terminal picks it up.
pub const DEVICE_SHELL_TIMEOUT_MS: u64 = 25_000;
/// Output kept from a command: its start and its end, with the middle left out (sizes estimated; /// Output kept from a command: its start and its end, with the middle left out (sizes estimated;
/// the original reports output_head / output_tail / elided_chars). /// the original reports output_head / output_tail / elided_chars).
pub const OUTPUT_HEAD_CHARS: usize = 6_000; pub const OUTPUT_HEAD_CHARS: usize = 6_000;
@ -49,7 +52,7 @@ pub fn specs() -> Vec<Value> {
json!({"name": "edit_file", "description": "Change a file. Replaces old_text (which must appear exactly once) with new_text. With empty old_text the file is created or overwritten with new_text.", json!({"name": "edit_file", "description": "Change a file. Replaces old_text (which must appear exactly once) with new_text. With empty old_text the file is created or overwritten with new_text.",
"parameters": {"type": "object", "properties": { "parameters": {"type": "object", "properties": {
"path": {"type": "string"}, "old_text": {"type": "string"}, "new_text": {"type": "string"}}, "required": ["path", "new_text"]}}), "path": {"type": "string"}, "old_text": {"type": "string"}, "new_text": {"type": "string"}}, "required": ["path", "new_text"]}}),
json!({"name": "shell", "description": "Run a shell command in the work folder. If it runs longer than timeout_ms it keeps running in the background and you get a terminal id to check with read_terminal.", json!({"name": "shell", "description": "Run a shell command in the work folder. If it runs longer than timeout_ms it keeps running in the background and you get a terminal id to check with read_terminal (pass the same computer to read_terminal/write_terminal).",
"parameters": {"type": "object", "properties": { "parameters": {"type": "object", "properties": {
"command": {"type": "string"}, "command": {"type": "string"},
"timeout_ms": {"type": "integer", "description": "How long to wait before moving it to the background (default 30000)."}}, "required": ["command"]}}), "timeout_ms": {"type": "integer", "description": "How long to wait before moving it to the background (default 30000)."}}, "required": ["command"]}}),
@ -330,7 +333,17 @@ pub fn decide(name: &str, args: &str, workdir: &Path, grants: &[String]) -> Deci
return Decision::Deny("沒有指令".into()); return Decision::Deny("沒有指令".into());
} }
let Some(parts) = split_commands(&cmd) else { let Some(parts) = split_commands(&cmd) else {
return Decision::Ask { reason: format!("執行指令（含有隱藏的子指令，無法自動判斷）：{cmd}"), grant: String::new() }; // Commands the splitter can't read ($(), backticks, eval…) ask once and remember
// by their first word: 「一律允許」 on `git` covers `git … $(…)` afterwards, so
// the same shape of command doesn't ask again (models use $(…) constantly).
if let Some(d) = denied_hit(&cmd) {
return Decision::Deny(format!("「{d}」這類指令一律不允許。"));
}
let grant = shell_grant(&cmd);
if !writes_files(&cmd) && granted(&grant) {
return Decision::Allow;
}
return Decision::Ask { reason: format!("執行指令（含有隱藏的子指令，無法自動判斷）：{cmd}"), grant };
}; };
// deny // deny
for part in &parts { for part in &parts {
@ -338,14 +351,8 @@ pub fn decide(name: &str, args: &str, workdir: &Path, grants: &[String]) -> Deci
return Decision::Deny(format!("「{d}」這類指令一律不允許。")); return Decision::Deny(format!("「{d}」這類指令一律不允許。"));
} }
} }
// ignore: commands that point at secret files // (No word-by-word path scan here: secret folders are the sandbox's job, and
for part in &parts { // scanning words as paths denied perfectly ordinary commands.)
for w in words(part).iter().skip(1) {
if is_ignored(&resolve(workdir, w.trim_matches(['"', '\'']))) {
return Decision::Deny(format!("指令碰到私密或系統檔案：{w}"));
}
}
}
// allow: every part is read-only or already granted // allow: every part is read-only or already granted
let ok = |part: &String| !writes_files(part) && (READ_ONLY_COMMANDS.iter().any(|r| starts_with_words(part, r)) || granted(&shell_grant(part))); let ok = |part: &String| !writes_files(part) && (READ_ONLY_COMMANDS.iter().any(|r| starts_with_words(part, r)) || granted(&shell_grant(part)));
let unapproved: Vec<&String> = parts.iter().filter(|p| !ok(p)).collect(); let unapproved: Vec<&String> = parts.iter().filter(|p| !ok(p)).collect();
@ -466,9 +473,14 @@ pub fn protected_dirs() -> Vec<PathBuf> {
} }
/// Inside the protected folders, what commands may still read: the chat's attachments (a picture /// Inside the protected folders, what commands may still read: the chat's attachments (a picture
/// the person sent is the bot's to look at; the keys next to it are not). /// the person sent is the bot's to look at; the keys next to it are not) and the ssh keys, so
/// git and ssh can reach remote repos (writing them stays denied).
pub fn readable_dirs() -> Vec<PathBuf> { pub fn readable_dirs() -> Vec<PathBuf> {
crate::model::State::path().parent().map(|d| vec![d.join("attachments")]).unwrap_or_default() let mut v: Vec<PathBuf> = crate::model::State::path().parent().map(|d| vec![d.join("attachments")]).unwrap_or_default();
if let Some(h) = dirs::home_dir() {
v.push(h.join(".ssh"));
}
v
} }
fn sandbox_exec() -> Option<&'static str> { fn sandbox_exec() -> Option<&'static str> {
@ -525,6 +537,9 @@ pub fn build_command_sealed(cmd: &str, workdir: &Path, protected: &[PathBuf], re
for k in KEY_ENV_VARS { for k in KEY_ENV_VARS {
c.env_remove(k); c.env_remove(k);
} }
// The app started from Finder or launchd has no shell PATH (no /opt/homebrew/bin): a
// command run on an always-on Mac (a Mac mini under LaunchAgent) must find its tools too.
c.env("PATH", crate::cloudbox::tool_path());
c.env(ENV_CONVERSATION_ID, conversation).env(ENV_REQUEST_ID, request); c.env(ENV_CONVERSATION_ID, conversation).env(ENV_REQUEST_ID, request);
#[cfg(unix)] #[cfg(unix)]
{ {
@ -616,8 +631,9 @@ pub enum Place {
Local { workdir: PathBuf }, Local { workdir: PathBuf },
/// Another computer: `ssh target`, starting in `dir` (empty = its home). /// Another computer: `ssh target`, starting in `dir` (empty = its home).
Ssh { target: String, dir: String }, Ssh { target: String, dir: String },
/// 裝置連結: another AnyBot of the same account (through the relay), by its host id. /// 裝置連結: another AnyBot of the same account (through the relay), by its host id,
Device { host: String }, /// starting in `dir` (empty = its home).
Device { host: String, dir: String },
} }
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
@ -641,9 +657,8 @@ pub struct Roles {
pub fn specs_for(computers: &[String], default: &str) -> Vec<Value> { pub fn specs_for(computers: &[String], default: &str) -> Vec<Value> {
let mut v = specs(); let mut v = specs();
for t in v.iter_mut() { for t in v.iter_mut() {
if t["name"] == "read_terminal" || t["name"] == "write_terminal" { // Every tool that works on a computer names one, background terminals included: a command
continue; // moved to the background on the Mac mini is read on the Mac mini.
}
// Each computer goes by its own name; the instructions say which is which. Working on // Each computer goes by its own name; the instructions say which is which. Working on
// one and calling it another (the server as "your computer") is what confused people. // one and calling it another (the server as "your computer") is what confused people.
let desc = format!("Which computer to work on, by its name. Default: {default} (the one the person chose). Use another only when the person names it; when unsure which one they mean, ask."); let desc = format!("Which computer to work on, by its name. Default: {default} (the one the person chose). Use another only when the person names it; when unsure which one they mean, ask.");
@ -760,10 +775,30 @@ pub fn run_on(name: &str, args: &str, target: &Target, cancel: &AtomicBool, ids:
}; };
match &target.place { match &target.place {
Place::Local { workdir } => run(name, args, workdir, cancel, ids), Place::Local { workdir } => run(name, args, workdir, cancel, ids),
Place::Device { host } => match crate::remote::run_on_device(host, name, args, "") { Place::Device { host, dir } => {
Ok(v) => (v["text"].as_str().unwrap_or("").to_string(), v["ok"].as_bool().unwrap_or(false)), // The device runs the tool in the bot's work folder (its own home when unset), and
// must answer before the relay gives up at 35 s: long commands move to the
// background there, and read_terminal now names this computer to find them.
let mut v = v.clone();
let cap = timeout_of(&v).min(DEVICE_SHELL_TIMEOUT_MS);
if let Some(o) = v.as_object_mut() {
if !dir.is_empty() && o.get("workdir").and_then(Value::as_str).unwrap_or("").is_empty() {
o.insert("workdir".into(), serde_json::Value::String(dir.clone()));
}
o.insert("timeout_ms".into(), serde_json::Value::from(cap));
}
let args = v.to_string();
match crate::remote::run_on_device(host, name, &args, "") {
Ok(v) => {
let mut text = v["text"].as_str().unwrap_or("").to_string();
if text.contains("read_terminal") {
text.push_str(&format!("（這台是「{}」：read_terminal 的 computer 填「{}」。）", target.name, target.name));
}
(text, v["ok"].as_bool().unwrap_or(false))
}
Err(e) => (e, false), Err(e) => (e, false),
}, }
}
Place::Sealed { root } => { Place::Sealed { root } => {
let _ = std::fs::create_dir_all(root); let _ = std::fs::create_dir_all(root);
if name == "shell" { if name == "shell" {
@ -1235,7 +1270,12 @@ mod tests {
let w = wd(); let w = wd();
assert_eq!(decide("shell", r#"{"command":"ls && git status"}"#, &w, &[]), Decision::Allow); assert_eq!(decide("shell", r#"{"command":"ls && git status"}"#, &w, &[]), Decision::Allow);
assert!(matches!(decide("shell", r#"{"command":"ls && sudo rm x"}"#, &w, &[]), Decision::Deny(_))); assert!(matches!(decide("shell", r#"{"command":"ls && sudo rm x"}"#, &w, &[]), Decision::Deny(_)));
assert!(matches!(decide("shell", r#"{"command":"cat .env"}"#, &w, &[]), Decision::Deny(_))); // Secrets are the sandbox's job; the permission layer no longer denies words that
// merely look like secret paths. Read-only commands run (cat .env is cat), and the
// sandbox hides the protected folders from them.
assert_eq!(decide("shell", r#"{"command":"cat .env"}"#, &w, &[]), Decision::Allow);
assert!(matches!(decide("shell", r#"{"command":"echo $(whoami)"}"#, &w, &[]), Decision::Ask { ref grant, .. } if grant == "shell:echo"));
assert_eq!(decide("shell", r#"{"command":"echo $(whoami)"}"#, &w, &["shell:echo".into()]), Decision::Allow);
assert!(matches!(decide("shell", r#"{"command":"npm install"}"#, &w, &[]), Decision::Ask { ref grant, .. } if grant == "shell:npm install")); assert!(matches!(decide("shell", r#"{"command":"npm install"}"#, &w, &[]), Decision::Ask { ref grant, .. } if grant == "shell:npm install"));
assert_eq!(decide("shell", r#"{"command":"npm install && ls"}"#, &w, &["shell:npm install".into()]), Decision::Allow); assert_eq!(decide("shell", r#"{"command":"npm install && ls"}"#, &w, &["shell:npm install".into()]), Decision::Allow);
assert!(matches!(decide("shell", r#"{"command":"echo hi > out.txt"}"#, &w, &[]), Decision::Ask { .. })); assert!(matches!(decide("shell", r#"{"command":"echo hi > out.txt"}"#, &w, &[]), Decision::Ask { .. }));