Compare commits

...

4 Commits

Author SHA1 Message Date
王性驊 212d18b542 Merge branch 'smooth-remote': people watch and help on the bots' computer through VNC (noVNC) over the account server; unused and duplicated code removed
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 23:27:11 +08:00
王性驊 2867c8e822 Remove what nothing uses and keep one copy of what was written several times
Unused (found by taking away the dead-code allowances; nothing else used them, tests aside):
computer::specs, quiet_button, the never-made Poke::Spin, ten icons and their SVGs, the
theme's chrome/bubble_agent colours, ImportRow.line, voice's multipart/transcribe/speak
wrappers, and the old keychain writers vault::save/remove/index_add with their live test
(the vault store replaced them; reading old keychain entries for the import stays).

One copy each: percent-encoding (httpd::url_encode, was four), tool arguments (tools::args_of,
was three), the WebSocket handshake (httpd::accept_websocket, was three), the read pacing
(wire::Pace, was two: relay and app), docker with input and a time limit (dk_with_input, was
two). Docs: FEATURES §看畫面 (VNC), protocol (v* frames, /screen, tickets), architecture.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 23:21:47 +08:00
王性驊 02bca39b98 People watch and help on the bots' computer through VNC (noVNC), carried by the account server
The core: the bot works the computer; a person watches it like a colleague, and takes over
when it needs them. VNC is made for that (only what changed, every mouse move and key, touch on
phones), so the pictures sent one by one go, and with them their own input path.

- vnc.rs: a VNC proxy to the box's x11vnc through one docker exec (a Python bridge that ends
  with either side). It signs in to the box with the box's own password (its VNC_PASSWORD) and
  offers the viewer no password, so the password never leaves the brain. The person's clicks
  and keys make the bot's steps wait until 8 s after they stop; text typed in the viewer's box
  (any language) is pasted in the box.
- Account server: /screen (the viewer page) and /novnc.js (noVNC 1.6.0, MPL-2.0, bundled);
  /relay/vnc/<host> takes the browser's WebSocket (account session, or a 10-minute ticket the
  app asks for with POST /relay/screen-ticket; owners and admins) and carries it over that
  AnyBot's lasting connection (vopen / vdata / vtext / vclose). Both ends of that connection
  wait 2 ms instead of 40 ms to read while things move: a request and its answer took up to
  160 ms more than needed.
- Phone page: 看 Bot 的電腦 and 接管 open the viewer page (also on the local network: /api/vnc on
  the AnyBot's own phone server). This Mac's own screen moved to the ⋯ menu.
- Mac app: 看畫面 opens the viewer in the browser (a screen of a brain: the brain's computer by
  ticket; else this computer's box, noVNC straight from it). The in-app picture viewer, the
  live JPEG stream, the person's click/move/scroll/key endpoints and the input shell are gone.
- Opening a Mac app uses LaunchServices (`open -a`): agent-desktop's launch waited on the window
  through accessibility and failed without 輔助使用 now and then.

Measured on this Mac (made-up account, local relay + headless brain + the real box): viewer
signed in in 0.4–0.7 s, a request and its answer through relay + brain + box 25–58 ms median;
headless Chrome shows the desktop live in /screen. remote_e2e 28/28 ×2, 149 unit tests, relay 11.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 23:05:44 +08:00
王性驊 6c0c4d7ecd Watching and using the bots' computer from afar: pictures as the screen changes, input through a lasting shell
- One lasting docker exec takes the desktop as a JPEG inside the box and sends a picture only
  when it changed (quicker while it changes, slower while still). Viewers ask for the picture
  after the one they have and get it at once. It was one exec per picture (a 1 MB PNG decoded
  and encoded again on the brain) and a fixed 0.7 s / 1 s pause: about one picture a second.
  Measured on the Mac's box: first picture 45 ms, ~90 KB (was ~1.1 MB), 9–19 pictures a second
  while it changes; viewers take at most ~8 a second over the network.
- The loop in the box ends the one before it, ends itself when a write fails ("0" lines while
  still) and is ended when nobody watched for 10 s (ending docker exec on this side left it
  running in the box).
- The person's clicks and keys go through one lasting shell in the box (text as base64), not a
  docker exec each.
- Answers over the lasting connection keep their headers (the relay dropped them): a picture's
  number reaches the app. JPEG/PNG/PDF/zip bodies are no longer deflated again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 22:34:36 +08:00
39 changed files with 930 additions and 996 deletions

10
Cargo.lock generated
View File

@ -345,6 +345,7 @@ dependencies = [
"chrono-tz", "chrono-tz",
"cpal", "cpal",
"crossterm", "crossterm",
"des",
"dirs", "dirs",
"eframe", "eframe",
"flate2", "flate2",
@ -1356,6 +1357,15 @@ dependencies = [
"syn 2.0.119", "syn 2.0.119",
] ]
[[package]]
name = "des"
version = "0.8.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ffdd80ce8ce993de27e9f063a444a4d53ce8e8db4c1f00cc03af5ad5a9867a1e"
dependencies = [
"cipher",
]
[[package]] [[package]]
name = "digest" name = "digest"
version = "0.10.7" version = "0.10.7"

View File

@ -25,6 +25,8 @@ serde_json = "1"
dirs = "7" dirs = "7"
image = { version = "0.25", default-features = false, features = ["png", "jpeg", "gif", "webp", "bmp", "tiff", "ico"] } image = { version = "0.25", default-features = false, features = ["png", "jpeg", "gif", "webp", "bmp", "tiff", "ico"] }
base64 = "0.22" base64 = "0.22"
# VNC sign-in (the 16-byte challenge) for people watching the bots' computer.
des = "0.8"
chrono = "0.4" chrono = "0.4"
chrono-tz = "0.10" chrono-tz = "0.10"
iana-time-zone = "0.1" iana-time-zone = "0.1"

View File

@ -729,15 +729,17 @@ AnyBot 是一個桌面聊天 App,介面和運作方式照 Grok Bot 重做。
- 換位置後會在新的地方重新建立;舊檔案可以先「儲存快照」,換過去再「從快照復原」。 - 換位置後會在新的地方重新建立;舊檔案可以先「儲存快照」,換過去再「從快照復原」。
- **「網路流量改走這台 Mac」**(原版:網路流量可以改走你的本機電腦,「僅對新連線生效」,管理員可以關掉):打開後,Bot 的電腦之後執行的指令會經過這台 Mac 上的代理伺服器出去(例如要連公司內網或 VPN 才看得到的服務)。代理只聽這台 Mac 自己(127.0.0.1),用原版的埠 8791,被占用就往後找。設定頁顯示經過的連線數。Bot 的電腦在別的主機上時不能用。管理員可以用 `defaults write com.anybot.AnyBot AllowBoxTrafficViaThisMac false` 鎖住,這時顯示「🔒 由你的管理員設定,無法變更」。 - **「網路流量改走這台 Mac」**(原版:網路流量可以改走你的本機電腦,「僅對新連線生效」,管理員可以關掉):打開後,Bot 的電腦之後執行的指令會經過這台 Mac 上的代理伺服器出去(例如要連公司內網或 VPN 才看得到的服務)。代理只聽這台 Mac 自己(127.0.0.1),用原版的埠 8791,被占用就往後找。設定頁顯示經過的連線數。Bot 的電腦在別的主機上時不能用。管理員可以用 `defaults write com.anybot.AnyBot AllowBoxTrafficViaThisMac false` 鎖住,這時顯示「🔒 由你的管理員設定,無法變更」。
### 看畫面、取得控制權 ### 看畫面、取得控制權(像同事一樣看,需要時接手)
- 設定頁的「開啟畫面」,或聊天頂部的「電腦」按鈕。 - 核心只有三件事:Bot 操作它的電腦;你像同事一樣即時看它在做什麼;它需要你(登入、驗證碼)時你接手幫忙。看和接手用 **VNC(noVNC)**:只傳畫面上變了的地方,每個滑鼠移動和按鍵即時送過去,手機可以用觸控。原版也是用 noVNC 看 box。
- 視窗:螢幕圖示+「Bot 的電腦」+狀態圓點;右邊是「取得控制權」「實際大小/調整至視窗大小」「全螢幕」和關閉。 - **在哪裡看**:手機網頁的「看 Bot 的電腦」、需要處理卡片的「接管」,Mac 和其他電腦的「看畫面」「開啟電腦」都打開同一個網頁 `/screen`(電腦上在瀏覽器開)。
- 畫面大約每 0.45 秒更新一次(估計值)。Bot 正在操作時顯示「● Bot 正在操作」。 - 大腦在別台(伺服器):網頁在帳號伺服器上,VNC 經過帳號伺服器和大腦的長連線到大腦的 Bot 的電腦。電腦上的 App 會先向帳號伺服器要一個 10 分鐘有效的連結(不用在瀏覽器登入);手機網頁用自己的登入。只有擁有者和管理員能看。
- **取得控制權**:畫面外框變成強調色,你的點擊、右鍵、連點、捲動、打字(包括中文)和快速鍵都會送到 Bot 的電腦(⌘ 會當成 Ctrl)。這時 Bot 的滑鼠鍵盤操作會暫停,它會收到「使用者正在自己操作 Bot 的電腦,請等他交還控制權」。按「交還給 Bot」結束。 - 同一個網路的手機(不經帳號伺服器):同一個網頁,連 AnyBot 自己的 `/api/vnc`。
- **僅觀看**:沒有取得控制權時,畫面只能看。 - 這台電腦自己的 Bot 的電腦:直接開它的 noVNC。
- **全螢幕**:在瀏覽器用 noVNC 開啟(有控制權時可以操作,沒有時只能看)。 - **網頁**:最上面是返回、電腦名稱、狀態(即時觀看/你在操作)和「接管/交還給 Bot」。預設只看,按「接管」才會把你的滑鼠鍵盤送過去;需要處理卡片的「接管」直接打開接管。接管時有「打字」:打中文或任何語言(貼進目前的視窗),還有 Enter、Esc、Tab、⌫。斷線會自己重連。
- 電腦休眠或還沒建立時,畫面中間有「喚醒」按鈕。 - **你在操作時 Bot 會等**:你點擊或按鍵後 8 秒內,Bot 的滑鼠鍵盤操作會收到「使用者正在自己操作這台電腦」。不用另外按什麼;停下 8 秒它就繼續。需要處理卡片的「接管」會讓 Bot 一直等,直到你按「我完成了」或「跳過」。
- **VNC 密碼不出大腦**:大腦用 Bot 的電腦自己的 VNC 密碼登入,給看的人的是「不用密碼」;能連到這裡的人已經過帳號伺服器(或手機密碼)。
- 以前的做法(每 0.45~1 秒拍一張 1 MB 的圖、解開再壓、經過帳號伺服器傳;點擊另走一條路)已經拆掉。
--- ---

View File

@ -2,8 +2,7 @@
# Build and test the Linux version from a Mac (or anywhere with Docker), on a virtual screen. # Build and test the Linux version from a Mac (or anywhere with Docker), on a virtual screen.
# docker build -t anybot-linux-dev deploy/linux/dev (once) # docker build -t anybot-linux-dev deploy/linux/dev (once)
# deploy/linux/dev/test.sh (from app/) # deploy/linux/dev/test.sh (from app/)
# Runs: every unit test; the live X11 test (screenshot, clicks, keys, recording a demo); the # Runs: every unit test; the live X11 test (screenshot, clicks, keys, recording a demo); the live Wayland test (headless sway: screenshot,
# keyring test (save / find / remove a login); the live Wayland test (headless sway: screenshot,
# clicks, Chinese typing, keys, scroll); and screenshots of the window into ./linux-shots. # clicks, Chinese typing, keys, scroll); and screenshots of the window into ./linux-shots.
set -e set -e
cd "$(dirname "$0")/../../.." cd "$(dirname "$0")/../../.."
@ -17,7 +16,6 @@ export DISPLAY=:99 LANG=zh_TW.UTF-8
sleep 1 sleep 1
cargo test --release 2>&1 | grep -E "test result|FAILED|panicked" cargo test --release 2>&1 | grep -E "test result|FAILED|panicked"
cargo test --release x11_live -- --ignored 2>&1 | grep -E "test result|panicked" cargo test --release x11_live -- --ignored 2>&1 | grep -E "test result|panicked"
dbus-run-session -- sh -c "echo -n test | gnome-keyring-daemon --unlock --components=secrets >/dev/null 2>&1; sleep 1; cargo test --release keyring_live -- --ignored 2>&1 | grep -E \"test result|panicked\""
useradd -m t 2>/dev/null; chmod 777 /out useradd -m t 2>/dev/null; chmod 777 /out
runuser -u t -- env CARGO_TARGET_DIR=/target HOME=/home/t sh deploy/linux/dev/wayland_live.sh 2>&1 | sed -n "/--- wev heard/,\$p" runuser -u t -- env CARGO_TARGET_DIR=/target HOME=/home/t sh deploy/linux/dev/wayland_live.sh 2>&1 | sed -n "/--- wev heard/,\$p"
for v in chat computers general; do for v in chat computers general; do

View File

@ -42,7 +42,8 @@ AnyBot 是一個 Rust 程式(`anybot`),用 egui/eframe 畫介面。同一
| `tools.rs` | 檔案、指令、背景指令;指令在 macOS 沙盒裡跑(見「安全」) | | `tools.rs` | 檔案、指令、背景指令;指令在 macOS 沙盒裡跑(見「安全」) |
| `policy.rs` | 哪些動作直接做、先問、拒絕 | | `policy.rs` | 哪些動作直接做、先問、拒絕 |
| `cloudbox.rs`、`box/` | Bot 自己的電腦:一個 Docker 容器(有桌面、瀏覽器) | | `cloudbox.rs`、`box/` | Bot 自己的電腦:一個 Docker 容器(有桌面、瀏覽器) |
| `computer.rs`、`desktop.rs`、`axread.rs` | 看畫面、滑鼠鍵盤;讀 App 的按鈕清單(agent-desktop) | | `computer.rs`、`desktop.rs`、`axread.rs` | 看畫面、滑鼠鍵盤;讀 App 的按鈕清單(agent-desktop,只讀) |
| `vnc.rs`、`mobile/screen.html` | 人看 Bot 的電腦、需要時接手:VNC 代理(經帳號伺服器或同一個網路),noVNC 網頁 |
| `botbrowser.rs`、`browser.rs`、`web.rs` | AnyBot 自己的瀏覽器(Chrome DevTools 協定)、讀你的瀏覽器、抓網頁 | | `botbrowser.rs`、`browser.rs`、`web.rs` | AnyBot 自己的瀏覽器(Chrome DevTools 協定)、讀你的瀏覽器、抓網頁 |
| `routines.rs`、`triggers.rs` | 例行任務和觸發 | | `routines.rs`、`triggers.rs` | 例行任務和觸發 |
| `channels.rs` | 訊息管道(Slack、Discord、Telegram) | | `channels.rs` | 訊息管道(Slack、Discord、Telegram) |

View File

@ -54,6 +54,14 @@
轉送時伺服器附上 `user`、`role` 和 `from`(問的那台主機 id)。對方不在線時立刻回 503「目前離線」;收到了但 35 秒內沒回答,回 504。 轉送時伺服器附上 `user`、`role` 和 `from`(問的那台主機 id)。對方不在線時立刻回 503「目前離線」;收到了但 35 秒內沒回答,回 504。
看 Bot 的電腦(VNC,一條串流,不是一問一答):
| 路徑 | 誰 | 說明 |
|---|---|---|
| `GET /screen`、`GET /novnc.js` | 任何人 | 看畫面的網頁和 noVNC 1.6.0(MPL-2.0) |
| `POST /relay/screen-ticket` `{host}` | 同帳號的主機(host token) | → `{ticket}`:10 分鐘內有效,讓這台的 App 開瀏覽器看那台的電腦 |
| WebSocket `/relay/vnc/<host>?t=` 或 `?ticket=` | 擁有者、管理員 | 二進位訊息是 VNC;文字訊息是要打進去的字。伺服器把它接到那台 AnyBot 的長連線(下面的 `v*`) |
## AnyBot 提供的介面(`remote::handle`) ## AnyBot 提供的介面(`remote::handle`)
只有經過帳號伺服器(或同一個網路、用手機密碼登入)才到得了。 只有經過帳號伺服器(或同一個網路、用手機密碼登入)才到得了。
@ -63,7 +71,11 @@
| GET | `/api/state?since=` | 手機 | Bot 清單、誰在工作、等核准的事(有變化才回,最多等 20 秒) | | GET | `/api/state?since=` | 手機 | Bot 清單、誰在工作、等核准的事(有變化才回,最多等 20 秒) |
| GET | `/api/bots/<id>?since=` | 手機 | 一個 Bot 的訊息 | | GET | `/api/bots/<id>?since=` | 手機 | 一個 Bot 的訊息 |
| POST | `/api/bots/<id>/<動作>` | 手機 | `send`、`stop`、`answer`、`read`、`new`、`upload`… | | POST | `/api/bots/<id>/<動作>` | 手機 | `send`、`stop`、`answer`、`read`、`new`、`upload`… |
| GET/POST | `/api/screen/<box\|mac>` | 手機 | 看畫面、點擊打字 | | GET/POST | `/api/screen/mac` | 手機 | 看這台 Mac 的畫面、點擊打字(開了「從手機操作」才可以) |
| GET | `/screen`、`/novnc.js` | 任何人 | 看 Bot 的電腦的網頁(noVNC);本身沒有資料 |
| WebSocket | `/api/vnc?t=` | 手機(同一個網路) | VNC,直接到這台的 Bot 的電腦(見 vnc.rs) |
| GET | `/api/sync/screen` | mirror | Bot 的電腦的小縮圖(JPEG,寬 320) |
| POST | `/api/sync/screen` | mirror | `{kind: "hold", on}`(接管時 Bot 等你)、`{kind: "wake"}` |
| POST | `/api/device/run` | device | `{tool, args, who}` → `{text, ok, image}`:在這台跑一個工具 | | POST | `/api/device/run` | device | `{tool, args, who}` → `{text, ok, image}`:在這台跑一個工具 |
| GET | `/api/sync?since=&client=` | mirror | 大腦的變化(見 architecture.md) | | GET | `/api/sync?since=&client=` | mirror | 大腦的變化(見 architecture.md) |
| POST | `/api/sync/edit` | mirror | `{ops, client, seq}` → `{ok, applied}` | | POST | `/api/sync/edit` | mirror | `{ops, client, seq}` → `{ok, applied}` |
@ -88,9 +100,13 @@ WebSocket 的二進位訊息,每則是:
| `job` | 伺服器 → AnyBot | `id, method, path, query, user, role, from, z, z_ok` | 給這台的要求 | | `job` | 伺服器 → AnyBot | `id, method, path, query, user, role, from, z, z_ok` | 給這台的要求 |
| `reply` | AnyBot → 伺服器 | `id, status, ctype, headers, z` | 回答 job | | `reply` | AnyBot → 伺服器 | `id, status, ctype, headers, z` | 回答 job |
| `call` | AnyBot → 伺服器 | `id, to, method, path, query, z` | 要問帳號裡的另一台(只能是 `/api/sync…` 或 `/api/device/run`) | | `call` | AnyBot → 伺服器 | `id, to, method, path, query, z` | 要問帳號裡的另一台(只能是 `/api/sync…` 或 `/api/device/run`) |
| `ret` | 伺服器 → AnyBot | `id, status, ctype, z` | call 的回答 | | `ret` | 伺服器 → AnyBot | `id, status, ctype, z, headers` | call 的回答 |
| `vopen` | 伺服器 → AnyBot | `sid` | 有人要看這台的 Bot 的電腦:接上它的 VNC(AnyBot 先回 VNC 的開頭) |
| `vdata` | 雙向 | `sid` | VNC 的位元組(內容) |
| `vtext` | 伺服器 → AnyBot | `sid` | 看的人打的字(UTF-8),貼進 Bot 的電腦 |
| `vclose` | 雙向 | `sid`(AnyBot 送的有 `why`) | 結束這條 |
心跳用 WebSocket 的 ping/pong。連線斷掉時,伺服器把還在等的 job 都回 503「斷線了」,AnyBot 把還在等的 call 都回「和帳號伺服器的連線斷了」。 兩邊的迴圈有東西在跑時每 2 毫秒、安靜時每 40 毫秒看一次有沒有要讀的(`wire::Pace`),之後立刻送出排隊的訊息。心跳用 WebSocket 的 ping/pong。連線斷掉時,伺服器把還在等的 job 都回 503「斷線了」,AnyBot 把還在等的 call 都回「和帳號伺服器的連線斷了」。
## 同步的改動(`/api/sync/edit` 的 ops) ## 同步的改動(`/api/sync/edit` 的 ops)

View File

@ -1,15 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-chevron-down"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="m6 9 6 6 6-6" />
</svg>

Before

Width:  |  Height:  |  Size: 319 B

View File

@ -1,17 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-external-link"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="M15 3h6v6" />
<path d="M10 14 21 3" />
<path d="M18 13v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6" />
</svg>

Before

Width:  |  Height:  |  Size: 416 B

View File

@ -1,17 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-globe"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<circle cx="12" cy="12" r="10" />
<path d="M12 2a14.5 14.5 0 0 0 0 20 14.5 14.5 0 0 0 0-20" />
<path d="M2 12h20" />
</svg>

Before

Width:  |  Height:  |  Size: 407 B

View File

@ -1,17 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-info"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<circle cx="12" cy="12" r="10" />
<path d="M12 16v-4" />
<path d="M12 8h.01" />
</svg>

Before

Width:  |  Height:  |  Size: 369 B

View File

@ -1,23 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-keyboard"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="M10 8h.01" />
<path d="M12 12h.01" />
<path d="M14 8h.01" />
<path d="M16 12h.01" />
<path d="M18 8h.01" />
<path d="M6 8h.01" />
<path d="M7 16h10" />
<path d="M8 12h.01" />
<rect width="20" height="16" x="2" y="4" rx="2" />
</svg>

Before

Width:  |  Height:  |  Size: 540 B

View File

@ -1,16 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-mail"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="m22 7-8.991 5.727a2 2 0 0 1-2.009 0L2 7" />
<rect x="2" y="4" width="20" height="16" rx="2" />
</svg>

Before

Width:  |  Height:  |  Size: 391 B

View File

@ -1,16 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-panel-right"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<rect width="18" height="18" x="3" y="3" rx="2" />
<path d="M15 3v18" />
</svg>

Before

Width:  |  Height:  |  Size: 367 B

View File

@ -1,15 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-phone"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="M13.832 16.568a1 1 0 0 0 1.213-.303l.355-.465A2 2 0 0 1 17 15h3a2 2 0 0 1 2 2v3a2 2 0 0 1-2 2A18 18 0 0 1 2 4a2 2 0 0 1 2-2h3a2 2 0 0 1 2 2v3a2 2 0 0 1-.8 1.6l-.468.351a1 1 0 0 0-.292 1.233 14 14 0 0 0 6.392 6.384" />
</svg>

Before

Width:  |  Height:  |  Size: 513 B

View File

@ -1,17 +0,0 @@
<!-- @license lucide-static v1.67.0 - ISC -->
<svg
class="lucide lucide-volume-2"
xmlns="http://www.w3.org/2000/svg"
width="24"
height="24"
viewBox="0 0 24 24"
fill="none"
stroke="currentColor"
stroke-width="2"
stroke-linecap="round"
stroke-linejoin="round"
>
<path d="M11 4.702a.705.705 0 0 0-1.203-.498L6.413 7.587A1.4 1.4 0 0 1 5.416 8H3a1 1 0 0 0-1 1v6a1 1 0 0 0 1 1h2.416a1.4 1.4 0 0 1 .997.413l3.383 3.384A.705.705 0 0 0 11 19.298z" />
<path d="M16 9a5 5 0 0 1 0 6" />
<path d="M19.364 18.364a9 9 0 0 0 0-12.728" />
</svg>

Before

Width:  |  Height:  |  Size: 555 B

View File

@ -272,7 +272,7 @@ a.plink{color:var(--accent);text-decoration:underline;text-underline-offset:3px;
<section class="view" id="v-screen"> <section class="view" id="v-screen">
<header> <header>
<button class="icon-btn" id="b-back2"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><path d="M15 5l-7 7 7 7"/></svg></button> <button class="icon-btn" id="b-back2"><svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2"><path d="M15 5l-7 7 7 7"/></svg></button>
<div class="seg" id="seg"><button data-w="box" class="on">Bot 的電腦</button><button data-w="mac">這台 Mac</button></div> <div class="title">這台 Mac</div>
<span style="flex:1"></span> <span style="flex:1"></span>
<span class="sub" id="screen-live"></span> <span class="sub" id="screen-live"></span>
</header> </header>
@ -287,7 +287,6 @@ a.plink{color:var(--accent);text-decoration:underline;text-underline-offset:3px;
<button class="btn" data-scroll="up">↑ 捲動</button> <button class="btn" data-scroll="up">↑ 捲動</button>
<button class="btn" data-scroll="down">↓ 捲動</button> <button class="btn" data-scroll="down">↓ 捲動</button>
<button class="btn" id="k-right">右鍵:關</button> <button class="btn" id="k-right">右鍵:關</button>
<button class="btn" id="k-wake">叫醒</button>
</div> </div>
</section> </section>
</div> </div>
@ -295,6 +294,7 @@ a.plink{color:var(--accent);text-decoration:underline;text-underline-offset:3px;
<div class="sheet-bg" id="sheet"><div class="sheet"> <div class="sheet-bg" id="sheet"><div class="sheet">
<button id="m-notify">開啟通知</button> <button id="m-notify">開啟通知</button>
<div class="note" id="m-note">Bot 需要你允許、或在你不在時做完事,手機會跳通知。</div> <div class="note" id="m-note">Bot 需要你允許、或在你不在時做完事,手機會跳通知。</div>
<button id="m-mac" style="display:none">看這台 Mac 的螢幕</button>
<button id="m-pair" style="display:none">加一台電腦(Linux、伺服器)</button> <button id="m-pair" style="display:none">加一台電腦(Linux、伺服器)</button>
<button id="m-host" style="display:none">切換到別台 AnyBot</button> <button id="m-host" style="display:none">切換到別台 AnyBot</button>
<button class="no" id="m-out">登出這支手機</button> <button class="no" id="m-out">登出這支手機</button>
@ -692,7 +692,7 @@ function renderAsk() {
// 需要處理 (a step only you can do on a computer): its screen is one tap away, then 我完成了 / 跳過. // 需要處理 (a step only you can do on a computer): its screen is one tap away, then 我完成了 / 跳過.
if (a.kind === "handoff") { if (a.kind === "handoff") {
$("ask").innerHTML = `<div class="ask"><div class="q">需要處理 · ${esc(a.computer || "")}</div><div class="why">${esc(a.reason)}</div><div class="btns"><button class="btn" data-screen="1">接管</button><button class="btn main" data-a="once">我完成了</button><button class="btn no" data-a="reject" title="取消此要求且不執行該步驟;Bot 會跳過它並繼續">跳過</button></div></div>`; $("ask").innerHTML = `<div class="ask"><div class="q">需要處理 · ${esc(a.computer || "")}</div><div class="why">${esc(a.reason)}</div><div class="btns"><button class="btn" data-screen="1">接管</button><button class="btn main" data-a="once">我完成了</button><button class="btn no" data-a="reject" title="取消此要求且不執行該步驟;Bot 會跳過它並繼續">跳過</button></div></div>`;
$("ask").querySelector("[data-screen]").onclick = () => { S.back = S.view; show("screen"); $("seg").children[0].textContent = ownComputer(); $("seg").children[1].style.display = S.mac ? "" : "none"; startFrames(); }; $("ask").querySelector("[data-screen]").onclick = () => openBox(true);
} else { } else {
$("ask").innerHTML = `<div class="ask"><div class="q">${esc(a.title)}</div><div class="why">${esc(a.reason)}</div><div class="btns"><button class="btn main" data-a="once">允許一次</button>${a.can_always ? `<button class="btn" data-a="always">一律允許</button>` : ""}<button class="btn no" data-a="reject">拒絕</button></div></div>`; $("ask").innerHTML = `<div class="ask"><div class="q">${esc(a.title)}</div><div class="why">${esc(a.reason)}</div><div class="btns"><button class="btn main" data-a="once">允許一次</button>${a.can_always ? `<button class="btn" data-a="always">一律允許</button>` : ""}<button class="btn no" data-a="reject">拒絕</button></div></div>`;
} }
@ -737,27 +737,35 @@ function renderChips() {
// ---------- screens ---------- // ---------- screens ----------
// Each bot's own computer goes by the bot's name (「Master 的電腦」). // Each bot's own computer goes by the bot's name (「Master 的電腦」).
function ownComputer() { const n = botName(S.open); return n ? n + " 的電腦" : "Bot 的電腦"; } function ownComputer() { const n = botName(S.open); return n ? n + " 的電腦" : "Bot 的電腦"; }
$("b-screen").onclick = $("b-screen2").onclick = () => { S.back = S.view; show("screen"); $("seg").children[0].textContent = ownComputer(); $("seg").children[1].style.display = S.mac ? "" : "none"; startFrames(); }; // The bots' computer: live in the viewer page (VNC), where you can also take over.
function openBox(control) {
const at = S.relay ? "host=" + encodeURIComponent(S.host) : "lan=1";
location.href = "screen#" + at + "&name=" + encodeURIComponent(ownComputer()) + (control ? "&control=1" : "");
}
$("b-screen").onclick = $("b-screen2").onclick = () => openBox(false);
$("m-mac").onclick = () => { $("sheet").classList.remove("on"); S.back = S.view; S.screen = "mac"; show("screen"); startFrames(); };
$("b-back2").onclick = () => show(S.back === "chat" ? "chat" : "list"); $("b-back2").onclick = () => show(S.back === "chat" ? "chat" : "list");
document.querySelectorAll("#seg button").forEach((b) => b.onclick = () => { document.querySelectorAll("#seg button").forEach((x) => x.classList.toggle("on", x === b)); S.screen = b.dataset.w; $("k-wake").style.display = S.screen === "box" ? "" : "none"; startFrames(); }); // This Mac's screen: a picture at a time.
let busyFrame = false, lastUrl = null; let lastUrl = null;
function startFrames() { stopFrames(); frame(); S.frameTimer = setInterval(frame, 1000); } const nap = (ms) => new Promise((r) => setTimeout(r, ms));
function stopFrames() { clearInterval(S.frameTimer); S.frameTimer = null; } function startFrames() { stopFrames(); const run = S.frameRun = {}; frames(run); }
async function frame() { function stopFrames() { S.frameRun = null; }
if (busyFrame || document.hidden) return; busyFrame = true; async function frames(run) {
try { while (S.frameRun === run) {
const r = await api(base() + "api/screen/" + S.screen, {raw: true}); if (document.hidden) { await nap(800); continue; }
if (!r.ok) { const v = await r.json().catch(() => ({})); $("stage").innerHTML = `<div class="msg">${esc(v.error || "看不到畫面")}</div>`; $("screen-live").textContent = ""; } try {
else { const r = await api(base() + "api/screen/mac", {raw: true});
const sz = (r.headers.get("X-Size") || "0x0").split("x").map(Number); S.size = sz; if (S.frameRun !== run) return;
if (!r.ok) { const v = await r.json().catch(() => ({})); $("stage").innerHTML = `<div class="msg">${esc(v.error || "看不到畫面")}</div>`; $("screen-live").textContent = ""; await nap(1500); continue; }
S.size = (r.headers.get("X-Size") || "0x0").split("x").map(Number);
const url = URL.createObjectURL(await r.blob()); const url = URL.createObjectURL(await r.blob());
let img = $("stage").querySelector("img"); let img = $("stage").querySelector("img");
if (!img) { $("stage").innerHTML = ""; img = document.createElement("img"); img.draggable = false; $("stage").appendChild(img); bindStage(img); } if (!img) { $("stage").innerHTML = ""; img = document.createElement("img"); img.draggable = false; $("stage").appendChild(img); bindStage(img); }
img.src = url; if (lastUrl) URL.revokeObjectURL(lastUrl); lastUrl = url; img.src = url; if (lastUrl) URL.revokeObjectURL(lastUrl); lastUrl = url;
$("screen-live").innerHTML = `<span class="live"></span>即時`; $("screen-live").innerHTML = `<span class="live"></span>即時`;
} await nap(700);
} catch (e) { $("screen-live").textContent = ""; } } catch (e) { $("screen-live").textContent = ""; await nap(1500); }
busyFrame = false; }
} }
function point(img, e) { function point(img, e) {
const r = img.getBoundingClientRect(); const r = img.getBoundingClientRect();
@ -777,7 +785,7 @@ function act(img, e, button) {
input2({kind: "click", x: p.x, y: p.y, button}); input2({kind: "click", x: p.x, y: p.y, button});
} }
async function input2(body) { async function input2(body) {
try { await api(base() + "api/screen/" + S.screen, {json: body}); setTimeout(frame, 250); } catch (e) { toast(e.message); } try { await api(base() + "api/screen/mac", {json: body}); } catch (e) { toast(e.message); }
} }
document.querySelectorAll("[data-key]").forEach((b) => b.onclick = () => input2({kind: "key", key: b.dataset.key})); document.querySelectorAll("[data-key]").forEach((b) => b.onclick = () => input2({kind: "key", key: b.dataset.key}));
document.querySelectorAll("[data-scroll]").forEach((b) => b.onclick = () => input2({kind: "scroll", x: Math.round(S.size[0] / 2), y: Math.round(S.size[1] / 2), direction: b.dataset.scroll, amount: 5})); document.querySelectorAll("[data-scroll]").forEach((b) => b.onclick = () => input2({kind: "scroll", x: Math.round(S.size[0] / 2), y: Math.round(S.size[1] / 2), direction: b.dataset.scroll, amount: 5}));
@ -785,9 +793,8 @@ $("k-type").onclick = () => { $("typebar").classList.toggle("on"); if ($("typeba
$("b-type").onclick = () => { const t = $("typein").value; if (!t) return; $("typein").value = ""; input2({kind: "type", text: t}); }; $("b-type").onclick = () => { const t = $("typein").value; if (!t) return; $("typein").value = ""; input2({kind: "type", text: t}); };
$("typein").addEventListener("keydown", (e) => { if (e.key === "Enter" && !e.isComposing) { e.preventDefault(); $("b-type").click(); } }); $("typein").addEventListener("keydown", (e) => { if (e.key === "Enter" && !e.isComposing) { e.preventDefault(); $("b-type").click(); } });
$("k-right").onclick = () => { S.right = !S.right; $("k-right").textContent = "右鍵:" + (S.right ? "開" : "關"); }; $("k-right").onclick = () => { S.right = !S.right; $("k-right").textContent = "右鍵:" + (S.right ? "開" : "關"); };
$("k-wake").onclick = () => { toast("正在叫醒" + ownComputer() + "…"); input2({kind: "wake"}); };
// ---------- menu, notifications ---------- // ---------- menu, notifications ----------
$("b-more").onclick = () => { $("m-host").style.display = S.relay ? "" : "none"; $("m-pair").style.display = S.relay ? "" : "none"; notifyLabel(); $("sheet").classList.add("on"); }; $("b-more").onclick = () => { $("m-mac").style.display = S.mac ? "" : "none"; $("m-host").style.display = S.relay ? "" : "none"; $("m-pair").style.display = S.relay ? "" : "none"; notifyLabel(); $("sheet").classList.add("on"); };
$("sheet").onclick = (e) => { if (e.target.id === "sheet") $("sheet").classList.remove("on"); }; $("sheet").onclick = (e) => { if (e.target.id === "sheet") $("sheet").classList.remove("on"); };
$("m-cancel").onclick = () => $("sheet").classList.remove("on"); $("m-cancel").onclick = () => $("sheet").classList.remove("on");
$("m-out").onclick = () => { $("sheet").classList.remove("on"); if (confirm("要登出這支手機嗎?")) signOut(); }; $("m-out").onclick = () => { $("sheet").classList.remove("on"); if (confirm("要登出這支手機嗎?")) signOut(); };
@ -861,7 +868,7 @@ if ("serviceWorker" in navigator) navigator.serviceWorker.addEventListener("mess
if (d.host && d.host !== S.host && S.relay) { S.host = d.host; localStorage.setItem("anybot.host", S.host); S.version = 0; } if (d.host && d.host !== S.host && S.relay) { S.host = d.host; localStorage.setItem("anybot.host", S.host); S.version = 0; }
if (d.bot) openChat(+d.bot); if (d.bot) openChat(+d.bot);
}); });
document.addEventListener("visibilitychange", () => { if (!document.hidden && S.view === "screen") frame(); }); document.addEventListener("visibilitychange", () => { if (!document.hidden && S.view === "screen") startFrames(); });
hello(); hello();
</script> </script>
</body> </body>

8
mobile/novnc.js Normal file

File diff suppressed because one or more lines are too long

109
mobile/screen.html Normal file
View File

@ -0,0 +1,109 @@
<!doctype html>
<html lang="zh-Hant">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover, user-scalable=no">
<meta name="theme-color" content="#16181d">
<title>Bot 的電腦</title>
<link rel="icon" href="icon.svg">
<style>
:root { --bg: #16181d; --bar: #1f2228; --line: #2e323a; --text: #eceef2; --dim: #9aa1ad; --accent: #4f8cff; --warn: #e5a13a; --ok: #3fb67a; }
* { box-sizing: border-box; }
html, body { margin: 0; height: 100%; background: var(--bg); color: var(--text); font: 15px/1.4 -apple-system, BlinkMacSystemFont, "PingFang TC", "Noto Sans TC", sans-serif; overflow: hidden; }
body { display: flex; flex-direction: column; height: 100dvh; }
header { display: flex; align-items: center; gap: 8px; padding: 8px 12px; padding-top: max(8px, env(safe-area-inset-top)); background: var(--bar); border-bottom: 1px solid var(--line); }
header .name { font-weight: 600; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }
header .state { color: var(--dim); font-size: 13px; white-space: nowrap; display: flex; align-items: center; gap: 6px; }
.dot { width: 8px; height: 8px; border-radius: 50%; background: var(--dim); flex: none; }
.dot.live { background: var(--ok); }
.dot.hold { background: var(--warn); }
.grow { flex: 1; min-width: 0; display: flex; flex-direction: column; }
button { font: inherit; color: var(--text); background: #2b2f37; border: 1px solid var(--line); border-radius: 8px; padding: 6px 12px; cursor: pointer; white-space: nowrap; }
button.main { background: var(--accent); border-color: var(--accent); color: #fff; font-weight: 600; }
button.icon { padding: 6px 10px; }
#screen { flex: 1; min-height: 0; position: relative; background: #000; }
#screen > div { background: #000 !important; }
#msg { position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; text-align: center; padding: 24px; color: var(--dim); pointer-events: none; }
#type { display: none; gap: 6px; padding: 8px 12px; padding-bottom: max(8px, env(safe-area-inset-bottom)); background: var(--bar); border-top: 1px solid var(--line); flex-wrap: wrap; }
#type.on { display: flex; }
#type input { flex: 1 1 160px; min-width: 0; font: inherit; color: var(--text); background: #111318; border: 1px solid var(--line); border-radius: 8px; padding: 7px 10px; }
@media (max-width: 420px) { header .hint { display: none; } }
</style>
</head>
<body>
<header>
<button class="icon" id="back" title="返回">←</button>
<div class="grow"><div class="name" id="name">Bot 的電腦</div><div class="state"><span class="dot" id="dot"></span><span id="state">連線中…</span></div></div>
<button id="kb" title="打字(任何語言)">打字</button>
<button class="main" id="take">接管</button>
</header>
<div id="screen"><div id="msg">連線中…</div></div>
<div id="type">
<input id="text" placeholder="要打的字(中文也可以),按送出" autocomplete="off" enterkeyhint="send">
<button class="main" id="send">送出</button>
<button data-k="Return">Enter</button><button data-k="Escape">Esc</button><button data-k="Tab">Tab</button><button data-k="BackSpace">⌫</button>
</div>
<script type="module">
import RFB from "./novnc.js";
// #host=…&name=…&ticket=… (the app's link) or #host=… with this site's sign-in (the phone page);
// #lan=1 on an AnyBot's own phone page. &control=1 starts with control (a bot asked for help).
const q = new URLSearchParams(location.hash.slice(1));
const name = q.get("name") || "Bot 的電腦";
const token = localStorage.getItem("anybot.token") || "";
const wsBase = (location.protocol === "https:" ? "wss://" : "ws://") + location.host;
const url = q.get("lan") === "1"
? `${wsBase}/api/vnc?t=${encodeURIComponent(token)}`
: `${wsBase}/relay/vnc/${encodeURIComponent(q.get("host") || "")}?` + (q.get("ticket") ? "ticket=" + encodeURIComponent(q.get("ticket")) : "t=" + encodeURIComponent(token));
const $ = (id) => document.getElementById(id);
document.title = name;
$("name").textContent = name;
let rfb = null, ws = null, control = q.get("control") === "1", tries = 0, connected = false;
const KEYS = { Return: 0xff0d, Escape: 0xff1b, Tab: 0xff09, BackSpace: 0xff08 };
function state(text, kind) { $("state").textContent = text; $("dot").className = "dot" + (kind ? " " + kind : ""); }
function message(text) { $("msg").textContent = text; $("msg").style.display = text ? "" : "none"; }
function showControl() {
$("take").textContent = control ? "交還給 Bot" : "接管";
$("take").className = control ? "" : "main";
$("kb").style.display = control ? "" : "none";
if (!control) $("type").classList.remove("on");
if (rfb) { rfb.viewOnly = !control; if (control) rfb.focus(); }
if (connected) state(control ? "你在操作(Bot 會等你停下)" : "即時觀看", control ? "hold" : "live");
}
function connect() {
message("連線中…"); state("連線中…");
ws = new WebSocket(url);
rfb = new RFB($("screen"), ws, { shared: true });
rfb.scaleViewport = true;
rfb.resizeSession = false;
rfb.qualityLevel = 6;
rfb.compressionLevel = 2;
rfb.showDotCursor = true;
rfb.viewOnly = !control;
rfb.addEventListener("connect", () => { tries = 0; connected = true; message(""); showControl(); });
rfb.addEventListener("disconnect", () => {
rfb = null; connected = false;
tries += 1;
state("連線中斷,正在重新連線…");
message(tries > 3 ? "連不上這台電腦。它的 AnyBot 可能關了,或連結已過期:回到 AnyBot 再打開一次。" : "連線中斷,正在重新連線…");
setTimeout(connect, Math.min(1000 * tries, 8000));
});
}
$("take").onclick = () => { control = !control; showControl(); };
$("kb").onclick = () => { $("type").classList.toggle("on"); if ($("type").classList.contains("on")) $("text").focus(); };
// Typed text goes over the same connection as words (any language), pasted in the bots' computer.
function send() { const t = $("text").value; if (!t || !ws || ws.readyState !== 1) return; ws.send(t); $("text").value = ""; }
$("send").onclick = send;
$("text").addEventListener("keydown", (e) => { if (e.key === "Enter" && !e.isComposing) { e.preventDefault(); send(); } });
document.querySelectorAll("[data-k]").forEach((b) => b.onclick = () => { if (rfb) rfb.sendKey(KEYS[b.dataset.k], b.dataset.k); });
$("back").onclick = () => { if (history.length > 1) history.back(); else window.close(); };
showControl();
connect();
</script>
</body>
</html>

View File

@ -29,6 +29,9 @@ use std::sync::{Arc, Condvar, Mutex};
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
const PAGE: &str = include_str!("../../mobile/index.html"); const PAGE: &str = include_str!("../../mobile/index.html");
/// The bots' computer, watched and used in a browser (noVNC).
const SCREEN_PAGE: &str = include_str!("../../mobile/screen.html");
const NOVNC: &str = include_str!("../../mobile/novnc.js");
const MANIFEST: &str = r##"{"name":"AnyBot","short_name":"AnyBot","start_url":"./","display":"standalone","background_color":"#121212","theme_color":"#121212","icons":[{"src":"icon.svg","sizes":"any","type":"image/svg+xml"},{"src":"icon-192.png","sizes":"192x192","type":"image/png","purpose":"any maskable"},{"src":"icon-512.png","sizes":"512x512","type":"image/png","purpose":"any maskable"}]}"##; const MANIFEST: &str = r##"{"name":"AnyBot","short_name":"AnyBot","start_url":"./","display":"standalone","background_color":"#121212","theme_color":"#121212","icons":[{"src":"icon.svg","sizes":"any","type":"image/svg+xml"},{"src":"icon-192.png","sizes":"192x192","type":"image/png","purpose":"any maskable"},{"src":"icon-512.png","sizes":"512x512","type":"image/png","purpose":"any maskable"}]}"##;
/// AnyBot's icon (the same drawing as the app's), and PNGs for the iPhone home screen and Android. /// AnyBot's icon (the same drawing as the app's), and PNGs for the iPhone home screen and Android.
const ICON: &str = include_str!("../../mobile/icon.svg"); const ICON: &str = include_str!("../../mobile/icon.svg");
@ -200,6 +203,10 @@ struct Relay {
/// The brain each AnyBot shows (when it is a screen of another), and which lend their hands. /// The brain each AnyBot shows (when it is a screen of another), and which lend their hands.
uses: Mutex<HashMap<String, String>>, uses: Mutex<HashMap<String, String>>,
hands: Mutex<std::collections::HashSet<String>>, hands: Mutex<std::collections::HashSet<String>>,
/// People watching a bots' computer: stream id → (where the host's side goes, that host).
streams: Mutex<HashMap<String, (Sender<Vec<u8>>, String)>>,
/// One-time links for a browser to watch a host's computer: ticket → (account, host, made).
tickets: Mutex<HashMap<String, (String, String, Instant)>>,
/// 配對碼: code → (account, the AnyBot it is for, when made). Ten minutes, once. /// 配對碼: code → (account, the AnyBot it is for, when made). Ten minutes, once.
pairs: Mutex<HashMap<String, (String, String, Instant)>>, pairs: Mutex<HashMap<String, (String, String, Instant)>>,
/// When each host last asked (in memory; saved now and then). /// When each host last asked (in memory; saved now and then).
@ -210,6 +217,9 @@ struct Relay {
contact: String, contact: String,
} }
/// How long a link the app opens for a browser works (reconnects included).
const TICKET_LIFE: Duration = Duration::from_secs(600);
fn now() -> i64 { fn now() -> i64 {
chrono::Utc::now().timestamp() chrono::Utc::now().timestamp()
} }
@ -229,6 +239,8 @@ impl Relay {
flights: Mutex::new(HashMap::new()), flights: Mutex::new(HashMap::new()),
uses: Mutex::new(HashMap::new()), uses: Mutex::new(HashMap::new()),
hands: Mutex::new(std::collections::HashSet::new()), hands: Mutex::new(std::collections::HashSet::new()),
streams: Mutex::new(HashMap::new()),
tickets: Mutex::new(HashMap::new()),
pairs: Mutex::new(HashMap::new()), pairs: Mutex::new(HashMap::new()),
seen: Mutex::new(HashMap::new()), seen: Mutex::new(HashMap::new()),
fails: Mutex::new(HashMap::new()), fails: Mutex::new(HashMap::new()),
@ -294,6 +306,88 @@ impl Relay {
e.1 = Instant::now(); e.1 = Instant::now();
} }
/// A person's browser watching (and maybe using) an AnyBot's computer: its VNC stream goes
/// to that AnyBot over its lasting connection, and back. Signed in by the account session
/// (?t=, the phone page) or a ticket the account's app asked for (?ticket=); owners and
/// admins only.
fn vnc_viewer(self: &Arc<Self>, req: Req, mut c: std::net::TcpStream, host: &str) {
use tungstenite::{protocol::Role, Message, WebSocket};
let user = match req.query.get("ticket").filter(|t| !t.is_empty()) {
Some(t) => self.tickets.lock().unwrap().get(t).filter(|(_, h, made)| h == host && made.elapsed() < TICKET_LIFE).map(|(u, _, _)| u.clone()),
None => self.session(&req),
};
let Some(user) = user else { return httpd::write_response(&mut c, &Resp::error(401, "請先登入")) };
let allowed = self.data.lock().unwrap().hosts.iter().any(|x| x.id == host && matches!(x.role_of(&user).as_deref(), Some("owner") | Some("admin")));
if !allowed {
return httpd::write_response(&mut c, &Resp::error(404, "找不到這台 AnyBot"));
}
let Some(to_host) = self.sockets.lock().unwrap().get(host).map(|s| s.tx.clone()) else {
return httpd::write_response(&mut c, &Resp::error(503, "這台 AnyBot 現在沒有連線"));
};
if !httpd::accept_websocket(&req, &mut c) {
return;
}
let mut ws = WebSocket::from_raw_socket(c, Role::Server, None);
let sid = httpd::random_hex(8);
let (tx, rx) = channel::<Vec<u8>>();
self.streams.lock().unwrap().insert(sid.clone(), (tx, host.to_string()));
if to_host.send(wire::pack(&json!({"t": "vopen", "sid": sid}), &[])).is_err() {
self.streams.lock().unwrap().remove(&sid);
return;
}
let (mut heard, mut pinged) = (Instant::now(), Instant::now());
let mut pace = wire::Pace::new();
'live: loop {
pace.tune(ws.get_ref());
let up = match ws.read() {
Ok(Message::Binary(b)) => Some(("vdata", b.to_vec())),
Ok(Message::Text(t)) => Some(("vtext", t.as_bytes().to_vec())),
Ok(Message::Close(_)) => break,
Ok(_) => None,
Err(tungstenite::Error::Io(e)) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => None,
Err(_) => break,
};
if let Some((t, b)) = up {
heard = Instant::now();
pace.moved();
if to_host.send(wire::pack(&json!({"t": t, "sid": sid}), &b)).is_err() {
break;
}
}
loop {
match rx.try_recv() {
Ok(b) => {
pace.moved();
if ws.write(Message::Binary(b.into())).is_err() {
break 'live;
}
}
Err(std::sync::mpsc::TryRecvError::Empty) => break,
// The host closed it (or went away).
Err(std::sync::mpsc::TryRecvError::Disconnected) => break 'live,
}
}
if pinged.elapsed().as_secs() >= WS_PING_SECS {
pinged = Instant::now();
if ws.write(Message::Ping(Vec::new().into())).is_err() {
break;
}
}
match ws.flush() {
Err(tungstenite::Error::Io(e)) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => {}
Err(_) => break,
Ok(()) => {}
}
if heard.elapsed().as_secs() >= WS_DEAD_SECS {
break;
}
}
let _ = ws.close(None);
let _ = ws.flush();
self.streams.lock().unwrap().remove(&sid);
let _ = to_host.send(wire::pack(&json!({"t": "vclose", "sid": sid}), &[]));
}
/// Hand a request to a host and wait for its answer. /// Hand a request to a host and wait for its answer.
fn forward(&self, host_id: &str, method: &str, path: &str, query: &HashMap<String, String>, body: &[u8], who: (&str, &str)) -> Resp { fn forward(&self, host_id: &str, method: &str, path: &str, query: &HashMap<String, String>, body: &[u8], who: (&str, &str)) -> Resp {
self.forward_raw(host_id, method, path, query, body, false, who, false, "").into_resp() self.forward_raw(host_id, method, path, query, body, false, who, false, "").into_resp()
@ -356,22 +450,18 @@ impl Relay {
/// 長連線: an AnyBot keeps one WebSocket open; requests for it come down it at once, and its /// 長連線: an AnyBot keeps one WebSocket open; requests for it come down it at once, and its
/// own requests to the account's other AnyBots go up it. /// own requests to the account's other AnyBots go up it.
fn ws(self: &Arc<Self>, req: Req, mut c: std::net::TcpStream) { fn ws(self: &Arc<Self>, req: Req, mut c: std::net::TcpStream) {
use std::io::Write;
use tungstenite::{protocol::Role, Message, WebSocket}; use tungstenite::{protocol::Role, Message, WebSocket};
if let Some(host) = req.path.strip_prefix("/relay/vnc/") {
let host = host.to_string();
return self.vnc_viewer(req, c, &host);
}
if req.path != "/relay/host/ws" { if req.path != "/relay/host/ws" {
return httpd::write_response(&mut c, &Resp::error(404, "沒有這個網址")); return httpd::write_response(&mut c, &Resp::error(404, "沒有這個網址"));
} }
let Some(h) = self.host(&req) else { return httpd::write_response(&mut c, &Resp::error(401, "這台電腦的金鑰無效")) }; let Some(h) = self.host(&req) else { return httpd::write_response(&mut c, &Resp::error(401, "這台電腦的金鑰無效")) };
let key = req.header("sec-websocket-key").to_string(); if !httpd::accept_websocket(&req, &mut c) {
if key.is_empty() {
return httpd::write_response(&mut c, &Resp::error(400, "bad request"));
}
let accept = tungstenite::handshake::derive_accept_key(key.as_bytes());
if c.write_all(format!("HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: {accept}\r\n\r\n").as_bytes()).is_err() {
return; return;
} }
let _ = c.set_read_timeout(Some(Duration::from_millis(40)));
let _ = c.set_nodelay(true);
let cfg = tungstenite::protocol::WebSocketConfig::default().max_message_size(Some(httpd::MAX_BODY * 2)).max_frame_size(Some(httpd::MAX_BODY * 2)); let cfg = tungstenite::protocol::WebSocketConfig::default().max_message_size(Some(httpd::MAX_BODY * 2)).max_frame_size(Some(httpd::MAX_BODY * 2));
let mut ws = WebSocket::from_raw_socket(c, Role::Server, Some(cfg)); let mut ws = WebSocket::from_raw_socket(c, Role::Server, Some(cfg));
let serial = SERIAL.fetch_add(1, Ordering::SeqCst); let serial = SERIAL.fetch_add(1, Ordering::SeqCst);
@ -379,10 +469,13 @@ impl Relay {
self.sockets.lock().unwrap().insert(h.id.clone(), Sock { tx: tx.clone(), serial }); self.sockets.lock().unwrap().insert(h.id.clone(), Sock { tx: tx.clone(), serial });
self.seen.lock().unwrap().insert(h.id.clone(), Instant::now()); self.seen.lock().unwrap().insert(h.id.clone(), Instant::now());
let (mut heard, mut pinged, mut marked) = (Instant::now(), Instant::now(), Instant::now()); let (mut heard, mut pinged, mut marked) = (Instant::now(), Instant::now(), Instant::now());
let mut pace = wire::Pace::new();
'live: loop { 'live: loop {
pace.tune(ws.get_ref());
match ws.read() { match ws.read() {
Ok(Message::Binary(b)) => { Ok(Message::Binary(b)) => {
heard = Instant::now(); heard = Instant::now();
pace.moved();
self.on_frame(&h, &b, &tx); self.on_frame(&h, &b, &tx);
} }
Ok(Message::Close(_)) => break, Ok(Message::Close(_)) => break,
@ -391,6 +484,7 @@ impl Relay {
Err(_) => break, Err(_) => break,
} }
while let Ok(f) = rx.try_recv() { while let Ok(f) = rx.try_recv() {
pace.moved();
if ws.write(Message::Binary(f.into())).is_err() { if ws.write(Message::Binary(f.into())).is_err() {
break 'live; break 'live;
} }
@ -414,7 +508,8 @@ impl Relay {
self.seen.lock().unwrap().insert(h.id.clone(), Instant::now()); self.seen.lock().unwrap().insert(h.id.clone(), Instant::now());
} }
} }
// Gone: whoever waits on it hears so now, not after a timeout. // Gone: its viewers' streams end, and whoever waits on it hears so now, not after a timeout.
self.streams.lock().unwrap().retain(|_, (_, host)| *host != h.id);
{ {
let mut socks = self.sockets.lock().unwrap(); let mut socks = self.sockets.lock().unwrap();
if socks.get(&h.id).map(|s| s.serial) == Some(serial) { if socks.get(&h.id).map(|s| s.serial) == Some(serial) {
@ -434,6 +529,20 @@ impl Relay {
let Some((head, body)) = wire::unpack(b) else { return }; let Some((head, body)) = wire::unpack(b) else { return };
let s = |k: &str| head[k].as_str().unwrap_or("").to_string(); let s = |k: &str| head[k].as_str().unwrap_or("").to_string();
match head["t"].as_str().unwrap_or("") { match head["t"].as_str().unwrap_or("") {
// A viewer's stream: only the host it was opened on may write to it.
"vdata" => {
if let Some((v, host)) = self.streams.lock().unwrap().get(&s("sid")) {
if *host == h.id {
let _ = v.send(body);
}
}
}
"vclose" => {
let mut all = self.streams.lock().unwrap();
if all.get(&s("sid")).map(|(_, host)| *host == h.id).unwrap_or(false) {
all.remove(&s("sid"));
}
}
"reply" => { "reply" => {
// Only the host the job went to may answer it. // Only the host the job went to may answer it.
let w = { let w = {
@ -469,7 +578,9 @@ impl Relay {
let (me, tx, user, method, z, from) = (self.clone(), tx.clone(), h.user.clone(), s("method"), head["z"].as_bool().unwrap_or(false), h.id.clone()); let (me, tx, user, method, z, from) = (self.clone(), tx.clone(), h.user.clone(), s("method"), head["z"].as_bool().unwrap_or(false), h.id.clone());
std::thread::spawn(move || { std::thread::spawn(move || {
let a = me.forward_raw(&to, &method, &path, &query, &body, z, (&user, role), true, &from); let a = me.forward_raw(&to, &method, &path, &query, &body, z, (&user, role), true, &from);
let _ = tx.send(wire::pack(&json!({"t": "ret", "id": id, "status": a.status, "ctype": a.ctype, "z": a.z}), &a.body)); // The answer's headers too (a live picture's number, a screen's size).
let headers: Vec<[&str; 2]> = a.headers.iter().map(|(k, v)| [k.as_str(), v.as_str()]).collect();
let _ = tx.send(wire::pack(&json!({"t": "ret", "id": id, "status": a.status, "ctype": a.ctype, "z": a.z, "headers": headers}), &a.body));
}); });
} }
"hello" => { "hello" => {
@ -497,6 +608,25 @@ impl Relay {
("GET", []) | ("GET", ["index.html"]) => Resp::bytes("text/html; charset=utf-8", PAGE.as_bytes().to_vec()) ("GET", []) | ("GET", ["index.html"]) => Resp::bytes("text/html; charset=utf-8", PAGE.as_bytes().to_vec())
.with_header("Content-Security-Policy", "default-src 'self'; img-src 'self' blob: data:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'; connect-src 'self'") .with_header("Content-Security-Policy", "default-src 'self'; img-src 'self' blob: data:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'; connect-src 'self'")
.with_header("Strict-Transport-Security", "max-age=31536000"), .with_header("Strict-Transport-Security", "max-age=31536000"),
("GET", ["screen"]) => Resp::bytes("text/html; charset=utf-8", SCREEN_PAGE.as_bytes().to_vec())
.with_header("Content-Security-Policy", "default-src 'self'; img-src 'self' blob: data:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'; connect-src 'self' wss: ws:")
.with_header("Strict-Transport-Security", "max-age=31536000"),
("GET", ["novnc.js"]) => Resp::bytes("text/javascript; charset=utf-8", NOVNC.as_bytes().to_vec()).with_header("Cache-Control", "public, max-age=86400"),
// A link for this account's browser to watch one of its AnyBots' computer (the app
// opens it; the browser has no session of its own).
("POST", ["relay", "screen-ticket"]) => {
let Some(h) = self.host(&req) else { return Resp::error(401, "這台電腦的金鑰無效") };
let want = req.json()["host"].as_str().unwrap_or("").to_string();
let ok = self.data.lock().unwrap().hosts.iter().any(|x| x.id == want && matches!(x.role_of(&h.user).as_deref(), Some("owner") | Some("admin")));
if !ok {
return Resp::error(404, "找不到這台 AnyBot");
}
let t = httpd::random_hex(24);
let mut all = self.tickets.lock().unwrap();
all.retain(|_, (_, _, made)| made.elapsed() < TICKET_LIFE);
all.insert(t.clone(), (h.user.clone(), want, Instant::now()));
Resp::json(&json!({"ticket": t}))
}
("GET", ["manifest.webmanifest"]) => Resp::bytes("application/manifest+json", MANIFEST.as_bytes().to_vec()), ("GET", ["manifest.webmanifest"]) => Resp::bytes("application/manifest+json", MANIFEST.as_bytes().to_vec()),
("GET", ["icon.svg"]) => Resp::bytes("image/svg+xml", ICON.as_bytes().to_vec()), ("GET", ["icon.svg"]) => Resp::bytes("image/svg+xml", ICON.as_bytes().to_vec()),
("GET", [name]) if ICON_PNGS.iter().any(|(n, _)| n == name) => Resp::bytes("image/png", ICON_PNGS.iter().find(|(n, _)| n == name).map(|(_, b)| b.to_vec()).unwrap_or_default()), ("GET", [name]) if ICON_PNGS.iter().any(|(n, _)| n == name) => Resp::bytes("image/png", ICON_PNGS.iter().find(|(n, _)| n == name).map(|(_, b)| b.to_vec()).unwrap_or_default()),

View File

@ -371,13 +371,6 @@ impl App {
app.state.selected = 2; app.state.selected = 2;
app.show_routines = Some(2); app.show_routines = Some(2);
} }
Ok("viewer") => {
crate::cloudbox::start_background();
let _ = crate::cloudbox::ensure_running();
app.cu.demo_live = true;
app.cu.viewer_open = true;
app.cu.fit = true;
}
Ok("attach") => { Ok("attach") => {
app.state.selected = 1; app.state.selected = 1;
let dir = std::env::temp_dir().join("anybot-shot-attach"); let dir = std::env::temp_dir().join("anybot-shot-attach");
@ -511,7 +504,6 @@ impl App {
// The real box, so the 電腦 card shows its screen. // The real box, so the 電腦 card shows its screen.
crate::cloudbox::start_background(); crate::cloudbox::start_background();
let _ = crate::cloudbox::ensure_running(); let _ = crate::cloudbox::ensure_running();
app.cu.demo_live = true;
app.state.selected = 1; app.state.selected = 1;
app.panel = Some(panel_ui::PanelTab::Activity); app.panel = Some(panel_ui::PanelTab::Activity);
} }
@ -3427,11 +3419,6 @@ impl App {
ctx.send_viewport_cmd(egui::ViewportCommand::InnerSize(vec2(w, h))); ctx.send_viewport_cmd(egui::ViewportCommand::InnerSize(vec2(w, h)));
} }
} }
// Real data: the computer viewer opens a little after start, as a person would open it (a
// screen of a brain has said so by then, so it shows the brain's computer).
if self.frames == 10 && !self.demo && std::env::var("ANYBOT_SHOT_VIEW").as_deref() == Ok("viewer") {
self.open_viewer();
}
// ANYBOT_SHOT_PAGE=id: open that page late (a screen gets its pages from the brain first). // ANYBOT_SHOT_PAGE=id: open that page late (a screen gets its pages from the brain first).
if self.frames == need * 3 / 4 { if self.frames == need * 3 / 4 {
if let Some(id) = std::env::var("ANYBOT_SHOT_PAGE").ok().and_then(|v| v.parse().ok()) { if let Some(id) = std::env::var("ANYBOT_SHOT_PAGE").ok().and_then(|v| v.parse().ok()) {
@ -3767,7 +3754,6 @@ impl eframe::App for App {
self.delete_dialog(ctx); self.delete_dialog(ctx);
self.routines_window(ctx); self.routines_window(ctx);
self.rebuild_dialog(ctx); self.rebuild_dialog(ctx);
self.viewer_window(ctx);
self.local_control_bar(ctx); self.local_control_bar(ctx);
self.teach_bar(ctx); self.teach_bar(ctx);
self.approval_popup(ctx); self.approval_popup(ctx);
@ -4146,12 +4132,6 @@ fn icon_button(ui: &mut egui::Ui, icon: &str, tip: &str, p: Palette) -> egui::Re
ui.add(egui::Button::new(RichText::new(icon).size(15.0).color(p.text2)).frame(false).min_size(vec2(26.0, 26.0))).on_hover_text(t(tip)) ui.add(egui::Button::new(RichText::new(icon).size(15.0).color(p.text2)).frame(false).min_size(vec2(26.0, 26.0))).on_hover_text(t(tip))
} }
/// A header action: text only, a soft fill on hover.
#[allow(dead_code)]
fn quiet_button(ui: &mut egui::Ui, text: &str, p: Palette) -> egui::Response {
ui.add(egui::Button::new(RichText::new(t(text)).size(FONT_SM).color(p.text2)).frame_when_inactive(false).corner_radius(8).min_size(vec2(0.0, 28.0)))
}
fn small_button(ui: &mut egui::Ui, text: &str, p: Palette) -> egui::Response { fn small_button(ui: &mut egui::Ui, text: &str, p: Palette) -> egui::Response {
ui.add(egui::Button::new(RichText::new(t(text)).size(11.5).color(p.text3)).frame_when_inactive(false).corner_radius(6).min_size(vec2(0.0, 22.0))) ui.add(egui::Button::new(RichText::new(t(text)).size(11.5).color(p.text3)).frame_when_inactive(false).corner_radius(6).min_size(vec2(0.0, 22.0)))
} }

View File

@ -7,10 +7,6 @@
use super::*; use super::*;
use crate::cloudbox::{self, Rebuild, Run}; use crate::cloudbox::{self, Rebuild, Run};
/// How often the viewer asks the box for a new picture (estimated).
const VIEWER_FRAME_MS: u64 = 450;
/// The brain's computer comes over the account server: a picture about every 0.7 s.
const REMOTE_FRAME_MS: u64 = 700;
/// The floating bar stays this long after the bot's last mouse/keyboard action on this Mac. /// The floating bar stays this long after the bot's last mouse/keyboard action on this Mac.
const LOCAL_BAR_SECS: u64 = 8; const LOCAL_BAR_SECS: u64 = 8;
@ -31,16 +27,8 @@ pub struct ComputerUi {
pub confirm_reset: bool, pub confirm_reset: bool,
/// 一律允許的應用程式 is expanded. /// 一律允許的應用程式 is expanded.
pub apps_open: bool, pub apps_open: bool,
pub viewer_open: bool, /// 接管 during a hand-off: this app told the bots' computer to let the person have it.
/// 取得控制權: the person's clicks and keys go to the box.
pub control: bool, pub control: bool,
pub fit: bool,
texture: Option<egui::TextureHandle>,
frame: Arc<Mutex<Option<(u64, egui::ColorImage)>>>,
shown_seq: u64,
frame_size: [usize; 2],
worker: Option<Arc<AtomicBool>>,
input_tx: Option<Sender<Input>>,
pub confirm: Option<Rebuild>, pub confirm: Option<Rebuild>,
pub note: Arc<Mutex<String>>, pub note: Arc<Mutex<String>>,
disk: Arc<Mutex<Option<String>>>, disk: Arc<Mutex<Option<String>>>,
@ -52,8 +40,6 @@ pub struct ComputerUi {
teach_name: String, teach_name: String,
teach_for: Option<u64>, teach_for: Option<u64>,
teach_error: String, teach_error: String,
/// Screenshot mode: show the real box in the viewer.
pub demo_live: bool,
schedule_edit: String, schedule_edit: String,
pub update_ready: bool, pub update_ready: bool,
update_checked: Option<Instant>, update_checked: Option<Instant>,
@ -63,16 +49,6 @@ pub struct ComputerUi {
thumb_img: Arc<Mutex<Option<egui::ColorImage>>>, thumb_img: Arc<Mutex<Option<egui::ColorImage>>>,
thumb_busy: Arc<AtomicBool>, thumb_busy: Arc<AtomicBool>,
thumb_at: Option<Instant>, thumb_at: Option<Instant>,
/// The viewer shows the brain's computer (this window is a screen of a brain): how to reach it.
remote: Option<crate::remote::Settings>,
}
enum Input {
Click(i64, i64, &'static str, u32),
Move(i64, i64),
Scroll(i64, i64, bool, u32),
Key(String),
Text(String),
} }
fn status_color(r: &Run, p: Palette) -> Color32 { fn status_color(r: &Run, p: Palette) -> Color32 {
@ -104,16 +80,6 @@ fn status_pill(ui: &mut egui::Ui, r: &Run, p: Palette) {
} }
/// A little monitor drawn with lines (no emoji font needed). /// A little monitor drawn with lines (no emoji font needed).
fn monitor_icon(ui: &mut egui::Ui, size: f32, col: Color32) {
let (rect, _) = ui.allocate_exact_size(vec2(size, size), egui::Sense::hover());
let s = size / 20.0;
let screen = egui::Rect::from_min_size(rect.min + vec2(2.0 * s, 3.0 * s), vec2(16.0 * s, 11.0 * s));
ui.painter().rect_stroke(screen, 2.5 * s, Stroke::new(1.5, col), egui::StrokeKind::Inside);
ui.painter().line_segment([rect.min + vec2(10.0 * s, 14.0 * s), rect.min + vec2(10.0 * s, 17.0 * s)], Stroke::new(1.5, col));
ui.painter().line_segment([rect.min + vec2(6.5 * s, 17.5 * s), rect.min + vec2(13.5 * s, 17.5 * s)], Stroke::new(1.5, col));
}
impl App { impl App {
/// Called once at start. /// Called once at start.
pub(super) fn computers_start(&mut self) { pub(super) fn computers_start(&mut self) {
@ -947,444 +913,66 @@ impl App {
self.cu.thumb.as_ref().map(|t| (t.id(), t.size_vec2())) self.cu.thumb.as_ref().map(|t| (t.id(), t.size_vec2()))
} }
/// 看畫面: the bots' computer in the browser (noVNC), live, where the person can also take
/// over. On a screen of a brain it is the brain's computer, through the account server (a
/// link made for this once); here it is this computer's own box.
pub(super) fn open_viewer(&mut self) { pub(super) fn open_viewer(&mut self) {
self.cu.viewer_open = true; self.open_screen(false);
self.cu.fit = true; }
fn open_screen(&mut self, control: bool) {
self.show_settings = false; self.show_settings = false;
// A screen of a brain watches the brain's computer (where its bots really work). let name = self.state.bots.iter().find(|b| b.id == self.state.selected).map(|b| format!("{} 的電腦", b.name)).unwrap_or_else(|| "Bot 的電腦".into());
if self.mir.on { if self.mir.on {
let s = self.state.remote.clone(); let s = self.state.remote.clone();
self.cu.remote = Some(s.clone()); self.box_task(move || {
if let Some(stop) = self.cu.worker.take() { let ticket = crate::remote::screen_ticket(&s)?;
stop.store(true, Ordering::SeqCst); let url = format!("{}/screen#host={}&ticket={ticket}&name={}{}", s.relay_url.trim_end_matches('/'), s.brain_id, crate::httpd::url_encode(&name), if control { "&control=1" } else { "" });
} crate::preview::open_with_system(&url);
self.cu.texture = None; Ok(String::new())
std::thread::spawn(move || {
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "viewer", "on": true}));
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "wake"}));
}); });
return; return;
} }
if self.cu.remote.take().is_some() { if self.demo {
if let Some(stop) = self.cu.worker.take() { return;
stop.store(true, Ordering::SeqCst);
}
self.cu.texture = None;
}
cloudbox::set_viewer_open(true);
if !cloudbox::run_state().is_running() && !self.demo {
self.box_task(|| cloudbox::ensure_running().map(|_| String::new()));
} }
self.box_task(move || {
let url = cloudbox::viewer_url(control)?;
crate::preview::open_with_system(&url);
Ok(String::new())
});
} }
/// 接管: open the computer and take control (the bot waits for the person anyway). /// 接管: open the computer with control (the bot waits for the person meanwhile).
pub(super) fn take_over_computer(&mut self) { pub(super) fn take_over_computer(&mut self) {
self.open_viewer(); self.open_screen(true);
if self.cu.control { if self.cu.control {
return; return;
} }
self.cu.control = true; self.cu.control = true;
match self.cu.remote.clone() { self.hold_computer(true);
Some(s) => {
std::thread::spawn(move || {
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "hold", "on": true}));
});
}
None => cloudbox::HELD_BY_PERSON.store(true, Ordering::SeqCst),
}
} }
/// 我完成了 / 跳過: the bot gets its computer back (its steps would otherwise wait for the /// 我完成了 / 跳過: the bot gets its computer back.
/// person to let go).
pub(super) fn give_back_computer(&mut self) { pub(super) fn give_back_computer(&mut self) {
if !self.cu.control { if !self.cu.control {
return; return;
} }
self.cu.control = false; self.cu.control = false;
match self.cu.remote.clone() { self.hold_computer(false);
Some(s) => {
std::thread::spawn(move || {
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "hold", "on": false}));
});
}
None => cloudbox::HELD_BY_PERSON.store(false, Ordering::SeqCst),
}
} }
fn close_viewer(&mut self) { fn hold_computer(&self, on: bool) {
self.cu.viewer_open = false; if self.mir.on {
self.cu.control = false; let s = self.state.remote.clone();
if let Some(s) = self.cu.remote.clone() {
std::thread::spawn(move || { std::thread::spawn(move || {
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "hold", "on": false})); let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "hold", "on": on}));
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "viewer", "on": false}));
}); });
} else { } else {
cloudbox::HELD_BY_PERSON.store(false, Ordering::SeqCst); cloudbox::HELD_BY_PERSON.store(on, Ordering::SeqCst);
cloudbox::set_viewer_open(false);
}
if let Some(stop) = self.cu.worker.take() {
stop.store(true, Ordering::SeqCst);
}
self.cu.input_tx = None;
}
fn ensure_viewer_threads(&mut self, ctx: &egui::Context) {
if self.cu.worker.is_some() || (self.demo && !self.cu.demo_live) {
return;
}
let stop = Arc::new(AtomicBool::new(false));
self.cu.worker = Some(stop.clone());
let slot = self.cu.frame.clone();
let c = ctx.clone();
let s2 = stop.clone();
let brain = self.cu.remote.clone();
if let Some(s) = brain.clone() {
// The brain's computer: pictures and the person's input go through the account server.
let s_frames = s.clone();
std::thread::spawn(move || {
let s = s_frames;
let mut seq = 0u64;
while !s2.load(Ordering::SeqCst) {
match crate::remote::sync_screen(&s, false).ok().and_then(|b| image::load_from_memory(&b).ok()) {
Some(img) => {
let img = img.to_rgba8();
seq += 1;
let size = [img.width() as usize, img.height() as usize];
*slot.lock().unwrap() = Some((seq, egui::ColorImage::from_rgba_unmultiplied(size, img.as_raw())));
c.request_repaint();
std::thread::sleep(std::time::Duration::from_millis(REMOTE_FRAME_MS));
}
None => {
std::thread::sleep(std::time::Duration::from_millis(1500));
c.request_repaint();
}
}
}
});
let (tx, rx) = channel::<Input>();
self.cu.input_tx = Some(tx);
std::thread::spawn(move || {
let mut last_move = Instant::now();
while let Ok(i) = rx.recv() {
let v = match i {
Input::Click(x, y, b, n) => serde_json::json!({"kind": "click", "x": x, "y": y, "button": b, "count": n}),
// Hovering sends many moves; a few a second are enough over the network.
Input::Move(x, y) => {
if last_move.elapsed().as_millis() < 250 {
continue;
}
last_move = Instant::now();
serde_json::json!({"kind": "move", "x": x, "y": y})
}
Input::Scroll(x, y, down, n) => serde_json::json!({"kind": "scroll", "x": x, "y": y, "down": down, "steps": n}),
Input::Key(k) => serde_json::json!({"kind": "key", "key": k}),
Input::Text(t) => serde_json::json!({"kind": "text", "text": t}),
};
let _ = crate::remote::sync_post(&s, "screen", &v);
}
});
return;
}
std::thread::spawn(move || {
let mut seq = 0u64;
while !s2.load(Ordering::SeqCst) {
let got = cloudbox::frame();
if let Some(img) = got {
seq += 1;
let size = [img.width() as usize, img.height() as usize];
let ci = egui::ColorImage::from_rgba_unmultiplied(size, img.as_raw());
*slot.lock().unwrap() = Some((seq, ci));
c.request_repaint();
std::thread::sleep(std::time::Duration::from_millis(VIEWER_FRAME_MS));
} else {
std::thread::sleep(std::time::Duration::from_millis(1000));
c.request_repaint();
}
}
});
let (tx, rx) = channel::<Input>();
self.cu.input_tx = Some(tx);
std::thread::spawn(move || {
while let Ok(i) = rx.recv() {
match i {
Input::Click(x, y, b, n) => cloudbox::viewer_click(x, y, b, n),
Input::Move(x, y) => cloudbox::viewer_move(x, y),
Input::Scroll(x, y, down, n) => cloudbox::viewer_scroll(x, y, down, n),
Input::Key(k) => cloudbox::viewer_key(&k),
Input::Text(t) => cloudbox::viewer_text(&t),
}
}
});
}
pub(super) fn viewer_window(&mut self, ctx: &egui::Context) {
if !self.cu.viewer_open {
return;
}
self.ensure_viewer_threads(ctx);
let p = self.pal;
if let Some((seq, img)) = self.cu.frame.lock().unwrap().take() {
if seq != self.cu.shown_seq {
self.cu.shown_seq = seq;
self.cu.frame_size = img.size;
match &mut self.cu.texture {
Some(t) => t.set(img, egui::TextureOptions::LINEAR),
None => self.cu.texture = Some(ctx.load_texture("box-screen", img, egui::TextureOptions::LINEAR)),
}
}
}
let brain = self.cu.remote.is_some();
// The brain's computer reports its state with the brain's answers.
let run = if brain {
match (self.mir.box_state["running"].as_bool().unwrap_or(false), self.mir.box_state["available"].as_bool().unwrap_or(true)) {
(true, _) => Run::Running,
(false, false) => Run::NoDocker("大腦那台沒有 Docker".into()),
_ => Run::Asleep,
}
} else {
cloudbox::run_state()
};
let screen = ctx.content_rect();
let open = true;
let default = vec2((screen.width() * 0.62).min(900.0), (screen.width() * 0.62).min(900.0) * 0.625 + 52.0);
let busy = if brain { self.mir.box_state["busy"].as_bool().unwrap_or(false) } else { cloudbox::is_busy() };
let title = self.state.bot(self.state.selected).map(|b| crate::model::own_computer_name(&b.name)).unwrap_or_else(|| crate::model::BOT_COMPUTER_NAME.to_string());
let mut close_clicked = false;
let mut done_clicked = false;
let selected = self.state.selected;
let handoff = self.approvals.get(&selected).map(|a| a.kind == crate::providers::ApprovalKind::Handoff).unwrap_or(false);
egui::Window::new("box-viewer")
.id(egui::Id::unique("box-viewer"))
.title_bar(false)
.default_size(default)
.default_pos(screen.center() - default / 2.0)
.min_size(vec2(420.0, 300.0))
.resizable(true)
.collapsible(false)
.frame(
Frame::new()
.fill(p.elevated)
.corner_radius(14)
.inner_margin(Margin::symmetric(14, 12))
.stroke(Stroke::new(1.0, p.border))
.shadow(egui::epaint::Shadow { offset: [0, 14], blur: 40, spread: 0, color: Color32::from_black_alpha(60) }),
)
.show(ctx, |ui| {
// Buttons first, from the right; the title and state get what is left and end in
// … when it's short (the state used to slide under 「取得控制權」 in a narrow window).
ui.horizontal(|ui| {
ui.with_layout(Layout::right_to_left(Align::Center), |ui| {
let (rect, r) = ui.allocate_exact_size(vec2(28.0, 28.0), egui::Sense::click());
if r.hovered() {
ui.painter().rect_filled(rect, 8, p.hover);
}
let c = rect.center();
let st = Stroke::new(1.5, if r.hovered() { p.text } else { p.text2 });
ui.painter().line_segment([c + vec2(-5.0, -5.0), c + vec2(5.0, 5.0)], st);
ui.painter().line_segment([c + vec2(-5.0, 5.0), c + vec2(5.0, -5.0)], st);
if r.on_hover_text(t("關閉")).clicked() {
close_clicked = true;
}
ui.add_space(4.0);
// The bot is waiting for the person here: done, and it carries on.
if handoff && ui.add(egui::Button::new(RichText::new(t("我完成了")).strong().color(p.on_primary)).fill(p.primary).corner_radius(8).min_size(vec2(84.0, 30.0))).clicked() {
done_clicked = true;
}
if !brain && ui.add(secondary_button("全螢幕")).on_hover_text(t("在瀏覽器用 noVNC 全螢幕開啟")).clicked() {
let control = self.cu.control;
self.box_task(move || {
let url = cloudbox::viewer_url(control)?;
crate::preview::open_with_system(&url);
Ok(String::new())
});
}
let fit_label = if self.cu.fit { "實際大小" } else { "調整至視窗大小" };
if ui.add(secondary_button(fit_label)).clicked() {
self.cu.fit = !self.cu.fit;
}
let (label, fill, col) = if self.cu.control { ("交還給 Bot", p.accent, Color32::WHITE) } else { ("取得控制權", p.primary, p.on_primary) };
let btn = egui::Button::new(RichText::new(label).strong().color(col)).fill(fill).corner_radius(8).min_size(vec2(96.0, 30.0));
if ui.add_enabled(run.is_running(), btn).on_hover_text(if self.cu.control { "Bot 可以繼續操作" } else { "你自己點擊和輸入;Bot 的操作會先暫停" }).clicked() {
self.cu.control = !self.cu.control;
match self.cu.remote.clone() {
Some(s) => {
let on = self.cu.control;
std::thread::spawn(move || {
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "hold", "on": on}));
});
}
None => cloudbox::HELD_BY_PERSON.store(self.cu.control, Ordering::SeqCst),
}
}
ui.add_space(8.0);
ui.with_layout(Layout::left_to_right(Align::Center), |ui| {
monitor_icon(ui, 20.0, p.text);
let note = if self.cu.control { Some(t("你正在操作,Bot 已暫停")) } else if busy { Some(t("Bot 正在操作")) } else { None };
let room = (ui.available_width() - 96.0 - if note.is_some() { 120.0 } else { 0.0 }).max(60.0);
ui.label(RichText::new(super::sidebar_ui::shorten_width(ui, &title, room, FONT_LG)).strong().size(FONT_LG));
if ui.available_width() > 80.0 {
status_pill(ui, &run, p);
}
if let Some(n) = note.filter(|_| ui.available_width() > 110.0) {
crate::icons::label(ui, if self.cu.control { crate::icons::Icon::Users } else { crate::icons::Icon::Bot }, &n, FONT_XS, p.accent);
}
});
});
});
ui.add_space(6.0);
let avail = ui.available_size();
match (self.cu.texture.as_ref().map(|t| t.id()), run.is_running()) {
(Some(tex_id), true) => {
let [fw, fh] = self.cu.frame_size;
let (fw, fh) = (fw.max(1) as f32, fh.max(1) as f32);
let size = if self.cu.fit {
let k = (avail.x / fw).min((avail.y.max(120.0)) / fh);
vec2(fw * k, fh * k)
} else {
vec2(fw, fh)
};
egui::ScrollArea::both().auto_shrink([false, false]).show(ui, |ui| {
let sense = if self.cu.control { egui::Sense::click_and_drag() } else { egui::Sense::hover() };
// Centred in the window (it sat at the left with an empty band beside it).
let resp = if self.cu.fit {
let pad = ((ui.available_width() - size.x) / 2.0).max(0.0);
ui.horizontal(|ui| {
ui.add_space(pad);
ui.add(egui::Image::new((tex_id, size)).corner_radius(8).sense(sense))
})
.inner
} else {
ui.add(egui::Image::new((tex_id, size)).corner_radius(8).sense(sense))
};
if self.cu.control {
ui.painter().rect_stroke(resp.rect, 8, Stroke::new(2.0, p.accent), egui::StrokeKind::Outside);
self.forward_input(ui, &resp, (fw, fh));
}
});
}
_ => {
let (rect, _) = ui.allocate_exact_size(vec2(avail.x, avail.y.max(200.0)), egui::Sense::hover());
ui.painter().rect_filled(rect, 8, Color32::from_rgb(0x2b, 0x2f, 0x36));
let text = match &run {
Run::Running => "正在連線…".to_string(),
Run::NoDocker(why) => why.clone(),
Run::Failed(e) => format!("發生問題:{e}"),
other => format!("{}…", other.label()),
};
ui.painter().text(rect.center(), egui::Align2::CENTER_CENTER, text, egui::FontId::proportional(FONT_BASE), Color32::from_gray(220));
if matches!(run, Run::Asleep | Run::Absent) {
let b = egui::Rect::from_center_size(rect.center() + vec2(0.0, 34.0), vec2(90.0, 30.0));
if ui.put(b, primary_button("喚醒", p)).clicked() {
match self.cu.remote.clone() {
Some(s) => {
std::thread::spawn(move || {
let _ = crate::remote::sync_post(&s, "screen", &serde_json::json!({"kind": "wake"}));
});
}
None => self.box_task(|| cloudbox::ensure_running().map(|_| String::new())),
}
}
}
}
}
});
if done_clicked {
self.give_back_computer();
self.answer(ctx, selected, crate::providers::ApprovalAnswer::Once);
self.close_viewer();
} else if close_clicked || !open {
self.close_viewer();
}
ctx.request_repaint_after(std::time::Duration::from_millis(VIEWER_FRAME_MS));
}
/// Clicks, scrolls and keys on the picture go to the box while the person has control.
fn forward_input(&mut self, ui: &mut egui::Ui, resp: &egui::Response, frame: (f32, f32)) {
let Some(tx) = self.cu.input_tx.clone() else { return };
let to_box = |pos: egui::Pos2| -> (i64, i64) {
let r = resp.rect;
(((pos.x - r.left()) / r.width() * frame.0).round() as i64, ((pos.y - r.top()) / r.height() * frame.1).round() as i64)
};
if resp.clicked() || resp.secondary_clicked() || resp.double_clicked() {
resp.request_focus();
}
if let Some(pos) = resp.interact_pointer_pos().or(resp.hover_pos()) {
let (x, y) = to_box(pos);
if resp.double_clicked() {
let _ = tx.send(Input::Click(x, y, "left", 2));
} else if resp.clicked() {
let _ = tx.send(Input::Click(x, y, "left", 1));
} else if resp.secondary_clicked() {
let _ = tx.send(Input::Click(x, y, "right", 1));
} else if resp.hovered() {
let d = ui.input(|i| i.smooth_scroll_delta.y);
if d.abs() > 0.5 {
let _ = tx.send(Input::Scroll(x, y, d < 0.0, ((d.abs() / 40.0).ceil() as u32).max(1)));
} else if ui.input(|i| i.pointer.delta().length() > 3.0) {
let _ = tx.send(Input::Move(x, y));
}
}
}
if !resp.has_focus() {
return;
}
ui.memory_mut(|m| m.set_focus_lock_filter(resp.id, egui::EventFilter { tab: true, horizontal_arrows: true, vertical_arrows: true, escape: true }));
let events = ui.input(|i| i.events.clone());
for e in events {
match e {
egui::Event::Text(t) => {
let _ = tx.send(Input::Text(t));
}
egui::Event::Paste(t) => {
let _ = tx.send(Input::Text(t));
}
egui::Event::Key { key, pressed: true, modifiers, .. } => {
let name = match key {
Key::Enter => "Return",
Key::Tab => "Tab",
Key::Backspace => "BackSpace",
Key::Delete => "Delete",
Key::Escape => "Escape",
Key::ArrowUp => "Up",
Key::ArrowDown => "Down",
Key::ArrowLeft => "Left",
Key::ArrowRight => "Right",
Key::Home => "Home",
Key::End => "End",
Key::PageUp => "Prior",
Key::PageDown => "Next",
_ if modifiers.command || modifiers.ctrl || modifiers.alt => {
let n = key.name();
if n.chars().count() == 1 {
n
} else {
continue;
}
}
_ => continue,
};
let mut combo = vec![];
if modifiers.command || modifiers.ctrl {
combo.push("ctrl".to_string());
}
if modifiers.alt {
combo.push("alt".to_string());
}
if modifiers.shift && (name.len() > 1 || !combo.is_empty()) {
combo.push("shift".to_string());
}
combo.push(if name.len() == 1 { name.to_lowercase() } else { name.to_string() });
let _ = tx.send(Input::Key(combo.join("+")));
}
_ => {}
}
} }
} }
// ---------- the floating bar while a bot drives this Mac ----------
pub(super) fn local_control_bar(&mut self, ctx: &egui::Context) { pub(super) fn local_control_bar(&mut self, ctx: &egui::Context) {
if self.demo { if self.demo {
return; return;

View File

@ -29,9 +29,7 @@ impl Face {
} }
#[derive(Clone, Copy, Debug, PartialEq)] #[derive(Clone, Copy, Debug, PartialEq)]
#[allow(dead_code)]
pub enum Poke { pub enum Poke {
Spin,
Bounce, Bounce,
Nod, Nod,
/// Glance left, then right (the greeting). /// Glance left, then right (the greeting).
@ -44,7 +42,6 @@ const SPLASH_FADE: f64 = 0.35;
fn duration(p: Poke) -> f64 { fn duration(p: Poke) -> f64 {
match p { match p {
Poke::Spin => 0.9,
Poke::Bounce => 0.62, Poke::Bounce => 0.62,
Poke::Nod => 0.5, Poke::Nod => 0.5,
Poke::Look => 0.7, Poke::Look => 0.7,
@ -121,7 +118,6 @@ impl App {
f.poke = None; f.poke = None;
} else { } else {
match kind { match kind {
Poke::Spin => pose.yaw = 180.0 * 2.0 * (1.0 - (PI * e).cos()),
Poke::Nod => pose.pitch = 16.0 * bump(e, 0.0, 1.0), Poke::Nod => pose.pitch = 16.0 * bump(e, 0.0, 1.0),
Poke::Look => pose.gaze = vec2(-(2.0 * PI * e).sin() * 0.9, 0.1), Poke::Look => pose.gaze = vec2(-(2.0 * PI * e).sin() * 0.9, 0.1),
Poke::Bounce => { Poke::Bounce => {

View File

@ -206,6 +206,16 @@ pub fn vnc_password() -> String {
pw pw
} }
/// The password the box's VNC server really has: the one it was made with (its VNC_PASSWORD), so a
/// viewer signs in even when this AnyBot's settings were made anew; else this AnyBot's own.
pub fn box_vnc_password() -> String {
dk_timeout(&["exec", container(), "printenv", "VNC_PASSWORD"], 5_000)
.ok()
.map(|b| String::from_utf8_lossy(&b).trim().to_string())
.filter(|p| !p.is_empty())
.unwrap_or_else(vnc_password)
}
static BUSY: AtomicUsize = AtomicUsize::new(0); static BUSY: AtomicUsize = AtomicUsize::new(0);
/// The person took control in the viewer: the bot's screen actions wait. /// The person took control in the viewer: the bot's screen actions wait.
pub static HELD_BY_PERSON: AtomicBool = AtomicBool::new(false); pub static HELD_BY_PERSON: AtomicBool = AtomicBool::new(false);
@ -417,6 +427,13 @@ fn docker() -> Command {
c c
} }
/// A program in the box with its stdin and stdout piped to this side (as the bot user).
pub fn exec_piped(args: &[&str]) -> Result<std::process::Child, String> {
let mut a = vec!["exec", "-i", "-u", "bot", container()];
a.extend_from_slice(args);
docker().args(&a).stdin(Stdio::piped()).stdout(Stdio::piped()).stderr(Stdio::null()).spawn().map_err(|e| format!("找不到 docker:{e}"))
}
/// The `docker …` prefix for shell strings (used by the command tools). /// The `docker …` prefix for shell strings (used by the command tools).
fn docker_prefix() -> String { fn docker_prefix() -> String {
let host = docker_host(); let host = docker_host();
@ -436,9 +453,24 @@ fn dk(args: &[&str]) -> Result<String, String> {
} }
fn dk_timeout(args: &[&str], ms: u64) -> Result<Vec<u8>, String> { fn dk_timeout(args: &[&str], ms: u64) -> Result<Vec<u8>, String> {
use std::io::Read; dk_with_input(args, None, ms)
let mut child = docker().args(args).stdout(Stdio::piped()).stderr(Stdio::piped()).spawn().map_err(|e| format!("找不到 docker:{e}"))?; }
// Drain both pipes while waiting, or a big screenshot fills the pipe and stalls docker.
/// Run docker with `input` on its stdin (if any): Ok(stdout) or Err(stderr), killed after `ms`
/// ("逾時"). Both pipes are drained while waiting, or a big screenshot fills a pipe and stalls
/// docker.
fn dk_with_input(args: &[&str], input: Option<&[u8]>, ms: u64) -> Result<Vec<u8>, String> {
use std::io::{Read, Write};
let mut child = docker()
.args(args)
.stdin(if input.is_some() { Stdio::piped() } else { Stdio::null() })
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.map_err(|e| format!("找不到 docker:{e}"))?;
if let (Some(b), Some(mut si)) = (input, child.stdin.take()) {
si.write_all(b).map_err(|e| e.to_string())?;
}
let mut so = child.stdout.take().ok_or("沒有輸出")?; let mut so = child.stdout.take().ok_or("沒有輸出")?;
let mut se = child.stderr.take().ok_or("沒有輸出")?; let mut se = child.stderr.take().ok_or("沒有輸出")?;
let out_t = std::thread::spawn(move || { let out_t = std::thread::spawn(move || {
@ -461,7 +493,7 @@ fn dk_timeout(args: &[&str], ms: u64) -> Result<Vec<u8>, String> {
let _ = child.wait(); let _ = child.wait();
return Err("逾時".into()); return Err("逾時".into());
} }
std::thread::sleep(Duration::from_millis(10)); std::thread::sleep(Duration::from_millis(5));
}; };
let out = out_t.join().unwrap_or_default(); let out = out_t.join().unwrap_or_default();
let err = err_t.join().unwrap_or_default(); let err = err_t.join().unwrap_or_default();
@ -1188,13 +1220,14 @@ pub fn thumbnail(width: u32) -> Option<image::RgbaImage> {
image::open(&path).ok().map(|i| i.to_rgba8()) image::open(&path).ok().map(|i| i.to_rgba8())
} }
/// A frame for the viewer (no waking: returns None when the box isn't running). /// The screen now, for a small picture (no waking: None when the box isn't running). People watch
/// it live through VNC (vnc.rs).
pub fn frame() -> Option<image::RgbaImage> { pub fn frame() -> Option<image::RgbaImage> {
if !run_state().is_running() { if !run_state().is_running() {
return None; return None;
} }
let png = dk_timeout(&["exec", container(), "import", "-window", "root", "-display", DISPLAY, "-quality", "1", "png:-"], 5_000).ok()?; let jpeg = dk_timeout(&["exec", container(), "import", "-window", "root", "-display", DISPLAY, "-quality", "80", "jpeg:-"], 5_000).ok()?;
image::load_from_memory(&png).ok().map(|i| i.to_rgba8()) image::load_from_memory(&jpeg).ok().map(|i| i.to_rgba8())
} }
/// The desktop's size in pixels (asked once per box). /// The desktop's size in pixels (asked once per box).
@ -1341,51 +1374,14 @@ fn type_script(text: &str) -> String {
/// Run a step's script in the box; with `picture`, wait for the screen to settle and return it. /// Run a step's script in the box; with `picture`, wait for the screen to settle and return it.
fn box_exec(script: &str, stdin: Option<&str>, picture: bool) -> Result<Option<image::RgbImage>, String> { fn box_exec(script: &str, stdin: Option<&str>, picture: bool) -> Result<Option<image::RgbImage>, String> {
use std::io::{Read, Write};
let full = if picture { format!("{{ {script}; }} || exit $?; sleep {:.2}; import -window root png:- 2>/dev/null || true", crate::computer::SETTLE_MS as f64 / 1000.0) } else { script.to_string() }; let full = if picture { format!("{{ {script}; }} || exit $?; sleep {:.2}; import -window root png:- 2>/dev/null || true", crate::computer::SETTLE_MS as f64 / 1000.0) } else { script.to_string() };
let mut child = docker() // Always an stdin (closed at once when there's no text), as `docker exec -i` expects.
.args(["exec", "-i", "-u", "bot", "-e", "DISPLAY=:1", container(), "sh", "-c", &full]) let out = match dk_with_input(&["exec", "-i", "-u", "bot", "-e", "DISPLAY=:1", container(), "sh", "-c", &full], Some(stdin.unwrap_or("").as_bytes()), 30_000) {
.stdin(Stdio::piped()) Ok(o) => o,
.stdout(Stdio::piped()) Err(e) if e == "逾時" => return Err("這台電腦太久沒有回應".into()),
.stderr(Stdio::piped()) Err(e) if e.is_empty() => return Err("這一步沒有成功".into()),
.spawn() Err(e) => return Err(format!("xdotool:{e}")),
.map_err(|e| format!("找不到 docker:{e}"))?;
{
let mut si = child.stdin.take().ok_or("寫不進 Bot 的電腦")?;
if let Some(t) = stdin {
si.write_all(t.as_bytes()).map_err(|e| e.to_string())?;
}
}
let mut so = child.stdout.take().ok_or("沒有輸出")?;
let mut se = child.stderr.take().ok_or("沒有輸出")?;
let out_t = std::thread::spawn(move || {
let mut v = vec![];
let _ = so.read_to_end(&mut v);
v
});
let err_t = std::thread::spawn(move || {
let mut v = vec![];
let _ = se.read_to_end(&mut v);
v
});
let start = Instant::now();
let status = loop {
if let Ok(Some(st)) = child.try_wait() {
break st;
}
if start.elapsed() > Duration::from_secs(30) {
let _ = child.kill();
let _ = child.wait();
return Err("這台電腦太久沒有回應".into());
}
std::thread::sleep(Duration::from_millis(5));
}; };
let out = out_t.join().unwrap_or_default();
let err = err_t.join().unwrap_or_default();
if !status.success() {
let e = String::from_utf8_lossy(&err).trim().to_string();
return Err(if e.is_empty() { "這一步沒有成功".into() } else { format!("xdotool:{e}") });
}
if !picture || out.is_empty() { if !picture || out.is_empty() {
return Ok(None); return Ok(None);
} }
@ -1456,8 +1452,8 @@ pub fn computer_run(name: &str, v: &Value) -> Result<(String, Option<String>), S
} }
_ => {} _ => {}
} }
if HELD_BY_PERSON.load(Ordering::SeqCst) { if HELD_BY_PERSON.load(Ordering::SeqCst) || crate::vnc::person_active() {
return Err("使用者正在自己操作這台電腦(已取得控制權)。請等他交還控制權後再操作,或先截圖看看畫面。".into()); return Err("使用者正在自己操作這台電腦。請等他操作完(停下幾秒後)再繼續,或先截圖看看畫面。".into());
} }
let (note, picture) = if name == "computer_use_open_app" { let (note, picture) = if name == "computer_use_open_app" {
let note = open_on_box(v["app"].as_str().unwrap_or(""))?; let note = open_on_box(v["app"].as_str().unwrap_or(""))?;
@ -1474,40 +1470,10 @@ pub fn computer_run(name: &str, v: &Value) -> Result<(String, Option<String>), S
} }
} }
// ---------- the viewer's own input (when the person takes control) ---------- // ---------- text a person types in a viewer ----------
// Same scripts, at the desktop's own pixels (the viewer shows it full size), no picture back.
fn viewer(name: &str, v: Value) {
if let Ok((script, stdin, _)) = box_script(name, &v, (1.0, 1.0)) {
let _ = box_exec(&script, stdin.as_deref(), false);
}
touch();
}
pub fn viewer_click(x: i64, y: i64, button: &str, count: u32) {
viewer("computer_use_click", serde_json::json!({"x": x, "y": y, "button": button, "count": count.clamp(1, 3)}));
}
pub fn viewer_move(x: i64, y: i64) {
viewer("computer_use_mouse_move", serde_json::json!({"x": x, "y": y}));
}
pub fn viewer_scroll(x: i64, y: i64, down: bool, steps: u32) {
viewer("computer_use_scroll", serde_json::json!({"x": x, "y": y, "direction": if down { "down" } else { "up" }, "amount": steps.clamp(1, 20)}));
}
/// A key from a viewer: an X key name (「BackSpace」「Return」, ctrl+c…) or the bot's names.
pub fn viewer_key(combo: &str) {
let c = combo.trim();
if c.is_empty() || !c.chars().all(|ch| ch.is_ascii_alphanumeric() || "+_-".contains(ch)) {
return;
}
// X key names pass straight through (they are what the viewers send).
let x = combo_to_x(c).unwrap_or_else(|_| c.to_string());
let _ = box_exec(&format!("xdotool key --clearmodifiers {x}"), None, false);
touch();
}
/// Text from a person's viewer (any language): typed or pasted into the window in front, like
/// the bot's typing. (Their mouse and keys go through VNC itself.)
pub fn viewer_text(text: &str) { pub fn viewer_text(text: &str) {
if !text.is_empty() { if !text.is_empty() {
let _ = box_exec(&type_script(text), Some(text), false); let _ = box_exec(&type_script(text), Some(text), false);
@ -1686,8 +1652,7 @@ mod tests {
} }
/// Uses the real Docker: `cargo test --release -- --ignored box_screen_is_exact --nocapture`. /// Uses the real Docker: `cargo test --release -- --ignored box_screen_is_exact --nocapture`.
/// The bot's clicks land where its picture says, a phone's tap where the phone's picture says, /// The bot's clicks land where its picture says, the same spot twice doesn't hang, typing (ASCII, Chinese, a key by name) reaches the app,
/// the same spot twice doesn't hang, typing (ASCII, Chinese, a key by name) reaches the app,
/// and every step comes back with the screen after it. /// and every step comes back with the screen after it.
#[test] #[test]
#[ignore] #[ignore]
@ -1713,9 +1678,6 @@ mod tests {
println!("two clicks with pictures: {} ms", t.elapsed().as_millis()); println!("two clicks with pictures: {} ms", t.elapsed().as_millis());
assert!(t.elapsed() < Duration::from_secs(6)); assert!(t.elapsed() < Duration::from_secs(6));
assert_eq!(pointer(), ((40.0 * w as f64 / fw as f64).round() as i64, (40.0 * h as f64 / fh as f64).round() as i64)); assert_eq!(pointer(), ((40.0 * w as f64 / fw as f64).round() as i64, (40.0 * h as f64 / fh as f64).round() as i64));
// A phone's tap is in the full-size frame it was sent.
viewer_click(321, 123, "left", 1);
assert_eq!(pointer(), (321, 123));
// Typing reaches the app in front: a fresh terminal, ASCII then Chinese, a key by name, return. // Typing reaches the app in front: a fresh terminal, ASCII then Chinese, a key by name, return.
let _ = dk(&["exec", "-u", "bot", container(), "rm", "-f", "/home/bot/exact.txt"]); let _ = dk(&["exec", "-u", "bot", container(), "rm", "-f", "/home/bot/exact.txt"]);
let (txt, img) = computer_run("computer_use_open_app", &serde_json::json!({"app": "終端機"})).unwrap(); let (txt, img) = computer_run("computer_use_open_app", &serde_json::json!({"app": "終端機"})).unwrap();

View File

@ -215,12 +215,9 @@ pub fn label(name: &str) -> &'static str {
} }
} }
fn args_of(args: &str) -> Value {
serde_json::from_str(if args.trim().is_empty() { "{}" } else { args }).unwrap_or(Value::Null)
}
pub fn describe(name: &str, args: &str) -> String { pub fn describe(name: &str, args: &str) -> String {
let v = args_of(args); let v = crate::tools::args_of(args);
// By number from the app's list (e7), else where on the screen (it said "(0, 0)" for every // By number from the app's list (e7), else where on the screen (it said "(0, 0)" for every
// click by number). // click by number).
let xy = || match v["ref"].as_str().map(str::trim).filter(|r| !r.is_empty()) { let xy = || match v["ref"].as_str().map(str::trim).filter(|r| !r.is_empty()) {
@ -251,11 +248,6 @@ pub fn describe(name: &str, args: &str) -> String {
} }
} }
#[allow(dead_code)]
pub fn specs() -> Vec<Value> {
specs_for(&[], "")
}
/// The screen tools, each with a `computer` choice. One way to act (a real click or keystroke), /// The screen tools, each with a `computer` choice. One way to act (a real click or keystroke),
/// two ways to see: the picture, or an app's controls as numbered text. /// two ways to see: the picture, or an app's controls as numbered text.
pub fn specs_for(computers: &[String], default: &str) -> Vec<Value> { pub fn specs_for(computers: &[String], default: &str) -> Vec<Value> {
@ -445,7 +437,6 @@ fn flags_of(v: &Value) -> u64 {
} }
/// Run a computer_use tool. Returns (text, ok, screenshot as base64 JPEG). /// Run a computer_use tool. Returns (text, ok, screenshot as base64 JPEG).
#[allow(dead_code)]
pub fn run(name: &str, args: &str) -> (String, bool, Option<String>) { pub fn run(name: &str, args: &str) -> (String, bool, Option<String>) {
run_by(name, args, "") run_by(name, args, "")
} }
@ -455,7 +446,7 @@ pub fn run_by(name: &str, args: &str, who: &str) -> (String, bool, Option<String
if moves_things(name) { if moves_things(name) {
*LAST_ACTION.lock().unwrap_or_else(|e| e.into_inner()) = Some((std::time::Instant::now(), who.to_string())); *LAST_ACTION.lock().unwrap_or_else(|e| e.into_inner()) = Some((std::time::Instant::now(), who.to_string()));
} }
match run_inner(name, &args_of(args)) { match run_inner(name, &crate::tools::args_of(args)) {
Ok((t, img)) => (t, true, img), Ok((t, img)) => (t, true, img),
Err(e) => (e, false, None), Err(e) => (e, false, None),
} }
@ -536,13 +527,29 @@ fn open_app(app: &str) -> Result<String, String> {
if app.is_empty() { if app.is_empty() {
return Err("請給要打開的 App 名稱".into()); return Err("請給要打開的 App 名稱".into());
} }
crate::desktop::launch(app).or_else(|e| match app_named(app) { launch(app).or_else(|e| match app_named(app) {
Some(real) if real != app => crate::desktop::launch(&real), Some(real) if real != app => launch(&real),
_ => Err(e), _ => Err(e),
})?; })?;
Ok(format!("已打開「{app}」,放在最前面。")) Ok(format!("已打開「{app}」,放在最前面。"))
} }
/// Open an app or bring it forward. macOS: LaunchServices (`open -a`), which needs no
/// permission (agent-desktop's launch waited on the window through accessibility and failed
/// without 輔助使用 now and then); elsewhere agent-desktop's.
fn launch(app: &str) -> Result<(), String> {
#[cfg(target_os = "macos")]
{
let out = std::process::Command::new("open").args(["-a", app]).output().map_err(|e| e.to_string())?;
if out.status.success() {
return Ok(());
}
Err(format!("找不到「{app}」這個 App"))
}
#[cfg(not(target_os = "macos"))]
crate::desktop::launch(app)
}
/// Bring the app a control belongs to in front, so a click at its spot lands in it (and not in /// Bring the app a control belongs to in front, so a click at its spot lands in it (and not in
/// a window lying over it). Nothing to do when it is in front already. /// a window lying over it). Nothing to do when it is in front already.
fn bring_forward(app: &str) { fn bring_forward(app: &str) {
@ -555,7 +562,7 @@ fn bring_forward(app: &str) {
if app.is_empty() || front.eq_ignore_ascii_case(app) { if app.is_empty() || front.eq_ignore_ascii_case(app) {
return; return;
} }
if crate::desktop::launch(app).is_ok() { if launch(app).is_ok() {
std::thread::sleep(std::time::Duration::from_millis(300)); std::thread::sleep(std::time::Duration::from_millis(300));
} }
} }
@ -773,7 +780,7 @@ pub fn person_act(name: &str, v: &Value) -> Result<(), String> {
/// The app a screen action will land in: the window under the point for clicks, else the /// The app a screen action will land in: the window under the point for clicks, else the
/// frontmost app. Used for the per-app permission (「AnyBot 可以看到 {app},並在其中點擊和輸入」). /// frontmost app. Used for the per-app permission (「AnyBot 可以看到 {app},並在其中點擊和輸入」).
pub fn target_app(name: &str, args: &str) -> Option<String> { pub fn target_app(name: &str, args: &str) -> Option<String> {
let v = args_of(args); let v = crate::tools::args_of(args);
if let Some(app) = v["app"].as_str().filter(|a| !a.trim().is_empty() && matches!(name, "computer_use_read_app" | "computer_use_open_app")) { if let Some(app) = v["app"].as_str().filter(|a| !a.trim().is_empty() && matches!(name, "computer_use_read_app" | "computer_use_open_app")) {
return Some(app.trim().to_string()); return Some(app.trim().to_string());
} }

View File

@ -259,7 +259,8 @@ pub fn read_app(app: &str) -> Result<(String, Vec<String>), String> {
Ok((app_name, lines)) Ok((app_name, lines))
} }
/// Open an app, or bring it to the front when it runs. /// Open an app, or bring it to the front when it runs (macOS uses LaunchServices instead).
#[cfg(not(target_os = "macos"))]
pub fn launch(app: &str) -> Result<(), String> { pub fn launch(app: &str) -> Result<(), String> {
run("launch", &serde_json::json!({"app": app})).map(|_| ()) run("launch", &serde_json::json!({"app": app})).map(|_| ())
} }

View File

@ -9,6 +9,25 @@ use std::net::{TcpListener, TcpStream};
use std::sync::Arc; use std::sync::Arc;
use std::time::Duration; use std::time::Duration;
/// Answer a WebSocket upgrade: "101 Switching Protocols" (or 400 without a key). True when the
/// connection is now a WebSocket.
pub fn accept_websocket(req: &Req, c: &mut TcpStream) -> bool {
let key = req.header("sec-websocket-key");
if key.is_empty() {
write_response(c, &Resp::error(400, "bad request"));
return false;
}
let accept = tungstenite::handshake::derive_accept_key(key.as_bytes());
let ok = c.write_all(format!("HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: {accept}\r\n\r\n").as_bytes()).is_ok();
let _ = c.set_nodelay(true);
ok
}
/// Percent-encode a query value or form field (RFC 3986: letters, digits and -_.~ stay).
pub fn url_encode(s: &str) -> String {
s.bytes().map(|b| if b.is_ascii_alphanumeric() || b"-_.~".contains(&b) { (b as char).to_string() } else { format!("%{b:02X}") }).collect()
}
/// Largest request body accepted (a phone photo fits). /// Largest request body accepted (a phone photo fits).
pub const MAX_BODY: usize = 30 * 1024 * 1024; pub const MAX_BODY: usize = 30 * 1024 * 1024;

View File

@ -7,7 +7,6 @@ use std::collections::HashMap;
use std::sync::{Mutex, OnceLock}; use std::sync::{Mutex, OnceLock};
#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)] #[derive(Clone, Copy, PartialEq, Eq, Hash, Debug)]
#[allow(dead_code)]
pub enum Icon { pub enum Icon {
SidebarCollapse, SidebarCollapse,
SidebarExpand, SidebarExpand,
@ -19,7 +18,6 @@ pub enum Icon {
Stop, Stop,
Monitor, Monitor,
Mic, Mic,
Speaker,
Sparkle, Sparkle,
Chat, Chat,
Shield, Shield,
@ -29,8 +27,6 @@ pub enum Icon {
Person, Person,
Eye, Eye,
CloudDownload, CloudDownload,
PanelRight,
Phone,
Clock, Clock,
Calendar, Calendar,
Wave, Wave,
@ -56,7 +52,6 @@ pub enum Icon {
NewChat, NewChat,
History, History,
Eraser, Eraser,
Info,
Bot, Bot,
Users, Users,
Template, Template,
@ -70,10 +65,6 @@ pub enum Icon {
File, File,
Lock, Lock,
Update, Update,
Keyboard,
Globe,
ChevronRight,
ChevronDown,
Zap, Zap,
Alert, Alert,
Summary, Summary,
@ -81,10 +72,8 @@ pub enum Icon {
MapPin, MapPin,
CloudSun, CloudSun,
ShoppingBag, ShoppingBag,
ExternalLink,
Message, Message,
Alarm, Alarm,
Mail,
CircleCheck, CircleCheck,
CircleX, CircleX,
Minus, Minus,
@ -93,6 +82,7 @@ pub enum Icon {
Pause, Pause,
FileJson, FileJson,
FileWord, FileWord,
ChevronRight,
} }
impl Icon { impl Icon {
@ -115,6 +105,7 @@ impl Icon {
/// The Lucide file. /// The Lucide file.
fn svg(self) -> &'static str { fn svg(self) -> &'static str {
match self { match self {
Icon::ChevronRight => include_str!("../icons/lucide/chevron-right.svg"),
Icon::SidebarCollapse => include_str!("../icons/lucide/panel-left-close.svg"), Icon::SidebarCollapse => include_str!("../icons/lucide/panel-left-close.svg"),
Icon::SidebarExpand => include_str!("../icons/lucide/panel-left-open.svg"), Icon::SidebarExpand => include_str!("../icons/lucide/panel-left-open.svg"),
Icon::Plus => include_str!("../icons/lucide/plus.svg"), Icon::Plus => include_str!("../icons/lucide/plus.svg"),
@ -125,7 +116,6 @@ impl Icon {
Icon::Stop => include_str!("../icons/lucide/square.svg"), Icon::Stop => include_str!("../icons/lucide/square.svg"),
Icon::Monitor => include_str!("../icons/lucide/monitor.svg"), Icon::Monitor => include_str!("../icons/lucide/monitor.svg"),
Icon::Mic => include_str!("../icons/lucide/mic.svg"), Icon::Mic => include_str!("../icons/lucide/mic.svg"),
Icon::Speaker => include_str!("../icons/lucide/volume-2.svg"),
Icon::Sparkle => include_str!("../icons/lucide/sparkles.svg"), Icon::Sparkle => include_str!("../icons/lucide/sparkles.svg"),
Icon::Chat => include_str!("../icons/lucide/messages-square.svg"), Icon::Chat => include_str!("../icons/lucide/messages-square.svg"),
Icon::Shield => include_str!("../icons/lucide/shield-check.svg"), Icon::Shield => include_str!("../icons/lucide/shield-check.svg"),
@ -135,8 +125,6 @@ impl Icon {
Icon::Person => include_str!("../icons/lucide/user-round.svg"), Icon::Person => include_str!("../icons/lucide/user-round.svg"),
Icon::Eye => include_str!("../icons/lucide/eye.svg"), Icon::Eye => include_str!("../icons/lucide/eye.svg"),
Icon::CloudDownload => include_str!("../icons/lucide/cloud-download.svg"), Icon::CloudDownload => include_str!("../icons/lucide/cloud-download.svg"),
Icon::PanelRight => include_str!("../icons/lucide/panel-right.svg"),
Icon::Phone => include_str!("../icons/lucide/phone.svg"),
Icon::Clock => include_str!("../icons/lucide/clock.svg"), Icon::Clock => include_str!("../icons/lucide/clock.svg"),
Icon::Calendar => include_str!("../icons/lucide/calendar.svg"), Icon::Calendar => include_str!("../icons/lucide/calendar.svg"),
Icon::Wave => include_str!("../icons/lucide/audio-lines.svg"), Icon::Wave => include_str!("../icons/lucide/audio-lines.svg"),
@ -162,7 +150,6 @@ impl Icon {
Icon::NewChat => include_str!("../icons/lucide/square-pen.svg"), Icon::NewChat => include_str!("../icons/lucide/square-pen.svg"),
Icon::History => include_str!("../icons/lucide/history.svg"), Icon::History => include_str!("../icons/lucide/history.svg"),
Icon::Eraser => include_str!("../icons/lucide/eraser.svg"), Icon::Eraser => include_str!("../icons/lucide/eraser.svg"),
Icon::Info => include_str!("../icons/lucide/info.svg"),
Icon::Bot => include_str!("../icons/lucide/bot.svg"), Icon::Bot => include_str!("../icons/lucide/bot.svg"),
Icon::Users => include_str!("../icons/lucide/users.svg"), Icon::Users => include_str!("../icons/lucide/users.svg"),
Icon::Template => include_str!("../icons/lucide/layout-template.svg"), Icon::Template => include_str!("../icons/lucide/layout-template.svg"),
@ -176,10 +163,6 @@ impl Icon {
Icon::File => include_str!("../icons/lucide/file.svg"), Icon::File => include_str!("../icons/lucide/file.svg"),
Icon::Lock => include_str!("../icons/lucide/lock.svg"), Icon::Lock => include_str!("../icons/lucide/lock.svg"),
Icon::Update => include_str!("../icons/lucide/circle-arrow-up.svg"), Icon::Update => include_str!("../icons/lucide/circle-arrow-up.svg"),
Icon::Keyboard => include_str!("../icons/lucide/keyboard.svg"),
Icon::Globe => include_str!("../icons/lucide/globe.svg"),
Icon::ChevronRight => include_str!("../icons/lucide/chevron-right.svg"),
Icon::ChevronDown => include_str!("../icons/lucide/chevron-down.svg"),
Icon::Zap => include_str!("../icons/lucide/zap.svg"), Icon::Zap => include_str!("../icons/lucide/zap.svg"),
Icon::Alert => include_str!("../icons/lucide/circle-alert.svg"), Icon::Alert => include_str!("../icons/lucide/circle-alert.svg"),
Icon::Summary => include_str!("../icons/lucide/notebook-pen.svg"), Icon::Summary => include_str!("../icons/lucide/notebook-pen.svg"),
@ -187,10 +170,8 @@ impl Icon {
Icon::MapPin => include_str!("../icons/lucide/map-pin.svg"), Icon::MapPin => include_str!("../icons/lucide/map-pin.svg"),
Icon::CloudSun => include_str!("../icons/lucide/cloud-sun.svg"), Icon::CloudSun => include_str!("../icons/lucide/cloud-sun.svg"),
Icon::ShoppingBag => include_str!("../icons/lucide/shopping-bag.svg"), Icon::ShoppingBag => include_str!("../icons/lucide/shopping-bag.svg"),
Icon::ExternalLink => include_str!("../icons/lucide/external-link.svg"),
Icon::Message => include_str!("../icons/lucide/message-circle.svg"), Icon::Message => include_str!("../icons/lucide/message-circle.svg"),
Icon::Alarm => include_str!("../icons/lucide/alarm-clock.svg"), Icon::Alarm => include_str!("../icons/lucide/alarm-clock.svg"),
Icon::Mail => include_str!("../icons/lucide/mail.svg"),
Icon::CircleCheck => include_str!("../icons/lucide/circle-check.svg"), Icon::CircleCheck => include_str!("../icons/lucide/circle-check.svg"),
Icon::CircleX => include_str!("../icons/lucide/circle-x.svg"), Icon::CircleX => include_str!("../icons/lucide/circle-x.svg"),
Icon::Minus => include_str!("../icons/lucide/minus.svg"), Icon::Minus => include_str!("../icons/lucide/minus.svg"),

View File

@ -47,6 +47,7 @@ mod tools;
mod triggers; mod triggers;
mod vault; mod vault;
mod vault_store; mod vault_store;
mod vnc;
mod voice; mod voice;
mod tunnel; mod tunnel;
mod web; mod web;

View File

@ -35,9 +35,7 @@ pub fn twilio_auth(token: &str) -> Result<(String, String), String> {
} }
fn form(pairs: &[(&str, &str)]) -> String { fn form(pairs: &[(&str, &str)]) -> String {
fn enc(s: &str) -> String { let enc = crate::httpd::url_encode;
s.bytes().map(|b| if b.is_ascii_alphanumeric() || b"-_.~".contains(&b) { (b as char).to_string() } else { format!("%{b:02X}") }).collect()
}
pairs.iter().map(|(k, v)| format!("{}={}", enc(k), enc(v))).collect::<Vec<_>>().join("&") pairs.iter().map(|(k, v)| format!("{}={}", enc(k), enc(v))).collect::<Vec<_>>().join("&")
} }

View File

@ -322,6 +322,8 @@ pub fn handle(req: Req, via_relay: bool) -> Resp {
let path: Vec<&str> = parts.iter().map(|s| *s).collect(); let path: Vec<&str> = parts.iter().map(|s| *s).collect();
match (req.method.as_str(), path.as_slice()) { match (req.method.as_str(), path.as_slice()) {
("GET", []) | ("GET", ["index.html"]) => return page(), ("GET", []) | ("GET", ["index.html"]) => return page(),
("GET", ["screen"]) => return Resp::bytes("text/html; charset=utf-8", SCREEN_PAGE.as_bytes().to_vec()).with_header("Content-Security-Policy", "default-src 'self'; img-src 'self' blob: data:; style-src 'self' 'unsafe-inline'; script-src 'self' 'unsafe-inline'; connect-src 'self' ws: wss:"),
("GET", ["novnc.js"]) => return Resp::bytes("text/javascript; charset=utf-8", NOVNC.as_bytes().to_vec()).with_header("Cache-Control", "public, max-age=86400"),
("GET", ["manifest.webmanifest"]) => return Resp::bytes("application/manifest+json", MANIFEST.as_bytes().to_vec()), ("GET", ["manifest.webmanifest"]) => return Resp::bytes("application/manifest+json", MANIFEST.as_bytes().to_vec()),
("GET", ["icon.svg"]) => return Resp::bytes("image/svg+xml", ICON.as_bytes().to_vec()), ("GET", ["icon.svg"]) => return Resp::bytes("image/svg+xml", ICON.as_bytes().to_vec()),
("GET", [name]) if ICON_PNGS.iter().any(|(n, _)| n == name) => return Resp::bytes("image/png", ICON_PNGS.iter().find(|(n, _)| n == name).map(|(_, b)| b.to_vec()).unwrap_or_default()), ("GET", [name]) if ICON_PNGS.iter().any(|(n, _)| n == name) => return Resp::bytes("image/png", ICON_PNGS.iter().find(|(n, _)| n == name).map(|(_, b)| b.to_vec()).unwrap_or_default()),
@ -435,24 +437,16 @@ pub fn handle(req: Req, via_relay: bool) -> Resp {
Err(e) => Resp::error(503, &e), Err(e) => Resp::error(503, &e),
}, },
("POST", ["api", "sync", "upload", id]) => upload(id.parse().unwrap_or(0), &req), ("POST", ["api", "sync", "upload", id]) => upload(id.parse().unwrap_or(0), &req),
// A screen of this brain watches (and may take over) the bots' own computer, which lives // A screen of this brain shows the bots' own computer, which lives here: a small picture
// here: its picture on that screen used to be its own computer's, not this one's. // (people watch it live through VNC, see vnc.rs).
("GET", ["api", "sync", "screen"]) => { ("GET", ["api", "sync", "screen"]) => match crate::cloudbox::thumbnail(320) {
if req.query.get("thumb").map(|t| t == "1").unwrap_or(false) { Some(img) => match crate::computer::encode_jpeg(&image::DynamicImage::ImageRgba8(img).to_rgb8()) {
match crate::cloudbox::thumbnail(320) { Some(j) => Resp::bytes("image/jpeg", j),
Some(img) => { None => Resp::error(500, "畫面編碼失敗"),
let rgb = image::DynamicImage::ImageRgba8(img).to_rgb8(); },
match crate::computer::encode_jpeg(&rgb) { None => Resp::error(404, "還沒有畫面"),
Some(j) => Resp::bytes("image/jpeg", j), },
None => Resp::error(500, "畫面編碼失敗"), // 接管 / 我完成了 from a screen: the bot's steps wait (or not) for the person.
}
}
None => Resp::error(404, "還沒有畫面"),
}
} else {
screen("box", &s)
}
}
("POST", ["api", "sync", "screen"]) => { ("POST", ["api", "sync", "screen"]) => {
let v = req.json(); let v = req.json();
match v["kind"].as_str().unwrap_or("") { match v["kind"].as_str().unwrap_or("") {
@ -460,25 +454,11 @@ pub fn handle(req: Req, via_relay: bool) -> Resp {
crate::cloudbox::HELD_BY_PERSON.store(v["on"].as_bool().unwrap_or(false), Ordering::SeqCst); crate::cloudbox::HELD_BY_PERSON.store(v["on"].as_bool().unwrap_or(false), Ordering::SeqCst);
Resp::json(&json!({"ok": true})) Resp::json(&json!({"ok": true}))
} }
"viewer" => { "wake" => match crate::cloudbox::ensure_running() {
crate::cloudbox::set_viewer_open(v["on"].as_bool().unwrap_or(false)); Ok(()) => Resp::json(&json!({"ok": true})),
Resp::json(&json!({"ok": true})) Err(e) => Resp::error(500, &e),
} },
// The person's own hands on the picture (frame pixels, like this computer's own _ => Resp::error(400, "不認得的操作"),
// viewer; not the bot's tools, which stop while the person holds it).
"click" | "move" | "scroll" | "key" | "text" => {
crate::cloudbox::touch();
let (x, y) = (v["x"].as_i64().unwrap_or(0), v["y"].as_i64().unwrap_or(0));
match v["kind"].as_str().unwrap_or("") {
"click" => crate::cloudbox::viewer_click(x, y, if v["button"].as_str() == Some("right") { "right" } else { "left" }, v["count"].as_u64().unwrap_or(1).clamp(1, 3) as u32),
"move" => crate::cloudbox::viewer_move(x, y),
"scroll" => crate::cloudbox::viewer_scroll(x, y, v["down"].as_bool().unwrap_or(true), v["steps"].as_u64().unwrap_or(1).clamp(1, 20) as u32),
"key" => crate::cloudbox::viewer_key(v["key"].as_str().unwrap_or("")),
_ => crate::cloudbox::viewer_text(v["text"].as_str().unwrap_or("")),
}
Resp::json(&json!({"ok": true}))
}
_ => screen_input("box", &v, &s),
} }
} }
("POST", ["api", "sync", "models"]) => match hub().ask_sync_for(SyncKind::Models(req.json()["provider"].as_str().unwrap_or("").to_string()), 45) { ("POST", ["api", "sync", "models"]) => match hub().ask_sync_for(SyncKind::Models(req.json()["provider"].as_str().unwrap_or("").to_string()), 45) {
@ -606,21 +586,10 @@ fn upload(bot: u64, req: &Req) -> Resp {
} }
/// A frame of the bot's computer ("box") or this Mac ("mac") as JPEG. /// A frame of the bot's computer ("box") or this Mac ("mac") as JPEG.
/// This Mac's screen for the phone (when allowed). The bots' computer is watched through VNC
/// (the page /screen, vnc.rs).
fn screen(which: &str, s: &Settings) -> Resp { fn screen(which: &str, s: &Settings) -> Resp {
match which { match which {
"box" => {
crate::cloudbox::touch();
match crate::cloudbox::frame() {
Some(img) => {
let rgb = image::DynamicImage::ImageRgba8(img).to_rgb8();
match crate::computer::encode_jpeg(&rgb) {
Some(j) => Resp::bytes("image/jpeg", j).with_header("X-Size", &format!("{}x{}", rgb.width(), rgb.height())),
None => Resp::error(500, "畫面編碼失敗"),
}
}
None => Resp::error(503, "Bot 的電腦在休息,按「叫醒」"),
}
}
"mac" if s.mac_from_phone => { "mac" if s.mac_from_phone => {
let (text, ok, img) = crate::computer::run_by("computer_use_screenshot", "{}", "手機"); let (text, ok, img) = crate::computer::run_by("computer_use_screenshot", "{}", "手機");
match (ok, img) { match (ok, img) {
@ -638,41 +607,15 @@ fn screen(which: &str, s: &Settings) -> Resp {
} }
} }
/// The phone taps, types or scrolls on a screen, as the person, in the pixels of the picture it /// The phone taps, types or scrolls on this Mac's screen, as the person, in the pixels of the
/// was sent: the box's frame is its desktop at full size, this computer's is the bot-sized /// picture it was sent (shot_size). The bots' computer takes the person's input through VNC.
/// picture (shot_size). It used to go through the bot's tools, which mapped the tap with the
/// bot's last screenshot (a different size: taps landed a quarter off, or failed with "take a
/// screenshot first") and refused it while the person held the screen.
fn screen_input(which: &str, v: &Value, s: &Settings) -> Resp { fn screen_input(which: &str, v: &Value, s: &Settings) -> Resp {
let kind = v["kind"].as_str().unwrap_or("");
if which == "box" && kind == "wake" {
return match crate::cloudbox::ensure_running() {
Ok(()) => Resp::json(&json!({"ok": true})),
Err(e) => Resp::error(500, &e),
};
}
let (x, y) = (v["x"].as_f64().unwrap_or(0.0), v["y"].as_f64().unwrap_or(0.0)); let (x, y) = (v["x"].as_f64().unwrap_or(0.0), v["y"].as_f64().unwrap_or(0.0));
let down = v["direction"].as_str().unwrap_or("down") != "up"; let down = v["direction"].as_str().unwrap_or("down") != "up";
let amount = v["amount"].as_i64().unwrap_or(3).clamp(1, 20); let amount = v["amount"].as_i64().unwrap_or(3).clamp(1, 20);
match which { match which {
"box" => {
if !matches!(kind, "click" | "type" | "key" | "scroll") {
return Resp::error(400, "不認得的操作");
}
if let Err(e) = crate::cloudbox::ensure_running() {
return Resp::error(500, &e);
}
let (xi, yi) = (x.round() as i64, y.round() as i64);
match kind {
"click" => crate::cloudbox::viewer_click(xi, yi, if v["button"].as_str() == Some("right") { "right" } else { "left" }, v["count"].as_u64().unwrap_or(1) as u32),
"type" => crate::cloudbox::viewer_text(v["text"].as_str().unwrap_or("")),
"key" => crate::cloudbox::viewer_key(v["key"].as_str().unwrap_or("")),
_ => crate::cloudbox::viewer_scroll(xi, yi, down, amount as u32),
}
Resp::json(&json!({"ok": true}))
}
"mac" if s.mac_from_phone => { "mac" if s.mac_from_phone => {
let (name, args) = match kind { let (name, args) = match v["kind"].as_str().unwrap_or("") {
"click" => ("computer_use_click", json!({"x": x, "y": y, "button": v["button"].as_str().unwrap_or("left"), "count": v["count"].as_u64().unwrap_or(1)})), "click" => ("computer_use_click", json!({"x": x, "y": y, "button": v["button"].as_str().unwrap_or("left"), "count": v["count"].as_u64().unwrap_or(1)})),
"type" => ("computer_use_typing", json!({"value": v["text"]})), "type" => ("computer_use_typing", json!({"value": v["text"]})),
"key" => ("computer_use_press_key", json!({"key": v["key"]})), "key" => ("computer_use_press_key", json!({"key": v["key"]})),
@ -684,7 +627,8 @@ fn screen_input(which: &str, v: &Value, s: &Settings) -> Resp {
Err(e) => Resp::error(500, &e), Err(e) => Resp::error(500, &e),
} }
} }
_ => Resp::error(403, "這台 Mac 沒有開放從手機操作"), "mac" => Resp::error(403, "這台 Mac 沒有開放從手機操作"),
_ => Resp::error(404, "沒有這個畫面"),
} }
} }
@ -862,10 +806,65 @@ pub fn set_lan(on: bool) -> Result<u16, String> {
.ok_or("找不到可用的連接埠")?; .ok_or("找不到可用的連接埠")?;
let port = listener.local_addr().map_err(|e| e.to_string())?.port(); let port = listener.local_addr().map_err(|e| e.to_string())?.port();
*LAN_BOUND.lock().unwrap() = Some(port); *LAN_BOUND.lock().unwrap() = Some(port);
std::thread::spawn(move || httpd::serve(listener, Arc::new(|r| handle(r, false)), Arc::new(move || LAN_GEN.load(Ordering::SeqCst) == gen))); std::thread::spawn(move || httpd::serve_upgrade(listener, Arc::new(|r| handle(r, false)), Some(Arc::new(lan_vnc)), Arc::new(move || LAN_GEN.load(Ordering::SeqCst) == gen)));
Ok(port) Ok(port)
} }
/// The phone page on the local network watching the bots' computer: /api/vnc, signed in with
/// the phone's password like every other request here; the VNC stream goes straight to the box.
fn lan_vnc(req: Req, mut c: std::net::TcpStream) {
use tungstenite::{protocol::Role, Message, WebSocket};
if req.path != "/api/vnc" {
return httpd::write_response(&mut c, &Resp::error(404, "沒有這個網址"));
}
if devices_ok(&req.bearer()).is_none() {
return httpd::write_response(&mut c, &Resp::error(401, "請先登入"));
}
if !httpd::accept_websocket(&req, &mut c) {
return;
}
let _ = c.set_read_timeout(Some(Duration::from_millis(2)));
let mut ws = WebSocket::from_raw_socket(c, Role::Server, None);
let (tx, rx) = std::sync::mpsc::channel::<Vec<u8>>();
let send = move |b: &[u8]| tx.send(b.to_vec()).is_ok();
let Ok(mut session) = crate::vnc::open(send.clone()) else { return };
if !send(crate::vnc::GREETING) {
return;
}
'live: loop {
match ws.read() {
Ok(Message::Binary(b)) => {
if !session.feed(&b, &send) {
break;
}
}
Ok(Message::Text(t)) => crate::vnc::person_typed(&t),
Ok(Message::Close(_)) => break,
Ok(_) => {}
Err(tungstenite::Error::Io(e)) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => {}
Err(_) => break,
}
loop {
match rx.try_recv() {
// An empty send: the box's side ended.
Ok(b) if b.is_empty() => break 'live,
Ok(b) => {
if ws.write(Message::Binary(b.into())).is_err() {
break 'live;
}
}
Err(_) => break,
}
}
match ws.flush() {
Err(tungstenite::Error::Io(e)) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => {}
Err(_) => break,
Ok(()) => {}
}
}
let _ = ws.close(None);
}
/// Addresses a phone can open (local network and Tailscale). /// Addresses a phone can open (local network and Tailscale).
pub fn addresses(port: u16) -> Vec<String> { pub fn addresses(port: u16) -> Vec<String> {
// macOS: ifconfig ("inet 192.168.1.5 netmask…"); Linux: `ip -4 -o addr` ("… inet 192.168.1.5/24 …"). // macOS: ifconfig ("inet 192.168.1.5 netmask…"); Linux: `ip -4 -o addr` ("… inet 192.168.1.5/24 …").
@ -1212,7 +1211,7 @@ fn to_brain_raw(s: &Settings, method: &str, rest: &str, query: &[(&str, String)]
if let Some(r) = crate::tunnel::call(&s.brain_id, method, &format!("/api/sync{rest}"), query, body, secs) { if let Some(r) = crate::tunnel::call(&s.brain_id, method, &format!("/api/sync{rest}"), query, body, secs) {
return r.map(|a| (a.status, a.body)); return r.map(|a| (a.status, a.body));
} }
let q: Vec<String> = query.iter().map(|(k, v)| format!("{k}={}", url_escape(v))).collect(); let q: Vec<String> = query.iter().map(|(k, v)| format!("{k}={}", httpd::url_encode(v))).collect();
let url = brain_url(s, &format!("{rest}{}{}", if q.is_empty() { "" } else { "?" }, q.join("&")))?; let url = brain_url(s, &format!("{rest}{}{}", if q.is_empty() { "" } else { "?" }, q.join("&")))?;
let auth = format!("Bearer {}", s.host_token); let auth = format!("Bearer {}", s.host_token);
let r = if method == "GET" { agent(secs).get(&url).header("Authorization", &auth).call() } else { agent(secs).post(&url).header("Authorization", &auth).header("Content-Type", "application/octet-stream").send(body) }; let r = if method == "GET" { agent(secs).get(&url).header("Authorization", &auth).call() } else { agent(secs).post(&url).header("Authorization", &auth).header("Content-Type", "application/octet-stream").send(body) };
@ -1247,6 +1246,23 @@ pub fn sync_screen(s: &Settings, thumb: bool) -> Result<Vec<u8>, String> {
Ok(bytes) Ok(bytes)
} }
/// A link for this account's browser to watch the brain's computer (/screen on the account
/// server; one-time, it works for ten minutes).
pub fn screen_ticket(s: &Settings) -> Result<String, String> {
if s.relay_url.is_empty() || s.brain_id.is_empty() {
return Err("這台沒有連到帳號的大腦".into());
}
let url = format!("{}/relay/screen-ticket", s.relay_url.trim_end_matches('/'));
let mut r = agent(15)
.post(&url)
.header("Authorization", &format!("Bearer {}", s.host_token))
.header("Content-Type", "application/json")
.send(json!({"host": s.brain_id}).to_string().as_bytes())
.map_err(|e| format!("連不到帳號伺服器:{}", why_unreachable(&e)))?;
let v: Value = r.body_mut().read_json().map_err(|_| "帳號伺服器的回答看不懂".to_string())?;
v["ticket"].as_str().map(String::from).ok_or_else(|| "帳號伺服器沒有給看畫面的連結(它可能是舊版,請更新)".into())
}
/// A file the brain keeps (an attachment shown in a chat), as bytes. /// A file the brain keeps (an attachment shown in a chat), as bytes.
pub fn sync_file(s: &Settings, path: &str) -> Result<Vec<u8>, String> { pub fn sync_file(s: &Settings, path: &str) -> Result<Vec<u8>, String> {
let (st, bytes) = to_brain_raw(s, "GET", "/file", &[("path", path.to_string())], &[], 120)?; let (st, bytes) = to_brain_raw(s, "GET", "/file", &[("path", path.to_string())], &[], 120)?;
@ -1256,15 +1272,6 @@ pub fn sync_file(s: &Settings, path: &str) -> Result<Vec<u8>, String> {
Ok(bytes) Ok(bytes)
} }
fn url_escape(s: &str) -> String {
s.bytes()
.map(|b| match b {
b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'-' | b'_' | b'.' | b'~' => (b as char).to_string(),
_ => format!("%{b:02X}"),
})
.collect()
}
/// A notification on the person's phones (through the relay; phones that turned them on). /// A notification on the person's phones (through the relay; phones that turned them on).
pub fn notify(title: &str, body: &str, bot: u64) { pub fn notify(title: &str, body: &str, bot: u64) {
let s = hub().settings(); let s = hub().settings();
@ -1503,6 +1510,9 @@ pub fn devices_online(s: &Settings) -> Vec<(String, String, bool)> {
// ---------- the phone page ---------- // ---------- the phone page ----------
const PAGE: &str = include_str!("../mobile/index.html"); const PAGE: &str = include_str!("../mobile/index.html");
/// The bots' computer, watched and used in a browser (noVNC; see vnc.rs).
const SCREEN_PAGE: &str = include_str!("../mobile/screen.html");
const NOVNC: &str = include_str!("../mobile/novnc.js");
pub const MANIFEST: &str = r##"{"name":"AnyBot","short_name":"AnyBot","start_url":"./","display":"standalone","background_color":"#121212","theme_color":"#121212","icons":[{"src":"icon.svg","sizes":"any","type":"image/svg+xml"},{"src":"icon-192.png","sizes":"192x192","type":"image/png","purpose":"any maskable"},{"src":"icon-512.png","sizes":"512x512","type":"image/png","purpose":"any maskable"}]}"##; pub const MANIFEST: &str = r##"{"name":"AnyBot","short_name":"AnyBot","start_url":"./","display":"standalone","background_color":"#121212","theme_color":"#121212","icons":[{"src":"icon.svg","sizes":"any","type":"image/svg+xml"},{"src":"icon-192.png","sizes":"192x192","type":"image/png","purpose":"any maskable"},{"src":"icon-512.png","sizes":"512x512","type":"image/png","purpose":"any maskable"}]}"##;
/// The app's icon, the same drawing everywhere (icons/anybot.svg is the Mac/Linux one). /// The app's icon, the same drawing everywhere (icons/anybot.svg is the Mac/Linux one).
pub const ICON: &str = include_str!("../mobile/icon.svg"); pub const ICON: &str = include_str!("../mobile/icon.svg");

View File

@ -24,16 +24,12 @@ fn agent() -> ureq::Agent {
} }
fn form(url: &str, fields: &[(&str, &str)]) -> Result<Value, String> { fn form(url: &str, fields: &[(&str, &str)]) -> Result<Value, String> {
let body: String = fields.iter().map(|(k, v)| format!("{}={}", k, urlencode(v))).collect::<Vec<_>>().join("&"); let body: String = fields.iter().map(|(k, v)| format!("{}={}", k, crate::httpd::url_encode(v))).collect::<Vec<_>>().join("&");
let mut resp = agent().post(url).header("Content-Type", "application/x-www-form-urlencoded").send(body.as_bytes()).map_err(|e| e.to_string())?; let mut resp = agent().post(url).header("Content-Type", "application/x-www-form-urlencoded").send(body.as_bytes()).map_err(|e| e.to_string())?;
let text = resp.body_mut().read_to_string().unwrap_or_default(); let text = resp.body_mut().read_to_string().unwrap_or_default();
serde_json::from_str(&text).map_err(|_| text.chars().take(200).collect()) serde_json::from_str(&text).map_err(|_| text.chars().take(200).collect())
} }
pub fn urlencode(s: &str) -> String {
s.bytes().map(|b| if b.is_ascii_alphanumeric() || b"-_.~".contains(&b) { (b as char).to_string() } else { format!("%{b:02X}") }).collect()
}
/// Step 1: ask for a code. Returns (user code, address, device code, poll seconds). /// Step 1: ask for a code. Returns (user code, address, device code, poll seconds).
pub fn device_code(client_id: &str, tenant: &str) -> Result<(String, String, String, u64), String> { pub fn device_code(client_id: &str, tenant: &str) -> Result<(String, String, String, u64), String> {
let v = form(&format!("{}/{}/oauth2/v2.0/devicecode", login_base(), tenant_or_default(tenant)), &[("client_id", client_id.trim()), ("scope", SCOPES)])?; let v = form(&format!("{}/{}/oauth2/v2.0/devicecode", login_base(), tenant_or_default(tenant)), &[("client_id", client_id.trim()), ("scope", SCOPES)])?;
@ -266,6 +262,6 @@ mod tests {
let (cur, items) = new_messages(&v, "2026-11-04T01:00:01Z", "ME"); let (cur, items) = new_messages(&v, "2026-11-04T01:00:01Z", "ME");
assert_eq!(cur, "2026-11-04T01:00:03Z"); assert_eq!(cur, "2026-11-04T01:00:03Z");
assert_eq!(items, vec![("小美".to_string(), "週報 好了嗎?".to_string(), "2".to_string())]); assert_eq!(items, vec![("小美".to_string(), "週報 好了嗎?".to_string(), "2".to_string())]);
assert_eq!(urlencode("a b/c"), "a%20b%2Fc"); assert_eq!(crate::httpd::url_encode("a b/c"), "a%20b%2Fc");
} }
} }

View File

@ -6,8 +6,6 @@ use eframe::egui::Color32;
#[derive(Clone, Copy)] #[derive(Clone, Copy)]
pub struct Palette { pub struct Palette {
pub pane: Color32, pub pane: Color32,
#[allow(dead_code)]
pub chrome: Color32,
pub sidebar: Color32, pub sidebar: Color32,
pub elevated: Color32, pub elevated: Color32,
pub hover: Color32, pub hover: Color32,
@ -18,8 +16,6 @@ pub struct Palette {
pub text3: Color32, pub text3: Color32,
pub on_primary: Color32, pub on_primary: Color32,
pub bubble_user: Color32, pub bubble_user: Color32,
#[allow(dead_code)]
pub bubble_agent: Color32,
pub border: Color32, pub border: Color32,
pub focus: Color32, pub focus: Color32,
pub accent: Color32, pub accent: Color32,
@ -32,7 +28,6 @@ const fn hexa(r: u8, g: u8, b: u8, a: u8) -> Color32 {
pub const LIGHT: Palette = Palette { pub const LIGHT: Palette = Palette {
pane: hexa(0xff, 0xff, 0xff, 0xff), pane: hexa(0xff, 0xff, 0xff, 0xff),
chrome: hexa(0xfc, 0xfc, 0xfc, 0xff),
sidebar: hexa(0xf5, 0xf5, 0xf5, 0xff), sidebar: hexa(0xf5, 0xf5, 0xf5, 0xff),
elevated: hexa(0xff, 0xff, 0xff, 0xff), elevated: hexa(0xff, 0xff, 0xff, 0xff),
hover: hexa(0x1c, 0x1c, 0x1c, 0x0f), hover: hexa(0x1c, 0x1c, 0x1c, 0x0f),
@ -43,7 +38,6 @@ pub const LIGHT: Palette = Palette {
text3: hexa(0x1c, 0x1c, 0x1c, 0x8f), text3: hexa(0x1c, 0x1c, 0x1c, 0x8f),
on_primary: hexa(0xff, 0xff, 0xff, 0xff), on_primary: hexa(0xff, 0xff, 0xff, 0xff),
bubble_user: hexa(0x1c, 0x1c, 0x1c, 0xff), bubble_user: hexa(0x1c, 0x1c, 0x1c, 0xff),
bubble_agent: hexa(0x1c, 0x1c, 0x1c, 0x0f),
border: hexa(0x1c, 0x1c, 0x1c, 0x1f), border: hexa(0x1c, 0x1c, 0x1c, 0x1f),
focus: hexa(0x1e, 0x86, 0xfc, 0xff), focus: hexa(0x1e, 0x86, 0xfc, 0xff),
accent: hexa(0x16, 0x80, 0xf3, 0xff), accent: hexa(0x16, 0x80, 0xf3, 0xff),
@ -57,7 +51,6 @@ pub const LIGHT: Palette = Palette {
// Measured from the original's dark window: chat rgb 19, sidebar 16, bubbles/cards 31–33. // Measured from the original's dark window: chat rgb 19, sidebar 16, bubbles/cards 31–33.
pub const DARK: Palette = Palette { pub const DARK: Palette = Palette {
pane: hexa(0x13, 0x13, 0x13, 0xff), pane: hexa(0x13, 0x13, 0x13, 0xff),
chrome: hexa(0x13, 0x13, 0x13, 0xff),
sidebar: hexa(0x10, 0x10, 0x10, 0xff), sidebar: hexa(0x10, 0x10, 0x10, 0xff),
elevated: hexa(0x1f, 0x1f, 0x1f, 0xff), elevated: hexa(0x1f, 0x1f, 0x1f, 0xff),
hover: hexa(0xf5, 0xf5, 0xf5, 0x0f), hover: hexa(0xf5, 0xf5, 0xf5, 0x0f),
@ -68,7 +61,6 @@ pub const DARK: Palette = Palette {
text3: hexa(0xf5, 0xf5, 0xf5, 0x8a), text3: hexa(0xf5, 0xf5, 0xf5, 0x8a),
on_primary: hexa(0x1f, 0x1f, 0x1f, 0xff), on_primary: hexa(0x1f, 0x1f, 0x1f, 0xff),
bubble_user: hexa(0xf5, 0xf5, 0xf5, 0x2e), bubble_user: hexa(0xf5, 0xf5, 0xf5, 0x2e),
bubble_agent: hexa(0xf5, 0xf5, 0xf5, 0x0a),
border: hexa(0xf5, 0xf5, 0xf5, 0x1a), border: hexa(0xf5, 0xf5, 0xf5, 0x1a),
focus: hexa(0x18, 0x84, 0xfc, 0xff), focus: hexa(0x18, 0x84, 0xfc, 0xff),
accent: hexa(0x18, 0x84, 0xfc, 0xff), accent: hexa(0x18, 0x84, 0xfc, 0xff),

View File

@ -90,8 +90,9 @@ pub fn label(name: &str) -> &'static str {
} }
} }
/// A tool call's arguments (none = an empty object).
pub fn args_of(args: &str) -> Value { pub fn args_of(args: &str) -> Value {
serde_json::from_str(args).unwrap_or(Value::Null) serde_json::from_str(if args.trim().is_empty() { "{}" } else { args }).unwrap_or(Value::Null)
} }
fn arg_str(v: &Value, k: &str) -> String { fn arg_str(v: &Value, k: &str) -> String {

View File

@ -51,7 +51,10 @@ pub fn zip(b: &[u8]) -> Vec<u8> {
/// The body to send and whether it is compressed: only when that saves a good part of it /// The body to send and whether it is compressed: only when that saves a good part of it
/// (pictures and PDFs are compressed already). /// (pictures and PDFs are compressed already).
fn maybe_zip(b: Vec<u8>) -> (Vec<u8>, bool) { fn maybe_zip(b: Vec<u8>) -> (Vec<u8>, bool) {
if b.len() <= ZIP_OVER { // JPEG, PNG, GIF, WebP, PDF, zip/docx, gzip: compressed already, not worth a try (the live
// picture is a JPEG several times a second).
const PACKED: [&[u8]; 7] = [b"\xff\xd8\xff", b"\x89PNG", b"GIF8", b"RIFF", b"%PDF", b"PK\x03\x04", b"\x1f\x8b"];
if b.len() <= ZIP_OVER || PACKED.iter().any(|m| b.starts_with(m)) {
return (b, false); return (b, false);
} }
let z = zip(&b); let z = zip(&b);
@ -226,7 +229,7 @@ pub fn run(url: &str, token: &str, alive: &dyn Fn() -> bool, status: &dyn Fn(&st
} }
fn keep(mut ws: WebSocket<Stream>, serial: u64, alive: &dyn Fn() -> bool) { fn keep(mut ws: WebSocket<Stream>, serial: u64, alive: &dyn Fn() -> bool) {
let _ = ws.get_ref().tcp().set_read_timeout(Some(Duration::from_millis(40))); let mut pace = wire::Pace::new();
let (tx, rx) = channel::<Vec<u8>>(); let (tx, rx) = channel::<Vec<u8>>();
let calls: Calls = Arc::default(); let calls: Calls = Arc::default();
*link().lock().unwrap() = Some(Link { tx: tx.clone(), calls: calls.clone(), serial }); *link().lock().unwrap() = Some(Link { tx: tx.clone(), calls: calls.clone(), serial });
@ -234,9 +237,11 @@ fn keep(mut ws: WebSocket<Stream>, serial: u64, alive: &dyn Fn() -> bool) {
let _ = tx.send(wire::pack(&json!({"t": "hello", "brain": s.brain_id, "hands": s.device_link}), &[])); let _ = tx.send(wire::pack(&json!({"t": "hello", "brain": s.brain_id, "hands": s.device_link}), &[]));
let (mut heard, mut pinged) = (Instant::now(), Instant::now()); let (mut heard, mut pinged) = (Instant::now(), Instant::now());
'live: while alive() { 'live: while alive() {
pace.tune(ws.get_ref().tcp());
match ws.read() { match ws.read() {
Ok(Message::Binary(b)) => { Ok(Message::Binary(b)) => {
heard = Instant::now(); heard = Instant::now();
pace.moved();
if let Some((head, body)) = wire::unpack(&b) { if let Some((head, body)) = wire::unpack(&b) {
// The relay says this machine was unregistered (signed out): stop for good. // The relay says this machine was unregistered (signed out): stop for good.
if head["t"] == "bye" { if head["t"] == "bye" {
@ -251,6 +256,7 @@ fn keep(mut ws: WebSocket<Stream>, serial: u64, alive: &dyn Fn() -> bool) {
Err(_) => break, Err(_) => break,
} }
while let Ok(f) = rx.try_recv() { while let Ok(f) = rx.try_recv() {
pace.moved();
if ws.write(Message::Binary(f.into())).is_err() { if ws.write(Message::Binary(f.into())).is_err() {
break 'live; break 'live;
} }
@ -276,14 +282,72 @@ fn keep(mut ws: WebSocket<Stream>, serial: u64, alive: &dyn Fn() -> bool) {
*l = None; *l = None;
} }
drop(l); drop(l);
streams().lock().unwrap().clear();
// Whoever waits on an answer hears now, not after a timeout. // Whoever waits on an answer hears now, not after a timeout.
for (_, w) in calls.lock().unwrap().drain() { for (_, w) in calls.lock().unwrap().drain() {
let _ = w.send(Err(CUT.into())); let _ = w.send(Err(CUT.into()));
} }
} }
/// People watching the bots' computer through the account server: one VNC stream each, by the
/// relay's stream id (vopen / vdata / vtext / vclose).
fn streams() -> &'static Mutex<HashMap<String, crate::vnc::Session>> {
static S: OnceLock<Mutex<HashMap<String, crate::vnc::Session>>> = OnceLock::new();
S.get_or_init(Default::default)
}
fn on_stream(kind: &str, sid: String, body: Vec<u8>, tx: &Sender<Vec<u8>>) {
let send = {
let (tx, sid) = (tx.clone(), sid.clone());
move |b: &[u8]| tx.send(wire::pack(&json!({"t": "vdata", "sid": sid}), b)).is_ok()
};
let close = {
let (tx, sid) = (tx.clone(), sid.clone());
move |why: &str| {
streams().lock().unwrap().remove(&sid);
let _ = tx.send(wire::pack(&json!({"t": "vclose", "sid": sid, "why": why}), &[]));
}
};
match kind {
"vopen" => {
std::thread::spawn(move || {
let out = {
let (send, close) = (send.clone(), close.clone());
move |b: &[u8]| if b.is_empty() { close("這台電腦的畫面關了"); false } else { send(b) }
};
match crate::vnc::open(out) {
Ok(s) => {
streams().lock().unwrap().insert(sid, s);
// Only now: the viewer answers this at once.
if !send(crate::vnc::GREETING) {
close("");
}
}
Err(e) => close(&e),
}
});
}
"vdata" => {
let mut all = streams().lock().unwrap();
let ok = all.get_mut(&sid).map(|s| s.feed(&body, &send)).unwrap_or(false);
drop(all);
if !ok {
close("這台電腦的畫面關了");
}
}
"vtext" => crate::vnc::person_typed(&String::from_utf8_lossy(&body)),
_ => {
streams().lock().unwrap().remove(&sid);
}
}
}
fn on_frame(head: Value, body: Vec<u8>, tx: &Sender<Vec<u8>>, calls: &Calls) { fn on_frame(head: Value, body: Vec<u8>, tx: &Sender<Vec<u8>>, calls: &Calls) {
match head["t"].as_str().unwrap_or("") { let t = head["t"].as_str().unwrap_or("");
if matches!(t, "vopen" | "vdata" | "vtext" | "vclose") {
return on_stream(t, head["sid"].as_str().unwrap_or("").to_string(), body, tx);
}
match t {
"job" => { "job" => {
let tx = tx.clone(); let tx = tx.clone();
std::thread::spawn(move || { std::thread::spawn(move || {

View File

@ -203,29 +203,6 @@ pub fn find(host: &str) -> Option<Login> {
None None
} }
/// Save a login in the macOS Keychain (the person types it in AnyBot's settings).
/// (Kept for the live tests and older callers.)
#[cfg_attr(not(test), allow(dead_code))]
pub fn save(site: &str, username: &str, password: &str) -> Result<(), String> {
let host = host_of(site);
let host = if host.is_empty() { site.trim().to_lowercase() } else { host };
if host.is_empty() || password.is_empty() {
return Err("請填網站和密碼".into());
}
if !cfg!(target_os = "macos") {
let label = format!("{LABEL}:{host}");
let o = secret_tool(&["store", "--label", &label, "application", "anybot", "server", &host, "user", username], Some(password))
.ok_or("找不到 secret-tool:請安裝 libsecret-tools(系統鑰匙圈的指令)。")?;
return if o.status.success() { Ok(()) } else { Err(format!("存不進系統鑰匙圈:{}", String::from_utf8_lossy(&o.stderr).trim())) };
}
let out = Command::new("security").args(["add-internet-password", "-U", "-s", &host, "-a", username, "-l", LABEL, "-w", password]).output().map_err(|e| e.to_string())?;
if out.status.success() {
index_add(&host, username);
Ok(())
} else {
Err(String::from_utf8_lossy(&out.stderr).trim().to_string())
}
}
/// AnyBot's list of the logins it saved: site and account only (the passwords stay in the /// AnyBot's list of the logins it saved: site and account only (the passwords stay in the
/// keychain). macOS prints a non-ASCII label like 「AnyBot 登入」 as hex bytes, so the keychain /// keychain). macOS prints a non-ASCII label like 「AnyBot 登入」 as hex bytes, so the keychain
@ -254,15 +231,6 @@ fn write_index(list: &[(String, String)]) {
let _ = std::fs::write(index_path(), serde_json::to_string_pretty(list).unwrap_or_default()); let _ = std::fs::write(index_path(), serde_json::to_string_pretty(list).unwrap_or_default());
} }
/// (Kept for the live tests and older callers.)
#[cfg_attr(not(test), allow(dead_code))]
fn index_add(site: &str, account: &str) {
let mut list = read_index().unwrap_or_default();
if !list.iter().any(|(s, a)| s == site && a == account) {
list.push((site.to_string(), account.to_string()));
write_index(&list);
}
}
/// The keychain has this login (no password is read, so no prompt). /// The keychain has this login (no password is read, so no prompt).
fn keychain_has(site: &str, account: &str) -> bool { fn keychain_has(site: &str, account: &str) -> bool {
@ -305,19 +273,6 @@ pub fn saved_sites() -> Vec<(String, String)> {
kept kept
} }
/// (Kept for the live tests and older callers.)
#[cfg_attr(not(test), allow(dead_code))]
pub fn remove(site: &str, account: &str) {
if !cfg!(target_os = "macos") {
let _ = secret_tool(&["clear", "application", "anybot", "server", site, "user", account], None);
return;
}
let _ = Command::new("security").args(["delete-internet-password", "-s", site, "-a", account]).output();
if let Some(mut list) = read_index() {
list.retain(|(s, a)| !(s == site && a == account));
write_index(&list);
}
}
/// The JavaScript that types the login into the page (as a JSON-encoded string literal). /// The JavaScript that types the login into the page (as a JSON-encoded string literal).
pub fn fill_script(username: &str, password: &str, submit: bool) -> String { pub fn fill_script(username: &str, password: &str, submit: bool) -> String {
@ -355,9 +310,6 @@ pub fn fill_script(username: &str, password: &str, submit: bool) -> String {
) )
} }
fn args_of(args: &str) -> Value {
serde_json::from_str(if args.trim().is_empty() { "{}" } else { args }).unwrap_or_default()
}
/// The current tab's host in AnyBot's browser. /// The current tab's host in AnyBot's browser.
fn current_host() -> Result<String, String> { fn current_host() -> Result<String, String> {
@ -393,7 +345,7 @@ impl LoginPlan {
} }
pub fn plan_login(args: &str) -> Result<LoginPlan, String> { pub fn plan_login(args: &str) -> Result<LoginPlan, String> {
let v = args_of(args); let v = crate::tools::args_of(args);
let tab = current_host()?; let tab = current_host()?;
let host = v["site"].as_str().filter(|s| !s.trim().is_empty()).map(host_of).filter(|h| !h.is_empty()).unwrap_or(tab); let host = v["site"].as_str().filter(|s| !s.trim().is_empty()).map(host_of).filter(|h| !h.is_empty()).unwrap_or(tab);
if let Some(id) = v["login_id"].as_str().filter(|s| !s.is_empty()) { if let Some(id) = v["login_id"].as_str().filter(|s| !s.is_empty()) {
@ -532,13 +484,13 @@ fn filled_summary(r: &Value) -> (Vec<&'static str>, u64) {
/// For the approval question: (current host, card label) without secrets. /// For the approval question: (current host, card label) without secrets.
pub fn plan_card(args: &str) -> Result<(String, crate::vault_store::Item), String> { pub fn plan_card(args: &str) -> Result<(String, crate::vault_store::Item), String> {
let id = args_of(args)["card_id"].as_str().unwrap_or("").to_string(); let id = crate::tools::args_of(args)["card_id"].as_str().unwrap_or("").to_string();
let card = crate::vault_store::get(&id).filter(|i| i.kind == crate::vault_store::Kind::Card).ok_or("保險庫裡沒有這張卡片(先用 vault_list 看 id)")?; let card = crate::vault_store::get(&id).filter(|i| i.kind == crate::vault_store::Kind::Card).ok_or("保險庫裡沒有這張卡片(先用 vault_list 看 id)")?;
Ok((current_host()?, card)) Ok((current_host()?, card))
} }
pub fn fill_card(args: &str) -> Result<String, String> { pub fn fill_card(args: &str) -> Result<String, String> {
let id = args_of(args)["card_id"].as_str().unwrap_or("").to_string(); let id = crate::tools::args_of(args)["card_id"].as_str().unwrap_or("").to_string();
let c = crate::vault_store::card_secret(&id).ok_or("保險庫裡沒有這張卡片")?; let c = crate::vault_store::card_secret(&id).ok_or("保險庫裡沒有這張卡片")?;
let values = serde_json::json!({ let values = serde_json::json!({
"cc-number": c.number, "cc-number": c.number,
@ -562,7 +514,7 @@ pub fn fill_card(args: &str) -> Result<String, String> {
} }
pub fn plan_contact(args: &str) -> Result<(String, crate::vault_store::Item), String> { pub fn plan_contact(args: &str) -> Result<(String, crate::vault_store::Item), String> {
let id = args_of(args)["contact_id"].as_str().unwrap_or("").to_string(); let id = crate::tools::args_of(args)["contact_id"].as_str().unwrap_or("").to_string();
let c = crate::vault_store::get(&id).filter(|i| i.kind == crate::vault_store::Kind::Contact).ok_or("保險庫裡沒有這筆聯絡資訊(先用 vault_list 看 id)")?; let c = crate::vault_store::get(&id).filter(|i| i.kind == crate::vault_store::Kind::Contact).ok_or("保險庫裡沒有這筆聯絡資訊(先用 vault_list 看 id)")?;
Ok((current_host()?, c)) Ok((current_host()?, c))
} }
@ -615,21 +567,6 @@ mod tests {
assert!(js.contains("input[type=password]") && js.contains("if (true && (pw || user))")); assert!(js.contains("input[type=password]") && js.contains("if (true && (pw || user))"));
} }
/// Needs a keyring: on Linux run under `dbus-run-session` with gnome-keyring unlocked.
#[test]
#[ignore]
fn keyring_live() {
save("https://login.example.org/x", "me@example.org", "s3cret pass").unwrap();
let l = find("login.example.org").expect("found");
assert_eq!((l.username.as_str(), l.password.as_str()), ("me@example.org", "s3cret pass"));
assert!(saved_sites().iter().any(|(s, a)| s == "login.example.org" && a == "me@example.org"));
// Logins from before the list existed (label printed as hex by `security`) show up too.
if cfg!(target_os = "macos") {
assert!(saved_sites().iter().any(|(s, _)| s == "legacy.example.org"), "legacy entry: {:?}", saved_sites());
}
remove("login.example.org", "me@example.org");
assert!(find("login.example.org").is_none());
}
/// The whole bot-side flow on this computer: local pages in AnyBot's browser (a throwaway /// The whole bot-side flow on this computer: local pages in AnyBot's browser (a throwaway
/// profile), items in a scratch vault, and the fill tools. Run with ANYBOT_DATA_DIR set to a /// profile), items in a scratch vault, and the fill tools. Run with ANYBOT_DATA_DIR set to a

View File

@ -790,8 +790,6 @@ pub enum RowStatus {
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
pub struct ImportRow { pub struct ImportRow {
#[allow(dead_code)]
pub line: usize,
pub kind: Kind, pub kind: Kind,
pub label: String, pub label: String,
pub secondary: String, pub secondary: String,
@ -864,7 +862,7 @@ pub fn plan_import(text: &str) -> Result<ImportPlan, String> {
if format == CsvFormat::LastPass && url.trim() == "http://sn" { if format == CsvFormat::LastPass && url.trim() == "http://sn" {
let extra = get(r, c_notes); let extra = get(r, c_notes);
if !extra.starts_with("NoteType:Credit Card") { if !extra.starts_with("NoteType:Credit Card") {
out.push(ImportRow { line, kind: Kind::Login, label: if title.is_empty() { format!("第 {line} 列") } else { title }, secondary: String::new(), status: RowStatus::Skipped("不是登入資訊或卡片"), site: String::new(), username: String::new(), password: String::new(), notes: String::new(), card: None }); out.push(ImportRow { kind: Kind::Login, label: if title.is_empty() { format!("第 {line} 列") } else { title }, secondary: String::new(), status: RowStatus::Skipped("不是登入資訊或卡片"), site: String::new(), username: String::new(), password: String::new(), notes: String::new(), card: None });
continue; continue;
} }
let f = lastpass_fields(&extra); let f = lastpass_fields(&extra);
@ -899,7 +897,6 @@ pub fn plan_import(text: &str) -> Result<ImportPlan, String> {
}; };
let expiry = exp.map(|(m, y)| format!("{m:02}/{}", y % 100)).unwrap_or_default(); let expiry = exp.map(|(m, y)| format!("{m:02}/{}", y % 100)).unwrap_or_default();
out.push(ImportRow { out.push(ImportRow {
line,
kind: Kind::Card, kind: Kind::Card,
label: if title.is_empty() { card_brand(&number).to_string() } else { title.clone() }, label: if title.is_empty() { card_brand(&number).to_string() } else { title.clone() },
secondary: format!("•••• {last4}"), secondary: format!("•••• {last4}"),
@ -940,7 +937,7 @@ pub fn plan_import(text: &str) -> Result<ImportPlan, String> {
RowStatus::Ready RowStatus::Ready
} }
}; };
out.push(ImportRow { line, kind: Kind::Login, label, secondary: if user.is_empty() { "••••••••".into() } else { format!("{user} · ••••••••") }, status, site: host, username: user, password: pw, notes, card: None }); out.push(ImportRow { kind: Kind::Login, label, secondary: if user.is_empty() { "••••••••".into() } else { format!("{user} · ••••••••") }, status, site: host, username: user, password: pw, notes, card: None });
} }
Ok(ImportPlan { format, rows: out }) Ok(ImportPlan { format, rows: out })
} }

263
src/vnc.rs Normal file
View File

@ -0,0 +1,263 @@
// People watching and helping on the bots' computer: VNC, carried to them as a stream.
//
// The box already runs a VNC server (x11vnc, inside the container, with a password). A person's
// viewer (noVNC in a browser: the phone, or any computer through the account server) speaks VNC
// to this proxy, which reaches the box's server through one `docker exec` (a few lines of
// Python in the box joining its stdin/stdout to 127.0.0.1:5900). VNC sends only what changed on
// the screen and every mouse move and key as it happens, so watching is live and helping is
// like sitting at it — unlike pictures sent one by one.
//
// The proxy signs in to the box with its password itself and offers the viewer "no password",
// so the password never leaves this AnyBot; whoever reaches the proxy was already let in (an
// account session or one-time ticket at the account server, or the phone's password here).
// While the person clicks or types, the bot's own steps wait (HELD_BY_PERSON), and go on a
// little after they stop (person_active).
use std::io::{Read, Write};
use std::process::{Child, ChildStdin};
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Mutex;
use std::time::{Duration, Instant};
/// The bot waits this long after the person's last click or key before it goes on.
const HELD_AFTER_INPUT: Duration = Duration::from_secs(8);
/// Joins the box's VNC server to stdin/stdout, and ends when either side does (a loop left in
/// the box would hold the connection open).
const BRIDGE: &str = r#"import os,select,socket
s=socket.create_connection(('127.0.0.1',5900))
while True:
r=select.select([s,0],[],[])[0]
if s in r:
d=s.recv(65536)
if not d: break
os.write(1,d)
if 0 in r:
d=os.read(0,65536)
if not d: break
s.sendall(d)
"#;
/// Viewers connected now (the box stays awake while there is one).
static OPEN: AtomicUsize = AtomicUsize::new(0);
/// When the person last clicked or typed through a viewer.
static LAST_INPUT: Mutex<Option<Instant>> = Mutex::new(None);
/// VNC authentication: the 16-byte challenge encrypted with DES, the key being the password's
/// first 8 bytes with each byte's bits reversed (RFC 6143 §7.2.2, as every VNC server does it).
pub fn vnc_response(password: &str, challenge: &[u8; 16]) -> [u8; 16] {
use des::cipher::{BlockEncrypt, KeyInit};
let mut key = [0u8; 8];
for (k, b) in key.iter_mut().zip(password.bytes()) {
*k = b.reverse_bits();
}
let c = des::Des::new_from_slice(&key).expect("8-byte key");
let mut out = *challenge;
for block in out.chunks_mut(8) {
c.encrypt_block(des::cipher::generic_array::GenericArray::from_mut_slice(block));
}
out
}
/// This proxy's first words to a viewer, as a VNC server: version 3.8 (then "no password"; see
/// Session::feed).
pub const GREETING: &[u8] = b"RFB 003.008\n";
/// One person's viewer joined to the box.
pub struct Session {
child: Child,
stdin: ChildStdin,
/// The viewer's side of the handshake still to read: its version (12 bytes) and its choice
/// of security type (1 byte).
greeting: usize,
/// Viewer → box messages not yet complete (to notice clicks and keys).
pending: Vec<u8>,
sniffing: bool,
/// The viewer's ClientInit (one byte) is still to come.
init: bool,
}
/// Open a viewer's way to the box: sign in to its VNC server, and start sending what the server
/// says to `out` (from a thread of its own; `out` returns false when the viewer is gone, and
/// gets an empty slice when the box's side ended). The caller then sends the viewer GREETING
/// (once it is ready for the viewer's answer).
pub fn open(out: impl Fn(&[u8]) -> bool + Send + 'static) -> Result<Session, String> {
crate::cloudbox::ensure_running()?;
let mut child = crate::cloudbox::exec_piped(&["python3", "-c", BRIDGE])?;
let mut stdin = child.stdin.take().ok_or("連不上這台電腦的畫面")?;
let mut stdout = child.stdout.take().ok_or("連不上這台電腦的畫面")?;
let fail = |child: &mut Child, why: &str| -> String {
let _ = child.kill();
let _ = child.wait();
format!("連不上這台電腦的畫面:{why}")
};
let mut read = |n: usize| -> Result<Vec<u8>, String> {
let mut b = vec![0u8; n];
stdout.read_exact(&mut b).map(|_| b).map_err(|_| "畫面伺服器沒有回應".to_string())
};
// The box's side, as a VNC client: version 3.8, then its password.
let signed_in = (|| -> Result<(), String> {
let v = read(12)?;
if !v.starts_with(b"RFB 003.") {
return Err("不是 VNC 伺服器".into());
}
stdin.write_all(b"RFB 003.008\n").map_err(|e| e.to_string())?;
let n = read(1)?[0] as usize;
if n == 0 {
return Err("畫面伺服器拒絕連線".into());
}
let types = read(n)?;
if types.contains(&2) {
stdin.write_all(&[2]).map_err(|e| e.to_string())?;
let c: [u8; 16] = read(16)?.try_into().map_err(|_| "認證失敗".to_string())?;
stdin.write_all(&vnc_response(&crate::cloudbox::box_vnc_password(), &c)).map_err(|e| e.to_string())?;
} else if types.contains(&1) {
stdin.write_all(&[1]).map_err(|e| e.to_string())?;
} else {
return Err("畫面伺服器要的登入方式不支援".into());
}
if read(4)? != [0, 0, 0, 0] {
return Err("畫面伺服器的密碼不對".into());
}
Ok(())
})();
if let Err(e) = signed_in {
return Err(fail(&mut child, &e));
}
OPEN.fetch_add(1, Ordering::SeqCst);
crate::cloudbox::set_viewer_open(true);
std::thread::spawn(move || {
let mut buf = vec![0u8; 64 * 1024];
loop {
match stdout.read(&mut buf) {
Ok(0) | Err(_) => break,
Ok(n) => {
if !out(&buf[..n]) {
return;
}
}
}
}
// The box's side ended: an empty send says so.
out(&[]);
});
Ok(Session { child, stdin, greeting: 13, pending: Vec::new(), sniffing: true, init: true })
}
impl Session {
/// Bytes from the viewer. False when the box's side is gone.
pub fn feed(&mut self, mut b: &[u8], reply: impl Fn(&[u8]) -> bool) -> bool {
if self.greeting > 0 {
let take = self.greeting.min(b.len());
let before = self.greeting;
self.greeting -= take;
b = &b[take..];
// After the version (12 bytes): the one security type, "None".
if before > 1 && self.greeting <= 1 && !reply(&[1, 1]) {
return false;
}
// After the choice: OK.
if self.greeting == 0 && !reply(&[0, 0, 0, 0]) {
return false;
}
if b.is_empty() {
return true;
}
}
self.notice_input(b);
self.stdin.write_all(b).and_then(|_| self.stdin.flush()).is_ok()
}
/// Follow the viewer's messages (RFC 6143 §7.5) to see clicks and keys. Anything it doesn't
/// know stops the watching, never the forwarding.
fn notice_input(&mut self, b: &[u8]) {
if !self.sniffing {
return;
}
self.pending.extend_from_slice(b);
// The first byte after the handshake is ClientInit (the shared flag).
if self.init && !self.pending.is_empty() {
self.pending.remove(0);
self.init = false;
}
let mut at = 0;
loop {
let rest = &self.pending[at..];
let Some(&kind) = rest.first() else { break };
let len = match kind {
0 => 20,
2 if rest.len() >= 4 => 4 + 4 * u16::from_be_bytes([rest[2], rest[3]]) as usize,
3 => 10,
4 => 8,
5 => 6,
6 if rest.len() >= 8 => 8 + u32::from_be_bytes([rest[4], rest[5], rest[6], rest[7]]) as usize,
2 | 6 => break,
_ => {
self.sniffing = false;
self.pending.clear();
return;
}
};
if rest.len() < len {
break;
}
match kind {
4 if rest[1] == 1 => person_acted(),
5 if rest[1] != 0 => person_acted(),
_ => {}
}
at += len;
}
self.pending.drain(..at);
}
}
impl Drop for Session {
fn drop(&mut self) {
let _ = self.child.kill();
let _ = self.child.wait();
if OPEN.fetch_sub(1, Ordering::SeqCst) == 1 {
crate::cloudbox::set_viewer_open(false);
}
}
}
/// The person clicked or typed: the bot's steps wait until a little after they stop.
pub fn person_acted() {
*LAST_INPUT.lock().unwrap_or_else(|e| e.into_inner()) = Some(Instant::now());
}
/// The person is using the box through a viewer right now (input in the last few seconds).
pub fn person_active() -> bool {
LAST_INPUT.lock().unwrap_or_else(|e| e.into_inner()).map(|t| t.elapsed() < HELD_AFTER_INPUT).unwrap_or(false)
}
/// Text the person typed in the viewer's own box (any language): pasted into the box.
pub fn person_typed(text: &str) {
if !text.is_empty() {
person_acted();
crate::cloudbox::viewer_text(text);
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn the_bridge_keeps_its_indentation() {
// Python needs its indentation (a Rust "\" line continuation would strip it).
assert!(BRIDGE.contains("\n r=select") && BRIDGE.contains("\n if not d: break"));
}
#[test]
fn vnc_password_answer_matches_the_standard() {
// Password "password", challenge 00..0f: DES-ECB with the bit-reversed key 0e86ceceeef64e26
// (worked out with `openssl enc -des-ecb -nopad`).
let c: [u8; 16] = core::array::from_fn(|i| i as u8);
let want = [0xb8, 0x66, 0x92, 0x41, 0x25, 0xc8, 0xee, 0xbb, 0x9d, 0xeb, 0xc1, 0xdb, 0x61, 0xc5, 0x38, 0xe2];
assert_eq!(vnc_response("password", &c), want);
// Only the first 8 bytes of a password count.
assert_eq!(vnc_response("password-and-more", &c), want);
}
}

View File

@ -196,11 +196,6 @@ impl Recording {
} }
} }
/// multipart/form-data body for /audio/transcriptions.
#[allow(dead_code)]
pub fn multipart(model: &str, file_name: &str, audio: &[u8], boundary: &str) -> Vec<u8> {
multipart_lang(model, file_name, audio, boundary, "")
}
/// Same, telling the service the spoken language (ISO 639-1, e.g. "zh"; empty = detect). /// Same, telling the service the spoken language (ISO 639-1, e.g. "zh"; empty = detect).
pub fn multipart_lang(model: &str, file_name: &str, audio: &[u8], boundary: &str, lang: &str) -> Vec<u8> { pub fn multipart_lang(model: &str, file_name: &str, audio: &[u8], boundary: &str, lang: &str) -> Vec<u8> {
@ -219,11 +214,6 @@ pub fn multipart_lang(model: &str, file_name: &str, audio: &[u8], boundary: &str
b b
} }
/// Audio → text through `{base}/audio/transcriptions`.
#[allow(dead_code)]
pub fn transcribe(base_url: &str, api_key: &str, file: &std::path::Path) -> Result<String, String> {
transcribe_lang(base_url, api_key, file, "")
}
/// The spoken languages offered for voice chat (the original: 語言 Auto-detect…): (code, name). /// The spoken languages offered for voice chat (the original: 語言 Auto-detect…): (code, name).
pub const SPOKEN: [(&str, &str); 6] = [("", "自動偵測"), ("zh", "中文"), ("en", "English"), ("ja", "日本語"), ("ko", "한국어"), ("es", "Español")]; pub const SPOKEN: [(&str, &str); 6] = [("", "自動偵測"), ("zh", "中文"), ("en", "English"), ("ja", "日本語"), ("ko", "한국어"), ("es", "Español")];
@ -364,11 +354,6 @@ pub fn voice_for(text: &str) -> Option<&'static str> {
text.chars().any(|c| ('\u{4e00}'..='\u{9fff}').contains(&c)).then_some(if cfg!(target_os = "macos") { "Meijia" } else { "cmn" }) text.chars().any(|c| ('\u{4e00}'..='\u{9fff}').contains(&c)).then_some(if cfg!(target_os = "macos") { "Meijia" } else { "cmn" })
} }
/// Read an answer aloud (stops whatever was being read).
#[allow(dead_code)]
pub fn speak(text: &str) {
speak_as(text, "", 1.0);
}
/// Words per minute of the system voices at 1x (`say`'s usual rate). /// Words per minute of the system voices at 1x (`say`'s usual rate).
pub const BASE_RATE: f32 = 175.0; pub const BASE_RATE: f32 = 175.0;
@ -461,12 +446,12 @@ mod tests {
}); });
let f = std::env::temp_dir().join(format!("anybot-voice-test-{}.m4a", std::process::id())); let f = std::env::temp_dir().join(format!("anybot-voice-test-{}.m4a", std::process::id()));
std::fs::write(&f, vec![7u8; 2048]).unwrap(); std::fs::write(&f, vec![7u8; 2048]).unwrap();
let text = transcribe(&format!("http://127.0.0.1:{port}/v1"), "sk-test", &f).unwrap(); let text = transcribe_lang(&format!("http://127.0.0.1:{port}/v1"), "sk-test", &f, "").unwrap();
let req = seen.join().unwrap(); let req = seen.join().unwrap();
let _ = std::fs::remove_file(&f); let _ = std::fs::remove_file(&f);
assert_eq!(text, "你好,幫我排行程"); assert_eq!(text, "你好,幫我排行程");
assert!(req.starts_with("POST /v1/audio/transcriptions")); assert!(req.starts_with("POST /v1/audio/transcriptions"));
assert!(req.contains("Bearer sk-test") && req.contains("name=\"model\"\r\n\r\ngpt-4o-mini-transcribe") && req.contains("filename=\"voice.m4a\"")); assert!(req.contains("Bearer sk-test") && req.contains("name=\"model\"\r\n\r\ngpt-4o-mini-transcribe") && req.contains("filename=\"voice.m4a\""));
assert!(transcribe("http://127.0.0.1:1", "", &std::path::PathBuf::from("/x")).unwrap_err().contains("OpenAI")); assert!(transcribe_lang("http://127.0.0.1:1", "", &std::path::PathBuf::from("/x"), "").unwrap_err().contains("OpenAI"));
} }
} }

View File

@ -28,6 +28,32 @@ pub fn unpack(b: &[u8]) -> Option<(Value, Vec<u8>)> {
Some((head, b[4 + n..].to_vec())) Some((head, b[4 + n..].to_vec()))
} }
/// How long a connection's loop waits to read before it sends what is queued: 2 ms while things
/// move (a live screen), 40 ms when quiet. A fixed 40 ms added up to 40 ms to every message on
/// each side of each hop.
pub struct Pace {
moved: std::time::Instant,
busy: Option<bool>,
}
impl Pace {
pub fn new() -> Pace {
Pace { moved: std::time::Instant::now() - std::time::Duration::from_secs(60), busy: None }
}
/// Something was read or sent.
pub fn moved(&mut self) {
self.moved = std::time::Instant::now();
}
/// Set the socket's read wait for now (only when it changes).
pub fn tune(&mut self, c: &std::net::TcpStream) {
let busy = self.moved.elapsed() < std::time::Duration::from_secs(2);
if self.busy != Some(busy) {
self.busy = Some(busy);
let _ = c.set_read_timeout(Some(std::time::Duration::from_millis(if busy { 2 } else { 40 })));
}
}
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
#[test] #[test]