diff --git a/README.md b/README.md index 3765e7e..632fabb 100644 --- a/README.md +++ b/README.md @@ -61,9 +61,9 @@ See [team setup and behavior](docs/TEAM.md) for commands, budgets, privacy bound | Explicit local files | `external_list_dir`, `external_read_file`, `external_edit_file`, `external_write_file`, `external_grep` (regex), `external_glob`, `external_search_files` (literal) | | Web | `web_fetch` (direct anonymous HTTP GET, HTML → text, short cache), `web_search` (remote search service); no browser login; Docker browser for login-gated pages | | Background | `spawn_subagent` (returns immediately; `kind=computerUse` for desktop GUI), `check_subagent`, `message_subagent`, `stop_subagent`. Subagents and external command completion revive the turn; observe detached box commands with `await_shell`. | -| Desktop GUI | Parent: `screenshot` (read-only). Clicks: `spawn_subagent kind=computerUse`, which gets `computer` (screenshot/click/move/drag/type/key/scroll/wait via xdotool on `DISPLAY=:1`). One computerUse per agent computer. Passwords still `request_box_help`. | +| Desktop GUI | Parent: `screenshot` (read-only). Clicks: `spawn_subagent kind=computerUse`, which gets `computer` (screenshot/snapshot/click/move/drag/type/key/scroll/wait via xdotool on `DISPLAY=:1`; `snapshot` lists on-screen elements with centre coordinates from the AT-SPI accessibility tree). One computerUse per agent computer. Passwords still `request_box_help`. | | MCP | `get_mcp_tools`, `call_mcp_tool`, `get_mcp_server_status`, `add_mcp_server`, `remove_mcp_server`. Config: `~/.lazyboy/mcp.json` (or `LAZYBOY_MCP_CONFIG`). Prefer a connector over the browser for that service. | -| My computer (box) | Docker Linux desktop (XFCE + **Chromium** + 終端機 + xdotool). Named agents each get their own container; opening an agent starts it and the UI confirms ready/error. Session CLI keeps `lazyboy-box`. Default tools: `shell`, `read`, `await_shell`, `screenshot`. Profile: `/home/box/chrome-profile` on that seat. Viewer：`lazyboy computer` or the agent overlay. | +| My computer (box) | Docker Linux desktop (TigerVNC + minimal XFCE in lazyBoy's Arc-Dark/粉圓 look, zh-TW/Asia/Taipei, **Chromium** + 終端機 + xdotool + AT-SPI). Named agents each get their own container; opening an agent starts it and the UI confirms ready/error. Session CLI keeps `lazyboy-box`. Default tools: `shell`, `read`, `await_shell`, `screenshot`. Profile: `/home/box/chrome-profile` on that seat. Viewer：`lazyboy computer` or the agent overlay. | | Browser observation | `browser_navigate`, `browser_snapshot`, `browser_read_page`, `browser_tabs` | | Browser actions | `browser_click`, `browser_type`, `browser_press`, `browser_select`, `browser_scroll`, `browser_wait` | | Browser file exchange | `browser_upload`, `browser_download` | diff --git a/box/Dockerfile b/box/Dockerfile index 4be1608..cbaf7f7 100644 --- a/box/Dockerfile +++ b/box/Dockerfile @@ -1,26 +1,81 @@ +# LazyBoy's computer: a real Linux desktop the human (noVNC) and the agent +# (xdotool + AT-SPI + CDP) share. Looks like lazyBoy's computer image: +# Arc-Dark + Papirus + jf open 粉圓, Traditional Chinese, Taipei time. +# One TigerVNC X server replaces Xvfb + x11vnc; no XFCE session manager. FROM debian:bookworm-slim ENV DEBIAN_FRONTEND=noninteractive -RUN apt-get update && apt-get install -y --no-install-recommends \ - xvfb x11-utils x11-xserver-utils x11vnc novnc websockify autocutsel xclip imagemagick xdotool \ - dbus-x11 xfce4-session xfce4-panel xfwm4 xfdesktop4 xfce4-settings \ - xfce4-terminal thunar \ - fonts-noto-core fonts-noto-cjk \ +# dpkg never unpacks docs/man/foreign locales; zh_TW catalogs are kept so the +# XFCE menus are in Traditional Chinese. +RUN printf '%s\n' \ + 'path-exclude=/usr/share/doc/*' \ + 'path-include=/usr/share/doc/*/copyright' \ + 'path-exclude=/usr/share/man/*' \ + 'path-exclude=/usr/share/info/*' \ + 'path-exclude=/usr/share/gtk-doc/*' \ + 'path-exclude=/usr/share/backgrounds/*' \ + 'path-exclude=/usr/share/locale/*' \ + 'path-include=/usr/share/locale/locale.alias' \ + 'path-include=/usr/share/locale/zh_TW/*' \ + 'path-include=/usr/share/locale/zh/*' \ + > /etc/dpkg/dpkg.cfg.d/zz-lazyboy-slim \ + && apt-get update && apt-get install -y --no-install-recommends \ + tigervnc-standalone-server tigervnc-tools x11-utils x11-xserver-utils \ + novnc websockify xclip imagemagick xdotool \ + dbus-x11 xfconf xfce4-panel xfwm4 xfdesktop4 xfce4-settings xfce4-terminal \ + arc-theme papirus-icon-theme adwaita-icon-theme gnome-themes-extra librsvg2-common \ + at-spi2-core libatk-adaptor python3-gi gir1.2-atspi-2.0 gir1.2-gtk-3.0 \ + dconf-gsettings-backend gsettings-desktop-schemas libglib2.0-bin \ + fonts-dejavu-core fonts-liberation fonts-noto-cjk fonts-noto-color-emoji \ + locales tzdata \ python3 curl ca-certificates git bash tini procps \ chromium \ + && echo "zh_TW.UTF-8 UTF-8" >> /etc/locale.gen \ + && echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen \ + && locale-gen \ + && ln -sf /usr/share/zoneinfo/Asia/Taipei /etc/localtime \ + && echo "Asia/Taipei" > /etc/timezone \ + && rm -rf /usr/share/icons/ePapirus /usr/share/icons/ePapirus-Dark /usr/share/icons/Papirus-Light \ + && test -e /usr/share/icons/Papirus-Dark/48x48/apps/web-browser.svg \ + && test -d /usr/share/themes/Arc-Dark/xfwm4 \ && rm -rf /var/lib/apt/lists/* +# jf open 粉圓 (OFL) as the desktop UI font, pinned like lazyBoy. +RUN mkdir -p /usr/share/fonts/truetype/huninn \ + && curl -fsSL https://github.com/justfont/open-huninn-font/releases/download/v2.1/jf-openhuninn-2.1.ttf \ + -o /usr/share/fonts/truetype/huninn/jf-openhuninn-2.1.ttf \ + && echo '9d5bf4932d31fe94c18cd8cfddc98bc1b14ce10f4e354c682179db290a99c825 /usr/share/fonts/truetype/huninn/jf-openhuninn-2.1.ttf' | sha256sum -c \ + && fc-cache -f COPY --from=node:22-bookworm-slim /usr/local/ /usr/local/ COPY playwright/ /opt/lazyboy/playwright/ RUN cd /opt/lazyboy/playwright && PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm install --omit=dev -RUN mkdir -p /workspace /home/box /usr/share/lazyboy \ +RUN mkdir -p /workspace /home/box /usr/share/lazyboy/xfce-skel /etc/xdg/xfce4 /usr/share/xfce4/helpers /etc/gtk-3.0 /etc/fonts/conf.d /usr/share/licenses/huninn \ && chmod 1777 /workspace -COPY start-desktop.sh stop-browser.py stop-browser.mjs box-chrome /usr/local/bin/ -COPY welcome.html /usr/share/lazyboy/welcome.html -COPY autostart/ /etc/xdg/autostart/ -COPY desktop/ /usr/share/lazyboy/desktop/ -RUN chmod +x /usr/local/bin/start-desktop.sh /usr/local/bin/box-chrome -ENV DISPLAY=:1 +COPY --chmod=755 start-desktop.sh stop-browser.py stop-browser.mjs box-chrome box-terminal box-a11y /usr/local/bin/ +COPY --chmod=644 welcome.html /usr/share/lazyboy/welcome.html +COPY --chmod=644 xfce/wallpaper.svg /usr/share/lazyboy/wallpaper.svg +COPY --chmod=644 xfce/xfce4-panel.xml xfce/xfwm4.xml xfce/xfce4-desktop.xml /usr/share/lazyboy/xfce-skel/ +COPY --chmod=644 xfce/helpers.rc /etc/xdg/xfce4/helpers.rc +COPY --chmod=644 xfce/helper-browser.desktop /usr/share/xfce4/helpers/lazyboy-browser.desktop +COPY --chmod=644 xfce/helper-terminal.desktop /usr/share/xfce4/helpers/lazyboy-terminal.desktop +COPY --chmod=644 xfce/browser.desktop /usr/share/applications/lazyboy-browser.desktop +COPY --chmod=644 xfce/terminal.desktop /usr/share/applications/lazyboy-terminal.desktop +COPY --chmod=644 fonts.conf /etc/fonts/conf.d/99-lazyboy-cjk.conf +COPY --chmod=644 gtk3-settings.ini /etc/gtk-3.0/settings.ini +COPY --chmod=644 licenses/huninn-OFL.txt /usr/share/licenses/huninn/OFL.txt +# Hide the stock launchers so the menu shows only 瀏覽器 / 終端機 (box-chrome +# is the only supported way to start Chromium: same profile + CDP port). +RUN for f in chromium xfce4-terminal debian-xterm xterm uxterm; do \ + if [ -f "/usr/share/applications/${f}.desktop" ]; then \ + printf '\nNoDisplay=true\nHidden=true\n' >> "/usr/share/applications/${f}.desktop"; \ + fi; \ + done \ + && test -f /usr/share/locale/zh_TW/LC_MESSAGES/xfce4-panel.mo +# Traditional Chinese locale, Taipei time and AT-SPI on for every process in +# the box, including `docker exec` launches. +ENV DISPLAY=:1 \ + LANG=zh_TW.UTF-8 LC_ALL=zh_TW.UTF-8 LANGUAGE=zh_TW:zh:en TZ=Asia/Taipei \ + GTK_MODULES=atk-bridge GTK_A11Y=atspi GNOME_ACCESSIBILITY=1 NO_AT_BRIDGE=0 WORKDIR /workspace EXPOSE 6080 -LABEL lazyboy.box.revision=computer-use-5 +LABEL lazyboy.box.revision=computer-use-7 ENTRYPOINT ["/usr/bin/tini", "--"] CMD ["/usr/local/bin/start-desktop.sh"] diff --git a/box/autostart/lazyboy-browser.desktop b/box/autostart/lazyboy-browser.desktop deleted file mode 100644 index 0250922..0000000 --- a/box/autostart/lazyboy-browser.desktop +++ /dev/null @@ -1,5 +0,0 @@ -[Desktop Entry] -Type=Application -Name=Browser -Exec=box-chrome /usr/share/lazyboy/welcome.html -X-XFCE-Autostart-Enabled=true diff --git a/box/autostart/lazyboy-terminal.desktop b/box/autostart/lazyboy-terminal.desktop deleted file mode 100644 index ff42d76..0000000 --- a/box/autostart/lazyboy-terminal.desktop +++ /dev/null @@ -1,5 +0,0 @@ -[Desktop Entry] -Type=Application -Name=Terminal -Exec=xfce4-terminal --title=終端機 --working-directory=/workspace -X-XFCE-Autostart-Enabled=true diff --git a/box/box-a11y b/box/box-a11y new file mode 100755 index 0000000..3866b9f --- /dev/null +++ b/box/box-a11y @@ -0,0 +1,258 @@ +#!/usr/bin/env python3 +"""Accessibility snapshot of the box desktop (AT-SPI). + +Prints one line per interactive/visible element with its role, name and the +screen-pixel centre the computer tool can click: + + [12] button "登入" @ (640,412) 96x36 win="蝦皮購物 - Chromium" + +Usage: box-a11y [--json] [--limit N] [--budget SECONDS] +Exit 0 with an empty list when nothing is exposed; exit 2 when AT-SPI itself is +unreachable (the desktop or registry is not up). +""" +import argparse +import json +import os +import sys +import time +import warnings + +warnings.filterwarnings("ignore", category=DeprecationWarning) + +# Always listed when on screen, even without a name (the agent can still +# target them by position/value). +INTERACTIVE = { + "push button", "toggle button", "radio button", "check box", "link", + "menu item", "check menu item", "radio menu item", "tab", "page tab", + "list item", "tree item", "combo box", "spin button", "slider", + "entry", "password text", +} +# Listed only when they carry a name or text (headings, labels, cells...). +NAMED = { + "heading", "label", "static", "text", "image", "table cell", + "column header", "row header", "icon", "menu", "tool tip", "alert", + "dialog", "frame", +} +ROLE_SHORT = { + "push button": "button", "toggle button": "toggle", "radio button": "radio", + "check box": "checkbox", "menu item": "menuitem", "check menu item": "menuitem", + "radio menu item": "menuitem", "page tab": "tab", "list item": "listitem", + "tree item": "treeitem", "combo box": "combobox", "spin button": "spinner", + "password text": "password", "entry": "textbox", "text": "text", + "static": "text", "document frame": "document", "table cell": "cell", + "column header": "header", "row header": "header", "internal frame": "frame", +} + + +def load_dbus_env(): + if os.environ.get("DBUS_SESSION_BUS_ADDRESS"): + return + try: + with open("/tmp/lazyboy/dbus.env", encoding="utf-8") as fh: + for line in fh: + key, _, value = line.strip().partition("=") + if key == "DBUS_SESSION_BUS_ADDRESS" and value: + os.environ[key] = value + except OSError: + pass + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--json", action="store_true") + ap.add_argument("--limit", type=int, default=300) + ap.add_argument("--budget", type=float, default=6.0) + ap.add_argument("--max-depth", type=int, default=40) + args = ap.parse_args() + + load_dbus_env() + os.environ.setdefault("DISPLAY", ":1") + try: + import gi + gi.require_version("Atspi", "2.0") + from gi.repository import Atspi + except Exception as exc: # noqa: BLE001 + print(f"AT-SPI bindings unavailable: {exc}", file=sys.stderr) + return 2 + + try: + Atspi.init() + desktop = Atspi.get_desktop(0) + app_count = desktop.get_child_count() + except Exception as exc: # noqa: BLE001 + print(f"AT-SPI registry unreachable: {exc}", file=sys.stderr) + return 2 + + screen_w = int(os.environ.get("LAZYBOY_SCREEN_W", "0") or 0) + screen_h = int(os.environ.get("LAZYBOY_SCREEN_H", "0") or 0) + deadline = time.monotonic() + args.budget + out = [] + truncated = False + seen_apps = [] + + def showing(node): + try: + st = node.get_state_set() + return st.contains(Atspi.StateType.SHOWING) and st.contains(Atspi.StateType.VISIBLE) + except Exception: # noqa: BLE001 + return False + + def extents(node): + try: + comp = node.get_component() if hasattr(node, "get_component") else node.get_component_iface() + if comp is None: + return None + r = comp.get_extents(Atspi.CoordType.SCREEN) + return int(r.x), int(r.y), int(r.width), int(r.height) + except Exception: # noqa: BLE001 + return None + + def onscreen(box): + x, y, w, h = box + if w <= 0 or h <= 0: + return False + if screen_w and (x >= screen_w or x + w <= 0): + return False + if screen_h and (y >= screen_h or y + h <= 0): + return False + return True + + def walk(node, depth, win_title): + nonlocal truncated + if truncated or time.monotonic() > deadline: + truncated = True + return + if depth > args.max_depth or len(out) >= args.limit: + truncated = truncated or len(out) >= args.limit + return + try: + role = node.get_role_name() + except Exception: # noqa: BLE001 + return + if depth > 1 and not showing(node): + return + box = extents(node) + if depth > 1 and (box is None or not onscreen(box)): + return + name = "" + try: + name = (node.get_name() or "").strip() + except Exception: # noqa: BLE001 + pass + if role in ("frame", "dialog", "window") and name: + win_title = name + try: + st = node.get_state_set() + focused = st.contains(Atspi.StateType.FOCUSED) + editable = st.contains(Atspi.StateType.EDITABLE) + checked = st.contains(Atspi.StateType.CHECKED) + selected = st.contains(Atspi.StateType.SELECTED) + disabled = not st.contains(Atspi.StateType.ENABLED) + except Exception: # noqa: BLE001 + focused = editable = checked = selected = disabled = False + value = "" + if role in ("entry", "password text", "text", "spin button", "slider") or editable: + try: + txt = node.get_text() if hasattr(node, "get_text") else node.get_text_iface() + if txt is not None and role != "password text": + value = (txt.get_text(0, min(txt.get_character_count(), 80)) or "").strip() + except Exception: # noqa: BLE001 + pass + printable = depth > 1 and box is not None and ( + role in INTERACTIVE + or (role in NAMED and (name or value)) + or (editable and role == "text") + ) + if printable: + x, y, w, h = box + out.append({ + "role": ROLE_SHORT.get(role, role.replace(" ", "")), + "name": name[:120], + "value": value[:80], + "x": x, "y": y, "w": w, "h": h, + "cx": x + w // 2, "cy": y + h // 2, + "focused": focused, "checked": checked, "selected": selected, + "disabled": disabled, "editable": editable, + "window": win_title, + }) + try: + count = node.get_child_count() + except Exception: # noqa: BLE001 + return + for i in range(min(count, 2000)): + if truncated: + return + try: + child = node.get_child_at_index(i) + except Exception: # noqa: BLE001 + continue + if child is None: + continue + walk(child, depth + 1, win_title) + + for i in range(app_count): + try: + app = desktop.get_child_at_index(i) + if app is None: + continue + app_name = app.get_name() or "" + except Exception: # noqa: BLE001 + continue + if app_name in ("xfce4-panel", "wrapper-2.0", "xfdesktop", "xfwm4", "box-a11y", "python3"): + # The desktop shell is visible in the screenshot already; keep the + # list for the windows the agent actually works in. + continue + seen_apps.append(app_name) + walk(app, 0, app_name) + + # Chromium mirrors some nodes (omnibox rows, nested list items); one entry + # per role/name/box is enough for targeting. Then top-to-bottom, + # left-to-right reading order. + unique = {} + for e in out: + unique.setdefault((e["role"], e["name"], e["cx"], e["cy"], e["w"], e["h"]), e) + window_order = {} + for e in unique.values(): + window_order.setdefault(e["window"], len(window_order)) + out = sorted(unique.values(), key=lambda e: (window_order[e["window"]], e["cy"] // 12, e["cx"])) + for i, e in enumerate(out): + e["id"] = i + if args.json: + json.dump({"apps": seen_apps, "elements": out, "truncated": truncated}, sys.stdout, ensure_ascii=False) + print() + return 0 + lines = [] + current_window = None + for e in out: + if len(window_order) > 1 and e["window"] != current_window: + current_window = e["window"] + lines.append(f"## {current_window}") + flags = [] + if e["focused"]: + flags.append("focused") + if e["checked"]: + flags.append("checked") + if e["selected"]: + flags.append("selected") + if e["disabled"]: + flags.append("disabled") + label = f'"{e["name"]}"' if e["name"] else "" + if e["value"]: + label += f' value="{e["value"]}"' + parts = [f"[{e['id']}]", e["role"]] + if label: + parts.append(label) + parts.append(f"@ ({e['cx']},{e['cy']}) {e['w']}x{e['h']}") + if flags: + parts.append("(" + ", ".join(flags) + ")") + lines.append(" ".join(parts)) + if not lines: + lines.append("(no accessible elements exposed; apps: " + ", ".join(seen_apps) + ")") + if truncated: + lines.append(f"... truncated at {len(out)} elements; scroll or focus a window and snapshot again") + print("\n".join(lines)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/box/box-chrome b/box/box-chrome old mode 100644 new mode 100755 index 99855e7..892427c --- a/box/box-chrome +++ b/box/box-chrome @@ -3,6 +3,25 @@ # attach to the browser the human sees (Firefox profiles cannot). set -euo pipefail export DISPLAY="${DISPLAY:-:1}" +# Chromium reads HOME for its config/crash-report dirs; keep them on the +# persisted /home/box volume rather than in /root. +export HOME=/home/box +export XDG_CONFIG_HOME=/home/box/.config +export LANG="${LANG:-zh_TW.UTF-8}" +export LC_ALL="${LC_ALL:-zh_TW.UTF-8}" +export LANGUAGE="${LANGUAGE:-zh_TW:zh:en}" +export TZ="${TZ:-Asia/Taipei}" +export GTK_MODULES="${GTK_MODULES:-atk-bridge}" +export GTK_A11Y="${GTK_A11Y:-atspi}" +export GNOME_ACCESSIBILITY="${GNOME_ACCESSIBILITY:-1}" +export NO_AT_BRIDGE=0 +# Join the desktop's session bus so Chromium registers with the same AT-SPI +# registry that box-a11y reads (launches over `docker exec` start without it). +if [[ -z "${DBUS_SESSION_BUS_ADDRESS:-}" && -r /tmp/lazyboy/dbus.env ]]; then + # shellcheck disable=SC1091 + source /tmp/lazyboy/dbus.env + export DBUS_SESSION_BUS_ADDRESS +fi PROFILE=/home/box/chrome-profile CDP_PORT=9222 LAUNCH_LOCK=/tmp/.box-chrome-launch.lock @@ -31,7 +50,11 @@ flock -w 30 9 || true if profile_in_use; then flock -u 9 exec 9>&- - exec chromium --no-sandbox --user-data-dir="$PROFILE" "$@" + # This invocation only forwards its URLs to the running instance, but + # Chromium treats *this* command line as the startup command line when it + # decides whether to show the "unsupported flag: --no-sandbox" infobar in + # the new tab, so --test-type must be present here too. + exec chromium --no-sandbox --test-type --user-data-dir="$PROFILE" "$@" fi # The profile lives on a volume that outlives the container. Chromium's @@ -42,16 +65,33 @@ fi # stale by definition. rm -f "$PROFILE/SingletonLock" "$PROFILE/SingletonSocket" "$PROFILE/SingletonCookie" +# Flags mirror lazyBoy's lazyboy-browser (the desktop that passes Shopee's +# checks): Traditional Chinese UI + Accept-Language, maximized normal window, +# no translate/crash bubbles. Never add --kiosk or --app. chromium \ --no-sandbox \ - --disable-dev-shm-usage \ - --disable-gpu \ - --no-first-run --restore-last-session \ - --no-default-browser-check \ - --password-store=basic \ - --user-data-dir="$PROFILE" \ --remote-debugging-address=127.0.0.1 \ --remote-debugging-port="$CDP_PORT" \ + --test-type \ + --disable-gpu \ + --disable-dev-shm-usage \ + --disable-features=TranslateUI \ + --disable-blink-features=AutomationControlled \ + --no-first-run \ + --no-default-browser-check \ + --disable-session-crashed-bubble \ + --hide-crash-restore-bubble \ + --disable-infobars \ + --force-renderer-accessibility \ + --disable-background-networking \ + --disable-component-update \ + --renderer-process-limit=4 \ + --password-store=basic \ + --lang=zh-TW \ + --accept-lang=zh-TW,zh,en-US,en \ + --start-maximized \ + --restore-last-session \ + --user-data-dir="$PROFILE" \ "$@" 9>&- & pid=$! trap 'kill -TERM "$pid" 2>/dev/null' TERM INT diff --git a/box/box-terminal b/box/box-terminal new file mode 100755 index 0000000..54b81db --- /dev/null +++ b/box/box-terminal @@ -0,0 +1,15 @@ +#!/bin/bash +# Terminal on my computer: xfce4-terminal in /workspace. Plain bash keeps the +# image small; --disable-server so each window is its own process. +export LANG="${LANG:-zh_TW.UTF-8}" +export LC_ALL="${LC_ALL:-zh_TW.UTF-8}" +export LANGUAGE="${LANGUAGE:-zh_TW:zh:en}" +export TERM="${TERM:-xterm-256color}" +export COLORTERM="${COLORTERM:-truecolor}" +cd /workspace 2>/dev/null || cd / +exec xfce4-terminal \ + --disable-server \ + --geometry=100x28+40+40 \ + --title=終端機 \ + --working-directory=/workspace \ + "$@" diff --git a/box/desktop/browser.desktop b/box/desktop/browser.desktop deleted file mode 100644 index 59bf6aa..0000000 --- a/box/desktop/browser.desktop +++ /dev/null @@ -1,9 +0,0 @@ -[Desktop Entry] -Version=1.0 -Type=Application -Name=瀏覽器 -Comment=Chrome on my computer -Exec=box-chrome -Icon=web-browser -Terminal=false -Categories=Network; diff --git a/box/fonts.conf b/box/fonts.conf new file mode 100644 index 0000000..b1faef7 --- /dev/null +++ b/box/fonts.conf @@ -0,0 +1,41 @@ + + + + + + 粉圓 + jf open 粉圓 2.1 + + + jf open 粉圓 + jf open 粉圓 2.1 + + + Huninn + jf open 粉圓 2.1 + + + + sans-serif + + jf open 粉圓 2.1 + + + + zh + sans-serif + + jf open 粉圓 2.1 + + + + + monospace + + Noto Sans Mono CJK TC + + + DejaVu Sans Mono + + + diff --git a/box/gtk3-settings.ini b/box/gtk3-settings.ini new file mode 100644 index 0000000..5a63f3f --- /dev/null +++ b/box/gtk3-settings.ini @@ -0,0 +1,5 @@ +[Settings] +gtk-font-name=jf open 粉圓 2.1 12 +gtk-icon-theme-name=Papirus-Dark +gtk-theme-name=Arc-Dark +gtk-application-prefer-dark-theme=1 diff --git a/box/licenses/huninn-OFL.txt b/box/licenses/huninn-OFL.txt new file mode 100644 index 0000000..7058b6a --- /dev/null +++ b/box/licenses/huninn-OFL.txt @@ -0,0 +1,101 @@ +[Kosugi Maru] +Copyright (c) 2010 MOTOYA CO.,LTD. + +[Varela Round] +Copyright (c) 2011-2016 The Varela Round Project Authors (https://github.com/alefalefalef/Varela-Round-Hebrew/), with Reserved Font Names 'Varela' and 'Varela Round'. + +[jf open huninn] +Copyright (c) 2020-2024 The jf open huninn font is redistributed by justfont Co., LTD., with Reserved Font Names 'open huninn' and 'huninn'. under SIL Open Font License. The Hanzi part of this project was derived from Kosugi Maru under Apache-2.0 (https://www.apache.org/licenses/LICENSE-2.0). + + +This Font Software is licensed under the SIL Open Font License, Version 1.1. +This license is copied below, and is also available with a FAQ at: +http://scripts.sil.org/OFL + + +----------------------------------------------------------- +SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007 +----------------------------------------------------------- + +PREAMBLE +The goals of the Open Font License (OFL) are to stimulate worldwide +development of collaborative font projects, to support the font creation +efforts of academic and linguistic communities, and to provide a free and +open framework in which fonts may be shared and improved in partnership +with others. + +The OFL allows the licensed fonts to be used, studied, modified and +redistributed freely as long as they are not sold by themselves. The +fonts, including any derivative works, can be bundled, embedded, +redistributed and/or sold with any software provided that any reserved +names are not used by derivative works. The fonts and derivatives, +however, cannot be released under any other type of license. The +requirement for fonts to remain under this license does not apply +to any document created using the fonts or their derivatives. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this license and clearly marked as such. This may +include source files, build scripts and documentation. + +"Reserved Font Name" refers to any names specified as such after the +copyright statement(s). + +"Original Version" refers to the collection of Font Software components as +distributed by the Copyright Holder(s). + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to a +new environment. + +"Author" refers to any designer, engineer, programmer, technical +writer or other person who contributed to the Font Software. + +PERMISSION & CONDITIONS +Permission is hereby granted, free of charge, to any person obtaining +a copy of the Font Software, to use, study, copy, merge, embed, modify, +redistribute, and sell modified and unmodified copies of the Font +Software, subject to the following conditions: + +1) Neither the Font Software nor any of its individual components, +in Original or Modified Versions, may be sold by itself. + +2) Original or Modified Versions of the Font Software may be bundled, +redistributed and/or sold with any software, provided that each copy +contains the above copyright notice and this license. These can be +included either as stand-alone text files, human-readable headers or +in the appropriate machine-readable metadata fields within text or +binary files as long as those fields can be easily viewed by the user. + +3) No Modified Version of the Font Software may use the Reserved Font +Name(s) unless explicit written permission is granted by the corresponding +Copyright Holder. This restriction only applies to the primary font name as +presented to the users. + +4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font +Software shall not be used to promote, endorse or advertise any +Modified Version, except to acknowledge the contribution(s) of the +Copyright Holder(s) and the Author(s) or with their explicit written +permission. + +5) The Font Software, modified or unmodified, in part or in whole, +must be distributed entirely under this license, and must not be +distributed under any other license. The requirement for fonts to +remain under this license does not apply to any document created +using the Font Software. + +TERMINATION +This license becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT +OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM +OTHER DEALINGS IN THE FONT SOFTWARE. diff --git a/box/start-desktop.sh b/box/start-desktop.sh old mode 100644 new mode 100755 index eacda0b..ca6d958 --- a/box/start-desktop.sh +++ b/box/start-desktop.sh @@ -1,14 +1,54 @@ #!/bin/bash +# Boot the box desktop: one TigerVNC X server (framebuffer + VNC in a single +# event-driven process), a minimal XFCE (xfwm4 + xfdesktop + panel, no session +# manager), AT-SPI for element-level automation, noVNC, then Chromium. +# Never a kiosk, --app, or bare HTML shell: the human and the agent share a +# normal Linux desktop. set -euo pipefail export DISPLAY="${DISPLAY:-:1}" -mkdir -p /workspace /home/box/Desktop /home/box/.config/autostart /tmp -chmod 1777 /workspace /tmp -cp -n /usr/share/lazyboy/desktop/*.desktop /home/box/Desktop/ 2>/dev/null || true -chmod +x /home/box/Desktop/*.desktop 2>/dev/null || true +# Traditional Chinese desktop + Taipei time; everything launched from here +# (panel, box-chrome, terminals) inherits these. +export LANG="${LANG:-zh_TW.UTF-8}" +export LC_ALL="${LC_ALL:-zh_TW.UTF-8}" +export LANGUAGE="${LANGUAGE:-zh_TW:zh:en}" +export TZ="${TZ:-Asia/Taipei}" +# AT-SPI: GTK apps and Chromium publish their accessibility trees so the +# agent can read element names/coordinates instead of guessing pixels. +export GTK_MODULES="${GTK_MODULES:-atk-bridge}" +export GTK_A11Y="${GTK_A11Y:-atspi}" +export GNOME_ACCESSIBILITY="${GNOME_ACCESSIBILITY:-1}" +export NO_AT_BRIDGE=0 +export XDG_CURRENT_DESKTOP=XFCE +export XDG_CONFIG_HOME=/home/box/.config +export XDG_CACHE_HOME=/tmp/box-cache +export XDG_RUNTIME_DIR=/tmp/box-runtime + +ROOT=/tmp/lazyboy +SCREEN_GEOMETRY="${LAZYBOY_SCREEN_GEOMETRY:-1280x720}" +mkdir -p /workspace /home/box /tmp "$ROOT" "$XDG_CACHE_HOME" "$XDG_RUNTIME_DIR" /tmp/.X11-unix +chmod 1777 /workspace /tmp /tmp/.X11-unix +chmod 700 "$XDG_RUNTIME_DIR" +rm -f "$ROOT/ready" + +# XFCE configuration is re-seeded from the image on every boot so the desktop +# always looks the way the image defines it; only /home/box/chrome-profile +# (Chromium) is meant to carry state across boots. +hydrate_xfce() { + local xml="$XDG_CONFIG_HOME/xfce4/xfconf/xfce-perchannel-xml" + mkdir -p "$xml" "$XDG_CONFIG_HOME/xfce4/panel" "$XDG_CONFIG_HOME/glib-2.0/settings" + local f + for f in xfce4-panel.xml xfwm4.xml xfce4-desktop.xml; do + cp -f "/usr/share/lazyboy/xfce-skel/$f" "$xml/$f" + done + cp -f /etc/xdg/xfce4/helpers.rc "$XDG_CONFIG_HOME/xfce4/helpers.rc" +} if [[ -z "${DBUS_SESSION_BUS_ADDRESS:-}" ]]; then eval "$(dbus-launch --sh-syntax)" fi +# Tools started later over `docker exec` (box-a11y, box-chrome) must join this +# bus to reach the AT-SPI registry that the desktop apps registered with. +printf 'DBUS_SESSION_BUS_ADDRESS=%s\n' "$DBUS_SESSION_BUS_ADDRESS" >"$ROOT/dbus.env" display_number="${DISPLAY#:}" display_number="${display_number%%.*}" @@ -29,30 +69,62 @@ if ! xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then rm -f -- "$lock_file" "/tmp/.X11-unix/X${display_number}" fi -Xvfb "$DISPLAY" -screen 0 1280x720x24 -ac +extension GLX +render -noreset >/tmp/xvfb.log 2>&1 & -for _ in 1 2 3 4 5 6 7 8 9 10; do +# Only websockify (same container) talks to VNC; keep the passwordless port +# off the Docker bridge so sibling containers cannot attach. +Xtigervnc "$DISPLAY" -geometry "$SCREEN_GEOMETRY" -depth 24 \ + -rfbport 5900 -localhost -SecurityTypes None -AlwaysShared -AcceptCutText -SendCutText \ + -desktop LazyBoy -ac -noreset >/tmp/xvnc.log 2>&1 & +for _ in $(seq 1 30); do if xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then break fi - sleep 0.3 + sleep 0.2 done if ! xdpyinfo -display "$DISPLAY" >/dev/null 2>&1; then - cat /tmp/xvfb.log >&2 + cat /tmp/xvnc.log >&2 exit 1 fi -startxfce4 >/tmp/xfce.log 2>&1 & -# Only websockify (same container) talks to VNC; keep the passwordless port -# off the Docker bridge so sibling containers cannot attach. (libvncserver -# still binds [::]:5900, but the container has no routable IPv6 address.) -x11vnc -display "$DISPLAY" -forever -shared -nopw -rfbport 5900 -listen localhost -xkb -repeat -noxdamage >/tmp/x11vnc.log 2>&1 & -if command -v autocutsel >/dev/null 2>&1; then - autocutsel -fork -selection CLIPBOARD >/dev/null 2>&1 || true - autocutsel -fork -selection PRIMARY >/dev/null 2>&1 || true -fi -websockify --web /usr/share/novnc 6080 localhost:5900 >/tmp/novnc.log 2>&1 & +# Accessibility bus + registry. toolkit-accessibility is what Chromium and GTK +# consult before exposing their trees over AT-SPI. +for c in /usr/libexec/at-spi-bus-launcher /usr/lib/at-spi2-core/at-spi-bus-launcher; do + if [[ -x "$c" ]]; then + "$c" --launch-immediately >/tmp/atspi-bus.log 2>&1 & + break + fi +done +for c in /usr/libexec/at-spi2-registryd /usr/lib/at-spi2-core/at-spi2-registryd; do + if [[ -x "$c" ]]; then + "$c" >/tmp/atspi-registry.log 2>&1 & + break + fi +done +HOME=/home/box gsettings set org.gnome.desktop.interface toolkit-accessibility true 2>/tmp/gsettings.log || true -echo "[lazyboy-box] desktop up display=$DISPLAY vnc=6080 xfce chrome terminal" >/tmp/start-desktop.log +hydrate_xfce +xfconfd --daemon >/tmp/xfconfd.log 2>&1 || true +sleep 0.2 +xfwm4 --compositor=off --display="$DISPLAY" --sm-client-disable >/tmp/xfwm4.log 2>&1 & +sleep 0.3 +xfdesktop --disable-wm-check --sm-client-disable >/tmp/xfdesktop.log 2>&1 & +xfce4-panel --disable-wm-check --sm-client-disable >/tmp/xfce4-panel.log 2>&1 & + +# Xvnc keeps the VNC clipboard in sync with the X selections through vncconfig. +if command -v vncconfig >/dev/null 2>&1; then + vncconfig -nowin -display "$DISPLAY" >/tmp/vncconfig.log 2>&1 & +fi +websockify --heartbeat=30 --web /usr/share/novnc 6080 127.0.0.1:5900 >/tmp/novnc.log 2>&1 & + +# Chromium: --restore-last-session brings back the previous tabs, so only the +# very first boot of a profile gets the welcome page (otherwise it doubled up). +if [[ -d /home/box/chrome-profile/Default/Sessions ]]; then + box-chrome >/tmp/lazyboy-browser.log 2>&1 & +else + box-chrome /usr/share/lazyboy/welcome.html >/tmp/lazyboy-browser.log 2>&1 & +fi + +echo "[lazyboy-box] desktop up display=$DISPLAY geometry=$SCREEN_GEOMETRY vnc=6080 xfce chrome" >/tmp/start-desktop.log +touch "$ROOT/ready" trap 'node /usr/local/bin/stop-browser.mjs || python3 /usr/local/bin/stop-browser.py; exit 0' TERM INT while true; do sleep 3600 & diff --git a/box/welcome.html b/box/welcome.html index f7028d0..a03f149 100644 --- a/box/welcome.html +++ b/box/welcome.html @@ -4,15 +4,15 @@ LazyBoy 的電腦

這是 LazyBoy 的電腦

登入、驗證碼、付款請在這個畫面完成。帳密只在這裡輸入，對話裡看不到。

-

桌面上有「瀏覽器」和「終端機」。做完後回到聊天回覆「好了」。

+

下方工作列有「瀏覽器」和「終端機」。做完後回到聊天回覆「好了」。

diff --git a/box/xfce/browser.desktop b/box/xfce/browser.desktop new file mode 100644 index 0000000..288858b --- /dev/null +++ b/box/xfce/browser.desktop @@ -0,0 +1,11 @@ +[Desktop Entry] +Version=1.0 +Type=Application +Name=瀏覽器 +Name[zh_TW]=瀏覽器 +Comment=Chromium on my computer +Exec=/usr/local/bin/box-chrome +Icon=web-browser +Terminal=false +Categories=Network;WebBrowser;GTK; +StartupNotify=true diff --git a/box/xfce/helper-browser.desktop b/box/xfce/helper-browser.desktop new file mode 100644 index 0000000..ddbedd5 --- /dev/null +++ b/box/xfce/helper-browser.desktop @@ -0,0 +1,11 @@ +[Desktop Entry] +Version=1.0 +Type=X-XFCE-Helper +Icon=web-browser +Name=瀏覽器 +Name[zh_TW]=瀏覽器 +StartupNotify=true +X-XFCE-Binaries=box-chrome;chromium; +X-XFCE-Category=WebBrowser +X-XFCE-Commands=/usr/local/bin/box-chrome; +X-XFCE-CommandsWithParameter=/usr/local/bin/box-chrome "%s"; diff --git a/box/xfce/helper-terminal.desktop b/box/xfce/helper-terminal.desktop new file mode 100644 index 0000000..6bfde4a --- /dev/null +++ b/box/xfce/helper-terminal.desktop @@ -0,0 +1,11 @@ +[Desktop Entry] +Version=1.0 +Type=X-XFCE-Helper +Icon=utilities-terminal +Name=終端機 +Name[zh_TW]=終端機 +StartupNotify=true +X-XFCE-Binaries=box-terminal;xfce4-terminal; +X-XFCE-Category=TerminalEmulator +X-XFCE-Commands=/usr/local/bin/box-terminal; +X-XFCE-CommandsWithParameter=/usr/local/bin/box-terminal -e %s; diff --git a/box/xfce/helpers.rc b/box/xfce/helpers.rc new file mode 100644 index 0000000..b8e4935 --- /dev/null +++ b/box/xfce/helpers.rc @@ -0,0 +1,2 @@ +WebBrowser=lazyboy-browser +TerminalEmulator=lazyboy-terminal diff --git a/box/desktop/terminal.desktop b/box/xfce/terminal.desktop similarity index 53% rename from box/desktop/terminal.desktop rename to box/xfce/terminal.desktop index f2da43f..128d26b 100644 --- a/box/desktop/terminal.desktop +++ b/box/xfce/terminal.desktop @@ -2,8 +2,10 @@ Version=1.0 Type=Application Name=終端機 +Name[zh_TW]=終端機 Comment=Terminal on my computer -Exec=xfce4-terminal --working-directory=/workspace +Exec=/usr/local/bin/box-terminal Icon=utilities-terminal Terminal=false -Categories=System; +Categories=System;TerminalEmulator;GTK; +StartupNotify=true diff --git a/box/xfce/wallpaper.svg b/box/xfce/wallpaper.svg new file mode 100644 index 0000000..687bed4 --- /dev/null +++ b/box/xfce/wallpaper.svg @@ -0,0 +1,32 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/box/xfce/xfce4-desktop.xml b/box/xfce/xfce4-desktop.xml new file mode 100644 index 0000000..620c3ed --- /dev/null +++ b/box/xfce/xfce4-desktop.xml @@ -0,0 +1,53 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/box/xfce/xfce4-panel.xml b/box/xfce/xfce4-panel.xml new file mode 100644 index 0000000..a959b13 --- /dev/null +++ b/box/xfce/xfce4-panel.xml @@ -0,0 +1,66 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/box/xfce/xfwm4.xml b/box/xfce/xfwm4.xml new file mode 100644 index 0000000..13a3955 --- /dev/null +++ b/box/xfce/xfwm4.xml @@ -0,0 +1,20 @@ + + + + + + + + + + + + + + + + + + + + diff --git a/crates/lazyboy-core/src/box_runtime.rs b/crates/lazyboy-core/src/box_runtime.rs index b6e1f98..9204304 100644 --- a/crates/lazyboy-core/src/box_runtime.rs +++ b/crates/lazyboy-core/src/box_runtime.rs @@ -21,7 +21,7 @@ const SESSION_WORKSPACE_VOLUME: &str = "grokboy-box-workspace"; const SESSION_HOME_VOLUME: &str = "grokboy-box-home"; const SESSION_VIEWER_PORT: u16 = 6080; pub const SESSION_SEAT: &str = "session"; -const BOX_REVISION: &str = "computer-use-5"; +const BOX_REVISION: &str = "computer-use-7"; const BROWSER_PROFILE: &str = "/home/box/chrome-profile"; /// Per-stream cap on shell output returned to the model; the middle is elided. const JOB_OUTPUT_HEAD_CHARS: usize = 16_000; @@ -627,19 +627,20 @@ print(json.dumps({{'path': str(p), 'offset': start+1, 'lines': len(lines), 'cont validate_bounds(actions, screen)?; let names: Vec<&str> = actions.iter().map(ComputerAction::name).collect(); let shot = ShotFile::new(); + let mut snapshot = None; for (index, action) in actions.iter().enumerate() { let settle = settle_before(actions, index); if settle > 0 { tokio::time::sleep(Duration::from_millis(u64::from(settle))).await; } - self.run_computer_action(action, &shot).await?; + self.run_computer_action(action, &shot, screen, &mut snapshot).await?; } if !actions.iter().any(ComputerAction::is_screenshot) { self.capture_root_png(&shot).await?; } shot.fetch(self.container(), dest).await?; let cursor = self.cursor_position().await.ok(); - Ok(json!({ + let mut payload = json!({ "ok": true, "actions": names, "path": dest, @@ -647,13 +648,53 @@ print(json.dumps({{'path': str(p), 'offset': start+1, 'lines': len(lines), 'cont "cursor_position": cursor, "screen": {"width": screen.width, "height": screen.height}, "instruction": "Computer action ran on my computer. Read this screenshot before the next action. A batched then sequence returns only this final screen." - })) + }); + if let Some(text) = snapshot { + payload["snapshot"] = json!(text); + payload["instruction"] = json!( + "Computer action ran on my computer. `snapshot` lists the on-screen elements as `[id] role \"name\" @ (x,y) WxH`; click the (x,y) centre of the element you mean. Read the screenshot too before the next action." + ); + } + Ok(payload) } - async fn run_computer_action(&self, action: &ComputerAction, shot: &ShotFile) -> Result<()> { + async fn run_computer_action( + &self, + action: &ComputerAction, + shot: &ShotFile, + screen: ScreenSize, + snapshot: &mut Option, + ) -> Result<()> { for step in steps_for(action)? { match step { ComputerStep::Screenshot => self.capture_root_png(shot).await?, + ComputerStep::A11ySnapshot => { + let width = screen.width.to_string(); + let height = screen.height.to_string(); + let out = self + .exec_env( + &[ + ("DISPLAY", ":1"), + ("LAZYBOY_SCREEN_W", width.as_str()), + ("LAZYBOY_SCREEN_H", height.as_str()), + ], + &["box-a11y"], + ) + .await?; + if out.status != 0 { + let detail = out.stderr.trim(); + if detail.contains("not found") { + return Err(anyhow!( + "My computer image is missing box-a11y. Rebuild the box image and retry." + )); + } + return Err(anyhow!( + "computer snapshot failed: {}", + if detail.is_empty() { out.stdout.trim() } else { detail } + )); + } + *snapshot = Some(out.stdout.trim_end().to_string()); + } ComputerStep::SleepMs(0) => {} ComputerStep::SleepMs(ms) => { let secs = format!("{:.3}", f64::from(ms) / 1000.0); diff --git a/crates/lazyboy-core/src/computer.rs b/crates/lazyboy-core/src/computer.rs index ff65ce9..958aed0 100644 --- a/crates/lazyboy-core/src/computer.rs +++ b/crates/lazyboy-core/src/computer.rs @@ -1,5 +1,6 @@ -//! Grok Bot Computer tool: screenshot / click / move / drag / type / key / scroll / wait. -//! Parent agents do not get this tool. A `computerUse` subagent drives the box via xdotool. +//! Grok Bot Computer tool: screenshot / snapshot / click / move / drag / type / key / scroll / wait. +//! Parent agents do not get this tool. A `computerUse` subagent drives the box via xdotool; +//! `snapshot` reads the AT-SPI accessibility tree (element names + pixel centres) via box-a11y. use anyhow::{anyhow, Result}; use serde_json::{json, Value}; @@ -98,6 +99,8 @@ impl ScrollDir { #[derive(Debug, Clone, PartialEq, Eq)] pub enum ComputerAction { Screenshot, + /// Accessibility-tree listing of on-screen elements with clickable centres. + Snapshot, Click { x: Option, y: Option, @@ -133,6 +136,7 @@ impl ComputerAction { pub fn name(&self) -> &'static str { match self { Self::Screenshot => "screenshot", + Self::Snapshot => "snapshot", Self::Click { .. } => "click", Self::Move { .. } => "move", Self::Drag { .. } => "drag", @@ -148,7 +152,7 @@ impl ComputerAction { } pub fn needs_screenshot_settle(&self) -> bool { - !matches!(self, Self::Screenshot | Self::Wait { .. }) + !matches!(self, Self::Screenshot | Self::Snapshot | Self::Wait { .. }) } fn points(&self) -> Vec<(i32, i32)> { @@ -232,11 +236,13 @@ pub enum ComputerStep { Xdotool(Vec), SleepMs(u32), Screenshot, + A11ySnapshot, } pub fn steps_for(action: &ComputerAction) -> Result> { match action { ComputerAction::Screenshot => Ok(vec![ComputerStep::Screenshot]), + ComputerAction::Snapshot => Ok(vec![ComputerStep::A11ySnapshot]), ComputerAction::Wait { duration_ms } => Ok(vec![ComputerStep::SleepMs(*duration_ms)]), ComputerAction::Type { text } if text.is_empty() => Ok(vec![]), ComputerAction::Type { text } => Ok(vec![ComputerStep::Xdotool(vec![ @@ -316,7 +322,8 @@ pub fn tool_definition() -> Value { let action_fields = json!({ "action": { "type": "string", - "enum": ["screenshot", "click", "move", "drag", "type", "key", "scroll", "wait"] + "enum": ["screenshot", "snapshot", "click", "move", "drag", "type", "key", "scroll", "wait"], + "description": "snapshot lists the on-screen UI elements (role, name, value, centre x/y) from the accessibility tree; click those centres instead of estimating pixels from the image." }, "x": {"type": "integer", "description": "Desktop pixel X (origin top-left)"}, "y": {"type": "integer", "description": "Desktop pixel Y"}, @@ -365,7 +372,7 @@ pub fn tool_definition() -> Value { "type": "function", "function": { "name": "computer", - "description": "Drive MY computer's desktop (screenshot, click, move, drag, type, key, scroll, wait). Only a computerUse subagent may call this. See-act-verify: read the screenshot returned after this call before the next one. `then` batches up to 9 follow-ups that need no intermediate check; a screenshot is always captured at the end. Coordinates are desktop pixels. Do not type passwords; if a human is needed, stop and report so the parent can call request_box_help.", + "description": "Drive MY computer's desktop (screenshot, snapshot, click, move, drag, type, key, scroll, wait). Only a computerUse subagent may call this. Start with snapshot: it returns the visible elements with exact centre coordinates (plus a screenshot), so clicks land on the element, not a guessed pixel. See-act-verify: read the screenshot returned after this call before the next one. `then` batches up to 9 follow-ups that need no intermediate check; a screenshot is always captured at the end. Coordinates are desktop pixels. Do not type passwords; if a human is needed, stop and report so the parent can call request_box_help.", "parameters": { "type": "object", "properties": primary, @@ -389,6 +396,7 @@ fn parse_action(args: &Value, follow_up: bool) -> Result { } match name { "screenshot" => Ok(ComputerAction::Screenshot), + "snapshot" | "a11y" | "elements" => Ok(ComputerAction::Snapshot), "click" => { let (x, y) = optional_point(args, "click")?; Ok(ComputerAction::Click { @@ -436,7 +444,7 @@ fn parse_action(args: &Value, follow_up: bool) -> Result { duration_ms: bounded_u32(args.get("durationMs"), 1_000, 0, MAX_WAIT_MS, "durationMs")?, }), other => Err(anyhow!( - "unknown computer action {other}; use screenshot, click, move, drag, type, key, scroll, or wait" + "unknown computer action {other}; use screenshot, snapshot, click, move, drag, type, key, scroll, or wait" )), } } @@ -634,6 +642,15 @@ mod tests { assert!(seq[2].is_screenshot()); } + #[test] + fn snapshot_reads_the_tree_then_screenshots_without_settling() { + let seq = parse_computer_call(&json!({"action": "snapshot"})).unwrap(); + assert_eq!(seq, vec![ComputerAction::Snapshot, ComputerAction::Screenshot]); + assert_eq!(settle_before(&seq, 1), 0); + assert_eq!(steps_for(&seq[0]).unwrap(), vec![ComputerStep::A11ySnapshot]); + assert_eq!(parse_computer_call(&json!({"action": "elements"})).unwrap()[0], ComputerAction::Snapshot); + } + #[test] fn screenshot_only_does_not_duplicate() { let seq = parse_computer_call(&json!({"action": "screenshot"})).unwrap(); diff --git a/crates/lazyboy-core/src/subagents.rs b/crates/lazyboy-core/src/subagents.rs index d3d0100..b8da087 100644 --- a/crates/lazyboy-core/src/subagents.rs +++ b/crates/lazyboy-core/src/subagents.rs @@ -20,9 +20,10 @@ pub const COMPUTER_USE_SYSTEM: &str = "\ OVERRIDE: you are a computerUse subagent. You DO have the computer tool and you SHOULD click, type, scroll, and key on MY computer's desktop. The parent restriction (\"screenshot is read-only\") does not apply to you. ## Computer -Drive this box's desktop with computer (screenshot, click, move, drag, type, key, scroll, wait): GUI apps, file dialogs, drag, and sites that defeat page-level automation. +Drive this box's desktop with computer (screenshot, snapshot, click, move, drag, type, key, scroll, wait): GUI apps, file dialogs, drag, and sites that defeat page-level automation. - Stay inside the task you were handed. Do that step and its success criteria, then stop. If it is bigger or more ambiguous than scoped, stop and report what you found. -- See-act-verify: screenshot to see the real state, act, then read the one fresh screenshot returned after the entire computer call before the next one. A batched `then` sequence returns only its final screen, so batch only steps that need no intermediate check. Never fire actions blind off a remembered layout. +- Start with `snapshot`, not a bare screenshot: it lists every visible element as `[id] role \"name\" @ (x,y) WxH` (buttons, links, textboxes with their current value, tabs, menus) straight from the app's accessibility tree, plus the screenshot. Click the listed (x,y) centre; only estimate pixels from the image for things the list does not show (canvas, images, custom widgets). Re-snapshot after the page changes. +- See-act-verify: snapshot/screenshot to see the real state, act, then read the one fresh screenshot returned after the entire computer call before the next one. A batched `then` sequence returns only its final screen, so batch only steps that need no intermediate check. Never fire actions blind off a remembered layout. - If the screen is mid-load, wait and re-screenshot rather than clicking a moving target. If a click missed, study the new screenshot and re-target; never type into a field you have not just focused. - Before typing into a field that may already hold text, key ctrl+a then BackSpace. Open Chrome with shell `box-chrome 'https://example.com'` or `box-chrome --new-window`, then confirm with one computer screenshot. Never launch another browser. - Your desktop is display :1 — the display computer screenshots and clicks — and Chromium's CDP is http://127.0.0.1:9222. Those are given facts. Do not probe other displays or ports. diff --git a/docs/SURFACES.md b/docs/SURFACES.md index a23a0e1..b54e406 100644 --- a/docs/SURFACES.md +++ b/docs/SURFACES.md @@ -60,6 +60,15 @@ Gateway 需要提供 JSON Connect `POST /aiserver.v1.AiService/RunWebSearch`（s 本次驗證：一般 Rust 測試 120 項通過；另以實際 Docker 通過 `computer_use_click_and_screenshot_on_box`（xdotool click + screenshot）。Clippy 通過。Box 映像 revision `computer-use-2` 含 xdotool 與 procps；`box-chrome` 以啟動鎖序列化 Chromium 冷啟動並清除跨 container 殘留的 `SingletonLock`，x11vnc 只監聽 localhost。 +Box 映像 revision `computer-use-6`：對齊 lazyBoy `image/computer` 的瀏覽器環境，避免蝦皮等站直接判定為機器人。容器與 `box-chrome` 設 `LANG/LC_ALL=zh_TW.UTF-8`、`LANGUAGE=zh_TW:zh:en`、`TZ=Asia/Taipei`（`locale-gen` zh_TW/en_US、`/etc/localtime` 指向台北）；字型補 `fonts-liberation`、`fonts-dejavu-core`、`fonts-noto-color-emoji`；Chromium 旗標改用 lazyBoy 的 `lazyboy-browser` 組合（`--lang=zh-TW --accept-lang=zh-TW,zh,en-US,en --start-maximized --test-type --disable-features=TranslateUI --disable-session-crashed-bubble --hide-crash-restore-bubble --disable-infobars`），另加 `--disable-blink-features=AutomationControlled`；`HOME=/home/box` 讓 Chromium 設定與 Crash Reports 留在 volume。 + +Box 映像 revision `computer-use-7`：桌面改成 lazyBoy `image/computer` 的樣子並瘦身、加上元件級操控。 +- 顯示層：一個 `Xtigervnc :1`（framebuffer + VNC 同一程序、事件驅動）取代 `Xvfb + x11vnc + autocutsel`；剪貼簿由 `vncconfig -nowin` 同步。noVNC/websockify、xdotool、`import` 截圖與 `DESKTOP_PROBE`（WM、5900、6080）不變，解析度仍 1280x720（可用 `LAZYBOY_SCREEN_GEOMETRY` 覆寫）。 +- 桌面：不再跑 `startxfce4`（xfce4-session／xfsettingsd／Thunar／自動開終端機都拿掉），只起 `xfconfd + xfwm4 --compositor=off + xfdesktop + xfce4-panel`。外觀沿用 lazyBoy：Arc-Dark、Papirus-Dark、jf open 粉圓、底部中文面板（應用程式／瀏覽器／終端機啟動器／工作列／時鐘）、同一張桌布；XFCE 設定每次開機從 `/usr/share/lazyboy/xfce-skel` 重灌到 `/home/box/.config`。 +- Chromium：只在 profile 第一次啟動時開 welcome 頁（之前 `--restore-last-session` 加 URL 會出現兩個相同分頁）；另加 `--force-renderer-accessibility --disable-background-networking --disable-component-update --renderer-process-limit=4`。 +- 元件級操控：`box-a11y`（python3 + AT-SPI）列出畫面上元件的角色／名稱／值／中心座標；`computer` 工具新增 `snapshot` 動作（`ComputerStep::A11ySnapshot`），回傳 `snapshot` 文字並照常附截圖。computerUse 提示改為先 `snapshot` 再點列出的座標。`docker exec` 啟動的程序透過 `/tmp/lazyboy/dbus.env` 加入桌面的 session bus 才能看到同一個 AT-SPI registry；`gsettings toolkit-accessibility=true`（dconf 在 `/home/box/.config`）讓 Chromium 願意輸出無障礙樹。 +- 量測：桌面＋Chromium 閒置約 400 MiB（原 486 MiB），`box-a11y` 對一個網頁約 0.2 秒。 + Box 執行期行為：截圖前會等 `SCREENSHOT_SETTLE_MS` 讓桌面重繪；每次截圖使用獨立的暫存檔，父 agent 的 `screenshot` 與 computerUse 子 agent 不會互相讀到對方的畫面。`shell` 輸出以 UTF-8 容錯讀取，超過上限時保留頭尾並在 `full_output` 給出可用 `read` 讀取的 `/tmp/gb-jobs//` 完整路徑。多個程序（CLI、browser helper、測試）同時初始化 box 時，以主機端 flock 序列化映像建置與 container 重建。 一般工具名 `shell`／`read`／`await_shell` 一律指向 box；`box_shell`／`box_read`／`box_await` 保留為舊 box 呼叫的別名。本地檔案與命令工具全部改成 `external_*`；舊模糊名稱如 `exec_command`／`read_file` 會報錯，不會隱含操作本地。Box 命令等待時間到後保留執行，以 `await_shell` 收取結果。關閉桌面瀏覽器不會阻止 shell/read 工作，下一次 browser 工具會重開原 profile。