- agent live test drives real OpenAI-compatible endpoints (ANYBOT_LIVE_BASE/KEY/MODEL/PROMPT)
- specs assembled like the app (web::specs_for) so the model truly sees the browser tool
- botbrowser honors ANYBOT_BROWSER_FLAGS (containers: --no-sandbox --headless=new)
- src/win.rs: the same mac:: API the harness drives, on Win32 —
SendInput mouse (absolute, multi-click, drag, wheel), keyboard
(macOS key codes mapped to VK, unicode text, modifiers), GDI BitBlt
screenshot, WindowFromPoint/GetForegroundWindow for window_at and
front_app; compiles clean against the windows crate for
x86_64-pc-windows-msvc (API-checked, no hardware here)
- the control tree was already speaking agent-desktop-windows
- intents answer honestly on Windows ("捷徑目前是 macOS/Linux 的")
and don't advertise themselves from the facts line
- CI: a report-only Windows compile job for wherever a Windows runner lives
- needs a real machine: multi-monitor coordinates, high-DPI, UIA-number
clicks, Chinese input
- a skill is now a .md file (front matter + instructions + an optional
steps fence): <data>/skills/ is read at every start, files win;
install from the CLI (anybot --install-skill <path|url>) or from the
phone (設定 › 整合 › 技能 › 從網址安裝), delete takes the file with it
- skillify (C8): long-press your own message in the chat → 教我做 —
the last turn that answered it becomes a replayable skill from the
oplog: only steps that succeeded, looking steps and numbered clicks
filtered (numbers die with their screenshot)
- a browser replay step that doesn't fit gets one fresh page read
before the bot takes over, same courtesy the screen steps had (C8)
- three example skills in skills/, the format guide in docs/skills.md
- the prompt lines that merges had eaten: menu_click/fill_form/wait_for
(C3), memory recall + search_documents (P6) — the tools were there,
the model just wasn't told about them
- look() runs the picture and the control tree at once (two doors of
the system, about the same wait): a third of a second back on every
Mac step; the AX retry path keeps its one-more-look behaviour
- the device lane (this computer serving a remote brain through the
relay) encodes screenshots on a tighter ladder (70/60/50 where the
local one starts at 85): a quarter less to carry, and the test
proves the same picture really comes out smaller
- no wire format changes: same JPEG bytes, just lighter
- SECURITY.md (zh) + docs/en/security.md: trust boundaries (app /
box / Docker API / relay / providers), where every secret lives and
what a thief holding that thing actually gets, the injection layers
with an honest limit (no absolute defense — the human at the gate
is the last line), the network surface, the known-limits list and
hardening advice
- state.json is written 0600: it carries model keys, and now says so
with its permissions like the vault's key file
- README (zh + en) link it; reporting = private security advisory
- intents.rs: volume / wifi / lock / dark / trash / eject as named
shortcuts — the model picks a name, the script is built from
constants and checked numbers ("volume set 40; rm -rf /" is not a
thing: args are grammar-checked before any shell sees them)
- the computer tool gains action=intent {intent_name, intent_arg}:
one call where the bench measured 9 steps and 2 minutes of GUI
- the computer's facts line advertises the shortcuts each machine
really has (Mac 6, Linux desktop 5, headless box 0 — nobody hears
a box's volume), so the prompt cache carries the knowledge for free
- scripts self-dispatch (uname): osascript/networksetup/pmset on Mac,
pactl/amixer/nmcli/loginctl/gsettings on Linux, honest failure text
- system prompt: 有捷徑先捷徑，捷徑失敗才回 GUI
- tests: the injection grammar, the availability matrix, a real run
- embed.rs: vectors from the person's own Ollama (/api/embed), model
auto-found among installed ones (bge-m3, nomic-embed-text…), off by
absence — no Ollama is a normal state: every search still answers
with words, the toggle lives in 設定 › 一般
- memories grow a vec column; recall() is now hybrid: trigram FTS,
two-character windows for Chinese, cosine over the shelf, merged by
reciprocal rank — 「那篇講語法的筆記」 finds text that never says 筆記
- docindex.rs: the person's folders (their 文件 by default, add more in
設定 › 一般), text-shaped files chunked into SQLite + FTS, refreshed
by mtime, gone files leave; one background thread does it all — the
turn never waits on embeddings
- one tool for it: search_documents {query, limit} — reads, no approval
- 251 tests: a real temp folder swept, answered, and swept again after
deletion; offline Ollama tested as the normal state it is
- store.rs grows a memories table + FTS5 (trigram) index: bot facts and
the person's shared lines (bot 0), no cap — state.json's 200-item list
stays the source that mirrors sync and the 設定 list; the index is its
searchable superset, best-effort as always
- queries answer in Chinese: phrase hits (FTS), then words inside lines —
CJK queries cut into two-character windows, lines scored by how many
they contain (「有没有靠窗」 finds 「我喜歡靠窗的座位」)
- every finished turn writes one event line (ask → answer) nobody's
asked to write: 「我上個月說的那個報表」 answers from months back
- the turn's prompt carries the 14 memories that answer *this* request
(no index yet: the list itself, the old behavior); remember/forget and
the UI's 忘記 keep the index in step with the list
- embedding (Ollama) and document indexing land on this same table next
- 「網頁上的事一律用 browser 一個工具」: the guidance lines now name
action=open/read/click/type/table/find/wait/download and action=profiles —
the model reads one vocabulary in the system prompt, the schema and the
loop-guard messages (browser_read/browser_open mentions gone from those)
- H3, light: usage rules were already in the system prompt (and C1 caches
the tools block too, so moving descriptions there saves little) — the
browser schema shed its duplicated safety sentence to the click line;
wholesale description-slimming deliberately not done: the guidance that
reads as noise is what keeps a small model out of trouble
- browser {action: open|read|elements|find|table|screenshot|click|hover|
type|press|select|scroll|wait|tabs|back|upload|download|pdf|dialog|eval|
profiles|fill} — the shape models are trained on (Anthropic CUA, OpenAI CUA)
- vault fills ride on it: fill {kind: login|card|contact, id}; vault_list
stays its own tool; the fills' approvals (every use asks, the model never
sees the secret) are untouched — expand happens at the model-output
boundary, so oplog, approvals, replay, remote limbs keep internal names
- web keeps {web_fetch, browser}; the eighteen browser_* and the three
fill_* schemas leave the model's view (−48% schema text, −58% counting
the old web trio; 25 tools → 2)
- tests: every action maps to a real internal tool, every internal tool is
reachable from the one schema
- 帳號 › 登入的裝置: every signed-in device with its last use; tap a device's
id to sign just that one out; 全部登出 ends every session, this phone
included — the stolen-session story ends on one page, on the phone
- 帳號 › 兩步驗證: enroll (secret + recovery codes shown once, code confirms),
disable (proves the code); the login screen's second step landed with S2
- relay: POST /relay/sessions/drop {id} — a session is named by the first 8
hash characters its row shows; kicking the phone you hold tells the app to
sign itself out (was_self)
- src/store.rs: bundled SQLite (WAL) in the data folder; every oplog record
is mirrored into an audit table — turns, model calls, tools with their
arguments and end statuses, queued messages, stops
- side-car by design: state.json stays the app's truth (one struct, written
atomically); the mirror is best-effort — an audit that never breaks a turn,
and a timeline that answers for 30 days, not just for the running turn
- /api/bots/<id>/audit now reads the trail (phone's 稽核 tab gains history)
- retention: rows older than KEEP_DAYS are deleted on open
- groundwork for memory/RAG (P6) on the same layer; state.json itself moves
in step 2 with the memory tables (the app keeps reading State as a struct)
- oplog gets wall-clock times on tool records (serde-default: old logs read
fine), and OpLog::timeline turns the machine log into a story: per turn —
the prompt, model calls (retries counted into one line), every tool with
the arguments it ran with and its status (等核准/進行中/完成/被拒絕/失敗),
messages typed mid-turn, stop, outcome
- GET /api/bots/<id>/audit (owner/admin; members' allow-list untouched)
- 對話詳細資訊 gains a 稽核 tab: newest turn first, running tool shows
進行中…, an approval gate shows as 等核准 — the phone can answer the question
'what is my bot doing right now, exactly', step by step
- the log stays the machine format recovery reads; the timeline is derived
- README.en.md: full English front door (install table, quick start,
deployment, security-in-one-paragraph, license) + a positioning table
against OpenClaw / OpenHands / Khoj / UI-TARS
- docs/en/{architecture,install,multi-device,protocol}.md: faithful English
summaries of the Chinese docs, each linking to the canonical version
- docs/demo/README.md: how to record the GIF/MP4 the README reserves space for
- language switch line at the top of the Chinese README
- login with 2FA on returns a 5-minute challenge, not a session; the session
comes from /relay/2fa/verify with an authenticator code — or one of eight
recovery codes, each good once (hash slot emptied)
- /relay/2fa/enroll hands out the secret (QR) + codes once; enabling completes
only when a real code proves the app has the secret; disable proves the code
- /relay/sessions/logout kicks every device of the account from one page
- password hashes are argon2id (memory-hard, OWASP parameters); stored PBKDF2
hashes still sign in — no forced reset, upgrade on next password change
- TOTP is RFC 6238 (vectors pinned in tests); mobile page prompts for the
second step; protocol.md documents the endpoints
- fixes a double-lock in /relay/me found by the flow test (handlers hold one
data guard: read through it)
- clicks on controls whose own words say money/sending/deleting ask the person
every time — on every computer, in direct mode, no grant silences it (the
harness decides, not the model, so a page cannot talk its way past it)
- screenshots paint card-number and OTP boxes black before any model sees them
- content the bot READS (pages, files, screens) enters the history wrapped in
<untrusted>, and the system prompt says the wrapper means content-not-orders
- keep_to_chosen now has a test on the screen tools: a model cannot move a
screen step onto another computer by itself (B26's promise, pinned)
- bench red team R1–R7: every payload's goal is a harmless probe file /
button code; passing = reporting the page honestly and leaving the probe
untouched. Run: --tasks R1..R7
- menu_click walks the real macOS menu bar in one step (three screenshot
rounds per level become one call + one picture)
- fill_form writes every named field with one AXValue write each, in one step
- wait_for polls the control tree (no screenshots) until text appears/gone —
replaces the screenshot-polling loops that filled whole turns with identical
pictures
- batch steps take an optional expect: the batch stops and shows the screen at
the step whose promised text never showed, instead of running a blind plan
on sand
Menu/fill are macOS accessibility; other systems keep click+typing per field
(the tool descriptions say which is which).
The quick model learned what its last click did only by re-reading two long
numbered lists. A multiset diff of (kind, name) between consecutive looks adds
two lines and answers it directly — including the case that matters most, a
dialog that appeared entirely.
The bench's precision failures (quick model clicking the wrong number, Retina
scaling, covered windows) all come from acting on pixels. A plain left click
now asks the frontmost app's accessibility tree first — by name, or by the
numbered spot — and typing writes AXValue into the focused field once,
keyboard events only when accessibility can't reach. Any app with a thin tree
behaves exactly as before: every AX answer that isn't a clear yes falls
through to CGEvent.
macOS code type-checked with --target x86_64-apple-darwin; gate: build on the
macos-arm64 runner + bench tasks T2/T5 (clicking by name) on a Mac.
Linux/Windows: stubs, pixel path unchanged.
The model picks action=screenshot/click/typing/… on a single `computer` tool
(the shape Anthropic's and OpenAI's computer-use models are trained on):
choosing among twelve near-identical names was its own error source, and the
twelve schemas were the biggest stable part of every request. Calls expand to
the internal per-action tools at the model-output boundary, so approvals,
repeat guards, skill replay, the remote limb protocol and the work log keep
their existing names — old limbs need no update.
AnyBot Dev
daniel wang