P4: human browser handoff + product notes

This commit is contained in:
王性驊 2026-09-13 16:15:27 +08:00
parent 829739e5b4
commit 5507e5b4b5
10 changed files with 538 additions and 25 deletions

View File

@ -1,6 +1,6 @@
# GrokBoy # GrokBoy
Minimal local **GrokBot-like** CLI agent. Phase **P3**: optional Playwright DOM browser tools. Minimal local **GrokBot-like** CLI agent. Phase **P4**: human browser handoff (login/OTP/captcha).
## Status ## Status
@ -10,8 +10,9 @@ Minimal local **GrokBot-like** CLI agent. Phase **P3**: optional Playwright DOM
| P1 shell / files + ReAct | done | | P1 shell / files + ReAct | done |
| P2 completion / loop guard / truncation | done | | P2 completion / loop guard / truncation | done |
| P3 browser (Playwright DOM) | done (optional) | | P3 browser (Playwright DOM) | done (optional) |
| P4 human handoff | done |
No Docker desktop, no Codex/LazyBoy fork. No Docker desktop, no Codex/LazyBoy fork. Product notes: [`docs/PRODUCT.md`](docs/PRODUCT.md).
## Setup (macOS) ## Setup (macOS)
@ -21,6 +22,7 @@ export GROKBOY_API_KEY=your_key # or XAI_API_KEY
# export GROKBOY_BASE_URL=https://api.x.ai/v1 # export GROKBOY_BASE_URL=https://api.x.ai/v1
# export GROKBOY_MODEL=grok-4.6 # export GROKBOY_MODEL=grok-4.6
# export GROKBOY_CONTEXT_CHARS=100000 # export GROKBOY_CONTEXT_CHARS=100000
# export GROKBOY_BROWSER_HEADED=1 # visible Chromium (recommended for handoff)
cd ~/GrokBoy cd ~/GrokBoy
cargo run -p grokboy -- chat cargo run -p grokboy -- chat
@ -36,29 +38,48 @@ npm install
npx playwright install chromium npx playwright install chromium
``` ```
Then the agent can use `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval` (DOM/selector/role — not screenshot-first). Then the agent can use `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`, and **`browser_handoff`** (DOM/selector/role — not screenshot-first).
### Human handoff (P4)
When the agent hits a login / OTP / captcha wall it calls `browser_handoff`:
1. Chromium opens **headed** (visible) — or relaunches headed if it was headless.
2. Terminal prints why it paused (ZH-TW + English) and what to do.
3. You complete the wall in the browser, then press **Enter** in that terminal (or type `abort`).
4. Agent resumes with a fresh **DOM snapshot**.
```bash
# Prefer headed for runs that may need handoff:
export GROKBOY_BROWSER_HEADED=1
cargo run -p grokboy -- run "打開需要登入的頁面並完成任務"
```
Tests / CI: `GROKBOY_HANDOFF_AUTO=1` auto-resumes (no interactive Enter).
## Commands ## Commands
- `grokboy chat` — interactive streaming chat (no tools) - `grokboy chat` — interactive streaming chat (no tools)
- `grokboy run "<prompt>"` — one-shot agent with tools - `grokboy run "<prompt>"` — one-shot agent with tools
- `grokboy run --session <id> "<prompt>"` — continue a saved session - `grokboy run --session <id> "<prompt>"` — continue a saved session
- `grokboy smoke` — offline checks (no API key / no Chromium required) - `grokboy smoke` — offline checks (no API key / no interactive handoff)
- `grokboy help` - `grokboy help`
Tools: `shell`, `list_dir`, `read_file`, `write_file`, `report_done`, `report_blocked`, Tools: `shell`, `list_dir`, `read_file`, `write_file`, `report_done`, `report_blocked`,
`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`. `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`,
`browser_handoff`.
Sessions are stored under `~/.grokboy/sessions/<id>.json` (may include `last_browser_url`). Sessions are stored under `~/.grokboy/sessions/<id>.json` (may include `last_browser_url`).
The agent stops on `report_done` / `report_blocked`, blocks identical tool rounds (×3), and truncates old context when over budget. The agent stops on `report_done` / `report_blocked`, blocks identical tool rounds (×3), and truncates old context when over budget.
## 繁體中文 ## Traditional Chinese
本機終端機 coding assistant。P3 可選 Playwright DOM 瀏覽器工具(非截圖優先) 本機終端機 coding assistant。P4 支援瀏覽器人工接手登入OTP驗證碼代理暫停 → 你在可見 Chromium 完成 → 終端機按 Enter 繼續
```bash ```bash
export GROKBOY_API_KEY=你的金鑰 export GROKBOY_API_KEY=你的金鑰
export GROKBOY_BROWSER_HEADED=1
cd ~/GrokBoy cd ~/GrokBoy
cargo run -p grokboy -- smoke cargo run -p grokboy -- smoke
# 可選瀏覽器: # 可選瀏覽器:
@ -73,5 +94,6 @@ cargo run -p grokboy -- chat
crates/grokboy-core/ # config, model, tools, browser, agent, session crates/grokboy-core/ # config, model, tools, browser, agent, session
crates/grokboy/ # CLI binary crates/grokboy/ # CLI binary
tools/playwright/ # optional Node Playwright helper (JSONL) tools/playwright/ # optional Node Playwright helper (JSONL)
docs/PRODUCT.md
docs/ACCEPTANCE.md docs/ACCEPTANCE.md
``` ```

View File

@ -15,8 +15,9 @@ pub const AGENT_SYSTEM: &str = "\
You are GrokBoy, a concise local coding assistant with tools. You are GrokBoy, a concise local coding assistant with tools.
Use tools when they help solve the task; otherwise answer directly. Use tools when they help solve the task; otherwise answer directly.
Prefer short, clear answers. Traditional Chinese is welcome when the user writes in Chinese. Prefer short, clear answers. Traditional Chinese is welcome when the user writes in Chinese.
Available tools: shell, list_dir, read_file, write_file, report_done, report_blocked, and optional browser_* (Playwright DOM: browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval). Available tools: shell, list_dir, read_file, write_file, report_done, report_blocked, and optional browser_* (Playwright DOM: browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval, browser_handoff).
For web pages prefer DOM snapshot + selector/role click/type not screenshots or pixel XY clicks. For web pages prefer DOM snapshot + selector/role click/type not screenshots or pixel XY clicks.
If you hit a login / OTP / captcha wall you cannot pass alone, call browser_handoff with a clear reason so the human can help in the visible browser, then continue from the returned snapshot.
When the task is finished, call report_done with a short summary. When the task is finished, call report_done with a short summary.
If you are stuck or cannot proceed, call report_blocked with the reason do not invent results or loop. If you are stuck or cannot proceed, call report_blocked with the reason do not invent results or loop.
Do not invent tool results call the tools."; Do not invent tool results call the tools.";
@ -314,6 +315,7 @@ mod tests {
assert!(AGENT_SYSTEM.contains("report_done")); assert!(AGENT_SYSTEM.contains("report_done"));
assert!(AGENT_SYSTEM.contains("report_blocked")); assert!(AGENT_SYSTEM.contains("report_blocked"));
assert!(AGENT_SYSTEM.contains("browser_navigate") || AGENT_SYSTEM.contains("browser_")); assert!(AGENT_SYSTEM.contains("browser_navigate") || AGENT_SYSTEM.contains("browser_"));
assert!(AGENT_SYSTEM.contains("browser_handoff"));
assert!(AGENT_SYSTEM.contains("Traditional Chinese") || AGENT_SYSTEM.contains("Chinese")); assert!(AGENT_SYSTEM.contains("Traditional Chinese") || AGENT_SYSTEM.contains("Chinese"));
} }

View File

@ -1,8 +1,12 @@
//! Optional Playwright DOM browser tools (P3). //! Optional Playwright DOM browser tools (P3) + human handoff (P4).
//! //!
//! Spawns `tools/playwright/browser_helper.mjs` (JSONL over stdin/stdout). //! Spawns `tools/playwright/browser_helper.mjs` (JSONL over stdin/stdout).
//! Fail closed with an install hint when Node / Playwright / Chromium is missing. //! Fail closed with an install hint when Node / Playwright / Chromium is missing.
//! Prefer selectors / roles — not screenshot-first control. //! Prefer selectors / roles — not screenshot-first control.
//!
//! Env:
//! - `GROKBOY_BROWSER_HEADED=1` — launch Chromium headed (visible) by default
//! - `GROKBOY_HANDOFF_AUTO=1` — auto-resume handoff (tests); `abort` to auto-abort
use anyhow::{Context, Result, anyhow}; use anyhow::{Context, Result, anyhow};
use serde_json::{Value, json}; use serde_json::{Value, json};
@ -10,6 +14,7 @@ use std::io::{BufRead, BufReader, Write};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use std::process::{Child, ChildStdin, ChildStdout, Command, Stdio}; use std::process::{Child, ChildStdin, ChildStdout, Command, Stdio};
use std::sync::{Arc, Mutex, OnceLock}; use std::sync::{Arc, Mutex, OnceLock};
use std::sync::mpsc;
use std::time::{Duration, Instant}; use std::time::{Duration, Instant};
pub const INSTALL_HINT: &str = "Playwright browser tools unavailable. Install with: \ pub const INSTALL_HINT: &str = "Playwright browser tools unavailable. Install with: \
@ -399,6 +404,27 @@ pub fn browser_tool_definitions() -> Vec<Value> {
} }
} }
}), }),
json!({
"type": "function",
"function": {
"name": "browser_handoff",
"description": "Pause for human help on auth walls (login, OTP, captcha). Shows a headed Chromium window, prints terminal instructions (ZH+EN), waits for Enter (or abort), then returns a DOM snapshot so the agent can continue. Prefer when automation cannot pass the wall.",
"parameters": {
"type": "object",
"properties": {
"reason": {
"type": "string",
"description": "Why human help is needed (e.g. login page, OTP, captcha)"
},
"timeout_secs": {
"type": "integer",
"description": "Seconds to wait for user (default 300)"
}
},
"required": ["reason"]
}
}
}),
] ]
} }
@ -411,6 +437,7 @@ pub fn is_browser_tool(name: &str) -> bool {
| "browser_click" | "browser_click"
| "browser_type" | "browser_type"
| "browser_eval" | "browser_eval"
| "browser_handoff"
) )
} }
@ -420,6 +447,10 @@ pub async fn execute_browser_tool(
name: &str, name: &str,
args: &Value, args: &Value,
) -> Result<Value> { ) -> Result<Value> {
if name == "browser_handoff" {
return execute_browser_handoff(cwd, last_url, args).await;
}
// Run blocking helper I/O off the async runtime. // Run blocking helper I/O off the async runtime.
let cwd = cwd.to_path_buf(); let cwd = cwd.to_path_buf();
let name = name.to_string(); let name = name.to_string();
@ -460,6 +491,185 @@ pub async fn execute_browser_tool(
.map_err(|e| anyhow!("browser task join: {e}"))? .map_err(|e| anyhow!("browser task join: {e}"))?
} }
/// Outcome of waiting for the human during handoff.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum HandoffWait {
Resumed,
Aborted(String),
TimedOut,
}
const DEFAULT_HANDOFF_TIMEOUT_SECS: u64 = 300;
/// Wait for stdin line (Enter to continue, `abort` to cancel) or timeout.
///
/// `GROKBOY_HANDOFF_AUTO=1|resume` skips the wait (tests).
/// `GROKBOY_HANDOFF_AUTO=abort` auto-aborts.
pub fn wait_for_handoff_resume(timeout_secs: u64) -> HandoffWait {
match std::env::var("GROKBOY_HANDOFF_AUTO") {
Ok(v) => {
let v = v.trim().to_ascii_lowercase();
if v == "1" || v == "true" || v == "yes" || v == "resume" || v == "continue"
{
return HandoffWait::Resumed;
}
if v == "abort" || v == "0" || v == "false" || v == "no" {
return HandoffWait::Aborted(format!("GROKBOY_HANDOFF_AUTO={v}"));
}
}
Err(_) => {}
}
let (tx, rx) = mpsc::channel::<Result<String, String>>();
std::thread::spawn(move || {
let stdin = std::io::stdin();
let mut line = String::new();
match stdin.lock().read_line(&mut line) {
Ok(0) => {
let _ = tx.send(Err("stdin closed (EOF)".into()));
}
Ok(_) => {
let _ = tx.send(Ok(line));
}
Err(e) => {
let _ = tx.send(Err(format!("stdin read error: {e}")));
}
}
});
match rx.recv_timeout(Duration::from_secs(timeout_secs.max(1))) {
Ok(Ok(line)) => {
let t = line.trim().to_ascii_lowercase();
if t == "abort" || t == "q" || t == "quit" || t == "cancel" {
HandoffWait::Aborted(format!("user typed {t}"))
} else {
// Empty line (Enter) or any other input → resume.
HandoffWait::Resumed
}
}
Ok(Err(msg)) => HandoffWait::Aborted(msg),
Err(mpsc::RecvTimeoutError::Timeout) => HandoffWait::TimedOut,
Err(mpsc::RecvTimeoutError::Disconnected) => {
HandoffWait::Aborted("handoff wait thread disconnected".into())
}
}
}
fn print_handoff_instructions(reason: &str, timeout_secs: u64) {
let banner = format!(
"
GrokBoy P4 Human handoff /
/ Why paused
{reason}
/ What to do
1. Chromium headed / visible
Look at the visible Chromium window.
2. OTP
Complete login / OTP / captcha (or whatever is blocking).
3. Enter
When done, return here and press Enter to continue.
4. abort Enter
To give up, type abort then Enter.
Timeout / : {timeout_secs}s
(tests: GROKBOY_HANDOFF_AUTO=1 to auto-resume)
waiting for Enter / Enter
"
);
eprintln!("{banner}");
let _ = std::io::Write::flush(&mut std::io::stderr());
}
/// `browser_handoff`: prepare headed browser, wait for human, return snapshot.
pub async fn execute_browser_handoff(
cwd: &Path,
last_url: &LastUrlSlot,
args: &Value,
) -> Result<Value> {
let reason = args
.get("reason")
.and_then(|v| v.as_str())
.map(str::trim)
.filter(|s| !s.is_empty())
.ok_or_else(|| anyhow!("browser_handoff: missing 'reason'"))?
.to_string();
let timeout_secs = args
.get("timeout_secs")
.and_then(|v| v.as_u64())
.filter(|&n| n > 0)
.unwrap_or(DEFAULT_HANDOFF_TIMEOUT_SECS);
let cwd = cwd.to_path_buf();
let last_url = last_url.clone();
tokio::task::spawn_blocking(move || {
// 1) Ensure headed Chromium (JSONL daemon keeps page state when already headed).
let prep = browser_request(
&cwd,
json!({
"op": "handoff_prepare",
"reason": reason,
}),
)?;
if prep.get("ok") != Some(&json!(true)) {
return Ok(response_to_tool_json(prep));
}
update_last_url(&last_url, &prep);
// 2) Instruct the human (bilingual).
print_handoff_instructions(&reason, timeout_secs);
// 3) Block until Enter / abort / timeout.
let wait = wait_for_handoff_resume(timeout_secs);
match wait {
HandoffWait::Aborted(msg) => {
return Ok(json!({
"status": "blocked",
"blocked": true,
"handoff": "aborted",
"reason": format!("human handoff aborted: {msg}"),
"original_reason": reason,
}));
}
HandoffWait::TimedOut => {
return Ok(json!({
"status": "blocked",
"blocked": true,
"handoff": "timeout",
"reason": format!(
"human handoff timed out after {timeout_secs}s (fail-closed)"
),
"original_reason": reason,
}));
}
HandoffWait::Resumed => {}
}
// 4) Snapshot so the model can continue from post-login DOM.
let snap = browser_request(&cwd, json!({ "op": "snapshot" }))?;
update_last_url(&last_url, &snap);
let mut out = response_to_tool_json(snap);
if let Some(obj) = out.as_object_mut() {
obj.insert("handoff".into(), json!("resumed"));
obj.insert("handoff_reason".into(), json!(reason));
obj.insert(
"message".into(),
json!("Human handoff resumed; DOM snapshot attached."),
);
}
Ok(out)
})
.await
.map_err(|e| anyhow!("browser handoff join: {e}"))?
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
@ -524,12 +734,44 @@ mod tests {
#[test] #[test]
fn browser_defs_count() { fn browser_defs_count() {
assert_eq!(browser_tool_definitions().len(), 5); assert_eq!(browser_tool_definitions().len(), 6);
assert!(is_browser_tool("browser_navigate")); assert!(is_browser_tool("browser_navigate"));
assert!(is_browser_tool("browser_snapshot")); assert!(is_browser_tool("browser_snapshot"));
assert!(is_browser_tool("browser_handoff"));
assert!(!is_browser_tool("shell")); assert!(!is_browser_tool("shell"));
} }
#[test]
fn handoff_auto_resume_and_abort() {
// SAFETY: tests run serially for this env in practice; restore after.
let prev = std::env::var("GROKBOY_HANDOFF_AUTO").ok();
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") };
assert_eq!(wait_for_handoff_resume(1), HandoffWait::Resumed);
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "abort") };
match wait_for_handoff_resume(1) {
HandoffWait::Aborted(msg) => assert!(msg.contains("abort"), "{msg}"),
other => panic!("expected aborted, got {other:?}"),
}
match prev {
Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) },
None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") },
}
}
#[test]
fn oneshot_ping_lists_handoff_prepare() {
let root = repo_cwd();
if which_node().is_none() {
return;
}
let resp = browser_oneshot(&root, &json!({"op": "ping", "id": "h1"})).unwrap();
let ops = resp["ops"].as_array().expect("ops");
assert!(
ops.iter().any(|o| o.as_str() == Some("handoff_prepare")),
"{resp}"
);
}
#[test] #[test]
fn install_hint_constant() { fn install_hint_constant() {
assert!(INSTALL_HINT.contains("npx playwright install chromium")); assert!(INSTALL_HINT.contains("npx playwright install chromium"));

View File

@ -12,7 +12,7 @@ pub use agent::{
context_char_budget, message_char_len, messages_char_len, round_signature, run_agent, context_char_budget, message_char_len, messages_char_len, round_signature, run_agent,
run_agent_with, tool_call_signature, truncate_messages, run_agent_with, tool_call_signature, truncate_messages,
}; };
pub use browser::{INSTALL_HINT as BROWSER_INSTALL_HINT, browser_oneshot, browser_self_test, find_helper_script}; pub use browser::{INSTALL_HINT as BROWSER_INSTALL_HINT, HandoffWait, browser_oneshot, browser_self_test, find_helper_script, wait_for_handoff_resume};
pub use config::Config; pub use config::Config;
pub use model::{ChatMessage, FunctionCall, Role, ToolCall, chat_completion, stream_chat}; pub use model::{ChatMessage, FunctionCall, Role, ToolCall, chat_completion, stream_chat};
pub use session::{Session, load_or_create, load_session, save_session, sessions_dir}; pub use session::{Session, load_or_create, load_session, save_session, sessions_dir};

View File

@ -1,4 +1,4 @@
//! Built-in tools: shell, files, completion, and optional Playwright browser (P3). //! Built-in tools: shell, files, completion, Playwright browser (P3), human handoff (P4).
use anyhow::{Context, Result, anyhow}; use anyhow::{Context, Result, anyhow};
use serde_json::{Value, json}; use serde_json::{Value, json};
@ -496,7 +496,7 @@ mod tests {
fn tool_defs_include_core_and_browser() { fn tool_defs_include_core_and_browser() {
let defs = tool_definitions(); let defs = tool_definitions();
let arr = defs.as_array().unwrap(); let arr = defs.as_array().unwrap();
assert_eq!(arr.len(), 11); // 6 core + 5 browser assert_eq!(arr.len(), 12); // 6 core + 6 browser (incl. handoff)
let names: Vec<&str> = arr let names: Vec<&str> = arr
.iter() .iter()
.map(|t| t["function"]["name"].as_str().unwrap()) .map(|t| t["function"]["name"].as_str().unwrap())
@ -512,6 +512,38 @@ mod tests {
assert!(names.contains(&"browser_click")); assert!(names.contains(&"browser_click"));
assert!(names.contains(&"browser_type")); assert!(names.contains(&"browser_type"));
assert!(names.contains(&"browser_eval")); assert!(names.contains(&"browser_eval"));
assert!(names.contains(&"browser_handoff"));
}
#[tokio::test]
async fn browser_handoff_auto_resume_protocol() {
// Offline: with GROKBOY_HANDOFF_AUTO=1, missing Chromium still fail-closes
// OR (if Chromium present) resumes and returns snapshot/error JSON — never hangs.
let prev = std::env::var("GROKBOY_HANDOFF_AUTO").ok();
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") };
let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..");
let root = root.canonicalize().unwrap_or(root);
let ctx = ToolContext::new(root);
let out = execute_tool(
&ctx,
"browser_handoff",
&json!({"reason": "unit test OTP wall", "timeout_secs": 5}).to_string(),
)
.await;
let v: Value = serde_json::from_str(&out).unwrap();
// Either resumed snapshot, or fail-closed blocked/error (no Chromium) — must not panic.
let okish = v.get("handoff").and_then(|h| h.as_str()) == Some("resumed")
|| v.get("error").is_some()
|| v.get("blocked") == Some(&json!(true));
assert!(okish, "unexpected handoff result: {v}");
// missing reason fails closed
let bad = execute_tool(&ctx, "browser_handoff", &json!({}).to_string()).await;
let bad: Value = serde_json::from_str(&bad).unwrap();
assert!(bad.get("error").is_some(), "{bad}");
match prev {
Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) },
None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") },
}
} }
#[tokio::test] #[tokio::test]

View File

@ -50,7 +50,7 @@ async fn run() -> Result<()> {
fn print_help() { fn print_help() {
println!( println!(
"\ "\
GrokBoy minimal local CLI agent (P3: optional Playwright DOM browser) GrokBoy minimal local CLI agent (P4: human browser handoff)
USAGE: USAGE:
grokboy chat Interactive streaming chat (no tools) grokboy chat Interactive streaming chat (no tools)
@ -65,11 +65,15 @@ ENV:
GROKBOY_BASE_URL default https://api.x.ai/v1 GROKBOY_BASE_URL default https://api.x.ai/v1
GROKBOY_MODEL default grok-4.6 GROKBOY_MODEL default grok-4.6
GROKBOY_CONTEXT_CHARS context budget (default 100000) GROKBOY_CONTEXT_CHARS context budget (default 100000)
GROKBOY_BROWSER_HEADED 1 = always launch Chromium headed (visible)
GROKBOY_HANDOFF_AUTO 1 = auto-resume handoff (tests); abort = auto-abort
Tools: shell, list_dir, read_file, write_file, report_done, report_blocked, Tools: shell, list_dir, read_file, write_file, report_done, report_blocked,
browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval,
browser_handoff
Sessions: ~/.grokboy/sessions/<id>.json Sessions: ~/.grokboy/sessions/<id>.json
Browser (optional): cd tools/playwright && npm i && npx playwright install chromium Browser (optional): cd tools/playwright && npm i && npx playwright install chromium
Handoff: agent pauses on login/OTP/captcha you fix in headed Chromium Enter
" "
); );
} }
@ -182,7 +186,7 @@ async fn cmd_run(args: &[String]) -> Result<()> {
} }
async fn cmd_smoke() -> Result<()> { async fn cmd_smoke() -> Result<()> {
println!("GrokBoy smoke (offline P3)…"); println!("GrokBoy smoke (offline P4)…");
let stamp = uuid_like(); let stamp = uuid_like();
let dir = std::env::temp_dir().join(format!("grokboy-smoke-{stamp}")); let dir = std::env::temp_dir().join(format!("grokboy-smoke-{stamp}"));
std::fs::create_dir_all(&dir).context("temp dir")?; std::fs::create_dir_all(&dir).context("temp dir")?;
@ -281,11 +285,20 @@ async fn cmd_smoke() -> Result<()> {
} }
println!(" completion ok"); println!(" completion ok");
// tool definitions present (6 core + 5 browser) // tool definitions present (6 core + 6 browser incl. handoff)
let defs = tool_definitions(); let defs = tool_definitions();
let n_tools = defs.as_array().map(|a| a.len()).unwrap_or(0); let n_tools = defs.as_array().map(|a| a.len()).unwrap_or(0);
if n_tools != 11 { if n_tools != 12 {
return Err(anyhow!("expected 11 tool defs, got {n_tools}")); return Err(anyhow!("expected 12 tool defs, got {n_tools}"));
}
let tool_names: Vec<&str> = defs
.as_array()
.unwrap()
.iter()
.filter_map(|t| t["function"]["name"].as_str())
.collect();
if !tool_names.contains(&"browser_handoff") {
return Err(anyhow!("browser_handoff missing from tool defs"));
} }
println!(" tool defs ok"); println!(" tool defs ok");
@ -314,6 +327,37 @@ async fn cmd_smoke() -> Result<()> {
println!(" browser fail ok"); println!(" browser fail ok");
} }
// handoff protocol (auto-resume; no interactive stdin)
// SAFETY: smoke is single-threaded for this env toggle.
let prev_auto = std::env::var("GROKBOY_HANDOFF_AUTO").ok();
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") };
let handoff = execute_tool(
&ToolContext::new(repo.clone()),
"browser_handoff",
&json!({"reason": "smoke handoff check", "timeout_secs": 3}).to_string(),
)
.await;
let handoff_v: serde_json::Value = serde_json::from_str(&handoff)?;
// Fail-closed without Chromium, or resumed with snapshot if browsers installed.
if handoff_v.get("error").is_none()
&& handoff_v.get("blocked").is_none()
&& handoff_v.get("handoff").and_then(|h| h.as_str()) != Some("resumed")
{
return Err(anyhow!("unexpected handoff smoke result: {handoff_v}"));
}
println!(" handoff auto ok");
// abort path
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "abort") };
let abort_wait = grokboy_core::wait_for_handoff_resume(1);
if !matches!(abort_wait, grokboy_core::HandoffWait::Aborted(_)) {
return Err(anyhow!("expected HandoffWait::Aborted, got {abort_wait:?}"));
}
println!(" handoff abort ok");
match prev_auto {
Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) },
None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") },
}
// context truncation // context truncation
let mut msgs = vec![ let mut msgs = vec![
ChatMessage::system(AGENT_SYSTEM), ChatMessage::system(AGENT_SYSTEM),

View File

@ -26,3 +26,14 @@
- [x] Thin Node helper under `tools/playwright/` (JSONL / one-shot JSON) - [x] Thin Node helper under `tools/playwright/` (JSONL / one-shot JSON)
- [x] Fail closed with install hint when Node/Playwright/Chromium missing - [x] Fail closed with install hint when Node/Playwright/Chromium missing
- [x] `cargo test` / `grokboy smoke` pass without Playwright browsers installed - [x] `cargo test` / `grokboy smoke` pass without Playwright browsers installed
## P4 — human browser handoff
- [x] Spec: `docs/PRODUCT.md` (north star, P0P3, P4, non-goals)
- [x] Tool `browser_handoff` (`reason` required, optional `timeout_secs`)
- [x] Headed Chromium for handoff (`handoff_prepare`; relaunch if was headless)
- [x] Bilingual (繁中 + EN) terminal instructions; Enter continue / `abort` / timeout → fail-closed
- [x] After resume: DOM snapshot returned as tool result
- [x] Env docs: `GROKBOY_BROWSER_HEADED`, `GROKBOY_HANDOFF_AUTO`
- [x] Wired into tool defs + agent system prompt; fail-closed if no browser
- [x] Offline tests / smoke without API key or interactive stdin (`GROKBOY_HANDOFF_AUTO`)
- [x] README status table updated

48
docs/PRODUCT.md Normal file
View File

@ -0,0 +1,48 @@
# GrokBoy product notes
**North star:** a local **Grok Botlike** agent — thin CLI core, tool-using ReAct loop, optional Playwright **DOM** browser (not screenshot-first). GrokBoy is the sole main line; LazyBoy is reference only (no fork).
## Done (P0P3)
| Phase | What |
|-------|------|
| **P0** | Streaming chat CLI (`grokboy chat`), xAI/OpenAI-compatible API |
| **P1** | Tools + ReAct (`shell`, files), sessions under `~/.grokboy/sessions/` |
| **P2** | Completion contract (`report_done` / `report_blocked`), loop guard, context truncation |
| **P3** | Optional Playwright DOM tools: navigate / snapshot / click / type / eval (fail-closed) |
## P4 — Human handoff (this slice)
Auth walls (login, OTP, captcha) often cannot be automated safely. P4 adds **`browser_handoff`**:
1. Agent calls `browser_handoff` with a `reason` (optional `timeout_secs`).
2. Helper ensures Chromium is **headed** (visible); may relaunch from headless and restore URL.
3. Terminal prints bilingual (繁中 + English) instructions.
4. Loop **blocks** until you press **Enter** (continue) or type **`abort`**, or timeout → fail-closed blocked.
5. On resume, a **DOM snapshot** is returned so the model can continue.
### Env
- `GROKBOY_BROWSER_HEADED=1` — always launch Chromium headed (recommended when handoff is likely).
- `GROKBOY_HANDOFF_AUTO=1` — auto-resume (tests / CI); `abort` to auto-abort.
### Non-goals (this slice)
- Multi-agent orchestration
- Desktop accessibility / native UI automation
- External connectors / SaaS integrations
- Forking LazyBoy or Codex
### Acceptance (summary)
See `docs/ACCEPTANCE.md` section P4. Short list:
- `browser_handoff` registered and wired into the agent loop
- Headed Chromium for handoff; JSONL daemon keeps state within one run when already headed
- Bilingual terminal prompt; Enter / abort / timeout fail-closed
- Post-resume DOM snapshot as tool result
- Offline `cargo test` / `grokboy smoke` without API key or interactive stdin
## Roadmap hint (later)
P5+ may deepen persistence, richer session UX, or more tools — still thin core, DOM-first browser.

View File

@ -1,6 +1,6 @@
# GrokBoy Playwright helper (optional) # GrokBoy Playwright helper (optional)
Thin Node script used by Rust `browser_*` tools. Thin Node script used by Rust `browser_*` tools (P3) and human handoff (P4).
## Install ## Install
@ -16,6 +16,14 @@ npx playwright install chromium
- **One-shot**: `node browser_helper.mjs --cmd '{"op":"ping"}'` - **One-shot**: `node browser_helper.mjs --cmd '{"op":"ping"}'`
- **Self-test** (no Chromium): `npm run self-test` - **Self-test** (no Chromium): `npm run self-test`
Ops: `ping`, `navigate`, `snapshot`/`dom`, `click`, `type`, `eval`, `close`, `status`. Ops: `ping`, `navigate`, `snapshot`/`dom`, `click`, `type`, `eval`, `close`, `status`, **`handoff_prepare`**.
### `handoff_prepare`
Ensures a **headed** (visible) Chromium window and brings it to front. If the daemon was headless, closes and relaunches headed, then navigates back to the last URL when possible.
Note: relaunch uses a new browser context (cookies from the prior headless session are not carried unless storage state is added later). Prefer `GROKBOY_BROWSER_HEADED=1` so the first launch is already headed and state survives across handoff within one run.
Env: `GROKBOY_BROWSER_HEADED=1` launches headed for normal navigate as well.
Prefer CSS selector or `role`+`name` over screenshots. Prefer CSS selector or `role`+`name` over screenshots.

View File

@ -7,7 +7,7 @@
* One-shot: node browser_helper.mjs --cmd '<json>' * One-shot: node browser_helper.mjs --cmd '<json>'
* Self-test: node browser_helper.mjs --self-test (no Chromium required) * Self-test: node browser_helper.mjs --self-test (no Chromium required)
* *
* Ops: ping, navigate, snapshot, click, type, eval, close, status * Ops: ping, navigate, snapshot, click, type, eval, close, status, handoff_prepare
* Prefer CSS selector or role+name. Screenshots are NOT the primary control surface. * Prefer CSS selector or role+name. Screenshots are NOT the primary control surface.
*/ */
@ -25,6 +25,20 @@ let browser = null;
let context = null; let context = null;
let page = null; let page = null;
let lastUrl = null; let lastUrl = null;
/** Whether the current browser was launched headed (visible). */
let browserHeaded = false;
function envHeadedDefault() {
const v = (process.env.GROKBOY_BROWSER_HEADED || "").trim().toLowerCase();
if (v === "1" || v === "true" || v === "yes") return true;
if (v === "0" || v === "false" || v === "no") return false;
return false;
}
function wantHeaded(forceHeaded) {
if (forceHeaded === true) return true;
return envHeadedDefault();
}
function ok(id, extra = {}) { function ok(id, extra = {}) {
return { id: id ?? null, ok: true, ...extra }; return { id: id ?? null, ok: true, ...extra };
@ -50,11 +64,11 @@ async function loadPlaywright() {
} }
} }
async function ensurePage() { async function launchBrowser(headed) {
if (page) return page;
const { chromium } = await loadPlaywright(); const { chromium } = await loadPlaywright();
try { try {
browser = await chromium.launch({ headless: true }); browser = await chromium.launch({ headless: !headed });
browserHeaded = headed;
} catch (e) { } catch (e) {
const err = new Error( const err = new Error(
`${INSTALL_HINT} (chromium launch failed: ${e.message})` `${INSTALL_HINT} (chromium launch failed: ${e.message})`
@ -67,6 +81,78 @@ async function ensurePage() {
return page; return page;
} }
async function ensurePage(opts = {}) {
const headed = wantHeaded(opts.headed === true);
if (page) {
// Already open; if caller needs headed and we are headless, relaunch below via handoff_prepare.
return page;
}
return launchBrowser(headed);
}
/** Close current browser and reopen headed, restoring lastUrl when possible. */
async function ensureHeadedForHandoff() {
const restore = lastUrl || (page ? page.url() : null);
if (browser && browserHeaded && page) {
try {
await page.bringToFront();
} catch {
/* ignore */
}
return {
prepared: true,
headed: true,
relaunched: false,
url: page.url(),
restored: false,
};
}
if (browser) {
await browser.close().catch(() => {});
browser = null;
context = null;
page = null;
browserHeaded = false;
}
await launchBrowser(true);
let restored = false;
if (restore && restore !== "about:blank") {
try {
await page.goto(String(restore), {
waitUntil: "domcontentloaded",
timeout: 30000,
});
lastUrl = page.url();
restored = true;
} catch (e) {
// Keep headed page even if restore fails; caller still sees visible window.
lastUrl = page.url();
return {
prepared: true,
headed: true,
relaunched: true,
url: lastUrl,
restored: false,
restore_error: String(e.message || e),
};
}
}
try {
await page.bringToFront();
} catch {
/* ignore */
}
return {
prepared: true,
headed: true,
relaunched: true,
url: page ? page.url() : null,
restored,
};
}
/** Accessibility-ish text snapshot: roles, names, and useful selectors — not pixels. */ /** Accessibility-ish text snapshot: roles, names, and useful selectors — not pixels. */
async function buildSnapshot(p) { async function buildSnapshot(p) {
const nodes = await p.evaluate(() => { const nodes = await p.evaluate(() => {
@ -242,14 +328,27 @@ async function handle(req) {
"eval", "eval",
"close", "close",
"status", "status",
"handoff_prepare",
], ],
}); });
case "status": { case "status": {
return ok(id, { return ok(id, {
browser_open: !!browser, browser_open: !!browser,
headed: browserHeaded,
last_url: lastUrl, last_url: lastUrl,
page_url: page ? page.url() : null, page_url: page ? page.url() : null,
env_headed: envHeadedDefault(),
});
}
case "handoff_prepare": {
// Bring a visible Chromium window forward for human login/OTP/captcha.
const info = await ensureHeadedForHandoff();
return ok(id, {
...info,
message:
"Browser is headed (visible). Complete login/OTP/captcha in the window, then resume in the terminal.",
}); });
} }
@ -342,6 +441,7 @@ async function handle(req) {
browser = null; browser = null;
context = null; context = null;
page = null; page = null;
browserHeaded = false;
return ok(id, { closed: true, last_url: lastUrl }); return ok(id, { closed: true, last_url: lastUrl });
} }
@ -375,6 +475,10 @@ async function runSelfTest() {
// Protocol: ping without playwright // Protocol: ping without playwright
let r = await handle({ id: "t1", op: "ping" }); let r = await handle({ id: "t1", op: "ping" });
cases.push(["ping", r.ok === true && r.pong === true]); cases.push(["ping", r.ok === true && r.pong === true]);
cases.push([
"ping_lists_handoff_prepare",
Array.isArray(r.ops) && r.ops.includes("handoff_prepare"),
]);
// Missing fields fail closed // Missing fields fail closed
r = await handle({ id: "t2", op: "navigate" }); r = await handle({ id: "t2", op: "navigate" });