diff --git a/README.md b/README.md index d085849..2c31661 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # GrokBoy -Minimal local **GrokBot-like** CLI agent. Phase **P3**: optional Playwright DOM browser tools. +Minimal local **GrokBot-like** CLI agent. Phase **P4**: human browser handoff (login/OTP/captcha). ## Status @@ -10,8 +10,9 @@ Minimal local **GrokBot-like** CLI agent. Phase **P3**: optional Playwright DOM | P1 shell / files + ReAct | done | | P2 completion / loop guard / truncation | done | | P3 browser (Playwright DOM) | done (optional) | +| P4 human handoff | done | -No Docker desktop, no Codex/LazyBoy fork. +No Docker desktop, no Codex/LazyBoy fork. Product notes: [`docs/PRODUCT.md`](docs/PRODUCT.md). ## Setup (macOS) @@ -21,6 +22,7 @@ export GROKBOY_API_KEY=your_key # or XAI_API_KEY # export GROKBOY_BASE_URL=https://api.x.ai/v1 # export GROKBOY_MODEL=grok-4.6 # export GROKBOY_CONTEXT_CHARS=100000 +# export GROKBOY_BROWSER_HEADED=1 # visible Chromium (recommended for handoff) cd ~/GrokBoy cargo run -p grokboy -- chat @@ -36,29 +38,48 @@ npm install npx playwright install chromium ``` -Then the agent can use `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval` (DOM/selector/role — not screenshot-first). +Then the agent can use `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`, and **`browser_handoff`** (DOM/selector/role — not screenshot-first). + +### Human handoff (P4) + +When the agent hits a login / OTP / captcha wall it calls `browser_handoff`: + +1. Chromium opens **headed** (visible) — or relaunches headed if it was headless. +2. Terminal prints why it paused (ZH-TW + English) and what to do. +3. You complete the wall in the browser, then press **Enter** in that terminal (or type `abort`). +4. Agent resumes with a fresh **DOM snapshot**. + +```bash +# Prefer headed for runs that may need handoff: +export GROKBOY_BROWSER_HEADED=1 +cargo run -p grokboy -- run "打開需要登入的頁面並完成任務" +``` + +Tests / CI: `GROKBOY_HANDOFF_AUTO=1` auto-resumes (no interactive Enter). ## Commands - `grokboy chat` — interactive streaming chat (no tools) - `grokboy run ""` — one-shot agent with tools - `grokboy run --session ""` — continue a saved session -- `grokboy smoke` — offline checks (no API key / no Chromium required) +- `grokboy smoke` — offline checks (no API key / no interactive handoff) - `grokboy help` Tools: `shell`, `list_dir`, `read_file`, `write_file`, `report_done`, `report_blocked`, -`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`. +`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`, +`browser_handoff`. Sessions are stored under `~/.grokboy/sessions/.json` (may include `last_browser_url`). The agent stops on `report_done` / `report_blocked`, blocks identical tool rounds (×3), and truncates old context when over budget. -## 繁體中文 +## Traditional Chinese -本機終端機 coding assistant。P3 可選 Playwright DOM 瀏覽器工具(非截圖優先)。 +本機終端機 coding assistant。P4 支援瀏覽器人工接手(登入/OTP/驗證碼):代理暫停 → 你在可見 Chromium 完成 → 終端機按 Enter 繼續。 ```bash export GROKBOY_API_KEY=你的金鑰 +export GROKBOY_BROWSER_HEADED=1 cd ~/GrokBoy cargo run -p grokboy -- smoke # 可選瀏覽器: @@ -73,5 +94,6 @@ cargo run -p grokboy -- chat crates/grokboy-core/ # config, model, tools, browser, agent, session crates/grokboy/ # CLI binary tools/playwright/ # optional Node Playwright helper (JSONL) +docs/PRODUCT.md docs/ACCEPTANCE.md ``` diff --git a/crates/grokboy-core/src/agent.rs b/crates/grokboy-core/src/agent.rs index 2b04f1e..5fbb984 100644 --- a/crates/grokboy-core/src/agent.rs +++ b/crates/grokboy-core/src/agent.rs @@ -15,8 +15,9 @@ pub const AGENT_SYSTEM: &str = "\ You are GrokBoy, a concise local coding assistant with tools. Use tools when they help solve the task; otherwise answer directly. Prefer short, clear answers. Traditional Chinese is welcome when the user writes in Chinese. -Available tools: shell, list_dir, read_file, write_file, report_done, report_blocked, and optional browser_* (Playwright DOM: browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval). +Available tools: shell, list_dir, read_file, write_file, report_done, report_blocked, and optional browser_* (Playwright DOM: browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval, browser_handoff). For web pages prefer DOM snapshot + selector/role click/type — not screenshots or pixel XY clicks. +If you hit a login / OTP / captcha wall you cannot pass alone, call browser_handoff with a clear reason so the human can help in the visible browser, then continue from the returned snapshot. When the task is finished, call report_done with a short summary. If you are stuck or cannot proceed, call report_blocked with the reason — do not invent results or loop. Do not invent tool results — call the tools."; @@ -314,6 +315,7 @@ mod tests { assert!(AGENT_SYSTEM.contains("report_done")); assert!(AGENT_SYSTEM.contains("report_blocked")); assert!(AGENT_SYSTEM.contains("browser_navigate") || AGENT_SYSTEM.contains("browser_")); + assert!(AGENT_SYSTEM.contains("browser_handoff")); assert!(AGENT_SYSTEM.contains("Traditional Chinese") || AGENT_SYSTEM.contains("Chinese")); } diff --git a/crates/grokboy-core/src/browser.rs b/crates/grokboy-core/src/browser.rs index 77623da..dcb8812 100644 --- a/crates/grokboy-core/src/browser.rs +++ b/crates/grokboy-core/src/browser.rs @@ -1,8 +1,12 @@ -//! Optional Playwright DOM browser tools (P3). +//! Optional Playwright DOM browser tools (P3) + human handoff (P4). //! //! Spawns `tools/playwright/browser_helper.mjs` (JSONL over stdin/stdout). //! Fail closed with an install hint when Node / Playwright / Chromium is missing. //! Prefer selectors / roles — not screenshot-first control. +//! +//! Env: +//! - `GROKBOY_BROWSER_HEADED=1` — launch Chromium headed (visible) by default +//! - `GROKBOY_HANDOFF_AUTO=1` — auto-resume handoff (tests); `abort` to auto-abort use anyhow::{Context, Result, anyhow}; use serde_json::{Value, json}; @@ -10,6 +14,7 @@ use std::io::{BufRead, BufReader, Write}; use std::path::{Path, PathBuf}; use std::process::{Child, ChildStdin, ChildStdout, Command, Stdio}; use std::sync::{Arc, Mutex, OnceLock}; +use std::sync::mpsc; use std::time::{Duration, Instant}; pub const INSTALL_HINT: &str = "Playwright browser tools unavailable. Install with: \ @@ -399,6 +404,27 @@ pub fn browser_tool_definitions() -> Vec { } } }), + json!({ + "type": "function", + "function": { + "name": "browser_handoff", + "description": "Pause for human help on auth walls (login, OTP, captcha). Shows a headed Chromium window, prints terminal instructions (ZH+EN), waits for Enter (or abort), then returns a DOM snapshot so the agent can continue. Prefer when automation cannot pass the wall.", + "parameters": { + "type": "object", + "properties": { + "reason": { + "type": "string", + "description": "Why human help is needed (e.g. login page, OTP, captcha)" + }, + "timeout_secs": { + "type": "integer", + "description": "Seconds to wait for user (default 300)" + } + }, + "required": ["reason"] + } + } + }), ] } @@ -411,6 +437,7 @@ pub fn is_browser_tool(name: &str) -> bool { | "browser_click" | "browser_type" | "browser_eval" + | "browser_handoff" ) } @@ -420,6 +447,10 @@ pub async fn execute_browser_tool( name: &str, args: &Value, ) -> Result { + if name == "browser_handoff" { + return execute_browser_handoff(cwd, last_url, args).await; + } + // Run blocking helper I/O off the async runtime. let cwd = cwd.to_path_buf(); let name = name.to_string(); @@ -460,6 +491,185 @@ pub async fn execute_browser_tool( .map_err(|e| anyhow!("browser task join: {e}"))? } +/// Outcome of waiting for the human during handoff. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum HandoffWait { + Resumed, + Aborted(String), + TimedOut, +} + +const DEFAULT_HANDOFF_TIMEOUT_SECS: u64 = 300; + +/// Wait for stdin line (Enter to continue, `abort` to cancel) or timeout. +/// +/// `GROKBOY_HANDOFF_AUTO=1|resume` skips the wait (tests). +/// `GROKBOY_HANDOFF_AUTO=abort` auto-aborts. +pub fn wait_for_handoff_resume(timeout_secs: u64) -> HandoffWait { + match std::env::var("GROKBOY_HANDOFF_AUTO") { + Ok(v) => { + let v = v.trim().to_ascii_lowercase(); + if v == "1" || v == "true" || v == "yes" || v == "resume" || v == "continue" + { + return HandoffWait::Resumed; + } + if v == "abort" || v == "0" || v == "false" || v == "no" { + return HandoffWait::Aborted(format!("GROKBOY_HANDOFF_AUTO={v}")); + } + } + Err(_) => {} + } + + let (tx, rx) = mpsc::channel::>(); + std::thread::spawn(move || { + let stdin = std::io::stdin(); + let mut line = String::new(); + match stdin.lock().read_line(&mut line) { + Ok(0) => { + let _ = tx.send(Err("stdin closed (EOF)".into())); + } + Ok(_) => { + let _ = tx.send(Ok(line)); + } + Err(e) => { + let _ = tx.send(Err(format!("stdin read error: {e}"))); + } + } + }); + + match rx.recv_timeout(Duration::from_secs(timeout_secs.max(1))) { + Ok(Ok(line)) => { + let t = line.trim().to_ascii_lowercase(); + if t == "abort" || t == "q" || t == "quit" || t == "cancel" { + HandoffWait::Aborted(format!("user typed {t}")) + } else { + // Empty line (Enter) or any other input → resume. + HandoffWait::Resumed + } + } + Ok(Err(msg)) => HandoffWait::Aborted(msg), + Err(mpsc::RecvTimeoutError::Timeout) => HandoffWait::TimedOut, + Err(mpsc::RecvTimeoutError::Disconnected) => { + HandoffWait::Aborted("handoff wait thread disconnected".into()) + } + } +} + +fn print_handoff_instructions(reason: &str, timeout_secs: u64) { + let banner = format!( + " +╔══════════════════════════════════════════════════════════════╗ +║ GrokBoy P4 — Human handoff / 人工接手 ║ +╚══════════════════════════════════════════════════════════════╝ + +【為什麼暫停 / Why paused】 + {reason} + +【請你做什麼 / What to do】 + 1. 看著已開啟的 Chromium 視窗(headed / visible)。 + Look at the visible Chromium window. + 2. 完成登入、OTP、驗證碼或其他真人操作。 + Complete login / OTP / captcha (or whatever is blocking). + 3. 完成後回到這個終端機,按 Enter 繼續。 + When done, return here and press Enter to continue. + 4. 若要放棄,輸入 abort 再按 Enter。 + To give up, type abort then Enter. + + Timeout / 逾時: {timeout_secs}s + (tests: GROKBOY_HANDOFF_AUTO=1 to auto-resume) + +─── waiting for Enter / 等待 Enter ─── +" + ); + eprintln!("{banner}"); + let _ = std::io::Write::flush(&mut std::io::stderr()); +} + +/// `browser_handoff`: prepare headed browser, wait for human, return snapshot. +pub async fn execute_browser_handoff( + cwd: &Path, + last_url: &LastUrlSlot, + args: &Value, +) -> Result { + let reason = args + .get("reason") + .and_then(|v| v.as_str()) + .map(str::trim) + .filter(|s| !s.is_empty()) + .ok_or_else(|| anyhow!("browser_handoff: missing 'reason'"))? + .to_string(); + + let timeout_secs = args + .get("timeout_secs") + .and_then(|v| v.as_u64()) + .filter(|&n| n > 0) + .unwrap_or(DEFAULT_HANDOFF_TIMEOUT_SECS); + + let cwd = cwd.to_path_buf(); + let last_url = last_url.clone(); + + tokio::task::spawn_blocking(move || { + // 1) Ensure headed Chromium (JSONL daemon keeps page state when already headed). + let prep = browser_request( + &cwd, + json!({ + "op": "handoff_prepare", + "reason": reason, + }), + )?; + + if prep.get("ok") != Some(&json!(true)) { + return Ok(response_to_tool_json(prep)); + } + update_last_url(&last_url, &prep); + + // 2) Instruct the human (bilingual). + print_handoff_instructions(&reason, timeout_secs); + + // 3) Block until Enter / abort / timeout. + let wait = wait_for_handoff_resume(timeout_secs); + match wait { + HandoffWait::Aborted(msg) => { + return Ok(json!({ + "status": "blocked", + "blocked": true, + "handoff": "aborted", + "reason": format!("human handoff aborted: {msg}"), + "original_reason": reason, + })); + } + HandoffWait::TimedOut => { + return Ok(json!({ + "status": "blocked", + "blocked": true, + "handoff": "timeout", + "reason": format!( + "human handoff timed out after {timeout_secs}s (fail-closed)" + ), + "original_reason": reason, + })); + } + HandoffWait::Resumed => {} + } + + // 4) Snapshot so the model can continue from post-login DOM. + let snap = browser_request(&cwd, json!({ "op": "snapshot" }))?; + update_last_url(&last_url, &snap); + let mut out = response_to_tool_json(snap); + if let Some(obj) = out.as_object_mut() { + obj.insert("handoff".into(), json!("resumed")); + obj.insert("handoff_reason".into(), json!(reason)); + obj.insert( + "message".into(), + json!("Human handoff resumed; DOM snapshot attached."), + ); + } + Ok(out) + }) + .await + .map_err(|e| anyhow!("browser handoff join: {e}"))? +} + #[cfg(test)] mod tests { use super::*; @@ -524,12 +734,44 @@ mod tests { #[test] fn browser_defs_count() { - assert_eq!(browser_tool_definitions().len(), 5); + assert_eq!(browser_tool_definitions().len(), 6); assert!(is_browser_tool("browser_navigate")); assert!(is_browser_tool("browser_snapshot")); + assert!(is_browser_tool("browser_handoff")); assert!(!is_browser_tool("shell")); } + #[test] + fn handoff_auto_resume_and_abort() { + // SAFETY: tests run serially for this env in practice; restore after. + let prev = std::env::var("GROKBOY_HANDOFF_AUTO").ok(); + unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") }; + assert_eq!(wait_for_handoff_resume(1), HandoffWait::Resumed); + unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "abort") }; + match wait_for_handoff_resume(1) { + HandoffWait::Aborted(msg) => assert!(msg.contains("abort"), "{msg}"), + other => panic!("expected aborted, got {other:?}"), + } + match prev { + Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) }, + None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") }, + } + } + + #[test] + fn oneshot_ping_lists_handoff_prepare() { + let root = repo_cwd(); + if which_node().is_none() { + return; + } + let resp = browser_oneshot(&root, &json!({"op": "ping", "id": "h1"})).unwrap(); + let ops = resp["ops"].as_array().expect("ops"); + assert!( + ops.iter().any(|o| o.as_str() == Some("handoff_prepare")), + "{resp}" + ); + } + #[test] fn install_hint_constant() { assert!(INSTALL_HINT.contains("npx playwright install chromium")); diff --git a/crates/grokboy-core/src/lib.rs b/crates/grokboy-core/src/lib.rs index bbbf59a..f092825 100644 --- a/crates/grokboy-core/src/lib.rs +++ b/crates/grokboy-core/src/lib.rs @@ -12,7 +12,7 @@ pub use agent::{ context_char_budget, message_char_len, messages_char_len, round_signature, run_agent, run_agent_with, tool_call_signature, truncate_messages, }; -pub use browser::{INSTALL_HINT as BROWSER_INSTALL_HINT, browser_oneshot, browser_self_test, find_helper_script}; +pub use browser::{INSTALL_HINT as BROWSER_INSTALL_HINT, HandoffWait, browser_oneshot, browser_self_test, find_helper_script, wait_for_handoff_resume}; pub use config::Config; pub use model::{ChatMessage, FunctionCall, Role, ToolCall, chat_completion, stream_chat}; pub use session::{Session, load_or_create, load_session, save_session, sessions_dir}; diff --git a/crates/grokboy-core/src/tools.rs b/crates/grokboy-core/src/tools.rs index 713a49a..3600f15 100644 --- a/crates/grokboy-core/src/tools.rs +++ b/crates/grokboy-core/src/tools.rs @@ -1,4 +1,4 @@ -//! Built-in tools: shell, files, completion, and optional Playwright browser (P3). +//! Built-in tools: shell, files, completion, Playwright browser (P3), human handoff (P4). use anyhow::{Context, Result, anyhow}; use serde_json::{Value, json}; @@ -496,7 +496,7 @@ mod tests { fn tool_defs_include_core_and_browser() { let defs = tool_definitions(); let arr = defs.as_array().unwrap(); - assert_eq!(arr.len(), 11); // 6 core + 5 browser + assert_eq!(arr.len(), 12); // 6 core + 6 browser (incl. handoff) let names: Vec<&str> = arr .iter() .map(|t| t["function"]["name"].as_str().unwrap()) @@ -512,6 +512,38 @@ mod tests { assert!(names.contains(&"browser_click")); assert!(names.contains(&"browser_type")); assert!(names.contains(&"browser_eval")); + assert!(names.contains(&"browser_handoff")); + } + + #[tokio::test] + async fn browser_handoff_auto_resume_protocol() { + // Offline: with GROKBOY_HANDOFF_AUTO=1, missing Chromium still fail-closes + // OR (if Chromium present) resumes and returns snapshot/error JSON — never hangs. + let prev = std::env::var("GROKBOY_HANDOFF_AUTO").ok(); + unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") }; + let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../.."); + let root = root.canonicalize().unwrap_or(root); + let ctx = ToolContext::new(root); + let out = execute_tool( + &ctx, + "browser_handoff", + &json!({"reason": "unit test OTP wall", "timeout_secs": 5}).to_string(), + ) + .await; + let v: Value = serde_json::from_str(&out).unwrap(); + // Either resumed snapshot, or fail-closed blocked/error (no Chromium) — must not panic. + let okish = v.get("handoff").and_then(|h| h.as_str()) == Some("resumed") + || v.get("error").is_some() + || v.get("blocked") == Some(&json!(true)); + assert!(okish, "unexpected handoff result: {v}"); + // missing reason fails closed + let bad = execute_tool(&ctx, "browser_handoff", &json!({}).to_string()).await; + let bad: Value = serde_json::from_str(&bad).unwrap(); + assert!(bad.get("error").is_some(), "{bad}"); + match prev { + Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) }, + None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") }, + } } #[tokio::test] diff --git a/crates/grokboy/src/main.rs b/crates/grokboy/src/main.rs index aecca8f..b946004 100644 --- a/crates/grokboy/src/main.rs +++ b/crates/grokboy/src/main.rs @@ -50,7 +50,7 @@ async fn run() -> Result<()> { fn print_help() { println!( "\ -GrokBoy — minimal local CLI agent (P3: optional Playwright DOM browser) +GrokBoy — minimal local CLI agent (P4: human browser handoff) USAGE: grokboy chat Interactive streaming chat (no tools) @@ -65,11 +65,15 @@ ENV: GROKBOY_BASE_URL default https://api.x.ai/v1 GROKBOY_MODEL default grok-4.6 GROKBOY_CONTEXT_CHARS context budget (default 100000) + GROKBOY_BROWSER_HEADED 1 = always launch Chromium headed (visible) + GROKBOY_HANDOFF_AUTO 1 = auto-resume handoff (tests); abort = auto-abort Tools: shell, list_dir, read_file, write_file, report_done, report_blocked, - browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval + browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval, + browser_handoff Sessions: ~/.grokboy/sessions/.json Browser (optional): cd tools/playwright && npm i && npx playwright install chromium +Handoff: agent pauses on login/OTP/captcha → you fix in headed Chromium → Enter " ); } @@ -182,7 +186,7 @@ async fn cmd_run(args: &[String]) -> Result<()> { } async fn cmd_smoke() -> Result<()> { - println!("GrokBoy smoke (offline P3)…"); + println!("GrokBoy smoke (offline P4)…"); let stamp = uuid_like(); let dir = std::env::temp_dir().join(format!("grokboy-smoke-{stamp}")); std::fs::create_dir_all(&dir).context("temp dir")?; @@ -281,11 +285,20 @@ async fn cmd_smoke() -> Result<()> { } println!(" completion ok"); - // tool definitions present (6 core + 5 browser) + // tool definitions present (6 core + 6 browser incl. handoff) let defs = tool_definitions(); let n_tools = defs.as_array().map(|a| a.len()).unwrap_or(0); - if n_tools != 11 { - return Err(anyhow!("expected 11 tool defs, got {n_tools}")); + if n_tools != 12 { + return Err(anyhow!("expected 12 tool defs, got {n_tools}")); + } + let tool_names: Vec<&str> = defs + .as_array() + .unwrap() + .iter() + .filter_map(|t| t["function"]["name"].as_str()) + .collect(); + if !tool_names.contains(&"browser_handoff") { + return Err(anyhow!("browser_handoff missing from tool defs")); } println!(" tool defs ok"); @@ -314,6 +327,37 @@ async fn cmd_smoke() -> Result<()> { println!(" browser fail ok"); } + // handoff protocol (auto-resume; no interactive stdin) + // SAFETY: smoke is single-threaded for this env toggle. + let prev_auto = std::env::var("GROKBOY_HANDOFF_AUTO").ok(); + unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") }; + let handoff = execute_tool( + &ToolContext::new(repo.clone()), + "browser_handoff", + &json!({"reason": "smoke handoff check", "timeout_secs": 3}).to_string(), + ) + .await; + let handoff_v: serde_json::Value = serde_json::from_str(&handoff)?; + // Fail-closed without Chromium, or resumed with snapshot if browsers installed. + if handoff_v.get("error").is_none() + && handoff_v.get("blocked").is_none() + && handoff_v.get("handoff").and_then(|h| h.as_str()) != Some("resumed") + { + return Err(anyhow!("unexpected handoff smoke result: {handoff_v}")); + } + println!(" handoff auto ok"); + // abort path + unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "abort") }; + let abort_wait = grokboy_core::wait_for_handoff_resume(1); + if !matches!(abort_wait, grokboy_core::HandoffWait::Aborted(_)) { + return Err(anyhow!("expected HandoffWait::Aborted, got {abort_wait:?}")); + } + println!(" handoff abort ok"); + match prev_auto { + Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) }, + None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") }, + } + // context truncation let mut msgs = vec![ ChatMessage::system(AGENT_SYSTEM), diff --git a/docs/ACCEPTANCE.md b/docs/ACCEPTANCE.md index 7718529..80caf64 100644 --- a/docs/ACCEPTANCE.md +++ b/docs/ACCEPTANCE.md @@ -26,3 +26,14 @@ - [x] Thin Node helper under `tools/playwright/` (JSONL / one-shot JSON) - [x] Fail closed with install hint when Node/Playwright/Chromium missing - [x] `cargo test` / `grokboy smoke` pass without Playwright browsers installed + +## P4 — human browser handoff +- [x] Spec: `docs/PRODUCT.md` (north star, P0–P3, P4, non-goals) +- [x] Tool `browser_handoff` (`reason` required, optional `timeout_secs`) +- [x] Headed Chromium for handoff (`handoff_prepare`; relaunch if was headless) +- [x] Bilingual (繁中 + EN) terminal instructions; Enter continue / `abort` / timeout → fail-closed +- [x] After resume: DOM snapshot returned as tool result +- [x] Env docs: `GROKBOY_BROWSER_HEADED`, `GROKBOY_HANDOFF_AUTO` +- [x] Wired into tool defs + agent system prompt; fail-closed if no browser +- [x] Offline tests / smoke without API key or interactive stdin (`GROKBOY_HANDOFF_AUTO`) +- [x] README status table updated diff --git a/docs/PRODUCT.md b/docs/PRODUCT.md new file mode 100644 index 0000000..4048d3d --- /dev/null +++ b/docs/PRODUCT.md @@ -0,0 +1,48 @@ +# GrokBoy product notes + +**North star:** a local **Grok Bot–like** agent — thin CLI core, tool-using ReAct loop, optional Playwright **DOM** browser (not screenshot-first). GrokBoy is the sole main line; LazyBoy is reference only (no fork). + +## Done (P0–P3) + +| Phase | What | +|-------|------| +| **P0** | Streaming chat CLI (`grokboy chat`), xAI/OpenAI-compatible API | +| **P1** | Tools + ReAct (`shell`, files), sessions under `~/.grokboy/sessions/` | +| **P2** | Completion contract (`report_done` / `report_blocked`), loop guard, context truncation | +| **P3** | Optional Playwright DOM tools: navigate / snapshot / click / type / eval (fail-closed) | + +## P4 — Human handoff (this slice) + +Auth walls (login, OTP, captcha) often cannot be automated safely. P4 adds **`browser_handoff`**: + +1. Agent calls `browser_handoff` with a `reason` (optional `timeout_secs`). +2. Helper ensures Chromium is **headed** (visible); may relaunch from headless and restore URL. +3. Terminal prints bilingual (繁中 + English) instructions. +4. Loop **blocks** until you press **Enter** (continue) or type **`abort`**, or timeout → fail-closed blocked. +5. On resume, a **DOM snapshot** is returned so the model can continue. + +### Env + +- `GROKBOY_BROWSER_HEADED=1` — always launch Chromium headed (recommended when handoff is likely). +- `GROKBOY_HANDOFF_AUTO=1` — auto-resume (tests / CI); `abort` to auto-abort. + +### Non-goals (this slice) + +- Multi-agent orchestration +- Desktop accessibility / native UI automation +- External connectors / SaaS integrations +- Forking LazyBoy or Codex + +### Acceptance (summary) + +See `docs/ACCEPTANCE.md` section P4. Short list: + +- `browser_handoff` registered and wired into the agent loop +- Headed Chromium for handoff; JSONL daemon keeps state within one run when already headed +- Bilingual terminal prompt; Enter / abort / timeout fail-closed +- Post-resume DOM snapshot as tool result +- Offline `cargo test` / `grokboy smoke` without API key or interactive stdin + +## Roadmap hint (later) + +P5+ may deepen persistence, richer session UX, or more tools — still thin core, DOM-first browser. diff --git a/tools/playwright/README.md b/tools/playwright/README.md index 1214eb0..03e21e2 100644 --- a/tools/playwright/README.md +++ b/tools/playwright/README.md @@ -1,6 +1,6 @@ # GrokBoy Playwright helper (optional) -Thin Node script used by Rust `browser_*` tools. +Thin Node script used by Rust `browser_*` tools (P3) and human handoff (P4). ## Install @@ -16,6 +16,14 @@ npx playwright install chromium - **One-shot**: `node browser_helper.mjs --cmd '{"op":"ping"}'` - **Self-test** (no Chromium): `npm run self-test` -Ops: `ping`, `navigate`, `snapshot`/`dom`, `click`, `type`, `eval`, `close`, `status`. +Ops: `ping`, `navigate`, `snapshot`/`dom`, `click`, `type`, `eval`, `close`, `status`, **`handoff_prepare`**. + +### `handoff_prepare` + +Ensures a **headed** (visible) Chromium window and brings it to front. If the daemon was headless, closes and relaunches headed, then navigates back to the last URL when possible. + +Note: relaunch uses a new browser context (cookies from the prior headless session are not carried unless storage state is added later). Prefer `GROKBOY_BROWSER_HEADED=1` so the first launch is already headed and state survives across handoff within one run. + +Env: `GROKBOY_BROWSER_HEADED=1` launches headed for normal navigate as well. Prefer CSS selector or `role`+`name` over screenshots. diff --git a/tools/playwright/browser_helper.mjs b/tools/playwright/browser_helper.mjs index b7913a3..c466d68 100755 --- a/tools/playwright/browser_helper.mjs +++ b/tools/playwright/browser_helper.mjs @@ -7,7 +7,7 @@ * One-shot: node browser_helper.mjs --cmd '' * Self-test: node browser_helper.mjs --self-test (no Chromium required) * - * Ops: ping, navigate, snapshot, click, type, eval, close, status + * Ops: ping, navigate, snapshot, click, type, eval, close, status, handoff_prepare * Prefer CSS selector or role+name. Screenshots are NOT the primary control surface. */ @@ -25,6 +25,20 @@ let browser = null; let context = null; let page = null; let lastUrl = null; +/** Whether the current browser was launched headed (visible). */ +let browserHeaded = false; + +function envHeadedDefault() { + const v = (process.env.GROKBOY_BROWSER_HEADED || "").trim().toLowerCase(); + if (v === "1" || v === "true" || v === "yes") return true; + if (v === "0" || v === "false" || v === "no") return false; + return false; +} + +function wantHeaded(forceHeaded) { + if (forceHeaded === true) return true; + return envHeadedDefault(); +} function ok(id, extra = {}) { return { id: id ?? null, ok: true, ...extra }; @@ -50,11 +64,11 @@ async function loadPlaywright() { } } -async function ensurePage() { - if (page) return page; +async function launchBrowser(headed) { const { chromium } = await loadPlaywright(); try { - browser = await chromium.launch({ headless: true }); + browser = await chromium.launch({ headless: !headed }); + browserHeaded = headed; } catch (e) { const err = new Error( `${INSTALL_HINT} (chromium launch failed: ${e.message})` @@ -67,6 +81,78 @@ async function ensurePage() { return page; } +async function ensurePage(opts = {}) { + const headed = wantHeaded(opts.headed === true); + if (page) { + // Already open; if caller needs headed and we are headless, relaunch below via handoff_prepare. + return page; + } + return launchBrowser(headed); +} + +/** Close current browser and reopen headed, restoring lastUrl when possible. */ +async function ensureHeadedForHandoff() { + const restore = lastUrl || (page ? page.url() : null); + if (browser && browserHeaded && page) { + try { + await page.bringToFront(); + } catch { + /* ignore */ + } + return { + prepared: true, + headed: true, + relaunched: false, + url: page.url(), + restored: false, + }; + } + + if (browser) { + await browser.close().catch(() => {}); + browser = null; + context = null; + page = null; + browserHeaded = false; + } + + await launchBrowser(true); + let restored = false; + if (restore && restore !== "about:blank") { + try { + await page.goto(String(restore), { + waitUntil: "domcontentloaded", + timeout: 30000, + }); + lastUrl = page.url(); + restored = true; + } catch (e) { + // Keep headed page even if restore fails; caller still sees visible window. + lastUrl = page.url(); + return { + prepared: true, + headed: true, + relaunched: true, + url: lastUrl, + restored: false, + restore_error: String(e.message || e), + }; + } + } + try { + await page.bringToFront(); + } catch { + /* ignore */ + } + return { + prepared: true, + headed: true, + relaunched: true, + url: page ? page.url() : null, + restored, + }; +} + /** Accessibility-ish text snapshot: roles, names, and useful selectors — not pixels. */ async function buildSnapshot(p) { const nodes = await p.evaluate(() => { @@ -242,14 +328,27 @@ async function handle(req) { "eval", "close", "status", + "handoff_prepare", ], }); case "status": { return ok(id, { browser_open: !!browser, + headed: browserHeaded, last_url: lastUrl, page_url: page ? page.url() : null, + env_headed: envHeadedDefault(), + }); + } + + case "handoff_prepare": { + // Bring a visible Chromium window forward for human login/OTP/captcha. + const info = await ensureHeadedForHandoff(); + return ok(id, { + ...info, + message: + "Browser is headed (visible). Complete login/OTP/captcha in the window, then resume in the terminal.", }); } @@ -342,6 +441,7 @@ async function handle(req) { browser = null; context = null; page = null; + browserHeaded = false; return ok(id, { closed: true, last_url: lastUrl }); } @@ -375,6 +475,10 @@ async function runSelfTest() { // Protocol: ping without playwright let r = await handle({ id: "t1", op: "ping" }); cases.push(["ping", r.ok === true && r.pong === true]); + cases.push([ + "ping_lists_handoff_prepare", + Array.isArray(r.ops) && r.ops.includes("handoff_prepare"), + ]); // Missing fields fail closed r = await handle({ id: "t2", op: "navigate" });