P4: human browser handoff + product notes
This commit is contained in:
parent
829739e5b4
commit
5507e5b4b5
36
README.md
36
README.md
|
|
@ -1,6 +1,6 @@
|
||||||
# GrokBoy
|
# GrokBoy
|
||||||
|
|
||||||
Minimal local **GrokBot-like** CLI agent. Phase **P3**: optional Playwright DOM browser tools.
|
Minimal local **GrokBot-like** CLI agent. Phase **P4**: human browser handoff (login/OTP/captcha).
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
|
|
@ -10,8 +10,9 @@ Minimal local **GrokBot-like** CLI agent. Phase **P3**: optional Playwright DOM
|
||||||
| P1 shell / files + ReAct | done |
|
| P1 shell / files + ReAct | done |
|
||||||
| P2 completion / loop guard / truncation | done |
|
| P2 completion / loop guard / truncation | done |
|
||||||
| P3 browser (Playwright DOM) | done (optional) |
|
| P3 browser (Playwright DOM) | done (optional) |
|
||||||
|
| P4 human handoff | done |
|
||||||
|
|
||||||
No Docker desktop, no Codex/LazyBoy fork.
|
No Docker desktop, no Codex/LazyBoy fork. Product notes: [`docs/PRODUCT.md`](docs/PRODUCT.md).
|
||||||
|
|
||||||
## Setup (macOS)
|
## Setup (macOS)
|
||||||
|
|
||||||
|
|
@ -21,6 +22,7 @@ export GROKBOY_API_KEY=your_key # or XAI_API_KEY
|
||||||
# export GROKBOY_BASE_URL=https://api.x.ai/v1
|
# export GROKBOY_BASE_URL=https://api.x.ai/v1
|
||||||
# export GROKBOY_MODEL=grok-4.6
|
# export GROKBOY_MODEL=grok-4.6
|
||||||
# export GROKBOY_CONTEXT_CHARS=100000
|
# export GROKBOY_CONTEXT_CHARS=100000
|
||||||
|
# export GROKBOY_BROWSER_HEADED=1 # visible Chromium (recommended for handoff)
|
||||||
|
|
||||||
cd ~/GrokBoy
|
cd ~/GrokBoy
|
||||||
cargo run -p grokboy -- chat
|
cargo run -p grokboy -- chat
|
||||||
|
|
@ -36,29 +38,48 @@ npm install
|
||||||
npx playwright install chromium
|
npx playwright install chromium
|
||||||
```
|
```
|
||||||
|
|
||||||
Then the agent can use `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval` (DOM/selector/role — not screenshot-first).
|
Then the agent can use `browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`, and **`browser_handoff`** (DOM/selector/role — not screenshot-first).
|
||||||
|
|
||||||
|
### Human handoff (P4)
|
||||||
|
|
||||||
|
When the agent hits a login / OTP / captcha wall it calls `browser_handoff`:
|
||||||
|
|
||||||
|
1. Chromium opens **headed** (visible) — or relaunches headed if it was headless.
|
||||||
|
2. Terminal prints why it paused (ZH-TW + English) and what to do.
|
||||||
|
3. You complete the wall in the browser, then press **Enter** in that terminal (or type `abort`).
|
||||||
|
4. Agent resumes with a fresh **DOM snapshot**.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Prefer headed for runs that may need handoff:
|
||||||
|
export GROKBOY_BROWSER_HEADED=1
|
||||||
|
cargo run -p grokboy -- run "打開需要登入的頁面並完成任務"
|
||||||
|
```
|
||||||
|
|
||||||
|
Tests / CI: `GROKBOY_HANDOFF_AUTO=1` auto-resumes (no interactive Enter).
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
- `grokboy chat` — interactive streaming chat (no tools)
|
- `grokboy chat` — interactive streaming chat (no tools)
|
||||||
- `grokboy run "<prompt>"` — one-shot agent with tools
|
- `grokboy run "<prompt>"` — one-shot agent with tools
|
||||||
- `grokboy run --session <id> "<prompt>"` — continue a saved session
|
- `grokboy run --session <id> "<prompt>"` — continue a saved session
|
||||||
- `grokboy smoke` — offline checks (no API key / no Chromium required)
|
- `grokboy smoke` — offline checks (no API key / no interactive handoff)
|
||||||
- `grokboy help`
|
- `grokboy help`
|
||||||
|
|
||||||
Tools: `shell`, `list_dir`, `read_file`, `write_file`, `report_done`, `report_blocked`,
|
Tools: `shell`, `list_dir`, `read_file`, `write_file`, `report_done`, `report_blocked`,
|
||||||
`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`.
|
`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_eval`,
|
||||||
|
`browser_handoff`.
|
||||||
|
|
||||||
Sessions are stored under `~/.grokboy/sessions/<id>.json` (may include `last_browser_url`).
|
Sessions are stored under `~/.grokboy/sessions/<id>.json` (may include `last_browser_url`).
|
||||||
|
|
||||||
The agent stops on `report_done` / `report_blocked`, blocks identical tool rounds (×3), and truncates old context when over budget.
|
The agent stops on `report_done` / `report_blocked`, blocks identical tool rounds (×3), and truncates old context when over budget.
|
||||||
|
|
||||||
## 繁體中文
|
## Traditional Chinese
|
||||||
|
|
||||||
本機終端機 coding assistant。P3 可選 Playwright DOM 瀏覽器工具(非截圖優先)。
|
本機終端機 coding assistant。P4 支援瀏覽器人工接手(登入/OTP/驗證碼):代理暫停 → 你在可見 Chromium 完成 → 終端機按 Enter 繼續。
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
export GROKBOY_API_KEY=你的金鑰
|
export GROKBOY_API_KEY=你的金鑰
|
||||||
|
export GROKBOY_BROWSER_HEADED=1
|
||||||
cd ~/GrokBoy
|
cd ~/GrokBoy
|
||||||
cargo run -p grokboy -- smoke
|
cargo run -p grokboy -- smoke
|
||||||
# 可選瀏覽器:
|
# 可選瀏覽器:
|
||||||
|
|
@ -73,5 +94,6 @@ cargo run -p grokboy -- chat
|
||||||
crates/grokboy-core/ # config, model, tools, browser, agent, session
|
crates/grokboy-core/ # config, model, tools, browser, agent, session
|
||||||
crates/grokboy/ # CLI binary
|
crates/grokboy/ # CLI binary
|
||||||
tools/playwright/ # optional Node Playwright helper (JSONL)
|
tools/playwright/ # optional Node Playwright helper (JSONL)
|
||||||
|
docs/PRODUCT.md
|
||||||
docs/ACCEPTANCE.md
|
docs/ACCEPTANCE.md
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -15,8 +15,9 @@ pub const AGENT_SYSTEM: &str = "\
|
||||||
You are GrokBoy, a concise local coding assistant with tools.
|
You are GrokBoy, a concise local coding assistant with tools.
|
||||||
Use tools when they help solve the task; otherwise answer directly.
|
Use tools when they help solve the task; otherwise answer directly.
|
||||||
Prefer short, clear answers. Traditional Chinese is welcome when the user writes in Chinese.
|
Prefer short, clear answers. Traditional Chinese is welcome when the user writes in Chinese.
|
||||||
Available tools: shell, list_dir, read_file, write_file, report_done, report_blocked, and optional browser_* (Playwright DOM: browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval).
|
Available tools: shell, list_dir, read_file, write_file, report_done, report_blocked, and optional browser_* (Playwright DOM: browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval, browser_handoff).
|
||||||
For web pages prefer DOM snapshot + selector/role click/type — not screenshots or pixel XY clicks.
|
For web pages prefer DOM snapshot + selector/role click/type — not screenshots or pixel XY clicks.
|
||||||
|
If you hit a login / OTP / captcha wall you cannot pass alone, call browser_handoff with a clear reason so the human can help in the visible browser, then continue from the returned snapshot.
|
||||||
When the task is finished, call report_done with a short summary.
|
When the task is finished, call report_done with a short summary.
|
||||||
If you are stuck or cannot proceed, call report_blocked with the reason — do not invent results or loop.
|
If you are stuck or cannot proceed, call report_blocked with the reason — do not invent results or loop.
|
||||||
Do not invent tool results — call the tools.";
|
Do not invent tool results — call the tools.";
|
||||||
|
|
@ -314,6 +315,7 @@ mod tests {
|
||||||
assert!(AGENT_SYSTEM.contains("report_done"));
|
assert!(AGENT_SYSTEM.contains("report_done"));
|
||||||
assert!(AGENT_SYSTEM.contains("report_blocked"));
|
assert!(AGENT_SYSTEM.contains("report_blocked"));
|
||||||
assert!(AGENT_SYSTEM.contains("browser_navigate") || AGENT_SYSTEM.contains("browser_"));
|
assert!(AGENT_SYSTEM.contains("browser_navigate") || AGENT_SYSTEM.contains("browser_"));
|
||||||
|
assert!(AGENT_SYSTEM.contains("browser_handoff"));
|
||||||
assert!(AGENT_SYSTEM.contains("Traditional Chinese") || AGENT_SYSTEM.contains("Chinese"));
|
assert!(AGENT_SYSTEM.contains("Traditional Chinese") || AGENT_SYSTEM.contains("Chinese"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,8 +1,12 @@
|
||||||
//! Optional Playwright DOM browser tools (P3).
|
//! Optional Playwright DOM browser tools (P3) + human handoff (P4).
|
||||||
//!
|
//!
|
||||||
//! Spawns `tools/playwright/browser_helper.mjs` (JSONL over stdin/stdout).
|
//! Spawns `tools/playwright/browser_helper.mjs` (JSONL over stdin/stdout).
|
||||||
//! Fail closed with an install hint when Node / Playwright / Chromium is missing.
|
//! Fail closed with an install hint when Node / Playwright / Chromium is missing.
|
||||||
//! Prefer selectors / roles — not screenshot-first control.
|
//! Prefer selectors / roles — not screenshot-first control.
|
||||||
|
//!
|
||||||
|
//! Env:
|
||||||
|
//! - `GROKBOY_BROWSER_HEADED=1` — launch Chromium headed (visible) by default
|
||||||
|
//! - `GROKBOY_HANDOFF_AUTO=1` — auto-resume handoff (tests); `abort` to auto-abort
|
||||||
|
|
||||||
use anyhow::{Context, Result, anyhow};
|
use anyhow::{Context, Result, anyhow};
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
|
|
@ -10,6 +14,7 @@ use std::io::{BufRead, BufReader, Write};
|
||||||
use std::path::{Path, PathBuf};
|
use std::path::{Path, PathBuf};
|
||||||
use std::process::{Child, ChildStdin, ChildStdout, Command, Stdio};
|
use std::process::{Child, ChildStdin, ChildStdout, Command, Stdio};
|
||||||
use std::sync::{Arc, Mutex, OnceLock};
|
use std::sync::{Arc, Mutex, OnceLock};
|
||||||
|
use std::sync::mpsc;
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
|
|
||||||
pub const INSTALL_HINT: &str = "Playwright browser tools unavailable. Install with: \
|
pub const INSTALL_HINT: &str = "Playwright browser tools unavailable. Install with: \
|
||||||
|
|
@ -399,6 +404,27 @@ pub fn browser_tool_definitions() -> Vec<Value> {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}),
|
}),
|
||||||
|
json!({
|
||||||
|
"type": "function",
|
||||||
|
"function": {
|
||||||
|
"name": "browser_handoff",
|
||||||
|
"description": "Pause for human help on auth walls (login, OTP, captcha). Shows a headed Chromium window, prints terminal instructions (ZH+EN), waits for Enter (or abort), then returns a DOM snapshot so the agent can continue. Prefer when automation cannot pass the wall.",
|
||||||
|
"parameters": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"reason": {
|
||||||
|
"type": "string",
|
||||||
|
"description": "Why human help is needed (e.g. login page, OTP, captcha)"
|
||||||
|
},
|
||||||
|
"timeout_secs": {
|
||||||
|
"type": "integer",
|
||||||
|
"description": "Seconds to wait for user (default 300)"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": ["reason"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}),
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -411,6 +437,7 @@ pub fn is_browser_tool(name: &str) -> bool {
|
||||||
| "browser_click"
|
| "browser_click"
|
||||||
| "browser_type"
|
| "browser_type"
|
||||||
| "browser_eval"
|
| "browser_eval"
|
||||||
|
| "browser_handoff"
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -420,6 +447,10 @@ pub async fn execute_browser_tool(
|
||||||
name: &str,
|
name: &str,
|
||||||
args: &Value,
|
args: &Value,
|
||||||
) -> Result<Value> {
|
) -> Result<Value> {
|
||||||
|
if name == "browser_handoff" {
|
||||||
|
return execute_browser_handoff(cwd, last_url, args).await;
|
||||||
|
}
|
||||||
|
|
||||||
// Run blocking helper I/O off the async runtime.
|
// Run blocking helper I/O off the async runtime.
|
||||||
let cwd = cwd.to_path_buf();
|
let cwd = cwd.to_path_buf();
|
||||||
let name = name.to_string();
|
let name = name.to_string();
|
||||||
|
|
@ -460,6 +491,185 @@ pub async fn execute_browser_tool(
|
||||||
.map_err(|e| anyhow!("browser task join: {e}"))?
|
.map_err(|e| anyhow!("browser task join: {e}"))?
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Outcome of waiting for the human during handoff.
|
||||||
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||||
|
pub enum HandoffWait {
|
||||||
|
Resumed,
|
||||||
|
Aborted(String),
|
||||||
|
TimedOut,
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEFAULT_HANDOFF_TIMEOUT_SECS: u64 = 300;
|
||||||
|
|
||||||
|
/// Wait for stdin line (Enter to continue, `abort` to cancel) or timeout.
|
||||||
|
///
|
||||||
|
/// `GROKBOY_HANDOFF_AUTO=1|resume` skips the wait (tests).
|
||||||
|
/// `GROKBOY_HANDOFF_AUTO=abort` auto-aborts.
|
||||||
|
pub fn wait_for_handoff_resume(timeout_secs: u64) -> HandoffWait {
|
||||||
|
match std::env::var("GROKBOY_HANDOFF_AUTO") {
|
||||||
|
Ok(v) => {
|
||||||
|
let v = v.trim().to_ascii_lowercase();
|
||||||
|
if v == "1" || v == "true" || v == "yes" || v == "resume" || v == "continue"
|
||||||
|
{
|
||||||
|
return HandoffWait::Resumed;
|
||||||
|
}
|
||||||
|
if v == "abort" || v == "0" || v == "false" || v == "no" {
|
||||||
|
return HandoffWait::Aborted(format!("GROKBOY_HANDOFF_AUTO={v}"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Err(_) => {}
|
||||||
|
}
|
||||||
|
|
||||||
|
let (tx, rx) = mpsc::channel::<Result<String, String>>();
|
||||||
|
std::thread::spawn(move || {
|
||||||
|
let stdin = std::io::stdin();
|
||||||
|
let mut line = String::new();
|
||||||
|
match stdin.lock().read_line(&mut line) {
|
||||||
|
Ok(0) => {
|
||||||
|
let _ = tx.send(Err("stdin closed (EOF)".into()));
|
||||||
|
}
|
||||||
|
Ok(_) => {
|
||||||
|
let _ = tx.send(Ok(line));
|
||||||
|
}
|
||||||
|
Err(e) => {
|
||||||
|
let _ = tx.send(Err(format!("stdin read error: {e}")));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
match rx.recv_timeout(Duration::from_secs(timeout_secs.max(1))) {
|
||||||
|
Ok(Ok(line)) => {
|
||||||
|
let t = line.trim().to_ascii_lowercase();
|
||||||
|
if t == "abort" || t == "q" || t == "quit" || t == "cancel" {
|
||||||
|
HandoffWait::Aborted(format!("user typed {t}"))
|
||||||
|
} else {
|
||||||
|
// Empty line (Enter) or any other input → resume.
|
||||||
|
HandoffWait::Resumed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(Err(msg)) => HandoffWait::Aborted(msg),
|
||||||
|
Err(mpsc::RecvTimeoutError::Timeout) => HandoffWait::TimedOut,
|
||||||
|
Err(mpsc::RecvTimeoutError::Disconnected) => {
|
||||||
|
HandoffWait::Aborted("handoff wait thread disconnected".into())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn print_handoff_instructions(reason: &str, timeout_secs: u64) {
|
||||||
|
let banner = format!(
|
||||||
|
"
|
||||||
|
╔══════════════════════════════════════════════════════════════╗
|
||||||
|
║ GrokBoy P4 — Human handoff / 人工接手 ║
|
||||||
|
╚══════════════════════════════════════════════════════════════╝
|
||||||
|
|
||||||
|
【為什麼暫停 / Why paused】
|
||||||
|
{reason}
|
||||||
|
|
||||||
|
【請你做什麼 / What to do】
|
||||||
|
1. 看著已開啟的 Chromium 視窗(headed / visible)。
|
||||||
|
Look at the visible Chromium window.
|
||||||
|
2. 完成登入、OTP、驗證碼或其他真人操作。
|
||||||
|
Complete login / OTP / captcha (or whatever is blocking).
|
||||||
|
3. 完成後回到這個終端機,按 Enter 繼續。
|
||||||
|
When done, return here and press Enter to continue.
|
||||||
|
4. 若要放棄,輸入 abort 再按 Enter。
|
||||||
|
To give up, type abort then Enter.
|
||||||
|
|
||||||
|
Timeout / 逾時: {timeout_secs}s
|
||||||
|
(tests: GROKBOY_HANDOFF_AUTO=1 to auto-resume)
|
||||||
|
|
||||||
|
─── waiting for Enter / 等待 Enter ───
|
||||||
|
"
|
||||||
|
);
|
||||||
|
eprintln!("{banner}");
|
||||||
|
let _ = std::io::Write::flush(&mut std::io::stderr());
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `browser_handoff`: prepare headed browser, wait for human, return snapshot.
|
||||||
|
pub async fn execute_browser_handoff(
|
||||||
|
cwd: &Path,
|
||||||
|
last_url: &LastUrlSlot,
|
||||||
|
args: &Value,
|
||||||
|
) -> Result<Value> {
|
||||||
|
let reason = args
|
||||||
|
.get("reason")
|
||||||
|
.and_then(|v| v.as_str())
|
||||||
|
.map(str::trim)
|
||||||
|
.filter(|s| !s.is_empty())
|
||||||
|
.ok_or_else(|| anyhow!("browser_handoff: missing 'reason'"))?
|
||||||
|
.to_string();
|
||||||
|
|
||||||
|
let timeout_secs = args
|
||||||
|
.get("timeout_secs")
|
||||||
|
.and_then(|v| v.as_u64())
|
||||||
|
.filter(|&n| n > 0)
|
||||||
|
.unwrap_or(DEFAULT_HANDOFF_TIMEOUT_SECS);
|
||||||
|
|
||||||
|
let cwd = cwd.to_path_buf();
|
||||||
|
let last_url = last_url.clone();
|
||||||
|
|
||||||
|
tokio::task::spawn_blocking(move || {
|
||||||
|
// 1) Ensure headed Chromium (JSONL daemon keeps page state when already headed).
|
||||||
|
let prep = browser_request(
|
||||||
|
&cwd,
|
||||||
|
json!({
|
||||||
|
"op": "handoff_prepare",
|
||||||
|
"reason": reason,
|
||||||
|
}),
|
||||||
|
)?;
|
||||||
|
|
||||||
|
if prep.get("ok") != Some(&json!(true)) {
|
||||||
|
return Ok(response_to_tool_json(prep));
|
||||||
|
}
|
||||||
|
update_last_url(&last_url, &prep);
|
||||||
|
|
||||||
|
// 2) Instruct the human (bilingual).
|
||||||
|
print_handoff_instructions(&reason, timeout_secs);
|
||||||
|
|
||||||
|
// 3) Block until Enter / abort / timeout.
|
||||||
|
let wait = wait_for_handoff_resume(timeout_secs);
|
||||||
|
match wait {
|
||||||
|
HandoffWait::Aborted(msg) => {
|
||||||
|
return Ok(json!({
|
||||||
|
"status": "blocked",
|
||||||
|
"blocked": true,
|
||||||
|
"handoff": "aborted",
|
||||||
|
"reason": format!("human handoff aborted: {msg}"),
|
||||||
|
"original_reason": reason,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
HandoffWait::TimedOut => {
|
||||||
|
return Ok(json!({
|
||||||
|
"status": "blocked",
|
||||||
|
"blocked": true,
|
||||||
|
"handoff": "timeout",
|
||||||
|
"reason": format!(
|
||||||
|
"human handoff timed out after {timeout_secs}s (fail-closed)"
|
||||||
|
),
|
||||||
|
"original_reason": reason,
|
||||||
|
}));
|
||||||
|
}
|
||||||
|
HandoffWait::Resumed => {}
|
||||||
|
}
|
||||||
|
|
||||||
|
// 4) Snapshot so the model can continue from post-login DOM.
|
||||||
|
let snap = browser_request(&cwd, json!({ "op": "snapshot" }))?;
|
||||||
|
update_last_url(&last_url, &snap);
|
||||||
|
let mut out = response_to_tool_json(snap);
|
||||||
|
if let Some(obj) = out.as_object_mut() {
|
||||||
|
obj.insert("handoff".into(), json!("resumed"));
|
||||||
|
obj.insert("handoff_reason".into(), json!(reason));
|
||||||
|
obj.insert(
|
||||||
|
"message".into(),
|
||||||
|
json!("Human handoff resumed; DOM snapshot attached."),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
Ok(out)
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.map_err(|e| anyhow!("browser handoff join: {e}"))?
|
||||||
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests {
|
mod tests {
|
||||||
use super::*;
|
use super::*;
|
||||||
|
|
@ -524,12 +734,44 @@ mod tests {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn browser_defs_count() {
|
fn browser_defs_count() {
|
||||||
assert_eq!(browser_tool_definitions().len(), 5);
|
assert_eq!(browser_tool_definitions().len(), 6);
|
||||||
assert!(is_browser_tool("browser_navigate"));
|
assert!(is_browser_tool("browser_navigate"));
|
||||||
assert!(is_browser_tool("browser_snapshot"));
|
assert!(is_browser_tool("browser_snapshot"));
|
||||||
|
assert!(is_browser_tool("browser_handoff"));
|
||||||
assert!(!is_browser_tool("shell"));
|
assert!(!is_browser_tool("shell"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn handoff_auto_resume_and_abort() {
|
||||||
|
// SAFETY: tests run serially for this env in practice; restore after.
|
||||||
|
let prev = std::env::var("GROKBOY_HANDOFF_AUTO").ok();
|
||||||
|
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") };
|
||||||
|
assert_eq!(wait_for_handoff_resume(1), HandoffWait::Resumed);
|
||||||
|
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "abort") };
|
||||||
|
match wait_for_handoff_resume(1) {
|
||||||
|
HandoffWait::Aborted(msg) => assert!(msg.contains("abort"), "{msg}"),
|
||||||
|
other => panic!("expected aborted, got {other:?}"),
|
||||||
|
}
|
||||||
|
match prev {
|
||||||
|
Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) },
|
||||||
|
None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") },
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn oneshot_ping_lists_handoff_prepare() {
|
||||||
|
let root = repo_cwd();
|
||||||
|
if which_node().is_none() {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
let resp = browser_oneshot(&root, &json!({"op": "ping", "id": "h1"})).unwrap();
|
||||||
|
let ops = resp["ops"].as_array().expect("ops");
|
||||||
|
assert!(
|
||||||
|
ops.iter().any(|o| o.as_str() == Some("handoff_prepare")),
|
||||||
|
"{resp}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn install_hint_constant() {
|
fn install_hint_constant() {
|
||||||
assert!(INSTALL_HINT.contains("npx playwright install chromium"));
|
assert!(INSTALL_HINT.contains("npx playwright install chromium"));
|
||||||
|
|
|
||||||
|
|
@ -12,7 +12,7 @@ pub use agent::{
|
||||||
context_char_budget, message_char_len, messages_char_len, round_signature, run_agent,
|
context_char_budget, message_char_len, messages_char_len, round_signature, run_agent,
|
||||||
run_agent_with, tool_call_signature, truncate_messages,
|
run_agent_with, tool_call_signature, truncate_messages,
|
||||||
};
|
};
|
||||||
pub use browser::{INSTALL_HINT as BROWSER_INSTALL_HINT, browser_oneshot, browser_self_test, find_helper_script};
|
pub use browser::{INSTALL_HINT as BROWSER_INSTALL_HINT, HandoffWait, browser_oneshot, browser_self_test, find_helper_script, wait_for_handoff_resume};
|
||||||
pub use config::Config;
|
pub use config::Config;
|
||||||
pub use model::{ChatMessage, FunctionCall, Role, ToolCall, chat_completion, stream_chat};
|
pub use model::{ChatMessage, FunctionCall, Role, ToolCall, chat_completion, stream_chat};
|
||||||
pub use session::{Session, load_or_create, load_session, save_session, sessions_dir};
|
pub use session::{Session, load_or_create, load_session, save_session, sessions_dir};
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
//! Built-in tools: shell, files, completion, and optional Playwright browser (P3).
|
//! Built-in tools: shell, files, completion, Playwright browser (P3), human handoff (P4).
|
||||||
|
|
||||||
use anyhow::{Context, Result, anyhow};
|
use anyhow::{Context, Result, anyhow};
|
||||||
use serde_json::{Value, json};
|
use serde_json::{Value, json};
|
||||||
|
|
@ -496,7 +496,7 @@ mod tests {
|
||||||
fn tool_defs_include_core_and_browser() {
|
fn tool_defs_include_core_and_browser() {
|
||||||
let defs = tool_definitions();
|
let defs = tool_definitions();
|
||||||
let arr = defs.as_array().unwrap();
|
let arr = defs.as_array().unwrap();
|
||||||
assert_eq!(arr.len(), 11); // 6 core + 5 browser
|
assert_eq!(arr.len(), 12); // 6 core + 6 browser (incl. handoff)
|
||||||
let names: Vec<&str> = arr
|
let names: Vec<&str> = arr
|
||||||
.iter()
|
.iter()
|
||||||
.map(|t| t["function"]["name"].as_str().unwrap())
|
.map(|t| t["function"]["name"].as_str().unwrap())
|
||||||
|
|
@ -512,6 +512,38 @@ mod tests {
|
||||||
assert!(names.contains(&"browser_click"));
|
assert!(names.contains(&"browser_click"));
|
||||||
assert!(names.contains(&"browser_type"));
|
assert!(names.contains(&"browser_type"));
|
||||||
assert!(names.contains(&"browser_eval"));
|
assert!(names.contains(&"browser_eval"));
|
||||||
|
assert!(names.contains(&"browser_handoff"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn browser_handoff_auto_resume_protocol() {
|
||||||
|
// Offline: with GROKBOY_HANDOFF_AUTO=1, missing Chromium still fail-closes
|
||||||
|
// OR (if Chromium present) resumes and returns snapshot/error JSON — never hangs.
|
||||||
|
let prev = std::env::var("GROKBOY_HANDOFF_AUTO").ok();
|
||||||
|
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") };
|
||||||
|
let root = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../..");
|
||||||
|
let root = root.canonicalize().unwrap_or(root);
|
||||||
|
let ctx = ToolContext::new(root);
|
||||||
|
let out = execute_tool(
|
||||||
|
&ctx,
|
||||||
|
"browser_handoff",
|
||||||
|
&json!({"reason": "unit test OTP wall", "timeout_secs": 5}).to_string(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let v: Value = serde_json::from_str(&out).unwrap();
|
||||||
|
// Either resumed snapshot, or fail-closed blocked/error (no Chromium) — must not panic.
|
||||||
|
let okish = v.get("handoff").and_then(|h| h.as_str()) == Some("resumed")
|
||||||
|
|| v.get("error").is_some()
|
||||||
|
|| v.get("blocked") == Some(&json!(true));
|
||||||
|
assert!(okish, "unexpected handoff result: {v}");
|
||||||
|
// missing reason fails closed
|
||||||
|
let bad = execute_tool(&ctx, "browser_handoff", &json!({}).to_string()).await;
|
||||||
|
let bad: Value = serde_json::from_str(&bad).unwrap();
|
||||||
|
assert!(bad.get("error").is_some(), "{bad}");
|
||||||
|
match prev {
|
||||||
|
Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) },
|
||||||
|
None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") },
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::test]
|
#[tokio::test]
|
||||||
|
|
|
||||||
|
|
@ -50,7 +50,7 @@ async fn run() -> Result<()> {
|
||||||
fn print_help() {
|
fn print_help() {
|
||||||
println!(
|
println!(
|
||||||
"\
|
"\
|
||||||
GrokBoy — minimal local CLI agent (P3: optional Playwright DOM browser)
|
GrokBoy — minimal local CLI agent (P4: human browser handoff)
|
||||||
|
|
||||||
USAGE:
|
USAGE:
|
||||||
grokboy chat Interactive streaming chat (no tools)
|
grokboy chat Interactive streaming chat (no tools)
|
||||||
|
|
@ -65,11 +65,15 @@ ENV:
|
||||||
GROKBOY_BASE_URL default https://api.x.ai/v1
|
GROKBOY_BASE_URL default https://api.x.ai/v1
|
||||||
GROKBOY_MODEL default grok-4.6
|
GROKBOY_MODEL default grok-4.6
|
||||||
GROKBOY_CONTEXT_CHARS context budget (default 100000)
|
GROKBOY_CONTEXT_CHARS context budget (default 100000)
|
||||||
|
GROKBOY_BROWSER_HEADED 1 = always launch Chromium headed (visible)
|
||||||
|
GROKBOY_HANDOFF_AUTO 1 = auto-resume handoff (tests); abort = auto-abort
|
||||||
|
|
||||||
Tools: shell, list_dir, read_file, write_file, report_done, report_blocked,
|
Tools: shell, list_dir, read_file, write_file, report_done, report_blocked,
|
||||||
browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval
|
browser_navigate, browser_snapshot, browser_click, browser_type, browser_eval,
|
||||||
|
browser_handoff
|
||||||
Sessions: ~/.grokboy/sessions/<id>.json
|
Sessions: ~/.grokboy/sessions/<id>.json
|
||||||
Browser (optional): cd tools/playwright && npm i && npx playwright install chromium
|
Browser (optional): cd tools/playwright && npm i && npx playwright install chromium
|
||||||
|
Handoff: agent pauses on login/OTP/captcha → you fix in headed Chromium → Enter
|
||||||
"
|
"
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
@ -182,7 +186,7 @@ async fn cmd_run(args: &[String]) -> Result<()> {
|
||||||
}
|
}
|
||||||
|
|
||||||
async fn cmd_smoke() -> Result<()> {
|
async fn cmd_smoke() -> Result<()> {
|
||||||
println!("GrokBoy smoke (offline P3)…");
|
println!("GrokBoy smoke (offline P4)…");
|
||||||
let stamp = uuid_like();
|
let stamp = uuid_like();
|
||||||
let dir = std::env::temp_dir().join(format!("grokboy-smoke-{stamp}"));
|
let dir = std::env::temp_dir().join(format!("grokboy-smoke-{stamp}"));
|
||||||
std::fs::create_dir_all(&dir).context("temp dir")?;
|
std::fs::create_dir_all(&dir).context("temp dir")?;
|
||||||
|
|
@ -281,11 +285,20 @@ async fn cmd_smoke() -> Result<()> {
|
||||||
}
|
}
|
||||||
println!(" completion ok");
|
println!(" completion ok");
|
||||||
|
|
||||||
// tool definitions present (6 core + 5 browser)
|
// tool definitions present (6 core + 6 browser incl. handoff)
|
||||||
let defs = tool_definitions();
|
let defs = tool_definitions();
|
||||||
let n_tools = defs.as_array().map(|a| a.len()).unwrap_or(0);
|
let n_tools = defs.as_array().map(|a| a.len()).unwrap_or(0);
|
||||||
if n_tools != 11 {
|
if n_tools != 12 {
|
||||||
return Err(anyhow!("expected 11 tool defs, got {n_tools}"));
|
return Err(anyhow!("expected 12 tool defs, got {n_tools}"));
|
||||||
|
}
|
||||||
|
let tool_names: Vec<&str> = defs
|
||||||
|
.as_array()
|
||||||
|
.unwrap()
|
||||||
|
.iter()
|
||||||
|
.filter_map(|t| t["function"]["name"].as_str())
|
||||||
|
.collect();
|
||||||
|
if !tool_names.contains(&"browser_handoff") {
|
||||||
|
return Err(anyhow!("browser_handoff missing from tool defs"));
|
||||||
}
|
}
|
||||||
println!(" tool defs ok");
|
println!(" tool defs ok");
|
||||||
|
|
||||||
|
|
@ -314,6 +327,37 @@ async fn cmd_smoke() -> Result<()> {
|
||||||
println!(" browser fail ok");
|
println!(" browser fail ok");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// handoff protocol (auto-resume; no interactive stdin)
|
||||||
|
// SAFETY: smoke is single-threaded for this env toggle.
|
||||||
|
let prev_auto = std::env::var("GROKBOY_HANDOFF_AUTO").ok();
|
||||||
|
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "1") };
|
||||||
|
let handoff = execute_tool(
|
||||||
|
&ToolContext::new(repo.clone()),
|
||||||
|
"browser_handoff",
|
||||||
|
&json!({"reason": "smoke handoff check", "timeout_secs": 3}).to_string(),
|
||||||
|
)
|
||||||
|
.await;
|
||||||
|
let handoff_v: serde_json::Value = serde_json::from_str(&handoff)?;
|
||||||
|
// Fail-closed without Chromium, or resumed with snapshot if browsers installed.
|
||||||
|
if handoff_v.get("error").is_none()
|
||||||
|
&& handoff_v.get("blocked").is_none()
|
||||||
|
&& handoff_v.get("handoff").and_then(|h| h.as_str()) != Some("resumed")
|
||||||
|
{
|
||||||
|
return Err(anyhow!("unexpected handoff smoke result: {handoff_v}"));
|
||||||
|
}
|
||||||
|
println!(" handoff auto ok");
|
||||||
|
// abort path
|
||||||
|
unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", "abort") };
|
||||||
|
let abort_wait = grokboy_core::wait_for_handoff_resume(1);
|
||||||
|
if !matches!(abort_wait, grokboy_core::HandoffWait::Aborted(_)) {
|
||||||
|
return Err(anyhow!("expected HandoffWait::Aborted, got {abort_wait:?}"));
|
||||||
|
}
|
||||||
|
println!(" handoff abort ok");
|
||||||
|
match prev_auto {
|
||||||
|
Some(v) => unsafe { std::env::set_var("GROKBOY_HANDOFF_AUTO", v) },
|
||||||
|
None => unsafe { std::env::remove_var("GROKBOY_HANDOFF_AUTO") },
|
||||||
|
}
|
||||||
|
|
||||||
// context truncation
|
// context truncation
|
||||||
let mut msgs = vec![
|
let mut msgs = vec![
|
||||||
ChatMessage::system(AGENT_SYSTEM),
|
ChatMessage::system(AGENT_SYSTEM),
|
||||||
|
|
|
||||||
|
|
@ -26,3 +26,14 @@
|
||||||
- [x] Thin Node helper under `tools/playwright/` (JSONL / one-shot JSON)
|
- [x] Thin Node helper under `tools/playwright/` (JSONL / one-shot JSON)
|
||||||
- [x] Fail closed with install hint when Node/Playwright/Chromium missing
|
- [x] Fail closed with install hint when Node/Playwright/Chromium missing
|
||||||
- [x] `cargo test` / `grokboy smoke` pass without Playwright browsers installed
|
- [x] `cargo test` / `grokboy smoke` pass without Playwright browsers installed
|
||||||
|
|
||||||
|
## P4 — human browser handoff
|
||||||
|
- [x] Spec: `docs/PRODUCT.md` (north star, P0–P3, P4, non-goals)
|
||||||
|
- [x] Tool `browser_handoff` (`reason` required, optional `timeout_secs`)
|
||||||
|
- [x] Headed Chromium for handoff (`handoff_prepare`; relaunch if was headless)
|
||||||
|
- [x] Bilingual (繁中 + EN) terminal instructions; Enter continue / `abort` / timeout → fail-closed
|
||||||
|
- [x] After resume: DOM snapshot returned as tool result
|
||||||
|
- [x] Env docs: `GROKBOY_BROWSER_HEADED`, `GROKBOY_HANDOFF_AUTO`
|
||||||
|
- [x] Wired into tool defs + agent system prompt; fail-closed if no browser
|
||||||
|
- [x] Offline tests / smoke without API key or interactive stdin (`GROKBOY_HANDOFF_AUTO`)
|
||||||
|
- [x] README status table updated
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,48 @@
|
||||||
|
# GrokBoy product notes
|
||||||
|
|
||||||
|
**North star:** a local **Grok Bot–like** agent — thin CLI core, tool-using ReAct loop, optional Playwright **DOM** browser (not screenshot-first). GrokBoy is the sole main line; LazyBoy is reference only (no fork).
|
||||||
|
|
||||||
|
## Done (P0–P3)
|
||||||
|
|
||||||
|
| Phase | What |
|
||||||
|
|-------|------|
|
||||||
|
| **P0** | Streaming chat CLI (`grokboy chat`), xAI/OpenAI-compatible API |
|
||||||
|
| **P1** | Tools + ReAct (`shell`, files), sessions under `~/.grokboy/sessions/` |
|
||||||
|
| **P2** | Completion contract (`report_done` / `report_blocked`), loop guard, context truncation |
|
||||||
|
| **P3** | Optional Playwright DOM tools: navigate / snapshot / click / type / eval (fail-closed) |
|
||||||
|
|
||||||
|
## P4 — Human handoff (this slice)
|
||||||
|
|
||||||
|
Auth walls (login, OTP, captcha) often cannot be automated safely. P4 adds **`browser_handoff`**:
|
||||||
|
|
||||||
|
1. Agent calls `browser_handoff` with a `reason` (optional `timeout_secs`).
|
||||||
|
2. Helper ensures Chromium is **headed** (visible); may relaunch from headless and restore URL.
|
||||||
|
3. Terminal prints bilingual (繁中 + English) instructions.
|
||||||
|
4. Loop **blocks** until you press **Enter** (continue) or type **`abort`**, or timeout → fail-closed blocked.
|
||||||
|
5. On resume, a **DOM snapshot** is returned so the model can continue.
|
||||||
|
|
||||||
|
### Env
|
||||||
|
|
||||||
|
- `GROKBOY_BROWSER_HEADED=1` — always launch Chromium headed (recommended when handoff is likely).
|
||||||
|
- `GROKBOY_HANDOFF_AUTO=1` — auto-resume (tests / CI); `abort` to auto-abort.
|
||||||
|
|
||||||
|
### Non-goals (this slice)
|
||||||
|
|
||||||
|
- Multi-agent orchestration
|
||||||
|
- Desktop accessibility / native UI automation
|
||||||
|
- External connectors / SaaS integrations
|
||||||
|
- Forking LazyBoy or Codex
|
||||||
|
|
||||||
|
### Acceptance (summary)
|
||||||
|
|
||||||
|
See `docs/ACCEPTANCE.md` section P4. Short list:
|
||||||
|
|
||||||
|
- `browser_handoff` registered and wired into the agent loop
|
||||||
|
- Headed Chromium for handoff; JSONL daemon keeps state within one run when already headed
|
||||||
|
- Bilingual terminal prompt; Enter / abort / timeout fail-closed
|
||||||
|
- Post-resume DOM snapshot as tool result
|
||||||
|
- Offline `cargo test` / `grokboy smoke` without API key or interactive stdin
|
||||||
|
|
||||||
|
## Roadmap hint (later)
|
||||||
|
|
||||||
|
P5+ may deepen persistence, richer session UX, or more tools — still thin core, DOM-first browser.
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# GrokBoy Playwright helper (optional)
|
# GrokBoy Playwright helper (optional)
|
||||||
|
|
||||||
Thin Node script used by Rust `browser_*` tools.
|
Thin Node script used by Rust `browser_*` tools (P3) and human handoff (P4).
|
||||||
|
|
||||||
## Install
|
## Install
|
||||||
|
|
||||||
|
|
@ -16,6 +16,14 @@ npx playwright install chromium
|
||||||
- **One-shot**: `node browser_helper.mjs --cmd '{"op":"ping"}'`
|
- **One-shot**: `node browser_helper.mjs --cmd '{"op":"ping"}'`
|
||||||
- **Self-test** (no Chromium): `npm run self-test`
|
- **Self-test** (no Chromium): `npm run self-test`
|
||||||
|
|
||||||
Ops: `ping`, `navigate`, `snapshot`/`dom`, `click`, `type`, `eval`, `close`, `status`.
|
Ops: `ping`, `navigate`, `snapshot`/`dom`, `click`, `type`, `eval`, `close`, `status`, **`handoff_prepare`**.
|
||||||
|
|
||||||
|
### `handoff_prepare`
|
||||||
|
|
||||||
|
Ensures a **headed** (visible) Chromium window and brings it to front. If the daemon was headless, closes and relaunches headed, then navigates back to the last URL when possible.
|
||||||
|
|
||||||
|
Note: relaunch uses a new browser context (cookies from the prior headless session are not carried unless storage state is added later). Prefer `GROKBOY_BROWSER_HEADED=1` so the first launch is already headed and state survives across handoff within one run.
|
||||||
|
|
||||||
|
Env: `GROKBOY_BROWSER_HEADED=1` launches headed for normal navigate as well.
|
||||||
|
|
||||||
Prefer CSS selector or `role`+`name` over screenshots.
|
Prefer CSS selector or `role`+`name` over screenshots.
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@
|
||||||
* One-shot: node browser_helper.mjs --cmd '<json>'
|
* One-shot: node browser_helper.mjs --cmd '<json>'
|
||||||
* Self-test: node browser_helper.mjs --self-test (no Chromium required)
|
* Self-test: node browser_helper.mjs --self-test (no Chromium required)
|
||||||
*
|
*
|
||||||
* Ops: ping, navigate, snapshot, click, type, eval, close, status
|
* Ops: ping, navigate, snapshot, click, type, eval, close, status, handoff_prepare
|
||||||
* Prefer CSS selector or role+name. Screenshots are NOT the primary control surface.
|
* Prefer CSS selector or role+name. Screenshots are NOT the primary control surface.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
|
|
@ -25,6 +25,20 @@ let browser = null;
|
||||||
let context = null;
|
let context = null;
|
||||||
let page = null;
|
let page = null;
|
||||||
let lastUrl = null;
|
let lastUrl = null;
|
||||||
|
/** Whether the current browser was launched headed (visible). */
|
||||||
|
let browserHeaded = false;
|
||||||
|
|
||||||
|
function envHeadedDefault() {
|
||||||
|
const v = (process.env.GROKBOY_BROWSER_HEADED || "").trim().toLowerCase();
|
||||||
|
if (v === "1" || v === "true" || v === "yes") return true;
|
||||||
|
if (v === "0" || v === "false" || v === "no") return false;
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
function wantHeaded(forceHeaded) {
|
||||||
|
if (forceHeaded === true) return true;
|
||||||
|
return envHeadedDefault();
|
||||||
|
}
|
||||||
|
|
||||||
function ok(id, extra = {}) {
|
function ok(id, extra = {}) {
|
||||||
return { id: id ?? null, ok: true, ...extra };
|
return { id: id ?? null, ok: true, ...extra };
|
||||||
|
|
@ -50,11 +64,11 @@ async function loadPlaywright() {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function ensurePage() {
|
async function launchBrowser(headed) {
|
||||||
if (page) return page;
|
|
||||||
const { chromium } = await loadPlaywright();
|
const { chromium } = await loadPlaywright();
|
||||||
try {
|
try {
|
||||||
browser = await chromium.launch({ headless: true });
|
browser = await chromium.launch({ headless: !headed });
|
||||||
|
browserHeaded = headed;
|
||||||
} catch (e) {
|
} catch (e) {
|
||||||
const err = new Error(
|
const err = new Error(
|
||||||
`${INSTALL_HINT} (chromium launch failed: ${e.message})`
|
`${INSTALL_HINT} (chromium launch failed: ${e.message})`
|
||||||
|
|
@ -67,6 +81,78 @@ async function ensurePage() {
|
||||||
return page;
|
return page;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
async function ensurePage(opts = {}) {
|
||||||
|
const headed = wantHeaded(opts.headed === true);
|
||||||
|
if (page) {
|
||||||
|
// Already open; if caller needs headed and we are headless, relaunch below via handoff_prepare.
|
||||||
|
return page;
|
||||||
|
}
|
||||||
|
return launchBrowser(headed);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Close current browser and reopen headed, restoring lastUrl when possible. */
|
||||||
|
async function ensureHeadedForHandoff() {
|
||||||
|
const restore = lastUrl || (page ? page.url() : null);
|
||||||
|
if (browser && browserHeaded && page) {
|
||||||
|
try {
|
||||||
|
await page.bringToFront();
|
||||||
|
} catch {
|
||||||
|
/* ignore */
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
prepared: true,
|
||||||
|
headed: true,
|
||||||
|
relaunched: false,
|
||||||
|
url: page.url(),
|
||||||
|
restored: false,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
if (browser) {
|
||||||
|
await browser.close().catch(() => {});
|
||||||
|
browser = null;
|
||||||
|
context = null;
|
||||||
|
page = null;
|
||||||
|
browserHeaded = false;
|
||||||
|
}
|
||||||
|
|
||||||
|
await launchBrowser(true);
|
||||||
|
let restored = false;
|
||||||
|
if (restore && restore !== "about:blank") {
|
||||||
|
try {
|
||||||
|
await page.goto(String(restore), {
|
||||||
|
waitUntil: "domcontentloaded",
|
||||||
|
timeout: 30000,
|
||||||
|
});
|
||||||
|
lastUrl = page.url();
|
||||||
|
restored = true;
|
||||||
|
} catch (e) {
|
||||||
|
// Keep headed page even if restore fails; caller still sees visible window.
|
||||||
|
lastUrl = page.url();
|
||||||
|
return {
|
||||||
|
prepared: true,
|
||||||
|
headed: true,
|
||||||
|
relaunched: true,
|
||||||
|
url: lastUrl,
|
||||||
|
restored: false,
|
||||||
|
restore_error: String(e.message || e),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
await page.bringToFront();
|
||||||
|
} catch {
|
||||||
|
/* ignore */
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
prepared: true,
|
||||||
|
headed: true,
|
||||||
|
relaunched: true,
|
||||||
|
url: page ? page.url() : null,
|
||||||
|
restored,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/** Accessibility-ish text snapshot: roles, names, and useful selectors — not pixels. */
|
/** Accessibility-ish text snapshot: roles, names, and useful selectors — not pixels. */
|
||||||
async function buildSnapshot(p) {
|
async function buildSnapshot(p) {
|
||||||
const nodes = await p.evaluate(() => {
|
const nodes = await p.evaluate(() => {
|
||||||
|
|
@ -242,14 +328,27 @@ async function handle(req) {
|
||||||
"eval",
|
"eval",
|
||||||
"close",
|
"close",
|
||||||
"status",
|
"status",
|
||||||
|
"handoff_prepare",
|
||||||
],
|
],
|
||||||
});
|
});
|
||||||
|
|
||||||
case "status": {
|
case "status": {
|
||||||
return ok(id, {
|
return ok(id, {
|
||||||
browser_open: !!browser,
|
browser_open: !!browser,
|
||||||
|
headed: browserHeaded,
|
||||||
last_url: lastUrl,
|
last_url: lastUrl,
|
||||||
page_url: page ? page.url() : null,
|
page_url: page ? page.url() : null,
|
||||||
|
env_headed: envHeadedDefault(),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
case "handoff_prepare": {
|
||||||
|
// Bring a visible Chromium window forward for human login/OTP/captcha.
|
||||||
|
const info = await ensureHeadedForHandoff();
|
||||||
|
return ok(id, {
|
||||||
|
...info,
|
||||||
|
message:
|
||||||
|
"Browser is headed (visible). Complete login/OTP/captcha in the window, then resume in the terminal.",
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -342,6 +441,7 @@ async function handle(req) {
|
||||||
browser = null;
|
browser = null;
|
||||||
context = null;
|
context = null;
|
||||||
page = null;
|
page = null;
|
||||||
|
browserHeaded = false;
|
||||||
return ok(id, { closed: true, last_url: lastUrl });
|
return ok(id, { closed: true, last_url: lastUrl });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -375,6 +475,10 @@ async function runSelfTest() {
|
||||||
// Protocol: ping without playwright
|
// Protocol: ping without playwright
|
||||||
let r = await handle({ id: "t1", op: "ping" });
|
let r = await handle({ id: "t1", op: "ping" });
|
||||||
cases.push(["ping", r.ok === true && r.pong === true]);
|
cases.push(["ping", r.ok === true && r.pong === true]);
|
||||||
|
cases.push([
|
||||||
|
"ping_lists_handoff_prepare",
|
||||||
|
Array.isArray(r.ops) && r.ops.includes("handoff_prepare"),
|
||||||
|
]);
|
||||||
|
|
||||||
// Missing fields fail closed
|
// Missing fields fail closed
|
||||||
r = await handle({ id: "t2", op: "navigate" });
|
r = await handle({ id: "t2", op: "navigate" });
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue