38 KiB

Raw Blame History

LazyBoy → Cua Driver Migration Plan

Purpose: This document is an implementation specification for a coding agent.

Repository: https://code.30cm.net/daniel.w/lazyBoy

Cua: https://github.com/trycua/cua

Cua docs: https://cua.ai/docs

Main principle: Do NOT rewrite LazyBoy into Cua. Keep LazyBoy as the agent/workspace/product layer and replace the low-level computer-control implementation with Cua Driver incrementally.

0. Agent Instructions

You are modifying LazyBoy, a self-hosted AI agent workspace.

The goal is to migrate the low-level GUI/browser computer-control implementation from custom CDP / AT-SPI / X11 code to Cua Driver, without breaking LazyBoy's existing product architecture.

Non-negotiable rules

Do not perform a Big Bang rewrite.
Do not remove the legacy computer-control implementation in the first PR.
Keep LazyBoy's current public Agent tool schema stable unless absolutely necessary.
Keep:
- computer_observe
- computer_act
- browser
- shell
- file tools
- takeover flow
- memory
- schedule
- vault
- skills/playbooks
Cua must initially be an implementation detail behind LazyBoy abstractions.
Do not expose all Cua MCP tools directly to the LLM.
Do not replace DockerSandbox, Supervisor, noVNC, or persistent bot homes during Phase 1.
Add a feature flag / backend selector so legacy behavior can be restored immediately.
Every migrated action must have observable verification.
Do not guess Cua API/tool names from this document.

Cua API freshness rule

Cua changes quickly.

Before implementing anything:

cua-driver --version
cua-driver doctor
cua-driver list-tools

For every Cua tool you plan to call:

cua-driver describe <tool-name>

Use the schema reported by the installed Cua Driver version as the source of truth.

Do not hard-code assumptions from old blog posts, old examples, or this document when the installed version differs.

1. Current LazyBoy Architecture

The current architecture is approximately:

User
 │
 ▼
React Web UI
 │
 ▼
crates/api
 │
 ├── Agent loop
 ├── Sessions / Runs
 ├── Memory
 ├── Schedule
 ├── Vault
 ├── Skills
 ├── MCP
 └── Tool definitions
 │
 ▼
SandboxProvider
 │
 ▼
crates/sandbox
 │
 ▼
DockerSandbox
 │
 ▼
crates/supervisor
 │
 ▼
Linux Desktop Container
 │
 ├── XFCE
 ├── Chromium
 ├── Xvfb
 ├── x11vnc
 ├── websockify
 └── controld
       │
       ├── CDP
       ├── AT-SPI
       ├── X11
       └── screenshots

Relevant current components:

apps/web
crates/api
crates/contracts
crates/control
crates/controld
crates/harness
crates/sandbox
crates/supervisor

The existing SandboxProvider abstraction is an important migration seam and should be preserved.

Current methods include approximately:

async fn provision(...)
async fn prepare(...)
async fn capabilities(...)
async fn ensure_screen(...)
async fn reconnect(...)
async fn suspend(...)
async fn resume(...)
async fn execute(...)
async fn observe(...)
async fn act(...)
async fn connect_screen(...)
async fn list_files(...)
async fn read_file(...)
async fn write_file(...)
async fn stop(...)
async fn destroy(...)

Do not collapse this abstraction.

2. Target Architecture

Phase 1 target:

                         LazyBoy Web
                              │
                              ▼
                         LazyBoy API
                              │
                       LazyBoy Agent Loop
                              │
             ┌────────────────┼────────────────┐
             │                │                │
             ▼                ▼                ▼
          Memory            Skills          Schedule
                              │
                              ▼
                       LazyBoy Tool Layer
                              │
                  computer / browser / shell
                              │
                              ▼
                    ComputerController
                       abstraction
                         │       │
             ┌───────────┘       └────────────┐
             ▼                                ▼
     LegacyController                  CuaController
             │                                │
      CDP / AT-SPI / X11                 Cua Driver
                                               │
                                ┌──────────────┼──────────────┐
                                ▼              ▼              ▼
                              X11           AT-SPI         Browser
                                               │
                                               ▼
                                      Existing Desktop
                                         Container

Important

Cua is initially used as the computer-control backend.

It is not the Agent runtime.

It is not the memory system.

It is not the scheduler.

It is not the user-facing product.

It is not the first-phase sandbox lifecycle owner.

3. What Must Stay in LazyBoy

The following are LazyBoy product responsibilities and must remain owned by LazyBoy.

Keep unchanged unless required

Frontend

apps/web

Keep:

Chat UI
remote desktop
noVNC
mobile controls
takeover UI
bot management
session UI

API / Agent layer

crates/api

Keep:

Agent loop
model interaction
run management
session management
tool policy
takeover workflow
credential flow
memory
schedules
skills
MCP integrations

Harness

crates/harness

Keep model-provider support.

Cua Driver must not decide which model LazyBoy uses.

Supervisor

crates/supervisor

Phase 1: keep current behavior.

Keep:

Docker lifecycle
pause/resume
resource limits
persisted bot homes
screen lifecycle

Sandbox

crates/sandbox

Phase 1: keep DockerSandbox.

Do not replace it with Cua Sandbox yet.

4. What Cua Should Replace

The long-term goal is to reduce LazyBoy-owned OS automation.

Current files that are candidates for replacement or simplification:

crates/control/src/a11y.py
crates/control/src/a11y.rs

crates/control/src/cdp.py
crates/control/src/cdp.rs

crates/control/src/x11.rs
crates/control/src/screen.rs
crates/control/src/overlay.rs

Do not delete them during the initial migration.

Instead place them behind a legacy backend.

The responsibilities that should gradually move to Cua:

screenshots
application/window enumeration
accessibility tree
native UI element actions
pointer input
keyboard input
scrolling
browser state
browser semantic actions
window-scoped control
recording / trajectory capture

5. Do NOT Expose Raw Cua to the LLM

LazyBoy currently has a relatively compact Agent-facing tool surface.

Preserve this.

Preferred model:

LLM
 │
 ▼
LazyBoy tools
 │
 ├── computer_observe
 ├── computer_act
 ├── browser
 ├── shell
 ├── file tools
 └── request_takeover
 │
 ▼
LazyBoy policy / safety / vault / state
 │
 ▼
Cua adapter
 │
 ▼
Cua Driver

Avoid this:

LLM
 │
 ▼
50+ raw Cua tools
 │
 ▼
OS

Reasons:

larger tool schemas consume context
LazyBoy loses policy control
LazyBoy loses stable abstraction
Cua version changes would leak into prompts
takeover behavior becomes harder to control
credential handling becomes harder to constrain
Agent behavior becomes coupled to Cua implementation details

Cua should initially behave like a device driver.

6. Introduce a ComputerController Abstraction

Create a low-level computer-control abstraction separate from sandbox lifecycle.

Suggested location:

crates/control/src/controller.rs

Possible interface:

#[async_trait::async_trait]
pub trait ComputerController: Send + Sync {
    async fn health(&self, ctx: &ControlContext)
        -> Result<ControllerHealth, ControlError>;

    async fn observe(
        &self,
        request: ObserveRequest,
        ctx: &ControlContext,
    ) -> Result<ComputerObservation, ControlError>;

    async fn act(
        &self,
        request: ActionRequest,
        ctx: &ControlContext,
    ) -> Result<ActionResult, ControlError>;

    async fn browser(
        &self,
        request: BrowserRequest,
        ctx: &ControlContext,
    ) -> Result<BrowserResult, ControlError>;

    async fn start_recording(
        &self,
        request: RecordingRequest,
        ctx: &ControlContext,
    ) -> Result<RecordingSession, ControlError>;

    async fn stop_recording(
        &self,
        ctx: &ControlContext,
    ) -> Result<RecordingResult, ControlError>;
}

Names may be adjusted to match existing LazyBoy contracts.

Do not introduce unnecessary abstractions if equivalent types already exist.

7. Backend Implementations

Implement:

LegacyController
CuaController

Suggested layout:

crates/control/src/
├── controller.rs
├── legacy/
│   ├── mod.rs
│   ├── a11y.rs
│   ├── cdp.rs
│   ├── x11.rs
│   └── screen.rs
└── cua/
    ├── mod.rs
    ├── client.rs
    ├── translate.rs
    ├── observe.rs
    ├── actions.rs
    ├── browser.rs
    └── recording.rs

Do not spend the first PR moving all old files if it creates noisy diffs.

It is acceptable to initially keep existing file locations and only add:

controller.rs
cua.rs

Refactor structure after behavior is stable.

8. Backend Configuration

Add a configuration value:

LAZYBOY_COMPUTER_DRIVER=legacy

or:

LAZYBOY_COMPUTER_DRIVER=cua

Default during initial rollout:

LAZYBOY_COMPUTER_DRIVER=legacy

After Cua passes production-equivalent validation, the default may become:

LAZYBOY_COMPUTER_DRIVER=cua

Do not remove the legacy option until Cua has passed the migration acceptance suite.

If configuration already has a typed settings system, add this there instead of reading environment variables throughout the code.

Suggested enum:

pub enum ComputerDriver {
    Legacy,
    Cua,
}

9. Recommended Cua Integration Mode

LazyBoy is an application with built-in computer-use.

Preferred order for experimentation:

POC

Use the Cua Driver CLI / daemon boundary first if it allows fast validation.

Examples:

cua-driver call <tool> '<json>'

or a controlled local process interface.

This proves Cua works inside the existing Linux desktop container.

Production integration

Prefer a typed/stable integration boundary.

Evaluate, based on the installed Cua release:

direct in-process SDK if suitable for LazyBoy's process model
private worker / daemon
CLI JSON calls as fallback

Do not choose MCP merely because Agent frameworks often use MCP.

LazyBoy is embedding computer use inside a product. The internal integration does not have to look like the Agent-facing integration.

Rust note

Cua Driver's core runtime is Rust-based, but public application SDK support may differ by release.

Do not write custom unsafe bindings unless there is a strong reason.

Prefer an officially supported application integration surface.

10. Phase 0 — Discovery / Compatibility Check

Before changing LazyBoy behavior, create a short engineering report.

The coding agent must verify the actual runtime environment.

Inside a real LazyBoy desktop container:

echo "$DISPLAY"
ps aux
env | sort

Confirm:

X11 display exists
XFCE session exists
AT-SPI bus is available
Chromium exists
Cua Driver can start
Cua Driver can capture the current display
Cua Driver can enumerate apps/windows
Cua Driver can operate in Xvfb
Cua Driver can access Chromium in this environment

Install Cua Driver using the current official installation method.

Then run:

cua-driver --version
cua-driver doctor
cua-driver list-tools

Save results in:

docs/cua-compatibility.md

Include:

installed Cua version
Linux distribution
display server
Cua doctor output summary
supported Cua tools relevant to LazyBoy
missing dependencies
known limitations

Stop condition

If Cua cannot reliably control the existing XFCE + Xvfb environment, do not proceed with architecture replacement.

Instead document the blocker first.

11. Phase 1 — Five-Action POC

Do not begin by migrating the whole control crate.

First prove these five capabilities inside the existing LazyBoy desktop:

screenshot
accessibility/window observation
click native GUI element
type text
browser semantic action

Optional sixth:

scroll

Create a standalone POC path.

Example:

scripts/cua-smoke-test.sh

or:

crates/control/examples/cua_smoke.rs

POC flow

Suggested test:

1. Open a native/simple XFCE application.
2. Observe window state.
3. Click an accessible element.
4. Type text.
5. Launch/use Chromium.
6. Navigate to example.com.
7. Read browser state.
8. Trigger a semantic browser interaction if available.
9. Capture final screenshot.

POC success criteria

All five core actions succeed 10 consecutive times without:

wrong window actions
stale element actions
focus corruption
unexplained timeouts
leaving Cua processes behind
breaking noVNC control

Do not migrate production paths until this passes.

12. Phase 2 — Migrate `computer_observe`

First production migration target:

computer_observe

Current LazyBoy observation must remain compatible with the Agent.

Do not change the Agent prompt contract unless required.

Cua result must be translated into:

ComputerObservation

Preserve existing concepts where possible:

screenshot/image
dimensions
cursor
active window
UI elements

Translation layer

Implement:

Cua state
   ↓
CuaObservationAdapter
   ↓
LazyBoy ComputerObservation

Do not expose Cua-native short-lived element identifiers directly as permanent LazyBoy identifiers.

Element references may only be valid for a specific observation.

Treat each observation as a snapshot.

Observation ID

Strongly consider adding or preserving an observation/snapshot identifier.

Example:

pub struct ComputerObservation {
    pub observation_id: String,
    ...
}

If changing the contract is too invasive, keep this internal first.

This will later help detect stale actions.

13. Phase 3 — Migrate `computer_act`

Preserve the existing LazyBoy ComputerAction DSL.

Current behavior such as:

click
move
pointer down/up
hover
drag
type
keyboard
wait
semantic references

should remain Agent-facing LazyBoy concepts.

Create a translator:

LazyBoy ComputerAction
          │
          ▼
   CuaActionTranslator
          │
          ▼
       Cua Driver

Example conceptual mapping:

LazyBoy click(element)
    → semantic Cua action when possible

LazyBoy click(x, y)
    → pixel Cua action

LazyBoy type(text)
    → Cua text input

LazyBoy key(...)
    → Cua key action

LazyBoy scroll(...)
    → Cua scroll

Exact tool names MUST come from:

cua-driver list-tools
cua-driver describe ...

Preserve semantic-first behavior

Prefer:

Accessibility / browser semantic action

over:

screen coordinate

Coordinate action should be fallback, not default.

14. Preserve LazyBoy Action Safety Logic

Do not delete useful behavior from:

crates/control/src/actions.rs

Existing logic includes concepts such as:

action batch limits
coordinate validation
element lookup
stale click prevention
browser semantic-routing preference
double-click expansion
drag normalization

These should become policy/translation logic above Cua.

Desired layering:

Agent request
      │
      ▼
LazyBoy validation
      │
      ▼
LazyBoy action policy
      │
      ▼
Cua translation
      │
      ▼
Cua Driver

Cua is not a replacement for LazyBoy product policy.

15. Stale Element Handling

This is critical.

Do not assume:

element 12

from one Cua observation refers to the same UI element later.

The migration must treat element references as snapshot-scoped.

Preferred flow:

observe
  ↓
snapshot A
  ↓
Agent selects element
  ↓
act against snapshot A
  ↓
UI changes
  ↓
observe again
  ↓
snapshot B

If the UI changed materially, do not retry an old semantic reference blindly.

Retry rule

When an action fails:

re-observe
re-resolve the target
retry with bounded count

Never loop the same stale action indefinitely.

16. Phase 4 — Migrate Browser Control

LazyBoy currently separates browser actions from generic computer actions.

Keep that separation for the Agent.

Agent-facing:

browser

Internal:

LazyBoy BrowserRequest
        │
        ▼
Cua Browser Adapter
        │
        ▼
Cua Driver browser tools

Cua browser control should replace custom CDP behavior gradually.

Important

Preserve the current policy:

When semantic browser state is available,
prefer browser semantic actions over pixel clicking Chromium.

Do not make browser automation less reliable during migration.

Test cases

At minimum test:

navigate URL
inspect page
click semantic element
type into input
scroll
multiple tabs if LazyBoy currently depends on them
file picker transition
page refresh
browser restart
authenticated persistent profile

17. Browser Profile Compatibility

LazyBoy persists browser profiles per computer/bot.

This is product-critical.

Do not let Cua silently replace LazyBoy's profile with an ephemeral managed browser unless explicitly intended.

Verify:

existing Chromium profile path remains usable
cookies survive container pause/resume
login sessions survive LazyBoy restart behavior as expected
takeover user and Agent see the same session
Cua attaches to the correct browser/window

If Cua requires an explicit browser preparation/attach step, integrate that into LazyBoy lifecycle.

Do not auto-create a separate profile that breaks existing saved logins.

18. Phase 5 — Recording Integration

LazyBoy has an important feature:

Human demonstration
        ↓
record context/actions
        ↓
model generates playbook
        ↓
future run resolves current UI

Preserve this architecture.

Do not downgrade it into raw coordinate replay.

Cua recording should be used as richer source data.

Target:

Human / Agent demonstration
           │
           ▼
      Cua trajectory
           │
           ├── before state
           ├── action
           ├── after state
           ├── screenshots
           └── optional video
           │
           ▼
  LazyBoy Skill Compiler
           │
           ▼
     Semantic Playbook

Cua trajectory is evidence.

LazyBoy skill/playbook is the reusable automation.

Do not do

record x=312,y=441
replay x=312,y=441 forever

Do

Store semantic intent when possible:

Click the "Sign in" button

Then resolve it on the current screen during replay.

19. Takeover Must Keep Working

User takeover is a core LazyBoy feature.

Migration acceptance requires:

Agent running
   ↓
request_takeover
   ↓
Agent stops issuing input
   ↓
User controls noVNC desktop
   ↓
User releases takeover
   ↓
Agent re-observes
   ↓
Agent resumes from current state

Required rule

After takeover ends:

Always perform a fresh observation before the Agent performs another UI action.

Never reuse pre-takeover element references.

20. Credential / Vault Boundary

Cua must not get broad access to LazyBoy secrets by default.

Keep credential policy in LazyBoy.

Preferred flow:

Agent wants login
     │
     ▼
LazyBoy checks:
- allowed domain
- saved credential exists
- HTTPS / policy
     │
     ▼
LazyBoy authorizes injection
     │
     ▼
Cua performs allowed typing/action

Do not make the Cua integration read the entire Vault.

Secrets should not appear:

in command-line arguments
in logs
in Cua debug output
in trajectory metadata
in screenshots longer than unavoidable
in error messages

Audit recording behavior around passwords.

If recording is active, ensure sensitive typing can be masked or recording paused.

21. `controld` Migration Strategy

Do not delete crates/controld during Phase 1.

It is a useful compatibility boundary.

Current conceptual API:

POST /observe
POST /act

Recommended first migration:

POST /observe
    ↓
selected ComputerController
    ↓
CuaController or LegacyController

POST /act
    ↓
LazyBoy validation
    ↓
selected ComputerController
    ↓
CuaController or LegacyController

This keeps:

API
SandboxProvider
Supervisor

largely unchanged.

Later, if Cua integration makes controld unnecessary, remove it in a dedicated architectural PR.

Do not mix that cleanup into the initial migration.

22. Docker Image Changes

Add Cua dependencies to the LazyBoy desktop image.

The coding agent must locate the actual Dockerfile(s) used for desktop computers.

Do not assume the path.

Changes may include:

Cua Driver install
required X11 packages
AT-SPI dependencies
ffmpeg if recording/video is enabled
runtime directories
permissions
PATH configuration

Run:

cua-driver doctor

inside the built desktop container as part of the smoke test.

Image versioning

Pin a known-working Cua version for reproducible builds.

Do not install uncontrolled nightly builds in the default production image.

Optionally allow:

LAZYBOY_CUA_CHANNEL=stable
LAZYBOY_CUA_VERSION=<version>

or equivalent build args.

23. Cua Process Lifecycle

Do not start one uncontrolled global Cua process for every LazyBoy computer unless architecture requires it.

Determine which process should own:

display connection
accessibility session
browser connection
recordings
Cua lifecycle

For LazyBoy's existing architecture, the safest initial design is usually:

one desktop container
      │
      ├── XFCE/Xvfb
      ├── Chromium
      ├── noVNC
      ├── LazyBoy controld
      └── Cua runtime/driver

The driver should only see/control that computer's desktop session.

Isolation

Bot A must never control Bot B's display.

Tests must explicitly verify isolation.

24. Health Checks

Add a Cua health check.

Possible information:

pub struct ControllerHealth {
    pub backend: String,
    pub version: Option<String>,
    pub healthy: bool,
    pub degraded: bool,
    pub details: Vec<String>,
}

Surface useful failures:

driver not installed
X11 unavailable
AT-SPI unavailable
screenshot unavailable
browser unavailable
incompatible Cua version

Do not return generic:

computer failed

when a meaningful diagnosis is available.

25. Fallback Behavior

During migration:

LAZYBOY_COMPUTER_DRIVER=legacy

must work.

For cua mode, avoid silent fallback for individual actions unless explicitly designed.

Bad:

Cua click failed
→ silently xdotool click

This makes failures impossible to debug.

Preferred:

Cua action failed
→ return classified error
→ Agent re-observes / retries / requests takeover

A feature-flag-level fallback is acceptable.

An invisible per-action fallback is not.

26. Logging / Observability

Add structured logs around Cua calls.

Include:

operation ID
run ID
bot ID
screen ID
backend
Cua tool
duration
success/failure
action route if Cua reports it
observation ID if available

Never log secret text.

For typing actions:

text="<redacted>"
length=12

not:

text="actual-password"

27. Metrics

If LazyBoy has metrics infrastructure, add:

computer_action_total
computer_action_failed_total
computer_observe_duration_ms
computer_action_duration_ms
computer_browser_action_duration_ms
computer_stale_reference_total
computer_takeover_total
cua_driver_restart_total

Useful labels:

backend
action_type
route
result

Avoid high-cardinality IDs such as run_id as metric labels.

28. Testing Strategy

Unit tests

Test translation only.

Examples:

LazyBoy click semantic ref → expected Cua request

LazyBoy coordinate click → expected Cua request

LazyBoy type → expected Cua request

Cua observation → LazyBoy ComputerObservation

unsupported Cua response → classified error

Use fake/mocked Cua responses.

Contract tests

The Agent-facing tool results should remain equivalent between:

LegacyController
CuaController

for common scenarios.

Test:

observation shape
action result shape
error behavior
browser results
takeover interaction

Integration tests

Run against a real desktop container.

Test native GUI:

open app
observe
click
type
verify application state

Do not validate only that Cua returned "ok".

Validate independent application state.

Browser E2E

Use a deterministic local test page rather than an external website.

Create fixtures for:

button
input
checkbox
select
scroll area
delayed DOM update
modal
new tab
canvas fallback if needed

Verify actual DOM/application state.

29. Migration Acceptance Suite

Cua backend is not considered ready until all of the following pass.

Desktop

screenshot works
window enumeration works
active window works
accessibility elements work
semantic click works
coordinate click works
type text works
hotkey works
scroll works
drag works if supported
desktop stays usable through noVNC

Browser

attach to correct Chromium
use persistent LazyBoy profile
navigate
observe DOM/browser state
semantic click
type
scroll
refresh
profile survives pause/resume
same browser is visible to human takeover

Lifecycle

fresh container
existing persisted container
pause
resume
stop
restart
concurrent bots
no cross-bot control

Agent flow

computer_observe
computer_act
browser
shell
takeover
resume after takeover
skill recording
scheduled run

30. Performance Benchmark

Before defaulting to Cua, compare against legacy.

Create a benchmark report:

docs/cua-benchmark.md

Measure:

Scenario	Legacy	Cua	Winner
screenshot
observe desktop
native click
type text
browser snapshot
browser click
20-step browser task

Also measure:

tool-call count
bytes returned to model
screenshot count
total model-visible observation size
end-to-end task completion time
failure rate

The goal is not only faster pointer execution.

The real goal is:

fewer Agent turns
+
less context
+
higher task success rate

31. Rollout Plan

Step A

Add:

ComputerController
LegacyController

No behavior change.

All tests must pass.

Step B

Add Cua smoke test.

No production traffic.

Step C

Implement:

CuaController.observe

Feature flagged.

Step D

Implement simple actions:

click
type
keyboard
scroll

Step E

Migrate browser state/actions.

Step F

Add recording integration.

Step G

Run acceptance suite and benchmark.

Step H

Change default:

legacy
→
cua

but keep legacy rollback.

Step I

After stable operation, delete obsolete low-level code in separate PRs.

Possible deletion candidates:

a11y.py
a11y.rs
cdp.py
cdp.rs
x11.rs
screen implementation portions

Only delete code proven unused.

32. Suggested PR Sequence

Keep PRs small.

PR 1

refactor(control): introduce pluggable ComputerController

add interface
wrap legacy
no behavior change
tests

PR 2

build(desktop): install and validate Cua Driver

image changes
doctor
smoke test
compatibility doc

PR 3

feat(control): add Cua observation backend

PR 4

feat(control): route computer actions through Cua

PR 5

feat(browser): add Cua browser adapter

PR 6

feat(skills): ingest Cua trajectories for demonstrations

PR 7

test(control): add Cua acceptance and benchmark suite

PR 8

chore(control): make Cua the default backend

Later

chore(control): remove obsolete legacy OS automation

33. Error Model

Map Cua failures to typed LazyBoy errors.

Suggested categories:

pub enum ControlError {
    DriverUnavailable,
    DriverUnhealthy,
    DisplayUnavailable,
    AccessibilityUnavailable,
    BrowserUnavailable,
    TargetNotFound,
    StaleReference,
    PermissionDenied,
    Timeout,
    Unsupported,
    Busy,
    InvalidAction,
    Internal(String),
}

Do not expose Cua raw errors directly to the Agent when a stable LazyBoy error can represent them.

Log the raw cause internally.

34. Retry Policy

Do not blindly retry UI actions.

Recommended:

Observation failure

Retry a small bounded number if transport/runtime error appears transient.

Semantic target not found

re-observe
→ re-resolve
→ retry once

Stale target

re-observe
→ never retry original snapshot ref directly

Pixel miss

Do not repeat the same click indefinitely.

Preserve LazyBoy's stale-click protection concept.

Driver crash

Restart driver/runtime if safe, then require a fresh observation.

35. Screenshot Policy

Avoid sending screenshots to the model when structured UI state is sufficient.

Prefer:

semantic / accessibility observation

Use screenshots when:

layout matters
semantic state is incomplete
canvas/WebGL is involved
visual verification is needed

This reduces:

latency
model context
vision token cost
accidental secret exposure

36. Cua Permission Policy

If the chosen Cua deployment mode supports permission policies, use allow-list behavior.

LazyBoy only needs a subset of Cua capabilities.

Allow only required operations.

Example conceptual set:

health/doctor-like observation
window/app listing
screenshot
window state
click
type
key
scroll
browser state
browser click
browser type
browser navigation
recording

Do not allow unrelated capabilities automatically.

Use actual current Cua tool names from the installed version.

37. Security Requirements

Cua must not access the host Docker socket from Agent desktops.
Each bot must remain isolated.
Keep LazyBoy supervisor on internal network.
Keep controld localhost/internal where applicable.
Do not expose Cua daemon ports publicly.
Do not log passwords/tokens.
Verify trajectory storage permissions.
Verify screenshots are covered by LazyBoy retention policy.
Do not allow an Agent to change Cua policies.
Do not let the Agent select another bot's display/session.
Cua executable/version should be controlled by LazyBoy image/build process.

38. Multi-Screen Considerations

LazyBoy already has screen concepts:

screen_lease_id
screen_id
screen_slot
display

Do not discard these.

Cua must be bound to the LazyBoy-selected display/screen.

Verify multi-screen behavior before enabling Cua for multi-screen bots.

If Cua does not safely support LazyBoy's current multi-screen semantics:

support one screen first
return explicit capability information
do not fake support

39. Cua Capability Mapping

LazyBoy should expose backend capabilities.

Suggested internal structure:

pub struct ComputerCapabilities {
    pub multi_screen: bool,
    pub semantic_desktop: bool,
    pub semantic_browser: bool,
    pub pixel_actions: bool,
    pub recording: bool,
    pub background_input: bool,
}

Only add fields if needed and compatible with existing contract evolution.

Use capability checks instead of OS-name conditionals where possible.

Bad:

if linux {
    ...
}

Better:

if capabilities.semantic_browser {
    ...
}

40. Future Phase — Cua Sandbox

Do not implement this during the Driver migration.

Later, LazyBoy may support:

SandboxProvider
     │
     ├── LazyBoyDocker
     └── CuaSandbox

Possible future computer types:

Lightweight Linux
Linux VM
Windows VM
macOS VM
Cloud Computer

This should be a separate project after Cua Driver migration is stable.

The current SandboxProvider abstraction should make this possible.

41. Future Phase — Windows / macOS

One strategic reason for Cua is avoiding custom implementations for:

Linux AT-SPI/X11
Windows UIA/input/capture
macOS AX/input/capture

Do not add Windows/macOS support during the initial Linux migration.

First ensure the LazyBoy control contract is OS-neutral.

Then add backend capabilities and sandbox providers later.

42. Definition of Done — Phase 1

Phase 1 is complete when:

LazyBoy can run with:

LAZYBOY_COMPUTER_DRIVER=legacy

and:

LAZYBOY_COMPUTER_DRIVER=cua

Existing Agent tool schemas remain compatible.
Cua works in the existing LazyBoy Linux desktop container.
computer_observe works through Cua.
computer_act core actions work through Cua.
Browser semantic operations work through Cua.
noVNC human takeover still works.
Persistent Chromium sessions still work.
Docker pause/resume still works.
Multi-bot isolation is verified.
Legacy backend remains available for rollback.
No secret is exposed in logs or recording metadata.
Acceptance tests pass.
Benchmark results are documented.

43. First Task for the Coding Agent

Do this first and nothing larger:

Task

Create a branch:

feat/cua-driver-poc

Then:

inspect current LazyBoy computer-control flow
identify actual desktop Dockerfile/image entrypoint
install a pinned stable Cua Driver in that image
make no Agent-facing schema changes
create a Cua smoke test that runs inside a LazyBoy desktop
verify:
- cua-driver --version
- cua-driver doctor
- screenshot
- window/accessibility observation
- native click
- typing
- Chromium/browser state
write results to:

docs/cua-compatibility.md

stop after the POC
do not delete any legacy control code
report blockers before starting the controller refactor

Expected output

The first PR should answer only:

"Can Cua Driver reliably control the existing LazyBoy XFCE + Xvfb desktop container?"

If the answer is yes, proceed to the next PR.

44. Second Task After POC Passes

Create:

ComputerController

with:

LegacyController
CuaController

Initially make all production requests use:

LegacyController

Then route only test/flagged traffic to:

CuaController

Do not migrate browser and recording in the same PR.

45. Architecture Decision

The intended ownership after migration is:

Concern	Owner
Product UI	LazyBoy
Agent loop	LazyBoy
Model provider	LazyBoy
Memory	LazyBoy
Schedules	LazyBoy
Vault	LazyBoy
Takeover	LazyBoy
Skills/playbooks	LazyBoy
Sandbox lifecycle (Phase 1)	LazyBoy
Docker resources	LazyBoy
Remote desktop/noVNC	LazyBoy
Computer observation	Cua
Native UI control	Cua
Pointer/keyboard	Cua
Browser semantic control	Cua
OS-specific accessibility	Cua
Trajectory capture	Cua
Workflow abstraction from recording	LazyBoy

This is the central design decision.

Do not invert it.

46. Final Principle

LazyBoy's competitive/product value is:

Agent workspace
Multi-agent
Persistent computers
Human takeover
Memory
Schedules
Vault
Skills
Demo → reusable workflow
Web/mobile UX
Model choice

The goal of using Cua is to stop spending LazyBoy engineering effort on:

X11 automation
AT-SPI edge cases
CDP plumbing
screen capture
OS-specific input
window enumeration
cross-platform UI automation

Build LazyBoy on top of Cua.

Do not turn LazyBoy into Cua.

References

LazyBoy:

Cua:

38 KiB Raw Blame History

LazyBoy → Cua Driver Migration Plan

0. Agent Instructions

Non-negotiable rules

Cua API freshness rule

1. Current LazyBoy Architecture

2. Target Architecture

Important

3. What Must Stay in LazyBoy

Keep unchanged unless required

Frontend

API / Agent layer

Harness

Supervisor

Sandbox

4. What Cua Should Replace

5. Do NOT Expose Raw Cua to the LLM

6. Introduce a ComputerController Abstraction

7. Backend Implementations

8. Backend Configuration

9. Recommended Cua Integration Mode

POC

Production integration

Rust note

10. Phase 0 — Discovery / Compatibility Check

Stop condition

11. Phase 1 — Five-Action POC

POC flow

POC success criteria

12. Phase 2 — Migrate computer_observe

Translation layer

Observation ID

13. Phase 3 — Migrate computer_act

Preserve semantic-first behavior

14. Preserve LazyBoy Action Safety Logic

15. Stale Element Handling

Retry rule

16. Phase 4 — Migrate Browser Control

Important

Test cases

17. Browser Profile Compatibility

18. Phase 5 — Recording Integration

Do not do

Do

19. Takeover Must Keep Working

Required rule

20. Credential / Vault Boundary

21. controld Migration Strategy

22. Docker Image Changes

Image versioning

23. Cua Process Lifecycle

Isolation

24. Health Checks

25. Fallback Behavior

26. Logging / Observability

27. Metrics

28. Testing Strategy

Unit tests

Contract tests

Integration tests

Browser E2E

29. Migration Acceptance Suite

Desktop

Browser

Lifecycle

Agent flow

30. Performance Benchmark

31. Rollout Plan

Step A

Step B

Step C

Step D

Step E

Step F

Step G

Step H

Step I

32. Suggested PR Sequence

PR 1

PR 2

38 KiB

Raw Blame History

12. Phase 2 — Migrate `computer_observe`

13. Phase 3 — Migrate `computer_act`

21. `controld` Migration Strategy