1.7 KiB

Raw Blame History

Scout Chrome Crawler

Private Playwright service for Scout dev_mode=true. It is not a public API. It binds to 127.0.0.1 only, requires a bearer token, and receives decrypted Chrome storage state only from the backend worker/gateway process.

Install

cd apps/backend/crawler
npm install
npx playwright install --with-deps chromium

Configure

Set the same private token in the local-only apps/backend/etc/gateway.yaml:

Scout:
  SessionSecret: "a dedicated long random secret"
  CrawlerEndpoint: "http://127.0.0.1:8891"
  CrawlerToken: "a different long random token"

Start the crawler with the matching token:

cd apps/backend/crawler
SCOUT_CRAWLER_TOKEN='the CrawlerToken value' npm run start

Optional: SCOUT_CRAWLER_PORT=8891 changes the local port.

Runtime Rules

Never expose this port through nginx, Docker host publishing, or the public internet.
Never place CrawlerToken, SessionSecret, or storage state in a tracked file or log.
dev_mode=false does not use this service; it uses the configured API provider.
dev_mode=true requires a freshly synchronized Chrome session. Missing or expired sessions fail with a user-actionable error.
The service supports /v1/threads/search and /v1/threads/resolve; both require Authorization: Bearer <CrawlerToken>.

Operations

Start Mongo, Redis, gateway, worker, and this crawler service.
In Harbor Desk settings, enable dev mode and synchronize Chrome Threads session.
Start a Scout run. The worker chooses crawler mode and contacts only CrawlerEndpoint.
If Threads redirects to login, sync the session again. No fallback to API occurs in dev mode.

1.7 KiB Raw Blame History