thread-master/apps/backend/crawler

Scout Chrome Crawler

Private Playwright service for Scout dev_mode=true. It is not a public API. It binds to 127.0.0.1 only, requires a bearer token, and receives decrypted Chrome storage state only from the backend worker/gateway process.

Install

cd apps/backend/crawler
npm install
npx playwright install --with-deps chromium

Configure

Set the same private token in the local-only apps/backend/etc/gateway.yaml:

Scout:
  SessionSecret: "a dedicated long random secret"
  CrawlerEndpoint: "http://127.0.0.1:8891"
  CrawlerToken: "a different long random token"

Start the crawler with the matching token:

cd apps/backend/crawler
SCOUT_CRAWLER_TOKEN='the CrawlerToken value' npm run start

Optional: SCOUT_CRAWLER_PORT=8891 changes the local port.

Runtime Rules

• Never expose this port through nginx, Docker host publishing, or the public internet.
• Never place CrawlerToken, SessionSecret, or storage state in a tracked file or log.
• dev_mode=false does not use this service; it uses the configured API provider.
• dev_mode=true requires a freshly synchronized Chrome session. Missing or expired sessions fail with a user-actionable error.
• The service supports /v1/threads/search and /v1/threads/resolve; both require Authorization: Bearer <CrawlerToken>.

Operations

1. Start Mongo, Redis, gateway, worker, and this crawler service.
2. In Harbor Desk settings, enable dev mode and synchronize Chrome Threads session.
3. Start a Scout run. The worker chooses crawler mode and contacts only CrawlerEndpoint.
4. If Threads redirects to login, sync the session again. No fallback to API occurs in dev mode.