49 lines
1.7 KiB
Markdown
49 lines
1.7 KiB
Markdown
|
# Scout Chrome Crawler
|
||
|
|
|
||
|
|
Private Playwright service for Scout `dev_mode=true`. It is not a public API.
|
||
|
|
It binds to `127.0.0.1` only, requires a bearer token, and receives decrypted
|
||
|
|
Chrome storage state only from the backend worker/gateway process.
|
||
|
|
|
||
|
|
## Install
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd apps/backend/crawler
|
||
|
|
npm install
|
||
|
|
npx playwright install --with-deps chromium
|
||
|
|
```
|
||
|
|
|
||
|
|
## Configure
|
||
|
|
|
||
|
|
Set the same private token in the local-only `apps/backend/etc/gateway.yaml`:
|
||
|
|
|
||
|
|
```yaml
|
||
|
|
Scout:
|
||
|
|
SessionSecret: "a dedicated long random secret"
|
||
|
|
CrawlerEndpoint: "http://127.0.0.1:8891"
|
||
|
|
CrawlerToken: "a different long random token"
|
||
|
|
```
|
||
|
|
|
||
|
|
Start the crawler with the matching token:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
cd apps/backend/crawler
|
||
|
|
SCOUT_CRAWLER_TOKEN='the CrawlerToken value' npm run start
|
||
|
|
```
|
||
|
|
|
||
|
|
Optional: `SCOUT_CRAWLER_PORT=8891` changes the local port.
|
||
|
|
|
||
|
|
## Runtime Rules
|
||
|
|
|
||
|
|
- Never expose this port through nginx, Docker host publishing, or the public internet.
|
||
|
|
- Never place `CrawlerToken`, `SessionSecret`, or storage state in a tracked file or log.
|
||
|
|
- `dev_mode=false` does not use this service; it uses the configured API provider.
|
||
|
|
- `dev_mode=true` requires a freshly synchronized Chrome session. Missing or expired sessions fail with a user-actionable error.
|
||
|
|
- The service supports `/v1/threads/search` and `/v1/threads/resolve`; both require `Authorization: Bearer <CrawlerToken>`.
|
||
|
|
|
||
|
|
## Operations
|
||
|
|
|
||
|
|
1. Start Mongo, Redis, gateway, worker, and this crawler service.
|
||
|
|
2. In Harbor Desk settings, enable dev mode and synchronize Chrome Threads session.
|
||
|
|
3. Start a Scout run. The worker chooses crawler mode and contacts only `CrawlerEndpoint`.
|
||
|
|
4. If Threads redirects to login, sync the session again. No fallback to API occurs in dev mode.
|