thread-master/deploy/prod/README.md

# Harbor Desk Production Deployment

This deployment targets Ubuntu amd64 at `daniel@10.0.0.33` and serves
`threads-tool.30cm.net`.

## Layout

- Host Nginx serves immutable static web releases.
- Gateway uses blue/green systemd slots on `127.0.0.1:8888` and `:8889`.
- Worker uses matching systemd slots and guarded Mongo/Redis leases.
- Docker Compose runs MongoDB, Redis, MinIO, Prometheus, Grafana, Loki,
  Alertmanager, Alloy, node-exporter, cAdvisor, and blackbox-exporter.
- Runtime secrets stay in `/etc/harbor/harbor.env` with mode `0600`.
- Releases stay under `/opt/harbor/releases`; `/opt/harbor/current` changes
  atomically.

## First Install

The dedicated SSH key is expected at `~/.ssh/harbor_deploy`.

```bash
./deploy/prod/bootstrap-server.sh
./deploy/prod/release.sh
```

Bootstrap asks for remote sudo once. It then grants `daniel` passwordless sudo
only for the validated activate, rollback, and status scripts.

Edit external provider credentials after bootstrap:

```bash
ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33
sudoedit /etc/harbor/harbor.env
```

Restart the active services after changing runtime env values.

The first release creates `admin@haixun.local` once. Read its generated initial
password, sign in, then change it immediately:

```bash
ssh harbor-prod 'sudo cat /etc/harbor/initial-admin-credentials'
```

## Normal Release

```bash
./deploy/prod/release.sh
```

The command runs backend tests/vet, frontend tests, builds Linux amd64 binaries
and static assets, uploads a checksummed archive, migrates forward, starts the
inactive gateway, checks health, reloads Nginx, then hands work to the new
worker. The old gateway remains available for Nginx connection draining.

If an earlier inactive gateway still has established connections, activation
refuses to replace it. Retry after those requests finish.

To deploy an already built artifact:

```bash
./deploy/prod/release.sh deploy/prod/artifacts/harbor-*.tar.gz
```

## Rollback

```bash
./deploy/prod/rollback.sh
```

Rollback switches application code only. Never automatically run a down
migration; production migrations must remain backward compatible.

## Threads Profile Scrape (Playwright)

Persona analysis shells out to `node scripts/threads-profile/scrape.mjs` (a
headless Chromium scrape of a public Threads profile). `build-release.sh`
bundles the script with its production `node_modules`; the Chromium browser
binary itself is installed once on the host by `bootstrap.sh` (cached under
`/var/lib/harbor/.cache/ms-playwright`, shared across blue/green releases).

This host only has 1 vCPU. A scrape briefly saturates the CPU and can slow
down other requests while it runs; keep usage infrequent until the box is
upsized.

## Data Reset

To wipe Mongo, Redis, and MinIO (irreversible; monitoring volumes are untouched):

```bash
./deploy/prod/reset-data.sh --yes
```

This stops both app slots, recreates empty `mongo`/`redis`/`minio` containers using
the existing credentials in `harbor.env`, and clears the admin-seeded marker so the
next `release.sh` run creates a fresh `admin@haixun.local`. Take a backup first with
`backup.sh` if the data might still be worth keeping. Run `release.sh` immediately
after to bring the app back up with forward migrations and a reseeded admin.

## TLS

The public hostname is Cloudflare-proxied while the origin has a private IP.
Use a restricted Cloudflare DNS API token to obtain a DNS-01 certificate:

```bash
sudo install -m 0600 /dev/null /etc/letsencrypt/cloudflare.ini
sudoedit /etc/letsencrypt/cloudflare.ini
# dns_cloudflare_api_token = REPLACE_WITH_RESTRICTED_TOKEN
sudo certbot certonly --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  -d threads-tool.30cm.net
sudo /opt/harbor/deploy/remote/enable-tls.sh
```

Ensure Cloudflare Tunnel, router forwarding, or split DNS sends origin traffic
to `10.0.0.33`. A public Cloudflare DNS record alone cannot reach a private IP.

## Monitoring

All monitoring ports bind to localhost. Open Grafana through an SSH tunnel:

```bash
ssh -i ~/.ssh/harbor_deploy -L 3000:127.0.0.1:3000 daniel@10.0.0.33
```

Then browse `http://127.0.0.1:3000`. The generated Grafana password is in
`/etc/harbor/harbor.env` and should be read with sudo.

Useful commands:

```bash
ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  sudo /opt/harbor/deploy/remote/status.sh
ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  journalctl -u 'harbor-gateway@*' -f
```

### Background job health

`harbor-job-health.timer` runs every 5 minutes and writes a node-exporter
textfile-collector metric (`haixun_job_failures_recent`) counting jobs that
failed in the last 15 minutes, overall and specifically for
`persona_analyze_account` (the Threads profile scrape). Prometheus rules
`PersonaScrapeJobsFailing` / `BackgroundJobsFailing` /
`JobHealthMetricStale` in `monitoring/prometheus/rules/alerts.yml` alert on
this so a stuck worker (e.g. a missing Playwright browser) surfaces on its
own instead of only being noticed when a member reports it.

## Backups

`harbor-backup.timer` creates daily Mongo and MinIO backups under
`/var/backups/harbor` and retains 14 days. This protects against accidental
deletion but not loss of the machine or disk.

`harbor-offsite-backup.timer` runs `backup/offsite-sync.sh` an hour later and
pushes those local backups to an off-host [restic](https://restic.net/)
repository (client-side encrypted before upload). It is installed and enabled
by default but is a no-op until you set `RESTIC_REPOSITORY` and
`RESTIC_PASSWORD` (plus provider credentials, e.g. `AWS_ACCESS_KEY_ID` /
`AWS_SECRET_ACCESS_KEY` for an S3-compatible bucket) in
`/etc/harbor/harbor.env` — see `harbor.env.example` for the format. Once
configured, verify with:

```bash
ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  sudo systemctl start harbor-offsite-backup.service
ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  sudo journalctl -u harbor-offsite-backup.service -n 50
```