thread-master/deploy/prod

Harbor Desk Production Deployment

This deployment targets Ubuntu amd64 at daniel@10.0.0.33 and serves threads-tool.30cm.net.

Layout

• Host Nginx serves immutable static web releases.
• Gateway uses blue/green systemd slots on 127.0.0.1:8888 and :8889.
• Worker uses matching systemd slots and guarded Mongo/Redis leases.
• Docker Compose runs MongoDB, Redis, MinIO, Prometheus, Grafana, Loki, Alertmanager, Alloy, node-exporter, cAdvisor, and blackbox-exporter.
• Runtime secrets stay in /etc/harbor/harbor.env with mode 0600.
• Releases stay under /opt/harbor/releases; /opt/harbor/current changes atomically.

First Install

The dedicated SSH key is expected at ~/.ssh/harbor_deploy.

./deploy/prod/bootstrap-server.sh
./deploy/prod/release.sh

Bootstrap asks for remote sudo once. It then grants daniel passwordless sudo only for the validated activate, rollback, and status scripts.

Edit external provider credentials after bootstrap:

ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33
sudoedit /etc/harbor/harbor.env

Restart the active services after changing runtime env values.

The first release creates admin@haixun.local once. Read its generated initial password, sign in, then change it immediately:

ssh harbor-prod 'sudo cat /etc/harbor/initial-admin-credentials'

Normal Release

./deploy/prod/release.sh

The command runs backend tests/vet, frontend tests, builds Linux amd64 binaries and static assets, uploads a checksummed archive, migrates forward, starts the inactive gateway, checks health, reloads Nginx, then hands work to the new worker. The old gateway remains available for Nginx connection draining.

If an earlier inactive gateway still has established connections, activation refuses to replace it. Retry after those requests finish.

To deploy an already built artifact:

./deploy/prod/release.sh deploy/prod/artifacts/harbor-*.tar.gz

Rollback

./deploy/prod/rollback.sh

Rollback switches application code only. Never automatically run a down migration; production migrations must remain backward compatible.

Threads Profile Scrape (Playwright)

Persona analysis shells out to node scripts/threads-profile/scrape.mjs (a headless Chromium scrape of a public Threads profile). build-release.sh bundles the script with its production node_modules; the Chromium browser binary itself is installed once on the host by bootstrap.sh (cached under /var/lib/harbor/.cache/ms-playwright, shared across blue/green releases).

This host only has 1 vCPU. A scrape briefly saturates the CPU and can slow down other requests while it runs; keep usage infrequent until the box is upsized.

Data Reset

To wipe Mongo, Redis, and MinIO (irreversible; monitoring volumes are untouched):

./deploy/prod/reset-data.sh --yes

This stops both app slots, recreates empty mongo/redis/minio containers using the existing credentials in harbor.env, and clears the admin-seeded marker so the next release.sh run creates a fresh admin@haixun.local. Take a backup first with backup.sh if the data might still be worth keeping. Run release.sh immediately after to bring the app back up with forward migrations and a reseeded admin.

TLS

The public hostname is Cloudflare-proxied while the origin has a private IP. Use a restricted Cloudflare DNS API token to obtain a DNS-01 certificate:

sudo install -m 0600 /dev/null /etc/letsencrypt/cloudflare.ini
sudoedit /etc/letsencrypt/cloudflare.ini
# dns_cloudflare_api_token = REPLACE_WITH_RESTRICTED_TOKEN
sudo certbot certonly --dns-cloudflare \
  --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini \
  -d threads-tool.30cm.net
sudo /opt/harbor/deploy/remote/enable-tls.sh

Ensure Cloudflare Tunnel, router forwarding, or split DNS sends origin traffic to 10.0.0.33. A public Cloudflare DNS record alone cannot reach a private IP.

Monitoring

All monitoring ports bind to localhost. Open Grafana through an SSH tunnel:

ssh -i ~/.ssh/harbor_deploy -L 3000:127.0.0.1:3000 daniel@10.0.0.33

Then browse http://127.0.0.1:3000. The generated Grafana password is in /etc/harbor/harbor.env and should be read with sudo.

Useful commands:

ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  sudo /opt/harbor/deploy/remote/status.sh
ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  journalctl -u 'harbor-gateway@*' -f

Background job health

harbor-job-health.timer runs every 5 minutes and writes a node-exporter textfile-collector metric (haixun_job_failures_recent) counting jobs that failed in the last 15 minutes, overall and specifically for persona_analyze_account (the Threads profile scrape). Prometheus rules PersonaScrapeJobsFailing / BackgroundJobsFailing / JobHealthMetricStale in monitoring/prometheus/rules/alerts.yml alert on this so a stuck worker (e.g. a missing Playwright browser) surfaces on its own instead of only being noticed when a member reports it.

Backups

harbor-backup.timer creates daily Mongo and MinIO backups under /var/backups/harbor and retains 14 days. This protects against accidental deletion but not loss of the machine or disk.

harbor-offsite-backup.timer runs backup/offsite-sync.sh an hour later and pushes those local backups to an off-host restic repository (client-side encrypted before upload). It is installed and enabled by default but is a no-op until you set RESTIC_REPOSITORY and RESTIC_PASSWORD (plus provider credentials, e.g. AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY for an S3-compatible bucket) in /etc/harbor/harbor.env — see harbor.env.example for the format. Once configured, verify with:

ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  sudo systemctl start harbor-offsite-backup.service
ssh -i ~/.ssh/harbor_deploy daniel@10.0.0.33 \
  sudo journalctl -u harbor-offsite-backup.service -n 50