241 lines
7.8 KiB
Markdown
241 lines
7.8 KiB
Markdown
|
# Spec: 設定驅動的多主機正式部署
|
|||
|
|
|
|||
|
|
> Status: `draft-for-review`
|
|||
|
|
> Source: `requirements.md`
|
|||
|
|
> Last updated: `2026-07-18`
|
|||
|
|
> Implementation: not started
|
|||
|
|
|
|||
|
|
## 1. 目標拓撲
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
Internet
|
|||
|
|
|
|
|||
|
|
v
|
|||
|
|
Edge Nginx
|
|||
|
|
|-- / -> frontend-01, frontend-02, ...
|
|||
|
|
|-- /api -> backend-01, backend-02, ...
|
|||
|
|
`-- /haixun-assets -> MinIO
|
|||
|
|
|
|||
|
|
backend-* ----+---- MongoDB Atlas (fixed egress IP)
|
|||
|
|
worker-* ----+---- Redis over WireGuard
|
|||
|
|
`---- MinIO over WireGuard
|
|||
|
|
|
|||
|
|
worker-N <---- private Docker network ----> crawler-N
|
|||
|
|
|
|||
|
|
control machine -- SSH/Ansible --> all hosts
|
|||
|
|
control machine -- push --------> private Registry
|
|||
|
|
all Docker hosts -- pull -------> private Registry
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## 2. 部署控制檔
|
|||
|
|
|
|||
|
|
### 2.1 `deployment.yml`
|
|||
|
|
|
|||
|
|
非機密設定的單一入口：
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
environment: production
|
|||
|
|
|
|||
|
|
bootstrap:
|
|||
|
|
operatingSystem: ubuntu-24.04
|
|||
|
|
installDocker: true
|
|||
|
|
installWireGuard: true
|
|||
|
|
configureFirewall: true
|
|||
|
|
configureTimeSync: true
|
|||
|
|
installSecurityUpdates: true
|
|||
|
|
enableAutomaticUpdates: true
|
|||
|
|
acceptNewSSHHostKeys: true
|
|||
|
|
|
|||
|
|
ssh:
|
|||
|
|
user: ubuntu
|
|||
|
|
port: 22
|
|||
|
|
privateKey: ~/.ssh/id_ed25519
|
|||
|
|
|
|||
|
|
nodes:
|
|||
|
|
edge:
|
|||
|
|
- ip: 203.0.113.10
|
|||
|
|
registry:
|
|||
|
|
- ip: 203.0.113.11
|
|||
|
|
frontend:
|
|||
|
|
- ip: 203.0.113.20
|
|||
|
|
backend:
|
|||
|
|
- ip: 203.0.113.30
|
|||
|
|
worker:
|
|||
|
|
- ip: 203.0.113.40
|
|||
|
|
redis:
|
|||
|
|
- ip: 203.0.113.50
|
|||
|
|
minio:
|
|||
|
|
- ip: 203.0.113.60
|
|||
|
|
|
|||
|
|
network:
|
|||
|
|
wireguardCIDR: 10.80.0.0/16
|
|||
|
|
atlasEgressNode: edge
|
|||
|
|
|
|||
|
|
domains:
|
|||
|
|
web: app.example.com
|
|||
|
|
registry: registry.example.com
|
|||
|
|
letsEncryptEmail: admin@example.com
|
|||
|
|
|
|||
|
|
images:
|
|||
|
|
registry: registry.example.com/haixun
|
|||
|
|
tag: auto
|
|||
|
|
|
|||
|
|
database:
|
|||
|
|
provider: mongodb-atlas
|
|||
|
|
name: haixun
|
|||
|
|
runMigrations: true
|
|||
|
|
runInit: true
|
|||
|
|
seedAdmin: true
|
|||
|
|
|
|||
|
|
redis:
|
|||
|
|
namespace: haixun-prod
|
|||
|
|
persistencePath: /var/lib/haixun/redis
|
|||
|
|
appendOnly: true
|
|||
|
|
|
|||
|
|
minio:
|
|||
|
|
persistencePath: /var/lib/haixun/minio
|
|||
|
|
bucket: haixun-assets
|
|||
|
|
initializeBucket: true
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
每個 node 可選擇覆寫 `sshUser`、`sshPort`、`sshPrivateKey`、`wireguardIP` 與 `architecture`。preflight 必須拒絕重複 IP、重複 WireGuard IP、空角色、Redis/MinIO 多於一台（第一版）與缺少必要角色。
|
|||
|
|
|
|||
|
|
### 2.2 `secrets.vault.yml`
|
|||
|
|
|
|||
|
|
至少包含：
|
|||
|
|
|
|||
|
|
```yaml
|
|||
|
|
mongoAtlasURI: mongodb+srv://...
|
|||
|
|
registryUser: ...
|
|||
|
|
registryPassword: ...
|
|||
|
|
redisPassword: ...
|
|||
|
|
minioRootUser: ...
|
|||
|
|
minioRootPassword: ...
|
|||
|
|
authAccessSecret: ...
|
|||
|
|
authRefreshSecret: ...
|
|||
|
|
memberSettingsEncryptionKeyID: v1
|
|||
|
|
memberSettingsEncryptionKey: ...
|
|||
|
|
aiModelCacheFingerprintSecret: ...
|
|||
|
|
scoutSessionSecret: ...
|
|||
|
|
crawlerToken: ...
|
|||
|
|
initialAdmin:
|
|||
|
|
email: admin@example.com
|
|||
|
|
displayName: System Admin
|
|||
|
|
password: ...
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Provider、SMTP、OAuth、Stripe 等既有 runtime secret 也放同一 Vault。Vault password 不進 repo；遠端 role 只取得執行該角色必要的 secret。
|
|||
|
|
|
|||
|
|
## 3. Image 與執行單位
|
|||
|
|
|
|||
|
|
| Image | 內容 | 持久資料 |
|
|||
|
|
|------|------|----------|
|
|||
|
|
| `frontend` | Vite build + static Nginx | 無 |
|
|||
|
|
| `backend` | Go gateway + extension ZIP | 無 |
|
|||
|
|
| `worker` | Go worker + Node scraper + Chromium | 無；僅 ephemeral temp |
|
|||
|
|
| `crawler` | Node + Playwright crawler | 無 |
|
|||
|
|
| `ops` | migrate/init/seeder binaries 與 migration files | 無 |
|
|||
|
|
| `registry:2` | 私有 image registry | Registry volume |
|
|||
|
|
| `redis:7` | cache/lock + AOF | Redis volume |
|
|||
|
|
| `minio/minio` | object storage | MinIO volume |
|
|||
|
|
|
|||
|
|
應用 image 使用不可變 git SHA tag。`latest` 不可作為 production deploy 真相；inventory 另保存各角色目前與上一個成功 tag。
|
|||
|
|
|
|||
|
|
## 4. First deploy 狀態機
|
|||
|
|
|
|||
|
|
`make first-deploy` 僅是本機 orchestrator，按下列 checkpoint 執行：
|
|||
|
|
|
|||
|
|
1. **control doctor**：檢查 Ansible、Docker Buildx、SSH、Vault、Git worktree tag 能力。
|
|||
|
|
2. **preflight**：解析設定、檢查 DNS、SSH/sudo、Ubuntu 版本、CPU architecture、磁碟/RAM、Atlas 固定出口前置條件。
|
|||
|
|
3. **raw bootstrap**：在無 Python VM 先安裝 `python3`。
|
|||
|
|
4. **common role**：套件、安全更新、chrony、fail2ban、目錄、Docker、log rotation。
|
|||
|
|
5. **WireGuard**：配置 peer、防火牆與連通測試；所有私網測試成功才繼續。
|
|||
|
|
6. **Registry first**：DNS 驗證、Let's Encrypt、認證、持久 volume、所有 Docker host login。
|
|||
|
|
7. **local build/push**：測試並 build frontend/backend/worker/crawler/ops，push immutable tag。
|
|||
|
|
8. **data services**：啟動 Redis/MinIO，檢查 persistence 與私網存取；建立 bucket/policy。
|
|||
|
|
9. **Atlas prepare**：由單一受控 backend host 執行 ops container，依序 migration -> init -> optional first seed。
|
|||
|
|
10. **applications**：backend rolling start、worker+crawler start、frontend rolling start。
|
|||
|
|
11. **Edge**：產生 upstream、申請網站 TLS、驗證後 reload。
|
|||
|
|
12. **acceptance**：HTTP、job claim、crawler health、Redis/MinIO persistence、公網封鎖與版本報告。
|
|||
|
|
|
|||
|
|
任一步驟失敗即停止後續 checkpoint；重跑從 idempotent state 繼續，不執行 wipe。
|
|||
|
|
|
|||
|
|
## 5. 一般部署狀態機
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
test -> local build -> push -> remote pull while old runs
|
|||
|
|
-> optional migration(run_once)
|
|||
|
|
-> atomic config sync
|
|||
|
|
-> rolling recreate
|
|||
|
|
-> readiness
|
|||
|
|
-> edge upstream update
|
|||
|
|
-> record successful tag
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
- backend migration 失敗：不 recreate backend。
|
|||
|
|
- backend/frontend readiness 失敗：不加入 upstream，保留其他健康節點。
|
|||
|
|
- worker 失敗：回報 host，其他 worker 不回滾。
|
|||
|
|
- config-only deploy：跳過 build/push，但仍做 render validation、atomic sync 與 recreate。
|
|||
|
|
- rollback：切回上一個成功 image tag；資料庫只允許 forward-compatible migration，不自動執行 down migration。
|
|||
|
|
|
|||
|
|
## 6. 初始資料庫契約
|
|||
|
|
|
|||
|
|
- `ops migrate`：使用 Atlas URI 執行所有 forward migrations。
|
|||
|
|
- `ops init`：執行 `cmd/init`，只建立可重複建立的 operational indexes。
|
|||
|
|
- `ops seed-admin`：只在 `database.seedAdmin=true` 且目標管理員不存在時建立初始資料。
|
|||
|
|
- seeder 需把目前寫死的 email/display name 改成設定值；password 至少 12 字元。
|
|||
|
|
- 既有管理員存在時，seed 必須成功 no-op，不可更新 password hash。
|
|||
|
|
- 管理員改密碼另設 `make admin-reset-password`，要求明確 host/environment 與確認。
|
|||
|
|
|
|||
|
|
## 7. 網路規則
|
|||
|
|
|
|||
|
|
| 來源 | 目的 | Port | 公開 |
|
|||
|
|
|------|------|------|------|
|
|||
|
|
| Internet | Edge | 80/443 | 是 |
|
|||
|
|
| Control/Docker hosts | Registry | 443 | 認證 + TLS |
|
|||
|
|
| Edge WG IP | Frontend | container HTTP | 否 |
|
|||
|
|
| Edge WG IP | Backend | gateway port | 否 |
|
|||
|
|
| Backend/Worker WG IP | Redis | 6379 | 否 |
|
|||
|
|
| Backend/Worker/Edge WG IP | MinIO | 9000 | 否 |
|
|||
|
|
| Worker container | crawler sidecar | crawler port | 否，不 publish |
|
|||
|
|
| Backend/Worker | Atlas | 27017/SRV | 經固定出口 |
|
|||
|
|
|
|||
|
|
防火牆啟用前先 allow SSH；所有 role 完成後做反向驗證，確認 private port 從公網不可達。
|
|||
|
|
|
|||
|
|
## 8. 操作介面
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
make setup-control
|
|||
|
|
make doctor
|
|||
|
|
make preflight
|
|||
|
|
make bootstrap
|
|||
|
|
make bootstrap ROLE=worker
|
|||
|
|
make bootstrap HOST=worker-02
|
|||
|
|
make first-deploy
|
|||
|
|
make deploy-all
|
|||
|
|
make deploy-frontend
|
|||
|
|
make deploy-backend
|
|||
|
|
make deploy-worker
|
|||
|
|
make deploy-config ROLE=worker
|
|||
|
|
make db-check
|
|||
|
|
make db-migrate
|
|||
|
|
make db-init
|
|||
|
|
make db-seed
|
|||
|
|
make status
|
|||
|
|
make logs ROLE=worker HOST=worker-01
|
|||
|
|
make restart ROLE=backend
|
|||
|
|
make rollback ROLE=backend
|
|||
|
|
make vault-edit
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
具破壞性的資料 reset 不提供無參數捷徑，必須指定 environment、role 並互動確認。
|
|||
|
|
|
|||
|
|
## 9. Readiness 與擴縮
|
|||
|
|
|
|||
|
|
- gateway readiness 至少驗證程序、Redis、MongoDB；S3 狀態另外呈現，不能再只靠目前 Redis-only health。
|
|||
|
|
- frontend readiness 驗證靜態首頁與指定 build version。
|
|||
|
|
- crawler 增加 health endpoint；crawler token 不得記錄。
|
|||
|
|
- worker 第一版以 container running + Mongo/Redis connectivity + worker identity 啟動檢查為準；後續補 durable heartbeat/lag。
|
|||
|
|
- worker ID 由 hostname/role 派生，不得在多主機都使用 `worker-1`。
|
|||
|
|
- Outbox 全域 lock 是吞吐限制，部署完成不代表 Outbox 已水平擴充；須另立應用改造 task。
|