241 lines
7.8 KiB
Markdown
241 lines
7.8 KiB
Markdown
# Spec: 設定驅動的多主機正式部署
|
||
|
||
> Status: `draft-for-review`
|
||
> Source: `requirements.md`
|
||
> Last updated: `2026-07-18`
|
||
> Implementation: not started
|
||
|
||
## 1. 目標拓撲
|
||
|
||
```text
|
||
Internet
|
||
|
|
||
v
|
||
Edge Nginx
|
||
|-- / -> frontend-01, frontend-02, ...
|
||
|-- /api -> backend-01, backend-02, ...
|
||
`-- /haixun-assets -> MinIO
|
||
|
||
backend-* ----+---- MongoDB Atlas (fixed egress IP)
|
||
worker-* ----+---- Redis over WireGuard
|
||
`---- MinIO over WireGuard
|
||
|
||
worker-N <---- private Docker network ----> crawler-N
|
||
|
||
control machine -- SSH/Ansible --> all hosts
|
||
control machine -- push --------> private Registry
|
||
all Docker hosts -- pull -------> private Registry
|
||
```
|
||
|
||
## 2. 部署控制檔
|
||
|
||
### 2.1 `deployment.yml`
|
||
|
||
非機密設定的單一入口：
|
||
|
||
```yaml
|
||
environment: production
|
||
|
||
bootstrap:
|
||
operatingSystem: ubuntu-24.04
|
||
installDocker: true
|
||
installWireGuard: true
|
||
configureFirewall: true
|
||
configureTimeSync: true
|
||
installSecurityUpdates: true
|
||
enableAutomaticUpdates: true
|
||
acceptNewSSHHostKeys: true
|
||
|
||
ssh:
|
||
user: ubuntu
|
||
port: 22
|
||
privateKey: ~/.ssh/id_ed25519
|
||
|
||
nodes:
|
||
edge:
|
||
- ip: 203.0.113.10
|
||
registry:
|
||
- ip: 203.0.113.11
|
||
frontend:
|
||
- ip: 203.0.113.20
|
||
backend:
|
||
- ip: 203.0.113.30
|
||
worker:
|
||
- ip: 203.0.113.40
|
||
redis:
|
||
- ip: 203.0.113.50
|
||
minio:
|
||
- ip: 203.0.113.60
|
||
|
||
network:
|
||
wireguardCIDR: 10.80.0.0/16
|
||
atlasEgressNode: edge
|
||
|
||
domains:
|
||
web: app.example.com
|
||
registry: registry.example.com
|
||
letsEncryptEmail: admin@example.com
|
||
|
||
images:
|
||
registry: registry.example.com/haixun
|
||
tag: auto
|
||
|
||
database:
|
||
provider: mongodb-atlas
|
||
name: haixun
|
||
runMigrations: true
|
||
runInit: true
|
||
seedAdmin: true
|
||
|
||
redis:
|
||
namespace: haixun-prod
|
||
persistencePath: /var/lib/haixun/redis
|
||
appendOnly: true
|
||
|
||
minio:
|
||
persistencePath: /var/lib/haixun/minio
|
||
bucket: haixun-assets
|
||
initializeBucket: true
|
||
```
|
||
|
||
每個 node 可選擇覆寫 `sshUser`、`sshPort`、`sshPrivateKey`、`wireguardIP` 與 `architecture`。preflight 必須拒絕重複 IP、重複 WireGuard IP、空角色、Redis/MinIO 多於一台（第一版）與缺少必要角色。
|
||
|
||
### 2.2 `secrets.vault.yml`
|
||
|
||
至少包含：
|
||
|
||
```yaml
|
||
mongoAtlasURI: mongodb+srv://...
|
||
registryUser: ...
|
||
registryPassword: ...
|
||
redisPassword: ...
|
||
minioRootUser: ...
|
||
minioRootPassword: ...
|
||
authAccessSecret: ...
|
||
authRefreshSecret: ...
|
||
memberSettingsEncryptionKeyID: v1
|
||
memberSettingsEncryptionKey: ...
|
||
aiModelCacheFingerprintSecret: ...
|
||
scoutSessionSecret: ...
|
||
crawlerToken: ...
|
||
initialAdmin:
|
||
email: admin@example.com
|
||
displayName: System Admin
|
||
password: ...
|
||
```
|
||
|
||
Provider、SMTP、OAuth、Stripe 等既有 runtime secret 也放同一 Vault。Vault password 不進 repo；遠端 role 只取得執行該角色必要的 secret。
|
||
|
||
## 3. Image 與執行單位
|
||
|
||
| Image | 內容 | 持久資料 |
|
||
|------|------|----------|
|
||
| `frontend` | Vite build + static Nginx | 無 |
|
||
| `backend` | Go gateway + extension ZIP | 無 |
|
||
| `worker` | Go worker + Node scraper + Chromium | 無；僅 ephemeral temp |
|
||
| `crawler` | Node + Playwright crawler | 無 |
|
||
| `ops` | migrate/init/seeder binaries 與 migration files | 無 |
|
||
| `registry:2` | 私有 image registry | Registry volume |
|
||
| `redis:7` | cache/lock + AOF | Redis volume |
|
||
| `minio/minio` | object storage | MinIO volume |
|
||
|
||
應用 image 使用不可變 git SHA tag。`latest` 不可作為 production deploy 真相；inventory 另保存各角色目前與上一個成功 tag。
|
||
|
||
## 4. First deploy 狀態機
|
||
|
||
`make first-deploy` 僅是本機 orchestrator，按下列 checkpoint 執行：
|
||
|
||
1. **control doctor**：檢查 Ansible、Docker Buildx、SSH、Vault、Git worktree tag 能力。
|
||
2. **preflight**：解析設定、檢查 DNS、SSH/sudo、Ubuntu 版本、CPU architecture、磁碟/RAM、Atlas 固定出口前置條件。
|
||
3. **raw bootstrap**：在無 Python VM 先安裝 `python3`。
|
||
4. **common role**：套件、安全更新、chrony、fail2ban、目錄、Docker、log rotation。
|
||
5. **WireGuard**：配置 peer、防火牆與連通測試；所有私網測試成功才繼續。
|
||
6. **Registry first**：DNS 驗證、Let's Encrypt、認證、持久 volume、所有 Docker host login。
|
||
7. **local build/push**：測試並 build frontend/backend/worker/crawler/ops，push immutable tag。
|
||
8. **data services**：啟動 Redis/MinIO，檢查 persistence 與私網存取；建立 bucket/policy。
|
||
9. **Atlas prepare**：由單一受控 backend host 執行 ops container，依序 migration -> init -> optional first seed。
|
||
10. **applications**：backend rolling start、worker+crawler start、frontend rolling start。
|
||
11. **Edge**：產生 upstream、申請網站 TLS、驗證後 reload。
|
||
12. **acceptance**：HTTP、job claim、crawler health、Redis/MinIO persistence、公網封鎖與版本報告。
|
||
|
||
任一步驟失敗即停止後續 checkpoint；重跑從 idempotent state 繼續，不執行 wipe。
|
||
|
||
## 5. 一般部署狀態機
|
||
|
||
```text
|
||
test -> local build -> push -> remote pull while old runs
|
||
-> optional migration(run_once)
|
||
-> atomic config sync
|
||
-> rolling recreate
|
||
-> readiness
|
||
-> edge upstream update
|
||
-> record successful tag
|
||
```
|
||
|
||
- backend migration 失敗：不 recreate backend。
|
||
- backend/frontend readiness 失敗：不加入 upstream，保留其他健康節點。
|
||
- worker 失敗：回報 host，其他 worker 不回滾。
|
||
- config-only deploy：跳過 build/push，但仍做 render validation、atomic sync 與 recreate。
|
||
- rollback：切回上一個成功 image tag；資料庫只允許 forward-compatible migration，不自動執行 down migration。
|
||
|
||
## 6. 初始資料庫契約
|
||
|
||
- `ops migrate`：使用 Atlas URI 執行所有 forward migrations。
|
||
- `ops init`：執行 `cmd/init`，只建立可重複建立的 operational indexes。
|
||
- `ops seed-admin`：只在 `database.seedAdmin=true` 且目標管理員不存在時建立初始資料。
|
||
- seeder 需把目前寫死的 email/display name 改成設定值；password 至少 12 字元。
|
||
- 既有管理員存在時，seed 必須成功 no-op，不可更新 password hash。
|
||
- 管理員改密碼另設 `make admin-reset-password`，要求明確 host/environment 與確認。
|
||
|
||
## 7. 網路規則
|
||
|
||
| 來源 | 目的 | Port | 公開 |
|
||
|------|------|------|------|
|
||
| Internet | Edge | 80/443 | 是 |
|
||
| Control/Docker hosts | Registry | 443 | 認證 + TLS |
|
||
| Edge WG IP | Frontend | container HTTP | 否 |
|
||
| Edge WG IP | Backend | gateway port | 否 |
|
||
| Backend/Worker WG IP | Redis | 6379 | 否 |
|
||
| Backend/Worker/Edge WG IP | MinIO | 9000 | 否 |
|
||
| Worker container | crawler sidecar | crawler port | 否，不 publish |
|
||
| Backend/Worker | Atlas | 27017/SRV | 經固定出口 |
|
||
|
||
防火牆啟用前先 allow SSH；所有 role 完成後做反向驗證，確認 private port 從公網不可達。
|
||
|
||
## 8. 操作介面
|
||
|
||
```bash
|
||
make setup-control
|
||
make doctor
|
||
make preflight
|
||
make bootstrap
|
||
make bootstrap ROLE=worker
|
||
make bootstrap HOST=worker-02
|
||
make first-deploy
|
||
make deploy-all
|
||
make deploy-frontend
|
||
make deploy-backend
|
||
make deploy-worker
|
||
make deploy-config ROLE=worker
|
||
make db-check
|
||
make db-migrate
|
||
make db-init
|
||
make db-seed
|
||
make status
|
||
make logs ROLE=worker HOST=worker-01
|
||
make restart ROLE=backend
|
||
make rollback ROLE=backend
|
||
make vault-edit
|
||
```
|
||
|
||
具破壞性的資料 reset 不提供無參數捷徑，必須指定 environment、role 並互動確認。
|
||
|
||
## 9. Readiness 與擴縮
|
||
|
||
- gateway readiness 至少驗證程序、Redis、MongoDB；S3 狀態另外呈現，不能再只靠目前 Redis-only health。
|
||
- frontend readiness 驗證靜態首頁與指定 build version。
|
||
- crawler 增加 health endpoint；crawler token 不得記錄。
|
||
- worker 第一版以 container running + Mongo/Redis connectivity + worker identity 啟動檢查為準；後續補 durable heartbeat/lag。
|
||
- worker ID 由 hostname/role 派生，不得在多主機都使用 `worker-1`。
|
||
- Outbox 全域 lock 是吞吐限制，部署完成不代表 Outbox 已水平擴充；須另立應用改造 task。
|