265 lines
8.8 KiB
Markdown
265 lines
8.8 KiB
Markdown
# Nomad on the services VPS — install & operations runbook
|
|
|
|
Single-node Nomad agent (server+client) running the app services
|
|
(`api` + 5 workers) while stateful infra (postgres, redis, rabbitmq, minio),
|
|
the nginx edge cascade, and observability stay on docker compose.
|
|
|
|
Layout:
|
|
|
|
```
|
|
deploy/nomad/
|
|
├── nomad.hcl # agent config → /etc/nomad.d/nomad.hcl
|
|
├── policies/ci.hcl # ACL policy for the Forgejo deploy token
|
|
├── contract-check.nomad.hcl # job: api + worker groups
|
|
└── README.md # this runbook
|
|
```
|
|
|
|
Placeholders used below — substitute before running:
|
|
|
|
| Placeholder | Meaning |
|
|
|---|---|
|
|
| `<VPS_PUBLIC_IP>` | public IP of the services VPS |
|
|
| `<FORGEJO_SERVER_IP>` | public IP of the Forgejo VPS (runner) |
|
|
| `<FORGEJO_HOST>` | Forgejo hostname, e.g. `git.example.com` |
|
|
| `<OWNER>` | Forgejo user/org owning the repo + images |
|
|
|
|
---
|
|
|
|
## 1. Install Nomad
|
|
|
|
### 1a. Manual install via the official Docker image
|
|
|
|
The services VPS network 404s `apt.releases.hashicorp.com`,
|
|
`releases.hashicorp.com` and GitHub release assets, and Ubuntu `resolute`
|
|
is newer than the apt repo's suites — so extract the binary from the
|
|
official Docker image instead (Docker Hub is reachable there):
|
|
|
|
```bash
|
|
docker pull hashicorp/nomad:1.9
|
|
docker create --name nomad-extract hashicorp/nomad:1.9
|
|
docker cp nomad-extract:/bin/nomad /tmp/nomad
|
|
docker rm nomad-extract
|
|
chmod +x /tmp/nomad && /tmp/nomad version
|
|
sudo install -m 0755 /tmp/nomad /usr/local/bin/nomad
|
|
# if the cp path is wrong:
|
|
# docker run --rm --entrypoint sh hashicorp/nomad:1.9 -c 'command -v nomad'
|
|
```
|
|
|
|
**Why 1.9.x and not 2.x:** Nomad 2.x extracts the docker driver into an
|
|
external plugin distributed via the blocked hosts above. 1.9.x has it
|
|
built in — everything this stack uses (nomadVar templates, native
|
|
services, static ports, canary updates) is fully supported there.
|
|
|
|
No `.deb` ⇒ no systemd unit ships; install ours:
|
|
|
|
```bash
|
|
sudo cp deploy/nomad/nomad.service /etc/systemd/system/nomad.service
|
|
```
|
|
|
|
### 1b. apt install (only on networks that reach HashiCorp's repo)
|
|
|
|
```bash
|
|
wget -O- https://apt.releases.hashicorp.com/gpg | \
|
|
sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
|
|
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
|
|
https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
|
|
sudo tee /etc/apt/sources.list.d/hashicorp.list
|
|
sudo apt-get update && sudo apt-get install -y nomad
|
|
nomad version # expect 1.7+
|
|
```
|
|
|
|
Note: the `.deb`'s own unit lives in /lib/systemd/system — do NOT install
|
|
`deploy/nomad/nomad.service` in that case.
|
|
|
|
## 2. Install config + data dir
|
|
|
|
```bash
|
|
sudo mkdir -p /var/lib/nomad /etc/nomad.d/tls
|
|
# from the repo root on the VPS:
|
|
sudo cp deploy/nomad/nomad.hcl /etc/nomad.d/nomad.hcl
|
|
```
|
|
|
|
## 3. Generate TLS material
|
|
|
|
The API is reachable from the internet (Forgejo runner → VPS), so TLS is
|
|
mandatory. Use the script (CA + server cert with all required SANs in one
|
|
shot — hand-rolled openssl commands tend to miss `server.global.vps.nomad`,
|
|
which `verify_server_hostname` requires):
|
|
|
|
```bash
|
|
sudo deploy/nomad/gen-tls.sh <VPS_PUBLIC_IP>
|
|
sudo systemctl restart nomad
|
|
```
|
|
|
|
The runner gets **only** the CA cert (`nomad-ca.crt`) as `NOMAD_CACERT`.
|
|
|
|
Manual equivalent (what the script does):
|
|
```bash
|
|
cd /tmp && mkdir nomad-tls && cd nomad-tls
|
|
|
|
# CA (keep nomad-ca.key offline afterwards; only the .crt is ever copied)
|
|
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
|
|
-keyout nomad-ca.key -out nomad-ca.crt -subj "/CN=Contract-Check Nomad CA"
|
|
|
|
# server cert — SANs must cover the RPC hostname AND how clients reach it
|
|
openssl req -newkey rsa:2048 -nodes \
|
|
-keyout server.key -out server.csr \
|
|
-subj "/CN=server.global.vps.nomad"
|
|
|
|
cat > server.ext <<'EOF'
|
|
subjectAltName = DNS:server.global.vps.nomad, DNS:localhost, IP:127.0.0.1, IP:<VPS_PUBLIC_IP>
|
|
extendedKeyUsage = serverAuth, clientAuth
|
|
EOF
|
|
|
|
openssl x509 -req -in server.csr \
|
|
-CA nomad-ca.crt -CAkey nomad-ca.key -CAcreateserial \
|
|
-out server.crt -days 825 -extfile server.ext
|
|
|
|
sudo install -m 644 nomad-ca.crt server.crt /etc/nomad.d/tls/
|
|
sudo install -m 600 server.key /etc/nomad.d/tls/
|
|
```
|
|
|
|
Copy `nomad-ca.crt` to the Forgejo VPS (or a repo secret / secure storage)
|
|
— the deploy workflow needs it as `NOMAD_CACERT`.
|
|
|
|
## 4. Start the agent + bootstrap ACLs
|
|
|
|
```bash
|
|
sudo systemctl enable --now nomad
|
|
systemctl status nomad --no-pager
|
|
```
|
|
|
|
For every CLI call on the VPS, export:
|
|
|
|
```bash
|
|
export NOMAD_ADDR=https://127.0.0.1:4646
|
|
export NOMAD_CACERT=/etc/nomad.d/tls/nomad-ca.crt
|
|
```
|
|
|
|
Bootstrap ACLs once:
|
|
|
|
```bash
|
|
nomad acl bootstrap
|
|
# → prints an AccessorID/SecretID management token. Store the SecretID in a
|
|
# password manager; export it for the following commands:
|
|
export NOMAD_TOKEN=<management-secret-id>
|
|
```
|
|
|
|
Sanity checks:
|
|
|
|
```bash
|
|
nomad server members # self as leader
|
|
nomad node status # client ready, docker driver detected
|
|
```
|
|
|
|
## 5. CI policy + token + registry variables
|
|
|
|
```bash
|
|
# policy for the deploy token
|
|
nomad acl policy apply -description "Forgejo CI deploy" ci \
|
|
deploy/nomad/policies/ci.hcl
|
|
|
|
# long-lived token for the runner (no TTL)
|
|
nomad acl token create -name forgejo-ci -policy ci
|
|
# → AccessorID + SecretID. Forgejo repo secret: NOMAD_TOKEN = SecretID
|
|
```
|
|
|
|
Registry credentials and app secrets — the job file templates ALL of these
|
|
from `nomad/jobs/contract-check` (missing keys render empty; define every
|
|
key once). Connection strings point at the docker0 host gateway
|
|
`172.17.0.1` and the host-published infra ports (15432/17379/5672/9000) —
|
|
substitute real credentials where the compose `.env` deviates from
|
|
defaults:
|
|
|
|
```bash
|
|
nomad var put -in=json nomad/jobs/contract-check - <<'EOF'
|
|
{
|
|
"registry_host": "<FORGEJO_HOST>",
|
|
"registry_owner": "<OWNER>",
|
|
"registry_user": "<OWNER>",
|
|
"registry_token": "<forgejo token with write:package scope>",
|
|
|
|
"database_url": "postgresql+asyncpg://contract_check:<POSTGRES_PASSWORD>@172.17.0.1:15432/contract_check",
|
|
"redis_url": "redis://172.17.0.1:17379/0",
|
|
"rabbitmq_url": "amqp://contract_check:<RABBITMQ_PASS>@172.17.0.1:5672/",
|
|
"s3_endpoint_url": "http://172.17.0.1:9000",
|
|
"s3_access_key": "<S3_ACCESS_KEY>",
|
|
"s3_secret_key": "<S3_SECRET_KEY>",
|
|
"s3_bucket": "contract-check-docs",
|
|
|
|
"jwt_secret": "<openssl rand -hex 32 — reuse the value from compose .env>",
|
|
"telegram_bot_token": "<TELEGRAM_BOT_TOKEN>",
|
|
"ollama_api_key": "<OLLAMA_API_KEY>",
|
|
"yandexgpt_api_key": "<YANDEXGPT_API_KEY or empty>",
|
|
"smtp_host": "<SMTP_HOST or empty>",
|
|
"smtp_username": "<SMTP_USERNAME or empty>",
|
|
"smtp_password": "<SMTP_PASSWORD or empty>",
|
|
"metrics_bearer_token": "<METRICS_BEARER_TOKEN or empty>"
|
|
}
|
|
EOF
|
|
|
|
nomad var get nomad/jobs/contract-check # sanity check
|
|
```
|
|
|
|
## 6. Firewall
|
|
|
|
```bash
|
|
# 4646: only the Forgejo runner may talk to the API
|
|
sudo ufw allow from <FORGEJO_SERVER_IP> to any port 4646 proto tcp \
|
|
comment 'nomad http (forgejo runner)'
|
|
# single-node: no external rpc/gossip peers
|
|
sudo ufw deny 4647/tcp comment 'nomad rpc (local only)'
|
|
sudo ufw deny 4648/tcp comment 'nomad gossip (local only)'
|
|
sudo ufw status verbose
|
|
```
|
|
|
|
Not using ufw? Apply the equivalent (443/80 stay as-is; only 4646 needs a
|
|
source restriction) in iptables/nft/cloud-secgroup.
|
|
|
|
## 7. Web UI
|
|
|
|
The UI has no login of its own — it rides on TLS+ACL, and neither should be
|
|
public. Reach it through an SSH tunnel:
|
|
|
|
```bash
|
|
ssh -L 4646:127.0.0.1:4646 <vps>
|
|
# browser: https://localhost:4646/ui (self-signed warning is expected)
|
|
```
|
|
|
|
## 8. Everyday operations
|
|
|
|
First submission (the job file requires IMAGE_TAG — it is rendered by the
|
|
CLI at submit time; CI exports it automatically):
|
|
|
|
```bash
|
|
IMAGE_TAG=<git-sha> nomad job validate deploy/nomad/contract-check.nomad.hcl
|
|
IMAGE_TAG=<git-sha> nomad job plan deploy/nomad/contract-check.nomad.hcl
|
|
IMAGE_TAG=<git-sha> nomad job run deploy/nomad/contract-check.nomad.hcl
|
|
```
|
|
|
|
Replace `<FORGEJO_HOST>`/`<OWNER>` in the job file image paths first.
|
|
|
|
```bash
|
|
nomad job status contract-check # groups, allocs, deployments
|
|
nomad alloc logs -f <alloc-id> # or: nomad logs -f contract-check
|
|
nomad job scale contract-check worker-extract 3
|
|
nomad deployment list / promote / rollback <deployment-id>
|
|
nomad job revert contract-check <prior-job-version>
|
|
```
|
|
|
|
Health gates: the api group checks `/healthz`; every group sets
|
|
`update { auto_revert = true }` — a deployment that turns unhealthy rolls
|
|
back on its own.
|
|
|
|
## 9. Failure behavior (single server)
|
|
|
|
- Agent process dies → containers keep running untouched; deploys/scaling
|
|
impossible until `systemctl restart nomad`. Data in `/var/lib/nomad`
|
|
survives restarts.
|
|
- `systemctl status nomad`, `journalctl -u nomad -f` for diagnostics.
|
|
|
|
## 10. Image retention
|
|
|
|
Every merge pushes 6 SHA-tagged images to the Forgejo registry. Periodically
|
|
prune old tags (Forgejo UI → Packages, or the API), or keep `latest` + the
|
|
last few SHAs. Nomad also GCs unused images on the VPS automatically.
|