DealDocumentScreening/deploy/nomad/README.md

Nomad on the services VPS — install & operations runbook

Single-node Nomad agent (server+client) running the app services (api + 5 workers) while stateful infra (postgres, redis, rabbitmq, minio), the nginx edge cascade, and observability stay on docker compose.

Layout:

deploy/nomad/
├── nomad.hcl                      # agent config → /etc/nomad.d/nomad.hcl
├── policies/ci.hcl                # ACL policy for the Forgejo deploy token
├── contract-check.nomad.hcl       # job: api + worker groups
└── README.md                      # this runbook

Placeholders used below — substitute before running:

Placeholder	Meaning
<VPS_PUBLIC_IP>	public IP of the services VPS
<FORGEJO_SERVER_IP>	public IP of the Forgejo VPS (runner)
<FORGEJO_HOST>	Forgejo hostname, e.g. git.example.com
<OWNER>	Forgejo user/org owning the repo + images

---

1. Install Nomad

1a. Manual install via the official Docker image

The services VPS network 404s apt.releases.hashicorp.com, releases.hashicorp.com and GitHub release assets, and Ubuntu resolute is newer than the apt repo's suites — so extract the binary from the official Docker image instead (Docker Hub is reachable there):

docker pull hashicorp/nomad:1.9
docker create --name nomad-extract hashicorp/nomad:1.9
docker cp nomad-extract:/bin/nomad /tmp/nomad
docker rm nomad-extract
chmod +x /tmp/nomad && /tmp/nomad version
sudo install -m 0755 /tmp/nomad /usr/local/bin/nomad
# if the cp path is wrong:
#   docker run --rm --entrypoint sh hashicorp/nomad:1.9 -c 'command -v nomad'

Why 1.9.x and not 2.x: Nomad 2.x extracts the docker driver into an external plugin distributed via the blocked hosts above. 1.9.x has it built in — everything this stack uses (nomadVar templates, native services, static ports, canary updates) is fully supported there.

No .deb ⇒ no systemd unit ships; install ours:

sudo cp deploy/nomad/nomad.service /etc/systemd/system/nomad.service

1b. apt install (only on networks that reach HashiCorp's repo)

wget -O- https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
  https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
  sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y nomad
nomad version   # expect 1.7+

Note: the .deb's own unit lives in /lib/systemd/system — do NOT install deploy/nomad/nomad.service in that case.

2. Install config + data dir

sudo mkdir -p /var/lib/nomad /etc/nomad.d/tls
# from the repo root on the VPS:
sudo cp deploy/nomad/nomad.hcl /etc/nomad.d/nomad.hcl

3. Generate TLS material

The API is reachable from the internet (Forgejo runner → VPS), so TLS is mandatory. One mini-CA + one server cert; the runner gets only the CA cert.

cd /tmp && mkdir nomad-tls && cd nomad-tls

# CA (keep nomad-ca.key offline afterwards; only the .crt is ever copied)
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
  -keyout nomad-ca.key -out nomad-ca.crt -subj "/CN=Contract-Check Nomad CA"

# server cert — SANs must cover the RPC hostname AND how clients reach it
openssl req -newkey rsa:2048 -nodes \
  -keyout server.key -out server.csr \
  -subj "/CN=server.global.vps.nomad"

cat > server.ext <<'EOF'
subjectAltName = DNS:server.global.vps.nomad, DNS:localhost, IP:127.0.0.1, IP:<VPS_PUBLIC_IP>
extendedKeyUsage = serverAuth, clientAuth
EOF

openssl x509 -req -in server.csr \
  -CA nomad-ca.crt -CAkey nomad-ca.key -CAcreateserial \
  -out server.crt -days 825 -extfile server.ext

sudo install -m 644 nomad-ca.crt server.crt /etc/nomad.d/tls/
sudo install -m 600 server.key        /etc/nomad.d/tls/

Copy nomad-ca.crt to the Forgejo VPS (or a repo secret / secure storage) — the deploy workflow needs it as NOMAD_CACERT.

4. Start the agent + bootstrap ACLs

sudo systemctl enable --now nomad
systemctl status nomad --no-pager

For every CLI call on the VPS, export:

export NOMAD_ADDR=https://127.0.0.1:4646
export NOMAD_CACERT=/etc/nomad.d/tls/nomad-ca.crt

Bootstrap ACLs once:

nomad acl bootstrap
# → prints an AccessorID/SecretID management token. Store the SecretID in a
#   password manager; export it for the following commands:
export NOMAD_TOKEN=<management-secret-id>

Sanity checks:

nomad server members    # self as leader
nomad node status       # client ready, docker driver detected

5. CI policy + token + registry variables

# policy for the deploy token
nomad acl policy apply -description "Forgejo CI deploy" ci \
  deploy/nomad/policies/ci.hcl

# long-lived token for the runner (no TTL)
nomad acl token create -name forgejo-ci -policy ci
# → AccessorID + SecretID. Forgejo repo secret: NOMAD_TOKEN = SecretID

Registry credentials and app secrets — the job file templates ALL of these from nomad/jobs/contract-check (missing keys render empty; define every key once). Connection strings point at the docker0 host gateway 172.17.0.1 and the host-published infra ports (15432/17379/5672/9000) — substitute real credentials where the compose .env deviates from defaults:

nomad var put -in=json nomad/jobs/contract-check - <<'EOF'
{
  "registry_host": "<FORGEJO_HOST>",
  "registry_owner": "<OWNER>",
  "registry_user": "<OWNER>",
  "registry_token": "<forgejo token with write:package scope>",

  "database_url": "postgresql+asyncpg://contract_check:<POSTGRES_PASSWORD>@172.17.0.1:15432/contract_check",
  "redis_url": "redis://172.17.0.1:17379/0",
  "rabbitmq_url": "amqp://contract_check:<RABBITMQ_PASS>@172.17.0.1:5672/",
  "s3_endpoint_url": "http://172.17.0.1:9000",
  "s3_access_key": "<S3_ACCESS_KEY>",
  "s3_secret_key": "<S3_SECRET_KEY>",
  "s3_bucket": "contract-check-docs",

  "jwt_secret": "<openssl rand -hex 32 — reuse the value from compose .env>",
  "telegram_bot_token": "<TELEGRAM_BOT_TOKEN>",
  "ollama_api_key": "<OLLAMA_API_KEY>",
  "yandexgpt_api_key": "<YANDEXGPT_API_KEY or empty>",
  "smtp_host": "<SMTP_HOST or empty>",
  "smtp_username": "<SMTP_USERNAME or empty>",
  "smtp_password": "<SMTP_PASSWORD or empty>",
  "metrics_bearer_token": "<METRICS_BEARER_TOKEN or empty>"
}
EOF

nomad var get nomad/jobs/contract-check   # sanity check

6. Firewall

# 4646: only the Forgejo runner may talk to the API
sudo ufw allow from <FORGEJO_SERVER_IP> to any port 4646 proto tcp \
  comment 'nomad http (forgejo runner)'
# single-node: no external rpc/gossip peers
sudo ufw deny 4647/tcp comment 'nomad rpc (local only)'
sudo ufw deny 4648/tcp comment 'nomad gossip (local only)'
sudo ufw status verbose

Not using ufw? Apply the equivalent (443/80 stay as-is; only 4646 needs a source restriction) in iptables/nft/cloud-secgroup.

7. Web UI

The UI has no login of its own — it rides on TLS+ACL, and neither should be public. Reach it through an SSH tunnel:

ssh -L 4646:127.0.0.1:4646 <vps>
# browser: https://localhost:4646/ui (self-signed warning is expected)

8. Everyday operations

First submission (the job file requires IMAGE_TAG — it is rendered by the CLI at submit time; CI exports it automatically):

IMAGE_TAG=<git-sha> nomad job validate deploy/nomad/contract-check.nomad.hcl
IMAGE_TAG=<git-sha> nomad job plan    deploy/nomad/contract-check.nomad.hcl
IMAGE_TAG=<git-sha> nomad job run     deploy/nomad/contract-check.nomad.hcl

Replace <FORGEJO_HOST>/<OWNER> in the job file image paths first.

nomad job status contract-check          # groups, allocs, deployments
nomad alloc logs -f <alloc-id>           # or: nomad logs -f contract-check
nomad job scale contract-check worker-extract 3
nomad deployment list / promote / rollback <deployment-id>
nomad job revert contract-check <prior-job-version>

Health gates: the api group checks /healthz; every group sets update { auto_revert = true } — a deployment that turns unhealthy rolls back on its own.

9. Failure behavior (single server)

• Agent process dies → containers keep running untouched; deploys/scaling impossible until systemctl restart nomad. Data in /var/lib/nomad survives restarts.
• systemctl status nomad, journalctl -u nomad -f for diagnostics.

10. Image retention

Every merge pushes 6 SHA-tagged images to the Forgejo registry. Periodically prune old tags (Forgejo UI → Packages, or the API), or keep latest + the last few SHAs. Nomad also GCs unused images on the VPS automatically.