DealDocumentScreening/deploy/nomad

Nomad on the services VPS — install & operations runbook

Single-node Nomad agent (server+client) running the app services (api + 5 workers) while stateful infra (postgres, redis, rabbitmq, minio), the nginx edge cascade, and observability stay on docker compose.

Layout (also nomad.service for manual installs, gen-tls.sh for TLS):

deploy/nomad/
├── nomad.hcl                      # agent config → /etc/nomad.d/nomad.hcl
├── nomad.service                  # systemd unit (manual install route)
├── gen-tls.sh                     # CA + server cert regeneration
├── policies/ci.hcl                # ACL policy for the Forgejo deploy token
├── contract-check.nomad.hcl       # job: api + worker groups
└── README.md                      # this runbook

Registry (resolved): Forgejo at p2gnl.mu-dungeon.xyz, owner admin-git — images are p2gnl.mu-dungeon.xyz/admin-git/contract-check-<name>:<git-sha>.

Placeholders still used below — substitute before running:

Placeholder	Meaning
<VPS_PUBLIC_IP>	public IP of the services VPS
<FORGEJO_SERVER_IP>	public IP of the Forgejo VPS (runner)

---

1. Install Nomad

1a. Manual install via the official Docker image

The services VPS network 404s apt.releases.hashicorp.com, releases.hashicorp.com and GitHub release assets, and Ubuntu resolute is newer than the apt repo's suites — so extract the binary from the official Docker image instead (Docker Hub is reachable there):

docker pull hashicorp/nomad:1.9
docker create --name nomad-extract hashicorp/nomad:1.9
docker cp nomad-extract:/bin/nomad /tmp/nomad
docker rm nomad-extract
chmod +x /tmp/nomad && /tmp/nomad version
sudo install -m 0755 /tmp/nomad /usr/local/bin/nomad
# if the cp path is wrong:
#   docker run --rm --entrypoint sh hashicorp/nomad:1.9 -c 'command -v nomad'

Why 1.9.x and not 2.x: Nomad 2.x extracts the docker driver into an external plugin distributed via the blocked hosts above. 1.9.x has it built in — everything this stack uses (nomadVar templates, native services, static ports, canary updates) is fully supported there.

No .deb ⇒ no systemd unit ships; install ours:

sudo cp deploy/nomad/nomad.service /etc/systemd/system/nomad.service

1b. apt install (only on networks that reach HashiCorp's repo)

wget -O- https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
  https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
  sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y nomad
nomad version   # expect 1.7+

Note: the .deb's own unit lives in /lib/systemd/system — do NOT install deploy/nomad/nomad.service in that case.

2. Install config + data dir

sudo mkdir -p /var/lib/nomad /etc/nomad.d/tls
# from the repo root on the VPS:
sudo cp deploy/nomad/nomad.hcl /etc/nomad.d/nomad.hcl

3. Generate TLS material

The API is reachable from the internet (Forgejo runner → VPS), so TLS is mandatory. Use the script (CA + server cert with all required SANs in one shot — hand-rolled openssl commands tend to miss server.global.nomad, which verify_server_hostname requires):

sudo deploy/nomad/gen-tls.sh <VPS_PUBLIC_IP>
sudo systemctl restart nomad

The runner gets only the CA cert (nomad-ca.crt) as NOMAD_CACERT.

Manual equivalent (what the script does):

cd /tmp && mkdir nomad-tls && cd nomad-tls

# CA (keep nomad-ca.key offline afterwards; only the .crt is ever copied)
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
  -keyout nomad-ca.key -out nomad-ca.crt -subj "/CN=Contract-Check Nomad CA"

# server cert — SANs must cover the RPC hostname AND how clients reach it
openssl req -newkey rsa:2048 -nodes \
  -keyout server.key -out server.csr \
  -subj "/CN=server.global.nomad"

cat > server.ext <<'EOF'
subjectAltName = DNS:server.global.nomad, DNS:server.global.vps.nomad, DNS:localhost, IP:127.0.0.1, IP:<VPS_PUBLIC_IP>
extendedKeyUsage = serverAuth, clientAuth
EOF

openssl x509 -req -in server.csr \
  -CA nomad-ca.crt -CAkey nomad-ca.key -CAcreateserial \
  -out server.crt -days 825 -extfile server.ext

sudo install -m 644 nomad-ca.crt server.crt /etc/nomad.d/tls/
sudo install -m 600 server.key        /etc/nomad.d/tls/

Copy nomad-ca.crt to the Forgejo VPS (or a repo secret / secure storage) — the deploy workflow needs it as NOMAD_CACERT.

4. Start the agent + bootstrap ACLs

sudo systemctl enable --now nomad
systemctl status nomad --no-pager

For every CLI call on the VPS, export:

export NOMAD_ADDR=https://127.0.0.1:4646
export NOMAD_CACERT=/etc/nomad.d/tls/nomad-ca.crt

Bootstrap ACLs once:

nomad acl bootstrap
# → prints an AccessorID/SecretID management token. Store the SecretID in a
#   password manager; export it for the following commands:
export NOMAD_TOKEN=<management-secret-id>

Lost the management token? The API reset path (ResetIndex in the bootstrap call) does not work on this version. Since the failure window here is initial setup — the cluster is EMPTY (no jobs, no variables) — do a state reset (wipes raft: jobs, tokens, variables):

sudo systemctl stop nomad
# ensure acl { enabled = true } in /etc/nomad.d/nomad.hcl
sudo rm -rf /var/lib/nomad/*
sudo systemctl start nomad
sleep 5 && nomad acl bootstrap   # fresh management token — save it

On a POPULATED cluster never do this — keep the bootstrap SecretID in a password manager from day one.

Handy: keep the CLI env in a root-only file and source it on demand (never into .bashrc — tokens shouldn't leak to every shell):

umask 077
cat > /root/.nomadrc <<'EOF'
export NOMAD_ADDR=https://127.0.0.1:4646
export NOMAD_CACERT=/etc/nomad.d/tls/nomad-ca.crt
export NOMAD_TOKEN=<management-secret-id>
EOF
# per session: . /root/.nomadrc

Sanity checks:

nomad server members    # self as leader
nomad node status       # client ready, docker driver detected

5. CI policy + token + registry login + variables

# policy for the deploy token
nomad acl policy apply -description "Forgejo CI deploy" ci \
  deploy/nomad/policies/ci.hcl

# long-lived token for the runner (no TTL)
nomad acl token create -name forgejo-ci -policy ci
# → AccessorID + SecretID. Forgejo repo secret: NOMAD_TOKEN = SecretID

Registry login on the VPS — Nomad 1.9's docker driver predates the usernameFile/passwordFile auth fields, so the job file has NO auth block. Instead, log in once on the host (the agent runs as root and uses the host docker daemon → all task pulls read root's docker config):

docker login p2gnl.mu-dungeon.xyz -u admin-git -p <forgejo token, read:package>

Registry credentials and app secrets — the job file templates ALL of these from nomad/jobs/contract-check (missing keys render empty; define every key once). Connection strings point at the docker0 host gateway 172.17.0.1 and the host-published infra ports (15432/17379/5672/9000) — substitute real credentials where the compose .env deviates from defaults:

# NOTE: -in=json requires the {"Items": {...}} envelope — a flat key map
# is rejected with "variable missing required Items object".
nomad var put -in=json nomad/jobs/contract-check - <<'EOF'
{
  "Items": {
    "registry_host": "p2gnl.mu-dungeon.xyz",
    "registry_owner": "admin-git",
    "registry_user": "admin-git",
    "registry_token": "<forgejo token with write:package scope>",

    "database_url": "postgresql+asyncpg://contract_check:<POSTGRES_PASSWORD>@172.17.0.1:15432/contract_check",
    "redis_url": "redis://172.17.0.1:17379/0",
    "rabbitmq_url": "amqp://contract_check:<RABBITMQ_PASS>@172.17.0.1:5672/",
    "s3_endpoint_url": "http://172.17.0.1:9000",
    "s3_access_key": "<S3_ACCESS_KEY>",
    "s3_secret_key": "<S3_SECRET_KEY>",
    "s3_bucket": "contract-check-docs",

    "jwt_secret": "<openssl rand -hex 32 — reuse the value from compose .env>",
    "telegram_bot_token": "<TELEGRAM_BOT_TOKEN>",
    "ollama_api_key": "<OLLAMA_API_KEY>",
    "yandexgpt_api_key": "<YANDEXGPT_API_KEY or empty>",
    "smtp_host": "<SMTP_HOST or empty>",
    "smtp_username": "<SMTP_USERNAME or empty>",
    "smtp_password": "<SMTP_PASSWORD or empty>",
    "metrics_bearer_token": "<METRICS_BEARER_TOKEN or empty>"
  }
}
EOF

nomad var get nomad/jobs/contract-check   # sanity check

6. Firewall

# 4646: only the Forgejo runner may talk to the API
sudo ufw allow from <FORGEJO_SERVER_IP> to any port 4646 proto tcp \
  comment 'nomad http (forgejo runner)'
# single-node: no external rpc/gossip peers
sudo ufw deny 4647/tcp comment 'nomad rpc (local only)'
sudo ufw deny 4648/tcp comment 'nomad gossip (local only)'
sudo ufw status verbose

Not using ufw? Apply the equivalent (443/80 stay as-is; only 4646 needs a source restriction) in iptables/nft/cloud-secgroup.

7. Web UI

The UI has no login of its own — it rides on TLS+ACL, and neither should be public. Reach it through an SSH tunnel:

ssh -L 4646:127.0.0.1:4646 <vps>
# browser: https://localhost:4646/ui (self-signed warning is expected)

8. Everyday operations

First submission (the job file uses ${IMAGE_TAG} — rendered by envsubst at submit time; CI does this automatically):

IMAGE_TAG=<git-sha> envsubst < deploy/nomad/contract-check.nomad.hcl | nomad job validate -
IMAGE_TAG=<git-sha> envsubst < deploy/nomad/contract-check.nomad.hcl | nomad job plan -
IMAGE_TAG=<git-sha> envsubst < deploy/nomad/contract-check.nomad.hcl | nomad job run -

Or via Makefile (handles envsubst + nomad env automatically):

IMAGE_TAG=<git-sha> make nomad-validate
IMAGE_TAG=<git-sha> make nomad-plan
IMAGE_TAG=<git-sha> make nomad-deploy

nomad job status contract-check          # groups, allocs, deployments
nomad alloc logs -f <alloc-id>           # or: nomad logs -f contract-check
nomad job scale contract-check worker-extract 3
nomad deployment list / promote / rollback <deployment-id>
nomad job revert contract-check <prior-job-version>

Health gates: the api group checks /healthz; every group sets update { auto_revert = true } — a deployment that turns unhealthy rolls back on its own.

9. Failure behavior (single server)

• Agent process dies → containers keep running untouched; deploys/scaling impossible until systemctl restart nomad. Data in /var/lib/nomad survives restarts.
• systemctl status nomad, journalctl -u nomad -f for diagnostics.

10. Image retention

Every merge pushes 6 SHA-tagged images to the Forgejo registry. Periodically prune old tags (Forgejo UI → Packages, or the API), or keep latest + the last few SHAs. Nomad also GCs unused images on the VPS automatically.