# Nomad on the services VPS — install & operations runbook

Single-node Nomad agent (server+client) running the app services
(`api` + 5 workers) while stateful infra (postgres, redis, rabbitmq, minio),
the nginx edge cascade, and observability stay on docker compose.

Layout (also `nomad.service` for manual installs, `gen-tls.sh` for TLS):

```
deploy/nomad/
├── nomad.hcl                      # agent config → /etc/nomad.d/nomad.hcl
├── nomad.service                  # systemd unit (manual install route)
├── gen-tls.sh                     # CA + server cert regeneration
├── policies/ci.hcl                # ACL policy for the Forgejo deploy token
├── contract-check.nomad.hcl       # job: api + worker groups
└── README.md                      # this runbook
```

Registry (resolved): Forgejo at `p2gnl.mu-dungeon.xyz`, owner `admin-git` —
images are `p2gnl.mu-dungeon.xyz/admin-git/contract-check-<name>:<git-sha>`.

Placeholders still used below — substitute before running:

| Placeholder | Meaning |
|---|---|
| `<VPS_PUBLIC_IP>` | public IP of the services VPS |
| `<FORGEJO_SERVER_IP>` | public IP of the Forgejo VPS (runner) |

---

## 1. Install Nomad

### 1a. Manual install via the official Docker image

The services VPS network 404s `apt.releases.hashicorp.com`,
`releases.hashicorp.com` and GitHub release assets, and Ubuntu `resolute`
is newer than the apt repo's suites — so extract the binary from the
official Docker image instead (Docker Hub is reachable there):

```bash
docker pull hashicorp/nomad:1.9
docker create --name nomad-extract hashicorp/nomad:1.9
docker cp nomad-extract:/bin/nomad /tmp/nomad
docker rm nomad-extract
chmod +x /tmp/nomad && /tmp/nomad version
sudo install -m 0755 /tmp/nomad /usr/local/bin/nomad
# if the cp path is wrong:
#   docker run --rm --entrypoint sh hashicorp/nomad:1.9 -c 'command -v nomad'
```

**Why 1.9.x and not 2.x:** Nomad 2.x extracts the docker driver into an
external plugin distributed via the blocked hosts above. 1.9.x has it
built in — everything this stack uses (nomadVar templates, native
services, static ports, canary updates) is fully supported there.

No `.deb` ⇒ no systemd unit ships; install ours:

```bash
sudo cp deploy/nomad/nomad.service /etc/systemd/system/nomad.service
```

### 1b. apt install (only on networks that reach HashiCorp's repo)

```bash
wget -O- https://apt.releases.hashicorp.com/gpg | \
  sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] \
  https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
  sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt-get update && sudo apt-get install -y nomad
nomad version   # expect 1.7+
```

Note: the `.deb`'s own unit lives in /lib/systemd/system — do NOT install
`deploy/nomad/nomad.service` in that case.

## 2. Install config + data dir

```bash
sudo mkdir -p /var/lib/nomad /etc/nomad.d/tls
# from the repo root on the VPS:
sudo cp deploy/nomad/nomad.hcl /etc/nomad.d/nomad.hcl
```

## 3. Generate TLS material

The API is reachable from the internet (Forgejo runner → VPS), so TLS is
mandatory. Use the script (CA + server cert with all required SANs in one
shot — hand-rolled openssl commands tend to miss `server.global.nomad`,
which `verify_server_hostname` requires):

```bash
sudo deploy/nomad/gen-tls.sh <VPS_PUBLIC_IP>
sudo systemctl restart nomad
```

The runner gets **only** the CA cert (`nomad-ca.crt`) as `NOMAD_CACERT`.

Manual equivalent (what the script does):
```bash
cd /tmp && mkdir nomad-tls && cd nomad-tls

# CA (keep nomad-ca.key offline afterwards; only the .crt is ever copied)
openssl req -x509 -newkey rsa:2048 -nodes -days 3650 \
  -keyout nomad-ca.key -out nomad-ca.crt -subj "/CN=Contract-Check Nomad CA"

# server cert — SANs must cover the RPC hostname AND how clients reach it
openssl req -newkey rsa:2048 -nodes \
  -keyout server.key -out server.csr \
  -subj "/CN=server.global.nomad"

cat > server.ext <<'EOF'
subjectAltName = DNS:server.global.nomad, DNS:server.global.vps.nomad, DNS:localhost, IP:127.0.0.1, IP:<VPS_PUBLIC_IP>
extendedKeyUsage = serverAuth, clientAuth
EOF

openssl x509 -req -in server.csr \
  -CA nomad-ca.crt -CAkey nomad-ca.key -CAcreateserial \
  -out server.crt -days 825 -extfile server.ext

sudo install -m 644 nomad-ca.crt server.crt /etc/nomad.d/tls/
sudo install -m 600 server.key        /etc/nomad.d/tls/
```

Copy `nomad-ca.crt` to the Forgejo VPS (or a repo secret / secure storage)
— the deploy workflow needs it as `NOMAD_CACERT`.

## 4. Start the agent + bootstrap ACLs

```bash
sudo systemctl enable --now nomad
systemctl status nomad --no-pager
```

For every CLI call on the VPS, export:

```bash
export NOMAD_ADDR=https://127.0.0.1:4646
export NOMAD_CACERT=/etc/nomad.d/tls/nomad-ca.crt
```

Bootstrap ACLs once:

```bash
nomad acl bootstrap
# → prints an AccessorID/SecretID management token. Store the SecretID in a
#   password manager; export it for the following commands:
export NOMAD_TOKEN=<management-secret-id>
```

Lost the management token? The API reset path (`ResetIndex` in the
bootstrap call) does not work on this version. Since the failure window
here is initial setup — the cluster is EMPTY (no jobs, no variables) — do a
state reset (wipes raft: jobs, tokens, variables):

```bash
sudo systemctl stop nomad
# ensure acl { enabled = true } in /etc/nomad.d/nomad.hcl
sudo rm -rf /var/lib/nomad/*
sudo systemctl start nomad
sleep 5 && nomad acl bootstrap   # fresh management token — save it
```

On a POPULATED cluster never do this — keep the bootstrap SecretID in a
password manager from day one.

Handy: keep the CLI env in a root-only file and source it on demand
(never into .bashrc — tokens shouldn't leak to every shell):

```bash
umask 077
cat > /root/.nomadrc <<'EOF'
export NOMAD_ADDR=https://127.0.0.1:4646
export NOMAD_CACERT=/etc/nomad.d/tls/nomad-ca.crt
export NOMAD_TOKEN=<management-secret-id>
EOF
# per session: . /root/.nomadrc
```

Sanity checks:

```bash
nomad server members    # self as leader
nomad node status       # client ready, docker driver detected
```

## 5. CI policy + token + registry login + variables

```bash
# policy for the deploy token
nomad acl policy apply -description "Forgejo CI deploy" ci \
  deploy/nomad/policies/ci.hcl

# long-lived token for the runner (no TTL)
nomad acl token create -name forgejo-ci -policy ci
# → AccessorID + SecretID. Forgejo repo secret: NOMAD_TOKEN = SecretID
```

Registry login on the VPS — Nomad 1.9's docker driver predates the
`usernameFile`/`passwordFile` auth fields, so the job file has NO `auth`
block. Instead, log in once on the host (the agent runs as root and uses
the host docker daemon → all task pulls read root's docker config):

```bash
docker login p2gnl.mu-dungeon.xyz -u admin-git -p <forgejo token, read:package>
```

Registry credentials and app secrets — the job file templates ALL of these
from `nomad/jobs/contract-check` (missing keys render empty; define every
key once). Connection strings point at the docker0 host gateway
`172.17.0.1` and the host-published infra ports (15432/17379/5672/9000) —
substitute real credentials where the compose `.env` deviates from
defaults:

```bash
# NOTE: -in=json requires the {"Items": {...}} envelope — a flat key map
# is rejected with "variable missing required Items object".
nomad var put -in=json nomad/jobs/contract-check - <<'EOF'
{
  "Items": {
    "registry_host": "p2gnl.mu-dungeon.xyz",
    "registry_owner": "admin-git",
    "registry_user": "admin-git",
    "registry_token": "<forgejo token with write:package scope>",

    "database_url": "postgresql+asyncpg://contract_check:<POSTGRES_PASSWORD>@172.17.0.1:15432/contract_check",
    "redis_url": "redis://172.17.0.1:17379/0",
    "rabbitmq_url": "amqp://contract_check:<RABBITMQ_PASS>@172.17.0.1:5672/",
    "s3_endpoint_url": "http://172.17.0.1:9000",
    "s3_access_key": "<S3_ACCESS_KEY>",
    "s3_secret_key": "<S3_SECRET_KEY>",
    "s3_bucket": "contract-check-docs",

    "jwt_secret": "<openssl rand -hex 32 — reuse the value from compose .env>",
    "telegram_bot_token": "<TELEGRAM_BOT_TOKEN>",
    "ollama_api_key": "<OLLAMA_API_KEY>",
    "yandexgpt_api_key": "<YANDEXGPT_API_KEY or empty>",
    "smtp_host": "<SMTP_HOST or empty>",
    "smtp_username": "<SMTP_USERNAME or empty>",
    "smtp_password": "<SMTP_PASSWORD or empty>",
    "metrics_bearer_token": "<METRICS_BEARER_TOKEN or empty>"
  }
}
EOF

nomad var get nomad/jobs/contract-check   # sanity check
```

## 6. Firewall

```bash
# 4646: only the Forgejo runner may talk to the API
sudo ufw allow from <FORGEJO_SERVER_IP> to any port 4646 proto tcp \
  comment 'nomad http (forgejo runner)'
# single-node: no external rpc/gossip peers
sudo ufw deny 4647/tcp comment 'nomad rpc (local only)'
sudo ufw deny 4648/tcp comment 'nomad gossip (local only)'
sudo ufw status verbose
```

Not using ufw? Apply the equivalent (443/80 stay as-is; only 4646 needs a
source restriction) in iptables/nft/cloud-secgroup.

## 7. Web UI

The UI has no login of its own — it rides on TLS+ACL, and neither should be
public. Reach it through an SSH tunnel:

```bash
ssh -L 4646:127.0.0.1:4646 <vps>
# browser: https://localhost:4646/ui (self-signed warning is expected)
```

## 8. Everyday operations

First submission (the job file uses `${IMAGE_TAG}` — rendered by `envsubst`
at submit time; CI does this automatically):

```bash
IMAGE_TAG=<git-sha> envsubst < deploy/nomad/contract-check.nomad.hcl | nomad job validate -
IMAGE_TAG=<git-sha> envsubst < deploy/nomad/contract-check.nomad.hcl | nomad job plan -
IMAGE_TAG=<git-sha> envsubst < deploy/nomad/contract-check.nomad.hcl | nomad job run -
```

Or via Makefile (handles envsubst + nomad env automatically):
```bash
IMAGE_TAG=<git-sha> make nomad-validate
IMAGE_TAG=<git-sha> make nomad-plan
IMAGE_TAG=<git-sha> make nomad-deploy
```

```bash
nomad job status contract-check          # groups, allocs, deployments
nomad alloc logs -f <alloc-id>           # or: nomad logs -f contract-check
nomad job scale contract-check worker-extract 3
nomad deployment list / promote / rollback <deployment-id>
nomad job revert contract-check <prior-job-version>
```

Health gates: the api group checks `/healthz`; every group sets
`update { auto_revert = true }` — a deployment that turns unhealthy rolls
back on its own.

## 9. Failure behavior (single server)

- Agent process dies → containers keep running untouched; deploys/scaling
  impossible until `systemctl restart nomad`. Data in `/var/lib/nomad`
  survives restarts.
- `systemctl status nomad`, `journalctl -u nomad -f` for diagnostics.

## 10. Image retention

Every merge pushes 6 SHA-tagged images to the Forgejo registry. Periodically
prune old tags (Forgejo UI → Packages, or the API), or keep `latest` + the
last few SHAs. Nomad also GCs unused images on the VPS automatically.