Fix Vector config and OpenObserve auth for live E2E verification
Ticket 10 verification surfaced runtime issues that static validation
missed; all verified against running Docker stack (Vector 0.43 + OO 0.92.2):
- Rewrite VRL for Vector 0.43: no coalesce/default funcs, merge!/replace!
for guarded fallible calls, no two-value parse_json destructuring
- Quote env-interpolated sink credentials (empty env left a bare YAML null)
- Fix logs sink URI: OO needs /api/{org}/{stream}/_json (stream segment
was missing, causing 404); logs now land in contract_check stream
- Replace OPENOBSERVE_AUTH_TOKEN with root email/password Basic auth:
prometheus_remote_write ignores request.headers, so remote-write got 401
- Update .env.example, DEPLOY.md, ARCHITECTURE.md, ticket 05 accordingly
Verified live: logs with service + correlation_id searchable in
OpenObserve; contract_check_* metrics queryable via its Prometheus API;
no outbound port 4318 connections from app containers.
This commit is contained in:
parent
b279d6c61a
commit
85db70887d
5 changed files with 46 additions and 48 deletions
|
|
@ -48,10 +48,6 @@ REFUND_FULL_USAGE_THRESHOLD=0.20 # ≤20% consumed + within window ⇒ full ref
|
||||||
|
|
||||||
# --- Observability (leave empty to disable) ---
|
# --- Observability (leave empty to disable) ---
|
||||||
SENTRY_DSN=
|
SENTRY_DSN=
|
||||||
# Auth header for OpenObserve. Used by Vector in the `observer` profile to forward
|
|
||||||
# logs and metrics. Generate with: echo -n 'user:pass' | base64
|
|
||||||
# Default value is for root@example.com:Complexpass#123.
|
|
||||||
OPENOBSERVE_AUTH_TOKEN=cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM=
|
|
||||||
# OTEL_EXPORTER_OTLP_ENDPOINT is no longer used. The application does not push
|
# OTEL_EXPORTER_OTLP_ENDPOINT is no longer used. The application does not push
|
||||||
# OTLP; Vector/OpenObserve collects logs/metrics passively from stdout and
|
# OTLP; Vector/OpenObserve collects logs/metrics passively from stdout and
|
||||||
# /metrics endpoints. Remove this line from existing .env files.
|
# /metrics endpoints. Remove this line from existing .env files.
|
||||||
|
|
@ -59,6 +55,8 @@ OPENOBSERVE_AUTH_TOKEN=cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM=
|
||||||
# OTEL_EXPORTER_OTLP_HEADERS=
|
# OTEL_EXPORTER_OTLP_HEADERS=
|
||||||
|
|
||||||
# --- OpenObserve (profile: observer) ---
|
# --- OpenObserve (profile: observer) ---
|
||||||
|
# Vector (profile `observer`) authenticates to OpenObserve with these
|
||||||
|
# credentials, so keep them in sync with the OpenObserve root user below.
|
||||||
OPENOBSERVE_PORT=5080
|
OPENOBSERVE_PORT=5080
|
||||||
OPENOBSERVE_ROOT_USER_EMAIL=root@example.com
|
OPENOBSERVE_ROOT_USER_EMAIL=root@example.com
|
||||||
OPENOBSERVE_ROOT_USER_PASSWORD=Complexpass#123
|
OPENOBSERVE_ROOT_USER_PASSWORD=Complexpass#123
|
||||||
|
|
|
||||||
|
|
@ -26,7 +26,10 @@ sources:
|
||||||
- http://worker-billing:9105/metrics
|
- http://worker-billing:9105/metrics
|
||||||
- http://worker-notify:9103/metrics
|
- http://worker-notify:9103/metrics
|
||||||
scrape_interval_secs: 15
|
scrape_interval_secs: 15
|
||||||
# Auth header is injected via a remap transform when METRICS_BEARER_TOKEN is set.
|
# The token is optional: an empty METRICS_BEARER_TOKEN leaves /metrics open.
|
||||||
|
authorization:
|
||||||
|
strategy: bearer
|
||||||
|
token: "${METRICS_BEARER_TOKEN-}"
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
# Transforms
|
# Transforms
|
||||||
|
|
@ -38,43 +41,32 @@ transforms:
|
||||||
inputs:
|
inputs:
|
||||||
- docker_logs
|
- docker_logs
|
||||||
source: |
|
source: |
|
||||||
# Keep the raw message and container name; derive a short service label.
|
# Derive a short service label from the container metadata
|
||||||
.container = .container_name ?? .container_id ?? "unknown"
|
# (docker_logs always provides container_name).
|
||||||
.service = .container_name ?? "unknown"
|
.container = .container_name
|
||||||
|
.service = .container_name
|
||||||
|
|
||||||
# If the log line is JSON from our structured logger, merge its fields
|
# If the log line is JSON from our structured logger, merge its fields
|
||||||
# without overwriting Vector/container metadata.
|
# without overwriting Vector/container metadata.
|
||||||
raw = .message ?? ""
|
raw = .message
|
||||||
if is_string(raw) {
|
if is_string(raw) {
|
||||||
parsed, err = parse_json(raw)
|
parsed = parse_json(raw) ?? null
|
||||||
if err == null && is_object(parsed) {
|
if is_object(parsed) {
|
||||||
for_each(parsed) -> |key, value| {
|
. = merge!(parsed, .)
|
||||||
if !exists(.) || !exists(.[key]) {
|
service = parsed.service
|
||||||
.[key] = value
|
if service != null {
|
||||||
|
.service = service
|
||||||
}
|
}
|
||||||
}
|
correlation_id = parsed.correlation_id
|
||||||
# Ensure OpenObserve can filter by the canonical service label even when
|
if correlation_id != null {
|
||||||
# the JSON payload carries its own "service" field.
|
.correlation_id = correlation_id
|
||||||
if exists(parsed.service) {
|
|
||||||
.service = parsed.service
|
|
||||||
}
|
|
||||||
if exists(parsed.correlation_id) {
|
|
||||||
.correlation_id = parsed.correlation_id
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
# Trim container runtime prefixes for a cleaner service label.
|
# Trim the compose project prefix for a cleaner service label.
|
||||||
.service = replace(.service, "^contract_check-", "")
|
if is_string(.service) {
|
||||||
|
.service = replace!(.service, r'^contract_check-', "")
|
||||||
add_metrics_auth:
|
|
||||||
type: remap
|
|
||||||
inputs:
|
|
||||||
- service_metrics
|
|
||||||
source: |
|
|
||||||
token = get_env_var("METRICS_BEARER_TOKEN") ?? ""
|
|
||||||
if token != "" {
|
|
||||||
.headers = {"Authorization": "Bearer " + token}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
# -----------------------------------------------------------------------------
|
# -----------------------------------------------------------------------------
|
||||||
|
|
@ -86,12 +78,12 @@ sinks:
|
||||||
type: http
|
type: http
|
||||||
inputs:
|
inputs:
|
||||||
- enrich_logs
|
- enrich_logs
|
||||||
uri: http://openobserve:5080/openobserve/api/default/_json
|
uri: http://openobserve:5080/openobserve/api/default/contract_check/_json
|
||||||
method: post
|
method: post
|
||||||
auth:
|
auth:
|
||||||
strategy: basic
|
strategy: basic
|
||||||
user: ""
|
user: "${OPENOBSERVE_ROOT_USER_EMAIL-}"
|
||||||
password: ${OPENOBSERVE_AUTH_TOKEN}
|
password: "${OPENOBSERVE_ROOT_USER_PASSWORD-}"
|
||||||
encoding:
|
encoding:
|
||||||
codec: json
|
codec: json
|
||||||
batch:
|
batch:
|
||||||
|
|
@ -104,12 +96,14 @@ sinks:
|
||||||
openobserve_metrics:
|
openobserve_metrics:
|
||||||
type: prometheus_remote_write
|
type: prometheus_remote_write
|
||||||
inputs:
|
inputs:
|
||||||
- add_metrics_auth
|
- service_metrics
|
||||||
|
# OpenObserve answers Vector's healthcheck probe with 405; this is
|
||||||
|
# harmless — actual remote-write POSTs succeed (200).
|
||||||
endpoint: http://openobserve:5080/openobserve/api/default/prometheus/api/v1/write
|
endpoint: http://openobserve:5080/openobserve/api/default/prometheus/api/v1/write
|
||||||
auth:
|
auth:
|
||||||
strategy: basic
|
strategy: basic
|
||||||
user: ""
|
user: "${OPENOBSERVE_ROOT_USER_EMAIL-}"
|
||||||
password: ${OPENOBSERVE_AUTH_TOKEN}
|
password: "${OPENOBSERVE_ROOT_USER_PASSWORD-}"
|
||||||
batch:
|
batch:
|
||||||
max_events: 100
|
max_events: 100
|
||||||
timeout_secs: 1
|
timeout_secs: 1
|
||||||
|
|
|
||||||
|
|
@ -390,8 +390,9 @@ services:
|
||||||
# profile: obs → Grafana + Prometheus + Loki (mature, heavier).
|
# profile: obs → Grafana + Prometheus + Loki (mature, heavier).
|
||||||
# Use for production-grade visibility.
|
# Use for production-grade visibility.
|
||||||
#
|
#
|
||||||
# For the `observer` profile, set OPENOBSERVE_AUTH_TOKEN so Vector can write
|
# Vector (profile `observer`) authenticates to OpenObserve with
|
||||||
# to OpenObserve. No OTLP endpoint configuration is needed in the app.
|
# OPENOBSERVE_ROOT_USER_EMAIL / OPENOBSERVE_ROOT_USER_PASSWORD. No OTLP
|
||||||
|
# endpoint configuration is needed in the app.
|
||||||
|
|
||||||
# ── LIGHTWEIGHT OBSERVABILITY (profile: observer) ───────────────────────────
|
# ── LIGHTWEIGHT OBSERVABILITY (profile: observer) ───────────────────────────
|
||||||
openobserve:
|
openobserve:
|
||||||
|
|
@ -428,7 +429,8 @@ services:
|
||||||
container_name: contract_check-vector
|
container_name: contract_check-vector
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
environment:
|
environment:
|
||||||
OPENOBSERVE_AUTH_TOKEN: ${OPENOBSERVE_AUTH_TOKEN}
|
OPENOBSERVE_ROOT_USER_EMAIL: ${OPENOBSERVE_ROOT_USER_EMAIL:-root@example.com}
|
||||||
|
OPENOBSERVE_ROOT_USER_PASSWORD: ${OPENOBSERVE_ROOT_USER_PASSWORD:-Complexpass#123}
|
||||||
METRICS_BEARER_TOKEN: ${METRICS_BEARER_TOKEN:-}
|
METRICS_BEARER_TOKEN: ${METRICS_BEARER_TOKEN:-}
|
||||||
volumes:
|
volumes:
|
||||||
- /var/run/docker.sock:/var/run/docker.sock:ro
|
- /var/run/docker.sock:/var/run/docker.sock:ro
|
||||||
|
|
|
||||||
|
|
@ -1087,7 +1087,8 @@ None of 1–5 requires touching `core/` application code — only compose/infra.
|
||||||
| `REFUND_WINDOW_DAYS` | `14` | full-refund window (§8a) |
|
| `REFUND_WINDOW_DAYS` | `14` | full-refund window (§8a) |
|
||||||
| `REFUND_FULL_USAGE_THRESHOLD` | `0.20` | usage ratio for full refund |
|
| `REFUND_FULL_USAGE_THRESHOLD` | `0.20` | usage ratio for full refund |
|
||||||
| `SENTRY_DSN` | (empty) | if set, init sentry |
|
| `SENTRY_DSN` | (empty) | if set, init sentry |
|
||||||
| `OPENOBSERVE_AUTH_TOKEN` | default creds base64 | Basic auth token Vector uses to forward logs/metrics to OpenObserve |
|
| `OPENOBSERVE_ROOT_USER_EMAIL` | `root@example.com` | OpenObserve root user; Vector uses it for Basic auth |
|
||||||
|
| `OPENOBSERVE_ROOT_USER_PASSWORD` | `Complexpass#123` | OpenObserve root password; Vector uses it for Basic auth |
|
||||||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | (removed) | no longer used; the application does not push OTLP |
|
| `OTEL_EXPORTER_OTLP_ENDPOINT` | (removed) | no longer used; the application does not push OTLP |
|
||||||
| `OTEL_EXPORTER_OTLP_HEADERS` | (removed) | no longer used |
|
| `OTEL_EXPORTER_OTLP_HEADERS` | (removed) | no longer used |
|
||||||
| `OTEL_SERVICE_NAME` | (removed) | service name is set by `configure_logging()` and compose labels |
|
| `OTEL_SERVICE_NAME` | (removed) | service name is set by `configure_logging()` and compose labels |
|
||||||
|
|
@ -1400,9 +1401,11 @@ enforcing the adapter boundary even in dependency ordering.
|
||||||
compose network (`api:8000/metrics`, workers on `:9101`/`:9102`/`:9104`/`:9105`/`:9103`).
|
compose network (`api:8000/metrics`, workers on `:9101`/`:9102`/`:9104`/`:9105`/`:9103`).
|
||||||
- Vector enriches logs with `service` (container name) and preserves
|
- Vector enriches logs with `service` (container name) and preserves
|
||||||
`correlation_id` from JSON log lines.
|
`correlation_id` from JSON log lines.
|
||||||
- Logs are sent to `http://openobserve:5080/openobserve/api/default/_json` and
|
- Logs are sent to
|
||||||
|
`http://openobserve:5080/openobserve/api/default/contract_check/_json` and
|
||||||
metrics to `http://openobserve:5080/openobserve/api/default/prometheus/api/v1/write`,
|
metrics to `http://openobserve:5080/openobserve/api/default/prometheus/api/v1/write`,
|
||||||
both authenticated with `OPENOBSERVE_AUTH_TOKEN`.
|
both authenticated with the OpenObserve root credentials
|
||||||
|
(`OPENOBSERVE_ROOT_USER_EMAIL` / `OPENOBSERVE_ROOT_USER_PASSWORD`).
|
||||||
- OpenObserve UI is exposed under `/openobserve/` via nginx.
|
- OpenObserve UI is exposed under `/openobserve/` via nginx.
|
||||||
- Worker metrics ports are no longer published on the Docker host; they are only
|
- Worker metrics ports are no longer published on the Docker host; they are only
|
||||||
reachable inside the compose network for scraping.
|
reachable inside the compose network for scraping.
|
||||||
|
|
|
||||||
|
|
@ -144,9 +144,10 @@ PRICE_PER_DOC_KOPECKS=19900 # 199 ₽ за документ без подпис
|
||||||
|
|
||||||
# --- Observability (опционально) ---
|
# --- Observability (опционально) ---
|
||||||
SENTRY_DSN=https://...@sentry.io/...
|
SENTRY_DSN=https://...@sentry.io/...
|
||||||
# OpenObserve auth token used by Vector in the `observer` profile.
|
# Vector (профиль `observer`) авторизуется в OpenObserve с этими кредами
|
||||||
# Generate with: echo -n 'user:pass' | base64
|
# (совпадают с root-пользователем OpenObserve).
|
||||||
OPENOBSERVE_AUTH_TOKEN=cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM=
|
OPENOBSERVE_ROOT_USER_EMAIL=root@example.com
|
||||||
|
OPENOBSERVE_ROOT_USER_PASSWORD=Complexpass#123
|
||||||
# OTEL_EXPORTER_OTLP_ENDPOINT больше не используется: приложение не шлёт OTLP.
|
# OTEL_EXPORTER_OTLP_ENDPOINT больше не используется: приложение не шлёт OTLP.
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue