nixos-servers/docs/plans/monitoring-migration-victoriametrics.md

Monitoring Stack Migration to VictoriaMetrics

Overview

Migrate from Prometheus to VictoriaMetrics on a new host (monitoring02) to gain better compression and longer retention. Run in parallel with monitoring01 until validated, then switch over using a monitoring CNAME for seamless transition.

Current State

monitoring02 (10.69.13.24) - PRIMARY:

• 4 CPU cores, 8GB RAM, 60GB disk
• VictoriaMetrics with 3-month retention
• vmalert with alerting enabled (routes to local Alertmanager)
• Alertmanager -> alerttonotify -> NATS notification pipeline
• Grafana with Kanidm OIDC (grafana.home.2rjus.net)
• Loki (log aggregation)
• CNAMEs: monitoring, alertmanager, grafana, grafana-test, metrics, vmalert, loki

monitoring01 (10.69.13.13) - SHUT DOWN:

• No longer running, pending decommission

Decision: VictoriaMetrics

Per docs/plans/long-term-metrics-storage.md, VictoriaMetrics is the recommended starting point:

• Single binary replacement for Prometheus
• 5-10x better compression (30 days could become 180+ days in same space)
• Same PromQL query language (Grafana dashboards work unchanged)
• Same scrape config format (existing auto-generated configs work)

If multi-year retention with downsampling becomes necessary later, Thanos can be evaluated.

Architecture

                     ┌─────────────────┐
                     │  monitoring02   │
                     │  VictoriaMetrics│
                     │  + Grafana      │
     monitoring      │  + Loki         │
     CNAME ──────────│  + Alertmanager │
                     │  (vmalert)      │
                     └─────────────────┘
                            ▲
                            │ scrapes
            ┌───────────────┼───────────────┐
            │               │               │
       ┌────┴────┐    ┌─────┴────┐    ┌─────┴────┐
       │  ns1    │    │  ha1     │    │  ...     │
       │ :9100   │    │ :9100    │    │ :9100    │
       └─────────┘    └──────────┘    └──────────┘

Implementation Plan

Phase 1: Create monitoring02 Host [COMPLETE]

Host created and deployed at 10.69.13.24 (prod tier) with:

• 4 CPU cores, 8GB RAM, 60GB disk
• Vault integration enabled
• NATS-based remote deployment enabled
• Grafana with Kanidm OIDC deployed as test instance (grafana-test.home.2rjus.net)

Phase 2: Set Up VictoriaMetrics Stack [COMPLETE]

New service module at services/victoriametrics/ for VictoriaMetrics + vmalert + Alertmanager. Imported by monitoring02 alongside the existing Grafana service.

1. VictoriaMetrics (port 8428):

   • services.victoriametrics.enable = true
   • retentionPeriod = "3" (3 months)
   • All scrape configs migrated from Prometheus (22 jobs including auto-generated)
   • Static user override (DynamicUser disabled) for credential file access
   • OpenBao token fetch service + 30min refresh timer
   • Apiary bearer token via vault.secrets

2. vmalert for alerting rules:

   • Points to VictoriaMetrics datasource at localhost:8428
   • Reuses existing services/monitoring/rules.yml directly via settings.rule
   • Notifier sends to local Alertmanager at localhost:9093

3. Alertmanager (port 9093):

   • Same configuration as monitoring01 (alerttonotify webhook routing)
   • alerttonotify imported on monitoring02, routes alerts via NATS

4. Grafana (port 3000):

   • VictoriaMetrics datasource (localhost:8428) as default
   • Loki datasource pointing to localhost:3100

5. Loki (port 3100):

   • Same configuration as monitoring01 in standalone services/loki/ module
   • Grafana datasource updated to localhost:3100

Note: pve-exporter and pushgateway scrape targets are not included on monitoring02. pve-exporter requires a local exporter instance; pushgateway is replaced by VictoriaMetrics native push support.

Phase 3: Parallel Operation [COMPLETE]

Ran both monitoring01 and monitoring02 simultaneously to validate data collection and dashboards.

Phase 4: Add monitoring CNAME [COMPLETE]

Added CNAMEs to monitoring02: monitoring, alertmanager, grafana, metrics, vmalert, loki.

Phase 5: Update References [COMPLETE]

• Moved alertmanager, grafana, prometheus CNAMEs from http-proxy to monitoring02
• Removed corresponding Caddy reverse proxy entries from http-proxy
• monitoring02 Caddy serves alertmanager, grafana, metrics, vmalert directly

Phase 6: Enable Alerting [COMPLETE]

• Switched vmalert from blackhole mode to local Alertmanager
• alerttonotify service running on monitoring02 (NATS nkey from Vault)
• prometheus-metrics Vault policy added for OpenBao scraping
• Full alerting pipeline verified: vmalert -> Alertmanager -> alerttonotify -> NATS

Phase 7: Cutover and Decommission [IN PROGRESS]

• monitoring01 shut down (2026-02-17)
• Vault AppRole moved from approle.tf to hosts-generated.tf with extra_policies support

Remaining cleanup (separate branch):

• Update system/monitoring/logs.nix - Promtail still points to monitoring01
• Update hosts/template2/bootstrap.nix - Bootstrap Loki URL still points to monitoring01
• Remove monitoring01 from flake.nix and host configuration
• Destroy monitoring01 VM in Proxmox
• Remove monitoring01 from terraform state
• Remove or archive services/monitoring/ (Prometheus config)

Completed

• 2026-02-08: Phase 1 - monitoring02 host created
• 2026-02-17: Phase 2 - VictoriaMetrics, vmalert, Alertmanager, Loki, Grafana configured
• 2026-02-17: Phase 6 - Alerting enabled, CNAMEs migrated, monitoring01 shut down

VictoriaMetrics Service Configuration

Implemented in services/victoriametrics/default.nix. Key design decisions:

• Static user: VictoriaMetrics NixOS module uses DynamicUser, overridden with a static victoriametrics user so vault.secrets and credential files work correctly
• Shared rules: vmalert reuses services/monitoring/rules.yml via settings.rule path reference (no YAML-to-Nix conversion needed)
• Scrape config reuse: Uses the same lib/monitoring.nix functions and services/monitoring/external-targets.nix as Prometheus for auto-generated targets

Notes

• VictoriaMetrics uses port 8428 vs Prometheus 9090
• PromQL compatibility is excellent
• VictoriaMetrics native push replaces Pushgateway (remove from http-proxy if not needed)
• monitoring02 deployed via OpenTofu using create-host script
• Grafana dashboards defined declaratively via NixOS, not imported from monitoring01 state
• Tempo and Pyroscope deferred (not actively used; can be added later if needed)