monitoring02: add Loki service
Some checks failed
Run nix flake check / flake-check (push) Failing after 3m19s

Add standalone Loki service module (services/loki/) with same config as
monitoring01 and import it on monitoring02. Update Grafana Loki datasource
to localhost. Defer Tempo and Pyroscope migration (not actively used).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
2026-02-17 19:42:19 +01:00
parent 471f536f1f
commit 74e7c9faa4
4 changed files with 69 additions and 18 deletions

View File

@@ -14,8 +14,8 @@ a `monitoring` CNAME for seamless transition.
- Alertmanager (routes to alerttonotify webhook)
- Grafana (dashboards, datasources)
- Loki (log aggregation from all hosts via Promtail)
- Tempo (distributed tracing)
- Pyroscope (continuous profiling)
- Tempo (distributed tracing) - not actively used
- Pyroscope (continuous profiling) - not actively used
**Hardcoded References to monitoring01:**
- `system/monitoring/logs.nix` - Promtail sends logs to `http://monitoring01.home.2rjus.net:3100`
@@ -44,9 +44,7 @@ If multi-year retention with downsampling becomes necessary later, Thanos can be
│ VictoriaMetrics│
│ + Grafana │
monitoring │ + Loki │
CNAME ──────────│ + Tempo
│ + Pyroscope │
│ + Alertmanager │
CNAME ──────────│ + Alertmanager
│ (vmalert) │
└─────────────────┘
@@ -94,16 +92,11 @@ Imported by monitoring02 alongside the existing Grafana service.
4. **Grafana** (port 3000): [DONE]
- VictoriaMetrics datasource (localhost:8428) as default
- monitoring01 Prometheus datasource kept for comparison during parallel operation
- Loki datasource pointing to monitoring01 (until Loki migrated)
- Loki datasource pointing to localhost (after Loki migrated to monitoring02)
5. **Loki** (port 3100):
- TODO: Same configuration as current
6. **Tempo** (ports 3200, 3201):
- TODO: Same configuration
7. **Pyroscope** (port 4040):
- TODO: Same Docker-based deployment
5. **Loki** (port 3100): [DONE]
- Same configuration as monitoring01 in standalone `services/loki/` module
- Grafana datasource updated to localhost:3100
**Note:** pve-exporter and pushgateway scrape targets are not included on monitoring02.
pve-exporter requires a local exporter instance; pushgateway is replaced by VictoriaMetrics
@@ -147,7 +140,6 @@ Update hardcoded references to use the CNAME:
- prometheus.home.2rjus.net -> monitoring.home.2rjus.net:8428
- alertmanager.home.2rjus.net -> monitoring.home.2rjus.net:9093
- grafana.home.2rjus.net -> monitoring.home.2rjus.net:3000
- pyroscope.home.2rjus.net -> monitoring.home.2rjus.net:4040
Note: `hosts/template2/bootstrap.nix` stays pointed at monitoring01 until decommission.
@@ -172,8 +164,8 @@ Once ready to cut over:
## Current Progress
- **Phase 1** complete (2026-02-08): monitoring02 host created, Grafana with Kanidm OIDC validated
- **Phase 2** in progress (2026-02-17): VictoriaMetrics, vmalert, Alertmanager, Grafana datasources configured
- Remaining: Loki, Tempo, Pyroscope migration
- **Phase 2** complete (2026-02-17): VictoriaMetrics, vmalert, Alertmanager, Loki, Grafana datasources configured
- Tempo and Pyroscope deferred (not actively used; can be added later if needed)
## Open Questions