monitoring02: add Loki service
Some checks failed
Run nix flake check / flake-check (push) Failing after 3m19s
Some checks failed
Run nix flake check / flake-check (push) Failing after 3m19s
Add standalone Loki service module (services/loki/) with same config as monitoring01 and import it on monitoring02. Update Grafana Loki datasource to localhost. Defer Tempo and Pyroscope migration (not actively used). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -14,8 +14,8 @@ a `monitoring` CNAME for seamless transition.
|
||||
- Alertmanager (routes to alerttonotify webhook)
|
||||
- Grafana (dashboards, datasources)
|
||||
- Loki (log aggregation from all hosts via Promtail)
|
||||
- Tempo (distributed tracing)
|
||||
- Pyroscope (continuous profiling)
|
||||
- Tempo (distributed tracing) - not actively used
|
||||
- Pyroscope (continuous profiling) - not actively used
|
||||
|
||||
**Hardcoded References to monitoring01:**
|
||||
- `system/monitoring/logs.nix` - Promtail sends logs to `http://monitoring01.home.2rjus.net:3100`
|
||||
@@ -44,9 +44,7 @@ If multi-year retention with downsampling becomes necessary later, Thanos can be
|
||||
│ VictoriaMetrics│
|
||||
│ + Grafana │
|
||||
monitoring │ + Loki │
|
||||
CNAME ──────────│ + Tempo │
|
||||
│ + Pyroscope │
|
||||
│ + Alertmanager │
|
||||
CNAME ──────────│ + Alertmanager │
|
||||
│ (vmalert) │
|
||||
└─────────────────┘
|
||||
▲
|
||||
@@ -94,16 +92,11 @@ Imported by monitoring02 alongside the existing Grafana service.
|
||||
4. **Grafana** (port 3000): [DONE]
|
||||
- VictoriaMetrics datasource (localhost:8428) as default
|
||||
- monitoring01 Prometheus datasource kept for comparison during parallel operation
|
||||
- Loki datasource pointing to monitoring01 (until Loki migrated)
|
||||
- Loki datasource pointing to localhost (after Loki migrated to monitoring02)
|
||||
|
||||
5. **Loki** (port 3100):
|
||||
- TODO: Same configuration as current
|
||||
|
||||
6. **Tempo** (ports 3200, 3201):
|
||||
- TODO: Same configuration
|
||||
|
||||
7. **Pyroscope** (port 4040):
|
||||
- TODO: Same Docker-based deployment
|
||||
5. **Loki** (port 3100): [DONE]
|
||||
- Same configuration as monitoring01 in standalone `services/loki/` module
|
||||
- Grafana datasource updated to localhost:3100
|
||||
|
||||
**Note:** pve-exporter and pushgateway scrape targets are not included on monitoring02.
|
||||
pve-exporter requires a local exporter instance; pushgateway is replaced by VictoriaMetrics
|
||||
@@ -147,7 +140,6 @@ Update hardcoded references to use the CNAME:
|
||||
- prometheus.home.2rjus.net -> monitoring.home.2rjus.net:8428
|
||||
- alertmanager.home.2rjus.net -> monitoring.home.2rjus.net:9093
|
||||
- grafana.home.2rjus.net -> monitoring.home.2rjus.net:3000
|
||||
- pyroscope.home.2rjus.net -> monitoring.home.2rjus.net:4040
|
||||
|
||||
Note: `hosts/template2/bootstrap.nix` stays pointed at monitoring01 until decommission.
|
||||
|
||||
@@ -172,8 +164,8 @@ Once ready to cut over:
|
||||
## Current Progress
|
||||
|
||||
- **Phase 1** complete (2026-02-08): monitoring02 host created, Grafana with Kanidm OIDC validated
|
||||
- **Phase 2** in progress (2026-02-17): VictoriaMetrics, vmalert, Alertmanager, Grafana datasources configured
|
||||
- Remaining: Loki, Tempo, Pyroscope migration
|
||||
- **Phase 2** complete (2026-02-17): VictoriaMetrics, vmalert, Alertmanager, Loki, Grafana datasources configured
|
||||
- Tempo and Pyroscope deferred (not actively used; can be added later if needed)
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
Reference in New Issue
Block a user