monitoring02: enable alerting and migrate CNAMEs from http-proxy #42

Merged

torjus merged 1 commits from monitoring02-enable-alerting into master 2026-02-17 20:24:16 +00:00

Conversation 0 Commits 1 Files Changed 13 +204 -253

torjus commented 2026-02-17 20:24:11 +00:00

Owner

Copy Link

Summary

• Enable alerting on monitoring02 by switching vmalert from blackhole mode to the local Alertmanager, completing the monitoring stack migration from monitoring01
• Migrate alertmanager, grafana, and prometheus CNAMEs and Caddy reverse proxy entries from http-proxy to monitoring02, which now serves these endpoints directly
• Update Promtail, pipe-to-loki, and bootstrap scripts to use the authenticated loki.home.2rjus.net endpoint instead of the now-decommissioned monitoring01
• Move monitoring02 Vault AppRole to hosts-generated.tf with extra_policies support for OpenBao metrics scraping

Changes

• services/victoriametrics/default.nix - Enable vmalert notifier, add Alertmanager Caddy entry
• hosts/monitoring02/default.nix - Import alerttonotify service
• hosts/monitoring02/configuration.nix - Add monitoring, alertmanager, grafana CNAMEs
• services/grafana/default.nix - Add grafana.home.2rjus.net Caddy virtual host
• hosts/http-proxy/configuration.nix - Remove migrated CNAMEs
• services/http-proxy/proxy.nix - Remove prometheus, alertmanager, grafana proxy entries
• system/monitoring/logs.nix - Remove unauthenticated monitoring01 Promtail client
• system/pipe-to-loki.nix - Use authenticated loki.home.2rjus.net endpoint
• hosts/template2/bootstrap.nix - Use authenticated loki.home.2rjus.net endpoint
• terraform/vault/ - Move monitoring02 AppRole to hosts-generated.tf, add extra_policies support
• docs/plans/ - Move migration plan to completed

## Summary - Enable alerting on monitoring02 by switching vmalert from blackhole mode to the local Alertmanager, completing the monitoring stack migration from monitoring01 - Migrate alertmanager, grafana, and prometheus CNAMEs and Caddy reverse proxy entries from http-proxy to monitoring02, which now serves these endpoints directly - Update Promtail, pipe-to-loki, and bootstrap scripts to use the authenticated `loki.home.2rjus.net` endpoint instead of the now-decommissioned monitoring01 - Move monitoring02 Vault AppRole to hosts-generated.tf with extra_policies support for OpenBao metrics scraping ## Changes - `services/victoriametrics/default.nix` - Enable vmalert notifier, add Alertmanager Caddy entry - `hosts/monitoring02/default.nix` - Import alerttonotify service - `hosts/monitoring02/configuration.nix` - Add monitoring, alertmanager, grafana CNAMEs - `services/grafana/default.nix` - Add `grafana.home.2rjus.net` Caddy virtual host - `hosts/http-proxy/configuration.nix` - Remove migrated CNAMEs - `services/http-proxy/proxy.nix` - Remove prometheus, alertmanager, grafana proxy entries - `system/monitoring/logs.nix` - Remove unauthenticated monitoring01 Promtail client - `system/pipe-to-loki.nix` - Use authenticated loki.home.2rjus.net endpoint - `hosts/template2/bootstrap.nix` - Use authenticated loki.home.2rjus.net endpoint - `terraform/vault/` - Move monitoring02 AppRole to hosts-generated.tf, add extra_policies support - `docs/plans/` - Move migration plan to completed

torjus added 1 commit 2026-02-17 20:24:11 +00:00

monitoring02: enable alerting and migrate CNAMEs from http-proxy ... a6013d3950

- Switch vmalert from blackhole mode to sending alerts to local
  Alertmanager
- Import alerttonotify service so alerts route to NATS notifications
- Move alertmanager and grafana CNAMEs from http-proxy to monitoring02
- Add monitoring CNAME to monitoring02
- Add Caddy reverse proxy entries for alertmanager and grafana
- Remove prometheus, alertmanager, and grafana Caddy entries from
  http-proxy (now served directly by monitoring02)
- Move monitoring02 Vault AppRole to hosts-generated.tf with
  extra_policies support and prometheus-metrics policy
- Update Promtail to use authenticated loki.home.2rjus.net endpoint
  only (remove unauthenticated monitoring01 client)
- Update pipe-to-loki and bootstrap to use loki.home.2rjus.net with
  basic auth from Vault secret
- Move migration plan to completed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

torjus merged commit 1bba6f106a into master 2026-02-17 20:24:16 +00:00

torjus deleted branch monitoring02-enable-alerting 2026-02-17 20:24:16 +00:00

torjus referenced this issue from a commit 2026-02-17 20:24:18 +00:00

Merge pull request 'monitoring02: enable alerting and migrate CNAMEs from http-proxy' (#42) from monitoring02-enable-alerting into master

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

torjus

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: torjus/nixos-servers#42