torjus/nixos-servers
1
0
Fork 0
Code Issues 1 Pull Requests Actions 1 Packages Projects 1 Releases Wiki Activity

monitoring: auto-generate Prometheus scrape targets from host configs #16

Merged
torjus merged 2 commits from monitoring-improvements into master 2026-02-04 23:53:46 +00:00
Conversation 0 Commits 2 Files Changed 13 +21 -2
1 changed files with 21 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit e7980978c7 - Show all commits

23
CLAUDE.md
View File

@@ -122,9 +122,10 @@ This ensures documentation matches the exact nixpkgs version (currently NixOS 25
- Core modules: nix.nix, sshd.nix, sops.nix, acme.nix, autoupgrade.nix
- Monitoring: node-exporter and promtail on every host
- `/modules/` - Custom NixOS modules
- `homelab/` - Homelab-specific options (DNS automation, etc.)
- `homelab/` - Homelab-specific options (DNS automation, monitoring scrape targets)
- `/lib/` - Nix library functions
- `dns-zone.nix` - DNS zone generation functions
- `monitoring.nix` - Prometheus scrape target generation functions
- `/services/` - Reusable service modules, selectively imported by hosts
- `home-assistant/` - Home automation stack
- `monitoring/` - Observability stack (Prometheus, Grafana, Loki, Tempo)
@@ -156,6 +157,7 @@ All hosts automatically get:
- Internal ACME CA integration (ca.home.2rjus.net)
- Daily auto-upgrades with auto-reboot
- Prometheus node-exporter + Promtail (logs to monitoring01)
- Monitoring scrape target auto-registration via `homelab.monitoring` options
- Custom root CA trust
- DNS zone auto-registration via `homelab.dns` options
@@ -310,7 +312,7 @@ This means:
11. Deploy by running `nixos-rebuild boot --flake URL#<hostname>` on the host.
12. Run auto-upgrade on DNS servers (ns1, ns2) to pick up the new host's DNS entry
**Note:** DNS A records are auto-generated from the host's `systemd.network.networks` static IP configuration. No manual zone file editing is required.
**Note:** DNS A records and Prometheus node-exporter scrape targets are auto-generated from the host's `systemd.network.networks` static IP configuration. No manual zone file or Prometheus config editing is required.
### Important Patterns
@@ -333,6 +335,23 @@ All hosts ship metrics and logs to `monitoring01`:
- **Tracing**: Tempo for distributed tracing
- **Profiling**: Pyroscope for continuous profiling
**Scrape Target Auto-Generation:**
Prometheus scrape targets are automatically generated from host configurations, following the same pattern as DNS zone generation:
- **Node-exporter**: All flake hosts with static IPs are automatically added as node-exporter targets
- **Service targets**: Defined via `homelab.monitoring.scrapeTargets` in service modules
- **External targets**: Non-flake hosts defined in `/services/monitoring/external-targets.nix`
- **Library**: `lib/monitoring.nix` provides `generateNodeExporterTargets` and `generateScrapeConfigs`
Host monitoring options (`homelab.monitoring.*`):
- `enable` (default: `true`) - Include host in Prometheus node-exporter scrape targets
- `scrapeTargets` (default: `[]`) - Additional scrape targets exposed by this host (job_name, port, metrics_path, scheme, scrape_interval, honor_labels)
Service modules declare their scrape targets directly (e.g., `services/ca/default.nix` declares step-ca on port 9000). The Prometheus config on monitoring01 auto-generates scrape configs from all hosts.
To add monitoring targets for non-NixOS hosts, edit `/services/monitoring/external-targets.nix`.
### DNS Architecture
- `ns1` (10.69.13.5) - Primary authoritative DNS + resolver