homelab-deploy: enable prometheus metrics

- Update homelab-deploy input to get metrics support
- Enable metrics endpoint on port 9972
- Add scrape target for prometheus auto-discovery

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

.claude/skills/observability/SKILL.md

@@ -0,0 +1,250 @@
+---
+name: observability
+description: Reference guide for exploring Prometheus metrics and Loki logs when troubleshooting homelab issues. Use when investigating system state, deployments, service health, or searching logs.
+---
+
+# Observability Troubleshooting Guide
+
+Quick reference for exploring Prometheus metrics and Loki logs to troubleshoot homelab issues.
+
+## Available Tools
+
+Use the `lab-monitoring` MCP server tools:
+
+**Metrics:**
+- `search_metrics` - Find metrics by name substring
+- `get_metric_metadata` - Get type/help for a specific metric
+- `query` - Execute PromQL queries
+- `list_targets` - Check scrape target health
+- `list_alerts` / `get_alert` - View active alerts
+
+**Logs:**
+- `query_logs` - Execute LogQL queries against Loki
+- `list_labels` - List available log labels
+- `list_label_values` - List values for a specific label
+
+---
+
+## Logs Reference
+
+### Label Reference
+
+Available labels for log queries:
+- `host` - Hostname (e.g., `ns1`, `monitoring01`, `ha1`)
+- `systemd_unit` - Systemd unit name (e.g., `nsd.service`, `nixos-upgrade.service`)
+- `job` - Either `systemd-journal` (most logs) or `varlog` (file-based logs)
+- `filename` - For `varlog` job, the log file path
+- `hostname` - Alternative to `host` for some streams
+
+### Log Format
+
+Journal logs are JSON-formatted. Key fields:
+- `MESSAGE` - The actual log message
+- `PRIORITY` - Syslog priority (6=info, 4=warning, 3=error)
+- `SYSLOG_IDENTIFIER` - Program name
+
+### Basic LogQL Queries
+
+**Logs from a specific service on a host:**
+```logql
+{host="ns1", systemd_unit="nsd.service"}
+```
+
+**All logs from a host:**
+```logql
+{host="monitoring01"}
+```
+
+**Logs from a service across all hosts:**
+```logql
+{systemd_unit="nixos-upgrade.service"}
+```
+
+**Substring matching (case-sensitive):**
+```logql
+{host="ha1"} |= "error"
+```
+
+**Exclude pattern:**
+```logql
+{host="ns1"} != "routine"
+```
+
+**Regex matching:**
+```logql
+{systemd_unit="prometheus.service"} |~ "scrape.*failed"
+```
+
+**File-based logs (caddy access logs, etc):**
+```logql
+{job="varlog", hostname="nix-cache01"}
+{job="varlog", filename="/var/log/caddy/nix-cache.log"}
+```
+
+### Time Ranges
+
+Default lookback is 1 hour. Use `start` parameter for older logs:
+- `start: "1h"` - Last hour (default)
+- `start: "24h"` - Last 24 hours
+- `start: "168h"` - Last 7 days
+
+### Common Services
+
+Useful systemd units for troubleshooting:
+- `nixos-upgrade.service` - Daily auto-upgrade logs
+- `nsd.service` - DNS server (ns1/ns2)
+- `prometheus.service` - Metrics collection
+- `loki.service` - Log aggregation
+- `caddy.service` - Reverse proxy
+- `home-assistant.service` - Home automation
+- `step-ca.service` - Internal CA
+- `openbao.service` - Secrets management
+- `sshd.service` - SSH daemon
+- `nix-gc.service` - Nix garbage collection
+
+### Extracting JSON Fields
+
+Parse JSON and filter on fields:
+```logql
+{systemd_unit="prometheus.service"} | json | PRIORITY="3"
+```
+
+---
+
+## Metrics Reference
+
+### Deployment & Version Status
+
+Check which NixOS revision hosts are running:
+
+```promql
+nixos_flake_info
+```
+
+Labels:
+- `current_rev` - Git commit of the running NixOS configuration
+- `remote_rev` - Latest commit on the remote repository
+- `nixpkgs_rev` - Nixpkgs revision used to build the system
+- `nixos_version` - Full NixOS version string (e.g., `25.11.20260203.e576e3c`)
+
+Check if hosts are behind on updates:
+
+```promql
+nixos_flake_revision_behind == 1
+```
+
+View flake input versions:
+
+```promql
+nixos_flake_input_info
+```
+
+Labels: `input` (name), `rev` (revision), `type` (git/github)
+
+Check flake input age:
+
+```promql
+nixos_flake_input_age_seconds / 86400
+```
+
+Returns age in days for each flake input.
+
+### System Health
+
+Basic host availability:
+
+```promql
+up{job="node-exporter"}
+```
+
+CPU usage by host:
+
+```promql
+100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+Memory usage:
+
+```promql
+1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
+```
+
+Disk space (root filesystem):
+
+```promql
+node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
+```
+
+### Service-Specific Metrics
+
+Common job names:
+- `node-exporter` - System metrics (all hosts)
+- `nixos-exporter` - NixOS version/generation metrics
+- `caddy` - Reverse proxy metrics
+- `prometheus` / `loki` / `grafana` - Monitoring stack
+- `home-assistant` - Home automation
+- `step-ca` - Internal CA
+
+### Instance Label Format
+
+The `instance` label uses FQDN format:
+
+```
+<hostname>.home.2rjus.net:<port>
+```
+
+Example queries filtering by host:
+
+```promql
+up{instance=~"monitoring01.*"}
+node_load1{instance=~"ns1.*"}
+```
+
+---
+
+## Troubleshooting Workflows
+
+### Check Deployment Status Across Fleet
+
+1. Query `nixos_flake_info` to see all hosts' current revisions
+2. Check `nixos_flake_revision_behind` for hosts needing updates
+3. Look at upgrade logs: `{systemd_unit="nixos-upgrade.service"}` with `start: "24h"`
+
+### Investigate Service Issues
+
+1. Check `up{job="<service>"}` for scrape failures
+2. Use `list_targets` to see target health details
+3. Query service logs: `{host="<host>", systemd_unit="<service>.service"}`
+4. Search for errors: `{host="<host>"} |= "error"`
+5. Check `list_alerts` for related alerts
+
+### After Deploying Changes
+
+1. Verify `current_rev` updated in `nixos_flake_info`
+2. Confirm `nixos_flake_revision_behind == 0`
+3. Check service logs for startup issues
+4. Check service metrics are being scraped
+
+### Debug SSH/Access Issues
+
+```logql
+{host="<host>", systemd_unit="sshd.service"}
+```
+
+### Check Recent Upgrades
+
+```logql
+{systemd_unit="nixos-upgrade.service"}
+```
+
+With `start: "24h"` to see last 24 hours of upgrades across all hosts.
+
+---
+
+## Notes
+
+- Default scrape interval is 15s for most metrics targets
+- Default log lookback is 1h - use `start` parameter for older logs
+- Use `rate()` for counter metrics, direct queries for gauges
+- The `instance` label includes the port, use regex matching (`=~`) for hostname-only filters
+- Log `MESSAGE` field contains the actual log content in JSON format

.claude/skills/quick-plan/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: quick-plan
+description: Create a planning document for a future homelab project. Use when the user wants to document ideas for future work without implementing immediately.
+argument-hint: [topic or feature to plan]
+---
+
+# Quick Plan Generator
+
+Create a planning document for a future homelab infrastructure project. Plans are for documenting ideas and approaches that will be implemented later, not immediately.
+
+## Input
+
+The user provides: $ARGUMENTS
+
+## Process
+
+1. **Understand the topic**: Research the codebase to understand:
+   - Current state of related systems
+   - Existing patterns and conventions
+   - Relevant NixOS options or packages
+   - Any constraints or dependencies
+
+2. **Evaluate options**: If there are multiple approaches, research and compare them with pros/cons.
+
+3. **Draft the plan**: Create a markdown document following the structure below.
+
+4. **Save the plan**: Write to `docs/plans/<topic-slug>.md` using a kebab-case filename derived from the topic.
+
+## Plan Structure
+
+Use these sections as appropriate (not all plans need every section):
+
+```markdown
+# Title
+
+## Overview/Goal
+Brief description of what this plan addresses and why.
+
+## Current State
+What exists today that's relevant to this plan.
+
+## Options Evaluated (if multiple approaches)
+For each option:
+- **Option Name**
+  - **Pros:** bullet points
+  - **Cons:** bullet points
+  - **Verdict:** brief assessment
+
+Or use a comparison table for structured evaluation.
+
+## Recommendation/Decision
+What approach is recommended and why. Include rationale.
+
+## Implementation Steps
+Numbered phases or steps. Be specific but not overly detailed.
+Can use sub-sections for major phases.
+
+## Open Questions
+Things still to be determined. Use checkbox format:
+- [ ] Question 1?
+- [ ] Question 2?
+
+## Notes (optional)
+Additional context, caveats, or references.
+```
+
+## Style Guidelines
+
+- **Concise**: Use bullet points, avoid verbose paragraphs
+- **Technical but accessible**: Include NixOS config snippets when relevant
+- **Future-oriented**: These are plans, not specifications
+- **Acknowledge uncertainty**: Use "Open Questions" for unresolved decisions
+- **Reference existing patterns**: Mention how this fits with existing infrastructure
+- **Tables for comparisons**: Use markdown tables when comparing options
+- **Practical focus**: Emphasize what needs to happen, not theory
+
+## Examples of Good Plans
+
+Reference these existing plans for style guidance:
+- `docs/plans/auth-system-replacement.md` - Good option evaluation with table
+- `docs/plans/truenas-migration.md` - Good decision documentation with rationale
+- `docs/plans/remote-access.md` - Good multi-option comparison
+- `docs/plans/prometheus-scrape-target-labels.md` - Good implementation detail level
+
+## After Creating the Plan
+
+1. Tell the user the plan was saved to `docs/plans/<filename>.md`
+2. Summarize the key points
+3. Ask if they want any adjustments before committing

.gitignore

@@ -1,5 +1,6 @@
 .direnv/
 result
+result-*
 
 # Terraform/OpenTofu
 terraform/.terraform/

.mcp.json

@@ -22,6 +22,17 @@
         "ALERTMANAGER_URL": "https://alertmanager.home.2rjus.net",
         "LOKI_URL": "http://monitoring01.home.2rjus.net:3100"
       }
+    },
+    "homelab-deploy": {
+      "command": "nix",
+      "args": [
+        "run",
+        "git+https://git.t-juice.club/torjus/homelab-deploy",
+        "--",
+        "mcp",
+        "--nats-url", "nats://nats1.home.2rjus.net:4222",
+        "--nkey-file", "/home/torjus/.config/homelab-deploy/test-deployer.nkey"
+      ]
     }
   }
 }

.sops.yaml

@@ -10,7 +10,6 @@ keys:
   - &server_nix-cache01 age1w029fksjv0edrff9p7s03tgk3axecdkppqymfpwfn2nu2gsqqefqc37sxq
   - &server_pgdb1 age1ha34qeksr4jeaecevqvv2afqem67eja2mvawlmrqsudch0e7fe7qtpsekv
   - &server_nats1 age1cxt8kwqzx35yuldazcc49q88qvgy9ajkz30xu0h37uw3ts97jagqgmn2ga
-  - &server_auth01 age16prza00sqzuhwwcyakj6z4hvwkruwkqpmmrsn94a5ucgpkelncdq2ldctk
 creation_rules:
   - path_regex: secrets/[^/]+\.(yaml|json|env|ini)
     key_groups:
@@ -26,7 +25,6 @@ creation_rules:
         - *server_nix-cache01
         - *server_pgdb1
         - *server_nats1
-        - *server_auth01
   - path_regex: secrets/ca/[^/]+\.(yaml|json|env|ini|)
     key_groups:
       - age:
@@ -52,8 +50,3 @@ creation_rules:
       - age:
         - *admin_torjus
         - *server_http-proxy
-  - path_regex: secrets/auth01/[^/]+\.(yaml|json|env|ini|)
-    key_groups:
-      - age:
-        - *admin_torjus
-        - *server_auth01

CLAUDE.md

@@ -35,6 +35,21 @@ nix build .#create-host
 
 Do not automatically deploy changes. Deployments are usually done by updating the master branch, and then triggering the auto update on the specific host.
 
+### Testing Feature Branches on Hosts
+
+All hosts have the `nixos-rebuild-test` helper script for testing feature branches before merging:
+
+```bash
+# On the target host, test a feature branch
+nixos-rebuild-test boot <branch-name>
+nixos-rebuild-test switch <branch-name>
+
+# Additional arguments are passed through to nixos-rebuild
+nixos-rebuild-test boot my-feature --show-trace
+```
+
+When working on a feature branch that requires testing on a live host, suggest using this command instead of the full flake URL syntax.
+
 ### Flake Management
 
 ```bash
@@ -63,6 +78,8 @@ Legacy sops-nix is still present but only actively used by the `ca` host. Do not
 
 **Important:** Never commit directly to `master` unless the user explicitly asks for it. Always create a feature branch for changes.
 
+**Important:** Never amend commits to `master` unless the user explicitly asks for it. Amending rewrites history and causes issues for deployed configurations.
+
 When starting a new plan or task, the first step should typically be to create and checkout a new branch with an appropriate name (e.g., `git checkout -b dns-automation` or `git checkout -b fix-nginx-config`).
 
 ### Plan Management
@@ -177,6 +194,51 @@ node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
 node_filesystem_avail_bytes{mountpoint="/"}
 ```
 
+### Deploying to Test Hosts
+
+The **homelab-deploy** MCP server enables remote deployments to test-tier hosts via NATS messaging.
+
+**Available Tools:**
+
+- `deploy` - Deploy NixOS configuration to test-tier hosts
+- `list_hosts` - List available deployment targets
+
+**Deploy Parameters:**
+
+- `hostname` - Target a specific host (e.g., `vaulttest01`)
+- `role` - Deploy to all hosts with a specific role (e.g., `vault`)
+- `all` - Deploy to all test-tier hosts
+- `action` - nixos-rebuild action: `switch` (default), `boot`, `test`, `dry-activate`
+- `branch` - Git branch or commit to deploy (default: `master`)
+
+**Examples:**
+
+```
+# List available hosts
+list_hosts()
+
+# Deploy to a specific host
+deploy(hostname="vaulttest01", action="switch")
+
+# Dry-run deployment
+deploy(hostname="vaulttest01", action="dry-activate")
+
+# Deploy to all hosts with a role
+deploy(role="vault", action="switch")
+```
+
+**Note:** Only test-tier hosts with `homelab.deploy.enable = true` and the listener service running will respond to deployments.
+
+**Verifying Deployments:**
+
+After deploying, use the `nixos_flake_info` metric from nixos-exporter to verify the host is running the expected revision:
+
+```promql
+nixos_flake_info{instance=~"vaulttest01.*"}
+```
+
+The `current_rev` label contains the git commit hash of the deployed flake configuration.
+
 ## Architecture
 
 ### Directory Structure
@@ -402,6 +464,8 @@ This means:
 
 **Firewall**: Disabled on most hosts (trusted network). Enable selectively in host configuration if needed.
 
+**Shell scripts**: Use `pkgs.writeShellApplication` instead of `pkgs.writeShellScript` or `pkgs.writeShellScriptBin` for creating shell scripts. `writeShellApplication` provides automatic shellcheck validation, sets strict bash options (`set -euo pipefail`), and allows declaring `runtimeInputs` for dependencies. When referencing the executable path (e.g., in `ExecStart`), use `lib.getExe myScript` to get the proper `bin/` path.
+
 ### Monitoring Stack
 
 All hosts ship metrics and logs to `monitoring01`:

docs/plans/completed/nixos-exporter.md

@@ -0,0 +1,176 @@
+# NixOS Prometheus Exporter
+
+## Overview
+
+Build a generic Prometheus exporter for NixOS-specific metrics. This exporter should be useful for any NixOS deployment, not just our homelab.
+
+## Goal
+
+Provide visibility into NixOS system state that standard exporters don't cover:
+- Generation management (count, age, current vs booted)
+- Flake input freshness
+- Upgrade status
+
+## Metrics
+
+### Core Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `nixos_generation_count` | Number of system generations | Count entries in `/nix/var/nix/profiles/system-*` |
+| `nixos_current_generation` | Active generation number | Parse `readlink /run/current-system` |
+| `nixos_booted_generation` | Generation that was booted | Parse `/run/booted-system` |
+| `nixos_generation_age_seconds` | Age of current generation | File mtime of current system profile |
+| `nixos_config_mismatch` | 1 if booted != current, 0 otherwise | Compare symlink targets |
+
+### Flake Metrics (optional collector)
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `nixos_flake_input_age_seconds` | Age of each flake.lock input | Parse `lastModified` from flake.lock |
+| `nixos_flake_input_info` | Info gauge with rev label | Parse `rev` from flake.lock |
+
+Labels: `input` (e.g., "nixpkgs", "home-manager")
+
+### Future Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `nixos_upgrade_pending` | 1 if remote differs from local | Compare flake refs (expensive) |
+| `nixos_store_size_bytes` | Size of /nix/store | `du` or filesystem stats |
+| `nixos_store_path_count` | Number of store paths | Count entries |
+
+## Architecture
+
+Single binary with optional collectors enabled via config or flags.
+
+```
+nixos-exporter
+├── main.go
+├── collector/
+│   ├── generation.go    # Core generation metrics
+│   └── flake.go         # Flake input metrics
+└── config/
+    └── config.go
+```
+
+## Configuration
+
+```yaml
+listen_addr: ":9971"
+collectors:
+  generation:
+    enabled: true
+  flake:
+    enabled: false
+    lock_path: "/etc/nixos/flake.lock"  # or auto-detect from /run/current-system
+```
+
+Command-line alternative:
+```bash
+nixos-exporter --listen=:9971 --collector.flake --flake.lock-path=/etc/nixos/flake.lock
+```
+
+## NixOS Module
+
+```nix
+services.prometheus.exporters.nixos = {
+  enable = true;
+  port = 9971;
+  collectors = [ "generation" "flake" ];
+  flake.lockPath = "/etc/nixos/flake.lock";
+};
+```
+
+The module should integrate with nixpkgs' existing `services.prometheus.exporters.*` pattern.
+
+## Implementation
+
+### Language
+
+Go - mature prometheus client library, single static binary, easy cross-compilation.
+
+### Phase 1: Core
+1. Create git repository
+2. Implement generation collector (count, current, booted, age, mismatch)
+3. Basic HTTP server with `/metrics` endpoint
+4. NixOS module
+
+### Phase 2: Flake Collector
+1. Parse flake.lock JSON format
+2. Extract lastModified timestamps per input
+3. Add input labels
+
+### Phase 3: Packaging
+1. Add to nixpkgs or publish as flake
+2. Documentation
+3. Example Grafana dashboard
+
+## Example Output
+
+```
+# HELP nixos_generation_count Total number of system generations
+# TYPE nixos_generation_count gauge
+nixos_generation_count 47
+
+# HELP nixos_current_generation Currently active generation number
+# TYPE nixos_current_generation gauge
+nixos_current_generation 47
+
+# HELP nixos_booted_generation Generation that was booted
+# TYPE nixos_booted_generation gauge
+nixos_booted_generation 46
+
+# HELP nixos_generation_age_seconds Age of current generation in seconds
+# TYPE nixos_generation_age_seconds gauge
+nixos_generation_age_seconds 3600
+
+# HELP nixos_config_mismatch 1 if booted generation differs from current
+# TYPE nixos_config_mismatch gauge
+nixos_config_mismatch 1
+
+# HELP nixos_flake_input_age_seconds Age of flake input in seconds
+# TYPE nixos_flake_input_age_seconds gauge
+nixos_flake_input_age_seconds{input="nixpkgs"} 259200
+nixos_flake_input_age_seconds{input="home-manager"} 86400
+```
+
+## Alert Examples
+
+```yaml
+- alert: NixOSConfigStale
+  expr: nixos_generation_age_seconds > 7 * 24 * 3600
+  for: 1h
+  labels:
+    severity: warning
+  annotations:
+    summary: "NixOS config on {{ $labels.instance }} is over 7 days old"
+
+- alert: NixOSRebootRequired
+  expr: nixos_config_mismatch == 1
+  for: 24h
+  labels:
+    severity: info
+  annotations:
+    summary: "{{ $labels.instance }} needs reboot to apply config"
+
+- alert: NixpkgsInputStale
+  expr: nixos_flake_input_age_seconds{input="nixpkgs"} > 30 * 24 * 3600
+  for: 1d
+  labels:
+    severity: info
+  annotations:
+    summary: "nixpkgs input on {{ $labels.instance }} is over 30 days old"
+```
+
+## Open Questions
+
+- [ ] How to detect flake.lock path automatically? (check /run/current-system for flake info)
+- [ ] Should generation collector need root? (probably not, just reading symlinks)
+- [ ] Include in nixpkgs or distribute as standalone flake?
+
+## Notes
+
+- Port 9971 suggested (9970 reserved for homelab-exporter)
+- Keep scope focused on NixOS-specific metrics - don't duplicate node-exporter
+- Consider submitting to prometheus exporter registry once stable

docs/plans/homelab-exporter.md

@@ -0,0 +1,179 @@
+# Homelab Infrastructure Exporter
+
+## Overview
+
+Build a Prometheus exporter for metrics specific to our homelab infrastructure. Unlike the generic nixos-exporter, this covers services and patterns unique to our environment.
+
+## Current State
+
+### Existing Exporters
+- **node-exporter** (all hosts): System metrics
+- **systemd-exporter** (all hosts): Service restart counts, IP accounting
+- **labmon** (monitoring01): TLS certificate monitoring, step-ca health
+- **Service-specific**: unbound, postgres, nats, jellyfin, home-assistant, caddy, step-ca
+
+### Gaps
+- No visibility into Vault/OpenBao lease expiry
+- No ACME certificate expiry from internal CA
+- No Proxmox guest agent metrics from inside VMs
+
+## Metrics
+
+### Vault/OpenBao Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_vault_token_expiry_seconds` | Seconds until AppRole token expires | Token metadata or lease file |
+| `homelab_vault_token_renewable` | 1 if token is renewable | Token metadata |
+
+Labels: `role` (AppRole name)
+
+### ACME Certificate Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_acme_cert_expiry_seconds` | Seconds until certificate expires | Parse cert from `/var/lib/acme/*/cert.pem` |
+| `homelab_acme_cert_not_after` | Unix timestamp of cert expiry | Certificate NotAfter field |
+
+Labels: `domain`, `issuer`
+
+Note: labmon already monitors external TLS endpoints. This covers local ACME-managed certs.
+
+### Proxmox Guest Metrics (future)
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_proxmox_guest_info` | Info gauge with VM ID, name | QEMU guest agent |
+| `homelab_proxmox_guest_agent_running` | 1 if guest agent is responsive | Agent ping |
+
+### DNS Zone Metrics (future)
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_dns_zone_serial` | Current zone serial number | DNS AXFR or zone file |
+
+Labels: `zone`
+
+## Architecture
+
+Single binary with collectors enabled via config. Runs on hosts that need specific collectors.
+
+```
+homelab-exporter
+├── main.go
+├── collector/
+│   ├── vault.go     # Vault/OpenBao token metrics
+│   ├── acme.go      # ACME certificate metrics
+│   └── proxmox.go   # Proxmox guest agent (future)
+└── config/
+    └── config.go
+```
+
+## Configuration
+
+```yaml
+listen_addr: ":9970"
+collectors:
+  vault:
+    enabled: true
+    token_path: "/var/lib/vault/token"
+  acme:
+    enabled: true
+    cert_dirs:
+      - "/var/lib/acme"
+  proxmox:
+    enabled: false
+```
+
+## NixOS Module
+
+```nix
+services.homelab-exporter = {
+  enable = true;
+  port = 9970;
+  collectors = {
+    vault = {
+      enable = true;
+      tokenPath = "/var/lib/vault/token";
+    };
+    acme = {
+      enable = true;
+      certDirs = [ "/var/lib/acme" ];
+    };
+  };
+};
+
+# Auto-register scrape target
+homelab.monitoring.scrapeTargets = [{
+  job_name = "homelab-exporter";
+  port = 9970;
+}];
+```
+
+## Integration
+
+### Deployment
+
+Deploy on hosts that have relevant data:
+- **All hosts with ACME certs**: acme collector
+- **All hosts with Vault**: vault collector
+- **Proxmox VMs**: proxmox collector (when implemented)
+
+### Relationship with nixos-exporter
+
+These are complementary:
+- **nixos-exporter** (port 9971): Generic NixOS metrics, deploy everywhere
+- **homelab-exporter** (port 9970): Infrastructure-specific, deploy selectively
+
+Both can run on the same host if needed.
+
+## Implementation
+
+### Language
+
+Go - consistent with labmon and nixos-exporter.
+
+### Phase 1: Core + ACME
+1. Create git repository (git.t-juice.club/torjus/homelab-exporter)
+2. Implement ACME certificate collector
+3. HTTP server with `/metrics`
+4. NixOS module
+
+### Phase 2: Vault Collector
+1. Implement token expiry detection
+2. Handle missing/expired tokens gracefully
+
+### Phase 3: Dashboard
+1. Create Grafana dashboard for infrastructure health
+2. Add to existing monitoring service module
+
+## Alert Examples
+
+```yaml
+- alert: VaultTokenExpiringSoon
+  expr: homelab_vault_token_expiry_seconds < 3600
+  for: 5m
+  labels:
+    severity: warning
+  annotations:
+    summary: "Vault token on {{ $labels.instance }} expires in < 1 hour"
+
+- alert: ACMECertExpiringSoon
+  expr: homelab_acme_cert_expiry_seconds < 7 * 24 * 3600
+  for: 1h
+  labels:
+    severity: warning
+  annotations:
+    summary: "ACME cert {{ $labels.domain }} on {{ $labels.instance }} expires in < 7 days"
+```
+
+## Open Questions
+
+- [ ] How to read Vault token expiry without re-authenticating?
+- [ ] Should ACME collector also check key/cert match?
+
+## Notes
+
+- Port 9970 (labmon uses 9969, nixos-exporter will use 9971)
+- Keep infrastructure-specific logic here, generic NixOS stuff in nixos-exporter
+- Consider merging Proxmox metrics with pve-exporter if overlap is significant

docs/plans/long-term-metrics-storage.md

@@ -0,0 +1,122 @@
+# Long-Term Metrics Storage Options
+
+## Problem Statement
+
+Current Prometheus configuration retains metrics for 30 days (`retentionTime = "30d"`). Extending retention further raises disk usage concerns on the homelab hypervisor with limited local storage.
+
+Prometheus does not support downsampling - it stores all data at full resolution until the retention period expires, then deletes it entirely.
+
+## Current Configuration
+
+Location: `services/monitoring/prometheus.nix`
+
+- **Retention**: 30 days
+- **Scrape interval**: 15s
+- **Features**: Alertmanager, Pushgateway, auto-generated scrape configs from flake hosts
+- **Storage**: Local disk on monitoring01
+
+## Options Evaluated
+
+### Option 1: VictoriaMetrics
+
+VictoriaMetrics is a Prometheus-compatible TSDB with significantly better compression (5-10x smaller storage footprint).
+
+**NixOS Options Available:**
+- `services.victoriametrics.enable`
+- `services.victoriametrics.prometheusConfig` - accepts Prometheus scrape config format
+- `services.victoriametrics.retentionPeriod` - e.g., "6m" for 6 months
+- `services.vmagent` - dedicated scraping agent
+- `services.vmalert` - alerting rules evaluation
+
+**Pros:**
+- Simple migration - single service replacement
+- Same PromQL query language - Grafana dashboards work unchanged
+- Same scrape config format - existing auto-generated configs work as-is
+- 5-10x better compression means 30 days of Prometheus data could become 180+ days
+- Lightweight, single binary
+
+**Cons:**
+- No automatic downsampling (relies on compression alone)
+- Alerting requires switching to vmalert instead of Prometheus alertmanager integration
+- Would need to migrate existing data or start fresh
+
+**Migration Steps:**
+1. Replace `services.prometheus` with `services.victoriametrics`
+2. Move scrape configs to `prometheusConfig`
+3. Set up `services.vmalert` for alerting rules
+4. Update Grafana datasource to VictoriaMetrics port (8428)
+5. Keep Alertmanager for notification routing
+
+### Option 2: Thanos
+
+Thanos extends Prometheus with long-term storage and automatic downsampling by uploading data to object storage.
+
+**NixOS Options Available:**
+- `services.thanos.sidecar` - uploads Prometheus blocks to object storage
+- `services.thanos.compact` - compacts and downsamples data
+- `services.thanos.query` - unified query gateway
+- `services.thanos.query-frontend` - query caching and parallelization
+- `services.thanos.downsample` - dedicated downsampling service
+
+**Downsampling Behavior:**
+- Raw resolution kept for configurable period (default: indefinite)
+- 5-minute resolution created after 40 hours
+- 1-hour resolution created after 10 days
+
+**Retention Configuration (in compactor):**
+```nix
+services.thanos.compact = {
+  retention.resolution-raw = "30d";   # Keep raw for 30 days
+  retention.resolution-5m = "180d";   # Keep 5m samples for 6 months
+  retention.resolution-1h = "2y";     # Keep 1h samples for 2 years
+};
+```
+
+**Pros:**
+- True downsampling - older data uses progressively less storage
+- Keep metrics for years with minimal storage impact
+- Prometheus continues running unchanged
+- Existing Alertmanager integration preserved
+
+**Cons:**
+- Requires object storage (MinIO, S3, or local filesystem)
+- Multiple services to manage (sidecar, compactor, query)
+- More complex architecture
+- Additional infrastructure (MinIO) may be needed
+
+**Required Components:**
+1. Thanos Sidecar (runs alongside Prometheus)
+2. Object storage (MinIO or local filesystem)
+3. Thanos Compactor (handles downsampling)
+4. Thanos Query (provides unified query endpoint)
+
+**Migration Steps:**
+1. Deploy object storage (MinIO or configure filesystem backend)
+2. Add Thanos sidecar pointing to Prometheus data directory
+3. Add Thanos compactor with retention policies
+4. Add Thanos query gateway
+5. Update Grafana datasource to Thanos Query port (10902)
+
+## Comparison
+
+| Aspect | VictoriaMetrics | Thanos |
+|--------|-----------------|--------|
+| Complexity | Low (1 service) | Higher (3-4 services) |
+| Downsampling | No | Yes (automatic) |
+| Storage savings | 5-10x compression | Compression + downsampling |
+| Object storage required | No | Yes |
+| Migration effort | Minimal | Moderate |
+| Grafana changes | Change port only | Change port only |
+| Alerting changes | Need vmalert | Keep existing |
+
+## Recommendation
+
+**Start with VictoriaMetrics** for simplicity. The compression alone may provide 6+ months of retention in the same disk space currently used for 30 days.
+
+If multi-year retention with true downsampling becomes necessary, Thanos can be evaluated later. However, it requires deploying object storage infrastructure (MinIO) which adds operational complexity.
+
+## References
+
+- VictoriaMetrics docs: https://docs.victoriametrics.com/
+- Thanos docs: https://thanos.io/tip/thanos/getting-started.md/
+- NixOS options searched from nixpkgs revision e576e3c9 (NixOS 25.11)

docs/plans/nats-deploy-service.md

@@ -0,0 +1,371 @@
+# NATS-Based Deployment Service
+
+## Overview
+
+Create a message-based deployment system that allows triggering NixOS configuration updates on-demand, rather than waiting for the daily auto-upgrade timer. This enables faster iteration when testing changes and immediate fleet-wide deployments.
+
+## Goals
+
+1. **On-demand deployment** - Trigger config updates immediately via NATS message
+2. **Targeted deployment** - Deploy to specific hosts or all hosts
+3. **Branch/revision support** - Test feature branches before merging to master
+4. **MCP integration** - Allow Claude Code to trigger deployments during development
+
+## Current State
+
+- **Auto-upgrade**: All hosts run `nixos-upgrade.service` daily, pulling from master
+- **Manual testing**: `nixos-rebuild-test <action> <branch>` helper exists on all hosts
+- **NATS**: Running on nats1 with JetStream enabled, using NKey authentication
+- **Accounts**: ADMIN (system) and HOMELAB (user workloads with JetStream)
+
+## Architecture
+
+```
+┌─────────────┐                        ┌─────────────┐
+│  MCP Tool   │  deploy.test.>         │  Admin CLI  │  deploy.test.> + deploy.prod.>
+│  (claude)   │────────────┐     ┌─────│  (torjus)   │
+└─────────────┘            │     │     └─────────────┘
+                           ▼     ▼
+                      ┌──────────────┐
+                      │    nats1     │
+                      │  (authz)     │
+                      └──────┬───────┘
+                             │
+           ┌─────────────────┼─────────────────┐
+           │                 │                 │
+           ▼                 ▼                 ▼
+     ┌──────────┐      ┌──────────┐      ┌──────────┐
+     │ template1│      │   ns1    │      │   ha1    │
+     │ tier=test│      │ tier=prod│      │ tier=prod│
+     └──────────┘      └──────────┘      └──────────┘
+```
+
+## Repository Structure
+
+The project lives in a **separate repository** (e.g., `homelab-deploy`) containing:
+
+```
+homelab-deploy/
+├── flake.nix           # Nix flake with Go package + NixOS module
+├── go.mod
+├── go.sum
+├── cmd/
+│   └── homelab-deploy/
+│       └── main.go     # CLI entrypoint with subcommands
+├── internal/
+│   ├── listener/       # Listener mode logic
+│   ├── mcp/            # MCP server mode logic
+│   └── deploy/         # Shared deployment logic
+└── nixos/
+    └── module.nix      # NixOS module for listener service
+```
+
+This repo imports the flake as an input and uses the NixOS module.
+
+## Single Binary with Subcommands
+
+The `homelab-deploy` binary supports multiple modes:
+
+```bash
+# Run as listener on a host (systemd service)
+homelab-deploy listener --hostname ns1 --nats-url nats://nats1:4222
+
+# Run as MCP server (for Claude Code)
+homelab-deploy mcp --nats-url nats://nats1:4222
+
+# CLI commands for manual use
+homelab-deploy deploy ns1 --branch feature-x --action switch  # single host
+homelab-deploy deploy --tier test --all --action boot          # all test hosts
+homelab-deploy deploy --tier prod --all --action boot          # all prod hosts (admin only)
+homelab-deploy deploy --tier prod --role dns --action switch   # all prod dns hosts
+homelab-deploy status
+```
+
+## Components
+
+### Listener Mode
+
+A systemd service on each host that:
+- Subscribes to multiple subjects for targeted and group deployments
+- Validates incoming messages (revision, action)
+- Executes `nixos-rebuild` with specified parameters
+- Reports status back via NATS
+
+**Subject structure:**
+```
+deploy.<tier>.<hostname>      # specific host (e.g., deploy.prod.ns1)
+deploy.<tier>.all             # all hosts in tier (e.g., deploy.test.all)
+deploy.<tier>.role.<role>     # all hosts with role in tier (e.g., deploy.prod.role.dns)
+```
+
+**Listener subscriptions** (based on `homelab.host` config):
+- `deploy.<tier>.<hostname>` - direct messages to this host
+- `deploy.<tier>.all` - broadcast to all hosts in tier
+- `deploy.<tier>.role.<role>` - broadcast to hosts with matching role (if role is set)
+
+Example: ns1 with `tier=prod, role=dns` subscribes to:
+- `deploy.prod.ns1`
+- `deploy.prod.all`
+- `deploy.prod.role.dns`
+
+**NixOS module configuration:**
+```nix
+services.homelab-deploy.listener = {
+  enable = true;
+  timeout = 600;  # seconds, default 10 minutes
+};
+```
+
+The listener reads tier and role from `config.homelab.host` (see Host Metadata below).
+
+**Request message format:**
+```json
+{
+  "action": "switch" | "boot" | "test" | "dry-activate",
+  "revision": "master" | "feature-branch" | "abc123...",
+  "reply_to": "deploy.responses.<request-id>"
+}
+```
+
+**Response message format:**
+```json
+{
+  "status": "accepted" | "rejected" | "started" | "completed" | "failed",
+  "error": "invalid_revision" | "already_running" | "build_failed" | null,
+  "message": "human-readable details"
+}
+```
+
+**Request/Reply flow:**
+1. MCP/CLI sends deploy request with unique `reply_to` subject
+2. Listener validates request (e.g., `git ls-remote` to check revision exists)
+3. Listener sends immediate response:
+   - `{"status": "rejected", "error": "invalid_revision", "message": "branch 'foo' not found"}`, or
+   - `{"status": "started", "message": "starting nixos-rebuild switch"}`
+4. If started, listener runs nixos-rebuild
+5. Listener sends final response:
+   - `{"status": "completed", "message": "successfully switched to generation 42"}`, or
+   - `{"status": "failed", "error": "build_failed", "message": "nixos-rebuild exited with code 1"}`
+
+This provides immediate feedback on validation errors (bad revision, already running) without waiting for the build to fail.
+
+### MCP Mode
+
+Runs as an MCP server providing tools for Claude Code.
+
+**Tools:**
+| Tool | Description | Tier Access |
+|------|-------------|-------------|
+| `deploy` | Deploy to test hosts (individual, all, or by role) | test only |
+| `deploy_admin` | Deploy to any host (requires `--enable-admin` flag) | test + prod |
+| `deploy_status` | Check deployment status/history | n/a |
+| `list_hosts` | List available deployment targets | n/a |
+
+**CLI flags:**
+```bash
+# Default: only test-tier deployments available
+homelab-deploy mcp --nats-url nats://nats1:4222
+
+# Enable admin tool (requires admin NKey to be configured)
+homelab-deploy mcp --nats-url nats://nats1:4222 --enable-admin --admin-nkey-file /path/to/admin.nkey
+```
+
+**Security layers:**
+1. **MCP flag**: `deploy_admin` tool only exposed when `--enable-admin` is passed
+2. **NATS authz**: Even if tool is exposed, NATS rejects publishes without valid admin NKey
+3. **Claude Code permissions**: Can set `mcp__homelab-deploy__deploy_admin` to `ask` mode for confirmation popup
+
+By default, the MCP only loads test-tier credentials and exposes the `deploy` tool. Claude can:
+- Deploy to individual test hosts
+- Deploy to all test hosts at once (`deploy.test.all`)
+- Deploy to test hosts by role (`deploy.test.role.<role>`)
+
+### Tiered Permissions
+
+Authorization is enforced at the NATS layer using subject-based permissions. Different deployer credentials have different publish rights:
+
+**NATS user configuration (on nats1):**
+```nix
+accounts = {
+  HOMELAB = {
+    users = [
+      # MCP/Claude - test tier only
+      {
+        nkey = "UABC...";  # mcp-deployer
+        permissions = {
+          publish = [ "deploy.test.>" ];
+          subscribe = [ "deploy.responses.>" ];
+        };
+      }
+      # Admin - full access to all tiers
+      {
+        nkey = "UXYZ...";  # admin-deployer
+        permissions = {
+          publish = [ "deploy.test.>" "deploy.prod.>" ];
+          subscribe = [ "deploy.responses.>" ];
+        };
+      }
+      # Host listeners - subscribe to their tier, publish responses
+      {
+        nkey = "UDEF...";  # host-listener (one per host)
+        permissions = {
+          subscribe = [ "deploy.*.>" ];
+          publish = [ "deploy.responses.>" ];
+        };
+      }
+    ];
+  };
+};
+```
+
+**Host tier assignments** (via `homelab.host.tier`):
+| Tier | Hosts |
+|------|-------|
+| test | template1, nix-cache01, future test hosts |
+| prod | ns1, ns2, ha1, monitoring01, http-proxy, etc. |
+
+**Example deployment scenarios:**
+
+| Command | Subject | MCP | Admin |
+|---------|---------|-----|-------|
+| Deploy to ns1 | `deploy.prod.ns1` | ❌ | ✅ |
+| Deploy to template1 | `deploy.test.template1` | ✅ | ✅ |
+| Deploy to all test hosts | `deploy.test.all` | ✅ | ✅ |
+| Deploy to all prod hosts | `deploy.prod.all` | ❌ | ✅ |
+| Deploy to all DNS servers | `deploy.prod.role.dns` | ❌ | ✅ |
+
+All NKeys stored in Vault - MCP gets limited credentials, admin CLI gets full-access credentials.
+
+### Host Metadata
+
+Rather than defining `tier` in the listener config, use a central `homelab.host` module that provides host metadata for multiple consumers. This aligns with the approach proposed in `docs/plans/prometheus-scrape-target-labels.md`.
+
+**Status:** The `homelab.host` module is implemented in `modules/homelab/host.nix`.
+Hosts can be filtered by tier using `config.homelab.host.tier`.
+
+**Module definition (in `modules/homelab/host.nix`):**
+```nix
+homelab.host = {
+  tier = lib.mkOption {
+    type = lib.types.enum [ "test" "prod" ];
+    default = "prod";
+    description = "Deployment tier - controls which credentials can deploy to this host";
+  };
+
+  priority = lib.mkOption {
+    type = lib.types.enum [ "high" "low" ];
+    default = "high";
+    description = "Alerting priority - low priority hosts have relaxed thresholds";
+  };
+
+  role = lib.mkOption {
+    type = lib.types.nullOr lib.types.str;
+    default = null;
+    description = "Primary role of this host (dns, database, monitoring, etc.)";
+  };
+
+  labels = lib.mkOption {
+    type = lib.types.attrsOf lib.types.str;
+    default = { };
+    description = "Additional free-form labels";
+  };
+};
+```
+
+**Consumers:**
+- `homelab-deploy` listener reads `config.homelab.host.tier` for subject subscription
+- Prometheus scrape config reads `priority`, `role`, `labels` for target labels
+- Future services can consume the same metadata
+
+**Example host config:**
+```nix
+# hosts/nix-cache01/configuration.nix
+homelab.host = {
+  tier = "test";      # can be deployed by MCP
+  priority = "low";   # relaxed alerting thresholds
+  role = "build-host";
+};
+
+# hosts/ns1/configuration.nix
+homelab.host = {
+  tier = "prod";      # requires admin credentials
+  priority = "high";
+  role = "dns";
+  labels.dns_role = "primary";
+};
+```
+
+## Implementation Steps
+
+### Phase 1: Core Binary + Listener
+
+1. **Create homelab-deploy repository**
+   - Initialize Go module
+   - Set up flake.nix with Go package build
+
+2. **Implement listener mode**
+   - NATS subscription logic
+   - nixos-rebuild execution
+   - Status reporting via NATS reply
+
+3. **Create NixOS module**
+   - Systemd service definition
+   - Configuration options (hostname, NATS URL, NKey path)
+   - Vault secret integration for NKeys
+
+4. **Create `homelab.host` module** (in nixos-servers)
+   - Define `tier`, `priority`, `role`, `labels` options
+   - This module is shared with Prometheus label work (see `docs/plans/prometheus-scrape-target-labels.md`)
+
+5. **Integrate with nixos-servers**
+   - Add flake input for homelab-deploy
+   - Import listener module in `system/`
+   - Set `homelab.host.tier` per host (test vs prod)
+
+6. **Configure NATS tiered permissions**
+   - Add deployer users to nats1 config (mcp-deployer, admin-deployer)
+   - Set up subject ACLs per user (test-only vs full access)
+   - Add deployer NKeys to Vault
+   - Create Terraform resources for NKey secrets
+
+### Phase 2: MCP + CLI
+
+7. **Implement MCP mode**
+   - MCP server with deploy/status tools
+   - Request/reply pattern for deployment feedback
+
+8. **Implement CLI commands**
+   - `deploy` command for manual deployments
+   - `status` command to check deployment state
+
+9. **Configure Claude Code**
+   - Add MCP server to configuration
+   - Document usage
+
+### Phase 3: Enhancements
+
+10. Add deployment locking (prevent concurrent deploys)
+11. Prometheus metrics for deployment status
+
+## Security Considerations
+
+- **Privilege escalation**: Listener runs as root to execute nixos-rebuild
+- **Input validation**: Strictly validate revision format (branch name or commit hash)
+- **Rate limiting**: Prevent rapid-fire deployments
+- **Audit logging**: Log all deployment requests with source identity
+- **Network isolation**: NATS only accessible from internal network
+
+## Decisions
+
+All open questions have been resolved. See Notes section for decision rationale.
+
+## Notes
+
+- The existing `nixos-rebuild-test` helper provides a good reference for the rebuild logic
+- Uses NATS request/reply pattern for immediate validation feedback and completion status
+- Consider using NATS headers for metadata (request ID, timestamp)
+- **Timeout decision**: Metrics show no-change upgrades complete in 5-55 seconds. A 10-minute default provides ample headroom for actual updates with package downloads. Per-host override available for hosts with known longer build times.
+- **Rollback**: Not needed as a separate feature - deploy an older commit hash to effectively rollback.
+- **Offline hosts**: No message persistence - if host is offline, deploy fails. Daily auto-upgrade is the safety net. Avoids complexity of JetStream deduplication (host coming online and applying 10 queued updates instead of just the latest).
+- **Deploy history**: Use existing Loki - listener logs deployments to journald, queryable via Loki. No need for separate JetStream persistence.
+- **Naming**: `homelab-deploy` - ties it to the infrastructure rather than implementation details.

docs/plans/prometheus-scrape-target-labels.md

@@ -4,6 +4,8 @@
 
 Add support for custom per-host labels on Prometheus scrape targets, enabling alert rules to reference host metadata (priority, role) instead of hardcoding instance names.
 
+**Related:** This plan shares the `homelab.host` module with `docs/plans/nats-deploy-service.md`, which uses the same metadata for deployment tier assignment.
+
 ## Motivation
 
 Some hosts have workloads that make generic alert thresholds inappropriate. For example, `nix-cache01` regularly hits high CPU during builds, requiring a longer `for` duration on `high_cpu_load`. Currently this is handled by excluding specific instance names in PromQL expressions, which is brittle and doesn't scale.
@@ -32,24 +34,82 @@ Values: free-form string, e.g. `"dns"`, `"build-host"`, `"database"`, `"monitori
 
 Recommendation: start with a single primary role string. If multi-role matching becomes a real need, switch to separate boolean labels.
 
+### `dns_role`
+
+For DNS servers specifically, distinguish between primary and secondary resolvers. The secondary resolver (ns2) receives very little traffic and has a cold cache, making generic cache hit ratio alerts inappropriate.
+
+Values: `"primary"`, `"secondary"`
+
+Example use case: The `unbound_low_cache_hit_ratio` alert fires on ns2 because its cache hit ratio (~62%) is lower than ns1 (~90%). This is expected behavior since ns2 gets ~100x less traffic. With a `dns_role` label, the alert can either exclude secondaries or use different thresholds:
+
+```promql
+# Only alert on primary DNS
+unbound_cache_hit_ratio < 0.7 and on(instance) unbound_up{dns_role="primary"}
+
+# Or use different thresholds
+(unbound_cache_hit_ratio < 0.7 and on(instance) unbound_up{dns_role="primary"})
+or
+(unbound_cache_hit_ratio < 0.5 and on(instance) unbound_up{dns_role="secondary"})
+```
+
 ## Implementation
 
-### 1. Add `labels` option to `homelab.monitoring`
+This implementation uses a shared `homelab.host` module that provides host metadata for multiple consumers (Prometheus labels, deployment tiers, etc.). See also `docs/plans/nats-deploy-service.md` which uses the same module for deployment tier assignment.
 
-In `modules/homelab/monitoring.nix`, add:
+### 1. Create `homelab.host` module
+
+**Status:** Step 1 (Create `homelab.host` module) is complete. The module is in
+`modules/homelab/host.nix` with tier, priority, role, and labels options.
+
+Create `modules/homelab/host.nix` with shared host metadata options:
 
 ```nix
-labels = lib.mkOption {
-  type = lib.types.attrsOf lib.types.str;
-  default = { };
-  description = "Custom labels to attach to this host's scrape targets";
-};
+{ lib, ... }:
+{
+  options.homelab.host = {
+    tier = lib.mkOption {
+      type = lib.types.enum [ "test" "prod" ];
+      default = "prod";
+      description = "Deployment tier - controls which credentials can deploy to this host";
+    };
+
+    priority = lib.mkOption {
+      type = lib.types.enum [ "high" "low" ];
+      default = "high";
+      description = "Alerting priority - low priority hosts have relaxed thresholds";
+    };
+
+    role = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      description = "Primary role of this host (dns, database, monitoring, etc.)";
+    };
+
+    labels = lib.mkOption {
+      type = lib.types.attrsOf lib.types.str;
+      default = { };
+      description = "Additional free-form labels (e.g., dns_role = 'primary')";
+    };
+  };
+}
 ```
 
+Import this module in `modules/homelab/default.nix`.
+
 ### 2. Update `lib/monitoring.nix`
 
-- `extractHostMonitoring` should carry `labels` through in its return value.
-- `generateNodeExporterTargets` currently returns a flat list of target strings. It needs to return structured `static_configs` entries instead, grouping targets by their label sets:
+- `extractHostMonitoring` should also extract `homelab.host` values (priority, role, labels).
+- Build the combined label set from `homelab.host`:
+
+```nix
+# Combine structured options + free-form labels
+effectiveLabels =
+  (lib.optionalAttrs (host.priority != "high") { priority = host.priority; })
+  // (lib.optionalAttrs (host.role != null) { role = host.role; })
+  // host.labels;
+```
+
+- `generateNodeExporterTargets` returns structured `static_configs` entries, grouping targets by their label sets:
 
 ```nix
 # Before (flat list):
@@ -62,7 +122,7 @@ labels = lib.mkOption {
 ]
 ```
 
-This requires grouping hosts by their label attrset and producing one `static_configs` entry per unique label combination. Hosts with no custom labels get grouped together with no extra labels (preserving current behavior).
+This requires grouping hosts by their label attrset and producing one `static_configs` entry per unique label combination. Hosts with default values (priority=high, no role, no labels) get grouped together with no extra labels (preserving current behavior).
 
 ### 3. Update `services/monitoring/prometheus.nix`
 
@@ -76,17 +136,29 @@ static_configs = [{ targets = nodeExporterTargets; }];
 static_configs = nodeExporterTargets;
 ```
 
-### 4. Set labels on hosts
+### 4. Set metadata on hosts
 
-Example in `hosts/nix-cache01/configuration.nix` or the relevant service module:
+Example in `hosts/nix-cache01/configuration.nix`:
 
 ```nix
-homelab.monitoring.labels = {
-  priority = "low";
+homelab.host = {
+  tier = "test";       # can be deployed by MCP (used by homelab-deploy)
+  priority = "low";    # relaxed alerting thresholds
   role = "build-host";
 };
 ```
 
+Example in `hosts/ns1/configuration.nix`:
+
+```nix
+homelab.host = {
+  tier = "prod";
+  priority = "high";
+  role = "dns";
+  labels.dns_role = "primary";
+};
+```
+
 ### 5. Update alert rules
 
 After implementing labels, review and update `services/monitoring/rules.yml`:

flake.lock

@@ -21,6 +21,27 @@
         "url": "https://git.t-juice.club/torjus/alerttonotify"
       }
     },
+    "homelab-deploy": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unstable"
+        ]
+      },
+      "locked": {
+        "lastModified": 1770447502,
+        "narHash": "sha256-xH1PNyE3ydj4udhe1IpK8VQxBPZETGLuORZdSWYRmSU=",
+        "ref": "master",
+        "rev": "79db119d1ca6630023947ef0a65896cc3307c2ff",
+        "revCount": 22,
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/homelab-deploy"
+      },
+      "original": {
+        "ref": "master",
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/homelab-deploy"
+      }
+    },
     "labmon": {
       "inputs": {
         "nixpkgs": [
@@ -42,6 +63,26 @@
         "url": "https://git.t-juice.club/torjus/labmon"
       }
     },
+    "nixos-exporter": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unstable"
+        ]
+      },
+      "locked": {
+        "lastModified": 1770422522,
+        "narHash": "sha256-WmIFnquu4u58v8S2bOVWmknRwHn4x88CRfBFTzJ1inQ=",
+        "ref": "refs/heads/master",
+        "rev": "cf0ce858997af4d8dcc2ce10393ff393e17fc911",
+        "revCount": 11,
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/nixos-exporter"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/nixos-exporter"
+      }
+    },
     "nixpkgs": {
       "locked": {
         "lastModified": 1770136044,
@@ -60,11 +101,11 @@
     },
     "nixpkgs-unstable": {
       "locked": {
-        "lastModified": 1770181073,
-        "narHash": "sha256-ksTL7P9QC1WfZasNlaAdLOzqD8x5EPyods69YBqxSfk=",
+        "lastModified": 1770197578,
+        "narHash": "sha256-AYqlWrX09+HvGs8zM6ebZ1pwUqjkfpnv8mewYwAo+iM=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "bf922a59c5c9998a6584645f7d0de689512e444c",
+        "rev": "00c21e4c93d963c50d4c0c89bfa84ed6e0694df2",
         "type": "github"
       },
       "original": {
@@ -77,7 +118,9 @@
     "root": {
       "inputs": {
         "alerttonotify": "alerttonotify",
+        "homelab-deploy": "homelab-deploy",
         "labmon": "labmon",
+        "nixos-exporter": "nixos-exporter",
         "nixpkgs": "nixpkgs",
         "nixpkgs-unstable": "nixpkgs-unstable",
         "sops-nix": "sops-nix"

flake.nix

@@ -17,6 +17,14 @@
       url = "git+https://git.t-juice.club/torjus/labmon?ref=master";
       inputs.nixpkgs.follows = "nixpkgs-unstable";
     };
+    nixos-exporter = {
+      url = "git+https://git.t-juice.club/torjus/nixos-exporter";
+      inputs.nixpkgs.follows = "nixpkgs-unstable";
+    };
+    homelab-deploy = {
+      url = "git+https://git.t-juice.club/torjus/homelab-deploy?ref=master";
+      inputs.nixpkgs.follows = "nixpkgs-unstable";
+    };
   };
 
   outputs =
@@ -27,6 +35,8 @@
       sops-nix,
       alerttonotify,
       labmon,
+      nixos-exporter,
+      homelab-deploy,
       ...
     }@inputs:
     let
@@ -42,6 +52,20 @@
         alerttonotify.overlays.default
         labmon.overlays.default
       ];
+      # Common modules applied to all hosts
+      commonModules = [
+        (
+          { config, pkgs, ... }:
+          {
+            nixpkgs.overlays = commonOverlays;
+            system.configurationRevision = self.rev or self.dirtyRev or "dirty";
+          }
+        )
+        sops-nix.nixosModules.sops
+        nixos-exporter.nixosModules.default
+        homelab-deploy.nixosModules.default
+        ./modules/homelab
+      ];
       allSystems = [
         "x86_64-linux"
         "aarch64-linux"
@@ -58,15 +82,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ns1
-            sops-nix.nixosModules.sops
           ];
         };
         ns2 = nixpkgs.lib.nixosSystem {
@@ -74,15 +91,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ns2
-            sops-nix.nixosModules.sops
           ];
         };
         ha1 = nixpkgs.lib.nixosSystem {
@@ -90,15 +100,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ha1
-            sops-nix.nixosModules.sops
           ];
         };
         template1 = nixpkgs.lib.nixosSystem {
@@ -106,15 +109,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/template
-            sops-nix.nixosModules.sops
           ];
         };
         template2 = nixpkgs.lib.nixosSystem {
@@ -122,15 +118,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/template2
-            sops-nix.nixosModules.sops
           ];
         };
         http-proxy = nixpkgs.lib.nixosSystem {
@@ -138,15 +127,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/http-proxy
-            sops-nix.nixosModules.sops
           ];
         };
         ca = nixpkgs.lib.nixosSystem {
@@ -154,15 +136,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ca
-            sops-nix.nixosModules.sops
           ];
         };
         monitoring01 = nixpkgs.lib.nixosSystem {
@@ -170,15 +145,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/monitoring01
-            sops-nix.nixosModules.sops
             labmon.nixosModules.labmon
           ];
         };
@@ -187,15 +155,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/jelly01
-            sops-nix.nixosModules.sops
           ];
         };
         nix-cache01 = nixpkgs.lib.nixosSystem {
@@ -203,15 +164,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/nix-cache01
-            sops-nix.nixosModules.sops
           ];
         };
         pgdb1 = nixpkgs.lib.nixosSystem {
@@ -219,15 +173,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/pgdb1
-            sops-nix.nixosModules.sops
           ];
         };
         nats1 = nixpkgs.lib.nixosSystem {
@@ -235,31 +182,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/nats1
-            sops-nix.nixosModules.sops
-          ];
-        };
-        auth01 = nixpkgs.lib.nixosSystem {
-          inherit system;
-          specialArgs = {
-            inherit inputs self sops-nix;
-          };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
-            ./hosts/auth01
-            sops-nix.nixosModules.sops
           ];
         };
         testvm01 = nixpkgs.lib.nixosSystem {
@@ -267,15 +191,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/testvm01
-            sops-nix.nixosModules.sops
           ];
         };
         vault01 = nixpkgs.lib.nixosSystem {
@@ -283,33 +200,19 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = commonOverlays;
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/vault01
-            sops-nix.nixosModules.sops
           ];
         };
-      vaulttest01 = nixpkgs.lib.nixosSystem {
-        inherit system;
-        specialArgs = {
-          inherit inputs self sops-nix;
+        vaulttest01 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/vaulttest01
+          ];
         };
-        modules = [
-          (
-            { config, pkgs, ... }:
-            {
-              nixpkgs.overlays = commonOverlays;
-            }
-          )
-          ./hosts/vaulttest01
-          sops-nix.nixosModules.sops
-        ];
-      };
       };
       packages = forAllSystems (
         { pkgs }:
@@ -322,11 +225,12 @@
         { pkgs }:
         {
           default = pkgs.mkShell {
-            packages = with pkgs; [
-              ansible
-              opentofu
-              openbao
+            packages = [
+              pkgs.ansible
+              pkgs.opentofu
+              pkgs.openbao
               (pkgs.callPackage ./scripts/create-host { })
+              homelab-deploy.packages.${pkgs.system}.default
             ];
           };
         }

hosts/auth01/configuration.nix

@@ -1,67 +0,0 @@
-{
-  pkgs,
-  ...
-}:
-
-{
-  imports = [
-    ../template/hardware-configuration.nix
-
-    ../../system
-    ../../common/vm
-  ];
-
-  homelab.dns.cnames = [ "ldap" ];
-
-  nixpkgs.config.allowUnfree = true;
-  # Use the systemd-boot EFI boot loader.
-  boot.loader.grub = {
-    enable = true;
-    device = "/dev/sda";
-    configurationLimit = 3;
-  };
-
-  networking.hostName = "auth01";
-  networking.domain = "home.2rjus.net";
-  networking.useNetworkd = true;
-  networking.useDHCP = false;
-  services.resolved.enable = true;
-  networking.nameservers = [
-    "10.69.13.5"
-    "10.69.13.6"
-  ];
-
-  systemd.network.enable = true;
-  systemd.network.networks."ens18" = {
-    matchConfig.Name = "ens18";
-    address = [
-      "10.69.13.18/24"
-    ];
-    routes = [
-      { Gateway = "10.69.13.1"; }
-    ];
-    linkConfig.RequiredForOnline = "routable";
-  };
-  time.timeZone = "Europe/Oslo";
-
-  nix.settings.experimental-features = [
-    "nix-command"
-    "flakes"
-  ];
-  nix.settings.tarball-ttl = 0;
-  environment.systemPackages = with pkgs; [
-    vim
-    wget
-    git
-  ];
-
-  services.qemuGuest.enable = true;
-
-  # Open ports in the firewall.
-  # networking.firewall.allowedTCPPorts = [ ... ];
-  # networking.firewall.allowedUDPPorts = [ ... ];
-  # Or disable the firewall altogether.
-  networking.firewall.enable = false;
-
-  system.stateVersion = "23.11"; # Did you read the comment?
-}

hosts/auth01/default.nix

@@ -1,8 +0,0 @@
-{ ... }:
-{
-  imports = [
-    ./configuration.nix
-    ../../services/lldap
-    ../../services/authelia
-  ];
-}

hosts/ha1/configuration.nix

@@ -57,6 +57,7 @@
 
   # Vault secrets management
   vault.enable = true;
+  homelab.deploy.enable = true;
   vault.secrets.backup-helper = {
     secretPath = "shared/backup/password";
     extractKey = "password";
@@ -76,6 +77,7 @@
     timerConfig = {
       OnCalendar = "daily";
       Persistent = true;
+      RandomizedDelaySec = "2h";
     };
     pruneOpts = [
       "--keep-daily 7"

hosts/http-proxy/configuration.nix

@@ -21,8 +21,6 @@
     "prometheus"
     "alertmanager"
     "jelly"
-    "auth"
-    "lldap"
     "pyroscope"
     "pushgw"
   ];
@@ -63,6 +61,7 @@
     "flakes"
   ];
   vault.enable = true;
+  homelab.deploy.enable = true;
 
   nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [

hosts/jump/configuration.nix

@@ -8,6 +8,9 @@
     ];
 
   nixpkgs.config.allowUnfree = true;
+
+  homelab.host.role = "bastion";
+
   # Use the systemd-boot EFI boot loader.
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/sda";

hosts/monitoring01/configuration.nix

@@ -58,6 +58,7 @@
 
   # Vault secrets management
   vault.enable = true;
+  homelab.deploy.enable = true;
   vault.secrets.backup-helper = {
     secretPath = "shared/backup/password";
     extractKey = "password";
@@ -72,6 +73,7 @@
     timerConfig = {
       OnCalendar = "daily";
       Persistent = true;
+      RandomizedDelaySec = "2h";
     };
     pruneOpts = [
       "--keep-daily 7"
@@ -88,6 +90,7 @@
     timerConfig = {
       OnCalendar = "daily";
       Persistent = true;
+      RandomizedDelaySec = "2h";
     };
     pruneOpts = [
       "--keep-daily 7"

hosts/nix-cache01/configuration.nix

@@ -13,6 +13,8 @@
 
   homelab.dns.cnames = [ "nix-cache" "actions1" ];
 
+  homelab.host.role = "build-host";
+
   fileSystems."/nix" = {
     device = "/dev/disk/by-label/nixcache";
     fsType = "xfs";
@@ -53,6 +55,7 @@
     "flakes"
   ];
   vault.enable = true;
+  homelab.deploy.enable = true;
 
   nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [

hosts/ns1/configuration.nix

@@ -48,6 +48,12 @@
     "flakes"
   ];
   vault.enable = true;
+  homelab.deploy.enable = true;
+
+  homelab.host = {
+    role = "dns";
+    labels.dns_role = "primary";
+  };
 
   nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [

hosts/ns2/configuration.nix

@@ -48,6 +48,12 @@
     "flakes"
   ];
   vault.enable = true;
+  homelab.deploy.enable = true;
+
+  homelab.host = {
+    role = "dns";
+    labels.dns_role = "secondary";
+  };
 
   environment.systemPackages = with pkgs; [
     vim

hosts/template/configuration.nix

@@ -11,6 +11,11 @@
   # Template host - exclude from DNS zone generation
   homelab.dns.enable = false;
 
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+  };
+
 
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/sda";

hosts/template/scripts.nix

@@ -1,7 +1,9 @@
 { pkgs, ... }:
 let
-  prepare-host-script = pkgs.writeShellScriptBin "prepare-host.sh"
-    ''
+  prepare-host-script = pkgs.writeShellApplication {
+    name = "prepare-host.sh";
+    runtimeInputs = [ pkgs.age ];
+    text = ''
       echo "Removing machine-id"
       rm -f /etc/machine-id || true
 
@@ -24,8 +26,9 @@ let
       echo "Generate age key"
       rm -rf /var/lib/sops-nix || true
       mkdir -p /var/lib/sops-nix
-      ${pkgs.age}/bin/age-keygen -o /var/lib/sops-nix/key.txt
+      age-keygen -o /var/lib/sops-nix/key.txt
     '';
+  };
 in
 {
   environment.systemPackages = [ prepare-host-script ];

hosts/template2/configuration.nix

@@ -32,6 +32,11 @@
     datasource_list = [ "ConfigDrive" "NoCloud" ];
   };
 
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+  };
+
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";
   networking.hostName = "nixos-template2";

hosts/template2/scripts.nix

@@ -1,7 +1,9 @@
 { pkgs, ... }:
 let
-  prepare-host-script = pkgs.writeShellScriptBin "prepare-host.sh"
-    ''
+  prepare-host-script = pkgs.writeShellApplication {
+    name = "prepare-host.sh";
+    runtimeInputs = [ pkgs.age ];
+    text = ''
       echo "Removing machine-id"
       rm -f /etc/machine-id || true
 
@@ -24,8 +26,9 @@ let
       echo "Generate age key"
       rm -rf /var/lib/sops-nix || true
       mkdir -p /var/lib/sops-nix
-      ${pkgs.age}/bin/age-keygen -o /var/lib/sops-nix/key.txt
+      age-keygen -o /var/lib/sops-nix/key.txt
     '';
+  };
 in
 {
   environment.systemPackages = [ prepare-host-script ];

hosts/testvm01/configuration.nix

@@ -16,6 +16,11 @@
   # Test VM - exclude from DNS zone generation
   homelab.dns.enable = false;
 
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+  };
+
   nixpkgs.config.allowUnfree = true;
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";

hosts/vault01/configuration.nix

@@ -16,6 +16,8 @@
 
   homelab.dns.cnames = [ "vault" ];
 
+  homelab.host.role = "vault";
+
   nixpkgs.config.allowUnfree = true;
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";

hosts/vaulttest01/configuration.nix

@@ -5,6 +5,32 @@
   ...
 }:
 
+let
+  vault-test-script = pkgs.writeShellApplication {
+    name = "vault-test";
+    text = ''
+      echo "=== Vault Secret Test ==="
+      echo "Secret path: hosts/vaulttest01/test-service"
+
+      if [ -f /run/secrets/test-service/password ]; then
+        echo "✓ Password file exists"
+        echo "Password length: $(wc -c < /run/secrets/test-service/password)"
+      else
+        echo "✗ Password file missing!"
+        exit 1
+      fi
+
+      if [ -d /var/lib/vault/cache/test-service ]; then
+        echo "✓ Cache directory exists"
+      else
+        echo "✗ Cache directory missing!"
+        exit 1
+      fi
+
+      echo "Test successful!"
+    '';
+  };
+in
 {
   imports = [
     ../template2/hardware-configuration.nix
@@ -13,6 +39,12 @@
     ../../common/vm
   ];
 
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+    role = "vault";
+  };
+
   nixpkgs.config.allowUnfree = true;
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";
@@ -49,6 +81,7 @@
     vim
     wget
     git
+    htop # test deploy verification
   ];
 
   # Open ports in the firewall.
@@ -60,6 +93,7 @@
   # Testing config
   # Enable Vault secrets management
   vault.enable = true;
+  homelab.deploy.enable = true;
 
   # Define a test secret
   vault.secrets.test-service = {
@@ -79,27 +113,7 @@
       Type = "oneshot";
       RemainAfterExit = true;
 
-      ExecStart = pkgs.writeShellScript "vault-test" ''
-        echo "=== Vault Secret Test ==="
-        echo "Secret path: hosts/vaulttest01/test-service"
-
-        if [ -f /run/secrets/test-service/password ]; then
-          echo "✓ Password file exists"
-          echo "Password length: $(wc -c < /run/secrets/test-service/password)"
-        else
-          echo "✗ Password file missing!"
-          exit 1
-        fi
-
-        if [ -d /var/lib/vault/cache/test-service ]; then
-          echo "✓ Cache directory exists"
-        else
-          echo "✗ Cache directory missing!"
-          exit 1
-        fi
-
-        echo "Test successful!"
-      '';
+      ExecStart = lib.getExe vault-test-script;
 
       StandardOutput = "journal+console";
     };

modules/homelab/default.nix

@@ -1,7 +1,9 @@
 { ... }:
 {
   imports = [
+    ./deploy.nix
     ./dns.nix
+    ./host.nix
     ./monitoring.nix
   ];
 }

modules/homelab/deploy.nix

@@ -0,0 +1,16 @@
+{ config, lib, ... }:
+
+{
+  options.homelab.deploy = {
+    enable = lib.mkEnableOption "homelab-deploy listener for NATS-based deployments";
+  };
+
+  config = {
+    assertions = [
+      {
+        assertion = config.homelab.deploy.enable -> config.vault.enable;
+        message = "homelab.deploy.enable requires vault.enable to be true (needed for NKey secret)";
+      }
+    ];
+  };
+}

modules/homelab/host.nix

@@ -0,0 +1,28 @@
+{ lib, ... }:
+{
+  options.homelab.host = {
+    tier = lib.mkOption {
+      type = lib.types.enum [ "test" "prod" ];
+      default = "prod";
+      description = "Deployment tier - controls which credentials can deploy to this host";
+    };
+
+    priority = lib.mkOption {
+      type = lib.types.enum [ "high" "low" ];
+      default = "high";
+      description = "Alerting priority - low priority hosts have relaxed thresholds";
+    };
+
+    role = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      description = "Primary role of this host (dns, database, monitoring, etc.)";
+    };
+
+    labels = lib.mkOption {
+      type = lib.types.attrsOf lib.types.str;
+      default = { };
+      description = "Additional free-form labels (e.g., dns_role = 'primary')";
+    };
+  };
+}

scripts/create-host/templates/configuration.nix.j2

@@ -13,6 +13,11 @@
     ../../common/vm
   ];
 
+  # Host metadata (adjust as needed)
+  homelab.host = {
+    tier = "test";  # Start in test tier, move to prod after validation
+  };
+
   nixpkgs.config.allowUnfree = true;
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";

secrets/auth01/secrets.yaml

@@ -1,29 +0,0 @@
-authelia_ldap_password: ENC[AES256_GCM,data:x2UDMpqQKoRVSlDSmK5XiC9x4/WWzmjk7cwtFA70waAD7xYQfXEOV+AeX1LlFfj0qHYrhyn//TLsa+tJzb7HPEAfl8vYR4MdkVFOm5vjPWWoF5Ul8ZVn8+B1VJLbiXkexv0/hfXL8NMzEcp/pF4H0Yei7xaKezu9OPtGzKufHws=,iv:88RXaOj8Zy9fGeDLAE0ItY7TKCCzxn6F0+kU5+Zy/XU=,tag:yPdCJ9d139iO6J97thVVgA==,type:str]
-authelia_jwt_secret: ENC[AES256_GCM,data:9ZHkT2o5KZLmml95g8HZce8fNBmaWtRn+175Gaz0KhsndNl3zdgGq3hydRuoZuEgLVsherJImVmb5DQAZpv04lUEsDKCYeFNwAyYl4Go2jCp1fI53fdcRCKlNVZA37pMi4AYaCoe8vIl/cwPOOBDEwK5raOBnklCzVERoO0B8a0=,iv:9CTWCw0ImZR0OSrl2znbhpRHlzAxA5Cpcy98JeH9Z+Y=,tag:L+0xKqiwXTi7XiDYWA1Bcw==,type:str]
-authelia_storage_encryption_key_file: ENC[AES256_GCM,data:RfbcQK8+rrW/Krd2rbDfgo7YI2YvQKqpLuDtk5DZJNNhw4giBh5nFp/8LNeo8r39/oiJLYTe6FjTLBu72TZz2wWrJFsBqjwQ/3TfATQGdLUsaXXRDr88ezHLTiYvEHIHJhUS5qsr7VMwBam5e7YGWBe5sGZCE/nX41ijyPUjtOY=,iv:sayYcAC38cApAtL+cDhgGNjWaHn+furKRowKL6AmfdU=,tag:1IZpnlpvDWGLLpZyU9iJUw==,type:str]
-authelia_session_secret: ENC[AES256_GCM,data:4PaLv4RRA7/9Z8QzETXLwo3OctJ0mvzQkYmHsGGF97nq9QeB3eo0xj4FyuCbkJGGZ/huAyRgmFBTyscY3wgxoc4t+8BdlYcSbefEk1/xRFjmG8ooXLKhvGJ5c6t72KJRcqsEGTiC0l9CFJWQ2qYcjM4dPwG8z0tjUZ6j25Zfx4M=,iv:QORJkf0w6iyuRHM/xuql1s7K75Qa49ygq+lwHfrm9rk=,tag:/HZ/qI80fKjmuTRwIwmX8g==,type:str]
-lldap_user_pass: ENC[AES256_GCM,data:56gF7uqVQ+/J5/lY/N904Q==,iv:qtY1XhHs4WWA4kPY56NigPvX4OslO0koZepgdv947zg=,tag:UDmJs8FPXskp7rUS2Sxinw==,type:str]
-sops:
-    age:
-        - recipient: age1lznyk4ee7e7x8n92cq2n87kz9920473ks5u9jlhd3dczfzq4wamqept56u
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlc1dxK3FKU2ZGWTNGUmxZ
-            aWx1NngySjVHclJTd3hXejJRTmVHRExReHcwCk55c0xMbGcyTktySkJZdHRZbzhK
-            bEI3RzBHQkROTU1qWXBoU1RqTXppdVkKLS0tIHkwZ0QyNTMydWRqUlBtTEdhZ05r
-            YVpuT1JadnlyN1hqNnJxYzVPT3pXN1UKDCeIv0xv+5pcoDdtYc+rYjwi8SLrqWth
-            vdWepxmV2edajZRqcwFEC9weOZ1j2lh7Z3hR6RSN/+X3sFpqkpw+Yg==
-            -----END AGE ENCRYPTED FILE-----
-        - recipient: age16prza00sqzuhwwcyakj6z4hvwkruwkqpmmrsn94a5ucgpkelncdq2ldctk
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbU0wNmFLelRmNmJTRlho
-            dTEwVXZqUVI5NHZkb1QyNUZ4R0pLVFZWVDM4CkhVc00zY2FKaVdNRXdGVk1ranpG
-            MlRWWGJmd2FWeFE1dXU4WHVFL0FHZ3MKLS0tIGt2ZWlaOW5wNkJnQVkrTDZWTnY0
-            RW5HRjA3cERCUU1CVWZhck12SGhTRUkK6k/zQ87TIETYouRBby7ujtwgpqIPKKv+
-            2aLJW6lSWMVzL/f3ZrIeg12tJjHs3f44EXR6j3tfLfSKog2iL8Y57w==
-            -----END AGE ENCRYPTED FILE-----
-    lastmodified: "2025-12-06T10:03:56Z"
-    mac: ENC[AES256_GCM,data:SRNqx5n+xg/cNGiyze3CGKufox3IuXmOKLqNRDeJhBNMBHC1iYYCjRdHEVXsl7XSiYe51dSwjV0KrJa/SG1pRVkuyT+xyPrTjT2/DyXN7A/CESSAkBIwI7lkZmIf8DkxB3CELF1PgjIr1o2isxlBnkAnhEBTxQ7t8AzpcH7I5yU=,iv:P3FGQurZrL0ed5UuBPRFk11T0VRFtL6xI4iQ4LmYTec=,tag:8gQL08ojjIMyCl5E0Qs/Ww==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.11.0

secrets/secrets.yaml

@@ -7,110 +7,101 @@ sops:
         - recipient: age1lznyk4ee7e7x8n92cq2n87kz9920473ks5u9jlhd3dczfzq4wamqept56u
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0elpybDFQMmlXV21XaTBR
-            TGExNEVPa3N2VzBCRzJpN2lSVzNFN09CWGowCkFUbTA1MmtNelJZZHgwMHpJcEQ1
-            dXNmRy9yODBrU01FYXh4RkJ2MzJmMU0KLS0tIDZMWSthOHovVWhSQ1pSYmcrQXFh
-            R3JBaDM1R2VxcUI4OFhyRUFlZEMxNkkKxTb8QBnxBQ2zfbTEZuQ3QIv9bKwm2c0p
-            wWSxxSI2u3crC17Vb8yVX8p5tZuKxierxOuIVXLxxvU51ldIQquKPw==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBuWXhzQWFmeCt1R05jREcz
+            Ui9HZFN5dkxHNVE0RVJGZUJUa3hKK2sxdkhBCktYcGpLeGZIQzZIV3ZZWGs3YzF1
+            T09sUEhPWkRkOWZFWkltQXBlM1lQV1UKLS0tIERRSlRUYW5QeW9TVjJFSmorOWNI
+            ZytmaEhzMjVhRXI1S0hielF0NlBrMmcK4I1PtSf7tSvSIJxWBjTnfBCO8GEFHbuZ
+            BkZskr5fRnWUIs72ZOGoTAVSO5ZNiBglOZ8YChl4Vz1U7bvdOCt0bw==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1hz2lz4k050ru3shrk5j3zk3f8azxmrp54pktw5a7nzjml4saudesx6jsl0
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5Wk05REFwZSszWWlaZWJV
-            UFNzK3g0TXhGd1N4YjJpSUQvaFJCM21BT1FVCkd6d210cndtVVEyeUFhUXJvR0lM
-            N0p2aHExZlBibW1OTERiQ1JtZ29hbFUKLS0tIHVLYWtIZUFRUDBXK3BZYU9KdUlU
-            bXl0VnVZTEJ6clljeTVnVGxKOXhwYTgKUGw+3Ry03lsYOrM8zBT3Q0lGVFnaQ9Ca
-            nLWJEwZXrqTstBxVtcVO8EbQHIhs0FH1PnvmXZWDS7ADABXlSEjwYQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQcXM0RHlGcmZrYW4yNGZs
+            S1ZqQzVaYmQ4MGhGaTFMUVIwOTk5K0tZZjB3ClN0QkhVeHRrNXZHdmZWMzFBRnJ6
+            WTFtaWZyRmx2TitkOXkrVkFiYVd3RncKLS0tIExpeGUvY1VpODNDL2NCaUhtZkp0
+            cGNVZTI3UGxlNWdFWVZMd3FlS3pDR3cKBulaMeonV++pArXOg3ilgKnW/51IyT6Z
+            vH9HOJUix+ryEwDIcjv4aWx9pYDHthPFZUDC25kLYG91WrJFQOo2oA==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1w2q4gm2lrcgdzscq8du3ssyvk6qtzm4fcszc92z9ftclq23yyydqdga5um
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMUytmK2JmMnNPNVdpUE5u
-            RlhJS3JSdm1sSW1CUnVKcXo1STI5WkhsTncwCndua0dzam9VeEY3RnR2S0I4NXg4
-            a1dTNlZ0VmFpdmo1R1hoNzVrRzl4MWsKLS0tIDFvT2JwZWxJMFRwUkFUMFNyaHgy
-            a3hpSDQzaHN2M1JWTG82TU4wOGo4RkEKlF/YdB/l5WqPrWR+gHS4CDnQ2WLD0emV
-            ScxDCgHnFYdKkv4TTaVV6opcB5t7uJECqUqBNxTyvwBrN9+n6m7Edg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBabTdsZWxZQjV2TGx2YjNM
+            ZTgzWktqTjY0S0M3bFpNZXlDRDk5TSt3V2k0CjdWWTN0TlRlK1RpUm9xYW03MFFG
+            aWN4a3o4VUVnYzBDd2FrelUraWtrMTAKLS0tIE1vTGpKYkhzcWErWDRreml2QmE2
+            ZkNIWERKb1drdVR6MTBSTnVmdm51VEkKVNDYdyBSrUT7dUn6a4eF7ELQ2B2Pk6V9
+            Z5fbT75ibuyX1JO315/gl2P/FhxmlRW1K6e+04gQe2R/t/3H11Q7YQ==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1d2w5zece9647qwyq4vas9qyqegg96xwmg6c86440a6eg4uj6dd2qrq0w3l
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlcnNCZmNTRWdDUER3Tlpl
-            S0dMc25qTzRiYlBsWE05OWZGRUJhYnNUWGt3CkNZcGNQaGJDbWdrQUNNa1d0emhI
-            UmtkL2dBbEEzNFp5ZnVFeHV2dDR0QzgKLS0tIG0xVE1LQjBHUUx2bklFVy9lVXBu
-            NzRMb1dnSTU2MlRtVkhLdjVlalFQOUkKYMY2yykgH8Qgmw7xyPf8dYybBuiRxQwy
-            hh2tgikE/90asVQTmW9ioRMy/e4cKnJGi8irGXoK4rkM/+fOVMWQ7Q==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVSFhDOFRVbnZWbVlQaG5G
+            U0NWekU0NzI1SlpRN0NVS1hPN210MXY3Z244CmtFemR5OUpzdlBzMHBUV3g0SFFo
+            eUtqNThXZDJ2b01yVVVuOFdwQVo2Qm8KLS0tIHpXRWd3OEpPRkpaVDNDTEJLMWEv
+            ZlZtaFpBdzF0YXFmdjNkNUR3YkxBZU0KAub+HF/OBZQR9bx/SVadZcL6Ms+NQ7yq
+            21HCcDTWyWHbN4ymUrIYXci1A/0tTOrQL9Mkvaz7IJh4VdHLPZrwwA==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1gq8434ku0xekqmvnseeunv83e779cg03c06gwrusnymdsr3rpufqx6vr3m
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKSFI4bUJXOS9zV082Ykho
-            ZnFYazVyb2hheUVTb0k5czlqRDRIVXJSTjNzClZ6TndTRnRwQ0ZZUkFld2c2WFl4
-            N0l3UHB1SnN4YUx5YTM3bDkrdzFScG8KLS0tIE5jYmVmelcxZGxPRjBIV1dobHF5
-            d2QxRzlRaWZ2ZjB2UEwyNHQrTDNwZDAKyWp3vMfeE1/oT7hRcAdoxnZKPnZYRF5F
-            YrRBIGJdVaC6h9YwlzsQ3Ew3TRg65dq+h4xew/227ZY7Qg9uVuHk5Q==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBWkhBL1NTdjFDeEhQcEgv
+            Z3c3Z213L2ZhWGo0Qm5Zd1A1RTBDY3plUkh3CkNWV2ZtNWkrUjB0eWFzUlVtbHlk
+            WTdTQjN4eDIzY0c0dyt6ajVXZ0krd1UKLS0tIHB4aEJqTTRMenV3UkFkTGEySjQ2
+            YVM1a3ZPdUU4T244UU0rc3hVQ3NYczQK10wug4kTjsvv/iOPWi5WrVZMOYUq4/Mf
+            oXS4sikXeUsqH1T2LUBjVnUieSneQVn7puYZlN+cpDQ0XdK/RZ+91A==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1288993th0ge00reg4zqueyvmkrsvk829cs068eekjqfdprsrkeqql7mljk
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXR2xSd0pTd04wemhqZHNH
-            UVJ1ZjFEWG9OZGtQQUVNUnJBR2dLeXFNM0F3ClhkLzA3cWVTR01XZzNmaUgwdnlR
-            bEExTjluYXpIZmRvdURBdkFIY2VubTAKLS0tIGVsWmlPNCtWbWxMWFQ4Ky9jZVcr
-            VHhlNnV1cTlEd3U4YjV3UGlLYVRWVUEKhjbs9nRhu5s1SD3CJTDkW8s0koPvW6LY
-            jJlw8dPctC1bfWgzca3WxhuBIE14TWoxI2+ec9y6x8yYzdvIQhNIIg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYcEtHbjNWRkdodUxYdHRn
+            MDBMU08zWDlKa0Z4cHJvc28rZk5pUjhnMjE0CmdzRmVGWDlYQ052Wm1zWnlYSFV6
+            dURQK3JSbThxQlg3M2ZaL1hGRzVuL0UKLS0tIEI3UGZvbEpvRS9aR2J2Tnc1YmxZ
+            aUY5Q2MrdHNQWDJNaGt5MWx6MVRrRVEKRPxyAekGHFMKs0Z6spVDayBA4EtPk18e
+            jiFc97BGVtC5IoSu4icq3ZpKOdxymnkqKEt0YP/p/JTC+8MKvTJFQw==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1vpns76ykll8jgdlu3h05cur4ew2t3k7u03kxdg8y6ypfhsfhq9fqyurjey
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBULy91QnFLSmxrNlU1U0RV
-            Mnprc2dBVVRHMzdQTzhHL2d5ejB5cEYxSVZzClp4UXZNbWdJZk5LWnZlSVdEM0Vk
-            MEV3WmlLVlVsWXduSFpVQW9KU1d6WlEKLS0tIE8xYjRxY1ZySlZMbG5acm5RSU1Z
-            c2Y5aXJSMFJNcVp0YS96MGtMTEJHMEEKm2jRWDsdpMnDXPMOhA56Qld3yjlJe246
-            6Xbc4924WparHwPh8YmVKP3IYsrNYw2WxFmLZpDGVQmd5Tz1lD4s9w==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQL3ZMUkI1dUV1T2tTSHhn
+            SjhyQ3dKTytoaDBNcit1VHpwVGUzWVNpdjBnCklYZWtBYzBpcGxZSDBvM2tIZm9H
+            bTFjb1ZCaDkrOU1JODVBVTBTbmxFbmcKLS0tIGtGcS9kejZPZlhHRXI5QnI5Wm9Q
+            VjMxTDdWZEltWThKVDl0S24yWHJxZHcKgzH79zT2I7ZgyTbbbvIhLN/rEcfiomJH
+            oSZDFvPiXlhPgy8bRyyq3l47CVpWbUI2Y7DFXRuODpLUirt3K3TmCA==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1hchvlf3apn8g8jq2743pw53sd6v6ay6xu6lqk0qufrjeccan9vzsc7hdfq
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkUitINVFScFY5R2dKTWtC
-            ai83UmNVbzdWNTNMWUhHc2lRTW1ZVnVHdVc0CjlSVmVOc0FvOUVvZnVuQUVCells
-            eW9uc21sZ0dpTjQ4N2ZvbGsyYVo5dlUKLS0tIDdsSGdZcVZLbXowUzNsYTNlR3VP
-            N1JNQmhDVWdid0pHOEZxM1dBSmRrSjAKP9z3b9b1huO/iFxUVf34W4P/Xnok9It7
-            ENRMctqEmHIp3Je/p/fMWUArSznMpxm0ukmBb9bGn3NCRxG5sEs1lw==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPcm9zUm1XUkpLWm1Jb3Uw
+            RncveGozOW5SRThEM1Y4SFF5RDdxUEhZTUE4CjVESHE5R3JZK0krOXZDL0RHR0oy
+            Z3JKaEpydjRjeFFHck1ic2JTRU5yZTQKLS0tIGY2ck56eG95YnpDYlNqUDh5RVp1
+            U3dRYkNleUtsQU1LMWpDbitJbnRIem8K+27HRtZihG8+k7ZC33XVfuXDFjC1e8lA
+            kffmxp9kOEShZF3IKmAjVHFBiPXRyGk3fGPyQLmSMK2UOOfCy/a/qA==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1w029fksjv0edrff9p7s03tgk3axecdkppqymfpwfn2nu2gsqqefqc37sxq
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0K0xxVUkyNWJtekFBdW0r
-            YUNBaUlzbmdNbktIUDEzVVlhSUtJTENHRDNFCjJpRHgycGFQbkhTUHRFNGpsNlJU
-            L2puZkhwSlExb3pXTXZMNHFhL0pjZVkKLS0tIHgza01pZ2hzUDlITGlYYnVDTWNF
-            RkpIbUJMRlJ2ZXJPSHRUTlpZYUUxOG8KF27qYEyAyt8kN8H7mFO0wf8IkXH0NcWR
-            w7Y1Nea6yMXHhEIazONJsmAkmLvQA+j7RxcTUI0Ej8qIxnJ0ZtT6RQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTZHlldDdSOEhjTklCSXQr
+            U2pXajFwZnNqQzZOTzY5b3lkMzlyREhXRWo4CmxId2F6NkNqeHNCSWNrcUJIY0Nw
+            cGF6NXJaQnovK1FYSXQ2TkJSTFloTUEKLS0tIHRhWk5aZ0lDVkZaZEJobm9FTDNw
+            a29sZE1GL2ZQSk0vUEc1ZGhkUlpNRkEK9tfe7cNOznSKgxshd5Z6TQiNKp+XW6XH
+            VvPgMqMitgiDYnUPj10bYo3kqhd0xZH2IhLXMnZnqqQ0I23zfPiNaw==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1ha34qeksr4jeaecevqvv2afqem67eja2mvawlmrqsudch0e7fe7qtpsekv
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpenhpVHJDajBMaVExeHJD
-            NFhuM2x4Y2xzR2I2S1JybkJVd1pZWDhoUVY0CklEVDRRcFBGeFMrbUwrOVh5ZUt3
-            WW9DTDhMNWUvOFFEYnB1RFNUelg3TjAKLS0tIC9Ed3dVaTZRZjJSMHJIS0M5cmZ3
-            eTlyWlZIS1VxcHlpSnBBaG1aUTVtR1kKE4DLKal6eYRf4N9ni7vd7lUcEJKeaIBJ
-            AOQYspAD8NSNVc1QlVzClb9sipUxoCDLNOaKjlPLMkN0fOQbNmzhlQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5bk9NVjJNWmMxUGd3cXRx
+            amZ5SWJ3dHpHcnM4UHJxdmh6NnhFVmJQdldzCm95dHN3R21qSkE4Vm9VTnVPREp3
+            dUQyS1B4MWhhdmd3dk5LQ0htZEtpTWMKLS0tIGFaa3MxVExFYk1MY2loOFBvWm1o
+            L0NoRStkeW9VZVdpWlhteC8yTnRmMUkKMYjUdE1rGgVR29FnhJ5OEVjTB1Rh5Mtu
+            M/DvlhW3a7tZU8nDF3IgG2GE5xOXZMDO9QWGdB8zO2RJZAr3Q+YIlA==
             -----END AGE ENCRYPTED FILE-----
         - recipient: age1cxt8kwqzx35yuldazcc49q88qvgy9ajkz30xu0h37uw3ts97jagqgmn2ga
           enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGS1JKc092ZmRza0wydklU
-            NUhTVHJtbzBpU1NBb3ZIYXgzMnlLVXBCcFU0Ci9idmJWd2RUaGM2V0VqVjY3SjBW
-            dTZLNHVYUEhvOEx2QzJVN0RzL2RPOWMKLS0tIHlpV3RmR0F1b3BBK3hjWjFHb2pj
-            WnJkUVowU3M0L09CSmxmeFBkUGRvQ3cKDS24pnHugCvkMCbiXd0R4Rk5xqn9IWC6
-            CErAOoAITdfrhoci4SG6LZu28de+OrKnO3W4wWm4DioSQgn3mVRmdg==
-            -----END AGE ENCRYPTED FILE-----
-        - recipient: age16prza00sqzuhwwcyakj6z4hvwkruwkqpmmrsn94a5ucgpkelncdq2ldctk
-          enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSY25GWkVoMk9jaGJlL2lj
-            cjQ1QW9XTTJVanRiS28rbmNMNmVKVTRDblZNCnJZUTNMYWpQOHlEbHI0eXZZQS91
-            bjdsdDFxL2VOYUoyblZhNEp3UXVtTncKLS0tIFFlU3BReWpYaHRjM2hBUlFiR2V5
-            S0t2dFdScW9RY2t6Y0hYN0N3d2dwa3MKNB9nsg3t6T0QzwB0tKk5JMxNGVZXH1cr
-            DJ/D8lE9sSV43oFx19p2ckzHigtFJQeS/bKaiWIR972vaoYmpLetSg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBU0xYMnhqOE0wdXdleStF
+            THcrY2NBQzNoRHdYTXY3ZmM5YXRZZkQ4aUZnCm9ad0IxSWxYT1JBd2RseUdVT1pi
+            UXBuNzFxVlN0OWNTQU5BV2NiVEV0RUUKLS0tIGJHY0dzSDczUzcrV0RpTjE0czEy
+            cWZMNUNlTzBRcEV5MjlRV1BsWGhoaUUKGhYaH8I0oPCfrbs7HbQKVOF/99rg3HXv
+            RRTXUI71/ejKIuxehOvifClQc3nUW73bWkASFQ0guUvO4R+c0xOgUg==
             -----END AGE ENCRYPTED FILE-----
     lastmodified: "2025-02-11T21:18:22Z"
     mac: ENC[AES256_GCM,data:5//boMp1awc/2XAkSASSCuobpkxa0E6IKf3GR8xHpMoCD30FJsCwV7PgX3fR8OuLEhOJ7UguqMNQdNqG37RMacreuDmI1J8oCFKp+3M2j4kCbXaEo8bw7WAtyjUez+SAXKzZWYmBibH0KOy6jdt+v0fdgy5hMBT4IFDofYRsyD0=,iv:6pD+SLwncpmal/FR4U8It2njvaQfUzzpALBCxa0NyME=,tag:4QN8ZFjdqck5ZgulF+FtbA==,type:str]

services/authelia/default.nix

@@ -1,98 +0,0 @@
-{ config, ... }:
-{
-  homelab.monitoring.scrapeTargets = [{
-    job_name = "authelia";
-    port = 9959;
-  }];
-
-  sops.secrets.authelia_ldap_password = {
-    format = "yaml";
-    sopsFile = ../../secrets/auth01/secrets.yaml;
-    key = "authelia_ldap_password";
-    restartUnits = [ "authelia-auth.service" ];
-    owner = "authelia-auth";
-    group = "authelia-auth";
-  };
-  sops.secrets.authelia_jwt_secret = {
-    format = "yaml";
-    sopsFile = ../../secrets/auth01/secrets.yaml;
-    key = "authelia_jwt_secret";
-    restartUnits = [ "authelia-auth.service" ];
-    owner = "authelia-auth";
-    group = "authelia-auth";
-  };
-  sops.secrets.authelia_storage_encryption_key_file = {
-    format = "yaml";
-    key = "authelia_storage_encryption_key_file";
-    sopsFile = ../../secrets/auth01/secrets.yaml;
-    restartUnits = [ "authelia-auth.service" ];
-    owner = "authelia-auth";
-    group = "authelia-auth";
-  };
-  sops.secrets.authelia_session_secret = {
-    format = "yaml";
-    key = "authelia_session_secret";
-    sopsFile = ../../secrets/auth01/secrets.yaml;
-    restartUnits = [ "authelia-auth.service" ];
-    owner = "authelia-auth";
-    group = "authelia-auth";
-  };
-
-  services.authelia.instances."auth" = {
-    enable = true;
-    environmentVariables = {
-      AUTHELIA_AUTHENTICATION_BACKEND_LDAP_PASSWORD_FILE =
-        config.sops.secrets.authelia_ldap_password.path;
-      AUTHELIA_SESSION_SECRET_FILE = config.sops.secrets.authelia_session_secret.path;
-    };
-    secrets = {
-      jwtSecretFile = config.sops.secrets.authelia_jwt_secret.path;
-      storageEncryptionKeyFile = config.sops.secrets.authelia_storage_encryption_key_file.path;
-    };
-    settings = {
-      telemetry = {
-        metrics = {
-          enabled = true;
-          address = "tcp://0.0.0.0:9959";
-        };
-      };
-      access_control = {
-        default_policy = "two_factor";
-      };
-      session = {
-        # secret = "{{- fileContent \"${config.sops.secrets.authelia_session_secret.path}\" }}";
-        cookies = [
-          {
-            domain = "home.2rjus.net";
-            authelia_url = "https://auth.home.2rjus.net";
-            default_redirection_url = "https://dashboard.home.2rjus.net";
-            name = "authelia_session";
-            same_site = "lax";
-            inactivity = "1h";
-            expiration = "24h";
-            remember_me = "30d";
-          }
-        ];
-      };
-      notifier = {
-        filesystem.filename = "/var/lib/authelia-auth/notification.txt";
-      };
-      storage = {
-        local.path = "/var/lib/authelia-auth/db.sqlite3";
-      };
-      authentication_backend = {
-        password_reset = {
-          disable = false;
-        };
-        ldap = {
-          address = "ldap://127.0.0.1:3890";
-          implementation = "lldap";
-          timeout = "5s";
-          base_dn = "dc=home,dc=2rjus,dc=net";
-          user = "uid=authelia_ldap_user,ou=people,dc=home,dc=2rjus,dc=net";
-          # password = "{{- fileContent \"${config.sops.secrets.authelia_ldap_password.path}\" -}}";
-        };
-      };
-    };
-  };
-}

services/home-assistant/default.nix

@@ -78,15 +78,15 @@
         # Override battery calculation using voltage (mV): (voltage - 2100) / 9
         "0x54ef441000a547bd" = {
           friendly_name = "0x54ef441000a547bd";
-          homeassistant.sensor_battery.value_template = "{{ (((value_json.voltage | float) - 2100) / 9) | round(0) | int | min(100) | max(0) }}";
+          homeassistant.battery.value_template = "{{ (((value_json.voltage | float) - 2100) / 9) | round(0) | int | min(100) | max(0) }}";
         };
         "0x54ef441000a54d3c" = {
           friendly_name = "0x54ef441000a54d3c";
-          homeassistant.sensor_battery.value_template = "{{ (((value_json.voltage | float) - 2100) / 9) | round(0) | int | min(100) | max(0) }}";
+          homeassistant.battery.value_template = "{{ (((value_json.voltage | float) - 2100) / 9) | round(0) | int | min(100) | max(0) }}";
         };
         "0x54ef441000a564b6" = {
           friendly_name = "temp_server";
-          homeassistant.sensor_battery.value_template = "{{ (((value_json.voltage | float) - 2100) / 9) | round(0) | int | min(100) | max(0) }}";
+          homeassistant.battery.value_template = "{{ (((value_json.voltage | float) - 2100) / 9) | round(0) | int | min(100) | max(0) }}";
         };
 
         # Other sensors

services/http-proxy/proxy.nix

@@ -86,22 +86,6 @@
         }
         reverse_proxy http://jelly01.home.2rjus.net:8096
       }
-      lldap.home.2rjus.net {
-        log {
-          output file /var/log/caddy/auth.log {
-            mode 644
-          }
-        }
-        reverse_proxy http://auth01.home.2rjus.net:17170
-      }
-      auth.home.2rjus.net {
-        log {
-          output file /var/log/caddy/auth.log {
-            mode 644
-          }
-        }
-        reverse_proxy http://auth01.home.2rjus.net:9091
-      }
       pyroscope.home.2rjus.net {
         log {
           output file /var/log/caddy/pyroscope.log {

services/lldap/default.nix

@@ -1,38 +0,0 @@
-{ config, ... }:
-{
-  sops.secrets.lldap_user_pass = {
-    format = "yaml";
-    key = "lldap_user_pass";
-    sopsFile = ../../secrets/auth01/secrets.yaml;
-    restartUnits = [ "lldap.service" ];
-    group = "acme";
-    mode = "0440";
-  };
-
-  services.lldap = {
-    enable = true;
-    settings = {
-      ldap_base_dn = "dc=home,dc=2rjus,dc=net";
-      ldap_user_email = "admin@home.2rjus.net";
-      ldap_user_dn = "admin";
-      ldap_user_pass_file = config.sops.secrets.lldap_user_pass.path;
-      ldaps_options = {
-        enabled = true;
-        port = 6360;
-        cert_file = "/var/lib/acme/auth01.home.2rjus.net/cert.pem";
-        key_file = "/var/lib/acme/auth01.home.2rjus.net/key.pem";
-      };
-    };
-  };
-  systemd.services.lldap = {
-    serviceConfig = {
-      SupplementaryGroups = [ "acme" ];
-    };
-  };
-  security.acme.certs."auth01.home.2rjus.net" = {
-    listenHTTP = ":80";
-    reloadServices = [ "lldap" ];
-    extraDomainNames = [ "ldap.home.2rjus.net" ];
-    enableDebugLogs = true;
-  };
-}

services/monitoring/prometheus.nix

@@ -1,21 +1,78 @@
-{ self, lib, ... }:
+{ self, lib, pkgs, ... }:
 let
   monLib = import ../../lib/monitoring.nix { inherit lib; };
   externalTargets = import ./external-targets.nix;
 
   nodeExporterTargets = monLib.generateNodeExporterTargets self externalTargets;
   autoScrapeConfigs = monLib.generateScrapeConfigs self externalTargets;
+
+  # Script to fetch AppRole token for Prometheus to use when scraping OpenBao metrics
+  fetchOpenbaoToken = pkgs.writeShellApplication {
+    name = "fetch-openbao-token";
+    runtimeInputs = [ pkgs.curl pkgs.jq ];
+    text = ''
+      VAULT_ADDR="https://vault01.home.2rjus.net:8200"
+      APPROLE_DIR="/var/lib/vault/approle"
+      OUTPUT_FILE="/run/secrets/prometheus/openbao-token"
+
+      # Read AppRole credentials
+      if [ ! -f "$APPROLE_DIR/role-id" ] || [ ! -f "$APPROLE_DIR/secret-id" ]; then
+        echo "AppRole credentials not found at $APPROLE_DIR" >&2
+        exit 1
+      fi
+
+      ROLE_ID=$(cat "$APPROLE_DIR/role-id")
+      SECRET_ID=$(cat "$APPROLE_DIR/secret-id")
+
+      # Authenticate to Vault
+      AUTH_RESPONSE=$(curl -sf -k -X POST \
+        -d "{\"role_id\":\"$ROLE_ID\",\"secret_id\":\"$SECRET_ID\"}" \
+        "$VAULT_ADDR/v1/auth/approle/login")
+
+      # Extract token
+      VAULT_TOKEN=$(echo "$AUTH_RESPONSE" | jq -r '.auth.client_token')
+      if [ -z "$VAULT_TOKEN" ] || [ "$VAULT_TOKEN" = "null" ]; then
+        echo "Failed to extract Vault token from response" >&2
+        exit 1
+      fi
+
+      # Write token to file
+      mkdir -p "$(dirname "$OUTPUT_FILE")"
+      echo -n "$VAULT_TOKEN" > "$OUTPUT_FILE"
+      chown prometheus:prometheus "$OUTPUT_FILE"
+      chmod 0400 "$OUTPUT_FILE"
+
+      echo "Successfully fetched OpenBao token"
+    '';
+  };
 in
 {
-  # OpenBao token for scraping metrics
-  vault.secrets.openbao-token = {
-    secretPath = "hosts/monitoring01/openbao-token";
-    extractKey = "token";
-    outputDir = "/run/secrets/prometheus/openbao-token";
-    mode = "0400";
-    owner = "prometheus";
-    services = [ "prometheus" ];
+  # Systemd service to fetch AppRole token for Prometheus OpenBao scraping
+  # The token is used to authenticate when scraping /v1/sys/metrics
+  systemd.services.prometheus-openbao-token = {
+    description = "Fetch OpenBao token for Prometheus metrics scraping";
+    after = [ "network-online.target" ];
+    wants = [ "network-online.target" ];
+    before = [ "prometheus.service" ];
+    requiredBy = [ "prometheus.service" ];
+
+    serviceConfig = {
+      Type = "oneshot";
+      ExecStart = lib.getExe fetchOpenbaoToken;
+    };
   };
+
+  # Timer to periodically refresh the token (AppRole tokens have 1-hour TTL)
+  systemd.timers.prometheus-openbao-token = {
+    description = "Refresh OpenBao token for Prometheus";
+    wantedBy = [ "timers.target" ];
+    timerConfig = {
+      OnBootSec = "5min";
+      OnUnitActiveSec = "30min";
+      RandomizedDelaySec = "5min";
+    };
+  };
+
   services.prometheus = {
     enable = true;
     # syntax-only check because we use external credential files (e.g., openbao-token)

services/monitoring/rules.yml

@@ -75,12 +75,12 @@ groups:
           description: "Based on the last 6h trend, the root filesystem on {{ $labels.instance }} is predicted to run out of space within 24 hours."
       - alert: systemd_not_running
         expr: node_systemd_system_running == 0
-        for: 5m
+        for: 10m
         labels:
-          severity: critical
+          severity: warning
         annotations:
           summary: "Systemd not in running state on {{ $labels.instance }}"
-          description: "Systemd is not in running state on {{ $labels.instance }}. The system may be in a degraded state."
+          description: "Systemd is not in running state on {{ $labels.instance }}. The system may be in a degraded state. Note: brief degraded states during nixos-rebuild are normal."
       - alert: high_file_descriptors
         expr: node_filefd_allocated / node_filefd_maximum > 0.8
         for: 5m
@@ -227,13 +227,13 @@ groups:
           summary: "Mosquitto not running on {{ $labels.instance }}"
           description: "Mosquitto has been down on {{ $labels.instance }} more than 5 minutes."
       - alert: zigbee_sensor_stale
-        expr: (time() - hass_last_updated_time_seconds{entity=~"sensor\\.(0x[0-9a-f]+|temp_server)_temperature"}) > 3600
+        expr: (time() - hass_last_updated_time_seconds{entity=~"sensor\\.(0x[0-9a-f]+|temp_server)_temperature"}) > 7200
         for: 5m
         labels:
           severity: warning
         annotations:
           summary: "Zigbee sensor {{ $labels.friendly_name }} is stale"
-          description: "Zigbee temperature sensor {{ $labels.entity }} has not reported data for over 1 hour. The sensor may have a dead battery or connectivity issues."
+          description: "Zigbee temperature sensor {{ $labels.entity }} has not reported data for over 2 hours. The sensor may have a dead battery or connectivity issues."
   - name: smartctl_rules
     rules:
       - alert: smart_critical_warning
@@ -414,24 +414,6 @@ groups:
         annotations:
           summary: "PostgreSQL connection pool near exhaustion on {{ $labels.instance }}"
           description: "PostgreSQL is using over 80% of max_connections on {{ $labels.instance }}."
-  - name: auth_rules
-    rules:
-      - alert: authelia_down
-        expr: node_systemd_unit_state{instance="auth01.home.2rjus.net:9100", name="authelia-auth.service", state="active"} == 0
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          summary: "Authelia not running on {{ $labels.instance }}"
-          description: "Authelia has been down on {{ $labels.instance }} more than 5 minutes."
-      - alert: lldap_down
-        expr: node_systemd_unit_state{instance="auth01.home.2rjus.net:9100", name="lldap.service", state="active"} == 0
-        for: 5m
-        labels:
-          severity: critical
-        annotations:
-          summary: "LLDAP not running on {{ $labels.instance }}"
-          description: "LLDAP has been down on {{ $labels.instance }} more than 5 minutes."
   - name: jellyfin_rules
     rules:
       - alert: jellyfin_down

services/nats/default.nix

@@ -1,16 +1,18 @@
 { ... }:
 {
-  homelab.monitoring.scrapeTargets = [{
-    job_name = "nats";
-    port = 7777;
-  }];
+  homelab.monitoring.scrapeTargets = [
+    {
+      job_name = "nats";
+      port = 7777;
+    }
+  ];
 
   services.prometheus.exporters.nats = {
     enable = true;
     url = "http://localhost:8222";
     extraFlags = [
-      "-varz"    # General server info
-      "-connz"   # Connection info
+      "-varz" # General server info
+      "-connz" # Connection info
       "-jsz=all" # JetStream info
     ];
   };
@@ -38,6 +40,48 @@
             }
           ];
         };
+
+        DEPLOY = {
+          users = [
+            # Shared listener (all hosts use this)
+            {
+              nkey = "UCCZJSUGLCSLBBKHBPL4QA66TUMQUGIXGLIFTWDEH43MGWM3LDD232X4";
+              permissions = {
+                subscribe = [
+                  "deploy.test.>"
+                  "deploy.prod.>"
+                  "deploy.discover"
+                ];
+                publish = [
+                  "deploy.responses.>"
+                  "deploy.discover"
+                ];
+              };
+            }
+            # Test deployer (MCP without admin)
+            {
+              nkey = "UBR66CX2ZNY5XNVQF5VBG4WFAF54LSGUYCUNNCEYRILDQ4NXDAD2THZU";
+              permissions = {
+                publish = [
+                  "deploy.test.>"
+                  "deploy.discover"
+                ];
+                subscribe = [
+                  "deploy.responses.>"
+                  "deploy.discover"
+                ];
+              };
+            }
+            # Admin deployer (full access)
+            {
+              nkey = "UD2BFB7DLM67P5UUVCKBUJMCHADIZLGGVUNSRLZE2ZC66FW2XT44P73Y";
+              permissions = {
+                publish = [ "deploy.>" ];
+                subscribe = [ "deploy.>" ];
+              };
+            }
+          ];
+        };
       };
       system_account = "ADMIN";
       jetstream = {

system/default.nix

@@ -3,7 +3,9 @@
   imports = [
     ./acme.nix
     ./autoupgrade.nix
+    ./homelab-deploy.nix
     ./monitoring
+    ./motd.nix
     ./packages.nix
     ./nix.nix
     ./root-user.nix
@@ -11,7 +13,5 @@
     ./sops.nix
     ./sshd.nix
     ./vault-secrets.nix
-
-    ../modules/homelab
   ];
 }

system/homelab-deploy.nix

@@ -0,0 +1,37 @@
+{ config, lib, ... }:
+
+let
+  hostCfg = config.homelab.host;
+in
+{
+  config = lib.mkIf config.homelab.deploy.enable {
+    # Fetch listener NKey from Vault
+    vault.secrets.homelab-deploy-nkey = {
+      secretPath = "shared/homelab-deploy/listener-nkey";
+      extractKey = "nkey";
+    };
+
+    # Enable homelab-deploy listener
+    services.homelab-deploy.listener = {
+      enable = true;
+      tier = hostCfg.tier;
+      role = hostCfg.role;
+      natsUrl = "nats://nats1.home.2rjus.net:4222";
+      nkeyFile = "/run/secrets/homelab-deploy-nkey";
+      flakeUrl = "git+https://git.t-juice.club/torjus/nixos-servers.git";
+      metrics.enable = true;
+    };
+
+    # Expose metrics for Prometheus scraping
+    homelab.monitoring.scrapeTargets = [{
+      job_name = "homelab-deploy";
+      port = 9972;
+    }];
+
+    # Ensure listener starts after vault secret is available
+    systemd.services.homelab-deploy-listener = {
+      after = [ "vault-secret-homelab-deploy-nkey.service" ];
+      requires = [ "vault-secret-homelab-deploy-nkey.service" ];
+    };
+  };
+}

system/monitoring/metrics.nix

@@ -13,5 +13,26 @@
   services.prometheus.exporters.systemd = {
     enable = true;
     # Default port: 9558
+    extraFlags = [
+      "--systemd.collector.enable-restart-count"
+      "--systemd.collector.enable-ip-accounting"
+    ];
   };
+
+  services.prometheus.exporters.nixos = {
+    enable = true;
+    # Default port: 9971
+    flake = {
+      enable = true;
+      url = "git+https://git.t-juice.club/torjus/nixos-servers.git";
+    };
+  };
+
+  # Register nixos-exporter as a Prometheus scrape target
+  homelab.monitoring.scrapeTargets = [
+    {
+      job_name = "nixos-exporter";
+      port = 9971;
+    }
+  ];
 }

system/motd.nix

@@ -0,0 +1,28 @@
+{ config, lib, self, ... }:
+
+let
+  hostname = config.networking.hostName;
+  domain = config.networking.domain or "";
+  fqdn = if domain != "" then "${hostname}.${domain}" else hostname;
+
+  # Get commit hash (handles both clean and dirty trees)
+  shortRev = self.shortRev or self.dirtyShortRev or "unknown";
+
+  # Format timestamp from lastModified (Unix timestamp)
+  # lastModifiedDate is in format "YYYYMMDDHHMMSS"
+  dateStr = self.sourceInfo.lastModifiedDate or "unknown";
+  formattedDate = if dateStr != "unknown" then
+    "${builtins.substring 0 4 dateStr}-${builtins.substring 4 2 dateStr}-${builtins.substring 6 2 dateStr} ${builtins.substring 8 2 dateStr}:${builtins.substring 10 2 dateStr} UTC"
+  else
+    "unknown";
+
+  banner = ''
+    ####################################
+     ${fqdn}
+     Commit: ${shortRev} (${formattedDate})
+    ####################################
+  '';
+in
+{
+  users.motd = lib.mkDefault banner;
+}

system/nix.nix

@@ -1,5 +1,25 @@
-{ lib, ... }:
+{ lib, pkgs, ... }:
+let
+  nixos-rebuild-test = pkgs.writeShellApplication {
+    name = "nixos-rebuild-test";
+    runtimeInputs = [ pkgs.nixos-rebuild ];
+    text = ''
+      if [ $# -lt 2 ]; then
+        echo "Usage: nixos-rebuild-test <action> <branch>"
+        echo "Example: nixos-rebuild-test boot my-feature-branch"
+        exit 1
+      fi
+
+      action="$1"
+      branch="$2"
+      shift 2
+
+      exec nixos-rebuild "$action" --flake "git+https://git.t-juice.club/torjus/nixos-servers.git?ref=$branch" "$@"
+    '';
+  };
+in
 {
+  environment.systemPackages = [ nixos-rebuild-test ];
   nix = {
     gc = {
       automatic = true;

system/vault-secrets.nix

@@ -8,6 +8,48 @@ let
   # Import vault-fetch package
   vault-fetch = pkgs.callPackage ../scripts/vault-fetch { };
 
+  # Helper to create fetch scripts using writeShellApplication
+  mkFetchScript = name: secretCfg: pkgs.writeShellApplication {
+    name = "fetch-${name}";
+    runtimeInputs = [ vault-fetch ];
+    text = ''
+      # Set Vault environment variables
+      export VAULT_ADDR="${cfg.vaultAddress}"
+      export VAULT_SKIP_VERIFY="${if cfg.skipTlsVerify then "1" else "0"}"
+    '' + (if secretCfg.extractKey != null then ''
+      # Fetch to temporary directory, then extract single key
+      TMPDIR=$(mktemp -d)
+      trap 'rm -rf $TMPDIR' EXIT
+
+      vault-fetch \
+        "${secretCfg.secretPath}" \
+        "$TMPDIR" \
+        "${secretCfg.cacheDir}"
+
+      # Extract the specified key and write as a single file
+      if [ ! -f "$TMPDIR/${secretCfg.extractKey}" ]; then
+        echo "ERROR: Key '${secretCfg.extractKey}' not found in secret" >&2
+        exit 1
+      fi
+
+      # Ensure parent directory exists
+      mkdir -p "$(dirname "${secretCfg.outputDir}")"
+      cp "$TMPDIR/${secretCfg.extractKey}" "${secretCfg.outputDir}"
+      chown ${secretCfg.owner}:${secretCfg.group} "${secretCfg.outputDir}"
+      chmod ${secretCfg.mode} "${secretCfg.outputDir}"
+    '' else ''
+      # Fetch secret as directory of files
+      vault-fetch \
+        "${secretCfg.secretPath}" \
+        "${secretCfg.outputDir}" \
+        "${secretCfg.cacheDir}"
+
+      # Set ownership and permissions
+      chown -R ${secretCfg.owner}:${secretCfg.group} "${secretCfg.outputDir}"
+      chmod ${secretCfg.mode} "${secretCfg.outputDir}"/*
+    '');
+  };
+
   # Secret configuration type
   secretType = types.submodule ({ name, config, ... }: {
     options = {
@@ -162,44 +204,7 @@ in
           RemainAfterExit = true;
 
           # Fetch the secret
-          ExecStart = pkgs.writeShellScript "fetch-${name}" (''
-            set -euo pipefail
-
-            # Set Vault environment variables
-            export VAULT_ADDR="${cfg.vaultAddress}"
-            export VAULT_SKIP_VERIFY="${if cfg.skipTlsVerify then "1" else "0"}"
-          '' + (if secretCfg.extractKey != null then ''
-            # Fetch to temporary directory, then extract single key
-            TMPDIR=$(mktemp -d)
-            trap "rm -rf $TMPDIR" EXIT
-
-            ${vault-fetch}/bin/vault-fetch \
-              "${secretCfg.secretPath}" \
-              "$TMPDIR" \
-              "${secretCfg.cacheDir}"
-
-            # Extract the specified key and write as a single file
-            if [ ! -f "$TMPDIR/${secretCfg.extractKey}" ]; then
-              echo "ERROR: Key '${secretCfg.extractKey}' not found in secret" >&2
-              exit 1
-            fi
-
-            # Ensure parent directory exists
-            mkdir -p "$(dirname "${secretCfg.outputDir}")"
-            cp "$TMPDIR/${secretCfg.extractKey}" "${secretCfg.outputDir}"
-            chown ${secretCfg.owner}:${secretCfg.group} "${secretCfg.outputDir}"
-            chmod ${secretCfg.mode} "${secretCfg.outputDir}"
-          '' else ''
-            # Fetch secret as directory of files
-            ${vault-fetch}/bin/vault-fetch \
-              "${secretCfg.secretPath}" \
-              "${secretCfg.outputDir}" \
-              "${secretCfg.cacheDir}"
-
-            # Set ownership and permissions
-            chown -R ${secretCfg.owner}:${secretCfg.group} "${secretCfg.outputDir}"
-            chmod ${secretCfg.mode} "${secretCfg.outputDir}"/*
-          ''));
+          ExecStart = lib.getExe (mkFetchScript name secretCfg);
 
           # Logging
           StandardOutput = "journal";

terraform/vault/approle.tf

@@ -4,6 +4,17 @@ resource "vault_auth_backend" "approle" {
   path = "approle"
 }
 
+# Shared policy for homelab-deploy (all hosts need this for NATS-based deployments)
+resource "vault_policy" "homelab_deploy" {
+  name = "homelab-deploy"
+
+  policy = <<EOT
+path "secret/data/shared/homelab-deploy/*" {
+  capabilities = ["read", "list"]
+}
+EOT
+}
+
 # Define host access policies
 locals {
   host_policies = {
@@ -15,6 +26,7 @@ locals {
     #     "secret/data/services/grafana/*",
     #     "secret/data/shared/smtp/*"
     #   ]
+    #   extra_policies = ["some-other-policy"]  # Optional: additional policies
     # }
 
     # Example: ha1 host
@@ -38,6 +50,7 @@ locals {
         "secret/data/shared/backup/*",
         "secret/data/shared/nats/*",
       ]
+      extra_policies = ["prometheus-metrics"]
     }
 
     # Wave 1: hosts with no service secrets (only need vault.enable for future use)
@@ -87,6 +100,12 @@ locals {
         "secret/data/hosts/nix-cache01/*",
       ]
     }
+
+    "vaulttest01" = {
+      paths = [
+        "secret/data/hosts/vaulttest01/*",
+      ]
+    }
   }
 }
 
@@ -109,9 +128,12 @@ EOT
 resource "vault_approle_auth_backend_role" "hosts" {
   for_each = local.host_policies
 
-  backend        = vault_auth_backend.approle.path
-  role_name      = each.key
-  token_policies = ["${each.key}-policy"]
+  backend   = vault_auth_backend.approle.path
+  role_name = each.key
+  token_policies = concat(
+    ["${each.key}-policy", "homelab-deploy"],
+    lookup(each.value, "extra_policies", [])
+  )
 
   # Token configuration
   token_ttl     = 3600  # 1 hour

terraform/vault/policies.tf

@@ -1,21 +1,10 @@
 # Generic policies for services (not host-specific)
 
 resource "vault_policy" "prometheus_metrics" {
-  name = "prometheus-metrics"
+  name   = "prometheus-metrics"
   policy = <<EOT
 path "sys/metrics" {
   capabilities = ["read"]
 }
 EOT
 }
-
-# Long-lived token for Prometheus to scrape OpenBao metrics
-resource "vault_token" "prometheus_metrics" {
-  policies  = [vault_policy.prometheus_metrics.name]
-  ttl       = "8760h" # 1 year
-  renewable = true
-
-  metadata = {
-    purpose = "prometheus-metrics-scraping"
-  }
-}

terraform/vault/secrets.tf

@@ -93,11 +93,20 @@ locals {
       data          = { token = var.actions_token_1 }
     }
 
-    # Prometheus OpenBao token for scraping metrics
-    # Token is created by vault_token.prometheus_metrics in policies.tf
-    "hosts/monitoring01/openbao-token" = {
+    # Homelab-deploy NKeys
+    "shared/homelab-deploy/listener-nkey" = {
       auto_generate = false
-      data          = { token = vault_token.prometheus_metrics.client_token }
+      data          = { nkey = var.homelab_deploy_listener_nkey }
+    }
+
+    "shared/homelab-deploy/test-deployer-nkey" = {
+      auto_generate = false
+      data          = { nkey = var.homelab_deploy_test_deployer_nkey }
+    }
+
+    "shared/homelab-deploy/admin-deployer-nkey" = {
+      auto_generate = false
+      data          = { nkey = var.homelab_deploy_admin_deployer_nkey }
     }
   }
 }

terraform/vault/variables.tf

@@ -52,3 +52,24 @@ variable "actions_token_1" {
   sensitive   = true
 }
 
+variable "homelab_deploy_listener_nkey" {
+  description = "NKey seed for homelab-deploy listeners"
+  type        = string
+  default     = "PLACEHOLDER"
+  sensitive   = true
+}
+
+variable "homelab_deploy_test_deployer_nkey" {
+  description = "NKey seed for test-tier deployer"
+  type        = string
+  default     = "PLACEHOLDER"
+  sensitive   = true
+}
+
+variable "homelab_deploy_admin_deployer_nkey" {
+  description = "NKey seed for admin deployer"
+  type        = string
+  default     = "PLACEHOLDER"
+  sensitive   = true
+}
+