homelab-deploy: enable prometheus metrics

- Update homelab-deploy input to get metrics support
- Enable metrics endpoint on port 9972
- Add scrape target for prometheus auto-discovery

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

.claude/skills/observability/SKILL.md

@@ -0,0 +1,250 @@
+---
+name: observability
+description: Reference guide for exploring Prometheus metrics and Loki logs when troubleshooting homelab issues. Use when investigating system state, deployments, service health, or searching logs.
+---
+
+# Observability Troubleshooting Guide
+
+Quick reference for exploring Prometheus metrics and Loki logs to troubleshoot homelab issues.
+
+## Available Tools
+
+Use the `lab-monitoring` MCP server tools:
+
+**Metrics:**
+- `search_metrics` - Find metrics by name substring
+- `get_metric_metadata` - Get type/help for a specific metric
+- `query` - Execute PromQL queries
+- `list_targets` - Check scrape target health
+- `list_alerts` / `get_alert` - View active alerts
+
+**Logs:**
+- `query_logs` - Execute LogQL queries against Loki
+- `list_labels` - List available log labels
+- `list_label_values` - List values for a specific label
+
+---
+
+## Logs Reference
+
+### Label Reference
+
+Available labels for log queries:
+- `host` - Hostname (e.g., `ns1`, `monitoring01`, `ha1`)
+- `systemd_unit` - Systemd unit name (e.g., `nsd.service`, `nixos-upgrade.service`)
+- `job` - Either `systemd-journal` (most logs) or `varlog` (file-based logs)
+- `filename` - For `varlog` job, the log file path
+- `hostname` - Alternative to `host` for some streams
+
+### Log Format
+
+Journal logs are JSON-formatted. Key fields:
+- `MESSAGE` - The actual log message
+- `PRIORITY` - Syslog priority (6=info, 4=warning, 3=error)
+- `SYSLOG_IDENTIFIER` - Program name
+
+### Basic LogQL Queries
+
+**Logs from a specific service on a host:**
+```logql
+{host="ns1", systemd_unit="nsd.service"}
+```
+
+**All logs from a host:**
+```logql
+{host="monitoring01"}
+```
+
+**Logs from a service across all hosts:**
+```logql
+{systemd_unit="nixos-upgrade.service"}
+```
+
+**Substring matching (case-sensitive):**
+```logql
+{host="ha1"} |= "error"
+```
+
+**Exclude pattern:**
+```logql
+{host="ns1"} != "routine"
+```
+
+**Regex matching:**
+```logql
+{systemd_unit="prometheus.service"} |~ "scrape.*failed"
+```
+
+**File-based logs (caddy access logs, etc):**
+```logql
+{job="varlog", hostname="nix-cache01"}
+{job="varlog", filename="/var/log/caddy/nix-cache.log"}
+```
+
+### Time Ranges
+
+Default lookback is 1 hour. Use `start` parameter for older logs:
+- `start: "1h"` - Last hour (default)
+- `start: "24h"` - Last 24 hours
+- `start: "168h"` - Last 7 days
+
+### Common Services
+
+Useful systemd units for troubleshooting:
+- `nixos-upgrade.service` - Daily auto-upgrade logs
+- `nsd.service` - DNS server (ns1/ns2)
+- `prometheus.service` - Metrics collection
+- `loki.service` - Log aggregation
+- `caddy.service` - Reverse proxy
+- `home-assistant.service` - Home automation
+- `step-ca.service` - Internal CA
+- `openbao.service` - Secrets management
+- `sshd.service` - SSH daemon
+- `nix-gc.service` - Nix garbage collection
+
+### Extracting JSON Fields
+
+Parse JSON and filter on fields:
+```logql
+{systemd_unit="prometheus.service"} | json | PRIORITY="3"
+```
+
+---
+
+## Metrics Reference
+
+### Deployment & Version Status
+
+Check which NixOS revision hosts are running:
+
+```promql
+nixos_flake_info
+```
+
+Labels:
+- `current_rev` - Git commit of the running NixOS configuration
+- `remote_rev` - Latest commit on the remote repository
+- `nixpkgs_rev` - Nixpkgs revision used to build the system
+- `nixos_version` - Full NixOS version string (e.g., `25.11.20260203.e576e3c`)
+
+Check if hosts are behind on updates:
+
+```promql
+nixos_flake_revision_behind == 1
+```
+
+View flake input versions:
+
+```promql
+nixos_flake_input_info
+```
+
+Labels: `input` (name), `rev` (revision), `type` (git/github)
+
+Check flake input age:
+
+```promql
+nixos_flake_input_age_seconds / 86400
+```
+
+Returns age in days for each flake input.
+
+### System Health
+
+Basic host availability:
+
+```promql
+up{job="node-exporter"}
+```
+
+CPU usage by host:
+
+```promql
+100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
+```
+
+Memory usage:
+
+```promql
+1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)
+```
+
+Disk space (root filesystem):
+
+```promql
+node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}
+```
+
+### Service-Specific Metrics
+
+Common job names:
+- `node-exporter` - System metrics (all hosts)
+- `nixos-exporter` - NixOS version/generation metrics
+- `caddy` - Reverse proxy metrics
+- `prometheus` / `loki` / `grafana` - Monitoring stack
+- `home-assistant` - Home automation
+- `step-ca` - Internal CA
+
+### Instance Label Format
+
+The `instance` label uses FQDN format:
+
+```
+<hostname>.home.2rjus.net:<port>
+```
+
+Example queries filtering by host:
+
+```promql
+up{instance=~"monitoring01.*"}
+node_load1{instance=~"ns1.*"}
+```
+
+---
+
+## Troubleshooting Workflows
+
+### Check Deployment Status Across Fleet
+
+1. Query `nixos_flake_info` to see all hosts' current revisions
+2. Check `nixos_flake_revision_behind` for hosts needing updates
+3. Look at upgrade logs: `{systemd_unit="nixos-upgrade.service"}` with `start: "24h"`
+
+### Investigate Service Issues
+
+1. Check `up{job="<service>"}` for scrape failures
+2. Use `list_targets` to see target health details
+3. Query service logs: `{host="<host>", systemd_unit="<service>.service"}`
+4. Search for errors: `{host="<host>"} |= "error"`
+5. Check `list_alerts` for related alerts
+
+### After Deploying Changes
+
+1. Verify `current_rev` updated in `nixos_flake_info`
+2. Confirm `nixos_flake_revision_behind == 0`
+3. Check service logs for startup issues
+4. Check service metrics are being scraped
+
+### Debug SSH/Access Issues
+
+```logql
+{host="<host>", systemd_unit="sshd.service"}
+```
+
+### Check Recent Upgrades
+
+```logql
+{systemd_unit="nixos-upgrade.service"}
+```
+
+With `start: "24h"` to see last 24 hours of upgrades across all hosts.
+
+---
+
+## Notes
+
+- Default scrape interval is 15s for most metrics targets
+- Default log lookback is 1h - use `start` parameter for older logs
+- Use `rate()` for counter metrics, direct queries for gauges
+- The `instance` label includes the port, use regex matching (`=~`) for hostname-only filters
+- Log `MESSAGE` field contains the actual log content in JSON format

.claude/skills/quick-plan/SKILL.md

@@ -0,0 +1,89 @@
+---
+name: quick-plan
+description: Create a planning document for a future homelab project. Use when the user wants to document ideas for future work without implementing immediately.
+argument-hint: [topic or feature to plan]
+---
+
+# Quick Plan Generator
+
+Create a planning document for a future homelab infrastructure project. Plans are for documenting ideas and approaches that will be implemented later, not immediately.
+
+## Input
+
+The user provides: $ARGUMENTS
+
+## Process
+
+1. **Understand the topic**: Research the codebase to understand:
+   - Current state of related systems
+   - Existing patterns and conventions
+   - Relevant NixOS options or packages
+   - Any constraints or dependencies
+
+2. **Evaluate options**: If there are multiple approaches, research and compare them with pros/cons.
+
+3. **Draft the plan**: Create a markdown document following the structure below.
+
+4. **Save the plan**: Write to `docs/plans/<topic-slug>.md` using a kebab-case filename derived from the topic.
+
+## Plan Structure
+
+Use these sections as appropriate (not all plans need every section):
+
+```markdown
+# Title
+
+## Overview/Goal
+Brief description of what this plan addresses and why.
+
+## Current State
+What exists today that's relevant to this plan.
+
+## Options Evaluated (if multiple approaches)
+For each option:
+- **Option Name**
+  - **Pros:** bullet points
+  - **Cons:** bullet points
+  - **Verdict:** brief assessment
+
+Or use a comparison table for structured evaluation.
+
+## Recommendation/Decision
+What approach is recommended and why. Include rationale.
+
+## Implementation Steps
+Numbered phases or steps. Be specific but not overly detailed.
+Can use sub-sections for major phases.
+
+## Open Questions
+Things still to be determined. Use checkbox format:
+- [ ] Question 1?
+- [ ] Question 2?
+
+## Notes (optional)
+Additional context, caveats, or references.
+```
+
+## Style Guidelines
+
+- **Concise**: Use bullet points, avoid verbose paragraphs
+- **Technical but accessible**: Include NixOS config snippets when relevant
+- **Future-oriented**: These are plans, not specifications
+- **Acknowledge uncertainty**: Use "Open Questions" for unresolved decisions
+- **Reference existing patterns**: Mention how this fits with existing infrastructure
+- **Tables for comparisons**: Use markdown tables when comparing options
+- **Practical focus**: Emphasize what needs to happen, not theory
+
+## Examples of Good Plans
+
+Reference these existing plans for style guidance:
+- `docs/plans/auth-system-replacement.md` - Good option evaluation with table
+- `docs/plans/truenas-migration.md` - Good decision documentation with rationale
+- `docs/plans/remote-access.md` - Good multi-option comparison
+- `docs/plans/prometheus-scrape-target-labels.md` - Good implementation detail level
+
+## After Creating the Plan
+
+1. Tell the user the plan was saved to `docs/plans/<filename>.md`
+2. Summarize the key points
+3. Ask if they want any adjustments before committing

.gitignore

@@ -1 +1,22 @@
 .direnv/
+result
+result-*
+
+# Terraform/OpenTofu
+terraform/.terraform/
+terraform/.terraform.lock.hcl
+terraform/*.tfstate
+terraform/*.tfstate.*
+terraform/terraform.tfvars
+terraform/*.auto.tfvars
+terraform/crash.log
+terraform/crash.*.log
+
+terraform/vault/.terraform/
+terraform/vault/.terraform.lock.hcl
+terraform/vault/*.tfstate
+terraform/vault/*.tfstate.*
+terraform/vault/terraform.tfvars
+terraform/vault/*.auto.tfvars
+terraform/vault/crash.log
+terraform/vault/crash.*.log

.mcp.json

@@ -0,0 +1,39 @@
+{
+  "mcpServers": {
+    "nixpkgs-options": {
+      "command": "nix",
+      "args": ["run", "git+https://git.t-juice.club/torjus/labmcp#nixpkgs-search", "--", "options", "serve"],
+      "env": {
+        "NIXPKGS_SEARCH_DATABASE": "sqlite:///run/user/1000/labmcp/nixpkgs-search.db"
+      }
+    },
+    "nixpkgs-packages": {
+      "command": "nix",
+      "args": ["run", "git+https://git.t-juice.club/torjus/labmcp#nixpkgs-search", "--", "packages", "serve"],
+      "env": {
+        "NIXPKGS_SEARCH_DATABASE": "sqlite:///run/user/1000/labmcp/nixpkgs-search.db"
+      }
+    },
+    "lab-monitoring": {
+      "command": "nix",
+      "args": ["run", "git+https://git.t-juice.club/torjus/labmcp#lab-monitoring", "--", "serve", "--enable-silences"],
+      "env": {
+        "PROMETHEUS_URL": "https://prometheus.home.2rjus.net",
+        "ALERTMANAGER_URL": "https://alertmanager.home.2rjus.net",
+        "LOKI_URL": "http://monitoring01.home.2rjus.net:3100"
+      }
+    },
+    "homelab-deploy": {
+      "command": "nix",
+      "args": [
+        "run",
+        "git+https://git.t-juice.club/torjus/homelab-deploy",
+        "--",
+        "mcp",
+        "--nats-url", "nats://nats1.home.2rjus.net:4222",
+        "--nkey-file", "/home/torjus/.config/homelab-deploy/test-deployer.nkey"
+      ]
+    }
+  }
+}
+

.sops.yaml

@@ -2,13 +2,14 @@ keys:
   - &admin_torjus age1lznyk4ee7e7x8n92cq2n87kz9920473ks5u9jlhd3dczfzq4wamqept56u
   - &server_ns1 age1hz2lz4k050ru3shrk5j3zk3f8azxmrp54pktw5a7nzjml4saudesx6jsl0
   - &server_ns2 age1w2q4gm2lrcgdzscq8du3ssyvk6qtzm4fcszc92z9ftclq23yyydqdga5um
-  - &server_ns3 age1snmhmpavqy7xddmw4nuny0u4xusqmnqxqarjmghkm5zaluff84eq5xatrd
-  - &server_ns4 age12a3nyvjs8jrwmpkf3tgawel3nwcklwsr35ktmytnvhpawqwzrsfqpgcy0q
   - &server_ha1 age1d2w5zece9647qwyq4vas9qyqegg96xwmg6c86440a6eg4uj6dd2qrq0w3l
-  - &server_nixos-test1 age1gcyfkxh4fq5zdp0dh484aj82ksz66wrly7qhnpv0r0p576sn9ekse8e9ju
-  - &server_inc1 age1g5luz2rtel3surgzuh62rkvtey7lythrvfenyq954vmeyfpxjqkqdj3wt8
   - &server_http-proxy age1gq8434ku0xekqmvnseeunv83e779cg03c06gwrusnymdsr3rpufqx6vr3m
   - &server_ca age1288993th0ge00reg4zqueyvmkrsvk829cs068eekjqfdprsrkeqql7mljk
+  - &server_monitoring01 age1vpns76ykll8jgdlu3h05cur4ew2t3k7u03kxdg8y6ypfhsfhq9fqyurjey
+  - &server_jelly01 age1hchvlf3apn8g8jq2743pw53sd6v6ay6xu6lqk0qufrjeccan9vzsc7hdfq
+  - &server_nix-cache01 age1w029fksjv0edrff9p7s03tgk3axecdkppqymfpwfn2nu2gsqqefqc37sxq
+  - &server_pgdb1 age1ha34qeksr4jeaecevqvv2afqem67eja2mvawlmrqsudch0e7fe7qtpsekv
+  - &server_nats1 age1cxt8kwqzx35yuldazcc49q88qvgy9ajkz30xu0h37uw3ts97jagqgmn2ga
 creation_rules:
   - path_regex: secrets/[^/]+\.(yaml|json|env|ini)
     key_groups:
@@ -16,25 +17,36 @@ creation_rules:
         - *admin_torjus
         - *server_ns1
         - *server_ns2
-        - *server_ns3
-        - *server_ns4
         - *server_ha1
-        - *server_nixos-test1
-        - *server_inc1
         - *server_http-proxy
         - *server_ca
-  - path_regex: secrets/ns3/[^/]+\.(yaml|json|env|ini)
-    key_groups:
-      - age:
-        - *admin_torjus
-        - *server_ns3
+        - *server_monitoring01
+        - *server_jelly01
+        - *server_nix-cache01
+        - *server_pgdb1
+        - *server_nats1
   - path_regex: secrets/ca/[^/]+\.(yaml|json|env|ini|)
     key_groups:
       - age:
         - *admin_torjus
         - *server_ca
+  - path_regex: secrets/monitoring01/[^/]+\.(yaml|json|env|ini)
+    key_groups:
+      - age:
+        - *admin_torjus
+        - *server_monitoring01
   - path_regex: secrets/ca/keys/.+
     key_groups:
       - age:
         - *admin_torjus
         - *server_ca
+  - path_regex: secrets/nix-cache01/.+
+    key_groups:
+      - age:
+        - *admin_torjus
+        - *server_nix-cache01
+  - path_regex: secrets/http-proxy/.+
+    key_groups:
+      - age:
+        - *admin_torjus
+        - *server_http-proxy

CLAUDE.md

@@ -0,0 +1,519 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Repository Overview
+
+This is a Nix Flake-based NixOS configuration repository for managing a homelab infrastructure consisting of 16 server configurations. The repository uses a modular architecture with shared system configurations, reusable service modules, and per-host customization.
+
+## Common Commands
+
+### Building Configurations
+
+```bash
+# List all available configurations
+nix flake show
+
+# Build a specific host configuration locally (without deploying)
+nixos-rebuild build --flake .#<hostname>
+
+# Build and check a configuration
+nix build .#nixosConfigurations.<hostname>.config.system.build.toplevel
+```
+
+**Important:** Do NOT pipe `nix build` commands to other commands like `tail` or `head`. Piping can hide errors and make builds appear successful when they actually failed. Always run `nix build` without piping to see the full output.
+
+```bash
+# BAD - hides errors
+nix build .#create-host 2>&1 | tail -20
+
+# GOOD - shows all output and errors
+nix build .#create-host
+```
+
+### Deployment
+
+Do not automatically deploy changes. Deployments are usually done by updating the master branch, and then triggering the auto update on the specific host.
+
+### Testing Feature Branches on Hosts
+
+All hosts have the `nixos-rebuild-test` helper script for testing feature branches before merging:
+
+```bash
+# On the target host, test a feature branch
+nixos-rebuild-test boot <branch-name>
+nixos-rebuild-test switch <branch-name>
+
+# Additional arguments are passed through to nixos-rebuild
+nixos-rebuild-test boot my-feature --show-trace
+```
+
+When working on a feature branch that requires testing on a live host, suggest using this command instead of the full flake URL syntax.
+
+### Flake Management
+
+```bash
+# Check flake for errors
+nix flake check
+```
+Do not run `nix flake update`. Should only be done manually by user.
+
+### Development Environment
+
+```bash
+# Enter development shell (provides ansible, python3)
+nix develop
+```
+
+### Secrets Management
+
+Secrets are managed by OpenBao (Vault) using AppRole authentication. Most hosts use the
+`vault.secrets` option defined in `system/vault-secrets.nix` to fetch secrets at boot.
+Terraform manages the secrets and AppRole policies in `terraform/vault/`.
+
+Legacy sops-nix is still present but only actively used by the `ca` host. Do not edit any
+`.sops.yaml` or any file within `secrets/`. Ask the user to modify if necessary.
+
+### Git Workflow
+
+**Important:** Never commit directly to `master` unless the user explicitly asks for it. Always create a feature branch for changes.
+
+**Important:** Never amend commits to `master` unless the user explicitly asks for it. Amending rewrites history and causes issues for deployed configurations.
+
+When starting a new plan or task, the first step should typically be to create and checkout a new branch with an appropriate name (e.g., `git checkout -b dns-automation` or `git checkout -b fix-nginx-config`).
+
+### Plan Management
+
+When creating plans for large features, follow this workflow:
+
+1. When implementation begins, save a copy of the plan to `docs/plans/` (e.g., `docs/plans/feature-name.md`)
+2. Once the feature is fully implemented, move the plan to `docs/plans/completed/`
+
+### Git Commit Messages
+
+Commit messages should follow the format: `topic: short description`
+
+Examples:
+- `flake: add opentofu to devshell`
+- `template2: add proxmox image configuration`
+- `terraform: add VM deployment configuration`
+
+### Clipboard
+
+To copy text to the clipboard, pipe to `wl-copy` (Wayland):
+
+```bash
+echo "text" | wl-copy
+```
+
+### NixOS Options and Packages Lookup
+
+Two MCP servers are available for searching NixOS options and packages:
+
+- **nixpkgs-options** - Search and lookup NixOS configuration option documentation
+- **nixpkgs-packages** - Search and lookup Nix packages from nixpkgs
+
+**Session Setup:** At the start of each session, index the nixpkgs revision from `flake.lock` to ensure documentation matches the project's nixpkgs version:
+
+1. Read `flake.lock` and find the `nixpkgs` node's `rev` field
+2. Call `index_revision` with that git hash (both servers share the same index)
+
+**Options Tools (nixpkgs-options):**
+
+- `search_options` - Search for options by name or description (e.g., query "nginx" or "postgresql")
+- `get_option` - Get full details for a specific option (e.g., `services.loki.configuration`)
+- `get_file` - Fetch the source file from nixpkgs that declares an option
+
+**Package Tools (nixpkgs-packages):**
+
+- `search_packages` - Search for packages by name or description (e.g., query "nginx" or "python")
+- `get_package` - Get full details for a specific package by attribute path (e.g., `firefox`, `python312Packages.requests`)
+- `get_file` - Fetch the source file from nixpkgs that defines a package
+
+This ensures documentation matches the exact nixpkgs version (currently NixOS 25.11) used by this flake.
+
+### Lab Monitoring Log Queries
+
+The **lab-monitoring** MCP server can query logs from Loki. All hosts ship systemd journal logs via Promtail.
+
+**Loki Label Reference:**
+
+- `host` - Hostname (e.g., `ns1`, `ns2`, `monitoring01`, `ha1`). Use this label, not `hostname`.
+- `systemd_unit` - Systemd unit name (e.g., `nsd.service`, `prometheus.service`, `nixos-upgrade.service`)
+- `job` - Either `systemd-journal` (most logs) or `varlog` (file-based logs like caddy access logs)
+- `filename` - For `varlog` job, the log file path (e.g., `/var/log/caddy/nix-cache.log`)
+
+Journal log entries are JSON-formatted with the actual log message in the `MESSAGE` field. Other useful fields include `PRIORITY` and `SYSLOG_IDENTIFIER`.
+
+**Example LogQL queries:**
+```
+# Logs from a specific service on a host
+{host="ns2", systemd_unit="nsd.service"}
+
+# Substring match on log content
+{host="ns1", systemd_unit="nsd.service"} |= "error"
+
+# File-based logs (e.g., caddy access logs)
+{job="varlog", hostname="nix-cache01"}
+```
+
+Default lookback is 1 hour. Use the `start` parameter with relative durations (e.g., `24h`, `168h`) for older logs.
+
+### Lab Monitoring Prometheus Queries
+
+The **lab-monitoring** MCP server can query Prometheus metrics via PromQL. The `instance` label uses the FQDN format `<host>.home.2rjus.net:<port>`.
+
+**Prometheus Job Names:**
+
+- `node-exporter` - System metrics from all hosts (CPU, memory, disk, network)
+- `caddy` - Reverse proxy metrics (http-proxy)
+- `nix-cache_caddy` - Nix binary cache metrics
+- `home-assistant` - Home automation metrics
+- `jellyfin` - Media server metrics
+- `loki` / `prometheus` / `grafana` - Monitoring stack self-metrics
+- `step-ca` - Internal CA metrics
+- `pve-exporter` - Proxmox hypervisor metrics
+- `smartctl` - Disk SMART health (gunter)
+- `wireguard` - VPN metrics (http-proxy)
+- `pushgateway` - Push-based metrics (e.g., backup results)
+- `restic_rest` - Backup server metrics
+- `labmon` / `ghettoptt` / `alertmanager` - Other service metrics
+
+**Example PromQL queries:**
+```
+# Check all targets are up
+up
+
+# CPU usage for a specific host
+rate(node_cpu_seconds_total{instance=~"ns1.*", mode!="idle"}[5m])
+
+# Memory usage across all hosts
+node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
+
+# Disk space
+node_filesystem_avail_bytes{mountpoint="/"}
+```
+
+### Deploying to Test Hosts
+
+The **homelab-deploy** MCP server enables remote deployments to test-tier hosts via NATS messaging.
+
+**Available Tools:**
+
+- `deploy` - Deploy NixOS configuration to test-tier hosts
+- `list_hosts` - List available deployment targets
+
+**Deploy Parameters:**
+
+- `hostname` - Target a specific host (e.g., `vaulttest01`)
+- `role` - Deploy to all hosts with a specific role (e.g., `vault`)
+- `all` - Deploy to all test-tier hosts
+- `action` - nixos-rebuild action: `switch` (default), `boot`, `test`, `dry-activate`
+- `branch` - Git branch or commit to deploy (default: `master`)
+
+**Examples:**
+
+```
+# List available hosts
+list_hosts()
+
+# Deploy to a specific host
+deploy(hostname="vaulttest01", action="switch")
+
+# Dry-run deployment
+deploy(hostname="vaulttest01", action="dry-activate")
+
+# Deploy to all hosts with a role
+deploy(role="vault", action="switch")
+```
+
+**Note:** Only test-tier hosts with `homelab.deploy.enable = true` and the listener service running will respond to deployments.
+
+**Verifying Deployments:**
+
+After deploying, use the `nixos_flake_info` metric from nixos-exporter to verify the host is running the expected revision:
+
+```promql
+nixos_flake_info{instance=~"vaulttest01.*"}
+```
+
+The `current_rev` label contains the git commit hash of the deployed flake configuration.
+
+## Architecture
+
+### Directory Structure
+
+- `/flake.nix` - Central flake defining all NixOS configurations
+- `/hosts/<hostname>/` - Per-host configurations
+  - `default.nix` - Entry point, imports configuration.nix and services
+  - `configuration.nix` - Host-specific settings (networking, hardware, users)
+- `/system/` - Shared system-level configurations applied to ALL hosts
+  - Core modules: nix.nix, sshd.nix, sops.nix (legacy), vault-secrets.nix, acme.nix, autoupgrade.nix
+  - Monitoring: node-exporter and promtail on every host
+- `/modules/` - Custom NixOS modules
+  - `homelab/` - Homelab-specific options (DNS automation, monitoring scrape targets)
+- `/lib/` - Nix library functions
+  - `dns-zone.nix` - DNS zone generation functions
+  - `monitoring.nix` - Prometheus scrape target generation functions
+- `/services/` - Reusable service modules, selectively imported by hosts
+  - `home-assistant/` - Home automation stack
+  - `monitoring/` - Observability stack (Prometheus, Grafana, Loki, Tempo)
+  - `ns/` - DNS services (authoritative, resolver, zone generation)
+  - `http-proxy/`, `ca/`, `postgres/`, `nats/`, `jellyfin/`, etc.
+- `/secrets/` - SOPS-encrypted secrets with age encryption (legacy, only used by ca)
+- `/common/` - Shared configurations (e.g., VM guest agent)
+- `/docs/` - Documentation and plans
+  - `plans/` - Future plans and proposals
+  - `plans/completed/` - Completed plans (moved here when done)
+- `/playbooks/` - Ansible playbooks for fleet management
+- `/.sops.yaml` - SOPS configuration with age keys (legacy, only used by ca)
+
+### Configuration Inheritance
+
+Each host follows this import pattern:
+```
+hosts/<hostname>/default.nix
+  └─> configuration.nix (host-specific)
+      ├─> ../../system (ALL shared system configs - applied to every host)
+      ├─> ../../services/<service> (selective service imports)
+      └─> ../../common/vm (if VM)
+```
+
+All hosts automatically get:
+- Nix binary cache (nix-cache.home.2rjus.net)
+- SSH with root login enabled
+- OpenBao (Vault) secrets management via AppRole
+- Internal ACME CA integration (ca.home.2rjus.net)
+- Daily auto-upgrades with auto-reboot
+- Prometheus node-exporter + Promtail (logs to monitoring01)
+- Monitoring scrape target auto-registration via `homelab.monitoring` options
+- Custom root CA trust
+- DNS zone auto-registration via `homelab.dns` options
+
+### Active Hosts
+
+Production servers managed by `rebuild-all.sh`:
+- `ns1`, `ns2` - Primary/secondary DNS servers (10.69.13.5/6)
+- `ca` - Internal Certificate Authority
+- `ha1` - Home Assistant + Zigbee2MQTT + Mosquitto
+- `http-proxy` - Reverse proxy
+- `monitoring01` - Full observability stack (Prometheus, Grafana, Loki, Tempo, Pyroscope)
+- `jelly01` - Jellyfin media server
+- `nix-cache01` - Binary cache server
+- `pgdb1` - PostgreSQL database
+- `nats1` - NATS messaging server
+
+Template/test hosts:
+- `template1` - Base template for cloning new hosts
+
+### Flake Inputs
+
+- `nixpkgs` - NixOS 25.11 stable (primary)
+- `nixpkgs-unstable` - Unstable channel (available via overlay as `pkgs.unstable.<package>`)
+- `sops-nix` - Secrets management (legacy, only used by ca)
+- Custom packages from git.t-juice.club:
+  - `alerttonotify` - Alert routing
+  - `labmon` - Lab monitoring
+
+### Network Architecture
+
+- Domain: `home.2rjus.net`
+- Infrastructure subnet: `10.69.13.x`
+- DNS: ns1/ns2 provide authoritative DNS with primary-secondary setup
+- Internal CA for ACME certificates (no Let's Encrypt)
+- Centralized monitoring at monitoring01
+- Static networking via systemd-networkd
+
+### Secrets Management
+
+Most hosts use OpenBao (Vault) for secrets:
+- Vault server at `vault01.home.2rjus.net:8200`
+- AppRole authentication with credentials at `/var/lib/vault/approle/`
+- Secrets defined in Terraform (`terraform/vault/secrets.tf`)
+- AppRole policies in Terraform (`terraform/vault/approle.tf`)
+- NixOS module: `system/vault-secrets.nix` with `vault.secrets.<name>` options
+- `extractKey` option extracts a single key from vault JSON as a plain file
+- Secrets fetched at boot by `vault-secret-<name>.service` systemd units
+- Fallback to cached secrets in `/var/lib/vault/cache/` when Vault is unreachable
+- Provision AppRole credentials: `nix develop -c ansible-playbook playbooks/provision-approle.yml -e hostname=<host>`
+
+Legacy SOPS (only used by `ca` host):
+- SOPS with age encryption, keys in `.sops.yaml`
+- Shared secrets: `/secrets/secrets.yaml`
+- Per-host secrets: `/secrets/<hostname>/`
+
+### Auto-Upgrade System
+
+All hosts pull updates daily from:
+```
+git+https://git.t-juice.club/torjus/nixos-servers.git
+```
+
+Configured in `/system/autoupgrade.nix`:
+- Random delay to avoid simultaneous upgrades
+- Auto-reboot after successful upgrade
+- Systemd service: `nixos-upgrade.service`
+
+### Proxmox VM Provisioning with OpenTofu
+
+The repository includes automated workflows for building Proxmox VM templates and deploying VMs using OpenTofu (Terraform).
+
+#### Building and Deploying Templates
+
+Template VMs are built from `hosts/template2` and deployed to Proxmox using Ansible:
+
+```bash
+# Build NixOS image and deploy to Proxmox as template
+nix develop -c ansible-playbook -i playbooks/inventory.ini playbooks/build-and-deploy-template.yml
+```
+
+This playbook:
+1. Builds the Proxmox image using `nixos-rebuild build-image --image-variant proxmox`
+2. Uploads the `.vma.zst` image to Proxmox at `/var/lib/vz/dump`
+3. Restores it as VM ID 9000
+4. Converts it to a template
+
+Template configuration (`hosts/template2`):
+- Minimal base system with essential packages (age, vim, wget, git)
+- Cloud-init configured for NoCloud datasource (no EC2 metadata timeout)
+- DHCP networking on ens18
+- SSH key-based root login
+- `prepare-host.sh` script for cleaning machine-id, SSH keys, and regenerating age keys
+
+#### Deploying VMs with OpenTofu
+
+VMs are deployed from templates using OpenTofu in the `/terraform` directory:
+
+```bash
+cd terraform
+tofu init     # First time only
+tofu apply    # Deploy VMs
+```
+
+Configuration files:
+- `main.tf` - Proxmox provider configuration
+- `variables.tf` - Provider variables (API credentials)
+- `vm.tf` - VM resource definitions
+- `terraform.tfvars` - Actual credentials (gitignored)
+
+Example VM deployment includes:
+- Clone from template VM
+- Cloud-init configuration (SSH keys, network, DNS)
+- Custom CPU/memory/disk sizing
+- VLAN tagging
+- QEMU guest agent
+
+OpenTofu outputs the VM's IP address after deployment for easy SSH access.
+
+#### Template Rebuilding and Terraform State
+
+When the Proxmox template is rebuilt (via `build-and-deploy-template.yml`), the template name may change. This would normally cause Terraform to want to recreate all existing VMs, but that's unnecessary since VMs are independent once cloned.
+
+**Solution**: The `terraform/vms.tf` file includes a lifecycle rule to ignore certain attributes that don't need management:
+
+```hcl
+lifecycle {
+  ignore_changes = [
+    clone,            # Template name can change without recreating VMs
+    startup_shutdown, # Proxmox sets defaults (-1) that we don't need to manage
+  ]
+}
+```
+
+This means:
+- **clone**: Existing VMs are not affected by template name changes; only new VMs use the updated template
+- **startup_shutdown**: Proxmox sets default startup order/delay values (-1) that Terraform would otherwise try to remove
+- You can safely update `default_template_name` in `terraform/variables.tf` without recreating VMs
+- `tofu plan` won't show spurious changes for Proxmox-managed defaults
+
+**When rebuilding the template:**
+1. Run `nix develop -c ansible-playbook -i playbooks/inventory.ini playbooks/build-and-deploy-template.yml`
+2. Update `default_template_name` in `terraform/variables.tf` if the name changed
+3. Run `tofu plan` - should show no VM recreations (only template name in state)
+4. Run `tofu apply` - updates state without touching existing VMs
+5. New VMs created after this point will use the new template
+
+### Adding a New Host
+
+1. Create `/hosts/<hostname>/` directory
+2. Copy structure from `template1` or similar host
+3. Add host entry to `flake.nix` nixosConfigurations
+4. Configure networking in `configuration.nix` (static IP via `systemd.network.networks`, DNS servers)
+5. (Optional) Add `homelab.dns.cnames` if the host needs CNAME aliases
+6. Add `vault.enable = true;` to the host configuration
+7. Add AppRole policy in `terraform/vault/approle.tf` and any secrets in `secrets.tf`
+8. Run `tofu apply` in `terraform/vault/`
+9. User clones template host
+10. User runs `prepare-host.sh` on new host
+11. Provision AppRole credentials: `nix develop -c ansible-playbook playbooks/provision-approle.yml -e hostname=<host>`
+12. Commit changes, and merge to master.
+13. Deploy by running `nixos-rebuild boot --flake URL#<hostname>` on the host.
+14. Run auto-upgrade on DNS servers (ns1, ns2) to pick up the new host's DNS entry
+
+**Note:** DNS A records and Prometheus node-exporter scrape targets are auto-generated from the host's `systemd.network.networks` static IP configuration. No manual zone file or Prometheus config editing is required.
+
+### Important Patterns
+
+**Overlay usage**: Access unstable packages via `pkgs.unstable.<package>` (defined in flake.nix overlay-unstable)
+
+**Service composition**: Services in `/services/` are designed to be imported by multiple hosts. Keep them modular and reusable.
+
+**Hardware configuration reuse**: Multiple hosts share `/hosts/template/hardware-configuration.nix` for VM instances.
+
+**State version**: All hosts use stateVersion `"23.11"` - do not change this on existing hosts.
+
+**Firewall**: Disabled on most hosts (trusted network). Enable selectively in host configuration if needed.
+
+**Shell scripts**: Use `pkgs.writeShellApplication` instead of `pkgs.writeShellScript` or `pkgs.writeShellScriptBin` for creating shell scripts. `writeShellApplication` provides automatic shellcheck validation, sets strict bash options (`set -euo pipefail`), and allows declaring `runtimeInputs` for dependencies. When referencing the executable path (e.g., in `ExecStart`), use `lib.getExe myScript` to get the proper `bin/` path.
+
+### Monitoring Stack
+
+All hosts ship metrics and logs to `monitoring01`:
+- **Metrics**: Prometheus scrapes node-exporter from all hosts
+- **Logs**: Promtail ships logs to Loki on monitoring01
+- **Access**: Grafana at monitoring01 for visualization
+- **Tracing**: Tempo for distributed tracing
+- **Profiling**: Pyroscope for continuous profiling
+
+**Scrape Target Auto-Generation:**
+
+Prometheus scrape targets are automatically generated from host configurations, following the same pattern as DNS zone generation:
+
+- **Node-exporter**: All flake hosts with static IPs are automatically added as node-exporter targets
+- **Service targets**: Defined via `homelab.monitoring.scrapeTargets` in service modules
+- **External targets**: Non-flake hosts defined in `/services/monitoring/external-targets.nix`
+- **Library**: `lib/monitoring.nix` provides `generateNodeExporterTargets` and `generateScrapeConfigs`
+
+Host monitoring options (`homelab.monitoring.*`):
+- `enable` (default: `true`) - Include host in Prometheus node-exporter scrape targets
+- `scrapeTargets` (default: `[]`) - Additional scrape targets exposed by this host (job_name, port, metrics_path, scheme, scrape_interval, honor_labels)
+
+Service modules declare their scrape targets directly (e.g., `services/ca/default.nix` declares step-ca on port 9000). The Prometheus config on monitoring01 auto-generates scrape configs from all hosts.
+
+To add monitoring targets for non-NixOS hosts, edit `/services/monitoring/external-targets.nix`.
+
+### DNS Architecture
+
+- `ns1` (10.69.13.5) - Primary authoritative DNS + resolver
+- `ns2` (10.69.13.6) - Secondary authoritative DNS (AXFR from ns1)
+- All hosts point to ns1/ns2 for DNS resolution
+
+**Zone Auto-Generation:**
+
+DNS zone entries are automatically generated from host configurations:
+
+- **Flake-managed hosts**: A records extracted from `systemd.network.networks` static IPs
+- **CNAMEs**: Defined via `homelab.dns.cnames` option in host configs
+- **External hosts**: Non-flake hosts defined in `/services/ns/external-hosts.nix`
+- **Serial number**: Uses `self.sourceInfo.lastModified` (git commit timestamp)
+
+Host DNS options (`homelab.dns.*`):
+- `enable` (default: `true`) - Include host in DNS zone generation
+- `cnames` (default: `[]`) - List of CNAME aliases pointing to this host
+
+Hosts are automatically excluded from DNS if:
+- `homelab.dns.enable = false` (e.g., template hosts)
+- No static IP configured (e.g., DHCP-only hosts)
+- Network interface is a VPN/tunnel (wg*, tun*, tap*)
+
+To add DNS entries for non-NixOS hosts, edit `/services/ns/external-hosts.nix`.

README.md

@@ -1,11 +1,125 @@
 # nixos-servers
 
-Nixos configs for my homelab servers.
+NixOS Flake-based configuration repository for a homelab infrastructure. All hosts run NixOS 25.11 and are managed declaratively through this single repository.
 
-## Configurations in use
+## Hosts
 
-* ha1
-* ns1
-* ns2
-* template1
+| Host | Role |
+|------|------|
+| `ns1`, `ns2` | Primary/secondary authoritative DNS |
+| `ca` | Internal Certificate Authority |
+| `ha1` | Home Assistant + Zigbee2MQTT + Mosquitto |
+| `http-proxy` | Reverse proxy |
+| `monitoring01` | Prometheus, Grafana, Loki, Tempo, Pyroscope |
+| `jelly01` | Jellyfin media server |
+| `nix-cache01` | Nix binary cache |
+| `pgdb1` | PostgreSQL |
+| `nats1` | NATS messaging |
+| `vault01` | OpenBao (Vault) secrets management |
+| `template1`, `template2` | VM templates for cloning new hosts |
 
+## Directory Structure
+
+```
+flake.nix              # Flake entry point, defines all host configurations
+hosts/<hostname>/      # Per-host configuration
+system/                # Shared modules applied to ALL hosts
+services/              # Reusable service modules, selectively imported per host
+modules/               # Custom NixOS module definitions
+lib/                   # Nix library functions (DNS zone generation, etc.)
+secrets/               # SOPS-encrypted secrets (legacy, only used by ca)
+common/                # Shared configurations (e.g., VM guest agent)
+terraform/             # OpenTofu configs for Proxmox VM provisioning
+terraform/vault/       # OpenTofu configs for OpenBao (secrets, PKI, AppRoles)
+playbooks/             # Ansible playbooks for template building and fleet ops
+scripts/               # Helper scripts (create-host, vault-fetch)
+```
+
+## Key Features
+
+**Automatic DNS zone generation** - A records are derived from each host's static IP configuration. CNAME aliases are defined via `homelab.dns.cnames`. No manual zone file editing required.
+
+**OpenBao (Vault) secrets** - Hosts authenticate via AppRole and fetch secrets at boot. Secrets and policies are managed as code in `terraform/vault/`. Legacy SOPS remains only for the `ca` host.
+
+**Daily auto-upgrades** - All hosts pull from the master branch and automatically rebuild and reboot on a randomized schedule.
+
+**Shared base configuration** - Every host automatically gets SSH, monitoring (node-exporter + Promtail), internal ACME certificates, and Nix binary cache access via the `system/` modules.
+
+**Proxmox VM provisioning** - Build VM templates with Ansible and deploy VMs with OpenTofu from `terraform/`.
+
+**OpenBao (Vault) secrets** - Centralized secrets management with AppRole authentication, PKI infrastructure, and automated bootstrap. Managed as code in `terraform/vault/`.
+
+## Usage
+
+```bash
+# Enter dev shell (provides ansible, opentofu, openbao, create-host)
+nix develop
+
+# Build a host configuration locally
+nix build .#nixosConfigurations.<hostname>.config.system.build.toplevel
+
+# List all configurations
+nix flake show
+```
+
+Deployments are done by merging to master and triggering the auto-upgrade on the target host.
+
+## Provisioning New Hosts
+
+The repository includes an automated pipeline for creating and deploying new hosts on Proxmox.
+
+### 1. Generate host configuration
+
+The `create-host` tool (available in the dev shell) generates all required files for a new host:
+
+```bash
+create-host \
+  --hostname myhost \
+  --ip 10.69.13.50/24 \
+  --cpu 4 \
+  --memory 4096 \
+  --disk 50G
+```
+
+This creates:
+- `hosts/<hostname>/` - NixOS configuration (networking, imports, hardware)
+- Entry in `flake.nix`
+- VM definition in `terraform/vms.tf`
+- Vault AppRole policy and wrapped bootstrap token
+
+Omit `--ip` for DHCP. Use `--dry-run` to preview changes. Use `--force` to regenerate an existing host's config.
+
+### 2. Build and deploy the VM template
+
+The Proxmox VM template is built from `hosts/template2` and deployed with Ansible:
+
+```bash
+nix develop -c ansible-playbook -i playbooks/inventory.ini playbooks/build-and-deploy-template.yml
+```
+
+This only needs to be re-run when the base template changes.
+
+### 3. Deploy the VM
+
+```bash
+cd terraform && tofu apply
+```
+
+### 4. Automatic bootstrap
+
+On first boot, the VM automatically:
+1. Receives its hostname and Vault credentials via cloud-init
+2. Unwraps the Vault token and stores AppRole credentials
+3. Runs `nixos-rebuild boot` against the flake on the master branch
+4. Reboots into the host-specific configuration
+5. Services fetch their secrets from Vault at startup
+
+No manual intervention is required after `tofu apply`.
+
+## Network
+
+- Domain: `home.2rjus.net`
+- Infrastructure subnet: `10.69.13.0/24`
+- DNS: ns1/ns2 authoritative with primary-secondary AXFR
+- Internal CA for TLS certificates (migrating from step-ca to OpenBao PKI)
+- Centralized monitoring at monitoring01

TODO.md

@@ -0,0 +1,669 @@
+# TODO: Automated Host Deployment Pipeline
+
+## Vision
+
+Automate the entire process of creating, configuring, and deploying new NixOS hosts on Proxmox from a single command or script.
+
+**Desired workflow:**
+```bash
+./scripts/create-host.sh --hostname myhost --ip 10.69.13.50
+# Script creates config, deploys VM, bootstraps NixOS, and you're ready to go
+```
+
+**Current manual workflow (from CLAUDE.md):**
+1. Create `/hosts/<hostname>/` directory structure
+2. Add host to `flake.nix`
+3. Add DNS entries
+4. Clone template VM manually
+5. Run `prepare-host.sh` on new VM
+6. Add generated age key to `.sops.yaml`
+7. Configure networking
+8. Commit and push
+9. Run `nixos-rebuild boot --flake URL#<hostname>` on host
+
+## The Plan
+
+### Phase 1: Parameterized OpenTofu Deployments ✅ COMPLETED
+
+**Status:** Fully implemented and tested
+
+**Implementation:**
+- Locals-based structure using `for_each` pattern for multiple VM deployments
+- All VM parameters configurable with smart defaults (CPU, memory, disk, IP, storage, etc.)
+- Automatic DHCP vs static IP detection based on `ip` field presence
+- Dynamic outputs showing deployed VM IPs and specifications
+- Successfully tested deploying multiple VMs simultaneously
+
+**Tasks:**
+- [x] Create module/template structure in terraform for repeatable VM deployments
+- [x] Parameterize VM configuration (hostname, CPU, memory, disk, IP)
+- [x] Support both DHCP and static IP configuration via cloud-init
+- [x] Test deploying multiple VMs from same template
+
+**Deliverable:** ✅ Can deploy multiple VMs with custom parameters via OpenTofu in a single `tofu apply`
+
+**Files:**
+- `terraform/vms.tf` - VM definitions using locals map
+- `terraform/outputs.tf` - Dynamic outputs for all VMs
+- `terraform/variables.tf` - Configurable defaults
+- `terraform/README.md` - Complete documentation
+
+---
+
+### Phase 2: Host Configuration Generator ✅ COMPLETED
+
+**Status:** ✅ Fully implemented and tested
+**Completed:** 2025-02-01
+**Enhanced:** 2025-02-01 (added --force flag)
+
+**Goal:** Automate creation of host configuration files
+
+**Implementation:**
+- Python CLI tool packaged as Nix derivation
+- Available as `create-host` command in devShell
+- Rich terminal UI with configuration previews
+- Comprehensive validation (hostname format/uniqueness, IP subnet/uniqueness)
+- Jinja2 templates for NixOS configurations
+- Automatic updates to flake.nix and terraform/vms.tf
+- `--force` flag for regenerating existing configurations (useful for testing)
+
+**Tasks:**
+- [x] Create Python CLI with typer framework
+  - [x] Takes parameters: hostname, IP, CPU cores, memory, disk size
+  - [x] Generates `/hosts/<hostname>/` directory structure
+  - [x] Creates `configuration.nix` with proper hostname and networking
+  - [x] Generates `default.nix` with standard imports
+  - [x] References shared `hardware-configuration.nix` from template
+- [x] Add host entry to `flake.nix` programmatically
+  - [x] Text-based manipulation (regex insertion)
+  - [x] Inserts new nixosConfiguration entry
+  - [x] Maintains proper formatting
+- [x] Generate corresponding OpenTofu configuration
+  - [x] Adds VM definition to `terraform/vms.tf`
+  - [x] Uses parameters from CLI input
+  - [x] Supports both static IP and DHCP modes
+- [x] Package as Nix derivation with templates
+- [x] Add to flake packages and devShell
+- [x] Implement dry-run mode
+- [x] Write comprehensive README
+
+**Usage:**
+```bash
+# In nix develop shell
+create-host \
+  --hostname test01 \
+  --ip 10.69.13.50/24 \  # optional, omit for DHCP
+  --cpu 4 \               # optional, default 2
+  --memory 4096 \         # optional, default 2048
+  --disk 50G \            # optional, default 20G
+  --dry-run               # optional preview mode
+```
+
+**Files:**
+- `scripts/create-host/` - Complete Python package with Nix derivation
+- `scripts/create-host/README.md` - Full documentation and examples
+
+**Deliverable:** ✅ Tool generates all config files for a new host, validated with Nix and Terraform
+
+---
+
+### Phase 3: Bootstrap Mechanism ✅ COMPLETED
+
+**Status:** ✅ Fully implemented and tested
+**Completed:** 2025-02-01
+**Enhanced:** 2025-02-01 (added branch support for testing)
+
+**Goal:** Get freshly deployed VM to apply its specific host configuration
+
+**Implementation:** Systemd oneshot service that runs on first boot after cloud-init
+
+**Approach taken:** Systemd service (variant of Option A)
+- Systemd service `nixos-bootstrap.service` runs on first boot
+- Depends on `cloud-config.service` to ensure hostname is set
+- Reads hostname from `hostnamectl` (set by cloud-init via Terraform)
+- Supports custom git branch via `NIXOS_FLAKE_BRANCH` environment variable
+- Runs `nixos-rebuild boot --flake git+https://git.t-juice.club/torjus/nixos-servers.git?ref=$BRANCH#${hostname}`
+- Reboots into new configuration on success
+- Fails gracefully without reboot on errors (network issues, missing config)
+- Service self-destructs after successful bootstrap (not in new config)
+
+**Tasks:**
+- [x] Create bootstrap service module in template2
+  - [x] systemd oneshot service with proper dependencies
+  - [x] Reads hostname from hostnamectl (cloud-init sets it)
+  - [x] Checks network connectivity via HTTPS (curl)
+  - [x] Runs nixos-rebuild boot with flake URL
+  - [x] Reboots on success, fails gracefully on error
+- [x] Configure cloud-init datasource
+  - [x] Use ConfigDrive datasource (Proxmox provider)
+  - [x] Add cloud-init disk to Terraform VMs (disks.ide.ide2.cloudinit)
+  - [x] Hostname passed via cloud-init user-data from Terraform
+- [x] Test bootstrap service execution on fresh VM
+- [x] Handle failure cases (flake doesn't exist, network issues)
+  - [x] Clear error messages in journald
+  - [x] No reboot on failure
+  - [x] System remains accessible for debugging
+
+**Files:**
+- `hosts/template2/bootstrap.nix` - Bootstrap service definition
+- `hosts/template2/configuration.nix` - Cloud-init ConfigDrive datasource
+- `terraform/vms.tf` - Cloud-init disk configuration
+
+**Deliverable:** ✅ VMs automatically bootstrap and reboot into host-specific configuration on first boot
+
+---
+
+### Phase 4: Secrets Management with OpenBao (Vault)
+
+**Status:** 🚧 Phases 4a, 4b, 4c (partial), & 4d Complete
+
+**Challenge:** Current sops-nix approach has chicken-and-egg problem with age keys
+
+**Current workflow:**
+1. VM boots, generates age key at `/var/lib/sops-nix/key.txt`
+2. User runs `prepare-host.sh` which prints public key
+3. User manually adds public key to `.sops.yaml`
+4. User commits, pushes
+5. VM can now decrypt secrets
+
+**Selected approach:** Migrate to OpenBao (Vault fork) for centralized secrets management
+
+**Why OpenBao instead of HashiCorp Vault:**
+- HashiCorp Vault switched to BSL (Business Source License), unavailable in NixOS cache
+- OpenBao is the community fork maintaining the pre-BSL MPL 2.0 license
+- API-compatible with Vault, uses same Terraform provider
+- Maintains all Vault features we need
+
+**Benefits:**
+- Industry-standard secrets management (Vault-compatible experience)
+- Eliminates manual age key distribution step
+- Secrets-as-code via OpenTofu (infrastructure-as-code aligned)
+- Centralized PKI management with ACME support (ready to replace step-ca)
+- Automatic secret rotation capabilities
+- Audit logging for all secret access (not yet enabled)
+- AppRole authentication enables automated bootstrap
+
+**Current Architecture:**
+```
+vault01.home.2rjus.net (10.69.13.19)
+  ├─ KV Secrets Engine (ready to replace sops-nix)
+  │   ├─ secret/hosts/{hostname}/*
+  │   ├─ secret/services/{service}/*
+  │   └─ secret/shared/{category}/*
+  ├─ PKI Engine (ready to replace step-ca for TLS)
+  │   ├─ Root CA (EC P-384, 10 year)
+  │   ├─ Intermediate CA (EC P-384, 5 year)
+  │   └─ ACME endpoint enabled
+  ├─ SSH CA Engine (TODO: Phase 4c)
+  └─ AppRole Auth (per-host authentication configured)
+       ↓
+   [✅ Phase 4d] New hosts authenticate on first boot
+   [✅ Phase 4d] Fetch secrets via Vault API
+   No manual key distribution needed
+```
+
+**Completed:**
+- ✅ Phase 4a: OpenBao server with TPM2 auto-unseal
+- ✅ Phase 4b: Infrastructure-as-code (secrets, policies, AppRoles, PKI)
+- ✅ Phase 4d: Bootstrap integration for automated secrets access
+
+**Next Steps:**
+- Phase 4c: Migrate from step-ca to OpenBao PKI
+
+---
+
+#### Phase 4a: Vault Server Setup ✅ COMPLETED
+
+**Status:** ✅ Fully implemented and tested
+**Completed:** 2026-02-02
+
+**Goal:** Deploy and configure Vault server with auto-unseal
+
+**Implementation:**
+- Used **OpenBao** (Vault fork) instead of HashiCorp Vault due to BSL licensing concerns
+- TPM2-based auto-unseal using systemd's native `LoadCredentialEncrypted`
+- Self-signed bootstrap TLS certificates (avoiding circular dependency with step-ca)
+- File-based storage backend at `/var/lib/openbao`
+- Unix socket + TCP listener (0.0.0.0:8200) configuration
+
+**Tasks:**
+- [x] Create `hosts/vault01/` configuration
+  - [x] Basic NixOS configuration (hostname: vault01, IP: 10.69.13.19/24)
+  - [x] Created reusable `services/vault` module
+  - [x] Firewall not needed (trusted network)
+  - [x] Already in flake.nix, deployed via terraform
+- [x] Implement auto-unseal mechanism
+  - [x] **TPM2-based auto-unseal** (preferred option)
+    - [x] systemd `LoadCredentialEncrypted` with TPM2 binding
+    - [x] `writeShellApplication` script with proper runtime dependencies
+    - [x] Reads multiple unseal keys (one per line) until unsealed
+    - [x] Auto-unseals on service start via `ExecStartPost`
+- [x] Initial Vault setup
+  - [x] Initialized OpenBao with Shamir secret sharing (5 keys, threshold 3)
+  - [x] File storage backend
+  - [x] Self-signed TLS certificates via LoadCredential
+- [x] Deploy to infrastructure
+  - [x] DNS entry added for vault01.home.2rjus.net
+  - [x] VM deployed via terraform
+  - [x] Verified OpenBao running and auto-unsealing
+
+**Changes from Original Plan:**
+- Used OpenBao instead of HashiCorp Vault (licensing)
+- Used systemd's native TPM2 support instead of tpm2-tools directly
+- Skipped audit logging (can be enabled later)
+- Used self-signed certs initially (will migrate to OpenBao PKI later)
+
+**Deliverable:** ✅ Running OpenBao server that auto-unseals on boot using TPM2
+
+**Documentation:**
+- `/services/vault/README.md` - Service module overview
+- `/docs/vault/auto-unseal.md` - Complete TPM2 auto-unseal setup guide
+
+---
+
+#### Phase 4b: Vault-as-Code with OpenTofu ✅ COMPLETED
+
+**Status:** ✅ Fully implemented and tested
+**Completed:** 2026-02-02
+
+**Goal:** Manage all Vault configuration (secrets structure, policies, roles) as code
+
+**Implementation:**
+- Complete Terraform/OpenTofu configuration in `terraform/vault/`
+- Locals-based pattern (similar to `vms.tf`) for declaring secrets and policies
+- Auto-generation of secrets using `random_password` provider
+- Three-tier secrets path hierarchy: `hosts/`, `services/`, `shared/`
+- PKI infrastructure with **Elliptic Curve certificates** (P-384 for CAs, P-256 for leaf certs)
+- ACME support enabled on intermediate CA
+
+**Tasks:**
+- [x] Set up Vault Terraform provider
+  - [x] Created `terraform/vault/` directory
+  - [x] Configured Vault provider (uses HashiCorp provider, compatible with OpenBao)
+  - [x] Credentials in terraform.tfvars (gitignored)
+  - [x] terraform.tfvars.example for reference
+- [x] Enable and configure secrets engines
+  - [x] KV v2 engine at `secret/`
+  - [x] Three-tier path structure:
+    - `secret/hosts/{hostname}/*` - Host-specific secrets
+    - `secret/services/{service}/*` - Service-wide secrets
+    - `secret/shared/{category}/*` - Shared secrets (SMTP, backups, etc.)
+- [x] Define policies as code
+  - [x] Policies auto-generated from `locals.host_policies`
+  - [x] Per-host policies with read/list on designated paths
+  - [x] Principle of least privilege enforced
+- [x] Set up AppRole authentication
+  - [x] AppRole backend enabled at `approle/`
+  - [x] Roles auto-generated per host from `locals.host_policies`
+  - [x] Token TTL: 1 hour, max 24 hours
+  - [x] Policies bound to roles
+- [x] Implement secrets-as-code patterns
+  - [x] Auto-generated secrets using `random_password` provider
+  - [x] Manual secrets supported via variables in terraform.tfvars
+  - [x] Secret structure versioned in .tf files
+  - [x] Secret values excluded from git
+- [x] Set up PKI infrastructure
+  - [x] Root CA (10 year TTL, EC P-384)
+  - [x] Intermediate CA (5 year TTL, EC P-384)
+  - [x] PKI role for `*.home.2rjus.net` (30 day max TTL, EC P-256)
+  - [x] ACME enabled on intermediate CA
+  - [x] Support for static certificate issuance via Terraform
+  - [x] CRL, OCSP, and issuing certificate URLs configured
+
+**Changes from Original Plan:**
+- Used Elliptic Curve instead of RSA for all certificates (better performance, smaller keys)
+- Implemented PKI infrastructure in Phase 4b instead of Phase 4c (more logical grouping)
+- ACME support configured immediately (ready for migration from step-ca)
+- Did not migrate existing sops-nix secrets yet (deferred to gradual migration)
+
+**Files:**
+- `terraform/vault/main.tf` - Provider configuration
+- `terraform/vault/variables.tf` - Variable definitions
+- `terraform/vault/approle.tf` - AppRole authentication (locals-based pattern)
+- `terraform/vault/pki.tf` - PKI infrastructure with EC certificates
+- `terraform/vault/secrets.tf` - KV secrets engine (auto-generation support)
+- `terraform/vault/README.md` - Complete documentation and usage examples
+- `terraform/vault/terraform.tfvars.example` - Example credentials
+
+**Deliverable:** ✅ All secrets, policies, AppRoles, and PKI managed as OpenTofu code in `terraform/vault/`
+
+**Documentation:**
+- `/terraform/vault/README.md` - Comprehensive guide covering:
+  - Setup and deployment
+  - AppRole usage and host access patterns
+  - PKI certificate issuance (ACME, static, manual)
+  - Secrets management patterns
+  - ACME configuration and troubleshooting
+
+---
+
+#### Phase 4c: PKI Migration (Replace step-ca)
+
+**Status:** 🚧 Partially Complete - vault01 and test host migrated, remaining hosts pending
+
+**Goal:** Migrate hosts from step-ca to OpenBao PKI for TLS certificates
+
+**Note:** PKI infrastructure already set up in Phase 4b (root CA, intermediate CA, ACME support)
+
+**Tasks:**
+- [x] Set up OpenBao PKI engines (completed in Phase 4b)
+  - [x] Root CA (`pki/` mount, 10 year TTL, EC P-384)
+  - [x] Intermediate CA (`pki_int/` mount, 5 year TTL, EC P-384)
+  - [x] Signed intermediate with root CA
+  - [x] Configured CRL, OCSP, and issuing certificate URLs
+- [x] Enable ACME support (completed in Phase 4b, fixed in Phase 4c)
+  - [x] Enabled ACME on intermediate CA
+  - [x] Created PKI role for `*.home.2rjus.net`
+  - [x] Set certificate TTLs (30 day max) and allowed domains
+  - [x] ACME directory: `https://vault01.home.2rjus.net:8200/v1/pki_int/acme/directory`
+  - [x] Fixed ACME response headers (added Replay-Nonce, Link, Location to allowed_response_headers)
+  - [x] Configured cluster path for ACME
+- [x] Download and distribute root CA certificate
+  - [x] Added root CA to `system/pki/root-ca.nix`
+  - [x] Distributed to all hosts via system imports
+- [x] Test certificate issuance
+  - [x] Tested ACME issuance on vaulttest01 successfully
+  - [x] Verified certificate chain and trust
+- [x] Migrate vault01's own certificate
+  - [x] Created `bootstrap-vault-cert` script for initial certificate issuance via bao CLI
+  - [x] Issued certificate with SANs (vault01.home.2rjus.net + vault.home.2rjus.net)
+  - [x] Updated service to read certificates from `/var/lib/acme/vault01.home.2rjus.net/`
+  - [x] Configured ACME for automatic renewals
+- [ ] Migrate hosts from step-ca to OpenBao
+  - [x] Tested on vaulttest01 (non-production host)
+  - [ ] Standardize hostname usage across all configurations
+    - [ ] Use `vault.home.2rjus.net` (CNAME) consistently everywhere
+    - [ ] Update NixOS configurations to use CNAME instead of vault01
+    - [ ] Update Terraform configurations to use CNAME
+    - [ ] Audit and fix mixed usage of vault01.home.2rjus.net vs vault.home.2rjus.net
+  - [ ] Update `system/acme.nix` to use OpenBao ACME endpoint
+  - [ ] Change server to `https://vault.home.2rjus.net:8200/v1/pki_int/acme/directory`
+  - [ ] Roll out to all hosts via auto-upgrade
+- [ ] Configure SSH CA in OpenBao (optional, future work)
+  - [ ] Enable SSH secrets engine (`ssh/` mount)
+  - [ ] Generate SSH signing keys
+  - [ ] Create roles for host and user certificates
+  - [ ] Configure TTLs and allowed principals
+  - [ ] Distribute SSH CA public key to all hosts
+  - [ ] Update sshd_config to trust OpenBao CA
+- [ ] Decommission step-ca
+  - [ ] Verify all ACME services migrated and working
+  - [ ] Stop step-ca service on ca host
+  - [ ] Archive step-ca configuration for backup
+  - [ ] Update documentation
+
+**Implementation Details (2026-02-03):**
+
+**ACME Configuration Fix:**
+The key blocker was that OpenBao's PKI mount was filtering out required ACME response headers. The solution was to add `allowed_response_headers` to the Terraform mount configuration:
+```hcl
+allowed_response_headers = [
+  "Replay-Nonce",  # Required for ACME nonce generation
+  "Link",          # Required for ACME navigation
+  "Location"       # Required for ACME resource location
+]
+```
+
+**Cluster Path Configuration:**
+ACME requires the cluster path to include the full API path:
+```hcl
+path     = "${var.vault_address}/v1/${vault_mount.pki_int.path}"
+aia_path = "${var.vault_address}/v1/${vault_mount.pki_int.path}"
+```
+
+**Bootstrap Process:**
+Since vault01 needed a certificate from its own PKI (chicken-and-egg problem), we created a `bootstrap-vault-cert` script that:
+1. Uses the Unix socket (no TLS) to issue a certificate via `bao` CLI
+2. Places it in the ACME directory structure
+3. Includes both vault01.home.2rjus.net and vault.home.2rjus.net as SANs
+4. After restart, ACME manages renewals automatically
+
+**Files Modified:**
+- `terraform/vault/pki.tf` - Added allowed_response_headers, cluster config, ACME config
+- `services/vault/default.nix` - Updated cert paths, added bootstrap script, configured ACME
+- `system/pki/root-ca.nix` - Added OpenBao root CA to trust store
+- `hosts/vaulttest01/configuration.nix` - Overrode ACME server for testing
+
+**Deliverable:** ✅ vault01 and vaulttest01 using OpenBao PKI, remaining hosts still on step-ca
+
+---
+
+#### Phase 4d: Bootstrap Integration ✅ COMPLETED (2026-02-02)
+
+**Goal:** New hosts automatically authenticate to Vault on first boot, no manual steps
+
+**Tasks:**
+- [x] Update create-host tool
+  - [x] Generate wrapped token (24h TTL, single-use) for new host
+  - [x] Add host-specific policy to Vault (via terraform/vault/hosts-generated.tf)
+  - [x] Store wrapped token in terraform/vms.tf for cloud-init injection
+  - [x] Add `--regenerate-token` flag to regenerate only the token without overwriting config
+- [x] Update template2 for Vault authentication
+  - [x] Reads wrapped token from cloud-init (/run/cloud-init-env)
+  - [x] Unwraps token to get role_id + secret_id
+  - [x] Stores AppRole credentials in /var/lib/vault/approle/ (persistent)
+  - [x] Graceful fallback if Vault unavailable during bootstrap
+- [x] Create NixOS Vault secrets module (system/vault-secrets.nix)
+  - [x] Runtime secret fetching (services fetch on start, not at nixos-rebuild time)
+  - [x] Secrets cached in /var/lib/vault/cache/ for fallback when Vault unreachable
+  - [x] Secrets written to /run/secrets/ (tmpfs, cleared on reboot)
+  - [x] Fresh authentication per service start (no token renewal needed)
+  - [x] Optional periodic rotation with systemd timers
+  - [x] Critical service protection (no auto-restart for DNS, CA, Vault itself)
+- [x] Create vault-fetch helper script
+  - [x] Standalone tool for fetching secrets from Vault
+  - [x] Authenticates using AppRole credentials
+  - [x] Writes individual files per secret key
+  - [x] Handles caching and fallback logic
+- [x] Update bootstrap service (hosts/template2/bootstrap.nix)
+  - [x] Unwraps Vault token on first boot
+  - [x] Stores persistent AppRole credentials
+  - [x] Continues with nixos-rebuild
+  - [x] Services fetch secrets when they start
+- [x] Update terraform cloud-init (terraform/cloud-init.tf)
+  - [x] Inject VAULT_ADDR and VAULT_WRAPPED_TOKEN via write_files
+  - [x] Write to /run/cloud-init-env (tmpfs, cleaned on reboot)
+  - [x] Fixed YAML indentation issues (write_files at top level)
+  - [x] Support flake_branch alongside vault credentials
+- [x] Test complete flow
+  - [x] Created vaulttest01 test host
+  - [x] Verified bootstrap with Vault integration
+  - [x] Verified service secret fetching
+  - [x] Tested cache fallback when Vault unreachable
+  - [x] Tested wrapped token single-use (second bootstrap fails as expected)
+  - [x] Confirmed zero manual steps required
+
+**Implementation Details:**
+
+**Wrapped Token Security:**
+- Single-use tokens prevent reuse if leaked
+- 24h TTL limits exposure window
+- Safe to commit to git (expired/used tokens useless)
+- Regenerate with `create-host --hostname X --regenerate-token`
+
+**Secret Fetching:**
+- Runtime (not build-time) keeps secrets out of Nix store
+- Cache fallback enables service availability when Vault down
+- Fresh authentication per service start (no renewal complexity)
+- Individual files per secret key for easy consumption
+
+**Bootstrap Flow:**
+```
+1. create-host --hostname myhost --ip 10.69.13.x/24
+   ↓ Generates wrapped token, updates terraform
+2. tofu apply (deploys VM with cloud-init)
+   ↓ Cloud-init writes wrapped token to /run/cloud-init-env
+3. nixos-bootstrap.service runs:
+   ↓ Unwraps token → gets role_id + secret_id
+   ↓ Stores in /var/lib/vault/approle/ (persistent)
+   ↓ Runs nixos-rebuild boot
+4. Service starts → fetches secrets from Vault
+   ↓ Uses stored AppRole credentials
+   ↓ Caches secrets for fallback
+5. Done - zero manual intervention
+```
+
+**Files Created:**
+- `scripts/vault-fetch/` - Secret fetching helper (Nix package)
+- `system/vault-secrets.nix` - NixOS module for declarative Vault secrets
+- `scripts/create-host/vault_helper.py` - Vault API integration
+- `terraform/vault/hosts-generated.tf` - Auto-generated host policies
+- `docs/vault-bootstrap-implementation.md` - Architecture documentation
+- `docs/vault-bootstrap-testing.md` - Testing guide
+
+**Configuration:**
+- Vault address: `https://vault01.home.2rjus.net:8200` (configurable)
+- All defaults remain configurable via environment variables or NixOS options
+
+**Next Steps:**
+- Gradually migrate existing services from sops-nix to Vault
+- Add CNAME for vault.home.2rjus.net → vault01.home.2rjus.net
+- Phase 4c: Migrate from step-ca to OpenBao PKI (future)
+
+**Deliverable:** ✅ Fully automated secrets access from first boot, zero manual steps
+
+---
+
+### Phase 6: Integration Script
+
+**Goal:** Single command to create and deploy a new host
+
+**Tasks:**
+- [ ] Create `scripts/create-host.sh` master script that orchestrates:
+  1. Prompts for: hostname, IP (or DHCP), CPU, memory, disk
+  2. Validates inputs (IP not in use, hostname unique, etc.)
+  3. Calls host config generator (Phase 2)
+  4. Generates OpenTofu config (Phase 2)
+  5. Handles secrets (Phase 4)
+  6. Updates DNS (Phase 5)
+  7. Commits all changes to git
+  8. Runs `tofu apply` to deploy VM
+  9. Waits for bootstrap to complete (Phase 3)
+  10. Prints success message with IP and SSH command
+- [ ] Add `--dry-run` flag to preview changes
+- [ ] Add `--interactive` mode vs `--batch` mode
+- [ ] Error handling and rollback on failures
+
+**Deliverable:** `./scripts/create-host.sh --hostname myhost --ip 10.69.13.50` creates a fully working host
+
+---
+
+### Phase 7: Testing & Documentation
+
+**Status:** 🚧 In Progress (testing improvements completed)
+
+**Testing Improvements Implemented (2025-02-01):**
+
+The pipeline now supports efficient testing without polluting master branch:
+
+**1. --force Flag for create-host**
+- Re-run `create-host` to regenerate existing configurations
+- Updates existing entries in flake.nix and terraform/vms.tf (no duplicates)
+- Skip uniqueness validation checks
+- Useful for iterating on configuration templates during testing
+
+**2. Branch Support for Bootstrap**
+- Bootstrap service reads `NIXOS_FLAKE_BRANCH` environment variable
+- Defaults to `master` if not set
+- Allows testing pipeline changes on feature branches
+- Cloud-init passes branch via `/etc/environment`
+
+**3. Cloud-init Disk for Branch Configuration**
+- Terraform generates custom cloud-init snippets for test VMs
+- Set `flake_branch` field in VM definition to use non-master branch
+- Production VMs omit this field and use master (default)
+- Files automatically uploaded to Proxmox via SSH
+
+**Testing Workflow:**
+
+```bash
+# 1. Create test branch
+git checkout -b test-pipeline
+
+# 2. Generate or update host config
+create-host --hostname testvm01 --ip 10.69.13.100/24
+
+# 3. Edit terraform/vms.tf to add test VM with branch
+# vms = {
+#   "testvm01" = {
+#     ip = "10.69.13.100/24"
+#     flake_branch = "test-pipeline"  # Bootstrap from this branch
+#   }
+# }
+
+# 4. Commit and push test branch
+git add -A && git commit -m "test: add testvm01"
+git push origin test-pipeline
+
+# 5. Deploy VM
+cd terraform && tofu apply
+
+# 6. Watch bootstrap (VM fetches from test-pipeline branch)
+ssh root@10.69.13.100
+journalctl -fu nixos-bootstrap.service
+
+# 7. Iterate: modify templates and regenerate with --force
+cd .. && create-host --hostname testvm01 --ip 10.69.13.100/24 --force
+git commit -am "test: update config" && git push
+
+# Redeploy to test fresh bootstrap
+cd terraform
+tofu destroy -target=proxmox_vm_qemu.vm[\"testvm01\"] && tofu apply
+
+# 8. Clean up when done: squash commits, merge to master, remove test VM
+```
+
+**Files:**
+- `scripts/create-host/create_host.py` - Added --force parameter
+- `scripts/create-host/manipulators.py` - Update vs insert logic
+- `hosts/template2/bootstrap.nix` - Branch support via environment variable
+- `terraform/vms.tf` - flake_branch field support
+- `terraform/cloud-init.tf` - Custom cloud-init disk generation
+- `terraform/variables.tf` - proxmox_host variable for SSH uploads
+
+**Remaining Tasks:**
+- [ ] Test full pipeline end-to-end on feature branch
+- [ ] Update CLAUDE.md with testing workflow
+- [ ] Add troubleshooting section
+- [ ] Create examples for common scenarios (DHCP host, static IP host, etc.)
+
+---
+
+## Open Questions
+
+1. **Bootstrap method:** Cloud-init runcmd vs Terraform provisioner vs Ansible?
+2. **Secrets handling:** Pre-generate keys vs post-deployment injection?
+3. **DNS automation:** Auto-commit or manual merge?
+4. **Git workflow:** Auto-push changes or leave for user review?
+5. **Template selection:** Single template2 or multiple templates for different host types?
+6. **Networking:** Always DHCP initially, or support static IP from start?
+7. **Error recovery:** What happens if bootstrap fails? Manual intervention or retry?
+
+## Implementation Order
+
+Recommended sequence:
+1. Phase 1: Parameterize OpenTofu (foundation for testing)
+2. Phase 3: Bootstrap mechanism (core automation)
+3. Phase 2: Config generator (automate the boilerplate)
+4. Phase 4: Secrets (solves biggest chicken-and-egg)
+5. Phase 5: DNS (nice-to-have automation)
+6. Phase 6: Integration script (ties it all together)
+7. Phase 7: Testing & docs
+
+## Success Criteria
+
+When complete, creating a new host should:
+- Take < 5 minutes of human time
+- Require minimal user input (hostname, IP, basic specs)
+- Result in a fully configured, secret-enabled, DNS-registered host
+- Be reproducible and documented
+- Handle common errors gracefully
+
+---
+
+## Notes
+
+- Keep incremental commits at each phase
+- Test each phase independently before moving to next
+- Maintain backward compatibility with manual workflow
+- Document any manual steps that can't be automated

common/vm/default.nix

@@ -0,0 +1,6 @@
+{ ... }:
+{
+  imports = [
+    ./qemu-guest.nix
+  ];
+}

common/vm/qemu-guest.nix

@@ -0,0 +1,4 @@
+{ ... }:
+{
+  services.qemuGuest.enable = true;
+}

docs/infrastructure.md

@@ -0,0 +1,282 @@
+# Homelab Infrastructure
+
+This document describes the physical and virtual infrastructure components that support the NixOS-managed servers in this repository.
+
+## Overview
+
+The homelab consists of several core infrastructure components:
+- **Proxmox VE** - Hypervisor hosting all NixOS VMs
+- **TrueNAS** - Network storage and backup target
+- **Ubiquiti EdgeRouter** - Primary router and gateway
+- **Mikrotik Switch** - Core network switching
+
+All NixOS configurations in this repository run as VMs on Proxmox and rely on these underlying infrastructure components.
+
+## Network Topology
+
+### Subnets
+
+VLAN numbers are based on third octet of ip address.
+
+TODO: VLAN naming is currently inconsistent across router/switch/Proxmox configurations. Need to standardize VLAN names and update all device configs to use consistent naming.
+
+- `10.69.8.x`  - Kubernetes (no longer in use)
+- `10.69.12.x` - Core services
+- `10.69.13.x` - NixOS VMs and core services
+- `10.69.30.x` - Client network 1
+- `10.69.31.x` - Clients network 2
+- `10.69.99.x` - Management network
+
+### Core Network Services
+
+- **Gateway**: Web UI exposed on 10.69.10.1
+- **DNS**: ns1 (10.69.13.5), ns2 (10.69.13.6)
+- **Primary DNS Domain**: `home.2rjus.net`
+
+## Hardware Components
+
+### Proxmox Hypervisor
+
+**Purpose**: Hosts all NixOS VMs defined in this repository
+
+**Hardware**:
+- CPU: AMD Ryzen 9 3900X 12-Core Processor
+- RAM: 96GB (94Gi)
+- Storage: 1TB NVMe SSD (nvme0n1)
+
+**Management**:
+- Web UI: `https://pve1.home.2rjus.net:8006`
+- Cluster: Standalone
+- Version: Proxmox VE 8.4.16 (kernel 6.8.12-18-pve)
+
+**VM Provisioning**:
+- Template VM: ID 9000 (built from `hosts/template2`)
+- See `/terraform` directory for automated VM deployment using OpenTofu
+
+**Storage**:
+- ZFS pool: `rpool` on NVMe partition (nvme0n1p3)
+  - Total capacity: ~900GB (232GB used, 667GB available)
+  - Configuration: Single disk (no RAID)
+  - Scrub status: Last scrub completed successfully with 0 errors
+
+**Networking**:
+- Management interface: `vmbr0` - 10.69.12.75/24 (VLAN 12 - Core services)
+- Physical interface: `enp9s0` (primary), `enp4s0` (unused)
+- VM bridges:
+  - `vmbr0` - Main bridge (bridged to enp9s0)
+  - `vmbr0v8` - VLAN 8 (Kubernetes - deprecated)
+  - `vmbr0v13` - VLAN 13 (NixOS VMs and core services)
+
+### TrueNAS
+
+**Purpose**: Network storage, backup target, media storage
+
+**Hardware**:
+- Model: Custom build
+- CPU: AMD Ryzen 5 5600G with Radeon Graphics
+- RAM: 32GB (31.2 GiB)
+- Disks:
+  - 2x Kingston SA400S37 240GB SSD (boot pool, mirrored)
+  - 2x Seagate ST16000NE000 16TB HDD (hdd-pool mirror-0)
+  - 2x WD WD80EFBX 8TB HDD (hdd-pool mirror-1)
+  - 2x Seagate ST8000VN004 8TB HDD (hdd-pool mirror-2)
+  - 1x NVMe 2TB (nvme-pool, no redundancy)
+
+**Management**:
+- Web UI: `https://nas.home.2rjus.net` (10.69.12.50)
+- Hostname: `nas.home.2rjus.net`
+- Version: TrueNAS-13.0-U6.1 (Core)
+
+**Networking**:
+- Primary interface: `mlxen0` - 10GbE (10Gbase-CX4) connected to sw1
+- IP: 10.69.12.50/24 (VLAN 12 - Core services)
+
+**ZFS Pools**:
+- `boot-pool`: 206GB (mirrored SSDs) - 4% used
+  - Mirror of 2x Kingston 240GB SSDs
+  - Last scrub: No errors
+- `hdd-pool`: 29.1TB total (3-way mirror, 28.4TB used, 658GB free) - 97% capacity
+  - mirror-0: 2x 16TB Seagate ST16000NE000
+  - mirror-1: 2x 8TB WD WD80EFBX
+  - mirror-2: 2x 8TB Seagate ST8000VN004
+  - Last scrub: No errors
+- `nvme-pool`: 1.81TB (single NVMe, 70.4GB used, 1.74TB free) - 3% capacity
+  - Single NVMe drive, no redundancy
+  - Last scrub: No errors
+
+**NFS Exports**:
+- `/mnt/hdd-pool/media` - Media storage (exported to 10.69.0.0/16, used by Jellyfin)
+- `/mnt/hdd-pool/virt/nfs-iso` - ISO storage for Proxmox
+- `/mnt/hdd-pool/virt/kube-prod-pvc` - Kubernetes storage (deprecated)
+
+**Jails**:
+TrueNAS runs several FreeBSD jails for media management:
+- nzbget - Usenet downloader
+- restic-rest - Restic REST server for backups
+- radarr - Movie management
+- sonarr - TV show management
+
+### Ubiquiti EdgeRouter
+
+**Purpose**: Primary router, gateway, firewall, inter-VLAN routing
+
+**Model**: EdgeRouter X 5-Port
+
+**Hardware**:
+- Serial: F09FC20E1A4C
+
+**Management**:
+- SSH: `ssh ubnt@10.69.10.1`
+- Web UI: `https://10.69.10.1`
+- Version: EdgeOS v2.0.9-hotfix.6 (build 5574651, 12/30/22)
+
+**WAN Connection**:
+- Interface: eth0
+- Public IP: 84.213.73.123/20
+- Gateway: 84.213.64.1
+
+**Interface Layout**:
+- **eth0**: WAN (public IP)
+- **eth1**: 10.69.31.1/24 - Clients network 2
+- **eth2**: Unused (down)
+- **eth3**: 10.69.30.1/24 - Client network 1
+- **eth4**: Trunk port to Mikrotik switch (carries all VLANs)
+  - eth4.8: 10.69.8.1/24 - K8S (deprecated)
+  - eth4.10: 10.69.10.1/24 - TRUSTED (management access)
+  - eth4.12: 10.69.12.1/24 - SERVER (Proxmox, TrueNAS, core services)
+  - eth4.13: 10.69.13.1/24 - SVC (NixOS VMs)
+  - eth4.21: 10.69.21.1/24 - CLIENTS
+  - eth4.22: 10.69.22.1/24 - WLAN (wireless clients)
+  - eth4.23: 10.69.23.1/24 - IOT
+  - eth4.99: 10.69.99.1/24 - MGMT (device management)
+
+**Routing**:
+- Default route: 0.0.0.0/0 via 84.213.64.1 (WAN gateway)
+- Static route: 192.168.100.0/24 via eth0
+- All internal VLANs directly connected
+
+**DHCP Servers**:
+Active DHCP pools on all networks:
+- dhcp-8: VLAN 8 (K8S) - 91 addresses
+- dhcp-12: VLAN 12 (SERVER) - 51 addresses
+- dhcp-13: VLAN 13 (SVC) - 41 addresses
+- dhcp-21: VLAN 21 (CLIENTS) - 141 addresses
+- dhcp-22: VLAN 22 (WLAN) - 101 addresses
+- dhcp-23: VLAN 23 (IOT) - 191 addresses
+- dhcp-30: eth3 (Client network 1) - 101 addresses
+- dhcp-31: eth1 (Clients network 2) - 21 addresses
+- dhcp-mgmt: VLAN 99 (MGMT) - 51 addresses
+
+**NAT/Firewall**:
+- Masquerading on WAN interface (eth0)
+
+### Mikrotik Switch
+
+**Purpose**: Core Layer 2/3 switching
+
+**Model**: MikroTik CRS326-24G-2S+ (24x 1GbE + 2x 10GbE SFP+)
+
+**Hardware**:
+- CPU: ARMv7 @ 800MHz
+- RAM: 512MB
+- Uptime: 21+ weeks
+
+**Management**:
+- Hostname: `sw1.home.2rjus.net`
+- SSH access: `ssh admin@sw1.home.2rjus.net` (using gunter SSH key)
+- Management IP: 10.69.99.2/24 (VLAN 99)
+- Version: RouterOS 6.47.10 (long-term)
+
+**VLANs**:
+- VLAN 8: Kubernetes (deprecated)
+- VLAN 12: SERVERS - Core services subnet
+- VLAN 13: SVC - Services subnet
+- VLAN 21: CLIENTS
+- VLAN 22: WLAN - Wireless network
+- VLAN 23: IOT
+- VLAN 99: MGMT - Management network
+
+**Port Layout** (active ports):
+- **ether1**: Uplink to EdgeRouter (trunk, carries all VLANs)
+- **ether11**: virt-mini1 (VLAN 12 - SERVERS)
+- **ether12**: Home Assistant (VLAN 12 - SERVERS)
+- **ether24**: Wireless AP (VLAN 22 - WLAN)
+- **sfp-sfpplus1**: Media server/Jellyfin (VLAN 12) - 10Gbps, 7m copper DAC
+- **sfp-sfpplus2**: TrueNAS (VLAN 12) - 10Gbps, 1m copper DAC
+
+**Bridge Configuration**:
+- All ports bridged to main bridge interface
+- Hardware offloading enabled
+- VLAN filtering enabled on bridge
+
+## Backup & Disaster Recovery
+
+### Backup Strategy
+
+**NixOS VMs**:
+- Declarative configurations in this git repository
+- Secrets: SOPS-encrypted, backed up with repository
+- State/data: Some hosts are backed up to nas host, but this should be improved and expanded to more hosts.
+
+**Proxmox**:
+- VM backups: Not currently implemented
+
+**Critical Credentials**:
+
+TODO: Document this
+
+- OpenBao root token and unseal keys: _[offline secure storage location]_
+- Proxmox root password: _[secure storage]_
+- TrueNAS admin password: _[secure storage]_
+- Router admin credentials: _[secure storage]_
+
+### Disaster Recovery Procedures
+
+**Total Infrastructure Loss**:
+1. Restore Proxmox from installation media
+2. Restore TrueNAS from installation media, import ZFS pools
+3. Restore network configuration on EdgeRouter and Mikrotik
+4. Rebuild NixOS VMs from this repository using Proxmox template
+5. Restore stateful data from TrueNAS backups
+6. Re-initialize OpenBao and restore from backup if needed
+
+**Individual VM Loss**:
+1. Deploy new VM from template using OpenTofu (`terraform/`)
+2. Run `nixos-rebuild` with appropriate flake configuration
+3. Restore any stateful data from backups
+4. For vault01: follow re-provisioning steps in `docs/vault/auto-unseal.md`
+
+**Network Device Failure**:
+- EdgeRouter: _[config backup location, restoration procedure]_
+- Mikrotik: _[config backup location, restoration procedure]_
+
+## Future Additions
+
+- Additional Proxmox nodes for clustering
+- Backup Proxmox Backup Server
+- Additional TrueNAS for replication
+
+## Maintenance Notes
+
+### Proxmox Updates
+
+- Update schedule: manual
+- Pre-update checklist: yolo
+
+### TrueNAS Updates
+
+- Update schedule: manual
+
+### Network Device Updates
+
+- EdgeRouter: manual
+- Mikrotik: manual
+
+## Monitoring
+
+**Infrastructure Monitoring**:
+
+TODO: Improve monitoring for physical hosts (proxmox, nas)
+TODO: Improve monitoring for networking equipment
+
+All NixOS VMs ship metrics to monitoring01 via node-exporter and logs via Promtail. See `/services/monitoring/` for the observability stack configuration.

docs/plans/auth-system-replacement.md

@@ -0,0 +1,192 @@
+# Authentication System Replacement Plan
+
+## Overview
+
+Replace the current auth01 setup (LLDAP + Authelia) with a modern, unified authentication solution. The current setup is not in active use, making this a good time to evaluate alternatives.
+
+## Goals
+
+1. **Central user database** - Manage users across all homelab hosts from a single source
+2. **Linux PAM/NSS integration** - Users can SSH into hosts using central credentials
+3. **UID/GID consistency** - Proper POSIX attributes for NAS share permissions
+4. **OIDC provider** - Single sign-on for homelab web services (Grafana, etc.)
+
+## Options Evaluated
+
+### OpenLDAP (raw)
+
+- **NixOS Support:** Good (`services.openldap` with `declarativeContents`)
+- **Pros:** Most widely supported, very flexible
+- **Cons:** LDIF format is painful, schema management is complex, no built-in OIDC, requires SSSD on each client
+- **Verdict:** Doesn't address LDAP complexity concerns
+
+### LLDAP + Authelia (current)
+
+- **NixOS Support:** Both have good modules
+- **Pros:** Already configured, lightweight, nice web UIs
+- **Cons:** Two services to manage, limited POSIX attribute support in LLDAP, requires SSSD on every client host
+- **Verdict:** Workable but has friction for NAS/UID goals
+
+### FreeIPA
+
+- **NixOS Support:** None
+- **Pros:** Full enterprise solution (LDAP + Kerberos + DNS + CA)
+- **Cons:** Extremely heavy, wants to own DNS, designed for Red Hat ecosystems, massive overkill for homelab
+- **Verdict:** Overkill, no NixOS support
+
+### Keycloak
+
+- **NixOS Support:** None
+- **Pros:** Good OIDC/SAML, nice UI
+- **Cons:** Primarily an identity broker not a user directory, poor POSIX support, heavy (Java)
+- **Verdict:** Wrong tool for Linux user management
+
+### Authentik
+
+- **NixOS Support:** None (would need Docker)
+- **Pros:** All-in-one with LDAP outpost and OIDC, modern UI
+- **Cons:** Heavy stack (Python + PostgreSQL + Redis), LDAP is a separate component
+- **Verdict:** Would work but requires Docker and is heavy
+
+### Kanidm
+
+- **NixOS Support:** Excellent - first-class module with PAM/NSS integration
+- **Pros:**
+  - Native PAM/NSS module (no SSSD needed)
+  - Built-in OIDC provider
+  - Optional LDAP interface for legacy services
+  - Declarative provisioning via NixOS (users, groups, OAuth2 clients)
+  - Modern, written in Rust
+  - Single service handles everything
+- **Cons:** Newer project, smaller community than LDAP
+- **Verdict:** Best fit for requirements
+
+### Pocket-ID
+
+- **NixOS Support:** Unknown
+- **Pros:** Very lightweight, passkey-first
+- **Cons:** No LDAP, no PAM/NSS integration - purely OIDC for web apps
+- **Verdict:** Doesn't solve Linux user management goal
+
+## Recommendation: Kanidm
+
+Kanidm is the recommended solution for the following reasons:
+
+| Requirement | Kanidm Support |
+|-------------|----------------|
+| Central user database | Native |
+| Linux PAM/NSS (host login) | Native NixOS module |
+| UID/GID for NAS | POSIX attributes supported |
+| OIDC for services | Built-in |
+| Declarative config | Excellent NixOS provisioning |
+| Simplicity | Modern API, LDAP optional |
+| NixOS integration | First-class |
+
+### Key NixOS Features
+
+**Server configuration:**
+```nix
+services.kanidm.enableServer = true;
+services.kanidm.serverSettings = {
+  domain = "home.2rjus.net";
+  origin = "https://auth.home.2rjus.net";
+  ldapbindaddress = "0.0.0.0:636";  # Optional LDAP interface
+};
+```
+
+**Declarative user provisioning:**
+```nix
+services.kanidm.provision.enable = true;
+services.kanidm.provision.persons.torjus = {
+  displayName = "Torjus";
+  groups = [ "admins" "nas-users" ];
+};
+```
+
+**Declarative OAuth2 clients:**
+```nix
+services.kanidm.provision.systems.oauth2.grafana = {
+  displayName = "Grafana";
+  originUrl = "https://grafana.home.2rjus.net/login/generic_oauth";
+  originLanding = "https://grafana.home.2rjus.net";
+};
+```
+
+**Client host configuration (add to system/):**
+```nix
+services.kanidm.enableClient = true;
+services.kanidm.enablePam = true;
+services.kanidm.clientSettings.uri = "https://auth.home.2rjus.net";
+```
+
+## NAS Integration
+
+### Current: TrueNAS CORE (FreeBSD)
+
+TrueNAS CORE has a built-in LDAP client. Kanidm's read-only LDAP interface will work for NFS share permissions:
+
+- **NFS shares**: Only need consistent UID/GID mapping - Kanidm's LDAP provides this
+- **No SMB requirement**: SMB would need Samba schema attributes (deprecated in TrueNAS 13.0+), but we're NFS-only
+
+Configuration approach:
+1. Enable Kanidm's LDAP interface (`ldapbindaddress = "0.0.0.0:636"`)
+2. Import internal CA certificate into TrueNAS
+3. Configure TrueNAS LDAP client with Kanidm's Base DN and bind credentials
+4. Users/groups appear in TrueNAS permission dropdowns
+
+Note: Kanidm's LDAP is read-only and uses LDAPS only (no StartTLS). This is fine for our use case.
+
+### Future: NixOS NAS
+
+When the NAS is migrated to NixOS, it becomes a first-class citizen:
+
+- Native Kanidm PAM/NSS integration (same as other hosts)
+- No LDAP compatibility layer needed
+- Full integration with the rest of the homelab
+
+This future migration path is a strong argument for Kanidm over LDAP-only solutions.
+
+## Implementation Steps
+
+1. **Create Kanidm service module** in `services/kanidm/`
+   - Server configuration
+   - TLS via internal ACME
+   - Vault secrets for admin passwords
+
+2. **Configure declarative provisioning**
+   - Define initial users and groups
+   - Set up POSIX attributes (UID/GID ranges)
+
+3. **Add OIDC clients** for homelab services
+   - Grafana
+   - Other services as needed
+
+4. **Create client module** in `system/` for PAM/NSS
+   - Enable on all hosts that need central auth
+   - Configure trusted CA
+
+5. **Test NAS integration**
+   - Configure TrueNAS LDAP client to connect to Kanidm
+   - Verify UID/GID mapping works with NFS shares
+
+6. **Migrate auth01**
+   - Remove LLDAP and Authelia services
+   - Deploy Kanidm
+   - Update DNS CNAMEs if needed
+
+7. **Documentation**
+   - User management procedures
+   - Adding new OAuth2 clients
+   - Troubleshooting PAM/NSS issues
+
+## Open Questions
+
+- What UID/GID range should be reserved for Kanidm-managed users?
+- Which hosts should have PAM/NSS enabled initially?
+- What OAuth2 clients are needed at launch?
+
+## References
+
+- [Kanidm Documentation](https://kanidm.github.io/kanidm/stable/)
+- [NixOS Kanidm Module](https://search.nixos.org/options?query=services.kanidm)
+- [Kanidm PAM/NSS Integration](https://kanidm.github.io/kanidm/stable/pam_and_nsswitch.html)

docs/plans/completed/dns-automation.md

@@ -0,0 +1,61 @@
+# DNS Automation
+
+**Status:** Completed (2026-02-04)
+
+**Goal:** Automatically generate DNS entries from host configurations
+
+**Approach:** Leverage Nix to generate zone file entries from flake host configurations
+
+Since most hosts use static IPs defined in their NixOS configurations, we can extract this information and automatically generate A records. This keeps DNS in sync with the actual host configs.
+
+## Implementation
+
+- [x] Add optional CNAME field to host configurations
+  - [x] Added `homelab.dns.cnames` option in `modules/homelab/dns.nix`
+  - [x] Added `homelab.dns.enable` to allow opting out (defaults to true)
+  - [x] Documented in CLAUDE.md
+- [x] Create Nix function to extract DNS records from all hosts
+  - [x] Created `lib/dns-zone.nix` with extraction functions
+  - [x] Parses each host's `networking.hostName` and `systemd.network.networks` IP configuration
+  - [x] Collects CNAMEs from `homelab.dns.cnames`
+  - [x] Filters out VPN interfaces (wg*, tun*, tap*, vti*)
+  - [x] Generates complete zone file with A and CNAME records
+- [x] Integrate auto-generated records into zone files
+  - [x] External hosts separated to `services/ns/external-hosts.nix`
+  - [x] Zone includes comments showing which records are auto-generated vs external
+- [x] Update zone file serial number automatically
+  - [x] Uses `self.sourceInfo.lastModified` (git commit timestamp)
+- [x] Test zone file validity after generation
+  - [x] NSD validates zone at build time via `nsd-checkzone`
+- [x] Deploy process documented
+  - [x] Merge to master, run auto-upgrade on ns1/ns2
+
+## Files Created/Modified
+
+| File | Purpose |
+|------|---------|
+| `modules/homelab/dns.nix` | Defines `homelab.dns.*` options |
+| `modules/homelab/default.nix` | Module import hub |
+| `lib/dns-zone.nix` | Zone generation functions |
+| `services/ns/external-hosts.nix` | Non-flake host records |
+| `services/ns/master-authorative.nix` | Uses generated zone |
+| `services/ns/secondary-authorative.nix` | Uses generated zone |
+
+## Usage
+
+View generated zone:
+```bash
+nix eval .#nixosConfigurations.ns1.config.services.nsd.zones.'"home.2rjus.net"'.data --raw
+```
+
+Add CNAMEs to a host:
+```nix
+homelab.dns.cnames = [ "alias1" "alias2" ];
+```
+
+Exclude a host from DNS:
+```nix
+homelab.dns.enable = false;
+```
+
+Add non-flake hosts: Edit `services/ns/external-hosts.nix`

docs/plans/completed/host-cleanup.md

@@ -0,0 +1,23 @@
+# Host Cleanup
+
+## Overview
+
+Remove decommissioned/unused host configurations that are no longer reachable on the network.
+
+## Hosts to review
+
+The following hosts return "no route to host" from Prometheus scraping and are likely no longer needed:
+
+- `media1` (10.69.12.82)
+- `ns3` (10.69.13.7)
+- `ns4` (10.69.13.8)
+- `nixos-test1` (10.69.13.10)
+
+## Steps
+
+1. Confirm each host is truly decommissioned (not just temporarily powered off)
+2. Remove host directory from `hosts/`
+3. Remove `nixosConfigurations` entry from `flake.nix`
+4. Remove host's age key from `.sops.yaml`
+5. Remove per-host secrets from `secrets/<hostname>/` if any
+6. Verify DNS zone and Prometheus targets no longer include the removed hosts after rebuild

docs/plans/completed/monitoring-gaps.md

@@ -0,0 +1,128 @@
+# Monitoring Gaps Audit
+
+## Overview
+
+Audit of services running in the homelab that lack monitoring coverage, either missing Prometheus scrape targets, alerting rules, or both.
+
+## Services with No Monitoring
+
+### PostgreSQL (`pgdb1`)
+
+- **Current state:** No scrape targets, no alert rules
+- **Risk:** A database outage would go completely unnoticed by Prometheus
+- **Recommendation:** Enable `services.prometheus.exporters.postgres` (available in nixpkgs). This exposes connection counts, query throughput, replication lag, table/index stats, and more. Add alerts for at least `postgres_down` (systemd unit state) and connection pool exhaustion.
+
+### Authelia (`auth01`)
+
+- **Current state:** No scrape targets, no alert rules
+- **Risk:** The authentication gateway being down blocks access to all proxied services
+- **Recommendation:** Authelia exposes Prometheus metrics natively at `/metrics`. Add a scrape target and at minimum an `authelia_down` systemd unit state alert.
+
+### LLDAP (`auth01`)
+
+- **Current state:** No scrape targets, no alert rules
+- **Risk:** LLDAP is a dependency of Authelia -- if LDAP is down, authentication breaks even if Authelia is running
+- **Recommendation:** Add an `lldap_down` systemd unit state alert. LLDAP does not expose Prometheus metrics natively, so systemd unit monitoring via node-exporter may be sufficient.
+
+### Vault / OpenBao (`vault01`)
+
+- **Current state:** No scrape targets, no alert rules
+- **Risk:** Secrets management service failures go undetected
+- **Recommendation:** OpenBao supports Prometheus telemetry output natively. Add a scrape target for the telemetry endpoint and alerts for `vault_down` (systemd unit) and seal status.
+
+### Gitea Actions Runner
+
+- **Current state:** No scrape targets, no alert rules
+- **Risk:** CI/CD failures go undetected
+- **Recommendation:** Add at minimum a systemd unit state alert. The runner itself has limited metrics exposure.
+
+## Services with Partial Monitoring
+
+### Jellyfin (`jelly01`)
+
+- **Current state:** Has scrape targets (port 8096), metrics are being collected, but zero alert rules
+- **Metrics available:** 184 metrics, all .NET runtime / ASP.NET Core level. No Jellyfin-specific metrics (active streams, library size, transcoding sessions). Key useful metrics:
+  - `microsoft_aspnetcore_hosting_failed_requests` - rate of HTTP errors
+  - `microsoft_aspnetcore_hosting_current_requests` - in-flight requests
+  - `process_working_set_bytes` - memory usage (~256 MB currently)
+  - `dotnet_gc_pause_ratio` - GC pressure
+  - `up{job="jellyfin"}` - basic availability
+- **Recommendation:** Add a `jellyfin_down` alert using either `up{job="jellyfin"} == 0` or systemd unit state. Consider alerting on sustained `failed_requests` rate increase.
+
+### NATS (`nats1`)
+
+- **Current state:** Has a `nats_down` alert (systemd unit state via node-exporter), but no NATS-specific metrics
+- **Metrics available:** NATS has a built-in `/metrics` endpoint exposing connection counts, message throughput, JetStream consumer lag, and more
+- **Recommendation:** Add a scrape target for the NATS metrics endpoint. Consider alerts for connection count spikes, slow consumers, and JetStream storage usage.
+
+### DNS - Unbound (`ns1`, `ns2`)
+
+- **Current state:** Has `unbound_down` alert (systemd unit state), but no DNS query metrics
+- **Available in nixpkgs:** `services.prometheus.exporters.unbound.enable` (package: `prometheus-unbound-exporter` v0.5.0). Exposes query counts, cache hit ratios, response types (SERVFAIL, NXDOMAIN), upstream latency.
+- **Recommendation:** Enable the unbound exporter on ns1/ns2. Add alerts for cache hit ratio drops and SERVFAIL rate spikes.
+
+### DNS - NSD (`ns1`, `ns2`)
+
+- **Current state:** Has `nsd_down` alert (systemd unit state), no NSD-specific metrics
+- **Available in nixpkgs:** Nothing. No exporter package or NixOS module. Community `nsd_exporter` exists but is not packaged.
+- **Recommendation:** The existing systemd unit alert is likely sufficient. NSD is a simple authoritative-only server with limited operational metrics. Not worth packaging a custom exporter for now.
+
+## Existing Monitoring (for reference)
+
+These services have adequate alerting and/or scrape targets:
+
+| Service | Scrape Targets | Alert Rules |
+|---|---|---|
+| Monitoring stack (Prometheus, Grafana, Loki, Tempo, Pyroscope) | Yes | 7 alerts |
+| Home Assistant (+ Zigbee2MQTT, Mosquitto) | Yes (port 8123) | 3 alerts |
+| HTTP Proxy (Caddy) | Yes (port 80) | 3 alerts |
+| Nix Cache (Harmonia, build-flakes) | Via Caddy | 4 alerts |
+| CA (step-ca) | Yes (port 9000) | 4 certificate alerts |
+
+## Per-Service Resource Metrics (systemd-exporter)
+
+### Current State
+
+No per-service CPU, memory, or IO metrics are collected. The existing node-exporter systemd collector only provides unit state (active/inactive/failed), socket stats, and timer triggers. While systemd tracks per-unit resource usage via cgroups internally (visible in `systemctl status` and `systemd-cgtop`), this data is not exported to Prometheus.
+
+### Available Solution
+
+The `prometheus-systemd-exporter` package (v0.7.0) is available in nixpkgs with a ready-made NixOS module:
+
+```nix
+services.prometheus.exporters.systemd.enable = true;
+```
+
+**Options:** `enable`, `port`, `extraFlags`, `user`, `group`
+
+This exporter reads cgroup data and exposes per-unit metrics including:
+- CPU seconds consumed per service
+- Memory usage per service
+- Task/process counts per service
+- Restart counts
+- IO usage
+
+### Recommendation
+
+Enable on all hosts via the shared `system/` config (same pattern as node-exporter). Add a corresponding scrape job on monitoring01. This would give visibility into resource consumption per service across the fleet, useful for capacity planning and diagnosing noisy-neighbor issues on shared hosts.
+
+## Suggested Priority
+
+1. **PostgreSQL** - Critical infrastructure, easy to add with existing nixpkgs module
+2. **Authelia + LLDAP** - Auth outage affects all proxied services
+3. **Unbound exporter** - Ready-to-go NixOS module, just needs enabling
+4. **Jellyfin alerts** - Metrics already collected, just needs alert rules
+5. **NATS metrics** - Built-in endpoint, just needs a scrape target
+6. **Vault/OpenBao** - Native telemetry support
+7. **Actions Runner** - Lower priority, basic systemd alert sufficient
+
+## Node-Exporter Targets Currently Down
+
+Noted during audit -- these node-exporter targets are failing:
+
+- `nixos-test1.home.2rjus.net:9100` - no route to host
+- `media1.home.2rjus.net:9100` - no route to host
+- `ns3.home.2rjus.net:9100` - no route to host
+- `ns4.home.2rjus.net:9100` - no route to host
+
+These may be decommissioned or powered-off hosts that should be removed from the scrape config.

docs/plans/completed/nixos-exporter.md

@@ -0,0 +1,176 @@
+# NixOS Prometheus Exporter
+
+## Overview
+
+Build a generic Prometheus exporter for NixOS-specific metrics. This exporter should be useful for any NixOS deployment, not just our homelab.
+
+## Goal
+
+Provide visibility into NixOS system state that standard exporters don't cover:
+- Generation management (count, age, current vs booted)
+- Flake input freshness
+- Upgrade status
+
+## Metrics
+
+### Core Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `nixos_generation_count` | Number of system generations | Count entries in `/nix/var/nix/profiles/system-*` |
+| `nixos_current_generation` | Active generation number | Parse `readlink /run/current-system` |
+| `nixos_booted_generation` | Generation that was booted | Parse `/run/booted-system` |
+| `nixos_generation_age_seconds` | Age of current generation | File mtime of current system profile |
+| `nixos_config_mismatch` | 1 if booted != current, 0 otherwise | Compare symlink targets |
+
+### Flake Metrics (optional collector)
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `nixos_flake_input_age_seconds` | Age of each flake.lock input | Parse `lastModified` from flake.lock |
+| `nixos_flake_input_info` | Info gauge with rev label | Parse `rev` from flake.lock |
+
+Labels: `input` (e.g., "nixpkgs", "home-manager")
+
+### Future Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `nixos_upgrade_pending` | 1 if remote differs from local | Compare flake refs (expensive) |
+| `nixos_store_size_bytes` | Size of /nix/store | `du` or filesystem stats |
+| `nixos_store_path_count` | Number of store paths | Count entries |
+
+## Architecture
+
+Single binary with optional collectors enabled via config or flags.
+
+```
+nixos-exporter
+├── main.go
+├── collector/
+│   ├── generation.go    # Core generation metrics
+│   └── flake.go         # Flake input metrics
+└── config/
+    └── config.go
+```
+
+## Configuration
+
+```yaml
+listen_addr: ":9971"
+collectors:
+  generation:
+    enabled: true
+  flake:
+    enabled: false
+    lock_path: "/etc/nixos/flake.lock"  # or auto-detect from /run/current-system
+```
+
+Command-line alternative:
+```bash
+nixos-exporter --listen=:9971 --collector.flake --flake.lock-path=/etc/nixos/flake.lock
+```
+
+## NixOS Module
+
+```nix
+services.prometheus.exporters.nixos = {
+  enable = true;
+  port = 9971;
+  collectors = [ "generation" "flake" ];
+  flake.lockPath = "/etc/nixos/flake.lock";
+};
+```
+
+The module should integrate with nixpkgs' existing `services.prometheus.exporters.*` pattern.
+
+## Implementation
+
+### Language
+
+Go - mature prometheus client library, single static binary, easy cross-compilation.
+
+### Phase 1: Core
+1. Create git repository
+2. Implement generation collector (count, current, booted, age, mismatch)
+3. Basic HTTP server with `/metrics` endpoint
+4. NixOS module
+
+### Phase 2: Flake Collector
+1. Parse flake.lock JSON format
+2. Extract lastModified timestamps per input
+3. Add input labels
+
+### Phase 3: Packaging
+1. Add to nixpkgs or publish as flake
+2. Documentation
+3. Example Grafana dashboard
+
+## Example Output
+
+```
+# HELP nixos_generation_count Total number of system generations
+# TYPE nixos_generation_count gauge
+nixos_generation_count 47
+
+# HELP nixos_current_generation Currently active generation number
+# TYPE nixos_current_generation gauge
+nixos_current_generation 47
+
+# HELP nixos_booted_generation Generation that was booted
+# TYPE nixos_booted_generation gauge
+nixos_booted_generation 46
+
+# HELP nixos_generation_age_seconds Age of current generation in seconds
+# TYPE nixos_generation_age_seconds gauge
+nixos_generation_age_seconds 3600
+
+# HELP nixos_config_mismatch 1 if booted generation differs from current
+# TYPE nixos_config_mismatch gauge
+nixos_config_mismatch 1
+
+# HELP nixos_flake_input_age_seconds Age of flake input in seconds
+# TYPE nixos_flake_input_age_seconds gauge
+nixos_flake_input_age_seconds{input="nixpkgs"} 259200
+nixos_flake_input_age_seconds{input="home-manager"} 86400
+```
+
+## Alert Examples
+
+```yaml
+- alert: NixOSConfigStale
+  expr: nixos_generation_age_seconds > 7 * 24 * 3600
+  for: 1h
+  labels:
+    severity: warning
+  annotations:
+    summary: "NixOS config on {{ $labels.instance }} is over 7 days old"
+
+- alert: NixOSRebootRequired
+  expr: nixos_config_mismatch == 1
+  for: 24h
+  labels:
+    severity: info
+  annotations:
+    summary: "{{ $labels.instance }} needs reboot to apply config"
+
+- alert: NixpkgsInputStale
+  expr: nixos_flake_input_age_seconds{input="nixpkgs"} > 30 * 24 * 3600
+  for: 1d
+  labels:
+    severity: info
+  annotations:
+    summary: "nixpkgs input on {{ $labels.instance }} is over 30 days old"
+```
+
+## Open Questions
+
+- [ ] How to detect flake.lock path automatically? (check /run/current-system for flake info)
+- [ ] Should generation collector need root? (probably not, just reading symlinks)
+- [ ] Include in nixpkgs or distribute as standalone flake?
+
+## Notes
+
+- Port 9971 suggested (9970 reserved for homelab-exporter)
+- Keep scope focused on NixOS-specific metrics - don't duplicate node-exporter
+- Consider submitting to prometheus exporter registry once stable

docs/plans/completed/sops-to-openbao-migration.md

@@ -0,0 +1,86 @@
+# Sops to OpenBao Secrets Migration Plan
+
+## Status: Complete (except ca, deferred)
+
+## Remaining sops cleanup
+
+The `sops-nix` flake input, `system/sops.nix`, `.sops.yaml`, and `secrets/` directory are
+still present because `ca` still uses sops for its step-ca secrets (5 secrets in
+`services/ca/default.nix`). The `services/authelia/` and `services/lldap/` modules also
+reference sops but are only used by auth01 (decommissioned).
+
+Once `ca` is migrated to OpenBao PKI (Phase 4c in host-migration-to-opentofu.md), remove:
+- `sops-nix` input from `flake.nix`
+- `sops-nix.nixosModules.sops` from all host module lists in `flake.nix`
+- `inherit sops-nix` from all specialArgs in `flake.nix`
+- `system/sops.nix` and its import in `system/default.nix`
+- `.sops.yaml`
+- `secrets/` directory
+- All `sops.secrets.*` declarations in `services/ca/`, `services/authelia/`, `services/lldap/`
+
+## Overview
+
+Migrate all hosts from sops-nix secrets to OpenBao (vault) secrets management. Pilot with ha1, then roll out to remaining hosts in waves.
+
+## Pre-requisites (completed)
+
+1. Hardcoded root password hash in `system/root-user.nix` (removes sops dependency for all hosts)
+2. Added `extractKey` option to `system/vault-secrets.nix` (extracts single key as file)
+
+## Deployment Order
+
+### Pilot: ha1
+- Terraform: shared/backup/password secret, ha1 AppRole policy
+- Provision AppRole credentials via `playbooks/provision-approle.yml`
+- NixOS: vault.enable + backup-helper vault secret
+
+### Wave 1: nats1, jelly01, pgdb1
+- No service secrets (only root password, already handled)
+- Just need AppRole policies + credential provisioning
+
+### Wave 2: monitoring01
+- 3 secrets: backup password, nats nkey, pve-exporter config
+- Updates: alerttonotify.nix, pve.nix, configuration.nix
+
+### Wave 3: ns1, then ns2 (critical - deploy ns1 first, verify, then ns2)
+- DNS zone transfer key (shared/dns/xfer-key)
+
+### Wave 4: http-proxy
+- WireGuard private key
+
+### Wave 5: nix-cache01
+- Cache signing key + Gitea Actions token
+
+### Wave 6: ca (DEFERRED - waiting for PKI migration)
+
+### Skipped: auth01 (decommissioned)
+
+## Terraform variables needed
+
+User must extract from sops and add to `terraform/vault/terraform.tfvars`:
+
+| Variable | Source |
+|----------|--------|
+| `backup_helper_secret` | `sops -d secrets/secrets.yaml` |
+| `ns_xfer_key` | `sops -d secrets/secrets.yaml` |
+| `nats_nkey` | `sops -d secrets/secrets.yaml` |
+| `pve_exporter_config` | `sops -d secrets/monitoring01/pve-exporter.yaml` |
+| `wireguard_private_key` | `sops -d secrets/http-proxy/wireguard.yaml` |
+| `cache_signing_key` | `sops -d secrets/nix-cache01/cache-secret` |
+| `actions_token_1` | `sops -d secrets/nix-cache01/actions_token_1` |
+
+## Provisioning AppRole credentials
+
+```bash
+export BAO_ADDR='https://vault01.home.2rjus.net:8200'
+export BAO_TOKEN='<root-token>'
+nix develop -c ansible-playbook playbooks/provision-approle.yml -e hostname=<host>
+```
+
+## Verification (per host)
+
+1. `systemctl status vault-secret-*` - all secret fetch services succeeded
+2. Check secret files exist at expected paths with correct permissions
+3. Verify dependent services are running
+4. Check `/var/lib/vault/cache/` is populated (fallback ready)
+5. Reboot host to verify boot-time secret fetching works

docs/plans/completed/zigbee-sensor-battery-monitoring.md

@@ -0,0 +1,109 @@
+# Zigbee Sensor Battery Monitoring
+
+**Status:** Completed
+**Branch:** `zigbee-battery-fix`
+**Commit:** `c515a6b home-assistant: fix zigbee sensor battery reporting`
+
+## Problem
+
+Three Aqara Zigbee temperature sensors report `battery: 0` in their MQTT payload, making the `hass_sensor_battery_percent` Prometheus metric useless for battery monitoring on these devices.
+
+Affected sensors:
+- **Temp Living Room** (`0x54ef441000a54d3c`) — WSDCGQ12LM
+- **Temp Office** (`0x54ef441000a547bd`) — WSDCGQ12LM
+- **temp_server** (`0x54ef441000a564b6`) — WSDCGQ12LM
+
+The **Temp Bedroom** sensor (`0x00124b0025495463`) is a SONOFF SNZB-02 and reports battery correctly.
+
+## Findings
+
+- All three sensors are actively reporting temperature, humidity, and pressure data — they are not dead.
+- The Zigbee2MQTT payload includes a `voltage` field (e.g., `2707` = 2.707V), which indicates healthy battery levels (~40-60% for a CR2032 coin cell).
+- CR2032 voltage reference: ~3.0V fresh, ~2.7V mid-life, ~2.1V dead.
+- The `voltage` field is not exposed as a Prometheus metric — it exists only in the MQTT payload.
+- This is a known firmware quirk with some Aqara WSDCGQ12LM sensors that always report 0% battery.
+
+## Device Inventory
+
+Full list of Zigbee devices on ha1 (12 total):
+
+| Device | IEEE Address | Model | Type |
+|--------|-------------|-------|------|
+| temp_server | 0x54ef441000a564b6 | WSDCGQ12LM | Temperature sensor (battery fix applied) |
+| (Temp Living Room) | 0x54ef441000a54d3c | WSDCGQ12LM | Temperature sensor (battery fix applied) |
+| (Temp Office) | 0x54ef441000a547bd | WSDCGQ12LM | Temperature sensor (battery fix applied) |
+| (Temp Bedroom) | 0x00124b0025495463 | SNZB-02 | Temperature sensor (battery works) |
+| (Water leak) | 0x54ef4410009ac117 | SJCGQ12LM | Water leak sensor |
+| btn_livingroom | 0x54ef441000a1f907 | WXKG13LM | Wireless mini switch |
+| btn_bedroom | 0x54ef441000a1ee71 | WXKG13LM | Wireless mini switch |
+| (Hue bulb) | 0x001788010dc35d06 | 9290024688 | Hue E27 1100lm (Router) |
+| (Hue bulb) | 0x001788010dc5f003 | 9290024688 | Hue E27 1100lm (Router) |
+| (Hue ceiling) | 0x001788010e371aa4 | 915005997301 | Hue Infuse medium (Router) |
+| (Hue ceiling) | 0x001788010d253b99 | 915005997301 | Hue Infuse medium (Router) |
+| (Hue wall) | 0x001788010d1b599a | 929003052901 | Hue Sana wall light (Router, transition=5) |
+
+## Implementation
+
+### Solution 1: Calculate battery from voltage in Zigbee2MQTT (Implemented)
+
+Override the Home Assistant battery entity's `value_template` in Zigbee2MQTT device configuration to calculate battery percentage from voltage.
+
+**Formula:** `(voltage - 2100) / 9` (maps 2100-3000mV to 0-100%)
+
+**Changes in `services/home-assistant/default.nix`:**
+- Device configuration moved from external `devices.yaml` to inline NixOS config
+- Three affected sensors have `homeassistant.sensor_battery.value_template` override
+- All 12 devices now declaratively managed
+
+**Expected battery values based on current voltages:**
+| Sensor | Voltage | Expected Battery |
+|--------|---------|------------------|
+| Temp Living Room | 2710 mV | ~68% |
+| Temp Office | 2658 mV | ~62% |
+| temp_server | 2765 mV | ~74% |
+
+### Solution 2: Alert on sensor staleness (Implemented)
+
+Added Prometheus alert `zigbee_sensor_stale` in `services/monitoring/rules.yml` that fires when a Zigbee temperature sensor hasn't updated in over 1 hour. This provides defense-in-depth for detecting dead sensors regardless of battery reporting accuracy.
+
+**Alert details:**
+- Expression: `(time() - hass_last_updated_time_seconds{entity=~"sensor\\.(0x[0-9a-f]+|temp_server)_temperature"}) > 3600`
+- Severity: warning
+- For: 5m
+
+## Pre-Deployment Verification
+
+### Backup Verification
+
+Before deployment, verified ha1 backup configuration and ran manual backup:
+
+**Backup paths:**
+- `/var/lib/hass` ✓
+- `/var/lib/zigbee2mqtt` ✓
+- `/var/lib/mosquitto` ✓
+
+**Manual backup (2026-02-05 22:45:23):**
+- Snapshot ID: `59704dfa`
+- Files: 77 total (0 new, 13 changed, 64 unmodified)
+- Data: 62.635 MiB processed, 6.928 MiB stored (compressed)
+
+### Other directories reviewed
+
+- `/var/lib/vault` — Contains AppRole credentials; not backed up (can be re-provisioned via Ansible)
+- `/var/lib/sops-nix` — Legacy; ha1 uses Vault now
+
+## Post-Deployment Steps
+
+After deploying to ha1:
+
+1. Restart zigbee2mqtt service (automatic on NixOS rebuild)
+2. In Home Assistant, the battery entities may need to be re-discovered:
+   - Go to Settings → Devices & Services → MQTT
+   - The new `value_template` should take effect after entity re-discovery
+   - If not, try disabling and re-enabling the battery entities
+
+## Notes
+
+- Device configuration is now declarative in NixOS. Future device additions via Zigbee2MQTT frontend will need to be added to the NixOS config to persist.
+- The `devices.yaml` file on ha1 will be overwritten on service start but can be removed after confirming the new config works.
+- The NixOS zigbee2mqtt module defaults to `devices = "devices.yaml"` but our explicit inline config overrides this.

docs/plans/homelab-exporter.md

@@ -0,0 +1,179 @@
+# Homelab Infrastructure Exporter
+
+## Overview
+
+Build a Prometheus exporter for metrics specific to our homelab infrastructure. Unlike the generic nixos-exporter, this covers services and patterns unique to our environment.
+
+## Current State
+
+### Existing Exporters
+- **node-exporter** (all hosts): System metrics
+- **systemd-exporter** (all hosts): Service restart counts, IP accounting
+- **labmon** (monitoring01): TLS certificate monitoring, step-ca health
+- **Service-specific**: unbound, postgres, nats, jellyfin, home-assistant, caddy, step-ca
+
+### Gaps
+- No visibility into Vault/OpenBao lease expiry
+- No ACME certificate expiry from internal CA
+- No Proxmox guest agent metrics from inside VMs
+
+## Metrics
+
+### Vault/OpenBao Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_vault_token_expiry_seconds` | Seconds until AppRole token expires | Token metadata or lease file |
+| `homelab_vault_token_renewable` | 1 if token is renewable | Token metadata |
+
+Labels: `role` (AppRole name)
+
+### ACME Certificate Metrics
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_acme_cert_expiry_seconds` | Seconds until certificate expires | Parse cert from `/var/lib/acme/*/cert.pem` |
+| `homelab_acme_cert_not_after` | Unix timestamp of cert expiry | Certificate NotAfter field |
+
+Labels: `domain`, `issuer`
+
+Note: labmon already monitors external TLS endpoints. This covers local ACME-managed certs.
+
+### Proxmox Guest Metrics (future)
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_proxmox_guest_info` | Info gauge with VM ID, name | QEMU guest agent |
+| `homelab_proxmox_guest_agent_running` | 1 if guest agent is responsive | Agent ping |
+
+### DNS Zone Metrics (future)
+
+| Metric | Description | Source |
+|--------|-------------|--------|
+| `homelab_dns_zone_serial` | Current zone serial number | DNS AXFR or zone file |
+
+Labels: `zone`
+
+## Architecture
+
+Single binary with collectors enabled via config. Runs on hosts that need specific collectors.
+
+```
+homelab-exporter
+├── main.go
+├── collector/
+│   ├── vault.go     # Vault/OpenBao token metrics
+│   ├── acme.go      # ACME certificate metrics
+│   └── proxmox.go   # Proxmox guest agent (future)
+└── config/
+    └── config.go
+```
+
+## Configuration
+
+```yaml
+listen_addr: ":9970"
+collectors:
+  vault:
+    enabled: true
+    token_path: "/var/lib/vault/token"
+  acme:
+    enabled: true
+    cert_dirs:
+      - "/var/lib/acme"
+  proxmox:
+    enabled: false
+```
+
+## NixOS Module
+
+```nix
+services.homelab-exporter = {
+  enable = true;
+  port = 9970;
+  collectors = {
+    vault = {
+      enable = true;
+      tokenPath = "/var/lib/vault/token";
+    };
+    acme = {
+      enable = true;
+      certDirs = [ "/var/lib/acme" ];
+    };
+  };
+};
+
+# Auto-register scrape target
+homelab.monitoring.scrapeTargets = [{
+  job_name = "homelab-exporter";
+  port = 9970;
+}];
+```
+
+## Integration
+
+### Deployment
+
+Deploy on hosts that have relevant data:
+- **All hosts with ACME certs**: acme collector
+- **All hosts with Vault**: vault collector
+- **Proxmox VMs**: proxmox collector (when implemented)
+
+### Relationship with nixos-exporter
+
+These are complementary:
+- **nixos-exporter** (port 9971): Generic NixOS metrics, deploy everywhere
+- **homelab-exporter** (port 9970): Infrastructure-specific, deploy selectively
+
+Both can run on the same host if needed.
+
+## Implementation
+
+### Language
+
+Go - consistent with labmon and nixos-exporter.
+
+### Phase 1: Core + ACME
+1. Create git repository (git.t-juice.club/torjus/homelab-exporter)
+2. Implement ACME certificate collector
+3. HTTP server with `/metrics`
+4. NixOS module
+
+### Phase 2: Vault Collector
+1. Implement token expiry detection
+2. Handle missing/expired tokens gracefully
+
+### Phase 3: Dashboard
+1. Create Grafana dashboard for infrastructure health
+2. Add to existing monitoring service module
+
+## Alert Examples
+
+```yaml
+- alert: VaultTokenExpiringSoon
+  expr: homelab_vault_token_expiry_seconds < 3600
+  for: 5m
+  labels:
+    severity: warning
+  annotations:
+    summary: "Vault token on {{ $labels.instance }} expires in < 1 hour"
+
+- alert: ACMECertExpiringSoon
+  expr: homelab_acme_cert_expiry_seconds < 7 * 24 * 3600
+  for: 1h
+  labels:
+    severity: warning
+  annotations:
+    summary: "ACME cert {{ $labels.domain }} on {{ $labels.instance }} expires in < 7 days"
+```
+
+## Open Questions
+
+- [ ] How to read Vault token expiry without re-authenticating?
+- [ ] Should ACME collector also check key/cert match?
+
+## Notes
+
+- Port 9970 (labmon uses 9969, nixos-exporter will use 9971)
+- Keep infrastructure-specific logic here, generic NixOS stuff in nixos-exporter
+- Consider merging Proxmox metrics with pve-exporter if overlap is significant

docs/plans/host-migration-to-opentofu.md

@@ -0,0 +1,224 @@
+# Host Migration to OpenTofu
+
+## Overview
+
+Migrate all existing hosts (provisioned manually before the OpenTofu pipeline) into the new
+OpenTofu-managed provisioning workflow. Hosts are categorized by their state requirements:
+stateless hosts are simply recreated, stateful hosts require backup and restore, and some
+hosts are decommissioned or deferred.
+
+## Current State
+
+Hosts already managed by OpenTofu: `vault01`, `testvm01`, `vaulttest01`
+
+Hosts to migrate:
+
+| Host | Category | Notes |
+|------|----------|-------|
+| ns1 | Stateless | Primary DNS, recreate |
+| ns2 | Stateless | Secondary DNS, recreate |
+| nix-cache01 | Stateless | Binary cache, recreate |
+| http-proxy | Stateless | Reverse proxy, recreate |
+| nats1 | Stateless | Messaging, recreate |
+| auth01 | Decommission | No longer in use |
+| ha1 | Stateful | Home Assistant + Zigbee2MQTT + Mosquitto |
+| monitoring01 | Stateful | Prometheus, Grafana, Loki |
+| jelly01 | Stateful | Jellyfin metadata, watch history, config |
+| pgdb1 | Stateful | PostgreSQL databases |
+| jump | Decommission | No longer needed |
+| ca | Deferred | Pending Phase 4c PKI migration to OpenBao |
+
+## Phase 1: Backup Preparation
+
+Before migrating any stateful host, ensure restic backups are in place and verified.
+
+### 1a. Expand monitoring01 Grafana Backup
+
+The existing backup only covers `/var/lib/grafana/plugins` and a sqlite dump of `grafana.db`.
+Expand to back up all of `/var/lib/grafana/` to capture config directory and any other state.
+
+### 1b. Add Jellyfin Backup to jelly01
+
+No backup currently exists. Add a restic backup job for `/var/lib/jellyfin/` which contains:
+- `config/` — server settings, library configuration
+- `data/` — user watch history, playback state, library metadata
+
+Media files are on the NAS (`nas.home.2rjus.net:/mnt/hdd-pool/media`) and do not need backup.
+The cache directory (`/var/cache/jellyfin/`) does not need backup — it regenerates.
+
+### 1c. Add PostgreSQL Backup to pgdb1
+
+No backup currently exists. Add a restic backup job with a `pg_dumpall` pre-hook to capture
+all databases and roles. The dump should be piped through restic's stdin backup (similar to
+the Grafana DB dump pattern on monitoring01).
+
+### 1d. Verify Existing ha1 Backup
+
+ha1 already backs up `/var/lib/hass`, `/var/lib/zigbee2mqtt`, `/var/lib/mosquitto`. Verify
+these backups are current and restorable before proceeding with migration.
+
+### 1e. Verify All Backups
+
+After adding/expanding backup jobs:
+1. Trigger a manual backup run on each host
+2. Verify backup integrity with `restic check`
+3. Test a restore to a temporary location to confirm data is recoverable
+
+## Phase 2: Declare pgdb1 Databases in Nix
+
+Before migrating pgdb1, audit the manually-created databases and users on the running
+instance, then declare them in the Nix configuration using `ensureDatabases` and
+`ensureUsers`. This makes the PostgreSQL setup reproducible on the new host.
+
+Steps:
+1. SSH to pgdb1, run `\l` and `\du` in psql to list databases and roles
+2. Add `ensureDatabases` and `ensureUsers` to `services/postgres/postgres.nix`
+3. Document any non-default PostgreSQL settings or extensions per database
+
+After reprovisioning, the databases will be created by NixOS, and data restored from the
+`pg_dumpall` backup.
+
+## Phase 3: Stateless Host Migration
+
+These hosts have no meaningful state and can be recreated fresh. For each host:
+
+1. Add the host definition to `terraform/vms.tf` (using `create-host` or manually)
+2. Commit and push to master
+3. Run `tofu apply` to provision the new VM
+4. Wait for bootstrap to complete (VM pulls config from master and reboots)
+5. Verify the host is functional
+6. Decommission the old VM in Proxmox
+
+### Migration Order
+
+Migrate stateless hosts in an order that minimizes disruption:
+
+1. **nix-cache01** — low risk, no downstream dependencies during migration
+2. **nats1** — low risk, verify no persistent JetStream streams first
+4. **http-proxy** — brief disruption to proxied services, migrate during low-traffic window
+5. **ns1, ns2** — migrate one at a time, verify DNS resolution between each
+
+For ns1/ns2: migrate ns2 first (secondary), verify AXFR works, then migrate ns1. All hosts
+use both ns1 and ns2 as resolvers, so one being down briefly is tolerable.
+
+## Phase 4: Stateful Host Migration
+
+For each stateful host, the procedure is:
+
+1. Trigger a final restic backup
+2. Stop services on the old host (to prevent state drift during migration)
+3. Provision the new VM via `tofu apply`
+4. Wait for bootstrap to complete
+5. Stop the relevant services on the new host
+6. Restore data from restic backup
+7. Start services and verify functionality
+8. Decommission the old VM
+
+### 4a. pgdb1
+
+1. Run final `pg_dumpall` backup via restic
+2. Stop PostgreSQL on the old host
+3. Provision new pgdb1 via OpenTofu
+4. After bootstrap, NixOS creates the declared databases/users
+5. Restore data with `pg_restore` or `psql < dumpall.sql`
+6. Verify database connectivity from gunter (`10.69.30.105`)
+7. Decommission old VM
+
+### 4b. monitoring01
+
+1. Run final Grafana backup
+2. Provision new monitoring01 via OpenTofu
+3. After bootstrap, restore `/var/lib/grafana/` from restic
+4. Restart Grafana, verify dashboards and datasources are intact
+5. Prometheus and Loki start fresh with empty data (acceptable)
+6. Verify all scrape targets are being collected
+7. Decommission old VM
+
+### 4c. jelly01
+
+1. Run final Jellyfin backup
+2. Provision new jelly01 via OpenTofu
+3. After bootstrap, restore `/var/lib/jellyfin/` from restic
+4. Verify NFS mount to NAS is working
+5. Start Jellyfin, verify watch history and library metadata are present
+6. Decommission old VM
+
+### 4d. ha1
+
+1. Verify latest restic backup is current
+2. Stop Home Assistant, Zigbee2MQTT, and Mosquitto on old host
+3. Provision new ha1 via OpenTofu
+4. After bootstrap, restore `/var/lib/hass`, `/var/lib/zigbee2mqtt`, `/var/lib/mosquitto`
+5. Start services, verify Home Assistant is functional
+6. Verify Zigbee devices are still paired and communicating
+7. Decommission old VM
+
+**Note:** ha1 currently has 2 GB RAM, which is consistently tight. Average memory usage has
+climbed from ~57% (30-day avg) to ~70% currently, with a 30-day low of only 187 MB free.
+Consider increasing to 4 GB when reprovisioning to allow headroom for additional integrations.
+
+**Note:** ha1 is the highest-risk migration due to Zigbee device pairings. The Zigbee
+coordinator state in `/var/lib/zigbee2mqtt` should preserve pairings, but verify on a
+non-critical time window.
+
+**USB Passthrough:** The ha1 VM has a USB device passed through from the Proxmox hypervisor
+(the Zigbee coordinator). The new VM must be configured with the same USB passthrough in
+OpenTofu/Proxmox. Verify the USB device ID on the hypervisor and add the appropriate
+`usb` block to the VM definition in `terraform/vms.tf`. The USB device must be passed
+through before starting Zigbee2MQTT on the new host.
+
+## Phase 5: Decommission jump and auth01 Hosts
+
+### jump
+1. Verify nothing depends on the jump host (no SSH proxy configs pointing to it, etc.)
+2. Remove host configuration from `hosts/jump/`
+3. Remove from `flake.nix`
+4. Remove any secrets in `secrets/jump/`
+5. Remove from `.sops.yaml`
+6. Destroy the VM in Proxmox
+7. Commit cleanup
+
+### auth01
+1. Remove host configuration from `hosts/auth01/`
+2. Remove from `flake.nix`
+3. Remove any secrets in `secrets/auth01/`
+4. Remove from `.sops.yaml`
+5. Remove `services/authelia/` and `services/lldap/` (only used by auth01)
+6. Destroy the VM in Proxmox
+7. Commit cleanup
+
+## Phase 6: Decommission ca Host (Deferred)
+
+Deferred until Phase 4c (PKI migration to OpenBao) is complete. Once all hosts use the
+OpenBao ACME endpoint for certificates, the step-ca host can be decommissioned following
+the same cleanup steps as the jump host.
+
+## Phase 7: Remove sops-nix
+
+Once `ca` is decommissioned (Phase 6), `sops-nix` is no longer used by any host. Remove
+all remnants:
+- `sops-nix` input from `flake.nix` and `flake.lock`
+- `sops-nix.nixosModules.sops` from all host module lists in `flake.nix`
+- `inherit sops-nix` from all specialArgs in `flake.nix`
+- `system/sops.nix` and its import in `system/default.nix`
+- `.sops.yaml`
+- `secrets/` directory
+- All `sops.secrets.*` declarations in `services/ca/`, `services/authelia/`, `services/lldap/`
+- Template scripts that generate age keys for sops (`hosts/template/scripts.nix`,
+  `hosts/template2/scripts.nix`)
+
+See `docs/plans/completed/sops-to-openbao-migration.md` for full context.
+
+## Notes
+
+- Each host migration should be done individually, not in bulk, to limit blast radius
+- Keep the old VM running until the new one is verified — do not destroy prematurely
+- The old VMs use IPs that the new VMs need, so the old VM must be shut down before
+  the new one is provisioned (or use a temporary IP and swap after verification)
+- Stateful migrations should be done during low-usage windows
+- After all migrations are complete, the only hosts not in OpenTofu will be ca (deferred)
+- Since many hosts are being recreated, this is a good opportunity to establish consistent
+  hostname naming conventions before provisioning the new VMs. Current naming is inconsistent
+  (e.g. `ns1` vs `nix-cache01`, `ha1` vs `auth01`, `pgdb1` vs `http-proxy`). Decide on a
+  convention before starting migrations — e.g. whether to always use numeric suffixes, a
+  consistent format like `service-NN`, role-based vs function-based names, etc.

docs/plans/long-term-metrics-storage.md

@@ -0,0 +1,122 @@
+# Long-Term Metrics Storage Options
+
+## Problem Statement
+
+Current Prometheus configuration retains metrics for 30 days (`retentionTime = "30d"`). Extending retention further raises disk usage concerns on the homelab hypervisor with limited local storage.
+
+Prometheus does not support downsampling - it stores all data at full resolution until the retention period expires, then deletes it entirely.
+
+## Current Configuration
+
+Location: `services/monitoring/prometheus.nix`
+
+- **Retention**: 30 days
+- **Scrape interval**: 15s
+- **Features**: Alertmanager, Pushgateway, auto-generated scrape configs from flake hosts
+- **Storage**: Local disk on monitoring01
+
+## Options Evaluated
+
+### Option 1: VictoriaMetrics
+
+VictoriaMetrics is a Prometheus-compatible TSDB with significantly better compression (5-10x smaller storage footprint).
+
+**NixOS Options Available:**
+- `services.victoriametrics.enable`
+- `services.victoriametrics.prometheusConfig` - accepts Prometheus scrape config format
+- `services.victoriametrics.retentionPeriod` - e.g., "6m" for 6 months
+- `services.vmagent` - dedicated scraping agent
+- `services.vmalert` - alerting rules evaluation
+
+**Pros:**
+- Simple migration - single service replacement
+- Same PromQL query language - Grafana dashboards work unchanged
+- Same scrape config format - existing auto-generated configs work as-is
+- 5-10x better compression means 30 days of Prometheus data could become 180+ days
+- Lightweight, single binary
+
+**Cons:**
+- No automatic downsampling (relies on compression alone)
+- Alerting requires switching to vmalert instead of Prometheus alertmanager integration
+- Would need to migrate existing data or start fresh
+
+**Migration Steps:**
+1. Replace `services.prometheus` with `services.victoriametrics`
+2. Move scrape configs to `prometheusConfig`
+3. Set up `services.vmalert` for alerting rules
+4. Update Grafana datasource to VictoriaMetrics port (8428)
+5. Keep Alertmanager for notification routing
+
+### Option 2: Thanos
+
+Thanos extends Prometheus with long-term storage and automatic downsampling by uploading data to object storage.
+
+**NixOS Options Available:**
+- `services.thanos.sidecar` - uploads Prometheus blocks to object storage
+- `services.thanos.compact` - compacts and downsamples data
+- `services.thanos.query` - unified query gateway
+- `services.thanos.query-frontend` - query caching and parallelization
+- `services.thanos.downsample` - dedicated downsampling service
+
+**Downsampling Behavior:**
+- Raw resolution kept for configurable period (default: indefinite)
+- 5-minute resolution created after 40 hours
+- 1-hour resolution created after 10 days
+
+**Retention Configuration (in compactor):**
+```nix
+services.thanos.compact = {
+  retention.resolution-raw = "30d";   # Keep raw for 30 days
+  retention.resolution-5m = "180d";   # Keep 5m samples for 6 months
+  retention.resolution-1h = "2y";     # Keep 1h samples for 2 years
+};
+```
+
+**Pros:**
+- True downsampling - older data uses progressively less storage
+- Keep metrics for years with minimal storage impact
+- Prometheus continues running unchanged
+- Existing Alertmanager integration preserved
+
+**Cons:**
+- Requires object storage (MinIO, S3, or local filesystem)
+- Multiple services to manage (sidecar, compactor, query)
+- More complex architecture
+- Additional infrastructure (MinIO) may be needed
+
+**Required Components:**
+1. Thanos Sidecar (runs alongside Prometheus)
+2. Object storage (MinIO or local filesystem)
+3. Thanos Compactor (handles downsampling)
+4. Thanos Query (provides unified query endpoint)
+
+**Migration Steps:**
+1. Deploy object storage (MinIO or configure filesystem backend)
+2. Add Thanos sidecar pointing to Prometheus data directory
+3. Add Thanos compactor with retention policies
+4. Add Thanos query gateway
+5. Update Grafana datasource to Thanos Query port (10902)
+
+## Comparison
+
+| Aspect | VictoriaMetrics | Thanos |
+|--------|-----------------|--------|
+| Complexity | Low (1 service) | Higher (3-4 services) |
+| Downsampling | No | Yes (automatic) |
+| Storage savings | 5-10x compression | Compression + downsampling |
+| Object storage required | No | Yes |
+| Migration effort | Minimal | Moderate |
+| Grafana changes | Change port only | Change port only |
+| Alerting changes | Need vmalert | Keep existing |
+
+## Recommendation
+
+**Start with VictoriaMetrics** for simplicity. The compression alone may provide 6+ months of retention in the same disk space currently used for 30 days.
+
+If multi-year retention with true downsampling becomes necessary, Thanos can be evaluated later. However, it requires deploying object storage infrastructure (MinIO) which adds operational complexity.
+
+## References
+
+- VictoriaMetrics docs: https://docs.victoriametrics.com/
+- Thanos docs: https://thanos.io/tip/thanos/getting-started.md/
+- NixOS options searched from nixpkgs revision e576e3c9 (NixOS 25.11)

docs/plans/nats-deploy-service.md

@@ -0,0 +1,371 @@
+# NATS-Based Deployment Service
+
+## Overview
+
+Create a message-based deployment system that allows triggering NixOS configuration updates on-demand, rather than waiting for the daily auto-upgrade timer. This enables faster iteration when testing changes and immediate fleet-wide deployments.
+
+## Goals
+
+1. **On-demand deployment** - Trigger config updates immediately via NATS message
+2. **Targeted deployment** - Deploy to specific hosts or all hosts
+3. **Branch/revision support** - Test feature branches before merging to master
+4. **MCP integration** - Allow Claude Code to trigger deployments during development
+
+## Current State
+
+- **Auto-upgrade**: All hosts run `nixos-upgrade.service` daily, pulling from master
+- **Manual testing**: `nixos-rebuild-test <action> <branch>` helper exists on all hosts
+- **NATS**: Running on nats1 with JetStream enabled, using NKey authentication
+- **Accounts**: ADMIN (system) and HOMELAB (user workloads with JetStream)
+
+## Architecture
+
+```
+┌─────────────┐                        ┌─────────────┐
+│  MCP Tool   │  deploy.test.>         │  Admin CLI  │  deploy.test.> + deploy.prod.>
+│  (claude)   │────────────┐     ┌─────│  (torjus)   │
+└─────────────┘            │     │     └─────────────┘
+                           ▼     ▼
+                      ┌──────────────┐
+                      │    nats1     │
+                      │  (authz)     │
+                      └──────┬───────┘
+                             │
+           ┌─────────────────┼─────────────────┐
+           │                 │                 │
+           ▼                 ▼                 ▼
+     ┌──────────┐      ┌──────────┐      ┌──────────┐
+     │ template1│      │   ns1    │      │   ha1    │
+     │ tier=test│      │ tier=prod│      │ tier=prod│
+     └──────────┘      └──────────┘      └──────────┘
+```
+
+## Repository Structure
+
+The project lives in a **separate repository** (e.g., `homelab-deploy`) containing:
+
+```
+homelab-deploy/
+├── flake.nix           # Nix flake with Go package + NixOS module
+├── go.mod
+├── go.sum
+├── cmd/
+│   └── homelab-deploy/
+│       └── main.go     # CLI entrypoint with subcommands
+├── internal/
+│   ├── listener/       # Listener mode logic
+│   ├── mcp/            # MCP server mode logic
+│   └── deploy/         # Shared deployment logic
+└── nixos/
+    └── module.nix      # NixOS module for listener service
+```
+
+This repo imports the flake as an input and uses the NixOS module.
+
+## Single Binary with Subcommands
+
+The `homelab-deploy` binary supports multiple modes:
+
+```bash
+# Run as listener on a host (systemd service)
+homelab-deploy listener --hostname ns1 --nats-url nats://nats1:4222
+
+# Run as MCP server (for Claude Code)
+homelab-deploy mcp --nats-url nats://nats1:4222
+
+# CLI commands for manual use
+homelab-deploy deploy ns1 --branch feature-x --action switch  # single host
+homelab-deploy deploy --tier test --all --action boot          # all test hosts
+homelab-deploy deploy --tier prod --all --action boot          # all prod hosts (admin only)
+homelab-deploy deploy --tier prod --role dns --action switch   # all prod dns hosts
+homelab-deploy status
+```
+
+## Components
+
+### Listener Mode
+
+A systemd service on each host that:
+- Subscribes to multiple subjects for targeted and group deployments
+- Validates incoming messages (revision, action)
+- Executes `nixos-rebuild` with specified parameters
+- Reports status back via NATS
+
+**Subject structure:**
+```
+deploy.<tier>.<hostname>      # specific host (e.g., deploy.prod.ns1)
+deploy.<tier>.all             # all hosts in tier (e.g., deploy.test.all)
+deploy.<tier>.role.<role>     # all hosts with role in tier (e.g., deploy.prod.role.dns)
+```
+
+**Listener subscriptions** (based on `homelab.host` config):
+- `deploy.<tier>.<hostname>` - direct messages to this host
+- `deploy.<tier>.all` - broadcast to all hosts in tier
+- `deploy.<tier>.role.<role>` - broadcast to hosts with matching role (if role is set)
+
+Example: ns1 with `tier=prod, role=dns` subscribes to:
+- `deploy.prod.ns1`
+- `deploy.prod.all`
+- `deploy.prod.role.dns`
+
+**NixOS module configuration:**
+```nix
+services.homelab-deploy.listener = {
+  enable = true;
+  timeout = 600;  # seconds, default 10 minutes
+};
+```
+
+The listener reads tier and role from `config.homelab.host` (see Host Metadata below).
+
+**Request message format:**
+```json
+{
+  "action": "switch" | "boot" | "test" | "dry-activate",
+  "revision": "master" | "feature-branch" | "abc123...",
+  "reply_to": "deploy.responses.<request-id>"
+}
+```
+
+**Response message format:**
+```json
+{
+  "status": "accepted" | "rejected" | "started" | "completed" | "failed",
+  "error": "invalid_revision" | "already_running" | "build_failed" | null,
+  "message": "human-readable details"
+}
+```
+
+**Request/Reply flow:**
+1. MCP/CLI sends deploy request with unique `reply_to` subject
+2. Listener validates request (e.g., `git ls-remote` to check revision exists)
+3. Listener sends immediate response:
+   - `{"status": "rejected", "error": "invalid_revision", "message": "branch 'foo' not found"}`, or
+   - `{"status": "started", "message": "starting nixos-rebuild switch"}`
+4. If started, listener runs nixos-rebuild
+5. Listener sends final response:
+   - `{"status": "completed", "message": "successfully switched to generation 42"}`, or
+   - `{"status": "failed", "error": "build_failed", "message": "nixos-rebuild exited with code 1"}`
+
+This provides immediate feedback on validation errors (bad revision, already running) without waiting for the build to fail.
+
+### MCP Mode
+
+Runs as an MCP server providing tools for Claude Code.
+
+**Tools:**
+| Tool | Description | Tier Access |
+|------|-------------|-------------|
+| `deploy` | Deploy to test hosts (individual, all, or by role) | test only |
+| `deploy_admin` | Deploy to any host (requires `--enable-admin` flag) | test + prod |
+| `deploy_status` | Check deployment status/history | n/a |
+| `list_hosts` | List available deployment targets | n/a |
+
+**CLI flags:**
+```bash
+# Default: only test-tier deployments available
+homelab-deploy mcp --nats-url nats://nats1:4222
+
+# Enable admin tool (requires admin NKey to be configured)
+homelab-deploy mcp --nats-url nats://nats1:4222 --enable-admin --admin-nkey-file /path/to/admin.nkey
+```
+
+**Security layers:**
+1. **MCP flag**: `deploy_admin` tool only exposed when `--enable-admin` is passed
+2. **NATS authz**: Even if tool is exposed, NATS rejects publishes without valid admin NKey
+3. **Claude Code permissions**: Can set `mcp__homelab-deploy__deploy_admin` to `ask` mode for confirmation popup
+
+By default, the MCP only loads test-tier credentials and exposes the `deploy` tool. Claude can:
+- Deploy to individual test hosts
+- Deploy to all test hosts at once (`deploy.test.all`)
+- Deploy to test hosts by role (`deploy.test.role.<role>`)
+
+### Tiered Permissions
+
+Authorization is enforced at the NATS layer using subject-based permissions. Different deployer credentials have different publish rights:
+
+**NATS user configuration (on nats1):**
+```nix
+accounts = {
+  HOMELAB = {
+    users = [
+      # MCP/Claude - test tier only
+      {
+        nkey = "UABC...";  # mcp-deployer
+        permissions = {
+          publish = [ "deploy.test.>" ];
+          subscribe = [ "deploy.responses.>" ];
+        };
+      }
+      # Admin - full access to all tiers
+      {
+        nkey = "UXYZ...";  # admin-deployer
+        permissions = {
+          publish = [ "deploy.test.>" "deploy.prod.>" ];
+          subscribe = [ "deploy.responses.>" ];
+        };
+      }
+      # Host listeners - subscribe to their tier, publish responses
+      {
+        nkey = "UDEF...";  # host-listener (one per host)
+        permissions = {
+          subscribe = [ "deploy.*.>" ];
+          publish = [ "deploy.responses.>" ];
+        };
+      }
+    ];
+  };
+};
+```
+
+**Host tier assignments** (via `homelab.host.tier`):
+| Tier | Hosts |
+|------|-------|
+| test | template1, nix-cache01, future test hosts |
+| prod | ns1, ns2, ha1, monitoring01, http-proxy, etc. |
+
+**Example deployment scenarios:**
+
+| Command | Subject | MCP | Admin |
+|---------|---------|-----|-------|
+| Deploy to ns1 | `deploy.prod.ns1` | ❌ | ✅ |
+| Deploy to template1 | `deploy.test.template1` | ✅ | ✅ |
+| Deploy to all test hosts | `deploy.test.all` | ✅ | ✅ |
+| Deploy to all prod hosts | `deploy.prod.all` | ❌ | ✅ |
+| Deploy to all DNS servers | `deploy.prod.role.dns` | ❌ | ✅ |
+
+All NKeys stored in Vault - MCP gets limited credentials, admin CLI gets full-access credentials.
+
+### Host Metadata
+
+Rather than defining `tier` in the listener config, use a central `homelab.host` module that provides host metadata for multiple consumers. This aligns with the approach proposed in `docs/plans/prometheus-scrape-target-labels.md`.
+
+**Status:** The `homelab.host` module is implemented in `modules/homelab/host.nix`.
+Hosts can be filtered by tier using `config.homelab.host.tier`.
+
+**Module definition (in `modules/homelab/host.nix`):**
+```nix
+homelab.host = {
+  tier = lib.mkOption {
+    type = lib.types.enum [ "test" "prod" ];
+    default = "prod";
+    description = "Deployment tier - controls which credentials can deploy to this host";
+  };
+
+  priority = lib.mkOption {
+    type = lib.types.enum [ "high" "low" ];
+    default = "high";
+    description = "Alerting priority - low priority hosts have relaxed thresholds";
+  };
+
+  role = lib.mkOption {
+    type = lib.types.nullOr lib.types.str;
+    default = null;
+    description = "Primary role of this host (dns, database, monitoring, etc.)";
+  };
+
+  labels = lib.mkOption {
+    type = lib.types.attrsOf lib.types.str;
+    default = { };
+    description = "Additional free-form labels";
+  };
+};
+```
+
+**Consumers:**
+- `homelab-deploy` listener reads `config.homelab.host.tier` for subject subscription
+- Prometheus scrape config reads `priority`, `role`, `labels` for target labels
+- Future services can consume the same metadata
+
+**Example host config:**
+```nix
+# hosts/nix-cache01/configuration.nix
+homelab.host = {
+  tier = "test";      # can be deployed by MCP
+  priority = "low";   # relaxed alerting thresholds
+  role = "build-host";
+};
+
+# hosts/ns1/configuration.nix
+homelab.host = {
+  tier = "prod";      # requires admin credentials
+  priority = "high";
+  role = "dns";
+  labels.dns_role = "primary";
+};
+```
+
+## Implementation Steps
+
+### Phase 1: Core Binary + Listener
+
+1. **Create homelab-deploy repository**
+   - Initialize Go module
+   - Set up flake.nix with Go package build
+
+2. **Implement listener mode**
+   - NATS subscription logic
+   - nixos-rebuild execution
+   - Status reporting via NATS reply
+
+3. **Create NixOS module**
+   - Systemd service definition
+   - Configuration options (hostname, NATS URL, NKey path)
+   - Vault secret integration for NKeys
+
+4. **Create `homelab.host` module** (in nixos-servers)
+   - Define `tier`, `priority`, `role`, `labels` options
+   - This module is shared with Prometheus label work (see `docs/plans/prometheus-scrape-target-labels.md`)
+
+5. **Integrate with nixos-servers**
+   - Add flake input for homelab-deploy
+   - Import listener module in `system/`
+   - Set `homelab.host.tier` per host (test vs prod)
+
+6. **Configure NATS tiered permissions**
+   - Add deployer users to nats1 config (mcp-deployer, admin-deployer)
+   - Set up subject ACLs per user (test-only vs full access)
+   - Add deployer NKeys to Vault
+   - Create Terraform resources for NKey secrets
+
+### Phase 2: MCP + CLI
+
+7. **Implement MCP mode**
+   - MCP server with deploy/status tools
+   - Request/reply pattern for deployment feedback
+
+8. **Implement CLI commands**
+   - `deploy` command for manual deployments
+   - `status` command to check deployment state
+
+9. **Configure Claude Code**
+   - Add MCP server to configuration
+   - Document usage
+
+### Phase 3: Enhancements
+
+10. Add deployment locking (prevent concurrent deploys)
+11. Prometheus metrics for deployment status
+
+## Security Considerations
+
+- **Privilege escalation**: Listener runs as root to execute nixos-rebuild
+- **Input validation**: Strictly validate revision format (branch name or commit hash)
+- **Rate limiting**: Prevent rapid-fire deployments
+- **Audit logging**: Log all deployment requests with source identity
+- **Network isolation**: NATS only accessible from internal network
+
+## Decisions
+
+All open questions have been resolved. See Notes section for decision rationale.
+
+## Notes
+
+- The existing `nixos-rebuild-test` helper provides a good reference for the rebuild logic
+- Uses NATS request/reply pattern for immediate validation feedback and completion status
+- Consider using NATS headers for metadata (request ID, timestamp)
+- **Timeout decision**: Metrics show no-change upgrades complete in 5-55 seconds. A 10-minute default provides ample headroom for actual updates with package downloads. Per-host override available for hosts with known longer build times.
+- **Rollback**: Not needed as a separate feature - deploy an older commit hash to effectively rollback.
+- **Offline hosts**: No message persistence - if host is offline, deploy fails. Daily auto-upgrade is the safety net. Avoids complexity of JetStream deduplication (host coming online and applying 10 queued updates instead of just the latest).
+- **Deploy history**: Use existing Loki - listener logs deployments to journald, queryable via Loki. No need for separate JetStream persistence.
+- **Naming**: `homelab-deploy` - ties it to the infrastructure rather than implementation details.

docs/plans/nixos-improvements.md

@@ -0,0 +1,27 @@
+# NixOS Infrastructure Improvements
+
+This document contains planned improvements to the NixOS infrastructure that are not directly part of the automated deployment pipeline.
+
+## Planned
+
+### Custom NixOS Options for Service and System Configuration
+
+Currently, most service configurations in `services/` and shared system configurations in `system/` are written as plain NixOS module imports without declaring custom options. This means host-specific customization is done by directly setting upstream NixOS options or by duplicating configuration across hosts.
+
+The `homelab.dns` module (`modules/homelab/dns.nix`) is the first example of defining custom options under a `homelab.*` namespace. This pattern should be extended to more of the repository's configuration.
+
+**Goals:**
+
+- Define `homelab.*` options for services and shared configuration where it makes sense, following the pattern established by `homelab.dns`
+- Allow hosts to enable/configure services declaratively (e.g. `homelab.monitoring.enable`, `homelab.http-proxy.virtualHosts`) rather than importing opaque module files
+- Keep options simple and focused — wrap only the parts that vary between hosts or that benefit from a clearer interface. Not everything needs a custom option.
+
+**Candidate areas:**
+
+- `system/` modules (e.g. auto-upgrade schedule, ACME CA URL, monitoring endpoints)
+- `services/` modules where multiple hosts use the same service with different parameters
+- Cross-cutting concerns that are currently implicit (e.g. which Loki endpoint promtail ships to)
+
+## Completed
+
+- [DNS Automation](completed/dns-automation.md) - Automatically generate DNS entries from host configurations

docs/plans/prometheus-scrape-target-labels.md

@@ -0,0 +1,173 @@
+# Prometheus Scrape Target Labels
+
+## Goal
+
+Add support for custom per-host labels on Prometheus scrape targets, enabling alert rules to reference host metadata (priority, role) instead of hardcoding instance names.
+
+**Related:** This plan shares the `homelab.host` module with `docs/plans/nats-deploy-service.md`, which uses the same metadata for deployment tier assignment.
+
+## Motivation
+
+Some hosts have workloads that make generic alert thresholds inappropriate. For example, `nix-cache01` regularly hits high CPU during builds, requiring a longer `for` duration on `high_cpu_load`. Currently this is handled by excluding specific instance names in PromQL expressions, which is brittle and doesn't scale.
+
+With per-host labels, alert rules can use semantic filters like `{priority!="low"}` instead of `{instance!="nix-cache01.home.2rjus.net:9100"}`.
+
+## Proposed Labels
+
+### `priority`
+
+Indicates alerting importance. Hosts with `priority = "low"` can have relaxed thresholds or longer durations in alert rules.
+
+Values: `"high"` (default), `"low"`
+
+### `role`
+
+Describes the function of the host. Useful for grouping in dashboards and targeting role-specific alert rules.
+
+Values: free-form string, e.g. `"dns"`, `"build-host"`, `"database"`, `"monitoring"`
+
+**Note on multiple roles:** Prometheus labels are strictly string values, not lists. For hosts that serve multiple roles there are a few options:
+
+- **Separate boolean labels:** `role_build_host = "true"`, `role_cache_server = "true"` -- flexible but verbose, and requires updating the module when new roles are added.
+- **Delimited string:** `role = "build-host,cache-server"` -- works with regex matchers (`{role=~".*build-host.*"}`), but regex matching is less clean and more error-prone.
+- **Pick a primary role:** `role = "build-host"` -- simplest, and probably sufficient since most hosts have one primary role.
+
+Recommendation: start with a single primary role string. If multi-role matching becomes a real need, switch to separate boolean labels.
+
+### `dns_role`
+
+For DNS servers specifically, distinguish between primary and secondary resolvers. The secondary resolver (ns2) receives very little traffic and has a cold cache, making generic cache hit ratio alerts inappropriate.
+
+Values: `"primary"`, `"secondary"`
+
+Example use case: The `unbound_low_cache_hit_ratio` alert fires on ns2 because its cache hit ratio (~62%) is lower than ns1 (~90%). This is expected behavior since ns2 gets ~100x less traffic. With a `dns_role` label, the alert can either exclude secondaries or use different thresholds:
+
+```promql
+# Only alert on primary DNS
+unbound_cache_hit_ratio < 0.7 and on(instance) unbound_up{dns_role="primary"}
+
+# Or use different thresholds
+(unbound_cache_hit_ratio < 0.7 and on(instance) unbound_up{dns_role="primary"})
+or
+(unbound_cache_hit_ratio < 0.5 and on(instance) unbound_up{dns_role="secondary"})
+```
+
+## Implementation
+
+This implementation uses a shared `homelab.host` module that provides host metadata for multiple consumers (Prometheus labels, deployment tiers, etc.). See also `docs/plans/nats-deploy-service.md` which uses the same module for deployment tier assignment.
+
+### 1. Create `homelab.host` module
+
+**Status:** Step 1 (Create `homelab.host` module) is complete. The module is in
+`modules/homelab/host.nix` with tier, priority, role, and labels options.
+
+Create `modules/homelab/host.nix` with shared host metadata options:
+
+```nix
+{ lib, ... }:
+{
+  options.homelab.host = {
+    tier = lib.mkOption {
+      type = lib.types.enum [ "test" "prod" ];
+      default = "prod";
+      description = "Deployment tier - controls which credentials can deploy to this host";
+    };
+
+    priority = lib.mkOption {
+      type = lib.types.enum [ "high" "low" ];
+      default = "high";
+      description = "Alerting priority - low priority hosts have relaxed thresholds";
+    };
+
+    role = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      description = "Primary role of this host (dns, database, monitoring, etc.)";
+    };
+
+    labels = lib.mkOption {
+      type = lib.types.attrsOf lib.types.str;
+      default = { };
+      description = "Additional free-form labels (e.g., dns_role = 'primary')";
+    };
+  };
+}
+```
+
+Import this module in `modules/homelab/default.nix`.
+
+### 2. Update `lib/monitoring.nix`
+
+- `extractHostMonitoring` should also extract `homelab.host` values (priority, role, labels).
+- Build the combined label set from `homelab.host`:
+
+```nix
+# Combine structured options + free-form labels
+effectiveLabels =
+  (lib.optionalAttrs (host.priority != "high") { priority = host.priority; })
+  // (lib.optionalAttrs (host.role != null) { role = host.role; })
+  // host.labels;
+```
+
+- `generateNodeExporterTargets` returns structured `static_configs` entries, grouping targets by their label sets:
+
+```nix
+# Before (flat list):
+["ns1.home.2rjus.net:9100", "ns2.home.2rjus.net:9100", ...]
+
+# After (grouped by labels):
+[
+  { targets = ["ns1.home.2rjus.net:9100", "ns2.home.2rjus.net:9100", ...]; }
+  { targets = ["nix-cache01.home.2rjus.net:9100"]; labels = { priority = "low"; role = "build-host"; }; }
+]
+```
+
+This requires grouping hosts by their label attrset and producing one `static_configs` entry per unique label combination. Hosts with default values (priority=high, no role, no labels) get grouped together with no extra labels (preserving current behavior).
+
+### 3. Update `services/monitoring/prometheus.nix`
+
+Change the node-exporter scrape config to use the new structured output:
+
+```nix
+# Before:
+static_configs = [{ targets = nodeExporterTargets; }];
+
+# After:
+static_configs = nodeExporterTargets;
+```
+
+### 4. Set metadata on hosts
+
+Example in `hosts/nix-cache01/configuration.nix`:
+
+```nix
+homelab.host = {
+  tier = "test";       # can be deployed by MCP (used by homelab-deploy)
+  priority = "low";    # relaxed alerting thresholds
+  role = "build-host";
+};
+```
+
+Example in `hosts/ns1/configuration.nix`:
+
+```nix
+homelab.host = {
+  tier = "prod";
+  priority = "high";
+  role = "dns";
+  labels.dns_role = "primary";
+};
+```
+
+### 5. Update alert rules
+
+After implementing labels, review and update `services/monitoring/rules.yml`:
+
+- Replace instance-name exclusions with label-based filters (e.g. `{priority!="low"}` instead of `{instance!="nix-cache01.home.2rjus.net:9100"}`).
+- Consider whether any other rules should differentiate by priority or role.
+
+Specifically, the `high_cpu_load` rule currently has a nix-cache01 exclusion that should be replaced with a `priority`-based filter.
+
+### 6. Consider labels for `generateScrapeConfigs` (service targets)
+
+The same label propagation could be applied to service-level scrape targets. This is optional and can be deferred -- service targets are more specialized and less likely to need generic label-based filtering.

docs/plans/remote-access.md

@@ -0,0 +1,122 @@
+# Remote Access to Homelab Services
+
+## Status: Planning
+
+## Goal
+
+Enable remote access to some or all homelab services from outside the internal network, without exposing anything directly to the internet.
+
+## Current State
+
+- All services are only accessible from the internal 10.69.13.x network
+- Exception: jelly01 has a WireGuard link to an external VPS
+- No services are directly exposed to the public internet
+
+## Constraints
+
+- Nothing should be directly accessible from the outside
+- Must use VPN or overlay network (no port forwarding of services)
+- Self-hosted solutions preferred over managed services
+
+## Options
+
+### 1. WireGuard Gateway (Internal Router)
+
+A dedicated NixOS host on the internal network with a WireGuard tunnel out to the VPS. The VPS becomes the public entry point, and the gateway routes traffic to internal services. Firewall rules on the gateway control which services are reachable.
+
+**Pros:**
+- Simple, well-understood technology
+- Already running WireGuard for jelly01
+- Full control over routing and firewall rules
+- Excellent NixOS module support
+- No extra dependencies
+
+**Cons:**
+- Hub-and-spoke topology (all traffic goes through VPS)
+- Manual peer management
+- Adding a new client device means editing configs on both VPS and gateway
+
+### 2. WireGuard Mesh (No Relay)
+
+Each client device connects directly to a WireGuard endpoint. Could be on the VPS which forwards to the homelab, or if there is a routable IP at home, directly to an internal host.
+
+**Pros:**
+- Simple and fast
+- No extra software
+
+**Cons:**
+- Manual key and endpoint management for every peer
+- Doesn't scale well
+- If behind CGNAT, still needs the VPS as intermediary
+
+### 3. Headscale (Self-Hosted Tailscale)
+
+Run a Headscale control server (on the VPS or internally) and install the Tailscale client on homelab hosts and personal devices. Gets the Tailscale mesh networking UX without depending on Tailscale's infrastructure.
+
+**Pros:**
+- Mesh topology - devices communicate directly via NAT traversal (DERP relay as fallback)
+- Easy to add/remove devices
+- ACL support for granular access control
+- MagicDNS for service discovery
+- Good NixOS support for both headscale server and tailscale client
+- Subnet routing lets you expose the entire 10.69.13.x network or specific hosts without installing tailscale on every host
+
+**Cons:**
+- More moving parts than plain WireGuard
+- Headscale is a third-party reimplementation, can lag behind Tailscale features
+- Need to run and maintain the control server
+
+### 4. Tailscale (Managed)
+
+Same as Headscale but using Tailscale's hosted control plane.
+
+**Pros:**
+- Zero infrastructure to manage on the control plane side
+- Polished UX, well-maintained clients
+- Free tier covers personal use
+
+**Cons:**
+- Dependency on Tailscale's service
+- Less aligned with self-hosting preference
+- Coordination metadata goes through their servers (data plane is still peer-to-peer)
+
+### 5. Netbird (Self-Hosted)
+
+Open-source alternative to Tailscale with a self-hostable management server. WireGuard-based, supports ACLs and NAT traversal.
+
+**Pros:**
+- Fully self-hostable
+- Web UI for management
+- ACL and peer grouping support
+
+**Cons:**
+- Heavier to self-host (needs multiple components: management server, signal server, TURN relay)
+- Less mature NixOS module support compared to Tailscale/Headscale
+
+### 6. Nebula (by Defined Networking)
+
+Certificate-based mesh VPN. Each node gets a certificate from a CA you control. No central coordination server needed at runtime.
+
+**Pros:**
+- No always-on control plane
+- Certificate-based identity
+- Lightweight
+
+**Cons:**
+- Less convenient for ad-hoc device addition (need to issue certs)
+- NAT traversal less mature than Tailscale's
+- Smaller community/ecosystem
+
+## Key Decision Points
+
+- **Static public IP vs CGNAT?** Determines whether clients can connect directly to home network or need VPS relay.
+- **Number of client devices?** If just phone and laptop, plain WireGuard via VPS is fine. More devices favors Headscale.
+- **Per-service vs per-network access?** Gateway with firewall rules gives per-service control. Headscale ACLs can also do this. Plain WireGuard gives network-level access with gateway firewall for finer control.
+- **Subnet routing vs per-host agents?** With Headscale/Tailscale, can either install client on every host, or use a single subnet router that advertises the 10.69.13.x range. The latter is closer to the gateway approach and avoids touching every host.
+
+## Leading Candidates
+
+Based on existing WireGuard experience, self-hosting preference, and NixOS stack:
+
+1. **Headscale with a subnet router** - Best balance of convenience and self-hosting
+2. **WireGuard gateway via VPS** - Simplest, most transparent, builds on existing setup

docs/plans/truenas-migration.md

@@ -0,0 +1,151 @@
+# TrueNAS Migration Planning
+
+## Current State
+
+### Hardware
+- CPU: AMD Ryzen 5 5600G with Radeon Graphics
+- RAM: 32GB
+- Network: 10GbE (mlxen0)
+- Software: TrueNAS-13.0-U6.1 (Core)
+
+### Storage Status
+
+**hdd-pool**: 29.1TB total, **28.4TB used, 658GB free (97% capacity)** ⚠️
+- mirror-0: 2x Seagate ST16000NE000 16TB HDD (16TB usable)
+- mirror-1: 2x WD WD80EFBX 8TB HDD (8TB usable)
+- mirror-2: 2x Seagate ST8000VN004 8TB HDD (8TB usable)
+
+## Goal
+
+Expand storage capacity for the main hdd-pool. Since we need to add disks anyway, also evaluating whether to upgrade or replace the entire system.
+
+## Decisions
+
+### Migration Approach: Option 3 - Migrate to NixOS
+
+**Decision**: Replace TrueNAS with NixOS bare metal installation
+
+**Rationale**:
+- Aligns with existing infrastructure (16+ NixOS hosts already managed in this repo)
+- Declarative configuration fits homelab philosophy
+- Automatic monitoring/logging integration (Prometheus + Promtail)
+- Auto-upgrades via same mechanism as other hosts
+- SOPS secrets management integration
+- TrueNAS-specific features (WebGUI, jails) not heavily utilized
+
+**Service migration**:
+- radarr/sonarr: Native NixOS services (`services.radarr`, `services.sonarr`)
+- restic-rest: `services.restic.server`
+- nzbget: NixOS service or OCI container
+- NFS exports: `services.nfs.server`
+
+### Filesystem: BTRFS RAID1
+
+**Decision**: Migrate from ZFS to BTRFS with RAID1
+
+**Rationale**:
+- **In-kernel**: No out-of-tree module issues like ZFS
+- **Flexible expansion**: Add individual disks, not required to buy pairs
+- **Mixed disk sizes**: Better handling than ZFS multi-vdev approach
+- **RAID level conversion**: Can convert between RAID levels in place
+- Built-in checksumming, snapshots, compression (zstd)
+- NixOS has good BTRFS support
+
+**BTRFS RAID1 notes**:
+- "RAID1" means 2 copies of all data
+- Distributes across all available devices
+- With 6+ disks, provides redundancy + capacity scaling
+- RAID5/6 avoided (known issues), RAID1/10 are stable
+
+### Hardware: Keep Existing + Add Disks
+
+**Decision**: Retain current hardware, expand disk capacity
+
+**Hardware to keep**:
+- AMD Ryzen 5 5600G (sufficient for NAS workload)
+- 32GB RAM (adequate)
+- 10GbE network interface
+- Chassis
+
+**Storage architecture**:
+
+**Bulk storage** (BTRFS RAID1 on HDDs):
+- Current: 6x HDDs (2x16TB + 2x8TB + 2x8TB)
+- Add: 2x new HDDs (size TBD)
+- Use: Media, downloads, backups, non-critical data
+- Risk tolerance: High (data mostly replaceable)
+
+**Critical data** (small volume):
+- Use 2x 240GB SSDs in mirror (BTRFS or ZFS)
+- Or use 2TB NVMe for critical data
+- Risk tolerance: Low (data important but small)
+
+### Disk Purchase Decision
+
+**Options under consideration**:
+
+**Option A: 2x 16TB drives**
+- Matches largest current drives
+- Enables potential future RAID5 if desired (6x 16TB array)
+- More conservative capacity increase
+
+**Option B: 2x 20-24TB drives**
+- Larger capacity headroom
+- Better $/TB ratio typically
+- Future-proofs better
+
+**Initial purchase**: 2 drives (chassis has space for 2 more without modifications)
+
+## Migration Strategy
+
+### High-Level Plan
+
+1. **Preparation**:
+   - Purchase 2x new HDDs (16TB or 20-24TB)
+   - Create NixOS configuration for new storage host
+   - Set up bare metal NixOS installation
+
+2. **Initial BTRFS pool**:
+   - Install 2 new disks
+   - Create BTRFS filesystem in RAID1
+   - Mount and test NFS exports
+
+3. **Data migration**:
+   - Copy data from TrueNAS ZFS pool to new BTRFS pool over 10GbE
+   - Verify data integrity
+
+4. **Expand pool**:
+   - As old ZFS pool is emptied, wipe drives and add to BTRFS pool
+   - Pool grows incrementally: 2 → 4 → 6 → 8 disks
+   - BTRFS rebalances data across new devices
+
+5. **Service migration**:
+   - Set up radarr/sonarr/nzbget/restic as NixOS services
+   - Update NFS client mounts on consuming hosts
+
+6. **Cutover**:
+   - Point consumers to new NAS host
+   - Decommission TrueNAS
+   - Repurpose hardware or keep as spare
+
+### Migration Advantages
+
+- **Low risk**: New pool created independently, old data remains intact during migration
+- **Incremental**: Can add old disks one at a time as space allows
+- **Flexible**: BTRFS handles mixed disk sizes gracefully
+- **Reversible**: Keep TrueNAS running until fully validated
+
+## Next Steps
+
+1. Decide on disk size (16TB vs 20-24TB)
+2. Purchase disks
+3. Design NixOS host configuration (`hosts/nas1/`)
+4. Plan detailed migration timeline
+5. Document NFS export mapping (current → new)
+
+## Open Questions
+
+- [ ] Final decision on disk size?
+- [ ] Hostname for new NAS host? (nas1? storage1?)
+- [ ] IP address allocation (keep 10.69.12.50 or new IP?)
+- [ ] Timeline/maintenance window for migration?

docs/vault-bootstrap-implementation.md

@@ -0,0 +1,560 @@
+# Phase 4d: Vault Bootstrap Integration - Implementation Summary
+
+## Overview
+
+Phase 4d implements automatic Vault/OpenBao integration for new NixOS hosts, enabling:
+- Zero-touch secret provisioning on first boot
+- Automatic AppRole authentication
+- Runtime secret fetching with caching
+- Periodic secret rotation
+
+**Key principle**: Existing sops-nix infrastructure remains unchanged. This is new infrastructure running in parallel.
+
+## Architecture
+
+### Component Diagram
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│ Developer Workstation                                       │
+│                                                             │
+│  create-host --hostname myhost --ip 10.69.13.x/24          │
+│       │                                                     │
+│       ├─> Generate host configs (hosts/myhost/)            │
+│       ├─> Update flake.nix                                 │
+│       ├─> Update terraform/vms.tf                          │
+│       ├─> Generate terraform/vault/hosts-generated.tf      │
+│       ├─> Apply Vault Terraform (create AppRole)           │
+│       └─> Generate wrapped token (24h TTL) ───┐            │
+│                                               │            │
+└───────────────────────────────────────────────┼────────────┘
+                                                │
+                    ┌───────────────────────────┘
+                    │ Wrapped Token
+                    │ (single-use, 24h expiry)
+                    ↓
+┌─────────────────────────────────────────────────────────────┐
+│ Cloud-init (VM Provisioning)                                │
+│                                                             │
+│  /etc/environment:                                          │
+│    VAULT_ADDR=https://vault01.home.2rjus.net:8200            │
+│    VAULT_WRAPPED_TOKEN=hvs.CAES...                         │
+│    VAULT_SKIP_VERIFY=1                                     │
+└─────────────────────────────────────────────────────────────┘
+                    │
+                    ↓
+┌─────────────────────────────────────────────────────────────┐
+│ Bootstrap Service (First Boot)                              │
+│                                                             │
+│  1. Read VAULT_WRAPPED_TOKEN from environment              │
+│  2. POST /v1/sys/wrapping/unwrap                           │
+│  3. Extract role_id + secret_id                            │
+│  4. Store in /var/lib/vault/approle/                       │
+│     ├─ role-id     (600 permissions)                       │
+│     └─ secret-id   (600 permissions)                       │
+│  5. Continue with nixos-rebuild boot                       │
+└─────────────────────────────────────────────────────────────┘
+                    │
+                    ↓
+┌─────────────────────────────────────────────────────────────┐
+│ Runtime (Service Starts)                                    │
+│                                                             │
+│  vault-secret-<name>.service (ExecStartPre)                │
+│    │                                                        │
+│    ├─> vault-fetch <secret-path> <output-dir>             │
+│    │     │                                                 │
+│    │     ├─> Read role_id + secret_id                     │
+│    │     ├─> POST /v1/auth/approle/login → token          │
+│    │     ├─> GET /v1/secret/data/<path> → secrets         │
+│    │     ├─> Write /run/secrets/<name>/password            │
+│    │     ├─> Write /run/secrets/<name>/api_key             │
+│    │     └─> Cache to /var/lib/vault/cache/<name>/        │
+│    │                                                        │
+│    └─> chown/chmod secret files                           │
+│                                                             │
+│  myservice.service                                         │
+│    └─> Reads secrets from /run/secrets/<name>/            │
+└─────────────────────────────────────────────────────────────┘
+```
+
+### Data Flow
+
+1. **Provisioning Time** (Developer → Vault):
+   - create-host generates AppRole configuration
+   - Terraform creates AppRole + policy in Vault
+   - Vault generates wrapped token containing role_id + secret_id
+   - Wrapped token stored in terraform/vms.tf
+
+2. **Bootstrap Time** (Cloud-init → VM):
+   - Cloud-init injects wrapped token via /etc/environment
+   - Bootstrap service unwraps token (single-use operation)
+   - Stores unwrapped credentials persistently
+
+3. **Runtime** (Service → Vault):
+   - Service starts
+   - ExecStartPre hook calls vault-fetch
+   - vault-fetch authenticates using stored credentials
+   - Fetches secrets and caches them
+   - Service reads secrets from filesystem
+
+## Implementation Details
+
+### 1. vault-fetch Helper (`scripts/vault-fetch/`)
+
+**Purpose**: Fetch secrets from Vault and write to filesystem
+
+**Features**:
+- Reads AppRole credentials from `/var/lib/vault/approle/`
+- Authenticates to Vault (fresh token each time)
+- Fetches secret from KV v2 engine
+- Writes individual files per secret key
+- Updates cache for fallback
+- Gracefully degrades to cache if Vault unreachable
+
+**Usage**:
+```bash
+vault-fetch hosts/monitoring01/grafana /run/secrets/grafana
+```
+
+**Environment Variables**:
+- `VAULT_ADDR`: Vault server (default: https://vault01.home.2rjus.net:8200)
+- `VAULT_SKIP_VERIFY`: Skip TLS verification (default: 1)
+
+**Error Handling**:
+- Vault unreachable → Use cache (log warning)
+- Invalid credentials → Fail with clear error
+- No cache + unreachable → Fail with error
+
+### 2. NixOS Module (`system/vault-secrets.nix`)
+
+**Purpose**: Declarative Vault secret management for NixOS services
+
+**Configuration Options**:
+
+```nix
+vault.enable = true;  # Enable Vault integration
+
+vault.secrets.<name> = {
+  secretPath = "hosts/monitoring01/grafana";  # Path in Vault
+  outputDir = "/run/secrets/grafana";         # Where to write secrets
+  cacheDir = "/var/lib/vault/cache/grafana";  # Cache location
+  owner = "grafana";                          # File owner
+  group = "grafana";                          # File group
+  mode = "0400";                              # Permissions
+  services = [ "grafana" ];                   # Dependent services
+  restartTrigger = true;                      # Enable periodic rotation
+  restartInterval = "daily";                  # Rotation schedule
+};
+```
+
+**Module Behavior**:
+
+1. **Fetch Service**: Creates `vault-secret-<name>.service`
+   - Runs on boot and before dependent services
+   - Calls vault-fetch to populate secrets
+   - Sets ownership and permissions
+
+2. **Rotation Timer**: Optionally creates `vault-secret-rotate-<name>.timer`
+   - Scheduled restarts for secret rotation
+   - Automatically excluded for critical services
+   - Configurable interval (daily, weekly, monthly)
+
+3. **Critical Service Protection**:
+   ```nix
+   vault.criticalServices = [ "bind" "openbao" "step-ca" ];
+   ```
+   Services in this list never get auto-restart timers
+
+### 3. create-host Tool Updates
+
+**New Functionality**:
+
+1. **Vault Terraform Generation** (`generators.py`):
+   - Creates/updates `terraform/vault/hosts-generated.tf`
+   - Adds host policy granting access to `secret/data/hosts/<hostname>/*`
+   - Adds AppRole configuration
+   - Idempotent (safe to re-run)
+
+2. **Wrapped Token Generation** (`vault_helper.py`):
+   - Applies Vault Terraform to create AppRole
+   - Reads role_id from Vault
+   - Generates secret_id
+   - Wraps credentials in cubbyhole token (24h TTL, single-use)
+   - Returns wrapped token
+
+3. **VM Configuration Update** (`manipulators.py`):
+   - Adds `vault_wrapped_token` field to VM in vms.tf
+   - Preserves other VM settings
+
+**New CLI Options**:
+```bash
+create-host --hostname myhost --ip 10.69.13.x/24
+  # Full workflow with Vault integration
+
+create-host --hostname myhost --skip-vault
+  # Create host without Vault (legacy behavior)
+
+create-host --hostname myhost --force
+  # Regenerate everything including new wrapped token
+```
+
+**Dependencies Added**:
+- `hvac`: Python Vault client library
+
+### 4. Bootstrap Service Updates
+
+**New Behavior** (`hosts/template2/bootstrap.nix`):
+
+```bash
+# Check for wrapped token
+if [ -n "$VAULT_WRAPPED_TOKEN" ]; then
+  # Unwrap to get credentials
+  curl -X POST \
+    -H "X-Vault-Token: $VAULT_WRAPPED_TOKEN" \
+    $VAULT_ADDR/v1/sys/wrapping/unwrap
+
+  # Store role_id and secret_id
+  mkdir -p /var/lib/vault/approle
+  echo "$ROLE_ID" > /var/lib/vault/approle/role-id
+  echo "$SECRET_ID" > /var/lib/vault/approle/secret-id
+  chmod 600 /var/lib/vault/approle/*
+
+  # Continue with bootstrap...
+fi
+```
+
+**Error Handling**:
+- Token already used → Log error, continue bootstrap
+- Token expired → Log error, continue bootstrap
+- Vault unreachable → Log warning, continue bootstrap
+- **Never fails bootstrap** - host can still run without Vault
+
+### 5. Cloud-init Configuration
+
+**Updates** (`terraform/cloud-init.tf`):
+
+```hcl
+write_files:
+  - path: /etc/environment
+    content: |
+      VAULT_ADDR=https://vault01.home.2rjus.net:8200
+      VAULT_WRAPPED_TOKEN=${vault_wrapped_token}
+      VAULT_SKIP_VERIFY=1
+```
+
+**VM Configuration** (`terraform/vms.tf`):
+
+```hcl
+locals {
+  vms = {
+    "myhost" = {
+      ip = "10.69.13.x/24"
+      vault_wrapped_token = "hvs.CAESIBw..."  # Added by create-host
+    }
+  }
+}
+```
+
+### 6. Vault Terraform Structure
+
+**Generated Hosts File** (`terraform/vault/hosts-generated.tf`):
+
+```hcl
+locals {
+  generated_host_policies = {
+    "myhost" = {
+      paths = [
+        "secret/data/hosts/myhost/*",
+      ]
+    }
+  }
+}
+
+resource "vault_policy" "generated_host_policies" {
+  for_each = local.generated_host_policies
+  name = "host-${each.key}"
+  policy = <<-EOT
+    path "secret/data/hosts/${each.key}/*" {
+      capabilities = ["read", "list"]
+    }
+  EOT
+}
+
+resource "vault_approle_auth_backend_role" "generated_hosts" {
+  for_each = local.generated_host_policies
+
+  backend        = vault_auth_backend.approle.path
+  role_name      = each.key
+  token_policies = ["host-${each.key}"]
+  secret_id_ttl  = 0      # Never expire
+  token_ttl      = 3600   # 1 hour tokens
+}
+```
+
+**Separation of Concerns**:
+- `approle.tf`: Manual host configurations (ha1, monitoring01)
+- `hosts-generated.tf`: Auto-generated configurations
+- `secrets.tf`: Secret definitions (manual)
+- `pki.tf`: PKI infrastructure
+
+## Security Model
+
+### Credential Distribution
+
+**Wrapped Token Security**:
+- **Single-use**: Can only be unwrapped once
+- **Time-limited**: 24h TTL
+- **Safe in git**: Even if leaked, expires quickly
+- **Standard Vault pattern**: Built-in Vault feature
+
+**Why wrapped tokens are secure**:
+```
+Developer commits wrapped token to git
+  ↓
+Attacker finds token in git history
+  ↓
+Attacker tries to use token
+  ↓
+❌ Token already used (unwrapped during bootstrap)
+  ↓
+❌ OR: Token expired (>24h old)
+```
+
+### AppRole Credentials
+
+**Storage**:
+- Location: `/var/lib/vault/approle/`
+- Permissions: `600 (root:root)`
+- Persistence: Survives reboots
+
+**Security Properties**:
+- `role_id`: Non-sensitive (like username)
+- `secret_id`: Sensitive (like password)
+- `secret_id_ttl = 0`: Never expires (simplicity vs rotation tradeoff)
+- Tokens: Ephemeral (1h TTL, not cached)
+
+**Attack Scenarios**:
+
+1. **Attacker gets root on host**:
+   - Can read AppRole credentials
+   - Can only access that host's secrets
+   - Cannot access other hosts' secrets (policy restriction)
+   - ✅ Blast radius limited to single host
+
+2. **Attacker intercepts wrapped token**:
+   - Single-use: Already consumed during bootstrap
+   - Time-limited: Likely expired
+   - ✅ Cannot be reused
+
+3. **Vault server compromised**:
+   - All secrets exposed (same as any secret storage)
+   - ✅ No different from sops-nix master key compromise
+
+### Secret Storage
+
+**Runtime Secrets**:
+- Location: `/run/secrets/` (tmpfs)
+- Lost on reboot
+- Re-fetched on service start
+- ✅ Not in Nix store
+- ✅ Not persisted to disk
+
+**Cached Secrets**:
+- Location: `/var/lib/vault/cache/`
+- Persists across reboots
+- Only used when Vault unreachable
+- ✅ Enables service availability
+- ⚠️ May be stale
+
+## Failure Modes
+
+### Wrapped Token Expired
+
+**Symptom**: Bootstrap logs "token expired" error
+
+**Impact**: Host boots but has no Vault credentials
+
+**Fix**: Regenerate token and redeploy
+```bash
+create-host --hostname myhost --force
+cd terraform && tofu apply
+```
+
+### Vault Unreachable
+
+**Symptom**: Service logs "WARNING: Using cached secrets"
+
+**Impact**: Service uses stale secrets (may work or fail depending on rotation)
+
+**Fix**: Restore Vault connectivity, restart service
+
+### No Cache Available
+
+**Symptom**: Service fails to start with "No cache available"
+
+**Impact**: Service unavailable until Vault restored
+
+**Fix**: Restore Vault, restart service
+
+### Invalid Credentials
+
+**Symptom**: vault-fetch logs authentication failure
+
+**Impact**: Service cannot start
+
+**Fix**:
+1. Check AppRole exists: `vault read auth/approle/role/hostname`
+2. Check policy exists: `vault policy read host-hostname`
+3. Regenerate credentials if needed
+
+## Migration Path
+
+### Current State (Phase 4d)
+
+- ✅ sops-nix: Used by all existing services
+- ✅ Vault: Available for new services
+- ✅ Parallel operation: Both work simultaneously
+
+### Future Migration
+
+**Gradual Service Migration**:
+
+1. **Pick a non-critical service** (e.g., test service)
+2. **Add Vault secrets**:
+   ```nix
+   vault.secrets.myservice = {
+     secretPath = "hosts/myhost/myservice";
+   };
+   ```
+3. **Update service to read from Vault**:
+   ```nix
+   systemd.services.myservice.serviceConfig = {
+     EnvironmentFile = "/run/secrets/myservice/password";
+   };
+   ```
+4. **Remove sops-nix secret**
+5. **Test thoroughly**
+6. **Repeat for next service**
+
+**Critical Services Last**:
+- DNS (bind)
+- Certificate Authority (step-ca)
+- Vault itself (openbao)
+
+**Eventually**:
+- All services migrated to Vault
+- Remove sops-nix dependency
+- Clean up `/secrets/` directory
+
+## Performance Considerations
+
+### Bootstrap Time
+
+**Added overhead**: ~2-5 seconds
+- Token unwrap: ~1s
+- Credential storage: ~1s
+
+**Total bootstrap time**: Still <2 minutes (acceptable)
+
+### Service Startup
+
+**Added overhead**: ~1-3 seconds per service
+- Vault authentication: ~1s
+- Secret fetch: ~1s
+- File operations: <1s
+
+**Parallel vs Serial**:
+- Multiple services fetch in parallel
+- No cascade delays
+
+### Cache Benefits
+
+**When Vault unreachable**:
+- Service starts in <1s (cache read)
+- No Vault dependency for startup
+- High availability maintained
+
+## Testing Checklist
+
+Complete testing workflow documented in `vault-bootstrap-testing.md`:
+
+- [ ] Create test host with create-host
+- [ ] Add test secrets to Vault
+- [ ] Deploy VM and verify bootstrap
+- [ ] Verify secrets fetched successfully
+- [ ] Test service restart (re-fetch)
+- [ ] Test Vault unreachable (cache fallback)
+- [ ] Test secret rotation
+- [ ] Test wrapped token expiry
+- [ ] Test token reuse prevention
+- [ ] Verify critical services excluded from auto-restart
+
+## Files Changed
+
+### Created
+- `scripts/vault-fetch/vault-fetch.sh` - Secret fetching script
+- `scripts/vault-fetch/default.nix` - Nix package
+- `scripts/vault-fetch/README.md` - Documentation
+- `system/vault-secrets.nix` - NixOS module
+- `scripts/create-host/vault_helper.py` - Vault API client
+- `terraform/vault/hosts-generated.tf` - Generated Terraform
+- `docs/vault-bootstrap-implementation.md` - This file
+- `docs/vault-bootstrap-testing.md` - Testing guide
+
+### Modified
+- `scripts/create-host/default.nix` - Add hvac dependency
+- `scripts/create-host/create_host.py` - Add Vault integration
+- `scripts/create-host/generators.py` - Add Vault Terraform generation
+- `scripts/create-host/manipulators.py` - Add wrapped token injection
+- `terraform/cloud-init.tf` - Inject Vault credentials
+- `terraform/vms.tf` - Support vault_wrapped_token field
+- `hosts/template2/bootstrap.nix` - Unwrap token and store credentials
+- `system/default.nix` - Import vault-secrets module
+- `flake.nix` - Add vault-fetch package
+
+### Unchanged
+- All existing sops-nix configuration
+- All existing service configurations
+- All existing host configurations
+- `/secrets/` directory
+
+## Future Enhancements
+
+### Phase 4e+ (Not in Scope)
+
+1. **Dynamic Secrets**
+   - Database credentials with rotation
+   - Cloud provider credentials
+   - SSH certificates
+
+2. **Secret Watcher**
+   - Monitor Vault for secret changes
+   - Automatically restart services on rotation
+   - Faster than periodic timers
+
+3. **PKI Integration** (Phase 4c)
+   - Migrate from step-ca to Vault PKI
+   - Automatic certificate issuance
+   - Short-lived certificates
+
+4. **Audit Logging**
+   - Track secret access
+   - Alert on suspicious patterns
+   - Compliance reporting
+
+5. **Multi-Environment**
+   - Dev/staging/prod separation
+   - Per-environment Vault namespaces
+   - Separate AppRoles per environment
+
+## Conclusion
+
+Phase 4d successfully implements automatic Vault integration for new NixOS hosts with:
+
+- ✅ Zero-touch provisioning
+- ✅ Secure credential distribution
+- ✅ Graceful degradation
+- ✅ Backward compatibility
+- ✅ Production-ready error handling
+
+The infrastructure is ready for gradual migration of existing services from sops-nix to Vault.

docs/vault-bootstrap-testing.md

@@ -0,0 +1,419 @@
+# Phase 4d: Vault Bootstrap Integration - Testing Guide
+
+This guide walks through testing the complete Vault bootstrap workflow implemented in Phase 4d.
+
+## Prerequisites
+
+Before testing, ensure:
+
+1. **Vault server is running**: vault01 (vault01.home.2rjus.net:8200) is accessible
+2. **Vault access**: You have a Vault token with admin permissions (set `BAO_TOKEN` env var)
+3. **Terraform installed**: OpenTofu is available in your PATH
+4. **Git repository clean**: All Phase 4d changes are committed to a branch
+
+## Test Scenario: Create vaulttest01
+
+### Step 1: Create Test Host Configuration
+
+Run the create-host tool with Vault integration:
+
+```bash
+# Ensure you have Vault token
+export BAO_TOKEN="your-vault-admin-token"
+
+# Create test host
+nix run .#create-host -- \
+  --hostname vaulttest01 \
+  --ip 10.69.13.150/24 \
+  --cpu 2 \
+  --memory 2048 \
+  --disk 20G
+
+# If you need to regenerate (e.g., wrapped token expired):
+nix run .#create-host -- \
+  --hostname vaulttest01 \
+  --ip 10.69.13.150/24 \
+  --force
+```
+
+**What this does:**
+- Creates `hosts/vaulttest01/` configuration
+- Updates `flake.nix` with new host
+- Updates `terraform/vms.tf` with VM definition
+- Generates `terraform/vault/hosts-generated.tf` with AppRole and policy
+- Creates a wrapped token (24h TTL, single-use)
+- Adds wrapped token to VM configuration
+
+**Expected output:**
+```
+✓ All validations passed
+✓ Created hosts/vaulttest01/default.nix
+✓ Created hosts/vaulttest01/configuration.nix
+✓ Updated flake.nix
+✓ Updated terraform/vms.tf
+
+Configuring Vault integration...
+✓ Updated terraform/vault/hosts-generated.tf
+Applying Vault Terraform configuration...
+✓ Terraform applied successfully
+Reading AppRole credentials for vaulttest01...
+✓ Retrieved role_id
+✓ Generated secret_id
+Creating wrapped token (24h TTL, single-use)...
+✓ Created wrapped token: hvs.CAESIBw...
+⚠️  Token expires in 24 hours
+⚠️  Token can only be used once
+✓ Added wrapped token to terraform/vms.tf
+
+✓ Host configuration generated successfully!
+```
+
+### Step 2: Add Test Service Configuration
+
+Edit `hosts/vaulttest01/configuration.nix` to enable Vault and add a test service:
+
+```nix
+{ config, pkgs, lib, ... }:
+{
+  imports = [
+    ../../system
+    ../../common/vm
+  ];
+
+  # Enable Vault secrets management
+  vault.enable = true;
+
+  # Define a test secret
+  vault.secrets.test-service = {
+    secretPath = "hosts/vaulttest01/test-service";
+    restartTrigger = true;
+    restartInterval = "daily";
+    services = [ "vault-test" ];
+  };
+
+  # Create a test service that uses the secret
+  systemd.services.vault-test = {
+    description = "Test Vault secret fetching";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "vault-secret-test-service.service" ];
+
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+
+      ExecStart = pkgs.writeShellScript "vault-test" ''
+        echo "=== Vault Secret Test ==="
+        echo "Secret path: hosts/vaulttest01/test-service"
+
+        if [ -f /run/secrets/test-service/password ]; then
+          echo "✓ Password file exists"
+          echo "Password length: $(wc -c < /run/secrets/test-service/password)"
+        else
+          echo "✗ Password file missing!"
+          exit 1
+        fi
+
+        if [ -d /var/lib/vault/cache/test-service ]; then
+          echo "✓ Cache directory exists"
+        else
+          echo "✗ Cache directory missing!"
+          exit 1
+        fi
+
+        echo "Test successful!"
+      '';
+
+      StandardOutput = "journal+console";
+    };
+  };
+
+  # Rest of configuration...
+  networking.hostName = "vaulttest01";
+  networking.domain = "home.2rjus.net";
+
+  systemd.network.networks."10-lan" = {
+    matchConfig.Name = "ens18";
+    address = [ "10.69.13.150/24" ];
+    gateway = [ "10.69.13.1" ];
+    dns = [ "10.69.13.5" "10.69.13.6" ];
+    domains = [ "home.2rjus.net" ];
+  };
+
+  system.stateVersion = "25.11";
+}
+```
+
+### Step 3: Create Test Secrets in Vault
+
+Add test secrets to Vault using Terraform:
+
+Edit `terraform/vault/secrets.tf`:
+
+```hcl
+locals {
+  secrets = {
+    # ... existing secrets ...
+
+    # Test secret for vaulttest01
+    "hosts/vaulttest01/test-service" = {
+      auto_generate = true
+      password_length = 24
+    }
+  }
+}
+```
+
+Apply the Vault configuration:
+
+```bash
+cd terraform/vault
+tofu apply
+```
+
+**Verify the secret exists:**
+```bash
+export VAULT_ADDR=https://vault01.home.2rjus.net:8200
+export VAULT_SKIP_VERIFY=1
+
+vault kv get secret/hosts/vaulttest01/test-service
+```
+
+### Step 4: Deploy the VM
+
+**Important**: Deploy within 24 hours of creating the host (wrapped token TTL)
+
+```bash
+cd terraform
+tofu plan   # Review changes
+tofu apply  # Deploy VM
+```
+
+### Step 5: Monitor Bootstrap Process
+
+SSH into the VM and monitor the bootstrap:
+
+```bash
+# Watch bootstrap logs
+ssh root@vaulttest01
+journalctl -fu nixos-bootstrap.service
+
+# Expected log output:
+# Starting NixOS bootstrap for host: vaulttest01
+# Network connectivity confirmed
+# Unwrapping Vault token to get AppRole credentials...
+# Vault credentials unwrapped and stored successfully
+# Fetching and building NixOS configuration from flake...
+# Successfully built configuration for vaulttest01
+# Rebooting into new configuration...
+```
+
+### Step 6: Verify Vault Integration
+
+After the VM reboots, verify the integration:
+
+```bash
+ssh root@vaulttest01
+
+# Check AppRole credentials were stored
+ls -la /var/lib/vault/approle/
+# Expected: role-id and secret-id files with 600 permissions
+
+cat /var/lib/vault/approle/role-id
+# Should show a UUID
+
+# Check vault-secret service ran successfully
+systemctl status vault-secret-test-service.service
+# Should be active (exited)
+
+journalctl -u vault-secret-test-service.service
+# Should show successful secret fetch:
+# [vault-fetch] Authenticating to Vault at https://vault01.home.2rjus.net:8200
+# [vault-fetch] Successfully authenticated to Vault
+# [vault-fetch] Fetching secret from path: hosts/vaulttest01/test-service
+# [vault-fetch] Writing secrets to /run/secrets/test-service
+# [vault-fetch]   - Wrote secret key: password
+# [vault-fetch] Successfully fetched and cached secrets
+
+# Check test service passed
+systemctl status vault-test.service
+journalctl -u vault-test.service
+# Should show:
+# === Vault Secret Test ===
+# ✓ Password file exists
+# ✓ Cache directory exists
+# Test successful!
+
+# Verify secret files exist
+ls -la /run/secrets/test-service/
+# Should show password file with 400 permissions
+
+# Verify cache exists
+ls -la /var/lib/vault/cache/test-service/
+# Should show cached password file
+```
+
+## Test Scenarios
+
+### Scenario 1: Fresh Deployment
+✅ **Expected**: All secrets fetched successfully from Vault
+
+### Scenario 2: Service Restart
+```bash
+systemctl restart vault-test.service
+```
+✅ **Expected**: Secrets re-fetched from Vault, service starts successfully
+
+### Scenario 3: Vault Unreachable
+```bash
+# On vault01, stop Vault temporarily
+ssh root@vault01
+systemctl stop openbao
+
+# On vaulttest01, restart test service
+ssh root@vaulttest01
+systemctl restart vault-test.service
+journalctl -u vault-secret-test-service.service | tail -20
+```
+✅ **Expected**:
+- Warning logged: "Using cached secrets from /var/lib/vault/cache/test-service"
+- Service starts successfully using cached secrets
+
+```bash
+# Restore Vault
+ssh root@vault01
+systemctl start openbao
+```
+
+### Scenario 4: Secret Rotation
+```bash
+# Update secret in Vault
+vault kv put secret/hosts/vaulttest01/test-service password="new-secret-value"
+
+# On vaulttest01, trigger rotation
+ssh root@vaulttest01
+systemctl restart vault-secret-test-service.service
+
+# Verify new secret
+cat /run/secrets/test-service/password
+# Should show new value
+```
+✅ **Expected**: New secret fetched and cached
+
+### Scenario 5: Expired Wrapped Token
+```bash
+# Wait 24+ hours after create-host, then try to deploy
+cd terraform
+tofu apply
+```
+❌ **Expected**: Bootstrap fails with message about expired token
+
+**Fix (Option 1 - Regenerate token only):**
+```bash
+# Only regenerates the wrapped token, preserves all other configuration
+nix run .#create-host -- --hostname vaulttest01 --regenerate-token
+cd terraform
+tofu apply
+```
+
+**Fix (Option 2 - Full regeneration with --force):**
+```bash
+# Overwrites entire host configuration (including any manual changes)
+nix run .#create-host -- --hostname vaulttest01 --force
+cd terraform
+tofu apply
+```
+
+**Recommendation**: Use `--regenerate-token` to avoid losing manual configuration changes.
+
+### Scenario 6: Already-Used Wrapped Token
+Try to deploy the same VM twice without regenerating token.
+
+❌ **Expected**: Second bootstrap fails with "token already used" message
+
+## Cleanup
+
+After testing:
+
+```bash
+# Destroy test VM
+cd terraform
+tofu destroy -target=proxmox_vm_qemu.vm[\"vaulttest01\"]
+
+# Remove test secrets from Vault
+vault kv delete secret/hosts/vaulttest01/test-service
+
+# Remove host configuration (optional)
+git rm -r hosts/vaulttest01
+# Edit flake.nix to remove nixosConfigurations.vaulttest01
+# Edit terraform/vms.tf to remove vaulttest01
+# Edit terraform/vault/hosts-generated.tf to remove vaulttest01
+```
+
+## Success Criteria Checklist
+
+Phase 4d is considered successful when:
+
+- [x] create-host generates Vault configuration automatically
+- [x] New hosts receive AppRole credentials via cloud-init
+- [x] Bootstrap stores credentials in /var/lib/vault/approle/
+- [x] Services can fetch secrets using vault.secrets option
+- [x] Secrets extracted to individual files in /run/secrets/
+- [x] Cached secrets work when Vault is unreachable
+- [x] Periodic restart timers work for secret rotation
+- [x] Critical services excluded from auto-restart
+- [x] Test host deploys and verifies working
+- [x] sops-nix continues to work for existing services
+
+## Troubleshooting
+
+### Bootstrap fails with "Failed to unwrap Vault token"
+
+**Possible causes:**
+- Token already used (wrapped tokens are single-use)
+- Token expired (24h TTL)
+- Invalid token
+- Vault unreachable
+
+**Solution:**
+```bash
+# Regenerate token
+nix run .#create-host -- --hostname vaulttest01 --force
+cd terraform && tofu apply
+```
+
+### Secret fetch fails with authentication error
+
+**Check:**
+```bash
+# Verify AppRole exists
+vault read auth/approle/role/vaulttest01
+
+# Verify policy exists
+vault policy read host-vaulttest01
+
+# Test authentication manually
+ROLE_ID=$(cat /var/lib/vault/approle/role-id)
+SECRET_ID=$(cat /var/lib/vault/approle/secret-id)
+vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"
+```
+
+### Cache not working
+
+**Check:**
+```bash
+# Verify cache directory exists and has files
+ls -la /var/lib/vault/cache/test-service/
+
+# Check permissions
+stat /var/lib/vault/cache/test-service/password
+# Should be 600 (rw-------)
+```
+
+## Next Steps
+
+After successful testing:
+
+1. Gradually migrate existing services from sops-nix to Vault
+2. Consider implementing secret watcher for faster rotation (future enhancement)
+3. Phase 4c: Migrate from step-ca to OpenBao PKI
+4. Eventually deprecate and remove sops-nix

docs/vault/auto-unseal.md

@@ -0,0 +1,178 @@
+# OpenBao TPM2 Auto-Unseal Setup
+
+This document describes the one-time setup process for enabling TPM2-based auto-unsealing on vault01.
+
+## Overview
+
+The auto-unseal feature uses systemd's `LoadCredentialEncrypted` with TPM2 to securely store and retrieve an unseal key. On service start, systemd automatically decrypts the credential using the VM's TPM, and the service unseals OpenBao.
+
+## Prerequisites
+
+- OpenBao must be initialized (`bao operator init` completed)
+- You must have at least one unseal key from the initialization
+- vault01 must have a TPM2 device (virtual TPM for Proxmox VMs)
+
+## Initial Setup
+
+Perform these steps on vault01 after deploying the service configuration:
+
+### 1. Save Unseal Key
+
+```bash
+# Create temporary file with one of your unseal keys
+echo "paste-your-unseal-key-here" > /tmp/unseal-key.txt
+```
+
+### 2. Encrypt with TPM2
+
+```bash
+# Encrypt the key using TPM2 binding
+systemd-creds encrypt \
+  --with-key=tpm2 \
+  --name=unseal-key \
+  /tmp/unseal-key.txt \
+  /var/lib/openbao/unseal-key.cred
+
+# Set proper ownership and permissions
+chown openbao:openbao /var/lib/openbao/unseal-key.cred
+chmod 600 /var/lib/openbao/unseal-key.cred
+```
+
+### 3. Cleanup
+
+```bash
+# Securely delete the plaintext key
+shred -u /tmp/unseal-key.txt
+```
+
+### 4. Test Auto-Unseal
+
+```bash
+# Restart the service - it should auto-unseal
+systemctl restart openbao
+
+# Verify it's unsealed
+bao status
+# Should show: Sealed = false
+```
+
+## TPM PCR Binding
+
+The default `--with-key=tpm2` binds the credential to PCR 7 (Secure Boot state). For stricter binding that includes firmware and boot state:
+
+```bash
+systemd-creds encrypt \
+  --with-key=tpm2 \
+  --tpm2-pcrs=0+7+14 \
+  --name=unseal-key \
+  /tmp/unseal-key.txt \
+  /var/lib/openbao/unseal-key.cred
+```
+
+PCR meanings:
+- **PCR 0**: BIOS/UEFI firmware measurements
+- **PCR 7**: Secure Boot state (UEFI variables)
+- **PCR 14**: MOK (Machine Owner Key) state
+
+**Trade-off**: Stricter PCR binding improves security but may require re-encrypting the credential after firmware updates or kernel changes.
+
+## Re-provisioning
+
+If you need to reprovision vault01 from scratch:
+
+1. **Before destroying**: Back up your root token and all unseal keys (stored securely offline)
+2. **After recreating the VM**:
+   - Initialize OpenBao: `bao operator init`
+   - Follow the setup steps above to encrypt a new unseal key with TPM2
+3. **Restore data** (if migrating): Copy `/var/lib/openbao` from backup
+
+## Handling System Changes
+
+**After firmware updates, kernel updates, or boot configuration changes**, PCR values may change, causing TPM decryption to fail.
+
+### Symptoms
+- Service fails to start
+- Logs show: `Failed to decrypt credentials`
+- OpenBao remains sealed after reboot
+
+### Fix
+1. Unseal manually with one of your offline unseal keys:
+   ```bash
+   bao operator unseal
+   ```
+
+2. Re-encrypt the credential with updated PCR values:
+   ```bash
+   echo "your-unseal-key" > /tmp/unseal-key.txt
+   systemd-creds encrypt \
+     --with-key=tpm2 \
+     --name=unseal-key \
+     /tmp/unseal-key.txt \
+     /var/lib/openbao/unseal-key.cred
+   chown openbao:openbao /var/lib/openbao/unseal-key.cred
+   chmod 600 /var/lib/openbao/unseal-key.cred
+   shred -u /tmp/unseal-key.txt
+   ```
+
+3. Restart the service:
+   ```bash
+   systemctl restart openbao
+   ```
+
+## Security Considerations
+
+### What This Protects Against
+- **Data at rest**: Vault data is encrypted and cannot be accessed without unsealing
+- **VM snapshot theft**: An attacker with a VM snapshot cannot decrypt the unseal key without the TPM state
+- **TPM binding**: The key can only be decrypted by the same VM with matching PCR values
+
+### What This Does NOT Protect Against
+- **Compromised host**: If an attacker gains root access to vault01 while running, they can access unsealed data
+- **Boot-time attacks**: If an attacker can modify the boot process to match PCR values, they may retrieve the key
+- **VM console access**: An attacker with VM console access during boot could potentially access the unsealed vault
+
+### Recommendations
+- **Keep offline backups** of root token and all unseal keys in a secure location (password manager, encrypted USB, etc.)
+- **Use Shamir secret sharing**: The default 5-key threshold means even if the TPM key is compromised, an attacker needs the other keys
+- **Monitor access**: Use OpenBao's audit logging to detect unauthorized access
+- **Consider stricter PCR binding** (PCR 0+7+14) for production, accepting the maintenance overhead
+
+## Troubleshooting
+
+### Check if credential exists
+```bash
+ls -la /var/lib/openbao/unseal-key.cred
+```
+
+### Test credential decryption manually
+```bash
+# Should output your unseal key if TPM decryption works
+systemd-creds decrypt /var/lib/openbao/unseal-key.cred -
+```
+
+### View service logs
+```bash
+journalctl -u openbao -n 50
+```
+
+### Manual unseal
+```bash
+bao operator unseal
+# Enter one of your offline unseal keys when prompted
+```
+
+### Check TPM status
+```bash
+# Check if TPM2 is available
+ls /dev/tpm*
+
+# View TPM PCR values
+tpm2_pcrread
+```
+
+## References
+
+- [systemd.exec - Credentials](https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Credentials)
+- [systemd-creds man page](https://www.freedesktop.org/software/systemd/man/systemd-creds.html)
+- [TPM2 PCR Documentation](https://uapi-group.org/specifications/specs/linux_tpm_pcr_registry/)
+- [OpenBao Documentation](https://openbao.org/docs/)

flake.lock

@@ -1,49 +1,111 @@
 {
   "nodes": {
-    "backup-helper": {
+    "alerttonotify": {
       "inputs": {
         "nixpkgs": [
           "nixpkgs-unstable"
         ]
       },
       "locked": {
-        "lastModified": 1727998045,
-        "narHash": "sha256-BOvQHqs50Hk1sevvuJQai83kYuwTN27FTgmTitPsJtw=",
+        "lastModified": 1739310461,
+        "narHash": "sha256-GscftfATX84Aae9FObrQOe+hr5MsEma2Fc5fdzuu3hA=",
         "ref": "master",
-        "rev": "162c35769cc06b117b6753eb93460af650b64921",
-        "revCount": 31,
+        "rev": "53915cec6356be1a2d44ac2cbd0a71b32d679e6f",
+        "revCount": 7,
         "type": "git",
-        "url": "https://git.t-juice.club/torjus/backup-helper"
+        "url": "https://git.t-juice.club/torjus/alerttonotify"
       },
       "original": {
         "ref": "master",
         "type": "git",
-        "url": "https://git.t-juice.club/torjus/backup-helper"
+        "url": "https://git.t-juice.club/torjus/alerttonotify"
+      }
+    },
+    "homelab-deploy": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unstable"
+        ]
+      },
+      "locked": {
+        "lastModified": 1770447502,
+        "narHash": "sha256-xH1PNyE3ydj4udhe1IpK8VQxBPZETGLuORZdSWYRmSU=",
+        "ref": "master",
+        "rev": "79db119d1ca6630023947ef0a65896cc3307c2ff",
+        "revCount": 22,
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/homelab-deploy"
+      },
+      "original": {
+        "ref": "master",
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/homelab-deploy"
+      }
+    },
+    "labmon": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unstable"
+        ]
+      },
+      "locked": {
+        "lastModified": 1748983975,
+        "narHash": "sha256-DA5mOqxwLMj/XLb4hvBU1WtE6cuVej7PjUr8N0EZsCE=",
+        "ref": "master",
+        "rev": "040a73e891a70ff06ec7ab31d7167914129dbf7d",
+        "revCount": 17,
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/labmon"
+      },
+      "original": {
+        "ref": "master",
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/labmon"
+      }
+    },
+    "nixos-exporter": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs-unstable"
+        ]
+      },
+      "locked": {
+        "lastModified": 1770422522,
+        "narHash": "sha256-WmIFnquu4u58v8S2bOVWmknRwHn4x88CRfBFTzJ1inQ=",
+        "ref": "refs/heads/master",
+        "rev": "cf0ce858997af4d8dcc2ce10393ff393e17fc911",
+        "revCount": 11,
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/nixos-exporter"
+      },
+      "original": {
+        "type": "git",
+        "url": "https://git.t-juice.club/torjus/nixos-exporter"
       }
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1729307008,
-        "narHash": "sha256-QUvb6epgKi9pCu9CttRQW4y5NqJ+snKr1FZpG/x3Wtc=",
+        "lastModified": 1770136044,
+        "narHash": "sha256-tlFqNG/uzz2++aAmn4v8J0vAkV3z7XngeIIB3rM3650=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "a9b86fc2290b69375c5542b622088eb6eca2a7c3",
+        "rev": "e576e3c9cf9bad747afcddd9e34f51d18c855b4e",
         "type": "github"
       },
       "original": {
         "owner": "nixos",
-        "ref": "nixos-24.05",
+        "ref": "nixos-25.11",
         "repo": "nixpkgs",
         "type": "github"
       }
     },
     "nixpkgs-unstable": {
       "locked": {
-        "lastModified": 1729256560,
-        "narHash": "sha256-/uilDXvCIEs3C9l73JTACm4quuHUsIHcns1c+cHUJwA=",
+        "lastModified": 1770197578,
+        "narHash": "sha256-AYqlWrX09+HvGs8zM6ebZ1pwUqjkfpnv8mewYwAo+iM=",
         "owner": "nixos",
         "repo": "nixpkgs",
-        "rev": "4c2fcb090b1f3e5b47eaa7bd33913b574a11e0a0",
+        "rev": "00c21e4c93d963c50d4c0c89bfa84ed6e0694df2",
         "type": "github"
       },
       "original": {
@@ -55,7 +117,10 @@
     },
     "root": {
       "inputs": {
-        "backup-helper": "backup-helper",
+        "alerttonotify": "alerttonotify",
+        "homelab-deploy": "homelab-deploy",
+        "labmon": "labmon",
+        "nixos-exporter": "nixos-exporter",
         "nixpkgs": "nixpkgs",
         "nixpkgs-unstable": "nixpkgs-unstable",
         "sops-nix": "sops-nix"
@@ -65,17 +130,14 @@
       "inputs": {
         "nixpkgs": [
           "nixpkgs-unstable"
-        ],
-        "nixpkgs-stable": [
-          "nixpkgs"
         ]
       },
       "locked": {
-        "lastModified": 1729394972,
-        "narHash": "sha256-fADlzOzcSaGsrO+THUZ8SgckMMc7bMQftztKFCLVcFI=",
+        "lastModified": 1770145881,
+        "narHash": "sha256-ktjWTq+D5MTXQcL9N6cDZXUf9kX8JBLLBLT0ZyOTSYY=",
         "owner": "Mic92",
         "repo": "sops-nix",
-        "rev": "c504fd7ac946d7a1b17944d73b261ca0a0b226a5",
+        "rev": "17eea6f3816ba6568b8c81db8a4e6ca438b30b7c",
         "type": "github"
       },
       "original": {

flake.nix

@@ -2,16 +2,27 @@
   description = "Homelab v5 Nixos Server Configurations";
 
   inputs = {
-    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-24.05";
+    nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-25.11";
     nixpkgs-unstable.url = "github:nixos/nixpkgs?ref=nixos-unstable";
 
     sops-nix = {
       url = "github:Mic92/sops-nix";
       inputs.nixpkgs.follows = "nixpkgs-unstable";
-      inputs.nixpkgs-stable.follows = "nixpkgs";
     };
-    backup-helper = {
-      url = "git+https://git.t-juice.club/torjus/backup-helper?ref=master";
+    alerttonotify = {
+      url = "git+https://git.t-juice.club/torjus/alerttonotify?ref=master";
+      inputs.nixpkgs.follows = "nixpkgs-unstable";
+    };
+    labmon = {
+      url = "git+https://git.t-juice.club/torjus/labmon?ref=master";
+      inputs.nixpkgs.follows = "nixpkgs-unstable";
+    };
+    nixos-exporter = {
+      url = "git+https://git.t-juice.club/torjus/nixos-exporter";
+      inputs.nixpkgs.follows = "nixpkgs-unstable";
+    };
+    homelab-deploy = {
+      url = "git+https://git.t-juice.club/torjus/homelab-deploy?ref=master";
       inputs.nixpkgs.follows = "nixpkgs-unstable";
     };
   };
@@ -22,7 +33,10 @@
       nixpkgs,
       nixpkgs-unstable,
       sops-nix,
-      backup-helper,
+      alerttonotify,
+      labmon,
+      nixos-exporter,
+      homelab-deploy,
       ...
     }@inputs:
     let
@@ -33,6 +47,33 @@
           config.allowUnfree = true;
         };
       };
+      commonOverlays = [
+        overlay-unstable
+        alerttonotify.overlays.default
+        labmon.overlays.default
+      ];
+      # Common modules applied to all hosts
+      commonModules = [
+        (
+          { config, pkgs, ... }:
+          {
+            nixpkgs.overlays = commonOverlays;
+            system.configurationRevision = self.rev or self.dirtyRev or "dirty";
+          }
+        )
+        sops-nix.nixosModules.sops
+        nixos-exporter.nixosModules.default
+        homelab-deploy.nixosModules.default
+        ./modules/homelab
+      ];
+      allSystems = [
+        "x86_64-linux"
+        "aarch64-linux"
+        "x86_64-darwin"
+        "aarch64-darwin"
+      ];
+      forAllSystems =
+        f: nixpkgs.lib.genAttrs allSystems (system: f { pkgs = import nixpkgs { inherit system; }; });
     in
     {
       nixosConfigurations = {
@@ -41,15 +82,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ns1
-            sops-nix.nixosModules.sops
           ];
         };
         ns2 = nixpkgs.lib.nixosSystem {
@@ -57,64 +91,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ns2
-            sops-nix.nixosModules.sops
-          ];
-        };
-        ns3 = nixpkgs.lib.nixosSystem {
-          inherit system;
-          specialArgs = {
-            inherit inputs self sops-nix;
-          };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
-            ./hosts/ns3
-            sops-nix.nixosModules.sops
-          ];
-        };
-        ns4 = nixpkgs.lib.nixosSystem {
-          inherit system;
-          specialArgs = {
-            inherit inputs self sops-nix;
-          };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
-            ./hosts/ns4
-            sops-nix.nixosModules.sops
-          ];
-        };
-        nixos-test1 = nixpkgs.lib.nixosSystem {
-          inherit system;
-          specialArgs = {
-            inherit inputs self sops-nix;
-          };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
-            ./hosts/nixos-test1
-            sops-nix.nixosModules.sops
-            backup-helper.nixosModules.backup-helper
           ];
         };
         ha1 = nixpkgs.lib.nixosSystem {
@@ -122,50 +100,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ha1
-            sops-nix.nixosModules.sops
-            backup-helper.nixosModules.backup-helper
-          ];
-        };
-        inc1 = nixpkgs.lib.nixosSystem {
-          inherit system;
-          specialArgs = {
-            inherit inputs self sops-nix;
-          };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
-            ./hosts/inc1
-            sops-nix.nixosModules.sops
-            # backup-helper.nixosModules.backup-helper
-          ];
-        };
-        inc2 = nixpkgs.lib.nixosSystem {
-          inherit system;
-          specialArgs = {
-            inherit inputs self sops-nix;
-          };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
-            ./hosts/inc2
-            sops-nix.nixosModules.sops
-            # backup-helper.nixosModules.backup-helper
           ];
         };
         template1 = nixpkgs.lib.nixosSystem {
@@ -173,15 +109,17 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/template
-            sops-nix.nixosModules.sops
+          ];
+        };
+        template2 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/template2
           ];
         };
         http-proxy = nixpkgs.lib.nixosSystem {
@@ -189,15 +127,8 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/http-proxy
-            sops-nix.nixosModules.sops
           ];
         };
         ca = nixpkgs.lib.nixosSystem {
@@ -205,17 +136,104 @@
           specialArgs = {
             inherit inputs self sops-nix;
           };
-          modules = [
-            (
-              { config, pkgs, ... }:
-              {
-                nixpkgs.overlays = [ overlay-unstable ];
-              }
-            )
+          modules = commonModules ++ [
             ./hosts/ca
-            sops-nix.nixosModules.sops
+          ];
+        };
+        monitoring01 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/monitoring01
+            labmon.nixosModules.labmon
+          ];
+        };
+        jelly01 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/jelly01
+          ];
+        };
+        nix-cache01 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/nix-cache01
+          ];
+        };
+        pgdb1 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/pgdb1
+          ];
+        };
+        nats1 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/nats1
+          ];
+        };
+        testvm01 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/testvm01
+          ];
+        };
+        vault01 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/vault01
+          ];
+        };
+        vaulttest01 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self sops-nix;
+          };
+          modules = commonModules ++ [
+            ./hosts/vaulttest01
           ];
         };
       };
+      packages = forAllSystems (
+        { pkgs }:
+        {
+          create-host = pkgs.callPackage ./scripts/create-host { };
+          vault-fetch = pkgs.callPackage ./scripts/vault-fetch { };
+        }
+      );
+      devShells = forAllSystems (
+        { pkgs }:
+        {
+          default = pkgs.mkShell {
+            packages = [
+              pkgs.ansible
+              pkgs.opentofu
+              pkgs.openbao
+              (pkgs.callPackage ./scripts/create-host { })
+              homelab-deploy.packages.${pkgs.system}.default
+            ];
+          };
+        }
+      );
     };
 }

hosts/ca/configuration.nix

@@ -8,6 +8,7 @@
     ../template/hardware-configuration.nix
 
     ../../system
+    ../../common/vm
   ];
 
   nixpkgs.config.allowUnfree = true;
@@ -35,7 +36,7 @@
       "10.69.13.12/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };

hosts/ha1/configuration.nix

@@ -1,12 +1,17 @@
-{ config, lib, pkgs, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
 
 {
-  imports =
-    [
-      ../template/hardware-configuration.nix
+  imports = [
+    ../template/hardware-configuration.nix
 
-      ../../system
-    ];
+    ../../system
+    ../../common/vm
+  ];
 
   nixpkgs.config.allowUnfree = true;
   # Use the systemd-boot EFI boot loader.
@@ -33,13 +38,16 @@
       "10.69.13.9/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };
   time.timeZone = "Europe/Oslo";
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
   nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [
     vim
@@ -47,16 +55,36 @@
     git
   ];
 
+  # Vault secrets management
+  vault.enable = true;
+  homelab.deploy.enable = true;
+  vault.secrets.backup-helper = {
+    secretPath = "shared/backup/password";
+    extractKey = "password";
+    outputDir = "/run/secrets/backup_helper_secret";
+    services = [ "restic-backups-ha1" ];
+  };
+
   # Backup service dirs
-  sops.secrets."backup_helper_secret" = { };
-  backup-helper = {
-    enable = true;
-    password-file = "/run/secrets/backup_helper_secret";
-    backup-dirs = [
+  services.restic.backups.ha1 = {
+    repository = "rest:http://10.69.12.52:8000/backup-nix";
+    passwordFile = "/run/secrets/backup_helper_secret";
+    paths = [
       "/var/lib/hass"
       "/var/lib/zigbee2mqtt"
       "/var/lib/mosquitto"
     ];
+    timerConfig = {
+      OnCalendar = "daily";
+      Persistent = true;
+      RandomizedDelaySec = "2h";
+    };
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 6"
+      "--keep-within 1d"
+    ];
   };
 
   # Open ports in the firewall.
@@ -67,4 +95,3 @@
 
   system.stateVersion = "23.11"; # Did you read the comment?
 }
-

hosts/http-proxy/configuration.nix

@@ -8,6 +8,21 @@
     ../template/hardware-configuration.nix
 
     ../../system
+    ../../common/vm
+  ];
+
+  homelab.dns.cnames = [
+    "nzbget"
+    "radarr"
+    "sonarr"
+    "ha"
+    "z2m"
+    "grafana"
+    "prometheus"
+    "alertmanager"
+    "jelly"
+    "pyroscope"
+    "pushgw"
   ];
 
   nixpkgs.config.allowUnfree = true;
@@ -35,7 +50,7 @@
       "10.69.13.11/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };
@@ -45,6 +60,9 @@
     "nix-command"
     "flakes"
   ];
+  vault.enable = true;
+  homelab.deploy.enable = true;
+
   nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [
     vim

hosts/http-proxy/default.nix

@@ -3,5 +3,6 @@
   imports = [
     ./configuration.nix
     ../../services/http-proxy
+    ./wireguard.nix
   ];
 }

hosts/http-proxy/wireguard.nix

@@ -0,0 +1,40 @@
+{ config, ... }:
+{
+  vault.secrets.wireguard = {
+    secretPath = "hosts/http-proxy/wireguard";
+    extractKey = "private_key";
+    outputDir = "/run/secrets/wireguard_private_key";
+    services = [ "wireguard-wg0" ];
+  };
+
+  networking.wireguard = {
+    enable = true;
+    useNetworkd = true;
+
+    interfaces = {
+      wg0 = {
+        ips = [ "10.69.222.3/24" ];
+        mtu = 1384;
+        listenPort = 51820;
+        privateKeyFile = "/run/secrets/wireguard_private_key";
+        peers = [
+          {
+            name = "docker2.t-juice.club";
+            endpoint = "docker2.t-juice.club:51820";
+            publicKey = "32Rb13wExcy8uI92JTnFdiOfkv0mlQ6f181WA741DHs=";
+            allowedIPs = [ "10.69.222.0/24" ];
+            persistentKeepalive = 25;
+          }
+        ];
+      };
+    };
+  };
+  homelab.monitoring.scrapeTargets = [{
+    job_name = "wireguard";
+    port = 9586;
+  }];
+
+  services.prometheus.exporters.wireguard = {
+    enable = true;
+  };
+}

hosts/inc1/configuration.nix

@@ -1,96 +0,0 @@
-# Edit this configuration file to define what should be installed on
-# your system. Help is available in the configuration.nix(5) man page, on
-# https://search.nixos.org/options and in the NixOS manual (`nixos-help`).
-
-{ config, lib, pkgs, ... }:
-
-{
-  imports =
-    [
-      # Include the results of the hardware scan.
-      ./hardware-configuration.nix
-      ../../system
-      ../../services/incus
-    ];
-
-  # Use the systemd-boot EFI boot loader.
-  boot.loader.systemd-boot.enable = true;
-  boot.loader.efi.canTouchEfiVariables = true;
-
-  boot.kernel.sysctl = {
-    "net.ipv4.ip_forward" = 1;
-  };
-
-  networking.hostName = "inc1";
-  networking.domain = "home.2rjus.net";
-  networking.useNetworkd = true;
-  networking.useDHCP = false;
-  networking.nftables.enable = true;
-  networking.firewall.trustedInterfaces = [ "vlan13" ];
-
-  services.resolved.enable = true;
-  networking.nameservers = [
-    "10.69.13.5"
-    "10.69.13.6"
-  ];
-
-  systemd.network.enable = true;
-  # Primary interface
-  systemd.network.networks."enp2s0" = {
-    matchConfig.Name = "enp2s0";
-    address = [
-      "10.69.12.80/24"
-    ];
-    networkConfig = {
-      VLAN = [ "enp2s0.13" ];
-    };
-    routes = [
-      { routeConfig.Gateway = "10.69.12.1"; }
-    ];
-    linkConfig.RequiredForOnline = "routable";
-  };
-
-  # VLAN 13 netdev
-  systemd.network.netdevs."enp2s0.13" = {
-    enable = true;
-    netdevConfig = {
-      Kind = "vlan";
-      Name = "enp2s0.13";
-    };
-    vlanConfig = {
-      Id = 13;
-    };
-  };
-
-  # # Bridge netdev
-  # systemd.network.netdevs."br13" = {
-  #   netdevConfig = {
-  #     Name = "br13";
-  #     Kind = "bridge";
-  #   };
-  # };
-
-  # # Bridge network
-  # systemd.network.networks."br13" = {
-  #   matchConfig.Name = "enp2s0.13";
-  #   networkConfig.Bridge = "br13";
-  # };
-
-  time.timeZone = "Europe/Oslo";
-
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
-  nix.settings.tarball-ttl = 0;
-  environment.systemPackages = with pkgs; [
-    tcpdump
-    vim
-    wget
-    git
-  ];
-
-  # Enable the OpenSSH daemon.
-  # services.openssh.enable = true;
-  # services.openssh.settings.PermitRootLogin = "yes";
-
-  system.stateVersion = "24.05"; # Did you read the comment?
-}
-

hosts/inc1/hardware-configuration.nix

@@ -1,41 +0,0 @@
-# Do not modify this file!  It was generated by ‘nixos-generate-config’
-# and may be overwritten by future invocations.  Please make changes
-# to /etc/nixos/configuration.nix instead.
-{ config, lib, pkgs, modulesPath, ... }:
-
-{
-  imports =
-    [ (modulesPath + "/installer/scan/not-detected.nix")
-    ];
-
-  boot.initrd.availableKernelModules = [ "xhci_pci" "nvme" "ahci" "usbhid" "usb_storage" "sd_mod" "rtsx_usb_sdmmc" ];
-  boot.initrd.kernelModules = [ ];
-  boot.kernelModules = [ "kvm-amd" ];
-  boot.extraModulePackages = [ ];
-
-  fileSystems."/" =
-    { device = "/dev/disk/by-uuid/faa60038-b3a4-448a-8909-49857818c955";
-      fsType = "xfs";
-    };
-
-  fileSystems."/boot" =
-    { device = "/dev/disk/by-uuid/7A94-A91C";
-      fsType = "vfat";
-      options = [ "fmask=0077" "dmask=0077" ];
-    };
-
-  swapDevices =
-    [ { device = "/dev/disk/by-uuid/f7a4f85e-0b4b-492d-a611-f50d2b915c2c"; }
-    ];
-
-  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
-  # (the default) this is the recommended approach. When using systemd-networkd it's
-  # still possible to use this option, but it's recommended to use it in conjunction
-  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
-  networking.useDHCP = lib.mkDefault true;
-  # networking.interfaces.enp2s0.useDHCP = lib.mkDefault true;
-  # networking.interfaces.wlp3s0.useDHCP = lib.mkDefault true;
-
-  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
-  hardware.cpu.amd.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
-}

hosts/inc2/configuration.nix

@@ -1,96 +0,0 @@
-# Edit this configuration file to define what should be installed on
-# your system. Help is available in the configuration.nix(5) man page, on
-# https://search.nixos.org/options and in the NixOS manual (`nixos-help`).
-
-{ config, lib, pkgs, ... }:
-
-{
-  imports =
-    [
-      # Include the results of the hardware scan.
-      ./hardware-configuration.nix
-      ../../system
-      ../../services/incus
-    ];
-
-  # Use the systemd-boot EFI boot loader.
-  boot.loader.systemd-boot.enable = true;
-  boot.loader.efi.canTouchEfiVariables = true;
-
-  boot.kernel.sysctl = {
-    "net.ipv4.ip_forward" = 1;
-  };
-
-  networking.hostName = "inc2";
-  networking.domain = "home.2rjus.net";
-  networking.useNetworkd = true;
-  networking.useDHCP = false;
-  networking.nftables.enable = true;
-  networking.firewall.trustedInterfaces = [ "vlan13" ];
-
-  services.resolved.enable = true;
-  networking.nameservers = [
-    "10.69.13.5"
-    "10.69.13.6"
-  ];
-
-  systemd.network.enable = true;
-  # Primary interface
-  systemd.network.networks."enp2s0" = {
-    matchConfig.Name = "enp2s0";
-    address = [
-      "10.69.12.81/24"
-    ];
-    networkConfig = {
-      VLAN = [ "enp2s0.13" ];
-    };
-    routes = [
-      { routeConfig.Gateway = "10.69.12.1"; }
-    ];
-    linkConfig.RequiredForOnline = "routable";
-  };
-
-  # VLAN 13 netdev
-  systemd.network.netdevs."enp2s0.13" = {
-    enable = true;
-    netdevConfig = {
-      Kind = "vlan";
-      Name = "enp2s0.13";
-    };
-    vlanConfig = {
-      Id = 13;
-    };
-  };
-
-  # # Bridge netdev
-  # systemd.network.netdevs."br13" = {
-  #   netdevConfig = {
-  #     Name = "br13";
-  #     Kind = "bridge";
-  #   };
-  # };
-
-  # # Bridge network
-  # systemd.network.networks."br13" = {
-  #   matchConfig.Name = "enp2s0.13";
-  #   networkConfig.Bridge = "br13";
-  # };
-
-  time.timeZone = "Europe/Oslo";
-
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
-  nix.settings.tarball-ttl = 0;
-  environment.systemPackages = with pkgs; [
-    tcpdump
-    vim
-    wget
-    git
-  ];
-
-  # Enable the OpenSSH daemon.
-  # services.openssh.enable = true;
-  # services.openssh.settings.PermitRootLogin = "yes";
-
-  system.stateVersion = "24.05"; # Did you read the comment?
-}
-

hosts/inc2/hardware-configuration.nix

@@ -1,33 +0,0 @@
-{ config, lib, pkgs, modulesPath, ... }:
-
-{
-  imports =
-    [
-      (modulesPath + "/installer/scan/not-detected.nix")
-    ];
-
-  boot.initrd.availableKernelModules = [ "xhci_pci" "ahci" "usb_storage" "usbhid" "sd_mod" "rtsx_usb_sdmmc" ];
-  boot.initrd.kernelModules = [ ];
-  boot.kernelModules = [ "kvm-amd" ];
-  boot.extraModulePackages = [ ];
-
-  fileSystems."/" =
-    {
-      device = "/dev/disk/by-uuid/3e7c311c-b1a3-4be7-b8bf-e497cba64302";
-      fsType = "btrfs";
-    };
-
-  fileSystems."/boot" =
-    {
-      device = "/dev/disk/by-uuid/F0D7-E5C1";
-      fsType = "vfat";
-      options = [ "fmask=0022" "dmask=0022" ];
-    };
-
-  swapDevices =
-    [{ device = "/dev/disk/by-uuid/1a06a36f-da61-4d36-b94e-b852836c328a"; }];
-
-  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
-  hardware.cpu.amd.updateMicrocode = lib.mkDefault config.hardware.enableRedistributableFirmware;
-}
-

hosts/jelly01/configuration.nix

@@ -1,19 +1,25 @@
-{ config, lib, pkgs, ... }:
+{
+  pkgs,
+  ...
+}:
 
 {
-  imports =
-    [
-      ../template/hardware-configuration.nix
+  imports = [
+    ../template/hardware-configuration.nix
 
-      ../../system
-    ];
+    ../../system
+    ../../common/vm
+  ];
 
   nixpkgs.config.allowUnfree = true;
   # Use the systemd-boot EFI boot loader.
-  boot.loader.grub.enable = true;
-  boot.loader.grub.device = "/dev/sda";
+  boot.loader.grub = {
+    enable = true;
+    device = "/dev/sda";
+    configurationLimit = 3;
+  };
 
-  networking.hostName = "nixos-test1";
+  networking.hostName = "jelly01";
   networking.domain = "home.2rjus.net";
   networking.useNetworkd = true;
   networking.useDHCP = false;
@@ -27,16 +33,19 @@
   systemd.network.networks."ens18" = {
     matchConfig.Name = "ens18";
     address = [
-      "10.69.13.10/24"
+      "10.69.13.14/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };
   time.timeZone = "Europe/Oslo";
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
   nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [
     vim
@@ -44,24 +53,17 @@
     git
   ];
 
+  services.qemuGuest.enable = true;
+
   # Open ports in the firewall.
   # networking.firewall.allowedTCPPorts = [ ... ];
   # networking.firewall.allowedUDPPorts = [ ... ];
   # Or disable the firewall altogether.
   networking.firewall.enable = false;
 
-  # Secrets
-  # Backup helper
-  sops.secrets."backup_helper_secret" = { };
-  backup-helper = {
+  zramSwap = {
     enable = true;
-    password-file = "/run/secrets/backup_helper_secret";
-    backup-dirs = [
-      "/etc/machine-id"
-      "/etc/os-release"
-    ];
   };
 
   system.stateVersion = "23.11"; # Did you read the comment?
 }
-

hosts/jelly01/default.nix

@@ -0,0 +1,7 @@
+{ ... }:
+{
+  imports = [
+    ./configuration.nix
+    ../../services/jellyfin
+  ];
+}

hosts/jump/configuration.nix

@@ -8,6 +8,9 @@
     ];
 
   nixpkgs.config.allowUnfree = true;
+
+  homelab.host.role = "bastion";
+
   # Use the systemd-boot EFI boot loader.
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/sda";
@@ -29,7 +32,7 @@
       "10.69.13.10/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };

hosts/monitoring01/configuration.nix

@@ -0,0 +1,165 @@
+{
+  pkgs,
+  ...
+}:
+
+{
+  imports = [
+    ../template/hardware-configuration.nix
+
+    ../../system
+    ../../common/vm
+  ];
+
+  nixpkgs.config.allowUnfree = true;
+  # Use the systemd-boot EFI boot loader.
+  boot.loader.grub = {
+    enable = true;
+    device = "/dev/sda";
+    configurationLimit = 3;
+  };
+
+  networking.hostName = "monitoring01";
+  networking.domain = "home.2rjus.net";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = true;
+  networking.nameservers = [
+    "10.69.13.5"
+    "10.69.13.6"
+  ];
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+    address = [
+      "10.69.13.13/24"
+    ];
+    routes = [
+      { Gateway = "10.69.13.1"; }
+    ];
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    vim
+    wget
+    git
+    sqlite
+  ];
+
+  services.qemuGuest.enable = true;
+
+  # Vault secrets management
+  vault.enable = true;
+  homelab.deploy.enable = true;
+  vault.secrets.backup-helper = {
+    secretPath = "shared/backup/password";
+    extractKey = "password";
+    outputDir = "/run/secrets/backup_helper_secret";
+    services = [ "restic-backups-grafana" "restic-backups-grafana-db" ];
+  };
+
+  services.restic.backups.grafana = {
+    repository = "rest:http://10.69.12.52:8000/backup-nix";
+    passwordFile = "/run/secrets/backup_helper_secret";
+    paths = [ "/var/lib/grafana/plugins" ];
+    timerConfig = {
+      OnCalendar = "daily";
+      Persistent = true;
+      RandomizedDelaySec = "2h";
+    };
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 6"
+      "--keep-within 1d"
+    ];
+  };
+
+  services.restic.backups.grafana-db = {
+    repository = "rest:http://10.69.12.52:8000/backup-nix";
+    passwordFile = "/run/secrets/backup_helper_secret";
+    command = [ "${pkgs.sqlite}/bin/sqlite3" "/var/lib/grafana/data/grafana.db" ".dump" ];
+    timerConfig = {
+      OnCalendar = "daily";
+      Persistent = true;
+      RandomizedDelaySec = "2h";
+    };
+    pruneOpts = [
+      "--keep-daily 7"
+      "--keep-weekly 4"
+      "--keep-monthly 6"
+      "--keep-within 1d"
+    ];
+  };
+
+  labmon = {
+    enable = true;
+
+    settings = {
+      ListenAddr = ":9969";
+      Profiling = true;
+      StepMonitors = [
+        {
+          Enabled = true;
+          BaseURL = "https://ca.home.2rjus.net";
+          RootID = "3381bda8015a86b9a3cd1851439d1091890a79005e0f1f7c4301fe4bccc29d80";
+        }
+      ];
+
+      TLSConnectionMonitors = [
+        {
+          Enabled = true;
+          Address = "ca.home.2rjus.net:443";
+          Verify = true;
+          Duration = "12h";
+        }
+        {
+          Enabled = true;
+          Address = "jelly.home.2rjus.net:443";
+          Verify = true;
+          Duration = "12h";
+        }
+        {
+          Enabled = true;
+          Address = "grafana.home.2rjus.net:443";
+          Verify = true;
+          Duration = "12h";
+        }
+        {
+          Enabled = true;
+          Address = "prometheus.home.2rjus.net:443";
+          Verify = true;
+          Duration = "12h";
+        }
+        {
+          Enabled = true;
+          Address = "alertmanager.home.2rjus.net:443";
+          Verify = true;
+          Duration = "12h";
+        }
+        {
+          Enabled = true;
+          Address = "pyroscope.home.2rjus.net:443";
+          Verify = true;
+          Duration = "12h";
+        }
+      ];
+    };
+  };
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  system.stateVersion = "23.11"; # Did you read the comment?
+}

hosts/monitoring01/default.nix

@@ -0,0 +1,7 @@
+{ ... }:
+{
+  imports = [
+    ./configuration.nix
+    ../../services/monitoring
+  ];
+}

hosts/nats1/configuration.nix

@@ -1,25 +1,29 @@
-{ config, lib, pkgs, ... }:
+{
+  pkgs,
+  ...
+}:
 
 {
-  imports =
-    [
-      ../template/hardware-configuration.nix
+  imports = [
+    ../template/hardware-configuration.nix
 
-      ../../system
-      ../../services/ns/master-authorative.nix
-      ../../services/ns/resolver.nix
-    ];
+    ../../system
+    ../../common/vm
+  ];
 
   nixpkgs.config.allowUnfree = true;
   # Use the systemd-boot EFI boot loader.
-  boot.loader.grub.enable = true;
-  boot.loader.grub.device = "/dev/sda";
+  boot.loader.grub = {
+    enable = true;
+    device = "/dev/sda";
+    configurationLimit = 3;
+  };
 
-  networking.hostName = "ns3";
+  networking.hostName = "nats1";
   networking.domain = "home.2rjus.net";
   networking.useNetworkd = true;
   networking.useDHCP = false;
-  services.resolved.enable = false;
+  services.resolved.enable = true;
   networking.nameservers = [
     "10.69.13.5"
     "10.69.13.6"
@@ -29,16 +33,20 @@
   systemd.network.networks."ens18" = {
     matchConfig.Name = "ens18";
     address = [
-      "10.69.13.7/24"
+      "10.69.13.17/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };
   time.timeZone = "Europe/Oslo";
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [
     vim
     wget
@@ -53,4 +61,3 @@
 
   system.stateVersion = "23.11"; # Did you read the comment?
 }
-

hosts/nats1/default.nix

@@ -1,5 +1,7 @@
-{ ... }: {
+{ ... }:
+{
   imports = [
     ./configuration.nix
+    ../../services/nats
   ];
 }

hosts/nix-cache01/configuration.nix

@@ -0,0 +1,76 @@
+{
+  pkgs,
+  ...
+}:
+
+{
+  imports = [
+    ../template/hardware-configuration.nix
+
+    ../../system
+    ../../common/vm
+  ];
+
+  homelab.dns.cnames = [ "nix-cache" "actions1" ];
+
+  homelab.host.role = "build-host";
+
+  fileSystems."/nix" = {
+    device = "/dev/disk/by-label/nixcache";
+    fsType = "xfs";
+  };
+  nixpkgs.config.allowUnfree = true;
+  # Use the systemd-boot EFI boot loader.
+  boot.loader.grub = {
+    enable = true;
+    device = "/dev/sda";
+    configurationLimit = 3;
+  };
+
+  networking.hostName = "nix-cache01";
+  networking.domain = "home.2rjus.net";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = true;
+  networking.nameservers = [
+    "10.69.13.5"
+    "10.69.13.6"
+  ];
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+    address = [
+      "10.69.13.15/24"
+    ];
+    routes = [
+      { Gateway = "10.69.13.1"; }
+    ];
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  vault.enable = true;
+  homelab.deploy.enable = true;
+
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    vim
+    wget
+    git
+  ];
+
+  services.qemuGuest.enable = true;
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  system.stateVersion = "24.05"; # Did you read the comment?
+}

hosts/nix-cache01/default.nix

@@ -0,0 +1,9 @@
+{ ... }:
+{
+  imports = [
+    ./configuration.nix
+    ../../services/nix-cache
+    ../../services/actions-runner
+    ./zram.nix
+  ];
+}

hosts/nix-cache01/zram.nix

@@ -0,0 +1,6 @@
+{ ... }:
+{
+  zramSwap = {
+    enable = true;
+  };
+}

hosts/ns1/configuration.nix

@@ -1,14 +1,19 @@
-{ config, lib, pkgs, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
 
 {
-  imports =
-    [
-      ../template/hardware-configuration.nix
+  imports = [
+    ../template/hardware-configuration.nix
 
-      ../../system
-      ../../services/ns/master-authorative.nix
-      ../../services/ns/resolver.nix
-    ];
+    ../../system
+    ../../services/ns/master-authorative.nix
+    ../../services/ns/resolver.nix
+    ../../common/vm
+  ];
 
   nixpkgs.config.allowUnfree = true;
   # Use the systemd-boot EFI boot loader.
@@ -32,13 +37,24 @@
       "10.69.13.5/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };
   time.timeZone = "Europe/Oslo";
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  vault.enable = true;
+  homelab.deploy.enable = true;
+
+  homelab.host = {
+    role = "dns";
+    labels.dns_role = "primary";
+  };
+
   nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [
     vim
@@ -54,4 +70,3 @@
 
   system.stateVersion = "23.11"; # Did you read the comment?
 }
-

hosts/ns2/configuration.nix

@@ -1,14 +1,19 @@
-{ config, lib, pkgs, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
 
 {
-  imports =
-    [
-      ../template/hardware-configuration.nix
+  imports = [
+    ../template/hardware-configuration.nix
 
-      ../../system
-      ../../services/ns/secondary-authorative.nix
-      ../../services/ns/resolver.nix
-    ];
+    ../../system
+    ../../services/ns/secondary-authorative.nix
+    ../../services/ns/resolver.nix
+    ../../common/vm
+  ];
 
   nixpkgs.config.allowUnfree = true;
   # Use the systemd-boot EFI boot loader.
@@ -32,13 +37,24 @@
       "10.69.13.6/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };
   time.timeZone = "Europe/Oslo";
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  vault.enable = true;
+  homelab.deploy.enable = true;
+
+  homelab.host = {
+    role = "dns";
+    labels.dns_role = "secondary";
+  };
+
   environment.systemPackages = with pkgs; [
     vim
     wget
@@ -53,4 +69,3 @@
 
   system.stateVersion = "23.11"; # Did you read the comment?
 }
-

hosts/ns3/hardware-configuration.nix

@@ -1,36 +0,0 @@
-{ config, lib, pkgs, modulesPath, ... }:
-
-{
-  imports =
-    [
-      (modulesPath + "/profiles/qemu-guest.nix")
-    ];
-
-  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "virtio_pci" "virtio_scsi" "sd_mod" "sr_mod" ];
-  boot.initrd.kernelModules = [ ];
-  # boot.kernelModules = [ ];
-  # boot.extraModulePackages = [ ];
-
-  fileSystems."/" =
-    {
-      device = "/dev/disk/by-uuid/6889aba9-61ed-4687-ab10-e5cf4017ac8d";
-      fsType = "xfs";
-    };
-
-  fileSystems."/boot" =
-    {
-      device = "/dev/disk/by-uuid/BC07-3B7A";
-      fsType = "vfat";
-    };
-
-  swapDevices =
-    [{ device = "/dev/disk/by-uuid/64e5757b-6625-4dd2-aa2a-66ca93444d23"; }];
-
-  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
-  # (the default) this is the recommended approach. When using systemd-networkd it's
-  # still possible to use this option, but it's recommended to use it in conjunction
-  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
-  # networking.interfaces.ens18.useDHCP = lib.mkDefault true;
-
-  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
-}

hosts/ns4/hardware-configuration.nix

@@ -1,36 +0,0 @@
-{ config, lib, pkgs, modulesPath, ... }:
-
-{
-  imports =
-    [
-      (modulesPath + "/profiles/qemu-guest.nix")
-    ];
-
-  boot.initrd.availableKernelModules = [ "ata_piix" "uhci_hcd" "virtio_pci" "virtio_scsi" "sd_mod" "sr_mod" ];
-  boot.initrd.kernelModules = [ ];
-  # boot.kernelModules = [ ];
-  # boot.extraModulePackages = [ ];
-
-  fileSystems."/" =
-    {
-      device = "/dev/disk/by-uuid/6889aba9-61ed-4687-ab10-e5cf4017ac8d";
-      fsType = "xfs";
-    };
-
-  fileSystems."/boot" =
-    {
-      device = "/dev/disk/by-uuid/BC07-3B7A";
-      fsType = "vfat";
-    };
-
-  swapDevices =
-    [{ device = "/dev/disk/by-uuid/64e5757b-6625-4dd2-aa2a-66ca93444d23"; }];
-
-  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
-  # (the default) this is the recommended approach. When using systemd-networkd it's
-  # still possible to use this option, but it's recommended to use it in conjunction
-  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
-  # networking.interfaces.ens18.useDHCP = lib.mkDefault true;
-
-  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
-}

hosts/pgdb1/configuration.nix

@@ -1,25 +1,29 @@
-{ config, lib, pkgs, ... }:
+{
+  pkgs,
+  ...
+}:
 
 {
-  imports =
-    [
-      ../template/hardware-configuration.nix
+  imports = [
+    ../template/hardware-configuration.nix
 
-      ../../system
-      ../../services/ns/secondary-authorative.nix
-      ../../services/ns/resolver.nix
-    ];
+    ../../system
+    ../../common/vm
+  ];
 
   nixpkgs.config.allowUnfree = true;
   # Use the systemd-boot EFI boot loader.
-  boot.loader.grub.enable = true;
-  boot.loader.grub.device = "/dev/sda";
+  boot.loader.grub = {
+    enable = true;
+    device = "/dev/sda";
+    configurationLimit = 3;
+  };
 
-  networking.hostName = "ns4";
+  networking.hostName = "pgdb1";
   networking.domain = "home.2rjus.net";
   networking.useNetworkd = true;
   networking.useDHCP = false;
-  services.resolved.enable = false;
+  services.resolved.enable = true;
   networking.nameservers = [
     "10.69.13.5"
     "10.69.13.6"
@@ -29,16 +33,20 @@
   systemd.network.networks."ens18" = {
     matchConfig.Name = "ens18";
     address = [
-      "10.69.13.8/24"
+      "10.69.13.16/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.13.1"; }
+      { Gateway = "10.69.13.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };
   time.timeZone = "Europe/Oslo";
 
-  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
   environment.systemPackages = with pkgs; [
     vim
     wget
@@ -53,4 +61,3 @@
 
   system.stateVersion = "23.11"; # Did you read the comment?
 }
-

hosts/pgdb1/default.nix

@@ -0,0 +1,7 @@
+{ ... }:
+{
+  imports = [
+    ./configuration.nix
+    ../../services/postgres
+  ];
+}

hosts/template/configuration.nix

@@ -8,6 +8,14 @@
       ../../system
     ];
 
+  # Template host - exclude from DNS zone generation
+  homelab.dns.enable = false;
+
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+  };
+
 
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/sda";
@@ -28,7 +36,7 @@
       "10.69.8.250/24"
     ];
     routes = [
-      { routeConfig.Gateway = "10.69.8.1"; }
+      { Gateway = "10.69.8.1"; }
     ];
     linkConfig.RequiredForOnline = "routable";
   };

hosts/template/hardware-configuration.nix

@@ -1,4 +1,10 @@
-{ config, lib, pkgs, modulesPath, ... }:
+{
+  config,
+  lib,
+  pkgs,
+  modulesPath,
+  ...
+}:
 
 {
   imports = [
@@ -13,17 +19,17 @@
     "sr_mod"
   ];
   boot.initrd.kernelModules = [ "dm-snapshot" ];
-  boot.kernelModules = [ ];
+  boot.kernelModules = [
+    "ptp_kvm"
+  ];
   boot.extraModulePackages = [ ];
 
-  fileSystems."/" =
-    {
-      device = "/dev/disk/by-label/root";
-      fsType = "xfs";
-    };
+  fileSystems."/" = {
+    device = "/dev/disk/by-label/root";
+    fsType = "xfs";
+  };
 
-  swapDevices =
-    [{ device = "/dev/disk/by-label/swap"; }];
+  swapDevices = [ { device = "/dev/disk/by-label/swap"; } ];
 
   # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
   # (the default) this is the recommended approach. When using systemd-networkd it's
@@ -34,4 +40,3 @@
 
   nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
 }
-

hosts/template/scripts.nix

@@ -1,7 +1,9 @@
 { pkgs, ... }:
 let
-  prepare-host-script = pkgs.writeShellScriptBin "prepare-host.sh"
-    ''
+  prepare-host-script = pkgs.writeShellApplication {
+    name = "prepare-host.sh";
+    runtimeInputs = [ pkgs.age ];
+    text = ''
       echo "Removing machine-id"
       rm -f /etc/machine-id || true
 
@@ -24,8 +26,9 @@ let
       echo "Generate age key"
       rm -rf /var/lib/sops-nix || true
       mkdir -p /var/lib/sops-nix
-      ${pkgs.age}/bin/age-keygen -o /var/lib/sops-nix/key.txt
+      age-keygen -o /var/lib/sops-nix/key.txt
     '';
+  };
 in
 {
   environment.systemPackages = [ prepare-host-script ];

hosts/template2/bootstrap.nix

@@ -0,0 +1,120 @@
+{ pkgs, config, lib, ... }:
+let
+  bootstrap-script = pkgs.writeShellApplication {
+    name = "nixos-bootstrap";
+    runtimeInputs = with pkgs; [ systemd curl nixos-rebuild jq git ];
+    text = ''
+      set -euo pipefail
+
+      # Read hostname set by cloud-init (from Terraform VM name via user-data)
+      # Cloud-init sets the system hostname from user-data.txt, so we read it from hostnamectl
+      HOSTNAME=$(hostnamectl hostname)
+      echo "DEBUG: Hostname from hostnamectl: '$HOSTNAME'"
+
+      echo "Starting NixOS bootstrap for host: $HOSTNAME"
+      echo "Waiting for network connectivity..."
+
+      # Verify we can reach the git server via HTTPS (doesn't respond to ping)
+      if ! curl -s --connect-timeout 5 --max-time 10 https://git.t-juice.club >/dev/null 2>&1; then
+        echo "ERROR: Cannot reach git.t-juice.club via HTTPS"
+        echo "Check network configuration and DNS settings"
+        exit 1
+      fi
+
+      echo "Network connectivity confirmed"
+
+      # Unwrap Vault token and store AppRole credentials (if provided)
+      if [ -n "''${VAULT_WRAPPED_TOKEN:-}" ]; then
+        echo "Unwrapping Vault token to get AppRole credentials..."
+
+        VAULT_ADDR="''${VAULT_ADDR:-https://vault01.home.2rjus.net:8200}"
+
+        # Unwrap the token to get role_id and secret_id
+        UNWRAP_RESPONSE=$(curl -sk -X POST \
+          -H "X-Vault-Token: $VAULT_WRAPPED_TOKEN" \
+          "$VAULT_ADDR/v1/sys/wrapping/unwrap") || {
+          echo "WARNING: Failed to unwrap Vault token (network error)"
+          echo "Vault secrets will not be available, but continuing bootstrap..."
+        }
+
+        # Check if unwrap was successful
+        if [ -n "$UNWRAP_RESPONSE" ] && echo "$UNWRAP_RESPONSE" | jq -e '.data' >/dev/null 2>&1; then
+          ROLE_ID=$(echo "$UNWRAP_RESPONSE" | jq -r '.data.role_id')
+          SECRET_ID=$(echo "$UNWRAP_RESPONSE" | jq -r '.data.secret_id')
+
+          # Store credentials
+          mkdir -p /var/lib/vault/approle
+          echo "$ROLE_ID" > /var/lib/vault/approle/role-id
+          echo "$SECRET_ID" > /var/lib/vault/approle/secret-id
+          chmod 600 /var/lib/vault/approle/role-id
+          chmod 600 /var/lib/vault/approle/secret-id
+
+          echo "Vault credentials unwrapped and stored successfully"
+        else
+          echo "WARNING: Failed to unwrap Vault token"
+          if [ -n "$UNWRAP_RESPONSE" ]; then
+            echo "Response: $UNWRAP_RESPONSE"
+          fi
+          echo "Possible causes:"
+          echo "  - Token already used (wrapped tokens are single-use)"
+          echo "  - Token expired (24h TTL)"
+          echo "  - Invalid token"
+          echo ""
+          echo "To regenerate token, run: create-host --hostname $HOSTNAME --force"
+          echo ""
+          echo "Vault secrets will not be available, but continuing bootstrap..."
+        fi
+      else
+        echo "No Vault wrapped token provided (VAULT_WRAPPED_TOKEN not set)"
+        echo "Skipping Vault credential setup"
+      fi
+
+      echo "Fetching and building NixOS configuration from flake..."
+
+      # Read git branch from environment, default to master
+      BRANCH="''${NIXOS_FLAKE_BRANCH:-master}"
+      echo "Using git branch: $BRANCH"
+
+      # Build and activate the host-specific configuration
+      FLAKE_URL="git+https://git.t-juice.club/torjus/nixos-servers.git?ref=$BRANCH#''${HOSTNAME}"
+
+      if nixos-rebuild boot --flake "$FLAKE_URL"; then
+        echo "Successfully built configuration for $HOSTNAME"
+        echo "Rebooting into new configuration..."
+        sleep 2
+        systemctl reboot
+      else
+        echo "ERROR: nixos-rebuild failed for $HOSTNAME"
+        echo "Check that flake has configuration for this hostname"
+        echo "Manual intervention required - system will not reboot"
+        exit 1
+      fi
+    '';
+  };
+in
+{
+  systemd.services."nixos-bootstrap" = {
+    description = "Bootstrap NixOS configuration from flake on first boot";
+
+    # Wait for cloud-init to finish setting hostname and network to be online
+    after = [ "cloud-config.service" "network-online.target" ];
+    wants = [ "network-online.target" ];
+    requires = [ "cloud-config.service" ];
+
+    # Run on boot
+    wantedBy = [ "multi-user.target" ];
+
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+      ExecStart = "${bootstrap-script}/bin/nixos-bootstrap";
+
+      # Read environment variables from cloud-init (set by cloud-init write_files)
+      EnvironmentFile = "-/run/cloud-init-env";
+
+      # Logging to journald
+      StandardOutput = "journal+console";
+      StandardError = "journal+console";
+    };
+  };
+}

hosts/template2/configuration.nix

@@ -0,0 +1,75 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+{
+  imports = [
+    ./hardware-configuration.nix
+    ../../system/sshd.nix
+  ];
+
+  # Root user with no password but SSH key access for bootstrapping
+  users.users.root = {
+    hashedPassword = "";
+    openssh.authorizedKeys.keys = [
+      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAwfb2jpKrBnCw28aevnH8HbE5YbcMXpdaVv2KmueDu6 torjus@gunter"
+    ];
+  };
+
+  # Proxmox image-specific configuration
+  # Configure storage to use local-zfs instead of local-lvm
+  image.modules.proxmox = {
+    proxmox.qemuConf.virtio0 = lib.mkForce "local-zfs:vm-9999-disk-0";
+    proxmox.qemuConf.boot = lib.mkForce "order=virtio0";
+    proxmox.cloudInit.defaultStorage = lib.mkForce "local-zfs";
+  };
+
+  # Configure cloud-init to use ConfigDrive datasource (used by Proxmox)
+  services.cloud-init.settings = {
+    datasource_list = [ "ConfigDrive" "NoCloud" ];
+  };
+
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+  };
+
+  boot.loader.grub.enable = true;
+  boot.loader.grub.device = "/dev/vda";
+  networking.hostName = "nixos-template2";
+  networking.domain = "home.2rjus.net";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = true;
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+    networkConfig.DHCP = "ipv4";
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    age
+    vim
+    wget
+    git
+  ];
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  system.stateVersion = "25.11";
+}

hosts/template2/default.nix

@@ -0,0 +1,10 @@
+{ ... }:
+{
+  imports = [
+    ./hardware-configuration.nix
+    ./configuration.nix
+    ./scripts.nix
+    ./bootstrap.nix
+    ../../system/packages.nix
+  ];
+}

hosts/template2/hardware-configuration.nix

@@ -0,0 +1,45 @@
+{
+  config,
+  lib,
+  pkgs,
+  modulesPath,
+  ...
+}:
+
+{
+  imports = [
+    (modulesPath + "/profiles/qemu-guest.nix")
+  ];
+  boot.initrd.availableKernelModules = [
+    "ata_piix"
+    "uhci_hcd"
+    "virtio_pci"
+    "virtio_scsi"
+    "sd_mod"
+    "sr_mod"
+  ];
+  boot.initrd.kernelModules = [ "dm-snapshot" ];
+  boot.kernelModules = [
+    "ptp_kvm"
+    "virtio_rng"  # Provides entropy from host for fast SSH key generation
+  ];
+  boot.extraModulePackages = [ ];
+
+  # Filesystem configuration matching Proxmox image builder output
+  fileSystems."/" = lib.mkDefault {
+    device = "/dev/disk/by-label/nixos";
+    fsType = "ext4";
+    options = [ "x-systemd.growfs" ];
+  };
+
+  swapDevices = lib.mkDefault [ ];
+
+  # Enables DHCP on each ethernet and wireless interface. In case of scripted networking
+  # (the default) this is the recommended approach. When using systemd-networkd it's
+  # still possible to use this option, but it's recommended to use it in conjunction
+  # with explicit per-interface declarations with `networking.interfaces.<interface>.useDHCP`.
+  networking.useDHCP = lib.mkDefault true;
+  # networking.interfaces.ens18.useDHCP = lib.mkDefault true;
+
+  nixpkgs.hostPlatform = lib.mkDefault "x86_64-linux";
+}

hosts/template2/scripts.nix

@@ -0,0 +1,36 @@
+{ pkgs, ... }:
+let
+  prepare-host-script = pkgs.writeShellApplication {
+    name = "prepare-host.sh";
+    runtimeInputs = [ pkgs.age ];
+    text = ''
+      echo "Removing machine-id"
+      rm -f /etc/machine-id || true
+
+      echo "Removing SSH host keys"
+      rm -f /etc/ssh/ssh_host_* || true
+
+      echo "Restarting SSH"
+      systemctl restart sshd
+
+      echo "Removing temporary files"
+      rm -rf /tmp/* || true
+
+      echo "Removing logs"
+      journalctl --rotate || true
+      journalctl --vacuum-time=1s || true
+
+      echo "Removing cache"
+      rm -rf /var/cache/* || true
+
+      echo "Generate age key"
+      rm -rf /var/lib/sops-nix || true
+      mkdir -p /var/lib/sops-nix
+      age-keygen -o /var/lib/sops-nix/key.txt
+    '';
+  };
+in
+{
+  environment.systemPackages = [ prepare-host-script ];
+  users.motd = "Prepare host by running 'prepare-host.sh'.";
+}

hosts/testvm01/configuration.nix

@@ -0,0 +1,69 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+{
+  imports = [
+    ../template2/hardware-configuration.nix
+
+    ../../system
+    ../../common/vm
+  ];
+
+  # Test VM - exclude from DNS zone generation
+  homelab.dns.enable = false;
+
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+  };
+
+  nixpkgs.config.allowUnfree = true;
+  boot.loader.grub.enable = true;
+  boot.loader.grub.device = "/dev/vda";
+
+  networking.hostName = "testvm01";
+  networking.domain = "home.2rjus.net";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = false;
+  networking.nameservers = [
+    "10.69.13.5"
+    "10.69.13.6"
+  ];
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+    address = [
+      "10.69.13.101/24"
+    ];
+    routes = [
+      { Gateway = "10.69.13.1"; }
+    ];
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    vim
+    wget
+    git
+  ];
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  system.stateVersion = "25.11"; # Did you read the comment?
+}

hosts/testvm01/default.nix

@@ -2,4 +2,4 @@
   imports = [
     ./configuration.nix
   ];
-}
+}

hosts/vault01/configuration.nix

@@ -0,0 +1,67 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+{
+  imports = [
+    ../template2/hardware-configuration.nix
+
+    ../../system
+    ../../common/vm
+    ../../services/vault
+  ];
+
+  homelab.dns.cnames = [ "vault" ];
+
+  homelab.host.role = "vault";
+
+  nixpkgs.config.allowUnfree = true;
+  boot.loader.grub.enable = true;
+  boot.loader.grub.device = "/dev/vda";
+
+  networking.hostName = "vault01";
+  networking.domain = "home.2rjus.net";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = true;
+  networking.nameservers = [
+    "10.69.13.5"
+    "10.69.13.6"
+  ];
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+    address = [
+      "10.69.13.19/24"
+    ];
+    routes = [
+      { Gateway = "10.69.13.1"; }
+    ];
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    vim
+    wget
+    git
+  ];
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  system.stateVersion = "25.11"; # Did you read the comment?
+}
+

hosts/vault01/default.nix

@@ -2,4 +2,4 @@
   imports = [
     ./configuration.nix
   ];
-}
+}

hosts/vaulttest01/configuration.nix

@@ -0,0 +1,135 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+let
+  vault-test-script = pkgs.writeShellApplication {
+    name = "vault-test";
+    text = ''
+      echo "=== Vault Secret Test ==="
+      echo "Secret path: hosts/vaulttest01/test-service"
+
+      if [ -f /run/secrets/test-service/password ]; then
+        echo "✓ Password file exists"
+        echo "Password length: $(wc -c < /run/secrets/test-service/password)"
+      else
+        echo "✗ Password file missing!"
+        exit 1
+      fi
+
+      if [ -d /var/lib/vault/cache/test-service ]; then
+        echo "✓ Cache directory exists"
+      else
+        echo "✗ Cache directory missing!"
+        exit 1
+      fi
+
+      echo "Test successful!"
+    '';
+  };
+in
+{
+  imports = [
+    ../template2/hardware-configuration.nix
+
+    ../../system
+    ../../common/vm
+  ];
+
+  homelab.host = {
+    tier = "test";
+    priority = "low";
+    role = "vault";
+  };
+
+  nixpkgs.config.allowUnfree = true;
+  boot.loader.grub.enable = true;
+  boot.loader.grub.device = "/dev/vda";
+
+  networking.hostName = "vaulttest01";
+  networking.domain = "home.2rjus.net";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = true;
+  networking.nameservers = [
+    "10.69.13.5"
+    "10.69.13.6"
+  ];
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+    address = [
+      "10.69.13.150/24"
+    ];
+    routes = [
+      { Gateway = "10.69.13.1"; }
+    ];
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    vim
+    wget
+    git
+    htop # test deploy verification
+  ];
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  # Testing config
+  # Enable Vault secrets management
+  vault.enable = true;
+  homelab.deploy.enable = true;
+
+  # Define a test secret
+  vault.secrets.test-service = {
+    secretPath = "hosts/vaulttest01/test-service";
+    restartTrigger = true;
+    restartInterval = "daily";
+    services = [ "vault-test" ];
+  };
+
+  # Create a test service that uses the secret
+  systemd.services.vault-test = {
+    description = "Test Vault secret fetching";
+    wantedBy = [ "multi-user.target" ];
+    after = [ "vault-secret-test-service.service" ];
+
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+
+      ExecStart = lib.getExe vault-test-script;
+
+      StandardOutput = "journal+console";
+    };
+  };
+
+  # Test ACME certificate issuance from OpenBao PKI
+  # Override the global ACME server (from system/acme.nix) to use OpenBao instead of step-ca
+  security.acme.defaults.server = lib.mkForce "https://vault01.home.2rjus.net:8200/v1/pki_int/acme/directory";
+
+  # Request a certificate for this host
+  # Using HTTP-01 challenge with standalone listener on port 80
+  security.acme.certs."vaulttest01.home.2rjus.net" = {
+    listenHTTP = ":80";
+    enableDebugLogs = true;
+  };
+
+  system.stateVersion = "25.11"; # Did you read the comment?
+}
+

hosts/vaulttest01/default.nix

@@ -2,4 +2,4 @@
   imports = [
     ./configuration.nix
   ];
-}
+}

inventory

@@ -0,0 +1,27 @@
+#!/usr/bin/env python
+
+import json
+import subprocess
+
+IGNORED_HOSTS = [
+    "inc1",
+    "inc2",
+    "template1",
+]
+
+result = subprocess.run(["nix", "flake", "show", "--json"], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+results = json.loads(result.stdout)
+
+configs = results.get("nixosConfigurations")
+hosts = [x for x in configs.keys() if x not in IGNORED_HOSTS]
+
+output = {
+    "all": {
+        "hosts": hosts,
+        "vars": {
+            "ansible_python_interpreter": "/run/current-system/sw/bin/python3"
+        },
+    }
+}
+
+print(json.dumps(output))

lib/dns-zone.nix

@@ -0,0 +1,160 @@
+{ lib }:
+let
+  # Pad string on the right to reach a fixed width
+  rightPad = width: str:
+    let
+      len = builtins.stringLength str;
+      padding = if len >= width then "" else lib.strings.replicate (width - len) " ";
+    in
+    str + padding;
+
+  # Extract IP address from CIDR notation (e.g., "10.69.13.5/24" -> "10.69.13.5")
+  extractIP = address:
+    let
+      parts = lib.splitString "/" address;
+    in
+    builtins.head parts;
+
+  # Check if a network interface name looks like a VPN/tunnel interface
+  isVpnInterface = ifaceName:
+    lib.hasPrefix "wg" ifaceName ||
+    lib.hasPrefix "tun" ifaceName ||
+    lib.hasPrefix "tap" ifaceName ||
+    lib.hasPrefix "vti" ifaceName;
+
+  # Extract DNS information from a single host configuration
+  # Returns null if host should not be included in DNS
+  extractHostDNS = name: hostConfig:
+    let
+      cfg = hostConfig.config;
+      # Handle cases where homelab module might not be imported
+      dnsConfig = (cfg.homelab or { }).dns or { enable = true; cnames = [ ]; };
+      hostname = cfg.networking.hostName;
+      networks = cfg.systemd.network.networks or { };
+
+      # Filter out VPN interfaces and find networks with static addresses
+      # Check matchConfig.Name instead of network unit name (which can have prefixes like "40-")
+      physicalNetworks = lib.filterAttrs
+        (netName: netCfg:
+          let
+            ifaceName = netCfg.matchConfig.Name or "";
+          in
+          !(isVpnInterface ifaceName) && (netCfg.address or [ ]) != [ ])
+        networks;
+
+      # Get addresses from physical networks only
+      networkAddresses = lib.flatten (
+        lib.mapAttrsToList
+          (netName: netCfg: netCfg.address or [ ])
+          physicalNetworks
+      );
+
+      # Get the first address, if any
+      firstAddress = if networkAddresses != [ ] then builtins.head networkAddresses else null;
+
+      # Check if host uses DHCP (no static address)
+      usesDHCP = firstAddress == null ||
+        lib.any
+          (netName: (networks.${netName}.networkConfig.DHCP or "no") != "no")
+          (lib.attrNames networks);
+    in
+    if !(dnsConfig.enable or true) || firstAddress == null then
+      null
+    else
+      {
+        inherit hostname;
+        ip = extractIP firstAddress;
+        cnames = dnsConfig.cnames or [ ];
+      };
+
+  # Generate A record line
+  generateARecord = hostname: ip:
+    "${rightPad 20 hostname}IN      A       ${ip}";
+
+  # Generate CNAME record line
+  generateCNAME = alias: target:
+    "${rightPad 20 alias}IN      CNAME   ${target}";
+
+  # Generate zone file from flake configurations and external hosts
+  generateZone =
+    { self
+    , externalHosts
+    , serial
+    , domain ? "home.2rjus.net"
+    , ttl ? 1800
+    , refresh ? 3600
+    , retry ? 900
+    , expire ? 1209600
+    , minTtl ? 120
+    , nameservers ? [ "ns1" "ns2" ]
+    , adminEmail ? "admin.test.2rjus.net"
+    }:
+    let
+      # Extract DNS info from all flake hosts
+      nixosConfigs = self.nixosConfigurations or { };
+      hostDNSList = lib.filter (x: x != null) (
+        lib.mapAttrsToList extractHostDNS nixosConfigs
+      );
+
+      # Sort hosts by IP for consistent output
+      sortedHosts = lib.sort (a: b: a.ip < b.ip) hostDNSList;
+
+      # Generate A records for flake hosts
+      flakeARecords = lib.concatMapStringsSep "\n" (host:
+        generateARecord host.hostname host.ip
+      ) sortedHosts;
+
+      # Generate CNAMEs for flake hosts
+      flakeCNAMEs = lib.concatMapStringsSep "\n" (host:
+        lib.concatMapStringsSep "\n" (cname:
+          generateCNAME cname host.hostname
+        ) host.cnames
+      ) (lib.filter (h: h.cnames != [ ]) sortedHosts);
+
+      # Generate A records for external hosts
+      externalARecords = lib.concatStringsSep "\n" (
+        lib.mapAttrsToList (name: ip:
+          generateARecord name ip
+        ) (externalHosts.aRecords or { })
+      );
+
+      # Generate CNAMEs for external hosts
+      externalCNAMEs = lib.concatStringsSep "\n" (
+        lib.mapAttrsToList (alias: target:
+          generateCNAME alias target
+        ) (externalHosts.cnames or { })
+      );
+
+      # NS records
+      nsRecords = lib.concatMapStringsSep "\n" (ns:
+        "                    IN      NS      ${ns}.${domain}."
+      ) nameservers;
+
+      # SOA record
+      soa = ''
+        $ORIGIN ${domain}.
+        $TTL ${toString ttl}
+        @       IN      SOA     ns1.${domain}.      ${adminEmail}. (
+                                ${toString serial}                    ; serial number
+                                ${toString refresh}                    ; refresh
+                                ${toString retry}                     ; retry
+                                ${toString expire}                 ; expire
+                                ${toString minTtl}                     ; ttl
+                                )'';
+    in
+    lib.concatStringsSep "\n\n" (lib.filter (s: s != "") [
+      soa
+      nsRecords
+      "; Flake-managed hosts (auto-generated)"
+      flakeARecords
+      (if flakeCNAMEs != "" then "; Flake-managed CNAMEs\n${flakeCNAMEs}" else "")
+      "; External hosts (not managed by this flake)"
+      externalARecords
+      (if externalCNAMEs != "" then "; External CNAMEs\n${externalCNAMEs}" else "")
+      ""
+    ]);
+
+in
+{
+  inherit extractIP extractHostDNS generateARecord generateCNAME generateZone;
+}

lib/monitoring.nix

@@ -0,0 +1,145 @@
+{ lib }:
+let
+  # Extract IP address from CIDR notation (e.g., "10.69.13.5/24" -> "10.69.13.5")
+  extractIP = address:
+    let
+      parts = lib.splitString "/" address;
+    in
+    builtins.head parts;
+
+  # Check if a network interface name looks like a VPN/tunnel interface
+  isVpnInterface = ifaceName:
+    lib.hasPrefix "wg" ifaceName ||
+    lib.hasPrefix "tun" ifaceName ||
+    lib.hasPrefix "tap" ifaceName ||
+    lib.hasPrefix "vti" ifaceName;
+
+  # Extract monitoring info from a single host configuration
+  # Returns null if host should not be included
+  extractHostMonitoring = name: hostConfig:
+    let
+      cfg = hostConfig.config;
+      monConfig = (cfg.homelab or { }).monitoring or { enable = true; scrapeTargets = [ ]; };
+      dnsConfig = (cfg.homelab or { }).dns or { enable = true; };
+      hostname = cfg.networking.hostName;
+      networks = cfg.systemd.network.networks or { };
+
+      # Filter out VPN interfaces and find networks with static addresses
+      physicalNetworks = lib.filterAttrs
+        (netName: netCfg:
+          let
+            ifaceName = netCfg.matchConfig.Name or "";
+          in
+          !(isVpnInterface ifaceName) && (netCfg.address or [ ]) != [ ])
+        networks;
+
+      # Get addresses from physical networks only
+      networkAddresses = lib.flatten (
+        lib.mapAttrsToList
+          (netName: netCfg: netCfg.address or [ ])
+          physicalNetworks
+      );
+
+      firstAddress = if networkAddresses != [ ] then builtins.head networkAddresses else null;
+    in
+    if !(monConfig.enable or true) || !(dnsConfig.enable or true) || firstAddress == null then
+      null
+    else
+      {
+        inherit hostname;
+        ip = extractIP firstAddress;
+        scrapeTargets = monConfig.scrapeTargets or [ ];
+      };
+
+  # Generate node-exporter targets from all flake hosts
+  generateNodeExporterTargets = self: externalTargets:
+    let
+      nixosConfigs = self.nixosConfigurations or { };
+      hostList = lib.filter (x: x != null) (
+        lib.mapAttrsToList extractHostMonitoring nixosConfigs
+      );
+      flakeTargets = map (host: "${host.hostname}.home.2rjus.net:9100") hostList;
+    in
+    flakeTargets ++ (externalTargets.nodeExporter or [ ]);
+
+  # Generate scrape configs from all flake hosts and external targets
+  generateScrapeConfigs = self: externalTargets:
+    let
+      nixosConfigs = self.nixosConfigurations or { };
+      hostList = lib.filter (x: x != null) (
+        lib.mapAttrsToList extractHostMonitoring nixosConfigs
+      );
+
+      # Collect all scrapeTargets from all hosts, grouped by job_name
+      allTargets = lib.flatten (map
+        (host:
+          map
+            (target: {
+              inherit (target) job_name port metrics_path scheme scrape_interval honor_labels;
+              hostname = host.hostname;
+            })
+            host.scrapeTargets
+        )
+        hostList
+      );
+
+      # Group targets by job_name
+      grouped = lib.groupBy (t: t.job_name) allTargets;
+
+      # Generate a scrape config for each job
+      flakeScrapeConfigs = lib.mapAttrsToList
+        (jobName: targets:
+          let
+            first = builtins.head targets;
+            targetAddrs = map
+              (t:
+                let
+                  portStr = toString t.port;
+                in
+                "${t.hostname}.home.2rjus.net:${portStr}")
+              targets;
+            config = {
+              job_name = jobName;
+              static_configs = [{
+                targets = targetAddrs;
+              }];
+            }
+            // (lib.optionalAttrs (first.metrics_path != "/metrics") {
+              metrics_path = first.metrics_path;
+            })
+            // (lib.optionalAttrs (first.scheme != "http") {
+              scheme = first.scheme;
+            })
+            // (lib.optionalAttrs (first.scrape_interval != null) {
+              scrape_interval = first.scrape_interval;
+            })
+            // (lib.optionalAttrs first.honor_labels {
+              honor_labels = true;
+            });
+          in
+          config
+        )
+        grouped;
+
+      # External scrape configs
+      externalScrapeConfigs = map
+        (ext: {
+          job_name = ext.job_name;
+          static_configs = [{
+            targets = ext.targets;
+          }];
+        } // (lib.optionalAttrs (ext ? metrics_path) {
+          metrics_path = ext.metrics_path;
+        }) // (lib.optionalAttrs (ext ? scheme) {
+          scheme = ext.scheme;
+        }) // (lib.optionalAttrs (ext ? scrape_interval) {
+          scrape_interval = ext.scrape_interval;
+        }))
+        (externalTargets.scrapeConfigs or [ ]);
+    in
+    flakeScrapeConfigs ++ externalScrapeConfigs;
+
+in
+{
+  inherit extractHostMonitoring generateNodeExporterTargets generateScrapeConfigs;
+}

modules/homelab/default.nix

@@ -0,0 +1,9 @@
+{ ... }:
+{
+  imports = [
+    ./deploy.nix
+    ./dns.nix
+    ./host.nix
+    ./monitoring.nix
+  ];
+}

modules/homelab/deploy.nix

@@ -0,0 +1,16 @@
+{ config, lib, ... }:
+
+{
+  options.homelab.deploy = {
+    enable = lib.mkEnableOption "homelab-deploy listener for NATS-based deployments";
+  };
+
+  config = {
+    assertions = [
+      {
+        assertion = config.homelab.deploy.enable -> config.vault.enable;
+        message = "homelab.deploy.enable requires vault.enable to be true (needed for NKey secret)";
+      }
+    ];
+  };
+}

modules/homelab/dns.nix

@@ -0,0 +1,20 @@
+{ config, lib, ... }:
+let
+  cfg = config.homelab.dns;
+in
+{
+  options.homelab.dns = {
+    enable = lib.mkOption {
+      type = lib.types.bool;
+      default = true;
+      description = "Include this host in DNS zone generation";
+    };
+
+    cnames = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ ];
+      description = "CNAME records pointing to this host";
+      example = [ "web" "api" ];
+    };
+  };
+}

modules/homelab/host.nix

@@ -0,0 +1,28 @@
+{ lib, ... }:
+{
+  options.homelab.host = {
+    tier = lib.mkOption {
+      type = lib.types.enum [ "test" "prod" ];
+      default = "prod";
+      description = "Deployment tier - controls which credentials can deploy to this host";
+    };
+
+    priority = lib.mkOption {
+      type = lib.types.enum [ "high" "low" ];
+      default = "high";
+      description = "Alerting priority - low priority hosts have relaxed thresholds";
+    };
+
+    role = lib.mkOption {
+      type = lib.types.nullOr lib.types.str;
+      default = null;
+      description = "Primary role of this host (dns, database, monitoring, etc.)";
+    };
+
+    labels = lib.mkOption {
+      type = lib.types.attrsOf lib.types.str;
+      default = { };
+      description = "Additional free-form labels (e.g., dns_role = 'primary')";
+    };
+  };
+}

modules/homelab/monitoring.nix

@@ -0,0 +1,50 @@
+{ config, lib, ... }:
+let
+  cfg = config.homelab.monitoring;
+in
+{
+  options.homelab.monitoring = {
+    enable = lib.mkOption {
+      type = lib.types.bool;
+      default = true;
+      description = "Include this host in Prometheus node-exporter scrape targets";
+    };
+
+    scrapeTargets = lib.mkOption {
+      type = lib.types.listOf (lib.types.submodule {
+        options = {
+          job_name = lib.mkOption {
+            type = lib.types.str;
+            description = "Prometheus scrape job name";
+          };
+          port = lib.mkOption {
+            type = lib.types.port;
+            description = "Port to scrape metrics from";
+          };
+          metrics_path = lib.mkOption {
+            type = lib.types.str;
+            default = "/metrics";
+            description = "HTTP path to scrape metrics from";
+          };
+          scheme = lib.mkOption {
+            type = lib.types.str;
+            default = "http";
+            description = "HTTP scheme (http or https)";
+          };
+          scrape_interval = lib.mkOption {
+            type = lib.types.nullOr lib.types.str;
+            default = null;
+            description = "Override the global scrape interval for this target";
+          };
+          honor_labels = lib.mkOption {
+            type = lib.types.bool;
+            default = false;
+            description = "Whether to honor labels from the scraped target";
+          };
+        };
+      });
+      default = [ ];
+      description = "Additional Prometheus scrape targets exposed by this host";
+    };
+  };
+}

playbooks/build-and-deploy-template.yml

@@ -0,0 +1,101 @@
+---
+- name: Build and deploy NixOS Proxmox template
+  hosts: localhost
+  gather_facts: false
+
+  vars:
+    template_name: "template2"
+    nixos_config: "template2"
+    proxmox_node: "pve1.home.2rjus.net"  # Change to your Proxmox node name
+    proxmox_host: "pve1.home.2rjus.net"  # Change to your Proxmox host
+    template_vmid: 9000  # Template VM ID
+    storage: "local-zfs"
+
+  tasks:
+    - name: Build NixOS image
+      ansible.builtin.command:
+        cmd: "nixos-rebuild build-image --image-variant proxmox --flake .#template2"
+        chdir: "{{ playbook_dir }}/.."
+      register: build_result
+      changed_when: true
+
+    - name: Find built image file
+      ansible.builtin.find:
+        paths: "{{ playbook_dir}}/../result"
+        patterns: "*.vma.zst"
+        recurse: true
+      register: image_files
+
+    - name: Fail if no image found
+      ansible.builtin.fail:
+        msg: "No QCOW2 image found in build output"
+      when: image_files.matched == 0
+
+    - name: Set image path
+      ansible.builtin.set_fact:
+        image_path: "{{ image_files.files[0].path }}"
+
+    - name: Extract image filename
+      ansible.builtin.set_fact:
+        image_filename: "{{ image_path | basename }}"
+
+    - name: Display image info
+      ansible.builtin.debug:
+        msg: "Built image: {{ image_path }} ({{ image_filename }})"
+
+- name: Deploy template to Proxmox
+  hosts: proxmox
+  gather_facts: false
+
+  vars:
+    template_name: "template2"
+    template_vmid: 9000
+    storage: "local-zfs"
+
+  tasks:
+    - name: Get image path and filename from localhost
+      ansible.builtin.set_fact:
+        image_path: "{{ hostvars['localhost']['image_path'] }}"
+        image_filename: "{{ hostvars['localhost']['image_filename'] }}"
+
+    - name: Set destination path
+      ansible.builtin.set_fact:
+        image_dest: "/var/lib/vz/dump/{{ image_filename }}"
+
+    - name: Copy image to Proxmox
+      ansible.builtin.copy:
+        src: "{{ image_path }}"
+        dest: "{{ image_dest }}"
+        mode: '0644'
+
+    - name: Check if template VM already exists
+      ansible.builtin.command:
+        cmd: "qm status {{ template_vmid }}"
+      register: vm_status
+      failed_when: false
+      changed_when: false
+
+    - name: Destroy existing template VM if it exists
+      ansible.builtin.command:
+        cmd: "qm destroy {{ template_vmid }} --purge"
+      when: vm_status.rc == 0
+      changed_when: true
+
+    - name: Import image
+      ansible.builtin.command:
+        cmd: "qmrestore {{ image_dest }} {{ template_vmid }}"
+      changed_when: true
+
+    - name: Convert VM to template
+      ansible.builtin.command:
+        cmd: "qm template {{ template_vmid }}"
+      changed_when: true
+
+    - name: Clean up uploaded image
+      ansible.builtin.file:
+        path: "{{ image_dest }}"
+        state: absent
+
+    - name: Display success message
+      ansible.builtin.debug:
+        msg: "Template VM {{ template_vmid }} created successfully on {{ storage }}"

playbooks/inventory.ini

@@ -0,0 +1,5 @@
+[proxmox]
+pve1.home.2rjus.net
+
+[proxmox:vars]
+ansible_user=root

playbooks/provision-approle.yml

@@ -0,0 +1,78 @@
+---
+# Provision OpenBao AppRole credentials to an existing host
+# Usage: nix develop -c ansible-playbook playbooks/provision-approle.yml -e hostname=ha1
+# Requires: BAO_ADDR and BAO_TOKEN environment variables set
+
+- name: Fetch AppRole credentials from OpenBao
+  hosts: localhost
+  connection: local
+  gather_facts: false
+
+  vars:
+    vault_addr: "{{ lookup('env', 'BAO_ADDR') | default('https://vault01.home.2rjus.net:8200', true) }}"
+    domain: "home.2rjus.net"
+
+  tasks:
+    - name: Validate hostname is provided
+      ansible.builtin.fail:
+        msg: "hostname variable is required. Use: -e hostname=<name>"
+      when: hostname is not defined
+
+    - name: Get role-id for host
+      ansible.builtin.command:
+        cmd: "bao read -field=role_id auth/approle/role/{{ hostname }}/role-id"
+      environment:
+        BAO_ADDR: "{{ vault_addr }}"
+        BAO_SKIP_VERIFY: "1"
+      register: role_id_result
+      changed_when: false
+
+    - name: Generate secret-id for host
+      ansible.builtin.command:
+        cmd: "bao write -field=secret_id -f auth/approle/role/{{ hostname }}/secret-id"
+      environment:
+        BAO_ADDR: "{{ vault_addr }}"
+        BAO_SKIP_VERIFY: "1"
+      register: secret_id_result
+      changed_when: true
+
+    - name: Add target host to inventory
+      ansible.builtin.add_host:
+        name: "{{ hostname }}.{{ domain }}"
+        groups: vault_target
+        ansible_user: root
+        vault_role_id: "{{ role_id_result.stdout }}"
+        vault_secret_id: "{{ secret_id_result.stdout }}"
+
+- name: Deploy AppRole credentials to host
+  hosts: vault_target
+  gather_facts: false
+
+  tasks:
+    - name: Create AppRole directory
+      ansible.builtin.file:
+        path: /var/lib/vault/approle
+        state: directory
+        mode: "0700"
+        owner: root
+        group: root
+
+    - name: Write role-id
+      ansible.builtin.copy:
+        content: "{{ vault_role_id }}"
+        dest: /var/lib/vault/approle/role-id
+        mode: "0600"
+        owner: root
+        group: root
+
+    - name: Write secret-id
+      ansible.builtin.copy:
+        content: "{{ vault_secret_id }}"
+        dest: /var/lib/vault/approle/secret-id
+        mode: "0600"
+        owner: root
+        group: root
+
+    - name: Display success
+      ansible.builtin.debug:
+        msg: "AppRole credentials provisioned to {{ inventory_hostname }}"

playbooks/run-upgrade.yml

@@ -0,0 +1,9 @@
+---
+- name: Trigger nixos-upgrade job on all hosts
+  hosts: all
+  remote_user: root
+
+  tasks:
+    - ansible.builtin.systemd_service:
+        name: nixos-upgrade.service
+        state: started

rebuild-all.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# array of hosts
+HOSTS=(
+    "ns1"
+    "ns2"
+    "ca"
+    "ha1"
+    "http-proxy"
+    "jelly01"
+    "monitoring01"
+    "nix-cache01"
+    "pgdb1"
+)
+
+for host in "${HOSTS[@]}"; do
+    echo "Rebuilding $host"
+    nixos-rebuild boot --flake .#${host} --target-host root@${host}
+done

scripts/create-host/MANIFEST.in

@@ -0,0 +1 @@
+recursive-include templates *.j2

scripts/create-host/README.md

@@ -0,0 +1,268 @@
+# NixOS Host Configuration Generator
+
+Automated tool for generating NixOS host configurations, flake.nix entries, and Terraform VM definitions for homelab infrastructure.
+
+## Installation
+
+The tool is available in the Nix development shell:
+
+```bash
+nix develop
+```
+
+## Usage
+
+### Basic Usage
+
+Create a new host with DHCP networking:
+
+```bash
+python -m scripts.create_host.create_host create --hostname test01
+```
+
+Create a new host with static IP:
+
+```bash
+python -m scripts.create_host.create_host create \
+  --hostname test01 \
+  --ip 10.69.13.50/24
+```
+
+Create a host with custom resources:
+
+```bash
+python -m scripts.create_host.create_host create \
+  --hostname bighost01 \
+  --ip 10.69.13.51/24 \
+  --cpu 8 \
+  --memory 8192 \
+  --disk 100G
+```
+
+### Dry Run Mode
+
+Preview what would be created without making changes:
+
+```bash
+python -m scripts.create_host.create_host create \
+  --hostname test01 \
+  --ip 10.69.13.50/24 \
+  --dry-run
+```
+
+### Force Mode (Regenerate Existing Configuration)
+
+Overwrite an existing host configuration (useful for testing):
+
+```bash
+python -m scripts.create_host.create_host create \
+  --hostname test01 \
+  --ip 10.69.13.50/24 \
+  --force
+```
+
+This mode:
+- Skips hostname and IP uniqueness validation
+- Overwrites files in `hosts/<hostname>/`
+- Updates existing entries in `flake.nix` and `terraform/vms.tf` (doesn't duplicate)
+- Useful for iterating on configuration templates during testing
+
+### Options
+
+- `--hostname` (required): Hostname for the new host
+  - Must be lowercase alphanumeric with hyphens
+  - Must be unique (not already exist in repository)
+
+- `--ip` (optional): Static IP address with CIDR notation
+  - Format: `10.69.13.X/24`
+  - Must be in `10.69.13.0/24` subnet
+  - Last octet must be 1-254
+  - Omit this option for DHCP configuration
+
+- `--cpu` (optional, default: 2): Number of CPU cores
+  - Must be at least 1
+
+- `--memory` (optional, default: 2048): Memory in MB
+  - Must be at least 512
+
+- `--disk` (optional, default: "20G"): Disk size
+  - Examples: "20G", "50G", "100G"
+
+- `--dry-run` (flag): Preview changes without creating files
+
+- `--force` (flag): Overwrite existing host configuration
+  - Skips uniqueness validation
+  - Updates existing entries instead of creating duplicates
+
+## What It Does
+
+The tool performs the following actions:
+
+1. **Validates** the configuration:
+   - Hostname format (RFC 1123 compliance)
+   - Hostname uniqueness
+   - IP address format and subnet (if provided)
+   - IP address uniqueness (if provided)
+
+2. **Generates** host configuration files:
+   - `hosts/<hostname>/default.nix` - Import wrapper
+   - `hosts/<hostname>/configuration.nix` - Full host configuration
+
+3. **Updates** repository files:
+   - `flake.nix` - Adds new nixosConfigurations entry
+   - `terraform/vms.tf` - Adds new VM definition
+
+4. **Displays** next steps for:
+   - Reviewing changes with git diff
+   - Verifying NixOS configuration
+   - Verifying Terraform configuration
+   - Committing changes
+   - Deploying the VM
+
+## Generated Configuration
+
+### Host Features
+
+All generated hosts include:
+
+- Full system imports from `../../system`:
+  - Nix binary cache integration
+  - SSH with root login
+  - SOPS secrets management
+  - Internal ACME CA integration
+  - Daily auto-upgrades with auto-reboot
+  - Prometheus node-exporter
+  - Promtail logging to monitoring01
+
+- VM guest agent from `../../common/vm`
+- Hardware configuration from `../template/hardware-configuration.nix`
+
+### Networking
+
+**Static IP mode** (when `--ip` is provided):
+```nix
+systemd.network.networks."ens18" = {
+  matchConfig.Name = "ens18";
+  address = [ "10.69.13.50/24" ];
+  routes = [ { Gateway = "10.69.13.1"; } ];
+  linkConfig.RequiredForOnline = "routable";
+};
+```
+
+**DHCP mode** (when `--ip` is omitted):
+```nix
+systemd.network.networks."ens18" = {
+  matchConfig.Name = "ens18";
+  networkConfig.DHCP = "ipv4";
+  linkConfig.RequiredForOnline = "routable";
+};
+```
+
+### DNS Configuration
+
+All hosts are configured with:
+- DNS servers: `10.69.13.5`, `10.69.13.6` (ns1, ns2)
+- Domain: `home.2rjus.net`
+
+## Examples
+
+### Create a test VM with defaults
+
+```bash
+python -m scripts.create_host.create_host create --hostname test99
+```
+
+This creates a DHCP VM with 2 CPU cores, 2048 MB memory, and 20G disk.
+
+### Create a database server with static IP
+
+```bash
+python -m scripts.create_host.create_host create \
+  --hostname pgdb2 \
+  --ip 10.69.13.52/24 \
+  --cpu 4 \
+  --memory 4096 \
+  --disk 50G
+```
+
+### Preview changes before creating
+
+```bash
+python -m scripts.create_host.create_host create \
+  --hostname test99 \
+  --ip 10.69.13.99/24 \
+  --dry-run
+```
+
+## Error Handling
+
+The tool validates input and provides clear error messages for:
+
+- Invalid hostname format (must be lowercase alphanumeric with hyphens)
+- Duplicate hostname (already exists in repository)
+- Invalid IP format (must be X.X.X.X/24)
+- Wrong subnet (must be 10.69.13.0/24)
+- Invalid last octet (must be 1-254)
+- Duplicate IP address (already in use)
+- Resource constraints (CPU < 1, memory < 512 MB)
+
+## Integration with Deployment Pipeline
+
+This tool implements **Phase 2** of the automated deployment pipeline:
+
+1. **Phase 1**: Template building ✓ (build-and-deploy-template.yml)
+2. **Phase 2**: Host configuration generation ✓ (this tool)
+3. **Phase 3**: Bootstrap automation (planned)
+4. **Phase 4**: Secrets management (planned)
+5. **Phase 5**: DNS automation (planned)
+6. **Phase 6**: Full integration (planned)
+
+## Development
+
+### Project Structure
+
+```
+scripts/create-host/
+├── create_host.py          # Main CLI entry point (typer app)
+├── __init__.py            # Package initialization
+├── validators.py          # Validation logic
+├── generators.py          # File generation using Jinja2
+├── manipulators.py        # Text manipulation for flake.nix and vms.tf
+├── models.py              # Data models (HostConfig)
+├── templates/
+│   ├── default.nix.j2     # Template for default.nix
+│   └── configuration.nix.j2  # Template for configuration.nix
+└── README.md              # This file
+```
+
+### Testing
+
+Run the test cases from the implementation plan:
+
+```bash
+# Test 1: DHCP host with defaults
+python -m scripts.create_host.create_host create --hostname testdhcp --dry-run
+
+# Test 2: Static IP host
+python -m scripts.create_host.create_host create \
+  --hostname test50 --ip 10.69.13.50/24 --dry-run
+
+# Test 3: Custom resources
+python -m scripts.create_host.create_host create \
+  --hostname test51 --ip 10.69.13.51/24 \
+  --cpu 8 --memory 8192 --disk 100G --dry-run
+
+# Test 4: Duplicate hostname (should error)
+python -m scripts.create_host.create_host create --hostname ns1 --dry-run
+
+# Test 5: Invalid subnet (should error)
+python -m scripts.create_host.create_host create \
+  --hostname testbad --ip 192.168.1.50/24 --dry-run
+
+# Test 6: Invalid hostname (should error)
+python -m scripts.create_host.create_host create --hostname Test_Host --dry-run
+```
+
+## License
+
+Part of the nixos-servers homelab infrastructure repository.

scripts/create-host/__init__.py

@@ -0,0 +1,3 @@
+"""NixOS host configuration generator for homelab infrastructure."""
+
+__version__ = "0.1.0"

scripts/create-host/__main__.py

@@ -0,0 +1,6 @@
+"""Entry point for running the create-host module."""
+
+from .create_host import app
+
+if __name__ == "__main__":
+    app()

scripts/create-host/create_host.py

@@ -0,0 +1,452 @@
+"""CLI tool for generating NixOS host configurations."""
+
+import shutil
+import sys
+from pathlib import Path
+from typing import Optional
+
+import typer
+from rich.console import Console
+from rich.panel import Panel
+from rich.table import Table
+
+from generators import generate_host_files, generate_vault_terraform
+from manipulators import (
+    update_flake_nix,
+    update_terraform_vms,
+    add_wrapped_token_to_vm,
+    remove_from_flake_nix,
+    remove_from_terraform_vms,
+    remove_from_vault_terraform,
+    check_entries_exist,
+)
+from models import HostConfig
+from vault_helper import generate_wrapped_token
+from validators import (
+    validate_hostname_format,
+    validate_hostname_unique,
+    validate_ip_subnet,
+    validate_ip_unique,
+)
+
+app = typer.Typer(
+    name="create-host",
+    help="Generate NixOS host configurations for homelab infrastructure",
+    add_completion=False,
+)
+console = Console()
+
+
+def get_repo_root() -> Path:
+    """Get the repository root directory."""
+    # Use current working directory as repo root
+    # The tool should be run from the repository root
+    return Path.cwd()
+
+
+@app.callback(invoke_without_command=True)
+def main(
+    ctx: typer.Context,
+    hostname: Optional[str] = typer.Option(None, "--hostname", help="Hostname for the new host"),
+    ip: Optional[str] = typer.Option(
+        None, "--ip", help="Static IP address with CIDR (e.g., 10.69.13.50/24). Omit for DHCP."
+    ),
+    cpu: int = typer.Option(2, "--cpu", help="Number of CPU cores"),
+    memory: int = typer.Option(2048, "--memory", help="Memory in MB"),
+    disk: str = typer.Option("20G", "--disk", help="Disk size (e.g., 20G, 50G, 100G)"),
+    dry_run: bool = typer.Option(False, "--dry-run", help="Preview changes without creating files"),
+    force: bool = typer.Option(False, "--force", help="Overwrite existing host configuration / skip confirmation for removal"),
+    skip_vault: bool = typer.Option(False, "--skip-vault", help="Skip Vault configuration and token generation"),
+    regenerate_token: bool = typer.Option(False, "--regenerate-token", help="Only regenerate Vault wrapped token (no other changes)"),
+    remove: bool = typer.Option(False, "--remove", help="Remove host configuration and terraform entries"),
+) -> None:
+    """
+    Create a new NixOS host configuration.
+
+    Generates host configuration files, updates flake.nix, and adds Terraform VM definition.
+    """
+    # Show help if no hostname provided
+    if hostname is None:
+        console.print("[bold red]Error:[/bold red] --hostname is required\n")
+        ctx.get_help()
+        sys.exit(1)
+
+    # Get repository root
+    repo_root = get_repo_root()
+
+    # Handle removal mode
+    if remove:
+        handle_remove(hostname, repo_root, dry_run, force, ip, cpu, memory, disk, skip_vault, regenerate_token)
+        return
+
+    # Handle token regeneration mode
+    if regenerate_token:
+        # Validate that incompatible options aren't used
+        if force or dry_run or skip_vault:
+            console.print("[bold red]Error:[/bold red] --regenerate-token cannot be used with --force, --dry-run, or --skip-vault\n")
+            sys.exit(1)
+        if ip or cpu != 2 or memory != 2048 or disk != "20G":
+            console.print("[bold red]Error:[/bold red] --regenerate-token only regenerates the token. Other options (--ip, --cpu, --memory, --disk) are ignored.\n")
+            console.print("[yellow]Tip:[/yellow] Use without those options, or use --force to update the entire configuration.\n")
+            sys.exit(1)
+
+        try:
+            console.print(f"\n[bold blue]Regenerating Vault token for {hostname}...[/bold blue]")
+
+            # Validate hostname exists
+            host_dir = repo_root / "hosts" / hostname
+            if not host_dir.exists():
+                console.print(f"[bold red]Error:[/bold red] Host {hostname} does not exist")
+                console.print(f"Host directory not found: {host_dir}")
+                sys.exit(1)
+
+            # Generate new wrapped token
+            wrapped_token = generate_wrapped_token(hostname, repo_root)
+
+            # Update only the wrapped token in vms.tf
+            add_wrapped_token_to_vm(hostname, wrapped_token, repo_root)
+            console.print("[green]✓[/green] Regenerated and updated wrapped token in terraform/vms.tf")
+
+            console.print("\n[bold green]✓ Token regenerated successfully![/bold green]")
+            console.print(f"\n[yellow]⚠️[/yellow]  Token expires in 24 hours")
+            console.print(f"[yellow]⚠️[/yellow]  Deploy the VM within 24h or regenerate token again\n")
+
+            console.print("[bold cyan]Next steps:[/bold cyan]")
+            console.print(f"  cd terraform && tofu apply")
+            console.print(f"  # Then redeploy VM to pick up new token\n")
+
+            return
+
+        except Exception as e:
+            console.print(f"\n[bold red]Error regenerating token:[/bold red] {e}\n")
+            sys.exit(1)
+
+    try:
+        # Build configuration
+        config = HostConfig(
+            hostname=hostname,
+            ip=ip,
+            cpu=cpu,
+            memory=memory,
+            disk=disk,
+        )
+
+        # Validate configuration
+        console.print("\n[bold blue]Validating configuration...[/bold blue]")
+
+        config.validate()
+        validate_hostname_format(hostname)
+
+        # Skip uniqueness checks in force mode
+        if not force:
+            validate_hostname_unique(hostname, repo_root)
+            if ip:
+                validate_ip_unique(ip, repo_root)
+        else:
+            # Check if we're actually overwriting something
+            host_dir = repo_root / "hosts" / hostname
+            if host_dir.exists():
+                console.print(f"[yellow]⚠[/yellow]  Updating existing host configuration for {hostname}")
+
+        if ip:
+            validate_ip_subnet(ip)
+
+        console.print("[green]✓[/green] All validations passed\n")
+
+        # Display configuration summary
+        display_config_summary(config)
+
+        # Dry run mode - exit before making changes
+        if dry_run:
+            console.print("\n[yellow]DRY RUN MODE - No files will be created[/yellow]\n")
+            display_dry_run_summary(config, repo_root)
+            return
+
+        # Generate files
+        console.print("\n[bold blue]Generating host configuration...[/bold blue]")
+
+        generate_host_files(config, repo_root)
+        action = "Updated" if force else "Created"
+        console.print(f"[green]✓[/green] {action} hosts/{hostname}/default.nix")
+        console.print(f"[green]✓[/green] {action} hosts/{hostname}/configuration.nix")
+
+        update_flake_nix(config, repo_root, force=force)
+        console.print("[green]✓[/green] Updated flake.nix")
+
+        update_terraform_vms(config, repo_root, force=force)
+        console.print("[green]✓[/green] Updated terraform/vms.tf")
+
+        # Generate Vault configuration if not skipped
+        if not skip_vault:
+            console.print("\n[bold blue]Configuring Vault integration...[/bold blue]")
+
+            try:
+                # Generate Vault Terraform configuration
+                generate_vault_terraform(hostname, repo_root)
+                console.print("[green]✓[/green] Updated terraform/vault/hosts-generated.tf")
+
+                # Generate wrapped token
+                wrapped_token = generate_wrapped_token(hostname, repo_root)
+
+                # Add wrapped token to VM configuration
+                add_wrapped_token_to_vm(hostname, wrapped_token, repo_root)
+                console.print("[green]✓[/green] Added wrapped token to terraform/vms.tf")
+
+            except Exception as e:
+                console.print(f"\n[yellow]⚠️  Vault configuration failed: {e}[/yellow]")
+                console.print("[yellow]Host configuration created without Vault integration[/yellow]")
+                console.print("[yellow]You can add Vault support later by re-running with --force[/yellow]\n")
+        else:
+            console.print("\n[yellow]Skipped Vault configuration (--skip-vault)[/yellow]")
+
+        # Success message
+        console.print("\n[bold green]✓ Host configuration generated successfully![/bold green]\n")
+
+        # Display next steps
+        display_next_steps(hostname, skip_vault=skip_vault)
+
+    except ValueError as e:
+        console.print(f"\n[bold red]Error:[/bold red] {e}\n", style="red")
+        sys.exit(1)
+    except Exception as e:
+        console.print(f"\n[bold red]Unexpected error:[/bold red] {e}\n", style="red")
+        sys.exit(1)
+
+
+def handle_remove(
+    hostname: str,
+    repo_root: Path,
+    dry_run: bool,
+    force: bool,
+    ip: Optional[str],
+    cpu: int,
+    memory: int,
+    disk: str,
+    skip_vault: bool,
+    regenerate_token: bool,
+) -> None:
+    """Handle the --remove workflow."""
+    # Validate --remove isn't used with create options
+    incompatible_options = []
+    if ip:
+        incompatible_options.append("--ip")
+    if cpu != 2:
+        incompatible_options.append("--cpu")
+    if memory != 2048:
+        incompatible_options.append("--memory")
+    if disk != "20G":
+        incompatible_options.append("--disk")
+    if skip_vault:
+        incompatible_options.append("--skip-vault")
+    if regenerate_token:
+        incompatible_options.append("--regenerate-token")
+
+    if incompatible_options:
+        console.print(
+            f"[bold red]Error:[/bold red] --remove cannot be used with: {', '.join(incompatible_options)}\n"
+        )
+        sys.exit(1)
+
+    # Validate hostname exists (host directory must exist)
+    host_dir = repo_root / "hosts" / hostname
+    if not host_dir.exists():
+        console.print(f"[bold red]Error:[/bold red] Host {hostname} does not exist")
+        console.print(f"Host directory not found: {host_dir}")
+        sys.exit(1)
+
+    # Check what entries exist
+    flake_exists, terraform_exists, vault_exists = check_entries_exist(hostname, repo_root)
+
+    # Collect all files in the host directory recursively
+    files_in_host_dir = sorted([f for f in host_dir.rglob("*") if f.is_file()])
+
+    # Check for secrets directory
+    secrets_dir = repo_root / "secrets" / hostname
+    secrets_exist = secrets_dir.exists()
+
+    # Display summary
+    if dry_run:
+        console.print("\n[yellow][DRY RUN - No changes will be made][/yellow]\n")
+
+    console.print(f"\n[bold blue]Removing host: {hostname}[/bold blue]\n")
+
+    # Show host directory contents
+    console.print("[bold]Directory to be deleted (and all contents):[/bold]")
+    console.print(f"  • hosts/{hostname}/")
+    for f in files_in_host_dir:
+        rel_path = f.relative_to(host_dir)
+        console.print(f"    - {rel_path}")
+
+    # Show entries to be removed
+    console.print("\n[bold]Entries to be removed:[/bold]")
+    if flake_exists:
+        console.print(f"  • flake.nix (nixosConfigurations.{hostname})")
+    else:
+        console.print(f"  • flake.nix [dim](not found)[/dim]")
+
+    if terraform_exists:
+        console.print(f'  • terraform/vms.tf (locals.vms["{hostname}"])')
+    else:
+        console.print(f"  • terraform/vms.tf [dim](not found)[/dim]")
+
+    if vault_exists:
+        console.print(f'  • terraform/vault/hosts-generated.tf (generated_host_policies["{hostname}"])')
+    else:
+        console.print(f"  • terraform/vault/hosts-generated.tf [dim](not found)[/dim]")
+
+    # Warn about secrets directory
+    if secrets_exist:
+        console.print(f"\n[yellow]⚠️  Warning: secrets/{hostname}/ directory exists and will NOT be deleted[/yellow]")
+        console.print(f"   Manually remove if no longer needed: [white]rm -rf secrets/{hostname}/[/white]")
+        console.print(f"   Also update .sops.yaml to remove the host's age key")
+
+    # Exit if dry run
+    if dry_run:
+        console.print("\n[yellow][DRY RUN - No changes made][/yellow]\n")
+        return
+
+    # Prompt for confirmation unless --force
+    if not force:
+        console.print("")
+        confirm = typer.confirm("Proceed with removal?", default=False)
+        if not confirm:
+            console.print("\n[yellow]Removal cancelled[/yellow]\n")
+            sys.exit(0)
+
+    # Perform removal
+    console.print("\n[bold blue]Removing host configuration...[/bold blue]")
+
+    # Remove from terraform/vault/hosts-generated.tf
+    if vault_exists:
+        if remove_from_vault_terraform(hostname, repo_root):
+            console.print("[green]✓[/green] Removed from terraform/vault/hosts-generated.tf")
+        else:
+            console.print("[yellow]⚠[/yellow]  Could not remove from terraform/vault/hosts-generated.tf")
+
+    # Remove from terraform/vms.tf
+    if terraform_exists:
+        if remove_from_terraform_vms(hostname, repo_root):
+            console.print("[green]✓[/green] Removed from terraform/vms.tf")
+        else:
+            console.print("[yellow]⚠[/yellow]  Could not remove from terraform/vms.tf")
+
+    # Remove from flake.nix
+    if flake_exists:
+        if remove_from_flake_nix(hostname, repo_root):
+            console.print("[green]✓[/green] Removed from flake.nix")
+        else:
+            console.print("[yellow]⚠[/yellow]  Could not remove from flake.nix")
+
+    # Delete hosts/<hostname>/ directory
+    shutil.rmtree(host_dir)
+    console.print(f"[green]✓[/green] Deleted hosts/{hostname}/")
+
+    # Success message
+    console.print(f"\n[bold green]✓ Host {hostname} removed successfully![/bold green]\n")
+
+    # Display next steps
+    display_removal_next_steps(hostname, vault_exists)
+
+
+def display_removal_next_steps(hostname: str, had_vault: bool) -> None:
+    """Display next steps after successful removal."""
+    vault_file = " terraform/vault/hosts-generated.tf" if had_vault else ""
+    vault_apply = ""
+    if had_vault:
+        vault_apply = f"""
+3. Apply Vault changes:
+   [white]cd terraform/vault && tofu apply[/white]
+"""
+
+    next_steps = f"""[bold cyan]Next Steps:[/bold cyan]
+
+1. Review changes:
+   [white]git diff[/white]
+
+2. If VM exists in Proxmox, destroy it first:
+   [white]cd terraform && tofu destroy -target='proxmox_vm_qemu.vm["{hostname}"]'[/white]
+{vault_apply}
+4. Commit changes:
+   [white]git add -u hosts/{hostname} flake.nix terraform/vms.tf{vault_file}
+   git commit -m "hosts: remove {hostname}"[/white]
+"""
+    console.print(Panel(next_steps, border_style="cyan"))
+
+
+def display_config_summary(config: HostConfig) -> None:
+    """Display configuration summary table."""
+    table = Table(title="Host Configuration", show_header=False)
+    table.add_column("Property", style="cyan")
+    table.add_column("Value", style="white")
+
+    table.add_row("Hostname", config.hostname)
+    table.add_row("Domain", config.domain)
+    table.add_row("Network Mode", "Static IP" if config.is_static_ip else "DHCP")
+
+    if config.is_static_ip:
+        table.add_row("IP Address", config.ip)
+        table.add_row("Gateway", config.gateway)
+
+    table.add_row("DNS Servers", ", ".join(config.nameservers))
+    table.add_row("CPU Cores", str(config.cpu))
+    table.add_row("Memory", f"{config.memory} MB")
+    table.add_row("Disk Size", config.disk)
+    table.add_row("State Version", config.state_version)
+
+    console.print(table)
+
+
+def display_dry_run_summary(config: HostConfig, repo_root: Path) -> None:
+    """Display what would be created in dry run mode."""
+    console.print("[bold]Files that would be created:[/bold]")
+    console.print(f"  • {repo_root}/hosts/{config.hostname}/default.nix")
+    console.print(f"  • {repo_root}/hosts/{config.hostname}/configuration.nix")
+
+    console.print("\n[bold]Files that would be modified:[/bold]")
+    console.print(f"  • {repo_root}/flake.nix (add nixosConfigurations.{config.hostname})")
+    console.print(f"  • {repo_root}/terraform/vms.tf (add VM definition)")
+
+
+def display_next_steps(hostname: str, skip_vault: bool = False) -> None:
+    """Display next steps after successful generation."""
+    vault_files = "" if skip_vault else " terraform/vault/hosts-generated.tf"
+    vault_apply = ""
+
+    if not skip_vault:
+        vault_apply = """
+4a. Apply Vault configuration:
+   [white]cd terraform/vault
+   tofu apply[/white]
+"""
+
+    next_steps = f"""[bold cyan]Next Steps:[/bold cyan]
+
+1. Review changes:
+   [white]git diff[/white]
+
+2. Verify NixOS configuration:
+   [white]nix flake check
+   nix build .#nixosConfigurations.{hostname}.config.system.build.toplevel[/white]
+
+3. Verify Terraform configuration:
+   [white]cd terraform
+   tofu validate
+   tofu plan[/white]
+
+4. Commit changes:
+   [white]git add hosts/{hostname} flake.nix terraform/vms.tf{vault_files}
+   git commit -m "hosts: add {hostname} configuration"[/white]
+{vault_apply}
+5. Deploy VM (after merging to master or within 24h of token generation):
+   [white]cd terraform
+   tofu apply[/white]
+
+6. Host will bootstrap automatically on first boot
+   - Wrapped token expires in 24 hours
+   - If expired, re-run: create-host --hostname {hostname} --force
+"""
+    console.print(Panel(next_steps, border_style="cyan"))
+
+
+if __name__ == "__main__":
+    app()

scripts/create-host/default.nix

@@ -0,0 +1,39 @@
+{ lib
+, python3
+, python3Packages
+}:
+
+python3Packages.buildPythonApplication {
+  pname = "create-host";
+  version = "0.1.0";
+
+  src = ./.;
+
+  pyproject = true;
+
+  build-system = with python3Packages; [
+    setuptools
+  ];
+
+  propagatedBuildInputs = with python3Packages; [
+    typer
+    jinja2
+    rich
+    hvac  # Python Vault/OpenBao client library
+  ];
+
+  # Install templates to share directory
+  postInstall = ''
+    mkdir -p $out/share/create-host
+    cp -r templates $out/share/create-host/
+  '';
+
+  # No tests yet
+  doCheck = false;
+
+  meta = with lib; {
+    description = "NixOS host configuration generator for homelab infrastructure";
+    license = licenses.mit;
+    maintainers = [ ];
+  };
+}

scripts/create-host/generators.py

@@ -0,0 +1,199 @@
+"""File generation using Jinja2 templates."""
+
+import sys
+from pathlib import Path
+
+from jinja2 import Environment, BaseLoader, TemplateNotFound
+
+from models import HostConfig
+
+
+class PackageTemplateLoader(BaseLoader):
+    """Custom Jinja2 loader that works with both dev and installed packages."""
+
+    def __init__(self):
+        # Try to find templates in multiple locations
+        self.template_dirs = []
+
+        # Location 1: Development (scripts/create-host/templates)
+        dev_dir = Path(__file__).parent / "templates"
+        if dev_dir.exists():
+            self.template_dirs.append(dev_dir)
+
+        # Location 2: Installed via Nix (../share/create-host/templates from bin dir)
+        # When installed via Nix, __file__ is in lib/python3.X/site-packages/
+        # and templates are in ../../../share/create-host/templates
+        for site_path in sys.path:
+            site_dir = Path(site_path)
+            # Try to find the Nix store path
+            if "site-packages" in str(site_dir):
+                # Go up to the package root (e.g., /nix/store/xxx-create-host-0.1.0)
+                pkg_root = site_dir.parent.parent.parent
+                share_templates = pkg_root / "share" / "create-host" / "templates"
+                if share_templates.exists():
+                    self.template_dirs.append(share_templates)
+
+        # Location 3: Fallback - sys.path templates
+        for site_path in sys.path:
+            site_templates = Path(site_path) / "templates"
+            if site_templates.exists():
+                self.template_dirs.append(site_templates)
+
+    def get_source(self, environment, template):
+        for template_dir in self.template_dirs:
+            template_path = template_dir / template
+            if template_path.exists():
+                mtime = template_path.stat().st_mtime
+                source = template_path.read_text()
+                return source, str(template_path), lambda: mtime == template_path.stat().st_mtime
+
+        raise TemplateNotFound(template)
+
+
+def generate_host_files(config: HostConfig, repo_root: Path) -> None:
+    """
+    Generate host configuration files from templates.
+
+    Args:
+        config: Host configuration
+        repo_root: Path to repository root
+    """
+    # Setup Jinja2 environment with custom loader
+    env = Environment(
+        loader=PackageTemplateLoader(),
+        trim_blocks=True,
+        lstrip_blocks=True,
+    )
+
+    # Create host directory
+    host_dir = repo_root / "hosts" / config.hostname
+    host_dir.mkdir(parents=True, exist_ok=True)
+
+    # Generate default.nix
+    default_template = env.get_template("default.nix.j2")
+    default_content = default_template.render(hostname=config.hostname)
+    (host_dir / "default.nix").write_text(default_content)
+
+    # Generate configuration.nix
+    config_template = env.get_template("configuration.nix.j2")
+    config_content = config_template.render(
+        hostname=config.hostname,
+        domain=config.domain,
+        nameservers=config.nameservers,
+        is_static_ip=config.is_static_ip,
+        ip=config.ip,
+        gateway=config.gateway,
+        state_version=config.state_version,
+    )
+    (host_dir / "configuration.nix").write_text(config_content)
+
+
+def generate_vault_terraform(hostname: str, repo_root: Path) -> None:
+    """
+    Generate or update Vault Terraform configuration for a new host.
+
+    Creates/updates terraform/vault/hosts-generated.tf with:
+    - Host policy granting access to hosts/<hostname>/* secrets
+    - AppRole configuration for the host
+    - Placeholder secret entry (user adds actual secrets separately)
+
+    Args:
+        hostname: Hostname for the new host
+        repo_root: Path to repository root
+    """
+    vault_tf_path = repo_root / "terraform" / "vault" / "hosts-generated.tf"
+
+    # Read existing file if it exists, otherwise start with empty structure
+    if vault_tf_path.exists():
+        content = vault_tf_path.read_text()
+    else:
+        # Create initial file structure
+        content = """# WARNING: Auto-generated by create-host tool
+# Manual edits will be overwritten when create-host is run
+
+# Generated host policies
+# Each host gets access to its own secrets under hosts/<hostname>/*
+locals {
+  generated_host_policies = {
+  }
+
+  # Placeholder secrets - user should add actual secrets manually or via tofu
+  generated_secrets = {
+  }
+}
+
+# Create policies for generated hosts
+resource "vault_policy" "generated_host_policies" {
+  for_each = local.generated_host_policies
+
+  name = "host-\${each.key}"
+
+  policy = <<-EOT
+    # Allow host to read its own secrets
+    %{for path in each.value.paths~}
+    path "${path}" {
+      capabilities = ["read", "list"]
+    }
+    %{endfor~}
+  EOT
+}
+
+# Create AppRoles for generated hosts
+resource "vault_approle_auth_backend_role" "generated_hosts" {
+  for_each = local.generated_host_policies
+
+  backend            = vault_auth_backend.approle.path
+  role_name          = each.key
+  token_policies     = ["host-\${each.key}"]
+  secret_id_ttl      = 0  # Never expire (wrapped tokens provide time limit)
+  token_ttl          = 3600
+  token_max_ttl      = 3600
+  secret_id_num_uses = 0  # Unlimited uses
+}
+"""
+
+    # Parse existing policies from the file
+    import re
+
+    policies_match = re.search(
+        r'generated_host_policies = \{(.*?)\n  \}',
+        content,
+        re.DOTALL
+    )
+
+    if policies_match:
+        policies_content = policies_match.group(1)
+    else:
+        policies_content = ""
+
+    # Check if hostname already exists
+    if f'"{hostname}"' in policies_content:
+        # Already exists, don't duplicate
+        return
+
+    # Add new policy entry
+    new_policy = f'''
+    "{hostname}" = {{
+      paths = [
+        "secret/data/hosts/{hostname}/*",
+      ]
+    }}'''
+
+    # Insert before the closing brace
+    if policies_content.strip():
+        # There are existing entries, add after them
+        new_policies_content = policies_content.rstrip() + new_policy + "\n  "
+    else:
+        # First entry
+        new_policies_content = new_policy + "\n  "
+
+    # Replace the policies map
+    new_content = re.sub(
+        r'(generated_host_policies = \{)(.*?)(\n  \})',
+        rf'\1{new_policies_content}\3',
+        content,
+        flags=re.DOTALL
+    )
+
+    # Write the updated file
+    vault_tf_path.write_text(new_content)

scripts/create-host/manipulators.py

@@ -0,0 +1,312 @@
+"""Text manipulation for flake.nix and Terraform files."""
+
+import re
+from pathlib import Path
+from typing import Tuple
+
+from models import HostConfig
+
+
+def remove_from_flake_nix(hostname: str, repo_root: Path) -> bool:
+    """
+    Remove host entry from flake.nix nixosConfigurations.
+
+    Args:
+        hostname: Hostname to remove
+        repo_root: Path to repository root
+
+    Returns:
+        True if found and removed, False if not found
+    """
+    flake_path = repo_root / "flake.nix"
+    content = flake_path.read_text()
+
+    # Check if hostname exists
+    hostname_pattern = rf"^      {re.escape(hostname)} = nixpkgs\.lib\.nixosSystem"
+    if not re.search(hostname_pattern, content, re.MULTILINE):
+        return False
+
+    # Match the entire block from "hostname = " to "};"
+    replace_pattern = rf"^      {re.escape(hostname)} = nixpkgs\.lib\.nixosSystem \{{.*?^      \}};\n"
+    new_content, count = re.subn(replace_pattern, "", content, flags=re.MULTILINE | re.DOTALL)
+
+    if count == 0:
+        return False
+
+    flake_path.write_text(new_content)
+    return True
+
+
+def remove_from_terraform_vms(hostname: str, repo_root: Path) -> bool:
+    """
+    Remove VM entry from terraform/vms.tf locals.vms map.
+
+    Args:
+        hostname: Hostname to remove
+        repo_root: Path to repository root
+
+    Returns:
+        True if found and removed, False if not found
+    """
+    terraform_path = repo_root / "terraform" / "vms.tf"
+    content = terraform_path.read_text()
+
+    # Check if hostname exists
+    hostname_pattern = rf'^\s+"{re.escape(hostname)}" = \{{'
+    if not re.search(hostname_pattern, content, re.MULTILINE):
+        return False
+
+    # Match the entire block from "hostname" = { to }
+    replace_pattern = rf'^\s+"{re.escape(hostname)}" = \{{.*?^\s+\}}\n'
+    new_content, count = re.subn(replace_pattern, "", content, flags=re.MULTILINE | re.DOTALL)
+
+    if count == 0:
+        return False
+
+    terraform_path.write_text(new_content)
+    return True
+
+
+def remove_from_vault_terraform(hostname: str, repo_root: Path) -> bool:
+    """
+    Remove host policy from terraform/vault/hosts-generated.tf.
+
+    Args:
+        hostname: Hostname to remove
+        repo_root: Path to repository root
+
+    Returns:
+        True if found and removed, False if not found
+    """
+    vault_tf_path = repo_root / "terraform" / "vault" / "hosts-generated.tf"
+
+    if not vault_tf_path.exists():
+        return False
+
+    content = vault_tf_path.read_text()
+
+    # Check if hostname exists in the policies
+    if f'"{hostname}"' not in content:
+        return False
+
+    # Match the host entry block within generated_host_policies
+    # Pattern matches:  "hostname" = { ... }  with possible trailing newlines
+    replace_pattern = rf'\s*"{re.escape(hostname)}" = \{{\s*paths = \[.*?\]\s*\}}\n?'
+    new_content, count = re.subn(replace_pattern, "", content, flags=re.DOTALL)
+
+    if count == 0:
+        return False
+
+    vault_tf_path.write_text(new_content)
+    return True
+
+
+def check_entries_exist(hostname: str, repo_root: Path) -> Tuple[bool, bool, bool]:
+    """
+    Check which entries exist for a hostname.
+
+    Args:
+        hostname: Hostname to check
+        repo_root: Path to repository root
+
+    Returns:
+        Tuple of (flake_exists, terraform_vms_exists, vault_exists)
+    """
+    # Check flake.nix
+    flake_path = repo_root / "flake.nix"
+    flake_content = flake_path.read_text()
+    flake_pattern = rf"^      {re.escape(hostname)} = nixpkgs\.lib\.nixosSystem"
+    flake_exists = bool(re.search(flake_pattern, flake_content, re.MULTILINE))
+
+    # Check terraform/vms.tf
+    terraform_path = repo_root / "terraform" / "vms.tf"
+    terraform_content = terraform_path.read_text()
+    terraform_pattern = rf'^\s+"{re.escape(hostname)}" = \{{'
+    terraform_exists = bool(re.search(terraform_pattern, terraform_content, re.MULTILINE))
+
+    # Check terraform/vault/hosts-generated.tf
+    vault_tf_path = repo_root / "terraform" / "vault" / "hosts-generated.tf"
+    vault_exists = False
+    if vault_tf_path.exists():
+        vault_content = vault_tf_path.read_text()
+        vault_exists = f'"{hostname}"' in vault_content
+
+    return (flake_exists, terraform_exists, vault_exists)
+
+
+def update_flake_nix(config: HostConfig, repo_root: Path, force: bool = False) -> None:
+    """
+    Add or update host entry in flake.nix nixosConfigurations.
+
+    Args:
+        config: Host configuration
+        repo_root: Path to repository root
+        force: If True, replace existing entry; if False, insert new entry
+    """
+    flake_path = repo_root / "flake.nix"
+    content = flake_path.read_text()
+
+    # Create new entry
+    new_entry = f"""      {config.hostname} = nixpkgs.lib.nixosSystem {{
+        inherit system;
+        specialArgs = {{
+          inherit inputs self sops-nix;
+        }};
+        modules = [
+          (
+            {{ config, pkgs, ... }}:
+            {{
+              nixpkgs.overlays = commonOverlays;
+            }}
+          )
+          ./hosts/{config.hostname}
+          sops-nix.nixosModules.sops
+        ];
+      }};
+"""
+
+    # Check if hostname already exists
+    hostname_pattern = rf"^      {re.escape(config.hostname)} = nixpkgs\.lib\.nixosSystem"
+    existing_match = re.search(hostname_pattern, content, re.MULTILINE)
+
+    if existing_match and force:
+        # Replace existing entry
+        # Match the entire block from "hostname = " to "};"
+        replace_pattern = rf"^      {re.escape(config.hostname)} = nixpkgs\.lib\.nixosSystem \{{.*?^      \}};\n"
+        new_content, count = re.subn(replace_pattern, new_entry, content, flags=re.MULTILINE | re.DOTALL)
+
+        if count == 0:
+            raise ValueError(f"Could not find existing entry for {config.hostname} in flake.nix")
+    else:
+        # Insert new entry before closing brace of nixosConfigurations
+        # Pattern: "      };\n      packages = forAllSystems"
+        pattern = r"(      \};)\n(      packages = forAllSystems)"
+        replacement = rf"{new_entry}\g<1>\n\g<2>"
+
+        new_content, count = re.subn(pattern, replacement, content)
+
+        if count == 0:
+            raise ValueError(
+                "Could not find insertion point in flake.nix. "
+                "Looking for pattern: '      };\\n      packages = forAllSystems'"
+            )
+
+    flake_path.write_text(new_content)
+
+
+def update_terraform_vms(config: HostConfig, repo_root: Path, force: bool = False) -> None:
+    """
+    Add or update VM entry in terraform/vms.tf locals.vms map.
+
+    Args:
+        config: Host configuration
+        repo_root: Path to repository root
+        force: If True, replace existing entry; if False, insert new entry
+    """
+    terraform_path = repo_root / "terraform" / "vms.tf"
+    content = terraform_path.read_text()
+
+    # Create new entry based on whether we have static IP or DHCP
+    if config.is_static_ip:
+        new_entry = f'''    "{config.hostname}" = {{
+      ip        = "{config.ip}"
+      cpu_cores = {config.cpu}
+      memory    = {config.memory}
+      disk_size = "{config.disk}"
+    }}
+'''
+    else:
+        new_entry = f'''    "{config.hostname}" = {{
+      cpu_cores = {config.cpu}
+      memory    = {config.memory}
+      disk_size = "{config.disk}"
+    }}
+'''
+
+    # Check if hostname already exists
+    hostname_pattern = rf'^\s+"{re.escape(config.hostname)}" = \{{'
+    existing_match = re.search(hostname_pattern, content, re.MULTILINE)
+
+    if existing_match and force:
+        # Replace existing entry
+        # Match the entire block from "hostname" = { to }
+        replace_pattern = rf'^\s+"{re.escape(config.hostname)}" = \{{.*?^\s+\}}\n'
+        new_content, count = re.subn(replace_pattern, new_entry, content, flags=re.MULTILINE | re.DOTALL)
+
+        if count == 0:
+            raise ValueError(f"Could not find existing entry for {config.hostname} in terraform/vms.tf")
+    else:
+        # Insert new entry before closing brace
+        # Pattern: "  }\n\n  # Compute VM configurations"
+        pattern = r"(  \})\n\n(  # Compute VM configurations)"
+        replacement = rf"{new_entry}\g<1>\n\n\g<2>"
+
+        new_content, count = re.subn(pattern, replacement, content)
+
+        if count == 0:
+            raise ValueError(
+                "Could not find insertion point in terraform/vms.tf. "
+                "Looking for pattern: '  }\\n\\n  # Compute VM configurations'"
+            )
+
+    terraform_path.write_text(new_content)
+
+
+def add_wrapped_token_to_vm(hostname: str, wrapped_token: str, repo_root: Path) -> None:
+    """
+    Add or update the vault_wrapped_token field in an existing VM entry.
+
+    Args:
+        hostname: Hostname of the VM
+        wrapped_token: The wrapped token to add
+        repo_root: Path to repository root
+    """
+    terraform_path = repo_root / "terraform" / "vms.tf"
+    content = terraform_path.read_text()
+
+    # Find the VM entry
+    hostname_pattern = rf'^\s+"{re.escape(hostname)}" = \{{'
+    match = re.search(hostname_pattern, content, re.MULTILINE)
+
+    if not match:
+        raise ValueError(f"Could not find VM entry for {hostname} in terraform/vms.tf")
+
+    # Find the full VM block
+    block_pattern = rf'(^\s+"{re.escape(hostname)}" = \{{)(.*?)(^\s+\}})'
+    block_match = re.search(block_pattern, content, re.MULTILINE | re.DOTALL)
+
+    if not block_match:
+        raise ValueError(f"Could not parse VM block for {hostname}")
+
+    block_start = block_match.group(1)
+    block_content = block_match.group(2)
+    block_end = block_match.group(3)
+
+    # Check if vault_wrapped_token already exists
+    if "vault_wrapped_token" in block_content:
+        # Update existing token
+        block_content = re.sub(
+            r'vault_wrapped_token\s*=\s*"[^"]*"',
+            f'vault_wrapped_token = "{wrapped_token}"',
+            block_content
+        )
+    else:
+        # Add new token field (add before closing brace)
+        # Find the last field and add after it
+        block_content = block_content.rstrip()
+        if block_content and not block_content.endswith("\n"):
+            block_content += "\n"
+        block_content += f'      vault_wrapped_token = "{wrapped_token}"\n'
+
+    # Reconstruct the block
+    new_block = block_start + block_content + block_end
+
+    # Replace in content
+    new_content = re.sub(
+        rf'^\s+"{re.escape(hostname)}" = \{{.*?^\s+\}}',
+        new_block,
+        content,
+        flags=re.MULTILINE | re.DOTALL
+    )
+
+    terraform_path.write_text(new_content)

scripts/create-host/models.py

@@ -0,0 +1,54 @@
+"""Data models for host configuration."""
+
+from dataclasses import dataclass
+from typing import Optional
+
+
+@dataclass
+class HostConfig:
+    """Configuration for a new NixOS host."""
+
+    hostname: str
+    ip: Optional[str] = None
+    cpu: int = 2
+    memory: int = 2048
+    disk: str = "20G"
+
+    @property
+    def is_static_ip(self) -> bool:
+        """Check if host uses static IP configuration."""
+        return self.ip is not None
+
+    @property
+    def gateway(self) -> str:
+        """Default gateway for the network."""
+        return "10.69.13.1"
+
+    @property
+    def nameservers(self) -> list[str]:
+        """DNS nameservers for the network."""
+        return ["10.69.13.5", "10.69.13.6"]
+
+    @property
+    def domain(self) -> str:
+        """Domain name for the network."""
+        return "home.2rjus.net"
+
+    @property
+    def state_version(self) -> str:
+        """NixOS state version for new hosts."""
+        return "25.11"
+
+    def validate(self) -> None:
+        """Validate configuration constraints."""
+        if not self.hostname:
+            raise ValueError("Hostname cannot be empty")
+
+        if self.cpu < 1:
+            raise ValueError("CPU cores must be at least 1")
+
+        if self.memory < 512:
+            raise ValueError("Memory must be at least 512 MB")
+
+        if not self.disk:
+            raise ValueError("Disk size cannot be empty")

scripts/create-host/setup.py

@@ -0,0 +1,35 @@
+from setuptools import setup
+from pathlib import Path
+
+# Read templates
+templates = [str(p.relative_to(".")) for p in Path("templates").glob("*.j2")]
+
+setup(
+    name="create-host",
+    version="0.1.0",
+    description="NixOS host configuration generator for homelab infrastructure",
+    py_modules=[
+        "create_host",
+        "models",
+        "validators",
+        "generators",
+        "manipulators",
+        "vault_helper",
+    ],
+    include_package_data=True,
+    data_files=[
+        ("templates", templates),
+    ],
+    install_requires=[
+        "typer",
+        "jinja2",
+        "rich",
+        "hvac",
+    ],
+    entry_points={
+        "console_scripts": [
+            "create-host=create_host:app",
+        ],
+    },
+    python_requires=">=3.9",
+)

scripts/create-host/templates/configuration.nix.j2

@@ -0,0 +1,71 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+{
+  imports = [
+    ../template2/hardware-configuration.nix
+
+    ../../system
+    ../../common/vm
+  ];
+
+  # Host metadata (adjust as needed)
+  homelab.host = {
+    tier = "test";  # Start in test tier, move to prod after validation
+  };
+
+  nixpkgs.config.allowUnfree = true;
+  boot.loader.grub.enable = true;
+  boot.loader.grub.device = "/dev/vda";
+
+  networking.hostName = "{{ hostname }}";
+  networking.domain = "{{ domain }}";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = true;
+  networking.nameservers = [
+{% for ns in nameservers %}
+    "{{ ns }}"
+{% endfor %}
+  ];
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+{% if is_static_ip %}
+    address = [
+      "{{ ip }}"
+    ];
+    routes = [
+      { Gateway = "{{ gateway }}"; }
+    ];
+{% else %}
+    networkConfig.DHCP = "ipv4";
+{% endif %}
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    vim
+    wget
+    git
+  ];
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  system.stateVersion = "{{ state_version }}"; # Did you read the comment?
+}

scripts/create-host/validators.py

@@ -0,0 +1,159 @@
+"""Validation functions for host configuration."""
+
+import re
+from pathlib import Path
+from typing import Optional
+
+
+def validate_hostname_format(hostname: str) -> None:
+    """
+    Validate hostname format according to RFC 1123.
+
+    Args:
+        hostname: Hostname to validate
+
+    Raises:
+        ValueError: If hostname format is invalid
+    """
+    # RFC 1123: lowercase, alphanumeric, hyphens, max 63 chars
+    pattern = r"^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$"
+
+    if not re.match(pattern, hostname):
+        raise ValueError(
+            f"Invalid hostname '{hostname}'. "
+            "Must be lowercase alphanumeric with hyphens, "
+            "start and end with alphanumeric, max 63 characters."
+        )
+
+
+def validate_hostname_unique(hostname: str, repo_root: Path) -> None:
+    """
+    Validate that hostname is unique in the repository.
+
+    Args:
+        hostname: Hostname to check
+        repo_root: Path to repository root
+
+    Raises:
+        ValueError: If hostname already exists
+    """
+    # Check if host directory exists
+    host_dir = repo_root / "hosts" / hostname
+    if host_dir.exists():
+        raise ValueError(f"Host directory already exists: {host_dir}")
+
+    # Check if hostname exists in flake.nix
+    flake_path = repo_root / "flake.nix"
+    if flake_path.exists():
+        flake_content = flake_path.read_text()
+        # Look for pattern like "      hostname = "
+        hostname_pattern = rf'^\s+{re.escape(hostname)}\s*='
+        if re.search(hostname_pattern, flake_content, re.MULTILINE):
+            raise ValueError(f"Hostname '{hostname}' already exists in flake.nix")
+
+
+def validate_ip_format(ip: str) -> None:
+    """
+    Validate IP address format with CIDR notation.
+
+    Args:
+        ip: IP address with CIDR (e.g., "10.69.13.50/24")
+
+    Raises:
+        ValueError: If IP format is invalid
+    """
+    if not ip:
+        return
+
+    # Check CIDR notation
+    if "/" not in ip:
+        raise ValueError(f"IP address must include CIDR notation (e.g., {ip}/24)")
+
+    ip_part, cidr_part = ip.rsplit("/", 1)
+
+    # Validate CIDR is /24
+    if cidr_part != "24":
+        raise ValueError(f"CIDR notation must be /24, got /{cidr_part}")
+
+    # Validate IP format
+    octets = ip_part.split(".")
+    if len(octets) != 4:
+        raise ValueError(f"Invalid IP address format: {ip_part}")
+
+    try:
+        octet_values = [int(octet) for octet in octets]
+    except ValueError:
+        raise ValueError(f"Invalid IP address format: {ip_part}")
+
+    # Check each octet is 0-255
+    for i, value in enumerate(octet_values):
+        if not 0 <= value <= 255:
+            raise ValueError(f"Invalid octet value {value} in IP address")
+
+    # Check last octet is 1-254
+    if not 1 <= octet_values[3] <= 254:
+        raise ValueError(
+            f"Last octet must be 1-254, got {octet_values[3]}"
+        )
+
+
+def validate_ip_subnet(ip: str) -> None:
+    """
+    Validate that IP address is in the correct subnet (10.69.13.0/24).
+
+    Args:
+        ip: IP address with CIDR (e.g., "10.69.13.50/24")
+
+    Raises:
+        ValueError: If IP is not in correct subnet
+    """
+    if not ip:
+        return
+
+    validate_ip_format(ip)
+
+    ip_part = ip.split("/")[0]
+    octets = ip_part.split(".")
+
+    # Check subnet is 10.69.13.x
+    if octets[:3] != ["10", "69", "13"]:
+        raise ValueError(
+            f"IP address must be in 10.69.13.0/24 subnet, got {ip_part}"
+        )
+
+
+def validate_ip_unique(ip: Optional[str], repo_root: Path) -> None:
+    """
+    Validate that IP address is not already in use.
+
+    Args:
+        ip: IP address with CIDR to check (None for DHCP)
+        repo_root: Path to repository root
+
+    Raises:
+        ValueError: If IP is already in use
+    """
+    if not ip:
+        return  # DHCP mode, no uniqueness check needed
+
+    # Extract just the IP part without CIDR for searching
+    ip_part = ip.split("/")[0]
+
+    # Check all hosts/*/configuration.nix files
+    hosts_dir = repo_root / "hosts"
+    if hosts_dir.exists():
+        for config_file in hosts_dir.glob("*/configuration.nix"):
+            content = config_file.read_text()
+            if ip_part in content:
+                raise ValueError(
+                    f"IP address {ip_part} already in use in {config_file}"
+                )
+
+    # Check terraform/vms.tf
+    terraform_file = repo_root / "terraform" / "vms.tf"
+    if terraform_file.exists():
+        content = terraform_file.read_text()
+        if ip_part in content:
+            raise ValueError(
+                f"IP address {ip_part} already in use in {terraform_file}"
+            )

scripts/create-host/vault_helper.py

@@ -0,0 +1,178 @@
+"""Helper functions for Vault/OpenBao API interactions."""
+
+import os
+import subprocess
+from pathlib import Path
+from typing import Optional
+
+import hvac
+import typer
+
+
+def get_vault_client(vault_addr: Optional[str] = None, vault_token: Optional[str] = None) -> hvac.Client:
+    """
+    Get a Vault client instance.
+
+    Args:
+        vault_addr: Vault server address (defaults to BAO_ADDR env var or hardcoded default)
+        vault_token: Vault token (defaults to BAO_TOKEN env var or prompts user)
+
+    Returns:
+        Configured hvac.Client instance
+
+    Raises:
+        typer.Exit: If unable to create client or authenticate
+    """
+    # Get Vault address
+    if vault_addr is None:
+        vault_addr = os.getenv("BAO_ADDR", "https://vault01.home.2rjus.net:8200")
+
+    # Get Vault token
+    if vault_token is None:
+        vault_token = os.getenv("BAO_TOKEN")
+
+    if not vault_token:
+        typer.echo("\n⚠️  Vault token required. Set BAO_TOKEN environment variable or enter it below.")
+        vault_token = typer.prompt("Vault token (BAO_TOKEN)", hide_input=True)
+
+    # Create client
+    try:
+        client = hvac.Client(url=vault_addr, token=vault_token, verify=False)
+
+        # Verify authentication
+        if not client.is_authenticated():
+            typer.echo(f"\n❌ Failed to authenticate to Vault at {vault_addr}", err=True)
+            typer.echo("Check your BAO_TOKEN and ensure Vault is accessible", err=True)
+            raise typer.Exit(code=1)
+
+        return client
+
+    except Exception as e:
+        typer.echo(f"\n❌ Error connecting to Vault: {e}", err=True)
+        raise typer.Exit(code=1)
+
+
+def generate_wrapped_token(hostname: str, repo_root: Path) -> str:
+    """
+    Generate a wrapped token containing AppRole credentials for a host.
+
+    This function:
+    1. Applies Terraform to ensure the AppRole exists
+    2. Reads the role_id for the host
+    3. Generates a secret_id
+    4. Wraps both credentials in a cubbyhole token (24h TTL, single-use)
+
+    Args:
+        hostname: The host to generate credentials for
+        repo_root: Path to repository root (for running terraform)
+
+    Returns:
+        Wrapped token string (hvs.CAES...)
+
+    Raises:
+        typer.Exit: If Terraform fails or Vault operations fail
+    """
+    from rich.console import Console
+
+    console = Console()
+
+    # Get Vault client
+    client = get_vault_client()
+
+    # First, apply Terraform to ensure AppRole exists
+    console.print(f"\n[bold blue]Applying Vault Terraform configuration...[/bold blue]")
+    terraform_dir = repo_root / "terraform" / "vault"
+
+    try:
+        result = subprocess.run(
+            ["tofu", "apply", "-auto-approve"],
+            cwd=terraform_dir,
+            capture_output=True,
+            text=True,
+            check=False,
+        )
+
+        if result.returncode != 0:
+            console.print(f"[red]❌ Terraform apply failed:[/red]")
+            console.print(result.stderr)
+            raise typer.Exit(code=1)
+
+        console.print("[green]✓[/green] Terraform applied successfully")
+
+    except FileNotFoundError:
+        console.print(f"[red]❌ Error: 'tofu' command not found[/red]")
+        console.print("Ensure OpenTofu is installed and in PATH")
+        raise typer.Exit(code=1)
+
+    # Read role_id
+    try:
+        console.print(f"[bold blue]Reading AppRole credentials for {hostname}...[/bold blue]")
+        role_id_response = client.read(f"auth/approle/role/{hostname}/role-id")
+        role_id = role_id_response["data"]["role_id"]
+        console.print(f"[green]✓[/green] Retrieved role_id")
+
+    except Exception as e:
+        console.print(f"[red]❌ Failed to read role_id for {hostname}:[/red] {e}")
+        console.print(f"\nEnsure the AppRole '{hostname}' exists in Vault")
+        raise typer.Exit(code=1)
+
+    # Generate secret_id
+    try:
+        secret_id_response = client.write(f"auth/approle/role/{hostname}/secret-id")
+        secret_id = secret_id_response["data"]["secret_id"]
+        console.print(f"[green]✓[/green] Generated secret_id")
+
+    except Exception as e:
+        console.print(f"[red]❌ Failed to generate secret_id:[/red] {e}")
+        raise typer.Exit(code=1)
+
+    # Wrap the credentials in a cubbyhole token
+    try:
+        console.print(f"[bold blue]Creating wrapped token (24h TTL, single-use)...[/bold blue]")
+
+        # Use the response wrapping feature to wrap our credentials
+        # This creates a temporary token that can only be used once to retrieve the actual credentials
+        wrap_response = client.write(
+            "sys/wrapping/wrap",
+            wrap_ttl="24h",
+            # The data we're wrapping
+            role_id=role_id,
+            secret_id=secret_id,
+        )
+
+        wrapped_token = wrap_response["wrap_info"]["token"]
+        console.print(f"[green]✓[/green] Created wrapped token: {wrapped_token[:20]}...")
+        console.print(f"[yellow]⚠️[/yellow]  Token expires in 24 hours")
+        console.print(f"[yellow]⚠️[/yellow]  Token can only be used once")
+
+        return wrapped_token
+
+    except Exception as e:
+        console.print(f"[red]❌ Failed to create wrapped token:[/red] {e}")
+        raise typer.Exit(code=1)
+
+
+def verify_vault_setup(hostname: str) -> bool:
+    """
+    Verify that Vault is properly configured for a host.
+
+    Checks:
+    - Vault is accessible
+    - AppRole exists for the hostname
+    - Can read role_id
+
+    Args:
+        hostname: The host to verify
+
+    Returns:
+        True if everything is configured correctly, False otherwise
+    """
+    try:
+        client = get_vault_client()
+
+        # Try to read the role_id
+        client.read(f"auth/approle/role/{hostname}/role-id")
+        return True
+
+    except Exception:
+        return False

scripts/vault-fetch/README.md

@@ -0,0 +1,78 @@
+# vault-fetch
+
+A helper script for fetching secrets from OpenBao/Vault and writing them to the filesystem.
+
+## Features
+
+- **AppRole Authentication**: Uses role_id and secret_id from `/var/lib/vault/approle/`
+- **Individual Secret Files**: Writes each secret key as a separate file for easy consumption
+- **Caching**: Maintains a cache of secrets for fallback when Vault is unreachable
+- **Graceful Degradation**: Falls back to cached secrets if Vault authentication fails
+- **Secure Permissions**: Sets 600 permissions on all secret files
+
+## Usage
+
+```bash
+vault-fetch <secret-path> <output-directory> [cache-directory]
+```
+
+### Examples
+
+```bash
+# Fetch Grafana admin secrets
+vault-fetch hosts/monitoring01/grafana-admin /run/secrets/grafana /var/lib/vault/cache/grafana
+
+# Use default cache location
+vault-fetch hosts/monitoring01/grafana-admin /run/secrets/grafana
+```
+
+## How It Works
+
+1. **Read Credentials**: Loads `role_id` and `secret_id` from `/var/lib/vault/approle/`
+2. **Authenticate**: Calls `POST /v1/auth/approle/login` to get a Vault token
+3. **Fetch Secret**: Retrieves secret from `GET /v1/secret/data/{path}`
+4. **Extract Keys**: Parses JSON response and extracts individual secret keys
+5. **Write Files**: Creates one file per secret key in output directory
+6. **Update Cache**: Copies secrets to cache directory for fallback
+7. **Set Permissions**: Ensures all files have 600 permissions (owner read/write only)
+
+## Error Handling
+
+If Vault is unreachable or authentication fails:
+- Script logs a warning to stderr
+- Falls back to cached secrets from previous successful fetch
+- Exits with error code 1 if no cache is available
+
+## Environment Variables
+
+- `VAULT_ADDR`: Vault server address (default: `https://vault01.home.2rjus.net:8200`)
+- `VAULT_SKIP_VERIFY`: Skip TLS verification (default: `1`)
+
+## Integration with NixOS
+
+This tool is designed to be called from systemd service `ExecStartPre` hooks via the `vault.secrets` NixOS module:
+
+```nix
+vault.secrets.grafana-admin = {
+  secretPath = "hosts/monitoring01/grafana-admin";
+};
+
+# Service automatically gets secrets fetched before start
+systemd.services.grafana.serviceConfig = {
+  EnvironmentFile = "/run/secrets/grafana-admin/password";
+};
+```
+
+## Requirements
+
+- `curl`: For Vault API calls
+- `jq`: For JSON parsing
+- `coreutils`: For file operations
+
+## Security Considerations
+
+- AppRole credentials stored at `/var/lib/vault/approle/` should be root-owned with 600 permissions
+- Tokens are ephemeral and not stored - fresh authentication on each fetch
+- Secrets written to tmpfs (`/run/secrets/`) are lost on reboot
+- Cache directory persists across reboots for service availability
+- All secret files have restrictive permissions (600)

scripts/vault-fetch/default.nix

@@ -0,0 +1,18 @@
+{ pkgs, lib, ... }:
+
+pkgs.writeShellApplication {
+  name = "vault-fetch";
+
+  runtimeInputs = with pkgs; [
+    curl      # Vault API calls
+    jq        # JSON parsing
+    coreutils # File operations
+  ];
+
+  text = builtins.readFile ./vault-fetch.sh;
+
+  meta = with lib; {
+    description = "Fetch secrets from OpenBao/Vault and write to filesystem";
+    license = licenses.mit;
+  };
+}

scripts/vault-fetch/vault-fetch.sh

@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# vault-fetch: Fetch secrets from OpenBao/Vault and write to filesystem
+#
+# Usage: vault-fetch <secret-path> <output-directory> [cache-directory]
+#
+# Example: vault-fetch hosts/monitoring01/grafana-admin /run/secrets/grafana /var/lib/vault/cache/grafana
+#
+# This script:
+# 1. Authenticates to Vault using AppRole credentials from /var/lib/vault/approle/
+# 2. Fetches secrets from the specified path
+# 3. Writes each secret key as an individual file in the output directory
+# 4. Updates cache for fallback when Vault is unreachable
+# 5. Falls back to cache if Vault authentication fails or is unreachable
+
+# Parse arguments
+if [ $# -lt 2 ]; then
+    echo "Usage: vault-fetch <secret-path> <output-directory> [cache-directory]" >&2
+    echo "Example: vault-fetch hosts/monitoring01/grafana /run/secrets/grafana /var/lib/vault/cache/grafana" >&2
+    exit 1
+fi
+
+SECRET_PATH="$1"
+OUTPUT_DIR="$2"
+CACHE_DIR="${3:-/var/lib/vault/cache/$(basename "$OUTPUT_DIR")}"
+
+# Vault configuration
+VAULT_ADDR="${VAULT_ADDR:-https://vault01.home.2rjus.net:8200}"
+VAULT_SKIP_VERIFY="${VAULT_SKIP_VERIFY:-1}"
+APPROLE_DIR="/var/lib/vault/approle"
+
+# TLS verification flag for curl
+if [ "$VAULT_SKIP_VERIFY" = "1" ]; then
+    CURL_TLS_FLAG="-k"
+else
+    CURL_TLS_FLAG=""
+fi
+
+# Logging helper
+log() {
+    echo "[vault-fetch] $*" >&2
+}
+
+# Error handler
+error() {
+    log "ERROR: $*"
+    exit 1
+}
+
+# Check if cache is available
+has_cache() {
+    [ -d "$CACHE_DIR" ] && [ -n "$(ls -A "$CACHE_DIR" 2>/dev/null)" ]
+}
+
+# Use cached secrets
+use_cache() {
+    if ! has_cache; then
+        error "No cache available and Vault is unreachable"
+    fi
+
+    log "WARNING: Using cached secrets from $CACHE_DIR"
+    mkdir -p "$OUTPUT_DIR"
+    cp -r "$CACHE_DIR"/* "$OUTPUT_DIR/"
+    chmod -R u=rw,go= "$OUTPUT_DIR"/*
+}
+
+# Fetch secrets from Vault
+fetch_from_vault() {
+    # Read AppRole credentials
+    if [ ! -f "$APPROLE_DIR/role-id" ] || [ ! -f "$APPROLE_DIR/secret-id" ]; then
+        log "WARNING: AppRole credentials not found at $APPROLE_DIR"
+        use_cache
+        return
+    fi
+
+    ROLE_ID=$(cat "$APPROLE_DIR/role-id")
+    SECRET_ID=$(cat "$APPROLE_DIR/secret-id")
+
+    # Authenticate to Vault
+    log "Authenticating to Vault at $VAULT_ADDR"
+    AUTH_RESPONSE=$(curl -s $CURL_TLS_FLAG -X POST \
+        -d "{\"role_id\":\"$ROLE_ID\",\"secret_id\":\"$SECRET_ID\"}" \
+        "$VAULT_ADDR/v1/auth/approle/login" 2>&1) || {
+        log "WARNING: Failed to connect to Vault"
+        use_cache
+        return
+    }
+
+    # Check for errors in response
+    if echo "$AUTH_RESPONSE" | jq -e '.errors' >/dev/null 2>&1; then
+        ERRORS=$(echo "$AUTH_RESPONSE" | jq -r '.errors[]' 2>/dev/null || echo "Unknown error")
+        log "WARNING: Vault authentication failed: $ERRORS"
+        use_cache
+        return
+    fi
+
+    # Extract token
+    VAULT_TOKEN=$(echo "$AUTH_RESPONSE" | jq -r '.auth.client_token' 2>/dev/null)
+    if [ -z "$VAULT_TOKEN" ] || [ "$VAULT_TOKEN" = "null" ]; then
+        log "WARNING: Failed to extract Vault token from response"
+        use_cache
+        return
+    fi
+
+    log "Successfully authenticated to Vault"
+
+    # Fetch secret
+    log "Fetching secret from path: $SECRET_PATH"
+    SECRET_RESPONSE=$(curl -s $CURL_TLS_FLAG \
+        -H "X-Vault-Token: $VAULT_TOKEN" \
+        "$VAULT_ADDR/v1/secret/data/$SECRET_PATH" 2>&1) || {
+        log "WARNING: Failed to fetch secret from Vault"
+        use_cache
+        return
+    }
+
+    # Check for errors
+    if echo "$SECRET_RESPONSE" | jq -e '.errors' >/dev/null 2>&1; then
+        ERRORS=$(echo "$SECRET_RESPONSE" | jq -r '.errors[]' 2>/dev/null || echo "Unknown error")
+        log "WARNING: Failed to fetch secret: $ERRORS"
+        use_cache
+        return
+    fi
+
+    # Extract secret data
+    SECRET_DATA=$(echo "$SECRET_RESPONSE" | jq -r '.data.data' 2>/dev/null)
+    if [ -z "$SECRET_DATA" ] || [ "$SECRET_DATA" = "null" ]; then
+        log "WARNING: No secret data found at path $SECRET_PATH"
+        use_cache
+        return
+    fi
+
+    # Create output and cache directories
+    mkdir -p "$OUTPUT_DIR"
+    mkdir -p "$CACHE_DIR"
+
+    # Write each secret key to a separate file
+    log "Writing secrets to $OUTPUT_DIR"
+    for key in $(echo "$SECRET_DATA" | jq -r 'keys[]'); do
+        echo "$SECRET_DATA" | jq -j --arg k "$key" '.[$k]' > "$OUTPUT_DIR/$key"
+        echo "$SECRET_DATA" | jq -j --arg k "$key" '.[$k]' > "$CACHE_DIR/$key"
+        chmod 600 "$OUTPUT_DIR/$key"
+        chmod 600 "$CACHE_DIR/$key"
+        log "  - Wrote secret key: $key"
+    done
+
+    log "Successfully fetched and cached secrets"
+}
+
+# Main execution
+fetch_from_vault