nixos-servers/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Repository Overview

This is a Nix Flake-based NixOS configuration repository for managing a homelab infrastructure consisting of 16 server configurations. The repository uses a modular architecture with shared system configurations, reusable service modules, and per-host customization.

## Common Commands

### Building Configurations

```bash
# List all available configurations
nix flake show

# Build a specific host configuration locally (without deploying)
nixos-rebuild build --flake .#<hostname>

# Build and check a configuration
nix build .#nixosConfigurations.<hostname>.config.system.build.toplevel
```

**Important:** Do NOT pipe `nix build` commands to other commands like `tail` or `head`. Piping can hide errors and make builds appear successful when they actually failed. Always run `nix build` without piping to see the full output.

```bash
# BAD - hides errors
nix build .#create-host 2>&1 | tail -20

# GOOD - shows all output and errors
nix build .#create-host
```

### Deployment

Do not automatically deploy changes. Deployments are usually done by updating the master branch, and then triggering the auto update on the specific host.

### Testing Feature Branches on Hosts

All hosts have the `nixos-rebuild-test` helper script for testing feature branches before merging:

```bash
# On the target host, test a feature branch
nixos-rebuild-test boot <branch-name>
nixos-rebuild-test switch <branch-name>

# Additional arguments are passed through to nixos-rebuild
nixos-rebuild-test boot my-feature --show-trace
```

When working on a feature branch that requires testing on a live host, suggest using this command instead of the full flake URL syntax.

### Flake Management

```bash
# Check flake for errors
nix flake check
```
Do not run `nix flake update`. Should only be done manually by user.

### Development Environment

```bash
# Enter development shell (provides ansible, python3)
nix develop
```

### Secrets Management

Secrets are managed by OpenBao (Vault) using AppRole authentication. Most hosts use the
`vault.secrets` option defined in `system/vault-secrets.nix` to fetch secrets at boot.
Terraform manages the secrets and AppRole policies in `terraform/vault/`.

Legacy sops-nix is still present but only actively used by the `ca` host. Do not edit any
`.sops.yaml` or any file within `secrets/`. Ask the user to modify if necessary.

### Git Workflow

**Important:** Never commit directly to `master` unless the user explicitly asks for it. Always create a feature branch for changes.

When starting a new plan or task, the first step should typically be to create and checkout a new branch with an appropriate name (e.g., `git checkout -b dns-automation` or `git checkout -b fix-nginx-config`).

### Plan Management

When creating plans for large features, follow this workflow:

1. When implementation begins, save a copy of the plan to `docs/plans/` (e.g., `docs/plans/feature-name.md`)
2. Once the feature is fully implemented, move the plan to `docs/plans/completed/`

### Git Commit Messages

Commit messages should follow the format: `topic: short description`

Examples:
- `flake: add opentofu to devshell`
- `template2: add proxmox image configuration`
- `terraform: add VM deployment configuration`

### Clipboard

To copy text to the clipboard, pipe to `wl-copy` (Wayland):

```bash
echo "text" | wl-copy
```

### NixOS Options and Packages Lookup

Two MCP servers are available for searching NixOS options and packages:

- **nixpkgs-options** - Search and lookup NixOS configuration option documentation
- **nixpkgs-packages** - Search and lookup Nix packages from nixpkgs

**Session Setup:** At the start of each session, index the nixpkgs revision from `flake.lock` to ensure documentation matches the project's nixpkgs version:

1. Read `flake.lock` and find the `nixpkgs` node's `rev` field
2. Call `index_revision` with that git hash (both servers share the same index)

**Options Tools (nixpkgs-options):**

- `search_options` - Search for options by name or description (e.g., query "nginx" or "postgresql")
- `get_option` - Get full details for a specific option (e.g., `services.loki.configuration`)
- `get_file` - Fetch the source file from nixpkgs that declares an option

**Package Tools (nixpkgs-packages):**

- `search_packages` - Search for packages by name or description (e.g., query "nginx" or "python")
- `get_package` - Get full details for a specific package by attribute path (e.g., `firefox`, `python312Packages.requests`)
- `get_file` - Fetch the source file from nixpkgs that defines a package

This ensures documentation matches the exact nixpkgs version (currently NixOS 25.11) used by this flake.

### Lab Monitoring Log Queries

The **lab-monitoring** MCP server can query logs from Loki. All hosts ship systemd journal logs via Promtail.

**Loki Label Reference:**

- `host` - Hostname (e.g., `ns1`, `ns2`, `monitoring01`, `ha1`). Use this label, not `hostname`.
- `systemd_unit` - Systemd unit name (e.g., `nsd.service`, `prometheus.service`, `nixos-upgrade.service`)
- `job` - Either `systemd-journal` (most logs) or `varlog` (file-based logs like caddy access logs)
- `filename` - For `varlog` job, the log file path (e.g., `/var/log/caddy/nix-cache.log`)

Journal log entries are JSON-formatted with the actual log message in the `MESSAGE` field. Other useful fields include `PRIORITY` and `SYSLOG_IDENTIFIER`.

**Example LogQL queries:**
```
# Logs from a specific service on a host
{host="ns2", systemd_unit="nsd.service"}

# Substring match on log content
{host="ns1", systemd_unit="nsd.service"} |= "error"

# File-based logs (e.g., caddy access logs)
{job="varlog", hostname="nix-cache01"}
```

Default lookback is 1 hour. Use the `start` parameter with relative durations (e.g., `24h`, `168h`) for older logs.

### Lab Monitoring Prometheus Queries

The **lab-monitoring** MCP server can query Prometheus metrics via PromQL. The `instance` label uses the FQDN format `<host>.home.2rjus.net:<port>`.

**Prometheus Job Names:**

- `node-exporter` - System metrics from all hosts (CPU, memory, disk, network)
- `caddy` - Reverse proxy metrics (http-proxy)
- `nix-cache_caddy` - Nix binary cache metrics
- `home-assistant` - Home automation metrics
- `jellyfin` - Media server metrics
- `loki` / `prometheus` / `grafana` - Monitoring stack self-metrics
- `step-ca` - Internal CA metrics
- `pve-exporter` - Proxmox hypervisor metrics
- `smartctl` - Disk SMART health (gunter)
- `wireguard` - VPN metrics (http-proxy)
- `pushgateway` - Push-based metrics (e.g., backup results)
- `restic_rest` - Backup server metrics
- `labmon` / `ghettoptt` / `alertmanager` - Other service metrics

**Example PromQL queries:**
```
# Check all targets are up
up

# CPU usage for a specific host
rate(node_cpu_seconds_total{instance=~"ns1.*", mode!="idle"}[5m])

# Memory usage across all hosts
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# Disk space
node_filesystem_avail_bytes{mountpoint="/"}
```

## Architecture

### Directory Structure

- `/flake.nix` - Central flake defining all NixOS configurations
- `/hosts/<hostname>/` - Per-host configurations
  - `default.nix` - Entry point, imports configuration.nix and services
  - `configuration.nix` - Host-specific settings (networking, hardware, users)
- `/system/` - Shared system-level configurations applied to ALL hosts
  - Core modules: nix.nix, sshd.nix, sops.nix (legacy), vault-secrets.nix, acme.nix, autoupgrade.nix
  - Monitoring: node-exporter and promtail on every host
- `/modules/` - Custom NixOS modules
  - `homelab/` - Homelab-specific options (DNS automation, monitoring scrape targets)
- `/lib/` - Nix library functions
  - `dns-zone.nix` - DNS zone generation functions
  - `monitoring.nix` - Prometheus scrape target generation functions
- `/services/` - Reusable service modules, selectively imported by hosts
  - `home-assistant/` - Home automation stack
  - `monitoring/` - Observability stack (Prometheus, Grafana, Loki, Tempo)
  - `ns/` - DNS services (authoritative, resolver, zone generation)
  - `http-proxy/`, `ca/`, `postgres/`, `nats/`, `jellyfin/`, etc.
- `/secrets/` - SOPS-encrypted secrets with age encryption (legacy, only used by ca)
- `/common/` - Shared configurations (e.g., VM guest agent)
- `/docs/` - Documentation and plans
  - `plans/` - Future plans and proposals
  - `plans/completed/` - Completed plans (moved here when done)
- `/playbooks/` - Ansible playbooks for fleet management
- `/.sops.yaml` - SOPS configuration with age keys (legacy, only used by ca)

### Configuration Inheritance

Each host follows this import pattern:
```
hosts/<hostname>/default.nix
  └─> configuration.nix (host-specific)
      ├─> ../../system (ALL shared system configs - applied to every host)
      ├─> ../../services/<service> (selective service imports)
      └─> ../../common/vm (if VM)
```

All hosts automatically get:
- Nix binary cache (nix-cache.home.2rjus.net)
- SSH with root login enabled
- OpenBao (Vault) secrets management via AppRole
- Internal ACME CA integration (ca.home.2rjus.net)
- Daily auto-upgrades with auto-reboot
- Prometheus node-exporter + Promtail (logs to monitoring01)
- Monitoring scrape target auto-registration via `homelab.monitoring` options
- Custom root CA trust
- DNS zone auto-registration via `homelab.dns` options

### Active Hosts

Production servers managed by `rebuild-all.sh`:
- `ns1`, `ns2` - Primary/secondary DNS servers (10.69.13.5/6)
- `ca` - Internal Certificate Authority
- `ha1` - Home Assistant + Zigbee2MQTT + Mosquitto
- `http-proxy` - Reverse proxy
- `monitoring01` - Full observability stack (Prometheus, Grafana, Loki, Tempo, Pyroscope)
- `jelly01` - Jellyfin media server
- `nix-cache01` - Binary cache server
- `pgdb1` - PostgreSQL database
- `nats1` - NATS messaging server

Template/test hosts:
- `template1` - Base template for cloning new hosts

### Flake Inputs

- `nixpkgs` - NixOS 25.11 stable (primary)
- `nixpkgs-unstable` - Unstable channel (available via overlay as `pkgs.unstable.<package>`)
- `sops-nix` - Secrets management (legacy, only used by ca)
- Custom packages from git.t-juice.club:
  - `alerttonotify` - Alert routing
  - `labmon` - Lab monitoring

### Network Architecture

- Domain: `home.2rjus.net`
- Infrastructure subnet: `10.69.13.x`
- DNS: ns1/ns2 provide authoritative DNS with primary-secondary setup
- Internal CA for ACME certificates (no Let's Encrypt)
- Centralized monitoring at monitoring01
- Static networking via systemd-networkd

### Secrets Management

Most hosts use OpenBao (Vault) for secrets:
- Vault server at `vault01.home.2rjus.net:8200`
- AppRole authentication with credentials at `/var/lib/vault/approle/`
- Secrets defined in Terraform (`terraform/vault/secrets.tf`)
- AppRole policies in Terraform (`terraform/vault/approle.tf`)
- NixOS module: `system/vault-secrets.nix` with `vault.secrets.<name>` options
- `extractKey` option extracts a single key from vault JSON as a plain file
- Secrets fetched at boot by `vault-secret-<name>.service` systemd units
- Fallback to cached secrets in `/var/lib/vault/cache/` when Vault is unreachable
- Provision AppRole credentials: `nix develop -c ansible-playbook playbooks/provision-approle.yml -e hostname=<host>`

Legacy SOPS (only used by `ca` host):
- SOPS with age encryption, keys in `.sops.yaml`
- Shared secrets: `/secrets/secrets.yaml`
- Per-host secrets: `/secrets/<hostname>/`

### Auto-Upgrade System

All hosts pull updates daily from:
```
git+https://git.t-juice.club/torjus/nixos-servers.git
```

Configured in `/system/autoupgrade.nix`:
- Random delay to avoid simultaneous upgrades
- Auto-reboot after successful upgrade
- Systemd service: `nixos-upgrade.service`

### Proxmox VM Provisioning with OpenTofu

The repository includes automated workflows for building Proxmox VM templates and deploying VMs using OpenTofu (Terraform).

#### Building and Deploying Templates

Template VMs are built from `hosts/template2` and deployed to Proxmox using Ansible:

```bash
# Build NixOS image and deploy to Proxmox as template
nix develop -c ansible-playbook -i playbooks/inventory.ini playbooks/build-and-deploy-template.yml
```

This playbook:
1. Builds the Proxmox image using `nixos-rebuild build-image --image-variant proxmox`
2. Uploads the `.vma.zst` image to Proxmox at `/var/lib/vz/dump`
3. Restores it as VM ID 9000
4. Converts it to a template

Template configuration (`hosts/template2`):
- Minimal base system with essential packages (age, vim, wget, git)
- Cloud-init configured for NoCloud datasource (no EC2 metadata timeout)
- DHCP networking on ens18
- SSH key-based root login
- `prepare-host.sh` script for cleaning machine-id, SSH keys, and regenerating age keys

#### Deploying VMs with OpenTofu

VMs are deployed from templates using OpenTofu in the `/terraform` directory:

```bash
cd terraform
tofu init     # First time only
tofu apply    # Deploy VMs
```

Configuration files:
- `main.tf` - Proxmox provider configuration
- `variables.tf` - Provider variables (API credentials)
- `vm.tf` - VM resource definitions
- `terraform.tfvars` - Actual credentials (gitignored)

Example VM deployment includes:
- Clone from template VM
- Cloud-init configuration (SSH keys, network, DNS)
- Custom CPU/memory/disk sizing
- VLAN tagging
- QEMU guest agent

OpenTofu outputs the VM's IP address after deployment for easy SSH access.

#### Template Rebuilding and Terraform State

When the Proxmox template is rebuilt (via `build-and-deploy-template.yml`), the template name may change. This would normally cause Terraform to want to recreate all existing VMs, but that's unnecessary since VMs are independent once cloned.

**Solution**: The `terraform/vms.tf` file includes a lifecycle rule to ignore certain attributes that don't need management:

```hcl
lifecycle {
  ignore_changes = [
    clone,            # Template name can change without recreating VMs
    startup_shutdown, # Proxmox sets defaults (-1) that we don't need to manage
  ]
}
```

This means:
- **clone**: Existing VMs are not affected by template name changes; only new VMs use the updated template
- **startup_shutdown**: Proxmox sets default startup order/delay values (-1) that Terraform would otherwise try to remove
- You can safely update `default_template_name` in `terraform/variables.tf` without recreating VMs
- `tofu plan` won't show spurious changes for Proxmox-managed defaults

**When rebuilding the template:**
1. Run `nix develop -c ansible-playbook -i playbooks/inventory.ini playbooks/build-and-deploy-template.yml`
2. Update `default_template_name` in `terraform/variables.tf` if the name changed
3. Run `tofu plan` - should show no VM recreations (only template name in state)
4. Run `tofu apply` - updates state without touching existing VMs
5. New VMs created after this point will use the new template

### Adding a New Host

1. Create `/hosts/<hostname>/` directory
2. Copy structure from `template1` or similar host
3. Add host entry to `flake.nix` nixosConfigurations
4. Configure networking in `configuration.nix` (static IP via `systemd.network.networks`, DNS servers)
5. (Optional) Add `homelab.dns.cnames` if the host needs CNAME aliases
6. Add `vault.enable = true;` to the host configuration
7. Add AppRole policy in `terraform/vault/approle.tf` and any secrets in `secrets.tf`
8. Run `tofu apply` in `terraform/vault/`
9. User clones template host
10. User runs `prepare-host.sh` on new host
11. Provision AppRole credentials: `nix develop -c ansible-playbook playbooks/provision-approle.yml -e hostname=<host>`
12. Commit changes, and merge to master.
13. Deploy by running `nixos-rebuild boot --flake URL#<hostname>` on the host.
14. Run auto-upgrade on DNS servers (ns1, ns2) to pick up the new host's DNS entry

**Note:** DNS A records and Prometheus node-exporter scrape targets are auto-generated from the host's `systemd.network.networks` static IP configuration. No manual zone file or Prometheus config editing is required.

### Important Patterns

**Overlay usage**: Access unstable packages via `pkgs.unstable.<package>` (defined in flake.nix overlay-unstable)

**Service composition**: Services in `/services/` are designed to be imported by multiple hosts. Keep them modular and reusable.

**Hardware configuration reuse**: Multiple hosts share `/hosts/template/hardware-configuration.nix` for VM instances.

**State version**: All hosts use stateVersion `"23.11"` - do not change this on existing hosts.

**Firewall**: Disabled on most hosts (trusted network). Enable selectively in host configuration if needed.

**Shell scripts**: Use `pkgs.writeShellApplication` instead of `pkgs.writeShellScript` or `pkgs.writeShellScriptBin` for creating shell scripts. `writeShellApplication` provides automatic shellcheck validation, sets strict bash options (`set -euo pipefail`), and allows declaring `runtimeInputs` for dependencies.

### Monitoring Stack

All hosts ship metrics and logs to `monitoring01`:
- **Metrics**: Prometheus scrapes node-exporter from all hosts
- **Logs**: Promtail ships logs to Loki on monitoring01
- **Access**: Grafana at monitoring01 for visualization
- **Tracing**: Tempo for distributed tracing
- **Profiling**: Pyroscope for continuous profiling

**Scrape Target Auto-Generation:**

Prometheus scrape targets are automatically generated from host configurations, following the same pattern as DNS zone generation:

- **Node-exporter**: All flake hosts with static IPs are automatically added as node-exporter targets
- **Service targets**: Defined via `homelab.monitoring.scrapeTargets` in service modules
- **External targets**: Non-flake hosts defined in `/services/monitoring/external-targets.nix`
- **Library**: `lib/monitoring.nix` provides `generateNodeExporterTargets` and `generateScrapeConfigs`

Host monitoring options (`homelab.monitoring.*`):
- `enable` (default: `true`) - Include host in Prometheus node-exporter scrape targets
- `scrapeTargets` (default: `[]`) - Additional scrape targets exposed by this host (job_name, port, metrics_path, scheme, scrape_interval, honor_labels)

Service modules declare their scrape targets directly (e.g., `services/ca/default.nix` declares step-ca on port 9000). The Prometheus config on monitoring01 auto-generates scrape configs from all hosts.

To add monitoring targets for non-NixOS hosts, edit `/services/monitoring/external-targets.nix`.

### DNS Architecture

- `ns1` (10.69.13.5) - Primary authoritative DNS + resolver
- `ns2` (10.69.13.6) - Secondary authoritative DNS (AXFR from ns1)
- All hosts point to ns1/ns2 for DNS resolution

**Zone Auto-Generation:**

DNS zone entries are automatically generated from host configurations:

- **Flake-managed hosts**: A records extracted from `systemd.network.networks` static IPs
- **CNAMEs**: Defined via `homelab.dns.cnames` option in host configs
- **External hosts**: Non-flake hosts defined in `/services/ns/external-hosts.nix`
- **Serial number**: Uses `self.sourceInfo.lastModified` (git commit timestamp)

Host DNS options (`homelab.dns.*`):
- `enable` (default: `true`) - Include host in DNS zone generation
- `cnames` (default: `[]`) - List of CNAME aliases pointing to this host

Hosts are automatically excluded from DNS if:
- `homelab.dns.enable = false` (e.g., template hosts)
- No static IP configured (e.g., DHCP-only hosts)
- Network interface is a VPN/tunnel (wg*, tun*, tap*)

To add DNS entries for non-NixOS hosts, edit `/services/ns/external-hosts.nix`.