nixos-servers/docs/plans/host-migration-to-opentofu.md

Host Migration to OpenTofu

Overview

Migrate all existing hosts (provisioned manually before the OpenTofu pipeline) into the new OpenTofu-managed provisioning workflow. Hosts are categorized by their state requirements: stateless hosts are simply recreated, stateful hosts require backup and restore, and some hosts are decommissioned or deferred.

Current State

Hosts already managed by OpenTofu: vault01, testvm01, vaulttest01

Hosts to migrate:

Host	Category	Notes
ns1	Stateless	Primary DNS, recreate
ns2	Stateless	Secondary DNS, recreate
nix-cache01	Stateless	Binary cache, recreate
http-proxy	Stateless	Reverse proxy, recreate
nats1	Stateless	Messaging, recreate
auth01	Decommission	No longer in use
ha1	Stateful	Home Assistant + Zigbee2MQTT + Mosquitto
monitoring01	Stateful	Prometheus, Grafana, Loki
jelly01	Stateful	Jellyfin metadata, watch history, config
pgdb1	Stateful	PostgreSQL databases
jump	Decommission	No longer needed
ca	Deferred	Pending Phase 4c PKI migration to OpenBao

Phase 1: Backup Preparation

Before migrating any stateful host, ensure restic backups are in place and verified.

1a. Expand monitoring01 Grafana Backup

The existing backup only covers /var/lib/grafana/plugins and a sqlite dump of grafana.db. Expand to back up all of /var/lib/grafana/ to capture config directory and any other state.

1b. Add Jellyfin Backup to jelly01

No backup currently exists. Add a restic backup job for /var/lib/jellyfin/ which contains:

• config/ — server settings, library configuration
• data/ — user watch history, playback state, library metadata

Media files are on the NAS (nas.home.2rjus.net:/mnt/hdd-pool/media) and do not need backup. The cache directory (/var/cache/jellyfin/) does not need backup — it regenerates.

1c. Add PostgreSQL Backup to pgdb1

No backup currently exists. Add a restic backup job with a pg_dumpall pre-hook to capture all databases and roles. The dump should be piped through restic's stdin backup (similar to the Grafana DB dump pattern on monitoring01).

1d. Verify Existing ha1 Backup

ha1 already backs up /var/lib/hass, /var/lib/zigbee2mqtt, /var/lib/mosquitto. Verify these backups are current and restorable before proceeding with migration.

1e. Verify All Backups

After adding/expanding backup jobs:

1. Trigger a manual backup run on each host
2. Verify backup integrity with restic check
3. Test a restore to a temporary location to confirm data is recoverable

Phase 2: Declare pgdb1 Databases in Nix

Before migrating pgdb1, audit the manually-created databases and users on the running instance, then declare them in the Nix configuration using ensureDatabases and ensureUsers. This makes the PostgreSQL setup reproducible on the new host.

Steps:

1. SSH to pgdb1, run \l and \du in psql to list databases and roles
2. Add ensureDatabases and ensureUsers to services/postgres/postgres.nix
3. Document any non-default PostgreSQL settings or extensions per database

After reprovisioning, the databases will be created by NixOS, and data restored from the pg_dumpall backup.

Phase 3: Stateless Host Migration

These hosts have no meaningful state and can be recreated fresh. For each host:

1. Add the host definition to terraform/vms.tf (using create-host or manually)
2. Commit and push to master
3. Run tofu apply to provision the new VM
4. Wait for bootstrap to complete (VM pulls config from master and reboots)
5. Verify the host is functional
6. Decommission the old VM in Proxmox

Migration Order

Migrate stateless hosts in an order that minimizes disruption:

1. nix-cache01 — low risk, no downstream dependencies during migration
2. nats1 — low risk, verify no persistent JetStream streams first
3. http-proxy — brief disruption to proxied services, migrate during low-traffic window
4. ns1, ns2 — migrate one at a time, verify DNS resolution between each

For ns1/ns2: migrate ns2 first (secondary), verify AXFR works, then migrate ns1. All hosts use both ns1 and ns2 as resolvers, so one being down briefly is tolerable.

Phase 4: Stateful Host Migration

For each stateful host, the procedure is:

1. Trigger a final restic backup
2. Stop services on the old host (to prevent state drift during migration)
3. Provision the new VM via tofu apply
4. Wait for bootstrap to complete
5. Stop the relevant services on the new host
6. Restore data from restic backup
7. Start services and verify functionality
8. Decommission the old VM

4a. pgdb1

1. Run final pg_dumpall backup via restic
2. Stop PostgreSQL on the old host
3. Provision new pgdb1 via OpenTofu
4. After bootstrap, NixOS creates the declared databases/users
5. Restore data with pg_restore or psql < dumpall.sql
6. Verify database connectivity from gunter (10.69.30.105)
7. Decommission old VM

4b. monitoring01

1. Run final Grafana backup
2. Provision new monitoring01 via OpenTofu
3. After bootstrap, restore /var/lib/grafana/ from restic
4. Restart Grafana, verify dashboards and datasources are intact
5. Prometheus and Loki start fresh with empty data (acceptable)
6. Verify all scrape targets are being collected
7. Decommission old VM

4c. jelly01

1. Run final Jellyfin backup
2. Provision new jelly01 via OpenTofu
3. After bootstrap, restore /var/lib/jellyfin/ from restic
4. Verify NFS mount to NAS is working
5. Start Jellyfin, verify watch history and library metadata are present
6. Decommission old VM

4d. ha1

1. Verify latest restic backup is current
2. Stop Home Assistant, Zigbee2MQTT, and Mosquitto on old host
3. Provision new ha1 via OpenTofu
4. After bootstrap, restore /var/lib/hass, /var/lib/zigbee2mqtt, /var/lib/mosquitto
5. Start services, verify Home Assistant is functional
6. Verify Zigbee devices are still paired and communicating
7. Decommission old VM

Note: ha1 currently has 2 GB RAM, which is consistently tight. Average memory usage has climbed from ~57% (30-day avg) to ~70% currently, with a 30-day low of only 187 MB free. Consider increasing to 4 GB when reprovisioning to allow headroom for additional integrations.

Note: ha1 is the highest-risk migration due to Zigbee device pairings. The Zigbee coordinator state in /var/lib/zigbee2mqtt should preserve pairings, but verify on a non-critical time window.

USB Passthrough: The ha1 VM has a USB device passed through from the Proxmox hypervisor (the Zigbee coordinator). The new VM must be configured with the same USB passthrough in OpenTofu/Proxmox. Verify the USB device ID on the hypervisor and add the appropriate usb block to the VM definition in terraform/vms.tf. The USB device must be passed through before starting Zigbee2MQTT on the new host.

Phase 5: Decommission jump and auth01 Hosts

jump

1. Verify nothing depends on the jump host (no SSH proxy configs pointing to it, etc.)
2. Remove host configuration from hosts/jump/
3. Remove from flake.nix
4. Remove any secrets in secrets/jump/
5. Remove from .sops.yaml
6. Destroy the VM in Proxmox
7. Commit cleanup

auth01

1. Remove host configuration from hosts/auth01/
2. Remove from flake.nix
3. Remove any secrets in secrets/auth01/
4. Remove from .sops.yaml
5. Remove services/authelia/ and services/lldap/ (only used by auth01)
6. Destroy the VM in Proxmox
7. Commit cleanup

Phase 6: Decommission ca Host (Deferred)

Deferred until Phase 4c (PKI migration to OpenBao) is complete. Once all hosts use the OpenBao ACME endpoint for certificates, the step-ca host can be decommissioned following the same cleanup steps as the jump host.

Phase 7: Remove sops-nix

Once ca is decommissioned (Phase 6), sops-nix is no longer used by any host. Remove all remnants:

• sops-nix input from flake.nix and flake.lock
• sops-nix.nixosModules.sops from all host module lists in flake.nix
• inherit sops-nix from all specialArgs in flake.nix
• system/sops.nix and its import in system/default.nix
• .sops.yaml
• secrets/ directory
• All sops.secrets.* declarations in services/ca/, services/authelia/, services/lldap/
• Template scripts that generate age keys for sops (hosts/template/scripts.nix, hosts/template2/scripts.nix)

See docs/plans/completed/sops-to-openbao-migration.md for full context.

Notes

• Each host migration should be done individually, not in bulk, to limit blast radius
• Keep the old VM running until the new one is verified — do not destroy prematurely
• The old VMs use IPs that the new VMs need, so the old VM must be shut down before the new one is provisioned (or use a temporary IP and swap after verification)
• Stateful migrations should be done during low-usage windows
• After all migrations are complete, the only hosts not in OpenTofu will be ca (deferred)
• Since many hosts are being recreated, this is a good opportunity to establish consistent hostname naming conventions before provisioning the new VMs. Current naming is inconsistent (e.g. ns1 vs nix-cache01, ha1 vs auth01, pgdb1 vs http-proxy). Decide on a convention before starting migrations — e.g. whether to always use numeric suffixes, a consistent format like service-NN, role-based vs function-based names, etc.