nixos-servers/docs/plans/nixos-hypervisor.md

NixOS Hypervisor

Overview

Experiment with running a NixOS-based hypervisor as an alternative/complement to the current Proxmox setup. Goal is better homelab integration — declarative config, monitoring, auto-updates — while retaining the ability to run VMs with a Terraform-like workflow.

Motivation

• Proxmox works but doesn't integrate with the NixOS-managed homelab (no monitoring, no auto-updates, no vault, no declarative config)
• The PN51 units (once stable) are good candidates for experimentation — test-tier, plenty of RAM (32-64GB), 8C/16T
• Long-term: could reduce reliance on Proxmox or provide a secondary hypervisor pool
• VM migration: Currently all VMs (including both nameservers) run on a single Proxmox host. Being able to migrate VMs between hypervisors would allow rebooting a host for kernel updates without downtime for critical services like DNS.

Hardware Candidates

	pn01	pn02
CPU	Ryzen 7 5700U (8C/16T)	Ryzen 7 5700U (8C/16T)
RAM	64GB (2x32GB)	32GB (1x32GB, second slot available)
Storage	1TB NVMe	1TB SATA SSD (NVMe planned)
Status	Stability testing	Stability testing

Options

Option 1: Incus

Fork of LXD (after Canonical made LXD proprietary). Supports both containers (LXC) and VMs (QEMU/KVM).

NixOS integration:

• virtualisation.incus.enable module in nixpkgs
• Manages storage pools, networks, and instances
• REST API for automation
• CLI tool (incus) for management

Terraform integration:

• lxd provider works with Incus (API-compatible)
• Dedicated incus Terraform provider also exists
• Can define VMs/containers in OpenTofu, similar to current Proxmox workflow

Migration:

• Built-in live and offline migration via incus move <instance> --target <host>
• Clustering makes hosts aware of each other — migration is a first-class operation
• Shared storage (NFS, Ceph) or Incus can transfer storage during migration
• Stateful stop-and-move also supported for offline migration

Pros:

• Supports both containers and VMs
• REST API + CLI for automation
• Built-in clustering and migration — closest to Proxmox experience
• Good NixOS module support
• Image-based workflow (can build NixOS images and import)
• Active development and community

Cons:

• Another abstraction layer on top of QEMU/KVM
• Less mature Terraform provider than libvirt
• Container networking can be complex
• NixOS guests in Incus VMs need some setup

Option 2: libvirt/QEMU

Standard Linux virtualization stack. Thin wrapper around QEMU/KVM.

NixOS integration:

• virtualisation.libvirtd.enable module in nixpkgs
• Mature and well-tested
• virsh CLI for management

Terraform integration:

• dmacvicar/libvirt provider — mature, well-maintained
• Supports cloud-init, volume management, network config
• Very similar workflow to current Proxmox+OpenTofu setup
• Can reuse cloud-init patterns from existing terraform/ config

Migration:

• Supports live and offline migration via virsh migrate
• Requires shared storage (NFS, Ceph, or similar) for live migration
• Requires matching CPU models between hosts (or CPU model masking)
• Works but is manual — no cluster awareness, must specify target URI
• No built-in orchestration for multi-host scenarios

Pros:

• Closest to current Proxmox+Terraform workflow
• Most mature Terraform provider
• Minimal abstraction — direct QEMU/KVM management
• Well-understood, massive community
• Cloud-init works identically to Proxmox workflow
• Can reuse existing template-building patterns

Cons:

• VMs only (no containers without adding LXC separately)
• No built-in REST API (would need to expose libvirt socket)
• No web UI without adding cockpit or virt-manager
• Migration works but requires manual setup — no clustering, no orchestration
• Less feature-rich than Incus for multi-host scenarios

Option 3: microvm.nix

NixOS-native microVM framework. VMs defined as NixOS modules in the host's flake.

NixOS integration:

• VMs are NixOS configurations in the same flake
• Supports multiple backends: cloud-hypervisor, QEMU, firecracker, kvmtool
• Lightweight — shares host's nix store with guests via virtiofs
• Declarative network, storage, and resource allocation

Terraform integration:

• None — everything is defined in Nix
• Fundamentally different workflow from current Proxmox+Terraform approach

Pros:

• Most NixOS-native approach
• VMs defined right alongside host configs in this repo
• Very lightweight — fast boot, minimal overhead
• Shares nix store with host (no duplicate packages)
• No cloud-init needed — guest config is part of the flake

Migration:

• No migration support — VMs are tied to the host's NixOS config
• Moving a VM means rebuilding it on another host

Cons:

• Very niche, smaller community
• Different mental model from current workflow
• Only NixOS guests (no Ubuntu, FreeBSD, etc.)
• No Terraform integration
• No migration support
• Less isolation than full QEMU VMs
• Would need to learn a new deployment pattern

Comparison

Criteria	Incus	libvirt	microvm.nix
Workflow similarity	Medium	High	Low
Terraform support	Yes (lxd/incus provider)	Yes (mature provider)	No
NixOS module	Yes	Yes	Yes
Containers + VMs	Both	VMs only	VMs only
Non-NixOS guests	Yes	Yes	No
Live migration	Built-in (first-class)	Yes (manual setup)	No
Offline migration	Built-in	Yes (manual setup)	No (rebuild)
Clustering	Built-in	Manual	No
Learning curve	Medium	Low	Medium
Community/maturity	Growing	Very mature	Niche
Overhead	Low	Minimal	Minimal

Recommendation

Start with Incus. Migration and clustering are key requirements:

• Built-in clustering makes two PN51s a proper hypervisor pool
• Live and offline migration are first-class operations, similar to Proxmox
• Can move VMs between hosts for maintenance (kernel updates, hardware work) without downtime
• Supports both containers and VMs — flexibility for future use
• Terraform provider exists (less mature than libvirt's, but functional)
• REST API enables automation beyond what Terraform covers

libvirt could achieve similar results but requires significantly more manual setup for migration and has no clustering awareness. For a two-node setup where migration is a priority, Incus provides much more out of the box.

microvm.nix is off the table given the migration requirement.

Implementation Plan

Phase 1: Single-Node Setup (on one PN51)

1. Enable virtualisation.incus on pn01 (or whichever is stable)
2. Initialize Incus (incus admin init) — configure storage pool (local NVMe) and network bridge
3. Configure bridge networking for VM traffic on VLAN 12
4. Build a NixOS VM image and import it into Incus
5. Create a test VM manually with incus launch to validate the setup

Phase 2: Two-Node Cluster (PN51s only)

1. Enable Incus on the second PN51
2. Form a cluster between both nodes
3. Configure shared storage (NFS from NAS, or Ceph if warranted)
4. Test offline migration: incus move <vm> --target <other-node>
5. Test live migration with shared storage
6. CPU compatibility is not an issue here — both nodes have identical Ryzen 7 5700U CPUs

Phase 3: Terraform Integration

1. Add Incus Terraform provider to terraform/
2. Define a test VM in OpenTofu (cloud-init, static IP, vault provisioning)
3. Verify the full pipeline: tofu apply -> VM boots -> cloud-init -> vault credentials -> NixOS rebuild
4. Compare workflow with existing Proxmox pipeline

Phase 4: Evaluate and Expand

• Is the workflow comparable to Proxmox?
• Migration reliability — does live migration work cleanly?
• Performance overhead acceptable on Ryzen 5700U?
• Worth migrating some test-tier VMs from Proxmox?
• Could ns1/ns2 run on separate Incus nodes instead of the single Proxmox host?

Phase 5: Proxmox Replacement (optional)

If Incus works well on the PN51s, consider replacing Proxmox entirely for a three-node cluster.

CPU compatibility for mixed cluster:

Node	CPU	Architecture	x86-64-v3
Proxmox host	AMD Ryzen 9 3900X (12C/24T)	Zen 2	Yes
pn01	AMD Ryzen 7 5700U (8C/16T)	Zen 3	Yes
pn02	AMD Ryzen 7 5700U (8C/16T)	Zen 3	Yes

All three CPUs are AMD and support x86-64-v3. The 3900X (Zen 2) is the oldest, so it defines the feature ceiling — but x86-64-v3 is well within its capabilities. VMs configured with x86-64-v3 can migrate freely between all three nodes.

Being all-AMD also avoids the trickier Intel/AMD cross-vendor migration edge cases (different CPUID layouts, virtualization extensions).

The 3900X (12C/24T) would be the most powerful node, making it the natural home for heavier workloads, with the PN51s (8C/16T each) handling lighter VMs or serving as migration targets during maintenance.

Steps:

1. Install NixOS + Incus on the Proxmox host (or a replacement machine)
2. Join it to the existing Incus cluster with x86-64-v3 CPU baseline
3. Migrate VMs from Proxmox to the Incus cluster
4. Decommission Proxmox

Prerequisites

• PN51 units pass stability testing (see pn51-stability.md)
• Decide which unit to use first (pn01 preferred — 64GB RAM, NVMe, currently more stable)

Open Questions

• How to handle VM storage? Local NVMe, NFS from NAS, or Ceph between the two nodes?
• Network topology: bridge on VLAN 12, or trunk multiple VLANs to the PN51?
• Should VMs be on the same VLAN as the hypervisor host, or separate?
• Incus clustering with only two nodes — any quorum issues? Three nodes (with Proxmox replacement) would solve this
• How to handle NixOS guest images? Build with nixos-generators, or use Incus image builder?
• What CPU does the current Proxmox host have? AMD Ryzen 9 3900X (Zen 2) — x86-64-v3 confirmed, all-AMD cluster
• If replacing Proxmox: migrate VMs first, or fresh start and rebuild?