nixos-servers/docs/plans/truenas-migration.md

TrueNAS Migration Planning

Current State

Hardware

• CPU: AMD Ryzen 5 5600G with Radeon Graphics
• RAM: 32GB
• Network: 10GbE (mlxen0)
• Software: TrueNAS-13.0-U6.1 (Core)

Storage Status

hdd-pool: 29.1TB total, 28.4TB used, 658GB free (97% capacity) ⚠️

• mirror-0: 2x Seagate ST16000NE000 16TB HDD (16TB usable)
• mirror-1: 2x WD WD80EFBX 8TB HDD (8TB usable)
• mirror-2: 2x Seagate ST8000VN004 8TB HDD (8TB usable)

Goal

Expand storage capacity for the main hdd-pool. Since we need to add disks anyway, also evaluating whether to upgrade or replace the entire system.

Decisions

Migration Approach: Option 3 - Migrate to NixOS

Decision: Replace TrueNAS with NixOS bare metal installation

Rationale:

• Aligns with existing infrastructure (16+ NixOS hosts already managed in this repo)
• Declarative configuration fits homelab philosophy
• Automatic monitoring/logging integration (Prometheus + Promtail)
• Auto-upgrades via same mechanism as other hosts
• SOPS secrets management integration
• TrueNAS-specific features (WebGUI, jails) not heavily utilized

Service migration:

• radarr/sonarr: Native NixOS services (services.radarr, services.sonarr)
• restic-rest: services.restic.server
• nzbget: NixOS service or OCI container
• NFS exports: services.nfs.server

Filesystem: Keep ZFS

Decision: Keep existing ZFS pool, import on NixOS

Rationale:

• No data migration needed: Existing ZFS pool can be imported directly on NixOS
• Proven reliability: Pool has been running reliably on TrueNAS
• NixOS ZFS support: Well-supported, declarative configuration via boot.zfs and services.zfs
• BTRFS RAID5/6 unreliable: Research showed BTRFS RAID5/6 write hole is still unresolved
• BTRFS RAID1 wasteful: With mixed disk sizes, RAID1 wastes significant capacity vs ZFS mirrors
• Checksumming, snapshots, compression (lz4/zstd) all available

Hardware: Keep Existing + Add Disks

Decision: Retain current hardware, expand disk capacity

Hardware to keep:

• AMD Ryzen 5 5600G (sufficient for NAS workload)
• 32GB RAM (adequate)
• 10GbE network interface
• Chassis

Storage architecture:

hdd-pool (ZFS mirrors):

• Current: 3 mirror vdevs (2x16TB + 2x8TB + 2x8TB) = 32TB usable
• Add: mirror-3 with 2x 24TB = +24TB usable
• Total after expansion: ~56TB usable
• Use: Media, downloads, backups, non-critical data

Disk Purchase Decision

Decision: 2x 24TB drives (ordered, arriving 2026-02-21)

Migration Strategy

High-Level Plan

1. Expand ZFS pool (on TrueNAS):

   • Install 2x 24TB drives (may need new drive trays - order from abroad if needed)
   • If chassis space is limited, temporarily replace the two oldest 8TB drives (da0/ada4)
   • Add as mirror-3 vdev to hdd-pool
   • Verify pool health and resilver completes
   • Check SMART data on old 8TB drives (all healthy as of 2026-02-20, no reallocated sectors)
   • Burn-in: at minimum short + long SMART test before adding to pool

2. Prepare NixOS configuration:

   • Create host configuration (hosts/nas1/ or similar)
   • Configure ZFS pool import (boot.zfs.extraPools)
   • Set up services: radarr, sonarr, nzbget, restic-rest, NFS
   • Configure monitoring (node-exporter, promtail, smartctl-exporter)

3. Install NixOS:

   • zfs export hdd-pool on TrueNAS before shutdown (clean export)
   • Wipe TrueNAS boot-pool SSDs, set up as mdadm RAID1 for NixOS root
   • Install NixOS on mdadm mirror (keeps boot path ZFS-independent)
   • Import hdd-pool via boot.zfs.extraPools
   • Verify all datasets mount correctly

4. Service migration:

   • Configure NixOS services to use ZFS dataset paths
   • Update NFS exports
   • Test from consuming hosts

5. Cutover:

   • Update DNS/client mounts if IP changes
   • Verify monitoring integration
   • Decommission TrueNAS

Post-Expansion: Vdev Rebalancing

ZFS has no built-in rebalance command. After adding the new 24TB vdev, ZFS will write new data preferentially to it (most free space), leaving old vdevs packed at ~97%. This is suboptimal but not urgent once overall pool usage drops to ~50%.

To gradually rebalance, rewrite files in place so ZFS redistributes blocks across all vdevs proportional to free space:

# Rewrite files individually (spreads blocks across all vdevs)
find /pool/dataset -type f -exec sh -c '
  for f; do cp "$f" "$f.rebal" && mv "$f.rebal" "$f"; done
' _ {} +

Avoid zfs send/recv for large datasets (e.g. 20TB) as this would concentrate data on the emptiest vdev rather than spreading it evenly.

Recommendation: Do this after NixOS migration is stable. Not urgent - the pool will function fine with uneven distribution, just slightly suboptimal for performance.

Migration Advantages

• No data migration: ZFS pool imported directly, no copying terabytes of data
• Low risk: Pool expansion done on stable TrueNAS before OS swap
• Reversible: Can boot back to TrueNAS if NixOS has issues (ZFS pool is OS-independent)
• Quick cutover: Once NixOS config is ready, the OS swap is fast

Next Steps

1. Decide on disk size - 2x 24TB ordered
2. Install drives and add mirror vdev to ZFS pool
3. Check SMART data on 8TB drives - decide whether to keep or retire
4. Design NixOS host configuration (hosts/nas1/)
5. Document NFS export mapping (current -> new)
6. Plan NixOS installation and cutover

Open Questions

• Hostname for new NAS host? (nas1? storage1?)
• IP address allocation (keep 10.69.12.50 or new IP?)
• Boot drive: Reuse TrueNAS boot-pool SSDs as mdadm RAID1 for NixOS root (no ZFS on boot path)
• Retire old 8TB drives? (SMART looks healthy, keep unless chassis space is needed)
• Drive trays: do new 24TB drives fit, or order trays from abroad?
• Timeline/maintenance window for NixOS swap?