nixos-servers/docs/plans/truenas-migration.md

# TrueNAS Migration Planning

## Current State

### Hardware
- CPU: AMD Ryzen 5 5600G with Radeon Graphics
- RAM: 32GB
- Network: 10GbE (mlxen0)
- Software: TrueNAS-13.0-U6.1 (Core)

### Storage Status

**hdd-pool**: 29.1TB total, **28.4TB used, 658GB free (97% capacity)** ⚠️
- mirror-0: 2x Seagate ST16000NE000 16TB HDD (16TB usable)
- mirror-1: 2x WD WD80EFBX 8TB HDD (8TB usable)
- mirror-2: 2x Seagate ST8000VN004 8TB HDD (8TB usable)

## Goal

Expand storage capacity for the main hdd-pool. Since we need to add disks anyway, also evaluating whether to upgrade or replace the entire system.

## Decisions

### Migration Approach: Option 3 - Migrate to NixOS

**Decision**: Replace TrueNAS with NixOS bare metal installation

**Rationale**:
- Aligns with existing infrastructure (16+ NixOS hosts already managed in this repo)
- Declarative configuration fits homelab philosophy
- Automatic monitoring/logging integration (Prometheus + Promtail)
- Auto-upgrades via same mechanism as other hosts
- SOPS secrets management integration
- TrueNAS-specific features (WebGUI, jails) not heavily utilized

**Service migration**:
- radarr/sonarr: Native NixOS services (`services.radarr`, `services.sonarr`)
- restic-rest: `services.restic.server`
- nzbget: NixOS service or OCI container
- NFS exports: `services.nfs.server`

### Filesystem: Keep ZFS

**Decision**: Keep existing ZFS pool, import on NixOS

**Rationale**:
- **No data migration needed**: Existing ZFS pool can be imported directly on NixOS
- **Proven reliability**: Pool has been running reliably on TrueNAS
- **NixOS ZFS support**: Well-supported, declarative configuration via `boot.zfs` and `services.zfs`
- **BTRFS RAID5/6 unreliable**: Research showed BTRFS RAID5/6 write hole is still unresolved
- **BTRFS RAID1 wasteful**: With mixed disk sizes, RAID1 wastes significant capacity vs ZFS mirrors
- Checksumming, snapshots, compression (lz4/zstd) all available

### Hardware: Keep Existing + Add Disks

**Decision**: Retain current hardware, expand disk capacity

**Hardware to keep**:
- AMD Ryzen 5 5600G (sufficient for NAS workload)
- 32GB RAM (adequate)
- 10GbE network interface
- Chassis

**Storage architecture**:

**hdd-pool** (ZFS mirrors):
- Current: 3 mirror vdevs (2x16TB + 2x8TB + 2x8TB) = 32TB usable
- Add: mirror-3 with 2x 24TB = +24TB usable
- Total after expansion: ~56TB usable
- Use: Media, downloads, backups, non-critical data

### Disk Purchase Decision

**Decision**: 2x 24TB drives (ordered, arriving 2026-02-21)

## Migration Strategy

### High-Level Plan

1. **Expand ZFS pool** (on TrueNAS):
   - Install 2x 24TB drives (may need new drive trays - order from abroad if needed)
   - If chassis space is limited, temporarily replace the two oldest 8TB drives (da0/ada4)
   - Add as mirror-3 vdev to hdd-pool
   - Verify pool health and resilver completes
   - Check SMART data on old 8TB drives (all healthy as of 2026-02-20, no reallocated sectors)
   - Burn-in: at minimum short + long SMART test before adding to pool

2. **Prepare NixOS configuration**:
   - Create host configuration (`hosts/nas1/` or similar)
   - Configure ZFS pool import (`boot.zfs.extraPools`)
   - Set up services: radarr, sonarr, nzbget, restic-rest, NFS
   - Configure monitoring (node-exporter, promtail, smartctl-exporter)

3. **Install NixOS**:
   - `zfs export hdd-pool` on TrueNAS before shutdown (clean export)
   - Wipe TrueNAS boot-pool SSDs, set up as mdadm RAID1 for NixOS root
   - Install NixOS on mdadm mirror (keeps boot path ZFS-independent)
   - Import hdd-pool via `boot.zfs.extraPools`
   - Verify all datasets mount correctly

4. **Service migration**:
   - Configure NixOS services to use ZFS dataset paths
   - Update NFS exports
   - Test from consuming hosts

5. **Cutover**:
   - Update DNS/client mounts if IP changes
   - Verify monitoring integration
   - Decommission TrueNAS

### Post-Expansion: Vdev Rebalancing

ZFS has no built-in rebalance command. After adding the new 24TB vdev, ZFS will
write new data preferentially to it (most free space), leaving old vdevs packed
at ~97%. This is suboptimal but not urgent once overall pool usage drops to ~50%.

To gradually rebalance, rewrite files in place so ZFS redistributes blocks across
all vdevs proportional to free space:

```bash
# Rewrite files individually (spreads blocks across all vdevs)
find /pool/dataset -type f -exec sh -c '
  for f; do cp "$f" "$f.rebal" && mv "$f.rebal" "$f"; done
' _ {} +
```

Avoid `zfs send/recv` for large datasets (e.g. 20TB) as this would concentrate
data on the emptiest vdev rather than spreading it evenly.

**Recommendation**: Do this after NixOS migration is stable. Not urgent - the pool
will function fine with uneven distribution, just slightly suboptimal for performance.

### Migration Advantages

- **No data migration**: ZFS pool imported directly, no copying terabytes of data
- **Low risk**: Pool expansion done on stable TrueNAS before OS swap
- **Reversible**: Can boot back to TrueNAS if NixOS has issues (ZFS pool is OS-independent)
- **Quick cutover**: Once NixOS config is ready, the OS swap is fast

## Next Steps

1. ~~Decide on disk size~~ - 2x 24TB ordered
2. Install drives and add mirror vdev to ZFS pool
3. Check SMART data on 8TB drives - decide whether to keep or retire
4. Design NixOS host configuration (`hosts/nas1/`)
5. Document NFS export mapping (current -> new)
6. Plan NixOS installation and cutover

## Open Questions

- [ ] Hostname for new NAS host? (nas1? storage1?)
- [ ] IP address allocation (keep 10.69.12.50 or new IP?)
- [x] Boot drive: Reuse TrueNAS boot-pool SSDs as mdadm RAID1 for NixOS root (no ZFS on boot path)
- [ ] Retire old 8TB drives? (SMART looks healthy, keep unless chassis space is needed)
- [ ] Drive trays: do new 24TB drives fit, or order trays from abroad?
- [ ] Timeline/maintenance window for NixOS swap?