163 lines
7.0 KiB
Markdown
163 lines
7.0 KiB
Markdown
# NixOS Router — Replace EdgeRouter
|
|
|
|
Replace the aging Ubiquiti EdgeRouter (gw, 10.69.10.1) with a NixOS-based router.
|
|
The EdgeRouter is suspected to be a throughput bottleneck. A NixOS router integrates
|
|
naturally with the existing fleet: same config management, same monitoring pipeline,
|
|
same deployment workflow.
|
|
|
|
## Goals
|
|
|
|
- Eliminate the EdgeRouter throughput bottleneck
|
|
- Full integration with existing monitoring (node-exporter, promtail, Prometheus, Loki)
|
|
- Declarative firewall and routing config managed in the flake
|
|
- Inter-VLAN routing for all existing subnets
|
|
- DHCP server for client subnets
|
|
- NetFlow/traffic accounting for future ntopng integration
|
|
- Foundation for WireGuard remote access (see remote-access.md)
|
|
|
|
## Current Network Topology
|
|
|
|
**Subnets (known VLANs):**
|
|
| VLAN/Subnet | Purpose | Notable hosts |
|
|
|----------------|------------------|----------------------------------------|
|
|
| 10.69.10.0/24 | Gateway | gw (10.69.10.1) |
|
|
| 10.69.12.0/24 | Core services | nas, pve1, arr jails, restic |
|
|
| 10.69.13.0/24 | Infrastructure | All NixOS servers (static IPs) |
|
|
| 10.69.22.0/24 | WLAN | unifi-ctrl |
|
|
| 10.69.30.0/24 | Workstations | gunter |
|
|
| 10.69.31.0/24 | Media | media |
|
|
| 10.69.99.0/24 | Management | sw1 (MikroTik CRS326-24G-2S+) |
|
|
|
|
**DNS:** ns1 (10.69.13.5) and ns2 (10.69.13.6) handle all resolution. Upstream is
|
|
Cloudflare/Google over DoT via Unbound.
|
|
|
|
**Switch:** MikroTik CRS326-24G-2S+ — L2 switching with VLAN trunking. Capable of
|
|
L3 routing via RouterOS but not ideal for sustained routing throughput.
|
|
|
|
## Hardware
|
|
|
|
Needs a small x86 box with:
|
|
- At least 2 NICs (WAN + LAN trunk). Dual 2.5GbE preferred.
|
|
- Enough CPU for nftables NAT at line rate (any modern x86 is fine)
|
|
- 4-8 GB RAM (plenty for routing + DHCP + NetFlow accounting)
|
|
- Low power consumption, fanless preferred for always-on use
|
|
|
|
Candidates:
|
|
- Topton / CWWK mini PC with dual/quad Intel 2.5GbE (~100-150 EUR)
|
|
- Protectli Vault (more expensive, ~200-300 EUR, proven in pfSense/OPNsense community)
|
|
- Any mini PC with one onboard NIC + one USB 2.5GbE adapter (cheapest, less ideal)
|
|
|
|
The LAN port would carry a VLAN trunk to the MikroTik switch, with sub-interfaces
|
|
for each VLAN. WAN port connects to the ISP uplink.
|
|
|
|
## NixOS Configuration
|
|
|
|
### Stability Policy
|
|
|
|
The router is treated differently from the rest of the fleet:
|
|
- **No auto-upgrade** — `system.autoUpgrade.enable = false`
|
|
- **No homelab-deploy listener** — `homelab.deploy.enable = false`
|
|
- **Manual updates only** — update every few months, test-build first
|
|
- **Use `nixos-rebuild boot`** — changes take effect on next deliberate reboot
|
|
- **Tier: prod, priority: high** — alerts treated with highest priority
|
|
|
|
### Core Services
|
|
|
|
**Routing & NAT:**
|
|
- `systemd-networkd` for all interface config (consistent with rest of fleet)
|
|
- VLAN sub-interfaces on the LAN trunk (one per subnet)
|
|
- `networking.nftables` for stateful firewall and NAT
|
|
- IP forwarding enabled (`net.ipv4.ip_forward = 1`)
|
|
- Masquerade outbound traffic on WAN interface
|
|
|
|
**DHCP:**
|
|
- Kea or dnsmasq for DHCP on client subnets (WLAN, workstations, media)
|
|
- Infrastructure subnet (10.69.13.0/24) stays static — no DHCP needed
|
|
- Static leases for known devices
|
|
|
|
**Firewall (nftables):**
|
|
- Default deny between VLANs
|
|
- Explicit allow rules for known cross-VLAN traffic:
|
|
- All subnets → ns1/ns2 (DNS)
|
|
- All subnets → monitoring01 (metrics/logs)
|
|
- Infrastructure → all (management access)
|
|
- Workstations → media, core services
|
|
- NAT masquerade on WAN
|
|
- Rate limiting on WAN-facing services
|
|
|
|
**Traffic Accounting:**
|
|
- nftables flow accounting or softflowd for NetFlow export
|
|
- Export to future ntopng instance (see new-services.md)
|
|
|
|
### Monitoring Integration
|
|
|
|
Since this is a NixOS host in the flake, it gets the standard monitoring stack for free:
|
|
- node-exporter for system metrics (CPU, memory, NIC throughput per interface)
|
|
- promtail shipping logs to Loki
|
|
- Prometheus scrape target auto-registration
|
|
- Alertmanager alerts for host-down, high CPU, etc.
|
|
|
|
Additional router-specific monitoring:
|
|
- Per-VLAN interface traffic metrics via node-exporter (automatic for all interfaces)
|
|
- NAT connection tracking table size
|
|
- WAN uplink status and throughput
|
|
- DHCP lease metrics (if Kea, it has a Prometheus exporter)
|
|
|
|
This is a significant advantage over the EdgeRouter — full observability through
|
|
the existing Grafana dashboards and Loki log search, debuggable via the monitoring
|
|
MCP tools.
|
|
|
|
### WireGuard Integration
|
|
|
|
The remote access plan (remote-access.md) currently proposes a separate `extgw01`
|
|
gateway host. With a NixOS router, there's a decision to make:
|
|
|
|
**Option A:** WireGuard terminates on the router itself. Simplest topology — the
|
|
router is already the gateway, so VPN traffic doesn't need extra hops or firewall
|
|
rules. But adds complexity to the router, which should stay simple.
|
|
|
|
**Option B:** Keep extgw01 as a separate host (original plan). Router just routes
|
|
traffic to it. Better separation of concerns, router stays minimal.
|
|
|
|
Recommendation: Start with option B (keep it separate). The router should do routing
|
|
and nothing else. WireGuard can move to the router later if extgw01 feels redundant.
|
|
|
|
## Migration Plan
|
|
|
|
### Phase 1: Build and lab test
|
|
- Acquire hardware
|
|
- Create host config in the flake (routing, NAT, DHCP, firewall)
|
|
- Test-build on workstation: `nix build .#nixosConfigurations.router01.config.system.build.toplevel`
|
|
- Lab test with a temporary setup if possible (two NICs, isolated VLAN)
|
|
|
|
### Phase 2: Prepare cutover
|
|
- Pre-configure the MikroTik switch trunk port for the new router
|
|
- Document current EdgeRouter config (port forwarding, NAT rules, DHCP leases)
|
|
- Replicate all rules in the NixOS config
|
|
- Verify DNS, DHCP, and inter-VLAN routing work in test
|
|
|
|
### Phase 3: Cutover
|
|
- Schedule a maintenance window (brief downtime expected)
|
|
- Swap WAN cable from EdgeRouter to new router
|
|
- Swap LAN trunk from EdgeRouter to new router
|
|
- Verify connectivity from each VLAN
|
|
- Verify internet access, DNS resolution, inter-VLAN routing
|
|
- Monitor via Prometheus/Loki (immediately available since it's a fleet host)
|
|
|
|
### Phase 4: Decommission EdgeRouter
|
|
- Keep EdgeRouter available as fallback for a few weeks
|
|
- Remove `gw` entry from external-hosts.nix, replace with flake-managed host
|
|
- Update any references to 10.69.10.1 if the router IP changes
|
|
|
|
## Open Questions
|
|
|
|
- **Router IP:** Keep 10.69.10.1 or move to a different address? Each VLAN
|
|
sub-interface needs an IP (the gateway address for that subnet).
|
|
- **ISP uplink:** What type of WAN connection? PPPoE, DHCP, static IP?
|
|
- **Port forwarding:** What ports are currently forwarded on the EdgeRouter?
|
|
These need to be replicated in nftables.
|
|
- **DHCP scope:** Which subnets currently get DHCP from the EdgeRouter vs
|
|
other sources (UniFi controller for WLAN?)?
|
|
- **UPnP/NAT-PMP:** Needed for any devices? (gaming consoles, etc.)
|
|
- **Hardware preference:** Fanless mini PC budget and preferred vendor?
|