homelab-deploy/design.md

homelab-deploy Design Document

A message-based deployment system for NixOS configurations using NATS for messaging. This binary runs in multiple modes to enable on-demand NixOS configuration updates across a fleet of hosts.

Overview

The homelab-deploy binary provides three operational modes:

1. Listener mode - Runs on each NixOS host as a systemd service, subscribing to NATS subjects and executing nixos-rebuild when deployment requests arrive
2. MCP mode - Runs as an MCP (Model Context Protocol) server, exposing deployment tools for AI assistants
3. CLI mode - Manual deployment commands for administrators

Architecture

┌─────────────┐                        ┌─────────────┐
│  MCP Tool   │  deploy.test.>         │  Admin CLI  │  deploy.test.> + deploy.prod.>
│             │────────────┐     ┌─────│             │
└─────────────┘            │     │     └─────────────┘
                           ▼     ▼
                      ┌──────────────┐
                      │ NATS Server  │
                      │  (authz)     │
                      └──────┬───────┘
                             │
           ┌─────────────────┼─────────────────┐
           │                 │                 │
           ▼                 ▼                 ▼
     ┌──────────┐      ┌──────────┐      ┌──────────┐
     │  host-a  │      │  host-b  │      │  host-c  │
     │ tier=test│      │ tier=prod│      │ tier=prod│
     └──────────┘      └──────────┘      └──────────┘

Repository Structure

homelab-deploy/
├── flake.nix           # Nix flake with Go package + NixOS module
├── go.mod
├── go.sum
├── cmd/
│   └── homelab-deploy/
│       └── main.go     # CLI entrypoint with subcommands
├── internal/
│   ├── listener/       # Listener mode logic
│   ├── mcp/            # MCP server mode logic
│   ├── nats/           # NATS client wrapper
│   └── deploy/         # Shared deployment execution logic
└── nixos/
    └── module.nix      # NixOS module for listener service

CLI Interface

# Listener mode (runs as systemd service on each host)
homelab-deploy listener \
  --hostname <hostname> \
  --tier <test|prod> \
  --nats-url nats://server:4222 \
  --nkey-file /path/to/listener.nkey \
  --flake-url <git+https://...> \
  [--role <role>] \
  [--timeout 600]

# MCP server mode (for AI assistants)
homelab-deploy mcp \
  --nats-url nats://server:4222 \
  --nkey-file /path/to/mcp.nkey \
  [--enable-admin --admin-nkey-file /path/to/admin.nkey]

# CLI commands for manual use
homelab-deploy deploy <hostname> \
  --nats-url nats://server:4222 \
  --nkey-file /path/to/deployer.nkey \
  [--branch <branch>] \
  [--action <switch|boot|test|dry-activate>]

homelab-deploy deploy \
  --tier <test|prod> \
  --all \
  --nats-url nats://server:4222 \
  --nkey-file /path/to/deployer.nkey \
  [--branch <branch>] \
  [--action <switch|boot|test|dry-activate>]

homelab-deploy deploy \
  --tier <test|prod> \
  --role <role> \
  --nats-url nats://server:4222 \
  --nkey-file /path/to/deployer.nkey \
  [--branch <branch>] \
  [--action <switch|boot|test|dry-activate>]

NATS Subject Structure

Subjects follow the pattern deploy.<tier>.<target>:

Subject Pattern	Description
deploy.<tier>.<hostname>	Deploy to specific host (e.g., deploy.prod.ns1)
deploy.<tier>.all	Deploy to all hosts in tier (e.g., deploy.test.all)
deploy.<tier>.role.<role>	Deploy to hosts with role in tier (e.g., deploy.prod.role.dns)
deploy.responses.<request-id>	Response subject for request/reply pattern

Listener Mode

Responsibilities

1. Connect to NATS using NKey authentication
2. Subscribe to subjects based on hostname, tier, and role
3. Validate incoming deployment requests
4. Execute nixos-rebuild with the specified parameters
5. Report status back via NATS reply subject

Subject Subscriptions

A listener subscribes to multiple subjects based on its configuration:

• deploy.<tier>.<hostname> - Direct messages to this host
• deploy.<tier>.all - Broadcast to all hosts in tier
• deploy.<tier>.role.<role> - Broadcast to hosts with matching role (only if role is configured)

Example: A host with hostname=ns1, tier=prod, role=dns subscribes to:

• deploy.prod.ns1
• deploy.prod.all
• deploy.prod.role.dns

Message Formats

Request message:

{
  "action": "switch",
  "revision": "master",
  "reply_to": "deploy.responses.abc123"
}

Field	Type	Required	Description
action	string	yes	One of: switch, boot, test, dry-activate
revision	string	yes	Git branch name or commit hash
reply_to	string	yes	Subject to publish responses to

Response message:

{
  "hostname": "ns1",
  "status": "completed",
  "error": null,
  "message": "Successfully switched to generation 42"
}

Field	Type	Description
hostname	string	The responding host's name
status	string	One of: accepted, rejected, started, completed, failed
error	string or null	Error code if status is rejected or failed
message	string	Human-readable details

Error codes:

• invalid_revision - The specified branch/commit does not exist
• invalid_action - The action is not recognized
• already_running - A deployment is already in progress on this host
• build_failed - nixos-rebuild exited with non-zero status
• timeout - Deployment exceeded the configured timeout

Request/Reply Flow

1. Deployer sends request with unique reply_to subject
2. Deployer subscribes to the reply_to subject before sending
3. Listener validates request:
   • Checks revision exists using git ls-remote
   • Checks no other deployment is running
4. Listener sends immediate response:
   • {"status": "rejected", ...} if validation fails, or
   • {"status": "started", ...} if deployment begins
5. If started, listener executes nixos-rebuild
6. Listener sends final response:
   • {"status": "completed", ...} on success, or
   • {"status": "failed", ...} on failure

Deployment Execution

The listener executes nixos-rebuild with the following command pattern:

nixos-rebuild <action> --flake <flake-url>?ref=<revision>#<hostname>

Where:

• <action> is one of: switch, boot, test, dry-activate
• <flake-url> is the configured git flake URL (e.g., git+https://git.example.com/user/nixos-configs.git)
• <revision> is the branch name or commit hash from the request
• <hostname> is the listener's configured hostname

Environment requirements:

• Must run as root (nixos-rebuild requires root)
• Nix must be configured with proper git credentials if the flake is private
• Network access to the git repository

Concurrency Control

Only one deployment may run at a time per host. The listener maintains a simple lock:

• Before starting a deployment, acquire lock
• If lock is held, reject with already_running error
• Release lock when deployment completes (success or failure)
• Lock should be in-memory (no persistence needed - restarts clear it)

Logging

All deployment events should be logged to stdout/stderr (captured by systemd journal):

• Request received (with subject, action, revision)
• Validation result
• Deployment start
• Deployment completion (with exit code)
• Any errors

This enables integration with log aggregation systems (e.g., Loki via Promtail).

MCP Mode

Purpose

Exposes deployment functionality as MCP tools for AI assistants (e.g., Claude Code).

Tools

Tool	Description	Parameters
deploy	Deploy to test-tier hosts	hostname or all, optional role, branch, action
deploy_admin	Deploy to any tier (requires --enable-admin)	tier, hostname or all, optional role, branch, action
list_hosts	List available deployment targets	tier (optional)

Tool Schemas

deploy:

{
  "name": "deploy",
  "description": "Deploy NixOS configuration to test-tier hosts",
  "inputSchema": {
    "type": "object",
    "properties": {
      "hostname": {
        "type": "string",
        "description": "Target hostname, or omit to use 'all' or 'role' targeting"
      },
      "all": {
        "type": "boolean",
        "description": "Deploy to all test-tier hosts"
      },
      "role": {
        "type": "string",
        "description": "Deploy to all test-tier hosts with this role"
      },
      "branch": {
        "type": "string",
        "description": "Git branch or commit to deploy (default: master)"
      },
      "action": {
        "type": "string",
        "enum": ["switch", "boot", "test", "dry-activate"],
        "description": "nixos-rebuild action (default: switch)"
      }
    }
  }
}

deploy_admin:

{
  "name": "deploy_admin",
  "description": "Deploy NixOS configuration to any host (admin access required)",
  "inputSchema": {
    "type": "object",
    "properties": {
      "tier": {
        "type": "string",
        "enum": ["test", "prod"],
        "description": "Target tier"
      },
      "hostname": {
        "type": "string",
        "description": "Target hostname, or omit to use 'all' or 'role' targeting"
      },
      "all": {
        "type": "boolean",
        "description": "Deploy to all hosts in tier"
      },
      "role": {
        "type": "string",
        "description": "Deploy to all hosts with this role in tier"
      },
      "branch": {
        "type": "string",
        "description": "Git branch or commit to deploy (default: master)"
      },
      "action": {
        "type": "string",
        "enum": ["switch", "boot", "test", "dry-activate"],
        "description": "nixos-rebuild action (default: switch)"
      }
    },
    "required": ["tier"]
  }
}

list_hosts:

{
  "name": "list_hosts",
  "description": "List available deployment targets",
  "inputSchema": {
    "type": "object",
    "properties": {
      "tier": {
        "type": "string",
        "enum": ["test", "prod"],
        "description": "Filter by tier (optional)"
      }
    }
  }
}

Security Layers

1. MCP flag: deploy_admin tool only registered when --enable-admin is passed
2. NATS authz: MCP credentials can only publish to authorized subjects
3. AI assistant permissions: The assistant's configuration can require confirmation for admin operations

Multi-Host Deployments

When deploying to multiple hosts (via all or role), the MCP should:

1. Publish the request to the appropriate broadcast subject
2. Collect responses from all responding hosts
3. Return aggregated results showing each host's status

Timeout handling:

• Set a reasonable timeout for collecting responses (e.g., 30 seconds after last response, or max 15 minutes)
• Return partial results if some hosts don't respond
• Indicate which hosts did not respond

Host Discovery

The list_hosts tool needs to know available hosts. Options:

1. Static configuration: Read from a config file or environment variable
2. NATS request: Publish to a discovery subject and collect responses from listeners

Recommend option 2: Listeners respond to deploy.discover with their metadata:

{
  "hostname": "ns1",
  "tier": "prod",
  "role": "dns"
}

NixOS Module

The NixOS module configures the listener as a systemd service.

Module Options

{
  options.services.homelab-deploy.listener = {
    enable = lib.mkEnableOption "homelab-deploy listener service";

    hostname = lib.mkOption {
      type = lib.types.str;
      description = "Hostname for this listener (used for NATS subject)";
    };

    tier = lib.mkOption {
      type = lib.types.enum [ "test" "prod" ];
      description = "Deployment tier for this host";
    };

    role = lib.mkOption {
      type = lib.types.nullOr lib.types.str;
      default = null;
      description = "Role for role-based deployment targeting";
    };

    natsUrl = lib.mkOption {
      type = lib.types.str;
      description = "NATS server URL";
    };

    nkeyFile = lib.mkOption {
      type = lib.types.path;
      description = "Path to NKey seed file for NATS authentication";
    };

    flakeUrl = lib.mkOption {
      type = lib.types.str;
      description = "Git flake URL for nixos-rebuild";
    };

    timeout = lib.mkOption {
      type = lib.types.int;
      default = 600;
      description = "Deployment timeout in seconds";
    };
  };
}

Systemd Service

The module should create a systemd service with:

• Type=simple
• Restart=always
• RestartSec=10
• Run as root (required for nixos-rebuild)
• Proper ordering (after network-online.target)
• Resource limits if desired

NATS Authentication

All NATS connections use NKey authentication. NKeys are ed25519 keypairs where:

• The seed (private key) is stored in a file readable by the service
• The public key is configured in the NATS server's user list

Credential Types

Credential	Purpose	Publish Permissions	Subscribe Permissions
listener	Host listener service	deploy.responses.>	deploy.*.>
mcp-deployer	MCP test-tier access	deploy.test.>	deploy.responses.>, deploy.discover
admin-deployer	Full deployment access	deploy.test.>, deploy.prod.>	deploy.responses.>, deploy.discover

Flake Structure

The flake.nix should provide:

1. Package: The Go binary
2. NixOS module: The listener service configuration
3. Development shell: Go toolchain for development

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  };

  outputs = { self, nixpkgs }: {
    packages.x86_64-linux.default = /* Go package build */;
    packages.x86_64-linux.homelab-deploy = self.packages.x86_64-linux.default;

    nixosModules.default = import ./nixos/module.nix;
    nixosModules.homelab-deploy = self.nixosModules.default;

    devShells.x86_64-linux.default = /* Go dev shell */;
  };
}

Implementation Notes

Go Dependencies

Recommended libraries:

• github.com/nats-io/nats.go - NATS client
• github.com/spf13/cobra - CLI framework
• github.com/mark3labs/mcp-go - MCP server implementation (or similar)
• Standard library for JSON, logging, process execution

Error Handling

• NATS connection errors: Retry with exponential backoff
• nixos-rebuild failures: Capture stdout/stderr, report in response message
• Timeout: Kill the nixos-rebuild process, report timeout error

Testing

• Unit tests for message parsing and validation
• Integration tests using a local NATS server
• End-to-end tests with a NixOS VM (optional, can be done in consuming repo)

Security Considerations

• Privilege: Listener runs as root to execute nixos-rebuild
• Input validation: Strictly validate revision format (alphanumeric, dashes, underscores, dots, slashes for branch names; hex for commit hashes)
• Command injection: Never interpolate user input into shell commands without validation
• Rate limiting: Consider adding rate limiting to prevent rapid-fire deployments
• Audit logging: Log all deployment requests with full context
• Network isolation: NATS should only be accessible from trusted networks

Future Enhancements

These are not required for initial implementation:

1. Deployment locking - Cluster-wide lock to prevent fleet-wide concurrent deploys
2. Prometheus metrics - Export deployment count, duration, success/failure rates
3. Webhook triggers - HTTP endpoint for CI/CD integration
4. Scheduled deployments - Deploy at specific times (though this overlaps with existing auto-upgrade)