After a successful switch deployment, the listener now waits for Prometheus
to scrape the /metrics endpoint before exiting for restart. This ensures
deployment metrics are captured before the process restarts and resets
in-memory counters. Falls back to a 60 second timeout if no scrape occurs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Counter and histogram metrics were absent from Prometheus scrapes until
the first deployment occurred, making it impossible to distinguish
"no deployments" from "exporter not running" in dashboards and alerts.
Initialize all expected label combinations with zero values when the
collector is created so metrics appear in every scrape from startup.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The CLI was reporting deployment failures even when the listener showed
success. This was a race condition: after a successful switch deployment,
the listener would send the "completed" response then immediately signal
restart. The NATS connection closed before the buffered message was
actually sent to the broker, so the CLI never received it.
Adding Flush() after sending the completed response ensures the message
reaches NATS before the listener can exit.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Send periodic "running" status messages while nixos-rebuild executes,
preventing the idle timeout from triggering before deployments complete.
This fixes false "Some deployments failed" warnings in MCP when builds
take longer than 30 seconds.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add an optional Prometheus metrics HTTP endpoint to the listener for
monitoring deployment operations. Includes four metrics:
- homelab_deploy_deployments_total (counter with status/action/error_code)
- homelab_deploy_deployment_duration_seconds (histogram with action/success)
- homelab_deploy_deployment_in_progress (gauge)
- homelab_deploy_info (gauge with hostname/tier/role/version)
New CLI flags: --metrics-enabled, --metrics-addr (default :9972)
New NixOS options: metrics.enable, metrics.address, metrics.openFirewall
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The CLI was incorrectly reporting "some deployments failed" even when
deployments succeeded. This was because AllSucceeded() checked if every
response had StatusCompleted, but the Responses slice contains all
messages including intermediate ones like "started". Since started !=
completed, it returned false.
Now AllSucceeded() only examines final responses (using IsFinal()) and
checks that each host's final status is completed.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
After a successful switch deployment, the listener now exits gracefully
so systemd can restart it with the new binary. This works together with
stopIfChanged/restartIfChanged to ensure deployments complete before
the service restarts.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add nixos-rebuild to listener service PATH in NixOS module
- Fix CLI deploy command hanging after receiving final status by properly
tracking lastResponse time and exiting when all hosts have responded
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Reject NKey files that are readable by group or others (permissions
more permissive than 0600). This prevents accidental exposure of
private keys through overly permissive file permissions.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Implement the complete homelab-deploy system with three operational modes:
- Listener mode: Runs on NixOS hosts as a systemd service, subscribes to
NATS subjects with configurable templates, executes nixos-rebuild on
deployment requests with concurrency control
- MCP mode: MCP server exposing deploy, deploy_admin, and list_hosts
tools for AI assistants with tiered access control
- CLI mode: Manual deployment commands with subject alias support via
environment variables
Key components:
- internal/messages: Request/response types with validation
- internal/nats: Client wrapper with NKey authentication
- internal/deploy: Executor with timeout and lock for concurrency
- internal/listener: Subject template expansion and request handling
- internal/cli: Deploy logic with alias resolution
- internal/mcp: MCP server with mcp-go integration
- nixos/module.nix: NixOS module with hardened systemd service
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
