claude: add auditor agent and git-explorer MCP

Add new auditor agent for security-focused audit log analysis:
- SSH session tracking, command execution, sudo usage
- Suspicious activity detection patterns
- Can be used standalone or as sub-agent by investigate-alarm

Update investigate-alarm to delegate audit analysis to auditor
and add git-explorer MCP for configuration drift detection.

Add git-explorer to .mcp.json for repository inspection.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

.claude/agents/investigate-alarm.md

@@ -4,6 +4,7 @@ description: Investigates a single system alarm by querying Prometheus metrics a
 tools: Read, Grep, Glob
 mcpServers:
   - lab-monitoring
+  - git-explorer
 ---
 
 You are an alarm investigation specialist for a NixOS homelab infrastructure. Your task is to analyze a single alarm and determine its root cause.
@@ -33,9 +34,9 @@ Gather evidence about the current system state:
 - Use `list_targets` to verify the host/service is being scraped successfully
 - Look for correlated metrics that might explain the issue
 
-### 3. Check Logs
+### 3. Check Service Logs
 
-Search for relevant log entries using `query_logs`. **Be careful to avoid overly broad queries that return too much data.**
+Search for relevant log entries using `query_logs`. Focus on service-specific logs and errors.
 
 **Query strategies (start narrow, expand if needed):**
 - Start with `limit: 20-30`, increase only if needed
@@ -43,23 +44,35 @@ Search for relevant log entries using `query_logs`. **Be careful to avoid overly
 - Filter to specific services: `{host="<hostname>", systemd_unit="<service>.service"}`
 - Search for errors: `{host="<hostname>"} |= "error"` or `|= "failed"`
 
-**For audit logs (SSH sessions, command execution):**
-- Filter to just commands: `{host="<hostname>"} |= "EXECVE"`
-- Exclude verbose noise: `!= "PATH item" != "PROCTITLE" != "SYSCALL" != "BPF"`
-- Example: `{host="testvm01"} |= "EXECVE" != "systemd"` (user commands only)
-
 **Common patterns:**
 - Service logs: `{host="<hostname>", systemd_unit="<service>.service"}`
-- SSH activity: `{host="<hostname>", systemd_unit="sshd.service"}`
 - All errors on host: `{host="<hostname>"} |= "error"`
-- Specific command: `{host="<hostname>"} |= "EXECVE" |= "stress"`
+- Journal for a unit: `{host="<hostname>", systemd_unit="nginx.service"} |= "failed"`
 
 **Avoid:**
-- Querying all audit logs without filtering (very verbose)
 - Using `start: "1h"` with no filters on busy hosts
 - Limits over 50 without specific filters
 
-### 4. Check Configuration (if relevant)
+### 4. Investigate User Activity (if needed)
+
+If you suspect the issue may be related to user actions (manual commands, SSH sessions, service manipulation), **spawn the `auditor` agent** to investigate.
+
+Use the auditor when you need to know:
+- What commands were run around the time of an incident
+- Whether a service was manually stopped/restarted
+- Who was logged in and what they did
+- Whether there's suspicious activity on the host
+
+**Example prompt for auditor:**
+```
+Investigate user activity on <hostname> between <start_time> and <end_time>.
+Context: The nginx service stopped unexpectedly at 14:32. Check if it was
+manually stopped or if any commands were run around that time.
+```
+
+The auditor will return a focused report on relevant user activity that you can incorporate into your investigation.
+
+### 5. Check Configuration (if relevant)
 
 If the alert relates to a NixOS-managed service:
 - Check host configuration in `/hosts/<hostname>/`
@@ -67,9 +80,60 @@ If the alert relates to a NixOS-managed service:
 - Look for thresholds, resource limits, or misconfigurations
 - Check `homelab.host` options for tier/priority/role metadata
 
-### 5. Consider Common Causes
+### 6. Check for Configuration Drift
+
+Use the git-explorer MCP server to compare the host's deployed configuration against the current master branch. This helps identify:
+- Hosts running outdated configurations
+- Recent changes that might have caused the issue
+- Whether a fix has already been committed but not deployed
+
+**Step 1: Get the deployed revision from Prometheus**
+```promql
+nixos_flake_info{hostname="<hostname>"}
+```
+The `current_rev` label contains the deployed git commit hash.
+
+**Step 2: Check if the host is behind master**
+```
+resolve_ref("master")           # Get current master commit
+is_ancestor(deployed, master)   # Check if host is behind
+```
+
+**Step 3: See what commits are missing**
+```
+commits_between(deployed, master)  # List commits not yet deployed
+```
+
+**Step 4: Check which files changed**
+```
+get_diff_files(deployed, master)   # Files modified since deployment
+```
+Look for files in `hosts/<hostname>/`, `services/<relevant-service>/`, or `system/` that affect this host.
+
+**Step 5: View configuration at the deployed revision**
+```
+get_file_at_commit(deployed, "services/<service>/default.nix")
+```
+Compare against the current file to understand differences.
+
+**Step 6: Find when something changed**
+```
+search_commits("<service-name>")   # Find commits mentioning the service
+get_commit_info(<hash>)            # Get full details of a specific change
+```
+
+**Example workflow for a service-related alert:**
+1. Query `nixos_flake_info{hostname="monitoring01"}` → `current_rev: 8959829`
+2. `resolve_ref("master")` → `4633421`
+3. `is_ancestor("8959829", "4633421")` → Yes, host is behind
+4. `commits_between("8959829", "4633421")` → 7 commits missing
+5. `get_diff_files("8959829", "4633421")` → Check if relevant service files changed
+6. If a fix was committed after the deployed rev, recommend deployment
+
+### 7. Consider Common Causes
 
 For infrastructure alerts, common causes include:
+- **Configuration drift**: Host running outdated config, fix already in master
 - **Disk space**: Nix store growth, logs, temp files
 - **Memory pressure**: Service memory leaks, insufficient limits
 - **CPU**: Runaway processes, build jobs
@@ -133,6 +197,5 @@ Provide a concise report with one of two outcomes:
 - If the alert is a false positive or expected behavior, explain why
 - Consider the host's tier (test vs prod) when assessing severity
 - Build a timeline from log timestamps and metrics to show the sequence of events
-- Include precursor events (logins, config changes, restarts) that led to the issue
 - **Query logs incrementally**: start with narrow filters and small limits, expand only if needed
-- **Avoid broad audit log queries**: always filter to EXECVE and exclude noise (PATH, SYSCALL, BPF)
+- **Use the auditor agent** for user activity analysis (commands, SSH sessions, sudo usage)