nixos-exporter/README.md

# nixos-exporter

A Prometheus exporter for NixOS-specific metrics. Exposes system state information that standard exporters don't cover: generation management, flake input freshness, and upgrade status.

## Installation

### As a flake

```nix
{
  inputs.nixos-exporter.url = "git+https://git.t-juice.club/torjus/nixos-exporter";

  outputs = { self, nixpkgs, nixos-exporter, ... }: {
    nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
      modules = [
        nixos-exporter.nixosModules.default
        {
          services.prometheus.exporters.nixos = {
            enable = true;
            flake = {
              enable = true;
              url = "github:myuser/myconfig";
            };
          };
        }
      ];
    };
  };
}
```

### Manual

```bash
nix build
./result/bin/nixos-exporter --listen=:9971
```

## CLI Flags

| Flag | Default | Description |
|------|---------|-------------|
| `--listen` | `:9971` | Address to listen on |
| `--collector.flake` | `false` | Enable flake collector |
| `--flake.url` | | Flake URL for revision comparison (required if flake collector enabled) |
| `--flake.check-interval` | `1h` | Interval between remote flake checks |
| `--flake.nats.enable` | `false` | Enable NATS cache sharing |
| `--flake.nats.url` | `nats://localhost:4222` | NATS server URL |
| `--flake.nats.subject` | `nixos-exporter.remote-rev` | NATS subject for revision updates |
| `--flake.nats.credentials-file` | | NATS credentials file (optional) |
| `--flake.nats.nkey-seed-file` | | NATS NKey seed file (optional) |

## NixOS Module Options

```nix
services.prometheus.exporters.nixos = {
  enable = true;
  port = 9971;
  listenAddress = "0.0.0.0";
  openFirewall = false;

  flake = {
    enable = false;
    url = "";  # Required if flake.enable = true
    checkInterval = "1h";

    nats = {
      enable = false;
      url = "nats://localhost:4222";
      subject = "nixos-exporter.remote-rev";
      credentialsFile = null;  # Optional path to credentials file
      nkeySeedFile = null;     # Optional path to NKey seed file
    };
  };
};
```

## Metrics

### Generation Metrics (always enabled)

| Metric | Type | Description |
|--------|------|-------------|
| `nixos_generation_count` | Gauge | Total number of system generations |
| `nixos_current_generation` | Gauge | Currently active generation number |
| `nixos_booted_generation` | Gauge | Generation that was booted |
| `nixos_generation_age_seconds` | Gauge | Age of current generation in seconds |
| `nixos_config_mismatch` | Gauge | 1 if booted generation differs from current |

### Flake Metrics (optional)

| Metric | Type | Labels | Description |
|--------|------|--------|-------------|
| `nixos_flake_input_age_seconds` | Gauge | `input` | Age of flake input in seconds |
| `nixos_flake_input_info` | Gauge | `input`, `rev`, `type` | Info gauge with revision and type labels |
| `nixos_flake_info` | Gauge | `current_rev`, `remote_rev`, `nixpkgs_rev`, `nixos_version` | Info gauge with system version details |
| `nixos_flake_revision_behind` | Gauge | | 1 if current system revision differs from remote latest |

### Setting Configuration Revision

For `current_rev` to show the flake's git revision, you must set `system.configurationRevision` in your flake:

```nix
{
  outputs = { self, nixpkgs, ... }: {
    nixosConfigurations.myhost = nixpkgs.lib.nixosSystem {
      modules = [
        {
          system.configurationRevision = self.rev or self.dirtyRev or "dirty";
        }
        ./configuration.nix
      ];
    };
  };
}
```

This sets the revision to:
- `self.rev` - Git commit hash (only available when tree is clean and committed)
- `self.dirtyRev` - Dirty revision (e.g., `abc1234-dirty`) when uncommitted changes exist
- `"dirty"` - Fallback when neither is available

Without this setting, `current_rev` will be `"unknown"` and `nixos_flake_revision_behind` will always be 0.

## Example Prometheus Alerts

```yaml
groups:
  - name: nixos
    rules:
      - alert: NixOSConfigStale
        expr: nixos_generation_age_seconds > 7 * 24 * 3600
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "NixOS config on {{ $labels.instance }} is over 7 days old"

      - alert: NixOSRebootRequired
        expr: nixos_config_mismatch == 1
        for: 24h
        labels:
          severity: info
        annotations:
          summary: "{{ $labels.instance }} needs reboot to apply config"

      - alert: NixpkgsInputStale
        expr: nixos_flake_input_age_seconds{input="nixpkgs"} > 30 * 24 * 3600
        for: 1d
        labels:
          severity: info
        annotations:
          summary: "nixpkgs input on {{ $labels.instance }} is over 30 days old"

      - alert: NixOSRevisionBehind
        expr: nixos_flake_revision_behind == 1
        for: 1h
        labels:
          severity: info
        annotations:
          summary: "{{ $labels.instance }} is behind remote flake revision"
```

## Security Considerations

- The `/metrics` endpoint exposes system state and revision information. Only expose it on internal networks.
- Runs as non-root user; only reads symlinks and files that are world-readable.
- When using the flake collector, the exporter executes `nix flake metadata` to fetch remote data.

## Known Limitations

- Flake input metrics (`nixos_flake_input_age_seconds`, `nixos_flake_input_info`) reflect the remote flake state, not the currently deployed system. If the deployed system is behind, these will show newer data than what's actually deployed.

- The `nixos_flake_revision_behind` metric requires `system.configurationRevision` to be set. Without it, the metric will always be 0 since there's no local revision to compare against.

## NATS Cache Sharing

When running multiple nixos-exporter instances across different hosts, NATS cache sharing allows them to share remote revision data. This reduces the number of `nix flake metadata` calls and ensures all hosts see updates faster.

When enabled:
- Hosts publish their remote revision when they fetch new data
- Hosts subscribe to receive updates from other hosts
- Updates are filtered by flake URL to avoid cross-contamination
- Auto-reconnect handles NATS server restarts gracefully
- Falls back to local fetching if NATS is unavailable

Additionally, smart cache refresh is enabled: when the current system revision matches the cached remote revision, the exporter immediately checks for newer revisions instead of waiting for the next scheduled check.

## License

MIT