Merge pull request 'grafana-kanidm-oidc' (#35) from grafana-kanidm-oidc into master

Reviewed-on: #35

CLAUDE.md

@@ -39,6 +39,30 @@ Do not automatically deploy changes. Deployments are usually done by updating th
 
 Do not run SSH commands directly. If a command needs to be run on a remote host, provide the command to the user and ask them to run it manually.
 
+### Sharing Command Output via Loki
+
+All hosts have the `pipe-to-loki` script for sending command output or terminal sessions to Loki, allowing users to share output with Claude without copy-pasting.
+
+**Pipe mode** - send command output:
+```bash
+command | pipe-to-loki                  # Auto-generated ID
+command | pipe-to-loki --id my-test     # Custom ID
+```
+
+**Session mode** - record interactive terminal session:
+```bash
+pipe-to-loki --record                   # Start recording, exit to send
+pipe-to-loki --record --id my-session   # With custom ID
+```
+
+The script prints the session ID which the user can share. Query results with:
+```logql
+{job="pipe-to-loki"}                           # All entries
+{job="pipe-to-loki", id="my-test"}             # Specific ID
+{job="pipe-to-loki", host="testvm01"}          # From specific host
+{job="pipe-to-loki", type="session"}           # Only sessions
+```
+
 ### Testing Feature Branches on Hosts
 
 All hosts have the `nixos-rebuild-test` helper script for testing feature branches before merging:

docs/plans/auth-system-replacement.md

@@ -151,11 +151,30 @@ Rationale:
 - Well above NixOS system users (typically <1000)
 - Avoids Podman/container issues with very high GIDs
 
+### Completed (2026-02-08) - OAuth2/OIDC for Grafana
+
+**OAuth2 client deployed for Grafana on monitoring02:**
+- Client ID: `grafana`
+- Redirect URL: `https://grafana-test.home.2rjus.net/login/generic_oauth`
+- Scope maps: `openid`, `profile`, `email`, `groups` for `users` group
+- Role mapping: `admins` group → Grafana Admin, others → Viewer
+
+**Configuration locations:**
+- Kanidm OAuth2 client: `services/kanidm/default.nix`
+- Grafana OIDC config: `services/grafana/default.nix`
+- Vault secret: `services/grafana/oauth2-client-secret`
+
+**Key findings:**
+- PKCE is required by Kanidm - enable `use_pkce = true` in Grafana
+- Must set `email_attribute_path`, `login_attribute_path`, `name_attribute_path` to extract from userinfo
+- Users need: primary credential (password + TOTP for MFA), membership in `users` group, email address set
+- Unix password is separate from primary credential (web login requires primary credential)
+
 ### Next Steps
 
 1. Enable PAM/NSS on production hosts (after test tier validation)
 2. Configure TrueNAS LDAP client for NAS integration testing
-3. Add OAuth2 clients (Grafana first)
+3. Add OAuth2 clients for other services as needed
 
 ## References
 

docs/plans/monitoring-migration-victoriametrics.md

@@ -169,9 +169,30 @@ Once ready to cut over:
    - Destroy VM in Proxmox
    - Remove from terraform state
 
+## Current Progress
+
+### monitoring02 Host Created (2026-02-08)
+
+Host deployed at 10.69.13.24 (test tier) with:
+- 4 CPU cores, 8GB RAM, 60GB disk
+- Vault integration enabled
+- NATS-based remote deployment enabled
+
+### Grafana with Kanidm OIDC (2026-02-08)
+
+Grafana deployed on monitoring02 as a test instance (`grafana-test.home.2rjus.net`):
+- Kanidm OIDC authentication (PKCE enabled)
+- Role mapping: `admins` → Admin, others → Viewer
+- Declarative datasources pointing to monitoring01 (Prometheus, Loki)
+- Local Caddy for TLS termination via internal ACME CA
+
+This validates the Grafana + OIDC pattern before the full VictoriaMetrics migration. The existing
+`services/monitoring/grafana.nix` on monitoring01 can be replaced with the new `services/grafana/`
+module once monitoring02 becomes the primary monitoring host.
+
 ## Open Questions
 
-- [ ] What disk size for monitoring02? 100GB should allow 3+ months with VictoriaMetrics compression
+- [ ] What disk size for monitoring02? Current 60GB may need expansion for 3+ months with VictoriaMetrics
 - [ ] Which dashboards to recreate declaratively? (Review monitoring01 Grafana for current set)
 
 ## VictoriaMetrics Service Configuration

docs/plans/openbao-kanidm-oidc.md

@@ -0,0 +1,108 @@
+# OpenBao + Kanidm OIDC Integration
+
+## Overview
+
+Enable Kanidm users to authenticate to OpenBao (Vault) using OIDC, allowing access to secrets based on Kanidm group membership.
+
+## Current State
+
+**Kanidm:**
+- Server: `auth.home.2rjus.net` (kanidm01)
+- Domain: `home.2rjus.net`
+- Groups: `admins`, `users`, `ssh-users`
+- No OIDC clients configured yet
+
+**OpenBao:**
+- Server: `vault.home.2rjus.net` (vault01)
+- Auth: AppRole only (machine-to-machine)
+- No human user authentication configured
+
+## OpenBao OIDC Auth Method
+
+OpenBao includes the JWT/OIDC auth method in the open-source version (unlike Vault Enterprise which gates some auth features). Key points:
+
+- Enable with: `bao auth enable oidc`
+- Supports browser-based OIDC login flow
+- Maps OIDC claims/groups to OpenBao policies
+- Works with both CLI (`bao login`) and Web UI
+
+### Required Configuration
+
+```bash
+bao write auth/oidc/config \
+  oidc_discovery_url="https://auth.home.2rjus.net/oauth2/openid/<client_id>" \
+  oidc_client_id="<client_id>" \
+  oidc_client_secret="<client_secret>" \
+  default_role="default"
+```
+
+### Callback URIs
+
+OpenBao requires specific callback URIs registered in Kanidm:
+
+- **CLI:** `http://localhost:8250/oidc/callback`
+- **Web UI:** `https://vault.home.2rjus.net:8200/ui/vault/auth/oidc/oidc/callback`
+
+## Kanidm OAuth2 Configuration
+
+Kanidm supports declarative OAuth2 client provisioning via NixOS:
+
+```nix
+services.kanidm.provision.systems.oauth2.openbao = {
+  displayName = "OpenBao Secrets";
+  # originUrl - where the client lives
+  # originLanding - where to redirect after auth
+  # basicSecretFile - client secret
+  # scopeMaps - which scopes groups can request
+  # claimMaps - custom claims based on group membership
+};
+```
+
+The `basicSecretFile` should contain the client secret, fetched from Vault.
+
+## Implementation Approach
+
+### 1. Create OAuth2 Client in Kanidm
+
+Add to `services/kanidm/default.nix`:
+- OAuth2 client `openbao` with callback URIs
+- Scope maps for `admins` and `users` groups
+- Claim maps to expose group membership
+
+### 2. Enable OIDC Auth in OpenBao
+
+Options:
+- **Terraform:** Add `vault_jwt_auth_backend` resource in `terraform/vault/`
+- **NixOS:** Configure in vault01 host config
+
+Terraform is probably cleaner since we already manage OpenBao config there.
+
+### 3. Create OpenBao Roles
+
+Map Kanidm groups to policies:
+
+| Kanidm Group | OpenBao Role | Policy |
+|--------------|--------------|--------|
+| `admins` | `admin` | Full read access to secrets |
+| `users` | `user` | Limited read access |
+
+### 4. Chicken-and-Egg Problem
+
+The OAuth2 client secret needs to be stored in OpenBao, but OpenBao needs the secret to configure OIDC auth. Solutions:
+
+1. **Bootstrap manually:** Create initial secret via `bao` CLI
+2. **Two-phase Terraform:** First create the secret, then configure OIDC
+3. **Static secret:** Use a static secret for the OAuth2 client (less ideal)
+
+## Open Questions
+
+1. **Web UI access:** Do we want users logging into the OpenBao web UI, or just CLI?
+2. **Policy granularity:** What secrets should `admins` vs `users` access?
+3. **Token TTL:** How long should OIDC-issued tokens last?
+
+## References
+
+- [OpenBao JWT/OIDC Auth Method](https://openbao.org/docs/auth/jwt/)
+- [OpenBao OIDC Provider Configuration](https://openbao.org/docs/auth/jwt/oidc-providers/)
+- [Kanidm OAuth2 Documentation](https://kanidm.github.io/kanidm/stable/integrations/oauth2.html)
+- [NixOS Kanidm OAuth2 Options](https://search.nixos.org/options?query=services.kanidm.provision.systems.oauth2)

docs/user-management.md

@@ -43,11 +43,21 @@ kanidm person posix set-password <username>
 kanidm person posix set <username> --shell /bin/zsh
 ```
 
+### Setting Email Address
+
+Email is required for OAuth2/OIDC login (e.g., Grafana):
+
+```bash
+kanidm person update <username> --mail <email>
+```
+
 ### Example: Full User Creation
 
 ```bash
 kanidm person create testuser "Test User"
+kanidm person update testuser --mail testuser@home.2rjus.net
 kanidm group add-members ssh-users testuser
+kanidm group add-members users testuser  # Required for OAuth2 scopes
 kanidm person posix set testuser
 kanidm person posix set-password testuser
 kanidm person get testuser
@@ -129,6 +139,40 @@ Kanidm auto-assigns UIDs/GIDs from its configured range. For manually assigned G
 | 65,536+ | Users (auto-assigned) |
 | 68,000 - 68,999 | Groups (manually assigned) |
 
+## OAuth2/OIDC Login (Web Services)
+
+For OAuth2/OIDC login to web services like Grafana, users need:
+
+1. **Primary credential** - Password set via `credential update` (separate from unix password)
+2. **MFA** - TOTP or passkey (Kanidm requires MFA for primary credentials)
+3. **Group membership** - Member of `users` group (for OAuth2 scope mapping)
+4. **Email address** - Set via `person update --mail`
+
+### Setting Up Primary Credential (Web Login)
+
+The primary credential is different from the unix/POSIX password:
+
+```bash
+# Interactive credential setup
+kanidm person credential update <username>
+
+# In the interactive prompt:
+# 1. Type 'password' to set a password
+# 2. Type 'totp' to add TOTP (scan QR with authenticator app)
+# 3. Type 'commit' to save
+```
+
+### Verifying OAuth2 Readiness
+
+```bash
+kanidm person get <username>
+```
+
+Check for:
+- `mail:` - Email address set
+- `memberof:` - Includes `users@home.2rjus.net`
+- Primary credential status (check via `credential update` → `status`)
+
 ## PAM/NSS Client Configuration
 
 Enable central authentication on a host:

flake.nix

@@ -191,6 +191,15 @@
             ./hosts/kanidm01
           ];
         };
+        monitoring02 = nixpkgs.lib.nixosSystem {
+          inherit system;
+          specialArgs = {
+            inherit inputs self;
+          };
+          modules = commonModules ++ [
+            ./hosts/monitoring02
+          ];
+        };
       };
       packages = forAllSystems (
         { pkgs }:

hosts/monitoring02/configuration.nix

@@ -0,0 +1,75 @@
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+
+{
+  imports = [
+    ../template2/hardware-configuration.nix
+
+    ../../system
+    ../../common/vm
+  ];
+
+  # Host metadata (adjust as needed)
+  homelab.host = {
+    tier = "test";  # Start in test tier, move to prod after validation
+  };
+
+  # DNS CNAME for Grafana test instance
+  homelab.dns.cnames = [ "grafana-test" ];
+
+  # Enable Vault integration
+  vault.enable = true;
+
+  # Enable remote deployment via NATS
+  homelab.deploy.enable = true;
+
+  nixpkgs.config.allowUnfree = true;
+  boot.loader.grub.enable = true;
+  boot.loader.grub.device = "/dev/vda";
+
+  networking.hostName = "monitoring02";
+  networking.domain = "home.2rjus.net";
+  networking.useNetworkd = true;
+  networking.useDHCP = false;
+  services.resolved.enable = true;
+  networking.nameservers = [
+    "10.69.13.5"
+    "10.69.13.6"
+  ];
+
+  systemd.network.enable = true;
+  systemd.network.networks."ens18" = {
+    matchConfig.Name = "ens18";
+    address = [
+      "10.69.13.24/24"
+    ];
+    routes = [
+      { Gateway = "10.69.13.1"; }
+    ];
+    linkConfig.RequiredForOnline = "routable";
+  };
+  time.timeZone = "Europe/Oslo";
+
+  nix.settings.experimental-features = [
+    "nix-command"
+    "flakes"
+  ];
+  nix.settings.tarball-ttl = 0;
+  environment.systemPackages = with pkgs; [
+    vim
+    wget
+    git
+  ];
+
+  # Open ports in the firewall.
+  # networking.firewall.allowedTCPPorts = [ ... ];
+  # networking.firewall.allowedUDPPorts = [ ... ];
+  # Or disable the firewall altogether.
+  networking.firewall.enable = false;
+
+  system.stateVersion = "25.11"; # Did you read the comment?
+}

hosts/monitoring02/default.nix

@@ -0,0 +1,6 @@
+{ ... }: {
+  imports = [
+    ./configuration.nix
+    ../../services/grafana
+  ];
+}

services/grafana/default.nix

@@ -0,0 +1,98 @@
+{ config, pkgs, ... }:
+{
+  services.grafana = {
+    enable = true;
+    settings = {
+      server = {
+        http_addr = "127.0.0.1";
+        http_port = 3000;
+        domain = "grafana-test.home.2rjus.net";
+        root_url = "https://grafana-test.home.2rjus.net/";
+      };
+
+      # Disable anonymous access
+      "auth.anonymous".enabled = false;
+
+      # OIDC authentication via Kanidm
+      "auth.generic_oauth" = {
+        enabled = true;
+        name = "Kanidm";
+        client_id = "grafana";
+        client_secret = "$__file{/run/secrets/grafana-oauth2}";
+        auth_url = "https://auth.home.2rjus.net/ui/oauth2";
+        token_url = "https://auth.home.2rjus.net/oauth2/token";
+        api_url = "https://auth.home.2rjus.net/oauth2/openid/grafana/userinfo";
+        scopes = "openid profile email groups";
+        use_pkce = true;  # Required by Kanidm, more secure
+        # Extract user attributes from userinfo response
+        email_attribute_path = "email";
+        login_attribute_path = "preferred_username";
+        name_attribute_path = "name";
+        # Map admins group to Admin role, everyone else to Viewer
+        role_attribute_path = "contains(groups[*], 'admins') && 'Admin' || 'Viewer'";
+        allow_sign_up = true;
+      };
+    };
+
+    # Declarative datasources pointing to monitoring01
+    provision.datasources.settings = {
+      apiVersion = 1;
+      datasources = [
+        {
+          name = "Prometheus";
+          type = "prometheus";
+          url = "http://monitoring01.home.2rjus.net:9090";
+          isDefault = true;
+          uid = "prometheus";
+        }
+        {
+          name = "Loki";
+          type = "loki";
+          url = "http://monitoring01.home.2rjus.net:3100";
+          uid = "loki";
+        }
+      ];
+    };
+  };
+
+  # Vault secret for OAuth2 client secret
+  vault.secrets.grafana-oauth2 = {
+    secretPath = "services/grafana/oauth2-client-secret";
+    extractKey = "password";
+    services = [ "grafana" ];
+    owner = "grafana";
+    group = "grafana";
+  };
+
+  # Local Caddy for TLS termination
+  services.caddy = {
+    enable = true;
+    package = pkgs.unstable.caddy;
+    configFile = pkgs.writeText "Caddyfile" ''
+      {
+        acme_ca https://vault.home.2rjus.net:8200/v1/pki_int/acme/directory
+        metrics
+      }
+
+      grafana-test.home.2rjus.net {
+        log {
+          output file /var/log/caddy/grafana.log {
+            mode 644
+          }
+        }
+
+        reverse_proxy http://127.0.0.1:3000
+      }
+
+      http://${config.networking.hostName}.home.2rjus.net/metrics {
+        metrics
+      }
+    '';
+  };
+
+  # Expose Caddy metrics for Prometheus
+  homelab.monitoring.scrapeTargets = [{
+    job_name = "caddy";
+    port = 80;
+  }];
+}

services/kanidm/default.nix

@@ -30,6 +30,16 @@
       };
 
       # Regular users (persons) are managed imperatively via kanidm CLI
+
+      # OAuth2/OIDC clients for service authentication
+      systems.oauth2.grafana = {
+        displayName = "Grafana";
+        originUrl = "https://grafana-test.home.2rjus.net/login/generic_oauth";
+        originLanding = "https://grafana-test.home.2rjus.net/";
+        basicSecretFile = config.vault.secrets.grafana-oauth2.outputDir;
+        preferShortUsername = true;
+        scopeMaps.users = [ "openid" "profile" "email" "groups" ];
+      };
     };
   };
 
@@ -53,6 +63,15 @@
     group = "kanidm";
   };
 
+  # Vault secret for Grafana OAuth2 client secret
+  vault.secrets.grafana-oauth2 = {
+    secretPath = "services/grafana/oauth2-client-secret";
+    extractKey = "password";
+    services = [ "kanidm" ];
+    owner = "kanidm";
+    group = "kanidm";
+  };
+
   # Note: Kanidm does not expose Prometheus metrics
   # If metrics support is added in the future, uncomment:
   # homelab.monitoring.scrapeTargets = [

system/default.nix

@@ -9,6 +9,7 @@
     ./motd.nix
     ./packages.nix
     ./nix.nix
+    ./pipe-to-loki.nix
     ./root-user.nix
     ./pki/root-ca.nix
     ./sshd.nix

system/pipe-to-loki.nix

@@ -0,0 +1,140 @@
+{
+  config,
+  pkgs,
+  lib,
+  ...
+}:
+let
+  pipe-to-loki = pkgs.writeShellApplication {
+    name = "pipe-to-loki";
+    runtimeInputs = with pkgs; [
+      curl
+      jq
+      util-linux
+      coreutils
+    ];
+    text = ''
+      set -euo pipefail
+
+      LOKI_URL="http://monitoring01.home.2rjus.net:3100/loki/api/v1/push"
+      HOSTNAME=$(hostname)
+      SESSION_ID=""
+      RECORD_MODE=false
+
+      usage() {
+        echo "Usage: pipe-to-loki [--id ID] [--record]"
+        echo ""
+        echo "Send command output or interactive sessions to Loki."
+        echo ""
+        echo "Options:"
+        echo "  --id ID      Set custom session ID (default: auto-generated)"
+        echo "  --record     Start interactive recording session"
+        echo ""
+        echo "Examples:"
+        echo "  command | pipe-to-loki           # Pipe command output"
+        echo "  command | pipe-to-loki --id foo  # Pipe with custom ID"
+        echo "  pipe-to-loki --record            # Start recording session"
+        exit 1
+      }
+
+      generate_id() {
+        local random_chars
+        random_chars=$(head -c 2 /dev/urandom | od -An -tx1 | tr -d ' \n')
+        echo "''${HOSTNAME}-$(date +%s)-''${random_chars}"
+      }
+
+      send_to_loki() {
+        local content="$1"
+        local type="$2"
+        local timestamp_ns
+        timestamp_ns=$(date +%s%N)
+
+        local payload
+        payload=$(jq -n \
+          --arg job "pipe-to-loki" \
+          --arg host "$HOSTNAME" \
+          --arg type "$type" \
+          --arg id "$SESSION_ID" \
+          --arg ts "$timestamp_ns" \
+          --arg content "$content" \
+          '{
+            streams: [{
+              stream: {
+                job: $job,
+                host: $host,
+                type: $type,
+                id: $id
+              },
+              values: [[$ts, $content]]
+            }]
+          }')
+
+        if curl -s -X POST "$LOKI_URL" \
+          -H "Content-Type: application/json" \
+          -d "$payload" > /dev/null; then
+          return 0
+        else
+          echo "Error: Failed to send to Loki" >&2
+          return 1
+        fi
+      }
+
+      # Parse arguments
+      while [[ $# -gt 0 ]]; do
+        case $1 in
+          --id)
+            SESSION_ID="$2"
+            shift 2
+            ;;
+          --record)
+            RECORD_MODE=true
+            shift
+            ;;
+          --help|-h)
+            usage
+            ;;
+          *)
+            echo "Unknown option: $1" >&2
+            usage
+            ;;
+        esac
+      done
+
+      # Generate ID if not provided
+      if [[ -z "$SESSION_ID" ]]; then
+        SESSION_ID=$(generate_id)
+      fi
+
+      if $RECORD_MODE; then
+        # Session recording mode
+        SCRIPT_FILE=$(mktemp)
+        trap 'rm -f "$SCRIPT_FILE"' EXIT
+
+        echo "Recording session $SESSION_ID... (exit to send)"
+
+        # Use script to record the session
+        script -q "$SCRIPT_FILE"
+
+        # Read the transcript and send to Loki
+        content=$(cat "$SCRIPT_FILE")
+        if send_to_loki "$content" "session"; then
+          echo "Session $SESSION_ID sent to Loki"
+        fi
+      else
+        # Pipe mode - read from stdin
+        if [[ -t 0 ]]; then
+          echo "Error: No input provided. Pipe a command or use --record for interactive mode." >&2
+          exit 1
+        fi
+
+        content=$(cat)
+        if send_to_loki "$content" "command"; then
+          echo "Sent to Loki with id: $SESSION_ID"
+        fi
+      fi
+    '';
+  };
+in
+{
+  environment.systemPackages = [ pipe-to-loki ];
+}

terraform/vault/approle.tf

@@ -89,6 +89,23 @@ locals {
       ]
     }
 
+    # kanidm01: Kanidm identity provider
+    "kanidm01" = {
+      paths = [
+        "secret/data/hosts/kanidm01/*",
+        "secret/data/kanidm/*",
+        "secret/data/services/grafana/*",
+      ]
+    }
+
+    # monitoring02: Grafana test instance
+    "monitoring02" = {
+      paths = [
+        "secret/data/hosts/monitoring02/*",
+        "secret/data/services/grafana/*",
+      ]
+    }
+
   }
 }
 

terraform/vault/hosts-generated.tf

@@ -39,6 +39,11 @@ locals {
         "secret/data/kanidm/*",
       ]
     }
+    "monitoring02" = {
+      paths = [
+        "secret/data/hosts/monitoring02/*",
+      ]
+    }
   
   }
 

terraform/vault/secrets.tf

@@ -108,6 +108,12 @@ locals {
       auto_generate   = true
       password_length = 32
     }
+
+    # Grafana OAuth2 client secret (for Kanidm OIDC)
+    "services/grafana/oauth2-client-secret" = {
+      auto_generate   = true
+      password_length = 64
+    }
   }
 }
 

terraform/vms.tf

@@ -79,6 +79,13 @@ locals {
       disk_size = "20G"
       vault_wrapped_token = "s.OOqjEECeIV7dNgCS6jNmyY3K"
     }
+    "monitoring02" = {
+      ip        = "10.69.13.24/24"
+      cpu_cores = 4
+      memory    = 4096
+      disk_size = "60G"
+      vault_wrapped_token = "s.uXpdoGxHXpWvTsGbHkZuq1jF"
+    }
   }
 
   # Compute VM configurations with defaults applied