docs: update auth-system-replacement plan with PAM/NSS progress

- Mark PAM/NSS client module as complete
- Mark documentation as complete
- Update provisioning approach (declarative groups, imperative users)
- Add details on client module and verified functionality
- Update next steps

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs/plans/auth-system-replacement.md

@@ -66,9 +66,9 @@ This future migration path is a strong argument for Kanidm over LDAP-only soluti
    - Vault integration for idm_admin password
    - LDAPS on port 636
 
-2. **Configure declarative provisioning** ✅
-   - Groups: `admins`, `users`, `ssh-users`
-   - User: `torjus` (member of all groups)
+2. **Configure provisioning** ✅
+   - Groups provisioned declaratively: `admins`, `users`, `ssh-users`
+   - Users managed imperatively via CLI (allows setting POSIX passwords in one step)
    - POSIX attributes enabled (UID/GID range 65,536-69,999)
 
 3. **Test NAS integration** (in progress)
@@ -80,14 +80,16 @@ This future migration path is a strong argument for Kanidm over LDAP-only soluti
    - Grafana
    - Other services as needed
 
-5. **Create client module** in `system/` for PAM/NSS
-   - Enable on all hosts that need central auth
-   - Configure trusted CA
+5. **Create client module** in `system/` for PAM/NSS ✅
+   - Module: `system/kanidm-client.nix`
+   - `homelab.kanidm.enable = true` enables PAM/NSS
+   - Short usernames (not SPN format)
+   - Home directory symlinks via `home_alias`
+   - Enabled on test tier: testvm01, testvm02, testvm03
 
-6. **Documentation**
-   - User management procedures
-   - Adding new OAuth2 clients
-   - Troubleshooting PAM/NSS issues
+6. **Documentation** ✅
+   - `docs/user-management.md` - CLI workflows, troubleshooting
+   - User/group creation procedures verified working
 
 ## Progress
 
@@ -106,14 +108,37 @@ This future migration path is a strong argument for Kanidm over LDAP-only soluti
 - Prometheus monitoring scrape target configured
 
 **Provisioned entities:**
-- Groups: `admins`, `users`, `ssh-users`
-- User: `torjus` (member of all groups, POSIX enabled with GID 65536)
+- Groups: `admins`, `users`, `ssh-users` (declarative)
+- Users managed via CLI (imperative)
 
 **Verified working:**
 - WebUI login with idm_admin
 - LDAP bind and search with POSIX-enabled user
 - LDAPS with valid internal CA certificate
 
+### Completed (2026-02-08) - PAM/NSS Client
+
+**Client module deployed (`system/kanidm-client.nix`):**
+- `homelab.kanidm.enable = true` enables PAM/NSS integration
+- Connects to auth.home.2rjus.net
+- Short usernames (`torjus` instead of `torjus@home.2rjus.net`)
+- Home directory symlinks (`/home/torjus` → UUID-based dir)
+- Login restricted to `ssh-users` group
+
+**Enabled on test tier:**
+- testvm01, testvm02, testvm03
+
+**Verified working:**
+- User/group resolution via `getent`
+- SSH login with Kanidm unix passwords
+- Home directory creation with symlinks
+- Imperative user/group creation via CLI
+
+**Documentation:**
+- `docs/user-management.md` with full CLI workflows
+- Password requirements (min 10 chars)
+- Troubleshooting guide (nscd, cache invalidation)
+
 ### UID/GID Range (Resolved)
 
 **Range: 65,536 - 69,999** (manually allocated)
@@ -128,10 +153,9 @@ Rationale:
 
 ### Next Steps
 
-1. Deploy to monitoring01 to enable Prometheus scraping
+1. Enable PAM/NSS on production hosts (after test tier validation)
 2. Configure TrueNAS LDAP client for NAS integration testing
 3. Add OAuth2 clients (Grafana first)
-4. Create PAM/NSS client module for other hosts
 
 ## References
 

docs/user-management.md

@@ -0,0 +1,267 @@
+# User Management with Kanidm
+
+Central authentication for the homelab using Kanidm.
+
+## Overview
+
+- **Server**: kanidm01.home.2rjus.net (auth.home.2rjus.net)
+- **WebUI**: https://auth.home.2rjus.net
+- **LDAPS**: port 636
+
+## CLI Setup
+
+The `kanidm` CLI is available in the devshell:
+
+```bash
+nix develop
+
+# Login as idm_admin
+kanidm login --name idm_admin --url https://auth.home.2rjus.net
+```
+
+## User Management
+
+POSIX users are managed imperatively via the `kanidm` CLI. This allows setting
+all attributes (including UNIX password) in one workflow.
+
+### Creating a POSIX User
+
+```bash
+# Create the person
+kanidm person create <username> "<Display Name>"
+
+# Add to groups
+kanidm group add-members ssh-users <username>
+
+# Enable POSIX (UID is auto-assigned)
+kanidm person posix set <username>
+
+# Set UNIX password (required for SSH login, min 10 characters)
+kanidm person posix set-password <username>
+
+# Optionally set login shell
+kanidm person posix set <username> --shell /bin/zsh
+```
+
+### Example: Full User Creation
+
+```bash
+kanidm person create testuser "Test User"
+kanidm group add-members ssh-users testuser
+kanidm person posix set testuser
+kanidm person posix set-password testuser
+kanidm person get testuser
+```
+
+After creation, verify on a client host:
+```bash
+getent passwd testuser
+ssh testuser@testvm01.home.2rjus.net
+```
+
+### Viewing User Details
+
+```bash
+kanidm person get <username>
+```
+
+### Removing a User
+
+```bash
+kanidm person delete <username>
+```
+
+## Group Management
+
+Groups for POSIX access are also managed via CLI.
+
+### Creating a POSIX Group
+
+```bash
+# Create the group
+kanidm group create <group-name>
+
+# Enable POSIX with a specific GID
+kanidm group posix set <group-name> --gidnumber <gid>
+```
+
+### Adding Members
+
+```bash
+kanidm group add-members <group-name> <username>
+```
+
+### Viewing Group Details
+
+```bash
+kanidm group get <group-name>
+kanidm group list-members <group-name>
+```
+
+### Example: Full Group Creation
+
+```bash
+kanidm group create testgroup
+kanidm group posix set testgroup --gidnumber 68010
+kanidm group add-members testgroup testuser
+kanidm group get testgroup
+```
+
+After creation, verify on a client host:
+```bash
+getent group testgroup
+```
+
+### Current Groups
+
+| Group | GID | Purpose |
+|-------|-----|---------|
+| ssh-users | 68000 | SSH login access |
+| admins | 68001 | Administrative access |
+| users | 68002 | General users |
+
+### UID/GID Allocation
+
+Kanidm auto-assigns UIDs/GIDs from its configured range. For manually assigned GIDs:
+
+| Range | Purpose |
+|-------|---------|
+| 65,536+ | Users (auto-assigned) |
+| 68,000 - 68,999 | Groups (manually assigned) |
+
+## PAM/NSS Client Configuration
+
+Enable central authentication on a host:
+
+```nix
+homelab.kanidm.enable = true;
+```
+
+This configures:
+- `services.kanidm.enablePam = true`
+- Client connection to auth.home.2rjus.net
+- Login authorization for `ssh-users` group
+- Short usernames (`torjus` instead of `torjus@home.2rjus.net`)
+- Home directory symlinks (`/home/torjus` → UUID-based directory)
+
+### Enabled Hosts
+
+- testvm01, testvm02, testvm03 (test tier)
+
+### Options
+
+```nix
+homelab.kanidm = {
+  enable = true;
+  server = "https://auth.home.2rjus.net";  # default
+  allowedLoginGroups = [ "ssh-users" ];     # default
+};
+```
+
+### Home Directories
+
+Home directories use UUID-based paths for stability (so renaming a user doesn't
+require moving their home directory). Symlinks provide convenient access:
+
+```
+/home/torjus -> /home/e4f4c56c-4aee-4c20-846f-90cb69807733
+```
+
+The symlinks are created by `kanidm-unixd-tasks` on first login.
+
+## Testing
+
+### Verify NSS Resolution
+
+```bash
+# Check user resolution
+getent passwd <username>
+
+# Check group resolution
+getent group <group-name>
+```
+
+### Test SSH Login
+
+```bash
+ssh <username>@<hostname>.home.2rjus.net
+```
+
+## Troubleshooting
+
+### "PAM user mismatch" error
+
+SSH fails with "fatal: PAM user mismatch" in logs. This happens when Kanidm returns
+usernames in SPN format (`torjus@home.2rjus.net`) but SSH expects short names (`torjus`).
+
+**Solution**: Configure `uid_attr_map = "name"` in unixSettings (already set in our module).
+
+Check current format:
+```bash
+getent passwd torjus
+# Should show: torjus:x:65536:...
+# NOT: torjus@home.2rjus.net:x:65536:...
+```
+
+### User resolves but SSH fails immediately
+
+The user's login group (e.g., `ssh-users`) likely doesn't have POSIX enabled:
+
+```bash
+# Check if group has POSIX
+getent group ssh-users
+
+# If empty, enable POSIX on the server
+kanidm group posix set ssh-users --gidnumber 68000
+```
+
+### User doesn't resolve via getent
+
+1. Check kanidm-unixd service is running:
+   ```bash
+   systemctl status kanidm-unixd
+   ```
+
+2. Check unixd can reach server:
+   ```bash
+   kanidm-unix status
+   # Should show: system: online, Kanidm: online
+   ```
+
+3. Check client can reach server:
+   ```bash
+   curl -s https://auth.home.2rjus.net/status
+   ```
+
+4. Check user has POSIX enabled on server:
+   ```bash
+   kanidm person get <username>
+   ```
+
+5. Restart nscd to clear stale cache:
+   ```bash
+   systemctl restart nscd
+   ```
+
+6. Invalidate kanidm cache:
+   ```bash
+   kanidm-unix cache-invalidate
+   ```
+
+### Changes not taking effect after deployment
+
+NixOS uses nsncd (a Rust reimplementation of nscd) for NSS caching. After deploying
+kanidm-unixd config changes, you may need to restart both services:
+
+```bash
+systemctl restart kanidm-unixd
+systemctl restart nscd
+```
+
+### Test PAM authentication directly
+
+Use the kanidm-unix CLI to test PAM auth without SSH:
+
+```bash
+kanidm-unix auth-test --name <username>
+```

flake.nix

@@ -207,6 +207,7 @@
               pkgs.ansible
               pkgs.opentofu
               pkgs.openbao
+              pkgs.kanidm_1_8
               (pkgs.callPackage ./scripts/create-host { })
               homelab-deploy.packages.${pkgs.system}.default
             ];

hosts/testvm01/configuration.nix

@@ -25,6 +25,9 @@
   # Enable remote deployment via NATS
   homelab.deploy.enable = true;
 
+  # Enable Kanidm PAM/NSS for central authentication
+  homelab.kanidm.enable = true;
+
   nixpkgs.config.allowUnfree = true;
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";

hosts/testvm02/configuration.nix

@@ -25,6 +25,9 @@
   # Enable remote deployment via NATS
   homelab.deploy.enable = true;
 
+  # Enable Kanidm PAM/NSS for central authentication
+  homelab.kanidm.enable = true;
+
   nixpkgs.config.allowUnfree = true;
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";

hosts/testvm03/configuration.nix

@@ -25,6 +25,9 @@
   # Enable remote deployment via NATS
   homelab.deploy.enable = true;
 
+  # Enable Kanidm PAM/NSS for central authentication
+  homelab.kanidm.enable = true;
+
   nixpkgs.config.allowUnfree = true;
   boot.loader.grub.enable = true;
   boot.loader.grub.device = "/dev/vda";

services/kanidm/default.nix

@@ -17,7 +17,8 @@
       };
     };
 
-    # Provisioning - initial users/groups
+    # Provision base groups only - users are managed via CLI
+    # See docs/user-management.md for details
     provision = {
       enable = true;
       idmAdminPasswordFile = config.vault.secrets.kanidm-idm-admin.outputDir;
@@ -28,10 +29,7 @@
         ssh-users = { };
       };
 
-      persons.torjus = {
-        displayName = "Torjus";
-        groups = [ "admins" "users" "ssh-users" ];
-      };
+      # Regular users (persons) are managed imperatively via kanidm CLI
     };
   };
 
@@ -46,7 +44,7 @@
     extraDomainNames = [ "${config.networking.hostName}.home.2rjus.net" ];
   };
 
-  # Vault secret for idm_admin password
+  # Vault secret for idm_admin password (used for provisioning)
   vault.secrets.kanidm-idm-admin = {
     secretPath = "kanidm/idm-admin-password";
     extractKey = "password";

system/default.nix

@@ -4,6 +4,7 @@
     ./acme.nix
     ./autoupgrade.nix
     ./homelab-deploy.nix
+    ./kanidm-client.nix
     ./monitoring
     ./motd.nix
     ./packages.nix

system/kanidm-client.nix

@@ -0,0 +1,42 @@
+{ lib, config, pkgs, ... }:
+let
+  cfg = config.homelab.kanidm;
+in
+{
+  options.homelab.kanidm = {
+    enable = lib.mkEnableOption "Kanidm PAM/NSS client for central authentication";
+
+    server = lib.mkOption {
+      type = lib.types.str;
+      default = "https://auth.home.2rjus.net";
+      description = "URI of the Kanidm server";
+    };
+
+    allowedLoginGroups = lib.mkOption {
+      type = lib.types.listOf lib.types.str;
+      default = [ "ssh-users" ];
+      description = "Groups allowed to log in via PAM";
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    services.kanidm = {
+      package = pkgs.kanidm_1_8;
+      enablePam = true;
+
+      clientSettings = {
+        uri = cfg.server;
+      };
+
+      unixSettings = {
+        pam_allowed_login_groups = cfg.allowedLoginGroups;
+        # Use short names (torjus) instead of SPN format (torjus@home.2rjus.net)
+        # This prevents "PAM user mismatch" errors with SSH
+        uid_attr_map = "name";
+        gid_attr_map = "name";
+        # Create symlink /home/torjus -> /home/torjus@home.2rjus.net
+        home_alias = "name";
+      };
+    };
+  };
+}