fixup! vault: implement bootstrap integration

TODO.md

@@ -197,18 +197,18 @@ vault01.home.2rjus.net (10.69.13.19)
   ├─ SSH CA Engine (TODO: Phase 4c)
   └─ AppRole Auth (per-host authentication configured)
        ↓
-   [Phase 4d] New hosts authenticate on first boot
+   [✅ Phase 4d] New hosts authenticate on first boot
-   [Phase 4d] Fetch secrets via Vault API
+   [✅ Phase 4d] Fetch secrets via Vault API
    No manual key distribution needed
 ```
 
 **Completed:**
 - ✅ Phase 4a: OpenBao server with TPM2 auto-unseal
 - ✅ Phase 4b: Infrastructure-as-code (secrets, policies, AppRoles, PKI)
+- ✅ Phase 4d: Bootstrap integration for automated secrets access
 
 **Next Steps:**
 - Phase 4c: Migrate from step-ca to OpenBao PKI
-- Phase 4d: Bootstrap integration for automated secrets access
 
 ---
 
@@ -388,55 +388,99 @@ vault01.home.2rjus.net (10.69.13.19)
 
 ---
 
-#### Phase 4d: Bootstrap Integration
+#### Phase 4d: Bootstrap Integration ✅ COMPLETED (2026-02-02)
 
 **Goal:** New hosts automatically authenticate to Vault on first boot, no manual steps
 
 **Tasks:**
-- [ ] Update create-host tool
+- [x] Update create-host tool
-  - [ ] Generate AppRole role_id + secret_id for new host
+  - [x] Generate wrapped token (24h TTL, single-use) for new host
-  - [ ] Or create wrapped token for one-time bootstrap
+  - [x] Add host-specific policy to Vault (via terraform/vault/hosts-generated.tf)
-  - [ ] Add host-specific policy to Vault (via terraform)
+  - [x] Store wrapped token in terraform/vms.tf for cloud-init injection
-  - [ ] Store bootstrap credentials for cloud-init injection
+  - [x] Add `--regenerate-token` flag to regenerate only the token without overwriting config
-- [ ] Update template2 for Vault authentication
+- [x] Update template2 for Vault authentication
-  - [ ] Create Vault authentication module
+  - [x] Reads wrapped token from cloud-init (/run/cloud-init-env)
-  - [ ] Reads bootstrap credentials from cloud-init
+  - [x] Unwraps token to get role_id + secret_id
-  - [ ] Authenticates to Vault, retrieves permanent AppRole credentials
+  - [x] Stores AppRole credentials in /var/lib/vault/approle/ (persistent)
-  - [ ] Stores role_id + secret_id locally for services to use
+  - [x] Graceful fallback if Vault unavailable during bootstrap
-- [ ] Create NixOS Vault secrets module
+- [x] Create NixOS Vault secrets module (system/vault-secrets.nix)
-  - [ ] Replacement for sops.secrets
+  - [x] Runtime secret fetching (services fetch on start, not at nixos-rebuild time)
-  - [ ] Fetches secrets from Vault at nixos-rebuild/activation time
+  - [x] Secrets cached in /var/lib/vault/cache/ for fallback when Vault unreachable
-  - [ ] Or runtime secret fetching for services
+  - [x] Secrets written to /run/secrets/ (tmpfs, cleared on reboot)
-  - [ ] Handle Vault token renewal
+  - [x] Fresh authentication per service start (no token renewal needed)
-- [ ] Update bootstrap service
+  - [x] Optional periodic rotation with systemd timers
-  - [ ] After authenticating to Vault, fetch any bootstrap secrets
+  - [x] Critical service protection (no auto-restart for DNS, CA, Vault itself)
-  - [ ] Run nixos-rebuild with host configuration
+- [x] Create vault-fetch helper script
-  - [ ] Services automatically fetch their secrets from Vault
+  - [x] Standalone tool for fetching secrets from Vault
-- [ ] Update terraform cloud-init
+  - [x] Authenticates using AppRole credentials
-  - [ ] Inject Vault address and bootstrap credentials
+  - [x] Writes individual files per secret key
-  - [ ] Pass via cloud-init user-data or write_files
+  - [x] Handles caching and fallback logic
-  - [ ] Credentials scoped to single use or short TTL
+- [x] Update bootstrap service (hosts/template2/bootstrap.nix)
-- [ ] Test complete flow
+  - [x] Unwraps Vault token on first boot
-  - [ ] Run create-host to generate new host config
+  - [x] Stores persistent AppRole credentials
-  - [ ] Deploy with terraform
+  - [x] Continues with nixos-rebuild
-  - [ ] Verify host bootstraps and authenticates to Vault
+  - [x] Services fetch secrets when they start
-  - [ ] Verify services can fetch secrets
+- [x] Update terraform cloud-init (terraform/cloud-init.tf)
-  - [ ] Confirm no manual steps required
+  - [x] Inject VAULT_ADDR and VAULT_WRAPPED_TOKEN via write_files
+  - [x] Write to /run/cloud-init-env (tmpfs, cleaned on reboot)
+  - [x] Fixed YAML indentation issues (write_files at top level)
+  - [x] Support flake_branch alongside vault credentials
+- [x] Test complete flow
+  - [x] Created vaulttest01 test host
+  - [x] Verified bootstrap with Vault integration
+  - [x] Verified service secret fetching
+  - [x] Tested cache fallback when Vault unreachable
+  - [x] Tested wrapped token single-use (second bootstrap fails as expected)
+  - [x] Confirmed zero manual steps required
 
-**Bootstrap flow:**
+**Implementation Details:**
+
+**Wrapped Token Security:**
+- Single-use tokens prevent reuse if leaked
+- 24h TTL limits exposure window
+- Safe to commit to git (expired/used tokens useless)
+- Regenerate with `create-host --hostname X --regenerate-token`
+
+**Secret Fetching:**
+- Runtime (not build-time) keeps secrets out of Nix store
+- Cache fallback enables service availability when Vault down
+- Fresh authentication per service start (no renewal complexity)
+- Individual files per secret key for easy consumption
+
+**Bootstrap Flow:**
 ```
-1. terraform apply (deploys VM with cloud-init)
+1. create-host --hostname myhost --ip 10.69.13.x/24
-2. Cloud-init sets hostname + Vault bootstrap credentials
+   ↓ Generates wrapped token, updates terraform
+2. tofu apply (deploys VM with cloud-init)
+   ↓ Cloud-init writes wrapped token to /run/cloud-init-env
 3. nixos-bootstrap.service runs:
-   - Authenticates to Vault with bootstrap credentials
+   ↓ Unwraps token → gets role_id + secret_id
-   - Retrieves permanent AppRole credentials
+   ↓ Stores in /var/lib/vault/approle/ (persistent)
-   - Stores locally for service use
+   ↓ Runs nixos-rebuild boot
-   - Runs nixos-rebuild
+4. Service starts → fetches secrets from Vault
-4. Host services fetch secrets from Vault as needed
+   ↓ Uses stored AppRole credentials
-5. Done - no manual intervention
+   ↓ Caches secrets for fallback
+5. Done - zero manual intervention
 ```
 
-**Deliverable:** Fully automated secrets access from first boot, zero manual steps
+**Files Created:**
+- `scripts/vault-fetch/` - Secret fetching helper (Nix package)
+- `system/vault-secrets.nix` - NixOS module for declarative Vault secrets
+- `scripts/create-host/vault_helper.py` - Vault API integration
+- `terraform/vault/hosts-generated.tf` - Auto-generated host policies
+- `docs/vault-bootstrap-implementation.md` - Architecture documentation
+- `docs/vault-bootstrap-testing.md` - Testing guide
+
+**Configuration:**
+- Vault address: `https://vault01.home.2rjus.net:8200` (configurable)
+- All defaults remain configurable via environment variables or NixOS options
+
+**Next Steps:**
+- Gradually migrate existing services from sops-nix to Vault
+- Add CNAME for vault.home.2rjus.net → vault01.home.2rjus.net
+- Phase 4c: Migrate from step-ca to OpenBao PKI (future)
+
+**Deliverable:** ✅ Fully automated secrets access from first boot, zero manual steps
 
 ---
 

terraform/vault/secrets.tf

@@ -49,7 +49,7 @@ locals {
     # TODO: Remove after testing
     "hosts/vaulttest01/test-service" = {
       auto_generate = true
-      password_length = 24
+      password_length = 32
     }
   }
 }

terraform/vms.tf

@@ -51,7 +51,7 @@ locals {
       memory    = 2048
       disk_size = "20G"
       flake_branch = "vault-bootstrap-integration"
-      vault_wrapped_token = "s.aLlvvgIX4RegyBZKwnDIplJ4"
+      vault_wrapped_token = "s.HwNenAYvXBsPs8uICh4CbE11"
     }
   }