From a68e02aa0224800205a1f133e0b2850480ffe343 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Torjus=20H=C3=A5kestad?= Date: Tue, 3 Feb 2026 00:54:15 +0100 Subject: [PATCH] fixup! vault: implement bootstrap integration --- TODO.md | 130 +++++++++++++++++++++++++------------ terraform/vault/secrets.tf | 2 +- terraform/vms.tf | 2 +- 3 files changed, 89 insertions(+), 45 deletions(-) diff --git a/TODO.md b/TODO.md index ab93fc4..77a5068 100644 --- a/TODO.md +++ b/TODO.md @@ -197,18 +197,18 @@ vault01.home.2rjus.net (10.69.13.19) ├─ SSH CA Engine (TODO: Phase 4c) └─ AppRole Auth (per-host authentication configured) ↓ - [Phase 4d] New hosts authenticate on first boot - [Phase 4d] Fetch secrets via Vault API + [✅ Phase 4d] New hosts authenticate on first boot + [✅ Phase 4d] Fetch secrets via Vault API No manual key distribution needed ``` **Completed:** - ✅ Phase 4a: OpenBao server with TPM2 auto-unseal - ✅ Phase 4b: Infrastructure-as-code (secrets, policies, AppRoles, PKI) +- ✅ Phase 4d: Bootstrap integration for automated secrets access **Next Steps:** - Phase 4c: Migrate from step-ca to OpenBao PKI -- Phase 4d: Bootstrap integration for automated secrets access --- @@ -388,55 +388,99 @@ vault01.home.2rjus.net (10.69.13.19) --- -#### Phase 4d: Bootstrap Integration +#### Phase 4d: Bootstrap Integration ✅ COMPLETED (2026-02-02) **Goal:** New hosts automatically authenticate to Vault on first boot, no manual steps **Tasks:** -- [ ] Update create-host tool - - [ ] Generate AppRole role_id + secret_id for new host - - [ ] Or create wrapped token for one-time bootstrap - - [ ] Add host-specific policy to Vault (via terraform) - - [ ] Store bootstrap credentials for cloud-init injection -- [ ] Update template2 for Vault authentication - - [ ] Create Vault authentication module - - [ ] Reads bootstrap credentials from cloud-init - - [ ] Authenticates to Vault, retrieves permanent AppRole credentials - - [ ] Stores role_id + secret_id locally for services to use -- [ ] Create NixOS Vault secrets module - - [ ] Replacement for sops.secrets - - [ ] Fetches secrets from Vault at nixos-rebuild/activation time - - [ ] Or runtime secret fetching for services - - [ ] Handle Vault token renewal -- [ ] Update bootstrap service - - [ ] After authenticating to Vault, fetch any bootstrap secrets - - [ ] Run nixos-rebuild with host configuration - - [ ] Services automatically fetch their secrets from Vault -- [ ] Update terraform cloud-init - - [ ] Inject Vault address and bootstrap credentials - - [ ] Pass via cloud-init user-data or write_files - - [ ] Credentials scoped to single use or short TTL -- [ ] Test complete flow - - [ ] Run create-host to generate new host config - - [ ] Deploy with terraform - - [ ] Verify host bootstraps and authenticates to Vault - - [ ] Verify services can fetch secrets - - [ ] Confirm no manual steps required +- [x] Update create-host tool + - [x] Generate wrapped token (24h TTL, single-use) for new host + - [x] Add host-specific policy to Vault (via terraform/vault/hosts-generated.tf) + - [x] Store wrapped token in terraform/vms.tf for cloud-init injection + - [x] Add `--regenerate-token` flag to regenerate only the token without overwriting config +- [x] Update template2 for Vault authentication + - [x] Reads wrapped token from cloud-init (/run/cloud-init-env) + - [x] Unwraps token to get role_id + secret_id + - [x] Stores AppRole credentials in /var/lib/vault/approle/ (persistent) + - [x] Graceful fallback if Vault unavailable during bootstrap +- [x] Create NixOS Vault secrets module (system/vault-secrets.nix) + - [x] Runtime secret fetching (services fetch on start, not at nixos-rebuild time) + - [x] Secrets cached in /var/lib/vault/cache/ for fallback when Vault unreachable + - [x] Secrets written to /run/secrets/ (tmpfs, cleared on reboot) + - [x] Fresh authentication per service start (no token renewal needed) + - [x] Optional periodic rotation with systemd timers + - [x] Critical service protection (no auto-restart for DNS, CA, Vault itself) +- [x] Create vault-fetch helper script + - [x] Standalone tool for fetching secrets from Vault + - [x] Authenticates using AppRole credentials + - [x] Writes individual files per secret key + - [x] Handles caching and fallback logic +- [x] Update bootstrap service (hosts/template2/bootstrap.nix) + - [x] Unwraps Vault token on first boot + - [x] Stores persistent AppRole credentials + - [x] Continues with nixos-rebuild + - [x] Services fetch secrets when they start +- [x] Update terraform cloud-init (terraform/cloud-init.tf) + - [x] Inject VAULT_ADDR and VAULT_WRAPPED_TOKEN via write_files + - [x] Write to /run/cloud-init-env (tmpfs, cleaned on reboot) + - [x] Fixed YAML indentation issues (write_files at top level) + - [x] Support flake_branch alongside vault credentials +- [x] Test complete flow + - [x] Created vaulttest01 test host + - [x] Verified bootstrap with Vault integration + - [x] Verified service secret fetching + - [x] Tested cache fallback when Vault unreachable + - [x] Tested wrapped token single-use (second bootstrap fails as expected) + - [x] Confirmed zero manual steps required -**Bootstrap flow:** +**Implementation Details:** + +**Wrapped Token Security:** +- Single-use tokens prevent reuse if leaked +- 24h TTL limits exposure window +- Safe to commit to git (expired/used tokens useless) +- Regenerate with `create-host --hostname X --regenerate-token` + +**Secret Fetching:** +- Runtime (not build-time) keeps secrets out of Nix store +- Cache fallback enables service availability when Vault down +- Fresh authentication per service start (no renewal complexity) +- Individual files per secret key for easy consumption + +**Bootstrap Flow:** ``` -1. terraform apply (deploys VM with cloud-init) -2. Cloud-init sets hostname + Vault bootstrap credentials +1. create-host --hostname myhost --ip 10.69.13.x/24 + ↓ Generates wrapped token, updates terraform +2. tofu apply (deploys VM with cloud-init) + ↓ Cloud-init writes wrapped token to /run/cloud-init-env 3. nixos-bootstrap.service runs: - - Authenticates to Vault with bootstrap credentials - - Retrieves permanent AppRole credentials - - Stores locally for service use - - Runs nixos-rebuild -4. Host services fetch secrets from Vault as needed -5. Done - no manual intervention + ↓ Unwraps token → gets role_id + secret_id + ↓ Stores in /var/lib/vault/approle/ (persistent) + ↓ Runs nixos-rebuild boot +4. Service starts → fetches secrets from Vault + ↓ Uses stored AppRole credentials + ↓ Caches secrets for fallback +5. Done - zero manual intervention ``` -**Deliverable:** Fully automated secrets access from first boot, zero manual steps +**Files Created:** +- `scripts/vault-fetch/` - Secret fetching helper (Nix package) +- `system/vault-secrets.nix` - NixOS module for declarative Vault secrets +- `scripts/create-host/vault_helper.py` - Vault API integration +- `terraform/vault/hosts-generated.tf` - Auto-generated host policies +- `docs/vault-bootstrap-implementation.md` - Architecture documentation +- `docs/vault-bootstrap-testing.md` - Testing guide + +**Configuration:** +- Vault address: `https://vault01.home.2rjus.net:8200` (configurable) +- All defaults remain configurable via environment variables or NixOS options + +**Next Steps:** +- Gradually migrate existing services from sops-nix to Vault +- Add CNAME for vault.home.2rjus.net → vault01.home.2rjus.net +- Phase 4c: Migrate from step-ca to OpenBao PKI (future) + +**Deliverable:** ✅ Fully automated secrets access from first boot, zero manual steps --- diff --git a/terraform/vault/secrets.tf b/terraform/vault/secrets.tf index 72f3996..8c15df7 100644 --- a/terraform/vault/secrets.tf +++ b/terraform/vault/secrets.tf @@ -49,7 +49,7 @@ locals { # TODO: Remove after testing "hosts/vaulttest01/test-service" = { auto_generate = true - password_length = 24 + password_length = 32 } } } diff --git a/terraform/vms.tf b/terraform/vms.tf index 76eb802..92f71d2 100644 --- a/terraform/vms.tf +++ b/terraform/vms.tf @@ -51,7 +51,7 @@ locals { memory = 2048 disk_size = "20G" flake_branch = "vault-bootstrap-integration" - vault_wrapped_token = "s.aLlvvgIX4RegyBZKwnDIplJ4" + vault_wrapped_token = "s.HwNenAYvXBsPs8uICh4CbE11" } }