improve-bootstrap-visibility #29

Merged

torjus merged 11 commits from improve-bootstrap-visibility into master 2026-02-07 15:00:09 +00:00

Conversation 0 Commits 11 Files Changed 5 +160 -13

torjus commented 2026-02-07 15:00:03 +00:00

Owner

Copy Link

Summary

This branch improves visibility into the VM bootstrap process, making it possible to monitor bootstrap status without console access.

Bootstrap Logging to Loki

• Added log_to_loki function that sends structured log entries directly to Loki via curl during bootstrap
• Logs include labels: job="bootstrap", host, branch, stage
• Bootstrap stages tracked: starting, network_ok, vault_ok/vault_skip/vault_warn, building, success, failed
• Fails silently if Loki is unreachable (won't break bootstrap)

TTY Visibility

• Bootstrap output now displays on tty1 via journal+console
• Getty greeting updated to indicate bootstrap image status
• Added ncurses for clear command availability

Infrastructure Updates

• Ansible playbook auto-updates terraform template name after deploy
• Template updated to nixos-25.11.20260203.e576e3c

CLAUDE.md Improvements

• Document bootstrap log queries in Loki
• Prefer nix develop -c <command> for devshell tools
• Use tofu -chdir= instead of cd for OpenTofu commands

Query Examples

{job="bootstrap"}                      # All bootstrap logs
{job="bootstrap", host="testvm01"}     # Specific host
{job="bootstrap", stage="failed"}      # All failures

## Summary This branch improves visibility into the VM bootstrap process, making it possible to monitor bootstrap status without console access. ### Bootstrap Logging to Loki - Added `log_to_loki` function that sends structured log entries directly to Loki via curl during bootstrap - Logs include labels: `job="bootstrap"`, `host`, `branch`, `stage` - Bootstrap stages tracked: `starting`, `network_ok`, `vault_ok`/`vault_skip`/`vault_warn`, `building`, `success`, `failed` - Fails silently if Loki is unreachable (won't break bootstrap) ### TTY Visibility - Bootstrap output now displays on tty1 via `journal+console` - Getty greeting updated to indicate bootstrap image status - Added ncurses for `clear` command availability ### Infrastructure Updates - Ansible playbook auto-updates terraform template name after deploy - Template updated to nixos-25.11.20260203.e576e3c ### CLAUDE.md Improvements - Document bootstrap log queries in Loki - Prefer `nix develop -c <command>` for devshell tools - Use `tofu -chdir=` instead of `cd` for OpenTofu commands ### Query Examples ``` {job="bootstrap"} # All bootstrap logs {job="bootstrap", host="testvm01"} # Specific host {job="bootstrap", stage="failed"} # All failures ```

torjus added 11 commits 2026-02-07 15:00:03 +00:00

template2: show bootstrap progress on tty1 ... a1ae766eb8

- Display bootstrap banner and live progress on tty1 instead of login prompt
- Add custom getty greeting on other ttys indicating this is a bootstrap image
- Disable getty on tty1 during bootstrap so output is visible

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

playbooks: auto-update terraform template name after deploy ... 6a3a51407e

Add a third play to build-and-deploy-template.yml that updates
terraform/variables.tf with the new template name after deploying
to Proxmox. Only updates if the template name has changed.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

terraform: update template to nixos-25.11.20260203.e576e3c ... 0cf72ec191

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

template2: add ncurses for clear command in bootstrap ... 78e8d7a600

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

terraform: add flake_branch and token for testvm01 ... 4ca3c8890f

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

template2: revert to journal+console output for bootstrap ... 11261c4636

TTY output was causing nixos-rebuild to fail. Keep the custom
greeting line to indicate bootstrap image, but use journal+console
for reliable logging.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

template2: send bootstrap status to Loki for remote monitoring ... ae3039af19

Adds log_to_loki function that pushes structured log entries to Loki
at key bootstrap stages (starting, network_ok, vault_*, building,
success, failed). Enables querying bootstrap state via LogQL without
console access.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

terraform: regenerate wrapped token for testvm01 ... 09c9df1bbe

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

CLAUDE.md: prefer nix develop -c for devshell commands ... a90d9c33d5

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

CLAUDE.md: use tofu -chdir instead of cd ... f19ba2f4b6

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

CLAUDE.md: document bootstrap logs in Loki ... eea000b337

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

torjus merged commit 13d6d0ea3a into master 2026-02-07 15:00:09 +00:00

torjus deleted branch improve-bootstrap-visibility 2026-02-07 15:00:10 +00:00

torjus referenced this issue from a commit 2026-02-07 15:00:11 +00:00

Merge pull request 'improve-bootstrap-visibility' (#29) from improve-bootstrap-visibility into master

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

torjus

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: torjus/nixos-servers#29