torjus/nixos-servers
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects 1 Releases Wiki Activity

docs: update Loki queries from host to hostname label
Some checks failed
Run nix flake check / flake-check (push) Has been cancelled
Details

Browse Source 
Update all LogQL examples, agent instructions, and scripts to use
the hostname label instead of host, matching the Prometheus label
naming convention. Also update pipe-to-loki and bootstrap scripts
to push hostname instead of host.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Torjus Håkestad
2026-02-13 23:43:47 +01:00
parent 7b804450a3
commit d485948df0
8 changed files with 63 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
docs/plans/loki-improvements.md
View File

@@ -86,13 +86,13 @@ These are generous limits that shouldn't affect normal operation but protect aga
- The `varlog` scrape config uses `hostname` while journal uses `host` (different label name)
- No `tier` or `role` labels, making it hard to filter logs by deployment tier or host function
**Recommendations:**
**Implemented:** Standardized on `hostname` to match Prometheus labels. The journal scrape previously used a relabel from `__journal__hostname` to `host`; now both scrape configs use a static `hostname` label from `config.networking.hostName`. Also updated `pipe-to-loki` and bootstrap scripts to use `hostname` instead of `host`.
1. **Fix varlog label:** Rename `hostname` to `host` for consistency with journal scrape config
2. **Add `tier` label:** Static label from `config.homelab.host.tier` (`test`/`prod`) on both scrape configs
3. **Add `role` label:** Static label from `config.homelab.host.role` on both scrape configs, only when set (10 hosts have no role, so omit to keep labels clean)
1. **Standardized label:** Both scrape configs use `hostname` (matching Prometheus) via shared `hostLabels`
2. **Added `tier` label:** Static label from `config.homelab.host.tier` (`test`/`prod`) on both scrape configs
3. **Added `role` label:** Static label from `config.homelab.host.role` on both scrape configs (conditionally, only when non-null)
No cardinality impact - `tier` and `role` are 1:1 with `host`, so they add metadata to existing streams without creating new ones.
No cardinality impact - `tier` and `role` are 1:1 with `hostname`, so they add metadata to existing streams without creating new ones.
This enables queries like:
- `{tier="prod"} |= "error"` - all errors on prod hosts
@@ -167,10 +167,10 @@ For each service, check whether it supports a JSON log format option and whether
1. Add `compactor` section to `services/monitoring/loki.nix`
2. Add `limits_config` with 30-day retention and basic rate limits
3. Update `system/monitoring/logs.nix`:
- Fix `hostname` → `host` label in varlog scrape config
- Add `tier` static label from `config.homelab.host.tier` to both scrape configs
- Add `role` static label from `config.homelab.host.role` (conditionally, only when set) to both scrape configs
- Add pipeline stages to journal scrape config: `json` to extract PRIORITY, `template` to map to level name, `labels` to attach as `level`
- ~~Fix `hostname` → `host` label in varlog scrape config~~ Done: standardized on `hostname` (matching Prometheus)
- ~~Add `tier` static label from `config.homelab.host.tier` to both scrape configs~~ Done
- ~~Add `role` static label from `config.homelab.host.role` (conditionally, only when set) to both scrape configs~~ Done
- ~~Add pipeline stages to journal scrape config: `json` to extract PRIORITY, `template` to map to level name, `labels` to attach as `level`~~ Done
4. Deploy to monitoring01, verify compactor runs and old data gets cleaned
5. Verify `level` label works: `{level="error"}` should return results, and match cases where `detected_level="unknown"`