docs: add JSON logging audit to Loki improvements plan
Some checks failed
Run nix flake check / flake-check (push) Failing after 15m38s
Some checks failed
Run nix flake check / flake-check (push) Failing after 15m38s
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -125,7 +125,25 @@ This enables queries like:
|
|||||||
- `{level=~"critical|error", tier="prod"}` - prod errors and criticals
|
- `{level=~"critical|error", tier="prod"}` - prod errors and criticals
|
||||||
- `{level="warning", role="dns"}` - warnings from DNS servers
|
- `{level="warning", role="dns"}` - warnings from DNS servers
|
||||||
|
|
||||||
### 6. Monitoring CNAME for Promtail Target
|
### 6. Enable JSON Logging on Services
|
||||||
|
|
||||||
|
**Problem:** Many services support structured JSON log output but may be using plain text by default. JSON logs are significantly easier to query in Loki - `| json` cleanly extracts all fields, whereas plain text requires fragile regex or pattern matching.
|
||||||
|
|
||||||
|
**Recommendation:** Audit all configured services and enable JSON logging where supported. Candidates to check include:
|
||||||
|
- Caddy (already JSON by default)
|
||||||
|
- Prometheus / Alertmanager / Loki / Tempo
|
||||||
|
- Grafana
|
||||||
|
- NSD / Unbound
|
||||||
|
- Home Assistant
|
||||||
|
- NATS
|
||||||
|
- Jellyfin
|
||||||
|
- OpenBao (Vault)
|
||||||
|
- Kanidm
|
||||||
|
- Garage
|
||||||
|
|
||||||
|
For each service, check whether it supports a JSON log format option and whether enabling it would break anything (e.g., log volume increase from verbose JSON, or dashboards that parse text format).
|
||||||
|
|
||||||
|
### 7. Monitoring CNAME for Promtail Target
|
||||||
|
|
||||||
**Problem:** Promtail hardcodes `monitoring01.home.2rjus.net:3100`. The VictoriaMetrics migration plan already addresses this by switching to a `monitoring` CNAME.
|
**Problem:** Promtail hardcodes `monitoring01.home.2rjus.net:3100`. The VictoriaMetrics migration plan already addresses this by switching to a `monitoring` CNAME.
|
||||||
|
|
||||||
@@ -139,7 +157,8 @@ This enables queries like:
|
|||||||
| 2 | **Limits config** | Low | Medium | Do with retention - minimal additional effort |
|
| 2 | **Limits config** | Low | Medium | Do with retention - minimal additional effort |
|
||||||
| 3 | **Promtail label fix** | Trivial | Low | Quick fix, do with other label changes |
|
| 3 | **Promtail label fix** | Trivial | Low | Quick fix, do with other label changes |
|
||||||
| 4 | **Journal priority → level** | Low-medium | Medium | Reliable level filtering across the fleet |
|
| 4 | **Journal priority → level** | Low-medium | Medium | Reliable level filtering across the fleet |
|
||||||
| 5 | **Monitoring CNAME** | Low | Medium | Part of monitoring02 migration |
|
| 5 | **JSON logging audit** | Low-medium | Medium | Audit services, enable JSON where supported |
|
||||||
|
| 6 | **Monitoring CNAME** | Low | Medium | Part of monitoring02 migration |
|
||||||
|
|
||||||
## Implementation Steps
|
## Implementation Steps
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user