authoring-log-alerts

Authoring log alerts

Authoring an alert is a measurement problem, not a guessing problem. You are not trying to be exhaustive — you are trying to land thresholds that fire 0–3 times per week on real production patterns, on services that matter.

When to use this skill

The user asks to "set up alerts" / "suggest alerts" for their project.
The user wants to evaluate whether a service is producing alertable signal.
The user has just enabled log alerting and wants a starter set.

When not to use this skill

Tuning an alert that already exists — that's a different job (use posthog:logs-alerts-events-list to inspect fire/resolve cadence and posthog:logs-alerts-partial-update to adjust).
Investigating an active incident — pull rows with posthog:query-logs, don't author an alert mid-incident.

Tools

Tool	Job	Where it fits
`posthog:logs-services`	Top-25 services in window with log_count, error_count, error_rate, sparkline.	Step 1 — triage.
`posthog:logs-attributes-list` / `posthog:logs-attribute-values-list`	Discover keys/values for narrower filters.	Step 2, optional.
`posthog:logs-count-ranges`	Adaptive time-bucketed counts for a filter.	Step 3 — baseline.
`posthog:logs-alerts-simulate-create`	Replay a draft config against `-7d` history with full state machine.	Step 4 — validate.
`posthog:logs-alerts-create`	Persist the alert.	Step 5 — ship.
`posthog:logs-alerts-destinations-create`	Wire the alert to Slack, webhook, or Microsoft Teams.	Step 5: ship.

Do not call posthog:query-logs during authoring. You need distributions, not rows. Reserve posthog:query-logs for the very end if the user asks "show me a sample of what would have fired" — limit: 10 is plenty.

Workflow

1. Triage — pick candidate services

Call posthog:logs-services for the last 24h with no filters. The response is capped at 25 services and includes a sparkline, so it is small and bounded.

A service is a candidate when both are true:

log_count is non-trivial (≥ ~1k in 24h — quieter services produce too little signal to alert on).
error_rate is non-zero, or the user has named the service explicitly.

Skip services with high volume but error_rate == 0 unless the user wants a volume-shape alert (e.g. "warn me if api-gateway suddenly stops producing logs"). Volume-floor alerts use threshold_operator: below and need different reasoning — see references/volume-floor-alerts.md.

If the user names a service, treat it as a candidate even without error signal.

2. (Optional) Narrow the filter

If a service has many error sub-types, an alert on "all errors" is usually too broad. Use posthog:logs-attributes-list (try attribute_type: log) and posthog:logs-attribute-values-list to find a discriminator — common ones are http.status_code, error.type, k8s.container.name. Add the narrowing filter to your draft.

Keep it simple: one severity filter + one or two attribute filters is plenty. Multi-clause filters are harder to reason about and rarely improve precision.

3. Baseline — characterise the candidate over 7 days

Call posthog:logs-count-ranges with the candidate's filters, dateRange: { date_from: "-7d" }, and targetBuckets: 24 (one bucket ≈ 7h). The response gives you bucket counts.

Do not eyeball the percentiles or scale the threshold to the alert window manually. Pipe the count-ranges response into the helper script:

echo '<count-ranges JSON>' | python3 scripts/baseline_stats.py --window-minutes 5

The script returns:

{
  "n_buckets": 12,
  "bucket_minutes": 420.0,
  "alert_window_minutes": 5,

Use suggested_threshold_count as your starting threshold. Read health:

`health` flag	What it means	What to do
`sparse:N_of_M_buckets`	Too few non-empty buckets for a 7d baseline.	Widen filter, extend to `-30d`, or skip.
`empty`	All buckets are zero.	Skip — no signal.
`spiky`	`max` is 10×+ `p95`.	Count-threshold alerts work well. Proceed.
`flat`	`p95` ≈ `p50`.	Be cautious — either no incidents in lookback, or the metric is too smooth. Try a longer lookback or skip.
`[]` (empty)	Healthy distribution.	Proceed.

4. Draft and simulate

Pick a starter draft from these defaults — see references/threshold-defaults.md for the reasoning:

Setting	Default	Notes
`threshold_count`	`suggested_threshold_count` from the script	Already scaled to the alert window.
`threshold_operator`	`above`	Use `below` only for volume-floor alerts.
`window_minutes`	`5`	Allowed: 5, 10, 15, 30, 60. Must match what you passed to the script.
`evaluation_periods`	`3`	M in N-of-M.
`datapoints_to_alarm`	`2`	N in N-of-M. 2-of-3 reduces flap from a single noisy bucket.
`cooldown_minutes`	`30`	Minimum time between repeat fires.

Call posthog:logs-alerts-simulate-create with these settings and date_from: "-7d". The response gives you fire_count and resolve_count.

5. Iterate — three rounds, then ship or skip

Target: fire_count between 0 and ~3 over -7d. If outside the band:

Outcome	Adjustment
`fire_count` = 0 over 7d and the baseline was spiky	Lower `threshold_count` toward `stats.p95` from the script, or drop to 1-of-2.
`fire_count` = 0 and the baseline was flat	The service has no alertable signal. Skip it; log why.
`fire_count` > 5	Raise `threshold_count` toward `stats.max` from the script, or move to 3-of-5 for a smoother window.
`fire_count` is fine but resolve_count never matches fire_count	Cooldown is too long, or the underlying state is genuinely sticky. Acceptable for now.

When adjusting the threshold, read values from the script's stats block — never recompute percentiles by hand.

Cap iteration at 3 simulate calls per candidate. If you can't land in the band in 3 rounds, the metric is wrong — either the filter is too broad, the window is wrong, or the service genuinely doesn't have a threshold-shape signal. Note it and move on.

6. Ship — create + attach destination

Once a draft simulates cleanly:

Call posthog:logs-alerts-create with the validated config. Use a name like <service> error rate (auto) so the user can see at a glance which alerts came from this skill.
Call posthog:logs-alerts-destinations-create to wire it to a notification target. An alert with no destination is silent. Supported destination fields:
- Slack: type: "slack", slack_workspace_id, and slack_channel_id. slack_channel_name is optional.
- Webhook: type: "webhook" and webhook_url.
- Microsoft Teams: type: "teams" and webhook_url.
Always confirm the channel name or webhook URL with the user before attaching. Never wire an auto-generated alert to a production channel without explicit confirmation. If the user is unsure, suggest a low-traffic testing channel for the first few alerts.

If the user wants alerts created in enabled: false state for review-then-flip, pass enabled: false to -create and tell them how many drafts you produced.

Filter shape — required

The filters field on posthog:logs-alerts-create takes a subset of LogsViewerFilters and must contain at least one of:

severityLevels — list of ["trace","debug","info","warn","error","fatal"]
serviceNames — list of service name strings
filterGroup — property filter group

The same shape goes into posthog:logs-alerts-simulate-create's filters field. Match the simulate filters to the alert filters exactly — otherwise the simulation is testing a different alert than the one you ship.

Example minimum:

Token-economy rules

One posthog:logs-services call at the start, not per-candidate.
One posthog:logs-count-ranges call per candidate at targetBuckets: 24. Don't go above 30 during authoring.
≤ 3 posthog:logs-alerts-simulate-create calls per candidate.
Zero posthog:query-logs calls during the authoring loop.
Prefer reporting a small set of well-validated alerts over a long list of unvalidated drafts.

Output

Report what you did, in this shape:

For each shipped alert: name, filters, threshold, simulated fire_count over 7d, destination.
For each skipped candidate: service name + why (flat baseline, can't land threshold, low volume).
Total simulate calls made, total alerts created.

The user should be able to read this and decide whether to disable any drafts before they go live.