exploring-llm-evaluations

Exploring LLM evaluations

PostHog evaluations score $ai_generation events. Each evaluation is one of two types, both first-class:

• hog — deterministic Hog code that returns true/false (and optionally N/A). Best for objective rule-based checks: format validation (JSON parses, schema matches), length limits, keyword presence/absence, regex patterns, structural assertions, latency thresholds, cost guards. Cheap, fast, reproducible — no LLM call per run. Prefer this when the criterion can be expressed as code.
• llm_judge — an LLM scores generations against a prompt you write. Best for subjective or fuzzy checks: tone, helpfulness, hallucination detection, off-topic drift, instruction-following. Costs an LLM call per run and requires AI data processing approval at the org level.

Results from both types land in ClickHouse as $ai_evaluation events with the same schema, so the read/query/summary workflows are identical regardless of evaluator type — the only thing that changes is whether $ai_evaluation_reasoning was written by Hog code or by an LLM.

This skill covers the full lifecycle: list/inspect/manage evaluation configs (Hog or LLM judge), run them on specific generations, query individual results, and get an AI-generated summary of pass/fail/N/A patterns across many runs.

Tools

All llma-evaluation-* tools are defined in products/ai_observability/mcp/tools.yaml.

Event schema

Every run of an evaluation emits an $ai_evaluation event. Key properties:

When $ai_evaluation_applicable = false, the run counts as N/A regardless of $ai_evaluation_result. For evaluations that don't support N/A, this property may be null — treat null as "applicable".

Workflow: investigate why an evaluation is failing

Works the same way for llm_judge and hog evaluations — the differences only matter when you eventually go to fix the evaluator (edit the prompt vs. edit the Hog source).

Step 1 — Find the evaluation

posthog:llma-evaluation-list
{ "search": "hallucination", "enabled": true }

Look at the returned id, name, evaluation_type, and either:

• evaluation_config.prompt for an llm_judge
• evaluation_config.source for a hog evaluator

The Hog source is the ground truth for why a hog evaluator passes or fails — read it before assuming the failure is in the generation.

Step 2 — Get the AI-generated summary

posthog:llma-evaluation-summary-create
{
  "evaluation_id": "<uuid>",
  "filter": "fail"
}

Returns:

• overall_assessment — natural-language summary
• fail_patterns — grouped patterns with title, description, frequency, and example_generation_ids
• pass_patterns and na_patterns — same shape, populated when filter includes them
• recommendations — actionable next steps
• statistics — total_analyzed, pass_count, fail_count, na_count

The endpoint analyses the most recent ~250 runs (EVALUATION_SUMMARY_MAX_RUNS). Results are cached for one hour per (evaluation_id, filter, set_of_generation_ids). Pass force_refresh: true to recompute.

Compare filters in two calls to spot what's distinctive about failures vs passes:

Then diff the pass_patterns against the fail_patterns from Step 2.

Step 3 — Drill into example failing runs

Each pattern surfaces example_generation_ids. Pull the underlying trace for the most representative example:

(If you only have a generation ID, query for it via execute-sql first to find the parent trace ID — see below.)

Step 4 — Verify the pattern with raw SQL

The summary is LLM-generated and should be verified. Use execute-sql to count and spot-check:

The N/A guard (IS NULL OR != false) is important — it matches the same logic the backend uses to bucket runs.

Workflow: run an evaluation against a specific generation

Use this when the user pastes a trace/generation URL and asks "what would evaluation X say about this?".

The timestamp is required for an efficient ClickHouse lookup of the target event. Pass distinct_id if you have it — it speeds up the lookup further.

Workflow: build and test a new evaluator

Hog evaluator (deterministic, code-based)

Reach for this first when the criterion is rule-based — it's cheaper, faster, and reproducible. Prototype with llma-evaluation-test-hog (no save):

The handler returns the boolean result for each of the most recent N $ai_generation events. Iterate on the source until it behaves as expected, then promote it via llma-evaluation-create:

Hog evaluators have full access to the event and its properties — common patterns include schema validation, length/token limits, regex matches, and tool-call shape checks. Because they're deterministic, results are reproducible across reruns and trivially diff-able.

LLM-judge evaluator (subjective, prompt-based)

Use this when the criterion is fuzzy and a code rule would be brittle (tone, factuality, helpfulness, on-topic-ness). There's no equivalent of llma-evaluation-test-hog for LLM judges — the typical loop is to create the evaluator with enabled: false, run it manually against a handful of representative generations via llma-evaluation-run, inspect the results, refine the prompt with llma-evaluation-update, and then flip enabled: true when you're satisfied:

Then dry-run against a known-good and a known-bad generation:

LLM judges require organisation AI data processing approval. Hog evaluators do not.

Workflow: manage the evaluation lifecycle

llm_judge evaluations require AI data processing approval at the org level (is_ai_data_processing_approved). The same gate applies to llma-evaluation-summary-create. Hog evaluations do not require this gate — they run as plain code on the ingestion pipeline.

When to use Hog vs LLM judge

Reach for Hog by default. Switch to LLM judge only when the criterion can't be expressed as code.

A common pattern is to layer them: a Hog evaluator gates obvious format/length violations cheaply, and an LLM-judge evaluator only fires on the generations that pass the Hog gate (via conditions).

Investigation patterns

The summarisation tool works the same way regardless of whether the evaluator is hog or llm_judge — it analyses the resulting $ai_evaluation events, not the evaluator itself. The fix path differs (edit Hog source vs. edit prompt) but the diagnosis is identical.

"Why is evaluation X suddenly failing more?"

1. llma-evaluation-list — confirm the evaluation is still enabled and unchanged (compare evaluation_config.source or evaluation_config.prompt to the version you expect)

2. llma-evaluation-summary-create with filter: "fail" — get the dominant failure patterns and example IDs

3. SQL count of fails per day to confirm the regression window:

   SELECT toDate(timestamp) AS day, count() AS fails
   FROM events
   WHERE event = '$ai_evaluation'
       AND properties.$ai_evaluation_id = '<uuid>'
       AND properties.$ai_evaluation_result = false
       AND timestamp >= now() - INTERVAL 30 DAY
   GROUP BY day
   ORDER BY day

4. Drill into a representative trace per pattern via query-llm-trace

"Are passes and fails caused by the same root content?"

1. Generate two summaries: one with filter: "pass", one with filter: "fail"
2. If pass_patterns and fail_patterns describe similar content:
   • For an llm_judge: the prompt or rubric is probably ambiguous — reword evaluation_config.prompt and use llma-evaluation-update
   • For a hog evaluator: the rule is probably under- or over-matching — read the source via llma-evaluation-get, narrow the predicate, and retest with llma-evaluation-test-hog before pushing the fix via llma-evaluation-update

"Did a Hog evaluator regression after a code change?"

Hog evaluators are reproducible — if the source hasn't changed, identical inputs should yield identical outputs. When fail rates jump for a Hog evaluator:

1. llma-evaluation-get — note the current source and updated_at
2. Spot-check the latest failing runs with the SQL query from Step 4 above
3. Re-run the source against those exact generations using llma-evaluation-test-hog with a modified conditions filter that targets them
4. If the test results match the live results, the change is in the generations, not the evaluator (a model upgrade, prompt change upstream, etc.) — investigate the producer
5. If they diverge, the evaluator was edited; check git history of the source field via the activity log

"What kinds of generations does this evaluator skip as N/A?"

posthog:llma-evaluation-summary-create
{ "evaluation_id": "<uuid>", "filter": "na" }

Inspect na_patterns to see whether the N/A logic is doing the right thing. If a pattern in na_patterns looks like something that should have been scored:

• For an llm_judge: the applicability instruction in the prompt is too broad — narrow it
• For a hog evaluator with output_config.allows_na: true: the source is returning null (or whatever the N/A signal is) too eagerly — tighten the precondition

"Score this single generation right now"

llma-evaluation-run with the trace's generation ID and timestamp. Useful for spot-checking or wiring evaluations into a larger agent loop.

Constructing UI links

• Evaluations list: https://app.posthog.com/ai-evals/evaluations
• Single evaluation: https://app.posthog.com/ai-evals/evaluations/<evaluation_id>
• Underlying generation/trace: see the exploring-llm-traces skill's URL conventions

Always surface the relevant link so the user can verify in the UI.

Tips

• The summary tool is rate-limited (burst, sustained, daily) and caches results for one hour — repeated calls with the same (evaluation_id, filter) are cheap; use force_refresh: true only when you genuinely need fresh analysis
• Pass generation_ids: [...] to scope a summary to a specific cohort of runs (max 250)
• The statistics block in the summary response is computed from raw data, not the LLM — trust those counts even if a pattern's frequency field is qualitative
• For rich filtering not supported by llma-evaluation-list (e.g. by author or model configuration), fall back to execute-sql against the evaluations Postgres table or the $ai_evaluation ClickHouse events
• When showing failure patterns to the user, always include 1-2 example trace links so they can validate the pattern visually
• llma-evaluation-* tools use evaluation:read for read tools and evaluation:write for mutating tools; llma-evaluation-summary-create uses llm_analytics:write
• Hog evaluators are reproducible — if you suspect a regression, llma-evaluation-test-hog with the suspect source against the failing generations is the fastest way to bisect whether the change is in the evaluator or in the producer of the generations
• LLM-judge evaluators are non-deterministic across reruns; expect 1-5% noise even with a fixed prompt and model. If you're chasing a small regression in fail rate, prefer Hog or pin a deterministic provider/seed in the model_configuration