skill-judge

Skill Judge

Evaluate Agent Skills against official specifications and patterns derived from 17+ official examples.

Core Philosophy

What is a Skill?

A Skill is NOT a tutorial. A Skill is a knowledge externalization mechanism.

Traditional AI knowledge is locked in model parameters. To teach new capabilities:

Traditional: Collect data → GPU cluster → Train → Deploy new version
Cost: $10,000 - $1,000,000+
Timeline: Weeks to months

Skills change this:

Skill: Edit SKILL.md → Save → Takes effect on next invocation
Cost: $0
Timeline: Instant

This is the paradigm shift from "training AI" to "educating AI" — like a hot-swappable LoRA adapter that requires no training. You edit a Markdown file in natural language, and the model's behavior changes.

The Core Formula

Good Skill = Expert-only Knowledge − What Claude Already Knows

A Skill's value is measured by its knowledge delta — the gap between what it provides and what the model already knows.

• Expert-only knowledge: Decision trees, trade-offs, edge cases, anti-patterns, domain-specific thinking frameworks — things that take years of experience to accumulate
• What Claude already knows: Basic concepts, standard library usage, common programming patterns, general best practices

When a Skill explains "what is PDF" or "how to write a for-loop", it's compressing knowledge Claude already has. This is token waste — context window is a public resource shared with system prompts, conversation history, other Skills, and user requests.

Tool vs Skill

Tools define capability boundaries — without bash tool, model can't execute commands. Skills inject knowledge — without frontend-design Skill, model produces generic UI.

The equation:

General Agent + Excellent Skill = Domain Expert Agent

Same Claude model, different Skills loaded, becomes different experts.

Three Types of Knowledge in Skills

When evaluating, categorize each section:

The art of Skill design is maximizing Expert content, using Activation sparingly, and eliminating Redundant ruthlessly.

Evaluation Dimensions (120 points total)

D1: Knowledge Delta (20 points) — THE CORE DIMENSION

The most important dimension. Does the Skill add genuine expert knowledge?

Red flags (instant score ≤5):

• "What is [basic concept]" sections
• Step-by-step tutorials for standard operations
• Explaining how to use common libraries
• Generic best practices ("write clean code", "handle errors")
• Definitions of industry-standard terms

Green flags (indicators of high knowledge delta):

• Decision trees for non-obvious choices ("when X fails, try Y because Z")
• Trade-offs only an expert would know ("A is faster but B handles edge case C")
• Edge cases from real-world experience
• "NEVER do X because [non-obvious reason]"
• Domain-specific thinking frameworks

Evaluation questions:

1. For each section, ask: "Does Claude already know this?"
2. If explaining something, ask: "Is this explaining TO Claude or FOR Claude?"
3. Count paragraphs that are Expert vs Activation vs Redundant

D2: Mindset + Appropriate Procedures (15 points)

Does the Skill transfer expert thinking patterns along with necessary domain-specific procedures?

The difference between experts and novices isn't "knowing how to operate" — it's "how to think about the problem." But thinking patterns alone aren't enough when Claude lacks domain-specific procedural knowledge.

Key distinction:

What counts as valuable procedures:

• Workflows Claude hasn't been trained on (new tools, proprietary systems)
• Correct ordering that's non-obvious (e.g., "validate BEFORE packing, not after")
• Critical steps that are easy to miss (e.g., "MUST recalculate formulas after editing")
• Domain-specific sequences (e.g., MCP server's 4-phase development process)

What counts as redundant procedures:

• Generic file operations (open, read, write, save)
• Standard programming patterns (loops, conditionals, error handling)
• Common library usage that's well-documented

Expert thinking patterns look like:

Before [action], ask yourself:
- **Purpose**: What problem does this solve? Who uses it?
- **Constraints**: What are the hidden requirements?

Valuable domain procedures look like:

Redundant generic procedures look like:

The test:

1. Does it tell Claude WHAT to think about? (thinking patterns)
2. Does it tell Claude HOW to do things it wouldn't know? (domain procedures)

A good Skill provides both when needed.

D3: Anti-Pattern Quality (15 points)

Does the Skill have effective NEVER lists?

Why this matters: Half of expert knowledge is knowing what NOT to do. A senior designer sees purple gradient on white background and instinctively cringes — "too AI-generated." This intuition for "what absolutely not to do" comes from stepping on countless landmines.

Claude hasn't stepped on these landmines. It doesn't know Inter font is overused, doesn't know purple gradients are the signature of AI-generated content. Good Skills must explicitly state these "absolute don'ts."

Expert anti-patterns (specific + reason):

Weak anti-patterns (vague, no reasoning):

The test: Would an expert read the anti-pattern list and say "yes, I learned this the hard way"? Or would they say "this is obvious to everyone"?

D4: Specification Compliance — Especially Description (15 points)

Does the Skill follow official format requirements? Special focus on description quality.

Frontmatter requirements:

• name: lowercase, alphanumeric + hyphens only, ≤64 characters
• description: THE MOST CRITICAL FIELD — determines if skill gets used at all

Why description is THE MOST IMPORTANT field:

The brutal truth: A Skill with perfect content but poor description is useless — it will never be activated. The description is the only chance to tell the Agent "use me in these situations."

Description must answer THREE questions:

1. WHAT: What does this Skill do? (functionality)
2. WHEN: In what situations should it be used? (trigger scenarios)
3. KEYWORDS: What terms should trigger this Skill? (searchable terms)

Excellent description (all three elements):

Analysis:

• WHAT: creation, editing, analysis, tracked changes, comments
• WHEN: "When Claude needs to work with... for: (1)... (2)... (3)..."
• KEYWORDS: .docx files, tracked changes, professional documents

Poor description (missing elements):

Problems:

• WHAT: vague ("文档相关功能" — what specifically?)
• WHEN: missing (when should Agent use this?)
• KEYWORDS: missing (no ".docx", no specific scenarios)

Another poor example:

This is useless — Agent has no idea when to activate it.

Description quality checklist:

• Lists specific capabilities (not just "helps with X")
• Includes explicit trigger scenarios ("Use when...", "When user asks for...")
• Contains searchable keywords (file extensions, domain terms, action verbs)
• Specific enough that Agent knows EXACTLY when to use it
• Includes scenarios where this skill MUST be used (not just "can be used")

D5: Progressive Disclosure (15 points)

Does the Skill implement proper content layering?

Skill loading has three layers:

For Skills WITH references directory, check Loading Trigger Quality:

The loading problem:

Good loading trigger (embedded in workflow):

Bad loading trigger (just listed):

For simple Skills (no references, <100 lines): Score based on conciseness and self-containment.

D6: Freedom Calibration (15 points)

Is the level of specificity appropriate for the task's fragility?

Different tasks need different levels of constraint. This is about matching freedom to fragility.

The freedom spectrum:

High freedom (text-based instructions):

Medium freedom (pseudocode or parameterized):

Low freedom (specific scripts, exact steps):

The test: Ask "if Agent makes a mistake, what's the consequence?"

• High consequence → Low freedom
• Low consequence → High freedom

D7: Pattern Recognition (10 points)

Does the Skill follow an established official pattern?

Through analyzing 17 official Skills, we identified 5 main design patterns:

Pattern selection guide:

D8: Practical Usability (15 points)

Can an Agent actually use this Skill effectively?

Check for:

• Decision trees: For multi-path scenarios, is there clear guidance on which path to take?
• Code examples: Do they actually work? Or are they pseudocode that breaks?
• Error handling: What if the main approach fails? Are fallbacks provided?
• Edge cases: Are unusual but realistic scenarios covered?
• Actionability: Can Agent immediately act, or needs to figure things out?

Good usability (decision tree + fallback):

Poor usability (vague):

NEVER Do When Evaluating

• NEVER give high scores just because it "looks professional" or is well-formatted
• NEVER ignore token waste — every redundant paragraph should result in deduction
• NEVER let length impress you — a 43-line Skill can outperform a 500-line Skill
• NEVER skip mentally testing the decision trees — do they actually lead to correct choices?
• NEVER forgive explaining basics with "but it provides helpful context"
• NEVER overlook missing anti-patterns — if there's no NEVER list, that's a significant gap
• NEVER assume all procedures are valuable — distinguish domain-specific from generic
• NEVER undervalue the description field — poor description = skill never gets used
• NEVER put "when to use" info only in the body — Agent only sees description before loading

Evaluation Protocol

Step 1: First Pass — Knowledge Delta Scan

Read SKILL.md completely and for each section ask:

"Does Claude already know this?"

Mark each section as:

• [E] Expert: Claude genuinely doesn't know this — value-add
• [A] Activation: Claude knows but brief reminder is useful — acceptable
• [R] Redundant: Claude definitely knows this — should be deleted

Calculate rough ratio: E:A:R

• Good Skill: >70% Expert, <20% Activation, <10% Redundant
• Mediocre Skill: 40-70% Expert, high Activation
• Bad Skill: <40% Expert, high Redundant

Step 2: Structure Analysis

Step 3: Score Each Dimension

For each of the 8 dimensions:

1. Find specific evidence (quote relevant lines)
2. Assign score with one-line justification
3. Note specific improvements if score < max

Step 4: Calculate Total & Grade

Grade Scale (percentage-based):

Step 5: Generate Report

Common Failure Patterns

Pattern 1: The Tutorial

Pattern 2: The Dump

Pattern 3: The Orphan References

Pattern 4: The Checkbox Procedure

Pattern 5: The Vague Warning

Pattern 6: The Invisible Skill

Pattern 7: The Wrong Location

Pattern 8: The Over-Engineered

Pattern 9: The Freedom Mismatch

Quick Reference Checklist

The Meta-Question

When evaluating any Skill, always return to this fundamental question:

"Would an expert in this domain, looking at this Skill, say: 'Yes, this captures knowledge that took me years to learn'?"

If the answer is yes → the Skill has genuine value. If the answer is no → it's compressing what Claude already knows.

The best Skills are compressed expert brains — they take a designer's 10 years of aesthetic accumulation and compress it into 43 lines, or a document expert's operational experience into a 200-line decision tree.

What gets compressed must be things Claude doesn't have. Otherwise, it's garbage compression.

Self-Evaluation Note

This Skill (skill-judge) should itself pass evaluation:

• Knowledge Delta: Provides specific evaluation criteria Claude wouldn't generate on its own
• Mindset: Shapes how to think about Skill quality, not just checklist items
• Anti-Patterns: "NEVER Do When Evaluating" section with specific don'ts
• Specification: Valid frontmatter with comprehensive description
• Progressive Disclosure: Self-contained, no external references needed
• Freedom: Medium freedom appropriate for evaluation task
• Pattern: Follows Tool pattern with decision frameworks
• Usability: Clear protocol, report template, quick reference

Evaluate this Skill against itself as a calibration exercise.