judge-with-debate
Evaluate solutions through multi-agent debate where independent judges analyze, challenge each other's assessments, and iteratively refine their evaluations until reaching consensus or maximum rounds.
This command implements the Multi-Agent Debate pattern for high-quality evaluation where multiple perspectives and rigorous argumentation improve assessment accuracy. Unlike single-pass evaluation, debate forces judges to defend their positions with evidence and consider counter-arguments.
Key benefits:
- Structured evaluation - Meta-judge produces tailored rubrics and criteria before judging begins
- Multiple perspectives - Three independent judges reduce individual bias
- Evidence-based debate - Judges defend positions with specific evidence from the solution and evaluation specification
- Iterative refinement - Up to 3 debate rounds drive convergence on accurate scores
- Shared specification - Meta-judge runs once; all judges across all rounds share the same evaluation specification
Pattern: Debate-Based Evaluation
This command implements iterative multi-judge debate:
Process
Setup: Create Reports Directory
Before starting evaluation, ensure the reports directory exists:
Report naming convention: .specs/reports/{solution-name}-{YYYY-MM-DD}.[1|2|3].md
Where:
{solution-name} - Derived from solution filename (e.g., users-api from src/api/users.ts){YYYY-MM-DD} - Current date[1|2|3] - Judge number
Before independent analysis, dispatch a meta-judge agent to generate a tailored evaluation specification. The meta-judge runs ONCE and produces rubrics, checklists, and scoring criteria that ALL judges will use across ALL rounds.
Meta-judge prompt template:
Dispatch:
Wait for the meta-judge to complete and extract the evaluation specification YAML from its output before proceeding to Phase 1.
Phase 1: Independent Analysis
Launch 3 independent judge agents in parallel (Opus for rigor):
- Each judge receives:
- Path to solution(s) being evaluated
- The meta-judge's evaluation specification YAML
- Task description
- Each produces independent assessment saved to
.specs/reports/{solution-name}-{date}.[1|2|3].md
- Reports must include:
- Per-criterion scores with evidence
- Specific quotes/examples supporting ratings
- Overall weighted score
- Key strengths and weaknesses
Key principle: Independence in initial analysis prevents groupthink.
Prompt template for initial judges:
Output File
.specs/reports/{solution-name}-{date}.{N}.md
Instructions
Follow your full judge process as defined in your agent instructions!
Additional instructions:
- Read the solution thoroughly
- For each criterion from the evaluation specification:
- Find specific evidence (quote exact text)
- Score on the defined scale
- Justify with concrete examples
- Calculate weighted overall score
- Write comprehensive report to {output_file}
Add to report beginning Done by Judge {N}
Use Task tool:
- description: "Judge {N}: independent analysis of {solution-name}"
- prompt: {judge prompt with evaluation specification YAML}
- model: opus
- subagent_type: "sadd:judge"
Output File
.specs/reports/{solution-name}-{date}.{N}.md (append to existing file)
Instructions
Follow your full judge process as defined in your agent instructions!
Additional debate instructions:
- Read your previous assessment from {your_previous_report}
- Read all other judges' reports
- Identify disagreements (where your scores differ by >1 point)
- For each major disagreement:
- State the disagreement clearly
- Defend your position with evidence from the solution and evaluation specification
- Challenge the other judge's position with counter-evidence
- Consider whether their evidence changes your view
- Update your report file by APPENDING debate round section
- Reply whether you reached agreement, and with which judge. Include revisited scores and criteria scores.
CRITICAL:
- Ground your arguments in the evaluation specification criteria
- Only revise if you find their evidence compelling
- Defend your original scores if you still believe them
- Quote specific evidence from the solution
Use Task tool:
- description: "Judge {N}: debate round {R} for {solution-name}"
- prompt: {debate judge prompt with evaluation specification YAML}
- model: opus
- subagent_type: "sadd:judge"
The command produces:
- Reports directory:
.specs/reports/ (created if not exists)
- Initial reports:
.specs/reports/{solution-name}-{date}.1.md, .specs/reports/{solution-name}-{date}.2.md, .specs/reports/{solution-name}-{date}.3.md
- Debate updates: Appended sections in each report file per round
- Final synthesis: Replied to user (consensus or disagreement summary)
Best Practices
- Never skip meta-judge - Tailored evaluation criteria produce better judgments and more grounded debates
- Meta-judge runs once - Same specification for all 3 judges across all debate rounds
- Include CLAUDE_PLUGIN_ROOT - Both meta-judge and judges need the resolved plugin root path
- Meta-judge YAML - Pass only the YAML to judges, do not modify it
- Debate grounding - Judges should reference evaluation specification criteria when defending positions
Common Pitfalls
- Judges create new reports instead of appending - Loses debate history
- Orchestrator passes reports between judges - Violates filesystem communication principle
- Weak initial assessments - Garbage in, garbage out
- Too many debate rounds - Diminishing returns after 3 rounds
- Sycophancy in debate - Judges agree too easily without real evidence
- Modifying meta-judge YAML - Specification must be passed verbatim to all judges
- Re-running meta-judge between rounds - Specification is generated once and shared
Do This
- Judges append to their own report file
- Judges read other reports from filesystem directly
- Strong evidence-based initial assessments
- Maximum 3 debate rounds
- Require evidence for changing positions
- Ground debate arguments in the evaluation specification criteria
- Use same evaluation specification across all rounds
Example Usage
Evaluating an API Implementation
Phase 0.5 - Meta-Judge (assuming date 2025-01-15):
- Meta-judge generates evaluation specification YAML with criteria:
- Correctness (30%), Design (25%), Security (20%), Performance (15%), Documentation (10%)
- Rubrics, checklists, and scoring definitions for each criterion
Phase 1 - Independent Analysis (3 judges receive specification):
.specs/reports/users-api-2025-01-15.1.md - Judge 1 scores correctness 4/5, security 3/5.specs/reports/users-api-2025-01-15.2.md - Judge 2 scores correctness 4/5, security 5/5.specs/reports/users-api-2025-01-15.3.md - Judge 3 scores correctness 5/5, security 4/5
Disagreement detected: Security scores range from 3-5
Phase 2 - Debate Round 1 (judges reference evaluation specification):
- Judge 1 defends 3/5: "Missing rate limiting, input validation incomplete per specification checklist item 4"
- Judge 2 challenges: "Rate limiting exists in middleware (line 45), satisfies specification rubric"
- Judge 1 revises to 4/5: "Missed middleware, but input validation still weak per specification"
- Judge 3 defends 4/5: "Input validation adequate for requirements as defined in specification"
Debate Round 1 outputs:
- All judges now 4-5/5 on security (within 1 point)
- Disagreement on input validation remains
Debate Round 2 (same evaluation specification):
- Judges examine specific validation code against specification criteria
- Judge 2 revises to 4/5: "Upon re-examination, email validation regex is weak per specification checklist"
- Consensus: Security = 4/5
Final consensus: