tree-of-thoughts

tree-of-thoughts

Key benefits:

• Systematic exploration - Multiple agents explore different regions of the solution space
• Structured evaluation - Meta-judges produce tailored rubrics and criteria before judging
• Independent verification - Judges apply meta-judge specifications mechanically, reducing bias
• Adaptive strategy - Clear winners get polished, split decisions get synthesized, failures get redesigned

Pattern: Tree of Thoughts (ToT)

This command implements an eight-phase systematic reasoning pattern with meta-judge evaluation and adaptive strategy selection:

Process

Setup: Create Directory Structure

Before starting, ensure the directory structure exists:

Naming conventions:

• Proposals: .specs/research/{solution-name}-{YYYY-MM-DD}.proposals.[a|b|c].md
• Pruning: .specs/research/{solution-name}-{YYYY-MM-DD}.pruning.[1|2|3].md
• Selection: .specs/research/{solution-name}-{YYYY-MM-DD}.selection.md
• Evaluation: .specs/reports/{solution-name}-{YYYY-MM-DD}.[1|2|3].md

Where:

• {solution-name} - Derived from output path (e.g., users-api from output specs/api/users.md)
• {YYYY-MM-DD} - Current date

Note: Solutions remain in their specified output locations; only research and evaluation files go to .specs/

Phase 1: Exploration (Propose Approaches)

Launch 3 independent agents in parallel (recommended: Sonnet for speed):

1. Each agent receives identical task description and context
2. Each agent generates 6 high-level approaches (not full implementations)
3. For each approach, agent provides:
   • Approach description (2-3 paragraphs)
   • Key design decisions and trade-offs
   • Probability estimate (0.0-1.0)
   • Estimated complexity (low/medium/high)
   • Potential risks and failure modes
4. Proposals saved to .specs/research/{solution-name}-{date}.proposals.[a|b|c].md

Key principle: Systematic exploration through probabilistic sampling from the full distribution of possible approaches.

Prompt template for explorers:

Phase 1.5: Dispatch Pruning Meta-Judge

CRITICAL: Launch the pruning meta-judge in parallel with Phase 1 exploration agents. The meta-judge does not need exploration output to generate pruning criteria — it only needs the original task description.

The pruning meta-judge generates an evaluation specification (rubrics, checklist, scoring criteria) tailored to evaluating high-level proposals for pruning.

Prompt template for pruning meta-judge:

Dispatch:

Phase 2: Pruning (Vote for Top 3 Candidates)

Wait for BOTH Phase 1 exploration agents AND Phase 1.5 pruning meta-judge to complete before proceeding.

Launch 3 independent judges in parallel (recommended: Opus for rigor):

1. Each judge receives ALL proposal files (from .specs/research/) and the pruning meta-judge evaluation specification YAML
2. Judges evaluate each proposal against the meta-judge-generated pruning criteria
3. Each judge produces:
   • Scores for each proposal (with evidence)
   • Vote for top 3 proposals to expand
   • Rationale for selections
4. Votes saved to .specs/research/{solution-name}-{date}.pruning.[1|2|3].md

Key principle: Independent evaluation with meta-judge-generated criteria ensures consistent, tailored assessment without hardcoded weights.

CRITICAL: Provide to each judge the EXACT pruning meta-judge's evaluation specification YAML. Do not skip, add, modify, shorten, or summarize any text in it!

Prompt template for pruning judges:

You are evaluating {N} proposed approaches against an evaluation specification produced by the meta judge, to select the top 3 for full development.
 
CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`
 
## Task
{task_description}
 
## Proposals
{list of paths to all proposal files}
Read all proposals carefully before evaluating.
 
## Evaluation Specification
 
```yaml
{pruning meta-judge's evaluation specification YAML}

Output

{.specs/research/{solution-name}-{date}.pruning.[1|2|3].md}

Instructions

Follow your full judge process as defined in your agent instructions!

CRITICAL: You must reply with this exact structured evaluation report format in YAML at the START of your response!

 
**Dispatch:**
 

Use Task tool:

• description: "Pruning Judge {1|2|3}: {brief task summary}"
• prompt: {pruning judge prompt with exact meta-judge specification YAML}
• model: opus
• subagent_type: "sadd:judge"

 
### Phase 2b: Select Top 3 Proposals
 
After judges complete voting:
 
1. **Aggregate votes** using ranked choice:
   - 1st choice = 3 points
   - 2nd choice = 2 points
   - 3rd choice = 1 point
2. **Select top 3** proposals by total points
3. **Handle ties** by comparing average scores across criteria
4. **Document selection** in `.specs/research/{solution-name}-{date}.selection.md`:
   - Vote tallies
   - Selected proposals
   - Consensus rationale
 
### Phase 3: Expansion (Develop Full Solutions)
 
Launch **3 independent agents in parallel** (recommended: Opus for quality):
 
1. Each agent receives:
   - **One selected proposal** to expand
   - **Original task description** and context
   - **Judge feedback** from pruning phase (concerns, questions)
2. Agent produces **complete solution** implementing the proposal:
   - Full implementation details
   - Addresses concerns raised by judges
   - Documents key decisions made during expansion
3. Solutions saved to `solution.a.md`, `solution.b.md`, `solution.c.md`
 
**Key principle:** Focused development of validated approaches with awareness of evaluation feedback.
 
**Prompt template for expansion agents:**
 
```markdown
You are developing a full solution based on a selected proposal.
 
<task>
{task_description}
</task>
 
<selected_proposal>
{write selected proposal EXACTLY as it is. Including all details provided by the agent}
Read this carefully - it is your starting point.
</selected_proposal>
 
<judge_feedback>
{concerns and questions from judges about this proposal}
Address these in your implementation.
</judge_feedback>
 
<output>
solution.[*].md where [*] is your unique identifier (a, b, or c)
</output>
 
Instructions:
 
Let's work through this systematically to ensure we build a complete, high-quality solution.
 
**Step 1: Understand the proposal deeply**
Before implementing, analyze:
- What is the core insight or approach of this proposal?
- What are the key design decisions already made?
- What gaps need to be filled for a complete solution?
 
**Step 2: Address judge feedback**
For each concern raised by judges:
- What specific change or addition addresses this concern?
- How does this change integrate with the proposal's approach?
 
**Step 3: Decompose into implementation subproblems**
Break the solution into logical parts:
- What are the main components or sections?
- What must be defined first for other parts to build upon?
- What are the dependencies between parts?
 
**Step 4: Implement each subproblem**
For each component, work through:
- Core functionality and behavior
- Edge cases and error handling
- Integration points with other components
 
**Step 5: Self-verification**
Generate 3-5 verification questions about critical aspects, then answer them:
- Review solution against each question
- Identify gaps or weaknesses
- Fix identified issues
 
**Step 6: Document changes**
Explain what was changed from the original proposal and why.
 
<example>
**Example of good expansion thinking:**
 
Proposal: "Use event-driven architecture with message queue"
 
Step 1 Analysis:
- Core insight: Decouple components via async messaging
- Key decisions: Events as primary communication, eventual consistency
- Gaps: Need to define event schemas, queue technology, error handling
 
Step 2 - Addressing judge concern "What about message ordering?":
- Add partition keys for ordered processing within entity scope
- Document ordering guarantees and limitations
 
Step 3 - Subproblems:
1. Event schema definitions (foundational - others depend on this)
2. Producer interfaces (depends on schemas)
3. Consumer handlers (depends on schemas)
4. Error handling and dead letter queues (depends on both)
5. Integration patterns (builds on all above)
</example>
 
CRITICAL:
- Stay faithful to the selected proposal's core approach
- Do not switch to a different approach midway
- Address judge feedback explicitly
- Produce a complete, implementable solution

Phase 3.5: Dispatch Evaluation Meta-Judge

CRITICAL: Launch the evaluation meta-judge in parallel with Phase 3 expansion agents. The meta-judge does not need expansion output to generate evaluation criteria — it only needs the original task description.

The evaluation meta-judge generates an evaluation specification (rubrics, checklist, scoring criteria) tailored to evaluating full solution implementations.

Prompt template for evaluation meta-judge:

Dispatch:

Phase 4: Evaluation (Judge Full Solutions)

Wait for BOTH Phase 3 expansion agents AND Phase 3.5 evaluation meta-judge to complete before proceeding.

Launch 3 independent judges in parallel (recommended: Opus for rigor):

1. Each judge receives ALL solution files (solution.a.md, solution.b.md, solution.c.md) and the evaluation meta-judge specification YAML
2. Judges evaluate against the meta-judge-generated evaluation criteria
3. Each judge produces:
   • Comparative analysis (which solution excels where)
   • Evidence-based ratings (with specific quotes/examples)
   • Final vote (which solution they prefer and why)
4. Reports saved to .specs/reports/{solution-name}-{date}.[1|2|3].md

Key principle: Multiple independent evaluations with meta-judge-generated specifications and explicit evidence reduce bias and catch different quality aspects.

CRITICAL: Provide to each judge the EXACT evaluation meta-judge's evaluation specification YAML. Do not skip, add, modify, shorten, or summarize any text in it!

CRITICAL: NEVER provide score threshold to judges. Judge MUST not know what threshold for score is, in order to not be biased!!!

Prompt template for evaluation judges:

You are evaluating {number} full solutions against an evaluation specification produced by the meta judge.
 
CLAUDE_PLUGIN_ROOT=`${CLAUDE_PLUGIN_ROOT}`
 
## Task
{task_description}
 
## Solutions
{list of paths to all solution files}
Read all solutions carefully before evaluating.
 
## Evaluation Specification
 
```yaml
{evaluation meta-judge's evaluation specification YAML}

Output

Write full report to: .specs/reports/{solution-name}-{date}.[1|2|3].md

CRITICAL: You must reply with this exact structured header format:

VOTE: [Solution A/B/C] SCORES: Solution A: [X.X]/5.0 Solution B: [X.X]/5.0 Solution C: [X.X]/5.0 CRITERIA:

• {criterion_1}: [X.X]/5.0
• {criterion_2}: [X.X]/5.0 ...

[Summary of your evaluation]

Instructions

Follow your full judge process as defined in your agent instructions!

CRITICAL: You must reply with this exact structured evaluation report format in YAML at the START of your response!

 
**Dispatch:**
 

Use Task tool:

• description: "Evaluation Judge {1|2|3}: {brief task summary}"
• prompt: {evaluation judge prompt with exact meta-judge specification YAML}
• model: opus
• subagent_type: "sadd:judge"

 
### Phase 4.5: Adaptive Strategy Selection (Early Return)
 
**The orchestrator** (not a subagent) analyzes judge outputs to determine the optimal strategy.
 
#### Decision Logic
 
**Step 1: Parse structured headers from judge reply**
 
Parse the judges reply.
CRITICAL: Do not read report files themselves, as they can overflow your context.
 
**Step 2: Check for unanimous winner**
 
Compare all three VOTE values:
- If Judge 1 VOTE = Judge 2 VOTE = Judge 3 VOTE (same solution):
  - **Strategy: SELECT_AND_POLISH**
  - **Reason:** Clear consensus - all three judges prefer same solution
 
**Step 3: Check if all solutions are fundamentally flawed**
 
If no unanimous vote, calculate average scores:
1. Average Solution A scores: (Judge1_A + Judge2_A + Judge3_A) / 3
2. Average Solution B scores: (Judge1_B + Judge2_B + Judge3_B) / 3
3. Average Solution C scores: (Judge1_C + Judge2_C + Judge3_C) / 3
 
If (avg_A < 3.0) AND (avg_B < 3.0) AND (avg_C < 3.0):
- **Strategy: REDESIGN**
- **Reason:** All solutions below quality threshold, fundamental approach issues
 
**Step 4: Default to full synthesis**
 
If none of the above conditions met:
- **Strategy: FULL_SYNTHESIS**
- **Reason:** Split decision with merit, synthesis needed to combine best elements
 
#### Strategy 1: SELECT_AND_POLISH
 
**When:** Clear winner (unanimous votes)
 
**Process:**
1. Select the winning solution as the base
2. Launch subagent to apply specific improvements from judge feedback
3. Cherry-pick 1-2 best elements from runner-up solutions
4. Document what was added and why
 
**Benefits:**
- Saves synthesis cost (simpler than full synthesis)
- Preserves proven quality of winning solution
- Focused improvements rather than full reconstruction
 
**Prompt template:**
 
```markdown
You are polishing the winning solution based on judge feedback.
 
<task>
{task_description}
</task>
 
<winning_solution>
{path_to_winning_solution}
Score: {winning_score}/5.0
Judge consensus: {why_it_won}
</winning_solution>
 
<runner_up_solutions>
{list of paths to all runner-up solutions}
</runner_up_solutions>
 
<judge_feedback>
{list of paths to all evaluation reports}
</judge_feedback>
 
<output>
{final_solution_path}
</output>
 
Instructions:
 
Let's approach this polishing task methodically to improve without disrupting what works.
 
**Step 1: Understand why this solution won**
Analyze the winning solution:
- What are its core strengths that judges praised?
- What makes its approach superior to alternatives?
- Which parts should remain untouched?
 
**Step 2: Catalog improvement opportunities**
From judge feedback, identify:
- Specific weaknesses mentioned (list each one)
- Missing elements judges noted
- Areas where runner-ups were praised
 
**Step 3: Prioritize changes by impact**
For each improvement opportunity:
- High impact: Directly addresses judge criticism
- Medium impact: Adds praised element from runner-up
- Low impact: Nice-to-have refinement
 
Focus on high-impact changes first.
 
**Step 4: Apply improvements surgically**
For each change:
- Locate the specific section to modify
- Make the minimal change needed to address the issue
- Verify the change integrates cleanly with surrounding content
 
**Step 5: Cherry-pick from runners-up**
Review runner-up solutions for:
- 1-2 specific elements that judges praised
- Elements that complement (not conflict with) the winning approach
- Only incorporate if clearly superior to winning solution's version
 
**Step 6: Document all changes**
Record:
- What was changed and why (with reference to judge feedback)
- What was added from other solutions (cite source)
- What was intentionally left unchanged
 
CRITICAL: Preserve the winning solution's core approach. Make targeted improvements only.

Strategy 2: REDESIGN

When: All solutions scored <3.0/5.0 (fundamental issues across the board)

Process:

1. Launch new agent to analyze the failure modes and lessons learned
2. Return to Phase 3 (Expansion), provide to new implementation agents the lessons learned and new constraints

Note: If redesign fails twice, escalate to user for guidance.

Prompt template for new implementation:

Strategy 3: FULL_SYNTHESIS (Default)

When: No clear winner AND solutions have merit (scores >=3.0)

Process: Proceed to Phase 5 (Evidence-Based Synthesis)

Phase 5: Synthesis (Evidence-Based Combination)

Only executed when Strategy 3 (FULL_SYNTHESIS) selected in Phase 4.5

Launch 1 synthesis agent (recommended: Opus for quality):

1. Agent receives:
   • All solutions (from specified output location)
   • All evaluation reports (from .specs/reports/)
   • Selection rationale from pruning phase (from .specs/research/)
2. Agent analyzes:
   • Consensus strengths (what multiple judges praised)
   • Consensus weaknesses (what multiple judges criticized)
   • Complementary elements where solutions took different approaches
3. Agent produces final solution by:
   • Copying superior sections when one solution clearly wins
   • Combining approaches when hybrid is better
   • Fixing identified issues that judges caught
   • Documenting decisions (what was taken from where and why)

Key principle: Evidence-based synthesis leverages collective intelligence from exploration and evaluation.

Prompt template for synthesizer:

Outputs (All Strategies)

1. Research directory: .specs/research/ (created if not exists)

   • Proposals: .specs/research/{solution-name}-{date}.proposals.[a|b|c].md - High-level approaches with probabilities
   • Pruning: .specs/research/{solution-name}-{date}.pruning.[1|2|3].md - Judge evaluations and votes
   • Selection: .specs/research/{solution-name}-{date}.selection.md - Vote tallies and selected proposals

2. Expansion outputs:

   • solution.a.md, solution.b.md, solution.c.md - Full implementations (in specified output location)

3. Reports directory: .specs/reports/ (created if not exists)

   • Evaluation: .specs/reports/{solution-name}-{date}.[1|2|3].md - Final judge reports

4. Resulting solution: {output_path}

Strategy-Specific Outputs

• SELECT_AND_POLISH: Polished solution based on winning solution, with targeted improvements
• REDESIGN: Do not stop; return to Phase 3 with lessons learned; eventually finishes at SELECT_AND_POLISH or FULL_SYNTHESIS
• FULL_SYNTHESIS: Synthesized solution combining best elements from all solutions

Best Practices

Meta-Judge + Judge Verification

• Two meta-judges - Separate specs for pruning (proposals) and evaluation (full solutions)
• Meta-judges run in parallel with implementation - Don't block the pipeline; pruning meta-judge runs with Phase 1, evaluation meta-judge runs with Phase 3
• Include CLAUDE_PLUGIN_ROOT - Both meta-judges and judges need the resolved plugin root path
• Meta-judge YAML - Pass only the YAML to judges, do not modify it

Common Pitfalls

• Insufficient exploration - Agents propose similar approaches
• Ignoring judge feedback - Expansion ignores concerns from pruning
• Vague proposals - Can't properly evaluate without implementation details
• Over-exploration - Too many proposals, evaluation becomes expensive
• Forcing synthesis when clear winner exists - Wastes cost and risks degrading quality
• Synthesizing fundamentally flawed solutions - Better to redesign than polish garbage

Recommendations

• Encourage diverse exploration - Prompt for different regions of solution space
• Feed feedback forward - Expansion agents address pruning concerns
• Right level of detail - Proposals have enough detail to evaluate
• Prune aggressively - Only expand most promising 3 approaches
• Trust adaptive strategy selection - Polish clear winners, synthesize split decisions, redesign failures

Example: API Design

Phase 1 outputs (assuming date 2025-01-15):

• .specs/research/users-api-2025-01-15.proposals.a.md - 6 approaches from Agent A
• .specs/research/users-api-2025-01-15.proposals.b.md - 6 approaches from Agent B
• .specs/research/users-api-2025-01-15.proposals.c.md - 6 approaches from Agent C

Phase 1.5 output (runs in parallel with Phase 1):

• Pruning Meta-judge (Opus, sadd:meta-judge) generates pruning evaluation specification YAML

Phase 2 outputs (3 judges with pruning meta-judge spec):

• .specs/research/users-api-2025-01-15.pruning.1.md - Top 3: Resource-based REST, Pure REST, Monolithic
• .specs/research/users-api-2025-01-15.pruning.2.md - Top 3: Pure REST, Hybrid (services), Resource-based REST
• .specs/research/users-api-2025-01-15.pruning.3.md - Top 3: Resource-based REST, REST+GraphQL hybrid, Pure REST
• .specs/research/users-api-2025-01-15.selection.md - Selected: Resource-based REST (8 pts), Pure REST (7 pts), Monolithic (4 pts)

Phase 3 outputs:

• specs/api/users.a.md - Full resource-based design with nested routes
• specs/api/users.b.md - Flat REST design with simple endpoints
• specs/api/users.c.md - Monolithic API with service-oriented internals

Phase 3.5 output (runs in parallel with Phase 3):

• Evaluation Meta-judge (Opus, sadd:meta-judge) generates evaluation specification YAML

Phase 4 outputs (3 judges with evaluation meta-judge spec):

• .specs/reports/users-api-2025-01-15.1.md:

  VOTE: Solution A
  SCORES: A=4.2/5.0, B=3.8/5.0, C=3.4/5.0

  "Prefers A for RESTfulness, criticizes C complexity"

• .specs/reports/users-api-2025-01-15.2.md:

  VOTE: Solution B
  SCORES: A=3.9/5.0, B=4.1/5.0, C=3.5/5.0

  "Prefers B for simplicity, criticizes A deep nesting"

• .specs/reports/users-api-2025-01-15.3.md:

  VOTE: Solution A
  SCORES: A=4.3/5.0, B=3.6/5.0, C=3.2/5.0

  "Prefers A for discoverability, criticizes B lack of structure"

Phase 4.5 decision (orchestrator parses headers):

• Split votes: A, B, A (no unanimous winner)
• Average scores: A=4.1, B=3.8, C=3.4 (all >=3.0)
• Strategy: FULL_SYNTHESIS
• Reason: Split decision with merit, synthesis needed

Phase 5 output (synthesis):

• specs/api/users.md - Resource-based structure (from A), max 2-level nesting (from B), internal services (from C)