autobrowse

AutoBrowse — Self-Improving Browser Skill

Build reliable browser automation skills through iterative experimentation. An inner agent browses the site (evaluate.ts). You — the outer agent — read what happened and improve the instructions (strategy.md). Repeat until it passes consistently.

Entry Points

Invocation is flexible — both explicit flags and free-form natural language work:

/autobrowse --task google-flights
/autobrowse --task google-flights --iterations 10 --env remote
/autobrowse --task google-flights --browser-trace
/autobrowse --tasks google-flights,amazon-add-to-cart
/autobrowse --all

# Also fine — parse freely:
/autobrowse https://flights.google.com/
/autobrowse book a flight on delta.com
/autobrowse fix the existing google-flights skill

--browser-trace (default off, remote-only): pairs each iteration with the sibling browser-trace skill — wraps the inner agent in a CDP capture for per-page network/console/page-lifecycle evidence. Implies --env remote; errors if combined with --env local. Requires the sibling browser-trace skill present at ${CLAUDE_SKILL_DIR}/../browser-trace/, and the BROWSERBASE_API_KEY env var.

When the user drops a URL or free-form instruction instead of --task <name>:

• If an existing task in ${WORKSPACE}/tasks/ clearly matches the site/intent, use it.
• Otherwise, pick a short kebab-case name, create ${WORKSPACE}/tasks/<name>/task.md from ${CLAUDE_SKILL_DIR}/references/example-task.md, fill in the URL/goal based on what the user said, and proceed. Tell the user the chosen name in one line.

How to run

Step 1 — Parse arguments and orient

Check what was passed:

• --task <name> → single task mode
• --tasks a,b,c or --all → multi-task mode (spawn sub-agents)
• --iterations N → how many evaluate → improve cycles (default: 5)
• --env local|remote → browser environment (default: local; use remote for bot-protected sites)
• --browser-trace → opt in to the browser-trace integration (default off). Implies --env remote. If --env local --browser-trace are both passed explicitly, error with: browser-trace requires Browserbase; drop --env local or drop --browser-trace.

If the user passed free-form text instead, map it to one of the above before continuing.

Step 2 — Set up the workspace

All training artifacts (task definitions, strategy iterations, traces, reports) live in a workspace directory in the current working directory — NOT inside ~/.claude/skills/. This keeps the inner agent's file writes out of Claude's home dir and away from permission friction.

Default workspace: ${CWD}/autobrowse/

mkdir -p ./autobrowse/tasks ./autobrowse/traces ./autobrowse/reports

If the task directory (./autobrowse/tasks/<task>/task.md) doesn't exist yet, scaffold it:

mkdir -p ./autobrowse/tasks/

The skill source at ${CLAUDE_SKILL_DIR} stays read-only — only ./autobrowse/ in CWD gets written to during training. Graduation (final step) writes a single file to ~/.claude/skills/<task>/SKILL.md.

List available tasks:

Step 3 — Multi-task: spawn parallel sub-agents

If running multiple tasks, use the Agent tool to spawn one sub-agent per task simultaneously. Each sub-agent receives a self-contained prompt to run the full autobrowse loop for its task:

"You are running the autobrowse skill for task <name>. Workspace: <absolute-path-to-workspace> (e.g. /path/to/project/autobrowse). Run <N> iterations of: evaluate → read trace → improve strategy.md → repeat. Use --env <env>. Pass --workspace <workspace> to every evaluate.mjs invocation. If the parent invocation used --browser-trace, you MUST use the traced-path block of the SKILL.md loop for every iteration (pre-create session, attach bb-capture, pass --connect-url to evaluate.mjs, stop+bisect, release) — do not fall back to the default single-command path. Follow the autobrowse loop instructions exactly.

When graduating, install the skill to ~/.claude/skills/<task-name>/SKILL.md with proper agentskills frontmatter (name + description). Do not just copy strategy.md — write a self-contained skill.

At the end, output a structured summary with: task name, pass/fail on final run, total cumulative cost, iterations completed, per-iteration table (iter number, turns, cost, status, hypothesis tested), and 2-3 bullet key learnings."

Spawn all sub-agents in parallel, wait for all to complete, then collect their summaries and write the session report.

For single task, skip this step and run the loop directly below.

The Loop (run this for each task)

Iteration start

Check that ./autobrowse/tasks/<task>/task.md exists (scaffold it from the template if not — see Step 2). strategy.md is auto-created empty by the harness on first run.

Requirements

• ANTHROPIC_API_KEY must be in the environment (or in a .env file in CWD — evaluate.mjs auto-loads it). If missing, the harness prints a clear error and exits; don't hunt for keys in other paths.

Run the inner agent

Default path (no --browser-trace) — single command, no orchestration:

This runs the browser session and writes a full trace to ./autobrowse/traces/<task>/latest/.

Traced path (--browser-trace, remote only) — the outer harness pre-creates a Browserbase session, attaches bb-capture as a passive observer, and passes the session's connectUrl to evaluate.mjs so every inner browse call uses --cdp $connectUrl --session autobrowse-main (the canonical browser-trace pattern that gives observers full Network/Console events). Run this block once per iteration with $N set to the 1-indexed iteration number:

This writes the inner-agent trace to ./autobrowse/traces/<task-name>/latest/ and the CDP bisect to ./autobrowse/traces/<task-name>/latest/.o11y/<run-id>/. The traced browse CLI also emits per-command rich node descriptors to .o11y/<run-id>/cdp/descriptors.ndjson (one JSON object per page-driving call: target tag/id/role/accessibleName/attributes/xpath/bounding-rect). The descriptors file feeds downstream codegen; it is not required for hypothesis formation — skip it when reading the trace.

Read the trace

The summary has duration, cost, turns, the decision log, and the final JSON output.

If the agent failed or got stuck, look deeper:

• Read ./autobrowse/traces/<task-name>/latest/trace.json — search for the failure turn
• Read screenshots around the failure point with the Read tool

When --browser-trace was used — start with unified-events.jsonl. The harness joins the agent's turn log and the browser's CDP firehose into one time-ordered NDJSON stream at the run root. One file, source-tagged (source: "agent" | "browser"), interleaved by wall-clock timestamp. Skim it top-to-bottom; the failure cause is usually one or two adjacent lines (the agent issued command X, the browser responded with Y).

The structured files (trace.json, .o11y/<run-id>/cdp/*) are also agent-consumable as drill-downs when the unified stream points at something you need more of:

The unified stream is the default; drill into structured files only when you need a grouped query, a full-text payload, or filtering the stream can't give you.

Form one hypothesis

Find the exact turn where things went wrong. What single heuristic would have prevented it?

Under --browser-trace, the hypothesis must cite a specific event from unified-events.jsonl (line number or timestamp) — or name the drill-down file if you had to descend into one. This keeps updates evidence-grounded rather than vibes-driven. A hypothesis based only on the agent's commands might say "the click didn't work"; grounded in the unified stream, it can say "line 47 of unified-events.jsonl: browse open was followed by Network.responseReceived status 403 on /api/checkout — switch to --verified --proxies."

Examples:

• "After clicking the dropdown, wait 1s — options animate in before they're clickable"
• "Navigate directly to /pay-invoice/ — skip the landing page entirely"
• "Use browse fill #field_3 value not browse type — this field clears on focus"
• "The page shows a spinner at turn 8 — add browse wait timeout 2000 before snapshot"
• (with --browser-trace) "At line 47 of unified-events.jsonl, 3 consecutive Network.responseReceived events on /api/availability returned 403 right after browse open — the site is fingerprinting; the next iter needs --verified --proxies."

Update strategy.md

Edit ./autobrowse/tasks/<task-name>/strategy.md. Keep everything that worked. Fix the specific failure. Add a concrete heuristic.

Good strategies have:

• Fast path: direct URL or shortcuts to skip exploration
• Step-by-step workflow: exact sequence with timing notes
• Site-specific knowledge: selector IDs, form field names, success indicators
• Failure recovery: what to do when X goes wrong

Judge the result

Read the new summary. Did it pass? Make clear progress?

• Pass or progress → keep, next iteration
• No progress or regression → revert strategy.md to the previous version and try a different hypothesis

Generate a runnable script (optional)

Once the task has converged, you can produce a deterministic, runnable script in one or more frameworks via scripts/codegen.mjs. This is one shot of an LLM call per framework, cached by content hash, with optional verify-against- fresh-session and rewrite-on-failure.

Each framework gets its own subdirectory under tasks/<name>/<framework>/ with the emitted script and a self-contained scaffold (package.json, tsconfig.json). The directory is runnable standalone with cd tasks/<name>/playwright && npm install && npx tsx <name>.ts — the only runtime requirement is BROWSERBASE_API_KEY (plus ANTHROPIC_API_KEY for the Stagehand target).

Builtin frameworks: playwright, stagehand. Add a custom framework with --prompt-template <path> --frameworks custom (and provide your own runner or pass --no-verify).

Common flags:

Output is one JSON line per framework on stdout. Non-zero exit if any selected framework's final state is passed: false.

See references/playwright-cdp-bridge.md for the canonical connectOverCDP patterns the emitted scripts follow.

After all iterations — publish if ready

If the task passed on 2+ of the last 3 iterations or has reached the max iteration limit, install it as a Claude Code skill. Do not just copy strategy.md — the skill must be self-contained and useful to someone who has never seen this codebase. If graduating at max iterations without a clean pass, note the known failure point but still document everything learned.

Install by writing to ~/.claude/skills/<task-name>/SKILL.md:

Use this structure for the SKILL.md:

---
name: <task-name>
description: <1-2 sentences describing what this skill does and when to use it. Include trigger keywords.>
---
 
# <Task Title> — Browser Skill
 
## Purpose
<1-2 sentences: what this automates and why it exists.>
 
## When to Use
<When should someone reach for this skill.>
 
## Browse CLI Reference
The inner agent uses the `browse` CLI. Key commands for this task:
- `browse stop` — kill existing session (always run before switching to remote)
- `browse open <url> --remote` — start a fresh Browserbase cloud session and navigate
- `browse open <url> --local` — start a clean local browser and navigate
- `browse tab new <url>` — open URL in a new tab
- `browse wait load` — wait for page to finish loading
- `browse wait timeout <ms>` — wait a fixed amount of time for spinners or animations
- `browse wait selector "<selector>"` — wait for an element to become visible
- `browse get title` — verify you're on the right page
- `browse get text body` — extract all visible text (preferred for content extraction)
- `browse snapshot` — get accessibility tree; each node has a ref in `[X-Y]` format (e.g. `[0-5]`, `[2-147]`)
- `browse click [X-Y]` — click element by ref from the latest snapshot (include the brackets)
 
**Never use `--session <name>` flags in SKILL.md.** Named sessions are a parallel-run workaround — they contaminate skills with infrastructure concerns. Skills must work in isolation with the default session.
 
## Workflow
 
### Step 1 — Start session
<exact browse commands in order>
 
### Step 2 — Navigate
<exact URL and verification steps>
 
### Step 3 — Extract
<exact extraction commands>
 
### Step 4 — Output
<what JSON to emit, referencing the schema below>
 
## Site-Specific Gotchas
<Bullet list of every hard-won heuristic from the iterations. This is the core value of the skill.>
 
## Failure Recovery
<What to do when navigation fails, session is contaminated, or extraction returns garbage>
 
## Expected Output
```json
<paste the exact expected output schema from task.md>

 
After writing the SKILL.md, confirm it's installed:
```bash
ls ~/.claude/skills/<task-name>/SKILL.md

The skill is now available as /<task-name> in Claude Code.

Final report (multi-task mode)

After all sub-agents complete, print a markdown table:

Then write a persistent session report to ./autobrowse/reports/ so there's a durable record of the run inside the workspace:

mkdir -p ./autobrowse/reports

Write the file ./autobrowse/reports/YYYY-MM-DD-HH-MM-<tasks>.md with:

# AutoBrowse Session Report
**Date:** <ISO date>
**Tasks:** <comma-separated list>
**Environment:** remote|local
**Total cost:** $X.XX
 
## Results
 
| Task | Iterations | Pass Rate | Final Status | Graduated | Cost |
|------|-----------|-----------|--------------|-----------|------|
| ... | ... | X/5 | ✅/❌ | yes/no | $X.XX |
 
## Per-Task Learnings
 
### <task-name>
- **Key insight 1:** <what the agent learned>
- **Key insight 2:** <another heuristic>
- **Failure mode fixed:** <what was failing and how it was resolved>
 
## Iteration Log
 
### <task-name>
| Iter | Turns | Cost | Status | Hypothesis tested |
|------|-------|------|--------|-------------------|
| 1 | 79 | $18.75 | ❌ fail | baseline |
| 2 | 9 | $0.26 | ✅ pass | session contamination fix |
| ... | ... | ... | ... | ... |

Rules

• Only edit strategy.md — never touch task.md (unless creating it from the template) or evaluate.mjs
• Stay in the workspace — all training writes go to ./autobrowse/, never to ~/.claude/skills/autobrowse/. The skill source is read-only.
• One hypothesis per iteration — test one change at a time
• Build on wins — keep what worked, add to it
• Trust the trace — the inner agent shows exactly what it saw and did
• Graduate to ~/.claude/skills/ — the only file you write there is the final graduated SKILL.md
• Don't release before bisecting — under --browser-trace, the order at the end of each iteration is non-negotiable: stop-capture → bisect-cdp → browse cloud sessions update REQUEST_RELEASE. Bisect depends on the session still existing when the trace stops.