benchmark-sandbox

Benchmark Sandbox — Remote Eval via Vercel Sandboxes

Run benchmark scenarios inside Vercel Sandboxes — ephemeral Firecracker microVMs with node24. Each sandbox gets a fresh Claude Code + Vercel CLI + agent-browser install, the local vercel-plugin uploaded, and runs a 3-phase eval pipeline:

• Phase 1 (BUILD): Claude Code builds the app with --dangerously-skip-permissions --debug
• Phase 2 (VERIFY): A follow-up Claude Code session uses agent-browser to walk through user stories, fixing issues until all pass (20 min timeout)
• Phase 3 (DEPLOY): A third Claude Code session links to vercel-labs, runs vercel deploy, and fixes build errors (up to 3 retries). Deployed apps have deployment protection enabled by default.

Skills are tracked across all 3 phases — each phase may trigger additional skill injections as new files/patterns are created. After each phase, a haiku structured scoring step (claude -p --json-schema --model haiku) evaluates the results as structured JSON.

Proven Working Script

Use run-eval.ts — the proven eval runner:

# Run default scenarios with full 3-phase pipeline
bun run .claude/skills/benchmark-sandbox/run-eval.ts
 
# With dynamic scenarios from a JSON file (recommended — see "Dynamic Scenarios" below)
bun run .claude/skills/benchmark-sandbox/run-eval.ts --scenarios-file /tmp/my-scenarios.json
 
# Keep sandboxes alive overnight with public URLs
bun run .claude/skills/benchmark-sandbox/run-eval.ts --keep-alive

CLI Flags

Dynamic Scenarios (Recommended Approach)

Instead of hardcoding tech-specific prompts, generate scenarios dynamically as a JSON file. Prompts should describe real-world apps people want to build using user stories — no tech name-dropping. Let the plugin figure out what Vercel tech to inject.

Scenario JSON Format

Each scenario needs: slug (string), prompt (string), expectedSkills (string[]), userStories (tuple of exactly 3 strings).

Prompt Design Guidelines

• Focus on what the user wants, not what tech to use
• Describe real-world apps that solve real problems with friendly, stylish UX
• Include AI features naturally (recommendations, analysis, generation)
• Always end with: "Link the project to my vercel-labs team. After building all files, start the dev server on port 3000 with \npx next dev --port 3000`."`
• Include storage needs (photos, uploads) to trigger vercel-storage
• Include scheduled tasks (reminders, cleanup) to trigger cron-jobs
• Include auth/middleware to trigger routing-middleware

Structured Scoring (Haiku)

Each phase gets a structured JSON score via claude -p --json-schema --model haiku --setting-sources "" running inside the sandbox. This is a separate quick pass — no tools, no hooks — just reads the phase output and returns structured data.

Build Score Schema

Verify Score Schema (per user story)

Deploy Score Schema

Important: The claude -p --output-format json response wraps results — the actual schema data is in parsed.structured_output, not the top-level object.

Critical Sandbox Environment Facts

Key Discoveries (Hard-Won)

1. Snapshots work: sandbox.snapshot() preserves files AND npm globals. Use it after build to create a restore point before verify/deploy. Note: snapshotting stops the source sandbox — create a new one from the snapshot to continue.
2. Plugin install: Use npx add-plugin <path> -s project -y --target claude-code — works because claude is in PATH after npm install -g. The --target claude-code flag is required because add-plugin can't auto-detect Claude Code without an initialized ~/.claude/ dir.
3. File uploads: Use sandbox.writeFiles([{ path, content: Buffer }]) — NOT runCommand heredocs. Heredocs with special characters cause 400 errors from the sandbox API.
4. Claude flags: Always use --dangerously-skip-permissions --debug. The --debug flag writes to ~/.claude/debug/.
5. Auth: API key from macOS Keychain (ANTHROPIC_AUTH_TOKEN — a vck_* Vercel Claude Key for AI Gateway), Vercel token from ~/.local/share/com.vercel.cli/auth.json (a vca_* token).
6. OIDC for sandbox SDK: Run npx vercel link --scope vercel-labs -y + npx vercel env pull once before first use.
7. Port exposure: Pass ports: [3000] in Sandbox.create() to get a public URL immediately via sandbox.domain(3000). Works on v1.8.0 — URL is assigned at creation time, before anything listens.
8. extendTimeout: Use sandbox.extendTimeout(ms) to keep sandboxes alive past their initial timeout. Verified working — extends by the requested duration. Use this for overnight keep-alive.
9. Background commands: runCommand with backgrounded processes (& or nohup) may throw ZodError on v1. Write a script file first, then execute it.
10. Session cleanup race: The session-end-cleanup.mjs hook deletes /tmp/vercel-plugin-*-seen-skills.d/ on session end. Extract artifacts BEFORE the session completes, or rely on poll history data.
11. agent-browser works in sandboxes: Install via npm install -g agent-browser. Claude Code can use it for browser-based verification inside the sandbox.
12. No hobby tier cap: Early 301s timeouts were from lower default timeout values in earlier script iterations, not a tier limitation. Enterprise (vercel-labs) has no known sandbox time cap — sandboxes ran 10+ minutes successfully.
13. claude -p works inside sandboxes: claude -p --json-schema --output-format json --model haiku works for structured scoring passes. No nesting issue when running inside a sandbox (only fails when running Claude inside Claude on the same machine).
14. Deploy project naming: ALWAYS use timestamped slugs with minute precision (e.g., pet-adoption-board-202603101853) to avoid collisions when linking to vercel-labs team projects. These are demo projects — we generate many per day. Format: <slug>-<YYYYMMDDHHMM>.

When to Use This vs benchmark-agents

How It Works

1. Create fresh sandbox: Sandbox.create({ runtime: "node24", ports: [3000], env: { ANTHROPIC_API_KEY, ... } }) — no snapshot
2. Install tools: npm install -g @anthropic-ai/claude-code vercel agent-browser (~20s per sandbox)
3. Auth Vercel CLI: Write token to ~/.local/share/com.vercel.cli/auth.json
4. Upload plugin: sandbox.writeFiles() for 80 plugin files, then npx add-plugin
5. Phase 1 — BUILD: Claude Code builds the app (30 min timeout)
6. Score build: Haiku evaluates completeness, API routes, UI, AI features
7. Start dev server: If not already running, start npx next dev --port 3000
8. Extend timeout: sandbox.extendTimeout() for verify + deploy + keep-alive
9. Phase 2 — VERIFY: Claude Code uses agent-browser to test user stories (20 min timeout). Prompt tells Claude to start dev server itself if not running.
10. Score verify: Haiku evaluates each user story as pass/fail with reasons
11. Re-extract skills: Skills re-collected after verify phase (agent-browser + code fixes trigger more)
12. Phase 3 — DEPLOY: Claude Code runs vercel link + vercel deploy, fixes build errors (30 min timeout)
13. Score deploy: Haiku evaluates deploy success, URL extraction, errors
14. Re-extract skills: Skills re-collected after deploy phase
15. Write incremental results: Each scenario writes its own result.json immediately on completion (survives crashes)
16. Extract source archive: source.tar.gz of project files saved locally
17. Generate report: Markdown report with build/verify/deploy scores, skill coverage, URLs

Sandbox Session Flow (Per Scenario)

Verification Phase Details

The verify phase is the "closer" — its job is to make the app work and prove it. Key behaviors:

• Always runs if >1 project file exists (no longer gated on port 3000 being up)
• Starts dev server itself if not already running — the prompt tells Claude to check localhost:3000 and run npx next dev --port 3000 if needed
• 20 minute timeout — enough for agent-browser to open pages, screenshot, interact, fix broken code, restart server, and re-verify
• Triggers skill injection — the verify session creates/edits files, triggering PreToolUse and PostToolUse hooks
• Uses agent-browser workflow: open → wait --load networkidle → screenshot --annotate → snapshot -i → interact → fix → re-verify
• Results scored by haiku — no more parsing STORY_1: PASS from free text

Deploy Phase Details

The deploy phase uses a full Claude Code session (for skill tracking) to:

1. Run vercel link --yes --scope vercel-labs --project <slug>-YYYYMMDD
2. Run vercel deploy --yes
3. If build fails, fix code and retry (up to 3 attempts)
4. Important: unsets VERCEL_TOKEN env var so CLI falls back to ~/.local/share/com.vercel.cli/auth.json
5. Deployment protection is enabled by default on vercel-labs team

Deploy URL is extracted by regex from Claude's output, with haiku as fallback URL extractor.

DO NOT (Hard Rules)

Same rules as benchmark-agents, plus sandbox-specific:

• DO NOT use claude --print or -p flag for BUILD/VERIFY/DEPLOY phases — hooks don't fire without tool-calling sessions (use -p only for haiku scoring passes)
• DO NOT let sandboxes run without extracting artifacts — ephemeral filesystem is lost on stop
• DO NOT pass API keys via writeFiles() — use Sandbox.create({ env: { ... } })
• DO NOT skip snapshotting after build — it's your safety net if verify/deploy kills the sandbox
• DO NOT use v2 beta SDK — named sandbox endpoint returns 404 for this team; use v1.8.0
• DO NOT use runCommand heredocs to write file content — use sandbox.writeFiles() instead
• DO NOT assume /home/user/ exists — the home dir is /home/vercel-sandbox/
• DO NOT use simple project names without timestamps — always append -YYYYMMDDHHMM to avoid collisions across runs

Prerequisites

Commands

Run eval with dynamic scenarios (recommended)

Monitoring While Running

The orchestrator prints live status. For manual checks on a running sandbox:

Artifact Export Layout

Results are written to ~/dev/vercel-plugin-testing/sandbox-results/<run-id>/:

Each scenario result includes:

• slug, sandboxId, success, durationMs
• claimedSkills[], expectedSkills[], projectFiles[]
• appUrl — public https://sb-XXX.vercel.run URL (sandbox lifetime only)
• deployUrl — permanent https://xxx.vercel.app URL (if deploy succeeded)
• pollHistory[] — timestamped skill/file/port snapshots
• verification — { ran, exitCode, stories: [{ index, status }], output }
• buildScore — haiku structured completeness assessment
• deployScore — haiku structured deploy assessment

The markdown report (report.md / .reports/<timestamp>.md) includes:

1. Summary table — slug, build status, skills, files, verify results, deploy URL, duration
2. Per-scenario details — build score, deploy score, verification per-story pass/fail
3. Skill coverage — expected vs actual per scenario, missing/bonus breakdown
4. Total unique skills across all scenarios

Proven Results (2026-03-10)

Across 34 scenarios run in 5 batches:

Key findings:

• User-story-focused prompts (no tech name-dropping) work — plugin detects patterns from actual code
• ai-sdk, shadcn, nextjs, vercel-functions are the most consistently detected skills
• cron-jobs, routing-middleware need Claude to write specific file patterns to trigger
• Lexical prompt inject (UserPromptSubmit) working — skills injected before any files written
• session-end-cleanup deletes claim dirs — use poll history for final skill counts
• Enterprise tier (vercel-labs) — no sandbox time cap; builds ran 10+ minutes

Known Limitations

1. Snapshot stops the source sandbox: sandbox.snapshot() stops the original sandbox. Create a new sandbox from the snapshot to continue. Files and npm globals DO survive.
2. v2 beta incompatible: @vercel/sandbox@2.0.0-beta.3's named sandbox endpoint returns 404 for this team. Stick with v1.8.0.
3. Artifact window: Must extract before sandbox.stop() — filesystem is ephemeral. Session cleanup hook may delete claim dirs before extraction.
4. Amazon Linux paths: User is vercel-sandbox (home at /home/vercel-sandbox/). NOT /home/user/ or /root/.
5. --dangerously-skip-permissions parity: Sandbox evals auto-approve all tool calls. WezTerm evals use normal permission flow. Coverage results may differ.
6. runCommand timeout: Use { signal: AbortSignal.timeout(ms) } — the { timeout } option is silently ignored.
7. BrotliDecompressionError: Transient Vercel API errors can kill sandbox creation. Retry logic recommended for production runs.
8. Deploy reliability: Claude Code deploy sessions sometimes fail to output a parseable *.vercel.app URL. The haiku scoring step provides a fallback URL extraction attempt.
9. Verify timeout: Complex apps may need the full 20 minutes for agent-browser to test all stories. Simpler apps finish in 2-5 minutes.