mastra-smoke-test

Mastra Smoke Test

Comprehensive smoke testing for Mastra projects.

Release smoke workflows

Use progressive disclosure: stay in this file until the workflow branches, then read only the reference for the branch you are on. references/release-smoke.md is a short index if you need the full map.

Alpha release branch point

Before alpha smoke testing, identify the alpha versioning PR state. Prefer the standard Changesets release branch:

gh pr view changeset-release/main \
  --json number,title,state,url,headRefName,baseRefName,isDraft,mergeable,reviewDecision,updatedAt,mergedAt,mergeCommit

Expected shape:

title: chore: version packages (alpha)
head: changeset-release/main
base: main

If that branch lookup fails, search open and recently merged PRs:

gh pr list --state open --search 'version packages alpha in:title' --limit 20

Then branch:

• If the versioning PR is open, read references/alpha-versioning-pr.md.
• If the versioning PR is merged, read references/alpha-publish.md.
• If no versioning PR exists, report that and wait for the scheduled alpha versioning flow or user direction.

Do not create the alpha smoke-test project until the automatic alpha publish workflow has completed and the intended packages are installable.

Stable release branch point

If the user is running the stable/full release workflow, read references/stable-release-smoke.md. If that workflow fails after some packages publish, switch to references/stable-partial-publish-recovery.md.

Scope and targeted checks

After the release package is published and before running smoke tests, read references/release-scope-discovery.md to identify changed features. Use the default generated project for the baseline checklist, then add targeted checks for changed features the generated project does not exercise.

When scope discovery identifies a branch:

• For general changed-feature coverage, read references/targeted-feature-smoke.md.
• For storage/provider schema or migration changes, read references/storage-provider-migration-smoke.md.

⚠️ Mandatory Test Checklist

Use task_write to track progress. Run ALL tests unless --test specifies otherwise.

Do not skip tests unless you hit an actual blocker. "Seemed complex" or "wasn't sure" are not valid reasons. Attempt everything - only stop a test when you literally cannot proceed. Report what you tried and what blocked you.

Execution Flow

1. Read the reference file for each test you're about to run
2. Execute the steps in that reference file
3. Mark the test complete before moving to the next

Partial Testing (--test)

If --test is provided:

1. Always run Setup (step 1)
2. Run only the specified test(s)
3. Skip other tests

Example: --test agents,traces → Run steps 1, 2, and 5 only.

Local Studio Browser Smoke

For local release smoke tests, do both API/curl checks and a Studio browser pass unless --skip-browser is explicitly requested or browser access is genuinely blocked. API checks prove runtime endpoints work; browser checks prove the Playground/Studio UI can load, submit forms, and display results.

Before opening the browser:

1. Confirm the dev server is alive on the expected port:

   curl -s -o /dev/null -w '%{http_code}\n' http://localhost:4111
   lsof -i :4111 || true

2. If the process died, restart it from the generated project and wait for readiness:

   cd "$SMOKE_DIR/smoke-project"
   pnpm run dev > "$SMOKE_DIR/logs/dev-server-browser.log" 2>&1 &
    
   for i in {1..60}; do
     code=$(curl -s -o /dev/null -w '%{http_code}' http://localhost:4111 || true)
     [ "$code" = 200 ] && break
     sleep 1
   done

3. Use browser tools to navigate to http://localhost:4111. If networkidle times out but domcontentloaded succeeds and the UI is usable, continue and note the timeout.

Recommended browser task list:

1. Verify Studio shell loads
2. Smoke test agent chat UI
3. Smoke test tools UI
4. Smoke test workflows UI
5. Smoke test observability, scorers, and MCP pages
6. Report browser smoke results

Run these page checks:

If a browser interaction does not expose enough text in the accessibility snapshot, inspect document.body.innerText or take a screenshot, then record the visible evidence. Do not rely only on API output for browser smoke.

Append browser results to $SMOKE_DIR/smoke-report.md with a separate section, for example:

## Studio Browser Smoke Results
 
| Area         | Result | Evidence                                                          |
| ------------ | ------ | ----------------------------------------------------------------- |
| Studio shell | PASS   | Browser loaded localhost:4111; sidebar/nav visible; version shown |
| Agents UI    | PASS   | Weather Agent chat returned Tokyo weather and displayed tool call |
| Tools UI     | PASS   | get-weather form returned Paris weather JSON                      |
| Workflows UI | PASS   | weather-workflow Berlin run completed as success                  |
| Traces UI    | PASS   | Recent agent/workflow traces listed                               |
| Scorers UI   | PASS   | Expected scorers listed                                           |
| MCP UI       | PASS   | Expected empty MCP state shown                                    |

Call out separately whether browser smoke was local Studio only or cloud Studio/deployed server.

Usage

# Full smoke test
smoke test --env local --existing-project ~/my-app
smoke test --env staging -d ~/projects -n test-app
 
# Partial testing
smoke test --env local --existing-project ~/my-app --test agents
smoke test --env production --existing-project ~/my-app --test studio,server,traces
 
# Multi-environment: same project, different targets
smoke test --env staging --existing-project ~/my-app   # Uses .mastra-project-staging.json
smoke test --env production --existing-project ~/my-app # Uses .mastra-project.json

Multi-Environment Support

One project can target all environments using separate config files:

See references/tests/setup.md for setup details.

Parameters

* Either --directory + --name OR --existing-project required

Test Options (--test)

Prerequisites

All environments:

• Node.js + package manager
• LLM API key in env or .env

Local (--env local):

• Browser tools enabled (/browser on)

Cloud (--env staging/production):

• Mastra platform account

Quick Start Flow

1. Setup      → Read references/tests/setup.md, create/verify project
2. Start      → `pnpm run dev` (local) or deploy (cloud)
3. Test       → For each test, read its reference file and execute
4. Verify     → Check all items in reference file's checklist
5. Report     → Summarize pass/fail for each test

References

Platform Dashboards

• Production: https://projects.mastra.ai
• Staging: https://projects.staging.mastra.ai

For Gateway API testing (memory, threads, BYOK via gateway), use platform-smoke-test.

Result Reporting

After testing, provide:

## Smoke Test Results
 
**Environment**: local/staging/production
**Project**: <name>
 
| Test   | Status | Notes |
| ------ | ------ | ----- |
| Setup  | ✅/❌  |       |
| Agents | ✅/❌  |       |
| Tools  | ✅/❌  |       |
| ...    |        |       |
 
**Issues Found**: (list any)
**Warnings**: (list any deploy/runtime warnings)
**Skipped Tests**: (list with reason - e.g., "Server Deploy - not applicable in local environment")