cli-e2e-testing

Aspire CLI End-to-End Testing with Hex1b

This skill provides patterns and practices for writing end-to-end tests for the Aspire CLI using the Hex1b terminal automation library.

Overview

CLI E2E tests use the Hex1b library to automate terminal sessions, simulating real user interactions with the Aspire CLI. Tests run in CI with asciinema recordings for debugging.

Location: tests/Aspire.Cli.EndToEnd.Tests/

Supported Platforms: Linux only. Hex1b requires a Linux terminal environment. Tests are configured to skip on Windows and macOS in CI.

Key Components

Core Classes

• Hex1bTerminal: The main terminal class from the Hex1b library for terminal automation
• Hex1bTerminalAutomator: Async/await API for driving a Hex1bTerminal — the preferred approach for new tests
• Hex1bAutomatorTestHelpers (shared helpers): Async extension methods on Hex1bTerminalAutomator (WaitForSuccessPromptAsync, AspireNewAsync, etc.)
• CliE2EAutomatorHelpers (Helpers/CliE2EAutomatorHelpers.cs): CLI-specific async extension methods on Hex1bTerminalAutomator (PrepareDockerEnvironmentAsync, InstallAspireCliAsync, etc.)
• CellPatternSearcher: Pattern matching for terminal cell content
• SequenceCounter (Helpers/SequenceCounter.cs): Tracks command execution count for deterministic prompt detection
• CliE2ETestHelpers (Helpers/CliE2ETestHelpers.cs): Environment variable helpers and terminal factory methods
• TemporaryWorkspace: Creates isolated temporary directories for test execution
• Hex1bTerminalInputSequenceBuilder (legacy): Fluent builder API for building sequences of terminal input/output operations. Prefer Hex1bTerminalAutomator for new tests.

Test Architecture

Each test:

1. Creates a TemporaryWorkspace for isolation
2. Builds a Hex1bTerminal with headless mode and asciinema recording
3. Creates a Hex1bTerminalAutomator wrapping the terminal
4. Drives the terminal with async/await calls and awaits completion

Test Structure

public sealed class SmokeTests(ITestOutputHelper output)

TerminalRun Pattern

Always use CliE2ETestHelpers.StartRun to wrap the terminal run. This returns a TerminalRun (implements IAsyncDisposable) that automatically:

1. Captures Aspire diagnostics via CaptureAspireDiagnosticsAsync (best effort)
2. Types exit and presses Enter to close the terminal
3. Awaits the pending run task

This eliminates the need for manual exit/await pendingRun at the end of every test and ensures diagnostics are always captured, even when tests fail.

Running Tests Locally

CLI E2E tests run inside Docker containers on Linux. The workflow is: build a portable archive with localhive, then point the tests at it. This is the primary way to iterate on E2E tests during development.

Prerequisites

• Docker Desktop running (with Linux containers)
• .NET 10 SDK (installed via ./restore.sh or .\restore.cmd)

Quick Start (macOS / Linux)

Quick Start (Windows / PowerShell)

Choosing the Right RID

The archive must match the Docker container's architecture:

Development Workflow

The typical loop when writing or debugging E2E tests:

Install Modes

The CliInstallStrategy class auto-detects how to install the CLI in the test container. You can override via environment variables:

LocalHive (via ASPIRE_E2E_ARCHIVE) is the recommended mode for local development — it uses your locally-built CLI, packages, and bundle so you test exactly what you've changed.

Testing Against Released Versions

Useful for verifying tests pass against shipped versions or catching regressions:

Emulated channel matrix tests (identity sidecar)

A set of tests validates the CLI identity sidecar — the ability to make a locally built CLI emulate a different channel/version via ASPIRE_CLI_* env vars. They form an AppHost-language × channel-emulation matrix (one test per language because C# and TypeScript scaffold through different code paths and have diverged before):

EmulatedLocalReleaseBuildTests is the all-local "future release" row: it emulates a version (e.g. 13.5.0) that exists only in a locally built hive, so a successful resolve proves the CLI consulted ASPIRE_CLI_PACKAGES rather than nuget.org. Run it by building a stable-shaped archive with localhive --version:

These tests skip unless the CLI was installed from a LocalHive archive and that archive is stable-shaped (no prerelease suffix). In default CI the archive is a prerelease LocalArchive, so they skip and add zero CI cost — CI relies on --ignore-exit-code 8 (set in eng/Testing.props MtpBaseArgs) so an all-skipped class job still passes. The test also registers the hive as an ambient NuGet source (dotnet nuget add source) because MSBuild resolves the apphost's Aspire.AppHost.Sdk before restore from nuget.config sources only, ignoring ASPIRE_CLI_PACKAGES.

⚠️ Local rebuilds: isolate the NuGet global cache. This only affects local iteration of a stable-shaped emulation (the E2E tests run in fresh Docker containers, so CI is immune). NuGet's global packages folder (~/.nuget/packages/<id>/<version>/) caches extracted packages keyed by version. When you emulate a fixed stable version (e.g. 13.5.0) and rebuild it, a stale 13.5.0 in that shared cache silently shadows the freshly built one — same version, different content — so restore drifts (the stale AppHost SDK injects a prerelease floor and you get NU1603 warnings binding the graph to a stray 13.5.0-pr.…). Fix: point NUGET_PACKAGES at a per-emulation directory (export NUGET_PACKAGES=/tmp/aspire-localrelease/.nuget-packages). localhive … -o DIR's generated activate.sh/activate.ps1 already sets this up. See the cache hazard note in .agents/skills/cli-channel-debugging/SKILL.md (Scenario 7c) for the full mechanism.

SequenceCounter and Prompt Detection

The SequenceCounter class tracks the number of shell commands executed. This enables deterministic waiting for command completion via a custom shell prompt.

How It Works

1. PrepareDockerEnvironmentAsync() configures the shell with a custom prompt: [N OK] $ or [N ERR:code] $
2. Each command increments the counter
3. WaitForSuccessPromptAsync(counter) waits for a prompt showing the current count with OK

This approach is more reliable than arbitrary timeouts because it deterministically waits for each command to complete.

Pattern Searching with CellPatternSearcher

Use CellPatternSearcher to find text patterns in terminal output:

Find vs FindPattern

• Find(string): Literal string matching. Use this for most cases.
• FindPattern(string): Regex pattern matching. Use only when you need regex features like wildcards.

Important: If your search string contains regex special characters like (, ), /, ., *, +, ?, [, ], {, }, ^, $, |, or \, use Find() instead of FindPattern() to avoid regex interpretation.

Extension Methods

Hex1bAutomatorTestHelpers Extensions (Shared — Automator API)

See AspireNew Helper below for detailed usage.

CliE2EAutomatorHelpers Extensions on Hex1bTerminalAutomator

SequenceCounterExtensions

Legacy Builder Extensions

The following extensions on Hex1bTerminalInputSequenceBuilder are still available but should not be used in new tests:

DO: Use CellPatternSearcher for Output Detection

Wait for specific output patterns rather than arbitrary delays:

DO: Use WaitForSuccessPromptAsync After Commands

After running shell commands, use WaitForSuccessPromptAsync() to wait for the command to complete:

AspireNew Helper

The AspireNew extension method centralizes the multi-step aspire new interactive flow. Use it instead of manually building the prompt sequence.

AspireTemplate Enum

Parameters

Usage Examples

DO: Handle Interactive Prompts

For aspire new, use the AspireNewAsync helper instead of manually building the prompt sequence:

For other interactive CLI commands, wait for each prompt before responding:

DO: Use Ctrl+C to Stop Long-Running Processes

For processes like aspire run that don't exit on their own:

DO: Check IsRunningInCI for CI-Only Operations

Some operations only apply in CI (like installing CLI from PR artifacts):

DO: Get Environment Variables Using Helpers

Use CliE2ETestHelpers for CI environment variables:

DO: Always Include description: on WaitUntilAsync

Every WaitUntilAsync call requires a named description: parameter. This description appears in logs and asciinema recordings to make debugging easier when a wait times out.

DO: Inline Code Where ExecuteCallback Was Used

The old builder API used ExecuteCallback() to run synchronous operations mid-sequence. With the automator API, simply inline the code directly — no special wrapper is needed.

DON'T: Use Hard-coded Delays

Use WaitUntilAsync() with specific output patterns instead of arbitrary delays:

DON'T: Hard-code Prompt Sequence Numbers

Don't hard-code the sequence numbers in WaitForSuccessPromptAsync calls. Use the counter:

The counter automatically tracks which command you're waiting for, even if command sequences change.

Writing New Tests with Hex1b MCP Server

When writing new CLI E2E tests, use the Hex1b MCP server to interactively explore what terminal output to expect. The MCP server provides tools to start terminal sessions, send commands, and capture screenshots—helping you discover the exact strings and prompts to use in CellPatternSearcher.

Workflow for Discovering Patterns

1. Start a bash terminal session using the MCP server's terminal creation tools
2. Send commands (like aspire new or aspire run) and observe the output
3. Capture terminal screenshots (SVG or text) to see exact formatting
4. Use captured text to build your CellPatternSearcher patterns

Example: Finding Prompt Text for aspire new

Ask the MCP server to:

1. Start a new bash terminal
2. Run aspire new interactively
3. Capture the terminal text at each prompt

This reveals the exact strings like:

• "> Starter App" for template selection
• "Enter the project name" for name input
• "Press Ctrl+C to stop..." for run completion

Benefits

• See real output: No guessing what text appears in the terminal
• Exact formatting: Capture shows spacing, ANSI codes stripped, actual cell content
• Interactive exploration: Try different inputs and see responses before writing test code
• Debug patterns: If a CellPatternSearcher isn't matching, capture current terminal state to compare

Tips

• Use Capture Terminal Text to get plain text for pattern matching
• Use Capture Terminal Screenshot (SVG) for visual debugging
• The Wait for Terminal Text tool works similarly to WaitUntil in tests
• Terminal sessions persist, so you can step through multi-command sequences

Adding New Extension Methods

When adding new CLI operations as extension methods, define them on Hex1bTerminalAutomator:

Key points:

1. Define as async extension method on Hex1bTerminalAutomator
2. Accept SequenceCounter parameter for prompt tracking
3. Use CellPatternSearcher for output detection
4. Always include description: on WaitUntilAsync calls
5. Call WaitForSuccessPromptAsync(counter) after command completion
6. Return Task (no fluent chaining needed with async/await)

CI Configuration

Environment variables set in CI:

• GITHUB_PR_NUMBER: PR number for downloading CLI artifacts
• GITHUB_PR_HEAD_SHA: PR head commit SHA for version verification (not the merge commit)
• GH_TOKEN: GitHub token for API access
• GITHUB_WORKSPACE: Workspace root for artifact paths

Each test class runs as a separate CI job via the unified TestEnumerationRunsheetBuilder infrastructure (using SplitTestsOnCI=true) for parallel execution.

CI Troubleshooting

When CLI E2E tests fail in CI, follow these steps to diagnose the issue:

Flaky test investigation: for recurring/intermittent failures, see troubleshooting.md for a catalog of known flake classes (Y/n input race, prompt-counter desync, etc.) and the recipes to identify them from .cast recordings.

VS Code Extension E2E Diagnostics

For VS Code extension behavior or extension/CLI integration issues, strongly prefer adding or updating a reproducible test under extension/src/test-e2e/. Use agent-driven Playwright/VS Code UI driving only as exploratory diagnosis when the E2E scenario is not clear yet; convert any successful manual reproduction into an E2E test before fixing the bug unless there is a strong, explicit reason not to.

When running extension E2E tests against an older published CLI for compatibility validation, set ASPIRE_EXTENSION_E2E_SKIP_CURRENT_CLI_REGRESSIONS=true to skip tests that intentionally cover bugs fixed only by the current repo-built CLI.

VS Code extension E2E jobs upload shard-specific diagnostics as extension-e2e-diagnostics-<rid>-<shard>-attempt<N> artifacts. Linux shards include .mp4 display recordings from Xvfb by default; Windows shards do not record video and instead rely on screenshots, VS Code logs, state files, and workspace diagnostics.

Important paths inside the downloaded shard:

The workflow keeps Linux recordings by default with ASPIRE_EXTENSION_E2E_RECORDING_MODE=always. Use failure to keep only failed-run videos or off to disable recording for local runs.

Quick Start: Download and Play Recordings

The fastest way to debug a CLI E2E test failure is to download and play the asciinema recording.

Using the helper scripts (recommended):

Manual download steps:

Step 1: Find the CI Run

Step 2: Find CLI E2E Test Artifacts

Job names follow the pattern: Tests / Cli E2E Linux (<TestClass>) / <TestClass> (ubuntu-latest)

Artifact names follow the pattern: logs-<TestClass>-ubuntu-latest

Step 3: Download and Play Recording

Artifact Contents

Downloaded artifacts contain:

Workspace Capture

Tests annotated with [CaptureWorkspaceOnFailure] automatically copy the generated project workspace into the test artifacts when a test fails. This is invaluable for debugging template generation or aspire run failures — you can inspect the exact generated files including the SDK output in .aspire/modules/aspire.js.

To add workspace capture to a new test:

 
### One-Liner: Download Latest Recording
 
```bash
# Download and play the latest CLI E2E recording from current branch
RUN_ID=$(gh run list --branch $(git branch --show-current) --workflow CI --limit 1 --json databaseId --jq '.[0].databaseId') && \
  rm -rf /tmp/cli-e2e-debug && mkdir -p /tmp/cli-e2e-debug && \
  gh run download $RUN_ID -n logs-SmokeTests-ubuntu-latest -D /tmp/cli-e2e-debug && \
  CAST=$(find /tmp/cli-e2e-debug -name "*.cast" | head -1) && \
  echo "Recording: $CAST" && \
  asciinema play "$CAST"

Common Issues and Solutions