npx skills add ...
npx skills add microsoft/aspire --skill ci-test-failures
npx skills add microsoft/aspire --skill ci-test-failures
Guide for diagnosing GitHub Actions test failures, extracting failed tests from runs, and creating or updating failing-test issues. Use this when asked to investigate GitHub Actions test failures, download failure logs, create failing-test issues, or debug CI issues.
When the user asks to create an issue for a failing test, follow these steps. Always redirect full output to a log file (not tail) so you can inspect it if the command fails.
Omit --test to discover all failures. Redirect output to a log file:
Then read the result with jq:
If success is false, inspect the full log: cat /tmp/cfti-list.log (bash) or Get-Content $env:TEMP/cfti-list.log (PowerShell).
Ask the user which test to file for, then proceed to Step 2.
Then read the result:
If success is false, inspect the full log: cat /tmp/cfti-create.log (bash) or Get-Content $env:TEMP/cfti-create.log (PowerShell).
That's it — do not add analysis comments, do not use --dry-run unless the user explicitly asks for a preview.
Rules:
--output <file> to keep JSON clean. Do NOT try to parse JSON from stdout — it is interleaved with dotnet build progress output.jq to extract fields from the output file. Key paths:
.success — whether the operation succeeded.issue.createdIssue.number and .issue.createdIssue.url — the created/updated issue.diagnostics.availableFailedTests[] — test names when --test is omitted.errorMessage — error details when .success is falsegh issue create. The tool handles everything: resolving the run, finding the test, generating a template-compliant body, and creating the issue..github/ISSUE_TEMPLATE/50_failing_test.yml.diagnostics.log and the JSON output. Do not fall back to manual issue creation.To download and inspect failure artifacts without creating an issue:
Then search the downloaded logs and .trx files for errors.
Everything below is reference material for edge cases and deeper investigation.
Use this skill in two phases:
DownloadFailingJobLogs.cs to fetch failed job logs and artifacts.tools/CreateFailingTestIssue --create.| Tool | Purpose | Location |
|---|---|---|
DownloadFailingJobLogs.cs | Download failed job logs and test artifacts from a GitHub Actions run | tools/scripts/DownloadFailingJobLogs.cs |
CreateFailingTestIssue | Resolve a failing test from PR/run/job URLs and create/update issues | tools/CreateFailingTestIssue |
/create-issue workflow | Create, reopen, or comment on failing-test issues from issue/PR comments | .github/workflows/create-failing-test-issue.yml |
Get the run ID from the GitHub Actions URL or use the gh CLI:
Example:
The tool creates files in your current directory:
| File Pattern | Contents |
|---|---|
failed_job_<n>_<job-name>.log | Raw job logs from GitHub Actions |
artifact_<n>_<testname>_<os>.zip | Downloaded artifact zip files |
artifact_<n>_<testname>_<os>/ | Extracted directory with .trx files, logs, binlogs |
logs-{testShortName}-{os})After you know which test failed, use the branch automation to create a failing-test issue in the known-issues format.
/create-issue from a PR or issue commentComment on the PR or issue with:
Examples:
Notes:
--url is supplied, the workflow defaults to that PR URL.--workflow defaults to ci.--force-new bypasses issue reuse and always requests a fresh issue.The resolver accepts:
https://github.com/<owner>/<repo>/pull/<number>https://github.com/<owner>/<repo>/actions/runs/<run-id>https://github.com/<owner>/<repo>/actions/runs/<run-id>/attempts/<attempt>https://github.com/<owner>/<repo>/actions/runs/<run-id>/job/<job-id>Always use --output to write results to a file so JSON is not interleaved with build output:
To generate the JSON result locally without creating an issue (dry run):
To resolve the failure and create the issue on GitHub in one step:
Read the result with jq:
If --test is omitted, the tool emits structured JSON for all failing tests it found in the run (useful for picking which test to file).
The command writes a diagnostics.log file in the current directory. The JSON output (written to the --output file or stdout) contains:
--create is set, the created issue number and URLCreateFailingTestIssue:
.trx artifacts..github/ISSUE_TEMPLATE/50_failing_test.yml. The error details code block is wrapped in a collapsible <details> element when it exceeds 30 lines.For a run with multiple failures, first extract the candidate test names, then issue one /create-issue command per test:
If the resolver cannot match the requested test exactly, it returns availableFailedTests so you can retry with one of the discovered names.
The tool prints a summary for each failed job:
Sometimes job logs aren't available (404). Use annotations instead:
This returns structured error information even when full logs aren't downloadable.
Symptom: No space left on device in annotations or logs
Diagnosis:
Common fixes:
8-core-ubuntu-latest)/p:BuildTests=false)Symptom: exit code 127 or command not found
Diagnosis:
Common fixes:
Symptom: Test hangs, then fails with timeout
Diagnosis:
Common fixes:
Symptom: Build FAILED or MSBuild errors
Diagnosis:
Common fixes:
Downloaded artifacts typically contain:
CLI E2E tests annotated with [CaptureWorkspaceOnFailure] automatically capture the full generated project workspace when a test fails. This includes the generated SDK (.aspire/modules/aspire.js), template output, and config files — critical for debugging template generation or aspire run failures.
Look in testresults/workspaces/{TestClassName.MethodName}/ inside the downloaded artifact.
The tool creates files in the current directory, so run it from tools/scripts to keep things organized:
The downloaded log files can be large. Don't commit them to the repository:
gh) installed and authenticatedtools/scripts/README.md - Full documentationtools/scripts/Heartbeat.cs - System monitoring tool for diagnosing hangs.agents/skills/cli-e2e-testing/SKILL.md - CLI E2E test troubleshooting`jq '{ success, availableFailedTests: .diagnostics.availableFailedTests, errorMessage: .errorMessage }' /tmp/cfti-result.jsonGet-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, errorMessage, @{N='availableFailedTests';E={$_.diagnostics.availableFailedTests}}dotnet run --project tools/CreateFailingTestIssue -- \
--url "<the-url-the-user-gave>" \
--test "<test-name>" \
--create \
--output /tmp/cfti-result.json \
> /tmp/cfti-create.log 2>&1dotnet run --project tools/CreateFailingTestIssue -- `
--url "<the-url-the-user-gave>" `
--test "<test-name>" `
--create `
--output $env:TEMP/cfti-result.json `
> $env:TEMP/cfti-create.log 2>&1jq '{ success, issue: .issue.createdIssue, errorMessage: .errorMessage }' /tmp/cfti-result.jsonGet-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, errorMessage, @{N='issue';E={$_.issue.createdIssue}}cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>Set-Location tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>Set-Location tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id>Set-Location tools/scripts
dotnet run DownloadFailingJobLogs.cs -- <run-id># From URL: https://github.com/microsoft/aspire/actions/runs/19846215629
# ^^^^^^^^^^
# run ID
# Or find the latest run on a branch
gh run list --repo microsoft/aspire --branch <branch-name> --limit 1 --json databaseId --jq '.[0].databaseId'
# Or for a PR
gh pr checks <pr-number> --repo microsoft/aspire# From URL: https://github.com/microsoft/aspire/actions/runs/19846215629
# ^^^^^^^^^^
# run ID
# Or find the latest run on a branch
gh run list --repo microsoft/aspire --branch <branch-name> --limit 1 --json databaseId --jq '.[0].databaseId'
# Or for a PR
gh pr checks <pr-number> --repo microsoft/aspiredotnet run DownloadFailingJobLogs.cs -- 19846215629/create-issue --test "<test-name>" [--url <pr|run|job-url>] [--workflow <selector>] [--force-new]/create-issue --test "Tests.Namespace.Type.Method(input: 1)"
/create-issue --test "Tests.Namespace.Type.Method(input: 1)" --url https://github.com/microsoft/aspire/actions/runs/123
/create-issue "Tests.Namespace.Type.Method(input: 1)" https://github.com/microsoft/aspire/actions/runs/123/job/456
/create-issue --test "Tests.Namespace.Type.Method(input: 1)" --url https://github.com/microsoft/aspire/actions/runs/123/attempts/2/job/456?pr=321 --force-newdotnet run --project tools/CreateFailingTestIssue -- \
--url "https://github.com/microsoft/aspire/actions/runs/123" \
--test "<test-name>" \
--repo "microsoft/aspire" \
--output /tmp/cfti-result.jsondotnet run --project tools/CreateFailingTestIssue -- `
--url "https://github.com/microsoft/aspire/actions/runs/123" `
--test "<test-name>" `
--repo "microsoft/aspire" `
--output $env:TEMP/cfti-result.jsondotnet run --project tools/CreateFailingTestIssue -- \
--url "https://github.com/microsoft/aspire/actions/runs/123" \
--test "<test-name>" \
--repo "microsoft/aspire" \
--create \
--output /tmp/cfti-result.jsondotnet run --project tools/CreateFailingTestIssue -- `
--url "https://github.com/microsoft/aspire/actions/runs/123" `
--test "<test-name>" `
--repo "microsoft/aspire" `
--create `
--output $env:TEMP/cfti-result.jsonjq '{ success, issue: .issue, availableFailedTests: .diagnostics.availableFailedTests }' /tmp/cfti-result.jsonGet-Content $env:TEMP/cfti-result.json | ConvertFrom-Json | Select-Object success, @{N='issue';E={$_.issue}}, @{N='availableFailedTests';E={$_.diagnostics.availableFailedTests}}Get-ChildItem -Path "artifact_*" -Recurse -Filter "*.trx" | ForEach-Object {
[xml]$xml = Get-Content $_.FullName
$xml.TestRun.Results.UnitTestResult |
Where-Object { $_.outcome -eq "Failed" } |
Select-Object -ExpandProperty testName
}# 1. Check failed jobs on a PR
gh pr checks 14105 --repo microsoft/aspire 2>&1 | Where-Object { $_ -match "fail" }
# 2. Get the run ID
$runId = gh run list --repo microsoft/aspire --branch davidfowl/my-branch --limit 1 --json databaseId --jq '.[0].databaseId'
# 3. Download failure logs
cd tools/scripts
dotnet run DownloadFailingJobLogs.cs -- $runId
# 4. Search for errors in downloaded logs
Get-Content "failed_job_0_*.log" | Select-String -Pattern "error|Error:" -Context 2,3 | Select-Object -First 20
# 5. Check .trx files for test failures
Get-ChildItem -Recurse -Filter "*.trx" | ForEach-Object {
[xml]$xml = Get-Content $_.FullName
$xml.TestRun.Results.UnitTestResult | Where-Object { $_.outcome -eq "Failed" }
}
# 6. Create or update the failing-test issue from the PR or issue thread
/create-issue --test "Tests.Namespace.Type.Method(input: 1)" --url https://github.com/microsoft/aspire/actions/runs/$runId=== Failed Job 1/1 ===
Name: Tests / Integrations macos (Hosting.Azure) / Hosting.Azure (macos-latest)
ID: 56864254427
URL: https://github.com/microsoft/aspire/actions/runs/19846215629/job/56864254427
Downloading job logs...
Saved job logs to: failed_job_0_Tests___Integrations_macos__Hosting_Azure____Hosting_Azure__macos-latest_.log
Errors found (2):
- System.InvalidOperationException: Step 'provision-api-service' failed...# PowerShell
Get-Content "failed_job_*.log" | Select-String -Pattern "error|Error:" -Context 2,3
# Bash
grep -i "error" failed_job_*.log | head -50Get-Content "failed_job_*.log" | Select-String -Pattern "Build FAILED|error MSB|error CS"Get-Content "failed_job_*.log" | Select-String -Pattern "Failed!" -Context 5,0Get-Content "failed_job_*.log" | Select-String -Pattern "No space left|disk space"Get-Content "failed_job_*.log" | Select-String -Pattern "timeout|timed out|Timeout"gh api repos/microsoft/aspire/check-runs/<job-id>/annotationsgh api repos/microsoft/aspire/check-runs/<job-id>/annotations 2>&1Get-Content "failed_job_*.log" | Select-String -Pattern "command not found|exit code 127" -Context 3,1Get-Content "failed_job_*.log" | Select-String -Pattern "Test host process exited|Timeout|timed out"Get-Content "failed_job_*.log" | Select-String -Pattern "error CS|error MSB|Build FAILED" -Context 0,3artifact_0_TestName_os/
├── testresults/
│ ├── TestName_net10.0_timestamp.trx # Test results XML
│ ├── Aspire.*.Tests_*.log # Console output
│ ├── recordings/ # Asciinema recordings (CLI E2E tests)
│ └── workspaces/ # Captured project workspaces (CLI E2E tests)
│ └── TestClassName.MethodName/ # Full generated project for failed tests
│ ├── apphost.ts
│ ├── aspire.config.json
│ ├── .aspire/modules/ # Generated SDK (aspire.js) - key for debugging
│ └── ...
├── *.crash.dmp # Crash dump (if test crashed)
└── test.binlog # MSBuild binary log# Find all failed tests in .trx files
Get-ChildItem -Path "artifact_*" -Recurse -Filter "*.trx" | ForEach-Object {
Write-Host "=== $($_.Name) ==="
[xml]$xml = Get-Content $_.FullName
$xml.TestRun.Results.UnitTestResult | Where-Object { $_.outcome -eq "Failed" } | ForEach-Object {
Write-Host "FAILED: $($_.testName)"
Write-Host $_.Output.ErrorInfo.Message
Write-Host "---"
}
}Remove-Item *.log -Force -ErrorAction SilentlyContinue
Remove-Item *.zip -Force -ErrorAction SilentlyContinue
Remove-Item -Recurse artifact_* -Force -ErrorAction SilentlyContinue# Before committing
rm tools/scripts/*.log
rm tools/scripts/*.zip
rm -rf tools/scripts/artifact_*# Before committing
Remove-Item tools/scripts/*.log -Force -ErrorAction SilentlyContinue
Remove-Item tools/scripts/*.zip -Force -ErrorAction SilentlyContinue
Remove-Item tools/scripts/artifact_* -Recurse -Force -ErrorAction SilentlyContinue