analyze-ci-failures

vcpkg CI Failures Analyzer

When to Use

• Investigating CI failures on Azure DevOps
• Identifying regressions in a PR or scheduled build
• Triaging root causes before assigning bugs

Overview

Fetches build metadata and failure logs via Azure DevOps REST API, cross-references with baselines, produces a regression report.

MCP Tools

Prerequisites

• GitHub MCP server — needed for PR URL input

Workflow

OUTPUT RULE: Write report.md as SOON as step-log analysis (Phase 1) completes. Do not wait for artifact downloads. You can update the file later. Your response MUST also contain the complete report content — not a summary.

URL RULE: Your response text (not just the report file) MUST include:

• For scheduled/manual builds: the full Azure DevOps URL https://dev.azure.com/vcpkg/public/_build/results?buildId={buildId}
• For PR builds: both the full PR URL https://github.com/microsoft/vcpkg/pull/{prNumber} AND the Azure DevOps build URL

Before doing Azure-specific work, read .\.github\skills\shared\azure-vcpkg-ci-notes.md and follow its shared API and log-handling rules.

When you use the shared helper script, execute the .ps1 file directly by path. Do not open the script and inline its contents into a large PowerShell block.

Phase 1: Extract failures from step logs (REQUIRED — do first)

1. Parse input — Extract buildId from Azure DevOps URL. For PR URLs:

   $prNumber = 51515
   $builds = (Invoke-RestMethod "https://dev.azure.com/vcpkg/public/_apis/build/builds?reasonFilter=pullRequest&repositoryType=GitHub&repositoryId=microsoft/vcpkg&branchName=refs/pull/$prNumber/merge&api-version=7.0").value
   $buildId = ($builds | Sort-Object -Property id -Descending | Select-Object -First 1).id

   See references/azure-devops-api.md. For PR inputs where you want the failed step logs immediately, you may instead use the shared helper script:

   & '.\.github\skills\shared\Get-VcpkgAzureFailureLogs.ps1' -PrNumber <pr>

2. Fetch metadata — Build info, timeline, and artifacts list (parallelize these API calls).

3. Scan step logs — For each failed job, find "*** Test Modified Ports" task and fetch its log:

   $logText = Invoke-RestMethod "$($task.log.url)?api-version=7.0"
   $regressions = $logText -split "`n" | Where-Object { $_ -match 'REGRESSION:' }

   You may also use the shared helper script directly from a build id:

   & '.\.github\skills\shared\Get-VcpkgAzureFailureLogs.ps1' -BuildId <buildId>

   Captures all types: BUILD_FAILED, FILE_CONFLICTS, POST_BUILD_CHECKS_FAILED, CASCADED_DUE_TO_MISSING_DEPENDENCIES. Report every REGRESSION line using the EXACT failure type keyword from the log — never paraphrase. Also capture 2-3 lines around each error for root cause context. Extract the triplet name from each failed job (e.g., from job names like x64-windows Build or from log paths) — ensure ALL triplets with failures are listed in your report.

4. PR feature-test logs — PR builds may NOT have REGRESSION: lines. Instead scan for FAIL: or failed with lines showing per-feature failures. Capture and quote verbatim:

   • Compiler errors (missing headers, undefined symbols)
   • Post-build check failures (file path issues, misplaced files)
   • Version validation errors
   • Platform-specific feature guard messages

   Report each feature failure individually. Dependency ports that fail get their own entry.

5. Version validation — Check "Validate version files" task. If failed, scan for version database errors and quote them verbatim. Fix: vcpkg x-add-version.

6. Write report immediately — Generate and save report.md using all step-log data. This ensures output exists even if later steps time out.

Phase 2: Download logs and enhance (time permitting)

7. Download logs — Invoke-WebRequest or curl shell commands for artifact ZIPs (not web_fetch). Only download "failure logs for {triplet}" — skip "file lists". Extract into ci-failure-analysis/{scope}/logs/{triplet}/. If download fails, still create the directory with a placeholder noting the URL.
8. Analyze — Read stdout-{triplet}.log last lines. Classify per references/vcpkg-failure-patterns.md. Update report with additional root cause detail.
9. Baselines — Check both ci.baseline.txt and ci.feature.baseline.txt.

Report Requirements

Format per references/report-template.md:

• Full build URL: [{buildNumber}](https://dev.azure.com/vcpkg/public/_build/results?buildId={buildId})
• For PRs: [#{prNumber}](https://github.com/microsoft/vcpkg/pull/{prNumber})
• List every triplet by full name (e.g., x64-windows, arm64-linux) — never "N triplets". Only include triplets that actually had failures for this specific build.
• Use exact failure type keywords from logs: BUILD_FAILED, POST_BUILD_CHECKS_FAILED, FILE_CONFLICTS, CASCADED_DUE_TO_MISSING_DEPENDENCIES — never paraphrase
• Include error messages verbatim — quote the exact text from logs for compiler errors, path issues, and validation failures
• Dependency ports' failures as separate entries
• Include baseline/known failures — report them with their failure types, but classify them separately from new regressions

Output Structure

ci-failure-analysis/
├── ci-129315/         ← scheduled build
│   ├── report.md
│   └── logs/
│       ├── x64-windows/
│       └── arm64-linux/
└── pr-51202/          ← PR build
    ├── report.md
    └── logs/

Critical Rules

• Use Invoke-WebRequest or curl shell commands for ZIPs — web_fetch can't download binaries
• Artifact type is PipelineArtifact — Container API won't work
• Scan step logs first — FILE_CONFLICTS only appear there
• Check both baseline files
• Never suggest <= version constraints or VCPKG_BUILD_TYPE release
• If artifact download fails, still create logs/{triplet}/ directory with a placeholder noting the download URL**`