npx skills add ...
npx skills add openshift/release-tests --skill release-workflow
npx skills add openshift/release-tests --skill release-workflow
OpenShift z-stream release workflow and orchestration expert. Use when discussing release tasks, build promotion, test analysis, advisory workflows, or any aspect of the Konflux/Errata release pipeline. Provides context on task sequencing, checkpoints, and MCP execution for releases 4.12-4.20.
This document defines the complete workflow for OpenShift z-stream release orchestration under the Konflux release platform. All releases from 4.12 to 4.20 follow this flow.
Target Audience: AI agents (Claude Code) and human release engineers
Execution Method: All oar commands are executed via the MCP (Model Context Protocol) server, NOT local CLI. The MCP server exposes OAR commands as structured tools with proper input validation and output parsing.
_releases/{y-stream}/statebox/{release}.yamloar_get_release_status(release)_releases/ocp-test-result-{build}-amd64.jsonaggregated: true/false - All test jobs completed and results collectedaccepted: true/false - BO3 verification passedKey Characteristics:
analyze-candidate-build runs independently (tests already completed when flow starts)push-to-cdn-staging starts immediately after check-rhcos-security-alerts (runs while waiting for build promotion)Candidate Build:
4.20.0-0.nightly-2025-01-28-123456oar_get_release_metadata(release) → candidate_builds.x86_64Promoted Build:
X.Y.Z (z-stream version, e.g., 4.20.1)API Endpoint:
Success Criteria:
MCP Execution: Not a direct OAR command - AI must fetch URL and parse JSON response.
When to Check:
/release:drive {release} commandphase != "Accepted"), AI reports status to user/release:drive periodically until promotion completesWhen to run: Can start immediately when release flow begins (tests already completed)
Execution Flow:
Slash Command:
The slash command provides:
When to run: After build promotion detected (phase == "Accepted")
Execution Flow:
Slash Command:
ENHANCED LOGIC (Early Async Task Triggering):
The gate check has been optimized to trigger async tasks as soon as the stable build is promoted, without waiting for blocking tests analysis to complete.
Condition to Trigger Async Tasks:
Rationale: Once the stable build is accepted by ART and promoted to the release stream, we can immediately start parallel async tasks (image-consistency-check, stage-testing) to save time. The blocking tests analysis happens independently and doesn't need to gate these operations.
Condition to PROCEED to Final Approval:
Note: Candidate build analysis runs independently and doesn't block the gate check. It's informational for context.
If promoted build test analysis FAILS:
If promoted build test analysis PASSES:
Purpose: Initialize Google Sheets test report and ConfigStore entry
MCP Tool: oar_create_test_report(release)
Input:
release: Z-stream version (e.g., 4.20.1)Success Detection:
Failure Detection:
Expected Duration: 5 mins
Next Action: Proceed to take-ownership
Purpose: Assign release ownership to QE team member
MCP Tool: oar_take_ownership(release, email)
Input:
release: Z-stream versionemail: Owner email (e.g., user@redhat.com)AI Decision Logic:
/release:drive command: Use that emailoar_get_release_metadata(release) for current ownerSuccess Detection:
Expected Duration: 10 seconds
Next Action: Proceed to check-cve-tracker-bug
Purpose: Verify CVE tracker bug coverage for release
MCP Tool: oar_check_cve_tracker_bug(release)
Input:
release: Z-stream versionBehavior:
Success Detection:
Expected Duration: 1 minute
Next Action: Proceed to check-rhcos-security-alerts
Purpose: Check for blocking security alerts on RHCOS advisory (Konflux flow only)
When to run: Konflux release flow only (releases with shipment_mr in metadata)
Prerequisites: check-cve-tracker-bug completed
Implementation: Uses curl with Kerberos authentication (no existing MCP tool)
Execution Steps:
Step 1: Verify Kerberos ticket exists
If no ticket or ticket expired:
Step 2: Get RHCOS advisory ID
Step 3: Fetch security alerts from Errata Tool
Step 4: Parse response and check for blocking alerts
Success Detection:
Expected Duration: 10 seconds
Errata Tool API Response Format:
Key Fields:
.alerts.alerts[] (array) - List of individual alerts.alerts.alerts[].blocking (boolean) - Per-alert blocking status (THIS is what we check).alerts.blocking (boolean) - Top-level blocking status (informational only)Next Action:
Purpose: Push release images to CDN staging environment
MCP Tool: oar_push_to_cdn_staging(release)
Input:
release: Z-stream versionPrerequisites: check-cve-tracker-bug completed
Execution Phases:
Phase 1 - Trigger:
Phase 2 - Check Status:
When user invokes /release:drive, re-execute oar_push_to_cdn_staging(release) to check current status
Phase 3 - Complete:
Expected Duration: 30-60 minutes (user should check status every 5-10 minutes)
Note: This task runs in parallel with build promotion waiting. It doesn't depend on promotion status.
Failure Handling: If fails, mark overall status "Red", notify owner
Purpose: Evaluate test results from candidate nightly build
MCP Tool: Uses slash command /ci:analyze-build-test-results
Prerequisites: None - can run immediately when release flow starts
Input:
candidate_build: Retrieved from oar_get_release_metadata(release).candidate_builds.x86_64Execution Steps:
Step 1: Fetch test result file
Step 2: Check aggregation status
Step 3: Check acceptance status
Expected Duration: 2-5 minutes (if analysis needed)
Note: This task runs independently and provides context. It doesn't block the main pipeline gate check.
Purpose: Evaluate test results from promoted stable build
MCP Tool: Uses slash command /ci:analyze-build-test-results
Prerequisites: Build promotion detected (phase == "Accepted")
Input:
release: The z-stream version (e.g., 4.20.1)Execution Steps:
Step 1: Check for test result file
Step 2: Check for aggregation
Step 3: Check acceptance status and gate check
Expected Duration:
Next Action: If gate check passes, trigger async tasks (image-consistency-check, stage-testing)
Purpose: Verify payload images match shipment MR
MCP Tool: oar_image_consistency_check(release, job_id=None)
Input:
release: Z-stream versionjob_id: Optional Prow job ID (for status check)Prerequisites:
Execution Phases:
Phase 1 - Trigger:
Phase 2 - Check Status:
Phase 3 - Complete:
Expected Duration: 90-120 minutes after trigger succeeds
Failure Handling:
Purpose: Run stage testing jobs on Jenkins
MCP Tool: oar_stage_testing(release, build_number=None)
Input:
release: Z-stream versionbuild_number: Optional Jenkins build number (for status check)Prerequisites:
Execution Phases:
Phase 1 - Trigger:
Phase 2 - Check Status:
Phase 3 - Complete:
Expected Duration: 2-4 hours after trigger succeeds
Failure Handling:
Purpose: Verify release images are properly signed
MCP Tool: oar_image_signed_check(release)
Input:
release: Z-stream versionPrerequisites: All 3 async tasks (push-to-cdn-staging, image-consistency-check, stage-testing) must complete successfully
Success Detection:
Expected Duration: 2 minutes
Next Action: Proceed to change-advisory-status
Purpose: Change advisory status from QE to REL_PREP (final QE approval)
MCP Tool: oar_change_advisory_status(release)
Input:
release: Z-stream versionPrerequisites: All previous tasks must be "Pass"
Timing Guidance: This task should be run 1 day before the scheduled release date for optimal results.
How to determine release date:
Execution Flow:
Phase 1 - Trigger (Immediate Return):
IMPORTANT - Asynchronous Execution:
Phase 2 - Background Process (Runs Independently):
/tmp/oar_logs/metadata_checker_<release>.logPhase 3 - Completion:
If task times out:
oar_change_advisory_status(release) - checker restarts with fresh 2-day timeoutFinal Action: When background process succeeds, overall status marked "Green" and Slack notifications sent
Before making ANY decisions, AI must retrieve release state:
This returns task statuses, metadata, and any blocking issues from StateBox (or Google Sheets as fallback).
For Sequential Tasks:
For Test Result Analysis:
For Async Tasks:
For Parallel Tasks After Build Promotion (ENHANCED):
Transient Errors (Retry):
Permanent Errors (STOP):
Error Response:
Success Notifications:
Failure Notifications:
/ci:analyze-build-test-resultsSymptom: image-consistency-check or stage-testing fails to trigger
Diagnosis:
oar_get_release_metadata(release).shipment_mrCommon Causes:
Resolution:
/release:drivecd mcp_server && python3 server.pyhttps://amd64.ocp.releases.ci.openshift.org/api/v1/releasestream/{major_version}-stable/release/{release}
# Where {major_version} is extracted from {release} (e.g., "4" from "4.20.1", "5" from "5.0.1"){
"phase": "Accepted"
}1. Retrieve candidate build from oar_get_release_metadata(release).candidate_builds.x86_64
2. Fetch test result file from GitHub: _releases/ocp-test-result-{candidate_build}-amd64.json
3. Check attributes:
IF aggregated == false:
Report to user: "Candidate build tests still aggregating, check again later"
(Should rarely happen - tests complete before flow starts)
IF aggregated == true AND accepted == true:
Mark analyze-candidate-build as "Pass" (all tests passed)
No further action needed
IF aggregated == true AND accepted == false:
Trigger /ci:analyze-build-test-results {candidate_build}
Parse AI recommendation:
- ACCEPT: Mark task "Pass" (failures are waivable)
- REJECT: Mark task "Fail" (blocking issues found)/ci:analyze-build-test-results {candidate_build}1. [CHECKPOINT 1] Check for test result file existence
File: _releases/ocp-test-result-{release}-amd64.json
When user invokes /release:drive:
IF file does not exist:
Report to user: "Test result file not yet created, check again in 5-10 minutes"
Typical wait time: 10-120 minutes after promotion
ELSE:
Proceed to Checkpoint 2
2. [CHECKPOINT 2] Check for test aggregation
Read file and check: aggregated == true
When user invokes /release:drive:
IF aggregated == false:
Report to user: "Tests still running/aggregating, check again in 5-10 minutes"
Typical wait time: 10-30 minutes after file creation
ELSE:
Proceed to Checkpoint 3
3. [CHECKPOINT 3] Check acceptance status
IF accepted == true:
Mark analyze-promoted-build as "Pass" (all tests passed)
Proceed to gate check
IF accepted == false:
Trigger /ci:analyze-build-test-results {release}
Parse AI recommendation:
- ACCEPT: Mark task "Pass" (failures are waivable)
- REJECT: Mark task "Fail" (blocking issues found)/ci:analyze-build-test-results {release}Build promotion detected (phase == "Accepted")promoted_build_analysis == "Pass"
(either accepted == true OR AI recommendation == ACCEPT)
AND
all 3 async tasks complete successfullystdout contains: "task [Create test report] status is changed to [Pass]" OR exiting report urlstdout contains: "task [Create test report] status is changed to [Fail]"stdout contains: "task [Take ownership] status is changed to [Pass]"stdout contains: "task [Check CVE tracker bugs] status is changed to [Pass]"klistmetadata = oar_get_release_metadata(release)
rhcos_advisory_id = metadata.advisories.rhcoscurl -s -u : --negotiate 'https://errata.devel.redhat.com/api/v1/erratum/{rhcos_advisory_id}/security_alerts'response = json.loads(curl_output)
# Filter blocking alerts from the alerts array
blocking_alerts = [alert for alert in response.alerts.alerts if alert.blocking == true]
IF len(blocking_alerts) > 0:
Report to user with alert details and ask to email secalert@redhat.com
# Continue pipeline - this is not a hard blocker, but requires follow-up
ELSE:
Report to user: "No blocking security alerts found on RHCOS advisory"Task always passes - this is an informational check
Blocking alerts require manual follow-up but don't stop the pipeline{
"alerts": {
"alerts": [
{
"name": "erratum_missing_notes_link",
"text": "...",
"description": "...",
"how_to_resolve": "...",
"blocking": false
}
],
"blocking": false
}
}stdout contains: "task [Push to CDN staging] status is changed to [In Progress]"Success: stdout contains: "task [Push to CDN staging] status is changed to [Pass]"
Failure: stdout contains: "task [Push to CDN staging] status is changed to [Fail]"File: _releases/ocp-test-result-{candidate_build}-amd64.json
Location: GitHub repository on 'record' branchIF 'aggregated' not in file:
Report to user: "Candidate build tests still running, aggregation not started. Check again in 5-10 minutes"
RETURN
IF file.aggregated != true:
Report to user: "Candidate build tests still aggregating, check again in 5-10 minutes"
RETURNIF file.accepted == true:
oar_update_task_status(release, "analyze-candidate-build", "Pass")
Continue to next task
ELSE IF file.accepted == false:
Trigger: /ci:analyze-build-test-results {candidate_build} --arch amd64
Parse AI recommendation
IF recommendation == ACCEPT:
oar_update_task_status(release, "analyze-candidate-build", "Pass")
Continue to next task
ELSE IF recommendation == REJECT:
Report blocking issues to user
oar_update_task_status(release, "analyze-candidate-build", "Fail")
STOP pipeline - manual intervention requiredFile: _releases/ocp-test-result-{release}-amd64.json
IF file exists: Proceed to Step 2
ELSE: Report "Test result file not yet created, check again in 5-10 minutes", RETURNIF 'aggregated' not in file:
Report "Tests still running, aggregation not started. Check again in 5-10 minutes", RETURN
IF file.aggregated != true:
Report "Tests still aggregating, check again in 5-10 minutes", RETURNIF file.accepted == true:
oar_update_task_status(release, "analyze-promoted-build", "Pass")
Proceed to trigger async tasks (gate check passed)
ELSE IF file.accepted == false:
Trigger: /ci:analyze-build-test-results {release}
IF recommendation == ACCEPT:
oar_update_task_status(release, "analyze-promoted-build", "Pass")
Proceed to trigger async tasks
ELSE IF recommendation == REJECT:
oar_update_task_status(release, "analyze-promoted-build", "Fail")
Notify owner via Slack
BLOCK at gate check - manual intervention requiredExecute: oar_image_consistency_check(release)
# Success - Prow job triggered:
stdout contains: "task [Image consistency check] status is changed to [In Progress]"
AND capture Prow job ID from stdout (pattern: "Triggered image consistency check Prow job: {job_id}")
# Blocked - Stage-release pipeline not succeeded:
IF stage-release pipeline error detected:
Report to user with shipment MR URL and action steps
RETURN (do not mark as failed - this is a prerequisite wait state)Execute: oar_image_consistency_check(release, job_id={captured_job_id})Success: stdout contains: "task [Image consistency check] status is changed to [Pass]"
Failure: stdout contains: "task [Image consistency check] status is changed to [Fail]"Execute: oar_stage_testing(release)
# Success - Jenkins job triggered:
stdout contains: "task [Stage testing] status is changed to [In Progress]"
AND capture Jenkins build number from stdout
# Blocked - Stage-release pipeline not succeeded:
IF stage-release pipeline error detected:
Report to user with shipment MR URL and action steps
RETURN (do not mark as failed - this is a prerequisite wait state)Execute: oar_stage_testing(release, build_number={captured_build_number})Success: stdout contains: "task [Stage testing] status is changed to [Pass]"
Failure: stdout contains: "task [Stage testing] status is changed to [Fail]"stdout contains: "task [Image signature check] status is changed to [Pass]"metadata = oar_get_release_metadata(release)
release_date = metadata.release_date # Format: "2025-Nov-04"
# Calculate optimal execution date: release_date - 1 day
# If today < optimal_date: Wait to execute
# If today >= optimal_date: Safe to executeExecute: oar_change_advisory_status(release)
Action: Approves shipment MR + launches detached background process
Return: "SCHEDULED" - parent process terminates immediately
Google Sheets: Task status updated to "In Progress"state = oar_get_release_status(release="{release}")IF previous_task.status == "Pass":
Execute next_task
ELSE IF previous_task.status == "In Progress":
Report to user: "Task still in progress, check again later"
ELSE IF previous_task.status == "Fail":
Report to user: "Pipeline blocked - manual intervention required"
STOP pipelineresult_file = fetch_from_github(f"_releases/ocp-test-result-{build}-amd64.json")
IF file does not exist:
Report to user: "Test result file not yet created, check again later"
RETURN
IF 'aggregated' not in result_file:
Report to user: "Tests still running, aggregation not started. Check again in 5-10 minutes"
RETURN
IF result_file.aggregated != true:
Report to user: "Tests still aggregating, check again in 5-10 minutes"
RETURN
IF result_file.accepted == true:
Mark task "Pass"
ELSE:
Trigger /ci:analyze-build-test-results {build}
IF AI_recommendation == ACCEPT:
Mark task "Pass" (with waiver)
ELSE:
Mark task "Fail"
STOP pipelineWHEN trigger phase:
Execute command
Capture job ID from stdout (Prow job ID or Jenkins build number)
Report to user: "Task triggered, check status in X minutes"
WHEN user re-invokes /release:drive:
Execute command with job ID (Prow job ID or Jenkins build number)
Parse stdout for status
IF status == "In Progress":
Report to user: "Task still running, check again in X minutes"
ELSE IF status == "Pass":
Mark task complete, proceed to next task
ELSE IF status == "Fail":
Mark overall status "Red", notify owner, STOP pipelineWHEN build promotion detected (phase == "Accepted"):
Trigger 2 tasks immediately:
- image-consistency-check
- stage-testing
IF either task fails due to stage-release pipeline not ready:
Report blocking message with shipment MR and action steps
RETURN (tasks not triggered yet, will retry on next invocation)
# Both tasks triggered successfully
Report to user: "Build promoted! 2 async tasks triggered. Check status in 10-15 minutes"
THEN proceed to check test results in parallel
When user re-invokes /release:drive:
# Retry triggering any tasks blocked by stage-release
IF image-consistency-check or stage-testing not triggered yet:
Retry trigger
IF still blocked: Report same blocking message, RETURN
# Check BOTH conditions for final approval
1. Test analysis status - check file, aggregation, acceptance
2. Async task status - check all 3 tasks
IF gate check PASSED AND all 3 async tasks == "Pass":
Proceed to image-signed-check
ELSE:
Report current status and wait1. Mark task as "Fail"
2. Update overall status to "Red"
3. Notify owner via Slack with error details
4. Report to user: "Pipeline stopped - manual intervention required"