verify-samples-tool

verify-samples Tool

The verify-samples project (dotnet/eng/verify-samples/) is an automated tool that runs sample projects and verifies their output using deterministic checks and AI-powered verification.

Running verify-samples

Important: By default, samples must be pre-built before running verify-samples. Build the solution first, or pass --build to build samples during the run:

cd dotnet
dotnet build agent-framework-dotnet.slnx -f net10.0

Then run verify-samples:

Required Environment Variables

The tool itself needs:

AZURE_OPENAI_ENDPOINT — for the AI verification agent
AZURE_OPENAI_DEPLOYMENT_NAME (optional, defaults to gpt-5-mini)

Individual samples require their own env vars (e.g., AZURE_AI_PROJECT_ENDPOINT). The tool automatically checks and skips samples with missing env vars.

Output Files

--log results.log — detailed per-sample log with stdout/stderr, AI reasoning, and a summary
--csv results.csv — tabular summary with Sample, ProjectPath, Status, FailedChecks, and Failures columns
--md results.md — Markdown summary with results table and collapsible failure details (suitable for GitHub PR comments)

Sample Categories

Definitions are in the dotnet/eng/verify-samples/ directory:

Category	Config File	Registered Key
01-get-started	`GetStartedSamples.cs`	`01-get-started`
02-agents	`AgentsSamples.cs`	`02-agents`
03-workflows	`WorkflowSamples.cs`	`03-workflows`

Categories are registered in VerifyOptions.cs in the s_sampleSets dictionary.

SampleDefinition Properties

Each sample is defined as a SampleDefinition in the appropriate config file. Key properties:

How to Add a New Sample Definition

Check the sample's Program.cs to understand:
- What environment variables it reads (look for GetEnvironmentVariable)
- Whether it needs stdin input (look for Console.ReadLine, Application.GetInput)
- Whether it has an external loop (look for EXIT patterns in YAML workflows)
- What output it produces (section headers, markers, expected behavior)
- Whether it exits on its own or runs as a server
Choose the right verification strategy:
- Deterministic (IsDeterministic = true): Use MustContain for samples with fixed output strings. No AI verification.
- AI-verified (default): Use ExpectedOutputDescription with semantic descriptions. Write expectations that are flexible enough for non-deterministic LLM output.
- Both: Use MustContain for fixed markers AND ExpectedOutputDescription for LLM-generated content.
Set SkipReason only for structural issues:
- Web servers that don't exit
- Multi-process client/server architectures
- Samples requiring external infrastructure (MCP servers you can't reach, Docker, etc.)
- Do NOT skip for missing env vars — the tool checks those dynamically.
For interactive samples, provide Inputs:
- Samples using Application.GetInput(args) need one initial input
- Samples with Console.ReadLine() approval loops need "Y" inputs
- YAML workflows with externalLoop need "EXIT" as the last input
- Set InputDelayMs to 3000-8000ms for samples with LLM calls between inputs
Add the definition to the appropriate config file (e.g., AgentsSamples.cs) in the All list.
Register new categories (if needed) in VerifyOptions.cs s_sampleSets dictionary.

Writing Good ExpectedOutputDescription

Write descriptions that are semantically flexible — LLM output varies between runs
Each array entry should describe one independent aspect to verify
Always include "The output should not contain error messages or stack traces." as the last entry
Avoid exact wording expectations — use "should mention", "should contain information about", "should show"
Bad: "The output should say 'The weather in Amsterdam is cloudy with a high of 15°C'"
Good: "The output should contain weather information about Amsterdam mentioning cloudy weather with a high of 15°C."

Example: Simple LLM Sample

Example: Deterministic Sample

Example: Interactive Sample with Approval Loop

Example: Declarative Workflow with External Loop

Example: Skipped Sample