winapp-ui-automation

When to use

• Inspecting a running Windows app's UI from the command line
• AI agents interacting with Windows applications (clicking buttons, reading text, taking screenshots)
• Verifying UI state during development or testing
• Automating UI workflows without Playwright or Selenium
• Debugging WinUI 3, WPF, WinForms, Win32, or Electron app UIs

Prerequisites

• For UIA mode (any app): No setup needed — works with any running Windows app
• For input-injecting verbs (click, hover, drag, touch, pen, scroll --wheel, send-keys --via send-input): an unlocked, interactive desktop with the target window foregroundable. On a locked/secure desktop they fail fast with no_interactive_desktop. The UIA-pattern verbs (inspect, search, get-*, wait-for, set-value, invoke, scroll --direction/--to) are headless/locked-session friendly — prefer them in CI.
• screenshot is not in that group: it always takes an exclusive turn, so it queues behind other UI workflows, and capture can need a usable interactive desktop — the engine restores the target if it is minimized, and falls back to foregrounding it when frame capture is unavailable or --capture-screen is used.
• --capture-screen needs exactly one window. -w <hwnd> gives it one: that window's screen region, including anything visibly on top of it. If -a matches several top-level or owned windows the command fails with invalid_arguments before capturing; run winapp ui list-windows -a <app> and retry with -w <hwnd>.
• If other UI workflows may run at the same time, set one workflow id per logical workflow (see below). Nothing breaks without it, but your commands will not be recognized as belonging together.

Coordinating with other UI workflows

Windows has one foreground window, one keyboard focus, one cursor, and one input stream. winapp ui therefore makes desktop-driving commands take cooperative turns so concurrent workflows cannot steal each other's focus or dismiss each other's menus. That is always on. Read-only commands never wait.

Keeping the desktop across commands is opt-in — without an id, each command is a one-shot that releases the desktop the moment it finishes:

# Set once per logical UI workflow — same value for cooperating calls, different values for
# independent workflows (even from the same agent).
$env:WINAPP_UI_WORKFLOW_ID = [guid]::NewGuid().ToString()

Rules that matter when driving this from an agent:

• Each tool call usually gets a fresh shell, and there is no process-ancestry fallback, so commands are grouped ONLY by the id you inject. Without it every call is its own workflow.
• A workflow with an id keeps its turn for four seconds after its last command. That covers back-to-back commands in one script; it deliberately expires while you are reasoning. Treat it as a fallback for when you cannot say you are done — not as the way to finish.
• Run winapp ui yield when you finish a known sequence. It hands the desktop over immediately instead of making a waiting workflow sit out a four-second grace nobody needs. Yielding twice, or after the grace lapsed, is a harmless success.
• After a reasoning gap, replay your setup. Another workflow may have used the desktop, so reopen the menu / re-navigate, re-resolve the element, then act. Do not assume transient UI survived.
• Prefer one tight script over many round trips for a known sequence: winapp ui invoke View -w $hwnd; winapp ui search "Status bar" -w $hwnd; winapp ui click "Status bar" -w $hwnd; winapp ui yield.
• record shares the turn with its own workflow, so same-workflow clicks and typing are captured while it runs — but only if both commands carry the same id. A record with no id blocks everyone else for its whole duration.
• Ordering is owner affinity, then FIFO. An active workflow may keep issuing commands ahead of others already waiting; once it yields or its grace expires, waiters are served in arrival order.
• Waiting is indefinite and cancellable. A status line appears after one second; Ctrl+C exits 130 with error code cancelled and the command never ran.
• There is no hard cap — a long script, unbounded recording, or failure loop can block other mutating workflows until it finishes or is stopped.

Commands that never wait: status, list-windows, inspect, search, get-*, wait-for. Commands that wait for the turn but never take the desktop (headless/locked-session friendly): set-value, scroll-into-view, scroll --direction/--to, record. They mutate the app, so they queue behind another workflow. Inside your own workflow they overlap with other shared work — that is how record captures the set-value calls it is recording — but they still wait behind an earlier DesktopExclusive command of your own workflow, so a click followed by a set-value runs in the order you wrote it. Commands that take the desktop exclusively: invoke, click, drag, hover, scroll --wheel, touch, pen, focus, send-keys, screenshot.

# Finish a workflow deliberately rather than leaving the desktop reserved for four more seconds.

Common patterns

Read within a specific container

Use --root, --type, and --class-name together or separately on search, get-property, get-value, and wait-for. The root must be unique; only its descendants match. wait-for re-resolves it every poll, including when it is initially absent. Type names and literal whole ClassName values ignore case. The only type aliases are TextBox → Edit and TextBlock → Text. See Scoped and typed queries for the full type vocabulary, boundaries, and error behavior.

Discover and interact

Find visible text and click it

Navigate multi-page apps

Choose an exact action in tests

Use --action to avoid automatic pattern and ancestor fallback. Omit it for the existing automatic behavior. See the action reference for supported actions, idempotent toggles, and failure recovery, and the JSON envelope for action results.

Disambiguate duplicate elements

Key concepts

• Selector brackets: inspect and search output shows selectors in [brackets] — use the bracketed value with other ui commands. Selectors are either AutomationId (stable, developer-set) or generated slug (e.g., btn-name-hash).
• AutomationId selectors: When an element has a unique AutomationId, it becomes the selector directly (e.g., [MinimizeButton]). These survive layout changes and localization — preferred for stable targeting.
• Slug selectors: When no unique AutomationId exists, a generated slug is used (e.g., [btn-close-a2b3]). Format: prefix-name-hash. May go stale after UI changes.
• Plain text search: search and invoke accept plain text — search Minimize finds elements with "Minimize" in their Name or AutomationId (substring, case-insensitive). No special syntax needed.
• --interactive flag: Filters to invokable elements only with auto-depth 8 — the fastest way to see what you can click
• Invokable ancestor surfacing: When a search result isn't invokable, the nearest invokable parent is shown with its selector
• ; chaining: Chain commands with ; to run multiple operations in one call, reducing agent round-trips
• -a vs -w: Use -a to find apps by name/title/PID. Use -w <HWND> for stable window targeting
• Element markers: [on]/[off] for toggles, [collapsed]/[expanded], [scroll:v]/[scroll:h]/[scroll:vh] for scrollable containers, [offscreen], [disabled], value="..." for editable elements

Usage

Connect and discover

Inspect element tree

Find elements

Screenshot

Record video (H.264 MP4)

Record a window or element region to MP4. Prefer a positive --duration-sec N for agents and scripts; otherwise recording waits for a stop signal. npm helpers require durationSec (integer 1–86400). Their abort signal is forceful cancellation, not graceful stop.

• Default --duration-sec 0 records until Ctrl+C, a newline, or EOF on redirected stdin.
• --frames writes <output-name>.frames with a manifest, NDJSON index, and changed JPEGs. It supports 1-30 fps and --max-edge 64-4096 (default 1280), with a 1 GiB cap. Use elapsedMs to bound transitions.
• Existing recording outputs are rejected by default. Use a fresh path, or explicitly request --overwrite to replace them after the new take finishes. Previous frame directories are archived, not deleted. On partial failure, keep the reported evidence and follow recoveryHint.
• --capture-screen captures from the screen DC so overlays and popups are included; the window is brought to the foreground first. When WGC is unavailable and --capture-screen is not passed, the CLI returns an error — re-run with --capture-screen to consent to screen-DC capture. Because the screen DC captures whatever is genuinely in front, the target's foreground is verified immediately before capture; if activation was refused the command fails with foreground_not_target and writes nothing rather than returning an image of the wrong window.
• Providing a selector that doesn't match any element fails immediately with element_not_found (rather than silently recording the whole window).
• --json writes the final result to stdout and one JSON event per line to stderr.

Hover (for tooltips, flyouts, hover states)

--dwell-time <ms> sets how long to wait after hovering (default: 800, range: 0–10000).

Send keyboard input

Synthesize keystrokes — the keyboard counterpart to click. Use for arrow/Tab/Enter navigation, shortcuts, and per-keystroke typing (vs set-value's atomic write). Tokens are whitespace-separated: named keys (enter, down, tab, esc, f5), modifier combos (ctrl+shift+t), literal text (hello), and raw virtual keys (vk=0xNN).

• Default post-message is HWND-targeted and works across integrity levels, but can't fire WH_KEYBOARD_LL global hotkeys. It automatically retargets to the focused child control of the target window, so classic Win32/WinForms child-window controls (e.g. an edit box) receive the input. WinUI 3 / UWP / XAML controls are windowless and ignore posted WM_CHAR/WM_KEYDOWN — post-message can't deliver keys or text to them (the command emits a warning and still exits 0, since PostMessage can't confirm delivery). Use --via send-input for WinUI 3 / UWP / XAML apps.
• A token that collides with a key/modifier name (e.g. enter, down, ctrl+a) is pressed as that key. Prefix it with text= to type it as literal text instead — text=enter types the word "enter"; chain text= tokens to type a literal phrase like text=down text=down text=enter. Backslash escapes inside a text= value type whitespace the tokenizer would otherwise collapse: \s→space, \t→tab, \n→newline, \r→CR, \\→backslash (e.g. text=a\s\sb → "a b"). When the whole argument is literal text, pass --verbatim instead of escaping each token: it types the entire keys argument as-is (no key/combo/vk=/text= parsing) and preserves exact whitespace — send-keys "down down enter" --verbatim types the words. (--verbatim does not decode backslash escapes; use a text= token for control characters.)
• send-input is fully real input but goes to the foreground window and is UIPI-blocked when injecting from elevated → AppContainer/AppX. It rejects system-reserved combos (win+l, alt+f4, ctrl+shift+esc, ctrl+alt+del, alt+tab, …) because those act on the OS/shell, not just the target — pass --allow-system-keys to opt in (e.g. to fire a global hotkey such as PowerToys' win+shift+v or win+r), or use --via post-message (window-scoped) to send one straight to the window. win+l and ctrl+alt+del stay blocked even with --allow-system-keys — win+l locks the workstation via LockWorkStation() (unrecoverable from automation), and ctrl+alt+del is a Secure Attention Sequence (SAS) that Windows drops from injected input regardless of the flag, so it errors (invalid_arguments) instead of falsely reporting success. On a locked/secure desktop send-input fails fast with no_interactive_desktop.
• Per-keystroke events: named keys/combos fire a real KeyDown on both transports. For literal typed text, --via send-input maps each char to its VK (+Shift) so each character fires a real KeyDown + OS-composed WM_CHAR (TextChanged) — use it when downstream logic keys off KeyDown (e.g. WinUI 3/WPF TextBox); bring the target window to the foreground first. --via post-message posts a WM_CHAR per character to the focused child control (raises TextChanged, lands correct text into classic Win32 controls across integrity levels) but does not fire a per-character KeyDown, and — because WinUI 3 / UWP / XAML controls are windowless — does not reach them at all (named keys and text alike); use --via send-input there.
• Long literal text on --via send-input is auto-throttled: the text is split into small character chunks injected one SendInput at a time with a brief pause between the chunks of that one run, so the target can drain its input queue and every character lands. A single unbroken burst overruns the queue and silently drops characters even though the command reports success. The pacing is scoped to a single long text run — short text, key names, and modifier combos (including sequences like ctrl+a delete) inject with no added delay, and the command emits an informational warning when a payload is large enough to be throttled. Because each paced chunk lands on whatever window is foreground, send-input re-verifies the target still owns the foreground before every continuation chunk and aborts with foreground_not_target if focus leaves the target mid-injection, so a focus change partway through can't spray the rest of the text into another window (a focus-changing chord such as alt+tab is exempt and is never treated as drift). For large bulk text, prefer set-value (atomic, no keystrokes or foreground needed) on controls that support it.

Drag (reorder, resize, sliders, drag-and-drop)

Press the mouse button at one point, move to another, then release with drag <from> <to>, where each endpoint is an element selector (uses its center) or screen x,y coordinates from ui inspect. Uses SendInput with intermediate moves so apps see a realistic WM_MOUSEMOVE stream.

• A selector drags from/to the element's center; x,y are screen coordinates in the same space ui inspect/search report. Element endpoints are re-resolved just before the drag and fail with target_moved if still animating; on a locked/secure desktop the drag fails with no_interactive_desktop.

Touch gestures (tap, swipe, pinch, stretch, long-press)

Inject synthetic touch. The contact anchor is an element selector (its center) or an explicit --at x,y screen coordinate. Prefers the modern synthetic-pointer device and falls back to the legacy touch-injection API.

• Gestures: tap (default), double-tap, long-press, swipe, pinch, stretch. --fingers 1–10 (pinch/stretch always 2). Refuses without a non-zero foregrounded target (no_target/foreground_not_target/no_interactive_desktop); every coordinate is checked against the target window, and a point outside it produces a non-fatal warning (warnings[] in --json) while injection still proceeds — consistent with the mouse verbs. If injected touch is unsupported on the device, the command surfaces the real Win32 error rather than a false success. In a Remote Desktop/VM session delivery isn't guaranteed even on exit 0 — the command appends a delivery-uncertainty warning (warnings[] in --json); verify the effect with a screenshot.

Pen / stylus (taps and ink strokes)

Inject synthetic pen input (Windows 10 1809+). Target an element center, an explicit --at, or a full --path ink stroke, with pressure/tilt/eraser control.

• --pressure 0.0–1.0, --tilt-x/--tilt-y ±90°, --eraser for the eraser end. Same injection safety as touch: requires a non-zero foregrounded target window and checks every ink point (out-of-window → non-fatal warnings[] advisory, injection still proceeds). Pen routing is especially unreliable over Remote Desktop — exit 0 can mean the call succeeded but no pen reached the app; the command appends a delivery-uncertainty warning (warnings[] in --json). Validate pen flows on a local interactive desktop.

Read element state

get-property also accepts FontName, FontSize, ForegroundColor, IsItalic, and StrikethroughStyle. Omit --property to include all six formatting attributes. Treat Mixed, NotSupported, and Unavailable as distinct states, not formatting values. Names are case-sensitive; unknown names fail with invalid_arguments. See the formatting reference for units and state meanings, and references/ui-json-envelope.md for the JSON shape.

Set values

set-value writes programmatically (no keystrokes, no foreground) via a fallback chain: ValuePattern → RangeValuePattern (numeric) → LegacyIAccessible put_accValue for TextPattern-only edit controls.

• The LegacyIAccessible fallback reaches rich-edit/compose controls that expose no ValuePattern, as long as their accessibility implements put_accValue (native Win32 rich-edit and Chromium/Electron/WebView2 compose boxes typically do).
• WinUI 3 RichEditBox and WPF RichTextBox don't support programmatic value-setting — by design they're read-only to UI Automation's value APIs (Text pattern, no settable Value pattern). set-value fails on them with a clear error — use send-keys (needs an unlocked, foregrounded desktop) to type into those instead. Note get-value can still read them via TextPattern.

Scroll containers

Wait for UI state

Tips

• Use --interactive with inspect as your first command — it shows only what you can click
• Chain commands with ; to reduce round-trips (see note below on why not &&)
• Use slugs from output to target specific elements — they're hash-validated and shell-safe
• Use plain text search to find elements: search Minimize, invoke Submit
• When multiple elements match text search, the error shows slugs for each — pick the right one
• Use get-property --property ToggleState to verify checkbox/toggle state after invoke
• scroll auto-finds the nearest scrollable parent
• Use --capture-screen to capture popup overlays, dropdown menus, and flyouts (also brings the window to the foreground)
• Use hover before screenshot --capture-screen to capture tooltips and hover-triggered UI
• Use --focus to foreground the target window before capture without switching to screen-DC capture (default capture path uses Windows.Graphics.Capture and works while occluded)
• Use --hide-disabled and --hide-offscreen to reduce noise

Why ; instead of &&

Use ; (not &&) to chain commands. PowerShell's && operator can freeze when a native CLI writes to stderr or uses ANSI escape sequences — this causes a pipeline deadlock. ; runs each command unconditionally and avoids this issue. This is also better for agent workflows: you usually want the screenshot to run even if the invoke had a non-zero exit (to see what went wrong).

File dialog workaround

File open/save dialogs are standard Windows dialogs with UIA support. Interact with them using existing commands:

Note: The filename input in standard file dialogs typically has AutomationId 1148. Use inspect -w <dialog-hwnd> --interactive to discover the actual slugs.

JSON output envelopes

The --json envelope for ui inspect, ui get-focused, ui search, and ui wait-for was reshaped in v0.3.1. The DPI context and typed get-property element are available in v0.6.3+. Highlights:

• ui inspect --json now nests elements under windows[].elements[] (was a flat elements[]).
• Each inspected window and the ui status --json target reports windowDpi, scale, dpiAwareness, and coordinateSpace: "physical-screen-pixels". This is the target window's DPI context. The selected target fails fast on an unreadable DPI instead of defaulting to 96; a secondary window that disappears mid-walk carries dpiError and omits the four context fields.
• ui get-focused --json always emits an envelope — { "hasFocus": false } or { "hasFocus": true, "element": {...} } (was bare null).
• ui search --json returns { "matchCount", "hasMore", "matches" }; ui wait-for --json returns { "found", "waitedMs", "element"?, "timedOut" }.
• ui get-property --json preserves elementId and its string-valued properties map, and adds a typed, scrubbed element.
• Typed elements use type (not controlType) and numeric x, y, width, and height in physical screen pixels. 0,0,0,0 is UIA's empty/no-displayed-UI rectangle; isOffscreen remains independent.
• Search and wait-for elements may include an invokableAncestor field (element-shaped).
• Per-element id, parentSelector, and windowHandle are removed — use selector as the public handle.

Full schemas with examples: references/ui-json-envelope.md.

Automating in Windows Sandbox

Use --detach before follow-up UI commands, and retain --on sandbox with guest PIDs or window handles. Inject the same WINAPP_UI_WORKFLOW_ID into cooperating guest calls; finish with winapp ui yield --on sandbox after all of them complete.

Use winapp-sandbox for setup consent, connected-client requirements, brief focus changes during setup/reconnect, and capture recovery. App screenshots and recordings, including default filenames and --frames directories, return to the host.

Related skills

• winapp-sandbox for running and automating apps in an isolated Windows Sandbox
• winapp-setup for adding Windows SDK to your project
• winapp-package for packaging apps as MSIX

Troubleshooting

CLI reference

Run winapp <command> --help for current command options, or winapp --cli-schema for the complete machine-readable command schema.`