desktop-test-strategy-reference

Overview

Desktop GUI testing is fragmented along two axes: operating system (Windows / macOS / Linux) and UI toolkit (native, Electron, Qt, GTK, Cocoa, WPF, WinUI, Win32). Unlike the web - where a single DOM-driving tool (Playwright / Selenium) covers every browser - each desktop OS exposes its own accessibility tree, and that tree is the substrate every reliable desktop test driver uses.

Per Microsoft Learn - UI Automation Win32 overview:

"Microsoft UI Automation is an accessibility framework that enables Windows applications to provide and consume programmatic information about user interfaces (UIs). … UI Automation also allows automated test scripts to interact with the UI."

Per Apple - Testing with Xcode, UI Testing chapter:

"UI Testing rests upon two core technologies: the XCTest framework and Accessibility."

Per the at-spi2-core project:

"Base DBus XML interfaces for accessibility, the accessibility registry daemon, and atspi library."

This skill is a pure reference consumed by the per-tool skills below and by anyone choosing a desktop test stack.

How to use this reference

1. Pick OS + toolkit from the per-OS / per-toolkit decision matrix.
2. Choose stable locators for that backend from the locator-strategy table.
3. Apply the matching async-wait primitive for the OS (XCTest waiters / FlaUI Retry / AT-SPI polling) - see references/async-waits-and-concurrency.md.
4. Pre-handle the OS elevation hazard (UAC / TCC / AT-SPI) before it blocks the run - see references/platform-hazards-and-dpi.md.

When to use

• Starting a new desktop test program - which OS + toolkit needs which driver?
• Auditing an existing desktop test suite - is the right backend in use, or is the suite working around a wrong-tool fit?
• Cross-platform desktop app (Electron, Qt, .NET MAUI) - which test tools cover which platforms?
• Onboarding a tester from web E2E - what's the mental-model difference between DOM-driving and accessibility-tree-driving?

The three OS backends

Every reliable desktop driver routes through one of three OS-native accessibility trees. Direct pixel / coordinate scripting (SikuliX, AutoIt screen scraping) is in the anti-pattern column below.

Windows - Microsoft UI Automation (UIA)

UI Automation (UIA) is Microsoft's accessibility framework for Windows desktop applications, replacing the older Microsoft Active Accessibility (MSAA) interface (msuia). UIA exposes an AutomationElement tree with control patterns (InvokePattern, ValuePattern, SelectionPattern, etc.) that describe what each element does, not just how it looks.

Per msuia:

"UI Automation is designed for experienced C/C++ developers. In general, developers need a moderate level of understanding about Component Object Model (COM) objects and interfaces, Unicode, and Windows API programming."

Higher-level language bindings (C#, PowerShell, Python via pywinauto) sit on top. For test-automation purposes, the WinAppDriver and Appium-Windows projects expose UIA as a W3C WebDriver endpoint - which is what most QA toolchains actually drive.

macOS - Apple Accessibility + XCTest

macOS GUI testing uses the same XCTest framework that ships with Xcode for unit and integration tests. Per appleuit:

"UI testing rests upon two core technologies: the XCTest framework and Accessibility."

The three foundational classes (appleuit) are:

Class	Role
XCUIApplication	The application under test
XCUIElement	A single UI element in the accessibility tree
XCUIElementQuery	A query that resolves to zero or more XCUIElement

The general UI-test pattern per appleuit:

"Use an XCUIElementQuery to find an XCUIElement. Synthesize an event and send it to the XCUIElement. Use an assertion to compare the state of the XCUIElement against an expected reference state."

Xcode's Accessibility Inspector (bundled with Xcode → Xcode → Open Developer Tool → Accessibility Inspector) is the read-side companion: it walks the same tree the tests see, lets the engineer verify accessibility identifiers exist before writing the query, and flags WCAG-style accessibility gaps in the same pass.

Linux - AT-SPI

The Linux desktop accessibility stack is AT-SPI (Assistive Technology Service Provider Interface). Per atspi2core, the canonical implementation at-spi2-core provides "Base DBus XML interfaces for accessibility, the accessibility registry daemon, and atspi library." The registry daemon (at-spi2-registryd) exposes a system-wide D-Bus service that assistive tools - and test clients - connect to.

GTK applications expose AT-SPI automatically via the atk bridge (at-spi2-atk); Qt applications expose AT-SPI via the QAccessible infrastructure (see qt-test-framework SKILL for the Qt-specific path); Electron and Chromium apps on Linux expose their own AT-SPI surface.

Python clients (dogtail, pyatspi) and the GNOME inspector tool Accerciser walk the registry to drive tests.

The toolkit overlays

Two cross-toolkit families need explicit treatment because they ride on top of (or around) the OS backends:

Electron

Electron apps are Chromium + Node.js wrapped in a native window. They expose two parallel surfaces:

1. Renderer process - a normal Chromium DOM. Page-level Playwright / Selenium / Puppeteer drive this just like a browser tab.
2. Main process - the Node.js process that owns native windows, menus, IPC, file dialogs, lifecycle.

A packaged Electron app cannot be driven by browser-only Playwright - the entry point is the packaged binary, not a URL. The Playwright _electron API (electron-playwright SKILL in this plugin) launches the packaged binary, exposes the main process as a typed ElectronApplication handle, and returns Chromium-window pages for the renderer. Legacy alternative (deprecated 2022): Spectron - migration path in electron-playwright's references/spectron-migration.md.

Qt

Qt has its own first-party test framework - QtTest - that lives in the application's process and emits events directly into the QObject event queue. It does not go through the OS accessibility tree by default. For UI tests of Qt apps that need to be driven from outside the process, the OS-native drivers (WinAppDriver on Windows, XCTest on macOS, AT-SPI on Linux via Qt's QAccessible) are the path. See qt-test-framework SKILL.

Per-OS / per-toolkit decision matrix

App type	Windows driver	macOS driver	Linux driver
Win32 / WinForms / WPF	WinAppDriver, Appium-Windows	n/a	n/a
WinUI 3 / UWP	WinAppDriver, Appium-Windows	n/a	n/a
Cocoa / SwiftUI / AppKit	n/a	XCTest UI (XCUIApplication)	n/a
GTK	n/a	n/a	AT-SPI (dogtail / pyatspi)
Qt - in-process	QtTest (QTEST_MAIN)	QtTest	QtTest
Qt - out-of-process	WinAppDriver (via UIA)	XCTest UI	AT-SPI (via QAccessible)
Electron - renderer only	Playwright _electron page handle	Playwright _electron page handle	Playwright _electron page handle
Electron - main + IPC	Playwright _electron	Playwright _electron	Playwright _electron

Cross-references to per-tool SKILLs:

• winappdriver - the Windows UIA WebDriver service, covering both the direct Selenium-client path and the Appium 2.x invocation path
• xcuitest-suite (qa-mobile plugin) - XCTest UI; its references/macos.md covers the macOS desktop delta
• AT-SPI on Linux - driven via dogtail / pyatspi clients (no dedicated skill; the backend and bootstrap are covered in this reference)
• qt-test-framework - QtTest in-process
• electron-playwright - Playwright _electron API; its references/spectron-migration.md covers legacy Spectron migrations

Choosing a driver

To pick one driver for a concrete app, first infer the app type from the project file - never from a bare directory name or README:

Signal in project file	Inferred app type
*.csproj containing <UseWPF>true</UseWPF> or <TargetFramework>...-windows</TargetFramework> + PresentationFramework reference	wpf
*.csproj containing <UseWindowsForms>true</UseWindowsForms>	winforms
*.csproj containing <TargetPlatformIdentifier>Windows</TargetPlatformIdentifier> + UWP namespaces	uwp
*.csproj targeting net48 / older with no UseWPF/UseWindowsForms	win32 (managed Win32)
package.json with "electron" in dependencies or devDependencies	electron
*.pro file OR CMakeLists.txt with find_package(Qt6) / find_package(Qt5)	qt
*.xcodeproj / Package.swift targeting macOS	macos-native
*.in / configure.ac referencing GTK / GLib	linux-gtk
Avalonia or .NET MAUI references	cross-platform-unknown (host OS dictates the driver)

Then apply the decision table - one primary driver per app; a secondary fallback only where two are co-equal defensible (UWP, Win32, Qt):

App type	Recommended driver	Why	Read next
wpf	FlaUI (UIA3)	.NET-native, idiomatic C# API, no HTTP hop; UIA3 is preferred for WPF per the FlaUI README	flaui-tests
winforms	FlaUI (UIA2)	Managed System.Windows.Automation has better legacy WinForms compatibility per the FlaUI README	flaui-tests
uwp / Store App	FlaUI (UIA3) OR WinAppDriver	FlaUI when the test stack is .NET; WinAppDriver when cross-language clients are required	flaui-tests or winappdriver
win32	FlaUI (UIA3) OR WinAppDriver	Either works; FlaUI for .NET test stack, WinAppDriver for Java / Python / Ruby clients	flaui-tests or winappdriver
electron	electron-playwright	Drives main + renderer from one Playwright suite via _electron.launch	electron-playwright
qt (Windows / Linux)	qt-test-framework in-process; WinAppDriver (Windows) or AT-SPI (Linux) out-of-process	First-party in-process tests use QtTest; out-of-process driving needs a UIA / AT-SPI client	qt-test-framework
macos-native	XCUITest	Apple's first-party UI test harness, accessibility-tree backed	xcuitest-suite (qa-mobile) + its references/macos.md
linux-gtk / linux-qt	AT-SPI (dogtail / Accerciser)	The canonical Linux accessibility-tree backend	the AT-SPI sections of this reference
cross-platform-unknown (Avalonia / MAUI)	Per-platform per row above	Avalonia / MAUI render via the host OS's UI toolkit, so the OS dictates the driver	per-platform skill

Two constraints that flip or qualify a recommendation:

• Elevation (Windows). If the SUT requires admin privileges, the driver session itself must run elevated - UAC's secure desktop is outside the accessibility tree, so a non-elevated WinAppDriver / FlaUI session sees the entire elevated UI as empty (WinAppDriver #306, #2033). Signals: <requestedExecutionLevel level="requireAdministrator" /> in app.manifest; README "Run as administrator."
• Cross-OS Electron. electron-playwright is the same driver across Windows / macOS / Linux (_electron.launch() + electronApp.evaluate() per the Playwright ElectronApplication API); only the CI bootstrap differs per OS (see references/platform-hazards-and-dpi.md).

Never mix UIA2 and UIA3 in the same FlaUI test process (unsupported per the FlaUI README), and never emit one recommendation for "cross-platform desktop" without a per-OS breakout - each OS has a different accessibility backend.

Locator strategy across backends

The portable lesson from web E2E (per playwright-testing in the qa-web-e2e plugin) - accessibility-first locators - carries directly over: every desktop backend resolves elements through the same accessibility tree assistive technology uses.

Backend	Stable locator	Brittle locator
UIA (WinAppDriver, Appium-Windows)	AutomationId (preferred - locale-independent), ControlType + property combo (disambiguation when no AutomationId)	Name (last resort - Name is the localised label, fails across language packs); absolute screen coordinates; XPath (officially supported by WinAppDriver but the community-canonical guidance treats it as fragile and slow)
XCTest (macOS)	accessibilityIdentifier, role-based queries (buttons["Submit"])	Label-based queries when the label is localised; image matching; hard-coded coordinates
AT-SPI (Linux)	Object name field set explicitly on the widget (GTK widget.set_property('name', ...), Qt QObject::setObjectName)	Visible label or role-only queries (collapse under localisation and theme changes); pixel coordinates
QtTest in-process	Qt object name (setObjectName("…"))	child-index navigation through QWidget tree
Electron renderer (Playwright)	getByRole / getByLabel on the renderer DOM	CSS class chains; Electron debug-build internal IDs

The single biggest portability win across desktop platforms is: every app under test sets a stable accessibility identifier on every interactive widget. Without it, the tree's only resolution key is the human-visible label, which collapses under localisation.

Localisation rule of thumb: if the test passes on a US-English build but the same app ships in 20 locales, every Name / label-based locator is a latent failure. AccessibilityId / AutomationId / object name are locale-independent by design - make the developer set them at the source.

Coverage scope per layer

Layer	What it can cover	What it cannot cover
Unit test (XCTest / QtTest / .NET MSTest)	Pure logic, view-model bindings, QObject signals	OS-level dialog interactions, IME input, multi-window focus
In-process UI test (XCUIApplication-in-app, QtTest with QTest::mouseClick)	App-internal widget events	Cross-app drag-drop, system shortcuts, OS file pickers
Out-of-process UI driver (WinAppDriver / AT-SPI / Playwright _electron)	Full end-to-end including file dialogs, menus, system tray	Pre-login OS UI, GPU-accelerated rendered surfaces (canvas, WebGL) that don't expose the accessibility tree
Visual snapshot	Pixel-level regressions, font rendering	Logic, state, async behaviour

Cross-references for the upstream + downstream slots:

• Unit testing - see the per-language qa-unit-tests-* plugins.
• Visual regression - see qa-visual-regression for desktop screenshot comparison patterns.
• CI integration - desktop runners cost more than web runners; see winappdriver and qa-mobile's xcuitest-suite (references/macos.md) for GitHub-hosted vs self-hosted considerations.

Operating the tests - deep references

Once the stack is chosen, the operational depth lives in two companion references:

• Asynchronous waits + concurrency - per-OS retry primitives (XCTest waiters, FlaUI Retry, AT-SPI polling) and the per-OS parallel-test policy: references/async-waits-and-concurrency.md.
• Platform hazards + high-DPI - foreground-lock, UAC secure desktop, macOS TCC prompts, AT-SPI session enablement, and the per-monitor DPI test matrix: references/platform-hazards-and-dpi.md.

Review hazards - desktop test-code checklist

When reviewing an existing desktop UI test suite or a PR that touches desktop test files, walk these four axes. For generic, framework-agnostic test-file conventions (AAA, naming, magic numbers, assertion quality), use the test-code-critic agent in qa-test-review - this checklist covers only the desktop-specific hazards.

Axis	What to flag
Screen-object encapsulation	Raw locator calls (FindByAccessibilityId, FindFirstChild, XCUIApplication().descendants, findElementByAccessibilityId) inside a test method instead of behind a Screen Object class (Pattern 7 in qa-test-review's object-model-patterns)
Locator stability	FindByName / By.Name where an AutomationId is available (per Microsoft Learn - Use the AutomationID Property, AutomationIdProperty "uniquely identifies a UI Automation element from its siblings"); XPath deeper than 2 levels; integer-index child navigation (GetChildren()[2], childAtIndex(1)); locators composed from runtime string interpolation
Explicit waits over sleep	Any Thread.Sleep / Task.Delay / time.sleep in a test or screen-object body - replace with the per-OS retry primitive (FlaUI Retry.WhileNull / WhileFalse per the FlaUI Retry wiki, XCTest waitForExistence(timeout:), AT-SPI polling helper) - see references/async-waits-and-concurrency.md
Platform hazards	UIA event subscription from the test's main thread (STA threading); keyboard input without a preceding app.Activate() / window.Focus() (foreground-lock); elevation-requiring launches without an elevated-session note (UAC / TCC); AT-SPI tests that skip the gsettings + Xvfb / dbus-launch bootstrap - see references/platform-hazards-and-dpi.md

Hard-reject signal: a new test file where every element lookup is by Name or index (zero AutomationId / accessibilityIdentifier usage) - locale-dependent and structurally fragile; assign stable identifiers before authoring tests.

Anti-patterns

Anti-pattern	Why it fails	Fix
Image-recognition / pixel-matching as the primary driver (SikuliX-only, AutoIt screen scraping)	Brittle to font / theme / DPI / OS-chrome changes; opaque to localisation	Use the OS accessibility tree; reserve image matching for canvas-rendered content only
Hard-coded screen coordinates	Multi-monitor / DPI / window-state dependent	Resolve via accessibility identifier; click resolves to the element's hit-test centre
Win32 message sending (SendMessage / PostMessage) for modern apps	Doesn't reach UIA-only WinUI / WPF controls; bypasses event ordering	Route through UIA (WinAppDriver) - the documented control-pattern API
Using only XCTest on a packaged macOS app shipped to the App Store via TestFlight	Loses cross-platform consistency with Windows + Linux	Pair with a higher-level cross-platform layer (Appium Mac2 driver) only when the same test source must run on multiple OSes; otherwise XCTest is the right tool on macOS
Spectron for new Electron projects (archived 2022)	No maintenance, no support for modern Electron	Use Playwright _electron (electron-playwright SKILL)
Driving Qt apps with Win32-only tools when accessibility isn't wired in	Qt's QAccessible interface must be enabled; without it UIA sees no children	Verify with Accessibility Insights for Windows before writing tests; enable QAccessible in the Qt build
Running desktop UI tests on GitHub-hosted Linux runners with no virtual display	at-spi2-registryd requires a session bus	Self-hosted runners with a real (or Xvfb / dbus-launch synthetic) session, or use Windows / macOS hosted runners for those OS-specific suites

Limitations

• Apple developer documentation is behind a Cloudflare-style challenge on most "https://developer.apple.com/documentation/…" pages - automated fetches return the SPA shell only. This reference cites Apple's stable-archive testing_with_xcode chapter (appleuit) for prose; per-API surface (XCUIApplication, XCUIElement) is treated as a stable identifier in qa-mobile's xcuitest-suite SKILL.
• No single cross-platform driver. Tools that claim to be one (Appium across Windows + macOS + Linux desktop) are thin facades over the per-OS backends - coverage gaps follow the underlying backend's gaps.
• GPU-accelerated content (canvas, WebGL, custom-painted Qt widgets, Win32 DirectComposition surfaces) does not expose itself to the accessibility tree by default; any driver routed through the tree sees them as opaque rectangles.
• Cross-process focus + window-Z order is OS-policy-dependent (Windows foreground-lock, macOS Stage Manager); tests that assume a window is foreground often need explicit app.activate() calls.
• Localisation collapses label-based locators. The per-tool SKILLs all push toward AutomationId / accessibilityIdentifier / object name for this reason.

References

• Microsoft UI Automation overview - msuia.
• Apple Testing with Xcode - UI Testing chapter (appleuit).
• at-spi2-core - atspi2core.
• dogtail (Python AT-SPI client) - dogtail.
• Operational depth (with their own citations): async waits + concurrency in references/async-waits-and-concurrency.md; foreground / elevation hazards + high-DPI matrix in references/platform-hazards-and-dpi.md.
• Per-tool implementation SKILLs: winappdriver (direct + Appium invocation), qt-test-framework, electron-playwright (incl. Spectron migration), and qa-mobile's xcuitest-suite (references/macos.md for the macOS desktop delta).
• Web-side neighbour: playwright-testing - DOM-driving for browser apps (the contrast surface for desktop drivers).
• Architecture-pattern sibling: object-model-patterns (in the qa-test-review plugin) - Pattern 7 (Screen Object) is the desktop analog of POM and cites this skill for the backend substrate.