unreal-automation-system
Overview
This skill wraps the C++ Automation Test Framework (UE 4.x /
5.x) plus the two most commonly composed sub-systems. Per Epic's
Automation Test Framework documentation,
the framework spans five test categories (Unit, Feature, Smoke,
Content Stress, Screenshot Comparison) and supports multiple
authoring styles (traditional, BDD Spec, UI Driver, Functional,
Python / Blueprint).
The full authoring code (macros, Spec, Driver) lives in
references/authoring-macros-and-apis.md;
running, report parsing, and CI wiring live in
references/running-and-reporting.md.
This file is the decision surface.
Composes with:
platform-cert-overview-reference
for the canonical six test categories Unreal's five categories map
to, and for cert-gated requirements automation tests should cover.multiplayer-state-machine-coverage
for replication / dedicated-server state coverage authored as
automation tests.
When to use
- Unit under test is C++ code in an Unreal Engine project
(UE 4.x or UE 5.x) that needs the engine runtime, editor APIs,
or UMG UI surface.
- You want CI-runnable tests via Unreal's command-line
Automation entry point.
- You need BDD-style readable specs (Automation Spec) or scripted
UI input simulation (Automation Driver) on top of plain
assertion tests.
For Python / Blueprint editor tests outside C++, see
Editor Automation in Unreal Engine.
For end-to-end pipeline / build-farm orchestration, use
Gauntlet
on top of the framework this skill covers.
How to use
- Pick the category from the table below (Unit / Feature /
Smoke / Content Stress / Screenshot Comparison) and the matching
EAutomationTestFlags filter + application-context mask.
- Pick the authoring style - plain
RunTest macro, data-driven
complex macro, BDD Spec, or Automation Driver for UI - and lift
the pattern from
references/authoring-macros-and-apis.md.
- Name the test path as dot-separated segments
(
MyGame.Health.Damage_…); the tree keys the Session Frontend.
- Run in-editor via Session Frontend to iterate, then
run headless via
UnrealEditor-Cmd.exe … -ExecCmds="Automation RunTests …" for CI.
- Parse the
-ReportOutputPath JSON - treat any
"state": "Fail" as a failed build; full schema in
references/running-and-reporting.md.
- Split the CI matrix -
SmokeFilter on PRs (sub-second),
ProductFilter | StressFilter plus screenshot comparison nightly.
Test categories and flags
Per the
Automation Test Framework page,
Epic's five categories are:
| Category | Purpose |
|---|
| Unit | "API level verification tests." |
| Feature | "System-level tests that verify such things as PIE, in-game stats, and changing resolution." |
| Smoke | Tests that "complete within 1 second" and run automatically. |
| Content Stress | "More thorough testing of a particular system to avoid crashes." |
| Screenshot Comparison | For comparing renders "between versions or builds". |
Each test declares flags from EAutomationTestFlags that mix:
- Filter -
SmokeFilter, EngineFilter, ProductFilter,
PerfFilter, StressFilter, NegativeFilter.
- Application context -
EditorContext, ClientContext,
ServerContext, CommandletContext, plus the convenience mask
ApplicationContextMask.
Typical combo for a product-level test runnable in editor / client
/ server contexts:
EAutomationTestFlags::ProductFilter | EAutomationTestFlags::ApplicationContextMask.
Authoring styles
Pick the style that matches the unit under test, then lift the full
pattern from
references/authoring-macros-and-apis.md:
| Style | Macro / API | Use when |
|---|
| Simple test | IMPLEMENT_SIMPLE_AUTOMATION_TEST + RunTest | One assertion body over plain C++ |
| Data-driven | IMPLEMENT_COMPLEX_AUTOMATION_TEST + GetTests | One sub-test per enumerated row (assets, configs) |
| Multi-frame | ADD_LATENT_AUTOMATION_COMMAND | Test must yield to the tick loop across frames |
| BDD Spec | DEFINE_SPEC / BEGIN_DEFINE_SPEC + Describe / It / LatentIt | Readable specs; .spec.cpp files; async via FDoneDelegate |
| UI Driver | IAutomationDriverModule::Get().CreateDriver() + By::Id | Simulate cursor / click / type on UMG; runs off the GameThread |
Running
Iterate in-editor via Window → Test Automation (Session
Frontend), then run headless for CI:
UnrealEditor-Cmd.exe MyGame.uproject \
-ExecCmds="Automation RunTests MyGame.Inventory; Quit" \
-unattended -nopause -testexit="Automation Test Queue Empty" \
-ReportOutputPath="artifacts/automation" \
-log
Command variants, the -ReportOutputPath JSON schema, Gauntlet, and
a full GitHub Actions job are in
references/running-and-reporting.md.
Worked example
Goal: cover an inventory system's stacking rule as a CI-gating BDD
spec, plus a menu-close UI check.
- Category + flags. Stacking logic is pure C++ product logic,
so
ProductFilter | ApplicationContextMask; it runs in well
under a second, so a second SmokeFilter copy joins the PR job.
- Authoring. Write
FInventorySpec with DEFINE_SPEC and a
Describe("AddItem") block holding two It("should …") cases -
one asserting count increases by the stack amount, one asserting
an over-cap add is rejected (pattern in
references/authoring-macros-and-apis.md).
Save it as InventorySpec.spec.cpp.
- UI check. Add
FMenuDriverSpec with the Driver enabled in
BeforeEach on TaskGraphMainThread, the It running on
EAsyncExecution::ThreadPool, By::Id("SubmitButton") clicked,
and Disable() in AfterEach.
- Run headless.
UnrealEditor-Cmd.exe MyGame.uproject -ExecCmds="Automation RunTests MyGame.Inventory; Quit" -unattended -nopause -ReportOutputPath="artifacts/automation" -log.
- Gate. CI reads
artifacts/automation/index.json; the
over-cap It shows "state": "Fail" when the rule regresses, so
the job fails the build. Nightly re-runs the same specs under
ProductFilter | StressFilter.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|
All tests as SmokeFilter | Smoke tests "complete within 1 second" per framework docs - long tests break the smoke contract | Use ProductFilter for tests > 1 s |
| Running Automation Driver on GameThread | API "cannot run on the GameThread" per Driver docs | Use EAsyncExecution::ThreadPool on the driver-using It |
Spec without It descriptions starting with "should" | Runner output reads poorly | Per Spec docs, start descriptions with "should" |
Trusting By::Path locators | "Brittle" per Driver docs | Prefer By::Id with tagged metadata |
Sleep-style waits in latent commands | Flaky under CI load | Use FDoneDelegate (LatentIt) or custom IAutomationLatentCommand::Update() polling |
Spec test file without .spec.cpp extension | Build system may not pick it up | Per Spec docs, use .spec.cpp suffix and no "Test" in filename |
Cloning non-thread-safe shared pointers in async It | Crash under threadpool | Per Driver docs, cache them on the test class |
No BeforeEach / AfterEach cleanup of IAutomationDriverModule | Driver state leaks between specs | Pair Enable() / Disable() calls in BeforeEach / AfterEach per Driver docs |
Limitations
- C++ build required. Tests live in C++ modules; pure-content
Blueprint projects need a code module added to use this
framework. (Pure Blueprint projects can use Blueprint Functional
Tests in-level - see Epic's docs on Functional Testing.)
- No common exit code definition comparable to Unity's caveat -
parse the
-ReportOutputPath JSON or scrape the log for
LogAutomationController Fail lines.
- Spec parameterised tests are loop-generated per
Spec docs;
there is no
[TestCase] analogue from NUnit. The framework is
NUnit-inspired but not NUnit-derived (unlike Unity's UTF).
- Screenshot comparison baseline storage + tolerance
configuration is engine-version-specific - consult per-version
Epic docs.
- Editor-context tests cannot run in dedicated-server only
builds - declare
EditorContext on those tests; client / server
tests need their own flag set.
- Documentation source. The
dev.epicgames.com docs
are the public mirror; deeper detail (full macro implementations,
precise JSON report schema) lives in the engine source under
Engine/Source/Runtime/AutomationController/ and
Engine/Source/Developer/AutomationMessages/ - partners with
engine source access should consult those for authoritative
details.
References