Analyse scenario results

Paths. scripts/… and references/… below are relative to the directory holding this SKILL.md. Your working directory is the user's project, not that directory, so prefix them with its absolute path or the command is not found.

Three separate things get called "results", and they answer different questions:

Artefact	Produced by	Answers
criteria table	--output / --file / --json / --junit	did it pass, and which criterion failed
criteria JSON next to a recording	--record	the full criterion state, for tooling
CARLA recorder log	--record	everything that happened, replayable

The metrics module works off the third one, so anything you did not --record can only be re-measured by re-running.

Instructions

Progress:
- [ ] Step 1: Check prerequisites (bash scripts/check_env.sh)
- [ ] Step 2: Summarise the result files you have
- [ ] Step 3: For deeper analysis, run a metric over a recording
- [ ] Step 4: Compare runs if you have more than one

Step 2: Summarise

source scripts/env.sh

python3 scripts/analyze_results.py summary ./results            # a directory of results
python3 scripts/analyze_results.py summary ./results/*.json     # specific files
python3 scripts/analyze_results.py compare ./before ./after      # two runs, criterion by criterion

summary reads .json, .xml (junit) and .txt outputs, normalises them, and prints one line per scenario with the failing criteria named. compare diffs two sets by scenario name and criterion, which is the useful view when you changed a controller and want to know what got worse.

To produce these in the first place, run with the output flags:

cd ../run-scenario
OUTPUT=1 JSON=1 OUTPUT_DIR=./results bash scripts/run_scenario.sh group:ControlLoss

Note the file naming: ScenarioRunner writes <outputDir>/<config name><YYYY-MM-DD-HH-MM-SS>.json with no separator before the timestamp, so names look like ControlLoss_12025-08-14-10-22-31.json. analyze_results.py splits that back apart.

Step 3: Metrics over a recording

Record first — this writes both a .log recording and a <name>.json of criteria:

cd ../run-scenario
RECORD=recordings bash scripts/run_scenario.sh FollowLeadingVehicle_1
# -> $SCENARIO_RUNNER_ROOT/recordings/FollowLeadingVehicle_1.log
#    $SCENARIO_RUNNER_ROOT/recordings/FollowLeadingVehicle_1.json

Then run a metric:

source scripts/env.sh

python3 scripts/analyze_results.py metrics --list      # bundled example metrics

python3 scripts/analyze_results.py metrics \
    --metric "$SCENARIO_RUNNER_ROOT/srunner/metrics/examples/distance_between_vehicles.py" \
    --log recordings/FollowLeadingVehicle_1.log \
    --criteria recordings/FollowLeadingVehicle_1.json

which is a wrapper for

python3 "$SCENARIO_RUNNER_ROOT/metrics_manager.py" \
    --metric <metric.py> --log <recording.log> [--criteria <criteria.json>]

The metrics module needs a running server even though it computes offline: it replays the recording to recover the map and to resolve actor ids to positions. That is the one non-obvious prerequisite.

Bundled examples, in increasing complexity:

Metric	What it shows
basic_metric.py	the minimum: subclass BasicMetric, implement _create_metric
criteria_filter.py	pull specific fields out of the criteria JSON into a table
distance_between_vehicles.py	per-frame distance between two actors, plotted
distance_to_lane_center.py	lateral error against the lane centre, plotted

Writing your own is a subclass of BasicMetric with one method; the recording is exposed as a MetricsLog object with get_actor_transforms, get_ego_vehicle_id, get_all_frames and friends. Details in references/metrics.md.

Step 4: What the criteria mean

A failing criterion names the behaviour that broke, not the cause:

Criterion failed	Read as
CollisionTest	the ego hit something — check the recording for what
DrivenDistanceTest	the ego did not get far enough: blocked, or the scenario ended early
ActorBlockedTest	the ego stopped and stayed stopped
KeepLaneTest / WrongLaneTest / OnSidewalkTest	lateral control or a bad overtake
RunningRedLightTest / RunningStopTest	traffic-control handling
RouteCompletionTest < 100	route not finished (route mode)
InRouteTest	the ego left the route corridor — usually a wrong turn
ScenarioTimeoutTest	a scenario did not finish in its own budget

"All scenario tests were passed successfully!" on stdout means every criterion passed. "Not all scenario tests were successful" with no table means you forgot --output.

Examples

Example 1: "did my controller pass the control-loss scenarios?"

Run group:ControlLoss with OUTPUT=1 JSON=1 OUTPUT_DIR=./results, then summary ./results. 15 configs, one line each, failures named.

Example 2: "why did FollowLeadingVehicle_1 fail?"

summary says CollisionTest. Re-run with RECORD=recordings, then replay it ([[replay-recording]]) to watch the impact, and run distance_between_vehicles.py to see the gap closing rate.

Example 3: "compare before and after my change"

Two OUTPUT_DIRs, then compare ./before ./after. It reports criteria that flipped in either direction, so a fix that broke something else is visible.

Troubleshooting

Problem: "Not all scenario tests were successful" with no detail Cause: no output flag. Solution: add OUTPUT=1 (stdout) and/or JSON=1.

Problem: no result files in OUTPUT_DIR Cause: --outputDir only takes effect with one of --file/--json/--junit; --output alone prints to stdout. Solution: set JSON=1 as well.

Problem: metrics_manager.py fails with a connection error Cause: it needs a live server to replay against. Solution: start one ([[run-carla-server]]); the map does not matter, the recording names its own.

Problem: the metric runs but every transform is None Cause: actor ids in the recording do not match what the metric asks for, usually because it hard-codes ids from a different run. Solution: resolve ids through MetricsLog.get_ego_vehicle_id() and get_actor_ids_with_role_name() rather than by number.

Problem: --criteria file not found next to the recording Cause: the criteria JSON is written by _record_criteria only when --record was used, and its name is the recording name with .log replaced by .json. Solution: re-run with RECORD=; without it there is no criteria file to pass.

Problem: replaying a recording made by a different CARLA version does nothing Cause: recorder logs are version-specific. Solution: replay with the version that recorded it.

Outputs

A per-scenario pass/fail summary, a criterion-level diff between two runs, or a metric's table/plot over a recording. Nothing is modified: all three modes are read-only over files, plus a replay on the server for metrics.

The metrics API — BasicMetric, MetricsLog, and what the recorder actually stores — is in references/metrics.md.