Read leaderboard results

Paths. scripts/… and references/… below are relative to the directory holding this SKILL.md. Your working directory is the user's project, not that directory, so prefix them with its absolute path or the command is not found.

driving score = route completion × infraction penalty

Route completion is the percentage of the route driven. The infraction penalty starts at 1.0 and is reduced by what you did wrong — and how it is reduced is the one thing that changed between 2.0 and 2.1.

Instructions

Progress:
- [ ] Step 1: Read the file (summary)
- [ ] Step 2: Drill into the routes that scored badly
- [ ] Step 3: If the numbers disagree with the online leaderboard, rescore

Step 1: Summary

python3 scripts/read_results.py ~/results/results.json

Prints entry status and eligibility, the declared sensors, progress, a per-route table (driving score / route % / penalty / top infractions), and the global record with mean and standard deviation.

Two fields decide whether the run counts at all:

• entry_status — Finished, Started, Rejected, Crashed or Invalid.
• eligible — True only for Finished. One crashed route out of 90 makes the entire entry ineligible, not just that route.

Step 2: Drill in

python3 scripts/read_results.py results.json --route RouteScenario_3 -v

Full scores, meta (route length, game and wall-clock duration) and every infraction message with its location. The -v messages are what tell you where the ego hit something, which is the difference between "collided" and "collided at the Accident scenario in the third junction".

Reading a score:

Symptom	Read as
route % high, penalty low	drives the route but breaks rules
route % low, penalty 1.0	clean but stops early — check vehicle_blocked, route_dev, route_timeout
status Agent got blocked	ActorBlockedTest fired; the ego stood still too long
status Agent crashed	an exception in run_step, or a watchdog timeout
eligible: false with good scores	some route did not finish; find it and fix it
outside_route_lanes large	driving on the wrong lane; it reduces route completion, not the penalty

Step 3: Rescore across versions

python3 scripts/read_results.py results.json --as 2.1   # 2.0 result, 2.1 rules
python3 scripts/read_results.py results.json --as 2.0

This is exact for constant-penalty infractions, because the infraction counts are what both formulas consume and the file stores them precisely. Verified round-trip: rescoring a 2.0 file --as 2.0 reproduces its stored scores to the cent.

The difference is large and not a constant offset:

pedestrian collisions on a completed route	2.0	2.1
1	50.0	50.0
2	25.0	33.3
5	3.1	16.7

2.0 multiplies (P = Π p_j^n_j), so the first infraction is cheap and later ones are nearly free — which rewarded stopping early. 2.1 sums (P = 1/(1+Σ c_j n_j)), so every infraction costs the same absolute amount. Full coefficient tables in references/scoring.md.

min_speed_infractions and outside_route_lanes scale with a percentage the file does not store, so rescoring leaves them out and says so — those routes come out as an upper bound.

Compare and merge

python3 scripts/read_results.py before.json after.json --diff
python3 scripts/read_results.py shard0.json --merge shard1.json shard2.json --out all.json

--diff matches by route id and reports the score delta plus which infraction counts changed. --merge joins shards from parallel runs; it does not recompute the global record — use the leaderboard's own scripts/merge_statistics.py when you need an official global score.

Examples

Example 1: "what did my agent score?"

read_results.py results.json. The global driving score mean is the number that would rank you; the std dev tells you how much of it is luck.

Example 2: "why is my score 12 when route completion is 95%?"

--route <id> -v. A penalty of ~0.13 with 95% completion means several infractions. Under 2.0 that is roughly three collisions; the messages name them.

Example 3: "my local score doesn't match my submission"

Check which version produced the file: master and leaderboard-2.0 score multiplicatively, the live leaderboard has scored additively since March 2025. --as 2.1 converts without re-running. If the counts also differ, it is not the formula — it is a different CARLA build, TM seed or route subset.

Example 4: "did I improve?"

--diff before.json after.json. Score deltas plus infraction-count changes, so a higher mean that hides a new failure mode is visible.

Troubleshooting

Problem: results.json is not valid JSON Cause: the run was killed mid-write; the checkpoint is rewritten after every route. Solution: the previous route's data is lost. Re-run with --resume 1 after truncating, or use a shard from another run.

Problem: no completed routes Cause: only in-progress routes. The evaluator saves records with index != -1 only, so a route still running is not in the file. Solution: wait; watch _checkpoint.progress.

Problem: global_record missing Cause: it is written when the run finishes every route. Solution: expected for a partial run; per-route records are still there.

Problem: scores are 0 with no infractions Cause: route completion is 0 — the agent never moved, or the sensors were rejected. Solution: check entry_status; Rejected means sensor validation failed ([[write-leaderboard-agent]] validate).

Problem: infractions per km look tiny in the global record Cause: global infractions are normalised per kilometre; per-route records hold raw counts. Solution: expected. Compare like with like.

Problem: --merge drops routes Cause: duplicate route ids across shards are skipped, by design. Solution: shard by --routes-subset so ids do not overlap.

Outputs

A readable summary of any results file, a per-route drill-down, a cross-version rescore, a two-run diff, or a merged results file. All read-only except --merge, which writes a new file and never modifies its inputs.

The formulas, coefficient tables, every infraction key and the full results.json schema are in references/scoring.md.