Create a leaderboard route

Paths. scripts/… and references/… below are relative to the directory holding this SKILL.md. Your working directory is the user's project, not that directory, so prefix them with its absolute path or the command is not found.

A route is waypoints plus scenarios pinned to trigger points, in one XML file ([[run-route-scenario]] documents the format). Writing the coordinates by hand does not work in practice — Town12 is 10 km across and a trigger point a metre into the wrong lane silently never fires. So the leaderboard ships interactive tools that read positions off the spectator camera in a live simulator.

All of them live in $LEADERBOARD_ROOT/scripts/ and all need a running server with the right town loaded.

Instructions

Progress:
- [ ] Step 1: Check prerequisites (bash scripts/check_env.sh)
- [ ] Step 2: Load the town the route will live in (load-map)
- [ ] Step 3: Place waypoints (create), then scenarios (scenarios)
- [ ] Step 4: Inspect: display in the simulator, summarise as a table
- [ ] Step 5: Order the scenarios, then run it (run-leaderboard-evaluation)

Step 2-3: Build it

source scripts/env.sh

# 1. load the town first — every tool works on the *currently loaded* map
python3 -c "import carla; carla.Client('127.0.0.1',2000).load_world('Town12')"

# 2. place the route waypoints by flying the spectator and clicking
bash scripts/route_tools.sh create --file ~/routes/my_routes.xml

# 3. add scenarios along it
bash scripts/route_tools.sh scenarios --file ~/routes/my_routes.xml

# 4. capture the current weather as a <weather> element
bash scripts/route_tools.sh weather --route 0

route_creator and scenario_creator are terminal + spectator loops: you move the spectator in the CARLA window, and type commands in the terminal to record the current position as a waypoint or a scenario trigger. --file accepts several files (nargs="+") so it can append into an existing set.

scenario_creator -s/--show-only shows the route without editing, which is the safe way to look before touching.

Step 4: Inspect

# draw an existing route in the simulator
bash scripts/route_tools.sh display --file ~/routes/my_routes.xml --route 0 --keypoints --scenarios

# a table of every route: length, towns, scenario histogram
bash scripts/route_tools.sh summary --file ~/routes/my_routes.xml

# offline structural check — no server needed
bash scripts/route_tools.sh check --file ~/routes/my_routes.xml

check is this skill's own addition and the one to run first: it validates the XML against the 2.x schema shape, confirms every <scenario type> has a class in the paired scenario_runner checkout, flags routes whose town is not installed, and finds trigger points that are far from the route polyline — the failure that otherwise shows up as "the scenario never triggers".

route_displayer flags: -sr/--show-route, -sa/--show-all, -sk/--show-keypoints, -ss/--show-scenarios.

Step 5: Tidy and run

# sort the <scenarios> block by position along the route (cosmetic but makes review sane)
bash scripts/route_tools.sh order --file ~/routes/my_routes.xml

cd ../run-leaderboard-evaluation
ROUTES=~/routes/my_routes.xml TEAM_AGENT=.../npc_agent.py \
    bash scripts/run_leaderboard.sh --routes-subset 0

Always validate a new route with npc_agent.py before using it to judge your own agent: a route the NPC agent cannot complete is a broken route, not a hard one.

Converting 1.0 routes

bash scripts/route_tools.sh bridge --routes old_routes.xml \
    --scenarios all_towns_traffic_scenarios_public.json --endpoint new_routes.xml

scripts/route_bridge.py merges a 1.0 geometry file and its separate scenario JSON into a single 2.x route file. It exists on master only — it was removed from the leaderboard-2.0 and 2.1 branches, so run it from a master checkout and use the output elsewhere.

Examples

Example 1: "make a short route in Town12 to test one scenario"

Load Town12, create two or three waypoints along one road, scenarios to drop a single Accident trigger, check, then run with the NPC agent. Faster to iterate on than any of the shipped 90-route files.

Example 2: "why doesn't my scenario trigger?"

check first — it reports trigger points off the route polyline and scenario types with no class. If both are clean, display --scenarios and watch where the ego passes relative to the trigger.

Example 3: "what is actually in routes_training.xml?"

summary --file $LEADERBOARD_ROOT/data/routes_training.xml: 90 routes, all Town12, 4629 scenario instances across 38 types. check on the same file is a good sanity test of your own checkout pairing.

Example 4: "I want the same route in fog and at night"

weather with the simulation set as you want it prints a <weather> element ready to paste. Two entries with route_percentage="0" and "100" interpolate along the route; more entries give a profile.

Troubleshooting

Problem: a tool exits immediately or draws nothing Cause: no server, or the wrong town loaded. These tools operate on the currently loaded map and do not load one for you. Solution: load the town first; confirm with check_env.sh.

Problem: ModuleNotFoundError: No module named 'pygame' Cause: the interactive tools need pygame (in leaderboard/requirements.txt). Solution: pip install pygame.

Problem: scenarios in the file never trigger during a run Cause: the type has no class in this scenario_runner branch — RouteScenario skips unknown types silently — or the trigger point is off the driven path. Solution: check. Branch pairing is the usual culprit: master scenario_runner with a leaderboard-2.1 route set, or the reverse.

Problem: the ego spawns facing backwards Cause: the first waypoint's lane direction. Routes take their heading from the road, not from a yaw you set. Solution: move the first waypoint, or reverse the waypoint order.

Problem: route_bridge.py: No such file Cause: it only exists on leaderboard master. Solution: run it from a master checkout; the output is a plain 2.x route file.

Problem: edits to the route file did not take effect on --resume Cause: RouteIndexer.validate_and_resume compares the total route count and restarts from scratch if it changed — and silently continues at the old index if it did not. Solution: never edit a route file mid-evaluation; use a fresh checkpoint.

Problem: --routes-subset 3 runs a different route than expected Cause: the subset is an index into the file, not a route id. Solution: summary shows both; keep ids sequential from 0 so they coincide.

Outputs

A route XML in the 2.x format, plus debug drawings in the simulator and a table on stdout. check is read-only and exits non-zero on any problem that would make a route silently misbehave. The interactive tools write to the --file you name.

Route element semantics, the interactive key/command sets and what each shipped script does are in references/authoring.md.