rr Deterministic Debugger

rr provides deterministic record-replay debugging with full reverse execution capabilities.

Core Workflow

1. Record: rr record <program> [args]

   Sandbox exemption (documented decision): rr needs ptrace and perf counters, which libexec/raptor-run-sandboxed denies, so recording runs the untrusted binary unsandboxed — behavior observed under earlier sandboxed runs says nothing about what the payload does with this ambient authority. Keep the recorded invocation to exactly the reproduced crash command (never a broader test suite or build), and treat any unexpected network or filesystem activity during recording as a finding in itself. Replay carries no such residual: it re-executes the recorded trace under gdb, not the binary against the host.

2. Replay: rr replay -- -nx -iex 'set auto-load off' -iex 'set auto-load safe-path /dev/null' (enters gdb interface with reverse execution). The flags after -- go to gdb and disable auto-load with no trusted directory: the recorded binary is untrusted, and a permissive gdb config (e.g. an operator ~/.gdbinit widening auto-load safe-path) would otherwise execute scripts the binary or its build tree plants (.debug_gdb_scripts, *-gdb.py). Same hardening set as scripts/crash_trace.py — use it on EVERY manual replay.

Reverse Execution Commands

All standard gdb commands work, plus reverse variants:

• reverse-next / rn: Step back over function calls
• reverse-step / rs: Step back into functions
• reverse-continue / rc: Continue backward to previous breakpoint
• reverse-stepi / rsi: Step back one instruction
• reverse-nexti / rni: Step back over one instruction

Crash Trace Extraction

Regular Crashes

After rr record <crashing-program>:

rr replay -- -nx -iex 'set auto-load off' -iex 'set auto-load safe-path /dev/null'
# In gdb:
reverse-next 100    # Go back 100 steps (adjust N as needed)
# Now step forward to see execution leading to crash:
next
next
...

ASAN Crashes

After rr record <asan-program>:

rr replay -- -nx -iex 'set auto-load off' -iex 'set auto-load safe-path /dev/null'
# In gdb:
bt                  # View stack trace
up                  # Issue "up" commands until last app frame (before ASAN runtime)
break *$pc          # Set breakpoint at that location
reverse-continue    # Go back to last app instruction before ASAN
# Now step forward to see execution leading to fault:
next
next
...

Inspecting Variables and Memory

Standard gdb commands work at any point:

• print <var>: Print variable value
• print *<ptr>: Dereference pointer
• x/<format> <address>: Examine memory
  • x/10xb <addr>: 10 bytes in hex
  • x/s <addr>: String at address
• info locals: Show local variables
• info args: Show function arguments

Source vs Assembly View

• list: Show source code around current location
• disassemble: Show assembly around current location
• layout src: TUI source view
• layout asm: TUI assembly view
• set disassemble-next-line on: Show assembly with each step

Automation Script

Use scripts/crash_trace.py to automatically extract execution trace before crash.