DOCA DPA High-Level Tracer

Examples

• "My DPA kernel is doing the wrong thing — where do I look?" — this is in scope when there is no host-side DOCA_ERROR_*. Capture then decode; do not start with cycle-only profiles.
• "TRACE mode buried the DPA in observation overhead." — switch capture window and TRACE vs CRIT before changing the kernel.

Instructions

Follow Loading order below. Open the tasks companion for configure / run / debug. Open the capabilities companion for event taxonomy. Do not open every companion file.

Where to start: This is a tool skill for invoking doca_dpa_hl_tracer — the documented host-side CLI that captures DPA-side execution traces in higher-level terms (DPA programming events: kernel entry / exit, sync points, comm primitive calls, RDMA WR submission, completions) rather than raw cycle counts. Open the tasks companion and start at ## configure for the mode-vs-overhead decision and the JSON config layout, then ## run for the capture → decode → render pipeline. Open the capabilities companion when the question is what does this tool actually trace, which DPA programming events does it expose, what is the trace-overhead vs fidelity tradeoff, or how does it slot into a DPA debug loop alongside doca-dpa and doca-debug. If DPA is not the right surface for the user's question (e.g. the bug is host-side, the bug is in the DPACC-produced image, the user wants raw cycle counts), the path-selection rule in the capabilities companion routes the agent before any capture is attempted.

Example questions this skill answers well

The CLASSES of doca_dpa_hl_tracer questions this skill is built to answer, each with one worked example. The class is the load-bearing piece; the worked example is one instance.

• "My DPA kernel is doing the wrong thing — where do I look?" — worked example: "my host-side doca_dpa_kernel_launch_update_* completes, but the kernel's reported result is wrong; no host-side DOCA_ERROR_*". Answered by the when DPA-side high-level tracing is the right surface gate in the capabilities companion
  • the capture → decode → render flow in the tasks companion + the which DPA programming events to focus on rule in the tasks companion.
• "My DPA kernel is slow at a granularity that doesn't show up in cycle profiles — how do I see kernel-entry to first-comm-call latency?" — worked example: "my DPA kernel runs but the time between launch and the first RDMA WR submission is bigger than I expected". Answered by the event-taxonomy table in the capabilities companion
  • the iterative loop in the tasks companion which treats trace overhead, mode (TRACE vs CRIT), and capture window as axes to tune.
• "How do I capture a trace without burying the DPA in observation overhead?" — worked example: "TRACE mode is producing too much data and my measured DPA latency went up by 2x compared to without the tracer". Answered by the mode-vs-overhead tradeoff in the capabilities companion
  • the CRIT-first guidance in the tasks companion (start with critical-events-only; widen to TRACE only when the bug demands per-event detail).
• "My trace file got truncated mid-run — how should I configure the file-size limits?" — worked example: "binary trace file hit 5 GB and the capture stopped". Answered by the log_file_max_size_in_bytes / bin_file_max_size_in_bytes / file_size_limit_policy triple in the capabilities companion
  • the JSON config layout in the tasks companion.
• "Is the tracer on my install, and is it paired with the matching doca-dpa library and DPACC compiler version?" — worked example: "is the tracer ABI on my install compatible with the DPA image my DPACC just produced?". Answered by the version-overlay in the capabilities companion, which redirects to the canonical doca-version chain and adds the tracer ↔ doca-dpa library ↔ DPACC compiler match rule.
• "The capture file looks empty / decode failed — is the install broken, no events fired, or am I tracing the wrong thing?" — worked example: "doca_dpa_hl_tracer ran, wrote a file, but the parser shows zero events". Answered by the layered error taxonomy in the capabilities companion (install / device-binding / DPA-image-instrumented / capture-window / decode-vs-elf / overhead-saturated / version / cross-cutting) + the layered walk in the tasks companion.

Audience

This skill serves external developers, platform operators, and AI agents who have already brought up a DPA-side workload through doca-dpa and now need higher-level visibility into what the DPA kernel is actually doing on the wire — DPA programming events ordering, sync gaps, comm-call latencies, RDMA-WR / completion timing — without dropping all the way down to raw cycle counters. Concretely:

• A DPA developer who can launch their kernel cleanly from the host side but whose kernel's result is wrong or whose DPA-side performance is below expectation, and who needs a DPA-side ground truth before triaging.
• A platform operator running a DPA-using workload (RDMA offload from accelerator, custom CC algorithm via doca-pcc) and needs to localize a regression to the DPA side without instrumenting the application.
• An AI agent producing a DPA-side trace report as evidence for the host-side doca-dpa ladder when the host side reports clean completions but the DPA-side behaviour is wrong.

It is not for users debugging the tracer binary itself, not a substitute for the live public DOCA DPA Tools guide, not the right place for users learning how to write a DPA kernel (that audience belongs in doca-dpa plus the public DOCA DPA / DPACC / DPA-Comms / DPA-Verbs guides), and not the right place for raw per-instruction cycle profiling (different surface, different tool — route via doca-public-knowledge-map ## DOCA tools).

The tracer is shipped as a CLI binary under /opt/mellanox/doca/tools/, not a library you link against. The skill uses the same kind: tool three-file shape as the rest of the bundle so the agent's task-verb contract is uniform across libraries, services, and tools.

Language scope

doca_dpa_hl_tracer is a C++ host-side CLI. Its inputs are its JSON config file, the DPA-side ELF (the doca_dpa_app-class image produced by DPACC), and a running DPA-side workload that the host-side doca-dpa lifecycle already started. Its outputs are a binary trace file (bin_file) and a human-readable log file (log_file). The skill keeps the workflow guidance language-neutral — the DPA-side workload it traces can be C compiled by DPACC or any other DPA translation unit DPACC accepts — and routes per-language questions to the public DPA / DPACC guides via doca-public-knowledge-map.

When to load this skill

Load this skill when the user is — or the agent needs to — invoke doca_dpa_hl_tracer on a real host with DOCA installed against a BlueField with a DPA processor visible to the host, and the host-side doca-dpa lifecycle has already brought a DPA workload up at least once. Concretely:

• Capturing a DPA-side trace to localize a DPA kernel's wrong-result or wrong-ordering behaviour when the host-side doca-dpa lifecycle reports clean completions.
• Capturing a DPA-side trace to localize a DPA-side performance gap (kernel-entry to first-comm latency, RDMA-WR-issue to completion gap, sync-point dwell time) at a granularity above raw cycle counts.
• Choosing between TRACE and CRIT capture modes based on the bug-vs-overhead tradeoff and the available capture window.
• Tuning the JSON config (thread priorities, core affinities, file size limits, file-size-limit policy) so the capture itself does not perturb the workload more than the bug it is investigating.
• Decoding a captured bin_file against the matching DPA-side ELF to render the human-readable event stream.
• Capturing a side-effect-bounded trace as prerequisite evidence for a host-side doca-dpa ladder step.

Do not load this skill for general DOCA orientation, DPA-side programming model questions, raw cycle profiling, or DOCA / DPACC install. For those, route to doca-public-knowledge-map, doca-dpa, or doca-setup.

What this skill provides

This is a thin loader. Substantive material lives in two companion files:

• the capabilities companion — what doca_dpa_hl_tracer captures: the DPA programming event taxonomy (kernel entry / exit, sync points, comm primitive calls, RDMA WR submission and completion drain), the two documented capture modes (TRACE for full per-event, CRIT for critical-events only), the trace-overhead-vs-fidelity tradeoff, the config-file shape (receiver / binary-writer / file-writer / printer threads with priority + core affinity, file size limits, file_size_limit_policy), the capture-window + workload-must-be-running invariant, the ELF-must-match-image rule for decode, the version-availability overlay (tracer ↔ doca-dpa library ↔ DPACC compiler), the layered error taxonomy (install / device-binding / image-instrumented / capture-window / decode / overhead-saturated / version / cross-cutting), the observability surface (binary trace file + log file + tool's own stderr), and the safety policy (capture is bounded; tracing is not a production observability surface).
• the tasks companion — step-by-step workflows for the in-scope task verbs: install (route to host-side DOCA install + DPA prerequisites), configure (mode + JSON config layout + capture window), build (route to install — the binary is shipped, the DPA-side application is user-built by DPACC), modify (refuse — do not patch the binary; modify the JSON config and the invocation instead), run (the capture flow with --mode, --config-file, --output-file), test (iterative loop tuning mode, window, and overhead), debug (walk the error taxonomy), use (consume the decoded trace in a doca-dpa debug session), plus a Deferred task verbs block.

The skill assumes a host where DOCA is already installed at the standard location, a BlueField with a DPA processor is present and visible to the host, the DPACC compiler is installed at a version matched to the host-side DOCA, the DPA-side application image (the ELF the tracer decodes against) is on disk and matches what the doca-dpa lifecycle loaded, and the operator has the privileges the public DOCA DPA Tools guide requires.

What this skill deliberately does not ship

This skill is agent guidance, not a samples or scripts bundle. To keep the boundary clean, it deliberately does not contain — and pull requests should not add:

• Specific flag strings, event names, or mode tokens beyond what the public DOCA DPA Tools page and --help document. The DPA programming events surface evolves release to release; --help on the installed binary is the authoritative inventory.
• Pre-baked example traces or expected event timings. Trace output is workload-, DPA-image-, BlueField-, and firmware-specific; a captured example pinned to one setup misleads operators elsewhere.
• Wrappers, parsers, or rendering scripts in any language that consume the binary trace format. The format is documented; users who want to script against it should read the live guide and write the parser against their installed version.
• A specific tuning recommendation derived from a single trace. A DPA-side perf decision (move a sync, batch a comm call, change a launch argument) is a workload question and the skill prescribes how to capture and read traces — it refuses to translate a captured gap into a kernel-rewrite recommendation without the user's own analysis.
• A samples/ or reference/ subtree. This is a thin loader for a shipped CLI; substantive material lives on the public page, in --help, and in doca-dpa.

Loading order

1. Read this skill first to confirm the user's question is in scope (DPA-side high-level tracing, not DPA-side programming and not raw cycle profiling).
2. For the event taxonomy, capture modes, overhead tradeoff, JSON config layout, version overlay, error taxonomy, observability, and safety policy, see the capabilities companion.
3. For the documented invocations and the capture → decode → render workflow — install, configure, build, modify, run, test, debug, use — see the tasks companion.

Related skills

• doca-dpa — the host-side DPA control library whose loaded application image the tracer captures. Pair them in every DPA debug session: doca-dpa brings the workload up; the tracer captures what the workload does at the DPA programming event layer. Conflating the library with the tracer is the most common DPA-debug first-touch error.
• doca-debug — the cross-cutting debug ladder. The tracer slots in at the runtime layer as the DPA-side ground truth before any DPA-side perf or correctness conclusion is made.
• doca-public-knowledge-map — routing to the public DOCA DPA Tools page on docs.nvidia.com and the rest of the public DOCA documentation set.
• doca-version — canonical DOCA version-handling rules. The ## Version compatibility section in the capabilities companion is a concise overlay that redirects here for the body and adds the tracer ↔ doca-dpa library ↔ DPACC compiler matching rule.
• doca-setup — env preparation, install verification, DPACC compiler install / verification, BlueField mode (the DPA processor must be exposed before any tracing is meaningful), and the I have no install yet path with the public NGC DOCA container.
• doca-structured-tools-contract — the bundle's detect → prefer → fall back → report contract for structured helper tools. The command appendix in the tasks companion honors this contract.
• doca-programming-guide — general DOCA programming patterns shared by every library / tool surface, including the cross-library DOCA_ERROR_* taxonomy this tool's host-side error layer overlays on top of when host-side doca-dpa calls fail in tandem.

The DPA-side companion libraries doca-dpa-comms (comm primitives the DPA kernel itself calls) and doca-dpa-verbs (RDMA verbs the DPA kernel itself calls) are different artifacts that the tracer's DPA programming events surface visibly names; for the DPA-side programming model itself, route through doca-public-knowledge-map to the public DOCA DPA-Comms and DPA-Verbs guides and to the shipped /opt/mellanox/doca/samples/doca_dpa/ samples. This tool traces their use; it does not redefine them.