DOCA GPUNetIO ib_write_bw

Examples

• "What sustained RDMA-WRITE bandwidth can GPUNetIO deliver on this GPU-NIC pair?" — this is in scope. Stay on gpunetio_ib_write_bw; do not retune a live GPI app here.
• "Where is the bottleneck — GPU occupancy, NIC issue rate, or link saturation?" — decompose throughput before changing the CUDA kernel.

Instructions

Follow Loading order below. Open the tasks companion for configure / run / debug. Open the capabilities companion for GPU-NIC pairing and observability. Do not open every companion file.

Where to start: This is a tool skill for the GPUNetIO- flavored ib_write_bw benchmark shipped under doca/tools/gpunetio_ib_write_bw/ (a client + server pair, built from source against the installed DOCA via meson). It measures sustained RDMA WRITE bandwidth when the WRs are posted from a CUDA kernel through the doca-gpunetio device-side surface, with the GPU on the data path. Open the tasks companion and start at ## configure for the GPU-NIC pairing precondition and the build pattern; jump to ## run for the smoke-before-bulk flow. Open the capabilities companion when the question is what this tool actually measures, how the result decomposes (GPU occupancy vs NIC issue rate vs link saturation), or how the result reads against the GPI sister tool and the upstream CPU-initiated perftest ib_write_bw. If DOCA is not installed yet, route to doca-setup first; if the user is still deciding between the GPI and GPUNetIO programming surfaces, the picture in doca-gpunetio and doca-gpi is the first stop.

Example questions this skill answers well

The CLASSES of doca-gpunetio-ib-write-bw questions this skill is built to answer, each with one worked example. The class is the load-bearing piece; the worked example is one instance.

• "What sustained RDMA-WRITE bandwidth can the GPUNetIO path deliver on this GPU-NIC pair?" — worked example: "measure sustained WRITE BW between two hosts with an H100 + ConnectX-7 on each side". Answered by the GPU-NIC pairing precondition in the capabilities companion
  • the bring-up flow in the tasks companion + the tasks companion. The same shape answers "measure GPUNetIO-driven WRITE BW between a host GPU and a BlueField DPU".
• "Where is the bottleneck — GPU compute occupancy, NIC issue rate, or link saturation?" — worked example: "I see 120 Gbit/s on a 200 Gbit/s link; is the NIC saturated, am I CPU-bound on the client, or is the CUDA kernel not driving enough WRs in flight?". Answered by the throughput-decomposition rules in the capabilities companion
  • the eval-loop overlay in the tasks companion.
• "How does the result differ from the classic CPU- initiated perftest ib_write_bw?" — worked example: "my team has a CPU-initiated WRITE BW number on this same NIC; should I expect the GPUNetIO number to match or be different?". Answered by the "GPU-initiated path adds (or removes) overhead vs the CPU-initiated path" rule in the capabilities companion.
• "Is the doca-gpunetio path the right surface for my sustained-throughput workload class?" — worked example: "my application streams sensor data from GPU memory at line rate to a remote consumer". Answered by the "when GPUNetIO is the right surface vs GPI vs CPU- initiated" rule in the capabilities companion
  • the use-side decision in the tasks companion.
• "My BW number swings between runs. What do I check before quoting it?" — worked example: "three runs at the same flags gave 145, 187, and 160 Gbit/s; is the benchmark noisy or is my platform inconsistent?". Answered by the measurement-soundness rules in the capabilities companion layer 5 + the steady-state guidance in the tasks companion.
• "What version of DOCA + CUDA Toolkit do I need for this binary to build and run?" — worked example: "my install has DOCA at one semver and CUDA at another; will the ToT-shipped gpunetio_ib_write_bw even link?". Answered by the version overlay in the capabilities companion which cross-links the canonical detection chain in doca-version.

Audience

This skill serves external developers and performance engineers who need a reproducible measurement of sustained RDMA WRITE bandwidth when the WRs are posted from a CUDA kernel through doca-gpunetio, on the user's actual install and GPU-NIC pair. Concretely:

• A developer comparing the GPUNetIO path against the GPI path or the host-initiated perftest-style path before committing an application design to one of them.
• A platform operator validating a tuning change (NUMA pinning, GPU PCIe placement, IB device choice, GID index, NIC firmware burn) by re-running this benchmark against the new state.
• An SRE / performance engineer producing a "this is the GPUNetIO-driven WRITE BW on this GPU-NIC pair today" artifact downstream consumers can cite.
• An AI agent answering "is the doca-gpunetio path a win for my sustained-throughput workload class" honestly — with a measured number, the build + invocation that produced it, and the GPU + NIC + DOCA version that scopes it — rather than guessing from datasheet headlines.

It is not for users debugging the doca-gpunetio library itself (route to ../../libs/doca-gpunetio/SKILL.md), and not a substitute for the perftest upstream ib_write_bw (which measures CPU-initiated WRITE BW).

Language scope

The doca-gpunetio-ib-write-bw tool is shipped as C plus a CUDA .cu translation unit under doca/tools/gpunetio_ib_write_bw/, split into a client/ subtree and a server/ subtree. The verified surface (per client/{main.c,common.h,common.c,kernel.cu,perftest.c} and server/{main.c,common.h,common.c,perftest.c}): host-side build via meson against the installed DOCA pkg-config modules (doca-gpunetio, doca-rdma, doca-common); the device-side build via nvcc against the DOCA GPU NetIO device-side header set; the OOB descriptor exchange via a TCP socket between client and server. There is no Python / Rust / Go binding — the tool is a pair of CLI binaries. The skill's job is to keep the operator-side workflow language-neutral; the device-side CUDA surface is not wrappable in another language.

When to load this skill

Load this skill when the user is — or the agent needs to — build and run the gpunetio_ib_write_bw client + server on real hosts with DOCA installed plus a CUDA Toolkit matched to the DOCA install, and a GPU + IB device pair on the host's PCIe topology. Concretely:

• Measuring sustained kernel-initiated RDMA WRITE bandwidth between two hosts (or a host and a BlueField DPU) with the GPUNetIO surface.
• Deciding whether the GPUNetIO path is the right runtime surface for a class of workload vs the GPI programming surface (the doca-gpi library — doca/tools/ ships no GPI benchmark binary) or the classic CPU-initiated perftest path.
• Capturing a documented baseline (build + invocation + DOCA version + GPU + NIC + as-deployed environment + numbers) for later regression hunts.
• Diagnosing a build / link / run failure that surfaces the GPUNetIO + RDMA bring-up sequence under this tool's shipped scaffolding.

Do not load this skill for general DOCA orientation, library API work, or installation. For those, use doca-public-knowledge-map, ../../libs/doca-gpunetio/SKILL.md, or doca-setup. Do not load it for application-level end-to-end throughput either — this benchmark measures the WR-submission path through GPUNetIO, not the user's full pipeline.

What this skill provides

This is a thin loader. Substantive material lives in two companion files:

• the capabilities companion — what the tool measures (the sustained-WRITE-BW primitive driven by a client-side CUDA kernel through doca-gpunetio), the runtime-surface selection rule (GPUNetIO vs GPI vs CPU-initiated), the GPU-NIC pairing precondition, the throughput-decomposition guide (GPU compute occupancy vs NIC issue rate vs link saturation), the version overlay (DOCA .pc PLUS CUDA Toolkit), the layered error taxonomy (config-syntax / build-time / GPU-NIC- pairing / GPUNetIO-lifecycle / RDMA-connection / measurement-soundness / version / cross-cutting), the observability surface (stdout report, DOCA log levels, OOB-socket exchange), and the safety overlay (the "GPU-side handle is a credential" rule from doca-gpunetio; the cross-cutting hardware-safety meta-policy).
• the tasks companion — step-by-step workflows for the in-scope task verbs: install (preconditions — DOCA install, CUDA Toolkit, GPU + NIC pair, OOB connectivity), configure (build-tree under doca/tools/gpunetio_ib_write_bw/ and the meson build wrapping the shipped DOCA), build (the meson setup + meson compile pattern from the public DOCA build documentation), modify (do not patch the shipped tool source; modify the invocation and the surrounding environment instead), run (smoke- before-bulk; client + server bring-up order; reading the per-iteration report), test (the eval loop — steady-state, NUMA placement, NIC saturation cross- check), debug (walk the error taxonomy layer by layer), use (how a BW result feeds a class-of- workload decision), plus a Deferred task verbs block routing out-of-scope questions.

The skill assumes a host where DOCA is already installed, a CUDA Toolkit matched to the install is present, and the operator has whatever privileges the public install profile expects for binding a doca_dev, a doca_gpu, and an OOB TCP socket.

What this skill deliberately does not ship

This skill is agent guidance, not a samples or scripts bundle. To keep the boundary clean, it deliberately does not contain — and pull requests should not add:

• Specific flag strings or expected throughput numbers beyond what the tool's shipped --help and main.c ARGP registration establish. The flag surface is small (device name, GPU PCIe address, GID index, server IP on the client side); the agent re-reads the binary's --help on the installed version before quoting flag strings. Throughput numbers are device-, firmware-, version-, and topology-specific.
• Pre-written DOCA GPUNetIO or CUDA kernel source code that would compete with the shipped tool tree. The shipped client/{main.c,kernel.cu,perftest.c,common.{c,h}} and server/{main.c,perftest.c,common.{c,h}} files are the verified worked example; the agent's job is to route the user there and prescribe minimum-diff modification per the universal modify-a-sample workflow in doca-programming-guide.
• Wrappers, parsers, or scripts in any language that consume the tool's stdout. The output format is small and documented in the capabilities companion; if the user wants to script against it, the right answer is "read the live source, write the parser against your installed binary".
• A samples/, bindings/, or reference/ subtree. This is a thin loader for a shipped tool tree; substantive material lives in the source tree and in the GPUNetIO library docs.

Loading order

1. Read this skill first to confirm the user's question is in scope (the user actually wants to measure sustained kernel-initiated WRITE BW through GPUNetIO, not learn GPUNetIO as a library or do a CPU-initiated measurement).
2. For what the tool measures, the surface-selection rule against the GPI sister tool and the CPU-initiated perftest, the throughput-decomposition guide, the version overlay, the error taxonomy, the observability surface, and the safety overlay, see the capabilities companion.
3. For step-by-step workflows — install, configure, build, modify, run, test, debug, use — see the tasks companion.

Related skills

• ../../libs/doca-gpunetio/SKILL.md — the library this tool wraps. The per-GPU doca_gpu context, the GPU-visible doca_gpu_eth_* and RDMA-side handles, the CUDA-side persistent-kernel pattern, the dual capability-discovery rule (DOCA cap-query AND cudaGetDeviceProperties), and the env preconditions (nvidia_peermem loaded, CUDA buffers registered with DOCA) live there.
• ../../libs/doca-rdma/SKILL.md — the underlying RDMA library. The RDMA queue this tool binds is created and connected via doca-rdma; the queue lifecycle, transport type (RC vs UC vs UD), permission matrix, and connection method are owned there.
• ../../libs/doca-verbs/SKILL.md — the raw-verbs escape hatch beneath doca-rdma / doca-gpunetio. This tool stays on the higher-level surfaces; doca-verbs is the right place only if the user needs a specific WR flag / QP attribute the GPUNetIO + RDMA surfaces do not expose.
• ../doca-gpunetio-ib-write-lat/SKILL.md — the latency analog of this tool. Same physical operation; same runtime framework; different metric class (BW vs latency). The two together carry the full GPUNetIO-side throughput / latency picture.
• doca-gpi — the GPI programming surface (CUDA-kernel-initiated RDMA), the alternative runtime framework for the same physical operation. doca/tools/ ships no GPI ib_write_lat / ib_write_bw benchmark binary, so the GPI comparison is against the library surface, not a sibling tool. The selection rule in the capabilities companion is the decision aid.
• doca-version — the canonical version-detection chain, four-way match rule, NGC container semantics, and headers-win-over-docs rule. The ## Version compatibility section in this skill is a thin overlay; the body lives there.
• doca-setup — env preparation, install verification, GPU + CUDA Toolkit pairing, nvidia_peermem load, hugepages, NUMA, and the I have no install yet path with the public NGC DOCA container.
• doca-public-knowledge-map — routing to the public DOCA documentation set (DOCA GPU NetIO, DOCA RDMA pages on docs.nvidia.com) and the docs.nvidia.com/cuda/ pointer for the CUDA Toolkit.
• doca-debug — the cross-cutting debug ladder. The tool surfaces its own error taxonomy; when the cause is below DOCA, the taxonomy hands off here.
• doca-hardware-safety — the bundle-wide hardware-safety meta-policy. The ## Safety policy overlay cross-links it.