Tune NVSHMEM NIC-to-PE Mapping

Purpose

Produce a reproducible recommendation from the target node's actual GPU/NIC topology. Do not promise a performance improvement; present the mapping as a topology-informed starting point that must be validated.

Requirements

• Use an allocated target compute node with visible GPUs and RDMA devices; do not use login-node topology as evidence.
• Make nvidia-smi and /sys/class/infiniband readable on that node. If /sys is unavailable, make ibv_devinfo available or provide equivalent per-port evidence.
• Provide the NVSHMEM installation prefix, make nvshmem-info available on the target node, or supply the installed NVSHMEM version explicitly.
• Use the bundled read-only collector when possible. It requires no API keys, credentials, package installation, or writable filesystem access.

Inputs

Resolve every required input before emitting exact exports. If values from different sources conflict, identify the conflict and request clarification rather than selecting a value silently.

Required inputs

• NVSHMEM version: Prefer the user-confirmed installed version; otherwise use the collector's [nvshmem-version] output, then nvshmem-info -n on the target node. Retain the complete reported version after normalizing a leading v.
• Remote transport: Prefer the explicitly selected transport for the target run; otherwise use the target node's NVSHMEM initialization log. Do not infer a transport from topology; request a log when automatic selection is ambiguous.
• Local PE count: Prefer observed per-rank logs, then launcher configuration, then an explicit user-provided count. Confirm that all evidence describes the same node allocation.
• Ordered local PE -> GPU index or PCI BDF binding: Prefer observed per-rank binding logs, then launcher binding configuration, then an explicit user mapping. Use CUDA_VISIBLE_DEVICES only when its rank order is explicitly established.
• Target-node GPU/NIC and HCA-port topology: Prefer the newest complete output from scripts/collect-nic-topology.sh on the target node. Otherwise use the listed nvidia-smi commands plus per-port /sys/class/infiniband or ibv_devinfo evidence. Include GPU and NIC PCI BDFs, port state, and link layer.

Optional inputs

• Requested link layer: Use the user's explicit InfiniBand or Ethernet/RoCE preference. If absent, choose from observed active ports as described in step 5.
• Excess-port policy: Use the user's explicit request to select a closest subset or use multiple NICs per PE. If absent, present supported alternatives conditionally and make no selection.
• Current official documentation or citations: Obtain only when the user requests them or the version/transport matrix is insufficient. Use $nvshmem-docs for that check.

Scripts

Script	Use	Invocation
scripts/collect-nic-topology.sh	Collect read-only NVSHMEM version, GPU PCI, GPU/NIC topology, and RDMA-port evidence. It emits all evidence to stdout and may exit nonzero after producing useful partial output.	Resolve the script from this skill directory and invoke run_script("scripts/collect-nic-topology.sh", "--prefix", "/absolute/nvshmem/prefix") when the prefix is known, or omit --prefix to use environment/PATH discovery.

Read the Required References

• Read references/version-transport-matrix.md before interpreting or emitting any mapping variable.
• Read references/topology-and-output.md before ranking NICs or formatting a recommendation.
• Treat the bundled NVSHMEM 3.8 or later BLOCK contract as authoritative within its recorded transport scope. Do not invoke $nvshmem-docs merely to revalidate behavior that this contract resolves, including for releases later than 3.8.

Instructions

1. Resolve the NVSHMEM Version and Transport

Obtain the version from the user first. Otherwise use the collector's [nvshmem-version] evidence or run nvshmem-info -n when it is already available on the target node. Normalize a leading v and retain the complete reported version.

Classify the release as:

• 3.7 or earlier;
• 3.8 or later.

Do not assume latest or silently treat an unknown version as 3.8. If the version remains unknown, request the installed version before continuing.

Determine the selected remote transport: IBRC, IBDevX, IBGDA, GPUNetIO, UCX, or libfabric. If automatic selection makes the transport ambiguous, ask for the relevant initialization log or the intended transport.

2. Resolve Documentation Uncertainty

Use references/version-transport-matrix.md as the default authority for mapping behavior. Do not invoke $nvshmem-docs when the matrix or the bundled NVSHMEM 3.8 or later BLOCK contract resolves the version, transport, feature gate, and required mapping controls, including for a release later than 3.8.

Invoke the sibling $nvshmem-docs skill at ../nvshmem-docs/SKILL.md only when the matrix and bundled contract are insufficient or conflicting, exact feature-version applicability remains unclear, or the user explicitly requests live official documentation or citations. Follow that skill's version and citation workflow.

If a documentation check is needed but the sibling skill, an HTTPS retrieval tool, or the live official pages are unavailable:

• continue from the matrix only when it resolves the needed behavior;
• otherwise withhold final exports and identify the unresolved behavior;
• label Documentation freshness: unverified only when a documentation check was attempted;
• state which official routes could not be checked; and
• never invent a behavior that the matrix does not establish.

When a documentation check is performed, report the exact documentation edition consulted and any remaining uncertainty. Otherwise identify the applicable matrix section as the basis for the recommendation.

3. Collect Topology Evidence

When already running on the target compute node, tell the user that the skill is running a read-only collector. Before its first invocation in the current checkout, inspect the complete bundled collector script. Run only that inspected file; if it is unavailable or incomplete, do not execute it. Then run:

run_script("scripts/collect-nic-topology.sh")

When the NVSHMEM installation prefix is known, pass its absolute path explicitly:

run_script("scripts/collect-nic-topology.sh", "--prefix", "/opt/nvshmem")

Replace the example path with the absolute prefix. Without --prefix, discovery uses NVSHMEM_PREFIX, then nvshmem-info on PATH. Prefix libraries are scoped to that query. Exit 2 indicates incomplete evidence.

When not on the target node, ask the user for the collector's complete output from the target node. If the script is unavailable remotely, request all of:

nvshmem-info -n
nvidia-smi --query-gpu=index,pci.bus_id,name --format=csv,noheader
nvidia-smi topo -m

Also request, for every candidate HCA port, its HCA name, port number, PCI BDF, state, and link layer from /sys/class/infiniband or ibv_devinfo.

Do not run a collector from a login node and treat that topology as compute-node evidence.

4. Establish PE Placement

Obtain the number of local PEs per node. Assume nodes are homogeneous only after stating that assumption.

Obtain the ordered local PE -> GPU index or PCI BDF binding from explicit user input, launcher configuration, CUDA_VISIBLE_DEVICES, or per-rank logs. Do not infer a local PE's GPU solely from PE count or the order displayed by nvidia-smi topo -m.

If the binding is ambiguous, request it before producing an exact closest-NIC mapping. A conditional table is acceptable, but final exports are not. Do not bypass the missing binding by duplicating every usable NIC for every PE; that adds non-closest paths and is a different multi-port policy. Consider such an all-ports assignment only after the user explicitly requests it and accepts the distance tradeoff.

5. Filter and Rank Candidate Ports

Treat an HCA port, not merely an HCA device, as the selectable NIC unit.

• Exclude inactive or physically down ports and record why each was excluded.
• Partition active ports by link layer. Do not mix InfiniBand with Ethernet/RoCE by default.
• Honor an explicitly requested link layer. Otherwise choose the larger active partition; prefer InfiniBand on an exact tie. Ask the user if that default conflicts with their fabric or provider intent.
• Rank each bound GPU-to-port pair by PIX < PXB < PHB < NODE < SYS from nvidia-smi topo -m.
• Break equal-distance choices by lower assigned-port load, then canonical hca:port order.
• Share a port when fewer usable ports than PEs makes sharing necessary, while preserving distance first and balance second.

6. Resolve Excess NICs Explicitly

When usable NIC ports outnumber local PEs, stop before final exports and ask whether to:

1. select the closest balanced subset; or
2. use multiple NICs per PE.

Offer option 2 only when the resolved version and transport support it according to the matrix. Explain whether multi-NIC requires automatic distance assignment or supports an explicit BLOCK mapping. If the user does not answer, show both supported alternatives as conditional recommendations and do not choose silently.

If the user asked to use all NICs but the resolved version/transport does not support multi-NIC, explain the restriction and request approval to use the closest subset. Do not treat the unsupported request as implicit approval for a subset, and do not emit final exports until the user confirms it. When only the subset is supported, present that single supported choice for confirmation instead of offering a disabled multi-NIC option.

7. Synthesize Exactly One Selection Mechanism

Prefer NVSHMEM_HCA_PE_MAPPING for exact assignments. Use NVSHMEM_HCA_LIST only when selecting an eligible set and accepting transport-canonical ordering. Never set both variables.

Apply the version/transport rules in the matrix exactly:

• For 3.7 and earlier, preserve one expanded mapping slot per PE for explicit IB mappings. Use automatic distance assignment for supported multi-NIC paths.
• For 3.8 or later, apply the bundled BLOCK contract: set NVSHMEM_ENABLE_NIC_PE_MAPPING=1 for explicit assignments and NVSHMEM_ENABLE_MULTI_PORT=1 whenever recommending more than one NIC per PE.
• For UCX and libfabric, explain the provider-controlled boundary and do not fabricate common-HCA exact mappings.

Build NVSHMEM_HCA_PE_MAPPING with explicit hca:port:count entries. Expand every count before explaining which slots each PE receives. Preserve duplicates when intentional sharing is required.

Keep a mapping-only export block limited to NIC-assignment controls. Do not add NVSHMEM_REMOTE_TRANSPORT or provider/launcher settings merely because the transport is known; include them only when the user separately asks to select the transport and the matrix, or documentation when needed, verifies the value.

8. Return the Recommendation

Execute the Output Contract in order; that reference is the single authority for output fields, formatting, withholding messages, and validation guidance.

Treat its Mapping and Exact Exports sections as mandatory gates. Do not return until each gate is either completed or explicitly withheld for the reason allowed by the contract. Do not duplicate or paraphrase the contract in this file.

Examples

• “My 4 local PEs are bound to GPUs 0–3. Given this target-node collector output and NVSHMEM 3.8.0 IBRC, recommend the closest balanced InfiniBand port mapping and exact exports.”
• “Interpret this nvidia-smi topo -m and per-port state/link-layer output. NVSHMEM is 3.7 with IBGDA; identify whether multiple NICs per PE are supported and what extra evidence is needed before emitting exports.”
• “We have 2 PEs and 5 active Ethernet/RoCE ports. Use this PE-to-GPU binding and initialization log to show the supported subset and multi-NIC alternatives without selecting one silently.”

Troubleshooting

Symptom	Response
The collector reports status=partial, an unknown NVSHMEM version, missing GPUs, or no HCA ports.	Re-run it on an allocated target compute node with the intended NVSHMEM installation and NVIDIA/RDMA device visibility. Paste the complete labeled output; a nonzero exit may still contain usable evidence.
No exact mapping can be produced.	Obtain the NVSHMEM version, resolved remote transport, local PE-to-GPU binding, and active-port evidence. Give only conditional guidance until all are available.
Ports have mixed link layers or a requested port is inactive.	Exclude inactive ports and select one link-layer partition unless the user explicitly requests another supported fabric.
NVSHMEM behavior is unclear for the selected version or transport.	Check the version/transport matrix first, then invoke $nvshmem-docs only if it remains insufficient or the user requires current official citations.
Logs select unexpected ports after applying the mapping.	Verify the export block, PE binding, selected transport, and provider constraints; compare against the unforced automatic configuration before changing other settings.

Limitations

• Limit recommendations to the observed target-node topology and port state; repeat evidence collection after hardware, fabric, launcher, or binding changes.
• Treat topology proximity as a selection heuristic, not a performance guarantee; validate with a representative workload.
• Do not produce an exact mapping until the NVSHMEM version, transport, local PE-to-GPU binding, and usable-port evidence are resolved.
• For UCX and libfabric, remain within the provider-controlled boundary and do not fabricate common-HCA mapping settings.

Guardrails

• Do not select inactive ports or silently mix link layers.
• Do not assume that a numerically matching GPU and NIC index implies proximity.
• Do not turn an unknown PE-to-GPU binding into an all-ports-per-PE mapping.
• Do not emit 3.8-or-later BLOCK behavior for 3.7 or earlier.
• Do not emit legacy IBGDA multi-port controls as the primary 3.8-or-later control.
• Do not generate provider-native UCX or libfabric settings; explain them without inventing values.
• Do not overwrite user files or change the system while collecting evidence.

<!-- ci trigger -->