Coreweave Fabric Diagnostics

CoreWeave Fabric Diagnostics

Community-contributed. Not affiliated with, endorsed by, or sponsored by CoreWeave, Inc. CoreWeave is a registered trademark of CoreWeave, Inc.

Detects when a CoreWeave multi-node GPU job has silently fallen off the InfiniBand fabric onto TCP — the failure that makes distributed training run at a fraction of the hardware's speed while every GPU keeps billing at the full rate — and gives the exact fix.

Overview

On a CoreWeave multi-node job, NCCL should carry collectives over InfiniBand with GPUDirect RDMA (NET/IB). If any one of three conditions is missing, NCCL silently falls back to TCP sockets (NET/Socket): the job still runs, still converges, and raises no error — but all-reduce throughput collapses (commonly cited as 5-20x slower [nccl]) because it now crosses the Ethernet control plane instead of the 400 Gb/s-class fabric. You keep paying full GPU rate for a multi-node run that performs like a badly-connected one. This is the single highest-dollar invisible failure on the platform, and nothing in the default output flags it.

The diagnosis is deterministic: the bundled scripts/fabric-check.py greps the pasted NCCL_DEBUG=INFO log for the decisive Using network line (and NET/IB vs NET/Socket), parses the pod-spec's resources block for the RDMA device request, reads ibstat port state, and echoes any all_reduce_perf bus bandwidth — then emits a VERDICT with the rdma_engaged/transport call, the missing conditions, and the fix. The LLM never eyeballs which transport is in use; the script decides. Deep grounding lives in references/, loaded only when a leg of the diagnosis needs it.

Prerequisites

• An NCCL_DEBUG=INFO log from the actual run — the primary signal. Re-run the job (or one rank) with NCCL_DEBUG=INFO set and capture stderr. The decisive line is Using network IB (good) vs Using network Socket (the fallback). This is the one input the skill really needs; everything else corroborates.
• Optional, for a full diagnosis: the pod/job spec (kubectl get pod NAME -o yaml) to check the RDMA device request; ibstat output from the node for port health; and all_reduce_perf results from CoreWeave's nccl-tests to measure bus bandwidth.
• python3 to run the deterministic checker (stdlib only).
• kubectl (read-only) if corroborating the live pod spec / node cordon state.

Authentication. Nothing secret is read. If the pod spec is pulled live, kubectl uses the existing $KUBECONFIG; the skill only ever runs kubectl get (read-only) — it never cordons, drains, or applies.

The three required conditions (all must hold, or NCCL falls back)

1. The RDMA device is requested in BOTH resources.requests AND resources.limits (rdma/ib: 1). If it is in only one — or absent — the device plugin does not inject the IB device into the pod and NCCL never sees a HCA. [unverified — the exact resource key (e.g. rdma/ib) depends on the installed RDMA device-plugin config; confirm with kubectl describe node / kubectl get node -o yaml.]
2. NCCL_IB_HCA=ibp and NCCL_SOCKET_IFNAME=eth0 are set (CoreWeave's documented values [cw]) — unless you launch via the MPI Operator, which manages this network config for you [nt].
3. NCCL_DEBUG=INFO then confirms NET/IB (ideally a GPU Direct RDMA Enabled line). If it shows NET/Socket / Using network Socket, RDMA is not engaged.

Full checklist with verification commands: references/rdma-engagement-checklist.md.

Instructions

The pipeline is gather → verdict → fix → confirm. The script does the transport call; references/ carry the grounding:

1. Gather the NCCL_DEBUG=INFO log (required) plus any pod-spec / ibstat / all_reduce_perf output you have. Concatenate them into one paste — the checker keys on each signal independently.
2. Run the deterministic checker to get the VERDICT.
3. If the verdict is fallback (Socket), apply the three-condition fix and re-run.
4. If the verdict is IB but degraded, chase the degraded signal (port down / low busbw).
5. On NVSwitch systems with a stuck fabric, use the Fabric Manager reset order.

Step 1: Gather the evidence

The log is the load-bearing input. If the user has not run with NCCL_DEBUG=INFO, tell them to — without it, transport selection is unknowable. To pull the live pod spec:

kubectl get pod "$POD" -o yaml > pod.yaml

Step 2: Run the deterministic checker (it makes the call, not the model)

Pipe everything you gathered to fabric-check.py. It greps for the decisive Using network line, the resources block, ibstat state, and any Avg bus bandwidth:

cat nccl-debug.log pod.yaml ibstat.txt allreduce.txt 2>/dev/null | \
  python3 scripts/fabric-check.py

The verdict names rdma_engaged (yes/no/partial/unknown), the transport in use, the missing conditions, and the fix. Use --json to capture the structured result for further processing. Reading the log by eye is what this step exists to prevent — see references/nccl-debug-reading.md for what each line means.

Use Glob to gather multiple pasted log files when a run spans several ranks, Write the verdict report to the working directory, and Edit it to refine the fix as the user iterates on the manifest.

Step 3: If the verdict is fallback (NET/Socket) — apply the three-condition fix

This is the money case. Fix in order (the checker prints the same list):

1. Add rdma/ib: 1 to both resources.requests and resources.limits.
2. Set NCCL_IB_HCA=ibp and NCCL_SOCKET_IFNAME=eth0 (or launch via the MPI Operator).
3. Re-run with NCCL_DEBUG=INFO and confirm the log now shows NET/IB + GPU Direct RDMA Enabled, not NET/Socket.

If the log shows NCCL_IB_DISABLE=1, that alone forces sockets — set it to 0 (RoCE and IB both need the IB verbs transport enabled [env]).