GPU Cluster Evaluation

# 1. does the endpoint node think it is the host?
python -c "from torch.distributed.elastic.rendezvous.utils import _matches_machine_hostname as m; print(m('$MASTER_ADDR'))"
# 2. with is_host=1, does the store bind?           -> expect *:PORT
ss -ltnp | grep "$MASTER_PORT"
# 3. can a worker reach it?                          -> expect OPEN
timeout 5 bash -c "echo > /dev/tcp/$MASTER_ADDR/$MASTER_PORT" && echo OPEN
# 4. is the advertised master resolvable everywhere?
getent hosts "$(hostname)"      # 127.0.1.1 => you need --local_addr

nvidia-smi topo -m          # NV18 = 18 NVLink bonds; PHB = PCIe host bridge (no NVLink)
nvidia-smi nvlink -s        # per-link GBps; ×0.94 ≈ advertised (encoding)

ls /sys/class/infiniband/                 # mlx5_* IB/RoCE; rdmap* EFA
cat /sys/class/infiniband/*/ports/1/rate  # e.g. 200 Gb/sec (EFA) or 800 Gb/sec (CX-8)
rdma link
ibstat   # may print **nothing** on EFA — that is not “no fabric”; use rdma link + sysfs
nvidia-smi topo -m                        # NIC legend when present

./fio-scan "$NVME_PATH"
./fio-scan "$SHARED_PATH"

# Cluster eval: <name> (<N> nodes of <G>× <GPU>)

Example titles: `4 nodes of 8× H200`, `1 node of 8× B200`, `16 nodes of 8× H100`. Always include **node count and GPUs per node**, not only the GPU name.

- Datetime (UTC):
- Access:
- Nodes / GPUs:

Then a one-line pointer to the raw logs directory: `Raw logs: [\`raw-<name>/\`](raw-<name>/).`

## Table of Contents
Numbered list **before Environment**. Each item is **topic — tool/test that ran**, linking to that heading (GitHub-style `#anchor`: lowercase, punctuation stripped, spaces to `-`). Duplicate titles (`### Introduction`) get `-1`, `-2` suffixes; prefer linking the unique bench heading, not the intro. Only list tests that actually ran (plus inventory).

Compute TOC links are **purpose**, tools after the dash. Do **not** use DCGM / MAMF / Lemon GEMM as the TOC link text; those names live in the annotation or the section body. The `###` headings must match the TOC labels so the anchors agree.

The connectivity test has no section of its own — its result is item 1 of the Network `### Introduction`, so that TOC line links to `#introduction-1` (the second `### Introduction` in the document). Label it by the ranks it actually ran on: **Inter-node connectivity** only if it spanned every node, otherwise **Intra-node connectivity** with the rank count.

Network TOC: **Intra-node all-reduce benchmark** and **Inter-node all-reduce benchmark** — `all_reduce_bench.py`, then the rank count. Do not drop the word **benchmark**. Omit the inter-node line if 1 node.

Storage TOC: **Existing Partitions** — mount inventory. Then **Local disk IO benchmark** — `fio-scan` on `<local path>` and **Shared <product> IO benchmark** — `fio-scan` on `<shared path>` (same shape; path after `on`, never inside the link). Concurrent poke when ≥2 nodes: **Concurrent write shared fs benchmark** — `<N>-node 1 GiB O_DIRECT dd`.

Example:

```markdown
## Table of Contents

1. [Environment](#environment) — host, GPU, fabric, mounts, software (inventory, not a bench)
2. [Compute](#compute)
   1. [Hardware health](#hardware-health) — `dcgmi diag -r 2`
   2. [GPU performance benchmarks](#gpu-performance-benchmarks) — `mamf-finder-all-gpus.py`, one GPU at a time while the others run a matmul
   3. [Find underperforming GPUs](#find-underperforming-gpus) — fixed 16384³ `matmul` on every GPU
3. [Network](#network)
   1. [Inter-node connectivity](#…) — `torch-distributed-gpu-test.py`, R ranks (label it **Intra-node connectivity** if it only ran on one node)
   2. [Intra-node all-reduce benchmark](#intra-node-all-reduce-benchmark) — `all_reduce_bench.py`, 8 ranks
   3. [Inter-node all-reduce benchmark](#inter-node-all-reduce-benchmark) — `all_reduce_bench.py`, 32 ranks (omit this TOC line and the section if 1 node)
4. [Storage](#storage)
   1. [Existing Partitions](#existing-partitions) — mount inventory
   2. [Local disk IO benchmark](#local-disk-io-benchmark) — `fio-scan` on `/tmp`
   3. [Shared Lustre IO benchmark](#shared-lustre-io-benchmark) — `fio-scan` on `/code`
   4. [Concurrent write shared fs benchmark](#concurrent-write-shared-fs-benchmark) — 4-node 1 GiB `O_DIRECT` `dd` (omit if 1 node)
5. [Findings](#findings)
   1. [Healthy subsystems](#healthy-subsystems)
   2. [Underperforming subsystems](#underperforming-subsystems) (omit if nothing underperformed)
   3. [Needs operator input](#needs-operator-input) (omit if every target was known)
```

Rebuild the anchors from the headings you actually wrote. Omit TOC lines for phases that did not run — no placeholder line saying they are missing.

## Environment
Five small tables — **Host / GPU / Fabric / Filesystems / Software**. Do not pile every field into one table. The third and fourth are **Fabric** and **Filesystems**, never “Network” / “Storage”: those would take the `#network` / `#storage` anchors the top-level sections need.

### Host
**First row: Ubuntu** (`PRETTY_NAME`, e.g. 24.04.2 LTS). Then hostname, kernel (`uname`), CPU cores/NUMA, host RAM, PCIe gen×width (`pcie.link.gen` / `width` from nvidia-smi — board/host path, not a GPU property). Not Python.

### GPU
name × count, SM arch / SM count, HBM in **GiB only** (not `275040 MiB (268 GiB)`), driver / CUDA, **three** SM clocks — boost (`clocks.max.sm`), saturated (~TDP under dense GEMM), parked idle — power limit / idle draw. Do not merge boost and “loaded”; a dense GEMM is not at boost. No MAMF/MSMF names here. PCIe does **not** go here.

### Fabric
intra-node fabric + spec uni GBps; inter-node adapter + rate as **inventory** (`ibstat` / `rdma link`).

### Filesystems
Left column **Local** / **Shared network FS** — do not repeat “local” or “ceiling” in the value. Local value: mount + size; if the backing device is Linux md, write `md127 RAID` (or whatever `mdN`), not a bare `md127`. Shared: guest-visible type + path + size; add the vendor product name when it is not the same as `df -T`. **Capacity only — never how much is used or free.**

### Software
Python version, torch, NCCL (`torch.cuda.nccl.version()`). Note that torch was the newest release at eval time. **No eval venv path.** Isolated venv is how you run, not a report field.


## Compute

### Introduction
Name × count, SM count, HBM GiB, TDP, official BF16 TFLOPS. Not a benchmark dump. Then a **numbered list of the passes that follow**, each saying what the tool literally does:

1. `dcgmi diag -r 2` — NVIDIA's own health check (software, memory, PCIe).
2. `mamf-finder-all-gpus.py` — times many **matmul shapes** (M×N×K) on GPU0 to find the fastest, then measures the sustainable winner on every other GPU, each while all the others run a matmul; reports **MAMF** and **MSMF**, both vs the official spec.
3. Lemon GEMM — a throwaway inline script: one fixed matmul shape (M=N=K=16384) on every GPU, to catch a dead, slow, or throttled card.

### Hardware health
**Pass** (or **Fail** on named plugins/GPUs; or Gaps: no hostengine / unprivileged). Never **Pass** and **No Fail** together.

### GPU performance benchmarks
Official TFLOPS source = [TFLOPS comparison table](https://github.com/stas00/ml-engineering/blob/master/compute/accelerator/README.md#tflops-comparison-table), not a relative path. Open that section by explaining the measurement before any number: compute is measured by timing matrix multiplication; how fast a `matmul` runs depends on its **matmul shape** (the M×N×K dimensions), so the tool searches. State how many matmul shapes GPU0's search timed (the `Tried N shapes` line in `mamf/gpu0.txt`), and that every other GPU then measured GPU0's two winning shapes while all the rest of the node ran a continuous matmul. Then `Two numbers vs official BF16 <N> TFLOPS on <GPU>:` (name the GPU; do not leave the spec floating), then two bullets, each with the measured W/MHz the script printed:

- **MAMF** (Maximum Achievable Matmul FLOPS) — highest TFLOPS a short burst at boost reaches, with the board far under its power limit, measured on every GPU at GPU0's MAMF shape; the node figure is the median GPU. A ceiling, not a rate any sustained workload holds. One number, with its shape and % spec.
- **MSMF** (Maximum Sustainable Matmul FLOPS) — best that survives once the matmul shape is large enough to pin the board at the power limit and the clock settles, measured on every GPU while all the others compute. The node figure is the median GPU, so one weak or strong chip doesn't set it; synchronous work runs at the slowest GPU, which the summary also gives. Compare a sustained workload to the median, and a synchronous job's ceiling to the slowest.

Then the per-GPU table from `mamf/summary.txt`, and below it the node MAMF and MSMF (median GPU), the slowest GPU and the spread:

| GPU | MAMF | MSMF | MSMF shape | W / MHz | MSMF % spec |

TFLOPS are integers, as the script prints them; compute % spec from the integer. On a 1-GPU box there is no "while the others compute" and no median or spread; say it is the only GPU.

**Expand every acronym at first use in the report** — MAMF, MSMF, DCGM, NVLS, GEMM, busbw/algbw, MFU. The Terms table in this skill is for the agent; the reader of the report only sees the report.

**The eval does not know what the cluster is for.** Never assume training — the same allocation may be bought for inference, fine-tuning, or batch jobs. Write "sustained workload", not "training step"; say "training or inference" when you need an example.

After the bullets, say what the table's shape and `W/MHz` columns are: the M×N×K that won each regime, and the board power / SM clock measured while it ran.

### Find underperforming GPUs
**TLDR:** `no under-performing or dead GPUs` — name both outcomes, since a dead card and a slow one are different findings. If something is wrong, say which: `under-performing: <host>(node<I>):gpu<N>` or `dead: <host>(node<I>):gpu<N>`. Then define an under-performing GPU (slow or throttled; dead is the extreme case) and the pass: the **same** matmul shape (M=N=K=16384) on every GPU, siblings idle, flag a card well below its node's median — explicitly **not** MAMF (no search over matmul shapes, no boost/sustain split). Then 16384³ SM coverage. Table: all GPU scores per node you have, min (gpu), max. Do **not** prefix the heading with node count — ranks belong in the TOC annotation, not the heading.

## Network

### Introduction
Open with a **numbered list of tools** (same shape as Compute and Storage), each explained before any number: GitHub-link [`torch-distributed-gpu-test.py`](https://github.com/stas00/ml-engineering/blob/master/debug/torch-distributed-gpu-test.py) (connectivity — one process per GPU; ranks see each other and complete a collective; say how many ranks and over how many nodes it ran, and do not call a 1-node run inter-node) and [`all_reduce_bench.py`](https://github.com/stas00/ml-engineering/blob/master/network/benchmarks/all_reduce_bench.py) — say what it sweeps (payloads from 32 KiB to 16 GiB) and define `busbw` (bus bandwidth, what the fabric moves) and `algbw` (algorithm bandwidth, what the caller sees) **there**, plus which rank counts it ran at. Do not write “the book” or “MLE”. Then NVLS (see Terms): in-network **all-reduce** on the NVSwitch (~30% intra / ~25% inter vs ring `busbw`); does **not** help other collectives. InfiniBand switch SHARP is a different feature. Record `NCCL_NVLS_ENABLE`: **2** = auto — the eval default if unset; **0** = force off; **1** = require NVLS (can abort if unavailable). State whether NVLS was available for intra and for inter (`NCCL_DEBUG` when you have it). Name who set the env (container image / cluster), not that you “kept” it.

### Intra-node all-reduce benchmark
Intra-node: transport (e.g. NVLink 5, NV18), spec uni GBps, busbw table, peak % spec (note NVLS if on).
**Inline** `![Intra-node all-reduce](raw/<cluster>/busbw-mean-…-8.png)` next to that section.
### Inter-node all-reduce benchmark
Inter-node **only if ≥2 nodes**: transport, NIC inventory GBps, table, peak busbw vs 1-node at the same payload, NCCL path (not % of NIC spec). Do not name the launcher. Rank/node counts go in the TOC annotation. **1 node: delete this subsection and its TOC entry.**
**Inline** `![Inter-node all-reduce](raw/<cluster>/busbw-mean-…-R.png)` when the plot exists.
### Gaps
(only if needed, e.g. rdzv failed on ≥2 nodes — not “only 1 node”)

## Storage

### Existing Partitions
Inventory table: **always a `/` row**, then one row per **distinct** volume. Same backing store mounted twice (`/tmp` on `/`, bind-mount aliases) is still one row. No benchmarking comments. Do not name a network FS this allocation does not use. Shared facts go after the table.

| mount | size | type | device / server |
| --- | ---: | --- | --- |
| `/` | 28T | xfs | md127 RAID0 (8× ~3.5T NVMe) |
| `/code` | 2.3T | Lustre | `10.4.138.63@tcp:/xmym3bev` |

### Introduction
Same shape as Compute / Network: **numbered list of passes** (tool + path), not a mount inventory. GitHub-link `fio-scan` on the first item and explain what it literally does **there** — six `fio` runs against one directory — file sizes **16 KiB, 1 MiB and 1 GiB (correspondingly small, medium and large)**, each read and write — 16 parallel jobs, 4 KiB blocks, `libaio`, `O_DIRECT` (and say what `O_DIRECT` buys: bypasses the page cache so the numbers are the filesystem's, not RAM's), 3 min of timed IO each.

**Write IEC units, never `fio`'s shorthand.** The script prints `16k` / `1m` / `1g`; the report says **16 KiB** / **1 MiB** / **1 GiB**, in the `size` column of both tables and in every sentence that cites a row. `16k` is not a valid unit. Later mentions are bare. Then the table units note. All “we scanned X” comments live here.

Two tables, **same columns and same IOPS scale**:

| size | rw | latency (msec) | bw (MiBps) | IOPS (M) |
| ---: | --- | ---: | ---: | ---: |
| 16 KiB | read | … | 12_029.1 | 3.08 |
| … | … | … | 631.8 | 0.157 |

### Local disk IO benchmark
…
### Shared <product> IO benchmark
| size | rw | latency (msec) | bw (MiBps) | IOPS (M) | local/shared × |

Heading is **Shared <product> IO benchmark** (e.g. Shared Lustre IO benchmark), not a bare “Shared”. Optional extra when ≥2 nodes: **Concurrent write shared fs benchmark**. Omit on 1 node.

Local NVMe vs shared network FS (16 KiB / 1 MiB / 1 GiB, read+write). bw = MiBps (`_` on the fast table). IOPS = 10⁶ in every table. **`size` is right-aligned** (`---:`), like every other numeric column; only `rw` is left-aligned. Each shared table appends **`local/shared ×`** at the **end** (local bandwidth ÷ shared bandwidth) so the first five columns match local. Explain the column in a **sentence**, not a chain of equations: what it divides and what 1 would mean. Do not add an IOPS ratio: fixed 4 KiB blocks make it identical to the bandwidth ratio. **Quote table values verbatim** in prose — if the column says `26.5`, write `26.5×`, never `27×`. Do not express the comparison as a percentage, do not call it `%local`, and say "the local disk table above", not a device name. Raw unscaled numbers in `raw/`.
(omit Gaps unless something is missing)

## Findings

Up to three headings, in this order — **### Healthy subsystems**, **### Underperforming subsystems**, **### Needs operator input** (the noun is required; never a bare `### Healthy`). Include only the ones that have bullets. Bullet labels are the subsystem (GPUs, fabric, local disk, shared network FS, torchrun) — **never prefix with A./B./C.** (the report sections are Compute, Network, Storage). Split mixed areas (e.g. local disk vs shared FS). Do not put an underperforming item in Healthy with a caveat, and do not hide it in a Flags list that looks like the healthy bullets.

Healthy subsystems: fabric and GPUs that met spec. Network headline is **fabric** (intra vs NVLink spec + NVLS; inter busbw + path + vs 1-node all-reduce) — **never** “X% of NIC/rail,” and **not** a launcher. Do not credit DeepSpeed / torchrun / srun in busbw or Findings.

**Under-performing and dead GPUs** get one bullet each under Underperforming subsystems, from either GPU pass. Name each as `<host>(node<I>):gpu<N>`, e.g. `blah-176(node0):gpu7`: `<host>` is the hostname the Environment Host table uses, which an operator needs to drain or replace the card; `node<I>` is the node's 0-based position in the hostfile (the order the user gave the nodes), which stays readable when hostnames are cryptic; `<N>` is the GPU index `nvidia-smi` shows. Follow it with the pass that found it and its numbers against its node's median, e.g. "`blah-176(node0):gpu7` — MSMF 701 / 698 / 704 TFLOPS in three passes, 6.8–7.2% below the node median each time". List every one; never summarize as "two slow GPUs".

Underperforming subsystems: measured vs **vendor advertised** for **this** product; not good enough for code (16 KiB) / checkpoint (1 GiB → 1 TiB) / dataloader; torchrun/rdzv if it failed; leftover Gaps. A broken launcher is **not** “the network is unhealthy.” Do not name a network FS this allocation does not use.

**Needs operator input: a verdict requires a target, so a measurement without one goes here, not under Underperforming.** Calling a subsystem slow when you have nothing to compare it against is an opinion dressed as a finding — the reader cannot act on it and the operator can dismiss it. This heading is for a subsystem that was measured cleanly but whose expected figures only the operator has: a shared FS with no published provisioned throughput, a fabric whose purchased tier is unknown, any device whose spec sheet does not exist publicly. Open with one sentence saying these cannot be called healthy or underperforming until those figures arrive. Each bullet gives the measurements in full, then **what to obtain**.

**Write targets in the plural — a filesystem never has one.** Every file size and operation has its own rate, reads and writes differ, so it is "the figures they are supposed to hit" and "no published performance figures", never the singular. The singular quietly promises the reader one number to compare against, which is the mistake the whole category exists to avoid.

**Ask only about the mount you measured, and name its size.** Expected rates are per filesystem, so asking for "`/code` and `/data`" when six numbers came from `/code` alone invites an answer you cannot use. State which mount the figures describe, and if sibling mounts differ in size, say that each has its own target and needs its own scan — pointing at the one the workload will lean on hardest.

**Ask for the targets in the same shape as your measurement, or they will not settle anything.** "The provisioned throughput" is not askable — a parallel FS has no single rate, and a per-TiB headline number cannot be checked against a row of the table. Ask for **one expected value per measured cell, in the units you measured, under the conditions you measured**: for `fio-scan` that is bandwidth in MiB/s **or** IOPS at 16 KiB, 1 MiB and 1 GiB, read and write, from one client with 16 jobs, 4 KiB blocks and `O_DIRECT`. Either unit is enough because the fixed block size makes one derivable from the other. Say those conditions in the ask, because the same filesystem answers differently per client and in aggregate, and because small-file and large-file rates come from different limits — one headline figure hides both. Then add the conditions that would invalidate a comparison if the operator's figures assume something else: per-client vs per-filesystem, and stripe count. A comparison against another subsystem you did measure (shared FS vs local disk, `26.5×` slower) is a fact and belongs in the bullet; it is not a substitute for the target, because local NVMe was never what the shared FS was sold to match.

Judge before you place a bullet: is there a number this is supposed to hit? Official TFLOPS, NVLink spec GBps and a vendor-advertised FS figure are targets, so those bullets can be Healthy or Underperforming. If the answer is "the operator would have to tell me", the bullet goes in **Needs operator input** even when the measurement looks disappointing.

**Quote the whole measurement, compactly, not one cherry-picked row.** A storage bullet has six numbers behind it, so give all six in one line — `read / write: 16 KiB **a MiB/s / b MiB/s**, 1 MiB **c MiB/s / d MiB/s**, 1 GiB **e MiB/s / f MiB/s**` — instead of a single size, which leaves the reader asking what the others were. Same for any bench that swept a parameter. Compact, not partial.

**Every number outside a table carries its unit.** Putting `MiB/s` once in the lead-in and then listing bare `1_646.1 / 1_658.3` is a table habit; in prose each value needs its abbreviation — `1_646.1 MiB/s / 1_658.3 MiB/s`. `read / write` once is fine. Same for GBps, TFLOPS, msec, ×. The table header is the one place a unit may appear only once.

**When vendor figures are missing, say the targets are unknown, say why none are published, and ask for them — but do not narrate the search.** Three things, because a measurement with no target cannot be called a shortfall, and a reader who knows the filesystem's name will reasonably assume a spec sheet exists: (1) state plainly that the provisioned figures are unknown and the measured numbers therefore have nothing to be judged against; (2) explain in a sentence why — for a parallel FS, **the software publishes no performance figures**; Lustre, GPFS/Storage Scale, BeeGFS and NFS are filesystem software, and throughput is a property of the deployment (count of object storage targets / servers, their backing devices, the network to them, per-file stripe count), so the number that exists is the **service tier the operator purchased**, typically sold per TiB of capacity (managed Lustre commonly 125–1000 MB/s per TiB); (3) ask the user for that tier **and** the default stripe layout, since a single-stripe file is served by one storage target regardless of filesystem size and is the usual explanation for a single client seeing a few hundred MiB/s. What to leave out is the search: where you looked, what `df` or the mount table did not contain. A mount was never going to state IO speed, so ruling it out tells the reader nothing and reads as an excuse.

**Say who is slowed and by how much, in literal terms.** Hardware and files do not "feel", "suffer", "struggle", "get punished", or "care"; a filesystem is not "painful". Name the concrete operation and attach the measured rate to it — "a `git clone` or `pip install` on `/code` runs at the 16 KiB read rate of 204.7 MiB/s" — so the reader can check the claim against the table instead of trusting an adjective.

If nothing underperformed, **omit that heading and its TOC line** — never write `Underperforming subsystems: none`. Same for **Needs operator input** when every target was known. No extra **Flags** section.