GPU cluster commissioning: the checklist before you hand it to the AI team

Every GPU cluster eventually reaches the same moment: the racks are in, the lights are green, and someone asks whether the AI team can start. Saying yes too early is the most expensive mistake in this work. A single weak link, a GPU with a marginal memory error or a cable seated half a millimetre short, does not stop a cluster from booting. It quietly slows every training job, or fails one at hour 40. Commissioning is the discipline of finding those problems before anyone depends on the cluster.

This is the checklist we work from. It is deliberately vendor-neutral; your hardware vendor's own acceptance procedure always takes precedence where they differ.

1. Before the hardware arrives: the site

Power: confirm the real per-rack draw at full load, not the nameplate average. Current rack-scale systems are far beyond what most enterprise halls were built for; NVIDIA's GB200 NVL72, for example, draws roughly 120–132 kW per rack according to Supermicro's and Gigabyte's published specifications. Check that every rack's feeds, breakers and PDUs are sized for that, and that losing one feed does not trip the other.

Cooling: if the systems are liquid-cooled, the facility-water loop must be flushed, pressure-tested and running at the vendor's specified temperature and flow before any server is connected. If they are air-cooled, confirm the hall can hold inlet temperatures inside the vendor's range at full load — not just at idle.

Floor and space: rack-scale systems are heavy. Confirm floor loading, the delivery path from dock to hall (lifts, ramps, door widths) and service clearances front and rear.

2. Physical installation

Label every cable at both ends against a cabling plan before power-on. Most fabric faults found later are cabling faults: a crossed pair of links, a cable on the wrong rail, a connector not fully seated. Photograph each rack's rear when it is complete; it is the cheapest troubleshooting record you will ever create.

3. A known baseline: firmware and software

Before testing anything, make every node identical: BIOS, BMC, GPU, NIC and switch firmware at the versions the vendor validates together, then one operating-system image, driver and CUDA stack across the cluster. Record the versions. A cluster with two firmware versions is two clusters, and intermittent faults love the gap between them.

4. Burn-in, node by node

NVIDIA's Data Center GPU Manager (DCGM) includes diagnostics at increasing depth. The quickest level takes seconds and suits a health check before each job; NVIDIA's documentation puts the medium level at under about ten and a half minutes on an eight-GPU system, and the deeper levels run considerably longer. For acceptance, run the deepest level on every node, then keep the nodes under sustained full load for hours, logging GPU temperatures, clocks, power and memory errors throughout.

What you are looking for is outliers, not failures alone: one GPU running hotter than its neighbours, clocking lower, or logging correctable memory errors that the others do not. Those are the parts that fail in production. Replace or reseat them now, while the vendor is still on site.

A cluster is only as fast as its slowest link — and the slowest link never announces itself.

5. The fabric

First confirm every high-speed port is up at its rated speed — for InfiniBand, tools such as ibstat show each port's state and rate. Then measure collective performance with NVIDIA's open-source nccl-tests, the same communication pattern training jobs use. Report bus bandwidth (busbw) at large message sizes rather than algorithm bandwidth, which inflates with GPU count.

Scale in steps: one node, then two, four, eight and on to the full cluster. Large-message bus bandwidth should stay roughly flat as you scale. A drop at a particular step points to a specific bad link, switch or mis-wired rail. If bandwidth is far below expectations everywhere, check that traffic is actually using the high-speed fabric — running with NCCL's debug output enabled shows which transport it selected.

Agree the target with the integrator before the tests, not after: a bus-bandwidth figure for your exact GPU count and message size, with a stated tolerance. It turns 'it seems fine' into a pass or a fail.

6. Thermal and power under real load

Run the whole cluster flat out at once, not node by node. This is the only test that shows whether the hall — power distribution, cooling plant, coolant distribution units — can carry the cluster at full load together. Watch rack inlet temperatures, coolant supply and return temperatures, flow and power per rack. Throttling that appears only when every rack is busy is a facility problem, and it is far cheaper to find it now.

7. Monitoring and handover

Before sign-off, the cluster's GPU health, power, thermal and fabric telemetry should already be flowing into the operations team's monitoring, with alerts tested. Hand over the evidence: firmware versions, burn-in logs, fabric test results at every scale step, the cabling plan and the rack photographs. That pack is what lets the next engineer diagnose a fault in minutes rather than days.

Sign-off, in one list

Every node passes the deepest diagnostic level and a sustained burn-in with no outliers. Every fabric port is up at rated speed. Bus bandwidth meets the agreed target at full scale, within tolerance. The whole cluster runs at full load without thermal throttling. Telemetry and alerts are live. The evidence pack is delivered. Anything short of all six is not commissioned — it is installed.

Where Zyvark fits

Cluster commissioning and lifecycle work is one of our core services, delivered with the hardware vendor and the facility team, and NetPulse carries the telemetry into day-to-day operations. If you are planning a cluster, start with our free AI rack readiness check on the Tools page, or ask about a fixed-scope readiness assessment before procurement.

Sources

NVIDIA, Data Center GPU Manager (DCGM) documentation — diagnostics run levels. NVIDIA, nccl-tests (open-source collective-communication benchmarks) and NCCL documentation. Supermicro, SRS-GB200-NVL72 specification, and Gigabyte, GB200 NVL72 GIGAPOD specification (rack power). Public cluster-validation guides from GPU cloud providers (Crusoe, Nebius) on bus-bandwidth testing and scaling. Vendor acceptance procedures take precedence over this general checklist.