Common mistakes first-time AI builders make — and what they cost

Over the past year we've seen the same project shape again and again. A team raises money, orders a few hundred GPU servers, and pencils in a go-live date a week after the trucks arrive. The hardware almost never lets them down. The assumptions around it do.

None of this is exotic. These are just the things we keep running into on day one of an engagement — and every one of them shows up later as lost time, wasted capacity, or hardware that never quite hits spec.

"We'll have it running in a few days."

Delivery feels like the finish line. It's the start.

A rack of eight-GPU B300 systems doesn't come up the way a workstation does when you plug it in. Between the pallets landing on the dock and the cluster taking its first training job, there are weeks of work: racking and cabling — one node alone carries eight compute links plus storage and management, and it runs into the thousands across a full build — then firmware alignment on every node, per-node diagnostics, fabric validation, and a burn-in long enough to shake out the thermal and power faults that a quick test never touches.

Squeezing the schedule doesn't make that work disappear. It just pushes it past go-live, where it comes back as a cluster that's unstable the moment real load hits it. And chasing faults through a cluster that's already running jobs is a different, worse job than commissioning an empty one — reckoned in weeks of slip, and in production work that has to be paused or moved while you take the thing back down to fix it.

A rack is a power and thermal system, not a shelf of servers.

This is where physical planning falls over most often. Eight GPUs a node pull real power, and once you pack a rack with them it stops acting like a row of independent servers. It's one thermal and electrical system now, and it has to be designed as one.

The usual trap is reading a rack off the nameplate power and assuming the cooling takes care of itself. It doesn't. Nameplate isn't real load — and here's the part that catches people: a cluster can look perfectly healthy while it's ticking over, and only fall apart when GPU, HBM and the fabric all get pushed hard at the same moment. Which is exactly what a real training run does, and exactly what a light smoke test never will. Airflow, blanking panels, power distribution, and at this density rear-door or liquid cooling — none of that is polish. It's what decides whether the rack survives sustained load.

Get it wrong and the GPUs throttle. They clock themselves down to survive the heat, quietly, and nothing throws an error. The cluster passes every functional check and hands back a fraction of the throughput you paid for — thousands of GPU-hours of it — and you find out from the numbers, not from a log.

"Firmware is updated" is not one checkbox.

Firmware is the trouble nobody budgets for, because on any single node everything looks fine. The problem is that a cluster isn't one machine — it's thousands of parts that all have to agree. "Up to date" isn't one thing either: it's BIOS, BMC, GPU firmware, NIC firmware, NVSwitch and fabric firmware, the driver stack, CUDA, the networking stack, kernel versions, and every compatibility line between them.

And it's not about each piece being new. It's about the cluster being the same everywhere. One node on a different BIOS or NIC revision can sail through local diagnostics and still drag down collective operations across the whole fabric — a slow all-reduce, a stall that comes and goes, a job that hangs at scale for no reason you can see at first. The symptom is everywhere; the cause is one node out of step. Track that down after go-live, in a system already under load, and it's days of engineering per incident.

More GPUs is not more compute.

The most expensive misconception is a piece of arithmetic: 5,040 GPUs, so 5,040 GPUs' worth of training. For distributed training that just isn't true. What sets your throughput isn't how many GPUs you own — it's how well they can talk to each other.

Every iteration, a gradient makes the trip: GPU → NVLink/NVSwitch → NIC → fabric → NIC → NVLink/NVSwitch → GPU. The big bandwidth figure on a switch datasheet describes one link on a good day. It says nothing about what the fabric does when thousands of GPUs try to synchronise at once. Topology, oversubscription, rail alignment, how RDMA behaves, congestion control, packet loss — any of them can turn a network that looks fast on paper into one where your most expensive hardware sits idle waiting for data.

Here's the part that bites: a cluster can pass every ordinary network test and still be a bad training cluster. ping and iperf tell you the links are up and bytes move between two points. They tell you nothing about how a full-scale NCCL all-reduce behaves — and that collective, not point-to-point throughput, is what training actually runs on. It's why the only honest measure of a fabric is effective collective bandwidth as you scale from a handful of nodes up to the whole thing. Hold the line and it's built right. Watch it collapse at some step, and you've found where the problem lives. Miss it, and you're paying for idle GPUs — the single biggest line in the budget — stalled on a network that looked fine on paper.

No acceptance criteria means no way to say "done".

Everything above ends up here, and this is where an engineering problem turns into a money problem. If nobody wrote down — in numbers, agreed before the hardware shipped — what a healthy cluster actually looks like, then installation ends in an argument that neither side can win. How many GPUs are allowed to be down at handover. Which NCCL results have to be met. What error rate you'll accept. How many hours of burn-in. What thermal and power limits. What "success" even means for a real distributed workload.

Skip that, and you get the classic standoff. "The hardware works." — "It works at 78% of what we expected." Both true. With no measurable line agreed up front, there's nothing separating delivered from not delivered, and the side holding the money and the side holding the invoice look at the same cluster and see two different things. That dispute stalls acceptance, and payment, for weeks. Acceptance criteria aren't paperwork. They're the definition of what you're buying.

What this adds up to

The expensive mistake isn't buying too many GPUs. It's buying them without deciding, in advance, what "working" is going to mean once they all have to work together.

A large cluster isn't commissioned when every server powers on. It's commissioned when the whole system — compute, network, cooling, firmware, software — has actually shown it can do the thing it was bought to do.