The failure that doesn’t crash anything: fail-slow in large GPU clusters

Everyone plans for the node that dies. It’s visible, it’s loud, the job stops, you replace the hardware and move on. The failure that actually eats your budget is quieter: the node that keeps working but works slowly. Nothing crashes. No alert fires. The training run just takes longer than it should, and unless someone is looking at the right numbers, nobody can say why. The name for this is fail-slow, and at scale it does more damage than the failures people actually prepare for.

What fail-slow actually is

A fail-slow component hasn’t failed — it’s degraded. A GPU that’s throttling on temperature and running below its clock. A CPU losing the fight for cycles against something else on the box. A network link quietly choking on congestion. Each one keeps doing its job, just slower than the rest of the cluster, and in synchronised training that’s all it takes. Every GPU waits for the slowest one at each step, so a single degraded node drags the entire job down to its pace. Thousands of healthy GPUs, throttled to the speed of one sick one.

That’s what makes it worse than a clean crash. A dead node you find in seconds. A slow node can sit there for the length of a training run, bleeding throughput, while every dashboard shows green because technically everything is “up”.

How much it actually costs

This has been measured, and the numbers are worth sitting with. Alibaba published a study of fail-slow on a production cluster of more than 10,000 GPUs across 4,000 nodes (arXiv:2410.12588). Across the jobs they looked at, fail-slow stretched average job completion time by 1.34×. That’s a third more time, a third more cost, for work that would otherwise have finished — and nothing broke to cause it.

But the number that matters most is how it grows with scale. On a single node, the average slowdown was about 12%. On four nodes, 15%. On the large jobs — 512 GPUs and up — it jumped to 35%. And it wasn’t rare up there: of 27 large jobs they inspected, 16 hit fail-slow, and one in five of those lost more than half their speed. The bigger the job, the more links and nodes it spans, and the more surface it exposes to the one component that’s quietly lagging.

The causes split in a useful way. Compute-side slowdowns — a throttling GPU, CPU contention — tend to be short, around ten minutes on average, and less frequent. Network-side slowdowns are the real problem: more common, and longer, averaging around 24 minutes on smaller jobs and stretching to 72 minutes on the big ones. For large distributed jobs the network fabric is one of the most important sources of fail-slow behaviour — and one of the easiest to overlook during commissioning.

Why it’s so hard to catch

Here’s the trap. A dead component announces itself. A slow one doesn’t — it looks exactly like a healthy one that happens to be busy. Your monitoring says the node is up, the GPU responds, the link carries traffic. All true, and none of it tells you the thing is running at 80% of what it should. To catch fail-slow you have to be watching performance over time and comparing nodes against each other, not just checking whether each one answers.

Which is why, in most shops, the way fail-slow gets found is that an engineer eventually notices the run is slow and goes hunting by hand — pausing the job, benchmarking components, trying to isolate the laggard. On a cluster of thousands of GPUs that’s slow, painful, and expensive, and it only starts after the money’s already been lost.

When they stack

The worst case isn’t one slow component — it’s two at once, and at scale that stops being hypothetical. The Alibaba study caught it on a 1,024-GPU job: network congestion first cut throughput by 80%, and then, while that was still going, a single GPU started throttling on temperature. Together they drove the job down to about 10% of its normal speed. Either problem alone would have been a bad afternoon. Both together nearly stopped a thousand-GPU run cold — and neither one “failed” in any way a basic health check would flag.

This is the through-line with thermal throttling generally: a GPU that overheats doesn’t error out, it just slows down, and if you’re not watching for it you pay for the heat in lost throughput without ever seeing a fault.

What this means for a cluster you’re taking on

Fail-slow often reveals itself in production, but many of the conditions that create it can — and should — be identified during commissioning. Thermal throttling, power-related degradation, uneven GPU performance, fabric misconfiguration, NUMA and PCIe problems, wrong firmware or settings — these are visible under the right load before a cluster is ever handed over. The groundwork for catching fail-slow is laid at commissioning, and that’s the part worth getting right before anyone signs off.

A cluster that’s only been checked for “does every node respond” is a cluster where fail-slow is invisible by construction. Catching it needs a performance baseline established under real load — what throughput each node actually delivers when pushed, so a node running slow later can be spotted against the others instead of hiding in the average. It needs monitoring that watches for degradation, not just for outages. And it needs the network held to the same standard as the GPUs, because the fabric is where most fail-slow comes from and it’s the part people check least.

Where GTVA fits

GTVA treats performance as part of commissioning, not something to be measured after the first production incident.

Before handover, we establish a performance baseline under sustained load, validate GPU and system behaviour under thermal and power conditions, exercise the interconnect fabric under realistic traffic, and identify nodes or links that perform outside the expected range.

That baseline then becomes the reference point for operations. When a node starts delivering less performance than the rest of the cluster, there is something to compare it against.

A cluster doesn’t have to crash to have a problem. Sometimes the most expensive failure is the one that keeps running.