Lemon nodes: why a green health check doesn’t mean a GPU server is ready

There’s a failure pattern that only shows up once you’re running real workloads at scale, and it’s more expensive than the dramatic ones. Not a GPU that falls off the bus, not a node that goes dark — those are easy, you see them and you replace them. The costly one is the node that looks fine every time you check it and fails every time you actually use it. The industry has a name for it now: the lemon node.

What a lemon node actually is

A lemon node passes health checks while it’s idle and fails under real load. A job lands on it, the job dies, the node gets flagged, someone pulls it, runs the standard checks — and everything comes back green. So it goes back into the pool, picks up the next job, and kills that one too. Without something tracking the pattern across jobs, that node circulates through your fleet indefinitely, quietly breaking one workload after another.

The causes are spread across the whole stack: a GPU that’s marginal, memory that’s starting to go, a PCIe link, a flaky NIC, a firmware revision that’s slightly off. What ties them together is the thing that makes them dangerous — none of them show up under a synthetic health check. They only surface under real production load.

Why burn-in can miss it

The instinct is: run a burn-in, problem solved. It helps, but standard health checks and short synthetic tests may fail to expose intermittent or workload-dependent faults — a light or generic GPU burn-in can walk straight past a lemon node, because the fault often needs a specific combination to appear: GPU and HBM and the NIC and the fabric all driven hard at once, at temperature, with real traffic on the wire. A test that exercises one subsystem at a time, or runs for an hour and calls it done, never creates that combination. The node passes, because you never asked it the question that makes it fail.

This is the core of it: a health check confirms a node responds. It says nothing about whether the node performs under the load it was bought for. Those are different questions, and only one of them matters at acceptance.

Why swapping the node doesn’t end it

Replacement feels like the fix, and for a cleanly dead node it is. For a lemon node it often isn’t — because the replacement goes through the same shallow validation the original passed, and if that validation couldn’t catch the fault the first time, it won’t catch it on the way back in either. You’ve spent the swap, the revalidation, the engineer’s time, and the node is back in rotation waiting for its next victim. The loop closes and starts again.

Breaking it takes something most first-time operators don’t build in: tracking failures against specific hardware over time, so a node that fails repeatedly gets pulled permanently instead of cycling back after every green check.

What it costs — with real numbers

This isn’t theoretical, and the numbers are public. Meta published an analysis across 150 million A100 GPU-hours (arXiv:2410.21680). Two findings matter here.

First, scale makes it inevitable. At 8 GPUs, mean time between failures is around 47 days — a human can handle that by hand. At 1,024 GPUs it drops to roughly 8 hours. At 16,384 GPUs it’s about 1.8 hours. More components, more chances for something to be wrong, and on a large cluster a rare fault stops being rare — it’s a near-certainty that some node, somewhere, has it. And the damage is lopsided: failures hit only 0.2% of jobs but consumed 18.7% of total runtime, because the big, long training runs are exactly the ones that keep tripping over them.

Second — and this is the number worth remembering — when Meta implemented systematic lemon-node detection and removed those nodes from scheduling, the failure rate for large jobs dropped from 14% to 4%. Not from better GPUs or a faster network. Just from identifying the nodes that repeatedly cause problems and taking them out. More than a threefold improvement from finding the lemons.

Put that in money. When a node dies mid-run, a synchronised training job halts and rolls back to the last checkpoint. In a 1,000-GPU cluster, the other 999 sit idle waiting for the recovery. A lemon node doesn’t do this once — it does it on every job it touches until someone breaks the cycle.

Why this gets worse on B200, B300 and HGX clusters

Everything above scales with the cluster, and Blackwell-class builds — B200, B300, HGX — are large and dense by design. More nodes mean the rare fault is more likely to be present somewhere. Higher density means more heat and more power draw, which is exactly the regime where marginal hardware tips over — a GPU that’s fine at moderate load throttling or erroring only when the whole rack is pushed at once. The bigger and denser the cluster, the more certain it is that at least one lemon is hiding in it at handover, and the more each undetected one costs.

What real acceptance has to include

The takeaway isn’t complicated, and it’s the whole point of GPU cluster commissioning. A cluster is not proven — not production-ready — when every server powers on and the health dashboard is green. It’s proven when it has held up under sustained, realistic load — long enough and hard enough to force the marginal hardware to reveal itself.

That means acceptance has to go past POST, firmware inventory and health checks to include an extended run that drives GPU, memory, interconnect and power together, at temperature, the way a real workload does — with failures tracked against specific nodes so the repeat offenders are identified and pulled, not cycled back in. This is what separates a real GPU cluster acceptance test from a power-on check. Anything less accepts the lemons along with the good hardware, and hands the cost of finding them to whoever runs the first real training job.

Where GTVA fits

This is the part of commissioning we treat as the point of the exercise, not a formality at the end. Physical acceptance and diagnostics, firmware brought to a single baseline, a burn-in that loads the whole system rather than one piece at a time, a real workload run — and node-level tracking so the hardware that fails under load is caught and removed before handover, not after. What the client receives is a cluster that has already been made to fail everywhere it was going to, and had those nodes taken out. The lemons are our problem to find. They shouldn’t be the client’s to discover in production.