The 18% You Cannot Name

The Slack message arrives at 03:47. Training throughput on the 504-GPU Llama run is sitting at 82% of Tuesday's baseline, stable for six hours, zero errors in the NCCL log, zero ECC events, every InfiniBand port reporting active, and the product lead wants to know if we should fail over to the spare nodes or ride it out until the morning standup. I want to know that too.

A single degraded GPU in a 512-node training cluster can reduce overall throughput by 40%, and the failure mode everyone builds runbooks for is the hard stop: a GPU falls off the PCIe bus, an InfiniBand link goes down, ECC errors pile up, something throws a NCCL timeout and the job crashes. If even a single node is slow, the entire training job stalls, and you get a clean signal. The operator's job in that world is triage and restart. The tooling is decent. The contract with the hardware is that it either works or it visibly does not.

The 18% throughput gap lives in a different place. Many operational failures do not manifest as immediate hard faults, and a significant class occurs abruptly with little or no numeric precursor. Abrupt GPU failures often manifest as device detachments at the driver or interconnect level, and temperatures, power consumption, clocks, and utilization may remain nominal until the failure event. But this is not that either. Every device is present. Every temperature is in range. Every clock is locked. The cluster is simply 18% slower than it was two days ago, and the only people who know that are the two of us awake at 03:47 staring at a Grafana dashboard that has forty-seven green panels and one orange line trending down.

In large-scale GPU clusters, accidental link failures occur frequently, especially with network speed beyond 400 Gbps, leading to stragglers and unexpected crashes of collective communication due to timeout error. But timeout is the exit. Before timeout, there is degradation, and degradation at 400 Gbps InfiniBand NDR scale does not always announce itself. Under a degraded fabric with 10% fewer links, the p01 bandwidth gracefully degrades by 11%, closely tracking the actual physical network capacity. Graceful is the operational problem. A 10% fabric fault that produces an 11% slowdown does not light up an alert. It sits inside the noise floor of your training throughput variance unless you are logging baselines per communicator, per collective, per rank, and comparing them against Tuesday.

NVIDIA recently released NCCL Inspector, a low-overhead profiler plugin shipped with NCCL 2.28 that logs detailed performance data and metadata at runtime for every collective operation in each communicator, covering message size, execution time, algorithmic and bus bandwidth, and communicator placement. NCCL Inspector went GA in December 2025. It is the first time NCCL has shipped with a tool designed to make silent degradation visible at the collective level without requiring an external profiler attached to every rank. Before that, you had ib_send_lat, you had DCGM, you had Prometheus scraping NIC counters, and you had a forty-minute MTTR because the tooling told you the network was fine and the GPUs were fine and the training was simply 18% slower for reasons the dashboards could not name.

The operational tell is not in the hardware counters. InfiniBand fabric issues multiply across distributed training jobs, and single link errors cause MPI_Allreduce operations to hang indefinitely, while symbol errors exceeding 100 per hour indicate cable degradation requiring replacement. But 82 errors per hour do not. A cable that is failing but has not yet failed produces intermittent retransmits, elevated tail latency on a subset of paths, and a training throughput number that drifts down and then stabilizes at the new wrong number. Large-language-model training runs collective operations, chiefly AllReduce, where every GPU must share its gradients with every other GPU before the next iteration can begin, and a single slow link holds up the entire job. The AllReduce completes. The gradient sync completes. The step completes. It just completes slower, and unless you are logging step time per rank and per communication primitive, you do not know which of the 504 GPUs is waiting on which of the 3.2 Tbps of InfiniBand bandwidth you have provisioned per node.

For large-scale training beyond thousands of GPUs, NCCLX enables fast initialization and fault-tolerant communication to support a high ratio of effective training, and the major cloud ML platforms have all shipped internal NCCL forks with better observability and failure recovery than the upstream library provided until late 2025. When Meta introduced distributed GPU-based training, the company decided to construct specialized data center networks tailored for these GPU clusters and opted for RoCE as the inter-node communication transport, and Meta's infrastructure automatically removes nodes showing excessive InfiniBand errors from training pools. That works if you define excessive as a threshold you can measure. It does not work if the node is producing 70% of expected RDMA throughput with zero logged errors, because the failure mode is not packet loss or link down, but S3-over-RDMA transfers into GPU memory failing with a remote operation error, and frequently only the GPU physically closest to the NIC failing while other GPUs succeed. The symptom is not uniform. The correlation is not obvious. The dashboard is green.

I tell the product lead we are not failing over. The spare nodes would take 90 minutes to warm up, synchronize checkpoint state, and rejoin the communicator, and we do not yet know if the fault is in the fabric, the NIC, the NUMA topology, or a memory bandwidth bottleneck somewhere in the HBM stack. AMD MI300X GPUs provide 192 GB of HBM3 memory and 5.3 TB/s memory bandwidth, and the MI325X upgrades to 256 GB of HBM3e memory with 6 TB/s bandwidth, but a 70B parameter FP16 model requires moving approximately 140 GB of weights per generated token, and at 8 TB/s that yields only approximately 57 tokens/second at theoretical peak before KV-cache, batching overhead, or memory fragmentation, with even NVIDIA's best GPU spending a large fraction of inference time waiting for memory, not computing. Training is worse. Every backward pass moves gradients across every GPU, and as AI chip computing power continues to grow much faster than memory bandwidth, system performance is increasingly constrained by data transfer efficiency. A memory bottleneck and a fabric bottleneck produce the same operational symptom: the job runs slower and nothing errors out.

The NCCL Inspector log will tell me. It logs every AllReduce, every ReduceScatter, every broadcast, with the message size, the algorithm NCCL selected, the bus bandwidth it achieved, and the rank placement. If one rank is consistently 30% slower on AllReduce and it is the rank on the node whose InfiniBand NIC is in the far NUMA domain from its GPUs, I have a NUMA config error. If every rank on two specific nodes is 15% slower and those two nodes are on the same leaf switch, I have a fabric fault. If the slowdown is distributed evenly across all ranks and the bus bandwidth numbers are all within 5% of nominal, the bottleneck is not the network and I need to look at the HBM.

I do not have that log. NCCL Inspector shipped three months ago. We are still running NCCL 2.25 because the H200 nodes were deployed in May 2026 and nobody budgeted the validation time to upgrade NCCL mid-training on a production cluster. So I have the old tools. I SSH into four nodes at random, run nvidia-smi, check clocks and thermals, run ibstat and confirm all eight 400 Gbps NDR ports are active, check ib_send_bw between two GPUs on different nodes and confirm I am seeing 380 Gbps, which is within variance, and confirm that I have no evidence of anything wrong except the throughput number the training framework is reporting.

I add a line to the incident log: degraded throughput, cause unknown, monitoring. I do not page the network team because I do not have a test that failed. I do not fail over to the spares because I do not have confidence the fault will not follow. I sit with the dashboard and I wait for something to get worse.

NVIDIA Quantum InfiniBand with self-healing network capabilities overcomes link failures, enabling network recovery 5,000 times faster than any other software-based solution, and the vendor pitch is that the fabric heals itself before you notice. The operational reality is that partial degradation below the failure threshold does not heal and does not alert. It just costs you 18% of your training throughput until you can isolate which of the 504 GPUs, 63 nodes, 504 InfiniBand ports, 12 leaf switches, or 4 spine switches is the one that is failing slowly enough that nothing errors out.

The job finishes. It finishes 22% over the time budget, which means the product lead will ask why in the retrospective, and I will not have an answer that names a component. The post-mortem will read: intermittent performance degradation, likely fabric or memory subsystem, unable to isolate root cause with available tooling, recommend NCCL upgrade to 2.28+ for Inspector telemetry on future runs. The next time this happens, I will have the telemetry. This time, I have a Slack thread and a line chart and eighteen percent I cannot name.

Tarry Singh is the founder and CEO of Real AI (realai.eu), an enterprise AI advisory and deployment firm working with global enterprises on production agent systems, model risk, and AI sovereignty strategy. He also leads Earthscan (earthscan.io) for Energy AI, and is a founding contributor to the EU-funded HCAIM and PANORAIMA programmes for responsible AI education across European universities. He writes at tarrysingh.com.