AWS Machine Learning BlogT1·Infra
72 pts
Amazon added three silent data corruption (SDC) checks to the PyTorch DDP training loop on SageMaker HyperPod, detecting anomalies from the job side when hardware reports nothing, then mapping the suspect rank to its node, containing the corruption and recovering. The approach targets large-scale distributed training where wrong-but-plausible numbers silently poison gradients and weights; Meta attributed 6 of 419 unexpected interruptions during Llama 3 405B pre-training to SDC.