Amazon HyperPod Adds Silent Data Corruption Detection and Recovery

1 reports archived1 independent sourcesupdated 11/12/2026, 07:30:03

Synthesis & Latest Updates

1 Sources Cross-Validated

Amazon added three silent data corruption (SDC) checks to the PyTorch DDP training loop on SageMaker HyperPod, detecting anomalies from the job side when hardware reports nothing, then mapping the suspect rank to its node, containing the corruption and recovering. The approach targets large-scale distributed training where wrong-but-plausible numbers silently poison gradients and weights; Meta attributed 6 of 419 unexpected interruptions during Llama 3 405B pre-training to SDC.

LATEST/Amazon added three silent data corruption (SDC) checks to the PyTorch DDP training loop on SageMaker HyperPod, detecting anomalies from the job side when hardware reports nothing, then mapping the suspect rank to its node, containing the corruption and recovering. The approach targets large-scale distributed training where wrong-but-plausible numbers silently poison gradients and weights; Meta attributed 6 of 419 unexpected interruptions during Llama 3 405B pre-training to SDC.

Hourly heat curve

2 fully observed hours

Fewer than 3 fully observed hours; trend not plotted yet.

Coverage timeline

Total 1 reports · Latest first

AWS Machine Learning BlogT1Nov 12, 07:30·72 pts
Amazon HyperPod Adds Silent Data Corruption Detection and Recovery
- SDC raises no hardware error, so health checks and node replacement miss it; detection must come from the training job itself
- The approach adds three checks to the DDP loop, then maps the suspect rank to its node for containment and recovery
- Meta attributed 6 of 419 interruptions over 54 days of Llama 3 405B pre-training to SDC
View Related Reports & Analysis Original Source