Fault tolerant distributed training on Amazon EKS using NVRx

Integrate NVIDIA Resiliency Extension (NVRx) into PyTorch FSDP training on Amazon EKS to overlap checkpoint I/O with training and recover from GPU faults in seconds. This post covers async checkpointing, in-process restart, and ft_launcher…

Summary from AWS Machine Learning Blog.

Coverage 1 article · 1 outlet

1. AWS Machine Learning Blog 16 Sep
   Fault tolerant distributed training on Amazon EKS using NVRx ↗