DiLoCo is a distributed-optimization method for training LLMs across slow or geographically separated networks. Each worker runs many local AdamW steps on its own data; only every ~500 steps do the workers send compact “pseudo-gradients” to a global Nesterov-momentum optimizer, slashing communication by orders of magnitude. This infrequent-synchronization design makes training feasible over poor links and resilient to stragglers or shifting resources, although all workers must still rendezvous at the same global step, which can leave fast machines idle.