LoRDO: Distributed Low-Rank Optimization with Infrequent Communication

Jovanović, Andrej and Iacob, Alex and Safaryan, Mher and Modoranu, Ionut-Vlad and Sani, Lorenzo and Shen, William F. and Qiu, Xinchi and Alistarh, Dan and Lane, Nicholas D. (2026) LoRDO: Distributed Low-Rank Optimization with Infrequent Communication. Other. Arxiv.

Text (2602.04396v1)
2602.04396v1.pdf - Published Version
Available under License Creative Commons Attribution.
Download (4MB)

Abstract

Distributed training of foundation models via $\texttt{DDP}$ is limited by interconnect bandwidth. While infrequent communication strategies reduce synchronization frequency, they remain bottlenecked by the memory and communication requirements of optimizer states. Low-rank optimizers can alleviate these constraints; however, in the local-update regime, workers lack access to the full-batch gradients required to compute low-rank projections, which degrades performance. We propose $\texttt{LoRDO}$, a principled framework unifying low-rank optimization with infrequent synchronization. We first demonstrate that, while global projections based on pseudo-gradients are theoretically superior, they permanently restrict the optimization trajectory to a low-rank subspace. To restore subspace exploration, we introduce a full-rank quasi-hyperbolic update. $\texttt{LoRDO}$ achieves near-parity with low-rank $\texttt{DDP}$ in language modeling and downstream tasks at model scales of $125$M--$720$M, while reducing communication by $\approx 10 \times$. Finally, we show that $\texttt{LoRDO}$ improves performance even more in very low-memory settings with small rank/batch size.