Skip to content

SC25 Talks at the UMD Booth

SC25 Talks at the UMD Booth

Understanding and Mitigating Communication Bottlenecks in Multi-Node LLM Inference

As LLMs scale across GPUs and nodes, efficient distributed inference becomes critical. In this talk, we present our work on conducting a detailed performance study of multi-node distributed inference using LLMs on GPU-based supercomputers and identifying performance bottlenecks. Furthermore, having identified all-reduce communication as a primary bottleneck in tensor model parallel inference, we present our new all-reduce implementation that achieves up to 1.9-3.6x lower latency than NCCL, for messages in the range of 128 KB to 2 MB on HPE Slinghost and InfiniBand networks, and improves end-to-end batch latency in tensor parallel inference workloads by up to 1.72x.

Speaker Bio - Prajwal Singhania

Prajwal Singhania

Prajwal is a third-year CS PhD student at UMD. His research focuses on building efficient and scalable AI training and inference systems