Skip to content

SC25 Talks at the UMD Booth

SC25 Talks at the UMD Booth

Layer-Aware Asymmetric Tensor Parallelism for Heterogeneous LLM Inference

Running LLM inference in heterogeneous GPU environments presents unique challenges and is generally underexplored. Existing tensor-parallelism (TP) strategies typically assume homogeneous environments, which leads to severe load-imbalance. We propose and implement a novel layer-aware asymmetric tensor parallelism (LAATP) approach, which enables different layers to be partitioned independently in a way that is optimal for their specific performance characteristics. Additionally, we introduce and evaluate an analytical performance model for static load balancing. The model is based primarily on the arithmetic intensity (AI) of relevant kernels and their attainable performance (AP) on the heterogeneous hardware in question. Our model consistently finds optimal or near optimal asymmetric TP configurations. When using LAATP configured with the predictions of our performance models, we observe up to 27.4% higher throughput than naïve symmetric tensor parallelism on heterogeneous GPUs.

Speaker Bio - Lannie Hough

Lannie Hough

Lannie Hough completed his BS in CS at Stetson University and worked at JPMorgan Chase and i2k Connect as a software developer before starting his PhD at UMD. He is currently a second-year PhD student at UMD. His research interests are systems for machine learning and reducing communication volumes in HPC applications.