As model sizes grow, deep neural networks that operate on different forms of sparse data, ranging from graphs to regularly sparse attention matrices, are becoming increasingly popular. However, efficiently performing training and inference on such models has been challenging. Compared to optimizing isolated sparse primitives, such models have many interacting components. As a result, optimizations and abstractions should be thought of holistically to achieve high efficiency and programming productivity. For example, when optimizing for graph machine learning, one has to reason about interactions between sparse, dense, and other components such as temporal information. In this talk, I will dive into such optimizations and abstractions we built to increase the productivity and performance of both static and temporal graph neural networks, sparse convolutional networks, and sparse attention mechanisms, achieving superior performance benefits while increasing end-user productivity.

Charith Mendis is an Assistant Professor at the University of Illinois at Urbana-Champaign. His research interests are in automating compiler construction using both formal methods and ML techniques and in building high-performance ML systems. He received his Ph.D. and Master’s from the Massachusetts Institute of Technology and his B.Sc. from the University of Moratuwa. He recently co-led the DARPA ISAT study on “ML Optimized Compilers for Heterogeneous Architectures (MOCHA).” He is the recipient of a DARPA Young Faculty Award, an NSF CAREER Award, an IEEE Micro Top Picks honorable mention, the William A. Martin outstanding master’s thesis award at MIT, a best student paper award, a best paper award, and the university gold medal for his B.Sc.