We discuss parallel scheduling methods targeting various deep learning computation on top of parallel computing environments. The assumed parallel computing environments are based on many processing elements with ring connections or two-dimensional mesh connections. These are easy to implement architectures, and hence they can be utilized in various AI system integration. We first discuss the problems of dense matrix-vector multiplications and its extension to convolutional neural networks. Our approach has three steps. In the first step, small instances of the parallel scheduling problem are formulated as SAT problems and are optimally solved. In the second step, general scheduling algorithms which works for any sizes of problems are interactively inferred with our template based synthesis methods. In the third and final step, the correctness of the synthesized scheduling algorithms are formally verified. The proposed method has also been applied to the cases of sparse matrices and self-attention which is now one of the hottest topics in the field. For these cases as well, we have come up with the general parallel scheduling algorithms which have been proven to be very close to the optimum ones through experiments.

Bio:
Masahiro Fujita is a professor in the VLSI Design and Education Center at the University of Tokyo, Japan. He received his Ph.D. in Information Engineering from the University of Tokyo in 1985. From 1993 to 2000, he was the Director at Fujitsu Laboratories of America and headed the hardware formal verification group. He has co-authored 10 books, and more than 300 publications in the areas of hardware verification, high-level synthesis, testing, and software verification targeting embedded software and web-based programs. He has been involved as a program and steering committee member in many prestigious conferences on VLSI CAD and software engineering. His current research interests include system-on-chip synthesis and verification, hardware/software co-design targeting, digital/analog co-design, and formal analysis/verification/synthesis of cyber-physical systems.

Event Details
Topic: Parallel Scheduling Modern Deep Learning Implementations: Synthesis and Generalization of Parallel Algorithms Considering Communication Constraints
Day/Date: Tuesday, October 6, 2020
Time: 6pm-7pm EST
Host: Prabhat Mishra, Professor, University of Florida – UF CISE
Zoom link:
https://ufl.zoom.us/j/99610213865?pwd=YlBFeFRmK2Rpc3ZFRUVHL2s3dFhtUT09
Meeting ID: 996 1021 3865
Passcode: IOT_NELM