Synchronization-avoiding graph algorithms and runtime aspects

Abstract

Massively parallel computers provide unprecedented computing power that is only expected to grow. General-purpose Asynchronous Many-Task (AMT) runtimes exposes significant fine-grained parallelism. However, traditional Bulk-Synchronous Parallel (BSP) approach and variants thereof fail to utilize the full potential of AMT runtimes. In such execution model, parallel overheads may dominate execution time and hinder performance, especially in distributed memory settings. The synchronization overhead in particular is deeply rooted in the programming practice because it makes algorithms easier to design and implement. However, irregular applications such as graph algorithms can suffer performance bottlenecks due to the straggler effect induced by global and vertex-centric barriers. In the effort to eliminate barriers, we design and study unordered, data-driven graph algorithms, relying on optimistic (speculative) execution. Our design of algorithms allows work to be performed in any order, refining the result as the algorithm progresses. This flexibility in ordering facilitates parallel computation without global or vertex-centric synchronization. To avoid “wasted” work, our approach relies on local work prioritization. However, performance of such algorithms is marred by two competing trends: on one hand, there is a substantial level of parallelism, which, on the other hand, necessitates runtime support with potentially high overhead and scheduling complexity. Specifically, the global work order obtained by local prioritization is susceptible to the interference from the runtime. As such, we also investigate important runtime aspects that influence the performance of graph algorithms. In particular, we study the interaction between default runtime scheduling policies and synchronization-avoiding distributed graph algorithms. We propose plug-in scheduling policies in the application-layer for speculative graph algorithms that can provide feedback to the runtime to adjust the network progress frequency and flow of messages to the remote destinations. These techniques are useful for unordered distributed graph algorithms that necessitate a balanced interleaving of communication and computation to achieve better performance. Graph algorithms designed in different programming models have vastly different workload characteristics. In the final part of the thesis, we propose adaptivity of the runtime parameters such as message coalescing and flow control to adjust the “pressure points” in the runtime due to variable workload characteristics over the execution of an algorithm.