Journal of the Association for Computing Machinery, Vol. 42, No. 1, January 1995, pp. 204–231.

Bounds on the Speedup and Efficiency of Partial Synchronization in Parallel Processing Systems

C. S. CHANG

AND

R. NELSON

Abstract. In this paper, we derive bounds on the speedup and efficiency of applications that schedule tasks on a set of parallel processors. We assume that the application runs an algorithm that consists of N iterations and before starting its i + 1st iteration, a processor must wait for data (i.e., synchronize) evaluated in the ith iteration by a subset of the other processors of the system. Processing times and interconnections between iterations are modeled by random variables with possibly deterministic distributions. Scientific applications consisting of iterations of recursive equations are examples of applications that can be modeled within this formulation. We consider the efficiency of such applications and show that, although efficiency decreases with an increase in the number of processors, it has a nonzero limit when the number of processors increases to infinity. We obtain a lower bound for the efficiency by solving an equation that depends on the distribution of task service times and the expected number of tasks needed to be synchronized. We also show that the lower bound is approached if the topology of the processor graph is “spread-out,” a notion we define in the paper.

Categories and Subject Descriptors. C.1.2[parallel processors]; C.4[performance attributes]; G.M[queuing theory]

General Terms. Measurement, Performance

Additional Key words and Phrases. Large deviations theory, synchronization

Copyright 1995 by ACM, Inc. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or direct commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Publications Dept, ACM Inc., fax +1 (212) 869-0481, or permissions@acm.org.