Performance engineering of the Totem group communication system

Group communication systems simplify the development of fault-tolerant distributed applications; however, concerns exist about the performance and overheads associated with such systems. This paper presents measurements of the performance of the Totem group communication system, operating on a single local-area network (LAN) and on multiple LANs interconnected by gateways. The Totem system runs in user space with standard commercial off-the-shelf (COTS) software and hardware. For 1 kbyte messages, a throughput of over 5000 messages per second has been measured for Totem on a single LAN, and an aggregate throughput of 10 000 messages per second has been measured for Totem on three LANs. At 2000 messages per second, the message latency is less than 3 ms. The paper also discusses some of what has been learned in engineering a group communication system for high performance.