The flush protocol used by the MBRSHIP layer scales to perhaps a hundred members (if the membership is fairly static), but will then start showing signs of severe performance degradation. To allow scaling to groups with thousands of members (or even more), we designed a second membership protocol in which a tree-structure is superimposed on large groups.
In this layer, the members are divided into two sets: servers and clients. The servers run over the MBRSHIP layer. Each server is responsible for a set of clients, and forwards any events that it receives. By default, a client communicates through its responsible server. If a server fails, another server will adopt the orphaned clients. This way, the protocol simulates exactly the same group model as the MBRSHIP layer, with better scalability but at the cost of a higher average latency for clients. Scalability and latency can be traded off further by stacking multiple CLTSVR layers on top of each other (creating multiple levels in the hierarchy). Ordering properties used by the server members are inherited by clients. For example, if the servers use the TOTAL layer, messages will be received in a total order at the clients as well as the servers.
We anticipate two primary uses for the CLTSVR layer. One of these is in support of Replicated Remote Procedure Call (RRPC), where the set of replicas (or the primary and its backups) coincides with the set of servers. RRPC can be used to provide fault-tolerance, load balancing, and parallel request processing in servers shared by large sets of clients. The second expected use is for dissemination of data from a small set of sources to a large set of processes (e.g., for use in a brokerage or factory automation system).
Although our current CLTSVR layer provides the full semantics of the group mechanisms to the clients, we also intend to support optionally weakened semantics for greater scalability in these applications. For example, clients may not need to know who the other clients are (or even the membership of the set of servers), and by not reporting this information, overhead can be reduced. Moreover, if some small risk of message atomicity violations can be tolerated by clients, latency of client-server communication can be greatly reduced. Finally, we are exploring options for integrating security features with the CLTSVR layer [11].