The Horus Object Tools: Horus Group Members and Clients/Servers

This document is a part of the online Horus Documentation, under Horus Object Tools.

The main component of HOT is a hierarchy of classes that implement group members and clients/servers with state transfer. The classes are HorusGroupMember (implements group members), HorusClSv (implements clients/servers and state transfer protocol), and HorusCSX (clients/servers with higher-level state transfer interface). They are discussed in detail in the following sections.


HorusGroupMember

The HorusGroupMember is an "anchor" class that sits on top of Horus Uniform Group Interface and implements an abstraction of a Horus group member. There are public downcalls for joining a group, sending messages to it, etc., and (protected) upcalls that are invoked when a group event arives. Each HorusGroupMember object has a unique entity by which it is identified in a group view. The entity may be obtained with the method myEntity.


Creating HorusGroupMember Objects

When creating a group member object, you pass a HorusGrpMemb_Options structure as the argument. The HorusGrpMemb_Options structure contains the following fields:

	struct HorusGrpMemb_Options { 		
		HorusEntity groupEntity; 		
		char *groupName; 		
		char *protocolStack; 	
		HorusErrorHandler *errorHandler; 	
		HorusNameServer *nameServer; 	
	};

  • The groupEntity option specifies the address of the group (it is initialized to an illegal null address by default).

  • If the groupName option is not NULL (the default is NULL), then it is assumed to be the name under which the address of the group is installed in a directory server. Thus, if groupName is set, the address of the group will be looked up using the name server specified by the nameServer option (by default, nameServer is NULL, in which case a HorusNameServer object is used). Only if the lookup fails, the value contained in the groupEntity will be used.

    Thus, either groupEntity has to be initialized, or the group address must be retrievable from the name server.

    Two other options are protocolStack and errorHandler. The protocolStack option specifies the stack of Horus layers used in the group (the default stack is defined in the file config.h), and errorHandler points to an error-handler object. If errorHandler is NULL (the default), then HorusErrorHandler is used. For example:

    	HorusGrpMemb_Options ops; 	
    	ops.groupName = "lapa"; 	
    	ops.protocolStack = "BUFFER:ORDER:CAUSAL:UNPACK:TOTAL:MERGE(auto=0):VIEWS";
    	HorusGroupMember memb(ops);	
    The groupName has been specified; therefore the address of the group will be looked up with the name server under "lapa". Since the nameServer option has not been set, the default name server (HorusNameServer) will be used.


    Joining a Group

    After a HorusGroupMember object has been created, you should call the join downcall to actually join the group.

    NOTE: A HorusGroupMember object may join one group only.

    The join downcall accepts an (optional) HorusGrpMemb_JoinOptions argument, which contains the following fields:

    	struct HorusGrpMemb_JoinOptions { 		
    		char *contactName; 		
    		HorusMessage msg; 	
    	};
  • The contactName option should be set when the MERGE layer is not being used. It specifies the name under which the contact of the group is (or should be) installed in the name server (HorusGroupMember looks up and (re)installs group contacts when necessary). If the MERGE layer is present on the protocol stack, the group contacts are not used.

  • The msg field contains the message that is delivered to the coordinator with the join request. It is ignored when the MERGE layer is used. For example:
    	HorusGrpMemb_JoinOptions jops; 	
    	jops.contactName = "lapa_contact"; 	 	
    	memb.join(jops);
    The join downcall will block until the first VIEW upcall has been received.


    Sending/Receiving Messages

    After the first VIEW upcall has been received, you may send messages to the group using the send and cast downcalls. For example:
    	HorusMessage msg; 	
    	msg << HorusString("hello world!"); 	
    	memb.cast(msg);	 	// send msg to all members of the group 	
    
    	HorusEntity dest; 	
    	....			 	
    	memb.send(dest, msg);	// send msg to dest (must be in the view)
    Each invocation of a send or cast downcall eventually results in the SEND or CAST upcall method invoked at the message destination(s). The message upcalls are defined with no-op bodies, and are to be overloaded by subclasses of HorusGroupMember as required by the application.

    The SEND/CAST upcalls have the following declarations:

    	void grpMembCast_Upcall(HorusEntity& origin, 
    				HorusMessage &msg); 	
    
    	void grpMembSend_Upcall(HorusEntity& origin, 
    				HorusMessage &msg);
    The origin argument specifies the originator of the message, and msg contains the message.


    Monitoring Group Membership Changes

    Whenever a membership change occurs, the VIEW upcall is invoked at each member of the group. The VIEW upcall has the no-op body (to be overloaded by subclasses of HorusGroupMember), and is declared as follows:

    	void grpMembView_Upcall(HorusGrpMemb_ViewData&);
    The HorusGrpMemb_ViewData structure contains membership information, and is defined as follows:
    	struct HorusGrpMemb_ViewData { 		
    		HorusEntity myEntity, myGroup;   		
    		HorusEntityList members, newMembers, departedMembers; 
    		int nmembers, myRank, myOldRank;   		
    		HorusEntity coordinator, oldCoordinator;   	
    		HorusMessage msg; 	
    	};
  • The member's address is contained in myEntity, and the group address is in myGroup.

  • The current view (list of group members) is in the members list.

  • The newMembers list contains the entities that appear in the members list for the first time.

  • The departedMembers list contains the entities that were present in the previous view, but not in the current one.

  • The nmembers field tells the size of the current view.

  • The myRank and myOldRank fields tell the rank of this member in the current and the previous views respectively.

  • The coordinator and oldCoordinator fields contain the entity of the coordinator for the current and the previous views respectively.

  • The msg field contains the message that has arrived with the VIEW event.

    For example:

    	class myMember: public HorusGroupMember { 	
    	public: 		
    
    		myMember(HorusGrpMemb_Options &ops): HorusGroupMember(ops) {} 		 
    		
    		void grpMembView_Upcall(HorusGrpMemb_ViewData& vd) { 			
    			cout << "GOT A VIEW:  " << endl << vd.members; 	
    		}
    
    		void grpMembCast_Upcall(HorusEntity& origin, HorusMessage &msg) { 	
    			cout << "GOT A CAST FROM " << endl << origin; 		
    		}	 		 		
    
    		void grpMembSend_Upcall(HorusEntity& origin, HorusMessage &msg) { 	
    			cout << "GOT A SEND FROM " << endl << origin;
    		}
    	};


    Overloading Default Behavior

    The HorusGroupMember class defines a number of protected downcalls and upcalls that implement some default behavior of Horus group members, but can be overloaded in derived classes if necessary:

  • The view downcall is invoked by the group coordinator when a new view is being installed. The downcall is declared as follows:
    	void view(HorusMessage&);
    By default, an empty message is sent with the VIEW event. However, the view method can be overloaded to include some useful information into the view message.

  • There is an upcall method corresponding to each of the Horus event types, for example
    	void JOIN_REQUEST_upcall(HorusEvent *event); 	
    	void JOIN_DENIED_upcall(HorusEvent *event); 	
    	void JOIN_READY_upcall(HorusEvent *event); 	
    	void JOIN_FAILED_upcall(HorusEvent *event); 	
    	void FLUSH_upcall(HorusEvent *event);
    and so on. The upcalls may be overloaded if it is necessary to redefine the membership-level behavior of a HorusGroupMember subclass.


    HorusClSv

    HorusClSv is a subclass of HorusGroupMember. It provides the abstraction of clients/servers and implements a state transfer protocol (the protocol is discussed in detail in the section on State Transfer in HOT ).


    Creating HorusClSv Objects

    The options for the HorusClSv constructor are the same as for HorusGroupMember. For example:

    	HorusClSv_Options ops; 	
    	ops.groupName = "lapa"; 	
    	ops.protocolStack = "BUFFER:ORDER:CAUSAL:UNPACK:TOTAL:MERGE(auto=0):VIEWS";
    	HorusClSv memb(ops);	


    Joining a Group as Client or a Server

    Whereas in HorusGroupMember all members of the group have the same properties, the HorusClSv class makes a distinction between clients and servers. When a HorusClSv object joins a group, its membership type (either client or server) is specified.

    The options structure for the join downcall is a subclass of HorusGrpMemb_Options, and is defined as follows:

    	struct HorusClSv_JoinOptions : HorusGrpMemb_JoinOptions {   
    		HorusMbrshipOption mbrshipType;    	
    		HorusXferType xferType;
    	};
  • By default, the mbrshipType option is set to HORUS_CLIENT. It should be initialized to HORUS_SERVER if the object is to join the group as a server. In the latter case, the xferType option specifies whether or not the state transfer is required, and if yes, what level of protection for state transfer should be maintained.

  • The default value of the xferType option is HORUS_NO_XFER, which means no state transfer will be initiated for the object when it joins the group. The other possible values of the xferType option are HORUS_FREE_XFER, HORUS_PROTECTED_XFER, and HORUS_ATOMIC_XFER. For example:
    	HorusClSv_Options ops; 	
    	ops.groupName = "lapa"; 	
    	HorusClSv memb(ops);			// create a member of "lapa"
    
    	HorusClSv_JoinOptions jops; 	
    	jops.mbrshipType = HORUS_SERVER; 	
    	jops.xferType = HORUS_ATOMIC_XFER; 	
    	memb.join(jops);			// join "lapa" as a server
    
    	assert(memb.isServer()); 	
    	assert(!memb.isClient());
    The difference between FREE, PROTECTED, and ATOMIC state transfer is in the safety levels of messages that can be sent during state transfer (see the discussion in the section below).


    Sending/Receiving Messages

    The safety level of a message is specified in the options argument to a message-sending downcall. The options structure is defined as follows:

    struct HorusClSv_MsgOptions { HorusMsgXferSafety msgXferSafety; HorusEntityList destList; };

  • The msgXferSafety option sets the safety level of a message. It is initialized to HORUS_MSG_GENERIC by default. Two other safety types are HORUS_MSG_SAFE and HORUS_MSG_XFER.

  • The destList parameter is used by downcalls that send messages to a subset of the view.

    The correspondence between message safety levels and state transfer types is as follows:

  • During FREE state transfer, all user messages are allowed through, and it is left up to the application to guarantee the consistency of the global state.

  • During PROTECTED state transfer, user messages of SAFE or XFER safety level (as specified by the application programmer) are allowed through. However, GENERIC messages are delayed until the completion of state transfer by all joining servers. Then they are actually sent.

  • During ATOMIC state transfer, only XFER messages can be sent. All other messages will be delayed until the transfer ocmpletes.

    Here is an example:

    	// create a member of group "lapa"
    	HorusClSv_Options ops;
    	ops.groupName = "lapa";
    	MyClSv memb(ops);
    
    	// join "lapa" as a server, with PROTECTED state transfer
    	HorusClSv_JoinOptions jops;
    	jops.mbrshipType = HORUS_SERVER;
    	jops.xferType = HORUS_PROTECTED_XFER; 
    	memb.join(jops);
    
    	HorusClSv_MsgOptions mops;
    
    	// suppose the messages below are sent during state transfer..
    
    	// send a GENERIC message (*delayed* during PROTECTED xfer)
    	mops.msgXferSafety = HORUS_MSG_GENERIC;
    	HorusMessage msg;
    	memb.cast(msg, mops);
    
    	HorusMessage msg1;
    	memb.cast(msg1);		// by default, a message is GENERIC (delayed)
    
    	// send a SAFE message (not delayed during PROTECTED xfer)
    	mops.msgXferSafety = HORUS_MSG_SAFE;
    	HorusMessage msg2;
    	memb.cast(msg, mops);
    
    	// send an XFER message (never delayed)
    	mops.msgXferSafety = HORUS_MSG_XFER;
    	HorusMessage msg3;
    	memb.cast(msg3, mops);
    Besides send and cast downcalls, the HorusClSv class defines two new methods, called scast ("server cast") and lsend ("list send").

  • An scast sends a message to all servers and to a (empty by default) list of clients. The client destinations are specified by the destList option. For example:
    	HorusMessage msg; 	
    	msg << HorusString("hello"); 	
    	memb.scast(msg);		// send msg to servers only
    	....
    	HorsEntity ent;
    	....
    	HorusClSv_MsgOptions mops; 	
    	mops.destList += ent;		// ent must be in the view!
    
    	memb.scast(msg, mops);		// send msg to servers and ent only
  • The lsend downcall sends a message to the (empty by default) list of destinations contained in the destList. For example:
    	HorusMessage msg; 	
    	msg << HorusString("world");
    	HorusEntity ent1, ent2;
    	....
    	HorusClSv_MsgOptions mops; 	
    	mops.destList += ent1; 	
    	mops.destList += ent2;		// ent1 and ent2 must be in the view!
    
    	memb.lsend(msg, mops);
    An invocation of a message downcall (send, cast, scast, or lsend) eventually results in a corresponding upcall method invoked at the destination(s). The message upcall methods of the HorusClSv class are declared as follows:
    	void clSvCast_Upcall(HorusEntity &origin, HorusMessage &msg); 	
    	void clSvScast_Upcall(HorusEntity &origin, HorusMessage &msg); 	
    	void clSvSend_Upcall(HorusEntity &origin, HorusMessage &msg); 	
    	void clSvLsend_Upcall(HorusEntity &origin, HorusMessage &msg);


    Monitoring Group Membership Changes

    When a VIEW event arrives, the VIEW upcall is invoked. It is declared as follows:
    	void clSvView_Upcall(HorusClSv_ViewData& viewData);
    The message and VIEW upcalls are defined with default no-op bodies, which are to be overloaded by the application as necessary.

    The HorusClSv_ViewData structure is a subclass of HorusGrpMemb_ViewData and adds a number of new fields:

    	struct HorusClSv_ViewData : HorusGrpMemb_ViewData {   
    		HorusEntityList servers, newServers, departedServers;   
    		HorusEntityList xferServers, newXferServers, departedXferServers;   	
    		HorusEntityList clients, newClients, departedClients;   
    		HorusEntity serverCoordinator, oldServerCoordinator;   	
    		HorusViewType myXferType, viewType;   	
    		HorusClSvState state, oldState;   	
    		int myServerRank, myOldServerRank;   	
    		HorusMbrshipOption myMbrshipType;  // client or server   	
    		int startXfer;           
    	};
  • The servers, xferServers, and clients lists are disjoint. Each member of the group belongs to exactly one of those lists.

  • The servers field contains the list of normal servers in the current view. The xferServers field contains the list of servers that are in the process of state transfer. The clients field contains the list of clients in the current view.

  • The myXferType and myMbrshipType fields are the same as the xferType and mbrshipType options specified in the call to join.

  • The viewType field tells whether a state transfer is currently being done within the group. If that's the case, the value of viewType specifies the protection level for the current view. If a message has a safety level not appropriate for the view in which it is sent, then it will be delayed until the completion of all state transfers (at which time the viewType will become NO_XFER, and all delayed messages will be replayed).

    NOTE: If an "unsafe" point-to-point message is attempted to be sent during a state transfer, that will result in panic. Only multicast messages may be delayed and replayed. [this is not enforced yet, but will be]

  • The state and oldState fields tell the current and previous states of the group member. Their possible values are
    	HORUS_CLSV_STATE_CLIENT_NORMAL,
    	HORUS_CLSV_STATE_BECOMING_SERVER, 
    	HORUS_CLSV_STATE_SERVER_XFER, 
    	HORUS_CLSV_STATE_SERVER_XFER_DONE, 
    	HORUS_CLSV_STATE_SERVER_NORMAL. 

  • The startXfer flag is set when a state transfer is to be (re)started. That happens when the state of the group member changes to SERVER_XFER. The interface to the state transfer functionality provided by HorusClSv is discussed in the section below.


    State Transfer

    See the section on State Transfer in HOT for a discussion of the protocol.

    Once the startXfer flag is set (see a description of the HorusClSv_ViewData structure in the previous section), the joining server may start requesting the state from normal servers. The state is requested by invocations of the askState downcall, which is declared as follows:

    	void askState(HorusEntity &svr, HorusMessage &msg);
    The svr argument specifies the server from which a portion of the state is requested. The msg argument contains the request message that tells the server which portion of the state is being asked for.

    An invocation of the askState downcall results in the askState_Upcall method invoked at the destination server:

    	void askState_Upcall(HorusEntity& origin, HorusMessage &msg);
    The server may call the sendState method to send a portion of the state to the joining server:
    	void sendState(HorusEntity &joiningSvr, HorusMessage &msg);
    The joiningSvr argument is the address of the joining server that has asked for the state. The msg argument contains the reply message with the requested portion of the global state in it.

    An invocation of the sendState downcall results in the rcvState_Upcall method invoked at the joiningSvr:

    	void rcvState_Upcall(HorusEntity& origin, HorusMessage &msg);
    The origin argument tells who the state message has arrived from. The msg argument contains the reply message with the requested portion of the state in it.

    When a joining server decides it has received all the state, it should call the xferDone method. After that, a new view will eventually arrive, in which that server will be in the servers list.

    The state transfer interface of the HorusClSv class is rather low-level, although most general. The HorusCSX class (discussed in the section below) provides a higher-level state transfer interface, which is implemented on top of HorusClSv. Most applications will use HorusCSX rather than HorusClSv.


    HorusCSX

    HorusCSX (a subclass of HorusClSv) provides a higher-level interface to state transfer. When state transfer needs to be (re)started, a special XFER upcall is invoked in a separate thread. The body of the upcall function is to be overloaded by the programmer to actually do the state transfer.

    It may happen (for example, as a result of a group merge) that state transfer will be started more than once at a given server. On the other hand, a state transfer may be terminated before completion (for example, if all normal servers crash during state transfer). To insure state transfer consistency in the possible presence of multiple state transfers, HorusCSX assigns a unique ID to every state transfer.

    The interface to the state transfer functionality is described in the sections below.



    XFER_Upcall

    		void XFER_Upcall(HorusXferID &xferID);
  • This upcall method is invoked at the joining server in a separate thread when state transfer is (re)started. The argument, xferID, identifies this instance of the transfer. The default implementation of XFER_Upcall only invokes xferDone(xferID) (which completes the state transfer), and returns. This method will be overloaded in a derived class to implement an application-specific state transfer functionality.



    getState

    		void getState(HorusXferID &xferID, 
    			      HorusMessage &requestMsg,
    			      HorusMessage &stateMsg,
    			      HorusXferStatus &xferStatus);
  • This blocking method is invoked by a joining server (usually from within XFER_Upcall) to request a portion of the state from one of normal servers. It can be called as many times as necessary. xferID is the same as the argument to XFER_Upcall; requestMsg contains the request message specifying which portion of the state is being requested. When getState returns, stateMsg contains the reply message with the requested part of the state in it.

    The xferStatus is an OUT argument. When a call to getState returns, xferStatus will have the value of HORUS_XFER_OK if the request has succeeded, and HORUS_XFER_TERMINATED if the transfer has been prematurely terminated (usually because of a group merge or a total failure of all normal servers). If the transfer has been terminated, XFER_Upcall should return without further attempts to get the state.



    xferDone

    		void xferDone(HorusXferID &xferID);
    This method must be called by a joining server (usually from XFER_Upcall) when its state transfer has completed. The xferID argument is the same as the one passed to XFER_Upcall.



    askState_Upcall

    		void askState_Upcall(HorusEntity &origin, 
    				     HorusXferID &xferID,
    				     HorusMessage &requestMsg);
    This upcall method is invoked at a normal server when a state request from a joining server arrives (as a result of calling getState by the joining server). The origin argument is the entity of the joining server; xferID identifies the state transfer; requestMsg contains the request message, specifying which portion of the state is being requested.

    Every invocation of askState_Upcall must eventually be followed by a call to the sendState function (discussed below), which sends the requested part of the state to the joining server.



    sendState

    		void sendState(HorusEntity &dest,
    			       HorusXferID &xferID, 
    			       HorusMessage &stateMsg);
    This function should be eventually called after every invocation of askState_Upcall at a normal server. The dest argument is the same as the origin argument to the askState_Upcall, and xferID is the same as in the askState_Upcall. The stateMsg contains the requested portion of the state.



    Below is an example of using HorusCSX state transfer interface. We assume the global state is represented by two HorusEntityList's, stateList1 and stateList2. The state is transferred in two steps (stateList1 first, then stateList2).

    	class MyCSX : public HorusCSX {
    	public:    
    
    		MyCSX(HorusCSX_Options ops): HorusCSX(ops) {}
    	
    	protected:
    
    		void XFER_Upcall(HorusXferID &xferID) {
    			// ***** starting state xfer *****
    		
    			resetState();			
    			HorusMessage requestMsg, stateMsg;
    			HorusXferStatus xferStatus;
    
    			// request first half of the state
    			requestMsg << 1;
    			getState(xferID, requestMsg, 
    				stateMsg, xferStatus);
    			if (xferStatus == HORUS_XFER_TERMINATED) {
    				cerr << "** xfer terminated **" << endl;
    				return;
    			}
    		
    			// extract 1st half of the state
    			stateMsg >> stateList1;
    
    			// request second half of the state
    			requestMsg.reset();
    			requestMsg << 2;
    			getState(xferID, requestMsg, stateMsg, xferStatus);
    			if (xferStatus == HORUS_XFER_TERMINATED) {
    				cerr << "** xfer terminated **" << endl;
    				resetState();
    				return;
    			}    
    
    			// extract 2nd half of the state
    			stateMsg >> stateList2;
    
    			// complete the xfer
    			xferDone(xferID);
    		}
    
    		void askState_Upcall(HorusEntity &origin,
    					HorusXferID &xferID,
    					HorusMessage &requestMsg) {
    			// ***** got state request *****
        		
    			// figure out which part of the state to send
    			int locator;
    			requestMsg >> locator;
    
    			// inject requested portion of the state
    			HorusMessage stateMsg;
    			if (locator == 1) {
    				stateMsg << stateList1;
    			else if (locator == 2)
        				stateMsg << stateList2;
    
    			// send state to the joining server
    			sendState(origin, xferID, stateMsg);	
    		}
    	
    		void resetState() {
    			// clean up the state lists
    			stateList1.clear();
    			stateList2.clear();
    		}
    
    		void csxView_Upcall(HorusCSX_ViewData &viewData) {
    			// ***** GOT A VIEW *****"
    		}
    
    		void csxCast_Upcall(HorusEntity& origin, HorusMessage& msg) {
    			// ****** GOT A CAST ******* 
    		}
    
    		void csxSend_Upcall(HorusEntity& origin, HorusMessage& msg) {
    			// ****** GOT A SEND *******
    		}
    
    		void csxScast_Upcall(HorusEntity& origin, HorusMessage& msg) {
    			// ****** GOT AN SCAST *******
    		}
    
    		void csxLsend_Upcall(HorusEntity& origin, HorusMessage& msg) {
    			// ****** GOT AN LSEND *******
    		}
    
    	private:
    
    		HorusEntityList stateList1, stateList2;
    	};

    send mail to alexey@cs.cornell.edu