Maestro Open Toolkit: Group Members

This document is a part of online Ensemble documentation, under Maestro Open Toolkit.

Maestro_GroupMember

The Maestro_GroupMember class implements an abstract data type for group members in the Ensemble system. There are public downcalls for joining groups, sending messages, etc., and (protected) callbacks that are invoked when group events arive. Typically an application will define a subclass of the Maestro_GroupMember class, which will overload callbacks/downcalls of Maestro_GroupMember to implement application-specific functionality.

Code Examples:

  • Using Maestro_GroupMember in the multi-threaded mode
  • Using Maestro_GroupMember in the single-threaded mode

    Index of Topics:

  • Starting Ensemble
  • Creating Maestro_GroupMember Objects
  • Joining and leaving groups
  • Sending and receiving messages
  • Participating in membership changes
  • Using external failure detectors
  • Changing protocol stacks

    Index of Downcalls/Callbacks:

  • start
  • join
  • leave
  • send
  • cast
  • suspect
  • changeProtocol
  • requestNewView
  • myEndpID
  • grpMemb_ReceiveCast_Callback
  • grpMemb_ReceiveSend_Callback
  • grpMemb_ViewMsg_Callback
  • grpMemb_AcceptedView_Callback
  • grpMemb_Block_Callback
  • grpMemb_Exit_Callback
  • grpMemb_Heartbeat_Callback


    Starting Ensemble

    The initialization of Ensemble is done automatically when running in the multi-threaded mode. However, in the single-threaded mode, Ensemble has to be started by the application explicitly by making a call to the static Maestro_GroupMember::start() function:
    		void start(char **argv);
    The argv argument specifies the command-line parameters (should be set to NULL if none are specified).


    Creating Maestro_GroupMember Objects

    The Maestro_GroupMember class provides the following constructor:

    	Maestro_GroupMember(Maestro_GrpMemb_Options &membOps);
    Initialization parameters are specified with the membOps argument. The Maestro_GrpMemb_Options structure contains the following fields:
  • unsigned heartbeatRate -- The frequency of heartbeat callbacks, in milliseconds.
  • Maestro_String groupName -- The name of the group to be joined.
  • Maestro_String protocol -- The Ensemble protocol stack to be used. Protocols in the stack are separated by colons, for example:
    "Top:Heal:Switch:Leave:Inter:Intra:Elect:Merge:Sync:Suspect:Top_appl:Pt2pt:Frag:Stable:Mnak:Bottom", The use of raw protocol stacks is discouraged. Instead, protocol properties should be specified (see below).
  • Maestro_String properties -- Group protocol properties required by the application. The following properties are supported:
  • Agree -- Agreed (safe) delivery
  • Gmp -- Group-membership properties
  • Sync -- View synchronization
  • Total -- Totally ordered messages
  • Heal -- Partition healing
  • Switch -- Protocol switching
  • Auth -- Authentication
  • Causal -- Causally ordered broadcasts
  • Subcast -- Subcast pt2pt messages
  • Frag -- Fragmentation-reassembly
  • Scale -- Scalability
  • Suspect -- Failure detection
  • Flow -- Flow control
  • Migrate -- Process migration
  • Privacy -- Encryption of application data
  • Rekey -- Support for rekeying the group
  • Primary -- Primary partition detection
  • Properties are separated by colons, for example: "Gmp:Sync:Heal:Switch:Suspect:Flow:Primary".
  • Maestro_String params -- Parameters for the group. TODO: List available parameters
  • Maestro_String transports -- The list of available transport protocols. Protocols in the list are separated by colons, for example: "UDP:ATM".
  • Maestro_ErrorHandler *errorHandler -- The error handler to be used by the group-member object. If errorHandlerset field is set to NULL, the default error handler is used.
  • Maestro_ThreadMode threadMode -- Specifies the threading mode for Maestro. The value of the threadMode field can be set to MAESTRO_MODE_SINGLE_THREADED or MAESTRO_MODE_MULTI_THREADED (the default) to select single-threaded/ multi-threaded mode respectively. All group member objects created within a process must be initialized with the same threading mode.
  • int groupdFlag -- The flag should be set if Ensemble Membership Service is to be used.
  • int viewMsgFlag -- The flag should be set if view messages are required, in which case the grpMemb_ViewMsg_Callback is invoked at the coordinator allowing it to distribute additional data to view members along with the view structure.
  • char **argv -- Command-line parameters.

  • Joining and Leaving a Group

    A Maestro_GroupMember object does not automatically join the group at the initialization time. To join the group, the join() method should be invoked:
    	void join();
  • A call to join() is eventually followed by an invocation of the grpMemb_AcceptedView_Callback() by Ensemble.

  • In the multi-threaded mode, a call to join() returns only after the grpMemb_AcceptedView_Callback() method has been invoked by Ensemble.

  • A member object cannot join multiple groups and cannot join a group more than once.

  • A member can leave the group by calling the leave() method:
    	void leave():
    After the member completes leaving the group, the grpMemb_Exit_Callback() method is invoked by Ensemble. grpMemb_Exit_Callback() is defined in the Maestro_GroupMember class as a no-op function (to be overloaded in subclasses of Maestro_GroupMember):
    	void grpMemb_Exit_Callback() {}
    It is guaranteed that grpMemb_Exit_Callback() will be the last callback received by a group member.

  • After a group member leaves the group, it can rejoin by calling the join() method again.

  • A Maestro_GroupMember object leaves the group automatically at the time of deallocation, by calling the leave() method from the destructor.


    Sending and Receiving Messages

    After a Maestro_GroupMember object has joined a group, it can send point-to-point and multicast messages to it.

    The Maestro_GroupMember class provides the following downcall methods for sending point-to-point (send) and multicast (cast) messages:

    	void send(Maestro_EndpID &dest, 
    		  Maestro_Message &msg);
    
      	void send(Maestro_EndpID &dest, 
    		  Maestro_Message &msg,
    		  Maestro_MsgSendView &sendView);
    
    	void send(Maestro_EndpID &dest, 
    		  Maestro_Message &msg, 
    		  Maestro_MsgSendView &sendView,
    		  Maestro_GrpMemb_MsgOptions &msgOps);
    
    	void cast(Maestro_Message &msg);
    
    	void cast(Maestro_Message &msg, 
    		  Maestro_MsgSendView &sendView);
    
    	void cast(Maestro_Message &msg, 
    		  Maestro_MsgSendView &sendView, 
    		  Maestro_GrpMemb_MsgOptions &msgOps);
  • The dest argument in send methods specifies the endpoint ID of the destination.

  • The msg argument contains the message to be sent.

  • sendView is an "out" argument. It specifies the view in which the message will be delivered to its destination(s). The Maestro_MsgSendView type is defined as follows:
    	enum Maestro_MsgSendView {
    		MAESTRO_MSG_SEND_UNSPECIFIED_VIEW, // The view of msg delivery is unspecified
    		MAESTRO_MSG_SEND_CURRENT_VIEW,	   // The msg will be delivered in the current view	
    		MAESTRO_MSG_SEND_NEXT_VIEW	   // The msg will be delivered in the next view
    	};

    If a Maestro_GroupMember object is created over a protocol stack that provides the Virtual Synchrony property, all messages are delivered in the same view in which they are sent. However, if a send or a cast method is invoked during a view change, the message will be sent (and hence delivered) in the future view. The sendView argument can then be used to determine the view in which a given message will be sent and delivered.

  • The msgOps argument has no effect (the Maestro_GrpMemb_MsgOptions structure is empty). It is intended to be overloaded in subclasses of the Maestro_GroupMember class.
  • When a point-to-point or a multicast message is received by a group member, the grpMemb_ReceiveSend_Callback (respectively, grpMemb_ReceiveCast_Callback) method is invoked by Ensemble. The message callbacks are defined in Maestro_GroupMember as follows:
    	void grpMemb_ReceiveCast_Callback(Maestro_EndpID &origin,
    					  Maestro_Message &msg) {}
    
    	void grpMemb_ReceiveSend_Callback(Maestro_EndpID &origin,
    					  Maestro_Message &msg) {}
    By default, no action is taken when a message is received. Message callbacks are intended to be overloaded in subclasses of the Maestro_GroupMember class to implement application-specific functionality.


    Participating in Membership Changes

    When member objects join or leave the group (or become unavailable), Ensemble starts the view-change protocol to bring the view of group members up to date.

    In the first phase of the view-change protocol, the group is blocked, so that members cannot send any new messages. Ensemble invokes the grpMemb_Block_Callback() method of a Maestro_GroupMember object to notify it that the group is about to be blocked. The application can send messages for the current view from within grpMemb_Block_Callback(). However, after the callback returns, all subsequent message downcalls will be delayed by Ensemble until the next view is installed.

    The grpMemb_Block_Callback() method is defined in Maestro_GroupMember as follows:

    	void grpMemb_Block_Callback() {}
    By default, no action is taken when a group is being blocked. However, grpMemb_Block_Callback() can be overloaded in subclasses of the Maestro_GroupMember class to implement application-specific behavior.

    When the new view is ready to be installed, the grpMemb_ViewMsg_Callback() method is invoked at the coordinator. The grpMemb_ViewMsg_Callback() function is defined in Maestro_GroupMember as follows:

    	void grpMemb_ViewMsg_Callback(Maestro_GrpMemb_ViewData &viewData,
    				      Maestro_Message &viewMsg) {}
  • viewData describes the new view.
  • viewMsg is an "out" parameter. The coordinator can include some data with the view message (by writing to viewMsg), which will be delivered to group members along with the new view.

    By default, the grpMemb_ViewMsg_Callback is implemented in Maestro_GroupMember as a no-op function. However, it can be overloaded in subclasses of the Maestro_GroupMember class to implement application-specific behavior.

    After the call to grpMemb_ViewMsg_Callback returns, the new view is installed at group members included in it. When a group member receives a new view, the grpMemb_AcceptedView_Callback method is invoked. Upon returning from the call to grpMemb_AcceptedView_Callback, Ensemble unblocks the group, so that messages can be sent again. The grpMemb_AcceptedView_Callback method is defined in Maestro_GroupMember as follows:

    	void grpMemb_AcceptedView_Callback(Maestro_GrpMemb_ViewData &viewData,
    					   Maestro_Message &msg) {}
  • viewData describes the new view. The Maestro_GrpMemb_ViewData structure has the following fields:
  • Maestro_String version -- The version of Ensemble in use.
  • Maestro_String groupName -- Name of the group.
  • Maestro_EndpID myEndpID -- The endpoint ID of the member.
  • Maestro_EndpList members -- The list of members in the new view.
  • Maestro_EndpList newMembers -- The list of members added to the member's old view.
  • Maestro_EndpList departedMembers -- The list of members removed from the member's old view.
  • int nmembers -- The size of the new view.
  • int myRank -- The member's rank in the new view.
  • int myOldRank -- The member's rank in its old view.
  • Maestro_EndpID coordinator -- The endpoint ID of the coordinator of the new view.
  • Maestro_EndpID oldCoordinator -- The endpoint ID of the coordinator of the member's old view.
  • Maestro_String protocol -- The protocol stack used by Ensemble in the new view.
  • int groupdFlag -- The flag is set if group daemon (Ensemble Membership Service) is in use.
  • Maestro_ViewID viewID -- The ID of the view.
  • Maestro_String params -- Group parameters.
  • int primaryFlag -- The flag is set if the view is primary.
  • msg contains the view message filled in by the coordinator.


    Using External Failure Detectors

    Maestro supports external failure detectors via the suspect() method, which is defined in Maestro_GroupMember as follows:
    	void suspect(Maestro_EndpList &suspects);
    The suspects parameter contains the list of members which are believed to be faulty and should be removed from the view.

    NOTE: The suspect() function is not supported in the current revision of Ensemble (0.35).


    Changing Protocol Stacks

    The protocol stack of a view can be changed with a call to the changeProtocol method of Maestro_GroupMembers. The changeProtocol function is declared as follows:
    	 void changeProtocol(Maestro_String &protocol);
    The protocol parameter specifies the new stack. Protocols in a stack are separated by colons, such as in
    "Top:Heal:Switch:Leave:Inter:Intra:Elect:Merge:Sync:Suspect:Top_appl:Pt2pt:Frag:Stable:Mnak:Bottom".

    Only the coordinator of a view can request a replacement of a protocol stack. Calls to changeProtocol made by non-coordinator view members are ignored.


    Miscellaneous Functions

  • The coordinator of a view can request a view change with a requestNewView() method, which is declared as follows:
    	void requestNewView();
  • The endpoint ID of a group member object can be obtained (after the member has joined the group) with the myEndpID() method, which is declared as follows:
    	void myEndpID(Maestro_EndpID &eid);
  • The Maestro_GroupMember class defines the grpMemb_Heartbeat_Callback() function, which is periodically invoked by Ensemble as a callback. grpMemb_Heartbeat_Callback() is defined as a no-op function as follows:
    	void grpMemb_Heartbeat_Callback(unsigned time) {}
    It is intended that grpMemb_Heartbeat_Callback() will be overloaded in subclasses of Maestro_GroupMember to implement application-specific functionality.


    send mail to alexey@cs.cornell.edu