Maestro Open Toolkit: Clients/Servers with State Transfer Protocol

This document is a part of online Ensemble documentation, under Maestro Open Toolkit.


Maestro_ClSv

Maestro_ClSv is a public subclass of Maestro_GroupMember. It implements the abstraction of clients/servers with a state transfer protocol.

Index of Topics:

  • Creating client/server objects
  • Sending messages
  • Participating in membership changes
  • State transfer

    Index of Downcalls/Callbacks:

  • cast
  • send
  • scast
  • lsend
  • askState
  • sendState
  • xferDone
  • clSv_ReceiveCast_Callback
  • clSv_ReceiveSend_Callback
  • clSv_ReceiveScast_Callback
  • clSv_ReceiveLsend_Callback
  • clSv_Block_Callback
  • clSv_ChangingView_Callback
  • clSv_ProposeView_Callback
  • clSv_AcceptedView_Callback
  • clSv_askState_Callback
  • clSv_rcvState_Callback


    Creating Maestro_ClSv Objects

    The Maestro_ClSv class provides the following constructor:
    	Maestro_ClSv(Maestro_ClSv_Options &ops);
    Maestro_ClSv_Options is a public subclass of Maestro_grpMemb_Options. In addition to fields inherited from Maestro_grpMemb_Options, Maestro_ClSv_Options has the following two public fields:
  • Maestro_MbrshipOption mbrshipType -- Specifies whether the member is created as a server (mbrshipType = MAESTRO_SERVER) or as a client (mbrshipType = MAESTRO_CLIENT, the default value).
  • Maestro_XferType xferType -- Specifies the protection level for state transfer. The value of xferType is one of the following:
  • MAESTRO_NO_XFER -- State transfer is not required (the default value).
  • MAESTRO_FREE_XFER -- Group members can send all types of messages during state transfer.
  • MAESTRO_PROTECTED_XFER -- Group members can only send "safe" (state-neutral) and state-transfer messages during state transfer.
  • MAESTRO_ATOMIC_XFER -- Only state-transfer messages can be sent during state transfer.
  • Note that clients do not receive multicasts addressed to servers and do not participate in state transfer.


    Sending/Receiving Messages

    The Maestro_ClSv class defines downcall methods for sending point-to-point messages (send), multicasts to the entire group (cast), multicasts to servers only (scast), and multicasts to group subsets (lsend). Message downcalls are declared in Maestro_ClSv as follows:
    	void cast(Maestro_Message &msg);
    
    	void cast(Maestro_Message &msg, 
    		  Maestro_MsgSendView &sendView);
    
    	void cast(Maestro_Message &msg, 
    		  Maestro_MsgSendView &sendView,
    		  Maestro_ClSv_MsgOptions &msgOps);  
    
    	void send(Maestro_EndpID &dest, 
    		  Maestro_Message &msg);  
    
    	void send(Maestro_EndpID &dest, 
    		  Maestro_Message &msg, 
    		  Maestro_MsgSendView &sendView);
    
    	void send(Maestro_EndpID &dest, 
    		  Maestro_Message &msg,
    		  Maestro_MsgSendView &sendView,
    		  Maestro_ClSv_MsgOptions &msgOps);
    		    
    	void scast(Maestro_Message &msg);
    
    	void scast(Maestro_Message &msg, 
    		   Maestro_MsgSendView &sendView);
    
    	void scast(Maestro_Message &msg, 
    		   Maestro_MsgSendView &sendView,
    		   Maestro_ClSv_MsgOptions &msgOps);
    
    	void lsend(Maestro_Message &msg, 
    		   Maestro_MsgSendView &sendView,
    		   Maestro_ClSv_MsgOptions &msgOps);
  • The dest argument specifies the endpoint ID of the destination of a point-to-point message.

  • The sendView argument has the same semantics as in message downcalls of the Maestro_GroupMember class.

  • The msg parameter contains the message to be sent.

  • The Maestro_ClSv_MsgOptions class (of the msgOps argument) is defined in Maestro_ClSv with the following public fields:
  • Maestro_MsgXferSafety msgXferSafety -- Specifies the state-transfer safety level of a message. The value of msgXferSafety can be one of the following: MAESTRO_MSG_GENERIC (the default value) -- for regular messages; MAESTRO_MSG_SAFE -- for "state-safe" messages; MAESTRO_MSG_XFER -- for state-transfer messages.

  • During MAESTRO_FREE_XFER state transfers, all messages may be sent, and it is left up to the application to guarantee consistency of the global state.

  • During MAESTRO_PROTECTED_XFER state transfers, only messages with the MAESTRO_MSG_SAFE or MAESTRO_MSG_XFER safety levels may be sent. Messages with MAESTRO_MSG_GENERIC safety level are delayed by Maestro until state transfer completes.

  • During MAESTRO_ATOMIC_XFER state transfers, only state-transfer messages (with MAESTRO_MSG_XFER safety level) may be sent. All other messages are delayed by Maestro until state transfer completes.
  • Maestro_EndpList destList -- Specifies the list of client destinations for an scast message; specifies the list of all destinations for an lsend message. By default, destList is empty.
  • An scast downcall sends a message to all servers and to an (empty by default) list of clients. The client destinations are specified with the destList option. For example:
    	Maestro_Message msg; 	
    	msg << Maestro_String("hello"); 	
    	memb.scast(msg);		// Send msg to servers only.
    	....
    	HorsEndpID eid;
    	....
    	Maestro_ClSv_MsgOptions mops; 	
    	mops.destList += eid;		// Assuming eid is a client member of the group.
    
    	memb.scast(msg, mops);		// Send msg to servers and to eid.
  • The lsend downcall sends a message to the (empty by default) list of destinations included in destList. For example:
    	Maestro_Message msg; 	
    	msg << Maestro_String("world");
    	Maestro_EndpID endp1, endp2;
    	....
    	Maestro_ClSv_MsgOptions mops; 	
    	mops.destList += endp1; 	
    	mops.destList += endp2;		// Assuming endp1 and endp2 are members of the group.
    
    	memb.lsend(msg, mops);		// Send msg to endp1 and endp2.
    When a message is received at a destination, Ensemble invokes the corresponding message callback. The Maestro_ClSv class defines the following message callbacks corresponding to cast, send, scast, and lsend downcalls respectively:
    	void clSv_ReceiveCast_Callback(Maestro_EndpID &origin, 
    				       Maestro_Message &msg)  {}
    
    	void clSv_ReceiveSend_Callback(Maestro_EndpID &origin, 
    				       Maestro_Message &msg)  {}
    
    	void clSv_ReceiveScast_Callback(Maestro_EndpID &origin, 
    					Maestro_Message &msg) {}
    
    	void clSv_ReceiveLsend_Callback(Maestro_EndpID &origin, 
    					Maestro_Message &msg) {}
    The message callbacks are defined in Maestro_ClSv as no-op functions. They are intended to be overloaded in subclasses of the Maestro_ClSv class to implement application-specific functionality.


    Participating in Group Membership Changes

    The Maestro_ClSv class defines the following callback methods invoked by Ensemble during membership changes:
    	void clSv_Block_Callback() {}
    
    	void clSv_ProposeView_Callback(Maestro_ClSv_ViewData &viewData, 
    				       Maestro_MessageList &stateMsgs, 
    				       Maestro_Message &viewMsg) {}
      
    	void clSv_AcceptedView_Callback(Maestro_ClSv_ViewData& viewData,
    					Maestro_Message &viewMsg) {}
    
    	void clSv_ChangingView_Callback(Maestro_ClSv_ViewData &viewData,
    					Maestro_Message &stateMsg) {}
    The clSv_Block_Callback(), clSv_ChangingView_Callback(), clSv_ProposeView_Callback, and clSv_AcceptedView_Callback() methods have the same semantics as corresponding membership-change-protocol callbacks of the Maestro_GroupMember class. The callbacks are defined in Maestro_ClSv as no-op functions, and are intended to be overloaded in subclasses of the Maestro_ClSv class to implement application-specific functionality.

    The Maestro_ClSv_ViewData structure adds the following fields in addition to those inherited from the Maestro_GrpMemb_ViewData class:

  • Maestro_EndpList servers -- The list of servers in the new view.
  • Maestro_EndpList newServers -- Servers added since the previous view.
  • Maestro_EndpList departedServers -- Servers removed from the previous view.
  • Maestro_EndpList xferServers -- Servers doing state transfer in the new view.
  • Maestro_EndpList newXferServers -- State-transfer servers added since the previous view.
  • Maestro_EndpList departedXferServers -- State-transfer servers removed from the previous view.
  • Maestro_EndpList clients -- The list of clients in the new view.
  • Maestro_EndpList newClients -- Clients added since the previous view.
  • Maestro_EndpList departedClients -- Clients removed from the previous view.
  • Maestro_EndpID serverCoordinator -- The server with the lowest rank in the view.
  • Maestro_EndpID oldServerCoordinator -- The server coordinator of the previous view.
  • Maestro_ViewType myXferType -- Same as the value passed to the member object's constructor.
  • Maestro_ViewType viewType -- The type of state transfer in progress (if any).
  • Maestro_ClSvState state -- The state of the member in the new view (one of MAESTRO_CLSV_STATE_CLIENT_NORMAL, MAESTRO_CLSV_STATE_BECOMING_SERVER, MAESTRO_CLSV_STATE_SERVER_XFER, MAESTRO_CLSV_STATE_SERVER_XFER_DONE, MAESTRO_CLSV_STATE_SERVER_NORMAL).
  • Maestro_ClSvState oldState -- The state of the member in the previous view.
  • int myServerRank -- The rank of the member among servers in the new view.
  • int myOldServerRank -- The rank of the member among servers in the previous view.
  • Maestro_MbrshipOption myMbrshipType -- Same as the value passed to the member object's constructor.
  • int startXfer -- The flag is set when state transfer has to be (re)started in the new view (which happens when the state of the group member changes to MAESTRO_CLSV_SERVER_XFER).
  • If state transfer is in progress in a view, the viewType field in the corresponding viewData structure specifies the required safety level (one of MAESTRO_FREE_XFER, MAESTRO_PROTECTED_XFER, MAESTRO_ATOMIC_XFER). If the safety level of a message is lower than that required by state transfer, then Maestro will delay sending the message until state transfer completes.


    State Transfer

    The Maestro_ClSv class implements a State Transfer Protocol and provides the following interface to it.

  • Maestro notifies a server member that state transfer has to be (re)started by setting the startXfer flag in the viewData structure. To request a portion of the state, the member can invoke the askState downcall, which is declared in Maestro_ClSv as follows:
    	void askState(Maestro_EndpID &svr, Maestro_Message &msg);
    The svr argument specifies the server from which a portion of the state is being requested. The msg argument contains the state request message which tells the server which part of the state is being asked for.

  • When a normal server receives a state request (submitted with a call to askState), it invokes the askState_Callback method, which is defined in Maestro_ClSv as follows:
    	void askState_Callback(Maestro_EndpID& origin, Maestro_Message &msg) {}
    The origin argument identifies the member receiving the state. The msg argument contains the state-request message.

  • When a normal server receives a state request, it should eventually respond by sending the requested portion of the state to the state-recepient member. To send a state message, the server can invoke the sendState downcall, which is declared in Maestro_ClSv as follows:
    	void sendState(Maestro_EndpID &xferSvr, Maestro_Message &msg);
    The xferSvr argument specifies the endpoint ID of the state-recepient member (it should be the same as the origin argument in the corresponding invocation of askState_Callback). The msg argument contains the reply message with the requested portion of the state in it.

  • After a state-recepient member receives a state message (sent by a normal server with a call to sendState), the member invokes the rcvState_Callback method, which is defined in Maestro_ClSv as follows:
    	void rcvState_Callback(Maestro_EndpID& origin, Maestro_Message &msg) {}
    The origin argument identifies the sender of the state message. The message is passed with the msg argument.

  • The askState_Callback and rcvState_Callback methods are defined in Maestro_ClSv as no-op functions. They are intended to be overloaded in subclasses of Maestro_ClSv to implement application-specific functionality.

  • After a state-recepient member completes state transfer, it should invoke the xferDone method. Following that, Ensemble will eventually install a new view in which the member will be included in the list of normal servers.

    The state transfer interface of the Maestro_ClSv class is at a rather low level, albeit most general. The Maestro_CSX class (a subclass of Maestro_ClSv) implements a higher-level state transfer interface, which should be adequate for many applications.


    send mail to alexey@cs.cornell.edu