This document is a part of the online Horus Documentation, under Horus Object Tools.
The main component of HOT is a hierarchy of classes that implement group members and clients/servers with state transfer. The classes are HorusGroupMember (implements group members), HorusClSv (implements clients/servers and state transfer protocol), and HorusCSX (clients/servers with higher-level state transfer interface). They are discussed in detail in the following sections.
When creating a group member object, you pass a HorusGrpMemb_Options structure as the argument. The HorusGrpMemb_Options structure contains the following fields:
struct HorusGrpMemb_Options { HorusEntity groupEntity; char *groupName; char *protocolStack; HorusErrorHandler *errorHandler; HorusNameServer *nameServer; };
Thus, either groupEntity has to be initialized, or the group address must be retrievable from the name server.
Two other options are protocolStack and errorHandler. The protocolStack option specifies the stack of Horus layers used in the group (the default stack is defined in the file config.h), and errorHandler points to an error-handler object. If errorHandler is NULL (the default), then HorusErrorHandler is used. For example:
HorusGrpMemb_Options ops; ops.groupName = "lapa"; ops.protocolStack = "BUFFER:ORDER:CAUSAL:UNPACK:TOTAL:MERGE(auto=0):VIEWS"; HorusGroupMember memb(ops);The groupName has been specified; therefore the address of the group will be looked up with the name server under "lapa". Since the nameServer option has not been set, the default name server (HorusNameServer) will be used.
After a HorusGroupMember object has been created, you should call the join downcall to actually join the group.
NOTE: A HorusGroupMember object may join one group only.
The join downcall accepts an (optional) HorusGrpMemb_JoinOptions argument, which contains the following fields:
struct HorusGrpMemb_JoinOptions { char *contactName; HorusMessage msg; };
HorusGrpMemb_JoinOptions jops; jops.contactName = "lapa_contact"; memb.join(jops);The join downcall will block until the first VIEW upcall has been received.
HorusMessage msg; msg << HorusString("hello world!"); memb.cast(msg); // send msg to all members of the group HorusEntity dest; .... memb.send(dest, msg); // send msg to dest (must be in the view)Each invocation of a send or cast downcall eventually results in the SEND or CAST upcall method invoked at the message destination(s). The message upcalls are defined with no-op bodies, and are to be overloaded by subclasses of HorusGroupMember as required by the application.
The SEND/CAST upcalls have the following declarations:
void grpMembCast_Upcall(HorusEntity& origin, HorusMessage &msg); void grpMembSend_Upcall(HorusEntity& origin, HorusMessage &msg);The origin argument specifies the originator of the message, and msg contains the message.
Whenever a membership change occurs, the VIEW upcall is invoked at each member of the group. The VIEW upcall has the no-op body (to be overloaded by subclasses of HorusGroupMember), and is declared as follows:
void grpMembView_Upcall(HorusGrpMemb_ViewData&);The HorusGrpMemb_ViewData structure contains membership information, and is defined as follows:
struct HorusGrpMemb_ViewData { HorusEntity myEntity, myGroup; HorusEntityList members, newMembers, departedMembers; int nmembers, myRank, myOldRank; HorusEntity coordinator, oldCoordinator; HorusMessage msg; };
For example:
class myMember: public HorusGroupMember { public: myMember(HorusGrpMemb_Options &ops): HorusGroupMember(ops) {} void grpMembView_Upcall(HorusGrpMemb_ViewData& vd) { cout << "GOT A VIEW: " << endl << vd.members; } void grpMembCast_Upcall(HorusEntity& origin, HorusMessage &msg) { cout << "GOT A CAST FROM " << endl << origin; } void grpMembSend_Upcall(HorusEntity& origin, HorusMessage &msg) { cout << "GOT A SEND FROM " << endl << origin; } };
void view(HorusMessage&);By default, an empty message is sent with the VIEW event. However, the view method can be overloaded to include some useful information into the view message.
void JOIN_REQUEST_upcall(HorusEvent *event); void JOIN_DENIED_upcall(HorusEvent *event); void JOIN_READY_upcall(HorusEvent *event); void JOIN_FAILED_upcall(HorusEvent *event); void FLUSH_upcall(HorusEvent *event);and so on. The upcalls may be overloaded if it is necessary to redefine the membership-level behavior of a HorusGroupMember subclass.
The options for the HorusClSv constructor are the same as for HorusGroupMember. For example:
HorusClSv_Options ops; ops.groupName = "lapa"; ops.protocolStack = "BUFFER:ORDER:CAUSAL:UNPACK:TOTAL:MERGE(auto=0):VIEWS"; HorusClSv memb(ops);
Whereas in HorusGroupMember all members of the group have the same properties, the HorusClSv class makes a distinction between clients and servers. When a HorusClSv object joins a group, its membership type (either client or server) is specified.
The options structure for the join downcall is a subclass of HorusGrpMemb_Options, and is defined as follows:
struct HorusClSv_JoinOptions : HorusGrpMemb_JoinOptions { HorusMbrshipOption mbrshipType; HorusXferType xferType; };
HorusClSv_Options ops; ops.groupName = "lapa"; HorusClSv memb(ops); // create a member of "lapa" HorusClSv_JoinOptions jops; jops.mbrshipType = HORUS_SERVER; jops.xferType = HORUS_ATOMIC_XFER; memb.join(jops); // join "lapa" as a server assert(memb.isServer()); assert(!memb.isClient());The difference between FREE, PROTECTED, and ATOMIC state transfer is in the safety levels of messages that can be sent during state transfer (see the discussion in the section below).
The safety level of a message is specified in the options argument to a message-sending downcall. The options structure is defined as follows:
struct HorusClSv_MsgOptions { HorusMsgXferSafety msgXferSafety; HorusEntityList destList; };
The correspondence between message safety levels and state transfer types is as follows:
Here is an example:
// create a member of group "lapa" HorusClSv_Options ops; ops.groupName = "lapa"; MyClSv memb(ops); // join "lapa" as a server, with PROTECTED state transfer HorusClSv_JoinOptions jops; jops.mbrshipType = HORUS_SERVER; jops.xferType = HORUS_PROTECTED_XFER; memb.join(jops); HorusClSv_MsgOptions mops; // suppose the messages below are sent during state transfer.. // send a GENERIC message (*delayed* during PROTECTED xfer) mops.msgXferSafety = HORUS_MSG_GENERIC; HorusMessage msg; memb.cast(msg, mops); HorusMessage msg1; memb.cast(msg1); // by default, a message is GENERIC (delayed) // send a SAFE message (not delayed during PROTECTED xfer) mops.msgXferSafety = HORUS_MSG_SAFE; HorusMessage msg2; memb.cast(msg, mops); // send an XFER message (never delayed) mops.msgXferSafety = HORUS_MSG_XFER; HorusMessage msg3; memb.cast(msg3, mops);Besides send and cast downcalls, the HorusClSv class defines two new methods, called scast ("server cast") and lsend ("list send").
HorusMessage msg; msg << HorusString("hello"); memb.scast(msg); // send msg to servers only .... HorsEntity ent; .... HorusClSv_MsgOptions mops; mops.destList += ent; // ent must be in the view! memb.scast(msg, mops); // send msg to servers and ent only
HorusMessage msg; msg << HorusString("world"); HorusEntity ent1, ent2; .... HorusClSv_MsgOptions mops; mops.destList += ent1; mops.destList += ent2; // ent1 and ent2 must be in the view! memb.lsend(msg, mops);An invocation of a message downcall (send, cast, scast, or lsend) eventually results in a corresponding upcall method invoked at the destination(s). The message upcall methods of the HorusClSv class are declared as follows:
void clSvCast_Upcall(HorusEntity &origin, HorusMessage &msg); void clSvScast_Upcall(HorusEntity &origin, HorusMessage &msg); void clSvSend_Upcall(HorusEntity &origin, HorusMessage &msg); void clSvLsend_Upcall(HorusEntity &origin, HorusMessage &msg);
void clSvView_Upcall(HorusClSv_ViewData& viewData);The message and VIEW upcalls are defined with default no-op bodies, which are to be overloaded by the application as necessary.
The HorusClSv_ViewData structure is a subclass of HorusGrpMemb_ViewData and adds a number of new fields:
struct HorusClSv_ViewData : HorusGrpMemb_ViewData { HorusEntityList servers, newServers, departedServers; HorusEntityList xferServers, newXferServers, departedXferServers; HorusEntityList clients, newClients, departedClients; HorusEntity serverCoordinator, oldServerCoordinator; HorusViewType myXferType, viewType; HorusClSvState state, oldState; int myServerRank, myOldServerRank; HorusMbrshipOption myMbrshipType; // client or server int startXfer; };
NOTE: If an "unsafe" point-to-point message is attempted to be sent during a state transfer, that will result in panic. Only multicast messages may be delayed and replayed. [this is not enforced yet, but will be]
HORUS_CLSV_STATE_CLIENT_NORMAL, HORUS_CLSV_STATE_BECOMING_SERVER, HORUS_CLSV_STATE_SERVER_XFER, HORUS_CLSV_STATE_SERVER_XFER_DONE, HORUS_CLSV_STATE_SERVER_NORMAL.
Once the startXfer flag is set (see a description of the HorusClSv_ViewData structure in the previous section), the joining server may start requesting the state from normal servers. The state is requested by invocations of the askState downcall, which is declared as follows:
void askState(HorusEntity &svr, HorusMessage &msg);The svr argument specifies the server from which a portion of the state is requested. The msg argument contains the request message that tells the server which portion of the state is being asked for.
An invocation of the askState downcall results in the askState_Upcall method invoked at the destination server:
void askState_Upcall(HorusEntity& origin, HorusMessage &msg);The server may call the sendState method to send a portion of the state to the joining server:
void sendState(HorusEntity &joiningSvr, HorusMessage &msg);The joiningSvr argument is the address of the joining server that has asked for the state. The msg argument contains the reply message with the requested portion of the global state in it.
An invocation of the sendState downcall results in the rcvState_Upcall method invoked at the joiningSvr:
void rcvState_Upcall(HorusEntity& origin, HorusMessage &msg);The origin argument tells who the state message has arrived from. The msg argument contains the reply message with the requested portion of the state in it.
When a joining server decides it has received all the state, it should call the xferDone method. After that, a new view will eventually arrive, in which that server will be in the servers list.
The state transfer interface of the HorusClSv class is rather low-level, although most general. The HorusCSX class (discussed in the section below) provides a higher-level state transfer interface, which is implemented on top of HorusClSv. Most applications will use HorusCSX rather than HorusClSv.
It may happen (for example, as a result of a group merge) that state transfer will be started more than once at a given server. On the other hand, a state transfer may be terminated before completion (for example, if all normal servers crash during state transfer). To insure state transfer consistency in the possible presence of multiple state transfers, HorusCSX assigns a unique ID to every state transfer.
The interface to the state transfer functionality is described in the sections below.
void XFER_Upcall(HorusXferID &xferID);
void getState(HorusXferID &xferID, HorusMessage &requestMsg, HorusMessage &stateMsg, HorusXferStatus &xferStatus);
The xferStatus is an OUT argument. When a call to getState returns,
xferStatus
will have the value of HORUS_XFER_OK if the request has succeeded, and
HORUS_XFER_TERMINATED if the transfer has been prematurely
terminated (usually because of a group merge or a total failure of all normal
servers). If the transfer has been terminated, XFER_Upcall should return
without further attempts to get the state.
void xferDone(HorusXferID &xferID);This method must be called by a joining server (usually from XFER_Upcall) when its state transfer has completed. The xferID argument is the same as the one passed to XFER_Upcall.
void askState_Upcall(HorusEntity &origin, HorusXferID &xferID, HorusMessage &requestMsg);This upcall method is invoked at a normal server when a state request from a joining server arrives (as a result of calling getState by the joining server). The origin argument is the entity of the joining server; xferID identifies the state transfer; requestMsg contains the request message, specifying which portion of the state is being requested.
Every invocation of askState_Upcall must eventually be followed by a call to
the sendState function (discussed below), which sends the requested part of the
state to the joining server.
void sendState(HorusEntity &dest, HorusXferID &xferID, HorusMessage &stateMsg);This function should be eventually called after every invocation of askState_Upcall at a normal server. The dest argument is the same as the origin argument to the askState_Upcall, and xferID is the same as in the askState_Upcall. The stateMsg contains the requested portion of the state.
Below is an example of using HorusCSX state transfer interface. We assume the global state is represented by two HorusEntityList's, stateList1 and stateList2. The state is transferred in two steps (stateList1 first, then stateList2).
class MyCSX : public HorusCSX { public: MyCSX(HorusCSX_Options ops): HorusCSX(ops) {} protected: void XFER_Upcall(HorusXferID &xferID) { // ***** starting state xfer ***** resetState(); HorusMessage requestMsg, stateMsg; HorusXferStatus xferStatus; // request first half of the state requestMsg << 1; getState(xferID, requestMsg, stateMsg, xferStatus); if (xferStatus == HORUS_XFER_TERMINATED) { cerr << "** xfer terminated **" << endl; return; } // extract 1st half of the state stateMsg >> stateList1; // request second half of the state requestMsg.reset(); requestMsg << 2; getState(xferID, requestMsg, stateMsg, xferStatus); if (xferStatus == HORUS_XFER_TERMINATED) { cerr << "** xfer terminated **" << endl; resetState(); return; } // extract 2nd half of the state stateMsg >> stateList2; // complete the xfer xferDone(xferID); } void askState_Upcall(HorusEntity &origin, HorusXferID &xferID, HorusMessage &requestMsg) { // ***** got state request ***** // figure out which part of the state to send int locator; requestMsg >> locator; // inject requested portion of the state HorusMessage stateMsg; if (locator == 1) { stateMsg << stateList1; else if (locator == 2) stateMsg << stateList2; // send state to the joining server sendState(origin, xferID, stateMsg); } void resetState() { // clean up the state lists stateList1.clear(); stateList2.clear(); } void csxView_Upcall(HorusCSX_ViewData &viewData) { // ***** GOT A VIEW *****" } void csxCast_Upcall(HorusEntity& origin, HorusMessage& msg) { // ****** GOT A CAST ******* } void csxSend_Upcall(HorusEntity& origin, HorusMessage& msg) { // ****** GOT A SEND ******* } void csxScast_Upcall(HorusEntity& origin, HorusMessage& msg) { // ****** GOT AN SCAST ******* } void csxLsend_Upcall(HorusEntity& origin, HorusMessage& msg) { // ****** GOT AN LSEND ******* } private: HorusEntityList stateList1, stateList2; };