Membership and message atomicity is implemented at this layers,
the heart of which is the flush protocol. The flush
protocol is run when a member crash
is detected, or when views merge. Its intention is to finalize the
view, so that all surviving members have seen exactly the same set of
messages (virtual synchrony). The protocol works as follows: one of
the members (usually the oldest surviving member) is elected
as the coordinator of the flush (see figure 2).
The coordinator broadcasts a FLUSH message
to the (surviving) members in its view. All members first return any
unstable (see next section) messages from failed members, followed by a
FLUSH_OK reply message (note that for this it is necessary that all members
log all unstable messages). Subsequently, the members ignore
messages that they may receive from supposedly failed members, and await
another VIEW installation.
Figure 2: This picture depict four processes: A, B, C, and D. D crashes
right after sending a message M, and only C received a copy. After the
crash is detected, A starts the flush protocol by multicasting to B and
C. C sends a copy of M to A, which forwards it to B. After A has
received replies from everyone, it installs a new view by multicasting it.
Upon receiving all FLUSH_OK replies, the coordinator broadcasts any messages from failed members that are still unstable. At this point a new view may be installed. When all messages stabilize, the flush is completed. If processes fail during the process, a new round of the flush protocol may start up immediately.
Although the MBRSHIP layer is able to do its own failure recovery, it allows for external failure detection. In this case, an external service picks up communication problem-reports and other failure information, and decides whether a process is to be considered faulty or not. The output of this service can be fed to the MBRSHIP layers of any set of groups, so that they have the same (consistent) view of the environment.