

		    Things to do better next time
		    =============================


Headers
-------

Protocols add/remove headers to/from messages to carry protocol-specific
information. The Down() method may for example add a header whenever it
treated that message. It might not add a header if it did not treat the
message. Other protocols will always add a header; if they did not treat
the message, they just add a dummy header.

In both cases, we have the problem, that - if a protocol forgets to remove
or add a header - the next protocol gets confused, and may stop to work
correctly. Actually, a forgotten header might propagate up the entire
stack, causing other protocols to malfunction.

Therefore pushing/popping of headers should be replaced with a hashtable of
headers: keys are either protocol names (strings), or protocol integer
numbers (both have to be unique) and values would be headers. This prevents
errors stemming from problems like removing a header twice, or failing to
remove/add a header. As a hashtable (instead of a stack) is added to every
message, they have to be written to use as little space as possible (no use
for java.util.Hashtable !).



Interaction between protocols
-----------------------------

The concept of protocols is to have independent modules (of small
functionality). However, breaking up monolithic code into a bigger set of
small modules increases the need for modules to interact with each
other. The reason is that none of the modules implements all of the
functionality, e.g. the GMS protocol does no flushes, therefore it needs to
know of and interact with the FLUSH protocol.

Interaction between protocols is via events, e.g. the GMS protocol sends
down a FLUSH event and the waits for a FLUSH_OK event. The dependency here
is not syntactically, but semantically: GMS has to know that (a) there is a
FLUSH event that flushes the stack, (b) that there is a protocol answering
to the FLUSH event *below* it and (c) that the FLUSH_OK event (received
from below) is the reply to the FLUSH event sent down.

So, although there are no syntactic dependencies between protocols, there
are usually a great number of *semantic* dependencies. The problem with
semantic dependencies is that events are used to interact between
protocols. Their use is often synchronous, the caller needs to wait until a
response to the event is received. Since this is modelling a method call,
but is not statically (= at compile time) type-checked, all sorts of errors
can occur.

Therefore, recognizing that semantic dependencies exist between protocols,
I think it is best to model them as local (i.e. intra-stack) method calls
between different protocols. An interaction between protocols is
represented via an interface. The caller obtains a reference to a protocol
that implements this interface, and simply calls the method. The callee
implements the interface and registers itself under this interface, to be
able to be looked up by the caller. All protocols access a common
registration service (e.g. shared data structure in the protocol stack, or
a singleton object), where service providers register themselves, and
service users lookup the required service. So the above FLUSH protocol
interaction might look as follows:


	    class FLUSH extends RpcProtocol implements FlushProvider {
		  ...
		  public void Flush() {}
                  ...
                  reg.Register("FlushProvider", this);
                  ...
            }

            ...

            FlushProvider flush_svc=(FlushProvider)reg.Get("FlushProvider");
            flush_svc.Flush(timeout);


Syntactic dependencies can be check at protocol stack creation time:
service requesters specify what interfaces they will need to be present in
the stack, and service providers specify what services they offer by
registering the interface(s) they implement in the registry. If a requester
specifies a service that no other protocol implements, the stack creation
would fail. Thus, we have minimal syntactic dependency checking.





Common data structures
----------------------

Frequently different protocols need to access the same data structures,
e.g. sliding window containing messages ordered according to
seqnos. Currently, these protocols interact using events, or each of them
duplicates the structures. An example is STABLE and NAKACK. STABLE needs to
interact with NAKACK to obtain the highest seqnos seen so far (unless it
maintained its own sliding window table, but this would entail too much
overhead).

Since this is a similar dependency as shown above, it would be better to
have shared data structures in a protocol stack that can be accessed by
different layers. Thus, storage is used only once, and protocols do not
need to interact to obtain values from data structures maintained by other
protocols.

A registry could be used to register and lookup references to data
structures (objects).




