3 Configuration
3.1 Command-line Arguments and Environment Variables
Ensemble applications typically support a variety of configuration
parameters. Most of these can be configured through command-line options as
well as through setting environment variables. In all cases, command-line
options override environment variables. Look in appl/harg.ml for the
authoritative list of the configuration parameters. Some are listed below as
command-line options. The corresponding environment variable for
-group_name (for example) is ENS_GROUP_NAME (the name is
capitalized and the `-' is replace with ENS_).
-
-modes arg :
- Set the default modes for an application to use. The
modes are specified giving their names in all-uppercase, each separated by
single colons (`:') and no white-space.
- -udp_port port :
- set the default UDP port for Ensemble
applications. For point to point UDP communication, this is the port
number Ensemble first tries to bind to for UDP communication (if it
is already in use Ensemble will then fail). It can be set to any
value. The default is to let the operating system choose a port to
use.
- -deering_port port :
- This is the port that Ensemble will
use for Deering IP multicast communication (if enabled). All
processes must use the same port number.
- -gossip_port port :
- sets the port that the gossip servers
use.
- -gossip_hosts arg :
- sets the hosts where applications using UDP
communication can look for gossip servers. The value should be a
colon-separated list of hostnames. The gossip server application will
only execute on these hosts. Note that you only have to execute a gossip
server on one of these hosts: applications will try each of the hosts in turn
while looking for a gossip server. However, multiple servers can be executed
for increased availability.
- -id name :
- used to give applications unique identifiers.
Usually this is set to be your user id. Setting this variable
prevents Ensemble applications run by other users from interacting
with yours. In case you do want them to interact, you should set
their variables to have the same value. If using DEERING IP
multicast, their -deering_port variable should also be set to
the same value.
- -groupd_port port :
- sets the port that the membership groupd
servers use.
- -groupd_hosts arg :
- sets the hosts that the membership
groupd servers use. Format is the same as for -gossip_hosts.
- -groupd :
- Use the membership service on the local host (see
section ??. This option may override others.
- -group_name name :
- Set the name of the application's group.
[Currently, only the ensemble application supports this.]
- -key key :
- Set the key to use for a particular application. All
messages sent and received by the application will be authenticated with this
key.
- -secure :
- Enable security enforcement. This prevents any insecure
communication transports from being initialized.
- -add_prop property :
- Adds a specific property to the
Ensemble protocol stack. See Section ?? for more
information on supported Ensemble properties.
- -remove_prop property :
- The dual of add_prop.
- -sock_buf size :
- The size of socket buffers to request
from the operating system. The default size is 52428 (the
traditional limit on Unix). If you are using Ensemble in
high-performance setting and are experiencing message loss, this is a
parameter that should be increased.
- -refcount :
- Enables reference counting of message buffers.
The default is to rely on the garbage collector to detect when a
message is no longer needed. Setting this will improve performance,
but may expose reference counting bugs in Ensemble.
- -multiread :
- Enable multiple reads on sockets. The default
is to receive and process one message from the operating system
at a time. Setting this will cause all available messages to be read
from sockets before processing any of them, which may reduce message
loss due to buffer overflow in the operating system.
- -pollcount count :
- The number of times to query the
operating system before blocking. Ensemble blocks after checking
(via the select() system call) the operating system for
messages and not finding any. Setting this to 1 will cause
Ensemble to block immediately when there are no more messages.
Setting this to a large number will cause Ensemble to busy-poll for
a longer time before blocking.
- -ranking trans:
- this is used to set the priority of
transport mechanisms. This is useful if you wish to use TCP as a
transport instead of UDP (the default). Using -ranking TCP will
cause TCP to be ranked higher than any other transport.
The following configuration parameter can only be set as an
environment variable.
-
ENS_TRACE
- : enables module initialization tracing. With this set (to
any value), modules print out their names as they initialize. This is useful
if an exception occurs during initialization because because it
enables you to narrow the problem down to one module.
3.2 Transports
[If you are only using regular UDP sockets for communication,
then you do not need to read this section.]
Perhaps the most confusing part of running Ensemble applications
comes from selecting communication transports. Communication
transports are the bottom-most part of Ensemble and are used for
sending and receiving messages on a network. There are several ways
this can be confusing and often Ensemble cannot detect that there is
a problem, so you do not get a warning. For instance, if you
configure an application so that one process is using UDP sockets for
communication and another is using NETSIM, then the two processes will
stall waiting for other processes to communicate with them on their
selected medium.
A confusing aspect of transport is that an application typically uses
two different kinds of transports: a primary transport and a gossip
transport. Normal application communication is all done over the
primary transport, which must support point-to-point communication and
may also support multicast communication. Communication between
different partitions of a group of applications uses the gossip
transport which must support ``anonymous'' multicast communication.
Applications occasionally send ``gossip'' messages with their gossip
transport to the rest of the ``world'' in order to inform other
partitions about their presence. When two partitions learn of each
other, they can then merge the partitions together. After they have
merged together, they communicate over their primary transport. This
gossip-and-merge mechanism is used when applications first start up:
an application creates its own singleton group and then merges with
any other already existing partitions through gossiping and merging.
Thus, if there is a problem with the gossip transport, you will tend
to have a bunch of applications in singleton groups that never merge.
If there is a problem with the primary transport, the merging will
occur, but then the various members will be unable to communicate.
This will cause them to repeatedly break into partitions (when they
decide that the other members must have failed) and then re-merge
again.
The various primary and gossip transports are presented in the
following table. The ``P'' and ``G'' columns specify whether a
transport can be used for primary communication and/or gossip
communication.
transport |
P |
G |
description |
UDP |
Ö |
Ö |
UDP (+ gossip server) |
DEERING |
Ö |
Ö |
UDP/IP multicast |
TCP |
Ö |
|
TCP/IP |
NETSIM |
Ö |
Ö |
network simulator |
The NETSIM transports are used only in applications that are
simulating the behavior of a group inside a single process. The
rand and fifo demos use this, for instance. All other
currently supported modes run over IP. UDP requires running the gossip
server. With TCP as a transport, a fully connected mesh of TCP
connections is used to move messages between group members. This is
not very efficient for multicast messages, however, it can
sometimes be useful.
There are several ways to change the communication transports that Ensemble
uses. These are listed below in order of highest ``precedence.''
-
Command-line argument: with the -modes argument (see the
command-line argument documentation).
-
Application setting: a particular application may differ from the
Ensemble defaults.
-
Environment variable: ENS_MODES variable (see the environment
variable documentation below).
-
Ensemble defaults: UDP.
2
3.3 Using Deering IP Multicast
The method described above for running the mtalk demo is the best way
to first run mtalk because it uses UDP for both
transports. UDP does not use IP multicast communication,
which can be a source of problems because of variations in how it is
configured at different sites. [IP multicast is only available
when using the Socket library on Unix. It is not currently supported
by Ensemble on Windows NT.] If your machines support Deering IP
multicast communication, it is preferable to use DEERING
transports because you will then not have to run the gossip server
with Ensemble applications. You can try out the IP multicast
transport by using the command-line arguments. (Note the problems
section at the end of this section, however, which describes some of
the problems you may have.) This is done by executing the
applications with the -modes command-line arguments. With IP
multicast you no longer need to have a gossip server running. Run
the application on the hosts with these arguments (see below for a
description of the arguments):
% mtalk -modes DEERING
To always use IP multicast by default, modify the ENS_MODES
environment variable so that it includes DEERING. Also set the
ENS_DEERING_PORT environment variable to an unused port number.
You will probably wish to add these to your standard shell
environment:
setenv ENS_DEERING_PORT 1234
setenv ENS_MODES DEERING:UDP
3.4 Notes and Problems
See also the problems mentioned in the Ensemble reference manual.
-
IP Multicast problems :
- Some problems may occur with IP Multicast.
The time-to-live value for multicast messages may be too small in
some environments, preventing multicast messages from reaching all
members. The TTL value can be adjusted by editing the file
socket/multicasts.c.