Previous Contents Next

3   Configuration

3.1   Command-line Arguments and Environment Variables

Ensemble applications typically support a variety of configuration parameters. Most of these can be configured through command-line options as well as through setting environment variables. In all cases, command-line options override environment variables. Look in appl/harg.ml for the authoritative list of the configuration parameters. Some are listed below as command-line options. The corresponding environment variable for -group_name (for example) is ENS_GROUP_NAME (the name is capitalized and the `-' is replace with ENS_).
-modes arg :
Set the default modes for an application to use. The modes are specified giving their names in all-uppercase, each separated by single colons (`:') and no white-space.
-udp_port port :
set the default UDP port for Ensemble applications. For point to point UDP communication, this is the port number Ensemble first tries to bind to for UDP communication (if it is already in use Ensemble will then fail). It can be set to any value. The default is to let the operating system choose a port to use.
-deering_port port :
This is the port that Ensemble will use for Deering IP multicast communication (if enabled). All processes must use the same port number.
-gossip_port port :
sets the port that the gossip servers use.
-gossip_hosts arg :
sets the hosts where applications using UDP communication can look for gossip servers. The value should be a colon-separated list of hostnames. The gossip server application will only execute on these hosts. Note that you only have to execute a gossip server on one of these hosts: applications will try each of the hosts in turn while looking for a gossip server. However, multiple servers can be executed for increased availability.
-id name :
used to give applications unique identifiers. Usually this is set to be your user id. Setting this variable prevents Ensemble applications run by other users from interacting with yours. In case you do want them to interact, you should set their variables to have the same value. If using DEERING IP multicast, their -deering_port variable should also be set to the same value.
-groupd_port port :
sets the port that the membership groupd servers use.
-groupd_hosts arg :
sets the hosts that the membership groupd servers use. Format is the same as for -gossip_hosts.
-groupd :
Use the membership service on the local host (see section 2.5. This option may override others.
-group_name name :
Set the name of the application's group. [Currently, only the ensemble application supports this.]
-key key :
Set the key to use for a particular application. All messages sent and received by the application will be authenticated with this key.
-secure :
Enable security enforcement. This prevents any insecure communication transports from being initialized.
-add_prop property :
Adds a specific property to the Ensemble protocol stack. See Section 5.8 for more information on supported Ensemble properties.
-remove_prop property :
The dual of add_prop.
-sock_buf size :
The size of socket buffers to request from the operating system. The default size is 52428 (the traditional limit on Unix). If you are using Ensemble in high-performance setting and are experiencing message loss, this is a parameter that should be increased.
-refcount :
Enables reference counting of message buffers. The default is to rely on the garbage collector to detect when a message is no longer needed. Setting this will improve performance, but may expose reference counting bugs in Ensemble.
-multiread :
Enable multiple reads on sockets. The default is to receive and process one message from the operating system at a time. Setting this will cause all available messages to be read from sockets before processing any of them, which may reduce message loss due to buffer overflow in the operating system.
-pollcount count :
The number of times to query the operating system before blocking. Ensemble blocks after checking (via the select() system call) the operating system for messages and not finding any. Setting this to 1 will cause Ensemble to block immediately when there are no more messages. Setting this to a large number will cause Ensemble to busy-poll for a longer time before blocking.
-ranking trans:
this is used to set the priority of transport mechanisms. This is useful if you wish to use TCP as a transport instead of UDP (the default). Using -ranking TCP will cause TCP to be ranked higher than any other transport.
The following configuration parameter can only be set as an environment variable.
ENS_TRACE
: enables module initialization tracing. With this set (to any value), modules print out their names as they initialize. This is useful if an exception occurs during initialization because because it enables you to narrow the problem down to one module.

3.2   Transports

[If you are only using regular UDP sockets for communication, then you do not need to read this section.]

Perhaps the most confusing part of running Ensemble applications comes from selecting communication transports. Communication transports are the bottom-most part of Ensemble and are used for sending and receiving messages on a network. There are several ways this can be confusing and often Ensemble cannot detect that there is a problem, so you do not get a warning. For instance, if you configure an application so that one process is using UDP sockets for communication and another is using NETSIM, then the two processes will stall waiting for other processes to communicate with them on their selected medium.

A confusing aspect of transport is that an application typically uses two different kinds of transports: a primary transport and a gossip transport. Normal application communication is all done over the primary transport, which must support point-to-point communication and may also support multicast communication. Communication between different partitions of a group of applications uses the gossip transport which must support ``anonymous'' multicast communication.

Applications occasionally send ``gossip'' messages with their gossip transport to the rest of the ``world'' in order to inform other partitions about their presence. When two partitions learn of each other, they can then merge the partitions together. After they have merged together, they communicate over their primary transport. This gossip-and-merge mechanism is used when applications first start up: an application creates its own singleton group and then merges with any other already existing partitions through gossiping and merging. Thus, if there is a problem with the gossip transport, you will tend to have a bunch of applications in singleton groups that never merge. If there is a problem with the primary transport, the merging will occur, but then the various members will be unable to communicate. This will cause them to repeatedly break into partitions (when they decide that the other members must have failed) and then re-merge again.

The various primary and gossip transports are presented in the following table. The ``P'' and ``G'' columns specify whether a transport can be used for primary communication and/or gossip communication.
transport P G description
UDP Ö Ö UDP (+ gossip server)
DEERING Ö Ö UDP/IP multicast
TCP Ö TCP/IP
NETSIM Ö Ö network simulator

The NETSIM transports are used only in applications that are simulating the behavior of a group inside a single process. The rand and fifo demos use this, for instance. All other currently supported modes run over IP. UDP requires running the gossip server. With TCP as a transport, a fully connected mesh of TCP connections is used to move messages between group members. This is not very efficient for multicast messages, however, it can sometimes be useful.

There are several ways to change the communication transports that Ensemble uses. These are listed below in order of highest ``precedence.''
  1. Command-line argument: with the -modes argument (see the command-line argument documentation).
  2. Application setting: a particular application may differ from the Ensemble defaults.
  3. Environment variable: ENS_MODES variable (see the environment variable documentation below).
  4. Ensemble defaults: UDP.
2

3.3   Using Deering IP Multicast

The method described above for running the mtalk demo is the best way to first run mtalk because it uses UDP for both transports. UDP does not use IP multicast communication, which can be a source of problems because of variations in how it is configured at different sites. [IP multicast is only available when using the Socket library on Unix. It is not currently supported by Ensemble on Windows NT.] If your machines support Deering IP multicast communication, it is preferable to use DEERING transports because you will then not have to run the gossip server with Ensemble applications. You can try out the IP multicast transport by using the command-line arguments. (Note the problems section at the end of this section, however, which describes some of the problems you may have.) This is done by executing the applications with the -modes command-line arguments. With IP multicast you no longer need to have a gossip server running. Run the application on the hosts with these arguments (see below for a description of the arguments):
% mtalk -modes DEERING
To always use IP multicast by default, modify the ENS_MODES environment variable so that it includes DEERING. Also set the ENS_DEERING_PORT environment variable to an unused port number. You will probably wish to add these to your standard shell environment:
setenv ENS_DEERING_PORT 1234
setenv ENS_MODES DEERING:UDP

3.4   Notes and Problems

See also the problems mentioned in the Ensemble reference manual.
IP Multicast problems :
Some problems may occur with IP Multicast. The time-to-live value for multicast messages may be too small in some environments, preventing multicast messages from reaching all members. The TTL value can be adjusted by editing the file socket/multicasts.c.

Previous Contents Next