Ensemble 1.33

Release Notes and Installation Instructions

Copyright ©1997 Cornell University, Ithaca, NY, USA, Hebrew University 2000, Jerusalem Israel.
All rights reserved.

Maintainers: Ohad Rodeh,

Introduction

Thank you for choosing Ensemble as your message transport system. Its platform independence allows you to interconnect a wide range of computing environments.

The following instructions guide you through the process of installing Ensemble. Click on the hypertext links for the details about each step. The installation instructions are designed to be as simple as possible. However, please read them through carefully before starting and follow them carefully. Also, please read the file RELEASE_NOTES for the latest information on this release.

These installation instructions are aimed at getting Ensemble up and running on your system as quickly as possible. After successfully installing the system, you may wish to compile Ensemble for the native instruction set of your computer. The difference is only significant when Ensemble is used in high-performance settings.


Required Software

To compile Ensemble you need Version 3.01 of the Objective Caml (O'Caml) compiler. For unix systems, you also need the make version by GNU, also known as GNU-make. For windows, you'll need the Visual Studio compiler verion 6 and the nmake tool.

O'Caml is freely available via the World Wide Web at http://caml.inria.fr/ocaml/. Please ensure you have the correct version. See the instructions below on how to obtain it. This is a different compiler version than required by previous versions of Ensemble! Version 0.40 of Ensemble used ocaml-1.05; Version 0.50 of Ensemble used ocaml-1.07; version 0.60 used ocaml-2.01; and we are now up to ocaml-3.01.

On Unix platforms, to compile the HOT C interface, you need a POSIX compliant threads package. (On Solaris, you can use the native threads library). To compile the Maestro tools C++ interface, you will need a C++ compiler. We use the Microsoft C++ compiler on WIN32 and gcc on Unix platforms, but other compilers should work. On windows to build the optimized versions of HOT C and Maestro, you will also need the masm (MS Compatibility Driver) and ml (MS Macro Assembler) executables installed and located in your path.


Licensing

Ensemble code has a BSD-type license. It is copyrighted by Cornell and the Hebrew universities. Ensemble is freely available software, but you must agree to the terms of the license before you can install it. We ask that you read the licensing agreement before installing Ensemble. It can be found in ensemble/LICENSE.


Installation

Install Ensemble by following the step-by-step instructions. You need to get the O'Caml compiler, extract the Ensemble software, and then compile the portions you wish to use.


How to obtain Objective Caml 3.01

To compile Ensemble you need Objective Caml version 3.01. Other versions of the compiler may NOT work. This is because the O'Caml bytecode format and libraries are not guaranteed to be compatible across versions.

If you already have OCaml you can check which version you have by typing:

    ocamlc -v
This gives you the version of the compiler in your path (if there is one). The first line of the output should be:
    The Objective Caml compiler, version 3.01
If you have some other version, then obtain version 3.01 and install it. O'Caml is freely available from http://caml.inria.fr/ocaml and is easy to install (precompiled binaries are available for Windows NT). You do not need the native code compiler, although you can compile Ensemble with it.

The O'Caml Tcl/Tk Interface

Previously, the Tcl/Tk caml library was distributed separately from the compiler. However, it now comes with the main distribution, under the name LablTk. You'll need this library for building the whiteboard and game-of-life demos. Both can be found in the demo subdirectory.

Back to installation guide.


How to download and extract the Ensemble software

Download Ensemble from http://www.cs.cornell.edu/Info/Projects/Ensemble/ftp.html. Ensemble is no longer encrypted. Simply unzip and untar the distribution. It will put the sources in a directory called "ensemble". On Windows NT you can use the winzip program. On Unix platforms, do it this way:

    # uncompress and untar the distribution
    gunzip ensemble-1_0.tar.gz
    tar xvf ensemble.tar
Back to installation guide.


Building Ensemble

Ensemble is supported on Unix (and Unix like systems), and WIN32 platforms. It is tested mainly on Linux and WIN2000, but also on sparc-solaris and NT4.

First, make sure the CAMLLIB environment variable is set, and the Ocaml binaries are accessible from your PATH. Then compile the Ensemble system and demonstration programs as follows:

    # make the Ensemble system dependencies
    # directory: ensemble
    make depend
   
    # make the system
    make all
If all went smoothly, the initial installation is now complete, and you can try out the demo programs following the instructions below. Note that currently, the dependencies for windows come with the distribution, so you do need to make the dependencies. Due to a bug in ocaml-3.01, the dependecies will be erased. This bug will be fixed in version 3.03, in the mean time, simply refrain from the first step. You can also compile for your native instruction set. instruction set. In order to clean up the object files do:
    # clean up
    # directory: ensemble
    make clean
This will leave only the binaries (in bin) and libraries (in lib).

How to run applications

You can now refer to the tutorial, ensemble/doc/tut.ps, for instructions on executing the demo applications. This involves setting a few environment variables and then executing the applications. Come back to here to install additional parts of Ensemble.

Graphical Demos

Compile the white board demo and "Game of Life" demo. Note: your platform must support the LablTk library. These applications will not compile if the Ocaml TK interface has not been installed.

    # make the Ensemble tk library
    # directory: ensemble
    make tk
   
    # make the white board demo
    # directory: ensemble/demo/tk
    make
   
    # make the game-of-life demo
    # directory: ensemble/demo/life
    make
To make the optimized version of the tk library do 'make tk_opt'.

Environment variables

In order to compile platform-dependent parts (the Socket library, the native code, the HOT C interface, and the Maestro C++ interface) some configuration needs to be done.

Another important variable is ENS_ABSROOT. This denotes the absolute path to the root of the distribution. There are several other environment variables that are useful when using the system. A glossary appears here.

Building the Socket library

The Socket library supports extensions to the Unix library concerned with communication. Strictly speaking, the optimized socket library is not necessary, because the basic communication functionality comes with CAML. However, it contains many features not found in the basic Caml library: IP-multicast, handling C iovecs, and more. Using the Socket library increases Ensemble performance and decreases memory usage. The default is to use it, however, if you wish to use something safer, you can use the "unix" library, which is the simple non-optimized interface to the Caml Unix interface.

    # In ensemble/mk/config.mk, edit the macro definition for HSYS_TYPE to be "unix".
    # directory: ensemble
    make depend
    make 

The native C interface (CE)

The native C interface is automatically built as part of the build process. There is no need for special actions on the user part. The source files are in directory ce, atutorial appears as a chapter in the doc/tut.ps. Documentation for the header files appears under 'doc/ce' as well. The build process generates the file ensemble/lib/PLATFORM/libce.a (where PLATFORM is the name of your platform) and installs the CE library files into 'ensemble/lib/PLATFORM'. The demos are installed in 'ensemble/bin/PLATFORM'

Building the HOT tools

The HOT library is also automatically built as part of the build process. The build process generates the file ensemble/lib/PLATFORM/libhot.a (where PLATFORM is the name of your platform) and installs the HOT library files into 'ensemble/lib/PLATFORM'. The demos are installed in 'ensemble/bin/PLATFORM'.

There are auxilliary HOT test programs, which can be compiled as follows (for case of the hot_test2 demo):

    # build the hot_test2 demo program
    # directory: ensemble/hot
    make hot_test2 
This will build and install the program hot_test2 in the hot directory. The program hot_test2 is a two-process RPC performance test.
The HOT tools can also be built with cryptographic support as follows. First you will need to edit the file, ensemble/mk/config.mk, and uncomment the CRYPTO_LINK lines. You also need to edit the ensemble/hot/Makefile and modify the CLIBS and CRYPTO_CLIBS variables as shown in the makefile. Then goto the top level directory and recompile.

Building the Maestro tools

You need to have compiled the HOT library before compiling the Maestro C++ interface. You need a C++ compiler in order to compile Maestro. This step generates the file ensemble/maestro/conf/PLATFORM/libmae.a (where platform is the name of your platform).

    # build the Maestro and IIOP libraries and utilities
    # directory: ensemble/def
    make maestro
   
    # build the Maestro and IIOP test applications
    # directory: ensemble/def
    make maestro_test
(**NOTE** You can also run "make" in the maestro directory)

You should now have five test programs,

maestro-test is a single process sanity test in which several group-member objects are created which then randomly send messages and leave/rejoin the group. maestro-perf is a two-process RPC performance test. Both test-replicated and test-simple are Maestro IIOP test applications. group is an application which tests the new, simpler state transfer algorithm. See ensemble/maestro/REF.ps for more information on the Maestro IIOP tools. To get started with Maestro and Ensemble, we also recommend you read the Maestro documentation and start from the test programs (in the corresponding .C files) -- you can modify and experiment with them.

Building the Maestro tools for Windows NT

**NOTE**The Maestro IIOP software, built when compiling on Unix, has not yet been ported to NT.
You need to have compiled the HOT library before compiling the Maestro C++ interface. This step generates the file ensemble\maestro\maestro-nt\libmae-nt.lib as well as the maestro test programs:
    # build the Maestro tools
    # directory: ensemble\opt
    nmake -f Makefile.nt maestro-nt
You should now have three test programs: maestro_test is a single process sanity test in which several group-member objects are created which then randomly send messages and leave/rejoin the group. maestro_perf is a two-process RPC performance test. group is an application which tests the new, simpler state transfer algorithm. To get started with Maestro and Ensemble, we recommend you read the Maestro documentation and start from the test programs (in the corresponding .C files) -- you can modify and experiment with them.

Building the native code version

Compilation for your native instruction set results in faster code, and reduction in memory use. The build process in this case also generates optimized versions of the CE, HOT, and socket libraries. This step compiles all of the Ensemble sources (including the demos) and will take awhile.

    # make the Ensemble system dependencies
    # directory: ensemble
    make depend
   
    # build native code libraries
    # directory: ensemble
    make opt
Note: On windows do not use 'make depend', because it will erase your dependencies. This is a bug in the Caml dependency generator, it will be fixed in version 3.03. In the meanwhile, use the dependencies that come pre-computed with the system.

Building the EJava Interface

The EJava interface to Ensemble was built and tested using Sun Microsystem's JDK1.2. Things should also work with the newer 1.3 and 1.4 JDKs. Before trying to build this interface, please check and make sure you have access to JDK1.2. Then follow these steps:

    # build the EJava interface
    # directory: ensemble/ejava
    make
You can then run one of the demo programs, EnsDemoOutboard or ThreadTest as follows:
    java EnsDemoOutboard
EnsDemoOutboard is a graphical version of the mtalk demo. It allows processes to "chat" to each other. The demo application, ThreadTest, is a test which stresses multi-threading. ThreadTest, requires one parameter, the number of group members to wait for before beginning the test. For example, to run the test with 2 members, start the test on 2 machines (or 2 consoles) as follows:
    java ThreadTest 2

Installation is complete

This concludes the instructions for building Ensemble on Unix systems.
Back to installation guide.


Building Ensemble on Multiple Platforms

Multiple versions of Ensemble can co-exist within the same directory tree. All platform-dependent files now have a separate directory describing the platform for which they were compiled. For Unix platforms, you just need to ensure that the ENS_MACHTYPE and ENS_OSTYPE environment variables are correctly set. For Windows the name of the platfrom ('nt') is embedded in the Windows NT makefiles and does not need to be set. For instance, the platform-independent bytecode library for Ensemble is called 'libens.cma' ('cma' is the O'Caml suffix used for bytecode libraryies). The platform-dependent, native code library for Ensemble on i386, Linux platforms is called 'lib/i386-linux/libens.cmxa' ('cmxa' is the O'Caml suffix for native code libraries).

All the demos are install in bin/PLATFORM, this includes the ML, CE, and HOT demos. The Ejava and Maestro demos are built within there own directories.

This concludes the instructions for building Ensemble on multiple platforms.
Back to installation guide.

"Make Clean"

Calling 'make clean' from the top-level directory will erase all object files in the system, and leave just the libraries and executables in the lib and bin directories. Calling 'make realclean' will erase these as well. When compiling for multiple platforms make sure to call 'make clean' between builds.


POSIX threads

If your system does not have a native POSIX threads library, install the FSU pthreads.tar.gz package. You can get it from:

After installing the library. Update in ensemble/mk/config the macros GTHREADS, GTHREADS_INC to point to your installation. If necessary, add new HOT_CFLAGS_arch_os, HOT_MLLINK_arch_os, HOT_LINK_arch_os, and HOT_THREAD_OBJ_arch_os macros to ensemble/mk/config.mk for your platform (and send them to us so we can add them to the next distribution).


Back to installation guide.


SECURITY

Ensemble supports several security properties, including user message encryption, authentication, and group member authorization. Ensemble requires the user to install PGP6.5 and OpenSSL0.9.5a. Previous version of PGP (2.6.3, 5.0) also work, however, we have had correctness probelms with them. In order to make ensemble with cryptographic support, FIRST, you must edit mk/config.mk. You will see the CRYPTO_LINK define commented out by default. Comment out the top (empty) CRYPTO_LINK line and uncomment the rest of the CRYPTO related lines. On NT, simply uncomment the lines following the CRYPTO_LINK line. This will now link in the crypto libraries when you build executables. Now perform the following steps: Unix:

    # Make the regular crypto library
    # directory: ensemble
    make crypto
   
    # Make the optimized crypto library
    # directory: ensemble
    make crypto_opt
   
    # Make the system
    # directory: ensemble
    make 
   
    # Make the optimized version
    # directory: ensemble
    make opt
NOTE: on NT, change "make" to: "nmake -f Makefile.nt" and things should compile smoothly.

Windows not fully supported

Currently, cryptographic support is not perfect for NT, though we plan to fully support it in the future. PGP works on both windows and unix, however, Openssl does not work with Microsoft Visual Studio. Openssl needs the cygwin libraries, which we do not currently support.

Testing CRYPTO

In order to check that your system can use the various cryptographic libraries and interfaces, use the the armadillo program in the demo directory. In order to set up PGP you'll need to use (at least) the following command:
    pgp -kg
This will generate a public/private key pair for a principal of your choice. The key-pairs are stored in your home directory under the subdirectory .pgp. Armadillo requires that the principals o1,o2 are installed. To see the set of principals you currently use:
   pgp -kv
   pub   512/2F045569 1998/06/15 o2
   pub   512/A2358EED 1998/06/15 o1
   2 matching keys found.
    armadillo -prog pgp To test the PGP interface. You should see something like:
   PGP works
   check_background
   got a ticket
   background PGP works
On NT, PGP only works in the foreground, in the background due to socket select issues. If something is broken, trace the PGP execution using:
    armadillo -prog pgp -trace PGP 
If more information is required use the flags:
   -trace PGP1 -trace PGP2
For a wider scale test use the exchange program. This is a test that creates a set of endpoints, with principal names: o1, o2, .., and merges them securely together into one group. Each group merge requires that group-leaders properly authenticate themselves using PGP. The test is started with all members in components containing themselves, and ends when a single secure component is created. Note that it will keep running until reaching the timeout. The timeout is set by default to 20 seconds. To invoke the test do:
    armadillo -prog exchange -n 2 
To use more endpoints, increase the -n command line argument, and add more principals to your PGP database. If something goes wrong, a trace of the authentication protocol is available through
 -trace EXCHANGE
. To test security with two separate processes do the following:
   #setup and run gossip
   gossip &
   #On one machine run:
    mtalk -key 11112222333344441111222233334444 -add_prop Auth -pgp o1
   #On a second machine run:
    mtalk -key 01234567012345670123456701234567 -add_prop Auth -pgp o2
The two mtalk processes should authenticate each other and merge. The three command line arguments specify:

Performance issues with PGP

The way Ensemble uses PGP is not very efficient at the moment. However, this only effects authentication, which occurs rarely. Common operations such as Diffie-Hellman exchanges, symmetric encrtyption/decryption, and MD5 hashing are performed using the OpenSSL toolkit.

Glossary of environment variables

Here is a list of environment variables that one needs to set for Ensemble to work. The complete description can be found in the tutorial.
# The set of communication transports.
setenv ENS_MODES "UDP" #"TCP:DEERING"

# The user-id
setenv ENS_ID    $USER

# The port number used by the system
setenv ENS_PORT  6789

# The port number of the gossip service
setenv ENS_GOSSIP_PORT 6788

# The set of gossip hosts.
setenv ENS_GOSSIP_HOSTS "127.0.0.1" #"hfstore1:hfstore2:hfstore4:hfstore5"

# The set of groupd hosts
setenv ENS_GROUPD_HOSTS "hfstore1"

#the port number of the group-daemon service
setenv ENS_GROUPD_PORT 6790

#The port used for IP-multicast
setenv ENS_DEERING_PORT 6793

#The pgp password
setenv ENS_PGP_PASS "Wild Boar"

#The pgp version used (2.6,5.0,6.5)
setenv ENS_PGP_VERSION "2.6"

#where the outboard executable resides
setenv ENS_OUTBOARD "/home/orodeh/e/1.20a/ensemble/demo/outboard"

#The source of the distribution
setenv ENS_ABSROOT "/home/orodeh/e/1.20a"


Back to installation guide.
Last modified: Thu Dec 06 10:11:53 E. Europe Standard Time 2001