Previous Contents Next

8   The Ensemble Security Architecture (by Ohad Rodeh)

This section describes the Ensemble security architecture. We believe that Ensemble completely supports the fortress security model. Only trusted, authorized members are allowed into the group. Once a member is allowed into a group, it is completely trusted. Ensemble is not secure against attacks from members that have been admitted into the group: any group member can break the protocols by sending bad messages.

The goal of our architecture is to secure group messages from tampering and eavesdropping. To this end, all group messages are signed and (possibly) encrypted. While it is possible to use public key cryptography for this task, we find this approach unacceptably expensive. Since all group members are mutually trusted, we share a symmetric encryption key, and a MAC 1 key among them. These keys are used to seal all group messages, making the seal/unseal operation very fast2. As a shorthand, we shall refer to the key-pair as the group key. Using a group key raises two challenges:

A rekeying mechanism:
allowing secure replacement of the current group key once it is deemed insecure, or if there is danger that it was leaked to the adversary. Dissemination of the new key should be performed without relying on the old (compromised) group key.

Secure key agreement in a group:
i.e., a protocol that creates secure agreement among group members on a mutual group key.
We focus on benign failures and assume that authenticated members will not be corrupted. Byzantine fault tolerant systems suffer from poor performance since they use costly protocols and make extensive use of public key cryptography. We believe that our failure model is sufficient for the needs of most practical applications.

The user may specify a security policy for an application. The policy specifies for each address3 whether it is trusted or not. Each application maintains its own policy, it is up to Ensemble to enforce it and to allow only mutually trusted members into the same subgroup. A policy allows an application to specify the members that it trusts and exclude untrusted members from its subgroup.

8.1   Cryptographic Infrastructure

Our design supports the use of a variety of authentication and encryption mechanisms. Ensemble has been interfaced with the OpenSSL (see http://www.openssl.org/) cryptographic library, the PGP authentication engine, and the Kerberos centralized authentication system (this is out of date). By default, messages are signed using MD5, encrypted using RC4, and authentication is performed using PGP. Because these three functionalities are carried out independently any combination of supported authentication, signature, and encryption systems can be used. A future goal is to allow multiple systems to be supported concurrently. Under such a system, processes would be able to compare the systems they have support for and select any system that both have support for.

8.2   Rekeying

Ensemble rekeying uses the notion of secure channels. A secure channel between endpoints p and q is essentially a symmetric encryption key kpq agreed upon between p and q. This key is known only to p and q and is different than the group key. Whenever confidential information needs to be passed between p and q it is encrypted using kpq and sent using Ensemble reliable point-to-point messaging.

The basic rekeying protocol supported uses a binary tree structure. In order to rekey the group, a complete binary tree spanning the group is created. Member 0 is the father of 1 and 2, 1 is the father of 3 and 4, etc.. The leader chooses a new key knew and sends it securely to 1 and 2; member number 1 sends knew securely down to 3 and 4, etc.. When a tree leaf receives a new key it sends up a clear-text acknowledgment. When acknowledgments reach the leader (0) it prompts the group for a view change in which the new key will be used.

knew is disseminated confidentially using secure channels. We cannot use the old key to protect knew since the old key is assumed to be compromised. Secure channels are created upon demand by Ensemble, they are then cached for future use. Creating a secure channel is a costly operation taking hundreds of milliseconds even on fast CPUs. It is performed in the background so as not to block the application.

Recently, we have added faster rekeying protocols to the system. A complete implementation of the dWGL algorithm has been added, in the form of several layers. There are two new algorithms rekey_dt, and rekey_diam. There are described in the reference manual.

8.3   A secure stack

The Security architecture is comprised of 5 layers:
Exchange:
secure key agreement. This layer is responsible for securely handing the group key to new joining group components. Component leaders mutually authenticate and check authorization policies prior to handing the group key securely between them.
Encrypt:
chain-encryption of all user messages.
Secchan:
create and manage a cache of secure channels.
PerfRekey:
handling common rekeying tasks. For example, after a new key has been disseminated to the group, acknowledgments must be collected from all group members.
Rekey_dt:
Binary tree rekeying. Rekeying a group is very fast once secure channels have been setup. We logged an average rekey operation for a 20 member group at 100 milliseconds. Rekey_dt assumes that the Secchan and PerfRekey layers are in the stack.
The regular and secure Ensemble stacks are depicted in Figure 2. The Top and Bottom layer cap the stack from both sides. The membership layers compute the current set of live and connected machines, the Appl_top layer interfaces with the application and provides reliable send and receive capabilities for point-to-point and multicast messages. The RFifo layers provide reliable per-source fifo messaging. The Exchange and Rekey layers are related to the membership layers since the group key is a part of the view information. The Encrypt layer encrypts all user messages hence it is below the Appl_top layer.


Regular security additions
Top
Exchange
Rekey_dt
PerfRekey
  Secchan
Gmp  
Top_appl Interface to the application
  Encrypt
Rfifo
Bottom

Table 2: The Ensemble stack. On the left is the default stack that includes an application interface, the membership algorithm and a reliable-fifo module. To the right is a secure stack with the Exchange, Encrypt, Rekey_dt, and Secchan layers in place.


8.4   Security events

There are three security events to note: The Vs_key field was added to the view state was to allow for group keys. It holds the current group key.

8.5   Using Security

Ensemble has three security properties:
  1. Rekey: Add rekeying to the stack.
  2. OptRekey: Use the dWGL algorithm for rekeying.
  3. Auth: Authenticate all messages.
  4. Privacy: Encrypt all user messages.
An application wishing for strong security should choose all of the above properties in its stack and perform a Control Rekey action once every several hours. Note that there are two flavors to application Rekey-ing: An example command line, for application appl, with pgp user name James_Joyce:

appl -add_prop Auth -add_prop Privacy -key 01234567012345670123456701234567 
     -pgp James_Joyce

In order to add authorization to the stack, thereby controlling which members are allowed to join a group, one must do:


  val policy_function : Addr.set -> bool
  val interface : Appl_intf.New.t

  let state = Layer.new_state interface in
  let state = Layer.set_exchange (Some policy_function) state in 
  Appl.config_new_full state (ls,vs)

Instead of simply:

  Appl.config_new interface state (ls,vs)

Authorization is not linked to the Security architecture, regular stacks can perform authorization. Control of joining members is delegated to the group leader that checks its authorization list and allows/disallows join. Every view change the authorization list is checked and existing members that are not authorized are removed.

In practice, if an application changes its authorization list dynamically, it must perform a Prompt and a Rekey whenever such a change occurs.

8.6   Checking that things work

To check that PGP has been installed correctly, that Ensemble can talk to it without fault, and the cryptographic support is running correctly, one can use the armadillo demo program.

In order to set up PGP, one must create principals and corresponding public and private keys. These are installed by PGP in its local key repository. The basic PGP key-generation command is:

zigzag ~/ensemble/demo> pgp -kg

To work with the armadillo demo, you'll need to create principals in the group o1, o2, .... Armadillo creates a set of endpoints, and then runs a test between them. To this end, the program has a ``-n'' flag that describes the number of endpoints to use. For example, the command line armadillo -n 2 ... tells armadillo that use a two members configuration. These members will have principal names o1 and o2 respectively.

To view the set of principals in the repository do:

zigzag ~/ensemble/demo> pgp -kv
pub   512/2F045569 1998/06/15 o2
pub   512/A2358EED 1998/06/15 o1
2 matching keys found.

To check that PGP runs correctly do:

zigzag ~/ensemble/demo> armadillo -prog pgp 
PGP works
check_background
got a ticket
background PGP works

If something is broken, the PGP execution trace can be viewed using:

zigzag ~/ensemble/demo> armadillo -prog pgp  -trace PGP 

If more information is required use the flags -trace PGP1 -trace PGP2. The default version of PGP that Ensemble works with is 2.6. If, however, you'd like to use a different version, set your environment variable ENS_PGP_VERSION to the version number. Versions 5.0 and 6.5 are also supported.

To check that OpenSSL is installed correctly, one can do:

zigzag ~/ensemble/demo> armadillo -prog perf

For a wider scale test use the exchange test. This is a test that creates a set of endpoints, with principal names: o1, o2, ..., and merges them securely together into one group. Each group merge requires that group-leaders properly authenticate themselves using PGP. The test is started with all members in components containing themselves, and ends when a single secure component is created. Note that it will keep running until reaching the timeout. The timeout is set by default to 20 seconds. To invoke the test do:

zigzag ~/ensemble/demo> armadillo -prog exchange -n 2 -real_pgp

If something goes wrong, a trace of the authentication protocol is available through -trace EXCHANGE.

The -real_pgp flag tells armadillo not to simulate PGP. Simulation is the default mode for armadillo, since we use it to test communication protocol correctness.

To check that rekeying works do:

zigzag ~/ensemble/demo> armadillo -prog rekey  -n 5

To test security with two separate processes do the following:

zigzag ~/ensemble/demo> gossip &
zigzag ~/ensemble/demo> mtalk -key 11112222333344441111222233334444 
                  -add_prop Auth -pgp o1
zigzag ~/ensemble/demo> mtalk -key 01234567012345670123456701234567 
                 -add_prop Auth -pgp o2

The two mtalk processes should authenticate each other and merge.

The three command line arguments specify:

8.7   Using security from HOT and EJava

The security options have been added to the HOT interface. For a demonstration program look at hot_sec_test.c in the hot subdirectory. The only steps one needs to make are: (1) Set the program's principal name (2) Set the security bit. Both of these options are specified in the join-options structure. For example, in hot_sec_test.c:

static void join(
		 int i,
		 char **argv
) 
  state *s ;
  s = (state *) hot_mem_Alloc(memory, sizeof(*s)) ;
  memset(s,0,sizeof(*s)) ;
  
  s->status = BOGUS;
  s->magic = HOT_TEST_MAGIC;

  ...

  strcpy(s->jops.transports, "UDP");
  strcpy(s->jops.group_name, "HOT_test");
  
  ...

  sprintf(s->jops.princ, "Pgp(o%d)",i);
  s->jops.secure = 1;

  ...
  
  /* Join the group.
   */
  err = hot_ens_Join(&s->jops, &s->gctx);
  if (err != HOT_OK) 
    hot_sys_Panic(hot_err_ErrString(err));
  



EJava is interfaced with HOT, so they share a similar interface. Note that the outboard mode, supported by both interface is insecure. The messages passing on the TCP connection between the client and server are neither MACed nor encrypted. Therefore, they can be used securely only when situated on a single machine.


Previous Contents Next