U-Net/NT/MM Device Reference

Last modified 11-Feb-97 by M. Welsh and X. Huang

This is the device interface guide for U-Net version 0.9 for NT, revised from U-Net version 2.0 for Unix.

Introduction

This document describes the application-visible view of the U-Net device, and describes how to open the device, enable endpoints and channels, send data, receive data, manage buffers, and closing the device.

Many of the methods describe here have been encapsulated in the libunet library, which demonstrates use of the U-Net device. While you may find libunet useful, it is by no means the "standard" interface, libunet is not the most efficient way the U-Net Application can be written. Libunet's behavior is subject to change with each release. It's our hope that the U-Net device interface, described below, will be somewhat stable between releases.

You should look at the libunet code to see how the methods described here can be easily implemented, and use the library as the starting point for your own U-Net communication layer. libunet is by no means suitable for all applications --- it just presents a simplified API which makes certain assumptions, useful for some programs.

This guide assumes that you are familiar with the concepts used by U-Net such as endpoints and channels. See the U-Net documentation tree for information.

The primary definition of the U-Net device interface is the header file unet.h, found in the udi\ subdirectory of the U-Net distribution. This header file defines all aspects of the U-Net device which are not dependent on the particular implementation used.

Memory layout

There is a major change in memory management in this release. In the previous release, a U-Net endpoint layout is as follows:

The new layout is as following, the transmit and receive buffer is outside the user endpoint:

In the previous release, the transmit/receive buffers are allocated by the device driver from the kernel during endpoint creation, they are non-swappable memory. In this new release, with the memory management module incorporated into U-Net, a U-Net application can use any buffer for sending, and can allocate its own receive buffer in user space to achieve zero copy transmit/receive. To read more about memory management, please refer to our paper Incorporating Memory Management into User-Level Network Interfaces .

Tx FIFO: The Tx FIFO is a ring of Unet_txbd structures. Each structure points to a buffer in the users space and includes a channel number and a length. The user process pushes new entries onto the Tx FIFO as a ring (that is, wrapping around to the top of the FIFO after reaching the bottom). These entries are consumed by the U-Net device in FIFO order.
Tx wall pointer: After the Tx FIFO is an unsigned int which contains the Tx wall pointer. This is a byte offset into the Tx FIFO pointing to the most recently-transmitted Tx FIFO entry. It is initialized by the U-Net device to point to the last entry and is incremented (wrapping around to the top of the Tx FIFO) when an entry of the Tx FIFO is consumed. It is updated by the U-Net device and should only be read by the user process.
The Tx wall pointer indicates to the user when the Tx FIFO is full; the Tx FIFO entry pointed to by the wall pointer should always have a channel field of CHAN_FREE. That is, the user process should not attempt to push a Tx descriptor into the entry pointed to by the wall pointer.
Free FIFO: The Free FIFO is a ring of Unet_rxbd structures. Each entry points to an buffer in the Rx buffer region which is available for newly-received packets to be placed into by the U-Net device. The user process should push new entries as a ring. Entries are consumed by the U-Net device in FIFO order.
Free wall pointer: After the Free FIFO is an unsigned int which contains the Free wall pointer. This is a pointer pointing to the most recently-consumed Free FIFO entry. It is initialized by the U-Net device to point to the last entry and is incremented (wrapping around to the top of the FIFO) when an entry of the Free FIFO is consumed. It is updated by the U-Net device and should only be ready by the user process.
The Free wall pointer is used by the user process to determine if the Free FIFO is full.
Rx FIFO: The Rx FIFO is a ring of Unet_rxd structures. When a message arrives for a U-Net endpoint, the U-Net device pushes a descriptor onto the Rx FIFO, containing the destination channel number, length in bytes, and either
(a) Up to info.rxd_payload_size bytes of message data (if the message is small enough to fit into the receive descriptor itself), or
(b) Up to RXD_PAYLOAD_ENTRIES of pointers to buffers in the Tx/Rx buffer region. Each buffer contains a portion of the received message.
Rx wall pointer: After the Rx FIFO is an unsigned int which contains the Rx wall pointer. This is a byte offset into the Rx FIFO pointing to the Rx FIFO entry most recently consumed by the user process. It should be initialized by the user process to point to the last Rx FIFO entry. The Rx wall pointer should be updated by the user process upon consumption of Rx FIFO entries; whenever an entry is popped off of the Rx FIFO by the user process, the Rx wall pointer should be incremented (wrapping around to the top of the FIFO when reaching the end). The U-Net device uses the Rx wall pointer to determine if the Rx FIFO is full.

Device Initialization

Opening the U-Net device

This can be done through DeviceIoControl calls, but we have provided a function named unet_opendev() in libunet to do this. In this release, we support two types of device UNET_DEVID_TULIP and UNET_DEVID_ZEITNET. Your application should select the appropriate device entry and a valid device index for the U-Net device and open it with unet_opendev().

Obtaining information about the device

Issuing the UNET_INFO DeviceIoControl() will return the information about the U-Net device, such as the following:

info.devid can be used to determine which implementation of the U-Net device is being used (constants are defined in unet.h).
info.version should be equal to UNET_DEV_VERSION as defined in unet.h; if not, your kernel driver is incompatible with the unet.h which you are using.
info.minor is used to provide the appropriate minor device ID (corresponding to the endpoint and device number) if a kernel trap is required by the U-Net implementation (more on this below).
The maximum number of endpoints and channels supported by the device, transmit/receive buffer alignment, and min/max sizes for each of the U-Net FIFOs (see below). These fields should be used to determine meaningful values for endpoint and channel creation.

Creating an endpoint

To create an endpoint, fill in a UNET_CREATE_ENDPOINT_REQUEST structure as described in unet.h, and call the UNET_CREATE_ENDPT DeviceIoControl(). In this way you indicate the desired sizes for the U-Net FIFOs and buffer regions. If EINVAL is returned, at least one of the requested size values is out of range for the device.

The kernel will fill in the remaining fields in the Unet_create_endpt structure, indicating byte offsets from the beginning of the endpoint mapping to each of the areas in the endpoint: The three FIFOs and wall pointers, and the buffer area offset. These offsets are determined by the kernel driver and correspond with any alignment restrictions imposed by the device.

After creating the endpoint, the user process should first allocate a region for RX buffers, then initialize the chan field of each Rx FIFO entry to CHAN_FREE, and initialize the Rx wall pointer to the byte offset of the last Rx FIFO entry.

Current we have provided a function in libunet called unet_create_endpt() that does the above.

Activating a channel

To create a point-to-point communications channel for use with an endpoint, the user process should fill in a UNET_DEACTIVATE_CHANNEL_REQUEST structure and call the UNET_ACTIVATE_CHAN DeviceIoControl() on the U-Net device. The UNET_DEACTIVATE_CHANNEL_REQUEST has fields for the transmit and receive address, each of which is a union unet_chan_addr. This union is used as different U-Net implementations have different address formats; the devid field of Unet_info structure can be used to determine which to use. For example, struct unet_fe_addr is used for Fast Ethernet MAC addresses, while struct unet_atm_addr is used for ATM VPI/VCI pairs.

The UNET_ACTIVATE_CHAN DeviceIoControl() will return the channel number which should be used for all subsequent communication through the U-Net endpoint. Each endpoint can support up to info.max_chans channels.

Transmit

To transmit data, push a Unet_txbd descriptor onto the Tx FIFO in ring order. The Unet_txbd contains the destination channel number, message length, and a pointer to the transmit buffer . The U-Net device will only transmit messages up to info.max_pdu bytes in length, and transmit buffers should be aligned to info.tx_buf_align bytes.

It is important that the entire Unet_txbd be pushed onto the FIFO at once (e.g., using a structure assignment), as the U-Net device may trigger a transmit when just one field of the structure has been pushed onto the ring.

The Tx wall pointer should be used to determine if the Tx FIFO is full; the entry pointed to by the Tx wall pointer should not be pushed into by the user process. If a Tx FIFO entry is free it will have a chan value of CHAN_FREE.

If the Unet_txbd.buf offset has its low-order bit set to one (as indicated by the TX_CONTINUE macro), then the next Tx FIFO will be transmitted as the same network packet. This allows multiple Tx FIFO entries (and multiple Tx buffers) to be chained into a single packet.

Some implementations of U-Net require a kernel trap in order for the Tx FIFO to be serviced. Servicing the Tx FIFO consists of transmitting new entries and marking already-transmitted entries as free. You can use the info.devid field to determine if a trap is required.

If info.devid == UNET_DEVID_TULIP, TULIP_UNET_TRAP should be called to service the Tx FIFO.
Otherwise, no trap is necessary.

The trap macro takes one argument: The minor device ID, as returned in info.minor.

Note that even if a trap is required, the Tx FIFO may be updated asynchronously by the U-Net device (for example, as a side-effect of other applications invoking the trap).

The U-Net device will set the channel field of entries in the Tx FIFO to CHAN_FREE once the entry has been transmitted, or to CHAN_ERROR if an error occurred during transmit. The U-Net device will update the Tx wall pointer to correspond to the last Tx FIFO entry which has been transmitted.

Receive

As message data is received, the U-Net device will push new entries onto the Rx FIFO. Each entry contains the destination channel tag, length of the message in bytes, and either the message data itself or pointers to buffers containing the message.

If the incoming message is larger than info.rxd_payload_size bytes, a receive buffer contained in the Free FIFO will be allocated. After consuming an entry on the Free FIFO, the U-Net device will set the buf value of the Free FIFO entry to RXB_FREE and increment the Free wall pointer. Buffers on the Free FIFO should be aligned to info.rx_buf_align bytes.

If the message is smaller than (or equal to) info.rxd_payload_size bytes, the pay_or_buf field of the Rx FIFO entry contains the message data itself. Otherwise, the pay_or_buf field contains up to RXD_PAYLOAD_ENTRIES byte offsets into the transmit/receive buffer region, pointing to buffers where the message data is located. Each buffer consumed by the U-Net device stores up to create.rx_buffer_size bytes (as indicated by the user at endpoint creation time).

After consuming an Rx FIFO entry (and its corresponding buffers, if any) the user process should set the chan field of the Rx FIFO entry to CHAN_FREE and update the Rx wall pointer to contain the byte offset (from the beginning of the Rx FIFO) of the last Rx FIFO entry consumed. The U-Net device will not attempt to use the Rx FIFO entry corresponding to the Rx wall; it is used to mark the end of the ring.

When buffers corresponding to incoming messages are consumed they may be pushed back onto the Free FIFO by the user.

If an error occurs during message reception, the U-Net device may set the chan field to CHAN_ERROR. This may happen if, for example, no buffers are available on the Free FIFO, or if a message arrives with a bad CRC. In other cases messages may be silently dropped on reception: Notably in the case where the Rx FIFO itself is full.

Device Shutdown

Deactivating a channel

The user should call the UNET_DEACTIVATE_CHAN DeviceIoControl() on the U-Net device with the channel tag as the argument. This will close down the named channel, making it available for re-use.

Destroying an endpoint

The user should call the UNET_DESTROY_ENDPT DeviceIoControl() on the U-Net device. This will implicitly deactivate any channels left activated by the user.

Closing the device

Upon cloing the U-Net device, the device endpoint and all active channels will be deactivated.

Return to U-Net Documentation.

M. Welsh, mdw@cs.cornell.edu X. Huang, xwh@cs.cornell.edu