A light-Weight Communication System for a High Performance System Area Network Amelia De Vivo

Communication Systems for SANs

As we saw in the previous section, SANs was introduced mainly to support user-level communication systems, so that the operating system involvement in the communication task can be reduced as much as possible. Such communication systems can be substantially divided in two software layers. At the bottom, above the network hardware, there are the network interface protocols, that control the network device and implement a low level communication abstraction that is used by the higher layers. The second layer is present in most communication systems, but not all, and consists of a communication library that implements message abstractions and higher level communication primitives.

User-level communication systems can be very different among them, according to several design choices and specific SAN architectures. Various factors influence the performance and semantics of a communication system, mainly the lowest layer implementation. In [BBR98] six issues on the network interface protocols are indicated as basic for the communication system designers: data transfer, address translation, protection, control transfer, reliability and multicast.

The data transfer mechanism significantly affects latency and throughput. Generally a SAN is provided with a DMA engine for moving data from host memory to NIC and vice versa, but in many cases programmed I/O is allowed too. DMA engines can transfer entire packets in large bursts and proceed in parallel with host computation, but they have a high start-up cost. With programmed I/O, instead, the host processor must write and read data to and from the I/O bus, but it can do it typically one or two words at a time resulting in a lot of bus transactions. Choosing the suitable type of data transfer depends on the host CPU, the DMA engine and the packet size. A good solution can be using programmed I/O for short messages and DMA for longer ones, where the definition of short message changes according to the host CPU and the DMA engine. This is effective for data transfers from host memory to NIC, but reads over the I/O bus are generally much slower than DMA transfers, so most protocols use only DMA in this direction. Because DMA engines work asynchronously, host memory being source or destination of a DMA transfer cannot be swapped out by the operating system. Some communication systems use reserved and pinned areas for DMA transfers, others allow user processes to pin a limited number of memory pages in their address space. The first solution imposes a memory copy into a reserved area, the second requires a system call utilization.

The address translation is necessary because DMA engines must know the physical addresses of the memory pages they access. If the protocol uses reserved memory areas for DMA transfers, each time a user process opens the network device, the operating system allocates one of such areas as a contiguous chunk of physical memory and passes its physical address and size to the network interface. Then the process can specify send and receive buffers using offsets that the NIC adds to the starting address of the respective DMA area. The drawback of this solution is that the user process must copy its data in the DMA area increasing the software overhead. If the protocol does not make use of DMA areas, user processes must dynamically pin and unpin the memory pages containing send and receive buffers and the operating system must translate their virtual addresses. Some protocols provide a kernel module for this purpose, so that user processes, afte