The Rbuf design separates the three issues of I/O buffering, namely:
An I/O channel is comprised of a data area (for the actual data) and some control areas (for the aggregation information). The memory allocation is managed independently of the I/O channel by the owner of the memory.
DATA AREA
The Rbuf Data Area consists of a small number of large contiguous regions of the virtual address space. These areas are allocated by the system and are always backed by physical memory. Revocation of this memory is subject to out of band discussion with the memory system. To as large an extent as possible the memory allocator will keep these contiguous regions of virtual addresses backed by contiguous regions of physical addresses (this is clearly a platform dependent factor).
The system provides a fast mechanism for converting Rbuf Data Area virtual addresses into physical addresses for use by drivers that perform DMA. On many platforms a page table mapping indexed by virtual page number exists for use by the TLB miss handler; on such platforms this table can be made accessible to device driver domain with read only status.
Protection of the data area is determined by the use of the I/O channel. It must be at least writable in the domain generating the data and at least readable in the domain receiving the data. Other domains may also have access to the data area especially when an I/O channel spanning multiple domains (see section vi-C.4) is in use.
One of the domains is logically the owner in the sense that it allocates the addresses within the data area which are to be used.
The Rbuf data area is considered volatile and is always updateable by the domain generating the data.
DATA AGGREGATION
A collection of regions in the data area may be grouped together (e.g. to form a packet) using a data structure known as an I/O Record or iorec. An iorec is closest in form to the UNIX concept of an iovec. It consists of a header followed by a sequence of base and length pairs. The header indicates the number of such pairs which follow it and is padded to make it the same size as a pair.
This padding could be used on some channels where appropriate to carry additional information. For example the exact time at which the packet arrived or partial checksum information if this is computed by the hardware. [37] points out that for synchronisation it is more important to know exactly when something happened than getting to process it immediately.
CONTROL AREAS
A control area is a circular buffer used in a producer / consumer arrangement. A pair of event channels is provided between the domains to control access to this circular buffer. One of these event channels (going from writer to reader) indicates the head position and the other (going from reader to writer) indicates the tail.
A circular buffer is given memory protection so that it is writable by the writing domain and read-only to the reading domain. A control area is used to transfer iorec information in a simplex direction in an I/O channel. Two of these control areas are thus required to form an I/O channel and their sizes are chosen at the time that the I/O channel is established.
Figure 6 shows a control area with two iorecs in it. The first iorec describes two regions within the Rbuf data area whereas the second describes a single contiguous region.
Figure 6: Rbuf memory arrangement
USAGE
Figure 7 shows two domains A and B using control areas to send iorecs between them. Each control area, as described above, provides a fifo queue of iorecs between the two ends of an I/O channel. Equivalently, an I/O channel is composed of two simplex control area fifos to form a duplex management channel. The control areas are used indistinguishably no matter how the I/O channel is being used.
Figure 7: Control Areas for an I/O channel
between A and B
A typical I/O channel is in fact a simplex data channel operating in one of two modes. The purpose of these two modes is to allow for the support of ADUs in various contexts. Note that there is no requirement for either end of the I/O channel to process the data in a FIFO manner, that is merely how the buffering between the two ends is implemented.
In Transmit Master Mode (TMM), the originator of the data chooses the addresses in the Rbuf data area, places the data into the Rbufs, and places the records into the control area. It then updates the head event for that control buffer indicating to the receiver that there is at least one record present. As soon as the downstream side has read these records from the control buffer it updates the other (tail) event, freeing the control buffer space for more records. When the downstream side is finished with the data it places the records into the control area for the queue in the other direction and signals its head event on that control buffer. The originator likewise signals when it has read the returned acknowledgement from the control buffer. The originator is then free to reuse the data indicated in the returning control buffer.
In Receive Master Mode (RMM), the operation of the control areas is indistinguishable from TMM; the difference is that the Rbuf data area is mapped with the permissions reversed and the data is placed in the allocated areas by the downstream side. It is the receiver of the data which chooses the addresses in the Rbuf data area and passes iorecs which indicate where it wishes the data to be placed to the downstream side. The downstream side uses the other control area to indicate when it has filled these areas with data.
The Master end, which is choosing the addresses, is responsible for managing the data area and keeping track of which parts of it are ``free'' and which are ``busy''. This can be done in whatever way is deemed appropriate. For some applications, where FIFO processing occurs at both ends, it may be sufficient to partition the data area into iorecs at the initiation of an I/O channel, performing no subsequent allocation management.
Table i presents a summary of the differences between TMM and RMM for the diagram shown in figure 7; without loss of generality A is the master - it chooses the addresses within the data area.
Table i: TMM and RMM properties
Since the event counts for both control areas are available to a user of an I/O channel it is possible to operate in a non-blocking manner. By reading the event counts associated with the circular buffers, instead of blocking on them, a domain can ensure both that there is an Rbuf ready for collection and also that there will be space to dispose of it in the other buffer. This functions reliably because event counts never lose events. Routines for both blocking and non-blocking access are standard parts of the Rbuf library.