Xilinx components have been extensively used in the project as their reprogrammability provided the ability to migrate functions from software to hardware in the light of experience. Furthermore as both de jure and de facto standards have been accepted, it has been possible to modify the firmware to track these changes - this enables us continue to use the Fairisle equipment for switch and network experiments while integrating it with commercially available equipment.
The firmware underwent several redesigns to add functionality; the relevant features of ``Xi2'', ``Xi3'' and ``Xi5'' designs (i.e. those that have seen service) are discussed.
Initially the Xilinx chip did little more than to read cells into and out of the SRAM into buffer slots allocated by software and passed to the hardware via two shared 8 entry circular buffers, one each for reception and transmission. On reception the hardware would copy the cell to the first reception buffer and interrupt the processor. The interrupt routine would check the received VCI, perform the VCI translation, add the relevant routeing information, and append to a queue. The service process of these queues, invoked on a successful transmission, would remove entries and pass to the transmission hardware via the circular buffer.
At an early stage, VCI mapping was added to the hardware, as it was observed that the synchronisation cost of the external memory cycles required to read and write the header was having an adverse effect on the processor performance. The VCI mapping and routeing information are stored in the SRAM buffer memory and the mapping function can be invoked on a per cell basis by use of a control bit in the transmit descriptor; transmission without mapping is used for control cells which originate from the local processor.
The first traffic on the network was that generated by various packet based sources (e.g. switches booting and managing themselves!) and it was observed that, even at comparatively low mean utilisations, the occurrence of NACK events from the fabric was sufficiently frequent that the cost of handling such events in software (which often involved reloading the transmit queue) was unacceptable. This cost was decreased by the addition of a 1 bit retry count to the transmit descriptor.
After these modifications the Xilinx design was known as ``Xi2''. This design is discussed in detail in [Orange93].
Performance evaluation of the cell forwarding software on the port controller revealed that much of the time was spent manipulating the short circular buffers and the free queue; furthermore the off-processor operations required to perform these were expensive. While it was desirable to continue to have the processor active in implementing the cell forwarding queues (this is one of the research areas the equipment is designed to investigate), the free queue management and the interface between software and hardware were redesigned.
The cell buffers fall into four classes: waiting collection by the reception FIQ, on the free queue, on the transmit queue, and being used by software. The cell buffers in the first three classes are linked together by cell buffer numbers in their link field. These form a single list with pointers in the Xilinx chip marking the boundaries as shown in figure 4.
Figure 4: Xi3 queue structure.
The queue must be set up by the processor before starting the Xilinx.
The queue is automatically maintained because the processor adds
buffers for transmission to the tail. Once these are sent they form
part of the free queue before being used for reception. The length of
the transmission section is limited to a maximum of eight in a trade
off between the desires to pipeline the transmission function and
provide software with a reasonably timely indication when congestion
is experienced - without this limit and indication the most cunning
queueing software invented would degenerate at times of
congestion
to fifo
behaviour. The same requirement for a short feedback loop exists in
the design of a host interface (and associated device driver) which
aims to implement per channel priority and rate control.
Buffers not in the queue are assumed in use by software, and must be entirely managed by software. This is straightforward while the processor is not a net source or sink of cells (e.g. signalling cells); if it is, both dummy transmissions and receptions are implemented to enable software to correct any imbalance.
The link field in each buffer contains not only the next buffer in the list, but also the number of retries (between zero and seven inclusive). Sensible use of retry counts actually reduces the demand on the processor during contention in the fabric, so that at these critical times, cycles are available for more complex queueing algorithms.
The interrupt interface was also modified so that in a single operation it is possible for the software to read the interrupt condition, the buffer number at the head of the list, and if a cell arrival has occurred cause the pointer to the head of the list to reload as a side effect. Such interfaces carefully constructed to match the commonly required operations of the software considerably speeded the interaction. With this design, known as ``Xi3'' [Orange93] almost the full line rate could be handled (see section 5).
The most recent iteration of the Port Controller Hardware, done in the context of a British Telecom funded follow-on project, includes a larger Xilinx device (3090). The design for this chip, known as ``Xi5'' [Orange93], is from the point of view of cell management the same as the ``Xi3'' design; the additional space has been used to implement telemetry support for more detailed measurements and experimental purposes.
The requirements of the theoreticians and modellers were for traces of cell arrivals and departures from switches; for each cell handled it was desired to record the channel identifier of each cell and these two times. This presents some problems, as each 100Mbit/sec link is generating about 10Mbit/sec of trace, and not satisfied with one port's observations, we are required to do it for whole switches.
The solution adopted is based on the observation that when performing such experiments, the important aspect of some streams is simply their temporal behaviour, rather than contents (e.g consider an open loop variable rate coded video stream). Hence the Xilinx component now contains a free running counter (which ticks at the rate of the switch fabric framing signal) and may be configured per VCI to stamp the value of this counter into the data portion of the cell when a cell arrives and leaves the system. The position of the stamp is determined by some bits in the VCI, so different switches can be configured to place the information at different points in the cell.
The ARM fast interrupt (FIQ) is reserved for use by the cell forwarding hardware. When handling a FIQ, the ARM processor remaps part of its register set to provide private registers for the FIQ handler - this enables state to be maintained between FIQ invocations without having to save and restore state to memory, greatly reducing the interrupt entry and exit overhead.
The structure of the list data structure shared between the processor and hardware is shown in figure 4. An interrupt is generated when a cell arrives, if there are less than eight buffers on the transmit section of the queue, or if a NACK occurs. On entering the FIQ handler, the interrupt status is obtained and the used to dispatch to the correct handler routine; this process takes a maximum of 7 instructions of which one is an non-cached data read.
On reception, the incoming VCI is read from the received buffer, and, after range checking, used to index into the VCI table. Having obtained the queue pointer for this VCI from the table, the buffer number is appended to the queue. Worst case for this path is 24 instructions of which one is a non-cached data read.
On transmission, the default configuration serves three queues in strict priority - one of these queues is used for all forwarded cells. In this configuration, the instruction count for transmission of forwarded cells is 16, with an additional 5 instructions for each extra queue that must be checked. More complex strategies with multiple queues and fair service disciplines are also available for specific experiments.