
   ************************************************************
   * Arsenic 1.0: User-Accessible Gigabit Ethernet Networking *
   ************************************************************

   Ian Pratt and Keir Fraser
   Systems Research Group
   Computer Laboratory
   University of Cambridge



OVERVIEW
========

The Arsenic software distribution contains the following major software
components:
 
 * new `flow-aware' firmware for the Alteon ACEnic Gigabit Ethernet adaptor;

 * a kernel module for Linux 2.3.29 allowing user-space protocols over IP
   to coexist with their kernel equivalents;

 * a full port of the Linux 2.3.29 TCP/IP protocol-processing stack to 
   user-space, in the form of a shared library.

The design and implementation of the above are described in our paper 
"Arsenic: A User-Accessible Gigabit Ethernet Interface", which has been
submitted to INFOCOM 2001. A draft version of the paper is available for 
download at:
http://www.cl.cam.ac.uk/Research/SRG/netos/arsenic/gige.ps



*** CAVEAT ***
==============

To use Arsenic you will need an x86 machines and an
Alteon ACEnic 1000BASE-SX adaptor. The adaptor _must_ contain a Revision 5
Tigon II ASICs and 1MB of SRAM.

If you do not have the above hardware, DO NOT BOTHER TO CONTINUE!

This release has had various sections of proprietary code removed. It is 
believed to work, but lacks some functionality described in the paper.



LICENSE AGREEMENT
=================

This software distribution is protected by a number of different license
agreements. Please read the LICENSE file for more details.



BUILDING & INSTALLING ARSENIC
=============================

This involves building six main components:
 1. Cross-compilation tools
 2. Flow-aware firmware
 3. Patched Linux 2.3.29 kernel
 4. In-kernel support modules
 5. User-threads library 
 6. User-level TCP stack and test utilities

Some users may wish to build their own user-level protocol stacks. They may
wish to only build the first four components, and refer to the code in 
user-tcp/lowlevel.c which provides a simple API for accessing the network 
adaptor from user space.


STAGE 1: Cross-compilation tools
--------------------------------

To build the flow-aware firmware you will need a cross-compiler targeted
for MIPS-compatible processors. The first stage is to build and install
the cross-compilation tools.

 1. Download gcc-2.8.1 and binutils-2.9.1 (main download site is ftp.gnu.org)
    and untar them in the arsenic-1.0 directory. This will provide you with
    gcc-2.8.1/ and binutils-2.9.1/ subdirectories containing the source trees
    for the two packages.

 2. Build and install binutils. Your configuration parameters should be
    along the lines of:
     --host=i686-pc-linux-gnu
     --prefix=$(MIPS_CROSS_COMPILE_INSTALL)
     --target=mips-any-elf
     --with-gnu-ld
     --enable-commonbfdlib
     --enable-64-bit-bfdmake
    (NB. MIPS_CROSS_COMPILE_INSTALL is a suitable installation directory)

 3. Ensure that $(MIPS_CROSS_COMPILE_INSTALL)/bin is on your PATH.

 4. Build and install gcc. Your configuration parameters should be along the
    lines of:
     --host=i686-pc-linux-gnu
     --prefix=$(MIPS_CROSS_COMPILE_INSTALL)
     --target=mips-any-elf
     --with-gnu-as
     --with-gnu-ld
    Then:
     make LANGUAGES=c && make install LANGUAGES=c

 5. Apply the Alteon-supplied patch to the binutils source tree, and rebuild:
     > cd binutils-2.9.1
     > patch <../patches/alteon-binutils-2.9.1-patch
     > make clean && make && make install


STAGE 2: Flow-aware firmware
----------------------------

Now that the cross-compilation tools are installed, you can build the 
flow-aware firmware. 

 1. After ensuring that $(MIPS_CROSS_COMPILE_INSTALL)/bin is on your PATH:
     > cd acenic-fw-12.3.10/nic/fw2/common
     > make


STAGE 3: Patched Linux 2.3.29 kernel
------------------------------------

The Arsenic device driver and protocol support module require a patched
Linux 2.3.29 kernel. This stage builds such a kernel.

 1. Untar and patch a Linux 2.3.29 source tree inside the arsenic-1.0
    directory:
     > tar -xvzf linux-2.3.29.tar.gz
     > mv linux linux-2.3.29
     > cd linux-2.3.29
     > patch -p1 <../patches/bigphysarea-2.3.29-patch
     > patch -p1 <../patches/arsenic-2.3.29-patch

 2. Configure the kernel as usual, making sure you enable standard networking
    options. Don't worry about selecting the Alteon ACEnic driver: that is
    compiled outside tke kernel tree. An example configuration file is
    included if you wish to use it:
     > cp example-2.3.29-config linux-2.3.29/.config

 3. Build and install the kernel as described in the kernel documentation.


STAGE 4: In-kernel support modules
----------------------------------

This stage builds the Arsenic device driver and user-level networking
support module.
     > cd gige-mods
     > make


STAGE 5: User-level threads library
-----------------------------------

The user-level TCP implementation provided with the Arsenic distribution
requires a modified version of the GNU Pth user-level threads library. This
is used to separate protocol processing from application-level processing.

Building the modified threads library is likely to be the biggest stumbling 
block for most people. See the README.FIRST file for details and
hints about building and installing the package.


STAGE 6: User-level networking utilities
----------------------------------------

 1. Enter the user-tcp directory and edit the PTH_INSTALL variable in 
    Rules.mk to the same value you set `prefix' in the Pth Makefile.

 2. There are various other interesting things to tweak in Rules.mk:
     LINUXROOT and MODULES_ROOT: If you built Linux and/or the support
                                 modules outside the arsenic-1.0 directory.
     CFLAGS: -D__TEST_NO_CHECKSUMS == disables receive checksumming
     DEBUG:  -DNDEBUG              == disables _all_ tracing and assertions

 3. Things to tweak in the Makefile:
     TRC: This is described in the Makefile, and allows you to specify
          per-file trace levels.

 4. Build the library and test suite:
     > make depend && make



TRYING IT OUT
=============

Install the patched kernel on two test machines connected by a pair of 
Alteon ACEnic 1000BASE-SX adaptors. The adaptors _must_ contain Revision 5
ASICs and 1MB of SRAM.

When the patched kernel is booted, do the following (or similar) as root:
 > insmod gige-mods/acenic.o trace=0
 > ifconfig eth1 $(GIGE_ADDR) up mtu=8244
 > insmod gige-mods/user.o
 (IP_ADDR could be 10.0.0.1 on one machine and 10.0.0.2 on the other, 
  for instance)

Now, on machine A:
 > user-tcp/test/recv_tcp 2025

On machine B:
 > user-tcp/test/send_tcp 10.0.0.1 2025 1 1000000

With any luck, you should see a bulk TCP transfer going at up to 980Mbps :-)



TESTING & DEBUGGING
===================

Kernel modules
--------------

Both kernel modules have debugging support which can be enabled from the
Makefile. Tracing goes to the kernel message ring.


User-space utilities
--------------------

The TCP library tracing can be controlled on a per-file basis. Remove
-DNDEBUG from Rules.mk and edit the TRC variable in the Makefile.


Firmware
--------

Enter the fw-debugging directory and `make'. This will build the trace_dump
utility which can be used as follows (as root):
 > fw-debugging/trace_dump eth1

This will dump the contents of the network card's trace buffers to stdout.
Firmware tracing can be enabled at compile time by editing 
acenic-fw-12.3.10/nic/fw2/common/alt.tg1.gnu and setting TRACE and/or
TRACE_ALL to 1 rather than 0. TRACE_ALL should not usually be enabled, as
it floods the trace ring.

Firmware events can be selected for tracing when the device driver is 
installed:
 > insmod gige-mods/acenic.o trace=$(TRACE)

TRACE is a bitmask selecting event types to trace (assuming you set TRACE to
1 when you built the firmware!). The event types are listed in
acenic-fw-21.3.10/nic/common/nic_api.h:

#define TRACE_TYPE_SEND         0x00000001
#define TRACE_TYPE_RECV         0x00000002
#define TRACE_TYPE_DMA          0x00000004
#define TRACE_TYPE_EVENT        0x00000008
#define TRACE_TYPE_COMMAND      0x00000010
#define TRACE_TYPE_MAC          0x00000020
#define TRACE_TYPE_STATS        0x00000040
#define TRACE_TYPE_TIMER        0x00000080
#define TRACE_TYPE_DISP         0x00000100
#define TRACE_TYPE_MAILBOX      0x00000200
#define TRACE_TYPE_RECV_BD      0x00000400
#define TRACE_TYPE_LNK_PHY      0x00000800
#define TRACE_TYPE_LNK_NEG      0x00001000
#define TRACE_TYPE_TX2_TRACE    0x00002000
#define TRACE_TYPE_TX2_DEBUG    0x00004000
#define TRACE_TYPE_TX2_STATS    0x00008000
#define TRACE_TYPE_RX2_TRACE    0x00010000
#define TRACE_TYPE_RX2_DEBUG    0x00020000
#define TRACE_TYPE_RX2_STATS    0x00040000
#define TRACE_TYPE_RX2_IRQ      0x00080000
#define TRACE_TYPE_TX2_IRQ      0x00100000
#define TRACE_TYPE_TX2_SCHED    0x00200000
#define TRACE_LEVEL_1           0x10000000
#define TRACE_LEVEL_2           0x20000000

Our default setting for TRACE is 0x6c000.


 ---------------------------------------------
 Ian Pratt and Keir Fraser, 2nd September 2000
