README_BUILDING - Guide to building and installing Nprobe
=========================================================


REPOSITORY STRUCTURE
====================

The Nprobe system comprises two major components: the on-line monitoring 
 software (Nprobe), and the post collection analysis suite. Nprobe itself 
 is divided into two parts: kernel level (drivers, buffer handlers, etc.), 
 and data extraction code (Wan). The Wan software is duplicated by an off-line 
 data extraction program (Owan) which extracts data from tcpdump-type logs.

Directory			Contents
---------			--------

analysis			Post-collection log analysis interface, code
				 and tools

analysis/components		Data-retrieval interface and  Python analysis
				 modules
analysis/mogul			Some data-extraction stuff
analysis/nut_bolts		Tools for observing Nprobe activity from logs
analysis/plotter		Data plotter
analysis/py-trivial		Trivial 'try out Python' scripts
analysis/scripts		Executable log analysis scripts
analysis/tkinter_examples	Trivial 'try out Python tkinter' scripts
analysis/tools			Analysis specific Python tools
analysis/utility		Various Python analysis utility modules

atm				Various stuff required for building Nprobe as 
				 an atm monitor

common				C files and headers shared between monitoring
				 and analysis systems

linux				Linux stuff required for building Nprobe

make_tools			Tools used by the make system

minilzo				Log compression stuff (currently unused)

nprobe-modules*			Kernel level Nprobe components

python				Python specific stuff

python/misc			Miscellaneous
python/tools			Various Python tools

runtime				The Nprobe monitor code (Wan/Owan)
runtime/slurp			Stuff to observe Nprobe CPU usage

test_data			Data for built-system testing



REQUIREMENTS
============

Python
------

Most of the analysis code is written in Python - and requires Python 2.2 or 
 above. It should mostly work with later releases, but some features 
 (e.g. generators) will require statements such as 'from future import foo' 
 to be ammended to directly import from the newly introduced modules.

The Python installation used should include TKinter - the Python Tk interface 
 - normally included in standard Python installs.

Python additions
----------------

*Must have*

Some analysis code requires the fast array handling provided by the Python 
 Numerical module (aka numpy) - for details and download see 
 http://www.python.org/topics/scicomp/numpy.html or http://www.numpy.org/

*Desirable*

Data plotting during analysis is implemented using Nprobe's own graphing tool
 np_plot.py (which also makes a good stand-alone plotter) but, although not 
 currently used, Gnuplot has been invoked via the Python Gnuplot interface 
 - for details and download see http://gnuplot-py.sourceforge.net/

Similarly future analysis code may well use the Python stats package - see 
 http://py.vaults.ca/parnassus/apyllo.py/684222876.912038798


SWIG
----

The analysis data retrieval interface is automatically generated using SWIG - 
 for details and download see http://www.swig.org/index.html. The stable 
 release SWIG1.1p5 is preferred - generation using later, enhanced, versions 
 has not yet proved successful.


 
BUILDING AND INSTALLING
=======================

Most dirs/subdirs have their own Makefiles, and 'make' or 'make target' can be 
invoked at that level. Make invoked from the top-level Makefile will descend 
recursively as appropriate to a specific target.

BEFORE ATTEMPTING A MAKE it is necessary to edit the top-level Makefile.paths 
appropriately to reflect the location of your SWIG, Tk, Python, and Python 
headers. Thereafter all make paths will take care of themselves.

It is highly advisable to 'make depend' prior to ammending any code.

'Make', 'make all', or 'make install' should build and install everything. The 
default installation will be to the repository root/install directory - to 
install elsewhere edit INSTALL_DIR in Makefile.paths.

For a complete list and explanation of all top level make targets see 
docs/MAKE_TARGETS or 'make help' at the top level.

The ambitious may wish to jump straight in with 'make wread_test', 'make
 analysis_test', or 'make plotter_test', which will build, install, and run a 
test on the supplied data.

Building the analysis data-retrieval interface and plotter involves the use of 
at least one Python script, so it will be necessary to have at least a 
rudimentary, Python on your PATH before commencing (although not necessarily 
the same Python as used to run analysis scripts) - see PATHS below. 

PATHS
=====

The shell script make_tools/paths_setup.sh, when called with the correct 
arguments, will prepend the necessary paths to the environmental variables 
PATH, LD_LIBRARY_PATH, and PYTHONPATH to locate and execute the Nprobe monitor,
 analysis software and scripts, and for Python to locate analysis modules. 
It will also add the Python installation to PATH. 

For convenience, 'make paths' and its variants will print the full command and 
arguments necessary to run paths_setup.sh. Note that the 'paths_raw' variant 
will add the appropriate source directories to paths, while the basic variant 
adds the install directories. Simply 'make paths' or similar, then cut, paste, 
and execute its output.

Note also that paths_setup.sh will not prepend paths which are also included 
in the existing *PATH, so problems will occur if executables are to be found 
on the wrong path if occuring in the variable before the desired path.

'Make python_path' will produce the text of the command necessary to add the 
Python specified in Makefile.paths to your PATH if required before commencing 
a build.

AUTO-GENERATED FILES
====================

The following files are generated by make tools:

File				Sources				Purpose
	
runtime/probe_config.c/h	probe_config 		Probe configuration
				probe_config.tmpl

runtime/parse_http_fields.c	http_fields		HTTP header field
				parse_http_fields.tmpl	parser

runtime/content_t.c		content_types		HTTP Content-type
common/hdr/content_t.h		content_t.tmpl		header field parser
common/src/print_type.c


Auto-generation is triggered by 'make config_probe', 'make content_t', 
'make parse_http_fields', or 'make generate'/'make depend' which will do the 
lot. 

See runtime/probe_config, http_fields, content_types for a further explanation 
of the contents of these configuration files. In particular see section in 
content_types dealing with saving auto-allocated type codes for legacy logs.

PROBE_CONFIGURATION
===================

Various aspects of probe behaviour, protocol treatment, etc., can be controlled
by appropriate parameters in runtime/probe_config. See this file and above 
Section for details.

Currently configuration is hard coded by auto-generated probe_config.c/h files 
and reconfiguration requires a recompile. Hopefully to come soon - dynamic 
(re)configuration of live probes.
