
                                                                               
    Copyright 2005 University of Cambridge Computer Laboratory.                
                                                                               
    This file is part of Nprobe.                                               
                                                                               
    Nprobe is free software; you can redistribute it and/or modify             
    it under the terms of the GNU General Public License as published by       
    the Free Software Foundation; either version 2 of the License, or          
    (at your option) any later version.                                        
                                                                               
    Nprobe is distributed in the hope that it will be useful,                  
    but WITHOUT ANY WARRANTY; without even the implied warranty of             
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the              
    GNU General Public License for more details.                               
                                                                               
    You should have received a copy of the GNU General Public License          
    along with Nprobe; if not, write to the Free Software                      
    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA  
                                                                               



The sk98 clock runs at (about) 31.25MHz (or sometimes 78.11MHz,
depending on the exact model), and is completely independent of the
host clock.  It also varies slightly depending on temperature.  On the
other hand, it does stamp packets as the come off the wire, and is
very, very low overhead.

When packets come out of the card, they have a timestamp A which gives
the value of the sk98 tick counter when the packet was received.  If
we're using the packet for timer calibration, we also sample the
system time when we start processing the packet, B.  The aim is to
find a linear equation which gives B from A.

Life is significantly easier if say that time 0 is when we started
calibration for both clocks, so that's what we do.

sk98_timers.c is responsible for the runtime calibration.  It gets fed
(via doTimer) every packet which we receive, and is supposed to
calculate from that what the parameters of the sk98 clock are.  We do
this via a linear regression between two quantities, time and tdiff,
where time is how long it took from getting a packet to the sk98 and
getting the system time, and tdiff is how far through the calibration
run we are according to the system clock.  time is *estimated* using
the current clock calibration parameters.

We extract from these the formula time = inter + grad * tdiff.  At
tdiff = 0, time is clearly the delta between system and sk98 time, and
so we just subtract that out from future timestamps.  If we have a
good calibration, then time will be a constant, and so grad gives the
drift rate of the clock: this can be used to get a more accurate
frequency estimate.  mon will print out an estimate of the standard
deviation of the absolute error when it makes a frequency estimate;
there's no estimate of frequency error, but it should be roughly one
part per million (because if it's significantly more than that, mon
tries again).

The thing to note here is that while mon's clock calibration algorithm
is pretty clever (the error in the offset is swamped by the error in
the system clock, if we're using NTP to sync it, and the drift error
is less than 1ppm), it relies on us having a pretty good initial
estimate of the clock parameters.  Offset can be safely approximated
to zero, but the frequency needs to be obtained from somewhere, and so
we have a separate program, timer, to calculate it.

If mon is run with the -o option, it will produce a drift file.  This
is the input to timer.  For an end-to-end frequency estimation (i.e.
estimating the average frequency over the whole run), you want to
first run timer as: timer -f /path/to/drift_file -o /dev/null.  This
will tell you how many end records are available.  Divide that number
by two to get X, and then go
timer -f /path/to/drift_file -o /dev/null -e X.  This will tell you an
estimate of the frequency, which can then be fed to mon via its -n
parameter.

Note that timer needs *lots* of records in order to give you a good
answer.  A few thousand is the minimum.  In general, the value fed to
mon needs to be good to within about one part in a thousand before mon
can get a reasonable frequency lock.  If you get lots of messages
about either dumping or losing samples, the starting frequency isn't
good enough.



Programming model
-----------------

The timestamp calibration stuff is all in sk98_timers.c.  The
interface is in sk98_timers.h.  When your application starts, call
initialise_timestamps(a, b, c).

a -> enable clock recalibration if 1, disable if 0.  If recalibration is
     disabled, we assume the offset to be 0 and take the passed in
     frequency as gospel.
b -> Initial frequency.  Can be 0 to take a reasonable guess based on the
     card design.
c -> Filename for drift statistics.  Can be NULL to disable drift collection.

You then need to create a clock retimer for each interface.  This is
done by calling new_clock_retimer(name, number).  Name is the
interface name, e.g. "eth2", and number is an entirely arbitrary
interface number which gets put in drift files.

Packets can be fed into the recalibration code using doTimer(timer,
tstamp, total_pkts), where timer is the earlier created clock retimer,
tstamp is the sk98 timestamp for the packet, and total_pkts is the
number of packets received so far.  total_pkts is only used in the
drift file.  This function should be called shortly after receiving
the packet from the card.  It returns 1 iff the card has transitioned
from a non-calibrated state to a calibrated one.

To convert an sk98 timestamp to a system time, call getTime(timer,
tstamp, &tv, &ts), where timer is the retimer and tstamp is the sk98
timestamp.  tv is a struct timeval which will be filled out with the
system time corresponding to tstamp; ts is a struct timespec which
fills a similar role.  Either or both of tv and ts can be NULL.

There is also a function skb_to_nictstamp(skb) which will extract a
timestamp from an escalator-style skb, and
card_tstamp_getcurtime("eth2"), which will get the current value of
the sk98 timer.

The global variable drift_period is used to select the frequency at
which we take samples; we wait at least drift_period tics between
samples, and at most drift_period * 9.  Note that drift_period is
measured relative to the sk98 clock, rather than the system clock.

Validation
----------

The only way I've found to validate the results coming out of this is
to use a fibre splitter to send data to both an sk98 card running this
calibration and a card with a very accurate clock.  DAG boards work
quite nicely in that respect.  Plot the differences in reported times
between the two cards against time.  The y-intercept gives you the
offset error, while the gradient gives you the drift rate.

You can check NTP sync by running tcpdump on both machines while
pinging between them.  They will report seeing packets at different
times, and the time difference tells you how far out of sync their
clocks are.  You can eliminate network delay by looking at packets in
both directions.  Once you've got the difference between the host
clocks, it's fairly easy to calculate the amount of error due to the
sk98 clock sync.  This isn't hugely accurate, but when I tried it the
sk98 offset error estimate was about a tenth the NTP offset error
estimate, so I didn't bother going any further.

You'll probably find that, at any given point in time, the difference
between the two cards will jump backwards and forwards between two
values which are about a microsecond apart.  This is due to truncating
the timestamps to 1us for dumping to pcap format, and allows you to
bound the jitter as not more than 2us.

FUTURE WORK
-----------

-- Make invoking timer less unpleasant
-- Make it so that timer isn't necessary any more
-- Recalibrate as we run.  Frequency is probably more important than
   offset here, and would also be less likely to violate the
   time-is-monotonically-increasing bit.
