Summarizing Benchmarks Is No Mean Feat
John Mashey *
For decades, computer benchmarkers have fought a War of Means, arguing over
proper uses of Arithmetic, Harmonic, and Geometric Means, starting in the
mid-1980s. One would think this simple issue of computer performance
analysis would have been long resolved, but new papers still argue about it,
and contradictions are still present in some excellent and widely-used
textbook.
This argument seems to persist only because the argument was wrongly or
incompletely framed years ago, and people's mindsets have stayed within that
framing, which has tended to use algebra, but not statistics. However,
evidence is accumulating that CPU performance ratios are well-characterized
by lognormal distributions, which provide means, standard deviation, and
confidence limits, i.e., all the good things statisticians use routinely,
but for some reason appear rarely in computer architecture. These require
use of the Geometric Mean, which becomes obvious if one rewrites the common
form of the GM into a mathematical equivalent based on the sum of
logarithms. It is common practice to say "Performance is not a single
number," and then give a single number anyway.
No matter how badly people want a single number for performance, it is
better described by a distribution than by a mean alone. Hopefully this
work provides a good mathematical basis for some better performance analysis
techniques. Examples are taken from SPEC CPU benchmarks and Livermore
Fortran Kernels.
*
Dr. John Mashey is an ancient [1973-1983] Bell Labs UNIX software person,
who moved to Silicon Valley in 1983, joining Convergent Technologies to
ventually manage software for the server division. In late 1984, he got
recruited to join MIPS Computer Systems, managing operating systems,
networking and software QA, and was one of the designers of the MIPS R2000
and later architectures. He was later VP Systems Technology at MIPS, and
one of the founders of the SPEC benchmarking consortium in 1988. Joining
Silicon graphics in 1992, he was an troubleshooter/toublemaker in hardware
and software, ending as VP and Chief Scientist. He was active in 64-bit
hardware and software design, and was one of the architects for the SGI
NUMAflex systems (Origin 3000 and Altix). Currently he consults for venture
capitalists and high-tech companies, sits on Technical Advisory Boards for
hardware and software startups, and is a Trustee of the Computer History
Museum, writes occasional technical articles, and gives occasional lectures.
Dr. Mashey has given 500+ public talks over the years, of which the "Small
is Beautiful" talk was frequently requested during 4 years of being an ACM
National Lecturer in the early 1980s. Recently, he's given it again as
invited talks at several conferences, as old-timers decided that it needed
be heard again, and as a strong believer in permanent storage, he still had
the original foils.
He has long connections to the UK, being married to a Yorkshire (& Cambridge
& Imperial College) lady, having often met with GCHQ, Scotland Yard, Glaxo,
Unilever, etc, and having once spent several hours with Peter Mandelson
explaining how Silicon Valley worked.
