Computer Laboratory Home Page Search A-Z Directory Help
University of Cambridge Home Computer Laboratory
Computer Laboratory > Abstracts

Seminars will be held in the Lecture Theatre 1 - William Gates Building, Computer Laboratory at 4.15pm
for the research groups' seminars:

 Networks & OS seminars
 Security seminars
 Logic and Semantics seminars
 weekly timetable

Summarizing Benchmarks Is No Mean Feat

John Mashey *

For decades, computer benchmarkers have fought a War of Means, arguing over proper uses of Arithmetic, Harmonic, and Geometric Means, starting in the mid-1980s. One would think this simple issue of computer performance analysis would have been long resolved, but new papers still argue about it, and contradictions are still present in some excellent and widely-used textbook.

This argument seems to persist only because the argument was wrongly or incompletely framed years ago, and people's mindsets have stayed within that framing, which has tended to use algebra, but not statistics. However, evidence is accumulating that CPU performance ratios are well-characterized by lognormal distributions, which provide means, standard deviation, and confidence limits, i.e., all the good things statisticians use routinely, but for some reason appear rarely in computer architecture. These require use of the Geometric Mean, which becomes obvious if one rewrites the common form of the GM into a mathematical equivalent based on the sum of logarithms. It is common practice to say "Performance is not a single number," and then give a single number anyway.

No matter how badly people want a single number for performance, it is better described by a distribution than by a mean alone. Hopefully this work provides a good mathematical basis for some better performance analysis techniques. Examples are taken from SPEC CPU benchmarks and Livermore Fortran Kernels.

* Dr. John Mashey is an ancient [1973-1983] Bell Labs UNIX software person, who moved to Silicon Valley in 1983, joining Convergent Technologies to ventually manage software for the server division. In late 1984, he got recruited to join MIPS Computer Systems, managing operating systems, networking and software QA, and was one of the designers of the MIPS R2000 and later architectures. He was later VP Systems Technology at MIPS, and one of the founders of the SPEC benchmarking consortium in 1988. Joining Silicon graphics in 1992, he was an troubleshooter/toublemaker in hardware and software, ending as VP and Chief Scientist. He was active in 64-bit hardware and software design, and was one of the architects for the SGI NUMAflex systems (Origin 3000 and Altix). Currently he consults for venture capitalists and high-tech companies, sits on Technical Advisory Boards for hardware and software startups, and is a Trustee of the Computer History Museum, writes occasional technical articles, and gives occasional lectures.

Dr. Mashey has given 500+ public talks over the years, of which the "Small is Beautiful" talk was frequently requested during 4 years of being an ACM National Lecturer in the early 1980s. Recently, he's given it again as invited talks at several conferences, as old-timers decided that it needed be heard again, and as a strong believer in permanent storage, he still had the original foils.

He has long connections to the UK, being married to a Yorkshire (& Cambridge & Imperial College) lady, having often met with GCHQ, Scotland Yard, Glaxo, Unilever, etc, and having once spent several hours with Peter Mandelson explaining how Silicon Valley worked.

Wednesday Seminars

Click here for previous seminars.