It is sometimes all too easy to focus too much on the results of a particular benchmark and let this "tunnel vision" take over all performance work.

It is very important that a wide selection of standard industry benchmarks exists, as they will be used by anyone from hardware vendors to undergraduate researchers, for ultimately increasing performance for various runtimes and applications. Conclusions drawn, given a benchmark setup, can have wide-ranging implications.

One danger is, of course, if a mainstream benchmark gets too well adopted. Classic examples here are the SPECjvm benchmark suite and later the SPECjbb benchmark.

To put it bluntly, if a graduate student can run a benchmark from his workstation with a simple command line, the problems it addresses will get much love and research. If not, they won't. SPECjbb has a simple command line, SPECjAppServer doesn't. SPECjAppServer, in itself, is an excellent benchmark that pretty much stresses any desired portion of a Java platform stack and can be used to test everything from application servers to network cards in a realistic environment. SPECjAppServer requires quite a lot of hardware and has an extensive hard-to-configure setup. Consequently, few relevant publications have been released using SPECjAppServer in the academic world, but plenty of SPECjbb results are still published, even though it may not be that relevant for all aspects of modern performance research.