Craig B. Zilles
Abstract
In the past couple of years, a number of software and architectural techniques have been proposed for improving the performance of linked data structrues. These research ideas are often evaluated using the Olden benchmark suite [1]. Frequently, in such experients, the largest speed-up is attained for the benchmark called health. This article demonstrates that this benchmark is a micro-benchmark for enormous linked lists traversals, and not a good one at that. Given that linked lists of such size are not an efficient data structure, it is unlikely that this benchmark corresponds to any real program. Hence the benchmark should not be used. To demonstrate the inherent inefficiency in its use of linked data structures, the health program was modified algorithmically to generate the same output, while improving the execution time by over a factor of 200 on a 500Mhz Pentium II Xeon.
- A. Rogers, M. Carlisle, J. Reppy, and L. Hendren. Supporting dynamic data structures on distributed memory machines. ACM Trans. on Programming Languages and Systems, March 1995. Google Scholar
Digital Library
- http:\\www.cs.wisc.edu\~zilles\llubenchmark.html.Google Scholar
Index Terms
- Benchmark health considered harmful
Recommendations
Subsetting the SPEC CPU2006 benchmark suite
On August 24, 2006, the Standard Performance Evaluation Corporation (SPEC) announced CPU2006 -- the next generation of industry-standardized CPU-intensive benchmark suite. The SPEC CPU benchmark suite has become the most frequently used suite for ...
A Benchmark Characterization of the EEMBC Benchmark Suite
Benchmark consumers expect benchmark suites to be complete, accurate, and consistent, and benchmark scores serve as relative measures of performance. However, it is important to understand how benchmarks stress the processors that they aim to test. This ...
Requirements-driven database systems benchmark method
Benchmarks are the vital tools in the performance measurement, evaluation, and comparison of relational database management systems (RDBMS). Standard benchmarks such as the TP1, TPC-A, TPC-B, TPC-C, TPC-D, TPC-H, TPC-R, TPC-W, Wisconsin, and AS3 Ap ...
Comments