18.5. Using the Benchmark

Since the goal of this benchmark is to test individual XML query operations, we do not propose a single benchmark number that can be computed from the individual query execution times. While having a single benchmark number can be very effective in summarizing the performance of an application benchmark, for a non-application-specific benchmark, such as this benchmark, it may be meaningless.

Similarly, it may be useful to run the benchmark queries in both hot and cold modes, corresponding to running the queries using a buffer pool that is warmed up by a previous invocation of the same query and running the query with no previously cached data in the buffer pool, respectively.

In our own use of the benchmark, we have found it useful to produce two tables: a summary table that presents a single number for a group of related queries and a detail table that shows the query execution time for each individual query. For the summary table, we use the groups that are shown in Table 18.2. For each group, we compute the geometric mean of the execution times of the queries in that group. When comparing different systems, or when evaluating the scalability of a system using the benchmark, the summary table quickly identifies the key strengths and weaknesses of the system(s) being evaluated. The detailed table then provides more precise information on the performance of the individual query operations. We expect that this approach of using two tables to summarize the benchmark results will also be useful to other users of this benchmark.