As discussed in Chapter 3, “Requirements and challenges” on page 13, the reading at least 1 million records per second was expected to satisfy the minimum requirements of the application. This chapter describes the initial testing that benchmarked the I/O rates and determined the feasibility of the objective.

To provide the capabilities that the application required, the (extended pointer set) EPS search service followed this process:

This section describes the components that were used to perform these preliminary functions.

In this approach, the search criteria is sent from the client to the EPS service through the distributed cloud application that is described in 2.3.2, “Walmart Event Processing System (EPS)” on page 10. The primary means of communication for this approach is Hypertext Transfer Protocol (HTTP). A pre-existing custom format (developed by Walmart as part of an earlier service) was used to accommodate these queries. In this format, you pass a specialized SQL-like (Structured Query Language) query string on HTTP GET requests by prefixing the string with the "zQL" (zServices Query Language) keyword.

As described in 3.5, “Data Structure” on page 14, the searchable events are canonical objects (typically JSON) that are converted into a flat columnar format for storage, as you see in Figure 4-1.

The field or column names can be referenced in query strings in the zQL format via HTTP GET requests. The example in 3.5.1, “Event object” on page 14 describes a single event that is related to vehicle number 2e7d0f3f00d5acdc497b358779fa1b6f (identified by the producerID value) of the U.S. Transportation fleet (derived from the eventKey value). Suppose that you needed to satisfy the following related request:

This request in the zQL format would resemble this query string:

CICS gets this request through the WEB RECEIVE and WEB EXTRACT commands, then parses the request to determine the search criteria. The specific syntax of these query strings is not relevant for this document. In general, the strings confirm these facts:

Our initial test relied on basic sequential processing. Upon receiving the search criteria, the EPS service read the data set in search of the requested events. 3.5, “Data Structure” on page 14 and 3.6, “Fundamental I/O Requirements” on page 16 explained how the key in a KSDS VSAM/RLS data store gives direct access to the appropriate subset of data that must be processed. Using the search example from 4.1.1, “Search Criteria” on page 19, the following parameters (all contained within the eventKey field) can be used as an example:

With this information, the EPS search process navigates directly to the first relevant entry in the data store, as illustrated in Figure 4-2.

The EPS search process uses the EXEC CICS STARTBR command to begin browsing the records at this starting point. It examines this record and looks for other matching search criteria in the payload portion of the record. The process might find, for example, the producerID of 2e7d0f3f00d5acdc497b358779fa1b6f). For any matches, it extracts the requested field values from the record and stores the result in memory.

It then issues an EXEC CICS READNEXT command to read the next record in the file and process it accordingly. This action is repeated sequentially, record by record. The command stops when it finds a record that contains a data/timestamp that is greater than the End date/time Range value that was provided in the search request. At this point, the service gathers the matching results from memory and returns them to the client.

This approach yielded read operations of approximately 5,000 records per second. This is obviously, and unsurprisingly, well below the minimum required throughput. The inability to achieve the desired amount of reads with strictly sequential processing was anticipated, but this testing was necessary. It provided an initial baseline to be used in subsequent projections and testing.

Establishing this benchmark was useful, but it did prompt questions of how this single-threaded throughput might be increased. Although there was no expectation that any serial process would satisfy the needs of the application, this question leads to additional testing with less common mechanics.

Additional testing was performed by using native VSAM/RLS operations, as opposed to the CICS browse APIs. This was done with the understanding that while low-level operations can improve efficiency by eliminating overhead, you forfeit valuable features and capabilities of the APIs. This section summarizes the results of this testing and some of the considerations that are given to this approach.

The native VSAM/RLS operations were performed while running as THREADSAFE on an open task control block (TCB). With this method, we were able to process approximately 60,000 records per second. As expected, this is still not enough to satisfy the established minimum requirement, although it is substantially higher than the previous test. This benchmark proved to be useful for Walmart's use-case, but might not be meaningful in less extreme scenarios.

As mentioned in the opening statements of this section, taking this approach comes with costs. For example, ease of use is compromised to some degree. Of particular note is the loss of debugging capabilities in CICS APIs. In Walmart's case, feature flags for I/O types were incorporated into the programs. These flags allowed dynamic switching between native VSAM/RLS and EXEC CICS file control commands. These commands provide the many diagnostic capabilities of CICS APIs, such as CEDF and tracing. While this is helpful for some development activities, there are still compromises when you bypass the standard APIs.

The intricacies of doing native VSAM/RLS I/O processing are beyond the scope of this publication. In fact, it is likely unnecessary in most processing scenarios. The Walmart use case that is discussed here is so demanding that it warrants exceptional measures. Consequently, the metric of 60,000 records per second serves as a reference point throughout the assessment in this document. Yet, this specific metric is not important to the overall context that this publication describes. The relevance of concepts related to asynchronous processing extend beyond the particular details in Walmart's situation.

Our initial tests established that sequential processing was not sufficient to achieve the necessary I/O rates for this use-case. However, the tests did provide valuable benchmarks for use in further iterations of a solution. The sequential read rates that we achieved were especially useful:

These sequential processing metrics might be adequate in some other less-demanding situations. Accommodations for these situations are briefly covered in Chapter 8. However, the need to satisfy the particular requirements of this application still existed, and as the saying goes, necessity is the mother of invention. Chapter 5, “Homegrown asynchronous solution” on page 23 explores the inventive homegrown solution that was developed to meet these objectives.