As the number and variety of cloud service providers continue to expand, consumers face a growing information disparity concerning the level of service they should expect. This challenge is especially perplexing when the consumer tackles the complexity of selecting cloud services and service configurations that best meet the price and performance requirements of applications selected for cloud deployment.

For a cloud solution architect, this challenge is managed by comparing prospective CSP service levels and capabilities against industry benchmarks for those services. Challenges to being able to establish useful cloud computing industry benchmarks include the following:

• The sheer number of cloud service providers and the variety of cloud services in the market
• The broad geographic expanse of CSP platforms that typically span many different locations
• Geopolitical requirements and restrictions
• Wide area networking performance
• Variety of CSP business, pricing, and service models
• Multiplicity of service price changes
• Variability of performance within the same service at different times and from different locations

Additionally, services can be consumed by the hour, month, annually, or through a spot market. New products are introduced on an almost daily basis, and pricing changes weekly. Amazon, for instance, is known to make price changes monthly. One of the best industry benchmark studies was a collaboration between Rice University and Burstorm Inc., the result of which was the industry's first comprehensive and continuous price-performance benchmark:

Using a highly automated process, the first benchmark was across seven suppliers (Amazon, Rackspace, Google, Microsoft, HP, IBM, and Linode), across three continents (North America, Asia, and Europe) , as shown in the preceding tables, with a total of 266 compute products spread over three locations per vendor. The benchmark was executed every day, for 15 days. The results were normalized to a 720-hour, monthly pricing model to establish the price-performance metrics. These results showed the following:

• The range of performance within a single provider can vary widely:

• The diversity of platforms and solutions offered by different CSPs can result in a 1-core instance performance variance of as much as 622%:

• There is a 4-core compute price performance variation of 1,000%:

• There can be instance performance fluctuations of as much as 60% over time:

• There is a wide variance in the availability and performance of instance types when measured at different locations.

Rapid changes over time in instance types, pricing, performance, and availability of services by location demonstrates that benchmarking of a small set of instance types in a unique event is not sufficient for cloud computing. Even within the short span of this study, Google updated their infrastructure and pricing: