The EC2 instances you choose are directly dependent on your application characteristics. Ensure that you have a good understanding of these characteristics, for example, is the application CPU-intensive, or is it more IO bound? What are the typical workloads? Is there variability in demand over a period of time? Are there any special events when the demand is unusually high? What are the average resource requirements for good user experience?
Based on the application characteristics shortlist a few instance types available from AWS. EC2 types include several families of related instances available in sizes ranging from micro to extra large. These classes include general purpose, compute-optimized, memory-optimized, storage-optimized, and accelerated computing instances.
You should then do a few tests to analyze the performance of the shortlisted instances against increasing loads. It is a good idea to understand the upper limit of these instances in terms of number of concurrent users or throughput they can support.
For example, let's assume you want to select EC2 instances for your web servers. These web servers proxy API calls to the application servers, that is, handle CPU-intensive traffic and support heavy payloads. Based on these requirements, let's say that you shortlist two instance types – one a CPU-optimized (say, c3.xlarge) and a general purpose (m3.xlarge) instance type. Typically, you should choose a general purpose and a special purpose instance types for comparison purposes. In order to conduct the performance analysis, create a set of test cases to simulate the expected scenarios in your application. Monitor the CPU utilization for these instances at different loads (say 1,000, 2,000, and 3,000 users). Increase the load to a point where you max out on the CPU. It is very likely that you will hit max CPU utilization at different loads for each of the chosen instances.
In cases where you want to test at very high loads, you should contact Amazon before conducting the load test to have your load balancer "pre-warmed". They can configure the load balancer to have the appropriate level of capacity based on the expected traffic during your load tests. It is also important to load test from multiple distributed agents to ensure your IP address is not flagged. At this stage, you should provision multiple smaller instances (from the same families) that match the xlarge instance’s compute power and conduct the same load tests. This is done to check whether we can achieve the same performance, at a higher level of resiliency, using multiple smaller instances in place of a bigger instance.
Instance selection is not only about the instance size or type alone but also about the available network bandwidth. Hence, you should also compare the results of your network bandwidth assessment for each of your instance types. If your application requires increased network bandwidth turn on the enhanced networking option, which is available on certain instance types, for example, enhanced networking option is available on compute-optimized C3, C4, D2, I2, and R3 instances, but not on general purpose instances.
Compare the costs against different throughput levels. It is possible that the general purpose instance type costs more than the compute optimized instance type for your application’s expected workload.
Availability and costs of instance types differ by region. Hence, your EC2 instance type and size decision will need to take into consideration the availability of instance types and then strike the right balance in terms of performance, resiliency, and costs.
Typically, in start-ups, the development and test environments are provisioned in the most economical region using minimum sizes of general purpose EC2 instances to minimize development infrastructure costs.
Finally, revisit and rightsize your EC2 instances choices every 3 to 6 months as instance families and workloads can change. New instance types that are more appropriate for your workloads can reduce your overall costs. You can use the Amazon Trusted Advisor for cost optimization hints for low network/CPU utilization, unused or low utilization instances, and so on. For example, if you observe a 15-20% CPU maximum utilization on an instance, then it is a trivial change to lower instance. If this step leads to an improvement in CPU utilization to 60-70%, then you would saved on costs with sufficient headroom for growth and/or auto scaling.
Choosing the right distribution of instances across AZs can also make a significant difference to your overall costs while meeting your availability requirements. For example, if 12 instances are required in order to maintain HA, then instead of splitting 24 instances across two AZs, we split them across three AZs containing 6 instances each so that the required 12 are available even when an AZ is down. This might sound a little obvious but make yourself well aware of various facilities offered by AWS to drive additional savings.