Appendix B. Setting up your cloud

Most of the architecture described in this book is built on top of AWS. This means you need a clear understanding of AWS from the perspectives of security, alerting, and costs. It doesn’t matter whether you use Lambda alone or have a large mix of services. Being able to configure security, knowing how to set up alerts, and controlling cost are important. This appendix is designed so that you can understand these concerns and learn where to look for important information in AWS.

AWS security is a complex subject, but this appendix gives you an overview of the difference between users and roles and shows you how to create policies. This information is needed to configure a system in which services can communicate effectively and securely. Some of the time, you will not need to create or configure policies directly; tools like Serverless framework will do it for you. But it’s still important to understand how the pieces fit together and where to look for help if things go wrong.

Cost is an important consideration when using a platform such as AWS and implementing serverless architecture. It’s essential to understand the cost calculation of the services you’re going to use. This is useful not only for avoiding bill shock but also for predicting next month’s bill and beyond. We look at estimating the cost of services and discuss strategies for tracking costs and keeping them under control. This appendix is not an exhaustive guide to AWS. If you have further questions after reading this appendix, take a look at AWS documentation (https://aws.amazon.com/documentation).

B.1 Security model and identity management

In chapter 2, you created an Identity and Access Management (IAM) user and a number of roles in order to use Lambda, S3, and MediaConvert. In this section, you’ll take your new-found knowledge and develop it further by learning about users, groups, roles, and policies in more detail.

B.1.1 Creating and managing IAM users

As you’ll recall, an IAM user is an entity in AWS that identifies a human user, an application, or a service. A user normally has a set of credentials and permissions that can be used to access resources and services across AWS.

An IAM user typically has a friendly name to help you identify the user and an Amazon Resource Name (ARN) that uniquely identifies it across AWS. Figure B.1 shows a summary page and an ARN for a fictional user named Alfred. You can get to this summary in the AWS console by clicking IAM, clicking Users in the navigation pane, and then clicking the name of the user you want to view.

Figure B.1 The IAM console shows metadata such as the ARN, groups, and creation time for every IAM user in your account.

You can create IAM users to represent human users, applications, or services. IAM users created to work on behalf of an application or a service sometimes are referred to as service accounts. These types of IAM users can access AWS service APIs using an access key. An access key for an IAM user can be generated when the user is initially created, or you can create it later by clicking Users in the IAM console, clicking the required user name, selecting Security Credentials, and then clicking the Create Access Key button.

The two components of an access key are the Access Key ID and the Secret Access Key. The Access Key ID can be shared publicly, but the Secret Access Key must be kept hidden. If the Secret Access Key is revealed, the whole key must be immediately invalidated and recreated. An IAM user can have, at most, two active access keys.

If an IAM user is created for a real person, then that user should be assigned a password. This password allows a human user to log into the AWS console and use services and APIs directly. To create a password for an IAM user, follow these steps:

1. In the IAM console, click Users in the navigation pane.

2. Click the required username to open the user’s settings.

3. Click the Security Credentials tab and then click Manage next to Console password (figure B.2).

   
   

   Figure B.2 IAM users have a number of options including being able to set a password, change access keys, and enable multifactor authentication.

4. In the popup, choose whether to enable or disable console access, type in a new custom password, or let the system autogenerate one. You can also force the user to create a new password at the next sign-in (figure B.3).

Figure B.3 Make sure to create a good password policy with a high degree of complexity if you allow users to log into the AWS console. Password policy can be set up in Account Settings of the IAM console.

After a user is assigned a password, they can log into the AWS console by navigating to https://<Account-ID>.signin.aws.amazon.com/console. To get the account ID, click Support in the upper-right navigation bar, and then click Support Center. The account ID (or account number) is shown at the top of the console. You may want to set up an alias for the account ID also so that your users don’t have to remember it (for more information about aliases, see http://amzn.to/1MgvWvf).

B.1.2. Groups

Groups represent a collection of IAM users. They provide an easy way to specify permissions for multiple users at once. For example, you may want to create a group for developers or testers in your organization or have a group called Lambda to allow all members of that group to execute Lambda functions. Amazon recommends using groups to assign permissions to IAM users rather than defining permissions individually. Any user who joins a group inherits permissions assigned to the group. Similarly, if a user leaves a group, the group’s permissions are removed from the user. Furthermore, groups can contain only users, not other groups or entities such as roles.

B.1.3 Roles

A role is a set of permissions that a user, application, or a service can assume for a period of time. A role is not uniquely coupled to a specific user, nor does it have associated credentials such as passwords or access keys. It’s designed to grant permissions to a user or a service that typically doesn’t have access to the required resource.

Delegation is an important concept associated with roles. Put simply, delegation is concerned with the granting of permissions to a third party to allow access to a particular resource. It involves establishing a trust relationship between a trusting account that owns the resource and a trusted account that contains the users or applications that need to access the resource. Figure B.4 shows a role with a trust relationship established for a service called CloudCheckr.

Figure B.4 This role grants CloudCheckr access to the AWS account to perform analysis of costs and recommend improvements.

Federation is another concept that’s discussed often in the context of roles. Federation is the process of creating a trust relationship between an external identity provider such as Facebook, Google, or an enterprise identity system that supports Security Assertion Markup Language (SAML) 2.0 and AWS. It enables users to log in via one of those external identity providers and assume an IAM role with temporary credentials.

B.1.4 Resources

Permissions in AWS are either identity-based or resource-based. Identity-based permissions specify what an IAM user or a role can do. Resource-based permissions specify what an AWS resource such as an S3 bucket or an SNS topic is allowed to do or who can have access to it. A resource-based policy often specifies who has access to the given resource. This allows trusted users to access the resource without having to assume a role. The AWS user guide at http://mng.bz/VBJP states:

Cross-account access with a resource-based policy has an advantage over a role. With a resource that is accessed through a resource-based policy, the user still works in the trusted account and does not have to give up his or her user permissions in place of the role permissions. In other words, the user continues to have access to resources in the trusted account at the same time as he or she has access to the resource in the trusting account.

Not all AWS services support resource-based policies (the user guide at http://mng.bz/xX8W lists all the services that do).

B.1.5 Permissions and policies

When you initially create an IAM user, it’s not able to access or do anything in your account. You need to grant the user permissions by creating a policy that describes what the user is allowed to do. The same goes for a new group or role. A new group or a role needs to be assigned a policy to have any effect.

The scope of any policy can vary. You can give your user or role administrator access to the whole account or specify individual actions. It’s better to be granular and specify only permissions that are needed to get the job done (least privilege access). Start with a minimum set of permissions and add additional permissions only if necessary.

There are two types of policies: managed and inline. Managed policies apply to users, groups, and roles but not to resources. Managed policies are standalone. Some managed policies are created and maintained by AWS. You also can also create and maintain customer-managed policies. Managed policies are great for reusability and change management. If you use a customer-managed policy and decide to modify it, all changes are automatically applied to all IAM users, roles, and groups that the policy is attached to. Managed policies allow for easier versioning and rollbacks.

Inline policies are created and attached directly to a specific user, group, or role. When an entity is deleted, the inline policies embedded within it are deleted also. Resource-based policies are always inline. To add an inline or a managed policy, click the required user, group, or role and then click the Permissions tab. You can attach, view, or detach a managed policy and similarly create, view, or remove an inline policy.

A policy is specified using JSON notation. The following listing shows a managed AWSLambdaExecute policy.

Listing B.1 AWSLambdaExecute policy

{  
   "Version":"2012-10-17",                  ❶
   "Statement":[                            ❷
      {  
         "Effect":"Allow",
         "Action": "logs:*",
         "Resource":"arn:aws:logs:*:*:*"
      },
      {  
         "Effect":"Allow",                  ❸
         "Action":[                         ❹
            "s3:GetObject",
            "s3:PutObject"
         ],
         "Resource":"arn:aws:s3:::*"        ❺
      }
   ]
}

❶ Version specifies the policy language version; the current version is 2012-10-17. If you’re creating a custom policy, make sure to include the version and set it to 2012-10-17.

❷ Contains one or more statements that specify the actual permissions that make up the policy

❸ The Effect element is required and specifies whether the statement allows or denies access to the resource. The only two available options are Allow and Deny.

❹ Specifies the specific actions on the resource that should be allowed or denied. The use of a wildcard (*) character is allowed (for example, “Action”: “s3:*”).

❺ The Resource element identifies the object or objects that the statement applies to. It can be specific or include a wildcard to refer to multiple entities.

Many IAM policies contain additional elements such as Principal, Sid, and Condition. The Principal element specifies an IAM user, an account, or a service that’s allowed or denied access to a resource. The Principal element isn’t used in policies that are attached to IAM users or groups. Instead, it’s used in roles to specify who can assume the role. It’s also common to resource-based policies. Statement ID (Sid) is required in policies for certain AWS services, such as SNS. A condition allows you to specify rules that dictate when a policy should apply. An example of a condition is presented in the next listing.

Listing B.2 Policy condition

"Condition": {
   "DateLessThan": {                                         ❶
               "aws:CurrentTime": "2020-09-12T12:00:00Z"
        },
        "IpAddress": {
               "aws:SourceIp": "127.0.0.1"                   ❷
        }
   }    

❶ You can use a number of conditional elements, which include DateEquals, DateLessThan, DateMoreThan, StringEquals, StringLike, StringNotEquals, and ArnEquals.

❷ The condition keys represent values that come from the request issued by a user. Possible keys include SourceIp, CurrentTime, Referer, SourceArn, userid, and username. The value can be either a specific literal value such as “127.0.0.1” or a policy variable.

Amazon recommends using conditions to the extent that is practical for security. The next listing, for example, shows an S3 bucket policy that forces content to be served only over HTTPS/SSL. This policy refuses connections over unencrypted HTTP.

Listing B.3 Policy to enforce HTTPS/SSL

{
    "Version": "2012-10-17",
    "Id": "123",
    "Statement": [
        {
            "Effect": "Deny",                        ❶
            "Principal": "*",                        ❶
            "Action": "s3:*",                        ❶
            "Resource": "arn:aws:s3:::my-bucket/*",
            "Condition": {
                "Bool": {
                    "aws:SecureTransport": false     ❷
                }
            }
        }
    ]
}

❶ Explicitly denies access to s3 if the condition is met

❷ The condition is met when requests are not sent using SSL. This forces the policy to block access to the bucket if a user tries to access it over regular, unencrypted HTTP.

B.2 Cost

Receiving an unpleasant surprise in the form of a large bill at the end of the month is disappointing and stressful. Amazon CloudWatch can create billing alarms that send notifications if total charges for the month exceed a predefined threshold. This is useful not only to avoid unexpectedly large bills but also to catch potential misconfigurations of your system.

For example, it’s easy to misconfigure a Lambda function and inadvertently allocate 3.0 GB of RAM to it. The function might not do anything useful except wait for 15 s to receive a response from a database. In a heavy-duty environment, the system might perform 2 M invocations of the function a month, costing a little over $1,462. The same function with 128 MB of RAM would cost around $56 per month. If you perform cost calculations up front and have a sensible billing alarm, you’ll quickly realize that something is going on when billing alerts begin to come through.

B.2.1 Creating billing alerts

Follow these steps to create a billing alert:

1. In the main AWS console, click your name (or the name of the IAM user that’s representing you) and then click My Billing Dashboard.

2. Click Billing Preferences in the navigation pane and then enable the check box next to Receive Billing Alerts.

3. Click Save preferences, then go back to the main AWS console and find the CloudWatch service.

4. Open the CloudWatch service, click Alarms, and select All Alarms in the navigation pane. Click the Create alarm button and then click the Select metric button.

5. Under the Metrics heading, select Billing and click Total Estimated Charges. (If you don’t see Billing it means you may not have enabled the Receive Billing Alerts option in step 2).

6. Tick the checkbox for EstimatedCharges and click Select metric to continue.

7. Make sure that the Threshold type is set to Static and that Whenever EstimatedCharges is set to Greater.

8. In the Define the threshold value, enter the amount that you’d like to trigger the alarm (for example, 200 as seen in figure B.5).

9. Click Next to continue to the next page.

   Here you can set or create a new SNS topic to notify you when the alarm is triggered. This is important! You need an SNS topic to receive emails to alert you what is happening.

10. Click the Add notification button.

11. Choose Create new topic, enter a name for it, and then type in your email address. Click Create topic button to save your SNS topic settings. When you are ready to proceed click Next.

12. Type in a name for your Alarm and click Next again.

13. Finally, at the bottom, click the Create alarm button to finish.

Figure B.5 It’s good practice to create multiple billing alarms to keep you informed of ongoing costs.

B.2.2 Monitoring and optimizing costs

Services such as CloudCheckr (http://cloudcheckr.com) can help to track costs, send alerts, and even suggest savings by analyzing services and resources in use. CloudCheckr comprises several different AWS services including S3, CloudSearch, SES, SNS, and DynamoDB. It’s richer in features and easier to use than some of the standard AWS features. It’s worth considering for its recommendations and daily notifications.

AWS also has a service called Trusted Advisor that suggests improvements to performance, fault tolerance, security, and cost optimization. Unfortunately, the free version of Trusted Advisor is limited, so if you want to explore all of the features and recommendations it has to offer, you must upgrade to a paid monthly plan or access it through an AWS enterprise account.

Cost Explorer (figure B.6) is a useful, albeit high-level reporting and analytics tool built into AWS. You must activate it first by clicking your name (or the IAM username) in the top-right corner of the AWS console, selecting My Billing Dashboard, then clicking Cost Explorer from the navigation pane and enabling it. Cost Explorer analyzes your costs for the current month and the past four months. It then creates a forecast for the next three months. Initially, you may not see any information because it takes 24 hours for AWS to process data for the current month. Processing data for previous months make take even longer. More information about Cost Explorer is available at http://amzn.to/1KvN0g2.

Figure B.6 Cost Explorer allows you to review historical costs and estimate what future costs may be.

B.2.3 Using the Simple Monthly Calculator

The AWS Pricing Calculator (https://calculator.aws) is a web application developed by Amazon to help model costs for many of its services. This tool allows you to select a service, enter information related to the consumption of that particular resource, and get an indicative cost.

B.2.4 Calculating Lambda and API Gateway costs

The cost of running serverless architecture often can be a lot less than running traditional infrastructure. Naturally, the cost of each service you might use will be different, but you can look at what it takes to run a serverless system with Lambda and the API Gateway.

Amazon’s pricing for Lambda (https://aws.amazon.com/lambda/pricing/) is based on the number of requests, duration of execution, and the amount of memory allocated to the function. The first million requests are free with each subsequent million charged at $0.20. Duration is based on how long the function takes to execute measured to the millisecond (ms). Amazon charges in 1 ms increments, while also factoring in the amount of memory reserved for the function. A function created with 1 GB of memory will cost $0.000001667 per 100 ms of execution time, whereas a function created with 128 MB of memory will cost $0.000000208 per 100 ms.

Note Amazon prices may differ depending on the region and that they’re subject to change at any time.

Amazon provides a perpetual free tier with 1M free requests and 400,000 GB-seconds of compute time per month. This means that a user can perform a million requests and spend an equivalent of 400,000 seconds running a function created with 1 GB of memory before they have to pay. As an example, consider a scenario where you have to run a 256 MB function, 5 million times a month. The function executes for 2 seconds each time. The cost calculation follows:

• Monthly request charge:

  • The free tier provides 1 million requests, which means that there are only 4 million billable requests (5M requests - 1M free requests = 4M requests).

  • Each million is priced at $0.20, which makes the request charge $0.80 (4M requests × $0.2/M = $0.80).

• Monthly compute charge:

  • The compute price for a function per GB-second is $0.00001667. The free tier provides 400,000 GB-seconds free.

  • In the compute price scenario, the function runs for 10 ms (5M × 2s).

  • 10M seconds at 256 MB of memory equates to 2,500,000 GB-seconds (10,000,000 × 256 MB / 1024 = 2,500,000).

  • The total billable amount of GB-seconds for the month is 2,100,000 (2,500,000 GB-seconds - 400,000 free tier GB-seconds = 2,100,000). The compute charge is therefore $35.007 (2,100,000 GB-seconds × $0.00001667 = $35.007).

  • The total cost of running Lambda in this example is $35.807.

The API Gateway pricing is based on the number of API calls received and the amount of data transferred out of AWS. In the eastern United States, Amazon charges $3.50 for each million API calls received and $0.09/GB for the first 10 TB transferred out. Given the previous example and assuming that monthly outbound data transfer is 100 GB a month, the API Gateway pricing is as follows:

• Monthly API charge:

  • The free tier includes 1M API calls per month but is valid for only 12 months. Given that it’s not a perpetual free tier, it won’t be included in this calculation.

  • The total API cost is $17.50 (5M requests × $3.50/M = $17.50).

• The monthly data charge is $9.00 (100 GB × $0.09/GB = $9).

• The API Gateway cost in this example is $26.50.

• The total cost of Lambda and the API Gateway is $62.307 per month.

It’s worthwhile to attempt to model how many requests and operations you may have to handle on an ongoing basis. If you expect 2M invocations of a Lambda function that uses only 128 MB of memory and runs for 1 second, you’ll pay approximately $0.20 month. If you expect 2M invocations of a function with 512 MB of RAM that runs for 5 seconds, you’ll pay a little more than $75.00. With Lambda, you have an opportunity to assess costs, plan ahead, and pay for only what you actually use. Finally, don’t forget to factor in other services such as S3 or SNS, no matter how insignificant their cost may seem to be.