Execution Role

Now consider a case in which a Lambda function interacts with an S3 bucket. For the Lambda function to be able to access the bucket, you need to create a policy and attach it to a Lambda execution role. This role is then specified in the configuration of the Lambda function and allows the code to access the S3 bucket.

In the policy in Example 5-3, you simply allow all actions (s3:*) over the S3 bucket everyonelovesaws and all objects in the bucket denoted as everyonelovesaws/*.

Example 5-3 IAM Policy That Allows Access to S3 and the Logs Required by Lambda

Click here to view code image

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": "s3:*",
           "Resource": [
               "arn:aws:s3:::everyonelovesaws",
               "arn:aws:s3::: everyonelovesaws/*"
           ]
       },
       {
           "Effect": "Allow",
           "Action": [
               "logs:CreateLogGroup",
               "logs:CreateLogStream",
               "logs:PutLogEvents"
           ],
           "Resource": "arn:aws:logs:*:*:*"
       }
   ]
}

On top of the S3 access, the role also requires additional permissions to grant access to logs:CreateLogGroup, logs:CreateLogStream, and logs:PutLogEvents over arn:aws:logs:*:*:*, which allows the Lambda function to write the log output to CloudWatch logs. Other than the logs part, this is a completely standard IAM policy that you could apply to any user, group, or role, meaning that if you already have a user that has been given a certain level of access to test your code or if you are already using an EC2 instance with a role attached, you can simply copy your existing policy and add the logs section to create a new policy that you can use with the Lambda execution role. AWS also provides quite a number of different managed policies that you can use with your Lambda execution role or base your custom Lambda policy on.

Identity-Based IAM Policy

To grant a user, group, or role the permission to work with a Lambda function, you can simply apply a policy that allows the required level of access. AWS provides three different prebuilt identity-based policies that allow three different levels of access:

• AWSLambdaFullAccess: Allows a user, group, or role full control over creation, management, and invocation of Lambda functions.

• AWSLambdaReadOnlyAccess: Allows a user, group, or role read-only access to all the existing Lambda functions in the account.

• AWSLambdaRole: Allows a user, group, or role the ability to invoke a Lambda function.

Example 5-4 shows the AWSLambdaRole role. You can see that the policy is quite simple, allowing only the action lambda:InvokeFunction on all resources.

Example 5-4 The Lambda Function Invocation Permission IAM Document

Click here to view code image

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Effect": "Allow",
           "Action": [
               "lambda:InvokeFunction"
           ],
           "Resource": [
               "*"
           ]
       }
   ]
}

You can use this policy as a template for allowing an invocation of a specific function by replacing the resource section with your function ARN. For example, if you would like to allow access only to a function called test in the Ohio (us-east-2) region in account 111222333444, you can use the following ARN:

Click here to view code image

"Resource": "arn:aws:lambda:us-east-2:111222333444:function:test"

You can also look up the ARN of a particular function in a particular region by running the following CLI command:

Click here to view code image

aws lambda get-function --function-name test --region us-east-2

A Resource Policy

To allow an AWS service to invoke your Lambda function, you can either use the Lambda designer in the AWS Management Console or the AWS CLI to assign a permission for a certain AWS service to your Lambda function. Example 5-5 allows the S3 bucket everyonelovesaws from the AWS account number 111222333444 to invoke the test function.

Example 5-5 CLI Script to Add the Lambda Permission to the S3 Bucket

Click here to view code image

aws lambda add-permission --function-name test 
--action lambda:InvokeFunction 
--statement-id s3-account 
--principal s3.amazonaws.com 
--source-arn arn:aws:s3:::everyonelovesaws 
--source-account 111222333444

The --source-account option is not required, but it makes good sense to use it because S3 bucket names are global. If you didn’t use the --source-account option and you deleted the everyonelovesaws bucket, another account would be able to create that bucket, and the resource policy would allow the other account’s bucket to invoke your Lambda function. It is thus always recommended to use the --source-account option as it protects your account from undesired actions.

Workflows

An SWF workflow represents your business process. The workflow contains activities and tasks as well as all required orchestration and data transition logic. In the SWF, you can easily define the sequence, circumstances, and actions that your business process has and assign an AWS service to each task.

Each workflow also contains deciders that perform tasks when a certain condition is met and that are able to pass data to the tasks and schedule execution. At the end of the workflow, a final decider is implemented to close the workflow when completed or upon failure.

The workflow also contains the execution history, which enables you to take a look at the existing state and previous states of the workflow execution and determine any anomalies. The execution history is completely authoritative and is generated during the progress of each workflow execution.

Activity and Decider Tasks

A workflow contains the activity task, which defines the actions (activities) that need to be completed, as well as the decision task, which interacts with the decider and informs it of the task completion. To define an activity task, you need to define a list of all the actions (task list), name the activity task, and give it a version. You also need to define the workflow domain it belongs to, an optional schedule to run on, and the timeout for the task. Each activity task is given a unique ID that can help you track the progress of the task in the workflow.

Actors

Actions being performed within a workflow are always executed by actors. An actor or an activity worker needs to be some kind of entity that performs the task, either an AWS service, a software component for data processing, or a person who will execute a manual task.

Domains

Workflow domains are used to logically group workflows that belong to a certain application. An application can have multiple workflows that have the ability to interact and send data and invoke tasks on each other. There is no interaction between workflows allowed across domains.

Visibility Timeout

With standard queues, the delivery is guaranteed with an at-least-once delivery approach. When a message is picked up from the queue, the message is retained in the queue but made invisible to other recipients. This enables the message to be processed in a certain amount of time but still be available in case the recipient processing the message suffers a failure. Once the message is processed, the recipient needs to remove the message from the queue, which prevents it from being read again.

This approach helps maintain guaranteed message delivery and processing, but it also means that a message might be delivered and processed multiple times. The structure of the data and the processing logic must be designed so that the message processing output is idempotent, which prevents multiple duplicate results from being created in the processing output.

For example, when processing a message from SQS, try to refrain from adding a processing timestamp to the output data. If you had a timestamp, you could have duplicate entries. Suppose you have a message that has been successfully processed, but it became visible before the recipient deleted it. The message is inadvertently picked up by another recipient and processed, and then the same output with a different processing timestamp is generated and stored on the output data store. These two entries would not have been created as duplicates and would have been idempotent if the processing timestamp were not included in the output data.

The time for which a message becomes invisible is controlled through the visibility timeout. For each standard queue, you define a default visibility timeout. This visibility timeout should be long enough to allow your recipients to process the message but short enough to allow the message to be reread in a timely manner if the processing on the initial reader fails. A reader can also set a custom visibility timeout at the time when it picks up the message or at any other time during which the message is still invisible. This means that you can override the default visibility timeout at any time, even after processing has already started. Figure 5-11 illustrates the process of retrieving a message from the SQS queue.

When a message is read from the SQS queue, you receive a receipt handle, which is the unique identifier that has set in motion the visibility timeout clock and with which you can address the message in the queue and extend the visibility timeout or delete the message when the processing is completed.

The next example shows how to create an SQS queue and post a message with the AWS CLI. You then see how to retrieve the message and delete it from the queue. First, to create a queue.attributes file, you add the following line, which sets the visibility timeout to 60 seconds:

VisibilityTimeout=60

Next, run the following command to create the queue named everyonelovesaws:

Click here to view code image

aws sqs create-queue --queue-name everyonelovesaws 

--attributes file://queue.attributes

Next, you describe the newly created queue to get the queue URL and see if the visibility timeout attribute worked. You run the following command to get the queue URL that you will use in all the following commands:

Click here to view code image

aws sqs get-queue-url --queue-name everyonelovesaws

The response should look like this:

Click here to view code image

{

     "QueueUrl": "https://us-east-2.queue.amazonaws.com/
111222333444/everyonelovesaws"
}

To get the visibility timeout, run the following command:

Click here to view code image

aws sqs get-queue-attributes 

--attribute-names VisibilityTimeout --queue-url 

https://us-east-2.queue.amazonaws.com/111222333444/everyonelovesaws

This should result in a response similar to the following:

Click here to view code image

{

     "Attributes": {
         "VisibilityTimeout": "60"
     }
}

To get all the attributes of the queue, you can simply replace --attribute-names VisibilityTimeout with --attribute-names All.

Now send a message to the queue, modifying the message body to your liking:

Click here to view code image

aws sqs send-message --queue-url 

https://s-east-2.queue.amazonaws.com/111222333444/
everyonelovesaws 

--message-body "My first message"

You should receive a message ID as the response:

Click here to view code image

{

     "MD5OfMessageBody": "77a8f18b5388a3865556cc642424151c",

     "MessageId": "d489d625-e649-426f-9e21-dc432a06e778"

}

Now you can use this command to receive a message:

Click here to view code image

aws sqs receive-message --queue-url 

https://s-east-2.queue.amazonaws.com/111222333444/everyonelovesaws

The output of the command should provide the message, its content, and the receipt handle that you can use to manipulate this exact message during its visibility timeout, as demonstrated in Example 5-7.

Example 5-7 Response from the receive-message Command That Shows the Receipt Handle

Click here to view code image

{
     "Messages": [
         {

             "Body": "My first message",

             "ReceiptHandle": "AQEBa+DIuyBk4RpEBmyOKRmbDHNnRZeOdU8kWf0j4KTYqqr6wWpz9v
tTABppeku8O4TCOX0cKVMSRU7EW6i2tJAJdr8mvV9B7/VPx8cizuR0oa9wAL1jvoObp8bLC9PlQYWc6nDXV
J3orXelekeFbZBtAuK7csOsCLIIDFEQ9U2PHF0kBCc/i/tscrcFY+4U7Swyclg/nkugePklxumlyBj66hE
AxatdqJDXYhcLiAu0PwP5p5vHElRB7hsEr+gG9QGyXgy5fHEOl89xPgP2ki/f7wstQS1dGwDA4ceqy9
qmfFJkq5/6W0AK4/paQQ56SyQ/w+p/fOpND6M6meEloQCNFZCmORenAUcgYhnlaV4bk/emoC9s4vXG/
xIQwEPg5fIkbLeo/OXF71weYWQrIxLW/Q==",

                "MD5OfBody": "77a8f18b5388a3865556cc642424151c",

                "MessageId": "d489d625-e649-426f-9e21-dc432a06e778"
        }

    ]

}

Now that you have received a message, you can run the following command to see if the message became invisible:

Click here to view code image

aws sqs get-queue-attributes 

--attribute-names ApproximateNumberOfMessagesNotVisible
--queue-url  https://us-east-2.queue.amazonaws.com/111222333444/
everyonelovesaws

The output should look like this:

Click here to view code image

{

     "Attributes": {
         "ApproximateNumberOfMessagesNotVisible": "1"
     }
}

Now you can simulate processing the message and deleting it from the queue. You will need to use the receipt handle from the receive-message command like so:

Click here to view code image

aws sqs delete-message --queue-url 

https://s-east-2.queue.amazonaws.com/111222333444/
everyonelovesaws 

--receipt-handle 

"AQEBa+DIuyBk4RpEBmyOKRmbDHNnRZeOdU8kWf0j4KTYqqr6wWpz9vtTABppeku8
O4TCOX0cKVMSRU7EW6i2tJAJdr8mvV9B7/VPx8cizuR0oa9wAL1jvoObp8bLC9
PlQYWc6nDXVJ3orXelekeFbZBtAuK7csOsCLIIDFEQ9U2PHF0kBCc/i/tscrcFY+
4U7Swyclg/nkugePklxumlyBj66hEAxatdqJDXYhcLiAu0PwP5p5vHElRB7hsEr+
gG9QGyXgy5fHEOl89xPgP2ki/f7wstQS1dGwDA4ceqy9qmfFJkq5/6W0AK4/
paQQ56SyQ/w+p/fOpND6M6meEloQCNFZCmORenAUcgYhnlaV4bk/emoC9s4vXG/
xIQwEPg5fIkbLeo/OXF71weYWQrIxLW/Q=="

The message with the matching receipt handle is deleted, and you can check the number of messages in the queue by looking at the ApproximateNumberOfMessages attribute:

Click here to view code image

aws sqs get-queue-attributes 

--attribute-names ApproximateNumberOfMessages --queue-url 

https://us-east-2.queue.amazonaws.com/111222333444/everyonelovesaws

The output should look like this:

Click here to view code image

{
     "Attributes": {
         "ApproximateNumberOfMessages": "0"
     }
}

You can see that the SQS service is really simple to use. Experiment a bit with the queue to see what happens when you add many messages and try to read from many systems at once. You can simulate many systems by simply opening several consoles in your Linux environment and run multiple AWS CLI commands in parallel.

Dead Letter Queues

When working with SQS queues, an important concept is the dead letter queue. A dead letter queue is intended for storing any messages that cannot be processed due to the content of the messages and not due to a failure of receivers. For example, say that a recipient receives the message, the message cannot be processed, and if the dead letter queue is not implemented, the message becomes visible again. Another recipient can read the message but cannot process it, and the message becomes visible again. As you can imagine, the queue would quickly fill up to the maximum of approximately 120,000 messages that can exist in a queue at the same time.

To avoid this situation, you need to implement some kind of logic into your processing engine to identify messages that error out due to the content of the message. The processor needs to delete the message from your production queue and post the same message into the dead letter queue. Once a message hits a dead letter queue, you can simply create a CloudWatch alarm for a non-empty queue. The alarm can in turn trigger an SNS notification to an incident response team so that it can take a look at the message and figure out why it was not able to be processed.

Topics and Subscriptions

To send messages, you need to create a topic. Each topic has a unique URL created, similarly to the SQS URL used in the previous example of sending and receiving a message with SQS. Subscribers can subscribe themselves, or they can be added by a user or an application (through a role) with the permissions to manage the SNS topic. Once a topic is created and subscribers are subscribed, a publisher, represented by any service or user with permissions to publish to the SNS topic, can create a message.

The maximum size of each message sent to the SNS topic is 256 KB, which includes the message body, any metadata attributes, and the JSON formatting of the message. The message is sent out in a push-based fashion and cannot be recalled once posted due to the distribution mechanisms in place. For example, SMTP and SMS do not have a way to recall a message once it has been delivered to the recipient’s server.

The subscribers have ability to specify a filter messages based on a metadata attribute. When a client subscribes, the filtering allows the client to listen to only messages that it wants to receive. This can be very useful when subscribing different automated systems to an SNS topic so that each part of the system only receives the correct type of notification.

For example a fan-out queue design can be created, where SNS sends a message to multiple SQS queues. You can either deliver all the messages to all the queues or selectively filter the messages according to a certain attribute value. This can be very useful if you want to ensure that one or more queues receive only a certain subnet of messages.

Working with SNS Topics

Let’s look at an example of how to create an SNS topic and subscribe the SQS queue from Chapter 4, “Storing Data in AWS,” to the SNS topic and then publish a message. You can create an email or SMS subscription to your email address or phone number to test different protocols and see the differences in operation.

First, you can create a new SNS topic:

aws sns create-topic --name sqstopic

The output should look like this:

Click here to view code image

{
     "TopicArn": "arn:aws:sns:us-east-2:111222333444:sqstopic"
}

Record the TopicArn value as you will need it to grant permission in the SQS policy and to address the SNS topic.

Next, you can get the ARN for the SQS queue that you created before:

Click here to view code image

aws sqs get-queue-attributes --attribute-names QueueArn
--queue-url 

https://us-east-2.queue.amazonaws.com/111222333444/everyonelovesaws

The output should look like this:

Click here to view code image

{
     "Attributes": {
         "QueueArn": "arn:aws:sqs:us-east-2:111222333444:everyonelo
vesaws"
     }
}

Now you can use both ARNs to edit the following policy and replace arn:aws:sqs:us-east-2: 111222333444:everyonelovesaws with your SQS ARN and arn:aws:sns:us-east-2: 111222333444:sqstopic with your SNS ARN:

Click here to view code image

{ "Policy" : "{"Statement": [{ "Effect": "Allow",
"Principal": {"AWS": "*"},"Action": "SQS:SendMessage
","Resource": "arn:aws:sqs:us-east-2:111222333444:everyone
lovesaws","Condition": {"ArnEquals": {"aws:SourceArn": "
arn:aws:sns:us-east-2:111222333444:sqstopic"}}}]}"}

Save the policy to the sqspolicy.json file. This policy statement, which needs to be formatted exactly, allows the SNS topic to send messages to the SQS queue and will occupy a single line in the sqspolicy.json file.

Next, you apply this policy to the SQS queue:

Click here to view code image

aws sqs set-queue-attributes --queue-url 

https://s-east-2.queue.amazonaws.com/111222333444/
everyonelovesaws 

--attributes file://sqspolicy.json

Now you are ready to subscribe the queue to the SNS topic. If you want to subscribe your email instead, you can choose email as the protocol and your email address as the notification endpoint. The same goes for SMS:

Click here to view code image

aws sns subscribe --topic-arn 

arn:aws:sns:us-east-2:111222333444:sqstopic 

--protocol sqs --notification-endpoint 

arn:aws:sqs:us-east-2:111222333444:everyonelovesaws

The result, which should look like this, indicates that the subscription has been created:

Click here to view code image

{

     "SubscriptionArn": "arn:aws:sns:us-east-2:111222333444:
sqstopic:49fea96b-87c5-4720-9d71-a8dce33dc08c"
}

An email or SMS subscription needs to be confirmed, but an SQS subscription is created without confirmation.

Now all that you need to do is publish a message:

Click here to view code image

aws sns publish --topic-arn arn:aws:sns:us-east-2:111222333444:
sqstopic 

--message "My first SNS message"

After publishing, you should receive a message ID:

Click here to view code image

{
     "MessageId": "3c8bf044-c7fa-52c4-8aeb-a5ac0c805ab4"
}

You can now check whether the message has been delivered to the SQS queue by checking the number of messages.

Click here to view code image

aws sqs get-queue-attributes --attribute-names 

ApproximateNumberOfMessages --queue-url 

https://us-east-2.queue.amazonaws.com/111222333444/everyonelovesaws

If the number is still 0, try sending another message with the previous command or even sending a few messages. Even if the subscription is confirmed, it might take a minute for everything to go into effect. SNS does not retry the message delivery to SQS; it is essentially “fire and forget.”

The output should now show a message in the queue:

Click here to view code image

{

     "Attributes": {
          "ApproximateNumberOfMessages": "1"
     }

}

Now you can receive the message from the SQS queue:

Click here to view code image

aws sqs receive-message --queue-url 

https://us-east-2.queue.amazonaws.com/111222333444/everyonelovesaws

The content should look much different than before since the SMS message is JSON formatted and includes an SNS signature and an unsubscribe URL, as demonstrated in Example 5-8. This can be used to cancel your subscription if you subscribed via SMS or email.

Example 5-8 The Response of the sqs receive-message Command Showing the Message Formatted in JSON as Would Be Expected Since It Was Delivered to the Queue Through SNS

Click here to view code image

{
     "Messages": [
         {
             "Body": "{
 "Type" : "Notification",
 "MessageId" : "86cbb
f6d-7c55-5af1-8350-ea85aee60365",
 "TopicArn" : "arn:aws:sns:us-east-2:
111222333444:sqstopic",
 "Message" : "My first SNS message",
 
"Timestamp" : "2019-09-03T20:43:31.531Z",
 "SignatureVersion" : "1",

 "Signature" : "q7rKIF8piO+Q8pnl7JKCFZg/0kvjLSHQTK1diLfbxS6eoJyxQtJPMjR8FQ
e9Cx/aJ6GqfPTsevwAwPBMWo8i+rejjKbkXvp8mTotLpa+mzHokzWlq8Npyjyz73vISsY1X8IeTASoeU+
imD6+H6vAgE5xXBNLAl8Bh9Kt3mMzKdC3eceQKWOJNTbf3RSuEIwhnzbbZ9Q2QRCRg/5J7bL6eg1atY0D5/
jxUyIdnpCDxAxHL/9fookjn4bJRSFc/VeGQT3LfjAiSMUrTteSf+O8Dk7M7znKZgs9XOcbcbg1Poa96nY9Zi
PxRvJyDHWNahIE1Mq14A2DQg9lC4UAUrKZPg==",
 "SigningCertURL" : "https://
sns.us-east-2.amazonaws.com/SimpleNotificationService-6aad65c2f9911b05cd53ef
da11f913f9.pem",
 "UnsubscribeURL" : "https://sns.us-east-2.amazonaws.com/
?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-2:111222333444:sqstopic:
49fea96b-87c5-4720-9d71-a8dce33dc08c"
}",

            "ReceiptHandle": "AQEBrk+x+zvFISEzz4XuidS3F/ZzfglN6QA06Hz5SvUIwAJyQic
J0egnxyLEaQCNWqQHlrgbEi9ai7XOnRS7Dq2BH1ThfxzBCqS1qtI7kjS00Fva4g1flw1OQ3FcyZhuaw2Wu
WkVKrFcFuQjnsj/qaMa8vij3QXL4UjT/45wLiU90VbtoRCNW3iWMhmb9GvgAfhKRoO97z650LeSWgg47EqzJ
VLvndpxg5eNLqR3ckgOKTzEo3hAO6PEtOo7aOcy/Rh+AGK6Qkl37lZGkb/rdSjGIqtKUDd/7nIt
WgxaXIc3+g0LAZLak+5lTgeFcxmyG9rjfPmfjss4mFm7DwGOWzbVYnQcftDsvcdX5i2y/
tRXv7dWrrapS0kdRdKv9emRc1Pm0WFwheXbWYVOTR0gc3IhcA==",

            "MD5OfBody": "e32b6a9e6ee10391388ed97b80cd022c",

            "MessageId": "1f9b9a72-7316-4fef-a38a-970eda015f89"
        }
    ]
}

This example has demonstrated how to use SNS and SQS. By subscribing multiple SQS queues, you would essentially create a fan-out queue.

Exam Preparation Tasks

To prepare for the exam, use this section to review the topics covered and the key aspects that will allow you to gain the knowledge required to pass the exam. To gain the necessary knowledge, complete the exercises, examples, and questions in this section in combination with Chapter 9, “Final Preparation,” and the exam simulation questions in the Pearson Test Prep Software Online.