In order to react to the changes that happen in a table, we need to know what the changes are. That's where DynamoDB Streams come into play. A stream is like a message queue or notification topic that you can subscribe to in order to learn about the changes that are made to a table. Messages on a stream are asynchronous, arrive exactly once, and the stream itself scales with the table.

After creating a new stream on a table, DynamoDB will publish a number of change records on the stream in response to items in the table being inserted, deleted, or updated. A change record will include the old item, the new item, the primary key, and the type of change that was made to the item.

Here's an example message with one change record. We can see that the difference in the two images is that the value for the FLASH_FIRED key changed to true. The following change record has been simplified for brevity:

{
    "eventID": "1",
    "dynamodb": {
    "OldImage": {
        "id": { "N": "2" },
        ...
        "FLASH_FIRED": { "BOOL": false }
    },
    "Keys": { "id": { "N": "2" } },
    "NewImage": {
        "id": { "N": "2" },
        ...
        "FLASH_FIRED": { "BOOL": true }
    },
    "eventName": "MODIFY"
    "StreamViewType": "NEW_AND_OLD_IMAGES"
}

The data on a stream lasts for 24 hours, and a Lambda consuming from the stream can have an unlimited number of retries.

Let's have a look at how to create a stream on an existing table. If you don't already have one, create a new photos-metadata table:

1. Navigate to the DynamoDB console and click Create table. You can use the following configurations to create your table:

For this example, we will create a new stream on the photos-metadata table.

2. On the table overview page, click the Manage Stream button and enable a stream with a view type of New and old images. Your view should look similar to the following:

This creates a new stream with its own ARN to refer to. Choosing the New and old images view type means that, when a new event is triggered by a table update, the event object that goes into the stream will contain both the original item and the updated item with the associated values. You can also choose to have just the old items, just the new items, or just the attributes. The following diagram shows a high-level view of what happens when an item is added to a table that has a stream enabled:

Now, let's create a new Lambda function so that we can hook it up to our new stream:

1. First, we'll need to make sure that the execution role that we assign to our Lambda has the appropriate permissions. It's likely that you'll need to add a statement to the policy document to allow the function to access the stream:

{
    "Effect": "Allow",
    "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
    ],
    "Resource": "arn:aws:dynamodb:${region}:${account-id}:table/photos-metadata/stream/*"
}

2. Create a new lambda function. I've called mine processMetadata and used a blueprint called dynamodb-process-stream. This will save me some time in writing the implementation.

3. Next up is selecting the trigger:

When you click on point number (2), a configuration box will appear where you can select which table to listen to, the batch size, and the starting position. These are some new configuration elements that need some explanation. Batch size is the number of records that are passed to the Lambda function every time there is a change triggered. This number can be up to 1,000 MB or up to 6 MB in total size. There are two options for the starting position:

• • Latest, where all the new records are included from the tail of the stream
  • Trim horizon, where the Lambda receives all the records in the stream shard

4. When you've added the new trigger, remember to hit Save.
5. Test the new integration by adding an item to your table!

Just like how we learned about S3, we are going to move on from event sourcing to interacting with the service via the SDK. In the next section, we will learn how to add data to a table in DynamoDB.