Appendix A. Services for your serverless architecture

This appendix covers

• Useful services for serverless architecture
• Products and services appropriate for source control and DevOps

AWS has an array of services that you can use for building serverless architectures. Lambda is a key service, but other services can be very useful, if not essential, for solving certain problems. There are many excellent non-AWS services too, which you should consider when putting together your architecture. The following is a sample of services that we’ve found useful, but it’s not an exhaustive list. This book describes and shows how to use these services, and many others, to create robust serverless architectures. You can use this list as a quick reference if you find that we refer to a service somewhere in the book and you need a quick refresher on what it’s all about.

A.1. API Gateway

The Amazon API Gateway is a service that you can use to create an API layer between the front-end and back-end services. The lifecycle management of the API Gateway allows multiple versions of the API to be run at the same time, and it supports multiple release stages such as development, staging, and production. API Gateway also comes with useful features like caching and throttling of requests.

The API is defined around resources and methods. A resource is a logical entity such as a user or product. A method is a combination of an HTTP verb, such as GET, POST, PUT, or DELETE, and the resource path. The API Gateway integrates with Lambda and, as you’ll see in chapter 7, makes moving data in and out of Lambda straightforward. API Gateway can also connect to various AWS services via an AWS service proxy and forward requests to regular HTTP endpoints.

A.2. Simple Notification Service

Amazon Simple Notification Service (SNS) is a scalable pub-sub service designed to deliver messages. Producers or publishers create and send messages to a topic. Subscribers or consumers subscribe to a topic and receive messages over one of the supported protocols. SNS stores messages across multiple servers and data centers for redundancy and guarantees at-least-once delivery. At-least-once delivery stipulates that a message will be delivered at least once to a subscriber but, on rare occasions due to the distributed nature of SNS, it may be delivered multiple times.

In cases when a message can’t be delivered by SNS to HTTP endpoints, it can be configured to retry deliveries at a later time. SNS can also retry failed deliveries to Lambda in cases where throttling is applied. SNS supports message payloads of up to 256 KB.

A.3. Simple Storage Service

Simple Storage Service (S3) is Amazon’s scalable storage solution. Data in S3 is stored redundantly across multiple facilities and servers. The event notifications system allows S3 to send events to SNS, SQS, or Lambda when objects are created or deleted. S3 is secure by default, with only owners having access to the resources they create, but it’s possible to set more granular and flexible access permissions using access control lists and bucket policies.

S3 uses the concept of buckets and objects. Buckets are high-level directories or containers for objects. Objects are a combination of data, metadata, and a key. A key is a unique identifier for an object in a bucket. S3 also supports the concept of a folder as a means of grouping objects in the S3 console. Folders work by using key name prefixes. A forward slash character “/” in the key name delineates a folder. For example, an object with the key name documents/personal/myfile.txt is represented as a folder called documents, containing a folder called personal, containing the file myfile.txt in the S3 console.

A.4. Simple Queue Service

Simple Queue Service (SQS) is Amazon’s distributed and fault-tolerant queuing service. It ensures at-least-once delivery of messages similar to SNS and supports message payloads of up to 256 KB. SQS allows multiple publishers and consumers to interact with the same queue, and it has a built-in message lifecycle that automatically expires and deletes messages after a preset retention period. As with most AWS products, there are access controls to help control access to the queue. SQS integrates with SNS to automatically receive and queue messages.

A.5. Simple Email Service

Simple Email Service (SES) is a service designed to send and receive email. SES handles email-receiving operations such as scanning for spam and viruses and rejection of email from untrusted sources. Incoming email can be delivered to an S3 bucket, or used to invoke a Lambda notification or create an SNS notification. These actions can be configured as part of the receipt rule, which tells SES what to do with the email once it arrives.

Sending emails with SES is straightforward but there are limits, which are in place to regulate the rate and the number of messages being sent out. SES will automatically increase the quota as long as high-quality email, and not spam, is being sent.

A.6. Relational Database Service and DynamoDB

Amazon Relational Database Service (RDS) is a web service that helps with the setup and operation of a relational database in the AWS infrastructure. RDS supports the Amazon Aurora, MySQL, MariaDB, Oracle, MS-SQL, and PostgreSQL database engines. It takes care of routine tasks such as provisioning, backup, patching, recovery, repair, and failure detection. Monitoring and metrics, database snapshots, and multiple availability zone (AZ) support are provided out of the box. RDS uses SNS to deliver notifications when an event occurs. This makes it easy to respond to database events such as creation, deletion, failover, recovery, and restoration when they happen.

DynamoDB is Amazon’s NoSQL solution. Tables, items, and attributes are Dynamo’s main concepts. A table stores a collection of items. An item is made up of a collection of attributes. Each attribute is a simple piece of data such as a person’s name or phone number. Every item is uniquely identifiable. Lambda integrates with DynamoDB tables and can be triggered by a table update.

A.7. CloudSearch

CloudSearch is a search solution from AWS that supports structured data and plain text. CloudSearch takes snippets of data as JSON or XML and generates an index that can be queried. This service supports Boolean, prefix, range, and full-text search, as well as faceting, highlighting, and autocomplete. Every document provided to CloudSearch is supplied with an ID, generated by the user, that makes the document uniquely identifiable. Search requests can be carried out using GET requests. Results can be returned as JSON or XML, and they can be sorted and paginated and include useful metadata such as a relevance score.

A.8. Elastic Transcoder

Elastic Transcoder is an AWS service for transcoding media to other formats, resolutions, and bitrates. This service is useful if you need to have versions of your media playable on different devices. Elastic Transcoder comes with a number of presets, or templates, that define how a video should be transcoded. And, if needed, you can create your own.

Elastic Transcoder integrates with S3 and SNS, which it uses for notifications when a job is completed or an error condition is raised. Elastic Transcoder has additional features such as watermarking, transcoding of captions, and digital rights management support.

A.9. Kinesis Streams

Kinesis Streams is a service for real-time processing of streaming big data. It’s typically used for quick log and data intake, metrics, analytics, and reporting. It’s different from SQS in that Amazon recommends that Kinesis Streams be used primarily for streaming big data, whereas SQS is used as a reliable hosted queue, especially if more fine-grained control over messages, such as visibility timeouts or individual delays, is required.

In Kinesis Streams, shards specify the throughput capacity of a stream. The number of shards needs to be stipulated when the stream is created, but resharding is possible if throughput needs to be increased or reduced. In comparison, SQS makes scaling much more transparent. Lambda can integrate with Kinesis to read batches of records from a stream as soon as they’re detected.

A.10. Cognito

Amazon Cognito is an identity management service. It integrates with public identity providers such as Google, Facebook, Twitter, and Amazon or with your own system. Cognito supports user pools, which allow you to create your own user directory. This allows you to register and authenticate users without having to run a separate user database and authentication service. Cognito supports synchronization of user application data across different devices and has offline support that allows mobile devices to function even when there’s no internet access.

A.11. Auth0

Auth0 is a non-AWS identity management product that has a few features that Cognito doesn’t. Auth0 integrates with more than 30 identity providers, including Google, Facebook, Twitter, Amazon, LinkedIn, and Windows Live. It provides a way to register new users through the use of its own user database, without having to integrate with an identity provider. In addition, it has a facility to import users from other databases.

As expected, Auth0 supports standard industry protocols including SAML, OpenID Connect, OAuth 2.0, OAuth 1.0, and JSON Web Token. It’s dead simple to integrate with AWS Identity and Access Management and with Cognito.

A.12. Firebase

Firebase (owned by Google) is a company and also a suite of interesting products. One of the products that we particularly like is its NoSQL real-time database. Data in Firebase is stored as JSON. One of the nice things about Firebase is its real-time synchronization. It allows all connected users to receive updates as soon as they happen. Firebase can be accessed through a REST API and through client libraries, which are available for different languages and platforms. Firebase also has services for static hosting of files and authentication of users.

A.13. Other services

The list of services provided in this section is a short sample of the different products you can use to build your application. There are many more services, including those provided by large cloud-focused companies such as Google and Microsoft and smaller, independent companies like Auth0.

There are also auxiliary services that you need to be aware of. These can help you be more efficient and build software faster, improve performance, or achieve other goals. When building software, consider the following products and services:

• Content delivery networks (CDN) such as CloudFront
• DNS management (Route 53)
• Caching (ElastiCache)
• Source control (GitHub)
• Continuous integration and deployment (Travis CI)

For every service suggestion, you can find alternatives that may be just as good or even better, depending on your circumstances. We urge you to do more research and explore the various services that are currently available.