Appendix A. Services for your serverless architecture
AWS is a giant playground of different services and products you can use to build serverless applications. Lambda is a key service that we discussed in this book, but other services and products can be just as useful, if not crucial, for solving certain problems. There are many excellent non-AWS products too, so don’t feel obligated to use only what Amazon has to offer. Have a look at the offerings from Microsoft and Google too. The following sections provide a sample of services that we’ve found useful. You can use this appendix as a guide to various services and products we’ll discuss throughout the book.
A.1 API Gateway
The Amazon API Gateway is a service that you can use to create an API layer between the frontend and backend services. The lifecycle management of the API Gateway allows multiple versions of the API to be run at the same time, and it supports multiple release stages such as development, staging, and production. API Gateway also comes with useful features like caching and throttling requests.
The API is defined around resources and methods. A resource is a logical entity such as a user or product. A method is a combination of an HTTP verb (such as GET, POST, PUT, or DELETE) and the resource path. API Gateway integrates with Lambda and other AWS services. It can be used as a proxy service and forward requests to regular HTTP endpoints.
A.2 Simple Notification Service (SNS)
Amazon Simple Notification Service (SNS) is a scalable pub/sub service designed to deliver messages. Producers or publishers create and send messages to a topic. Subscribers or consumers subscribe to a topic and receive messages over one of the supported protocols. SNS stores messages across multiple servers and data centers for redundancy and guarantees at-least-once delivery. At-least-once delivery stipulates that a message will be delivered at least once to a subscriber, but on rare occasions, due to the distributed nature of SNS, it may be delivered multiple times.
In cases where a message can’t be delivered by SNS to HTTP endpoints, it can be configured to retry deliveries at a later time. SNS can also retry failed deliveries to Lambda when throttling is applied. SNS supports message payloads of up to 256 KB.
A.3 Simple Storage Service (S3)
Simple Storage Service (S3) is Amazon’s scalable storage solution. Data in S3 is stored redundantly across multiple facilities and servers. The event notifications system allows S3 to send events to SNS, SQS, or Lambda when objects are created or deleted. S3 is secure, by default, with only owners having access to the resources they create, but it’s possible to set more granular and flexible access permissions using access control lists and bucket policies.
S3 uses the concept of buckets and objects. Buckets are high-level directories or containers for objects. Objects are a combination of data, metadata, and a key. A key is a unique identifier for an object in a bucket.
S3 also supports the concept of a folder as a means of grouping objects in the S3 console. Folders work by using key name prefixes. A forward slash character (/) in the key name delineates a folder. For example, an object with the key name documents/personal/myfile.txt is represented as a folder called documents, containing a folder called personal, containing the file myfile.txt in the S3 console.
A.4 Simple Queue Service (SQS)
Simple Queue Service (SQS) is Amazon’s distributed and fault-tolerant queuing service. It ensures at-least-once delivery of messages similar to SNS and supports message payloads of up to 256 KB. SQS allows multiple publishers and consumers to interact with the same queue, and it has a built-in message lifecycle that automatically expires and deletes messages after a preset retention period. As with most AWS products, there are access controls to help control access to the queue. SQS integrates with SNS to automatically receive and queue messages.
A.5 Simple Email Service (SES)
Simple Email Service (SES) is a service designed to send and receive email. SES handles email-receiving operations such as scanning for spam and viruses and rejection of email from untrusted sources. Incoming email can be delivered to an S3 bucket or used to invoke a Lambda notification, or create an SNS notification. These actions can be configured as part of the receipt rule, which tells SES what to do with the email once it arrives.
Sending emails with SES is straightforward, but there are limits that are in place to regulate the rate and the number of messages sent. SES automatically increases the quota as long as high-quality email, and not spam, is sent.
A.6 Relational Database Service (RDS)
Amazon Relational Database Service (RDS) is a web service that helps with the setup and operation of a relational database in the AWS infrastructure. RDS supports the Amazon Aurora, MySQL, MariaDB, Oracle, MS-SQL, and PostgreSQL database engines. It takes care of routine tasks such as provisioning, backup, patching, recovery, repair, and failure detection. Monitoring and metrics, database snapshots, and multiple availability zone (AZ) support are provided out of the box. RDS uses SNS to deliver notifications when an event occurs. This makes it easy to respond to database events such as creation, deletion, failover, recovery, and restoration when they happen.
A.7 DynamoDB
DynamoDB is Amazon’s NoSQL database. Tables, items, and attributes are Dynamo’s main concepts. A table stores a collection of items. An item is made up of a collection of attributes. Each attribute is a simple piece of data such as a person’s name or phone number. Every item is uniquely identifiable. Lambda integrates with DynamoDB tables and can be triggered by a table update. Global tables is a notable feature of Dynamo that seamlessly replicates tables across different AWS regions and resolves any data conflicts (using “last writer wins” reconciliation to handle concurrent updates). It makes DynamoDB a good database for scalable, global applications. Finally, an in-memory cache (DAX) is available for DynamoDB. It shortens the response time but comes at a price.
A.8 Algolia
Algolia is a (non-AWS) managed search engine API. It can search through semi-structured data and has APIs to allow developers to integrate search directly into their websites and mobile applications. One of Algolia’s outstanding capabilities is its speed. Algolia can distribute and synchronize data across 15 regions around the world and direct queries to the closest data center.
Algolia has a concept of indices (“. . . an entity where you import the data you want to search . . . analogous to a table within a database . . .”), records (“. . . a JSON schemaless object that you want to be searchable . . .”) and operations (which are essentially atomic actions such as update or delete). These concepts are straightforward and make Algolia one of the easier search platforms to use. Paid plans begin from about $35 per month but can quickly grow in cost, depending on the number of records and operations performed by your application and users.
A.9 Media Services
AWS Media Services is a new product designed for developers to build video workflows. Media Services consist of the following products:
-
MediaConvert is designed to transcode between different video formats at scale.
-
MediaLive is a live video-processing service. It takes a live video source and compresses it into smaller versions for distribution.
-
MediaPackage enables developers to implement video features such as pause and rewind. It can also be used to add Digital Right Management (DRM) to content.
-
MediaStore is a storage service optimized for media. Its aim is to provide a low-latency storage system for live and on-demand video content.
-
MediaTailor enables developers to insert individually targeted ads in to the video stream.
Media Services provide an advanced suite of services that are superior to Elastic Transcoder. Nevertheless, Elastic Transcoder has a few features (such as the ability to create WebM files and animated GIFs) that Media Services is missing.
A.10 Kinesis Streams
Kinesis Streams is a service for real-time processing of streaming big data. It’s typically used for quick log and data intake, metrics, analytics, and reporting. It’s different from SQS in that Amazon recommends that Kinesis Streams be used primarily for streaming big data, whereas SQS is used as a reliable hosted queue, especially if more fine-grained control over messages such as visibility timeouts or individual delays is required.
In Kinesis Streams, shards specify the throughput capacity of a stream. The number of shards needs to be stipulated when the stream is created, but resharding is possible if throughput needs to be increased or reduced. In comparison, SQS makes scaling much more transparent. Lambda can integrate with Kinesis to read batches of records from a stream as soon as they’re detected.
A.11 Athena
AWS bills Athena as a serverless interactive query service. Essentially, this service allows you to query data placed into S3 using standard SQL. In a lot of cases, there’s no need to run ETL (extract, transform, and load) jobs to transform your data before querying can take place (although you can combine Athena with AWS Glue if you needed to transform your data a certain way). As a user, you upload data to S3, prepare a schema, and begin querying almost immediately.
A.12 AppSync
AppSync is billed as allowing developers to create “ . . . data driven apps with real-time and offline capabilities.” In reality, AppSync is a managed GraphQL endpoint provided by AWS. It integrates with DynamoDB, Lambda, and Amazon Elasticsearch. If you are familiar with GraphQL and GraphQL schemas, you can get started with AppSync straight away. If you are not familiar with GraphQL, we recommend doing a bit of reading beforehand (http://graphql.org/learn/). GraphQL has certainly been finding its share of acclaim over the past few years, particularly among adopters of serverless technologies.
A.13 Cognito
Amazon Cognito is an identity management service. It integrates with public identity providers such as Google, Facebook, Twitter, and Amazon or with your own system. Cognito supports user pools, which allow you to create your own user directory. This lets you register and authenticate users without having to run a separate user database and authentication service. Cognito supports synchronization of user application data across different devices and has offline support that allows mobile devices to function even when there’s no internet access.
A.14 Auth0
Auth0 (recently acquired by Okta) is a non-AWS identity management product that has a few features that Cognito doesn’t. Auth0 integrates with more than 30 identity providers including Google, Facebook, Twitter, Amazon, LinkedIn, and Windows Live. It provides a way to register new users through the use of its own user database, without having to integrate with an identity provider. In addition, it has a facility to import users from other databases. As expected, Auth0 supports standard industry protocols including SAML, OpenID Connect, OAuth 2.0, OAuth 1.0, and JSON Web Token (JWT). It’s simple to integrate with AWS Identity, Access Management, and Cognito.
A.15 Other services
The list of services provided in this section is a short sample of the different products you can use to build your application. There are many more services, including those provided by large cloud-focused companies such as Google and Microsoft and smaller, independent companies like Auth0. There are also auxiliary services that you need to be aware of. These can help you be more efficient and build software faster, improve performance, or achieve other goals. When building software, consider the following products and services:
For every service suggestion, you can find alternatives that may be just as good or even better, depending on your circumstances. We urge you to do more research and explore the various services that are currently available.