Prefixes and Delimiters

As you've seen, S3 stores objects within a bucket on a flat surface without subfolder hierarchies. However, you can use prefixes and delimiters to give your buckets the appearance of a more structured organization.

A prefix is a common text string that indicates an organization level. For example, the word contracts when followed by the delimiter / would tell S3 to treat a file with a name like contracts/acme.pdf as an object that should be grouped together with a second file named contracts/dynamic.pdf.

S3 recognizes folder/directory structures as they're uploaded and emulates their hierarchical design within the bucket, automatically converting slashes to delimiters. That's why you'll see the correct folders whenever you view your S3‐based objects through the console or the API.

Working with Large Objects

Although there's no theoretical limit to the total amount of data you can store within a bucket, a single object may be no larger than 5 TB. Individual uploads can be no larger than 5 GB. To reduce the risk of data loss or aborted uploads, AWS recommends that you use a feature called Multipart Upload for any object larger than 100 MB.

As the name suggests, Multipart Upload breaks a large object into multiple smaller parts and transmits them individually to their S3 target. If one transmission should fail, it can be repeated without impacting the others.

Multipart Upload will be used automatically when the upload is initiated by the AWS CLI or a high‐level API, but you'll need to manually break up your object if you're working with a low‐level API.

An application programming interface (API) is a programmatic interface through which operations can be run through code or from the command line. AWS maintains APIs as the primary method of administration for each of its services. AWS provides low‐level APIs for cases when your S3 uploads require hands‐on customization, and it provides high‐level APIs for operations that can be more readily automated. This page contains specifics:

docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html

If you need to transfer large files to an S3 bucket, the Amazon S3 Transfer Acceleration configuration can speed things up. When a bucket is configured to use Transfer Acceleration, uploads are routed through geographically nearby AWS edge locations and, from there, routed using Amazon's internal network.

You can find out whether Transfer Acceleration would actually improve transfer speeds between your location and a particular AWS region by using the Amazon S3 Transfer Acceleration Speed Comparison tool (s3-accelerate-speedtest.s3-accelerate.amazonaws.com/en/accelerate-speed-comparsion.html). If your transfers are, in fact, good candidates for Transfer Acceleration, you should enable the setting in your bucket. You can then use special endpoint domain names (like bucketname.s3-accelerate.amazonaws.com) for your transfers.

Work through Exercise 3.1 and create your own bucket.

Encryption

Unless it's intended to be publicly available—perhaps as part of a website—data stored on S3 should always be encrypted. You can use encryption keys to protect your data while it's at rest within S3 and—by using only Amazon's encrypted API endpoints for data transfers—protect data during its journeys between S3 and other locations.

Data at rest can be protected using either server‐side or client‐side encryption.

Logging

Tracking S3 events to log files is disabled by default—S3 buckets can see a lot of activity, and not every use case justifies the log data that S3 can generate.

When you enable logging, you'll need to specify both a source bucket (the bucket whose activity you're tracking) and a target bucket (the bucket to which you'd like the logs saved). Optionally, you can also specify delimiters and prefixes (such as the creation date or time) to make it easier to identify and organize logs from multiple source buckets that are saved to a single target bucket.

S3‐generated logs, which sometimes appear only after a short delay, will contain basic operation details, including the following:

• The account and IP address of the requestor
• The source bucket name
• The action that was requested (GET, PUT, POST, DELETE, etc.)
• The time the request was issued
• The response status (including error code)

S3 buckets are also used by other AWS services—including CloudWatch and CloudTrail—to store their logs or other objects (like EBS Snapshots).

Durability

S3 measures durability as a percentage. For instance, the 99.999999999 percent durability guarantee for most S3 classes and Amazon Glacier is as follows:

… corresponds to an average annual expected loss of 0.000000001% of objects. For example, if you store 10,000,000 objects with Amazon S3, you can on average expect to incur a loss of a single object once every 10,000 years.

Source: aws.amazon.com/s3/faqs

In other words, realistically, there's pretty much no way that you can possibly lose data stored on one of the standard S3/Glacier platforms because of infrastructure failure. However, it would be irresponsible to rely on your S3 buckets as the only copies of important data. After all, there's a real chance that a misconfiguration, account lockout, or unanticipated external attack could permanently block access to your data. And, as crazy as it might sound right now, it's not unthinkable to suggest that AWS could one day go out of business. Kodak and Blockbuster Video once dominated their industries, right? You should always back up your data to multiple locations, using different services and media types. You'll learn how to do that in Chapter 10, “The Reliability Pillar.” The high durability rates delivered by S3 are largely because they automatically replicate your data across at least three availability zones. This means that even if an entire AWS facility was suddenly wiped off the map, copies of your data would be restored from a different zone.

There is, however, one storage class that isn't quite so resilient. Amazon S3 Reduced Redundancy Storage (RRS) is rated at only 99.99 percent durability (because it's replicated across fewer servers than other classes). The RRS class is still available for historical reasons, but it's officially not recommended that you actually use it.

You can balance increased/decreased durability against other features like availability and cost to get the balance that's right for you. While all currently recommended classes are designed to withstand data loss 99.999999999% (11 nines) of the time, most will be maintained in at least three availability zones. The exception is S3 One Zone‐IA that, as the name suggests, stores its data in only a single zone. The difference shows up in its slightly lower availability, which we'll discuss next.

Availability

Object availability is also measured as a percentage; this time, though, it's the percentage you can expect a given object to be instantly available on request through the course of a full year. The Amazon S3 Standard class, for example, guarantees that your data will be ready whenever you need it (meaning it will be available) for 99.99% of the year. That means there will be less than nine hours each year of downtime. If you feel downtime has exceeded that limit within a single year, you can apply for a service credit. Amazon's durability guarantee, by contrast, is designed to provide 99.999999999% data protection. This means there's practically no chance your data will be lost, even if you might sometimes not have instant access to it.

S3 Intelligent‐Tiering is a relatively new storage class that saves you money while optimizing availability. For a monthly automation fee, Intelligent‐Tiering will monitor the way you access data within the class over time. It will automatically move an object to the lower‐cost infrequent access tier after it hasn't been accessed for 30 consecutive days.

Table 3.1 illustrates the availability guarantees for all S3 classes.

Eventually Consistent Data

It's important to bear in mind that S3 replicates data across multiple locations. As a result, there might be brief delays while updates to existing objects propagate across the system. Uploading a new version of a file or, alternatively, deleting an old file altogether can result in one site reflecting the new state with another still unaware of any changes.

To ensure that there's never a conflict between versions of a single object—which could lead to serious data and application corruption—you should treat your data according to an eventually consistent standard. That is, you should expect a delay (usually just two seconds or less) and design your operations accordingly.

Because there isn't the risk of corruption when creating new objects, S3 provides read‐after‐write consistency for creation (PUT) operations.

Versioning

Within many file system environments, saving a file using the same name and location as a preexisting file will overwrite the original object. That ensures you'll always have the most recent version available to you, but you will lose access to older versions—including versions that were overwritten by mistake.

By default, objects on S3 work the same way. But if you enable versioning at the bucket level, then older overwritten copies of an object will be saved and remain accessible indefinitely. This solves the problem of accidentally losing old data, but it replaces it with the potential for archive bloat. Here's where lifecycle management can help.

Lifecycle Management

In addition to the S3 Intelligent‐Tiering storage class we discussed earlier, you can manually configure lifecycle rules for a bucket that will automatically transition an object's storage class after a set number of days. You might, for instance, have new objects remain in the S3 Standard class for their first 30 days, after which they're moved to the cheaper One Zone IA for another 30 days. If regulatory compliance requires that you maintain older versions, your files could then be moved to the low‐cost, long‐term storage service Glacier for 365 more days before being permanently deleted.

Try it yourself with Exercise 3.2.

Access Control

Out of the box, new S3 buckets and objects will be fully accessible to your account but to no other AWS accounts or external visitors. You can strategically open up access at the bucket and object levels using access control list (ACL) rules, finer‐grained S3 bucket policies, or Identity and Access Management (IAM) policies.

There is more than a little overlap between those three approaches. In fact, ACLs are really leftovers from before AWS created IAM. As a rule, Amazon recommends applying S3 bucket policies or IAM policies instead of ACLs.

S3 bucket policies—which are formatted as JavaScript Object Notation (JSON) text and attached to your S3 bucket—will make sense for cases where you want to control access to a single S3 bucket for multiple external accounts and users. On the other hand, IAM policies—because they exist at the account level within IAM—will probably make sense when you're trying to control the way individual users and roles access multiple resources, including S3.

The following code is an example of an S3 bucket policy that allows both the root user and the user Steve from the specified AWS account to access the S3 MyBucket bucket and its contents. Both users are considered principals within this rule.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": ["arn:aws:iam::xxxxxxxxxxxx:root",
        "arn:aws:iam::xxxxxxxxxxxx:user/Steve"]
      },
      "Action": "s3:*",
      "Resource": ["arn:aws:s3:::MyBucket",
                   "arn:aws:s3:::MyBucket/*"]
    }
  ]
}

When it's attached to an IAM entity (a user, group, or role), the following IAM policy will accomplish the same thing as the previous S3 bucket policy:

{
  "Version": "2012-10-17",
  "Statement":[{
    "Effect": "Allow",
    "Action": "s3:*",
    "Resource": ["arn:aws:s3:::MyBucket",
                 "arn:aws:s3:::MyBucket/*"]
    }
  ]
}

IAM roles and policies will be discussed in greater detail in Chapter 6, “Authentication and Authorization—AWS Identity and Access Management.”

You can also closely control the way users and services access objects within your buckets by using Amazon S3 Access Points. An access point is a hostname that can point to a carefully defined subset of objects in a bucket. Depending on how you configure your access points, clients invoking the hostname will be able to read or write only the data you allow, and only as long as you allow it.

A simple AWS CLI command to request an access point might look something like this:

aws s3control create-access-point --name my-vpc-ap 
   --account-id 123456789012 --bucket my-bucket 
   --vpc-configuration VpcId=vpc-2b9d3c

Presigned URLs

If you want to provide temporary access to an object that's otherwise private, you can generate a presigned URL. The URL will be usable for a specified period of time, after which it will become invalid. You can build presigned URL generation into your code to provide object access programmatically.

The following AWS CLI command will return a URL that includes the required authentication string. The authentication will become invalid after 10 minutes (600 seconds). The default expiration value is 3,600 seconds (one hour).

aws s3 presign s3://MyBucketName/PrivateObject --expires-in 600

Try it yourself with Exercise 3.3.

Static Website Hosting

S3 buckets can be used to host the HTML files for entire static websites. A website is static when the system services used to render web pages and scripts are all client‐ rather than server‐based. This architecture permits simple and lean HTML code that's designed to be executed by the client browser.

S3, because it's such an inexpensive yet reliable platform, is an excellent hosting environment for such sites. When an S3 bucket is configured for static hosting, traffic directed at the bucket's URL can be automatically made to load a specified root document, usually named index.html. Users can click links within HTML pages to be sent to the target page or media resource. Error handling and redirects can also be incorporated into the profile.

If you want requests for a DNS domain name (like mysite.com) routed to your static site, you can use Amazon Route 53 to associate your bucket's endpoint with any registered name. This will work only if your domain name is also the name of the S3 bucket. You'll learn more about domain name records in Chapter 8, “The Domain Name System and Network Routing: Amazon Route 53 and Amazon CloudFront.”

You can also get a free SSL/TLS certificate to encrypt your site by requesting a certificate from AWS Certificate Manager (ACM) and importing it into a CloudFront distribution that specifies your S3 bucket as its origin.

Build your own static website using Exercise 3.4.

AWS provides a different way to access data stored on either S3 Select or Glacier Select. The feature lets you apply SQL‐like queries to stored objects so that only relevant data from within objects is retrieved, permitting significantly more efficient and cost‐effective operations.

One possible use case would involve large comma‐separated values (CSV) files containing sales and inventory data from multiple retail sites. Your company's marketing team might need to periodically analyze only sales data and only from certain stores. Using S3 Select, they'll be able to retrieve exactly the data they need—just a fraction of the full data set—while bypassing the bandwidth and cost overhead associated with downloading the whole thing.

Storage Pricing

To give you a sense of what S3 and Glacier might cost you, here's a typical usage scenario. Imagine you make weekly backups of your company sales data that generate 5 GB archives. You decide to maintain each archive in the S3 Standard Storage and Requests class for its first 30 days and then convert it to S3 One Zone (S3 One Zone‐IA), where it will remain for 90 more days. At the end of those 120 days, you will move your archives once again, this time to Glacier, where they will be kept for another 730 days (two years) and then deleted.

Once your archive rotation is in full swing, you'll have a steady total of (approximately) 20 GB in S3 Standard, 65 GB in One Zone‐IA, and 520 GB in Glacier. Table 3.3 shows what that storage will cost in the US East region at rates current as of this writing.

Of course, storage is only one part of the mix. You'll also be charged for operations including data retrievals; PUT, COPY, POST, or LIST requests; and lifecycle transition requests. Full, up‐to‐date details are available at aws.amazon.com/s3/pricing.

Exercise 3.5 will introduce you to an important cost‐estimating tool.

Amazon Elastic File System

The Elastic File System (EFS) provides automatically scalable and shareable file storage to be accessed from Linux instances. EFS‐based files are designed to be accessed from within a virtual private cloud (VPC) via Network File System (NFS) mounts on EC2 Linux instances or from your on‐premises servers through AWS Direct Connect connections. The goal is to make it easy to enable secure, low‐latency, and durable file sharing among multiple instances.

Amazon FSx

Amazon FSx comes in two flavors: FSx for Lustre and Amazon FSx for Windows File Server. Lustre is an open source distributed filesystem built to give Linux clusters access to high‐performance filesystems for use in compute‐intensive operations. Amazon's FSx service brings Lustre capabilities to your AWS infrastructure.

FSx for Windows File Server, as you can tell from the name, offers the kind of file‐sharing service EFS provides but for Windows servers rather than Linux. FSx for Windows File Server integrates operations with Server Message Block (SMB), NTFS, and Microsoft Active Directory.

AWS Storage Gateway

Integrating the backup and archiving needs of your local operations with cloud storage services can be complicated. AWS Storage Gateway provides software gateway appliances (based on an on‐premises hardware appliance or virtual machines built on VMware ESXi, Microsoft Hyper‐V, Linux KVM, VMware Cloud on AWS, or EC2 images) with multiple virtual connectivity interfaces. Local devices can connect to the appliance as though it's a physical backup device like a tape drive, and the data itself is saved to AWS platforms like S3 and EBS. Data can be maintained in a local cache to make it locally available.

AWS Snowball

Migrating large data sets to the cloud over a normal Internet connection can sometimes require far too much time and bandwidth to be practical. If you're looking to move terabyte‐ or even petabyte‐scaled data for backup or active use within AWS, ordering a Snowball device might be the best option.

When requested, AWS will ship you a physical, 256‐bit, encrypted Snowball storage device onto which you'll copy your data. You then ship the device back to Amazon, where its data will be uploaded to your S3 bucket(s).

Choosing the best method for transferring data to your AWS account will require a bit of arithmetic. You'll have to know the real‐world upload speeds you get from your Internet connection and how much of that bandwidth wouldn't be used by other operations. In this chapter, you've learned about Multipart Upload and Transfer Acceleration for moving larger objects into an S3 bucket. But some objects are just so large that uploading them over your existing Internet connection isn't practical. Think about it; if your Internet service provider (ISP) gives you 10 MB/second, then, assuming no one else is using the connection, uploading a one‐terabyte archive would take you around 10 days!

So, if you really need to move that data to the cloud, you're going to have to either invest in an expensive AWS Direct Connect connection or introduce yourself to an AWS Snowball device (or, for really massive volumes of data, AWS Snowmobile at aws.amazon.com/snowmobile).

AWS DataSync

DataSync specializes in moving on‐premises data stores into your AWS account with a minimum of fuss. It works over your regular Internet connection, so it's not as useful as Snowball for really large data sets. But it is much more flexible, since you're not limited to S3 (or RDS as you are with AWS Database Migration Service). Using DataSync, you can drop your data into any service within your AWS account. That means you can do the following:

• Quickly and securely move old data out of your expensive data center into cheaper S3 or Glacier storage.
• Transfer data sets directly into S3, EFS, or FSx, where it can be processed and analyzed by your EC2 instances.
• Apply the power of any AWS service to any class of data as part of an easy‐to‐configure automated system.

DataSync can handle transfer rates of up to 10 Gbps (assuming your infrastructure has that capacity) and offers both encryption and data validation.

1. Your organization runs Linux‐based EC2 instances that all require low‐latency read/write access to a single set of files. Which of the following AWS services are your best choices? (Choose two.)
   1. AWS Storage Gateway
   2. AWS S3
   3. Amazon Elastic File System
   4. AWS Elastic Block Store
2. Your organization expects to be storing and processing large volumes of data in many small increments. When considering S3 usability, you'll need to know whether you'll face any practical limitations in the use of AWS account resources. Which of the following will normally be available only in limited amounts?
   1. PUT requests/month against an S3 bucket
   2. The volume of data space available per S3 bucket
   3. Account‐wide S3 storage space
   4. The number of S3 buckets within a single account
3. You have a publicly available file called filename stored in an S3 bucket named bucketname. Which of the following addresses will successfully retrieve the file using a web browser?
   1. s3.amazonaws.com/bucketname/filename
   2. filename/ bucketname.s3.amazonaws.com
   3. s3://bucketname/filename
   4. s3://filename/bucketname
4. If you want the files stored in an S3 bucket to be accessible using a familiar directory hierarchy system, you'll need to specify prefixes and delimiters. What are prefixes and delimiters?
   1. A prefix is the name common to the objects you want to group, and a delimiter is the bar character (|).
   2. A prefix is the DNS name that precedes the amazonaws.com domain, and a delimiter is the name you want to give your file directory.
   3. A prefix is the name common to the objects you want to group, and a delimiter is a forward slash character (/).
   4. A prefix is the name common to the file type you want to identify, and a delimiter is a forward slash character (/).
5. Your web application relies on data objects stored in AWS S3 buckets. Compliance with industry regulations requires that those objects are encrypted and that related events can be closely tracked. Which combination of tools should you use? (Choose two.)
   1. Server‐side encryption
   2. Amazon S3‐Managed Keys
   3. AWS KMS‐Managed Keys
   4. Client‐side encryption
   5. AWS End‐to‐End managed keys
6. You are engaged in a deep audit of the use of your AWS resources and you need to better understand the structure and content of your S3 server access logs. Which of the following operational details are likely to be included in S3 server access logs? (Choose three.)
   1. Source bucket name
   2. Action requested
   3. Current bucket size
   4. API bucket creation calls
   5. Response status
7. You're assessing the level of durability you'll need to sufficiently ensure the long‐term viability of a new web application you're planning. Which of the following risks are covered by S3's data durability guaranties? (Choose two.)
   1. User misconfiguration
   2. Account security breach
   3. Infrastructure failure
   4. Temporary service outages
   5. Data center security breach
8. Which of the following explains the difference in durability between S3's One Zone‐IA and Reduced Redundancy classes?
   1. One Zone‐IA data is heavily replicated but only within a single availability zone, whereas Reduced Redundancy data is only lightly replicated.
   2. Reduced Redundancy data is heavily replicated but only within a single availability zone, whereas One Zone‐IA data is only lightly replicated.
   3. One Zone‐IA data is replicated across AWS regions, whereas Reduced Redundancy data is restricted to a single region.
   4. One Zone‐IA data is automatically backed up to Amazon Glacier, whereas Reduced Redundancy data remains within S3.
9. Which of the following is the 12‐month availability guarantee for the S3 Standard‐IA class?
   1. 99.99 percent
   2. 99.9 percent
   3. 99.999999999 percent
   4. 99.5 percent
10. Your application regularly writes data to an S3 bucket, but you're worried about the potential for data corruption as a result of conflicting concurrent operations. Which of the following data operations would not be subject to concerns about eventual consistency?
    1. Operations immediately preceding the deletion of an existing object
    2. Operations subsequent to the updating of an existing object
    3. Operations subsequent to the deletion of an existing object
    4. Operations subsequent to the creation of a new object
11. You're worried that updates to the important data you store in S3 might incorrectly overwrite existing files. What must you do to protect objects in S3 buckets from being accidentally lost?
    1. Nothing. S3 protects existing files by default.
    2. Nothing. S3 saves older versions of your files by default.
    3. Enable versioning.
    4. Enable file overwrite protection.
12. Your S3 buckets contain many thousands of objects. Some of them could be moved to less expensive storage classes and others still require instant availability. How can you apply transitions between storage classes for only certain objects within an S3 bucket?
    1. By specifying particular prefixes when you define your lifecycle rules
    2. This isn't possible. Lifecycle rules must apply to all the objects in a bucket.
    3. By specifying particular prefixes when you create the bucket
    4. By importing a predefined lifecycle rule template
13. Which of the following classes will usually make the most sense for long‐term storage when included within a sequence of lifecycle rules?
    1. Glacier
    2. Reduced Redundancy
    3. S3 One Zone‐IA
    4. S3 Standard‐IA
14. Which of the following are the recommended methods for providing secure and controlled access to your buckets? (Choose two.)
    1. S3 access control lists (ACLs)
    2. S3 bucket policies
    3. IAM policies
    4. Security groups
    5. AWS Key Management Service
15. In the context of an S3 bucket policy, which of the following statements describes a principal?
    1. The AWS service being defined (S3 in this case)
    2. An origin resource that's given permission to alter an S3 bucket
    3. The resource whose access is being defined
    4. The user or entity to which access is assigned
16. You don't want to open up the contents of an S3 bucket to anyone on the Internet, but you need to share the data with specific clients. Generating and then sending them a presigned URL is a perfect solution. Assuming you didn't explicitly set a value, how long will the presigned URL remain valid?
    1. 24 hours
    2. 3,600 seconds
    3. 5 minutes
    4. 360 seconds
17. Which non‐S3 AWS resources can improve the security and user experience of your S3‐hosted static website? (Choose two.)
    1. AWS Certificate Manager
    2. Elastic Compute Cloud (EC2)
    3. Relational Database Service (RDS)
    4. Route 53
    5. AWS Key Management Service
18. What is the largest single archive supported by Amazon Glacier?
    1. 5 GB
    2. 40 TB
    3. 5 TB
    4. 40 GB
19. You need a quick way to transfer very large (peta‐scale) data archives to the cloud. Assuming your Internet connection isn't up to the task, which of the following will be both (relatively) fast and cost‐effective?
    1. Direct Connect
    2. Server Migration Service
    3. Snowball
    4. Storage Gateway
20. Your organization runs Windows‐based EC2 instances that all require low‐latency read/write access to a single set of files. Which of the following AWS services is your best choice?
    1. Amazon FSx for Windows File Server
    2. Amazon FSx for Lustre
    3. Amazon Elastic File System
    4. Amazon Elastic Block Store