To use authentication, we need to acquire some credentials.

We will set this up through the AWS Console. Once you've registered with AWS, log into the Console at https://console.aws.amazon.com.

Once you are logged in, you need to perform the steps shown here:

You'll see a new page saying that the user has been created successfully. Click on the Download credentials button at the bottom right corner to download a CSV file, which contains the Access ID and Access Secret for this user. These are important because they will help in authenticating ourselves to the S3 API. Make sure that you store them securely, as they will allow full access to your S3 files.

Then, click on Close at the bottom of the screen, and click on the new user in the list that will appear, and then click on the Attach Policy button. A list of policy templates will appear. Scroll down this list and select the AmazonS3FullAccess policy, as shown in the following screenshot:

Finally, click on the Attach Policy button at the bottom right-hand side when it appears. Now, our user has full access to the S3 service.

Whenever we write a client for an API, our main point of reference is the API documentation. The documentation tells us how to construct the HTTP requests for performing operations. The S3 documentation can be found at http://docs.aws.amazon.com/AmazonS3/latest/API/APIRest.html. The http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketPUT.html URL will provide the details of bucket creation.

This documentation tells us that to create a bucket we need to make an HTTP request to our new bucket's endpoint by using the HTTP PUT method. It also tells us that the request body must contain some XML, which specifies the AWS region that we want the bucket to be created in.

So, now we know what we're aiming for, let's discuss our function. First, let's create the XML. Replace the content of create_bucket() with the following code block:

def create_bucket(bucket):
    XML = ET.Element('CreateBucketConfiguration')
    XML.attrib['xmlns'] = ns
    location = ET.SubElement(XML, 'LocationConstraint')
    location.text = auth.region
    data = ET.tostring(XML, encoding='utf-8')
    xml_pprint(data)

Here we create an XML tree following the format given in the S3 documentation. If we run our client now, then we will see the XML shown here:

$ python3.4 s3_client.py create_bucket mybucket.example.com
<?xml version="1.0" ?>
<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006- 03-01/">
    <LocationConstraint>eu-west-1</LocationConstraint>
</CreateBucketConfiguration>

This matches the format specified in the documentation. You can see that we've used the ns variable to fill the xmlns attribute. This attribute pops up throughout the S3 XML, having the ns variable pre-defined makes it quicker to work with it.

Now, let's add the code to make the request. Replace the xml_pprint(data) at the end of create_bucket() with the following:

    url = 'http://{}.{}'.format(bucket, endpoint)
    r = requests.put(url, data=data, auth=auth)
    if r.ok:
        print('Created bucket {} OK'.format(bucket))
    else:
        xml_pprint(r.text)

The first line shown here will generate the full URL from our bucket name and endpoint. The second line will make the request to the S3 API. Notice that we have used the requests.put() function to make this request using the HTTP PUT method, rather than by using either the requests.get()method or the requests.post() method. Also, note that we have supplied our auth object to the call. This will allow Requests to handle all the S3 authentication for us!

If all goes well , then we print out a message. In case everything does not go as expected, we print out the response body. S3 returns error messages as XML in the response body. So we use our xml_pprint() function to display it. We'll look at working with these errors in the Handling errors section, later on.

Now run the client, and if everything works as expected, then we will get a confirmation message. Make sure that you have picked a bucket that hasn't already been created:

$ python3.4 s3_client.py create_bucket mybucket.example.com
Created bucket mybucket.example.com OK

When we refresh the S3 Console in our browser, we will see that our bucket has been created.

Now that we've created a bucket, we can upload some files. Writing a function for uploading a file is similar to creating a bucket. We check the documentation to see how to construct our HTTP request, figure out what information should be collected at the command line, and then write the function.

We need to use an HTTP PUT again. We need the name of the bucket that we want to store the file in and the name that we want the file to be stored under in S3. The body of the request will contain the file data. At the command line, we'll collect the bucket name, the name we want the file to have in the S3 service and the name of the local file to upload.

Add the following function to your s3_client.py file after the create_bucket() function:

def upload_file(bucket, s3_name, local_path):
    data = open(local_path, 'rb').read()
    url = 'http://{}.{}/{}'.format(bucket, endpoint, s3_name)
    r = requests.put(url, data=data, auth=auth)
    if r.ok:
        print('Uploaded {} OK'.format(local_path))
    else:
        xml_pprint(r.text)

In creating this function, we follow a pattern similar to that for creating a bucket:

Note that we open the local file in binary mode. The file could contain any type of data, so we don't want text transforms applied. We could pull this data from anywhere, such as a database or another web API. Here, we just use a local file for simplicity.

The URL is the same endpoint that we constructed in create_bucket() with the S3 object name appended to the URL path. Later, we can use this URL to retrieve the object.

Now, run the command shown here to upload a file:

$ python3.4 s3_client.py mybucket.example.com test.jpg ~/test.jpg
Uploaded ~/test.jpg OK

You'll need to replace mybucket.example.com with your own bucket name. Once the file gets uploaded, you will see it in the S3 Console.

I have used a JPEG image that was stored in my home directory as the source file. You can use any file, just change the last argument to an appropriate path. However, using a JPEG image will make the following sections easier for you to reproduce.

By default, S3 applies restrictive permissions for buckets and objects. The account that creates them has full read-write permissions, but access is completely denied for anyone else. This means that the file that we've just uploaded can only be downloaded if the download request includes authentication for our account. If we try the resulting URL in a browser, then we'll get an access denied error. This isn't very useful if we're trying to use S3 for sharing files with other people.

The solution for this is to use one of S3's mechanisms for changing the permissions. Let's look at the simple task of making our uploaded file public. Change upload_file() to the following:

def upload_file(bucket, s3_name, local_path, acl='private'):
    data = open(local_path, 'rb').read()
    url = 'http://{}.{}/{}'.format(bucket, endpoint, s3_name)
    headers = {'x-amz-acl': acl}
    r = requests.put(url, data=data, headers=headers, auth=auth)
    if r.ok:
        print('Uploaded {} OK'.format(local_path))
    else:
        xml_pprint(r.text)

We have now included a header in our HTTP request, x-amz-acl, which specifies a permission set to be applied to the object. We've also added a new argument to our function signature so that we can specify the permission set on the command line. We have used the so-called canned ACLs (canned Access Control Lists), which have been provided by S3, and are documented at http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl.

The ACL that we're interested in is called public-read. This will allow anyone to download the file without needing any kind of authentication. We can now re-run our upload, but this time it will apply this ACL to it:

$ python3.4 s3_client.py mybucket.example.com test.jpg ~/test.jpg public-read
Uploaded test.jpg OK

Now, visiting the file's S3 URL in a browser will give us the option to download the file.

If you have uploaded an image, then you may be wondering why the browser had asked us to save it instead of just displaying it. The reason is that we haven't set the file's Content-Type.

If you remember from the last chapter, the Content-Type header in an HTTP response tells the client, which in this case is our browser, the type of file that is in the body. By default, S3 applies the content type of binary/octet-stream. Because of this Content-Type, the browser can't tell that it's downloading an image, so it just presents it as a file that can be saved. We can fix this by supplying a Content-Type header in the upload request. S3 will store the type that we specify, and it will use it as the Content-Type in the subsequent download responses.

Add the code block shown here to the import at the beginning of s3_client.py:

import mimetypes

Then change upload_file() to this:

def upload_file(bucket, s3_name, local_path, acl='private'):
    data = open(local_path, 'rb').read()
    url = 'http://{}.{}/{}'.format(bucket, endpoint, s3_name)
    headers = {'x-amz-acl': acl}
    mimetype = mimetypes.guess_type(local_path)[0]
    if mimetype:
        headers['Content-Type'] = mimetype
    r = requests.put(url, data=data, headers=headers, auth=auth)
    if r.ok:
        print('Uploaded {} OK'.format(local_path))
    else:
        xml_pprint(r.text)

Here, we have used the mimetypes module to guess a suitable Content-Type by looking at the file extension of local_path. If mimetypes can't determine a Content-Type from local_path, then we don't include the Content-Type header, and let S3 apply the default binary/octet-stream type.

Unfortunately, in S3 we won't be able to overwrite the metadata for an existing object by using a simple PUT request. It's possible to do it by using a PUT copy request, but that's beyond the scope of this chapter. For now, it's better to just delete the file from S3 by using the AWS Console before uploading it again. We only need to do this once. Now, our code will automatically add the Content-Type for any new file that we upload.

Once you've deleted the file, re-run the client just as shown in the last section, that is, upload the file with the new Content-Type and try to download the file in a browser again. If all goes well, then the image will be displayed.

Downloading a file through the S3 API is similar to uploading it. We simply take the bucket name, the S3 object name and the local filename again but issue a GET request instead of a PUT request, and then write the data received to disk.

Add the following function to your program, underneath the upload_file() function:

def download_file(bucket, s3_name, local_path):
    url = 'http://{}.{}/{}'.format(bucket, endpoint, s3_name)
    r = requests.get(url, auth=auth)
    if r.ok:
        open(local_path, 'wb').write(r.content)
        print('Downloaded {} OK'.format(s3_name))
    else:
        xml_pprint(r.text)

Now, run the client and download a file, which you have uploaded previously, by using the following command:

$ python3.4 s3_client.py download_file mybucket.example.com test.jpg ~/test_downloaded.jpg
Downloaded test.jpg OK

Printing all of the XML is too much for an error message. There's a lot of extraneous information which isn't useful to us. It would be better if we could just pull out the useful parts of the error message and display them.

Well, ElementTree gives us some powerful tools for extracting such information from XML. We're going back to XML for a while to explore these tools a little.

First we need to open an interactive Python shell, and then generate the aforementioned error message again by using the following command:

>>> import requests
>>> import requests_aws4auth
>>> auth = requests_aws4auth.AWS4Auth('<ID>', '', 'eu-west-1', '')
>>> r = requests.get('http://s3.eu-west-1.amazonaws.com', auth=auth)

You'll need to replace <ID> with your AWS access ID. Print out r.text to make sure that you get an error message, which is similar to the one that we generated earlier.

Now, we can explore our XML. Convert the XML text into an ElementTree tree. A handy function for doing this is:

>>> import xml.etree.ElementTree as ET
>>> root = ET.fromstring(r.text)

We now have an ElementTree instance, with root as the root element.

The simplest way of navigating the tree is by using the elements as iterators. Try doing the following:

>>> for element in root:
...     print('Tag: ' + element.tag)
Tag: Code
Tag: Message
Tag: AWSAccessKeyId
Tag: StringToSign
Tag: SignatureProvided
...

Iterating over root returns each of its child elements, and then we print out the tag of an element by using the tag attribute.

We can apply a filter to the tags that we iterate over by using the following command:

>>> for element in root.findall('Message'):
...     print(element.tag + ': ' + element.text)
Message: The request signature we calculated does not match the signature you provided. Check your key and signing method.

Here, we have used the findall() method of the root element. This method will provide us with a list of all the direct children of the root element that match the specified tag, which in this case is <Message>.

And this will solve our problem of just extracting the text of the error message. Now, let's update our error handling.

We can go back and add this to our s3_client.py file, but let's include a little more information in the output, and structure the code to allow re-use. Add the following function to the file underneath the download_file() function:

def handle_error(response):
    output = 'Status code: {}
'.format(response.status_code)
    root = ET.fromstring(response.text)
    code =  root.find('Code').text
    output += 'Error code: {}
'.format(code)
    message = root.find('Message').text
    output += 'Message: {}
'.format(message)
    print(output)

You'll notice that we have used a new function here, namely, root.find(). This works in the same way as findall() except that it only returns the first matching element, as opposed to a list of all matching elements.

Then, replace each instance of xml_pprint(r.text) in your file with handle_error(r) and then run the client again with the incorrect access secret. Now, you will see a more informative error message:

$ python3.4 s3_client.py create_bucket failbucket.example.com
Status code: 403
Error code: SignatureDoesNotMatch
Message: The request signature we calculated does not match the signature you provided. Check your key and signing method.