Every OpenStack Object Storage cluster has a limit on how large the uploaded objects can be. Usually that limit is set to 5 GB, though each cluster can have its own limit. However, this doesn't mean that you are limited to uploading only 5 GB or smaller objects to OpenStack Object Storage. Swift provides large object support via already configured and deployed middleware by splitting up large objects into smaller parts. There are two types of large object support: dynamic and static.
Ensure that you are logged on to a correctly configured OpenStack client and can access the OpenStack environment as a user with the swiftoperator privileges. We will use the developer user created in the Common OpenStack identity tasks recipe in Chapter 2, The OpenStack Client with the cookbook4 password. We also granted this user with the swiftoperator privileges.
Since the OpenStack CLI does not provide all the functionality required through the individual OpenStack project client, we will need to use the Swift CLI for this recipe. Ensure that you have the Swift command-line client installed. If you do not, install it:
pip install python-swiftclient
To upload large objects to OpenStack Object Storage container, follow these steps:
- First, we can list available containers with the following command:
swift listThis will give a list of the containers we have:
books - To upload a large file, we will specify the segment size of the objects in bytes with the
–Sflag:swift upload -S 25000 books nova.txtThis will show the object being split into segments in the following output:
nova.txt segment 1 nova.txt segment 0 nova.txt segment 5 nova.txt segment 2 nova.txt segment 3 nova.txt segment 4 nova.txt
In this example, the segment size is 25000 bytes, and our 128 K
nova.txtfile was split into 6 segments before being uploaded to the Swift cluster. - Verify that the file has been uploaded by listing the available objects in the container:
swift list booksThis shows our
nova.txtfile available:nova.txtEven though the file was split into 6 parts for uploading, it appears as one file in Swift's
bookscontainer. The segments are listed separately in a new container. - To view the available containers, issue this command:
swift listThis gives us our containers with a new one automatically created:
books books_segments
You will notice that a new
books_segmentscontainer has been created automatically. - List objects in the
books_segmentscontainer to view individual objects:swift list books_segmentsThis shows our segmented file objects:
nova.txt/1512361523.073429/131024/25000/00000000 nova.txt/1512361523.073429/131024/25000/00000001 nova.txt/1512361523.073429/131024/25000/00000002 nova.txt/1512361523.073429/131024/25000/00000003 nova.txt/1512361523.073429/131024/25000/00000004 nova.txt/1512361523.073429/131024/25000/00000005
- View the details of the uploaded object in the
bookscontainer using thestatcommand:swift stat books nova.txtThis gives a number of details about our
nova.txtfile in thebookscontainer:Account: AUTH_402e8fe274c143ea91fe905a1b8c7614 Container: books Object: nova.txt Content Type: text/plain Content Length: 131024 Last Modified: Sun, 17 Dec 2017 06:36:32 GMT ETag: "cd0ef1b80a6261f0b6b7db9efa739938" Manifest: books_segments/nova.txt/1512361523.073429/131024/25000/Meta Mtime: 1512361523.073429 Accept-Ranges: bytes X-Timestamp: 1513492591.13264 X-Trans-Id: tx4fa6e7a4e53a459a86430-005a3611ad X-Openstack-Request-Id: tx4fa6e7a4e53a459a86430-005a3611ad
Take a look at the
manifestfield in the information provided. Themanifestfield will include thecontainerdetails that was created for individual segments, the original file size, and segment size.
Since the OpenStack CLI does not provide large file upload support at the time of this recipe's writing, we will use the Swift command-line client. In order to store large files in OpenStack Object Storage, objects have to be split up or segmented before uploading to a container. This segmentation is done for us by specifying the -S flag in the swift upload command:
swift upload -S size_in_bytes container large_object
When uploading large files, Swift automatically creates a new container and pseudo-folders for individual segments of the uploaded object. Individual segments may be listed the same way as regular objects; however, do not manipulate them directly.