C H A P T E R  4

Windows Azure Storage Part II – Queues

The Windows Azure Queue service is an Internet-scale message queuing system for cross-service communications. Even though the service is called a queue, the messages aren't guaranteed to follow the First In First Out (FIFO) pattern. The design focus of the Queue service is on providing a highly scalable and available asynchronous message communication system that's accessible anywhere, anytime. The Queue service is not a replacement for your on-premises Microsoft Message Queuing (MSMQ), because it lacks some of the features like transactional messaging, distributed transactions, and integration with domain security. But, most of the applications that do not use these MSMQ features should be able to replace it by the Queue service with minor modifications. The Queue service provides a REST API for applications to use the large-scale Queue service infrastructure. If you want to build an application that is agnostic to the type of queuing system it uses, you should build an abstraction layer using interface contracts and then let the implementation decide the type of queue. In real-world programming, you should always start with interfaces whether you have multiple implementations or not.

The Queue service is scoped at the account level. So, when you create an account on the Azure Services Developer Portal, you get access to the Queue service. Figure 4-1 illustrates the Management Portal page for the storage account that I created in the previous chapter and URL endpoint for the Queue service.

Figure 4-1. Queue endpoint URL

The account endpoint for the Queue service is <account name>.queue.core.windows.net, where <account name> is the unique name you created for your storage account. The secret key associated with the account provides security for accessing the storage account. You can use the secret key to create a Hash-based Message Authentication Code (HMAC) SHA256 signature for each request. The storage server uses the signature to authenticate the request.

Queue Limitations and Constraints

Even though the Queue service provides a scalable and highly available infrastructure for asynchronous message communications in the cloud, it has some limitations and constraints that are important to understand before diving deep into architecture and programming. The limitations of the Queue service are as follows:

• The Queue service supports an unlimited number of messages, but individual messages in the Queue service can't be more than 64KB in size. If your object is larger than 64KB, you can store the actual object in Blob or Table storage and just send the link as a queue message.
• The FIFO behavior of the messages sent to the Queue service isn't guaranteed.
• Messages can be received in any order.
• The Queue service doesn't offer guaranteed-once delivery. This means a message may be received more than once.
• Messages sent to the Queue service can be in either text or binary format, but received messages are always in the Base64 encoded format.
• The expiration time for messages stored in the Queue service is seven days. After seven days, the messages are garbage-collected. Therefore, don't use Queue service as a permanent storage solution. Use Blob or Table storage for permanent storage.

Queue Service Architecture

The Queue service architecture consists of a three-level hierarchy: accounts, queues, and messages, as shown in Figure 4-2.

Figure 4-2. Queue service architecture

Your Windows Azure storage account is the entry point to the Queue service via the REST API.

Windows Azure Storage Account

The URI scheme for accessing the Queue service via your storage account is

where <account name> is the unique name you created for your storage account. The <account name> must be globally unique.

For example, the Queue service for the storage account that I created in the previous chapter can be referenced as

Queues

A queue is a logical destination for sending messages. There can be any number of queues in an account in the Queue service. A queue stores messages and makes them available to applications via the REST API. Queues can have metadata in the form of name-value pairs up to 8KB in size per queue. The Queue service support only private access; that means you need to have account privileges in order to access queues in a Queue service.

You can access a queue using the URI

where <queue name> is the name of the queue you want to access.

For example, if you create a queue named logsqueue in the proazurestorage account, you can reference it using the following URI:

The naming constraints for a queue are as follows:¹

• The queue name must be a valid DNS name.
• Queue names must be unique within an account.
• Queue names must start with a letter or a number.
• Container names can't contain any special characters other than the dash (-) character.
• The dash (-) character must be immediately followed by a character or a number.
• All the characters in the queue name must be lowercase.
• Queue names can't be less than 3 or more than 63 characters in length.

If a queue name or the URI violates the naming convention, an HTTP status code 400 (Bad Request) is returned by the server.

Messages

Messages are stored in queues. There is no limit to the number of messages that can be stored in a queue, but the size of each individual message can't exceed 8KB. To communicate large object messages, you can put the large object in a Blob and then send the URI of that object as a message to a queue.

When you send a message, it can be in either text or binary format; but when you receive a message from the queue, it's always in Base64-encoded format. A GUID MessageID assigned by the Queue service uniquely identifies a message in the queue.

A message has the following attributes:

__________

• MessageID – Uniquely identifies a message in a queue and is created by the Queue service when you send the message to the Queue service.
• PopReceipt – An attribute of the message used for deleting or popping the message out from the queue.
• Visibilitytimeout – An integer value representing the visibility of the message in seconds after it's received by a receiving application. The default visibilitytimeout value is 30 seconds, which means that after a message is received, it remains invisible to other applications for 30 seconds (unless it's deleted by the receiving application). The maximum visibilitytimeout value is two hours. The visibilitytimeout value is passed to the Queue service while a message is being retrieved from a queue. From the message receiver, you can control the visibility timeout in such a way that processing of the message is completed within the visibility timeout period, which includes deleting the message from the queue. If the processing is not complete and message not explicitly deleted within the visibility timeout period, the system will assume that the processing failed and the message will be visible again in the queue for other receivers to process.
• Messagettl – An integer value representing the time-to-live value in seconds. When you send a message to a queue, you can specify the messagettl, which instructs the Queue service to keep the message only for the specified number of seconds. The default value for messagettl is seven days. That means if you don't specify a messagettl value, the Queue service keeps the message for seven days before it's garbage-collected.

You can access messages in a queue using this URI

where <queue name> is the unique name of the queue within the scope of the account specified in the URI, and messages is a constant string representing all the messages in the specified queue. For example, if you create a queue named logsqueue, you can get messages from it by calling the following URI:

REST API

The REST API for the Queue service is available at the account, queue, and message levels. In this section, you learn about the Queue service REST API with specific examples. You also learn to interact with the Queue service programmatically, and you explore the queue methods in the available storage client libraries.

The REST API enables you to make HTTP calls to the Queue service and its resources. REST is an HTTP-based protocol that lets you specify the URI of the resource as well as the function you want to execute on the resource. Every REST call involves an HTTP request to the storage service and an HTTP response from the storage service.

Request

The Queue service REST API's HTTP request components are described in the following sections.

HTTP Verb

The HTTP verb represents the action or operation you can execute on the resource indicated in the URI. The Queue service REST API supports the following verbs: GET, PUT, POST, HEAD, and DELETE. Each verb behaves differently when executed on a different resource.

Request URI

The request URI represents the URI of a resource you're interested in accessing or executing a function on. Example resources in the Queue service include accounts, queues, and messages. An example URI for creating a queue named logsqueue in an account named proazurestorage is

The HTTP verb PUT instructs the service to create the queue, and the URI points to the resource that needs to be created.

URI Parameters

The URI parameters are the extra parameters you specify to fine-tune your operation execution. They may include operation parameters or filter parameters for the results. In the Queue service API, the URI parameters depend on the type of resource and the HTTP verb used. For example, a URI for retrieving a list of queues from an account looks like this:

The HTTP verb GET instructs the Queue service to retrieve results, and the parameter ?comp=list specifies that the data requested is a list of queues.

Request Headers

Request headers follow the standard HTTP 1.1 name-value pair format. Depending on the type of request, the header may contain security, date/time, metadata, or instructions embedded as name-value pairs. In the Storage service REST API, the request header must include the authorization information and a Coordinated Universal Time (UTC) timestamp for the request. The timestamp can be in the form of either an HTTP/HTTPS Date header or the x-ms-Date header.

The authorization header format is as follows:

Where SharedKey|SharedKeyLite is the authentication scheme, <Account Name> is the storage service account name, and <Signature> is an HMAC of the request computed using the SHA256 algorithm and then encoded by using Base64 encoding.

To create the signature, follow these steps:

1. Create the signature string for signing. The signature string for the Storage service request consists of the following format:
   VERB
Content - MD5
Content - Type
Date
CanonicalizedHeaders
CanonicalizedResource

   where VERB is the uppercase HTTP verb such as GET, PUT, and so on; Content — MD5 is the MD5 hash of the request content; CanonicalizedHeaders is the portion of the signature string created using a series of steps described in the “Authentication Schemes” section of the Windows Azure SDK documentation (http://msdn.microsoft.com/en-us/library/dd179428.aspx); and CanonicalizedResource is the storage service resource in the request URI. The CanonicalizedResource string is also constructed using a series of steps described in the “Authentication Schemes” section of the Windows Azure SDK documentation.

2. Use the System.Security.Cryptography.HMACSHA256.ComputeHash() method to compute the SHA256 HMAC-encoded string.
3. Use the System.Convert.ToBase64String() method to convert the encoded signature to Base64 format.

Listing 4-1 shows an example request header that sets the metadata values of a queue.

In Listing 4-1, the request header consists of x-ms-date, x-ms-version, x-ms-[name]:[value], and Authorization values. x-ms-date represents the UTC timestamp, and x-ms-version specifies the version of the storage service API you're using. x-ms-version isn't a required parameter, but if you don't specify, you have to make sure the operation you're calling is available in the default version of the Queue service. Before making the REST call, be sure you match the operation you're calling with the API version it's supported in. It's always safe to match the operation with the version to get the expected results. The x-ms-meta values represent the queue metadata name-value pairs the operation should set. The last header value is the Authorization SharedKey used by the Storage service to authenticate and authorize the caller.

Request Body

The request body consists of the contents of the request operation. Some operations require a request body and some don't. For example, the Put Message operation request body consists of the message data in XML format, whereas the Get Messages operation requires an empty request body.

Response

The HTTP response of the Queue service API typically includes the following components.

Status Code

The status code is the HTTP status code that indicates the success or failure of the request. The most common status codes for the Queue service API are 200 (OK), 201 (Created), 204 (No Content), 400 (BadRequest), 404 (NotFound), and 409 (Conflict).

Response Headers

The response headers include all the standard HTTP 1.1 headers plus any operation-specific headers returned by the Queue service. The x-ms-request-id response header uniquely identifies a request. Listing 4-2 shows an example response header for a List Queues operation.

Response Body

The response body consists of data returned by the operation. This data is specific to each operation. For example, the List Queues operation returns the list of queues in an account, whereas the Get Messages operation returns the messages in a queue. Listing 4-3 shows an example of the response body for a List Queues operation. The response contains four queues.

Storage Client API

Even though the REST API and the operations in the REST API are easily readable, the API doesn't automatically create client stubs like the ones created by WDSL-based web services. You have to create your own client API and stubs for REST API operations. The Windows Azure SDK team has created a client helper managed code library: Microsoft.WindowsAzure.StorageClient from Windows Azure SDK. Behind the scenes, the client API invokes the REST APIs of the Windows Azure Queue Storage service. The Microsoft.WindowsAzure.StorageClient library abstracts this by providing a closed-source interface and therefore is easier for developers to extend it and use it directly from your code.

In the following sections, I will cover the class structure and calling mechanisms from the Microsoft.WindowsAzure.StorageClient assembly.

Windows Azure Storage Client Queue API

The Microsoft.WindowsAzure.StorageClient namespace consists of classes representing the entire queue hierarchy. Figure 4-3 illustrates the core classes for programming Queue service applications.

Figure 4-3. Queue class hierarchy

As shown in Figure 4-3, four core classes are required for queue operations. Table 4-1 provides a short description of each of them.

In addition to these core classes, classes like QueueAttributes and QueueErrorCodeStrings represent more details about the queue.

The steps for programming simple queue applications with the queue classes listed in Table 4-1 are as follows:

1. Add the following using statement to your C# class:
   using Microsoft.WindowsAzure.StorageClient;
2. Instantiate the CloudStorageAccount class from the configuration file:
   CloudStorageAccount  storageAccountInfo =
CloudStorageAccount.FromConfigurationSetting(configurationSettingName);
3. Or, instantiate the CloudStorageAccount class using account information:
   CloudStorageAccount  storageAccountInfo = new CloudStorageAccount(new
StorageCredentialsAccountAndKey(accountName, accountKey), new Uri(blobEndpointURI), new
Uri(queueEndpointURI), new Uri(tableEndpointURI));
4. Create an instance of CloudQueueClient:
   CloudQueueClient queueStorageType = storageAccountInfo. CreateCloudQueueClient ();

When you have an instance of the CloudQueueClient class, you can execute operations on the queue storage service as follows:

List queues:

Add a message:

Get messages:

Peek messages:

Delete a message:

Set queue metadata:

The call to SetMetadata() method calls the method on the queue service API in the cloud.

In the next few sections, you learn how to call some of these functions at every level of the Queue service hierarchy.

Account Operations

The storage account provides an entry point to the Queue service via the Queue service endpoint URI. At the account level of the hierarchy, the Queue service supports only one operation: List Queues. The URI of a specific account is of the format <account name>.queue.core.windows.net. Table 4-2 describes the List Queues operation, and Table 4-3 lists some important characteristics of the List Queues function.

<account name> is the storage account name, such as proazurestorage; and <devstorageaccount> is the account name for the development storage. The HTTP verb used in this operation is GET. The table lists the URI format for accessing the cloud Queue service as well as the development storage URI. Port 10001 is the default Queue service port in the development fabric.

The URI for the List Queues operation supports additional optional parameters, as listed in Table 4-4.

The sample REST request for List Queues in raw format looks like Listing 4-4.

The characteristics of the REST request in Listing 4-4 are as follows:

• The parameter comp=list at the account level of the Queue service yields the list of all the queues.
• The prefix=test filters the results by queue names starting with “test.”
• The maxresults=50 returns 50 queues or less.
• The x-ms-date is the UTC timestamp of the request.

* The Authorization header contains the SharedKey of the request.

• The Host header points to the Queue service in the cloud.

Because the request is sending a maxresults parameter, it makes sense to keep the HTTP connection alive, because it's highly likely that the user will retrieve the next set of results by making another call to the Queue service.

Listing 4-5 shows the response for the List Queues request.

In Listing 4-5, the header consists of the HTTP status (200 OK) indicating the success of the operation. The response body is in XML format with <EnumerationResults /> as the root element. The <Queues /> element contains the retrieved queues. The Queue element encompasses queue attributes like the queue name and the queue URI. An empty <NextMarker /> element indicates that all the results have been retrieved.

To help you understand the Queue service programming model, open the Windows Azure Storage Operations project from Ch4Solution.sln. The project consists of a Windows form and uses the StorageClient project from the same solution for making calls to all the Windows Azure storage. The StorageClient project is shipped with the Windows Azure SDK. I also created a helper class named WAStorageHelper in the ProAzureCommonLib project for wrapping the StorageClient methods. Figure 4-4 shows the user interface for the Windows Azure Storage Operations application as it pertains to the Operations account of the Queue service.

Figure 4-4. Windows Azure storage Queue service account operations

In Figure 4-4, the top Account section displays the account name and SharedKey of the storage account. When the Windows Azure Storage Operations.exe application starts, it loads the account information from the configuration file. Listing 4-6 shows the account configuration in the project's app.config file.

The StorageAccountConnectionString is loaded when the application starts and displays in the textbox. Before starting the application, make sure you modify this connection string to point to your storage account, except when you are using development storage. The QueueStorageEndpoint is the URI of the Queue service.

The WAStorageHelper class in ProAzureCommonLib project has a ListQueue() method for retrieving queue names. Listing 4-7 shows the code for ListQueues() method.

In Listing 4-7, the ListQueues() method calls the ListQueues() method on the QueueClient object, which is of type CloudQueueClient from the StorageClient assembly. The first method returns all the queues in an account or filtered based on the prefix. Figure 4-5 illustrate the execution of the ListQueues operation in the Windows Azure Storage Operations application.

Figure 4-5. List Queues operation

When you click the List Queues button, the application retrieves queues from the Queue service and displays the names of queues in the Queues ListBox. In the parameter section, you can specify the prefix.

Queue Operations

Queues support several operations, as listed in Table 4-5.

Table 4-6 lists some of the important characteristics of the queue operations listed in Table 4-5.

Table 4-6 lists the HTTP verb, cloud URI, development storage URI, HTTP version, and access control for the queues. The <account name> is the storage account name in the cloud, and the <devstorageaccount> is the development storage account. Observe that unlike blob containers, all the operations can be called only with the account owner privileges.

The following sections discuss some of the operations from Table 4-7 in detail. Even though the operations are different, the programming concepts behind them are similar. To keep the book at a conceptual level, I discuss just the Create Queue and Set Queue Metadata operations. By studying these operations in detail, you can understand the programming concepts behind all the queue operations. The Windows Azure Storage Operations application included with this chapter's source code contains an implementation of all the queue operations.

Create Queue

The Create Queue operation creates a queue in a storage account. The URI for the Create Queue operation is of the format account name>.queue.core.windows.net/<queue name>. You can think of Queue as a message queuing system in the cloud. For example, if you want to send and receive messages across diverse applications in different domains, Windows Azure Queue may fit your requirement. Because of its standard REST interface and Internet scale, you can send and receive queue messages anywhere, anytime, and in any programming language that supports Internet programming. The Create Queue REST request looks like Listing 4-8.

Listing 4-8 shows the request for creating a queue named myfirstazurequeue. The PUT HTTP verb instructs the Queue service to create a queue. There is no metadata information for the queue, so the queue is created without any metadata. You can add x-ms-meta-[name]:[value] to the header to create metadata values. For the Create Queue operation, the Queue service responds with a status code of HTTP/1.1 201 Created, or HTTP/1.1 409 Conflict if a queue with the same name already exists. The Create Queue response is shown in Listing 4-9.

In Listing 4-9, the first line represents the status code of the operation. The x-ms-request-id represents a unique request identifier that can be used for debugging or tracing.

Figure 4-6 shows the working of the Create Queue operation in the Windows Azure Storage Operations application.

Figure 4-6. Create Queue from Windows Azure Storage Operations.exe

As shown in Figure 4-6, to create a queue, you need to do the following:

1. Go to the Queue Operations tab.
2. Enter a queue name (such as myfirstazurequeue) in the Queue Name text field.
3. Select the Create Queue operation from the Operations list box.
4. Click the Execute button. After the queue is created, the queues list box in the Account section is refreshed with the newly created queue name in it.

To help you understand the programming model of the Create Queue operation, open the Visual Studio Solution Chapter4.sln from the Chapter 4 source directory. The WAStorageHelper class in the ProAzureCommonLib contains a helper function called CreateQueue, as shown in Listing 4-10.

The CreateQueue() method calls the GetQueueReference() method to get a reference to the CloudQueue object. The CloudQueue object is a local instance of the Queue object and may not represent a queue that already exists. This instance doesn't create a queue when you instantiate it. To create a queue, you have to call the CreateQueue() method on the CloudQueue object explicitly. The CloudQueue object creates the accurate URI and metadata headers for calling the Queue service.

Under the hood, the API uses System.Net.HttpWebRequest to send the REST message over HTTP. Upon success or failure of the operation, the Queue service returns an HTTP status code: HTTP/1.1 201 for success or HTTP/1.1 409 for conflict or failure. The CreateQueue() method translates the HTTP status code into true for success and false for failure or conflict. The Boolean value is passed all the way to the Windows Azure Storage Operations application as a return parameter of the CreateQueue() method.

Set Queue Metadata

Queues can contain name-value pairs of metadata values. You can store values like the time of creation, creator, last modified by user, and so on in the metadata fields of a queue. The size of the metadata can be 8KB per queue. The Set Queue Metadata operation sets the metadata of a queue independently. The URI for the Set Queue Metadata operation is of the format account name>.queue.core.windows.net/<queue name>?comp=metadata. The Set Queue Metadata REST request looks like Listing 4-11.

In Listing 4-11, the HTTP verb used is PUT, and the URI parameter is ?comp=metadata. This parameter instructs the Queue service to set the queue metadata instead of creating the queue. The Create Queue operation doesn't have this parameter. The x-ms-meta.[name]:[value] entries represent the metadata name-value pairs you want to set on the queue.

Figure 4-7 illustrates how to execute the Set Queue Metadata operation in Windows Azure Storage Operations application.

Figure 4-7. Set Queue Metadata in the Windows Azure Storage Operations application

As shown in Figure 4-7, to execute the Set Queue Metadata operation, you do the following:

1. Go to the Queue Operations tab.
2. In the Account section, click the List Queues button to get a list of queues in your account.
3. Select one of the queues from the list (such as myfirstazurequeue).
4. Make sure the Queue Name text box in the Queues section displays the name of the selected queue.
5. In the Queues section, select the Set Queue Metadata operation from list of queue operations.
6. In the Queues section, enter metadata name-value pairs in the Metadata section.
7. Click the Execute button to execute the operation.
8. To verify the success of the operation, click the Clear Results button in the Queues section, and re-select the queue from the Queues list in the Account section to retrieve the newly set metadata values.

To help you understand the programming model of the Set Queue Metadata operation, open the Visual Studio Solution Chapter4.sln from the Chapter 4 source directory. The WAStorageHelper class in the ProAzureCommonLib contains a helper function called SetQueueMetadata(), as shown in Listing 4-12.

In Listing 4-12, the System.Collections.Specialized.NameValueCollection represents the metadata name-value pairs. The queue name is used to create a local instance of the CloudQueueQueue object. The code then clears all the metadata and adds new metadata to the queue. Note that you need to call SetMetadata() operation to actually commit the metadata changes.

The SetMetadata() method creates the REST message and sends it synchronously to the Windows Azure Queue service to set the queue metadata values. It uses the System.Net.HttpWebRequest to send the REST message over HTTP. Upon success or failure of the operation, the Windows Azure Queue service returns an HTTP status code: HTTP/1.1 200 for success or HTTP/1.1 204 (No content

Message Operations

Messages support several operations, as listed in Table 4-7.

Table 4-8 lists some of the important characteristics of the message operations.

The <account name> is the storage account name in the cloud, and the <devstorageaccount> is the development storage account. The <queue name> is the name of the queue in which messages are stored. The following sections discuss some of the operations from Table 4-8 in detail. Even though the operations are different, the programming concepts behind them are similar. To keep the book at a conceptual level, I discuss just the Put Message and Get Messages operations. By studying these two operations in detail, you can understand the programming concepts behind all the message operations. The Windows Azure Storage Operations application included with this chapter's source code contains implementations of most of the message operations.

Put Message

The Put Message operation en-queues (puts) a message at the end of the queue. The URI of a Put Message operation is of the format account name>.queue.core.windows.net/<queue name>/messages. You can send a message with size up to 8KB. To send larger files, you can save the message as a blob and send the URI of the blob to the queue. The body of the message while sending can be text or binary, but it should support inclusion in an XML body with UTF-8 encoding. This is because a message received from the queue is always returned in Base64-encoded format within an XML response body. You see this in the Get Messages operation. The URI for the Put Message operation supports an additional optional parameter, listed in Table 4-9.

The Put Message REST request looks like Listing 4-13.

In Listing 4-13, a string message “myfirstazuremessage” is sent to the queue named myfirstazurequeue. The time-to-live seconds for the message is 120, which means if the message isn't received or deleted by an application within 120 seconds in the queue, the message will be marked for deletion and won't be visible to any applications. The request body consists of the message content wrapped in the <QueueMessage> element. Note that the content of the message within the <MessageText /> element is in Base64-encoded format. Listing 4-14 shows the response from the Queue service.

As shown in Listing 4-14, the Queue service responds with an HTTP/1.1 201 Created status code for a successful Put Message operation. Figure 4-8 shows the working of the Put Message operation in the Windows Azure Storage Operations application.

Figure 4-8. Put Message in Windows Azure Storage Operations.exe

As illustrated in Figure 4-8, you can send a text message using the Windows Azure Storage Operations application. The steps for sending a message to a queue are as follows:

1. Create a new queue (called myfirstazurequeue).
2. Select the new queue from the Queues List Box in the Accounts section.
3. Add some text to the Message Body text box in the Queues section.
4. Select the Put Message operation from the Operations text box.
5. Make sure the Queue Name text box is populated with the selected queue name.
6. Optionally, you can specify the time-to-live in the “Time to live (secs)” text box.
7. Click the Execute button to execute the Put Message operation.

To help you understand the programming model of the Put Message operation, open the Visual Studio Solution Chapter4.sln from the Chapter 4 source directory. The WindowsAzureStorage.cs file in the Windows Azure Storage Operations project consists of a PutMessage() method, as shown in Listing 4-15.

The AddMessage() method of the CloudQueue object creates the REST message request and sends it synchronously to the Queue service. It uses the System.Net.HttpWebRequest to send the REST message over HTTP. Upon the success of the operation, the Queue service returns an HTTP status code: HTTP/1.1 201 Created.

Get Messages

In the previous section, you learned to send messages to queues in the Queue service. In this section, you learn to retrieve these messages using the Get Messages operation. The URI for the Get Messages operation is of the format account name>.queue.core.windows.net/<queue name>/messages. The URI for the Get Messages operation supports additional optional parameters, as listed in Table 4-10.

Listing 4-16 shows the REST API request for the Get Messages operation.

In Listing 4-16, the URI points to the myfirstazurequeue queue. numofmessages=10 instructs the Queue service to retrieve only 10 messages. visibilitytimeout=60 instructs the Queue service to make the retrieved messages invisible to other applications for 60 seconds, unless the receiving application deletes them. Listing 4-17 shows the REST API response from the Queue service for the Get Messages operation.

Listing 4-17 shows the HTTP header and body of the Get Messages operation response. For the sake of brevity, only three messages are shown. The HTTP response body consists of a list of messages in XML format. Every <QueueMessage /> element represents a message. When you retrieve a message, the MessageId and the PopReceipt properties of the message are important for deletion purposes. The recommended pattern is to receive the message, process it, and then delete it before it becomes visible to other applications when the visibilitytimeout period expires. The TimeNextVisible value specifies the expiration time of the visibilitytimeout period. The ExpirationTime specifies the time when the message will be marked for deletion if not retrieved and/or deleted by a receiving application. This value was set when the message was sent to the queue. Figure 4-9 shows the working of the Get Messages operation in the Windows Azure Storage Operations.exe application.

Figure 4-9. Get Messages in the Windows Azure Storage Operations application

As illustrated in Figure 4-9, you can use the Get Messages operation for a queue using the Windows Azure Storage Operations application. The steps for retrieving messages are as follows:

1. Select a queue (such as myfirstazurequeue) that already contains some messages.
2. In the Queues section, select the Get Messages operation.
3. Click the Execute button to get a list of messages from the selected queue.
4. Optionally, you can specify the Number of Messages and Visibility Timeout in the Parameters section.

The retrieved messages are populated in the DataGridView control in the Messages section. Each message is represented by a row in the DataGridView control. The control displays all the properties of the retrieved messages. To delete a message, select a row in the DataGridView and press the Delete button on your keyboard.

To help you understand the programming model of the Get Messages operation, open the Visual Studio Solution Chapter4.sln from the Chapter 4 source directory. The WAStorageHelper.cs file in the Windows Azure Storage Operations project consists of two overloaded GetMessages() methods, as shown in Listing 4-18.

The numofmessages parameter represents the number of messages to retrieve. The visibility timeout represents the length of time these retrieved messages will be invisible to other clients. If you don't delete these messages in the visibilitytimeout period specified, these messages will be read by other clients or the same client when it tries to read again. The visibility timeout parameter is used in making sure the message is processed at least once when multiple clients are accessing the same queue. If the processing of a message fails, then it will be automatically read by other clients. If the processing of the message succeeds, it must be deleted by the client processing the message. In order to avoid processing of the message multiple times, you need to make sure the visibilitytimeout period is longer than the message processing time.

Unlike MSMQ, the Microsoft.WindowsAzure.StorageClient API does not provide any queue listener events. But, the event objects in .NET Framework enables you to build your own. In the ProAzureCommonLib project, I have created an event class MessageReceivedEventArgs, an event handler delegate MessageReceivedEventHandler and a listener class QueueListener that defines the MessageReceived event. See Listing 4-10.

Figure 4-10. Custom QueueListener

Figure 4-10 illustrates class diagram for the event hander and QueueListener. The client class can implement the MessageReceived event to receive messages from the Queue service.

Listing 4-19 shows the usage of the QueueListener class for receiving a MessageReceived event whenever a new message arrives in the specified queue.

Asynchronous API

Until now, I have covered only synchronous methods for calling Queue service. In a real-world application, I recommend using asynchronous API instead of synchronous because in asynchronous method invocations, you are not blocking the calling thread and therefore the chances of getting a deadlock are limited. Especially in scenarios where the managed API (in this case, the Storage Client API) is making asynchronous calls to the service. The Storage Client API makes asynchronous REST calls to the Windows Azure Queue service and waits on the same thread for the response. If your synchronous call is waiting for the call to return on a thread and the asynchronous call is waiting on the ThreadPool to release a thread, there is a deadlock because your synchronous call will not return until the REST asynchronous call from within the API returns and the REST asynchronous call will not return because all the threads in the ThreadPool are exhausted.

As a workaround to this issue, and a best practice anyways, I recommend you to use asynchronous methods in the Storage Client API wherever possible. In stateless web applications, it involves a bit more work, because the request thread is synchronous, but the efforts in building asynchronous calling mechanisms in such applications will definitely pay off in terms of scalability. The Storage Client API for Queue Service consists of asynchronous methods for most of the operations. Listing 4-20 shows a pattern for invoking the asynchronous methods BeginAddMessage() and BeginGetMessage(). You can use the same pattern for invoking all the asynchronous methods in the Storage Client API, including Blob and Table storage.

In both the methods, note that I am creating a manual event that will be reset using the evt.Set() method after the operations is complete. I am not using the IAsyncCallback object's WaitOne() method to wait because when I use a lambda expression, the reset automatically happens even before the lambda expression code segment gets executed. If you are using a separate function to end the asynchronous method call instead of a lambda expression, you don't need to manually set the event. The above mentioned code pattern can be reused in all the asynchronous method calls in the StorageClient library.

Now that you understand Windows Azure Storage Queue service, let's look at some common scenarios in which Queues are used.

Queue Scenarios

In the previous sections, you saw the details of working with the Windows Azure Queue service. This section covers some of the basic application communication scenarios that can use the Windows Azure Queue service.

Scenario 1: Windows Azure Web and Worker Role Communications

Consider a scenario in which you're designing an ecommerce web application in Windows Azure with a Web role front end and several Worker roles for back-end processing work. The Web role instances continuously send purchase order information to the Worker roles for order processing. In this scenario, you can use the Windows Azure Queue service to queue purchase order messages for the Worker roles, as shown in Figure 4-11.

Figure 4-11. Web role/Worker role communication using the Queue service

In Figure 4-11, Web role instances 1 and 2 send orders to the order-processing queue. Worker Roles 1 and 2 dequeue the order messages and process the orders. Because not all orders have to be processed immediately, Worker roles can pick up from the queue only the orders that are ready to be processed. This way, you can create an effective message communication system between Web roles and Worker roles, taking advantage of the scalable and highly available Queue service infrastructure. If the order message size exceeds 8KB, you can store the message body in the Blob service and pass a link to the blob as a queue message, as shown in Figure 4-11. When the Worker role dequeues the message, it can retrieve the contents of the order from the Blob service.

Scenario 2: Worker Role Load Distribution

Continuing Scenario 1, depending on the volume of messages, you can either adjust the number of queues or the number of instances of Worker roles for processing orders. For example, if you identify during your testing phase that one Worker role can process only ten orders at a time, you can configure Worker roles to pick up only ten messages from the queue. If the number of messages in the queue keeps increasing beyond the number that Worker roles can process, you can create more instances of Worker roles on demand and increase the order-processing capacity. Similarly, if the queue is under-utilized, you can reduce the Worker role instances for processing orders.

In this scenario, the Queue service plays the role of capacity indicator. You can think of the queues in the Queue service as indicators of the system's processing capacity. You can also use this pattern to process scientific calculations and perform business analysis. Figure 4-12 illustrates the Worker role load-distribution scenario.

Figure 4-12. Worker role load distribution

In Figure 4-12, Worker Roles 1 through 3 can process average order loads. When the number of orders backs up into the queue, you can spawn more Worker roles (4 through n) depending on demand and the need for overall order-processing capacity.

Scenario 3: Interoperable Messaging

Large enterprises use applications from different vendors, and these applications seldom interoperate with each other. An enterprise may end up buying an expensive third-party tool that acts as the interoperability bridge between these applications. Instead, the enterprise could use the Queue service to send messages across the applications that don't interoperate with each other naturally. The Queue service exposes a REST API based on open standards. Any programming language or application capable of Internet programming can send and receive messages from the Windows Azure Queue service using the REST API. Figure 4-13 illustrates the use of the Queue service to interoperate between a Java-based Sales application and a .NET-based CRM application.

Figure 4-13. Interoperable messaging

Scenario 4: Guaranteed Processing

In Scenario 1, every order needs guaranteed processing. Any loss in orders can cause financial damage to the company. So, the Worker roles and the Queue service must make sure every order in the queue is processed. You can implement guaranteed processing using the following four simple principles:

• Set the visibilitytimeout parameter to a value large enough to last beyond the average processing time for the messages.
• Set the visibilitytimeout parameter to a value small enough to make the message visible if message processing fails in a consumer (Worker role) or a consumer crashes.
• Don't delete a message until it's processed completely.
• Design the message consumers (Worker roles) to be idempotent (that is, they should account for handling the same message multiple times without an adverse effect on the application's business logic).

Figure 4-14 illustrates guaranteed message processing in the context of the order-processing example discussed in Scenario 1.

Figure 4-14. Guaranteed processing

In Figure 4-14, two Web roles create orders, and three Worker roles process orders. Consider the following steps:

1. Worker Role 1 reads order O1 for processing. Worker roles typically take 15 seconds to process an order. The visibilitytimeout for messages is set to 60 seconds.
2. Worker Role 1 starts processing order O1. At this point, O1 isn't visible to other Worker roles for 60 seconds.
3. Worker Role 1 crashes after 10 seconds.
4. After 60 seconds, O1 becomes visible again because Worker Role 1 wasn't able to delete it.
5. Worker Role 2 reads O1 and processes it.
6. After processing is complete, Worker Role 2 deletes O1 from the queue.

The important points to note here are that Worker Role 1 didn't delete the message from the queue before processing was complete, and the visibilitytimeout was set to an appropriate time window to exceed the processing time of an order. This pattern is commonly used in batch processing systems.

Summary

The Queue service provides a scalable and highly available store and delivery mechanism for exchanging messages across distributed applications. It provides reliable message delivery from message producers to message consumers.

Don't expect the performance of the Queue service to match your on-premises message brokers like MSMQ or ServiceBroker, because of its reliance on HTTP REST protocol. The Queue service exposes a REST API, making it easily accessible across multiple platforms and programming languages. In this chapter, you saw some of the important operations and scenarios for using the Queue service. The next chapter covers Windows Azure tables.

Bibliography

MSDN. (n.d.). ADO.NET Data Services Specification. Retrieved from MSDN Developer's Network: http://msdn.microsoft.com/en-us/library/cc668808.aspx.

MSDN. (2009, May). Windows Azure Blob — Programming Blob Storage. Retrieved from MSDN: http://go.microsoft.com/fwlink/?LinkId=153400.

MSDN. (2009, May). Windows Azure Queue — Programming Queue Storage. Retrieved from MSDN: http://go.microsoft.com/fwlink/?LinkId=153402.

MSDN. (2009, May). Windows Azure SDK. Retrieved from MSDN: http://msdn.microsoft.com/en-us/library/dd179367.aspx.

MSDN. (2009, May). Windows Azure Table — Programming Table Storage. Retrieved from MSDN: http://go.microsoft.com/fwlink/?LinkId=153401.