Cloud computing is the principal technology provider for any big data solution from small to mid-sized applications, although it is also available for huge big data needs with different cloud features.
In the previous chapter, we saw some usage of Microsoft Azure. Let's see how to develop solutions using such technologies.
Within the Microsoft Azure offering, we will focus on the Service Bus sub-offering. Within this category, we can find technologies such as the Relay (a WCF router service for cross-premise communication), Queue and Topics (one-to-one or one-to-many messaging queues), ACS (a federated security service), Event Hub (a multistream-like event router similar to a queue), and a centralized and customizable Push Notification service.
To use a Topic within Microsoft Azure, we must create a Service Bus and get an access key for the specific Topic. To create a Service Bus item within Microsoft Azure, following the wizard on the website will suffice. Once created, the connection information can be found through the button on the lower part of the page with the name Connection Information.
The Service Bus page of the Microsoft Azure web portal
In the information popup, we will find the full connection string:
Connection information popup of a Service Bus account
Once we have the Service Bus name and the full connection string, we simply need to add a name to a Topic (always following the wizard). Here's an example of how to do this:
//topic connection string
static string connectionString = "[CONNECTIONSTRING]";
//the topic name
static string topicName = "[TOPICNAME]";
static void Main(string[] args)
{
//a topic client
var client = TopicClient.CreateFromConnectionString(connectionString, topicName);
StartSubscription1Async();
StartSubscription2Async();
while (true)
{
Console.Clear();
Console.WriteLine("Write a message");
var message = Console.ReadLine();
//send the message to the topic
client.Send(new BrokeredMessage(message));
Console.WriteLine("RETURN TO CONTINUE");
Console.ReadLine();
}
}
//a helper for handling all subscribers
private static Task SubcribeToTopicAsync(string subscriptionName, Action<BrokeredMessage> messageHandler)
{
return Task.Factory.StartNew(() =>
{
//the subscription client
var client = SubscriptionClient.CreateFromConnectionString(connectionString, topicName, subscriptionName);
//always dequeue messages...
while (true)
{
//try dequeue message from subscription's inner queue
var msg = client.Receive(TimeSpan.FromSeconds(3));
if (msg != null)
//handle message with subscription specific logic
messageHandler(msg);
}
}, TaskCreationOptions.LongRunning);
}
private static void StartSubscription2Async()
{
var subscription = "sub1";
SubcribeToTopicAsync(subscription, msg =>
{
//read message body containing user message
var body = msg.GetBody<string>();
//some subscriber specific logic
Thread.SpinWait(1000000);
Console.WriteLine("{0} -> {1}", SubscriptionClient.FormatSubscriptionPath(topicName, subscription), body);
//once completed, the message is flagged as completed
//otherwise the message will flow again in the next
//message subscriber available
msg.Complete();
});
}
private static void StartSubscription1Async()
{
var subscription = "sub2";
SubcribeToTopicAsync(subscription, msg =>
{
//read message body containing user message
var body = msg.GetBody<string>();
//some subscriber specific logic
Thread.SpinWait(2000000);
Console.WriteLine("{0} -> {1}", SubscriptionClient.FormatSubscriptionPath(topicName, subscription), body);
//once completed, the message is flagged as completed
//otherwise the message will flow again in the next
//message subscriber available
msg.Complete();
});
}Note
For more information on Bus Topic, please visit http://azure.microsoft.com/it-it/documentation/articles/service-bus-dotnet-how-to-use-topics-subscriptions/.
The AppFabric cache is available for the Windows Server operating system. In Azure, the same caching engine is available within the cloud services offering or as a Managed Cache Service, where we can buy a portion of a cache in a pay-per-use way. The API is always the same.
//cache provider
//will use the "default" configured cache
//within the .config file
var cache = new DataCache();
var key = "time";
while (true)
{
//retrieve cached value
var value = (DateTime?)cache[key];
//if not available, drill the cache with a new value
if (value == null)
{
value = DateTime.Now;
//adds the value to the cache with a TTL of 5 seconds
cache.Add(key, value, TimeSpan.FromSeconds(5));
}
Console.WriteLine(" {0}", value);
//slow down
Thread.Sleep(1000);
}Using the AppFabric API is easy. A lot of configuration is available, but is not mandatory.
A great benefit in using such an API is the ability to put anything that is serializable within the cache. Another great benefit is the local cache feature that adds an in-memory cache within the running executable memory, for example, as the MemoryCache class does from the System.Runtime.Caching namespace.
When enabled, this local cache becomes a primary-level cache that reduces the usage of the cloud cache, which acts as a second-level cache. This dual caching reduces the network usage of the remote cache service, also reducing the latency considerably.
Note
To know more about creating cache for Azure Managed Cache Service, refer to https://msdn.microsoft.com/en-us/library/dn448831.aspx.
Redis was born as an object-oriented NoSQL database. Later, because of the high throughput and reduced latency offered by the Redis database—because of its intrinsic architecture of being an in-memory database—Redis became a cache provider that is widely used across multiple programming languages and platforms.
Unlike the easiness of the AppFabric cache, Redis has a verbose API with the ability to do several things that are unavailable within the AppFabric SDK, such as locking an item or using a cache item, such as the Interlocked class. By default, the Redis API for .NET serializes only strings and primitive numbers or binary data. Date and time is not supported. Here is an example:
//redis connection string
var cs = "[URL],ssl=true,password=[PWD]";
//cache connector
var client = ConnectionMultiplexer.Connect(cs);
//cache proxy
var cache = client.GetDatabase();
var key = "time";
while (true)
{
//retrieve cached value
var value = (string)cache.StringGet(key);
//if not available, drill the cache with a new value
if (value == null)
{
value = DateTime.Now.ToString();
//adds the value to the cache with a TTL of 5 seconds
cache.StringSet(key, value, TimeSpan.FromSeconds(5));
}
Console.WriteLine(" {0}", value);
//slow down
Thread.Sleep(1000);
}The two biggest drawbacks of the Redis cache are the missing support for built-in serialization of custom types and the unavailability of a built-in local cache.
Caching must happen at the layer/tier boundary. Sharing cache data is important; this is why distributed cache systems exist. However, avoiding wastage of network usage is important. A second-level cache is mandatory when dealing with highly accessed data. Within .NET, the help to create such a local cache comes with the MemoryCache class, the following is an example:
//local cache
var local = new MemoryCache("local");
//redis connection string
var cs = "[URL],ssl=true,password=[PWD]";
//cache connector
var client = ConnectionMultiplexer.Connect(cs);
//cache proxy
var cache = client.GetDatabase();
var key = "time";
while (true)
{
//retrieve value from local cache
string value = (string)local[key];
if (value != null)
Console.WriteLine("local -> {0}", value);
//if unavailable retrieve if from
//second level cache from cloud
else if (value == null)
{
value = (string)cache.StringGet(key);
if (value != null)
Console.WriteLine("redis -> {0}", value);
}
//if not available, drill both caches with a new value
if (value == null)
{
value = DateTime.Now.ToString();
Console.WriteLine("new -> {0}", value);
//as an example we will use different TTLs
cache.StringSet(key, value, TimeSpan.FromSeconds(8));
local.Add(key, value, new CacheItemPolicy { AbsoluteExpiration = DateTime.Now.AddSeconds(5) });
}
//slow down
Thread.Sleep(1000);
}This example produces the following result:
new -> 22/03/2015 17:50:18 local -> 22/03/2015 17:50:18 local -> 22/03/2015 17:50:18 local -> 22/03/2015 17:50:18 local -> 22/03/2015 17:50:18 redis -> 22/03/2015 17:50:18 redis -> 22/03/2015 17:50:18 new -> 22/03/2015 17:50:26 local -> 22/03/2015 17:50:26 local -> 22/03/2015 17:50:26
Carefully use multilevel cache because of the increased difficulty in synchronizing cache items' expiration.
Note
A complete guide on using the Azure Redis cache is available at http://azure.microsoft.com/en-us/documentation/articles/cache-dotnet-how-to-use-azure-redis-cache/.