Redundancy, Resiliency, and Reliance

These three concepts shouldn't be new to you now. Take a moment to go over them in your head and then see whether what you have learned so far matches the descriptions that follow:

• Redundancy   Redundancy has to do with having multiple copies of your system configurations, source code, and data in many places. This is important in the case of mid- to long-term outages and, in some cases, helpful in recovery in the case of permanent data loss. To implement redundancies in Azure, concepts such as Availability Zones, fault and update domains, failover database instances and ZRS and GRS storage options should all be considered.
• Resiliency   Resiliency has to do with the next section that discusses recovering from transient issues automatically. How well does your application respond and recover from failures? One pattern discussed later, the Retry pattern, will wait for a few seconds if an error is received when attempting to execute code and then try it again. That is a much more resilient experience than simply throwing an exception stating, “Something unexpected happened; please try again later.”
• Reliance   The experiences users, customers, employees, or clients have with the application determine how much they can rely on the system. If they lose faith in it, meaning they don't know if the system will be working or not when they need it, you have failed in making the system resilient because it is not reliable. Reliance is the outcome of a system that is resilient and redundant.

Self and Automatic Healing

The components used in the cloud are commoditized. Their lifespan may not be what you would expect from a trademarked brand of computer components. When a commoditized memory module, for example, fails, it is simply replaced instead of trying to find out why it failed and fixing it. Simply replacing it is much faster than performing root-cause analysis and fixing it, and the latter really isn't possible at cloud speed anyway. When there is a failure of a hardware component, there will be a disruption. How large of a disruption is dependent on how heavily your application depends on the component that failed. Regardless, you need to place code into your applications to manage these kinds of short disruptions. Making sure that your transactions are ACID compliant (atomic, consistent, isolated, and durable) will help your application respond better in these situations.

If your software fails for some reason, make sure to catch the exception. This results in a handled exception and provides an interface for responding and retrying the execution. If you do not catch exceptions and one happens (an unhandled exception), it has the likelihood of crashing the process. When a process such as W3WP.exe crashes, it is possible that it will not be respawned automatically. It may require a reboot or manual restart to get all the bits realigned. There are platform features on Azure that monitor the health of the operating system and sometimes the application itself. When the OS or application appears to be unhealthy, the platform monitor may trigger a reimage of the VM on which your application is (or was) running. Simply, your application must be able to handle sudden stoppages in the execution of your transactions. It will happen, and if you don't manage it, your application will not be reliant!

Scaling and Decoupling

One of the primary features in the cloud and on Azure is the ability to provision and scale on demand. You can use large amounts of compute when required and then deallocate the computer when the demand diminishes. Your application needs to be able to work in that scenario, for example, if your web application requires state to be maintained throughout an operation. When a customer is placing an order online, then the process is usually fulfilled across numerous pages and numerous steps. Do any of these steps store some of the information on the server? If yes, then you need to make sure requests are sent back to the same server each time if you have multiple servers running. This can be achieved by a context called client affinity or sticky sessions. Those simply make sure that once a session is started on a specific server, all additional requests route to the same server for that session. There are scenarios in which your scaling is automated and will scale in or out the number of instances your workload is consuming. However, there is some built-in intelligence that waits some time before reimaging and returning the VM to the pool. Long-running tasks, like a WebJob, a batch job, or a background job, might get shut down in midprocess. You need to code for that by perhaps having some startup logic in the background job to check that everything is OK before proceeding with normal operations. The logic is application-specific. Making sure everything is OK could be as simple as having a processing status column on a row in a database and checking that status at startup. If the status shows “in process” for a long period, the code likely failed, and the data needs reprocessing.

The next topic discussed in this section covers decoupling and has a lot to do with the messaging services discussed in Chapter 6. It also has to do with removing long-running or resource-intensive jobs like a WebJob, batch job, or another background job to an offline service. For example, the multitiered (n-tiered) architecture style is a tightly coupled process. In that scenario, the client would wait for the request to the web server to flow through the application server and to the database server and then back before proceeding further. One of the problems with that architecture is that a successful response is dependent on the availability of all three of those tiers. If you could somehow decouple that flow by implementing the web-queue-worker model, then a request from a client wouldn't be dependent on the availability and resiliency of many systems. Your application would appear more reliant.

Use SaaS, PaaS, and IaaS, in That Order

You would likely agree that the fewer tasks you are directly responsible for, the higher probability of being successful in the areas where your responsibilities increase. In other words, if you can focus your efforts more directly on what you feel is the most important and be successful at those few things, then overall you will be more successful. If you can offload the management of your network infrastructure, the configuration, and the maintenance of your computer hardware, as well as the administration of the operating system, then by all means do that. If all you need is the product, then take that one too. The point is, if you can run your workload on PaaS, choose that one over IaaS. There is less to worry about. In reality, if you are like me, you can always find things to worry about. Even if you have only two tasks, not having to worry about network, hardware, and operating system administration is great. Then you can worry about just the important aspects of your work.

Design for Change

Things change, and sometimes they change fast. Can you recall any mention of the cloud five or ten years ago? Think about where we were then and what we have now. Designing for change is focused on, but not limited to, factors such as decoupling, asynchronous models, and microservice/serverless concepts. In short, the more you can isolate a specific piece of logic from the whole, the easier it is to change. Recall the discussion about NoSQL databases like Azure Cosmos DB and how quickly you can get a container to store data into it. Contrast that with a relational database where the tables and columns have an applied meaning, and the changes would likely have greater impact. When compared to a nonrelational data source, making a change to a relational model is possible, but it would take much longer and require much more application knowledge.

Retry

The Retry pattern is probably the most important one to implement. You can implement the pattern in all code running on all kinds of platforms, but it is of additional importance when the code is running in the cloud. This is because of the higher probability of transient faults. Historically, there have been patterns that recommend catching exceptions using a code snippet like the following:

try
{ …}
catch (Exception ex)
{…}

The interesting fact about this pattern is that in many cases there is no coded logic added inside the catch expression to perform any kind of recovery activity. I cannot explain this behavior; however, I review a lot of code, and this is the scenario I see often. The more sophisticated applications do at least log the ex.Message that is included in the Exception and maybe provide it back to the client that triggered the transaction, but nothing more. When running in the cloud, transient faults may happen more often when compared to running on-premise, trademarked hardware. Here, you should now implement some code within the catch expression to retry the transactions that triggered the exception. You need to define a retry policy that describes the scenario in which a retry should happen and the actions to take on retry.

The Exception defined in the previous code snippet is considered a catchall exception; however, there are numerous kinds of exceptions. This is not the place to cover all the possible exceptions. One that can be called out is an HttpRequestException . This kind of exception is one that could be successful if you were to try it again after waiting a few seconds. Some pseudocode that defines how a retry would look is shown here:

catch (HttpRequestException ex)
{
    writeLog($"This exception happened {ex.Message}")
    if currentRetryCount < allowedRetryCount && IsTransient(ex)
        await Task.Delay(5000)
        theMethodWhichFailed()
}

The retry policy applied in the previous code snippet requires that:

• The exception be logged
• The retry happen only a specific number of times
• The retry validate the exception to confirm it may be transient
• The retry wait for five seconds and then recall the method again

The example provided here was one that would occur when using the System.Net.Http.HttpClient class. When you use Azure product-specific SDKs for communicating with, for example, AAD, Azure Storage, or Cosmos DB, the SDKs typically have built-in retry capabilities. For a list of products that have built-in retry capabilities, take a look at https://docs.microsoft.com/en-us/azure/architecture/best-practices/retry-service-specific

Gatekeeper

It makes a lot of sense to name things that perform the action it actually performs. The Gatekeeper pattern does exactly as the name implies, like an orchestrator orchestrates and like a programmer inherits attributes from a parent class. A gatekeeper monitors an entry point to some service and makes sure that only those who should have access to it actually do. The location of the gatekeeper is in front of the resource that it protects so that unauthorized entities attempting to access it can be denied access before they are inside or close to being inside. (If it had been up to me, I would have named this Bouncer, which seems a bit cooler, but I guess Gatekeeper is acceptable.)

There are many Azure resources that have a global endpoint that in the given scenario is already too close to the protected resource for comfort. The measurement of “too close” depends greatly on the sensitivity and criticality of the data and application. However, let's assume doing authentication on a service that runs in full trust mode is, for you, too much of a risk. You may know that when you have a process (like W3WP.exe ) running on an operating system, it runs within the context of an identity. Some identities have full access, while other custom identities may have restricted access only to the specific resources required to do a job. In reality, the latter is hard to do, but the most common scenario is somewhere in between with a tendency to provide too much privilege. Therefore, if for some reason the entry point at your front door is compromised, then full access to the house is a relative certainty. It's interesting that I used “front door” there because there is an Azure product named Azure Front Door that can be used to protect endpoints from an architectural perspective. You may be thinking, well, what about service endpoints, NSGs, and IP restrictions? You can achieve the Gatekeeper pattern by using Azure Firewall, a service endpoint, and some other Azure products. As illustrated in Figure 7.2, any Azure product or feature that can act as a firewall is an option for implementing this pattern.

IP restrictions and NSGs (default Azure VM configurations) are typically configured on the resource themselves, which negates the purpose of the pattern. The purpose is to keep the tier that exposes a public endpoint separate from the tier that runs the sensitive or mission-critical processing.

Throttling

There is an HTTPS status code that exists just for this scenario; it's a status code of 429 and means Too Many Requests. This is usually handled by some custom code or third-party application that is monitoring the number of requests to a website for a specific period of time. What throttling does is limit the number of requests or amount of compute resources for a given time period. In Internet Information Services (IIS), it is possible to restrict the amount of consumed CPU for a given time frame. When the threshold is breached, any requests going to that specific process are not processed and instead return an error. This kind of setting can also be useful in a distributed denial-of-service (DDoS) scenario (intentional or not) where a large number of requests are coming from a specific IP address. The feature is called Dynamic IP Restrictions (DIPR), and it counts the number of concurrent connections or the number of total connection in a given time frame from the same IP address. When that restriction is breached, DIPR no longer allows access from that IP address for a configurable amount of time. Throttling is an important pattern that needs some attention, not only preventing it when it happens to your system but also preventing your system from inadvertently doing it to another system.

Sharding

Sharding was covered in many data storage scenarios in Chapter 5. The concept of sharding is the division of large sets of data into smaller, logically structured subsets. Review Figure 5.14 and Figure 5.15, which demonstrate examples of horizontal and vertical sharding. Horizontal sharding may break the data into rows depending on the alphabetical order of the row key. If you know there is an actual division of load across data starting and ending with A–K, L–P, and Q–Z, then it would make sense to configure three horizontal shards to improve performance and stability. On the other hand, if you see no pattern of such data access but instead notice there are three or four columns on each row that are retrieved much more frequently than the others, you can consider a vertical shard and separate the data structure into data columns that are accessed frequently versus a shard with columns that are not accessed as often.

Circuit Breaker

I was trimming my hedges once with an electric trimmer, and somehow the cord got between the blades and the leaves and—boom—it got cut, and the electricity shut off. To get the electricity flowing again, I had to switch the circuit back on. Had the circuit breaker not reacted, there would have been electricity flowing out the end of the electric cord causing damage but effectively doing nothing. When the circuit broke, it automatically caused the stoppage of electricity flow, which happened in less than a second. The Circuit Breaker pattern from a cloud design pattern operates in the same context. When one of the remote resources or services is not available, then the flow of all communications to them stop. You might be thinking that this is the same as the Retry pattern, and you are right—almost. The difference here is that you are certain that the resource or service your application is remotely connecting to is unavailable, and you have no idea when it will be back online. In this scenario, it doesn't make as much sense to retry. In the trimmer scenario, I knew something happened, and I knew trying to use the trimmer wouldn't work, so it was worthless to try. The outage took some time to resolve as I traversed down into the basement and found the switch needing administration. It took about 10 minutes; imagine how many exceptions your application would throw if your IT processes 100 or more transactions per second. Also, imagine how those using the application would feel getting the exceptions over and over again. They would think your system is unreliable. You could make the application more resilient by implementing the Circuit Breaker pattern.

A method for implementing a circuit breaker is to monitor the state of the dependent resources or services. For example, perhaps you look at the Closed, Open, and Half-Open states. If the state is Closed, the dependent resources are functional, just like if an electric circuit is closed, it is all flowing as expected, and there are no breaks in the connectivity. If there is a break, then the state changes to Open and requests that the dependent resource stop. In that scenario, you should have code that displays a temporary out-of-service page or routes and stores the transaction details to a temporary location for offline processing when the service comes back online. Half-Open is the state in which the exception has stopped occurring and the resource is likely back online, but it's in the early stages, so you throttle the number of requests you are passing through. After a given threshold based on a configured time frame is breached, the state is set back to Closed and you are back in business.

Antipatterns

When you are coding and designing your application, keep in mind the antipatterns, listed here:

• Superfluous Fetching
• Not Caching
• Synchronous I/O
• Monolithic Persistence
• Improperly Instantiating Objects

The first one, Superfluous Fetching, has to do with a concept called projection. This means you need to have only the columns that perform your task in the SELECT clause. You will need a very tight WHERE clause, as well. You do not want to retrieve any more data from a data store than is absolutely necessary. The same goes for caching. Don't store more data in a cache than the application needs to run because data loaded into the cache consumes memory. RAM has a cost in the cloud, sometimes a variable one. By all means, cache when you can. Give caching some close analysis and load only the data that is often retrieved from a data source into cache. This not only reduces load on your data store but also decreases the latency of your application since it doesn't have to perform unnecessary queries on the database to retrieve the same data over and over again.

When you are using a library in your programming and there exists asynchronous methods, by all means use them. When you use synchronous methods for performing I/O tasks such as accessing content on a hard disk or performing a task where the thread needs to pause for a response from some service on another server as the thread waits, that thread remains blocked until a response is returned from the I/O task. You must understand that the number of threads a process can generate is finite, and you should take actions where you can to reduce the number of them. Some latency happens when the process needs to create a new thread. When you use asynchronous methods that employ the async/await keywords, which make I/O calls, then the thread is not stuck waiting on the response. It is instead reallocated to the thread pool for other tasks to use. Once the I/O function is complete, the thread is pulled from the pool and is used to complete the action. Use asynchronous coding patterns when possible.

Monolithic Persistence has to do with having all your data stored on the same data store regardless of whether the data is related. When possible, you should separate online transactional processing (OLTP) data stores from data stores that contain logging, telemetry, or historical data used for reporting purposes. The reason for this is that OLTP data stores need to be responsive. They could be impacted if a large annual sales report is run against the same data store that orders are placed into. Separating them would likely have a positive impact on performance.

When you make a connection to a database or to a REST API, part of that process is to instantiate a connection object. Running your application on-premise, where you likely have almost unrestricted outbound connections, the most common pattern creates the connection within a using statement, like the following:

using (var httpClient = new HttpClient())
{
  …
}

This pattern follows a guideline for scoping the use of the httpClient object. As soon as the code execution leaves the closing bracket, the httpClient is marked for garbage collection. The negative aspect of this is that for every invocation of this code block, an object needs to be instantiated and a new connection made. That implementation is improper and will not scale in the cloud, from a performance perspective, but mostly from a connection perspective. Remember from Chapter 3 there was a discussion about SNAT ports. Recognize that there are rather stringent limitations on the number that you can have concurrently open. The previous pattern where the httpClient was instantiated has a high probability of using many SNAT ports under high load. The alternative and recommended best practice creates a static instance in your class and instantiates it in the constructor.

public class CsharpGuitarHttpClientController : ApiController
{
    private static readonly HttpClient httpClient;
    static CsharpGuitarHttpClientController()
    {
        httpClient = new HttpClient();
    }
    public async Task<Product> GetTheCoolestGuitarEver(string id)
    {
        …
    }
}

When you do this, the same connection object is used for all outbound connections to the same server. The pattern is the same when you are making connections to an Azure SQL database, an Azure Cosmos DB, or any system that requires the instantiation of a connection object.

Forms Authentication

Forms Authentication is implemented into an ASP.NET application starting with the installation of the IIS module. Then, its configuration is installed into the web.config file using syntax similar to the following snippet:

<configuration>
    <system.web>
        <authentication mode="Forms"/>
    </system.web>
</configuration>

Figure 7.3 shows that the IIS module for Forms Authentication is installed but not yet enabled.

Enabling the IIS module and the web.config file results in the authentication provider being enabled. The scope of this configuration is for the website running within and underneath the directory structure in direct relation to the location of the physical web.config file.

The reason for the note about where the configuration is applied in relation to the location of the web.config file is that this authentication provider can be enabled on the web server, but it's not used unless you enabled it specifically on a website. In addition to configuring web.config , you must also have an Authentication (AuthN), in which your custom user identity database stores the credentials. Any group associations that define whether the user has access to specific resources within the application once authenticated, would utilize Authorization (AuthZ) to confirm the access. The code required to perform the AuthN and AuthZ procedure must be written by a developer working in the application itself. This is not an “out-of-the-box” authentication solution. It is, however, a solid, sophisticated, and valid approach for implementing security into an internet application.

Certificate Authentication

A certificate is a file that contains an encrypted key or token that is created and administered by a certificate authority (CA). Recall from Chapter 5, which covered cryptography, that you encrypted a column on an Azure SQL database in EXERCISE 5.3. Certificate authentication falls into the same category and is related to cryptography. Something to call out here specifically has to do with public and private keys. Here are the fundamental concepts:

• A private key is used to encrypt data.
• The private key decrypts data encrypted by the public key.

Based on those definitions of public and private keys, it may be obvious that the private key is the one that needs the greatest protection. Making it possible to encrypt data using a key wouldn't cause any harm; however, the ability to decrypt that data and use it is something that needs greater control. Let's not go much deeper into this as it was mentioned before; this area of technology is career worthy and book worthy, but not essential for the exam. Some knowledge here is helpful. Certificate authentication has dependencies on public and private keys because those keys must be validated against the certificate to consider the certificate authentication successful. Look at Figure 7.3. You will see an authentication provider is installed as well. Figure 7.4 illustrates the process of this authentication (recall also from Figure 2.28 that a certificate was called out as a means for validating an identity using MFA).

An interesting point regarding the previous figure is that it isn't always just the server side that is concerned about the identity of the client/user making the connection. You should also be aware and confident that you are connecting to the server that you really intend to. It is easy to scrape a web page from a bank that includes the user ID and password textbox. You enter them on my page, and I store them when you click the submit button and then redirect you to the real bank site so you can reenter them and access. Having this client-server certificate validation is helpful for both parties, not just the server side. The process of client certification is as follows:

1. The client requests a protected resource.
2. The server returns a server certificate.
3. The client validates the certificate with the CA.
4. The client sends a client certificate.
5. The server validates the certificate with the CA.
6. The server grants or denies access based on the validation.

Notice that the client makes a request to the resource before sending the certificate. This makes sense because you wouldn't want to send a certificate with each request unless it is required to do so. This would increase the data transmission with each request. However, there is a feature called uploadReadAheadSize that defines how much of the data body content can be sent along with the certificate from the client in step 4. This improves performance as the data transmission can be optimized by utilizing all the available transmission buffer.

Windows Authentication

Both NTLM and Kerberos were introduced in Chapter 2 when discussing Azure Active Directory Domain Services. Both of those security protocols, NTLM and Kerberos, are included when you enable Windows Authentication for an intranet website. The key point here is that Windows Authentication is used with intranets and not the internet because intranets typically require an Active Directory (AD) and some additional infrastructure to implement. As you can see in Figure 7.3, there is an authentication provider named Windows Authentication aka Integrated Windows Authentication (IWA). If you want to enable IWA, you need to disable Anonymous Authentication to protect your website. Recall from the previous section where you learned that the initial request from the client to the server is an anonymous one. This is because the client doesn't yet know which security protocol the server requires for access to the requested resource. Therefore, if anonymous authentication is enabled, IIS uses that one as the default; once authentication is granted, it doesn't proceed to check for other enabled authentication providers.

Integrated Windows Authentication does support NTLM but mostly uses Kerberos, which is the default as it is more secure. You can see this in the advanced settings for IWA in the IIS Management console, as shown in Figure 7.5.

IWA provides a single sign-on (SSO) solution. When an employee successfully logs into their Windows workstation, they receive a Kerberos token. This token is cached and can again be used when accessing a website without having to log in again. In most cases, the browser that is used to access the intranet site has the code necessary to check whether the website is an intranet and then look and use the Kerberos token automatically. If you are coding a custom client application that is not web-based or doesn't use a browser, then you would need to code this logic into your app manually.

Multifactor Authentication

There isn't much more to add about multifactor authentication (MFA) that hasn't already been covered in Chapter 2 (see Figure 2.28). In summary, MFA is an added level of security that takes place after someone successfully provides a user ID and password, which is something they know. The second layer is based on something they have, such as a certificate, a fingerprint, or a code sent to a mobile device in your possession. These days, you need to strongly consider MFA as a default because the user ID and password security solution has many vulnerabilities and should no longer be solely depended on.

Open Standards

Protocols like OAuth and OpenID are used mostly for delegated authentication of internet-based applications. If you have a Google, Apple, Microsoft, or Facebook account and have clicked a button similar to those shown in Figure 7.6, then you have been authenticated using an open standard authentication protocol.

This protocol delegates the authentication to an identity provider instead of your application using one that is owned and operated by yourself or your company. Figure 7.7 illustrates how authentication works with OAuth, for example.

In this example, you try to access some application that is protected by an identity provider, as shown in Figure 7.6. You select the one that you will use, which triggers a routing of the request to the selected identity provider. If the credentials are not cached locally already, then you are prompted to enter those necessary for the selected provider. If the credentials are cached on your machine already, then they get used. If the credentials are valid, then the identity provider returns a token that must then be validated by the application code to which you want access. In some respects, this is like Forms Authentication without the need to store user credentials in your own data store. You will, however, need to have logic to enforce authorization access, using groups, claims, and permissions.

Managed Identities

Managed Identities (MI) are in many ways a service principle. The difference is that they can be used only with Azure resources. Chapter 2 includes a section about managed identities, so take a look at that. In a majority of cases, credentials usually identify a human individual. Some scenarios let you give an application an identity and grant the application access to the application instead of granting access to individuals. A scenario in which MI is used is to access Azure Key Vault where a connection string or token is stored to access some other resource such as a database or an Event Hub partition. Using Managed Identity can prevent credential leaking since the developer and/or administrator of the application never sees the credentials of the resource that are stored in the Azure Key Vault.

Service Principal Authentication

A service principal is an identity that is used to access a resource but isn't linked to any living being. It is simply an account with a user ID and password that applications can use to make sure whatever is accessing it is allowed to. In the service principal scenario, someone has created the account and will need to manage it. For example, someone must be made responsible for resetting the password and performing other tasks. Doing that may cause some disruption if you are unaware of all the applications using it. You need to notify them all and give them the new password and the date and time, and it will change so they can update the connected systems. In summary, using a service principle requires some management, which is why the abstracted layer around the service principal is called a managed identity. All the creation and administration tasks of the service principal identity are managed for you. The approach to use depends on your needs.