In this chapter, we will look at Integrated Caching and Compression, two HTTP features that offer enormous performance benefits and help with loading pages faster. In the caching section, we will look at:
- The various caching terms employed (these aren't NetScaler specific)
- How cache policies are evaluated on the NetScaler
- What the exchange between a client and NetScaler looks like
- Guidance on what kind of content to cache and not cache
- How to monitor and troubleshoot caching
In the compression section, we will look at:
- Guidance on what content should and shouldn't be compressed
- What a compression-related exchange looks like
- How to monitor compression
- Troubleshooting considerations for compression
A good majority of content on the web is fairly static. Granted, the current nature of the web means that new content is created all the time, but it doesn't change with every request and some items, like the branding on a company's homepage, stay unchanged for years. Integrated Caching is an HTTP feature on the NetScaler that allows potentially massive savings on bandwidth and server CPU utilization by storing a copy of previously requested objects in its RAM and serving them to users over and over again as long as it is considered valid.
Let's look at some caching-related terms that we will use in this chapter:
- Origin server: This is the server in the backend that generates and owns the original content
- Cache hit: This means that a request that has landed on the NetScaler can be responded to with an object from the cache
- Cache miss: The request has to be sent to the origin server
- Revalidation: This is the NetScaler going back to the server to verify that the content is still fresh
- Invalidation: This is the action of marking cached content as stale
Understanding Integrated Caching on the NetScaler is in many ways the same as understanding what RFC2616 (The RFC for HTTP) says.
As such, it makes sense to understand the headers involved and what content should and shouldn't be stored.
Caching or more accurately web caching is an HTTP feature as we discussed. As such, it relies on certain HTTP headers to function correctly. This feature only works well because these headers and their behavior are described well in the RFC. Let's look at some of the headers involved. As you start to troubleshoot, these headers play a definitive role in determining whether caching is or isn't working as planned and what device, if any, is at fault. These headers are:
Via: This is the most useful header when it comes to verifying that the NetScaler is participating in caching. This is of course configurable to help you determine from which of your several NetScalers the cached responses are originating:
If-Modified-Since: This is a request side header that the client uses to ask that a cache, like the NetScaler provide the full object if it has changed since the indicated date. In a large percentage of cases (talking about static objects here) those objects wouldn't have changed, which will result in a much smaller HTTP 304 response Not Modified instead of the much larger object.Cache-Control: This is a header that carries directives that must be obeyed by all caches between the User and the server. They allow the server to control whether the object can be cached or alternatively modified along the way.
The main options are:
no-cache: This means NetScaler should check the objects validity with server before serving hte object from cache.no-store: This option ensures that no intermediate devices (including NetScaler) cache the object. This is how sensitive items such as cookies should be treated.Private/public: This is used to indicate whether content is meant for a specific User or can be served to all users.no-transform: This tells intermediate caches not to transform (for example, compress) responses on their way to the User.max-age: The content must be revalidated if it is older than the indicated max-age.must-revalidate: Enforces revalidation strictly. It requires the cache to go back to check with the server as soon as it determines that the server-set expiry has reached (stalein caching terms). Proxy-revalidate is the same except that it's meant for proxies as is the case with the NetScaler.
There are five possible actions for a cache policy. Apart from the obvious ones, CACHE, NOCACHE, and INVALIDATION, there are MAY_CACHE and MAY_NOCACHE. These are conditional and help to extend the decision making to include response side processing. To help digest this processing flow, let's take a look at a flow chart:
Let's compare a response coming from the server with one that's being served from the NetScaler cache. Fiddler is an excellent tool for troubleshooting HTTP. Check it out at http://www.telerik.com/fiddler:
Here are the observations we can make from the preceding screenshot:
- There is an
Ageheader, which implies that the response came from a cache as only caches calculate and update this header's value. - There is an
Entity Tag(ETag) header that every cache (the ones in browsers included) relies on to validate cached objects. AnETagheader is a hash or a fingerprint of an HTTP object. A change inETagmeans that the cached object has changed at the server and hence should no longer be served from cache. NetScaler only inserts anETagheader if the server already didn't. - There is the
Viaheader we talked about earlier which helps identify which cache the request is coming from. This is a configurable value on NetScaler.
You could also add other Cache-Control directives when serving cached content from the NetScaler using the Configure Cache Control section that you will find within each content group:
Generally idempotent requests, that is, those that don't change the state of the server are cacheable (for example GET). The following are items you should not cache as they are generally meant for individual users:
Cache-Control: no-cacheCache-Control: no-storeCache-Control: private items
Cookies should not be stored; having them served to multiple users could be a security disaster. Imagine users being shown each other's account. Thus, you will see that this setting to remove cookies is enabled by default. The NetScaler will for example, cache images but remove the cookies associated with them:
PUT, POST, and DELETE must always go back to the server and they should result in invalidation of any cached data on intermediate caches.
Nonstorable HTTP response codes, as listed in the Citrix documentation, are as follows:
- 201, 202, 204, 205, 206 status codes
- All 4xx codes, except 403, 404, and 410
- 5xx status codes
- Responses that contain the
Authorizationheader should not be cached
NetScaler's caching behavior is entirely driven by policies. Hence, looking at the default policies is the best way to get an understanding of its default caching behavior. All the policies that you see starting with an underscore are default policies. These default policies being preconfigured on the NetScaler is of benefit as it saves the Administrator the time needed to mull over the HTTP RFCs to try and identify what is and what isn't a good practice.
NetScaler Default Policies:
It is possible to deactivate these should you wish to, by unbinding them from the policy group, but it is not advised to do so without a very clear reason.
The NetScaler is largely RFC 2616 compliant. There are some deviations chosen considering industry practices for optimization but these are very minor. Two such deviations are:
- User attempts to bypass cache (using
Cache-Control:no-cacheandmax-age=0) are ignored by default. This is to preventDoSattempts by an attacker. - Content with
Varyheader is not cached except for compressed responses. The NetScaler caches images by default, dropping any cookies if present.
Dynamic content, that is, objects that change frequently, such as news, stocks, and weather updates, need extra care while handling. In theory, dynamic content is not to be cached at all and you will have servers setting such content with Cache-Control set to no-cache, no-store, and must-revalidate to discourage caching. While this helps content to always stay fresh, there are often performance and server offload benefits to be had from caching some of the dynamic content, albeit for a very short period of time.
It is very important when you are caching such dynamic content that you understand what part of it is User specific; anything that is User specific is strictly a no-no. Anything that can be normalized, for example, by dropping User specific info like cookies, can be cached.
The following requirements are crucial when caching dynamic content:
- A solid understanding of how frequently the content changes.
- An understanding of what changes between requests, ID, or name while the rest of the response data remains the same. This type of data is called parameterized data and once identified correctly, you can cache one object per unique ID. Such as one for
UserID=1and one forUserID=2. The way to handle such requests is by using cache selectors (hit parameters is also an option, but selectors are recommended).
show cache stats –detail is an excellent command to understand how the hit ratio is at the current moment. Ideally, you would want to bring that as close as possible to 100 percent without breaking security or the application itself.
Lower down in the output, you will be able to see what percentage of hits are 304 versus non-304. The HTTP header code 304 is sent in the response instead of a full (larger) response when a client indicates that they already have a certain object and are only checking whether there is a newer copy.
The images in the following table have been taken from a trace that compares how the two differ:
- For non 304 hit:
- For 304 hit:
You can see that they both come from the cache; that is evident from the Via header. Also, the object is the same in both cases going by the ETag value. The difference is that in the case of the 304 hit, the client helps to conserve bandwidth by requesting (conditionally, through the If-Modified-Since header) that the full response be given only if the object has changed since Sun, 19 Apr 2015 11:05:34 GMT. The bandwidth saved in this case is only 689 bytes, but we can easily see how this could be beneficial when it happens at scale. The caching term for Last-Modified, which is saved as a property of the object, is a validator.
Tip
When troubleshooting caching, I find it very useful to look at how the stats are changing over a few seconds of time; this is a good way to evaluate the new policy you've just put in place, or a setting you've just changed on the Content Group.
For this, you can use the –ntimes option, in the following example, where we ask for statistics to be shown three times (there will always be a 7 second difference between each output). It is handy to use Grep to make it easier to notice changes:
> show cache stats -detail -ntimes 3 | grep 304
You can get a list of all objects by using the show cache object, which will help verify whether objects are getting cached in general or perhaps even search for a specific object. You can then take a closer look at the object by using its locator:
When working with objects already in the cache, you might notice that there's two ways of dealing with an object that you no longer want to serve or consider stale. Flush directly deletes the object from cache. Expire would allow that object to be served but not before it's revalidated with the Server.
Flash cache is invaluable in the right situation and dangerous otherwise, so it needs a mention. An example of a right situation is when a large online event (for example, the Super Bowl) is expected to cause a sudden big surge in traffic that might place a hard-to-handle amount of load on the servers. The users are referred to as being part of a flash crowd (in caching terms). When Flash cache is enabled, the NetScaler forwards the first request to the server and queues the rest. Once the response is received, this is fanned out to all the users in the queue. So the performance benefit is obvious. The reason you need to be careful though is that even if the response is strictly not to be cached or shared, it is still served served to all users in the queue.
When troubleshooting caching, try to use a lightweight client that doesn't have its own cache; WFetch is an excellent example of one such tool:
- The first troubleshooting step, especially if you find that caching is not working at all is the
show cacheparameter to see if memory is still at its default of0, in which case NetScaler cannot cache any objects:
The Max memory limit shows as 794 MB; the NetScaler makes this calculation based on the total amount of memory available, setting aside memory that it needs to perform its usual functions. Also, where possible, always reboot NetScaler after making memory-related changes, especially when de-allocating memory assigned to cache. The de-allocation only happens at boot time.
- Secondly, verify whether the necessary policies are getting hit. Start with a simple enough rule (I commonly use
http.REQ.IS_VALID) to ensure there is a hit in any case before getting more specific. - Focus on any time differences between the server, NetScaler, and client. The object cannot be served from cache if its age is past max-age. Here's one way to find if that is what's happening.
- Check whether you are hitting any memory allocation failures. These are an indication that what you have allocated in step one is not sufficient.
- Check whether the size of the object is larger than the max size configured on the Content Group. The default is only 80 K, which is fine for most deployments, but if your site has high resolution images, this default limit will easily get saturated.
- Check whether there are any marker objects. In the following screenshot, the object,
listheader_selected.png, is in the cache, however it isn't getting served from the cache yet as it is a marker object.
We can determine this by examining the object using its locator. The reason for it being a marker object is that it hasn't been requested often enough, as the
Reasonclarifies.