The most interesting problems you may encounter with Nginx or any other piece of complex software (or any other human endeavor, for that matter) are usually firmly situated in the category of "Misc". This dreaded class of troubles contains everything that does not fall into other convenient classes that you as a reader and a professional spent time and efforts in previous chapters. Nginx is known for its stability and robustness, so you might never get a chance to encounter anything we describe here in your professional career. Still, in the spirit of "better safe than sorry", we would highly recommend reading the chapter just in case.
In this chapter, you will learn about the following topics:
- Different security warnings that users may encounter on your websites and how to fix them
- A couple of reasons why users may see very obsolete versions of web pages that were updated and how to fix such cases
- Several curious configuration problems that will help you better understand the inner workings of Nginx and how to solve them
Anyway, let us start from what we consider more frequent and easier to fix and then move to less obvious and much more obscure issues.
It is a fact that the web is moving to HTTPS. Even 3 – 4 years ago, plain text HTTP was a perfectly normal choice unless you build a bank interface. Today, in the age of global personalization and sophisticated attackers, websites are slowly embracing total encryption. There are a number of reasons for that, and deep discussion is out of the scope of this book. Instead, you are kindly referred to this document https://www.eff.org/encrypt-the-web-report. Basically, in the next 2-3 years, HTTP will become de facto deprecated as an end user protocol and that brings us to a world of problems dealing with the public key infrastructure of HTTPS. HTTPS relies on TLS, which uses X.509 PKI with CAs, CRLs, OSCP, and so on. The abundance of abbreviations in the previous sentence is deliberate; this is a complex topic in itself, which regularly confuses the most experienced specialists. The design of the X.509 key and certificate infrastructure is known to be very complex, but the task it solves is not a simple one either. One of the most interesting recent initiatives to simplify the solution is the project Let's Encrypt, which is available at https://letsencrypt.org. They advertise as the free certificate vendor (certification authority, or CA in X.509 lingo), but they also provide a set of protocols, services, and tools which allow painless and transparent certificate management. They are not yet fully operational as of March 2016, so watch that space.
Setting up HTTPS on Nginx is a topic thoroughly described in many articles and books around the web, so we won't spend much time on it. You have probably done it several times.
There are some cases in which your visitors may encounter HTTPS-related security warnings when requesting pages from your website.
Let's say that you have something like this in your website configuration:
listen 443 ssl;
ssl_certificate "/site/example.com/conf/https_public_cert.pem";
ssl_certificate_key "/site/example.com/conf/https_priv.key";When the X.509 infrastructure was invented in the end of 1980s, one of the tasks it tried to solve was the issue of mutual authentication and trust between parties in a secure communication channel. While encryption does not strictly require this kind of authentication, it is still considered important that your browser trusts the server on the other side of an encrypted HTTPS connection at least in the sense of that server presenting some undeniable proof that it is what it claims to be. The implementation of that proof are the digital certificates that servers present while negotiating the connection, and a number of policies on a client. Amusingly, if you are familiar with other PKI schemes, for example, PGP/GPG, then you probably asked yourself why X.509 requires a separate entity in addition to the obvious key pair (public and private), which is actually required to implement asymmetrical cryptography. Well, the idea is that the certificate is a document from a third party about the server, whereas the keys are technical data used during encryption. The loose PGP analog of the certificate are the signatures from other people your keys may or may not have.
The most common certificates HTTPS server use are of Domain Validation (DV) type, and the most important policy that a client will enforce against a connection to a server with a DV certificate is that the domain name mentioned inside the certificate matches the domain name of the actual TCP connection endpoint.
A problem with this policy manifests itself with these nasty scary full-screen errors:
The earlier-mentioned image is from Chromium, the open browser that is the base for Google Chrome. The next example is from Firefox:
The next example is from mobile Safari on iOS:
The root of this problem lies in the historic convention of having one and only one fully qualified domain name per certificate. If you have aliases (and most websites usually do have at least the common pair—one with "www." and one without "www."), you either have to purchase separate certificates or use extensions to the original X.509. Fortunately, those extensions are pretty widely supported; the last problems we remember were with default Windows Phone 7 browsers, and if you have significant number of such clients, you probably know what to do and have resources to solve that problem with a dedicated project.
The extensions you need are: wildcard certificates and multidomain certificates or SAN certificates. Your HTTPS certificates vendor will have those in store. They are usually a bit more expensive but too useful to ignore. Wildcard certificates allow you to request certificates for domain patterns that look like *.example.com, whereas Subject Alternative Names (SANs) are a way to enumerate the list of domains that this certificate is valid for.
The next most common error message that you may encounter is as follows:
Now we will show Chromium and Microsoft Internet Explorer errors as examples:
iOS and Mobile Safari chose the strategy to show one type of HTTPS error message for both of the most common errors.
The philosophy behind digital certificate expiration is rather simple. Because a certificate is an electronic document asserting someone's identity as verified by a third party (that is your certificates vendor or Certificate Authority (CA) in X.509 speak) in the absence of said third party, it should have expiration date to ensure regular re-evaluations of that identity. Vendors will go out of their way to remind you about your certificates, but in spite of this, expirations happen a little too often in the wild.
This is a very embarrassing moment for any system administrator responsible for a web server. Almost as embarrassing as forgetting to renew the domain delegation (this also happens a lot). Most of the monitoring solutions, both standalone and hosted, will have a sensor for that. We even found a specific monitoring tool just for certificate expiration at https://snitch.io/ although we have not had the chance to use it. One of the most robust tools to ensure that your TLS certificates are renewed on time is, surprisingly, Google Calendar. Acquire the habit of always creating an event with a reminder 4 weeks before the expiration date right after you receive each new certificate file. This will save you, we promise.
You may ask how a problematic certificate could end up in production. Well, most of the time, production certificates are not used on development and test deployments because of the old convention mentioned earlier—having one domain name per certificate. Because this limit is long obsolete, you may include your test hostnames inside certificates and that will remove one additional difference between your production and test (or stage, depending on your release life cycle) environments.
There are hundreds of different Certificate Authorities (CAs) now; some of them worked for many years and some are just starting. If you had time to get familiar with X.509 PKI, you will know that clients should have the root certificate for each and every issuer (that is, CA) in their trusted storage. Otherwise, all the server certificates issued by that particular CA will be invalid. In an ideal world, all user computers and mobile devices have a very up-to-date list of trusted root certificates. Unfortunately, an ideal world does not exist, and you may face a real problem when some of the more conservative clients' browsers consider your web servers insecure because they do not yet have the relevant CA root certificate in their trusted storage.
To work around such problems, your CA may provide you with their own certificate that you should concatenate with your own and present to the clients as a certificate set (or chain). This will work because X.509 supports a certificate chain to verify the endpoint certificate. Your certificate will refer to the CA certificate, which if provided will refer the client further down the chain until one of the referred intermediate CA certificates is found in the trusted root certificate storage of the client.
The easiest way to do this is with a simple cat command:
$ cat your_example_domain.pem CA.pem > certs.pem
Then, specify the path to this compound file in your ssl_certificate directive.
Although the two types of HTTPS problems we demonstrated earlier are true errors and web browsers will actively prevent users from working around them, there are some less critical problems that still may compromise the trust of your clients.
One of the most massive and, at the same time, elusive warnings is the so-called "Mixed content" warning. The idea is that any HTTPS page should never embed HTTP objects because the overall security level of the page is the level of its least secure component. If you have even one image object that was fetched via plaintext HTTP, then it may compromise the whole page and even the whole application at times. All modern browsers give warnings about this situation although the actual interface implementation may vary.
These are some examples. The first one is from a desktop version of Firefox:
The second example is from a desktop version of Chromium:
Also, modern browsers make a distinction between active and passive content, the former including scripts, active objects, such as Flash, style sheets, and whole external documents as IFrames. Active mixed content is usually blocked outright, whereas passive mixed content only issues warnings. There is a recommendation from W3C about "Mixed content" that contains all the best practices and recommendations about handling the issue. You may well use it as a guide about what to expect. Refer to http://www.w3.org/TR/mixed-content/.
Linking insecurely to your own content is an obvious error on the side of the developers. They should never ever use direct absolute links with the http: scheme anymore. Fixing this may be quite easy using a couple of global full-text searches through all the documents and templates. Just ensure that all your secondary hosts are reachable via HTTPS and change http:// in links to either https:// or just // if you feel like saving a few bytes per link by using a clever trick of schemeless (or schemaless) URLs. A URL without a scheme looks like the following:
<img src="//img.example.com/images/face.png">
The idea exploits a rarely used but required by all standards feature of relative URL resolution. This is also the reason for the second name: protocol-relative. URLs with two slashes will inherit the scheme from the base document. Refer to RFC3986 http://tools.ietf.org/html/rfc3986#section-4.2 for more information.
As a workaround solution (that may become semi-permanent), you may use the power of Nginx to help your developers change all links to internal resources using on-the-fly substitution. Nginx source contains a special module named ngx_http_sub_module, which is not usually built by default although it depends on the author of the Nginx package in your distribution. To check whether it is available, run this command:
$ nginx -V 2>&1 | fgrep -c http_sub_module 1 $
If you see 1, then your Nginx is linked with sub_module. Otherwise, you need to compile it using --with-http_sub_module parameter to configure.
This is the example that you will need to modify for your own situation:
location /path {
sub_filter_last_modified on;
subs_filter_types text/css;
sub_filter ' src="http://img.example.com' ' src="https://img.example.com';
sub_filter ' href="http://img.example.com' ' href="https://img.example.com';
}The second line with sub_filter_types directive is only needed if your CSS files contain absolute URLs of images. It is as dirty a hack as many sub_module applications are, but it may solve at least some of the immediate problems you have. Remember to include all your assets hosts.
There are two main sources of insecure external content your website may contain. The first includes external trackers, advertisement networks, commenting systems, and the like. In 2016, all of these have the support for HTTPS websites. The only reason they may the mixed content warnings is incorrect embedding code (for example, very old).
The other source of insecure objects is User-generated Content (UGC). If your website has a way for users to post some data that may be displayed in the context of your pages afterwards, then you might have this problem. Examples include commenting systems, blogs, forums, messaging, and so on. This is not as rare as it might seem at first thought.
One way to find the culprits of the mixed content violation is using the browser console. Recent browsers display warnings about which objects are insecure. There are also tools for crawling websites and identifying insecure embeds, but these may not be relied upon, especially if you have a complex website that may not be easily crawled. Refer to, for example, https://www.whynopadlock.com/ or https://www.jitbit.com/sslcheck/.
Mozilla provides a good page on Mixed Content too. You are very welcome to consult it at https://developer.mozilla.org/en-US/docs/Security/MixedContent.
While fixing the embedding code of external components is rather straightforward, dealing with UGC content is much harder. Suppose that you have a way for your users to specify their image avatars by entering URLs pointing to those images. You cannot just change the URL from http: to https: because this may just break the link. You cannot be sure that all those far-away hosts support and will always support HTTPS. The only way to provide such a service for your own users is to serve all that remote content yourself by proxying it.
This is an important hack that involves some of the less popular Nginx magic and requires collaboration with your developer team, but in the end, you will have a very efficient proxy for the external content. Brace yourself.
Here is an example of a simple but relatively secure proxy for external images. It may be extended to other types of content with ease.
The relevant part of Nginx configuration looks like this:
server_name proxy.example.com;
location @1.gif {
empty_gif;
}
location / {
proxy_cache ext_images_cache;
proxy_cache_valid 200 48h;
proxy_cache_valid 301 302 304 404 1h;
secure_link_secret "some-secret";
if ($secure_link = "") {
return 403;
}
set $proto "http";
if ($uri ~ "/secure/")
{
set $proto "https";
}
image_filter_buffer 10M;
image_filter test;
proxy_connect_timeout 5;
proxy_read_timeout 5;
proxy_send_timeout 5;
proxy_ignore_headers Expires Cache-Control X-Accel-Redirect X-Accel-Expires;
proxy_intercept_errors on;
proxy_pass $proto://$secure_link;
error_page 301 302 401 403 404 415 500 501 502 503 504 505 =200 @1.gif;
access_log /var/log/nginx/cache/access-secure.log proxy;
}It uses several Nginx modules to implement resources that look like:
https://proxy.example.com/insecure/5f814704a38d9bc1915fd19fa0c7a00a/images.external.com/image.gif
The prefix "insecure" may also look like "secure" and encodes the protocol part of the original URL. When requested, this URL will either generate the response from a local cache or request an external image via HTTP, cache it locally, and send to the client.
The first named location block provides a fallback, that is, an empty 1 x 1 image that we will serve on all invalid requests.
The second big location block anchored at / is the main configuration. Since we have a dedicated hostname for the proxy, we work right from the root. First, there are declarations of caching and secure link parameters. After checking the validity of the request by using a condition on $secure_link variable, we compute the original, source URL schema or protocol. We use /secure/ as the prefix for HTTPS, and any other prefix will mean simple insecure HTTP.
A couple of image_filter_* directives configure the image filter to only ever check the first 10 megabytes. Proxy timeouts provide us with reasonably robust HTTP client. We do not want to hang endlessly on very slow (or maliciously slow) servers while also processing those servers that are not as fast as everyone hopes.
The interesting parts of the configuration are the secure link and image filter functionality that employ ngx_http_secure_link and ngx_http_image_filter modules, respectively.
The image filter module is the simpler of the two. It runs several heuristic checks against the contents of an image file to ensure that it is indeed a GIF, PNG, or JPEG image. This protects from several of the older browser security bugs that could be exploited with specially crafted responses masquerading as images. See http://nginx.org/en/docs/http/ngx_http_image_filter_module.html for more information.
The secure link module checks the cryptographic signature in the URL. The idea is that without the signature, you will create an HTTP proxy open to the whole world, which is a helpful resource for malicious actors of all kinds. The signature should be generated on the application side by your development team. The algorithm is described in the module documentation at http://nginx.org/en/docs/http/ngx_http_secure_link_module.html.
This module has a second, even more secure mode of operation that will also check the signature for expiration. We recommend that you implement that one too, see the documentation for details. The example mentioned earlier uses the easiest possible mode for the sake of brevity.
This proxy is not a final solution that we usually install in production but a simple version. For example, it does not properly process redirected images. As you may see from the last lines, many HTTP response codes including those 3xx that are responsible for redirects are considered errors and get redirected to an empty GIF. A solution for that is a good exercise in the Nginx configuration.