The first chapter in this book naturally is a review of concepts that are key to the rest of the book. In this chapter, we will look at:
The NetScaler code runs on top of FreeBSD as a userspace process, so it pays to understand the underlying file system structure.
A question that comes up often, given the system is based on FreeBSD, is, "Is the system automatically vulnerable to any FreeBSD security issues that (as with any software systems) are reported by advisories?" The answer is, it depends; the NetScaler is a hardened appliance with several of its modules customized to reduce a potential security impact. For a definitive answer, you will need to contact Citrix Support, who would then work with a Security Response team for a validated response.
While the base layout will be familiar to anyone familiar with UNIX-based systems, the files that we would look at when troubleshooting are custom to the NetScaler.
Start by using df
. This is also a great way to see how you are doing in terms of disk space:
The command df
stands for diskfree, a Unix command to show disk usage statistics. By using the –ah
option, we are asking for all the folders to be displayed in a human readable format, with percentages, for easy comprehension.
Let's take a look at the two important ones here for troubleshooting: /flash
and /var
.
/flash
, as you've probably guessed, maps directly to the Flash drive/SSD installed in the NetScaler. This is the most important partition on the NetScaler as it contains the operating system along with the configuration, license, and essentially everything needed to boot the NetScaler.
The /var
, which is the largest of partition and equals the hard disk on the NetScaler, contains: logs, crashes, traces, and other items that are to do with the maintenance and monitoring of the NetScaler.
In the case of a VPX, which is a virtual appliance with no physical drives, these folders become references to virtual partitions on the drive. Let's have a brief look at the important subfolders among these.
/flash
contains the following folders:
/nsconfig/
: This contains the NetScaler configuration files (ns.conf.*
). Each time you make a configuration change, it does get applied but doesn't get committed to the disk. To commit changes you need to click on Save config. Five such files, each resulting from a "save config", are saved in the /nsconfig/
folder. So, you can get back to a last known good configuration if you are in trouble after saving configuration changes./nsconfig/
folder is also home to other configuration files, most notably that of the routing engine ZebOS:/nsconfig/license
: This contains the license files./nsconfig/ssl
: This contains the SSL certificates, keys, and requests.ns-root.*
and ns-server.*
: These files come by default; the ns-root.*
files are used for signing, while the ns-server.*
files are bound to the internal services, so care must be exercised with any folder cleanup here. The ns-server
certificate is what you are presented with when accessing the NetScaler, and consequently, this is the key-pair that you would change as a best practice with a signed certificate you trust./nsconfig/monitors
: This is the folder for any USER
(script-based) monitors that you upload to the NetScaler.Monitors provided as Perl
files are used when creating a monitor of the type USER
. Going by the list in the following screenshot, you can guess that these are usually monitors that provide application knowledge beyond basic port or protocol response checks. In newer versions, the home for these files is /netscaler/monitors/;
it's when you upload any with modifications that they are stored in /nsconfig/monitors
.
/nsconfig/
and not /flash/nsconfig/
- that's because /nsconfig/
is a link to /flash/nsconfig/
and they represent the same folder.
/var/log
contains text based logs. Let's look at some of the important ones:
ns.log
: This is of paramount importance when troubleshooting, and as you will see during the course of this book, it is a file that we often turn to get a live view in easily readable messages in order to understand what is happening in the background, such as why is that User denied access, or why is the request blocked?messages.*
: These are your standard Linux/FreeBSD system logs. This is where I would look first if I were to go back in time to understand when a reported reboot of the unit happened, or when a shell command was run.license.log
: This is the log to look at if the licenses fail to apply, such as when the hostID
used during allocation is incorrect./var/nslog
: This folder contains the binary newnslog
files. While the ns.log
files we just discussed are very easy to read by text and are our preferred go to when dealing in a controlled environment, we most certainly can't digest patterns in a heavily used production unit, and that's where newnslogs
come in. This is a highly efficient form of logging that tracks the value of each counter (there's a few thousands of them) at every 7 second interval. These logs also serve as a basis for any external tool's plotting patterns, such as traffic, CPU, or errors.This folder also contains the AppFirewall learning related logs:
/var/core
and /var/crash
: This is where the crash files go./var/core
: This contains any crashes related to the NetScaler software, and you will almost always have them labeled in the format NSPPE-0x-xxxx
where the NSPPE
stands for the NetScaler packet engine, the first x
for the packet engine number, and the rest for the PID; recall what we said about the Packet Engines running as User processes, therefore they will have PIDs./var/crash
: This is where any core dumps by the Kernel will go.13.59.123.182