To access the system logs, navigate to Status | System Logs. There are several tabs in this section, but the default tab is System. Note that different subcategories (for example, Firewall and DHCP) have their own tabs where you can view log entries related to such activity, which simultaneously makes it easier to find log activity for a specific subcategory and also reduces clutter on the System tab. The System tab is itself divided into several subcategories: General, Gateways, Routing, DNS Resolver, and Wireless.

pfSense logs are stored in such a way as to not overflow the available disk space. The logs have a binary circular log file format; these log files are a fixed size and they store a maximum of 50 entries. If the limit is reached, older log entries are overwritten by newer ones. If you want to retain these logs, you can copy them to another server with syslog.

Displaying the system logs in pfSense.

As you can see in the sample log file displayed here, the General tab includes entries for several different services, including pfBlocker, VPN tunnels, and Dynamic DNS. The default log order is chronological, although you can show the log entries in reverse order by clicking on the Settings tab and checking the Forward/Reverse Display checkbox. Note that there is an Advanced Log Filter section at the top of the page (this section can be expanded by clicking on the plus icon on the right of the section heading). This section allows you to filter log entries by several criteria: by time, process, process ID (PID), the quantity of entries displayed (the default is 50), and the message contained in the log entry. Each of these fields except for Quantity can contain a regular expression as well. To filter the logs, click on the Apply Filter button.

You can control many log settings by clicking on the Settings tab. We already mentioned the Forward/Reverse Display checkbox, which allows you to show the log entries in reverse order. The GUI Log Entries edit box allows you to control the number of log entries displayed in the GUI (but not the number of entries in the actual log files). The next option, the Log file size (Bytes) edit box, allows you to change the size of each log file. By default, each log file is approximately 500 KB. Since there are about 20 log files, the disk space used by log files by default is about 10 MB. If you want to retain more than 50 entries per log file, you can increase this number. Be aware, however, that increasing this value increases every log file size, so make sure you have enough disk space available. For example, if you specify 1,048,576 here (1 MB), the total amount of disk space used will be 20 MB, and each log will contain 100 entries.

The next subsection is Log firewall default blocks. The Log packets matched from the default block rules in the ruleset checkbox, if checked, will log packets that are blocked by the implicit default block rule. By default, all internetwork traffic is blocked; unless traffic is explicitly allowed elsewhere, if this option is set this blocked traffic will be logged. If the log packets matched from the default pass rules in the ruleset are checked, pfSense will log packets that are allowed by the implicit default pass rule. Since you generally don't want to log traffic that is allowed to pass, by default this option is not checked. The Log packets blocked by 'Block Bogon Networks' rules and the Log packets blocked by 'Block Private Networks' rules checkboxes, if checked, logs packets blocked by those rules.

If the Web Server Log checkbox is checked, errors from the web server process for the pfSense GUI or the captive portal will appear in the main system log. The Raw Logs checkbox, if checked, will show the logs without being interpreted by the log parser. The raw log file, though more difficult to read, can be helpful in troubleshooting as it provides detailed information that is left out in the parsed log output.

The next option is the Where to show rule descriptions drop-down box. This option allows you to show a description of the applied rule in the firewall log. The options are as follows:

The Local Logging checkbox, if checked, will disable writing log files to the local disk. Clicking the Reset Log Files button will clear all local log files and reinitialize them as empty logs. This will also restart the DHCP daemon. If you have made any changes to settings on this page, you should click on the Save button before clearing the log files.

The next section is the Remote Logging Options section. Checking the Enable Remote Logging checkbox allows you to send log messages to a remote syslog server. If you check this option, a number of other options will appear. The Source Address drop-down box allows you to choose to which IP address the syslog daemon will bind. The choices include each interface on your pfSense system (which normally would include at least the WAN and LAN interfaces) and localhost. If one of these options is selected, then remote syslog servers must all be of that IP type (either IPv4 or IPv6). In order to mix IPv4 and IPv6 syslog servers, select Default (any) to bind to all interfaces. Also, if an IP address cannot be located on the chosen interface, the daemon will bind to all addresses.

The IP Protocol drop-down box allows you to select the IP type of the address specified in the Source Address drop-down box. However, if an IP address of the type selected here is not found, the other type will be tried. The Remote log servers edit boxes allow you to specify the IP addresses and ports of up to three syslog servers. Finally, the Remote Syslog Contents checkboxes allow you to select what events are sent to the syslog server(s). Keep in mind that you must configure the syslog daemon on the remote server to accept syslog messages from pfSense. When you are done making changes, click on the Save button.

You can also gather a great deal of information from the pfSense dashboard, which you can access by navigating to Status | Dashboard (the dashboard is also the first page you see when you log into the web GUI). The dashboard contains a great deal of information about your pfSense system, such as the uptime, CPU usage, memory usage, as well as the version being run and whether an upgrade is available. The dashboard has been redesigned for version 2.3; you can choose the number of columns in the display under General Setup. If you resize the width of your web browser, the dashboard will resize to a single column, thus ensuring that you do not have to scroll left and right. There is also an Interfaces widget which displays the interfaces, their speed, and their IP addresses. You can add widgets to the page by clicking on the plus sign on the right side of the title bar. There are widgets for gateways, traffic graphs, CARP status, load balancer status, and many packages have their own widgets. The dashboard updates every few seconds, so you don't have to hit the Reload button.

You can view information about the status of interfaces by navigating to Status | Interfaces. Information about all interfaces is available here, including the following:

If an interface has been configured to receive its IP address via DHCP (this is likely true for all WAN-type interfaces on your system), you can renew the DHCP lease via this page.

Most system and package services display their status on the Services page, which can be viewed by navigating to Status | Services. On this page, you will find a table that lists the name of the service (Service), a brief description (Description), and whether the service is running or stopped (Status). There is also an Actions column. By clicking on the appropriate icon for a service, you can either start a stopped service or restart/stop a running service. Normally it is not necessary to control services in such a way, but you may need to do so in a troubleshooting scenario.

There are three additional icons that appear in some, but not all, entries in the Services table:

By navigating to System | Monitoring, you can view another useful set of data relating to the real-time operation of your pfSense system. There are two sections on this page: a graph (Interactive Graph) and a summary of the information in the graph (Data Summary). There are several pieces of information available on this page, and they relate to the percentage of CPU usage attributable to different processes:

There is also a column representing the grand total of processes. Each entry includes the minimum, maximum, and average percentage of CPU usage for each category.

You can view traffic graphs for each interface by navigating to Status | Traffic Graph. You can select the interface for which a graph is generated in the Interface drop-down box. Information displayed in the table adjacent to the graph is sorted in descending order based on either bandwidth in or bandwidth out, depending on what is selected in the Sort by drop-down box. The Filter drop-down box allows you to display either only local traffic or only remote traffic in the table (the default selection is All). The Display drop-down box allows you to select what is displayed in the Host Name or IP column: the IP address, the hostname, a description, or the Fully Qualified Domain Name (FQDN).

Sometimes when troubleshooting, it is helpful to view information about the firewall states. These states can be viewed several different ways from within the pfSense web GUI and the console.

pfTop

pfTop is available in both the web GUI (by navigating to Diagnostics | pfTop), and at the console (it is 9 on the console menu). pfTop provides a live view of the state table and the total amount of bandwidth utilized by each state. If you are using pfTop from the console, type q to quit; this will return you to the console menu.

Running pfTop at the console.

Most of the column headings in pfTop are self-explanatory. For example, the default view provides the following column headings: PR, D, SRC, DEST, STATE, AGE, EXP, PKTS, BYTES.

PR stands for protocol; D stands for direction (in or out); SRC and DEST stand for source and destination, respectively. AGE indicates how long it has been since the entry was created; EXP indicates when the entry expires; PKTS indicates the number of packets that have been handled by the rule, and BYTES indicates the number of bytes.

STATE indicates the state of the connection in the format client:server. Since the states will not fit into an 80-column table, pfTop uses integers, such as 1:0. The numbers signify the following:

Number	State
0	TCP_CLOSED
1	TCP_LISTEN
2	TCP_SYN_SENT
3	TCP_SYN_RECEIVED
4	TCP_ESTABLISHED
5	TCP_CLOSE_WAIT
6	TCP_FIN_WAIT_1
7	TCP_CLOSING
8	TCP_LAST_ACK
9	TCP_FIN_WAIT_2
10	TCP_TIME_WAIT

Thus an entry of 1:0 indicates that the state on the client side is TCP_LISTEN, and the state on the server side is TCP_CLOSED.

One of the advantages of using pfSense within the web GUI is that it is very easy to change the output to suit your needs. The View drop-down menu allows you to choose how pfTop displays its output. There are several options, including:

• label: The LABEL column represents the rule that is being invoked, and how many packets, bytes, and states are accounted for by the rule
• long: Displays the protocol, source, destination, gateway, state, and age of each entry
• queue: If the traffic shaper is configured, it will display results organized by queue
• rules: This option will display each rule being invoked in the rightmost column, and the number of states associated with each rule

There is also a Sort by drop-down box, which allows you to sort output in descending order by several categories (for example, Bytes, Age, Destination Address, Source Address, and others). The Maximum # of States drop-down box allows you to control the number of states that appear on the page.

If you run pfTop from the console, it will be running in interactive mode, which means that pfTop will read commands from the terminal and act upon them accordingly; characters will be processed as soon as they are typed, and the display will be updated immediately after the characters are processed.

Note

Refer to the pfTop man page for a full listing of commands available in interactive mode, as well as pfTop command-line options.

Often the most effective way of troubleshooting a networking problem is through packet capturing, also known as packet sniffing. One way of capturing packets is to use the command-line tool tcpdump, which is part of the default pfSense installation. Tcpdump is a command-line utility used to capture and analyze packets; details can either be displayed on the screen or saved to a file. It uses the libpcap library for packet capturing.

The results of packet capture will differ depending on which interface's traffic you capture. As a result, you should give some consideration as to which interface's traffic you choose to capture, and in some cases, you may want to capture traffic from several interfaces at the same time. In order to use tcpdump, you will have to use the underlying device names of the interfaces. If you don't remember what they are, you can navigate to Interfaces | (assign) within the web GUI. The console menu also lists each interface and has a separate column for the device name. Another way of retrieving a list of interface names is to issue to following command from the console shell:

tcpdump –D

Then, to run tcpdump on a single interface, type the following:

tcpdump –iinterface_name

where interface_name is the device name (for example, fxp0, em1, and so on). Alternatively, you can run tcpdump without any command-line options to capture packets from all interfaces.

If you run tcpdump, you may notice that the hostname of the source and destination is displayed. By default, tcpdump does a DNS lookup on IP addresses. As a result, tcpdump can generate a considerable amount of DNS traffic.

By default, tcpdump runs continuously until you press Ctrl + C, but you can limit the number of packets captured with the –c option. For example:

tcpdump –c 10

This will cause tcpdump to capture 10 packets and then stop running. The default maximum capture size for each packet is 64 K, but in many cases you may only want to see what's in the header. You can use the –s parameter to limit the amount of each packet captured; for example:

tcpdump –s 96

This will only capture the first 96 bytes of each packet.

Tcpdump allows you to save packet capture files in pcap format for later analysis. This is useful, especially if you want to load the files onto another computer running Wireshark or some other graphical network protocol analyzer. To save the output to a file, use the –w option, like this:

tcpdump –w filename

Be aware that, when you are using this option, the frames will not be displayed on the screen, as they otherwise would be.

By default, tcpdump puts your network interface into promiscuous mode, which shows every frame on the wire, not just frames being sent to its MAC address. In modern networks, this should not be much of a problem, as most networks employ switches, and the interface generally will only receive traffic it should receive. If you have hubs on your network, however, running tcpdump in promiscuous mode can result in you capturing a great deal of traffic that may not be of interest to you. By using the –p option, which runs tcpdump in non-promiscuous mode, you can improve the signal-to-noise ratio and focus on traffic destined for the interface on which you are capturing packets.

You can control the verbosity of tcpdump's output with the –v flag. This flag only controls the output on the screen and not the contents of tcpdump output saved to a file (assuming that output is being saved). In addition to –v, you may also choose –vv or –vvv, which provide additional verbosity for screen output. If you invoked the –w option to write to a file along with one of the verbosity options, then tcpdump will report the number of packets captured at 10-second intervals.

The –e option causes tcpdump to display the MAC addresses of the source and destination of the packet as well as 802.1Q VLAN tag information.

You may notice that tcpdump displays packet sequence numbers. You may also notice that when displaying multiple packets from the same source/destination, the first packet in a series of packets has large sequence numbers, but all subsequent packets have smaller numbers. This is because tcpdump switches to relative sequence numbers in order to save display space. To see only actual sequence numbers, use the –S flag.

If you want a simple frontend for tcpdump, you can use the tcpdump page in the web GUI instead. To do so, navigate to Diagnostics | Packet Capture. Once there, use the Interface drop-down box to select the interface whose packets will be captured (note that there does not seem to be an option to capture all interfaces on this page). Checking the Promiscuous checkbox enables promiscuous mode. The Address Family drop-down box allows you to select IPv4 packets, IPv6 packets, or both. The Protocol drop-down box has several options: you can capture any packets (Any), or the following: ICMP, Exclude ICMP, ICMPv6, Exclude ICMPv6, TCP, Exclude TCP, UDP, Exclude UDP, ARP, Exclude ARP, CARP, Exclude CARP, pfsync, Exclude pfsync, ESP, and Exclude ESP.

The Host Address edit box allows you to specify a source or destination IP address or subnet (in CIDR notation). Tcpdump will look for the address specified in either field. You can negate the IP address by preceding the value with !, in which case tcpdump will match everything except the IP address. Multiple IP addresses or CIDR subnets may be specified here; comma-separated values (,) perform a Boolean AND, while separating addresses with a pipe (|) performs a Boolean OR. If this field is left blank, then all packets on the specified interface that meet the other criteria specified will be captured, regardless of the source or destination IP address.

If you specify a port in the Port edit box, tcpdump will look for the port in either field. If you leave this field blank, tcpdump will not filter by port. The Packet Length edit box lets you specify the number of bytes of each packet that will be captured. The default value is 0, which will cause the entire frame to be captured. The Count edit box allows you to specify the number of packets tcpdump will grab. The default value is 100; specifying 0 will result in tcpdump continuously capturing packets.

The Level of detail drop-down box controls the amount of detail that will be displayed after you hit Stop when packets have been captured. The options are Normal, Medium, High, and Full. This option does not affect the level of detail in the packet capture file if you choose to download it when the packet capture completes.

The Reverse DNS Lookup checkbox, if checked, will result in tcpdump performing a reverse DNS lookup on all IP addresses. As noted when discussing the command-line options for tcpdump, doing a reverse DNS lookup generates considerable DNS traffic and also creates delays, and therefore is not generally recommended. When you are done selecting options on this page, click on the Start button.

Once you click on Start, you should see a Packet capture is running message across the bottom of the page, and the Start button should become a Stop button. Once you click on the Stop button, a Packets Captured listbox will appear at the bottom of the page with information about the packets captured. You can change the level of detail by changing the value in the Level of Detail edit box and clicking on View Capture to update the display. Finally, you can save the packet capture by clicking on the Download Capture button; this will save the capture as a .cap file, which can be opened by many network protocol analyzers such as Wireshark.

tcpflow, like tcpdump, allows you to view the text contents of network packets in real time. Whereas tcpdump is more suited to capturing packets as well as protocol information, tcpflow is better suited for viewing the actual data flow between two hosts. One significant difference between tcpflow and tcpdump is that, while tcpdump displays output to the console by default, tcpflow writes the output to a file by default. In order to display tcpflow's output on the console, you can use the –c option.

Much of the syntax of tcpflow is similar to that of tcpdump. For example:

tcpflow –i fxp0 –c host 172.16.1.2 and port 80

This would capture packets on the fxp0 interface with either a source or destination of 172.16.1.2 port 80. Here are some of the options available for tcpflow:

ping, traceroute, and netstat are old command-line utilities used to test the reachability of hosts, provide routing information, and information about network connections. Often they are the first tools used by network technicians when testing networks. They can prove invaluable when troubleshooting networks. Both ping and traceroute are accessible from pfSense's web GUI, although they are more commonly used from the console.

ping

The purpose of ping is to measure the round-trip time (RTT) for messages sent from a source host to a destination host that are then echoed back to the source. It uses Internet Control Message Protocol (ICMP), sending ICMP Echo Request packets to the destination host and waiting for an ICMP Echo Reply. The following displays typical ping output under Linux:

The first item reported is the size of the packet received. The default size is 56 bytes, but an ICMP ECHO_REQUEST packet contains an additional 8 bytes as an ICMP header followed by an arbitrary amount of data. Thus, the size reported is 64 bytes. Next is the destination IP address. By default, ping displays the IP address to which the hostname resolves rather than the hostname.

The icmp_seq field reveals the ordering of the ICMP packets. ping reports on each packet as it is received, and the packets are not necessarily received in the same order as they are sent, although when the networks are functioning properly, they usually are. ttl stands for time to live. The TTL field is reduced by one by every router en route to its destination. If the TTL field reaches zero before the packet arrives, then an ICMP error is sent back (ICMP Time Exceeded). As you may have guessed, the default start value used by ping in Linux is 64. Finally, the last field is the RTT of each packet, which is a good measure of the latency of a connection.

Once the results for each packet are reported, ping reports aggregate statistics for the ping session. The number of packets transmitted and received is reported, as well as the percentage of packet loss. On the final line, we see: the minimum RTT, the average RTT, the maximum RTT, and the standard deviation.

One caveat that should be made concerning ping is that many firewalls block ICMP traffic, rendering the ping utility useless with hosts behind restrictive firewalls. In fact, pfSense blocks such traffic by default, so if you want to ping your hosts from the other side of the firewall, you will have to explicitly allow such traffic. Even so, you may have occasion to ping a network you don't control that blocks ICMP traffic. In such cases, you may be better off utilizing a utility that relies on TCP or UDP for sending packets, since such protocols are much less likely to be blocked by most firewalls. One such utility is tcpping, and it has a similar syntax to ping. If you are pinging to local hosts, you can use arping, which uses the Address Resolution Protocol (ARP) request method to resolve IP addresses.

Note

To install tcpping, you must first install tcptraceroute and then tcpping, which is a script that utilizes tcptraceroute. You can install tcptraceroute from the repositories. If you are using Debian/Ubuntu/Mint Linux, type the following at the console:

sudo apt-get install tcptraceroute

For CentOS/Red Hat Enterprise Level, the command is:

sudo yum install tcptraceroute

Then you have to install tcpping, which can be done with the wget command:

$ cd /usr/bin
$ sudowget http://www.vdberg.org/~richard/tcpping

You'll also want to set permissions for tcpping, which you can do with chmod:

$ sudochmod 755 tcpping

To see the command line options for tcpping, type the following at the console:

tcpping --help

This caveat aside, the ping utility is useful in a number of different troubleshooting scenarios:

• ping can help us determine if there is network connectivity between two hosts.
• ping can help us determine if there is an unacceptable rate of packet loss. We may have connectivity between two hosts, but if the packet loss rate is consistently high, network performance will undoubtedly suffer.
• ping is a good tool for measuring latency between two hosts.

As an example of the last of these scenarios, you might consider pinging a well-known host (for example, google.com) and measuring the latency in a number of different scenarios: for example, on a broadband connection, on a DSL connection, on a mobile connection, through a VPN, and so on.

You may have noticed that, when we invoked the ping command under Linux, we used one flag: the –c flag, which limits the number of packets sent. Without the –c flag, ping would have sent packets continuously until we pressed Ctrl + C at the console. This is just one of many flags and options available for ping. The following table covers some of the more commonly used ping options:

Option	Description	Windows equivalent
-c count	Stop after receiving count ECHO_RESPONSE packets	-n count
-D	Set the DF bit	-f
-f	Flood ping; output packets as fast as they come back (use with caution)	NA
-i wait	Wait seconds before sending each packet	NA
-mttl	Set the ttl for each packet.	-I ttl
-S source_addr	Use source_addr as the source address in outgoing packets; useful for forcing the IP address to be something other than the IP address on which the ping packet is sent out (only works if the IP address is one of the host's IP addresses)	-S source_addr
-spacketsize	Specify the number of data bytes to be sent (the default is 56)	-lpacketsize
-t timeout	Specify a timeout, in seconds, before ping exits regardless of how many packets have been received	NA
-v	Verbose output; ICMP packets other than ECHO_RESPONSE packets are also displayed	NA

Be aware that this is not an exhaustive list of ping options; consult the ping man page for a complete.

If you are running ping from the Windows command prompt, the output is similar, with some exceptions:

• By default, ping sends four packets instead of sending all the packets. To send packets continuously, use the –t option. To send an arbitrary number of packets, use the –n count option.
• The default packet size is 32 bytes.
• The summary does not show the standard deviation.

Other than that, the behavior of ping under Windows is similar to its behavior under Linux, although it seems to have fewer command-line options. The preceding table lists some of the Windows ping flag equivalents.

You can also invoke ping from within the pfSense web GUI. To do so, navigate to Diagnostics | Ping. In the Hostname edit box, specify the hostname or IP address to ping. You can specify the protocol in the IP Protocol drop-down box (IPv4 or IPv6). In the Source Address drop-down box, you can set a source address for the ping. Finally, in the Maximum number of pings edit box, you can set the maximum number of pings (the default is 3). When you are done configuring the ping settings, click on the Ping button.

traceroute

traceroute (or tracert, as it is known under Windows) is a network diagnostic tool for IP networks. Its purpose is twofold: to display the path of packets, and the transit delays along each step (known as a hop). The RTT of each hop is recorded, and the sum of the mean times in each hop is a measure of the total time taken to establish the connection. By default, traceroute outputs the results of each hop, as well as the final results. traceroute sends three packets, and it proceeds unless all three are lost more than twice. In this case, the connection is considered lost and the route/path cannot be evaluated.

Using traceroute at the Windows command prompt.

traceroute is available at the Windows command prompt (as tracert), but it is not part of most default Linux installations. Instead, it is available from the repositories, both as a standalone package (traceroute) and as part of the inetutils utilities (inetutils-traceroute). The output of traceroute is relatively simple. The first column displays the hop count. The final column displays the IP address and hostname (if available) of the host/router. The middle three columns display the RTT of each of the three packets sent.

The only required parameter is the hostname or IP address of the destination host. There are, however, many other options available, as shown in the following table:

Option	Description
-e	Firewall evasion mode; uses fixed destination ports for UDP, UDP-lite, TCP, and SCTP probes
-ffirst_ttl	Set the ttl for the first outgoing packet
-F	Set the DF bit
-d	Enable socket-level debugging
-I	Use ICMP ECHO instead of UDP datagrams
-M first_ttl	Set the ttl value used in outgoing probe packets
-P proto	Set the protocol (proto) used in outgoing probe packets; the currently supported values are UDP, UDP-Lite, TCP, SCTP, GRE, and ICMP
-ssrc_addr	Use src_addr as the IP address in outgoing probe packets to force the source address to something other than the IP address of the interface the probe packet is sent on (only works if the IP address is the address of one of the interfaces on the host)
-S	Print a summary of how many probes were not answered at each hop
-v	Verbose output (all received ICMP packets shown)
-w	Set the time to wait for a response to a probe (default is 5 seconds)

I omitted the Windows Equivalent column on this table, since few of the options available for traceroute under Linux exist for the Windows version. If you need to use another protocol, you can use the –P option; there is also a utility called tcptraceroute (available for Linux), which sends TCP probe packets.

You can also invoke traceroute from the web GUI. To do so, navigate to Diagnostics | Traceroute. Type in the hostname or IP address in the Hostname edit box. You can select the protocol (IPv4 or IPv6) in the IP Protocol drop-down box. You can select the source address for the trace in the Source Address drop-down box. In the Maximum number of hops drop-down box, you can set the maximum number of network hops to trace (the maximum number is 20; the default is 18). You can enable DNS lookup by checking the Reverse Address Lookup checkbox. Finally, you can change the protocol used by traceroute from UDP to ICMP by checking the Use ICMP checkbox. When you are done configuring settings, click on the Traceroute button.