Any advanced network security product needs a set of troubleshooting tools. This chapter covers the debugging and troubleshooting features available on the Check Point NGX R65 suite. Many of the troubleshooting methods detailed in this chapter will work on any NG- or NGX-based Check Point firewall.
To understand troubleshooting methodology, a deep understanding of the Check Point internal workings and directory structure is in order. As mentioned earlier in this book, Check Point VPN-1 is a software-based firewall product. Check Point can run on Check Point Secure Platform, Red Hat Linux, Sun Solaris, HP-UX, IBM AIX, Nokia IPSO (BSD), and Windows Server. The only major differences between the different operating system installs are the install scripts and the kernel bindings. The Check Point directory structure is the same across the different operation systems, with the exception of Windows, which adds an extra line feed to the various text files.
The firewall creates two directory trees. The SVN Foundation tree (a.k.a. the cpshared tree) is installed in the $CPDIR directory. This Windows install installs in %CPDIR%, following the Microsoft environment variable format. The firewall package installs in the $FWDIR directory. Note that the $CPDIR/bin and $FWDIR/bin directories are in the root/admin user’s path environment variable.
Here are some of the directories created that are important to mention:
$CPDIR/bin. Contains the cpconfig and cplic utilities
$FWDIR/bin. Contains the firewall and management daemons
$FWDIR/conf. Contains the firewall configuration
$FWDIR/lib. Contains the custom Check Point Inspect code for protocols
$FWDIR/log. Contains the log files
$FWDIR/spool. Contains the e-mail in the Simple Mail Transfer Protocol (SMTP) queue used by the SMTP resource
$FWDIR/tmp. Contains temporary files, used for utilities such as fw monitor
When a SmartDashboard client connects to a Management Server, the following files are downloaded to the client:
The objects_5_0.C file contains all firewall properties, objects, automatic network address translator (NAT) settings, global properties, user groups, and implicit rules. The rulebases_5_0.fws file contains all the explicit rules and manual NAT rules of every rulebase, concatenated. All rulebases use the same objects file, which is a primary reason for using Database Revision Control instead of creating new rulebases for internal policy backups. The fwauth. NBD file contains all of the users, and is also known as the User Database.
When a policy is pushed to a firewall, the firewall will take the particular rulebase being pushed and save it to a W file. If the policy name is Standard, the Standard rulebase from the rulebases_5_0.fws file is saved to $FWDIR/conf/Standard.W. The compile process takes this file along with the objects, the user database, and the contents of the $FWDIR/lib directory to create the policy, $FWDIR/conf/ Standard.pf.
fw debug fwd on TDERROR_ALL_ALL=5
The firewall daemon logging will output to $FWDIR/log/fwd.elg. This file will be in clear text and can be read into a text editor or parsed with grep. To stop logging issue the following command:
fw debug fwd off TDERROR_ALL_ALL=0
You also can place the Management Server into debug mode by substituting “fwm” for “fwd” in the preceding commands. The management daemon logging will output to $FWDIR/log/fwm.elg. The Check Point resources, also known as security servers and proxies, can also be placed into debug mode. To be able to debug a resource, the resource must have a process id (PID). The PID references are in the $FWDIR/tmp directory and end with .pid. Resources are named in the following manner: in.a<PROTOCOL>d. For example, to start debugging the Hypertext Transfer Protocol (HTTP) security server issue the following command:
fw debug in.ahttpd on FWAHTTPD_LEVEL=3
To stop debugging the HTTP security server use this command:
fw debug in.ahttpd off FWAHTTPD_LEVEL=3
The output of the debugging is stored in $FWDIR/log/ahttpd.elg in this instance. You can achieve more advanced troubleshooting using the fw ctl debug command, which will dump the actual kernel memory to a file.
Check Point communicates internally using secure internal communication (SIC). You can encrypt this communication using certificates, cut from the certificate authority (CA) running on the primary management station, and it is based on OpenSSL.
There is no built-in interface to manage the CA; however, there is a Knowledge Base article on Check Point’s support site that explains how to install one. You can use these certificates for authentication and encryption. This system replaced Check Point’s older “putkey” system, which is still there for backward compatibility.
You can resolve most SIC issues by resetting the communication on the remote end. Using the cpconfig command on a firewall offers an option to Reset Secure Internal Communication. This will ask for a one-time password, which is used in the SmartDashboard GUI to establish a new certificate for the firewall.
There is a chance the CA can become corrupt. This can happen in certain scenarios when the name or Internet Protocol (IP) address of the Management Server is changed without taking the proper steps to ensure a smooth transition. Once Check Point is configured on a Management Server, the name or IP address should not be changed in the firewall object, or on the operating system. Follow the R65 Upgrade Guide for procedures on changing these settings properly.
You can reinitialize the CA using the fwm sic_reset command. You need to run this command in expert mode if you are running it on Secure Platform. This will reset all certificate operations and require new one-time passwords to be generated for every firewall, along with a reimport of the licensing. After running the command on the management station, you need to run cpconfig again and choose the Certificate Authority option to tell Check Point to initialize a new CA.
SIC uses Transmission Control Protocol (TCP) ports 18209 through 18211. You can verify access to SIC services by accessing port 18211 or by using netstat –na and searching for the open TCP port. The fwd and fwm debug logs, as well as the cpd daemon logs, can help you to troubleshoot SIC issues. You can put the SVN Foundation process into debug mode by issuing the following commands:
cpd_admin debug on TDERROR_ALL_ALL=5 cpd_admin debug on OPSEC_DEBUG_LEVEL=9
The following commands will turn off cpd debugging:
cpd_admin debug off TDERROR_ALL_ALL=0 cpd_admin debug off OPSEC_DEBUG_LEVEL=0
If all else fails, try to delete the firewall object and create a new one. It is possible for independent objects to contain problematic configuration options.
Sometimes you need more information than what is available in Check Point’s logging mechanisms. Several utilities are available to troubleshoot packet issues, including snoop, tcpdump, and fw monitor.
snoop is the packet sniffer built into the Sun Solaris operating system. A long time ago, Check Point was a Solaris-only product and the Check Point documentation contained many references to snoop. Some of those references still exist. snoop is similar to tcpdump, as it grabs raw packets and either displays them in a readable format or dumps them to a file. snoop has the ability to parse packet dump files and format them for text-based display as well.
tcpdump is the packet sniffer utility that is available for most flavors of UNIX and Linux. This utility is installed by default on Check Point’s Secure Platform. Like snoop, tcpdump can read raw packets from any or all interfaces and display them in a text-based terminal or dump them into a file. It is recommended that you use filters to limit what tcpdump writes to a file.
Check Point has a built-in packet sniffer that you can access by running the command fw monitor. This is the sniffer of choice on a Check Point appliance. The fw monitor command is available on every operating system on which Check Point will run. The output is displayed on the console by default. If redirected to a file, the fw monitor utility stores its packet dumps in libpcap format, which is the same format that tcpdump uses. You can use a graphical utility such as Wireshark, CPethereal, or Sniffer Pro to analyze these capture files later.
Several command-line switches are available for fw monitor. Some of the more important ones are:
fw monitor –o <filename>. This outputs to a file instead of the console.
fw monitor –m. The capture mask tells the monitor where to sniff out of four places.
fw monitor –p. The position allows for a more granular inspection.
fw monitor –e. Expressions allow for filtering of what to capture.
fw monitor –f. Multiple expressions can be placed into a filter file.
fw monitor –x. The offset allows for capture of the data portion of the packets.
fw monitor –c. The count option limits how many packets are captured.
The fw monitor –m switch allows you to pick where the firewall captures packets. The firewall kernel is loaded in two places: on the inbound direction and on the outbound direction, with routing in between. By default, fw monitor sniffs in four places. Here is the process:
The packet arrives at the inbound interface. This is signified by –m i.
The packet flows through the inbound firewall kernel. This is signified by –m I.
The operating system routes the packet.
The packet arrives at the outbound interface. This is signified by –m o.
The packet flows through the outbound interface. This is signified by –m O.
So, the default capture contains all four places: fw monitor –m iIoO. You can pick and choose which ones you want to capture. This is sufficient for most needs, but what if you wanted to capture in between firewall processes? The fw monitor –p command allows for more granular capturing.
Before looking into the –p parameter, let’s reference the output of the fw ctl chain command, as shown in Figure 10.1.
This command displays most of the positions that a packet traverses through a Check Point firewall. More processes handle packets, but the fw ctl chain output displays everything that is possible to capture using the –p parameter. The first number is a position ID, and the other numbers are memory offsets and absolute memory locations, which vary depending on the operating system and firewall version. The text in parentheses at the end represents aliases. You can use the ID or the alias using the –p parameter. The fw monitor –m command captures at the firewall VM inbound listed in the chain output. Here is an example of using the –p parameter to capture at the inbound virtual private network (VPN) decryption:
fw monitor -pi 2
Here is the same command using an alias:
fw monitor -pi -fw
You can also sniff in every possible place using the –p all parameter. This will log 15 packets for every one packet. Running this on a busy firewall without a filter would be a recipe for disaster, so let’s get into filtering our capture to get just what we need. The –e parameter allows for custom expressions to be written. This expression is written in Check Point’s proprietary Inspect language. Here is an example:
fw monitor -e 'accept [9:1]=6;'
The preceding command captures packets whose ninth byte, the length of 1 byte, is equal to 6. In an IP header, the ninth byte is the protocol ID. The protocol ID for TCP is 6. This expression captures all TCP packets that the firewall rulebase accepted.
When an fw monitor is compiled, it includes the $FWDIR/lib/fwmonitor.def file, which also includes the $FWDIR/lib/tcpip.def file. These files contain aliases that you can use in expressions. For example, one of the lines in the tcpip.def file contains the text #define ip_p [ 9 : 1 ]. This allows us to issue the following command instead:
fw monitor -e 'accept ip_p=6;'
See the tcpip.def file for more references. When you execute fw monitor the following text appears:
monitor: getting filter (from command line) monitor: compiling monitorfilter: Compiled OK. monitor: loading monitor: monitoring (control-C to stop)
When the monitor is invoked a miniature inspect code base is generated and stored in the $FWDIR/tmp/monitorfilter.pf file. If you used an elaborate expression, you can copy this monitorfilter.pf file to another location so that you can use it in the future using the –f parameter. Your custom filter file can contain your own custom #define aliases as well.
The expressions also understand logical operations and order of operations. Let’s say we want to capture all traffic between 192.168.1.1 and 10.1.1.1, using the Internet Control Message Protocol (ICMP):
fw monitor -e 'accept ip_p=1, ((src=192.168.1.1, dst=10.1.1.1) or (src=10.1.1.1, dst=192.168.1.1));'
The –x parameter allows for data capture. For example, let’s say we want to capture the first 100 bytes of the data portion of the packets. An offset is required, and with TCP packets the data portion starts at offset 52:
fw monitor -x 52,100
The last option that can be useful is the –c parameter, which stops monitoring after a specified number of packets. You can specify inbound and outbound directions using this command. Keep in mind that it will still log as many positions as you specify. For example, if you use defaults, four entries are created for each packet, so if you count five packets, you will capture 20 packets (5 × 4). For example:
fw monitor -ci 5 -co 5
More command-line options are available for fw monitor, but the preceding command-line options are the ones that are most frequently used. Good references for capture scripts are available on the Check Point User Group (CPUG) forums, Check Point’s own Knowledge Base, Nokia’s Knowledge Base, and the Check Point “How to use fw monitor” document.
Check Point has a client application available for analysis of tcpdump and fw monitor capture files. CPethereal is available from Check Point’s Support Download Web site under NG Utilities. The advantage of this application is that it has an additional field for the FW Chain position of the packet. The downside of this application is that it is based on a rather dated version of Ethereal.
You also can load the fw monitor output into the latest version of Wireshark, which used to be Ethereal. Finding the FW Chain location is still possible by looking at the first several bytes in the hex output at the bottom of the interface, where the hex dump is displayed. The first byte is the basic location: i, I, o, or O. The second byte contains a number 0 through 9 signifying the absolute FW Chain location. The next several bytes spell out the inbound or outbound interface name.
All Check Point firewalls can generate logs locally. The fw log command will allow an administrator to view log entries on the command shell. Most issues with logging are the result of connectivity issues with the Management Server and/or Logging Server. Logging issues can also occur when there is an issue with the SIC certificates, as Check Point uses these certificates to secure the logging traffic.
Check Point will populate an empty $FWDIR/log directory on startup of various services. If you have a corrupt log file and fw repairlog does not seem to resolve your issues, you can issue a cpstop and move or delete the contents of the $FWDIR/log directory, and then issue a cpstart to bring the services back up. Do not remove the actual log directory or symlink in $FWDIR. The directory has to exist for Check Point to initialize.
When troubleshooting VPN traffic, you can resolve most issues by studying the Info field of the SmartView Tracker logs. Usually a break in an existing VPN is caused by a recent configuration change in the policy. If a Database Revision Control backup has been made recently, open it in a new window and compare the current policy with the backup. Also, double-check that the encryption domains have not been modified, as the encryption domain group may be used in other rules in the policy and is sometimes modified for other purposes by accident. It may also be a good idea to check CPU usage and memory usage of the VPN gateway. If a runaway process uses up all available resources, all VPN traffic can be affected.
If there are certificate issues, switching to a preshared secret is a valid step to fix the VPN issue while the certificate issues are being worked on. Another common issue, which directly affects VPNs that are using certificates, is time and date settings on the gateways. Make sure the time and date settings are within the certificate valid date fields.
Tip
Make sure routing has not been affected. It may seem obvious that two gateways need to be able to route to each other for a VPN tunnel to establish, but many VPN tunnels have failed as a result of improper routing.
Now we’ll discuss some examples of errors that show up in the Info view of the SmartView Tracker.
If you receive the error Encryption failure, decrypted methods did not match rule, this typically means there are overlapping encryption domains. VPN-1 supports proper subsets, which is an encryption domain fully contained within another encryption domain, and fully overlapping encryption domains when clusters are involved. Otherwise, any encryption domain overlap will cause the VPN to fail.
The error message Received notification from peer: no proposal chosen is rather common. This usually means the VPN settings do not match on both ends. This error could also mean there is a subnet negotiation mismatch. Make sure the Support key exchanges for subnets setting is enabled on older versions of Check Point, or that VPN tunnel sharing settings are configured properly in the advanced section of the VPN community on the R65. Both ends must agree on these settings for a proper VPN tunnel to be established.
You will receive the error message Cannot identify peer for encrypted connection when NAT is not being applied properly. Make sure that if NAT is applied to a packet the NAT IP address is part of the encryption domain.
The SA is the Security Association in a VPN tunnel. The error Encryption failure: packet is dropped as there is no valid SA can occur when packets are corrupted before they reach the other VPN gateway. Look into data link errors or links that are over capacity. This error can also occur when the fwha_sync_outbound_sa setting in the firewall object is set to false. To resolve this issue use guidbedit or dbedit to set this variable to true in the firewall properties.
If you receive the error message that reads Encryption failure: Clear text packet should be encrypted or clear text packet received within an encrypted packet, it means that the IP and subnet mask are incorrect in the firewall Topology tab, or the VPN is terminating to the wrong interface. Make sure the external IP address of the gateway is the IP address in the General tab of the firewall object. In a cluster object, the external routable Virtual Internet Protocol (VIP) address of the cluster should be the IP on the main gateway properties page.
An improper VPN domain can cause the error message Encryption Failure: Packet was decrypted, but policy says connection should not be decrypted. Double-check that the VPN domains on the Topology tab of the VPN objects are set properly.
If all else fails, you can run a VPN debug on a VPN gateway. To start VPN debugging issue the following commands:
vpn debug on vpn debug ikeon
The output of these commands dumps logging information in the $FWDIR/log/ vpnd.elg file. You can open this log file in the IKEView.exe utility, which is available from the NG Utilities section of Check Point’s support Web site. To stop VPN debugging issue the following commands:
vpn debug ikeoff vpn debug off
Troubleshooting issues with SecuRemote and Secure Client is typically very similar to standard IKE VPN troubleshooting. Check the VPN domains, as well as connectivity to the external side of the VPN gateways from the client network.
The most common issue is connectivity to the encryption gateway on the proper ports. Check Point uses several ports with client-side VPN. If these ports are not open, you may need to connect via Visitor Mode, which emulates a Secure Sockets Layer (SSL) VPN on TCP port 443. Check Point has a limited SSL VPN built into VPN-1, called SSL Network Extender (SNX). For a full-blown SSL VPN, Check Point offers Connectra, which requires an additional host. As of R65, Connectra management is built into the SmartCenter Server.
A packet capture utility is available on all SecuRemote clients. The srfw monitor command allows for client-side analysis and is similar to the fw monitor utility.
You can cluster Check Point firewalls in several ways. In this section, we will discuss how to diagnose issues with Check Point’s built-in clustering functionality, ClusterXL.
ClusterXL has four modes: Legacy High Availability, New Mode High Availability, Load-Sharing Multicast, and Load-Sharing Unicast. Communication among these nodes is facilitated using the Cluster Control Protocol (CCP) using User Datagram Protocol (UDP) port 8116.
State synchronization is optional with the High Availability modes and mandatory with the Load-Sharing modes. To check the status of state sync, use the fw ctl pstat command. The status of the sync will be at the end of the output. Another way to check the status of state sharing is to view the number of state table entries on both nodes by running the following command:
fw tab -s -t connections
The output under the VALS parameter is the number of state table entries in memory. Check the physical statistics on the Ethernet port that state sync is using and make sure there are no data link errors. Also make sure you can ping the other nodes as well, but keep in mind that a rule may by required to allow ICMP between the nodes.
Verify that there are no IP conflicts on the firewall networks. New Mode High Availability and the Load-Sharing modes require a physical IP address per network as well as a VIP. IP conflicts will show up in the Info view of the SmartView Tracker as well as the various devices that have the IP address causing the conflict.
The Load-Sharing Multicast method has the most issues on many networks, as it requires that all locally connected devices support Multicast traffic. Static Address Resolution Protocol (ARP) and/or static Content Addressable Memory (CAM) entries may be necessary on switches and routers. Also make sure the spanning tree is configured on switches. If the local switches do not understand Multicast, they may treat the traffic as broadcast traffic. Without the spanning tree, this traffic may cause routing loops and essentially cause an internal denial of service (DoS). In case of this emergency, immediately change the load-sharing method to Unicast until the internal network is reconfigured to handle the Multicast traffic.
For advanced troubleshooting techniques, refer to the Check Point Advanced Technical Reference Guide for NGX. This guide contains the specifications and detailed analysis of the Check Point High Availability (CPHA) and state sync packets.
In this chapter, we covered several different methods of troubleshooting various issues on Check Point gateways. When troubleshooting, check the obvious first. Make sure basic connectivity is established and the link layer is operating properly.
You can resolve most issues by studying the logs in the SmartView Tracker. If further analysis is required, Check Point’s built-in packet capturing utilities can gather packets for additional study. The capture format is libpcap, which is compatible with all of the popular packet analysis client software.
All of the Check Point daemons can run in debug mode for troubleshooting issues with internal functionality. Debug output is placed into text files with an .elg extension in the $FWDIR/log directory.
| The fw debug command is used to put various firewall daemons into debug mode. |
| The cpd_admin debug command is used for SIC troubleshooting. |
| For Layer 2 troubleshooting use snoop or tcpdump to capture packets. |
| The fw monitor utility built into the Check Point firewall can capture packets in multiple places. |
| The fw ctl chain command displays the path a packet takes through VPN-1. |
| Use the CPethereal or Wireshark utility to analyze packet data on a client PC. |
| Check Point logs are stored in the $FWDIR/log directory. |
| The fw log command allows for viewing of logs in a local command shell. |
| The $FWDIR/log directory can be empty when Check Point services start. |
| Study the Info field of the Check Point logs for Phase 1 and Phase 2 initiation details. |
| Use the vpn debug command for detailed IPSec analysis. |
| Verify that both sides of a VPN connection have the same settings. |
| Verify that the correct ports are open from the client to the firewall. |
| Use the srfw monitor command for client-side packet analysis. |
| If all else fails, look into SNX or the full-blown SSL VPN, Connectra. |