After learning how to architect a custom Zscaler Internet Access (ZIA) solution, it is time to put that solution into a day-to-day operation. To make the most out of the deployed ZIA solution, the enterprise administrator needs to take care of a few aspects.
Anyone who has been in any type of steady-state operation will almost immediately tell you that it involves working with trouble reports initiated either proactively created by a network monitoring tool or reactive tickets created by end users over the phone or through a web portal.
These trouble tickets are then routed to a generic help desk, where the associates need to know how to identify a potential Zscaler problem and engage the proper points of contact. For this reason, there needs to be a comprehensive and standardized troubleshooting process documented for the help desk. This documentation needs to include the various points of contact within the enterprise, such as the firewall team, the network team, IT operations, and the proxy team. The troubleshooting process also needs to have one or more customized flows described for each type of problem. Otherwise, it might create a nuisance if everyone is engaged for every problem.
When the trouble is finally escalated to the appropriate Zscaler enterprise administrator, the administrator will have to start at a baseline. The dashboards offered by the ZIA Admin Portal is the first place the administrator should reference. Custom widgets can be created by the administrator to suit the enterprise's needs. Streamlining this entire end-to-end operations process enables the enterprise to provide a great end user experience while saving on costs.
In this chapter, we are going to cover the following main topics:
Knowledge of operating system commands, such as command prompt, terminal, ping, and traceroute, is helpful to gain an understanding of the content in this chapter.
We have already mentioned that reactive tickets are usually created when end users call the help desk or via an online ticketing portal. In this section, we will look at the various ticketing and alerting options available with ZIA. First, let's begin with the native alerting mechanism offered by the ZIA Admin Portal.
The enterprise administrator can log in to the Admin Portal and then navigate to Administration > Alerts. On the first tab, Define Alerts, click on the + icon to add a new alert definition. The resulting pop-up window has the following options:
On the second tab, Publish Alerts, the alert defined just now can be sent to the interested subscribing parties. Click on the + icon to Add Alert Subscription. The pop-up window that appears has the following options:
Under each of the types of alert class, select the severity level that this subscribing party will receive. For example, the Tier 1 help desk can subscribe to the Minor level of alerts and the dedicated security team can subscribe to the Critical alerts. This way, the alert levels can be segmented based on the subscribing party.
Under the third and final tab, Global Configuration, the frequency of the alerts can be set. For the Resend Active Alerts Every option, the drop-down choices are 30 minutes, 1 Hour, 6 Hours, 12 Hours, and 24 Hours. The administrator can choose the right option based on the security needs of the enterprise.
The Admin Portal does not offer any native ticketing tools. If the organization has a need for proactive ticketing, they need to set up their SIEMs to receive the real-time web and firewall logs from Zscaler NSS virtual machines, and then build the event correlation and ticket generation logic into their SIEM and integrate it with their preferred ticketing platform.
The ZIA Admin Portal offers several types of NSS feeds that can be consumed by the SIEM:
Next, let's find out what kinds of reports can be generated for management review purposes.
From time to time, the enterprise administrator may be called upon by upper management to generate easy-to-understand reports that show whether the Zscaler ZIA solution is working optimally for the enterprise. ZIA Admin Portal makes it easy to create industry-standard and customized reports. Let's explore the built-in reports.
ZIA Admin Portal offers several types of default, system-defined reports. The two most common choices are the Executive Report and the Industry Peer Comparison Report. Let's see what is in each type of report.
The executive report contains an overall security view of the enterprise in an HTML format. It shows the value derived from the ZIA service. It contains details such as how many security threats and/or company security policy violations were detected for the enterprise during a certain time frame.
After logging into the ZIA Admin Portal, the administrator can go to Analytics > Reporting > Executive Reports. Once the report loads, the administrator can click on the time frame dropdown on the top left of the page and adjust it as needed. The report thus customized by time frame can then be printed by clicking on Print on the top right of the page.
If this report needs to be sent to management contacts on a periodic basis, click on Schedule on the top right of the page, just to the left of Print. Enter the details for the following options on the pop-up window.
Any further changes can be done by clicking on Edit Scheduled Report that is shown after a report has already been scheduled. Let's move on to the Industry Peer Comparison Report. Please note that this report will be discontinued and Zscaler recommends using the Executive Insights Email Report from the Analytics menu instead.
A second, popular built-in report is the Industry Peer Comparison Report. Often, security tool vendors provide an insight into how an enterprise compares to the industry (such as banking or manufacturing) it is in. This report provides those details and in addition, it also provides a comparison against all organizations using the ZIA service.
To get to this report, the enterprise administrator must log into the ZIA Admin Portal and navigate to Analytics > Reporting > Industry Peer Comparison. If this report looks inaccurate, then the administrator needs to click on the blue i icon right beside the Industry Peer Comparison Report in the upper-left corner. This shows the Business Vertical, Geographic Region, and Business Size for the enterprise. If this information is not correct, it needs to be corrected by contacting Zscaler.
There are several other types of reports that are available or being added by Zscaler on a regular basis. A list of reports from any given time can be found by logging into the ZIA Admin Portal and navigating to the Analytics > Reporting section or by consulting the ZIA documentation.
In addition to the built-in reports, the ZIA Admin Portal offers an on-demand query option, called Insights. Let's take a look at the various available Insights options.
Insights are sort of like on-demand, customizable reports that can be generated by the enterprise administrator. They are especially useful when troubleshooting a specific problem. They can be accessed by navigating to Analytics > Insights. The most popular Insights option is Web Insights.
Once this option is selected, the administrator can adjust the following choices.
Once the administrator makes the necessary adjustments and clicks on Apply Filters, the appropriate data is shown on the right-hand side that matches the criteria. This on-demand report can be printed using the Print icon on the top right-hand corner of the page. By default, the data shown is for Overall Traffic and it can be further narrowed down using available suboptions.
Alternatively, the Logs option (in the top-right corner beside the Insights option) can be selected to view and download the raw logs that match the same criteria. The timeframe can be narrowed down to as little as the Last 1 Minute and as large as the Previous Month. There is also a field called Number of Records Displayed that can be used to limit the number of records displayed on the screen. Underneath the Select Filters option, there are additional options (compared to the Insights options) such as Client IP, and Device Owner.
Be careful as they may run into several thousand entries. Narrow your criteria as much as you can before proceeding to download the logs in a Comma-Separated Value (CSV) format, which can then be used for further offline analysis. The maximum number of records that can be downloaded is 100,000.
Let's now look at how to customize the widgets on the ZIA Dashboard.
As soon as the enterprise administrator logs into the ZIA Admin Portal, the Web Overview dashboard is displayed by default. This dashboard offers some predefined widgets. These widgets can be edited or deleted, and new custom widgets can be created. Let's take a look at that customization process.
The Dashboard page is loaded by default when an enterprise administrator logs into the ZIA Admin Portal so, there is no special navigation required after login. Current widgets on the dashboard can be edited by hovering the mouse near the top-right corner of the individual widget to reveal a pencil icon. Click on the pencil icon, and you will see two options presented there: Edit Widget and Remove. Clicking on Remove will ask for confirmation before proceeding with the deletion of this widget. Clicking on the Edit Widget will open a pop-up menu with the various options. We will discuss those options in the next section.
On the top right of the Dashboard page, click on the blue + icon to add a new widget to the dashboard. The options available on the resulting pop-up window are as follows.
The top menu bar specifies the type of data that the widget will display. The default selected option is the Web. The other options are Mobile, Firewall, and DNS:
Once all these fields have been entered, click on the blue OK button to add this widget to the dashboard. Let's now look at the ZIA troubleshooting guide creation.
Either through proactive alerting or reactive ticketing, trouble tickets eventually reach the help desk, and they must be worked upon to resolution. When the enterprise adopts a logical and consistent troubleshooting approach, the resolution time for these trouble tickets can be decreased, thus alleviating the pressure on the Zscaler enterprise administrator.
The basic information that should be gathered by a help desk associate applicable to many common scenarios is as follows.
If the end user can log into their computer using domain credentials, they should be asked to open a company-approved internet browser and navigate to ip.zscaler.com. This can tell us if the end user is accessing the web using the ZIA service or through an alternate path. If the end user is not going through ZIA, this web page will say "The request received from you did not have an XFF header, so you are quite likely not going through the Zscaler proxy service."
If the end user is indeed traversing through the ZIA service for their web traffic, the ip.zscaler.com page will say "You are accessing this host via <> hosted at <> in the <cloud name> cloud," and provides the following additional details. The end user can then either take a screenshot type in these details into a notepad or an email client:
Depending on the operating system platform of the end user, get them to run the ipconfig /all or ifconfig command (on Command Prompt in Windows and Terminal in Mac, respectively), which tells us whether the end user has a valid IP address to be able to access the internet via the network they are connected to.
If the end user does not have a valid IP address, check whether their wired or wireless connection says, "Media disconnected". If the physical connection is OK, then check whether the end user has a valid IP address as per the enterprise DHCP allocation scheme. If not, check whether the end user can release and renew the IP address lease. In the case of a failure, check the connection to the enterprise DHCP server or engage the appropriate DHCP support team.
If the end user has a valid IP address, check whether they can ping the default gateway. If that does not work, engage the network team or the IT support team as needed. If the ping to the default gateway works, proceed to the next section.
In the same Command Prompt or Terminal window, have the end user run a ping command and a traceroute (tracert on Windows) to a well-known URL, such as www.google.com, and to the destination IP address to see whether the Layer 3 connectivity and DNS resolution are working. If the ping to www.google.com does not work, or the DNS resolution fails, engage the DNS support team.
Sometimes, in addition to the steps that we listed in the Basic troubleshooting section, Zscaler Support may ask for additional troubleshooting information. Two of the commonly used pieces of information are the HTTP headers and the ZCC App packet capture. Let's gain a better understanding of them now.
When an end user requests a web page using the HTTP protocol, capturing the HTTP headers from the user request gives us more details, such as the exact URL requested by the end user. The three most popular browsers, namely, Google Chrome, Mozilla Firefox, and Safari, offer additional developer tools that enable this capture. The links to each of these have been provided in the Further reading section of this chapter.
The ZCC App offers a packet capture of the end user traffic at a lower level and, hence, captures traffic that may be missed by other packet capture tools such as Wireshark. For the packet capture to work in the ZCC App, the end user must use version 1.3, or later, and this captures the packets at the driver level. The restriction is that only packets that are passing through the filter driver are captured. For ZCC App version 1.5, or later, all packet captures are done at the adapter level. To turn on this feature, and for instructions on how it works, please refer to the link in the Further reading section of this chapter.
Now that we are done with the common troubleshooting procedures, let's look at various scenarios that could be referenced in a problem ticket.
The end user may report that they are unable to access websites. The message they may be getting is that they are not connected to the internet. Follow these troubleshooting steps:
If trouble remains, engage higher-tier support.
In this scenario, the end users accessing certain website categories could be getting a message that says "Website blocked." Please follow these troubleshooting steps:
Next, let's take a look at the captive portal error.
In this scenario, the end user is not at a trusted location and is trying to access the internet from a public Wi-Fi, such as a hotel, airport, or a café. Such public Wi-Fi access points typically ask the end user to acknowledge their Acceptable Use Policy (AUP) by clicking on Accept or OK. Follow these steps:
If the issue persists, escalate it to the Zscaler enterprise administrator.
When the ZCC App displays a Network Error message, it usually means that an active network interface was not found. Follow these resolution steps:
If trouble persists, engage higher-tier support.
This error message displayed by the ZCC App usually means that it has encountered an internal socket error. Follow these steps to address this issue:
If the issue persists, engage higher-tier support.
A connection error usually means that the ZCC App is unable to reach the Zscaler PSE. Follow these steps to try and resolve this issue:
Next, let's examine a local firewall or anti-virus error.
This error occurs when the ZCC App is being blocked by a local anti-virus application or a firewall. Follow these steps:
If the issue persists, engage higher-tier support to see if they can whitelist the ZCC App in the anti-virus and/or the local firewall.
This error usually means that the ZCC App cannot enable the tunnel interface because of a possible driver installation failure. Attempt these resolution steps:
If the trouble persists, engage higher-tier support.
Usually, end users first authenticate using the Identity Provider (IdP) and Security Assertion Markup Language (SAML) which enables the exchange of this authentication and authorization information between the IdP and Zscaler. Let's take a look at a family of SAML, Lightweight Directory Access Protocol (LDAP), and Kerberos authentication errors experienced by the end users.
Transit errors are usually temporary and clear on their own. In this scenario, the end user gets one of these error codes – E5503, E5507, E5508, E5611, E5612, E5614, E5619, E5623, E5629, A002, A003, A00C, A00D, A00E, A011, A019, A023, A029, or A02A. Follow these steps:
If the issue persists after several minutes, engage higher-tier support.
Account errors are encountered by users due to account-related issues such as the account is not activated, disabled, or never provisioned. Check whether the end users report one of these error codes – E5616, E5621, E5624, E5628, or A010. Follow these steps:
If trouble persists, engage higher-tier support.
Login format errors occur when end users do not use the proper format for their username during authentication. In these scenarios, they get an error code of A021. Follow these resolution steps:
If trouble persists, engage higher-tier support.
When the enterprise is using LDAP as the configured authentication method and end users experience authentication errors due to a wrong password, they usually get an error code of 101. Follow these troubleshooting steps:
If trouble persists, engage higher-tier support.
If the enterprise is using LDAP and the end user account is not activated or found on the LDAP server, then they will experience account-related errors. They might get one of these error codes – 103, 106, or 113. Follow these resolution steps:
If trouble persists, engage higher-tier support.
LDAP transit errors usually clear on their own. In these situations, the end users get one of these error codes – 102, 107, 111, 114, or 115. Use these troubleshooting steps:
If trouble persists, engage higher-tier support.
When the enterprise is using Kerberos as the authentication method and the user's account is not activated or found on the server, they can experience account authentication errors with one of these error codes – 441000 or 461000. Use these troubleshooting steps:
So far, we have looked at SAML, LDAP, and Kerberos authentication errors. Let's now examine some more trouble scenarios that might be seen by the help desk.
In this scenario, end users report the inability to download or upload files to certain websites such as Google Drive or Dropbox. This could happen either due to a ZIA-enforced policy or a browser compatibility issue. Follow these troubleshooting steps:
Engage the next tier support or Zscaler administrator as needed.
In this scenario, end users report slow loading websites or erratic video streaming. This could be caused due to browser compatibility issues or due to a bandwidth enforcement policy of the enterprise. Follow these steps to troubleshoot these issues:
If trouble persists, engage higher-tier support or a Zscaler administrator so that the link bandwidth can be checked for contention or a Zscaler bandwidth enforcement policy.
In addition to the issues seen by the end users, it is worth mentioning a few issues that could be seen by ZIA administrators.
When the ZIA administrator is whitelisting certain domains, it is important to understand the behavior of non-Zscaler proxy solutions such as Bluecoat or other similar products. In those products, if a domain (such as example.com) is whitelisted, the product automatically and implicitly whitelists all the subdomains (such as download.example.com and mail.example.com).
However, in ZIA, if you need to whitelist multiple subdomains, you need to specify a dot or a period (.) to the left of the URL, and it will allow up to five subdomains deep. Failure to understand the behavior could result in the accidental blocking of the intended subdomains.
The ZIA administrators need to be aware of the behavior of the URL and firewall filtering policies when SSL inspection is turned off for a specific location or a sublocation. In such cases, the firewall and URL filtering policies may fail intermittently.
Many legacy enterprise applications do not support SAML or interactive authentication. As such, it is not recommended that you use service accounts as the username as the application authentication may fail inside the proxy. In these cases, the solution is usually to bypass authentication and create a separate location or sublocation for such sources.
In this chapter, we looked at the various built-in tools provided by the ZIA Admin Portal, by default, and customizable tools to suit the needs of the enterprise operations staff. They come in various forms such as Insights, Logs, Dashboards, Reports, and more.
We also looked at some of the most frequently seen troubles with the ZIA end users and how a streamlined troubleshooting approach can help the enterprise resolve them effectively.
In the next chapter, we will start addressing the Zero Trust Network Architecture (ZTNA) using Zscaler Private Access (ZPA).
As we conclude, here is a list of questions for you to test your knowledge regarding this chapter's material. You will find the answers in the Assessments section of the Appendix:
a. True
b. False
a. True
b. False
a. End user errors
b. Misconfiguration errors
c. Provisioning errors
d. None of the above
e. All the above
3.141.27.244