Before you attempt to deal with a problem, you need to first determine what the problem is and what the scope of the problem is. You need to know what you are getting yourself into so you can properly allocate time and resources for the task.

NDS/eDirectory problems can manifest in a number of ways. They can appear directly in the form of error or warning messages in utilities or on the server console. They can also show up indirectly as part of an error message when you attempt to perform a DS operation. Sometimes a DS error shows up as a side effect of other system or network problems. For instance, a -625 error is a DS error that indicates communication failure. However, the cause of this error is not DS itself but has to do with the subsystem that is responsible for data communication—which DS has no control over. Therefore, you need to be able to differentiate between the symptoms and the problem.

Users tend to cry wolf fairly easily. Therefore, before jumping to any conclusions, you should not panic; rather, you should confirm the error condition by either looking for concrete evidence of a problem or try to duplicate the problem. When you have established that there is indeed a problem, you must determine the extent of that problem.

You need to determine two facets of the problem. First, you need to determine the size of impact, such as how many people the error is affecting (including whether the CEO is one of the affected users) and the potential dollar cost of the error to the company. You also need to determine in which subcomponent the error condition happens. You can use these two facets to help prioritize your work schedule and troubleshooting efforts.

Our experience indicates that most of the time the DS errors you are mostly likely to encounter fall into the following three categories:

During the discovery process, you should check each of these areas in the order listed. You should make note of the following information, as it will help you determine the likely cause of the error:

It is important to quickly ascertain whether the error condition is occurring at the server, partition, replica, or object level. For instance, assume that the error code you receive is -601 when you’re trying to read the Title attribute of a User object from Server A. Do you get the same error when you try to access the Title attribute from another server in the replica ring? If you do not get the error when you’re trying to access this attribute from another server, the likely source of the problem is Server A itself. Otherwise, all the servers in the replica ring may be suspect.

eDirectory reports problems or generates errors if a certain condition within the network prevents one or more eDirectory background processes from starting or completing. In most cases, your only indication of a DS error is in the form of an error code, which you see only when you perform a health check or enable the DSTrace screen. The following is a list of common DS errors:

To determine or narrow down the cause of the problem, you need to first ascertain whether it is an actual eDirectory problem or some other type of error that is manifesting itself as an eDirectory issue. When you are certain that the problem is indeed an eDirectory problem, you need to analyze the information you have gathered about the error condition and narrow it down to the particular background process that caused it. Then, using the various resources at your disposal, such as online error code listings that show possible causes and the Novell Knowledge Base, you should try to gain a handle on the real cause of the problem.

You can almost always use more than one method or tool to fix a given eDirectory problem. When you search the TIDs for a possible solution to a particular problem, you shouldn’t be surprised to find multiple documents that suggest different ways to resolve the problem. However, before you implement any of the solutions, you should do the following:

When installing a patch, you should always select to back up files that will be replaced when the installer prompts you. If the installer does not offer such an option, you should abort the update process, perform a backup of your system files (including those on the DOS partition, in the case of NetWare servers), and then restart the patch update. Furthermore, it is always good practice to keep a library of older patches that you have installed in case the new patch doesn’t fix your problem or causes more problems; this way, you can roll back to an older patch that you know to work.

When you have a list of possible solutions to the problem you’re experiencing, you then need to rank them based the following criteria:

There is a medium to high degree of risk that if a corrective action does not work, it may lead to more harm to the system, and a sign-off from upper management is often warranted. You should present to management the rational for selecting the action but also lay out for them the possible consequences and rollback options.

If at all possible, you should test solutions in a lab environment before attempting them on a production network. At the very least, you should always have a test server available so you can dry-run a procedure before attempting it on a production server.

When you have decided on a course of action and received the necessary management approval, you need to make a backup of the Directory Information Base (DIB) files on the affected servers before implementing the fix. This will provide you with a back-out option in case something goes wrong.

You can find information on how to do this in the “Backing Up a Server’s DS Database” section of Chapter 13, “eDirectory Health Checks.”

If you are running eDirectory 8.7 or higher, you can use either the hot backup or the cold backup feature of eDirectory Backup eMTool, as discussed in Chapter 8, “eDirectory Data Recovery Tools,” to take a snapshot of the current view of the replicas on the servers that you will be working with.

The effects of your corrective actions may not show up immediately. You should allow some time for the various eDirectory background processes to perform their tasks. For example, assume that you were unable to merge two partitions because of stuck obituaries. After implementing your fix, you might not be able to merge the partitions right away because you need to allow the obit processes to advance the flags through the different stages and clear out the dead objects. Because this generally involves communication between a couple servers, at least a few minutes will be required before the stuck obits are cleared out. After that, you can perform the partition merge.

After the eDirectory processes report “All processed = YES” (and you might want to wait until you see this being reported a couple times to be on the safe side), you should verify that the problem has indeed been resolved. You can accomplish this in one of two ways:

If the problem persists, you should return to your list of possible solutions and consider taking another action. You might need to go back and reexamine and reconsider the possible causes of the problem to ensure that you are on the correct track.

You should keep a logbook that contains information about your servers, network configuration, and any special NDS/eDirectory information (such as any special naming convention used for designating administrative users or ACL assignments made to protect admin users). This logbook should also contain maintenance data about your servers and network, such as what operating system or NDS/eDirectory patch was applied and when. This allows you to determine whether the DS or other error condition may be a result of one of these updates. It is also a good idea to keep a copy of the patches installed as part of the logbook so you have ready access to older patches in case you need to roll back an update.

In your logbook, you should also document the DS errors that have occurred, their causes, and the solutions you have identified. Even if an error was not resolved by a given solution, you should note it for future reference and label it as an “unlikely” solution. Whatever you do, you shouldn’t totally discount it from consideration the next time around because the same or a similar error condition could show up again. Even if a solution didn’t work the first time around, it might very well work the next time because the condition that causes the same or similar error message or code might be different the next time around.

Depending on the type of error, you are likely to encounter it again sometime down the road. For instance, no matter how careful you are about your tree, stuck obits will continue to surface from time to time. Having the steps on error resolution clearly documented helps to ensure that if you need to solve a given problem, you have a “play book” that you can follow.

It is estimated that more than 90% of the software-related problems you encounter are due to human error. They can be caused by unintentional actions taken by an inexperienced administrator or procedures not being properly followed. They can also be due to a number of other causes. For example, someone might simply turn off the server and remove it from the network instead of first removing DS from the server before removing it from the network.

Once you have identified the source of a DS problem, you should review existing published procedures to ensure that any oversights or omissions that resulted in the error are promptly amended, and you should be sure that the information is passed on to your co-workers.

At the end of a problem-resolution session, depending on the scope of the error involved, you should hold a meeting with your support staff to go over lessons learned and discuss how the problem can be avoided in the future. You should not, however, use such a meeting to assign blame of any sort.

NDS iMonitor first shipped with eDirectory 8.5 and is available for all supported operating system platforms. It provides a Web-based portal for accessing server-centric eDirectory information, using a Web browser from any workstation on the network.

You don’t need to hunt through a number of different screens in DSRepair and DSBrowse for the necessary information because NDS iMonitor presents you with the easy-to-locate data about a partition, a replica, or a server. Figure 9.1 shows the synchronization status for a partition; you can obtain the server-centric replica status (see Figure 9.2) by clicking the Replica Synchronization link to the right of the partition name. By clicking the NCP server object name in the Replica Status screen, you can retrieve information about the server.

FIGURE 9.1 The partition synchronization status screen.

FIGURE 9.2 The Replica Status screen.

When diagnosing a DS problem, you generally make use of the Partition Synchronization and Replica Synchronization screens in NDS iMonitor. You start by looking at the big picture at the partition level. When you notice a partition showing a nonzero error count, you can drill down to the replica level. From there, you can examine each of the servers in the replica ring that are reporting DS errors. Depending on your screen resolution, you might need to scroll to the far right of the entry in order to see the error code. Then you click the error code listed to see its meaning, possible causes, and possible actions to take. You can also enable DSTrace with various filter settings to gather additional diagnostic information (see Figure 9.3).

FIGURE 9.3 The DSTrace configuration screen.

When trace is enabled, you can use the additional Live Trace link to view the DSTrace messages in real-time, as shown in Figure 9.4. You can adjust how often the screen is updated; the default is every 15 seconds.

FIGURE 9.4 Live DSTrace messages.

You can also use the Agent Health link and its sublinks to get a quick overview of the status of the partitions and local replicas. Figure 9.5 shows the overall health status of the local DS agent (DSA).

FIGURE 9.5 DSA health status.

If you do not have time to perform health checks on a regular basis, you can use NDS iMonitor to periodically check the agent and replica/partition health status and run health reports on each server. Refer to Chapter 13 for more information about performing DS health checks.

Similar to NDS iMonitor, iManager is a Web-based management utility from Novell. It, too, first shipped with eDirectory 8.5 and is available for all supported operating system platforms. Designed to be a replacement for ConsoleOne and NetWare Administrator, iManager does not require a Novell Client to be installed on the workstation—all its functions are performed through Tomcat servlets that are running on the server.

iManager provides a functionality called role-based services (RBS) that uses the concepts of roles (what the job functions are) and tasks (responsibilities associated with the job) such that members of the role will see only the tools necessary for their assigned roles. Two predefined DS management-related roles are included with iManager:

The tasks associated with these roles are identical to many of the DSRepair and DSMerge functions, and you can access them through simple wizards. As you can see in Figure 9.6, you have access to the same repair functions in iManager as you do in DSRepair—tasks such as single object repair and local database repair. You also have access to advanced repair functions such as schema maintenance and replica ring repair. Also available are other functions, such as tree merge (a function otherwise provided by the DSMerge utility).

FIGURE 9.6 iManager’s basic repair options.

By using the View Partition Information task (see Figure 9.7) and Sync Repair task (see Figure 9.8), you can easily get a quick, overall view of your tree’s health. The wizards offer some advanced options in DSRepair but do not allow you to use any of the special command-line options, such as -XK2. However, you can pop over to NDS iMonitor easily via the Repair option under the iMonitor task, and you can use DSRepair command-line switches there.

FIGURE 9.7 Viewing partition synchronization status.

FIGURE 9.8 Viewing server synchronization status.

Instead of using Tomcat servlets, iManager accesses the eDirectory database via the eDirectory Management Tool Box (eMBox) utility. This means that if you have the proper DS rights, you can perform all the RBS-based eDirectory Maintenance and Partition and Replicas tasks without using iManager by using the eMBox client instead. Because the eMBox client is Java based, you can run it from a server or workstation and perform tasks for a number of servers (as long as they are running the eMBox server, which is part of eDirectory 8.7 and above).

You can perform a large number of repair tasks by using the eMBox client. When you’re performing DS troubleshooting and repair procedures, it is easiest—and best—to run it in the graphical mode, as illustrated in Figure 9.9. You don’t want to use the command-line text mode and enter a wrong option mistakenly or leave out a necessary flag and have some crucial information not be reported.

FIGURE 9.9 Performing a local database repair by using the eMBox client.

The one drawback of using the eMBox client instead of iManager or other tools (such as DSRepair) is the inability to browse for an object. You need to know in advance the distinguished name (DN) of the object/partition or the object’s ID for certain operations and enter that manually (see Figure 9.10). This can be a dangerous limitation when you are in a rush or are under pressure to expedite a fix.

FIGURE 9.10 Manually entering information in the eMBox client.

Web-based tools such as NDS iMonitor and iManager are great when you are working from a remote location. However, sometimes, network security policies (such as firewall configuration) or communication disruptions force you to work directly at the server console, where using a Web-based tool becomes unpractical. Therefore, you should also be familiar with using the server-based “legacy” tools: DSRepair, DSTrace, and so on.

The other advantage of these server-based tools (with the exception of Windows versions) is that they are generally “processor efficient” because they are text based. As a result, these tools are generally fast in nature, and most of the processor cycles they consume go directly to collecting and reporting DS information. Note that the Windows versions of these server-based tools are also not overly processor unfriendly. The graphics (dialog boxes and so on) use standard Microsoft Foundation Class APIs and are fairly processor efficient. When compared with text-based applications, however, the GUI-oriented ones always seem to be not as efficient. The tradeoff is in the data representation: It is a lot easier to gain an overall picture of the situation from a graphical view than by having to wade through lines of messages for the necessary information.

When you use server-based tools for troubleshooting DS issues, you should generally start with DSTrace because it is non-intrusive (that is, it doesn’t modify the DIB files in any way) and can point to the general problematic area where you should focus your attention. From there, you can move on to using DSRepair to get a closer view of partition and replica status. The following sections cover how you can get the most information out of the available tools by using a number of them so you can see the problem from different angles.