Like I mentioned earlier, the diagnostic tool will not identify any runtime issues that might occur during the runtime processing. The runtime issues could emanate from various sources including incorrect input data, insufficient system resources, and so on and so forth. In the next several sections let us discuss how to debug and troubleshoot the runtime problems by quoting specific issues.
Before even attempting to search for solutions on Google, please make sure you have done a decent level of debugging on your configuration. Typically, when something fails, you should look for clues in the following files:
- OpenSSO debug logs
- OpenSSO audit logs
- Web Container's logs
OpenSSO audit logs are by default turned on and available for you to view under the<config-dir>/<deply-uri>/log directory. Whereas the debug logs are not turned on for apparent performance reasons in the out of the box configuration. When you have run into problems, the first thing you would do is to turn on the debug logs by setting the following property using the admin console or by means of ssoadm CLI:
./ssoadm update-server-cfg -s http://opensso.packt-services.net:9090/opensso -u amadmin -f /tmp/.passwd_of_amadmin -a com.iplanet.services.debug.level=message
This will cause the server to run in debug mode; the value message means (for the property com.iplanet.services.debug.level) the highest level of verbose possible by the server. This will dump the entire conversation between the server components and clients in the debug files located under the<conf-dir>/<deploy-uri>/debug directory. In this case all the debug logs will be written into multiple files based on the functional components. For example, authentication-related debug logs will be stored in the file named Authentication. Categorizing the debug information is good but storing the debug logs in multiple files complicates the debug process as it is difficult to correlate the events at a given timestamp. To make the debug process more simple, one can configure the system to write all the debug logs into a single file by enabling the following property:
./ssoadm update-server-cfg -s http://opensso.packt-services.net:9090/opensso -u amadmin -f /tmp/.passwd_of_amadmin -a com.sun.services.debug.mergeall=on
This enables the server to consolidate the debug messages into a single file, debug.out, so the administrator can look into one file to extricate the sequence of events that occurred at any given time frame.
In general, both of these properties are hot swappable, so there is no need to restart the web container that hosts the OpenSSO web application. In the next section, let us discuss some specific problematic scenarios and the tips to debug them.
We'll consider a few scenarios to discuss what could go wrong here.
Symptom: While configuring the OpenSSO server the configurator failed, with the following message:
The application encountered an unexpected error:
[#|2010-08-13T15:21:05.955-0700|INFO|glassfish3.0.1|javax.enterprise.system.std.com.sun.enterprise.v3.services.impl|_ThreadID=24;_ThreadName=Thread-1;|amSDK:08/13/2010 03:21:05:954 PM PDT: Thread[http-thread-pool-8080-(2),10,Grizzly] ERROR: Crypt: failed to set password-based key java.security.NoSuchProviderException: no such provider: SunJCE atsun.security.jca.GetInstance.getService(GetInstance.java:66) at javax.crypto.SunJCE_b.a(DashoA13*..)
Possible Causes: These kinds of issues are common in the configuration phase of the OpenSSO server, sometimes this will happen during the authentication time as well. In this case the cause for the failure is relatively apparent as it complains about the SunJCE provider not being found. As this provider comes from the JDK it could not find the SunJCE provider in its $JAVA_HOME/jre/lib/security/java.security file. There is an exception for this when it comes into IBM Java Virtual Machine, which will be discussed shortly.
Tips for Troubleshooting: IF you are using Sun JDK then to fix this issue, add the SunJCE provider to the top of the list of providers. If it already exists, make sure that it is on top of the list, as follows:
security.provider.1=com.sun.crypto.provider.SunJCE security.provider.2=sun.security.rsa.SunRsaSign security.provider.3=com.sun.net.ssl.internal.ssl.Provider security.provider.4=sun.security.provider.Sun security.provider.5=sun.security.jgss.SunProvider security.provider.6=com.sun.security.sasl.Provider security.provider.7=org.jcp.xml.dsig.internal.dom.XMLDSigRI security.provider.8=sun.security.smartcardio.SunPCSC
It is easy to configure an already existing OpenSSO system to run in "debug" mode, but what if the system is being configured and you want to view the debug traces of OpenSSO? There is a way to do that in OpenSSO.
It is always useful to get the debug information of the configurator, especially if things are not moving well. When you encounter problems with OpenSSO while configuring them, just add the following Java VM options into your containers' Java virtual machine:
-D"com.iplanet.services.debug.directory=/tmp/debug" -D"com.iplanet.services.debug.level=message"
After adding these parameters restart the container and invoke the OpenSSO configurator. Now you should be able to view the debug logs from the /tmp/debug directory. This is one quick way to debug the configurator-related problems.
Symptom: Configuration of OpenSSO was successful without login problems. However, after restarting the login page returned the configuration page.
Possible Causes: When this happens, either the bootstrap file has been removed or is not readable by the web container's Java virtual machine process.
Tips for Troubleshooting: In order to recover back you should check a couple of things:
- Ensure the configuration directory is indeed present in the file system
- Ensure the configuration directory and the
bootstrapfile is readable by the web container's JVM process - Ensure the external configuration store (in the case of the configuration store is ODSEE) is up and running
- Ensure the OpenSSO server is not running out of resources such as file descriptors, process limits, and so on
It is not uncommon that once you restart the JBOSS application server, subsequent access to the OpenSSO server will show you the configurator page. Do not panic, this is something known, this is due to the ServletContext.getRealPath() method which does not always return the same value after the server is restarted.
To fix this problem edit the<JBOSS_DEPLOY_BASE>/server/default/deploy/opensso.war/WEB-INF/classes/bootstrap.properties file and update the property configuration.dir=<opensso-config-dir> where<opensso-config-dir> is the directory that contains the bootstrap file.
After performing the preceding step, restart the JBOSS server and you will be able to see the login page from OpenSSO. You can also use this property when the system user that is running the web/application server process does not have a home directory, that is, System.getProperty("user.home") returns null.
Symptom: OpenSSO web application deployed successfully, but failed to initialize with the following exceptions:
[#|2010-09-04T16:43:47.708-0700|SEVERE|sun-appserver2.1|com.sun.jersey.core.spi.component.ProviderFactory|_ThreadID=16;_ThreadName=Timer-6;_RequestID=94ec2927-49ac-495f-97ad-42bf54b1fbe2;|The provider class, class com.sun.jersey.core.impl.provider.xml.XMLStreamReaderContextProvider, could not be instantiated. Processing will continue but the class will not be utilized java.security.AccessControlException: access denied (java.lang.reflect.ReflectPermission suppressAccessChecks) atjava.security.AccessControlContext.checkPermission(AccessControlContext.java:323) at java.security.AccessController.checkPermission(AccessController.java:546) at java.lang.SecurityManager.checkPermission(SecurityManager.java:532) at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:107) atcom.sun.jersey.core.spi.component.ComponentInjector$2.run(ComponentInjector.java:143)
Possible Causes: These kinds of exceptions happen when the Java EE security is enabled in the container.
Tips for Troubleshooting: You can add the necessary Java EE permissions as documented in the OpenSSO guide to overcome this problem. One can also disable the Java EE security manager to troubleshoot when the problem is really with the security permissions.
There are many conditions that could trigger a runtime error in these components. Let us see some of the very common errors that occur in the authentication and session areas.
Symptom: Authentication keeps on failing even though the user ID and password are correct.
Possible Causes: There could be more than one user ID matching, for the given username. The LDAP authentication module configuration could be wrong, including incorrect LDAP server name/port, SSL handshake error, or a wrong search scope/base.
Tips for Troubleshooting: Make sure you properly configured the LDAP module to rule out the above possibilities. If there is more than one username matching, then enable the username uniqueness in the underlying LDAP server.
Symptom: OpenSSO server keeps on showing the login page even after submitting the valid user and credentials.
Possible Causes: Most likely this could be a browser or OpenSSO platform service cookie domain-related configuration issue.
Tips for Troubleshooting: Check your browser setting for JavaScript and cookie settings. Both should be enabled for the login process to be successful. For example, in some newer versions of Internet Explorer, JavaScript is not enabled in the default configuration. Next, check your platform service configuration to make sure it has a proper cookie domain value with a dot(.) prefix or a hostname if you are running on a standalone host.
Symptom: Windows NT authentication keeps on failing.
Possible Causes: This kind of failure happens mostly due to an issue in the connectivity between the OpenSSO server and the backend Windows domain.
Tips for Troubleshooting: While debugging these kinds of issues first make sure the connection between the OpenSSO server and the Windows NT is working properly. This can be done in a simple way by just pinging, to actually verify the Samba Client connectivity to the Windows Active Directory domain. There is a wealth of information available for troubleshooting the Windows Desktop SSO module (found at. http://wikis.sun.com/display/OpenSSO/Troubleshooting+OpenSSO+Enterprise+8.0+Windows+Desktop+SSO).
In order to debug the NT authentication issues, please follow these check lists:
- Ensure the
smbclientbinary is available under<conf-dir>/<deploy-uri>/bin. - Execute the
smbclient -W domainame -L hostname -U ntuser%passwd. For example:
smbclient -W AMQA -L ad-host -U andy%secret993
This should be successful with no errors despite all the NT authentication failures, then it must be OpenSSO issue that needs to be debugged.
Symptom: When the sessions are viewed from the session tab, some of the sessions show negative session time values, especially for the especially in the session failover setup.
Possible Causes: This kind of scenario could occur when one or more of the servers in the site are not properly time-synchronized with the other servers.
Tips for Troubleshooting: First ensure that the OpenSSO servers in the site are properly time-synchronized either through the NTP protocol or manually.
A few scenarios that are possible here are:
Symptom: I have added a custom attribute and objectclass in the backend LDAP server. I then created a new identity repository for the plugin in the OpenSSO, but when I try to create a user it does not contain the new attribute and objectclass. Similarly, I could not get this attribute retrieved from the OpenSSO clients, such as agents or REST calls.
Possible Causes: It could be a configuration issue. Either you have not added these custom attributes to the identity repository plugin configuration or there could be a typo in the attribute names, or else the underlying LDAP server does not allow these attribute values to be read.
Tips for Troubleshooting: First make sure the attributes have been added to the configuration and spelt correctly. If it still does not work, look out for any ACIs that are blocking the attributes being queried by the LDAP clients. The user in the identity repository plugin configuration should be able to search for these custom attributes.
Symptom: While performing identity repository-related operations the server returned the following response:
Plug-in com.sun.identity.idm.plugins.ldapv3.LDAPv3Repo encountered an ldap exception. LDAP Error 65: The requested operation will add or change data so that the data no longer complies with the schema.
Possible Causes: This message is thrown mainly due to a LDAP schema violation in the backend identity repository.
Tips for Troubleshooting: OpenSSO server reads the IdRepo configuration, and based on this configuration, it tries to perform a ADD/MOD/DEL/SRCH operation in the backend LDAP server. This message will appear only when ADD or MOD operation is attempted. When this message appears you should check the LDAP server to see whether schema exists for the attributes you are trying to set from OpenSSO. Consider checking for typos in the attribute names as well.
Symptom: Customer updated the attributes of an identity in the identity repository, but the changes are not shown in the administration console of the OpenSSO.
Possible Causes: Mostly this is due to the change notification issue.
Tips for Troubleshooting: In the IdRepo configuration there is a persistent search notification configuration item present, please check whether this property and the associated properties are set with a legitimate value for the IdRepo to submit a psearch request to the LDAP server. It is also possible sometimes that the backend LDAP servers could have maxed out the persistent search connections. In any case, the debug file IdRepo will contain the details of the persistent connections that were submitted to the LDAP server.
Symptom: Customer received the following message after attempting to reset the forgotten user password:
Password Reset Confirmation Could not connect to SMTP host: localhost, port: 25; nested exception is: java.net.ConnectException: Connection refused
Possible Causes: These kinds of messages typically indicate that the password reset application configuration is not set properly at the server side. The SMTP host name or port could be wrong.
Tips for Troubleshooting: Login to the administration console and check the SMTP port and the host name for the respective realm. It should be pointing to a valid SMTP server and port. If this is correct, then check whether the SMTP server and port are accessible from the OpenSSO server.
Symptom: Customer received the following message when trying to reset the end user password from the OpenSSO password reset web application:
Password Reset Confirmation Your password has been reset but we are unable to send it to you. Contact your administrator.
Possible Causes: These kinds of message will be shown in the browser when the end user does not have a valid e-mail address in their profile, or a runtime error could have occurred while attempting to send the password to the user's e-mail address.
Tips for Troubleshooting: First check the concerned user's profile for the existence of a valid e-mail address. If it is present, then you probably should check the SMTP server configuration. Try to send a test mail using a common mail client such as mailx.
In the Policy and agents area most of the issues could be resolved by time synchronizing between the OpenSSO server and agent hosts. There may otehr scenarios where the agent or server could run it to problems. The next few scenrios high light the common problems.
Symptom: Protected resource not accessible, the browser keeps on looping between the OpenSSO server and the agent.
Possible Causes: There are many reasons why one could run into these scenarios. It is basically because the server and agents could not communicate reliably due to a configuration issue.
Tips for Troubleshooting: The fault could be in the agents or server configuration or both. You could check the following:
- Ensure both server and agents are in the same domain. If they are in different domains, CDSSO should have been enabled.
- Make sure the protected URL is a FQHN (fully qualified host name).
- Make sure the browser is accepting the cookie.
- Clear the cookies in the browser and retry.
- Ensure that proper cookie encoding is enabled for certain containers.
- Make sure the domain cookie value is set with a prefixed dot in the server-side platform service.
Symptom: Customers noticing the following error after successful agent installation:
Exception starting filter Agent java.lang.ClassNotFoundException: com.sun.identity.agents.filter.AmAgentFilter at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1498) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:235) at org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:369) at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:103) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4381) at org.apache.catalina.core.StandardContext.start(StandardContext.java:5181) at com.sun.enterprise.web.WebModule.start(WebModule.java:327)
Possible Causes: Agent installer did not place the required JARs in the configuration folder due to permissions issues, or the configuration changes made by agent installation could have been overwritten.
Tips for Troubleshooting: You can check in the domain.xml/server.xml for the agents-related JARs in the classpath. Also make sure the proper server instance block has the agents JARs. In the glassfish server there are two configurations, one for the administration server, and the other for the customer applications. One should choose the correct instance to install the agents. Also if the domain is running during agent installation, then the changes made by agent installer will be overwritten the next time the glassfish server is restarted.
Symptom: My agents and server are configured perfectly, but the protected resources are always denied even though the customer is using the correct subject and condition.
Possible Causes: This could be due to time-synchronization issues between the agents and servers.
Tips for Troubleshooting: Check whether the OpenSSO server and the agent's machines are time-synchronized by means of a standard protocol such as NTP. You can also view the agent-side debug logs to see what the policy server response was.
In the next few sections discuss some of the frequently encountered issues in the command line area including the ssoadm tool.
Symptom: When the ssoadm tool is invoked, the customer receives the following message:
com.sun.identity.cli.CLIException: AdminTokenAction: FATAL ERROR: Cannot obtain Application SSO token. Check AMConfig.properties for the following properties com.sun.identity.agents.app.username com.iplanet.am.service.password at com.sun.identity.cli.LogWriter.log(LogWriter.java:109) at com.sun.identity.cli.Authenticator.ldapLogin(Authenticator.java:169) at com.sun.identity.cli.AuthenticatedCommand.ldapLogin(AuthenticatedCommand.java:158) at com.sun.identity.cli.serverconfig.ListServers.handleRequest(ListServers.java:59) at com.sun.identity.cli.SubCommand.execute(SubCommand.java:290) at com.sun.identity.cli.CLIRequest.process(CLIRequest.java:212) at com.sun.identity.cli.CLIRequest.process(CLIRequest.java:134) at com.sun.identity.cli.CommandManager.serviceRequestQueue(CommandManager.java:568) at com.sun.identity.cli.CommandManager.<init>(CommandManager.java:164) at com.sun.identity.cli.CommandManager.main(CommandManager.java:141) AdminTokenAction: FATAL ERROR: Cannot obtain Application SSO token. Check AMConfig.properties for the following properties com.sun.identity.agents.app.username com.iplanet.am.service.password
Possible Causes: This issue may be due to the fact that the OpenSSO server is not directly accessible by the ssoadm tool. Most likely the servers are behind a load balancer or a firewall.
Tips for Troubleshooting: You may want to try the option suggested in Chapter 2. That is use of -D"com.iplanet.am.naming.map.site.to.server".
Symptom: While invoking the setup program or ssoadm tool (or the amverifyarchive or ampassword utility) in the IBM Java environment, the following exception is thrown:
Failed to create debug directory java.security.NoSuchProviderException: no such provider: SunJCE atsun.security.jca.GetInstance.getService(GetInstance.java:82) at javax.crypto.b.a(Unknown Source) at javax.crypto.SecretKeyFactory.getInstance(Unknown Source) atcom.iplanet.services.util.JCEEncryption.setPassword(JCEEncryption.java:369) atcom.iplanet.services.util.Crypt.createInstance(Crypt.java:126) at com.iplanet.services.util.Crypt.initialize(Crypt.java:97) at com.iplanet.services.util.Crypt.<clinit>(Crypt.java:89)
Possible Causes: Unlike the previous one that we have discussed in the beginning of this chapter this one is specific to IBM Java, and that has to be fixed in a different manner as described in the following section.
Tips for Troubleshooting: In this scenario it is very clear that the problem was due to the missing Sun JCE provider, but we can force the tool to use the IBMJCE provider by adding the following JVM options in the script:
-D"amCryptoDescriptor.provider=IBMJCE" -D"amKeyGenDescriptor.provider=IBMJCE"