Troubleshooting and tracing
A problem can manifest itself in many different ways. Often, the root cause of the problem is not obvious. This chapter describes an approach to problem determination for the IBM Content Manager OnDemand (Content Manager OnDemand) system administrator. It guides you through the initial steps in problem diagnosis. In addition, this chapter helps you gather the documentation that is most likely required by the IBM Support Center for further diagnosis.
In this chapter, we cover the following topics:
18.1 Troubleshooting common problems
While administrators and users work with Content Manager OnDemand systems, they might encounter various problems. These problems can occur in several main areas. We classify them into the following categories:
In this section, we describe several of the common problems and provide possible solutions to them. At the end of the section, we also include a list of common server messages.
 
Tip for determining the cause of the problems: For the UNIX platform, the console message might help determine the cause of the problem. However, if you use Telnet from your personal computer, you might miss the important console message. For AIX, you can switch the console to your current terminal by running the swcons 'tty' command. To switch it back to the console, run the swcons command.
18.1.1 Client issues
Users often encounter the following common problems when they run client-side applications:
•Problem: Client connection received the error Server failed while ....
Reasons: The following areas might cause this problem:
 – Server is not up, or the server is up but not responding to a request
 – TCP/IP problems between Windows and the OnDemand host
 – Protocol problem
 – Firewall problem
Resolution: Check the following conditions:
 – OnDemand library server is up, and it accepts requests. For example, log on to the OnDemand host and issue an arsdoc query against the OnDemand system log.
 – You can ping the OnDemand host. If not, consult your OS support.
 – The port OnDemand server is listening and ready for a supported protocol. For example, issue the netstat -tulpn command.
 – The Linux fire wall is not on. For example, use the systemctl status firewalld command to turn off the firewall by issuing the systemctl disable firewalld command.
•Problem: Content Navigator or a custom application encounters an error with OnDemand.
Resolution: Test the scenario with the OnDemand Windows client first. If the problem cannot be reproduced, turn on the OnDemand Web Enablement Kit (ODWEK) trace. (See 18.2, “Information collection” on page 389). After the ODWEK application programming interface (API) in question is identified from the trace, run a stand-alone program to test the API. (A sample program can be requested from IBM.)
18.1.2 Indexing and loading issues
The following common problems are encountered while users index or load documents:
•Problem: When you attempt to index a report with a large record length, you see the error message that is shown in Figure 18-1.
0425-422 AN ERROR OCCURRED WHILE ATTEMPTING TO READ /filename RETURN CODE 310.
Figure 18-1 Error message with return code 310
Reason or resolution: The maximum record length, which is 32 KB, for AFP Conversion and Indexing Facility (ACIF) might be exceeded.
Return code 310 explanation: An attempt to read a dataset failed. This message is informational and further action takes place in higher-level modules, if required. The file format is not valid.
•Problem: The arsload program performs progressively slower over time.
Reason or resolution: Performance problems can be caused by various reasons, and they require careful examination. Content Manager OnDemand issues an SQL DELETE against the ARSLOAD table before Content Manager OnDemand adds that same information to the ARSLOAD table to guarantee uniqueness. Duplicate information cannot be in the ARSLOAD table. This SQL DELETE is a single action against the ARSLOAD table and it uses an index that is formed from the application group identifier (AGID) and NAME.
The ARSLOAD table has two indexes:
 – ARSLOAD_NAME_IDX, which contains the AGID and NAME
 – ARSLOAD_IDX, which contains the AGID and EXP_DATE columns
Without the ARSLOAD_NAME_IDX, each load performs a complete table scan of the ARSLOAD table.
Check to see whether you already have these indexes. In addition to index creation, gather statistics by running the following command:
arsdb -I <INSTANCE_NAME> -mv
•Problem: The arsload program terminates when an unrecoverable error occurs during index, database, or storage manager processing.
Reason or resolution: Open the Content Manager OnDemand system log folder and view the messages that the arsload program generated during the load process. Search for message number 88 in the system log.
If the arsload program failed during indexing, correct the problem and then restart the load process from the beginning. Common causes of problems during indexing include invalid input files, invalid indexing parameters, invalid indexing parameter files, and insufficient temporary space.
If the failure occurred during database processing or storage manager processing, check the database or storage manager for errors.
Check the message log for a load ID that the arsload program saved in the system log. Before you attempt to reload the input data, you must remove the data that was created during the failed load process by using the UNLOAD function of the arsadmin program.
Restart the load process from the beginning.
•Problem: Content Manager OnDemand indexing fails when only one field is defined for an application group.
Reason or resolution: The Content Manager OnDemand file name indexing feature needs at minimum one index and a field value that are defined in the application indexer parameters.
Verify that you are using a file name index with one field that is defined in the application group and no field or indexing parameter defined for the application. If these conditions are true, you must use a field. You can define a dummy literal index and field value in the application indexing parameter as a placeholder. This dummy value is not processed, but it allows the file name to be indexed successfully.
•Problem: Content Manager OnDemand does not break up the PDF file into separate reports when TRIGGERs are defined correctly and indexing is successful. For certain reports, the trigger is not honored and the reports are grouped.
Reason or resolution: The field value must change for Content Manager OnDemand to indicate a report break. In Figure 18-2, there are several pages of a document. Page 1 is the TRIGGER, and the name is the field that is placed into the index.
Figure 18-2 Sample document for indexing
In this example, because the string Page 2 does not match the TRIGGER, it is ignored, and that page is included in report 1. Moreover, the report does not break until the name John Smith is read because it is different from the name John Doe.
•Problem: When the user views a document that is loaded with large object (LOB), the client receives the message:
Viewer Page count does not match Load Page Count. Viewing may be adversely affected. Contact your system administrator.
If the user clicks OK, the document can be viewed in its entirety, except that the page number is incorrect.
Reason or resolution: When a document is loaded as LOB, the Loader must count the pages, because a certain number of pages go into a LOB segment. When the client retrieves a LOB segment, the client also counts the pages that it receives from the server. The user will receive the message “Viewer Page count does not match Load Page Count” when the two page counts do not match.
This problem is usually caused by the user running ACIF to load a document that contains the form feed character x'0C'. ACIF does not support the form feed character as the start of a new page, but the line data viewer does support the form feed character as the start of a new page. Therefore, the viewer ends up with a different count of pages than the loader.
If you use the asciinp exit, ensure that the exit was not modified and recompile the exit.
If you use any exit other than the asciinp exit, do not use the exit so that troubleshooting will be easier.
If the file was transferred as text from a Windows system to an AIX system, the x'0D0A' will be changed to x'0A', which can affect the indexing.
Check your input file in a hex editor to verify whether the delimiter is x'0D0A' and that the file contains x'0C'.
18.1.3 Content Manager OnDemand maintenance issues
The following problems relate to Content Manager OnDemand maintenance:
•Problem: One of the Content Manager OnDemand database file systems is reaching 100% usage, and the file system size cannot be increased. How do you determine whether an application group is using this file system?
Reason or resolution: Follow these steps:
a. Run the arstblsp command to list the open table for the application group. For example, the application group that you want to find is called AppGrpName. Use the following command:
arstblsp -a 3 -g AppGrpName
The command returns table name CAA1:
Table still open for loading: ApplGroup(AppGrpName) Agid(5016) Table (CAA1)
b. List the table space ID, table space, and table name for the application group data table that is opened, for example:
su - archive
db2 connect to archive
db2 “select tbspaceid, tbspace, tabname from syscat.tables where tabname='CAA1'”
The command returns the following output for table space ID 3:
TBSPACEID TBSPACE TABNAME
3 ROOT_CAA1 CAA1
c. Determine the containers for this table space ID by running the following command:
db2 "list tablespace containers for 3"
The command returns with the table space containers for table space 3, as shown in Figure 18-3 on page 382.
Tablespace Containers for Tablespace 3
 
Container ID = 0
Name = /arsdb/db1/SMS/ARCHIVE/root/CAA1.0.0
Type = Path
 
Container ID = 1
Name = /arsdb/db1/SMS/ARCHIVE/root/CAA1.1.0
Type = Path
 
Container ID = 2
Name = /arsdb/db1/SMS/ARCHIVE/root/CAA1.2.0
Type = Path
 
Container ID = 3
Name = /arsdb/db1/SMS/ARCHIVE/root/CAA1.3.0
Type = Path
Figure 18-3 Output of db2 list tablespace 3 command
d. Check whether any of the containers that were listed previously belong to the file system that is full:
 • If any of the containers that were listed previously belong to the full file system, close the opened application group data table by using the following command:
arstblsp -a 1 -g AppGrpName
The following message indicates that the table closed successfully:
Closed table successfully: ApplGroup(AppGrpName) Agid(5016) Table(CAA1)
 • If none of the containers that were listed previously belong to the full file system, continue to find the next application group.
When the application group data table is closed, Content Manager OnDemand creates a table on a file system as defined in ARS.DBFS when data is next loaded. Content Manager OnDemand also searches for a file system with more free space to create the new table.
•Problem: The arsmaint program fails to complete.
Reason or resolution: The problem that is most commonly encountered is that the cache file system is full or a link is broken.
For a full cache file system, check to determine which file system is full, and expand the file system, if possible.
For a broken link problem, the system log displays errors that relate to arsmaint.
If neither situation is the case, check to see whether arsload is running at the same time. If arsload is running at the same time that you run the arsmaint -r command, arsmaint might fail.
•Problem: When you create new application groups, you see the error that is shown in Figure 18-4.
DB2 z/OS SQLCODE-497
THE MAXIMUM LIMIT OF INTERNAL IDENTIFIERS HAS BEEN EXCEEDED FOR DATABASE <database-name>
Figure 18-4 Error message SQLCODE 497
Reason or resolution: The following actions can help decrease the internal limits:
 – If the DBID limit is exceeded, DROP all unused databases and issue a COMMIT.
 – If the OBID limit is exceeded, DROP all unused objects in the database and issue a COMMIT. Specify a different database, or run the MODIFY utility to reclaim unused OBIDs.
 – Consider dropping indexes on application group data tables for indexes that are not frequently used for access. Review NODX reports to identify possible indexes that can be dropped.
 – Find any obsolete application groups and related data whose definitions can be deleted by using the Administrator Client.
 – Analyze application group data tables that became multiple segment tables, and change the MAX_ROWS column of the current table in the ARSSEG table to a larger value so that another table will not be created.
 – Check your application group expiration settings. Ensure that your expiration processes are performed on a timely basis. Ensure that your expiration processes complete successfully so that application group data segment tables, table spaces, and indexes are dropped in DB2 at the same time that the expiration processing occurs.
 – Consolidate applications into fewer application groups, if possible.
 – Create another database and modify the ARSUTBL exit to change the default created table space to a new database. For more information, see Chapter 11, “Exits” on page 241.
 – Use Content Manager OnDemand Administrator Client to define application groups to go to different databases.
18.1.4 Monitoring the main server task arssockd
The following problem is common:
•Problem: At the start, you need more information about the main server task that is named arssockd.
Reason or resolution: You can now monitor the main server started task arssockd by displaying the process usage information for the instance. By using the parameter /-I ARC900 -x -p for z/OS, add the following PARM statement to the arssockd started task:
PARM='/-I ARC900 -x -p'
-p displays the process information.
For Content Manager OnDemand, you can monitor the library server from any machine with Content Manager OnDemand installed on it by running the following command:
arssockd -I <INSTANCE> -p -x
The object servers also can be monitored from the object server by running the following command:
arsobjd -I <obj_hostname,port> -p -x
If you want to see all of the parameters that are available for arssockd, run arssockd with -p without any other parameters. You receive the output that is shown in Figure 18-5.
/usr/lpp/ars/V9R0M0/bin>arssockd -p
ARS0980I Usage: arssockd [options]
Version: 9.0.0.1
-h <od_inst> OnDemand instance name or host name (same as -I)
-I <od_inst> OnDemand instance name or host name (same as -h)
-p Display process usage information for the given instance
-P Ping the OnDemand Instance
-q Display configuration and version information for the given instance
-r <iterations> Number of iterations (defaults to 1)
-s <seconds> Number of seconds between iterations (defaults to 1)
-S Start the OnDemand server for the given instance
-T Stop the OnDemand server for the given instance
-v Verbose output
-x Extended information (when used with -p)
-1 <trace_file> Trace file
-2 <trace_level> Trace level
Figure 18-5 Options for running arssockd with the -p parameter
18.1.5 Installation and migration issues
The following problems might be encountered when you install or migrate Content Manager OnDemand systems:
•Problem: Various errors occur during the installation of Content Manager OnDemand for Multiplatforms V9.5.
Reason or resolution: First, look at the installation directory. The new installer does not change the directory location to where the installation occurs as the installer did in the previous release. This situation might cause installation errors.
 
Important: Before you install the Content Manager OnDemand system, note the installation directory location because changing the directory location affects upgrade instructions. In version 9.0 and earlier releases, the installer removes the previous version of Content Manager OnDemand.
For special installation and configuration instructions, see the installation readme file.
Table 18-1 on page 385 shows the new default installation directory locations.
Table 18-1 Default installation directory
Operating system
Content Manager OnDemand server installation directory
ODWEK installation directory
AIX, HP-UX, and Solaris
/opt/IBM/ondemand/V9.5
/opt/IBM/odwek/V9.5
Linux (both x64 and z Systems)
/opt/ibm/ondemand/V9.5
/opt/ibm/odwek/V9.5
Microsoft Windows
C:Program FilesIBMOnDemand for WindowsV9.5
C:Program FilesIBMOnDemand Web Enablement KitV9.5
•Problem: When you start Content Manager OnDemand, you see the error message that is shown in Figure 18-6.
ARS0013E ARSSOCKD DB Error: STARTUP -- SQLSTATE=1 -
ARS_ORIGINAL_CODEPAGE is not defined in ars.cfg. Run arsdb -u to
determine setting of ARS_ORIGINAL_CODEPAGE, SQLCODE=0, File=arssys.c,
Line=370
Figure 18-6 Error message ARSSOCKD DB error
Reason or resolution: For version 8.5.0, a new ars.cfg parameter, which is ARS_ORIGINAL_CODEPAGE, is now required.
 
Important: When you use Content Manager OnDemand for z/OS V8.5.0 to access a pre-version 8.5 Content Manager OnDemand database, this parameter must be set to the code page that the pre-version 8.5 server was running in. Failure to set this parameter prevents the Content Manager OnDemand server from starting. Setting this parameter incorrectly results in data corruption. This information is in the Content Manager OnDemand z/OS 8.5 readme file.
For information about the correct setting for ARS_ORIGINAL_CODEPAGE, run the arsdb -u command without ARS_ORIGINAL_CODEPAGE in the ars.cfg file, for example:
arsdb: Unable to initialize environment. The return code is -1 -
Define ARS_ORIGINAL_CODEPAGE in the ars.cfg file:
 – If this instance is new, use the following setting:
ARS_ORIGINAL_CODEPAGE=37
 – If this instance is an existing instance, use the following setting:
ARS_ORIGINAL_CODEPAGE=1047
 
Important: After you define ARS_ORIGINAL_CODEPAGE, you must never change it.
For an existing instance (created before version 8.5), the code page that is displayed must match the code page in the ARS0220I message that is issued by the pre-version 8.5 server when it starts, regardless of the setting of ARS_ORIGINAL_CODEPAGE. See this example message:
ARS0220I Server code page is 1047
Note: An 8.5 server will always display
ARS0220I Server code page is 1200
•Problem: You encounter an error while arsload is running, as shown in Figure 18-7.
ARSLOAD Command: An error occurred. Contact your system administrator and/or consult the System Log. File=arsadmp.c, Line=1608 Failed while attempting to load the database. The last row successfully loaded was 117461. Loaded 117461 rows into the database
Figure 18-7 ARSLOAD error message
Reason or resolution: This issue might relate to an incorrect ARS_ORIGINAL_CODEPAGE setting. Check that the value is correct by using the following method. For more information, see Technote 1616768.
 – Use this method for UNIX servers:
For information about the correct setting for ARS_ORIGINAL_CODEPAGE, run arsdb -u without ARS_ORIGINAL_CODEPAGE in the ars.cfg file, for example:
arsdb -u -I <OD_INSTANCE>
You might receive this message:
arsdb: Unable to initialize environment. The return code is -1.
Define ARS_ORIGINAL_CODEPAGE in the ars.cfg file:
 • If this instance is a new (created in version 8.5) instance, use the following setting:
ARS_ORIGINAL_CODEPAGE=819
 • If this instance is an existing instance (created before version 8.5), use the following setting:
ARS_ORIGINAL_CODEPAGE=923
After it is set, ARS_ORIGINAL_CODEPAGE must never change.
Edit the ars.cfg file and add the ARS_ORIGINAL_CODEPAGE parameter, which is set to the appropriate value by the arsdb command.
 – Use this method for Windows servers:
For information about the correct setting for ARS_ORIGINAL_CODEPAGE, run arsdb.exe -u -I <OD_INSTANCE>, for example:
arsdb.exe -I <OD_INSTANCE> -u
You might receive the following output:
arsdb: Unable to initialize environment. The return code is -1.
Define ARS_ORIGINAL_CODEPAGE in the ars.cfg file:
 • If this instance is a new (created in version 8.5) instance, set ARS_ORIGINAL_CODEPAGE with a value of 1208 in the registry.
 • If this instance is an existing instance (created before version 8.5), set ARS_ORIGINAL_CODEPAGE with a value of 5348 in the registry.
The registry setting must be in the HKEY_LOCAL_MACHINESOFTWAREIBMOnDemand for Windows@SRV@_<OD_INSTANCE>)CFG registry key.
After it is set, ARS_ORIGINAL_CODEPAGE must never change.
Run regedit.exe and update the Windows registry key that is specified in the output of the arsdb command and add the ARS_ORIGINAL_CODEPAGE string value, which is set to the appropriate value by the arsdb command.
•Problem: When you run the arsdb -I ARCHIVE -vu command from a BPXBATCH job, you see errors in STDOUT or in the output of arsdb -c, as shown in Figure 18-8.
DB Error: {DB2 for OS/390}{ODBC DRIVER} SQLSTATE=58004 ERRLOC=2:170:9
CAF “CONNECT” failed using DB2 system:DB2K
RC=08 and REASON=00f30002 -- SQLSTATE=58004, SQLCODE=-99999
arsdb:.Unable to connect to DB2 ARSDBASE database
or
arsdb: Unable to determine the database engine
Figure 18-8 ARSDB error
Reason or resolution: Enter an export command for DSNAOINI in the BPXBATCH job, or ensure that export DSNAOINI='/etc/ars/cli.ini' was defined. Verify that your cli.ini file is in this directory. Also, check that your cli.ini file references the correct DB2 subsystem, for example:
[COMMON]
MVSDEFAULTSSID=DB1M
[DB1M]
PLANNAME=DSNACLI
•Problem: During installation and migration, when you run arsdb -u -I or arsdb -I ARCHIVE -vx ARSSYS, you receive an error similar to Figure 18-9.
arsdb -u -I
CEE3204S The system detected a protection exception (System Completion Code=0C4).
From entry point u_file_write_44_arsxh at compile unit offset +00000054 at entry offset +00000054 at address
Figure 18-9 ARSDB error during installation and migration
Reason or resolution: Multiple resolutions can be attempted when you encounter this error:
 – Check the ARSSOCKD region parameter. We recommend that you use region=0. Also, check your TSO logon region size. Increase TSO, log out of TSO, and log back in.
 – When you run export (x), the contents of the ARSxxx table are exported to a flat file. The command attempts to write the file to the directory in which the arsdb command runs. Ensure that the user has write permissions for the file. Also, ensure that the user has permissions for the ARS_TMP intermediate file.
 – Check the OMVS size to see whether it can be increased to the system limit
(D OMVS,L). Check your OMVS limits: MAXMMAPAREA and MAXSHAREPAGES. The following example is output of the D OMVS,L command:
                CURRENT HIGHWATER SYSTEM
                 USAGE      USAGE         LIMIT
MAXMMAPAREA   3107       3107          4096
The example shows current MAXMMAPAREA is 3107 and the maximum size that MAXMMAPAREA can go is 4096. In our experience, version 8.5 required at least 3646 for the value of MAXMMAPAREA. In this example, the minimum memory size needed for CMOD will be 3107 + 3646 = 6753. Therefore, you must increase the MAXMMAPAREA from 4096 to a higher value, for example, 40960, to solve the problem.
•Problem: When you run /usr/lpp/ars/V9R0M0/bin/arsdb -I ARCHIVE -iv arsag, you receive an error message similar to Figure 18-10.
unable to import table arsag. err=1904
Figure 18-10 ARSDB error message 1904
Reason or resolution: The arsdb command cannot open a temporary file for messages. ARS_TMP from the ARCHIVE instance is not being used. If ARS_TMP is not present, root '/' is used. To resolve this error, define the directories correctly so that they are pointed to by the ARS_TMP= parm in the ars.cfg file.
18.1.6 Common server messages
Several common messages can occur in a Content Manager OnDemand environment:
•ARS0066I message
ARS0066I Application Group Document Get: Name(appl_grp_name) Agid(agid) ApplName(appl_name) Aid(aid)NodeName(node_name) Nid(nid) Server(server) Time(time) Flds(fields)
This message is received during document retrieval from a specific application group. You can find this message in the Content Manager OnDemand system log. The message is for your information only.
This message is valuable because it records the document that was retrieved and other information about the document and the retrieval time, for example:
ApplGroup DocGet: Name(QPJOBLOG) Agid(5081) ApplName(QPJOBLOG)Aid(5082)NodeName(-CACHE-)Nid(1)Server(-LOCAL-) Time(0.322) Flds()
•ARS0067I message
ARS0067I Application Group Resource Get: Name(appl_grp_name) Agid(agid) NodeName(node_name) Nid(nid)Server(server) Time(time)
This message is received when a resource is retrieved from a specific application group. You can find this message in the Content Manager OnDemand system log. The message is for your information only.
This message is valuable because it records the name of the application group that the resource is associated with and the time of the resource retrieval, for example:
ApplGroup ResGet: Name(INS) Agid(6843) NodeName(-CACHE-) Nid(25) Server(-LOCAL-) Time(0.069)
•ARS0087I message
ARS0087I Application Group Load: Name(appl_grp_name) LoadId(load_id) File(file) InputSize(input_size)OutputSize(output_size)
This message is received when arsload is running. A report is loaded into the system. The message identifies the application group, the input file, and the load ID. This message is for your information only.
This message is valuable because it records information about the load, such as the application group name, load ID, file name, and sizes of the files at load time, for example:
ApplGroup Load: Name(MOSUNPO)LoadId(5535-2-0-1FAA-12349-12349) File(/QIBM/USERDATA/ONDEMAND/QUSROND/TMP/SP_MOSUNPO_WTH7HTWCXA_DBRYANT_064315_000009_RDR400M_1031023_210136) InputSize(225789) OutputSize(16380)
•ARS0088E message
ARS0088E Application Group Failed Load: Name(appl_grp_name) LoadId(load_id) File(file)
This message is received when the load process fails. You can find this message in the Content Manager OnDemand system log.
This message is valuable because it records the name of the application group, load-id, and file name of the failed load, for example:
ApplGroup Failed Load: Name(LATECHARGE) LoadId() File(/QIBM/USERDATA/ONDEMAND/QUSROND/TMP/SP_QPRLR133_QPRTJOB_DBRYANT_001467_000022_RDR400M_1021226_132052)
Response: To correct the problem, see the other messages that were generated by the ARSLOAD program and see the messages in the Content Manager OnDemand system log. Then, resubmit the command.
18.2 Information collection
If the guidance in 18.1, “Troubleshooting common problems” on page 378 does not help you determine and resolve your problem, speak to IBM Support. In this section, we explain the information to gather for IBM Support so that they can help you more efficiently.
When you report a problem to IBM Support, you must provide the version of the software that you are using. For Content Manager OnDemand, this version might include the numbers of the operating system, DB2, Oracle, IBM Tivoli Storage Manager, Content Manager OnDemand, and ODWEK. This information helps IBM Support determine whether the software version is still supported and whether known issues exist with that software version.
We also advise that you apply the latest maintenance level to Content Manager OnDemand before you contact IBM Support to ensure that you are not experiencing a problem that is resolved.
Table 18-2 shows commands that are used to determine the version of Content Manager OnDemand on various operating systems.
Table 18-2 Determine the version of Content Manager OnDemand
Operating system
Example of the command to determine the version
IBM AIX
lslpp -l | grep ars (OnDemand server and ODWEK)
Sun Solaris
pkginfo -l ondemand
HP-UX
swlist -l product | grep (OnDemand server and ODWEK)
Linux
Look for the highest version for the package name in the list:
rpm –qa | grep ondemand (Take highest version that is listed.)
Windows
From the Content Manager OnDemand configurator, click Help → About.
Windows client
From the Windows client, click Help → About OnDemand.
ODWEK
Check the logon message.
Content Manager OnDemand commands
Starting in Content Manager OnDemand 8.5, the response to all Content Manager OnDemand commands includes the release and fix pack level.
After you obtain the correct version number of the software that you are using, you must collect information that is specific to the problem.
Problems can occur in several main areas. We divide them into the following areas in this section:
•Indexing or loading
•Database
•Tivoli Storage Manager
•Content Manager OnDemand Client logon
•Performance
•ODWEK
•Content Manager OnDemand server hangs or crashes
In 18.2.8, “Exporting information to a local server” on page 397, we demonstrate how to export Content Manager OnDemand information, such as an application group, application, and folder, to a local server.
18.2.1 Indexing or loading
This section describes the logs to collect that relate to indexing or a loading problem.
Common loading issues
Table 18-3 shows the information to collect if a problem occurs with loading.
Table 18-3 Information to collect for loading issues
File
Description
ARSSOCKD.ERR
This log file is for the arssockd daemon process. The process is instance-dependent if multiple instances are running.
ARSLOAD error message
The ARSLOAD error message shows whether ARSLOAD failed at the indexing or loading phase. (See ARSLOAD common error messages in 18.1.6, “Common server messages” on page 388.)
ARS.INI
This file is the Content Manager OnDemand instance configuration file. Each instance has a section in the ARS.INI file.
Content Manager OnDemand system log
This Content Manager OnDemand system log is in the system log folder. Various message numbers about warnings or errors at the time of failure are included.
Export folder, application group, and application files and sample data
The export files are used to import to the test server for problem replication.
CORE
This file holds the core memory dump that is generated by the operating system.
Version or level of DB2, Oracle, or SQL Server and Content Manager OnDemand
This file name contains the version or level of software that the server is using. Sometimes, a problem might be resolved by upgrading to the latest program temporary fix (PTF) or maintenance level.
IBM Content Manager OnDemand - Messages and Codes, SC19-3356, describes the error message codes that are in the Content Manager OnDemand system log.
Common AFP indexing problems
Content Manager OnDemand cannot load AFP data without indexes; therefore, you must first ensure that your AFP data is already indexed. Therefore, AFP must have Tag Logical Elements (TLEs).
Table 18-4 shows the information to collect when you have problems with AFP.
Table 18-4 Information to collect for common AFP problems
File
Description
Export folder, application group, and application files and sample data
The export files are imported to the test server for problem replication.
ACIF indexer error message
This file contains the error messages that are generated by the ACIF indexer. In z/OS, this file can be the job log, which has the indexer information from the failed job.
AFP sample data file
This file is a non-confidential data file that can be viewed by the IBM Support team to verify the AFP syntax.
AFP interim files that are used by AFP viewer within Content Manager OnDemand Windows Client
These files are created in the user’s temporary directory. They are deleted automatically after the document is closed by AFP viewer. They are useful in determining whether the issue is a server or client issue. In the Windows client, click File → Show Temporary File Locations to see the names of the directories where the client stores the data and resource files.
AFP trace report
AFP viewer trace can be turned on by modifying the FTDPORT2.INI file in the Content Manager OnDemand installation directory. For Windows 7, the default path is shown:
C:Program Files (x86)IBMOnDemand ClientsV9.5in
AFP resource and font files
Sometimes, this file is useful for various AFP issues, such as overlay, company logo, or globalized fonts.
Before you log a problem with IBM Support, use the information in Table 18-4 to look for clues about your problem. You can check the error codes from the ACIF indexer in IBM Content Manager OnDemand - Messages and Codes, SC19-3356. You might find the solution immediately. If you have an AFP memory dump tool, you can also dump the AFP data file to check for an invalid AFP data stream, which is a common problem.
 
Note: Because the AFP data stream can be printed by an AFP printer, it does not necessarily have the correct AFP structure for loading into Content Manager OnDemand. The loading of AFP data requires a more specific AFP structure than printing AFP data. IBM Content Manager OnDemand for Multiplatforms - Indexing, SC19-3354, provides information about the correct AFP data stream structure.
18.2.2 Database
For DB2 problems, collect the information in Table 18-5 on page 392 for problem determination.
Table 18-5 Information to collect for DB2
File
Description
db2diag.log
This file is the DB2 diagnosis file. It is in the $HOME/sqllib/db2dump directory, where $HOME is the home directory of the DB2 instance.
CLI trace (Open Database Connectivity (ODBC))
This file contains the call-level interface (CLI) trace file for diagnosing SQL statements. The CLI trace option must be turned on to collect the file. (See “Setting the CLI trace for DB2” on page 392.)
Text summary
Collect a text summary of the application group and folder with the database problem by logging on to the Content Manager OnDemand server through the Content Manager OnDemand Administrator Client.
Content Manager
OnDemand system log
Gather the Content Manager OnDemand system log messages
10 minutes before and after the database problem.
SQLCODE error message
If the SQLCODE error message is available, collect this information to determine whether the problem is from Content Manager OnDemand or the database.
DB2 configuration report of the Content Manager OnDemand instance
This report is generated by the db2 command:
db2 get db cfg for instance_name
ars.ini
If you cannot determine the database engine, check the path of the ars.ini file. In UNIX System Services, run set > /tmp/set.txt and send a binary copy of the ars.ini file to IBM Support.
ars.cfg
Send a binary copy of the ars.cfg file to IBM Support.
cli.ini
Verify the contents of the cli.ini file. Verify that MVSDEFAULTSSID=xxxx points to the correct DB2 Subsystem ID. Also, check whether the value can be set within your installation by a DSANOINI DD statement in JCL or as a hierarchical file system (HFS) file.
On Windows and other platforms
For Windows, collect the Content Manager OnDemand server configuration settings:
HKEY_LOCAL_MACHINESOFTWAREIBMOnDemand
 
For all other platforms, collect the ars.ini, ars.cfg, ars.cache, and ars.dbfs files.
Application group report
The application group ID is the name of the related DB2 tables.
Setting the CLI trace for DB2
We list two methods to turn on the CLI trace for DB2. One method is to edit the db2cli.ini file directly. The other method is to use the DB2 command line.
The examples show the common options for the DB2 CLI trace. The IBM Support team might suggest a different option to collect information that is appropriate to your situation. Modify these options as advised.
In both cases, the trace file that is collected (as shown in Example 18-1 on page 393 and Example 18-2 on page 393) is in the /tmp/db2trace.dmp file.
Method 1: Setting up the trace by editing the db2cli.ini file
You can set up trace by editing the db2cli.ini file by completing the following steps:
1. Add a section that is similar to the section that is shown in Example 18-1 or Example 18-2 to the db2cli.ini file, depending on your platform.
For Windows, this file is in the sqllib path, for example, C:Program FilesIBMSQLLIB. For UNIX, this file is placed in the /sqllib/cfg path of the home directory of the instance owner, such as /home/archive/sqllib/cfg. For z/OS, this file is in the UNIX System Services /tmp file.
Example 18-1 Common section of the db2cli.ini file
[COMMON]
TRACE=1
TRACEREFRESHINTERVAL=5
TRACEFILENAME=/tmp/db2trace.dmp
TRACEFLUSH=1
TRACECOMM=1
Example 18-2 Common section of the cli.ini file for DB2 z/OS
[COMMON]
MVSDEFAULTSSID=DB1X
APPLTRACEFILENAME=/tmp/db2trace
APPLTRACE=1
TRACETIMESTAMP=3
[DB1X]
PLANNAME=DSNACLI
For z/OS, controls also exist for the diagnostic trace:
DIAGTRACE=1
DIAGTRACE_BUFFER_SIZE=6291456
DIAGTRACE_NO_WRAP=1
The full path of the TRACEFILENAME must be a valid directory with permission for everyone to write.
For z/OS, ensure that you refer to the following file:
//DSNAOINI DD PATH=´/usr/lpp/ars/V9R0M0/config/cli.ini
2. Restart the application (in this case arssockd) for the changes to take effect.
3. Re-create the DB2 problem and capture the trace information.
4. To turn off the trace, modify the db2cli.ini file again and set TRACE=0.
5. Restart arssockd.
Method 2: Setting up the trace by using the DB2 command line
Alternatively, you can use the DB2 command line to activate the trace by completing the following steps:
1. In the DB2 instance, run the DB2 commands that are shown in Example 18-3.
Example 18-3 Turning on the trace through the DB2 command line
db2 UPDATE CLI CFG FOR SECTION COMMON USING Trace 1
db2 UPDATE CLI CFG FOR SECTION COMMON USING TraceRefreshInterval 5
db2 UPDATE CLI CFG FOR SECTION COMMON USING TraceFileName /tmp/db2trace.dmp
db2 UPDATE CLI CFG FOR SECTION COMMON USING TraceComm 1
db2 UPDATE CLI CFG FOR SECTION COMMON USING TraceFlush 1
2. Restart the application (in this case arssockd) for the changes to take effect.
3. Re-create the DB2 problem and capture the trace information.
4. Run the following command to turn off the traces:
db2 UPDATE CLI CFG FOR SECTION COMMON USING Trace 0
5. Restart arssockd.
18.2.3 Tivoli Storage Manager
For Content Manager OnDemand problems that relate to Tivoli Storage Manager, collect the information that is shown in Table 18-6. For specific Tivoli Storage Manager errors, see Collecting Data: Read First for Tivoli Storage Manager Products, reference number 1263547.
Table 18-6 Information to collect for Tivoli Storage Manager
File
Description
Application group report
The summary information for storage management shows the storage set name, which relates to Tivoli Storage Manager.
Storage set report
This information provides the node name at Tivoli Storage Manager.
Tivoli Storage Manager activity log
This log shows the events in the Tivoli Storage Manager server. You can retrieve the log by running the Query actlog command.
Tivoli Storage Manager error message
Tivoli Storage Manager error messages are prefixed with ANS, ANR, and so on. This error is generated by Tivoli Storage Manager and can be used for Tivoli Storage Manager support for further diagnosis.
You can gather the various object reports, such as the application group report and storage set report, by right-clicking the object and selecting Summarize.
18.2.4 Content Manager OnDemand Client logon
If a Content Manager OnDemand Client fails to log on to the server, check that arssockd is running on the server. Then, check the network connectivity by performing a ping test from the command window of the client. Open the command window and ping the host name or the IP address of the Content Manager OnDemand server.
Collect the files that are listed in Table 18-7 on page 395 for client problems, such as logging in to Content Manager OnDemand.
Table 18-7 Information to collect for client logon problems
File
Description
ARS.INI
This file is the Content Manager OnDemand instance configuration file. The instance is configured in each section in the ARS.INI file.
ARS.CFG
This file is the Content Manager OnDemand configuration file.
ARSSOCKD.ERR
This file is the log file for the arssockd daemon process. The process is instance-dependent if multiple instances are running. This file is in the
path that is defined for ARS_TMP.
Content Manager OnDemand system log
Check the Content Manager OnDemand system log for any specific messages that relate to logging in to the client.
Print screen
Print a screen capture of any client errors you might receive when you log on to the client for further analysis.
18.2.5 Performance
For Content Manager OnDemand performance issues, gather the information that is shown in Table 18-7.
Table 18-8 Information to collect for performance issues
File
Description
Application group report
Check these fields in the report, whether they are indexed or filters. Simply reviewing this report might resolve the issue.
Text summary
Collect a text summary of the application group and folder with the performance problem by logging on to the Content Manager OnDemand server through the Content Manager OnDemand Administrator Client.
CLI trace (ODBC)
This file contains the call-level interface (CLI) trace file for diagnosing SQL statements. The CLI trace option must be turned on to collect the file.
Content Manager OnDemand system log
Gather the Content Manager OnDemand system log messages
10 minutes before and after the search problem.
On Windows and other platforms
For Windows, collect the Content Manager OnDemand server configuration settings from the following registry key:
HKEY_LOCAL_MACHINESOFTWAREIBMOnDemand
 
For all other platforms, collect the following files:
ars.ini, ars.cfg, ars.cache, and ars.dbfs
Database reorganization information
This file is used to check whether the arsdb command ran to reorganize -m for DB2 and SQL Server, run maintenance on the Content Manager OnDemand database, and reorganize the Content Manager OnDemand system tables. This option refreshes the tables and optimizes access to information in the database. You might also want to run the reorg command on data tables. After you reorganize the tables, run the runstats command on the tables arsdb -s, and run database statistics.
Memory information
This file contains the amount of physical memory and the memory setting in the server, such as the output from the ulimit command.
ARSSOCKD.ERR
This log file is for the arssockd daemon process. The process is instance-dependent if multiple instances are running. This file is in the path that is defined for ARS_TMP.
Indexer information from the application report
This file helps you determine whether the report has a single index, which uses up memory if the report is huge. Also, if a large report is not using the large object option, a user might experience a long download time.
IBM Tivoli OMEGAMON® XE for DB2 Performance Monitor on z/OS
If necessary, you are instructed by IBM Support to run the Performance Monitor to further analyze a performance problem. The initial setup includes accounting and statistics reports.
18.2.6 ODWEK
For ODWEK problems, gather the information that is shown in Table 18-9. Depending on the environment and the specific failure, part of the information might not be present in your environment. See MustGather: ODWEK Java API terminating without warning:
Table 18-9 Information to collect for ODWEK
File
Description
ODWEK trace
This file is the arswww.trace file.
IBM WebSphere Application Server
WebSphere MustGather crash files.
HTTPd log
This file is the IBM HTTP Server log file.
Test case
This file is a skeleton test case that can re-create the crash.
Core
This core file is generated by the operating system.
Screen captures of the problem
Provide a file that contains screen captures of the error message or document. This file is also useful for non-English error messages and documents.
Plug-ins or applets information
Provide a file of any plug-ins or applets that are used. This information helps to verify the version in use. Sometimes, a problem is resolved by using the latest version.
Version or level of ODWEK, DB2, Oracle, or SQL Server, and Content Manager OnDemand
Provide the version or level of software that the server is using. Certain issues might be resolved by simply upgrading to the latest PTF.
18.2.7 Content Manager OnDemand server hangs or crashes
For problems where Content Manager OnDemand server hangs or crashes, you can search for a few MustGather technotes by going to the following website:
Search this website by using the MustGather keyword to find the following technotes:
•MustGather: Content Manager OnDemand Server for Windows - Hang or performance degradation, reference # 1223907
•MustGather: Content Manager OnDemand Server for Windows - Crash, reference # 1226443
•MustGather: IBM Content Manager OnDemand server hang or performance degradation on AIX, reference # 1222374
•MustGather: IBM Content Manager OnDemand server crash on AIX, reference # 1223109
Follow the instructions from the technotes to gather information when the server hangs or crashes.
18.2.8 Exporting information to a local server
IBM Support might require information about the Content Manager OnDemand application group, application, and folder for problem determination.
To create a local server to export object information, complete the following steps:
1. Create a local server on your workstation:
a. From your Content Manager OnDemand Administrator Client, select OnDemand Servers and then click File → New Server. See Figure 18-11.
Figure 18-11 Setting up a local server
b. From the Add a Server window that opens, for the Protocol field, select Local, and enter the information that is shown in Figure 18-12. Click OK. A local server with the name ODlocal is created.
Figure 18-12 Add a Server window
2. The local server cannot be used until it is set up. Right-click the ODlocal server and select Setup, as shown in Figure 18-13 on page 398.
Figure 18-13 Setting up the local server
When you see the prompt “Are you sure?”, click OK.
When the setup is complete, the local server is ready to use. By default, the local server has a user that is named admin without any password.
3. Export the requested information from your server to the local server. Right-click the object and select Export. For example, if you want to export the application group with the name Redbk, right-click the object Redbk and select Export, as shown in Figure 18-14.
Figure 18-14 Export Application Groups
4. In the Export Application Groups window (Figure 18-15) that opens, export your application groups by completing the following steps:
a. From the Server list, select the server to be exported.
b. Click Export. The information of the application group that you chose starts transferring to ODlocal.
c. Check the message at the end of the export to ensure that the export is successful.
d. You can select either of the following options:
 • Select Ignore Warnings if you want Content Manager OnDemand to add an item regardless of any warnings. Otherwise, Content Manager OnDemand stops transferring the item when the first warning is encountered. For example, if the application group has users and groups permissions that are defined in the source server, but the users and groups are not present in the local server, the export fails. If the item to be exported exists on the destination server, the export also fails.
 • Select No Storage Set if you do not want Content Manager OnDemand to assign a storage set to the application group.
Figure 18-15 Export local server
5. When all of the requested information is exported to the local server, compress the entire directory from the directory of the local server. In this example, the directory of the local server is C:ODlocal, as shown in Figure 18-12 on page 397.
 
Tip: When you export all of the requested information, we recommend this order:
•Printers
•Users and groups
•Storage sets
•Application groups, applications, and folders
18.3 Content Manager OnDemand trace facility
Content Manager OnDemand incorporated a trace facility to help IBM Support perform problem determination. In this section, we show you how to enable trace. The trace affects Content Manager OnDemand server performance. Enable the trace only when this action is requested by IBM Support. The trace is enabled to gather documentation and it must be disabled afterward.
 
Note: The information from the Content Manager OnDemand trace facility is for informational purposes only and must be used only under the direction of IBM Support personnel. How tracing is enabled and what gets traced is subject to change at any time. Do not use trace as a programming interface.
18.3.1 Enabling the trace facility
This information is also covered in the Technote How to enable trace in Content Manager OnDemand server, 1330810.
To enable the server trace facility, complete the following steps:
1. Locate your trace.settings file:
 – On AIX, it is in the /opt/IBM/ondemand/V9.5/config directory.
 – On Sun, it is in the /opt/IBM/ondemand/V9.5/config directory.
 – On Windows, it is in the C:Program FilesIBMOnDemand ServerV9.5config directory.
 – On z/OS, it is in the /SYSTEM/etc/ars directory.
 – On other platforms, it is in the /opt/ondemand/v9.5/config directory.
2. Edit the trace.settings file to trace server startup routines by setting the following trace parameter:
TRACE_LEVELS=ALL=15
3. Edit your ars.cfg file by adding the ARS_TRACE_SETTINGS parameter and referencing the full path to your trace.settings file:
ARS_TRACE_SETTINGS=/usr/lpp/ars/V9R0M0/config/trace.settings
For Windows, the ars.cfg configuration is in the registry. Edit the registry settings for the registry key HKEY_LOCAL_MACHINESOFTWAREIBMOnDemand for WinNT@SRV@_ARCHIVECFG, where ARCHIVE is the name of your Content Manager OnDemand instance. Create a string value ARS_TRACE_SETTINGS and set the value to the full path to your trace.settings file.
After you enable tracing, start the Content Manager OnDemand Administrator Client.
18.3.2 Setting trace parameters
After you enable tracing, you can set the appropriate option for a runtime trace by using the Content Manager OnDemand Administrator Client.
Log on to the Content Manager OnDemand Administrator Client and configure tracing by completing the following steps:
1. Right-click the server name and select Trace Parameters, as shown in Figure 18-16.
Figure 18-16 Configure trace parameters
2. In the System Trace Setting window (Figure 18-17), complete the following steps:
a. Select the Activate System Trace check box to turn on tracing for the whole system.
b. Enter information in the Trace Parameters entry field. The trace parameters can be name=value pairs that are separated by commas to define the trace level. These name=value pairs are provided by IBM Support. For an example, see Figure 18-17.
Figure 18-17 System trace settings
c. Click Update. You do not need to restart Content Manager OnDemand.
After the trace is collected, you can send the trace file to IBM Support.
 
Note: You can stop or start the runtime trace from the Content Manager OnDemand Administrator Client anytime without restarting arssockd.
Important: Use trace only with the help of IBM Support because activating trace might severely affect the performance of the Content Manager OnDemand system.
18.4 Other tracing options
In addition to Content Manager OnDemand tracing, you can run other traces for additional diagnosis, as needed. Other traces are enabled to focus on specific areas and gather the necessary documentation. You need to disable the other traces afterward.
18.4.1 ARSLOAD
The ARSLOAD program is the main Content Manager OnDemand data loading and indexing program where you can trace data loading and indexing issues.
In most cases, trace parameters can be specified on the command line when you run arsload.
With the ARSLOAD command, the trace is enabled by the -1 and -2 parameters:
•-1 <trace_file> (fully qualified trace file name)
•-2 <level> (trace level number)
•The trace level number values are additive (default is 3):
 – 1: Errors
 – 2: Warnings
 – 4: Info
 – 8: Flow
These trace levels provide entry and exit information for functions.
The trace level numbers are added up, and the default level is 3, which is used to report errors and warnings that occur during loading.
You can also use name=value pairs that are separated by commas to define the trace level. The name=value pairs are provided by IBM Support when they request the trace, for example:
-1 trace.out -2 ALL=15,ARSRD=3,ARSLOAD=3
The trace no longer generates textual output, so it performs better. The output is now in binary format and requires the use of the arstfmt command, which is in the bin directory of your Content Manager OnDemand server (version 8.5 and later). You can format the output in either text or XML format.
To produce a text formatted trace file, run the following command:
/usr/lpp/ars/V9R0M0/bin/arstfmt -i /tmp/arswww.trace -o /tmp/arswww.trace.txt
To produce an XML formatted trace file, run the following command:
/usr/lpp/ars/V9R0M0/bin/arstfmt -x -i /tmp/arswww.trace -o /tmp/arswww.trace.xml
 
Limitation: Use the -1 and -2 parameters only under the supervision of IBM Support because they might affect performance.
18.4.2 MidServer trace (z/OS only)
To collect data that processed by the MidServer, complete the following steps:
1. Locate the MidServer arsMSVR.cfg configuration file. The file is in the following directory:
/MountPoint/config/midserver
2. Turn on MidServer tracing by setting MIDSERVERTRACE=1.
To collect the input data that is returned to the application, set traceLevel to 2 before you issue the logon function request. This traceLevel indicates that a full trace by the C stub is requested.
3. Run the application to re-create the problem.
4. Send the following files to IBM Support for further analysis. All of these files that are sent must be from the same test.
File name Location
arswww.ini /MountPoint/config/midserver
arswww.trace As specified in the arswww.ini TraceDir= file
arsMSVR.err STDERR path in the MidServer start procedure
arsMSVR.out STDOUT path in the MidServer start procedure
MVS job output SDSF arsMVSR job
18.4.3 ODWEK trace
Sometimes, it is necessary to collect data for IBM Content Manager OnDemand Web Enablement Kit (ODWEK). Gather this information to assist with the troubleshooting process before you contact IBM Support.
Remember that IBM Support cannot debug custom application code. The purpose of this information is to collect diagnostic test results to help identify a possible problem in ODWEK and provide additional documentation to IBM Support. This information is in IBM Technote 1240220:
By using the ODConfig class, you can set the trace as shown in Example 18-4.
Example 18-4 Setting up trace in the ODConfig class
OConfig cfg = new ODConfig(
/*AfpViewer*/ ODConstant.PLUGIN,
/*LineViewer*/ ODConstant.APPLET,
/*MetaViewer default*/ null,
/*MaxHits*/ 500,
/*AppletDir*/ "/applets",
/*Language*/ "ENU",
/*TempDir*/ "c:\temp",
/*TraceDir*/ "c:\path\to\trace",
/*TraceLevel*/ 4);
Your ODWEK application must be recompiled and restarted for the changes to take effect.
After you enable tracing, re-create the issue and send all arswww.trace* files to IBM Support.
In ODWEK V8.5.0.0 and later, the trace file is written in binary. To convert the trace file from its binary format, use the arstfmt command that is in the bin directory of your Content Manager OnDemand server (version 8.5 and later). The following sample command shows how to convert the ODWEK trace file:
/usr/lpp/ars/v9.5/bin/arstfmt -i arswww.trace -o arswww.trace.txt
Tracing can be set to different levels with the trace parameter. When you troubleshoot an ODWEK issue, set the trace level to the highest level, unless you are instructed otherwise by IBM Support.
Setting the trace at lower levels, such as Trace=1, creates a minimal impact while it alerts you only about error conditions. Setting the lower trace levels are ideal for monitoring an ODWEK application that is in a steady state. Higher levels are used for troubleshooting an ODWEK issue.
18.4.4 TCP/IP packet trace
If a problem exists with TCP/IP, a TCP/IP packet trace might be helpful in troubleshooting.
Example 18-5 show a procedure for tracing to an external writer.
Example 18-5 Sample writer procedure
//CTWTR1 PROC
//IEFPROC EXEC PGM=ITTTRCWR
//TRCOUT01 DD DSNAME=SYS1.CTRACE1,VOL=SER=xxxxxx,
// UNIT=xxxxx,SPACE=(CYL,(100),,CONTIG),
// DISP=(NEW,CATLG)
//SYSPRINT DD SYSOUT=*
To obtain a TCP/IP packet trace, complete the following steps:
1. Start your CTRACE external writer procedure by running the following command:
TRACE CT,WTRSTART=CTWTR1
2. Start and connect a packet component trace to your writer procedure by running the following command:
TRACE CT,ON,COMP=SYSTCPDA,SUB=(TCPIP_Proc)
reply,WTR=CTWTR1,END
3. Start and connect a SYSTCPIP component trace to your writer procedure by running the following command:
TRACE CT,ON,COMP=SYSTCPIP,SUB=(TCPIP_Proc)
reply,WTR=CTWTR1,JOBNAME=(RM_Jobname)
reply,WTR=CTWTR1,OPTIONS=(socket,pfs,tcp,sockapi),END
4. Check whether the component trace is ready to gather data by running the following command:
D TRACE,WTR=CTWTR1
The display must show a status of ACTIVE.
5. Set packet trace filters by running the following command:
V TCPIP,TCPIP_Proc,PKT,clear # resets filters to none
V TCPIP,TCPIP_Proc,PKT,ON,ip=10.253.0.35 (client IP address)
6. Run your recreation scenario.
7. Stop the packet trace by running the following command:
V TCPIP,TCPIP_Proc,PKT,clear # resets filters to none
8. Disconnect SYSTCPIP CTRACE from the external writer by running the following command:
TRACE CT,ON,COMP=SYSTCPIP,SUB=(TCPIP_Proc)
reply,WTR=DISCONNECT,JOBNAME=(),OPTIONS=(),END
9. Disconnect SYSTCPDA CTRACE from the external writer by running the following command:
TRACE CT,ON,COMP=SYSTCPDA,SUB=(TCPIP_Proc)
reply,WTR=DISCONNECT,END
10. Stop the CTRACE external writer by running the following command:
TRACE CT,WTRSTOP=CTWTR1,FLUSH
Send the non-formatted packet trace dataset information to IBM Support.
18.4.5 Language Environment (z/OS only)
Many of the Content Manager OnDemand utilities use the Language Environment, and it offers its own customized traces.
To start the trace, rerun the job with the following DD statement. Set the CEE environmental variable in the JCL (or use env var for the UNIX System Services command line):
//STDENV DD *
_CEE_RUNOPTS='HEAPCHK(ON),HEAPPOOLS(OFF)'
The output writes to the job log.
This trace is good for problems with starting a service:
//CEEDUMP DD
//ARSSOCKD EXEC ...,PARM='TRACE(ON,8M,,LE=1)/'
18.4.6 ARSSUPPORT utility
You can use the ARSSUPPORT utility to gather log entry diagnostic information. The ARSSUPPORT utility is in the arssupport.jar file. To run the utility, run the following command:
java -jar arssupport.jar
The following prerequisites apply to using the command:
•Ensure that you have Java Runtime Environment version 1.5 or higher to run this program.
•Ensure that you are logged on to the operating system by using an ID that has administrator authority on Windows or root authority on UNIX.
•On Windows systems, run the ARSSUPPORT utility from the Content Manager OnDemand command prompt.
•To retrieve system log entries, ensure that the Content Manager OnDemand server is running.
•The data is collected from the computer where the ARSSUPPORT utility is run.
The ARSSUPPORT utility generates information about a Content Manager OnDemand server. This information includes information about its configuration and system environment. ARSSUPPORT archives all files into one compressed file, arssupport.zip and places this file in the odsupport subdirectory of the output directory.
When you get the compressed file, send it to IBM Support.
18.4.7 ARSJESD
The ARSJESD program is the server component of Download. Download can be used to transmit output datasets of application programs automatically from the JES spool to Content Manager OnDemand server file systems. If problems occur during the transmission of the ARSJESD program, complete the following steps:
1. Stop the arsjesd program.
2. Add the -t parameter, for example:
arsjesd -d /tmp/1 -d /tmp/2 -p 6001 -t
3. If you are using Windows, add the -t parameter to the following entry in your registry, and then uninstall and reinstall the arsjesd service by using the configurator:
HKEY_LOCAL_MACHINESOFTWAREIBMOnDemand for Windows@SRV@_ARCHIVEServicesarsjesd (ARCHIVE)ProgramParms
The example is from an instance named ARCHIVE. Go to the registry key for your respective instance.
4. Start the arsjesd service.
The trace file is named trace.log.<port #> and is written to the first arsjesd directory that is specified by the -d parameter. In the example from step 2, this directory is /tmp/1.
5. Re-create the issue.
18.4.8 PDF Indexer trace
If a problem occurs during the indexing and loading of PDF documents, you might want to run the PDF Indexer trace.
The PDF Indexer tracing can be performed by using either of two methods:
•Add the following lines to the indexing parameters:
TRACEDD=<trace file name>
TRACELEVEL=PDF=15
For example:
TRACEDD= emppdf_tracefile.bin
TRACELEVEL=PDF=15
•Run the PDF Indexer from the command line and add the trace parameters:
arspdoci parmdd=filen.parms inputdd=filen.pdf outputdd=filen.out indexdd=filen.ind tracedd=filen.trace tracelevel=pdf=15
Where:
 – arspdoci: Name of the command-line version of the PDF Indexer program
 – parmdd: Specifies the name of the input file that contains the indexing parameters
 – inputdd: Specifies the name of the PDF input file to process
 – outputdd: Specifies the name of the output file that contains the indexed PDF documents that are created by the PDF Indexer
 – indexdd: Specifies the name of the output file that contains the index information that is loaded into the database
 – tracedd: Specifies the name of the output file that contains the trace information
18.4.9 Trace resolver
The trace resolver output is useful and can be used by IBM Support, programmers, and networking system programmers to diagnose problems in resolving IP host names to
IP addresses or IP addresses to IP host names.
The trace resolver helps determine the values of the TCPIP.DATA statements and where the values were obtained.
The details of collecting this trace are in IBM Technote II13398:
18.4.10 Conclusion
The traces are enabled for troubleshooting and information gathering. When the task is complete, do not forget to disable the traces.
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.142.133.54