Problem handling
There are several aspects to handling zPDT problems, including these:
•Problems starting a zPDT operation
•Problems during a zPDT operation
•Problems with emulated device files
•Problems with the underlying Linux system
•Problems with the System z operating system and applications
This chapter uses the term zPDT service provider. This may be an IBM Business Partner or some other zPDT provider that offers service. Not all zPDT users may have such service providers, and some of the information in this chapter may not apply in these cases.
The underlying Linux system, and whatever System z operating system and applications are being used, are not part of the zPDT and are not part of any zPDT support activity. zPDT support includes only the direct zPDT components. As a practical matter, there may be some overlap between Linux issues and zPDT problems, and you may need assistance from your zPDT service provider to isolate the problem.
19.1 Problems starting zPDT operation
Many problems are related to devmap errors, and are often due to errors in Linux file names in the devmap. The solution is to check the devmap carefully, remembering that file names are case sensitive in Linux.
Problems obtaining a license (from the zPDT token) are typically due to an expired or unactivated token.
Messages about Time Cheat errors indicate a more serious problem. These are typically caused either by (1) moving a token between multiple PCs that do not have their time-of-day clocks closely synchronized, or (2) manipulation of the clock in the PC.
The zPDT system uses several Linux shared storage (virtual memory) areas. These are normally freed when zPDT is ended with an awsstop command. A failure or incorrect handling during zPDT startup or operation might result in these shared storage areas not being released. This can prevent zPDT from being started again. There are Linux commands to individually free shared storage areas, but this requires multiple detailed steps.1 Rebooting Linux is a primitive but effective way to solve this particular situation.
If you have problems with the token or license consider taking the following actions before seeking help, first resolving any errors detected by these steps:
•Try the uimcheck command.
•Verify that the local token can be accessed, using the following Linux commands:
$ lsusb | grep Rain (You see “Rainbow Technologies”)
$ ps -ef | grep usbdaemon | grep -v grep (You see safenet_sentinel)
$ ps -ef | grep Sntl | grep -v grep (You see SntlKeysSrvrlnx)
$ netstat -tlp (You see "sntlkeysrvr" in the fourth column)
•Try starting zPDT with a simple devmap, such as this one:
[system]
memory 1048m
processors 1
3270port 3270
•If a license problem persists, try deleting /usr/z1090/uim/uimclient.db. You need root authority to do this. Then restart zPDT.
•Try starting a local browser to "localhost:7002". This should produce a SafeNet browser page.
On this page, click on "Keys Information" in the menu list on the right-hand side of the page. This should produce a line showing the serial number of your SENTINEL KEY plus other data. On this line click on the "1" under "Keys#" This should display three lines showing the license numbers installed in your token. These license numbers should be 0xabcd, 0xd98, and 0x8198
•Try this command; you might need to save the output if further help is needed:
$ cat /usr/z1090/bin/sntlconfig.xml
If you seek help, be prepared to provide the following information:
•Has your zPDT system worked before the current problem? If so, what has changed?
•What are the software levels involved?
– What is the exact zPDT level? A number such as 51.14.01. Use the following Linux command to determine the zPDT level:
$ rpm -qa | grep 109 (Use “109” not “1090” or “1091”)
– What is the Linux level? For example, openSUSE 15.
– What is the operating system level (and is it an AD-CD system)?
•Are you running a virtual environment?
•How much memory is available on the PC (or your virtual machine)
•What type of token do you have (a 1090 or 1091)? Is it initialized? Are you certain? (Use Resource Link or your zPDT provider to initialize 1090 tokens. Use IBM Passport Advantage® or your zPDT provider for 1091 tokens, if initialization is needed.)
•Are you using a remote license server (as opposed to a token in your PC)?
•Are you trying to use a cloned system for the first time?
19.2 Problems during zPDT operation
The zPDT system maintains several logs and traces during operation. The zPDT programs might detect a problem and capture the logs or traces at the time of the problem. You can also capture logs and traces with a snapdump command2. This command may be used when there is no indication from zPDT that a problem exists, but you detect a problem and might want to work with your zPDT service provider3 to resolve it.
It is important to remember that this discussion is solely about zPDT operation. The snapdump data is not meaningful for addressing other problems, such as a problem with the System z operating system or System z applications.
A snapdump command typically creates a megabyte of data in /home/ibmsys1/z1090/logs, contained in various files. You may use snapdump as often you want, remembering that each one takes space in the logs directory. Your zPDT service provider might want several, or one, or none of these dumps.
Files in the logs directory created by snapdump are retained until you remove them. Most other log and trace files in this directory are automatically deleted by zPDT when appropriate. However, over time there may be a buildup of unwanted files in the logs directory. Assuming you are not working on an outstanding zPDT problem, you can simply delete all the files in the logs directory (doing this when zPDT is not running). An easy way to do this is to use the --clean option of the awsstart command. Again assuming you are not working with a zPDT problem, you might use the --clean option every time you perform an awsstart. Conversely, do not use the --clean option while you are working on a problem; some of the older log and trace files might be wanted at a later time.
The senderrdata command is used to package snapdump data into a .tar file, which is typically a little less than a megabyte. This file can be sent to IBM, through your zPDT service provider, or simply kept on your Linux system for potential use later. The senderrdata command can manage the FTP operation to IBM. These files (on the receiving system at IBM) are automatically deleted after a few days unless a formal problem (PMR or equivalent) event is opened; this can be done by your zPDT service provider.
Figure 19-1 provides an overview of problem data handling. If snapdump data is sent to IBM (as outlined in the figure), the senderrdata option to create a configuration file should also be used and the results sent to IBM.4
Figure 19-1 Problem data capture and reporting
If a problem incident is opened through your IBM Business Partner, you might be requested to send additional files. In general, after a problem incident is opened, do not delete anything from the logs directory.
If device managers fail they automatically create a trace file and are automatically restarted. This restart will not occur more than three times per minute. If a fourth failure occurs within the same minute the device manager is not restarted and the devices it controls become not operational for the remainder of that zPDT session. The zPDT design limits the size of the logs and traces and should never create more than about 30 MB per emulated device (and there is normally much less than this).
19.3 Core images
Severe problems might cause core image files to be created. If these are created by zPDT, they should go into the log subdirectory and be cleaned up with --clean option of awsstart. If you are actively working on a problem with your zPDT provider, these files may be useful. Otherwise they may be deleted because they can be rather large and might create a disk space problem.
Consistent dumps (“core images”) when zPDT is started can occur if you have a relatively large number of emulated I/O devices (more than 100, for example) and you have not considered memory management adjustments.
The snapdump command is used when zPDT is running. If you are unable to start zPDT (with the awsstart command) or zPDT ends immediately (before a snapdump can be taken), the problem may have created a core image file. In this case, the core image might help with problem analysis and should be preserved while the problem is under investigation.
19.4 Logs
There are many logs that can be references. Some are Linux logs (in /var/log) and some are zPDT logs (in /home/ibmsys1/z1090/logs, assuming userid ibmsys1 is used for zPDT operation). The two most useful logs for initially looking at a problem are these:
/var/log/message (for Linux messages)
/home/ibmsys1/z1090/logs/log_console_......txt (for zPDT messages)
The log_console names tend to be long because they include a PID and time/date, as in this example:
log_console_p6780_t2014-11-21_13-31-07.txt
19.5 Emulated volume problems
An emulated 3390 volume is a single Linux file that was created with the alcckd command. Three variations of this command are useful for problem handling:
$ alcckd /z/WORK03 -rs (scan emulated volume for format errors)
$ alcckd /z/WORK03 -rf (replace bad track with zeros)
$ alcckd /z/WORK03 -r (display volser and size)
The -rs function scans the emulated volume and verifies that it is in the correct 1090 emulation format.5 The -rf function replaces improperly formatted tracks with a properly formatted track containing zeros. The original contents of the track are lost, but the functionality of the volume is maintained.
Assuming that your emulated 3390 volumes are in the /z directory and there are no other file types in this directory, you could verify the format of all the volumes with the following Linux shell commands:
$ cd /z (location of emulated ckd, and nothing else)
$ for i in *; do alcckd $i -rs; done
The ckdPrint command can be used to examine the contents of an emulated CKD volume. This command prompts for the beginning cylinder and head numbers and the ending cylinder and head numbers. These numbers are in decimal. The following example lists the records on cylinder 0, head 0 of volume Z9SYS1:
$ ckdPrint /z/Z9SYS1
DeviceType 3390, Cylinders-3339, Trks/Cyl-15, TrkSize-56832
Input extent in decimal - CC-low HH-low CC-high HH-high
00 00 00 00
....
....
The tapeCheck command may be used to verify the format of an awstape file. That is, this command reads the awstape file and verifies that the awstape control blocks are logically correct.
19.6 Special problem-related commands
The senderrdata command provides a menu of options:
1. rassummary (uses the Linux less command)
2. rassummary -s (for an overview of snapdump incidents)
3. FTP/dump snapdump data
4. FTP/dump PE directed files (used only at IBM request)
5. Create configuration information file
6. Logs Directory maintenance
7. FTP/dump rassummary created files
8. FTP/dump all files in logs directory
9. snapdump
The dump options in this menu mean that a .tar file is created, containing the selected files, and the .tar file is saved in /tmp. The dump option (to create a .tar file) is especially useful if the zPDT machine is not connected to the Internet.
The senderrdata command also creates a configuration file when you choose option 5. This file contains the following information such as the version of zPDT, the version of linux, the tokens plugged, memory usage, network configuration, file system information, contents of the z1090 directories, active Linux processes, copy of the last devmap used, and when the root password is provided) a copy of the Linux /var/log/messages. This file is useful when debugging events where a linux issue may be impacting the zPDT application.
A number of 1090 commands are intended to be used only at the direction of IBM support personnel and they supply the specific commands and operands to be used. This category includes the following commands:
$ awslog (including --logsize and --logcount operands)
$ doOSAcmd (various subcommands)
$ dreg (shared resource registry)
$ dshrmem (shared memory)
$ printlog (only for some .gz logs)
$ printtrace (only for some .gz traces)
$ qpci (displays zEDC information)
The contents and formats of the various log and trace files are not documented and are intended only for diagnostic use. Our experience is that these files are not useful for solving general user-level problems.
19.7 Linux monitoring
Some monitoring of Linux statistics may be helpful. The vmstat command is useful and causes very little interference with other processes in Linux.
$ vmstat 3 4 (4 samples at intervals of 3 seconds)
procs -----------memory------------ ---swap--- -----io----- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
2 0 0 397600 101936 2398160 0 0 109 52 154 350 14 2 81 3
r = number of Linux processes waiting for run time
b = number of Linux processes sleeping
swpd = amount of swap space used (KB)
free = amount of idle memory (KB)
buff = amount of memory used as buffers (KB)
cache = amount of memory used as cache (KB)
si, so = Linux paging activity in KB/second
bi, bo = I/O activity in blocks/second
in = interrupts per second
cs = context switches per second
us = percent of CPU time in user mode
sy = percent of CPU time in kernel mode
id = percent of CPU time idle
wa = percent of CPU time waiting for I/O
Important data from vmstat includes the Linux swap rates (si, so); any number here can degrade zPDT performance. If zPDT suffers a page fault, then the whole System z CP operation must wait for the page fault to be resolved. The wa statistic (CPU percentage waiting for I/0) provides an indirect indication of I/O overload.
The top command provides data about individual Linux processes. We are most interested in the emily process (which is the name of the Linux process representing the CP), but information about various device manager processes can be useful in spotting bottlenecks. Enter q from the keyboard to terminate the top command.
$ top
top - 11:14:14 up 1:24, 3 users, load avg: 1.26, 1.15, 1.02
tasks: 128 total, 1 running, 121 sleeping, 6 stopped, 0 zombies
CPU(s): 49.9% us, 0.2% sy, 0.0% ni, 49.7% id, 0.0% wa 0.0% hi, 0.0% si
mem: 3111660k total, 2719204k used, 392456k free, 105356k buffers
swap: 2104504k total, 0k used, 2104504k free, 2397740k cached
PID USER PR NI Virt RES SHR S %CPU %MEM TIME COMMAND
7040 ibmsys1 25 0 1549m 1.5g 1.5g S 100 49.6 35:41.2 emily
7051 ibmsys1 16 0 1510m 36m 36m S 0 1.2 0:03.1 awsckd
1 The Linux ipcrm command can be used to remove shared resources that have been orphaned.
2 The snapdump command is valid only while zPDT is operational.
3 IBM internal users would communicate through the z1090 forum instead of through a zPDT service provider. Other users, without a defined service provider, might use another zPDT forum.
4 Later 1090 versions may automatically combine the configuration data with the snapdump data.
5 Only the emulation format is checked. There is no check for data content or operating system metadata (label, VTOC, and such).
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.