Problem handling
This chapter uses the term ISV zPDT service provider. IBM is not a formal “service provider” for Independent Software Vendor (ISV) IBM Z Program Development Tool (IBM zPDT)
(ISV zPDT), but an equivalent service might be provided (as an option) by an ISV zPDT vendor. Not all ISV zPDT users might have such service providers, and some of the information in this chapter might not apply in these cases. Also, various web online forums or discussion groups might informally help with problems.
(ISV zPDT), but an equivalent service might be provided (as an option) by an ISV zPDT vendor. Not all ISV zPDT users might have such service providers, and some of the information in this chapter might not apply in these cases. Also, various web online forums or discussion groups might informally help with problems.
The underlying Linux systems and the IBM zSystems operating system and applications that are used, are not a formal part of ISV zPDT. Your “service provider” options might include assistance with expanded software issues.
Various chapters throughout this publication contain text that might help with different problem areas. This chapter summarizes higher-level steps for dealing with some common issues.
19.1 Problems starting an ISV zPDT operation
Many problems are related to device map (devmap) errors, which are often due to errors in Linux file names in the devmap. The solution is to check the devmap carefully, remembering that file names are case-sensitive in Linux. Use the awsckmap command for some basic devmap checking.
Problems with obtaining a license (from the ISV zPDT token) are typically due to an expired or unactivated token.
Messages about time cheat errors indicate a more serious problem. These errors are typically caused either by moving a token between multiple PCs that do not have their time-of-day (TOD) clocks closely synchronized, or manipulation of the TOD clock on the personal computer (PC).
The ISV zPDT system uses several Linux shared storage (virtual memory) areas. These areas are normally freed when ISV zPDT ends with an awsstop command. A failure or incorrect handling during ISV zPDT operation might result in these shared storage areas not being released, which can prevent ISV zPDT from being started again. There are Linux commands to individually free shared storage areas, but they require multiple detailed steps. Restarting Linux is a primitive but effective way to solve this situation.
If you have problems with the token or license, complete the following steps before seeking help:
1. Run the uimcheck command.
2. Verify that the local token can be accessed by using the following Linux commands:
$ lsusb | grep Rain (You see “Rainbow Technologies”.)
$ ps -ef | grep usbdaemon | grep -v grep (You see safenet_sentinel.)
$ ps -ef | grep Sntl | grep -v grep (You see SntlKeysSrvrlnx.)
# netstat -tlp (Misc, including remote license servers)
3. Start ISV zPDT with a simple devmap:
[system]
memory 4096m
processors 1
3270port 3270
4. If a license problem persists, delete /usr/z1090/uim/uimclient.db. You need root authority to do this task. Then, restart ISV zPDT.
5. Run following command. Save the output if further help is needed.
$ cat /usr/z1090/bin/sntlconfig.xml
If you seek help, provide the following information:
•Has your ISV zPDT system worked before the current problem? If so, what changed?
•What are the software levels that are involved?
– What is the exact ISV zPDT level? The level is a number like z1090-1-11.57.03.x86_64. To determine the ISV zPDT level, use the following Linux command:
$ rpm -qa | grep 109 (Use “109”, not “1090” or “1091”.)
– What is the exact Linux distribution name and level?
– What is the IBM zSystems operating system level (and is it an Application Development Controlled Distribution (ADCD) system)?
•Are you running a virtual or container environment?
•How much memory is available on the PC (or your virtual machine (VM) or container)?
•What type of token do you have (a 1090 or 1091)? Is it initialized? Are you certain? (Use your ISV zPDT provider to initialize 1090 tokens. If it is a 1091 (indicating an IBM Z Development and Test Environment (ZD&T) system), then contact your IBM ZD&T vendor.
•Are you using a remote license server (as opposed to a token on your PC)?
•Are you trying to use a cloned system for the first time?
19.2 Problems during an ISV zPDT operation
The ISV zPDT system maintains several logs and traces during operation. The ISV zPDT programs might detect a problem and capture the logs or traces at the time of the problem.
|
Attention: The zPDT logs and traces are meaningful only to the zPDT developers. They are not as useful for normal zPDT operation.
|
Over time, there can be a buildup of unwanted files in the logs directory. Assuming that you are not working on an outstanding ISV zPDT problem, you can delete all the files in the logs directory (do this task when ISV zPDT is not running). An easy way to do this task is to use the --clean option of the awsstart command. Again, assuming that you are not working with an existing ISV zPDT problem, you might use the --clean option every time that you perform an awsstart. Conversely, do not use the --clean option while you are working on a problem because some of the older log and trace files might be wanted later.
If device managers fail, they automatically create a trace file and are automatically restarted. This restart will not occur more than three times per minute. If a fourth failure occurs within the same minute, the device manager is not restarted, and the devices that it controls become not operational for the remainder of that ISV zPDT session. The ISV zPDT design limits the size of the logs and traces and should never create more than about 30 MB per emulated device (and there is normally much less than this amount).
19.3 Core images
Severe problems might cause Linux core image files to be created. If they are created by
ISV zPDT, they should go into the log subdirectory and be cleaned up with the --clean option of awsstart. If you are actively working on a problem with your ISV zPDT provider, these files might be useful. Otherwise, they can be deleted because they can be rather large and might create a disk space problem.
ISV zPDT, they should go into the log subdirectory and be cleaned up with the --clean option of awsstart. If you are actively working on a problem with your ISV zPDT provider, these files might be useful. Otherwise, they can be deleted because they can be rather large and might create a disk space problem.
Consistent dumps (core images) when ISV zPDT is started can occur if you have a many emulated I/O devices (more than 100, for example) and you have not made memory management adjustments.
If you cannot start ISV zPDT (with the awsstart command), there might be a core image file. In this case, the core image file might help with problem analysis, and it should be preserved while the problem is under investigation.
19.4 Logs
The term “log” can have multiple meanings in a Linux environment. Among these meanings are the classic Linux logs (in /var/log) and the specific ISV zPDT logs (in /home/ibmsys1/z1090/logs, if user ID ibmsys1 is used for ISV zPDT operation). The logs that might be helpful for initially looking at a problem are the following ones:
/var/log/message (for Linux messages)
/home/ibmsys1/z1090/logs/log_console_......txt (for ISV zPDT messages)
The log_console names tend to be long because they include a process ID (PID) and time and date, as in this example:
log_console_p6780_t2014-11-21_13-31-07.txt
19.5 Emulated volume problems
An emulated 3390 volume is a single Linux file that was created with the alcckd command. Three variations of this command are useful for problem handling:
$ alcckd /z/WORK03 -rs (Scans the emulated volume for format errors.)
$ alcckd /z/WORK03 -rf (Replaces a bad track with zeros.)
$ alcckd /z/WORK03 -r (Displays a volser and its size.)
The -rs function scans the emulated volume and verifies that it is in the correct 1090 emulation format.1 The -rf function replaces improperly formatted tracks with a properly formatted track containing zeros. The original contents of the track are lost, but the remaining function of the volume is maintained.
Assuming that your emulated 3390 volumes are in the /z directory and there are no other file types in this directory, you can verify the format of all the volumes with the following Linux shell commands:
$ cd /z (location of emulated CKD, and nothing else)
$ for i in *; do alcckd $i -rs; done
The ckdPrint command can be used to examine the contents of an emulated CKD volume. This command prompts for the beginning cylinder and head numbers and the ending cylinder and head numbers. These numbers are in decimal format. The following example lists the records on cylinder 0, head 0 of volume Z9SYS1:
$ ckdPrint /z/Z9SYS1
DeviceType 3390, Cylinders-3339, Trks/Cyl-15, TrkSize-56832
Input extent in decimal - CC-low HH-low CC-high HH-high
00 00 00 00
....
....
The tapeCheck command can be used to verify the format of an awstape file. This command reads the awstape file and verifies that the awstape control blocks are logically correct.
19.6 Special problem-related commands
A number of ISV zPDT commands are intended to be used only at the direction of IBM development personnel, and they supply the specific commands and operands to be used. This category includes the following commands:
$ awslog (including --logsize and --logcount operands)
$ doOSAcmd (various subcommands)
$ dreg (shared resource registry)
$ dshrmem (shared memory)
$ printlog (only for some .gz logs)
$ printtrace (only for some .gz traces)
$ qpci (Displays zEDC information.)
The contents and formats of the various log and trace files are not documented and intended only for diagnostic use. Our experience is that these files are not useful for solving general user-level problems.
19.7 Linux monitoring
Some monitoring of Linux statistics might be helpful. The vmstat command is useful and causes little interference with other processes in Linux. Here is its syntax:
$ vmstat -w 3 4 (4 samples at intervals of 3 seconds)
procs -----------memory------------ ---swap--- -----io----- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 0 397600 101936 2398160 0 0 109 52 154 350 14 2 81 3 0
r = number of Linux processes waiting for run time
b = number of Linux processes sleeping
swpd = amount of swap space used (KB)
free = amount of idle memory (KB)
buff = amount of memory used as buffers (KB)
cache = amount of memory used as cache (KB)
si, so = Linux paging activity in KB/second
bi, bo = I/O activity in blocks/second
in = interrupts per second
cs = context switches per second
us = percent of CPU time in user mode
sy = percent of CPU time in kernel mode
id = percent of CPU time idle
wa = percent of CPU time waiting for I/O
st = timeslices stolen for virtual machine
-w = creates a wider report line with better column organization
Important data from vmstat includes the Linux swap rates (si, so). Any number here can degrade ISV zPDT performance. If ISV zPDT suffers a page fault, then the whole
IBM zSystems CP operation must wait for the page fault to be resolved. The wa statistic (CPU percentage waiting for I/O) provides an indirect indication of I/O overload.
IBM zSystems CP operation must wait for the page fault to be resolved. The wa statistic (CPU percentage waiting for I/O) provides an indirect indication of I/O overload.
The top command provides data about individual Linux processes. We are most interested in the emily process (which is the name of the Linux process representing the CP), but information about various device manager processes can be useful in spotting bottlenecks. Enter q from the keyboard to terminate the top command:
$ top
top - 11:14:14 up 1:24, 3 users, load avg: 1.26, 1.15, 1.02
tasks: 128 total, 1 running, 121 sleeping, 6 stopped, 0 zombies
CPU(s): 49.9% us, 0.2% sy, 0.0% ni, 49.7% id, 0.0% wa 0.0% hi, 0.0% si
mem: 3111660k total, 2719204k used, 392456k free, 105356k buffers
swap: 2104504k total, 0k used, 2104504k free, 2397740k cached
PID USER PR NI Virt RES SHR S %CPU %MEM TIME COMMAND
7040 ibmsys1 25 0 1549m 1.5g 1.5g S 100 49.6 35:41.2 emily
7051 ibmsys1 16 0 1510m 36m 36m S 0 1.2 0:03.1 awsckd
19.8 Summary
The material that is presented in this chapter (and in this publication) is intended to point to some basic elements that might help resolve ISV zPDT issues or provide starting points for investigating problems. Before requesting help from a service provider (if one exists) or posting problems on various websites, make a sincere effort to reduce the description of a problem to the most basic components.
1 Only the emulation format is checked. There is no check for data content or operating system metadata (label, Volume Table of Contents (VTOC), and such).
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.