Tips for critical message detection: •OPERLOG must be active. •The list of messages for which OPERLOG is searched is a hardcoded list defined by IBM and cannot be modified. •OPERLOG is searched back one hour from the current time (if available). •The Runtime Diagnostics event contains a summary of the message and the recommended actions. •Some messages are grouped into one event that describes a single problem symptom. •The detection is performed on the system specified in the SYSNAME parameter. If SYSNAME was not specified, the detection will be performed on the system that the command was issued on.

Tips for enqueue contention detection: •The address space must be in the list of system address spaces as defined by Runtime Diagnostics. This list cannot be modified. •The address space must be waiting for an enqueue for at least five seconds. •The detection is performed on the system specified in the SYSNAME parameter, or on the home system if SYSNAME was not specified.

Tips for GRS latch contention detection: •This function is available starting with z/OS 1.13. •The event is created for any address space waiting for a latch for more than five minutes. •Contention in z/OS UNIX System Services file system latches are handled separately. •The top waiter is the address space that has been waiting for the latch set the longest.

Tips for z/OS UNIX System Services file system latch contention detection: •This function is available starting with z/OS 1.13. •The event is created when a thread in any address space is waiting for a UNIX System Services latch for more than five minutes. •The normal action to take is to issue the D OMVS,W,A command.

Tips for CPU analysis: •Two snapshots are taken one second apart. •If any task is using more than 95 percent of a single CPU, a high CPU event is created. •The event might show usage greater than 100 percent. •A high CPU event combined with a loop event for the same job usually indicates the job is truly in a loop.

Tip for local lock suspension analysis: •If an address space is suspended for more than 50 percent of the time, a high local lock suspension rate event is created.

Tips for TCB loop detection: •The analysis is done for all tasks in all address spaces. •A high CPU event combined with a loop event for the same job usually indicates that the job is truly in a loop.

Tip: PFA does not take the time of day or the day of the week into account when comparing current values to projections of what the current values should be. However, by using CPU consumption to normalize its rate metrics, it effectively takes into account whether the system is busy or quiet. In practice, this has been found to be effective and negates the need to take time or day into account when creating its model of system behavior.

Display all parameters and obtain status of PFA checks: IBM Health Checker for z/OS is unaware of the cumulative nature of the PFA checks’ parameters. Therefore, if you display a PFA check using the IBM Health Checker for z/OS modify command to display details, or you look at the parameters in the reports, only the parameters that were last modified are displayed. To properly display all the parameters and to obtain the status of the PFA checks, use the PFA MODIFY command with the display details option such as shown: f pfa,display,check(pfa_*),detail Variations of the MODIFY PFA command and a description of the parameters can be found in z/OS Problem Management, G325-2564.

Important points about PFA modeling: •Modeling is done in Java and is zAAP eligible. It is strongly advised to offload PFA’s Java processing to a zAAP. •Even though the MODELINT value is configurable, you cannot force PFA to model at a specific time of day due to the fact that PFA models more frequently when your system is unstable to ensure that the model is current. When the system stabilizes, PFA resumes using the specified MODELINT value.

Important points about PFA comparisons: •If the value of the INTERVAL parameter in IBM Health Checker for z/OS is set to ONETIME for a PFA check, it is a check that runs after every collection. Do not change this setting. If you do, the comparison uses the data collected at the last collection rather than retrieving new current usage data. •Even though IBM Health Checker for z/OS allows you to run a health check on demand by using IBM Health Checker for z/OS interfaces, it is best to allow PFA to run the checks when PFA determines it is time for the comparisons to be performed. When a check is run manually that has the INTERVAL parameter set to ONETIME, message HZS0400I is issued with message AIR023I embedded. You can display the current INTERVAL value by using the SDSF CK command and scrolling to the right to the column titled “INTERVAL.”

Resource or metric	Causes detected	Type of analysis	Release
Common storage usage	Detects resource exhaustion in common storage by looking for spikes, leaks, and creeps.	Future prediction	1.10 SPE
LOGREC record arrival rate	Detects a damaged address space or system.	Expected rate	1.10 SPE
Console message arrival rate	Detects a damaged or hung address space or system based on an abnormal rate in the WTOs and WTORs issued, normalized by the amount of CPU being used.	Expected rate	1.11
SMF record arrival rate	Detects a damaged or hung address space or system based on an abnormal rate of SMF record arrivals, normalized by the amount of CPU being used.	Expected rate	1.12
JES spool usage	Detects a damaged persistent address space based on an abnormal increase in the number of track groups used.	Expected value	1.13
Enqueue request rate	Detects a damaged or hung address space or system based on an abnormal rate of enqueue requests, normalized by the amount of CPU being used.	Expected rate	1.13

Apply the PTF for APAR OA40065: If you are familiar with PFA, you might have noticed that the check for frames and slots usage (PFA_FRAMES_AND_SLOTS_USAGE) was omitted from Table 2-1. This check has been permanently removed from PFA with APAR OA40065 because it caused unwarranted exceptions that could not be avoided with the available mechanisms. It is advisable to apply the PTF for APAR OA40065 as soon as possible. To remove the check from PFA prior to applying the PTF, use the following IBM Health Checker for z/OS command: f hzsproc,delete,check(ibmpfa,pfa_f*)

PFA_COMMON_STORAGE_USAGE check: •This check requires that the following DIAGxx parmlib member options are set:  VSM TRACK CSA(ON) SQA(ON) •When an exception occurs, refer to the best practices defined for this check in z/OS Problem Management, G325-2564.

PFA_LOGREC_ARRIVAL_RATE check: •The record must be a software LOGREC record with a usable SDWA to be counted as an arrival. •z/OS LOGREC provides two options for the LOGREC recording medium: a System Logger log stream, or the LOGREC data set. When a LOGREC record is produced, PFA is notified through an ENF listener. If you are writing a LOGREC record to the LOGREC data set and that data set fills up, PFA stops getting notified when a LOGREC is produced. Therefore, for the best reliability, use the log stream method. •If you change the COLLECTINT, perform the following steps to ensure that the values being modelled in the check remain consistent:  – First, stop PFA.  – Next, delete the files in the PFA_LOGREC_ARRIVAL_RATE/data directory.  – Finally, restart PFA. •If an exception occurs, refer to the best practices in z/OS Problem Management, G325-2564.

PFA_MESSAGE_ARRIVAL_RATE check: So that address spaces that produce erratic numbers of messages do not skew the predictions, the following exclusions are done: •The CONSOLE address space is excluded from this check’s processing. This behavior cannot be changed. •Any address space that starts with “JES” is excluded from this check’s processing by default through the PFA_MESSAGE_ARRIVAL_RATE/config/EXCLUDED_JOBS file. This behavior can be changed, but is not suggested.

PFA_SMF_ARRIVAL_RATE check: If a change to your SMF definitions would result in a significant change in the number of SMF records being produced, it is advisable to do the following: 1. Stop PFA. 2. Delete the files in the PFA_SMF_ARRIVAL_RATE/data directory. 3. Restart PFA. This ensures that PFA does not use the previously collected data in the models so that the models accurately reflect the new configuration.

PFA_JES_SPOOL_USAGE check: This check is only active if JES2 is the primary JES. A WTO message is issued when PFA starts if JES2 is not the primary JES, and the check will not be initialized.

PFA_ENQUEUE_REQUEST_RATE check: If you change the maximum number of concurrent ENQ, ISGENQ, RESERVE, GQSCAN and ISGQUERY requests, or change system-wide defaults using the SETGRS command or through the GRSCNFxx Parmlib member, or the system is radically different after an IPL (such as a change from a test system to a production system), perform the following steps: 1. Stop PFA. 2. Delete the files in the PFA_ENQUEUE REQUEST_RATE/data directory. 3. Restart PFA. This ensures that PFA is collecting relevant information and that your models accurately reflect the behavior of your system.

The address spaces tracked	Which checks track this data	Description
Total system as a group	Common Storage Usage LOGREC Arrival Rate Message Arrival Rate SMF Arrival Rate ENQ Request Rate	For some metrics, the data is required to be tracked for the entire system to determine whether exhaustion is occurring or if the cumulative failures on the system indicate a damaged system.   For example, for exhaustion of common storage, it tracks the common storage usage for the entire system. Similarly, detecting a damaged system based on LOGREC arrival rates by key groupings requires accumulating the arrivals for the entire system.
Every persistent address space individually	JES Spool Usage	A persistent address space in PFA is defined to be any address space that was started within the first hour after IPL. This definition was created so that PFA could detect critical system address spaces without using a hardcoded list.   Address spaces with duplicate names are not tracked. If a persistent address space restarts while PFA is running, it continues to be considered persistent even if it is assigned a different ASID.   This category is used to detect a damaged persistent address space.
Tracked, persistent address spaces	Message Arrival Rate SMF Arrival Rate ENQ Request Rate	Tracking address spaces that have the highest rates individually allows PFA to detect damaged or hung address spaces among the address spaces that tend to have the highest rates for this metric.   PFA checks determine which persistent address spaces to track by entering a six-hour warm-up phase for the check to determine which address spaces have the highest rates in that period. The warm-up phase waits for one hour after IPL and then collects data for six hours. At the end of the warm-up phase, the rates are calculated for each persistent address space and those with the highest rates are chosen to be tracked.   If PFA is restarted or an IPL occurs, PFA still tracks the same address spaces without entering the warm-up period if those address spaces still exist and the maximum PFA chooses to track were being tracked prior to the IPL or PFA restart. When PFA can track the same persistent address spaces, it can use the data collected prior to the IPL in its modeling.   Address spaces with duplicate names are not tracked individually, but if an address space that is tracked ends and restarts, it continues to be tracked even if it was assigned a different ASID. In this case, the data collected from the previous address space is used when making predictions for this address space.
All other persistent address spaces as a group	Message Arrival Rate SMF Arrival Rate	Some metrics also categorize all persistent address spaces that are not being tracked individually as a group and compare the actual usage of those persistent address spaces in this category to the group average.
Non-persistent address spaces as a group	Message Arrival Rate SMF Arrival Rate	Some metrics track the address spaces that started more than an hour after IPL as the non-persistent address spaces as a group.

PFA uses Runtime Diagnostics only for “too low” problems: PFA only uses Runtime Diagnostics for “too-low” problems because PFA is unable to determine whether or not the lack of activity is really a problem. For “too-high” problems, PFA is able to determine the problem on its own and is usually able to present helpful diagnostic information based on the information that is available to it. For this reason, and also to eliminate the CPU cost of running Runtime Diagnostics when it is not really needed, PFA does not invoke Runtime Diagnostics for “too-high” problems.

Parameter	Supported checks	Description
THRESHOLD	Common storage usage	This parameter is often confused and thought to be a hardcoded threshold (such as 80 percent) that the location must be using in order for the exception to be issued.   Instead, the THRESHOLD is a percentage of the capacity that the prediction must be before the comparisons are performed after domain knowledge is applied. That is, it sets the sensitivity level of the comparison to know when exhaustion of the available capacity is being predicted.   For example, if PFA’s domain knowledge says that the location should allow 100 percent of the available capacity and THRESHOLD(2) is used, the prediction must be at 102 percent of the available capacity before comparisons for exhaustion are performed.   This design allows PFA to detect exhaustion even though current usage might not be nearing exhaustion yet.   To make this check less sensitive so it issues exceptions less frequently, set THRESHOLD to a higher value so that a larger prediction is required before exceptions can be issued.
STDDEV	LOGREC Arrival Rate JES Spool Usage Message Arrival Rate SMF Arrival Rate Enqueue Request Rate	This value is used as a multiplier to calculate the comparison value for comparisons detecting if the current usage or rate is too high compared to the expected value.   A higher STDDEV value requires the difference between the current value and the prediction to be greater in order for an exception to be issued. Therefore, exceptions are issued less frequently with a higher STDDEV.   To reduce exceptions indicating the metric is too high for checks that have the STDDEV parameter, incrementally increase the value until you are satisfied with the results.   If an exception is issued, a systemNameRUN.LOG file is created in the EXC_timestamp directory for this exception. For some checks, the STDDEV value needed to avoid the exception might be printed to help you tune the check if the exception was unwarranted.
STDDEVLOW	Message Arrival Rate SMF Arrival Rate Enqueue Request Rate	This parameter is similar to STDDEV except that it is used in the algorithms detecting “too low” conditions.   This value is used to set the variance between the current value and the comparison value when detecting rates that are too low.   A larger value requires the current rate to be significantly lower than the expected rate in order for an exception to be issued.   To reduce exceptions indicating the metric is too low for checks that have the STDDEVLOW parameter, incrementally increase the value until you are satisfied with the results.   If an exception is issued, a systemNameRUN.LOG file is created in the EXC_timestamp directory for this exception. For some checks, the STDDEVLOW value needed to avoid the exception might be printed to help you tune the check if the exception was unwarranted.
EXCEPTIONMIN	LOGREC Arrival Rate JES Spool Usage Message Arrival Rate SMF Arrival Rate Enqueue Request Rate	This parameter is used when detecting abnormally high values and rates.   A higher EXCEPTIONMIN value requires the current value (and sometimes the prediction) to be higher than this value before an exception can be issued.   This parameter is used to eliminate exceptions for “noise” on the system. That is, if your system tends to have a low rate for a given metric, you can set this parameter to the minimum value that should be compared so that the normal, low rates are not even compared.   To reduce indicating the metric is too high when your system is stable and normally has a relatively low rate or value, set EXCEPTIONMIN to the minimum value you want to be compared.   For example, if your system typically has a message arrival rate of 20 and you do not want comparisons for high rates to be performed for rates that are less than 20, set EXCEPTIONMIN(20).
LIMITLOW	Message Arrival Rate SMF Arrival Rate Enqueue Request Rate	This parameter is available on all checks that detect hung address spaces or systems based on a determination that the rates were too low.   Its purpose it to avoid “too low” conditions for rates that are higher than this value that indicate normal system activity is occurring.   That is, comparisons for “too low” conditions are only performed for rates that are less than this value.   To reduce exceptions indicating the metric is too low for rates that are relatively high, set LIMITLOW to the maximum value you want compared for “too low” conditions.   For example, if your system typically has a message arrival rate of 3, set LIMITLOW to a value less than 3 so that low values are not even compared.
CHECKLOW	Message Arrival Rate SMF Arrival Rate Enqueue Request Rate	This parameter is available on all checks that detect hung address spaces or systems based on a determination that the rates were too low.   Use this parameter to bypass comparisons for low rates and values if you do not want PFA to perform this function.
TRACKEDMIN	Message Arrival Rate SMF Arrival Rate Enqueue Request Rate	This parameter is used by the checks that find individual persistent address spaces to track.   It defines the minimum rate that an address space must have at the end of the warm-up period in order for it to be tracked.   To allow an address space with any rate to be considered for tracking, set this value to 0.   If you notice that you have many tracked jobs with normally low rates and you receive unwarranted exceptions for this normal, low behavior, consider setting TRACKEDMIN to a higher value.   To force PFA to track a different set of address spaces after changing this value, you must follow these steps: 1. Stop PFA. 2. Delete the /data subdirectory for the check for which you are changing the TRACKEDMIN parameter. 3. Start PFA.

Tip: You might find that IBM zAware appears to be selective about which systems are able to be successfully trained. This is particularly the case for test or development systems. However, remember that IBM zAware is trying to emulate what you do when presented with a system problem, which is, identifying what is different about the system now, compared to its normal state. The best way that IBM zAware has of creating a reliable model of the normal state of a system is to require that monitored systems exhibit consistent behavior and a statistically significant number of messages. Successfully creating a model for every system, even one that produced few messages or exhibited no consistency, would simply result in IBM zAware marking normal messages as being anomalous. If it behaved in this manner, you would soon learn to ignore its results, thus reducing its value as a reliable tool. The topic of identifying suitable systems for IBM zAware to monitor is discussed further in 3.2, “Selecting which systems to monitor with IBM zAware” on page 96.

Column heading	Description
Interval Contribution Score	This shows how this message has contributed to the overall anomaly score for this interval. The higher the Interval Contribution Score, the more influence that message has on the interval anomaly score.
Message Context	This indicates whether this message appears in any clusters contained in the model. The possible values are: •NEW - This means that this message was not found in the model database for this system. •UNCLUSTERED - This means that the message is in the model, but IBM zAware did not identify any other messages that always appear together with this message. •IN_CONTEXT - This means that the message has appeared in a cluster in the model and in this interval, the message was issued together with the other messages in that cluster. •OUT_OF_CONTEXT - This means that this message was issued in the interval, but other messages in the cluster were not issued in this interval.
Rules Status	This shows the application of domain knowledge, where metadata has been stored about the context of a message. The possible values (in order of significance) are: •CRITICAL - the rule indicates that this message might be important when diagnosing a problem. •IMPORTANT - the rule identifies this message is likely to indicate a problem. •INTERESTING - the rule identifies this message is indicative of a diagnostically useful event, such as a health check exception. •None - no rule was found for this message. •NON-INTERESTING - the rule for this message indicates that it is not a significant message.
Appearance Count	This shows how often the message appeared in the 10-minute interval1.
Rarity score	This is an indicator of how often this message was issued during the model interval. A score of 101 means that the message was not found in the model. A low score means that this message was found frequently in the model.
Cluster ID	If this message is part of a cluster, the cluster ID identifies the cluster. All the messages in a cluster have the same Cluster ID.

---

Retraining consideration: You must wait until at least after midnight of the day the anomalies occurred because training uses only full days of data. The effect of retraining depends on the stability of the system and the degree to which the new messages cluster. Therefore, waiting more than one day to collect additional data allows the patterns to be more obvious, which can result in a more valid model.

Runtime Diagnostics message processing information: •Runtime Diagnostics looks for a hardcoded list of messages in the last hour of the OPERLOG. •Runtime Diagnostics is intended to be used “after the fact” when you think a system problem exists.

PFA message processing information: •PFA looks for abnormally high or low message arrival rates, normalized by CPU utilization, to detect address spaces or the system being hung or damaged. •PFA includes the current data in the historical data when creating a new model so that current, predictive trends can be established. •PFA is intended to detect and alert you to the abnormal behavior before it can be seen externally and to provide the potential address spaces that are the source of the problem to help you diagnose the problem.

IBM zAware message processing information: •IBM zAware understands message IDs and message patterns. •IBM zAware resides outside of z/OS, which means it can be used when the system is non-functional. •IBM zAware analysis is based on models that depict historical normal behavior for the system, rather than on predictive trends. •IBM zAware is intended to be used to validate major software changes, detect anomalies before they are externally visible, and help identify the cause of the problem after the fact when you think the system is experiencing a problem.

Function	Metric used	Data source	Intent	Analysis method	Output
Runtime Diag- nostics	Hardcoded list of critical component messages.	•OPERLOG •LPAR-specific	Diagnose a problem by finding specific messages in OPERLOG after event was reported.	When invoked, search OPERLOG back for 1 hour (if available) to find a critical message on the targeted system and perform additional analysis on some of them.	Runtime Diagnostics Critical Message Event sent to the console (default) with actions suggested.
PFA	Rate of the count of WTO and WTORs, divided by the CPU used in the collection interval. (Not counted by individual message ID).	•Console address space (before Message Flooding Automation) •LPAR-specific	•Detect and alert you to a problem before it is externally visible by detecting potentially damaged address spaces or system and issuing a WTO (which could be processed by your automation product) before your business is impacted. •Helps you diagnose a problem by identifying the potential address spaces causing the problem. Information is externalized using IBM Health Checker for z/OS.	•Predictive, trending model of historical and current rates stored for three time ranges (to distinguish workload variances) creates an expected rate for this point in time. •Applies statistical comparison algorithms using the current rates and expected rates and issues an exception if the current rate is statistically abnormal when compared to the expected rate. •Invokes Runtime Diagnostics to corroborate “too low” condition. •Compares top 10 individual jobs. •Compares other persistent jobs. •Compares non-persistent jobs. •Compares total system. •Performs comparisons every 15 minutes by default.	IBM Health Checker for z/OS exception sent as WTO (default).
IBM zAware	Scores based on message pattern analysis, using message IDs and message clusters, against learned historical message data for each system.	•OPERLOG (before Message Flooding Automation). •LPAR-specific .	•Make sure system is likely to work (validate new workloads, new releases). •Detect events before your business is impacted by detecting message anomalies. •Diagnose a problem by identifying the potential cause of the problem after event was reported.	•Creates a behavioral model of normal, historical behavior based on message ID and by message pattern. •Applies analytics to determine how far message activity deviates from the model of normal behavior. •Results updated every two minutes.	•IBM zAware GUI shows scores and results. •Can view all LPARs at once. •API can be used to make results visible to automation products.

Function	Benefits	Other considerations
Runtime Diagnostics	•Simple setup. •Simple command interface. •Output provides recommended actions. •Can search through volumes of messages and isolate critical messages quickly. •Only consumes CPU time when needed. •Applies domain knowledge.	•IBM-defined list is not modifiable. •No proactive detection capability (diagnostic only). •Returns only messages found in last hour (must be used in real-time immediately after symptoms detected). •Not able to use if system is unresponsive. •Must be run for each LPAR (when needed).
PFA	•Reasonably simple setup. •Identifies address spaces potentially causing problem. •Can detect potentially hung address space or system based on rate that is too high or too low. •Attempts to understand workload changes across time so that unwarranted exceptions are not produced when workloads change. •Alerts operator through WTO (by default). •Alert can be acted on by your automation product. •Can be customized to exclude erratic address spaces from all processing. •Requires minimal data to model and little storage for persistence. •Counts all WTO and WTOR messages for this LPAR, including ISV and application-generated messages (except for address spaces specifically excluded) •Self-learning based on individual system workload. •Remodels as necessary to improve understanding of recent trends. •Exclusion list is modifable. •Applies domain knowledge and excludes some address spaces by default (such as JES* address spaces and CONSOLE, which cause “noise”).	•Relies solely on rates - has no message ID knowledge. •Has no understanding of which messages are critical. •Does not provide an option to exclude times when problems were occurring from next model. •Requires z/OS system resources to run •Analysis occurs after every collection (15 minutes by default), but is configurable •Not able to use if system unresponsive. •Must be running on each LPAR you want analyzed.
IBM zAware	•Independent of the operating system; can be used even if system is unresponsive. •Consumes minimal additional z/OS system resources. •Analyzes by message ID and the group of messages that they normally appear with (this is referred to as “clustering” in IBM zAware terminology). •Analyzes messages by count to determine whether rate is higher than expected •Can analyze large volumes of messages and isolate intervals of anomalous behavior quickly. •Finds anomalous behavior of newly changed workloads, newly installed products, or changes due to a new release based on message ID and patterns. •Tells you about anomalies, in contrast to Runtime Diagnostics, which looks for a set of known problems. IBM zAware tells you what is different, and you determine whether it is a problem. •Near real-time analysis with the IBM zAware GUI, which is refreshed every two minutes. •The IBM zAware APIs can be used to automatically monitor for intervals with an anomaly score above a threshold that you specify. •Can exclude data from next model by day. •Groups analysis results by sysplex (not consolidated; each LPAR is analyzed separately). •Analyzes all messages in OPERLOG for each LPAR, including ISV, suppressed, and application-generated messages. •Model can be updated to reflect a “new normal.” •Analysis of messages can be done for problems that started days or weeks ago (if datis available). •Applies domain knowledge; gives more weight to critical messages and removes specific messages as “noise” (such as JES2 messages).	•More effort required to set up than Runtime Diagnostics or PFA. •Cannot exclude specific address spaces or specific messages. •Cannot detect message traffic that is unusually low or messages that are missing. •The message text listed in the interval view might not be representative of the message related to the problem (for message IDs that have multiple possible message texts). •Requires more data for training and more storage for persistence. •Must be retrained if workloads or system changes significantly from model. •Each LPAR analyzed must be connected to IBM zAware. •No automated alerts (by default), but can be automated through APIs and XML output.

---