Tivoli Workload Scheduler: Architecture and concepts
This chapter focuses on important availability aspects in the architecture of Tivoli Workload Scheduler that are related to the development of the Tivoli System Automation for Multiplatforms policies. Knowing about the policies can help you react to certain events in the network.
This chapter contains the following topics:
24.1 Components
This section concentrates on the availability of critical components in a Tivoli Workload Scheduler network. If a specific part is not available, certain services that are provided to the user can be affected. Understanding how a particular failure might affect the network and which symptoms can arise from that error is important. This knowledge can significantly improve problem determination.
24.1.1 Engine
The Tivoli Workload Scheduler engine is the core component installed on each workstation. The most important role is fulfilled by the engine on the master domain manager, which generates the production plan and routes messages to all workstations.
If the master domain manager is not available, the production plan is not extended, and any message is not routed in the network. An unavailability of the engine must therefore be resolved as quickly as possible. Migrating the role of master domain manager to any full status agent in the top-level domain can be achieved by running conman switchmgr command. See 24.3.4, “Switch Manager” on page 739 for a detailed description of the mechanism.
24.1.2 Database
The Tivoli Workload Scheduler database contains the definitions of all scheduling objects and event rules. Moreover, for each triggered event rule, an instance is created in the database and the result of the triggered action is logged.
If the database is unavailable, the following problems might arise:
•Extension of the plan fails because planman is unable to establish a connection with the database.
•All composer and optman commands break down.
•Ad-hoc submission from the database fails because the scheduling objects cannot be retrieved. As a result, conman sbs and conman sbj commands do not function properly.
•Creation, modification, deployment and triggering of event rules stops functioning.
•Submission of dynamic jobs breaks down.
Although planned workload does not suffer from a problem with the database, a problem with it must certainly be fixed prior to the start of the following production day. If not, planned jobs are affected because the extension of the plan fails. The issue might even be more pressing because event rules are no longer triggered. Moreover, dynamic scheduling is also unavailable during an outage of the database.
If the issue cannot be fixed quickly, a switch to a standby database must be made. For an in-depth discussion of how this switch can be achieved for a DB2 database, see 24.3.1, “DB2: ACR and HADR” on page 733.
24.1.3 The embedded WebSphere Application Server
To interact with the engine or the database, the client interfaces use a connector which is based on embedded WebSphere Application Server. By default, the embedded WebSphere Application Server instance is installed on the master domain manager and backup master domain managers. Managing the connector is handled by the appservman process which reads its requests from the Appserverbox.msg message queue.
If the connector is unavailable on the master domain manager, the following symptoms might be observed:
•Extension of the plan fails because planman cannot contact the database.
•All composer and optman commands break down because no connection with the database can be established.
•Ad-hoc submission from the database fails because the scheduling objects cannot be retrieved. As a result, conman sbs and conman sbj commands do not function.
•If the Event Processor was running inside the application server, it is also unavailable. As a result, no event rules are triggered.
•The Dynamic Workload Broker Component fails if it was hosted by the failing embedded WebSphere Application Server instance. As a result, no dynamic jobs are launched. See 24.1.6, “Dynamic Workload Broker Component” on page 723 for a detailed discussion.
All planned workload during the production day does not experience any issues in case of a problem with the connector. Similar to an issue with the database, a problem with the application server must be addressed prior to the start of the following production day to avoid affecting the planned workload. If the error cannot be resolved, a connection to a different embedded WebSphere Application Server instance must be established. For command-line tools (composer, conman, optman), this step can be achieved by altering the embedded WebSphere Application Server connection properties in the localopts configuration file, as shown in Example 24-1.
Example 24-1 Embedded WebSphere Application Server connection properties in localopts
#HOST = #PROTOCOL = https
#PROTOCOL = http
#PORT =
#PROXY =
#PROXYPORT =
#TIMEOUT = 3600
#CLISSLSERVERAUTH = yes
#CLISSLCIPHER = MD5
#CLISSLSERVERCERTIFICATE =
#CLISSLTRUSTEDDIR =
#DEFAULTWS =
#USEROPTS =
The state of the application server can be observed in the local Symphony™ file by using the conman showcpus command, as shown in the reduced output of Example 24-2. State ‘A’ indicates that an embedded WebSphere Application Server instance is active on the particular workstation.
Example 24-2 The embedded WebSphere Application Server and Event Processor state
TWSPOC1 UNIX MASTER LTI JW EA FTA_1 *UNIX FTA I J MD
FTA_2 UNIX FTA
TWSPOC2 UNIX FTA A
XA_VMDM UNIX X-AGENT
The appservman process regularly checks the status of the embedded WebSphere Application Server instance. The interval can be controlled by the Appserver check interval localopts tunable. Moreover, as the status of the application server is visible in the Symphony file on each workstation, the appservman process is also responsible for generating state messages. These messages are put in the local Mailbox.msg message queue where the connector is installed. Subsequently, state messages are transferred to the master domain manager that will broadcast them further in the network. For more details about the interprocess communication, see 24.2.2, “Workstation interprocess communication” on page 725.
24.1.4 Event Processor
The Event Processor is a J2EE application running inside an embedded WebSphere Application Server in the network. Any Full Status workstation in the top-level domain that is running the connector is eligible for its hosting. The most important components of the Event Processor are as follows:
•The Event Correlation Engine is the core component of the Event Processor that receives all events from the monitoring agents. Each incoming event is sequentially passed to all active rules to verify the correlation conditions.
•The Rule Builder is responsible for building event rules and deploying them on the target workstations. See 24.2.4, “Event driven workload automation” on page 728 for more details about the deployment process.
•After all correlation conditions are satisfied, the Action Helper makes an instance of the event rule in the database and executes the associated actions. The result of the triggered actions is also logged in the Tivoli Workload Scheduler database.
During an unavailability of the Event Processor, triggering and deployment of event rules no longer function properly; creation and modification of existing rules experience no issues if the database is available. The monitoring agents notice the unavailability of the Event Processor and queue events in a local cache. For more information about this mechanism, see 24.2.6, “Store-and-forward ” on page 732. In case of a permanent failure, another embedded WebSphere Application Server instance in the network can be promoted to house the Event Processor. This promotion can be achieved by issuing the conman switchevtproc command. See 24.3.5, “Switch Event Processor” on page 740 for an in-depth discussion of the mechanism.
Like the embedded WebSphere Application Server, the state of the Event Processor can be examined by the conman showcpus command. State ‘E’ implies that the Event Processor is running on the specific workstation. Similar to the application server, the Event Processor will put a message in the local Mailbox.msg queue when it is started. Next, the local engine will forward the message to the master domain manager, which will distribute it further in the network. As a result, all workstations in the network will be able to determine the location of the Event Processor and update their configuration files.
|
Note: If Event Processor is switched to another embedded WebSphere Application Server instance, unlinked workstations are not aware of this migration. As a result, connections to the wrong Event Processor instance are attempted.
|
24.1.5 Tivoli Dynamic Workload Console
As of Tivoli Workload Scheduler version 8.5, running the Tivoli Dynamic Workload Console inside a single Tivoli Workload Automation instance is possible. In case of a failure of the embedded WebSphere Application Server, the web console will not be accessible. Therefore, be sure to separate the Tivoli Dynamic Workload Console from the embedded WebSphere Application Server instance.
24.1.6 Dynamic Workload Broker Component
Starting with version 8.5.1, the Dynamic Workload Broker Component can be installed as part of the Tivoli Workload Scheduler infrastructure. The workload broker extends the capabilities of the traditional engine by enabling dynamic scheduling. The Dynamic Workload Broker Component runs inside an embedded WebSphere Application Server instance and is installed by default on the master domain manager and backup master domain managers. The workload broker consists of the following important components:
•The Resource Advisor component of the workload broker server receives real-time information from the Resource Advisor Agent component of the workload broker agents that periodically scan for available physical resources (both hardware and operating system information). Along with logical resources, this data is stored in the Resource Repository, which is located in the Tivoli Workload Scheduler database.
•When a dynamic job is submitted, the Job Dispatcher interacts with the Resource Advisor and allocates the required resources in the Allocation Repository. This repository keeps track of the resources that are assigned to each dynamic job and is also located in the Tivoli Workload Scheduler database.
•The Job Dispatcher on the workload broker server instructs the workload broker agent to launch and monitor the dynamic job.
The sequence outlined shows that the workload broker server is highly dependent on the Tivoli Workload Scheduler database. As a result, if the database or the embedded WebSphere Application Server that is hosting the workload broker server becomes unavailable, no more dynamic jobs are launched. In case of a failure, the Dynamic Workload Broker Component can be migrated to a backup master domain manager in the top-level domain. See 24.2.5, “Dynamic scheduling” on page 731 for an in-depth discussion on the dynamic scheduling internals.
|
Note: The Dynamic Workload Broker Component must be collocated with the master domain manager. If a failure of the Dynamic Workload Broker Component occurs, the master domain manager role will be migrated also.
|
24.2 Network communication
Before developing Tivoli System Automation for Multiplatforms policies for Tivoli Workload Scheduler, a deeper understanding of how the network operates and how the workstation interprocess communication works internally are required. This section will discuss the details of the building blocks in a Tivoli Workload Scheduler network and their mutual relationships. The integration of Event Driven Workload Automation and the Dynamic Workload Broker Component will also be dealt with thoroughly. The study of these critical components and their common interaction is important when developing the intelligence of the automation policies.
24.2.1 Engine initialization
When the plan is extended, the entire network is restarted and the new Symphony file is distributed to all workstations and subordinate domain managers. The following operations take place during network initialization:
1. On the domain manager, the mailman process examines the Symphony file and learns which workstations it needs to connect to.
2. For each discovered fault tolerant agent (FTA), the mailman process on the domain manager establishes a TCP/IP connection to the remote netman process on the workstation.
3. The netman process on the FTA notices that the request is originating from a remote mailman process on the domain manager and subsequently creates a writer child process to handle the incoming connection.
4. The spawned writer process on the workstation examines the local Symphony file (if it exists) and reports the current run number back to the mailman process on the domain manager. If wr enable compression is set to yes in the localopts configuration file, compression of the Sinfonia file is also requested by the writer process on the FTA.
5. The mailman process on the domain manager compares its Symphony run number with the run number of the Symphony file on the workstation. If the run numbers differ, the mailman process on the domain manager sends the Sinfonia file to the writer process on the FTA using the unidirectional link.
6. The writer process on the workstation decompresses the Sinfonia file, if needed, and copies it to Symphony.
7. The mailman process on the domain manager issues a start operation for the workstation from the netman process on the FTA.
8. The netman process on the workstation starts the local mailman process which in turn examines the local Symphony file. Subsequently, the local mailman process on the FTA discovers its domain manager and links back to it, resulting in a writer process for that workstation on the domain manager.
9. Finally, the batchman process on the FTA is started by mailman and the jobman process is started by batchman.
Figure 24-1 on page 725 shows a graphical representation of these operations.
Figure 24-1 Engine initialization
|
Notes:
•To balance the load, there can be multiple mailman processes on the domain manager. The main mailman process on the domain manager remains in charge of distributing the Sinfonia file to all workstations and subordinate domain managers. The use of additional mailman processes might reduce the time to initialize a workstation.
•The sequence assumes that the workstation has the autolink flag set in its definition. See 24.2.6, “Store-and-forward ” on page 732 for more details about autolink.
|
24.2.2 Workstation interprocess communication
Now that the plan has been distributed, each workstation can start processing jobs. The following sequence of events occurs on a workstation throughout the production day:
1. The batchman process periodically checks the Symphony file to determine which jobs are ready to run. The interval used by the batchman process is controlled by the bm look localopts parameter.
2. If a job is ready to run, the batchman process writes a record in the Courier.msg message queue. The jobman process periodically scans Courier.msg for new requests. The interval used by the jobman process is regulated by the jm read localopts tunable.
3. The jobman process discovers the new message in the Courier.msg message queue. Subsequently, the requested job is launched and an entry is written in the Mailbox.msg queue indicating the job has started. The mailman process periodically scans Mailbox.msg for new messages. The time-out used by the mailman process is governed by the mm read localopts parameter. The jobman process will monitor the progress of the executing job. The interval used by the jobman process is controlled by the jm look localopts parameter.
4. The mailman process detects the new entry in the Mailbox.msg queue and puts a record in both the Mailbox.msg on the domain manager and the local Intercom.msg queue. The local batchman process periodically checks the Intercom.msg queue for new records. The interval used by the batchman process is regulated by the bm read localopts tunable. The mailman process on the domain manager propagates the message to remote writer processes in the network that are interested in the event.
5. The batchman process retrieves the message in the Intercom.msg queue and updates the local copy of the Symphony file indicating the job is executing.
6. When the job finishes, the jobman process puts a job termination record in the Mailbox.msg queue.
7. The mailman process discovers the entry in Mailbox.msg and puts a message in both the Mailbox.msg on the domain manager and the local Intercom.msg queue. Again, the mailman process on the domain manager further propagates the change in the network.
8. The batchman process retrieves the message in Intercom.msg and updates the Symphony file indicating the job has finished.
Figure 24-2 gives a schematic overview of the steps.
Figure 24-2 Workstation interprocess communication
|
Notes:
•The writer process on the domain manager receives messages from the remote mailman process on the FTA and puts them into the local Mailbox.msgqueue. The writer process on the workstation receives messages from a remote mailman process on the domain manager and puts them into the local Mailbox.msg. Together, a bidirectional link is formed in which state messages are exchanged.
•The mm read tunable is currently not present in the localopts file. APAR IZ74286 was created to address this issue. The default is 15 seconds.
|
24.2.3 Job execution
When batchman instructs jobman to launch a job, the following events take place:
1. The jobman process reads the BL record from Courier.msg message queue and forks a copy of itself. The effective user ID of the forked child process (Job Monitor) will be the STREAMLOGON user that is defined for the job.
2. The spawned jobman process forks again. The child process performs an execve system call to run the jobmanrc script. The script or command that must be executed, is passed as an argument to the jobmanrc script.
3. If a local configuration jobmanrc script exists for the user and its use is allowed, it is also executed.
4. The job is launched.
5. Upon successful completion (the job’s return code matches the return code condition, or zero if no condition is defined), the jobman process puts a JS (Job Successful) message in the Mailbox.msg message queue.
6. Regardless of the return code, a JT (Job Termination) message is written in the Mailbox.msg queue.
Figure 24-3 summarizes the events graphically.
Figure 24-3 Job execution details
|
Notes:
•If the job ends unsuccessfully, no JS record is written in the Mailbox.msg queue. The job will be marked as ABEND.
•The jobman executable is a setuid root program.
•The maximum number of concurrent Job Monitor processes is governed by the CPU limit of the workstation.
|
24.2.4 Event driven workload automation
After an event rule is saved in the database and marked as complete, it will be activated and deployed by the Rule Builder component of the Event Processor. The deployment process can be initiated either manually, using the planman deploy command, or automatically, which is governed by the deploymentFrequency global option. The deployment process consists of the following steps:
1. The Rule Builder generates configuration files for each specific workstation and compresses them. A dedicated compressed file is created for each FTA. The event rule is activated in the Event Correlation Engine by the Rule Builder.
2. The monman process on the affected workstation is notified about the new available configuration.
3. The monman process downloads the compressed file using the HTTP/HTTPS protocol.
4. The monman process decompresses the content of the deployconf.zip file into the monconf subfolder on the FTA.
5. The monman process starts the ssmagent process.
Figure 24-4 illustrates the deployment process.
Figure 24-4 Event Driven Workload Automation: deployment process
|
Note: Although the ssmagent process is started by the monman process, it is not a child of monman process.
|
The communication mechanism employed for the exchange of status messages between the Event Processor and the monitoring agents relies on the Event Integration Facility (EIF), which originates from Tivoli Enterprise Console. Note that the communication mechanism that is used by the EIF has no affinity with traditional Tivoli Workload Scheduler engine communication. As a result, event rules can still be deployed or triggered even if the workstation is unlinked.
Tivoli Workload Scheduler Object Monitor
The Tivoli Workload Scheduler Object Monitor built-in event provider is responsible for monitoring specific events related to the current plan, which are categorized as internal events. The objects that can be monitored include workstations, jobs, job streams, prompts and application server instances. These objects are monitored on the following locations in the network:
•Job streams are monitored on the master domain manager.
•Jobs are monitored by the workstation on which they run. If a job is run on a standard agent, the status will be monitored by the corresponding domain manager. If a job is run on an extended agent, the status will be monitored by the host on which the extended agent runs.
•A workstation is monitored by itself.
•Global prompts are monitored by the master domain manager.
•Local prompts are monitored by the workstation that is responsible for the job of job stream that has a dependency on it.
•An application server is monitored by the workstation on which it is running.
State changes for jobs, job streams, and prompts are monitored by the local batchman process. Workstation state changes are watched by the mailman process and the state of an application server is examined by the appservman process. All events are written to the Monbox.msg message queue, which is read by the monman process. Finally, the monman process forwards the events to the Event Correlation Engine on the Event Processor.
Figure 24-5 displays all components on a single workstation.
Figure 24-5 Tivoli Workload Scheduler Object Monitor
File Monitor
The File Monitor built-in event provider delivers file-based monitoring on the workstation where it is deployed. These events are classified as external events. The Netcool SSM (System Service Monitors) agent provides several subagents which are capable of monitoring and parsing text files. These subagents generate events, which are dispatched to the Event Correlation Engine on the Event Processor, as illustrated in Figure 24-6.
Figure 24-6 File Monitor
|
Note: The monman process merely delivers an interface to control the ssmagent process (start/stop). There is no affinity between these processes and each one provides distinct monitoring functions.
|
24.2.5 Dynamic scheduling
The Dynamic Workload Broker Component extends the capabilities of the traditional Tivoli Workload Scheduler engine by allowing a dynamic election of the workstation on which a job can run. The corresponding Tivoli Workload Scheduler job is launched on the workload broker server, which emulates a standard agent in the Tivoli Workload Scheduler network. The following steps explain briefly how a dynamic job is launched:
1. The Resource Advisor Agent component of the JobManager process periodically scans the local system for physical resources. The interval used by the JobManager process is regulated by the ScannerPeriodSeconds in the JobManager.ini configuration file. The Resource Advisor Agent will send the result from the scan to the Resource Advisor on the workload broker server every NotifyToResourceAdvisorPeriodSeconds seconds.
2. When a dynamic job is submitted, the Job Dispatcher interacts with the Resource Advisor to find the most suitable candidate for running the dynamic job. After a candidate is found, the job submission request is sent to it. The JobManager process on the elected agent launches a taskLauncher process.
3. The taskLauncher process launches and monitors the job.
4. After the dynamic job has completed, the taskLauncher process informs the JobManager process about the exit code.
5. Finally, the JobManager process reports the status of the dynamic job back to the Job Dispatcher component of the workload broker server.
Figure 24-7 illustrates the sequence graphically.
|
Important: The regular Tivoli Workload Scheduler network is not used by the Dynamic Workload Broker Component for delivering messages. As a result, dynamic jobs can be launched even if the workstation is unlinked.
|
Figure 24-7 Dynamic scheduling
24.2.6 Store-and-forward
To guarantee a consistent network, Tivoli Workload Scheduler employs a store-and-forward mechanism by queuing messages while a TCP/IP connection is not available. If the link with the domain manager is not available, the workstation queues outgoing messages in the tomaster.msg message queue located in the pobox subdirectory. If a link with a workstation is broken, the domain manager queues outbound messages in a XYZ.msg file in the pobox subfolder, where XYZ is name of the unlinked FTA.
An important aspect in the store-and-forward technology is the opening and closing of links, which is mainly controlled by the autolink flag in the workstation definition. This flag indicates whether the domain manager should open the link to the FTA when the domain manager starts. As mentioned, this will be important when the plan is extended, which causes a restart of the entire network. Moreover, if a workstation is stopped gracefully, its link to the domain manager will be closed because the mailman process on the FTA is halted. As a result, the corresponding writer process on the domain manager for this particular workstation will also cease to exist. However, the link from the domain manager to the FTA remains open until one of the following situations occurs:
•The mailman process on the domain manager times out the connection to the workstation. As a result of this unlink operation, the writer process on the FTA is halted. The time-out used by the mailman process on the domain manager is regulated by the mm unlink localopts tunable.
•A manual unlink operation is issued on the domain manager.
Alternatively, if the link from the domain manager is closed unexpectedly (such as by killing the writer process on the FTA), the mailman process on the domain manager tries to reestablish the link to the workstation. The interval that is used by the mailman process is governed by the mm retrylink localopts parameter. If the workstation is unlinked manually, the remote mailman process does not try to restore the link.
The event driven workload automation framework has store-and-forward techniques similar to the ones of a regular Tivoli Workload Scheduler network. If the Event Processor cannot be contacted, the monitoring agents will queue outbound messages for a later delivery. Each monitoring agent uses a separate queue. The monman process uses the monmancache.dat file in the EIF subfolder; the ssmagent process employs the ssmcache.dat file. The sendevent command, which is used to generate events from the command line, has its own queue sendeventcache.dat. Figure 24-8 and Figure 24-9 on page 733 each show a view of all store-and-forward queues during a network outage.
Figure 24-8 Store-and-forward Event Driven Workload Automation
Figure 24-9 Store-and-forward Tivoli Workload Scheduler
24.3 High availability
This section will concentrate on the concepts of high availability features for the master domain manager, which are delivered ready to use. To guarantee a transparent failover to a backup master domain manager, these features will be used by the automation policies. The use of DB2 as backend RDBMS will be assumed.
24.3.1 DB2: ACR and HADR
By default, DB2 includes several features that can be used to improve availability. The most important are automatic client reroute (ACR) and high availablity disaster recovery (HADR):
•ACR enables the DB2 JDBC driver in the embedded WebSphere Application Server to recover from a loss of communication with the primary database instance by rerouting the connection to an alternate server. This connection rerouting occurs automatically without any manual intervention. However, ACR does not synchronize the databases between the primary and standby DB2 servers.
•HADR is a database replication technology which provides high availability and disaster recovery for both partial and complete site failures. Both ACR and HADR can be combined to achieve a higher degree of availability.
|
Important: If ACR and HADR are combined, be sure to verify that the initial connection is pointing to the primary database. If the primary and standby databases switch roles, the embedded WebSphere Application Server data source properties should be updated accordingly.
|
ACR
The location of the alternate DB2 server is propagated to the JDBC driver when the embedded WebSphere Application Server makes an initial connection to the primary DB2 server. If the communication between the connector and the database server is interrupted, the JDBC driver on the embedded WebSphere Application Server attempts to reestablish the connection automatically by using the alternate server information. The following sequence occurs when the embedded WebSphere Application Server is started:
1. The JDBC driver of the embedded WebSphere Application Server connects to the primary DB2 server.
2. The primary DB2 server returns information about the standby DB2 server to the JDBC driver.
3. The JDBC driver caches this information in memory.
4. If a network interruption or failure of the primary DB2 server occurs, the JDBC driver of the embedded WebSphere Application Server automatically establishes a connection to the standby DB2 server.
Figure 24-10 depicts the sequence.
Figure 24-10 DB2 ACR
|
Note: ACR is enabled, by default, in the JDBC driver. No additional configuration is required on the client side. On the server side, the alternate server must be defined in the database directory.
|
HADR
DB2 HADR is a database replication mechanism in which log records use TCP/IP to transmit from the primary database server to a standby server. The standby server is in a continuous roll-forward mode and replays all log records to its copy of the database, keeping it tightly synchronized with the primary instance. The standby database in a HADR pair cannot be used by clients, not even for read-only access. If the primary database is not available, the standby database can take over its role. Be sure to either manually perform this operation or employ an automation tool.
DB2 HADR delivers three modes synchronization modes, each providing a level of protection:
•SYNC
In synchronous mode, log writes are considered successful only when the following situations occur:
– Logs are written to log files on the primary database.
– The primary database has received an acknowledgement from the standby database that the logs have successfully been written to log files.
Using this mode, ensures no loss of data if the HADR pair is synchronized.
•NEARSYNC
In near synchronous mode, log writes are considered successful only when the following situations occur:
– Log records are written to log files on the primary database.
– The primary database has received an acknowledgement from the standby database that the logs are in main memory.
If the primary database goes down and the standby also fails before takeover is completed, there will be loss of data, though this is unlikely to occur.
•ASYNC
In this mode, the primary database does not wait for an acknowledgement from the standby database. Log writes are considered successful only when the following situations occur:
– Log records are written to log files on the primary database.
– Log records have been delivered to the TCP/IP stack on the primary database. No acknowledgement is expected from the standby database.
Data loss is more likely to happen when using this synchronization mode.
Figure 24-11 illustrates the HADR synchronization modes and the point at which DB2 commits the transaction.
Figure 24-11 DB2 HADR synchronization modes
24.3.2 Fault tolerance
If the TCP/IP connection between the domain manager and the workstation is interrupted during the production day, the workstation is still able to submit local workloads to the workstation because it has a copy of the Symphony file. As described in 24.2.6, “Store-and-forward ” on page 732, the local workstation queues the message indicating the job has finished in the tomaster.msg queue if the connection with the domain manager is broken. When the connection is restored, the message is forwarded to the domain manager, which in turn broadcasts the message to other workstations that are interested in the event. When using dynamic scheduling, the concepts of fault tolerance differ because there is no production plan on the local workstation for dynamically submitted workloads. As illustrated in Figure 24-7 on page 731, the Job Dispatcher on the workload broker server contacts the agent synchronously if a job needs to run on the workstation. Therefore, while the communication between the workload broker server and the broker client is broken, the job does not start on the client.
The following example scenario illustrates the subtle differences between both technologies (dynamic and non-dynamic scheduling). Assume that job ABC, which has no predecessor dependencies, must run at 8:00 p.m. on workstation XYZ. Job ABC will take five minutes to complete. At 7.50 p.m., a planned network outage will last for 20 minutes. If job ABC is a traditional Tivoli Workload Scheduler job, the following chronological events take place:
1. (7:50 p.m.) The TCP/IP connection between the domain manager and XYZ is broken. Workstation XYZ is unlinked from the domain manager.
2. (8:00 p.m.) Workstation XYZ starts the job because it is in the local Symphony file. The state of the job in the local Symphony file changes from HOLD to READY/INTRO+/EXEC+.
3. (8:05 p.m.) Job ABC ends successfully. The state of the job in the local Symphony file is updated to SUCC. The state of the job in the Symphony file on the domain manager remains in the state READY.
4. (8:10 p.m.) TCP/IP connectivity between the domain manager and XYZ is restored.
5. (8.10 p.m. - 8:20 p.m.) After a maximum of mm retrylink seconds (default 600), the link with the domain manager is restored. The contents of the tomaster.msg queue on workstation XYZ, which contains the state changes of job ABC, is transferred to the domain manager.
6. (8:10 p.m. - 8:20 p.m.) The domain manager updates the local Symphony file and broadcasts the successful completion of job ABC to workstations that are interested in the event.
Figure 24-12 summarizes the events.
Figure 24-12 Fault tolerance I
If job ABC is a dynamic job that is bound to computer XYZ, the following chronological events take place:
1. (7:50 p.m.) The TCP/IP connection between the dynamic workload broker server and computer XYZ is interrupted.
2. (7:56 p.m. - 7:59 p.m.) From 360 seconds (MissedHeartBeatCount * NotifyToResourceAdvisorAgentSeconds) to 540 seconds ((MissedHeartBeatCount +1) * NotifyToResourceAdvisorAgentSeconds) after the network interruption, computer XYZ is marked Unavailable in the dynamic workload broker server. The MissedHeartBeatCount tunable (default 2) in ResourceAdvisorConfig.properties specifies the number of missed heartbeats after which a computer is listed as Unavailable.
3. (8:00 p.m.) The Tivoli Workload Scheduler engine on the master domain manager will instruct the dynamic workload broker server to start the job. Because the broker server cannot contact the broker client, the dynamic job will enter the state Waiting for resources for a maximum of MaxWaitingTime seconds. The global MaxWaitingTime tunable in ResourceAdvisorConfig.properties on the dynamic workload broker specifies the number of seconds a dynamic job can wait for a resource to become available. The state of the corresponding Tivoli Workload Scheduler job changes from HOLD to WAIT.
4. (8:10 p.m.) TCP connectivity between the dynamic workload broker server and XYZ is restored.
5. (8:10 p.m. - 8:13 p.m.) After a maximum of NotifyToResourceAdvisorPeriodSeconds seconds (default 180), the broker client will be able to connect to the broker server again. As a result, the dynamic workload broker server updates the state of computer XYZ to Available. Next, the Job Dispatcher will dispatch the job to computer XYZ. The state of the dynamic job changes to Running whereas the state of the corresponding Tivoli Workload Scheduler job changes to EXEC+.
6. (8:15 p.m. - 8:18 p.m.) After five minutes of run time, job ABC ends successfully. The state of the dynamic job is updated to Completed Successfully; the state of the corresponding Tivoli Workload Scheduler job is changed to SUCC.
Figure 24-13 illustrates the events.
Figure 24-13 Fault tolerance II
|
Important: Alternatively, the maximum resource waiting time for a dynamic job can be defined in the JSDL. If defined, the global tunable MaxWaitingTime in the ResourceAdvisorConfig.properties on the workload broker server is overridden.
|
24.3.3 Switch Fault Tolerance
In fullstatus mode, a workstation is updated with the status of jobs and job streams running on all other FTAs in its domain and in subordinate domains. As a result, the Symphony file on the fullstatus FTA contains the same information as the Symphony file on the domain manager, which acts as a message broker.
As described in 24.2.6, “Store-and-forward ” on page 732, each workstation stores outgoing messages in the tomaster.msg message queue if the domain manager is not available. If the domain manager is available, the mailman process on the workstation sends the message to the writer process on the domain manager, which will put the message in the local Mailbox.msg queue. Subsequently, the mailman process on the domain manager retrieves the record and forwards it to all workstations that are interested in that particular event, including all fullstatus agents. In the unlikely event that the domain manager fails before broadcasting the message to the fullstatus agents in the domain, the following situations might occur:
•The message is located in the Mailbox.msg queue on the domain manager, which is temporary unavailable. As soon as the domain manager is available again, it forwards the message.
•The message is located in the Mailbox.msg queue on the domain manager, which is permanently unavailable. The message is lost.
To overcome the possible loss of messages, Switch Fault Tolerance can be enabled by issuing optman chg sw=YES. The change will become effective when the Symphony plan is extended or created. If enabled, messages are no longer routed from the domain manager to fullstatus agents. When a workstation sends a message to its domain manager, it also sends the event to all fullstatus agents in the domain. If the FTA is unable to deliver the message to any of them, the event is stored in the pobox subdirectory on the workstation. When a fullstatus agent receives a direct message from the workstation, the local Symphony is updated and the event is buffered in a cyclical queue in the ftbox subdirectory, which acts as a recovery queue. As a result, all processed or partially processed messages are stored on at least two workstations. Events can be resent and reprocessed, thus eliminating the single point of failure of communication from the domain manager to fullstatus agents in the same domain.
Figure 24-14 shows the setup when Switch Fault Tolerance is enabled.
Figure 24-14 Switch Fault Tolerance
|
Tip: Limit the number of Full Status agents in the domain because having a high number of agents implies increased network traffic.
|
24.3.4 Switch Manager
If the domain manager is unavailable, any fullstatus agent in the domain can take over its role because the Symphony file on a fullstatus agent has the same content as the Symphony file on the domain manager. The fullstatus agent will be restarted and assume the role of the domain manager until the next plan extension, often referred as a short term switch. Subsequently, the old domain manager will be converted to a fullstatus workstation in the domain. The mechanism consists of the following steps and can be triggered from any FTA in the domain:
1. The conman switchmgr DOMAIN;XYZ command is issued on a workstation in the domain. XYZ is the name of the target workstation that will become the new domain manager of the domain.
2. The FTA establishes a TCP/IP connection to the netman process on the fullstatus agent XYZ, requesting the workstation to become the new domain manager.
3. Fullstatus workstation XYZ is restarted and becomes the new domain manager. A special message will be put in the local Mailbox.msg message queue indicating the domain manager switch.
4. The new domain manager XYZ links to all workstations in the domain and delivers the message.
5. On each workstation in the domain, the mailman process retrieves the event in the Mailbox.msg queue, unlinks from the old domain manager and relinks to the new domain manager XYZ. Messages that are stored in the tomaster.msg queue are transferred to the new domain manager XYZ. Moreover, the local batchman process is also notified to update the local Symphony file using the Intercom.msg queue.
6. The batchman and jobman process on the old domain manager are restarted.
The next time the plan is extended, the domain acts as though another switchmgr command was run and the old domain manager automatically resumes domain management responsibilities. If the old domain manager will not be available by the next plan extension, additional actions must be executed. The following steps are of a long term switch:
1. The definition of the old domain manager must be altered in the database. Its role should change from MANAGER to FTA.
2. The definition of the new domain manager must be altered in the database also. Its role needs to change from ‘FTA’ to ‘MANAGER’.
If the affected domain is the top-level domain, the following additional steps are required to rectify the plan extension:
1. The old FINAL job stream needs to be cancelled.
2. The definitions of jobs contained in the FINAL job stream need to be linked to the new domain manager.
3. The updated FINAL job stream needs to be submitted to the plan.
This procedure is quite cumbersome and can be optimized. See 24.4, “Virtualization” on page 741 for a detailed discussion.
|
Important: The network is not restarted during a switchmgr operation.
|
24.3.5 Switch Event Processor
Any workstation in the top-level domain that is running an embedded WebSphere Application Server instance is eligible for hosting the Event Processor. If a problem occurs, the Event Processor’s function can be easily migrated by running the conman switchevtproc command. The mechanism consists of the following steps:
1. The conman switchevtproc XYZ command is issued on workstation ABC in the network. XYZ is the name of the target FTA that will host the Event Processor. DEF is the name of the workstation that is currently hosting the Event Processor.
2. FTA ABC connects to the netman process on workstation XYZ, requesting it to become the Event Processor.
3. Subsequently, FTA XYZ connects to the netman process on workstation DEF, requesting it to release its role as the Event Processor.
4. FTA ABC puts a special message in the local Mailbox.msg message queue, indicating that workstation XYZ has become the new Event Processor.
5. The message is delivered to the upper domain manager and broadcasted on the entire network.
6. On each workstation in the network, the mailman process retrieves the event in the Mailbox.msg and asks batchman through the Intercom.msg queue to update the local Symphony file. Moreover, the mailman process will also put a message in the local Moncmd.msg message queue, which is examined by the monman process.
7. The monman process picks up the message in the Moncmd.msg queue and updates the location of the Event Processor in the configuration files.
Figure 24-15 presents a schematic overview.
Figure 24-15 Switch Event Processor
|
Tip: Before running the conman switchevtproc command, deploy the event rules on the old Event Processor by using the planman deploy command. As a result, the latest changes will be deployed before the Event Processor is switched.
|
24.4 Virtualization
After the product is installed, the composer add Sfinal command adds the FINAL stream and associated jobs to the database. These objects are linked to the master domain manager workstation. In case the master is running on UNIX, the FINAL stream can be virtualized. As a result, the long-term switch operation can be simplified significantly.
24.4.1 The $MASTER keyword and unixlocl method
When defining an extended agent, the $MASTER keyword can be used to indicate that the agent’s host workstation is the master domain manager. If the role of the master is migrated to another workstation, the $MASTER keyword automatically sees the new master domain manager.
Instead of defining multiple FTAs on the same UNIX node, the unixlocl extended agent access method can be used to virtualize workload. Its use implies reduced network traffic, improved network initialization and less CPU consumption of the Tivoli Workload Scheduler processes on the node, as shown in Figure 24-16.
Figure 24-16 unixlocl access method
When launching a job on a unixlocl extended agent, the access method starts by running the standard configuration script (.jobmanrc). If the job’s logon user is allowed to use a local configuration script (.jobmanrc),it is also run. Finally, the job is executed.
24.4.2 The mm resolve master
The localopts tunable mm resolve master controls if the $MASTER keyword is expanded to the real value of the master domain manager by the mailman process. When set to ‘yes’, the $MASTER keyword is resolved when the new plan is distributed. As a result, any extended agent that is linked to $MASTER switches to the new master domain manager when the network is restarted. When set to no, the keyword is not translated, resulting in an immediate switch for extended agents linked to $MASTER, triggered by a switchmgr operation.
Be careful when using $MASTER because it has a limitation. However, for this limitation, an enhancement request has been created. When the network is restarted during the plan extension, the jobman process will look at the jobtable to examine which jobs are still running. Only jobs that were started on FTAs and standard agents are currently taken into account. If a job was started on an extended agent linked to $MASTER prior to a switchmgr operation, the job will be marked ABEND immediately after the plan extension even though it is still running smoothly on the old master domain manager. Enhancement request MR011007250 was created to address this limitation.
24.4.3 FINAL stream
Using the $MASTER keyword and the unixlocl access method, the standard FINAL stream can now be modified as follows:
1. Define an extended agent linked to $MASTER using the unixlocl access method, as shown in Example 24-3.
Example 24-3 Virtual Master
CPUNAME XA_VMDM
OS OTHER
NODE NotApplicable TCPADDR 31111
FOR MAESTRO HOST $MASTER ACCESS "unixlocl"
TYPE X-AGENT
AUTOLINK OFF
BEHINDFIREWALL OFF
FULLSTATUS OFF
END
2. Link the FINAL jobs CREATEPOSTREPORTS, MAKEPLAN, STARTAPPSERVER, SWITCHPLAN and UPDATESTATS to the virtual master, as shown in Example 24-4.
Example 24-4 FINAL jobs
XA_VMDM#CREATEPOSTREPORTS
SCRIPTNAME "/opt/IBM/TWA/TWS/CreatePostReports"
STREAMLOGON twsusr
TASKTYPE UNIX
RECOVERY CONTINUE
XA_VMDM#MAKEPLAN
SCRIPTNAME "/opt/IBM/TWA/TWS/MakePlan"
STREAMLOGON twsusr
TASKTYPE UNIX
RCCONDSUCC "(RC=0) OR (RC=4)"
RECOVERY STOP
XA_VMDM#STARTAPPSERVER
SCRIPTNAME "/opt/IBM/TWA/TWS/../wastools/startWas.sh"
STREAMLOGON twsusr
TASKTYPE UNIX
RECOVERY CONTINUE
XA_VMDM#SWITCHPLAN
SCRIPTNAME "/opt/IBM/TWA/TWS/SwitchPlan"
STREAMLOGON twsusr
TASKTYPE UNIX
RECOVERY STOP
XA_VMDM#UPDATESTATS
SCRIPTNAME "/opt/IBM/TWA/TWS/UpdateStats"
STREAMLOGON twsusr
TASKTYPE UNIX
RECOVERY CONTINUE
3. Modify the FINAL stream as outlined in Example 24-5.
Example 24-5 FINAL stream
SCHEDULE XA_VMDM#FINAL
DESCRIPTION "FINAL stream"
ON RUNCYCLE RULE1 "FREQ=DAILY"
AT XXXX
CARRYFORWARD
:
XA_VMDM#STARTAPPSERVER
XA_VMDM#MAKEPLAN
FOLLOWS STARTAPPSERVER
XA_VMDM#SWITCHPLAN
FOLLOWS MAKEPLAN
XA_VMDM#CREATEPOSTREPORTS
FOLLOWS SWITCHPLAN
XA_VMDM#UPDATESTATS
FOLLOWS SWITCHPLAN
END
..................Content has been hidden....................
You can't read the all page of ebook, please click here login for view all page.