The pattern engine
This chapter describes the various components of the pattern engine. It provides examples of plug-ins, and of the Maestro Framework elements used in pattern transformation. This chapter also provides an in-depth overview of transformers and their purposes, and a high-level overview of the steps of a transform. The following topics are covered in this chapter:
2.1 Pattern transformation
The various components of the Maestro Framework choreograph and metamorphose the pattern from the application model into a topology that the deployment engine can understand. This is used to implement the actual virtual machines (VMs) and middleware components of the virtual application into the cloud. This topology also controls the various options and features (scaling, load balancing, and so on) while running in the cloud.
When using the Virtual Application Builder (VAB) WebUI, the GUI cannot always actually reflect the complexity of what it takes to instantiate and implement the end product. Figure 2-1 shows the JavaScript Object Notation file (appmodel.json) of an IBM Cognos® Business Intelligence Pattern.
Figure 2-1 Cognos Virtual Application topology as viewed in the VAB WebUI
Example 2-1 is the source view of the appmodel.json file as seen from the VAB. In the VAB, click the Source tab to get this view of your pattern. This code is what the pattern engine transforms into topology data to be employed for the deployment and use of the virtual application lifecycle.
Example 2-1 Cognos Business Intelligence sample appmodel.json
{
"model": {
"name": "Cognos BI Sample",
"nodes": [
{
"attributes": {
"DEFAULT_THROUGHPUT_TYPE": "LOW",
"ADMIN_CREDENTIALS_NAMESPACE": "NONE",
"ADMIN_CREDENTIALS_USERNAME": "virtuser",
"ADMIN_CREDENTIALS_PASSWORD": "<xor>Lz4sLCgwLTs=",
"PROXY_HOST_ALIAS": "SAMPLE",
"VALID_DOMAINS_AND_OR_HOSTS": "*"
},
"id": "Business Intelligence",
"type": "cognosbi",
"groups": {
"DATASOURCE_CONNECTORS": true,
"DEFAULT": true,
"ADVANCED": false,
"DB2": true,
"Notification": true,
"ADMIN_CREDENTIALS": true,
"Oracle": false,
"Microsoft": false
}
},
{
"attributes": {
"LDAP_USER_LOOKUP": "(uid=${userID})"
},
"id": "IBM Tivoli Directory Server",
"type": "xTDS",
"groups": {
}
},
{
"attributes": {
"intialInstanceNumber": 2
},
"id": "Scaling Policy",
"type": "ScalingPolicyofCognosBI",
"groups": {
"None": true,
"Basic": false
}
}
],
"links": [
{
"source": "Business Intelligence",
"target": "IBM Tivoli Directory Server",
"attributes": {
},
"type": "COGNOSBIxUR",
"id": "COGNOSBIxUR_1",
"groups": {
}
},
{
"source": "Scaling Policy",
"target": "Business Intelligence",
"attributes": {
},
"type": "policy.ScalingPolicyofCognosBI",
"id": "policy.ScalingPolicyofCognosBI_1",
"groups": {
}
}
],
"description": "Application template for IBM Business Intelligence Pattern using a scaling policy and an existing IBM Tivoli Directory Server user registry (LDAP namespace)",
"app_type": "application",
"locked": false,
"patterntype": "ibmbi",
"version": "1.0"
},
"layers": [
{
"id": "layer",
"nodes": [
"Business Intelligence",
"IBM Tivoli Directory Server",
"Scaling Policy"
]
}
]
}
 
By comparing Figure 2-1 on page 18 and Example 2-1 on page 18, you can see that they actually describe the same pattern from two different views. The VAB view provides a simplified method of interaction, rather than having to edit and manipulate the code in the appmodel.json file.
In short, although the pattern looks simple, it is usually not. There are many things that the pattern engine handles to implement a successful deployment and utilization of a virtual application pattern (VAP).
High-level overview of a virtual application transformation
These are the steps you follow to implement a virtual application transformation:
1. Pre-deployment
Before the deployment of a virtual application, the plug-ins to be used in its pattern type must be defined in the config.json file contained in the plug-in archive. See 2.2, “Basic plug-in overview” on page 21 for information about the actual structure and layout of a plug-in, and where the config.json file is located.
The various middleware packages, hardware, plug-in ID, and the pattern type for the plug-in to be associated with are all defined in this file.
2. Deployment
After you have saved the pattern configuration that you want in the VAB, that data is stored as an application model in the appmodel.json file. This file is stored in the Storehouse of the management module of PureApplication System.
The application model is composed of components, links, and policies. This application model goes though several iterations and progressive steps to achieve deployment:
a. Convert the application model into an unresolved topology. Do not include host names or identifying information, just the generalities such as disk space, what middleware to install, how much RAM and CPU, and so on.
In this step, Component and Link transforms use Java or the Apache Velocity files <template_name>.vm to set these values.
b. Take the unresolved topology and use the definitions in the config.json file to help transform that unresolved topology into a resolved topology. The appmodel is used to map key components to the parts provided by the plug-ins. The packages (binary files) are defined in the config.json file.
c. After the resolved topology document has been created, the pattern engine uses the resolved topology document, reserves and provisions the required resources, and writes the final topology document to use for deployment of the actual VMs that comprise the virtual application.
d. Deploy the VMs into a cloud using the final topology document.
e. Conduct the middleware and software installation and configuration, and then start the software on the deployed VMs. These actions are completed using a set of lifecycle management scripts that are defined for each component, such as the start.py, install.py, and configure.py Python scripts.
Figure 2-2 illustrates the steps described to realize a virtual application from beginning to end.
Figure 2-2 Flowchart diagram of the deployment process of a virtual application
2.2 Basic plug-in overview
Plug-ins are specifically structured archives of files that offer components, links, and policies to the pattern engine and operations for later lifecycle management of the virtual application. The pattern engine represents these in the virtual application WebUI as units to be added, deleted, or modified by dragging to create VAPs. The plug-ins define various attributes of the deployment of the actual VMs comprising the pattern and other virtual application lifecycle events.
Here are the various parts that can be included in a plug-in:
parts/<plug-in_id>.tgz This compressed tape archive (TAR) file (.tgz) describes the artifacts to be installed by the workload agent. These artifacts and files are used to communicate with the PureApplication System console about how the virtual application will be managed on the actual VMs.
nodeparts/<plug-in_id>.tgz Contains artifacts to be installed by the activation script. The workload agent is itself a node part.
appmodel/tweak.json Contains code for changing a virtual application in the VAB.
appmodel/metadata.json Contains the specific links, components, and policies that the plug-in shows to the user in the VAB.
appmodel/operation.json Contains code for changing a virtual application in the VAB.
bundles/<plug-in_id>.tgz Specifies the Java archive (.jar) file that contains the transformers, scanners, and provisioners of the plug-in.
config.json Contains the plug-in configuration file information.
 
Important: The config.json file is the only required file for a plug-in. The other files are used as needed.
Plug-ins are covered in more detail in Chapter 4, “Plug-in Development Kit” on page 45. This chapter includes steps for plug-in development, and for virtual application lifecycle scripting.
2.3 Transforms
Transformers are kernel services that convert the virtual application model and its logical descriptions into the topology document that the pattern engine can use for the deployment of the VMs in the virtual application. The actual process is referred to as a transformation.
The entire topology document is a JavaScript Object Notation (JSON) object document. The topology fragments are JSON objects as well.
Transforms are multi-part operations, and there are basically two types:
•Template-based transforms
•Java implementations
Template-based transforms
The most common transform is implemented as a template of a JSON document. Dependent objects are used for links, and topology fragments are used for components. Both the PureApplication System W1500 and W1700 series embed Apache Velocity 1.7 as their template engine. The user guide for Apache Velocity is located at the following website:
You can also see the Javadoc in the Plug-in Development Kit (PDK), which can be downloaded from the following website:
There is a javadoc.zip compressed file inside the pdk.zip.
Each plug-in has a specific file structure in which information about the plug-in’s deployment and lifecycle can be recorded and referenced by policies, links, or components. The component document’s name must match the ID of the component, policy, and link, as defined in the appmodel/metadata.json file of the plug-in.
Specified component properties in template files use path values relative to the plug-in root, as illustrated in Example 2-2.
Example 2-2 Name value and the relative path values
<?xml version="1.0" encoding="UTF-8"?>
<scr:component xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0" name="starget">
<implementation class="com.ibm.maestro.model.transform.template.TemplateTransformer"/>
<service>
<provide interface="com.ibm.maestro.model.transform.TopologyProvider"/>
</service>
<property name="component.template" type="String"
value="templates/starget_component.vm"/>
<property name="link.template" type="String"
value="templates/starget_link.vm"/>
</src:component>
The pattern engine sets the data in an Apache Velocity context to enable the use of variables that can be referenced. Continuing with the <starget> example, you can see the use of variables in the starget_component.vm file. The $attributes are defined in the appmodel/metadata.json file of the plug-in, as illustrated in Example 2-3.
Example 2-3 The appmodel/metadata.json file of the plug-in
{
"vm-templates": [
{
"scaling":{
"min": 1,
"max": 1,
},
"name": "${prefix}-starget",
"roles": [
{
'parms': {
"st1": "$attributes.st1"
},
"type": "starget",
"name": "starget"
}
]
}
]
}
Link transformations use a depends array of the source role to store the generated JSON data (see Example 2-4).
Example 2-4 The sample usage of a link transformer
## Link templates render the depends objects to be added to the source role.
 
## sourceRole is required to locate the source of the link. Value is the type of the source role.
#set( $sourceRole = "ssource" )
 
## sourcePackages is an optional array. Values in the array are added to the packages of the
## vm-template that is hosting the source role.
#set( $sourcePackages = ["pkg2"] )
 
## Obtain a tuple related to the matching target role:
## target.template == vm-template that holds the target role; target.role == role
## String argument is the type of the target role.
#set( $target = $provider.getMatchedRole($targetFragment, "starget") )
 
## Validate target. If not found, throw HttpException
#if( $target == $null )
$provider.throwHttpException("Target Role starget not found.")
#end
 
[
{
"role": "${target.template.name}.${target.role.name}",
"type": "starget",
"parms": {
"sl1": "$attributes.sl1"
}
}
]
Both the if_value and the if_else_value (for which there can be a null value) can be used to create conditional statements, as shown in Example 2-5 and in Example 2-6 on page 25.
Example 2-5 Sample of if_value used to create conditional events
{
"vm-templates": [
{
"scaling":{
"min": 1,
"max": 1
},
"name": "${prefix}-ssource",
"roles": [
{
'parms': {
## Handling optional attributes:
## macro syntax: #macro( if_value $map $key $format_str )
## String value:
#if_value( $attributes, "ss_s", '"ss_s": "$value",' )
## Number value:
#if_value( $attributes, "ss_n", '"ss_n": $value,' )
## Boolean value:
#if_value( $attributes, "ss_b", '"ss_b": $value,' )
## Missing value -- will not render:
#if_value( $attributes, "not_defined", '"not_defined": "$value",' )
 
## For artifacts, Inlet may send app model with absolute URLs for artifacts;
## other request paths might invoke with relative URLs.
## So use provider.generateArtifactPath(), which invokes URI.resolve() that handles both cases.
 
## Handling required attributes; throws an exception if the attribute is null/empty/not defined
"ss_f": "$provider.generateArtifactPath( $applicationUrl, ${attributes.ss_s} )",
 
## Handling range value (ss3)
"ss_r_min":"$attributes.ss_r.get(0)",
"ss_r_max":"$attributes.ss_r.get(1)",
## Handling policies: spolicy is defined; not_policy is not
#set( $spattrs = $provider.getPolicyAttributes($component, "spolicy") )
#if_value( $spattrs, "sp1", '"sp1": "$value",' )
#if_value( $spattrs, "not_defined", '"not_defined": "$value",' )
#set( $npattrs = $provider.getPolicyAttributes($component, "no_policy") )
#if_value( $npattrs, "np1", '"np1": "$value",' )
## Handling required config parms; throws an exception if the parm is null/empty/not defined
"cp1": "$config.cp1"
},
"type": "ssource",
"name": "ssource"
}
]
}
]
}
Example 2-6 Sample of if_else_value
"Users" : #if_else_value($attributes, "Users", $attributes.Users.serialize(), []),
# Render a formatted string if the mapped value exists and is not empty.
#macro( if_value $map $key $format )
 
# Render a formatted string if the mapped value exists and is not empty, else a different string.
#macro( if_else_value $map $key $format_if, $format_else )
Lastly, Java static classes can be used in templates, which can be useful if other static methods are used on these classes in the template. Just as in Example 2-6, the $attributes are defined in the appmodel/metadata.json file of the plug-in, as shown in Example 2-7.
Example 2-7 Java static class used in a template
#set( $Math = $provider.getClassForName("java.lang.Math") )
"ss_r_math_max":$Math.max($attributes.ss_r.get(0), $attributes.ss_r.get(1)),
 
ss_r
Java implementations
For cases where templates are not sufficient, Java implementations can be used. Java implementations can generate the JSON documents with the included JSON APIs (com.ibm.json.java.*), or by modifying templates. Another preferred option is to use templates, and to enhance them with Java functions.
Template transforms can be enhanced with Java code. Template transforms can be quite useful. If you need Java code, do as much as you can with the template first, then add Java methods that you start from your template, as shown in the following steps:
1. Create your Java class and have it extend TemplateTransformer.
2. Add your Java methods. The public methods can be started from your template by using $provider.myMethod(). You can pass parameters into the Java methods.
3. Update your Open Services Gateway Initiative (OSGi) component document to set the implementation class to your new Java class, not TemplateTransformer. There are two methods for updating the Apache Velocity context:
a. The Apache Velocity context is available in the VelocityContext method. Therefore, you can pass it into a Java method by using $context.
b. Implement the protected VelocityContext createContext(<String applicationUrl>, <JSONObject component>) method. In your method, call super.createContext(<applicationUrl>, <component>). This returns the Apache Velocity context, VelocityContext, to which you can add your custom objects.
The Java JSON APIs can be used to generate the required JSON fragments.
4. Just as in the Template implementations, the ID must match the policy, link, and component that are defined in the appmodel/metadata.json file of the plug-in (see Example 2-8).
Example 2-8 Value of name must match the ID of the links and policies of that plug-in
<?xml version="1.0" encoding="UTF-8"?>
<scr:component xmlns:scr="http://www.osgi.org/xmlns/scr/v1.1.0" name="WAR">
<implementation class="com.ibm.maestro.model.transform.was.WARTransformer"/>
<service>
<provide interface="com.ibm.maestro.model.transform.TopologyProvider"/>
</service>
</src:component>
5. To implement component and link transformations by overriding their corresponding methods, Java implementations can use the extension of com.ibm.maestro.model.transform.TopologyProvider, as shown in Example 2-9.
Example 2-9 Link and Component example of Java implementation
public JSONObject transformComponent(
String vmTemplateNamePrefix,
String applicationUrl,
JSONObject applicationComponent,
Transformer transformer)
throws Exception {
return new JSONObject();
}
 
public void transformLink(
JSONObject sourceFragment,
JSONObject targetFragment,
String applicationUrl,
JSONObject applicationLink,
Transformer transformer)
throws Exception {
}
6. You can invoke templates by using these methods of the TopologyProvider class, as shown in Example 2-10 on page 27.
Example 2-10 TopologyProvider method invocation
public static JSONArtifact renderTemplateToJSON(Bundle b, String template, String logTag, Context context) throws HttpException;
 
public static String renderTemplate(Bundle b, String template, String logTag, Context context) throws HttpException;
7. Example 2-11 illustrates how you can use the WebSphere Application Server in a transform from a template to generate the topology fragment for the server instance.
Example 2-11 WebSphere Application Server TopologyProvider method invocation
protected void activate(ComponentContext context){
_bundle = context.getBundleContext().getBundle();
}
 
@Override
public JSONObject transformComponent(String prefix, String applicationUrl, JSONObject component, Transformer transformer) throws Exception {
JSONObject topology;
JSONObject scalingPolicy = getPolicy(component, "ScalingPolicyofWAS");
String vmTemplateName = prefix + "-was";
 
if (scalingPolicy == null) {
VelocityContext context = new VelocityContext();
context.put(TemplateTransformer.PREFIX, prefix);
context.put(TemplateTransformer.APPLICATION_URL, applicationUrl); // Value ends with a slash.
context.put(TemplateTransformer.COMPONENT, component);
JSONObject attributes = (JSONObject) component.get("attributes");
context.put(TemplateTransformer.ATTRIBUTES, new RequiredMap(attributes));
context.put(TemplateTransformer.CONFIG, new RequiredMap(getConfigParms()));
context.put(TemplateTransformer.PROVIDER, this);
String logTag = "WAS:templates/SingleWAS.vm";
topology = (JSONObject) renderTemplateToJSON(_bundle, "templates/SingleWAS.vm", logTag, context);
2.4 Methods for use in virtual application lifecycle management
Plug-ins are also used to define other aspects of the deployment and virtual application lifecycle. Storage space can be defined as a template. The template can even be named and referenced later in the topology document, with more specific attributes in a different array (see Example 2-12). The parameters for the size of the storage, the type of file system, and the type of mounting are all described for the additional storage in a vm-template.
Example 2-12 The db2-storage id is assigned to the storage template
{
"vm-templates":[…],
"storage-templates":[
{
"parms":{
"size":4,
"format":"ext3",
"type":"auto"
},
"name":"db2-storage"
}
]
}
Using these extensibility features, you can design plug-ins for specific deployment environments, or use the existing VAPs if you are developing less-specific applications. Example 2-13 defines the specific hard disk drive (HDD) and file system location to which to attach the storage shown in Example 2-12 on page 27. The vm-template is referenced via the storage-ref tag in another array.
Example 2-13 The storage-ref tag references the db2-storage template to define file system attributes
"storage":[
{
"storage-ref":"db2-storage",
"device":"/dev/sdb",
"mount-point":"/home/db2inst1"
}
]
The pattern engine also provides limited support for OSGi services inside plug-ins. The following specific PureApplication System interfaces can be implemented:
TopologyProvider Contains process components, links, and policies to create the unresolved topology document.
TopologyProcessor Updates the unresolved topology document.
RegistryProvider Provides a shared service. A replacement service for the deprecated PostProvisioner.
ServiceProvisioner Provisions a resource from an external or shared service, such as the external or shared service client.
PostProvisioner Looks at the topology document after it has been finalized and stored. The topology document is written once. A deprecated interface, PostProvisioner was replaced by RegistryProvider.
AppBindingService Scans component artifacts in the appmodel.json file. For example, the type-webapp WebSphere Application Server plug-in scans the application. It returns data, such as the data source used by the application, from a given application, such as enterprise archive (EAR), web archive (WAR), or another application not based on Java Platform, Enterprise Edition (Java EE).
 
Important: Review the Javadoc shipped with the PDK for exact details of implementing these service interfaces for that specific version of the PDK. Changes might occur from version to version.
2.4.1 Maestro module for Python
There is also a Maestro module that can be imported into Python scripts by including the import statement in the Python script, as shown in Example 2-14. After it is imported, the Maestro package and the tools can be used.
Example 2-14 Import statement
import maestro
2.4.2 Maestro global parameters
You can also include the virtual applications global parameters used by Maestro for the deployment and lifecycle of the virtual application (see Example 2-15).
Example 2-15 Global parameters
role = maestro.role
peers = maestro.peers
node = maestro.node
parms = maestro.parms
role_status = maestro.role_status
 
def isQuorum( r=role ):
return r.get('QUORUM') == 'QUORUM'
 
logger.debug('quorum leader: %s', isQuorum())
 
if role_status != 'INSTALLING' and role_status != 'CONFIGURING' and role_status != 'INITIAL'
and not isQuorum(role) and not maestro.node['template']['scaling']['max'] == 1:
for p in peers:
if isQuorum(peers[p]):
# update local SSL key
break
2.5 Pattern types
Pattern types are a collection of related plug-ins consisting of components, links, and policies for deploying virtual applications and the actions governing their lifecycle. Virtual application domains are defined by pattern types.
For example, the WebSphere Community Edition Sample Pattern Type contains plug-ins. These plug-ins are relative to deployments of simple or complicated web or Java EE applications on the WebSphere Community Server. Deployment is dependent on certain other features, such as middleware, being installed.
Different pattern types have different dependencies, and different middleware requirements that are related to those dependencies. The subject of pattern types and their relation to plug-ins, especially with regard to their structure and developmental requirements, is covered in greater detail in later chapters of this book.
 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.222.115.215