Resource monitoring and performance tuning

One of the most important steps in a load testing exercise is resource monitoring and performance tuning. In Chapter 1, Architectural Best Practices, we looked at the reference architecture of a Liferay-Portal-based solution. In the reference architecture, we have used different components to build a high performing portal solution. Performance of the solution depends upon each of the components of reference architecture. Hence, during a load test, monitoring the performance of every component is required. In this section, we will talk about resource monitoring of various components. We will also learn about how to read resource monitoring data and tune the system.

Liferay Portal server – monitoring and tuning

As we know, Liferay Portal runs on an application server. In our reference architecture, we used Tomcat as the application server. There are many resources of Liferay Portal, such as JVM, thread pool, or cache engine, which can affect the overall performance of the system. It is required to closely monitor these resources during a load test to optimize the performance. Before we proceed with individual resource monitoring and tuning, let's learn about some of the key monitoring tools and how to configure them with Liferay Portal.

JConsole

JConsole is a GUI-based tool for monitoring applications launched using JVM. It is a Java Management Extension (JMX) compliant tool. It can be used to monitor JVM Heap, CPU usage, garbage collection, threads, and JMX-enabled beans. It is a very lightweight tool and adds a minor overhead on the running application. The JConsole utility comes as a part of Oracle JDK installation. It can connect to any remote or local Java-based applications. In order to connect JConsole with a remote application, the JMX port on the remote JVM has to be configured. Let's learn how to configure our Liferay Portal nodes to enable JConsole-based monitoring:

  1. Stop both the Liferay Portal nodes if they are running.
  2. Now add the following environment variable in the node-01liferay-portal-6.1.1-ce-ga2 tomcat-7.0.27insetEnv.bat file of liferay-node-01:
    set CATALINA_OPTS=-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false

    Note

    If the Liferay Portal server is deployed on a Linux- or Unix-based environment, the same changes need to be done in the setEnv.sh file.

  3. Apply the same changes as mentioned in step 2 on liferay-node-02.
  4. Restart the two Liferay Portal nodes.
  5. Now from the command prompt run the jconsole command from the <JAVA_HOME>in directory.
  6. The system will show a connection dialog box, as shown in the following screenshot:
    JConsole
  7. Enter the values in the Remote Process field as shown in the preceding screenshot and then press Connect. The system will open the Overview tab.

In the startup configuration file of the Liferay Portal Tomcat server, we enabled JMX-based monitoring and configured the JMX port. We will talk about how to read and analyze monitoring output in the following sections.

VisualVM

VisualVM is an open source resource monitoring tool for Java. It is actually considered as the next generation of monitoring tools as compared to the lightweight JConsole. VisualVM is included as part of the Oracle JDK installation. It is implemented using the plugin-based architecture, hence, it allows additional plugins for resource monitoring. JConsole can also be used as one of the plugins of VisualVM. VisualVM also includes profiling capabilities. VisualVM allows taking snapshots of monitoring data at any time. This enables us to compare application states at certain events. Similar to JConsole, VisualVM also allows us to connect to local as well as remote Java applications. To connect VisualVM with a remote Liferay Portal Tomcat server, we will need to enable the JMX port. We learned how to enable the JMX port in the previous section. Let's learn how to connect VisualVM with the Liferay Portal Tomcat server.

  1. From the command prompt run the following command to start VisualVM:
    <JDK_HOME>/bin/jvisualvm
  2. The system will open the VisualVM application. Now click on File | Add JMX Connection…. The system will open a dialog box as shown in the following screenshot:
    VisualVM
  3. Enter the value in the Connection test field as shown in the screenshot and then click on the OK button.
  4. Now from the Applications section, double-click on Local | localhost:9999.
  5. On the right tab view, the system will add a new tab and open the Overview subtab.

JVM – monitoring and tuning

We learned how to configure the Liferay Portal server with JConsole and VisualVM. Both of these tools can be used to monitor Liferay Portal JVM. In Chapter 3, Configuration Best Practices, we learned how to configure Liferay Portal JVM parameters. We also learned the recommended JVM parameters for the Liferay Portal server. In most cases, the recommended parameters will work well. But there is scope to optimize them according to the developed solution during load testing. The most important areas that need close monitoring during a load test are heap memory and garbage collection. JConsole's Memory tab (as shown in the following screenshot) allows us to closely monitor the heap memory and garbage collection activities of the Liferay Portal server:

JVM – monitoring and tuning

JVM divides heap memory into multiple regions or memory pools. Objects are moved from one pool to the other after garbage collection. Before we talk about what should be monitored on JConsole's Memory tab, let's briefly understand heap memory pools:

  • The Eden space: When an object is created, it will occupy memory space in this pool.
  • The Survivor space: When the objects that are stored in the Eden space survive at least one garbage collection, they are moved to the Survivor space.
  • The Tenured space (old generation): When the objects stay in the Survivor space for some time they are then moved to the Tenured Space.
  • Permanent generation space: This space mainly stores the reflective data of JVM such as class objects. It is also considered as non-heap memory.
  • Code cache: When HotSpot JVM is used, the Just-in-time (JIT) compiler stores the compiled classes in this memory pool. It is also considered as non-heap memory.

Objects move from Eden to Survivor and Survivor to Tenured space by garbage collection until they are cleared from the memory. When the garbage collector cleans up objects from the Tenured space, it is called major garbage collection (major GC). Major garbage collection consumes more resources and also pauses other threads for some time. Frequent major GCs will affect the overall performance of the system. When to go in for major GC or minor GC depends upon the size of these pools. Hence, the size of these memory pools needs to be monitored and optimized during a load test.

As shown in the preceding screenshot, JConsole by default shows a line chart representing the heap memory usage over time. In the bottom-right corner of the Memory tab, it shows the memory usage by different memory pools. We can monitor memory usage of individual memory pools by selecting the respective memory pool from the Chart drop-down box.

In the Details section of the Memory tab, JConsole displays the following memory matrices:

  • Used: This field displayed the memory occupied by all the objects in heap.
  • Committed: This field displays memory that is occupied by JVM irrespective of whether it is used by objects or not. The JVM may release—time to time—unoccupied memory to the system. The value of this field will be less than or equal to the maximum heap size allocated using the JVM parameter.
  • Max: This field shows the maximum memory that can be used for memory management.
  • GC Time: This field shows the total GC time spent by the garbage collector. It shows the time for which other threads were stopped to perform GC.

During monitoring, if we find that the Survivor space is full most of the time, it indicates the size of the young generation (Eden plus Survivor space) is not enough. Because of that it will move more objects into the old generation space. This will increase the possibility of a major GC. Hence, the young generation size should be tuned. The following JVM parameters help in the configuration of the young generation:

  • NewRatio: This parameter controls the size of the young generation. If the value of this parameter is four, the size of the young generation space (Eden plus Survivor) will be one fourth of the total heap. The rest of the space will be used by the Tenured space.
  • NewSize: This parameter defines the minimum size of the young generation space.
  • MaxNewSize: This parameter defines the upper limit of the young generation size.
  • SurvivorRatio: This parameter defines the ratio of the Survivor and Eden space. If it is configured to six, the Survivor space will be one sixth of the total young generation space.

During a load test, if the trend of the heap memory chart is going upward throughout the load test, it indicates the possibility of a memory leak in the application. In order to conclude this we should run the load test for a longer duration.

During the load test, if it is found that garbage collection is happening again and again, the heap size needs to be tuned. It may be because the young generation is not configured properly or the total heap size is not configured correctly.

During the load test, if it is found that the Permanent generation size is reaching near the maximum Permanent generation size, it is recommended to increase the Permanent generation space.

Note

JVM tuning in itself is a vast topic. We learned a few of the important JVM tuning options. For more information about garbage collection tuning, please refer to the following URL:

http://www.kgs.ku.edu/Publications/ancient/f15_snails.html

Tomcat thread – monitoring and tuning

In Chapter 3, Configuration Best Practices, we learned to configure the Liferay Portal-Tomcat server with the recommended thread pool configuration. We configured the maximum and minimum size thread pool. The recommended thread pool configuration works in most of the cases, but depending upon the concurrent user requirements, the thread pool configuration should be tuned. In order to fine-tune the thread pool configuration, we will need to monitor the thread pool during load tests. The Tomcat server exposes thread pool statistics using JMX MBeans. JConsole supports monitoring JMX MBeans. Let's learn how to monitor a Tomcat thread pool:

  1. Open JConsole and connect JConsole with the respective Liferay Portal node.
  2. In JConsole, navigate to the MBeans tab.
  3. In the MBeans tab navigate to Catalina | Thread Pool in the tree.
  4. In the Thread Pool node, you will find AJP and/or HTTP connector nodes. Expand the respective connector and then click on the Attributes subnode. The system will show thread pool attributes as shown in the following screenshot:
    Tomcat thread – monitoring and tuning

As shown in the preceding screenshot, we can get current values of various thread pool attributes. We can use the Refresh button to refresh the values. We can keep monitoring the values of thread pool attributes to find out any issues. In order to fine-tune thread pool sizing, the following attributes should be closely monitored:

  • currentThreadCount: This attribute tell us how many threads are created by the Liferay Portal-Tomcat server. It includes both busy threads and idle threads.
  • currentThreadsBusy: This attribute tell us how many threads are busy in serving requests.
  • maxThreads: This attribute tells us how many maximum threads are configured.

During the load test, if it is found that the value of the currentThreadsBusy attribute is nearing the maxThreads value, it indicates some issue with the thread pool configuration. The issue could be with the maxThreads value. In such a situation, the maxThreads value should be increased. If the same issue persists even after increasing the maxThreads value, further analysis of threads should be done. This can be done by taking the thread dump. We can take the thread dump of the Liferay Portal server through the control panel. We can also monitor individual threads using JConsole. The Threads tab of JConsole provides a way to monitor all the threads as shown:

Tomcat thread – monitoring and tuning

Here, JConsole displays a line chart representing the number of threads over time. It considers all the threads of the Tomcat server and not only thread pool threads;

it also allows the reviewing of the stack trace by individual threads.

During a load test, if it is found that CPU usage of Liferay Portal is very high all the time, one of the reasons could be because of a thread deadlock. Using JConsole we can detect such thread locks. As shown in the preceding screenshot, at any time during the test we can click on the Detect Deadlock button to find any thread deadlocks.

During a load test, if it is found that the value of the currentThreadsBusy attribute is always very low compare to the value of the maxThreads attribute, it indicates that the thread pool might be oversized. We can reduce the thread pool size by modifying the value of the maxThreads attribute.

Database connection pool – monitoring and tuning

A database connection pool impacts a lot on the overall performance of the Liferay Portal server. In Chapter 3, Configuration Best Practices, we learned how to configure a JNDI-based database connection pool. We learned about the recommended database pool configuration. For most of the Liferay Portal solutions, this configuration works well. But there is always scope for improvement, so it is recommended to monitor the database connection pool during a load test. Database connection pool statistics are exposed using JMX MBeans by the Liferay Portal-Tomcat server. Here are the steps to monitor a database connection pool using JConsole:

  1. Open JConsole and connect JConsole with the respective Liferay Portal node.
  2. In JConsole, navigate to the MBeans tab and then navigate to the com.mchange.v2.c3p0 node in the MBeans tree.
  3. From this node, select a subnode starting with the name PooledDataSource.
  4. Then, select the Attributes subnode. The system will display the database connection pool attributes as shown in the following screenshot. Keep refreshing the values of the attribute by using the Refresh button at the bottom.
Database connection pool – monitoring and tuning

There are many attributes exposed by the database connection pool MBean. The following list outlines the important attributes that should be tracked during a load test:

  • numBusyConnections: This attribute tells us how many database connections are in use by the Liferay Portal server.
  • maxPoolSize: This attribute tells us the maximum number of connections that can be created in a database connection pool. This attribute is static and the value of this attribute is controlled by the database connection pool configuration.
  • numConnections: This attribute tells us how many connections are created in the database connection pool. It includes both busy and idle connections.

During the load test, if it is found that the value of the numBusyConnections attribute is always nearing the value of the maxPoolSize attribute, it indicates an issue with the database connection pool. It could be because of the undersized database connection pool. If the same issue persists even after increasing the database connection pool size then the issue might be because of open database connections or slow query executions.

Similarly, if the value of the numBusyConnections attribute is very low as compared to the value of the maxPoolSize attribute, it indicates that the database connection pool is oversized.

Cache – monitoring and tuning

In Chapter 4, Caching Best Practices, we spoke about how to provide custom configuration for default Ehcache-based caching. Liferay creates many cache instances for both multi-VM and Hibernate cache managers. Liferay includes the default cache configuration for each cache instance. Depending upon the Portal, requirement-specific cache instances can be tuned to improve performance. Liferay Portal exposes cache information using JMX MBeans. Liferay Portal exposes two types of cache information: one is related to actual cache objects and the other one is for overall cache statistics. During load testing, cache statistics should be monitored. By default, cache statistics are not exposed via JMX MBeans. It is required to enable cache statistics using the Liferay Portal configuration. Here are the steps to enable cache statistics in our setup:

  1. Stop both Liferay Portal nodes if they are already running.
  2. Now add the following properties in portal-ext.properties of the two nodes:
    #
# To enable cache statistics for Single VM, Multi VM
# Cache Managers
#
ehcache.statistics.enabled=true
#
# To enable cache statistics for Hibernate cache manager
#
hibernate.generate_statistics=true
  3. Restart both the Liferay portal nodes.

We enabled cache statistics for both Liferay Portal's cache manager and Hibernate's cache manager. Once cache statistics are enabled, we can monitor cache statistics using JConsole. Here are the steps to monitor the cache using JConsole:

  1. Open JConsole and connect JConsole with the Liferay Portal node.
  2. In JConsole, navigate to the MBeans tab and then expand net.sf.ehcache | CacheStatistics | liferay-multi-vm-clustered.

    Under the liferay-multi-vm-clustered tree node, we can find all the cache instances as subnodes. Click on the Attributes subnode of the specific cache. The system will display cache statistics of the selected cache instance as shown in the following screenshot:

    Cache – monitoring and tuning

Cache statistics provide information of various attributes, but here are the key attributes that should be monitored during a load test:

  • ObjectCount: This attribute tells us how many objects there are in the cache.
  • OnDiskHits: This attribute tells us how many requests are successful in locating objects from a filesystem-based cache. This attribute is useful if we enabled overflow to the disk attribute of the cache instance.
  • InMemoryHits: This attribute tells us how many requests are successful in retrieving objects from an in-memory cache.
  • CacheHits: This attribute tells us how many requests are successful in retrieving objects from the cache. It includes both in-memory and disk-based caches.
  • CacheMisses: This attribute tells us how many requests are unsuccessful in retrieving objects from the cache.

We learned that there are many cache instances in Liferay Portal. It is not possible to monitor every instance during load tests. Hence, cache monitoring is done based on specific performance issues. Depending on the most used functionalities of the Portal, cache instances should be identified and monitoring those cache instances should be done during a load test.

During a load test, if it is found that the value of the CacheMisses attribute is very high, it indicates that the cache is undersized. In that case, the size of the cache should be increased.

Apache web server – monitoring and tuning

In our reference architecture, we have used an Apache web server in front of Liferay Portal servers. During a load test, we need to monitor the following resources of the Apache web server.

  • CPU and memory
  • Worker threads

There are many tools available in the market to monitor CPU and memory consumption of the Apache web server. On a Linux- or Unix-based server, we can simply use the TOP command to monitor CPU and memory consumption. With this option, it will be difficult to monitor resource usage over time. For this kind of monitoring, any SNMP-based monitoring tool can be used. Nagios is one of the most powerful open source monitoring tools. The Apache web server can be configured with Nagios to monitor CPU and memory consumption. With the use of Nagios, we can monitor worker threads as well.

The Apache web server also comes up with a simple monitoring module called mod_status. It can be used to monitor worker threads. The following are the steps to enable this tool:

  1. Locate the httpd.conf file in the <APACHE_HOME>/conf directory on the Apache web server and add the following configuration into it:
    LoadModule status_module modules/mod_status.so
ExtendedStatus On
<location /server-status>
  SetHandler server-status
  Order allow,deny
  Allow from all
</location>
  2. Restart the Apache web server.
  3. From the browser, access http://localhost/server-status.

We enabled the mod_status module and configured the server's status page. The server status page provides the following monitoring statistics:

  • The number of on-going requests
  • The number of idle worker threads
  • Process details
  • The total access requests
  • The number of requests served per second

During a load test, if it is found that most of the time there are no idle workers and requests are not going through, it is recommended to resize Apache threads / max client configuration. If the problem still persists, requests might not be processed because the Liferay Portal server is taking more time to respond. If the memory usage is consistently high, it is recommended to reduce the Apache thread / max client pool size.

Monitoring the database server

Liferay Portal is database agnostic. We can configure any JDBC-supported database server with the Liferay Portal server. In our reference architecture, we have used MySQL. Most of the database products provide their own monitoring and tuning tools. In this section we will discuss which items should be monitored during load tests.

CPU and memory usage

CPU and memory usage of the database server must be monitored during a load test to find any performance bottlenecks. As discussed in the previous section, the easiest way to monitor CPU and memory is through the TOP command. But it is recommended to configure SNMP-based tools such as Nagios for CPU and memory usage monitoring. There could be multiple reasons of high CPU or memory usage. After the load test, further investigation will be required to find out the root cause of this.

Slow queries

It is very important to identify database queries that are taking more time. It is also important to find out database queries that are executed many times during the load test. Every database product provides one way or an other to get slow queries or the top n queries.

There are multiple reasons for queries being slow. It could be because of improper indexing, improper query logic, or improper database configuration parameters. During a load test, a list of such queries should be identified, and then before the next load test for run, the necessary performance-related changes should be carried out.

Connections

We learned to monitor and tune the database connection pool in the Database connection pool – monitoring and tuning section. But that is one side of it. Performance issues may arise because of improper connection configuration at the database server level as well. So it is very important to monitor connections at the database server level. Every database server provides one way or another to monitor a number of open and idle connection objects in the database. During the load test, these statistics must be closely monitored.

Lock monitoring

Database servers use the locking mechanism to support concurrent access. Sometimes heavy database queries lock database objects for a long time. It will slow down the processing of other requests that are dependent on the same objects. It could be one of the causes of a high number of busy connections on the Liferay Portal server. Most of the database products provide lock-monitoring features. During a load test, database locks should be closely monitored.

Monitoring logfiles

We talked about monitoring various resources using tools. But, sometimes, performance bottlenecks are because of errors in some of the components. Hence, as part of the load testing process, all the logfiles should be monitored after the load test. It is recommended to clear all logfiles before starting the load test. Here is a list of logfiles that should be monitored:

  • Liferay Portal logfile
  • Application server logfile
  • Apache web server access logfile
  • Apache web server error logfile
  • Apache web server mod_jk logfile
  • Database server error logfile
  • Error logfiles of every application-specific integration components
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.148.144.228