Zope 2 is an application server. Therefore, every page it delivers involves "doing work", which results in server and network load. If possible, to conserve the system resources, you may want to cache your site's content as much as is practical and possible.

Caching agents store resources needed to serve the site's content and answer site requests with these resources, until the browser or caching agent is no longer able to do so (for example, when a caching rule tells the browser or caching agent that the page has expired).

At this point, it must contact the application server again for new data.

Traditionally, the very first thing many people do to improve Plone's performance is install the CacheFu add-on product.

CacheFu has been around since the Plone 2.5.x days, and has been a welcome addition for folks trying to improve the performance of an increasingly complex and slower running software stack.

Although it is not the magic elixir that many hope it to be, it does provide a decent, measurable improvement for very little effort; so it is usually a good idea to explore.

It does this by creating and configuring several cache-related Zope objects such as RAMCacheManager with a reasonable set of defaults. CacheFu saves you from having to learn and fully understand the necessary Zope 2 concepts needed to be able to create and configure these objects by hand. It also saves you from having to know about complex web caching, in general. You should probably learn these things too, but you don't have to learn them now.

The addition of CacheFu alone can provide a better performance, but a lot more so when it is combined with a caching agent like Varnish or Squid. We will get to caching agents later in this chapter.

For the next section, we will focus primarily on adding CacheFu to our buildout, and getting it installed and running properly.

In 06-deployment-optimization-cachefu.cfg, we have the following:

[buildout]
extends = 05-deployment-maintenance-rotate2.cfg
[instance]
eggs += Products.CacheSetup

Now, run Buildout:

$ bin/buildout -c 06-deployment-optimization-cachefu.cfg

You should see:

$ bin/buildout -c 06-deployment-optimization-cachefu.cfg
…
Getting distribution for 'Products.CacheSetup'.
File "build/bdist.macosx-10.6-i386/egg/Products/CacheSetup/skins/cache_setup/cache_policy_redirect.py", line 15
return ct.setDisplayPolicy(policy_id, camefrom=camefrom, redirect=True)
SyntaxError: 'return' outside function
…
Got Products.CacheSetup 1.2.1.
Getting distribution for 'Products.PolicyHTTPCacheManager'.
Got Products.PolicyHTTPCacheManager 1.2.
Getting distribution for 'Products.PageCacheManager'.
Got Products.PageCacheManager 1.2.
Getting distribution for 'Products.CMFSquidTool'.
Got Products.CMFSquidTool 1.5.1.
…

Now start Plone:

$ bin/instance fg

To install CacheFu, browse to http://localhost:8080/Plone and click on Site Setup | Add-on Products. You should see:

Select the box next to CacheSetup 1.2.1.jarn.2 and click on Install.

Once installed, click on the Cache Configuration Tool in the Add-on Product Configuration section.

To enable CacheFu, check the Enable CacheFu box and click on Save:

You should experience a significant performance improvement as a result.

For example with CacheFu enabled, using a server-benchmarking tool like ab (http://httpd.apache.org/docs/2.2/programs/ab.html) you may experience serving approximately 80 requests per second:

$ ab -c 1 -n 100 http://127.0.0.1:8080/Plone/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient).....done
…
Requests per second: 80.08 [#/sec] (mean)
…

Now disable CacheFu (uncheck the Enable CacheFu box and click on Save).

At this point, you may experience serving only approximately 10 requests per second:

$ ab -c 1 -n 100 http://127.0.0.1:8080/Plone/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 127.0.0.1 (be patient).....done
…
Requests per second: 10.19 [#/sec] (mean)
…

It is not the exact numbers that are important here, just the variation in results.

You can also use the Firefox add-on Live HTTP Headers (http://livehttpheaders.mozdev.org/) to examine the HTTP headers.

To do so:

You should see:

http://localhost:8080/Plone
GET /Plone HTTP/1.1
Host: localhost:8080
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3
…
X-Cache-Headers-Set-By: CachingPolicyManager: /Plone/caching_policy_manager
Expires: Tue, 23 May 2000 13:09:40 GMT
Vary: Accept-Encoding
Etag: |admin|Plone Default|en-us;en;q=0.5|1|159||||354013
X-Caching-Rule-Id: plone-content-types
Cache-Control: max-age=0, s-maxage=0, private, must-revalidate

In particular, you will notice the caching-related headers like Cache-Control, Etag, Vary, and so on.

At this point, we have done nothing but install CacheFu. We are still serving our site directly from Zope 2.

In production, we would most likely deploy this site behind Apache or Nginx, inside a virtual host that would proxy/rewrite requests to Zope 2. A Zope 2 deployment proxied by Apache or Nginx is fine for many sites, but some sites require more.

So let us now add a caching agent between the web server and Zope 2 to help improve caching even more (of course, we have not actually added the frontend web server yet).

We have a couple of options when it comes to adding a caching agent. We will begin by covering the state-of-the-art, high-performance HTTP accelerator Varnish followed by the venerable Squid caching proxy for the Web.

To add Varnish to your Buildout, you can use one of the CMMI recipes (configure; make; make install) like zc.recipe.cmmi (http://pypi.python.org/pypi/zc.recipe.cmmi) or hexagonit.recipe.cmmi (http://pypi.python.org/pypi/hexagonit.recipe.cmmi).

You can also use an operating system vendor package. Whichever you choose, you may want to use Buildout to create a Varnish configuration file suitable to use with your Plone site.

Advanced users may create their own configuration file template, and generate a configuration file dynamically when Buildout runs, with the help of the collective.recipe.template recipe (http://pypi.python.org/pypi/collective.recipe.template).

We will not cover installation with vendor packages, or the advanced template generation approach.

What we will cover is Varnish installation and configuration with zc.recipe.cmmi (http://pypi.python.org/pypi/zc.recipe.cmmi) and plone.recipe.varnish (http://pypi.python.org/pypi/plone.recipe.varnish).

In 06-deployment-optimization-varnish.cfg, we have:

[buildout]
extends = 06-deployment-optimization-cachefu.cfg
parts +=
varnish-install
varnish
[varnish-install]
recipe = zc.recipe.cmmi
url = http://downloads.sourceforge.net/project/varnish/varnish/2.1.2/
varnish-2.1.2.tar.gz
[varnish]
recipe = plone.recipe.varnish
daemon = ${varnish-install:location}/sbin/varnishd

Now run Buildout:

$ bin/buildout -c 06-deployment-optimization-varnish.cfg

You should see:

$ bin/buildout -c 06-deployment-optimization-varnish.cfg
…
Installing varnish-install.
varnish-install: Downloading http://downloads.sourceforge.net/project/varnish/varnish/2.1.2/varnish-2.1.2.tar.gz
varnish-install: Unpacking and configuring
…
Installing varnish

Now start Varnish with:

$ bin/varnish -F

Browse to http://localhost:8000/Plone. (Note that the port in this URL is 8000.)

Assuming your ZEO and Zope 2 instances are running, you should now see your Plone site (because the default Varnish configuration binds to port 8000 and proxies to port 8080).

If you do not see your Plone site, you may want stop everything and restart as follows:

$ bin/zeo start
$ bin/instance start
$ bin/varnish -F

In this way, we have configured ZEO and Plone to run in the background, while Varnish is running in the foreground (thanks to F).

You can always check the status of ZEO and Plone with the status command.

In the case of ZEO, you should see:

$ bin/zeo status
program running; pid=1088

In the case of Plone, you should see:

$ bin/instance status
program running; pid=10944

That takes care of Varnish. Now let us move on to Squid.

To install squid, we will use the zc.recipe.cmmi recipe (http://pypi.python.org/pypi/zc.recipe.cmmi) and plone.recipe.squid recipe (http://pypi.python.org/pypi/plone.recipe.squid).

In 06-deployment-optimization-squid.cfg, we have:

[buildout]
extends = 06-deployment-optimization-varnish.cfg
parts +=
squid-install
squid
[squid-install]
recipe = zc.recipe.cmmi
url = http://www.squid-cache.org/Versions/v3/3.0/
squid-3.0.STABLE21.tar.gz
[squid]
recipe = plone.recipe.squid:instance
cache-size = 1000
daemon = ${squid-install:location}/sbin/squid
backends = 127.0.0.1:8080
bind = 127.0.0.1:3128

Now stop Varnish and run Buildout:

$ bin/buildout -c 06-deployment-optimization-squid.cfg

Next, be sure to create some cache swap directories for Squid, with the following command:

$ bin/squid -z

You should see:

$ bin/squid -z
2010/05/21 11:54:41| Creating Swap Directories

Next, start Squid in the foreground (with N), as shown:

$ bin/squid -N
Now, browse to http://localhost:3128/Plone. (Note that the port in this URL is 3128.)

Assuming your ZEO and Zope 2 instances are running, you should now see your Plone site (because the default Squid configuration binds to port 3128 and proxies to port 8080).

At this point, we should take a minute to explain the direction we are heading in.

We have configured various caching agents in front of Plone, as well as configured Plone to be more cache-aware via CacheFu.

The next step to improve performance is to divide the application server work amongst one or more Zope 2 instances, assuming of course you have the appropriate hardware to handle such a configuration.

This process will leave you with two or more instances running on various ports.

But you will need a way to aggregate traffic, and direct it in a sane way to the various idle instances waiting to serve you.

This is similar to the checkout line of a grocery store, where instead of having your choice of lines, you are directed through a single line first then arbitrarily (or sequentially) directed to one of the cashiers (who may or may not have a bagger).

In this way, everyone gets a fair turn and your grocery store (or website) performs much better.

To create multiple instances, we will use Buildout macros.

As of version 1.4 of zc.buildout, it is possible to extend sections (except for the buildout section) with the use of the special< parameter. So, to create a macro, create a section with whatever you parameters you want in it, for example foo:

[foo]
bar = baz

Then create another section or sections such as foo1 and foo2 and set the< parameter with the value of the section you want to copy:

[foo1]
< = foo
[foo2]
< = foo

So foo1 and foo2 now contain:

bar = baz

You can now override and add whatever parameters you like:

[foo3]
< = foo
# bar gets bilge, instead of baz
bar = bilge
# car gets baz
car = ${foo:bar}

And so on.

That means we can now define a base instance section (which we did in buildout.cfg) and use it as a macro to create additional instance sections.

To demonstrate this, we will create two additional instances, based on the instance section in buildout.cfg.

In 06-deployment-optimization-macros.cfg, we have:

[buildout]
extends = 06-deployment-optimization-squid.cfg
parts +=
instance1
instance2
[instance1]
< = instance
http-address = 8081
[instance2]
< = instance
http-address = 8082

You should be aware that by doing this, we inherit various settings from the instance section in buildout.cfg, and then change the http-address parameter for each new instance.

Now run buildout with:

$ bin/buildout -c 06-deployment-optimization-macros.cfg

You should see:

$ bin/buildout -c 06-deployment-optimization-macros.cfg
…
Installing instance1.
Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/instance1'.
Installing instance2.
Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/instance2'.

Of course at this point to run our Plone site, we have to start four different services, for example:

$ bin/zeo start
$ bin/instance1 start
$ bin/instance2 start
$ bin/varnish

Also, we have not yet aggregated the instance traffic.

We will address these two problems next, starting with load balancing.

Load balancing (in the Plone context) is the process of distributing requests amongst two or more Zope 2 instances, capable of handling the load. There are a variety of tools available to help with load balancing in Plone including Apache, Varnish, HAProxy, and Pound.

Of these, we will focus primarily on HAProxy and Pound, as they provide the most full-featured and flexible implementations.

[buildout]
extends = 06-deployment-optimization-macros.cfg
parts +=
haproxy-install
[haproxy-install]
recipe = plone.recipe.haproxy

[haproxy-install]
recipe = plone.recipe.haproxy
url = http://my.dist.server/haproxy-1.4.4.zip

$ bin/buildout -c 06-deployment-optimization-haproxy.cfg

Getting distribution for 'plone.recipe.haproxy'.
Got plone.recipe.haproxy 1.1.
…
haproxy-install: Adding script wrapper for haproxy

[buildout]
extends = 06-deployment-optimization-haproxy.cfg
parts +=
haproxy-config
[haproxy-config]
recipe = collective.recipe.template
input = ${buildout:directory}/templates/haproxy.cfg.in
output = ${buildout:directory}/etc/haproxy.cfg

${buildout:directory}

$ bin/haproxy f etc/haproxy.cfg

$ bin/buildout -c 06-deployment-optimization-haproxy-config.cfg

$ bin/buildout -c 06-deployment-optimization-haproxy-config.cfg
…
Installing haproxy-config.

$ bin/haproxy f etc/haproxy.cfg

[buildout]
extends = 06-deployment-optimization-haproxy-config.cfg
parts =
pound-install
[pound-install]
recipe = hexagonit.recipe.cmmi
url = http://www.apsis.ch/pound/Pound-2.4.5.tgz
make-targets =
keep-compile-dir = true

make-targets =

keep-compile-dir = true

$ ls ./parts/pound-install__compile__/Pound-2.4.5/pound

$ ls ./parts/pound-install__compile__/Pound-2.4.5/pound
./parts/pound-install__compile__/Pound-2.4.5/pound*

[buildout]
extends = 06-deployment-optimization-pound.cfg
parts += pound-config
[pound-config]
recipe = collective.recipe.template
input = ${buildout:directory}/templates/pound.cfg.in
output = ${buildout:directory}/etc/pound.cfg

$ ./parts/pound-install__compile__/Pound-2.4.5/pound -f etc/pound.cfg

We have begun to accumulate quite a few of programs in our software stack.

In order to facilitate easy and consistent operation of these programs, we can use an excellent tool called Supervisor (http://supervisord.org) to help us.

As it so happens, there is a Buildout recipe to make configuration file generation easy called collective.recipe.supervisor (http://pypi.python.org/pypi/collective.recipe.supervisor). This one is too easy and simple to resist (hence its popularity).

In 06-deployment-optimization-supervisor.cfg, we have:

[buildout]
extends = 06-deployment-optimization-pound-config.cfg
parts += supervisor
[pound]
directory = ${buildout:directory}/parts/pound-install__compile__
[supervisor]
recipe = collective.recipe.supervisor
programs =
#Prio Name Program Params
00 zeo ${zeo:location}/bin/runzeo
00 instance1 ${instance1:location}/bin/runzope
00 instance2 ${instance2:location}/bin/runzope
00 haproxy ${buildout:directory}/bin/haproxy 
[-f etc/haproxy.cfg]
00 pound ${pound:directory}/Pound-2.4.5/pound 
[-f etc/pound.cfg]
00 varnish ${buildout:directory}/bin/varnish [-F]
00 squid ${buildout:directory}/bin/squid [-N]

We use the pound section to make the path to Pound installation a little more manageable. And we use the programs parameter in the supervisor section to tell Supervisor which programs to run.

You will notice that the priority is set to zero for all of the programs. That is because according to the author Chris McDonough (http://plope.com), the priority setting in Supervisor does not behave as expected.

So when in doubt, you can just give all the programs the same priority.

Now run Buildout:

$ bin/buildout c 06-deployment-optimization-supervisor.cfg

You should see:

$ bin/buildout c 06-deployment-optimization-supervisor.cfg
…
Installing supervisor.
Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/supervisord'.
Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/supervisorctl'.

This configures the entire stack to run all at once; of course, we probably do not need two load balancers and two caching agents.

To check if everything is in order, you can run this command:

$ bin/supervisord e debug n

You should see:

$ bin/supervisord -e debug -n
…
2010-05-21 19:03:04,652 INFO supervisord started with pid 19129
2010-05-21 19:03:05,654 INFO spawned: 'haproxy' with pid 19131
2010-05-21 19:03:05,657 INFO spawned: 'varnish' with pid 19132
2010-05-21 19:03:05,660 INFO spawned: 'pound' with pid 19133
2010-05-21 19:03:05,677 INFO spawned: 'squid' with pid 19134
2010-05-21 19:03:05,682 INFO spawned: 'zeo' with pid 19135
2010-05-21 19:03:05,707 INFO spawned: 'instance2' with pid 19136
2010-05-21 19:03:05,723 INFO spawned: 'instance1' with pid 19140
2010-05-21 19:03:05,724 DEBG fd 10 closed, stopped monitoring
…
2010-05-21 19:03:06,954 INFO spawned: 'haproxy' with pid 19150
2010-05-21 19:03:06,955 INFO success: varnish entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2010-05-21 19:03:06,955 INFO success: pound entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2010-05-21 19:03:06,955 INFO success: squid entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2010-05-21 19:03:06,955 INFO success: zeo entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2010-05-21 19:03:06,956 INFO success: instance2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2010-05-21 19:03:06,956 INFO success: instance1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

It should stay that way. If there are any problems, you will see them immediately.

This chapter has been a mouthful, but there is still more to go!

In order to put the finishing touches on the optimization chapter, we need to take a look at how to figure what parts of our site need optimizing.

One way to do that is to use Munin, using the Munin plugins for Zope 2 and Plone.

With this impressive program, you can easily create beautiful graphs of Zope 2 and Plone performance data.

First, let's review the complex software stack we have created in this chapter.

This is just one of the many scenarios you could deploy within production to handle various types of production environments you may encounter.

Our stack currently looks like this:

In production, it may look like this:

That is to say that in production, you may place a web server like Apache or Nginx in front (on port 80), and then proxy requests to the web server to a web cache like Varnish or Squid, which may then proxy requests down the stack to the Zope 2 instances as needed.

RewriteEngine On
RewriteRule ^(.*) http://127.0.0.1:8080/VirtualHostBase/http/mysite.com:80/Plone/
VirtualHostRoot/$1 [P,L]

proxy_pass http://localhost:8080/VirtualHostBase/http/mysite.com:80/Plone/
VirtualHostRoot/;

As we mentioned earlier, Munin produces graphs that contain diagnostic system information.

With Zope 2, you can analyze four different parameters over time:

To create graphs with Munin, you must install the Munin plugins in Zope 2 as well as install the Munin software on your production server (that is, outside the buildout).

Munin installation should be fairly trivial on most UNIX-like operating systems, including our target operating system Ubuntu Linux. Munin software is typically bundled into three separate packages munin, munin-node, and munin-plugins.

Assuming you have one server, you should have at least the munin and munin-node packages installed to facilitate monitoring of that server and the Plone site running on it.

If you have more than one server (running Plone), you should add munin-nodes to each additional server and configure them to be nodes of the server with the munin package installed.

To install the Munin plugins in Zope 2, we will use the munin.zope package (http://pypi.python.org/pypi/munin.zope/).

[buildout]
extends = 06-deployment-optimization-supervisor.cfg
parts += munin
[instance]
eggs += munin.zope
zcml += munin.zope
[munin]
recipe = zc.recipe.egg
eggs = munin.zope
arguments = http_address='${instance:http-address}',
user='${instance:user}'

$ bin/buildout c 06-deployment-optimization-munin.cfg

$ bin/zeo

$ bin/instance fg

Testing the munin.zope plugins through the Web

After logging in (via http://localhost:8080/manage), browse to http://localhost:8080/@@munin.zope.plugins/zopememory and you should see (something like):

Next, browse to http://localhost:8080/@@munin.zope.plugins/zopethreads and you should see (something like):

After that, browse to http://localhost:8080/@@munin.zope.plugins/zopecache and you should see (something like):

Finally, browse to http://localhost:8080/@@munin.zope.plugins/zodbactivity and you should see (something like):

$ sudo bin/munin install /etc/munin/plugins

$ sudo bin/munin install /etc/munin/plugins
installed symlink /etc/munin/plugins/aclark_zopecache_aclark
installed symlink /etc/munin/plugins/aclark_zopememory_aclark
installed symlink /etc/munin/plugins/aclark_zodbactivity_aclark
installed symlink /etc/munin/plugins/aclark_zopethreads_aclark

$ /etc/munin/plugins/aclark_zopecache_aclark
Total_objects_in_database.value 22959.0
Total_objects_in_all_caches.value 3867.0
Target_number_to_cache.value 20000.0
$ /etc/munin/plugins/aclark_zopememory_aclark
VmPeak.value 495312896.0
VmSize.value 429277184.0
VmLck.value 0.0
VmHWM.value 140443648.0
VmRSS.value 75329536.0
VmData.value 226422784.0
VmStk.value 405504.0
VmExe.value 1392640.0
VmLib.value 10489856.0
VmPTE.value 704512.0
$ /etc/munin/plugins/aclark_zopecache_aclark
Total_objects_in_database.value 22959.0
Total_objects_in_all_caches.value 3867.0
Target_number_to_cache.value 20000.0
$ /etc/munin/plugins/aclark_zopethreads_aclark
Total_threads.value 3.0
Free_threads.value 1.0
aclark@aclark:~/ > /etc/munin/plugins/aclark_zodbactivity_aclark
Total_objects_loaded.value 0.0
Total_objects_stored.value 0.0
Total_connections.value 11.0

Munin graphs

Now let us take a look at some sample data (from the author's server). What follows is a selection of the most interesting graphs chosen from the day, month, week, and year graphs for each plugin.

ZODB activity

The following graph shows a correlation between total connections and total objects loaded, perhaps indicating that most connections reference the same set of objects:

Zope cache parameters

The next graph shows a correlation between the total number of objects in the database, the target number of objects to cache, and the total objects in all caches, perhaps indicating that the target number of objects to cache should be set higher:

Zope memory usage

The next graph shows a correlation between various memory usage statistics and is perhaps the most interesting of the group:

In particular, the point where VmPeak rose above VmSize, perhaps indicates that more memory is needed.

Zope 2 server threads

The last graph shows a correlation between total threads and free threads, apparently indicating that three out of four threads are in use most of the time:

These graphs are at the very least interesting, and at the most can be invaluable when you are experiencing performance-related issues and you do not know the cause.

In addition to munin.zope, there are two more potentially useful plugin packages you may wish to experiment with:

• http://pypi.python.org/pypi/munin.plone

• http://pypi.python.org/pypi/munin.varnish

In this chapter, you have learned:

See you in Chapter 7, where we will adjust and configure various security-related settings.