If you have made it this far, you should understand how to use Buildout to deploy your site to production by executing specific, production-ready configuration files.
We will not cover any version control systems in this book (for example, RCS, CVS, Subversion, Bazaar, Mercurial, Git), but a brief mention of version control systems in the context of our development and deployment efforts is now in order.
Throughout this book, we have been working with various Buildout configuration files. As a result, you may want to consider the following technique to manage them properly:
Check in buildout's configuration files to Subversion (or a version control system of your choice) when you begin your next project.
Check out the buildout's configuration files to the development environment.
Develop the buildout and commit the results.
Check out the buildout's configuration files to the staging and/or production server.
In the future, technologies like Silver Lining (http://cloudsilverlining.org/) may enable one-click deployments. In fact, this is possible today with some Python-based (and other) frameworks and hosting providers, like Pylons and Rackspace Cloud, respectively.
Unfortunately in Plone's case, this technology is not so common as to have made it into this book. It may be closer to reality on Amazon EC2; visit the link http://www.slideshare.net/Jazkarta/plone-in-the-cloud-an-ondemand-cms-hosted-on-amazon-ec2.
In the meantime, checking in your Plone buildouts (and add-on package source code) to Subversion (or a version control system of your choice) is a reasonable approach to help manage the tasks at hand, which brings us back to this chapter.
By now:
In theory, we have checked out the buildout to staging and production, and have automated various maintenance tasks
In practice, we may have already performed various optimization tasks too, though we are just now getting to them here
In this chapter, you will learn:
About caching in the context of Plone
Installing CacheFu a caching add-on for Plone
Installing Varnish a caching agent
Installing Squid a caching agent
About load balancing in the context of Plone
Using Buildout macros to create multiple Zope 2 instances
Installing HAProxy a load balancer
Installing Pound a load balancer
Installing Supervisor a process manager
Understanding the software stack
Installing Munin plugins to analyze performance
Windows folks, consider your options
It may be more difficult to follow along here, as various technologies become more difficult to install and configure, or may not work at all on Windows.
Our apologies!
In the case of a caching agent, you may find that Squid is better supported than Varnish; visit http://wiki.squid-cache.org/SquidFaq/BinaryPackages#Windows for more information.
Because Supervisor does not work on Windows, you can use Windows services to achieve the same effect (which is what the Plone installer for Windows does for you).
If everything else fails, you may seek help from the good folks at Enfold Systems, and perhaps experiment with some of their software like Enfold Server: http://www.enfoldsystems.com/software/server/.
Alternatively, we suggest an Ubuntu Linux virtual machine, running on the freely available VirtualBox (http://www.virtualbox.org/wiki/Downloads).
Zope 2 is an application server. Therefore, every page it delivers involves "doing work", which results in server and network load. If possible, to conserve the system resources, you may want to cache your site's content as much as is practical and possible.
Caching agents store resources needed to serve the site's content and answer site requests with these resources, until the browser or caching agent is no longer able to do so (for example, when a caching rule tells the browser or caching agent that the page has expired).
At this point, it must contact the application server again for new data.
Traditionally, the very first thing many people do to improve Plone's performance is install the CacheFu add-on product.
CacheFu has been around since the Plone 2.5.x days, and has been a welcome addition for folks trying to improve the performance of an increasingly complex and slower running software stack.
Although it is not the magic elixir that many hope it to be, it does provide a decent, measurable improvement for very little effort; so it is usually a good idea to explore.
It does this by creating and configuring several cache-related Zope objects such as RAMCacheManager with a reasonable set of defaults. CacheFu saves you from having to learn and fully understand the necessary Zope 2 concepts needed to be able to create and configure these objects by hand. It also saves you from having to know about complex web caching, in general. You should probably learn these things too, but you don't have to learn them now.
The addition of CacheFu alone can provide a better performance, but a lot more so when it is combined with a caching agent like Varnish or Squid. We will get to caching agents later in this chapter.
For the next section, we will focus primarily on adding CacheFu to our buildout, and getting it installed and running properly.
About CacheFu
CacheFu is covered in much more detail in "Practical Plone 3", Packt Publishing (https://www.packtpub.com/practical-plone-3-beginners-guide-to-building-powerful-websites/book). However, you should be aware that it is beginning to show its age.
If you are interested in the next generation caching add-on for Plone, please check out plone.app.caching on http://pypi.python.org/pypi/plone.app.caching, currently in the alpha release. Watch out for a beta release in the middle of 2010 (around the time this book is published).
In 06-deployment-optimization-cachefu.cfg
, we have the
following:
[buildout] extends = 05-deployment-maintenance-rotate2.cfg [instance] eggs += Products.CacheSetup
Now, run Buildout:
$ bin/buildout -c 06-deployment-optimization-cachefu.cfg
You should see:
$ bin/buildout -c 06-deployment-optimization-cachefu.cfg … Getting distribution for 'Products.CacheSetup'. File "build/bdist.macosx-10.6-i386/egg/Products/CacheSetup/skins/cache_setup/cache_policy_redirect.py", line 15 return ct.setDisplayPolicy(policy_id, camefrom=camefrom, redirect=True) SyntaxError: 'return' outside function … Got Products.CacheSetup 1.2.1. Getting distribution for 'Products.PolicyHTTPCacheManager'. Got Products.PolicyHTTPCacheManager 1.2. Getting distribution for 'Products.PageCacheManager'. Got Products.PageCacheManager 1.2. Getting distribution for 'Products.CMFSquidTool'. Got Products.CMFSquidTool 1.5.1. …
SyntaxErrors are not "real"
Remember that the syntax errors above are not "real" errors. They are caused
by Distribute trying to compile Zope 2 script (Python) files located on the
filesystem into byte code (because they end with .py)
.
However, Zope 2 uses these files when publishing objects to the Web, so
compiling these makes no sense.
Typically, these scripts are found in directories called as Filesystem Directory Views, which are used by CMF Skin Layers. For more information about Zope 2 script (Python) objects, please visit http://docs.zope.org/zope2/zope2book/BasicScripting.html#creating-python-based-scripts.
For more information about why you can ignore these errors, refer to Chapter 2, Site Basics of this book.
Now start Plone:
$ bin/instance fg
To install CacheFu, browse to
http://localhost:8080/Plone
and click on Site
Setup | Add-on Products. You
should see:
Select the box next to CacheSetup 1.2.1.jarn.2 and click on Install.
Once installed, click on the Cache Configuration Tool in the Add-on Product Configuration section.
To enable CacheFu, check the Enable CacheFu box and click on Save:
You should experience a significant performance improvement as a result.
For example with CacheFu enabled, using a server-benchmarking tool like ab (http://httpd.apache.org/docs/2.2/programs/ab.html) you may experience serving approximately 80 requests per second:
$ ab -c 1 -n 100 http://127.0.0.1:8080/Plone/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done … Requests per second: 80.08 [#/sec] (mean) …
Now disable CacheFu (uncheck the Enable CacheFu box and click on Save).
At this point, you may experience serving only approximately 10 requests per second:
$ ab -c 1 -n 100 http://127.0.0.1:8080/Plone/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking 127.0.0.1 (be patient).....done … Requests per second: 10.19 [#/sec] (mean) …
It is not the exact numbers that are important here, just the variation in results.
You can also use the Firefox add-on Live HTTP Headers (http://livehttpheaders.mozdev.org/) to examine the HTTP headers.
To do so:
Download Live HTTP Headers from http://livehttpheaders.mozdev.org/ and install it in Firefox.
Select Tools | Live HTTP Headers.
Browse to http://localhost:8080/Plone
.
You should see:
http://localhost:8080/Plone GET /Plone HTTP/1.1 Host: localhost:8080 User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 … X-Cache-Headers-Set-By: CachingPolicyManager: /Plone/caching_policy_manager Expires: Tue, 23 May 2000 13:09:40 GMT Vary: Accept-Encoding Etag: |admin|Plone Default|en-us;en;q=0.5|1|159||||354013 X-Caching-Rule-Id: plone-content-types Cache-Control: max-age=0, s-maxage=0, private, must-revalidate
In particular, you will notice the caching-related headers like
Cache-Control, Etag, Vary
, and so on.
At this point, we have done nothing but install CacheFu. We are still serving our site directly from Zope 2.
In production, we would most likely deploy this site behind Apache or Nginx, inside a virtual host that would proxy/rewrite requests to Zope 2. A Zope 2 deployment proxied by Apache or Nginx is fine for many sites, but some sites require more.
So let us now add a caching agent between the web server and Zope 2 to help improve caching even more (of course, we have not actually added the frontend web server yet).
We have a couple of options when it comes to adding a caching agent. We will begin by covering the state-of-the-art, high-performance HTTP accelerator Varnish followed by the venerable Squid caching proxy for the Web.
To add Varnish to your Buildout, you can use one of the CMMI recipes
(configure; make; make install)
like
zc.recipe.cmmi (http://pypi.python.org/pypi/zc.recipe.cmmi)
or hexagonit.recipe.cmmi
(http://pypi.python.org/pypi/hexagonit.recipe.cmmi).
You can also use an operating system vendor package. Whichever you choose, you may want to use Buildout to create a Varnish configuration file suitable to use with your Plone site.
Advanced users may create their own configuration file template, and generate a configuration file dynamically when Buildout runs, with the help of the collective.recipe.template recipe (http://pypi.python.org/pypi/collective.recipe.template).
We will not cover installation with vendor packages, or the advanced template generation approach.
What we will cover is Varnish installation and configuration with zc.recipe.cmmi (http://pypi.python.org/pypi/zc.recipe.cmmi) and plone.recipe.varnish (http://pypi.python.org/pypi/plone.recipe.varnish).
Which CMMI recipe should I use?
CMMI stands for configure; make; make install
, which are a
set of UNIX commands typically used to install software from source code
distributions (based on conventions set by the
autoconf/automake/libtool
developer tools).
Take a look at the various features provided by each, described on http://pypi.python.org/pypi/zc.recipe.cmmi and http://pypi.python.org/pypi/hexagonit.recipe.cmmi.
The former provides basic CMMI functionality, with some advanced features like setting environment variables.
The latter provides a few more advanced features, like pre and post-make hooks
to allow you to execute arbitrary commands in addition to running
configure; make; make install
.
In 06-deployment-optimization-varnish.cfg
, we have:
[buildout] extends = 06-deployment-optimization-cachefu.cfg parts += varnish-install varnish [varnish-install] recipe = zc.recipe.cmmi url = http://downloads.sourceforge.net/project/varnish/varnish/2.1.2/ varnish-2.1.2.tar.gz [varnish] recipe = plone.recipe.varnish daemon = ${varnish-install:location}/sbin/varnishd
Now run Buildout:
$ bin/buildout -c 06-deployment-optimization-varnish.cfg
$ bin/buildout -c 06-deployment-optimization-varnish.cfg … Installing varnish-install. varnish-install: Downloading http://downloads.sourceforge.net/project/varnish/varnish/2.1.2/varnish-2.1.2.tar.gz varnish-install: Unpacking and configuring … Installing varnish
Now start Varnish with:
$ bin/varnish -F
Browse to
http://localhost:8000/Plone
. (Note that the port in this URL is
8000.)
Assuming your ZEO and Zope 2 instances are running, you should now see your Plone site (because the default Varnish configuration binds to port 8000 and proxies to port 8080).
If you do not see your Plone site, you may want stop everything and restart as follows:
$ bin/zeo start $ bin/instance start $ bin/varnish -F
In this way, we have configured ZEO and Plone to run in the background, while
Varnish is running in the foreground (thanks to F)
.
You can always check the status of ZEO and Plone with the
status
command.
In the case of ZEO, you should see:
$ bin/zeo status program running; pid=1088
In the case of Plone, you should see:
$ bin/instance status program running; pid=10944
To install squid, we will use the zc.recipe.cmmi recipe (http://pypi.python.org/pypi/zc.recipe.cmmi) and plone.recipe.squid recipe (http://pypi.python.org/pypi/plone.recipe.squid).
In 06-deployment-optimization-squid.cfg, we have:
[buildout] extends = 06-deployment-optimization-varnish.cfg parts += squid-install squid [squid-install] recipe = zc.recipe.cmmi url = http://www.squid-cache.org/Versions/v3/3.0/ squid-3.0.STABLE21.tar.gz [squid] recipe = plone.recipe.squid:instance cache-size = 1000 daemon = ${squid-install:location}/sbin/squid backends = 127.0.0.1:8080 bind = 127.0.0.1:3128
Now stop Varnish and run Buildout:
$ bin/buildout -c 06-deployment-optimization-squid.cfg
Next, be sure to create some cache swap directories for Squid, with the following command:
$ bin/squid -z
You should see:
$ bin/squid -z 2010/05/21 11:54:41| Creating Swap Directories
Next, start Squid in the foreground (with N)
, as shown:
$ bin/squid -N Now, browse to http://localhost:3128/Plone. (Note that the port in this URL is 3128.)
Assuming your ZEO and Zope 2 instances are running, you should now see your Plone site (because the default Squid configuration binds to port 3128 and proxies to port 8080).
At this point, we should take a minute to explain the direction we are heading in.
We have configured various caching agents in front of Plone, as well as configured Plone to be more cache-aware via CacheFu.
The next step to improve performance is to divide the application server work amongst one or more Zope 2 instances, assuming of course you have the appropriate hardware to handle such a configuration.
Choosing the right number of instances
For more information about choosing the appropriate number of Zope 2 instances for your hardware configuration, see Elizabeth Leddy's awesome Plone Conference 2009 presentation Unloading Plone.
Visit http://www.slideshare.net/eleddy/unloading-plone for the slides and http://plone.blip.tv/file/3042000/ for the video.
This process will leave you with two or more instances running on various ports.
But you will need a way to aggregate traffic, and direct it in a sane way to the various idle instances waiting to serve you.
This is similar to the checkout line of a grocery store, where instead of having your choice of lines, you are directed through a single line first then arbitrarily (or sequentially) directed to one of the cashiers (who may or may not have a bagger).
In this way, everyone gets a fair turn and your grocery store (or website) performs much better.
To create multiple instances, we will use Buildout macros.
As of version 1.4 of zc.buildout
, it is possible to extend
sections (except for the buildout
section) with the use of the
special<
parameter. So, to create a macro, create a
section with whatever you parameters you want in it, for example
foo:
[foo] bar = baz
Then create another section or sections such as foo1
and foo2
and set the<
parameter with
the value of the section you want to copy:
[foo1] < = foo [foo2] < = foo
So foo1
and foo2
now contain:
bar = baz
You can now override and add whatever parameters you like:
[foo3] < = foo # bar gets bilge, instead of baz bar = bilge # car gets baz car = ${foo:bar}
And so on.
That means we can now define a base instance section (which we did in
buildout.cfg)
and use it as a macro to create additional instance
sections.
To demonstrate this, we will create two additional instances, based on the
instance section in buildout.cfg
.
In 06-deployment-optimization-macros.cfg
, we have:
[buildout] extends = 06-deployment-optimization-squid.cfg parts += instance1 instance2 [instance1] < = instance http-address = 8081 [instance2] < = instance http-address = 8082
You should be aware that by doing this, we inherit various settings from the
instance section in buildout.cfg
, and then change the
http-address
parameter for each new instance.
Now run buildout with:
$ bin/buildout -c 06-deployment-optimization-macros.cfg
You should see:
$ bin/buildout -c 06-deployment-optimization-macros.cfg … Installing instance1. Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/instance1'. Installing instance2. Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/instance2'.
Of course at this point to run our Plone site, we have to start four different services, for example:
$ bin/zeo start $ bin/instance1 start $ bin/instance2 start $ bin/varnish
Also, we have not yet aggregated the instance traffic.
We will address these two problems next, starting with load balancing.
Load balancing (in the Plone context) is the process of distributing requests amongst two or more Zope 2 instances, capable of handling the load. There are a variety of tools available to help with load balancing in Plone including Apache, Varnish, HAProxy, and Pound.
Of these, we will focus primarily on HAProxy and Pound, as they provide the most full-featured and flexible implementations.
One popular load balancer is HAProxy (http://haproxy.1wt.eu/).
You may reach for it when you would like a more fully-featured load balancer than Pound (covered a little later, not to impugn Pound's work).
To make installation with Buildout easy, there is a recipe called plone.recipe.haproxy (http://pypi.python.org/pypi/plone.recipe.haproxy).
In 06-deployment-optimization-haproxy.cfg
, we
have:
[buildout] extends = 06-deployment-optimization-macros.cfg parts += haproxy-install [haproxy-install] recipe = plone.recipe.haproxy
Firstly, you will notice that the source URL is configured by default to http://dist.plone.org/thirdparty/haproxy-1.4.4.zip which you can override if you like, as shown:
[haproxy-install] recipe = plone.recipe.haproxy url = http://my.dist.server/haproxy-1.4.4.zip
Secondly, you will also notice that we did not use any CMMI recipe, due to
the fact that the package does not follow the CMMI standard
(configure; make; make install)
. Therefore, this
recipe is a workaround.
You will also notice that we did not use the author's
.tar.gz
distribution (available here:
http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.6.tar.gz) due
to some difficulties encountered during untar/ungzip.
Now run buildout:
$ bin/buildout -c 06-deployment-optimization-haproxy.cfg
You should see:
Unfortunately, this recipe does not generate a configuration file for us; neither does plone.recipe.pound (http://pypi.python.org/pypi/plone.recipe.pound).
Others such as isotoma.recipe.pound (http://pypi.python.org/pypi/isotoma.recipe.pound) generate a configuration file.
In the case of application-specific recipes that do not generate configuration files, this is not necessarily a drawback as some folks prefer more control, or more features than the application-specific recipes (for example, Varnish, Pound, and so on) can provide.
Another common complaint filed against the application-specific recipes, is that they provide too much complexity, and too many layers of indirection.
So let us take this opportunity to demonstrate the ultimate in a simple configuration file generation with flexibility with the help collective.recipe. template (http://pypi.python.org/pypi/collective.recipe.template).
In the book examples, you will find a templates directory that contains two files:
haproxy.conf.in
pound.conf.in
Each of these files contains sample configuration in comments, followed by simple configurations designed to work with Plone.
In 06-deployment-optimization-haproxy-config.cfg
, we
have:
[buildout] extends = 06-deployment-optimization-haproxy.cfg parts += haproxy-config [haproxy-config] recipe = collective.recipe.template input = ${buildout:directory}/templates/haproxy.cfg.in output = ${buildout:directory}/etc/haproxy.cfg
You will notice that in the haproxy-config
section we
tell collective.recipe.template about its input
and output
files.
The input
template's variable substitution syntax
is the same as Buildout's very simple.
So, if you want to refer to the buildout directory in a configuration file, you can use:
${buildout:directory}
One word of warning comments will not stop collective.recipe.template from trying to substitute variables (although it might be nice if they did).
Now, anytime you want to edit HAProxy's configuration, you should
edit templates/haproxy.cfg.in
and run
Buildout.
This will produce a "real" configuration file in
etc/haproxy.cfg
, which you can use to run HAProxy, like
this:
$ bin/haproxy f etc/haproxy.cfg
Now run Buildout:
$ bin/buildout -c 06-deployment-optimization-haproxy-config.cfg
You should see:
$ bin/buildout -c 06-deployment-optimization-haproxy-config.cfg … Installing haproxy-config.
Now you can run HAProxy as shown:
$ bin/haproxy f etc/haproxy.cfg
And browse to
http://localhost:10000/Plone
.
Assuming your ZEO and Zope 2 instances are running, you should now see
your Plone site (based on the configuration in
etc/haproxy.cfg)
.
Another popular load balancer is Pound (http://www.apsis.ch/pound/).
Instead of using plone.recipe.pound, we will try a simpler more generic approach with hexagonit.recipe.cmmi.
In 06-deployment-optimization-pound.cfg
, we
have:
[buildout] extends = 06-deployment-optimization-haproxy-config.cfg parts = pound-install [pound-install] recipe = hexagonit.recipe.cmmi url = http://www.apsis.ch/pound/Pound-2.4.5.tgz make-targets = keep-compile-dir = true
Here, you will notice we have configured an empty value for the
make-targets
parameter. That is because we do not want to
configure any make-targets
. Normally, the
make-targets
include make; make install
,
but we do not want to call make install
because it tries
to change ownership of files to the daemon user which every system does not
have:
make-targets =
Instead, we will use the binary in the installation, which we leave in place with:
keep-compile-dir = true
You can verify that Pound has been installed with the following command:
$ ls ./parts/pound-install__compile__/Pound-2.4.5/pound
You should see:
$ ls ./parts/pound-install__compile__/Pound-2.4.5/pound ./parts/pound-install__compile__/Pound-2.4.5/pound*
To create a configuration file we can use with Pound, we will again use collective.recipe.template (http://pypi.python.org/pypi/collective.recipe.template).
In 06-deployment-optimization-pound-config.cfg
, you
should see:
[buildout] extends = 06-deployment-optimization-pound.cfg parts += pound-config [pound-config] recipe = collective.recipe.template input = ${buildout:directory}/templates/pound.cfg.in output = ${buildout:directory}/etc/pound.cfg
We know from our previous experience how collective.recipe.template
uses templates/pound.cfg.in
to produce
etc/pound.cfg
.
Now start Pound with:
$ ./parts/pound-install__compile__/Pound-2.4.5/pound -f etc/pound.cfg
And browse to
http://localhost:10001/Plone
.
Assuming your ZEO and Zope 2 instances are running, you should now see
your Plone site (based on the configuration in
etc/pound.cfg)
.
We have begun to accumulate quite a few of programs in our software stack.
In order to facilitate easy and consistent operation of these programs, we can use an excellent tool called Supervisor (http://supervisord.org) to help us.
As it so happens, there is a Buildout recipe to make configuration file generation easy called collective.recipe.supervisor (http://pypi.python.org/pypi/collective.recipe.supervisor). This one is too easy and simple to resist (hence its popularity).
In 06-deployment-optimization-supervisor.cfg
, we have:
[buildout] extends = 06-deployment-optimization-pound-config.cfg parts += supervisor [pound] directory = ${buildout:directory}/parts/pound-install__compile__ [supervisor] recipe = collective.recipe.supervisor programs = #Prio Name Program Params 00 zeo ${zeo:location}/bin/runzeo 00 instance1 ${instance1:location}/bin/runzope 00 instance2 ${instance2:location}/bin/runzope 00 haproxy ${buildout:directory}/bin/haproxy [-f etc/haproxy.cfg] 00 pound ${pound:directory}/Pound-2.4.5/pound [-f etc/pound.cfg] 00 varnish ${buildout:directory}/bin/varnish [-F] 00 squid ${buildout:directory}/bin/squid [-N]
We use the pound
section to make the path to Pound installation
a little more manageable. And we use the programs
parameter in
the supervisor
section to tell Supervisor which programs to
run.
You will notice that the priority is set to zero for all of the programs. That is because according to the author Chris McDonough (http://plope.com), the priority setting in Supervisor does not behave as expected.
So when in doubt, you can just give all the programs the same priority.
$ bin/buildout c 06-deployment-optimization-supervisor.cfg
You should see:
$ bin/buildout c 06-deployment-optimization-supervisor.cfg … Installing supervisor. Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/supervisord'. Generated script '/Users/aclark/Developer/plone-site-admin/buildout/bin/supervisorctl'.
This configures the entire stack to run all at once; of course, we probably do not need two load balancers and two caching agents.
To check if everything is in order, you can run this command:
$ bin/supervisord e debug n
You should see:
$ bin/supervisord -e debug -n … 2010-05-21 19:03:04,652 INFO supervisord started with pid 19129 2010-05-21 19:03:05,654 INFO spawned: 'haproxy' with pid 19131 2010-05-21 19:03:05,657 INFO spawned: 'varnish' with pid 19132 2010-05-21 19:03:05,660 INFO spawned: 'pound' with pid 19133 2010-05-21 19:03:05,677 INFO spawned: 'squid' with pid 19134 2010-05-21 19:03:05,682 INFO spawned: 'zeo' with pid 19135 2010-05-21 19:03:05,707 INFO spawned: 'instance2' with pid 19136 2010-05-21 19:03:05,723 INFO spawned: 'instance1' with pid 19140 2010-05-21 19:03:05,724 DEBG fd 10 closed, stopped monitoring … 2010-05-21 19:03:06,954 INFO spawned: 'haproxy' with pid 19150 2010-05-21 19:03:06,955 INFO success: varnish entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2010-05-21 19:03:06,955 INFO success: pound entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2010-05-21 19:03:06,955 INFO success: squid entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2010-05-21 19:03:06,955 INFO success: zeo entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2010-05-21 19:03:06,956 INFO success: instance2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2010-05-21 19:03:06,956 INFO success: instance1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
It should stay that way. If there are any problems, you will see them immediately.
This chapter has been a mouthful, but there is still more to go!
In order to put the finishing touches on the optimization chapter, we need to take a look at how to figure what parts of our site need optimizing.
One way to do that is to use Munin, using the Munin plugins for Zope 2 and Plone.
With this impressive program, you can easily create beautiful graphs of Zope 2 and Plone performance data.
First, let's review the complex software stack we have created in this chapter.
This is just one of the many scenarios you could deploy within production to handle various types of production environments you may encounter.
Our stack currently looks like this:
In production, it may look like this:
That is to say that in production, you may place a web server like Apache or Nginx in front (on port 80), and then proxy requests to the web server to a web cache like Varnish or Squid, which may then proxy requests down the stack to the Zope 2 instances as needed.
In Apache, the proxy configuration typically looks like this
(assuming mod_rewrite
and mod_proxy
have been loaded):
RewriteEngine On RewriteRule ^(.*) http://127.0.0.1:8080/VirtualHostBase/http/mysite.com:80/Plone/ VirtualHostRoot/$1 [P,L]
You will typically find these entries inside a VirtualHost container (but not always).
In Nginx, it typically looks like this:
proxy_pass http://localhost:8080/VirtualHostBase/http/mysite.com:80/Plone/ VirtualHostRoot/;
You will typically find these entries inside a server { location / {
} }
block (but not always).
In both of these cases, you may typically rely on the operating system vendor packages to install the frontend Apache or Nginx.
Please note that this is only one of many possible production deployment scenarios. The right way to deploy is a decision you must make, based on the details of each production environment.
We will leave these things to the professionals you, or your system administrator.
Now, we get back to Munin.
As we mentioned earlier, Munin produces graphs that contain diagnostic system information.
With Zope 2, you can analyze four different parameters over time:
Memory
Cache
Threads
ZODB
To create graphs with Munin, you must install the Munin plugins in Zope 2 as well as install the Munin software on your production server (that is, outside the buildout).
Munin installation should be fairly trivial on most UNIX-like operating systems, including our target operating system Ubuntu Linux. Munin software is typically bundled into three separate packages munin, munin-node, and munin-plugins.
Assuming you have one server, you should have at least the munin and munin-node packages installed to facilitate monitoring of that server and the Plone site running on it.
If you have more than one server (running Plone), you should add munin-nodes to each additional server and configure them to be nodes of the server with the munin package installed.
To install the Munin plugins in Zope 2, we will use the munin.zope package (http://pypi.python.org/pypi/munin.zope/).
This package can be a little confusing to install, but the process is
relatively simple once you understand that the installation creates a
bin/munin
script to do two things:
Create symbolic links from the Munin plugins
directory to the bin/munin
script
Produce input for Munin when invoked via the symbolic link
In 06-deployment-optimization-munin.cfg
, we have:
[buildout] extends = 06-deployment-optimization-supervisor.cfg parts += munin [instance] eggs += munin.zope zcml += munin.zope [munin] recipe = zc.recipe.egg eggs = munin.zope arguments = http_address='${instance:http-address}', user='${instance:user}'
You will notice we pass the HTTP address and credentials to log in to our
Plone site, in the arguments
parameter.
You will also notice we use the zc.recipe.egg recipe
(http://pypi.python.org/pypi/zc.recipe.egg) to create a
bin/munin
script.
Now run Buildout:
$ bin/buildout c 06-deployment-optimization-munin.cfg
Now start ZEO:
And start Plone:
After logging in (via
http://localhost:8080/manage
), browse to
http://localhost:8080/@@munin.zope.plugins/zopememory
and you should
see (something like):
Next, browse to
http://localhost:8080/@@munin.zope.plugins/zopethreads
and you
should see (something like):
After that, browse to
http://localhost:8080/@@munin.zope.plugins/zopecache
and you
should see (something like):
Finally, browse to
http://localhost:8080/@@munin.zope.plugins/zodbactivity
and you
should see (something like):
At this point, the Zope 2 instance is configured properly, but Munin knows nothing about the Zope 2 instance.
To configure the plugins for use in Munin, run bin/munin
on
your production server with the install
command and the
location of the plug-ins
directory, as following:
$ sudo bin/munin install /etc/munin/plugins
You should see:
You can also test the plugins on the command line, as shown:
$ /etc/munin/plugins/aclark_zopecache_aclark Total_objects_in_database.value 22959.0 Total_objects_in_all_caches.value 3867.0 Target_number_to_cache.value 20000.0 $ /etc/munin/plugins/aclark_zopememory_aclark VmPeak.value 495312896.0 VmSize.value 429277184.0 VmLck.value 0.0 VmHWM.value 140443648.0 VmRSS.value 75329536.0 VmData.value 226422784.0 VmStk.value 405504.0 VmExe.value 1392640.0 VmLib.value 10489856.0 VmPTE.value 704512.0 $ /etc/munin/plugins/aclark_zopecache_aclark Total_objects_in_database.value 22959.0 Total_objects_in_all_caches.value 3867.0 Target_number_to_cache.value 20000.0 $ /etc/munin/plugins/aclark_zopethreads_aclark Total_threads.value 3.0 Free_threads.value 1.0 aclark@aclark:~/ > /etc/munin/plugins/aclark_zodbactivity_aclark Total_objects_loaded.value 0.0 Total_objects_stored.value 0.0 Total_connections.value 11.0
After five minutes (the default length of time between Munin runs), check your
Munin web directory, for example /var/www/munin
for images
and HTML files.
Now let us take a look at some sample data (from the author's server). What follows is a selection of the most interesting graphs chosen from the day, month, week, and year graphs for each plugin.
The following graph shows a correlation between total connections and total objects loaded, perhaps indicating that most connections reference the same set of objects:
The next graph shows a correlation between the total number of objects in the database, the target number of objects to cache, and the total objects in all caches, perhaps indicating that the target number of objects to cache should be set higher:
The next graph shows a correlation between various memory usage statistics and is perhaps the most interesting of the group:
In particular, the point where VmPeak rose above VmSize, perhaps indicates that more memory is needed.
The last graph shows a correlation between total threads and free threads, apparently indicating that three out of four threads are in use most of the time:
These graphs are at the very least interesting, and at the most can be invaluable when you are experiencing performance-related issues and you do not know the cause.
In addition to munin.zope, there are two more potentially useful plugin packages you may wish to experiment with:
In this chapter, you have learned:
Web caching, particularly as it relates to Plone
How to install CacheFu in Plone to configure cache settings
How to add Varnish and Squid to your buildout to use as caching agents
Load balancing, particularly as it relates to Plone
How to use the<
parameter in Buildout to create
macros
How to add HAProxy and Pound to your buildout to do load balancing
How to add Supervisor to your buildout to manage processes
How the production stack works
How to add Munin plugins to your buildout
See you in Chapter 7, where we will adjust and configure various security-related settings.
18.224.54.255