Persons grouped around a fire or candle for warmth or light are less able to pursue independent thoughts, or even tasks, than people supplied with electric light. In the same way, the social and educational patterns latent in automation are those of self-employment and artistic autonomy. | ||
| --Marshall McLuhan | ||
One frequently overlooked aspect of building a Rails application is, believe it or not, launching and running it in production. In some cases you may not be responsible for this part of your project, but it is important to understand how a modern web application operates in a production environment. In this chapter we’re going to show you how to build a simple production “stack,” and how to get your Rails application running on it. Many concerns play a role in how you design your production stack. We are going to stick with a basic, common configuration so that you can understand the key components and best practices. We will review some of the common concerns that require more complex configurations, which will be helpful if this is your first production deployment.
Even if you have run a Rails app in a production environment before, you will probably find this chapter worth reading, as we discuss some simple ways to automate your production system and keep its configuration clean and simple.
Fortunately it has become a relatively straightforward procedure to get a Rails application running in a production environment, but that has not always been the case. Many of the time-saving design philosophies and best practices that are essential to Rails, including the well-known maxims like “convention over configuration” and “don’t repeat yourself,” have also made their way into production configurations and deployment practices as well. Another way of saying this would be that automation is key. Rails enables developers to focus on the unique behavior of their application and automates the rest of the application behavior for you. In the same vein, tools like Mongrel, Mongrel Cluster, and Capistrano simplify and automate the tedious parts of running an application in production. Many of the lessons that have been learned about running a Rails app in production have been baked into tools like Capistrano, Mongrel, and Mongrel Cluster.
When Rails first came out, your options for running it behind a real web server like Apache were limited to CGI, Apache’s mod_ruby, or FastCGI (aka FCGI), all of which had their shortcomings. Scripts in your Rails script directory like spawner and reaper were just workarounds for FastCGI issues.
In 2006 Zed Shaw wrote a production-capable, mostly Ruby HTTP web server called Mongrel. Mongrel was designed from the start to replace the existing options while staying small and simple. Instead of forcing a front-facing web server like Apache to convert an incoming HTTP request to CGI just in order to load the Ruby environment, Mongrel,[1] as an HTTP-speaking Ruby process, could speak to Apache directly. In doing so, Mongrel cut out two middlemen (converting the request from HTTP to CGI, and loading Ruby), reduced the number of moving parts, increased the reliability and predictability of the production environment, and sped up the performance. It also provided a fast web server for local development.
In order to successfully set up the production stack we’re going to build, you’ll need at least the following:
A working understanding of Unix
By a large margin most Rails applications in production are running on some variant of Unix (FreeBSD, OSX, Solaris, any Linux distro, etc.). Unless you have some severe production environment constraints (i.e., you work at a big company and are trying to sneak Rails through the back door), you should be setting your application up on Unix too. We’ll assume that you understand how to use the command-line interface, and you can execute basic commands, install packages, start/stop services, and so on. The examples we list in this chapter will be through the Bash shell.[2]
Access to a fresh server with sudo access
We’ll be installing everything onto one server, so you’ll need to have access to one and be able to install software and perform deployments onto it. We’ll list the specific applications and types of access in the section “The Stack Checklist.” There are hosting options we’ll list as well, in case you weren’t able to find any via your favorite search engine.
Respect for the production environment and a strong desire to learn
Hopefully, your Rails application will live a long, healthy life in production. If you are not usually the person who is responsible for handling the production life cycle of an application, it’s critical to understand how important this part is. You should avoid thinking about the setup and maintenance of your production environment as a chore. These tasks may not be as fun as writing an application, but they are at least as important. Like any technology (e.g., cars, ships, etc.), after your app is out in the open it will take on a different set of concerns. This is particularly important for modern web applications, which undergo frequent changes and iterative development cycles. Your production life cycle will feed back into your development life cycle, and so on.
If you have ever run or maintained an application in production before, you understand how many of the peskier bugs arise here, and that this is the last place you want cluttered, poorly organized logs and configuration files getting in your way when you are the one getting up at 4 a.m. if something blows up. Respecting the importance and cleanliness of your production environment will inevitably save many headaches down the line, and many of the best practices you’ll find here (and in the tools and libraries we recommend) are the product of learning how to avoid them.
Willingness to shed old habits and learn new ones
One key concept in production environments is automation. Automate everything! All of the deployments and configurations make your life simple by doing all of the work for you. If you like to tweak files by hand or modify deployments already on the server, you will need to break those old habits. These kinds of behaviors (which we have seen in developers and sysadmins alike) will make your production environment brittle, unpredictable, and otherwise doomed.
The ability to operate a heavily automated production system requires you to use it as much as possible and remove any chance of manual human interference. You may think it’s faster to make a quick change by hand by editing some file on the live production server, but those habits result in death by a thousand cuts. They are difficult to track over time, and automated systems do not respond well to manual changes.
Willing suspension of disbelief
This is somewhat of a corollary to the preceding point. Sometimes developers and system administrators like to complicate things, especially when they think they have a better way. If you are one of these people (even if you think you aren’t) and read this chapter and think our approach is wrong, overly simplified, or whatever, just take it easy. Most simple Rails applications run on environments just like this, and the packages we recommend are among the most heavily used by professional Rails hosting environments. For example, some people don’t like the idea of using any web server other than Apache, usually because it’s been around for years and it’s the only thing they know. Don’t be afraid of Nginx; you might actually like it after you see how simple and fast (and bug-free) it is.
Before we get into the details of the software you’ll need, we’ll cover the general assumptions about what kind of configuration we’re going to build. As we said earlier, there are many ways to build a production stack, so we’re sticking with a simple, proven configuration that will work well for many Rails applications.
The following sections describe the key components for the production stack we’ll be building. You will notice that in addition to the standard Web, Application, and Database tiers we also discuss two other critical parts that are frequently left out of discussion: the server and network environment and monitoring components.
Of course, we need to run our application somewhere...
Since we are going to run everything from one server, you will need to have access to either a dedicated server, or a VPS slice. Some VPS hosting services are already tailored for hosting Rails applications and will take care of all the dirty work for you (e.g., Rails Machine, EngineYard, or Slicehost). For our purposes we’re going to start from a naked server and work our way up, though ultimately the configuration will largely be the same.
A fresh install of any popular Linux distro should work. We prefer Debian, Gentoo, CentOS, or RedHat. Some of us have personal preferences for other distros, but these are usually the ones hosting companies prefer to use.
Depending on your comfort level or preference, you can install the required packages using your favorite package manager or you can compile them from source. In our examples we’ll compile from source or install precompiled binaries (e.g., MySQL), for the sake of staying simple.
As we mentioned at the beginning of the chapter, you’ll need to have a user with sudo privileges (or just root). If you don’t have this, it’s not impossible to get a Rails app running, but you’ll need to coordinate with the sysadmin who does have sudo access to make sure everything can be installed correctly.
We are running everything from one server, but you’ll still need to get to it for administration, and since it’s a web site, your users will need to get to it through a browser. You will need SSH access, and ports 80 and 443 open. If your server is running behind a firewall, the firewall will need to keep those open. We strongly recommend running SSH on a nonstandard port, just to avoid some common security attacks. See the security section for more details on ports.
The current best practice for running your Rails application in production is to use a fast “static” web server like Apache or Nginx running in the front that points to a cluster of mongrels running in the back. The fast web server will receive an incoming request and reverse-proxy it to one of the available Mongrel processes when appropriate (e.g., you can set up the rules so it will serve static assets or cached files directly). This approach is used by most professional Rails hosting companies and in many production clusters and is much more reliable than FastCGI, SCGI, or any of the previous approaches.
Apache 2.2.x and Nginx are the two preferred web servers to run in the front. We’re going to use Nginx here, because it is fast, stable, far less complicated than Apache, and it has a tiny memory footprint. It’s relatively new to the Western hemisphere, but the Nginx site states that about one-fifth of Russian websites are running on it.[3] One critical reason we prefer it to Apache is that its reverse-proxying is less error-prone. Apache is still a good solution, but if you only prefer it because you already know it, we suggest giving Nginx a chance. You’ll be happily surprised. If you must use Apache, the Mongrel site has some excellent resources about how to set up mod_proxy_balancer to talk to your Mongrel Cluster.[4]
The tools that you need to run at the application tier are minimal. Ruby, RubyGems, Rails, and their dependent gems will be enough to get you going. We’ll list the specific gems needed for running the stack in the “Installations” section, but you should know which gems your own application is dependent upon.
Our configuration will be based on the deployment of a simple, single Rails application. If you have more complex requirements, such as a BackrounDRb process, or cron jobs, and so on, you will have to handle those details yourself. Keep in mind that if you have a more complex configuration you’ll need to make sure your Capistrano tasks are all set up properly. You can read more about Capistrano in the next chapter.
Most Rails applications use a single database. In a simple production configuration like ours we will run the database on the same server. In more complex configurations you can set up your database to address redundancy, failover, or performance concerns. As long as you’re doing a simple database server setup, this won’t change much from your local development environment.
We recommend the MySQL version 5.x branch; however, version 4.x is largely interchangeable and will work well for most projects. You will know your own application’s database requirements, so if you prefer another database, make sure you have reviewed Rails support for it.
Monitoring tools are not technically required in order to run a production environment, but in the interest of best practices we’re making them required here. Running your Rails application without them will be like driving a car without any gauges. Since this is a production environment, you’ll probably want to know when the site becomes unavailable, or when someone uploads a file and it pegs the CPU or MySQL processes.
We recommend Subversion for version control. You will likely already have your application in version control (and if you don’t, now is a good time to start). We also recommend storing other important files in version control, which we discuss later in the section “Configurations.”
In this section we’re going to install all of the necessary applications and tools before you can perform a deployment. Before you begin you should make sure you meet the requirements described in the preceding section, or know enough to work within your own constraints.
In this section we’re installing the tools directly from source, but you can install them using your favorite package manager if you prefer. In some cases the package managers make it easier to maintain subsequent updates, but they tend to be a few revisions behind what are sometimes important updates to tools.
Note
Everyone has their preferences about where to store libraries on their systems. If you prefer to install into different paths than what you see here, feel free. We’re going to install into /usr/local and will be putting the application in /var/www/apps/railsway/, which you will see referenced in some of the configurations.
The current recommended version of Ruby is the 1.8.5 branch. There have been some problems reported with Ruby 1.8.6, so unless you are fully confident that 1.8.6 works fine in your development and staging environments, we recommend sticking with the latest patch release from the 1.8.5 branch. As of this writing, the latest patch release is 52.
The following commands will download and install Ruby from source.
$ curl -O ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.5-p52.tar.gz $ tar zxvf ruby-1.8.5-p52.tar.gz ... $ cd ruby-1.8.5-p52 $ ./configure -prefix=/usr/local $ make $ sudo make install
The current version of RubyGems is 0.9.4. Once you install this, all other Ruby-related libraries can be installed by installing gems. The following commands will download and install RubyGems from source. Once it is installed, RubyGems can upgrade itself when new versions are released.
$ curl -O http://files.rubyforge.mmmultiworks.com/rubygems/rubygems- 0.9.4.tgz $ tar zxvf rubygems-0.9.4.tgz ... $ cd rubygems-0.9.4 $ sudo ruby setup.rb ...
We are working on the latest 1.2.x version, which at the time of this writing is 1.2.3. The -y switch here is the equivalent of -include-dependencies. The following command will install the Rails gems (one for each subframework) into your system gems.
$ sudo gem install rails -y
We are using Mongrel 1.0.1, which is the version most widely used in production environments. On your local development machine you usually call the mongrel_rails (or script/server) command directly. Once we set up the init scripts later in the chapter, you will use that to control the start, stop, and restart of the cluster. The following command installs the Mongrel gem and its dependencies.
$ sudo gem install mongrel -y
Mongrel Cluster is a gem that allows you to run a set of mongrel processes with a common configuration so they can be proxied to from Nginx. We are using version 1.0.2.
The following command will install the mongrel_cluster gem, which will allow you to configure and run mongrels in a “pack.” When we set up the static web server we will point to a configured mongrel_cluster configuration. Mongrel Cluster’s commands will automatically be available to you through Mongrel’s mongrel_rails command.
$ sudo gem install mongrel_cluster -y
Nginx is the fast, simple, static web server that will sit at the front of your production environment. It will be responsible for handling incoming HTTP requests, either on its own (such as for static assets on disk already) or by proxying the requests to one of the mongrel processes running in the mongrel cluster. The following commands will download and compile the source for the current stable 0.5.x branch of Nginx.
$ curl -O http://sysoev.ru/nginx/nginx-0.5.28.tar.gz $ tar zxvf nginx-0.5.28.tar.gz $ cd nginx-0.5.28 $ ./configure --sbin-path=/usr/local/sbin --with-http_ssl_module ... $ make ... $ sudo make install
Version control should be part of your standard arsenal. You can use other version management systems with Capistrano but Subversion is the preferred default. The following commands will download and install Subversion and some of its additional dependencies from source:
$ curl -O http://subversion.tigris.org/downloads/subversion- 1.4.4.tar.gz $ curl -O http://subversion.tigris.org/downloads/subversion-deps- 1.4.4.tar.gz $ tar zxvf subversion-1.4.4.tar.gz $ tar zxvf subversion-deps-1.4.4.tar.gz $ cd subversion-1.4.4 $ ./configure --prefix=/usr/local --with-openssl --with-ssl --with-zlib ... $ make ... $ sudo make install
MySQL 5.x can be installed using your package manager, from source, or from the precompiled binaries for your platform. The way you install it depends on your preferences and constraints. For our purposes we’ll install the generic binaries.
You can find the appropriate package at http://dev.mysql.com/downloads/mysql/5.0.html.
We strongly recommend that you lock down your MySQL installation by setting a root password and limiting access to the local machine. You can find out more about MySQL at http://www.securityfocus.com/infocus/1726.
Monit[5] is an excellent monitoring tool for managing processes and keeping tabs on your resource usage. Monit is highly configurable and can be set up to notify you based on many key metrics (CPU usage, disk usage, and so on). The following commands will download and install Monit from source files.
$ curl -O http://www.tildeslash.com/monit/dist/monit-4.9.tar.gz $ tar zxvf monit-4.9.tar.gz ... $ cd monit-4.9 $ ./configure ... $ make ... $ sudo make install ...
You do not need to install Capistrano on the server, only on your local machine. We are using the latest version of Capistrano, which right now is 2.x. We cover Capistrano thoroughly in Chapter 21, “Capistrano.” The following command will download and install the Capistrano Ruby gem and its dependencies:
$ sudo gem install capistrano -y
Now that we have everything installed, we’re going to set up the configurations for each of the tools. For most straightforward Rails applications you will only need to configure a few things: Mongrel Cluster, Nginx, and Monit.
One technique for simplifying and automating your system is to keep a deploy directory right in your config/ directory of your app, and use a separate subdirectory for each of the deployments you have (e.g., dev, staging, production). Each subdirectory will have its own copy of mongrel_cluster.yml, database.yml, nginx.conf, railsway.conf, and so on. In your post-deploy Capistrano tasks you can push these files into /var/www/railsway/shared/config and symlink to them from where they are expected (e.g., /etc/nginx/nginx.conf -> /var/www/railsway/shared/config/nginx.conf).
We’re going to generate a basic mongrel_cluster config file using the mongrel cluster gem’s configure command. You can always write and edit this configuration by hand, but it’s convenient to generate at least the initial version. You can do this like so:
$ mongrel_rails cluster::configure -p 8000 -e production -a 127.0.0.1 -N 2 --user deploy --group deploy -P /var/www/apps/railsway/shared/pids/mongrel.pid -c /var/www/apps/railsway/current
That will produce the file mongrel_cluster.yml, whose contents are shown in the following listing. You can store it in the shared directory of your Capistrano deployment tree (e.g., /var/www/railsway/shared/config/mongrel_cluster.yml).
-- cwd: /var/www/apps/railsway/current port: '8000' user: deploy group: deploy environment: production address: 127.0.0.1 pid_file: /var/www/apps/railsway/shared/pids/mongrel.pid servers: 2
We’re going to split the total Nginx configuration into two separate files: nginx.conf and railsway.conf. This pattern is a simple way to keep the application-specific details out of the global config file.
This is your primary Nginx configuration file. It sets up the pid, basic logging, gzip compression, and mime-types. Notice at the bottom that it includes the railsway.conf file. You could just as easily paste the contents of railsway.conf in its place, but by keeping it separate it will be easier to read and change over time. This approach also works well when you have multiple applications running on the same server.
# user and group to run as
user deploy deploy;
# number of nginx workers
worker_processes 4;
# pid of nginx master process
pid /var/run/nginx.pid;
error_log /var/log/nginx/default.error.log debug;
# Number of worker connections. 1024 is a good default
events {
worker_connections 8192;
use epoll; # linux only!
}
# start the http module where we config http access.
http {
# pull in mime-types. You can break out your config
# into as many include's as you want to make it cleaner
include /etc/nginx/mime.types;
# set a default type for the rare situation that
# nothing matches from the mime-type include
default_type application/octet-stream;
# configure log format
log_format main '$remote_addr - $remote_user [$time_local] '
''$request' $status $body_bytes_sent
'$http_referer' '
''$http_user_agent' '$http_x_forwarded_for'';
# no sendfile on OSX
sendfile on;
# These are good default values.
tcp_nopush on;
tcp_nodelay on;
# output compression saves bandwidth
gzip on;
gzip_http_version 1.0;
gzip_comp_level 2;
gzip_proxied any;
gzip_types text/plain text/html text/css application/
x-javascript text/xml application/xml application/xml+rss text/
javascript;
access_log /var/log/nginx.default.access.log main;
error_log /var/log/nginx.default.error.log info;
# Hosted applications
include /etc/nginx/railsway.conf;
}This is your specific application’s config file. It will be included inside the main nginx.conf in the preceding listing. Just as in Apache, when you have SSL set up, you need to basically copy the set of configuration details. We include an example of an SSL config here, but if you don’t need SSL you can just remove it.
upstream railsway {
server 127.0.0.1:8000;
server 127.0.0.1:8001;
}
server {
# port to listen on. Can also be set to an IP:PORT
listen 80 default;
# Set the max size for file uploads to 50Mb
client_max_body_size 50M;
# sets the domain[s] that this vhost server requests for
server_name railsway.com;
# doc root
root /var/www/apps/railsway/current/public;
# vhost specific logs
access_log /var/www/apps/railsway/shared/log/railsway.access.log
main;
error_log /var/www/apps/railsway/shared/log/railsway.error.log
notice;
# this rewrites all the requests to the maintenance.html
# page if it exists in the doc root. This is for capistrano's
# disable web task
if (-f $document_root/system/maintenance.html) {
rewrite ^(.*)$ /system/maintenance.html last;
break;
}
# Block access to paths containing .svn
location ~* ^.*.svn.*$ {
internal;
}
location / {
index index.html index.htm;
# Forward the user's IP address to Rails
proxy_set_header X-Real-IP $remote_addr;
# needed for HTTPS
proxy_set_header X_FORWARDED_PROTO https;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect false;
proxy_max_temp_file_size 0;
location ~ ^/(images|javascripts|stylesheets)/ {
expires 10y;
}
if (-f $request_filename) {
break;
}
if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}
if (! -f $request_filename) {
proxy_pass http://railsway;
break;
}
}
error_page 500 502 503 504 /500.html;
location = /500.html {
root /var/www/apps/railsway/current/public;
}
}
server {
# port to listen on. Can also be set to an IP:PORT
listen 443 default;
# Set the max size for file uploads to 50Mb
client_max_body_size 50M;
# sets the domain[s] that this vhost server requests for
server_name railsway.com;
# SSL certificate configuration
ssl on;
ssl_certificate /etc/nginx/ssl/railsway.cert;
ssl_certificate_key /etc/nginx/ssl/railsway.key;
keepalive_timeout 70;
add_header Front-End-Https on;
# doc root
root /var/www/apps/railsway/current/public;
# vhost specific logs
access_log /var/www/apps/railsway/shared/log/railsway.access.log
main;
error_log /var/www/apps/railsway/shared/log/railsway.error.log
notice;
# this rewrites all the requests to the maintenance.html
# page if it exists in the doc root. This is for capistrano's
# disable web task
if (-f $document_root/system/maintenance.html) {
rewrite ^(.*)$ /system/maintenance.html last;
break;
}
# Block access to paths containing .svn
location ~* ^.*.svn.*$ {
internal;
}
location / {
index index.html index.htm;
# Forward the user's IP address to Rails
proxy_set_header X-Real-IP $remote_addr;
# needed for HTTPS
proxy_set_header X_FORWARDED_PROTO https;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect false;
proxy_max_temp_file_size 0;
location ~ ^/(images|javascripts|stylesheets)/ {
expires 10y;
}
if (-f $request_filename) {
break;
}
# Rails page caching, part 1
# Add '/index.html' to the end of the current request's path
in the URL
# and look for that file on the file system.
if (-f $request_filename/index.html) {
rewrite (.*) $1/index.html break;
}
# Rails page caching, part 2
if (-f $request_filename.html) {
rewrite (.*) $1.html break;
}
if (! -f $request_filename) {
proxy_pass http://railsway;
break;
}
}
error_page 500 502 503 504 /500.html;
location = /500.html {
root /var/www/apps/railsway/current/public;
}
}You can test the configs by running the following command against them:
$ sudo /usr/local/sbin/nginx -t -c config/nginx.conf
Monit’s configuration file is quite easy to read. This fairly extensive configuration example will check the key processes on your server in one-minute intervals and alert you if any of the conditions you specify are met. You can check system- or process-level usage of disk, CPU, and average load. You can check whether processes are running, and you can even see if a process has been restarted a certain number of times within a given number of intervals. You can find more examples of useful Monit configurations in the example monitrc included with the tool’s source.
set daemon 60 # Poll at 1-minute intervals set logfile /var/log/monit.log set alert [email protected] set mail-format { from: [email protected] subject: $SERVICE service - $EVENT message: $ACTION $SERVICE on $HOST: $DESCRIPTION } set httpd port 1380 allow localhost # Allow localhost to connect check system railsway.com alert [email protected] but not on { instance } if loadavg(1min) > 4 for 3 cycles then alert if loadavg(5min) > 3 for 3 cycles then alert if memory usage > 80% for 3 cycles then alert if cpu usage (user) > 70% for 5 cycles then alert if cpu usage (system) > 30% for 5 cycles then alert if cpu usage (wait) > 20% for 5 cycles then alert check process nginx with pidfile /var/run/nginx.pid start program = "/etc/init.d/nginx start" stop program = "/etc/init.d/nginx stop" if 2 restarts within 3 cycles then timeout if failed host localhost port 80 protocol http then restart if failed host localhost port 443 then restart check process sendmail with pidfile /var/run/sendmail.pid start program = "/etc/init.d/sendmail start" stop program = "/etc/init.d/sendmail stop" check process mysql with pidfile /var/run/mysqld/mysqld.pid start program = "/etc/init.d/mysqld start" stop program = "/etc/init.d/mysqld stop" check process mongrel_8000 with pidfile /var/www/railsway/shared/pids/ mongrel.8000.pid start program = "/usr/bin/mongrel_rails cluster::start -C /var/www/railsway/shared/config/mongrel_cluster.yml --clean --only 8000" stop program = "/usr/bin/mongrel_rails cluster::stop -C /var/www/railsway/shared/config/mongrel_cluster.yml --clean --only 8000" if failed port 8000 protocol http with timeout 10 seconds then restart if totalmem is greater than 128 MB for 4 cycles then restart # eating up memory? if cpu is greater than 60% for 2 cycles then alert # send an email to admin if cpu is greater than 90% for 5 cycles then restart # hung process? if loadavg(5min) greater than 10 for 8 cycles then restart # bad, bad, bad if 3 restarts within 5 cycles then timeout # something is wrong, call the sys-admin group mongrel check process mongrel_8001 with pidfile /var/www/railsway/shared/ pids/mongrel.8001.pid start program = "/usr/bin/mongrel_rails cluster::start -C /var/www/ railsway/shared/config/mongrel_cluster.yml --clean --only 8001" stop program = "/usr/bin/mongrel_rails cluster::stop -C /var/www/ railsway/shared/config/mongrel_cluster.yml --clean --only 8001" if failed port 8001 protocol http with timeout 10 seconds then restart if totalmem is greater than 128 MB for 4 cycles then restart # eating up memory? if cpu is greater than 60% for 2 cycles then alert # send an email to admin if cpu is greater than 90% for 5 cycles then restart # hung process? if loadavg(5min) greater than 10 for 8 cycles then restart # bad, bad, bad if 3 restarts within 5 cycles then timeout # something is wrong, call the sys-admin group mongrel
As we mentioned earlier in this chapter, many of the time- and cost-saving benefits of setting up your production environment the way we are recommending are based on automation. Process management is a good candidate for heavy automation. This is done using init scripts, which serve two purposes: They are run on system startup (when your server boots), and they are run either through the shell (by you, manually when necessary) or when you execute Capistrano commands when running start/stop/restart tasks. We include code samples for Mongrel, Monit, and Nginx, though the last two are shell scripts and are OS-specific. The Mongrel init script is written in Ruby and is generic enough for general use.
You will want to find an Nginx init script so you can start, stop, or reconfigure Nginx. You should find one (using your friendly search engine) that is specific to the OS you are using. The following example is for CentOS. This file should be named /etc/init.d/nginx.
#!/bin/sh
# v1.0
# nginx - Start, stop, or reconfigure nginx
#
# chkconfig: - 60 50
# description: nginx [engine x] is light http web/proxy server
# that answers incoming ftp service requests.
# processname: nginx
# config: /etc/nginx/nginx.conf
# pidfile: /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ ${NETWORKING} = "no" ] && exit 0
BINARY=/usr/sbin/nginx
CONF_FILE=/etc/nginx/nginx.conf
PID_FILE=/var/run/nginx.pid
[ -x $BINARY ] || exit 0
RETVAL=0
prog="nginx"
start() {
# Start daemons.
if [ -e $BINARY ] ; then
echo -n $"Starting $prog: "
$BINARY -c $CONF_FILE
RETVAL=$?
[ $RETVAL -eq 0 ] && {
touch /var/lock/subsys/$prog
success $"$prog"
}
echo
else
RETVAL=1
fi
return $RETVAL
}
stop() {
# Stop daemons.
echo -n $"Shutting down $prog: "
kill -s QUIT `cat $PID_FILE 2>/dev/null`
RETVAL=$?
echo
[ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/$prog
return $RETVAL
}
# See how we were called.
case "$1" in
start)
start
;;
stop)
stop
;;
reconfigure)
if [ -f /var/lock/subsys/$prog ]; then
kill -s HUP `cat $PID_FILE 2>/dev/null`
RETVAL=$?
fi
;;
status)
status $prog
RETVAL=$?
;;
*)
echo $"Usage: $0 {start|stop|reconfigure|status}"
exit 1
esac
exit $RETVALThis Ruby script goes in /etc/init.d/mongrel (your OS may have a different place to store init scripts). You can use it to start, stop, and restart mongrel processes. If you place this script in init.d, it will be run when the server boots.
#! /usr/bin/env ruby
#
# mongrel Startup script for Mongrel clusters.
#
# chkconfig: 345 85 00
#
# description: mongrel_cluster manages multiple Mongrel
processes for use
# behind a load balancer.
#
MONGREL_RAILS = '/usr/bin/mongrel_rails'
CONF_FILE = '/etc/railsway/mongrel_cluster.yml'
SUBSYS = '/var/lock/subsys/mongrel'
SUDO = '/usr/bin/sudo'
case ARGV.first
when 'start'
'#{MONGREL_RAILS} cluster::start -C #{CONF_FILE}'
'#{SUDO} touch #{SUBSYS}'
when 'stop'
'#{MONGREL_RAILS} cluster::stop -C #{CONF_FILE}'
'#{SUDO} rm -f #{SUBSYS}'
when 'restart'
'#{MONGREL_RAILS} cluster::restart -C #{CONF_FILE}'
when 'status'
'#{MONGREL_RAILS} cluster::status -C #{CONF_FILE}'
else
puts 'Usage: /etc/init.d/mongrel {start--stop--restart--status}'
exit 1
end
exit $?You will also want to have your init script for Monit. If you use Monit to manage your processes, then Monit will actually call the other scripts in the Capistrano tasks. You will want to make sure that Monit is responsible for starting, stopping, and restarting tasks, because if you stop the process manually, Monit will discover at the next specified interval that the process is not running and will start it back up. This behavior is sometimes exactly what you do not want to happen if you are having issues on your production server and need to stop a process.
As with the Nginx init script previously, this script is specific to CentOS. You can find one specific to your OS online.
#! /bin/sh # # monit Monitor Unix systems # # Author: Clinton Work, <[email protected]> # # chkconfig: 2345 98 02 # description: Monit is a utility for managing and monitoring processes, # files, directories and devices on a Unix system. # processname: monit # pidfile: /var/run/monit.pid # config: /etc/mcommons/monitrc # Source function library. . /etc/rc.d/init.d/functions # Source networking configuration. . /etc/sysconfig/network MONIT=/usr/local/bin/monit CONFIG=/etc/monitrc # Source monit configuration. if [ -f /etc/sysconfig/monit ] ; then . /etc/sysconfig/monit fi [ -f $MONIT ] || exit 0 RETVAL=0 # See how we were called. case "$1" in start) echo -n "Starting monit: " daemon $NICELEVEL $MONIT -c $CONFIG RETVAL=$? echo [ $RETVAL = 0 ] && touch /var/lock/subsys/monit ;; stop) echo -n "Stopping monit: " killproc monit RETVAL=$? echo [ $RETVAL = 0 ] && rm -f /var/lock/subsys/monit ;; restart) $0 stop $0 start RETVAL=$? ;; condrestart) [ -e /var/lock/subsys/monit ] && $0 restart ;; status) status monit RETVAL=$? ;; *) echo "Usage: $0 {start|stop|restart|condrestart|status}" exit 1 esac exit $RETVAL
Now that you have your server running, you should be ready to try deploying to it. We are going to defer to Chapter 21 for instructions in setting up your application for deployments. You can see from the directory structure we’ve referred to that we are expecting to run the application from the base directory of /var/www/apps/railsway, and that we expect you to use monit to control your process management.
You should be aware of where the logs will be stored for your application, for the Mongrels, for Nginx, for Monit, and so on. These are often the first place you need to look when troubleshooting. You can use curl to hit your site quickly without needing to use a browser.
The following kinds of concerns should always be factored into how you design your production environment. Each of these topics on their own can result in huge labor and hardware costs, depending on your level of priority and, in some cases, paranoia. Be aware that addressing these concerns is always a compromise of time, money, and efficiency.
What happens if the database goes down, or if a disk fails? Redundancy and failover concerns deal with your ability to react to a failure at one or more levels of software or hardware. We are not going to cover these topics here, as there are many levels of redundancy and failover at each level of your production stack.
Caching is intended to improve the performance of your system by storing the results of common requests and making them easier and faster to access than processing the entire request from scratch. Common ways to cache include storing previous HTTP responses as HTML files on disk (aka page caching), storing them in memory (e.g., Memcache), and so on. There are many ways to cache at each level of your production configuration. You can read an excellent tutorial on caching in Rails at http://www.railsenvy.com/2007/2/28/rails-caching-tutorial.
The terms “performance” and “scaling” are often used interchangeably, and although they are related, they represent different concerns and have different solutions. Performance is a measure of the behavior of a given “unit” of resources. For example, you can benchmark the performance of the login system receiving 20,000 requests for the login page within one minute, running on a single web server and database connection. Repeated benchmarks across various parts of the application stack will establish the baseline performance you can expect from a single “unit” of resources.
Scalability, on the other hand, refers to how efficiently the system architecture can grow in proportion to increasing demand. In other words, it is a measure of change in the ratio between units of demand and units of resources as the units of demand increase. For example, let’s say you have a three-server stack, with one box each for web, app, and database tiers. If that stack can handle 1,000 req/sec based on your performance measurements, how much hardware will you need to handle 2,000 req/sec? Ideally an architecture should be able to scale “horizontally” (i.e., proportionately), meaning that you could add more “units” of resources without reducing the performance of the entire system. In the preceding example, that would mean we just double the hardware, but it is not always that simple.
In practical production environments, your ability to scale easily will depend on a number of factors. Some aspects of scaling behavior depend on the overall behavior of your application. For example, write-heavy applications (i.e., those that perform a lot of database INSERTs)—such as social networks or financial transaction applications—will require more complex database cluster solutions than read-heavy applications such as blog or news aggregators. Another factor in scaling behavior is that different parts of your application may scale at different rates. Vendor or third-party integration points—such as a mail delivery vendor, an e-commerce payment gateway, or external asset storage service—will often scale at different rates than the rest of your application.
You can read more about these at http://en.wikipedia.org/wiki/Scalability. Scaling a stack is far too intricate a topic to be able to cover with any depth in this chapter, but you can find some excellent resources online by searching for presentations and blog posts that deal with it. Rails uses “share-nothing architecture” similar to its LAMP-stack predecessors, which has proven to be an effective, low-cost way to structure an application. This approach has evolved as the web industry has matured over the past decade or so, and works well for many applications. Many of the veterans of the “share-nothing architecture” earned their badges with LAMP, but you can apply most of their wisdom to Rails without losing much value. Because our production stack is simple, it will not be difficult to change into more complex configurations later.
It is largely impossible to address security here with any degree of depth, but it is probably worth noting the following simple steps to at least protect yourself from obvious attacks.
You can lock down your server and network and bury the hardware in an old missile silo, but that won’t keep you safe if your application is open for attacks. Fortunately, Rails’ tools and methods will make it easy to keep your application relatively safe out of the box, and by employing some best practices you can avoid the most common attacks, such as SQL injection and Cross-Site Scripting (XSS). You can find out more from the ROR security blog at http://www.rorsecurity.info.
You should lock down your ports so that you can only access 80, 443, and your nonstandard SSH port. If you have a firewall and can configure it yourself, this should be done at the firewall level. If you do not have access to the firewall you can lock down these ports at the server level using iptables.[6]
After you actually launch your application, you’ll want to keep tabs on it and generally keep your system clean. You’ll also want it to be pretty straightforward to hunt down problems and troubleshoot them when they happen. All kinds of odd things happen in production environments that are tough or impossible to reproduce on your local development machine.
Setting up your production environment is not as daunting an endeavor as it used to be. Developers and sysadmins tend to stick to what they know, which is understandably all too easy in our age of specialties. Rails, for its part, has helped to chip away at this tendency by encouraging developers to crawl out of their holes and play DBA, sysadmin, or “front-end designer” by simply putting them within reach. In general, the web as a production, publication, and distribution medium has vastly simplified the skills and efforts required to get ideas, products, and services out into the open. Rails and the wealth of libraries and tools that developers have created to support it have served this paradigm well by simplifying even the most mundane areas of web development such as deployments and system administration.
In this chapter we have reviewed the key technical and philosophical components of setting up a modern production environment. There are, of course, millions of possible permutations, a few of which will be most suitable for your needs. Even if you use PostgreSQL, FreeBSD, and Perforce, you should be able to adapt our recommended configuration for your own system easily. You should always seek the advice of your peers, especially if this is your first production environment setup, but you can consider the approach that we’re recommending as the generally accepted best practices, at least for Rails applications.
The big remaining step of getting your application into production is the actual configuration of your application with Capistrano, so it can be deployed to your fancy new production environment. We cover this in the next chapter.
| 1. | Mongrel project homepage: http://mongrel.rubyforge.org |
| 2. | Information about the Bash shell: http://en.wikipedia.org/wiki/Bash |
| 3. | Nginx project homepage: http://nginx.net |
| 4. | Instructions for setting up Apache to talk to a Mongrel Cluster using |
| 5. | iptables project homepage: http://www.netfilter.org/projects/iptables/index.html |
| 6. | Monit project homepage: http://www.tildeslash.com/monit/ |