Cycle Long-Running Instances
Let’s first look at how our program runs, and decide what we can do to make it run faster.
Imagine you started a program. Let’s say it’s a web browser, and after some time it’s become slow. What do you do? It’s not a trick question. You know what you should do: restart it, and it’ll be fast again.
Can we apply the same principle with Ruby programs? It turns out we can. Any long-running Ruby application will become faster after the restart.
Back in Chapter 6, Profile Memory we talked about how garbage collector performance declines when the amount of memory allocated by the Ruby process increases. Spoiler alert: I’m going to jump ahead of myself and add to this one more fact. In most cases the Ruby process will never give the memory allocated for Ruby objects back to the operating system. Take a peek at the next chapter if you’re curious what happens, and why.
For now, let’s concentrate only on these two facts: (a) the performance declines with the increased memory usage, and (b) the amount of memory allocated by the Ruby process only grows with time.
What does this add up to? Slowdown. The longer our Ruby program runs, the slower it gets. No amount of code optimization can prevent this. Only a restart solves this slowdown!
So if you have a long-running Ruby instance, you’ll need to cycle it. And by cycling I mean restarting it when it uses too much memory.
You can cycle Ruby applications in several ways:
-
Use a hosting platform that does it for you. For example, Heroku cycles its “dynos” daily[27] and also aborts the process when it exceeds the dyno’s memory limit.
-
Use a process management tool, like monit,[28] god,[29] upstart,[30] runit,[31] or foreman with systemd.[32]
-
Ask the operating system to kill your application if it exceeds the memory limit. Note that this still depends on a process management tool to restart the application after it gets killed.
If you deploy on Heroku, its daily cycling might work well for you. But when I worked on Acunote, we had our own deployment infrastructure and dedicated servers. So we had to use the other techniques to combat excessive memory usage. I’ll show a few examples of how we did it.
Example: Configure Monit to Cycle Ruby Processes
With monit we can check totalmem and loadavg variables and restart based on their values, like this:
| | check process my_ruby_process |
| | with pidfile /var/run/my_ruby_process/my_ruby_process.pid |
| | start program = "my_ruby_process start" |
| | stop program = "my_ruby_process stop" |
| | # eating up memory? |
| | if totalmem is greater than 300.0 MB for 3 cycles then restart |
| | # bad, bad, bad |
| | if loadavg(5min) greater than 10 for 8 cycles then restart |
| | # something is wrong, call the sys-admin |
| | if 20 restarts within 20 cycles then timeout |
Example: Set Operating System Memory Limit
On Unix systems we can use the setrlimit system call to enforce the process memory limit.
For example, on Linux and Mac OS X, set RLIMIT_AS:
| | # 600 MB RSS limit |
| | Process.setrlimit(Process::RLIMIT_AS, 600 * 1024 * 1024) |
Example: Cycle Unicorn Workers in the Rails Application
Rails applications running on the Unicorn web server can cycle themselves without any external process management tool. The idea is to set a memory limit for workers and let the master process restart them once they get killed.
For example, this is what I have in my config/unicorn.rb:
| | after_fork do |server, worker| |
| | worker.set_memory_limits |
| | end |
| | |
| | class Unicorn::Worker |
| | MEMORY_LIMIT = 600 #MB |
| | |
| | def set_memory_limits |
| | Process.setrlimit(Process::RLIMIT_AS, MEMORY_LIMIT * 1024 * 1024) |
| | end |
| | end |
That works for Unicorn 4.4, so you might need to change it to work with newer versions.
Modern Rails applications take from 100 MB to 200 MB after startup. So I wouldn’t set this limit lower than 250 MB. And you probably want to go higher. How high? To get a good approximation, take the amount of memory available to Rails and divide by the number of Unicorn workers.
The only problem with this approach is that the operating system may kill your worker while serving the request. To avoid that, I also set up what I call the kind memory limit.
We can set this limit to a value lower than the RSS memory limit, and check for it after every request. Once the worker reaches the kind memory limit, it gracefully shuts itself down.
This way, in most cases workers quit before reaching the RSS memory limit enforced by the operating system. That becomes a safeguard only against long-running requests that grow too big in memory.
Here’s how I set up the kind limit with Unicorn:
| | class Unicorn::HttpServer |
| | KIND_MEMORY_LIMIT = 250 #MB |
| | |
| | alias process_client_orig process_client |
| | undef_method :process_client |
| | def process_client(client) |
| | process_client_orig(client) |
| | exit if get_memory_size(Process.pid) > KIND_MEMORY_LIMIT |
| | end |
| | |
| | def get_memory_size(pid) |
| | status = File.read("/proc/#{pid}/status") |
| | matches = status.match(/VmRSS:s+(d+)/) |
| | matches[1].to_i / 1024 |
| | end |
| | end |
This example will work only for Linux and other Unixes because it gets the current process memory usage from the /proc filesystem. If you’d like to port it for Mac OS or Windows, you’ll have to rewrite the get_memory_size function.
There are, of course, many other ways to keep the Ruby process from growing in memory. I can’t describe all of them in this book, but by now you should have the general idea. Whatever tool you use, make sure it restarts the long-running Ruby application before it grows too big in memory.