Saving memory in Ruby on Rails with fork() and copy-on-write

Some of you may have heard of Ruby on Rails. It is a web development framework, based on the Ruby language. Last year I developed a Rails web application. I like Ruby on Rails a lot, and I strongly prefer it over PHP for serious web applications. With Rails I was able to develop a robust, stable and maintainable web application in a much shorter time than I could have when I used PHP.

Unfortunately, Rails seems to use a bit more resources than PHP, namely:

  1. Memory usage.
  2. CPU usage.

I’m not really concerned about CPU usage – my server has enough CPU to handle the load, but memory usage is more problematic. My web server has “only” 1 GB of RAM. It runs quite a lot of services so it’s a bit short on RAM.

A little introduction on how Ruby on Rails interfaces with the HTTP client

But first, let us take a look at how Rails interfaces with a HTTP client (your web browser). My usual setup is as follows:

  1. The web browser connects to the web server software (in my case, Lighttpd).
  2. The web server launches one or more Ruby on Rails FastCGI processes. FastCGI is like CGI, but instead of launching a process every time a HTTP request is made, FastCGI keeps the process in memory, so that it is capable of processing more than one request. Note that Apache 2 doesn’t support FastCGI (its support is broken, last time I checked), so I have to use Lighttpd.
  3. The web server proxies the HTTP request to one of the Rails FastCGI processes.

This setup is illustrated in the following picture:
rails-server-architecture.png
(SVG version)

There are, of course, different setups. Mongrel – a web server designed to run Ruby on Rails – seems to be becoming more and more popular. Unlike Lighttpd which uses FastCGI to spawn several Rails worker processes, Mongrel embeds Rails directly. Rails is not thread safe, so Mongrel can only process one request at a time (unlike Lighttpd which can proxy requests to one of the many worker processes). So what people generally do is to launch several Mongrel processes, and use Apache 2.2 with mod_proxy_balancer to proxy requests to one of the Mongrel processes.

I have no idea why it’s better than using Lighttpd with FastCGI. I heard that Lighttpd’s mod_proxy module was unstable, but I don’t know whether that’s still the case. My web server runs Lighttpd 1.4 and uses mod_proxy to proxy requests to some backend web servers, and I’ve never had problems with it.

But what about memory usage?

A process which embeds Ruby on Rails (that is, either a Rails FastCGI process or a Mongrel process) uses between 20 MB to 30 MB. It is usually a good idea to launch more than one FastCGI process (or Mongrel process) so that your web server can process more than 1 requests concurrently. But the memory usage quickly adds up. If you load 4 Rails processes then you’re already using about 100 MB of memory. My web server has “only” 1 GB of RAM (it runs quite a lot of services so it’s a bit short on RAM). So I’ve been looking for ways to reduce Rails memory usage.

Luckily, there is a way, and it’s called fork and copy on write.

As you might know, processes’ memory are isolated. That is, one process cannot read or write another process’s memory. On modern Unix operating systems, when a parent process forks a child process, almost all of the memory between the parent and child process is shared. So if your 200 MB bloated application forks a child process, the child process actually only uses a few kilobytes. Only when either the parent process or the child process writes to a piece of memory, that piece of memory is copied, so that the parent process’s memory changes won’t affect the child (and vice versa). This is why it’s called “copy on write”.
We can use this simple fact to save memory. For example, mod_perl uses this to reduce web server Perl scripts’ startup time and memory usage. mod_perl loads all required Perl modules in advance. When the web server forks a new child process to process a HTTP requests, the memory used by the already loaded Perl modules will be shared between the parent and the child process. Because most (or all) of the modules are already loaded, Perl will not load them again, thus significantly reducing loading time.
This technique has only one disadvantage: it doesn’t work on Windows! :) Windows has no fork() system call.

We can use the same technique on Ruby on Rails. Though I find it strange that not many Rails users seem to care – they happily spawn multiple Mongrel processes but I haven’t seen many people asking why Mongrel doesn’t use fork() to save memory. Kirk Haines said that it’s because ActiveRecord doesn’t like fork (the connection to the database seems to break after a fork). But I don’t know why Mongrel doesn’t just fork before any requests are processed.

How much memory does Ruby on Rails use, really?

I decided to measure Rails’s memory usage. My Rails environment was as follows:

config.cache_classes     = true
config.whiny_nils        = true
config.breakpoint_server = false
config.action_controller.consider_all_requests_local = true
config.action_controller.perform_caching             = false
config.action_view.cache_template_extensions         = false
config.action_view.debug_rjs                         = false

This was my memory usage before launching Rails:

             total       used       free     shared    buffers     cached
Mem:          1011        365        645          0          1         97
-/+ buffers/cache:        267        744
Swap:          996          0        996

I then proceeded to launch 10 independent Rails FastCGI processes, which do not share memory with each other (other than the memory used by the Ruby interpreter itself). After the launch, the memory usage was as follows:

             total       used       free     shared    buffers     cached
Mem:          1011        537        474          0          1         97
-/+ buffers/cache:        438        573
Swap:          996          0        996

The 10 processes used 171 MB in total, or 17.1 MB per process. But we’re not there yet: memory usage is likely to increase when I make a HTTP request. So I used the httperf tool and ran httperf --uri /rails/ --port 3501 --num-conns 100 --rate 20
This was the memory usage after httperf was done:

             total       used       free     shared    buffers     cached
Mem:          1011        551        459          0          1         97
-/+ buffers/cache:        452        558
Swap:          996          0        996

Memory usage was increased by 10 MB. So that’s 1.5 MB extra memory per process, and 186 MB in total for all 10 processes.

The experiment

I wrote a script which loads all the Ruby on Rails library. It will then fork and spawn x Rails FastCGI processes, where x is a number which you can configure. Each FastCGI process will have its own Unix socket for communication with the web server.

This was my memory usage before I launched the script:

             total       used       free     shared    buffers     cached
Mem:          1011        334        676          0          2         95
-/+ buffers/cache:        236        775
Swap:          996          0        996

After instructing the script to launch 10 processes, memory usage become:

             total       used       free     shared    buffers     cached
Mem:          1011        364        646          0          3         95
-/+ buffers/cache:        265        745
Swap:          996          0        996

Nice! All 10 processes only used 30 MB in total, which is a far cry from the previously measured 171 MB. Now, let us see what happens after we run httperf:

             total       used       free     shared    buffers     cached
Mem:          1011        381        629          0          3         96
-/+ buffers/cache:        282        729
Swap:          996          0        996

Memory usage has gone up by 17 MB. That’s 1.7 MB extra memory per process.

Conclusion

By using preforking I was able to reduce memory usage for 10 Rails processes from 186 to 47 MB. That’s a memory saving of 75%! The Rails application seems to work fine. So far I haven’t been able to detect any strange behavior.

Without preforking, the memory usage, in MB, follows this formula:

memusage(n) = 18.6 * n

…where n is the number of Rails processes. With preforking, the memory usage is:

memusage(n) = 30 + 1.7 * n

The graph looks like this:
Rails memory usage

Script usage

The script can be downloaded here. Put it in your ‘scripts’ folder. Launch the script as follows:

./script/fork.rb NUM

…where NUM is the number of Rails processes you want to launch. Each process will be given its own Unix socket, named ‘log/fastcgi.socket-x’, where x is the process’s sequence number (which starts from 0). So if you launch 3 processes, the following Unix sockets will be created:
log/fastcgi.socket-0
log/fastcgi.socket-1
log/fastcgi.socket-2

You must also setup Lighttpd to communicate with Rails through the sockets. I use this configuration:

fastcgi.server = (
    ".fcgi" => (
        ("socket" => "/path-to-your-rails-root-folder/log/fastcgi.socket-0"),
        ("socket" => "/path-to-your-rails-root-folder/log/fastcgi.socket-1"),
        ("socket" => "/path-to-your-rails-root-folder/log/fastcgi.socket-2")
    )
)

25 Comments »

  1. Mike Hearn said,

    April 5, 2007 @ 12:57 pm

    That’s pretty cool. I wonder if you could make it even better by fixing ActiveRecord to survive a fork. You can use pthread_atfork to register a fork() handler function.

  2. Hongli said,

    April 5, 2007 @ 1:01 pm

    I’m not sure what exactly makes ActiveRecord bail out after a fork. Kirk Haines said that the following workaround works:

    fork do
        ... do your stuff ...
        exec("echo -n")
    end

    This suggests that the problem is likely to be caused by an exit handler – when the child process calls exit the exit handler does things which screws up the database connection. Ruby doesn’t seem to have a way to call _exit() to bypass exit handlers. The exec() call is an alternative way to bypass exit handlers that.

  3. Ninh Bui’s Weblog » Hongli on Ruby + Rails said,

    April 5, 2007 @ 4:01 pm

    [...] Ninhteresting! [...]

  4. Saimon Moore said,

    April 5, 2007 @ 6:03 pm

    This is very interesting indeed. I’t would be interesting to hack mongrel_cluster to check this out.
    You should get Zed (mongrel authors) opinion on this.

  5. Hongli said,

    April 5, 2007 @ 7:52 pm

    Good idea Saimon. I’ve contacted Zed, and I’ll post more information in a new blog entry soon.

  6. Kevin said,

    April 11, 2007 @ 5:51 am

    Please correct me if I’m wrong, as I’m also trying to debug some memory usage problems I’m having.

    Aren’t you reading the ‘free’ stats wrong?

    You need to subtract the change in the ‘cache’ line from the equation – in the last set, when you say 17 MB was added, 0 MB was actually added, it’s just that your machine used 17 MB to cache files on the system in memory (linux does lots of wicked cool optimizations with your ‘extra’ ram).

    Simiarly, all 10 process seem to use only 1 MB of ram then? Hmm, that doesn’t make much sense. I must be wrong? The man command for ‘free’ is horrible, not explaining that damned ‘-/+ buffers/cache’ line at all. Actually, worse yet it says it subtracts something unless the -o command is given, but both commands give the same two lines for Mem and Swap.

    Help! And thanks for your great work with the prefork system. Litespeed does something similar with their lsapi, but I’m intereseted in double-checking their work (theirs is in C), as I’m worried it’s causing me memory problems.

  7. Kevin said,

    April 11, 2007 @ 5:58 am

    I see now, finally. Honestly, some of the documentation on linux sucks.

    We’re supposed to basically ONLY look at the +/- buffers/cache line. It just takes the ‘free’ line from the line above and adds the memory used for buffers/cache back into it. And vice-versa for the ‘used’ line, it removes them from the memory ‘used’, since it’s really from the OS wierdness, and not the memory stats.

  8. Hongli said,

    April 11, 2007 @ 9:45 am

    I got the numbers by reading the ‘free’ line, not the ‘cache/buffers’ line.

    Good thing to know you’re interested in this. :) I’ll keep you informed on my progress.

  9. Kevin said,

    April 13, 2007 @ 6:15 am

    Since environment.rb calls Rails::Initializer.run, why do you call it in the fork.rb script? Did you find that it wasn’t really loading everything first?

    In fact, mind sharing why you did some of the other parts of the script as well, such as removing the breakpoint and including application.rb?

    Also, here is some code I was playing with to preload the controllers/models:
    Dir.foreach( “#{RAILS_ROOT}/app/controllers” ) {|f| $logger.d “r #{f}”; silence_warnings{require_dependency f} if f =~ /\.rb$/}
    Dir.foreach( “#{RAILS_ROOT}/app/models” ) {|f| $logger.d “r #{f}”; silence_warnings{require_dependency f} if f =~ /\.rb$/}

    I tried preloading the views too, but ran into lots of problems, and I can’t find the code for it now…

  10. Hongli said,

    April 13, 2007 @ 9:59 am

    Since environment.rb calls Rails::Initializer.run, why do you call it in the fork.rb script? Did you find that it wasn’t really loading everything first?

    Yes. Rails::Initializer.run loads the rest of the Rails framework, as well as plugins (and maybe other things).

    In fact, mind sharing why you did some of the other parts of the script as well, such as removing the breakpoint and including application.rb?

    I didn’t remove the breakpoint. This line:

    require 'breakpoint' if defined?(BREAKPOINT_SERVER_PORT)

    *loads* the breakpoint library, but only if the environment file says it should be loaded. The ‘application.rb’ requirement is to load a few more things which don’t appear to be loaded by Rails::Initializer.run.

    Also, here is some code I was playing with to preload the controllers/models:

    Great, thanks. :) Have you checked whether you’re loading the exact same filename as Rails does? Because as far as the Ruby interpreter is concerned, “foo.rb” and “/absolute-path-to/ruby.rb” are different files, and Ruby will load them twice.

    And if you’d like I can post the latest version of my prefork script.

  11. Kevin said,

    April 18, 2007 @ 5:21 pm

    I’ve been testing this some more, and from what I can tell the Rails::Initializer.run line isn’t needed since it’s called in environment.rb (at least in mine…)

    The require_depenency(‘application’) does help, as would loading any other controllers /etc. Please let me know if you’ve found different results! I was wondering if Rails::Initializer does strange stuff with configuring and not running vs. just running, but it seems (relatively) simple.

    Could you share your latest prefork script, if you’ve made many changes?

    I’ve sprinkled memory checks throughout my code with the following line, so I can better track how it’s doing:
    I’ve set $logger to Activerecord::Base.logger in my environment.rb
    $logger.d `ps h -p #{$$} -o rss -o vsz`

    forks() don’t need to do anything special to take advantage of copy-on-write, do they? Or the code to avoid having that memory artifically ‘touched’ ove time?

  12. Hongli said,

    April 18, 2007 @ 6:36 pm

    Yes, you are right, Rails::Initializer.run is called from environment.rb. I didn’t notice that.

    I’ve posted the latest version of my script in my newest blog entry.

    And yes, fork() doesn’t need anything special to take advantage of copy-on-write. The operating system automatically takes care of that. You need to be careful of what you write to memory though. Any memory you write is not shared with the parent process or other child processes.

  13. 赖洪礼的 blog » Latest Ruby on Rails prefork script said,

    April 18, 2007 @ 6:37 pm

    [...] my previous blog entries (here and here), I blogged about using fork() and copy-on-write semantics to reduce memory usage in Ruby [...]

  14. Florian Frank said,

    May 24, 2007 @ 11:11 am

    @Hongli:
    > This suggests that the problem is likely to be caused by an exit handler – when the child process calls exit the exit handler does things which screws up the database connection. Ruby doesn’t seem to have a way to call _exit() to bypass exit handlers.

    Actually you can bypass exit handlers in Ruby, if you call the exit! (with bang) method.

  15. 赖洪礼的 blog » Saving memory in Ruby on Rails with fork() - failed said,

    July 23, 2007 @ 10:50 am

    [...] been working on mongrel_light_cluster, an extension for mongrel_cluster which automatically uses copy-on-write semantics to save memory in Ruby on Rails applications. The initial measurements were exciting – one can [...]

  16. asdf said,

    September 6, 2007 @ 1:11 am

    Linux NOOB,

    try using ps to gauge your mongrel memory consumption…using free..what a joke…stupid linux NOOB

    Your blog page is 100% junk science…way to add to the lies and misinformation

  17. Hongli said,

    September 6, 2007 @ 2:50 pm

    Eh, mr “asdf”, if I’m so dumb and you’re so smart then why don’t you teach me how to gauge memory the right way, oh great l33t master?

    Secondly, why is this “junk science”? I posted the entire process in a verifiable way. Anybody can review the process and point out mistakes. Isn’t this the entire point of the scientific method?

    And why are you posting anonymously anyway? If you have constructive criticism you shouldn’t have anything to fear. You do realize that you’re undermining your credibility by posting anonymously, don’t you?

  18. Matt said,

    September 19, 2007 @ 7:36 pm

    Something like the following will prevent AR in the child from sharing the connection with the parent, so that one of the processes closing it doesn’t break the other.

    dbconfig = ActiveRecord::Base.remove_connection
    pid = fork do
    begin
    ActiveRecord::Base.establish_connection(dbconfig)
    ensure
    ActiveRecord::Base.remove_connection
    end
    end
    ActiveRecord::Base.establish_connection(dbconfig)

  19. Dan42 said,

    October 12, 2007 @ 2:19 am

    I think there’s one little problem with this picture: when the garbage collector runs, ALL memory pages are touched. In other words, copy-on-write offers memory savings only until the first run of the garbage collector. For long-running processes, there will be no memory savings (well, except for static libraries maybe). Forking does help to reduce startup time but IMHO your memory measurements are flawed.

  20. Hongli said,

    October 12, 2007 @ 10:13 am

    Yes, I discovered that later. That’s what my garbage collector blog posts are devoted on.

  21. Ben Knight said,

    December 10, 2007 @ 3:28 pm

    Are there any good solutions to this problem? I’m facing this problem currently.

    Is JRuby a good workaround to this problem since it uses threads to service multiple HTTP requests?

  22. Hongli said,

    December 10, 2007 @ 10:35 pm

    Ben Knight: Yes. Actually I’ve already solved it. Please click on the “Optimizing Rails” category for more recent posts on this subject.

    I’m not sure whether JRuby will solve the problem. Java is a memory hog on its own, and Ruby on Rails is not thread-safe so RoR on JRuby will not use threads (I think).

  23. bandsxbands said,

    March 3, 2010 @ 9:48 am

    Virtual Memory sure is becoming cheaper and cheaper and cheaper. I’m curious as to when we will eventually reach the ratio of 1c to 1 Gigabyte.I’m quietly waiting for the day when I will finally be able to afford a 20 TB hard disk, hahaha. But for now I guess I will be satisfied with having a 32 GB Micro SD Card in my R4i.(Submitted from FFBrows for R4i Nintendo DS.)

  24. Why Ruby 1.8.7 isn’t memory efficient? « Computellect said,

    September 11, 2011 @ 5:09 pm

    [...] outside the object. For more information checkout this. For a longer and better explanation, go here. Share this:TwitterFacebookLike this:LikeBe the first to like this post. Tags copy-on-write, [...]

  25. Muhamad Akbar Bin Widayat said,

    August 16, 2013 @ 4:45 am

    This article is nice. I more learn about how to manage the article because I developer rails. :)

RSS feed for comments on this post · TrackBack URI

Leave a Comment