In my previous blog entries (here and here), I blogged about using fork() and copy-on-write semantics to reduce memory usage in Ruby on Rails.
Kevin requested that I post the latest version of my prefork script, so here it is. For the time being, I call it “RailsFork”. The code has been cleaned up a lot. This version kills all child processes when you kill RailsFork with Ctrl-C.
Linux has supported antialiased fonts for quite some years now, but the default (antialiased) terminal fonts have always bothered me. By default, GNOME Terminal 2.x uses an antialiased Monospace or Fixed font:
I antialiasing makes terminals looks ugly. I much prefer the font as used by xterm, or GNOME Terminal 1.x. Back when Linux didn’t have good support for antialiased fonts, GNOME Terminal 1.x used a non-antialiased (bitmap) version of the Fixed font.
Luckily, I can use bitmap fonts in all fontconfig-enabled applications (which automatically includes all GTK and Qt applications). Although the “Fixed” font is antialiased now, the “MiscFixed” font is not - in fact it’s exactly the same as the old non-antialiased “Fixed” font. After using the MiscFixed font, by terminal now looks like this:
Not only does this look nicer, it also makes GNOME Terminal faster. 
It’s quite a hassle to find this font. I don’t remember where I got this font from. So for my convenience, as well as for the convenience of the reader, I have attached the font to this blog post:
MiscFixed font - download here
Extract the contents to ~/.fonts, restart your terminal, and select “MiscFixed” as font in your terminal.
About two months ago I switched from Fedora 4 to Ubuntu 6.10. My scanner, an Epson Perfection 2480 Photo, worked fine on Fedora 4. Today I tried to scan something and I found out that it wouldn’t work on Ubuntu for some reason. usb-find-scanner could find my scanner just fine. scanimage -L also lists my scanner. But whenever I start xsane, I get the message
open of device snapscan:libusb:002:007 failed: Invalid argument
I already installed Epson Kowa’s iscan driver from source and it still wouldn’t work.
After some searching I found this site. I followed the instructions: I downloaded the iscan RPMs and converted them to DEB with alien. After installing the DEBs the scanner works! 
I had to remove libsane-extras though. That conflicts with the generated DEB files.
[EDIT] I also had to edit /etc/sane.d/dll.conf, and add ‘epkowa’ as entry.
Every framework has its own insanities, and Ruby on Rails (and Ruby!) is no different.
I’m writing an application which handles image uploads. Rail’s API documentation doesn’t mention anywhere what the type of an uploaded file parameter is. The book “Rails Solutions” doesn’t mention it either, but it seems to have the same interface as the IO class. At first, I thought that the type is Tempfile (I found that out with some print statements). But that’s not entirely true:
Insanity #1: For some reason, Rails doesn’t unlink the Tempfile object after the HTTP request has been processed, so unless I unlink it manually the file will stay on the hard disk!
Insanity #2: Although StringIO and Tempfile have the same methods as the IO class, they aren’t derived from IO, or even have IO as mixin! As a result, is_a?(IO) will return false.
I spent an hour debugging my application because of this. RMagick accepts either a filename or an IO object to read the image from. The problem is, RMagick doesn’t recognize the variable for the uploaded file as an IO object. So I had to write code for 2 cases: if the uploaded file is a Tempfile, pass the tempfile’s path to RMagick. If not, create a new Tempfile, write the uploaded file’s data to that tempfile, and pass the tempfile’s path name to RMagick.
Insanity #3: According to Rails Solutions, the uploaded file has a method ‘content_type’, which isn’t part of Tempfile or part of StringIO. Again, this isn’t documented in the Rails API documentation (ActionController::TestUploadedFile mentions the method, but it’s not exactly an intuitive place to look in). Rails seems to magically add this method to the StringIO and Tempfile classes.
I do hope that they improve the API docs.
In my previous blog entries (here and here), I blogged about using fork() and copy-on-write semantics to reduce memory usage in Ruby on Rails.
I was wondering whether I’ve preloaded enough Ruby libraries. If I didn’t, then each child process will load its own copy of the required libraries, and that memory will not be shared between the child processes. Unfortunately, Ruby doesn’t seem to have a variable somewhere which lists all loaded Ruby files in the interpreter. So I hacked Ruby, by inserting the following code in ruby.c, function load_file(fname, script):
static void
load_file(fname, script)
const char *fname;
int script;
{
extern VALUE rb_stdin;
VALUE f;
int line_start = 1;
...
}
fprintf(stderr, “RUBY-LOAD: %s\n”, fname);
if (script) {
This will make the Ruby interpreter print all loaded files to stderr.
I compared the difference between the two files. I found out that only dispatch.fcgi had to be loaded by the child processes! Great.
I then launched a web browser and visited a page in my application. I compared the files again. This time I found out that each child process had to individually load the source code for the Rails application itself.
The prefork script doesn’t preload the source files for the Rails application itself. It should - this will save us a few hundred kilobytes to a few megabytes of memory, depending on the size of the application.
Based on this finding, as well as the results of my initial research on memory usage in Rails, I can conclude that most of the memory in a Rails application is spent on storing the Ruby on Rails (and dependencies) code, and that only a small fraction is spent on storing application code.
In other news, I cleaned up and restructured the prefork script. It can now gracefully terminate child processes whenever it receives a termination signal. It also performs more error checking. I’m going to do more testing. If this technique turns out to be reliable I’ll publish the final version of the script to the public.
In my previous blog entry, I blogged about using fork() and copy-on-write semantics to reduce memory usage in Ruby on Rails. Saimon Moore suggested that I should contact Zed Shaw, author of Mongrel. I asked him on his opinion and potential problems. Unfortunately I don’t have permission to quote him, so I’ll just summarize the issues (with preforking in Rails) and my own findings.
It is said that Ruby leaks I/O handles when it forks. I really don’t know how in the world that is possible - when the child exits all of its resources are freed, there is no way for it to leak anything unless the parent process forgets to clean up something that it created before forking.
I wrote a script to test this:
require 'socket'
serv = TCPServer.new(2202)
puts "*** File descriptors in parent process:"
system("ls --color -l /proc/#{Process.pid}/fd")
pid = fork do
serv = TCPServer.new(2203)
puts "*** File descriptors in child process:"
system("ls --color -l /proc/#{Process.pid}/fd")
exit
end
Process.waitpid(pid)
puts "*** File descriptors in parent process:"
system("ls --color -l /proc/#{Process.pid}/fd")
The script creates a TCP server socket, then lists the process’s file descriptors. It then forks, creates another TCP server socket, and lists the child process’s file descriptors. The parent process waits for the child, closes its own server socket, then lists its file descriptors again. The output is:
*** File descriptors in parent process: total 4 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 0 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 1 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 2 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 3 -> socket:[2300615] *** File descriptors in child process: total 5 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 0 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 1 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 2 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 3 -> socket:[2300615] lrwx------ 1 hongli hongli 64 2007-04-05 21:48 4 -> socket:[2300628] *** File descriptors in parent process: total 3 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 0 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 1 -> /dev/pts/0 lrwx------ 1 hongli hongli 64 2007-04-05 21:48 2 -> /dev/pts/0
Conclusion: Everything looks perfectly normal to me. I have no idea what “leaking IO handles” means.
It is said that preforking will cause issues with database reconnections. I gave it a try.
Conclusion: I have no idea what database reconnection issues people are talking about. I can’t find any.
I don’t use SQLite, and don’t plan on using them any time soon, so I didn’t test this. I use SQLSessionStore for storing session data in MySQL, so pstore issues don’t affect me directly. Pstore is the default session storage in Rails.
Pstore stores session data in files. Imagine two HTTP clients, with the same session ID, accessing two different Rails processes. Both Rails processes write session data to disk. What happens? Will the pstore session file be corrupted? Zed said that even Mongrel (without preforking) has problems with pstore sharing, so it’s possible that Rails doesn’t lock the pstore session file.
I tested my own Rails application, which uses SQLSessionStore:
def read if session[:rand].nil? render :text => "No random number set." else render :text => session[:rand] end end def write session[:rand] = rand read end
The write method generates a random number and saves it in the session. The read method reads the last saved number.
Conclusion: I don’t know whether pstore has problems, but SQLSessionStore seems to work fine. It’s a good idea to use SQLSessionStore anyway, as pstore slows down when you have a lot of sessions, and SQLSessionStore makes it easy to wipe idle session data.
According to this page, Ruby’s mark-and-sweep garbage collection makes all memory pages dirty, causing almost the entire child process’s to be copied. In my previous blog, I ran httperf to test preforked Rails. Rails creates a new ActionController object every time a HTTP request comes in, so using httperf will definitely activate garbage collection. Yet the memory usage didn’t increase as much as the page predicted it would.
I have a Perl application which uses about 35 MB of memory. 25 MB of that is spent on storing the parsed Perl optree, and only 10 MB is spent on storing runtime data. I suspect that Ruby is similar: most of the memory is spent on storing Rails code, not variable data. Code is probably never garbage collected (why would it be? in a dynamic language one cannot predict whether a function will be used in the future) so the garbage collector probably wouldn’t mark the pages containing Ruby opcodes as dirty. This explains why memory usage doesn’t go up a lot, after having made some HTTP requests.
Conclusion: I can’t find the problem. Nothing to worry about.
I couldn’t find any large problems that were relevant to me. In the future I will test this preforking technique on a busy (non-commercial) website to see how well it works.
Some of you may have heard of Ruby on Rails. It is a web development framework, based on the Ruby language. Last year I developed a Rails web application. I like Ruby on Rails a lot, and I strongly prefer it over PHP for serious web applications. With Rails I was able to develop a robust, stable and maintainable web application in a much shorter time than I could have when I used PHP.
Unfortunately, Rails seems to use a bit more resources than PHP, namely:
I’m not really concerned about CPU usage - my server has enough CPU to handle the load, but memory usage is more problematic. My web server has “only” 1 GB of RAM. It runs quite a lot of services so it’s a bit short on RAM.
But first, let us take a look at how Rails interfaces with a HTTP client (your web browser). My usual setup is as follows:
This setup is illustrated in the following picture:
(SVG version)
There are, of course, different setups. Mongrel - a web server designed to run Ruby on Rails - seems to be becoming more and more popular. Unlike Lighttpd which uses FastCGI to spawn several Rails worker processes, Mongrel embeds Rails directly. Rails is not thread safe, so Mongrel can only process one request at a time (unlike Lighttpd which can proxy requests to one of the many worker processes). So what people generally do is to launch several Mongrel processes, and use Apache 2.2 with mod_proxy_balancer to proxy requests to one of the Mongrel processes.
I have no idea why it’s better than using Lighttpd with FastCGI. I heard that Lighttpd’s mod_proxy module was unstable, but I don’t know whether that’s still the case. My web server runs Lighttpd 1.4 and uses mod_proxy to proxy requests to some backend web servers, and I’ve never had problems with it.
A process which embeds Ruby on Rails (that is, either a Rails FastCGI process or a Mongrel process) uses between 20 MB to 30 MB. It is usually a good idea to launch more than one FastCGI process (or Mongrel process) so that your web server can process more than 1 requests concurrently. But the memory usage quickly adds up. If you load 4 Rails processes then you’re already using about 100 MB of memory. My web server has “only” 1 GB of RAM (it runs quite a lot of services so it’s a bit short on RAM). So I’ve been looking for ways to reduce Rails memory usage.
Luckily, there is a way, and it’s called fork and copy on write.
As you might know, processes’ memory are isolated. That is, one process cannot read or write another process’s memory. On modern Unix operating systems, when a parent process forks a child process, almost all of the memory between the parent and child process is shared. So if your 200 MB bloated application forks a child process, the child process actually only uses a few kilobytes. Only when either the parent process or the child process writes to a piece of memory, that piece of memory is copied, so that the parent process’s memory changes won’t affect the child (and vice versa). This is why it’s called “copy on write”.
We can use this simple fact to save memory. For example, mod_perl uses this to reduce web server Perl scripts’ startup time and memory usage. mod_perl loads all required Perl modules in advance. When the web server forks a new child process to process a HTTP requests, the memory used by the already loaded Perl modules will be shared between the parent and the child process. Because most (or all) of the modules are already loaded, Perl will not load them again, thus significantly reducing loading time.
This technique has only one disadvantage: it doesn’t work on Windows! Windows has no fork() system call.
We can use the same technique on Ruby on Rails. Though I find it strange that not many Rails users seem to care - they happily spawn multiple Mongrel processes but I haven’t seen many people asking why Mongrel doesn’t use fork() to save memory. Kirk Haines said that it’s because ActiveRecord doesn’t like fork (the connection to the database seems to break after a fork). But I don’t know why Mongrel doesn’t just fork before any requests are processed.
I decided to measure Rails’s memory usage. My Rails environment was as follows:
config.cache_classes = true config.whiny_nils = true config.breakpoint_server = false config.action_controller.consider_all_requests_local = true config.action_controller.perform_caching = false config.action_view.cache_template_extensions = false config.action_view.debug_rjs = false
This was my memory usage before launching Rails:
total used free shared buffers cached Mem: 1011 365 645 0 1 97 -/+ buffers/cache: 267 744 Swap: 996 0 996
I then proceeded to launch 10 independent Rails FastCGI processes, which do not share memory with each other (other than the memory used by the Ruby interpreter itself). After the launch, the memory usage was as follows:
total used free shared buffers cached Mem: 1011 537 474 0 1 97 -/+ buffers/cache: 438 573 Swap: 996 0 996
The 10 processes used 171 MB in total, or 17.1 MB per process. But we’re not there yet: memory usage is likely to increase when I make a HTTP request. So I used the httperf tool and ran httperf --uri /rails/ --port 3501 --num-conns 100 --rate 20
This was the memory usage after httperf was done:
total used free shared buffers cached Mem: 1011 551 459 0 1 97 -/+ buffers/cache: 452 558 Swap: 996 0 996
Memory usage was increased by 10 MB. So that’s 1.5 MB extra memory per process, and 186 MB in total for all 10 processes.
I wrote a script which loads all the Ruby on Rails library. It will then fork and spawn x Rails FastCGI processes, where x is a number which you can configure. Each FastCGI process will have its own Unix socket for communication with the web server.
This was my memory usage before I launched the script:
total used free shared buffers cached Mem: 1011 334 676 0 2 95 -/+ buffers/cache: 236 775 Swap: 996 0 996
After instructing the script to launch 10 processes, memory usage become:
total used free shared buffers cached Mem: 1011 364 646 0 3 95 -/+ buffers/cache: 265 745 Swap: 996 0 996
Nice! All 10 processes only used 30 MB in total, which is a far cry from the previously measured 171 MB. Now, let us see what happens after we run httperf:
total used free shared buffers cached Mem: 1011 381 629 0 3 96 -/+ buffers/cache: 282 729 Swap: 996 0 996
Memory usage has gone up by 17 MB. That’s 1.7 MB extra memory per process.
By using preforking I was able to reduce memory usage for 10 Rails processes from 186 to 47 MB. That’s a memory saving of 75%! The Rails application seems to work fine. So far I haven’t been able to detect any strange behavior.
Without preforking, the memory usage, in MB, follows this formula:
memusage(n) = 18.6 * n
…where n is the number of Rails processes. With preforking, the memory usage is:
memusage(n) = 30 + 1.7 * n
The graph looks like this:
The script can be downloaded here. Put it in your ’scripts’ folder. Launch the script as follows:
./script/fork.rb NUM
…where NUM is the number of Rails processes you want to launch. Each process will be given its own Unix socket, named ‘log/fastcgi.socket-x’, where x is the process’s sequence number (which starts from 0). So if you launch 3 processes, the following Unix sockets will be created:
log/fastcgi.socket-0
log/fastcgi.socket-1
log/fastcgi.socket-2
You must also setup Lighttpd to communicate with Rails through the sockets. I use this configuration:
fastcgi.server = (
".fcgi" => (
("socket" => "/path-to-your-rails-root-folder/log/fastcgi.socket-0″),
(”socket” => “/path-to-your-rails-root-folder/log/fastcgi.socket-1″),
(”socket” => “/path-to-your-rails-root-folder/log/fastcgi.socket-2″)
)
)