daemon_controller: a library for robust daemon management
Phusion has recently released a library for robust daemon management. Check it out. Description and tutorials are available on that page.
Archive for August, 2008
Re: Strange HTTP header?
Yesterday I challenged people to look at sandbox.phusion.nl‘s HTTP headers and check whether they notice anything weird. The HTTP response header of the front page is:
HTTP/1.1 200 OK
Server: nginx/0.6.32
…
X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.1.0
X-Runtime: 0.00173
…
Wow, I got a lot more responses than I expected.
Sorry guys, there’s a reason why I didn’t post this on the Phusion blog, but on my personal blog instead. 
Chu Yeow said:
Wow Passenger on Nginx (I think that’s it – doubt you’d run Nginx on top of Apache+Passenger
).
Well actually… we are running Nginx on top of Apache+Passenger.
The first reaction of many people is probably “WTF, are you out of your mind? Why would you do such a thing?” Let me explain a little bit about our server.
Initial motivation: security
This server is shared by many users, including a few which we don’t fully trust. It not only runs Rails applications but also a bunch of PHP applications, and in the not too distant past some mod_perl applications. In the usual Apache setup, all those PHP/mod_perl applications will run under the same user and have the same rights. This means that there is no security between different people’s web applications: Jane’s PHP script can read Joe’s forum database password file. Not so nice.
Now, how do we solve this? These days, server virtualization is the latest hype: just give Joe and Joe different virtual machines! But virtualization wastes a lot of memory. Joe and Jane’s websites are really low-traffic compared to mine. The server “only” has 1 GB of RAM, and allocating a fixed amount of RAM (which must be at least 128 MB for a more or less usable server OS) is really wasteful.
Our solution was simple. Each user got his own Apache installation and runs all his web applications under his own user account. Users cannot read from and write to other users’ home folders. Each of these backend Apache installations are firewalled, and a frontend web server proxies requests to these backend Apache installations.
But the setup is of course not limited to one-Apache-per-real-user. blog.phusion.nl is running on WordPress, which doesn’t exactly have a good security track record. My personal WordPress installation had been hacked once: apparently some spam bot changed the file upload folder to /tmp and put a .exe in there. It also disabled Akismet. I wouldn’t be surprised if someone one day finds a remote shell code execution vulnerability. One really wouldn’t want to run WordPress with the same rights as all the other web applications. So we gave WordPress its own user account and Apache installation. WordPress is now completely sandboxed and cannot do any harm to the other websites.
Efficiency
Indeed, what about efficiency? We’ve been using this setup for almost 2 years now, and it’s actually running quite well. Not too long ago, this server hosted a website which got about 30 000 unique visitors per day (about 120 000 requests per day on this server; we load balanced that website over multiple web servers) and it was able to handle the load with ease. We noticed no delay in response times compared to when the website was running on the frontend web server directly. That said, we did go through several stages of optimization:
- A long long time ago, the frontend web server was Apache 1.3, which proxies requests via mod_accel. mod_accel is like mod_proxy, but you can specify a list of URI extensions that it won’t proxy. For example, you can tell mod_accel only to proxy requests that don’t end with .css, .jpg, .png, etc.
- Unfortunately Apache 1.3 was ancient and not well-supported, so we switched to Apache 2 with mod_proxy instead. mod_proxy provides no way to skip proxying certain URIs, so we had to live with this. Performance was acceptable, though the backend web servers are being hit harder than before because static asset requests are now also being proxied.
- Apache 2 proved to be too memory- and CPU-hungry for a reverse proxy, so we switched the frontend web server to Lighttpd instead. This reduced our CPU- and memory usage dramatically. We configured Lighttpd to serve static assets directly, so that the backend web servers are only there to serve PHP.
- Unfortunately Lighttpd leaks memory: after a few days, memory usage would jump to 200 MB. From time to time it will also “go out of control” and consume 100% CPU, although it’s still serving requests just fine. 2 days ago I finally got tired of that, and replaced Lighttpd with Nginx.
Finally, we used Apache with the worker MPM and Phusion Passenger development version (from the git repository) for hosting our Rails applications. The worker MPM, which uses a combination of threads and processes, is a lot more memory efficient than the default prefork MPM, which only use processes. This is our Apache worker MPM setup:
StartServers 1 ThreadsPerChild 10 MaxClients 10 MinSpareThreads 1 MaxSpareThreads 1 MaxRequestsPerChild 50000 ThreadStackSize 500000
This tells Apache to use only one process. That process is multi-threaded and will have 10 threads for serving requests. Furthermore, each thread will have a stack size of 500 KB. The default system stack size is usually something along the lines of 8 MB, so setting such a small stack size reduces Apache’s VM size a lot. 500 KB has proven to be sufficient for Apache.
Now, let’s compare the memory usage between Nginx and our Apache installation:
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11700 0.0 0.2 3452 2012 ?? Is 6:23AM 0:00.00 nginx: master process /usr/local/sbin/nginx www 11701 0.0 0.3 3452 2880 ?? S 6:23AM 2:28.91 nginx: worker process (nginx) www 11702 0.0 0.3 3452 2880 ?? S 6:23AM 2:47.10 nginx: worker process (nginx) app 82548 0.0 0.3 7656 3572 ?? Ss Tue03PM 0:05.79 /home/app/apache/bin/httpd -k start app 89467 0.0 0.4 10144 4632 ?? I 5:11AM 0:02.45 /home/app/apache/bin/httpd -k start
The server’s running on FreeBSD, not Linux, so we can’t measure memory usage excluding any copy-on-write savings (i.e. the private dirty RSS). But let’s compare the total Resident Set Sizes (RSS):
- Nginx: 7772 KB (7.6 MB)
- Apache: 8204 KB (8.0 MB)
Not a big difference.
Apache’s slowness and Nginx’s performance, both overrated?
People commented:
Pretty fast runtime?
X-Runtime: 0.00171
and
Damn!! It is impossible!
0.00173 per request?!
So the Rails application is running in Apache and behind an Nginx reverse proxy, and it’s still fast.
FastCGI
Lighttpd and Nginx both support PHP via FastCGI, so why didn’t we use that instead? The answer is ease of use. Setting up a PHP-FastCGI process pool for every user is quite a hassle. Plus, the user might be running CGI or mod_perl applications as well. Giving each user his own Apache installation is by far the easiest way. Apache also supports .htaccess, which Lighttpd and Nginx don’t support. WordPress’s URI rewriting feature writes mod_rewrite rules to .htaccess. Configuring the same rules in Lighttpd was a total pain, and I wouldn’t want to do that again.
Conclusion
I believe that all the fuss about web server performance is usually overrated. As we can see, Apache can be memory-efficient. Running Rails applications on Phusion Passenger behind an Nginx reverse proxy is viable. You just need to know how to tweak and mix-and-match the two.
What we’re doing is not very unlike proxying to a Mongrel cluster from Nginx. Instead of proxying to a Mongrel cluster, we proxy to Apache. This still makes Rails deployment a lot easier because Phusion Passenger will take care of managing the Rails processes for me. The only redundant thing that I have to do now is having to setup 2 virtual host definitions: one in Apache and one in Nginx.
Morale of the story: it’s all HTTP, you can proxy everything in any way you want. Some people on the Phusion Passenger mailing list asked how to horizontally scale Phusion Passenger. The answer is: the same way you’re used to when you were using Mongrel clusters.
This also shows that it is possible to run multiple Apache installations on the same server. It’s only a matter of specifying different configuration files for each installation. It seems that a lot of people aren’t aware of that. In a recent Google talk about Rails scalability, a speaker claimed that there is a limit to the amount of hardware resources that Apache can utilize. He said that if you have 16 cores and 20 GB of RAM, one Apache instance cannot utilize all those resources, and that in order to make full use of your hardware, one must virtualize. But why? It’s easier and more efficient to run multiple Apache instances on the same machine.
By the way, we use the following Nginx config snippet for Phusion Passenger-powered hosts:
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_redirect http://localhost/ $scheme://$http_host/;
server {
listen 80;
server_name sandbox.phusion.nl;
root /u/apps/sandbox/current/public;
location / {
proxy_redirect http://localhost:1234/ $scheme://$http_host/;
if (!-f $request_filename) {
proxy_pass http://localhost:1234;
break;
}
if ($request_method != GET) {
proxy_pass http://localhost:1234;
break;
}
}
}
This forwards all non-static-asset requests to Apache. Static assets are served directly by Nginx.
Strange HTTP header?
I challenge you to type the following command:
curl -i http://sandbox.phusion.nl/ | head
This shows the HTTP output of http://sandbox.phusion.nl/ (which is, unsurprisingly, a Rails app), including HTTP headers.
Do you notice anything strange about this HTTP header? (I’m not going to comment for a few days. I’ll let you guys speculate.)
Making Ruby’s garbage collector copy-on-write friendly, part 8
Hi folks, it has been a while since the last “Making Ruby’s garbage collector copy-on-write friendly” post. Many things have happened in the mean time, and my copy-on-write work is now usable (and used) in production environments, but it seems that there is still confusion. So I’ve decided to write a new post which explains the situation.
Copy-on-write updates
In March I submitted my work to the Ruby core mailing list. There has been some discussion. As a result, various people, including myself, have made a number of improvements.
The improvements are as follows:
- The copy-on-write friendly garbage collector is now a few % faster thanks to various micro-optimizations.
- The mark table implementation is now pluggable.
On Windows, a copy-on-write friendly garbage collector is totally useless because fork() is not supported on Windows. Furthermore, not all Ruby applications call fork(). So I’ve made two mark table implementations: one based on the old one (which marks objects directly by setting a flag on the object) and a copy-on-write friendly one. It is now possible to change the mark table implementation during runtime by calling GC.copy_on_write_friendly = (boolean value).
This has huge performance implications. The copy-on-write friendly mark table makes the garbage collector about 0%-20% slower, depending on the application and the workload. However, the non-copy-on-write friendly mark table is enabled by default, so by default there is only a 1% performance penalty. This performance penalty comes from the fact that marking an object now requires a function call which sets the mark flag, instead of setting the mark flag directly. But I think 1% is acceptable.
- Various little bugs in the debugging code have been fixed.
- Me and Ninh are working on a scientific paper regarding the copy-on-write work.
Unfortunately the discussion stranded. Matz had some concerns about performance, which is why I made the mark table implementation pluggable. I will re-submit the patch for further evaluation when the time is right.
Ruby Enterprise Edition
Many of you have probably heard of Ruby Enterprise Edition. There has been, and still is, a lot of fuss about the name. But that’s intentional and is all part of the plan — if people make a fuss about the name then it means we’re not in the Zone of Mediocrity.
What is Ruby Enterprise Edition? People thought it’s a closed source product, but in fact the website’s front page and download page has the following huge sticker:
(We actually added this sticker after we’ve seen that people think it’s going to be closed source.)
In one sentence:
Ruby Enterprise Edition is an easy to install Ruby interpreter that includes, among other things, my copy-on-write work.
Facts and myths:
- It’s open source, not closed source. It’s freely available to all.
- It’s not an entirely new Ruby implementation. It’s based on the official Ruby interpreter (MRI), version 1.8.6-p286. This means that all your existing Ruby applications are compatible with Ruby Enterprise Edition.
- It does not only include my copy-on-write work. There’s more. Read on.
- It is not a hostile fork, but a friendly one. The work included in Ruby Enterprise Edition is meant to be merged back to upstream at some point in the future.
- The copy-on-write work has been submitted to the Ruby core team in the past.
- Phusion Passenger is very well-integrated with Ruby Enterprise Edition. If you use Phusion Passenger in combination with Ruby Enterprise Edition, then your Rails applications will transparently use 33% less memory and will be faster, as if it’s magic. You don’t need to do anything special, it just works.
The only condition is that you must not be using conservative spawning in your application. But if you don’t know what conservative spawning is then you’re not using it, and you’ll have nothing to worry about.
Why was Ruby Enterprise Edition made?
Consider the following facts:
- My copy-on-write work can potentially save a lot of memory in Rails applications.
- The patch has been submitted to upstream, but hasn’t been accepted yet.
- There is a demand for lower memory usage in Rails applications, right now, not X months/years in the future.
Given the circumstances, and to satisfy the demands (including that of ourselves), we have decided that it would be best to maintain our own Ruby fork which includes these patches.
You might be wondering: Why not just release the patch? Why create a fork?
The answer is user friendliness. Telling people to download Ruby’s source code and apply a patch is not user friendly. In fact, to many people, it’s downright scary. Imagine that you want a transparent and easy way to make your Rails applications “magically” use 33% less memory. Which of the following instructions would you prefer?
Use Phusion Passenger to deploy your application. Then download the Ruby interpreter source code from www.ruby-lang.org. Download it and extract the tarball. Then, download this patch and apply it with this and that command. Then, run ‘./configure –prefix=/somewhere’. Make sure that /somewhere is not /usr in order to prevent overwriting your old Ruby installation, you don’t want that to happen. Then type ‘make’, and then ‘sudo make install’. Then download RubyGems, extract it, and type ‘sudo /somewhere/bin/ruby setup.rb’ in the RubyGems source folder. Then type ‘/somewhere/bin/gem install rails’ to install Ruby on Rails and whatever other gems you might need.
or:
Use Phusion Passenger to deploy your application. Then download Ruby Enterprise Edition. Run the installer and follow the instructions. Done.
The first one contains a lot of caveats. Many many things can go wrong. Many many people aren’t experienced in installing Ruby from source. It’s just easier if there’s a vendor that takes care of everything for you. And we are that vendor.
We want Phusion Passenger and everything surrounding it to have a “just works” experience.
So if it’s not just the copy-on-write work, then what else does Ruby Enterprise Edition include?
- This one is huge: by using Google’s tcmalloc, an alternative memory allocator, Ruby becomes 20% faster even with the copy-on-write friendly garbage collector! Furthermore, tcmalloc seems to be more copy-on-write friendly than ptmalloc2, Linux’s default memory allocator, so by using tcmalloc we can save even more memory!
We discovered this shortly after submitting the patch to the Ruby core mailing list. So Ruby Enterprise Edition also includes tcmalloc. - Ruby Enterprise Edition includes an easy-to-use installer which takes care of installing tcmalloc, Ruby, RubyGems and important/useful gems for you. It also teaches you how to tell Phusion Passenger to use Ruby Enterprise Edition instead of normal Ruby.
- In the future we might include more patches that might be useful in production environments.
Who’s already using Ruby Enterprise Edition?
I’m not sure because we’ve never asked our users. But the Ruby on Rails Wiki is running on it, and it has been great. I’ve been monitoring the Wiki for a while now, and ever since we’ve switched it to Phusion Passenger + Ruby Enterprise Edition, it has been rock-solid (before, it used to crash often). We also observed a great reduction in memory usage.
Michael Koziarski, a Rails core developer, runs Phusion Passenger with Ruby Enterprise Edition on his blog. He said that he downgraded his server because Phusion Passenger + Ruby Enterprise Edition saved him so much memory.
Final words
I hope this post has shed some light on matters. I’m just a little surprised that there’s all this confusion going on because all of this is also documented on the Ruby Enterprise Edition website’s FAQ. eustaquiorangel.com recently interviewed me and asked similar questions. You should check it out.
I’m also a little surprised that people seem to be reluctant about installing Ruby Enterprise Edition. If I have the choice between two products A and B, and B is the same as A but is much more efficient and is easy to install, then I’d choose B.
It is that people are suspicious about our claims? We’ve published a performance and memory usage comparison. Anybody can read this comparison, perform it himself, and check whether our claims are true. Everything we claim is verifiable so I don’t understand what there is to be suspicious about.
Please feel free to post your thoughts on this, I’d really like to hear what people have to say.
Phusion Passenger’s development cost according to Ohloh.org
From http://www.ohloh.net/projects/passenger:
Cool, we’re supposed to be millionaires!
FYI that’s is not entirely correct. Ohloh counts the vendorized Boost code base as well. Those who are interested in similar statistics, but without counting Boost, should type “rake sloccount” in the Phusion Passenger source tree (requires sloccount). Here’s the output:
SLOC Directory SLOC-by-Language (Sorted) 5357 top_dir ruby=4261,cpp=928,ansic=168 4494 apache2 cpp=4381,ansic=113 810 oxt cpp=810 479 railz ruby=479 194 wsgi python=133,ruby=61 137 rack ruby=137 0 templates (none) Totals grouped by language (dominant language first): cpp: 6119 (53.34%) ruby: 4938 (43.05%) ansic: 281 (2.45%) python: 133 (1.16%) Total Physical Source Lines of Code (SLOC) = 11,471 Development Effort Estimate, Person-Years (Person-Months) = 2.59 (31.10) (Basic COCOMO model, Person-Months = 2.4 * (KSLOC**1.05)) Schedule Estimate, Years (Months) = 0.77 (9.23) (Basic COCOMO model, Months = 2.5 * (person-months**0.38)) Estimated Average Number of Developers (Effort/Schedule) = 3.37 Total Estimated Cost to Develop = $ 350,125 (average salary = $56,286/year, overhead = 2.40). SLOCCount, Copyright (C) 2001-2004 David A. Wheeler SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL. SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to redistribute it under certain conditions as specified by the GNU GPL license; see the documentation for details. Please credit this data as "generated using David A. Wheeler's 'SLOCCount'."