Benchmark: Passenger (mod_rails) vs Mongrel vs Thin
Since the initial release of the teaser screencast, a lot of people have asked about Passenger’s (a.k.a. mod_rails) performance compared to Mongrel and Thin. Here’s the benchmark that you’ve all been waiting for. 
In this benchmark, we compare the following software:
System specification:
- Intel Core 2 Duo, T5300 @ 1.73GHz
- 2 GB RAM
- Ubuntu Linux 7.10
- Apache 2.2.4 (Ubuntu package)
- Ruby 1.8.6 (Ubuntu package)
Test applications:
- Typo 5.0.3. This is an out-of-the-box setup with only a hello world post. Uses semi-static caching.
- Petstore revision 360, an application that the JRuby guys use for benchmarking. At first it wouldn’t start in Passenger because of a bug Petstore: they forgot to add
require 'forwardable'in app/models/cart_items.rb. I added this line myself. - Eldorado 0.9.1. This is an out-of-the-box setup with only a single forum and a single post added.
Typo and Eldorado both use SQLite 3. Petstore uses MySQL, because for some reason Petstore performs very badly when SQLite 3 is used.
People have also recommended Lovd By Less as test application, but I couldn’t get it working (database setup failed).
The applications, when served by Passenger, are setup at the following test domains:
- http://typo.test/
- http://petstore.test/
- http://eldorado.test/
Both the Mongrel cluster and the Thin cluster are setup at http://mongrel_cluster.test/. (Only either Mongrel or Thin is running during the benchmark, not both at the same time.)
And of course, we always use the production environment.
Backend configuration:
- Default Apache configuration, as provided by Ubuntu.
- Passenger: RailsMaxPoolSize is set to 10. This means that Passenger will never spawn more than 10 Rails application instances. Note that Passenger is compiled without optimizations, so if you install Passenger yourself and benchmark it, you might find that Passenger is slightly faster.
- Mongrel cluster: a cluster of 10 servers for each test application, behind mod_proxy_balancer, using default settings:
ProxyPass / balancer://mongrel_cluster/ ProxyPassReverse / balancer://mongrel_cluster/ <Proxy balancer://mongrel_cluster> BalancerMember http://127.0.0.1:3000 BalancerMember http://127.0.0.1:3001 BalancerMember http://127.0.0.1:3002 BalancerMember http://127.0.0.1:3003 BalancerMember http://127.0.0.1:3004 BalancerMember http://127.0.0.1:3005 BalancerMember http://127.0.0.1:3006 BalancerMember http://127.0.0.1:3007 BalancerMember http://127.0.0.1:3008 BalancerMember http://127.0.0.1:3009 </Proxy>
- Thin cluster: same configuration as Mongrel cluster.
Benchmark: Mongrel clusters
First we benchmark Typo served by Mongrel clusters. Because the actual application code is loaded by Mongrel during the first request, we do not want to count the first request’s time. So we run the following command to “warm up” the Mongrels, and discard its result:
ab -n 1000 -c 100 http://mongrel_cluster.test/
Next, we test 10,000 requests with 100 concurrent users:
ab -n 10000 -c 100 http://mongrel_cluster.test/
Result:
Concurrency Level: 100 Time taken for tests: 32.628480 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 54890000 bytes HTML transferred: 51480000 bytes Requests per second: 306.48 [#/sec] (mean) Time per request: 326.285 [ms] (mean) Time per request: 3.263 [ms] (mean, across all concurrent requests) Transfer rate: 1642.83 [Kbytes/sec] received
We repeat these steps Petstore and Eldorado.
Petstore:
(for Petstore we benchmark the URI “/shop/viewCategory.shtml?category=DOGS” because it’s written in the INSTALL file)
Concurrency Level: 100 Time taken for tests: 109.786026 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 40750000 bytes HTML transferred: 37680000 bytes Requests per second: 91.09 [#/sec] (mean) Time per request: 1097.860 [ms] (mean) Time per request: 10.979 [ms] (mean, across all concurrent requests) Transfer rate: 362.47 [Kbytes/sec] received
Eldorado:
Concurrency Level: 100 Time taken for tests: 70.889388 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 41130000 bytes HTML transferred: 35530000 bytes Requests per second: 141.06 [#/sec] (mean) Time per request: 708.894 [ms] (mean) Time per request: 7.089 [ms] (mean, across all concurrent requests) Transfer rate: 566.60 [Kbytes/sec] received
Benchmark: Thin clusters
The benchmarking steps for Thin are the same.
Typo:
Concurrency Level: 100 Time taken for tests: 25.795470 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 54910000 bytes HTML transferred: 51480000 bytes Requests per second: 387.66 [#/sec] (mean) Time per request: 257.955 [ms] (mean) Time per request: 2.580 [ms] (mean, across all concurrent requests) Transfer rate: 2078.78 [Kbytes/sec] received
Petstore:
(for Petstore we benchmark the URI “/shop/viewCategory.shtml?category=DOGS” because it’s written in the INSTALL file)
Concurrency Level: 100 Time taken for tests: 96.874308 seconds Complete requests: 10000 Failed requests: 42 (Connect: 0, Length: 42, Exceptions: 0) Write errors: 0 Non-2xx responses: 42 Total transferred: 40630350 bytes HTML transferred: 37547028 bytes Requests per second: 103.23 [#/sec] (mean) Time per request: 968.743 [ms] (mean) Time per request: 9.687 [ms] (mean, across all concurrent requests) Transfer rate: 409.58 [Kbytes/sec] received
Eldorado:
Concurrency Level: 100 Time taken for tests: 71.311031 seconds Complete requests: 10000 Failed requests: 5 (Connect: 0, Length: 5, Exceptions: 0) Write errors: 0 Non-2xx responses: 5 Total transferred: 41137070 bytes HTML transferred: 35518568 bytes Requests per second: 140.23 [#/sec] (mean) Time per request: 713.110 [ms] (mean) Time per request: 7.131 [ms] (mean, across all concurrent requests) Transfer rate: 563.34 [Kbytes/sec] received
Benchmark: Passenger
The benchmarking steps for Passenger are the same.
Typo:
Concurrency Level: 100 Time taken for tests: 25.131980 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 55200000 bytes HTML transferred: 51260000 bytes Requests per second: 397.90 [#/sec] (mean) Time per request: 251.320 [ms] (mean) Time per request: 2.513 [ms] (mean, across all concurrent requests) Transfer rate: 2144.92 [Kbytes/sec] received
Petstore:
(for Petstore we benchmark the URI “/shop/viewCategory.shtml?category=DOGS” because it’s written in the INSTALL file)
Concurrency Level: 100 Time taken for tests: 104.428742 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 40300000 bytes HTML transferred: 36700000 bytes Requests per second: 95.76 [#/sec] (mean) Time per request: 1044.287 [ms] (mean) Time per request: 10.443 [ms] (mean, across all concurrent requests) Transfer rate: 376.86 [Kbytes/sec] received
Eldorado:
Concurrency Level: 100 Time taken for tests: 72.587018 seconds Complete requests: 10000 Failed requests: 0 Write errors: 0 Total transferred: 41660000 bytes HTML transferred: 35530000 bytes Requests per second: 137.77 [#/sec] (mean) Time per request: 725.870 [ms] (mean) Time per request: 7.259 [ms] (mean, across all concurrent requests) Transfer rate: 560.47 [Kbytes/sec] received
Summary
Here’s a nice summary of the requests per second, in a table and in a graph:
| Typo | Petstore | Eldorado | |
|---|---|---|---|
| Mongrel | 306.48 | 91.09 | 141.06 |
| Thin | 387.66 | 103.23 | 140.23 |
| Passenger | 397.90 | 95.60 | 139.27 |
Conclusion
I was pleasently surprised by the results. During Passenger’s development, our performance baseline was Mongrel behind mod_proxy. According to earlier (simple and naive) tests, Passenger performed similar to Mongrel. I expected Thin to be a bit faster than Passenger. These new benchmarks however suggest Passenger is faster than Mongrel, and is on par with Thin. 
But you know what they say: there are lies, damn lies, and statistics. Your hardware and software is different, so if you benchmark it yourself, you’re likely to get (slightly?) different results.
labria said,
March 31, 2008 @ 3:29 pm
C’mon guys! If you’ll get performance like mongrel but with many times less the pain installing the whole thing, you’ll already be the heroes of the community =)
Gerjan Stokkink said,
March 31, 2008 @ 3:48 pm
Keep up the good work, can’t wait to try this out myself
Steven Hilton said,
March 31, 2008 @ 4:18 pm
Kudos!
Do you have any info on memory requirements?
Say, completely hypothetically, I have a small VPS that’s running a single rails app. And, hypothetically, it’s configured to run three mongrels behind an apache2 with mod_proxy_balancer. Since the framework and app code is loaded on startup, does it exist in memory only once (via copy-on-write)? And therefore, hypothetically, I’ll save on memory and be able serve more requests in my memory-constrained environment. Hypothetically.
Also, has anyone tested passenger running multiple rails apps/vhosts in a single apache? Does a rails app in vhost 1 and a rails app in vhost 2 share any resources?
Frédéric de Villamil said,
March 31, 2008 @ 5:27 pm
Very interesting benchmark, and really thank you for choosing Typo as a guinea pig. Thos results somehow show that our refactoring towards performances was the ood choice.
Phill Kenoyer said,
March 31, 2008 @ 5:45 pm
Yeah, but how does it do against Nginx + Mongrel? I have found it to be way easier and lighter to setup Nginx than Apache. But if Apache give me more performance then…
Mike Judge said,
March 31, 2008 @ 5:46 pm
10 Mongrel instances is way too many. You’ll see its metrics improve if you reduce the number. In my experience, the sweet spot is usually [CPUs + 1] instances (e.g., Dual Intel processor, run 3 mongrels.)
Aleksandr said,
March 31, 2008 @ 7:11 pm
You guys rock! Can’t wait for the release.
Rahsun McAfee said,
March 31, 2008 @ 8:33 pm
Freaking “A”! Keep up the good work. Can’t wait to test this setup out.
Charles said,
March 31, 2008 @ 8:42 pm
Great write-up! My mouth is watering to taste the Passenger goodness!
Ninh’s Weblog » Blog Archive » Passenger and The Road to Railsconf 2008 said,
April 1, 2008 @ 12:44 am
[…] Okay, so this may probably already be a stale update by the time of this writing, but I thought I’d point you to our article on the performance of Passenger against Mongrel and Thin. […]
Bob Aman said,
April 1, 2008 @ 1:40 am
Ok, so this is all very cool, but can it run Merb, the other small frameworks, or better yet, Rack? I’m pretty much done with Rails for non-commercial work at this point. Rails is old and busted, Merb (and the others) are the new hotness. Or something to that effect.
Ninh said,
April 1, 2008 @ 1:46 am
Hi Bob Aman,
A similar question has been asked on http://weblog.rubyonrails.com/2008/3/26/mod_rails-is-on-the-way and for completeness sake, I’ve included Hongli’s answer to this question below:
“There is the following saying: jack of all trades, master of none. At the moment, our goal is to make Passenger a great deployment system for Ruby on Rails, one that is matched by none in terms of ease of use and low maintenance overhead.
This is the reason why other frameworks such as Camping and Merb are not supported at the moment. However, members from the community have expressed interest for support for other frameworks, so we’re definitely keeping the option open. And of course, because Passenger will be open source, interested parties may contribute support for other frameworks themselves.”
click2asia said,
April 1, 2008 @ 2:44 am
mod_rails? What about running a rails app with mod_ruby?
Sam said,
April 1, 2008 @ 10:25 am
Great work guys. Now release it already with a git repo and a forum so we can help ya build/fix the darned thing!
@click2asia - time to do some homework.
Klaas Jan Wierenga said,
April 1, 2008 @ 2:25 pm
I understand that for a fair comparision you want to compare applications that don’t do any caching. But what if my application does use caching? How does that work with mod_rails / apache? Will there be configuration examples that show how to set up caching so apache serves the static files?
Will there be support for the X-Sendfile header?
Hongli said,
April 1, 2008 @ 4:28 pm
Klaas, page caching is fully accelerated. I.e. the .html files generated through page caching are automatically and directly served by Apache.
No special configuration is required for page caching to work! If configuration is required then we’d have missed one of our biggest goals, namely ease of use.
X-Sendfile is supported via mod_xsendfile.
links for 2008-04-01 « Brent Sordyl’s Blog said,
April 1, 2008 @ 4:37 pm
[…] 赖洪礼的 blog » Benchmark: Passenger (mod_rails) vs Mongrel vs Thin These new benchmarks however suggest Passenger (mod_rails) is faster than Mongrel, and is on par with Thin. (tags: mongrel rubyonrails mod_rails) […]
links for 2008-04-01 « Richesh said,
April 2, 2008 @ 7:52 am
[…] 赖洪礼的 blog » Benchmark: Passenger (mod_rails) vs Mongrel vs Thin (tags: rubyonrails mod_rails mongrel) […]
Arie said,
April 3, 2008 @ 9:29 am
Exciting stuff, can’t wait to check out mod_rails to see if it lives up to your claims
I’ve been benchmarking with nginx in combination with mongrel_cluster, thin and ebb, (ebb being the fastest), but these solutions aren’t as easy to setup as your mod_rails (as shown in your video).
http://ariekanarie.nl/archives/142/ruby-web-server-performance
http://ariekanarie.nl/archives/145/yet-another-ruby-server
gaw.in said,
April 4, 2008 @ 1:11 pm
How does Passenger perform with Ruby 1.9?
Hongli said,
April 4, 2008 @ 1:33 pm
gaw.in: I haven’t had the chance to test that yet. The SQLite3 Gem is not compatible with 1.9 by default (one has to manually apply some patch) so we’ve postponed 1.9 testing.
gaw.in said,
April 4, 2008 @ 3:12 pm
k, thx for the quick respone.
Than I just have to be patient
Crónica de una vida - Phusion Passenger - un modrails para apache said,
April 4, 2008 @ 9:01 pm
[…] que en cuanto a rendimiento y estabilidad también deberá tener sus ventajas ya que como se puede ver los primeros resultados son […]
Ninh’s Weblog » Blog Archive » Robustness comparisson between Phusion Passenger, Thin, Ebb and Mongrel said,
April 7, 2008 @ 10:02 am
[…] our previous article we’ve compared the performance of Passenger (mod_rails) vs Mongrel vs Thin, which has led to some pretty interesting results to say the least. Even though performance […]
This Week in Ruby (April 7, 2008) | Zen and the Art of Programming said,
April 7, 2008 @ 1:10 pm
[…] week I mentioned Passenger (aka mod_rails). This week Hongli Lai published some interesting benchmarks that compared it against Mongrel and Thin for three Rails applications (Typo, Petstore and El […]
RX-7乗りの適当な日々 said,
April 12, 2008 @ 8:25 pm
[IT][Linux] Passenger (mod_rails for Apache) を使ってRedmineを動かしてみたよ！…
d:id:rx7:20080327:p1 で紹介されていたRails用のApacheモジュール「Passenger (mod_rails for Apache)」が、とうとうベールを脱いだ模様。 先日、mod_railsを使うと何が嬉しいのよ、って聞かれたんですが、…
胡言乱语 » Blog Archive » Rails部署也可以简单得像PHP一样 said,
April 13, 2008 @ 2:33 am
[…] http://izumi.plan99.net/blog/index.php/2008/03/31/benchmark-passenger-mod_rails-vs-mongrel-vs-thin/ […]
e-moka Bazar » Blog Archive » Passenger: deployment facile per Ruby on Rails said,
April 13, 2008 @ 6:14 pm
[…] quanto riguarda le prestazioni, Passenger riesce a rivaleggiare tranquillamente con Mongrel, ed anche a superarlo in alcune […]
michael’s web wide world » mod_rails out! said,
April 14, 2008 @ 11:13 am
[…] it’s free and open source! … and faster than Mongrel (comparison done by the […]
RunRails-Blog » Blog Archive » Ganz frisch: mod_rails verspricht einfacheres Server-Setup mit Apache said,
April 14, 2008 @ 1:46 pm
[…] Vergleich zwischen Passenger und Mongrel zeigt, dass sich Passenger zumindest auf gleichem Niveau von Mongrel befindet (bei der […]
Mod_Rails for Apache | trevoro.ca | blog said,
April 15, 2008 @ 1:59 am
[…] modules like Mod_Security, without having to use an additional transparent proxy. There are some benchmarks out, and it looks like Passenger edges out Mongrel in most tests. This is very exciting news […]
BrainBakery » Blog Archive » mod_rails - The Crash Course said,
April 16, 2008 @ 10:52 pm
[…] Couple days ago Phussion guys released mod_rails Apache module. The promise is very simple: deploy Rails applications in the PHP way. This is not the best marketing they could have. Rails application deployment is not a big thing - I was deploying applications with shell scripts at no cost at all. Since there is Capistrano tool I don’t think this is a problem at all. The upload-single-file-through-ftp approach is making more troubles than it solves (which files did I change?) and it is not encouraging me to peek at the new toy at all. What I want to see is simplistic installation procedure and the amazing performance (speed is promoted right after the promise of easy deployment). […]
Chris said,
April 24, 2008 @ 10:12 am
So, how does RubyOnRails comapare to Ocsigen performancewise?
http://nleyten.com/2008/04/21/simple-benchmarks-on-the-ocsigen-server.aspx
JumbaBox » Blog Archive » Ruby on Rails using Apache (Passenger) setup guide for Ubuntu 8.1 said,
May 7, 2008 @ 3:38 am
[…] is designed for performance, scalability and security. In terms of its performance I came across a post by the Phusion team which shows some benchmark results of Passenger compared to Mongrel and […]
links for 2008-05-09 « Mike Does Tech said,
May 9, 2008 @ 1:44 am
[…] 赖洪礼的 blog » Benchmark: Passenger (mod_rails) vs Mongrel vs Thin (tags: rails rubyonrails mongrel mod_rails apache deployment) […]
Using Capistrano with Passenger (mod_rails) | Jim Neath said,
May 10, 2008 @ 10:40 am
[…] Benchmark: Passenger (mod_rails) vs Mongrel vs Thin […]