Benchmark: Passenger (mod_rails) vs Mongrel vs Thin

Since the initial release of the teaser screencast, a lot of people have asked about Passenger’s (a.k.a. mod_rails) performance compared to Mongrel and Thin. Here’s the benchmark that you’ve all been waiting for. :)

In this benchmark, we compare the following software:

System specification:

  • Intel Core 2 Duo, T5300 @ 1.73GHz
  • 2 GB RAM
  • Ubuntu Linux 7.10
  • Apache 2.2.4 (Ubuntu package)
  • Ruby 1.8.6 (Ubuntu package)

Test applications:

  • Typo 5.0.3. This is an out-of-the-box setup with only a hello world post. Uses semi-static caching.
  • Petstore revision 360, an application that the JRuby guys use for benchmarking. At first it wouldn’t start in Passenger because of a bug Petstore: they forgot to add require 'forwardable' in app/models/cart_items.rb. I added this line myself.
  • Eldorado 0.9.1. This is an out-of-the-box setup with only a single forum and a single post added.

Typo and Eldorado both use SQLite 3. Petstore uses MySQL, because for some reason Petstore performs very badly when SQLite 3 is used.

People have also recommended Lovd By Less as test application, but I couldn’t get it working (database setup failed).

The applications, when served by Passenger, are setup at the following test domains:

  • http://typo.test/
  • http://petstore.test/
  • http://eldorado.test/

Both the Mongrel cluster and the Thin cluster are setup at http://mongrel_cluster.test/. (Only either Mongrel or Thin is running during the benchmark, not both at the same time.)

And of course, we always use the production environment.

Backend configuration:

  • Default Apache configuration, as provided by Ubuntu.
  • Passenger: RailsMaxPoolSize is set to 10. This means that Passenger will never spawn more than 10 Rails application instances. Note that Passenger is compiled without optimizations, so if you install Passenger yourself and benchmark it, you might find that Passenger is slightly faster.
  • Mongrel cluster: a cluster of 10 servers for each test application, behind mod_proxy_balancer, using default settings:
    ProxyPass / balancer://mongrel_cluster/
    ProxyPassReverse / balancer://mongrel_cluster/
    <Proxy balancer://mongrel_cluster>
    	BalancerMember http://127.0.0.1:3000
    	BalancerMember http://127.0.0.1:3001
    	BalancerMember http://127.0.0.1:3002
    	BalancerMember http://127.0.0.1:3003
    	BalancerMember http://127.0.0.1:3004
    	BalancerMember http://127.0.0.1:3005
    	BalancerMember http://127.0.0.1:3006
    	BalancerMember http://127.0.0.1:3007
    	BalancerMember http://127.0.0.1:3008
    	BalancerMember http://127.0.0.1:3009
    </Proxy>
  • Thin cluster: same configuration as Mongrel cluster.

Benchmark: Mongrel clusters

First we benchmark Typo served by Mongrel clusters. Because the actual application code is loaded by Mongrel during the first request, we do not want to count the first request’s time. So we run the following command to “warm up” the Mongrels, and discard its result:

ab -n 1000 -c 100 http://mongrel_cluster.test/

Next, we test 10,000 requests with 100 concurrent users:

ab -n 10000 -c 100 http://mongrel_cluster.test/

Result:

Concurrency Level:      100
Time taken for tests:   32.628480 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      54890000 bytes
HTML transferred:       51480000 bytes
Requests per second:    306.48 [#/sec] (mean)
Time per request:       326.285 [ms] (mean)
Time per request:       3.263 [ms] (mean, across all concurrent requests)
Transfer rate:          1642.83 [Kbytes/sec] received

We repeat these steps Petstore and Eldorado.

Petstore:
(for Petstore we benchmark the URI “/shop/viewCategory.shtml?category=DOGS” because it’s written in the INSTALL file)

Concurrency Level:      100
Time taken for tests:   109.786026 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      40750000 bytes
HTML transferred:       37680000 bytes
Requests per second:    91.09 [#/sec] (mean)
Time per request:       1097.860 [ms] (mean)
Time per request:       10.979 [ms] (mean, across all concurrent requests)
Transfer rate:          362.47 [Kbytes/sec] received

Eldorado:

Concurrency Level:      100
Time taken for tests:   70.889388 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      41130000 bytes
HTML transferred:       35530000 bytes
Requests per second:    141.06 [#/sec] (mean)
Time per request:       708.894 [ms] (mean)
Time per request:       7.089 [ms] (mean, across all concurrent requests)
Transfer rate:          566.60 [Kbytes/sec] received

Benchmark: Thin clusters

The benchmarking steps for Thin are the same.

Typo:

Concurrency Level:      100
Time taken for tests:   25.795470 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      54910000 bytes
HTML transferred:       51480000 bytes
Requests per second:    387.66 [#/sec] (mean)
Time per request:       257.955 [ms] (mean)
Time per request:       2.580 [ms] (mean, across all concurrent requests)
Transfer rate:          2078.78 [Kbytes/sec] received

Petstore:
(for Petstore we benchmark the URI “/shop/viewCategory.shtml?category=DOGS” because it’s written in the INSTALL file)

Concurrency Level:      100
Time taken for tests:   96.874308 seconds
Complete requests:      10000
Failed requests:        42
   (Connect: 0, Length: 42, Exceptions: 0)
Write errors:           0
Non-2xx responses:      42
Total transferred:      40630350 bytes
HTML transferred:       37547028 bytes
Requests per second:    103.23 [#/sec] (mean)
Time per request:       968.743 [ms] (mean)
Time per request:       9.687 [ms] (mean, across all concurrent requests)
Transfer rate:          409.58 [Kbytes/sec] received

Eldorado:

Concurrency Level:      100
Time taken for tests:   71.311031 seconds
Complete requests:      10000
Failed requests:        5
   (Connect: 0, Length: 5, Exceptions: 0)
Write errors:           0
Non-2xx responses:      5
Total transferred:      41137070 bytes
HTML transferred:       35518568 bytes
Requests per second:    140.23 [#/sec] (mean)
Time per request:       713.110 [ms] (mean)
Time per request:       7.131 [ms] (mean, across all concurrent requests)
Transfer rate:          563.34 [Kbytes/sec] received

Benchmark: Passenger

The benchmarking steps for Passenger are the same.

Typo:

Concurrency Level:      100
Time taken for tests:   25.131980 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      55200000 bytes
HTML transferred:       51260000 bytes
Requests per second:    397.90 [#/sec] (mean)
Time per request:       251.320 [ms] (mean)
Time per request:       2.513 [ms] (mean, across all concurrent requests)
Transfer rate:          2144.92 [Kbytes/sec] received

Petstore:
(for Petstore we benchmark the URI “/shop/viewCategory.shtml?category=DOGS” because it’s written in the INSTALL file)

Concurrency Level:      100
Time taken for tests:   104.428742 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      40300000 bytes
HTML transferred:       36700000 bytes
Requests per second:    95.76 [#/sec] (mean)
Time per request:       1044.287 [ms] (mean)
Time per request:       10.443 [ms] (mean, across all concurrent requests)
Transfer rate:          376.86 [Kbytes/sec] received

Eldorado:

Concurrency Level:      100
Time taken for tests:   72.587018 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      41660000 bytes
HTML transferred:       35530000 bytes
Requests per second:    137.77 [#/sec] (mean)
Time per request:       725.870 [ms] (mean)
Time per request:       7.259 [ms] (mean, across all concurrent requests)
Transfer rate:          560.47 [Kbytes/sec] received

Summary

Here’s a nice summary of the requests per second, in a table and in a graph:

Typo Petstore Eldorado
Mongrel 306.48 91.09 141.06
Thin 387.66 103.23 140.23
Passenger 397.90 95.60 139.27

passenger_mongrel_thin_benchmark.png

Conclusion

I was pleasently surprised by the results. During Passenger’s development, our performance baseline was Mongrel behind mod_proxy. According to earlier (simple and naive) tests, Passenger performed similar to Mongrel. I expected Thin to be a bit faster than Passenger. These new benchmarks however suggest Passenger is faster than Mongrel, and is on par with Thin. :D

But you know what they say: there are lies, damn lies, and statistics. Your hardware and software is different, so if you benchmark it yourself, you’re likely to get (slightly?) different results.

50 Comments »

  1. labria said,

    March 31, 2008 @ 3:29 pm

    C’mon guys! If you’ll get performance like mongrel but with many times less the pain installing the whole thing, you’ll already be the heroes of the community =)

  2. Gerjan Stokkink said,

    March 31, 2008 @ 3:48 pm

    Keep up the good work, can’t wait to try this out myself :)

  3. Steven Hilton said,

    March 31, 2008 @ 4:18 pm

    Kudos!

    Do you have any info on memory requirements?

    Say, completely hypothetically, I have a small VPS that’s running a single rails app. And, hypothetically, it’s configured to run three mongrels behind an apache2 with mod_proxy_balancer. Since the framework and app code is loaded on startup, does it exist in memory only once (via copy-on-write)? And therefore, hypothetically, I’ll save on memory and be able serve more requests in my memory-constrained environment. Hypothetically.

    Also, has anyone tested passenger running multiple rails apps/vhosts in a single apache? Does a rails app in vhost 1 and a rails app in vhost 2 share any resources?

  4. Frédéric de Villamil said,

    March 31, 2008 @ 5:27 pm

    Very interesting benchmark, and really thank you for choosing Typo as a guinea pig. Thos results somehow show that our refactoring towards performances was the ood choice.

  5. Phill Kenoyer said,

    March 31, 2008 @ 5:45 pm

    Yeah, but how does it do against Nginx + Mongrel? I have found it to be way easier and lighter to setup Nginx than Apache. But if Apache give me more performance then…

  6. Mike Judge said,

    March 31, 2008 @ 5:46 pm

    10 Mongrel instances is way too many. You’ll see its metrics improve if you reduce the number. In my experience, the sweet spot is usually [CPUs + 1] instances (e.g., Dual Intel processor, run 3 mongrels.)

  7. Aleksandr said,

    March 31, 2008 @ 7:11 pm

    You guys rock! Can’t wait for the release.

  8. Rahsun McAfee said,

    March 31, 2008 @ 8:33 pm

    Freaking “A”! Keep up the good work. Can’t wait to test this setup out.

  9. Charles said,

    March 31, 2008 @ 8:42 pm

    Great write-up! My mouth is watering to taste the Passenger goodness!

  10. Ninh’s Weblog » Blog Archive » Passenger and The Road to Railsconf 2008 said,

    April 1, 2008 @ 12:44 am

    [...] Okay, so this may probably already be a stale update by the time of this writing, but I thought I’d point you to our article on the performance of Passenger against Mongrel and Thin. [...]

  11. Bob Aman said,

    April 1, 2008 @ 1:40 am

    Ok, so this is all very cool, but can it run Merb, the other small frameworks, or better yet, Rack? I’m pretty much done with Rails for non-commercial work at this point. Rails is old and busted, Merb (and the others) are the new hotness. Or something to that effect.

  12. Ninh said,

    April 1, 2008 @ 1:46 am

    Hi Bob Aman,

    A similar question has been asked on http://weblog.rubyonrails.com/2008/3/26/mod_rails-is-on-the-way and for completeness sake, I’ve included Hongli’s answer to this question below:

    “There is the following saying: jack of all trades, master of none. At the moment, our goal is to make Passenger a great deployment system for Ruby on Rails, one that is matched by none in terms of ease of use and low maintenance overhead.

    This is the reason why other frameworks such as Camping and Merb are not supported at the moment. However, members from the community have expressed interest for support for other frameworks, so we’re definitely keeping the option open. And of course, because Passenger will be open source, interested parties may contribute support for other frameworks themselves.”

  13. click2asia said,

    April 1, 2008 @ 2:44 am

    mod_rails? What about running a rails app with mod_ruby?

  14. Sam said,

    April 1, 2008 @ 10:25 am

    Great work guys. Now release it already with a git repo and a forum so we can help ya build/fix the darned thing!

    @click2asia – time to do some homework.

  15. Klaas Jan Wierenga said,

    April 1, 2008 @ 2:25 pm

    I understand that for a fair comparision you want to compare applications that don’t do any caching. But what if my application does use caching? How does that work with mod_rails / apache? Will there be configuration examples that show how to set up caching so apache serves the static files?

    Will there be support for the X-Sendfile header?

  16. Hongli said,

    April 1, 2008 @ 4:28 pm

    Klaas, page caching is fully accelerated. I.e. the .html files generated through page caching are automatically and directly served by Apache.

    No special configuration is required for page caching to work! If configuration is required then we’d have missed one of our biggest goals, namely ease of use. :)

    X-Sendfile is supported via mod_xsendfile.

  17. links for 2008-04-01 « Brent Sordyl’s Blog said,

    April 1, 2008 @ 4:37 pm

    [...] 赖洪礼的 blog » Benchmark: Passenger (mod_rails) vs Mongrel vs Thin These new benchmarks however suggest Passenger (mod_rails) is faster than Mongrel, and is on par with Thin. (tags: mongrel rubyonrails mod_rails) [...]

  18. links for 2008-04-01 « Richesh said,

    April 2, 2008 @ 7:52 am

    [...] 赖洪礼的 blog » Benchmark: Passenger (mod_rails) vs Mongrel vs Thin (tags: rubyonrails mod_rails mongrel) [...]

  19. Arie said,

    April 3, 2008 @ 9:29 am

    Exciting stuff, can’t wait to check out mod_rails to see if it lives up to your claims :)

    I’ve been benchmarking with nginx in combination with mongrel_cluster, thin and ebb, (ebb being the fastest), but these solutions aren’t as easy to setup as your mod_rails (as shown in your video).

    http://ariekanarie.nl/archives/142/ruby-web-server-performance
    http://ariekanarie.nl/archives/145/yet-another-ruby-server

  20. gaw.in said,

    April 4, 2008 @ 1:11 pm

    How does Passenger perform with Ruby 1.9?

  21. Hongli said,

    April 4, 2008 @ 1:33 pm

    gaw.in: I haven’t had the chance to test that yet. The SQLite3 Gem is not compatible with 1.9 by default (one has to manually apply some patch) so we’ve postponed 1.9 testing.

  22. gaw.in said,

    April 4, 2008 @ 3:12 pm

    k, thx for the quick respone.
    Than I just have to be patient :-)

  23. Crónica de una vida - Phusion Passenger - un modrails para apache said,

    April 4, 2008 @ 9:01 pm

    [...] que en cuanto a rendimiento y estabilidad también deberá tener sus ventajas ya que como se puede ver los primeros resultados son [...]

  24. Ninh’s Weblog » Blog Archive » Robustness comparisson between Phusion Passenger, Thin, Ebb and Mongrel said,

    April 7, 2008 @ 10:02 am

    [...] our previous article we’ve compared the performance of Passenger (mod_rails) vs Mongrel vs Thin, which has led to some pretty interesting results to say the least. Even though performance [...]

  25. This Week in Ruby (April 7, 2008) | Zen and the Art of Programming said,

    April 7, 2008 @ 1:10 pm

    [...] week I mentioned Passenger (aka mod_rails). This week Hongli Lai published some interesting benchmarks that compared it against Mongrel and Thin for three Rails applications (Typo, Petstore and El [...]

  26. RX-7乗りの適当な日々 said,

    April 12, 2008 @ 8:25 pm

    [IT][Linux] Passenger (mod_rails for Apache) を使ってRedmineを動かしてみたよ!…

    d:id:rx7:20080327:p1 で紹介されていたRails用のApacheモジュール「Passenger (mod_rails for Apache)」が、とうとうベールを脱いだ模様。 先日、mod_railsを使うと何が嬉しいのよ、って聞かれたんですが、…

  27. 胡言乱语 » Blog Archive » Rails部署也可以简单得像PHP一样 said,

    April 13, 2008 @ 2:33 am

    [...] http://izumi.plan99.net/blog/index.php/2008/03/31/benchmark-passenger-mod_rails-vs-mongrel-vs-thin/ [...]

  28. e-moka Bazar » Blog Archive » Passenger: deployment facile per Ruby on Rails said,

    April 13, 2008 @ 6:14 pm

    [...] quanto riguarda le prestazioni, Passenger riesce a rivaleggiare tranquillamente con Mongrel, ed anche a superarlo in alcune [...]

  29. michael’s web wide world » mod_rails out! said,

    April 14, 2008 @ 11:13 am

    [...] it’s free and open source! … and faster than Mongrel (comparison done by the [...]

  30. RunRails-Blog » Blog Archive » Ganz frisch: mod_rails verspricht einfacheres Server-Setup mit Apache said,

    April 14, 2008 @ 1:46 pm

    [...] Vergleich zwischen Passenger und Mongrel zeigt, dass sich Passenger zumindest auf gleichem Niveau von Mongrel befindet (bei der [...]

  31. Mod_Rails for Apache | trevoro.ca | blog said,

    April 15, 2008 @ 1:59 am

    [...] modules like Mod_Security, without having to use an additional transparent proxy. There are some benchmarks out, and it looks like Passenger edges out Mongrel in most tests. This is very exciting news [...]

  32. BrainBakery » Blog Archive » mod_rails - The Crash Course said,

    April 16, 2008 @ 10:52 pm

    [...] Couple days ago Phussion guys released mod_rails Apache module. The promise is very simple: deploy Rails applications in the PHP way. This is not the best marketing they could have. Rails application deployment is not a big thing – I was deploying applications with shell scripts at no cost at all. Since there is Capistrano tool I don’t think this is a problem at all. The upload-single-file-through-ftp approach is making more troubles than it solves (which files did I change?) and it is not encouraging me to peek at the new toy at all. What I want to see is simplistic installation procedure and the amazing performance (speed is promoted right after the promise of easy deployment). [...]

  33. Chris said,

    April 24, 2008 @ 10:12 am

    So, how does RubyOnRails comapare to Ocsigen performancewise?

    http://nleyten.com/2008/04/21/simple-benchmarks-on-the-ocsigen-server.aspx

  34. JumbaBox » Blog Archive » Ruby on Rails using Apache (Passenger) setup guide for Ubuntu 8.1 said,

    May 7, 2008 @ 3:38 am

    [...] is designed for performance, scalability and security. In terms of its performance I came across a post by the Phusion team which shows some benchmark results of Passenger compared to Mongrel and [...]

  35. links for 2008-05-09 « Mike Does Tech said,

    May 9, 2008 @ 1:44 am

    [...] 赖洪礼的 blog » Benchmark: Passenger (mod_rails) vs Mongrel vs Thin (tags: rails rubyonrails mongrel mod_rails apache deployment) [...]

  36. Using Capistrano with Passenger (mod_rails) | Jim Neath said,

    May 10, 2008 @ 10:40 am

    [...] Benchmark: Passenger (mod_rails) vs Mongrel vs Thin [...]

  37. Art said,

    May 13, 2008 @ 10:16 pm

    Nice work and nice benchmark. – Maybe you can rerun the benchmark with a reduced number of mongrels ? (As suggested by Mike Judge, March 31, 2008 @ 5:46 pm)

    Can’t wait to try it myself :)

  38. LRGeek » Phusion’s mod_rails not so hot according to Railsplayground said,

    May 15, 2008 @ 5:12 pm

    [...] providers using shared hosting, or just anyone looking for an easy Rails deployment behind Apache. Performance benchmarks for the project seeme promising (a bit better than mongrel). However, when I asked to launch an [...]

  39. 28 mod_rails / Passenger Resources To Help You Deploy Rails Applications Faster said,

    May 23, 2008 @ 9:37 pm

    [...] Had With Passenger – A roundup of various impressions and experiences people have had so far. mod_rails vs Mongrel vs Thin – Benchmarks performed by Phusion themselves with three alternative deployment [...]

  40. Skiptree Thoughts » Blog Archive » mod_rails vs. thin vs. ebb said,

    May 24, 2008 @ 4:32 am

    [...]  here comes another performance analysis… or if you prefer more text, look here Leave a [...]

  41. Scott Motte » Blog Archive » Passenger (mod_rails) versus Thin on nginx said,

    July 13, 2008 @ 5:35 pm

    [...] a good post on Passenger versus Thin, and here is a better benchmark that includes thin on [...]

  42. Dinooz said,

    August 5, 2008 @ 8:41 pm

    I wonder how does Memory performance reflect on those benchmarks… what is more efficient in the server a lots of Thin processes… or a single Apache with many childs…

    =)

  43. lix said,

    August 17, 2008 @ 1:08 am

    hmm hmm the problem is that you measure the client performance with such a test like ab, check the system load etc. on the client host….

  44. Live in code | mod_rails 安装配置 [Ubuntu版] said,

    October 7, 2008 @ 2:41 am

    [...] 是赖洪礼和他的团队开发的一种基于 apache module 机制的 RoR 部署运行方式. [...]

  45. eXpand yOur cReativity » Blog Archive » Easy and robust deployments of Ruby On Rails applications on Apache webservers said,

    October 24, 2008 @ 7:53 am

    [...] According to our tests, Passenger is a bit faster than Mongrel, and seems to be on par with Thin: [...]

  46. Easy and robust deployments of Ruby On Rails applications on Apache webservers - eXpand yOur cReativity said,

    November 20, 2008 @ 6:10 am

    [...] According to our tests, Passenger is a bit faster than Mongrel, and seems to be on par with Thin: [...]

  47. Vikki Chan said,

    December 1, 2008 @ 5:25 am

    我想了解一下,配置的apache+mod_rails是否可以同时支持两个ruby应用的运行。

  48. brian said,

    February 5, 2009 @ 9:20 pm

    If this means I don’t have to deal with Mongrel memory leaks any more I’m all for it. Even if the performance was a little off it would be worth it. I’m doing some testing now of Mongrel 1.1.5 on Ubuntu 8.04 on a VPS with 400Meg of Ram. Basically mongrel eats up all the memory and kills the vps, even with no load!
    My personal opinion is that the Rails community needs to solve the deployment problem for higher load environments in the near future or RoR will just be an alternative to PHP and won’t compete in the broader market with Java or .Net. I agree with another person that jRuby could be the best option long term. Maybe mod_rails can save the day for lower volume VPS hosting. Hopefully it does. After I switch over I’ll post what I see from our experience. Thank you to the mod_rails team for the great working trying to solve this problem.

  49. Robert. Tsai said,

    January 15, 2010 @ 4:22 pm

    Great work on the load test. I had a question about running passenger on vmware guest. Has any one seen the performance on a virtual machine yet? I’m just asking that beacuse we ran a load test with 200 concurrent connections against Thin on a Centos 5.4 VM and it failed. We are seeing high CPU usage per Thin. We tried setting running 5 Thins to 15 Thins and the load was very high compared to the app running natively. I want to see if anyone else is seeing the same performance degradation as we are seeing. I’m hoping Passenger would give us better performance.

  50. Tyler said,

    August 27, 2011 @ 6:25 am

    This isn’t an apples-to-apples comparison.

    These results reflect, to a large degree, the level of concurrency employed. With only 10 pools in the mongrel cluster, you can serve at most 10 concurrent requests. But you’re sending 100 concurrent requests! Apache, on the other hand, is by default configured to handle 150 concurrent requests. So it’s no surprise that the mechanism that uses Apache’s process space is faster.

    If, on the other hand, you either increased the mongrel pool size to 100 or tested with only 10 concurrent requests, you would see just how well it ACTUALLY compares to passenger/mod_ruby.

RSS feed for comments on this post · TrackBack URI

Leave a Comment