赖洪礼的 blog
This is my personal blog. For Phusion's corporate blog, please go to http://blog.phusion.nl

Luke Kanies wrote an article, in which he says that Ruby has a distribution problem. It criticizes Rails applications for vendoring a lot of stuff, and criticizes RubyGems for not being able to handle native package dependencies, among other things.

I beg to disagree. The described problem is not a pure Ruby problem: it’s a general software distribution problem. I’m concerned that people would use this as another “reason” to oppose Ruby, even though it’s not specific to Ruby.

My response

I posted a reply to Luke’s article, and it was as follows:

Luke, I don’t really understand what you’re expecting from the Ruby/Rails community. You’re dealing with cross-platform distribution issues. That’s hard by its very nature. I was a developer in the Autopackage project (www.autopackage.org), a software packaging system that works across multiple Linux distributions. I’m currently a developer of Passenger (www.modrails.com). We’ve run into many of the issues that you mention here.

If I understand it correctly, you’re claiming that vendoring stuff is bad because:

• You cannot vendor everything (glibc, web browser, etc).
• As someone else has mentioned, vendoring stuff creates potential security issues. This is not unlike static linking in C/C++ projects.

On the other hand, vendoring stuff does have benefits:

• Guaranteed compatibility. Suppose you rely on GTK 2.2. On a faithful day, GTK 2.2.5 was released, but accidentally introduced a regression, and now your application fails left and right. Uh oh. If you vendored GTK then that wouldn’t have happened. This is not a theoretical possibility: GTK 2.2.15 or something actually broke AbiWord. (I’m not saying that vendoring GTK is a good idea, as GTK is quite large. I’m just pointing out a benefit of vendoring.)
• Less installation hassle. Not all platforms have good packaging systems. As far as I know, Debian-based distros are the only ones. RedHat-based YUM repositories tend to be quite small compared to Debian’s APT repositories. MacOS X and Windows don’t have native package management systems at all. If your app is cross-platform, then vendoring stuff is a lot easier for both the developer and the end user.

Vendoring stuff is not only common in the Java world, but also in the Windows and MacOS X world. Windows apps tend to bundle all their dependencies (with the exception of obvious stuff, such as Internet Explorer). How many games have you seen that bundle DirectX? Actually I’d say that not vendoring stuff is only common in Linux, and languages that have strong ties to Linux, such as Perl. Here’s where Debian’s package management system and huge package repositories really shine.

There’s almost no common ground in the world of package management. We at Autopackage had to invent our own dependency resolution mechanism (similar to APT) because there’s no lowest common denominator, even amongst Linux distros. RubyGems is probably created for the same reason: not all Ruby-supported platforms have (decent) package management, so they just wrote their own. Autopackage experimented with native package management integration (i.e. being able to use the system’s native package manager to resolve dependencies) but that proved to be much, much harder than initially expected, and up until today that feature still isn’t finished. So I don’t think you can reasonably expect the Ruby community to do something about this. It’s not a pure Ruby problem: it’s a general software distribution problem.

As for being FHS-compliant: I can only say “don’t bother” if your application is cross-platform. It seems that only hardcore Linux users care about that. Outside the Linux world, FHS is being criticized by pretty much everyone. Windows and OS X users complain that application files in Linux are scattered everywhere, instead of being self-contained. Scattering files isn’t a problem in Linux because of package management, but it is a problem in all other platforms that don’t have decent package management. I’ve found that being FHS-compliant is more trouble than it’s worth.

So how do you solve this problem? I don’t think it’s possible to come up with a general silver bullet solution. So that leaves the following choices to you, the developer:

• Create a native package for every platform that you support, i.e. .deb for Ubuntu/Debian, .rpm for RedHat, another .rpm for Mandriva and SuSE because their package names are different, a .exe for Windows, .dmg for MacOS X, .tgz for Slackware, .??? for Solaris, etc.
• Write a cross-platform installer. Passenger/mod_rails chose this option because it’s a lot easier than the first one. Passenger depends on Ruby packages (Rails, fastthread, Rake, etc.) as well as native packages (GCC, Apache, APR). It checks whether all dependencies are available, and if not, it tells the user how to install those dependencies. We’ve put platform autodetection and Linux distro autodetection code in the installer. So on Debian/Ubuntu it would tell you to run “apt-get install apache2-prefork-dev” while on Fedora/RHEL/CentOS it would tell you to run “yum install apache-devel”. We’ve found that this approach works extremely well.

Finally, we at Autopackage fully recognized the pros and cons and vendoring/static linking. Autopackage recommends the following: dynamically depend on stuff that are common, but vendor/static link stuff that are uncommon. We believe that this is a good trade off between the pros and cons. Passenger follows this recommendation as well: we vendor the Boost C++ library. Few people have Boost installed, and when they have it installed it often isn’t the version that Passenger requires. Installing Boost is a huge, huge pain on MacOS X. In this case, the benefits that vendoring Boost gives us outweight the cons by far.

On the other hand, Apache is fairly common, and easy to install on most platforms. Rake, fastthread, etc. are also easy to install because of RubyGems. That’s why we dynamically depend on those things instead of vendoring them.

So it all boils down to making the right choices and correctly balancing the pros and cons of vendoring. There’s no silver bullet.

Gunnar Wolf’s follow-up

Luke’s article was quickly followed by a blog post by Debian developer Gunnar Wolf:

By using Ruby Gems, you dramatically increase entropy and harm your systems’ security.

To this, I say nonsense. It’s pretty well-known that Debian, and Linux distros in general, dislike foreign packaging systems, regardless of their merits. I see quite a lot of conversations on #rubyonrails @ irc.freenode.net that are somewhat like this:

Person A: hi, I’m using Debian/Ubuntu/some-other-Debian-derived-distro. I typed “gem install rails”. But when I type “rails foo”, it says “command not found”. what’s going on?
Person B: type “gem update –system && gem install rails”
Person A: wow, it worked! thank you!

It is painfully obvious that Debian did something to RubyGems. I was bitten by this very issue as well. Debian’s RubyGems package places binaries in /var/gems (or something like that, I don’t remember the exact location) instead of /usr/bin. Fine, I understand that they don’t want foreign packages to pollute /usr/bin, which is a managed directly. But the least they can do is adding /var/gems to $PATH by default, something which they didn’t. As a result, many people who installed Rails via Debian thought that Rails is broken, when it’s actually Debian who crippled RubyGems.

“Increase entropy”? I don’t even know what this is supposed to mean in the context of Ruby software distribution.
“Harm your systems’ security”? As I’ve already stated, vendoring has both pros and cons. If one dynamically depends on a library, then it means that the system administrator and library author are responsible for security updates, but it also means that a security update can potentially break the application. By vendoring stuff, the responsibility of security updates is mostly shifted to the application developer. This is a trade-off: there’s nothing wrong with it, and whether it’s the best thing to do depends on the situation. Gunnar however has an extremely purist view, along the lines of “if you don’t agree with us then you’re an idiot, regardless of the circumstances”.

It seems more like that Gunnar’s words are carefully picked, with the goal of creating knee-jerk reactions. Debian said pretty much the same thing about Autopackage in the past, and now they’re doing it again with RubyGems.

Luke, Gunnar, will you realize that all you’re doing is ranting about a problem, without offering any solutions? And no, creating native Debian packages is not a solution, the world isn’t comprised of just Debian.

Passenger 1.0.2 fixed MacOS X compatibility. Unfortunately, the OS X fix broke compatibility with 64-bit Linux. The 1.0.4 release was supposed to fix it for both platforms. Before we released 1.0.4, we had an OS X user test it, and the test result was positive. But soon after the release we got bug reports from people, telling us that 1.0.4 broke OS X compatibility.

The main problem lies in the file descriptor passing code. We’re struggling to get it right:
1. The original code was copied from Ruby. Apparently that didn’t work well on 64-bit MacOS X. This also means that file descriptor passing on 64-bit MacOS X Ruby is also broken, not just Passenger.
2. There is generally very little documentation about file descriptor passing. We found a bunch of code samples, but it turns out that all of the ones that we’ve tried have portability problems. The developer man pages don’t provide a lot of information on file descriptor passing.

We want to get this right, but we don’t have any 64-bit machines at all. So we’re looking for people who can help us with testing Passenger on 64-bit OS X and 64-bit Linux. We’d like to have at least 3 people from both categories. If you’re interested, please send a message to our discussion board, or join #passenger on irc.freenode.net.

Thank you.

Passenger version 1.0.2 has been released. We’ve finally finished our corporate blog now, so visit http://blog.phusion.nl/ for the release announcement, overview of changes, upgrade instructions, and more.

Now that we finally have an official corporate blog, people can stop complaining about my blog template. Note that our corporate blog’s template isn’t finished yet: it still needs some polish.

There are a few technical achievements in Passenger that we haven’t actively marketed. But we’d like the community to be aware of them, so we’ve written this blog post.

Unix domain sockets

Not too long ago, Thin announced support for Unix domain sockets. This gave Thin an incredible speed boost. Switchpipe soon followed, with alpha support for Unix domain sockets.

It has always surprised me that the Ruby/Rails web containers didn’t support Unix domain sockets until early 2008. Unix domain sockets aren’t exactly rocket science or exotic: the X Window System has used them for client-server communication on localhost for as long as I can remember. Database systems such as MySQL prefer to use Unix domain sockets on localhost as well. It’s also well-known that Unix domain sockets are faster than TCP sockets.

There’s one down side to Unix domain sockets though: you have to remove them. Usually, server processes remove them during exit. But if the system crashes, or if those processes crash, then the socket files are never removed, and the system administrator will have to do that manually. Ouch, maintenance overhead.

Passenger uses Unix sockets extensively, and has done so since the first release. But there’s one difference compared to Thin and Switchpipe: Passenger uses Unix sockets in the abstract namespace (as opposed to on the filesystem) whenever possible. Abstract namespace Unix sockets have filenames, just like regular Unix sockets, but they do not appear as files on the filesystem. This means that after a reboot, or if Apache crashes, no stale files will be left on the filesystem. This way, the system administrator doesn’t have to manually remove stale files if something went wrong.

Passenger strives for a concept that we call “zero maintenance”. We believe that software should Just Work(tm), and system maintenance overhead should be as low as possible. The system administrator shouldn’t have to worry about stuff like stale files, if he doesn’t have to. Ideally, the system administrator should be able to forget that he ever installed Passenger at all. The usage of abstract namespace Unix sockets is one of the mechanisms that we use to achieve that goal.

PID files (or the lack thereof)

Mongrel and Thin write so-called PID files to the filesystem. PID files contain the PIDs of background processes, and are used for shutting down those processes. But if the system crashes, or if those background processes crash, then the PID files will never be removed, and they become stale PID files. Early Mongrel versions refuse to start if there are stale PID files, forcing the system administrator to remove them manually. Passenger on the other hand doesn’t use PID files at all. If one shuts down Apache, then all Rails processes are shut down as well.

So what happens if Apache crashes? The Rails processes will exit as well. We use so-called “owner pipes” in Passenger, pipes that are shared between Apache and the Rails processes. Owner pipes are essentially a mechanism for reference counting processes. If Apache crashes, then the owner pipe is automatically closed. The Rails processes will detect this, and will shut down gracefully.

Community response: one week after the release

It has been a little more than one week since the first public release of Passenger. In this one week, lots and lots of people have blogged about us, including even O’Reilly. The feedback has been overwhelmingly positive!

A few nice quotes from the community:

• “As many of you have already done, I hopped on board to see how it worked and was amazed on how easy it was to get up and going.” — O’Reilly
• “So far I’m happy to report that, as advertised, it’s dead easy to use… and the performance seems solid.” — zerosum.com
• “Mod_rails seems to be a surprising stable and finished product, and this for a first public release! I’m anxious to check out their Ruby Enterprise Edition.” — Handermann.be

Thanks for the praise, everybody.

“Enterprise licenses” (a.k.a. donations)

One may wonder why our “Enterprise”/Donation page was taken down so soon. It’s because people have been very, very generous with regard to donating to us. In fact, so generous that Paypal decided to block our account under the guise of “anti-money laundering investigation”. But everything has been cleared up now, so the donations page is once again online.

If you’d like to help us survive the coming winters, or just want to support Passenger’s development, then please consider getting an “enterprise” license.

InfoQ interview

We’ve been interviewed by InfoQ. This interview elaborates some things about the name and the license. It also elaborates our upcoming memory optimizations for Passenger/Ruby, which allows them to use less memory. More information about this soon, honestly.

On a side note, we’re preparing the release of Passenger 1.0.2. It will soon be available.

Hi there,

Our apologies for the later than usual update, but things are getting really busy here at Phusion and this has been the first opportunity that we’ve been able to blog about the latest developments again! First off, we’d like to take the opportunity to ask all enterprise licensees to check their mail as they’re bound to find something nice there

Also, the next batch of enterprise licenses will become available very soon. We just wanted to let you guys know that we’re still alive and are working around the clock on some very interesting projects for our clients. Regardless of this, we want you guys to know that we’re still actively involved in the development of Phusion Passenger and Ruby Enterprise Edition and for those who want to know a bit more about Phusion Passenger and Ruby Enterprise Edition, we’ve recently done a mini interview with the fine people at infoq, and we encourage you to go check it out over here.

Even though we try to stay as actively involved as possible, we would also like to encourage the community to submit patches for the improvement of Phusion Passenger. We’re confident that we can make Passenger even better together (wow, that even rhymes so it must be true right? ;-)).

Also, it has come to our attention that some of you guys have expressed a concern about the memory use of Phusion Passenger and we have reason to believe that the memory wasn’t always measured in an accurate way. This is for the greater part the result of a well known problem with the innacurate display of memory usage by ps and/or top.

We’ll try to write an article on this as soon as possible where we will address this ‘issue’ with the appropriate measuring tools as well. Hopefully, this should set the record straight and take away any reluctance to try out Phusion Passenger as we’re pretty confident that Phusion Passenger uses a lot less memory than some people seemed to have inferred.

We’d like to leave it at this for now, and we’ll keep you guys posted and would love to stay in touch with you guys :-).

Cheers,
Hongli Lai
Ninh Bui

After weeks of teaching it hardcore computer science kung-fu, we’ve finally reached a stage in that we think our little baby is now all grown up. It is no longer solely dependent of our care, and from this moment on, Phusion Passenger is available for the masses.

Phusion Passenger has been the result of many many many (did I say many? I mean MANY) hours of hard work by both Hongli and I ( Ninh ), but it couldn’t have been made possible without the help of our beta testers. In particular, we’d like to thank Dallas of Dreamhost for being so patient and we’re looking forward to seeing Passenger being fully integrated in your hosting plans. Also, we’d like to thank Alex of Twitter for trying his best to test Passenger in time.

Also, we’d like to thank the Ruby on Rails core team for their insightful feedback and support, and in particular, we’d like to thank David Heinemeier Hansson, Jeremy Kemper and Pratik Naik for this.

Yukihiro Matsumoto (matz), thank you for the kind words on our project, it really kept us motivated.

Ryan Bates of Railscasts, thank you so much for doing the screencast on Phusion Passenger. We’re looking forward to meeting you in person at Railsconf and being able to thank you properly Depending on the paint we’re allowed to carry with us, shaking a green hand may be involved

Antonio Cangiano, thanks for the coverage. It was when we saw that a technical evangelist at IBM had written about us that it truly hit us that we were doing something important.

Russel Norris, you really should not underestimate the importance of your great sense of humor for an open source project :-D.

Since we’re from the Netherlands, we’d also like to thank our fellow dutchmen Stefan Fountain of Soocial and Thijs van der Vossen of Fingertips for their support on this project.

Also, thank you Weyert de Boer for being a great friend and helping us streamline the markup of our website. Even though we haven’t been able to complete it yet, we’re confident that we’ll be able to finish this with your help and we thank you for this.

Lastly, we’d like to thank Hans Scholten of the University of Twente for offering to help us on writing a scientific paper on our work with Ruby Enterprise Edition and Phusion Passenger.

To you the reader, we hope that you’ll enjoy using Passenger and we thank you for your patience. We’ll be seeing you guys at Railsconf USA 2008 (and probably Apachecon USA 2008 as well! )

Be sure to check us out there, since we’ll be doing a talk on Phusion Passenger (mod_rails) and our garbage collector optimization work on MRI. The talk material will be based on our scientific paper to be published on eprints, so the tech savvy among you guys will be in for a treat :-).

For science and all things pragmatic, this was Hongli Lai and Ninh Bui of Phusion, The Computer Science Company, wishing you a good weekend as we’ll be seeing you guys monday again. We’re going to take that well-deserved time off now ;-).

P.S. Even though we’re currently getting swamped with clients, that shouldn’t stop you from contacting us for business enquiries. As a matter of fact, we’re planning on expanding our team, so if you’re a computer scientist / developer located in the Netherlands with a lot of work experience and interested in working with us, please contact us as well

Ninh wrote this a while ago:

“Lastly, we’d like to take the opportunity to thank Ryan Bates of Railscasts for creating an excellent screencast on Passenger, and you’ll be able to see this for yourselves at the public release of Passenger this week. Needless to say, it totally kicks the *censored* out of our initial screencast :-D”

Are you guys ready?

So you guys are probably still waiting for the release of Passenger, and I’m pleased to say that we’ve been able to include the test results from Dreamhost (thanks Dallas :D) in the first public release that you’ll see within this week. (Hold on just a little longer guys , with a bit of luck, Twitter’s results will come in as well prior to the release of Passenger this week).

Instead of writing an elaborate blog post, I’ll keep it concise for the time being as we’re currently getting swamped at Phusion.

Today, after planning to do this for quite a long time now, we went to go see our Operating Systems/Computer Architecture teacher, Ir. Hans Scholten to discuss the possibilities for receiving ECTS for our work on both Passenger and Ruby Enterprise Edition (our ruby branch containing our garbage collector optimization).

From this, it became clear that our work wasn’t something trivial and Hans elaborated on the possibilities for publishing this work scientifically. Needless to say, we’re more than thrilled about this. After all, we’re “Phusion, the computer science company” right?

All kidding aside though, we do intend on taking Hans up on his offer to write a scientific paper on our work, seeing as this is just too great an opportunity to ignore. With his help, we’re more than confident about the quality of its contents.

Lastly, we’d like to take the opportunity to thank Ryan Bates of Railscasts for creating an excellent screencast on Passenger, and you’ll be able to see this for yourself at the public release of Passenger this week. Needless to say, it totally kicks the *censored* out of our initial screencast .

To all the people that have submitted a voice sample, we’ll be crediting you on the screencast section of the Passenger as well .

Hang on in there for just a little longer people , we’re almost there!

Thanks,

Hongli Lai
Ninh Bui

We’ve written a robustness comparison between Phusion Passenger (mod_rails), Thin, Ebb and Mongrel. See Ninh’s blog for the article.