赖洪礼的 blog
This is my personal blog. For Phusion's corporate blog, please go to http://blog.phusion.nl

http://www.RubyEnterpriseEdition.com/ has been “launched”.

Next hint to be revealed on May 20, 2008.

Please help us with modding/digging these up.

• Passenger makes Rails easy, scalable! Dreamhost confirms!
• Rails Escapes The Ghetto
• Rails Escapes the Ghetto, Phusion Passenger now used on Dreamhost

We’ve made some modifications to Passenger to make Linux packagers’ lives a bit easier. Also, we’re looking for volunteers who wish to maintain packages for their distribution.

See http://blog.phusion.nl/2008/05/13/building-native-linux-packages-for-passenger-is-now-possible/ for details.

Passenger 1.0.5 has just been released. Please read the Phusion corporate blog for the announcement and upgrade instructions.

AkitaOnRails had recently interviewed us. The interview covers our company’s background and information about Passenger, among other things. He asked us quite some interesting questions, so we recommend you to read the interview.

Luke Kanies wrote an article, in which he says that Ruby has a distribution problem. It criticizes Rails applications for vendoring a lot of stuff, and criticizes RubyGems for not being able to handle native package dependencies, among other things.

I beg to disagree. The described problem is not a pure Ruby problem: it’s a general software distribution problem. I’m concerned that people would use this as another “reason” to oppose Ruby, even though it’s not specific to Ruby.

My response

I posted a reply to Luke’s article, and it was as follows:

Luke, I don’t really understand what you’re expecting from the Ruby/Rails community. You’re dealing with cross-platform distribution issues. That’s hard by its very nature. I was a developer in the Autopackage project (www.autopackage.org), a software packaging system that works across multiple Linux distributions. I’m currently a developer of Passenger (www.modrails.com). We’ve run into many of the issues that you mention here.

If I understand it correctly, you’re claiming that vendoring stuff is bad because:

• You cannot vendor everything (glibc, web browser, etc).
• As someone else has mentioned, vendoring stuff creates potential security issues. This is not unlike static linking in C/C++ projects.

On the other hand, vendoring stuff does have benefits:

• Guaranteed compatibility. Suppose you rely on GTK 2.2. On a faithful day, GTK 2.2.5 was released, but accidentally introduced a regression, and now your application fails left and right. Uh oh. If you vendored GTK then that wouldn’t have happened. This is not a theoretical possibility: GTK 2.2.15 or something actually broke AbiWord. (I’m not saying that vendoring GTK is a good idea, as GTK is quite large. I’m just pointing out a benefit of vendoring.)
• Less installation hassle. Not all platforms have good packaging systems. As far as I know, Debian-based distros are the only ones. RedHat-based YUM repositories tend to be quite small compared to Debian’s APT repositories. MacOS X and Windows don’t have native package management systems at all. If your app is cross-platform, then vendoring stuff is a lot easier for both the developer and the end user.

Vendoring stuff is not only common in the Java world, but also in the Windows and MacOS X world. Windows apps tend to bundle all their dependencies (with the exception of obvious stuff, such as Internet Explorer). How many games have you seen that bundle DirectX? Actually I’d say that not vendoring stuff is only common in Linux, and languages that have strong ties to Linux, such as Perl. Here’s where Debian’s package management system and huge package repositories really shine.

There’s almost no common ground in the world of package management. We at Autopackage had to invent our own dependency resolution mechanism (similar to APT) because there’s no lowest common denominator, even amongst Linux distros. RubyGems is probably created for the same reason: not all Ruby-supported platforms have (decent) package management, so they just wrote their own. Autopackage experimented with native package management integration (i.e. being able to use the system’s native package manager to resolve dependencies) but that proved to be much, much harder than initially expected, and up until today that feature still isn’t finished. So I don’t think you can reasonably expect the Ruby community to do something about this. It’s not a pure Ruby problem: it’s a general software distribution problem.

As for being FHS-compliant: I can only say “don’t bother” if your application is cross-platform. It seems that only hardcore Linux users care about that. Outside the Linux world, FHS is being criticized by pretty much everyone. Windows and OS X users complain that application files in Linux are scattered everywhere, instead of being self-contained. Scattering files isn’t a problem in Linux because of package management, but it is a problem in all other platforms that don’t have decent package management. I’ve found that being FHS-compliant is more trouble than it’s worth.

So how do you solve this problem? I don’t think it’s possible to come up with a general silver bullet solution. So that leaves the following choices to you, the developer:

• Create a native package for every platform that you support, i.e. .deb for Ubuntu/Debian, .rpm for RedHat, another .rpm for Mandriva and SuSE because their package names are different, a .exe for Windows, .dmg for MacOS X, .tgz for Slackware, .??? for Solaris, etc.
• Write a cross-platform installer. Passenger/mod_rails chose this option because it’s a lot easier than the first one. Passenger depends on Ruby packages (Rails, fastthread, Rake, etc.) as well as native packages (GCC, Apache, APR). It checks whether all dependencies are available, and if not, it tells the user how to install those dependencies. We’ve put platform autodetection and Linux distro autodetection code in the installer. So on Debian/Ubuntu it would tell you to run “apt-get install apache2-prefork-dev” while on Fedora/RHEL/CentOS it would tell you to run “yum install apache-devel”. We’ve found that this approach works extremely well.

Finally, we at Autopackage fully recognized the pros and cons and vendoring/static linking. Autopackage recommends the following: dynamically depend on stuff that are common, but vendor/static link stuff that are uncommon. We believe that this is a good trade off between the pros and cons. Passenger follows this recommendation as well: we vendor the Boost C++ library. Few people have Boost installed, and when they have it installed it often isn’t the version that Passenger requires. Installing Boost is a huge, huge pain on MacOS X. In this case, the benefits that vendoring Boost gives us outweight the cons by far.

On the other hand, Apache is fairly common, and easy to install on most platforms. Rake, fastthread, etc. are also easy to install because of RubyGems. That’s why we dynamically depend on those things instead of vendoring them.

So it all boils down to making the right choices and correctly balancing the pros and cons of vendoring. There’s no silver bullet.

Gunnar Wolf’s follow-up

Luke’s article was quickly followed by a blog post by Debian developer Gunnar Wolf:

By using Ruby Gems, you dramatically increase entropy and harm your systems’ security.

To this, I say nonsense. It’s pretty well-known that Debian, and Linux distros in general, dislike foreign packaging systems, regardless of their merits. I see quite a lot of conversations on #rubyonrails @ irc.freenode.net that are somewhat like this:

Person A: hi, I’m using Debian/Ubuntu/some-other-Debian-derived-distro. I typed “gem install rails”. But when I type “rails foo”, it says “command not found”. what’s going on?
Person B: type “gem update –system && gem install rails”
Person A: wow, it worked! thank you!

It is painfully obvious that Debian did something to RubyGems. I was bitten by this very issue as well. Debian’s RubyGems package places binaries in /var/gems (or something like that, I don’t remember the exact location) instead of /usr/bin. Fine, I understand that they don’t want foreign packages to pollute /usr/bin, which is a managed directly. But the least they can do is adding /var/gems to $PATH by default, something which they didn’t. As a result, many people who installed Rails via Debian thought that Rails is broken, when it’s actually Debian who crippled RubyGems.

“Increase entropy”? I don’t even know what this is supposed to mean in the context of Ruby software distribution.
“Harm your systems’ security”? As I’ve already stated, vendoring has both pros and cons. If one dynamically depends on a library, then it means that the system administrator and library author are responsible for security updates, but it also means that a security update can potentially break the application. By vendoring stuff, the responsibility of security updates is mostly shifted to the application developer. This is a trade-off: there’s nothing wrong with it, and whether it’s the best thing to do depends on the situation. Gunnar however has an extremely purist view, along the lines of “if you don’t agree with us then you’re an idiot, regardless of the circumstances”.

It seems more like that Gunnar’s words are carefully picked, with the goal of creating knee-jerk reactions. Debian said pretty much the same thing about Autopackage in the past, and now they’re doing it again with RubyGems.

Luke, Gunnar, will you realize that all you’re doing is ranting about a problem, without offering any solutions? And no, creating native Debian packages is not a solution, the world isn’t comprised of just Debian.

Passenger 1.0.2 fixed MacOS X compatibility. Unfortunately, the OS X fix broke compatibility with 64-bit Linux. The 1.0.4 release was supposed to fix it for both platforms. Before we released 1.0.4, we had an OS X user test it, and the test result was positive. But soon after the release we got bug reports from people, telling us that 1.0.4 broke OS X compatibility.

The main problem lies in the file descriptor passing code. We’re struggling to get it right:
1. The original code was copied from Ruby. Apparently that didn’t work well on 64-bit MacOS X. This also means that file descriptor passing on 64-bit MacOS X Ruby is also broken, not just Passenger.
2. There is generally very little documentation about file descriptor passing. We found a bunch of code samples, but it turns out that all of the ones that we’ve tried have portability problems. The developer man pages don’t provide a lot of information on file descriptor passing.

We want to get this right, but we don’t have any 64-bit machines at all. So we’re looking for people who can help us with testing Passenger on 64-bit OS X and 64-bit Linux. We’d like to have at least 3 people from both categories. If you’re interested, please send a message to our discussion board, or join #passenger on irc.freenode.net.

Thank you.

Passenger version 1.0.2 has been released. We’ve finally finished our corporate blog now, so visit http://blog.phusion.nl/ for the release announcement, overview of changes, upgrade instructions, and more.

Now that we finally have an official corporate blog, people can stop complaining about my blog template. Note that our corporate blog’s template isn’t finished yet: it still needs some polish.

There are a few technical achievements in Passenger that we haven’t actively marketed. But we’d like the community to be aware of them, so we’ve written this blog post.

Unix domain sockets

Not too long ago, Thin announced support for Unix domain sockets. This gave Thin an incredible speed boost. Switchpipe soon followed, with alpha support for Unix domain sockets.

It has always surprised me that the Ruby/Rails web containers didn’t support Unix domain sockets until early 2008. Unix domain sockets aren’t exactly rocket science or exotic: the X Window System has used them for client-server communication on localhost for as long as I can remember. Database systems such as MySQL prefer to use Unix domain sockets on localhost as well. It’s also well-known that Unix domain sockets are faster than TCP sockets.

There’s one down side to Unix domain sockets though: you have to remove them. Usually, server processes remove them during exit. But if the system crashes, or if those processes crash, then the socket files are never removed, and the system administrator will have to do that manually. Ouch, maintenance overhead.

Passenger uses Unix sockets extensively, and has done so since the first release. But there’s one difference compared to Thin and Switchpipe: Passenger uses Unix sockets in the abstract namespace (as opposed to on the filesystem) whenever possible. Abstract namespace Unix sockets have filenames, just like regular Unix sockets, but they do not appear as files on the filesystem. This means that after a reboot, or if Apache crashes, no stale files will be left on the filesystem. This way, the system administrator doesn’t have to manually remove stale files if something went wrong.

Passenger strives for a concept that we call “zero maintenance”. We believe that software should Just Work(tm), and system maintenance overhead should be as low as possible. The system administrator shouldn’t have to worry about stuff like stale files, if he doesn’t have to. Ideally, the system administrator should be able to forget that he ever installed Passenger at all. The usage of abstract namespace Unix sockets is one of the mechanisms that we use to achieve that goal.

PID files (or the lack thereof)

Mongrel and Thin write so-called PID files to the filesystem. PID files contain the PIDs of background processes, and are used for shutting down those processes. But if the system crashes, or if those background processes crash, then the PID files will never be removed, and they become stale PID files. Early Mongrel versions refuse to start if there are stale PID files, forcing the system administrator to remove them manually. Passenger on the other hand doesn’t use PID files at all. If one shuts down Apache, then all Rails processes are shut down as well.

So what happens if Apache crashes? The Rails processes will exit as well. We use so-called “owner pipes” in Passenger, pipes that are shared between Apache and the Rails processes. Owner pipes are essentially a mechanism for reference counting processes. If Apache crashes, then the owner pipe is automatically closed. The Rails processes will detect this, and will shut down gracefully.

Community response: one week after the release

It has been a little more than one week since the first public release of Passenger. In this one week, lots and lots of people have blogged about us, including even O’Reilly. The feedback has been overwhelmingly positive!

A few nice quotes from the community:

• “As many of you have already done, I hopped on board to see how it worked and was amazed on how easy it was to get up and going.” — O’Reilly
• “So far I’m happy to report that, as advertised, it’s dead easy to use… and the performance seems solid.” — zerosum.com
• “Mod_rails seems to be a surprising stable and finished product, and this for a first public release! I’m anxious to check out their Ruby Enterprise Edition.” — Handermann.be

Thanks for the praise, everybody.

“Enterprise licenses” (a.k.a. donations)

One may wonder why our “Enterprise”/Donation page was taken down so soon. It’s because people have been very, very generous with regard to donating to us. In fact, so generous that Paypal decided to block our account under the guise of “anti-money laundering investigation”. But everything has been cleared up now, so the donations page is once again online.

If you’d like to help us survive the coming winters, or just want to support Passenger’s development, then please consider getting an “enterprise” license.

InfoQ interview

We’ve been interviewed by InfoQ. This interview elaborates some things about the name and the license. It also elaborates our upcoming memory optimizations for Passenger/Ruby, which allows them to use less memory. More information about this soon, honestly.

On a side note, we’re preparing the release of Passenger 1.0.2. It will soon be available.