Making PStore reaaaally fast (and stable)

March 26, 2008 at 7:04 pm · Filed under Optimizing Rails, Ruby, Ruby on Rails

PStore is a library in Ruby that “implements a file based persistance mechanism based on a Hash”. Ruby on Rails up until version 1.2 uses PStore as the default mechanism for storing sessions. But it’s pretty well-known that PStore “sucks”: people say that it’s slow, causes file corruptions, etc. This is one of the reasons why Rails 2.0 uses the cookie session store by default: it’s faster and needs less maintenance at the cost of a few (easy-to-avoid) caveats.

If you look at the Mongrel FAQ, then you’ll see that 3 items are devoted to telling you that PStore is the work of satan and that any sane web developer should ritually burn it. But this particular sentence caught my eye:

“Other things that can cause big pauses are:
- …
- Locking files wrong. Multiple processes locking files is a delicate thing to do. “

Locking files wrong? How can that possibly be? File locking, just like mutex locking, isn’t really rocket science.

Note that PStore’s RDoc advertises itself as transactional:

“# The transactional behavior ensures that any changes succeed or fail together.
# This can be used to ensure that the data store is not left in a transitory
# state, where some values were upated but others were not.”

I decided to take a look at its source code. What possibly could have gone wrong?

Well, I’m not sure what is wrong with it, but the relevant code, PStore#transaction, is a bit messy. I had a hard figuring out what it is exactly doing and why, but I figured that it does these things:

  • It locks the file.
  • It reads and unmarshals the file.
  • It generates an MD5 checksum of the file.
  • It runs the transaction block, then generates an MD5 checksum of the new contents of the file. The file is only written to if the MD5 checksum or the size of the new content doesn’t match that of the original file.
  • It writes new data to a temp file, then renames that to the original file (or at least, I think that’s what it does; it seems to use 2 temp files files). File renames are atomic on Unix but not Windows. This is why on Windows, one needs to implement file recovery as well. PStore seems to have some code for recovery, but it’s unclear how well that works. It doesn’t match the algorithm given on MSDN.

It’s not clear whether PStore has locked everything correctly. Oh, and PStore isn’t thread-safe (though it is reentrant).

  • PStore is usually used for writing small amounts of data, so calculating an MD5 is definitely not worth it - just write the file already!
  • Writing to a temp file ensures atomicity on Unix, but it adds another system call, and system calls are expensive. If one is able to open the file for writing-and-appending, then writing to it shouldn’t raise any errors except in rare conditions, such as out-of-disk-space conditions or hardware errors. Furthermore, I’ve never seen anybody using PStore for anything other than for session data and other not-so-important stuff, so performance is more important.
  • Aah, Windows…. All the recovery code just to work around the fact that Windows doesn’t support atomic file renames, cause a lot of performance loss.

So I rewrote PStore. The code is now easier to read and is faster. And as far as I know, everything is locked correctly so there shouldn’t be any concurrent issues.

By default, it doesn’t try as hard to ensure file integrity because harddisk I/O errors are very rare, but if file integrity is really an issue, then you can set pstore.ultra_safe_transactions = true to enable it. This option only has effect on Unix though: as of now, I haven’t bothered writing complex recovery code for Windows.

This is the benchmark program that I used:
http://pastie.org/170997

The results are as follows:
benchmark.png

As you can see, PStore has become 2 times of even 3 times faster, depending on whether is ultra_safe_transactions is enabled! But let’s see what kind of effect it has on a dummy Rails 2.0 app that uses the PStore session store:

  • Dummy Rails 2.0 app, using original PStore library: 170.22 requests/sec
  • Dummy Rails 2.0 app, using my PStore library: 196.72 requests/sec

A small performance increase. :) (Though the cookie session store is still faster: 221.74 requests/sec.)

You can download it here:
http://pastebin.com/f163702f2

If you want your Rails app to make use of it, simply save it as “lib/pstore.rb”. For maximum stability, Rails’s PStore handler should be modified to ignore any errors encountered during unmarshalling.

One of these days I’ll send a patch to ruby-core so that this can be merged back upstream. But for now Passenger (a.k.a. mod_rails) has priority.

12 Comments »

  1. mla said,

    March 26, 2008 @ 8:16 pm

    Thanks for working on this!

    There are a couple more performance issues I found. First, by default all the sessions are stored in a single directory. As you probably know, most filesystems use a sequential scan on directories, so as the number of sessions increases, performance drops considerably. You can easily fix that by partitioning the sessions into subdirectories.

    I haven’t looked at the code yet, but you’re saying “calculating an MD5 is definitely not worth it - just write the file already” … does that mean you’re always writing the file on every request? I’d like to see the numbers for that. In a Perl implementation I can tell you that with an active site, skipping the writes, even for small files, significantly boosted performance.

    Even so, there’s an issue there. With Ruby, the marshaling of the data is not ordered. It’s fairly easy to construct a situation where the marshaled data “flip-flops” on every read/write, causing a write regardless of whether the data has changed. In Perl hash keys can always be ordered and the Storable module has an option ($Storable::canonical) to always sort the keys. I don’t know what the fix would be for Ruby.

    I have some example code for this code if you’re interested.

  2. Hongli said,

    March 26, 2008 @ 9:52 pm

    Hi mla.

    As you probably know, most filesystems use a sequential scan on directories

    This is true for listing all files in a directory, but are you sure it’s also true for reading the contents of a single of which the filename is already known?

    I haven’t looked at the code yet, but you’re saying “calculating an MD5 is definitely not worth it - just write the file already” … does that mean you’re always writing the file on every request?

    Not really. The current PStore implementation does something like this:

    original_data = read_from_file
original_md5 = calculate_md5_for(original_data)
@table = Marshal.dump(original_data)
yield
new_data = Marshal.dump(@table)
if new_data.size != original_data.size || calculate_md5_for(new_data) != original_md5
   write_to_file(new_data)
end

    I removed the MD5 calculation.

    Read-only transactions will not result in a write. Read-write transactions will always result in a write, unless the transaction was aborted. I’ve found that calculating the MD5 takes too much time, and writing to the file without checking the MD5 results in a performance improvement.

  3. mla said,

    March 26, 2008 @ 10:58 pm

    This is true for listing all files in a directory, but are you sure it’s also true for reading the contents of a single of which the filename is already known?

    I believe so. Take a look at this graph that shows the performance gain by partitioning the sessions
    into directories (this is on ext3 filesystem):

    http://mla.homeunix.com/tmp/pstore.png

    I’ll look at the marshal stuff later, but here’s a example of how it can appear that the contents has changed when it hasn’t:

    http://mla.homeunix.com/tmp/marshal-test.rb
    http://mla.homeunix.com/tmp/marshal-test.txt

    I’m very much interested in gettting an improved version of the file session store into rails. Esp. when using a load balancer that has session affinity, this is a very scalable solution if you can’t or don’t want to use the cookie store approach.

  4. Hongli said,

    March 27, 2008 @ 12:10 am

    Hm, interesting. :) I’ll look into this, thanks.

    As for Marshal: it’s implemented in C. Changing it would require a bit of work as well as patching the interpreter.

  5. Hongli said,

    March 27, 2008 @ 1:39 am

    By the way, what benchmark script did you use to generate the results seen in your pstore.png graph?

  6. mla said,

    March 27, 2008 @ 2:33 am

    By the way, what benchmark script did you use to generate the results seen in your pstore.png graph?

    Here is the patch to add partitioning:
    http://mla.homeunix.com/tmp/pstore_hashed_directories.patch.txt

    I think this is the script I used to test it. See the database_manager param which
    I toggled for successive runs:

    http://mla.homeunix.com/tmp/pstore-bench.rb

    That was one of my first forays into Ruby so be gentle ;/

    I think you should take another look at the md5 logic. Under Rails, the session is always being opened for R/W, correct?
    Which means the session will always be written even if it hasn’t changed. But in most environments, the read/write ratio is often 10:1 or more.

    In a quick benchmark of your new version, it appears to be 2x as slow when the session has not changed, suggesting that the write is still substantially more expensive than the md5 check.

  7. Hongli said,

    March 27, 2008 @ 7:40 pm

    I’ve modified Marshal.dump and added a ‘canonical’ option. :) A patch has been sent to ruby-core. I’ve also restored the MD5 checking code. It makes use of the ‘canonical’ option whenever possible. Code is here: http://pastebin.com/mb293607
    Could you benchmark it?

    (If you don’t want to apply the patch, then you can simulate the ‘canonical’ behavior by calling rehash() on all your Hash elements, including the @table instance variable in PStore.)

  8. mla said,

    March 27, 2008 @ 9:54 pm

    Fantastic. I don’t see the marshal patch in the archives yet. Would you post it to pastebin too?

  9. Hongli said,

    March 27, 2008 @ 10:01 pm

    Yeah, weird. I sent the email hours ago but I still don’t see it.

    http://pastebin.com/m12689e12
    Patch is against 1.8.6.

  10. mla said,

    March 28, 2008 @ 7:44 am

    The patch to marshal seems to work perfectly :)

    For pstore physical writes your version is 2x faster for me:

    user system total real
    std 20.240000 36.490000 56.730000 ( 58.234056)
    new_canonical_marshal 13.090000 12.830000 25.920000 ( 26.656768)

    That’s using the fast store strategy and read/writing the same file 50k times.

    For logical writes where the data hasn’t changed, the new version is
    roughly 15-20% slower:

    user system total real
    std 6.500000 3.160000 9.660000 ( 9.841138)
    new_canonical_marshal 8.640000 3.300000 11.940000 ( 12.119400)

    I did some quick profiling and I don’t see anything obvious. I think it’s the
    overhead of the added method calls. I inlined most of them and got pretty
    close to the standard version.

    new_mla 6.810000 3.110000 9.920000 ( 10.008198)

    OTOH, even with the slight slow-down we’re still talking 4k+ ops/sec
    which is unlikely to be a bottleneck.

  11. mla said,

    March 28, 2008 @ 7:07 pm

    Just for my understanding, what made the original version
    not thread-safe? I see your synchronize call, I’m just not
    clear on exactly what it’s serializing access to. Is the
    file I/O not thread safe even when flock’d?

  12. Hongli said,

    March 28, 2008 @ 7:14 pm

    No, the file I/O is probably safe. But bad things may happen if two threads enter the same transaction. For example, suppose thread 1 enters a read-only transaction, and it just passed this line:

    @rdonly = read_only

    Then a context switch occurs. Thread 2 enters a read-write transaction and passes the same line as well, setting @rdonly to false. Thread 2 exits transaction. Context switch occurs. Thread 1, which was supposed to be executing a read-only transaction, now thinks that the transaction is read-write and writes stuff to disk. Boom.

RSS feed for comments on this post · TrackBack URI

Leave a Comment