I benchmarked Rails 1.2 vs 2.0 in my last blog post, and Rails 2.0 turned out to be 30%-50% faster. Then Eric Allam said:
“Hmm, another benchmark that completely ignores memory use and memory usage growth.”
Okay, let’s give it a try.
Wait, hold your horses!
When measuring memory, people usually measure the wrong thing. There are 3 things that one can measure:
- 1. Virtual memory size
- All modern operating systems – probably including yours – use virtual memory. The memory that an application sees is not the same as your physical memory. Even if you only have 256 MB RAM installed, all applications on your system still see 4 GB RAM, and can use more than 256 MB RAM (on x86, that is). See the Wikipedia entry.
Even if you only have 256 MB RAM, the virtual memory size reported by a process can still be 4 GB! This is because processes can map files, hardware and other resources into its memory address space, without really loading them into physical memory. This is complicated by the fact that processes can share memory with each other. The VM size is quite meaningless when measuring real memory usage.
- 2. Resident Set Size (RSS)
- This is the amount of the process’s memory that’s currently in physical memory. At first, this seems to be a good candidate for measuring real memory usage. But parts of the process may be swapped to the hard disk. The RSS also seems to include things like shared memory, which we clearly don’t want to measure. It’s unclear whether the RSS also contains kernel buffers and caches, which we also don’t want to measure.
- 3. Private dirty memory
- This is the best measurement. It does not count shared memory. This blog post explains it all. The private dirty memory is easy to measure on Linux (I’m using Ubuntu 7.10), but I’m not sure how to do it on other operating systems.
- 4. Ruby heap information
- 3 is usually the best measurement – but not for Ruby! In the past I’ve explained how Ruby’s heap is implemented. Ruby allocates several heaps, and the heaps contain equally-sized slots in which Ruby objects are stored. If we measure the private dirty memory (that is, assuming
fork()is never called), we’ll be effectively measuring the total heap size, including any free heap slots. Free heap slots are reused by Ruby, so they really shouldn’t be counted.
We want to count the number of used heap slots. Usually, there’s no way to do this. But I maintain a private Ruby branch (which will be released soon) with statistics information. We’ll use that for memory measurement. (My GC patch also provides statistics information, though the version I’m working on provides more accurate statistics.)
People interested in details should read a good book about operating systems. I recommend “Operating System Concepts” by Silberschatz et al. My “Operating Systems” classes professor has Dutch slides on the subject.
After a cold start with Mongrel 1.1.3, the private dirty memories were as follows:
- Rails 1.2.6: 25.5 MB
- Rails 2.0.2: 19.7 MB
Nice! A 6 MB memory reduction after a cold boot!
Though, as I’ve said earlier, these numbers don’t really mean much. The application code (i.e. controller and model code) haven’t been loaded into memory yet. And Ruby doesn’t garbage collect until its heap is full. So let’s find out what happens after 3000 requests and a garbage collection run. I added these actions to PostsController:
def gc headers["Content-Type"] = "text/plain" GC.start render :text => ObjectSpace.statistics end
And this route:
map.connect '/gc', :controller => "posts", :action => 'gc'
Then I ran, for both apps:
ab -n 3000 http://localhost:3000/ links http://localhost:3000/gc
The memory usages were:
- Rails 1.2.6
- Private dirty memory:28.1 MB
Total heap size: 14,702 KB
Free heap space: 7,789 KB
- Rails 2.0.2
- Private dirty memory: 20.9 MB
Total heap size: 8,059 KB
Free heap space: 2,098 KB
Nice! I didn’t expect this, but apparently Rails 2.0 uses a lot less memory!
[EDIT: Some people have contributed a chart. Thanks.]