A year ago in October I took my first full-time real no kidding development position. I was finally going to get to use Rails every day. I’d never really been a fan of all those Mac loving Rubyists, but I figured it worked fine on Linux so I’d be OK.

Well, I don’t have a lot of tolerance for frustration, so after a few months of struggling to learn Linux and Ruby at the same time, and trying to find a decent editor for Linux that I liked I ended up buying a MacBook Pro so I could “just use Textmate.” I wanted my tools to be out of my way and everyone was telling me that the Mac was the way to do it. I RVM’d it up and installed a Textmate trial. Of course, I also had to install XCode so that I could get a working GCC and homebrew to get some other dev tools.

Textmate, it turned out, was too slow and clunky for me. I traded it in for MacVim + Janus for a long while, ran RubyMine for a while, tried emacs a few times and am currently using SublimeText 2. I didn’t find what I was looking for in an editor completely, but being on the Mac made trying many different editors an easy task.

Everything was going pretty well, with the occasional hiccup until I started really pushing my system. I started running into issues with memory management (inactive memory, ahem) bringing my system to a grinding halt right at a critical moment. I’ve got an SSD in my computer, but moving large files takes hours because of TimeMachine and Spotlight.

It seems that one of the biggest features of OSX is designed to sell newer hardware. Which frankly sucks. Not only do you have to pay a huge premium for Apple hardware to run OSX, but then critical components of the operating system prevent you from being able to take advantage of that overpriced but pretty piece of equipment you bought.

But the thing that really grinds my gears is Lion. Or more specifically XCode 4. Or even more specifically this:

% which gcc
/usr/bin/gcc
% gcc --version
i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

If you look closely, you should notice that GCC is in fact, not GCC. /usr/bin/gcc on Lion is actually the busted and unmaintained llvm-gcc. The amount of things that won’t compile against that is staggering, and I can’t really put the blame on the developers but it sure makes compiling new gems, rubies and updates to things from Homebrew a hassle.

It really just makes me want to go back to Linux. The high level stuff may have lacked polish (though I didn’t look everywhere I could’ve), but at least the stupid compilers work. And I’ve yet to run into a Linux distro where /usr/bin/gcc wasn’t GCC.

Over the last few months I’ve been experimenting with and testing various methods for using concurrency within a Rails application for improved performance. Specifically in handling large amounts of data. The app I’ve been testing with is for all intents and purposes simply a web scraper. I went through several designs of the underlying system and settled on a final design that I’m really happy with. The code doesn’t entirely work (it’s a database problem), but the theory is solid and the performance is purely amazing.

My initial inspiration was delayed_job since I was familiar with it and knew it would work and how I would implement that style but I ran into scaling problems even just on my iMac because just a few threads would hammer the database hard enough with reads and writes that things would get crazy in just a few minutes. I then decided it might work to use an array / queue to cache things needing written to the database and, well, don’t do that. The real problem was that the individual scrapers were reliant on a database for both getting a job and noting it was finished. This also led to a lot of duplication of logic so the scraper and controller mechanisms had to be tightly coupled and kept causing deadlocks and collisions.

I didn’t like that, not at all. I decided anything that constantly polled a database would never work at scale, and decided I’d fix the problem for real – with messaging. In my current implementation, I have a Padrino app that handles the web interface and could provide an API out to something else if you really wanted it to. On the backend I have two rake tasks, only one of which connects to the database.

When a job is created in the system, it sends a message to the scraper queue to process the initial URL. It also does some things like read robots.txt so that we don’t poll too hard or hit urls we shouldn’t. I mean, just because this is a toy app doesn’t mean we should be sloppy ;-).

The scraper queue is listened to by this rake task. The rake task manages a thread pool of 20 threads (in production that would probably be a lot higher) and simply processes requests as they come through using MetaInspector. Once it has fetched the data (which will almost always be more waiting on the remote server than anything) it packages it all up into a nice neat JSON blob (I’ll probably switch to MessagePack in the future) and queues it to hit the manager process.

The manager process is where the app’s intelligence lies. It listens to the write queue for completed scrapes, writes them to the database, figures out the next page to scrape and queues it. As you noticed there is very little code in either task, and even the models are pretty skinny. By relying on a messaging queue I’ve managed to greatly simplify the architecture of the app while still obeying things such as priority and scrape rates. I wasn’t able to accurately benchmark this mechanism due to system problems I didn’t have time to troubleshoot but it held up (using an sqlite database) to a fast enough write performance the database logs spinning past my screen were making me dizzy.

If you want to try it out, check out the code on my BitBucket. Also, feel free to share your thoughts, comments and suggestions either in the comments or directly.

If you’re using Unicorn, first off good job. If you’re not, bookmark this post and fix that. I can wait.

Next, you’ll want to make your app as fast as possible. One of the biggest penalties you’ll pay in a Rails app is Garbage Collection. MRI’s GC is just awful in terms of performance. But you don’t need to have people waiting on it anymore! In exchange for a little higher CPU and Memory usage, you can tell Unicorn to run garbage collection outside of the request/response cycle using OOB_GC. It’s really quite simple, and it works in any Rack application. Just open config.ru and add this:

require 'unicorn/oob_gc'

That’s all you have to do! There is some additional tuning if you have, for example some routes that are particularly memory intensive and have GC run more frequently than the every 5 requests it comes set out of the box for. You can read the docs for more info.

A word of warning: This increased memory usage ~100% on my Gitlab install. It made things much faster too, though. Coincidentally, the memory increase from doing this was the same as the decrease from switching from multiple thins to Unicorn.

Another nice trick is to preload your app in the master process so that your workers are a little lighter weight and spin up faster. With Rails it’s pretty easy, just add this to your unicorn config file:

preload_app true

before_fork do |server, worker|
  # the following is highly recomended for Rails + "preload_app true"
  # as there's no need for the master process to hold a connection
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.connection.disconnect!
end

after_fork do |server, worker|
  # the following is *required* for Rails + "preload_app true",
  defined?(ActiveRecord::Base) and
    ActiveRecord::Base.establish_connection
end

I’d encourage you to read the docs for Unicorn if you’re using it and spend some time tweaking it for your app. In just a few minutes I significantly sped up Gitlab over Thin and the default Unicorn configuration and everyone likes “free” performance bumps :-)