Keep Henry Ford Happy: Separate Out your Specialist Processes

Regardless of what your web framework of choice is, when developing an application of any reasonable complexity you are bound to come across the same couple of issues time and again.

By admin

1) Long running processes causing a tailback
2) Resource hungry processes happily devouring all your precious memory.

Within the rails community the long running process problem has generally been solved through the creation of a number of plugins and gems that manage the forking of these irksome jobs for you (we currently use spawn).  While this does solve the problem in a simple and effective way, in reality it is exactly these type of processes that most often have pain points, which may well cause the proxy/mongrel server to die.  Thats one less back-end to support your application until monit restarts it (assuming you are using it, or some other form of system monitoring), and any requests that were blithely directed at this same back-end are now lost to the Internet graveyard, leaving some potentially frustrated clients to receive request timeouts.

Resource hungry processes are guilty of the same disruptive crime, although perhaps fall into a slightly different pattern. Typical pain points are quite obvious - a swallowing of memory (perhaps leading the system to swap) and memory leaks preventing the timely release of this memory.

The thing is, these long running and memory hungry processes typically perform some specialised task so why keep this code so close to your core application?  Far better, surely, to separate it away into its own space, where it can run amok (or more hopefully run like a dream) without disrupting the performance of your core application.

Since we are happily tearing little chunks out of our application, lets get more adventurous and take some much bigger bites.  It isn’t just individual processes that perform isolated, or specialised tasks.  It is often very easy to take large chunks of an application that deal with a particular issue and isolate that also.  The most obvious of these would be administrative functions: we all have them but why not put them elsewhere.  This really allows you to manage your system usage more finely - and can be taken to some interesting (if somewhat implausible) levels: imagine if you will,  a car sales website.  Having done a fine job of implementing seo, the site is very popular and receives lots of hits from auto-loving folk keen for a vehicular bargain.  But being the humble and naive people that all used car salesmen are, this growth was somewhat unexpected, and they don’t have much server capacity.  On top of this, they only update their car stocks once a week - so why not separate the admin functions into a separate application, and turn it off during the week when it’s not being used? A trite example maybe, but the underlying concept of modularizing your code is an important one.  Merb seems to be  following this philosophy quite closely too; the whole framework is designed to be modular - add what you need none that you don’t, plus they have recently announced merb-slices which looks to be a very clean way of breaking an application apart into cleanly defined modules.

The infrastructure that we have built at SCT to support our application ‘Blocks’ has attempted to deal with these issues in a few complimentary ways.  Our servers use Xen virtualization to enable us to give each client their own slice.  Each of these slices run exactly the same configuration, allowing us to easily automate the setup and maintenance thanks to the excellent partners in crime Deprec and Capistrano.  We can provision for load by adjusting the RAM allocated to each client along with the number of mongrels we have running.  That provides great separation at the client level (and more and more people are using a similar setup) but what about within our application as we have just spoken of?  Well within each slice we have split Blocks into two applications.  The first is ‘Blocks-core’ - the muscle bound workhorse that helps people create and manage their websites, distribute assets and fire off HTML campaigns.  The other is ‘Blocks-live’ - the lean and fast fox which is responsible for managing the serving of published content to the general public.  This separation allows us to do some really nice performance/tweaking such as linking Blocks-live up to a slave database with read-only access.  It also allows us to take Blocks offline for regular upgrade deployments without ever taking the live sites away from the public.  And this is only a starting point, as we would like to be able to modularize further (our app is quite well suited to it).

But what about those lower level, slow and resource hungry embedded processes? Each of our slices has to run these so why not separate them out?  Well in some cases this is not appropriate, but where it is, we have done exactly this by creating new slices dedicated to one particular action.  A prime example of this is our image processing server. As and when client sites require an image to be dealt with in some way, it puts it on a starling queue, where airbrush (our soon-to-be gemified image-processor) deals with the request before dumping the results back on the queue to be picked up by the requester.

With each of these processes and modules you need to assess the appropriate level of isolation - but I think this can be a very positive way of tackling your code.  Plus it make testing far easier as you can isolate functionality more easily.