Put’em in the queue

Rails is very good web framework. It works very well, as long You use it as it supposed to be used. Serving web applications means processing HTTP requests (often a lot of them) as fast as possible and send output to the browser. All is fine as long processing is simple and can be done fast. But sometimes You will need to make some operations which are more time consuming – and not because server is overloaded, but task itself takes longer. Like image processing or other operations on huge datasets. Then You will hit performance wall very fast. And it is not only Rails issue (BTW – did You know that upcoming Rails 2.2 will be finally thread safe?) it is problem for each web framework.

There is one simple solution – to remove time consuming data crunching from HTTP request processing and mover to some other process. Idea seems simple, but implementation can be more complicated. You need to provide some way to communicate for frontend and processing part.

When we have started Friends Feed Me, at moment Facebook Platform was presented to public, I did processing OPML files with external process which was using DRb to provide communication between Rails application and worker process. Well, even with such simple application it required to write a bit code. FFM did not take off, so we did not have scalability issues, but if it would taken off then using DRb would be problematic. Sure it can be done, but there are much simpler and proven solutions to this. I wish I had knew then about Active Messaging.

It is plugin which brings Rails much closer to messaging. What is messaging or event driven architecture? To put it in simple words – between frontend application (Rails) and worker process (or many of them) You insert black box which we call queue. Now tasks to process are sent to queue by Rails as fast as possible, and are taken from this queue by worker whenever it has free resources. That way Rails can send response to browser very fast (of course this is response data is being processed type, not final result) and worker will do it when it be possible.

Queue
(c) andrewparnell

Active Messaging is only one of many parts of this black box, provides communication interface with queue. Queue need to be implemented separately. Luckily for us, there is one tool which makes our black box complete without much overhead from us. It is service from Amazon called Simple Queue Service. If You are familiar with Amazon S3 then SQS is for messaging that S3 is for file storage. Simple API, with reasonable pricing (pay-as-you-go), ready to use instantly.

So what this is about?

I will provide here simple example how You can start using messaging in own Rails application. Example is simple and is meant to allow You start in minutes with own AM and SQS usage.
Continue reading

Back online

Last two weeks I was basically offline, recharging my batteries as usual over Baltic sea. Now I have several new ideas, I hope they will turn into some interesting projects ;)

From this year’s regular vacations only memories will last. This time I had no time to take pictures – with three children, two of them under 3 yrs we had a lot other things to do :) And to keep this memories more vivid one of photos I got chance to take:

Jastarnia\'s Church

Now back to work :)