Put’em in the queue

Rails is very good web framework. It works very well, as long You use it as it supposed to be used. Serving web applications means processing HTTP requests (often a lot of them) as fast as possible and send output to the browser. All is fine as long processing is simple and can be done fast. But sometimes You will need to make some operations which are more time consuming – and not because server is overloaded, but task itself takes longer. Like image processing or other operations on huge datasets. Then You will hit performance wall very fast. And it is not only Rails issue (BTW – did You know that upcoming Rails 2.2 will be finally thread safe?) it is problem for each web framework.

There is one simple solution – to remove time consuming data crunching from HTTP request processing and mover to some other process. Idea seems simple, but implementation can be more complicated. You need to provide some way to communicate for frontend and processing part.

When we have started Friends Feed Me, at moment Facebook Platform was presented to public, I did processing OPML files with external process which was using DRb to provide communication between Rails application and worker process. Well, even with such simple application it required to write a bit code. FFM did not take off, so we did not have scalability issues, but if it would taken off then using DRb would be problematic. Sure it can be done, but there are much simpler and proven solutions to this. I wish I had knew then about Active Messaging.

It is plugin which brings Rails much closer to messaging. What is messaging or event driven architecture? To put it in simple words – between frontend application (Rails) and worker process (or many of them) You insert black box which we call queue. Now tasks to process are sent to queue by Rails as fast as possible, and are taken from this queue by worker whenever it has free resources. That way Rails can send response to browser very fast (of course this is response data is being processed type, not final result) and worker will do it when it be possible.

(c) andrewparnell

Active Messaging is only one of many parts of this black box, provides communication interface with queue. Queue need to be implemented separately. Luckily for us, there is one tool which makes our black box complete without much overhead from us. It is service from Amazon called Simple Queue Service. If You are familiar with Amazon S3 then SQS is for messaging that S3 is for file storage. Simple API, with reasonable pricing (pay-as-you-go), ready to use instantly.

So what this is about?

I will provide here simple example how You can start using messaging in own Rails application. Example is simple and is meant to allow You start in minutes with own AM and SQS usage.

When You install AM plugin You will get following features in Rails:

• publishes_to method in controller definition
• new generator processor
• new script poller in scripts

To get things working we need to:

• create processor
• define queues
• send messages to queue
• process data

Let’s start with plain Rails app (I’m using 2.1.0 version): rails sqs. Now we need to install AM plugin (mind that I’m cursed and use Windows for development so watch out for / and \):

ruby script\plugin install http://activemessaging.googlecode.com/svn/trunk/plugins/activemessaging

Now, You have new generator available (processor), so let’s make use of it:

ruby script\generate processor Simple
      create  app/processors/
      exists  test/functional/
      create  app/processors/simple_processor.rb
      create  test/functional/simple_processor_test.rb
      create  config/messaging.rb
      create  config/broker.yml
      create  app/processors/application.rb
      create  script/poller

As usual generator has created for us whole directory structure and some config files. So let’s start with config/broker.yml. Active Messaging is by default configured to use Stomp, but we don’t want setup any Stomp server, we just want to use Amazon SQS. So go to Amazon Web Services and login or create account, get Your Access Key and Secret Key.

Comment adapter: stomp in broker.yml in development environment and uncomment first three lines in Amazon SQS part:

    adapter: asqs
    access_key_id: XXXXXXXXXXXXXXXXXXXX
    secret_access_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Provide Your keys of course…

Now we can define some queues in config/messaging.rb.

AFAIK Queue names has to be unique for single access key. For different environments (development, production) You need different queue names. If You want to use single AWS account across several apps, they also need to be unique. It can be achieved with embedding application name and environment into queue name (don’t use / as separator in queue name – it has to be alpha numeric, underscore and hypen also does work).

ActiveMessaging::Gateway.define do |s|
  s.destination :simple, "sqs-queue_simple-#{RAILS_ENV}"
end

That way we have defined queue with embedded application name (sqs) and environment name. This queue will be referenced in Rails code with :simple.

OK, now we need simple controller:

class SimpleController < ApplicationController
  
  include ActiveMessaging::MessageSender
  
  publishes_to :simple
  def index
    return unless request.post?
    publish :simple, {
      :message => params[:msg],
      :time => Time.now
    }.to_yaml
    render :text => "Message has been sent"
    return
  end
end

Controller need to import ActiveMessaging::MessageSender to have access to publishes_to and publish methods. First one let know AM that we will use :simple queue in this controller (line 5). In lines 8-11 we are sending data to queue. SQS does not care about data format, we can choose what we want, here we are using YAML. We are sending message text entered by user and time, all wrapped in hash as payload.

So now we need some processor code (app/processors/simple_processor.rb created by generator and updated to match our choices):

class SimpleProcessor < ApplicationProcessor

  subscribes_to :simple

  def on_message(message)
    data = YAML.load message
    logger.debug "#{Time.now} SimpleProcessor received: #{data[:message]} on #{data[:time]}"
  end
end

Essential is on_message method which takes message payload as argument. In our example we just log that we have received data.

To make things more complete we need simple view for SimpleController (app/views/simple/index.html.erb):

<% form_tag do %>
  Put Your message: 

  
<% end %>

So start our application, go to http://localhost:3000/simple type in message, submit and examine log/development.log:

Processing SimpleController#index (for 127.0.0.1 at 2008-08-21 23:17:10) [GET]
  Session ID: BAh7BzoMY3NyZl9pZCIlNGIyY2M2ODk3NmVhZTU1NWY4ODRmN2NlZTJmOTBl
M2QiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh
c2h7AAY6CkB1c2VkewA=--2b2b357fe88d22c3558e6fd2ca4b58349736eee4
  Parameters: {"action"=>"index", "controller"=>"simple"}
Rendering simple/index
Completed in 0.01000 (100 reqs/sec) | Rendering: 0.01000 (100%) | DB: 0.00000 (0%) | 200 OK [http://localhost/simple]


Processing SimpleController#index (for 127.0.0.1 at 2008-08-21 23:17:30) [POST]
  Session ID: BAh7BzoMY3NyZl9pZCIlNGIyY2M2ODk3NmVhZTU1NWY4ODRmN2NlZTJmOTBl
M2QiCmZsYXNoSUM6J0FjdGlvbkNvbnRyb2xsZXI6OkZsYXNoOjpGbGFzaEhh
c2h7AAY6CkB1c2VkewA=--2b2b357fe88d22c3558e6fd2ca4b58349736eee4
  Parameters: {"msg"=>"Hello NetManiac readers", "authenticity_token"=>"202cbdf538a14be776c099a2fa76facee18e27c6", "action"=>"index", "controller"=>"simple"}
Completed in 0.82100 (1 reqs/sec) | Rendering: 0.00000 (0%) | DB: 0.00000 (0%) | 200 OK [http://localhost/simple]

Web browser talks to us Message has been sent but there is nothing in logs? Well, apparently we need something what will run processors. Active Messaging has provided us with poller script, so run it (owners of UNIX-like systems with fork implemented can use start instead run):

ruby script\poller run

In logs we have first poller startup sequence:

ActiveMessaging: adapter reliable_msg not loaded: no such file to load -- reliable-msg
ActiveMessaging: adapter stomp not loaded: no such file to load -- stomp
ActiveMessaging: adapter wmq not loaded: no such file to load -- wmq/wmq
ActiveMessaging: Loading E:/NB/devel/NetBeans/sqs/app/processors/application.rb
ActiveMessaging: Loading E:/NB/devel/NetBeans/sqs/app/processors/simple_processor.rb
ActiveMessaging: Loading E:/NB/devel/NetBeans/sqs/app/processors/application.rb
ActiveMessaging: Loading E:/NB/devel/NetBeans/sqs/app/processors/simple_processor.rb
=> Subscribing to sqs-queue_simple-development (processed by SimpleProcessor)

and finally queue is being read and processed:

Thu Aug 21 23:20:32 +0200 2008 SimpleProcessor received: Hello NetManiac readers on Thu Aug 21 23:17:30 +0200 2008

Some remarks

I hope this is sufficient to show You that using AM and Amazon SQS is not complicated.

Benefits? Separation between user interface and time consuming operations. Clean and simple code.

Pitfalls? Queue perform best on short messages so don't pass huge amount of data, rather some ID from DB or paths to files to process.

If You plan to move existing operations in application from controller to processor, remember that in processor context You wont have access to many methods/variables which belongs to controller - so in that case You will need to write some workarounds or just write this code differently. Nothing hard, but take it under consideration when planing.

Active Messaging - must have in most Rails applications

With AM You can speed up Your development, use more elegant and clean application architecture. So next time I hope You will consider using it in Your app.