How did I overcome TDD fear

I was using Rails for over two years, but I still was not truly converted to TDD. There was no special reason for it, I have tried few times with my own projects to take TDD approach. And I was quickly returning to plain, old code crunching.

Why? Maybe I haven’t approached TDD with enough determination to change my own habits – when I was starting new application, just write somehow seemed faster to me. Later I was creating some tests, just to check some most important (IMO) pieces of code. The thing which seemed being biggest obstacle was keeping set of fixtures to have data on which I can run tests. Problematic for me was having up to date fixtures – corresponding with current data model. I don’t know how about You, but when I approach new application I do usually have data model prepared only as very rough concept. This means that during development changes are often and notable. Editing every time set of fixtures… well each time it was reason I stopped TDD approach with my projects. I know. Maybe I should prepare data model in more detail before I start?

Recently I was preparing to bear with other type of project. For existing application I was supposed to create API providing some core features as RESTful interface. This was meant as wrap-up layer over existing code, not providing new features. But… Usually there is some but. Old code was not written by me, it was crappy and most of logic was in controllers instead of models.

I knew I need some test support (old code had no running tests – there were created at some point RSpec specifications, but models had changed since then, and RSpec was not updated). In other words – I wanted it or not, I was designated to take TDD approach.

Ride with TDD!

Ride with TDD! (c) Lachlan Hardy

But this time I got instant gratification – every test I wrote,it helped me understand what is going on in old code and let me be assured that I’m moving application logic from controllers to models without altering it. And fixtures – I have extracted them from real application data. With not changing data model two biggest obstacles were removed and… From now You can me count as another TDD zealot. Yes. I do see how much I have gained using TDD approach in this project, and could not imagine how I could work without it before. Well, we learn all the time.

Do You want to write less fixtures?

When writing functional test I would like sometimes to call methods in other controllers. Why, would You ask? Well… In most of cases it is related to fixtures or rather to my averse to maintain fixtures (I told You!). In case of API I was talking earlier, there were several models acting as helpers – they were representing some statuses of internal processes. Based on them final object was created. Sure I could write fixtures for them. But my point is – why to create those fixtures (and overhead to update them when model changes) when they can be prepared in test process?

What I’m talking about? Let’s assume we are preparing some reservations – to create Reservation based on search terms and set of available rates first we create Search (describes search terms), Result (set of matching rates with changes resulted from applied business logic – discounts, supercharges, etc). With those objects we can create final Reservation.

More – for those interim models I have wrote some tests, but treating whole process as a black box I could test it more efficient – I provide input data and I expect some final result – which is much more similar how this objects will be really used.

The only question was how to call those methods in other controllers inside of functional tests?

Methods for calling methods (put, get) available in functional test take action of current controller as argument, not path to run through Rails routing system. After short googling and looking into ActionController::TestCase I have noticed that current controller is defined by @controller variable (given method is just send to @controller). So, changing @controller is enough (storing it earlier for later usage):

class ExampleControllerTest < ActionController::TestCase
    def setup
      @controller = ExampleController.new
      @request    = ActionController::TestRequest.new
      @response   = ActionController::TestResponse.new
    end
    
  should "call other controller before main test" do
   orig_controller = @controller
   @controller = SomeOtherController.new
   post :action_in_some_other_controller, {:argument => "value"}
   #get some data from @response
   #.....
   #now we can return to original controller - just re-run setup
   @controller = orig_controller
   #let's tests begin!
  end
end

As You can see I’m shoulda plugin fan :)

It has worked for me. What are pitfalls of this approach? I did not encounter any problems, but YMMV – I expect that re-using @response and @request can lead to some problems.

What else can be done to make fixtures more robust tool

I’m aware of Factory Girl, but from simple test I was conducting it is not huge time saver (however fixtures are in nicer format). I will take look at Machinist and try to write here more about FG and Machinist.

Any other ideas? Maybe there is some totally different way to accomplish my goal and I’m just not aware of it? I would appreciate Your opinions in comments.

Rake and arguments for tasks

Rake is beautiful tool, helping automate many tasks using Ruby. When You want to pass some arguments You can add VAR=value on command line and then in task use ENV['VAR'] to access passed value. But it is not what You alway want. Sometimes You want more compact and clean way to pass arguments.

Starting with Rake 0.8.0 You can create tasks with arguments:

task :name, [:first_name, :last_name] do |t, args|
     args.with_defaults(:first_name => "John", :last_name => "Dough")
     puts "First name is #{args.first_name}"
     puts "Last  name is #{args.last_name}"
end

Now You can pass some arguments:

C:\>rake name
(in C:/)
First name is John
Last  name is Dough

C:\>rake name[Jim,Weirich]
(in C:/)
First name is Jim
Last  name is Weirich

Cool. I guess I don’t need to explain how arguments and setting default values does work?

However I have noticed some error. First documentation skip needed key in example when task have some dependency on other task (:needs) and argument names should be provided not as an array (for task without dependencies both array and list form are working). Here is example of task with two arguments and one dependency, which does work (I’m using Rake 0.8.3):

task :name, :first_name, :last_name, :needs => [:other] do |t, args|
     args.with_defaults(:first_name => "John", :last_name => "Dough")
     puts "First name is #{args.first_name}"
     puts "Last  name is #{args.last_name}"
   end

task :other  do
end

What You can do with args?

Here it is modified task from Chad Fowlers Rails Recipes to dump data from database to fixtures. Handy to populate fixtures with some data entered with WWW interface:

desc 'Create YAML test fixtures from data in an existing database.
Defaults to development database. Set RAILS_ENV to override. Use args
table and limit to dump one table and limit number of records'
task :extract_fixtures, :table, :limit, :needs => :environment do |t, args|
  args.with_defaults(:table => nil, :limit => nil)
  limit = args.limit.nil? ? "" : "LIMIT #{args.limit}"
  sql = "SELECT * FROM %s #{limit}"
  skip_tables = ["schema_info" ]
  if args.table.nil?
     tables = ActiveRecord::Base.connection.tables - skip_tables
  else
    tables = [ "#{args.table}"]
  end

  ActiveRecord::Base.establish_connection
  tables.each do |table_name|
    i = "000"
    File.open("#{RAILS_ROOT}/test/fixtures/#{table_name}.yml" , 'w' ) do |file|
      data = ActiveRecord::Base.connection.select_all(sql % table_name)
      file.write data.inject({}) { |hash, record|
        hash["#{table_name}_#{i.succ!}" ] = record
        hash
      }.to_yaml
    end
  end
end

You can dump all fixtures or only single table and limit it to first X entries. I find it useful recipe, I hope it can be used by someone else, too.

Put’em in the queue

Rails is very good web framework. It works very well, as long You use it as it supposed to be used. Serving web applications means processing HTTP requests (often a lot of them) as fast as possible and send output to the browser. All is fine as long processing is simple and can be done fast. But sometimes You will need to make some operations which are more time consuming – and not because server is overloaded, but task itself takes longer. Like image processing or other operations on huge datasets. Then You will hit performance wall very fast. And it is not only Rails issue (BTW – did You know that upcoming Rails 2.2 will be finally thread safe?) it is problem for each web framework.

There is one simple solution – to remove time consuming data crunching from HTTP request processing and mover to some other process. Idea seems simple, but implementation can be more complicated. You need to provide some way to communicate for frontend and processing part.

When we have started Friends Feed Me, at moment Facebook Platform was presented to public, I did processing OPML files with external process which was using DRb to provide communication between Rails application and worker process. Well, even with such simple application it required to write a bit code. FFM did not take off, so we did not have scalability issues, but if it would taken off then using DRb would be problematic. Sure it can be done, but there are much simpler and proven solutions to this. I wish I had knew then about Active Messaging.

It is plugin which brings Rails much closer to messaging. What is messaging or event driven architecture? To put it in simple words – between frontend application (Rails) and worker process (or many of them) You insert black box which we call queue. Now tasks to process are sent to queue by Rails as fast as possible, and are taken from this queue by worker whenever it has free resources. That way Rails can send response to browser very fast (of course this is response data is being processed type, not final result) and worker will do it when it be possible.

Queue
(c) andrewparnell

Active Messaging is only one of many parts of this black box, provides communication interface with queue. Queue need to be implemented separately. Luckily for us, there is one tool which makes our black box complete without much overhead from us. It is service from Amazon called Simple Queue Service. If You are familiar with Amazon S3 then SQS is for messaging that S3 is for file storage. Simple API, with reasonable pricing (pay-as-you-go), ready to use instantly.

So what this is about?

I will provide here simple example how You can start using messaging in own Rails application. Example is simple and is meant to allow You start in minutes with own AM and SQS usage.
Continue reading

Shuffling RubyGems versions

I do some Facebook work for my customer, and when I checked out current source into my dev environment, application didn’t want to work :) Basically Facebook related code did not initialized properly.

I was curious what is causing it and finally gave up (RFacebook plugin upgrade did the job), but it seemed that it is something related to RubyGems version. So I did RubyGems downgrade and upgrade several times. I did it via fetching RubyGems source and ruby setup.rb.

I’m not sure if it is FreeBSD specific, but if after RubyGems upgrade (I did upgrade from 0.9.4 to 1.1.1) You are experiencing:

/usr/local/bin/gem:23: uninitialized constant Gem::GemRunner (NameError)

Then just delete /usr/local/bin/gem and symlink /usr/local/bin/gem18 as /usr/local/bin/gem.