Automating Git Bisection for Rails Apps

Bisection is an awesome strategy for finding the introduction of a flaw. The basic idea is to recognize a failure in a particular version of your code, find a version where the failure did not exist, and use the SCM to automate finding change that introduced the bug.

I first used it in darcs a few years ago (where it's known as trackdown). mercurial and git both implement it as a bisect command.

While the concept is the same, the implementations vary across systems. In darcs, the trackdown command may only be used in an automated fashion (i.e. you have to write a test script), while in mercurial, the bisect command may only be used in an interactive fashion (i.e. you have to start bisection and manually test each revision as you go). git, however, supports both modes.

In practice, I find the darcs way generally preferable as it's faster (assuming you have a test ready) and harder to get wrong. Somehow, I manage to mark a revision as good when I mean bad or similar and have to start the whole thing over. In an automated mode, there's no thinking required.

Easy Case: An Existing Test Case That's Failing

If you have an existing test that's failing, you've got it quite easy. Find a version where it worked (we'll say HEAD~50) and just let it go:

% git bisect start HEAD HEAD~50
% git bisect run rake

That will spit out the change that caused the unit tests to start failing. If you've had multiple failures (or your tests are slow), you may want to tell it to just run a single test case:

% git bisect start HEAD HEAD~50
% git bisect run ruby test/unit/some_test.rb

Harder Case: Finding a Failure with a New Test

If the test didn't exist when the code was broken, bisection won't be helpful. I've found git stash to be very helpful in this case, however. Write the new, failing test case (that you believe would've succeeded before), and instead of committing it, just stash it (git stash) and write a quick shell script to run the test:

#!/bin/sh

git stash apply
ruby test/unit/modified_or_new_test.rb
rv=$?
git reset --hard
exit $rv

Once that script's in place (say /tmp/try.sh), you run the bisection as you normally would:

% git bisect start HEAD HEAD~50
% git bisect run /tmp/try.sh

A Really Hard Case: HTTP Request Needed to Show Problem

Recently, I had a bug in reloading a module in development mode that caused the second HTTP request sequence after a certain type of modification to attempt to reload a module that couldn't be reloaded. I wanted to bisect this, but I didn't want to use my browser and editor and stuff for every test during a bisection, so I automated it the following way:

#!/bin/sh

http_get() {
  curl -f -s $1 > /dev/null
  rv=$?
  echo "Requested $1 -> $rv"
  if [ $rv -ne 0 ]
    echo "Failed to fetch $1 (try #$2)"
    kill $pid
    exit $rv
  fi
}

http_sequence() {
  http_get http://127.0.0.1:3000/page1 $1
  http_get http://127.0.0.1:3000/page2 $1
  # [...]
}

# Start the dev server and capture the PID
./script/server &
pid=$!

# Give the server a chance to start before running sequences
waitforsocket 127.0.0.1 3000

http_sequence 1

touch app/[...]/somefile.rb

http_sequence 2

kill $pid
exit 0 # If we get this far, this version has no sequence bug

This script as my bisection command tracked down the first changeset with the reload issue very quickly and accurately. It's easy to adapt it to anything where you want to actually make an HTTP request and inspect the traffic/server/log/whatever.