Using Git Bisect for Finding When a Bug Was Introduced

Previously I wrote about updating a framework, automated tests, and included a brief mention of `git bisect`. I’d like to expand on the power of `git bisect` and your repository.

First a definition of the command:

git-bisect – Find by binary search the change that introduced a bug

The man page is quite helpful, but its application may not be immediately obvious. This has been my use case.

Step 1 – Prepare Your Bisect

$ git bisect start
$ git bisect bad
$ git bisect good 3f5ee0d32dd2a13c9274655de825d31c6a12313f

First, we tell git that we are starting the bisect. Then we indicate at what point we first noticed the bug – in this case HEAD. Finally we indicate at what point we were bug free – in this case at commit 3f5ee0d32dd2a13c9274655de825d31c6a12313f.

Step 2 – Start Your Engines

$ git bisect run ./path/to/test-script

We’ve told git-bisect which commits were good, and bad. Then with the above command, git will iteratively step through history and run the ./path/to/test-script. And what is ./path/to/test-script? It is any executable file that exits with a 0 status or not 0 status [learn more].

If the test-script exits with status 0, the current commit is considered good. Otherwise it is considered bad. Eventually git-bisect will converge on the commit that introduced the bad result and report the bad commit log entry.

Script Use Cases

So what is this ./path/to/test-script? Sometimes I’ve used `rake` for my Rails project. But that can be overkill. I’ve also ran bisect with one of the test files as the Good/Bad indicator (i.e. ruby ./test/unit/page_test.rb). In these cases, the tests were in my repository, which meant they were equally volatile.

I have also written a script that sat outside the repository I was bisecting. This was really powerful when my manager asked “When did this seemingly strange behavior get introduced?”

I wrote a Capybara test that automated the steps my manager reported for reproducing the error and an assertion of the expected behavior.

Sure enough, a few weeks prior, I had introduced the odd behavior as a side-effect. At the time I didn’t have test coverage for that particular behavior. I patched the error, answered my managers question, and had an automated test that I could drop into my repository to make sure I didn’t reintroduce that behavior.

As with most git utilities, I’m sure I’m only scratching the surface of git-bisect, but even with only using my above process, I’ve saved plenty of time and mental energy.

I have also created, long ago, a repository that highlights and automates several git commands. Follow the directions and it will walk you through a series of commits.

Updating Framework Versions – A Leisurely Stroll

I’m presently the primary application developer for conductor.nd.edu, map.nd.edu and a few other limited scope applications.  When I started working at the University of Notre Dame back in May 2009, I inherited these applications.

Fortunately, all of the applications I inherited had an automated test suite — a collection of programs that can be run to verify system behavior.  In the case of these applications, the test suites are developed and maintained by the application developer.

The test suites are a vital component of each of the applications I maintain.  Without them, I’d be lost.  Some test suites are better than others, but as I work on each application I also work to continually improve the test suite.

Recently, I just completed the process of migrating map.nd.edu’s authentication system from LDAP to CAS.  The advantages of CAS are pretty straightforward — I do not have to worry about maintaining the authentication component of each application.  This means I can remove code from the application — always a good thing. And the application doesn’t process user credentials, which eliminates one potential security hole from the application.

While performing this update, I also decided that I would update the underlying Ruby on Rails framework.  We were on version 2.3.5 and needed to increment to a more recent version — more recent versions of applications typically squash some bugs and close any discovered security holes.

The steps to increment the version were fairly simple, here is the script I followed:

10 increment version number
20 run tests
30 commit changes if tests were successful
40 update broken code and goto 20 if tests were unsuccessful
50 goto 10 if application's current version number is not latest version number

 

The key concept is that I walked from version 2.3.5 to version 2.3.14 by making sure 2.3.6, 2.3.7, etc. all passed their tests.  Never once did I open the browser during this process.

Once that was done, I began working on the CAS implementation.  Adding this feature went very smoothly.  When all of it was done, I began kicking around the application in the browser, making sure that things were working as expected.  I could automate these tests with Selenium, but have yet to invest the time in this tool.

I didn’t find any problems, but in an alternate universe, I’m sure there were issues.  After all, I had just incremented 9 minor versions of the application and implemented a whole new feature.

Enter Bizarro World

Let’s pretend for a moment that I did find problems.  It is possible the problem was untested from the beginning, introduced in the version update, introduced with the CAS feature, or something else entirely.

My first step is write a test that verifies the problem exists (i.e. the test fails).  With that automated test, I can begin to unravel what has happened. The first thing to do is go back to before any recent changes were made.  After all I want to know did my changes introduce the problem.

Given that I always use source control, it is easy to step back to a previous code state.

With the repository in it’s original state (i.e. before I incremented the Ruby on Rails version), I then run the test.  Did it still fail? If so, the changes likely had no effect.  If the test passes, then changes that I made broke the application.

Since we are still pretending, let’s pretend the test passed.  I now know that somewhere in my changes, I broke something.  At this point, I begin walking the repository one commit at a time to it’s most recent state (i.e. CAS has been implemented).  At each step, I run the test.  If it still passes, then I move to the following commit.  If it fails, I have found the commit that introduced the problem and can work to fix it.

Since we use git as our source control, I can automate the above process with the `git bisect` command.  I run `git bisect` by indicating where to start, finish, and what test to run at each step.  Then, I sit back and let my computer do the work. Note the test program that you are running will likely need to reside temporarily outside of the repository, as `git bisect` will checkout each version of the code.

Fortunately, in this universe, I didn’t encounter any of these problems, and instead was able to CAS-ify my first Rails application without breaking anything that I know of.