I’m presently the primary application developer for conductor.nd.edu, map.nd.edu and a few other limited scope applications. When I started working at the University of Notre Dame back in May 2009, I inherited these applications.
Fortunately, all of the applications I inherited had an automated test suite — a collection of programs that can be run to verify system behavior. In the case of these applications, the test suites are developed and maintained by the application developer.
The test suites are a vital component of each of the applications I maintain. Without them, I’d be lost. Some test suites are better than others, but as I work on each application I also work to continually improve the test suite.
Recently, I just completed the process of migrating map.nd.edu’s authentication system from LDAP to CAS. The advantages of CAS are pretty straightforward — I do not have to worry about maintaining the authentication component of each application. This means I can remove code from the application — always a good thing. And the application doesn’t process user credentials, which eliminates one potential security hole from the application.
While performing this update, I also decided that I would update the underlying Ruby on Rails framework. We were on version 2.3.5 and needed to increment to a more recent version — more recent versions of applications typically squash some bugs and close any discovered security holes.
The steps to increment the version were fairly simple, here is the script I followed:
10 increment version number 20 run tests 30 commit changes if tests were successful 40 update broken code and goto 20 if tests were unsuccessful 50 goto 10 if application's current version number is not latest version number
The key concept is that I walked from version 2.3.5 to version 2.3.14 by making sure 2.3.6, 2.3.7, etc. all passed their tests. Never once did I open the browser during this process.
Once that was done, I began working on the CAS implementation. Adding this feature went very smoothly. When all of it was done, I began kicking around the application in the browser, making sure that things were working as expected. I could automate these tests with Selenium, but have yet to invest the time in this tool.
I didn’t find any problems, but in an alternate universe, I’m sure there were issues. After all, I had just incremented 9 minor versions of the application and implemented a whole new feature.
Enter Bizarro World
Let’s pretend for a moment that I did find problems. It is possible the problem was untested from the beginning, introduced in the version update, introduced with the CAS feature, or something else entirely.
My first step is write a test that verifies the problem exists (i.e. the test fails). With that automated test, I can begin to unravel what has happened. The first thing to do is go back to before any recent changes were made. After all I want to know did my changes introduce the problem.
Given that I always use source control, it is easy to step back to a previous code state.
With the repository in it’s original state (i.e. before I incremented the Ruby on Rails version), I then run the test. Did it still fail? If so, the changes likely had no effect. If the test passes, then changes that I made broke the application.
Since we are still pretending, let’s pretend the test passed. I now know that somewhere in my changes, I broke something. At this point, I begin walking the repository one commit at a time to it’s most recent state (i.e. CAS has been implemented). At each step, I run the test. If it still passes, then I move to the following commit. If it fails, I have found the commit that introduced the problem and can work to fix it.
Since we use git as our source control, I can automate the above process with the `git bisect` command. I run `git bisect` by indicating where to start, finish, and what test to run at each step. Then, I sit back and let my computer do the work. Note the test program that you are running will likely need to reside temporarily outside of the repository, as `git bisect` will checkout each version of the code.
Fortunately, in this universe, I didn’t encounter any of these problems, and instead was able to CAS-ify my first Rails application without breaking anything that I know of.