Tuesday, November 26, 2013

Quality Software and Safe Refactoring Done Right, (Hint: You need a lot of tests)

Building new software from scratch is one of life's great pleasures for a developer. It gives us a chance to "do it right" that we typically don't have when doing maintenance work. More often than not though, we find ourselves working on an older system and need to work around old decisions made by long gone developers.  For old applications that still serve the business requirements, maintenance is the name of the game.  We have to add new features, fix old bugs, and generally try to modernize the application in manageable steps.  Over time we end up with a codebase of mixed old and new code, and doing any serious refactoring can become a scary proposition.  Changing a few lines of old, uncommented business logic might fix the bug at hand, but then again it might release gremlins hidden in the code years ago.

There are a few simple practices, using free and open source tools, that can greatly simplify the problem and allow a developer to refactor old code with confidence.  In his book Working Effectively With Legacy Code, Michael Feathers defines "legacy code" as any code that does not have tests around it.  By eliminating legacy code as he defines it, we make the application more manageable.  The first layer of testing of course is a suite of unit tests.  This article is not about unit testing though, as that is often not enough to allow a developer to do the sort of large scale refactoring inherent in modernizing an old code base.  Instead we will cover some best practices using Selenium and Jenkins to ensure that releases are high quality and contain no regressions.

A few words about our infrastructure. We were already using Jenkins to compile and run all unit tests each time we push our changes. Since the Selenium tests take a while to run we made the decision to do so nightly as opposed to every commit. The Selenium tests are quite exhaustive, so we split them up to run concurrently across multiple machines, in order to complete in a timely fashion. The tests themselves are JUnit tests built with Selenium-java. We spin up a Selenium server instance for each run. We created separate Ant targets for deploying a test DB, compiling, deploying and starting the app on JBoss, starting the Selenium server, as well as running the Selenium tests. Since Jenkins works great with Ant (and Maven for those who use that) we did not have a hard time setting it up.

In order to get started we needed to write Selenium tests for every screen and feature in the application. That may sound like a daunting task, but since we are creating the tests programmatically we were able to get a lot out of code reuse. For example if you want to test 10 features on one page, you can reuse the all the code that gets you to that page and sets up the data in the background. Tasks like logging in, setting up common test data, and creating user accounts have all become one-line functions in our test code. 

One thing we did have to do some work in is creating functions for locating specific items in the DOM on webpages. At this point we have a large library of custom locator strategies coded up for our application. They save a ton of time as we can assert things like "Make sure the item in Row #5 in Column X is equal to 'foo'". We also use any downtime between releases to improve our testing infrastructure. 

As we wrote test code for the old parts of the application, we also resolved to write tests for every new feature as well as every bug that we fixed. This way we can be sure that the same bug does not reoccur in the future. In some instances we had to adjust the existing code to display id's and such so that Selenium would have an easier time picking up specific elements. These features are now second nature on the new pages we create. Every new feature we add has a corresponding Selenium test (or is tested as part of a larger test) before release. The only time we delay creating the Selenium test before delivering code to the client is if we are pushing out a hot fix fix for a production error.  It is a very rare occurrence since the product is throughly tested every night.

Our initial pass ensured complete coverage on the Firefox web browser. Even though Selenium supports all modern browsers in theory, the tests run in Internet Explorer were an order of a magnitude slower than the same tests in Firefox. Fortunately, since we have a lot of code reuse we were able to identify the bottlenecks and refactor them to boost the performance of the IE tests considerably. We recommend this approach of getting complete coverage on one browser first and then moving on to other browsers. We also recommend upgrading both Jenkins and Selenium. Selenium is constantly being improved to be more efficient and to support the latest browsers. We did have a couple of times where the Selenium version we had was incompatible with the very latest browser version, but these problems are often fixed by upgrading and refactoring some test code functions. 

Using this approach has a lot of benefits for the project. First and foremost, when we hand a release over to the client, we can be very confident that it will be of the highest quality. Indeed this has been proven over many releases, and most change requests we do get have to do with feature tweaking as opposed to functional errors in the codebase. Second, it creates the necessary safety net to allow us to do quite a bit of refactoring. We have refactored entire sections composed of multiple screens with several shared elements across the entire application with minimal consequences. All "feature level" bugs that can creep up when completely re-writing something for a different framework were caught by the Selenium tests.

The approach does come with several challenges. You will need to invest resources into creating the tests. This is a worthwhile investment that will save you a lot of time in the future, but it will require some work upfront. As stated before, Selenium tests are slow as far as tests go so as your suite grows you will have to contend with a long build cycle. Finally there are some hurdles with Selenium itself. These can be overcome through work-arounds, making your code more testable as well as upgrading.

Even though this through level of testing is not free, the benefits far outweigh the costs and we highly recommend this approach for your project as well.

Tuesday, November 19, 2013

The DataPhilly Meetups:

So many great user groups, but so little time!  This year I've started going to the DataPhilly meetups, and I think I'm hooked.  The bottom line is DataPhilly talks are very intriguing, expose you to topics you don't encounter everyday, and give you the chance to meet "non-traditional" developers (scientists and statisticians), whose ranks are rapidly growing.  

First off, a lot of the talks typically use Python (and sometimes R):  There are a ton of libraries in Python when it comes to statistics, data analysis, machine learning, NLP, etc. so that's not surprising.  What I like is getting exposure to a language I don't code in frequently and the ecosystem around it.  In addition, a lot of times the real gems are hidden in the questions and discussions during the talks and the lightning round sessions at the end.

What follows is a summary of the last meetup:

The first half was on mrjob (https://github.com/Yelp/mrjob), a pretty lightweight (from what I can tell) map reduce library written in Python that lets you run Hadoop streaming jobs locally, on Amazon EMR (Elastic MapReduce), or your own Hadoop cluster.  The talk started with word counting, which I swear is the "Hello World" of big data, but as promised quickly moved on to discuss detailed and specific examples (backed by code) of how its used at Monetate to generate product recommendations and gather statistics on user behavior.  On the surface it seems super easy to write the MR jobs.  The best part was when they discussed user behavior statistics and calculating variance: a Temple U. professor in the audience got on his soapbox and vehemently warned of the dangers of calculating variance using the "single pass algorithm" with single precision numbers due to underflow and overflow. (More on that at the end)

The second half was on Scrapy (http://scrapy.org/), a Python library - scratch that - actually an "appliance" to perform crawling and web scraping.  It was a super basic intro that went over its high level architecture, spiders, and how to parse the HTML.  For those who didn't know anything about it, it was a decent intro.  Of course "Little Bobby" asked the obvious question of how do you maintain the heaping pile of scraping code?  The speaker mentioned how Monetate used it heavily back in the day when they were smaller, but now some of their clients give them direct data feeds instead so they don't always have to resort to scraping.  Its sort of crazy to to think how much data is still "trapped" in web pages.  I don't think that problem is going away anytime soon. 

At the end of the night, two people presented during the lighting round:

1. "How airline crew schedules are made" - The speaker presented a modern twist on this combinatorial problem that now tries to minimize pilot fatigue when generating schedules.  He discussed the features of a schedule that take this into account (pilot awake time, trips at night, # of timezones crossed, etc) and combining these features along with empirical data on fatigue (cognitive tests based on sleep deprivation, etc.) to come up with a  "best fit" formula to use in a cost function.  Hey pilots are human - they get sleepy and jet lagged just like the rest of us!

2. The aforementioned Temple U professor quickly set up an example in R to back up his claim earlier, showing how dangerous it is to calculate variance with single precision numbers and using the single-pass algorithm - you can end up with NEGATIVE variance, which makes no sense.  He then showed it again using double precision and the two pass algorithm.  Here's a wikipedia link that describes it in more detail: http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance  -  What I personally took away from this: when you perform anything more than very simple computations, know what the library does under the hood and use Google to find out if there are good numeric algorithms out there already.  Remember computers are limited precision machines!

BTW, last month, one of the talks was on Python scikit-learn (http://scikit-learn.org/stable/) - a machine learning library.  The example used Wikipedia articles written in different languages as training data into scikit-learn and showed a program that identifies which language an arbitrary body of text is written in.  It was a gentle intro to machine learning, "document vectorizing", and classification.

Here is a link to the meetup:  http://www.meetup.com/DataPhilly/

Seriously, go check it out!