Quality Software and Safe Refactoring Done Right, (Hint: You need a lot of tests)

Building new software from scratch is one of life's great pleasures for a developer. It gives us a chance to "do it right" that we typically don't have when doing maintenance work. More often than not though, we find ourselves working on an older system and need to work around old decisions made by long gone developers.  For old applications that still serve the business requirements, maintenance is the name of the game.  We have to add new features, fix old bugs, and generally try to modernize the application in manageable steps.  Over time we end up with a codebase of mixed old and new code, and doing any serious refactoring can become a scary proposition.  Changing a few lines of old, uncommented business logic might fix the bug at hand, but then again it might release gremlins hidden in the code years ago.

There are a few simple practices, using free and open source tools, that can greatly simplify the problem and allow a developer to refactor old code with confidence.  In his book Working Effectively With Legacy Code, Michael Feathers defines "legacy code" as any code that does not have tests around it.  By eliminating legacy code as he defines it, we make the application more manageable.  The first layer of testing of course is a suite of unit tests.  This article is not about unit testing though, as that is often not enough to allow a developer to do the sort of large scale refactoring inherent in modernizing an old code base.  Instead we will cover some best practices using Selenium and Jenkins to ensure that releases are high quality and contain no regressions.

A few words about our infrastructure. We were already using Jenkins to compile and run all unit tests each time we push our changes. Since the Selenium tests take a while to run we made the decision to do so nightly as opposed to every commit. The Selenium tests are quite exhaustive, so we split them up to run concurrently across multiple machines, in order to complete in a timely fashion. The tests themselves are JUnit tests built with Selenium-java. We spin up a Selenium server instance for each run. We created separate Ant targets for deploying a test DB, compiling, deploying and starting the app on JBoss, starting the Selenium server, as well as running the Selenium tests. Since Jenkins works great with Ant (and Maven for those who use that) we did not have a hard time setting it up.

In order to get started we needed to write Selenium tests for every screen and feature in the application. That may sound like a daunting task, but since we are creating the tests programmatically we were able to get a lot out of code reuse. For example if you want to test 10 features on one page, you can reuse the all the code that gets you to that page and sets up the data in the background. Tasks like logging in, setting up common test data, and creating user accounts have all become one-line functions in our test code. 

One thing we did have to do some work in is creating functions for locating specific items in the DOM on webpages. At this point we have a large library of custom locator strategies coded up for our application. They save a ton of time as we can assert things like "Make sure the item in Row #5 in Column X is equal to 'foo'". We also use any downtime between releases to improve our testing infrastructure. 

As we wrote test code for the old parts of the application, we also resolved to write tests for every new feature as well as every bug that we fixed. This way we can be sure that the same bug does not reoccur in the future. In some instances we had to adjust the existing code to display id's and such so that Selenium would have an easier time picking up specific elements. These features are now second nature on the new pages we create. Every new feature we add has a corresponding Selenium test (or is tested as part of a larger test) before release. The only time we delay creating the Selenium test before delivering code to the client is if we are pushing out a hot fix fix for a production error.  It is a very rare occurrence since the product is throughly tested every night.

Our initial pass ensured complete coverage on the Firefox web browser. Even though Selenium supports all modern browsers in theory, the tests run in Internet Explorer were an order of a magnitude slower than the same tests in Firefox. Fortunately, since we have a lot of code reuse we were able to identify the bottlenecks and refactor them to boost the performance of the IE tests considerably. We recommend this approach of getting complete coverage on one browser first and then moving on to other browsers. We also recommend upgrading both Jenkins and Selenium. Selenium is constantly being improved to be more efficient and to support the latest browsers. We did have a couple of times where the Selenium version we had was incompatible with the very latest browser version, but these problems are often fixed by upgrading and refactoring some test code functions. 

Using this approach has a lot of benefits for the project. First and foremost, when we hand a release over to the client, we can be very confident that it will be of the highest quality. Indeed this has been proven over many releases, and most change requests we do get have to do with feature tweaking as opposed to functional errors in the codebase. Second, it creates the necessary safety net to allow us to do quite a bit of refactoring. We have refactored entire sections composed of multiple screens with several shared elements across the entire application with minimal consequences. All "feature level" bugs that can creep up when completely re-writing something for a different framework were caught by the Selenium tests.

The approach does come with several challenges. You will need to invest resources into creating the tests. This is a worthwhile investment that will save you a lot of time in the future, but it will require some work upfront. As stated before, Selenium tests are slow as far as tests go so as your suite grows you will have to contend with a long build cycle. Finally there are some hurdles with Selenium itself. These can be overcome through work-arounds, making your code more testable as well as upgrading.

Even though this through level of testing is not free, the benefits far outweigh the costs and we highly recommend this approach for your project as well.

The DataPhilly Meetups:

So many great user groups, but so little time!  This year I've started going to the DataPhilly meetups, and I think I'm hooked.  The bottom line is DataPhilly talks are very intriguing, expose you to topics you don't encounter everyday, and give you the chance to meet "non-traditional" developers (scientists and statisticians), whose ranks are rapidly growing.  

First off, a lot of the talks typically use Python (and sometimes R):  There are a ton of libraries in Python when it comes to statistics, data analysis, machine learning, NLP, etc. so that's not surprising.  What I like is getting exposure to a language I don't code in frequently and the ecosystem around it.  In addition, a lot of times the real gems are hidden in the questions and discussions during the talks and the lightning round sessions at the end.

What follows is a summary of the last meetup:

The first half was on mrjob (https://github.com/Yelp/mrjob), a pretty lightweight (from what I can tell) map reduce library written in Python that lets you run Hadoop streaming jobs locally, on Amazon EMR (Elastic MapReduce), or your own Hadoop cluster.  The talk started with word counting, which I swear is the "Hello World" of big data, but as promised quickly moved on to discuss detailed and specific examples (backed by code) of how its used at Monetate to generate product recommendations and gather statistics on user behavior.  On the surface it seems super easy to write the MR jobs.  The best part was when they discussed user behavior statistics and calculating variance: a Temple U. professor in the audience got on his soapbox and vehemently warned of the dangers of calculating variance using the "single pass algorithm" with single precision numbers due to underflow and overflow. (More on that at the end)

The second half was on Scrapy (http://scrapy.org/), a Python library - scratch that - actually an "appliance" to perform crawling and web scraping.  It was a super basic intro that went over its high level architecture, spiders, and how to parse the HTML.  For those who didn't know anything about it, it was a decent intro.  Of course "Little Bobby" asked the obvious question of how do you maintain the heaping pile of scraping code?  The speaker mentioned how Monetate used it heavily back in the day when they were smaller, but now some of their clients give them direct data feeds instead so they don't always have to resort to scraping.  Its sort of crazy to to think how much data is still "trapped" in web pages.  I don't think that problem is going away anytime soon. 

At the end of the night, two people presented during the lighting round:

1. "How airline crew schedules are made" - The speaker presented a modern twist on this combinatorial problem that now tries to minimize pilot fatigue when generating schedules.  He discussed the features of a schedule that take this into account (pilot awake time, trips at night, # of timezones crossed, etc) and combining these features along with empirical data on fatigue (cognitive tests based on sleep deprivation, etc.) to come up with a  "best fit" formula to use in a cost function.  Hey pilots are human - they get sleepy and jet lagged just like the rest of us!

2. The aforementioned Temple U professor quickly set up an example in R to back up his claim earlier, showing how dangerous it is to calculate variance with single precision numbers and using the single-pass algorithm - you can end up with NEGATIVE variance, which makes no sense.  He then showed it again using double precision and the two pass algorithm.  Here's a wikipedia link that describes it in more detail: http://en.wikipedia.org/wiki/Algorithms_for_calculating_variance  -  What I personally took away from this: when you perform anything more than very simple computations, know what the library does under the hood and use Google to find out if there are good numeric algorithms out there already.  Remember computers are limited precision machines!

BTW, last month, one of the talks was on Python scikit-learn (http://scikit-learn.org/stable/) - a machine learning library.  The example used Wikipedia articles written in different languages as training data into scikit-learn and showed a program that identifies which language an arbitrary body of text is written in.  It was a gentle intro to machine learning, "document vectorizing", and classification.

Here is a link to the meetup:  http://www.meetup.com/DataPhilly/

Seriously, go check it out!

5 Tips for Big Software Projects

Software development isn't easy.  And the bigger the software gets, the harder it is to build right from the ground up.  A string of high-profile failures has given us a timely reminder of this.  But it's not hopeless -- here are five things to consider to bump the odds in your favor:

1. Great software is never built on a poor architecture

The only way to build a large project is to divide and conquer.  But ten teams working in ten different ways will just produce ten times the mess.  The bigger a project gets, the more important it is to build on the best architecture and tools, modularizing where it makes sense without simply introducing more discrepancies and more complexity.  And the more people working on top of an architecture, the more important that it doesn't introduce obstacles that every developer may find different ways to avoid.

2. Generated code is terrible

Many situations land programmers with large amounts of tedious boilerplate code.  The gut reaction is to code-generate your way past it.  But if you think a junior programmer writes bad code, you should see a programmer try to program a computer to program.  It's just as convoluted as it sounds.  Worse, it has to be done over and over again, and it's one way to guarantee that some large chunk of code is rarely up to date.  Whether it's persistence or XML or other boilerplate code, there's a different set of tools that will eliminate the tedium with better code instead of worse code.

3. Best practices aren't always best

If you want to be really cruel, you can write terrible code that adheres to multiple best practices, and just watch an architect splutter.  The problem is that many individual best practices don't make sense when used outside of their associated tools and frameworks, or when used in combination, or when otherwise taken out of context.  Part of what makes experienced programmers better is that they can review all the tools and practices at their disposal, and use their judgement to select the right combination.

4. Software without automated tests is just a pile of code

A large project is built, never adequately tested, and rushed into production.  It doesn't work well.  Why is this a surprise?  But testing is also tedious, and requires a whole additional staff.  That is, if you don't automate it from the beginning.  Your computer is perfectly capable of driving an application with simulated load, an actual Web browser, or real-world integration requests.  There's no reason not to build a comprehensive automated test suite to ensure the software operates as intended.  And the more people who have their hands in the code, the more important it gets.

5. Integration begins with testing

Projects with many integration points tend to have a large number of interested parties, many of whom may be developing their parts of the system in parallel.  Then everybody finishes, and guess what?  It doesn't all work together.  But there's an easy way to avoid that.  If everybody starts by providing simulated requests and replies then testing can begin immediately, and each group can phase in real implementations over time.  Each system gets immediate feedback as to whether anything's breaking, and there's much less debugging required to fix a small change from a working state.

Come to Chariot's Data I/O Event on October 30th and learn about all-things data

We are holding an all-day event on October 30th, downtown in the Philadelphia Cira Centre, that shines a light on large-scale data processing and application management. In this article I'm going to explain a bit about the event's goals, and some information on the speakers and talks we've been lining up.

First, the logistics. You can find out the full details and register on this page, but here are the basics:

Date/Time, etc

When: October 30, 2013
Where: The Cira Centre, Philadelphia, PA
Cost: $80 - includes breakfast and lunch

Speakers include

• Camille Fournier - Camille is Head of Engineering at Rent the Runway, and a committer for Apache Zookeeper. She's focused on open-source infrastructure. Zookeeper is a set of glue services that keeps a distributed application platform like Hadoop running across multiple nodes.
• Lars George - Lars George is speaking about big data from experience. He's been working on the HBase, a Hadoop database tool, since 2007, serving as the EMEA (Europe, the Middle East, and Africa) Chief Architect for Cloudera. He also authored HBase - the Definitive Guide.
• Lance Ball - The word on the street lately has been all about node.js and Javascript. Node uses a great event-driven model to process web requests. However, while Node has gotten all the hype, Vert.x has been quietly creating a storm. Vert.x is also event-driven, but it easily scales - using the JVM and the Hazelcast-based clustering mechanism to quickly expand and manage application nodes. Vert.x isn't just programmable in Javascript, but in Java, Groovy, Ruby and even Python. It can run stand-alone, in clusters, even embedded in your own applications. Lance is the creator of Vert.x, which recently moved to the Eclipse Foundation where it enjoys a vendor-neutral standing. It is girded by the Java Netty fast native-I/O API. Come see how you can get Polyglot programming, Java, Javascript, AND scaling all in one platform.
• Max De Marzi - Max is passionate about graph-based data. Any items that can be related together in a relational model usually have inferred hierarchy, but are stuck in the world of the Tuple. In neo4j, the Graph-based database from Neo Technology, you design your datasets using objects and relationships in a graph-oriented way. Max will review some key use cases, such as social networking, recommendation engines, personalization, and a number of others to see how you can leverage Neo4j to quickly handle large, interconnected datasets.
• Claudia Perlich - Claudia was a keynoter at Chariot's 2013 Emerging Technologies for the Enterprise conference, where she talked about data science as an industry. She has a strong data analysis background, holding a PhD in Information Systems from NYU, and is the Chief Data Scientist at Media Six Degrees. Claudia holds several patents, has spoken at dozens of conferences and is an author on a number of papers, journals and books.
• Grant Ingersoll - As a founder of LucidWorks, he supports the active Lucene search community. He is a committer to Solr, which is an enterprise search platform that allows for full-text searching of large volumes of traffic.
• Eric Snyder Amazon Redshift is a SQL-based clustered data warehouse with instant on-demand scaling and pay-as-you-use techniques. Eric will show you the pros and cons of this service.
• Walt Mankowski - Angle - Walt Mankowski will review several Python APIs that can do math and science calculations at the speed of C/Fortran, but with simplicity of scripting. There is NumPy, a math library that can deal with multi-dimensional data, and SciPy, which extends NumPy with scientific calculation features.

We are awaiting abstracts from a few of our speakers, but hopefully this gives you an idea of the types of topics they'll be speaking about. For $80 and a day of your time, we are providing you with a day of talks in a number of key data analysis and processing areas. Sign up today and secure your seat.

Details and registration links

emergingtech.chariotsoutions.com/dataio2013.

Automated Testing of HTML5 Canvas/Single-Page Applications with Geb

This is the second in a series of articles about using automated testing tools on a Canvas-based Web application. Each article covers a different testing tool or technique. (See the first on WebDriver)

The specific application I'm testing is sized for a tablet, and uses a combination of a Canvas taking up roughly 2/3 of the space and a set of JQuery Mobile widgets taking up most of the remaining 1/3 of the space (there's also a small space on top showing a read-only "status bar" with key statistics). The basic concept is an editable diagram in the Canvas, with specific edit controls in the JQuery Mobile area. You can hover, drag, and click parts of the diagram in the Canvas, which can change what's displayed in the edit area, and changes you make in the edit area can add, remove, or change elements of the diagram in the canvas as well as updating the values shown in the status bar.

My goals for the automated testing are:

• Make it easy to write tests
• Run the tests in a real browser, to ensure I'm testing what a user would actually see
• Make it easy to run a suite of tests and drill into specific test failures
• Make the tests run fast enough that it's not terribly onerous to run before every commit
• Be able to verify the visible state of the HTML showing on the screen (e.g. in the edit area, in the status area)
• Be able to verify the underlying diagram model that dictates what is drawn to the Canvas. (I haven't attempted to go as far as testing the actual Canvas state -- as in the pixel color at some specific coordinate. The bugs I get aren't that the Canvas is drawing something that does not accurately represent its model state.)

Summary of Geb

Good: All the advantages of WebDriver (which it uses under the covers), plus lots of syntactic sugar for writing tests, plus you can write functional tests in Spock or traditional tests in JUnit, TestNG, or etc. using the lovely Geb syntax. (WebDriver launches the browser and runs through the tests, clicking on various things on the screen, waiting for screens to appear, executing JavaScript as needed, etc.) Supports multiple browsers.

Bad: There's no built-in way to record Geb tests by interacting with an existing Web site. Inherits the limited WebDriver support for interacting with a Canvas (requiring JavaScript workarounds). The documentation doesn't include the single simple step you need to drive single-page applications.

Bottom Line: All the advantages of WebDriver, plus fixes a lot of the WebDriver limitations. If I was considering WebDriver, I'd definitely use Geb instead.

Detailed Review

Instead of providing many language bindings like WebDriver, Geb focuses on producing an outstanding syntax using Groovy. On the one hand, it means that you have to learn and use Groovy. On the other hand, the syntax is really amazing. I think it's worth the trade.

However, most of the Geb examples assume you are using Spock and writing functional tests. The tests I'm writing are probably best considered integration tests -- they're long sequences of browser activity and I expect it all to work. I find this to be a better fit for JUnit (run a long sequence of mixed actions and validations) than Spock (break up the tests into small given/when/then type groups). So for my purposes I chose to write JUnit tests.

Installing Geb

I chose to set up a Maven project so I could run my tests via "mvn test" as well as generating an IntelliJ project and using IntelliJ to write my test code. The POM has several entries to include Geb, Selenium, JUnit, and Groovy, and to enable the Groovy compiler:

Geb JUnit Maven 3 POM

You should be able to run "mvn install" on that and at least confirm that the plumbing works.

Learning Curve:

Geb makes heavy use of Groovy closures, and the ability to invoke properties and methods that weren't explicitly defined. At first, I just followed the examples in the Book of Geb. Eventually things started to make a little more sense.

I'm not going to describe all the Geb syntax here, as the manual does a decent job of it. I'm just going to hit the things the manual didn't cover for me.

Some of my issues were:

• The manual does clarify how to get Geb to work with single-page applications. The issue for me is that page transitions are animated and therefore are not instantaneous, so the "at checker" ran before the new page was actually visible (and therefore failed).
• I had to piece together the config file since I didn't see a complete example
• I wasn't quite clear on how to spread my tests and page object across Groovy source files
• I wasn't quite clear on whether to use inheritance or modules to reflect that fact that the Canvas is visible for every "page" (meaning a JQuery Mobile page) of the application
• It wasn't obvious how to handle a Canvas click that caused a page change, since the basic click operation in WebDriver is broken for the Canvas (as described in the previous article)
• I wanted to automate the recording of tests like I did for WebDriver directly

Config File

Here's my Geb config file for Firefox. The last two lines are necessary if you don't want to configure them as system properties for all the different ways you might run the tests. Note the code within the "driver" block is running against the WebDriver Java API.

Single Page Applications

Since the browser never requests a new URL, there is no need to use "to" or "via" or set the URL for a page. You can simply point the browser to the base URL with go() and then navigate around within that.

It doesn't mean you don't use multiple "pages" within the Geb tests, though. I created a Page object for every JQuery Mobile screen. I just handle the page transitions via "at SomePage" rather than "to SomePage" to tell the test that I now expect a different JQuery Mobile screen to be showing even though the URL hasn't changed.

My first stab at an "at checker" looked like this:
That doesn't account for the page animation, though. The fix turns out to be easy:
That gives the page 5 seconds (by default) to show up, more than enough. Putting the wait logic in the "at checker" is important because some other things call the at checker as a side effect, and that would have to be avoided if the at checker itself didn't handle the waiting.

Many Pages Sharing a Canvas

Since my Canvas is always visible, no matter what JQuery Mobile page you're on, I ended up creating a base Page class with all the Canvas logic (clicking on the Canvas, clicking various widgets within the canvas such as the menu button or toolbar buttons shown there, etc.). All my other Page objects inherited from that one (except the dialog pages, because you can't interact with the Canvas while a dialog is showing on the screen).

Handling Canvas Clicks

I put my same JavaScript workaround for Canvas clicks in my base Page class:
Then I wanted my toolbar clicks to automatically change the current Page and run the new page's at checker, just like you can do with regular clicks in Geb. I ended up invoking the Canvas click manually, and then returning some innocuous page element that Geb could use its click-and-change-page logic on.

Normal link or button that forwards to a new page:
Toolbar button in the Canvas:
Since the content entry returns a legitimate Geb Navigator object (after clicking the Canvas), you can call click() on the content entry. When you call click on it, it changes the page per the to: NewPage directive (including calling the at checker on the new page), even though the click on #someBoringElement did nothing at all.

And the test ends up looking like this:
That's not strictly necessary -- you could put in all the at checks explicitly, and I suppose it could make the test clearer. But I prefer to have them all run as a convenient side effect of navigating.

Source Code Organization

I ended up putting all my Page definitions in a single TestSupport.groovy file. They were all pretty short and closely related, and I didn't want to clutter up my project with lots of tiny files. I think a consequence of this is that I needed to do a "full build" more often since the dependencies across files weren't as obvious, but builds were fast and that didn't bother me.
For my tests, I created several XYZTest.groovy files, since Maven by default runs tests with the *Test.* naming convention. It was easy to put several related tests in each file with the JUnit @Test annotation:

Recording Tests

There's an up side and a down side here. The up side is, I was able to record tests by including a custom "QA" JavaScript file in my app, which added listeners to buttons and links and pages and so on. It emits the appropriate Geb code to the Web console, which then I copy and paste into a test class. The down side is, it relies a lot on referencing page elements by their ID. That works fine, but it sort of misses the point of the PageObjects pattern. It's very easy in Geb to define all the page widgets in a content block, but when recording the tests, I know that an element with a particular ID was clicked, but I don't know what you decided to call that element in the content block of the page element.

It's possible I could settle on a convention of always naming elements in the content block the same as their ID, but even that avoids some of the nice syntax available with Geb.

For instance, I have a screen with four items in a glorified list, but every time you go to the screen, the specific content of the four items might be different (one screen tries to pick the best four for you, another is the same screen no matter which of many categories you're viewing, etc.). So the IDs are pretty generic, like ListItem0, ListItem1, ListItem2, ListItem3. It makes it easy to loop over them and so on.

I can set up a content entry in Geb like this:
Then my test can click an item and cause an implicit navigation to the detail page and run the at checker for the detail page all very easily:
But again, the test recorder doesn't know about that, so it can only generate code like this:
Does it do the same thing? Yes. But it's more lines, uglier lines, and the test would break if the list IDs ever changed -- exactly the reasons the PageObjects pattern exists.

So I end up with a mix of somewhat uglier tests that I recorded, and somewhat prettier ones that I wrote by hand.

Here's what the recorder script looks like now:
And a generated test:
Whereas a handcrafted test might look a little cleaner:

Running Tests

Running tests is as easy as "mvn test" or right-clicking on a test file or test method in IntelliJ (or however else you prefer to run JUnit tests). I get a summary of tests run, passed, and failed, green or red bars, etc. I find that IntelliJ is a little better at putting the specific error in front of me for a test failure, whereas I have to hunt a little more with Maven, but either way works. It's a big improvement over plain WebDriver code.

PhoneGap 3.0

PhoneGap 3.0 was released at PhoneGap day in Portland, OR last week. There are some great changes with this new release. Historically PhoneGap and Cordova, the open source project behind PhoneGap, were almost identical. This has changed with the release of 3.0.

Prior to 3.0, PhoneGap was distributed as an archive which was downloaded and unzipped to install. Now, both PhoneGap and Cordova are distributed as command line tools, which are installed via the node.js package manager, npm. Both Cordova and PhoneGap can be installed on your machine at the same time.

$ npm install phonegap -g
$ npm install cordova -g

The command line tools provide scripts to create projects, add platforms (iOS, Android, BlackBerry, Windows Phone, etc), compile and deploy projects. The new structure makes it easier to support multiple platforms with one project.

$ cordova create hello com.example.hello Hello
$ cd hello
$ cordova platform add ios
$ cordova platform add android
$ cordova emulate

With 3.0, the Cordova distribution provides the core functionality to embed a webview (browser) into a native application. Cordova is distributed without any of the plugins installed. This allows you to only install the functionality you need, simplifies your code base, and reduces the amount of code in your app.

To use the Camera APIs, we'd install the camera plugin.

$ cordova plugin add https://git-wip-us.apache.org/repos/asf/cordova-plugin-camera.git

Raymond Camden's blog post about Cordova 3.0 for a list of core plugins.

PhoneGap is a distribution of Cordova, similar to how Safari and Chrome are based on Webkit. The PhoneGap distribution of Cordova come with all of the core plugins installed. Additionally the PhoneGap command line adds features like PhoneGap build support. If you don't have a native SDK installed, the compiling the project can be delegated to PhoneGap Build.

$ phonegap create example
$ cd example
$ phonegap run wp7

Overall these are great changes for PhoneGap, but may require a changes to your workflow. The payoff is that it's easier to create projects, manage plugins, and support multiple platforms.

Android Advanced Cursors

If you have been doing Android development for even a little while, you have probably used the provided SQLite functionality to store and retrieve data in a local database on the Android device.

If not, you can head over to Lars Vogel's excellent tutorial website to see how it all works, or to get a refresher.

For many applications, the basic Cursor is all you need. But what about more advanced requirements? What follows is an overview of some of the more advanced cursor options.

CursorJoiner

If you need data from two or more database tables, the most straightforward thing to do is to use a SQL query that joins the two tables. However, sometimes you already have sources for two separate queries (such as a CursorLoader based on a ContentProvider), and you may wish to just re-use them, without creating a new query (without, e.g., modifying the ContentProvider).

This is where CursorJoiner comes in handy. Unlike the other options listed below, CursorJoiner doesn't produce a new cursor. Instead, it provides an Iterable interface over the joined cursors.

If the only reason that you want to join the two tables is to produce a Cursor, say, for use by a CursorAdapter, then you should probably just stick with writing SQL that joins the two tables. However, if you just want to iterate the data once, then CursorJoiner can be very helpful.

One of the more interesting uses that I've seen for CursorJoiner is for determining what has changed when a LoaderManager.LoaderCallbacks<Cursor> implementation gets an updated Cursor in it's onLoadFinished method.

You can use the CursorJoiner to join the 'old' cursor and the 'new' cursor on their primary key, and then compare them. As you iterate the CursorJoiner, you get a CursorJoiner.Result enum that indicates whether the current row exits in the left cursor, the right cursor, or both.

Here is some sample code that determines what has been added to or deleted from a ContentProvider as a CursorLoader driving a CursorAdapter is updated:

   
@Override
public void onLoadFinished(Loader<cursor> loader, Cursor updatedCursor) {
   if(adapter.getCursor() != null) {
      Cursor oldCursor = adapter.getCursor();
      CursorJoiner joiner = 
         new CursorJoiner(oldCursor, 
             new String[] {Thing.ID},
             updatedCursor,
             new String[] {Thing.ID}
      );
            
      for(CursorJoiner.Result joinerResult : joiner) {
         switch (joinerResult) {

            case RIGHT:
               int thingId = updatedCursor.getInt(
                  updatedCursor.getColumnIndex(Thing.ID)
               );
               
               String thingName = updatedCursor.getString(
                  updatedCursor.getColumnIndex(Thing.NAME)
               );
                      
               Log.d(TAG, "new Thing added - id: " + thingId 
                    + ", name: " + thingName);
            break;

            case LEFT:
               int thingId = updatedCursor.getInt(
                  oldCursor.getColumnIndex(Thing.ID)
               );
               
               String thingName = updatedCursor.getString(
                  oldCursor.getColumnIndex(Thing.NAME)
               );
                      
               Log.d(TAG, "Thing deleted - id: " 
                    + thingId + ", name: " + thingName);
            break;

            case BOTH:
               int thingId = updatedCursor.getInt(
                  oldCursor.getColumnIndex(Thing.ID)
               );
               
               String thingName = updatedCursor.getString(
                  oldCursor.getColumnIndex(Thing.NAME)
               );  

               Log.d(TAG, "Thing unchanged - id: " 
                    + thingId + ", name: " + thingName);
            break;                                   
      }
   }        
   updatedCursor.moveToFirst();
   adapter.swapCursor(updatedCursor);
}

MatrixCursor

MatrixCursor is useful if you have a collection of data that is not in the database, and you want to create a cursor for it.  You simply construct it with an array of column names, and optionally an initial capacity. Then, you can add one row at a time to it by passing either an array of objects or an Iterable to its addRow() method. Then use it like any other cursor:

  
String[] columnNames = {"Column1", "Column2"};
MatrixCursor matrixCursor = new MatrixCursor(columnNames);
matrixCursor.addRow(new String[]{"value1", "value2"});
adapter.swapCursor(matrixCursor);

MergeCursor

MergeCursor allows you to present two or more cursors as a single cursor. You could get almost the same effect by doing a sql UNION query. However, MergeCursor allows you to have different columns in the various cursors, and still merge them.

This is very useful if you are presenting a list of heterogeneous items. I've used it with Emil Sjölander's StickyListHeaders library, using a different cursor for each 'section', all presented to the StickyListHeader implementation as a single cursor via a MergeCursor.

One thing to be aware of when presenting heterogeneous items through a MergeCursor - you'll probably want to add a pseudo-column to each of the cursors to make it easy for your ListAdapter (or whatever will be using the cursor) to be able to determine which type of data it's dealing with on each row. A seemingly little-known trick here is that you can include literal columns when creating a cursor using the ContentProvider's query method (and similar methods in other database classes). Just include the literal in the projection parameter to the query:

  
String projection = {Thing.ID, Thing.NAME, "'TYPE_THING' as TYPE"};
Cursor c = qBuilder.query(mDb,
                projection,
                selection,
                selectionArgs,
                groupBy,
                having,
                sortOrder);

CursorWrapper

Finally, CursorWrapper is a Wrapper class for Cursor that delegates all of its calls to the actual cursor object. As the documentation stated, the primary use for this class is to extend a cursor while overriding only a subset of its methods.

One example for the use of CursorWrapper is when you already have a cursor, but only want to present the first few rows of the cursor to a CursorAdapter. Override the getCount() method to return a number smaller than the actual cursor's count, and the adapter will only present that number of results. This is similar to what you would have gotten had you added a LIMIT clause to the sql that produced the cursor.

With this collection of advanced Cursor tricks, you should be able to accomplish whatever you want with Cursors and Adapters.