Wednesday, September 26, 2012

So You Want To Build An Android App?

If you’re reading this, you likely have an iPhone/iPad (iOS) application and are interested in expanding to the Android platform. You've probably already asked the hard questions to define what you want, why you want it, and how it fits into your roadmap. This article will help your business understand the impact of the Android effort.

Android ≠ iOS

This should be obvious, but too often business think "I have an iOS app, how different can it be?" The challenges on each platform are not the same and some factors make Android applications a bit trickier to develop and maintain.

It is unrealistic to think that the exact time (or less) taken to build your iOS application can yield an Android app. The underlying assumption is that all the mobile problems were solved with the iOS effort and perfect conditions are now in place to start a new application. This thinking doesn't leave room to reevaluate infrastructure decisions that could impact the device or scale to new demands. It assumes that there are no developer or institutional knowledge gaps. Finally, it doesn't leave room evaluate the current set of features and how the design translates to the new platform.

Development & Testing

Android does not have a small subset of form factors and OS versions like the iOS ecosphere. There are a multitude of devices that officially (and unofficially) run the Android OS. The OS version distribution impacts both hardware and software features that you can leverage in your application. Also, there are many legacy phones still in use.

This increases complexity of creating an application that looks great on all devices. However, the Android platform has many ways to mitigate these challenges. It's critical to hire engineers and designers that have a good grasp of the platform so that you can reach a wide audience while preserving your brand.

Your organization should define a strategy around which devices to support (preferably when an Android expert is in the room). Different approaches come into play here, but a common one is to pick a representative sampling of devices and make sure the application works well on them. Whichever way you go, the decision will impact the level of testing and development effort.

This may seem daunting, but your application will have a much further reach - on everything from phones to televisions!

Design

There is tremendous value in making your users happy by giving them an intuitive and optimal user experience. This drives adoption and makes your users much more likely to continue to engage with your app over time.

You can’t simply take your current application’s look and feel and shoehorn it into an Android device. This is disrespectful to your users and makes it clear that you view Android as an afterthought. Navigation, user interactions, and widget standards are very different and should be respected on each platform.

It’s critical to evaluate your current features and determine how they will translate to the Android platform. The upside to this fresh perspective is that you will discover ways to leverage platform features that are unique to Android (e.g. rich notifications and widgets).
--
Bottom line: The design will be different, the code base will be different, and the challenges will be different. Make your users happy and follow Android design standards.

Are you building a team or an app?

Once an application is in the market, it will need care and feeding. There will be bug fixes and a growing list of new features to implement. The goal is to find an approach that fits into your mobile strategy and creates a sustainable process around the application. The main trade off is creating an organization in-house or using external firms. Let's look at both approaches in detail.

External team

Outsourcing could provide many advantages for your organization. Although consulting has a higher up front cost, you won't be paying to train and maintain a team. You pay-as-you-go and build only business critical features. The downside is that you’ll continue to pay the firm for maintenance, bug fixes, and feature growth.

Although it's tempting to think about this as something you can just hand off to another firm, you should be involved at all stages. Communication is key to making this approach successful because any external party won't understand your business as well as you do. Being involved in the design and feature prioritization process will ensure that the application created is what you envisioned. You don't want your app built in a vacuum and end up surprised by the end product!

There are several options, including hiring a contracting firm, a boutique firm, an independent contractor, or an overseas team. Contracting firms typically have several contractors that either create apps from scratch or embed with existing teams. Boutique shops have extensive experience with a particular platform and know how to best leverage its features. Independent contractors are typically self employed and work on several applications at once. Finally, you can hire an overseas team for cheaper than what you'd pay a local contractor.

Experience varies widely in the Android development realm. Make sure to ask the right technical questions (covered in detail in the next blog in the series) to ensure you are getting a high quality software product. Aggressively check references and contact past clients to gauge how happy they are with the end product. Install their published applications and check ratings and comments in the Google Play market. Finally, determine how maintainable and easy to extend the app will be (just in case you or someone else needs to update it).

In-house team

Creating a team internally makes sense for many businesses. It allows you to leverage your existing developers and have complete control over the roadmap. You won’t need to pay outside firms each time you want a new feature. Finally, you won’t have to manage the risk of having external developers create your app.

The drawback of this approach is creating and maintaining a team. It's a massive commitment and fiscal investment to support this team. You need to recruit, train, and have enough work for developers to do over the long haul. Depending on the scope of your application and your business objectives, creating your own team can be overkill.

Do you have the right knowledge to select developers to help you with this effort? There is no shortage of developers interested in switching to mobile development because it's an exciting corner of technology. However, it can be challenging to find the talent with the right experience. One strategy is to recruit internal developers and retrain them as mobile developers. Skills like Java and front-end development translate well to Android because it's written in a mixture of XML and Java. This technique is especially effective when combined with hiring an architect to help build the team.

Organizations tend to face the same challenges when beginning a team and an experienced architect can help you avoid these pitfalls. Team augmentation, which is hiring one or more experts to mentor new developers and increase the team Android knowledge, is one way to make this an easier process. Hiring this expert can help you get the right architecture in place, recruit new members, implement best practices, set up continuous integration, mentor team members, and inform the organization about the challenges and constraints of mobile development. Team augmentation can help you make a sustainable team much earlier. Plus there is an endpoint - you decide when to remove the training wheels.
--
Stay tuned for more articles in the series that will focus on choosing the best developers, designers, and testers for your Android project.

Tuesday, August 14, 2012

Going mobile? The key to any successful project is asking tough questions before you write code

The following is an article written by Don Coleman, Director of Mobile Development, which was originally published in the July issue of NJ Tech News.


About half of U.S. businesses will be in the mobile market by the end of 2012, according to a recent survey by Robert Half Technology (“Mad for Mobile.” New Jersey TechNews, April 2012). Nearly a quarter of the CIOs surveyed said they planned to develop a mobile application for the first time this year. The mobile boom is in full swing. Developing mobile apps can be scary and exciting, especially for managers who must make difficult business decisions in a rapidly changing technology landscape. Asking tough questions before you begin mobile development can save a lot of headaches down the road.

Are you focused?
It’s a good strategy to kick off a project with some blue-sky brainstorming. Sometimes ignoring reality is the best way to get all the ideas on the table. But the hard part is taking your big ideas and breaking them down into much smaller ideas that you can actually execute.

Do you have a clear idea of what you’re going to build? There is a wide spectrum of possibilities. You could be building a mobile version of your website. You could be building a task-specific app for clients or customers. Perhaps you are building a set of tools to help your employees work smarter and more efficiently.

Make sure you are not trying to do too much. Often it’s a good idea to start with the minimum number of features you need to satisfy your users. You can always add features later.

Probably the biggest thing to keep in mind is this: it’s not an effective use of resources to go mobile just for the sake of it. The rush to mobile now feels a bit like the dot com boom in the late 90s—building websites just to build websites. Be thoughtful. Focus.

Have you considered your audience?
This might seem like an obvious one, but managers and developers often make assumptions without really understanding their users.

If you are building something for your customers, do you know what they want or need? What is you target market? Do you have any studies to back this up? What devices do your users have? Android and iPhone are the most popular, but you might also have to consider other devices, such as the Blackberry or Windows Phone.

It’s also important to be realistic about how much time your customers will spend using your app. Some product managers have this fantasy that they will hold a user’s attention for an hour and a half. Try five minutes.


If you are building tools for your employees, do you understand their workflow? Do you know what kind of hardware they are using? Does the tool need to work offline and online? You want to build something that your employees will easily (and willingly) adopt. It’s important to understand how these mobile tools will be integrated with other software and hardware systems in your company.

Also, don’t forget about the bigger picture for your users. How are you making their lives better or easier? In addition to solving problems and meeting needs, you might consider ways to bring a little fun into their mobile experience. Create something that delights them.

Is your team ready?
This is a tough question, in part because it may force you to consider longer-term business strategies. Is your mobile project a one-time endeavor or do you intend to make mobile development part of your company’s skillset? Is it better to hire someone to build the app for you? Or are you willing to spend time and money training your existing team?

If you have good software developers they can become mobile developers. A consultant can help you plan the project and get your team up to speed. This could be a good strategy if you have developers willing and excited to learn. If your developers aren’t willing and excited to learn new technologies, you have much bigger problems.

Which technologies will you use?
In some ways, choosing technologies is the easy part—easier, at least, than understanding your users and getting developers up to speed. It’s also easy to get into the trap of using shiny, new, cool tools without putting people first.

The first decision you’ll have to make is whether to develop a native app or a web app. With a native app—for instance, one developed specifically for the iPhone or Android platforms—you can easily access each phone’s built-in devices, like the camera, microphone, and file system.

If you are going to build native applications for multiple platforms, start with one platform first, instead of trying to develop on multiple platforms in parallel. Your applications won't launch at the same time, but you can take what you learned on the first platform and apply it to the others.

The mobile web allows you to build applications that are not platform specific and can run in the browser on any phone. But browser capabilities vary across devices. Geolocation support is good. Camera support exists in newer Android and Blackberry but is unavailable on the iPhone.

One solution is to use a framework like PhoneGap that will allow you to build cross-platform native apps using HTML5. PhoneGap provides access to device features (camera and file system) that are not yet available in all web browsers. PhoneGap applications are packaged like native apps and delivered to users through the app stores.


The good news is that the line between web apps and native apps is blurring, and a year from now this distinction might not be a sticking point.

How much are you willing to invest in the project?
Obviously, developer and designer time will be the most expensive resource. All of the plumbing around the project—planning, marketing, and managing—can also add up. Building custom software is expensive, and your first mobile project could be more expensive than you think, especially if you have to train your team. The scope of the project and your vision for your business will likely determine the cost of going mobile.

A consultant could set you on a good path, but only if you do your legwork first. Although there’s no silver bullet here, asking tough questions before you dive in could lead to a more successful mobile app, one that makes your users happy and won’t break the bank.



Bio: Don Coleman, director of consulting at Chariot Solutions, helps businesses put together clear roadmaps for mobile development. He loves equally solving interesting technical problems and making businesses work better.

Tuesday, August 7, 2012

Android Unit Testing With Robolectric

Are you interested in adding Robolectric to your development process? Here's a quick introduction to the most common types of test cases and some best practices to consider when deciding your testing strategy.

I've created two example apps for your reference. The first is a glorified Hello World and the other is a simple animation app. If you are unfamiliar with Robolectric, I'd start with my introductory blog that details how to integrate IntelliJ, Robolectric, and Maven into your project.

What To Test

You should focus your unit testing on business logic, basic view validation, and functionality. You can validate certain requirements very well with unit testing (e.g. "when I click this bell image, it whistles" or "all activities have a logo on the screen").

Here are some common examples:
  • Ensure the activity and view elements are not null.
  • Ensure expected view elements are visibile on the screen.
  • Ensure that the proper resources are used for view elements.
  • Ensure items are hidden when expected.
  • Ensure that your intent starts the correct thing.
  • Ensure that fragments are present.

What Not To Test

I don't recommend investing much time in verifying less important styling details and XML attributes. Your user interface is likely to change many times a week, but typically the basic view elements and their behavior remain the same. If you updated a test each time a button had new padding, that would get very tedious indeed!

Example: The app logo is 250dp high with a proportional width, the file name is "main_logo.png", it has a 20dp left margin and 12dp right margin, has a content description of "Text," and is centered vertically on every screen.

You are likely able to test all of the attributes above (I haven't tried), but your time is best spent on items that are most important to the business. From this example, likely the most bang for your buck will come from testing that the proper image is used and making sure that it appears on all expected screens.

Returns start diminishing pretty quickly when you verify every possible XML attribute - unless that detail has an important business outcome for your app. Many of the attributes you don't test can be caught by designers, quality assurance, business, product, and/or the engineers. A better stategy is to sweep the app near deployment with a designer to make sure the app appears as expected on a wide range of devices.

Best Practices

  • Simplify your architecture. As with most object oriented realms, attempt to simplify the architecture as much as possible. Do this by creating abstract or base objects or interfaces. This way you can test shared behavior in one place instead of several, which simplifies tests and reduces code duplication.
  • Mock/stub functionality. Often we want to ensure that a certain action was performed but not actually run the code. To do this, you create a test class which extends the class under test and override/mock out/track that a particular function was called. For example, if you want a particular click to make an API call, there is no need to make the actual call (since this would be an integration level test). You can use this strategy to track that the function that invokes the call has been called, but avoid making the actual call. This is also helpful when an operation crashes the unit testing framework (e.g. specialty library that causes funky side effects), doesn't play nice, or is otherwise unnecessary to invoke directly.
  • Create a custom test runner. Create a helper test runner to gives your entire unit test suite access to handy functionality. My simplest version includes string, drawable, and color resource fetching as well as view visibility helper functions. I also add a helper function to start fragments without a hosting activity. Finally, this is a good place to bind custom shadows you create and set up any external library configurations that you may need.
  • Centralize test constants. Android strongly encourages the use of abstracting string resources into one place over using hard-coded string constants. Use a test constants class when you have lots of test data (e.g. When creating a user with lots of fields to mock out a JSON response).
  • Have a robust testing strategy. Don't rely solely on unit tests to ensure the proper implementation of your app. These tests, though quite useful, only give you a small window into your app and it's behavior. At minimum, you should include integration level tests and have some level of manual QA in the process to ensure your app is everything that you want it to be.

Notes

  • Unit testing Android apps is not as simple as with other platforms and Robolectric is still a work in progress. You will find things that you'd like to test that the library doesn't suport. It's up to your organization to determine how much they would like to invest to give back the the community by writing and fleshing out shadow classes.
  • When running the sample tests you may notice a few warnings about binding map-related shadow classes (because the library is not available in the maven central repository). To solve this issue use the maven android SDK deployer.
    Warning: an error occurred while binding shadow class: ShadowGeoPoint
    Warning: an error occurred while binding shadow class: ShadowItemizedOverlay
    Warning: an error occurred while binding shadow class: ShadowMapController
    Warning: an error occurred while binding shadow class: ShadowMapActivity
    Warning: an error occurred while binding shadow class: ShadowMapView
    Warning: an error occurred while binding shadow class: ShadowOverlayItem
I hope you found this blog useful for getting started with Robolectric! If you are already using it, what do you find most useful or helpful? How does it fit into your testing strategy?

Friday, August 3, 2012

Integrating Robolectric with IntelliJ

Unit testing is a best practice, but has been particularly difficult in Android. One library that makes this process much simpler is Robolectric. Integration into your project can be tricky and most methods start with maven and create an Android project from scratch.

I'm a huge fan of living in IDE-land as much as possible, so this approach minimizes command line work and manual project configuration. This tutorial will walk you through using IntelliJ to generate the project and maven integration to make installation of Robolectric simpler.
After creating an Android project with the wizard, you move the source files into a mvn-like structure. Then you configure module settings in IntelliJ. To install Robolectric, you will add a pom file (maven dependency file) into the project. Finally, you will write your first test and run it inside the IDE.

Note: There are other projects and methods out there, but this is what I found to be the most effective way to get up and running. If you'd prefer an all in one solution, check out this project.

Assumptions:

  • You are starting a new project.
  • You are familiar with the IntelliJ and the Android wizard (CE edition is fine).
  • You have installed the Android SDK and configured IntelliJ to use it.
  • You have maven installed.

Step 1: Create Android project using IntelliJ wizard

Use the IntelliJ Android wizard to create a Hello World project.

Step 2: Set up directories for maven

Use IntelliJ or the command line to move files into the following structure:

src/main/java/com/...
src/test/java/com/...

After you are done, your project should look something like this (minus the test files):
Note: IntelliJ might not like it when you add subdirectories to the folder marked as source. If you are having trouble setting up the directory structure, open the screen shown in the next step and remove the old source folder before adding new subfolders.

Step 3: Configure IntelliJ source and test folders

Update module settings in IntelliJ to recognize the source and test directories (below: blue folder is source and test folder is green).


Step 4: Install Robolectric

Use this sample pom.xml to get you started. Change the groupId, artifactId, version, and name tags to suit your project. The file should live in the root of the project.
And some of my environment set up (zsh):
# Shell Settings (one long line, broken for view online)
export ANDROID_HOME="/usr/bin/android-sdk-macosx"
export PATH="/usr/bin/android-sdk-macosx/platform-tools:/usr/bin:
             /usr/bin/android-sdk-macosx/tools:
             /usr/local/apache-maven-2.1.2/bin:$PATH"
After updating the pom and environment variables, check the project in IntelliJ to make sure the "Maven Projects" view looks something like this (you may need to refresh):


If you see errors, you may just need to wait for it to finish indexing and importing your libraries. If you have git configured, once you see this dialog, you should be moments away from running your first test.

Step 5: Create test

I use the IDE to generate a test. On a Mac, the key combination is Apple+Shift+T.

Step 6: Write test & run

Right click on the test file name to run the test or use keyboard shortcut. TODO - resolve - If you are seeing issues similar to this, rebuild your project a few times to convince IntelliJ to generate the R.java file (which links to all your applications resources) and/or run "mvn install" from the command line.

If you are still having issues, update your Android facet here:
Success!

Optional Step 7: Create a run config for all tests

You could run each test individually as you work with new classes, however it should be a common task to run all the unit tests (e.g. prior to commit). I create a run configuration for the whole project:  


Next Steps

Now that you've got it integrated into your project, why not try adding a few tests? Check out the next blog in the series that shows you how to write test for your project.

Monday, July 30, 2012

Using Cocoapods to Manage Private Libraries

Introduction

Cocoapods (http://cocoapods.org/) is a dependency management framework for XCode. It allows you to declaratively define project dependencies and have them included in the build of your project. It's like Apache Maven or Ruby Gems for XCode. There are many tutorials on getting started with Cocoapods, so in this blog post I will present a simple case where we have a project that depends upon two internally developed projects or private libraries.
We will start by creating two projects, each with a simple UI screen and some 3rd party dependency. Each of these projects will contain a file known as a Podspec that defines what gets included when another project depends upon it. Next, we will create a third project that depends upon these internally developed libraries and see how everything gets tied together.
In my contrived example, I'll create two projects, each with a dependency on AFNetworking (a nice network abstraction layer). AFNetworking is already available via the public cocoapods spec library. Cocoapods will be used to add this dependency. Each project will also use YQL to query a weather service (Yahoo versus Wunderground) and present the forecast for a given zip code. The master project, which I like to call the Weather Comparator, will pull in the Yahoo and Wunderground projects and use their code to present a comparison of the services.

The Private Libraries

We start by creating the first project, Yahoo Weather. Once we have Cocoapods installed on our system, we create an XCode project for our first framework. I started using the Empty iOS application template. Once the XCode project is created, exit XCode. Create a text file called "Podfile" at the root directory of the project. The contents of the Podfile looks like the following:
platform :ios, '5.1'

xcodeproj 'YahooWeather.xcodeproj'

pod 'AFNetworking'
The first line defines the platform and SDK version, the next line defines the name of the XCode project file, and subsequent lines define the dependencies for this project. Podfiles can be much more complex, but this post will keep things simple. Check out the Cocoapods site for more information. Once the Podfile is complete, drop to a terminal and execute pod install at the root of the project directory. You'll see some messages about the dependencies being downloaded and configured and when the process is complete, there will be a message indicating NOT to use the YahooWeather.xcodeproj anymore, but use the YahooWeather.xcworkspace file now. This is a workspace that includes your project and the Pods project.
Under the Pods project, there is a Pods directory, which contains the libraries specified in the Podfile (AFNetworking in this case). Besides creating the workspace, Cocoapods configures your project to include a static library called libPods. This is the output of the Pods project build. Essentially it includes the compiled version of the pods for which you asked to be included in the project. We can now use the AFNetworking classes within our code. The source code for the YahooWeather project can be found on my github account: https://github.com/stevenpsmith/YahooWeatherService. The finished app can actually be run and looks like this (I know, gorgeous, right):
Following the same steps, we create a second project called WunderWeather. Much of the code is similar, except now we are using the Wunderground weather API. There are some other small changes, but for simplicity sake I reused much of the same code, just using different names for the classes (notice the class name prefix). Again, remember this is a contrived example, so reusing the code was merely for time efficiency. The source code for WunderWeather can be also be found on my github at https://github.com/stevenpsmith/WunderWeatherService. And for your viewing pleasure, here is what it looks like:
One important thing to note here, is that each of our frameworks uses a slightly different class prefix. This ensures that when both of these projects get used in a third project (the WeatherComparator), there are no name collisions. Wouldn't namespaces be nice?
When we are done coding each of these projects, we can commit them to our "private" git repository. In this case I am using github public repositories, but these projects can reside anywhere that is accessible from the projects that will depend upon them. That could a private repo, file system, etc. Cocoapods also support SVN and Mercurial, but git is what all the cool kids are using, so that is what this example will use.

Creating the Podspec

The next step is to configure a podspec for each of our frameworks. The podspec defines the source files, resources, etc. that our libraries/frameworks expose for use by other projects. In our case, we need to expose all the classes in the API, Model, and ViewController directories within our projects. We also want to expose our XIB files. Those are resources which need to get compiled and added to our final ".app" file. In a similar way, resources might include images that will get added into the ".app" file during the final build process. To create a podspec, execute pod spec create projectname at the root of the project directory. In our case, it would be pod spec create YahooWeather or pod spec create WunderWeather
The file that gets created has comments and examples of all the properties that can be set. For our projects, the podpsec looks like this (with all the extraneous comments and unused attributes removed):
Pod::Spec.new do |s|
  s.name         = "YahooWeather"
  s.version      = "0.0.1"
  s.summary      = "Provides a Yahoo weather forecast, with basic UI, for a given zip code."
  s.homepage     = "https://github.com/stevenpsmith/YahooWeatherService"
  s.license      = 'MIT'
  s.author       = { "stevenpsmith" => "ssmith@chariotsolutions.com" }
  s.source       = { :git => "https://github.com/stevenpsmith/YahooWeatherService.git", :tag => 'v0.0.3' }
  s.platform     = :ios, '5.1'
  s.source_files = 'YahooWeather/API/*.{h,m}', 'YahooWeather/Model/*.{h,m}', 'YahooWeather/ViewController/*.{h,m}'
  s.resources = "YahooWeather/ViewController/*.{xib}"
  s.requires_arc = true
  s.dependency 'AFNetworking'
end
Notice the source property. This points to our "private" repo. You can also access local repositories (instead of server based repos) by using a URI like this: '/Users/steve/projects/foo', which points to the root directory of your version controlled code. Also note the use of the tag key (which could also be a :commit). If consumers of this library do not specify a tag or commit, this specifies the snapshot of the code that will be provided. Without specifying anything, the latest commit of the master branch will be used. The source_files and resources specify the assets within this project that will be provided to the consuming projects. The dependency property specifies any pods that this pods would depend upon (in our case, AFNetworking). This file needs to be committed to the repository, and if a tag or commit is specified, that must exist within the repo as well. It's a good practice to validate the podspec by executing pod spec lint <podspec_name>
This will save a lot of grief with typos, incorrect source directories, etc. Once your podspec is clean, commit it to the repository and tag if needed. The podspec for WunderWeather looks eerily similar.

Using the private libraries

The next step is to use these libraries in a new project. This is the WeatherComparator project, and it's source can be found at https://github.com/stevenpsmith/WeatherComparator. We get started like we did before, creating the project within XCode, then exiting and creating a Podfile that references the dependencies. Except this Podfile will look a little different, as shown below:
platform :ios, '5.1'

xcodeproj 'WeatherComparator.xcodeproj'

pod "YahooWeather", :git => 'https://github.com/stevenpsmith/YahooWeatherService.git', :tag => 'v0.0.3'
pod "WunderWeather", :git => 'https://github.com/stevenpsmith/WunderWeatherService.git', :tag => 'v0.0.2'
Here we are adding the :git key to the pod declaration. Again, this could be subversion, mercurial, and even a filesystem reference. Notice there are no dependencies on AFNetworking. Our new project doesn't directly depend upon those, but since the pods do, it will get pulled into our new workspace as it did in the other projects. Execute pod install in a terminal at the root of the new project directory, open the workspace as we did before, and we are good to go. Here is a screen shot of our weather comparator:
If you take a look at the code in the WeatherComparator project, it is very simple. One interesting thing to look at is the Pods-resources.sh script located in the TargetsSupportFiles/Pods directory within the Pods project. Cocoapods generates this script file based on the contents of the podspec for the libraries. This script actually compiles the XIB files and adds them to the projects final build. As you can see, pods can use storyboards as well. This provides a nice way to create modules that include UIs and then combine them into applications.
Even though we used github public repositories, this post shows how to use Cocoapods to manage your project's public and private dependencies. To try out the project, install Cocoapods, download the WeatherComparator project, execute pod install at the root, open the workspace, run the project, and see how different Yahoo and Wunderground weather forecasts can be. Pretty surprising actually :).

Thursday, July 12, 2012

Using Maven to Integrate Flyway in an Existing Database-
Part 2

This is the second article of a two-part series on using Maven to integrate Flyway in an existing database. It addresses several key issues including using Maven profiles to manage promotion of migrations to hierarchical levels of a database as well as managing migrations across multiple disparate projects that use the same database.

Unless you are an experienced Maven and Flyway user, it is suggested you review the first article of this series, Using Maven to Integrate Flyway in an Existing Database- Part 1, as a foundation for the details presented in this article. You can find it here. We will be using the same project as an example.

Continuing Flyway’s Configuration…


As we stated in the first article, Flyway has proven to be a good tool for managing database versioning and migrations, but it lacks a “built-in” facility to manage migrations to multiple levels of the same database, e.g., development, QA, staging, and production. However, we did learn the concept of using configuration parameters to communicate database specifics such as the JDBC driver, database URL, database user name, and user password among others to the Flyway tasks.

Unfortunately, the format of the configuration medium varies depending on the method used to employ Flyway. So depending on which Flyway component(s) you use, you may need to maintain several versions of the configuration data.
  1. The Flyway Maven plugin configuration may be specified via system properties or via the <configuration> group of tags inside the definition of the plugin in the <pluginManagement> section of a parent POM or in the <plugins>section of the Maven POM for the build in which the plugin is to be executed.
  2. The Flyway Ant task configuration may be specified via system properties or via either task attributes or specific configuration tags inside the execution of the task in the Ant build file.
  3. The Flyway Command Line Tool uses a properties file in a specific location within a specific directory structure.
  4. And finally, the Java migrations may set a Properties object on the Flyway core class object or set them individually on the object.
Although system properties could be used to convey the database specifics to most of the Flyway components since they all reference the same system properties, that would preclude running the Flyway component on the same system for more than one database, unless, of course, the execution script itself sets the system properties. This would include multiple levels of the same database.

At a minimum, you should maintain at least one set of configurations for each of the levels of the database for which you are intending to automate migrations. You should also devise a methodology to use those differing configurations in executing the Flyway tasks against the different levels of your database. In this article we are going to modify the sample project we used in the first article to use Maven profiles for this purpose.

Maven Profiles to the rescue…


Again, this is not a tutorial on the purpose of or how to use Maven profiles. Suffice it to say Maven profiles can be used for a myriad of things, one of which is the setting of property values. This permits one to define properties and values in a profile and then use that profile for a given execution of Maven. Maven then substitutes the reference of the property in all POMs of the execution with the value specified in the profile.

Maven profiles can be defined in two contexts: in the Maven settings file and in a POM for a particular project. The obvious advantage of defining a profile in the Maven settings file is that it can then be referenced in any Maven execution on the system on which it is defined. However, that may be a disadvantage as well in that, generally, properties whose values are specific to a project should be defined within the project’s POM, not in a global settings file.

So we modified the parent POM for our project adding four profiles, one for each of the levels of our database: development, QA, staging, and production. Within each of these profiles, we defined properties for each of the Flyway configuration items that may differ from one database level to the next, specifically the JDBC driver, the database URL, the database schema, the database user credentials, and the database name.
NOTE: We have also employed the Maven password encryption feature to eliminate clear text passwords in our Maven POM file (see Password Encryption in the Maven Guide). When using this feature, you must encrypt a master password and define it in the Maven security-settings.xml file as well as encrypt the password for each individual database. The database passwords as well as the database user names are then specified via a <server> definition in the Maven settings.xml file. The id of the server is then specified via the <serverId> Flyway <configuration> parameter in place of the individual database user's credentials.
We added these server definitions to the <servers> section of the Maven settings.xml file:
<settings>
   . . .
   <servers>
      <server>
         <id>flyway-dev</id>
         <username>SA</username>
         <password>{QUXCK6DU8FtcS5P=}</password>
      </server>
      <server>
         <id>flyway-QA</id>
         <username>SA-QA</username>
         <password>{AXWK8jU8ob5dse#}</password>
      </server>
      <server>
         <id>flyway-staging</id>
         <username>SA-ST</username>
         <password>{MCVFo59x0#sdlk=}</password>
      </server>
      <server>
         <id>flyway-prod</id>
         <username>SA-PR</username>
         <password>{ZYpl57s2kchD3%}</password>
      </server>
   </servers>
</settings>

We added these profiles to the <profiles> section of our parent POM:
<profiles>
   . . .
   <profile>
      <id>flyway-dev</id>
      <properties>
         <database.name>flyway_dev</database.name>
         <database.driver>
            com.microsoft.sqlserver.jdbc.SQLServerDriver
         </database.driver>
         <database.url>
            jdbc:sqlserver://localhost:1433;databaseName=flyway_dev
         </database.url>
         <database.serverId>flyway-dev</database.serverId>
         <database.schemas>dbo</database.schemas>
      </properties>
   </profile>
   <profile>
      <id>flyway-QA</id>
      <properties>
         <database.name>flyway_QA</database.name>
         <database.driver>
            com.microsoft.sqlserver.jdbc.SQLServerDriver
         </database.driver>
         <database.url>
            jdbc:sqlserver://QAserver:1040;databaseName=flyway_QA
         </database.url>
         <database.serverId>flyway-QA</database.serverId>
         <database.schemas>dbo</database.schemas>
      </properties>
   </profile>
   <profile>
      <id>flyway-staging</id>
      <properties>
         <database.name>flyway_staging</database.name>
         <database.driver>
            com.microsoft.sqlserver.jdbc.SQLServerDriver
         </database.driver>
         <database.url>
            jdbc:sqlserver://stagingDB:1040;databaseName=flyway_staging
         </database.url>
         <database.serverId>flyway-staging</database.serverId>
         <database.schemas>dbo</database.schemas>
      </properties>
   </profile>
   <profile>
      <id>flyway-prod</id>
      <properties>
         <database.name>flyway_prod</database.name>
         <database.driver>
            com.microsoft.sqlserver.jdbc.SQLServerDriver
         </database.driver>
         <database.url>
            jdbc:sqlserver://prodDB:1040;databaseName=flyway_prod
         </database.url>
         <database.serverId>flyway-prod</database.serverId>
         <database.schemas>dbo</database.schemas>
      </properties>
   </profile>
</profiles>

We also changed the configuration in our plugin definitions to reference the properties.
<plugin>
   <groupId>com.googlecode.flyway</groupId>
   <artifactId>flyway-maven-plugin</artifactId>
   . . .
   <configuration>
      <driver>${database.driver}</driver>
      <url>${database.url}</url>
      <serverId>${database.serverId}</serverId>
      <schemas>${database.schemas}</schemas>
   </configuration>
</plugin>

One more note about Maven profiles. You can, if you wish, specify a default active profile so that when executing Maven for the project you do not need to specify the profile to use on the command line. If you were creating a Maven project (POM) exclusively for Flyway migrations, you might want to set the flyway-dev profile as the default using the <activeByDefault> tag so you wouldn’t need to specify it when running migrations against your dev database.

Alternatively, if you’ve modified an existing Maven project as we have you may wish to just add the Flyway properties to existing profiles you use for the differing environments. In either case, activation of the profile, either by default or explicitly via the command line -P argument, will cause Maven to use the values for the properties defined in the respective profile. In our example, we specify the profile via the -P command line argument but the results are the same.

[:flyway-test ]$ mvn flyway:history -P flyway-dev
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO]
[INFO] MyApp :: Parent
[INFO] MyApp :: XSD :: XMLBeans Generated
[INFO] MyApp :: Domain Layer
[INFO] MyApp :: Web Application
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building MyApp :: Parent 3.15.BUILD-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:1.6.2-SNAPSHOT:history (default-cli) @ myapp-parent ---
[INFO] <skip>true</skip> detected.
[WARNING] Skipping execution.
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building MyApp :: XSD :: XMLBeans Generated 3.15.BUILD-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:1.6.2-SNAPSHOT:history (default-cli) @ myapp-xmlbeans-generated ---
[INFO] <skip>true</skip> detected.
[WARNING] Skipping execution.
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building MyApp :: Domain Layer 3.15.BUILD-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:1.6.2-SNAPSHOT:history (default-cli) @ myapp-domain ---
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | Version | Description | Installed on | State |
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | 0 | << Flyway Init >> | 2012-06-27 08:39:40 | SUCCESS |
[INFO] | 1.0 | load-flyway test-schem | 2012-06-27 08:40:02 | SUCCESS |
[INFO] | 1.1 | add-flyway test-table | 2012-06-27 09:00:36 | SUCCESS |
[INFO] +-------------+------------------------+---------------------+---------+
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building MyApp :: Web Application 3.15.BUILD-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:1.6.2-SNAPSHOT:history (default-cli) @ myapp-web ---
[INFO] <skip>true</skip> detected.
[WARNING] Skipping execution.
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] MyApp :: Parent ..................................... SUCCESS [1.002s]
[INFO] MyApp :: XSD :: XMLBeans Generated .................. SUCCESS [0.057s]
[INFO] MyApp :: Domain Layer ............................... SUCCESS [0.894s]
[INFO] MyApp :: Web Application ............................ SUCCESS [0.188s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.211s
[INFO] Finished at: Wed Jun 27 14:21:50 EDT 2012
[INFO] Final Memory: 10M/554M
[INFO] ------------------------------------------------------------------------
[:flyway-test ]$


Flyway Placeholders…


You may have noticed that there is one property we defined in the profiles that we did not use in the configuration of the Flyway Maven plugin- <database.name>. Although we have not used it as yet in our example, we have used it in the real project.

Flyway has a feature much like the Maven profile <properties> feature in that you can specify a set of property-like value pairs in the Flyway component’s configuration which it then uses for substitution in SQL migration scripts prior to actually executing the script. These properties are called “placeholders” and are specified in a <placeholders> section of the configuration like the following:
<configuration>
   . . .
   <placeholders>
      <database>${database.name}</database>
   </placeholders>
   <placeholderPrefix>$[</placeholderPrefix>
   <placeholderSuffix>]</placeholderSuffix>
</configuration>

You may also specify the delimiters which you will use when including the placeholders in your SQL scripts. In this example, we’ve added the <database> placeholder definition to our configuration and specify the $[ ] notation for placeholders in our migration scripts. When the database name is required in our migration scripts, then, we use the $[database] notation. This allows us to use the same migration script for all levels of the database.

Versioning for horizontal migrations…


One last topic which is also not addressed by the “vanilla” Flyway implementation is how to automatically maintain the “version” of SQL migrations across multiple projects that use the same database. Basically, the Flyway documentation suggests a numbering scheme for versioning which must be specified as part of the SQL migration script file name in a specific position within the name.

In fact, it’s a requirement since Flyway uses the ascending values of the version to determine which migrations should be applied to a database. When applying migrations, Flyway looks at the highest value of this version that has already been applied and only applies migrations of a higher value (see “How Does Flyway Work”).

Although this concept may be satisfactory for a siloed project with a team of developers all using the same code base, i.e., src/main/resources/database/migrations, it presents a problem for multiple teams of developers using disparate code bases, each with their own src/main/resources/database/migrations directories, but programming to the same database. You may think this is a rarity and presents additional source code management issues beyond that which we describe here, but in my consulting experience I have seen this scenario more than once and, in fact, is the case with the client whose project I used as an example.

Although the Release.Version.Modification (1.0.1) numeric algorithm works fine for most projects, we needed a more universal numbering scheme if we were to be able to automatically manage version numbers across multiple projects. The naming requirement for the version portion of Flyway SQL migration file name requires all numeric digits separated by periods (.) to denote the levels of migrations.

We chose a date-time stamp to the hundredth of a second as our version numbering template. Shell script and batch files were developed to generate a SQL migration file to the project’s database migrations source directory. The file name follows the Flyway convention of V-version__description.sql. This script is then executed by all developers of all of the projects and virtually guarantees the scripts will have unique names and be in the correct sequence.

Using this concept for our example project, we renamed the two SQL migration script files from V-1.0__load-flyway_test-schema.sql and V-1.1__add-flyway_test-table.sql to V-20120627.13.03.55_00__load-flyway_test-schema.sql and V-20120627.13.05.12.00__add-flyway_test-table.sql, respectively.

After cleaning the database and running the renamed scripts, the history of database migrations looks like this:
[INFO] ------------------------------------------------------------------------
[INFO] Building MyApp :: Domain Layer 3.15.BUILD-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:1.6.2-SNAPSHOT:history (default-cli) @ myapp-domain ---
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | Version | Description | Installed on | State |
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | 0 | << Flyway Init >> | 2012-06-27 15:40:44 | SUCCESS |
[INFO] | 20120627.13.03.55.00| load-flyway test-schem | 2012-06-27 15:41:08 | SUCCESS |
[INFO] | 20120627.13.05.12.00| add-flyway test-table | 2012-06-27 15:41:08 | SUCCESS |
[INFO] +-------------+------------------------+---------------------+---------+


Handling horizontal migrations from multiple projects…


In reality, the sample project we are using in this example is one of several which use the same database, each with their own code base and, as a result, individual database migration source directories. Up to this point in our example we have been running the Flyway plugin in only the build for our domain module. We now need to add all of the projects that use the database to our Flyway migration project.

But, because of Flyway’s method of applying migrations, if we run the Flyway migrate goal in each of the modules independently, we cannot guarantee they will all be applied and in the correct sequence. Since Flyway uses the version of the last migration applied as a starting point in applying new migrations, we need to insure that ALL migrations from ALL projects are considered in the process. This means we need to maintain all migrations in the same directory that Flyway will reference in its migration algorithm. We can use a Maven plugin for this.

We will need to change our migration process. Rather than run Flyway in each of the modules that have database migrations, we will change the process to copy the database migrations from each individual module’s target directory to a parent target directory using the maven-resources-plugin. We will then add a final module to the build that will perform the Flyway tasks.

The maven-resources-plugin facilitates copying of resources from non-standard locations to a common target destination. By binding the maven-resources-plugin's copy-resources goal to the process-resources phase, it will execute after the default process-resources goal which copies and filters resources to the build output directory. We will configure the plugin to copy the database migration files from that build output directory to a top-level target/database/ migrations directory.

To do this, we added the plugin’s definition to the <pluginManagement> section of our parent POM so it is executed in each module. When additional modules are added to the parent project, their database migrations will be copied automatically since the plugin is inherited.

The plugin configuration specifies the top-level target/database/migrations directory as the output destination. The input directory, i.e., the source of the copy, is specified as a <resource>. Since this plugin is executed after the default process-resources goal, the resources to be copied, i.e., the database migrations, are in the project’s build output directory. Filtering is not required since the files would have been filtered in the default goal if requested.

The definition of the maven-resources-plugin looks like this:
<plugin>
   <artifactId>maven-resources-plugin</artifactId>
   <version>2.5</version>
   <executions>                                                                                   
      <execution>                                                                                
         <id>copy-resources</id>                                                                
         <phase>process-resources</phase>                                                       
         <goals>                                                                                
            <goal>copy-resources</goal>                                                        
         </goals>                                                                               
         <configuration>                                                                        
            <outputDirectory>
               ../target/database/migrations
            </outputDirectory>                   
            <encoding>UTF-8</encoding>                                                         
            <resources>                                                                        
               <resource>                                                                     
                  <directory>
                     ${project.build.outputDirectory}/database/migrations
                  </directory>
                  <filtering>false</filtering>                                               
               </resource>                                                                    
            </resources>                                                                       
         </configuration>                                                                       
      </execution>                                                                               
   </executions>                                                                                  
</plugin>

We test the plugin by running mvn compile flyway:history –P flyway-dev.
[INFO]
[INFO] --- maven-resources-plugin:2.5:resources (default-resources) @ myapp-domain ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 136 resources
[INFO] Copying 3 resources
[INFO] Copying 7 resources
[INFO]
[INFO] --- maven-resources-plugin:2.5:copy-resources (copy-resources) @ myapp-domain ---
[debug] execute contextualize
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ myapp-domain ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- flyway-maven-plugin:1.6.2-SNAPSHOT:history (default-cli) @ myapp-domain ---
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | Version | Description | Installed on | State |
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | 0 | << Flyway Init >> | 2012-06-27 15:40:44 | SUCCESS |
[INFO] | 20120627.13.03.55.00| load-flyway test-schem | 2012-06-27 15:41:08 | SUCCESS |
[INFO] | 20120627.13.05.12.00| add-flyway test-table | 2012-06-27 15:41:08 | SUCCESS |
[INFO] +-------------+------------------------+---------------------+---------+

We can see from the output that the copy-resources goal was run after the default-resources goal and copied two files. Those two files are the database migration files which were copied to the parent target/database/migrations directory.

Completing the modifications…


Next we added an additional child module to the parent POM. Like the domain child module, the services child module contains a database migration to add a new table to the database. Then we added a new module to the parent POM which we called database-migrations. The sole purpose of this module is to run the Flyway plugin.

We created the database-migrations directory under our parent project directory and created a pom.xml file in the new database-migrations directory including the Flyway plugin. This is the only file in that directory since the purpose of this module is to only run the Flyway plugin.
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
                             http://maven.apache.org/xsd/maven-4.0.0.xsd">

    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>com.myapp</groupId>
        <artifactId>myapp-parent</artifactId>
        <version>3.15.BUILD-SNAPSHOT</version>
    </parent>
    <artifactId>myapp-database</artifactId>
    <packaging>pom</packaging>
    <name>MyApp :: Database Migrations</name>

    <build>
        <outputDirectory>../target</outputDirectory>
        <plugins>
            <!-- Flyway database versioning / migration plugin -->
            <plugin>
                <groupId>com.googlecode.flyway</groupId>
                <artifactId>flyway-maven-plugin</artifactId>
                <version>1.6.2-SNAPSHOT</version>
                <configuration>
                    <skip>false</skip>
                    <baseDir>database/migrations</baseDir>
                </configuration>
            </plugin>
        </plugins>
    </build>
</project>

Since Flyway looks in the runtime classpath for its migrations files, we need to override the default build output directory for this module. The build output directory is always included in the runtime classpath. The default build output directory is target/classes. We override it by specifying ../target which points to the target directory one level above this project’s build directory, or to the target directory of the parent directory. Flyway will look in the classpath, i.e., the parent target directory, for a directory named database/migrations which we specified as the value of the <baseDir> configuration parameter.

We then added the definitions of the additional child modules to the parent POM. If you do this, make sure the database-migrations child module is specified last as in the following example:
<modules>
    <module>xsd</module>
    <module>domain</module>
    <module>services</module>
    <module>web</module>
    <module>database-migrations</module>
</modules>

Our last modification is to remove the Flyway plugin definition from the child domain module POM file so the Flyway plugin is not executed. To test our modifications we will clean the database using the flyway:clean goal, initialize it with the flyway:init goal, and then run flyway:migrate to insure Flyway uses the top-level parent target directory for its migration process.

After running the migration, we can see from the output of flyway:history that all three migrations were applied to our database:

[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building MyApp :: Database Migrations 3.15.BUILD-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- flyway-maven-plugin:1.6.2-SNAPSHOT:history (default-cli) @ myapp-database ---
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | Version | Description | Installed on | State |
[INFO] +-------------+------------------------+---------------------+---------+
[INFO] | 0 | << Flyway Init >> | 2012-06-28 13:27:05 | SUCCESS |
[INFO] | 20120627.13.03.55.00| load-flyway test-schem | 2012-06-28 13:27:32 | SUCCESS |
[INFO] | 20120627.13.05.12.00| add-flyway test-table | 2012-06-28 13:27:33 | SUCCESS |
[INFO] | 20120628.13.15.52.00| add-another-flyway test-table | 2012-06-28 13:27:33 | SUCCESS |
[INFO] +-------------+------------------------+---------------------+---------+


Mission Accomplished…


Using Maven profiles we are now able to apply the database migrations to each succeeding level of the database. And via a standard naming convention using a generated file name containing a date-timestamp as a version number we are able to guarantee database migration file names are unique throughout several projects using the same database. Using the maven-resources-plugin to copy all database migrations files to a common location, we can also guarantee that all database migrations will be applied in sequence by restricting execution of the Flyway plugin goals to only one child module, the last one specified.

The only desire left is to be able to apply any migration to a database if it has not already been applied. Currently, Flyway looks at the highest value of the version that has already been applied to a database which it has recorded in the schema_version table and only applies migrations in the <baseDir> directory whose file name’s version is of a higher value (see “How Does Flyway Work”). Hopefully, a future release of Flyway will support this behavior, as it will then solve the last of our database migrations issues, namely developers committing database migrations out of sequence in multiple projects.

Monday, July 2, 2012

Using Maven to Integrate Flyway in an Existing Database-
Part 1


I was recently asked by a client to investigate database versioning / migration tools. Having had previous experience with Rails db Rake tasks and Liquibase, both of which require a DSL, I looked for a more Java-friendly tool and found Flyway.

One of the pre-requisites was to be able to integrate the tool with an existing database and SQL scripts to reduce the learning curve for application development personnel. We were also looking for a tool we could run with either Ant or Maven as the client has multiple projects using both technologies. Flyway offered that with its support of native SQL migrations including support for stored procedures and triggers as well as a Maven plugin and an Ant task. The Flyway command line tool (CLT) and Java migrations are an added bonus.
Disclaimer: This is not a testimonial for Flyway versus any other tool, just an accounting of what it took to get it working for an existing database. Although the Flyway wiki and documentation is adequate, I had a difficult time getting all the wheels turning in the same direction. In fact, I found myself doing things that were not really explained nor even touched on in the Flyway wiki pages. I thought, “What better way to communicate that than through a blog?” BUT, YMMV- use this article at your own risk.
Java migrations are Java classes that invoke the Flyway Java core classes to perform database versioning and migration. These can be useful for “on-the-fly” migrations, i.e., at application startup, to automagically bring the target database up to speed with respect to its infrastructure, etc., as opposed to requiring a DBA or whomever to execute a series of steps prior to starting the application in a production environment. There is a sample available on the Flyway Downloads page that illustrates this feature.