Friday, December 31, 2010
Build and Release Management With Apache Maven
The Joel Test stands up pretty well even today. Joel's first two questions, "Do you use source control?" and "Can you make a build in one step?" are a good start at addressing the need for build automation. With the tools available today, however, I suggest that build management (not just automation) and release management should be added to the "must have" list for productive software development teams.
Since I mostly work in Java, I'll talk about Java build and release management. There are a few comprehensive tools for Java build and release management (actually many more build management tools than release management tools), but the most popular by far is Apache Maven.
Maven started in 2001 as a part of the Apache Alexandria project, moved to be a part of the Apache Turbine project in 2002, became a top-level Apache project in 2003, and had its first milestone release, v1.0, in October of 2005. Version 2.2.1 has been in wide use since its release in August of 2009. Version 3.0 was released in October of 2010, and the current version, 3.0.1 was released in November 2010.
Maven bills itself as a "software project management and comprehension tool". Its most basic use is as a build tool, commonly as a replacement for Apache Ant, but it does several other things, which taken together, really do form a useful software project management and comprehension tool. Maven is certainly not an absolute requirement for a good software development team, but the issues that it addresses must be addressed somehow. If your Java development team is not currently addressing these issues in a satisfactory way now, Maven is a good way to get started addressing them in a coherent way.
As a build tool, Maven takes the now-common "convention over configuration" approach that seems to have started around 2004 with the Ruby on Rails web framework. As such, Maven was designed to be a highly opinionated tool, giving structure to the free-for-all that Ant usage had become in many projects. Maven is based on the Project Object Model, an XML configuration file that specifies what goes into a project, but unlike Ant, generally not how to build it. This is where the convention over configuration comes in - you put your source code and other build artifacts in a directory structure that Maven expects, and then Maven can build it with no further configuration. This results in both simplified build management (no build script to write and maintain), and a project that anyone familiar with Maven can build. There is a well-defined build lifecycle, with well-defined targets: "mvn install" builds, tests, and packages any project built with Maven, and installs the resulting artifact in a local repository (more about repositories later). This is a great improvement over Ant, where the names of the build targets, and what they do are up to the whim of the person writing the Ant build script.
Maven is much more than just a build tool. It has many more features, but to me, the most important are Dependency Management and Release Management.
Dependency Management is all about dealing with the libraries (jar files) upon which your project depends (direct dependencies), and those jar files' dependencies (transitive dependencies). With Ant or any other build system that does not manage dependencies, this is left as a manual exercise for the developers. Typically, all of the dependencies are resolved manually, and all of the jar files are put into the project's 'lib' directory. This frequently means that multiple projects using the same third-party libraries will each need to keep a copy of them.
Maven's paradigm is to have the developer declare the project's direct dependencies in the POM. Maven then resolves the transitive dependencies, and pulls the required jar files (direct and transitive dependencies) from a Maven Repository at build time. Jar files are not kept within the project, just the specification of which jar files to use at build time. There is a "standard" external repository (the "central" repository) that is web-based, and hosted by maven.org, and a local repository in the filesystem on the developer's machine. Jar files are pulled from the central repository, or other external repositories that may be configured, and cached in the local repository.
This arrangement works well for small, one-off development efforts, but many shops have reusable jar files of their own that are used by multiple internal projects. Historically, many development teams have distributed these reusable in-house developed jar files by either putting them on a shared network directory, or by checking them into the source code version control system. This results in a process that is more manual and cumbersome than it should be, and generally violates the DRY principle.
Using Maven instead of Ant or a home-grown build system, and managing dependencies through it are a good first step, but for reusable code, Release Management becomes important, and again, Maven can help. Maven, through its release plugin, supports a well-defined release lifecycle, and allows teams to store their reusable code in a centralized Maven repository, just like the "standard" jar files that are kept in the central repository. Teams (or whole companies) can set up their own internal, web-based repository that either enhances or replaces the central repository. Using Maven, they can then "release" their jar files to this repository. Projects using Maven can then declare these jar files as dependencies the same way that they do for jar files in the central repository, and they will be resolved by Maven. No need for shared directories, or placing binaries in the source code version control system. The release plugin takes care of incrementing version numbers and updating them in the version control system, in a standard way.
The features that have been presented here are just the most prominent ones - Maven is capable of much more, especially with its extensive list of plugins. If your team is not using Maven (or something like it), or is only using Maven as a build tool, I encourage you to spend some time learning more about Maven. It will be time well spent. There are the Maven project's own "Getting Started" guides and a series of books by Sonatype (the commercial company behind Maven) for starters.
Sonatype provides online, instructor-led training courses, and here at Chariot Solutions, we offer public, on-site training in our Ft. Washington, PA facility, as well as custom private courses.
Tuesday, December 28, 2010
Painless Java Desktop Application Development with Griffon, MigLayout, and IntelliJ
We have become accustomed to depending on MVC web frameworks, and almost take for granted that at some level, we will be handling HTTPServletRequests, providing HTTPServletResponses, and deploying our applications to either a servlet container or a full J2EE server.
Every once in a while, however, the need arises to write an old-fashioned, desktop, thick-ui, non-web-centric application. If you have ever written a Java Swing application, you may have painful memories of dealing with things like BorderLayouts, GridBags, Event Listeners and Event Dispatching Threads, JFrames, and the like.
As a web developer, when you think about building your desktop application, you might start to think about how you will structure your application, and how you will build and package it. Will you have to re-package your code and all of your dependencies in one giant uber-jar?
Will you make it a web-start application? Fortunately, some folks have been thinking about, and working on these issues.
Griffon is a Groovy-based MVC framework for writing desktop applications. It is written by the Groovy Swing team, and is heavily influenced by Grails. If you have worked with Grails, then Griffon will be a very familiar environment. Griffon answers all of the above questions, and more.
Using the convention-over-configuration and command-line-driven scaffolding paradigms made popular by Ruby on Rails, Django, and Groovy on Grails, among others, Griffon generates the project structure, the model, view, and controller objects, and all of the configuration for you. You then customize these to produce your application.
Working in Groovy, of course, makes this a much more pleasurable experience, dispensing with the ceremony of Java development, and letting you write high-level, fairly terse, yet expressive code, and enabling you to accomplish quite a bit very quickly. The prime example of this in the Griffon framework is the SwingBuilder object - a Groovy Builder that provides a DSL for building Swing user interfaces. Views in Griffon are SwingBuilder scripts, and greatly simplify the task of creating Swing UIs.
Griffon uses the Gant build system (as does Grails), and by default, packages your application with all its dependencies as an uber-jar (with the correct manifest entries to make it 'executable'), as an applet, and as a java-webstart application.
When working with Swing, even with the SwingBuilder, one potentially still has to deal with the frustration of laying out the user interface by using a LayoutManager. GridBagLayout is flexible (too flexible?), but about as complicated as it can get. BorderLayout is fairly simple, but the simplicity comes at a cost - you typically wind up nesting lots of BorderLayouts in other BorderLayouts to get the desired behavior. Over the years, alternate layout managers have appeared, and some are quite useful, most notably the JGoodies Form Layout and RiverLayout. I tried MigLayout with my Griffon project, and found it to be very flexible and easy to use.
Here is a screenshot of the main UI of my application:
As mentioned previously, the View is a Groovy SwingBuilder script. The first part of the script sets up an 'application' - the application's "MainFrame", which Griffon interprets according to the runtime environment - for a stand-alone application, it will build a JFrame, and for an applet it will build a JApplet:
package servicetester
import java.awt.FlowLayout
import net.miginfocom.swing.MigLayout
application(title: 'Service Tester', minimumSize: [297, 200], maximumSize: [297,200], location: [50, 50], pack: true, locationByPlatform: true) {
Note that minimumSize, maximumSize, location, etc. are set as properties here. In straight Java Swing code, these would have been at least one line of code each, and several would have involved building Dimension objects.Next is setup of the actions and the menu bar that uses the actions, in this case, a simple 'File' menu with a 'Quit' menu item to exit the application.
Note the use of Groovy closures for associating the Action object with implementation of the action.
actions {
action(id: 'exitAction',
name: 'Quit',
closure: { System.exit(1) },
mnemonic: 'Q',
accelerator: shortcut('Q'),
shortDescription: 'Quit'
)
}
menuBar(id: 'menuBar') {
menu(text: 'File', mnemonic: 'F') {
menuItem(exitAction)
}
}
I chose to use the MigLayout for the main panel of the view. There are lots of ways to initialize a MigLayout, I used the "fill" parameter to the constructor, which tells it to grow to occupy all available space, which makes sense here because it is the top-level component in the frame.
panel(border: emptyBorder(12), layout: new MigLayout('fill')) {
As you can see in the screenshot, there are three groupings of controls - the group of fields above the separator (the input fields), the group of fields below the separator (the output fields), and the "button bar" at the bottom (only one button). To achieve these groupings, I added two panels, each with MigLayout initialized with "fill", as with the main frame, and a third panel with a FlowLayout for the button bar.
These MigLayouts are given constraints of "growy" and "wrap":
panel(layout: new MigLayout('fill'),
constraints: "growy, wrap") {
"growy" means "grow vertically", and "wrap" means that the next component should be placed below, instead of to the right.With this configuration, the MigLayout in the main frame gives the button bar it's preferred size, and resizes the other two panels as the main window is resized, giving equal weight to them.
The two MigLayout panels are very similar. Each has sets of label / input field pairs, which look like:
label(text: 'AMQ Broker URL:')
textField(
id: 'brokerUrl',
columns: 40,
text: bind(target: model, targetProperty: 'brokerUrl', value: model.brokerUrl),
enabled: bind {model.enabled},
constraints: 'wrap'
)
The "text:bind" property binds the value entered in the textField to a property in the model - the value entered by the user is placed into the model.One big "gotcha" here is that bi-directional binding is not supported. It is possible to bind in each direction, but not both directions at the same time. As one might expect, lots of people are asking for this capability, so I wouldn't be surprised to see it show up soon. I ran into this when trying to set the initial value of the field from the value in the model. The answer for this case is to use the "value" property as shown above - it gives the field it's initial value, and does not participate in the binding mechanism, so it doesn't present the same problem as bi-directional binding. If you do truly need bi-directional binding, at the moment, you'll have to roll your own.
Here is the full source code for this View.
Due to its highly-developed sense of structure and general lack of need for configuration, working on a Griffon application using the command line and a text editor is very straightforward. However, IDE support is always nice. Eclipse does not directly support Griffon, but does support Groovy. Both NetBeans and IntelliJ have direct support for Griffon projects. IntelliJ 9 and 10 have direct support for creating, importing, testing, running, and debugging Griffon applications, and I found these features to work quite well.
I found the combination of Groovy, Griffon, MigLayout, and IntelliJ to be an extremely productive environment for creating Swing applications, providing much of the application infrastructure, allowing me to dispense with ceremony and concentrate on writing the parts of the application that make it unique. Before embarking on a large or mission-critical Griffon project, however, you do need to be aware that Griffon is still in Beta (0.9.2-beta-3 was just released on Dec. 21st, 2010), and so is still in a bit of a state of flux.
If you are interested in reading about Griffon, there's documenatation at at the Griffon home page, and the Griffon In Action book is available from Manning.
Wednesday, December 1, 2010
Wednesday, November 24, 2010
Spring into Mobile Application Development
Spring into Mobile Application Development
Posted on November 19th, 2010 by Keith Donald in Spring.
At SpringOne2gx we announced exciting new initiatives in the areas of social media and mobile application development. A few weeks ago, Craig Walls released Spring Social. Today, Roy Clarkson released Spring Mobile and Spring Android. In this post, I'd like to highlight these projects and share how Spring aims to simplify mobile application development.
Choices in Mobile Application Development
If you attended SpringOne2gx this year, you've seen Greenhouse, an app we built for our community that also serves as a reference and driver for Spring technology. Craig showed you some of the social elements of Greenhouse, such as the ability to connect your account with Twitter and Facebook. There are also a number of mobile elements. Specifically, Greenhouse doubles as a mobile web app, and sports native Greenhouse for iPhone and Android clients.
Like many organizations today, we had to answer the basic question of "what mobile platforms to target?" In the end, we chose to invest in a native experience for iPhone and Android users, while also developing a cross-platform mobile web app. Our decision to go native was driven by the fact the app is consumer-oriented, and a large number of prospective consumers (application developers) own iPhone and Android devices. At the same time, the mobile web app aims to provide a good baseline experience that works cross-platform, something that's possible today with the rise of WebKit and HTML 5.
From our development work, grew Spring Framework contributions: first, a Spring Mobile project that provides extensions to Spring MVC for developing mobile web apps; and second, a Spring Android project that supports the development of native Android clients that communicate with Spring-based back-ends. I'll take you through each of these projects in turn.
Spring Mobile
The first problem we tackled was designing a web app that was pleasant for mobile visitors to use. While a smartphone may have a capable web browser, it still has a small screen and that needs to be accounted for. There are essentially two approaches to dealing with this problem:
- Detect the device that originated the web request and serve a separate site to mobile devices.
- Serve a single site, but progressively enhance it for desktop users by using CSS 3 media queries and JavaScript.
You can find examples of both techniques in the wild today; for example, the conference site Lanyrd enhances through client-side detection, while SpeakerRate uses server-side detection that redirects mobile visitors to a separate site (see for yourself by installing the Firefox User Agent Switcher and setting your user agent to iPhone).
In Greenhouse, we started with server-side detection. Specifically, we aimed to apply a different page layout if the device was a mobile device. Out of this grew a general purpose "Device Resolver Abstraction", which is the defining feature of Spring Mobile 1.0.0.M1. Some highlights of this feature include:
- A HandlerInterceptor that uses a DeviceResolver to detect the Device that originated the current HttpServletRequest.
- The ability to inject the detected Device into @Controller methods and view templates to vary logic by device type.
Below are some usage examples from the Greenhouse codebase. First, have a look at the interceptor definition from /WEB-INF/spring/appServlet/servlet-context.xml:
1<interceptors>
2<!-- On pre-handle, detect the device that originated the web request -->
3<beans:beanclass="org.springframework.mobile.device.mvc.DeviceResolvingHandlerInterceptor"/>
4</interceptors>Now, a JSP template that conditionally renders some content if the device is not a mobile device:
1Please try again<c:iftest="${!currentDevice.mobile}"> or <ahref="<c:url value="/signup" />">sign up</a></c:if>.Finally, a Tiles-based page layout that changes if the device is a mobile device:
1<definitionname="page"templateExpression="/WEB-INF/layouts/${currentDevice.mobile ? 'mobile/' : 'standard/'}page.jsp"/>In addition to these features, Spring Mobile 1.0.0.M1 ships:
- A HandlerInterceptor that redirects mobile visitors to another URL. This is useful if your mobile site truly is a separate application. Consider Flickr, for example, which will redirect you to m.flickr.com if you access www.flickr.com from your phone.
- A DeviceResolver implementation that delegates to WURFL for device detection. WURFL provides a large database of devices and their capabilities. It is useful when you need to know more about the Device that originated the request, such as its specific screen size, manufacturer, model, preferred markup, or other capabilities.
Recently, we have also begun exploring the use of CSS 3 media queries with JavaScript to perform client-side detection. This approach has the advantage of not requiring special server-side handling unless you aim to vary the semantic content you send to the device, and not just optimize the style. Not all browsers support media queries, so they may not be an option for you, but if you're targeting smartphones with WebKit-based browsers you should be fine. In general, the approach of first designing your app for mobile, then progressively enhancing it for desktop feels quite elegant to me.
Spring Android
For the Android client, a different set of challenges emerged. We needed to exchange data with the server over HTTPS via REST, and since that data is user-specific we require the user to sign in. Rather than use Basic Auth, where we would need to store username and password credentials on the device itself, we opted for OAuth.
OAuth is an emerging standard that provides a token-based authorization scheme. Essentially, a username and password is traded for an access token, and the access token is used to make requests for protected resources. This means you only need to store the access token with the device for "remember me" functionality. Furthermore, we chose to implement the sign in process whereby the client takes you to the server's web site for connection authorization. In this way, the client never sees your username and passsword, which is important if you allow third-party clients to be developed against your API (which we also want to encourage). Finally, in the event a user's phone is stolen or otherwise compromised, an access token can be invalidated without the risk of username and password compromise.
Out of this work, came the desire to get specific modules of the Spring Framework working in an Android environment. Specifically, we aimed to use RestTemplate for the REST API calls, and Spring Security for the OAuth client.
I'm pleased announce that the first milestone of Spring Android ships an "Android-ready" RestTemplate. We're using it in Greenhouse, and we encourage you to use it as the REST client in your own Android applications. Check out the example usage below:
1RestTemplate restTemplate =newRestTemplate(newCommonsClientHttpRequestFactory());
2Event event = restTemplate.getForObject("https://myapp.com/event/{name}", Event.class,"springone2gx");In future milestones, you can expect more of the Spring Framework to be supported in an Android environment, such as the Spring Security OAuth client.
Getting Started
The best way to get started is to see it live by walking through the Greenhouse app in your own local development environment. As a reference, Greenhouse uses the Spring Mobile and Spring Android projects, as well as Spring MVC, Security, Social, and Integration. The project page provides a guide that shows you how to get the web app, iPhone client, and Android client all running in your local environment in minutes.
If you have questions about features, roadmap, or just want to have a discussion with the development team, please do visit our community forums. We value your feedback.
Summary
I'm very excited about all the new initiatives we've got going, and especially what we're doing in the social and mobile space. The first milestones of these projects are just the beginning. I encourage you to get involved in the projects that are useful to you and help us make them the best they can possibly be.
via blog.springsource.com
Need to check this out for some old Spring stuff I have worked on :)
Tuesday, November 16, 2010
NFC. What does it all mean?
NFC: Near Field Communications: a short-range high frequency wireless communication technology which enables the exchange of data between devices over about a 10 centimeter (around 4 inches) distance (via Wikipedia).
Well that's great, but what does it mean, especially when mentioned in conjunction with Gingerbread?
As you may (or may not) have heard this week at the Web2.0 Summit Eric Schmidt talk about Android 2.3 and its support for NFC. This could indeed lead the way for mobile payments on devices. NFC will allow for the transmission of credit card data over a defined radio frequency (13.56 MHz).
This isn't new, Japanese handsets (Keitai) have had this type of technology for a while (I want to say up to 10 years, but I can't back that up right now). Go google Osaifu-Keitai/keitai credit.
However this *is* all exciting news. There is potential for integrating it with conferences, museums, travel, healthcare, payment systems, and countless other things. Lets not forget though, some of these things already exist already:
- Exxon SpeedPass
- Debit cards with Visa PayWave.
Now we have another new way to gain information about our surroundings. #mobilefirst anyone?
Some interesting articles to peruse at your leisure.
RFID in Museums - Another Growing Market
Who Will Win the Mobile Payments Battle: Gadgets, NFC or Apps?
Intuit Squares Off In Mobile Credit, With A Little Help From Apple
Future Credit Cards: Electronic and RFID?
Web 2.0 Summit 2010: Eric Schmidt, "A Conversation with Eric Schmidt"
Thursday, November 11, 2010
Configuring LogBack in Roo
LogBack is the more versatile logging strategy created by the same person that created log4j. Use the following steps to configure your Roo 1.1 project to use LogBack.
See Also: Reasons to Switch to LogBack
1. Create a logback.xml file in src/main/resources
Cont. here for the remainder of the post
Wednesday, November 10, 2010
Post Presentation Thoughts from our Mobile Seminar
Yesterday I presented at the first in a series of events for Chariot Solutions, the topic was 'Mobile Design Concepts'.
My goal was to try present things that I had learned/seen/been shown/done over the last couple of years, and try to get the audience out of the 'desktop browser' mind set when developing for mobile.
I drew from a lot of people that I have seen/meet including Luke Wroblewski (@LukeW), Josh Clark/Tapworthy (@globalmoxie), Mike Lee (@bmf) William Van Hecke (@fetjuel) and Brian Fling's book 'Mobile Design and Development' which I still think is the first book anyone planning on a mobile project should read. I was probably influenced by many more folks than that, but they are the obvious ones to me at the time of writing.
Cont. reading..
Monday, November 8, 2010
Grails Validation Constraints Quick Ref
After several iterations of Grails, we have to go to 17 (seventeen) pages to see all the validation constraints? I wanted to see them on one page.
Grails 1.3.5 – Validation Constraints
blank – To validate that a String value is not blankClick here for the rest of the article
Monday, October 4, 2010
Monthly Mobile Musings from September
Thursday, September 16, 2010
Setting up TeamCity CI w/ Cucumber, Flex, FunFX, and Firefox in a Headless Environment: Part Two
Recently I had the “opportunity” (read: challenge) to setup a continuous integration build for a project with a Flex frontend and a Ruby/Rails backend. The project had been converted from Java-Hibernate-Tomcat to Ruby on Rails. Cucumber was introduced for GUI testing using the FunFX adapter. It was my first experience with Cucumber, so the combination of Cucumber, FunFX, Flex, and Firefox was a little daunting at first. But we got through it and actually, it turned out to be quite an effective tool in our testing suite.
But the fun really started when I was tasked with building the TeamCity CI server for our project. And after it was all over, I felt I needed to document the trip so others might benefit from it, if not just for posterity. This is the second part of the two-part series on that journey.
Continuing…
In the first part we installed the open source software we are using: Java, Ruby, Rails, and all the required Ruby Gems, as well as the Flex 4.1.0 SDK, Firefox, FlashPlayer, and the Git client. The FunFX adapter for Cucumber uses JSSH to communicate with Firefox, so we configured and built Firefox with JSSH to run in a headless environment. We installed the latest Flex 4.1.0 SDK and the FlashPlayer 10.1 plugin for Firefox.
So what we have left is the installation and configuration of TeamCity itself as well as the setup of the build process for our Cucumber, Flex, and FunFX sample project.
So let’s get started.
Install TeamCity
Download the latest version of the Professional TeamCity software from the TeamCity web site for the appropriate platform. The Professional version is free but is limited in number of projects, build configurations, and users. We will only define one project and are well within the limit for build configurations.
We will be executing the installation instructions detailed on the TeamCity web site to install TeamCity bundled with the Tomcat servlet container on a Linux platform. They are relatively simple and are documented here. If you wish to install TeamCity to an existing Tomcat servlet container or on a different platform, please refer to the instructions for your specific environment.
We will be using the default admin user that we’ll define during installation. We will also be using HSQLDB and the default build agent. Most of the defaults will suffice for our installation.
Copy the TeamCity tar.gz file to the TeamCity server. SSH into the TeamCity server as yourself. Copy the TeamCity tar.gz file from /home/teamcity to /usr/local, unpackage the tar.gz file, and create a symlink to the TeamCity install directory. You will also need to change the owner of the TeamCity install directory to the 'teamcity' user.
$ cd /usr/local
$ sudo cp /home/teamcity/TeamCity-5.1.3.tar.gz .
$ sudo tar -xzvf TeamCity-5.1.3.tar.gz
$ sudo ln -s TeamCity teamcity
$ sudo chown -hR teamcity TeamCity
Logout as yourself and SSH into the TeamCity server as the ‘teamcity’ user. We will run the TeamCity CI server under this ‘teamcity’ user.
Since there is no physical display device attached to the TeamCity server, we need to indicate to TeamCity it will be running in headless mode. This is done by setting TeamCity’s java.awt.headless Java options property to true. You can either create a setenv.sh file in the bin directory or add the statement to the catalina.sh file. If you are installing TeamCity to execute in an existing Tomcat servlet container, add the statement to the catalina.sh file in its Tomcat installation’s bin directory.
$ cd /usr/local/teamcity/bin
$ echo 'JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"' > setenv.sh
$ chmod +x setenv.sh
You should also set the TEAMCITY_SERVER_MEM_OPTS and TEAMCITY_AGENT_MEM_OPTS environment variables as described in the instructions for the TeamCity startup options. Although these values may vary according to your build requirements, we set our values as follows:
| TEAMCITY_SERVER_MEM_OPTS | -Xms512m -Xmx1024m -XX:MaxPermSize=1024m |
| TEAMCITY_AGENT_MEM_OPTS | -Xms512m -Xmx1024m -XX:MaxPermSize=1024m |
Start the server with the following command:
$ sh /usr/local/teamcity/bin/runAll.sh start
Then bring up the TeamCity application in a local browser (http://localhost
Create a CI Project
After creating the administrator account, you are presented with the Projects page. Select the Create Project link. Enter the appropriate values in the respective fields of the resulting screen:
| Name: | FC Converter |
| Description: | Fahrenheit to Celsius Conversion project |
Press the Create button.
Next we will associate the project with a version control system. Select the VCS Roots tab and then the +Create VCS root link.
Enter an appropriate VCS rootname and select the type of VCS from the dropdown.
The remainder of the screen is refreshed with the fields for the respective VCS. We are illustrating Git here. Enter / select as appropriate for the respective fields:
| VCS rootname: | fc_converter-root |
| Type of VCS: | Git (Jetbrains) |
| Fetch URL: | git://github.com/landerson/FC-Converter.git |
| Push URL: | (leave blank to use Fetch URL) |
| Branch name: | master |
| Clone repository to: | checkout |
| User Name Style: | Userid(jsmith) |
| Submodules: | Checkout |
| Authentication Method: | Anonymous |
| User name: | (leave blank) |
| Path to git: | /usr/bin/git(or as appropriate) |
| Clean Policy: | (as appropriate) |
| Checking Interval: | (as appropriate) |
Press the Test Connection button to test the SSH handshake with github. The test should succeed. The test must succeed in order for the TeamCity build configuration to work properly. After the test succeeds, press the Save button.
Select the General tab to return to the main project page. It is from this page that we will create the build configuration for this CI project.
We will create the following build configuration steps:
- checkout the project from the source repository and update the Ruby gems,
- compile the Flex source modules,
- restart the webserver, and
- run the Cucumber FunFX tests.
First we will create a build step to gather the sources and update any required Ruby Gems. Select the +Create build configuration link. Enter the following in the respective fields of the resulting General Settings screen:
| Name: | Step 1- Collect Sources |
| Description: | Checkout sources and update gems. |
| Fail build if: | check ‘an error message is logged by build runner’ |
Leave the remaining fields blank or checked as-is to accept the defaults.
Press the VCS Settings button and enter the following in the respective fields of the resulting Version Control Settings screen:
| +Attach existing VCS root: | fc_converter-root |
| VCS checkout mode: | Automatically on server |
| Checkout directory: | checkout |
| VCS labeling mode: | Do not label |
| Choose VCS roots to label: | (leave blank- do not check) |
The “Automatically on server“ selection will cause the project source to be checked out of the source repository into the TeamCity project working directory.
Press the Choose Build Runner button and enter the following in the respective fields of the resulting Build Runner screen:
| Build runner: | Rake |
| Rake tasks: | gems:install |
Uncheck any Attached reporters that may be checked and leave the remaining fields blank.
Press the Save button to create the build configuration. The screen is refreshed and should show the following message:
Build configuration "Step 1- Collect Sources" has been created successfully.
Select FC Converter Project in the Administration > FC Converter Project > Step 1 ... banner line just under the tabs at the top of the page to return to the main project page. If you named your project differently, your project ‘s name will appear in the banner line.
Next we will create a build step to compile the Flex source modules. Select the +Create build configuration link. Enter the following in the respective fields of the resulting General Settings screen:
| Name: | Step 2- Flex Compiles |
| Description: | Compile the Flex source modules. |
| Fail build if: | check ‘an error message is logged by build runner’ |
Leave the remaining fields blank or checked as-is to accept the defaults.
Although we are not checking out the sources in this step (or any subsequent step), we still need to specify the VCS settings to tell TeamCity where the sources are, e.g., in the checkout directory. Press the VCS Settings button and enter the following in the respective fields of the resulting Version Control Settings screen:
| +Attach existing VCS root: | fc_converter-root |
| VCS checkout mode: | Do not checkout files automatically |
| Checkout directory: | checkout |
| VCS labeling mode: | Do not label |
| Choose VCS roots to label: | (leave blank) |
Press the Choose Build Runner button and enter the following in the respective fields of the resulting Build Runner screen:
| Build runner: | Rake |
| Rake tasks: | flex:compile |
Uncheck any Attached reporters that may be checked and leave the remaining fields blank.
Press the Save button to create the build configuration. The screen is refreshed and should show the following message:
Build configuration "Step 2- Flex Compiles" has been created successfully.
Under the right hand column titled Configuration Steps, select 6 Properties and Environment Variables. On the resulting screen, select +Add new variable under Environment Variables. Enter the following in the respective fields of the resulting screen:
| Name: | FLEX_HOME |
| Value: | /usr/local/flex_4_sdk |
and press the Save button.
Again, select FC Converter Project in the Administration > FC Converter Project > Step 1 ... banner line just under the tabs at the top of the page to return to the main project page. If you named your project differently, your project ‘s name will appear in the banner line.
Next we will create a build step to restart the Rails web server. Select the +Create build configuration link. Enter the following in the respective fields of the resulting General Settings screen:
| Name: | Step 3- Restart the Web Server |
| Description: | Stop and start the Rails web server. |
| Fail build if: | check ‘an error message is logged by build runner’ |
Leave the remaining fields blank or checked as-is to accept the defaults.
Although we are not checking out the sources in this step we still need to specify the VCS settings to tell TeamCity the sources are in the checkout directory. Press the VCS Settings button and enter the following in the respective fields of the resulting Version Control Settings screen:
| +Attach existing VCS root: | fc_converter-root |
| VCS checkout mode: | Do not checkout files automatically |
| Checkout directory: | checkout |
| VCS labeling mode: | Do not label |
| Choose VCS roots to label: | (leave blank) |
Press the Choose Build Runner button and enter the following in the respective fields of the resulting Build Runner screen:
| Build runner: | Rake |
| Rake tasks: | webserver:stop webserver:start |
| Additional Rake command line parameters: | RAILS_ENV=cucumber |
Uncheck any Attached reporters that may be checked and leave the remaining fields blank.
Press the Save button to create the build configuration. The screen is refreshed and should show the following message:
Build configuration "Step 3- Restart the Web Server" has been created successfully.
Again, select FC Converter Project in the Administration > FC Converter Project > Step 1 ... banner line just under the tabs at the top of the page to return to the main project page. If you named your project differently, your project ‘s name will appear in the banner line.
Finally, we will create a build step to run the Cucumber FunFX tests. Select the +Create build configuration link. Enter the following in the respective fields of the resulting General Settings screen:
| Name: | Step 4- Run Cucumber Tests |
| Description: | Run the Cucumber FunFX Tests. |
| Fail build if: | check ‘an error message is logged by build runner’ |
Leave the remaining fields blank or checked as-is to accept the defaults.
Although we are not checking out the sources in this step we still need to specify the VCS settings to tell TeamCity the sources are in the checkout directory. Press the VCS Settings button and enter the following in the respective fields of the resulting Version Control Settings screen:
| +Attach existing VCS root: | fc_converter-root |
| VCS checkout mode: | Do not checkout files automatically |
| Checkout directory: | checkout |
| VCS labeling mode: | Do not label |
| Choose VCS roots to label: | (leave blank) |
Press the Choose Build Runner button and enter the following in the respective fields of the resulting Build Runner screen:
| Build runner: | Rake |
| Rake tasks: | ci:features |
Uncheck any Attached reporters that may be checked and leave the remaining fields blank.
Press the Save button to create the build configuration. The screen is refreshed and should show the following message:
Build configuration "Step 4- Run Cucumber Tests" has been created successfully.
Under the right hand column titled Configuration Steps, select 6 Properties and Environment Variables. On the resulting screen, select +Add new variable under Environment Variables. Enter the following in the respective fields of the resulting screen:
| Name: | DISPLAY |
| Value: | localhost:1.0 |
and press the Save button.
Again select +Add new variable under Environment Variables. Enter the following in the respective fields of the resulting screen:
| Name: | CUCUMBER_FORMAT |
| Value: | progress |
and press the Save button.
Select the Projects tab in the top left of the screen to display the build configuration steps for the FC Converter project.
Perform First Build
We’re now ready to execute our first build. We will use the first build configuration step we created to clone the project source for the first time.
To run the first step, press the Run button to the left on the Step 1- Collect Sources line. This will add the step to the build queue and it should begin executing immediately. You should see the 'Running:...' message underneath the step title line within seconds. When the step has completed, 'Running:...' should be replaced with 'Success'.
To view the output of the build step, hover the mouse pointer over the down arrow adjacent to the 'Success' label and select the 'Full log' choice in the resulting drop down menu. Take note of the checkout directory in the log - we will reference this later.
| [10:19:14]: Checkout directory: /usr/local/TeamCity/buildAgent/work/fc_converter [10:19:14]: Updating sources: server side checkout... [10:19:14]: [Updating sources: server side checkout...] Will perform clean checkout. Reason: Checkout directory is empty or doesn't exist [10:19:14]: [Updating sources: server side checkout...] Building clean patch for VCS root: fc_converter-root [10:19:14]: [Updating sources: server side checkout...] Repository sources transferred: 2.59Mb total [10:19:14]: [Updating sources: server side checkout...] Removing /usr/local/TeamCity/buildAgent/work/fc_converter [10:19:14]: [Updating sources: server side checkout...] Updating /usr/local/TeamCity/buildAgent/work/fc_converter |
This is the directory that contains the source for the FC Converter project. Press the browser's back button to return to the FC Converter project page.
The remaining steps of the build process can be setup to automatically run after successful completion of the immediately previous step. We’ll describe how to do this using the second step as an example. Then you can repeat the process for the third and fourth steps, selecting the appropriate preceding step in each.
Hover the mouse pointer over the down arrow immediately following the title of the second step, Step 2- Flex Compiles, to expose the dropdown menu. Select Edit Settings from the dropdown menu and the General Settings page for the step is displayed.
Select Build Triggering from the list under the Configuration Steps heading on the right margin of the page. We are going to add a build dependency trigger. Select the +Add new trigger link and then select Build Dependency Trigger in the Trigger Type dropdown of the pop-up window. Select Step 1- Collect Sources from the Build Configuration drop down and press Save.
Select the Projects tab at the top of the page to return to the FC Converter project page. Repeat the steps above starting with hovering the mouse pointer for the remaining two steps, selecting the previous step as a build dependency.
When you have finished setting the dependencies, press the Run button in the right margin of the Step 2- Flex Compiles step to launch the remainder of the build process. Each step will execute upon successful completion of the previous step. You can view the results of an individual step by hovering the mouse pointer over the down arrow immediately following the ‘Success’ status label under the step title and selecting either ‘Short log’ or ‘Full log’.
Points of Interest…
Referencing the checkout source directory of the FC Converter project (/usr/local/TeamCity/buildAgent/work/fc_converter)…
In the features/support directory you will notice two files: env.rb and firewatir_firefox_patch.rb. Files in the support directory are loaded by Cucumber prior to running features. The purpose of env.rb is to set up the environment for running features. The purpose of firewatir_firefox_patch.rb is as its name indicates: a patch for the FireWatir::Firefox class. The patch is to support a headless Firefox environment by pre-pending DISPLAY to the command that invokes Firefox.
The features/step_definitions directory contains the fc_converter_steps.rb file which defines the templates for the Cucumber steps used in the features/fc_converter.feature file.
Some things you may want to investigate on your own:
- Setting a build trigger for the first step dependent on SCM updates.
- Adding more features and Cucumber steps to support them.
- Generating some reports from the TeamCity server.
- Setting up email notifications on successful and / or failed builds.
Conclusion…
Although our project is a very simple example of using Cucumber, Flex, FunFX, and Firefox, the purpose of this article is to illustrate the setup of the server environment to support a remote, headless, continuous integration environment. And although it’s a little complex, it’s a once-and-done thing. Setting up TeamCity is relatively simple.
Hopefully the log of this journey has helped you in your quest to get Cucumber FunFX tests running in a remote CI environment.
Oh, and if you’re inclined, Chariot Solutions is running a CI event this fall.
In the future, look for a sister post to this one where we’ll use Hudson as our CI server.
Friday, September 3, 2010
Setting up TeamCity CI w/ Cucumber, Flex, FunFX, and Firefox in a Headless Environment: Part One
Recently I had the “opportunity” (read: challenge) to setup a continuous integration build for a project with a Flex frontend and a Ruby/Rails backend. The project had been converted from Java-Hibernate-Tomcat to Ruby on Rails. Cucumber was introduced for GUI testing using the FunFX adapter. It was my first experience with Cucumber, so the combination of Cucumber, FunFX, Flex, and Firefox was a little daunting at first. But we got through it and, actually, it turned out to be quite an effective tool in our testing suite.
But the fun really started when I was tasked with building the TeamCity CI server for our project. I scoured the blogs and tech sites for anything about CI builds of a Flex-Ruby/Rails project with Cucumber and FunFX. I found bits and pieces, like how to setup Firefox in a headless environment and how to setup TeamCity in a headless environment. So I filled in the rest around that to get the TeamCity server up and running with a continuous integration build for our project.
So why is a headless environment required? Our TeamCity CI server was a remote server- there is no input device or display device. So all of the setup was done via SSH through a VPN. But more importantly, we needed a headless implementation since FunFX requires browser interaction. FunFX “attaches” itself to a browser (via JSSH) so we needed a windowing emulator. We use the X virtual frame buffer of the X Window System (Xvfb) for this purpose (more about Xvfb at http://en.wikipedia.org/wiki/Xvfb).
After it was all over, I felt I needed to document it, if not just for posterity, but also so others might benefit. Since setting up the environment is as much an effort as setting up TeamCity, I decided to break it into two parts. So we’ll cover the basics of setting up the server in this first part and then in the second part we’ll tackle getting TeamCity set up. I hope you enjoy the trip as much as I did.
We're using a Debian-based system in this prensentation. We’re going to assume you’re familiar with your operating system and know what most of these commands are. Hopefully you can translate to your operating system, if different.
Basically, this is what we need to do:
· Install Java
· Install Ruby and Rails
· Install Flex SDK
· Install and Configure Firefox
o Test JSSH
o Install FlashPlayer
· Install the SCM Client
· Install TeamCity
o Create the CI project
o Create the build steps
o Run the build
So Let’s Get Started…
Because our Flex-Ruby/Rails project was a bit more complicated than what’s needed, plus it involved other requirements that do not need to be documented here, we’re going to use a small sample project, FC_Convert, an interface to convert Fahrenheit to Celsius. I have uploaded the project to my public github account and we reference it in these procedures.
First create a user account on the TeamCity server for the ‘teamcity’ user. This will create a /home/teamcity directory which we will be using. All downloads are to a local workstation and then copied to this home directory of the ‘teamcity’ user on the TeamCity server. It is assumed you will also have your own account on the TeamCity server with sudo capability that you can use to SSH for all of these tasks.
Install Java
Since this is a Ruby/Rails and Flex project, we do not need the JDK. For our project we are installing the JRE since TeamCity requires either the JRE or a JDK. If your build process requires Java, you’ll need to install the JDK.
Visit the Java website and download either the JRE or the JDK for the operating platform of your TeamCity server. Install the JRE or JDK per the installation instructions. In addition to adding the Java bin directory to the PATH environment variable, don’t forget to set the JAVA_HOME environment variable to point to the Java installation directory- TeamCity requires it.
Install Ruby and Rails
As of this writing, the latest version of FunFX (0.2.2) had a conflict with Ruby 1.9.1. Ruby 1.9.1 now includes FasterCSV but with renamed variables and methods. If you install Ruby 1.9.1 and then run FunFX, you will get the following message:
Please switch to Ruby 1.9's standard CSV library. It's FasterCSV plus support for Ruby 1.9's m17n encoding engine. |
FunFX uses the FasterCSV library and requires modification to run with Ruby 1.9.1. Since there are no features of Ruby 1.9.1 that we need for this project, we are installing Ruby 1.8.7 instead.
Installing Ruby and Rails is pretty straightforward. Depending on the platform of the server, Ruby can be installed as a set of pre-compiled executables (Windows) or compiled for the specific platform (Linux, Unix). The distribution files for Ruby installation can be found on the Ruby downloads web page.
And if your server’s operating system has a package installer, i.e., apt-get or yum, it’s even easier to install the latest version of Ruby 1.8:
$ sudo apt-get install ruby-full |
Otherwise, download the source tar, unpackage it, and then configure, build, and install Ruby.
$ cd /usr/local/src $ sudo tar -xzvf ruby-1.8.7-p302.tgz $ cd ruby-1.8.7-p302 $ sudo ./configure |
After Ruby is installed, you need to install the Ruby Gems package manager. Ruby Gems binaries can be downloaded from the project's download page. Again, check your server package installer first.
$ sudo apt-get install rubygems |
If the package installer is not available, download the RubyGems distribution file from the site, unpackage it, and install the package manager by running setup.rb in the install directory:
$ cd /usr/local/src $ sudo tar -xzvf rubygems-1.3.7.tgz $ cd rubygems-1.3.7 |
Installing Rails is a snap using the gems package manager.
$ sudo gem install rails |
Next you need to install any gems required by your application. Add your required gems to the list. NOTE: FunFX requires version 1.6.2 of the firewatir gem. Install it separately specifying the version.
$ sudo gem sources -a http://gems.github.com $ sudo gem install –v 1.6.2 firewatir |
Install Flex SDK
The Flex SDK and Flex compiler come in several flavors: Flex 3 or Flex 4, and Adobe Flex or Open Source Flex (not quite as feature rich as Adobe Flex). For purposes of this presentation, we’re installing the open source edition of Flex 4 and the Flex compiler. Visit the Adobe Open Source Flex web site Flex 4 downloads page and download the Flex 4 Open Source SDK zip file (we are installing Flex 4.1.0 SDK) as well as the Adobe Add Ons zip file.
Create the flex_4_sdk directory, unpackage the zip files, and set execute permissions on the bin, lib, and frameworks directories. If prompted to replace any files, reply yes.
$ cd /usr/local $ cd flex_4_sdk $ sudo unzip /home/teamcity/flex_sdk_4.1.0.16076_mpl.zip $ sudo unzip /home/teamcity/flex_sdk_4.1.0.16076_add_on.zip $ sudo chmod –R 755 /usr/local/flex_4_sdk/bin $ sudo chmod –R 755 /usr/local/flex_4_sdk/lib $ sudo chmod –R 755 /usr/local/flex_4_sdk/frameworks |
Install and Configure Firefox
Now it gets a little more difficult. Firefox must run as a headless application (no display device). We use the X virtual framebuffer of the X Window System (Xvfb) for this purpose (more about Xvfb at http://en.wikipedia.org/wiki/Xvfb).
First, insure Xvfb is installed on your system.
$ which Xvfb |
If this command does not return the location of Xvfb, it is not installed. Install it with your system’s package installer.
$ sudo apt-get install Xvfb xinit |
Start Xvfb and create a virtual display. This only needs to be done once on the server and after every reboot. Ideally, it could be started in a script that is run whenever the server is restarted.
$ startx – ‘which Xvfb’ :1 -screen 0 1024x768x24 & |
And since we will not have a browser window to install the JSSH plugin, we must build Firefox from source as opposed to installing binaries. But prior to building Firefox, you must insure your system has all the necessary development tools required to build an application. Here is a list of the development libraries I needed to load prior to executing a successful build of Firefox from source. I used the apt-get package manager for my Debian-based system.
$ sudo apt-get install libgtk2.0-dev $ sudo apt-get install libdbus-glib-1-dev $ sudo apt-get install libasound2-dev $ sudo apt-get install libIDL-dev $ sudo apt-get install build-essential gcc-3.4 $ sudo apt-get install libxt-dev $ sudo apt-get build-dep mozilla-thunderbird |
As of this writing we are installing the latest release of Firefox 3.5, 3.5.11. The Firefox 3.5.11 source code is available from the Mozilla repo at ftp://ftp.mozilla.org/pub/mozilla.org/firefox/releases/3.5.11/source/. Download the firefox-3.5.11-source.tar.bz2 file from the site, copy it to the TeamCity server, and unpackage it.
$ cd /usr/local/src $ sudo mkdir firefox-3.5.11 $ cd firefox-3.5.11 |
Before building Firefox, we need to add configuration parameters to the mozilla configuration by creating a .mozconfig file if it does not exist. If it does exist, it should be modified to look like this:
$ cd mozilla |
This file is used in configuring the options to be used in building the Firefox application. Set the MOZCONFIG environment variable to point to this file and build Firefox.
$ export MOZCONFIG=/usr/local/src/firefox-3.5.11/mozilla/.mozconfig $ sudo make -f client.mk build |
Create a symlink to the firefox binary in /usr/local/bin:
$ sudo ln -s /usr/local/src/firefox-3.5.11/mozilla/firefox-jssh/dist/bin/firefox /usr/local/bin/firefox |
Test JSSH
We use FunFX for Flex testing. FunFX is a Watir / SafariWatir / FireWatir extension that lets you talk to an Adobe Flex application straight from Ruby. Firewatir is a Ruby gem that uses the Firefox JSSH connection. We need to first create a Firefox profile for use by the Cucumber FunFX tests so we can modify the default settings. We’ll name it firewatir.
SSH into the TeamCity server as the ‘teamcity’ user and create the firewatir profile.
$ DISPLAY=localhost:1.0 firefox -CreateProfile firewatir |
This creates a profile directory in the /home/teamcity/.mozilla/firefox directory for firewatir. The directory name for the profile is a two-part name: the first part is a randomly generated string of 8 letters and numbers; the second is the profile name specified in the -CreateProfile command- in our case, firewatir.
A skeleton file named prefs.js is created in this directory. The default configuration settings can be overridden for a particular profile by including them in the prefs.js file in this directory. Normally this file is maintained by Firefox. Typically, configuration changes are made via the about:config URL. But we need to modify this file since we do not have access to the about:config URL in a headless environment.
Modify the prefs.js file to contain the following:
$ cat .mozilla/firefox/vrvkdf9e.firewatir/prefs.js # Mozilla User Preferences /* Do not edit this file.
user_pref("app.update.auto", false); |
You will need two SSH sessions to test JSSH. SSH into the TeamCity host server as yourself for this test. Start Firefox in the teamcity user SSH session (Note: Xvfb must have been started prior to this. See above.):
$ DISPLAY=localhost:1.0 firefox -jssh 9997 -P firewatir |
Then use telnet to connect to JSSH in the other SSH session (enter Ctrl-] to exit the shell):
$ telnet localhost 9997 |
Terminate Firefox in the first SSH session via Ctrl-C. If you do not do this and Firefox remains executing, you may have difficulty running your first Cucumber test.
Install FlashPlayer
Since we’re using Flex for the client interface, we need to install the Adobe Flash Player as a plugin to Firefox. Unlike a development system that has a browser window with which to interact requiring the debug version of Flash Player, we do not need the debug version for the headless version of Firefox.
Flash Player can be downloaded from here. The installation instructions can be found here but are simple and are documented as follows.
Note: Firefox should not be executing when these installation procedures are executed.
Download the tar.gz file and copy it to the TeamCity host server. SSH into the host server as the 'teamcity' user and unpackage the tar.gz file. Create a plugins directory in the 'teamcity' user's .mozilla directory (if it does not exist) and move the unpackaged libflashplayer.so file to the plugins directory.
$ tar -xzvf libflashplayer-10.0.45.2.linux-x86_64.so.tar.gz |
Install the SCM Client
We are using Git for our sample project. Git binaries can be downloaded from here. You can also install git using your system’s package manager.
$ sudo apt-get install git |
If you are using your own project for this exercise, install your SCM client as appropriate.
That’s it for this session. Next time we’ll setup TeamCity and create a sample project.



