Friday, March 19, 2010

Developing the ETE Conference iPhone Application

As you probably know, Chariot is hosting its annual ETE conference this April. This year we've added mobile device support for obtaining all the conference information an attendee would need. This included a mobile browser friendly web site, and both iPhone and Android native applications. I had the pleasure of working on the iPhone application and would like to describe, at a high level, the process and design/implementation decisions that were made along the way.

The Importance of Storyboarding

The first step in designing the mobile application was storyboarding how a user would access the available information about the conference. This turned out to be more valuable than I had imagined. Not only did it define the screens that needed to be developed and the data required by each screen, but the extra value was identifying where screens could be re-used. Given a large part of the application is viewing various lists of sessions, the amount of re-use I was able to attain was impressive and time saving.



Also, when my co-worker Andy Oswald started to work in the Android version of the application, he could follow the same basic flow through the application that I had laid out for the iPhone, albeit designed more to the native Android look and feel.

What about the Data?

The next issue addressed was data access. The initial decision was to access the server side data on an as needed basis. This could be accomplished numerous ways including web services, REST, etc. Since the ETE website was a Rails application, REST seemed like the obvious answer. After a little research, the Objective Resource framework was found. This framework makes the interaction with Rails based web applications easy. It provides a nice abstraction of the REST calls (via either XML or JSON) and object serialization that makes remote data access simple. After some initial experimentation, it was realized that when generating JSON on the server side, Objective Resource did not support the serialization of nested objects on the client side. Since our data model had nested object graphs (Sessions have Speakers, etc.) we were forced to use XML. Sure, we could have modified the framework, but time was not on our side.

Is it really that fast?

So far, so good. We had our screen designs and our remote data access all defined. Development started and progressed nicely. Once the core functionality was developed, I moved the application from the simulator to my iPhone for more realistic testing. Suddenly, the remote data access, and therefore screen loading, seemed a bit sluggish. So, I retrieved the data on a separate thread, added progress screens while data was being retrieved from the server and loaded, and I added some local in-memory caching. This still seemed a bit sluggish, so I investigated using JSON rather than XML. It turns out the JSON was lighter weight and quicker to parse. This required some redesign on both the server side and client side, as we had to flatten out the data model being sent down to the iPhone. This wasn't too bad and it improved the user experience a good deal. But the application still didn't seem responsive enough. We kicked around different options for a more persistent local cache, but these tended to introduce what seemed to be too much complexity in staying synchronized with the server. Then Andy Oswald (developer of the ETE Android application) came up with a great solution.

Protocol buffers and local storage for a better experience

He suggested the use of Protocol Buffers to transmit data to the mobile applications. The Rails application would listen for changes to the data and subsequently generate a new protocol buffer file. Our mobile applications would make a request to the server indicating the date of the version of our local file (using the if-modified-since request header), and if a newer version existed in the server, it would return the file. If our version was current, we would receive an HTTP 304 response code. Our applications could then handle the response accordingly. This way, our local data was always synchronized with that on the server.

Since the iPhone application was now going to have a local copy of the server-side data, it was time to re-examine the data access for the application screens. Our data model was small and the relationships were pretty basic, so I decided to pursue the use of Core Data as the new data access and persistence architecture. Core Data provided the ability to deal with objects, abstracted away the actual persistence, and build queries as needed to support the various view the application presented to the users. Also it was quite performant, so the responsiveness and user experience improved greatly. Since the initial design included a nice separation between the view and data access code, this change was pretty non-invasive to implement.

So, in the end I learned the importance of laying out the user interface upfront, testing on the device early and often, and the value of local data (if possible) on the users mobile application experience. We also got a pretty cool application for the ETE 2010 Conference.

More info on the iPhone App can be found here
More info on the the Android App can be found here

Tuesday, March 9, 2010

Rapid Mobile Application Prototyping with PhoneGap

There is no doubt that carving out a mobile presence is a priority for many companies offering any type of product or service, or for start ups looking to gain rapid exposure. The Apple App Store has over 100,000 apps on it, and the Android platform is gaining ground. Those are just the cool kids – let's not forget the grandfather of mobile technology, Blackberry. Others in the class include Palm, Symbian and of course, Windows.

Multiple Platforms
With so many platforms available there is a risk of losing focus on developing and producing applications as resources are expended on supporting multiple platforms – a necessary evil as each presents different development environments and eccentricities.
This is why PhoneGap really stands out at the moment as four [1] of the aforementioned platforms share a common factor – a WebKit-based browser. This doesn't include BlackBerry's announcement [2] of their new WebKit based browser to come in late 2010, which takes us to five out of the six platforms.

Enter PhoneGap
Where does PhoneGap sit in this picture? Well, it jumps off the canvas and sits in your lap! PhoneGap allows you to develop applications using HTML, CSS and JavaScript and provides a JavaScript API which enables you to interact with a device's native functionality [3]. In addition to this simple application architecture, PhoneGap comes with build scripts/plugins (PhoneGap extension for XCode) which bundle up your pages into a deployable application.
The packaging and deployment to device for iPhone, Android and Palm was trivial; I can only speculate that the remaining platforms are equally easy to deploy. In light of recent Apple 'crackdowns', as of October 7th [4], 2009, it is worth noting that PhoneGap (version 0.8.0 and higher) is Apple approved.

The JavaScript advantage?
Anyone who has used a JavaScript library, Prototype, jQuery, MooTools etc, knows how powerful they can be. Anyone who has not used them can easily pick up JavaScript basics. The guys over at PhoneGap recommend using XUIJS[5] with PhoneGap as it does not contain a lot of cross-browser compatibility overhead and weighs in at < 10kb compared with jQuery at 24kb – although for simple functionality you may want to avoid using a extra library at all.
Due to the fact that we are ultimately aiming to work inside one browser implementation (WebKit), we are not held hostage by the browser compatibility issues that have haunted JavaScript for so many years. That's not to say that fragmentation in the mobile world is not an issue [6].

Now, I am not going to try to argue that these HTML/JavaScript apps offer greater performance or a better user experience over their native counterparts. However, I will say that PhoneGap will allow the rapid development of an application for multiple platforms, using a single code base [7]. This is ideal for prototyping or for apps that will have a limited life span and do not warrant the time and investment of producing native apps.Considering all of this in combination with developments coming down the pipeline, most specifically HTML5 (which includes WebSockets and Client-side database storage), the potential for these types of mobile applications is vast and genuinely exciting.[1] Android, iPhone (safari), Palm (WebOS) Symbian/S60(WRT) http://en.wikipedia.org/wiki/List_of_web_browsers#WebKit-based_browsers

[2]http://devblog.blackberry.com/2010/02/blackberry-webkit-browser-preview-at-mobile-world-congress/

[3]http://nachbaur.com/blog/phonegap-officially-permitted-on-the-app-store

[4]Checkout the road map for specifics: http://phonegap.pbworks.com/Roadmap

[5]http://code.google.com/p/xui-js/

[6]http://techcrunch.com/2010/03/04/mobile-fragmentation-forever/

[7]ok, you are going to have to tweak it for BlackBerry.

Wednesday, February 17, 2010

Mobile Platforms - Interview with Roy Rosin and Rick Rasansky

I am a little behind on editing a backlog of videos. As I continue to learn how to edit and not become frustrated with iMovie (why is it Apple can create so many great, intuitive products, yet this one is difficult to use! That will be another blog post for another time!)

This interview took place a few weeks ago, but it seems to be more relevant now with the release of Windows Mobile. I sat down with Rick Rasansky and Roy Rosin at their offices to discuss why it is important to be paying attention to mobile development. Since both of these gentleman have been in the mobile space since early on, they have a lot of experience to draw on.



Rick and Roy will be part of a panel at the Emerging Technologies for the Enterprise Conference (www.phillyemergingtech.com) on April 8th. They are part of a management track which is addressing different Buzz word topics (cloud, social media, mobile). The goal is to help managers have good, non-vendor driven information in their arsenal to guide their choices in 2010.

Sunday, February 7, 2010

Interview on JQuery

Two weeks ago, Chariot Solutions had a day long event for our consulting team. With the original name of Chariot Day, our team has a series of talks on all different subjects. Some of the talks are on topics which may never see the light of day in the enterprise, some are about more accepted tools. Some of those talks and presentations will be the beginnings for seminars we will host for the public.

I thought I would take advantage of having so many topics to choose from for interviews. I will be posting them as I have the time to edit them.

This first one, with Lyle Anderson, is on JQuery. Lyle gave a two-part presentation which was well attended by our architects. When I see that much interest from the team, I know this is a technology worth watching.




We are still waiting to confirm, but it looks like we will be having a talk at ETE 2010 (www.phillyemergingtech.com) by Yehuda Katz.

Saturday, January 30, 2010

Why You Need NoSql in Your Toolbox

Even if you work for Oracle, you still need NoSql databases in your toolbox.

One size does not fit all for programming languages, operating systems, IDEs, shoes, bailouts, or anything else. But for a long time now, many developers have been told that relational databases are really the only choice for persistence. If you need to store some data -- any data -- shove it in a relational database. It is the safe choice, and gives us developers one less thing to worry about, right? Well actually, wrong. RDBMS' are great solutions to many problems, but there are better alternatives for others.

Quick overview of NoSql

A simple definition of a NoSql database is a schema-less database that does not support joins or ACID transactions. In a broad simplification, there are 2 types of NoSql databases:

  1. Key/value -- A "giant hash table in the sky" that stores key value pairs.

    Pros:
    * very fast
    * very scalable
    * simple model
    * many are distributed

    Cons: many data structures (objects) can't be easily modeled as key value pairs

  2. Schema-less -- Databases somewhere in-between key/value and relational including: column-based (different rows in same table can have different columns), document-based, and graph-based

    Pros:
    * Schema-less data model is richer than key/value pairs
    * eventual consistency
    * many are distributed
    * still provide excellent performance and scalability

    Cons: typically no ACID transactions or joins

  3. Relational database

    Pros:
    * rich data models
    * ACID transactions
    * joins, reductions (group by, sum), ordering
    * lots of apps already integrate with RDBMS
    * mature, commercial support easy to find

    Cons:
    * complex
    * object to relational mismatch
    * can be difficult to scale horizontally

Performance and Scalability: Less Features = More Speed/Scaling

"Constraints proceed performance"
- Joe Gregorio, Google

"Constraints proceed performance" really sums up why many of the NoSql databases can be blazingly fast -- they implement fewer features and have some significant constraints when compared to a relational database. The key is determining whether you can live with those constraints. If you can, you reap the performance and scalability benefits. But before you become enamored with the allure of more performance, here's a list of things you'll probably have to give up:
  • joins
  • group by
  • order by
  • indexes
  • ACID transactions
  • SQL as a sometimes frustrating but still powerful query language
  • easy integration with all of the other applications that support SQL
Yikes. That's a pretty hefty price should you actually need one or more of those features. However, for many applications that perform CRUD operations on data largely partitioned by user, you might not need as many as you think.

On the scalability side of the equation, many of the NoSql databases were designed to be distributed so they can scale horizontally. Relational databases are not designed to scale horizontally. If you application really needs to scale up, vertically scaling your relational database is only going to get you so far because eventually you won't be able to buy a big enough box to scale any further. At that point you have to look at sharding your data across multiple database instances, and then you lose many of the benefits of a relational database across instances such as joins, transactions, etc.

In addition to get the performance you need, you very well may need to denormalize some of your data to eliminate slow joins. Additionally, to really scale up you may need to use a distributed caching layer like memcached that requires key/value semantics, in which case you are really starting to diverge from the pure relational model. So when making the comparison, it may be more fair to compare a sharded (data partitioned into multiple databases), partially denormalized relational database using a caching layer with an alternative NoSql database.

Productivity: Object to Relational Mapping Is Not Fun (Unless You Are Weird)

Object to relational mapping is difficult to implement, difficult to reason about, and easy to get wrong. As a developer, you either have to get in the weeds and write your own queries, or rely on a ORM tool like Hibernate which is great until something doesn't work and then you have try to find out why the magic isn't working anymore.

There is a lot of effort put into solving the object to relational mapping problem, and it still hasn't be completely perfected. This is where, depending on your data, a NoSql database can provide an abstraction that maps closer to your object model than relational tables. Column-based stores like BigTable and Cassandra essentially allow different rows in the same table to have different columns. Why is that useful? There a lots of cases where data is optional or slightly different between entities. If I was creating a ratings site, I could rate different things using different criteria, but they could all be stored in the same table. I won't need a table of "things" with a join table and a table of "criteria".

On the other hand, why is that dangerous? Obviously the application could get back data it didn't expect because the schema wasn't enforced by the database. Document-based stores like CouchDB give you even more flexibility where you store entire documents that can contain anything you want. No longer are you a slave to that crusty database administrator and all of his rules, but on the other hand you could easily hang yourself with all of this new rope. What happens later when you need to run reports on this data and you don't have a group by or sum function? You can write your own map reduce functions to essentially do the same thing, but you have to do the work.

Another point to consider is that many "rapid development" web frameworks like Ruby on Rails, Grails, Django, and Lift already have substantial ORM tools baked in. If you use NoSql for your persistence you won't be able to take full advantage of those frameworks. Some of the frameworks are starting to support some NoSql databases, but since NoSql is just a general label and not a specification like SQL, support is one off. For instance, there is a Grails plugin to provide support for Google App Engine's version of JPA.

You can't have it all; you have to decide between the complex features that a relational database supports versus the constraints/benefits that a NoSql database provides. But whether you put 99% of your data in a relational database and only 1% in a NoSql store, or the other way around, both types of databases are likely to be valuable to your business. I like the interpretation of NoSql as "Not Only SQL" meaning both types of databases can and should co-exist within your organization.

Why Now?

Non relational databases have been around since ENIAC, so why are we hearing so much about them now? My opinion is that there are 2 drivers:

1. The massive scalability requirements of some websites has exposed one of the weaknesses of relational databases: scalability. Sites like Google, Amazon, Twitter, and Facebook all use NoSql databases for major components of their site.

2. A strong open source community means that there are many high-quality, production-ready NoSql databases available. There probably have been hundreds of very good, proprietary non-RDMS databases created in the last 20 years, but since they are proprietary, they have been used at one company and no one has ever heard of them. Open source has changed that, and now companies like Amazon, Google, and Facebook are contributing incredibly valuable research and software to the community. Research papers such as Amazon's Dynamo and Google's BigTable inspired many open source implementations.

This is a fairly recently phenomena. I'm not aware of any production-ready open source non-relational databases that existed 10 years ago, and if there were, there weren't many and they weren't popular. Now there are well over 15 (and probably many more, I don't claim to know the exhaustive list) production-ready open source NoSql databases. Am I going to list them all? No, because if I do I'll miss one, and I'll get a comment saying "You didn't mention XYZ database, what's wrong with you?" But I'll list the ones I'm currently looking at in no particular order:
  • Voldemort
  • Cassandra
  • Redis
  • CouchDB
  • MongoDB
  • Terrastore
Some links for more reading:

Sunday, January 24, 2010

Mobile development and your team

Throughout the next several weeks, we will be conducting interviews with speakers who will be speaking at the Emerging Technologies for the Enterprise conference (www.phillyemergingtech.com). These interviews will give you some idea of the material we will be covering.

The first interview is with Mark Spence of Transmogrify and Rick Rasansky of Yorn. Rick and Mark will be on a panel discussion in the management track discussing why it is important to pay attention to mobile development for you company.

We tackle a different issue here. Helping your development team get up and running in this newer space.

(As a note, I am trying to get better with editing. Continue to bear with me while I learn "on the job". Please note, I finally purchased a tripod for a steadier picture!)



Here are the links Mark mentions in our interview:

http://cocoawithlove.com/ (cocoa + iphone dev blog)

http://akosma.com/ (iphone dev blog)

http://github.com/facebook/three20 (three20 on github)

http://iphonedevelopment.blogspot.com/ (Jeff LaMarche's iPhone Dev blog)

http://blog.iphone-dev.org/

http://blogs.msdn.com/windowsmobile/ (WinMo dev blog)

http://www.christec.co.nz/blog (Chris Fairbairn's mobile dev blog)
From a design point of view, we like to look at these sites for inspiration:

http://tapfancy.com/

http://www.appsafari.com/

http://wirelessdesignmag.com/

Other useful sites for keeping an eye on everything mobile:

http://gizmodo.com/

http://www.eweek.com/c/s/Mobile-and-Wireless/





Monday, January 4, 2010

Where is Java Headed?

Welcome to 2010!

As a way to start the new year, Joel Confino, one of our architects and future blog contributor, sat down for an interview to discuss his opinions on Java's future course. As you will hear in this interview, there are more options as Java is maturing.



We will be exploring some of these alternatives through 2010 on our blog, on our podcasts (Chariot Techcasts), and at our events, especially ETE 2010 (www.phillyemergingtech.com).