Monday, February 11, 2013

A Guiding "Light" to Learning Clojure


I've recently started learning Clojure after a little encouragement from co-worker of mine.  I've been developing in Java for a while now and spent the last 1 1/2 years with Groovy.  The Groovy experience was great and the use of closures certainly "whet my appetite" for functional programming.

Ok, for me, when I learn something new I need to have tools that are not distracting so that I'm focused on the material.  In this case, I wanted to take a little time to view the tools landscape for Clojure while trying not get carried away.  After doing some research, I narrowed it down to a few options.  First, I'm a long time Intellij IDEA user so I could just download the La Clojure plugin (I did) to get a REPL, syntax highlighting, source code navigation, etc.  This is fine but it's a bit of an overkill for just getting started with Clojure.  Intellij IDEA is a great IDE however, I wanted to stay open-minded to what else is out there.

Next, I tried configuring Vim for Clojure after seeing a co-worker work on a Clojure project with Vim.  The great thing about Vim is it's lightweight editor that's extremely fast.  If you're proficient Vim user then you get things done quickly.  If you're a basic Vim user (like me) then there's a learning curve.  Regardless, I thought I'd give it a try by downloading and configuring VimClojure.  There is a way to configure Vim to dynamically work with Clojure to provide documentation lookup, REPL running in a Vim buffer, etc.  You have to build a Nailgun interface by downloading a client program and so on.  I eventually got a dynamic Clojure development tool in Vim but I found that I was focusing more on learning this Vim/VimClojure/Nailgun environment than on Clojure!  So, I reluctantly moved on.

I thought about jumping into the deep end with Emacs, which is the choice for most Clojure developers but I'm not an Emacs user (please don't hold it against me).  I didn't want to get involved in a big learning curve again.  I was feeling a bit like, "Hi, my name is Rod and I'm a tools-aholic"...this was starting to consume a lot of my time and taking away from what I set out to do...learn Clojure.

Well the tools story continues, in November 2012 I attended Clojure-conj in Raleigh, NC.  That's where I saw Chris Granger present LightTable for the first time.  I was impressed with the presentation and thought to myself that I need to take a closer look at LightTable some time (not immediately but some time).  I was trying to resist the temptation and I did for a little while.  Well, three months later I "fell off the wagon" and took a "drink" of LightTable and I'm glad I did. :)


This is not a review of LightTable nor a "HowTo", I think of it more as getting the word out about a new and different kind of developer tool.  The UI is simple with a very modern look and color scheme.  My first impression was this could be a next generation Vim editor.  I found myself wanting to use the tool.  The intuitive UI is easy to use and it feels natural like you're closer to the terminal.

Instarepl


What appeals to me the most happens to be one of LightTable's guiding principles "Discover by doing".  The REPL (Read-Eval-Print-Loop) provides an interactive session to write and execute code.  This isn't something new to the "Lispers" since REPL has been around a long time.  However, the REPL (Instarepl) in LightTable takes things one step further.

Here's your typical REPL TryClojure.  Simply write some Clojure code and get the result.


Here's the same basic example in LightTable's Instarepl.


Ok, that's cool, you see what you type on the left, which is mirrored on the right with the result.  However, it's not much different than the Clojure REPL.  But wait, it gets better.


Here's a simple function that I pulled from Chris' blog post.  You can see that the right side of Instarepl is just showing the function as it was written.  However, things change after writing code to call the function.


The right side shows the data flowing through the function.  It is pretty awesome to try various inputs and watch how the values flow through the code.  I'm not sure if Chris has officially coined a term for this feature but I've read it being referenced as "live evaluation" or "real-time evaluation and visualization".  This is what makes me want to use LightTable even if nothing else works. :)  I have to wonder how this type of code "visualization" impacts TDD (Test-Driven Development).  It would be interesting to explore LightTable's impact on TDD...sounds like a good blog post. :)

There are many more features to LightTable and I highly recommend that you watch the video on the LightTable web page and see Chris drive the IDE.  The version I'm currently using is 0.2.7 and some of the features that he demonstrates in the video I wasn't able to get working.  However, there is a new 0.3.0 version coming out Feb.-Mar. timeframe and (according to Chris) this version is about "crafting an editing experience so a lot of things will be much, much better"...can't wait, I have seen the Light!

Monday, February 4, 2013

A Spring summary: The Spring framework is still relevant



This article by Ken Rimple is from a post on Serverside.com
A Spring summary: The Spring framework is still relevant 
With the advent of the simpler versions of Java EE comes a chorus of Java EE aficionados calling for developers to jettison proprietary application frameworks and embrace the core. Spring is one framework targeted for elimination, and several articles and tutorials have claimed that it's obsolete.
To kick off a regular monthly column on TSS, I'll tackle the question of why the Spring framework is still popular today. I'll discuss the platform's strengths, innovations, the power of the Spring container, contrasts and comparisons with Java EE, and some areas where SpringSource, a division of VMware, is totally ahead of the game.
Before I begin, let me lay my bias on the table. I've been working with Spring since 2006, when I ran into it on a project at my prior employer. I had been working with Java EE since the original 1.0 version and Java since 1.0.2. When I saw how the Spring framework simplified my life, I looked for a chance to work with it more. Having a background as a mentor, trainer and leader, I approached Chariot Solutions about teaching Spring in 2007, and I've been here ever since.
Chariot is a SpringSource/VMware training and solutions partner. I lead our training team, but these opinions are my own. Chariot also partners with a number of other technology firms in the enterprise application integration and development space.

Why is Spring so popular?

Simply put, Spring is so popular today because it simplifies development drastically for Java EE developers. Rod Johnson created Spring when he was writing a book on J2EE application design and felt the frustrations that most of us had with the specification. One chief concern was the emphasis on architecting separate Enterprise JavaBeans and processing them to generate deployment JARs. Another was the API's focus on framework interfaces -- home, local and remote interfaces -- not to mention that back in 2001, it took an expensive, heavyweight platform such as WebSphere or WebLogic to deploy these applications.

Wednesday, January 30, 2013

HTTPS with Client Certificates on Android

Many Android applications use REST or another HTTP based protocol to communicate with a server.

Working with HTTP and HTTPS on Android is generally fairly straightforward and well documented. Depending on the version of the Android OS, either HTTPClient or HttpURLConnection “just work”. Either one can be used, but the official recommendation is to use HttpURLConnection for Gingerbread or later, and HTTPClient for Froyo and earlier.

Things get interesting, however, once you go past “plain vanilla” HTTPS. On a recent project, we needed to communicate with an HTTPS server that required client certificates, and which used a self-signed server certificate. This presented some interesting challenges. The rest of this post describes how we met those challenges.

If you want to follow along by trying out the code, it’s available on GitHub. You will, of course, have to set up your own HTTPS server and your own server and client certificates. One way that you can do this is by using the Apache mod_ssl module.

Before starting out with using HTTP or HTTPS on Android, make sure that your application has permission to access the network by adding the following to your AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET"/>

Challenge #1: Using a Client Certificate

Our client certificate was issued in the PKCS 12 format, as a .p12 file.

To give our application access to the certificate, we used the DDMS utility to copy the certificate file to the root directory of the phone’s sdcard. If you are using an emulator, you can do the same thing to copy the certificate to the emulator’s sdcard.

You can also choose to mount the phone as a disk drive when you plug in the USB cable, and copy the file that way.

In our final application, we “imported” the certificate by copying it to application-specific storage, and then deleting the original from the sdcard root directory.

The secret to using our client certificate is setting up a custom KeyStore containing the certificate, and then using it to create a custom SSLContext.

Once we have a reference to a File containing the client certificate and the password for the certificate, we load it into an appropriate KeyStore (see the sample SSLContextFactory.java):

keyStore = KeyStore.getInstance("PKCS12");
fis = new FileInputStream(certificateFile);
keyStore.load(fis, clientCertPassword.toCharArray());

Now that we have the KeyStore containing the client certificate, we can use it to build an SSLContext:

KeyManagerFactory kmf = KeyManagerFactory.getInstance("X509");
kmf.init(keyStore, clientCertPassword.toCharArray());
KeyManager[] keyManagers = kmf.getKeyManagers();

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, null, null);

The SSLContext can then be used with an HTTPUrlConnection to connect to the server:

String result = null;

HttpURLConnection urlConnection = null;
try {
    URL requestedUrl = new URL(url);
    urlConnection = (HttpURLConnection) requestedUrl.openConnection();
    if(urlConnection instanceof HttpsURLConnection) {
        ((HttpsURLConnection)urlConnection).setSSLSocketFactory(sslContext.getSocketFactory());
    }
    urlConnection.setRequestMethod("GET");
    urlConnection.setConnectTimeout(1500);
    urlConnection.setReadTimeout(1500);

    lastResponseCode = urlConnection.getResponseCode();
    result = IOUtil.readFully(urlConnection.getInputStream());
    lastContentType = urlConnection.getContentType();

} catch(Exception ex) {
    result = ex.toString();
} finally {
    if(urlConnection != null) {
        urlConnection.disconnect();
    }
}

Challenge #2: Trusting a Self-Signed Server Certificate

We now have Android client code that can connect to an HTTPS server and present a client certificate.

However, this only works if the server’s certificate is trusted. In practice, this means that the server certificate must be signed by one of the major certificate authorities, such as VeriSign, Thawte, Geotrust, Comodo, etc. CA certificates for these well-known providers are pre-loaded into the phone’s default trust store.

In our case, the certificate was self-signed. This means that the default TrustManager in our SSLContext will not trust the server’s certificate, and the SSL connection will fail.

If you search the web for ways to trust self-signed server certificates, you’ll likely see a lot of really bad advice. Most responses to this sort of question involve setting up a custom TrustManager that simply trusts everything. This approach is both pointless and obviously insecure.

What we really want to do is to set up a custom TrustManager that trusts our self-signed certificate, and provide that TrustManager to our custom SSLContext. To do this, we need a copy of the server’s certificate chain, which will have to include at least the self-signed CA Certificate and the Intermediate Certificate that is signed by the CA. If you simply export the server’s certificate, you will actually get a file with three certificates - the CA certificate, the Intermediate Certificate, and the server’s certificate. This is fine - the extra certificate is not needed, but won’t hurt, either (usually - see Challenge #3 below).

The method for trusting the self-signed server certificate is very similar to the method for providing a client certificate - we load the certificate into a KeyStore, use that KeyStore to produce an array of TrustManagers, and then use those TrustManagers to create the SSLContext.

In our app, we included the server certificate file in the application resources (since it doesn’t change from user to user, and doesn’t change very often), but you could put this in an external file as well.

Here’s the code for loading server certificate chain into a KeyStore that will be used as a “trust store”, based on having the base-64-encoded (.pem) certificate as a String:

byte[] der = loadPemCertificate(new ByteArrayInputStream(certificateString.getBytes()));
ByteArrayInputStream derInputStream = new ByteArrayInputStream(der);
CertificateFactory certificateFactory = CertificateFactory.getInstance("X.509");
X509Certificate cert = (X509Certificate) certificateFactory.generateCertificate(derInputStream);
String alias = cert.getSubjectX500Principal().getName();

KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
trustStore.load(null);
trustStore.setCertificateEntry(alias, cert);

Now that we have the “trustStore” KeyStore with the server’s certificate, we use it to initialize the SSLContext. Adding to the code that we had before, production of the SSLContext now becomes:

KeyManagerFactory kmf = KeyManagerFactory.getInstance("X509");
kmf.init(keyStore, clientCertPassword.toCharArray());
KeyManager[] keyManagers = kmf.getKeyManagers();

TrustManagerFactory tmf = TrustManagerFactory.getInstance("X509");
tmf.init(trustStore);
TrustManager[] trustManagers = tmf.getTrustManagers();

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null);

Now, we can securely connect to our server, trust its certificate (but not others), and present our client certificate. If all goes well, you’re done, and can safely stop reading.

Challenge #3: Handling an Improperly Ordered Server Certificate Chain

As it turns out, some Apache mod_ssl installations (and possibly other SSL providers), whether due to a bug or mis-configuration, provide the server certificate chain in the wrong order.

To work properly, the certificates in the server’s certificate chain must start with the “root”, or CA certificate, followed by any intermediate certificates. If the server’s certificate is included, it must come last. Each certificate in the chain (other than the root) must be preceded by the certificate that was used to sign it.

In our case, a mis-configured server was producing a certificate chain that started with the root CA certificate, but which had the server’s certificate next, followed by the intermediate certificate. This caused the SSL handshake to fail.

To work properly, the certificates in the server’s certificate chain must start with the server certificare, followed by any intermediate certificates. If the “root”, or CA certificate, is included, it must come last. Each certificate in the chain (other than the root) must be followed by the certificate that was used to sign it.

In our case, a mis-configured server was producing a certificate chain that started with the server certificate, but which had the root CA certificate next, followed by the intermediate certificate. This caused the SSL handshake to fail.

The most straightforward thing to do in this situation, if you can, is to correct the server’s certificate chain. If you can’t, then the out-of-order certificate chain must be handled by the client.

In our Android app, we handled the out-of-order certificate chain by implementing a custom X509TrustManager that re-orders the chain.

The code for the custom X509TrustManager is too large to include here (see the full source on GitHub), but once we have it implemented, we use it in the creation of the SSLContext by replacing the ‘trustStores’ array with one containing the custom implementation:

KeyManagerFactory kmf = KeyManagerFactory.getInstance("X509");
kmf.init(keyStore, clientCertPassword.toCharArray());
KeyManager[] keyManagers = kmf.getKeyManagers();

TrustManager[] trustManagers = {new CustomTrustManager(trustStore)};

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null);

By loading our own certificates into our keystore and trust store, and re-ordering the server’s certificate chain, we were able to provide an SSLContext that let us connect to an SSL server with client a certificate and a badly-ordered, self-signed server certificate chain.

Wednesday, January 16, 2013

List of technologies we're watching in 2013

From cloud computing to Javascript, from micro services to repeatable environments, from Android and HTML 5 to Big Data and concurrent programming, Chariot's engineers are keeping watch.

The Internet of Things

The “Internet of Things” is all around us.. or, well, we’d like it to be. From “Just in Time” delivery of bus or train schedules as you leave your house, to sensors tracking air quality in the city, from phones to smart clothing, the Internet of Things is about integrating ourselves with this data available all around us. Free of ‘apps’ and ‘services,’ the Internet of Things will make data available ‘just in time’ wherever and whenever you need it. Whether you get the data from WiFi, RFID or QR codes it’s out there, and we think that’s exciting!

Cloud Host Almost Everything!

If you're a savvy IT person, it's may have been a while since you provisioned your own Linux host on a Virtual Private Server or hosting plan. Cloud hosting isn't emerging, it's emerged and taking over. You're likely to use a cloud-based service such as Heroku, Amazon EC2, SliceHost, or even Microsoft's Windows Azure. There are private cloud partitioning services to protect your data from theft in co-location.  Companies such as CloudBees have been shifting so many services to the cloud that now they even provide development and QA platforms that provide your SVN or Git repositories, run Hudson or Jenkins, and provide Continuous Integration services.

Read the rest of this article, subscribe to our TechCast podcast or view all of our Philly Emerging Tech 2012 sessions at our emerging technology resources site, emergingtech.chariotsolutions.com.

Thursday, January 10, 2013

Science of Big Data - sessions coming online

If you didn't attend last month's "Science of Big Data" event in downtown Philadelphia, not to worry. We're beginning to post screencasts of the sessions now:  The Science of Big Data - Screencast Sessions.

Currently we have two - Basho's Tom Santero discusses "Eventual Consistency" and outlines the challenge of keeping data highly available and consistent enough without impacting performance.

The second screencast is the combination of Chariot's Andrew Oswald and Comcast's Josh Angotti's talks. Andy reviews the work that our team did for Comcast using Hadoop in analyzing set-top box users' click selections, and reducing the timeframe of finding answers from days to hours or minutes. Josh Angotti tells the story from Comcast's perspective.

More screencasts are expected soon - we had some video syncing difficulties and audio challenges so it'll take a bit to get them all online. Check the page soon for talks by Cliff Moon of Boundary, Stuart Sierra of Relevance, and Mark J. Headd, the Philadelphia Chief Data Officer.



Wednesday, December 5, 2012

The "Pivotal Initiative" - VMware and EMC spin off a new organization to focus on big data in the coud


Disclaimer - I am a Chariot Solutions employee, however this analysis is my own and should not be interpreted as the policy or official stance of Chariot Solutions. I direct our VMware Authorized Training Center, and provide training in Spring, tc Server and other technologies. Chariot is a VMware solutions partner as well.

On December 12th, EMC and VMware announced a spin-off that will merge units from both VMware and their parent company into a single, focused organization named the Pivotal Initiative.

The main focus of this operation, which will involve more than 1,400 employees from both companies, will be cloud-based big-data software. The merger brings together the vFabric and SpringSource teams (including Cloud Foundry, projects such as GemFire, Hyperic and tc Server), recently acquired Pivotal Labs, and data warehousing and data science teams from Greenplum, as well as recently acquired Cetas.


Thursday, November 15, 2012

Installing PhoneGap Plugins with pluginstall

Phonegap applications can be extended with plugins to add additional functionality. Historically, installing plugins meant copying a bunch of files into the correct places in your project.

Andrew Lunny wrote the Cordova Plugin Specification and pluginstall. Pluginstall is a tool that automates the installation of plugins which conform to the plugin specification.

In addition to demonstrating pluginstall, we’ll also use the PhoneGap create script to make a new project. The cordova build scripts will be used for compiling and installing the binary. Using these tools it’s possible to develop PhoneGap apps with a text editor, instead of an IDE.

Prerequisites

I’ll assume that you have Phonegap 2.2.0 and the Android SDK installed.

Pluginstall requires node.js and is installed through npm. If you’re on a mac and use homebrew, install node and npm with brew install node.

Setup

Install pluginstall

$ npm install -g pluginstall

Create a new Android project using the PhoneGap create script. Create takes the new project path, package name and activity name as arguments.

$ cd /usr/local/phonegap-2.2.0/lib/android
$ ./bin/create ~/foo com.example.foo Foo

The create script built a new PhoneGap project in directory ~/foo. Inside the project there is a cordova directory with some command line tools.

Let’s ensure the generated project works before we modify anything. Start the emulator.

$ cd ~/foo
$ corodova/emulate

Patiently wait for the emulator to start. Unlock the screen and keep the emulator running. Compile, build, install, and launch your app.

$ cordova/BOOM

The application should launch and say “Device is ready”.

Install the plugin

Get a local copy of the Child Browser Plugin.

$ cd /tmp
$ git clone git://github.com/alunny/ChildBrowser.git

Install the plugin using pluginstall. Pluginstall takes the platform, project directory and plugin directory as arguments.

$ pluginstall android ~/foo /tmp/ChildBrowser

The Child Browser plugin is now installed. Pluginstall used to the information in the plugin.xml manifest to update the project. It added an Activity to the Android manifest, listed the plugin in config.xml, added java source code, and added javascript code and images to our project.

Write some code to use the plugin

Edit ~/foo/assets/www/index.html

Include the javascript after cordova.js

<script type="text/javascript" src="cordova-2.2.0.js"></script>
<script type="text/javascript" src="childbrowser.js"></script>
<script type="text/javascript" src="js/index.js"></script>

Edit ~/foo/assets/www/js/index.js

In onDeviceReady, add a call that will open a web page in the Child Browser after 3 seconds

onDeviceReady: function() {
    app.receivedEvent('deviceready');
    setTimeout( function() {
        window.plugins.childBrowser.showWebPage(
            'http://phonegap.com', { showLocationBar: true }
        );
    }, 3000 );        
},

Compile, build, install, and launch your app into the running emulator.

$ cordova/BOOM

The sample application should run and open http://phonegap.com in the Child Browser Plugin.