Wednesday, January 30, 2013

HTTPS with Client Certificates on Android

Many Android applications use REST or another HTTP based protocol to communicate with a server.

Working with HTTP and HTTPS on Android is generally fairly straightforward and well documented. Depending on the version of the Android OS, either HTTPClient or HttpURLConnection “just work”. Either one can be used, but the official recommendation is to use HttpURLConnection for Gingerbread or later, and HTTPClient for Froyo and earlier.

Things get interesting, however, once you go past “plain vanilla” HTTPS. On a recent project, we needed to communicate with an HTTPS server that required client certificates, and which used a self-signed server certificate. This presented some interesting challenges. The rest of this post describes how we met those challenges.

If you want to follow along by trying out the code, it’s available on GitHub. You will, of course, have to set up your own HTTPS server and your own server and client certificates. One way that you can do this is by using the Apache mod_ssl module.

Before starting out with using HTTP or HTTPS on Android, make sure that your application has permission to access the network by adding the following to your AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET"/>

Challenge #1: Using a Client Certificate

Our client certificate was issued in the PKCS 12 format, as a .p12 file.

To give our application access to the certificate, we used the DDMS utility to copy the certificate file to the root directory of the phone’s sdcard. If you are using an emulator, you can do the same thing to copy the certificate to the emulator’s sdcard.

You can also choose to mount the phone as a disk drive when you plug in the USB cable, and copy the file that way.

In our final application, we “imported” the certificate by copying it to application-specific storage, and then deleting the original from the sdcard root directory.

The secret to using our client certificate is setting up a custom KeyStore containing the certificate, and then using it to create a custom SSLContext.

Once we have a reference to a File containing the client certificate and the password for the certificate, we load it into an appropriate KeyStore (see the sample SSLContextFactory.java):

keyStore = KeyStore.getInstance("PKCS12");
fis = new FileInputStream(certificateFile);
keyStore.load(fis, clientCertPassword.toCharArray());

Now that we have the KeyStore containing the client certificate, we can use it to build an SSLContext:

KeyManagerFactory kmf = KeyManagerFactory.getInstance("X509");
kmf.init(keyStore, clientCertPassword.toCharArray());
KeyManager[] keyManagers = kmf.getKeyManagers();

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, null, null);

The SSLContext can then be used with an HTTPUrlConnection to connect to the server:

String result = null;

HttpURLConnection urlConnection = null;
try {
    URL requestedUrl = new URL(url);
    urlConnection = (HttpURLConnection) requestedUrl.openConnection();
    if(urlConnection instanceof HttpsURLConnection) {
        ((HttpsURLConnection)urlConnection).setSSLSocketFactory(sslContext.getSocketFactory());
    }
    urlConnection.setRequestMethod("GET");
    urlConnection.setConnectTimeout(1500);
    urlConnection.setReadTimeout(1500);

    lastResponseCode = urlConnection.getResponseCode();
    result = IOUtil.readFully(urlConnection.getInputStream());
    lastContentType = urlConnection.getContentType();

} catch(Exception ex) {
    result = ex.toString();
} finally {
    if(urlConnection != null) {
        urlConnection.disconnect();
    }
}

Challenge #2: Trusting a Self-Signed Server Certificate

We now have Android client code that can connect to an HTTPS server and present a client certificate.

However, this only works if the server’s certificate is trusted. In practice, this means that the server certificate must be signed by one of the major certificate authorities, such as VeriSign, Thawte, Geotrust, Comodo, etc. CA certificates for these well-known providers are pre-loaded into the phone’s default trust store.

In our case, the certificate was self-signed. This means that the default TrustManager in our SSLContext will not trust the server’s certificate, and the SSL connection will fail.

If you search the web for ways to trust self-signed server certificates, you’ll likely see a lot of really bad advice. Most responses to this sort of question involve setting up a custom TrustManager that simply trusts everything. This approach is both pointless and obviously insecure.

What we really want to do is to set up a custom TrustManager that trusts our self-signed certificate, and provide that TrustManager to our custom SSLContext. To do this, we need a copy of the server’s certificate chain, which will have to include at least the self-signed CA Certificate and the Intermediate Certificate that is signed by the CA. If you simply export the server’s certificate, you will actually get a file with three certificates - the CA certificate, the Intermediate Certificate, and the server’s certificate. This is fine - the extra certificate is not needed, but won’t hurt, either (usually - see Challenge #3 below).

The method for trusting the self-signed server certificate is very similar to the method for providing a client certificate - we load the certificate into a KeyStore, use that KeyStore to produce an array of TrustManagers, and then use those TrustManagers to create the SSLContext.

In our app, we included the server certificate file in the application resources (since it doesn’t change from user to user, and doesn’t change very often), but you could put this in an external file as well.

Here’s the code for loading server certificate chain into a KeyStore that will be used as a “trust store”, based on having the base-64-encoded (.pem) certificate as a String:

byte[] der = loadPemCertificate(new ByteArrayInputStream(certificateString.getBytes()));
ByteArrayInputStream derInputStream = new ByteArrayInputStream(der);
CertificateFactory certificateFactory = CertificateFactory.getInstance("X.509");
X509Certificate cert = (X509Certificate) certificateFactory.generateCertificate(derInputStream);
String alias = cert.getSubjectX500Principal().getName();

KeyStore trustStore = KeyStore.getInstance(KeyStore.getDefaultType());
trustStore.load(null);
trustStore.setCertificateEntry(alias, cert);

Now that we have the “trustStore” KeyStore with the server’s certificate, we use it to initialize the SSLContext. Adding to the code that we had before, production of the SSLContext now becomes:

KeyManagerFactory kmf = KeyManagerFactory.getInstance("X509");
kmf.init(keyStore, clientCertPassword.toCharArray());
KeyManager[] keyManagers = kmf.getKeyManagers();

TrustManagerFactory tmf = TrustManagerFactory.getInstance("X509");
tmf.init(trustStore);
TrustManager[] trustManagers = tmf.getTrustManagers();

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null);

Now, we can securely connect to our server, trust its certificate (but not others), and present our client certificate. If all goes well, you’re done, and can safely stop reading.

Challenge #3: Handling an Improperly Ordered Server Certificate Chain

As it turns out, some Apache mod_ssl installations (and possibly other SSL providers), whether due to a bug or mis-configuration, provide the server certificate chain in the wrong order.

To work properly, the certificates in the server’s certificate chain must start with the “root”, or CA certificate, followed by any intermediate certificates. If the server’s certificate is included, it must come last. Each certificate in the chain (other than the root) must be preceded by the certificate that was used to sign it.

In our case, a mis-configured server was producing a certificate chain that started with the root CA certificate, but which had the server’s certificate next, followed by the intermediate certificate. This caused the SSL handshake to fail.

To work properly, the certificates in the server’s certificate chain must start with the server certificare, followed by any intermediate certificates. If the “root”, or CA certificate, is included, it must come last. Each certificate in the chain (other than the root) must be followed by the certificate that was used to sign it.

In our case, a mis-configured server was producing a certificate chain that started with the server certificate, but which had the root CA certificate next, followed by the intermediate certificate. This caused the SSL handshake to fail.

The most straightforward thing to do in this situation, if you can, is to correct the server’s certificate chain. If you can’t, then the out-of-order certificate chain must be handled by the client.

In our Android app, we handled the out-of-order certificate chain by implementing a custom X509TrustManager that re-orders the chain.

The code for the custom X509TrustManager is too large to include here (see the full source on GitHub), but once we have it implemented, we use it in the creation of the SSLContext by replacing the ‘trustStores’ array with one containing the custom implementation:

KeyManagerFactory kmf = KeyManagerFactory.getInstance("X509");
kmf.init(keyStore, clientCertPassword.toCharArray());
KeyManager[] keyManagers = kmf.getKeyManagers();

TrustManager[] trustManagers = {new CustomTrustManager(trustStore)};

SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(keyManagers, trustManagers, null);

By loading our own certificates into our keystore and trust store, and re-ordering the server’s certificate chain, we were able to provide an SSLContext that let us connect to an SSL server with client a certificate and a badly-ordered, self-signed server certificate chain.

Wednesday, January 16, 2013

List of technologies we're watching in 2013

From cloud computing to Javascript, from micro services to repeatable environments, from Android and HTML 5 to Big Data and concurrent programming, Chariot's engineers are keeping watch.

The Internet of Things

The “Internet of Things” is all around us.. or, well, we’d like it to be. From “Just in Time” delivery of bus or train schedules as you leave your house, to sensors tracking air quality in the city, from phones to smart clothing, the Internet of Things is about integrating ourselves with this data available all around us. Free of ‘apps’ and ‘services,’ the Internet of Things will make data available ‘just in time’ wherever and whenever you need it. Whether you get the data from WiFi, RFID or QR codes it’s out there, and we think that’s exciting!

Cloud Host Almost Everything!

If you're a savvy IT person, it's may have been a while since you provisioned your own Linux host on a Virtual Private Server or hosting plan. Cloud hosting isn't emerging, it's emerged and taking over. You're likely to use a cloud-based service such as Heroku, Amazon EC2, SliceHost, or even Microsoft's Windows Azure. There are private cloud partitioning services to protect your data from theft in co-location.  Companies such as CloudBees have been shifting so many services to the cloud that now they even provide development and QA platforms that provide your SVN or Git repositories, run Hudson or Jenkins, and provide Continuous Integration services.

Read the rest of this article, subscribe to our TechCast podcast or view all of our Philly Emerging Tech 2012 sessions at our emerging technology resources site, emergingtech.chariotsolutions.com.

Thursday, January 10, 2013

Science of Big Data - sessions coming online

If you didn't attend last month's "Science of Big Data" event in downtown Philadelphia, not to worry. We're beginning to post screencasts of the sessions now:  The Science of Big Data - Screencast Sessions.

Currently we have two - Basho's Tom Santero discusses "Eventual Consistency" and outlines the challenge of keeping data highly available and consistent enough without impacting performance.

The second screencast is the combination of Chariot's Andrew Oswald and Comcast's Josh Angotti's talks. Andy reviews the work that our team did for Comcast using Hadoop in analyzing set-top box users' click selections, and reducing the timeframe of finding answers from days to hours or minutes. Josh Angotti tells the story from Comcast's perspective.

More screencasts are expected soon - we had some video syncing difficulties and audio challenges so it'll take a bit to get them all online. Check the page soon for talks by Cliff Moon of Boundary, Stuart Sierra of Relevance, and Mark J. Headd, the Philadelphia Chief Data Officer.