Friday, July 1, 2011

Learnings from Actor Development

I spent a fair amount of time developing actor-based systems recently, specifically with the Scala Actor library. Regardless of whether you are implementing actors with the Scala library, Akka, Lift or Scalaz, some basic gotchas can present themselves until you get a feel for what you're doing. Here are some of them that I've learned the hard way.

Never Refer Directly to Other Actors
Actors are fragile and can die easily. While you typically create a supervisor with a strategy for how to recreate that actor, any other class with a direct reference to that actor that died now has an invalid reference. If you absolutely must have actors with references to others that do not have a supervisory relationship, use a proxy reference instead - if the actor behind the proxy dies, you only have to replace it in the proxy, not in every actor with that reference. Akka solves this problem nicely with ActorRef, where the reference behind it can be recreated without updating anyone holding the reference.

If You Do Have an Actor's Reference, Avoid Synchronous Method Calls Between Actors
Regardless of whether your actors are event- (shared thread pool) or thread-based (each actor has its own dedicated thread), avoid having actors make direct method calls on another actor. It introduces concurrency into classes that are designed to avoid that very situation - the receiving actor can be operating on a thread handling a mailbox message at the same time it is dealing with your call. Use blocking or future-based message sends instead, which allows the receiving actor to handle the request through its mailbox on its own thread. Not to harp on the virtues of Akka too much, but the ActorRef type also prevents this kind of behavior.

Write Business Logic in External Idempotent Functions
Testing actors is difficult, particularly those with side effects. If you are in a supervisor hierarchy, the receipt of a message may lead to the creation of child actors that have their own side effects which may be difficult to account for in a test environment. The goal of unit tests is not to test whether actor interaction works, but that the business logic that the actor performs is sound. Externalize your business logic into functions and partial functions that can be tested outside of actors, and use integration tests to prove only that the actors executing that logic behave as expected as part of an end-to-end functional test.

Beware the Thundering Herd
When you start creating structures of actors such as supervisor hierarchies, it can seem simplest to send generic messages that are passed through the tree. However, as actors react and send their own messages, this can lead to event "storms". This can be addressed using two strategies - 1) use granular messages that target specific events for specific actor instances, and 2) ignore messages of the same type with the same parameter data for a given time period. You can even implement a common trait for all of your actors that gives them the ability to not handle the same message for an externally-configurable period of time. Be judicious in how you use this, though - tune it for the loads of your system.

Garbage Collection
In the case of a supervisor hierarchy that is responsible for configuring servers in a cluster, you may want to implement garbage collecting actors that ensure that each server is pruned of configuration that it currently has but is no longer relevant. The actors in the supervisor hierarchy will take care of that if one was created to represent that particular configuration item, but if no actor already existed to represent that state, only a garbage collector whose role is specifically to clean up a dirty environment can take care of clearing bad data from the target server.

Always Pass Copies in Immutable Messages
Copy any object instance that will be passed in a message, so as to avoid accidentally sharing any state. In almost all cases, you should ensure that your messages themselves are immutable. Dean Wampler and Alex Payne make this point specifically in their book, Programming Scala. This, combined with very granular messages, can seem expensive in terms of resources. But it is worth the cost in memory and performance to ensure that your actor behavior is what you expect at design time.

Semantic Logging
Debugging actors isn't easy. Typically, you have multiple instances of the same class with asynchronous behavior, so it is difficult to discern flow. Create trace level log output for each actor type that displays specific information about it in a clearly-visible manner. Use line breaks and tabbed indentation to make it readable, but note that doing so can make your log files even larger than they already are. This has an unfortunate side effect of forcing you to be very granular in your log configuration as to what logging level is used - package-level logging may be too much information. It may help to put a timestamp into a message, so you can grep the log for specific messages as they flow through actors. Also, log the timestamp of when the actor received and handled it.

Deteriorating Retry
If your actors have side effects where a required resource (network connection, database access) may not be available or may fail, use deteriorating retry logic to allow the actor to send itself a message to try again in an increasingly longer interval. For a good example of this, go to Gmail, disconnect from all networks and watch as it tries to reconnect in longer and longer timeframes.

Instrument via JMX for Runtime Clarity
Register every actor instance with the JVM's MBeanServer, and have their supervisors clean up the instrumentation when they die. Yes, this comes at a performance cost, but you can make the registration asynchronous through a future while you perform other tasks in initialization and startup. While you'll still need to profile the threads involved to find threading issues, having the ability to view actor existence and state in JConsole or VisualVM is a wonderful help in knowing what is happening in your system in production.

Prepare for Race Conditions
As with any asynchronous programming, the timing of actor interactions can be unpredictable. Make your actor interactions recheck state they depend on so that they can reflect an appropriate state of their own. If Actor A needs Actor B to have a specific value for its own state to be appropriate, it should not send only one message to Actor B and assume that the value returned is correct that one time. Keep checking the value (again, possibly with deteriorating retry) until you can be certain you have a correct representation of Actor B's state.

Thursday, June 30, 2011

Associative Arrays in Bash 4

 John Shepard, one of Chariot's architects and newest bloggers, just recently posted this on his blog.
 A little different from our more focused software dev posts, we hope you will find this information useful.


I recently switched my work machine from OS X to Fedora. One of the first programs I noticed was missing was Time Machine. After being unable to get any of the backup software for Linux to behave as desired, a project emerged.

Ignoring most of the details of the backup scripts (all the scripts are available at the end of this post), I'm going to focus on the part working with associative arrays.

The backup program would need some way to keep only one backup per day for previous days. The obvious way to determine which days are affected is to keep a key value pair of dates and backup counts per day. Key value pairs have a few different names such as maps or dictionaries. In bash key value pairs are called associative arrays. 


To declare an associative array use -A
declare -A MY_VARIABLE

To access the value just reference the variable as an array element
KEY="some value"
MY_VARIABLE["${KEY}"]

To access the value, or use zero if there is no value, use a default value
${MY_VARIABLE["${KEY}"]:-0}

And to set a value (here the value for the key is being set to two)
MY_VARIABLE["${KEY}"]=2

For looping, to access all keys
${!MY_VARIABLE[@]}

The entire set of backup scripts is available on github. cleanup_day.sh contains the associative array code.

Tuesday, June 21, 2011

HTML 5 Offline Applications with JQuery Mobile

I can't believe it is a month since our last post.  Vacation got in the way, but our consultants have been blogging on various topics.  There is some catching up to do over the next week.

New to the blogsphere is John Shepard.  John recently posted around JQuery ad HTML5, and what he has been doing with his spare time.


I was recently playing around with JQuery Mobile and was wondering how easy it would be to create an HTML 5 offline application. I'm not very familiar with HTML or JavaScript, so I figured this would be an interesting project.

A simple application was required, so a calculator was semi-randomly selected. (The source for the calculator is available at the end of this post.)

The first step in creating the calculator was to add some HTML that had a text area and some buttons. jQuery Mobile handles a resizing display area and can put buttons in a grid. The hooks into jQuery Mobile seem to be through data-roles and classes. Putting the textarea ....read more

Friday, May 20, 2011

Debugging PhoneGap and JavaScript

This is a "reprint" of a guest blog post on the PhoneGap blog, by our resident PhoneGap expert, Hiedi Utley.  Hiedi tackles debugging JavaScript on PhoneGap.

If you have ever written any JavaScript and tried to get it working on a mobile phone then spent hours banging your head against the wall because it just doesn’t work, this post is just for you. Today, we will be looking at ways to troubleshoot, diagnose, and debug your mobile web project. While I will be focusing on debugging interactions with PhoneGap, the strategies outlined here are really applicable to any JavaScript or mobile web project. Since I have been focusing mainly on iOS development, this post will discuss the debugging of JavaScript and PhoneGap within an iPhone application but the same ideas should work for Android and other platforms as well.

Read more..

Monday, May 16, 2011

Spock It Like You Mean It!

This post was just put up on the blog of one of our architects, Anatoly Polinsky.  Anatoly has a playful take on Spock, a Groovy testing framework, and specifically, abotu connecting Spock with Spring and DbUnit.


So here we go.. Yesterday night we hacked our way into The Ancient Database where besides the data about ancients themselves, we found incredible stats about many different living species of all the planets ancients traveled to.
So what do we do now? Well, we need a way to query/read this data. So we asked Sam to develop a reader ( ancients called it a ‘DAO’ ) to find a number of living species on each planet since the “beginning of time”.
So she did, and called this “species reader” a ‘SpeciesDao’.
We, of course, trust Sam. But in ancient technology, trust is always based on challenging the assumption by testing all the possible permutations. So we are calling Mr. Spock to help us out…
Sam of course used a well known ancient technique ( code name: Spring Framework ) to create a “species reader”, so Mr. Spock will use all the goodies of this technique to inject all the dependencies ( e.g. data source, a reader itself, etc.. ) for a test.

Read more...



Sunday, May 1, 2011

Digital Philadelphia: A vision of Philadelphia as a regional technology center


Here's how to make it happen, says former CTO Allan Frank

By Todd R. Weiss

PHILADELPHIA – (Editor's note: Allan R. Frank, the former CTO of the City of Philadelphia, was scheduled to speak here last Thursday at the 6th annual Emerging Technologies for the Enterprise Conference (ETE). A sick child at home, however, canceled his plans for the day. Frank was kind enough, though, to talk with me by telephone to describe his vision for technology in Philadelphia – which was to have been the subject of his talk. Here are his thoughts.)

From July 2008 to this past February, Allan R. Frank worked as the City of Philadelphia's first-ever Chief Technology Officer (CTO). Yes, the city has maintained a separate Chief Information Officer post for a long time, but the CIO is in charge of the city's IT systems and staff and running IT for its government agencies and processes.
Instead, Frank was brought in to help find ways to use technology to make the city a better place to live, work and do business. His vision, he said, was to find ways of using technology in the city to bring business people, educators, residents and other movers and shakers together in the common good.
Frank took the CTO post after talking with Mayor Michael Nutter and convincing Nutter that someone was needed to help connect the area's technological capabilities with the issues that were central to the mayor and the city. To Frank, that meant showing Nutter how a technology-focused CTO could help the city solve its most pressing problems, from economic development to solving poverty to improving education and more.
Frank said he explained to Nutter how technology could have an impact on a wide range of issues, but that someone was needed to lead the effort.
"I started explaining to the mayor that I saw a tremendous opportunity for the city to take its place as a cornerstone of a regional technology center," he said. By bringing Frank in as CTO, he told Nutter that the mayor would finally have someone "sitting at the table to focus on all of his goals."
"The mayor obviously shared my vision around the opportunity for how technology plays a role in everything in a city," Frank said. "He saw the importance of that outside role."
Frank's vision was to continue what Philadelphia has always been – a city of firsts. Since the 1600s, the first library was set up here, as well as the first hospital, bank, library, medical school and computer – ENIAC.
With that history in mind, Frank said he envisions the city as the "ground zero" for the next wave of technological innovation in the U.S.
"My vision of what I call 'Digital Philadelphia' is asking the question, 'what should it be and what should it look like? This city must be the national model for urban transformation to the 21st century knowledge economy."
The tools are here in the region to make it happen, including such things as an urban core that has been again attracting young people to move downtown and work in the city, he said.
"We're also the home of Comcast," he said. "We could be the next Silicon Valley."
Much of Frank's ideas for the city were born, he said, after President Obama took office, nurtured by Obama's ideas of creating broadband stimulus efforts to help cities expand their technological infrastructures and abilities.
"In the 1960s, Philadelphia was manufacturing textiles," Frank said. "Today we should be talking about putting in fiber networks and technology parks."
There are 154,660 people employed in IT occupations in the Philadelphia region, employing 7.4% of the area's total workforce, he said. That gives the area the 6th
largest concentration of IT workers in the U.S.
Adding to the region's allure as a burgeoning tech center are the world-class universities here that train qualified technical people, he said.
"What about developing tech economic development zones around the city, buildings that are pre-wired for 1-gigabit networks and many more innovations?" he said. "I was trying to frame this vision of Philadelphia as this zone of network incubators, and setting up better ways to work with local tech community."
All of these things have been part of the regional discussion after he joined the city in 2008 and they will continue to be pursued even after he departed from his post to head back into private industry, he said.
"Government can't make all of this happen," Frank said. "It takes the community at large. And it takes the government to be supportive. Digital Philadelphia needs to have executives, citizens and tech companies involved."
Now all of those ideas will be the responsibilities of others. Frank, who has more than 35 years of experience in technology and business leadership, left his CTO role in February to return full-time to The AKA Group LLC, a strategic consulting firm that he founded.
Frank remains, however, on a steering committee to help continue these efforts and he believes in the city and the area, he said. His passions and energy will continue to help push for these changes, he said.
"Over the last couple of years, you can feel the energy here," he said. "Our kids should be staying here in Philadelphia to create for technology companies" so we need to help create a vibrant tech community here to employ thousands of people. "The future of the world is cities."
The power to make it all happen here is right in the region's hands, according to Frank. "What we really have to do is create a freaking wave."

Todd R. Weiss is a longtime technology journalist who worked as a staff writer for Computerworld.com from 2000 to 2008. Now a freelance tech journalist, Weiss contributes regularly to Computerworld, PCWorld.com and other publications. He has also written extensively for Linux.com, ForecastingClouds.com and TechTarget on a wide range of enterprise IT topics from Linux and open source to disaster recovery, cloud computing, virtualization, application development, IT education and mobile and wireless technologies. He began writing about computers in 1996 after a newspaper editor he worked for told him that "no one cares about technology." Apparently, the editor was wrong. Follow him on Twitter.com @TechManTalking or contact him at toddrweiss at gmail dot com.

Thursday, April 28, 2011

Meet your future bosses: Meet the girls of TechGirlz.org


By Todd R. Weiss

PHILADELPHIA – Yes, the halls here at the 6th annual Emerging Technologies for the Enterprise Conference (ETE) are filled with software developers, business people and technology visionaries.
But if you looked a little deeper, you might have met your next boss – and she could still be in elementary school.
In a conference room here this morning, nine girls – ages eight to 15 – met with two visionary women in technology who delivered the keynotes for the event – Stormy Peters of Mozilla and Molly Holzschlag of Opera Software – to get advice, encouragement and
support in pursuing their youthful interests in computers.
The girls were invited to visit by TechGirlz.org, a Philadelphia-area non-profit group begun two years ago to encourage young girls to explore their love of technology even when they think it's too geeky to pursue.
So who are the girls of TechGirlz?
They are young women like Eliana, 12, who one day decided she wanted to create a video game. First she began creating a site for the game using HTML, then she realized she needed to use a programming language to build it correctly. She chose Python after attending several technology camps and talking with others to gaining their expertise.
"I started by teaching myself HTML then I started learning Python," said the bright-eyed girl with braces, hot pink pants and long, curly dark hair. "What got me involved was that ever since I was really little, I was always attracted to video games and computers."
Since then, she's been working hard to further develop the game and continue to add features. She's even exploring the idea of releasing it as an open source project, but she said she's not sure she wants to give up control of the project.
Yes, I did say she's 12.
And those code camps that Eliana's been going to? They're with students who are in high school and college, yet there she is coding with the best of them. "I'm still kind of more like an amateur," she said.
Yeah, right.
I know of companies out there who are desperate today to find and hire good Python coders and here is Eliana, a 12-year-old who is building her own video game from scratch.
Classic.
Then there's Olivia, 12, in a pink shirt and ponytail, who loves to do art and design work on her computer, "making it go from clunky to smooth and shiny."
There was Megan, 13, who said she'd like to find ways of helping to add emotional context to online chat and other electronic communications. The problem, Megan said, is that it's hard to know what people are feeling when they are communicating using chat – are they mad at you or are they happy with you, she wondered? That's a great problem to solve.
Ten-year-old Kayla had another concern – she created a Web site for her business - making healthy homemade doggie treats (www.kaninekookies.webs.com) – and sometimes the technology overwhelms her, she said.
Haley, 8, is on her school's robotics team and likes to use computers.
Meanwhile, Logan, 15, also likes robotics and learning about how to make the devices operate. "It can be frustrating figuring out how to make them do something," she said.

So what advice did Peters and Holzschlag have for the girls?
First, Peters said, if the girls feel overwhelmed by the technology, she reminded them that it's not due to any shortcomings of their own.
"It's not you," Peters said. "It's the way that the people who are telling you about it are talking about it. I always think it's the fault of the person who's trying to tell you about it [if they can't explain it better]."
Asking lots of questions is a great way to learn, Peters added, lauding Olivia for asking what CSS Zen Garden is all about in the field of Web design.
"If you're brave enough to ask what CSS is at a technology conference, then I guarantee that 80% of the other people in the room are also wondering what CSS is," Peters said with a laugh.
Whatever they choose to do as they continue their educations, Peters encouraged the girls to follow what they are interested in and not give up on their dreams of pursuing a technology career.
"What frustrates me is that women are missing opportunities in a fantastic field," she said. "I started in technology because I really liked what computers could do. Now we have computers that can help blind people read, that help people find a movie, that enable people to get jobs and more. I just found computers fascinating and being able to make … them do what I wanted to do was another part of that."

Holzschlag got involved with computers by accident, she said. She was studying Spanish and linguistics at the University of Arizona when she suddenly had to drop out of school due to a long-term illness. For 10 years she was home-bound.
One day, a cousin send her an old Commodore 64 computer and a friend sent her a 300
baud modem. She hooked it all together and went online using an ancient dial-up service back in 1988.
Suddenly she went from home-bound and being isolated to joining a global community, years before the Internet ever appeared, she said.
"It was in that moment that I logged on and saw people from around the world," Holzschlag said. "I didn't know what I was looking at, but I knew that was 'it.'"
She's been involved in technology ever since.
"My life changed … because of that Commodore 64 and a 300 baud modem that somehow came into my life,"she said.

Those are the kinds of messages that TechGirlz.org is working to bring to young girls in the area, said Tracey Welson-Rossman, director of marketing and sales at Chariot Solutions, the Fort Washington, Pa.-based software development consulting company that hosts the annual ETE event.
Welson-Rossman, the founder of TechGirlz.org, said the group's work started last September when groups of 15-20 girls would come to participate in lectures, discussions and demonstrations of technology so they could gain exposure and increase their curiosity about high-tech topics.
The idea is to expose tech-minded girls at a young age to women who are working in technology so they can be inspired and empowered, Welson-Rossman said.
"We've seen research that says that the reason that girls drop out of pursuing technology interests starting in ninth grade in high school is because they think it's geeky," she said. "They think it's not creative. They think you're working in a cubicle and not talking to anybody."
To battle those stereotypes, TechGirlz strives to get to them early and show them how exciting, gratifying and worthwhile technology jobs can be. "One of the other issues is that they think it's not cool for girls to be in technology. And because of that they feel that they can't actually express that they actually like technology."
"So one of our goals is to create a safe place for them to be able to express this part of their personalities, of their interests," through TechGirlz.
About 100 girls are involved in the program so far. TechGirlz is growing its program by making connections with teachers and schools in the metropolitan Philadelphia area and through communications with home-schooling organizations, Scouting groups and others.
Men in technology are also welcome to volunteer and help with the TechGirlz program, Welson-Rossman said. "We also want men to be involved because we need to show all types of role models. These girls are going to be working with men."

Todd R. Weiss is a longtime technology journalist who worked as a staff writer for Computerworld.com from 2000 to 2008. Now a freelance tech journalist, Weiss contributes regularly to Computerworld, PCWorld.com and other publications. He has also written extensively for Linux.com, ForecastingClouds.com and TechTarget on a wide range of enterprise IT topics from Linux and open source to disaster recovery, cloud computing, virtualization, application development, IT education and mobile and wireless technologies. He began writing about computers in 1996 after a newspaper editor he worked for told him that "no one cares about technology." Apparently, the editor was wrong.