I recently posted some thoughts regarding Microsoft’s Windows-compatible Hadoop implementation, HDInsight. I was investigating it for a project that I figured would benefit from a distributed processing approach, although ultimately decided to pursue other alternatives. It led our team to make some quite interesting discoveries about Hadoop, and some scenarios of when current distributed processing solutions are and aren’t appropriate.

Hadoop. Everyone and their dog is talking about it. That and “Big Data”. There was an excellent post on Brent Ozar’s DBA Reactions Tumblr blog recently that encapsulated it perfectly, titled “When the executives ask if we’re Hadooping”. It’s a valid point though, Hadoop is mentioned in just about every article these days, along with the phrase “Big Data” (which I personally don’t like at all). The consensus, at least on the surface, seems to be that Hadoop will solve everyone’s problems, process anything, oh and bring world peace while it’s doing that. My sarcastic tone belies a genuine interest in playing about with it though. With so many people talking about Hadoop (in its many implementations), I was very keen to get an opportunity to try it for myself.