Sign In

Better Data Crunching Can Prepare Us for Black Swan Events

This week, we’re engulfed in the coverage around the unbelievable damage wreaked upon the Northeast by Hurricane Sandy. While tracking the path of Sandy ahead of time helped cities in the Northeast to take some proactive steps to prepare their communities - like evacuating residents, closing down the NYSE, and shutting down public transportation – the “Frankenstorm” had some unforeseen and surprising side effects: The flooding in Manhattan took down several major websites and services, including The Huffington Post, Buzzfeed and Gawker, that relied on Manhattan-based data centers to run their businesses. Some data centers weathered the store with backup generators, but as they run out of fuel, even more businesses are in danger of major disruption.

Gawker publisher Nick Denton summarized it concisely in a twitter post on Tuesday: “How dumb to locate datacenter in a flood zone. And how dumb to host Gawker servers there.”

Why weren’t these businesses better prepared, especially based on the availability of data?

No doubt, the risk was somewhere in the data, but the data apparently failed to not only alert these businesses of the danger, but to also prescribe a course of action. Isn’t this the promise of Big Data?

But that’s exactly the problem: it’s not the Data, Dummy! Data is now as plentiful as dirt. Most organizations are storing the data, processing the data, reporting on the data – but where’s the magic of foresight? How do organizations get the data to start talking to them, tapping them on the shoulder, telling them to pay attention to THIS, not that!

To achieve this level of insight without a team of data scientists hovering over massive super computers requires advanced mathematical algorithms that fuel more nimble software applications. This approach can sift through all the data without breaking it down to process it in order to look for patterns, automatically surfacing the critical connections and muting the unimportant ones (detecting the signal from the noise). The same technology that can tell a police department “Hey, this coffee shop might get robbed on Friday” can inform a business if they need to relocate data centers.

Clearly, this isn’t a crystal ball, wand-waving, pull-a-rabbit-out-of-the-hat-type of magic. It’s more of a butterfly effect – it’s building in the data – and we need smarter ways to detect it to prepare for that hurricane.

Then we won’t be some dumb about our business decisions.

-Shannon Rentner, Director of Marketing