By now, anyone who uses data has heard about the three "V"s of big data. Let's not debate whether there are four or five or 20 "V"s. Whether your concern is millions of marketing research panel records, billions of retailer transactional records, or trillions of website tracking records, anyone who uses data has the same problems. The volume of data increases exponentially, the variety of data keeps increasing, and the speed with which data becomes available is, well, let's think of the speed of light. These issues alone make dealing with big data a complicated procedure.
Add to that the fact that big data is rife with missing data. It's incomplete data, and it's complicated data. It needs specialized analytical tools and specialized analysts. But those problems are also not the reason we're failing when it comes to big data.
Why are we failing at big data? Well, let's take a step back to the survey and focus group world that market researchers and insight professionals love so much. When I think back to the last survey I wrote, it too was quite the beast. For just twelve minutes of respondent time, I spent many hours dreaming of, writing, tweaking, rewriting, and retweaking every single question and answer. I pondered every "the," "or," "if," "they," "you," and "why." I argued with myself about the possible ramifications that every single word might have on my results. In every case, I settled on the best solution, not the right solution. In the end, I had a survey that would carefully address every single hypothesis and research objective on my list. This survey was a beauty and the analysis was quick and easy.
Let's move forward to your big data project. You know, the one where someone dumped a giant SQL database with thousands of variables and billions of records on your plate and said, "Make our program better." You weren't really sure what the program was, you didn't know what was currently good or bad about it, and none of the database variables matched up with any project plans or research objectives. Actually, there were no research objectives. Except for "make it better." I can assure that is not a solid research objective.
Imagine if someone collected together a hundred surveys from a hundred projects and told you to "make it better." I can guarantee you would fail at that survey analysis regardless of how many years of survey analysis you had behind you.
The simple reason we continue to fail at big data is that we fail to create concrete and specific research plans and objectives as we do for every other research project. We know very well that a survey project will fail without carefully operationalized objectives. But when we work with big data, we ignore this essential step. We don't plan ahead with specific variables, we don't list out potential hypotheses, we don't have a game plan. "Find something cool" isn't a game plan. Nor is "how can we improve?" Big data needs big brains to plan and organize and be specific.
Do you want to succeed at big data? Then stop treating it like a magical panacea and do the work. Do the hard work.