Click here to read an original op-ed from the TED speaker who inspired this post and watch the TEDTalk below.
If you were to walk up to a stranger on the street and say the words, "Big Data," you'd get mixed reactions. Some folks would look at you curiously. Others would engage you in a heated debate on privacy concerns and the potential misuse of information. Yet, a third group would extol the virtues of big data and how it could be used to save the world. Regardless of which camp you're in, there's a growing trend of capturing, curating, and making sense of larger and larger sets of data. So, is big data a good thing?
In "The Birth of a Word," MIT researcher, Deb Roy, takes us on an emotive and fascinating journey on how a child acquires language. Watching the Roy family and their infant son through a time-lapsed sequence of videos, we learn about the importance of feedback loops, the imperceptible ways in which human beings help one another learn, the engagement properties of content, and the use of social graphs to understand, anticipate, and potentially shape conversations. Roy's talk begins with his son's journey to acquire language and winds its way to how Roy and his colleagues at MIT are applying what they've learned to mass media. By making a few inferences (and by looking at Roy's biography), we see that Roy is co-founder of Bluefin Labs. His social analytics company works with brands, agencies, and television networks to "measure and mine insights."
The way future generations will ultimately look upon big data depends on how we apply it. Will we use big data for commercial gain? Will we use big data for social good?- Alicia Arnold
The plot twist in Roy's talk took me by surprise and made me question the benefits and drawbacks of big data. While looking at 90,000 hours of video to understand the acquisition of language seemed like a virtuous use of big data, tracking what people are watching on television, what they're saying in social media, and what they're purchasing seems a bit creepy. Well, maybe not that creepy -- after all, people are choosing to put that information out in social media and analysis is often done in aggregate.
What's the verdict on big data?
Roy's ability to uncover the complex feedback loops involved in the acquisition of language was an amazing find. When Roy and his team learned the importance of social structures and scaffolding, they knew they were onto something. Feedback loops are valuable, but what happens when they go awry?
Opponents to big data point to the unpredictable nature of feedback loops. The most widely felt example of feedback loop failure was the financial meltdown, also known as, The Great Recession. In 2000, David X. Li published a paper where he introduced the Gaussian copula function as a way of modeling financial risk. Li analyzed large, disparate sets of data and found a connective tissue -- a way for very large, complex risks to be modeled more easily and more accurately than ever before. This paved the way for traders to sell huge quantities of securities. Thus, growing financial markets to previously unimaginable levels. Financial firms lauded Li's function and used it as the basis for evaluating risk. Unfortunately, there was a flaw in the feedback loop. Li's function failed to anticipate risks outside of its model. It fell apart when the markets behaved in a way that Li's formula didn't expect. This brought about cracks in the financial system that we are still cleaning up.
While opponents to big data have their reasons for doubting the merits of using large sets of data, proponents point to all the ways big data is helping to improve society. Some even predict social innovation may be the next frontier of big data.
One of the more widely known benefits of big data comes out of IBM's Smarter Planet initiative. A sampling of solutions shows how data, analytics, and sensors can predict when something will break, so technician's can fix it before it happens. Big data is also helping to analyze and predict crime. In 2005, IBM worked with the Memphis Police Department and the Department of Criminology and Criminal Justice at the University of Memphis to build a system that could predict crime. Called Blue CRUSH, the system charts and analyzes crime patterns. Using the system, the police are able to see crime trends in real-time so they can inform what tactics are used and where resources are deployed. Since implementing Blue CRUSH, there has been a 30 percent drop in serious crime and 15 percent drop in violent crime.
Big data is also being used to help lessen the spread of malaria. Researchers at Harvard School of Public Health are using cell phone data from millions of people in Kenya to understand how human travel patterns relate to the spread of malaria. In a one-year study, researchers mapped every call or text from 14,816,521 Kenyan phone subscribers to 11,920 cell towers in 692 settlements. From this information, they documented the destination and duration of each person's journey. Overlaying the data with a malaria prevalence map, researchers learned malaria is "imported" to Nairobi from residents returning from Lake Victoria and the coast. Knowing this helped officials determine where and how to combat malaria.
At the end of the day, big data is neither good nor bad. The way future generations will ultimately look upon big data depends on how we apply it. Will we use big data for commercial gain? Will we use big data for social good?
Regardless of how we apply big data, we cannot overlook the need for human oversight and intervention. History has shown that models can be flawed. With the growth of big data, human emotion, human intelligence, and human intuition become even more essential.
Ideas are not set in stone. When exposed to thoughtful people, they morph and adapt into their most potent form. TEDWeekends will highlight some of today's most intriguing ideas and allow them to develop in real time through your voice! Tweet #TEDWeekends to share your perspective or email tedweekends@hufﬁngtonpost.com to learn about future weekend's ideas to contribute as a writer.