We live in an era of Big Data and predictive analytics, and Nate Silver's not the only one who's noticed this. I recently sat down with Dr. Eric Siegel to talk about new book, Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die. The founder of Predictive Analytics World, Siegel is a former Columbia professor, but has been in the commercial world for a dozen years since. As I write this, I'm about halfway finished with his excellent book and it's rife with a wide variety of examples.
Nate Silver is so famous for predicting the presidential election -- did he use predictive analytics?
No -- but Obama did. Nate Silver made election forecasts for each overall state: which way would a state trend, as a whole? In the meantime, the Obama campaign was using predictive analytics to make per-voter predictions. True power comes in influencing the future rather than speculating on it: Nate Silver publicly competed to win election forecasting, while Obama's analytics team quietly competed to win the election itself.
So what is predictive analytics? How do you define it?
The shortest definition is my book's subtitle, the power to predict who will click, buy, lie, or die. Predictive analytics is the technology that learns from data to make predictions about what each individual will do -- from thriving and donating to stealing and crashing your car. By doing so, organizations boost the success of marketing, auditing, law-enforcing, medically treating, educating, and even running a political campaign for president.
What are the most important things predictive analytics has accomplished?
Prediction is the key to driving improved decisions, guiding millions of per-person actions. For health care, this saves lives. For law enforcement, it fights crime. For business, it decreases risk, lowers cost, improves customer service, and decreases junk mail and spam. It was a contributing factor to the reelection of the U.S. president. One of the most inspiration accomplishments of predictive analytics is IBM's Watson, which was able to compete against the all-time human champions on the TV quiz show Jeopardy! The questions can be about most any topic, are intended for humans to answer, and can be complex grammatically. It turns out that predictive modeling is the way in which Watson succeeds in narrowing down the answer to each question: It predicts, "Is this candidate answer the correct answer to this question?" It knocks off one correct answer after another -- incredible.
OK, I'll bite. Why does early retirement decrease life expectancy and why do vegetarians miss fewer flights?
These are two more colorful examples of the multitudes of predictive discoveries awaiting within data. University of Zurich discovered that, for a certain working category of males in Austria, each additional year of early retirement decreases life expectancy by 1.8 months. They conjecture that this could be due to unhealthy habits such as smoking and drinking following retirement. One airline discovered that customers who preorder a vegetarian meal are more likely to make their flight, with the interpretation that knowledge of a personalized or specific meal awaiting the customer provides an incentive, or establishes a sense of commitment. Predictive analytics seeks out such predictive connections and then works to see how they may combine together for more precise prediction.
How does predictive analytics work?
By leveraging all the data they collect today, organizations attain a position of power: They learn from the data how to predict human behavior. For example, a company with hundreds of thousands of customer records, some of which show cancellations, can learn from the experience encoded in this data -- a kind of pattern-detection -- to discover which combinations of factors about a customer makes the individual much more likely than average to cancel. These factors aren't always obvious or entirely intuitive; they are signals that reveal odds.
Does predictive analytics exacerbate privacy concerns and issues?
With the advent of predictive analytics, organizations gain power by predicting potent yet -- in some cases -- sensitive insights about individuals. The fact is, predictive technology reveals a future often considered private. These predictions are derived from existing data, almost as if creating new information out of thin air. Examples include Hewlett-Packard inferring an employee's intent to resign, retailer Target deducing a customer's pregnancy, and law enforcement in Oregon and Pennsylvania foretelling a convict's future repeat offense.