I'm the co-founder of a technology company called Greenplum. I'm a second-generation Asian American currently living in Noe Valley, a great San Francisco neighborhood with a population of 21,106, a median household income of $77,479, an average high temperature in March of 62 degrees, and an average commute time of 32.7 minutes. I recently ate a cheeseburger, french fries, and a milkshake with my family at Barney's Gourmet Hamburgers on 4138 24th Street and the bill was $52.47 -- I paid by credit card. We went there after I returned from a business trip to San Diego that included flying on Southwest Airlines, Flight #2646, seat 14C, and renting a Chevy Impala from Hertz that featured an iPod dock so I could listen to my "most played" song: "The Bends" by Radiohead.
So what? The point is that I know all this detailed information about myself, my neighbors, and my neighborhood because I live in the dawn of a new era--one that is changing the world as profoundly as the Industrial Revolution did. It's an era in which our ability to gather, store, and analyze unprecedented quantities of information is changing our lives, our economy, and our world.
Call It the "Your Data" Revolution
Every minute of every day, we're generating titanic volumes of numbers, words, pictures, and sounds through security cameras, RFID chips, cell phones, scanners, sensors, emails, Tweets, and Facebook updates. We're storing that ocean of data in server farms the size of airports and on flash drives smaller than my thumb. We're searching, manipulating, and cross-referencing data with ever-more powerful computers and software. That's "Your Data," and it is empowering us to do incredible, almost unimaginable things.
Your Data can lead you home with turn-by-turn directions on Mapquest. It can find you love by sorting through the profiles of 20 million other lonely hearts on eHarmony. It brings you up-to-the-second stock prices, sports scores, and flight delay alerts. It helps doctors fight diseases and engineers design safer cars. It gives environmentalists the power to track the movements of endangered animals and biologists the tools to map the structure of our genes.
Your Data, in short, is transforming everything.
The Revolution Is Already Everywhere
As Wired magazine recently put it, data on the current scale isn't just more -- it's different. It's transforming how we do old things and enabling us to do new things that were never before possible.
Your Data has given birth to entirely new industries. Countless companies make products to store and manipulate data -- hard disks, flash drives, and other devices -- not to mention companies that make database software (including Greenplum). Others are generating fortunes just by using data. Google, a company barely 11 years old, has become one of the biggest enterprises in the world without making anything you can hold in your hand. They're strictly data crunchers, ceaselessly sifting through the Internet's ever-changing ocean of information to connect people with what they want to know. Ebay performs a similar trick, tracking millions of items for sale by people all over the world, and matching them in real time with buyers. Pandora.com breaks down millions of songs into their constituent parts--rhythms, chordal patterns, beats per minute -- and then steers subscribers to new tunes similar to their old favorites.
Whole new categories of jobs are being created, while others wither away. Search engine optimization is a great trade to be in these days; newspaper reporting, not so much. Communities in places such as rural Oregon that used to depend on mining and forestry for jobs are now home to gigantic server farms hosting the world's cloud computing infrastructure. There's so much demand for people who know how to crunch information that the New York Times recently reported that statisticians fresh out of college are earning salaries of $125,000.
Even the way physical products move around the world is changing. Consider the Los Angeles/Long Beach port complex, the biggest in North America. Nearly half of everything Americans import every year -- from Hello Kitty dolls to crude oil -- passes through this port. It's a relentless blizzard of stuff -- and the movement of every item is anticipated, tracked, and ordered by a computerized overseer called the Terminal Operating System. The TOS keeps real-time tabs on every vehicle, machine, and container via a network of RFID scanners, GPS systems, optical character readers, infrared relays, and digital cameras. It knows exactly where every object is, what it weighs, who owns it, where it's from, and where it's supposed to go next. The result: that Hello Kitty doll gets to your local Target a whole lot faster than it used to.
However, a whole lot of information can easily become too much information. Too much data overloads the system, clogs bandwidth, and makes it harder to find the useful needle in the data haystack. IDC estimates that an organization employing 1,000 knowledge workers loses $5.3 million every year in time wasted looking for information. Another $5.7 million in person-hours gets blown reformatting information to move it between applications. Not to mention the cost of keeping that data secure. IDC reckons that corporations will spend $65 billion on security software in 2010. The cost of not keeping data secure is high, too: identity theft affects some 8.4 million Americans every year, incurring billions of dollars in fraudulent charges.
Getting Smart with Data
Here's the most important thing to understand about Your Data: it's only going to get bigger. There's no turning back the tide, no going back to an era when we knew less. You can opt out of the Best Buy mailing list when you register your new plasma TV, but Best Buy still knows you bought a plasma TV. So does your credit card company. And if you miss the payments, next time you apply for a loan the bank will know that, too.
Thanks to Your Data, the government and other huge institutions have more power than ever. But thanks to Your Data, so do you.
Battles are being fought every day by citizens armed with the kind of information that was once available only to governments and corporations. For example, environmentalists can challenge plans for a new shopping mall armed with technical details about how it might impact the area's wildlife, air quality, and small businesses. Information has become democratized.
We can help one another in ways that have never been possible before. Websites such as PatientsLikeMe let people who are suffering from an illness share information about their symptoms and treatments. Other websites let you compare your baby's sleep pattern with those of other kids, or share product reviews to help other people buy the best blender or power drill.
And we can help ourselves live longer and better by collecting data about ourselves. The latest in sports footwear is the Nike Plus, a sneaker with built-in sensors that track how far and fast you run. You can synch that information with iTunes, store it on a log charting your progress, and post the results on Nike's site. Other gadgets and websites let you track your diet, blood glucose level, and even sleep patterns so you can make changes to improve things.
That's right, Your Data isn't just for big institutions. It's something that's already part of our lives -- something we're already using and can use in ways we've only begun to discover. The tricky part is making sure that on balance, we use in the right ways.
For starters, we need to free up data: as much information as is feasible should be distributed as widely as possible. We need corporations, governments, and big organizations to be more transparent and share their data more widely. The Obama administration has taken a step in this direction with the appointment of the nation's first-ever chief information officer. Part of his mandate is to make huge amounts of Washington's trove of statistics, studies, and reports available through a new website, data.gov. Sites such as Public.resource.org, which posts court records for free, are helping, too.
At the same time, though, we also need rules to control certain kinds of information. We need laws to keep sensitive stuff such as credit records and medical histories from getting into the wrong hands. But even with such laws, it's inevitable that information will leak out. So we also need rules about how personal information that might come into the hands of employers, police, lenders, and others can be used. Groups such as the Electronic Privacy Information Center and the ACLU are already fighting these fights.
We also need to make sure we can use all the information we're collecting. That means better schools that will turn out kids who are able to cope with the age of Your Data. And we need better, cheaper technologies to enable companies of all sizes, as well as organizations and individuals, to get all the information they want and do something useful with it.
Knowledge is power, and we know more than any previous generation could even conceive. We're moving into a world of infinite information. The challenge we face is turning all that information into insights, conclusions, and revelations -- in other words, turning that knowledge into wisdom, without letting it be turned against us. We need to make sure Your Data doesn't oppress us, but serves us. And we need to do that fast, because the revolution is well underway. Your Data Rules the World.