(Thanks to Vint Cerf, Jacob Ziv and Marc Smith for advice and discussions)
Who doesn't want the survey-less society? It may be the next big thing after the paperless office. Web 2.0 is making it happen. Just like with the paperless office, it will be another paradigm shift for society. It will change the way we make decisions together.
So much is driven by analytics in this era of computing, all marketers live by it. Their thirst for data just grows they more they get. They have become dataholics. Ordinary people are increasingly nagged to answer surveys. It's soon not possible to make an ordinary phone call without opting in or out of a survey before being let through to a human being. But the value of that data is questionable. And, hopefully, these surveys will be redundant soon.
There is accumulating so much data in the IT-clouds, that probably most trends are to be found in what already is recorded. The good thing with this data is that it covers everything, it isn't limited to a narrow window of interest. Everything that people talk about is there.
The strategists who until now are working with surveys are always looking for the answer to one question: "who says what?".
Surveys assume which are the important groups of people and then go check what they say, or the other way around - they assume which are the important ideas and then check who say them. The answers are won't be better than the questions. But asking the right questions is really tricky. You don't know what you don't know, it's a lot of guessing. That will not do in the innovation society, when every important trend is driven by the introduction of things that never existed before.
It's time to go beyond surveys and other analytics that can't spot new trends without first assuming their existence.
Can it be done? Yes, it can, using the data in the clouds and introducing new ways of thought.
The picture with the red-blue scale is an example of the next generation analytics. It finds trends, rates and ranks - it even quantifies bias - without making any assumptions about any groups of people, or what they think or what they do.
The example shows the most discussed blogs, from blue (Democrat) to red (Republican) discussing Obama's State of the Union address. The choice of media and the assignment of political color was done by a general algorithm, which does not care about politics or media.
We simply fed "state of the union" into the algorithm and this is what came out.
We did NOT assume the existence of Democrats or Republicans. We did NOT assume anything about blogs or news outlets, nor select any to look at, nor guess political sympathies. We did not assume which opinions or sentiments mattered.
How do we do it? The understanding of how to do it comes from a concept I have named 'commemities'. The algorithm was done by combining two modern, usually separate, analytics approaches - social network analysis and search/semantics. It is a natural thing to combine them because social network analysis shows how people connect and semantics shows how words connect, while the reality is that people and words connect each other.
1) Words are connected by people. People are connected by words. 2) A community has a shared language. A shared language has a community. 3) The community and the shared language can be seen as one: the "commemity" - the community and its memes.
Now we can explain how we did the list. We make a social network analysis of Twitter, finding two big clusters chatting about the "state of the union". These two clusters are Democrats (red) and Republicans (blue). Now we determine their shared language, and we will have mapped the commemities. One way of doing it is to check the web pages they refer to. There are blue referrals and there are red referrals. We sum up the referrals to the web pages and mix the colors. As expected, Fox is republican language. The democratic congressional campaign committee is democrat language.
This simple notion of commemities enables new analytics tools for strategic analysis, marketing, governance and planning, that can reveal unknown disruptive trends and answer unasked questions.