Huffpost Technology
The Blog

Featuring fresh takes and real-time analysis from HuffPost's signature lineup of contributors

Otto Scharmer Headshot

From Big Data to Deep Data

Posted: Updated:

Over the past ten months I have chaired and co-facilitated MIT's IDEAS China program--a ten month innovation journey for a group of 30 or so senior Chinese business leaders. This year the IDEAS China program enrolled executives of a major state-owned Chinese bank. One goal of this team was to reinvent the future of their organization in the face of big data and other related disruptive changes, which provided me with a little more exposure to that aspect of the world economy. For example, Jack Ma, the visionary founder of Alibaba, says that "In five years, we anticipate that the human era will move from the information technology era to the data technology era."

But what does it mean to be in an era of "data technology" and "big data"? Until today, it has often meant that big companies like Google, Amazon, Facebook, and Apple--the same companies that we used to love and now increasingly begin to question, mistrust or fear--take your data without asking and sell it to other companies without your knowledge (until you notice the targeted Web commercials that appear on your screen). I find it interesting that people's initially very positive view of these American big data empires has been shifting first in Europe, but now also in many other parts of the world, including North America. Edward Snowden made all of us more sensitive to the misuse of big data. But that's just the surface issue. The real problem is on a deeper level.

The real problem of big data is that we are increasingly outsourcing our capacity to sense and think to algorithms programmed into machines. While this seems very convenient and cool at first and offers access to services that many of us want, it also raises a question about who actually owns big data, about the rights of individuals and citizens to own their personal data and to exercise choices regarding its use.

While big data has certainly opened up a whole new range of possibilities, I would like to suggest a distinction between surface big data and deep data. Surface data is just data about others: what others do and say. That is what almost all current big data is composed of.

Deep data is used to make people and communities see themselves. Deep data functions like a mirror: it makes you see yourself--both as an individual and as a community. Over the past twenty years of my professional life I have been helping teams and organizations go through processes of profound innovation and transformative change across sectors and cultures. The one thing that I have learned from all these projects is that the key to transformative change is to make the system see itself. That's why deep data matters. It matters to the future of our institutions, our societies, and our planet.

But what happens today with big data often is the opposite: big data is used to manipulate our behavior, to bombard us with commercials that we never asked for. Surface big data is used to outsource human thinking to algorithms, to reduce our level of awareness inside the boundaries of habitual thought. Deep data, if developed and cultivated in the right way, could help us to enhance the level of awareness and consciousness and to change the system by shifting the consciousness of stakeholders in that system from ego-system awareness (awareness of my own silo) to eco-system awareness (awareness of the whole).

Let me summarize the distinction between surface big data and deep with two simple drawings:

2014-07-18-IMG_6035.jpg
Big data (science 1.0): data that informs about the world (source: A. Oechsner).

2014-07-18-IMG_6033.jpg
Deep data (science 2.0): data that helps us to see ourselves (source: A. Oechsner).

The journey from science 1.0 to 2.0 is a journey of bending the beam of scientific observation back onto the observing self--both individually and collectively.

At the end of our closing meeting last week, the senior leaders of the Chinese state-owned bank reflected on their own journey of the past ten months. Every one of them reported a profound shift in how they think and operate. Here are two exemplary statements:

"This journey is not just about tools and knowledge; it shifts your way of thinking and it allows you in the face of challenges to jump out of the box of old thinking. It feels like my self has been shifting. I also felt that shift among my colleagues. We get to consensus more easily. I feel there is a shift of intention among my colleagues. As a result, we are more in touch with our experience and we are able to execute better."

"To me, the IDEAS journey is a journey of the heart. It opened a new way of thinking, a new way of relating, and a new way of being."

In essence, what the IDEAS participants described was a transformation of
• thinking: from downloading old patterns to thinking creatively
• conversing: from debate to generative dialogue
• collaborating: from ego-centric/reactive to more eco-centric and co-creative

Over the past months, while staying in their jobs, the participants split into four teams that each tried to prototype some new way of operating in order to explore future opportunities. What struck me was that each team ended up developing a new platform of cross-organizational collaboration that used data as a tool for transforming the way their stakeholders communicate. All of their prototypes are still in an early stage. But one lesson that was mentioned by the teams repeatedly was the importance of shifting their mindset from me to we, from ego to eco.

The question that their efforts have left me with is this: On a societal level, what types of deep data infrastructures might facilitate this 'bending of the beam of observation' back onto the observer on the level of entire eco-systems?

For example, today we use GDP to measure economic progress. GDP is an excellent measure of surface data. But what would the equivalent deep data tool be for measuring real economic progress in a community? I believe that it would include a new indicator system that is grounded in real outcomes (e.g., life expectancy), and in the wellbeing of individuals and their communities (e.g., quality of life). Last year we--the Presencing Institute, with the GIZ Global Leadership Academy (German Ministry for Development Cooperation) and the Gross National Happiness Centre in Bhutan-- launched the Global Wellbeing Lab, The lab links leaders from government, business, and civil society around the world who are pioneering new indicators and deep data tools that help communities and eco-systems to see themselves, in order to sense, and prototype new ways of operating.

Where are you seeing the seeds of such new indicator systems or deep data tools today? What can be learned from these first examples? What would deep data mean for your self? What are the real sources of well-being and happiness in your own life and work and what metrics could help you to see and sense your own developmental path in a more meaningful way? How can we co-pioneer the shift from big data to deep data in business, society and self?

From Our Partners