Co-written by Finn Birger Lie
Big Data is perceived as the next value opportunity for corporations to innovate and grow. To be of any real value, however, Big Data has to become more accessible and understandable to non-specialist users and leave the domain of the data specialists.
The advent of Internet shopping and social networks has provided organizations with the opportunity to collect huge amounts of social and behavioral information linked to consumers' preferences and purchasing decisions with every online click. In fact, one of the biggest challenges for organizations is to mine these terabytes of data to find interesting patterns that can be turned into useful offerings and profit.
Companies like Amazon, LinkedIn and Facebook have long used such data to build personalized offerings, discounts and direct marketing to enhance their customers' experiences.
Data science practitioners are using technologies that can store, analyze and visualize such information in order to extract interesting patterns that can be turned into profit for the organization. However, their work cannot be automated and like engineering, design and design research, this work is part art and part science, requiring tremendous creativity.
Most organizations do not have terra or petabytes of useful data available to them. However, the most important aspect of Big Data is not so much the volume of a dataset, but more in the insights derived from combining several, dispersed, smaller datasets. More data means more reliable insights but it does not automatically guarantee more insights.
To help create better insight from BIG DATA we have small data that is frequently used by Design Science Researchers to quickly establish non-obvious connections, which can lead to potential breakthrough innovation. Some define small data by the nature of its information, others, by the data's physical manifestation.
Applying lower-fidelity and iterative experiments these findings are "good enough" for supporting or rejecting simple assumptions and detecting simple causalities and a good starting point for incremental exploration and thorough verification.
Until recently, Design Quality Criteria (DQC) have relied on more insightful and higher level, albeit less accurate, small data sets to construct Minimum Viable Products due to its accessibility and speed of information processing. With the advancement of Big Data technology, DQC can now be constructed on more reliable enriched data, enabling a more differentiated picture of the original design science based insights with dynamic feedback practically obtained in "real time."
Combining the Big Data from data scientists along with design science, small data makes efficient use of the strength of the small data viewpoint to define, identify, package and then extract superior Big Data for insight, which can quickly form reliable (DQC) Design Quality Criteria for actionable development. Through platforms and interfaces that can easily distribute actionable small data, Big Data can connect people with timely and meaningful insights designed for easy access, understanding and for actionable value creation in the real world.
This is the way forward for Big Data, co-opted into management functions as IT architecture capable of extracting the desired insight/information to the point of decision making. The incorporation of analytical, visual and data sharing tools, especially those designed to support management processes like business and project planning, product/services and process design, will result in the incremental and breakthrough innovations will finally unlock the real value of Big Data.
Special thanks to Finn Birger Lie for researching and co-writing this article.