"In God We Trust. All Others Must Bring Data."
This quote is attributed, either correctly or incorrectly, to W. Edwards Deming, and he laid the groundwork for the application of statistical methods to improve product quality, testing and sales in business. His work is easily applicable to the data warehousing movement, which aims to store customer data and present it in a unified manner for base analytics. Unfortunately the way business has been using data to date is limited. The age of the data warehouse is behind us. It's gone and it's not coming back.
Data warehousing was an important stage for business. It emerged when the proverbial light went on and business had the epiphany that massive amounts of data on their customers could be put to use for driving efficiency and learning from their past efforts. The problem is the basic definition of a data warehouse is all about storage. A data warehouse is intended to serve like a locker where data is cleaned up and placed so businesspeople can run analytics, but analytics without action is pointless and too often businesses did not put that analysis to work because it was simply too difficult to do so. To be honest, they couldn't, which brings us to the most recent stage of business and the evolution of data management, or as I like to refer to it -- Data Activation.
Over the last 18 years, since the explosive growth of the Internet and digital marketing, the opportunities for creating and using data have grown exponentially. In the past you had data being created from a number of sources, but since the early '90s the number of sources have increased tenfold as has the desire for business to not only store data, but activate it and put it to use. Suddenly business was dropped into a world where every single interaction that a consumer can have with a company can be completely tailored to their needs and wants. Every single aspect of that experience, from the website, email and CRM, to direct mail can be tailored to tell a story. This is important because all good marketing tells a story and all good stories can sway their intended audience to at least consider a service or product. Marketers, when successful, are among the best at weaving a story to drive a specific response and they are quickly becoming technologists in order to keep up with the data their intended audience creates.
That data was rarely used because the majority of the organization had no way to access the data and the cost to do so was enormous. Some companies invested in analytics departments to make use of the data, but for the most part the cost outweighed the return and the majority of marketing succumbed to the "spray and pray" approach of buy it cheap and spit it out where you "think" your customers may be. Hence the ongoing love/hate relationship with television and print advertising.
Now we sit squarely at the crossroads of Big Data and Marketing. This is our area of Data Activation. Data Activation is more than data management, and it's significantly more than data warehousing. It's the biggest trend in marketing right now because everyone knows the future rests upon it, but not many people get exactly what "it" is! It involves taking a background in marketing and steeping yourself in a better understanding of consumer behavior, data aggregation and utilization of that data for the purposes of increasing efficiency in your marketing efforts. It also relies heavily on establishing a strategy for the unlocking of that customer data, creating classification schema and rich taxonomies for activating that data, and enabling permission based sharing of that data so it can be put to use without losing control of what remains a proprietary asset. This cannot happen effectively unless you have a full data activation system in place; a Data Management Platform, or DMP, is simply not enough. You need to be able to manage your data, gather more data quickly, and append that data to be used and shared with all of your marketing partners in real-time.
As you come to understand more about the world of data activation and the death of the data warehouse, you need to know about three primary things you should be looking for:
1. Don't be sold on the benefits of solely using data in banner advertising
There are lots of companies trying to gain a foothold in the category of data-driven marketing; however, most of them are focused solely on one execution area: banners. Banners are not the be-all, end-all of digital marketing. In many cases they are not even the centerpiece of your brand's marketing efforts. Banners are one aspect of what you do, and they should be informed by data, but you should also select a partner that will allow you to port that data to multiple places. Websites, direct mail, addressable television and other areas are all going to be informed by data in the coming years and you should plan now for that future.
2. Don't be sold on a partner who focuses on activating on their own execution layer
Another fact of the business is that if your Data Activation partner also owns the execution tools, like a DSP or ad exchange, then they are incentivized to activate your data on their own platforms. You want a partner that is agnostic about where the data is activated and who is not incentivized to run on their own inventory. You want to select an unbiased partner so you can be sure they have your best interests at heart, and not theirs.
3. Don't under-estimate the difficulty in ingesting offline data
Ingesting, categorizing and activating offline data is not easy. It requires a technology set that works, and it requires a classification system which is based on actual partner output requirements, not just an off-the-cuff methodology. When you select a partner, be sure they are able to ingest offline data and ask for examples of where they've done that as well as how it worked when transitioning from ingesting to integration through to activation.
These three pointers are crucial as you shift your business' attention from the stone dead world of data warehousing into the fast track category of data activation. When you select the right kind of system for your purposes, you have to understand the objectives. or what you aim to do. Another quote attributed to Deming is, "What is a system? A system is a network of interdependent components that work together to try to accomplish the aim of the system. A system must have an aim. Without an aim, there is no system."
He was right, and in the age of the "data warehouse" the aim was to "warehouse" the data. In this age of data activation, the aim is to "activate." Are you planning for the right objectives?
Data Cleansing is about improving the quality of your data. In broad terms, some examples could be removing duplicate customer accounts, fixing mis-spelt addresses, adding correct zip codes etc.
There are a number of tools widely available that can do this.If you use Microsoft SQL Server 2012, it now comes with a tool called Data Quality Services that can help you do this too.
I hope that helps!
Best regards,
Ian
He says that business uses it for analytics (true) but can't do anything with the results of that analysis. Utter drivel.
Everyone needs information to make a choice. Whether they act on that choice has no bearing on the information.
Where he is correct is that organisations need to rapidly add more data and analyse that too. Where his level of understanding ends is that he is still thinking of things in the old-fashioned way, that data warehouses must be designed first, modelled, built using ETL tools, and then provided to the business - a slow, outdated and cumbersome process.
There are new ways, there are new tools - products like WhereScape, BIReady, Kalido and even SAP HANA that spring to mind, that short-circuit the difficult process of building a data warehouse.
I can tell you now, data warehousing WILL exist in many forms for many years to come - especially while organisations grapple with the challenges of too much data and not enough information.
In this age of data activation, the aim is to "activate." '
is so very true and obvious, but not clear to interpret.
Pattern recognition tools "read" energy data in buildings for "actionable" anomalies see http://blog.KWIQly.com . The key observation overlooked in this article is that to extract the 'actionable' domain knowledge is needed, else it is simply data-mining .
We are in the age of codification of domain knowledge. Years back 'AI' focussed on Medical Diagnostics as people understood this as diagnosis (literally separating beliefs). Two factors made this tough.
1) Reliable data is hard to come buy when interfacing to people.
2) Biological systems are complex.
The modest task (relatively) of playing chess saw Big blue became world-class soon. Backgammon proved surprisingly more intractable (probabilities and "looping").
Now we recognise photos of faces and interpret voices. But so do babies.
The world of expert systems - is soon to be upon us -value will be in people who have diverse expertise in niche fields. Specialists will again be star in the age of Geek2 - not because Geek1 can make it happen, but when Geek2 explains to Geek1 what needs to happen.
We don't know what Geek2 will need as tools from many possibilities. Do http://en.wikipedia.org/wiki/Hidden_semi-Markov_model (s) solve a problem .
Interesting when automated football coaches know who is better ! - What's it worth ?