Matters of machine intelligence are topics of great interest these days. In circles of CTOs, it's all anyone talks about. It reminds me of a statement I read a couple years back from renowned technology writer and author, Kevin Kelly -- "the business plans of the next 10,000 startups are easy to forecast: take X and add A.I." An interesting remark.
Although that prediction proved to be a bit off the mark (or perhaps it's still too early for its time), the idea is certainly compelling.
Nearly every day I hear of a new startup announcing some new intelligence offering. It's an exciting time to be in the industry. Finally, intelligence-as-a-service is now a reality. Companies and organizations can buy intelligent systems and forgo the tremendous difficulty of building one from the ground up. For CTOs, the question they must answer now is how to incorporate intelligence platforms into their existing application stack. Simply put, how do they make their company smarter?
It is a difficult question and many businesses are willing to throw money at the wall just to see what sticks. The truth is, there is a right and wrong way to go about it. Unfortunately, most CTOs are only aware of the wrong way. Here's where I believe I can help.
First, let's examine the wrong way. Traditionally, companies and CTOs who want to add intelligence to their applications will take a holistic approach, creating what is known as a data lake. This is done because of the need for context surrounding the data gathered from applications -- the categorical information about particular points of data. The belief is that by capturing and organizing all data, businesses will be given a full 360 degree view of their users' preferences, habits, etc. It is a great payoff -- in theory.
Companies may hire a team of data scientists who set out to build the data lake. Then once that storage is in place and all the data is gathered, an intelligence application can parse it and draw new insights from the company's data.
On a surface level, it seems very effective. Put all the data in one place and then learn from it. Simple enough. Unfortunately, this strategy has a number of shortcomings. The first is that it can take years to implement. Building a data lake from scratch is a slow moving and laborious endeavor. Secondly, a holistic approach to intelligence is inefficient. Hiring a team of data scientists and building a data lake is terribly expensive and diverts valuable resources away from other priorities. The truth is you really don't need everything to do something. It's much simpler!
For a better approach to make your company smarter, look to APIs. By placing an API layer in between your company's applications and an intelligence-as-a-service provider (i.e. IBM Watson), you can create smarter systems that provide top notch analysis without the need of a data lake. APIs enable data to flow into the machine learning system (since all interactions with the application happens through the APIs, the API traffic represents the totality of all signals about an app's usage). But more importantly, APIs represent a very simple way of taking action. Recommendations can be pushed, for example, as part of the API responses. Threats can be blocked in the API layer. APIs help close the loop.
Moreover, this approach is very efficient. APIs working between the applications and intelligence can quickly process and contextualize data, giving you responsive insight and the coveted 360 degree view of users. Not to mention, you can cut costs dramatically by not hiring a team of data scientists and building an expansive data lake.
Data science is hard, but data within an API layer is easy. Adding API layers between applications and intelligence is already being done by dozens of companies, particularly those in retail.
The best advice I can give: don't build it, buy it.
Intelligence-as-a-service is no longer an impossible dream -- it is available today. It can make your company smarter and give you an edge against competitors. Save the time, effort, and investment by opting for an intelligence strategy that does not require any data lakes. Let nature do the lake building -- use APIs and make your company smarter instead.