Understanding the capabilities and disruptive threats of big data is a must for all sophisticated finance professionals today. This article considers the development and current state of Big Data Finance, as it relates to innovation in investment management, intermediation (broking), insurance, risk management, and most other aspects of Finance.
Big Data Finance is a revolution as well as an evolution that has come prominence several decades. Chartists and other technical analysts since at least the 1920s have utilized market data to derive upcoming patterns of prices. Perhaps the first formal breakthrough in Big Data Finance occurred in the 1980s, when companies like Bloomberg began packaging and delivering market data in large sets to investment professionals. Big Data Finance 1.0 allowed for extensive data mining, yet with still limited mathematical tools. Computer terminals were expensive, clunky and slow. Still, the inferences were valuable and much sought after.
Big Data Finance 2.0 came with the explosion of the Internet in the late 1990s. Suddenly, it became possible to receive streaming real-time or near-real-time financial data. The innovation enabled the growth of financial technologies such as electronic trading, and market making. Efficient online communication facilitated deal-making, globalization of trading, and much more. The efficiencies of Big Data Finance 2.0 streamlined financial services’ processes, and brought down previously prohibitive costs of financial transactions by a factor of 100.
Big Data Finance 3.0 is presently upon us. Big Data Finance 3.0 is really about managing the scale of data and extracting the information within. Today’s big data is about faster, better analytics, an ability to extract that “needle from the haystack” using the latest data science inferences, and storing, managing and integrating ultra-large sets of streaming and historical data of all kinds: market data, social media data, news, regulations, announcements, and so on.
Where can one learn about the latest developments in the field of Big Data Finance? The upcoming Big Data Finance conference in Miami on December 5 (just ahead of Art Basel Miami) is a great place to start. The conference covers Big Data Finance techniques such as clustering, identifying latent data points and sources, and many more machine learning and artificial intelligence topics from the top academics in the field. In addition, the conference presents informal discussion panels by the industry leaders covering successes and challenges in implementation, industry prognoses, and many more. In its 6th year, the Big Data Finance conference has a steady following, with regular attendees including employees of companies like T Rowe Price, Goldman Sachs, Millennium Partners, Canadian Pension Plan, and many more. Big Data Finance is not a fad, but a new way of competing in today’s marketplace, necessary for survival as well as prosperity.
Irene Aldridge is a Visiting Professor of Financial Mathematics and Microstructure at Cornell University and President of AbleMarkets.com, a real-time streaming financial research company. She is the author and co-author of several best-selling books related to data in financial markets, including “Real-Time Risk: What Investors Should Know About Fintech, High-Frequency Trading and Flash Crashes” (2017, Wiley, with Steve Krawciw, translated into Chinese) and “High-Frequency Trading: A Practical Guide to Algorithmic Strategies and Trading Systems” (2013, Wiley, 2nd edition, translated into Chinese). She will discuss her latest research on clustering applications to intraday statistical arbitrage at the Big Data Finance conference in Miami on December 5, 2017.