On April 21, 2015, the U.S. Department of Justice announced that it would press criminal charges against Mr. Navinder Singh Sarao, a 36-year-old small-time British day-trader. He is being blamed for nothing less than causing the "Flash Crash" of May 6, 2010, the second largest point swing (1010.14 points) and the biggest one-day point decline (998.5 points) in the history of the Dow Jones Industrial Average. This shocking revelation is just the most recent turn in a surreal story that exposes the technical inadequacy of present-day market regulatory agencies.
A bit of history
Following that infamous day, observers were quick to offer explanations for the flash crash, from a "fat-finger trade" in the "Procter and Gamble" stock, changes in the U.S. Dollar/Japanese Yen exchange rate, to a large purchase of put options by the hedge fund "Universa Investments." On September 30, 2010, after almost 5 months of analyses, the CFTC-SEC issued its final
. This report blamed the crash on a low-frequency trader (p.2):
At 2:32 p.m., against this backdrop of unusually high volatility and thinning liquidity, a large fundamental trader (a mutual fund complex) initiated a sell program to sell a total of 75,000 E-Mini contracts (valued at approximately $4.1 billion) as a hedge to an existing equity position.
The report rightly pointed that the "flash crash" was the result of a liquidity crisis. However, it made the mistake of blaming a low-frequency trader for causing that crisis. On November 19, 2010, a new study by Easley, Lopez de Prado and O'Hara (published in the Journal of Portfolio Management) explained that liquidity crises are a structural part of our new, high frequency world. A mutual fund was not the culprit. The market structure, sanctioned by the 2007 RegNMS legislation, set the conditions for an accident to happen. In particular, this Easley et al. study showed how order imbalance reached toxic levels one hour before the crash.
Order Imbalance (dotted red line) reached toxic levels one hour before the crash. See video here
It was order imbalance...
To understand the crash, we have to explain how modern markets work:
- A transaction occurs when two opposite sides (a buyer and a seller) agree on a price and an amount.
- One side provides liquidity and the other takes liquidity. The liquidity provider posts an order, and waits for a liquidity taker to match it.
- Liquidity providers are "sitting ducks." They are afraid of offering too many securities at a wrong price (selling too low or buying too high). They avoid this by monitoring order imbalance. Sellers will provide liquidity at a higher price if they see that buying orders are more prevalent than selling orders. Buyer will provide liquidity at a lower price if they see that selling orders are more prevalent than buying orders.
These operations were done mostly by humans until around 2008, when High Frequency Trading (HFT) became the most common way of transacting. High Frequency Trading (HFT) is the rapid exchange of securities enabled by sophisticated computer algorithms and advanced communications technology. Unlike humans, algorithms operate in a
. This means that rather than analyzing order imbalance every hour, or every minute, HFT algorithms analyze order imbalance in volume buckets. When activity spikes, they will make even more decisions in less time, which will trigger even more activity.
This sort of snowballing effect is characteristic of HFT's volume clock paradigm. Spoofers know this and exploit it to their advantage. A "spoofer" is a trader that generates one-sided orders in high volume with the intention of scaring liquidity providers of the opposite side.
The SEC-CFTC finally see the light
Every financial outlet seemed shocked last Tuesday (April 21, 2015) to hear the new explanation pursued by our regulators. Here is a
on the announcement (underline is ours):
"His conduct was at least significantly responsible for the order imbalance that in turn was one of the conditions that led to the flash crash," said Aitan Goelman, head of enforcement at the Commodity Futures Trading Commission, which filed parallel civil charges against Sarao on Tuesday.
Yes, you read the Reuters article well. There is no mention of a 75,000-contracts sell order originated from a mutual fund, just as if that never happened, similar to the fate of Nikolai Yezhov, who, after his fall from power under Stalin, was executed and excised from public photos and records (see above). Instead, all of the sudden, Regulators have admitted that the Flash Crash was due to order imbalance, as Easley et al. had explained almost five years earlier. Here is a quote from a Bloomberg News article:
Tuesday's developments fly in the face of the prevailing narratives of what happened. Regulators initially concluded that a mutual fund company -- said to be Waddell and Reed Financial Inc. of Overland Park, Kansas -- played a leading role. Many in the industry countered that a confluence of several forces, including high-frequency trading, was probably behind the crash.
Why did it take five years?
As evidenced by the work of
, the conflation of HFT, order imbalance and spoofing was known five years ago. Then, why did it take five years to find Mr. Sarao? Two reasons.
First, Easley et al. pointed out the mechanism of the crash. However, they could not pursue the particular actors, because that would have required access to FIX-Tag-50 data. This is the data that identifies the person that generated each order. Section 8 of the Commodity Exchange Act bars the CFTC from disclosing this data.
Second, regulators appear to have been extremely satisfied with the official culprit: A mutual fund. The alternative explanation, that the post-RegNMS market structure exacerbates snowballing, was more inconvenient. Regulators preside over this market structure, and blaming a system prone to failures makes them partly responsible.
How did the whistleblower gain access to this confidential Section 8 data? This data appears to have been leaked in the past, and perhaps this was the source used by the whistleblower. What it is clear is that the SEC and CFTC did a poor job. In the words of a Bloomberg report:
[Regulators'] analysis was upended Tuesday with the arrest of Navinder Singh Sarao -- a U.K.-based trader accused by U.S. authorities of abusive algorithmic trading dating back to 2009. Even that action was spurred not by regulators' own analysis but by that of a whistle-blower who studied the crash, according to Shayne Stevenson, a Seattle lawyer representing the person who reported the conduct. The episode shows fundamental cracks in the way some of the world's most important markets are regulated, from the exchanges that get to police themselves to government agencies that complain they don't have adequate resources to do their jobs.
Is Sarao responsible for the flash crash?
This is, of course, a pointless question. Proving Mr. Sarao's guilt would require replaying the events of May 6, 2010 in absence of Sarao's spoofing, only to find that no crash occurred. This is an exercise of fiction. And even if that was possible, it would only prove what we already know:
The current market structure is an accident waiting to happen
. We can blame Mr. Sarao as much as we could blame a massive accident on a Mini-Cooper's reckless driver crashing on a road poorly maintained by the Department of Transportation. Yes, someone triggered the crash, and that unfortunate random individual may have been Mr. Sarao.
The real villain
The real villain of this story is Section 8 of the Commodity Exchange Act. If this data had been made available, spoofers would have been outed and charged back in 2010. Make no mistake: It is this law that protects spoofers, and should be blamed for the next flash crash. Any experienced data scientist would have found Mr. Sarao in minutes. His footprint was unmissable to trained eyes. Think of it,
, and regulators didn't see that? It takes a simple query of sell orders, grouped by FIX-Tag-50, to solve this case. After all, we are talking about a ridiculously small dataset, compared to the datasets used in Big Data analyses every day, which include trillions of records.
Lessons for the next flash crash
There are several lessons to be learned:
- The market structure is more complex now than it was five years ago. Regulators do not appear to be willing to discuss the actual root of the problem. More flash crashes have and will continue to occur.
- The SEC-CFTC explain that it took five years because of the magnitude of the data. Such comments expose a certain naivety regarding data: a few billion records is small data. The problem is, the regulatory agencies are not staffed with data scientists but instead are populated by Excel users (at best!). This is not a matter of budget or resources. All it took to catch Sarao was one competent person with access to confidential data. Perhaps the SEC-CFTC should transform a lawyer-centered agency into something closer to the NSA. Or they could simply get out of the way and let private citizens do the research for them, by divulging the Section 8 data.
- Since regulators failed to identify someone as careless as Mr. Sarao, most likely they will not be more successful in chasing the real pros working at cutting-edge HFT firms. There are thousands of spoofers out there, far more sophisticated than Sarao. Perhaps regulators do not go after high-frequency firms, because billionaires are a tougher fight than a 36-year-old small-time British day-trader.
Our 2024 Coverage Needs You
It's Another Trump-Biden Showdown — And We Need Your Help
The Future Of Democracy Is At Stake
Our 2024 Coverage Needs You
Your Loyalty Means The World To Us
As Americans head to the polls in 2024, the very future of our country is at stake. At HuffPost, we believe that a free press is critical to creating well-informed voters. That's why our journalism is free for everyone, even though other newsrooms retreat behind expensive paywalls.
Our journalists will continue to cover the twists and turns during this historic presidential election. With your help, we'll bring you hard-hitting investigations, well-researched analysis and timely takes you can't find elsewhere. Reporting in this current political climate is a responsibility we do not take lightly, and we thank you for your support.
Contribute as little as $2 to keep our news free for all.
Can't afford to donate? Support HuffPost by creating a free account and log in while you read.
The 2024 election is heating up, and women's rights, health care, voting rights, and the very future of democracy are all at stake. Donald Trump will face Joe Biden in the most consequential vote of our time. And HuffPost will be there, covering every twist and turn. America's future hangs in the balance. Would you consider contributing to support our journalism and keep it free for all during this critical season?
HuffPost believes news should be accessible to everyone, regardless of their ability to pay for it. We rely on readers like you to help fund our work. Any contribution you can make — even as little as $2 — goes directly toward supporting the impactful journalism that we will continue to produce this year. Thank you for being part of our story.
Can't afford to donate? Support HuffPost by creating a free account and log in while you read.
It's official: Donald Trump will face Joe Biden this fall in the presidential election. As we face the most consequential presidential election of our time, HuffPost is committed to bringing you up-to-date, accurate news about the 2024 race. While other outlets have retreated behind paywalls, you can trust our news will stay free.
But we can't do it without your help. Reader funding is one of the key ways we support our newsroom. Would you consider making a donation to help fund our news during this critical time? Your contributions are vital to supporting a free press.
Contribute as little as $2 to keep our journalism free and accessible to all.
Can't afford to donate? Support HuffPost by creating a free account and log in while you read.
As Americans head to the polls in 2024, the very future of our country is at stake. At HuffPost, we believe that a free press is critical to creating well-informed voters. That's why our journalism is free for everyone, even though other newsrooms retreat behind expensive paywalls.
Our journalists will continue to cover the twists and turns during this historic presidential election. With your help, we'll bring you hard-hitting investigations, well-researched analysis and timely takes you can't find elsewhere. Reporting in this current political climate is a responsibility we do not take lightly, and we thank you for your support.
Contribute as little as $2 to keep our news free for all.
Can't afford to donate? Support HuffPost by creating a free account and log in while you read.
Dear HuffPost Reader
Thank you for your past contribution to HuffPost. We are sincerely grateful for readers like you who help us ensure that we can keep our journalism free for everyone.
The stakes are high this year, and our 2024 coverage could use continued support. Would you consider becoming a regular HuffPost contributor?
Dear HuffPost Reader
Thank you for your past contribution to HuffPost. We are sincerely grateful for readers like you who help us ensure that we can keep our journalism free for everyone.
The stakes are high this year, and our 2024 coverage could use continued support. If circumstances have changed since you last contributed, we hope you'll consider contributing to HuffPost once more.
Support HuffPostAlready contributed? Log in to hide these messages.