Markets are powerful and efficient. Markets fail.
Engineers study failure. When we design a structure or a system, it is our professional obligation to account for known failure mechanisms, and produce a robust, practical, safe design. We don't rely on invisible hands.
Economists will quietly and reluctantly acknowledge several conventional mechanisms for market failure.
The housing bubble and financial crisis illustrate a new mechanism, which we should add to our list of failure modes -- market failure by "group-think."
Group-think can present itself as social pressure to accept and conform to the particular conceptual framework of the peer group. In the case of our housing bubble, the conceptual convention was that housing prices would always go up. Within that convention, smart experienced people could not process observations that contradicted the prevailing wisdom.
Group-think can have an "eyes open" presentation, as well. This failure mode works through a perverse incentive -- sending faulty market signals to decision-makers.
A bank officer or executive may be perceptive enough to recognize the herd behavior, and see that the herd is heading to disaster. However, if the bank officer takes defensive action too soon, it can cost the bank money. As long as one "greater fool" still wants to buy an over-valued financial derivative, the bank will feel compelled to write more bad home loans or create one more synthetic derivative. If a competitor's bank stays in the bubble an extra six or nine months and realizes additional (unsustainable) gains, then the bank that dropped out too soon will be punished by its shareholders for poor performance. No bank can "afford" to break ranks with the herd.
Another example might be an oil company drilling in dangerous waters. Experienced workers, under pressure from cost and delivery schedules, may feel compelled to cut corners. If a driller acts responsibly, but a competitor always cuts corners, then the market will punish the responsible company in the short term. Market punishment will be immediate and lasting, while the benefit of avoiding a disaster by responsible behavior could be deferred decades into the future. Upton Sinclair said it this way:
"It is difficult to get a man to understand something, when his salary depends upon his not understanding it!"
On the other hand, no prestige is lost by staying in until the bubble bursts, then saying, "No one could have foreseen this." More often that not, when someone says, "No one could have foreseen this," many had foreseen it, but were dismissed by those caught in group-think.
Market fundamentalism excludes the possibility of group-think, assuming that buyers and sellers are rational, perfectly well informed, and motivated entirely by self-interest. Everyone acts independently, yet shares all knowable information.
In the housing crisis, group-think was pervasive, affecting rating agencies, regulators, legislators, lenders, borrowers, mortgage brokers, pundits, thousands of handsomely-paid securities dealers, and most of the economics profession. This is a towering departure from the perfectly rational and all-knowing behavior assumed by market theory.
Mechanisms of market failure are 100 percent meaningful to millions of people who lose their homes, their jobs, their life savings, or their children's college funds.
We are taught that the market's invisible hand will find the optimal, most efficient outcome. Our faith in the invisible hand cannot justify the trillions of dollars misallocated through an escalating series of market failures over the last few decades. With some regularity, the invisible hand just royally screws up.
Joseph Stiglitz draws a simple inference:
" ... the reason the invisible hand seems invisible is that it is not there."
Updating Sidney Harris' classic cartoon:
All market failure mechanisms deserve careful study and respect - externalities, moral hazard, principal-agent problems, market domination, asymmetric information and, I would add, group-think. Market failures are really structural weaknesses built into the rules we use to operate our economy.
Now that "financial engineering" dominates one of our largest industries, policy-makers should adopt another fundamental engineering practice -- systems solutions that account for failure mechanisms, as the only way to design systems that are robust, practical and safe.