In my last blog "Balanced Capitalism" (July 26) I tried to explain the urgent need in society-wide modeling. But I am a realist, meaning a pessimist. I do not expect to see such a project soon.
Nevertheless, the complexity, uncertainty, and perils of the 21st century undoubtedly will generate a large number of crucially important complex and long-term socioeconomic and technical problems. Solving these problems requires the application of advanced analytical tools that combine computers and sophisticated mathematical models. Using such a combination demands, however, embedding it in an ensemble of models and methods that would neutralize its potentially dangerous miscalculations. Models create illusions of knowledge and objectivity. Models need filters.
Which models? Fashionable econometric models are useless here; the historic data for non-crisis period would drastically change for the century of "black swans." Moreover, if they contain no such constraints as limits on the balance-of-trade, they are severely dangerous. The current crisis, and our inability to handle it, are greatly due to both the government's and Fed's stubborn and stupid use of such non-protective models.
Optimization models remain as the natural choice. A prominent economist Robert Dorfman (a co-author with Paul Samuelson and Robert Solow of Linear Programming and Economic Analysis) used to say that he saw the world as an enormous problem of linear programming. Similarly, I see the world as an enormous problem of optimization under radical, "uninsurable" uncertainty.
That is, we know almost nothing about the future. At best, we can get no more than "guesstimates." They may include some plausible predictions about technological trends, but they still would be scenario-specific, while we do not have any credible idea about the probabilities of potential scenarios. Even more important, we know almost nothing about what is the right way to long-term sustainable survival of society. Sure, this corrupt and dysfunctional political system has to change, and so has capitalism. Unanimous "yes," I think. But how? The best we can do is trying to prevent, mitigate, and postpone the most obvious catastrophes.
Until now, neither economics nor Operations Research, Decision Science, and similar disciplines had any methods of addressing such overwhelmingly difficult problems. Now Risk-Constrained Optimization (RCO) provides the required capability.
I've been fortunate to be able to develop the required ensemble of models and methods. Namely, RCO provides the necessary ensemble of many novel concepts, models and methods that reasonably neutralize dangers and are critical to each other's success. They may be combined with other approaches, and they may be modified - nothing in this world is perfect.
But all these techniques, associated with seven diverse disciplines plus use of computers, appear to be necessary basic components of any dependable methodology of decision-making for non-trivial (complex and long-term) problems. Of course, we have to guess a tremendous lot and to use subjective judgment, but RCO combines "guesstimated" knowledge about the future with subjectivity in a sensible manner.
Thus, for the first time in more than 60 years, RCO legitimizes the high-level analytical use of a "computer-optimization model" combination. That perhaps can be considered as a "basic innovation" - a major technological or scientific breakthrough.
Also, RCO is the only currently existing methodology for strategic risk management in organizational (corporate or public) planning under radical (and less severe, too) uncertainty. RCO constructs a set of robust and flexible, reasonably good and safe long-range candidate strategies. The final strategy is selected from that set subjectively. As with any protective equipment, RCO could reduce the need for knowledge about the future.
Perhaps the most important of the RCO component models and methods is imposing on optimization models an additional function of self-filtering, so that they become very efficient "optimizing filters."
As an inseparable component of the outlined above ensemble, RCO introduces a new paradigm of decision-making - "catastrophe avoidance," which replaces the current economic paradigm of utility or profit maximization. The new paradigm is good for all times, but it fits especially well the type of problems we face in this century.
Once more - time has come to address crucially important problems facing both this country and the mankind. RCO currently provides the only known, scientifically and practically viable possibility of modeling such problems. I consider it my duty to outline this novel fundamental possibility. Use it or lose it.
We have developed powerful computers that perform billions operations per second, and mathematical programming models and algorithms that can deal with millions of constraints and variables. (Compare that with statistical models that can at best handle dozens of variables.) But they are never employed where they are desperately needed and where they should bring real benefits - as high-level analytical tools in long-range social and business strategic planning and decision-making.
And our present economic crisis - crisis that destroyed many trillions of wealth, many millions of jobs, and is far from ended -- confirms that sad conclusion. In other words, our technology has created two new Wonders of the World, but - outside the Information Revolution in short-term operations -- we are using them like Neanderthals, to crack nuts.
From its very inception in 1947, linear programming (LP) was described as an activity aimed at the "optimum," or "best," allocation of limited resources. (There are many types of mathematical programming, besides linear; I will collectively call "LP" all of them.)
"Optimization" has a strong positive connotation: its models and methods are supposed to find "the best." But it does that only if we have perfect knowledge of the future. Even the slightest change in the given values of the model's coefficients, supply and demand data, costs, or prices may lead to complete change of solution. The LP solutions are notoriously unstable and therefore extremely dangerous.
We may have perfect knowledge only in the simplest - and short-term, too - problems, such as "rearranging chairs on the deck of the Titanic." For any serious problem, LP does not "optimization" but "extremization": it gets us out on a limb and, as a rule, too far.
Of course, the black belts of Operations Research, starting from the originators of LP, understood very well that perfect knowledge about the future is impossible. They created a branch of mathematical programming called "stochastic programming" (SP) that allowed to formulate and solve optimization problems in a larger universe - under "insurable risk," that is, if we know probabilities of the future scenarios.
The main tool of SP is multi-scenario model. Suppose we have 12 scenarios that differ in their values of, say, prices of resources and products. We put, side by side, 12 "scenario submodels" that have identical structures, but different price values. The submodels are integrated into a single model by, say, a constraint on how much of a raw material is available. (So that the total use of this raw material in all 12 scenario solutions does not exceed its amount available.)
To each submodel we attach a weight that equals the probability of the corresponding scenario. Then we solve the unified problem as a usual LP model. If the model's goal is maximizing profit, we maximize here the weighted average of profits obtained in 12 submodels. Depending on their price values, scenario solutions may be different. The optimal solution is a weighted compromise between 12 scenario solutions. It avoids extremes and is therefore less risky.
SP has some excellent properties. For instance, it somewhat improves the quality of solutions in comparison with the quality of input data, so it is, to some degree, self-correcting. If we include a proper constraint in the model, it will allow harmful emissions only up to the specified limit. But only "on the average" -- SP does not care what will be emission level, if we apply its derived solution, under any individual scenario. It may be less risky than the solution of a deterministic (that is, a one-scenario) model, but it still is risky. It creates a single solution, which looks like the "true optimum." Again, except for trivial problems, this is an illusion. In bad times, illusions are hazardous.
The main limitation of SP is, however, its low applicability. We do not have in any serious problem what SP absolutely requires - reliable information about scenario probabilities.
In his 1990 book The Fifth Discipline, Peter Senge proposes a great idea. He suggests that a "basic innovation" results only from combining of a special ensemble of efficient "component technologies" that come from diverse fields of science or technology, and only when all necessary components of that ensemble come together. Senge classifies as "basic innovations" the creation of the telephone, the digital computer, or commercial aircraft -- major technological or scientific breakthroughs.
Senge describes the creation of the commercial aircraft in the following terms: " [T]he McDonnell Douglass DC-3, introduced in 1935, ... for the first time brought together five critical component technologies that formed a successful ensemble. They were: the variable-pitch propeller, retractable landing gear, a type of lightweight molded body construction called 'monococque,' radial air-cooled engine, and wing flaps. To succeed, the DC-3 needed all five; four were not enough."
In the 21st century, we have many crucially important problems that may define the fate of the humankind. Given the gravity of these problems, getting help from computers and mathematical programming might be tantamount to a "basic innovation," perhaps no less important than the very creation of these tools. I will not mince words here. It is extremely important to design a powerful tool; it is no less important to make that tool truly useful. The more powerful a tool is, the more dangerous it may become when wrongly applied.
Senge strongly emphasizes that the power of the ensemble comes mainly not from the individual components, but from their combined impact within the process. In his words, they form an inseparable ensemble and " ... are critical to each others' success." In other words, we need a yin yang of technologies. Our task is to discover whether a set of technologies with such an inestimable property does exist in our field of decision-making under radical uncertainty, and if it does -- to develop and combine all its components.
Of course, each "basic innovation" is usually created by many people and organizations during a substantial period of time. According to Senge, about 30 years "is a typical time period for incubating basic innovations." Since I worked on RCO alone (of course, using some embryo results, previously obtained by others), the development of RCO took longer - starting from the 1960s, longer than Moses wandered in the desert.
Novel models and methods came through the years one at a time, each replacing some weak link. RCO was granted an USA patent in 1999, but its final features were developed in the 2000s. (From 2004 to 2009, I've published several articles on RCO and made presentations at international scientific conventions. A definitive 51-page article on RCO is to be published in the Fall of 2010. As mentioned in the end of this blog, all these materials are available for asking.) Work on RCO, including linking it with other decision-making approaches, continues.
In combination with computers, RCO merges in the required ensemble a number of technologies that are associated with seven fields: Economics, Decision Science, Operations Research/Management Science, Scenario Planning, Risk Management, Utility Theory, and Portfolio Theory, with a digression into the eighth field - Psychology. In each of the seven fields, RCO introduces novel concepts, models and methods. Exactly as required by Senge, models and filters are inseparable and " ... are critical to each others' success."
Not going into technical detail, I will mention here just two most important, closely interconnected novelties: change of paradigm and "strong screening" by special (enhanced) stochastic multiscenario models with risk-limiting constraints.
RCO demonstrates the need, and shows the way, of changing the general paradigm of economic decision-making. The present paradigm maximizes satisfaction of our needs and wants. This paradigm is incorrect, because it does not take into account two crucially important features: the income to make that satisfaction possible, and our desire for self-preservation, for ensuring safety of the existing comfortable situation. (A certain way to get into a crisis is to rely on a paradigm that denies the very possibility of a crisis. That's what we see now.)
Instead, RCO deals with a simpler, easier to solve, more realistic and pressing problem -- how to avoid a catastrophic outcome in any of multiple types of risk facing us in the future. But it continues to use, as a tool, enhanced multi-scenario LP models, screening its results through five consecutive filters. Those filters eliminate, modify, or scale back too risky decisions or strategy candidates developed by the system.
An LP model consists of two parts: the model's goal, "objective function" (it is a description of what we maximize - say, long-term profit), and constraints (what resources are available, how much product can we sell at a given price). LP solutions are especially sensitive to costs and prices, which define the value of the objective function. A price modification that changes the total value of that function by just one dollar may completely change the solution, locating a new factory on another continent. And who in the world can predict faraway prices and costs with such precision? As a preeminent economist, Oscar Morgenstern, used to say, we now and again may be lucky to determine just the correct sign (plus or minus) of some data.
The objective function is therefore not at all dependable: it indicates the direction of search for solution - at best, very approximately, and at worst very badly, maybe even opposite to the correct direction. The "constraint cages" are, however, much more reliable. The purpose of RCO is to construct the proper cage. How to do that?
Again, I will not go into technical detail. Sufficient is to say that RCO starts where SP ends. Suppose we have the solution of our special SP multi-scenario model and derived from it a strategy. If we see any undesirable outcomes in any type of risk and in any individual scenario that exceed what we consider acceptable for that scenario, the decision-maker adds to the model additional "risk-limiting" constraints of such type as "Emission of sulfur for Scenario 187 should nor exceed 1,000 tons." Then we re-run the model. Since it is an optimization model, it finds the best way to modify the strategy to meet that target.
The new strategy will be "worse" from the point if view of the overall profit, but it will be less risky. We repeat this "Dutch auction" procedure until all our apprehensions are met, within the limits of what is possible. Please notice: The procedure exactly matches the aim of the catastrophe avoidance paradigm. That is what I call a perfect yin yang of technologies.
Let me emphasize the following: it is the decision-maker, rather than the modeling professional, who has to define what is acceptable, and who has to impose the additional constraints. And even an unsophisticated decision-maker is able to do that because the constraints are simple expressions of his concerns about the outcomes; they do not require understanding the model's structure and complexity.
One way or another, somebody's subjectivity will be inserted into the model, and it better be subjective preferences and apprehensions of the decision-maker, who undoubtedly knows more about the hidden realities and not-modeled complexities of the problem.
The model is now "customized" to the decision-maker(s) and becomes a "self-filtering" combination of our "guesstimated" knowledge about the future, on the one hand, and subjective knowledge, intuition, and preferences of the decision-maker(s), on the other hand.
The "self-filtering" optimization model is the strongest filter of RCO, but it is only one of its five filters - which also, in accordance with a very wise admonition of Peter Senge, come from different disciplines (that diversifies the methodology of screening and thus increases its effectiveness).
RCO may produce trees of strategy candidates with different optimal trade-offs between improving the results and protecting from risks. Then it reduces that multitude to a small set of the best and reasonably safe candidates. The final choice of a strategy to be implemented is made from that set by the decision-maker(s) subjectively.
I apologize to the reader for abundance of semi-technical detail (which is not really difficult). But I guess that for everybody - including the professionals in the field -- reading this blog would be useful.
Let me repeat this truism: the time has come to address serious socioeconomic problems. I do not know -- maybe we are still able to do something. RCO is the only realistic way to combine optimization models and computers in trying to solve these problems. (Of course, RCO is also applicable to solving any organizational optimization problems under uncertainty.) Once again, it is my duty to outline this novel fundamental possibility. Anybody who is interested can get additional information and materials from me; see my email address in the bio.
Copyright © 2010
Ironically, once it is widely acknowledged, it is likely to motivate a surprising stimulus package.
The Arctic Global Warming Tipping Point has recently been found to be 400 parts per million of carbon in the atmosphere.
We are currently at 390ppm and adding 2ppm each year.
Replacing fossil fuels with renewable energy systems as fast as possible may soon be recognized as necessary to insure human survival.
See A 5 Point Program at http://www.aesopinstitute.org for a summary and supporting evidence.
If these facts are correct, we need to provide whatever incentives are necessary to produce and deploy renewable energy systems on a 24/7 basis, worldwide. And we need to do that fast!
The earth has experienced two Tipping Points in the distant past. Both destroyed most life on the planet. In one case down to single-celled organisms.
A Tipping Point is unstoppable once it begins. Preventing it still appears possible. If we fail, we can kiss our lives and the planet goodbye.
Check the facts. Then see if you can find ways to incorporate them in your work.
If we confront the challenge, ironically the economy will revive, in a manner analogous to the way in which WWII ended the great depression. The huge number of jobs generated by that war resulted in a 2% rate of unemployment.
Let's demonstrate that we can do as well in meeting a life-threatening peacetime emergency!
Unanimity, Dr. Masch? To wish....perchance to dream. Let's pick a person, and wager on the likelihood
of agreement with your assessment. I will choose Lloyd Blankfein, and wager $100K that he will vehemently disagree with you on the state of the economy, and what needs to be done in that area.
I very much respect and admire the work that you are presenting here, but in my opinion the sociopathic Alphas that have gained dominance in the U.S. will not act rationally or pragmatically.
I fear that the slow evolution of human kind suggests that violent upheaval may still be, in a sense, required.
Now the government has predicted the future, because of that, investors can invest and innovators can innovate. Now the home medical office is a reality.
References
EHR http://en.wikipedia.org/wiki/Electronic_health_record
XML http://en.wikipedia.org/wiki/XML
XML schema http://en.wikipedia.org/wiki/XML_schema
XForms http://en.wikipedia.org/wiki/Xforms
web-services http://en.wikipedia.org/wiki/Web_service
IETM Class V http://en.wikipedia.org/wiki/IETM
DITA http://en.wikipedia.org/wiki/Darwin_Information_Typing_Architecture
A presentation by IBM using DITA
IBM http://dita.xml.org/sites/dita.xml.org/files/IDCMSBlue.pdf
Cloud Computing http://en.wikipedia.org/wiki/Cloud_computing
SaaS http://en.wikipedia.org/wiki/Software_as_a_service
An excellent article from a Brookings Institute Study from a medical standpoint http://www.brookings.edu/reports/2009/0901_btc.aspx
We are facing an unrecognized emergency that appears to require a massive economic effort.
The Arctic Global Warming Tipping Point has recently been found to be 400 parts per million of carbon in the atmosphere.
We are currently at 390ppm and adding 2ppm each year.
Replacing fossil fuels with renewable energy systems as fast as possible may soon be recognized as a requirement for human survival.
See A 5 Point Program at http://www.aesopinstitute.org for a summary and some supporting evidence.
If these facts are correct, we may want to provide whatever incentives are necessary to produce and deploy renewable energy systems on a 24/7 basis, worldwide.
During WWII we had Overfull employment, defined as a 2% rate of unemployment.
The earth has experienced two Tipping Points in the distant past. Both destroyed most life on the planet. In one case down to single-celled organisms.
A Tipping Point is unstoppable once it begins. Preventing it still appears possible. If we fail, we can kiss it all goodbye.
Check the facts.
If we confront the challenge, ironically the economy will revive, in a manner analogous to the way in which WWII ended the great depression.
When I say, Maybe we still can do something, that is what I mean.
I have to mention that my original title was just
"Risk-Constrained Optimization."
The preceding part of the title was added by someone else.
not to be accused of suffering from mania grandiosa::
My title was just
"Risk-Constrained Optimization".
The preceding part of the title was added by HP.
I am using the computer.
Computer does not use me.