Calibration and Econometric Non-Practice
Frydman and Goldberg, Beyond Mechanical Markets http://tinyurl.com/dl20111017c, sends us to a 2005 interview of Thomas Sargent by Evans and Honkapohja. Here is the section on the origins of "calibration"--which Olivier Blanchard once defined as "a way of ignoring the fact that the data do not fit your model, and proceeding as if they did":
Evans and Honkapohja: What were the profession's most important responses to the Lucas Critique?
Sargent: There were two. The first and most optimistic response was complete rational expectations econometrics. A rational expectations equilibrium is a likelihood function. Maximize it.
Evans and Honkapohja: Why optimistic?
Sargent: You have to believe in your model to use the likelihood function. it provides a coherent way to estimate objects of interest (preferences, technologies, information sets, measurement processes) within the context of a trusted model.
Evans and Honkapohja: What was the second response?
Sargent: Various types of calibration. Calibration is less optimistic about what your theory can accomplish because you would only use it if you din't fully trust your entire model, meaning that you think your model is partly misspecified or incompetely specified, or if you trusted someone else's model and data set more than your own. My recollection is that Bob Lucas and Ed Prescott were initially very enthusiastic about rational expetations econometrics. After all, it simply involved imposing on ourselves the same high standards we had criticized the Keynesians for failing to live up to. But after about five years of doing likelihood ratio tests on rational expectations models, I recall Bob Lucas and Ed Prescott both telling me that those tests were rejecting too many good models. The idea of calibration is to ignore some of the probabilistic implications of your model but to retain others. Somehow, calibration was intended as a balanced response to professing that your model, although not correct, is still worthy as a vehicle for quantitative policy analysis....
Evans and Honkapohja: Do you think calibration in macroeconomics was an advance?
Sargent: In many ways, yes. I view it as a constructive response to Bob' remark that "your likelihood ratio tests are rejecting too many good models". In those days... there was a danger that skeptics and opponents would misread those likelihood ratio tests as rejections of an entire class of models, which of course they were not.... The unstated cse for calibration was that it was a way to continue the process of acquiring experience in matching rational expectations models to data by lowering our standards relative to maximum likelihood, and emphasizing those features of the data that our models could capture. Instead of trumpeting their failures in terms of dismal likelihood ratio statistics, celebrate the featuers that they could capture and focus attention on the next unexplained feature that ought to be explained. One can argue that this was a sensible response... a sequential plan of attack: let's first devote resources to learning how to create a range of compelling equilibrium models to incorporate interesting mechanisms. We'll be careful about the estimation in later years when we have mastered the modelling technology...
Yet that day of getting serious about building a model that could match the time series never came.
Moreover, I would say that it will not come.
Any rational-expectations model must match the ex post distribution of economic outcomes to the ex ante expectations of economic agents. Thus if it analyzes the post-World War II U.S., it must now must assume that 2008 was always in people's minds as a possible outcome that had some significant probability.
But, in fact, 2008 was a close-to-zero probability event.
James Cayne and John Fuld each lost about $1,000,000,000 and destroyed their firms in 2008--if they had seen any significant possibility of that coming, they would have acted very differently. The investors in and holders of options on Citigroup lost 93% of their money as a result of 2008. The investors in and holders of options on Bank of America lost 85% of their money as a result of 2008. The investors in and holders of options on Morgan Stanley lost more than 75% of their money.
It is not the case that the senior executives of these banks understood the risks that they were running, gambled, and lost. They had no clue that they were holding as much mortgage risk or house price risk or AIG risk as they were. They had little clue that the lower tail of the house price change distribution was as large as it turned out to be--and they confidently expected the Federal Reserve to save them and the economy and keep them out of that lower tail. They were not stupid. It was just that the world turned out to be stranger than they, given who they were and what their life-experience had been, imagined. As one former major Wall Street CEO once told me: you never know what your beta really is, because there is always some important of systemic risks that eludes you.
Any rational expectations econometric analysis of the U.S. financial sector will hae to assume that economic agents had an average expectation that 2% of the time the next year would be like 2008. And that is simply wrong: the expectation that 2008 would come was 0.02%, or at most 0.2%.
So it simply can't be done. The rational expectations assumption that agents anticipate the probability distribution of outcomes predicted by the correct economic model is simply wrong.
