Back to Portfolio for the Future™

Why are Bank Stress Tests like Student/Instructor Evaluations?

August 15, 2016

Two scholars affiliated with Columbia University have contributed to the ongoing discussion of Federal Reserve stress tests and their consequences in an article in the Journal of Alternative Investments.

Stress tests have become a big part of the U.S. regulatory scene since 2009, with the creation that year of the Supervisory Capital Assessment Program.

SCAP checked the ability of the largest bank holding companies to survive a further weakening in their assets’ values in the wake of the crisis of the year before.

In an important innovation, the results of SCAP were made public.

Publicity and Confidence

SCAP, and in particular the publication of its results, were generally considered successful, and this led to the creation of a Comprehensive Capital Analysis and Review, as well as a Dodd Frank Act Stress Testing (DFAST) program.

These programs, under the Federal Reserve’s authority, recapitulate SCAP: they begin with a small number of scenarios defined by economic variables. The banks and their supervisors must evaluate how much each would lose given each scenario. Usually the summary information on the evaluation is made public.

One of the points of all this is to create the data that justifies bank capital rules, and that rationalizes share buybacks or dividends.  Before the development of the CCAR/DFAST system, that is, before the global financial crisis, banks used their internal models to calculate risk-weighted assets, which in turn drove those decisions.

Studying the stress-testing process, Paul Glasserman and Gowtham Tangirala found that the results of the tests are predictable. The results from CCAR in particular for 2013 almost perfectly predict the results for 2014 for banks that participated in the process each year. The results did not start off so predictable, but they have become so over the years since the crisis, as banks develop the staff and database to treat the stress tests as a matter of routine.

One gets similar results if one groups test results by loan or bank category.

Why that is worrisome

The worry behind this research is of course that predictable results are unhelpful results. When I was a student, one filled out “teacher evaluation” forms at the end of most courses. These forms could take up several pages, with twenty or more questions per page, and little circles with the numbers 1 through 5 inside them next to each question. One filled in circle (1) to indicate that the instructor had the lowest possible skill in a certain respect, or (5) to indicate that he was the Amadeus of that particular skill. [The Salieri of that skill would get a (3).]

The student filling out this form would generally settle into a pattern. This teacher is, say, a 3. Then, once the pattern was settled upon (within the first page of the form) every question got the same answer. This made life easy and getting through this task quick. Of course, those results probably did little or no good for whatever administrative purposes had led to the creation of these forms in the first place. Glasserman and Tangirala seem to be suggesting that the stress test process is a little like that, and we’re past the first page, the settling-in has taken place.

Glasserman is Jack R. Anderson professor at Columbia University, Graduate School of Business, and is the author of Monte Carlo Methods in Financial Engineering. Tangirala is a Ph.D. candidate there.

Going a bit beyond the conclusions of Glasserman and Tangirala, phrased as they were with scholarly caution, I submit that we can also say this: whatever market confidence is founded upon stress tests looks very much like unfounded confidence. If the stress test regime has created a sense that buffers are in place, so any future crisis will be quickly resolved, then the tests have done the equivalent of soothing us to sleep, and they may have worsened that future crisis.

There are other reasons to believe that the regulators’ emphasis on stress-testable cushions is misguided. A study done three years ago, Viral Acharya et al., “Testing Macroprudential Stress Tests,” as a working paper for the NBER, suggested that “continued reliance on regulatory risk weights in stress tests appears to have left financial sectors under-capitalized….”