In 1994, Jim Collins and Jerry Porras, former and current Stanford professors, published the best-seller Built to Last that described how “long-term sustained performance can be engineered into the DNA of an enterprise.”  It sold over a million copies. Buoyed by that success, Collins and a research team set out to find the characteristics of companies that convert long-term mediocrity or worse into long-term superiority The result was his next book, Good to Great.

Collins and his team of 20 researchers devoted over 15,000 hours to the project, which included interviews with numerous CEO’s.  After reviewing the entire set of Fortune 500 companies, they identified only 11 firms that had made the transition to great. Taking pride in the fact that their data do not represent a sample, they said the fact that we studied the total set of companies that met our criteria should increase our confidence in the findings.

Very extensive analysis yielded a set of guiding principles for the great companies. One was the “hedgehog” principle. Basically, this can be translated to “simplify and focus”, which is (somewhat fuzzily) analogous to the hedgehog rolling up in a ball and greeting the world with spikes. Collins and his researchers also found that all but one of their great companies had CEO’s that came up from within the firm, as opposed to celebrity CEO’s hired from outside. The principles produced by the project all make sense, though one can imagine other principles that also sound good. But the principles delineated in Good to Great were validated by extensive research, and they unite the 11  companies. This is where the statistician’s ears prick up.

There are two related statistical pitfalls lurking here:

1. Selecting the best and attributing their greatness attributes they have in common reverses the usual flow of logic in statistical studies:

2. When selecting the best of anything, you are bound to be disappointed with its future performance.  The phenomenon is known as regression to the mean – the most extreme cases invariably owe a part of their extremeness to either good luck or bad luck, which will wash out in the future, returning both the superlative and the terrible performers towards the middle.

Collins’ book was published 18 years ago, so we can now go back and check. Did his 11 companies, embodying the principles of greatness, persist in their greatness?  Results were mixed, at best.

• In the 15 years following the end of the study period, roughly half out-performed the Dow Jones, while the others under-performed

• Circuit City went bankrupt

• Fannie Mae had to be bailed out by the government

• Wells Fargo continued above average for its 15 years, but then was engulfed in a scandal (over-eager sales associates were signing customers up for services they didn’t ask for)

Conclusion? It looks like a case of unintentional selection bias (biased sample selection), and resulting regression to the mean.