Fooled By Randomness

I am back.

I have just finished reading a book that has motivated me sufficiently to want to write a post: "Fooled by Randomness - The Hidden Role of Chance in Life and in the Markets" by Nissim Nicholas Taleb (2001). Taleb subsequently wrote a second book, "The Black Swan"(2007), the title of which has become popular since the recent credit crisis in America, to describe rare, difficult-to-predict events that have a large impact. Taleb was an options trader in New York, and while both books dwell a lot on financial markets, they should be interesting to anyone who is interested in epistemology, understanding the nature of knowledge.

Let's start with "The Black Swan". Till the 17th century, people in Europe 'knew' that all swans were white. That would have been a reasonable assumption to make considering that all the swans that people had seen, and their parents and grandparents before them had seen, were all white. This is an instance of inductive reasoning, where people drew a universal conclusion (all swans are white) from a set of observations (all the swans that I have seen are white). But what if someone, somewhere, saw a black swan? Everyone would have to change their ideas about swans and their color. This did happen. The early European explorers spotted black swans (Cygnus atratus) in Australia, in a place that had not been 'discovered' till the 17th century.The colour of swans is perhaps not a significant matter for most people, and I suspect that the discovery of the Black Australian Swan did not monopolize newspaper headlines in Europe for too long. But weak inductive reasoning abounds in the modern world in matters that have far greater significance to our lives.

What about this: you are approached by a mutual fund that solicits your money. The fund is run by a fund manager who has beaten his benchmark, usually some kind of index like the Sensex or the Dow, for the last five years. Would you invest your money with him? Are five observations (the fund manager's performance for each of the last five years), sufficient for you to reach the conclusion that he will grow your money in the future? What if the guy is just a lucky fool?

What if your fund manager is just a survivor? This is how it can happen. Imagine that your fund manager was one of 10,000 fund managers who started out 5 years ago. Assume that managers suceed purely because of luck, and the odds of being lucky in any given year are 50%. At the end of the first year, 5,000 managers would beat the index, while the remaining would get fired. In the second year,the number of surviving guys from that first batch would be down to 2,500, and 1,250, 625, 312 in the each of the remaining years. Your fund manager, with his 5 year streak of winning is one of those 312. He is considered a success in his profession, draws a huge salary and comes on CNBC several times a week voicing his opinion about the direction of the market. Everyone believes that he can tell a good stock from a bad one, including probably himself, till one day, his luck runs out, and he gets fired like the others before him. You lose your money.

There is evidence that this may actually be true. The New York Times had an article, titled appropriately, "The Prescient are Few", which discussed a study called “False Discoveries in Mutual Fund Performance: Measuring Luck in Estimating Alphas” by three researchers who looked the fund performance between 1975 and 2006 using a statistical test called the False Discovery Rate that eliminates the possibility that a conclusion (for e.g. skill matters in success as a Mutual Fund manager) is statistically significant when it is actually random, and the reverse. The conclusion: "the number of funds that have beaten the market over their entire histories is so small that the False Discovery Rate test can’t eliminate the possibility that the few that did were merely false positives — just lucky, in other words. Very few fund managers actually showed true stock picking ability over a long period of time." You can download the original research paper here.

What applies to fund managers applies to other actors in this world. People who are considered successful by conventional standards -wealth and fame- are very often are under the illusion, (and spread this illusion) that they are successful because they were thriftier, or smarter, or more hardworking than the rest of the population. Do all hardworking, thrifty, smart people achieve an equal measure of success? Perhaps not.

I looked up the web, and found some interesting data for the chances for commercial success for an inventor. Bob Shaver is a patent attorney in the United States who noticed that only 5 in 100 inventions ended in some sort of commercial success. Most people, including me, would think inventors are both hardworking and smart.

It may be more than political correctness, therefore, to address the poor as the "less fortunate".

As you can see, by the length of this post, I liked this book very much.