Statistics and Human Psychology

The human mind has made tremendous achievements in logically understanding the world around us in the past two centuries. Since the industrial revolution science has given us breakthrough technology, medicine as well as a “rational” way of looking at the world around us. Causality that had been explained using “gods of rain/wind” for example has slowly been discarded across cultures as more experimentally verified  explanations developed. Does that mean the human mind has achieved complete rationality and all human behavior can be explained using fundamental laws or the concept of equilibrium? The short answer is no.

There have been number of experiments and studies in the second half of the 20th century that show just how poor human mind (including ours) is at coping with randomness. It turns out that not only are people not rational in their decision-making, but they are completely fooled by the properties of randomness.

Its human tendency to take shortcuts (Kahneman). Rather than carefully evaluating risks, humans usually just look at the past and assume it reflects the future. Even when we extrapolate past statistics often times we only count events we remember, events that were significant. So frequencies of events measured in the past often times is used as the probability of an event in the future. Just like a fair coin toss. The probability of getting heads is calculated by repeating the coin tossing experiment a large number of times and dividing the # of heads by the total # of outcomes. So the probability of heads on each coin flip is 0.5. In most real world situations (like financial markets) you cannot necessarily rely on historical frequency  based probability. Just because an event has never happened in the past does not mean it wont happen in the future. This is because unlike a coin toss, the initial conditions keep on changing everyday and most of the times are unobservable or get too complicated. So any probability assigned to such an event in the future is based on belief and is referred to as belief-based or subjective probability.

Do you understand the weather ?

Just consider the weather predictions. A weatherman brings out a 30% chance of rain and you keep your umbrella tucked at your home. You go out and see the clouds thundering and water drops battering the ground and you curse the weatherman. But the poor guy has done no wrong actually. A forecast of less than 50% chance doesn’t mean no rain. It simply means that if weather conditions like today played over 100 times, 30 of those times it will rain.

Value of Money


This figure above shows two curves – blue and red. The blue curve has a constant slope (rise over run) whereas the red one rises less sharply in gains territory and falls more sharply in losses territory. The red curve indicates that losses hurt more than equal amount of gains. Kahneman and Tversky (see Prospect theory) found that generally people behave according to the red curve. Consider a situation where a person gains $100 and then loses all of them. The blue curve would suggest that the person got the same amount of happiness from winning that money as the sorrow she got from losing it and so she should be indifferent. But if that person has a value function similar to the red curve then the sorrow of losing will be greater than the happiness of winning.  Now this may seem strange because it seems like people do not care about the absolute wealth. This asymmetry can also be used to explain why investors still hold on to losing stocks despite it probably being a riskier proposition. It is because losses hurt them more psychologically. Loss aversion does not necessarily mean risk aversion. It is important to remember that this behavior is not absolute but depends upon the reference point and the change in wealth and varies for every individual.

Black Swan 

Most of us probably remember where we were, what we were doing when two planes flew  into the World Trade Center in New york City on the morning of September 11, 2001. The shocking images of the tall skyscrapers crumbling to the ground are probably vivid in most people’s mind. 9/11 is what we call a “black swan” in probability terms. It was unexpected and caused great destruction – low probability and high impact event. Any normal distribution would have failed to recognize the true impact of 9/11 not just in terms of loss in human lives but economic impact as well. yet we continue using normal probability distributions to quantify risk in many cases only because its simple.

Power law

In our daily lives, we often face situations where the errors are only minor. Consider a machine which we encounter at work. It looks like it breaks down only infrequently and with small defects. So you might come to the conclusion that you are never going to face a big headache due to the machine. Enter the power law!

This strange but simple looking distribution is probably one statistical tool we can use in this case. The mundane small failures are all on the left. You can see that these events have a high probability of happening. But have a look at the right tail – the dreaded uncommon events. it shows that the probability is low, but they can occur. This may not apply in all situations. But we have to be ready for the right tail. The nice looking yellow part is the one that’s going to matter. Be prepared for that!

Don’t discard common sense! 

In many situations however its better to follow a heuristic approach to decision making because its simply impractical to do a complex analysis. This approach is quite common with traders of financial assets. The problems at hand are very complex and so the mind takes short cuts without being fully aware of the process. But a smart trader is aware of this fact and learns from trial and error. We believe that trial and small error accompanied by a sound understanding of statistics is probably a better approach in such situations than relying on complex mathematical models because the real world situations really make the decision-making process too complicated due to incomplete information. Its very similar (but more difficult) to learning how to drive in traffic – switching lanes to reduce the time of travel gets better with trial and error.

Uncertainty is everywhere; Deal with it !

For most of us, death will be unexpected and for almost all of us (don’t wanna piss off those who claim to have risen from the dead) its a once in a lifetime event. But many of us do not go about worrying if the fateful day is going to be today. But it is our duty to ensure that after we are gone, the impact of our death is minimal on those who survive us. Failure to do so cannot be attributed to a sudden catastrophe; only to the flaw in our understanding of this great game of chance.