Statistics and Human Psychology

The human mind has made tremendous achievements in logically understanding the world around us in the past two centuries. Since the industrial revolution science has given us breakthrough technology, medicine as well as a “rational” way of looking at the world around us. Causality that had been explained using “gods of rain/wind” for example has slowly been discarded across cultures as more experimentally verified  explanations developed. Does that mean the human mind has achieved complete rationality and all human behavior can be explained using fundamental laws or the concept of equilibrium? The short answer is no.

There have been number of experiments and studies in the second half of the 20th century that show just how poor human mind (including ours) is at coping with randomness. It turns out that not only are people not rational in their decision-making, but they are completely fooled by the properties of randomness.

Its human tendency to take shortcuts (Kahneman). Rather than carefully evaluating risks, humans usually just look at the past and assume it reflects the future. Even when we extrapolate past statistics often times we only count events we remember, events that were significant. So frequencies of events measured in the past often times is used as the probability of an event in the future. Just like a fair coin toss. The probability of getting heads is calculated by repeating the coin tossing experiment a large number of times and dividing the # of heads by the total # of outcomes. So the probability of heads on each coin flip is 0.5. In most real world situations (like financial markets) you cannot necessarily rely on historical frequency  based probability. Just because an event has never happened in the past does not mean it wont happen in the future. This is because unlike a coin toss, the initial conditions keep on changing everyday and most of the times are unobservable or get too complicated. So any probability assigned to such an event in the future is based on belief and is referred to as belief-based or subjective probability.

Do you understand the weather ?

Just consider the weather predictions. A weatherman brings out a 30% chance of rain and you keep your umbrella tucked at your home. You go out and see the clouds thundering and water drops battering the ground and you curse the weatherman. But the poor guy has done no wrong actually. A forecast of less than 50% chance doesn’t mean no rain. It simply means that if weather conditions like today played over 100 times, 30 of those times it will rain.

Value of Money


This figure above shows two curves – blue and red. The blue curve has a constant slope (rise over run) whereas the red one rises less sharply in gains territory and falls more sharply in losses territory. The red curve indicates that losses hurt more than equal amount of gains. Kahneman and Tversky (see Prospect theory) found that generally people behave according to the red curve. Consider a situation where a person gains $100 and then loses all of them. The blue curve would suggest that the person got the same amount of happiness from winning that money as the sorrow she got from losing it and so she should be indifferent. But if that person has a value function similar to the red curve then the sorrow of losing will be greater than the happiness of winning.  Now this may seem strange because it seems like people do not care about the absolute wealth. This asymmetry can also be used to explain why investors still hold on to losing stocks despite it probably being a riskier proposition. It is because losses hurt them more psychologically. Loss aversion does not necessarily mean risk aversion. It is important to remember that this behavior is not absolute but depends upon the reference point and the change in wealth and varies for every individual.

Black Swan 

Most of us probably remember where we were, what we were doing when two planes flew  into the World Trade Center in New york City on the morning of September 11, 2001. The shocking images of the tall skyscrapers crumbling to the ground are probably vivid in most people’s mind. 9/11 is what we call a “black swan” in probability terms. It was unexpected and caused great destruction – low probability and high impact event. Any normal distribution would have failed to recognize the true impact of 9/11 not just in terms of loss in human lives but economic impact as well. yet we continue using normal probability distributions to quantify risk in many cases only because its simple.

Power law

In our daily lives, we often face situations where the errors are only minor. Consider a machine which we encounter at work. It looks like it breaks down only infrequently and with small defects. So you might come to the conclusion that you are never going to face a big headache due to the machine. Enter the power law!

This strange but simple looking distribution is probably one statistical tool we can use in this case. The mundane small failures are all on the left. You can see that these events have a high probability of happening. But have a look at the right tail – the dreaded uncommon events. it shows that the probability is low, but they can occur. This may not apply in all situations. But we have to be ready for the right tail. The nice looking yellow part is the one that’s going to matter. Be prepared for that!

Don’t discard common sense! 

In many situations however its better to follow a heuristic approach to decision making because its simply impractical to do a complex analysis. This approach is quite common with traders of financial assets. The problems at hand are very complex and so the mind takes short cuts without being fully aware of the process. But a smart trader is aware of this fact and learns from trial and error. We believe that trial and small error accompanied by a sound understanding of statistics is probably a better approach in such situations than relying on complex mathematical models because the real world situations really make the decision-making process too complicated due to incomplete information. Its very similar (but more difficult) to learning how to drive in traffic – switching lanes to reduce the time of travel gets better with trial and error.

Uncertainty is everywhere; Deal with it !

For most of us, death will be unexpected and for almost all of us (don’t wanna piss off those who claim to have risen from the dead) its a once in a lifetime event. But many of us do not go about worrying if the fateful day is going to be today. But it is our duty to ensure that after we are gone, the impact of our death is minimal on those who survive us. Failure to do so cannot be attributed to a sudden catastrophe; only to the flaw in our understanding of this great game of chance.


Science vs Faith


A guy sitting in a big dome shaped room staring intently into the eyepiece of a huge telescope; another person praying to his God hoping that it will give him some strength and bring happiness in his life. History shows that each thinks the other is foolish or misguided. But is it really so?

What is Science ?

A mind of a scientist is perhaps one of the most misunderstood things around. What actually is the motivation for a scientist to do what he does? Is it really about solving differential equations, writing pages of code or making complicated instruments? A casual observer might think so but it is certainly not the case.

The universe is an entity far bigger than a human mind can ever fully comprehend. We introduce hierarchies like an atom, a tree, a star or even an abstract concept such as love for clarity of thought but each time we move higher in complexity there are things we have not taken into consideration. But yes, the universe also happens to be such an awe inspiring place that it is almost impossible for a human mind to not wonder. Every child has looked up and wondered what the tiny twinkling lights that adorn our skies are; he has wondered about the beautiful colors of nature, about the wind and the rain, about the sounds and the smells. This is the essence of science –curiosity!

Is Science applicable everywhere ?

Agnosticism arises when humans have to deal with a universe that they do not fully understand. Science by its nature cannot prove or disprove anything. It can collect evidence supporting or rejecting a thesis but that is all it can infer. Science is an excellent method of acquiring knowledge. It teaches us how the world works. But science is inadequate when it comes to making decisions of what humans should do with that understanding. Human choices are extremely complex and humans do not solely rely on scientific methods to make decisions. Social phenomena are very different from laboratories in nature when it comes to internal interactions. For most parts, we can safely observe the phenomenon in nature without worrying about affecting what we observe. But in a social situation, our every observation, every move makes a difference to the whole picture; which in turn should feed back into our observations. Like financial markets; where our bets affect the market and people tend to act on instincts. Therefore it turns out that when there is inadequate information people do act on certain degree of beliefs. Faith has been important in development of arts across different human cultures and the role of arts in human evolution is undisputed. Faith is an important part of human psychology.

Science and Faith – A peaceful co-existence ?

But isn’t faith monopolized by organized religion? What if we put away the rules and customs of organized religion and look at what faith truly represents? What happens when a person prays to his God? Be it a Krishna or Jesus or Allah or even to an object or place, the intention is the same. One prays for getting strength to deal with one’s problems. Maybe for abstract things like hope, love, pain, this is what helps more than what science can do.

This is not a pitch for believing in God or religion. Both of us have never quite subscribed to the idea of a God in the generic sense. But there is definitely space for faith to co-exist with Science. We understand that we are trivial in front of the huge uncertainty that we witness in nature. Science and faith are our small efforts to understand these uncertainties. As Thomas Huxley said “The known is finite, the unknown infinite; intellectually we stand on an islet in the midst of an illimitable ocean of inexplicability. Our business in every generation is to reclaim a little more land, to add something to the extent and the solidity of our possession.”