Keep Calm & Trust Your Data: AI And Stock Market Irrationality

I Know First Research Team LogoThis article was written by the I Know First Research Team.

One of the most common themes you find in many mythologies across the globe is that of order and chaos. The world as we know it, in all its orderly (or at least more orderly than not) glory, is emerging from the state of complete and utter entropy. This can be seen as a metaphor for many things, from organic life coming into existence in the primordial broth to the way the first humans explored their environment, rationalizing their experiences in ways that made sense in the existing system of knowledge. Rationalizing is the keyword here: it does not take a stretch of the imagination to view this post-chaos world as being the rational one, the one that makes sense, while its pre-chaos version would be absolutely irrational.

Interestingly, a similar theme of rationality versus irrationality can be noted in the modern debate on what makes the stock markets behave the way they do. Here, rationality is one of the assumptions at the core of the Efficient Market Hypothesis (EMH). The key idea of the EMH is that the stock prices fully reflect all the information available to the investors, and all investors are fully rational in their decision-making. There are various implications and consequences here, all stretching beyond the scope of this article and over the horizon, but what matters for now is this idea of investors as cool-headed number-crunching returns generators.

(Source: Wikimedia Commons, public domain)

In the irrational corner, in the meantime, is the discipline of behavioral finance. This school of thought tends to view investors as humans first, and investors second. In other words, while all the appropriate number-crunching and strategizing are expected, the decisions of the investors can also be driven by the quirks of human psychology rather than the most recent market data. The said quirks are, again, diverse and numerous enough to write a whole lot of articles, but for now, the key takeaway is that there is more to market behavior than rational decision-making.

So what does this debate mean for investors themselves? We will take on this question, but first, let us take a quick step back and look at what is implied in rationality and the lack thereof, just for the sake of more clarity in the discussion.

Rationality And Irrationality – What’s The Buzz About?

Given the prominence of the EMH as one of the pillars of stock market theory, it is no wonder that a lot has been written on the rational, irrational and normal behavior in terms of investment decisions. So, without lingering on this for too long, let us take a look at what each of these words implies.

Rationality, for example, is most frequently associated with a behavior that is very much goal-oriented, consistent, coherent and based on the assessment of all available data. Academically, it is viewed as a behavioral model that seeks to maximize the utility for the individual. Maximum utility does not necessarily have to mean maximum benefit: for example, for a risk-averse person, an investment that generates lower returns at lower risks is more rational than investing into all things obscure and exciting.

The subjective nature of the notion of utility and, by extension, of economic rationality, however, can be taken to extremes that leave no room for further discussion. For example, a choice system where you buy scores of Apple stocks and set them on fire can be seen as rational as long as, well, that is what you are inclined to do. Not setting them on fire would be viewed as irrational here, but not if your plan is to go for a long-term blue-chip investment – in this case, the rational option would be to keep the matches as far from them as possible. As long as we allow your subjective I-wants to act a defining feature of how you rank the choices and outcomes available to you, and as long as this hierarchy is internally-consistent, the option at its top will be the rational one even if it includes a lighter and some gasoline.

Thus, in this article, we will take the freedom to view rational investment behavior as the one that seeks to obtain the highest level of returns possible for the individual risk tolerance level based on all information available to the investor. Irrational behavior, thus, would be seen as its opposite – the behavioral model that prioritizes options delivering suboptimal returns for reasons that do not logically follow from the available data and historic knowledge. These are both based on the assumption that our readers would want to maximize the bang they get for the buck.

Now, with this aside, let us dig into the issue of irrationality on the market.

Where The Wild Things Are: Rationality And Stock Markets

The EMH as a theory is a hotly-contested one these days. Many of its assumptions, such as the idea that stock prices are essentially on a random walk, and it is thus impossible to consistently beat the market, are being put into question (throw in some link or something). Among them, as noted earlier, is the idea that investors are fully rational.

For example, a paper by Edwin J. Elton, Martin J. Gruber and Jeffrey A. Busse demonstrates that in the late 1990s, the S&P 500 index fund investors have consistently underperformed in comparison with the index itself. While the returns, as the authors show, were easily predictable, the investors proceeded to pour a large share of the money into the worst-performing funds where even simple strategies based on calculating the previous returns or the share of the index fund in the total cash flow for the sector did better. Such less-than-behavior, they claimed, could only partially be explained by the funds’ marketing efforts.

Dalbar Inc., a Boston-based research and audit company specializing on the financial industry, has demonstrated a similar scenario play itself out in its 2015 report. It found investors persistently underperforming against the S&P 500 index due to irrationalities in their decision-making. Interestingly, it also presented its views on the psychological mechanisms that resulted in such poor performance, but we will keep those for the next section.

A paper by Al Mamun, Abu Syeedb and Farida Yasmeen, published in the Journal of Economic & Financial Studies in 2015, paints a similar picture and concludes that at the end of the day, investors are just your regular human beings. Some of their decisions are rational, some are taken in the heat of the moment. They are more likely to credit themselves with any gains, but in most cases, will prefer to blame failure on anyone or anything else, from the broker to the overall state of the market.

Now, these are, of course, just a few from the huge number of studies suggesting that investors are not as rational as some expect them to be. For those interested in further reading, we recommend Al Mamun’s paper, which includes a rigorous literature overview. For those less academically-inclined, it is now the time for us to take another step further and look at some of the psychological mechanisms that can lead to irrational decision-making.

The Burden Of Humanity?

We, humans, are an unruly, but amazing crowd. The way our brain function includes a whole variety of tricks that are designed to improve or speed up our cognitive process but can end up ruining our day.


For example, we tend to approach new things based on our prior experience, whether it comes to finding familiar patterns in things where they are not really present (ever seen those cool shapes in the clouds?) to more general cases, where a whole new scenario is being reviewed via the prism of some events that have already happened. This is, on the one hand, perfectly fine and understandable, if not statistically-speaking, then at least in terms of how formative our background is to who we are. However, on some occasions, a new scenario requires a new response, and its lack leads to failure. This is, in fact, one of the things that Dalbar has pointed out in its report: some of the irrational behavior results from investors’ over-reliance on experience.

Other problems include expecting high reward with virtually no risk, tunnel-visioning on certain aspects of the situation at the expense of a full picture, mental accounting on the risk-management front, diversification mistakes, excessive optimism, fear of regret, too much trust to media reports and following the crowd. The latter is especially interesting, since a whole school of through actually holds the so-called wisdom of the crowd in quite a high regard. However, as we discuss in more detail here, the decision to act on your conformism is not always the wisest one.

Mental accounting is another interesting example of how our cost-benefit assessment is not always purely rational. Imagine you were on your way to view a movie by your favorite director, with the ticket at $10. On your way to the movie theatre, you lost the $10 you put away for this special occasion in a pocket, but there is still enough money in your wallet – would you still buy a ticket? Now, let us say you forgot your ticket at home – would you still spend $10 at the box office? Many people said yes to the former, but no to the latter, while their real losses in these scenarios are equal. In investment, this can lead to, among other things, investors creating arbitrary “money jars” for safe money and funds to play around with, going too risk-averse on the former and taking unnecessary risks on the latter.

Fear of regret is another major factor, leading to a sunken cost fallacy. In essence, here things boil down to the fact that we, humans, tend to get unhappy when proven wrong. Thus, if we have placed our bet on a losing stock, we may hesitate to sell it, expecting it to rise back up as it continues to grow our losses. We also tend to pay more attention to our losses than to making the most out of our gains. We tend to overly pessimistic when the market sinks and expect too much when bulls reign.

The mental tricks we play on ourselves are pretty diverse, as you can see, and they can all lead to serious losses. So is there anything we can do to avoid these traps? Yes, we can – and this is where the machine comes in to rescue the human.

Machines To The Rescue: AI And Stock Market Irrationality

In this day and age, artificial intelligence is one of the buzzwords you hear almost every day. Behind it is a technology that, at least when it comes to its certain subsets, seeks to approximate human cognition through complex statistical mathematics. This, of course, has its own downsides, but also opens new opportunities.

Let us take market bubbles, for example – the scenarios in which the price of a stock surges for no specific reason other than investors getting carried away with this particular financial instrument. This is a simplification, of course, but human emotion and psychology is a very important factor in the formation of a bubble. AI algorithms, with their complete lack of emotion, can be quite good with identifying those. For example, an AI trained by an Israel-based company called I Know First has successfully anticipated the Apple bubble of 2012. Using historical data as well as the latest market updates, the AI estimated the point of equilibrium for the stock and warned the company’s clients when the price began to dangerously overshoot it.

The algorithm has been designed with the use of Deep Learning, while also drawing upon genetic algorithms technology. The former allows it to crunch through troves and troves of data, relying on artificial neural networks to analyze and model the market dynamics, and the latter makes the algorithm aware of its own successes and failures, increasing the accuracy of its predictions with each running cycle. Furthermore, the neural networks the AI uses can be re-configured to conduct simulations of various scenarios to produce what-if analysis.     

The AI creates forecasts for over 10,000 financial instruments, including stocks, ETFs, currencies, commodities and cryptocurrencies. The output is presented as a heatmap with two key indicators: signal and predictability. The signal represents the performance of each asset on the forecast as compared against the other financial instruments on the forecast. The predictability indicator shows how well the algorithm has been predicting the stock so far. The time horizons for forecasts range from 3 to 365 days, covering short, mid- and long-term outlooks.

The algorithm boasts a high accuracy rate that makes it possible to profit even from fully automated trading, relying on a bot that simply buys and sells the stocks with the highest expected returns and predictability. In fact, a recent evaluation demonstrated that a portfolio built on I Know First predictions beats the one based on the S&P 500 index.

AI And Stock Market Irrationality
AI And Stock Market Irrationality