The “Big Data” Solution For Wall Street
Co-Founder & CTO of I Know First Ltd. With over 35 years of research in AI and machine learning. Dr. Roitman earned a Ph.D from the Weizmann Institute of Science
While the financial markets are very intricate systems, determining the best components of a successful portfolio does not have to be. Investors are familiar with the saying, “buy low, sell high” but this does not provide enough context to make proper investment decisions. Every investors dream is prior knowledge of the direction of the market before it happens. Although this is incredibly difficult to do accurately and consistently, it is now possible to create financial market forecasts with algorithms. The quickly growing trend of financial advisors utilizing advanced algorithms, is part of a much larger trend of our entire society using “Big Data” solutions for a diversified pool of needs, including predicting credit risk, demand for goods and services, querying social networks to gage market sentiment, machine readable format company reports, discounts and advertising targeting as well as many more applications. In fact, the Chinese government and IBM (IBM) have teamed up utilizing Big Data to finally tackle the far-east nation’s austere pollution problem.
By incorporating popular types of convergence averages and moving averages that have been traditionally used to forecast assets for many years with more sophisticated technology and genetic algorithms, professionals are now capable of building complex and intelligent algorithms that can make these predictions more accurate and efficient. Even when financial bubbles and market corrections lurk, a proper understanding of how the markets function plus a vigilant risk management strategy has always been necessary to survive in the financial wilderness. However, investors today have the option to take advantage of state-of-the-art algorithms in conjunction with traditional forms of analysis in order to enhance portfolio performance, verify their own analysis and respond to opportunities faster.
This overview is intended to further divulge this mysticism surrounding Big Data analytics and provide insights about the potential return on investment analytics can enable for those who embrace these capabilities. Financial professionals that step ahead of the curve today with avant-garde strategies such as these will be the definitive beneficiaries of predictive analytics, leading Wall Street with a much more proactive and cost-effective approach of algorithmic trading.
What is “Big Data”
Big Data solutions are used for data sets that are too large and complex to manipulate or interrogate with customary methods or tools. The importance of this field has increased because it gives us better insight of our structured and unstructured data, leading to potentially more accurate analyses, which may lead to more confident decision-making. In 2001, industry analyst Doug Laney who currently with Gartner articulated the now conventional description of Big Data as the three V’s of big data: volume, velocity and variety. Figure 1 illustrates the three V’s of Big Data in a Venn diagram.
The United Parcel Service (UPS), for example, has been using Big Data for decades by tracking package movements since the 1980’s. While today, the company is using much more advanced processes to analyze an average of 39.5 million tracking requests from customers per day, they have also created arguably the largest operations research project in the world using Big Data. This initiative is called On-Road Integration Optimization and Navigation (ORION) and so far the project has already led to savings of more than 8.4 million gallons of fuel by eliminating 85 million miles off of the daily route in 2011.
Wall Street also stands to benefit from Big Data analytics by using advanced algorithms to track and predict the financial markets, as does the I Know First self-learning algorithm. Although many investors are unaware of how they can apply algorithmic trading to because they may be thinking about a completely different form of algorithmic trading.
Quantitative Trading vs. HFT
It is important to recognize that there are two types of algorithmic trading, which are very distinguishable. Generally when someone mentions algorithmic trading, the average investor automatically assumes high frequency trading or HFT. The advantage of HFT is to be quicker than the rest of the market, but only a select group of traders can utilize these systems and there are extensive consequences that affect the entire market. This system is not “smart” and does not postulate any real valuable insight for investors, as it only blindly follows short-term trends. HFT is also ethically debatable.
The second unique form of algorithmic trading is referred to as quantitative trading or long term algorithmic trading. This type of black box trading is completely different from HFT in that instead of using aggregated data dating back to five minutes of history to create a one-minute projection, quantitative trading such as the I Know First algorithm, analyzes the structure and the trends in the market, finds predictable patterns, and creates machine-derived forecasts. The differences between the two are described in further detail here.
Chaotic + Efficient Patterns = Complex Systems
In general, there are two common fallacies concerning how to determine which assets would provide the highest return while simultaneously limiting exposure to the potential risk involved. The first fallacy is that the markets are entirely efficient, and therefore unpredictable. According to the efficient market hypothesis, the markets instantaneously absorb the latest information as well as stock prices and then adjust accordingly. Under this assumption, no stock can be a better than another, as both are efficient where all investors have impeccable information available. Consequently, it would therefore be impossible to consistently realize returns beyond the average market return on a risk-adjusted basis; assuming that information is available each time an investment is made. This obviously does not truly reflect the reality, however neither does the opposite, where the markets are completely chaotic. The focal edict supporting the chaos theory, is the fundamental notion that small incidences drastically affect the outcomes of ostensibly extraneous events. In such a chaotic market, profits and losses would consistently sum up to zero over time.
The financial markets are not 100 percent efficient and they are not 100 percent chaotic but can actually be more accurately understood under the complexity theory because each market has a systematic and random component. The fundamental structure of these financial markets is supported, however the involvement of abundant multifarious investors with different strategies, experience, amount invested and goals, all while accommodating the convoluted dynamics of interdependencies and feedback loops between system elements.
The Stock Wave
I Know First which uses a self-learning algorithm to predict over 3,000 markets, recognizes this as stock waves. By analyzing the outcomes of a multitude of trades, systematic trend patterns often emerge, which helps us better understand the past as well as prospects for realistic market forecasts. There are three different regimes the market can alternate from – positive feedback, negative feedback and randomness. The goal of such analysis is to recognize whether a stock is mean reverting or trending, and on what time scale. This can become very difficult, as these regimes may be present simultaneously on different time scales. The goal of such analysis is to recognize the regime the market is at now, whether a stock is mean reverting or trending and at what time scale. Correctly analyzing this aspect is absolutely necessary for making accurate market predictions.
Positive feedback loop behavior can be characterized as when there is a positive effect in one variable, it increases the other variable, which in turn also increases the original variable as well. This leads to exponential growth in the system, moving it out of its equilibrium and eventually leading to a collapse of the system. Conversely, a negative feedback loop has a stabilizing effect, the system responds to a stress in the opposite direction. Figure 2 depicts how positive and negative feedback loops work.
Generally when an asset is performing well, trader’s notice and act upon this momentum. In a chain reaction, more traders begin to buy this particular asset. While this is not a trend or an ephemeral arbitrary surge, it is actually a recognizable pattern that allows us to predict the behavior of such systems. Terms such as “overbought” or “oversold” are implied in the stock wave approach. The interaction of both feedback loops is known as dynamic equilibrium or that an asset is trading around a certain price level. The price will constantly overshoot the real value in both directions. We can see positive feedback loops and negative feedback loops in the S&P 500 YTD in figure 3.
Finding The Global Minimum
While I cannot speak on every algorithm meant to predict the market, the I Know First market prediction system is based on artificial intelligence (AI), machine learning (ML), as well as utilizes elements of artificial neural networks and genetic algorithms. Machine learning provides distinctive insight to our comprehension of market dynamics and behavior. The algorithm has a built-in general mathematical framework that generates and verifies statistical hypotheses about stock price development. Machine learning tools such as artificial neural networks make this prediction system self-learning, and consistently determined to become more precise. New data is added daily to the 15-year database, where it runs a learning and prediction cycle that creates predictions for the short and long term, as can be seen in Figure 4.
This framework is used to generate initial testing models over a test sample of data. The goal of this phase is to validate the accuracy of the algorithm as well as to fine-tune the fitness function, which represents the actual goal of the algorithm expressed as a mathematical function. When the algorithm finds the global minimum of the fitness function attached to one of the models generated, it fulfills its goal.
From the mathematical perspective finding a global minimum is a very complex task and bears a risk of finding a local minimum, which seems to be global from the perspective of surroundings of the point, but there can be found points less than that one. This situation is illustrated in figure 5.
To increase the chance of finding the global minimum, we need to combine multiple searching procedures. When the algorithm proves its ability to generate valid results over the sample data, we can then use it to real data analysis. Every run the algorithm improves its predictive ability, as it generates new models and verifies them to the fitness function, therefore providing better and better results.
Every investor has their own strategy, such as particular fundamentals they tend to be fond of and level of risk they are willing to accept. These types of analysis alone are becoming outdated and more effective investing tools have become evident to lending a helping hand in increasing portfolio performance. Hedge funds and investment houses have already recognized the benefits of these advanced mathematical models as they play a significant role in their ability to perform, regardless of the overall market environment. Computer-based algorithms, which can analyze many stocks simultaneously and determine quantifiable founded objective predictions, are becoming increasingly more popular to investors as an improved strategy for optimizing returns and mitigating prospective risk. There still is no flawless way of picking stocks but utilizing advanced algorithms based on Big Data predictive analytics in conjunction with reasonable risk management and fundamental analysis can potentially help improve portfolio performance for retail and professional investors alike.
Business disclosure: I Know First Research is the analytic branch of I Know First, a financial services firm that specializes in quantitatively predicting the stock market. Joshua Martin, an I Know First Research analyst & Founder Dr. Lipa Roitman, wrote this article. We did not receive compensation for this article and we have no business relationship with any company whose stock is mentioned in this article.