The Conceptual Framework of Applying ML and AI Models to Analyze and Forecast Financial Assets

This article was written by:

Sergey Okun  Sergey Okun – Financial Analyst at I Know First, Ph.D. in Economics.

Eugene KalaidinEugene Kalaidin – Professor, Dept. of Mathematics and Computer science, The Financial University under the Government of the Russian Federation, Ph.D., D. Sci. (Habilitation) in Physics and Mathematics.


  • Knowledge significantly decreases the speculative risk of investment
  • ML and AI technologies allow us to get relevant knowledge about the financial market
  • Information asymmetry is a key factor in getting the arbitrage return
  • Models of nonlinear dynamic systems allow us to correctly evaluate financial assets by determining the backbone behavior of assets

The main question for each market is the formation of knowledge based on the asymmetric information from buying and selling sides. This knowledge significantly decreases the speculative risk of investment leaving a fundamental part of changing a financial asset value and its derivatives for additional analysis, which attracts minority shareholders and institutional investors to the financial market.  

There is a natural question, does the market form knowledge, and does the market have an interest in uncovering knowledge? Market experts can discourse for a long time for searching for the answer to this question. From our side, we believe that the answer could be obtained not from experts, but from the market, i.e. ask the market “what do you know?”.       

In this article, we do not try to attract new members to a camp of individuals using machine learning models. But we try to find a reasonable solution applying this revolutionary mathematical instrument in the analysis of financial assets.   

So, knowledge is an evolutionary result of trial and error, especially in any kind of human activity, and the financial market is not an exception. If we correctly receive this knowledge then there is a high probability that this knowledge will be relevant for some time while new knowledge is being formed. It is necessary to understand that knowledge can change. Here we speak not about abstract knowledge that allows constructing beautiful self-contained theories (thinking about fundamental mathematics that is based on a system of axioms), but we speak about a real word where axioms could change before a theory is completed and people ask for a new theory, which is based on new axioms and prove those previous axioms completely wrong or require some adjustment to be an actual one. In this context, we understand axioms as a set of elements properties which is a base for constructing a non-controversial theory.  For example, we can remember Arrow’s Impossibility Theorem that describes an opportunity of market manipulation in a dictator’s favor.

Thus, in order to survive the market evolves for funding new mechanisms of efficient functions, forming new instruments and playing rules with them. Financial instruments are more or less amenable to analysis and their properties are predetermined, but the rules, at least in some of their parts, are formed evolutionarily and, as a result, have information asymmetry. Of course, over time, this asymmetry uncovers that gives a fast profit to the owner of this information.

Such a passive approach is probably unsatisfactory for most market participants, and we would like to assure an individual investor that there is a way to get an arbitrage return. We need to underline that by the arbitrage return we understand getting a higher return to compare with alternative financial assets having the same risk level.         

Who can win in the capital market? Firstly, the one who does not lose money where it is obvious, the one who goes to professionals. But if you can do something well, it does not mean that there is not someone who can do it better. So, there is a competitive advantage that not only prevents you from losing your money but also allows you to outplay other players in the capital market. Here we are not saying that you can do some standard things faster, but that you can get knowledge faster about what is happening around you. Modern information technology could help in solving this problem, and especially AI and ML technologies.


ML models
Source: (

Interested people have a question, “If everyone has or could have the knowledge, then how can I beat them?” For example, despite the fact that children can have the same teachers in schools, eventually, all of them have different levels of knowledge. Possibly, not everything is simple in the world, and there is not a clear answer to the question “who should we trust our money with”. We make a choice not only in the area of investment but in more important everyday situations. We need to choose a doctor, when we are sick, find a house for long-term accommodation, choose a safe car, or determine a person with whom we want to spend all our lives. So, we make decisions based on our human values and our knowledge that was gotten from an information stream that touches our problems.    

It should be noted that complex investment models make it possible to partially reveal the asymmetry of the capital market, and not all experts clearly understand them. For constructing models that are later used in the learning process of artificially intelligent systems, we choose the fractal market concept, which can be interpreted as a self-similarity behavior on different time scales. Sometimes this approach is called the Fractal Market Hypothesis. The dimension of the phase space of a dynamic system that forms a time series of returns is a key characteristic in the construction of artificial intelligence learning systems. In simple words, if a ladybug, crawling on the surface of an isomorphic two-dimension space, does not know that there are spaces of higher dimension, then it will not fly. On the other hand, it will be isomorphic to a three-dimensional sphere in a space of any dimension greater than three. The phase space of a dynamic system works the same way, where there is a single attractor that determines the properties of a dynamic system. The Lyapunov exponent describes the trajectories divergence property of a dynamical system, that enables us to evaluate the forecast accuracy at different time horizons. In this case, the property of strong dependence on the initial conditions, which is common for nonlinear dynamic systems, is not a fatal problem anymore in the stock market forecasting, and the set of predicted values becomes a definable parameter of the problem.

So, let’s return to the stock market and investment. Each minority investor has two goals: save money (as a minimum), and increased its amount (effectively neutralizing inflation). A significant increase in investment return is more luck than a rational, calculated strategy. In this case, we mean that there is a low probability to constantly get higher (arbitrage) returns.

Why do we get returns when we work with financial assets? The theory provides the answer in the following form: return is a premium for the risk. When there is low demand, and consequently the price is low, we buy an asset to sell it at a higher price in the future, when demand increases. Of course, the reason, that the demand will increase, is the attractiveness of this asset in the further future. Also, the right time to sell this asset will bring the maximum return, after which the profitability of this asset will most likely go flat and a downward correction will occur.

Thus, evaluation of risk and return is the key component in the making of investment decisions. Here we need to refer to a classical portfolio theory, that was offered by Harry Markowitz, which is based on the Efficient Market Hypothesis, namely the random walk model of asset returns, and as a result, the ability to use mathematical statistics as an instrument for the analysis. The most important assumption is that the return can be determined as a random variable. Otherwise, variance or other moments of random variable do not make sense, since the corresponding integrals, which determine these random moments, become divergent, and this is the moment where theory ends.

As a result, among the capital market models, models of nonlinear dynamic systems appear, which determine the backbone assets behavior, leaving the possibility of random fluctuations but not giving them a decisive role in the return evaluation. Making decisions about investments becomes more conscious and the risk in obtaining return moves from the plane of statistical descriptors to the predictability plane of the asset trajectory behavior in the phase space of a dynamic system.  

I Know First Algorithm – Seeking the Key &  Generating Stock Market Forecast

Basic Principle of the "I Know First" Predictive Algorithm

The I Know First predictive algorithm is a successful attempt to discover the rules of the market that enable us to make accurate stock market forecasts. Taking advantage of artificial intelligence and machine learning and using insights of chaos theory and self-similarity (the fractals), the algorithmic system is able to predict the behavior of over 10,500 markets. The key principle of the algorithm lays in the fact that a stock’s price is a function of many factors interacting non-linearly. Therefore, it is advantageous to use elements of artificial neural networks and genetic algorithms. How does it work? At first, an analysis of inputs is performed, ranking them according to their significance in predicting the target stock price. Then multiple models are created and tested utilizing 15 years of historical data. Only the best-performing models are kept while the rest are rejected. Models are refined every day, as new data becomes available. As the algorithm is purely empirical and self-learning, there is no human bias in the models and the market forecast system adapts to the new reality every day while still following general historical rules.

ML Models: Conclusion

The formation of knowledge based on asymmetric information is a key component for taking advantage of the financial market. Modern ML and AI technologies allow exploring relevant market knowledge to get an arbitrage return. The financial market is a nonlinear dynamic system where factors, which determine the system behavior, are not constants and change all the time. The classical theory of efficient market can not provide an efficient framework for statistical models to analyze the behavior of a nonlinear dynamic system. At the same time, the capital market models based on the Fractal Market Hypothesis allow analyzing the behavior of nonlinear dynamic systems and evaluating financial assets by determining the backbone behavior of asset price, and leaving the possibility of random fluctuations but not giving them a decisive role in the return evaluation.