A Robust Time Series Prediction Model Using POMDP and Data Analysis

One of the most important applications of information technology is to summarize data and predict new data based on existing values. For example, in stock market analysis, many investors use technical analysis tools to create a model that helps them in decision making. To minimize the uncertainties of the stock market, investors implement prediction models modified with their opinions. An ETF, which has a strong mutual connectivity between different portfolios, gets attention of the public by its low risk, intraday tradability and tax efficiency. In this paper, we propose a model in which investors’ opinion can be applied via Partially Observable Markov Decision Processes (POMDP), so that investors can intervene in the model to improve the prediction and make greater profit. Since an ETF has a strong mutual connectivity, we also use historical data to find out the relative changes between the chosen portfolios. This helps the model work better in POMDP structure. 


I. INTRODUCTION
The stock market is hard to predict.It has a lot of information and high uncertainty.Many factors like companies, politics, individual changes and even climate influence the stock market.For decades, it has been the desire of many investors to make use of information technologies to accurately predict the rise and the fall in stock market prices.
Despite difficulties in prediction, the size of stock market has been enlarged steadily.Stories of success in stock market and propagation of the Internet helped the growth of the stock market.Also, many attempts have been made to reduce risks and to increase profits.Recently, 'Robo-advisor' [1] with Artificial Intelligence Manuscript received July 1, 2016; revised May 15, 2017.
is spotlighted as a low-risk cyber investment tool.Owing to Robo-advisor, the number of investors increased recently.In fact, the global assets under management by Robo-advisor have increased by $10 billion in 2015.The steep growth rate is forecasted to last at least up to 2020.
The item of interest in this paper is ETFs (Exchanged-Traded Funds).An ETF is a pooled investment vehicle with shares that can be bought or sold throughout the day on a stock exchange at a market-determined price.Like a mutual fund, an ETF offers investors a proportionate share in a pool of stocks, bonds, and other assets [2].Since the beginning of ETFs in the early 1990s, it has grown with its intraday tradability, tax efficiency and low-risk.
In this paper, we propose a simple but powerful combined model with POMDP and data mining.In the case of ETFs, a strong mutuality binds portfolios together so that historical data can be powerful.A POMDP, which is based on MDP (Markov Decision Processes), can help reduce feasible risks and make profits.The method we proposed is also expected to work well in other prediction subjects which has multiple variables with relativity.
In our proposed model, we use various opinion values in certain ranges which easily allow investors to apply their observations.Furthermore, our model defines the policy according to the relativities between the chosen portfolios.This method organically connects many portfolios so that it can increase the profit.In addition to the lucrative investment, it improves the accuracy of multiple portfolios' prediction by using historical data.

II. RELATED WORK
There are a number of past works proposing reliable prediction models in the stock market.Many of the works have been implemented with popular machine learning approaches.
A Support Vector Machine (SVM) model assigns new examples into one category or another, making it a nonprobabilistic binary linear classifier.An SVM with timescale feature extraction [3] performed well.Also, an SVM model in volatile stock market [4] was tried.
Data mining is a good approach to finding out the tendency of changes in stock market.Scanning financial message boards and extracting sentiments expressed by individual authors was a novel challenge [5].Furthermore, techniques using Twitter sentiment analysis were also implemented [6].
Neural network is also a popular field of study in prediction modeling.Even though it may be hard to train a neural network, it is a very powerful tool to classify the complex dataset.Echo state network [7], Genetic algorithm with neural network [8] and other efforts [9,10] have also been implemented.Another work [11] is performed by Deep Recurrent Q-learning Networks (DQN).It contains 3 convolutional layers in the stock prediction network.But ETFs are difficult to implement with neural network due to its short history.
Former prediction models using POMDP were not implemented for ETFs but for general individual stocks.In [12], they implemented three strategies for long period, short period and planner module investigation.Another work [13] is implemented in POMDP that is similar with [11].
In [12], their model contains 27 different states that are derived from only a single portfolio.They suggest 3 different market states (bull, bear or flat) and investors have 3 different positions (long, short or off) in the previous 6 days.A decision follows the trends in accordance with price variation of +1%.Also, they adopt the fixed amount of rewards (10, 5, 2, 0, -2, -5 and -10) and adopt fixed amount of investments (M and 2M).

A. Partially Observable Markov Decision Process
(POMDP) Although most real-life systems can be modeled as Markov processes, it is often the case that the agent trying to control or to learn to control these systems has not enough information to infer the real state of the process.The agent observes the process but does not know its state.The framework of Partially Observable Markov Decision Processes (POMDPs) has been especially designed to deal with this kind of situation where the agent only has a partial knowledge of the process to control [14] A POMDP is an MDP where the agent does not know the real state of the process: the agent can only access an incomplete and partial observation of the state.A POMDP is defined by a tuple (S, A, Ω, T, p, O, r,  0 ) where: -S is a state space; -A is an action space; -Ω is an observation space; -T is the time space; -p are transition probabilities between states; -O are observation probabilities; -r is a reward function defined on the transitions; - 0 is an initial probability distribution over states.
If b is a belief state, after a transition (executing action a and receiving observation o) of the POMDP process, it becomes  0  where We model our ETF as a POMDP, where the price of each portfolio is modelled as a state in the POMDP, bringing the general concept of multiple states into our process.

B. A Novel Method for Time Series Prediction using POMDP and Data Analysis
We derived the relativities between the portfolios since the beginning of certain ETFs.Figuring out the relativities is important because it indicates how similar their changes are.First, we survey all changes between consecutive days of each portfolio, and then we calculate the similarity between each of portfolios by comparing the direction of changes.
where  , ( = 1,2, … , ) represents the price of portfolio i on the day d.  is the number of days that portfolios i and j increase or decrease together.We derived the correlations   among all the portfolios which can be described as The value   ≡     +  defined the change similarity between portfolios i and j.If   is 1, it means two portfolios i and j always move together in the same direction (increase/increase, or decrease/decrease).Next, we initialize the investment ratio distribution  ,0 (∑  ,0  =1 = 1) according to the market condition. , is a state space of day d.The stronger the prediction to increase, the higher the ratio is given.This procedure depends on economic data only and the ratio should be determined by the short term prediction.
is a total reward (income) and we choose one portfolio that gives the biggest income,  * (charge can be ignored in the case of ETFs) Then we add the opinion value to the chosen portfolio investment ratio.Subsequently, we adjust the ratio distribution [ 1,+1 ,  2,+1 , ⋯ ,  ,+1 ] according to the policy and probability from (3) where   is an action that represents the opinion value on the day d which is determined by investors.If   > 0, then the investors believes that portfolio  * will make a profit the next day again.According to (1), a belief of transition   , is a function of p, O and b(s).So, (1) can be rewritten as follows: is only determined by financial observation.It contains the amount of financial information, the tendency of how long the portfolio keeps its pace.The new distribution will be applied the next day.The investors will follow the above procedure except the initialization of the investment ratio distribution.The outline of our model is in Fig. 1.Since we brought the past data, we can obtain the correlations between the portfolios.After that, once the one portfolio is chosen as  * , we just follow the policy we made.IV.EXPERIMENT

A. Dataset
The experiment is performed with 15 portfolios focused on Korea ETFs during the period from March 31, 2016 to September 30, 2016.The list of portfolios is 「KODEX 200, TIGER KOSDAQ150, TIGER S&P500 Futures(H), TIGER EuroStocks50, KINDEX Japan Nikkei225, TIGER China A300, KOSEF 10year Government Bond, Kstar High Quality Corporate Bond, TIGER Short term, KODEX Gold Futures, TIGER Crude oil Futures, KODEX Inverse, KOSEF Dollar Futures, KOSEF Dollar Inverse Futures and KOSEF Short term Capital」.Each of the portfolio began on a different date, and the number of data is also different for each portfolio.In fact, KODEX 200 and KODEX Inverse have 1741 historical data while TIGER KOSDAQ150 has only 214 historical data.

B. Tests
The value   is changed according to the market condition.It has a range from +0.05 to -0.05 to avoid unexpected loss and speculation.Also we reduced   in unpredictable situations.If the situation makes it hard to predict the market changes, like just after Brexit, we hide our opinion to prevent any huge damages.Trading is performed every day except the weekends.
V. RESULT AND DISCUSSION Fig. 2 shows the top 6 prediction results during the period.The orange curve is the actual price of the portfolio  , and the blue curve is the predicted investment ratio which denotes the value  , .The above graphs are results that produced the most lucrative prediction.The scale of two curves in each graph is slightly different because our model predicts the tendency of the chosen portfolio, not an actual price prediction.Table I show that the prediction tends to be accurate when the portfolio has more data and smaller   change during the period.The portfolios shown in Fig. 2 are portfolios number 3, 6, 7, 12, 13 and 14.All of these have large number of data and stable relativities that gave better results.The average correlation is 0.172 and standard deviation is 0.426.
The correlation between the prediction and the actual price is derived from the below.

𝐶𝑜𝑟𝑟𝑒𝑙(𝑝𝑟𝑒𝑑
where   ̅̅̅,   ̅ are the mean values of the investment ratio and the actual price of portfolio i during the period.Having high correlation cannot tell us it is the most lucrative model but it is a good prediction model.Table II shows that the high correlation does not guarantee the high profit.10 of the 15 portfolios have positive predictionactual price correlation which means the prediction works well for those 10 portfolios.However, the other 5 portfolios have negative investment ratioprice correlation.Portfolio 11, although negative, was not bad for its correlation -0.03.Early prediction was applied to portfolio 10 and the model predicted results by a few days late.The two curves on portfolio 10 matches well by just shifting the price curve to the left.Portfolios 1, 8 and 9 could not avoid the local minima since Brexit.
Local minima could occur in several portfolios.If one portfolio is in a strong uptrend, the investment ratio of that portfolio increases fast and gets a high value.According to our method, the portfolio which has strong negative correlation with the uptrend portfolio should have a decrease in its investment ratio.Once it has very low investment ratio, it will not be selected as  * .This portfolio's investment ratio can be increased according to other portfolios that have positive correlation with that portfolio.In this situation, investors can re-initialize the investment ratio distribution.This POMDP model is robust in the market's dramatic change by performing reinitialization and determining the opinion   .
Actually, the model shows that local minima affect the distribution of  * during the period.Portfolios 2, 4, 11, 12 and 13 made several local minima and caused the other portfolios not to be chosen.Despite the existence of local minima, the model predicted well by using relativities as shown in the results of portfolios 3, 6, 7, 12, 13 and 14 in Fig. 2. In our result, the profit with the average |  | = 0.031 is comparatively low due to the local minima and high risks.Finding the proper   is important but is not the main issue in this paper.
From Table I and Table III, we can see that our model brought about a good correlation and a considerable profit.It shows that our model could be a good prediction model and a lucrative investment model too.

VI. FUTURE WORK
Our approach might not work well since the correlations between all the portfolios are not perfect.This imperfection came from the lack of data and the ineffaceable uncertainty of the stock market.It might be solved with a weekly holding strategy.Many regions from the result curves show that the model predicts changes after a few days later.Holding strategy can take positive effect on the investment.We will derive a trading-period dependency of the model by controlling the trading period.
Furthermore, if we can choose a proper set of ETFs, a better result can be expected.A proper set means that a set of ETF portfolios which have a definite strong correlation between portfolios.

VII. CONCLUSION
Integrating a simple POMDP algorithm and Data mining, this study produces the good performances in both prediction accuracy and profit since the average correlation is 0.172 and the average profit is 0.697.It is an effective approach to time series prediction with the multiple portfolios.Even though we cannot show the correlation between the effectiveness of our method and the number of portfolios, but it shows that the relativities between the multiple portfolios make the prediction much better.Also it shows that the amount of data can affect prediction.
This model can be applied to the subjects which have relativities between the items.In weather forecasting, the Met Office has a lot of historical weather records with meteorological information.For instance, using temperature, we can establish a model with the adjacent locations and their temperature.There are several climate factors such as humidity, wind and cloud.Moreover, there are temperature relativities among the adjacent locations and all the climate factors can be utilize as an observation.Whenever we determine a policy  * , it is possible to do the large scale weather forecasting.

Figure 1 .
Figure 1.The overall structure of our POMDP model -A new investment ratio is distributed according to the correlations   which aregiven by historical dataset.

Figure 2 .
Figure 2. The result graphs of 6 portfoliosthe orange curve is the actual price of the portfolio and the blue curve is the predicted investment ratio , ) = ∑ ( , −  ̅̅̅̅)( , −  ̅

TABLE I .
THE NUMBER OF DATA, CORRELATION CHANGES BETWEEN PORTFOLIOS, THE PREDICTION ACCURACY AND THE NUMBER OF DAYS AS J * DURING THE PERIOD.|  | = 0.026.

TABLE II .
THE DIFFERENCE BETWEEN THE CORRELATION AND ACTUAL PROFIT.

TABLE III .
THE DIFFERENT RESULT BY CHANGING   .