Solving problems of the oil and gas sector using machine learning algorithms

DOI: https://doi.org/10.46544/AMS.v26i2.11 Abstract The article describes the tasks of the oil and gas sector that can be solved by machine learning algorithms. These tasks include the study of the interference of wells, the classification of wells according to their technological and geophysical characteristics, the assessment of the effectiveness of ongoing and planned geological and technical measures, the forecast of oil production for individual wells and the total oil production for a group of wells, the forecast of the base level of oil production, the forecast of reservoir pressures and mapping. For each task, the features of building machine learning models and examples of input data are described. All of the above tasks are related to regression or classification problems. Of particular interest is the issue of well placement optimisation. Such a task cannot be directly solved using a single neural network. It can be attributed to the problems of optimal control theory, which are usually solved using dynamic programming methods. A paper is considered where field management and well placement are based on a reinforcement learning algorithm with Markov chains and Bellman's optimality equation. The disadvantages of the proposed approach are revealed. To eliminate them, a new approach of reinforcement learning based on the Alpha Zero algorithm is proposed. This algorithm is best known in the field of gaming artificial intelligence, beating the world champions in chess and Go. It combines the properties of dynamic and stochastic programming. The article discusses in detail the principle of operation of the algorithm and identifies common features that make it possible to consider this algorithm as a possible promising solution for the problem of optimising the placement of a grid of wells.


Introduction
Recently machine learning algorithms began to spread to all areas of the industry, and the oil&gas sector was not an exception (Meng et al., 2020;Roscher et al., 2020;Goodfellow et al., 2016). And although in MLsystems, it is rather difficult to interpret the decisions made by them, in contrast to mathematical models and expert systems, they are simpler to implement and practically have no limitations as the complexity of modelling increases (Piscopo et al., 2019). At first glance, this statement may seem dubious: how can the implementation of machine learning make development easier and not complicate an already complex system? The reason is that machine learning allows you to get the desired result without any knowledge of the mathematical model of the analysed process -algorithms can be unified for any field of application (Toms et al., 2020;Blišťanová et al., 2014;Blišťanová et al., 2015). In addition, most machine learning algorithms have already been implemented in software libraries in various programming languages (Kuric et al., 2011). The main condition for implementation of Machine learning on the tasks is to have data from past calculations of those tasks. This data is then converted into a training dataset for machine learning algorithms (Katysheva et al., 2019;Poluyan et al., 2018).
Let us consider the range of tasks in the oil and gas sector that are currently being solved by machine learning tools: -prediction of oil production for individual wells and total oil production for a group of wells, forecast of the base level of production; -prediction of reservoir pressures and mapping tasks; -estimation of the effectiveness of ongoing and planned geological and technical measures (geological and technical measures); -estimation of well interference; -estimation of fail-safety of pumping equipment and pipelines; -classification of wells according to their technological and geophysical characteristics; -optimisation of well grid placement.
At first, it is necessary to clearly define which data can most accurately describe the modelled process and will be included in the structure of the training dataset. Depending on the type of task, the dataset structure can be horizontal or vertical (Lei et al., 2019).
The horizontal structure is used for the tasks of assessing the interference of wells and predicting the total oil production for fragments of the field. This structure has a number of disadvantages: a directly proportional increase in the number of dataset fields with an increase in the number of studied wells and sensitivity to information gaps for individual wells.
The vertical structure is suitable for various classification tasks and tasks of predicting oil production for individual wells (Arteeva and Zemenkov, 2020;Vankov et al., 2020). The disadvantages of this structure include the sensitivity to errors in the data on injection and production wells.
All data included in the training dataset can be divided by type into the following groups: -frequency of data acquisition; -characteristics of production wells; -characteristics of injection wells; -parameters of extractive facilities; -geological and hydrodynamic characteristics of the reservoir.
Vertical and horizontal datasets usually have the same characteristics. However, in the horizontal one, characteristics for each well are located in one sample.
Next, we will consider the tasks of the oil and gas sector in more detail.

Material and Methods
The prediction of total oil production or oil flow rate using neural networks is implemented in the NNet software, which was developed to solve a whole range of tasks in oil-field geology (Yuzevych et al., 2019). As input data for the neural network, data on the reservoir and bottom-hole pressures, pump immersion depths, dynamic level, technological characteristics of equipment, geological characteristics of the reservoir are used. The output parameter is the daily flow rate of an individual production well. The paper (Ivanenko, 2014) presents the results using a linear model, a simple multilayer perceptron with one hidden layer, consisting of 2 neurons and a sigmoidal activation function for the output neuron and networks of RBFN (radial basis function network) and GRNN architectures (generalised regression). In this example, the neural network predicts production for each well separately, and then the total values are compared. In Fig. 1, you can see how much the predicted values of the total oil production diverge from the real values when using various neural network architectures. The task of mapping is rather extensive due to the fact that when modelling a field, a large number of different maps are built: structural maps of the top and bottom of the reservoir, maps of porosity, permeability, water saturation, etc. Usually, this task is solved by spline interpolation methods. In (Ivanenko, 2014), a comparison of N-dimensional spline interpolation and neural network predictions for constructing contour maps is considered. 80 irregularly spaced interpolation nodes are plotted on a unit square, and the values of the desired function are calculated in them. The coordinates of interpolation nodes are supplied as a training dataset for the neural network, and the calculated values of the function are the output parameter. For predictions, the coordinates of the nodes calculated on a regular grid are taken. Figure 2 shows the calculation results. Thin lines are built using neural network predictions and thick ones -by spline method.

Fig. 2. Contour map constructed by spline interpolation and neural network
It can be seen that the results are quite close. This means that neural networks with the right architecture and hyperparameters can become a serious alternative in solving the problem of building maps.
The effectiveness of ongoing and planned geological and technical measures is usually assessed by comparing baseline indicators with actual ones (Akatov et al., 2019). Basic indicators are understood as indicators that can be obtained during field development by the initial method -before geological and technical measures. The most common methods for calculating production baselines are displacement characteristics (integral models) and decline curves (differential models) (Saga et al., 2020). With the neural network approach, this problem is essentially reduced to the previously considered regression problem of predicting the total oil production (Ivanenko, 2014). In this case, the parameters of the geological and technical measures carried out, for example, the depth of immersion of pumps, are added to the training set. An important advantage of the approach is the ability to predict the baseline production rate not only for the group of wells but for separate ones.
Estimation of the degree of interaction (interference) of wells is another important way to assess the effectiveness of geological and technical measures. To estimate the interference, methods of numerical modelling of the formation with wells interaction coefficients are usually used (Pástor et al., 2020). Let's move on to the neural network approach again -this is the same regression task as with oil production prediction. The parameters of a group of wells at once can be used as input features: production and injection wells. By changing these parameters, you can see how the final production changes. In (Ivanenko, 2014), a training dataset is considered, the input parameters of which are injectivity of injection wells, water and oil flow rates for producing wells, as well as the operating time and current time. Figure 3 shows the degree of interaction between the low-rate production well 1029 and the injection well 2026 from the neural network. The abscissa axis shows the injectivity of the injection well. On the ordinate axis, the values correspond to the response of the production well 1029. The injectivity value equal to zero corresponds to the real value of the well injectivity in the selected time interval: -100 corresponds to the complete shutdown of the well, and 100 corresponds to a two-fold increase in injection.

Fig. 3. Prediction of the interference for wells 1029 and 2026
In the field of defectology and estimation of equipment fault tolerance, there have been relevant studies to assess the useful life of pipelines (Koshovin et al., 2010;Ignatov et al., 2018;Kuric et al., 2019). This can lead to leaks and equipment failures, which entails great losses for the operator. Various models are used to predict the residual service life: deterministic, statistical, probabilistic, fuzzy logic models, and, more recently, artificial neural networks. Figure 4 shows the neural network architecture that was used to solve this problem. The input data are pipe material, condition, length and rate of failure of the pipeline. The output is an estimate of the residual resource in the range from 0 to 1.
In (Chudinová et al., 2018), the problem of classifying wells according to geological properties and technological parameters is considered. A dataset of 555 wells was used for training, including both production and injection wells. 18 well characteristics were selected as input data for the neural network. As a result, 4 groups of wells were identified, similar in technological parameters and geological properties. Figure 5 shows the distribution of wells of all four groups in the field.
The first group of wells is characterised by the highest productivity, large reserves and high stratification coefficient.
The second group has similar parameters to the first, but the average permeability of the wells of the second group is several times lower than in the first.
Wells of the third group can be characterised as unproductive. Their technological parameters differ significantly from the first two groups. The wells are characterised by high current reservoir pressure and a relatively low value of the current water cut.
The fourth group of wells is the most numerous and is characterised by high productivity. In terms of geological parameters (Li et al., 2020), the wells of this group are distinguished by high values of permeability, porosity, net sand to gross ratio and low values of the stratification coefficient. All considered problems were problems of regression or classification. Let's consider the last of our list and probably the most interesting problem -the optimisation of the placement of the grid of wells (Min et al., 2020). Typical ways to solve it can be considered simplex methods, brute-force methods, gradient methods, etc. (Guerra and Narayanasamy, 2006;Hanida et al., 2017). Significant experiments were carried out with the use of neural networks. For example, in (Jang et al., 2018), a neural network predicts the total production for all possible cells of the field and then the wells are installed in places with maximum performance (Dodok et al., 2017).
In (Girich, 2020), an oil-field exploitation system based on a reinforcement learning algorithm is considered. One of the objectives of this system is to find an optimal well drilling strategy in order to increase oil production at an early stage. The algorithm works on the basis of a Markov decision-making process. Markov decisionmaking process can be described using Markov chains (Kemeny and Snell, 1967;Yates et al., 2017), which in turn are described by graphs. Graph nodes describe states of the environment, and edges describe possible actions.
At the first stage, a Markov chain is built (Fig. 6) to select the optimal sequence of actions carried out in the field for calculating reserves, additional exploration deposits and an increase in oil production. According to the concept of reinforcement learning (François-Lavet et al., 2018), we have an agent (field developer) who can perform various actions on the field.
Next, an important step is to determine the methodology for calculating the agent's reward for the actions performed, depending on the strategy and the final goal (Ayoub et al., 2020). Consideration should be given to the change in the value of each step after performing any action. The described system uses the difference between the cost before the action and the cost after (1). However, an important factor is the likelihood of success or failure of the undertaken action since this directly affects the success of subsequent actions (Maiorov and Lozhkin, 2019). action before action after ormation where EV is the expected value calculated using the formula: Bayes' theorem is used to calculate conditional probability. ) , ( ' ' * a s q -the value of performing an action ', in the state ', guided by an optimal strategy.
The described system also implements the determination of the optimal well drilling queue. An example of the resulting strategy on the Markov chain is shown in Fig. 7.

Fig.7. Optimal drilling queue
These approaches are not devoid of significant drawbacks -a complete enumeration of placement options will require a huge number of calculations, and the neural network approach, although it shows good results, but poorly takes into account the history of the placement of each well.
In 2018 company DeepMind introduced the Alpha Zero (Bozek et al., 2018;Bozek et al., 2016) algorithm, which revolutionised game AI by defeating the world champions in chess, Go and other games. Alpha Zero is a decision tree search algorithm. It considers all possible actions but explores the most promising ones (Silver et al., 2018). Degrees of promising are set by the neural network.
Let us highlight the general features of the considered problem of optimising the placement of wells and problems where Alpha Zero was used: -a large space of states, which is computationally difficult to sort out completely; -the final result is important, and not at the current moment -we need to calculate NPV at the end of the field development period; -history of previous actions plays a significant role in decision making.
Based on these similarities, it can be assumed that the Alpha Zero algorithm can also effectively manifest itself in the problem of optimising an irregular well grid.
We further consider this algorithm in more detail below. The idea behind the tree search algorithms in various board games is to find the actions that lead to the best result. The result is determined by a certain sequence of such actions. But there can be a huge number of variants of various sequences, and calculations can take weeks or months, so it is necessary to make a decision by checking only a part of the possible sequences.
The first version of the algorithm was trained on a dataset of games played between real players. To navigate the Monte Carlo tree search, two neural networks were used, one predicting the assessment of the game state and the other one to predict the desirability of the next actions. In the second version, the dataset for training began to be generated by the system itself due to playing games with itself. Two neural networks were combined into one.
The Alpha Zero algorithm conducts a specified number of rounds (so-called simulations) from some game state. During each such simulation, new nodes are added to the tree, corresponding to new game states. The more simulations are carried out, the wider and deeper the tree becomes, and the decisions made become more reliable (Dalgaard et al., 2020). An example of a decision tree can be seen in Figure 8.

Fig. 8. The Monte Carlo Decision Tree for a Go Game
Each node represents a game state. From this state, the permissible actions are known -the branches of the node. The algorithm examines some of them (depending on how many simulations are performed). A new node is created for all investigated states (Pumperla and Feruson, 2019).
The tree branch has the following parameters: p is the a priori desirability of this action, which characterises the desirability of the action before its investigation; n is the number of visits to this branch during simulations; q is the expected score of this action for all visits through this thread.
During the simulation, the branch is selected with the maximum UCT value -upper confidence bound for trees, which is calculated by the formula: Where c is a coefficient that affects the balance between exploration new branches and choosing the most successful ones already explored earlier; N is the total number of visits for all branches from the node; the meaning of the parameters p, n, q is indicated above. Table 1 shows an example of calculating the UCT parameter for three branches A, B, and C. Branch A has the highest Q, indicating that it leads to a good game state below. We visited branch B one time, and it received a negative grade of Q. Branch C has not yet been investigated and, as a result, has the highest indicator of uncertainty (last column). The final selection of the branch will be influenced by the exploration coefficient c. The value of this parameter is selected experimentally. Suppose c = 4, and we want to explore more new branches, then according to formula (4), branch C will be chosen. If we set c = 1, then the choice will be in favour of branch A, as the most explored and valuable. When choosing an unexplored action, a new game state is transferred to the neural network, and then we get from it an estimate of this action and a priori desirability of the next actions from this state.
After expanding the new node, the parameters for all parent nodes are updated: the number of visits increases, the score of the last node is added and averaged over visits. In this case, the sign of the added estimate changes at each level of the tree depth since the nodes belong to different players.
The training process in this reinforcement learning algorithm consists of 6 steps: -Generate a huge batch of self-play games. -Train the model on the experience data.
-Test the updated model against the previous version.
-If the new version is measurably stronger, switch to the new version.
-If not, generate more self-play games and try again.
-Repeat as many times as needed. When all simulations are completed, an action is selected -the branch with the highest number of visits is selected.
If we move on to solving the problem of placing a grid of wells, then we can consider the field sectors cut from the hydrodynamic model as states. Possible actions at each step in the episode would be to drill a new well, convert an existing well to injection, or skip a stroke.

Conclusions
Thus, the article considers a number of problems in the oil and gas sector at various stages of the field life cycle, which are already being solved using various machine learning algorithms, the advantages and disadvantages of the algorithms are assessed, and the prospects for further study and application are identified.
So far, an intelligent approach to solving the problems described above has not been widely used, and the advantage is used as experimental projects, but every year it becomes more difficult to ignore the development of technologies and the industrial implementation of artificial intelligence in the oil and gas sector is inevitable.