Three-dimensional Inversion of Electromagnetic Geophysical Data with Parallel Computational Code on Supercomputer Complex “Lomonosov”

Usage of 2D inversion of magnetotelluric data for real geological objects can cause distortion, but it is more often used in commercial projects, because of its effectiveness and great experience. Whereas in the case of 3D inversion is not such a great experience and there are a number of global problems. When switching to 3D inversion of MT data, the requirement for computer technology is significantly increased. In this paper we will discuss a few examples of 3D inversion of electromagnetic geophysical field data with the usage of “Lomonosov” supercomputer and show its effectiveness on several geological objects. Each object is associated with a variety of problems: from search for shallow ore to regional hydrocarbon exploration. But all these objects contain a large volume of measurements obtaining qualitative results for which requires a huge amount of time. So that the use of 3D inversion with a high-performance computational complex makes it possible to obtain a qualitative result of solving a wide range of problems.


Introduction
In most cases, two-dimensional (2D) inversion of magnetotelluric (MT) data is used to solve practical problems and three-dimensional (3D) inversion is not frequently used in commercial projects. This is due to the fact that most of the commercial studies are done on single profiles, and with already well-developed 2D inversion algorithms, what allows you to get results on a very fine grid. There is some skepticism in commercial companies associated with the use of 3D inversion on real geological objects. Is it possible to get a good resolution of the resulting model, which would suit interpreters? Is the time spent on 3D inversion comparable with the time for which the 2D inversion will be executed? Where are the limits of the applicability of the 3D inversion, in terms of solved problems?
To answer these questions, it is necessary to accumulate a base for solving the inverse problem on real objects, which would demonstrate the effectiveness of the 3D inversion and its capabilities. When selecting such objects, several rules should be considered: 1. The ability to check the results and compare them with other methods or geology; 2. The objects should be aimed at solving various problems for understanding the possible limitations of the applied technology. For the study, two objects were selected that satisfy these conditions: an MT project on shallow ore deposits (Bystrynskoe field) and an Oil & Gas project in Bolivia. They were chosen from different areas and aimed at solving various problems. We will try to show the effectiveness and possibilities of applying 3D inversion. The article is organized as follows. Section 1 is devoted to methods that were used. In section 2, we describe the implementation of 3D inversion on the ore deposit. Section 3 contains our regional study in Bolivia aimed at hydrocarbon deposits. Conclusion summarizes the study.

Methods
In this paper we are solving regularized EM inverse problems, i.e. minimization of a penalty functional of the form: to recover an Earth conductivity model parameter vector m, which provides fit to a data vector d.
In (1), C d is the covariance of data errors, f (m) defines the forward mapping, m 0 is a prior or first guess model parameter, v is a trade-off parameter, and C m defines the model covariance or regularization term. The forward mapping requires solution of the frequency domain EM partial differential equation -finding electrical and magnetic fields with known conductivity of model with finite-difference methods f (m) = d. In MT methods forward problem is calculated independently for two modes of electrical field polarization: TE and TM modes. So, for both modes we need to calculate two electrical fields for each frequency and project it on each site. Then transform to magnetic field h = T e, where T is transform operator. Penalty functional can be directly minimized using a gradient-based optimization algorithm such as non-linear conjugate gradient (NLCG). The NLCG approach requires calculating gradient of (1) with respect to m: In equation (2) J is sensivity matrix which depends on model parameters and field data. So in each iteration to solve inverse problem (finding conductivity of the earth with known field data) m = f −1 (d) it is necessary to calculate a forward problem f (m) and sensivity matrix J. To control results of inversion we use nRMS -normalized on error floor root-mean-square deviation. For our tasks we use ModEM code [1] to provide 3D inversion. The main advantage of the ModEM code is the ability to use parallel approaches to the calculation of forward and inverse problems using the Message Passing Interface (MPI) technology, which significantly reduces the time spent on finding the solutions of the inverse and forward problem, since the forward problem for each period is calculated independently.
Another important segment of the study is application of the supercomputer complex "Lomonosov" [3]. Its peak performance is 2,962.3 TFlop/s, which is the No. 1 of Russia for today. A more important parameter for computing on a supercomputer for us is the number of CPUs used. Since on all investigated objects the number of periods was no more than 30, all calculations were performed on 64 CPUs.
The standard approach for determining the weights of data under inversion was used: percentage of Z xy * Z yx . Therefore, in the description below, the method for determining the weights will be omitted.
Following article [2] the smoothing was determined in the ModEM as the covariance matrix selected in the directions X = Y = 0.3 and Z = 0.4. All starting models were homogeneous half space with resistivity changes depending on prior information. In this paper, only the final results are demonstrated. Of course, the choice of the "most successful" result depends on the experience of the geophysicist. We tried different approaches and show what we think to be the best result.

Bystrinskoe Skarn Cu-Fe Ore Deposit, Russia
The first example is the application of a 3D inversion for a shallow mineral exploration in the Bystrinskoye ore field. This area was chosen for having a well-known geological structure, which allows evaluating the quality of the results obtained. The inversion was carried out for 546 stations. Therefore, only the main components of the impedance tensor were inverted. The error floor of the main components was chosen as 5%. The horizontal size of the grid was 50x50 m, which allows obtaining a good horizontal resolution. Since the task consisted of detecting bodies at depths of the first hundreds of meters, the vertical size of the first cell was 10 meters. Then it increased with a coefficient of 1.1 to 1.5 km, then to 40 km with a coefficient of 2. Total size of the grid is 60x85x40 cells. It took 46 hours to calculate 51 iterations with misfit (nRMS) 1.87. The comparison of the resistivity models ( Fig. 1) obtained on the MT inversion and drilling results indicate that ore boundaries can be determined with high resolution right up to a depth of 800-1000m but on 2D inversion result we cannot successfully determine the ore body. Successful application of MT methods potentially may reduce the amount of closely spaced expensive drilling.

Oil & Gas Regional Studies, Bolivia
A more complicated example of 3D inversion is a regional research aimed at finding oil and gas traps in Bolivian mountain, as well as clarifying the deep geological structure of the region. Figure 2 shows a comparison of the results of 3D inversion and 2D inversion in the area with strong 3D distortions in the deeper part at the profile. At depths greater than 4 km, it can be seen that the 2D inversion "drags" the conductor to a depth, whereas the 3D inversion clearly restores the position of the conducting syncline. To obtain a qualitative result (nRMS < 1.8) of a 3D inversion on 360 stations for 21 periods, 49 hours of calculations were required. At the same time, 2D inversion was performed in several stages for each profile to eliminate 3D distortions and took more than a few days.

Conclusions
The performed work shows that with the use of modern 3D inversion algorithms and highperformance computing system, it is possible for a short period of time achieve acceptable results for a huge amount of MT stations. We tried to find the universal parameters of 3D inversion for each object and there was a good correlation with a geological priori data for different types of projects. The effectiveness of using 3D inversion and good correlation of the obtained results with known geology were demonstrated on field examples.