TWO-PHASE FLOW REGIME THREE-DIMENSONAL VISUALIZATION USING ELECTRICAL CAPACITANCE TOMOGRAPHY – ALGORITHMS AND SOFTWARE

This paper presents the software for comprehensive processing and visualization of 2D and 3D electrical tomography data. The system name as TomoKIS Studio has been developed in the frame of DENIDIA international research project and has been improved in the frame of Polish Ministry of Science and Higher Education Project no 4664/B/T02/2010/38. This software is worldwide unique because it simultaneously integrates the process of tomographic data acquisition, numerical FEM modeling and tomographic images reconstruction. The software can be adapted to specific industrial applications, particularly to monitoring and diagnosis of two-phase flows. The software architecture is composed of independent modules. Their combination offers calibration, configuration and full-duplex communication with any tomographic acquisition system with known and open communication protocol. The other major features are: online data acquisition and processing, online and offline 2D/3D images linear and nonlinear reconstruction and visualization as well as raw data and tomograms processing. Another important ability is 2D/3D ECT sensor construction using FEM modeling. The presented software is supported with the multi-core GPU technology and parallel computing using Nvidia CUDA technology.


Introduction
In the frame of research carried out in the Institute of Applied Computer Science at Lodz University of Technology there was build a dedicated software named TomoKISStudio for the three dimensional tomographic data on-line visualization. There was implemented a set of modules for communication with any of the tomography devices held in the department's research laboratory and the modules for the 3D image reconstruction and visualization. The mentioned modules offer the following functionality:  full-duplex communication with electrical tomography devices,  calibration and configuration,  measurement data acquisition and processing,  raw and normalized measurement data visualization,  linear and non-linear iterative 3D image reconstruction including implementation of parallel computing based on Microsoft OpenMP, NVidia CUDA  on-line 3D image presentation,  a possibility of easy development by adding a new modules. The application was implemented using concurrent programming. That means that singular modules are working in their own threads. Additionally, the linear algebra computing engine is based on CUDA -parallel computing technique powered by Nvidia and the Microsoft Visual Studio C++ environment. Thanks to the special directives of work sharing constructs it was possible to share the processes between existing CPUs and GPUs.
As was mentioned before the application can communicate with three electrical tomography devices. There are implemented the modules for communication with capacitance ECT, resistance ERT and dual ECRT tomography. In all of the modules there is a possibility of usage of the rotatable sensor driver [8].
In addition the two of the tomography modules are equipped with the mechanism of automatic measurement channel gains configuration. In order to perform the remote measurement there is build the module with implemented TCP/IP stack for the measurement data transferring through the computer network. Additionally, TomoKISStudio application has two plug-ins which significantly enlarge the functionality of the software by the possibility of building the 3D computer model of the capacitance or resistance sensor. It is worth mentioning about the module for tomograph simulation. From the previously storied measurement data it is possible to play them with the slower frequency and to reconstruct images with the more demanding algorithm. The figure 2 presents the part of the configuration panel which allows to choose the described devices modules. The three-dimensional image presentation was performed applying the graphics engine VTK library (The Visualization ToolKit). Thanks to this the application presents the images using cut planes or iso-surfaces. Additionally, it is possible to see the raw and normalized measurement data in one graph. This is obviously done in on-line mode.

Module for automatic configuration of gains for the measurement channels of the tomograph
In the research department's laboratory for three dimensional capacitance tomography experiments purposes it is used the device named ET3 which was construct by the research team (dr R. Szabatin) from Institute of Radioelectronics at Warsaw University of Technology [3]. The main feature of this device is the possibility of freely gains configuration for each of the measurement channels. For the 32 electrodes mode there are 496 measurement channels (electrodes pairs) and each of them has 4 configurable elements. There are two amplifiers (4 values of setting for each) and feedback resistor and capacitor (it gives together 12 values of settings). Such diversity of settings caused that the manual tuning of the device is very time-consuming. Simultaneously, there is very important to tune the system as precisely as possible remembering that despite the big and different distances between electrodes and the different sensor geometry the small measurement capacitances should oscillate in the measurement range of the device and additionally should be sensitive enough to the changes of the permittivity distribution between full and empty sensor. Taking the above into consideration the authors of the paper implemented the specialist module which automatically sets the gains for all the measurement channels based on the user conditions and calibrates the measurement unit to the sensor geometry in the same time. In the figure 3 the configuration dialog for ET3 gains auto-configuration module is presented.
In the figure can also be seen the choice of manual gains settings. In case of the need of manual tune it is only required to click the measurement column on the graph and simply to change the gains settings. The user can quickly diagnose the wrong configuration of the measurement channel and easily correct it.

TCP/IP data communication module for tomographic data transmission
The measurement noise can be real issue in electrical capacitance system due to very small values of capacitance data. One of solutions for noise minimizing can be reduction of a measurement cables length. There is also another limitation which is a RS232 interface data transmission cable. Given restrictions result in difficulties of performing ECT measurement in two sections in the same time for real industrial applications. As an example of such application two semi-industrial testing facilities built in the author's Institute can be considered. There is 20 meters of distance between them.
To avoid this problem a new measurement data transfer software module has been developed based on TCP/IP networking protocol and client-server communication scheme. A part of this module was embedded in TomoKISStudio architecture and it is working as a network client. A server part is dedicated to be as a software component of the PC directly connected to the ECT acquisition hardware -see Figure 4.
The server module includes both: calibration routines and tomography system configuration data. The remaining module features (ex. planes number selection, electrode layout configuration) are included in client part of the TCP/IP module.

Offline tomographics data server module
Apart from real-time online communication between PC and ECT hardware the very important thing is to have a possibility of using offline measurement data previously saved and backed up on storage. Using the offline data we can perform image reconstruction process which is accurate and does not require online and fast view of industrial process. This work can be also done immediately after the research experiment will end. The new offline measurement data serving module has been developed for that purpose. The configuration of this module is simple and requires only setting up of offline measurement data path, selecting their format and choosing a speed of feeding them to a reconstruction module.
Using new offline data feeding module advanced reconstruction techniques can be applied including iterative methods and nonlinear reconstruction algorithms with electric field simulation and sensitivity distribution update. This techniques, on account of their computational complexity, are much time consuming and can need hours to generate single 3-dimensional image but they offer good accuracy and quality for further tomographic data processing. Examples of reconstructed images using different algorithms have been presented in Figure 6. In comparison to the LBP method that is able to achieve 100 frames per second the nonlinear algorithm can require 2h to provide single frame depending on computational power of used PC.

A driver module for rotatable sensor with ECT hardware synchronization and online image reconstruction
The new idea of rotatable ECT sensor [8] in the author's Institute industrial tomography laboratory was also developed. This concept can bring a significant improvement of spatial resolution in reconstructed images by using an increased number of measurement data from the limited number of electrodes. For this purpose an additional driver module for stepping motor control and configuration was designed and implemented. This module is responsible for synchronization of stepping motor rotation with DenidiaERCT hardware measurement data transmission. The stepper motor -PC communication is controlled by ActiveX technology that lets to configure main parameters like number of measurement rotation steps, number of rotated electrodes and speed of motor rotation. This module also includes a specialized reconstruction submodule that takes into consideration increased amount of measurement data and reconstruct and visualize images in a realtime. There is an important fact that increased number of capacitance data is much more time consuming comparing to a conventional non-rotatable ECT system. The system developed in the Institute of Applied Computer Science is able to provide 2 frames/s using 12-electrode sensor and 4 rotation positions but the number of capacitance data increases from 66 to 264. The current speed of the rotatable ECT system can bring some difficulties for its real especially dynamically unfolded industrial application but the preliminary results are promising for non-invasive monitoring of static or slowly-unfolded processes.

Mesh generator for computer model of electrical sensors
The design of an ECT capacitance sensor consists of many different steps. One of the most important is the project of the sensor's computer model, based on a 3D spatial mesh. Before our application has been developed, spatial or plain meshes have been generated with help of external, ready-to-use and mostly universal tools, e.g. NETGEN. However, such tools are intended for general use in many engineering fields and therefore they do not allow implementing all design ideas. For these reasons we developed a TomoKISStudio module for automated mesh generation, namely ERCT Mesh Generator, which is designed to for both plain (2D) and spatial (3D) meshes ubiquitous in capacitance and impedance tomography.
The presented software has the following features and functionalities:  ready-to-use and fully configurable mesh-generators for cylindrical sensors. There is no need to write scripts or configuration files, every parameter can be modified via windows dialog boxes,  implementation of functionality for Complete Model [1] approach. The Complete Model mesh consists of three layers: the inner layer (marked as M i in Fig. 8) used as a base for image reconstruction algorithms, the insulation layer (marked as M t ) which is often a material of sensor's tube with electrodes placed on the outer surface (S 2 ) and the outer layer (M o ) which is insulation between electrodes and shield (S 3 ) of the sensor,  The ability to generate meshes for both plain and spatial meshes,  The ability to set the sparseness value in every point of the sensor as a pair-wise linear function of radius; each value can be freely modified by the designer,  The ability to setup geometrical parameters of generated mesh, such as thickness of the sensor's tube (insulation layer) or the radius of the image layer (inner layer),  The ability to setup material parameters in every layer of the sensor, namely electrical permittivity (for capacitance sensors) or resistivity (for impedance sensors),  Instant generation of mesh data as text files for further use in other software than TomoKISStudio, e.g. Matlab,  The ability to generate image files for visualization in VTK toolkit, helpful in preparing presentations or papers. Developed software allows viewing (Fig. 9) in real time the results of entered configuration values as the top cap of the mesh. There are three layers shown, marked with different colors, which correspond to the illustrative draft in the main window (Fig. 8). In addition, useful numerical information about the mesh is displayed:  radius value for each ring in the top cap,  density (distance between nodes) value for each ring,  number of nodes on each ring, and other information important for the duration of electrical field simulations, like number of nodes or finite elements in plain or spatial mesh.
The development of the presented tool is also justified by the instantiation in application for cylindrical sensors in CT, the possibility of rapid expansion and adjustments to new ideas and fields of application. Existing software tools (e.g. Netgen) allow only the amount of flexibility possible in terms of their internal configuration and scripting languages. However, our own software allows fast and easy modifications and interference in any part of its algorithms. In our case, self-developed software allowed to improve research on electrical tomography, including the Complete Model of spatial capacitance and impedance sensors by immediate and automated generation of meshes with different internal configurations.

The module for designing spatial distribution of electrodes in electrical tomography sensors
The design and assemblage of a capacitance sensor for 3D ECT is a key task in the overall building process of 3D electrical tomography measurement system. The quality and precision of measurements is therefore strongly affected by two factors: physical (e.g. the materials, precision of electrical components and their assembly) and theoretical (e.g. distribution of electrodes, size of finite elements and density of the model mesh). Our solution partially covers later factor by combining mesh generation (and altering) with designing spatial distributions of electrodes in one monolithic CAD-like environment, namely Layout Builder [6] shown in Fig. 10. In following we present main features and functionalities of Layout Builder module:  utilization and processing of spatial tetrahedral meshes,  defining the geometric distribution of electrodes on proper surface of the sensor,  creating distribution of electrodes for cylindrical and rectangular sensors,  model data files can be generated and stored for later use in electrical field simulation and image reconstruction algorithms,  an image of spatial distribution of electrodes on the surface can be generated as a bitmap file and used in assembly process of the sensor. Directly, images can be quite helpful in etching of electrodes from copper laminate. In addition, developed software allows to visualize designed distribution of electrodes (in 3D View tab) and to export such visualization into data files, compatible with VTK toolkit.
The design process of a typical cylindrical capacitance sensor with help of Layout Builder is performed in three steps: First is to create a distribution of electrodes on the surface of a sensor. To achieve this, proper tools have been developed. They allow to group electrodes in rings (rows of electrodes) and precisely position them on the sensor's surface. Such design can be done coarsely using visual tools or precisely by entering the appropriate numerical values.

Fig. 10. Graphical User Interface of the Layout Builder software
Second step is to load spatial mesh which will be the sensor geometry base and input data for further reconstruction and visualization algorithms. Loaded mesh should be adjusted to the previously developed distribution of electrodes by moving or adding nodes. This is done in the edit panel of the spatial mesh editor. Similarly for the first step, one can use coarse visual editing tools and adjust mesh with mouse or one can specify exact nodes coordinates values.
Third step is to generate the adjusted mesh required to determine the sensitivity matrix as a vital part of reconstruction algorithms. Reconstructed image uses same mesh in visualization process, hence adjusted mesh is required in imaging process as well.
Preparation of a sensor corresponding to the assumptions of the design is inherently an iterative process. It consists of: creating a numerical model and determining its quality by evaluating the results of simulations, the proper adjustments of the model and resimulation. Evaluation is based on quality of the modeled electrical field distribution inside the sensor and the reliability of the results of reconstruction. On this basis, adjustments are made to the finite element mesh of the sensor and to the geometric parameters of electrodes -their location, number and size.
So far, such modifications have been performed manually, e.g. by creating source codes for modifying the finite element meshes or manually drawing the distribution of electrodes in graphics software. This was due to a lack of specialized tools for modeling sensors.
The Layout Builder software fully automates this process by providing tools that enable simultaneously: designing distribution of electrodes, matching them to the mesh and generating data for subsequent simulations. This result in a significant reduction in time needed to perform a series of numerical models needed to obtain the sensor with the best parameters. It should also add that the developed software holds consistency between the model mesh, the layout and sensor electrodes by eliminating the possibility of human error in various stages of the modelling process.

Sensitivity and voltage software analyser
The task of image reconstruction process for the threedimensional capacitance or resistance tomography is a crucial problem in a tomographic visualization process. The threedimensional image is built using a set of sophisticated algorithms based on the finite elements grid of 3D ECT or ERT sensor model. The aim of it is to as fast as possible reconstruct the material distribution from the measurement data. Because of the similarity between the 2D and 3D electrical systems the three-dimensional visualization is possible using the deterministic image reconstruction algorithms like methods based on back-projection (one-step or iterative scheme) including non-linear methods [9][10][11].
The main goal of the method is the linearization of the inverse problem. The linearization assumes that the total change of the measured capacitance for any electrodes pair is equal to the sum of the particle capacitance changes. The mentioned particle change is directly proportional to the change of the electrical field energy in the given image element as a result of the change of the material permittivity value in this element. Such linearization product in the image reconstruction process is the sensitivity matrix.
It is worth to mention that the image reconstruction process commonly performed on "voxels" (i.e. tetrahedrons) in case of 3D ECT is very time consuming. This is due to the large number of image elements which are repeatedly analysed during algorithm life time. The mentioned algorithms involve the matrix computations. The common known "voxel" approach uses matrices which significantly absorb the computer memory. The solution for this inadvertence could be the node-based sensitivity matrix [1]. Moreover, the node-based sensitivity matrix is able to significantly reduce the computation time needed for 3D image reconstruction processing. This reduction is directly proportional to the ratio of the mesh voxels number to the nodes number. The additional advantage is that the final 3D image is already determined directly in the nodes of the mesh and that case there is no need to approximate it in the sense of its further processing before presentation by the graphics engine.
For purposes of this problem the specialist module has been built which supports the two and three dimensional sensitivity analysis of any capacitance or resistance sensor which computer model was generated using the plug-ins described above. This module called Sensitivity Analyser (see figure 11) has implemented the following functionality:  calculation of the potentials distribution for each electrode respecting the assumed permittivity distribution using FEM method,  calculation the sensitivity matrix for given sensor geometry on the basis of calculated electrical field,  calculations can be performed for the 2D as well as for the 3D space,  the Complete Model can be considered,  three dimensional visualisation of each sensitivity map as well as the electrical field,  configuration of different visualization effects.

GPU acceleration
GPGPU (General-Purpose computing on Graphics Processing Units) is a technique of using graphic cards (GPUs), which normally handles graphics rendering, for computations that are usually handled by processor (CPU). Growing interest in GPU computations started with the inability to clock CPU above certain level, because of the limitations of silicon based transistor technology and constant demand for improvements. Because of this interest in multi-core technology and parallel computing started to emerge, even though sequential programs are easier to understand, more natural. There is a limit though of how fast such program can run. Sequential speed up is now based on architecture improvements of the CPU rather than higher clocks, but even this approach has limitations. Parallel programming is not a new idea [4,5] though till only recently it was reserved for high performance clusters with many processors, the cost of such solution was extremely high. This changed with the introduction of many-core processors to the mainstream market. GPUs fit well in that trend, even take it to another level. Compared to CPU which, today, have maximum of 2 to 12 cores, GPU consist, of dozens, and even hundreds of smaller, simpler cores designed for high-performance calculations. CPUs are built and designed to execute single thread no matter how unpredictable, diverse or complicated it may be as fast as possible. For that they require additional resources such as: complicated mechanisms for predicting branches, cache memory and data prefetch. On the other hand GPUs mostly take care of data computations that are much simpler in their nature and for that reason their execution units, or cores, can be much simpler, which also mean smaller. Thanks to that there can be much more of them on a single chip with numbers reaching dozens or even hundreds. Also, because graphics computations can be easily parallelizable cores in GPU work based on a SIMD (Single Instruction, Multiple Data), or more precisely. In the frame of the presented software the authors developed a CUDA code [7] that has been successfully verified using 3D ECT numerical model and 3D image iterative reconstruction approach. A significant speed-up of 3D image reconstruction IAPGOŚ 1/2017 p-ISSN 2083-0157, e-ISSN 2391-6761 process was achieved using cheap market-level graphics processors. Thanks to this module it is possible to perform i.e. Landweber or SIRT image reconstruction with more than 10 iterations and achieve the image rate equal to the 3D 32-electrodes ECT measurement frame rate.
TomoKISStudio just after its start loads CUDA library driver and then tries to find accessibility of any graphics card in the system. In case of successful result of this process in the dialog window with the image reconstruction process settings the CUDA acceleration and the methods supported with the parallel computations are enabled -see fig. 12.
In addition the user is always informed in the status bar of the application about details of the found graphics cards as well as the version of CUDA and VTK libraries -see fig. 13.

3D visualization effects
Because the visualization module for presenting the 3D tomograms, sensitivities and potentials distribution is supported by the VTK library TomoKISStudio application performs the image presentation applying different interesting effects. There are:  the colors values interpolation between the nodes,  three cross-sectional cut planes freely defined along three axes,  set of iso-surfaces defined by the threshold value,  algorithm HAVS (Hardware Assistance Visibility Sorting),  banded surface maps,  electric field lines,  freely configurable levels of transparency for all effects separately,  legend for the image,  export of the current image to the file in TIFF format.