Multi-task super resolution method for vector field critical points enhancement

It is a challenging task to handle the vector field visualization at local critical points. Generally, topological based methods firstly divide critical regions into different categories, and then process the different types of critical regions to improve the effect, which pipeline is complex. In the paper, a learning based multi-task super resolution (SR) method is proposed to improve the refinement of vector field, and enhance the visualization effect, especially at the critical region. In detail, the multi-task model consists of two important designs on task branches: one task is to simulate the interpolation of discrete vector fields based on an improved super-resolution network; and the other is a classification task to identify the types of critical vector fields. It is an efficient end-to-end architecture for both training and inferencing stages, which simplifies the pipeline of critical vector field visualization and improves the visualization effect. In experiment, we compare our method with both traditional interpolation and pure SR network on both simulation data and real data, and the reported results indicate our method lower the error and improve PSNR significantly.


Introduction
Vector field visualization has wide application in the field of science and engineering, and plays an important role in meteorology, aircraft design, automotive design, and computational fluid dynamics [1] . Although significant progress has been made in vector field visualization after years of research and development, quite a few questions worth exploring still exist. For example, during data collection tasks, due to the limited precision of sensors and radical vector changes near critical points, the collected discrete data do not represent well the local features.
Currently, vector field visualization related works are mainly focused on improving the visualization quality and effectiveness, including the research, application, and refinement of streamline generation strategies, color enhancements [2] , GPU load balancing [3] , and seeding strategies [4] . However, these methods do not work well for the problem of vector field visualization near critical points. Since the critical points in the vector field contain very distinct structural information and can be explicitly characterized by topological features, the topological featurebased approach is better for visualizing the critical region of the vector field [1] : vector fields are first classified according to the type of critical points, the visualization is then enhanced using a special interpolation according to its classification. This process however involves numerical-based classification algorithms, which is computationally intensive. At the same time, the visualization process is complicated, and error accumulation is still too high for radical vector direction changes.
Super-resolution algorithm achieves image enhancement by enlarging the local details of an image. It is a mature end-to-end machine learning algorithm in the image processing field. Since vector field data and image have the same characteristics, i.e. a fixed boundary and number of channels, this paper draws on the super-resolution algorithm to perform superresolution enhancement on the critical region of the vector field. In addition, combined with the idea of multi-tasking, and the fact that multiple tasks in the network model can enhance each other, we consider adding multiple supervised signals based on the super-resolution model to achieve further enhancement of the super-resolution effect of the vector field. Therefore, this paper proposes a multi-task based super-resolution neural network model to achieve the improvement of interpolation and visualization near the critical region of vector fields. For the super-resolution task, interpolation of discrete vector fields is implemented to improve data accuracy in order to highlight more detailed features of the data fields. For the classification task, the classification of critical points is implemented, replacing the numerical solving classification of critical points in traditional topology-based methods. Effective information is extracted at the shallow layers of the super-resolution neural network instead of interpolation of specific types of vector fields.
In summary, the approach this paper presents has two main novel contributions: (1) A deep learning based multi-task super-resolution network model is designed to achieve a refined enhancement of the critical region of the vector field.
(2) The proposed vector field enhancement method is an end-to-end efficient inference approach that helps simplify the process of vector field visualization.

Vector field visualization
Vector field visualization is part of visualization in scientific computing. In practice, vector field visualization methods can be divided into icon-based, geometry-based, texture-based, topology-based [1] , etc. Among them, texture-based methods are usually used for two-dimensional vector field visualization, and the most representative ones are the Line Integral Convolution (LIC) algorithm [5] and the improved algorithms derived from the Line Integral Convolution algorithm [2] . Extensive research has already been done on LIC, and as such LIC related works are abundant and mature. Since the LIC algorithm usually encounters problems of occlusion and low efficiency on 3D data, 3D vector fields currently are mostly visualized by flow lines or flow surfaces based methods.

Topology-based methods
The theoretical basis for the vector field visualization based on topological features is the Critical Point Theory, in which the critical point refers to a point in the vector field where each component is zero. The Critical Point Theory considers the topology of an arbitrary vector field consisting of critical points, and surfaces and curves connecting those critical points. As a result, it can help people extract the important information of the vector field by extracting the topological skeleton and ignoring the secondary information, thus improving the visualization and enhancing the understanding of the vector field data. The topology-based approach has a sound mathematical basis and can therefore be applied to any vector field. The classification of critical points in this paper is based on publication [6] .
A very important step for topology-based of vector field visualization is to classify the critical points so that the topological characteristics of the vector fields can be better studied. The eigenvalues and eigenvectors of the Jacobi matrix are calculated by evaluating the partial derivatives of the vector field for each critical point. Classification is then performed based on the eigenvalue distribution on the complex plane [7] . Generally, they can be classified into saddle points, nodes, foci, centers, etc. Better visualization can be done according to critical point classifications.

Deep learning and visualization
General data visualization methods have been commonly used in the field of deep learning, usually for visualizing the training process of deep learning models, assisting researchers in debugging models, and enhancing the interpretability of models [8] , etc. On the contrary, there are only a few related works on data field visualization using deep learning methods. For example, publication [9] trains a model for viewpoint recommendation in volumetric data visualization, and publication [10] uses deep learning methods to reconstruct vector fields from generated streamlines, etc. In conclusion, the intersection of artificial intelligence and visualization is currently a research focus in computer science, and is early in its infancy. There are still many issues worth studying [11] .

Super-resolution
Super-resolution technology was first used in image processing to restore low-resolution images to high-resolution images, increasing image clarity in the process. Over the past decade, deep learningbased image super-resolution enhancement algorithms have been emerging. SRCNN method was first proposed in the publication [12] , on the basis of which many improved super-resolution models were proposed, such as FSRCNN [13] , which improved the input of SRCNN, changed the feature dimension, and accelerated the training speed. Other improved models such as VDSR [14] , ESPCN [15] , and more were also proposed. The deep learning-based image superresolution technology has been developed into a fullfledged and mature image enhancement solution over the years.
Publication [16] used the SRCNN method in the scalar field sampling, publication [17] simulated and generated a huge fluid flow dataset and used a simple super-resolution model for interpolation training, and publication [18] trained a deep convolutional neural network to interpolate 3D vector fields. This is the first time someone proposed a neural network specifically for vector field interpolation. The three components of the vector are separately trained and then combined, the network consists of several residual blocks.

Problem statement
Objective Given a low-resolution vector field , our goal is to synthesize a high resolution vector field using some function . . can be a non-machine learning-based function, such as bilinear interpolation, or an ML-based function, such as a super-resolution neural network. They all have their own characteristics and bottlenecks in ensuring reasonable continuity of high-resolution discrete values.

Non-ML Based Issues
The sampling information which F uses to generate Y is from X only. This is more effective for discrete values with good smoothness on U. If complex ridge and line features emerge, the generated result will be unsatisfactory.

ML-Based
Issues The sampling information which F uses to generate Y is not only from X, but also from a sample domain {X} to which X belongs. This can interpolate features that are not visible in X but are present in the sample domain. However, if the critical points present in {X} is diverse and have radical changes, a single-task holistic deep learning approach based on {X} will have great difficulty achieving high accuracy.
In a two-dimensional vector field, the direction of vectors near the critical points is highly dependent on the critical point type. The network architecture we propose will add subcategorized supervised signals of critical point type to the SR model, and use the shared network to split the sample domain { } into smaller subdomains { 1 }{ 2 }{ 3 }{ 4 } during the learning process, which facilitates further improvement of the SR task on each subdomain.

Neural network architecture
The multi-tasking network architecture we proposed for two-dimensional vector field critical points is shown in Figure 1. It is a multi-task super-resolution network model that includes a super-resolution branch and a classification branch, as well as a base network shared by both branches. Input. The super-resolution branch of this model is modified from VDSR and adopts a similar approach to VDSR in terms of data input. The lowresolution vector field is first scaled up to the same size as the high-resolution vector field and subsequently fed into the network. Shared Network. Since our approach is multitask prediction, we adopt the end-to-end learning strategy considering the inference speed and model learning capability. The shared network is intended to find common features across the two branches. Specifically, common features are extracted by convolving the input data twice, and then fed into the super-resolution branch and the classification branch for their respective prediction tasks.
Super-resolution Branch. The super-resolution branch is composed of several convolutional layers, in which the Conv4 layer can be followed by several identical convolutional layers. In this experiment, we used three Conv4 layers. To improve the computational efficiency of the classification branch, the number of channels before the introduction of the classification branch is reduced. The original number of channels is then restored. We used the Leaky ReLU activation function in order to prevent the vanishing gradient problem in this experiment.

Classification Branch.
We employ a conventional classification architecture. Having elicited data from the super-resolution branch, a four-classification result is acquired after three convolutional layers and two fully connected layers.
Under the supervision of the classification branch, this architecture uses a shared network to split the input sampling domain of the vector field space domain into small subdomains. Data enhancement is performed via each super-resolution branch. The types and parameters used in each layer of the architecture are shown in Table 1.

Loss function
For the super-resolution of images, we usually use the root mean square error (MSE) to measure the accuracy of the model output results. Thus, we continue to employ MSE as the loss function for our model. However, in vector field super-resolution based on the critical point classification, it is not enough to consider only the root mean square error. As the loss function of the classification also needs to be taken into consideration, we have made the decision of combining multiple loss functions.

Vector Loss Measurement.
We utilize MSE to measure the loss of vectors ( ), it's defined as follows: (1) In Equation (1), represents real data, and ′ represents the model prediction result. We calculate the root mean square error by comparing the square of the difference between each component of the output vector and each component of the real value.
Classification Loss Measurement. We use the cross-entropy loss function to measure the accuracy of the classification ( ), defined as follows: In Equation (2), is the number of samples fed into the model for each training, is the number of categories for classification, and is the indicator variable, which is 1 if the ith data is equivalent to category and 0 otherwise.
is the predicted probability distribution of belonging to category for data .
Based on the above two loss functions, our final loss function is defined as: In practice, each component of the loss function can be weighted accordingly. In this experiment, we assign the same weight to both loss functions.

Data preparation
Data Simulation. Since we need to perform SR to the vector field according to the critical point type, and there is a lack of large-scale datasets for vector fields, the training data can only be generated via simulation. The theoretical basis of our simulation is derived from the publication [19] , where an experiment-based formula is given that can simulate various types of critical points. Different types of critical points can be generated by adjusting the parameters. We use this formula to generate the critical points of two-dimensional vector fields. Types of generated critical points include saddle points, centers, foci, and nodes. The characteristics of vector fields near different critical points are shown in Figure 2. 9,345 synthetic data entries were used for training and 763 entries were used for testing, each with a size of 200 × 200 and a channel number of 2. The data were listed in categories according to data type, as shown in Table 2:

Implementation details
The experiment was run on a server equipped with an Intel Xeon E5 2678 v3 processor with 64GB of memory, and two NVIDIA GeForce RTX 2080Ti GPUs with 11GB of memory. The model was trained for 7.2 hours.
In this experiment, in order to achieve a more accurate training effect, we also adopt the adaptive learning rate. The initial learning rate is set to 0.001, and thereafter the learning rate becomes one-tenth of the previous one every 30 epochs. Adam optimizer is utilized, and 60 shuffled data entries are fed into the neural network every epoch, for 120 a total of 120 epochs.

Evaluation metrics
We use the peak signal-to-noise ratio (PSNR) here to measure the accuracy of the output results. PSNR was originally used to evaluate the effectiveness in image super-resolution tasks, and the formula for calculating PSNR is based on the root mean square error. Here we can also use it to measure the results of our model output. In the task of evaluating vector field super-resolution models, PSNR is defined as: We have made some modifications to the PSNR formula, where ( ) is the difference between the maximum and minimum values of the vector field values, and in Equation (4) we evaluate the truthfulness of the model output by calculating the root mean square error between the predicted result and the true value ′ .

Comparative analysis of training process and end results
Here we compare the model in this paper with some other algorithms, including the SRCNN model for SR imaging, and we also compare the training effectiveness of this model with the VDSR model in order to verify the effect of multi-objective optimization. Figure 3 shows the changes of the root mean square error of the SRCNN model, the VDSR model, and the model proposed in this paper during the training process: As can be seen from Figure 3, during the training process, the root mean square error decreases faster due to the large initial learning rate, and then decreases by varying degrees later as the learning rate adjusts. Overall, after adding the prediction branch for multi-task training, the model converges better, and the error is lower than that of the model without the prediction branch. The adaptive learning rate also has a positive effect on multi-objective optimization. The output of the model tested on different kinds of critical points is shown below (Figure  4).
Subsequently, to validate the model, we tested the model using untrained test data, while adding a portion of the double gyres to verify the generalization ability of the model. The MSE and PSNR derived from each of these methods are shown in Tables 3 and Table 4, and the best results are marked in bold font.  From the above evaluation data, it can be seen that from the SRCNN model to the VDSR model to the multi-task SR model, the SR end result of the input vector field is getting better and better. Meanwhile, the VDSR model can perform the SR task more accurately compared to SRCNN due to the higher number of layers. Therefore, under certain conditions, the deeper the neural network is, the better the training result will be. Under the condition that model depth is fixed, the model with a prediction branch outperforms the model without a prediction branch.
For the double gyres, our proposed model is shown to be able to accurately determine the critical point classification, and combined with the evaluation metrics, we can see that our model has better results than other models for SR task of this data.

Conclusions
This paper proposes a new method for vector field visualization based on a multi-task super-resolution model. After experiments, it is proved that this model achieves better results than the conventional numerical interpolation methods and single-task super-resolution models, has higher accuracy for super-resolution from low-resolution vector fields to high-resolution vector fields, and has higher accuracy for the classification of critical points. The model also has good extensibility and is we believe it can be generalized to visualize 3D vector fields, which is one of the future research possibilities. Besides, our model can only identify a single critical point in a vector field, and further research is needed to classify and identify more complex vector fields with more complex topology.