An Ontology-Based Framework for Integrating Remote Sensing Imagery, Image Products, and In Situ Observations

Along with the rapid development of remote sensing satellites and sensor network technology, vast amounts of remote sensing imagery and in situ observations have been accumulated. Further, various researchers and agencies have released a variety of thematic image products. These heterogeneous observations are therefore difficult to utilize comprehensively. In this study, an ontology-based framework for integrating remote sensing imagery, image products, and in situ observations was developed. It was extended based on the Semantic Sensor Network (SSN) ontology in the Web Ontology Language (OWL). The detailed process of ontology construction and rule establishment was demonstrated. Combined with some actual remote sensing imagery, image products, and in situ observations, semantic queries based on DL Query and SPARQL were conducted to establish the rationality and feasibility of the ontology and framework.


Introduction
With the development of remote sensing technology and the improvement of the resolution of satellite sensors, the application of remote sensing to quantitatively obtain the required parameters on a large scale on the ground has become even more extensive. However, remote sensing, as a monitoring method for periodic acquisition, lacks timeliness and is not suitable for monitoring hot spot regions with rapid variations, and because of atmospheric interference, the overall accuracy is not sufficiently high. As a kind of ground-based monitoring system, wireless sensor networks and remote sensing data functionally complement each other and can dynamically monitor all kinds of parameters needed by the region in quasi-real time. Wireless sensor network technology is not only regarded as a new outgrowth of traditional remote sensing technology but also an extension of remote sensing, the geographic information system, and the global positioning system [1,2]. Therefore, the integration of the technical characteristics of both, each playing to their respective advantages, will become a development and application trend in the future [3,4].
A large number of studies have focused on integrating remote sensing and ground-based sensor networks [5][6][7][8][9][10]. A collaborative inversion method combining a ground sensor network and HJ-1 satellite remote sensing data was proposed to retrieve ground temperature [3]. In situ temperature measurement and thermal imaging from flyovers or drones or satellite remote imaging were used to track the dynamics of freshwater-seawater mixing behavior in a bay [9]. Warming trends of perialpine lakes were analyzed based on homogenized time series of historical satellite and in situ data [8]. A novel strategy combining global satellite multispectral datasets, environmental constraints, and in situ acquisition of geospatial data was presented for cash-crop mapping [7]. From the contents of these studies, it can be seen that research on integrating remote sensing images and ground sensor networks is often aimed at a single target, using specific remote sensing images. It is difficult to carry out composite and integrated research.
Constructing uniform resource representations is quite significant to improve resource sharing and interoperability, as well as the comprehensive process of decision-making [11]. Semantic Sensor Network (SSN) ontology [12], proposed by the World Wide Web Consortium (W3C), has been widely used as an ontology model to solve semantic barriers in sensor networks [13][14][15][16][17][18][19][20][21]. However, SSN ontology lacks descriptions about remote sensing imagery, and satellites only act as platforms for sensors or actuators. There have been several studies on the semantics of satellite-based sensors or remote sensing imagery. An ontology for associating sensor observation capability was developed to facilitate remote sensing satellite selection in certain earth observation tasks [22]. A light ontology was constructed to represent the content of remote sensing imagery based on existing geographic ontologies, the domain-related corpus, and the WordNet [23]. The WordNet, identifying concepts having a similar meaning but using different words, has been widely applied in evaluating the semantic similarity between words [24][25][26]. A two-phase methodology was built to facilitate the satellite parameterized description and remote sensing data classification based on the ontology concept [27]. Taking into the principal scaling factor and various spectral variability consideration, a novel spectral mixture model was proposed to improve the flexibility of the endmember dictionary by introducing the dictionary of spectral variables [28]. Spectral rules have also been used for semantic classification of remote sensing imagery [29][30][31]. In order to make better use of satellite image resources, the World Meteorological Organization (WMO) has provided most mission objectives for major satellite sensors [32]. However, semantic research about the combination of remote sensing images and ground sensor networks is still rare.
At present, a large number of image products based on remote sensing imagery have emerged. These image products are based on various remote sensing images, using object identification, parameter extraction, and other methods in combination with field measurements or referring to existing datasets if possible. After band selection and fusion, image enhancement processing, stitching, or cutting, these image products represent certain thematic information, such as land use/land cover, precipitation, and surface water. Although image products and remote sensing imagery are similar in structure, the specific means of utilizing them are completely different.
Considering the abovementioned themes, the objective of this study was to develop an ontology with semantic rules to integrate remote sensing imagery, image products, and in situ observations (named RSISO ontology) based on the W3C OWL 2 Web Ontology Language [33] and the software Protégé developed by the Stanford Center for Biomedical Informatics Research at the Stanford University School of Medicine [34]. The ontology was extended from the SSN ontology based on the concepts of spatial pixel and spectral rule. This new ontology can be utilized to retrieve available and potentially valuable remote sensing imageries, image products, and in situ observations. Specifically, the contribu-tions were summarized as follows: (1) an ontology-based framework was proposed to integrate remote sensing imagery, image products, and in situ observations; (2) the formal expression of the characteristic index is more conducive to the sharing and reuse of empirical knowledge in different fields; and (3) available data can be discovered more easily by using semantic technology. A new perspective will be provided by applications of this ontology, making sufficient use of remote sensing imageries, image products, and in situ observations. It will be powerful in serving complex and comprehensive environmental monitoring, disaster management, and decision-making processes.
The remainder of this paper is organized as follows. Section 2 analyzed the typical remote sensing images and image products; then, the ontology and related rules were constructed. The experimental data and results were described in Section 3, the discussion was presented in Section 4, and conclusions and future work are given in Section 5.

Materials and Methods
2.1. Overview. The RSISO ontology was designed for integrating remote sensing imagery, image products, and in situ observations. Typical instances of these three types of observation data are listed in Table 1. There are three dimensions of semantics in blending the three kinds of observation data: spatial, temporal, and thematic. It acts as a semantic framework to achieve validation, fusion, and provenance among the three kinds of observation data, as illustrated in Figure 1. The detailed steps of constructing the ontology are presented in the following subsections.

Procedure of Ontology
Modeling. It is necessary to establish an ontology in a standard process in order to ensure the rationality and reusability of the constructed ontology. The seven-step methodology proposed by Noy and McGuinness of Stanford University [37] was applied in this study. The seven-step methodology has been widely used [38][39][40][41][42][43] and is closely integrated with the ontology construction software Protégé [34]. Therefore, it was also used in modeling the RSISO ontology. The details are as follows.
2.2.1. Determine the Domain of the Ontology. The formal representation of earth observation was determined as a domain with scopes limited to remote sensing imagery, image products, and in situ observations, which were examined in Table 1 and Section 2.1.

Consider Reusing Existing
Ontologies. Because the SSN has had many successful applications in ground-based sensor networks, as described in Section 1, we reused the core concepts of SSN. Additionally, GeoSPARQL [44], Time [45], and WordNet [46,47] ontology were imported as semantic descriptions of space and time, respectively. For ontologies from different sources that were reused, it is necessary to provide a clear list of prefixes, as shown in Table 2.

List Important
Terms in the Ontology. The ontology's key terms for the application domain were defined and used for the class definition in the next step. These terms include 2 Journal of Sensors remote sensing imagery, image product, spatial pixel, band, and characteristic index. Descriptions, key properties, and examples of these terms are listed in Table 3.

Define the Classes and Their
Relations. Based on terms enumerated in the former step, five classes-RemoteSen-singImage, ImageProduct, SpatialPixel, Band, and Charac-teristicIndex-were created in the proposed ontology.
RemoteSensingImage and ImageProduct were set as subclasses of sosa:Result, SpatialPixel was a subclass of sosa:FeatureOfInterest, and Band was a subclass of sosa:ObservableProperty. CharacteristicIndex, as a property, cannot be observed directly, so it was inherited from ssn:Property. In this step, we defined value type restrictions of data. For example, the value of property hasID is restricted to be of datatype string and datatype double for the value of property hasCloudCover. Pixels with band "precipitation"; the value of the band is numerical (unit: mm/day)  There are tools and codes to achieve such coordinate conversion, such as the United States Geological Survey (USGS)'s Landsat Acquisition Tool [57] and converting coordinates with Java code provided by Salkosuo [58]. As such, a detailed conversion procedure is not described here.
As described in step 2 in Section 2.2, GeoSPARQL was used to implement spatial semantic queries in the RSISO ontology proposed in this paper. rsiso:RemoteSensedImagery and rsiso:ImageProduct were defined as a subclass of sosa:Result in Section 2.2, and they rsiso:hasPixel rsiso:SpatialPixel. Then, rsiso:SpatialPixel and sosa:FeatureOfInterest were set as rdfs:subClassOf geo:SpatialObject. Next, the topological Based on distinct concepts, English words are grouped into cognitive synonym datasets (synsets). WordNet is a huge database consisted of these lexical datasets.  Journal of Sensors relations between spatial objects, such as intersects, within, or contains, were established by the Topology Vocabulary Extension (relation_family) parameterized requirements class of the GeoSPARQL. The spatial semantic extension of RSISO through the GeoSPARQL is shown in Figure 3.

Temporal Rule.
In the time dimension, in situ data have a higher resolution, such as hourly precipitation data that represent rainfall within an hour of the day, while for remote sensing image data, based on the return period of the carrying satellite platform, there are different temporal resolutions, such as 10 days of Sentinel 2 MSI, where the imagery typically represents the result of the day and the temporal resolution of the image product varies (e.g., JRC Monthly Water History in months, JRC Yearly Water Classification History in years, and CHIRPS Daily). Time ontology can be used to define these times (instants or intervals). As shown in Figure 2, these remote sensing data and remote sensing products obtained the range time:Temporal Entity of the sosa:phenomenonTime object property or xsd:dateTime value by inheriting the sosa:Observation class. They were achieved in the W3C SSN ontology.

Thematic Rule.
Regarding the theme, in situ observation has explicit content information. For instance, rain gauges observe an amount of precipitation. Similarly, the application fields of the image products can be easily found because they are produced for specific topics, such as JRC Monthly Water History for surface water and CHIRPS Daily for precipitation.
Although the WMO has provided most of the application topics for remote sensing satellites, it does not explain the internal relationship between remote sensing satellites and application topics, so it is unable to expand the application field of remote sensing satellites according to a wide range of practical applications. Spectral rules were used for the classification of remote sensing images, and effective results were obtained [7,29,30]. Therefore, spectral rules were introduced into the RSISO ontology through the class RSISO:CharacteristicIndex.
According to the bands contained in the specific remote sensing images, the available characteristic indexes can be inferred. For example, according to the calculation formula of the Normalized Difference Water Index (NDWI), the green and near-infrared bands were needed to acquire the index. From the bands listed in Table 4, both Landsat 8 OLI and Sentinel-2 MSI can obtain the NDWI, and sequentially, the land cover of surface water can be obtained. The difference is that the near-infrared band corresponds to B5 and B8 in Landsat 8 OLI and Sentinel-2 MSI, respectively.
Considering the text description difference and semantic correlation of application objectives, the WordNet ontology [59] was introduced in the RSISO. The WordNet ontology, a lexical database of the English language, enables acquiring synonyms or related words through semantic relevance [60]. More available data can be discovered by extending the requirement text and matching the instances of sosa:Observation of in situ observation, RSISO:Band of image   As shown in Figure 5, the ontology determines the classes and their relationships, which define different types of observation data and information available from these results. The classes ImageProduct and RemoteSensedImagery were designed as a subClassof class Result of SSN. They have the hasPixel property to describe the SpatialPixel contained in them. The difference between them is that the Band contained in ImageProduct refers to specific application objectives; however, Band in RemoteSensedImagery represents the reflection value in a particular band range. For example, the values of the "water" band in the ImageProduct JRC Monthly Water History stand for permanent water, seasonal water, or no water. In contrast, the values of Band in Remote-SensedImagery cannot be used directly to identify the type of land surface cover. The values of particular bands can be used to obtain the land surface cover index, such as the NDVI and NDWI, through a mature and reliable normalized calculation method. In the proposed ontology, CharacteristicIndex represents the type of surface cover that can be inferred from the bands that RemoteSensedImagery has. Because Character-isticIndex cannot be observed by sensors, it was designed as a subClassOf Property. The parts of classes and properties created by Protégé are shown in Figure 5.
Classes or individuals are linked through the object properties defined in the ontology. These object properties, as interactions between different objects, are the significant elements in the ontology. The main object properties in the proposed ontology are listed in Table 5.

Inference Based on Rules.
Rules were defined in Protégé to provide principles for logical reasoning based on experiential knowledge. Experiential knowledge originates from the long-term practice of professionals. The ontology was proposed to find the potential application domains of remote sensing imagery, image products, and in situ sensors. The in situ sensors themselves have clear observation targets, and the image products can also distinguish their meaning by the band names. Therefore, the rules are mainly used to discover the potential application domains of remote sensing images by calculating different land cover indices. The parts of the rules created for the ontology are listed below. In situ Daily Water flow volume OB_PR20150826 In situ Daily Precipitation 1 The value of band "water" has three opinions: 0-no data; 1-not water; 2-water. 2 The value of band "precipitation" represents the amount of rainfall.    hasPixel RemoteSensedImagery ImageProduct SpatialPixel Relation between imagery and spatial pixels. These pixels own their own spatial information and can be combined into features of interest.
hasCharacteristic RemoteSensedImagery CharacteristicIndex Relation between remote sensing imagery and particular characteristics. This property is derived from the bands of the remote sensing imagery and the rules of calculating the characteristic index through the bands. On the one hand, we can acquire various heterogeneous observations which are observed in a specified period. For example, all observations, including between "2015-08-23" and "2015-09-10," can be obtained by following the SPARQL sentence based on the presented RSISO ontology model. The results are shown in Figure 6. "2015-08-23T00:00:00 +08:00"^^xsd:dateTime && ?datetime <= "2015-09-10T00:00:00+08:00"^^xsd:dateTime) } On the other hand, sosa:observedProperty can be used to obtain the properties of in situ observations, such as water level and flow volume and precipitation. Combined with the rule definition for remote sensing imagery in Section 3.2, observations with similar or complementary properties can be found, such as the water field in JRC Monthly Water History and NDWI in Landsat 8 OLI imagery. In addition, the retrieval of heterogeneous data can be achieved according to spatial conditions, by introducing the functions of GeoS-PARQL. Combining the semantic query conditions of time, space, and topic, the specific available data can be retrieved. Combining the formulation of reasoning rules and the semantic conditions of time, space, and topic, the specific available data can be obtained more easily. This is beneficial for the fusion analysis of satellite-earth observations.

Integrated Application with In Situ and Remote Sensing
Observation and Image Products. There are an increasing number of in situ and satellite-based sensors, and a large number of image products using remote sensing imagery are emerging. This provides a great opportunity for research work, such as satellite-ground cooperative monitoring, image product correction, and remote sensing image inversion evaluation. At the same time, it also brings challenges to the management mode, association technologies, and query methods of the heterogeneous observed data. Up to now, some studies have applied spectral rules to the automatic application of remote sensing images [29,30] without, however, considering in situ observations. The proposed ontology model, based on the mature SSN ontology for managing in situ sensor networks, extends the objects of remote sensing imagery and image products so that these observation results can be uniformly managed and queried, which will be very beneficial to the integrated application of heterogeneous observation data.

Prior Knowledge Expressed in SWRL Can
Be Extended and Shared Better. The reasoning capability provided by RDF description logic and SPARQL is limited. On the basis of OWL, we established some rules based on SWRL, so as to improve and optimize knowledge reasoning. Furthermore, based on the formal expression of SWRL, which is a W3C specification, experts in different fields, such as precision agriculture, climate change, extreme weather, and emergency decision-making, can transform their knowledge into the  Journal of Sensors rules of ontology so that reusing and sharing their knowledge becomes feasible.

Image Products Can
Be Used More Widely Based on Semantic Queries. With the accumulation of remote sensing image data, different kinds of thematic image products have been released by scientists or institutes, for example, Global Surface Water by the European Commission's Joint Research Centre [62] and Rainfall Estimates from Rain Gauge and Satellite Observations (CHIRPS) by the Climate Hazards Center at the University of California, Santa Barbara [63]. These image products have great reference value for related research, but because they are professional and not popular, their role is not easy to be recognized. The ontology proposed in this paper provides a channel to manage and discover these imaging products, taking into account both in situ and remote sensing observations.

Conclusions
In this work, we developed an ontology-based framework for integrating remote sensing imagery, image products, and in situ observations. It was developed by expanding the W3C SSN ontology with spatial, temporal, and thematic rules. The detailed process of ontology construction and rule establishment was demonstrated. Combined with some actual remote sensing images, image products, and in situ observations, some semantic queries based on DL Query and SPARQL were implemented, and the rationality and feasibility of the ontology and framework proposed in this paper were proved. In addition, more rules can be built up by SWRL based on existing experience and knowledge. It will be beneficial for the reuse and sharing of professional knowledge and taking full advantage of impact products.
The current work realizes the ontology prototype of integrated management and preliminary reasoning of remote sensing images, image products, and in situ observation data. However, more semantic obstacles will be encountered in the comprehensive application of multisource data. The ontology proposed in this paper needs to be further integrated with the existing ontology and improved in more complex engineering projects.

Data Availability
The ontology file developed in this study is available from the first author (Chao Wang, c.wang@whu.edu.cn) upon request.

Conflicts of Interest
The authors declare no conflict of interest.