Vessel detection and classi fi cation from spaceborne optical images : A literature survey Ur š

Highlights • A review of 119 papers on ship detection and classification from optical satellite.• From 1978 to March 2017, showing an exponential growth in the number of papers.• Most published methods have very limited validation.• While big steps have been made, automatic algorithms are still far from perfect.• Increase in new observation and processing capabilities promises rapid advances.


Introduction
Measuring and monitoring human activity at sea is a topic of major and increasing interest. From fishing, drilling, exploration or cargo transport, to carrying passengers or tourism, efforts are expendedby governmental as well as commercial actorsto remain aware of what is going on. Maritime Domain Awareness (MDA) has been defined as the effective understanding of any activity associated with the maritime domain that could impact upon the security, safety, economy, or environment (IMO, 2010); and the maritime domain has been defined as all areas and things of, on, under, relating to, adjacent to, or bordering on a sea, ocean, or other navigable waterway, including all maritimerelated activities, infrastructure, people, cargo, and vessels and other conveyances (EU, 2008;US Government, 2004). To achieve proper MDA, a lot of information is needed. MDA is enabled by the combination of information from land, sea-based, airborne, and satellite sensor systems, augmented with heterogeneous information from geographical information systems (GIS) and vessel information repositories (Dekker et al., 2013). But in any case, knowledge of positions and behaviours of vessels is a cornerstone. Vessel detection has a wide range of applications, in the areas of maritime safety, marine traffic, marine pollution, maritime spatial planning, fisheries management, illegal fishing, defence and maritime security, maritime piracy, irregular migration, border control, etc. Ships can be easily discerned in optical images taken from space. Hence, there is merit in considering the contribution of ship detection from satellite optical images.
As to wording, within the literature two terms are widely used for man-made objects on the sea surface: 'ship' and 'vessel'. Although the majority of authors use the term 'ship' in their research, the expression 'vessel' is more generic, covering a variety of sea targets, ranging from smaller to bigger objects, and including very small boats, canoes, etc., as well as uncommon objects such as floating docks. Both terms will be deployed throughout this paper without an intended difference in meaning.
Many systems are available that can contribute to collecting knowledge on the presence and activity of ships. Usually, a distinction is made between cooperative and non-cooperative systems. With the former, the ships communicate information about themselves. With the latter, observation systems are used to collect information without any cooperation from the ships.
In accordance with a number of global, regional and national regulations, particular classes of vessels have to be equipped with shipborne transponders that transmit the vessel's identity and position at certain repeat intervals. One of the most common tracking systems is the Automatic Identification System (AIS), which is designed to automatically provide continuous information on location to other vessels and to coastal authorities (IMO, 2000). AIS was originally designed for short-range operation, and beside it, Long-Range Identification and Tracking (LRIT) was established as an international system by IMO (IMO, 2006). As a third widely used automatic reporting system, the Vessel Monitoring System (VMS) is used by authorities to keep track of fishing fleets (FAO, 1998). Whereas the carriage requirements for AIS and LRIT are globally set by IMO for the medium and large ships (see the IMO references for details), VMS carriage is instead regulated nationally or regionally. For some other systems see e.g. JRC (2008). While these systems are powerful tools to track the participating vessels, they only give a partial picture of the situation. Most small (< 300 tons) vessels do not need to carry AIS or LRIT, and small or even all fishing vessels do not carry VMS depending on the region. In addition, it has been observed that illegally operating vessels turn off or even spoof their mandatory position reports. Furthermore, LRIT and VMS data have a restricted access as they are collected by specific government authorities on specific legal bases. Therefore, we cannot count on the cooperative systems to provide comprehensive maritime domain awareness.
Concerning (non-cooperative) observing systems, the main sensors for maritime surveillance are, apart from visual sighting, optical camera, infrared camera and radar. These can be deployed from shore, ship, aircraft or satellite. Each sensor type and each platform has its own strengths and drawbacks, related to characteristics such as spatial resolution, update rate, range, coverage, persistence, latency and cost. Satellite-based sensors have the specific advantages of remote access, global reach, regular update and high data collection volume, so that in some scenarios they are the only feasible option and in others the most economic one. The usage of satellite images is therefore an essential tool to find vessels on the sea. In particular, satellite-based radar images, usually as Synthetic Aperture Radar (SAR), have become popular for maritime surveillance: ships are detected relatively easily, and independently of the presence of clouds or daylight. But the interest in the potential of optical images for maritime surveillance has dramatically increased recently, maybe in the first place due to the increase in optical imaging satellites. Indeed, it will be shown later that both the number of optical satellites and the number of publications on ship detection in optical images have grown exponentially during the last decade. Under the EU's Copernicus program for Earth observation, data from the Sentinel-2 optical satellite are even collected routinely and provided free and open to all since December 2015 (EC, 2016). Therefore, a review on this topic seems timely.
From the point of view of image analysis, object detection has been one of the most popular research topics and challenging tasks in remote sensing science in the last decades. There are numerous examples of machine-assisted or fully automatic detection of (small) objects in the fields of geosciences, geography, planning, infrastructure control, engineering, nearshore fixed object mapping, and homeland security (Marshburn et al., 2009). Ship detection is a special case of (small) object detection, where the background has the particular characteristics of the sea surface (further discussed in Section 4.2.2). The demand for automated analytical methods for remote sensing data is driven by the plethora of existing Earth-orbiting sensors and their daily generation of terabytes of data with different spatial, spectral, radiometric and temporal resolution (Hay et al., 2005). The detection systems are therefore faced with the need to process massive amounts of incoming data, often with the requirement to react in near-real time (Bi et al., 2010). Automated real-time vessel detection is a key point to various maritime missions.
The word 'detection' is used in the literature sometimes in a wide and sometimes in a narrow sense. The complete vessel detection procedure refers to the detection in the wide sense, and is composed of three main sequential steps: 1. Vessel detection (in the narrow sense): finding vessel candidates in the image and locating them; 2. Vessel classification: discriminating detected targets between vessel/non-vessel and then getting the class of the vessel (e.g. fishing, tanker, cargo); and 3. Vessel identification: establishing the unique identity of the vessel (e.g. International Maritime Organization (IMO) number, Maritime Mobile Service Identity (MMSI) number, name).
In the literature, the term 'vessel recognition' is also found. Its precise meaning seems not to be the same for all authors, but it often overlaps with what is called here classification. The last step, identification, cannot generally be performed with the use of satellite imagery. Despite optimistic claims by some authorse.g. Heiselberg (2016) proposes that a spectral library of ship signatures will enable vessel identification using Sentinel-2 multispectral imageswe have seen no published proof of this, and it is hard to imagine that spectral or other signatures in optical satellite images can uniquely identify the hundreds of thousands of ships that exist in the world. The world merchant fleet alone counted over 87,000 ships in 2015 (EMSA, 2015), and only in a scenario where a-priori information leaves a few options, it may be possible that a satellite observation can decide on the identity of a ship. Therefore, vessel identification will not be comprehensively covered in this work, whereas the first two steps, detection and classification, as part of vessel detection workflow on optical images, will be taken up in detail (see Section 4).

Existing imaging systems for vessel detection
A large number of imaging systems can produce data for vessel detection. The most frequent image types can be divided into four categories: optical and reflected infrared, hyperspectral, thermal infrared, and radar. For optical and infrared images, two sub-categories can be further distinguished: video and night-time images; whereas for radar data, SAR (Synthetic Aperture Radar) can be additionally sub-categorised. These sensors can be deployed from various types of platform, and their usage is summarised in Table 1. The sensors may look downward, forward or sideways from their platform, and they have a wide range of complexity and operational cost.
Regarding the first sensor type in Table 1, optical and reflected infrared are grouped together because they have quite similar characteristics. Optical implies the visible spectrum detectable by the human eye (wavelengths approximately 400-700 nm), and reflected infrared covers the near-and short-wave infrared bands, up to 3 μm. These sensors are passive, meaning they rely on external illumination, generally from the Sun. The images are essentially photos (nowadays digital), and from a coastal location or from a ship they are partly being replaced by video (always in the context of vessel detection and maritime surveillance). However, for the satellite, this is still the main type of imaging sensor and the main topic of this review. Multi-spectral