‘ Sold by our own shadow ’ Human detection in uas based boundary monitoring

The opportunities provided by Unmanned Aerial Systems (UAS) in the collection of spatial data, create a need for geo-engineers to continuously contribute to novel application areas for the technology. This study looked at the application of UAS imagery in the detection of illegal migrants at unauthorised areas around international boundary regions. International boundaries can be made up of connected sections with crisp (man-made) and at times fuzzy (natural) areas that collectively distinguish neighboring territories from each other. Areas with fuzzy boundaries can easily become conflict hotspots as they are prone to attract cross national movements, resulting in crime, military action and illegal migration. This is a growing challenge in world economies. In developing countries, human security details often man and patrol international boundary areas depending heavily on visual detection to monitor cross border movement. Visual patrol has great limitations and is often ineffective especially during the night. The main objective of this research is to propose a conceptual design for an early detection alert system suitable for use at border controls that is based on UAS imagery. It utilises a shadow index on an RGB dataset to test and validate the detection of human candidates in designated hotspots. A detection rate of 92% is reported. Alert information derived from the image detection is to be sent directly to the smart phones of ground patrol staff who navigate to the identified location for verification and mitigation.


Introduction -Overview and Objectives
Boundaries have been at times been described as invisible planes connecting and controlling the use of sovereign air space up to the minerals below the earth (Asiwaju, 1985).International boundaries are defined as natural or man-made areas that separate two sovereign states in order to allow for each unit to organise their affairs efficiently (Boggs, 1940).The affairs referred to here include economic, military, political, social and cultural in nature.Recognition of boundaries has profound effects on migration flows (legal and illegal), environmental preservation, economic performances and bi-lateral relations.
According to (Prescott and Schofield, 2005) most African boundaries exist as a result of colonial heritage accredited to 19 th century agreements and treaties that have been accepted as is or adjusted in coherence with the current expectations of the concerned nations.The nature of these boundaries especially where no restricting physical demarcation exists can often be a source of conflict between the parties (Giroux et al., 2009).
Cross border conflicts are challenging to predict and handle.They arise due to the failure of the neighbours to concur on the activities of one party in the boundary area that puts the security or sovereignty of the other party at risk.Activities such as military advances or attacks, sprawling of illegal settlements in border area, cross-national crime and illegal cross border migration are a growing challenge in world economies.They represent one way in which nationals from one country may be found detrimental to a neighbour, prompting a need for physical demarcation or patrolling of the boundary to curb effects.In general, conflict can impact negatively on the economies and the livelihoods of neighbouring citizens resulting in loss of life, strained foreign relations and expensive boundary resolution exercises have been quoted at approximately $USD500,000 in some instances in cases where the borders are +-200km long (RCMRD technical reports, 2014).In the past decade geo-spatial technologies have begun to find a place in international boundary control and conflict resolution.
Remotely sensed image data sets have proved reliable in providing ground information that could affect border delimitation and impact on the sovereignty of the nations involved (Sandström, 2003).
Multi temporal imagery has begun to have extensively use in monitor historical, man-made and environmental changes around international boundary areas.High resolution image datasets are now acceptable as evidence in forums such as the International Court of Justice (ICJ).This paper focuses specifically on the use of a shadow index based system for human detection in border patrol.The main objective of this study is to present the image processing component that is part of an image based early warning system that can be instrumental in alerting controllers on unauthorised human movement in boundary transition areas.The novelty of this study lies in its methodology which includes the application of a shadow and feature detection index in form of the maximally stable exterminable region detection.

Related Work
International boundary conflicts require systems that pro-actively highlight any likely conflict related challenges in time for intervention and humanitarian mitigation.Early warning systems (EWS)are proactive approaches to crisis or disaster management that are able to detect the signs of a catastrophe prior its occurrence in order to alert reaction teams (Basher, 2006).The paper discusses the first component of two (Human detection by image processing), and also gives an overview of the 'Geographic Information Systems' (GIS) based short message service (sms) alert component.
As highlighted above, image processing has significantly impacted on methods of conflict resolution.
The Qatar / Bahrain (2001) case where imagery was presented as supporting documentation in successfully resolving the maritime boundary conflict (AUBP, 2013).In addition, in March 1994, when Cameroon filed a case before the International Court of Justice (ICJ) concerning a dispute questioning the sovereignty over the Bakassi Peninsula and remote sensing provided resolving solutions.In 2011, the Cartographic Section of the United Nations held a workshop where the potential of SPOT and Landsat imagery was tested for its suitability in resolving the Bakassi conflict.The imagery was used to produce different 3D visualisations of the ridge that helped decision makers reach an agreement on a more precise boundary location.In the separate case of South Sudan in 2011, SPOT satellite image maps along their entire boundary length were produced as part of a supporting tool for the negotiations.This shows the place of aerial image dataset in solving conflict and in realigning international boundaries for civil harmony.Commercial remotely sensed imagery has therefore emerged as a pivotal data set in boundary conflict management and territorial negotiations for international law.

Data Specifications and Study Area
The study area selected for the system is the Southern African Development Community (SADC) region.SADC is a regional economic community comprising 15 member states including Angola, Botswana, Democratic Republic of Congo, Lesotho, Madagascar, Malawi, Mauritus, Mocambique, Namibia, Seychelles, South Africa, Swaziland, Tanzania, Zambia and Zimbabwe.It was established in 1992 and is committed to regional integration and poverty eradication within Southern Africa through economic development and ensuring peace and security.Figure 1.Below illustrate the countries that constitute the SADC region study area.SADC still has countries within it that have delimited (not demarcated) international boundaries hence it is ideal for implementation of the border alert system.The overall status of most SADC boundaries is shown in Table 1 below

Human detection using RGB imagery
Automated approaches in image processing offer advantages over manual techniques as they are more 'elegant' in terms of logic, efficiency and output quality.Their limitations may include the fact that they require expert skills for their development and maintenance and may at times be insensitive to some imaging scenarios where a human interpreter can make judgements based on area knowledge or experience.The research proposes an automated approach to identify human detail in imagery.High resolution nadir RGB drone imagery is tested for human presence.The detection of humans in high resolution imagery is not a trivial problem.Complexities arise especially in nadir imagery due to the small object size which makes it difficult to distinguish target candidates from background clutter (See Figure 2).This overlap with background often results in a need for an increased number of verifying cues or a larger search space to positively identify required features.For the detection of human detail, the research hypothesis used in this study proposes that a human being can be detected by the morphology of their shadow and that this shadow can be separated from body in RGB imagery because shadow will be more spectrally homogenous than the body components.In order to validate the hypotheses, the study opts for a colour transform as part of the image pre-processing towards shadow identification.The research uses a colour space that provides better separation between chromacity and intensity than the RGB colour space.The hue, saturation, and value (HSV) colour space is selected as it highlights the presence of shadows.In the case of the HSV colour model, shadows tend to have high saturation and low value.
HSV has strong similarities with the human visual system and its advantage is that it is are able to separate luma / intensity from Chroma / colour information for practical applications more fully than traditional colour models like RGB.The HSV model specifies colour as a triple (H, S, V) where H is an angle counter clockwise from the vertical which begins at a line representing from white to red and works through the colour spectrum.Shadows appear darker than other objects in imagery hence spectral analysis can be used to differentiate shadows from other features.Image enhancement can assist in fully capturing the available shadow information.To initiate the identification of like human detail, the shadow index (Ma et al., 2008) in HSV is as follows: Where S and V represent the digital number values in the hue and value bands of the image composite Shadows ranges from -1<shadow index<0 in this ratio.Challenges arise however in the morphological complexities of humans whose shadow is never standard even in standing position.Having successfully isolated human shadow candidates in the imagery the study then separates shadow pixels from the body.
Shape constraints are used to mesh out smaller pixels and remain with typical shadow areas.The body not the shadow is the required candidate in this case but it is still connected in the processed image with shadow and background clutter.The separation of background clutter from foreground involves the extraction of pixels of interest into the image fore at the expense of any pixels that do not qualify into foreground.Background separation techniques in general are very commonly used in objects systems.
To then separate body and shadow elements a technique of identifying maximally stable regions is employed.
Maximally stable exterminable regions (MSER) are not commonly applied to image segmentation problems but they are usually useful for image registration problems and key point identification for image matching (Nistér, 2008).They therefore are commonly grouped in the key feature point extraction family together with the of scale invariant feature transformation (SIFT), binary robust invariant scalable key points (BRISK) (Leutenegger, 2011) etc. useful in template matching, panorama stitching and image mosaicking tasks.We adapt the MSER here as a novel an incorporation into our human identification task in order to identify invariant regions within our candidate blobs.A region is considered stable if its variation is less than the value of the maximum area variation parameter to be considered stable.Modifications of the MSER algorithm think of the intensity profile of the image as representing a series of buckets with a hose turned on at one of the buckets.As the water fills into the bucket, it overflows to the next bucket starts filling up.Smaller regions of water join and become bigger bodies of water, and finally the whole area gets filled.As water is filling up into a bucket, it is checked against the MSER stability criterion.Regions therefore appear, grow and merge at different intensity thresholds.
In the research context MSER regions in the original RGB image are initially determined.It is noticed that during this determination the number of regions formed as corresponding to the body detail are greater number of segments for the shadow elements which are more monotone resulting in one maximally stable region overally.This observation enables the researcher to use the MSER to split the body from its shadow by filtering the regions using an area threshold that only permits the smaller regions to be enforced as these are the desired human candidates.To evaluate the performance of the final candidates from the human detection method, the following equation is computed Where TP is the true positive and FN is the false negative rate respectively as defined below: True (TP) are those human candidates that have been correctly identified by the shack extraction algorithm as being a shack unit and are in fact labelled as human in the reference data.
False Negatives (FN) are those background details that have in fact been identified as human candidates in the reference data.
Figure 3 below is a summary of the human detection method from RGB imagery that feeds into the entire warning system.

Detected human candidate data transmission to border patrol
The images enable a location computation to be established of possible intrusion points within patrol areas.These details are transmitted and stored in a GIS based server at the ground receiving station.
Where a true positive was identified within the image data the system sends a warning short message service (sms) via sms gateway to the border patrol's tablets, mobiles and related gadgets.In response the patrollers scans the area, guided by the coordinates given in the alert and monitor any violations.Figure 4. is an extension of Figure 3, giving more detail on the overall system dynamics that will take place in ground station processer in order to successfully send an sms alert to the client.

Results and Analysis
The research hypothesis used in this study proposed that a human being can be detected by the morphology of their shadow and that this shadow can be separated from body in RGB imagery because shadow will be more spectrally homogenous than the body components.In the methods described above a shadow index was applied in the HSV image channel in order to isolate the morphology or shape of the shadow of human details.Shadows are often cast by elevated objects that are illuminated at the time of exposure.Although shadow is usually not desired in imagery as it can occlude key features it can at time provide additional cues on feature shapes and size.It was observed that in imagery with homogenous backgrounds the shadow elements are easily brought to the foreground using the index but at times these were extracted with a portion of the body itself.This is also due to the fact that there are two types of shadows; cast shadow and self-shadow.The detection performed satisfactorily, however there were some noisy false positives possibly due to self-shadow effects.The human body has some areas of selfshadow as well where light is occluded.The shadow index positively identified both cast and self-shadow resulting in blobs that have the true shadow connected to the body element.These were filtered away using an erosion kernel or morphological filter.Figure 5 below illustrates output from the shadow index and the filtering.Limitations included the fact that with a more complex background more misclassification may occur.Moreover, with different shadow silhouettes, the human body and its shadow may become increasingly difficult to separate.Having filtered away noisy shadow, the MSER algorithm was tested for its ability to separate the body from shadow as highlighted above.The strength of the MSER is in its ability to identify regions of minimum variance within the image guided by an area and variance parameter.The MSER when applied to the original RGB image in order to identify the body detail and shadow detail as separate stable regions.The body detail has less homogeneity than shadow and results in several small MSER regions while the shadow is located as a single stable region.The intersection of the boundary of the MSER regions in the RGB image with the detected shadow candidates is used to differentiate shadow from body.An area threshold on the MSER is used to ensure that smaller MSER regions are assigned to body.
Figure 6 below illustrates the steps taken in separating shadow and body using MSER algorithm.As highlighted above, the human shadow was isolated from the body detail and the final candidates were identified.The test image was evaluated for true positives and false positives as described in Equation 1 and it yielded a detection accuracy rate of 92% (from equation 2) for the identification of human detail.This is a satisfactory performance metric for larger scale applications of the method.RGB imagery however is based on the use of a passive sensor and will be best for daytime human detection.
At night the RGB system may fail to detect human candidates as the sensor is light dependent.In this instance infrared imagery may be a helpful alternative as it performs uniformly in both lighted and dark surroundings.Moreover, detection based on shadow is dependent on the posture and time of day.This may affect the morphologically determined shadow computations presented here.The human detector is part of a larger scale frame early warning detection architecture for conflict repair at international border regions where unathorised migration is rife as already highlighted in the literature above .The paper highlighted the image processing detail mainly and future work will focus on the early warning aspect.
It is hoped that on full implementation the system will be a positive addition to automated border patrol for peace and sustainable development in African countries.This human and vehicle model can be extended and adapted to other research problems such as that of identifying poachers or security surveillance applications.

Conclusion
International law and conflict resolution ca not be fully addressed without the use of spatial data derived from primary data such as aerial imagery.Remote sensing provides opportunities for the

Figure 1
Figure 1 SADC region of Africa (Source 1 )

A
system is designed to compute a change detection based on the comparison of drone derived scenes to the latest satellite image map of the area.Where human or vehicle detail is identified in the imagery an alert is immediately raised via a sms gateway to the ground patrol at border areas who can then verify the alert.The overall system flow begins with an unmanned aerial system (UAS) platform and an RGB imaging sensor in flight being controlled via radio signals by a ground based controller.Two possible sensor types are proposed for the image scans and these are an optical camera by day and an infrared camera by night.This will enable human and vehicle detail to be identified throughout the day without the limitations of the passive sensing component of an optical camera.During deployment the UAS collects successive image scenes of the area of interest and continuously sends these via wireless transmission through the facilitation of a file transfer protocol (FTP) transmitter.

Figure 2 .
Figure 2. RGB image showing humans in unvegetated area, red circles highlight background clutter 2 Figure 4.21 represents the hexagonal relationship of colour within HSV model as one transforms from the RGB colour space.RGB to HSV transformation is a non-linear and reversible transformation.

Figure 3 .
Figure 3. Boundary monitoring flow diagram for border patrol reaction

Figure 4 .
Figure 4. Boundary monitoring early warning system architecture

Figure 5
Figure 5. a) Shadow detection based on HSV index b) Further segmentation using morphological shape characteristics of shadow to remain with most probable shadow candidates (less noise)

Figure 6
Figure 6.a) and b) show detection of human shadow plus body by MSER while c) and d) show separation of shadow and body using area threshold applied on MSER collection of large scale ground data when Unmanned Aerial Vehicles (UAVs) and other mobile vehicles are used even in the absence of satellite imagery.UAV platforms and sensors have continued to improve impacting positively on the imaging quality.A model driven image scene reconstruction technique was presented in this study for the identification of vehicle and human detail from drone derived imagery.The technique is described as model driven as it is based on a pre-known model of human morphology in nadir imagery.Image morphology is used as a key component of the feature extraction and classification steps.A detection rate of 92% for human detail.The detection model is accepted as valid based on the accuracy matrix.

Table 1 .
: The status of some of the borders in Southern Africa adopted from(Mbano, 2011)