Pedestrian detection at night time in FIR domain: Comprehensive study about temperature and brightness and new benchmark
Introduction
As the number of traffic accidents and deaths have increased steadily, the interest in preventing traffic accidents of pedestrians and reducing damage has been popularized in the global area. The accident rate at night is generally more than two times higher than that at daylight. The invisible or hard visible case often increases the likelihood of accidents, which might cause death. In the darkness, a monitoring and autonomous emergency braking (AEB) system is needed for a pedestrian safety system because of the higher accident and mortality rate at night. The advanced driver assistance system (ADAS) is an exemplar that makes the car stop at an emergency by monitoring the front scene of the car using a variety of sensors. Detecting pedestrians accurately and quickly in various environmental changes is important for implementing the AEB system. In Euro NCAP, the detection of pedestrians is becoming more important for protecting pedestrians because since 2016, detecting pedestrians has been a five star requirement when evaluating an advanced emergency brake system (AEBS) [1], [2], [3]. Most recent studies on these pedestrian detection topics are based on color images. On the other hand, color-based pedestrian detection is ineffective in dark environments particularly at night. As sensor technology has developed, infrared (IR) cameras, which are used widely in the military field, have become commercially available, and far infrared (FIR) camera sensors have been used to solve the problems of color-based pedestrian detection. Because the FIR camera sensor is not influenced by lighting and can detect the heat distribution of object to spatially display the objects. The detection distance is longer than the driving beam, and a human's temperature is radiated optimally at the far infrared spectral band [4]. Based on the characteristics of infrared rays, the pedestrian detection field has been expanded by applying color-based detection techniques to infrared images and fusing the color and infrared signatures with performance improvement, thereby analyzing the different properties from the two domains [5], [6]. This paper reports the findings of a comprehensive study of the temperature and brightness for pedestrian detection. The representative disadvantage of FIR brightness is the variability due to thermal infrared data loss and the contrast of the scene, which makes the classification of pedestrians more difficult. On the other hand, the temperature data is unaffected directly by the background change, except for the variation by the seasonal effect, which is more stable than the brightness because the pedestrian temperature does not change abruptly and is limited to a specific thermal range with thermoregulation. Therefore, this paper addresses the novel pedestrian detection method using the temperature, which is a further development from previous work [50].
This paper presents three main contributions. First, a YU FIR pedestrian detection database was constructed across each season through data acquisition 6 times to compensate for the disadvantages of thermal infrared data, which are vulnerable to environmental changes. Based on this, four versions of the thermal infrared radiometry ACF method were introduced using the temperature input data with normalization using the commonsense maximum temperature of humans for pedestrian detection. Finally, the performance of the brightness-based baseline method was evaluated by contrast variations and compared with the performance of the temperature-based method. The change in performance of the temperature-based method by seasonal variations was assessed. The reason for why the temperature is superior to the brightness was also analyzed within the discrimination of the exemplar methods in all seasonal cases.
The sections of the remaining paper are organized as follows. Section 2 describes the FIR brightness-based method and dataset with the weak point of brightness. Section 3 explains the brightness-based baseline ACF method and presents four versions of temperature-based ACF methods with normalization using the commonsense maximum temperature of humans. Section 4 accounts for the experimental results regarding the radiometric temperature-based YU FIR pedestrian detection dataset and performance evaluation of the proposed and baseline ACF methods for the seasonal variation. Section 5 concludes this paper.
Section snippets
Related work
Because of the independence of illumination on the far infrared spectrum, pedestrians generally have high visibility at night in that the pedestrian's temperature is higher than the ambient temperature of the background. This is one of the strong infrared thermal signatures of pedestrians. The researchers generated the region of interests (ROIs) on these infrared properties. They then validated the ROIs to distinguish the clutter and the pedestrian with a high probability. This paper
Proposed method
In this section, the brightness-based baseline ACF method [49] is explained and temperature-based ACF methods are presented. This section suggests four versions of temperature-based ACF methods with simple and effective normalization using the commonsense maximum temperature of humans.
Experimental results
This section introduces the YU FIR pedestrian detection dataset. The performance variation was examined according to the change in contrast in the brightness image by different grayscale mapping parameters and the performance of the method using temperature was compared with the performance of the method based on the grayscale image on the seasonal variation. In addition, the performance of pedestrian detection was evaluated regarding the seasonal changes in temperature and data analysis was
Conclusions
This paper presents a novel method to detect pedestrians in the FIR domain at night. A search of the FIR-based pedestrian detection techniques and the database showed that the characteristics of the infrared data applied to their experiments mostly included the brightness. On the other hand, the brightness can be distorted easily by the contrast of the scene, causing the pedestrian detection performance to degrade. This paper introduces radiometric temperature-based methods to cope with this
Acknowledgment
This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2016R1D1A1B03930076).
Taehwan Kim received B.S. degree in Electronic Engineering from Yeungnam University, Gyeongsan-si, Korea, in 2016. He is currently a graduate student in the Dept. of Electronic Engineering, Yeungnam University, Gyeongsan-si, Korea. His current research topic includes object detection using infrared sensor camera and computer vision technology.
References (54)
- et al.
Pedestrians’ detection in thermal bands–critical survey
J. Electr. Syst. Inf. Technol.
(2015) - et al.
Robust and fast pedestrian detection method for far-infrared automotive driving assistance systems
Infrared Phys. Technol.
(2013) - et al.
Pedestrian detection in thermal images: an automated scale based region extraction with curvelet space validation
Infrared Phys. Technol.
(2016) - et al.
Pyramid binary pattern features for real-time pedestrian detection from infrared videos
Neurocomputing
(2011) - et al.
Robust pedestrian detection in thermal infrared imagery using a shape distribution histogram feature and modified sparse representation classification
Pattern Recognit.
(2015) - et al.
Fusion of color and infrared video for moving human detection
Pattern Recognit.
(2007) - et al.
Adaptive contour-based statistical background subtraction method for moving target detection in infrared video sequences
Infrared Phys. Technol.
(2014) - et al.
An iterative integrated framework for thermal–visible image registration, sensor fusion, and people tracking for video surveillance applications
Comput. Vis. Image Understand.
(2012) The next steps for vulnerable road user AEB assessment
Two years activities in WP29/ITS informal group
Development of pedestrian detection system using far-infrared ray camera
SEI Tech. Rev. Engl. Ed.
Detection and Tracking in Thermal Infrared Imagery
Night-time pedestrian detection based on temperature and hogi feature in infra-red images
Int. J. Simul. Syst. Sci. Technol.
A shape-independent method for pedestrian detection with far-infrared images
IEEE Trans. Veh. Technol.
Pedestrian detection and tracking with night vision
IEEE Trans. Intell. Transp. Syst.
Early detection of sudden pedestrian crossing for safe driving during summer nights
IEEE Trans. Circuits Syst. Video Technol.
Intensity self similarity features for pedestrian detection in far-infrared images
Pedestrian detection in far-infrared daytime images using a hierarchical codebook of SURF
Sensors
Robust background-subtraction for person detection in thermal imagery
A two-stage template approach to person detection in thermal imagery
Pedestrian detection for driver assistance using multiresolution infrared vision
IEEE Trans. Veh. Technol.
Pedestrian detection in far infrared images based on the use of probabilistic templates
Detection and tracking of pedestrians in infrared images
Pedestrian detection using infrared images and histograms of oriented gradients
Pedestrian detection in infrared images based on local shape features
Contrast invariant features for human detection in far infrared images
Cited by (28)
Improved robustness of vision transformers via prelayernorm in patch embedding
2023, Pattern RecognitionIs there a relationship between time pressure and pedestrian non-compliance? A systematic review
2023, Transportation Research Part F: Traffic Psychology and BehaviourMultispectral interaction convolutional neural network for pedestrian detection
2022, Computer Vision and Image UnderstandingCitation Excerpt :Since not all environments can be handled with a single sensor, the need for all-day detection has emerged (especially at night) since bad visibility impacts the visible sensor data, and pedestrians are almost impossible to detect from visible sensor data only (Leykin et al., 2007; Hwang et al., 2015; González et al., 2016). On the other hand, in the case of an IR sensor, the heat distribution from the pedestrian can be detected without being affected by illumination changes (Kim and Kim, 2018). Therefore, the IR sensor is inevitable for round-the-clock detection, and the necessity of sensor data fusion is emerging.
Pedestrian safety using the Internet of Things and sensors: Issues, challenges, and open problems
2022, Future Generation Computer SystemsMulti-task faster R-CNN for nighttime pedestrian detection and distance estimation
2021, Infrared Physics and TechnologyMonocular pedestrian orientation estimation based on deep 2D-3D feedforward
2020, Pattern RecognitionCitation Excerpt :In [20], multiple parts of pedestrians are respectively recorded and learned by different detectors, which are combined by a decision tree, to solve occlusion problems. In [21–23], thermal images are studied and applied in nighttime pedestrian detection. Thermal images take advantage of temperature features of human, and have positive effects on differentiating pedestrians from the background environment.
Taehwan Kim received B.S. degree in Electronic Engineering from Yeungnam University, Gyeongsan-si, Korea, in 2016. He is currently a graduate student in the Dept. of Electronic Engineering, Yeungnam University, Gyeongsan-si, Korea. His current research topic includes object detection using infrared sensor camera and computer vision technology.
Sungho Kim received the M.S., Ph.D. degrees in Electrical Engineering and Computer Science from Korea Advanced Institute of Science and Technology, Korea in 2002, 2007, respectively. Since 2010, he has been a Professor of Electronic Engineering at Yeungnam University. His current research interests include target detection, object recognition, multi-sensor fusion, and deep learning.