Bag-of-visual-phrases and hierarchical deep models for traffic sign detection and recognition in mobile laser scanning data

Abstract

This paper presents a novel algorithm for detection and recognition of traffic signs in mobile laser scanning (MLS) data for intelligent transportation-related applications. The traffic sign detection task is accomplished based on 3-D point clouds by using bag-of-visual-phrases representations; whereas the recognition task is achieved based on 2-D images by using a Gaussian-Bernoulli deep Boltzmann machine-based hierarchical classifier. To exploit high-order feature encodings of feature regions, a deep Boltzmann machine-based feature encoder is constructed. For detecting traffic signs in 3-D point clouds, the proposed algorithm achieves an average recall, precision, quality, and F-score of 0.956, 0.946, 0.907, and 0.951, respectively, on the four selected MLS datasets. For on-image traffic sign recognition, a recognition accuracy of 97.54% is achieved by using the proposed hierarchical classifier. Comparative studies with the existing traffic sign detection and recognition methods demonstrate that our algorithm obtains promising, reliable, and high performance in both detecting traffic signs in 3-D point clouds and recognizing traffic signs on 2-D images.

Introduction

Traffic signs provide road users with correct, detailed road information, thereby ensuring road users to rapidly approach their destinations. Traffic signs also function to regulate and control traffic activities, thereby ensuring traffic safety and traffic smoothness. To facilitate management and improve efficiency, an effective and automated traffic sign recognition system is urgently demanded by transportation agencies to monitor the status and measure the usability of traffic signs. In addition, the accurate functionality and localization information of traffic signs also provides important inputs to many intelligent transportation-related applications, such as driver assistance and safety warning systems (Zheng et al., 2004, Cheng et al., 2007) and autonomous driving (Choi et al., 2012, Broggi et al., 2013, Seo et al., 2015). Specifically, the absence or lack of visibility of necessary traffic signs might cause inconvenience to road users, sometimes even cause terrible traffic accidents and casualties. Therefore, exploring and developing effective, automated traffic sign detection and recognition techniques are essential to the transportation agencies to rapidly update traffic sign inventory and improve traffic quality and safety.

Traditionally, the measurement and maintenance of traffic signs were basically accomplished through field work, where field workers from transportation agencies conducted on-site inspections and maintenance on a regular basis. In fact, such field work was time consuming, labor intensive, costly, and inefficient to operate large-scale, complicated road networks. Recently, with the advance of optical imaging techniques, mobile mapping systems (MMS) using digital or video camera(s) (Murray et al., 2011, Brogan et al., 2013) have emerged as an effective tool for a wide range of transportation applications. The images collected by a camera-based MMS have provided a promising data source for rapid detection and recognition of traffic signs along roadways. However, the MMS images suffer greatly from object distortions, motion blurs, noises, and illumination variations. In addition, caused by viewpoint variations, traffic signs are sometimes occluded or partially occluded by the nearby objects (e.g., trees) in the images. Therefore, it is still a great challenge to achieve high-quality, high-accuracy, and automated detection and recognition of traffic signs from MMS images.

Since last decade, benefitting from the integration of laser scanning and position and orientation technologies, mobile laser scanning (MLS) systems have been designed and extensively used in fields of transportation, road feature inventory, computer games, cultural heritage documentation, and basic surveying and mapping (Williams et al., 2013). MLS systems can rapidly acquire highly dense and accurate 3-D point clouds along with color imagery. The 3-D point clouds provide accurate geometric and localization information of the objects; whereas the color imagery provides detailed texture and content information of the objects. Therefore, by fusing imagery and 3-D point clouds, MLS systems provide a promising solution to traffic sign detection (based on 3-D point clouds) and recognition (based on imagery).

In this paper, we present a novel algorithm combining bag-of-visual-phrases (BoVPs) and hierarchical deep models for detecting and recognizing traffic signs from MLS data. The proposed algorithm includes three stages: visual phrase dictionary generation, traffic sign detection, and traffic sign recognition. At the visual phrase dictionary generation stage, the training MLS data are supervoxelized to construct feature regions for generating a visual word vocabulary, and to construct spatial word patterns for generating a visual phrase dictionary. At the traffic sign detection stage, individual semantic objects are first segmented, supervoxelized, featured, and quantized to form BoVPs representations. Then, traffic signposts are detected based on similarity measures between the BoVPs of the query object and the semantic objects. Finally, traffic signs are located and segmented via percentile-based analysis. At the traffic sign recognition stage, a Gaussian-Bernoulli deep Boltzmann machine (DBM) based hierarchical classifier is applied to the registered traffic sign regions to recognize traffic signs.

The main contributions of this paper are as follows: (1) a DBM-based feature encoder is proposed to generate high-order feature encodings of feature regions; (2) a supervoxel-based BoVPs model is proposed to depict point cloud objects; (3) an extended voxel-based normalized cut segmentation method is developed to segment overlapped semantic objects. In this paper, “high-order feature” denotes the high-level abstraction of a set of features or the “feature of features”; whereas “low-order feature” denotes a specific single feature.