TRAFFIC SIGN RECOGNITION AND LOCALIZATION FOR DATABASES OF TRAFFIC SIGNS

This article describes a developed real-time traffic sign recognition system (TSR) along with the algorithm to get a position of recognized traffic sign in order to store both the recognized traffic sign and its calculated approximate GPS position to a database. Such database can be used for example in navigation systems for finding route according to the restrictions on the roads given by traffic signs, or for local authorities to register traffic signs in a region.


INTRODUCTION
The traffic signs are one of the most important elements in a transportation infrastructure.They control the flow of traffic, give the right of way, inform a road user about the directions and distances along with guide to a destination, or warn in dangerous places.
In order to increase safety and comfort of a driver, the traffic sign recognition (TSR) systems have included the advanced driver assistance systems as the crucial parts.Nowadays, the commercial TSR systems are being deployed to cars for the purpose of recognition of speed limit signs or no overtaking sign.So, these systems can easily warn the driver in certain situations, e.g. if the speed limit is being exceeded or if the traffic lane is being crossed at the place where the car must not cross over any continuous (unbroken) centre line.Moreover, the TSR can be used for collecting traffic signs and locating GPS position.Subsequently, the collected data are being stored (incorporated into) to map databases of navigation systems or databases for local authorities [1].Afterwards, the navigation system can find the optimal route with respect to the restrictions given by the traffic signs such as speed, weight, height limits and etc.These databases of the traffic signs might be used as well by the local authorities to register all traffic signs in a region.The other way to process recognized traffic signs is to store it in the database distributed and replicated throughout the vehicular ad-hoc network.Information would therefore be accessible by any other road user and could be used to enhance the road safety [2][3][4].
Currently, many publications could be found which propose various methods and techniques how to detect and recognise the traffic signs.Classical methods of detection are based on colour segmentation in various colour spaces such as RGB, HSV, HSI, CIELab, CIELUV, CIECAM97s and others [5][6][7][8].Some authors use another approaches based on shape which are more robust with respect of different light conditions.There are not such problems with colours in pictures in case of bad illumination.Other possible approaches use machine learning algorithms, either for detection, classification, or for both.Several types of algorithms are possible to use, e.g.artificial neural networks (ANN) [9], support vector machine (SVM) [5][10], boosting [11], genetic algorithms [12] and others.One exceptional survey of the state-ofthe-art approaches for detection and recognition in this field you can find in work [13] which was published by Meng-Yin Fu and Yuan-Shui Huang.
In this paper, we will briefly introduce our proposed TSR system, our method of approximate position determining and a developed demonstrative application of collecting traffic signs to a database.In this work, we are mainly focused on the traffic signs approved by Vienna Convention [14] located in the Slovak Republic.

TRAFFIC SIGN RECOGNITION
The traffic signs have been developed for a driver to make him distinguish and understand their meanings simpler.The fact is that they are designed with different colours, shapes and pictograms.In this view, these distinctive features are the basic elements which are being used for the detection and the recognition in proposed TSR system.The detection is based on the colour and shape information.This particular procedure of traffic sign detection and recognition is shown in Fig. 1.Firstly, the system acquires an image using a video source like a video camera or a video file.We have decided to use a colour HD web camera.
The next step after the image has been acquired is the image preprocessing phase.To put it in other words, the image file is prepared to the detection stage, e.g. the system removes the noise from the image file.This process is objectively necessary when the video source contains the compression artefacts.In this stage, we chose to use a smoothing filter such as a Gaussian filter.After the potential candidate of traffic sign has been successfully detected, it needs to be recognized.The result of this recognition process implies whether the potential candidate represents a traffic sign and in case it has been positively identified as a traffic sign, the task of this system is to recognize the pictogram of the traffic sign.
The aim of the above mentioned procedure is to detect and recognize the traffic signs only from a still image.To recognize the traffic sign in a video, the programme has to start tracking sign in the image since it appears there and has to keep tracking it until it disappears.
The detection and the recognition process are the most important stages of TSR system.Therefore, more information about them will be explained below.

Detection
In order to detect the traffic signs, the colour based segmentation is used.The advantage of this approach is that it maintains the low computational complexity in comparison with the segmentation based on shapes.For this purpose, we need to obtain three simple binary images using global thresholding by red, blue and yellow colour (Fig. 2).So, the result of the using of such algorithm is the detection of three types of traffic signs by means of segmentation of three colours.On the one hand, other colours such as green or brown cannot be recognized by the mentioned algorithm but on the other hand they can be simply implemented later.To develop easier segmentation procedure, we have decided to base our system on the CIELAB colour space.It has been defined to emulate human colour perception [15].The main achievement of using CIELAB instead of RGB colour space is device independent model [16] and the values of colours for thresholding could be determined much more easily.
Following this stage, these three binary images will be processed by morphological opening operation.It is a complex operation consisting of basic morphological transformations of erosion and dilation [17].This operation is aimed to remove the redundant noise and the small objects as well as to enlarge the bigger objects.
After the processing of all 3 binary images, the system employs an algorithm for finding contours.This algorithm identifies the contours of objects displayed on the binary images.Each contour represents continuous area, which has some properties such as height and width, bounding rectangle, size of the area or perimeter.In addition, convex hull, the smallest circumscribed rectangle and such characteristics as oblongness, squareness, roundness, and convexity [16] can be computed.The system utilizes these properties and characteristics to remove contours which most likely seem not to be the traffic signs.As the result of this utilization, the system finds the candidates of traffic signs.However, a few of them might not represent the traffic signs.Thus, they will be eliminated by the classification in the next recognition stage.

Recognition
The recognition operates by the means of classification with a machine learning algorithm, specifically with the artificial neural network (ANN).We have proposed overall 15 ANNs based on the colour and shape of traffic signs (Fig. 3).The type of all used ANNs is a multilayer perceptron with one hidden layer.
Each candidate considered as traffic sign is initially classified according to the shape criteria (in Fig. 3 there are three ANNs shape of candidates for a particular colour -orange rectangles in second column).The output of this first neural network is the decision whether the shape is the candidate (triangle, inverted triangle, circle, square etc.) or that "it is not a traffic sign".If the candidate is accepted, i.e. it has not been classified as "it is not a traffic sign", the process of classification will continue by the next neural network to identify the name of the traffic sign.
As a result of cascade of the ANNs we are able to classify the candidates and get the output in form of recognized appropriate traffic signs.

Tracking
The above mentioned proposed procedure of the traffic sign detection and recognition works only in one still image, not in dynamic environment.The aim of this work is to introduce the recognition system of the traffic signs derived from the video (the sequence of still images) taken by a moving vehicle.Therefore, it is necessary to track the traffic signs in order to determine that one sign in the first image is the same sign in the next image.After the sign has been disappeared out of the image, the tracking process of the traffic sign is accomplished.
Tracking of traffic signs was performed by Lucas-Kanade (LK) algorithm [17].It is based on calculating of new positions of tracking points which are located on the traffic signs.So LK compares two consecutive images, the first (previous) at the time t and the second (current) image at the time of t + ∆t.New positions of tracking points in previous image are being calculated into new points in the current image.

APPROXIMATE POSITION OF TRAFFIC SIGN
To determine and to calculate the approximate position of the recognized traffic sign we have applied the following expressions: where sx and sy represents the calculated longitude and latitude of the recognized traffic sign on the map.The values of variables x, y and a are obtained from a GPS device and represents the current longitude, latitude and azimuth (direction) respectively, p symbolizes the position of the recognized sign on the image.It takes the value of +1 if the sign is located on the left half image, otherwise its value equals to -1.Lasts, cx and cy are constants for shift of position in longitude and latitude.They were introduced due to the current position (longitude and latitude) which is obtained from a GPS device located in a moving vehicle and the traffic signs are usually located on the left or right side of the road.To simplify the operating system, the signs located straight above the road were not taken into consideration in the current system.We have determined the value of both constants to 0.00005 in decimal degrees by performed experiments.
Fig. 4 Recognized traffic signs showed alongside red line (GPS track) on the map

DEMO APPLICATION
We have developed a software application that is aimed to collect traffic signs from images captured by a video camera located in front of a windscreen in the vehicle.The application detects and recognizes traffic signs from the camera or a video file.It displays a position acquired from a GPS device and after recognizing a traffic sign it will show the traffic sign in list of recognized traffic signs along with its calculated position.
In addition, the recognized traffic signs might be displayed also on the map of Google Earth plugin in real time.The traffic signs with positions can be easily stored to the file databases and subsequently they might be exported with the path of the travelled track to a KML file.Such information databases could be used for future data analysing or for creating the traffic signs databases situated on selected roads.
Moreover, the application can be used to load stored traffic signs and train/test developed TSR algorithm, annotating traffic signs from still images and videos (determination name of traffic signs), viewing and editing.

CONCLUSION
In this paper, we have described the simple TSR system and the associated algorithm in order to obtain a position of recognized traffic sign from an image and GPS position.The conclusion of our research is the developed demonstrative application aimed to be used to recognize and collect the traffic signs for future data processing as we mentioned above.

Fig. 1
Fig. 1 Procedure of traffic sign recognition However, the problem of the detection phase is to find the proper objects -the real candidates of traffic signs in the preprocessed image.The detection stage consists of further steps, namely the colour based segmentation by

Fig. 2
Fig. 2 Example of image segmented by red, blue and yellow colour

Fig. 3
Fig. 3 Cascade of the ANNs to classification of traffic signs

Fig. 5
Fig. 5 Main screen of the application for collecting traffic signs