Relevance of Automatic Number Plate Recognition Systems in Vehicle Theft Detection ^†

Abstract

Smart vehicle technologies have revolutionized human life in the current era. Smart vehicles, referred to as connected and autonomous vehicles (CAV) are equipped with advanced technologies that increase their safety and security. These technologies have the potential to transform various aspects of society in terms of transformation. This research paper presents an analysis of automatic number plate recognition (ANPR) systems and a comparison at each stage in the aspect of technologies and algorithms involving computer vision. The research paper compares algorithms used for number plate recognition at various ANPR stages. ANPR is also known as the automatic license plate recognition (ALPR) system in many countries. These ANPR systems are generally used in different applications like security surveillance, traffic management, and electric toll collection systems, including law enforcement, parking enforcement, etc. Several factors can destroy the performance of ANPR systems. These factors can lead to inaccuracies in plate recognition or cause the system to fail to identify license plates correctly. Some common factors that can undermine ANPR performance include poor image quality, nonstandard plates, weather conditions, vehicle speed, plate obstructions, lighting conditions, and hardware-based constraints. These challenges make ANPR an interesting area for research. In addition to enhancing the performance of ANPR, other technologies like RFID, and GPS can be used. The paper also focuses on the number plate recognition rate after applying different algorithms. This research aimed to improve the state of knowledge of ANPR, which includes various algorithms and ANPR steps analysis for number plate detection through citing relevant previous work.

1. Introduction

Vehicle theft is a major issue faced by the world these days. Intelligent transportation systems (ITS) is playing an important role in increasing the detection rate of vehicle thefts. The automatic number plate recognition system is already contributing as a part of ITS in the recognition of stolen vehicles. This does not need additional transponders to recognize registered number plates like radio frequency identification (RFID) systems.

ANPR cameras are very useful for not only capturing vehicle images but also helping to get additional vehicle information such as the vehicle’s speed, direction, counting, and group of vehicle details. It is a cost-effective technology. These abilities make ANPR part of our lives and also promises to stay with us in the future [1]. ANPR has also been adopted in many applications such as electric toll collection, traffic management systems, delivery tracking, user billing, queue length estimation, parking management systems, etc. The working process of mobile and fixed ANPR is shown in Figure 1.

The general process of recognition is to capture the image as input and then the system extracts the number plate with the help of specific algorithms; the extracted image will then be processed for character segmentation. After segmentation, character recognition algorithms will be processed in the segmented output. This whole process makes use of different algorithms and can be performed in a few seconds.

Figure 1b shows the diversity of vehicle license plates in terms of style, color, quality, font, size, and other physical conditions. After capturing the image, the system has to localize the number plate and then segment the characters. If the extraction is not proper due to these diversities, the recognition rate of the overall system will be decreased. Although the ANPR is a challenging system due to its various stages, it is currently impossible to achieve 100% overall accuracy as each stage depends on the first step. Diversities such as different light conditions, shadows, unequal plate sizes, characters, variety of fonts, and background color means that it cannot produce adequate result accuracy in tough conditions [2].

2. Review of the Literature

ANPR technology is widely used in many regions because it helps to detect stolen vehicles and also helps to increase vehicle theft detection rates all over the world. ANPR is a combination of three stages. The recognition process is performed at the end of the third stage [1]. The literature review considered various stages of ANPR technology. It also discussed the various algorithms used as well as the performance rate of implementation. To draw inferences from the proposed methodology, a comparative analysis was also conducted on all three stages based on different features and algorithms. The following ANPR stages have to be followed step by step.

Stage 1 Number Plate Extraction: it is the important and typical stage of the ANPR system. After capturing the vehicle image, the system extracts the number plate of the vehicle by using different algorithms. To distinguish between the license plate and other objects of the image, the algorithm should be reliable. There are many methods evaluated by researchers. Some of these are:

• Edge Feature: several researchers used edge detection algorithms and filters to extract number plates. They used different datasets and different algorithms for the same such as vertical and horizontal edge histograms tested on 50 images with a 90% extraction rate [3]. The extraction rate of the vertical edge detection algorithm (VEDA) tested on 50 images in different light conditions was 96% [4]. Sobel, Gabor, and Canny edge detection filters [5], and also a combination of several algorithms were used by them.
• Texture Feature: extracting number plates using texture features includes various algorithms such as Local Binary Pattern (LBP) which was tested on 110 images with an 89.7% accuracy rate [6]. Histograms of Oriented Gradients (HOG) and techniques such as scan line technique had a 99.2% performance rate. The vector quantization technique, sliding concentric window, and the combinations of weight density maps and algorithms based on the neural network tested on 400 images in different light conditions achieved 97.23% accuracy [7]. The LBP algorithm is one of the best techniques to achieve a 100% extraction rate but it can only work with HD images and is not able to work on blur or various condition images [8].
• Character Feature: character feature-based techniques were also evaluated by researchers researching in the related field. These techniques are Hough transformation, scale-space analysis, adaboost classifiers, support vector machine (SVM), and scale invariant feature transformation (SIFT) [9].
• Color Feature: the number plate can have a color diversity depending on the country. The researchers use several techniques such as the hue, lightness, and saturation (HLS) color model and 90% of the images were recognized correctly [10]. The mean shift algorithm achieved 97.6% accuracy [11,12] and the fast mean shift method extracted the number plate from 400 images with an accuracy rate of 92.6% [13].

Table 1. Diverse proposals that apply on number plate extraction features.

Different Proposals	Features Extracted	Algorithms Used	Image Condition	Extraction Rate	Device Configuration	Plate Format	Problem Areas
Slimani et al., 2019 [14]	Edge Feature	Ostus Adaptive Thresholding, CCA, Canny Edge Detection	Various situations and different light conditions	96.37%	MATLAB	Moroccan	_
Kashyap et al., 2018 [15]	Edge Feature	Edge Statistics and Morphological Techniques	HD images	98%	MATLAB	Indian	Not suitable for different image conditions
Vaishnav et al., 2018 [16]	Edge Feature	Morphological Techniques	Various situations and different light conditions	92%	MATLAB	Indian	_
Sferle, R.M. et al., 2019 [17]	Texture Feature	Histogram analysis using HOG	Various situations and different light conditions	89.70%	OpenALPR	European	Not able to detect plate in blurred images
Laroca, R et al., 2018 [18]	Character Feature	Object Detection, CNN	Full HD Images	100%	_	Brazilian	Not able to detect plate in blurred images
Desai, G.G. et al., 2018 [19]	Texture Feature	Local Binary Pattern with a cascade classifier	HD images	98.35%	_	Indian	Able to work only on the fixed front side number plate
Lin, N.H. et al., 2018 [20]	Texture Feature	Local Binary Pattern with Edge information	HD images	100%	OpenALPR	Myanma	Not able to detect plate in blurred images and takes high processing time
Haider, S.A. et al., 2017 [3]	Edge Feature	Vertical and Horizontal Edge Histogram	Normal condition images only	90%	MATLAB	Pakistani	Limited dataset tested
Hommos, O. et al., 2016 [21]	Character Feature	Nearest Neighbor Interpolation, Preprocessing, Geometrical Conditions	HD and various condition images	98.10%	MATLAB	Qatari	Memory and time constraints, do not properly work on blurred images
Omran, S.S. et al., 2017 [22]	Edge Feature	Intensity detection and morphological operations	HD images	98.30%	MATLAB	Iraqi	Not able to detect plate in blurred images
Gao, Q. et al., 2007 [23]	Color Feature	Color features with the vertical sweep	HD images	96.60%	_	Iranian	works on daylight and HD images only

summarises different algorithms used for extraction techniques:

In various proposals, preacquired images were used for extraction. Out of these studies, it was found that the LBP was the best algorithm to extract number plates with a 100% performance rate [20].

Stage 2 Character Segmentation: This stage takes place after the successful number plate extraction. This extracted number plate has been taken as input for this stage and the characters are segmented by the different features mentioned below:

• Boundary Information: algorithms such as vertical edge detection, vertical histogram, closed curve techniques, morphological thickening, and morphological thinning on 1189 images with an 84.5% segmentation rate [24] were evaluated by researchers in their studies.
• Connected Component Analysis: techniques like pixel connectivity were tested on 958 HD images with a 99.75% segmentation rate [21], connected component labeling and morphological method on 50 images with a 91% success rate [3], and a hybrid method of blob coloring and connected components with a 93.7% accuracy rate also proposed in previous studies.
• Extracted Character Feature: an RGB color extractor was tested on 255 images with a 98.5% accuracy [25]. YOLO models, YOLOv2, and Hough binarization methods were applied to 332 blurred images with 96.4% performance rate [26]. Fast-YOLO model and a classification regression network (CR NET) was applied for segmentation by researchers.
• Projection Method: vertical and horizontal pixel project methods were used for segmentation tested on 30,000 images with a 99.2% success rate [27] and profile projection methods were also used [28] on 560 images with a 95.4% rate.

Table 2. Diverse proposals that apply on number plate segmentation features.

Different Proposals	Features Segmented	Algorithms Used	Image Condition	Segmentation Rate	Device Configuration	Plate Format	Problem Areas
Vaishnav et al., 2018 [16]	Boundary Information	Region props bounding box	Low light and contrast images	97.00%	MATLAB	Indian	_
Laroca, R et al., 2018 [18]	Boundary Information	CNN, Bounding Box	HD Images	98%	_	Brazilian	Not able to process blurred images
Haider, S.A. et al., 2017 [3]	Connected Component Analysis	Connected Component Labeling + Morphological Operations	HD Images	91.00%	MATLAB	Pakistani	Limited dataset tested
Hommos, O. et al., 2016 [21]	Connected Component Analysis	CCA Labeling and Morphological Operations	HD Images	99.75%	_	_	Memory and time constraints, do not properly work on blurred images
Molina-Moreno, M. et al., 2018 [29]	Projection Method	Scale-weighted linear interpolation	Blur Images	74%	PASCAL	American, Taiwanese, Spanish	The segmentation rate is low
Jia et al., 2016 [25]	Extracted Character Feature	RGB color extraction, Character Isolation, Thresholding	HD Images	98.50%	Open-source OCR engine	American	Not able to process blurred images
Mutholib, A. et al., 2012 [30]	Projection Method	Vertical Projection Method	Color images and HD images	83.50%	Eclipse IDE, Android Platform	Malaysian	Not able to process blurred images

summarises different algorithms applied for these techniques to find out a better segmentation rate.

The above techniques used for segmentation from past years. Out of these, CCA labeling with morphological operations performed with a 99.75% segmentation rate which was comparatively the best accuracy rate [21].

Stage 3 Character Recognition: recognition of segmented number plate characters is the final stage of image processing in the ANPR system. There are two major techniques used for optical character recognition (OCR) which are:

• Template Matching: it is the simplest method of recognition. In this technique, segmented characters are compared with the existing template characters set. This process is performed by scanning the character’s column vise, and the highest correspondence value is found as the best-matched character. The technique tested on 1200 blurred images in the dataset correctly recognized 90% of images [20] and when tested on 1300 images of size 640 ∗ 480 pixels in the dataset, achieved a recognition rate of 92.5% [19].
• Using Extracted Features: feature extraction is based on several algorithms and techniques such as SVM, hoteling transformation, etc. [31,32].

The techniques used for the same are use template matching, recognition using extracted features, etc.

Table 3. Diverse Proposals that Apply to Number Plate Recognition Feature.

Different Proposals	Features Recognition	Algorithms Used	Image Condition	Recognition Rate	Device Configuration	Plate Format	Problem Areas
Slimani et al., 2019 [14]	Template Matching	Template Matching	Various situations and different light conditions	98.10%	MATLAB	Moroccan	_
Kraisin, S. et al., 2018 [33]	Using Extracted Features	HOG Feature, Extreme learning machine	Low Resolution	90%	_	_	The recognition rate is low
Vaishnav et al., 2018 [16]	Template Matching	Template Matching	Contrast and low-light images	98%	MATLAB	Indian	_
Laroca, R. et al., 2019 [18]	Using Extracted Features	Data Augmentation, Distant CNN	Full HD Images	97.83%	_	Brazilian	Not able to detect plate in blurred images
Yogheedha, K. et al., 2018 [34]	Using Extracted Features	Tesseracts OCR	HD images	92.12%	_	Indian	Able to work only on the fixed front side number plate
Lin, N.H. et al., 2018 [20]	Using Extracted Features	Tesseracts OCR, Preprocessing techniques	HD images	90%	OpenALPR	Myanma	Not able to detect plate in blurred images and takes high processing time
Haider, S.A. et al., 2017 [3]	Using Extracted Features	Statistical Feature Matching	Normal condition images only	93.00%	MATLAB	Pakistani	Limited dataset tested
Hommos, O. et al., 2016 [21]	Using Extracted Features	OCR Algorithms	HD and various condition images	99.50%	MATLAB	Qatari	Memory and time constraints, does not properly work on blurred images
Omran, S.S. et al., 2017 [22]	Using Extracted Features	Back Propagation Neural Network(BPNN)	HD images	93.20%	MATLAB	Iraqi	Not able to detect plate in blurred images
Jia, Y. et al., 2016 [25]	Template Matching	Template Matching	HD Images	95.10%	Open-source OCR engine	American	Not able to process blurred images
Mutholib, A. et al., 2012 [30]	Using Extracted Features	OCR using ANN	Color images and HD images	92.00%	Eclipse IDE, Android Platform	Malaysian	Not able to process blurred images

shows the summary of these techniques.

Out of these, OCR performed with a 99.50% accuracy rate [21]. It is important to find out the best results at each stage to increase the overall recognition rate because the output of each stage is considered as the input of next stage.

All three stages perform an important role in number plate recognition. Whether there is a still image or a video, the technology can recognize number plates. After the recognition of vehicle number plates, the data can be used for postprocessing for vehicle theft detection. The retrieved dataset is sent to the connected server and compared with the existing vehicle theft database [35]. This whole dataset can be used for several ITS in the future because ANPR stores all data in a central server. The review focuses on each stage and finds out the best algorithms working for every stage for further use in the proposed methodology.

Several researchers used HD images for number plate recognition. Table 1, Table 2 and Table 3 show results after applying different HD and blurred images in the dataset for recognition. Different researchers combined different algorithms to improve the overall recognition rate. This combination of algorithms play an important role in vehicle theft detection. The tables illustrate fifteen diverse proposals for the ANPR system, out of which 64% of researchers employed HD-quality images and only 27% used images with various situations as input images shown in Figure 2.

In a real-time scenario, not all captured images can be HD if the vehicle is in motion or if the number plate is dirty or scrawled. Thus there is a need for a system that first clears and converts blurred images to HD and then all three ANPR stages will work step by step. However, in ANPR technology, the decision of using still image or video depends on a number of variables, applications, accuracy requirements, available resources, and budget; videos provide more contextual information and are more suitable for capturing moving vehicles. Hybrid strategies can incorporate the benefits of both for best results.

3. Experiment and Analysis

A controlled experiment was conducted for number plate detection using both HD and blurred image datasets. The successful implementation and evaluation of the system was tested using a 1920 × 1080 pixel HD dataset of 50 images at MATLAB device configuration. The experiment recognized and extracted number plates from various high-quality images with a high accuracy of 99%. Then, the performance of the system was rigorously evaluated on a separate test set composed of 30 different types of blurred image. The results underline the difficulties presented by unclear characters, touchy characters, and boundary problems, and an error was found. Through error analysis, the study was able to identify recurring failure patterns and gain knowledge of future scope and development.

4. Proposed Methodology

The existing studies show better results in the case of HD images, but in the case of blurred images, the recognition rate was too low. Therefore, in the proposed methodology, three new algorithms were introduced at the stage of preprocessing. The best extraction algorithm found was local binary pattern with edge information results with a 100% extraction rate [20], but it did not work with the best quality images. For segmentation, CCA labeling, and morphological operations, the results were 99.75%. Furthermore, based on the HD images dataset and for the last stage of ANPR, OCR algorithm results were 99.50% [21], as seen in Figure 6. Thus, in the proposed methodology, these algorithms will be applied with the addition of three image preprocessing algorithms to be applied to the blurred image dataset.

Figure 3 shows that the collected images were subjected to preprocessing. The combination of several techniques can help to enhance overall image quality because each algorithm focuses on different areas of image degradation. When a vehicle arrives at a toll plaza, the camera captures images of the vehicle’s number plate, then the captured image is processed by the ANPR system and the preprocessing takes place and the system starts to clear the number plate with proposed algorithms.

Starting with the motion blur removal, Gaussian blur removal or image sharpening algorithm, the image quality will be improved. These methods can help reduce noise, increase clarity, and improve overall image quality. Once the image is better, the number plate extraction method will be processed such as by LBP to gather texture data. Working with a clean and clear image will be beneficial as it can result in more accurate feature extraction. After extraction, the number plate characters will be segmented using CCA labeling and morphological operations and next and last stage of ANPR will be OCR to recognize the number plate. In the preprocessing step, we will implement the following algorithms systematically to improve image quality.

These three methods are often used for the following reasons:

Motion Blur Removal: in the process of capturing vehicle images or video, motion blur is a common problem caused by the movement of the camera or vehicle about the seen. The sharpness and details that were lost due to this blurring effect can be recovered by using the motion blur. This method is extremely relevant when working with photos and videos that have a noticeable blurring in some directions. The images are restored after enhancing image quality in the database to their original sharpness. The reviewed research proposed a system to restore these sharp images by combining a pair of noisy/blurred images taken quickly to form a clear image. Deblurr RNN and DeblurMerger, two neural network topologies, have been introduced to manipulate pairs of images sequentially and simultaneously [36].

Gaussian Blur Removal: gaussian blur can be caused by many things, including atmospheric effects and lens imperfections. Gaussian blurring reduction procedures can be used to improve small details and textures that may have been obscured by such blurring. It is useful to recover high-frequency features and edges in the image. Previous researchers found that this algorithm helped to filter images with heavy noise [37].

Image Sharpening: this algorithm is used to enhance the clarity of the images by its contrast at the edges, also useful for improving the sharpness of images. The algorithm can make edges appear crisper and more defined. According to the reviewed research, image sharpening using unsharp masking (UM) approaches supports moderate contrast detail enhancement. The contrast and brightness of the image are additionally enhanced via wavelet-based UM, which also provides good value parameter percentage correction [38].

The combination of these three algorithms enables us to treat a wide range of image degradation problems, thereby improving image quality overall. Depending on the exact implementation, dataset, and evaluation criteria employed, specific statistics and benchmarks are subject to change.

After improving image quality, the following algorithms shown in

Table 4. Performance summary of ANPR system technique (HD images dataset).

Different Proposals	ANPR Stages	Algorithms Used	Results	Device Configuration	Plate Format	Problem Areas
Lin, N.H. et al., 2018 [20]	Stage 1: Number Plate Extraction	Local Binary Pattern with Edge information	100%	OpenALPR	Myanma	Not able to detect plate in blurred images and takes high processing time
Hommos, O. et al., 2016 [21]	Stage 2: Character Segmentation	CCA Labeling and Morphological Operations	99.75%	_	_	Memory and time constraints, not properly working on blurred images
Hommos, O. et al., 2016 [21]	Stage 3: Character Recognition	OCR Algorithms	99.50%	MATLAB	Qatari	Memory and time constraints, does not properly work on blurred images

will be applied.

Overall, in the proposed methodology, first, we take a dataset of blurred images and then apply image preprocessing algorithms for motion blur removal, then gaussian blur removal, and finally image sharpening to enhance image quality. Then, we apply the algorithms mentioned in Table 4, step by step, and get the result. The results of both the HD and blurred image datasets will be compared to improve the recognition rate of the number plates as mentioned in the preprocessing phase. This proposed methodology will enhance image quality and help to improve the overall recognition rate.

Postprocessing of the system includes identification and comparison of the captured image data with the already existing blacklisted database of stolen vehicle number plates. This blacklisted data has been provided and circulated by the regional transport office (RTO) after lodging the first information report (FIR) by the vehicle owner at the nearest police station [9]. As resultant vehicle theft will be detected, the vehicle theft detection rate will be improved.

5. Conclusions and Future Scope

The paper has analyzed different algorithms used in number plate extraction, character segmentation, and character recognition in the automatic number plate recognition system. To a great degree, the success rate of the ANPR system highly depends on the quality of the captured vehicle number plate images. The study exhibits that the results of the ANPR system depend on the input images. In the process of picking out the best algorithms, local binary patterns with edge information for number plate extraction, CCA labeling, morphological operations for character segmentation, and OCR algorithms for character recognition were found to be the best algorithms at each stage, respectively. These algorithms are used to enhance the overall system recognition rate but with one precondition of the images being of HD quality.

Thus, the research further aimed to enhance results for blurred images or tampered images along with all of the best algorithms. For this, the research included three image quality-improving algorithms: 1. image sharpening, 2. motion blur removal and 3. gaussian blur removal at the stage of image preprocessing. The research is consistent with other ANPR research initiatives in several ways. (a) Addressing image challenges: previous work identified difficulties in identifying lower quality and poor images. (b) Leveraging enhancement techniques: to enhance image quality, the idea uses established methods from the field of computer vision. (c) Addressing the detection gap: the proposed method focuses on detection reduction for blurred images, aligning with ANPR research to enhance the overall performance of the system.

In short, the proposal advances previous ANPR work by tackling image quality issues with tried and true methods and improving ANPR accuracy. Overall, the result of the study proves the applicability of image quality-improving algorithms to improve the overall number plate recognition rate. The study presents proposals on the relevance of automatic number plate recognition systems in vehicle theft detection. The research promises enhanced results and the results will be encouraged for further research. For future research, there is a need to simulate and implement the results on blurred or tempered image datasets. The dataset can be collected from toll plazas as a primary dataset for future research. The results could be compared and the efficacy of the proposed algorithms could be found. Moreover, the proposed algorithms could be tested on real-time images rather than prestored datasets. The results can be improved by applying new algorithms at different ANPR stages. The device configuration can also be changed accordingly.