LICENSE PLATE RECOGNITION SYSTEM BASED ON MASK R-CNN

. Automatic license plate recognition (ALPR) systems can be found in many different applications. If a person has a driving license, the person probably had already seen smart road camera after a speed limit sign or on a crossroad. Every year number of cars on roads growth very fast. It is also obvious that such systems can be used out-of-road situations. For instance, this type of systems can be used for automatic access control on private property or smart parking, or even log system that are being used literally everywhere. Because of popularity of ALPR systems, there are two main goals, which are being pursued by researches: speed and accuracy of recognition. Speed of the detection is important for real-time systems. Accuracy is important for every system.


Introduction
License plate recognition systems (LPRS) have become a crucial technology in recent years, finding applications in various fields such as law enforcement, traffic management, and parking systems.LPRS plays a significant role in ensuring the safety and security of individuals and property, as well as facilitating efficient transportation management.One of the key challenges in LPRS is accurately detecting and recognizing license plates from images captured by cameras, which can be affected by various factors such as lighting conditions, vehicle orientation, and plate occlusion.
In recent years, deep learning-based methods, particularly those using convolutional neural networks (CNNs), have shown great promise in addressing these challenges.Among the many CNN architectures available, one particularly powerful and versatile option is the Mask R-CNN (Region-based Convolutional Neural Network) model.This model combines object detection and instance segmentation tasks in a single framework, allowing it to accurately identify and isolate different objects within an image.

Literature analysis
All ALPR systems can be classified as hardware [1] and software [2] systems.Software systems are used to perform analysis on obtained data (for example, on a server).Outdoor camera transmits a data stream and powerful computation server performs analysis of this stream.This approach allows installation of cameras in different environmental and weather conditions.Even if camera is getting broken, it can be easily and cheaply replaced.
Hardware systems uses another principle to perform analysis.Cameras are expensive and should be protected from several of weather conditions, but they perform on-site analysis (detection and extraction) of the license plate and distribute obtained information to a server.This approach has own benefits.For instance, because analysis are performed on-site it is allows manipulation of mechanisms without any delay.In addition, requirements to the server are significantly lower, as it should perform only persistence of the received data.
Firstly idea to create own ALPR system was based on software approach, but in order to understand the needs of the users and to test performance and capabilities of the system Raspberry PI 4 was chosen for hardware implementation.It is obvious that there are may be approach related problems with the process of recognition.Software implementations should be able to run multiple analysis separately to work with a bunch of incoming data flows.Hardware implementation should run as fast as possible in order to reach real-time detection.
Raspberry PI attracts interest of researches because of its simplicity and availability, that's why so many different projects were made.One type of these projects is LPR projects.Different methods were discovered and implemented, but firstly it is important to understand capabilities of the platform (Table 1).Raspberry PI doesn't have big computation power but still opens a wide field for experiments.
Table 1 - Another way to implement hardware system is to use Raspberry PI as obtaining device and then perform correct analysis on the server [4].Despite the good accuracy and speed, it is software based approach and not hardware based.
One of the possible hardware solutions is unmanned aerial vehicles.Concept of this solution will allow better adoption to changeable environment (Fig. 1).

Fig. 1 -Concept of ALPR drone
Despite the type of problems, related to implementation, there are also problems that all ALPR have in common.Camera position is the most important factor for accurate detection (Fig. 2).One of the goals of the work is to make the system efficient considering large camera shooting angles.Such restrictions complicate the use of modern ALPR systems in real situations on public roads, especially in the presence of multi-lane traffic, large camera shooting angles, bad visibility of license plates, and in difficult weather conditions [5,6].

Fig. 2 -Camera position invariance
Different methods may come in handy when developing artificial intelligence system for license plate detection [7,8,9] Most of the researches refer to machine learning methods such as artificial neural networks.ANN is great tool to predict the behaviour of complex functions.In field of computer vision, convolutional neural networks are favourite [10,11] (Fig. 3).

Fig. 3 -General convolutional neural network architecture
Semantic segmentation models are mostly based on object detection models.Mask R-CNN is one of them [12] (Fig. 4).

Results
For accuracy test another two license plate recognition systems were used: NomeroffNet [13] and SeeAuto (FF-Group) [14] (Fig. 5).To test models "Autoria Number Plate Dataset" was used.Default accuracy metrics was used:  =  +   +  +  +  true positive (number plate exists and was correctly predicted);  -true negative (number plate doesn't exist and wasn't predicted);  -false positive (number plate doesn't exist but was predicted); false negative (number plate exists, but wasn't predicted).

Fig. 5 -Accuracy comparison
To test another important metrics like precision, recall and F1-Score of the model Medialab LPR dataset was used [15] (Fig. 6).2.

Table 2 -Test metrics among datasets
Raspberry PI is best platform for embedded systems research [16].Large market of different modules makes this platform suitable for different projects.This platform allows testing of different potential hardware systems without any special knowledge or preparations.
In the beginning, two different cameras were considered: PI camera and default USB-Camera.To make the project more reproductive standard USB-Camera was chosen.Resolution of the camera is 640x480 pixels.Raspberry PI were connected to external monitor (Fig. 7) for results visualization, but it isn't necessary, because Raspberry PI has Wi-Fi and Ethernet interfaces for communication through net.

Fig. 7 -Raspberry PI connection scheme
Because of large modules like TensorFlow and OpenCV time required to process one image firstly was from 10 to 20 seconds.After installation of simpler TensorFlow Lite and OpenCV Lite results are better (around 5 to 15 seconds per image), but there is no way this system can be used in real-time.Significant optimization improvements must be made or platform computation capabilities must be greater.
Another way to apply developed system is drones.With the help of drones and great accuracy, it makes sense to use this system as part of the drone for automatic license plate detection in changing environment [17].Although the system can't be applied for real-time number plate detection, the accuracy of the model doesn't depends on the platform, so the system shows same great accuracy results (Fig. 8).

Fig. 8 -License plate detection on Raspberry PI
As mentioned earlier it takes 5 to 15 seconds to process image with Raspberry PI.First version of the system was deployed to server with next tech specification: Intel Xeon CPU E5-2630 v4 2.2 GHz.It takes up to 4 seconds per image to process.
Another performance test (main one) was performed on a laptop CPU Intel i7-9750H.Average time around 5 seconds was achieved.With modern GPU technologies the process takes a lot less time.GeForce GTX 1060 Ti shows result around 500 ms.

General conclusion
As the result of the research automated licence plate recognition system was build and tested in various of ways.LPR problem was studied to create accurate prediction system.A lot of information was processed in order to reach deep understanding of Mask R-CNN architecture and application possibilities.
As part of additional research current system was implemented with the help of Raspberry PI 4 for real-time tasks.Unfortunately, optimization improvements or more powerful platforms are needed to reach acceptable processing times delays.Great accuracy metrics show that the main goal of the project was reached.Studied system works better than compared in terms of large camera angles.
methods is corner detection algorithm[3].Using powerful open source computer vision library OpenCV different methods can be applied to perform extraction of the position of the number.The problem with such approach is low invariance capability.The slightest change in the color of the image may lead to crucial accuracy loss.Besides, no model can be trained, so all the situation should be processed manually at first.

Fig. 4 -
Fig. 4 -Mask R-CNN general architectureUnique technic is used in order to process images.CNN (in our case ResNet101) performs feature map extraction.Then another ANN generates regions that are classified later.Fitting process of this network can be performed with the help of next loss functions:(, ,   ,   ) =   (, ) +    (  ,   ) +   (, , )

Fig. 6 -
Fig. 6 -Difficult multi license plate situation Totally, almost 1 thousand images were tested.The results are shown in Table2.Table 2 -Test metrics among datasets