Application independent localisation of vehicle plate number using multi-window-size binarisation and semi-hybrid genetic algorithm

: This study introduces an enhanced version of a previously published genetic algorithm based technique to allow fast and accurate detection of the vehicle plate number independently of the used application. Hence, signi ﬁ cant enhancements are introduced to upgrade the genetic algorithm into a semi-hybrid category by preceding it with a sorting and subgrouping process that reduces the search space and following it by a rule-based local search to optimize its output. The updated population structure and its initiation with subsets of the sorted image foreground objects results in a linear relationship between the image complexity represented by the number of foreground objects and the genetic algorithm search space instead of being exponential in previous versions. Moreover, a novel extra variable-size-window adaptive binarisation step is introduced to overcome the problems of attached license symbols that cannot be solved by the skipping ability introduced in previous versions. Various image samples with a wide range in scale, orientation, and symbol connectivity have been experimented to verify the effects of the new improvements. Encouraging results with 99.2% detection accuracy have been reported using a public dataset outperform-ing the author of the dataset by more than 5.5% and the state of the art systems by 2%.


Introduction
Automatic license plate recognition (ALPR) is the most important module of Intelligent Transportation Systems. ALPR is used separately to perform vehicle identification for the purpose of toll payment, parking fee payment, freeway and arterial monitoring systems, detection of stolen vehicles, among others [1][2][3]. ALPR applications can be split into three categories [access control (AC), traffic law enforcement (LE), and road patrol (RP)] based on the problem variables such as camera viewpoint, plate size in the image, and illumination conditions [3]. An application-oriented system has been introduced in [3]. Although most ALPR systems are performed in three steps (detection, segmentation, and recognition) [1][2][3], the genetic algorithm (GA)-based system, introduced initially in [4] and enhanced in [5] combines the first two steps in one step. In this paper, I am going to evolve the enhanced GA-based solution in [5] to allow detection and segmentation of a variety of LPs in different application categories without resorting to select different system parameters for each category as in [3]. In ALPR systems based on the connected component analysis technique, the accuracy and performance are significantly dependent on the binarisation stage where each pixel in the image is decided to belong either to the background or to the foreground objects. In [4], an adaptive local binarisation stage has been adopted to overcome illumination changes in addition to the skipping ability introduced in the GA to overcome physical problems affecting the CCA techniques. The concept of skipping that was introduced in [4] and enhanced in [5] supports partial detection of the LP based on the detection of a few number of license symbols less than the defined number of symbols in the LP (L). The number of skipped symbols (SN) and their positions are then determined based on the relevant geometrical relationship matrix (GRM) that defines the geometrical relationships between the symbols in the LP layout. Applying the previous GA-based system on a different dataset such as [3], raises new concerns related to the CCA such as connected symbols that their large number results in ambiguous situations that cannot be handled by the skipping ability introduced in the previous systems [4,5]. To overcome these problems, extra processing and enhancements are introduced in this research at different stages to get an application independent system. In addition to the enhancement introduced in the binarisation stage, the GA itself is upgraded to a higher category such as hybrid algorithms by preceding it with a sorting stage that reduces the search space and hence the time required in the global search, enhancing the effect of the mutation by limiting its search space in a narrower set, and finally a local search is carried out after completion of each GA run to optimise the skipping locations based on smartly designed logical and heuristic rules. A detailed survey of ALPR solutions based on different representations of the LP problem and using different techniques can be found in [1,2]. In [4], the system was experimented based on a public dataset [6] while in [5], a proprietary dataset was built to prove the orientation independence using the full angle range from '−85°to 85°', which was not available in any published dataset. Although the research that built the dataset in [3] has been cited by more than 50 papers (according to Google scholar at the time of writing this paper), only two of them [7,8] are considered because they have detailed and well-defined methods and had reported results based on this dataset which has a main complexity inherited in the tiny symbols that are mostly connected to each other as will be shown in this research. Another recent paper [9] is considered because it has its implemented powerful code publicly published on the Web. Other related work may be considered in the experiments section. A main difficulty that perhaps prevented most of the researchers from using this dataset is the large scope of the LP rotation angle, which has been solved, in my previous system [5]. In this research, I am going to enhance the adaptive binarisation step to minimise the errors due to connected symbols by trying to zoom in (using a window size less than the CCA box size) through a variable-window-size binarisation step if the default one fails. If the zoom-in trial fails, a zoom-out (using a larger fixed-size window than the default one) trial is performed to correctly segment symbols that cannot be segmented in the first two trials. In addition, the system accuracy will be increased by mapping the inclined LP symbols to the new reduced-in-size GRM. The significant change that will be introduced in the GA is the use of subgrouping technique to speed up the GA but in our case, each subgroup will use a subset of the available objects, which are sorted based on a hashing function (HF) that brings LP objects near each other. This enhancement in the population architecture and its initiation and the limitation of the search space for the mutation operator upgrades the GA to the level of hybrid GAs especially after following the GA by a local search that optimises the skipping positions as will be shown later in the following sections. A complete overview of the enhanced system will be introduced in Section 2. The image-processing phase is summarised in Section 3 with detailed implementation of the binarisation steps. In Section 4, GA modifications are described in detail. Finally, the results are presented and discussed in Section 5 and the conclusion is given in Section 6.

System overview
In this research, the system will have three trials to overcome the image processing problems related to the CCA technique, in addition to the enhancement of the skipping ability introduced in the previous systems. In the first trial, the maximum skipping number MAXSN will be two or three based on L (MAXSN = L − 4). In the second and third trials, MAXSN will be increased by one. The purpose of the second trial is to separate the attached license symbols by applying a variable-size window local binarisation technique if the first trial fails in finding a solution with an acceptable horizontal objective distance (HOD) as defined in [5]. The third trial is done if the second trial fails to find a solution with acceptable HOD, where in this case a fixed-size window having double the default size is used in the binarisation stage. The separation of trials into three is intentionally done to determine the impact of each trial on the accuracy of the system and to search for different solutions for complex LPs. The complete system flow chart is shown in Fig. 1. The system is composed of three phases and each phase is composed of many stages as in [5]. Most of these phases and their stages will be enhanced and supported with new additions to get higher accuracy and performance as described in detail in the following sections.

Image processing phase
In this phase, an input colour image is exposed to a sequence of four stages to extract a number (N ) of the relevant two-dimensional objects that may represent the symbols constituting the LP. These four stages [colour-to-grey conversion, adaptive binarisation, morphological operations, and connected component analysis (CCA: MATLAB version is used although there are many efficient implementations [10])] are the same in the first trial as in [5]. The change is done in the adaptive binarisation stage that will include two extra versions that one of them is executed in the second trial and the other in the third trial when the system fails to localise the LP in a former trial. The third trial is a fixed-size window adaptive binarisation as the first one but with double the default size while the second stage is a variable-size-window local binarisation stage that will be demonstrated in the following section. The default window size value (DWSZ) is selected as half of the image width divided by the number of LP symbols (L), assuming that the LP occupies ∼60% of the image width DWSZ = 0.5 × ImageWidth/L (1)

Multi-window-size local adaptive stage
In the first trial, the system performs binarisation using a fixed-size window. The binarisation is based on a local threshold value (LT) which is evaluated for each pixel (x0, y0) based on the mean of the intensity value in the square surrounding the pixel and having a side length WS (=DWSZ) as follows: The above equation has been efficiently computed by the integral images [11]. WS is selected according to (1) which is sufficient for normal conditions. In some cases when the distances between the license symbols are very small, especially when the capturing device resolution is low or the camera-to-car distance is long (which depends on the application), the resultant binarised symbols are attached to each other in the LP image. This attachment problem is solved partially by the previous systems [4,5] if the number of attached symbols is small relative to the total number of the license symbols (L) by the skipping technique. In case of seven license symbols as for the Saudi and Greece LPs [4], a maximum of three symbols can be skipped and their locations can be estimated based on the prototype GRM [5]. If the number of attached symbols is more than 3, the GA may fail in the localisation process and even if partial detection happens, the expected locations of the skipped symbols will be ambiguous. For higher accuracy, the skipping in the first trial is limited to two skipping symbols (L = 6 in [3]) and if the GA fails, the localisation process is restarted by the new local adaptive stage. The new local adaptive stage will use a variable-size window, unlike the first trial which uses a default fixed-size window with side length (DWSZ). The variable window size VWS at any pixel (x, y) will be calculated based on the size of the CCA object (detected in the first trial) that contains this pixel. My concept is based on the assumption that: if the attached symbols constitute a rectangle whose shorter length represents the character height (H ), then a window which has approximately the character width (i.e. VWS = H multiplied by the default aspect ratio ASPR) will do better segmentation of the current object. To get a better understanding of the best window size for the binarisation problem the reader is referred to [12] where document binarisation is considered. Although [12] has been cited by more than 30 papers (as reported by Google Scholar at the time of writing this paper), only two of them [13,14] have considered the license plate problem. In [13], a different technique has been proposed to correctly segment symbols inside an already detected LP. Also, [14] suggested an iterative loop method to segment the LP after detection. Both methods used prior knowledge of the detected LP characteristics before doing the segmentation process. None of them introduced an enhancement of the technique proposed in [12]. The difference in my implementation compared to [12] lies in the application of the multi-window-size concept (without using the variance) in a scenario of three trials instead of one for experimentation issues. In addition, a simplification of the calculation of the zoom-in window size, which is based on the character thickness and height in [12], is done by considering it as the character width in the proposed solution. Moreover, if the rotation is considered, the solution in [12] should be modified because it was proposed for almost horizontal text lines in normal documents. The rotation independency is handled in the proposed solution by selecting the minimum of the object dimensions (H and W ) and then multiply it by the aspect ratio and the bounding box extension (BExt) parameter which represents the percentage of the foreground pixels in the rectangle surrounding the character (or block of connected characters BCC). To justify the proposed approximation, the general case is considered when a BCC is detected in the first trial with dimensions h×wand rotated by an angle (Θ) as shown in Fig. 2a. The detected height (H ) and width (W ) of the surrounding rectangle, in conjunction with the BExt are already given after performing the first CCA step.
The dimensions h, w, and the angle Θ are unknown but can be deduced based on the following equations: Solving these three non-linear equations will lead to the intermediate solution of h in terms of Θ as follows: where FNC (Θ) is given by the following formula: The changes of FNC versus Θ using different rectangle lengths (two to six solid rectangular characters) are shown in Fig. 2b. Assuming FNC (Θ) is equal to 1 at any angle Θ, the average width of any symbol inside the detected block will be If the BCC is completely solid, (8) will be correct but in real LPs, the BCC will have its own real extension RBExt which will linearly affect the measured extension (MExt = BExt × RBExt). RBExt depends on the symbols extensions inside the BCC and the space between symbols. Hence, the approximate width can be given in general as follows: Based on the experimented dataset [3] that has symbols with ASPR≈0.5 and assuming RBExt to be 0.45 for a BCC (this value is higher than the normal extension because we know that the symbols are connected inside the BCC), (9) can be approximated to The incorporation of MExt in (10) replaces the time-consuming method used in the calculation of the character thickness in [12] and at the same time approximates the width estimation at different angles that supports the independency on the concerned application. Figs. 2c and d illustrate the quality of the proposed approximation by displaying the detected widths for different character block widths versus the block angle based on the algebraic solution of (2)-(4) (Fig. 2c) and based on the proposed approximation (Fig. 2d).
The inexact results of the algebraic solution are due to the approximated value of RBExt that is needed to reach an exact solution. Since RBExt is not constant for different combinations of symbols and its measured value is affected by the previous inexact binarisation stage, then it is expected for the algebraic solution to suffer from non-perfect results as the approximated one. Moreover, if we consider the execution time we will be satisfied by the proposed approximation. The implementation algorithm will be based on the output of the first trial and will be explained as follows. The inputs to the proposed binarisation algorithm are the previous label matrix (LBM) output from the previous CCA stage (in trial 1), 3 arrays: WA, HA, and MEXTA containing W, H, and MExt of every CC object, respectively, the input grey image GIMG, the previous integral image of the input grey image IntGIMG. Based on these inputs, the binarisation algorithm will be as shown in (Fig. 3).
Based on the system flowchart in Fig. 1 and the work done in [5], we can show the intermediate and final outputs of an LP image as in Fig. 4. The output sequence starts in trial 1 at the fixed size window binarisation stage (Fig. 4a) and then passes through the objects filtering phase (Fig. 4b) and then designated as unaccepted LP with high HOD in the three runs of the GA with three different skipping values (SN = 0,1,2) as in (Figs. 4c-e) respectively. Since trial 1 failed, trial 2 continues the work by the variable-size binarisation stage that separates the two previous BCCs (119 and 160) into correctly segmented symbols (Fig. 4f ), performs size filtering (Fig. 4g), and finally detects the LP symbols by the GA (Fig. 4h).
Since the variable size binarisation converts only foreground pixels into background ones, it has the effect of segmenting larger objects into smaller ones or clearing small objects as in

GA phase
Significant enhancements of this phase have been carried out on the updated GA in [5] to speed up the detection process and maximise the true positives while minimising the false positive cases. The modifications are done in the architecture of the population and its initiation process, the mutation operators, the input GRM, the objective function and finally the fake detection and optimisation stage to guarantee accurate and relatively fast localisation of the vehicle plate symbols.

Initial population and subgrouping
Due to the large scope of both scale and angle of our GA-based system, partitioning the whole population Z into G subgroups where each subgroup gets a subset of the M objects (output from the objects filtering phase) based on their X or Y coordinates will result in a better performance because the GA will require smaller number of iterations to reach the optimal solution. In addition to fast convergence, subgrouping supports convergence to global optimum because a local minimum in one subgroup will not affect other subgroups.
In case of slightly inclined plates, distributing objects inside subgroups based on the Y-coordinates of the objects will enhance the GA performance. In case of fixed size symbols (which is true for most of the recent LPs in the world), a HF can be proposed to give LP symbols hashing values near each other while being far from values for non-LP objects. Many HFs have been tried where the following one gave excellent results: Applying this many-to-one HF clusters objects based on their areas, which brings LP objects near each other and orders them based on their normalised coordinates, which simplifies the job of the GA and results in getting global optimum in almost all cases and hence increases the system speed and accuracy. After experimentation for almost all of the plates in the dataset used in [3], we found that the LP objects would be found in a range (last LP symbol index −first LP symbol index + 1) of at most 3L (=18) objects in the sorted sequence of M objects based on the HF function even for large M (≥320) as shown in Fig. 6.
To prove the significance of this result, we will roughly compare the search space size (SSZ) in each case. In case of a single group having M objects and searching for a sequence of L objects that gives the minimum objective distance, the SSZ is M!/(M − L)!, that can be approximated in case of large M into (M − L/2) L . By partitioning the population (Z) into G subgroups where G equals (M/L − 3) and limiting each subgroup to only 4L objects (three objects are shared between every two successive subgroups) as in Fig. 7, the SSZ for each subgroup will be 4L!/3L!, that can be roughly approximated to (4L − L/2) L (or (3.5L) L ). Having approximately M/L subgroups, we can deduce that the total SSZ in case of subgrouping is M/L × (3.5L) L or simply M × (3.5L) (L−1) which is a linear function of M in comparison to the exponential function in case of a single group. Hence, we can deduce that the required detection time of a blind search algorithm will increase linearly with the number of objects found in the image instead of being exponential in case of a single group which is a great improvement although it may not be observable in case of small M(≤4L).
Another advantage of the introduced subgrouping is that each subgroup has its own optimal solution which can be used to support detection of multiple plates per image under the condition of preventing interlaced solutions. Unfortunately, the number of multi-plate images in the dataset under consideration is very small and cannot be used to evaluate this feature because many experiments should be done to assess the performance of this

Compact multi-dimensional GRM layout
In [5], the GRM was upgraded to support highly inclined plates by the inclusion of the geometrical relationship values of the concerned prototype LP in the full range of the angles Θ [−90°, 90°]. Since the relationship values are concerned with the differences between the coordinates and dimensions of the consecutive (and non-consecutive in case of skipping) symbols, then by noticing that the absolute values of these differences are repeated four times, each in one of the four 45°sectors, we reach to the conclusion that through mapping, the 180 directions can be represented by only one of the 45°sectors, reducing the GRM size to one-fourth of its original size. According to Figs. 8(a-d), in which symbols are represented as rectangles with fixed dimensions, we have

|DYa|=|DYc|, |DXb|=|DXd|
Ha = Wb = Hc = Wd and Wa = Hb = Wc = Hd Then the mapping will be as follows: After mapping, the GRM values will be calculated by substituting Θ with Θ m in the new compact GRM as follows: where Θ m represents the unknown mapped sequence direction angle, R represents the relation type (1: RX, 2: RY, 3: RH, 4: RW), i 1 represents the first gene index, and finally, i 2 represents the second gene index where i 2 > i 1 . Before evaluation of the objective distances using the mapped angle in the GRM, the relationships in the four sectors should be redefined based on the geometric rules that appear from Figs. 8 (a-d) as follows.
Sector a: all relationships are defined as in [4] as follows: Sector b: the relationships will be redefined as follows: The relationships for sectors c and d can be similarly deduced as for sector b.
In [5], it was proved that the variation in the measured symbol dimensions is increased with the increase of the inclined angle Θ, hence by performing this mapping, not only the GRM size will be reduced, but also more accurate objective distances will be gained since calculations will be performed in a limited and more stable range which will increase the rigidness of the geometric features and hence maximises the true positive rate. On the other hand, an important enhancement is gained by this modification, if we consider the maximum acceptable objective distance threshold, which was defined in [5] as follows: Where in [5], |Θ k | was changing from 0°to 85°(<90) but now it will be substituted by Θ m which changes from 0 to 45°. Hence, OD max will be less in the two sectors b and d, which should minimise the false positives also. For example, if a chromosome (k) has Θ k equals 85°, then Θ m will be 5°and OD max will be 4.5 instead of 12.5 which surely will reduce the false positives significantly and increase the accuracy of the detected plates at this angle direction.

Objective distance and fitness modifications
The objective distance is calculated as defined in [5], but based on the experiments on the new public dataset [3], some refinements appear to be necessary to increase the system accuracy. Unlike the previous systems where the concentration was on the general architecture of the system, in this paper a deeper solution is introduced by exploring different methods of fine-tuning, especially when faced with a new dataset as in [3]. In [4,5], the relationships (RX, RY, RH, RW) between any two symbols j 1 and j 2 were normalised based on the fir s ts y m b o lh e i g h tH j1 as in (17) This modification should be taken into consideration in the mapping defined in the previous section when performing calculations in the other three sectors. Another modification regards the weighting parameters, which should be chosen based on the stability of the geometric features of the concerned LP as stated in [4]. In case of the public dataset in [3], the X and Y distances between the centres of the symbols are more fixed than the distances between the coordinates of the corners of these symbols. Hence, by using the centre coordinates instead of the corner coordinates, the X and Y coordinate based distances will be nearly stable as the height based distances in case of semi-horizontal plates. Consequently, the same weighting value '3' can be used for all the three distances in the objective function that defines the distance between a prototype p and any chromosome k as follows: where DRX k,p , DRY k,p , and DRH k,p are calculated as in [5]. Hence, the fitness of a chromosome k (Fit k ) will be given as follows:

Mutation and crossover operators
In [4,5], two types of interchangeably used mutation operators have been implemented: substitution operator and swap operator. Unlike what is done in [4,5] and almost all GA based systems mutation will be performed before crossover to prevent disordering of the unevaluated offspring produced by the crossover and, at the same time, allow the crossover to benefit from the done changes by the mutation operator. After experimenting the benefits of the mutation operators and finding that, the work done (objects ordering) by the swap operator is performed more accurately by the sorting crossover a decision was made to use only the substitution operator. The crossover operators are left intact as in [5].

Substitution operator:
In this type of operators, a random position in the chromosome is selected and a new random object from the M available objects replaces the corresponding allele if the random position is in the non-skipped region. Due to the mentioned subgrouping and the sorting of objects based on HF defined in (13), the range of objects from which the random object is selected can now be limited to what we can call the near set. The near set, used to replace an object ( j), has the range [ j − L: j + L], which significantly enhances the effectiveness of the mutation operator in comparison to the previous systems which uses the full range [1: M ]. If the random position is located in the skipped region, a new random number from one to L substitutes the current value at this location as in Fig. 9.

Fake detection and skipping optimisation stage
In the previous version [5], a fake detection stage was introduced to detect false positive outputs from the GA by mapping the resultant sequence to the horizontal direction where the rigidness of the plate becomes maximum (or the variance of the different symbol dimensions becomes minimum) and the objective distance represents the actual difference between the output and the horizontal prototype template. In this paper, another enhancement will be done to optimise the output in case of skipping, where the positions of the skipped symbols can have different arrangements that are all accepted from the perspective of having a final objective distance less than OD max . The lowness of the OD than OD max criteria may result in accepting many cases with incorrect positions of the skipped symbols because the width relationship is not included in the objective distance due to the variation of the symbol widths of most LPs. To correct this, a type of exhaustive search is performed by testing all combinations of possible skipping positions in the horizontal version [instead of doing it in every chromosome in the population (Z ) for speed issues] to reach to the pattern having the minimum objective distance taking into consideration the following heuristic rules: (a) Select the most in-phase sequence where an out-of-phase penalty is measured as follows: (i) Count the number of positive values (XPOS) for the ith x-distance for chromosome k which is defined as follows:  (b) Support skipping in positions that follow wide symbols (connected symbols) where a symbol is considered wide if its aspect ratio is more than 0.7(=0.2 + ASPR(=0.5)) for horizontal plates or (0.7 + tan (Θ k ))/(1 + 0.7tan (Θ k ) for inclined plates. This will support splitting connected or attached symbols while at the same time of suggesting these positions as good skipping positions, the widths of the wide symbols are reduced to their normal values. The support is achieved by introducing a negative penalty of 5 to reduce the final objective distance when skipping is matched with each suggested position. This rule allows splitting the remaining attached symbols (counted as ATS), that may be due to physical reasons other than illumination conditions, at the end of each trial level in the output sequence of the GA phase as shown in Figs. 10e-h. (c) Prevent skipping positions that do not have enough space left between the neighbours of the skipped symbol. When the skipping position faces this condition, a penalty of 5 is added to a skipping error accumulator parameter PrvSk that is added to the final objective distance.
Hence, the optimised horizontal distance OHOD will be given by the following formula: OHOD is calculated at every possible skipping pattern (maximum L!/((L−3)! × 3!) = 20 for L = 6 and SN = 3) and the pattern that leads to the minimum is selected and its HOD is recalculated and checked as in [5] to decide stopping or going to the next run or trial.
With the above mentioned local optimisation, the subgrouping fed by the HF-based sorted objects and the near-set based mutation operator we do a kind of hybridisation although not traditional as defined in the literature [15], it has the same effect of reducing the search time and supporting optimal solution. Fig. 11 displays the number of generations (NumGen) needed to reach the optimal solution versus the number of objects (M ) in case of a single group (circle-markers curve) and in case of subgrouping (diamondmarker curve). Since the population size (Z) increases with the number of objects (M )a sd e fined in [5], it is better to get an indication of the processing overhead, independently of the platform and the implementation, in each case by plotting (Z×NumGen) versus M. Hence, from the (Z×NumGen) curves (double lines), we can deduce that there is a great enhancement in the execution time of the new system because the execution time is (approximately) linearly proportional to the product of Z and NumGen as experimentally proved in [16]. It should be also noted that the execution time in case of skipping will also be reduced significantly because the maximum number of generations in the stopping conditions will be 10 instead of 20 in the previous systems [4,5].

Experimental results and discussion
The evaluation of this semi-hybrid GA (SHGA) based system will be based on the benchmark dataset in [3]. This dataset has three categories of images (AC, LE, RP) based on the application in which these images were collected. The binarisation parameters have been  defined for the three categories as follows (the mentioned constants are used mainly in the object filtering stage in addition to the ASPR which is used in different stages): ASPR = width/height = 0.5, minimum symbol width (MNW) = 2 pixels, minimum symbol height (MNH) = 11 pixels, maximum symbol width (MXW) = ImageWidth/L, maximum symbol height = MXW/ASPR, minimum number of symbol foreground pixels (MNSFGP) = 6.
Although the dataset [3] has all LPs with six symbols, it has two styles. These two styles are detected in the GA using a single default style template due to the small difference between the two styles that happens only in the arrangement of the X-coordinates of the symbols inside the two prototype sequences. The effect of this simplification causes a problem when there are skipped symbols where one style will not be enough to detect the correct positions of the skipped symbols for the other style. To correct this, two GRMs (one for each style) are taken into consideration in the fake detection and skipping optimisation stage by performing the exhaustive search two times; one for each style, which corrects cases having the non-default style (see Figs. 10a-d).
Since the output of the enhanced system is in the form of segmented license plate symbols, the scores are defined in the two steps (detection and segmentation) in a fair way to be well compared with the results in [3]. First, as in [7], a plate is considered detected if the intersection of the detected area and the ground truth area divided by the union of both areas is larger or equal to 0.5. Since in [3] each LP has six symbols, then if four or more symbols are detected out of six, the detection is considered a true positive otherwise it is counted as a false positive. We will perform two experiments, Exp1 and Exp2. Exp1 will be performed based on the described scenario for the three trials. On the other hand, in Exp2, the scenario will be changed by performing the three binarisation steps in one trial and combining their labelled objects into one array after eliminating repeated objects. Also, the role of each trail in Exp2 is changed by using a different binarisation constant (BC) which will be 0.9 in trial 1, while it will be 0.50 in trial 2 and 0.95 in trial 3 to overcome extreme illumination conditions. Since the number of segmented objects will be increased in the same LP area, the GA will be able to get more fit solutions which will increase the true positives as well as false positives outside the LP area. To overcome the problem of the new introduced false positives, we will do two extra modifications. First, since the combined objects will be very close to each other (based on their X-coordinates), the discrimination between different individuals (chromosomes) should be increased to converge to the best by increasing the weight of the X-component in the objective distance and fitness functions to 6 instead of 3 in (27) and (28). Second, the area containing the detected objects is extended longitudinally on both sides by half its original length, if applicable, otherwise, it is increased from either side such that the new area is double the original one. After that, the average number of objects inside the extended area is calculated based on three different criteria imposed on the detected objects such as minimum objects heights (MNH>H/2, results in N h ), the minimum number of symbol foreground pixels (MNSFGP>H, results in N fgpx ), and the average number of longitudinal while-black and black-white transitions (results in N BW ). This number is normalised based on the extended area length (ExL) and orientation (|Θ m |). The resultant number (RN) is defined as follows: If RN is less than (2 × L + 2 = 14), the GA output is reported as TP otherwise it is considered as an FP and the objects inside the extended area are removed and a new run of the GA is restarted. The details of the detection results that represent the number of samples (NOS), number of FPs, number of FNs and how many samples are detected at each trial (Tri1-Tri3) and at each skipping number (SN) are shown in Tables 1 and 2, respectively, where values are displayed as pairs separated by a backslash to show the results of both experiments (Exp1\Exp2).
From the shown tables, the significance of the new binarisation method is recognised where about 13.1% increase in the detection accuracy is achieved by trials 2 and 3 in Exp1. This proves that the SHGA system outperforms the previous system [5] by more than 13% when applied to a dataset such as [3]. Also, the significance of the skipping ability of SHGA is shown where about 45% of the plates are detected in the skipping modes of the GA in Exp1. Although Exp2 consumes more time in each trial, it has a lower percentage of plates detected in the skipping mode (34%) which decreases the average detection time. Table 3 compares the results of SHGA (Exp2) with [3] based on the best module with RP settings (BestM) and the best settings for each category (BestC) considering detection (Det.), segmentation (Seg.), and the combined (Cmb.) accuracy rates, leading to the conclusion that SHGA outperforms the system in [3] by more than 6% (in case of BestM) and 4.2% (in case of BestC) in the detection and 7.8% (in case of BestM) and 5.9% (in case of BestC) in the combined accuracy rates. Table 4 considers the works done by Li and Shen [7] and Soora et al. [8] (in addition to [3]) and presents the results based on the detection rates only because Soora et al. [8] focusses only on the detection, and Li and Shen [7] combine the segmentation and recognition results into one stage. Soora et al. [8] performed LP detection based on clustering of CCA objects depending on the geometric layout of the LP symbols, while Li and Shen [7] used a deep convolutional neural network CNN to detect text in the image then another deeper CNN to filter out NON-LP textual regions based on a large dataset of LP and NON-LP images. Table 4 indicates that SHGA outperforms [8] by 5.5% in the detection of all categories while we mean by '?' that he did not mention these details. Although Li and Shen [7] use the state-of-the-art technology of CNN, SHGA has exceeded its overall detection rate by about 2.1% which is a great achievement if we consider the performance in the development and execution phases. An important shortcoming of [7] regards its execution time where it requires a very large detection time (2-3 s) with the aid of GPU compared to SHGA, which requires an average of 0.12 s (for 640 × 480 resolution) and 0.06 s (for 320 × 240 resolution) with platform as in [5]. A main difficulty with [7] regards its dependency on a large number of samples to train the two CNNs on two different datasets (The first is related to the fonts used and the second is related to the LP layout itself), compared to SHGA which uses only one template based on the geometrical relationships between the LP symbols' bounding boxes irrelevant to which language or fonts do they belong. Moreover, the segmentation in [7] which is combined with the recognition process will require a large number of training samples to train the bidirectional recurrent neural network on any new specific license plate layout. The system in [17] has overcome the speed problem in CNN-based systems by using Faster-RCNN [18] and the Exemplar-SVM, but it reported lower accuracy (93.8% on average; 98.09% for AC, 93.92 for LE, and 89.03 for RP). As a verification test of the robustness of the introduced work, we consider the dataset in [6] which has different types of difficulties such as shadows and blurring. Although the same dataset has been experimented in [4], we redo the test under the same settings of experiment 2 to prove the robustness of the new system. The only change is the enhancement of the role of trials 2 and 3 where the DWSZ in trial 2 will be ImageWidth/L while it will be 0.5 × ImageWidth/(2 × L) with BC = 0.99 in trial 3. The two fixedsize binarisation methods are kept intact in all trials where the first binarisation uses window size equals DWSZ and the second binarisation window size equals the double of DWSZ. The third binarisation method which we can call extension-based variablesize window binarisation is the same as defined in Section 3.1 but with a more robust enhancement that optimises the estimation of the variable window size (w in (10)) in case of high extension value (MExt>ASPR) of a semi-horizontal BCC (|Θ k |<10) as follows: Another enhancement has been gained in this test which regards narrow symbols such as I and 1, which enforce the system to process small non-LP objects, by giving an expression to estimate the values of the minimum symbol width (MNW) and its related MNH as follows: The previous estimations have the effect of filtering non-LP symbols and speeding up the GA iterations. The other modification related to narrow symbols regards their hashing values based on (13) which places them far away from their related neighbours having normal dimensions. The HF is modified to correct this situation and place thin symbols in their related subgroups with their related symbols as follows: (see (34)) In case of LPs having unequal symbol heights such as that of the Saudi LP where the first four digits have heights approximately double of the remaining three letters, we can replace the heights and/or widths in the above equation with their base-2 logarithmic values. The system with the mentioned enhancements has been tested again on the dataset in [3] giving same accuracies as reported earlier, in addition to the improvements in the results when tested on the dataset in [6] in comparison with [4] as shown in Table 5. From Table 5, we notice that there is a 1% enhancement added by SHGA system because the dataset in [6] is not suffering from illumination conditions that result in symbols being connected to each other. Nevertheless, we got better results with a dataset that is categorised based on the image complexity and not based on real operating applications. Considering the system in [9] which has overcome the speed problems in [7] and based his system on a proprietary deep CNN, we will base our comparison on a sample of test images from [3] with extra modified samples from the same dataset Table 4 Comparison of detection results produced by Hsu et al. [3] (BestM and BestC), the authors in [7,8] [3]. After running the published code on the prepared samples, we conclude the following: (i) the system is rotation invariant but with limits (|Θ k |<45) in comparison to SHGA (|Θ k |<90), (ii) the system is scale invariant but to some extent, while SHGA is completely scale invariant, and finally (iii) it may be using the vehicle data (logo position, car boundary, and headlights positions) to prevent FPs because it gives no information when these parts are occluded. Fig. 12 shows a variety of images detected in different categories at different angles and having different illumination conditions. Under each image four attributes are given, the category (AC, LE, or RP), the trial number, the SN, and finally the LP rotation angle. All the shown images are detected by SHGA. The first six rows are detected by Masood et al. [9] while the last two rows contain images that are not detected by Masood et al. [9] except image H3 which has only four symbols detected and images H4 and H5 which have their apparent symbols detected without indication of hidden or missing ones as indicated by SHGA. I think that there are many machine learning techniques that may claim high accuracy on the same dataset by doing tricky ideas as using part of the dataset to train the algorithm and do testing on all the dataset or the remaining parts forgetting the fact that there are many LPs common between the different segments of the dataset [3]. Others use the characteristic of the LP shape or colour to filter out non-LP objects which is not our case where only the geometrical layout of the symbols is considered. As a final demonstration of the capability of SHGA, we present in Fig. 13 three images with multiplates at different scales and orientations that are manually constructed in addition to two original images from [3] to prove the ability mentioned in Section 4.1. Future work may consider more covering and evaluation of the accuracy and performance of the system when faced with multi-plates and even multi-style LPs. Also, CNN can be used to filter out non-text objects to completely eliminate FPs. Experimentally, SHGA proved more ability than previous versions in detecting plates at a variety of conditions that makes it more independent on the type of the application.

Conclusion
An enhancement of the previously designed genetic-based prototype system for localising vehicle plate number inside plane images has been introduced to support fast and accurate detection of LPs having many connected symbols. Hence, a deeper solution of the CCA problems has been introduced at the low level of preprocessing in the binarisation step and at the higher level of postprocessing in the fake detection and skipping optimisation stage after the GA module. In the binarisation stage, a novel variable window size based local adaptive step has been introduced to overcome the problem of connected symbols that cannot be solved by the limited skipping ability introduced in the previous systems. At the fake and optimisation stage, optimum positions of the skipped symbols have been guaranteed by the exhaustive search for the optimum objective distance under all combinations of skipped positions and based on three guiding heuristic rules using two GRMs. Moreover, a more compact representation of the GRM is introduced by minimising its size to one-fourth its original size in [5], which at the same time minimises the maximum objective distance in the two sectors having the ranges [46°-90°] and [−46°, −90°], respectively. A highly significant enhancement has also been introduced by minimising the search space for the GA to guarantee fast and more accurate convergence by introducing the new initial population stage with subgrouping where each subgroup is initialised with a subset of the sorted CCA objects based on a HF that groups LP symbols near each other. The significance of this enhancement has been proven by showing that the search space will increase linearly with the increase of the image complexity represented by the total number of CCA objects instead of growing exponentially in case of a single group. In addition, allowing multiple plates to be detected per image given its symbols are placed in a separate subgroup. Moreover, the effectiveness of the mutation operator has been increased by limiting its mutation space to a narrower set. The introduced enhancements upgrade the original GA to a faster and more stable category as hybrid GAs. An important novelty is provided by maintaining scale and rotation independency in all the introduced enhancements. Finally, the enhancements have been verified by applying the new system on a benchmark dataset, which resulted in a high detection and segmentation accuracy with low FP rate. The accuracy of the new system now competes with the state-of-the-art techniques although maintaining the wide range of angle and scale with independency on the shape or the colour of the LP. The wide range of the different categories of the LP images in the experimented dataset proves the independency of the enhanced system on the intended application.