Variable Search Space Converging Genetic Algorithm for Solving System of Non-linear Equations

: This paper introduce a new variant of the Genetic Algorithm which is developed to handle multivariable, multi-objective and very high search space optimization problems like the solving system of non-linear equations. It is an integer coded Genetic Algorithm with conventional cross over and mutation but with Inverse algorithm is varying its search space by varying its digit length on every cycle and it does a fine search followed by a coarse search. And its solution to the optimization problem will converge to precise value over the cycles. Every equation of the system is considered as a single minimization objective function. Multiple objectives are converted to a single fitness function by summing their absolute values. Some difficult test functions for optimization and applications are used to evaluate this algorithm. The results prove that this algorithm is capable to produce promising and precise results.


Introduction
Genetic algorithm (GA) can be chosen as a meta-heuristic search algorithm when a best possible solution/s of non-linear and discontinuous functions is to be found. GA does not require complex mathematics to execute and it can nd the global optimum even when many local optima available. Genetic algorithm is probabilistic optimization method, that mimics the process of natural selection and it is a kind of evolutionary algorithms (EA). GA generates the solutions using operators such as Selection, Cross Over and Mutation which are inspired by natural evolution. This evolution is governed by a simple law which Charles Darwin named as -"Survival of the Fittest".
In the computer science eld of arti cial intelligence, a genetic algorithm (GA) is a search heuristic [8]. This heuristic (also sometimes called a meta-heuristic) is routinely used to generate useful solutions to optimization and search problems. Genetic algorithm is probabilistic optimization method, which generate solutions to optimization problems using techniques inspired by natural evolution, such as inheritance, mutation, selection, and crossover. The term genetic algorithm, almost universally abbreviated nowadays to GA, was rst used by John Holland [6], whose book Adaptation in Natural and Arti cial Systems of 1975 was instrumental in creating what is now a ourishing eld of research and application that goes much wider. John Holland's original GA is called as Simple Genetic Algorithm (SGA) which has its limitations to handle multi variable and multi objective functions. Subsequently in [7] JR Koza had extended his work and proposed Genetic Programming, the Programming of Computers by Means of Natural Selection.
Later, Akira Oyama et al. [1] have developed a Real-coded Adaptive Range Gas (ARGAs) to nd a solution to an aerodynamic airfoil shape optimization problem. The results show that the real-coded ARGAs nd better solutions than the conventional real-coded GAs. Subsequently, F.Herrera and M.Lozano [2] have developed a Two-loop Real-coded GA with Adaptive Control of Mutation Step Sizes (TRAMSS). TRAMSS adjusts the step size of a mutation operator applied during the inner loop, for producing e cient local turning, thus avoids premature convergence. It also controls the step size of a mutation operator used by a restart operator performed in the outer loop, for re-initializing the population in order to ensure that di erent promising search zeroes are focused by the inner loop throughout the run. Solving di cult nonlinear equation using evolutionary technique is an important research area [11]. Many researcher have attempted this by using neural network and genetic algorithms. The work carried by Crina Grosan and Ajith Abraham [3], presents an evolutionary approach for Solving Nonlinear Equations Systems. They propose an evolutionary technique for solving systems of complex nonlinear equations by simply viewing them as a multi-objective optimization problem. Their approach has given good results for benchmark problems compared to earlier results. Later Satoshi Tomioka et al. [4] also proposed an adaptive domain method (ADM) using real-coded GAs to solve non-linear problems. They have demonstrated the e ectiveness of the new method by citing an example problem. Recently parameter optimization in GA found to be very important. The book [5] describes in detail some aspects of GA parameter optimization for recent applications.
Solving system of equations is very old and an important research problem and many researchers have given important consideration [14][15][16][17]. In 2002 the paper [18] deploys a gradient descent approach to solve nonlinear system of equations. Authors in [24,25] have used a Hop eld type neural network and energy function approach for nding roots of characteristic equation. The computational cost in GA based nonlinear equations solver is an important concern there are many variants to GA have been proposed in the past. An excellent survey on various methods can be found at [26]. Authors in this paper did an excellent job on summarizing the various works related to this area. The article [20] presents the estimation of roots of nonlinear equations using Genetic Algorithm varying the population size, crossover rate, degree of mutation, and coe cient size. In this paper [21], the author rst converted single and simple set of nonlinear system of equations into unconstrained optimization problem, and complex set of systems into constrained optimization problem. Afterward, Genetic Algorithm tool is applied to solve the system. In a recent work [22] the author developed a new approach in which optimum solution of nonlinear system of equations is obtained by a method based on variants of Genetic Algorithm using evolutionary computational technique. The paper [23] describes a novel application of Genetic Algorithm for approximating solution of optimum problems by introducing pairs of harmonious and symmetric individuals. Although many methods have been proposed this paper tries to propose the application of variable search space in nding the solutions of system of equations. While nding solution to system of non-linear equations, the search algorithm has to handle multiple variables, multiple objectives and a very high search space. Hence a new Genetic Algorithm named as 'Variable Search Space Converging Genetic Algorithm (VSSCGA)' is developed, which e ciently handles multiple variables, multiple objectives and high search space. The VSSCGA has been validated with benchmark problems and got very good results compared to previous results.

Search Space
Let the VSSCGA has integer coded representation and each variable (x) is a chromosome and its each gene has integer numbers from 0 to 9. Here the chromosome's length or number of genes in a chromosome is called as Digit Length (nD), which is similar to Bit Length in binary representation. Let nD = . Hence the only possible chromosome representations are , , , , .., . Here it can be say that the VSSCGA's Representation Range (VRR) is 0 to 9. And if nD = , the possible chromosome representations are , , , ..., , , , ..., .
Here the VRR is 00 to 99. Thus if =3, then the VRR is [ , ], and so on. It is assumed that the integer coding representation of chromosome is following the 'base 10' number system. If   As another example let us assume that the problem has 8 variables and all the 8 variables of the solution are rounded to 3 decimal places. Then the GA has to nd a most tted solution among × = solutions or combinations. If the solution set is rounded to 6 decimal places, then the GA has to nd a most tted solution among × = solutions. Hence the required GA should have very high searching capability. And since it needs to handle multi variables, it should have better capability to exit/jump from the local minima than the mutation operator provides. With these requirements a new GA called 'Variable Search Space Converging Genetic Algorithm' is developed.

Variable Search Space Converging Genetic Algorithm
The main concept of this VSSCGA is, it varies its digit length over its cycles and thus it varies its search space. In its rst cycle (i.e. rst digit cycle), its population is represented with a single (which can be varied) digit number (integer) ranging from 0 to 9, for each variable of the population individual. Its second cycle works with two digit numbers ranging from 0 to 99 and the third cycle works with three digit numbers ranging from 0 to 999 and so on. The initial/lower cycles provide a platform for coarse search and higher digit cycles account for ne search. The term converging here meant is that the process of growth chromosome will not be of xed length. It will gradually attain the maximum allowed length. It may also happen that the growth of chromosome digit length will be stopped when the algorithm attains the desired solutions.

. Algorithms and Features
The VSSCGA has many advantages over conventional binary coded GA, which can be summarized as follows [3]: • The solutions can be represented more precisely and thus the computation complexity is amended and the computation e ciency is improved. • Since during initial cycles the SSS is low, the execution time is faster. It takes lesser time to arrive the solution to the expected precision and tness. • It avoids the premature convergence always by its selection procedure.
• As the design variables are coded by oating numbers in classical optimization algorithms, the VSSCGA is more convenient for combination with classical optimization algorithms.

. Structure of Population Individual
In order to GA operators work well with the Population Matrix, and to the objective and tness values also to get integrated with the Population Matrix, it's structure is designed in Matlab as shown Figure 1. An example of the structure of a Population Individual for nV = , nD = , nP = , nO = is shown in Figure 1 and its corresponding total population structure is shown in Figure 2. It is a three dimensional matrix having its rst dimension (row) as nV , second dimension (column) as nD + and third dimension (page) as nP. The value within variable's range V R is computed with an assumption of SSR=[ , ). The advantage of this integrated

. Initial Population Generation
At the beginning of every round a new population is initialized with random numbers (single digit integers) with dimension nV × nD × nP. An example of a new random new population is shown in Figure 3.

. Value Computation
After new population generation, the values V S and V R of each variable of the population individual is computed as explained in section 'Search Space'.

. Fitness Computation
The Value in range (V R ) of all variables of an individual is fed to Fitness function to evaluate the objective function value. Here it is assumed that all objective function values are of positive values or cost/distance value in nature or it is made as positive by adding 'absolute' function or by introducing suitable o set or by some other alternative methods possible.
If the objective of the optimization problem is to maximize, and if there is only one objective function, the Value of the Objective function is directly taken as the Fitness function value . And if more than one objective function is there, the sum of each objective function value is taken as i.e.
If the objective of the optimization problem is to minimize, and if there is only one objective function, the value of the Fitness function V F is calculated as V F = +Vo . And if more than one objective function is there, the Fitness function value V F is calculated as V F = +(Vo +Vo +Vo ) . Fitness function value is very important in GA, as it plays a vital role in selection of population for next generation.

. Selection -Inverse Ranked Roulette Wheel
Selection operator selects individuals in the population for reproduction. The tter the individuals, the more times it is likely to be selected to reproduce. Selection operator works on the principle of Charles Darwin's 'Survival of the Fittest'. Many types of selection are discussed in the literature including Roulette wheel selection, Rank selection, Tournament selection etc. Each method has its own advantages and disadvantages.
Here a combination of Roulette wheel selection with Rank selection is introduced. This algorithm assumes that all V F ≥ . First the individuals are sorted in descending order depending on their tness value. In this order, the highest t individual has the rst place and the lowest t individual has the last place (i.e. nP th place). Then each individual is assigned with a rank of /( + itsplacevalue). Hence the name is Inverse Ranked. The highest t individual or rst individual is having rank , the j th individual has the rank /( + j) and the last individual has the rank /( + nP).
Similar to the normal roulette wheel algorithm, in this method also, the probability of tness of each individual is computed and expected count of each individual is calculated. Then the Ranked Expected Count (rec) of each individual is computed by multiplying expected count with its rank and the value is rounded up (ceil) to integer values.
The Ranked Expected Count of j th individual The rec of an individual who is having V F = , is assigned as 1. With this expected count the remaining procedure of roulette wheel is carried out. Each individual is reproduced their corresponding rec times. Then rst/top nP individuals are selected for the next generation. This IRRW selection improves the searching capability of the GA. Its main advantage is no single individual can make the complete population saturated throughout the generations, thus it always avoids premature convergence.

. Cross Over -Binary Imitative
The crossover operator is the most important operator of GA, because it leads to convergence [9]. In crossover, generally two chromosomes, called parents, are combined together to form new chromosomes, called ospring. Random pairs are selected for cross over from the complete population of IRRW selection. Many real coded cross over types are discussed in the literature. To get advantage of complete randomness, the binary imitative uniform single point cross over is used. Two pseudo random numbers are generated, rst being real oating number having range 0 to 1, to check the 'go ahead status' of the cross over and second being integer having range 1 to nD, to decide the cross over point. If the rst random number is less than or equal to Pc, the cross over is carried out. The second random integer number gives the place/position of the cross over point. The genes after the cross over points of both parents are interchanged. An example of cross over is shown in Figure 4.

. Mutation -Binary Imitative
In genetic algorithms, mutation realized as a random deformation of alleles with a certain probability. Mutation can occur at each bit position in a string with very small probability (e.g., 1%). Mutation plays a critical role in GA. As discussed earlier, crossover leads the population to converge by making the chromosomes in the population alike. Mutation reintroduces genetic diversity back into the population and assists the search escape from local optima. Many real coded mutation types are discussed in the literature. Similar to cross over to get advantage of complete randomness, the binary imitative uniform single point mutation is used. All individuals from cross over undergo mutation operation. Instead of considering all genes, the whole chromosomes (i.e.variables) are considered for mutation with probability Pm. Thus in each chromosome a single gene may have the chance to get mutated. This will reduce the number of random numbers to be generated and thus improves the execution speed. In order to get the same mutation performance, the mutation probability Pm may be increased. Typically it may have value from to %. Here in mutation three pseudo random numbers are generated, rst being real oating number having range 0 to 1, to check the 'go ahead status' of the mutation and second being integer having range 1 to nD , to decide the mutation point and third also being integer having range 0 to 9, is the value of the digit to be interchanged. If the rst random number is less than or equal to Pm, the mutation is carried out. The second random integer number gives the place/position of the mutation point. The gene at the mutation point of the parent is interchanged with the third random integer. An example of mutation is shown in Figure 5.

. Rounds of Iterations
In this VSSCGA, multiple iterations are called as a round. Each round starts with a new random population. As such to introduce random new components to the overall algorithm, the algorithm can be run for many rounds. Since it re-initializes the whole population after every round, this prevents premature convergence and gives greater searching capability. In the rst digit cycle, since the search space is less, GA can nd the rounded or approximate value of the solution easily. It is similar to coarse search. In the successive digit cycles, as the search space increase, GA does a ne search or ne tuning of the solution.

. Elitism
De Jong (Jong) [10] suggested a policy to always include the best individual of one generation into next generation in order to prevent losing it due to sampling e ects or operator disruption. This strategy, which can be extended to copy the one or more best solutions to the next generation, is denoted as elitism. The idea is to avoid that the observed best-tted individuals dies out just by selecting it for the next generation without any random experiment. Elitism is widely used for speeding up the convergence of a GA. The negative e ect of the elitism i.e. premature convergence is taken care in VSSCGA by introducing IRRW selection.
In order to have more randomness and searching capability, VSSCGA has the provision to start the elitism operator at any iteration (i.e. at rst iteration or at higher iterations less than Max Itr ). Two settings are specied regarding elitism. First one is number of best individuals (nElit) to be preserved for elitism and the second one is iteration count (Elit Itr ) from which the elitism operation starts. The recommended value for nElit is at least 2 and at the maximum of 0.1×nP. And the recommended value for Elit Itr is . ×Max Itr , which gives equal chance for non converging search and converging search. It means that in every round until . × (Max Itr ) th iteration, elitism is not working, hence GA works on complete random search in each iteration, with very little probability for convergence.

. Migration
Elitism operator takes care of carrying the best individuals to next iterations. An another operator like elitism is required to take care of carrying the best individuals/solution to next rounds and digit cycles. This operator is named as Migration operator. In the rst digit cycle migration works after Round > , and for other digit cycles migration works on all rounds. That is, migration preserves the best individual of the rst round of the rst digit cycle and carries to the next rounds and next digit cycles. Elitism operator introduces the best individuals in every iteration after Elit Itr . But the migration operator injects the best individuals only once at the iteration Migr Itr in a round. If it is ensured that Migr Itr > Elit Itr , after the injection of best individuals by migration, the elitism carry them to next iterations. In order to get best converged result, the stopping criterion other than Max Itr , will get activated only after Migr Itr + and Elit Itr + . The best individuals of migration are carrying forward to next rounds within a digit cycle without any scaling or modi cation. But when the migration works to next digit cycle, the best individuals of the current digit cycle have to be transformed (i.e. to undergo scaling or suitable transformation) based on the tness function and variable range, in such a way that the values of variables of the best individual in the current cycle is equal to the values of variables in next cycle. .

Best Solution Selection
The best solution selection operator gives the answer of the GA optimization.  The stopping criterion of the digit cycle is only based on stop digit. The iteration loop stops at the Max Itr , irrespective of any other condition of the GA. The iteration loop stops, when the best t of iterations is consistent over the speci ed number of iterations (nCons Itr ). If the speci ed number of individuals (nSatu Pop ) in a generation population are having the same tness value ( tness saturation), then also the iteration loop stops. A minimum tness (min Fit ) threshold is provided and until GA achieves this threshold, the last two stopping criterion will not get activated. To implement this provision, the user should have the knowledge of minimum tness value which can be achieved as Best Fit . As it is discussed in the migration section, the last two stopping criterions (i.e. other than Max Itr ) will get activated only after Migr Itr + and Elit Itr + to get best converged result.
The round loop stops at the Max Rnd , irrespective of any other condition of the GA. And the round loop stops, when the best t of rounds is consistent over the speci ed number of rounds (nCons Rnd ). The digit cycle starts at St dgt and stops with Sp dgt .

Tests and Results
In order to verify the e ectiveness of this VSSCGA, it is tested with two E ati and Nazemi examples, 4 test functions for optimization and two benchmark optimization problems and the results are obtained.

. E ati and Nazemi Examples . . E ati and Nazemi Example 1
The E ati and Nazemi example 1 (E ati.S) [11] contains a system of two equations with two variables. The variable's range used in VSSCGA is ≤ x , x ≤ .
The settings of VSSCGA for this example are given in Table 1. The results of every digit cycle are shown in Table 2. The comparison of VSSCGA's results with earlier results (Abraham) is shown in Table 3. In Table 3 we have performed a comparison of the popular numerical methods and GA based methods including the proposed one on E ati and Nazemi Example function. We show here the values of xi and Functions values and moreover, the function values with the proposed method are very accurate. Further, it is interesting to observe in Table 2 that on how the variations in nD a ect the accuracy and precision of the solution. It is also seen here that going to a higher digit length is not required in certain cases. Hence, the variable search space idea does help in reducing the complexity implicitly without a ecting the solution accuracy.  Table 2 is only upto Sp dgt = . If the VSSCGA is allowed to do more number of digit cycles, it can give better precise results.  .

. E ati and Nazemi Example 2
The E ati and Nazemi Example 2 (E ati.S) also contains a system of two equations with two variables. The variable's range used in VSSCGA is ≤ x , x ≤ .
The settings of VSSCGA for this example are given in Table 4. The results of every digit cycle are shown in Table 5 . The comparison of VSSCGA's results with earlier results (Abraham) is shown in Table 6.    The ow of Max Fit over iterations, rounds and digit cycles is shown in Figure 6. It can be noted that VSS-CGA has found the solution in th iteration of the rst round and rst digit cycle itself. This graph explains many concepts of VSSCGA.

. Test functions for optimization
Since the e ciency of VSSCGA for two variables and two objective test functions is proved, the VSSCGA is going to be tested with higher dimension test functions and to precisely estimate the values of Scha er 4 function than the values given in the literature [13].

. Rosenbrock's function
The function de nition of Rosenbrock's function (wikipedia.org) is where i = , , , ..., n − It is a single objective, fundamentally 2 variable system. The variable range used is− ≤ x i ≤ . The function minimum is at x i = .
First 2 variables are considered for this function. The settings of VSSCGA for this example are given in Table 13 11. The results of every digit cycle are shown in Table 14.  Second four variables are considered for this function. The settings of VSSCGA for this example are given in Table 15. The results of every digit cycle are shown in Table 16. Finally eight variables are considered for this function. The settings of VSSCGA for this example are given in Table 17. The results of every digit cycle are shown in Table 18.  which the solution  obtained  ( , , , ) , , ( , , , ) ( , , , ) ( , , , )

. . Himmelblau function
The function de nition of Himmelblau's function (K. Rajan) is where i = , , , ..., n − This is also a single objective, fundamentally 2 variable system. The variable range used is ≤ x i ≤ . The function minimum is 0 at x i = ( , ) for n = .
First 2 variables are considered for this function. The settings of VSSCGA for this example are given in Table 19. The results of every digit cycle are shown in Table 20.  Second four variables are considered for this function. The settings of VSSCGA for this example are given in Table 21. The results of every digit cycle are shown in Table 22.  Finally eight variables are considered for this function. The settings of VSSCGA for this example are given in Table 23. The results of every digit cycle are shown in Table 24.   which the  solution obtained  ( , , , , , , , ) , , ( , , , , , , , ) ( , , , , , , , ) ( , , , , , , , )

. . Scha er 4 function
The Scha er 4 function (wikipedia.org) [12] is also a good example to test VSSCGA for its ability to get the solution very precisely i.e. it's converging feature to very accurate solution. The function de nition is These are two variable single objective functions. As per the literature (wikipedia.org) the variable's range is − ≤ x i ≤ , the function value f (x) = . at x = ( , . ). First VSSCGA is tested with range given in the literature i.e. − ≤ x i ≤ , and see that whether it is able to get the values of x and f as given. Then to nd the values of x and f more precisely, the range ≤ x , x ≤ is used. The settings of VSSCGA for the rst case are given in Table 29. The results of every digit cycle are shown in Table 30.  ) .
. , , The result shows that the VSSCGA provides promising results. Then it is decided to estimate the value of the function more precisely than the value given in literature (wikipedia.org) (in nity77.net) [13]. The settings of VSSCGA for this case are given in Table 31. The results of every digit cycle are shown in Table 32.  Here it is found that the minimum value of Scha er 4 function is . at ( , . ). The digit cycle setting Sp dgt is set to 10, because the tness value and function values have saturated at Matlab's capacity. Crina Grosan and Ajith Abraham have reported 12 solutions for this problem. Among those the twelfth solution shows least (minimum) function values. Hence this twelfth solution is compared with the nal results of VSSCGA. On reviewing the results of Crina Grosan and Ajith Abraham, the variable range kept for this case is − ≤ x i ≤ , i = , , ..., and the settings of VSSCGA for this case are given in Table 35. The comparison of results of VSSCGA with Crina Grosan and Ajith Abraham's twelfth solution (Abraham) is shown in Table 36.  Here one more set of results of VSSCGA are given, which shows that as like GA, the VSSCGA also gives better results over time. The settings of VSSCGA for this case are given in Table 3.41 37. The comparison of results of VSSCGA with Crina Grosan and Ajith Abraham's twelfth solution is shown Table 38. All the above test cases show that the VSSCGA is giving very precise and promising results. The choice of the various pa-  rameters in VSSCGA varies with the problem at hand. However, from our experiments we infer that like all other GA algorithms it is very easy to nd the appropriate settings for VSSCGA too. The advantage we get here is that now we have a variable search space and that intuitively will converge faster than that of traditional GAs.

Conclusion
A new Genetic Algorithm that can vary its search space size and nd the solution of non-linear equations system very precisely is introduced. A new Inverse Ranked Roulette Wheel selection is introduced to overcome the premature convergence. The binary imitative integer coded uniform cross over and mutations are adopted. The adopted mutation decreases the execution time. Elitism and Migration are adopted after some prede ned iterations, and thus improves the randomness and carry the best solution throughout the run. Even though it can solve many problems, some benchmark problems are tried to test the performance of the VSSCGA and very good and more precise results are obtained. VSSCGA con rms that it can handle multi variable, multi objective and high search space problems very e ciently.
In future variable rate mutation may be integrated with this GA to analyze its performance. The VSSCGA may be used for other applications where other optimization techniques do not give good results.