Mone: Mutation Oriented Norm Evolution

A norm regulates the run-time behavior of the agent with the action and condition to trigger the action. Because of the incomplete understanding of the world, the result of the action may be different for the same agent with the ‘same’ context, or unintended different context. To study this phenomenon, the classical norm definition is extended from the condition-action pair to cover the expectation, in order to verify the result of the action. The Norm evolution can be defined as a gradual process which makes a norm more complete and effective. In the terminology of evolution, a norm is called mutated if the result contradicts to the expectation, i.e. at least one of the expected conditions is invalid. At run-time, norms are executed in series. A mutation brings new knowledge to the following states and might affect the later execution of the norms. Such knowledge provides will help the norm designer to complete the definitions. A mutation based norm evolution (Mone) method is proposed in this paper to detect the mutations, to propagate the evidence and to crossover the norms for completeness. The method is formalized in the Description Logic, and implemented with the algorithms for mutation detection and norm crossover. The case study illustrates the Description Logic $\mathcal {ALCI}$ of the method and shows the potential to evolve the norms autonomously in the Blackboard system, GBBopen.


I. INTRODUCTION
One of the most famous multi-agent system frameworks is the blackboard based system (BBS) whose history goes back to 1980's [1]. Open source frameworks such as GBBopen, 1 Repast [2] and Repast HPC [3] are used as the backbones of advanced science, engineering, and policy analysis. The BBS architecture provides a flexible framework to solve complex problems with blackboard, knowledge source (KS) and control shell. The blackboard serves as a space for common knowledge sharing from various KSs. Each KS can be regarded as an agent that operates autonomously and changes the shared knowledge on the blackboard in a controlled manner with predefined functions and trigger events, which are the action and the conditions of the norm.
The flexibility of BBS leads to complex interactions between the KSs and blackboard, through the transition of the states by the execution of the predefined norms (or the norm's actions). Norms are designed to be executed in series such that the result of a norm triggers the next norm. But it is The associate editor coordinating the review of this manuscript and approving it for publication was Hiram Ponce . 1 http://gbbopen.org/index.html not always the case as the context can never be studied completely. Some executions may follow the expectation, some may lead to exceptions. However, there may be a number of exceptional results that are identified as significant for other norms. Such an exceptional case is called a mutation i.e. the action is successfully triggered but the result of the action contradicts the expected outcome (negation of one of the expected outcome conditions). Mutations often, if not always, emerge in the real world applications because the predefined norms are always an incomplete description of the regulatory knowledge.
To reconcile this incompleteness of predefined knowledge, the norms should and have to evolve. Then the questions arise as what is the new knowledge for the evolution of the norms? where does the new knowledge come from? and how does the norm evolve with such knowledge?
To answer these questions, the BBS system is abstracted into an agent-based scenario. The norms are extended to incorporate expected results as expectation, in addition to the action and the trigger conditions that fire the action. The KSs are abstracted as norm executers that verify the knowledge on the blackboard whether its norm is triggered and execute the action and publish the result to the blackboard. The blackboard is abstracted as a state transactor that a state is denoted by all the knowledge on the blackboard and transits to another state, denoted with all the knowledge provided by all the KSs that executed the triggered norms. The system operates with a series of actions, i.e. to execute several norms sequentially in the transitions of states.
For example a robot that sweeps the room should follow a norm like the following: • Trigger: Time≥8:00AM • Action: Sweep • Expectation: Finish sweeping and Time≤8:30AM The norm is executed autonomously if its trigger conditions are satisfied. It should finish sweeping as the expectation prescribed, but not necessarily as the norm might mutate because of power shortage, broken arms, obstacles, etc. which can never be exhausted in an open world. The robot may follow the same series of norms everyday according to the designed norms. Such kind of routine executions do not provide any 'new' knowledge to the designer therefore it is hard to revise the norms in such routines. On the other side, mutations may take place in some of the days, the unexpected result may provide extra knowledge to the execution series and affect the execution of later norms. This paper explores the evolutionary behaviors of norms, i.e. inheritance, mutation and crossover. The result of the executed action can be either coherent or contradictory to the expectation of the norm. The norm is called mutated if any of the expectation is violated by the result. In the different series of norm executions, if a norm, say x mutates on one track while performs normally on another, x is supposed to evolve with some extra knowledge. We use the Layered Execution Graph (LEG) defined in the modeling section to analyze the different norm executions. It is found that other than mutated, in the track that x does not mutate, there exist other norms, say y, z . . . which mutate ahead of x. The mutations of y, z . . . provide extra knowledge not prescribed in the norm construction stage. It is also found that such extra knowledge can be inherited to x through the tracks of the LEG. x can evolve by crossover with the mutated norms y, z . . . The paper proposes a novel mutation oriented norm evolution method: Mone. It enriches the trigger condition of the norm that mutates, with the expectations of some other mutated norms. The contributions of the paper lay in 3 aspects: • To find the relationship between different executions of the same norm, from a mutated one to a well-executed.
• To prove the existence and inheritance of the extra knowledge from mutations.
• To provide algorithms to construct the Layered Execution Graph, to detect the mutations and to evolve the norms.

II. STATE-OF-THE-ART
Knowledge based systems are always facing the incompleteness of the knowledge. This is especially challenging in a multi-agent blackboard based system [1] where common knowledge can be changed by different knowledge sources [4]. One of the most influential aspects leads to the 'abnormal' executions of the norms, which regulate the runtime behaviors of the agents. It demands the norm evolution with the growth of knowledge. Current norm evolution studies mostly fell into 3 categories, norm formalisation, norm conflict resolution and norm failure localisation. Fisher et al. discussed the relationship between computational logic and agents [5] in agent specification, implementation and analysis. Many logical frameworks are proposed to formalize the domain [6]- [8]. Besides, model checking [9] and belief-desire-intention framework [10] are also applied for norm representation. But this is not in our focus because we emphasize on the use of mutation rather than its representation.
Vasconcelos et al. proposed the norm with constraints and a framework to detect and resolute conflicts in [11]. Silvestre and Da Silva developed a method to resolute conflicts among multiple norms [12]. Santos et al. classified [13] the conflict resolution approaches into direct and indirect detections and surveyed the progresses. Kayal et al. proposed to resolve conflicts in social commitment automatically through a conflict resolution model based on relevant user values such as privacy and safety [14]. This thread focuses on the resolution of a given norm set, which is not the problem we want to solve.
Passos et al. found the error-prone nature of multi-agent systems and focused on the behavior failures via spectrumbased fault localisation [15]. It has been improved by the accuracy graph method in 2017 [16]. Alechina et al. discussed the imperfect monitor of runtime norm executions [6]. Huang and Alexander explored the semantic mutation testing to assess tests and program robustness [17]. The Silk framework was proposed to monitor and resolve the conflicts of norms [18]. The work in this thread focuses on the conflicts between different norms. The common solution is to evaluate and place a priority on the conflicting norms. It is different from the Mone method in that we focus on the different behaviors of the same norm on multiple executions and explore the possible reasons inherited from the execution results of other norms that are mutated and generate expected knowledge useful for the evolution.

III. MOTIVATING EXAMPLE
In an agent-based system, a sweeping robot cleans the room. The robot automatically starts cleaning from 8:00 AM if set in the Auto mode. It is expected to take the robot less than 30 minutes to complete the job. Everyday, the robot finishes sweeping before 8:30 AM; except one day, the robot was stuck by an obstacle and did not manage to remove it, which resulted in the failure of the task. To detect the reason, the robot was set to the monitor mode, and tested for exceptions. In this case, an obstacle was found and its weight was beyond the ability of the robot, therefore removed manually by the monitor. Then without the obstacle, the job was finished in time. Yet another day, the job failed again. In another run of the monitor mode, it was found that the robot ran out of battery half way of the job. By manually charging, the robot recovered and finished the job.
With the above findings, it's rational to refine the norms that regulate the robot with the evidences found in the monitor mode, such that the conditions to start the sweeping job should incorporate not only Time==8AM and Mode==Auto, but also Obstacle≤1kg and Battery≥80%.
To study the different behaviors of the agent in multiple series of norm executions, we introduce the so-called layered execution graph. To describe the execution path, let v i represent the name of norm, s i represent the state. Figure 1 gives an example of the graph.
The graph represents multiple series of executed norms. In the first state s 1 , it starts from a root vertex as the common head of the different execution series. Each state denotes a set of conditions, such as Time==8AM and Mode==Auto. States grow in a horizontal direction sequentially. Each vertex denotes the (multiple) execution of a norm, triggered in the conditions of the state. Directed edges represent the order of the executions. The multiple outgoing edges from a vertex denote the different execution results of the norm. The solid edge denotes that the norm has been executed as expected; the dashed one denotes the result has some condition that is not prescribed by the norm. There are some execution paths in Figure 1. One series of execution is (v 2 , v 8 ) that the task starts and completes normally, while another is (v 2 , v 9 ), which indicates the robot is stuck and the task fails. Yet another series is indicates that the obstacle detected by the test is removed and the sweeping can be completed. Yet another series is , which indicates the robot is start sweeping in the monitor mode and the power shortage problem is settled by recharging, after the power test which is in turn an internal system test.

IV. MODEL
For better formalize the scenario, the Description Logic ALCI (attributive concept description language) is used with a real world interpretation I of the TBox (Terminological Box) and ABox (Assertional Box). The Description Logic is divided into a TBox part and an ABox part. The TBox defines the structure of the domain and includes some axioms, the ABox describes examples of the domain and including some axioms asserting.
The major entities such as norm, condition, state, action are defined as DL classes and the auxiliary entities such as time, location, weight, etc. are left as predefined domain attributes for data properties. In a general agent-based context, a state describes the environment with the features as conditions; a norm regulates the way an agent functions given a certain state. Traditional norms describe the regulation in a pair (request, action), where request ⊆ state describes the necessary conditions to trigger the action.
But it is not good enough to study the outcome of the action in such a definition. In our findings, a norm should clarify two states: • The state satisfies the norm's request in order to trigger the action; • The state of the agent after the action, as the expected result prescribed in the norm's expectation. If the outcome of the norm execution is not as expected, a exceptional situation occurs. From a norm definition point of view, it is called a mutation that the prescription of the norm is not complete and effective. To clarify the scenario, we have the following definitions and theorems.
Definition 1 (Condition): A condition describes the state in one dimension/perspective. It is formalized as an individual c of the prime class Condition with the ABox assertion as

Condition(c)
In certain context, a condition can be either satisfied or unsatisfied. Therefore, we define the class Satisfied as the subclass of Condition for those interpreted true in the context, and Unsatisfied ≡ Condition ¬Satisfied. Then if a condition c w.l.o.g. is satisfied, there exists a pair-wise counter condition not c that is unsatisfied in the same context. Here in the syntax of DL, negation is not applicable on an individual, thereforẽ c is used for the counter condition of c.
The condition class can be further classified into subclasses as time-related condition, space-related condition and other facilitated set of domain individuals.
For example, there are two individual conditions p 1 , p 2 where p 1 indicates that the mode of the robot is set as auto and p 2 indicates the current time is NOT 8 AM. If the current context interprets that the robot is in auto mode and the time is 8 AM, then the following ABox assertions hold: Satisfied(p 1 ), Satisfied(p 2 ).
where < is the strict order in the time domain.
Suppose that p 3 stands for the condition that the robot finishes sweeping in half an hour. The condition p 4 stands for the robot changes its mode into Finish. If one day at 8:30 AM, it is observed that the sweeping task has not finished, then the state is modeled as the ABox assertion ¬hasCond(s 1 , p 3 ).
To describe the behavior of the agent, the prime concept Action is defined. where subState(tri, pre) and State(res) holds.
In most of the cases the agent executes the norm and the result state is coherent with the expectation of the norm. But this is not always the case because in the real world, a norm is hardly perfect defined for the partial understanding of the constantly changing context. The execution is called exceptional if its res is not coherent with the exp of the norm. Such exceptions provide extra information not expected in the definition of the norm system. In the exceptional execution, the norm is called mutated because it behaves differently from expectation.
Definition 6 (Mutation): A norm n=act(tri, exp) is in mutation iff there exists an execution e=act(pre, res) such that there exists at least one condition in res which is conflict from a condition in exp. Formally Mutation(n,e) iff the following class is satisfiable (has individual in any interpretation).
inState : res ¬inState : exp Here the complex class is satisfiable in DL semantics iff in any interpretation I there exists an individual c such that c I ∈ (inState : res ¬inState : exp) I , which says there exists a condition that is the result of the norm execution, but not expected in the norm. According to the Open-world Assumption of the DL semantics, negation as failure is not assumed. The satisfiability of the class is validated with all the named entities only.
The motivating example is shown in Figure 1. A norm n 1 = act 1 (tri 1 , exp 1 ) is applicable in any state s 1 such that subState(tri 1 , s 1 ), where exp 1 is the expectation with hasCond (exp 1 , p 3 ) and hasCond(exp 1 , p 4 ). If the state s 2 is observed to have the condition ¬p 3 after the a 1 executes the norm r, then r is mutated in s 2 as ¬hasCond(s 2 , p 3 ) contradicts the expectation e 1 . With a general purpose DL reasoner, it is easy to detect which norm has been mutated at runtime.
A mutated norm provides extra evidences not expected in the design phase of the norms. Such evidences may affect the execution of the norms afterwards. Therefore, it is rational to track the path of the norm executions.
Definition 7 (Path): A path describes a sequence of norm executions. It is formalized as a listē, with element e k = act k (pre k , res k ) that for each k, 1 ≤ k < |ē|, the following TBox holds inState : res k inState : tri k+1 where tri k+1 is the trigger state of the norm e k+1 executes. In path, the norms are executed in a series of states that triggers the corresponding norms. For a certain state, it is not restricted that only one norm can be executed, i.e. the state will trigger all the applicable norms. Given a sequence of state transitions, if each state triggers only one norm, and the execution results are not conflict to the following state, then a path is constructed; if in some state, multiple norms are triggered, the path will split to multiple successors e l k+1 while 1 ≤ l ≤ N tri , where N tri denotes the number of norm that triggered by some conditions. As norms are executed in the time domain, a path will grow into a tree and even an acyclic directed graph.
The agent should complete the task according to the designed norms. However, although the paths are predicted to be the same, mutations do exist because of the incomplete understanding of the constantly changing world. A mutation may terminate the path, while it may trigger other norms and continue the path as well. No matter in which position the norm is mutated, it marks the unexpected behavior in the trigger context, and indicates the necessity of evolution for the mutated norm. Therefore, the mutations in the paths should be the anchors for the norm evolution.
In the real world, agents are usually designed to complete a predefined task, repeatedly if necessary. For example the sweeping robot will sweep the floor everyday. The paths will be high-probably the same if the robot works normally. To find the mutation (i.e. to position which norm to evolve), it is rational to take as a whole such 'repeated' paths. By merging all the paths for the same agent on the same state transitions, a directed acyclic graph is constructed. LEG is constructed to position the mutation of norm execution, therefore it records the paths of the same agent in the 'same' state transitions. To differentiate the mutated execution from the normal ones, the former connects to the next vertex with a dashed edge in contrast to the common solid edges.
The agent can execute a series of norms multiple times in the 'same' context, such as in the motivating example, the robot sweeps the floor everyday. But the context may vary in an unexpected fashion that the action may have different result from the 'same' trigger state. In the real world semantics, the result state of the action may either obey the norm or mutate it. For example, the state of layer s 8 in Figure 1 incorporates the trigger state of the norm a 2 (hasCond : p 2 , hasCond : p 3 ). But the execution result is a 2 (hasCond : p 2 , ¬hasCond : p 3 ) that the agent did not finish the sweeping task by 8:30 AM. Recall the example that the condition p 1 : mode == auto and p 2 : Time == 8AM in Figure 1. The state s 1 satisfies that hasCond(s 1 , p 1 ) and hasCond(s 1 , p 2 ). Afterwards, the state s 2 still satisfies hasCond(s 1 , p 1 ), as there is no explicit change of the mode of the robot. But p 2 is changed for the time elapses then hasCond(s 2 , p 2 ) is not valid anymore. Theorem 1 shows that in a track of a LEG, the afterwards state tends to inherit the conditions if not explicit objected. In the Open-World Assumption, such resistance is of great value for inference with incomplete knowledge.
Theorem 2 (Evidence Existence): A norm is triggered in a state, and mutated in the state after, then there exists some condition in the former state which is not the trigger condition of the norm, that has been inherited to the state after and contradicts to the expectation of the norm. Formally, given before(S, S ), a norm R = A(T , E) and its execution A(T , S ), there exists a condition C that Proof: Assume that ¬inState : T inState : S ⊥, then for each condition C, hasCond(S, C) → hasCond(T , C); S triggers R ⇔ subState(T , S) ⇔ for each C, hasCond(T , C) → hasCond(S, C); Then T ≡ S. As the action A is functional, S ≡ E, which contradicts to the assumption.
Theorem 2 assumes the actions are executed functionally, i.e., for the same state, the action result should be the same. Although an action may be not functional in the real world, non-functional actions are not in the scope of this paper. Besides, there exist unexpected outer forces in the real world to change the conditions, which are not considered in our Theorem 3 (Pairwise Mutation): Given two tracks of norm execution sequences of the same agent, a norm is obeyed in one but mutated in another, there should be some extra evidence lead by another mutation ahead in the obeyed track, that is inherited to but not specified in the trigger state of the norm. Formally, given two tracks α and β that w.o.l.g. a norm R = (T , E) is mutated in α and obeyed in β. There exists some condition C as the result of the execution of the norm R = (T , E ) in β ahead of R, such that ¬hasCond(E , C) mutates R , and C is inherited to the norm R that the state ∃hasCond.({C} ∃inState.T ) triggers R and prevents R from mutation.
Proof: It is assumed that no outer force is detected to change the conditions in the states on the tracks, therefore the conditions may change only because of the action execution. All the actions are functional, then the norms on each vertex are either obeyed or mutated. As the norm R is applicable on both α and β, there exits some evidence inherited to the norm R through α that leads to the different performance from R on β. If no norm is mutated on α ahead of R, the inherited conditions should be the same as those on β, then the action on both tracks will produce the same result. This is against the fact that R is obeyed, so there exists some norm mutated on α before R.
In Figure 3, the left path illustrates α and the right pictures β. The dashed arrows illustrate the mutations of the norms. From Theorem 2, the reason R is mutated on Q but obeyed on Q is because there exists another norm mutated on R and the conditions are inherited to Q by Theorem 3. With the open world assumption, there is always some condition(s) not considered at the time of norm construction. Moreover, there might exist some unnecessary conditions that were added to the trigger of the norm not on purpose. Intuitively, the expectation of the norm is regarded as the precise result from the norm action.

V. ALGORITHMS
Norms execution process based on BBS is mainly implemented by Algorithm 1.
The Algorithm 1 shows how the norms execute on the blackboard system. The Control Shell (Line 1 and Line 16) is one of the three major components of a blackboard system Que ← TriggerEvent(NormList); 16 Control Shell Exit; along with KSs and the blackboard. The control shell directs the problem-solving process by managing how KSs respond to contributions that are placed on the blackboard by an executing KS and to other events that may be triggered by the application or received from external sources. The Trig-gerEvent method (Line 2, Line 15) is proposed to detect norms which can be triggered by the information of blackboard. The CreateSignal method (Line 7) generates a signal to trigger the corresponding KS. The Action method (Line 8) returns the results of KS. We can get the expected results of a norm using PostCon method (Line 9). If the results of norm N are consistent with the expected results, it will add N 1 to the execution path. In contrast, N 0 represents its results are not coherent with the expectation of norm N . The updateBB method (Line 14) updates the information on the blackboard according to ResCache.
The MONE method is designed in three steps: • To construct the LEG of the multiple norm execution tracks of the same agent; • To detect the pairwise mutations in the LEG; • To evolve the norms related to the mutations. We propose Algorithm 2 to construct the LEG for the different behaviors of the same agent on the norm tracks.
It starts with a special state s 0 that triggers at least a norm. A layer of conditions is constructed with the result of the action starts. The built-in methods are called to create a layer (Line 1), a vertex (Line 8), a dashed edge (Line 13) and a solid edge (Line 15). As a layer represents a state, and a vertex represents an executed norm, the get method returns a state (Line 4) and a norm (Line 10). The getResult method returns the effect of action a in the state s. If it contradicts to the expectation e of the norm r, the vertex v is marked as mutated.
The getTime method (Line 23) returns the current time when the algorithm runs. It is added with the updateState function (Line 24) to the inherited conditions in s together with the effect e of the action, following Theorem 1.
Algorithm 2 creates the LEG for one time series as the layers are built sequentially with time elapses. In a scenario with multiple agents of the same type simultaneously, multiple such LEGs can be constructed. It is expected that all the norms are executed as expected, and all such LEGs are the same. But as is explained above, mutations may happen and there exist different edges even for the same type of agents with the same set of predefined norms. The expectations of the norms are compact and precise when defined to prescribe the result of the execution. The mutated result offers more information to evolve the norm definitions.
Then Algorithm 3 is proposed to detect the mutations in the LEG.
As the vertices are merged representing the same norm, Algorithm 3 detects such vertices that have different results in execution processes. It queries for the vertices that have both a dashed edge (mutated) and a solid edge (obeyed) which represent the different effects of the same action. Then it tries to create a track between two such vertices (Line 13). Such tracks are used to evolve the norms as in Algorithm 4.
It takes a track that follows Theorem 2 as the input. The norm r on the tail of the track is the destination norm of the evolution. The algorithm takes the norm on the head (Line 2), verifies the norm execution through the track (Line 3) and crossovers the mutated norm with the destination one by one. The ⊕ operator (Line 7) takes all the conditions ∃inState.e i , verifies if its negation can be deduced from t. The condition is added to t if not contradictory to t.
There could be other kinds of evolution strategies but this simple version is exploited to illustrate the applicability of Mone in this paper.
Although the reasoning complexity of the DL language is high, Mone does not rely on a common sense reasoning service. The complexity on Algorithm 3 is O (N V N E ) where

Algorithm 4 Norm Evolution
Input: p: a track Output: r: the refined norm 1 r ← getNorm(getTail(p)) as a(t, e); 2 v 0 ← getHead(p); r ← a(t , e); N V is the number of vertices and N E is the number of edges in the LEG.
As far as we known from the state-of-the-art, there is no publications focusing on this kind of mutation detection and usage so we do not have a comparative experiment with other methods.

VI. VERIFICATION
We have designed a case study to validate the algorithms and simulate a scenario in sweeping robot system. Alice is a sweeping robot, and the interior of Alice is an agent-based system. Alice starts cleaning from 8 : 00AM every day if set in the Auto mode. But one day Alice was stuck by an obstacle, which resulted in the failure of the task. The agentbased system triggered a series of rules to detect problems. Figure 4a gives an example of the horizontal layered execution graph and the initial set of norms in our scenario are shown in Table 1.
There are two execution paths in Figure 4a. One series of execution is (v 1 , v 2 , v 3 , v 4 , v 2 , T ), note that there is a dashed edge between v 2 and v 3 , which means Alice didn't finish its work within limited time. So this series of execution indicates the robot is stuck in state s 2 , the agent system changes to the monitor mode. After a series of detections, the sweeping robot executes r 2 normally and completes the cleaning task in state s 6 . Another series of execution is (v 1 , v 2 , T ) which  indicates the robot completes the work normally. There are four different types of nodes (including N 1 , N 2 , N 3 and N 1 in Table 1) shown in Figure 4a, the result of node v 2 in state s 2 and node v 4 in state s 4 is Exception; the result of other nodes is Satisfaction. The information of execution nodes is shown in Table 2. Preliminary test shows that mutations can be detected from the LEG as the two edges shown in Figure 4a.
• e(v 2 , v 3 ) mutates the norm that the task should complete but failed.
• e(v 4 , v 2 ) mutates the norm that the obstacle should be found and removed, but no obstacle is found. In the execution track mentioned above, the norm r 2 in s 2 is finally executed correctly in another execution path, the sweeping robot successfully completes the task, therefore we can find some information that contribute to the evolution of the destination norm r 2 . The norm r 4 mutated in state s 4 , which means the system has not found the obstacle. As we mentioned in the earlier section, the mutated norm r 4 enriches the trigger of r 2 with no obstacle to be moved. In the part of norm evolution, the condition obstacle = false is added to the trigger of norm r 2 .
Mone enriches the trigger condition of the norm r 2 that mutates. The trigger of norm r 2 changed from Mode = AutoClean to Mode = AutoClean ∧ Obstacle = False. One day, sweeping robot removed the obstacle which was found during cleaning work and therefore the robot will not stop working because of obstacles. The correct execution process of norms is shown in Figure 4b. The information for each norm is shown in Table 3.
The paper presents the example to revise norms based on the Mone framework. The crossover of the mutated norms with the norm on vertex 2 enriches the norm's trigger with no obstacle to be moved. The experiments show that the mutations will be inherited by the well-executed norm and contribute to the evolution of the norm's trigger. The simulation experiment ensure our approach is able to detect all mutations and ensure that norms can be improved by some mutations.

A. LOW-LEVEL NORM
In this paper, a description logic is used to represent the scenario of low level norms without considerations of temporal constraint, permission/obligation/prohibition, event, and many other features of modern BBS based applications. This is because the purpose of the paper is to illustrate the findings of mutations and its usage in the evolution of norms. Norms can be extended with more complicated formalisms to support practical context constraints in practical industrial applications. However, the mutations in complex norms still focus on the unexpected behaviors of the agent, which is contradictory to the expectation.

B. OUTER EVIDENCES
The layer in the LEG is used to incorporate the conditions for norm execution. In practice, there are outer evidences in the state of the layer that may lead to the mutations. But in Theorem 3 we assume no such outer evidences in a closed-world manner. This does not ruin the assumption of an open-world fundamental to the incompleteness of the norm predefinitions. The theorem works in the cases that the outer evidences do not contribute to the mutations.

C. CROSSOVER
The crossover is one of the key operators in the evolution theory. It is not studied thoroughly in this paper, only as a few lines of code in Algorithm 4. This is because the crossover function may be as simple as adding some conditions to the triggers of the norm, and may be as complex as to verify the relationships between the norms and locate the features of the norms to be fixed. The complex version of the crossover operator could be in the scope of the next paper.

VIII. CONCLUSION
In the flexible and open BBS, norms are defined on incomplete knowledge so the runtime executions of the norms may produce unexpected results. Researchers and engineers are always willing to fix such incompleteness with 'new' knowledge. However in the practical industry applications, various of features exist simultaneously and it is challenging to identify which is useful and how to use it in the norm evolution.
From our experiences of the BBS applications, the results of mutations are not always un-welcomed in the real world. With the detailed study of the different behaviors of the norms, it is found that mutations can be used as a source of the 'new' knowledge, to evolve the incomplete norm definitions. The Mone method has been proposed to record the execution tracks of the norms into a LEG, detect the mutations of the norms through the tracks, and crossover the mutated norm(s) with the norm to be fixed. Such evolution steps can enhance the completeness of the norms in a novel perspective.