APPLICATION OF MODIFIED SOM NEURAL NETWORKS ON ACYCLIC DATA STRUCTURES

Graphs as data structures are used in many applications, for example image analysis, scene description, natural language processing. The paper deals with Acyclic Graph Data Structures (AGDS) and with a learning process in a model of a Self-Organizing Map (SOM) that has been modified for processing of AGDS. To the modified SOM Neural Network (SOM NN), there are added contexts and counters which are built in a training phase of the neural network. The trained SOM NN in active phase can compute information which is used to built some acyclic substructure. The prepared application was tested on the real data of study programs, the test results are evaluated using a winner differentiation and a confidence criterion.


INTRODUCTION
Graphs are very important models of data structures, they give possibilities to represent facts in vertices and relations among facts in edges.In some applications, the structured data are often connected to a learning process from examples.The graph G = (V, E), where V is the finite set of vertices (facts) |V | = n and E is the set of edges (relations) |E| ≤ n 2 should be represented by a matrix of type n × n.Problems can be seen here are the following: for each example (presented by some graph) used in the training procedure we can have a different type of the matrix; if the biggest type of the matrix is used for all graphs then there are too many free positions in the matrix, which do not represent anything (any relation).It means, it is important to find some other representations of graphs (data structures) and it is necessary to do it in a connection to a model of computation.
SOM NN have been proposed by many authors [2], [3], [4], [8], [12], as models which are very good for a learning of graph data structures.It means, the graph structures in connection to neural networks should be represented in some special representation (a trained neural network) and the neural networks can be trained to graph structures.In the training, it is possible to prepare input data of the graph in some parts, it is not necessary to put a full graph as one input data to the network.The shape of a output substructure is very important in many solved problems.In [1], we have prepared theoretical points of view to a presented application of using SOM neural networks to input and output acyclic data structures.
The paper is prepared in the following sections.In the first section we describe the motivation to the study of the problem, the second section is oriented to a description of acyclic data structures and to a description of formal problems solved in the paper, in the third section we describe the model of prepared neural network -Counter Propagation SOM for Structured Data (CP SOM SD).The next section contains a description of a data encoding and a training algorithm of modified SOM NN.In the last section, there are defined criteria of an evaluation of CP SOM SD and evaluated the prepared solution in some examples.In the conclusion, we summarize the work which was done and describe some plan for the following work.

MOTIVATION -PREREQUISITES IN STUDY PROGRAMS
Many study programs at universities are built in a system of three subject types: compulsory subjects, compulsory elective and elective subjects.From the theoretical point of view, all subjects built a partially ordered set according to the order of subject passings.

Fig. 1 Mapping of prerequisites for study program in informatics
The partial ordering is defined by prerequisites of subjects.The part of study program in informatics at Faculty of Science of P. J. Šafárik university in Košice is mapped in the Fig. 1.
At the beginning of academic year, the students choose new subjects which they will study in the current academic year.They register the chosen subjects in an Academic information system.For example, a registration of the subject Software engineering can be taken after passing subjects Object-oriented programming and Database systems.
The main problem: The system has information about the structure of all subjects in the study program (some acyclic graph structure).The information about passed subjects of the student can be put to the system in two ways: • The information is in the list of passed subjects in Academic information system.It is possible to obtain the acyclic structure of all student's passed subjects in automatic way.
• The way is not automatic.The student will give only some passed subjects.We will follow this way.
When the student gives some passed subjects and he chooses some subject, which he try to study for the next academic year, the system should answer him automatically NO (you can not to register the subject, you have not fulfilled prerequisites) in real case or it can answer Y ES (it is O.K.).Or it is possible to ask the system for some recommended subjects which the student can study in the following academic year.

ACYCLIC GRAPH DATA STRUCTURES AND TWO SOLVED PROBLEMS
A directed graph or digraph is a pair G = (V, E), where V is a set, whose elements are called vertices or nodes, E is a set of ordered pairs of vertices, called directed edges.
A Directed Acyclic Graph (DAG), is a directed graph with no directed cycles.It is construct by a collection of vertices and directed edges, each edge is connecting from one vertex to another, such that there is no way to start at some vertex v and follow a sequence of edges that eventually loops back to v again.In the DAG graph, there exists one vertex at least without input edges.
Let G = (V, E) be a DAG graph, V s be a set of all vertices without input edges.We call the elements in V s as sources.Some vertices in the DAG graph have some special marks.Let V m be a set of all marked vertices.

The first problem -a reachable vertex
Let v z ∈ V be a chosen vertex.Let G p = (V p , E p ) be a graph of all predecessors of the vertex v z of all levels (predecessors of predecessors, ...).(We construct all paths from sources to v z .)Is it V p ⊆ V s ?

The second problem -a set of successors
Let G t = (V t , E t ) is a graph of successors of all vertices in V s of all levels which belong to V m too.(We construct all longest paths from sources using vertices from V m , all vertices in the longest paths belong to V t .)V t ⊆ V m .Let V l t be the set of the last vertices in the longest paths from sources.The problem is to find a set V f of vertices -successors in the first level of vertices in V l t ?

Examples
We used the DAG to define the study program.The subjects are in the vertices and the oriented edges represent prerequisites.If the subject A is the prerequisite of the subject B then in DAG is the oriented edge from A to B. The subject A is called right prerequisite or the first level prerequisite to B. If C is the right prerequisite to A, the C is the second level prerequisite to B.
If we choose some vertices from DAG together with the oriented edges among them then we get a set of graphs (one or more subgraphs) and the previous information in the DAG is reduced.The reduced information is sometimes enough to compute results, but in more cases it is not enough.
Some study program is constructed as a DAG graph.The marked vertices in V m represent passed subjects of some student.The formal problems should be represented in the following way: • The first problem: Can the student register a subject v z ?
• The second problem: Construction of the set of subjects which can be register by student, if he passed subjects in V m .

MODEL OF CP SOM SD
We describe a model of a neural network which has been used for the computation of the answers to given queries about passing subjects.Our model was motivated by the model of the neural network working with tree structured data [6], [10], [9].

SOM for Tree Structure
The SOM for Structured Data (SOM SD) has been described for processing of labeled trees with fixed fan-out k [5], [11].We describe very shortly the model which has been used for the classification of artificially constructed pictures represented by tree structured data according to [6], [3].
It is assumed that the neurons are arranged in rectangular d-dimensional lattice structure.Each neuron can be enumerated by tuples be the number neurons in the lattice, n is a dimension of an input.Each neuron n i is equipped with a weight w i and k context vectors c i 1 , . . ., c i k in R d .They represent the contexts of proceed tree given by the k subtrees and the indexes of their respective winners.The winning neuron with index I(t) to given a tree t with root label a and subtrees t 1 , . . .,t k is recursively defined by α, β > 0 are parameters to control mediation between the amount of pattern match versus context match.The empty tree ε is represented by an index not contained in the lattice, (−1, . . ., −1).The weights of the neurons consist of compact representations of trees with prototypical labels and prototypical winner indices which represent the subtrees.
Starting at the leaves, the winner is recursively computed for the entire tree.The index i of the winner for the subtrees is computed.The attached weights (w i , c i 1 , . . ., c i k ) are moved into the direction (a, I(t 1 ), . . ., I(t k )) after each recursive processing step, where I(t i ) denotes the winning index of the subtree t i of the currently processed part of the tree.The weights of neighbor neurons are updated in the same direction with smaller learning rate.
In [4], it is described SOM SD in a supervised approach.The training set is defined as T = {(D i , d i ), i = 1, 2, . . ., m} where D i is the data structure with desired value d i , the input vector for each node of this data structure is extended by adding a new field containing information about target.The model is called sSOM SD and it was applied on a clas-sification task with 12 classes.

Model CP SOM for Acyclic Data Structures
The developed model is drawn in the picture Fig. 2. The described model can be divided into three part: A. The lattice of neurons, contexts and counters -CP SOM SD.The network (in the middle part of Fig. 2) is trained for the acyclic structure of the study program.In the process, there are built contexts for prerequisites of subjects and counters for all prerequisites of a given subject.
The contexts are r 1 x , r 1 y , r 2 x , r 2 y , . . ., r k x , r k y .The weights c 1 , c 2 , . . ., c k are the weights to contexts.The counters are u 1 , u 2 , . . .u K and each counter contains all prerequisites of one subject.C. Output part is built from the vectors a = (a 1 , a 2 , . . ., a K ), q = (q 1 , q 2 , . . ., q K ) and d = (d 1 , d 2 , . . ., d K ).The (0, 1) -vector a represents the passed subjects given on input (given by student), the (0, 1) -vector d explains if the subject has prerequisites or not.From the vectors a, q and d is computed result if the student can register the queried subject.The vector q is computed using counters and one bit information in the input.

Encoding of training structures and training sets
Let K = |V | be the number of subjects in the acyclic data structure of the study program.The subjects are encoded in the binary alphabet {0, 1}, it means each subject is the numbers of bits used to the encoding of each subject.
The preparation of training structures -to each subject is prepared a substructure with all prerequisites on the first and second level.The prepared substructures have some the same vertices from the structure.
Let T c be training set to the set of ADS S.
where p is the number of training patterns (the number of patterns is the same as the number of all vertices in all training data structures) and d l is an expected value to a data structure D l .

Training of the neural network
The distance of input x from the i-th neuron in the lattice and in connection to the contexts was computed according the formula (2).In the formula, there is the second part where the distances from contexts (r j , c (j) i ) are evaluated and a sum of them have an influence to the distance value.
where α, β > 0 are some parameters.Let the winner is i * , it means winner is one of the closest neurons to the input x.It is computed by 3 The adaptation of weights in the lattice is done according the formula ( 4) where 0 < η(t) < 1 is a learning rate, h i * ( j,t) = e − dist(i * , j) 2 2σ 2 (t) , and dist(i * , j) is a distance of j-th neuron to the winner neuron i * .The output layer contains K Grossberg's units with weights u i , i = 1, . . ., K. The counter propagation was applied for training u i .
where z is a state vector of neurons in the second layer of a counter propagation network and d r is an expected value.
The basic information on training neural networks can be found in [7].
The weigths between neurons in the lattice and in the output layer are modified by (6). where r is evaluation of the vertex r. • Output: Configuration of CP SOM SD network.
1. Initialize weights w i to random values, for each i-th neuron in the lattice.
2. Initialize weights u r to random values, for each r-th neuron in the output layer.
3. Repeat until the network is not in a stable stateu for each training pattern (D l , d l ) from T c .
(a) For each vertex v ∈ V inv (D) i. Find winner neuron i * according to (3).ii.Modify weights according the formula (4).(b) Compute output according to the counter propagation algorithm.(c) Modify weights between neurons in the lattice and in the output layer according to (6).
The space complexity of the network is N * log 2 K + 2 * N * K + 6 * K, where K is the number of vertices in an acyclic data structure and N is the number of neurons in the lattice.The time complexity of the training is l s * N * log 2 K, where l s is the number of training cycles.

RESULTS AND EVALUATION
From theoretical point of view, we are interested in quality of learning process of neural networks.The size of a lattice in SOM neural network is very important parameter.The differentiation of winners refers about spreading of winners in the lattice and it is dependent on its size.Confidence of right answers refers how good the network answers.

Criteria of evaluation
• Winner differentiation WD) [12] describes the ratio between the number of winner neurons and the number of different inputs.
If W D < 1, some winners are computed for different inputs.The situation is the best one if W D = 1.
• Confidence of right answers.Let p S be the sum of all the numbers of predecessors of the vertex v D together with the subject in the training set and p N be the sum of all the numbers of predecessors of the vertex v D together with the subject computed by a neural network, then the confidence of right answer is

Analysis of queries
The student can put in the input a set of his passed subjects (SPS = V m ) and then he put the subject Q = v z which he would like to register for study.We analyzed the behavior of the network for the different structures of SPS.The set SPS is the set from the student point of view but in the acyclic structure of study program the subjects have their positions.The student do not put full information, it means, it is interesting to compare real information following from the study program to the information computed by network.We describe some situations: • |SPS| = 0, |SRS| = 0.If the student put on input any passed subjects and he do not know any subject it means it is beginner and he will get an answer the set with the subjects without prerequisites.The situation is solved in the following query 2.
• |SPS| = 0, SRS ≥ 1.If the student put on input any passed subject SPS = / 0 it means he would like to know if the subjects in the set SRS are subjects without prerequisites (in the case he should register them).The situation is solved in the following query 1.
• |SPS| ≥ 1, |SRS| = 0.If the student put on input passed subjects only he would like to know the subjects he could register.The situation is solved in the query 2.
• |SPS| ≥ 1, |SRS| ≥ 1.It is a common situation.The student passed the set SPS of subjects and he would like to register the subjects from the set SRS (chosen set of subjects).Can he register the subject from the set SRS.The situation is solved in the query 1.
From the previous analysis we formulated to the following two queries: • Query 1: I have passed the set of subjects SPS.Can I register the subject Q?
• Query 2: I have passed the subject SPS.Could I register the subject from SRS?

Encoding of queries
The query (input to the neural network) is prepared in the following way: • a is the number of all subjects in the query; • the subjects in the query are prepared as a pairs (p l ,t l ), where p l is the code of subject and t l is one bit (for a passed subject it contains the value 1, for not passed the value 0).

Results
We used 19 study programs from Faculty of science P. J. Šafárik university in Košice.In the test of the prepared application for CP SOM SD we did many tests for the study of a network behavior some in comparison to real values.
We construct 30 queries to each study program.Each query was built in the following way: • The set of passed subjects SPS = V m was generated randomly from subjects in the study program V .The number of passed subjects in SPS was generated randomly in the interval 1, 2 3 |V | .• Let V comp = V −V m be the set of non passed subjects.
The set SRS represent subjects prepared to registration, SRS ⊆ V comp was built randomly.The number of subjects in SRS was computed according the rule: if The used real values in the training: the dimension of lattices from N = 15 × 15 to 54 × 54: parameters α and β in interval 0.6 − 0.9; the number of iterations 1000; the starting initial weights were chosen randomly from the interval −10; 10 ; the learning rate 0.1.
Results of both quality parameters are in the Table 1 and in the Table 2. Results represent values of two defined criteria and our conclusions are the following: • The better results were computed by networks with the bigger size of the lattice.

CONCLUSION
In the paper, we described the new developed model of the CP SOM SD neural network which could be used in the application of an academic information system.The tests of the model for acyclic data structures of study programs at university were evaluated and results give quite good starting point to the following work.The plan to the following activities is to continue in the work with an analysis of different sets SPS they have their special positions in the acyclic data structure and to compare it to real data.

Fig. 2
Fig.2The structure of CP SOM neural network

Algorithm 1 :
Counter propagation training for data structures • Input: Training set T c is defined by ( 1).