High-Dimensional Graphical Model Search with the gRapHD R Package

This paper presents the R package gRapHD for eﬃcient selection of high-dimensional undirected graphical models. The package provides tools for selecting trees, forests, and decomposable models minimizing information criteria such as AIC or BIC, and for displaying the independence graphs of the models. It has also some useful tools for analysing graphical structures. It supports the use of discrete, continuous, or both types of variables.


Introduction
We describe here the R (R Development Core Team 2010) package gRapHD which is designed to work with graphical models involving a large number of variables. These may be useful for modelling high dimensional networks in an wide range of biological applications (e.g., ecology, Dunne et al. 2002;gene expression, Faith et al. 2007;proteomics Yosef et al. 2009). Other applications are in computer sciences (e.g., internet, Dorogovtsev and Mendes 2003), engineering (e.g., complex electronic circuits), physics (e.g., Dhamodaran et al. 2008), sociology (e.g., Krause et al. 2007).
The graphical models used here are classes of multivariate distributions whose conditional independence properties are encoded by a graph in the following way. The random variables are represented as vertices (nodes), and two vertices are connected by an edge (line) when the corresponding variables are not conditionally independent given the other variables represented in the graph. Thus the absence of an edge connecting two vertices indicates conditional independence of the two corresponding variables given the other variables. This type of model has been used in high dimensional contingency tables (e.g., log-linear models Goodman 1973;Darroch et al. 1980), continuous variables (e.g., the covariance selection model described by Dempster 1972), and models containing both continuous and discrete variables (Lauritzen and Wermuth 1989). Modern accounts of graphical models can be found in Edwards (2000), Lauritzen (1996), and Whittaker (1990).
The use of graphical models for large numbers of variables can be very challenging, both because of computational limitations, and because of intrinsic statistical difficulties (particularly when the sample sizes are small). Consequently the use of such models has often been restricted to small problems. The package gRapHD presented here is intended for high-dimensional graphical modelling. The central functions, minForest and stepw, search respectively for the optimal forest and the optimal decomposable model, where optimality is typically defined in terms of an information criterion (AIC or BIC).
The gRapHD package is presented using three distinct examples, which are described in Section 2. The basic definitions and notations used throughout the paper are found in Section 3, and the structure of the gRapHD object class defined by the package is described in Section 4. Sections 5 to 7 present the functions in the package, using the examples previously introduced.

Three examples
The features of the gRapHD package will be presented using three different examples covering continuous, discrete, or both types of variables (examples 1, 2, and 3, respectively). We here describe the data and show the graphical model selected.

Example 1 -continuous: Periodontitis data
The data arise from a functional genomics study in gingival tissue (Demmer et al. 2008). The study investigates the differences in the gene expression profiles of interproximal papillae tissues of diseased (periodontitis) and health sites. Ninety healthy non-smoker patients with moderate to advanced periodontitis were used in this study. Each patient contributed with at least two diseased samples and a healthy papilla, if available. A total of 247 samples were collected (64 from healthy sites and 183 from diseased sites). The transcription profiles of the samples were evaluated using Affymetrix human genome arrays with 54,675 probe sets.
Only 64 arrays from independent diseased sites were considered. The data were pre-processed using justRMA from the affy package (Gautier et al. 2004). To reduce the amount of data, probes with variance < 0.62 were omitted from the analysis. Thus our dataset is composed of 1,545 probes, from 64 different patients. The objective is to characterize the gene coexpression network in patients with periodontitis. The model selected by the stepw function can be seen in Figure 1(A).

Example 2 -discrete: HapMap data
The goal of the International HapMap Project is to characterize human genetic variation (The International HapMap Consortium 2003). The project recorded differences in the sequence of bases that composes the DNA, the SNPs (single nucleotide polymorphisms). There are four different bases (A, T, G, and C) that can occur at each position in the DNA (locus), and if different individuals have different bases at a locus, the locus can be considered polymorphic (provided that none of the alleles are too rare). Each different base occurring in one locus is considered as an allele. We use here only the polymorphic SNPs from the Yoruba population (Ibadan, Nigeria -West Africa) with complete data. From this population, we selected the chromosome 17, representing 606 SNPs in 176 individuals, for synonymous coding SNPs (SNPs which even with different structure code the same amino acid) with minor allele frequency greater than 0.25 (download on 2009-06-03 from the HapMart website http://www.hapmap.org/). After eliminating 9 individuals with very high missing values percentage we obtained 334 loci without missing values. Using the information of the stated reference allele we codified the genotypes of the individuals as: "0" for homozygous wild type (individual with the reference allele in both DNA strands); "1" for heterozygous (individual with only one copy of the reference allele); and "2" for homozygous mutant (individual with no copy of the reference allele). The objective is to determine the relationship between different loci through the network structure. The optimal decomposable representation of the network (minimum BIC) is displayed in Figure 1(B).

Example 3 -mixed: Iris data
The iris flower data were introduced by Anderson in 1935, but is also known as the Fisher's iris data (1936). A sample of 150 plants, 50 from each of three species (Iris setosa, Iris versicolor, and Iris virginica), had the sepal and petal lengths and widths recorded. The objective is to describe the structure of correlation between the different measurements considering the three different species. The final model can be seen in Figure 1(C).

Basic definitions and notation
We here give a brief sketch of the theory of graphical models. For a more complete account see for example Lauritzen (1996). Graphical models combine graph theory and probability theory. Each vertex represents a random variable, and two vertices are connected when they are not conditionally independent given the remaining variables. For example, from the graph presented in Figure 2(A), we see that the variables 1 and 2 are not conditionally independent given variables 3, 4 and 5, but that variables 1 and 4 are conditionally independent given the variables 2, 3 and 5. A graph is complete if every pair of vertices is connected by an edge. If a subgraph is maximally complete, it is called a clique: in this case the addition of any other vertex would renders the subgraph incomplete. In Figure 2 In a graph G, two vertices, u and v, are said to be connected if there is a sequence u Figure 2(A), the vertices 1 and 5 are connected since there exists a path between them, for example 1, 2, 3, 5. A cycle is a path which the end vertices are the same (u = v), as 1, 2, 3, 1 in Figure 2 and v j are only connected by an edge when |i − j| = 1. A graph is called triangulated if it has no chordless cycles of length greater than three. For example, the graph in Figure 2(A) is triangulated, but the graph in Figure 2(B) is not, since the cycle 1, 2, 5, 4, 1 of length four is chordless.
This definition implies that a graph is decomposable if it is complete, or exists a decomposition (A, B, C), with A = ∅ and B = ∅, into decomposable subgraphs G A∪C and G B∪C . A graph G=(V,E) is decomposable if and only if it is triangulated (Lauritzen 1996). The example shown in Figure 2 The cliques C 1 , ..., C k in a triangulated graph can be numbered in such a way that for all j = 1, ..., k, H j = C i ∪ ... ∪ C j , R j = C j \H j−1 , and S j = H j−1 ∩ C j gives that (1) for all i > 1 there is a j < i such that S i ⊆ C j , and (2) the sets S i are complete for all i (Lauritzen 1996). This sequence of cliques is called a perfect sequence. The sets H j , R j , and S j are named histories, residuals, and separators, respectively. A perfect sequence of the graph in Figure 2 A key property of decomposable graphs is that the probability densities of such models can be factorized as: where C is the class of cliques in a perfect sequence, and ν(S) is the number of times that S occurs as a separator in this perfect sequence, possibly including the empty set, (Lauritzen 1996). Furthermore, these models have an explicit formula for the maximum likelihood estimators.
A forest is a graph containing no cycles. It may be composed of several connected components called trees, i.e. a tree is a connected acyclic graph (Bondy and Murty 2008). Given a set of edge weights, a minimum spanning forest G m = (V m , E m ) of a graph G=(V,E) is a forest with V m = V and E m ⊂ E that has minimum sum of edge weights, among all possible such forests. Examples of edge weights are the contribution of each edge to the BIC or the contribution to minus the log-likelihood. Figure 3(A) shows a graph with four connected components, and Figure 3(B) shows a spanning forest of this graph.
The variables in the model can be discrete, continuous, or both (mixed). In the first case, in which each variable assumes a value in a set of levels, the models are based on the multinomial distribution. In the second case, the models are based on the multivariate Gaussian distribution. In the mixed case the CG (conditional Gaussian) distribution is assumed; the variances can be homogeneous or heterogeneous across different levels of the discrete variables. Furthermore, a mixed model is strongly decomposable when its graph is triangulated and no forbidden paths occur. A forbidden path is a path between two non-adjacent discrete vertices passing through only continuous vertices, as showed in Figure 4. For more details see Lauritzen (1996, p. 7-12).
Figure 4: Forbidden path: Continuous variables are represented as circles, and discrete variables as dots. The edge (5,10) creates a forbidden path in the graph, since a continuous variable (variable 5) connects non-adjacent discrete variables (variables 3 and 4 to variables 8, 9, 10, and 11).
In this paper we describe the main features of the new R package gRapHD, and how it can be used for graphical model search. Most of the core functions are programmed in ANSI C, due to its computational efficiency. The package as well as the code are available from http://CRAN.R-project.org/package=gRapHD.

The gRapHD object
The gRapHD package groups a number of functions designed for efficient selection of highdimensional undirected graphical models. The set of variables can contain only discrete, continuous, or both types. All the information about the model is stored in a gRapHD object which contains the following elements: edges: Matrix with 2 columns, each row representing one edge, and each column one of the vertices in the edge.

Search for graphical models
The core functions in the package are minForest and stepw. The minForest function finds a minimal AIC or BIC forest, or the maximum likelihood tree for the data. The stepw function performs forward search for the triangulated graph that minimises a given measure. The minimized measure used can be either -LR (likelihood ratio), BIC (Bayesian Information Criterion), AIC (Akaike Information Criterion), or a function specified by the user.
All calculations use the data set specified by the dataset parameter, which holds the raw data, in dataframe format, with the column storage mode defining its type (discrete or continuous). The discrete variables are defined as factors, and the continuous as numerical. In the first case, there should be more than one level (otherwise there is no gain in using such variable), and all levels must be represented in the sample. If all variables are continuous, the dataset can be given as a numeric matrix. Missing values (NA) are not allowed. The variables are referred by their indexes in the vertNames attribute. In this way, the edge representation in the gRapHD object is in the format (v 1 , v 2 ), where v 1 and v 2 are the indexes in vertNames, and v 1 < v 2 . For example, the iris data is presented as The discrete variable (Species) is identified as a factor with three levels. All other variables are continuous (as indicated by the numerical type 'num'). The columns names (Sepal.Length, Sepal.Width, Petal.Length, Petal.Width, and Species) are stored in the vertNames attribute of the gRapHD object. The variables are always referred to as the original column number in the dataset, e.g., the variable Sepal.Width is variable 2, while Species is variable 5. The edge connecting Sepal.Width and Species is represented by a row in the edges attribute consisting of the vector c(2, 5).

Search for minimum spanning forests
The function minForest searches for an optimal tree or forest using the algorithm of Chow and Liu (1968). If the ML measure is used, the function returns a tree, but if the AIC or BIC is used, the function may return a forest or a tree. Per default the BIC measure is used. Starting from an empty edge set, the algorithm repeatedly adds the edge that optimizes the selected measure. Only edges that preserve the tree/forest structure can be added, i.e., no cycles can be generated. The procedure continues until no more edges can be added. For example, if the selected measure is the BIC, at first the pairwise values are computed and in each step the edge that reduces the most the total BIC is added, if it exists. The algorithm is Step 1: Calculate the BIC for all possible edges.
Step 2: Select the edge that improves the most the model's BIC.
Step 3: If there is no such edge, stop.
Step 4: Test if the addition of this edge creates a cycle or a forbidden path.
Step 5: If it does, select the next edge with best improvement and return to Step 3.
Step 6: Add the edge to E, remove it from the list of possible edges, and return to Step 2.
For mixed models, the algorithm finds the strongly decomposable forest that minimizes the selected measure (Edwards et al. 2010).
For the periodontitis data, the summary of the minimum spanning forest is presented below. The graph found is actually a tree and not a forest, since all vertices are connected (p − 1 edges). All variables are continuous, and the measure minimised was the BIC (default).

Forward search
The function stepw searches for decomposable models minimising a given measure by adding edges to a initial model, also decomposable. The algorithm is also iterative, determining at each step the add-eligible edges, i.e., the edges that if added preserve the triangularity. Among these edges, the one that reduces the most the selected measure is added to the graph. The algorithm, showed below, stops when no more add-eligible edges are found. The structure of components in the starting model is preserved as default, which means that if the function starts from a forest with k isolated components, the final model will also have k isolated components. This can be changed setting the option join to TRUE.
Step 1: Calculate the BIC for all add-eligible edges.
Step 2: Select the edge that improves the most the model's BIC.
Step 3: If there is no such edge, stop.
Step 4: Add the edge to E, and return to Step 1.
For example, if the function is applied on the tree found in the previous section, 999 iterations are necessary (998 edges are added to the tree). The default measure is the BIC. We can see that in the final model, the first 1,544 edges were found by minForest (edges 1 to 1,544), and the last 998 by stepw (edges 1,545 to 2,542). The final graph can be seem in Figure 1(A).
The algorithm can start from an empty model (which is always decomposable). If we use the iris data, the final model ( Figure 1C) found is the same as if it had started from the forest/tree graph. The code for it is showed below. Initially a gRapHD object is generated, from an empty model with 5 variables, being the first 4 continuous and the last discrete with 3 levels. The model is considered heterogeneous. As we are starting from 5 isolated components, they are allowed to be joined.

Computational performance
The performance of gRapHD was evaluated using a Intel(R) Xeon(R) CPU E5450 3.00GHz with 31Gb of RAM, running Linux 64 bits. The memory use and CPU time of the three examples are presented in Table 1.
We see that the minForest function is highly efficient in terms of CPU time but may require substantial memory.
Additionally, we performed a simulation study where the functions minForest and stepw were evaluated. Data sets from multivariate normal distributions were generated with 10 to 5,000 variables (at intervals of 20). Each of these data sets were used to infer the minimum spanning forest and a more complex decomposable graph starting from this forest. The computational performance is showed in Figure 5. As the stepw function depends on the final complexity of the model (number of edges added to the model), the CPU time shown is for one (average) iteration only. The memory demand for the minForest function grows quadratically in the number of vertices, while the memory demand for the stepw function grows linearly in the number of vertices. The CPU time also grows quadratically in both functions.
In conclusion, the minForest function runs much faster than the stepw, allowing the selection of a skeleton of the model in a forest-like structure (sparse) in a very short time. We remark  that it appears to be much more efficient to start the stepw function from the forest found by minForest, rather than from the null model.

Analysis of the graphical structure
When high-dimensional models are studied, plotting the model found is probably not the best way to visualise the result, as can be seen in Figure 1. The plotting algorithm used here is time consuming, and may need a large number of iterations to generate a graph with no edges crossing each other. Therefore different ways to analyse the resulting model have to be used. The package contains a number of functions which could be useful for this task: adjMat: Returns the adjacency matrix based on a list of edges.
Degree: Returns the degree of a set of vertices.
DFS: Finds all vertices reachable from one specific vertex (assuming that there are no cycles).
findEd: Finds all add-eligible edges to a given triangulated graph, preserving triangularity. In the case of a mixed graph, forbidden edges are not returned.
neighbours: Returns all vertices with a direct connection with a vertex v.
MCS: Returns a perfect numbering of the edges.
modelDim: Calculates the number of free parameters corresponding to each edge.
neighbourhood: Finds the set of vertices with up to a given distance from a given vertex.
perfSets: Finds a perfect sequence, returning the cliques, histories, residuals, and separators of a given triangulated graph.
shortPath: Calculates the shortest path between a vertex v and all other vertices.
SubGraph: Based on a list of vertices, generates a subgraph.
summary: Gives details about the model's structure.
jTree: Finds a junction tree of a graph.
CI.test: Calculates the deviance and adjusted degrees of freedom for the conditional independence test.
The degree of the periodontitis model shows two vertices with high number of direct neighbours (degree 20). These vertices could be "hubs", with important action in the network. We can also see that there is a high number of "leafs" in the graph, as 382 (24.72%) vertices present only one edge. We can zoom in the neighbourhood (up to the second neighbour, for example) of these two "hubs".
R> The perfSets function finds the cliques structure of the graph, also returning the lists of separators, histories, and residuals, as described in Lauritzen (1996, p. 14-15 The shortPath function returns the shortest path length between vertices in the graph (considering that each edge has length one). If two vertices are not connected, i.e., there is no path between them, it is returned Inf. In the periodontitis data, the vertex 670 has a direct connection with 20 other vertices, and the most far vertex has a distance of 26 from it, while the graph's diameter (longest shortest path) is 46. Note that vertex 670 has a distance of 0 to itself.

Plotting a graphical model
Some examples of plots generated by the plot.gRapHD function have been given above. The function uses S3 method for the class gRapHD, so the regular plot function can be used. As default, when a gRapHD object is plotted, all discrete variables are pictured as black circles, and the continuous variables as grey circles. But the function is flexible, and the user can define different colours, shapes, and sizes for each vertex. The Fruchterman and Reingold (1991) algorithm is used to place the vertices in the plotting area (Csardi and Nepusz 2006). The algorithm is iterative and uses attractive and repulsive forces for placing the vertices. This technique is time consuming and cannot guarantee a clear plot, as the one in Figure 1(A), which used 4,000 iterations, a number not sufficient to untangle it.
The more complex a graph is, the more difficult is its visualisation. For this reason, the plot.gRapHD function has a number of optional parameters that allow the user to manipulate the appearance of the plot. It is possible to plot only the edges, or not to label the vertices; to highlight some vertices with different colours, shapes, and/or sizes. For example we could show in the graph where the neighbourhood of the vertices 670 and 1,123 are, for the periodontitis data, as shown in Figure 7.

Concluding remarks
We have described an R package for efficient selection of high-dimensional undirected graphical models, with functions not available in other packages. The objective of the package is to provide an efficient way to search for models in the classes of forests and decomposable graphs (discrete, continuous, and mixed). The computational performance depends on the model complexity, where sparse models demand much less resources. It is easy to convert gRapHD objects into graph or model objects supported by other packages; it is also easy to extract information directly from them. There is a technical limitation of 65,000 variables, due to numerical representation.