Morphological Residues and a General Framework for Image Filtering and Segmentation

Morphological residues represent an image in a hierarchical way by means of a decomposition of its structures and according to a size parameter λ . From this decomposition, we can obtain a relation between the different residual levels associated with the complexity of the image structures. In this work, we introduce a new method to ﬁlter out components of gray-scale images based on the morphological residue decomposition which takes into account a size parameter and a certain level of complexity of the different structures to be ﬁltered. As we will illustrate,this complexity is associated with a set of new attributes of the image deﬁned according to the information contained in its multi-resolution representation.


INTRODUCTION
Since the 1960s, mathematical morphology [1,2,3] has become increasingly popular in the community of digital image processing due to its rigorous mathematical description and capacity to extract information based on shape transformations.
Most of the works developed in this area concern mainly pre-processing and segmentation applications. Preprocessing consists in enhancing the "image syntax" to increase the success of the other operations. One example of this technique is filtering [1,3,4,5,6,7,8]. The aim of segmentation is to partition the image into its constituent parts. Segmentation techniques and some related problems are treated, for example, in [1,2,5,6,9,10,11,12].
The computational cost of the mathematical morphology operations is relatively high. Most of the time, the need to extract the information in a cost-effective way yields the use of a set of images from the same scene containing different levels of representation. In [13] is defined an image multi-resolution decomposition scheme, the pyramid scheme, which illustrates this kind of multiple representation of an image. This scheme encompasses both the morphological concept of granulometry [14,15] and morphological residues [2]. Informally, as we will see in Section 2, the granulometry describes quantitatively the coarseness of an image by characterizing the signal as a collection of grains that can be sieved in a grain size distribution process. It decomposes an image into classes of components, according to a size parameter, whereas the morphological residues constitute a complete hierarchical representation of this image [1,2].
In this work, we consider the problem of filtering and segmentation through the decomposition of an image into its morphological residues. This decomposition can be very useful for the characterization of some image structure attributes, such as size, volume, and shape [4,8]. Here, we define new attributes of an image based on the notion of vanishing level of a point. As we will see later, these attributes are related to the irregularity of the image structures, when considered as a topographic surface, and concern the persistence of the information between different residual levels. Finally, based on the concept of granulometric residues and image attributes, we define a general filtering method taking into account this information.
This paper is organized as follows. In Section 2, we introduce the multi-resolution scheme considered in [13], associated with the concept of granulometry and morphological residues. In Section 3, we define new attributes for image filtering and segmentation based on the notion of residues by attributes. In Section 4, we show how to define sets of markers used in the morphological reconstruction algorithm and considered in the general filtering method described in Section 5. This section also presents some results of the proposed method, by considering synthetic and real images, and discusses some of its basic properties. Finally, conclusions are drawn in Section 6.

MULTI-RESOLUTION DECOMPOSITION
The multi-resolution decomposition scheme represents an image by different levels of resolution or "coarseness" which, in turn, can be associated with the different amount of information we want to analyze.
A general framework for image decomposition, the pyramid scheme, is defined in [13]. The method considers two basic operations: (a) analysis, that simplifies the image representation by reducing the amount of information, and (b) synthesis, which tries to recover the information lost in the analysis step.
The combination of these operations produces an approximation of an original image X due to a partial recuperation of the image by the synthesis step. In this case, it is possible to obtain an image of details Y , containing the information not recovered by the synthesis step, and given by the difference between the original image X and the image defined by the combination of the above operations.
We can easily see that the granulometry and the morphological residues can be represented by this framework, since, as we will see next, the granulometry decreases the amount of information of an image, according to a size parameter, while the morphological residues, defined as the image of details, contain the information lost between two successive granulometric levels.

Granulometry
The granulometry, (ψ λ ) λ≥0 , which depends on a size parameter λ, describes quantitatively the "coarseness" of an image and is a basic morphological concept used, for example, in segmentation, texture analysis, and pattern recognition [16,17,18,19,20]. The granulometry decomposes the image in classes of components according to the used structuring element. It can be defined as follows.
Definition 1 (granulometry [14]). Let (ψ λ ) λ≥0 be a set of image transformations depending on a parameter λ. This set constitutes a granulometry if and only if the following properties hold: Equation (3) implies an idempotent operation, that is, The set of transformations ψ λ considered in this work corresponds to the morphological opening operation [1,3] with convex structuring elements and structuring functions, for binary and gray-scale images, respectively, and their homotetic representations.
The concept of morphological residues, directly associated with the granulometry operation, is defined next.

Morphological residues
The morphological residues, R, [2] characterize the information extracted from an image based on a set of granulometric transformations. These residues are given by the difference between two consecutive granulometric levels, as follows.
Definition 2 (morphological residues [2]). Let (ψ λ ) λ≥0 be a granulometry. The morphological residues R λ , of residual level λ associated with the size parameters λ, are given by the difference between the result of two consecutive granulometric levels, that is, where X and f represent discrete binary and gray-scale images, respectively, and \ stands for the difference between sets.
Equations (5) define the morphological residues for binary and gray-scale images. They represent the components preserved at the granulometric level (λ − 1) which are eliminated at level λ.
According to the transformation ψ, the set of residues corresponding to (R λ ) λ≥1 contains the complete granulometric information and defines a complete hierarchical representation of an image. Thus, for the binary case, and for the gray-scale case, Therefore, in the image analysis process, we can limit the amount of information to be processed by considering only the data defined in a certain resolution, as we will see in the next section.

Basic definitions
More precisely, let Γ I ⊂ Z 2 be the domain of a binary or grayscale image I, where each point ν ∈ Γ I can assume discrete values in the range [0, L], L equals 1 for binary images. Also, let Φ φ (I) define a subset of points based on a transformation φ : Z 2 → Z + of image I. We define the binary residues of an image as follows.
Definition 3 (binary residue). The binary residue Φ Rλ (I), where λ ≥ 1, of an image I, related to the transformation R λ , is represented by subsets of points included in Γ I , such For a binary image X, it is easy to see that which means that the details of the image obtained at level λ are not present at a different level µ. Unlike the binary case, a gray-scale image g yields In this case, we do not necessarily have a successive suppression of points between two different levels of the binary residues, since the points of a gray-scale image can be only "smoothed" by the successive opening functions ψ λ . Figure 1 illustrates these aspects for a one-dimensional case. Here, each residual level λ represents the size parameter associated with the radius of a flat structuring element. The dark parts in Figures 1b, 1d, 1f, and 1h are the residues R λ of the original image f (see Figure 1a), whereas Figures 1c, 1e, 1g, and 1i represent the subsets Φ Rλ (f ). Remark, for example, that points m and n belong to three different subsets Φ R1 (f ), Φ R2 (f ), and Φ R8 (f ) which convey size parameter information of the image structures related to these points.
Based on the subset (Φ Rλ ) λ≥1 of points present at different residual levels, we can associate with each point of an image the following residue mapping notion.
Definition 4 (residue mapping). Let (Φ Rλ ) λ≥1 be a set of binary morphological residues. For all points ν ∈ Γ I of the image I, define a residue mapping, M, representing a onedimensional array conveying information about a size parameter associated with each residual level λ, so that By this mapping, we take into account the parameter information related to the different levels in which the image points vanish (they change their state from 1 to 0), based on the set (Φ Rλ ) λ≥1 defined from a sequence of residual operations (R λ ) λ≥1 . For example, the mapping of point n in Figure 1 is given by 9. (12) Now, we can define the notion of vanishing attributes of a pixel.
Definition 5 (vanishing attributes). The vanishing of a point is represented by a transition of its value from 1 to 0 in the residue mapping M. Informally, the vanishing attributes are related to the different moments a given point does not belong to the morphological residue anymore.
As we have seen before and according to (10), a point in a gray-scale image can change its state from 1 to 0 many times in the residue mapping M. Related to this vanishing information, we can define two new attributes of an image pixel ν: one concerning the order, (ν), in which the vanishing occurs in the residue mapping, and the other associated with the number of occurrence, η(ν), of this vanishing. The first attribute concerns the different moments a point vanishes at the different residual levels, while the second one concerns the number of transitions from 1 to 0 of this point in M. This vanishing number can be related, for example, to the irregularity (complexity) of the image in the analysis of its structures. We consider the dark regions A and B indicated in Figure 2. These structures have the same size parameter, λ = 4, but they present different complexity information that can be characterized by the number of vanishing of the pixels, η(ν), given by the residue mapping, as we will illustrate elsewhere. Table 1 illustrates the mapping M and the number of vanishing η(v) of the plateaus in Figure 1a.
Based on the notion of residue mapping and vanishing attributes, we can finally define residues by attributes.

Residues by attribute
By considering the persistence of the points of a gray-scale image along the residual levels (equation (10)), we can use the parameter λ and/or the information on the vanishing of a point in these residual levels as basic attributes for filtering. Here, the parameter λ is closely related to the size of the image components (small (big) structures are represented by low (high) residual levels), and the vanishing information gives an idea about the complexity or regularity of their shape (very regular components tend to have small η values in M). Definition 6 (residues by attribute-a general definition). Let (R λ ) λ≥1 be a set of morphological residues, Γ I ⊆ Γ I a specific subset of the original image pixels, and (M) ∀ν∈ΓI the residue mapping of all points in the domain Γ I . The residues by attribute, Ω, represent the information concerning the residual level λ, R λ , relative to a size parameter λ, and/or to the vanishing of the image points in M.
The characterization of significant structures of an image is directly related to the right choice of the considered attributes. Based on the above notion, we can define the following set of residues by attribute.
Definition 7 (residues by attribute of size). Let (R λ ) λ≥1 be a set of morphological residues. The residues by attribute of size concern directly the information about the size parameter λ of the image structures.
This well-known attribute takes into account only the size of the image components and is very useful in image segmentation [2,4,7]. The hierarchical representation of an image based on this attribute is such that their small structures are present at the first residual levels, while the big ones are found at the last residual levels. Figure 1 shows points of an image associated with residues by attribute of size λ = 1, 2, 4, 8 denoted by R 1 , R 2 , R 4 , and R 8 , respectively.      Table 1).
Definition 9 (residues by attribute of size and vanishing). Let (R λ ) λ≥1 be a set of morphological residues and (M) ∀ν∈ΓI the residue mapping of all points in the domain Γ I . The residues by attribute of size and vanishing represent the information at the residual level λ, R λ , relative to the size parameter λ and to the vanishing of the points in M. Figure 4 illustrates the case in which the subset Γ I of regional maxima has vanishing number equal to 1 and residual level λ = 4.
In Figure 5, we consider the order of vanishing and the size parameter as information for filtering. In such a case and according to Table 1, we show components of regional maxima with order of vanishing equal to 2 at residual level 5, that is, the second transition from 1 to 0 in M occurring at level λ = 5.
These attributes can be associated to a filtering/segmentation process which encompasses the following three steps:  From the above operations, different images can be obtained and combined yielding results which highlight the components of interest of a scene.

a b c d e f g h i j k l m n o p q r s t u v w x y z
To retrieve the structures characterized by the corresponding attributes, we need to define a set of markers identifying these structures in the image (as in the morphological segmentation paradigm [10,12]). The following section concerns this aspect.

MARKERS DEFINITION
The information to be extracted by the filtering process depends on the markers used in the morphological reconstruction process [7,12], since these markers define the image structures which should be or not preserved. The markers are defined according to the type of filtering which, in turn, depends on the related attributes. Thus, to obtain the desired filtering, we can define two sets of markers, named Preserve and Eliminate, with the same size as the original image. These sets constitute the marker functions for the image structures that should be preserved and eliminated, respectively.
For each type of attribute, we can define the following sets of markers.

Markers definition associated with the size parameter λ and independent of the residue mapping M
To obtain the structures of an image f , associated with a size parameter λ, the sets of markers are given simply by where again \ stands for the difference between sets. Based on Table 1, the set of points {b, c, d, e, f , g, h, i, q, r , s, t, u, v, w, x} (Figure 6b), defining the marker function Preserve, is related to the components in Figure 1a with size parameter λ = 4 (the darker regions in Figure 6a). The set {a, j, k, l, m, n, o} (Figure 6c), associated with the marker Eliminate, constitutes the points not included in Φ R4 which have binary residues at all levels µ greater than 4. As we will see in Section 5, to achieve the correct retrieval of the image components, the combination of the images reconstructed by these markers will be considered in our final filtering/segmentation algorithm.

Markers definition associated with the residue mapping M
As we have seen before, from the residue mapping M, we can obtain information about the number and the order of occurrence of the vanishing of a point. If η(ν) denotes the number of vanishing of a point ν and η is a given number of vanishing, then we can have the following set of marker functions concerning this vanishing information: otherwise, For example, according to Table 1, the points k and m (Figure 7) can be used to retrieve regional maxima of the image in Figure 1a with η = 2.  Now, let (ν) be the order of vanishing of a point ν, as before, and be a given order of vanishing. An example of marker definition related to this order value is given by the functions We remark that the order occurs for a point ν only a b c d e f g h i j k l m n o p q r s t u v w x y z Figure 8: Set of points characterizing the regional maxima with = 2 in M. The sets {d, f , g, h, k, m} and {q, r , s, t, u, v, w, x, z} concern the functions Preserve and Eliminate, respectively. when its number of vanishing η(ν) is greater than or equal to the value . Figure 8 shows the points of the markers Preserve and Eliminate above which characterize the regional maxima in Figure 1a with = 2.

Markers definition depending both on the size
parameter λ and on the residue mapping M We define θ(M(ν), η ) as the level λ in which the number of vanishing of a point ν is equal to η . A set of marker functions based on the size parameter λ and on the value η can be given by In Figure 9, we show the points of the marker Preserve related to the regional maxima in Figure 1a with η = 1 and Finally, if we define θ(M(ν), ) as the level λ in which the vanishing of a point occurs in a certain order , then the set of marker functions concerning this order and the size parameter λ is given by Figure 10: Set of points corresponding to regional maxima with = 2 occurring at level λ = 5. The sets {d, f , g, h} and  {q, r , s, t, u, v, w, x, z} concern the markers Preserve and Eliminate, respectively. Figure 10 shows the set of points characterizing the regional maxima in Figure 1a which have = 2 associated with size parameter λ = 5.

a b c d e f g h i j k l m n o p q r s t u v w x y z
After the definition of the suitable set of marker functions, we can execute the final filtering step represented by the morphological reconstruction algorithm, as discussed in the next section.

THE RECONSTRUCTION STEP AND THE FILTERING/SEGMENTATION METHOD
Given the above set of markers, we consider the morphological reconstruction algorithm [12] to finally retrieve the significant structures of the image. The reconstruction of a binary image X using a binary image Y as a marker, ρ Y (X), is given by the union of the k connected components of X having at least one point of Y : To better understand the morphological reconstruction of a gray-scale image f , we can consider this function as a topographic surface represented by a pile of sections given by binary images X i (f ) defined at level i as We can easily see that Now, we define two images g and f such that g ≤ f , that is, for all pixel ν, g(ν) ≤ f (ν) (Figure 11a). In this case, it is obvious that the sections of image g, X i (g), are included in the sections of image f , X i (f ). The binary reconstruction of X i (f ) using X i (g) as a marker, ρ Xi(f ) (X i (g)), for every level i, defines a pile of embedded binary images representing the morphological reconstruction of f by g, ρ f (g) (Figure 11b).
Efficient algorithms implementing the binary and the gray-scale reconstruction operations can be found, for example, in [12,21].
From the above notions, the general filtering algorithm based on residues by attribute can now be summarized as follows.  Figure 11: Gray-scale reconstruction of image f using g as marker. Table 2: Residue mapping of the plateaus in Figure 12a.   Table 2. The darker parts of these figures represent the different structures obtained by the filtering process. Figures 15a, 15b, 15c, and 15d show some specific cases which combine both contrast and size parameter information in the filtering/segmentation process. Remark that based on the new defined image attributes, we can easily distinguish components of the image having the same size and different gray-scale values. Figures 16,17,18,19, and 20 illustrate the filtering based on residues by attribute by considering different classes of real images. Figures 16 and 17 illustrate the extraction of different image components with the same gray-scale and size parameter, but with different vanishing order (the Portuguese strings   "COMO COMPRAR OU VENDER UM CARRO PELA IN-TERNET" and "??" in Figure 16, and the black components in Figure 17). Figure 18 shows another example of segmentation based on the information η and Figure 19 shows a segmentation example based on the threshold of a filtered image. In this case, the significant components, which cannot be extracted by a simple thresholding operation, have the same size and vanishing order in the original image. As shown in Figure 19d, the combination of both these information in our filtering/segmentation framework yields a correct extraction  of the image components. Finally, Figure 20 illustrates the noise smoothing of a radar image as well as the elimination of some of its irrelevant components.

Some real examples
The next section discusses briefly some basic properties of the operations defined here.

Properties
Let ρ f (g) be the morphological reconstruction of an image f using the marker g, and let Preserve and Eliminate be the markers used in a filtering by attributes of image f , as before. Shortly, this filtering by attributes can be expressed as We can see that this operation is anti-extensive since It is also an idempotent operation since the transformation Ω(Ω(f )) = ρ Ω(f ) (Preserve) − ρ Ω(f ) (Eliminate) does not modify the result obtained at the first iteration Ω(f ) which, originally, eliminates the undesired content of the image. At the next iteration, we have that Eliminate(ν) = 0, for all ν ∈ Γ Eliminate , and, thus, Ω(Ω(f )) = ρ Ω(f ) (Preserve) = Ω(f ).
Finally, we can see that the filtering by attributes preserves the original contours of the image, in the sense that it uses the morphological reconstruction method discussed in Section 5 which extends the plateaus of an image while preserves the contours of its components (see Figure 11) [22,23].

CONCLUSIONS
In this work, we introduced a new filtering technique based on the notion of residues by attribute. Basically, the method consists in a decomposition of the image through the morphological residues which constitute a complete and hierarchical representation of the original image. From this decom-  position, we consider the concept of binary residues, vanishing attributes and its corresponding attributes of order and number. Based on these attributes we define a set of filtering scheme that has been proved to be useful in image smoothing and segmentation. A common problem with this type of filtering refers to the algorithm performance. The computational time for the morphological residues definition is considerable, even when we take into account a decomposition method of the structuring element. A future work in this sense concerns the algorithmic optimization of the technique by considering, for example, component trees representations. Another aspect to be investigated is the introduction of other more complex attributes in the filtering scheme, such as the connected surface information of the components at a certain residual level.