Three categories of existing approaches.

Texture-based methods (e.g. [6], [7], [24]) treat text as a special type of texture and make use of its textural properties, such as local intensities, spatial variance, filter responses, and wavelet coefficients. Generally, these methods are computation demanding as all locations and scales are exhaustively scanned. Moreover, these algorithms mostly only detect horizontal texts.

In an early work, Zhong et al. [31] proposed a method for text localization in color images. Horizontal spatial variance was used to roughly localize texts and color segmentation was performed within the localized areas to extract text components. The system of Wu et al. [32] adopted a set of Gaussian derivatives to segment texts. Rectangular boxes surrounding the corresponding text strings were formed, based on certain heuristic rules on text strings, such as height similarity, spacing and alignment. The above steps were applied to an image pyramid and the results were fused to make final detections. Li et al. [33] presented a system for detecting and tracking texts in digital video. In this system, the mean and the second- and third-order central moments of wavelet decomposition responses are used as local features. Zhong et al. [34] proposed to localize candidate caption text regions directly in the discrete cosine transform (DCT) compressed domain using the intensity variation information encoded in the DCT domain. The method proposed by Gllavata et al. [24] utilized the distribution of high-frequency wavelet coefficients to statistically characterize text and non-text areas.

Different from the methods surveyed above, in which filter responses or transform domain coefficients are used as features, the algorithm of Kim et al. [6] relies merely on intensities of raw pixels. A Support Vector Machine (SVM) classifier is trained to generate probability maps, in which the positions and extents of texts are searched using adaptive mean shift. Lienhart and Wernicke [35] used complex-valued edge orientation maps computed from the original RGB image as features and trained neural network to distinguish between text and non-text patterns.

The method of Weinman et al. [36] used a rich representation that captures important relationships between responses to different scale- and orientation-selective filters. To improve the performance, conditional random field (CRF) was used to exploit the dependencies between neighboring image region labels. Based on the observation that areas with high edge density indicate text regions, text detection in [37] was carried out in a sequential multi-resolution paradigm.

To speed up text detection, Chen et al. [7] proposed an efficient text detector, which is a cascade Adaboost classifier. The weak classifiers are trained on a set of informative features, including mean and variance of intensity, horizontal and vertical derivatives, and histograms of intensity gradient. Recently, Wang et al. [10] present a method for spotting words in natural images. They first perform character detection for every letter in an alphabet and then evaluate the configuration scores for the words in a specified list to pick out the most probable one.

Component-based methods (e.g. [4], [9], [14], [38]) first extract candidate text components through various ways (e.g. color reduction [4], [14] and Maximally Stable Extremal Region detection [11], [25]) and then eliminate non-text components using heuristic rules or trained classifier, based on geometry and appearance properties. Component-based methods are usually more efficient than texture-based methods because the number of candidate components is relatively small. These methods are more robust to the variations of texts, such as changes of font, scale and orientation. Moreover, the detected text components can be directly used for character recognition. Due to these advantages, recent progresses in text detection and recognition in natural images have been largely advanced by this category of methods [9], [11], [14], [18], [38]–[40].

In [4], color reduction and multi-valued image decomposition are performed to partition the input image into multiple foreground components. Connected component analysis is applied to these foreground components, followed by a text identification module, to filter out non-text components.

The great success of sparse representation in face recognition [41] and image denoising [42] has inspired numerous researchers in the community. The authors of [43] and [12] apply classification procedure to candidate text components, using learned discriminative dictionaries.

The MSER-based methods [11], [18], [25], [40] have attracted much attention from the community, because of the excellent characteristics of MSERs (Maximally Stable Extremal Regions) [44]. MSERs can be computed efficiently (near linear complexity) and are robust to noise and affine transformation. In [11], MSERs are detected and taken as candidate text components. Neumann et al. [40] modified the original MSER algorithm to take region topology into consideration, leading to superior detection performance. Chen et al. [25] also proposed an extension to MSER, in which the boundaries of MSERs are enhanced via edge detection, to cope with image blur. Recently, Neumann et al. [18] further extend the work of [11], [40] to achieve real-time text detection and recognition.

Epshtein et al. [9] proposed a novel image operator, called Stroke Width Transform (SWT), which transforms the image data from containing color values per pixel to containing the most likely stroke width. Based on SWT and a set of heuristic rules, this algorithm can reliably detect horizontal texts.

While most existing algorithms are designed for horizontal or near-horizontal texts, Yi et al. [14] and Shivakumara et al. [16] consider the problem of detecting multi-oriented texts in images or video frames. After extracting candidate components using gradient and color based partition, the line grouping strategy in [14] aggregates the components into text strings. The text strings can be in any direction. However, the method of [14] relies on a large set of manually defined rules and thresholds. In [16], candidate text component clusters are identified by K-means clustering in the Fourier-Laplacian domain. The component clusters are divided into separate components using skeletonization. Even though this method can handle multi-oriented texts, it only detects text blocks, rather than characters, words or sentences.

Finally, hybrid methods (e.g. [13], [45]) are a mixture of texture-based and component-based methods. In [45], edge pixels of all possible text regions are extracted, using an elaborate edge detection method; the gradient and geometrical properties of region contours are verified to generate candidate text regions, followed by a texture analysis procedure to distinguish true text regions from non-text regions. Unlike [45], the hybrid method proposed by Pan et al. [13] extracts candidate components from probability maps at multiple scales. The probability maps are estimated by a classifier, which is trained using a set of texture features (HOG features [46]) computed in predefined patterns. Like most other algorithms, these two methods only detect horizontal texts.

Component extraction.

At this stage, edge detection is performed on the original image and the edge map is fed to the SWT [9] module to produce an SWT image. Neighboring pixels in the SWT image are grouped together recursively to form connected components using a simple association rule.

Component analysis.

Many components extracted at the component extraction stage are not parts of texts. The component analysis stage is aimed to identify and filter out those non-text components. First, the components are filtered using a set of heuristic rules that can distinguish between obvious spurious text regions and true text regions. Next, a component level classifier is applied to prune the non-text components that are hard for the simple filter.

Candidate linking.

The remaining components are taken as character candidates. In fact, components do not necessarily correspond to characters, because a single character in some languages may consist of several strokes; however, we still call them characters (or character candidates) hereafter for simplicity. The first step of the candidate linking stage is to link the character candidates into pairs. Two adjacent candidates are grouped into a pair if they have similar geometric properties and colors. At the next step, the candidate pairs are aggregated into chains in a recursive manner.

Chain analysis.

At the chain analysis stage, the chains determined at the former stage are verified by a chain level classifier. The chains with low classification scores (probabilities) are discarded. The chains may be in any direction, so a candidate might belong to multiple chains; the interpretation step is aimed to dispel this ambiguity. The chains that pass this stage are the final detected texts.

The remainder of this paper is organized as follows. Section Methodology presents the details of the proposed method, including the algorithm pipeline and the two sets of features. Section Dataset and Evaluation Protocol introduces the proposed dataset and evaluation protocol. The experimental results and discussions are given in Section Experiments and Discussions. Section Conclusions concludes the paper and points out potential directions for future research.

Component extraction.

To extract connected components from the image, SWT [9] is adopted for its effectiveness and efficiency. SWT is an image operator which computes per pixel width of the most likely stroke containing the pixel. It provides a way to discover connected components from edge map directly, which makes it unnecessary to consider the factors of scale and direction. See [9] for details.

SWT runs on edge map, so we use Canny edge detector [51] to produce an edge map (Fig. 3 (b) of [23]) from the original image (Fig. 3 (a) of [23]). The resulting SWT image is shown in Fig. 3 (c) of [23].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 3. Typical images from the proposed dataset along with ground truth rectangles.

Notice the red rectangles. They indicate the texts within them are labeled as difficult (due to blur or occlusion).

https://doi.org/10.1371/journal.pone.0070173.g003

The next step of this stage is to group the pixels in the SWT image into connected components. The pixels are associated using a simple rule that the ratio of SWT values of neighboring pixels is less than 3.0. The connected components are shown in Fig. 3 (d) of [23]. Note the red rectangles in the image, where each rectangle contains a connected component.

In fact, the proposed pipeline is general and not specific to any kind of low level operator for component extraction. Though SWT is employed to extract components in this paper, other methods (such as MSER [11], [25]) that are able to reliably generate connected components corresponding to character candidates can also be used. We leave evaluation and comparison of different component extraction methods for future research.

Component analysis.

The purpose of component analysis is to identify and eliminate the connected components that are unlikely parts of texts. To this end, we devise a two-layer filtering mechanism.

The first layer is a filter consists of a set of heuristic rules. This filter runs on a collection of statistical and geometric properties of components, which are very fast to compute. True text components usually have nearly constant stroke width and compact structure (not too thin and long), so width variation, aspect ratio and occupation ratio are chosen as the basic properties to filer out obvious non-text components.

For a connected component with foreground pixels (black pixels in the SWT image), we first compute its bounding box (its width and height are denoted by and , respectively) and the mean as well as standard deviation of the stroke widths, and . The basic properties are defined as follows:

• Width variation. Width variation measures the variation in stroke width of the component : .
• Aspect ratio. In horizontal conditions aspect ratio is defined as the ratio between the width and height of the component . To accommodate texts of different directions, we use a new definition of aspect ratio: .
• Occupation ratio. Occupation ratio is used to remove non-text components caused by spurious rays in the SWT image. This property is defined as the ratio between the number of foreground pixels and area of the component : .

The valid ranges of these basic properties are empirically set to [0,1], [0.1,1] and [0.1,1], respectively. Components with one or more invalid properties will be taken as non-text regions and discarded. A large portion of obvious non-text components are eliminated after this step (notice the difference between Fig. 3 (d) of [23] and Fig. 3 (e) of [23]), suggesting that this preliminary filter is effective.

The second layer is a classifier trained to identify and reject the non-text components that are hard to remove with the preliminary filter. A collection of component level features, which capture the differences of geometric and textural properties between text components and non-text components, are used to train this classifier. The criteria for feature design are: scale invariance, rotation invariance and low computational cost. To meet these criteria, we propose to estimate the center, characteristic scale and major orientation of each component (Fig. 4 of [23]) before computing the component level features. Based on these characteristics, features that are both effective and computationally efficient can be obtained. The details of these component level features are discussed in Section Component Level Features.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 4. Ground truth generation.

(a) Human annotation. The annotators are required to locate and bound each text line using a four-vertex polygon (red dots and yellow lines). (b) Ground truth rectangle (green). The ground truth rectangle is generated automatically by fitting a minimum area rectangle using the polygon.

https://doi.org/10.1371/journal.pone.0070173.g004

For a component , the barycenter , major axis , minor axis , and orientation are estimated using Camshift [52] by taking the SWT image of component as distribution map. The center, characteristic scale and major orientation of component are defined as:(1)(2)(3)

These characteristics are invariant to translation, scale and rotation to some degree (Fig. 4 of [23]). As we will explain in Section Component Level Features, this is the key to the scale and rotation invariance of the component level features.

We train a component level classifier using the component level features. Random Forest [53] is chosen as the strong classifier. The component level classifier is the first level of the two-level classification scheme. The probability of component , , is the fraction of votes for the positive class (text) from the trees. The components whose probabilities are lower than a threshold are eliminated and the remaining components are considered as character candidates (Fig. 3 (f) of [23]). To ensure high recall, the threshold is set very low, as high threshold may filter out true text components.

Candidate linking.

The character candidates are aggregated into chains at this stage. This stage also serves as a filtering step because the candidate characters cannot be linked into chains are taken as components accidentally formed by noises or background clutters, and thus are discarded.

Firstly, character candidates are linked into pairs. In [9], whether two candidates can be linked into a pair is determined based on the heights and widths of their bounding boxes. However, bounding boxes are not rotation invariant, so we use their characteristic scales instead. If two candidates have similar stroke widths (ratio between the mean stroke widths is less than 2), similar sizes (ratio between their characteristic scales does not exceed 2.5), similar colors and are close enough (distance between them is less than two times the sum of their characteristic scales), they are labeled as a pair. The above parameters are optimized using the training data of the ICDAR datasets [47]–[49], however, this parameter setting turns out to be effective for all the datasets used in this paper.

Unlike [9] and [11], which only consider horizontal or near-horizontal linkings, the proposed algorithm allows linkings of arbitrary directions. This endows the system with the ability of detecting multi-oriented texts, not limited to horizontal texts.

Next, a greedy hierarchical agglomerative clustering [54] method is applied to aggregate the pairs into candidate chains. Initially, each pair constitutes a chain. Then the similarity between each couple of chains that share at least one common candidate and have similar orientations is computed; chains with the highest similarity are merged together to form a new chain. The orientation consistency and population consistency between two chains and , which share at least one common candidate, are defined as:(4)and(5)where is the included angle of and while and are the candidate numbers of and . is used to judge whether two chains have similar orientations and is empirically set to . The similarity between two chains and is defined as the harmonic mean [55] of their orientation consistency and population consistency:

(6)According to this similarity definition, the chains with proximal sizes and orientations are merged with priority. This merging process proceeds until no chains can be merged.

At last, the character candidates not belonging to any chain are discarded. The candidate chains after aggregation are shown in Fig. 3 (g) of [23], in which each green line represents a chain.

Chain analysis.

The candidate chains formed at the previous stage might include false positives that are random combinations of scattered background clutters (such as leaves and grasses) and repeated patterns (such as bricks and windows). To eliminate these false positives, a chain level classifier is trained using the chain level features.

Random Forest [53] is again used. The chain level classifier is the second level of the two-level classification scheme. The probability of chain , , is the fraction of votes for the positive class (text) from the trees. The chains with probabilities lower than a threshold are eliminated.

To make better decisions, the total probability of each chain is also calculated. For a chain with candidates , the total probability is defined as:(7)

The chains whose total probabilities are lower than a threshold are discarded.

As texts of different orientations are considered, the remaining chains may be in any direction. Therefore, a candidate might belong to multiple chains. For example, in Fig. 3 (h) of [23] the character ‘ P’ in the first line is linked in three chains (note the green lines). In reality, however, a character is unlikely to belong to multiple text lines. If several chains compete for the same candidate, only the chain with the highest total probability will survive (note the difference between Fig. 3 (h) and (i) in [23]).

The survived chains are outputted by the system as detected texts (Fig. 3 (j) of [23]). For each detected text, its orientation is calculated through linear least squares [54] using the centers of the characters; its minimum area rectangle [56] is estimated using the orientation and the bounding boxes of the characters. Word partition, which divides text lines into separate words, is also implemented in the proposed algorithm; but it is not shown, since the general task of text detection does not require this step.

The whole algorithm described above is performed twice to handle both bright text on dark background and dark text on bright background, once along the gradient direction and once along the inverse direction. The results of two passes are fused to make final decisions. For clarity, only the results of one pass are presented.

Component level features.

Inspired by Shape Context [57] and Feature Context [58], we devise two templates (Fig. 5 (a) of [23]) to capture the regularities of each component in coarse and fine granularity, respectively. The radius and orientation of the templates are not stationary, but adaptive to the component. When computing descriptors for a component, each template is placed at the center and rotated to align with the major orientation of the component; the radius is set to the characteristic scale of the component. Different cues from the sectors are encoded and concatenated into histograms. In this paper, the following cues are considered for each sector:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 5. Calculation of overlap ratio between detection rectangle and ground truth rectangle.

https://doi.org/10.1371/journal.pone.0070173.g005

• Contour shape [59]. Contour shape is a histogram of oriented gradients. The gradients are computed on the component contour (Fig. 5 (c) of [23]).
• Edge shape [59]. Edge shape is also a histogram of oriented gradients; but the gradients are computed at all the pixels in the sector (Fig. 5 (d) of [23]).
• Occupation ratio. Occupation ratio is defined as the ratio between the number of the foreground pixels of the component within the sector and the sector area (Fig. 5 (e) of [23]).

To achieve rotation invariance, the gradient orientations are rotated by an angle , before computing contour shape and edge shape. Then, the gradient orientations are normalized to the range . Six orientation bins are used for computing histograms of contour shape and edge shape, to cope with different fonts and local deformations.

For each cue, the signals computed in all the sectors of all the templates are concatenated to form a descriptor. We call these descriptors scalable rotative descriptors, because they are computed based on templates that are scalable and rotative. Scalable rotative descriptors are similar to PHOG [60], as they both adopt spatial pyramid representation [61]. Different from the templates used for computing PHOG, our templates are circular and their scale and orientation are adaptive to the component being described. This is the key to the scale and rotation invariance of these descriptors.

It is widely accepted in the community that alignment is very important for recognition and classification tasks [62], [63], as it can moderate or even eliminate the negative effects caused by transformations and thus lead to more robust measurement and similarity. Our strategy for computing scalable rotative descriptors, i.e. estimating center, characteristic scale and major orientation of components and calculating features using adaptive templates, is actually a kind of implicit alignment of components. This strategy can be generalized to multi-oriented text recognition either.

The characteristic scale is crucial for the computation of scalable rotative descriptors, because it directly determines the scales of the templates. Too small templates may miss important information of components while too large templates may introduce noises and interferences from other components and background. The value of characteristic scale calculated using Eqn. 2 is a good trade-off in practice.

We have found through experiments (not shown in this paper) that using finer templates can slightly improve the performance, but will largely increase the computational burden.

In addition, another three types of rotation and scale invariant features are considered:

• Axial ratio. Axial ratio is computed by dividing the major axis of the component with its minor axis: .
• Width variation. This feature is the same as defined in Sec. Component Analysis.
• Density. The density of component is defined as the ratio between its pixel number and characteristic area (here the characteristic area is , not the area of the bounding box): .

Chain level features.

Eleven types of chain level features, which are not specific to rotation and scale, are designed to discriminate text lines from false positives (mostly repeated patterns and random clutters) that cannot be distinguished by the component level features.

For a candidate chain with () candidates , the features are defined as below and summarized in Tab. 1:

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Chain level features.

https://doi.org/10.1371/journal.pone.0070173.t001

• Candidate count. This feature is adopted based on the observation that false positives usually have very few (for random clutters) or too many (for repeated patterns) candidates.
• Average probability. The probabilities given by the component level classifier are reliable. This feature is the average of all the probabilities () of the candidates belonging to .
• Average turning angle. Most texts present in linear form, so for a text line the mean of the turning angles at the interior characters () is very small; however, for random clutters this property will not hold. is the included angle between the line and .
• Size variation. In most cases characters in a text line have approximately equal sizes; but it’s not that case for random clutters. The size of each component is measured by its characteristic scale .
• Distance variation. Another property of text is that characters in a text line are distributed uniformly, i.e. the distances between consecutive characters have small deviation. The distance between two consecutive components is the distance of their centers and .
• Average direction bias. For most text lines, the major orientations of the characters are nearly perpendicular to the major orientation of the text line. Direction bias of component , , is the included angle between and the orientation of the chain.
• Average axial ratio. Some repeated patterns (e.g. barriers) that are not texts consist of long and thin components, this feature can help differentiate them from true texts.
• Average density. On the contrary, other repeated patterns (e.g. bricks) consist of short and fat components, this feature can be used to eliminate this kind of false positives.
• Average width variation. False positives formed by foliage usually have varying widths while texts have constant widths. This feature is defined as the mean of all the width variation values of the candidates.
• Average color self-similarity. Characters in a text line usually have similar but not identical color distributions with each other; yet in false positive chains, color self-similarities [64] of the candidates are either too high (repeated patterns) or too low (random clutters). The color similarity is defined as the cosine similarity of the color histograms of the two candidates and .
• Average structure self-similarity. Likewise, characters in a text line have similar structure with each other while false positives usually have almost the same structure (repeated patterns) or diverse structures (random clutters). The structure similarity is defined as the cosine similarity of the edge shape descriptors of the two components and .