Learning Mixtures of Bernoulli Templates by Two-Round EM with Performance Guarantee

Dasgupta and Shulman showed that a two-round variant of the EM algorithm can learn mixture of Gaussian distributions with near optimal precision with high probability if the Gaussian distributions are well separated and if the dimension is sufficiently high. In this paper, we generalize their theory to learning mixture of high-dimensional Bernoulli templates. Each template is a binary vector, and a template generates examples by randomly switching its binary components independently with a certain probability. In computer vision applications, a binary vector is a feature map of an image, where each binary component indicates whether a local feature or structure is present or absent within a certain cell of the image domain. A Bernoulli template can be considered as a statistical model for images of objects (or parts of objects) from the same category. We show that the two-round EM algorithm can learn mixture of Bernoulli templates with near optimal precision with high probability, if the Bernoulli templates are sufficiently different and if the number of features is sufficiently high. We illustrate the theoretical results by synthetic and real examples.


Introduction
During the past decades, a large number of theoretical results have been obtained for supervised learning such as classification and regression [9]. For unsupervised learning, however, relatively few theoretical results are available. A main difficulty is that the objective functions in unsupervised learning are usually non-convex and multi-modal, so the optimization algorithms usually cannot find the global optima. As a result, it is generally difficult to obtain theoretical guarantees for the performances of the unsupervised learning algorithms. A simple and typical example of unsupervised learning is clustering or learning mixture models, and a typical algorithm for fitting the mixture models is the EM algorithm [3], which is a statistical counterpart of the k-means algorithm for clustering. Although the EM algorithm is simple and interpretable, and is known to converge monotonically to a local mode of the observed-data log-likelihood, little is known about its theoretical performance in terms of correctly recovering the mixture components. As such, the EM algorithm is often considered a heuristic algorithm.
A major recent advance in the theoretical understanding of the EM algorithm for fitting mixture models was made by Dasgupta and Shulman [1]. They proposed a two-round variant of the EM algorithm that consists of only two iterations of EM: the first iteration is initialized from a number of randomly selected training examples as the centers of the Gaussian distributions, and the second iteration is carried out after pruning the clusters learned from the first iteration. They showed that the two-round EM can learn the mixture of Gaussian distributions with near optimal precision with high probability if the Gaussian distributions are well separated and if the dimensionality of the Gaussian distributions is sufficiently high. Here near optimal precision means that one can estimate the parameters of the Gaussian distributions as if the memberships of the observations are known. In this paper, we generalize the theory of Dasgupta and Shulman [1] to learning mixture of Bernoulli templates. Each template is a binary vector, and it generates examples by independently switching its binary components with a certain probability. So the observed examples are also binary vectors. This setup is a version of the latent class model of [5] restricted to binary data. In potential applications in computer vision, a binary vector is a feature map of an image, where each binary component indicates whether a local feature or structure is present or absent within a certain cell of the image domain. Fig. 1 illustrates the basic idea by a synthetic example. The image domain is equally partitioned into squared cells (in the example in Fig. 1, there are a total of 9 × 9 = 81 cells in the image domain). There is an alphabet of sketch patterns that can appear in these cells ( Fig. 1 shows an alphabet of 18 types of sketch patterns). Each cell may contain one or more sketch patterns, so the binary vector for each image consists of 9 × 9 × 18 binary components, each component indicates whether a certain sketch pattern is present or not within a certain cell. Specifically, each component is a binary decision that can be made based on local edge detection, Gabor filter responses [2], beamlet transformation [6] or a pre-trained classifier. A Gabor filter is a 2D linear filter that has a prefered orientation. Along that orientation the Gabor filter resembles a Gaussian and along the perpen-dicular direction it resembles the derivative of a Gaussian. It was shown in [2] that the Gabor filters are a good approximation of the receptive field profiles of orientation-sensitive neurons in a cat's visual cortex.
The formulation is very general. One can design any alphabet of local features or patterns, and one can use any binary detector or classifier to decide the presence or absence of these features within each cell. The whole feature map is a composition of local image features and is in the form of a binary vector, usually high dimensional (on the order of 10 3 −10 5 ). A template itself is a binary vector that is subject to component-wise switching or Bernoulli noise to account for the variations of the feature maps of individual images. The reason we focus on binary feature maps in this article is that they are easy to design and we do not need to make strong assumptions on their distributions such as Gaussianity. As another illustration, Fig. 2 displays some examples of real images and their binary sketches based on a simple design of image features and binary decision rules. We partition the image domain into squared cells of equal size (in these images, the cells are relatively small, ranging from 5 × 5 pixels to 7 × 7 pixels). We convolve the image with Gabor filters [2] at 8 orientations. Within each cell, at each orientation, we pool a local maximum of the Gabor filter responses (in absolute values). If the local maximum is above a threshold, we then declare that there is a sketch within this cell at this orientation, and the sketch is depicted by a bar in the corresponding binary sketch image in Fig.  2. Clearly the sketch image captures a lot of information in the corresponding original image. Now back to the issue of learning mixture models by EM. We assume that there are k Bernoulli templates, and each observed example is a noisy observation of one of the k template. The question we want to answer is: given a number of training examples that are noisy observations of the k templates, can an EM-type algorithm reliably recover these k templates with high probability? The reason we are interested in this question is that it will shed light on unsupervised learning of templates of objects (or their parts) from real images, which is a crucial task for object modeling and recognition in computer vision. Many learning methods are based on fitting mixture models by EM-type algo-rithms, including the Active Basis model [8]. In the language of the And-Or graph [11] for object modeling, each template is an And-node, which is a composition of a number of sketches. The mixture of k templates is an Or-node, with each template being its child node. So the mixture of the templates is an Or-And structure. The theoretical results in this paper will be useful for us to understand the learning of the Or-And structure from training images.
To answer the above question, we shall generalize the theory of Dasgupta and Shulman [1] to Bernoulli distributions, and we shall show that the two-round EM algorithm can learn mixtures of Bernoulli templates with near optimal precision with high probability if the templates are sufficiently different and if the dimensions are sufficiently high.
Generalizing the theory of [1] from Gaussian mixtures to the mixtures of Bernoulli distributions is far from being straightforward. The sample space is no longer Euclidean, and some results for Gaussian distributions cannot be translated directly into those for the Bernoulli models. So we have to establish a theoretical foundation that is suitable for our purpose. For example, we will need bounds on the tails of the distribution of distances between a template P and the mean of m binary vectors obtained by perturbing P by Bernoulli noise. Similar bounds for the Gaussian case are easy to obtain because the moment generating function of X 2 is known when X is an isotropic Gaussian.
The rest of the paper is organized as follows. Section 2 describes the two-round EM algorithm and states the main theorem. Sections 3 to 4 present theoretical results that lead to the proof of the main theorem. Section 5 illustrates the theoretical results by some experiments on synthetic and real examples. Section 6 concludes with a discussion. In the text, we shall only state the theoretical results. The proofs can be found in the appendix.

Model and algorithm
Let P be a template. It is an n-dimensional binary vector, i.e., P ∈ Ω = {0, 1} n . In the example in Fig. 1, n = 9 × 9 × 18 = 1458. Let P(s) be the s-th component of P, s = 1, ..., n. An example x generated by P is a noisy version of P, and we write x ∼ P. Specifically, let x(s) be the s-th component of x. Then x(s) = P(s) with probability 1 − q, and x(s) = 1 − P(s) with probability q, i.e., q is the probability of switching a component of P, and it defines the level of Bernoulli noise. We assume that q ∈ (0, 1/2). We also assume that the components of x are independent given P. We call P a Bernoulli template because it is binary and is subject to Bernoulli noise.
Let {P i , i = 1, ..., k} be k Bernoulli templates with mixture weights {w i , i = 1, ..., k}. We assume that k is given. Otherwise, k can be determined by some model selection criteria such as BIC [4,7]. Let x 1 , ..., x m be m noisy observations of these k templates, where the noise level is q. The probability that x j is generated by P i is w i , and we let w min = min i=1,...,k w i . We define µ i to be the expectation of the examples generated by P i , i.e., µ i = E[x i ] where x i ∼ P i . Let S i be the set of examples coming from the template P i .
For two n-dimensional vectors P and Q, let D(P, Q) = n s=1 |P(s) − Q(s)| be the 1 distance between P and Q. Let c ij be the separation between P i and P j , i.e., D(P i , P j ) = d ij = nc ij .
We shall show that if the separation c is sufficiently large, then the two-round EM algorithm will reliably recover {P i , i = 1, ..., k}.
We use the notation T i to denote the estimated P i . In the two-round EM, the first round initializes {T (0) i , i = 1, ..., l} to be l randomly selected training examples. The initial number of clusters, l, is greater than the true number k. Specifically, we let l = 4 wmin ln 2 δwmin , where δ is the confidence parameter, i.e., with probability 1 − δ, the algorithm will succeed in recovering the mixture components. According to the coupon collector problem, the l examples cover all the k clusters with high probability. We estimate the Bernoulli noise level q 0 so that q 0 (1 − q 0 ) = min ij D(T After the first iteration, we prune the clusters by a starvation scheme. The pruning process consists of two stages. In the first stage, we remove all the templates {T (1) i } whose weights are below a threshold 1/4l. In the second stage, we keep only k templates that are far apart from each other through an inclusion process. Specifically, we start the inclusion process by randomly picking a template. Then in each subsequent step of the inclusion process, we add a template that is farthest away from the selected templates in terms of the minimum distance between the candidate template and the selected templates. We repeat this step until we get k templates. We let i = 1, ..., k to index the remaining k templates.
After the pruning process, we run another iteration of EM. The estimated templates from this second round EM are already near optimal as we will show.
To be more precise, Algorithm 1 describes the two-round EM. In Step 9 the templates {T (2) i } are to be converted to binary by rounding to the nearest integer.

Notation
For the convenience of reference, the following summarizes the notation used in this paper: • n is the dimension of Bernoulli templates, which generate examples in Ω = {0, 1} n . • m is the number of observations. • k is the true number of clusters.
wmin ln 2 δwmin . The parameter δ is the confidence level in Theorem 1.
4l is the threshold for pruning the clusters learned by the first round.
• C i collects the templates that are initialized from examples in the i-th cluster S i and survive the pruning process after the first round of EM, i.e.
Let m examples be generated from a mixture of k Bernoulli templates under Bernoulli noise of level q and mixing weights w i ≥ w min for all i. Let , δ ∈ (0, 1). If the following conditions hold: 1. The initial number of clusters is

The separation is
).

The dimension is
Then with probability at least 1 − δ, the estimated templates after the round 2 of EM satisfy: The above theorem states that with high probability, the estimated templates from the two-round EM is nearly as accurate as if we knew the memberships of the examples.

Sketch of the proof
The proof follows the steps of the two-round EM. We show that after the initialization, with high probability, the initial templates cover all the clusters and the estimated noise level q 0 is close to the true noise level q. Then after the first round, the estimated templates are likely to be close to the true templates of the same clusters. After the pruning process, we prove that it is very likely that exactly one template is kept for each cluster. Finally after the second round, the estimated templates are proved to be near optimal.

Basic facts
We shall first establish some basic facts about the Bernoulli templates perturbed by Bernoulli noise. They are concerned with the 1 distances among templates and their examples. Proposition 1. Let P, Q ∈ Ω be Bernoulli templates with noise level q. We have: Proposition 2. Let P, Q ∈ Ω be Bernoulli templates with noise level q. We have: a) If x ∼ P and λ ≥ 1 then then for any ∈ (0, 1) Prop. 2 states that the 1 distance between an example and its template is concentrated around nq, while the distance between two examples from two different templates is concentrated around ν(P, Q). This leads to the following proposition.
Proposition 3. Draw m samples from a c-separated mixture of k Bernoulli templates with mixing weights at least w min . Let 0 > 0. Then with probability at Proposition 4. (Average of subsets) Draw a set S 1 of m examples randomly from template P ∈ {0, 1} n with noise level q < 1/2. Then with probability at least 1 − δ for any subset of size at least t ≥ n there is no subset of S 1 of size at least t whose average µ has Prop. 4 states that the sample average is unlikely to deviate too far from P.
imsart-ejs ver. 2013/03/06 file: TwoStepEM-EJS.tex date: January 21, 2015 Prop. 5 states that the weighted average can be bounded by unweighted average. This result is needed because the templates are estimated as the weighted averages in both rounds of the EM algorithm and from Prop. 4 and 5 we can bound on the distance to the template.

Key steps of the proof
In this section we state the results that hold for the estimated template parameters after each EM iteration. We assume that the following technical conditions hold These conditions are a subset of the conditions of Theorem 1 that don't depend on w min and δ. They will be referred to in the proofs of the statements of this section.
We also assume that 0 ≤ E where condition C3 guarantees that E > 0. Observe that condition C3 imposes an upper bound on the noise level q since c < 1. In our experiments this upper bound was between 0.2 and 0.3.

Initialization
This section analyzes the initial estimates for the parameters before the first round of EM.
By initializing from more templates than the actual number of clusters, there is a high probability that the estimated templates cover all the clusters.

First Round of EM
In the cases when the conclusions of Proposition 3 hold, for any x ∈ S i the ratio between the probabilities p i and p j is Prop. 7 states that the first round of EM will likely give higher weights to the templates representing the correct cluster than to a wrong cluster.
Proposition 8. In the cases when the conclusions of Proposition 3 hold, any non-starved estimate T So the estimated template of a cluster is very likely to be close to the true template of this cluster.

Pruning
We prove that with high probability the pruning step will keep exactly one template from each cluster.
Proposition 9. In the cases when Propositions 3, 6 and 8 hold, the set C i obeys the following properties: a) Each C i is non-empty b) There exists τ ∈ R such that for any x ∈ C i and y, z ∈ C j , j = i we have D(y, z) ≤ τ and D(x, y) > τ . c) The pruning procedure finds exactly one member of each C i .

Second Round of EM
We permute the obtained templates T In the cases when Propositions 3, 6 and 8 hold, for any x ∈ S i the ratio between the probabilities p i and p j is Theorem 2. Suppose that l > k, w i > w min for all i and that conditions C1 − C3 hold. Then with probability at least 1−m 2 e −2n(1−q) 2 0 /3 −m 2 e −n min(c,0.5) 2 0 /3 − 2me −n 2 0 /3 − 2me −n min(c,0.5) 2 0 /3 − ke −mwmin/8 − k(l + 1)e −lwmin − ke lwmin/12 − k/n, the estimated templates after the round 2 of EM satisfy: We are now ready to prove Theorem 1.
Proof of Theorem 1.

Experiments
This section illustrates the theoretical results obtained in the previous sections by a simulation study as well as experiments on synthetic image sketches and real images.

Simulation study
In this section we conduct experiments showing that indeed, the true templates are found with high probability when the conditions of Theorem 1 hold. We will work with a mixture of two templates, P 1 = 0 and P 2 = (1, 1, ..., 1, 0, 0, .., 0) where the number of 1's is cn , to obtain a desired separation c ∈ [0, 1] in dimension n. We experiment with standard EM for 2, 10 and 20 iterations. The standard EM starts from k clusters, instead of l clusters followed by pruning as Domains where the two-round EM and the standard EM find the k = 2 binary templates correctly 90% of the time when m = 300. The first plot is for q = .01, with n = 2, 000, and the second plot is for q = .1 with n = 10, 000. Also shown is the domain theoretically guaranteed by Theorem 1. Each domain is above and to the right of the corresponding curve.
in the two-step EM. For the standard EM we also assumed the noise level q is a known parameter. All results are obtained from 100 runs. Fig. 3 and 4 show the domains where the two-step EM and the standard EM find the templates P 1 , P 2 with 90% probability, thus δ = 0.1.
In the two plots of Fig. 3, the horizontal axis is the minimum weight w min , and the vertical axis is the separation c. The domain for each algorithm is the region above and to the right of the corresponding curve. Two version of the two step EM algorithm were evaluated: the two-step EM, and 10-step version that does 9 EM steps after the pruning step. Five version of the original EM were evaluated, with 2 or 10 iterations, and 1, 5 or 10 random initializations (and selecting from the 5 or 10 obtained results the largest likelihood one as the final result). The first plot is obtained at the noise level q = .01, while the second plot is for the noise level q = .1. We take the number of observations m = 300. For the first plot the dimension is n = 2, 000 and for the second plot, n = 10, 000. One can see that for low noise, the two-step EM works better than the original EM. Also displayed is the domain where the conditions of our theorem are satisfied.
In the four plots from Fig. 4 the horizontal axis is the number m of observations and the vertical axis is the dimension n. The four plots show the domain where the two-step EM algorithm finds the templates P 1 , P 2 with 90% probability for the levels of noise q ∈ {0.0001, 0.01, 0.1, 0.2}. The curves corresponding to conditions 2-4 of Theorem 1 and the technical conditions C1-C3 are also displayed, as well as the domain where all conditions of our theorem are satisfied.
From the experiments we observe that the domain where the templates are found with high probability is larger than the domain where the conditions of Theorem 1 hold. The largest discrepancy is in the dimensionality conditions, where the gap between theory and experiments is considerable. This gap could be substantially decreased if tighter bounds could be obtained for Prop 4 and consequently for Prop 8 and Theorem 2.

Experiments on synthetic image sketches
In this experiment we work with a mixture of two Bernoulli templates, shown in the bottom row of Fig. 5, in a space of dimension n = 9 × 9 × 18 = 1458. By perturbing the entries with Bernoulli noise of level q we obtain images such as those shown in the top row of Fig. 5. Fig. 6 shows the success rate of finding the two templates exactly using the two-round EM algorithm vs. the number of training examples. The experiments are run for two levels of noise q ∈ {.1, .2} and two mixture weights w min ∈ {.2, .4}.
The separation between the two templates is quite small c = .02, because the two templates share a lot of zero components. So the separation conditions fail in this case. Since we are not in the conditions of the Theorem 1, the bound on the training examples is not expected to hold. We may achieve a better bound if we reduce the dimension n while increasing c by selecting those features that differentiate the templates. In any case, we see that in the given scenarios the two templates can be recovered with 100% certainty with the two-round EM given sufficiently many examples. So Theorem 1 might hold under milder assumptions  than ours.

Experiments on real images
We also performed experiments on real images. Each image is first convolved with Gabor filters tuned to 16 orientations. Then the image domain is partitioned into equal sized squared cells (the size ranges from 5 × 5 pixels to 7 × 7 pixels). Within each cell, at each orientation, we pool the maximum of the absolute values of the filter responses. If the maximum is above a threshold, we declare that there is a sketch within this cell at this orientation. Thus each cell produces a binary vector of 16 components. We then concatenate the binary vectors of all the cells into a large binary vector. So each image is transformed into a binary vector.
Evaluation metrics. To evaluate the clustering quality, we use two metrics from [10]: conditional purity and conditional entropy. Given the underlying ground-truth category labels X (which are unknown to the algorithm) and the obtained cluster labels Y , the conditional purity is defined as the mean of the maximum category probabilities for (X, Y ),   Table 1 Comparison of the two-step EM algorithm and the original EM for clustering motorcycles, bicycles and cars. Shown are the mean±std of the conditional purity, conditional entropy and log-likelihood. #Round is the number of steps of an EM algorithm, and #Init the number of random initializations used to select the best result (in terms of the log-likelihood).
N is the number of runs (out of 100 total runs) that recover the clusters perfectly. and the conditional entropy is defined as, where both p(y) and p(x|y) are estimated on the training set, and we would expect higher purity and lower entropy for a better clustering algorithm. We then use the two-round EM algorithm to cluster the images and learn a binary template for each cluster. We compare with the original EM algorithm running for different numbers of iterations (2, 10, 20, 100 in the experiments) and starting with desired number k of clusters (while the two-round EM starts with l > k clusters and prunes them). A more robust EM could be obtained by starting with many random initialization and choosing the clustering result that has the largest log-likelihood. Such robust versions with different number of initializations are also evaluated in Tables 1, 2 and 3. A ten-round version of the two-round EM (with eight additional EM iterations after the pruning step) is also evaluated. The methods are evaluated in terms of conditional purity and conditional entropy. From the experiments one could see that the two-round EM algorithm can only be outperformed with a five or ten random initializations of the standard EM algorithm. All the results are obtained based on 100 runs. Fig. 7 and 8 show the results of two experiments (vehicles and animal faces). Table 1 and 2 show the performance comparisons. Table 3 shows the performance all data combined. In the learned templates, the existence of a sketch at each cell is represented by a bar at the center of this cell and at the orientation of the sketch. In each experiment, there are 15 images in each cluster, and the two-round EM is able to separate the clusters perfectly. For the real images, the templates are denser than those in Fig. 5 because the numbers of cells are larger.
Currently we use a very simple sketch detector by thresholding the Gabor filter responses at different orientations. We will design more sophisticated features and associated detectors in future work.

Discussion
This paper obtains theoretical guarantees on the performance of a two-round EM algorithm for learning mixture of Bernoulli templates, by generalizing the theory of [1]. Unlike the theoretical results for supervised learning, results on  Table 2 Performance comparison of our two-round EM algorithm and the original EM algorithm for clustering cats, wolves and deers.  Table 3 Performance comparison of our two-round EM algorithm and the original EM algorithm for clustering cats, wolves, deers, motorcycles, bicycles and cars.
unsupervised learning such as clustering are relatively scarce. The results obtained in this paper can be useful for understanding the behavior of EM-type algorithms for unsupervised learning. In our future work, we shall improve the theoretical results by relaxing the conditions on the separation between the templates as well as the sample size. We shall also generalize Bernoulli templates to more general statistical models for images, such as templates with dependent switching of the binary components, as well as other non-Gaussian models such as exponential family models.

Acknowledgments
The authors wish to acknowledge support from DARPA MSEE grant FA 8650-11-1-7149 and NSF grant DMS 1007889. The authors thank Maria Pavlovskaia for her help with the experiment on synthetic sketches. Also thanks to Jianwen Xie for assistance with the animal face experiments, and to Prof. Song-Chun Zhu for valuable suggestions. and 3. In the case when x, y ∼ P we have 4. In the case when x ∼ P, y ∼ Q we have Proof of Prop. 2. Statements a), b) follow directly from the Chernoff inequality.
c) Let C be indices of the n − d common elements of P and Q. Let B i be the Bernoulli event that the i-th element of x and y are different. Then

Proof of
Proof of Lemma 1. The mean of mn Bernoullis B ij with E[B ij ] = q (the coordinates of the Z i ) satisfies and we take = λ/nq. Proof of Prop. 4. First, it is sufficient to prove it for subsets of size exactly t, otherwise we increase t. Without loss of generality, we can assume P = 0. From Lemma 1 we have The number of t-point subsets of S 1 is m t < (me/t) t , thus Solving for me t k e −tλ 2 /3nq = δ we get Proof of Prop 5. Sort the points x ∈ S by D(x, P) = n i=1 |x i − P i | and take T as the ones with |T | = x∈S w x largest values. Then Proof of Prop. 6. Let B i be the Bernoulli event that a random sample from the mixture comes from the i-th true template P i . Then E[B i ] = w i . Having l random samples B ij from the Bernoulli event B i , then Thus P(P i is represented twice) ≥ 1− (l + 1)e −lwmin , so P(P i is represented twice, ∀i = 1, k) ≥ (1 − (l + 1)e −lwmin ) k ≥ 1 − k(l + 1)e −lwmin .
3. As there exist T i , T j representing the same cluster, then 2nq Prop. 3, a). Also from Prop. 3, if the minimum is attained for two centers T i , T j representing the same cluster, we are done. Otherwise so both parts of the inequality are proved.
Proof of Prop 7 . We have since we have the following condition obtained from 0 ≤ E. We also have since 0 < 1/4 2q)).
Let i be such that T But |{j, T j ∈ S i }| ≤ 15lw i /8 so there is a j, T j ∈ S i such that j" ∈ C j for i = j. Then from Proposition 8 we have j" ) ≤ 2nq + 4 6nql while using Proposition 8 and the triangle inequality we get j ) ≥ D(P i , P j ) − 2nq − 4 6nql ≥ ≥ nc − 2nq − 4 6nql > 2nq + 4 6nql from condition c ≥ 4q + 8 6ql/n (C3), so we can take τ = 1 2 nc. c) There are k true clusters, exactly as many as selected templates. If two selected templates were from the same cluster, there should be a cluster that has no selected templates. But the two templates from the same cluster are at distance at most τ while the distance of a template from the unselected cluster has distance more than τ , we get a contradiction. where a = 1−q0 q0 > 1, and therefore p (2) i (x) p , P i ) = x p (2) i (x)D(x, P i ) x p (2) i (x) ≤ x∈Si p (2) i (x)D(x, P i ) x∈Si p (2) i (x) + j =i x∈Sj p (2) i (x)D(x, P i ) x∈Si p (2) i (x) From Proposition 10 we have for x ∈ S i , p (2) j (x) ≤ p (2) i (x)e −ncB/2 ≤ e −ncB/2 so p (2) i (x) = 1 − j =i p (2) j (x) ≥ 1 − ke −ncB/2