Discovering Potential Correlations via Hypercontractivity

Discovering a correlation from one variable to another variable is of fundamental scientific and practical interest. While existing correlation measures are suitable for discovering average correlation, they fail to discover hidden or potential correlations. To bridge this gap, (i) we postulate a set of natural axioms that we expect a measure of potential correlation to satisfy; (ii) we show that the rate of information bottleneck, i.e., the hypercontractivity coefficient, satisfies all the proposed axioms; (iii) we provide a novel estimator to estimate the hypercontractivity coefficient from samples; and (iv) we provide numerical experiments demonstrating that this proposed estimator discovers potential correlations among various indicators of WHO datasets, is robust in discovering gene interactions from gene expression time series data, and is statistically more powerful than the estimators for other correlation measures in binary hypothesis testing of canonical examples of potential correlations.


Introduction
Measuring the strength of an association between two random variables is a fundamental topic of broad scientific interest. Pearson's correlation coefficient [1] dates from over a century ago and has been generalized seven decades ago as maximal correlation (mCor) to handle nonlinear dependencies [2][3][4]. Novel correlation measures to identify different kinds of associations continue to be proposed in the literature; these include maximal information coefficient (MIC) [5] and distance correlation (dCor) [6]. Despite the differences, a common theme of measurement of the empirical average dependence unites the different dependence measures. Alternatively, these are factual measures of dependence and their relevance is restricted when we seek a potential dependence of one random variable on another. For instance, consider a hypothetical city with very few smokers. A standard measure of correlation on the historical data in this town on smoking and lung cancer will fail to discover the fact that smoking causes cancer, since the average correlation is very small. On the other hand, clearly, there is a potential correlation between smoking and lung cancer; indeed applications of this nature abound in several scenarios in modern data science, including a recent one on genetic pathway discovery [7].
Discovery of a potential correlation naturally leads one to ask for a measure of potential correlation that is statistically well-founded and addresses practical needs. Such is the focus of this work, where our proposed measure of potential correlation is based on a novel interpretation of the Information Bottleneck (IB) principle [8]. The IB principle has been used to address one of the fundamental tasks in supervised learning: , how do we find a compact summary of a variable X that is most informative in explaining another variable Y . The output of the IB principle is a compact summary of X that is most relevant to Y and has a wide range of applications [9,10].
We use this IB principle to create a measure of correlation based on the following intuition: if X is (potentially) correlated with Y , then a relatively compact summary of X can still be very informative about Y . In other words, the maximal ratio of how informative a summary can be in explaining Y to how compact a summary is with respect to X is, conceptually speaking, an indicator of potential correlation from X to Y .
Quantifying the compactness by I(U ; X) and the information by I(U ; Y ) we consider the rate of information bottleneck as a measure of potential correlation: where U − X − Y forms a Markov chain and the supremum is over all summaries U of X. This intuition is made precise in Section 2, where we formally define a natural notion of potential correlation (Axiom 6), and show that the rate of information bottleneck s(X; Y ) captures this potential correlation (Theorem 1) while other standard measures of correlation fail (Theorem 2). This ratio has only recently been identified as the hypercontractivity coefficient [11]. Hypercontractivity has a distinguished and central role in a large number of technical arenas including quantum physics [12,13], theoretical computer science [14,15], mathematics [16][17][18] and probability theory [19,20]. In this paper, we provide a novel interpretation to the hypercontractivity coefficient as a measure of potential correlation by demonstrating that it satisfies a natural set of axioms such a measure is expected to obey.
For practical use in discovering correlations, the standard correlation coefficients are equipped with corresponding natural sample-based estimators. However, for hypercontractivity coefficient, estimating it from samples is widely acknowledged to be challenging, especially for continuous random variables [21][22][23]. There is no existing algorithm to estimate the hypercontractivity coefficient in general [21], and there is no existing algorithm for solving IB from samples either [22,23]. We provide a novel estimator of the hypercontractivity coefficient -the first of its kind -by bringing together the recent theoretical discoveries in [11,24] of an alternate definition of hypercontractivity coefficient as ratio of Kullback-Leibler divergences defined in ( 10), and recent advances in joint optimization (the max step in Equation 1) and estimating information measures from samples using importance sampling [25].
Our main contributions are the following: • We postulate a set of natural axioms that a measure of potential correlation from X to Y should satisfy (Section 2.1).
• We show that s(X; Y ), our proposed measure of potential correlation, satisfies all the axioms we postulate (Section 2. 2). In comparison, we prove that existing standard measures of correlation not only fail to satisfy the proposed axioms, but also fail to capture canonical potential correlations captured by s(X; Y ) (Section 2. 3). Another natural candidate is mutual information, but it is not clear how to interpret the value of mutual information as it is unnormalized, unlike all other measures of correlation which are between zero and one.
• Computation of the hypercontractivity coefficient from samples is known to be a challenging open problem. We in troduce a novel estimator to compute hypercontractivity coefficient from i.i.d. samples in a statistically consistent manner for continuous random variables, using ideas from importance sampling and kernel density estimation (Section 3).
• In a series of synthetic experiments, we show empirically that our estimator for the hypercontractivity coefficient is statistically more powerful in discovering a potential correlation than existing correlation estimators; a larger power means a larger successful detection rate for a fixed false alarm rate (Section 4.1).
• We show applications of our estimator of hypercontractivity coefficient in two important datasets: In Section 4.2, we demonstrate that it discovers hidden potential correlations among various national indicators in WHO datasets, including how aid is potentially correlated with the income growth. In Section 4.3, we consider the following gene pathway recovery problem: we are given samples of four gene expressions time series. Assuming we know that gene A causes B, that B causes C, and that C causes D, the problem is to discover that these causations occur in the sequential order: A to B, and then B to C, and then C to D. We show empirically that the estimator of the hypercontractivity coefficient recovers this order accurately from a vastly smaller number of samples compared to other state-of-the art causal influence estimators.
2 Axiomatic approach to measure potential correlations We propose a set of axioms that we expect a measure of potential correlation to satisfy. We then show that hypercontractivity coefficient, first introduced in [19], satisfies all the proposed axioms, hence propose hypercontractivity coefficient as a measure of potential correlation. We also show that other standard correlation coefficients and mutual information, on the other hand, violate the proposed axioms.

Axioms for potential correlation
We postulate that a measure of potential correlation ρ * : X × Y → [0,1] between two random variables X ∈ X and Y ∈ Y should satisfy: 1. ρ * (X, Y ) is defined for any pair of non-constant random variables X and Y .

5.
If (X, Y ) ∼ N (µ, Σ), then ρ * (X, Y ) = |ρ|, where ρ is the Pearson correlation coefficient. 6. ρ * (X, Y ) = 1 if there exists a subset X r ⊆ X such that for a pair of continuous random variables (X, Y ) ∈ X r × Y, Y = f (X) for a Borel-measurable and non-constant continuous function f .  Figure 1: A measure of potential correlation should capture the rare correlation in X ∈ [0,1] in these examples which satisfy Axiom 6 for a linear and a quadratic function, respectively. Axioms 1-5 are identical to a subset of the celebrated axioms of Rényi in [4], which ensure that the measure is properly normalized and invariant under bijective transformations, and recovers the Pearson correlation for jointly Gaussian random variables. Rényi's original axioms for a measure of correlation in [4] included Axioms 1-5 and also that the measure ρ * of correlation should satisfy The Pearson correlation violates a subset (3, 4, and 6') of Rényi's axioms. Together with recent empirical successes in multimodal deep learning (e.g. [26][27][28]), Rényi's axiomatic approach has been a major justification of Hirschfeld-Gebelein-Rényi (HGR) maximal correlation coefficient defined as mCor(X, , which satisfies all Rényi's axioms [2]. Here, the supremum is over all measurable functions However, maximal correlation is not the only measure satisfying all of Rényi's axioms, as we show in the following. This follows from the fact that the hypercontractivity coefficient s(X; Y ) satisfies all but the symmetry in Axiom 7' (Theorem 1), and it follows that a symmetrized version satisfies all axioms, e.g. (1/2)( s(X; Y ) + s(Y ; X)) and (s(X; Y )s(Y ; X)) 1/4 . A formal proof is provided in Section 5.1. From the original Rényi's axioms, for a potential correlation measure, we remove Axiom 7' that ensures symmetry, as directionality is fundamental in measuring the potential correlation from X to Y . We further replace Axiom 6' by Axiom 6, as a variable X has a full potential to be correlated with Y if there exists a domain X r such that X and Y are deterministically dependent and non-degenerate (i.e. not a constant function), as illustrated in Figure 1 for a linear function and a quadratic function.

The hypercontractivity coefficient satisfies all axioms
We show that the hypercontractivity coefficient defined in Equation (1) satisfies all Axioms 1-6. Intuitively, s(X; Y ) measures how much potential correlation X has with Y . For example, if X and Y are independent, then s(X; Y ) = 0 as X has no correlation with Y (Axiom 3). By data processing inequality, it follows that it is a measure between zero and one (Axiom 2) and also invariant under bijective transformations (Axiom 4). For jointly Gaussian variables X and Y with the Pearson correlation ρ, we can show that s(X; Y ) = s(Y ; X) = ρ 2 . Hence, the squared-root of s(X; Y ) satisfies Axiom 5. In fact, s(X; Y ) satisfies all desired axioms for potential correlation, and we make this precise in the following theorem whose proof is provided in Section 5.2.
In particular, the hypercontractivity coefficient satisfies Axiom 6 for potential correlation, unlike other measures of correlation (see Theorem 2 for examples). If there is a potential for X in a possibly rare regime in X to be fully correlated with Y such that Y = f (X), then the hypercontractivity coefficient is maximum: s(X; Y ) = 1. In the following section, we show that existing correlation measures, on the other hand, violate the proposed axioms.

Standard correlation coefficients violate the Axioms.
We analyze existing measures of correlations under the scenario with potential correlation (Axiom 6), where we find that none of the existing correlation measures satisfy Axiom 6. Suppose X and Y are independent (i.e. no correlation) in a subset X d of the domain X , and allow X and Y to be arbitrarily correlated in the rest X r of the domain, such that X = X d ∪ X r . We further assume that the independent part is dominant and the correlated part is rare; let α := P(X ∈ X r ) and we consider the scenario when α is small. A good measure of potential correlation is expected to capture the correlation in X r even if it is rare (i.e., α is small). To make this task more challenging, we assume that the conditional distribution of Y |{X ∈ X r } is the same as Y |{X / ∈ X r }. Figure 1 (of this section) illustrates sampled points for two examples from such a scenario and more examples are in Figure 3. Our main result is the analysis of HGR maximal correlation (mCor) [2], distance correlation (dCor) [6], maximal information coefficients (MIC) [5], which shows that these measures are vanishing with α even if the dependence in the rare regime is very high. Suppose Y |(X ∈ X r ) = f (X), then all three correlation coefficients are vanishing as α gets small. This in particular violates Axiom 6. The reason is that standard correlation coefficients measure the average correlation whereas the hypercontractivity coefficient measures the potential correlation. The experimental comparisons on the power of these measures confirm our analytical predictions in Figure 4 in Section 4. The formal statement is below and the proof is provided in Section 5.3.

Theorem 2. Consider a pair of continuous random variables
the distance correlation coefficient is the maximal information coefficient is upper bounded by where X r is the random variable X conditioned on the rare domain X ∈ X r .
Under the rare/dominant scenario considered in Theorem 2, s(X; Y ) ≥ mCor 2 (X; Y ). It is well known that this inequality holds for any X and Y [19]. In particular, Theorem 3 in [29] shows that hypercontractivity coefficient is a natural extension of the popular HGR maximal correlation coefficient as follows.
Remark 1 (Connection between s(X; Y ) and mCor(X, Y ) [29]). The squared HGR maximal correlation is a special case of the hypercontractivity optimization in Equation ( 10) restricted to searching over a distribution r(x) in a close neighborhood of p(x).
As s(X; Y ) searches over a larger space, it is always larger than or equal to mCor 2 (X; Y ). This gives an intuitive justification for using s(X; Y ) as a measure of potential influence; we allow search over larger space, but properly normalized by the KL divergence, in a hope to find a potential distribution r(x) that can influence Y significantly. While hypercontractivity coefficient is a natural extension of HGR maximal correlation coefficient, there is an important difference between hypercontractivity coefficient and HGR maximal correlation coefficient (and other correlation measures); hypercontractivity is directional.
Remark 2 (Asymmetry of s(X; Y )). Hypercontractivity coefficient is asymmetric in X and Y while HGR maximal correlation, distance correlation, and MIC are symmetric.
Under the rare/dominant scenario considered in Theorem 2, the hypercontracitivy coefficient s(X; Y ) is large because it measures the potential correlation from X to Y . On the other hand, inverse hypercontractivity coefficient s(Y ; X), which measures the potential correlation from Y to X, is small as there is no apparent potential correlation from Y to X. This is made precise in the following proposition.

Proposition 2. Under the hypotheses of Theorem 2, the hypercontractivity coefficient from
where X r is the random variable X conditioned on the rare domain X ∈ X r .
Proof is provided in Section 5.4.

Mutual Information violates the Axioms
Beside standard correlation measures, another measure widely used to quantify the strength of dependence is mutual information. We can show that mutual information satisfies Axiom 6 if we replace 1 by ∞. However there are two key problems: • Practically, mutual information is unnormalized, i.e., I(X; Y ) ∈ [0, ∞). Hence, it provides no absolute indication of the strength of the dependence.
• Mathematically, we are looking for a quantity that tensorizes, i.e., doesn't change when there are many i.i.d. copies of the same pair of random variables.
Remark 3 (Tensorization property of s(X; Y ) [30]). Hypercontractivity coefficient tensorizes, i.e, On the other hand, mutual information is additive, i.e., Tensorizing quantities capture the strongest relationship among independent copies while additive quantities capture the sum. For instance, mutual information could be large because a small amount of information accumulates over many of the independent components of X and Y (when X and Y are high dimensional) while tensorizing quantities would rule out this scenario, where there is no strong dependence. When the components are not independent, hypercontractivity indeed pools information from different components to find the strongest direction of dependence, which is a desirable property.
One natural way to normalize mutual information is by the log of the cardinality of the input/output alphabets [31]. One can interpret a popular correlation measure MIC as a similar effort for normalizing mutual information and is one of our baselines.
Given that other correlation measures and mutual information do not satisfy our axioms, a natural question to ask is whether hypercontractivity is a unique solution that satisfies all the proposed axioms. In the following, we show that the hypercontractivity coefficient is not the only one satisfying all the proposed axioms -just as HGR correlation is not the only measure satisfying Rényi's original axioms.

Hypercontractivity ribbon
We show that a family of measures known as hypercontractivity ribbon, which includes hypercontractivity coefficient as a special case, satisfies all the axioms. The hypercontractivity ribbon [19,32] is a class of measures parametrized by α ≥ 1 as where D(r(x)||p(x)) denotes the KL divergence of r(x) and p(x). An alternative characterization of hypercontractivity ribbon in terms of mutual information is provided in [24,32]; from which we can see that hypercontractivity coefficient is a special case of hypercontractivity ribbon [11]: Proof. By definition, s α (X; Y ) is defined for any pair of non-constant random variables (Axiom 1) and is between 0 and 1 by data processing inequality (Axiom2). We can show that s α (X; Y ) satisfies Axioms 3 and 4, in a similar way to show s(X; Y ) satisfies Axioms 3 and 4. Also, s α (X; Y ) = ρ 2 for a jointly Gaussian X,Y with Pearson correlation ρ [24] (Axiom 5). Finally, s α (X; Y ) satisfies Axiom 6 because r α (X; Y ) is non-increasing in α, which implies that s α (X; Y ) = r α (X; Y ) = 1 if s(X; Y ) = 1.
Although hypercontracitivy ribbon satisfies all axioms, a few properties of the hypercontractivity coefficient make it more attractive than hypercontractivity ribbon for practical use; hypercontractivity coefficient can be efficiently estimated from samples (see Section 3). Hypercontractivity coefficient is a natural extension of the popular HGR maximal correlation coefficient (Remark 1).

Multidimensional X and Y
In this section, we discuss potential correlation of multidimensional X and Y . While most of the correlation coefficients, including the hypercontractivity coefficient, are well-defined for multi-dimensional X and Y , the axioms are specific to univariate X and Y . To bridge this gap, we propose replacing Axiom 5, as this is the only axiom specific to univariate random variables.
This recovers the original Axiom 5 when restricted to univariate X and Y . This naturally generalizes both Rényi's axioms and the proposed potential correlation axioms to multidimensional X and Y . Proposition 4. Axiom 5', together with original Rényi's Axioms 1-4, 6', and 7', recovers maximal correlation (mCor) as a measure satisfying all Axioms even in this multi-dimensional case. Axiom 5', together with our proposed Axioms 1-4, and 6, recovers the hypercontractivity coefficient s(X; Y ) as a measure satisfying all axioms.
The second statement in the proposition follows from the analyses of the hypercontractivity coefficient of Gaussian distributions in [33]. A formal proof is provided in Section 5.7.

Noisy, discrete, noisy and discrete potential correlations
In this section, we consider more general scenarios of potential correlation than the one in Axiom 6. We consider (i) noisy potential correlation where Y = f (X) + Z for a Gaussian noise Z for (X, Y ) ∈ X r × Y, (ii) discrete potential correlation, where X r = {1, · · · , k}, and (iii) noisy discrete potential correlation -a random corruption model. For these three examples, we obtain a lower bound on s(X; Y ).
Example 1. Suppose that for a pair of random variables (X, Y ) ∈ X × Y, there exists a subset X r ⊆ X for which P{X ∈ X r } = α (α > 0), and for (X, Proof is in Section 5.5. We now consider for discrete (X, Y ). We start with the case for which X and Y are perfectly correlated for (X, Y ) ∈ X r × Y.
We conjecture this lower bound is indeed tight for α ≤ 0.5 based on numerical simulations. From this lower-bound, we can see the trade-off between k and α. As k → ∞, the lower bounds approaches to 1. As α → 1, the lower bound approaches to 1. As α → 0, the lower bound approaches to 0. In the following, we consider the case where X and Y are not perfectly correlated in (X r × Y) for discrete (X, Y ). In particular, we consider a random corruption model for (X r × Y) and obtain a lower bound on s(X; Y ).
Example 3. Suppose that for a pair of random variables (X, Y ) ∈ X × Y, there exists a subset X r ⊆ X for which P{X ∈ X r } = α (α > 0), and for (X, Y ) ∈ X r × Y, On the other hand, Proof is in Section 5.6.
In Figure 2, we show plots of lower bounds on s(X; Y ) and mCor(X; Y ) in Examples 1-3; from these figures, we can see that s(X; Y ) increases as ρ → 1 and k → ∞. In comparison, mCor(X; Y ) remains small.

Estimator of the hypercontractivity coefficient from samples
In this section, we present an algorithm 1 to compute the hypercontractivity coefficient The computation of the hypercontractivity coefficient from samples is known to be challenging for continuous random variables [22,23], and to the best of our knowledge, there is no known efficient algorithm to compute the hypercontractivity coefficient from samples. Our estimator is the first efficient algorithm to compute the hypercontractivity coefficient, based on the following equivalent definition of the hypercontractivity coefficient, shown recently in [11]: There are two main challenges for computing s(X; Y ). The first challenge is -given a marginal distribution r x and samples from p xy , how do we estimate the KL divergences D(r y ||p y ) and D(r x ||p x ). The second challenge is the optimization over the infinite dimensional simplex. We need to combine estimation and optimization together in order to compute s(X; Y ). Our approach is to combine ideas from traditional kernel density estimates and from importance sampling. Let w i = r x (X i )/p x (X i ) be the likelihood ratio evaluated at sample i. We propose the estimation and optimization be solved jointly as follows: Estimation: To estimate KL divergence D(r x ||p x ), notice that Using empirical average to replace the expectation over p x , we propose For D(r y ||p y ), we follow the similar idea, but the challenge is in computing v j = r y (Y j )/p y (Y j ). To do this, notice that r xy = r x p y|x , so Replacing the expectation by empirical average again, we get the following estimator of v j : 1 Code available at https://github.com/wgao9/hypercontractivity We can write this expression in matrix form as v = A T w. We use a kernel density estimator from [34] to estimate the matrix A, but our approach is compatible with any density estimator of choice. Optimization: Given the estimators of the KL divergences, we are able to convert the problem of computing s(X; Y ) into an optimization problem over the vector w. Here a constraint of (1/n) n i=1 w i = 1 is needed to satisfy E px [r x /p x ] = 1. To improve numerical stability, we use log s(X; Y ) as the objective function. Then the optimization problem has the following form: Although this problem is not convex, we apply gradient descent to maximize the objective. In practice, we initialize w i = 1 + N (0, σ 2 ) for σ 2 = 0.01. Hence, the initial r x is perturbed mildly from p x . Although we are not guaranteed to achieve the global maximum, we consistently observe in extensive numerical experiments that we have 50%-60% probability of achieving the same maximum value, which we believed to be the global maximum. A theoretical analysis of the landscape of local and global optima and their regions of attraction with respect to gradient descent is an interesting and challenging open question, outside the scope of this paper.

Consistency of Estimation
While a theoretical understanding of the performance of gradient descent on the optimization step (where the number of samples is fixed) above is technically very challenging, we can study the performance of the solution as the number of samples increases. In particular we show below (under suitable simplifying assumptions to get to the essence of the proof) that the optimal solution to the finite sample optimization problem is consistent. Suppose that X is discrete. Further we restrict the optimization over a quantized and bounded set T ∆ , where w ∈ T ∆ is quantized by a gap ∆ and satisfies: We further assume that we have access of A = P xy (X i , Y j )/P x (X i )P y (Y j ). Define s ∆ (X; Y ) = max w∈T∆ w T A log(A T w)/w T log w, then with two further simplifying conditions on the joint distribution (formally stated in Section 5.8), we can prove consistency of our estimation procedure: Theorem 3. As n goes to infinity, s ∆ (X; Y ) converges to s(X; Y ) up to a resolution of quantization in probability, i.e., for any ε > 0, ∆ > 0 and s(∆) = O(∆), we have

Experimental results
We present experimental results on synthetic and real datasets showing that the hypercontractivity coefficient (a) is more powerful in detecting potential correlation compared to existing measures; (b) discovers hidden potential correlations among various national indicators in WHO datasets; and (c) is more robust in discovering pathways of gene interactions from gene expression time series data.

Synthetic data: power test on potential correlation
As our estimator (and the measure itself) involves a maximization, it is possible that we are sensitive to outliers and may capture spurious noise. Via a series of experiments we show that the hypercontractivity coefficient and our estimator are capturing the true potential correlation. As shown in Figure 3, we generate pairs of datasets -one where X and Y are independent and one where there is a potential correlation as per our scenario. We run experiment with eight types of functional associations, following the examples from [5,35,36]. For the correlated datasets, out of n samples {(x i , y i )} n i=1 , αn rare but correlated samples are in X = [0, 1] and (1 − α)n dominant but independent samples are in X ∈ [1, 1.1]. The rare but correlated samples are generated as Table 1 shows the hypercontractivity coefficient and the other correlation coefficients for correlated and independent datasets shown in Figure 3, along with the chosen value of α and σ 2 . Correlation estimates with the largest separation for each row is shown in bold. The hypercontractivity coefficient gives the largest separation between the correlated and the independent dataset for most functional types. Step function (correlated) x x x x y y Figure 3: Sample data points for eight functions with/without a potential correlation for n = 320.
A formal statistical approach to test the robustness as well as accuracy is to run power tests: testing for the power of the estimator in binary hypothesis tests. To compute the power of each estimator, we compare the false negative rate at a fixed false positive rate of, say, 5%. We generate 500 independent datasets and 500 correlated datasets. We compute the correlation estimates on 500 independent samples, and take the top 5% as a threshold. We compute the correlation estimates on 500 correlated samples. Power is defined as the fraction of correlated datasets for which the correlation estimate is larger than the threshold.
We show empirically that for linear, quadratic, sine with period 1/2, and the step function, the hypercontractivity coefficient is more powerful as compared to other measures. For a given setting, a larger power means a larger successful detection rate for a fixed false alarm rate. Figure 4 shows the power of correlation estimators as a function of the additive noise level, σ 2 , for α = 0.05 and n = 320. The hypercontractivity coefficient is more powerful than other correlation estimators for most functions. The power of all the estimators are very small for sine (period 1/8) and circle functions. This is not surprising given that it is very hard to discern the correlated and independent cases even visually, as shown in Figure 3.  Step function  On the other hand, hypercontractivity estimator is power deficient for the cubic function. This is because in estimating hypercontractivity coefficient, we estimate p(y j |x i )/p(y j ) using the kernel density estimator (KDE), which gives a smooth estimate of p(y j |x i )/p(y j ), i.e., for x i and x j close to each other, estimated p(y|x i ) and p(y|x j ) are close to each other. Hence, for a correlated dataset for a cubic function, shown in   To investigate the dependency of power on α more closely, in Figure 7, we plot the power vs. α or n = 320 and σ 2 = 0.1. Hypercontractivity estimator is more powerful than other estimators for most α, for all functions except for cubic function. For a sine with period 1/8, due to its high frequency, the powers of all the correlation estimators do not increase as α increases. Figure 8 plots the power vs. sample size n for α = 0.05 and σ 2 = 0.1. For sine with period 1/2, hypercontractivity estimator is much more powerful than the other estimators for all sample sizes. We can also see that for sine with period 1/8, powers of all correlation estimators do not increase as sample size increases.

Real data: correlation between indicators of WHO datasets
We computed the hypercontractivity coefficient, MIC, and Pearson correlation of 1600 pairs of indicators for 202 countries in the World Health Organization (WHO) dataset [5]. Figure 9 illustrates that the hypercontractivity coefficient discovers hidden potential correlation (e.g. in (E) -(H)), whereas other measures fail. Scatter plots of Pearson correlation vs. the hypercontractivity coefficient and MIC vs. the hypercontractivity coefficient for all pairs are presented in (A) and (D). The samples for pairs of indicators corresponding to B,C,E -J are shown in (B),(C),(E) -(J), respectively. In Figure 9 (B), it is reasonable to assume that the number of bad teeth per child is uncorrelated with the democracy score. The hypercontractivity coefficient, MIC, and Pearson correlation are all small, as expected. In Figure 9 (C), the correlation between CO 2 emissions and energy use is clearly visible, and all three correlation estimates are close to one.
However, only the hypercontractivity coefficient discovers the hidden potential correlation in Figure 9 (E) -(H). In Figure 9 (E), the data is a mixture of two types of countries -one with small amount of aid received (less than $5 × 10 8 ), and the other with large amount of aid received (larger than $5 × 10 8 ). Dominantly many countries (104 out of 146) belong to the first type (small aid), and for those countries, the amount of aid received and the income growth are independent. For the remaining countries with larger aid received, although those are rare, there is a clear correlation between the amount of aid received and the income growth.
Similarly in Figure 9 (F), there are two types of countries -one with small arms exports (less than $2 × 10 8 ) and the other with large arms exports (larger than $2 × 10 8 ). Dominantly many countries (71 out of 82) belong to the first type, for which the amount of arms exports and the health expenditure are independent. For the remaining countries that belong to the second type, on the other hand, there is a visible correlation between the arms exports and the health expenditure. This is expected as for those countries that export arms the GDP is positively correlated with both arms exports and health expenditure, whereas for those do not have arms industry, these two will be independent. q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q qq q q q q q q q q In Figure 9 (G), for dominant number of countries, the number of male deaths from the colon and rectum cancer is small (145 out of 169 countries have it less than 2000), and it is independent of the amount of health expenditure. On the other hand, for the remaining countries with larger number of male deaths from colon and rectum cancer, the two indicators are positively associated. This is expected as both indicators are positively correlated with the population. Only hypercontractivity discovers this hidden potential correlation. MIC and Pearson correlation are small.
In Figure 9 (H), for dominant number of countries, the number of broadband subscribers is very small and is independent of the private health expenditure; 155 out of 180 countries have broadband subscribers less than 10 6 . On the other hand, for the remaining countries, the number of broadband subscribers is positively correlated with the private health expenditure. This is as expected because both indicators are positively correlated with the population. Hypercontractivity is large for this dataset, discovering the hidden correlation, whereas all other correlations all small.
In Figure 9 (I), most countries do not have large hydroelectricity facilities, and for those countries, energy use and hydroelectricity consumption are independent (41 out of 53 countries have hydroelectricity ≤ 0.25). On the other hand, for the countries which have hydroelectrocity facilities, the amount of total energy use and the amount of hydroelectricity consumption are positively correlated. Hypercontractivity discovers this hidden potential correlation. Unlike in (G) and (H) for which the fraction of correlated samples was only about 14%, in (I), the fraction of correlated samples is about 23%. Hence, Pearson correlation is larger compared to Pearson correlation values for (G) and (H).
In Figure 9 (J), there is one country (Luxembourg) with very large amounts of foreign direct investment net inflow and outflow. Due to this outlier, Pearson correlation is close to 1. Hypercontractivity is also close to 1, whereas MIC is small. To analyze the effect of the outlier in correlation measures, in the following, we compute the correlation measures for samples without an outlier. On the middle and on the right are shown all samples but one outlier and all samples but two outliers, respectively. By comparing the hypercontractivity coefficients for the three datasets for each pair of indicators, we can analyze the effect of outliers on hypercontractivity. For a comparison, on the top of each figure, we show the estimated hypercontractivity (HC), MIC, Pearson correlation (Cor), distance correlation (dCor), maximal correlation (mCor), and the hypercontractivity for reversed direction (HCR). In Figures 10 and 11, we can see that hypercontractivity is more sensitive to an outlier than other correlation measures.   Figure 11: Samples for the pair of indicators shown in Figure 9-(F) from the entire WHO dataset (left), without one outlier (middle), and without two outliers (right).

How hypercontractivity changes as we remove outliers
In Figure 12 (left), the two countries with the largest number of male deaths from the colon and rectum cancer are China and United States. As China is removed from the dataset, in (middle), hypercontractivity remains unchanged. As we also remove United States, in (right), hypercontractivity becomes small, 0.17. This value is still larger than the typical coefficient for two independent indicators (≈ 0.05), we can see that hypercontractivity is more sensitive to the outlier than other correlation measures. In Figure 13, the two countries with the largest number of broadband subscribers are United States and China. When we remove United States from the samples, hypercontractivity becomes close to zero, which also shows hypercontractivity is sensitive to the outliers. In Figure 14, hypercontractivity remains large even after we remove outliers. The two countries with the largest amount of hydroelectricity consumption are Norway and Iceland. Even after we remove Norway from the samples, as shown in (middle), hypercontractivity remains large. As we further remove one outlier (Iceland) from the samples, as shown in (right), hypercontractivity becomes 0.49. In Figure 15, (middle), all samples but Luxembourg is shown. We can see that most countries have a very small absolute amount of foreign direct investment net outflows (For 126 out of 157 countries, it is between [−2, 2]), and for those countries, the foreign direct investment net outflow is independent of foreign direct investment net inflows. For the remaining countries, there is a positive association between the outflow and the inflow. Hypercontractivity captures this hidden correlation better than other correlations; hypercontractivity is 0.47, whereas MIC and Pearson correlation are small. If we further remove the rightmost sample, as shown in (right), hypercontractivity becomes small. q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q qq q q q q q q q q Whether we should consider a sample in a rare type as a meaningful sample or as an outlier depends on the application. If we use hypercontractivity to discover a pair of measures for which one variable can be potentially correlated with the other, then we would expect to discover that an aid for a country has potential correlation in the income growth. Other measures will fail. It is possible that hypercontractivity might have larger false positive rate, and depending on the application, one might prefer to error on the side of having more positive cases to be screened by further experiments, surveys, or human judgements.

Hypercontractivity detecting an outlier
In Figure 16 (A) and (B), we show examples of pairs of indicators for which there is one outlier and the remaining samples are independent, but hypercontractivity is large. As shown in Figure 16 (A) and (B) (left), hypercontractivity is close to 1, when there is an outlier. As shown in (right), hypercontractivity is close to 0, when the outlier is removed. This implies that one single outlier can make the hypercontractivity large. We can see similar patterns for other correlation measures, such as for Pearson correlation, distance correlation, and maximal correlation for both (A) and (B), and MIC for (B), but are less sensitive than hypercontractivity.
To further study how hypercontractivity estimator is affected by outliers, we ran simulations on synthetic data. We generated three sets of synthetic data shown in Figure 17 and computed hypercontractivity coefficients. In Figure 17 (left), an outlier is located far from the rest of samples, and the estimated hypercontractivity coefficient is 0.99. In Figure 17 (middle), an outlier is located close to the rest of samples, and the estimated hypercontractivity coefficient is 0.04. In Figure 17 (right), X and Y are potentially correlated, and the hypercontractivity estimate is 0.17. As can bee seen from this simulation and experimental results on WHO dataset, our hypercontractivity estimator is sensitive to outliers. If one wants to filter out the effect of outliers, one can combine methods for robust estimation, such as trimming and winsorizing [37][38][39], along with the hypercontractivity estimator. This is an interesting future research direction.

Gene pathway recovery from single cell data
We replicate the genetic pathway detection experiment from [7], and show that hypercontractivity correctly discovers the genetic pathways from smaller number of samples. A genetic pathway is a series of genes interacting with each other as a chain. Consider the following setup where four genes whose expression q q q q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q values in a single cell are modeled by random processes X t , Y t , Z t and W t respectively. These 4 genes interact with each other following a pathway X t → Y t → Z t → W t ; it is biologically known that X t causes Y t with a negligible delay, and later at time t , Y t causes Z t , and so on. Our goal is to recover this known gene pathway from sampled datapoints. For a sequence of time points ti } ni j=1 generated from the random process P (X ti , Y ti , Z ti , W ti ). We use the real data obtained by the single-cell mass flow cytometry technique [7].
Given these samples from time series, the goal of [7] is to recover the direction of the interaction along the known pathway using correlation measures as follows, where they proposed a new measure called DREMI. The DREMI correlation measure is evaluated on each pairs on the pathway, τ (X ti , Y ti ), τ (Y ti , Z ti ) and τ (Z ti , W ti ), at each time points t i . It is declared that a genetic pathway is correctly recovered if the peak of correlation follows the expected trend: arg max ti τ (X ti , Y ti ) ≤ arg max ti τ (Y ti , Z ti ) ≤ arg max ti τ (Z ti , W ti ).
In [25], the same experiment has been done with τ evaluated by UMI and CMI estimators. In this paper, we evaluate τ using our proposed estimator of hypercontractivity.
The Figure 18 shows the scatter plots pCD3ζ-pSLP76-pERK-pS6 chain at different time points after TCR activation. The data comes from CD4+ naïve T lymphocytes from B6 mice with CD3, CD28, and CD4 cross-linking. Each row represents a pair of data in the chain, and each column stands for a time point after TCR activation. Estimate of hypercontractivity is shown below the scatter plot for each pair of data and each time point and we highlight the time point where each pair of data is maximally correlated. We can see that the peak of the correlation of pCD3ζ-pSLP76, pSLP76-pERK and pERK-pS6 appears at 0.5 min, 1 min and 2 min respectively, hence the pathway is correctly identified. In Figure 19, the similar plots was shown for T-cells exposed with an antigen. Similarly, hypercontractivity is able to capture the trend.
We subsample the raw data from [7] to evaluate the ability to find the trend from smaller samples. Precisely, given a resampling rate γ ∈ (0, 1], we randomly select a subset of indices ti } j∈Si , and determine whether we can recover the trend successfully, i.e., whether arg max ti τ (X ti , Y ti ) ≤ arg max ti τ (Y ti , Z ti ) ≤ arg max ti τ (Z ti , W ti ). We repeat the experiment several times with independent subsamples and compute the probability of successfully recovering the trend. Figure 20 illustrates that when the entire dataset is available, all methods are able to recover the trend correctly. When only fewer samples are available, hypercontractivity improves upon other competing measures in recovering the hidden chronological order of interactions of the pathway. For completeness, we run datasets for both regular T-cells (shown in left figure) and T-cells exposed with an antigen (shown right figure), for which we expect distinct biological trends. Hypercontractivity method can capture the trend for both datasets correctly and sample-efficiently.    Figure 20: Accuracy vs. subsampling rate. Hypercontractivity method has higher probability to recover the trend when data size is smaller compared to other methods. Left: regular T-cells. Right: T-cells exposed with an antigen [7].

Proofs
In this section, we provide proofs for our main results and technical lemmas.
for any X r such that X r ⊆ X for P{X ∈ X r } =: α > 0.
Proof. Let Consider The proof is completed by combining (14) and (15). (13) in Lemma 1, we obtain the following lower bound: .
For any continuous random variable X and a non-constant continuous function f , I(f (X); f (X)|X ∈ X r ) = I(f (X); X|X ∈ X r ) = ∞, which implies that s(X; Y ) = 1.
Finally, we show that MIC(X, Y ) ≤ α MIC(X r , Y ) in (4). Let X Q (X) ∈ X Q (X) and Y Q (Y ) ∈ Y Q (Y ) denote a quantization of X and Y , respectively. Consider where step (a) holds because I X∈Xr |= Y implies I X∈Xr |= Y Q and X |= Y in X ∈ X d implies X Q |= Y Q in X ∈ X d .

Proof of Proposition 2
The inverse hypercontractivity s(Y ; X) is defined as Step (a) holds because Y |= X given I r = 0. Consider where step (a) folllows since U − Y − I r .

Noisy discrete potential correlation in Example 3
The inequality (8) follows by choosing r(x) = I {X=1} in (10). To show (9), we show that The rest follows because mCor(X; Y ) = √ α mCor(X r , Y ) by Proposition 2. To show (18), we use the fact that maximal correlation is the second eigenvalue of Q = P With a little abuse of notations, we define s(r x ) = D(r y ||p y )/D(r x ||p x ) and s(w) = w T A log(A T w)/w T log w. Therefore, s(X; Y ) = max rx∈R s(r x ) and s ∆ (X; Y ) = max w∈T∆ s(w). Here R is the probability simplex over all r x . We want to bound the error | s ∆ (X; Y ) − s(X; Y )|. First, consider the quantity: where the constraint set T ∆ (R) is defined as: Now we rewrite the error term as The first error comes from quantization. Let r * be the maximizer of s(X; Y ). By assumption, r * (x)/p x (x) ∈ [C 1 , C 2 ], for all x. Since T ∆ (R) is a quantization of the simplex R, so there exists an r 0 ∈ T ∆ (R) such that |r 0 (x) − r * (x)| < ∆ for all x ∈ X . Now we will bound the difference between s(r 0 ) and s(r * ) by the following lemma: for some positive constant L.
Next we have: Similarly, let r * * be the maximizer of s ∆ (X; Y ), we can also find a r 1 ∈ R such that |r 1 (x) − r * * (x)| < ∆ for all x ∈ X . Using Lemma 2 again, we will obtain s ∆ (X; Y ) ≤ s(X; Y ) + L∆. Therefore, the quantization error is bounded by O(∆) with probability 1. Now consider the second term. Upper bound on the second term relies on the convergence of estimation of s. We claim that for given r x , the estimator is convergent in probability, i.e., Here w r (x) = r x (x)/p x (x). Since the set T ∆ (R) is finite, by union bound, we have: Also, by the strong law of large numbers, we have that where n x = card{i ∈ [n] : x i = x}. We claim that if the events inside the probability in (25) and (26) happen simultaneously, then | s ∆ (X; Y ) − s ∆ (X; Y )| < ε + O(∆), which implies the desired claim.
Let w * = arg max w∈T∆ s(w). Define r 2 (x) = w * (x)p x (x). Since [r 2 (x)/p x (x)] ∈ T ∆ for all x and Therefore, r 2 ∈ T ∆ (R), so On the other hand, consider r * * = arg max rx∈T∆(R) s(r x ) again, and define w 0 (x) = r * * (x)/p x (x). We know that w 0 ∈ T |X | ∆ but not necessarily n i=1 w 0 (x i ) = n. But we claim that the sum is closed to n as follows: so we can find a w 1 ∈ T ∆ (R) such that |w 1 (x) − w 0 (x)| ≤ ∆ for all x. Let r 4 (x) = w 1 (x)p x (x), similar as (27), we know that r 4 ∈ T ∆ (R). Moreover, We conclude that | s ∆ (X; Y ) − s ∆ (X; Y )| < ε + O(∆); thus our proof is complete.

Proof of Lemma 3
Note that s(w r ) = w T A log(A T w)/w T log w. Define D(r y ||p y ) = w T A log(A T w) and D(r x ||p x ) = w T log w. We will prove that both D(r y ||p y ) converges to D(r y ||p y ) and D(r x ||p x ) converges to D(r x ||p x ) in probability. Since D(r On the other hand, for fixed Y j = y, we have py(Y ) in probability, by law of large number again. Therefore, we conclude that D(r y ||p y ) converges to D(r y ||p y ) in probability.