Quantifying unique information

We propose new measures of shared information, unique information and synergistic information that can be used to decompose the multi-information of a pair of random variables $(Y,Z)$ with a third random variable $X$. Our measures are motivated by an operational idea of unique information which suggests that shared information and unique information should depend only on the pair marginal distributions of $(X,Y)$ and $(X,Z)$. Although this invariance property has not been studied before, it is satisfied by other proposed measures of shared information. The invariance property does not uniquely determine our new measures, but it implies that the functions that we define are bounds to any other measures satisfying the same invariance property. We study properties of our measures and compare them to other candidate measures.


Introduction
Consider three random variables X, Y, Z with finite state spaces X , Y, Z. Suppose that we are interested in the value of X, but we can only observe Y or Z. If the tuple (Y, Z) is not independent of X, then the values of Y or Z or both of them contain information about X. The information about X contained in the tuple (Y, Z) can be distributed in different ways. For example, it may happen that Y contains information about X, but Z does not, or vice versa. In this case, it would suffice to observe only one of the two variables Y, Z, namely the one containing the information. It may also happen, that Y and Z contain different information, so it would be worthwhile to observe both of the variables. If Y and Z contain the same information about X, we could choose to observe either Y or Z. Finally, it is possible that neither Y nor Z taken for itself contains any information about X, but together they contain information about X. This The value of (4) can be interpreted as the union information, that is, the union of the informations contained in Y and in Z without the synergy.
The problem to separate the contributions of shared information and synergistic information to the co-information is probably as old as the definition of co-information itself. Nevertheless, the co-information has been widely used as a measure of synergy in the neurosciences; see, for example, [9,7] and references therein. The first general attempt to construct a consistent information decomposition into terms corresponding to different combinations of shared and synergistic information is due to Williams and Beer [10]. See also the references in [10] for other approaches to study multivariate information. While the general approach of [10] is intriguing, the proposed measure of shared information I min suffers from serious flaws, which prompted a series of other papers trying to improve these results [4,5,2].
In our current contribution, we propose to define the unique information as follows: Let ∆ be the set of all joint distributions of X, Y and Z. Define ∆ P = Q ∈ ∆ : Q(X = x, Y = y) = P (X = x, Y = y) and Q(X = x, Z = z) = P (X = x, Z = z) for all x ∈ X , y ∈ Y, z ∈ Z In Section 3 we show that the four functions U I, SI and CI are non-negative, and we study further properties. In Appendix 6 we describe the set ∆ P in terms of a parametrization.
Our approach is motivated by the idea that unique and shared information should only depend on the marginal distribution of the pairs (X, Z) and (X, Y ). This idea can be explained from an operational interpretation of unique information: Namely, if Y has unique information about X (with respect to Z), then there must be some way to extract this information. More precisely, there must be a situation in which Y can use this information to perform better at predicting the outcome of X. We make this idea precise in Section 2 and show how it naturally leads to the definition of the functions U I, SI and CI, as defined above. Section 3 contains basic properties of these three functions. In particular, Lemma 5 shows that all three functions are nonnegative. Corollary 7 proves that the function U I is consistent with the operational idea put forward in Section 2. In Section 4 we compare our function with other proposed information decompositions. Some examples are studied in Section 5. Remaining open problems are discussed in Section 6. The appendix contains some more technical aspects that help to compute the functions U I, SI and CI.

Operational interpretation
Our basic idea to characterize unique information is the following: If Y has unique information about X with respect to Z, then there must be some way to extract this information. That is, there must be a situation in which this unique information is useful. We formalize this idea in terms of decision problems as follows: Let X, Y , Z be three random variables, let p be the marginal distribution of X, and let κ ∈ [0, 1] X ×Y and µ ∈ [0, 1] X ×Z be (row) stochastic matrices describing the conditional distribution of Y and Z, respectively, given X. In other words, p, κ and µ satisfy Observe that, if p(x) > 0, then κ(x; y) and µ(x; z) are uniquely defined. Otherwise, κ(x; y) and µ(x; z) can be chosen arbitrarily. In this section, we will assume that the random variable X has full support. If this is not the case, our discussion will remain valid after replacing X by the support of X. In fact, the information quantities that we consider later will not depend on those matrix elements κ(x; y) and µ(x; z) which are not uniquely defined.
Suppose that an agent has a finite set of possible actions A. After the agent chooses her action a ∈ A, she receives a reward u(x, a), which not only depends on the chosen action a ∈ A, but also on the value x ∈ X of the random variable X. The tuple (p, A, u), consisting of the prior distribution p, the set of possible actions A and the reward function u is called a decision problem. If the agent can observe the value x of X before choosing her action, her best strategy is to chose a such that u(x, a) = max a ∈A u(x, a ). Suppose now, that the agent cannot observe X directly, but the agent knows the probability distribution p of X. Moreover, the agent observes a random variable Y with conditional distribution described by the row-stochastic matrix κ ∈ [0, 1] X ×Y . In this context, κ will also be called a channel from X to Y. When using a channel κ, the agent's optimal strategy is to choose her action such that her expected reward is maximal. Note that, in order to maximize (5), the agent has to know (or estimate) the prior distribution of X as well as the channel κ. Often, the agent is allowed to play a stochastic strategy. However, in the present setting, the agent cannot increase her expected reward by randomizing her actions, and therefore, we only consider deterministic strategies here.
be the maximal expected reward that the agent can achieve by always choosing the optimal action. In this setting we make the following definition: Definition 1. Let X, Y, Z be three random variables, and let p be the marginal distribution of X.
• Y has unique information about X (with respect to Z), if there is a set A and a reward function u ∈ R X ×A such that R(κ, p, u) > R(µ, p, u).
• Z has no unique information about X (with respect to Y ), if for any set A and reward function u ∈ R X ×A the inequality R(κ, p, u) ≥ R(µ, p, u) holds. In this situation we also say that Y knows everything that Z knows about X, and we write Y X Z.
This operational idea allows to distinguish when the unique information vanishes, but, unfortunately, does not allow to quantify the unique information.
As shown recently in [1], the question whether Y X Z or not, does not depend on the prior distribution p (but just on the support of p, which we assume to be X ). In fact, if p has full support, then, in order to check whether Y X Z, it suffices to know the stochastic matrices κ, µ representing the conditional distributions of Y and Z given X.
Consider the case Y = Z and κ = µ ∈ K(X ; Y), i.e. Y and Z use a similar channel. In this case, Y has no unique information with respect to Z, and Z has no unique information with respect to Y . Hence, in the decomposition (1) only the shared information and the synergistic information may be larger than zero. The shared information may be computed from Observe that in this case, the shared information can be computed from the marginal distribution of X and Y . Only the synergistic information depends on the joint distribution of X, Y and Z.
We argue that this should be the case in general: By what was said above, whether the unique information U I(X : Y \ Z) is greater than zero only depends on the two channels κ and µ. Even more is true: The set of decision problems (p, A, u) such that R(κ, p, u) > R(µ, p, u) only depends on κ and µ (and the support of p). To quantify the unique information, this set of decision problems must be measured in some way. It is reasonable to expect that this quantification can be achieved by taking into account only the marginal distribution p of X. Therefore, we believe that a sensible measure U I for unique information should satisfy the following property: U I(X : Y \ Z) only depends on p, κ and µ.
( * ) Although this condition seems to have not been considered before, many candidate measures of unique information satisfy this property; for example those defined in [10,5].
In the following, we explore the consequences of assumption ( * ).

Lemma 2.
Under assumption ( * ), the shared information only depends on p, κ and µ.
Proof. This follows from SI(X : Let ∆ be the set of all joint distributions of X, Y and Z. Fix P ∈ ∆, and assume that the marginal distribution of X, denoted by p, has full support. Denote by κ and µ the stochastic matrices corresponding to the conditional distributions of Y and Z given X. Let and Q(X = x, Z = z) = P (X = x, Z = z) for all x ∈ X , y ∈ Y, z ∈ Z be the set of all joint distributions which have the same marginal distributions on the pairs (X, Y ) and (X, Z), and let ∆ * P = Q ∈ ∆ P : Q(x) > 0 for all x ∈ X be the subset of distributions with full support. Lemma 2 says that, under assumption ( * ), the functions U I(X : Y \ Z), U I(X : Z \ Y ) and SI(X : Y ; Z) are constant on ∆ * P , and only the function CI(X : Y ; Z) depends on the joint distribution Q ∈ ∆ * P . If we further assume continuity, the same statement holds true for all Q ∈ ∆ P . To make clear that we now consider the synergistic information and the mutual information as a function of the joint distribution Q ∈ ∆, we write CI Q (X : Y ; Z) and M I Q (X : (Y, Z)) in the following; and we omit this subscript, if these information theoretic quantities are computed with respect to the "true" joint distribution P .
Consider the following functions: Observe that these minima and maxima are well-defined, since the set ∆ P is compact and the mutual informations and the co-information are continuous. The next lemma says that, under assumption ( * ), the quantities U I, SI and CI bound the unique, shared and synergistic information.
and CI(X : Y ; Z) be nonnegative continuous functions on ∆ satisfying equations (1) and (2) and assumption ( * ). Then If P ∈ ∆ and if there exists Q ∈ ∆ P such that CI Q (X : Y ; Z) = 0, then equality holds in all four inequalities. Conversely, if equality holds in one of the inequalities for a joint distribution P ∈ ∆, then there exists Q ∈ ∆ P such that CI Q (X : Y ; Z) = 0.
Proof. Fix a joint distribution P ∈ ∆. By Lemma 2, assumption ( * ) and continuity, the functions U I(X : Y \ Z), U I(X : Z \ Y ) and SI(X : Y ; Z) are constant on ∆ P , and only the function CI(X : Y ; Z) depends on the joint distribution Q ∈ ∆ P . The decomposition (1) rewrites to Using the non-negativity of synergistic information, this implies In total, this shows The chain rule of mutual information says Now, Q ∈ ∆ P implies M I Q (X : Z) = M I(X : Z), and therefore, where H Q (X|Z) = H(X|Z) for Q ∈ ∆ P , and so

By (3), the shared information satisfies
By (2), the unique information satisfies The inequality for U I(X : Z \ Y ) follows similarly.
Hence, in this case, all inequalities are tight. Conversely, assume that one of the inequalities is tight for some P ∈ ∆. The proof above shows that all four inequalities hold with equality. By assumption ( * ), the functions U I and SI are constant on ∆ P . Therefore, the inequalities are tight for all The proof of Lemma 3 shows that the optimization problems defining U I, SI and CI are in fact equivalent; that is, it suffices to solve one of them. Lemma 4 in Section 3 gives yet another formulation and shows that the solution is actually unique.
In the following, we interpret the functions U I, SI and CI as measures of unique, shared and complementary information. Under assumption ( * ), Lemma 3 says that choosing those measures is equivalent to saying that in each set ∆ P there exists a measure Q such that CI Q (X : Y ; Z) = 0. In other words, U I, SI and CI are the only measures of unique, shared and complementary information that satisfy the following property: It is not possible to decide whether or not there is synergistic information, when only the marginal distributions of (X, Y ) and (X, Z) are known. ( * * ) For any other combination of measures different from U I, SI and CI that satisfy assumption ( * ) there are combinations of (p, µ, κ) such that the existence of non-vanishing complementary information can be deduced. Since complementary information should capture precisely the information that is carried by the joint dependencies between X, Y and Z we find assumption ( * * ) natural, and we consider this observation as evidence in favour of our interpretation of the functions U I, SI and CI.

Characterization and Positivity
The next lemma shows that the optimization problems involved in the definitions of U I, SI and CI are easy to solve numerically, in the sense that they are convex optimization problems on convex sets. As always, theory is easier than practice, as discussed in Example 31 in Appendix A.2.
Lemma 4. Let P ∈ ∆ and Q P ∈ ∆ P . The following conditions are equivalent: Moreover, the functions M I Q (X : Y |Z), M I Q (X : Z|Y ) and M I Q (X : (Y, Z)) are convex on ∆ P ; and CoI Q (X; Y ; Z) and H Q (X|Y, Z) are concave. Therefore, for fixed P ∈ ∆, the set of all Q P ∈ ∆ P satisfying any of these conditions is convex.
Proof. The conditional entropy H Q (X|Y, Z) is a concave function on ∆; therefore, the set of maxima is convex. To show the equivalence of the five optimization problems and the convexity properties, it suffices to show that the difference of any two minimized functions and the sum of a minimized and a maximized function is constant on ∆ p . Except for H Q (X|Y, Z) this follows from the proof of Lemma 3. For H Q (X|Y, Z), this follows from the chain rule: The optimization problems mentioned in Lemma 4 will be studied more closely in Appendix 6.
Lemma 5 (Non-negativity). U I, SI and CI are non-negative functions.
Proof. CI is non-negative by definition. The functions U I are non-negative, because they are obtained by minimizing mutual informations, which are non-negative.
Consider the real function It is easy to check Q 0 ∈ ∆ P . Moreover, with respect to Q 0 , the two random variables Y and Z are conditionally independent given X, that is, M I Q 0 (Y : Z|X) = 0. But this implies showing that SI is a non-negative function.
In general, the measure Q 0 constructed in the proof of Lemma 5 does not satisfy the conditions of Lemma 4.

Vanishing shared and unique information
In this section we study when SI = 0 and when U I = 0. In particular, in Corollary 7 we show that U I conforms with the operational idea put forward in Section 2.
Proof. If M I Q (X : Y |Z) = 0 for some Q ∈ ∆ P , then X and Y are independent given Z with respect to Q. Therefore, there exists a stochastic matrix λ ∈ [0, 1] Z×Y satisfying Conversely, if such a matrix λ exists, then the equality defines a probability distribution Q which lies in ∆ P . Then The last result can be translated into the language of our motivational Section 2 and says that U I is consistent with our operational idea of unique information: Proof. We need to show that decision problems can be solved with the channel κ at least as well as with the channel µ if and only if µ = κλ for some stochastic matrix λ. This result is known as Blackwell's theorem [3]; see also [1]. In particular, under assumption ( * ), there is no unique information in this situation.
Proof. Apply Lemma 6 with the identity matrix in the place of λ. Proof. By assumption, P = Q 0 . Thus the statement follows from Lemma 9.

The bivariate PI axioms
In [10], Williams and Beer proposed axioms that a measure of shared information should satisfy. We call these axioms the PI axioms after the partial information decomposition framework derived from these axioms in [10]. In fact, the PI axioms apply to a measure of shared information that is defined for arbitrarily many random variables, while our function SI only measures the shared information of two random variables (about a third variable). The PI axioms are as follows: 1. The shared information of Y 1 , . . . , Y n about X is symmetric under permutations of Y 1 , . . . , Y n .
2. The shared information of Y 1 about X is equal to M I(X : Y 1 ). (self-redundancy) 3. The shared information of Y 1 , . . . , Y n about X is less than the shared information of Y 1 , . . . , Y n−1 about X, with equality if Y n−1 is a function of Y n . (monotonicity) Any measure SI of bivariate shared information that is consistent with the PI axioms must obviously satisfy the following two properties, which we call the bivariate PI axioms: (bivariate monotonicity) We do not claim that any function SI that satisfies A) and B) can be extended to a measure of multivariate shared information satisfying the PI axioms. In fact, such a claim is false, and as discussed in Section 6, our bivariate function SI is not extendable in this way.
The following two lemmas show that SI satisfies the bivariate PI axioms, and they show corresponding properties of U I and CI.
Lemma 11 (Symmetry). where the chain rule of mutual information was used.
The third equality from Lemma 11 is the consistency condition (4). The following lemma is the inequality condition of the monotonicity axiom. To finish the study of the bivariate PI axioms, only the equality condition in the monotonicity axiom is missing. We show that SI satisfies SI(X : Y ; Z) = M I(X : Y ) not only if Z is a deterministic function of Y , but also more generally, when Z is independent of X given Y . In this case, Z can be interpreted as a stochastic function of Y , independent of X.
Lemma 13. If X is independent of Z given Y , then P solves the optimization problems of Lemma 4. In particular,

Probability distributions with structure
In this section we compute the values of SI, CI and U I for probability distributions with special structure. If two of the variables are identical, then CI = 0 as a consequence of Lemma 13 (see Corollaries 15 and 16). When X = (Y, Z), then the same is true (Proposition 18). Moreover, in this case, SI((Y, Z) : Y ; Z) = M I(Y : Z). This equation has been postulated as an additional axiom, called identity axiom, in [5]. Proof. If Y = Z, then X is independent of Z given Y . Proof. If X = Y , then X is independent of Z given Y . The following Lemma shows that the quantities U I, SI and CI are additive when considering systems that can be decomposed into independent subsystems.
Lemma 19. Let X 1 , X 2 , Y 1 , Y 2 , Z 1 , Z 2 be random variables such that (X 1 , Y 1 , Z 1 ) is independent of (X 2 , Y 2 , Z 2 ). Then The proof of the last lemma is given in Appendix A.3.

Comparison with other measures
In this section we compare our information decomposition using U I, SI and CI with similar functions proposed in other papers; in particular, the function I min of [10] and the bivariate redundancy measure I red of [5]. We do not repeat their definitions here, since they are rather technical.
The first observation is that both I red and I min satisfy assumption ( * ). Therefore, I red ≥ SI and I min ≥ SI. According to [5], I min tends to be larger than I red , but there are some exceptions.
It is easy to find examples where I min is unreasonably large [5,2]. It is much more difficult to distinguish I red and SI. In fact, in many special cases the two measures I red and SI agree, as the following results show. Proof of Theorem 20. By Lemma 3, if I red (X : Y ; Z) = 0, then SI(X : Y ; Z) = 0. Now assume that SI(X : Y ; Z) = 0. Since both SI and I red are constant on ∆ P , we may assume that P = Q 0 ; that is, we may assume that Y ⊥ ⊥ Z |X . Then Y ⊥ ⊥ Z by Lemma 9. Therefore, Lemma 21 implies that I red (X : Y ; Z) = 0.
Denote by U I red the unique information defined from I red and (2). Then:  [5], this is equivalent to p(x|y) = p y Z (x) for all x, y. In this case, p(x|y) = z p(x|z)λ(z; y) for some λ(z; y) with z λ(z; y) = 1. Hence, Lemma 6 implies that U I(X : Y \ Z) = 0.
Theorem 22 implies that I red does not contradict our operational ideas introduced in Section 2.
Corollary 23. Suppose that one of the following conditions is satisfied: 2. X is independent of Z given Y .

CI SI I min Note
Rdn RdnXor  Although SI and I red often agree, they are different functions. An example where SI and I red have different values is the dice example given at the end of the next section. In particular, it follows that I red does not satisfy property ( * * ). Table 1 contains the values of CI and SI for some paradigmatic examples. The list of examples is taken from [5]; see also [4]. In all these examples, SI agrees with I red . In particular, in these examples the values of SI agree with the intuitively plausible values called "expected values" in [5].

Examples
As a more complicated example we treated the following system with two parameters λ ∈ [0, 1], α ∈ {1, 2, 3, 4, 5, 6}, also proposed by [5]. Let Y and Z be two dice, and define X = Y + αZ. To change the degree of dependence of the two dice, assume that they are distributed according to For λ = 0 the two dice are completely correlated, while for λ = 1 they are independent. The resulting shared information is shown in Figure 1. As a comparison, we reproduce Figure 8 from [5] showing the function I red in the same example. In fact, for α = 1, α = 5 and α = 6 the two functions agree. Moreover, they agree for λ = 0 and λ = 1. In all other cases, SI ≤ I red , in agreement with Lemma 3. For α = 1 and α = 6 and λ = 0 the fact that I red = SI follows from the results in Section 4; in the other cases we do not know a simple reason for this coincidence.
It is interesting to note that for small λ and α > 1 the function SI depends only weakly on α. In contrast, the dependence of I red on α is stronger. At the moment we do not have an argument that tells us which of these two behaviours is more intuitive.

Outlook
We defined a decomposition of the mutual information M I(X : (Y, Z)) of a random variable X with a pair of random variables (Y, Z) into non-negative terms which have an interpretation in terms of shared information, unique information and synergistic information. We have shown that the quantities SI, CI and U I have many properties that such a decomposition should intuitively fulfil; among them the PI axioms and the identity axiom. It is a natural question whether the same can be done when further random variables are added to the system.
The first question in this context is how the decomposition of M I(X : Y 1 , . . . , Y n ) should look like. How many terms do we need? In the bivariate case n = 2, many people agree that shared, unique and synergistic information should provide a complete decomposition (but it may well be worth to look for other types of decompositions). For n > 2, there is no universal agreement of this kind.
Williams and Beer proposed a framework that suggests to construct an information decomposition only in terms of shared information [10]. Their ideas naturally lead to a decomposition according to a lattice, called PI lattice. For example, in this framework, M I(X : Y 1 , Y 2 , Y 3 ) has to be decomposed into 18 terms with well-defined interpretation. The approach is very appealing, since it is only based on very natural properties of shared information (the PI axioms) and the idea that all information can be "localized," in the sense that, in an information decomposition, it suffices to classify information according to "who knows what," that is, which information is shared by which subsystems. Unfortunately, as shown in [2], our function SI cannot be generalized to the case n = 3 in the framework of the PI lattice. The problem is that the identity axiom is incompatible with a non-negative decomposition according to the PI lattice.
Even though we currently cannot extend our decomposition to n > 2, our bivariate decomposition can be useful for the analysis of larger systems consisting of more than two parts. For example, the quantity can still be interpreted as the unique information of Y i about X with respect to all other variables, and it can be used to assess the value of the ith variable, when synergistic contributions can be ignored. Furthermore the measure has the intuitive property that the unique information cannot grow when additional variables are taken into account: Proof. Let P k be the joint distribution of X, Y, Z 1 , . . . , Z k , and let P k+1 be the joint distribution of X, Y, Z 1 , . . . , Z k , Z k+1 . By definition, P k is a marginal of P k+1 . For any Q ∈ ∆ P k , the distribution Q defined by Q (x, y, z 1 , . . . , z k , z k+1 ) := Q(x,y,z 1 ,...,z k )P k+1 (x,z 1 ,...,z k ,z k+1 ) P k (x,z 1 ,...,z k ) , if P k (x, z 1 , . . . , z k ) > 0, 0, else, lies in ∆ P k+1 . Moreover, Q is the (X, Y, Z 1 , . . . , Z k )-marginal of Q , and Z k+1 is independent of Y given X, Z 1 , . . . , Z k . Therefore, The statement now follows by taking the minimum over Q ∈ ∆ P k .
Thus, we believe that our measure, which is well-motivated in operational terms, can serve as a good starting point towards a general decomposition of multi-variate information.
Appendix: Computing U I, SI and CI A.1 The optimization domain ∆ P By Lemma 4, to compute the quantities U I, SI and CI, we need to solve a convex optimization problem. In this section we study some aspects of this problem.
First we describe ∆ P . For any set S let ∆(S) be the set of probability distributions on S, and let A be the map ∆ → ∆(X × Y) × ∆(X × Z) that takes a joint probability distribution of X, Y and Z and computes the marginal distributions of the pairs (X, Y ) and (X, Z). Then A is a linear map, and ∆ P = (P + ker A) ∩ ∆. In particular, ∆ P is the intersection of an affine space and a simplex; hence ∆ P is a polytope.
The matrix describing A (and denoted by the same symbol in the following) is a wellstudied object. For example, A describes the graphical model associated with the graph Y -X-Z. The columns of A define a polytope, called marginal polytope. Moreover, the kernel of A is known: Let δ x,y,z ∈ R X ×Y×Z be the characteristic function of the point (x, y, z) ∈ X × Y × Z, and let γ x;y,y ;z,z = δ x,y,z + δ x,y ,z − δ x,y ,z − δ x,y,z .
• The functions γ x;y,y ;z,z for all x ∈ X , y, y ∈ Y and z, z ∈ Z span ker A.
• For any fixed y 0 ∈ Y, z 0 ∈ Z, the functions γ x;y 0 ,y;z 0 ,z for all x ∈ X , y ∈ Y \ {y 0 } and z ∈ Z \ {z 0 } form a basis of ker A.
The vectors γ x;y,y ;z,z for different values of x ∈ X have disjoint supports. As the next lemma shows, this can be used to write ∆ P as a Cartesian product of simpler polytopes. Unfortunately, the function M I(X : (Y, Z)) does not respect this product structures. In fact, the diagonal directions are important (see Example 31 below).
Lemma 27. Let P ∈ ∆. For all x ∈ X with P (x) > 0 denote by and Q(Z = z) = P (Z = z|X = x) the set of joint distributions of Y and Z such that the marginal distributions of Y and Z agree with the conditional distributions of Y and Z given X = x. Then the map π P : ∆ P → × x∈X :P (x)>0 ∆ P,x that maps each Q ∈ ∆ P to the family (Q(·|X = x)) x∈X :P (x)>0 of conditional distributions of Y and Z given X = x for those x ∈ X with P (X = x) > 0 is a linear bijection.
Proof. The image of π P is contained in × x∈X :P (x)>0 ∆ P,x by definition of ∆ P . The shows that π P is injective and surjective. Since π P is in fact a linear map, the domain and the codomain of π P are affinely equivalent.
Each Cartesian factor ∆ P,x of ∆ P is a fibre polytope of the independence model.
Proof. By assumption, both conditional probability distributions P (Y |X = x) and P (Z|X = x) are point measures. Therefore, each factor ∆ P,x consists of a single point; namely the conditional distribution P (Y, Z|X = x) of Y and Z given X. Hence, ∆ P is a singleton.

A.2 The critical equations
Lemma 29. The derivative of M I Q (X : (Y, Z)) in the direction γ x;y,y ;z,z is log Q(x, y, z)Q(x, y , z ) Q(x, y , z)Q(x, y, z ) Q(y , z)Q(y, z ) Q(y, z)Q(y , z ) .

Therefore, Q solves the optimization problems of Lemma 4 if and only if
log for all x, y, y , z, z with Q + γ x;y,y ;z,z ∈ ∆ P for > 0 small enough.
Proof. The proof is by direct computation.
Example 30 (The AND-example). Consider the binary case X = Y = Z = {0, 1}, assume that Y and Z are independent and uniformly distributed, and suppose that X = Y AND Z. The underlying distribution P is uniformly distributed on the four states {000, 001, 010, 111}. In this case, ∆ P,1 = {δ Y =1,Z=1 } is a singleton, and ∆ P,0 consists of all probability distributions Q of the form for some 0 ≤ α ≤ 1 3 . Therefore, ∆ P is a one-dimensional polytope consisting of all probability distributions of the form if (x, y, z) = (0, 1, 1), 1 4 , if (x, y, z) = (1, 1, 1), 0, else, for some 0 ≤ α ≤ 1 4 . To compute the minimum of M I Qα (X : (Y, Z)) over ∆ P , we compute the derivative with respect to α (which equals the directional derivative of M I Q (X : (Y, Z)) in the direction γ 0;0,1;0,1 at Q α ) and obtain: Since α In other words, in the AND-example there is no unique information, but only shared and synergistic information. This follows, of course, also from Corollary 8.
Example 31. The optimization problems in Lemma 4 can be very ill-conditioned, in the sense that there are directions in which the function varies fast, and other directions in which the function varies slowly. As an example, consider the example where P is the distribution of three i.i.d. uniform binary random variables. In this case, ∆ P is a square. Figure 2 contains a heat map of the function CoI Q on ∆ P , where ∆ P is parametrized by Q(a, b) = P + aγ 0;0,1;0,1 + bγ 1;0,1;0,1 , Clearly, the function varies very little along one of the diagonals. In fact, along this diagonal, X is independent of (Y, Z), corresponding to a very low synergy. Although in this case the optimising probability distribution Q P is unique, it can be difficult to find. For example, Mathematica's function FindMinimum does not always find the true optimum out of the box (apparently, FindMinimum cannot make use of the convex structure in the presence of constraints) [11].  ). Darker colours indicate larger values of CoI Q . In this example, ∆ P is a square. The uniform distribution lies at the centre of this square and is the maximum of CoI Q . In the two dark corners, X is independent of Y and Z, and either Y = Z or Y = ¬Z. In the two light corners, Y and Z are independent, and either X = Y XOR Z or X = ¬(Y XOR Z).
Proof of Lemma 21. We use the notation from [5]. The information divergence is jointly convex. Therefore, any critical point of the divergence restricted to a convex set is a global minimizer. Let y ∈ Y. Then it suffices to show: If P satisfies the two conditional independence statements, then the marginal distribution P X of X is a critical point of D(P (·|y) Q) for Q restricted to C cl ( Z X ); for if this statement is true, then P y Z = P X , thus I π X (Y Z) = 0, and finally I red (X : Y ; Z) = 0. Let z, z ∈ Z. The derivative of D(P (·|y) Q) at Q = P X in the direction P (X|z) − P (X|z ) is x∈X (P (x|z) − P (x|z )) P (x|y) P (x) = x∈X P (x, z)P (x, y) P (x)P (y)P (z) − P (x, z )P (x, y) P (x)P (y)P (z ) .
Now, Y ⊥ ⊥ Z |X implies x∈X P (x, z)P (x, y) P (x) = P (y, z) and x∈X P (x, z )P (x, y) P (x) = P (y, z ), and Y ⊥ ⊥ Z implies P (y)P (z) = P (y, z) and P (y)P (z ) = P (y, z ). Together, this shows that P X is a critical point.