Introducing a differentiable measure of pointwise shared information

Abdullah Makkeh, Aaron J. Gutknecht, and Michael Wibral

Phys. Rev. E 103, 032149 – Published 25 March 2021

Abstract

Partial information decomposition of the multivariate mutual information describes the distinct ways in which a set of source variables contains information about a target variable. The groundbreaking work of Williams and Beer has shown that this decomposition cannot be determined from classic information theory without making additional assumptions, and several candidate measures have been proposed, often drawing on principles from related fields such as decision theory. None of these measures is differentiable with respect to the underlying probability mass function. We here present a measure that satisfies this property, emerges solely from information-theoretic principles, and has the form of a local mutual information. We show how the measure can be understood from the perspective of exclusions of probability mass, a principle that is foundational to the original definition of mutual information by Fano. Since our measure is well defined for individual realizations of random variables it lends itself, for example, to local learning in artificial neural networks. We also show that it has a meaningful Möbius inversion on a redundancy lattice and obeys a target chain rule. We give an operational interpretation of the measure based on the decisions that an agent should take if given only the shared information.

Received 30 April 2020
Revised 18 December 2020
Accepted 19 February 2021

DOI:https://doi.org/10.1103/PhysRevE.103.032149

Physics Subject Headings (PhySH)

Information theory

Statistical Physics & ThermodynamicsNetworks

Authors & Affiliations

Abdullah Makkeh ^*, Aaron J. Gutknecht^†, and Michael Wibral ^‡

Campus Institute for Dynamics of Biological Networks, Georg-August Univeristy, Goettingen, Germany

^*abdullah.alimakkeh@uni-goettingen.de
^†agutkne@uni-goettingen.de; also at MEG Unit, Brain Imaging Center, Goethe University, Frankfurt, Germany.
^‡michael.wibral@uni-goettingen.de

Article Text (Subscription Required)

Click to Expand

References (Subscription Required)

Click to Expand

Issue

Vol. 103, Iss. 3 — March 2021

Reuse & Permissions

Access Options

Author publication services for translation and copyediting assistance advertisement

Images

Figure 1
Depiction of deriving the local mutual information $i (t : s)$ by excluding the probability mass of the impossible event $\bar{s}$ after observing $s$ . (a) Two events $t, \bar{t}$ partition the sample space $Ω$ . (b) Two event partitions $s, \bar{s}$ of the source variable $S$ in the sample space $Ω$ . The occurrence of $s$ renders $\bar{s}$ impossible [red (dark gray) stripes]. (c) $t$ may intersect with $s$ (gray region) and $\bar{s}$ [red (dark gray) hashed region]. The relative size of the two intersections determines whether we obtain information or misinformation, i.e., whether $t$ becomes relatively more likely after considering $s$ , or not (d), considering the necessary rescaling of the probability measure (e). Note that if the gray region in (e) is larger (resp. smaller) than that in (a), then $s$ is informative (resp. misinformative) about $t$ since observing $s$ hints that $t$ is more (reps. less) likely to occur compared to an ignorant prior. Panel (f) shows why the misinformative exclusion $P (t \cap \bar{s})$ [intersection of red (dark gray) hashes with gray region] cannot be cleanly separated from the informative exclusion, $P (\bar{t} \cap \bar{s})$ [dotted outline in (c)], as stated already in [22]. This is because these overlaps appear together in a sum inside the logarithm, but this logarithm in turn guarantees the additivity of information terms. Thus the additivity of (mutual) information terms is incompatible with an additive separation of informative and misinformative exclusions inside the logarithms of the information measures.
Reuse & Permissions
Figure 2
Shared exclusions in the three-source variable case. Upper left: A sample space with three events $s_{1}$ , $s_{2}$ , $s_{3}$ from three source variables [their complements events are depicted in (4)]. For clarity, $t$ is not shown, but may arbitrarily intersect with any intersections or unions of $s_{i}$ . The remaining panels show the induced exclusions by different combinations of $a_{i}$ . These exclusions arise by taking the corresponding unions and intersections of sets. Which unions and intersections were taken can be deduced by the shapes of the remaining, nonexcluded regions. For (1)–(3) we show the shared exclusions for combination of singletons [(1) and (2)] and those of singletons and coalitions, such as the events of the collections (left) and the shared exclusions (right). For (4)–(7) we only show shared exclusions. The online version uses the additional, nonessential color-based mark-up of unions and intersections: An intersection exclusion is indicated by the mix of the individual colors, e.g., the ${1} {2}$ exclusion is ${\bar{s}}_{1} \cap {\bar{s}}_{2}$ and mixes red and blue to purple, and a union exclusion is indicated by a pattern of the individual colors, e.g., the ${1, 2}$ exclusion is ${\bar{s}}_{1} \cup {\bar{s}}_{2}$ and takes a red-blue pattern.
Reuse & Permissions
Figure 3
Worked example of $i_{\cap}^{sx}$ for the classical xor. Let $T = XOR (S_{1}, S_{2})$ and $S_{1}, S_{2} \in {0, 1}$ be independent uniformly distributed and consider the realization $(s_{1}, s_{2}, t) = (1, 1, 0)$ . (a), (b) The sample space $Ω$ and the realized event [gold (gray) frame]. (c) The exclusion of events induced by learning that $S_{1} = 1$ , i.e., ${\bar{s}}_{1} = {0}$ (gray). (d) Same for ${\bar{s}}_{2} = {0}$ . (e) The union of exclusions fully determines the event (1,1,0) and yields 1 bit of $i (t = 0 : s_{1} = 1, s_{2} = 1)$ . (f) The shared exclusions by ${\bar{s}}_{1} = {0}$ and ${\bar{s}}_{2} = {0}$ , i.e., ${\bar{s}}_{1} \cap {\bar{s}}_{2}$ exclude only (0,0,0). This is a misinformative exclusion, as it raises the probability of events that did not happen ( $t = 1$ ) relative to those that did happen ( $t = 0$ ) compared to the case of complete ignorance. (g) Learning about one full variable, i.e., obtaining the statement that ${\bar{s}}_{1} = {0}$ adds additional probability mass to the exclusion [green (light gray)]. The shared exclusion [red (dark gray)] and the additional unique exclusion [green (light gray)] induced by $s_{1}$ create an exclusion that is uninformative, i.e., the probabilities for $t = 0$ and 1 remain unchanged by learning $s_{1} = 1$ . At the level of the $π^{sx}$ atoms, the shared and the unique information atom cancel each other. (h) Lattice with $i_{\cap}^{sx}$ and $π^{sx}$ terms for this realization. Other realizations are equivalent by the symmetry of xor, thus, the averages yield the same numbers. Note that the necessity to cancel the negative shared information twice to obtain both $i (t = 0 : s_{1} = 1) = 0$ , and $i (t = 0 : s_{2} = 1) = 0$ results in a synergy $< 1$ bit. Also note that while adding the shared exclusion from (f) and the unique exclusions for $s_{1}$ and $s_{2}$ results in the full exclusion from (e), information atoms add differently due to the nonlinear transformation of excluded probability mass into information via $- {log}_{2} p (\cdot)$ [compare (h)].
Reuse & Permissions
Figure 4
Worked example of $i_{\cap}^{sx}$ for a four source-variables case. We evaluate the shared information $i_{\cap}^{sx} (t : a_{1}; a_{2})$ with $a_{1} = {1, 2},$ $a_{2} = {3, 4}$ , $s = (s_{1}, s_{2}, s_{3}, s_{4}) = (0, 0, 1, 0),$ and $t = Parity (s) = 1$ . (a) Sample space: The relevant event is marked by the blue (gray) outline. (b) Exclusions induced by the two collections of source realization indices $a_{1}$ [brown (dark gray)], $a_{2}$ [yellow (light gray)], and the shared exclusion relevant for $i_{\cap}^{sx}$ [gold (gray)]. After removing and rescaling, the probability for the target event that was actually realized, i.e., $t = 1$ , is reduced from $1 / 2$ to $3 / 7$ . Hence the shared exclusion leads to negative shared information. Hence, $π^{sx} (t : {1, 2} {3, 4}) = - 0.0145 bit .$
Reuse & Permissions
Figure 5
The family of mappings introduced in Proposition t11 that preserve the probability mass difference. Let $α$ be the top node of $A ([3]) .$ The orange (gray dotted) region is $α^{-},$ the set of children of $α$ . Each color depicts one mapping in the family based on some $γ \in α^{-}$ . The dark red (solid line) mapping is based on $γ_{1},$ the red mapping (dash-dotted line) is based on $γ_{2}$ , and the salmon (dotted line) mapping is based on $γ_{3} .$
Reuse & Permissions
Figure 6
Depiction of set differences corresponding to the probability mass difference $d_{1}$ introduced in Proposition t11 and shown in Fig. 5, for the sets from Fig. 2.
Reuse & Permissions

Physical Review E

covering statistical, nonlinear, biological, and soft matter physics