Unsupervised nested Dirichlet finite mixture model for clustering

Abstract

The Dirichlet distribution is widely used in the context of mixture models. Despite its flexibility, it still suffers from some limitations, such as its restrictive covariance matrix and its direct proportionality between its mean and variance. In this work, a generalization over the Dirichlet distribution, namely the Nested Dirichlet distribution, is introduced in the context of finite mixture model providing more flexibility and overcoming the mentioned drawbacks, thanks to its hierarchical structure. The model learning is based on the generalized expectation-maximization algorithm, where parameters are initialized with the method of moments and estimated through the iterative Newton-Raphson method. Moreover, the minimum message length criterion is proposed to determine the best number of components that describe the data clusters by the finite mixture model. The Nested Dirichlet distribution is proven to be part of the exponential family, which offers several advantages, such as the calculation of several probabilistic distances in closed forms. The performance of the Nested Dirichlet mixture model is compared to the Dirichlet mixture model, the generalized Dirichlet mixture model, and the Convolutional Neural Network as a deep learning network. The excellence of the powerful proposed framework is validated through this comparison via challenging datasets. The hierarchical feature of the model is applied to real-world challenging tasks such as hierarchical cluster analysis and hierarchical feature learning, showing a significant improvement in terms of accuracy.

Appendices

Appendix A: The connection between NDD and Dirichlet-tree distributions

The derivation of the PDF of both the NDD [63] and Dirichlet-tree distributions [56] is based on the distribution introduced by [24]. Both distributions are solely based on the Dirichlet distribution being represented as a finite stochastic process with more generalization to the covariance.

1.1 A.1 Derivation of NDD

The NDD can be viewed as a representation of the original variables in the Dirichlet distribution in addition to the nesting variables where each nesting follows a Dirichlet distribution as well. Therefore, the NDD can be directly derived from the PDF of the Dirichlet distribution. Thus, ($x_{(d+1)},x_{(d+2)},\ldots ,x_{(d+k)}$) is determined by ($x_1,x_2,\ldots ,x_d$).

Consequently,

$$\begin{aligned} {\begin{matrix} &{}P(x_1,x_2,\ldots ,x_{(d+K)})=\\ {} &{}P(x_1,x_2,\ldots ,x_d\ ) P(x_{(d+1)},x_{(d+2)},\ldots ,x_{(d+K)}\mid x_1,x_2,\ldots ,x_d)=\\ {} &{}P(x_1,x_2,\ldots ,x_d) \end{matrix}} \end{aligned}$$

(A1)

By denoting $X_0\sim \mathcal {D}(\alpha _{0})$ for the unnested variables (root node variables) and $X_k \quad \forall k>0$, where $X_k=\tilde{X}_kx_n+k$ given that $\tilde{X}_k\sim \mathcal {D}(\alpha _{k})$, the following expression can be written:

$$\begin{aligned} P(x_1,x_2,\ldots ,x_d)= & {} P(X_0,X_1,\ldots ,X_K)\nonumber \\= & {} P(X_0)P(X_1,\ldots ,X_K\mid X_0)\nonumber \\{} & {} \textit{Using chain-rule}\nonumber \\= & {} P(X_0)f(X_1\mid X_0) P(X_2\mid X_0,X_1)\nonumber \\{} & {} \dots P(X_K\mid X_0,X_1,\dots X_{K-1})\nonumber \\{} & {} \textit{Using above notations}\nonumber \\= & {} P(X_0)P(X_1\mid x_{d+1})P(X_2\mid x_{d+2})\nonumber \\{} & {} \dots P(X_K \mid x_{d+K}) \end{aligned}$$

(A2)

Due to the change in the variable,

$$\begin{aligned} {\begin{matrix} P(X_k\mid x_{d+k})&{}=P(\tilde{X}_kx_n+k\mid x_n+k)|\textbf{J}_{X_k \longrightarrow \tilde{X}_k}|\\ {} &{}=P(\tilde{X}_k)|\textbf{J}_{X_k \longrightarrow \tilde{X}_k}|\end{matrix}} \end{aligned}$$

(A3)

where $|\textbf{J}_{X_k \longrightarrow \tilde{X}_k}|$ is the determinant of Jacobian matrix derived in [23]. The Jacobian matrix is added as a result of the change of variable rule. According to [23],

$$\begin{aligned} |\textbf{J}_{X_k \longrightarrow \tilde{X}_k}|=\begin{bmatrix} \frac{1}{x_{d+k}} &{}0 &{} \cdots &{} 0 \\ 0 &{}\frac{1}{x_{d+k}} &{} \cdots &{} 0 \\ \vdots &{} \vdots &{}\ddots &{} \vdots \\ 0 &{}0 &{} \cdots &{} \frac{1}{x_{d+k}} \end{bmatrix}=\left( \frac{1}{x_{d+k}}\right) ^{|X_j|-1} \end{aligned}$$

(A4)

Therefore,

$$\begin{aligned} P(X_k\mid x_{n+k})=P(\tilde{X}_k)\left( \frac{1}{x_{d+k}}\right) ^{|X_j|-1} \end{aligned}$$

(A5)

$$\begin{aligned} {\begin{matrix} &{}P(x_1,x_2,\ldots ,x_n)=P(X_0)\prod _{k=1}^KP(\tilde{X}_k)\left( \frac{1}{x_{d+k}}\right) ^{|X_k|-1}\\ &{}=\frac{1}{B\left( A_0\right) }\left( \prod _{j \in I_0} x_j^{\alpha _j-1}\right) \left( \prod _{k=1}^K \frac{1}{B\left( A_k\right) }\left( \prod _{j \in I_k}\left( \frac{x_j}{x_{d+k}}\right) ^{\alpha _j-1}\right) {\frac{1}{x_{d+k}}}^{|x_k|-1}\right) \\ &{}=\left( \prod _{k=0}^K \frac{1}{B\left( A_k\right) }\right) \left( \prod _{j=1}^{d+K}\left( x_j^{\alpha _j-1}\right) \right) \left( \prod _{k=1}^K \frac{1}{x_{d+k}} ^{\sum _{j \in I_k}{\alpha _j-1}}\right) \\ &{}=\left( \prod _{k=0}^K \frac{1}{B\left( A_k\right) }\right) \left( \prod _{j=1}^{d+K}\left( x_j^{\alpha _j-1}\right) \right) \left( \prod _{k=1}^K \frac{1}{x_{d+k}} ^{\bar{A}_k-1}\right) \\ &{}=\left( \prod _{k=0}^K \frac{1}{B\left( A_k\right) }\right) \left( \prod _{j=1}^{d+K}\left( x_j^{\alpha _j-1}\right) \right) \left( \prod _{k=1}^K x_{d+k}^{1-\bar{A}_k}\right) \\ &{}=\left( \prod _{k=0}^K \frac{1}{B\left( A_k\right) }\right) \left( \prod _{j=1}^d\left( x_j^{\alpha _j-1}\right) \right) \left( \prod _{k=1}^K x_{d+k} ^{\alpha _{d+k}-\bar{A}_k}\right) \\ &{}=\frac{\left( \prod _{j=1}^d x_j{ }^{\alpha _j-1}\right) \left( \prod _{k=1}^K x_{d+k}{ }^{\alpha _{d+k}-\bar{A}_k}\right) }{\prod _{k=0}^K B\left( A_k\right) } \end{matrix}} \end{aligned}$$

(A6)

1.2 A.2 Derivation of Dirichlet-tree distribution

In [56], the function ($\delta$) is introduced to control the contribution of each branch to the associated nodes as illustrated in this section.

The probability for a certain sample (x) for a set of leaf probabilities $(p_1\dots p_d)$ can be formulated as:

$$\begin{aligned} P(x\mid p_1\dots p_d)=\prod _{j=1}^d{p_j}^{\delta (x-j)} \end{aligned}$$

(A7)

where $P(p_1\dots p_d\mid \alpha ) \sim \mathcal {D}(\alpha _{j})$.

Analogously, the leaf probabilities in the Dirichlet-tree distribution for a given tree T for K nodes and B branch weights and C branches can be represented as

$$\begin{aligned} P(x\mid B,T)=\prod _{k=1}^K\prod _{c=1}^Cb^{\delta _{kc}(x)}_{kc} \end{aligned}$$

(A8)

where,

$$\begin{aligned} b_{kc}=\frac{\sum _{j=1}^d\delta _{kc}p_j}{\sum _{j=1}^d\delta _{k}p_j} \end{aligned}$$

(A9)

and,

$$\begin{aligned} \delta _{kc}={\left\{ \begin{array}{ll} 1,&{} \text {if branch { kc} leads to { x}}\\ 0,&{} \text {otherwise} \end{array}\right. } \end{aligned}$$

(A10)

Similar to the NDD, the interior nodes are defining another Dirichlet distribution for the descendant branches as shown in (A5).

$$\begin{aligned} P(B\mid \alpha )=\prod _{k=1}^KP(b_j\mid \alpha ) \end{aligned}$$

(A11)

where $P(b_j\mid \alpha )\sim \mathcal {D}(\alpha _{kc})$.

By combining (A2) with (A5) and multiplying by the Jacobian introduced by [23] in [24] the PDF of the Dirichlet-tree distribution can be written as:

$$\begin{aligned} P(x_1\dots x_d \mid \alpha ,T)=\prod _{j=1}^dx^{\alpha _{parent{(j)}}-1}_j\prod _{k=1}^K\frac{\Gamma (\sum _{c=1}^C\alpha _{kc})}{\prod _{c=1}^C{\Gamma (\alpha _{kc})}}\left( \sum _{j=1}^d\sum _{c=1}^C\delta _{kc(j)}x_j\right) ^{\beta _{k}} \end{aligned}$$

(A12)

where,

$$\begin{aligned} \beta _{k}={\left\{ \begin{array}{ll} 0,&{} \text {if { k} is the root node}\\ \alpha _{parent(k)-\sum _{c=1}^C\alpha _{kc}},&{} \text {otherwise} \end{array}\right. } \end{aligned}$$

(A13)

Fig. 10

Tree structures of the NDD and Dirichlet-tree distribution

In (A7), the consideration of the tree is given from the original variables upwards, resulting in the consideration of all the variables in one equation. Therefore, instead of dealing with each nesting independently as shown in the NDD derivation, ($\delta _{kc}$) determines which iteration is accepted or removed depending on ($\varvec{\delta }$); a 3D matrix that considers nodes, branches, and original variables. Therefore, ($\varvec{\delta }$) needs to be defined for all the original variables in advance. Moreover, any change to the tree structure would need to modify the ($\varvec{\delta }$) matrix.

Appendix B: Dirichlet-tree Mixture Model

1.1 B.1 Derivation of DTMM

In order to formulate the Dirichlet-tree distribution PDF, the tree shown in Fig. 1 needs to be redefined as in Fig. 14, to show the similarity between the NDD and the Dirichlet-tree distribution. The PDF shown in (B14) is the same as the one used in (A12), where $\bar{\alpha }_j=\alpha _{parent(j)}$.

$$\begin{aligned} P(X\mid \alpha )=\prod _{j=1}^dx^{\bar{\alpha }_j-1}_j\prod _{k=1}^K\dfrac{\Gamma (\sum _{c}^C\alpha _{kc})}{\prod _{c=1}^C\Gamma (\alpha _{kc})}\left( \sum _{j=1}^d\sum _{c=1}^C\delta _{kc}(j)x_j\right) ^{\beta _{k}} \end{aligned}$$

(B14)

where ($\delta _{kc}$) and ($\beta _{k}$) are defined in Eqs. A10 and A13, respectively. Accordingly,

$$\begin{aligned} P(X\mid \theta )=\sum _{m=1}^{M}P(X\mid \vec {\alpha }_m)P(m), \end{aligned}$$

(B15)

where,

$$\begin{aligned} \theta =\{\vec {\alpha }_1\ldots ,\vec {\alpha }_M,P(1),\ldots ,P(M)\}, \end{aligned}$$

and,

$$\begin{aligned} \vec {\alpha }_m=(\alpha _{kcm},\ldots , \alpha _{KCm}). \end{aligned}$$

Thus, for (N) independent observations $\mathcal {X}=\{X_i,\ldots ,X_N\}$

$$\begin{aligned} P(\mathcal {X}\mid \theta )=\prod _{i=1}^{N} {\sum _{m=1}^{M}P(X_i\mid \vec {\alpha }_m)P(m)}. \end{aligned}$$

(B16)

Accordingly, the likelihood shown in (B17) can be formulated for the two conditions shown in (A13).

$$\begin{aligned} \mathcal {L}(\mathcal {X},\mathcal {Z}\mid \theta )=\sum _{i=1}^{N} \sum _{m=1}^{M} z_{im}\log {[P(X_i\mid \vec {\alpha }_m)P(m)]}. \end{aligned}$$

(B17)

Based on Fig. 14, for root nodes $(k=1)$:

$$\begin{aligned} \mathcal {L}(\mathcal {X},\mathcal {Z}\mid \theta )= & {} \sum _{i=1}^{N}\sum _{m=1}^{M}z_{im}\Bigg [\sum _{j=1}^d\Bigg [\bar{\alpha }_{j} \log {x_j}-\log {x_j} \nonumber \\{} & {} +\sum _{k=1}^K\Bigg [\log {\Gamma (\sum _{c=1}^C\alpha _{kc})}-\sum _{c=1}^C\log {\Gamma (\alpha _{kc})}\Bigg ]\Bigg ]\nonumber \\{} & {} +\log {P(m)}\Bigg ] \end{aligned}$$

(B18)

For non-root nodes $(k=[2,3])$:

$$\begin{aligned} \mathcal {L}(\mathcal {X},\mathcal {Z}\mid \theta )= & {} \sum _{i=1}^N\sum _{m=1}^Mz_{im}\Bigg [\sum _{j=1}^d\Bigg [ \bar{\alpha }_{j} \log {x_j}-\log {x_j} \nonumber \\{} & {} +\sum _{k=1}^K\Bigg [\log {\Gamma (\sum _{c=1}^C\alpha _{kc})}-\sum _{c=1}^C\log {\Gamma (\alpha _{kc})}\nonumber \\{} & {} +\bar{\alpha }_k\log {\Bigg (\sum _{j=1}^d\sum _{c=1}^C\delta _{kc}(j)x_j\Bigg )} \\{} & {} -\sum _{c=1}^C\alpha _{kc}\log {\Bigg (\sum _{j=1}^d\sum _{c=1}^C\delta _{kc}(j)x_j\Bigg )}\Bigg ]\Bigg ]\nonumber \\{} & {} +\log {P(m)}\Bigg ] \nonumber \end{aligned}$$

(B19)

From the above equations, the relation of the ($\alpha _{kc}$) and ($\bar{\alpha }_k$) can be observed, as $\alpha _{kc} = \bar{\alpha }_k$ at specific (k). This is determined by the tree structure, hence, the use of ($\delta _{kc}$) is important to define this relationship. For example, according to Fig. 14, $\alpha _{11} = \bar{\alpha }_2$. This makes each tree structure distinct from the other structures. Therefore, each structure requires its own model derivation. In this paper, the derivations are made with respect to the tree shown in Fig. 14. To estimate the parameters for the EM method, the Newton-Raphson method in (26) is used for the distribution parameters, while (25) is used to define the prior parameters. The parameter estimation for non-root nodes $(k=[2,3],c=[1,2])$ is required for ($\alpha _{kc}$) and ($\bar{\alpha }_k$) while using deterministic annealing extension:

$$\begin{aligned} \frac{\partial \Theta (\theta ^*,\theta ,T)}{\partial \alpha _{kcm}^{(t)}}= & {} \sum _{i=1}^Nz_{im}\Bigg [\log {x_{kc}}+\Psi (\sum _{c=1}^C\alpha _{kcm})\nonumber \\{} & {} -\Psi (\alpha _{kcm})\Bigg ] \end{aligned}$$

(B20)

$$\begin{aligned} \frac{\partial \Theta (\theta ^*,\theta ,T)}{\partial \bar{\alpha }_{km}^{(t)}}=\sum _{i=1}^Nz_{im}\left( \sum _{j=1}^d\sum _{c=1}^C\delta _{kc}(j)x_j\right) \end{aligned}$$

(B21)

while, the parameter estimation for non-root nodes $(k=1,c=[1,2])$ is only required for $\alpha _{kc}$ as there are no parent nodes.

$$\begin{aligned} \frac{\partial \Theta (\theta ^*,\theta ,T)}{\partial \alpha _{kcm}^{(t)}}=\sum _{i=1}^Nz_{im}\left[ \Psi (\sum _{c=1}^C\alpha _{kcm})-\Psi (\alpha _{kcm})\right] . \end{aligned}$$

(B22)

However, the parent node for k=[2,3] is the actual node for k=1. Therefore, (B21) and (B22) can be combined for $\alpha _{kc}: \forall k \in 1$. Thus:

$$\begin{aligned} \frac{\partial \Theta (\theta ^*,\theta ,T)}{\partial \alpha _{kcm}^{(t)}}:\forall k \in [2,3]= & {} \sum _{i=1}^Nz_{im}\Bigg [\log {x_{kc}}+\Psi (\sum _{c=1}^C\alpha _{kcm})\nonumber \\{} & {} -\Psi (\alpha _{kcm})\Bigg ] \end{aligned}$$

(B23)

$$\begin{aligned} \frac{\partial \Theta (\theta ^*,\theta ,T)}{\partial \alpha _{kcm}^{(t)}}:\forall k \in [1]= & {} \sum _{i=1}^Nz_{im}\Bigg [\Bigg (\sum _{j=1}^d\sum _{c=1}^C\delta _{kc}(j)x_j\Bigg ) \nonumber \\{} & {} +\Psi (\sum _{c=1}^C\alpha _{kcm})-\Psi (\alpha _{kcm})\Bigg ] \end{aligned}$$

(B24)

To define the Hessian matrix,

$$\begin{aligned} \frac{\partial ^2 \Theta (\theta ^*,\theta ,T)}{\partial \alpha _{kc_1m}^{(t)} \alpha _{kc_2m}^{(t)}} = \sum _{i=1}^Nz_{im}\left[ \Psi ^\prime (\sum _{c=1}^C\alpha _{kcm})\right] \end{aligned}$$

(B25)

$$\begin{aligned} \frac{\partial ^2 \Theta (\theta ^*,\theta ,T)}{\partial {\alpha _{kc_m}^{(t)}}^2} = \sum _{i=1}^Nz_{im}\left[ \Psi ^\prime (\sum _{c=1}^C\alpha _{kcm})-\Psi ^\prime (\alpha _{kcm})\right] \end{aligned}$$

(B26)

Finally, $\delta _{kc}(j)$ is a ($K\times C\times d$) matrix, that is defined based on the tree structure. Therefore, based on Fig. 14, $\delta _{kc}(j)$ is a ($3\times 2\times 4$) matrix, and can be written as:

$$\begin{aligned} \delta (1)=\begin{bmatrix} 1 &{} 0 \\ 1 &{} 0 \\ 0 &{} 0 \end{bmatrix} , \delta (2)=\begin{bmatrix} 1 &{} 0 \\ 0 &{} 1 \\ 0 &{} 0 \end{bmatrix} , \delta (3)=\begin{bmatrix} 0 &{} 1 \\ 0 &{} 0 \\ 1 &{} 0 \end{bmatrix} , \delta (4)=\begin{bmatrix} 0 &{} 1 \\ 0 &{} 0 \\ 0 &{} 1 \end{bmatrix} \end{aligned}$$

(B27)

1.2 B.2 Results verification

This section shows the results for the NDMM and DTMM compared to the DMM and the original values under the same initialization hyperparameters. In Table 13, it is shown that NDMM and DTMM gave the same results, which proves that they are similar to each other.

Table 12 Comparison between the yielded $\alpha$ parameters for the DMM, NDMM and DTMM

Appendix C: Proof of exponential properties of NDD

In this section, the pertinence of the NDD to the exponential family, along with its properties to use the Bhattacharyya kernel and Kullback-Leibler divergence.

1.1 C. 1 Proof of exponential pertinence

First, the exponential of the natural logarithm is taken for the PDF of the NDD as shown in (C28).

$$\begin{aligned} \begin{aligned} P(X\mid \theta )&=\exp {[\log P(X\mid \theta )]} \\&=\frac{\prod _{j=1}^d x_j^{\alpha _j-1} \prod _{k=1}^k x_{d+k}^{\alpha _{d+k}-\bar{A}_k}}{\prod _{k=0}^k \frac{\prod _{j \in k} \Gamma \left( \alpha _j\right) }{\Gamma \left( \sum _{j \in k_k}^{\alpha _j}\right) }} \end{aligned} \end{aligned}$$

(C28)

To verify that the distribution belongs to the exponential family, it needs to be written in the form shown in (C29):

$$\begin{aligned} P(X\mid \theta )=\exp {[A(X)+\theta ^TT(X)-k(\theta )]} \end{aligned}$$

(C29)

where A is the measure function, T is the sufficient statistics, and K is the cumulant generating function [36]. Therefore, by expanding (C28),

$$\begin{aligned} P(X \mid \theta )= & {} \exp \Bigg [\sum _{j=1}^d\alpha _j\log {x_j}-\sum _{j=1}^d x_j+\sum _{k=1}^K\alpha _{d+k}\log {x_{d+k}}\nonumber \\{} & {} -\sum _{k=1}^K\bar{A}_k\log {x_{d+k}}+\sum _{k=0}^K\log {\Gamma (\sum _{j\in k}\alpha _{j})}\nonumber \\{} & {} -\sum _{k=0}^K\sum _{j\in k}\log {\Gamma (\alpha _j)} \Bigg ]\nonumber \\= & {} EXP\Bigg [\sum _{j=1}^d\alpha _j\log {x_j}-\sum _{j=1}^d x_j+\sum _{k=1}^K\alpha _{d+k}\log {x_{d+k}}\nonumber \\{} & {} -\sum _{k=1}^K\sum _{j\in k}\alpha _{d+j}\log {x_{d+k}}+\sum _{k=0}^K\log {\Gamma (\sum _{j\in k}\alpha _{j})}\nonumber \\{} & {} -\sum _{k=0}^K\sum _{j\in k}\log {\Gamma (\alpha _j)} \Bigg ] \end{aligned}$$

(C30)

Finally, (C30) can be written as:

$$\begin{aligned}{} & {} A(X)=-\sum _{j=1}^dx_j\nonumber \\{} & {} K(\theta )=\sum _{k=0}^K\log {\Gamma (\sum _{j\in k}\alpha _{j})}-\sum _{k=0}^K\sum _{j\in k}\log {\Gamma (\alpha _j)}\nonumber \\{} & {} \theta =(\alpha _{1},\dots ,\alpha _{d},\alpha _{d+1},\dots , \alpha _{d+K},\bar{A}_1,\dots ,\bar{A}_K)\nonumber \\{} & {} \theta =(\alpha _{1},\dots ,\alpha _{d},\alpha _{d+1},\dots , \alpha _{d+K},\sum _{j\in k(1)}\alpha _{j}),\dots ,\sum _{j\in k(K)}\alpha _{j})\nonumber \\{} & {} T(X)=(\log {x_1},\dots ,\log {x_d},\log {x_{d+1}},\dots ,\log {x{d+K}},\nonumber \\{} & {} \log {x_{d+1}},\dots ,\log {x{d+K}}) \end{aligned}$$

(C31)

1.2 C.2 Bhattacharyya distance

$$\begin{aligned}{} & {} BH(P(X \mid \theta ),P(X \mid \theta ^\prime ))\nonumber \\{} & {} \quad =\exp {\left[ K\left( \dfrac{1}{2}\theta -\dfrac{1}{2}\theta ^\prime \right) -\dfrac{1}{2}K(\theta )-\dfrac{1}{2}K(\theta ^\prime )\right] } \end{aligned}$$

(C32)

$$\begin{aligned}{} & {} BH(P(X \mid \theta ),P(X \mid \theta ^\prime ))\nonumber \\= & {} \exp \left[ \right. \sum _{k=0}^K\log {\Gamma \left( \right. \sum _{j\in k}\frac{\alpha _j}{2}-\frac{{\alpha _j}^\prime }{2}}\left. \right) \left. \right] \nonumber \\{} & {} -\sum _{k=0}^K\sum _{j\in k}\log {\Gamma \left( \frac{\alpha _j}{2}-\frac{{\alpha _j}^\prime }{2}\right) }-\frac{1}{2}\sum _{k=0}^K\log {\Gamma (\sum _{j\in k}\alpha _j)}\nonumber \\{} & {} +\frac{1}{2}\sum _{k=0}^K\sum _{j\in k}\log {\Gamma (\alpha _j)}\nonumber \\{} & {} -\frac{1}{2}\sum _{k=0}^K\log {\Gamma (\sum _{j\in k}{\alpha _j}^\prime )}+\frac{1}{2}\sum _{k=0}^K\sum _{j\in k}\log {\Gamma ({\alpha _j}^\prime )}\nonumber \\= & {} \frac{\prod _{k=0}^K \Gamma \left( \sum _{j \in k} \frac{\alpha _j-{\alpha _j}^{\prime }}{2}\right) \sqrt{\prod _{k=0}^K \prod _{j \in k} \Gamma \left( \alpha _j\right) \Gamma \left( {\alpha _j}^{\prime }\right) }}{\prod _{k=0}^K\sum _{j\in k} \Gamma \left( \frac{\alpha _j-{\alpha _j}^{\prime }}{2}\right) \sqrt{\prod _{k=0}^K \Gamma \left( \sum _{j \in k}{\alpha _j}\right) \Gamma \left( \sum _{j \in k} {\alpha _j}^{\prime }\right) }} \end{aligned}$$

(C33)

1.3 Kullback-Leibler divergence

$$\begin{aligned}{} & {} KL(P(X \mid \theta ),P(X \mid \theta ^\prime ))\nonumber \\{} & {} \quad =\int {p(x)\log {\frac{p(x)}{q(x)}}}dx=\left\langle \log {\frac{p(x)}{q(x)}}\right\rangle _{p(x)}\nonumber \\{} & {} \quad =\left\langle \log {\frac{\left( \dfrac{\prod _{j=1}^{d} {x_j}^{\alpha _j-1} \prod _{k=1}^{K} {x_{d+k}}^{\alpha _{d+k}-\bar{A}_k}}{\prod _{k=0}^{K} Beta(A_k)}\right) }{\left( \dfrac{\prod _{j=1}^{d} {x_j}^{\beta _j-1} \prod _{k=1}^{K} {x_{d+k}}^{\beta _{d+k}-\bar{B}_k}}{\prod _{k=0}^{K} Beta(B_k)}\right) }}\right) \rangle _{p(x)} \end{aligned}$$

(C34)

By Expansion,

$$\begin{aligned}{} & {} KL(P(X \mid \theta ),P(X \mid \theta ^\prime ))\nonumber \\{} & {} \qquad =\Bigg [\sum _{j=1}^d\alpha _j\log {x_j}-\sum _{j=1}^d x_j+\sum _{k=1}^K\alpha _{d+k}\log {x_{d+k}}\nonumber \\{} & {} \quad -\sum _{k=1}^K\bar{A}_k\log {x_{d+k}}+\sum _{k=0}^K\log {\Gamma (\sum _{j\in k}\alpha _{j})}\nonumber \\{} & {} \quad -\sum _{k=0}^K\sum _{j\in k}\log {\Gamma (\alpha _j)} \Bigg ] \nonumber \\{} & {} \quad -\Bigg [\sum _{j=1}^d\beta _j\log {x_j}-\sum _{j=1}^d x_j+\sum _{k=1}^K\beta _{d+k}\log {x_{d+k}}\nonumber \\ \end{aligned}$$

$$\begin{aligned}{} & {} \quad -\sum _{k=1}^K\bar{B}_k\log {x_{d+k}}\nonumber \\{} & {} \quad +\sum _{k=0}^K\log {\Gamma (\sum _{j\in k}\beta _{j})}-\sum _{k=0}^K\sum _{j\in k}\log {\Gamma (\beta _j)} \Bigg ] \end{aligned}$$

(C35)

$$\begin{aligned} {\begin{matrix} KL(P(X \mid \theta ),P(X \mid \theta ^\prime ))&{}=\sum _{j=1}^d(\alpha _j-\beta _j)\langle \log {x_j}\rangle _{p(x)}\\ {} &{}+\sum _{k=1}^K(\alpha _{d+k}+\bar{B}_k-\beta _{d+k}-\bar{A}_k)\langle \log {x_{d+k}}\rangle _{p(x)}\\ {} &{}+\sum _{k=0}^K\left( \log {\Gamma (\sum _{j\in k}\alpha _{j})}-\log {\Gamma (\sum _{j\in k}\beta _{j})}\right) \\ {} &{}+\sum _{k=0}^K\sum _{j\in k}\left( \log {\Gamma (\beta _{j})}-\log {\Gamma (\alpha _{j})}\right) \end{matrix}} \end{aligned}$$

(C36)

where,

$$\begin{aligned} {\begin{matrix} &{}\langle \log {x_j}\rangle _{p(x)}=\Psi (\alpha _{j})-\Psi \left( \sum _{j=1}^d\alpha _j\right) \\ &{}\langle \log {x_{d+k}}\rangle _{p(x)}=\Psi (\alpha _{d+k})-\Psi \left( \sum _{k=1}^K\alpha _{d+k}\right) \end{matrix}} \end{aligned}$$

(C37)