Alternative Dirichlet Priors for Estimating Entropy via a Power Sum Functional

Abstract

Entropy is a functional of probability and is a measurement of information contained in a system; however, the practical problem of estimating entropy in applied settings remains a challenging and relevant problem. The Dirichlet prior is a popular choice in the Bayesian framework for estimation of entropy when considering a multinomial likelihood. In this work, previously unconsidered Dirichlet type priors are introduced and studied. These priors include a class of Dirichlet generators as well as a noncentral Dirichlet construction, and in both cases includes the usual Dirichlet as a special case. These considerations allow for flexible behaviour and can account for negative and positive correlation. Resultant estimators for a particular functional, the power sum, under these priors and assuming squared error loss, are derived and represented in terms of the product moments of the posterior. This representation facilitates closed-form estimators for the Tsallis entropy, and thus expedite computations of this generalised Shannon form. Select cases of these proposed priors are considered to investigate the impact and effect on the estimation of Tsallis entropy subject to different parameter scenarios.

1. Introduction

Shannon entropy and related information measures are functionals of probability and a measurement of information contained in a system that arise in information theory, machine learning and text modelling, amongst others. Ref. [1] discussed quantifying the information carried by neural signals to estimating the dependency structure and inferring causal relations, uncertainty and dispersion in statistics being applied in fields such as molecular biology. Other interests include studies measuring complexity of dynamics in physics, to studies measuring diversity in ecology and genetics, fields of coding theory and cryptography [2], financial analysis and data compression [3]. Numerous inferential tasks rely on data-driven procedures to estimate these quantities. In these settings and utilising the estimated quantities, researchers are often confronted with data arising from an unknown discrete distribution, and seek to estimate its entropy. This motivates sustained research interest within entropy, coupled with the current data-driven and computing-rich era, for practitioners.

Entropy estimation remains an openly discussed challenge. Ref. [4] investigated how the maximum likelihood estimator (MLE) performed. This is also referred to as the plug-in principle in functional estimation, where a point estimate of the parameter is used to build an estimate for a functional of the parameter. The classical asymptotic theory of MLEs does not adequately address high-dimensional settings in this current data-driven era [4]. High-dimensional statistics arguably demand theoretical tools to address the needs of these high-dimensional settings. Ref. [5] investigated 18 different estimation measures and the suitability was determined experimentally based on the bias and the mean squared error. This work takes a Bayesian approach to entropy estimation, building upon work by [1,4,6,7].

Multivariate count data constrained to add up to a certain constant are commonly modelled using the multinomial distribution. This is widely used in modelling categorical data, of which features could be for example, words in the case of textual documents or visual words in the case of images. The Dirichlet distribution, closely related to the probabilistic behaviour of the multinomial distribution, is a conjugate prior for the multinomial distribution when a Bayes perspective is of interest. Ref. [8] highlights how the use of prior distributions in a Bayesian framework makes it possible to work with very limited data sets. Ref. [9] underscores the superior performance by using the hierarchical approach which introduces the construction of the statistical model. Some meaningful studies include [1,4,6,10]. Ref. [11] also showed how using different Dirichlet distributions for the bivariate case gives one the opportunity to include prior information and expert opinion to obtain more realistic results in certain situations. Ref. [4] also experimented with the estimation of entropy and this triggered more exploration with alternative priors. Experimentation on diverse data sets might necessitate parameter-rich priors; therefore, this study proposes these alternative Dirichlet priors to address this potential challenge.

The paper illustrates how a Bayesian approach is applied in a multinomial-Dirichlet family setup, which allows us to obtain a posterior distribution from where explicit expressions for the Tsallis entropy can be derived, by particularly focussing on the Product moment for the power sum functional, and assuming squared error loss. The first of two main contributions of this paper is the addition of flexible priors from a Dirichlet family, utilising them within an information-theoretical world, which also allows for positive correlation in addition to the usual negative correlation characteristic. The second shows that using elegant constructs of the complete product moments of the posteriors, gives one the comparative advantage of obtaining explicit estimators for entropy under these Dirichlet priors. Ref. [8] echos how the computation on moments accelerates the estimation of entropy.

The paper is outlined as follows. In Section 2, the essential components that are used in the paper are outlined. In Section 3, alternative Dirichlet priors will be introduced and studied, as candidates for the Bayesian analysis of entropy. In Section 4, analytical expressions for the entropy expressions under consideration will be derived and studied. Section 5 contains conclusions and final thoughts.

2. Essential Components

The countably discrete model under consideration in this paper is given by the well-motivated multinomial distribution. A discrete random variable $\mathbf{X}=(X_1,\cdots ,X_K)$ follows the multinomial distribution of order K (i.e., with K distinct classes of interest) with parameters $\mathbf{p}=(p_1,p_2,\cdots ,p_K)$ and $n>0$ if its probability mass function (pmf) is given by

$$f\left(\mathbf{x}\right|\mathbf{p})=\frac{n!}{{\prod }_{i=1}^K{x}_i!(n-{\sum }_{i=1}^K{x}_i)}\prod _{i=1}^K{p}_i^{x_i}{(1-\sum _{i=1}^K{p}_i)}^{n-{\sum }_{i=1}^K{x}_i}.$$

(1)

The Dirichlet distribution (of type 1, see [12]) of order $K\ge 2$ and parameters $\Pi =({\pi }_1,{\pi }_2,\cdots ,{\pi }_{K+1})$ for ${\pi }_i>0,i=1,\cdots ,k+1$, has a probability density function (pdf) with respect to the Lebesgue measure on the Euclidean space ${\mathbb{R}}^K$ given by

$$h(\mathbf{p};\Pi )=\frac{{\prod }_{i=1}^{k+1}\Gamma \left({\pi }_i\right)}{\Gamma \left({\sum }_{i=1}^{k+1}{\pi }_i\right)}\prod _{i=1}^{K+1}{p}_i^{{\pi }_i-1}$$

(2)

on the K dimensional simplex, defined by   

$$\begin{array}{c}\hfill p_1,p_2,\cdots ,p_K>0\\ \hfill p_1+p_2+\cdots +p_K<1\\ \hfill p_{K+1}=1-p_1-\cdots -p_K,\end{array}$$

and where $\Gamma (·)$ denotes the usual gamma function (the space and constraints of this K dimensional simplex is denoted by $\mathcal{A}$).

To derive a Bayesian engine, we need the likelihood function $f\left(\mathbf{x}\right|\mathbf{p})$ in addition to a suitable prior distribution $h\left(\mathbf{p}\right)$. The fundamental relationship between the likelihood function and the prior distribution to form the posterior distribution $f\left(\mathbf{p}\right|\mathbf{x})$ is given by

$$f\left(\mathbf{p}\right|\mathbf{x})=\frac{f\left(\mathbf{x}\right|\mathbf{p}\left)h\right(\mathbf{p})}{\int f\left(\mathbf{x}\right|\mathbf{p}\left)h\right(\mathbf{p})d\mathbf{p}}.$$

The most popular form of entropy is that of Shannon:

$$H\left(P\right)=\sum _{i=1}^{K+1}-p_{i}ln{p}_i.$$

Various generalised cases of this entropy exist, which relies on the power sum:

$$F_{\alpha }\left(P\right)=\sum _{i=1}^{K+1}{p}_i^{\alpha }$$

(3)

where $\alpha >0$. The power sum functional occurs in various operational problems ([4]). Under the assumption of squared error loss within Bayes estimation, the estimates of both these quantities is given by their expected values:

$$E\left(H\left(P\right)\right)=E(\sum _{i=1}^{K+1}-p_{i}ln{p}_i)$$

and

$${\widehat{F}}_{\alpha }\left(P\right)=E\left(F_{\alpha }\left(P\right)\right)=E(\sum _{i=1}^{K+1}{p}_i^{\alpha })=\sum _{i=1}^{K+1}E\left(p_i^{\alpha }\right).$$

(4)

Thus, it is of value to consider the expected value of $p_i^{\alpha }$ for all values of i.

Since there are cases, such as the non-extensive system like alignment processing, namely registration, which has complex behaviours associated with the phenomena of radar-imaging systems [13])which cannot be fully explained by Shannon entropy, other generalized forms were designed. The Tsallis entropy considered in this paper, which is a popular generalised entropy, tends to Shannon entropy as $\alpha$ tends to 1 [14] and is given by

$$T=\frac{{\sum }_{i=1}^{K+1}{p}_i^{\alpha }-1}{1-\alpha };\alpha \ge 0,\alpha \ne 1.$$

(5)

The estimate of this generalisation can be written in terms of the estimate of the power sum:

$$E\left(T\right)=E\left(\frac{{\sum }_{j=1}^{K+1}{p}_j^{\alpha }-1}{1-\alpha }\right)=\frac{{\widehat{F}}_{\alpha }\left(p\right)-1}{1-\alpha }.$$

Since the power sum is easier to estimate than the Shannon entropy, the power sum is used in our case. We consider the estimate as the expectation under the posterior distribution, thus under squared-error loss.

3. Alternative Dirichlet Priors

In this section, two previously unconsidered Dirichlet priors, namely the Dirichlet generator prior and the noncentral Dirichlet prior will be proposed. Positive correlation can be observed for special cases of the Dirichlet generator prior, which is a benefit of this generator form. These new contributions add to the field of generative models for count data and have not been previously considered for entropy.

3.1. Dirichlet Generator Prior

In this section, Dirichlet generator distributions are proposed as alternative candidates. From this form, numerous flexible candidates can be “generated”.

Definition 1.

Suppose $\mathbf{p}$ is Dirichlet-generator distributed. Then, its pdf is given by

$$h(p_1,\cdots ,p_K;\Pi )=C{p}_1^{{\pi }_1-1}{p}_2^{{\pi }_2-1}\cdots p_K^{{\pi }_K-1}{(1-\sum _{i=1}^K{p}_i)}^{{\pi }_{K+1}-1}g(\theta \sum _{i=1}^K{p}_i)$$

(6)

with C a normalising constant such that

$$\begin{array}{ccc}\hfill C^{-1}& =& \underset{\mathcal{A}}{\int \cdots \int }{p}_1^{{\pi }_1-1}{p}_2^{{\pi }_2-1}\cdots p_K^{{\pi }_K-1}{(1-\sum _{i=1}^K{p}_i)}^{{\pi }_{K+1}-1}g(\theta \sum _{i=1}^K{p}_i)d\mathbf{p}.\hfill \end{array}$$

(7)

The vector $\mathbf{p}\in \mathcal{A}$ is thus a Dirichlet generator variate with parameters $\Pi =({\pi }_1,\cdots ,{\pi }_{K+1})$, $\theta \in \mathbb{R}$, and whichever additional parameters $g(·)$ imposed, which ensures that the pdf $h(·)$ is non-negative. The following conditions also apply:

(1)

$g(·)$ is a Borel-measurable function;

(2)

$g(·)$ admits a Taylor series expansion;

(3)

$g\left(0\right)=1$.

The usual Dirichlet distribution with pdf (2) is thus a special case of (6) when $\theta =0$.

For illustration of the implementation of the Dirichlet generator prior, we focus on

$$\begin{array}{ccc}\hfill g(\theta \sum _{i=1}^K{p}_i)& =& {}_r{F}_q(a_1,\cdots ,a_r;b_1,\cdots ,b_q;\theta \sum _{i=1}^K{p}_i)\hfill \\ & =& \sum _{n=0}^{\infty }\frac{{\left(a_1\right)}_n\cdots {\left(a_r\right)}_n}{{\left(b_1\right)}_n\cdots {\left(b_q\right)}_n}\frac{{\left(\theta {\sum }_{i=1}^K{p}_i\right)}^n}{n!}\hfill \end{array}$$

where ${}_p{F}_q(·)$ denotes the generalised hypergeometric function (see [15]) and ${\left(a\right)}_k=\frac{\Gamma (a+k)}{\Gamma \left(a\right)}$ is the Pochhammer function.

The prior distribution (6) will then take on the following form with pdf

$$\begin{array}{ccc}\hfill h(p_1,\cdots ,p_k;\Pi )& =& C_{*}^{-1}\frac{\Gamma \left({\sum }_{j=1}^{k+1}{\pi }_j\right)}{{\prod }_{j=1}^{k+1}\Gamma \left({\pi }_j\right)}{p}_1^{{\pi }_1-1}\cdots ,p_k^{{\pi }_k-1}{\left(1-\sum _{j=1}^k{p}_j\right)}^{{\pi }_{k+1}-1}\hfill \\ & \times & {}_r{F}_q\left(a_1,\cdots ,a_r,b_1,\cdots ,b_q;\theta \sum _{j=1}^k{p}_j\right)\hfill \end{array}$$

(8)

where $C_{*}$ is equal to

$$\begin{array}{c}\hfill {}_{r+1}{F}_{q+1}\left(a_1,\cdots ,a_r,\sum _{j=1}^k{\pi }_j;b_1,\cdots ,b_q,\sum _{j=1}^{k+1}{\pi }_j;\theta \right).\end{array}$$

(9)

In this paper, three hypergeometric functions are considered (${}_0{F}_0$; ${}_0{F}_1{\mathrm{and}}_1{F}_1$), since these are commonly considered functions representing exponential, binomial, and the confluent hypergeometric functions. For illustrative investigation, bivariate observations from the corresponding distributions were simulated using Algorithm 1 and the associated pdfs are overlaid and presented in Figure 1, Figure 2 and Figure 3. The data were simulated from (8) using the following steps of the Acceptance/Rejection method:

Algorithm 1 Acceptance/Rejection method

1 Define $y_i\in (0,1)$ of size n for $i=1,2$; 2 Calculate the pdf of the Dirichlet (2) $h(y_1,y_2)$ for $y_1+y_2<1$; 3 Obtain $m=max\left(h(y_1,y_2)\right)$; 4 Simulate $p_i\sim Unif(0,1)$ of size n for $i=1,2$; 5 Calculate the pdf of the Dirichlet generator (8) $h(p_1,p_2)$ for $p_1+p_2<1$; 6 Simulate $z\sim Unif(0,1)$ of size n; 7 If $\frac{h(p_1,p_2)}{m}>z$, then keep $(p_1,p_2)$, else return to Step 4; 8 Repeat steps 4–7 k times.

Figure 1 and Figure 2 illustrate the three chosen hypergeometric functions for two choices of $\theta$ and for three different sets of $\Pi$s if $K=2$ with $a_1=4$ and $b_1=5$ for ${}_0{F}_1{\mathrm{and}}_1{F}_1$, respectively. This firstly illustrates the difference between the different hypergeometric candidates as well as the effect a change in ${\pi }_1$ has on these three functions (note that a symmetric observation would be made for ${\pi }_2$). The difference between Figure 1 and Figure 2 shows the effect that $\theta$ has on these different combinations with Figure 1 having a very small (almost negligible) $\theta$, while Figure 2 increases the value of $\theta$. An increase in ${\pi }_1$ results in a more highly dense concentration of the pdf for corresponding values of $p_1$ and $p_2$. This is observed for all three considered hypergeometric candidates, as seen in Figure 1 and Figure 2, also for an increase in $\theta$. For Figure 3, a single set of $\Pi$s were selected with $\theta =0.1$ to showcase the effect that the parameters a and b of the hypergeometric function have on the ${}_1{F}_0$ and ${}_1{F}_1$ functions. As a increases, an increased mass is observed closer to the restriction $p_1+p_2<1$ while an increase in b results in a lower pdf volume.

Next, the posterior distribution is derived, assuming the Dirichlet generator prior (8) together with a multinomial likelihood (1).

Theorem 1.

Suppose the likelihood function is given by (1) and the prior distribution for $\mathbf{p}$ is given by (8). Then, the pdf of the posterior distribution is given by

$$\begin{array}{ccc}\hfill f\left(\mathbf{p}\right|\mathbf{x})& \propto & p_1^{{\pi }_1+x_1-1}{p}_2^{{\pi }_2+x_2-1}\cdots p_K^{{\pi }_K+x_K-1}{(1-\sum _{i=1}^K{p}_i)}^{{\pi }_{K+1}+x_{K-1}-1}g(\theta \sum _{i=1}^K{p}_i)\hfill \end{array}$$

(10)

which is identifiable as a Dirichlet generator distribution with parameters $({\pi }_1+x_1,\cdots ,{\pi }_K+x_K,{\pi }_{K+1}+x_{K+1})$.

The complete product moment of the Dirichlet generator posterior (10) is of interest for the power sum (4), thus we are interested in $E(p_1^{k_1}{p}_2^{k_2}\cdots p_k^{k_k}{p}_{k+1}^{k_{k+1}})$.

Theorem 2.

Suppose that $\mathbf{p}|\mathbf{x}$ follows a Dirichlet generator posterior distribution with pdf given in (10). Then, the complete product moment is given by   

$$\begin{array}{ccc}& & E(p_1^{k_1}{p}_2^{k_2}\cdots p_k^{k_k}{p}_{k+1}^{k_{k+1}})\hfill \\ & =& \frac{{({\pi }_1+x_1)}_{k_1}{({\pi }_2+x_2)}_{k_2}\cdots {({\pi }_{k+1}+x_{k+1})}_{k_{k+1}}}{{({\pi }_1+x_1+\cdots +{\pi }_{k+1}+x_{k+1})}_{k_1+\cdots +k_{k+1}}}\hfill \\ & \times & \frac{{}_{r+1}{F}_{q+1}(a_1,\cdots ,a_r,{\sum }_{i=1}^k({\pi }_i+x_i+k_i);b_1,\cdots ,b_q,{\sum }_{i=1}^{k+1}({\pi }_i+x_i+k_i);\theta )}{{}_{r+1}{F}_{q+1}(a_1,\cdots ,a_r,{\sum }_{i=1}^k({\pi }_i+x_i);b_1,\cdots ,b_q,{\sum }_{i=1}^{k+1}({\pi }_i+x_i);\theta )}.\hfill \end{array}$$

(11)

Special cases of the above expression include setting $k_{k+1}=0$ to obtain an expression for the usual product moment of the Dirichlet generator distribution under investigation in this paper.

Proof. 

See Appendix A for the proof.    □

The product moment can then be used to investigate the correlation for the examples as illustrated in this section. Figure 4 displays the correlation for a range of $\theta$ values using (8) and the special cases. It is important to note the positive correlation obtained by the introduction of $g(·)=$${}_r{F}_q(·)$, which is a major benefit of using these alternative Dirichlet priors.

3.2. Noncentral Dirichlet Prior

In this section, a noncentral Dirichlet distribution will be constructed via the use of Poisson weights. Ref. [16] explored the use of a compounding method as a distributional building tool to obtain bivariate noncentral distributions and showed how this form of the distribution isolated the noncentrality parameter by retaining them in a Poisson probability form and hence introducing mathematical convenience. Ref. [17] extended on this work by introducing new bivariate gamma distributions emanating from a scale mixture of normal class.

Theorem 3.

Suppose $\mathbf{p}$ is Dirichlet distributed with pdf given by (2). Then, a noncentral Dirichlet distribution can be constructed in the following manner:

$$\begin{array}{ccc}& & h(\mathbf{p};\Pi ,\Lambda )\hfill \\ & =& \sum _{j_1=1}^{\infty }\cdots \sum _{j_{K+1}=1}^{\infty }\frac{exp\left(\frac{{\lambda }_1}{2}\right){\left(\frac{{\lambda }_1}{2}\right)}^{j_1}}{j_1!}\cdots \frac{exp\left(\frac{{\lambda }_K}{2}\right){\left(\frac{{\lambda }_K}{2}\right)}^{j_K}}{j_K!}\frac{exp\left(\frac{{\lambda }_{K+1}}{2}\right){\left(\frac{{\lambda }_{K+1}}{2}\right)}^{j_{K+1}}}{j_{K+1}!}h(\mathbf{p};\Pi |j_1,\cdots ,j_{K+1})\hfill \\ & =& \sum _{j_1=1}^{\infty }\cdots \sum _{j_{K+1}=1}^{\infty }\frac{exp\left(\frac{{\lambda }_1}{2}\right){\left(\frac{{\lambda }_1}{2}\right)}^{j_1}}{j_1!}\cdots \frac{exp\left(\frac{{\lambda }_K}{2}\right){\left(\frac{{\lambda }_K}{2}\right)}^{j_K}}{j_K!}\frac{exp\left(\frac{{\lambda }_{K+1}}{2}\right){\left(\frac{{\lambda }_{K+1}}{2}\right)}^{j_{K+1}}}{j_{K+1}!}\hfill \\ & \times & \frac{\Gamma ({\pi }_1+j_1+\cdots +{\pi }_K+j_K+{\pi }_{K+1}+j_{K+1})}{\Gamma ({\pi }_1+j_1)\Gamma ({\pi }_K+j_K)\Gamma ({\pi }_{K+1}+j_{K+1})}{p}_1^{{\pi }_1+j_1-1}\cdots p_K^{{\pi }_K+j_K-1}{(1-\sum _{i=1}^K{p}_i)}^{{\pi }_{K+1}+j_{K+1}-1}\hfill \end{array}$$

(12)

where $h\left(\mathbf{p}\right|j_1,\cdots ,j_{K+1})$ denotes the conditional (central) Dirichlet distribution (see (2)) with parameters ${\Pi }_{*}=({\pi }_1+j_1,\cdots ,{\pi }_{K+1}+j_{K+1})$, and Λ denotes the vector of noncentral parameters $({\lambda }_1,\cdots ,{\lambda }_K,{\lambda }_{K+1})$ with ${\lambda }_i>0\forall i$. After simplification (12) reflects

$$\begin{array}{ccc}& & h(\mathbf{p};\Pi ,\Lambda )\hfill \\ & =& h(\mathbf{p};\Pi )exp(-\sum _{i=1}^{K+1}\frac{{\lambda }_i}{2})\hfill \\ & \times & \sum _{\varphi }\frac{{({\pi }_1+\cdots +{\pi }_{K+1})}_{j_1+\cdots +j_{K+1}}}{{\left({\pi }_1\right)}_{j_1}\cdots {\left({\pi }_{K+1}\right)}_{j_{K+1}}{j}_1!\cdots j_{K+1}!}{\left(\frac{{\lambda }_1}{2}{p}_1\right)}^{j_1}\cdots {\left(\frac{{\lambda }_K}{2}{p}_K\right)}^{j_K}{\left(\frac{{\lambda }_{K+1}}{2}(1-\sum _{i=1}^K{p}_i)\right)}^{j_{K+1}}\hfill \end{array}$$

(13)

where $h(\mathbf{p};\Pi )$ denotes the (unconditional) Dirichlet distribution (see (2)) with parameter Π and where ${\sum }_{\varphi }={\sum }_{j_1=1}^{\infty }\cdots {\sum }_{j_{K+1}=1}^{\infty }$ .

Remark 1.

The pdf in equation (13) reflects a parametrization of the noncentral Dirichlet distribution of [12] and can be represented via the confluent hypergeometric function of several variables:

$$\begin{array}{ccc}& & f(\mathbf{p};\Lambda )\hfill \\ & =& h(\mathbf{p};\Pi )exp(-\sum _{i=1}^{K+1}\frac{{\lambda }_i}{2})\hfill \\ & & \times {\Psi }_2^{(K+1)}(\sum _{i=1}^{K+1}{\pi }_i;{\pi }_1,\cdots ,{\pi }_{K+1};\frac{{\lambda }_1}{2}{p}_1,\cdots ,\frac{{\lambda }_K}{2}{p}_K,\frac{{\lambda }_{K+1}}{2}(1-\sum _{i=1}^K{p}_i))\hfill \end{array}$$

where

$$\begin{array}{ccc}& & {\Psi }_2^{(K+1)}(\sum _{i=1}^{K+1}{\pi }_i;{\pi }_1,\cdots ,{\pi }_{K+1};\frac{{\lambda }_1}{2}{p}_1,\cdots ,\frac{{\lambda }_K}{2}{p}_K,\frac{{\lambda }_{K+1}}{2}(1-\sum _{i=1}^K{p}_i))\hfill \\ & =& \sum _{\varphi }\frac{{({\pi }_1+\cdots +{\pi }_{K+1})}_{j_1+\cdots +j_{K+1}}}{{\left({\pi }_1\right)}_{j_1}\cdots {\left({\pi }_{K+1}\right)}_{j_{K+1}}{j}_1!\cdots j_{K+1}!}{\left(\frac{{\lambda }_1}{2}{p}_1\right)}^{j_1}\cdots {\left(\frac{{\lambda }_K}{2}{p}_K\right)}^{j_K}{(\frac{{\lambda }_{K+1}}{2}(1-\sum _{i=1}^K{p}_i))}^{j_{K+1}}.\hfill \end{array}$$

In particular, when ${\lambda }_1={\lambda }_2=\cdots ={\lambda }_K={\lambda }_{K+1}=0$, see that

$$\begin{array}{ccc}& & h(\mathbf{p};\Pi )exp\left(0\right){\Psi }_2^{(K+1)}(\sum _{i=1}^{K+1}{\pi }_i;{\pi }_1,\cdots ,{\pi }_{K+1};0,\cdots ,0,0)\hfill \\ & =& h(\mathbf{p};\Pi )\hfill \end{array}$$

which illustrates that the model in (12) reduces to the usual (central) Dirichlet model in (2) when the noncentrality parameters are equal to 0. The model in (12) is thus the multivariate analogue of the doubly noncentral beta distribution (see [18]). In the case when ${\Psi }_2^{\left(K\right)}({\sum }_{i=1}^K{\pi }_i;{\pi }_1,\cdots ,$${\pi }_K;\frac{{\lambda }_1}{2}{p}_1,\cdots ,\frac{{\lambda }_K}{2}{p}_K)$ is considered in (12), this would represent the multivariate analogue of the singly noncentral beta distribution of [18].

Bivariate observations from the corresponding distributions were simulated using Algorithm 1 and the associated pdfs are overlaid and presented in in Figure 5 for different values of ${\lambda }_1$ and three combinations of $\Pi$s. These results showcase the effect that ${\lambda }_1$ has on these three different functions. Figure 5 clearly demonstrates the movement of the centroid of the contour plot.

Next, the posterior distribution is derived, assuming the noncentral Dirichlet prior (12) together with a multinomial likelihood (1).

Theorem 4.

Suppose the likelihood function is given by (1) and the prior distribution for $\mathbf{p}$ is given by (12). Then, the posterior distribution has pdf

$$\begin{array}{ccc}\hfill f\left(\mathbf{p}\right|\mathbf{x})& \propto & \sum _{\varphi }\frac{{({\pi }_1+\cdots +{\pi }_{k+1})}_{j_1+\cdots +j_{k+1}}}{{\left({\pi }_1\right)}_{j_1}\cdots {\left({\pi }_{k+1}\right)}_{j_{k+1}}{j}_1!\cdots j_{k+1}!}{\left(\frac{{\lambda }_1}{2}{p}_1\right)}^{j_1}\cdots {\left(\frac{{\lambda }_{k+1}}{2}{p}_{K+1}\right)}^{j_{k+1}}\hfill \\ & \times & \left[\frac{\Gamma \left({\sum }_{j=1}^{k+1}{\pi }_j+x_j\right)}{{\prod }_{j=1}^{K+1}\Gamma ({\pi }_j+x_j)}\right]p_1^{{\pi }_1+x_i-1}\cdots p_{k+1}^{{\pi }_{k+1}+x_{k+1}-1}\hfill \end{array}$$

(14)

which can be identified as a noncentral Dirichlet distribution with parameters $({\pi }_1+x_1,\cdots ,{\pi }_K+x_K,{\pi }_{K+1}+x_{K+1})$ and Λ.

Remark 2.

See that (14) can be represented using the confluent hypergeometric function from Remark 1 as

$$\begin{array}{ccc}& & f\left(\mathbf{p}\right|\mathbf{x};\Lambda )\hfill \\ & =& \frac{{\Psi }_2^{(K+1)}({\sum }_{i=1}^{K+1}{\pi }_i;{\pi }_1,\cdots ,{\pi }_{K+1};\frac{{\lambda }_1}{2}{p}_1,\cdots ,\frac{{\lambda }_K}{2}{p}_K,\frac{{\lambda }_{K+1}}{2}(1-{\sum }_{i=1}^K{p}_i))}{{\Psi }_2^{(K+1)}({\sum }_{i=1}^{K+1}{\pi }_i;{\pi }_1,\cdots ,{\pi }_{K+1};\frac{{\lambda }_1}{2},\cdots ,\frac{{\lambda }_K}{2},\frac{{\lambda }_{K+1}}{2})\frac{{\prod }_{j=1}^{K+1}\Gamma ({\pi }_j+x_j+j_j)}{\Gamma \left({\sum }_{j=1}^{k+1}{\pi }_j+x_j+j_j\right)}}\hfill \\ & \times & p_1^{{\pi }_1+x_i-1}\cdots p_{k+1}^{{\pi }_{k+1}+x_{k+1}-1}.\hfill \end{array}$$

The complete product moment of the noncentral Dirichlet posterior is of interest for the power sum, thus we are interested in $E(p_1^{k_1}{p}_2^{k_2}\cdots p_k^{k_k}{p}_{k+1}^{k_{k+1}})$.

Theorem 5.

Suppose that $\mathbf{p}|\mathbf{x}$ follows a noncentral Dirichlet distribution with pdf given in (14). Then, the complete product moment is given by

$$\begin{array}{ccc}& & E(p_1^{k_1}{p}_2^{k_2}\cdots p_k^{k_k}{p}_{k+1}^{k_{k+1}})\hfill \\ & =& \frac{{\sum }_{\varphi }\frac{{({\pi }_1+\cdots +{\pi }_{K+1})}_{j_1+\cdots +j_{K+1}}}{{\left({\pi }_1\right)}_{j_1}\cdots {\left({\pi }_{K+1}\right)}_{j_{K+1}}{j}_1!\cdots j_{K+1}!}{\left(\frac{{\lambda }_1}{2}\right)}^{j_1}\cdots {\left(\frac{{\lambda }_{K+1}}{2}\right)}^{j_{K+1}}\left[\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+j_i+k_i)}{\Gamma ({\sum }_{i=1}^{K+1}({\pi }_i+x_i+j_i+k_i)}\right]}{{\sum }_{{\varphi }^{*}}\frac{{({\pi }_1+\cdots +{\pi }_{K+1})}_{j_1+\cdots +j_{K+1}}}{{\left({\pi }_1\right)}_{j_1}\cdots {\left({\pi }_{K+1}\right)}_{j_{K+1}}{j}_1!\cdots j_{K+1}!}{\left(\frac{{\lambda }_1}{2}\right)}^{j_1}\cdots {\left(\frac{{\lambda }_{K+1}}{2}\right)}^{j_{K+1}}\left[\frac{{\prod }_{i=1}^{K+1}\Gamma ({\pi }_i+x_i+j_i)}{\Gamma ({\sum }_{i=1}^{K+1}({\pi }_i+x_i+j_i)}\right]}.\hfill \end{array}$$

(15)

Proof. 

See Appendix B for the proof.    □

4. Entropy Estimates

In this section, the Bayesian estimators (16) and (17) based on the posterior distributions (10) and (14) are derived for the power sum (3).

4.1. Dirichlet Generator Prior

Assuming the Dirichlet generator prior, the posterior distribution is given by (10). Using the complete product moments derived in (11), the Bayesian estimator for the power sum (3) can be derived by setting $k_i=\alpha$ with $i=1,\cdots ,k+1$ and $k_{\ne i}=0$.

Theorem 6.

Using (11), the Bayesian estimator for the power sum entropy under the Dirichlet generator posterior (10) is given by:

$$\begin{array}{ccc}& & {\widehat{F}}_{\alpha }\left(p\right)\hfill \\ & =& \sum _{j^{*}=1}^{K+1}\frac{\Gamma ({\pi }_{j^{*}}+x_{j^{*}}+\alpha ){\prod }_{j\ne j^{*}}^{K+1}\Gamma ({\pi }_j+x_j)\Gamma ({\sum }_{j=1}^{K+1}{\pi }_j+x_j)}{\Gamma ({\sum }_{j=1}^{K+1}{\pi }_j+x_j+\alpha ){\prod }_{j=1}^{K+1}({\pi }_j+x_j)}\hfill \\ & \times & \frac{{}_{r+1}{F}_{q+1}\left(a_1,\cdots ,a_r,\alpha +\sum _{j=1}^K{\pi }_j+x_j,b_1,\cdots ,b_q,\alpha +\sum _{j=1}^{K+1}{\pi }_j+x_j;\theta \right)}{{}_{r+1}{F}_{q+1}\left(a_1,\cdots ,a_r,\sum _{j=1}^K{\pi }_j+x_j,b_1,\cdots ,b_q,\sum _{j=1}^{K+1}{\pi }_j+x_j;\theta \right)}\hfill \\ & =& \sum _{j^{*}=1}^{K+1}\left(\frac{\frac{\Gamma \left(\sum _{j=1}^{K+1}{\pi }_j+x_j\right){\left(\prod _{j=1}^{K+1}\Gamma \left({\pi }_j+x_j\right)\right)}^{-1}}{{\left(\Gamma \left({\pi }_{j^{*}}+x_{j^{*}}+\alpha \right)\prod _{j\ne j^{*}}^{K+1}\Gamma \left({\pi }_j+x_j\right)\right)}^{-1}\Gamma \left(\alpha +\sum _{j=1}^{K+1}{\pi }_j+x_j\right)}}{{\left(\frac{{}_{r+1}{F}_{q+1}\left(a_1,\cdots ,a_r,\alpha +\sum _{j=1}^K{\pi }_j+x_j,b_1,\cdots ,b_q,\alpha +\sum _{j=1}^{K+1}{\pi }_j+x_j;\theta \right)}{{}_{r+1}{F}_{q+1}\left(a_1,\cdots ,a_r,\sum _{j=1}^K{\pi }_j+x_j,b_1,\cdots ,b_q,\sum _{j=1}^{K+1}{\pi }_j+x_j;\theta \right)}\right)}^{-1}}\right).\hfill \end{array}$$

(16)

Using the estimated power sum entropy, we can calculate and investigate the behaviour of Tsallis entropy for various parameter scenarios as illustrated in Figure 6 for the bivariate case $K=2$.

4.2. Noncentral Dirichlet Prior

Assuming the noncentral Dirichlet prior and the posterior distribution is given by (14). Using the complete product moments derived in (15), the Bayesian estimator for the power sum entropy (3) can be derived by setting $k_i=\alpha$ with $i=1,\cdots ,K+1$ and $k_{\ne i}=0$.

Theorem 7.

By using (15), the Bayesian estimator for the power sum (4) under the noncentral Dirichlet posterior (14) is given by:

$$\begin{array}{ccc}& & {\widehat{F}}_{\alpha }\left(p\right)\hfill \\ & =& \sum _{j^{*}=1}^{K+1}\left(\frac{\sum _{\varphi }\frac{{\left({\pi }_1+\cdots +{\pi }_K+{\pi }_{K+1}\right)}_{j_1+..+j_{K+1}}}{{\left({\pi }_1\right)}_{j_1}\cdots {\left({\pi }_{K+1}\right)}_{j_{K+1}}{j}_1!\cdots j_{K+1}!}{\left(\frac{{\lambda }_1}{2}\right)}^{j_1}\cdots {\left(\frac{{\lambda }_{K+1}}{2}\right)}^{j_{K+1}}\frac{\Gamma \left({\pi }_{j^{*}}+x_{j^{*}}+j_{j^{*}}+\alpha \right)\prod _{j\ne j^{*}}^{K+1}\Gamma \left({\pi }_j+x_j+j_j\right)}{\Gamma \left(\alpha +\sum _{j=1}^{K+1}{\pi }_j+x_j+j_j\right)}}{\sum _{{\varphi }^{*}}\frac{{\left({\pi }_1+\cdots +{\pi }_K+{\pi }_{K+1}\right)}_{j_1+\cdots +j_{K+1}}}{{\left({\pi }_1\right)}_{j_1}\cdots {\left({\pi }_{K+1}\right)}_{j_{K+1}}{j}_1!\cdots j_{K+1}!}{\left(\frac{{\lambda }_1}{2}\right)}^{j_1}\cdots {\left(\frac{{\lambda }_{K+1}}{2}\right)}^{j_{K+1}}\frac{\prod _{j=1}^{K+1}\Gamma \left({\pi }_j+x_j+j_j\right)}{\Gamma \left(\sum _{j=1}^{K+1}{\pi }_j+x_j+j_j\right)}}\right).\hfill \end{array}$$

(17)

Using the estimated power sum entropy, we can calculate the entropies for different parameters of interests as illustrated in Figure 7 for the bivariate case.

4.3. Numerical Experiments of Entropy

The following steps illustrate the empirical behaviour of the Tsallis entropy under consideration for the alternative priors under consideration (Algorithm 2).

Algorithm 2 Numerical Experiments of Entropy

1 Simulate $p_1$ and $p_2$ from the posterior distribution given by (10) and (14) using the Accept/Rejection method as described earlier for $n=50$; 2 Calculate $p_3=1-p_1-p_2$; 3 Calculate the $p_i^{\alpha }$ values for all the samples; 4 Determine ${\sum }_{i=1}^3{p}_i^{\alpha }$ for each calculation; 5 Calculate the median for the sample of quantities in Step 4 (Note that ${\sum }_{i=1}^3{p}_i^{\alpha }$ might not be symmetric, thus the median is used.) 6 The power sum (16) and (17) is used to calculate the Tsallis entropy (5); 7 Steps 1 to 5 are repeated for different parameters and plotted against the analytical entropy in order to illustrate the accuracy of the derived estimates.

Figure 8 and Figure 9 provides validation of the accuracy of the obtained theoretical expression and contribution for the Dirichlet generator ${}_0{F}_0$ and noncentral Dirichlet cases with $x_1=1$, $x_2=2$ and $x_3=10$. From these two figures, it can be seen that the Dirichlet generator prior resulted in empirical results that closely match the theoretical results, while the noncentral Dirichlet shows slight deviations. It is observed that as ${\pi }_1$ increases, the Tsallis entropy increases (indicating more uncertainly), while as ${\pi }_1$ decreases the Tsallis entropy also decreases (indicating less uncertainty). When considering the location of the density, the changing of ${\pi }_1$ leads to densities which tend to the margin of $p_2$ or towards a specific point along the $p_1+p_2=1$ line. This shows that as the concentration of the density moves toward a point along the $p_1+p_2=1$ line, the uncertainty increases and will decrease as the concentration moves towards the small values of $p_1$.

5. Conclusions

This study focussed on the power sum functional and its estimation as a key tool to model a generalised entropy form, namely Tsallis entropy, via a Bayesian approach. In particular, previous unconsidered Dirichlet priors have been proposed and studied, offering the practitioner more pliable options given experimental data. Specific choices of the proposed Dirichlet family allow for positive correlation in addition to the usual negative correlation characteristic. An example illustrated theoretical results accurately described empirical entropy. Future work could include further investigations into generalised functionals and their modeling in this information-theoretic environment.