Asymptotic Distribution of Certain Types of Entropy under the Multinomial Law

Simple Summary

We obtain expressions for the asymptotic distributions of the Rényi and Tsallis of order q entropies, and Fisher information when computed on the maximum likelihood estimator of probabilities from multinomial random samples. We recall results related to the Shannon entropy. We build a test for comparing entropies of different types and categories.

Abstract

We obtain expressions for the asymptotic distributions of the Rényi and Tsallis of order q entropies and Fisher information when computed on the maximum likelihood estimator of probabilities from multinomial random samples. We verify that these asymptotic models, two of which (Tsallis and Fisher) are normal, describe well a variety of simulated data. In addition, we obtain test statistics for comparing (possibly different types of) entropies from two samples without requiring the same number of categories. Finally, we apply these tests to social survey data and verify that the results are consistent but more general than those obtained with a ${\chi }^2$ test.

1. Introduction

The multinomial distribution is an adequate model for describing how observations fall into categories. Quoting Johnson et al. [1], “The Multinomial distribution, like the Multivariate Normal distribution among the continuous multivariate distributions, consumed a sizable amount of the attention that numerous theoretical as well as applied researchers directed towards the area of discrete multivariate distributions.”

The entropy of a (multivariate, in our case) random variable is a substantial quantity. It quantifies the predictability of a system whose outputs can be described by such a model. Entropy has several definitions, both conceptual and mathematical. The concept of entropy originated as a way to relate a system’s energy and temperature [2]. The same concept was used to describe the number of ways the particles of a system can be arranged.

Entropy has been seldom studied as a random variable. Hutcheson [3] and Hutcheson and Shenton [4] discussed the exact expected value and variance of the Shannon entropy under the multinomial model. These works also provided approximate expressions that circumvent the numerical issues when using the exact value.

Jacquet and Szpankowski [5] studied high-quality analytic approximations of the Rényi entropy, of which the Shannon entropy is a particular case, under the binomial model. With the same approach, Cichoń and Golębiewski [6] obtained expressions for more general functionals that include the multinomial distribution. These works treat the entropy as a fixed quantity. Cook et al. [7] studied almost unbiased estimators of functions of the parameter of the binomial distribution. The authors extended those results to find an almost-unbiased estimator for the entropy under multinomial laws.

Chagas et al. [8] treated the Shannon entropy as a random variable. The authors obtained its asymptotic distribution when indexing by the maximum likelihood estimators of the proportions under the multinomial distribution. This result allowed the devising of unilateral and bilateral tests for comparing the entropy from two samples in a very general way. These tests do not require having the same number of categories.

In this work, our attention is directed toward the asymptotic distribution of other forms of entropy under the multinomial model. This allows the comparison of large samples throughout their entropies and, with this, they may have different numbers of classes. The comparison also allows using different types of entropy. We firstly apply the multivariate delta method and, in the case of the Rényi entropy, we transform the resulting multivariate normal distribution into that of the logarithm of the absolute value of a normally distributed random variable. Then, we provide the general expression of a test statistic that suits our needs.

This paper unfolds as follows. Section 2 recalls the main properties of the multinomial distribution and defines the four types of entropies we will study. In Section 3, we present the central results, i.e., the asymptotic distribution of those entropies. We describe the techniques we used and left for Appendix A.1 technical details. We validate our results with simulation studies in Section 4: we show the adequacy of the normal distribution as limit law for the entropies under three probability models of different support, considering various sample sizes. In Section 5, we show that these asymptotic properties lead to a helpful hypothesis test between samples with different categories. We conclude the article in Section 6. Appendix A.2 comments on applications that justify our choices of the number of categories and sample sizes in the simulation studies. Appendix A.3 discloses relevant computational information, including reproducibility.

2. Entropies and the Multinomial Distribution

Consider a series of n independent trials, where only one of k mutually exclusive events ${\pi }^1,{\pi }^2,\dots ,{\pi }^k$ must be observed in each one, with probability $\mathit{p}=\{p_1,p_2,\dots ,p_k\}$ such that $p_{\ell }\ge 0$ and ${\sum }_{\ell =1}^k{p}_{\ell }=1$. Let $\mathit{N}=(N_1,N_2,\dots ,N_k)$ be the random vector that counts the number of occurrences of the events ${\pi }^1,{\pi }^2,\dots ,{\pi }^k$ in the n trials, with $N_{\ell }\ge 0$ and ${\sum }_{\ell =1}^k{N}_{\ell }=n$. A sample from $\mathit{N}$, say $\mathit{n}$, is a k-variate vector of integer values $\mathit{n}=(n_1,n_2,\dots ,n_k)$. Then, the joint distribution of $\mathit{N}$ is

$$Pr(\mathit{N}=\mathit{n})=Pr(N_1=n_1,N_2=n_2,\dots ,N_k=n_k)=n!\prod _{\ell =1}^k\frac{p_{\ell }^{n_{\ell }}}{n_{\ell }!}.$$

(1)

We denote this situation as $\mathit{N}\sim \mathrm{Mult}(n,\mathit{p})$.

In practice, one does not know the true values of $\mathit{p}$, the probabilities that index this multinomial distribution. Such values are estimated by computing ${\widehat{p}}_{\ell }$, the proportion of times the class (category, event) ${\pi }^{\ell }$ was observed among the k possible categories $\mathit{\pi }=\{{\pi }^1,{\pi }^2,\dots ,{\pi }^k\}$ during the n trials. The maximum likelihood estimator for $\widehat{\mathit{p}}=({\widehat{p}}_1,{\widehat{p}}_2,\dots ,{\widehat{p}}_k)$ is the random vector of proportions. This maximum likelihood estimator coincides with the intuitive estimator based on the distribution’s first moments, and is the most frequently used in applications.

We study the distribution of several forms of entropy of the random vector $\widehat{\mathit{p}}$ for fixed k. Notice that $\widehat{\mathit{p}}$ is computed over a single k-variate measurement of random proportions corresponding to a single random sample from $\mathit{N}\sim \mathrm{Mult}(n,\mathit{p})$. The asymptotic behaviors we derive hold for typical cases in which $n\gg k$.

The Shannon entropy measures the disorder or unpredictability of systems characterized by a probability distribution. On the one hand, the minimum Shannon value occurs when there is complete knowledge about the system behavior and total confidence in predicting the following observation. On the other hand, when a uniform distribution describes the system’s behavior, that is, when all possibilities have the same probability of occurrence, the knowledge about the behavior of the data is minimal. In Chagas et al. [8], we studied the asymptotic distribution of the Shannon entropy. In this work, we extend those results to three other forms of entropy.

Other types of descriptors have been proposed in the literature to extract additional information not captured by the Shannon entropy. Tsallis [9] and Rényi [10], for instance, proposed parametric versions, which include the Shannon entropy.

Fisher information [11] is defined by an average logarithm derivative of a continuous probability density function. In the case of discrete densities, this measure can be approximated using differences of probabilities between consecutive distribution elements. While the Shannon entropy captures the degree of unpredictability of a system, the Fisher information is related to the rate of change of consecutive observations and, thus, quantifies small changes and perturbations.

Given a type of entropy H, we are interested in the distribution of $H\left(\mathit{p}\right)$ when indexed by $\widehat{\mathit{p}}$, the maximum likelihood estimator of $\mathit{p}$. Our problem then becomes finding the distribution of $H\left(\widehat{\mathit{p}}\right)$ for the following:

• The Shannon entropy

  $$H_S\left(\widehat{\mathit{p}}\right)=-\sum _{\ell =1}^k{\widehat{p}}_{\ell }log{\widehat{p}}_{\ell },$$

  (2)
• The Tsallis entropy with index $q\in \mathbb{R}\backslash \left\{1\right\}$

  $$H_T^q\left(\widehat{\mathit{p}}\right)=\sum _{\ell =1}^k\frac{{\widehat{p}}_{\ell }-{\widehat{p}}_{\ell }^q}{q-1},$$

  (3)
• The Rényi entropy of order $q\in {\mathbb{R}}^{+}\backslash \left\{1\right\}$

  $$H_R^q\left(\widehat{\mathit{p}}\right)=\frac{1}{1-q}log\sum _{\ell =1}^k{\widehat{p}}_{\ell }^q,$$

  (4)
• The Fisher information, also termed “Fisher Information Measure” in the literature, with renormalization coefficient $F_0=4$

  $$H_F\left(\widehat{\mathit{p}}\right)=F_0\sum _{\ell =1}^{k-1}{\left(\sqrt{{\widehat{p}}_{\ell +1}}-\sqrt{{\widehat{p}}_{\ell }}\right)}^2.$$

  (5)
  Among other possibilities, we used Equation (2.7) from Ref. [12].

3. Asymptotic Distributions of Entropies

The main results of this section are the asymptotic distributions of the Shannon (2), Tsallis of order q (3), and Rényi of order q (4) entropies, and Fisher information (5). These results are presented, respectively, in Equations (30)–(32) and (35). Notice that the Rényi entropy is not asymptotically normally distributed, while the other three are.

We recall the following theorems known respectively as the delta method and its multivariate version. We refer to Lehmann and Casella [13] for their proofs.

Theorem 1.

Let $X_n$ be a sequence of independent and identically distributed random variables such that $\sqrt{n}[X_n-\theta ]$ converges in distribution to a $\mathcal{N}(0,{\sigma }^2)$. If $\partial h/\partial \theta$ exists and does not vanish, then $\sqrt{n}[h\left(X_n\right)-h\left(\theta \right)]$ converges in distribution to a $\mathcal{N}(0,{\sigma }^2{[\partial h/\partial \theta ]}^2)$.

Theorem 2.

Let ${\mathit{X}}_n=(X_{1n},X_{2n},\dots ,X_{kn})$ be a sequence of independent and identically distributed vectors of random variables such that $\sqrt{n}[X_{1n}-{\theta }_1,X_{2n}-{\theta }_2,\dots ,X_{kn}-{\theta }_k]$ converges in distribution to a multivariate normal distribution ${\mathcal{N}}_n(\mathbf{0},\mathsf{\Sigma })$, where Σ is the covariance matrix. Suppose that $h_1,h_2,\dots ,h_k$ are real functions continuously differentiable in a neighborhood of the parameter point $\mathit{\theta }=({\theta }_1,{\theta }_2,\dots ,{\theta }_k)$ and such that the matrix of partial derivatives $B={(\partial h_{\ell }/\partial {\theta }_j)}_{\ell ,j=1}^k$ is non-singular in the mentioned neighborhood. Then, the following convergence in distribution holds:

$$\sqrt{n}\left[h_1\left({\mathit{X}}_n\right)-h_1\left(\mathit{\theta }\right),h_2\left({\mathit{X}}_n\right)-h_2\left(\mathit{\theta }\right),\dots ,h_k\left({\mathit{X}}_n\right)-h_k\left(\mathit{\theta }\right)\right]\stackrel{\mathcal{D}}{\to }\mathcal{N}(\mathbf{0},B\mathsf{\Sigma }{B}^{\prime }),$$

where $B^{\prime }$ denotes the transpose of B.

Now, we focus on the case $\mathit{N}\sim \mathrm{Mult}(n,\mathit{p})$. Let $\widehat{\mathit{p}}=\mathit{N}/n$ be the vector of sample proportions which coincides with the maximum likelihood estimator (MLE) of $\mathit{p}$ and ${\mathit{Y}}_n=\sqrt{n}(\widehat{\mathit{p}}-\mathit{p})$. Then

$${\mathit{Y}}_n\stackrel{\mathcal{D}}{\to }\mathcal{N}(\mathbf{0},{\mathbf{D}}_{\mathit{p}}-\mathit{p}{\mathit{p}}^{\prime }),$$

where ${\mathbf{D}}_{\mathit{p}}=\mathrm{Diag}(p_1,p_2,\dots ,p_k)$.

Let us explore the covariance matrix in this case:

$$\begin{array}{cc}{\mathbf{D}}_{\mathit{p}}-\mathit{p}{\mathit{p}}^{\prime }\hfill & =\left(\begin{array}{cccc}{p}_1& 0& \cdots & 0\\ 0& p_2& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & p_k\end{array}\right)-\left(\begin{array}{c}{p}_1\\ p_2\\ ⋮\\ p_k\end{array}\right)\left(\begin{array}{cccc}{p}_1& p_2& \cdots & p_k\end{array}\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & =\left(\begin{array}{cccc}{p}_1-p_1^2& -p_1{p}_2& \cdots & -p_1{p}_k\\ -p_2{p}_1& p_2-p_2^2& \cdots & -p_2{p}_k\\ ⋮& ⋮& \ddots & ⋮\\ -p_k{p}_1& -p_k{p}_2& \cdots & p_k-p_k^2\end{array}\right)\hfill \end{array}$$

(7)

It means that the covariance matrix ${\mathsf{\Sigma }}_{\mathit{p}}\in {\mathbb{R}}^{k\times k}$ we are interested in is of the form

$${\left({\mathsf{\Sigma }}_{\mathit{p}}\right)}_{\ell j}=\left\{\begin{array}{cc}{p}_{\ell }(1-p_{\ell })\hfill & \mathrm{if}\ell =j,\hfill \\ -p_{\ell }{p}_j\hfill & \mathrm{if}\ell \ne j.\hfill \end{array}\right.$$

(8)

The above statements are generalized. In the following, we obtain new results for the Tsallis and Rényi entropies, and for the Fisher information. For the sake of completeness, we also include the results for the Shannon entropy.

In order to apply the delta method using Theorem 2, we consider the following functions:

$$\begin{array}{cc}\hfill h_{\ell }^S(p_1,p_2,\dots ,p_k)& =p_{\ell }log{p}_{\ell },\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill h_{\ell }^T(p_1,p_2,\dots ,p_k)& =p_{\ell }-p_{\ell }^q,\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill h_{\ell }^R(p_1,p_2,\dots ,p_k)& =p_{\ell }^q,\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill h_{\ell }^F(p_1,p_2,\dots ,p_k)& ={\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)}^2,\hfill \end{array}$$

(12)

for $\ell =1,2,\dots ,k$ except for () that holds for $\ell =1,2,\dots ,k-1$. The assumptions are verified, and thus,

$$\begin{array}{cc}\hfill & \frac{\partial h_{\ell }^S}{\partial p_{\ell }}=log{p}_{\ell }+1\mathrm{and}\frac{\partial h_{\ell }^S}{\partial p_j}=0\mathrm{if}j\ne \ell ,\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & \frac{\partial h_{\ell }^T}{\partial p_{\ell }}=1-q{p}_{\ell }^{q-1}\mathrm{and}\frac{\partial h_{\ell }^T}{\partial p_j}=0\mathrm{if}j\ne \ell ,\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill & \frac{\partial h_{\ell }^R}{\partial p_{\ell }}=q{p}_{\ell }^{q-1}\mathrm{and}\frac{\partial h_{\ell }^R}{\partial p_j}=0\mathrm{if}j\ne \ell ,\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & \frac{\partial h_{\ell }^F}{\partial p_j}=\frac{\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}}{{(-1)}^{\ell +j-1}\sqrt{p_j}}\mathrm{if}j=\ell ,\ell +1\mathrm{and}\frac{\partial h_{\ell }^F}{\partial p_j}=0\mathrm{if}j\ne \ell ,\ell +1.\hfill \end{array}$$

(16)

Finally, we need the covariance matrix of the multivariate normal limit distribution, which is

$${\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }\mathcal{M}}={\left(\frac{\partial h_{\ell }^{\mathcal{M}}}{\partial p_j}\right)}_{\ell ,j=1}^k{\mathsf{\Sigma }}_{\mathit{p}}{{\left(\frac{\partial h_{\ell }^{\mathcal{M}}}{\partial p_j}\right)}_{\ell ,j=1}^k}^{\prime },$$

(17)

where $\mathcal{M}\in \{S,T,R,F\}$. Since ${(\partial h_{\ell }^{\mathcal{M}}/\partial p_j)}_{\ell ,j=1}^k$ are diagonal matrices for $\mathcal{M}\in \{S,T,R\}$, we can use Equation (A1) to conclude that

$$\begin{array}{cc}\hfill & {\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }S}\right)}_{\ell j}=\left\{\begin{array}{cc}(p_{\ell }-p_{\ell }^2){(log{p}_{\ell }+1)}^2\hfill & \mathrm{if}\ell =j,\hfill \\ -p_{\ell }{p}_j(log{p}_{\ell }+1)(log{p}_j+1)\hfill & \mathrm{if}\ell \ne j;\hfill \end{array}\right.\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill & {\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }T}\right)}_{\ell j}=\left\{\begin{array}{cc}(p_{\ell }-p_{\ell }^2){(1-q{p}_{\ell }^{q-1})}^2\hfill & \mathrm{if}\ell =j,\hfill \\ -p_{\ell }{p}_j(1-q{p}_{\ell }^{q-1})(1-q{p}_j^{q-1})\hfill & \mathrm{if}\ell \ne j;\hfill \end{array}\right.\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & {\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }R}\right)}_{\ell j}=\left\{\begin{array}{cc}{q}^2(p_{\ell }-p_{\ell }^2)p_{\ell }^{2(q-1)}\hfill & \mathrm{if}\ell =j,\hfill \\ -q^2{\left(p_{\ell }{p}_j\right)}^q\hfill & \mathrm{if}\ell \ne j.\hfill \end{array}\right.\hfill \end{array}$$

(20)

In the case of ${\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }F}$, from Equations (A3) and (A4) we have the following:

• For $\ell ,j=1,2,\dots ,k-2$ and $\ell \ne j-1,j,j+1$:

  $${\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }F}\right)}_{\ell j}=\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)\left(\sqrt{p_{j+1}}-\sqrt{p_j}\right)\left(\sqrt{p_{\ell +1}{p}_j}+\sqrt{p_{\ell }{p}_{j+1}}-\sqrt{p_{\ell }{p}_j}-\sqrt{p_{\ell +1}{p}_{j+1}}\right).$$

  (21)
• For $\ell =1,2,\dots ,k-2$:

  $$\begin{array}{cc}{\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }F}\right)}_{\ell ,\ell -1}=\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)\hfill & \left(\sqrt{p_{\ell }}-\sqrt{p_{\ell -1}}\right)\hfill \\ \hfill & \left(\sqrt{p_{\ell +1}{p}_{\ell -1}}+p_{\ell }-1-\sqrt{p_{\ell }{p}_{\ell -1}}-\sqrt{p_{\ell +1}{p}_{\ell }}\right).\hfill \end{array}$$

  (22)
• For $\ell =1,2,\dots ,k-2$:

  $${\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }F}\right)}_{\ell \ell }={\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)}^2\left(2\sqrt{p_{\ell }{p}_{\ell +1}}+2-p_{\ell }-p_{\ell +1}\right).$$

  (23)
• For $\ell =1,2,\dots ,k-2$:

  $${\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }F}\right)}_{\ell ,\ell +1}=\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)\left(\sqrt{p_{\ell +2}}-\sqrt{p_{\ell +1}}\right)\left(p_{\ell +1}-1+\sqrt{p_{\ell }{p}_{\ell +2}}-\sqrt{p_{\ell }{p}_{\ell +1}}-\sqrt{p_{\ell +1}{p}_{\ell +2}}\right).$$

  (24)
• For $j=1,2,\dots ,k-2$:

  $${\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }F}\right)}_{k-1,j}=(\sqrt{p_k}-\sqrt{p_{k-1}})(\sqrt{p_{j+1}}-\sqrt{p_j})\left(\frac{p_k{p}_{j+1}}{\sqrt{p_{j+1}}}-\frac{p_{k-1}{p}_j}{\sqrt{p_j}}\right).$$

  (25)
• Finally,

  $${\left({\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }F}\right)}_{k-1,k-1}={(\sqrt{p_k}-\sqrt{p_{k-1}})}^2(1-p_{k-1}).$$

  (26)

Hence, we conclude that

$$\sqrt{n}\left[h_1^{\mathcal{M}}\left({\widehat{p}}_1\right)-h_1^{\mathcal{M}}\left(p_1\right),h_2^{\mathcal{M}}\left({\widehat{p}}_2\right)-h_2^{\mathcal{M}}\left(p_2\right),\dots ,h_{k^{\prime }}^{\mathcal{M}}{\widehat{p}}_{k^{\prime }}-h_{k^{\prime }}^{\mathcal{M}}\left(p_{k^{\prime }}\right)\right]\stackrel{\mathcal{D}}{\to }\mathcal{N}(\mathbf{0},{\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }\mathcal{M}}),$$

(27)

where $\mathcal{M}\in \{S,T,R,F\}$ and $k^{\prime }=k$ in all cases except for the case of the Fisher information in which $k^{\prime }=k-1$. An equivalent expression is

$$\sqrt{n}\left[h_1^{\mathcal{M}}\left({\widehat{p}}_1\right),h_2^{\mathcal{M}}\left({\widehat{p}}_2\right),\dots ,h_{k^{\prime }}^{\mathcal{M}}\left({\widehat{p}}_{k^{\prime }}\right)\right]\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(\sqrt{n}\left(\begin{array}{c}{h}_1^{\mathcal{M}}\left(p_1\right)\\ h_2^{\mathcal{M}}\left(p_2\right)\\ ⋮\\ h_{k^{\prime }}^{\mathcal{M}}\left(p_{k^{\prime }}\right)\end{array}\right),{\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }\mathcal{M}}\right).$$

(28)

If $\mathit{Y}$ is a vector of random variables such that $\sqrt{n}\mathit{Y}\stackrel{\mathcal{D}}{\to }\mathcal{N}(\sqrt{n}\mathit{\mu },\mathsf{\Sigma })$, then it can be proved that $E\left(\sqrt{n}\mathit{Y}\right)\to \sqrt{n}\mathit{\mu }$ and $Var\left(\sqrt{n}\mathit{Y}\right)\to \mathsf{\Sigma }$. Provided well-known properties, it holds that $E\left(\mathit{Y}\right)\to \mathit{\mu }$ and $Var\left(\mathit{Y}\right)\to 1/n\mathsf{\Sigma }$. Applying this to (28),

$$\left[h_1^{\mathcal{M}}\left({\widehat{p}}_1\right),h_2^{\mathcal{M}}\left({\widehat{p}}_2\right),\dots ,h_{k^{\prime }}^{\mathcal{M}}\left({\widehat{p}}_{k^{\prime }}\right)\right]\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(\left(\begin{array}{c}{h}_1^{\mathcal{M}}\left(p_1\right)\\ h_2^{\mathcal{M}}\left(p_2\right)\\ ⋮\\ h_{k^{\prime }}^{\mathcal{M}}\left(p_{k^{\prime }}\right)\end{array}\right),\frac{1}{n}{\mathsf{\Sigma }}_{\mathit{p}}^{\mathsf{\Delta }\mathcal{M}}\right).$$

(29)

Now, using (29), we find the asymptotic distribution of (2)–(5). In order to do so, we need to know the distribution of the sum of k Gaussian random variables with different means and an arbitrary covariance matrix.

For any k-dimensional multivariate normal distribution $\mathit{Z}\sim \mathcal{N}(\mathit{\mu },\mathsf{\Sigma })$, with $\mathit{\mu }\in {\mathbb{R}}^k$ and covariance matrix $\mathsf{\Sigma }=\left({\sigma }_{\ell j}\right)$, holds that the distribution of $W={\mathit{a}}^{\mathrm{T}}\mathit{Z}$, with $\mathit{a}\in {\mathbb{R}}^k$, is $\mathcal{N}\left({\mathit{a}}^{\mathrm{T}}\mathit{\mu },{\sum }_{\ell =1}^k{a}_{\ell }^2{\sigma }_{\ell \ell }+2{\sum }_{\ell =1}^{k-1}{\sum }_{j=i+1}^k{a}_{\ell }{a}_j{\sigma }_{\ell j}\right)$. Using the limit distribution presented in (29) and $\mathit{a}=(-1,-1,\dots ,-1)$, we directly have the asymptotic distribution of the Shannon entropy as follows:

$$\begin{array}{c}{H}_S\left(\widehat{\mathit{p}}\right)=-\sum _{\ell =1}^k{\widehat{p}}_{\ell }log{\widehat{p}}_{\ell }\stackrel{\mathcal{D}}{\to }\hfill \\ \hfill \mathcal{N}\left(-\sum _{\ell =1}^k{p}_{\ell }log{p}_{\ell },\frac{1}{n}\sum _{\ell =1}^k{p}_{\ell }(1-p_{\ell }){(log{p}_{\ell }+1)}^2-\frac{2}{n}\sum _{j=1}^{k-1}\sum _{\ell =j+1}^k{p}_{\ell }{p}_j(log{p}_{\ell }+1)(log{p}_j+1)\right).\end{array}$$

(30)

With similar arguments and $\mathit{a}=(1,1,\dots ,1)$, we obtain the asymptotic distribution for the Tsallis entropy of order q:

$$H_T^q\left(\widehat{\mathit{p}}\right)=\sum _{\ell =1}^k\frac{{\widehat{p}}_{\ell }-{\widehat{p}}_{\ell }^q}{q-1}\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(\sum _{\ell =1}^k\frac{p_{\ell }-p_{\ell }^q}{q-1},\sum _{\ell =1}^k\frac{(p_{\ell }-p_{\ell }^2){(1-q{p}_{\ell }^{q-1})}^2}{n{(q-1)}^2}-2\sum _{j=1}^{k-1}\sum _{\ell =j+1}^k\frac{p_{\ell }{p}_j(1-q{p}_{\ell }^{q-1})(1-q{p}_j^{q-1})}{n{(q-1)}^2}\right).$$

(31)

The procedure is analogous for the Fisher information but with $\mathit{a}=(1,1,\dots ,1)\in {\mathbb{R}}^{k-1}$. Hence, it can be proved that

$$H_F\left(\widehat{\mathit{p}}\right)=F_0\sum _{\ell =1}^{k-1}{(\sqrt{{\widehat{p}}_{\ell -1}}-\sqrt{{\widehat{p}}_{\ell }})}^2\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(F_0\sum _{\ell =1}^{k-1}{(\sqrt{p_{\ell -1}}-\sqrt{p_{\ell }})}^2,\frac{F_0}{n}{\mathsf{\Sigma }}^{*}\right),$$

(32)

where

$$\begin{array}{c}{\mathsf{\Sigma }}^{*}={\left(\sqrt{p_k}-\sqrt{p_{k-1}}\right)}^2(1-p_{k-1})+\sum _{\ell =1}^{k-2}{\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)}^2\left(2\sqrt{p_{\ell }{p}_{\ell +1}}-p_{\ell }-p_{\ell +1}+2\right)+\hfill \\ 2\sum _{\ell =3}^{k-2}\sum _{j=1}^{\ell -2}\left[\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)\left(\sqrt{p_{j+1}}-\sqrt{p_j}\right)\left(\sqrt{p_{\ell +1}{p}_j}+\sqrt{p_{\ell }{p}_{j+1}}-\sqrt{p_{\ell }{p}_j}-\sqrt{p_{\ell +1}{p}_{j+1}}\right)\right]+\\ 2\sum _{j=1}^{k-2}\left[\left(\sqrt{p_k}-\sqrt{p_{k-1}}\right)\left(\sqrt{p_{j+1}}-\sqrt{p_j}\right)\left(p_k\sqrt{p_{j+1}}-p_{k-1}\sqrt{p_j}\right)\right]+\\ \hfill 2\sum _{\ell =2}^{k-2}\left[\left(\sqrt{p_{\ell +1}}-\sqrt{p_{\ell }}\right)\left(\sqrt{p_{\ell }}-\sqrt{p_{\ell -1}}\right)\left(\sqrt{p_{\ell +1}{p}_{\ell -1}}-\sqrt{p_{\ell }{p}_{\ell -1}}-\sqrt{p_{\ell +1}{p}_{\ell }}+p_{\ell }-1\right)\right].\end{array}$$

(33)

To obtain expression (33), we use the symmetry of the covariance matrix which implies that ${\sum }_{\ell =1}^{k-1}{\sum }_{j=\ell +1}^k{a}_{\ell }{a}_j{\sigma }_{\ell j}={\sum }_{\ell =2}^{k-1}{\sum }_{j=1}^{\ell -1}{a}_{\ell }{a}_j{\sigma }_{\ell j}$. It is worth noticing that the expression of the covariance matrix for Fisher information is more complicated than the previously analyzed entropies since the matrix of partial derivatives is not diagonal in this case.

The case of Rényi entropy is different because, following the previous methodology, we can prove that

$$\sum _{\ell =1}^k{\widehat{p}}_{\ell }^q\stackrel{\mathcal{D}}{\to }\mathcal{N}\left(\sum _{\ell =1}^k{p}_{\ell }^q,\frac{1}{n}\sum _{\ell =1}^k{q}^2(p_{\ell }-p_{\ell }^2)p_{\ell }^{2(q-1)}-\frac{2}{n}\sum _{\ell =1}^{k-1}\sum _{j=\ell +1}^k{q}^2{\left(p_j{p}_{\ell }\right)}^q\right).$$

(34)

Hence,

$$H_R^q\left(\widehat{\mathit{p}}\right)=\frac{1}{1-q}log\sum _{\ell =1}^k{\widehat{p}}_{\ell }^q\stackrel{\mathcal{D}}{\to }{\mathcal{P}}_R^q,$$

(35)

where

$${\mathcal{P}}_R^q\left(x\right)=\frac{1-q}{{\sigma }^{*}\sqrt{2\pi }}exp[(1-q)xlog\left(k\right)]exp\left\{-\frac{1}{2}{\left(\frac{exp[(1-q)xlog\left(k\right)]-{\mu }^{*}}{{\sigma }^{*}}\right)}^2\right\},$$

(36)

with ${\mu }^{*}={\sum }_{\ell =1}^k{p}_{\ell }^q$ and ${\sigma }^{*}=n^{-1}{\sum }_{\ell =1}^k{q}^2(p_{\ell }-p_{\ell }^2)p_{\ell }^{2(q-1)}-2n^{-1}{\sum }_{\ell =1}^{k-1}{\sum }_{j=\ell +1}^k{q}^2{\left(p_{\ell }{p}_j\right)}^q$. Notice that this is not a normal distribution but the distribution of the logarithm of the absolute value of a normally distributed random variable.

Often, in practice, these entropies are scaled to be in $[0,1]$; these are called “normalized entropies”. The following modifications must be considered in the normalized versions of the entropies. For the normalized Shannon entropy, the asymptotic mean and variance are multiplied by $1/logk$ and $1/{(logk)}^2$, respectively. In the case of the normalized Tsallis entropy, the asymptotic mean and variance are multiplied by $(q-1)/(1-k^{1-q})$ and ${(q-1)}^2/{(1-k^{1-q})}^2$, respectively. Finally, the asymptotic distribution of the normalized Rényi entropy is ${\tilde{\mathcal{P}}}_R^q\left(x\right)=logk{\mathcal{P}}_R^q(xlogk)$. Notice that normalized entropies do not depend on the logarithm basis. The Fisher information is, as defined in (5), already normalized.

4. Analysis and Validation

In this section, we study the empirical distribution of the entropies computed from $\widehat{\mathit{p}}$ under three models, four categories ($k\in \{6,24,120,720\}$), and three sample sizes ($n\in \{{10}^{2}k,{10}^{3}k,{10}^{4}k\}$) that depend on the number of categories. These choices of k and n are based on the values that appear in signal analysis with ordinal patterns; see details of this technique in Appendix A.2.

We considered the following probability functions $\mathit{p}=(p_1,p_2,\dots ,p_k)$:

• Linear: $p_{\ell }=2\ell /\left(k(k+1)\right)$, $1\le \ell \le k$.
• One-Almost-Zero: $p_{\ell }=1/k$ for $1\le \ell \le k-2$, $p_{k-1}={ϵ}_0$, and $p_k=2/k-{ϵ}_0$ with ${ϵ}_0=2.220446\times {10}^{-16}$ (the smallest positive number for which, in our computer platform, $1+{ϵ}_0>1$).
• Half-and-Half: $p_{\ell }=1/k+ϵ/k$ for $1\le \ell \le k/2$, and $p_{\ell }=1/k-ϵ/k$ for $k/2+1\le \ell \le k$, with $ϵ\in \{0.1,0.3,0.5,0.8\}$.

These probability functions are illustrated, for $k=6$ and $ϵ=0.3$, in Figure 1. We studied the behavior of the Shannon entropy, the Rényi entropy with $q\in \{1/3,2/5\}$, the Tsallis entropy with $q\in \{1/2,3/2\}$, and the Fisher information computed on samples of sizes $n\in \{{10}^{2}k,{10}^{3}k,{10}^{4}k\}$. We used 300 independent samples (replicates).

Although Equation (35) shows that the Rényi entropy is not asymptotically normal, we verified that its density is similar to that of a Gaussian distribution. With this in mind, we also checked of the normality of Rényi entropies. We used the Anderson–Darling test to verify the null hypothesis that the data follow a normal distribution. We chose this test because it uses the hypothesized distribution in calculating critical values. This test is more sensitive than other alternatives; see, for instance, the book by Lehman and Romano [14].

From

Table 1. Situations for which the p-values of the Anderson–Darling test for the normality of samples of size 300 are less than $0.01$ (“HF” stands for the Fisher information; “HaH” and “OAZ” are the Half-And-Half and One-Almost-Zero models).

Type	Model	$\mathit{ϵ}$	k	n	p-Value
$H_F$	HaH	0.8	6	600	0.0030
$H_F$	HaH	0.1	6	600	0.0000
$H_F$	HaH	0.1	24	2400	0.0000
$H_F$	HaH	0.8	24	2400	0.0064
$H_F$	HaH	0.1	6	6000	0.0000
$H_F$	HaH	0.3	6	600	0.0000
$H_F$	HaH	0.1	120	120,000	0.0089
$H_F$	HaH	0.1	24	24000	0.0004
$H_F$	Linear	0	6	600	0.0000
$H_F$	Linear	0	24	2400	0.0000
$H_R^{1/3}$	HaH	0.1	6	600	0.0000
$H_R^{1/3}$	HaH	0.1	24	2400	0.0001
$H_R^{1/3}$	OAZ	0	24	2400	0.0000
$H_R^{2/3}$	HaH	0.1	6	600	0.0000
$H_R^{2/3}$	HaH	0.1	24	2400	0.0001
$H_R^{2/3}$	OAZ	0	24	2400	0.0000
$H_S$	HaH	0.1	6	600	0.0001
$H_S$	HaH	0.1	24	2400	0.0002
$H_S$	OAZ	0	24	2400	0.0000
$H_T^{1/2}$	HaH	0.1	6	600	0.0000
$H_T^{1/2}$	HaH	0.1	24	2400	0.0001
$H_T^{1/2}$	OAZ	0	24	2400	0.0000
$H_T^{3/2}$	HaH	0.1	6	600	0.0001
$H_T^{3/2}$	HaH	0.1	24	2400	0.0003
$H_T^{3/2}$	OAZ	0	24	2400	0.0000

, we notice that the Fisher information is the one that fails most times to pass the normality test at the 1% The situation that appears with $p-\mathrm{value}=0.0010$ in the table has, in fact, $p-\mathrm{value}=9.606130\times {10}^{-3}$; the table shows rounded values. Figure 2 shows four of these cases, namely for $k=6$, $n=600$, and $ϵ=0.1,0.3,0.5,0.8$. We notice that the deviation from the normal hypothesis is more prevalent in both tails, being that the observations are larger than the theoretical quantiles.

The normality hypothesis was rejected at the 1% level by the Anderson–Darling test in only 24 out of 432 situations, showing that the asymptotic Gaussian model for the entropies is a good description for these data. Table 1 shows those situations.

With the aim to assess the goodness of fit of the asymptotic models, we applied the Kolmogorov–Smirnov test to fifty replicates of samples.

Table 2. Situations for which the p-values of the Kolmogorov–Smirnov test of samples of size 50 are larger than or equal to $0.05$ (“HaH” and “OAZ” are the Half-And-Half and One-Almost-Zero models).

Type	Model	$\mathit{ϵ}$	k	n		Type	Model	$\mathit{ϵ}$	k	n
$H_S$	HaH	$0.1$	$6,24$	${10}^{3}k,{10}^{4}k$		$H_T^{3/2}$	HaH	0.1	$6,24$	${10}^{3}k,{10}^{4}k$
			$120,720$	${10}^{4}k$					$120,720$	${10}^{4}k$
		$0.3$	$6,120$	for all				$0.3$	$6,120$	for all
			24	${10}^{3}k$					24	${10}^{3}k$
			720	${10}^{3}k,{10}^{4}k$					720	${10}^{3}k,{10}^{4}k$
		$0.5$	$6,24$	for all				$0.5$	$6,24$	for all
			$120,720$	${10}^{3}k,{10}^{4}k$					$120,720$	${10}^{3}k,{10}^{4}k$
		$0.8$	6	${10}^{2}k,{10}^{3}k$				$0.8$	6	${10}^{2}k,{10}^{3}k$
			$24,720$	for all					$24,720$	for all
			120	${10}^{3}k,{10}^{4}k$					120	${10}^{3}k,{10}^{4}k$
	Linear	0	$6,24,120$	for all			Linear	0	$6,24,120$	for all
			720	${10}^{3}k,{10}^{4}k$					720	${10}^{2}k,{10}^{4}k$
	OAZ	0	$6,24$	for all			OAZ	0	$6,24$	for all
$H_F$	HaH	$0.1$	6	${10}^{3}k$		$H_R^{1/3}$	HaH	$0.3$	6	${10}^{2}k,{10}^{3}k$
		$0.3$	$6,24$	for all				0.5	6	for all
		$0.5$	6	${10}^{2}k,{10}^{3}k$				$0.8$	6	${10}^{2}k,{10}^{3}k$
			24	${10}^{3}k,{10}^{4}k$			Linear	0	6	for all
			120	${10}^{4}k$			OAZ	0	6	for all
		$0.8$	6	${10}^{2}k,{10}^{3}k$					24	${10}^{2}k$
			24	for all		$H_R^{2/3}$	HaH	$0.3$	6	for all
	Linear	0	6	${10}^{3}k,{10}^{4}k$				$0.5$	6	for all
			24	${10}^{4}k$				$0.8$	6	${10}^{2}k,{10}^{3}k$
	OAZ	0	$6,24$	for all			Linear	0	6	for all
$H_T^{1/2}$	HaH	$0.1$	$6,24$	${10}^{3}k,{10}^{4}k$			OAZ	0	6	${10}^{2}k,{10}^{3}k$
			$120,720$	${10}^{4}k$
		$0.3$	$6,120$	for all
			24	${10}^{3}k$
			720	${10}^{3}k,{10}^{4}k$
		$0.5$	$6,24$	for all
			$120,720$	${10}^{3}k,{10}^{4}k$
		$0.8$	6	${10}^{2}k,{10}^{3}k$
			24	for all
			$120,720$	${10}^{3}k,{10}^{4}k$
	Linear	0	for all	for all
	OAZ	0	$6,24$	for all
			120	${10}^{3}k,{10}^{4}k$
			720	${10}^{4}k$

shows the results where the p-value of the test is at least equal to $0.05$.

It is worth noticing that even in those cases where the p-value is lesser than $0.05$, the asymptotic models are a good fit to the data as can be seen in several examples exhibited in Figure 3. The Fisher information shows the worst fitting. Additionally, notice in Figure 3d that, although the asymptotic distribution of the Rényi entropy is not normal, the probability density function is visually very close to the Gaussian model. We verified this similarity in all the cases we considered.

5. Application

Inspired by an example from Agresti [16] (p. 200), we extracted data from the General Social Survey (GSS, a project of the independent research organization NORC at the University of Chicago, with principal funding from the National Science Foundation, available at https://gss.norc.org/. The data were downloaded on 24 December 2022).

Table 3. GSS data about religious intolerance.

Year	1998	2008	2018
STRONGLY AGREE	148	285	186
AGREE	429	602	496
NOT AGREE/DISAGREE	278	210	229
DISAGREE	275	196	181
STRONG DISAGREE	72	30	38
Total	1202	1323	1130

shows the level of agreement to the assertion “Religious people are often too intolerant” as measured in three years.

The p-values of pairwise ${\chi }^2$ tests for the null hypotheses that the underlying probabilities are equal are

• 1998 and 2008:$3.43\times {10}^{-22}$,
• 1998 and 2018:$2.01\times {10}^{-8}$,
• 2008 and 2018:$1.06\times {10}^{-3}$.

On the one hand, these values attest that 1998 and 2008 and 1998 and 2018 are very different. On the other hand, although significant, the change between 2008 and 2018 is not so significant.

Table 4. Asymptotic mean and variance of entropies.

	Mean			Variance
	1998	2008	2018	1998	2008	2018
$H_S$	0.914	0.839	0.863	0.0000724	0.0001081	0.0001175
$H_R^{1/3}$	0.967	0.933	0.947	0.0001172	0.0002375	0.0002122
$H_R^{2/3}$	0.939	0.881	0.902	0.0000317	0.0000500	0.0000512
$H_T^{1/2}$	0.932	0.868	0.892	0.0000500	0.0000871	0.0000842
$H_T^{3/2}$	0.919	0.849	0.870	0.0000622	0.0001089	0.0001183
$H_F$	0.516	0.702	0.642	0.0008028	0.0011355	0.0011917

shows the asymptotic mean and variance (in entropies normalized units) of the entropies of the proportions reported in Table 3.

We perform the same hypothesis test with the asymptotic quantities presented in Table 4.

Table 5. p-values of the hypothesis of equal entropies.

	1998–2008	1998–2018	2008–2018
$H_S$	0.0000000	0.0002606	0.1029
$H_R^{1/3}$	0.0705681	0.2520262	0.5316
$H_R^{2/3}$	0.0000000	0.0000478	0.0404
$H_T^{1/2}$	0.0000000	0.0004471	0.0690
$H_T^{3/2}$	0.0000001	0.0002316	0.1672
$H_F$	0.0000219	0.0045777	0.2118

shows the p-values of the null hypothesis that the entropies are equal, using the test discussed by Chagas et al. [8] (Section 5):

$$p-\mathrm{value}\approx 2\left(1-\mathsf{\Phi }\left(\frac{\left|H\left(\widehat{{\mathit{p}}_1}\right)-H\left(\widehat{{\mathit{p}}_2}\right)\right|}{\sqrt{{\widehat{{\sigma }^2}}_{n_1,\widehat{{\mathit{p}}_1}}+{\widehat{{\sigma }^2}}_{n_2,\widehat{{\mathit{p}}_2}}}}\right)\right),$$

(37)

where $\mathsf{\Phi }$ is the cumulative distribution function of a standard normal random variable, H is any of the considered entropies computed with the observed proportions $\widehat{{\mathit{p}}_i}$, $i=1,2$, and ${\widehat{{\sigma }^2}}_{n_i,\widehat{{\mathit{p}}_i}}$ is the corresponding sample asymptotic variance that takes into account the sample size $n_i$. Notice that the test based on entropies compares only these features, and not the underlying distribution.

The results in Table 5 are consistent with those provided by the ${\chi }^2$ tests, i.e., the most significant differences arise between 1998 and 2008 and between 1998 and 2018. Moreover, the tests based on entropies do not reject the null hypothesis in the pair 2008–2018, except for Rényi entropy of order $2/3$. The increased p-values are a consequence of the information reduction: whereas the ${\chi }^2$ test compares count-by-count, those based on entropies compare two scalars.

In the second part of this application, we will illustrate the use of test statistics based on entropies for comparing samples with different categories. Situations like this may appear when applying alternative versions of the same questionnaire in a series of surveys.

We collapsed the categories of 1998 into three: “agreement” (by adding “strongly agree” and “agree”), “indifference” (“not agree/disagree”), and “disagreement” (by adding “disagree” and “strong disagree”). The resulting asymptotic mean entropies and asymptotic variances are shown in

Table 6. Asymptotic mean and variance of entropies of the collapsed entries of 1998.

	Mean	Variance
$H_S$	0.955	0.0000678
$H_R^{1/3}$	0.985	0.0000171
$H_R^{2/3}$	0.970	0.0000083
$H_T^{1/2}$	0.971	0.0000281
$H_T^{3/2}$	0.949	0.0000896
$H_F$	0.192	0.0000678

.

Table 7. p-values of the hypotheses of equal entropies using collapsed data in 1998.

	1998–2008	1998–2018
$H_S$	0.00000	0.0000
$H_R^{1/3}$	0.00115	0.0107
$H_R^{2/3}$	0.00000	0.0000
$H_T^{1/2}$	0.00000	0.0000
$H_T^{3/2}$	0.00000	0.0000
$H_F$	0.00000	0.0000

presents the p-values of the tests that verify the null hypothesis of the same entropy between the collapsed 1998 data (three categories), and 2008 and 2018 (five categories). These results agree with those presented in Table 5. Such an agreement suggests that, although the number of categories was reduced in 1998 from five to three, the tests based on entropies cope with the loss of information.

6. Conclusions

We presented expressions for the asymptotic distribution of the Rényi and Tsallis entropies of order q, and Fisher information. The Fisher information and the Tsallis and Shannon entropies have limit normal distribution with means and variances that depend on the underlying probability of patterns and the number of patterns. The Rényi entropy follows, asymptotically, a different distribution, cf. (35), but a Gaussian law can well approximate it. Those expressions pose no numerical challenges other than setting $0log0\doteq 0$. We verified that these asymptotic distributions are good models for data arising from both simulations with a variety of models and from the analysis of actual data.

On the one hand, the Fisher information is the one that fails more frequently to pass the Anderson–Darling normality tests. On the other hand, it does not provide evidence to reject the same hypothesis under the One-Almost-Zero model.

The distributions we present here can be used for building test statistics, as discussed by Chagas et al. [8]. Moreover, Equation (37) allows performing tests with mixed types of distributions, a situation that may appear in Internet of Things applications, in which, citing Borges et al. [17], one has to deal with “large time series data generated at different rates, of different types and magnitudes, possibly having issues concerning uncertainty, inconsistency, and incompleteness due to missing readings and sensor failures.”