Equivalence of Partition Functions Leads to Classification of Entropies and Means

Abstract

We derive a two-parameter family of generalized entropies, S_pq, and means m_pq. To this end, assume that we want to calculate an entropy and a mean for n non-negative real numbers {x₁,…,x_n}. For comparison, we consider {m₁,…,m_k} where m_i = m for all i = 1,…,k and where m and k are chosen such that the l^p and l^q norms of {x₁,…,x_n} and {m₁,…,m_k} coincide. We formally allow k to be real. Then, we define k, log k, and m to be a generalized cardinality k_pq, a generalized entropy S_pq, and a generalized mean m_pq respectively. We show that this family of entropies includes the Shannon and Rényi entropies and that the family of generalized means includes the power means (such as arithmetic, harmonic, geometric, root-mean-square, maximum, and minimum) as well as novel means of Shannon-like and Rényi-like forms. A thermodynamic interpretation arises from the fact that the l^p norm is closely related to the partition function at inverse temperature β = p. Namely, two systems possess the same generalized entropy and generalized mean energy if and only if their partition functions agree at two temperatures, which is also equivalent to the condition that their Helmholtz free energies agree at these two temperatures.

1. Introduction

Two of the most basic concepts of thermodynamics are: (a) the average of measurement outcomes and (b) the uncertainty or entropy about measurement outcomes. Consider, for example, a physical system, A, that is in contact with a heat bath at some fixed temperature, i.e., a canonical ensemble. The measurement of the system’s energy can return any one of its energy eigenstates. What then is (a) the mean energy to expect and (b) how uncertain is the prediction of the measured energy state?

We notice that, in principle, many different notions of energy mean and many different measures of entropy could be employed here. Of course, in thermodynamics, the Boltzmann factor weighted mean as well as the Shannon/von Neumann entropy are of foremost importance. In this paper, we show that also other important notions of average such as the harmonic mean, the geometric mean and the arithmetic mean arise naturally, along with generalized notions of entropy including Rényi entropies [1], all unified in a two-parameter family of notions of means and notions of entropies.

To this end, consider systems (canonical ensembles) in a heat bath. We begin by considering the simplest kind of system, namely the type of system which possesses only one energy level, E. Let us denote its degeneracy by k. Unambiguously, we should assign that system the mean $m:=E$ and the entropy $S:=\log k$. Let us denote these simple one-level systems by the term reference system.

Now, let X be a system with arbitrary discrete energy levels. Our aim is to assign X a mean and an entropy by finding that reference system M which is in some sense equivalent to X. Then we assign X the same value for the mean and entropy as the reference system M.

But how do we decide if a reference system is in some sense equivalent to system X? Given that we want the reference system M and system X to share two properties, namely a mean and an entropy, we expect any such condition for the equivalence of two systems to require two equations to be fulfilled. Further, since the properties of systems are encoded in their partition function $Z(X)$, we expect that these two equations can be expressed in terms of the partition functions of the two systems in question.

To this end, let us adopt what may be considered the simplest definition. We choose two temperatures, T₁ and T₂ and we define that a reference system is (T₁, T₂)-equivalent to system X if the partition functions of the two systems coincide with each other at these two temperatures. Since the Helmholtz free energy $A_H$ obeys $A_H=-K_{B}T\log (Z)$, where $K_B$ is the Boltzmann constant, this is the same as saying that two systems are put in the same equivalence class if their Helmholtz free energies coincide at these two temperatures.

This allows us now to assign any system X a mean and an entropy. We simply find its unique (T₁ and T₂)-equivalent reference system M. Then the mean and entropy of X are defined to be the mean and the entropy of the reference system M.

Clearly, the so-defined mean and entropy of a system X now actually depend on two temperatures, namely (T₁, T₂). As we will show below, in the limit when we let the two temperatures become the same temperature, we recover the usual Boltzmann factor-weighted mean, i.e., the usual mean energy, along with the usual Shannon/von Neumann entropy.

For general (T₁, T₂), we cover more, however. Namely, we naturally obtain a unifying 2-parameter family of notions of mean that includes for example the geometric, the harmonic, the arithmetic and the root-mean-square (RMS) means. And we obtain a unifying 2-parameter family of notions of entropy that, for example, includes the Rényi family of entropies.

To be precise, let us assume that a system X has only discrete energy levels, {E_i}, where i enumerates all the energy levels counting also possible degeneracies. Notice that {E_i} is formally what is called a multiset, because its members are allowed to occur more than once. Similarly, let us also collect the exponentiation of the negative energies $x_i:=\exp (-{\mathsf{{\rm E}}}_i)$ in the multiset $\left\{x_i\right\}$. Either multiset can be used to describe the same thermodynamic system X. Let $\beta :=1/K_{B}T$ denote the inverse temperature, where $K_B$ is the Boltzmann constant. The partition function of system X, i.e., the sum of its Boltzmann factors, then reads:

$$Z_{\beta }(X)={\displaystyle {\sum }_i{e}^{-\beta {\mathsf{{\rm E}}}_i}}={\displaystyle {\sum }_i{x_i}^{\beta }}$$

(1)

For later reference, note that the partition function is therefore related to the l^p norm of X = {x₁,…,x_n} for $p=\beta$ through $Z_{\beta }(X)=\text{\hspace{0.17em}\hspace{0.17em}}\left|\right|X|{|}_{\beta }^{\beta }$.

Now the key definition is that we call two physical systems $({\beta }_1,{\beta }_2)$-equivalent if their partition functions coincide at the two inverse temperatures $({\beta }_1,{\beta }_2)$, i.e., systems $X$ and $M$ are $({\beta }_1,{\beta }_2)$-equivalent if $Z_{{\beta }_1}(X)=Z_{{\beta }_1}(M)$ and $Z_{{\beta }_2}(X)=Z_{{\beta }_2}(M)$. To be more explicit, one may also call such systems $({\beta }_1,{\beta }_2)$-partition function equivalent, or also $({\beta }_1,{\beta }_2)$-Helmholtz free energy equivalent, but we will here use the term $({\beta }_1,{\beta }_2)$-equivalent for short.

In particular, for any given system X, let us consider the $({\beta }_1,{\beta }_2)$-equivalent reference system M which possesses just one energy level, with energy E₀ and degeneracy k, where we formally allow k to be any positive number. E₀ and k are then determined by the two conditions that the partition function of M is to coincide with that of X at the two inverse temperatures ${\beta }_1$ and ${\beta }_2$. Then, we define $S_{{\beta }_1,{\beta }_2}(X):=\log k$ to be the generalized entropy, and $m_{{\beta }_1{\beta }_2}(X):={\mathsf{{\rm E}}}_0$ to be the generalized mean energy of system $X$ with respect to the temperatures $({\beta }_1,{\beta }_2)$.

We will explore the properties of these families of generalized entropies and means in the subsequent sections. First, however, let us consider the special limiting case when the two temperatures coincide (i.e., ${\beta }_1={\beta }_2=\beta$). As will be detailed in the subsequent sections of the manuscript, in this limiting case, the two equivalence conditions of partition functions can be shown to reduce to:

$${\left(Z_{\beta }(X)\right)}^{1/\beta }={\left(Z_{\beta }(M)\right)}^{1/\beta }$$

(2a)

$$\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial \beta $}\right.{\left(Z_{\beta }(X)\right)}^{1/\beta }=\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial \beta $}\right.{\left(Z_{\beta }(M)\right)}^{1/\beta }$$

(2b)

which can be shown to be equivalent to:

$$Z_{\beta }(X)=Z_{\beta }(M)$$

(3a)

$${\displaystyle {\sum }_i{\mathsf{{\rm E}}}_i\frac{e^{-\beta {\mathsf{{\rm E}}}_i}}{{\displaystyle {\sum }_j{e}^{-\beta {\mathsf{{\rm E}}}_j}}}}={\mathsf{{\rm E}}}_0$$

(3b)

The conditions (3a) and (3b) physically mean that systems $X$ and $M$ have the same partition function and average energy, respectively, at the inverse temperature $\beta$. Notice that this is also the same as saying that the two systems have the same average energy and the same Helmholtz free energy at the inverse temperature $\beta$. Now by employing either pair of conditions, (2) or (3), we then recover indeed the usual thermodynamic entropy of the system X, which is given in the Shannon form at the inverse temperature $\beta$ by

$$S_{\beta \beta }(X)=-{\displaystyle {\sum }_i\frac{e^{-\beta {\mathsf{{\rm E}}}_i}}{{\displaystyle {\sum }_j{e}^{-\beta {\mathsf{{\rm E}}}_j}}}\log \left(\frac{e^{-\beta {\mathsf{{\rm E}}}_i}}{{\displaystyle {\sum }_j{e}^{-\beta {\mathsf{{\rm E}}}_j}}}\right)}$$

(4)

The proofs of Equations (2)–(4) are straightforward. We will spell them out in detail in the subsequent sections where the setting is abstract mathematical.

Before we begin the mathematical treatment, let us remark that entropy is not only a cornerstone of thermodynamics but it is also crucial in information theory. Due to its universal significance, measures of uncertainty in the form of an entropy have been proposed by physicists and mathematicians for over a century [2]. Our approach here for deriving a generalized family of entropies was originally motivated by basic questions regarding the effective dimensionality of multi-antenna systems (e.g., [3,4,5]). After initial attempts in [6] and later in [5,7], we here give for the first time a comprehensive derivation with the proofs and we also include the family of generalized means.

The manuscript is organized as follows. In Section 2, we introduce the proposed family of entropies and means mathematically, and also show some special cases thereof. The axiomatic formulation is presented in Section 3, followed by the study of resulting properties in Section 4. Proofs are provided in the appendices.

2. Mathematical Definition of Generalized Entropies and Means

Let $X=\{x_i:i=1,2,\dots ,K_X\}$ be a multiset of real non-negative numbers, where $K_X$ denotes the cardinality of X. We assume that X possesses at least one non-zero element. Further, let $p,q$ be arbitrary fixed real numbers obeying $pq\ge 0$. Let $M=\{m_i:m_i=m,m>0,i=1,2,\dots ,k\}$ be a reference multiset possessing exactly one real positive element $m$, which is of multiplicity $k\ge 1$. We introduce the following definitions:

• $N_X$ is the number of non-zero elements of X, therefore $N_X\le K_X$. To simplify notation, the subscript X is omitted when dealing with one multiset at hand.
• $K_{X\max }$ is the multiplicity of the maximum elements of X
• $K_{X\min }$ is the multiplicity of the minimum elements of X

Our objective is to determine suitable values for $m$ and $k$, possibly non-integer, that can serve as mean and effective cardinality of X, respectively, namely by imposing a suitable criterion for the equivalence of X to a reference multiset M. Having two unknowns ($m$ and $k$) in M, we need two equivalence conditions. We choose to impose the equivalence of the p-norms and the q-norms:

$$\begin{array}{l}{\Vert X\Vert }_p\equiv {\Vert M\Vert }_p=m{k}^{1/p}\\ {\Vert X\Vert }_q\equiv {\Vert M\Vert }_q=m{k}^{1/q}\end{array}$$

(5)

Here, the p-norm ${\Vert \cdot \Vert }_p$ is defined as usual through:

$${\Vert X\Vert }_p:={\left({\displaystyle {\sum }_{i=1}^{K_X}{(x_i)}^p}\right)}^{1/p}$$

(6)

with the proviso that ${(x_i)}^p$ is replaced by 0 if $x_i=0$ and $p\le 0$. We remark that, for $p<1$, (6) is merely a quasi-norm since the triangle inequality does not hold. Note the singularity $\underset{p\to 0^{+}}{\lim }{\Vert X\Vert }_p=\infty$.

Solving for $k$ and $m$ in (5), we obtain:

$$\begin{array}{l}{k}_{pq}(X)={\left[{\Vert X\Vert }_p/{\Vert X\Vert }_q\right]}^{1/\left(\frac{1}{p}-\frac{1}{q}\right)}p\ne q\\ m_{pq}(X)={\left[{\Vert X\Vert }_p^p/{\Vert X\Vert }_q^q\right]}^{\frac{1}{p-q}}p\ne q\end{array}$$

(7)

We call $k_{pq}(X)$ and $m_{pq}(X)$ the norm-induced effective cardinality and generic mean of order $p,q$ for the multiset X, respectively. Let us now express (7) in a logarithmic form and define the entropy $S_{pq}(X)$ as follows:

$$S_{pq}(X):=\log k_{pq}(X)=\frac{\log {\Vert X\Vert }_p-\log {\Vert X\Vert }_q}{\frac{1}{p}-\frac{1}{q}}p\ne q$$

(8a)

$$\log m_{pq}(X)=\frac{p\log {\Vert X\Vert }_p-q\log {\Vert X\Vert }_q}{p-q}p\ne q$$

(8b)

Notice that $S_{pq}(X)=S_{qp}(X)$ and $m_{pq}(X)=m_{qp}(X)$, i.e., both the entropy and mean are symmetrical with respect to the order $p,q$.

Next, we express $S_{pq}(X)$ and $m_{pq}(X)$ in the limiting case when $p\to q$. For $S_{qq}(X)$ we find:

$$\begin{array}{l}{S}_{qq}(X):=\log k_{qq}(X)\\ \underset{p\to q}{=\lim }{S}_{pq}(X)\\ \underset{p\to q}{=\lim }\frac{\log {\Vert X\Vert }_p-\log {\Vert X\Vert }_q}{\frac{1}{p}-\frac{1}{q}}\\ \stackrel{\text{l}\text{'}\text{H}\widehat{\text{o}}\text{pital}}{=}-q^2\frac{\frac{\partial }{\partial q}{\Vert X\Vert }_q}{{\Vert X\Vert }_q}\\ =-{\displaystyle {\sum }_{i=1}^K\frac{{x_i}^q}{{\displaystyle {\sum }_{j=1}^K{x_j}^q}}}\log \left(\frac{{x_i}^q}{{\displaystyle {\sum }_{j=1}^K{x_j}^q}}\right)\end{array}$$

(9)

where the last step is obtained by straightforward manipulations. Similarly, we find for $m_{qq}(X)$:

$$\begin{array}{l}{m}_{qq}(X)\underset{p\to q}{=\lim }{m}_{pq}(X)={\Vert X\Vert }_q/k_{qq}^{1/q}\\ \log m_{qq}(X)\underset{p\to q}{=\lim }\frac{p\log {\Vert X\Vert }_p-q\log {\Vert X\Vert }_q}{p-q}\\ \stackrel{\text{l}\text{'}\text{H}\widehat{\text{o}}\text{pital}}{=}\log {\Vert X\Vert }_q+q\frac{\frac{\partial }{\partial q}{\Vert X\Vert }_q}{{\Vert X\Vert }_q}\\ =\frac{\frac{\partial }{\partial q}{\Vert X\Vert }_q^q}{{\Vert X\Vert }_q^q}\\ ={\displaystyle {\sum }_{i=1}^K\frac{{x_i}^q}{{\displaystyle {\sum }_{j=1}^K{x_j}^q}}}\log x_i\end{array}$$

(10)

where in the second to last step, we used the fact that $\frac{d}{dx}g{(x)}^{f(x)}=g^f\left\{\frac{df}{dx}\log g+\frac{f}{g}\frac{dg}{dx}\right\}$, and the last step is obtained by straightforward manipulations.

It is worthwhile to mention the following useful relation linking $S_{qq}(X)$, ${\Vert X\Vert }_q$ and $m_{qq}(X)$, which is readily deduced from (9) and (10):

$$\log m_{qq}(X)=\log {\Vert X\Vert }_q-S_{qq}(X)/q$$

(11)

We remark that in the early phase of this work [5], each author independently suggested either $k_{pq}(X)$ or $k_{qq}(X)$ as two possible distinct notions for the effective cardinality. In [7], it was reported that the average energy and Shannon entropy of a thermodynamic system are obtained by starting from equivalence of partition functions of two systems at two temperatures when the two temperatures coincide as mentioned in the introduction. Clearly, the limiting operation in (9) makes the connection and establishes (7) as the general definition of this norm-induced family of entropies and means.

In fact, for the case of degenerate order, ($p\to q$), the quantities $k_{qq}(X)$, $S_{qq}(X)$, and $m_{qq}(X)$, could have been obtained as well through a differential equivalence of the $q$-norm. To see this, we impose the following two conditions:

$$\begin{array}{l}{\Vert X\Vert }_q\equiv {\Vert M\Vert }_q\\ \frac{\partial }{\partial q}{\Vert X\Vert }_q\equiv \frac{\partial }{\partial q}{\Vert M\Vert }_q\end{array}$$

(12)

After employing $\frac{d}{dx}g{(x)}^{f(x)}=g^f\left\{\frac{df}{dx}\log g+\frac{f}{g}\frac{dg}{dx}\right\}$ and solving for $k$ and $m$, we ultimately obtain $k_{qq}(X)$ and $m_{qq}(X)$ as given by (9) and (10). The condition (12) is the mathematical equivalent of the aforementioned physical condition (2) imposed on the two thermodynamic systems, which yielded the Shannon entropy form (4).

From (9), it is obvious that $S_{qq}(X)$ is the Shannon entropy of the distribution $\left\{{x_i}^q/{\displaystyle {\sum }_{i=1}^K{x_i}^q}\right\}$, which is called the escort distribution of order $q$ [8] of $\left\{x_i/{\displaystyle {\sum }_{i=1}^K{x}_i}\right\}$. On the other hand, $S_{pq}(X)$ is a more general expression of the Rényi entropy of order $\alpha$. For a probability distribution $P=\left\{p_i\right\}$, the Rényi entropy of order $\alpha$ is given by [9]:

$$S_{\alpha }^{(R)}(P):=\frac{1}{1-\alpha }\log \frac{{\displaystyle {\sum }_i{p_i}^{\alpha }}}{{\displaystyle {\sum }_i{p}_i}},\alpha >0,\alpha \ne 1$$

(13)

By setting the order $p=1$ in $S_{pq}$, we obtain from (8):

$$\begin{array}{l}{S}_{1q}(X)=\frac{\log {\displaystyle {\sum }_{i=1}^K{x_i}^q}-q\log {\displaystyle {\sum }_{j=1}^K{x}_j}}{1-q}q>0,q\ne 1\\ =\frac{1}{1-q}\log {\displaystyle \sum _{i=1}^K{\left(\frac{x_i}{{\displaystyle {\sum }_{j=1}^K{x}_j}}\right)}^q}\end{array}$$

(14)

By comparing (13) and (14), we readily identify $S_{1q}(X)$ as the Rényi entropy of order $q$ for a complete statistical distribution given by $\left\{x_i/{\displaystyle {\sum }_{i=1}^K{x}_i}\right\}$, where the multiset elements add to 1. Formally:

$$S_{1q}(X)=S_q^{(R)}\left(\left\{x_i/{\displaystyle {\sum }_{i=1}^K{x}_i}\right\}\right)$$

(15)

In the degenerate case (when $q\to 1$), $S_{11}(X)$ is the Shannon entropy of the latter distribution. For $p\ne 1$, $S_{pq}(X)$ from (8) can be rearranged as a generalization of (13):

$$\begin{array}{l}{S}_{pq}(X)=\frac{p}{p-q}\log {\displaystyle \sum _{i=1}^K{\left(\frac{{x_i}^p}{{\displaystyle {\sum }_{j=1}^K{x_j}^p}}\right)}^{q/p}}\\ =\frac{1}{1-q/p}\log {\displaystyle \sum _{i=1}^K{\left(\frac{{x_i}^p}{{\displaystyle {\sum }_{j=1}^K{x_j}^p}}\right)}^{q/p}}\\ =S_{q/p}^{(R)}\left(\{{x_i}^p/{\displaystyle {\sum }_{i=1}^K{x_i}^p}\}\right)\end{array}$$

(16)

which can be viewed as the Rényi entropy of order $q/p$ for the $p^{th}$ order escort distribution $\left\{{x_i}^p/{\displaystyle {\sum }_{i=1}^K{x_i}^p}\right\}$.

Rényi defined his entropy for $p,q\ge 0$. We relax this condition further and allow $S_{pq}(X)$ and $m_{pq}(X)$ to be defined for any real indices $p,q$ such that $pq\ge 0$. Accordingly, we obtain the following properties:

$$\begin{array}{l}{S}_{-p,-q}(\{x_i\})=S_{p,q}(\{x_i^{-1}:x_i\ne 0\})\\ m_{-p,-q}(\{x_i\})=m_{p,q}^{-1}(\{x_i^{-1}:x_i\ne 0\})\end{array}$$

(17)

When at least one order $p,q$ is zero, we find the interesting results:

$$k_{0q}(X)=N_X$$

(18)

$$S_{0q}(X)=\log N_X$$

(19)

$$m_{0q}(X)=\{\begin{array}{cc}{\left(\frac{{\displaystyle {\sum }_{i=1}^{N_X}{x_i}^q}}{N_X}\right)}^{1/q}& x_i\ne 0,q\ne 0\\ {\left({\displaystyle {\prod }_{i=1}^{N_X}{x}_i}\right)}^{1/N_X}& x_i\ne 0,q\to 0\end{array}$$

(20)

We recognize $S_{0q}(X)$ in (19) as the Hartley entropy [10], which will be shown later to be the maximum value of any entropy. From (20), we obtain a famous family of generic p-means of the non-zero elements of X: particularly $m_{0,-\infty },m_{0,-1},m_{0,1},m_{0,2},m_{0,\infty }$ are the minimum, harmonic mean, arithmetic mean, root-mean-square mean, and maximum, respectively. In the limiting case $q\to 0$, we obtain $m_{0,0}$, which is the geometric mean. In Table 1, we summarize these and other particular cases of means and entropies at specific $p,q$. The key point is that each $p,q$ uniquely defines an entropy $S_{pq}$ with a corresponding mean $m_{pq}$, such that each pair of $S_{pq}$ and $m_{pq}$ is coupled in this sense.

Table 1. Special cases of $S_{pq}$ and $m_{pq}$. Note that $S_{pq}=S_{qp}$ and $m_{pq}=m_{qp}$ (Property 4.3).

Order $p,q$	$S_{pq}(X)$	$S_{pq}$ name	$m_{pq}(X)$	$m_{pq}$ name
$0,qq\ne 0$	$\log N_X$	Boltzmann-Hartley entropy	${\left(\frac{{\displaystyle {\sum }_{i=1}^{N_X}{x_i}^q}}{N_X}\right)}^{1/q}$	Generic $q$ mean. Specific q values are harmonic (-1), arithmetic (1), root-mean-square (2), maximum ($\infty$), minimum ($-\infty$)
0,0	$\log N_X$	Boltzmann-Hartley entropy	${\left({\displaystyle {\prod }_{i=1}^{N_X}{x}_i}\right)}^{1/N_X}$	Geometric mean
$\infty ,\infty$	$\log K_{X\max }$		$\max (X)$	maximum
$\infty ,qq>0$	$\log \left(K_{X\max }+{\displaystyle \sum _{x_i\ne \max (x_i)}{\left(\frac{x_i}{\max (x_i)}\right)}^q}\right)$		$\max (X)$	maximum
$-\infty ,-\infty$	$\log K_{X\min }$		$\min (X)$	minimum
$-\infty ,qq<0$	$\log \left(K_{X\min }+{\displaystyle \sum _{x_i\ne \min (x_i)}{\left(\frac{x_i}{\min (x_i)}\right)}^q}\right)$		$\min (X)$	minimum
$1,qq>0$	$\frac{1}{1-q}\log {\displaystyle \sum _{i=1}^K{\left(\frac{x_i}{{\displaystyle {\sum }_{j=1}^K{x}_j}}\right)}^q}$	Rényi entropy, order $q$, of the complete distribution $\left\{x_i/{\displaystyle {\sum }_{i=1}^K{x}_i}\right\}$	$\frac{{\displaystyle {\sum }_{i=1}^K{x}_i}}{\exp S_{1q}(X)}$	“Rényi-like” mean
1,1	$-{\displaystyle {\sum }_{i=1}^K\frac{x_i}{{\displaystyle {\sum }_{j=1}^K{x}_j}}}\log \left(\frac{x_i}{{\displaystyle {\sum }_{j=1}^K{x}_j}}\right)$	Gibbs-Shannon entropy	$\frac{{\displaystyle {\sum }_{i=1}^K{x}_i}}{\exp S_{11}(X)}$	“Shannon-like” mean

Table 1. Special cases of $S_{pq}$ and $m_{pq}$. Note that $S_{pq}=S_{qp}$ and $m_{pq}=m_{qp}$ (Property 4.3).

Order $p,q$	$S_{pq}(X)$	$S_{pq}$ name	$m_{pq}(X)$	$m_{pq}$ name
$0,qq\ne 0$	$\log N_X$	Boltzmann-Hartley entropy	${\left(\frac{{\displaystyle {\sum }_{i=1}^{N_X}{x_i}^q}}{N_X}\right)}^{1/q}$	Generic $q$ mean. Specific q values are harmonic (-1), arithmetic (1), root-mean-square (2), maximum ($\infty$), minimum ($-\infty$)
0,0	$\log N_X$	Boltzmann-Hartley entropy	${\left({\displaystyle {\prod }_{i=1}^{N_X}{x}_i}\right)}^{1/N_X}$	Geometric mean
$\infty ,\infty$	$\log K_{X\max }$		$\max (X)$	maximum
$\infty ,qq>0$	$\log \left(K_{X\max }+{\displaystyle \sum _{x_i\ne \max (x_i)}{\left(\frac{x_i}{\max (x_i)}\right)}^q}\right)$		$\max (X)$	maximum
$-\infty ,-\infty$	$\log K_{X\min }$		$\min (X)$	minimum
$-\infty ,qq<0$	$\log \left(K_{X\min }+{\displaystyle \sum _{x_i\ne \min (x_i)}{\left(\frac{x_i}{\min (x_i)}\right)}^q}\right)$		$\min (X)$	minimum
$1,qq>0$	$\frac{1}{1-q}\log {\displaystyle \sum _{i=1}^K{\left(\frac{x_i}{{\displaystyle {\sum }_{j=1}^K{x}_j}}\right)}^q}$	Rényi entropy, order $q$, of the complete distribution $\left\{x_i/{\displaystyle {\sum }_{i=1}^K{x}_i}\right\}$	$\frac{{\displaystyle {\sum }_{i=1}^K{x}_i}}{\exp S_{1q}(X)}$	“Rényi-like” mean
1,1	$-{\displaystyle {\sum }_{i=1}^K\frac{x_i}{{\displaystyle {\sum }_{j=1}^K{x}_j}}}\log \left(\frac{x_i}{{\displaystyle {\sum }_{j=1}^K{x}_j}}\right)$	Gibbs-Shannon entropy	$\frac{{\displaystyle {\sum }_{i=1}^K{x}_i}}{\exp S_{11}(X)}$	“Shannon-like” mean

A typical plot for ${\Vert X\Vert }_p$, $S_{pp}$ and $m_{pp}$ is shown in log scale in Figure 1, illustrating some of the properties to be discussed hereafter. In particular, we notice:

• $\log {\Vert X\Vert }_p$ has a two-sided singularity at $p=0$.
• $S_{pp}$ is non-decreasing/non-increasing for negative/positive $p$, respectively, and is guaranteed to be maximized at $p=0$. This is discussed more generally in Property 4.6.
• $m_{pp}$ ranges from $\min (X)$ to $\max (X)$ and is always non-decreasing with respect to $p$. This is discussed more generally in Property 4.6.
• $p=1$ has a specific property of making $m_{pp}{k}_{pp}={\Vert X\Vert }_p$.

Figure 1. Typical plot for ${\Vert X\Vert }_p$, $S_{pp}$ and $m_{pp}$ in log scale.

Figure 1. Typical plot for ${\Vert X\Vert }_p$, $S_{pp}$ and $m_{pp}$ in log scale.

3. An Axiomatic Approach to the Generalized Entropies and Means

In order to simplify the conceptual underpinnings, let us now describe the generalized entropies $S_{pq}$ and means $m_{pq}$, through two simple axioms.

3.1. Axioms for the Generalized Entropy

Let $p,q$ be fixed real numbers obeying $pq\ge 0$. Consider a map, $S_{pq}$, which maps multisets of positive real numbers into the real numbers. We call $S_{pq}$ a generalized entropy of order $p,q$ if it obeys the following two axioms:

Entropy Axiom 1: 

$S_{pq}(M)$ of a uniform multiset $M=\{m_0,m_0,\dots ,m_0\}$, $m_0>0$, with multiplicity $k$, equals $\log k$, (where the base of the logarithm is arbitrarily chosen), i.e.,

$$M=\{m_i:m_i=m_0,m_0>0,i=1,2,\dots ,k\}\Rightarrow S_{pq}(M)=\log k$$

(21)

Entropy Axiom 2: 

If $p\text{\hspace{0.17em}\hspace{0.17em}}\ne q$, the map $S_{pq}$ depends only on the ratio of the multiset’s p and q norms, i.e., $S_{pq}$ is some function, $f_{pq}$, of this ratio:

$$S_{pq}(X):=f_{pq}\left(\frac{{\Vert X\Vert }_p}{{\Vert X\Vert }_q}\right)p\ne q$$

(22a)

If $p=q$, the map $S_{qq}$ depends only on the ratio of the multiset’s q norm to its derivative, i.e., $S_{pq}$ is some function, $f_q$, of this ratio:

$$S_{qq}(X):=f_q\left(\frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert X\Vert }_q}{{\Vert X\Vert }_q}\right)p\to q$$

(22b)

To see that (22b) arises in the limit from (22a) we notice that, since the logarithm is strictly monotone, (22a) is equivalent to saying that $S_{pq}$ is some function $h_{pq}$ of some finite number times the logarithm of the ratio of norms:

$$\begin{array}{l}{S}_{pq}(X)=h_{pq}\left(\frac{1}{p-q}\log \frac{{\Vert X\Vert }_p}{{\Vert X\Vert }_q}\right)\\ =h_{pq}\left(\frac{\log {\Vert X\Vert }_p-\log {\Vert X\Vert }_q}{p-q}\right)\end{array}$$

(23)

Choosing $p=q+\epsilon$ and taking the limit $\epsilon \to 0$ we obtain (22b) with $f_q=h_{qq}$.

Proposition: The two entropy axioms in (21) and (22) uniquely define $S_{pq}(X)$ and $S_{qq}(X)$, namely as given in section 2 by (8) and (9).

Proof: 

Entropy Axiom 2 implies that for any two multisets $X$ and $Y$:

$$\frac{{\Vert X\Vert }_p}{{\Vert X\Vert }_q}=\frac{{\Vert Y\Vert }_p}{{\Vert Y\Vert }_q}\Rightarrow S_{pq}(X)=S_{pq}(Y)forp\ne q$$

(24a)

$$\frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert X\Vert }_q}{{\Vert X\Vert }_q}=\frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert Y\Vert }_q}{{\Vert Y\Vert }_q}\Rightarrow S_{qq}(X)=S_{qq}(Y)forp=q$$

(24b)

Choosing for$Y$the uniform multiset $M$ of Axiom 1, and taking the logarithm of both sides yields:

$$\begin{array}{l}\log \frac{{\Vert X\Vert }_p}{{\Vert X\Vert }_q}=\left(\frac{1}{p}-\frac{1}{q}\right)\log kp\ne q\\ \frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert X\Vert }_q}{{\Vert X\Vert }_q}=-\frac{\log k}{q^2}p=q\end{array}$$

(25)

Using that $S_{pq}(X)=S_{pq}(Y)=S_{pq}(M)=\log k$, we now uniquely obtain the formulas for $S_{pq}(X)$ and $S_{qq}(X)$ given in (8) and (9), respectively. We note that the functions $f_{pq}$ and $f_q$ are therefore:

$$\begin{array}{l}{f}_{pq}(·)=\frac{\log (·)}{\frac{1}{p}-\frac{1}{q}}p\ne q\\ f_q(·)=-q^2(·)p=q\end{array}$$

(26)

Remarks:

• Even though $k$ is treated as an integer representing the multiplicity in Axiom 1, this condition is tacitly relaxed in Axiom 2 to include non-integer values, which we may call the effective cardinality (or effective dimensionality) of order $p,q$.
• The logarithmic measure of Axiom 1 is directly connected to the celebrated Boltzmann entropy formula $S^{(B)}=K_B\log W$, where $K_B$ is the Boltzmann constant and $W$ is the number of the microstates in the system. The logarithmic measure is also connected to the so-called “Hartley’s measure” [11,12], which indicates the non-specificity [13] and does not require a probability distribution assumption. In Axiom 1, a multiset of equal positive numbers is all that is required. In fact, Axiom 1 encompasses the additivity and monotonicity axioms [12,13], which are equivalent to the earlier Khinchin’s axioms of additivity, maximality and expansibility [14].
• In Axiom 2, note that the p-norm definition is relaxed to include the values $p<1$, which would result in the triangle inequality to be violated should the multisets be treated as vectors.

3.2. Axioms for the Generalized Mean

We define the p^th moment of the multiset $X=\left\{x_i\right\}$ as:

$$p^{th}moment(X):={\Vert X\Vert }_p^p={\displaystyle {\sum }_i{x_i}^p}$$

(27)

The nomenclature “p^th moment” is motivated by the fact that for the density function $r(x):={\displaystyle {\sum }_i\delta (x-x_i)}$, where $\delta (x)$ is the Dirac delta function, the p^th moment is indeed:

$$\begin{array}{l}{p}^{th}moment(X):={\displaystyle \int x^{p}r(x)dx}\\ ={\displaystyle \int x^p{\displaystyle {\sum }_i\delta (x-x_i)}dx}\\ ={\displaystyle {\sum }_i{x_i}^p}={\Vert X\Vert }_p^p\end{array}$$

(28)

Let $p,q$ be fixed real numbers obeying $pq\ge 0$. Consider a map, $m_{pq}$, which maps multisets of positive real numbers into the real numbers. We call $m_{pq}$ a generalized mean of order $p,q$ if it obeys the following two axioms:

Mean Axiom 1: 

$m_{pq}(M)$ of a uniform multiset $M=\{m_0,m_0,\dots ,m_0\}$, $m_0>0$ is $m_0$. Formally:

$$M=\{m_i:m_i=m_0,m_0>0,i=1,2,\dots ,k\}\Rightarrow m(M):=m_0$$

(29)

Mean Axiom 2: 

If $p\text{\hspace{0.17em}\hspace{0.17em}}\ne q$, the map $m_{pq}$ depends for any multiset X only on the ratio of the multiset’s p^th and q^th moments, $\left|\right|X|{|}_p^p$ and $\left|\right|X|{|}_q^q$, i.e., $m_{pq}$ is some function, $g_{pq}$, of their ratio:

$$m_{pq}(X):=g_{pq}\left(\frac{{\Vert X\Vert }_p^p}{{\Vert X\Vert }_q^q}\right)p\ne q$$

(30a)

If $p=q$, the map $m_{qq}$ is a function only of the ratio:

$$m_{qq}(X):=g\left(\frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert X\Vert }_q^q}{{\Vert X\Vert }_q^q}\right)p=q$$

(30b)

The fact that (30b) is the limit of (30a) follows by the same reasoning as in (23).

Proposition: 

The two mean axioms in (29) and (30) uniquely define $m_{pq}$, namely as given in (7) and (10).

Proof: 

Axiom 2 implies for any two multisets $X$ and $Y$ that:

$$\begin{array}{l}\frac{{\Vert X\Vert }_p^p}{{\Vert X\Vert }_q^q}=\frac{{\Vert Y\Vert }_p^p}{{\Vert Y\Vert }_q^q}\Rightarrow m_{pq}(X):=m_{pq}(Y)p\ne q\\ \frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert X\Vert }_q^q}{{\Vert X\Vert }_q^q}=\frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert Y\Vert }_q^q}{{\Vert Y\Vert }_q^q}\Rightarrow m_{qq}(X):=m_{qq}(Y)p=q\end{array}$$

(31)

Choosing the multiset $Y$ to be the uniform multiset $M$ from (29) we obtain

$$\begin{array}{l}\frac{{\Vert X\Vert }_p^p}{{\Vert X\Vert }_q^q}=m_0^{p-q}p\ne q\\ \frac{\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert X\Vert }_q^q}{{\Vert X\Vert }_q^q}=\log m_{0}p=q\end{array}$$

(32)

We can now use that $m(X)=m(M)=m_0$, to obtain $m_{pq}(X)$ and $m_{qq}(X)$ as given in (7) and (10), respectively. Accordingly the functions $g_{pq}$ and $g$ are found to be

$$\begin{array}{l}{g}_{pq}(·)={(·)}^{\frac{1}{p-q}}p\ne q\\ g(·)=\exp (·)p=q\end{array}$$

(33)

We have obtained axiomatizations of the generalized entropies and means which revealed, in particular, that the generalized entropies can be characterized as those entropies that cover the reference multiset case (the multiset of equal elements) and that are functions of only the ratio of the multisets’ $l^p$ and $l^q$ norms. Similarly, the axiomatization also revealed that the generalized means can be characterized as those means which cover the reference multiset case and which are functions of only the ratio of the multisets’ p^th and q^th moments $\left|\right|X|{|}_p^p$ and $\left|\right|X|{|}_q^q$. We will now develop an axiomatization that links up with Section 2, yielding simultaneously a unique family of generalized entropies and means.

3.3. Unifying Axioms for Generalized Entropies and Means

We notice that, as is straightforward to verify:

$$\frac{{\Vert X\Vert }_p}{{\Vert X\Vert }_q}=\frac{{\Vert Y\Vert }_p}{{\Vert Y\Vert }_q}\text{\hspace{0.17em}and\hspace{0.17em}}\frac{{\Vert X\Vert }_p^p}{{\Vert X\Vert }_q^q}=\frac{{\Vert Y\Vert }_p^p}{{\Vert Y\Vert }_q^q}\iff {\Vert X\Vert }_p={\Vert Y\Vert }_p\text{\hspace{0.17em}and\hspace{0.17em}}{\Vert X\Vert }_q={\Vert Y\Vert }_q$$

(34)

This means that we can describe the generalized entropies and means also through a unifying set of axioms. To this end, let $p,q$ be fixed real numbers obeying $pq\ge 0$. Consider maps $S_{pq}$ and $m_{pq}$, which map multisets of positive real numbers into the real numbers. We call $S_{pq}$ and $m_{pq}$ generalized entropies and means of order $p,q$ respectively, if they obey the following two axioms:

Unifying Axiom 1: 

$S_{pq}$ and $m_{pq}$ applied to a multiset of k equal elements $M=\{m_0,m_0,\dots ,m_0\}$, $m_0>0$ yield the values $\log k$ and $m_0$, respectively.

Unifying axiom 2: 

$$\begin{array}{l}{\Vert X\Vert }_p={\Vert Y\Vert }_{p}and{\Vert X\Vert }_q={\Vert Y\Vert }_q\Rightarrow S_{pq}(X):=S_{pq}(Y)and{m}_{pq}(X):=m_{pq}(Y)p\ne q\\ {\Vert X\Vert }_q={\Vert Y\Vert }_{q}and\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert X\Vert }_q=\raisebox{1ex}{$\partial $}\!\left/ \!\raisebox{-1ex}{$\partial q$}\right.{\Vert Y\Vert }_q\Rightarrow S_{qq}(X):=S_{qq}(Y)and{m}_{qq}(X):=m_{qq}(Y)p=q\end{array}$$

(35)

Proposition: 

The maps $S_{pq}$ and $m_{pq}$ are unique and given by Equations (7)–(10).

Proof: 

The proofs are straightforward and proceed similarly to the proofs of the propositions related to the entropy and mean axioms.

4. Properties of $S_{pq}$ and $m_{pq}$

We list in this section useful properties of $m_{pq}$, $S_{pq}$ and $k_{pq}$ with proofs in the appendix. The definitions of $m_{pq}$, $S_{pq}$ and $k_{pq}$ are given by (7)–(10). We also add two plots for an example multiset $X=\{10;9;8;7;6;0.5;0.4;0.3;0.2;0.1\}$ in Figure 2 and Figure 3 in order to provide some numerical illustration of the properties hereunder.

Figure 2. Numerical example showing the multiset elements $\left\{x_i\right\}$ versus their index$i$; with their corresponding mean $m_{pq}$ and effective cardinality $k_{pq}$ for different values of $p,q$.

Figure 2. Numerical example showing the multiset elements $\left\{x_i\right\}$ versus their index$i$; with their corresponding mean $m_{pq}$ and effective cardinality $k_{pq}$ for different values of $p,q$.

Figure 3. Numerical example for the multiset $X$ showing its mean $m_{pq}$ and effective cardinality $k_{pq}$ plotted versus $q$ for fixed values of $p$.

Figure 3. Numerical example for the multiset $X$ showing its mean $m_{pq}$ and effective cardinality $k_{pq}$ plotted versus $q$ for fixed values of $p$.

4.1. Scaling

Given a multiset $X=\{x_i:i=1,2,\dots ,K\}$ and a constant $\gamma >0$, we have:

$$\begin{array}{l}{S}_{pq}(\{\gamma x_i:i=1,2,\dots ,K\})=S_{pq}(X)\\ m_{pq}(\{\gamma x_i:i=1,2,\dots ,K\})=\gamma m_{pq}(X)\end{array}$$

(36)

That is the entropy is invariant of the scaling, whereas the mean varies linearly with scaling. The proof is straightforward, based on (7).

4.2. Symmetry with Respect to the Elements of $X$

$S_{pq}(X)$ does not depend on the order of the elements of $X$.

4.3. Symmetry with Respect to the Order $p,q$

By exchanging$p$ and $q$ in (8), we readily find that:

$$S_{pq}(X)=S_{qp}(X)$$

(37)

4.4. Sign Change of the Order $p,q$

From (17), for $pq\ge 0$, we obtain:

$$\begin{array}{l}{S}_{-p,-q}(\{x_i\})=S_{p,q}(\{x_i^{-1}:x_i\ne 0\})\\ m_{-p,-q}(\{x_i\})=m_{p,q}^{-1}(\{x_i^{-1}:x_i\ne 0\})\end{array}$$

(38)

4.5. Range of $S_{pq}(X)$

Let $N_X\ge 1$ be the number of the non-zero elements in $X$. Therefore:

$$0\le S_{pq}(X)\le \log N_{X}pq\ge 0$$

(39)

The minimum value occurs when $X$ has exactly one non-zero element. For $p,q\ne 0$, the maximum value occurs when all the non-zero elements of $X$ are equal. When either $p$ or $q$ is zero, $S_{pq}(X)$ yields the maximum value, $\log N_X$, for any distribution of $X$, which is physically intuitive since the zero^th order renders all the non-zero multiset elements to an equal value and thus we reach the equiprobable case leading to maximum entropy. The proof of this property is in Appendix A.3.

4.6. Monotonicity of $S_{pq}(X)$ and $m_{pq}(X)$ with respect to $p,q$

We have the following results for the monotonicity of $S_{pq}(X)$ and $m_{pq}(X)$ with respect to $p,q$. The proofs are in Appendix B and Appendix C, respectively:

$$\begin{array}{l}q\frac{\partial }{\partial p}{S}_{pq}\le 0p\ne q,pq\ge 0\\ p\frac{\partial }{\partial p}{S}_{pp}\le 0p\to q\end{array}$$

(40)

$$\begin{array}{l}\frac{\partial }{\partial p}{m}_{pq}\ge 0p\ne q,pq\ge 0\\ \frac{\partial }{\partial p}{m}_{pp}\ge 0p\to q\end{array}$$

(41)

where equality holds when all the non-zero elements$x_i$are equal. Accordingly, for $p\ne q$ and $pq\ge 0$, by fixing one order (say $q$), $m_{pq}$ is always non-decreasing with respect to the other order ($p$); whereas $S_{pq}$ is non-decreasing for $p<0$, non-increasing for $p>0$, with a maximum value at $p=0$. The result is true when switching $p,q$ from the symmetry Property 4.3.

Similarly, for the degenerate case $p\to q$, $m_{pp}$ is always non-decreasing with respect to $p$; whereas $S_{pp}$ is non-decreasing for $p<0$, non-increasing for $p>0$, with a maximum value at $p=0$. In all cases, both the mean and entropy are invariant with respect to the order $p,q$ if and only if all the non-zero elements $x_i$ are equal.

This property explains the monotonicity of the curves in Figure 1 and Figure 3.

4.7. Range of $m_{pq}(X)$

From Property 4.6, $m_{pq}(X)$ is non-decreasing with respect to $p,q$. From Table 1, we know that $m_{p,\infty }(X)=\max (X)$ and $m_{p,-\infty }(X)=\min (X)$. Therefore, $\min (X)\le m_{pq}(X)\le \max (X)$, which is an intuitive range for a mean. This is also true in the degenerate case $p\to q$, i.e.,

$$\min (X)\le m_{pp}(X)\le \max (X)$$

(42)

where, as usual, equality holds when all the non-zero elements $x_i$ are equal.

4.8. Additivity of the Joint Multiset Entropy

For the two probability distribution multisets $X=\{x_i:i=1,2,\dots ,K_X\}$ and $Y=\{y_i:i=1,2,\dots ,K_Y\}$, we define the joint multiset $X\ast Y:=\left\{x_i{y}_j:\begin{array}{c}i=1,2,\dots ,K_X\\ j=1,2,\dots ,K_Y\end{array}\right\}$. Therefore, we have:

$$\begin{array}{l}S(X\ast Y)=S(X)+S(Y)\\ m(X\ast Y)=m(X)m(Y)\end{array}$$

(43)

The proof is straightforward by using the fact that ${\Vert X\ast Y\Vert }_p={\Vert X\Vert }_p{\Vert Y\Vert }_p$ along with (7)–(10), Property 4.8 is true for both $p\ne q$ and for the degenerate case $p\to q$.

4.9. Sub-Additivity of the Effective Cardinality Subject to the Multiset Additive Union Operation

Let $\uplus$ denote a multiset additive union operation [15] (page 50), e.g., $\{2,2\}\uplus \{1,1,2\}=\{1,1,2,2,2\}$. Let $X=\{x_i:i=1,2,\dots .K_X\}$ and $Y=\{y_j:j=1,2,\dots .K_Y\}$ be two multisets of non-negative real numbers. Moreover, let $\xi$ and $\eta$ be two positive real scaling factors of the elements of $X$ and $Y$, respectively. Then:

$$k_{pq}(\{\xi x_i\}\uplus \{\eta y_j\})\le k_{pq}(X)+k_{pq}(Y)$$

(44)

where the equality holds under the following condition for the value $\xi /\eta$:

$$\frac{{\xi }_0}{{\eta }_0}=\{\begin{array}{cc}{\left(\frac{{\displaystyle {\sum }_j{y_j}^q}/{\displaystyle {\sum }_j{y_j}^p}}{{\displaystyle {\sum }_i{x_i}^q}/{\displaystyle {\sum }_i{x_i}^p}}\right)}^{\frac{1}{q-p}}& p\ne q\\ \frac{{\Vert Y\Vert }_q}{{\Vert X\Vert }_q}{\left(\frac{k_{qq}(X)}{k_{qq}(Y)}\right)}^{1/q}& p\to q\end{array}$$

(45)

This property is a generalization of the effective alphabet size of two disjoint alphabets mixture as discussed in [16] (Problem 2.10). To see this, set $p=q=1$ and note that ${\Vert X\Vert }_1={\Vert Y\Vert }_1=1$ for a complete probability distribution. Accordingly, $k_{pq}(X)$ represents the effective alphabet size of $X$ corresponding to the entropy $S_{pq}(X)$. The proof is in Appendix D.

4.10. Effective Rank of a Matrix

Let $X$ be the multiset of the singular values of a matrix $M$. Then, $k_{0q}(X)=\text{rank}(M)$. Accordingly, for general $p,q$, $k_{pq}(X)$ can be viewed as a biased effective rank of $M$ corresponding to the order $p,q$. From Property 4.5, we have $1\le k_{pq}(X)\le \text{rank}(M)$. The minimum value occurs when $M$has exactly one non-zero singular value (the rank of $M$ is 1). The maximum value, $\text{rank}(M)$, is reached for any $pq>0$ when all the non-zero singular values are equal. The effective rank can be helpful to determine, in a well-defined $p,q$ sense, how to view an ill-conditioned matrix, which is a full-rank from a mathematical perspective, but is effectively behaving as if possessing a lower rank. Such ill-conditioned matrices often arise in problems of oversampling or determining the degrees of freedom, where the singular values, ordered in non-increasing order by definition, exhibit some sort of “knee cut-off”, similar to$\left\{x_i\right\}$in Figure 2. A biased effective rank can help to compare matrices when the knee cut-off is not sharp, thus giving more weight to the small or large singular values according to the order $p,q$ in a consistent manner for different matrices. An example thereof is the evaluation of the degrees of freedom of some applications such as multi-antenna systems [3,4,5,6], optical imaging systems [20], or in general any case of similar limitation to space-bandwidth product [21].

4.11. Geometrical Interpretation of $S_{pq}$ and $m_{pq}$ on Log Scale, with Thermodynamics Analogy

For the multiset $X=\left\{x_i\right\}$, after taking the logarithm of both sides of (5), we obtain a simple relation between ${\Vert X\Vert }_q$, $S_{pq}$ and $m_{pq}$ as in (11):

$$\begin{array}{l}\log {\Vert X\Vert }_q^q=\log {\Vert M\Vert }_q^q=S_{pq}+q\log m_{pq}\\ \log {\Vert X\Vert }_p^p=\log {\Vert M\Vert }_p^p=S_{pq}+p\log m_{pq}\end{array}$$

(46)

Accordingly, a secant cutting the function $f(q)=\log {\Vert X\Vert }_q^q$ at $q=p_0$ and $q=q_0$ will have a slope and intercept of $\log m_{p_0{q}_0}$ and $S_{p_0{q}_0}$, respectively, as shown in Figure 4. Based on the discussion of Section 1, (46) readily yields the following analogous expressions for a thermodynamics system described by the Boltzmann factors $\left\{e^{-{\mathsf{{\rm E}}}_i}\right\}$:

$$\begin{array}{l}\log Z_{\beta 1}(X)=S_{\beta 1,\beta 2}-{\beta }_1{E}_0\\ \log Z_{\beta 2}(X)=S_{\beta 1,\beta 2}-{\beta }_2{E}_0\end{array}$$

(47)

In the limiting case, when $p\to q,{\beta }_1\to {\beta }_2$, the secant in Figure 4 becomes a tangent and we get the Gibbs–Shannon entropy at the inverse temperature $\beta$. Consequently, (47) can be re-written after introducing the Boltzmann constant $K_B$ and using the absolute temperature $T=1/K_B\beta$ as:

$$\begin{array}{l}-K_{B}T\log Z_{\beta }:=A_H\\ =E_0-T{S}_{\beta }\end{array}$$

(48)

where $A_H$ is the Helmholtz free energy of the system.

Figure 4. A secant of $\log {\Vert X\Vert }_q^q$ versus $q$.

Figure 4. A secant of $\log {\Vert X\Vert }_q^q$ versus $q$.

5. Discussion and Conclusions

A two-parameter family of cardinalities, entropies and means has been derived for multisets of non-negative elements. Rather than starting from thermodynamic or information theoretic considerations to derive entropies and means, see e.g., [9,17,18,19], we here defined the generalized entropies and means through simple abstract axioms. There are other families of entropies in the literature (e.g., [23]), which are a generalization of Shannon entropy. The generalized entropy in this manuscript is shown to preserve the additivity (Property 4.8), which is not the case with the generalized entropies based on Tsallis non-additive entropy as in [23].

Our first two axiomatizations treat the generalized entropies and means separately. It revealed that the generalized entropies are exactly those entropies that are functions of only the ratio of the multisets’ $l^p$ and $l^q$ norms. It also revealed that the generalized means are exactly those means that are functions only of the ratio of the multisets’ p^th and q^th moments, $\left|\right|X|{|}_p^p$ and $\left|\right|X|{|}_q^q$. Subsequently, our unifying axiomatization characterized the generalized entropies and means together. This showed that if two multisets have exactly the same $l^p$ and $l^q$ norms, then they share the same generalized entropy and mean.

We presented several key features of the new families of generalized entropies and means, for example, that the family of generalized entropies contains and generalizes the Rényi family of entropies, of which the Shannon entropy is a special case, thus including some of the desiderata for entropies [22]. We also showed the monotonicity with respect to $p,q$, extreme values, symmetry with respect to $p,q$, and additivity preservation. The effective cardinality $k_{pq}$ measures the distribution uniformity of the multiset elements in the sense of the p- and q-norm equivalence to a reference flat multiset. From an information theory perspective, $S_{pq}$ and $k_{pq}$ represent a two-parameter entropy of order $p,q$ and its corresponding effective alphabet size, respectively, when a probability distribution is constructed after proper normalization of the multiset elements. Furthermore, we recall that knowing the ${\ell }^p$ and ${\ell }^q$ norms of a multiset is to know the multiset’s p^th and q^th moments. Our findings here therefore imply that knowledge of a multiset’s p^th and q^th moments is exactly enough information to deduce the multiset’s (p,q) entropy and (p,q) mean. Further, knowledge of sufficiently many moments of a multiset can be sufficient to reconstruct the multiset. Conversely, it should be interesting to examine how many (p,q)-entropies and/or (p,q)-means are required to completely determine the multiset.

Regarding the thermodynamic interpretation, we noticed that to require that the ${\ell }^p$ and ${\ell }^q$ norms of multisets coincide is mathematically equivalent to requiring that the two partition functions of two thermodynamic systems coincide at two temperatures. This in turn is equivalent to requiring that the Helmholtz free energy of the two thermodynamic systems coincide at two temperatures. The Helmholtz free energy represents the maximum mechanical work that can be extracted from a thermodynamic system under certain idealized circumstances. This suggests that there perhaps exists a thermodynamic interpretation of the generalized entropies and means in terms of the extractability of mechanical work. In this case, the fact that the generalized entropies and means depend on two rather than one temperature could be related to the fact that the maximum efficiency of a heat engine, obtained in Carnot cycles, is a function of two temperatures. We did show that in the limiting case, when the two temperatures become the same, one recovers the usual Boltzmann factor weighted mean energy as well as the usual Shannon/von Neumann entropy.