Attention to the Variation of Probabilistic Events: Information Processing with Message Importance Measure

Abstract

Different probabilities of events attract different attention in many scenarios such as anomaly detection and security systems. To characterize the events’ importance from a probabilistic perspective, the message importance measure (MIM) is proposed as a kind of semantics analysis tool. Similar to Shannon entropy, the MIM has its special function in information representation, in which the parameter of MIM plays a vital role. Actually, the parameter dominates the properties of MIM, based on which the MIM has three work regions where this measure can be used flexibly for different goals. When the parameter is positive but not large enough, the MIM not only provides a new viewpoint for information processing but also has some similarities with Shannon entropy in the information compression and transmission. In this regard, this paper first constructs a system model with message importance measure and proposes the message importance loss to enrich the information processing strategies. Moreover, the message importance loss capacity is proposed to measure the information importance harvest in a transmission. Furthermore, the message importance distortion function is discussed to give an upper bound of information compression based on the MIM. Additionally, the bitrate transmission constrained by the message importance loss is investigated to broaden the scope for Shannon information theory.

1. Introduction

In recent years, massive data has attracted much attention in various realistic scenarios. Actually, there exist many challenges for data processing such as distributed data acquisition, huge-scale data storage and transmission, as well as correlation or causality representation [1,2,3,4,5]. Facing these obstacles, it is a promising way to make good use of information theory and statistics to deal with mass information. For example, a method based on Max Entropy in Metric Space (MEMS) is utilized for local features extraction and mechanical system analysis [6]; as an information measure different from Shannon entropy, Voronoi entropy is discussed to characterize the random 2D patterns [7]; Category theory, which can characterize the Kolmogorov–Sinai and Shannon entropy as the unique functors, is used in autonomous and networked dynamical systems [8].

To some degree, probabilistic events attract different interests according to their probability. For example, considering that small probability events hidden in massive data contain more semantic importance [9,10,11,12,13], people usually pay more attention to the rare events (rather than the common events) and design the corresponding strategies of their information representation and processing in many applications including outliers detection in the Internet of Things (IoT), smart cities and autonomous driving [14,15,16,17,18,19,20,21,22]. Therefore, the probabilistic events processing has special values in the information technology based on semantics analysis of message importance.

In order to characterize the importance of probabilistic events, a new information measure named MIM is presented to generalize Shannon information theory [23,24,25]. Here, we shall investigate the information processing including compression (or storage) and transmission based on MIM to bring some new viewpoints in the information theory. Now, we first give a short review on MIM.

1.1. Review of Message Importance Measure

Essentially, the message importance measure (MIM) is proposed to focus on the probabilistic events importance [23]. In particular, the core idea of this information measure is that the weights of importance are allocated to different events according to the corresponding events’ probability. In this regard, as an information measure, MIM may provide an applicable criterion to characterize the message importance from the viewpoint of inherent property of events without the human subjective factors. For convenience of calculation, an exponential expression of MIM is defined as follows.

Definition 1.

For a discrete distribution $P\left(X\right)$ = $\{p\left(x_1\right)$, $p\left(x_2\right)$, …, $p\left(x_n\right)\}$, the exponential expression of message importance measure (MIM) is given by

$$\begin{array}{cc}\hfill L(\varpi ,X)& =\sum _{x_i}p\left(x_i\right)e^{\varpi \{1-p\left(x_i\right)\}},\hfill \end{array}$$

(1)

where the adjustable parameter ϖ is nonnegative and $p\left(x_i\right)e^{\varpi \{1-p\left(x_i\right)\}}$ is viewed as the self-scoring value of event i to measure its message importance.

Actually, from the perspective of generalized Fadeev’s postulates, the MIM is viewed as a rational information measure similar to Shannon entropy and Renyi entropy which are respectively defined by

$$\begin{array}{cc}\hfill & H\left(X\right)=-\sum _{x_i}p\left(x_i\right)logp\left(x_i\right),\hfill \end{array}$$

(2a)

$$\begin{array}{cc}\hfill & H_{\alpha }\left(X\right)=\frac{1}{1-\alpha }log\sum _{x_i}{\left\{p\left(x_i\right)\right\}}^{\alpha },(0<\alpha <\infty ,\alpha \ne 1),\hfill \end{array}$$

(2b)

where the condition of variable X is the same as that described in Definition 1. In particular, a postulate for the MIM weaker than that for Shannon entropy and Renyi entropy is given by

$$F\left(PQ\right)\le F\left(P\right)+F\left(Q\right),$$

(3)

while $F\left(PQ\right)=F\left(P\right)+F\left(Q\right)$ is satisfied for Shannon entropy and Renyi entropy [26], where P and Q are two independent random distributions and $F(·)$ denotes a kind of information measure.

Moreover, the crucial operator of MIM to handle probability elements is exponential function while the corresponding operators of Shannon and Renyi entropy are logarithmic function and polynomial function respectively. In this case, MIM can be viewed as a map for the assignments of events’ importance weights or the achievement for the self-scoring values of events different from conventional information measures.

As far as the application of MIM is concerned, it may be a better method by using this information measure to detect unbalanced events in signal processing. Ref. [27] has investigated the minor probability event detection by combining MIM and Bayes detection. Moreover, it is worth noting that the physical meaning of the components of MIM corresponds to the normalized optimal data recommendation distribution, which makes a trade-off between the users’ preference and system revenue [28]. In this respect, MIM plays a fundamental role in the recommendation system (a popular applications of big data) from the theoretic viewpoint. Therefore, MIM does not come from the imagination directly, whereas it is a meaningful information measure originated from the practical scenario.

1.2. The Importance Coefficient $\varpi$ in MIM

In general, the parameter $\varpi$ viewed as the importance coefficient has a great impact on the MIM. Actually, different parameter $\varpi$ can lead to different properties and performances for this information measure. In particular, to measure a distribution $P\left(X\right)=\{p\left(x_1\right),p\left(x_2\right),\dots ,p\left(x_n\right)\}$, there are three kinds of work regions of MIM which can be classified by the parameters, whose details are discussed as follows.

(i)

If the parameter satisfies $0\le \varpi \le 2/max\left\{p\left(x_i\right)\right\}$, the convexity of MIM is similar to Shannon entropy and Renyi entropy. Actually, these three information measures all have maximum value properties and allocate weights for probability elements of the distribution $P\left(X\right)$. It is notable that the MIM in this work region focuses on the typical sets rather than atypical sets, which implies that the uniform distribution reaches the maximum value. In brief, the MIM in this work region can be regarded as the same class of message measure as Shannon entropy and Renyi entropy to deal with the problems of information theory.

(ii)

If we have $\varpi >2/max\left\{p\left(x_i\right)\right\}$, the small probability elements will be the dominant factor for MIM to measure a distribution. That is, the small probability events can be highlighted more in this work region of MIM than those in the first one. Moreover, in this work region, MIM can pay more attention to atypical sets, which can be viewed as a magnifier for rare events. In fact, this property corresponds to some common scenarios where anomalies catch more eyes such as anomalous detection and alarm. In this case, some problems (including communication and probabilistic events processing) can be rehandled from the perspective of rare events importance. Particularly, the compression encoding and maximum entropy rate transmission are proposed based on the non-parametric MIM (namely NMIM) [24]; in addition, the distribution goodness-of-fit approach is also presented by use of the differential MIM (namely DMIM) [29].

(iii)

If the MIM has the parameter $\varpi <0$, the large probability elements will be the main part contributing to the value of this information measure. In other words, the normal events attract more attention in this work region of MIM than rare events. In practice, this can be used in many applications where regular events are popular such as filter systems and data cleaning.

As a matter of fact, by selecting the parameter $\varpi$ properly, we can exploit the MIM to solve several problems in different scenarios. The importance coefficient facilitates more flexibility of MIM in applications beyond Shannon entropy and Renyi entropy.

To focus on a concrete object, in this paper, we mainly investigate the first work region of MIM (namely $0\le \varpi \le 2/max\left\{p\left(x_i\right)\right\}$) and intend to dig out some novelties related to this metric for information processing.

1.3. Similarities and Differences between Shannon Entropy and MIM

In fact, when the parameter $\varpi$ satisfies $0\le \varpi \le 2/max\left\{p\left(x_i\right)\right\}$, MIM is similar to Shannon entropy in regard to the expression and properties. The exponential operator of MIM is a substitute for the logarithm operator of Shannon entropy. As a kind of tool based on probability distributions, the MIM with parameter $0\le \varpi \le 2/max\left\{p\left(x_i\right)\right\}$ has the same concavity and monotonicity as Shannon entropy, which can characterize the information otherness for different variables.

By resorting to the exponential operator of MIM, the weights for small probability elements are amplified more in some degree than those for large probability ones, which is considered as message importance allocation based on the self-scoring values. In this regard, the MIM may add fresh factors to the information processing, which takes into account the effects of probabilistic events’ importance from an objective viewpoint.

In the conventional Shannon information theory, data transmission and compression both can be viewed as the information transfer process from the variable X to Y. The capacity of information transmission is achieved by maximizing the mutual information between the X and Y. Actually, there exists distortion for probabilistic events during an information transfer process, which denotes the difference between the source and its corresponding reconstruction. Due to this fact, it is possible to compress data based on the allowable information loss in a certain extent [30,31,32]. In Shannon information theory, rate-distortion theory is investigated for lossy data compression, whose essence is mutual information minimization under the constraint of a certain distortion. However, in some cases involved with distortion, small probability events containing more message importance require higher reliability than those with large probability. In this sense, another aspect of information distortion may be essential, in which message importance is considered as a reasonable metric. Particularly, information transfer process is characterized by the MIM (rather than the entropy) with controlling the distortion, which can be viewed as a new kind of information compression, compared to the conventional scheme compressing redundancy to save resources. In fact, some information measures with respect to message importance have been investigated to extend the range of Shannon information theory [33,34,35,36,37]. In this regard, it is worthwhile exploring the information processing in the sense of MIM. Furthermore, it is also promising to investigate the Shannon mutual information constrained by the MIM in an information transfer process which may become a novel system invariant.

In addition, similar to Shannon conditional entropy, a conditional message importance measure for two distributions is proposed to process conditional probability.

Definition 2.

For the two discrete probability $P\left(X\right)$ = $\{p\left(x_1\right)$, $p\left(x_2\right)$, …, $p\left(x_n\right)\}$ and $P\left(Y\right)$ = $\{p\left(y_1\right)$, $p\left(y_2\right)$, …, $p\left(y_n\right)\}$, the conditional message importance measure (CMIM) is given by

$$\begin{array}{cc}\hfill L(\varpi ,X|Y)& =\sum _{y_j}p\left(y_j\right)\sum _{x_i}p\left(x_i\right|y_j)e^{\varpi \{1-p\left(x_i\right|y_j)\}},\hfill \end{array}$$

(4)

where $p\left(x_i\right|y_j)$ denotes the conditional probability between $y_j$ and $x_i$. The component $p\left(x_i\right|y_j)e^{\varpi \{1-p\left(x_i\right|y_j)\}}$ is similar to self-scoring value. Therefore, the CMIM can be considered as a system invariant which indicates the average total self-scoring value for an information transfer process.

Actually, the MIM is a metric with different mathematical and physical meaning from Shannon entropy and Renyi entropy, which provides its own perspective to process probabilistic events. However, due to the similarity between the MIM and Shannon entropy, they may have analogous performance in some aspects. To this end, the information processing based on the MIM is discussed in this paper.

1.4. Motivation and Contributions

The purpose of this paper is to characterize the probabilistic events processing including compression and transmission by means of MIM. Particularly, in terms of the information processing system model shown in Figure 1, the message source $\phi$ (regarded as a random variable whose support set corresponds to the set of events’ types) can be measured by the amount of information $H(·)$ and the message importance $L(·)$ according to the probability distribution. Then, the information transfer process whose details are presented in Section 2 can be characterized based on these two metrics. Different from the mathematically probabilistic characterization of traditional telecommunication system, this paper mainly discusses the information processing from the perspectives of message importance. In this regard, the information importance harvest in a transmission is characterized by the proposed message importance loss capacity. Moreover, the upper bound of information compression based on the MIM is described by the message importance distortion function. In addition, we also investigate the trade-off between bitrate transmission and message importance loss to bring some inspiration to the conventional information theory.

1.5. Organization

The rest of this paper is discussed as follows. In Section 2, a system model involved with message importance is constructed to help analyze the data compression and transmission in big data. In Section 3, we propose a kind of message transfer capacity to investigate the message importance loss in the transmission. In Section 4, message importance distortion function is introduced and its properties are also presented to give some details. In Section 5, we discuss the bitrate transmission constrained by message importance to widen the horizon for the Shannon theory. In Section 6, some numerical results are presented to validate propositions and the analysis in theory. Finally, we conclude this paper in Section 7. Additionally, the fundamental notations in this paper are summarized in

Table 1. Notations.

Notation	Description
$P\left(X\right)$ = $\{p\left(x_1\right)$, $p\left(x_2\right)$, …, $p\left(x_n\right)\}$	The discrete probability distribution with respect to the variable X
$\phi$	The message source in the information processing system model
$\tilde{\phi }$	The mapped or compressed message with respect to the $\phi$
$\tilde{\Omega }$	The received message transferred from the $\tilde{\phi }$
$\Omega$	The recovered message with respect to the $\phi$ by the decoding process
$\varpi$	The importance coefficient
$L(·)$	The message importance measure (MIM) described as Definition 1
$H(·)$	The Shannon entropy, $H\left(X\right)=-{\sum }_{x_i}p\left(x_i\right)logp\left(x_i\right)$
	or $H\left(p\right)=-plogp-(1-p)log(1-p),(0\le p\le 1)$
$H_{\alpha }(·)$	The Renyi entropy with the parameter $\alpha$ $H_{\alpha }\left(X\right)=\frac{1}{1-\alpha }log{\sum }_{x_i}{\left\{p\left(x_i\right)\right\}}^{\alpha }$
$L(·|·)$	The CMIM described as Definition 2
$H(·|·)$	The conditional Shannon entropy, $H\left(X\right|Y)={\sum }_{x_i}{\sum }_{y_j}p(x_i,y_j)log\frac{1}{p\left(x_i\right|y_j)}$
${\Phi }_{\varpi }(·||·)$	The message importance loss described as Definition 3
C	the message importance loss capacity (MILC) described as Definition 4
$p\left(y\right|x)$	An information transfer matrix from the variable X to Y
$\{X,p(y\left|x\right),Y\}$	An information transfer process from the variable X to Y
${\beta }_s$,${\beta }_e$,	The parameters in the binary symmetric matrix, binary eraser matrix and
${\beta }_k$	k-ary symmetric matrix respectively
$d(x,y)$	The distortion function, $d(x,y)\ge 0$
D	The allowable distortion ($D_{min}\le D\le D_{max}$)
$\overline{D}$	The average distortion, $\overline{D}={\sum }_{x_i}{\sum }_{y_j}p\left(x_i\right)p\left(y_j\right|x_i)d(x_i,y_j)$
$B_D$	The the allowable information transfer matrix set $B_D=\{q\left(y\right|x):\overline{D}\le D\}$
$R_{\varpi }\left(D\right)$	The message importance distortion function described as Definition 5
$I\left(X\right|\left|Y\right)$	Mutual information, $I\left(X\right|\left|Y\right)={\sum }_{x_i}{\sum }_{y_j}p(x_i,y_j)log\frac{p(x_i,y_j)}{p\left(x_i\right)p\left(y_j\right)}$

.

2. System Model with Message Importance

Considering an information processing system model shown in Figure 1, the information transfer process is discussed as follows. At first, a message source $\phi$ follows a distribution $P_{\phi }=\{p\left({\phi }_1\right),p\left({\phi }_2\right),...,p\left({\phi }_n\right)\}$ whose support set is $\{{\phi }_1,{\phi }_2,...,{\phi }_n\}$ corresponding to the events types. Then, the message $\phi$ is encoded or compressed into the variable $\tilde{\phi }$ following the distribution $P_{\tilde{\phi }}=\{p\left({\tilde{\phi }}_1\right),p\left({\tilde{\phi }}_2\right),...,p\left({\tilde{\phi }}_n\right)\}$ whose alphabet is $\{{\phi }_1,{\phi }_2,...,{\phi }_n\}$. After the information transfer process denoted by matrix $p\left({\tilde{\Omega }}_j\right|{\tilde{\phi }}_i)$, the received message $\tilde{\Omega }$ originating from $\tilde{\phi }$ is observed as a random variable, where the distribution of $\tilde{\Omega }$ is $P_{\tilde{\Omega }}=\{p\left({\tilde{\Omega }}_1\right),p\left({\tilde{\Omega }}_2\right),...,p\left({\tilde{\Omega }}_n\right)\}$ whose alphabet is $\{{\tilde{\Omega }}_1,{\tilde{\Omega }}_2,...,{\tilde{\Omega }}_n\}$. Finally, the receiver recovers the original message $\phi$ by decoding $\Omega =g\left(\tilde{\Omega }\right)$ where $g(·)$ denotes the decoding function and $\Omega$ is the recovered message with the alphabet $\{{\Omega }_1,{\Omega }_2,...,{\Omega }_n\}$.

From the viewpoint of generalized information theory, a two-layer framework is considered to understand this model, where the first layer is based on the amount of information characterized by Shannon entropy denoted by $H(·)$, while the second layer reposes on message importance measure of events denoted by $L(·)$. Due to the fact that the former is discussed pretty entirely, we mainly investigate the latter in the paper.

Considering the source-channel separation theorem [38], the above information processing model consists of two problems, namely data compression and data transmission. On one hand, the data compression of the system can be achieved by using classical source coding strategies to reduce more redundancy, in which the information loss is described by $H\left(\phi \right)-H\left(\phi \right|\tilde{\phi })$ under the information transfer matrix $p\left(\tilde{\phi }\right|\phi )$. Similarly, from the perspective of message importance, the data can be further compressed by discarding worthless messages, where the message importance loss can be characterized by $L\left(\phi \right)-L\left(\phi \right|\tilde{\phi })$. On the other hand, the data transmission is discussed to obtain the upper bound of the mutual information $H\left(\tilde{\phi }\right)-H\left(\tilde{\phi }\right|\tilde{\Omega })$, namely the information capacity. In a similar way, $L\left(\tilde{\phi }\right)-L\left(\tilde{\phi }\right|\tilde{\Omega })$ means the income of message importance in the transmission.

In essence, it is apparent that the data compression and transmission are both considered as an information transfer processes $\{X,p(y\left|x\right),Y\}$, and they can be characterized by the difference between $\left\{X\right\}$ and $\left\{X\right|Y\}$. In order to facilitate the analysis of the above model, the message importance loss is introduced as follows.

Definition 3.

For two discrete probability $P\left(X\right)$ = $\{p\left(x_1\right)$, $p\left(x_2\right)$, …, $p\left(x_n\right)\}$ and $P\left(Y\right)$ = $\{p\left(y_1\right)$, $p\left(y_2\right)$, …, $p\left(y_n\right)\}$, the message importance loss based on MIM and CMIM is given by

$$\begin{array}{cc}\hfill {\Phi }_{\varpi }\left(X\right|\left|Y\right)& =L(\varpi ,X)-L(\varpi ,X|Y),\hfill \end{array}$$

(5)

where $L(\varpi ,X)$ and $L(\varpi ,X|Y)$ are given by the Definitions 1 and 2.

In fact, according to the intrinsic relationship between $L(\varpi ,X)$ and $L(\varpi ,X|Y)$, it is readily seen that

$$\begin{array}{c}\hfill {\Phi }_{\varpi }\left(X\right|\left|Y\right)\ge 0,\end{array}$$

(6)

where $0<\varpi \le 2\le 2/max\left\{p\left(x_i\right|y_j)\right\}$.

Proof. 

Considering a function $f\left(x\right)=x{e}^{\varpi (1-x)}$ ($0\le x\le 1$ and $0<\varpi$), it is easy to have $\frac{{\partial }^{2}f\left(x\right)}{\partial x}=-\varpi e^{\varpi (1-x)}(2-\varpi x)$, which implies if $\varpi \le 2\le 2/x$, the function $f\left(x\right)$ is concave.

In the light of Jensen’s inequality, if $0<\varpi \le 2\le 2/max\left\{p\left(x_i\right|y_j)\right\}$ is satisfied, it is not difficult to see

$$\begin{array}{cc}\hfill L(\varpi ,X)& =\sum _{x_i}p\left(x_i\right)e^{\varpi (1-p\left(x_i\right))}\hfill \\ & =\sum _{x_i}\left\{\sum _{y_j}p\left(y_j\right)p\left(x_i\right|y_j)\right\}{e}^{\varpi (1-\left\{{\sum }_{y_j}p\left(y_j\right)p\left(x_i\right|y_j)\right\})}\hfill \\ & \ge \sum _{y_j}p\left(y_j\right)\sum _{x_i}\left\{p\left(x_i\right|y_j)e^{\varpi (1-p\left(x_i\right|y_j))}\right\}=L(\varpi ,X|Y).\hfill \end{array}$$

(7)

□

3. Message Importance Loss in Transmission

In this section, we will introduce the CMIM to characterize the information transfer processing. To do so, we define a kind of message transfer capacity measured by the CMIM as follows.

Definition 4.

Assume that there exists an information transfer process as

$$\begin{array}{c}\hfill \{X,p(y\left|x\right),Y\},\end{array}$$

(8)

where the $p\left(y\right|x)$ denotes a probability distribution matrix describing the information transfer from the variable X to Y. We define the message importance loss capacity (MILC) as

$$\begin{array}{cc}\hfill C& =\underset{p\left(x\right)}{max}\{{\Phi }_{\varpi }\left(X\right|\left|Y\right)\}\hfill \\ & =\underset{p\left(x\right)}{max}\{L(\varpi ,X)-L(\varpi ,X|Y)\},\hfill \end{array}$$

(9)

where $L(\varpi ,X)={\sum }_{x_i}p\left(x_i\right)e^{\varpi \{1-p\left(x_i\right)\}}$, $p\left(y_j\right)={\sum }_{x_i}p\left(x_i\right)p\left(y_j\right|x_i)$, $p\left(x_i\right|y_j)=\frac{p\left(x_i\right)p\left(y_j\right|x_i)}{p\left(y_j\right)}$, $L(\varpi ,X|Y)$ is defined by Equation (4), and $\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$.

In order to have an insight into the applications of MILC, some specific information transfer scenarios are discussed as follows.

3.1. Binary Symmetric Matrix

Consider the binary symmetric information transfer matrix, where the original variables are complemented with the transfer probability which can be seen in the following proposition.

Proposition 1.

Assume that there exists an information transfer process $\{X,p(y\left|x\right),Y\}$, where the information transfer matrix is

$$\begin{array}{c}\hfill p\left(y\right|x)=\left[\begin{array}{cc}1-{\beta }_s& {\beta }_s\\ {\beta }_s& 1-{\beta }_s\end{array}\right],\end{array}$$

(10)

which indicates that X and Y both follow binary distributions. In that case, we have

$$\begin{array}{c}\hfill C(\varpi ,{\beta }_s)=e^{\frac{\varpi }{2}}-L(\varpi ,{\beta }_s),\end{array}$$

(11)

where $L(\varpi ,{\beta }_s)={\beta }_s{e}^{\varpi (1-{\beta }_s)}+(1-{\beta }_s)e^{\varpi {\beta }_s}$ ($0\le {\beta }_s\le 1$) and $\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$.

Proof of Proposition 1.

Assume that the distribution of variable X is a binary distribution $(p,1-p)$. According to Equation (10) and Bayes’ theorem (namely, $p\left(x\right|y)=\frac{p\left(x\right)p\left(y\right|x)}{p\left(y\right)}$), it is not difficult to see that

$$\begin{array}{c}\hfill p\left(x\right|y)=\left[\begin{array}{cc}\frac{p(1-{\beta }_s)}{p(1-{\beta }_s)+(1-p){\beta }_s}& \frac{(1-p){\beta }_s}{p(1-{\beta }_s)+(1-p){\beta }_s}\\ \frac{p{\beta }_s}{p{\beta }_s+(1-p)(1-{\beta }_s)}& \frac{(1-p)(1-{\beta }_s)}{p{\beta }_s+(1-p)(1-{\beta }_s)}\end{array}\right].\end{array}$$

(12)

Furthermore, in accordance with Equations (4) and (9), we have

$$\begin{array}{c}C(\varpi ,{\beta }_s)\hfill \\ =\underset{p}{max}\left\{C(p,\varpi ,{\beta }_s)\right\}\hfill \\ =\underset{p}{max}\{L(\varpi ,p)-\{p(1-{\beta }_s)e^{\frac{\varpi (1-p){\beta }_s}{p(1-{\beta }_s)+(1-p){\beta }_s}}+(1-p){\beta }_s{e}^{\frac{\varpi p(1-{\beta }_s)}{p(1-{\beta }_s)+(1-p){\beta }_s}}+p{\beta }_s{e}^{\frac{\varpi (1-p)(1-{\beta }_s)}{p{\beta }_s+(1-p)(1-{\beta }_s)}}\hfill \\ +(1-p)(1-{\beta }_s)e^{\frac{\varpi p{\beta }_s}{p{\beta }_s+(1-p)(1-{\beta }_s)}}\left\}\right\},\hfill \end{array}$$

(13)

where $L(\varpi ,p)=p{e}^{\varpi (1-p)}+(1-p)e^{\varpi p}$ ($0<p<1$). Then, it is readily seen that

$$\begin{array}{cc}\hfill \frac{\partial C(p,\varpi ,{\beta }_s)}{\partial p}=& (1-\varpi p)e^{\varpi (1-p)}+[(1-p)\varpi -1]e^{\varpi p}\hfill \\ & -\{(1-{\beta }_s)\left\{1-\frac{\varpi p(1-{\beta }_s){\beta }_s}{{[p(1-{\beta }_s)+(1-p){\beta }_s]}^2}\right\}{e}^{\frac{\varpi (1-p)\beta }{p(1-\beta )+(1-p)\beta }}\hfill \\ & +(1-{\beta }_s)\left\{\frac{\varpi (1-p){\beta }_s(1-{\beta }_s)}{{[p{\beta }_s+(1-p)(1-{\beta }_s)]}^2}-1\right\}{e}^{\frac{\varpi p{\beta }_s}{p{\beta }_s+(1-p)(1-{\beta }_s)}}\hfill \\ & +{\beta }_s\left\{\frac{\varpi (1-p){\beta }_s(1-{\beta }_s)}{{[p(1-{\beta }_s)+(1-p){\beta }_s]}^2}-1\right\}{e}^{\frac{\varpi p(1-{\beta }_s)}{p(1-{\beta }_s)+(1-p){\beta }_s}}\hfill \\ & +{\beta }_s\left\{1-\frac{\varpi p(1-{\beta }_s){\beta }_s}{{[p{\beta }_s+(1-p)(1-{\beta }_s)]}^2}\right\}{e}^{\frac{\varpi (1-p)(1-{\beta }_s)}{p{\beta }_s+(1-p)(1-{\beta }_s)}}\}.\hfill \end{array}$$

(14)

In the light of the positivity for $\frac{\partial C(p,{\beta }_s)}{\partial p}$ in $\left\{p\right|p\in (0,1/2)\}$ and the negativity in $\left\{p\right|p\in (1/2,1)\}$ (if ${\beta }_s\ne 1/2$), it is apparent that $p=1/2$ is the only solution for $\frac{\partial C(p,{\beta }_s)}{\partial p}=0$. That is, if ${\beta }_s\ne 1/2$, the extreme value is indeed the maximum value of $C(p,\varpi ,{\beta }_s)$ when $p=1/2$. Similarly, if ${\beta }_s=1/2$, the solution $p=1/2$ also results in the same conclusion. □

Remark 1.

According to Proposition 1, on one hand, when ${\beta }_s=1/2$, that is, the information transfer process is just random, we will gain the lower bound of the MILC namely $C\left({\beta }_s\right)=0$. On the other hand, when ${\beta }_s=0$, namely there is a certain information transfer process, we will have the maximum MILC. As for the distribution selection for the variable X, the uniform distribution is preferred to gain the capacity.

3.2. Binary Erasure Matrix

The binary erasure information transfer matrix is similar to the binary symmetric one; however, in the former, a part of information is lost rather than corrupted. The MILC of this kind of information transfer matrix is discussed as follows.

Proposition 2.

Consider an information transfer process $\{X,p(y\left|x\right),Y\}$, in which the information transfer matrix is described as

$$\begin{array}{c}\hfill p\left(y\right|x)=\left[\begin{array}{ccc}1-{\beta }_e& 0& {\beta }_e\\ 0& 1-{\beta }_e& {\beta }_e\end{array}\right],\end{array}$$

(15)

which indicates that X follows the binary distribution and Y follows the 3-ary distribution. Then, we have

$$\begin{array}{cc}\hfill C(\varpi ,{\beta }_e)& =(1-{\beta }_e)\{e^{\frac{\varpi }{2}}-1\},\hfill \end{array}$$

(16)

where $0\le {\beta }_e\le 1$ and $0<\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$.

Proof of Proposition 2.

Assume the distribution of variable X is $(p,1-p)$. Furthermore, according to the binary erasure matrix and Bayes theorem, we have that the transmission matrix conditioned by the variable Y as follows:

$$\begin{array}{c}\hfill p\left(x\right|y)=\left[\begin{array}{cc}1& 0\\ 0& 1\\ p& 1-p\end{array}\right].\end{array}$$

(17)

Then, it is not difficult to have

$$\begin{array}{cc}\hfill L(\varpi ,X|Y)& ={\beta }_{e}p{e}^{\varpi (1-p)}+{\beta }_e(1-p)e^{\varpi p}+1-{\beta }_e.\hfill \end{array}$$

(18)

Furthermore, it is readily seen that

$$\begin{array}{cc}\hfill C(p,\varpi ,{\beta }_e)& =\underset{p}{max}\left\{L(\varpi ,p)-\left\{{\beta }_{e}p{e}^{\varpi (1-p)}+{\beta }_e(1-p)e^{\varpi p}+1-{\beta }_e\right\}\right\}\hfill \\ & =(1-{\beta }_e)\left\{\underset{p}{max}\left\{L(\varpi ,p)\right\}-1\right\},\hfill \end{array}$$

(19)

where $L(\varpi ,p)=p{e}^{\varpi (1-p)}+(1-p)e^{\varpi p}$. Moreover, we have the solution $p=1/2$ leads to $\frac{\partial L(\varpi ,p)}{\partial p}=0$ and the corresponding second derivative is

$$\begin{array}{cc}\hfill \frac{{\partial }^{2}L(\varpi ,p)}{\partial p^2}& =e^{\varpi (1-p)}(\varpi p-2)\varpi +e^{\varpi p}[(1-p)\varpi -2]\varpi <0,\hfill \end{array}$$

(20)

which results from the condition $0<\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$.

Therefore, it is readily seen that, in the case $p=1/2$, the capacity $C(p,\varpi ,{\beta }_e)$ reaches the maximum value. □

Remark 2.

Proposition 2 indicates that, in the case ${\beta }_e=1$, the lower bound of the capacity is obtained, that is $C\left({\beta }_e\right)=0$. However, if a certain information transfer process is satisfied (namely ${\beta }_e=0$), we will have the maximum MILC. Similar to Proposition 1, the uniform distribution is selected to reach the capacity in practice.

3.3. Strongly Symmetric Backward Matrix

As for a strongly symmetric backward matrix, it is viewed as a special example of information transmission. The discussion for the message transfer capacity in this case is similar to that in the symmetric matrix, whose details are given as follows.

Proposition 3.

For an information transmission from the source X to the sink Y, assume that there exists a strongly symmetric backward matrix as follows:

$$\begin{array}{c}\hfill p\left(x\right|y)=\left[\begin{array}{cccc}1-{\beta }_k& \frac{{\beta }_k}{K-1}& ...& \frac{{\beta }_k}{K-1}\\ \frac{{\beta }_k}{K-1}& 1-{\beta }_k& ...& \frac{{\beta }_k}{K-1}\\ ...& ...& ...& ...\\ \frac{{\beta }_k}{K-1}& ...& \frac{{\beta }_k}{K-1}& 1-{\beta }_k\end{array}\right],\end{array}$$

(21)

which indicates that X and Y both obey K-ary distribution. We have

$$\begin{array}{c}\hfill C(\varpi ,{\beta }_k)=e^{\frac{\varpi (K-1)}{K}}-\{(1-{\beta }_k)e^{\varpi {\beta }_k}+{\beta }_k{e}^{\varpi (1-\frac{{\beta }_k}{K-1})}\},\end{array}$$

(22)

where $0\le {\beta }_k\le 1$, $K\ge 2$ and $0<\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$.

Proof of Proposition 3.

For given K-ary variables X and Y whose distribution are $\{p\left(x_1\right),p\left(x_2\right),...,p\left(x_K\right)\}$ and $\{p\left(y_1\right),p\left(y_2\right),...,p\left(y_K\right)\}$ respectively, we can use the strongly symmetric backward matrix to obtain the relationship between the two variables as follows:

$$\begin{array}{cc}\hfill p\left(x_i\right)& =(1-{\beta }_k)p\left(y_i\right)+\frac{{\beta }_k}{K-1}[1-p\left(y_i\right)](i=1,2,...,K),\hfill \end{array}$$

(23)

which implies $p\left(x_i\right)$ is a one-to-one onto function for $p\left(y_i\right)$.

In accordance with Definition 2, it is easy to see that

$$\begin{array}{cc}\hfill L(\varpi ,X|Y)& =\sum _{x_i}\sum _{y_j}p\left(y_j\right)p\left(x_i\right|y_j)e^{\varpi (1-p\left(x_i\right|y_j))}\hfill \\ & =\sum _{y_j}p\left(y_j\right)\left\{(1-{\beta }_k)e^{\varpi {\beta }_k}+{\beta }_k{e}^{\varpi (1-\frac{{\beta }_k}{K-1})}\right\}\hfill \\ & =(1-{\beta }_k)e^{\varpi {\beta }_k}+{\beta }_k{e}^{\varpi (1-\frac{{\beta }_k}{K-1})}.\hfill \end{array}$$

(24)

Moreover, by virtue of the definition of MILC in Equation (9), it is readily seen that

$$\begin{array}{cc}\hfill C(\varpi ,{\beta }_k)& =\underset{p\left(x\right)}{max}\left\{L(\varpi ,X)\right\}-[(1-{\beta }_k)e^{\varpi {\beta }_k}+{\beta }_k{e}^{\varpi (1-\frac{{\beta }_k}{K-1})}],\hfill \end{array}$$

(25)

where $L(\varpi ,X)={\sum }_{x_i}p\left(x_i\right)e^{\varpi \{1-p\left(x_i\right)\}}$.

Then, by using Lagrange multiplier method, we have

$$\begin{array}{cc}\hfill G(p\left(x_i\right),{\lambda }_0)& =\sum _{x_i}p\left(x_i\right)e^{\varpi (1-p\left(x_i\right))}+{\lambda }_0\left[\sum _{x_i}p\left(x_i\right)-1\right].\hfill \end{array}$$

(26)

By setting $\frac{\partial G(p\left(x_i\right),{\lambda }_0)}{\partial p\left(x_i\right)}=0$ and $\frac{\partial G(p\left(x_i\right),{\lambda }_0)}{\partial {\lambda }_0}=0$, it can be readily verified that the extreme value of ${\sum }_{y_j}p\left(y_j\right)e^{\varpi (1-p\left(y_j\right))}$ is achieved by the uniform distribution as a solution, that is $p\left(x_1\right)=p\left(x_2\right)=...=p\left(x_K\right)=1/K$. In the case that $0<\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$, we have $\frac{{\partial }^{2}G(p\left(x_i\right),{\lambda }_0)}{\partial p^2\left(x_i\right)}<0$ with respect to $p\left(x_i\right)\in [0,1]$, which implies that the extreme value of ${\sum }_{x_i}p\left(x_i\right)e^{\varpi (1-p\left(x_i\right))}$ is the maximum value.

In addition, according to the Equation (23), the uniform distribution of variable X is resulted from the uniform distribution for variable Y.

Therefore, by substituting the uniform distribution for $p\left(x\right)$ into Equation (25), we will obtain the capacity $C(\varpi ,{\beta }_k)$. □

Furthermore, in light of Equation (22), we have

$$\begin{array}{cc}\hfill \frac{\partial C(\varpi ,{\beta }_k)}{\partial {\beta }_k}& =\{1-\varpi (1-{\beta }_k)\}{e}^{\varpi {\beta }_k}+\left\{\frac{\varpi {\beta }_k}{K-1}-1\right\}{e}^{\varpi (1-\frac{{\beta }_k}{K-1})}.\hfill \end{array}$$

(27)

By setting $\frac{\partial C(\varpi ,{\beta }_k)}{\partial {\beta }_k}=0$, it is apparent that $C(\varpi ,{\beta }_k)$ reaches the extreme value in the case that ${\beta }_k=\frac{K-1}{K}$. Additionally, when the parameter $\varpi$ satisfies $0<\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$, we also have the second derivative of the $C(\varpi ,{\beta }_k)$ as follows:

$$\begin{array}{cc}\hfill \frac{{\partial }^{2}C(\varpi ,{\beta }_k)}{\partial {\beta }_k^2}& =\varpi [2-(1-{\beta }_k)\varpi ]e^{\varpi {\beta }_k}+\frac{\varpi }{K-1}\left\{2-\frac{\varpi {\beta }_k}{K-1}\right\}{e}^{\varpi (1-\frac{{\beta }_k}{K-1})}>0,\hfill \end{array}$$

(28)

which indicates that the convex $C(\varpi ,{\beta }_k)$ reaches the minimum value 0 in the case ${\beta }_k=\frac{K-1}{K}$.

Remark 3.

According to Proposition 3, when ${\beta }_k=\frac{K-1}{K}$, namely, the channel is just random, we gain the lower bound of the capacity namely $C(\varpi ,{\beta }_k)=0$. On the contrary, when ${\beta }_k=0$ (that is, there is a certain channel), we will have the maximum capacity.

4. Distortion of Message Importance Transfer

In this section, we will focus on the information transfer distortion, a common problem of information processing. In a real information system, there exists inevitable information distortion caused by noises or other disturbances, though the devices and hardware of telecommunication systems are updating and developing. Fortunately, there are still some bonuses from allowable distortion in some scenarios. For example, in conventional information theory, rate distortion is exploited to obtain source compression such as predictive encoding and hybrid encoding, which can save a lot of hardware resources and communication traffic [39].

Similar to the rate distortion theory for Shannon entropy [38], a kind of information distortion function based on MIM and CMIM is defined to characterize the effect of distortion on the message importance loss. In particular, there are some details of discussion as follows.

Definition 5.

Assume that there exists an information transfer process $\{X,p(y\left|x\right),Y\}$ from the variable X to Y, where the $p\left(y\right|x)$ denotes a transfer matrix (distributions of X and Y are denoted by $p\left(x\right)$ and $p\left(y\right)$ respectively). For a given distortion function $d(x,y)$ ($d(x,y)\ge 0$) and an allowable distortion D, the message importance distortion function is defined as

$$\begin{array}{cc}\hfill R_{\varpi }\left(D\right)& =\underset{p\left(y\right|x)\in B_D}{min}{\Phi }_{\varpi }\left(X\right|\left|Y\right)\hfill \\ & =\underset{p\left(y\right|x)\in B_D}{min}\{L(\varpi ,X)-L(\varpi ,X|Y)\},\hfill \end{array}$$

(29)

in which $L(\varpi ,X)={\sum }_{x_i}p\left(x_i\right)e^{\varpi \{1-p\left(x_i\right)\}}$, $L(\varpi ,X|Y)$ is defined by Equation (4), $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$ and $B_D=\{q\left(y\right|x):\overline{D}\le D\}$ denotes the allowable information transfer matrix set where

$$\begin{array}{c}\overline{D}=\sum _{x_i}\sum _{y_j}p\left(x_i\right)p\left(y_j\right|x_i)d(x_i,y_j),\hfill \end{array}$$

(30)

which is the average distortion.

In this model, the information source X is given and our goal is to select an adaptive $p\left(y\right|x)$ to achieve the minimum allowable message importance loss under the distortion constraint. This provides a new theoretical guidance for information source compression from the perspective of message importance.

In contrast to the rate distortion of Shannon information theory, this new information distortion function just depends on the message importance loss rather than entropy loss to choose an appropriate information compression matrix. In practice, there are some similarities and differences between the rate distortion theory and the message importance distortion in terms of the source compression. On one hand, both two information distortion encodings can be regarded as special information transfer processes just with different optimization objectives. On the other hand, the new distortion theory tries to keep the rare events as high as possible, while the conventional rate distortion focuses on the amount of information itself. To some degree, by reducing more redundant common information, the new source compression strategy based on rare events (viewed as message importance) may save more computing and storage resources in big data.

4.1. Properties of Message Importance Distortion Function

In this subsection, we shall discuss some fundamental properties of rate distortion function based on message importance in details.

4.1.1. Domain of Distortion

Here, we investigate the domain of allowable distortion, namely $[D_{min},D_{max}]$, and the corresponding message importance distortion function values as follows.

(i) The lower bound $D_{min}$: Due to the fact $0\le d(x_i,y_j)$, it is easy to obtain the non-negative average distortion, namely $0\le \overline{D}$. Considering $\overline{D}\le D$, we readily have the minimum allowable distortion, that is

$$\begin{array}{c}\hfill D_{min}=0,\end{array}$$

(31)

which implies the distortionless case, namely Y is the same as X.

In addition, when the lower bound $D_{min}$ (namely the distortionless case) is satisfied, it is readily seen that

$$\begin{array}{cc}\hfill L(\varpi ,X|Y)& =L(\varpi ,X|X)=\sum _{x_i}p\left(x_i\right)p\left(x_i\right|x_i)e^{\varpi \{1-p\left(x_i\right|x_i)\}}=1,\hfill \end{array}$$

(32)

and according to the Equation (29) the message importance distortion function is

$$\begin{array}{cc}\hfill R_{\varpi }\left(D_{min}\right)& =L(\varpi ,X)-L(\varpi ,X|X)=L(\varpi ,X)-1,\hfill \end{array}$$

(33)

where $L(\varpi ,X)={\sum }_{x_i}p\left(x_i\right)e^{\varpi \{1-p\left(x_i\right)\}}$ and $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$.

(ii) The upper bound $D_{max}$: When the allowable distortion satisfies $D\ge D_{max}$, it is apparent that the variables X and Y are independent, that is, $p\left(y\right|x)=p(y)$. Furthermore, it is not difficult to see that

$$\begin{array}{cc}\hfill D_{max}& =\underset{p\left(y\right)}{min}\left\{\sum _{x_i}\sum _{y_j}p\left(x_i\right)p\left(y_j\right)d(x_i,y_j)\right\}\hfill \\ & =\sum _{y_j}p\left(y_j\right)\underset{p\left(y\right)}{min}\left\{\sum _{x_i}p\left(x_i\right)d(x_i,y_j)\right\}\hfill \\ & \ge \underset{y_j}{min}\left\{\sum _{x_i}p\left(x_i\right)d(x_i,y_j)\right\},\hfill \end{array}$$

(34)

which indicates that when the distribution of variable Y follows $p\left(y_j\right)=1$ and $p\left(y_l\right)=0$ ($l\ne j$), we have the upper bound

$$\begin{array}{c}\hfill D_{max}=\underset{y_j}{min}\left\{\sum _{x_i}p\left(x_i\right)d(x_i,y_j)\right\}.\end{array}$$

(35)

Additionally, on account of the independent X and Y, namely $p\left(x\right|y)=p(x)$, it is readily seen that

$$\begin{array}{cc}\hfill R_{\varpi }\left(D_{max}\right)& =L(\varpi ,X)-\sum _{y_j}p\left(y_j\right)L(\varpi ,X)=0.\hfill \end{array}$$

(36)

4.1.2. The Convexity Property

For two allowable distortions $D_a$ and $D_b$, whose optimal allowable information transfer matrixes are $p_a\left(y\right|x)$ and $p_b\left(y\right|x)$ respectively, we have

$$\begin{array}{c}{R}_{\varpi }(\delta D_a+(1-\delta )D_b)\le \delta R_{\varpi }\left(D_a\right)+(1-\delta )R_{\varpi }\left(D_b\right),\hfill \end{array}$$

(37)

where $0\le \delta \le 1$ and $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$.

Proof. 

Refer to the Appendix A. □

4.1.3. The Monotonically Decreasing Property

For two given allowable distortions $D_a$ and $D_b$, if $0\le D_a<D_b<D_{max}$ is satisfied, we have $R_{\varpi }\left(D_a\right)\ge R_{\varpi }\left(D_b\right)$, where $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$.

Proof. 

Considering that $0\le D_a<D_b<D_{max}$, we have $D_b=\gamma D_a+(1-\gamma )D_{max}$ where $\gamma =\frac{D_{max}-D_b}{D_{max}-D_a}$. On account of the Equation (36) and the convexity property mentioned in Equation (37), it is not difficult to see that

$$\begin{array}{cc}\hfill R_{\varpi }\left(D_b\right)& \le \gamma R_{\varpi }\left(D_a\right)+(1-\gamma )R_{\varpi }\left(D_{max}\right)=\gamma R_{\varpi }\left(D_a\right)<R_{\varpi }\left(D_a\right),\hfill \end{array}$$

(38)

where $0<\gamma <1$. □

4.1.4. The Equivalent Expression

For an information transfer process $\{X,p(y\left|x\right),Y\}$, if we have a given distortion function $d(x,y)$, an allowable distortion D and a average distortion $\overline{D}$ defined in Equation (30), the message importance distortion function defined in Equation (29) can be rewritten as

$$\begin{array}{c}{R}_{\varpi }\left(D\right)=\underset{\overline{D}=D}{min}\{L(\varpi ,X)-L(\varpi ,X|Y)\},\hfill \end{array}$$

(39)

where $L(\varpi ,X)$ and $L(\varpi ,X|Y)$ are defined by the Equations (1) and (4), as well as $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$.

Proof. 

For a given allowable distortion D, if there exists an allowable distortion $D^{*}$ ($D_{min}\le D^{*}<D<D_{max}$) and the corresponding optimal information transfer matrix $p^{*}\left(y\right|x)$ leads to $R_{\varpi }\left(D\right)$, we will have $R_{\varpi }\left(D\right)=R_{\varpi }\left(D^{*}\right),$ which contradicts the monotonically decreasing property. □

4.2. Analysis for Message Importance Distortion Function

In this subsection, we shall investigate the computation of message importance distortion function, which has a great impact on the probabilistic events analysis in practice. Actually, the definition of message importance distortion function in Equation (29) can be regarded as a special function, which is the minimization of the message importance loss with the symbol error less than or equal to the allowable distortion D. In particular, Definition 5 can also be expressed as the following optimization:

$$\begin{array}{cc}\hfill {\mathcal{P}}_1:\underset{p\left(y_j\right|x_i)}{min}& \left\{L\right(\varpi ,X)-L(\varpi ,X\left|Y\right)\}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \sum _{x_i}\sum _{y_j}p\left(x_i\right)p\left(y_j\right|x_i)d(x_i,y_j)\le D,\hfill \\ & \sum _{y_j}p\left(y_j\right|x_i)=1,\hfill \\ & p\left(y_j\right|x_i)\ge 0,\hfill \end{array}$$

(40)

where $L(\varpi ,X)$ and $L(\varpi ,X|Y)$ are MIM and CMIM defined in Equations (1) and (4), as well as $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$.

To take a computable optimization problem as an example, we consider Hamming distortion as the distortion function $d(x,y)$, namely

$$\begin{array}{c}\hfill d(x,y)=\left[\begin{array}{cccc}0& 1& \dots & 1\\ 1& 0& \dots & 1\\ \dots & \dots & \dots \\ 1& 1& \dots & 0\end{array}\right],\end{array}$$

(41)

which means $d(x_i,y_i)=0$ and $d(x_i,y_j)=1$ ($i\ne j$). In order to reveal some intrinsic meanings of $R_{\varpi }\left(D\right)$, we investigate an information transfer of Bernoulli source as follows.

Proposition 4.

For a Bernoulli(p) source denoted by a variable X and an information transfer process $\{X,p(y\left|x\right),Y\}$ with Hamming distortion, the message importance distortion function is given by

$$\begin{array}{c}\hfill \begin{array}{c}{R}_{\varpi }\left(D\right)=\{p{e}^{\varpi (1-p)}+(1-p)e^{\varpi p}\}-\{D{e}^{\varpi (1-D)}+(1-D)e^{\varpi D}\},\hfill \end{array}\end{array}$$

(42)

and the corresponding information transfer matrix is

$$\begin{array}{c}\hfill p\left(y\right|x)=\left[\begin{array}{cc}\frac{(1-D)(p-D)}{p(1-2D)}& \frac{(1-p-D)D}{p(1-2D)}\\ \frac{D(p-D)}{(1-p)(1-2D)}& \frac{(1-p-D)(1-D)}{(1-p)(1-2D)}\end{array}\right],\end{array}$$

(43)

where $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$ and $0\le D\le min\{p,1-p\}$.

Proof of Proposition 4.

Refer to the Appendix B. □

5. Bitrate Transmission Constrained by Message Importance

We investigate the information capacity in the case of a limited message importance loss in this section. The objective is to achieve the maximum transmission bitrate under the constraint of a certain message importance loss $ϵ$. The maximum transmission bitrate is one of system invariants in a transmission process, which provides a upper bound of amount of information obtained by the receiver.

In an information transmission process, the information capacity is the mutual information between the encoded signal and the received signal with the dimension bit/symbol. In a real transmission, there always exists an allowable distortion between the sending sequence X and the received sequence Y, while the maximum allowable message importance loss is required to avoid too much distortion of important events. From this perspective, message importance loss is considered to be another constraint for the information transmission capacity beyond the information distortion. Therefore, this might play a crucial role in the design of transmission in information processing systems.

In particular, we characterize the maximizing mutual information constrained by a controlled message importance loss as follows:

$$\begin{array}{cc}\hfill {\mathcal{P}}_2:\underset{p\left(x\right)}{max}& I\left(X\right|\left|Y\right)\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& L(\varpi ,X)-L(\varpi ,X|Y)\le ϵ,\hfill \\ & \sum _{y_j}p\left(x_i\right)=1,\hfill \\ & p\left(x_i\right)\ge 0,\hfill \end{array}$$

(44)

where $I\left(X\right|\left|Y\right)={\sum }_{x_i,y_j}p\left(x_i\right)p\left(y_j\right|x_i)log\frac{p\left(x_i\right)p\left(y_j\right|x_i)}{p\left(y_j\right)}$, $p\left(y_j\right)={\sum }_{x_i}p\left(x_i\right)p\left(y_j\right|x_i)$, $L(\varpi ,X)$ and $L(\varpi ,X|Y)$ are MIM and CMIM defined in Equations (1) and (4), as well as $0<\varpi \le \frac{2{min}_j\left\{p\left(y_j\right)\right\}}{{max}_i\left\{p\left(x_i\right)\right\}}$.

Actually, the bitrate transmission with a message importance loss constraint has a special solution for a certain scenario. In order to give a specific example, we investigate the optimization problem in the Bernoulli(p) source with a symmetric or erasure transfer matrix as follows.

5.1. Binary Symmetric Matrix

Proposition 5.

For a Bernoulli(p) source X whose distribution is $\{p,1-p\}$ ($0\le p\le 1/2$) and an information transfer process $\{X,p(y\left|x\right),Y\}$ with transfer matrix

$$\begin{array}{c}\hfill p\left(y\right|x)=\left[\begin{array}{cc}1-{\beta }_s& {\beta }_s\\ {\beta }_s& 1-{\beta }_s\end{array}\right],\end{array}$$

(45)

we have the solution for ${\mathcal{P}}_2$ defined in Equation (44) as follows:

$$\begin{array}{c}\underset{p\left(x\right)}{max}I\left(X\right|\left|Y\right)\hfill \\ =\left\{\begin{array}{c}1-H\left({\beta }_s\right),(ϵ\ge C_{{\beta }_s})\hfill \\ H(p_s(1-{\beta }_s)+(1-p_s){\beta }_s)-H\left({\beta }_s\right),(0<ϵ\le C_{{\beta }_s}),\hfill \end{array}\right.\hfill \end{array}$$

(46)

where $p_s$ is the solution of $L(\varpi ,X)-L(\varpi ,X|Y)=ϵ$ ($L(\varpi ,X)$ and $L(\varpi ,X|Y)$ mentioned in the optimization problem ${\mathcal{P}}_2$), whose approximate value is

$$\begin{array}{c}\hfill p_s\doteq \frac{1-\sqrt{\Theta }}{2},\end{array}$$

(47)

in which the parameter Θ is given by

$$\begin{array}{cc}\hfill \Theta & =1-\frac{4ϵ}{4\varpi +{\varpi }^2}-\frac{4\sqrt{{(1-2{\beta }_s)}^2{ϵ}^2+2(4\varpi +{\varpi }^2){\beta }_s(1-{\beta }_s)ϵ}}{(4\varpi +{\varpi }^2)|1-2{\beta }_s|},\hfill \end{array}$$

(48)

and $H(·)$ denotes the operator for Shannon entropy, that is $H\left(p\right)=-\left[\right(1-p)log(1-p)+plogp]$, $C_{{\beta }_s}=e^{\frac{\varpi }{2}}-\{{\beta }_s{e}^{\varpi (1-{\beta }_s)}+(1-{\beta }_s)e^{\varpi {\beta }_s}\}$ ($0\le {\beta }_s\le 1$) and $\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$.

Proof of Proposition 5.

Considering the Bernoulli(p) source X following $\{p,1-p\}$ and the binary symmetric matrix, it is not difficult to gain

$$\begin{array}{cc}\hfill I\left(X\right|\left|Y\right)& =H\left(Y\right)-H\left(Y\right|X)\hfill \\ & =-\{p\left(y_0\right)logp\left(y_0\right)+p\left(y_1\right)logp\left(y_1\right)\}-H\left({\beta }_s\right),\hfill \end{array}$$

(49)

where $p\left(y_0\right)=p(1-{\beta }_s)+(1-p){\beta }_s$, $p\left(y_1\right)=p{\beta }_s+(1-p)(1-{\beta }_s)$ and $H\left({\beta }_s\right)=-[(1-{\beta }_s)log(1-{\beta }_s)+{\beta }_{s}log{\beta }_s]$.

Moreover, define the Lagrange function as $G_s\left(p\right)=I\left(X\right|\left|Y\right)+{\lambda }_s(L(\varpi ,X)-L(\varpi ,X|Y)-ϵ)$ where $ϵ>0$, $0\le p\le 1/2$ and ${\lambda }_s\ge 0$. It is not difficult to have the partial derivative of $G_s\left(p\right)$ as follows:

$$\begin{array}{c}\frac{\partial G_s\left(p\right)}{\partial p}=\frac{\partial I\left(X\right|\left|Y\right)}{\partial p}+{\lambda }_s\frac{\partial C(p,\varpi ,{\beta }_s)}{\partial p},\hfill \end{array}$$

(50)

where $\frac{\partial C(p,\varpi ,{\beta }_s)}{\partial p}$ is given by the Equation (14) and

$$\begin{array}{c}\hfill \frac{\partial I\left(X\right|\left|Y\right)}{\partial p}=(1-2{\beta }_s)log\left\{\frac{(2{\beta }_s-1)p+1-{\beta }_s}{(1-2{\beta }_s)p+{\beta }_s}\right\}.\end{array}$$

(51)

By virtue of the monotonic increasing function $log\left(x\right)$ for $x>0$, it is easy to see the nonnegativity of $\frac{\partial I\left(X\right|\left|Y\right)}{\partial p}$ is equal to $(1-2{\beta }_s)\{(2{\beta }_s-1)p+1-{\beta }_s-\left[(1-2{\beta }_s)p+{\beta }_s\right]\}=(1-2p){(1-2{\beta }_s)}^2\ge 0$ in the case $0\le p\le 1/2$. Moreover, due to the nonnegative $\frac{\partial C(p,\varpi ,{\beta }_s)}{\partial p}$ in $p\in [0,1/2]$ which is mentioned in the proof of Proposition 1, it is readily seen that $\frac{\partial G_s\left(p\right)}{\partial p}\ge 0$ is satisfied under the condition $0\le p\le 1/2$.

Thus, the optimal solution $p_s^{*}$ is the maximal available p ($p\in [0,1/2]$) as follows:

$$\begin{array}{c}\hfill p_s^{*}=\left\{\begin{array}{c}\frac{1}{2},\mathrm{for}ϵ\ge C_{{\beta }_s},\hfill \\ p_s,\mathrm{for}0<ϵ\le C_{{\beta }_s},\hfill \end{array}\right.\end{array}$$

(52)

where $p_s$ is the solution of $L(\varpi ,X)-L(\varpi ,X|Y)=ϵ$, and $C_{{\beta }_s}$ is the MILC mentioned in Equation (11).

By using Taylor series expansion, the equation $L(\varpi ,X)-L(\varpi ,X|Y)=ϵ$ can be expressed approximately as follows:

$$\begin{array}{c}(2\varpi +\frac{{\varpi }^2}{2})\left\{(1-p)p-\frac{p(1-p){\beta }_s(1-{\beta }_s)}{\left[(2{\beta }_s-1)p+1-{\beta }_s\right]\left[(1-2{\beta }_s)p+{\beta }_s\right]}\right\}=ϵ,\hfill \end{array}$$

(53)

whose solution is the approximate $p_s$ as the Equation (47).

Therefore, by substituting the $p_s^{*}$ into Equation (49), we have Equation (46). □

Remark 4.

Proposition 5 gives the maximum transmission bitrate under the constraint of message importance loss. Particularly, there are growth regions and smooth regions for the maximum transmission bitrate in the receiver with respect to message importance loss ϵ. When the message importance loss ϵ is constrained in a little range, the real bitrate is less than the Shannon information capacity, which is involved with the entropy of the symmetric matrix parameter ${\beta }_s$.

5.2. Binary Erasure Matrix

Proposition 6.

Assume that there is a Bernoulli(p) source X following distribution $\{p,1-p\}$ ($0\le p\le 1/2$) and an information transfer process $\{X,p(y\left|x\right),Y\}$ with the binary erasure matrix

$$\begin{array}{c}\hfill p\left(y\right|x)=\left[\begin{array}{ccc}1-{\beta }_e& 0& {\beta }_e\\ 0& 1-{\beta }_e& {\beta }_e\end{array}\right],\end{array}$$

(54)

where $0\le {\beta }_e\le 1$. In this case, the solution for ${\mathcal{P}}_2$ described in Equation (44) is

$$\begin{array}{c}\underset{p\left(x\right)}{max}I\left(X\right|\left|Y\right)\hfill \\ =\left\{\begin{array}{c}1-{\beta }_e,(ϵ\ge C_{{\beta }_e})\hfill \\ (1-{\beta }_e)H\left(p_e\right),(0<ϵ\le C_{{\beta }_s}),\hfill \end{array}\right.\hfill \end{array}$$

(55)

where $p_e$ is the solution of $(1-{\beta }_e)\{p{e}^{\varpi (1-p)}+(1-p)e^{\varpi p}-1\}=ϵ$, whose approximate value is

$$\begin{array}{c}\hfill p_e\doteq \frac{1-\sqrt{1-\frac{8ϵ}{(1-{\beta }_e)(4\varpi +{\varpi }^2)}}}{2},\end{array}$$

(56)

and $H\left(x\right)=-\left[\right(1-x)log(1-x)+xlogx]$, $C_{{\beta }_e}=(1-{\beta }_e)(e^{\frac{\varpi }{2}}-1)$ and $\varpi <2\le 2/max\left\{p\left(x_i\right)\right\}$.

Proof of Proposition 6.

In the binary erasure matrix, considering the Bernoulli(p) source X whose distribution is $\{p,1-p\}$, it is readily seen that

$$\begin{array}{cc}\hfill I\left(X\right|\left|Y\right)& =H\left(Y\right)-H\left(Y\right|X)\hfill \\ & =(1-{\beta }_e)H\left(p\right),\hfill \end{array}$$

(57)

where $H(·)$ denotes the Shannon entropy operator, namely $H\left(p\right)=-\left[\right(1-p)log(1-p)+plogp]$.

Moreover, according to the Definitions 1 and 2, it is easy to see that

$$L(\varpi ,X)-L(\varpi ,X|Y)=(1-{\beta }_e)\{L(\varpi ,p)-1\},$$

(58)

where $L(\varpi ,p)=p{e}^{\varpi (1-p)}+(1-p)e^{\varpi p}$.

Similar to the proof of the Proposition 5 and considering the monotonically increasing $H\left(p\right)$ and $L(\varpi ,p)$ in $p\in [0,1/2]$, it is not difficult to see that the optimal solution $p_e^{*}$ is the maximal available p in the case $0\le p\le \frac{1}{2}$, which is given by

$$\begin{array}{c}\hfill p_e^{*}=\left\{\begin{array}{c}\frac{1}{2},\mathrm{for}ϵ\ge C_{{\beta }_e},\hfill \\ p_e,\mathrm{for}0<ϵ\le C_{{\beta }_e},\hfill \end{array}\right.\end{array}$$

(59)

where $p_e$ is the solution of $(1-{\beta }_e)\{L(\varpi ,p)-1\}=ϵ$, and the upper bound $C_{{\beta }_e}$ is gained in Equation (16).

By resorting to Taylor series expansion, the approximate equation for $(1-{\beta }_e)\{L(\varpi ,p)-1\}=ϵ$ is given by

$$\begin{array}{c}\hfill (1-{\beta }_e)(2\varpi +\frac{{\varpi }^2}{2})(1-p)p=ϵ,\end{array}$$

(60)

from which the approximate solution $p_e$ in Equation (56) is obtained.

Therefore, Equation (55) is obtained by substituting the $p_e^{*}$ into the Equation (57). □

Remark 5.

From Proposition 6, there are two regions for the maximum transmission bitrate with respect to message importance loss. The one depends on the message importance loss threshold ϵ. The other is just related to the erasure matrix parameter ${\beta }_e$.

Note that single-letter models are discussed to show some theoretical results for information transfer under the constraint of massage importance loss, which may be used in some special potential applications such as maritime international signal or switch signal processing. As a matter of fact, in practice, it is preferred to operate multi-letters models which can be applied to more scenarios such as the multimedia communication, cooperative communications and multiple access, etc. As for these complicated cases which may be different from conventional Shannon information theory, we shall consider it in the near future.

6. Numerical Results

This section shall provide numerical results to validate the theoretical results in this paper.

6.1. The Message Importance Loss Capacity

First of all, we give some numerical simulation with respect to the MILC in different information transmission cases. In Figure 2, it is apparent to see that if the Bernoulli source follows the uniform distribution, namely $p=0.5$, the message importance loss will reach the maximum in the cases of different matrix parameter ${\beta }_s$. That is, the numerical results of MILC are obtained as $\{0.4081,0.0997,0,0.2265\}$ in the case of parameter ${\beta }_s=\{0.1,0.3,0.5,0.8\}$ and $\varpi =1$, which corresponds to Proposition 1. Moreover, we also know that if ${\beta }_s=0.5$, namely the random transfer matrix is satisfied, the MILC reaches the lower bound that is $C=0$. In contrast, if the parameter ${\beta }_s$ satisfies ${\beta }_s=0$, the upper bound of MILC will be gained such as $\{0.1618,0.4191,0.6487,1.7183\}$ in the case $\varpi =\{0.3,0.7,1.0,2.0\}$.

Figure 3 shows that, in the binary erasure matrix, the MILC is reached under the same condition as that in the binary symmetric matrix, namely $p=0.5$. For example the numerical results of MILC with $\varpi =1$ are $\{0.5838,0.4541,0.3244,0.1297\}$ in the cases ${\beta }_e=\{0.1,0.3,0.5,0.8\}$. However, if ${\beta }_e=1$, the lower bound of MILC ($C=0$) is obtained in the erasure transfer matrix, different from the symmetric case.

From Figure 4, it is not difficult to see that the certain transfer matrix (namely ${\beta }_k=0$) leads to upper bound of MILC. For example, when the number of source symbols satisfies $K=\{4,6,8,10\}$, the numerical results of MILC with $\varpi =2$ are $\{3.4817,4.2945,4.7546,5.0496\}$. In addition, the lower bound of MILC is reached in the case that ${\beta }_k=1-\frac{1}{K}$.

6.2. Message Importance Distortion

We focus on the distortion of message importance transfer and give some simulations in this subsection. From Figure 5, it is illustrated that the message importance distortion function $R_{\varpi }\left(D\right)$ is monotonically non-increasing with respect to the distortion D, which can validate some properties mentioned in Section 4.1. Moreover, the maximum $R_{\varpi }\left(D\right)$ is obtained in the case $D=0$. Taking the Bernoulli(p) source as an example, the numerical results of $R_{\varpi }\left(D\right)$ with $\varpi =0.2$ are $\{0.0379,0.0674,0.0884,0.1010,0.1052\}$ and the corresponding probability satisfies $p=\{0.1,0.2,0.3,0.4,0.5\}$. Note that the turning point of $R_{\varpi }\left(D\right)$ is gained when the probability p equals to the distortion D, which conforms to Proposition 4.

6.3. Bitrate Transmission with Message Importance Loss

Figure 6 shows the allowable maximum bitrate (characterized by mutual information) constrained by a message importance loss $ϵ$ in a Bernoulli(p) source case. It is worth noting that there are two regions for the mutual information in the both transfer matrixes. In the first region, the mutual information is monotonically increasing with respect to the $ϵ$; however, in the second region, the mutual information is stable, namely the information transmission capacity is obtained. As for the numerical results, the turning points are obtained at $ϵ=\{0.0328,0.0185,0.0082,0.0021\}$ and the maximum mutual information values are $\{0.5310,0.2781,0.1187,0.0290\}$ in the binary symmetric matrix with the corresponding parameter ${\beta }_s=\{0.1,0.2,0.3,0.4\}$, while the turning points of erasure matrix are at $ϵ=\{0.0416,0.0410,0.0359,0.0308\}$ in the case that ${\beta }_e=\{0.1,0.2,0.3,0.4\}$ with the maximum mutual information values $\{0.9,0.8,0.7,0.6\}$. Consequently, Propositions 5 and 6 are validated from the numerical results.

6.4. Experimental Simulations

In this subsection, we take the binary stochastic process (in which the random variable follows Bernoulli distribution) as an example to validate theoretical results. In particular, the Bernoulli(p) source X (whose distribution is denoted by $P\left(X\right)=\{p,1-p\}$ where $0<p<1$) with the symmetric or erasure matrix (described by Equations (10) and (15)) is considered to reveal some properties of message importance loss capacity (in Section 3), message importance distortion function (in Section 4) as well as bitrate transmission constrained by message importance (in Section 5).

From Figure 7, it is seen that the uniform information source X (that is $P\left(X\right)=\{1/2,1/2\}$) leads to the maximum message importance loss (namely MILC) in both cases of symmetric matrix and erasure matrix, which implies Propositions 1 and 2. Moreover, with the increase of number of samples, the performance of massage importance loss tends to smooth. In addition, the MILC in symmetric transfer matrix is larger than that in the erasure one when the matrix parameters ${\beta }_s$ and ${\beta }_e$ are the same.

As for the distortion of message importance transfer, we investigate the message importance loss based on different transfer matrices, which is shown in Figure 8 where $p_{optimal}\left(y\right|x)$ is described as Equation (43), $p_{symmetric}\left(y\right|x)=\left[\begin{array}{cc}1-D& D\\ D& 1-D\end{array}\right]$, $p_{random1}\left(y\right|x)=\left[\begin{array}{cc}1-\frac{D}{10p}& \frac{D}{10p}\\ \frac{9D}{10(1-p)}& 1-\frac{9D}{10(1-p)}\end{array}\right]$, $p_{random2}\left(y\right|x)=\left[\begin{array}{cc}1-\frac{D}{5p}& \frac{D}{5p}\\ \frac{D}{5(1-p)}& 1-\frac{D}{5(1-p)}\end{array}\right]$, $p_{random3}\left(y\right|x)=\left[\begin{array}{cc}1-\frac{D}{10p}& \frac{D}{10p}\\ \frac{D}{10(1-p)}& 1-\frac{D}{10(1-p)}\end{array}\right]$, $p_{certain}\left(y\right|x)=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$, D is the allowable distortion and p is the probability element of Bernoulli(p) source. From Figure 8, it is illustrated that, when the $p_{optimal}\left(y\right|x)$ is selected as the transfer matrix, the massage importance loss reaches the minimum, which corresponds to Proposition 4. In addition, if the transfer matrix is not certain (existing distortion), message importance loss is decreasing with the increase of allowable distortion.

Considering the transmission with a message importance loss constraint, Figure 9 shows that, when the $p_s^{*}$ (given by Equation (52)) and $p_e^{*}$ (given by Equation (59)) are selected as the probability elements for the Bernoulli(p) source in the symmetric matrix and erasure matrix respectively, the corresponding mutual information values are larger than those based on other probability (such as $p_{random1}=(1-\sqrt{1-8ϵ})/2$ and $p_{random2}=(1-\sqrt{1-4ϵ})/2$). In addition, it is not difficult to see that, when the parameter ${\beta }_s$ is equal to ${\beta }_e$, the mutual information (constrained by a message importance loss) in symmetric transfer matrix is larger than that in the erasure one.

7. Conclusions

In this paper, we investigated the information processing from the perspective of an information measure i.e., MIM. Actually, with the help of parameter $\varpi$, the MIM has more flexibility and can be used widely. Here, we just focused on the MIM with $0\le \varpi \le 2/max\left\{p\left(x_i\right)\right\}$ which not only has properties of self-scoring values for probabilistic events but also has similarities with Shannon entropy in information compression and transmission. In particular, based on a system model with message importance processing, a message importance loss was presented. This measure can characterize the information distinction before and after a message transfer process. Furthermore, we have proposed the message importance loss capacity which can provide an upper bound for the message importance harvest in the information transmission. Moreover, the message importance distortion function, which is to select an information transfer matrix to minimize the message importance loss, was discussed to characterize the performance of information lossy compression from the viewpoint of message importance of events. In addition, we exploited the message importance loss to constrain the bitrate transmission so that the combined factors of message importance and amount of information are considered to guide an information transmission. To give the validation for theoretical analyses, some numerical results and experimental simulations were also presented in details. As the next step research, we are looking forward to exploiting real data to design some applicable strategies for information processing based on the MIM, as well as investigating the performance of multivariate systems in the sense of MIM.