Comparison Between Bayesian and Maximum Entropy Analyses of Flow Networks†

Abstract

We compare the application of Bayesian inference and the maximum entropy (MaxEnt) method for the analysis of flow networks, such as water, electrical and transport networks. The two methods have the advantage of allowing a probabilistic prediction of flow rates and other variables, when there is insufficient information to obtain a deterministic solution, and also allow the effects of uncertainty to be included. Both methods of inference update a prior to a posterior probability density function (pdf) by the inclusion of new information, in the form of data or constraints. The MaxEnt method maximises an entropy function subject to constraints, using the method of Lagrange multipliers, to give the posterior, while the Bayesian method finds its posterior by multiplying the prior with likelihood functions incorporating the measured data. In this study, we examine MaxEnt using soft constraints, either included in the prior or as probabilistic constraints, in addition to standard moment constraints. We show that when the prior is Gaussian, both Bayesian inference and the MaxEnt method with soft prior constraints give the same posterior means, but their covariances are different. In the Bayesian method, the interactions between variables are applied through the likelihood function, using second or higher-order cross-terms within the posterior pdf. In contrast, the MaxEnt method incorporates interactions between variables using Lagrange multipliers, avoiding second-order correlation terms in the posterior covariance. The MaxEnt method with soft prior constraints, therefore, has a numerical advantage over Bayesian inference, in that the covariance terms are avoided in its integrations. The second MaxEnt method with soft probabilistic constraints is shown to give posterior means of similar, but not identical, structure to the other two methods, due to its different formulation.

1. Introduction

The analysis of flow rates on networks is required for the design and monitoring of electrical, water, sewer, irrigation, fire suppression, drainage, oil, gas and any other networks through which fluids or energy are transported. Their analysis is an important engineering problem. Traditionally, these systems have been analysed using deterministic methods. These methods incorporate physical laws such as Kirchhoff’s first and second laws (conservation of mass and equivalence of potentials at nodes) and sufficient known parameter values, giving a closed-form set of equations which is solved for the (deterministic) solution. Deterministic methods yield precise parameter values but do not consider uncertainty, either due to a lack of knowledge of the state of the system or flow variability. To account for uncertainty, a probabilistic framework is required. There are two primary methods for probabilistic inference: Bayesian inference using Bayes’ rule, and maximum entropy (MaxEnt) analysis.

Bayes’ theorem comes from the product rule of probabilities. To use Bayes’ theorem, the prior and likelihood functions need to be chosen before the data are analysed. To analyse the data, a set of data values are incorporated in the likelihood function, which is then updated by Bayes’ rule to obtain the posterior. This process can be repeated for each data set by using the posterior as the prior for the next data set. The order in which the data sets are analysed should not impact the final result.

The analysis presented here follows the well-known Bayesian parameter estimation or regression procedure as described in [1]. Although the applications of this procedure are extremely vast, including least-squares regression, the authors are not aware of it being applied in its general form to estimate flows on a network. An example of using Bayes’ theorem with transient pipe flows is presented in Rougier and Goldstein [2] who solve the water hammer partial differential equations, incorporating uncertainty from the pipeline characteristics and the boundary conditions. Bayes’ theorem is used to estimate the flows, pressures and pipeline characteristics as time progresses, using data obtained through real-time monitoring of the pipeline in a few locations. As this method requires the solution of a partial differential equation which incorporates time and uncertainty, its computational cost is high and therefore is restricted to small networks; in the example, a single pipe is analysed. The Bayesian method can also be used to calibrate model parameters, often using least-squares regression. Savic et al. [3] provide a comprehensive review of calibration techniques used with water networks. As an alternative to predicting model parameters, Hutton et al. [4] use Bayes’ theorem to update the coefficients in an autoregressive, data-driven model to predict future flow rates (at two locations in their example) using current and previous flow rate observations. In their case study, they were able to provide accurate one-hour forecasts for the monitored locations. Hutton and Kapelan [5] extended their previous analysis to predict pipe bursts by considering the difference between their predicted and observed flow rates. They were able to detect abnormal flow conditions representing pipe bursts greater than 5% of normal flow conditions.

Entropy is a measure of uncertainty [6,7,8,9,10]. The MaxEnt method for inference can be derived from an axiomatic approach based on the axioms of locality, coordinate invariance and subsystem independence [11,12,13]. Alternatively the MaxEnt method can be derived from a combinatorial approach [14,15,16,17], which shows that the MaxEnt method infers the most probable distribution, subject to the constraints and prior [6,14,15,16,17,18,19,20]. The maximum relative entropy method (MaxEnt), equivalent to the minimum Kullback–Leibler divergence [21], is a method of inference used to infer or update a probability distribution describing an under-determined system, which respects all constraints imposed on the system and is closest to the prior distribution [8]. However, the MaxEnt method is a method of inference, with no guarantee that the inferred solution will be realised [6,8,9]. The validity of the distribution will depend on the assumptions used to construct the MaxEnt model. The MaxEnt method has been applied to predict the flows on water distribution networks [22,23,24,25], transportation networks [26], electrical networks [26] and generic flow networks [27,28,29].

There have been many studies on the connection between Bayes’ theorem and the MaxEnt method, with some authors suggesting that one can be obtained from the other (in either direction) [30,31,32,33]. Giffin and Caticha’s method [31,32,33] to obtain Bayes’ rule using MaxEnt requires the relative entropy function to be defined over the model parameters and the data. The normalisation constraint is applied, and the variables representing the data are constrained with a pdf (probability density function) representing the Bayesian likelihood function. The pdf constraint is applied to the parameters defined to be the prior in the Bayesian method. Bayes’ rule is obtained by dividing the inferred distribution by the pdf defined over the data parameters, i.e., dividing by the pdf constraint. Although this equivalence is mathematically correct, the second constraint appears somewhat contrived as it is applied over the parameters of the Bayesian prior.

The current authors have compared the probability distributions of quasi-Newton rules obtained when inferring the Jacobian or Hessian using Bayesian inference [34,35] and the MaxEnt method [36]. In both methods, the same Gaussian prior was used. In the MaxEnt method, secant equations were used as the constraints. In Bayes’ method, delta likelihood functions incorporating the secant equation were used to represent the data. It was found that both methods obtained the same posterior means, but the covariance matrices were found to be different.

In this study, we develop a Bayesian method to analyse flow networks (Section 2). This theory contains many features in common with the MaxEnt method of [25]. In Section 3.1, we present a MaxEnt theory using soft constraints that are implemented in the prior pdf. In Section 3.2, we compare the distributions obtained by the two methods. In Section 4.1, we also present a MaxEnt theory with soft constraints implemented using pdfs as constraints, and, in Section 4.2, we compare this to the Bayesian method. Finally, in Section 5, we discuss our findings.

2. Bayesian Analysis

Consider a flow network with N external flow rates and M internal flow rates, which can be assembled into the vectors Θ and $\mathit{Q}$, respectively, which, in turn, can be assembled into the vector

$$\mathbf{\Psi }=\left[\begin{array}{c}\mathbf{\Theta }\\ \mathit{Q}\end{array}\right],$$

(1)

In the Bayesian method, to avoid inconsistencies due to different network representations, we consider a basis set $\mathit{X}$ of n flow rates selected from Ψ as parameters of the pdf used to represent the uncertainty. The indices of the basis set $\mathit{X}$ in Ψ are given by the set $\mathcal{B}$, while the indices of the complementary non-basis set of flow rates in Ψ are given by $\mathcal{N}$. For closure, at least $N-1$ basis flow rates must be chosen but up to $N+M$ can be chosen. The derivation of the Bayesian method requires a prior belief of the state of the system, represented as a prior pdf $q(\mathit{X}$), which is updated using observed data to a posterior pdf according to Bayes’ rule:

$$p\left(\mathit{X}\right|\mathit{y})=\frac{p\left(\mathit{y}\right|\mathit{X}\left)q\right(\mathit{X})}{\underset{\mathrm{\Omega }}{\int \dots \int }p\left(\mathit{y}\right|\mathit{X})q\left(\mathit{X}\right)d\mathit{X}}\text{'}$$

(2)

where $p\left(\mathit{y}\right|\mathit{X})$ is the likelihood function, the denominator allows for normalisation, $\mathit{X}$ is the basis set of flow rates, $\mathit{y}$ is the vector of observed data and Ω is the domain of $\mathit{X}$. The flow rates $\overline{\mathit{X}}$ not included in the basis set are taken as functions of the model parameters $\mathit{X}$, using:

$$\overline{\mathit{X}}=\mathit{V}\mathit{X},$$

(3)

where

$$\mathit{V}=-{\mathit{A}}_{i\in \mathcal{V},j\in \mathcal{N}}^{-1}{\mathit{A}}_{i\in \mathcal{V},j\in \mathcal{B}},$$

(4)

$$\mathit{A}=\left[\begin{array}{c}\mathit{C}\\ \mathit{W}\mathrm{diag}\left(\mathit{K}\right)\\ \mathit{F}\\ \mathit{T}\mathrm{diag}\left(\mathit{K}\right)\end{array}\right],$$

(5)

in which

• diag() places the elements of a vector on the diagonal of a square matrix of zeros;
• the set $\mathcal{V}$ contains the $N+M-n$ indices of the equations required to uniquely define $\overline{\mathit{X}}$ from $\mathit{X}$;
• the matrix $\mathit{C}$ is an $N\times (N+M)$ connectivity matrix containing elements $\{-1,0,1\}$. Its entries $C_{i,r},\forall i\in \{1,\dots N\}$ indicate membership of edge r to the node i, given by 0 if the edge is not connected to the node, 1 if the assumed direction of $Q_m$ or ${\mathrm{\Theta }}_i$ is entering the node and $-1$ otherwise;
• the vector $\mathit{K}$ is an $(M+N)\times 1$ vector of flow resistances;
• the matrix $\mathit{W}$ is a $w\times (N+M)$ loop matrix containing elements $\{-1,0,1\}$, where w is the number of independent cycles (loops) within the network. Its entries $W_{i,r},\forall i\in \{1,\dots ,w\}$ indicate membership of edge r within loop i, given by 0 if the edge is not in the loop, 1 if the assumed direction of $Q_m$ is in a clockwise direction around the loop and $-1$ otherwise;
• the matrix $\mathit{F}$ is a $N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}\times N+M$ matrix containing either 0 or 1 in each of its elements. Each row will have a single 1 on the index corresponding to the dimension of the observed link, with the remaining elements set to 0;
• $N_{\widehat{\mathrm{\Theta }}}$ and $N_{\widehat{Q}}$ are the numbers of flow rate observation locations for flows entering/exiting or within the network respectively; and
• the matrix $\mathit{T}$ is a $h_c\times (M+N)$ pseudo-loop matrix containing $\{-1,0,1\}$, where $h_c$ is the number of potential difference constraints applied. The pseudo-loop matrix contains paths between nodes of known pressure or potential values. For convenience, these are referenced to the potential at a single reference node $H_0$; this gives $\langle {\mathit{Y}}_{\mathit{T}}\rangle$ as the $h_c\times 1$ vector of mean potential differences between $H_0$ and $H_j$, for all nodes with potential observations. The entries in $T_{i,r},\forall i\in \{1,\dots ,h_c\}$ indicate membership of edge r within the potential difference constraint index i, given by 0 if the edge is not in the constraint, 1 if the assumed direction of $Q_m$ is defined as in the direction from node 0 to node j, and $-1$ otherwise.

The prior is chosen to represent one’s belief of the system state before incorporating any measured data. Although any distribution which represents what is believed about the system state could be chosen, in this study, a multidimensional Gaussian distribution is selected, defined over the real domain:

$$q\left(\mathit{X}\right)=\frac{\exp \left(-\frac{1}{2}{\left(\mathit{X}-\mathit{m}\right)}^{\top }{\mathbf{\Sigma }}^{-1}\left(\mathit{X}-\mathit{m}\right)\right)}{{\left(2\pi \right)}^{\frac{n}{2}}{\left|\mathbf{\Sigma }\right|}^{\frac{1}{2}}},$$

(6)

where $\mathit{m}$ is the $n\times 1$ vector of prior mean flow rates and Σ is the $n\times n$ matrix of prior covariances.

In Bayes’ method, likelihood functions are used to incorporate the physics of the system as well as any observed data, as follows:

• The likelihood function to incorporate conservation of mass at each node or Kirchhoff’s first law (or the flow rate for incompressible systems) is given by a delta function

  $$p\left(\mathbf{0}\right|\mathit{X})=\delta \left(\mathbf{0}-\left({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\right)\mathit{X}\right),$$

  (7)

  where

  $${\mathit{C}}_{\mathit{X}}={\mathit{C}}_{i\notin \mathcal{V},j\in \mathcal{B}},$$

  (8)

  $${\mathit{C}}_{\overline{\mathit{X}}}={\mathit{C}}_{i\notin \mathcal{V},j\in \mathcal{N}}\mathit{V}.$$

  (9)
  This delta function is defined by the limit of a Gaussian distribution

  $$-2\ln \left(p\left(\mathbf{0}\right|\mathit{X})\right)\propto \underset{{\mathbf{\Sigma }}_{\mathit{C}}\to \mathbf{0}}{\lim }{\left(\mathbf{0}-\left({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{C}}^{-1}\left(\mathbf{0}-\left({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\right)\mathit{X}\right).$$

  (10)
• The likelihood function to incorporate the loop laws for each loop, Kirchhoff’s second law, is given by a delta function

  $$p\left(\mathbf{0}\right|\mathit{X})=\delta \left(\mathbf{0}-\left({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\right)\mathit{X}\right),$$

  (11)

  where

  $${\mathit{W}}_{\mathit{X}}={\mathit{W}}_{i\notin \mathcal{V},j\in \mathcal{B}}\mathrm{diag}\left({\mathit{K}}_{i\in \mathcal{B}}\right),$$

  (12)

  $${\mathit{W}}_{\overline{\mathit{X}}}={\mathit{W}}_{i\notin \mathcal{V},j\in \mathcal{N}}\mathrm{diag}\left({\mathit{K}}_{i\in \mathcal{N}}\right)\mathit{V}.$$

  (13)

  This delta function is defined by the limit of the Gaussian distribution

  $$-2\ln \left(p\left(\mathbf{0}\right|\mathit{X})\right)\propto \underset{{\mathbf{\Sigma }}_{\mathit{W}}\to \mathbf{0}}{\lim }{\left(\mathbf{0}-\left({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{W}}^{-1}\left(\mathbf{0}-\left({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\right)\mathit{X}\right).$$

  (14)
• Observed flow rates can be constrained using the likelihood function

  $$-2\ln \left(p\left({\mathit{Y}}_{\mathit{F}}\right|\mathit{X})\right)\propto {\left({\mathit{Y}}_{\mathit{F}}-\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left({\mathit{Y}}_{\mathit{F}}-\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}\right),$$

  (15)

  where ${\mathit{Y}}_{\mathit{F}}$ is a $N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}\times 1$ vector that has the flow rate of each observation for a link in its elements, ${\mathbf{\Sigma }}_{\mathit{F}}$ is the $N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}\times N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}$ covariance matrix of the observations and

  $${\mathit{F}}_{\mathit{X}}={\mathit{F}}_{i\notin \mathcal{V},j\in \mathcal{B}},$$

  (16)

  $${\mathit{F}}_{\overline{\mathit{X}}}={\mathit{F}}_{i\notin \mathcal{V},j\in \mathcal{N}}\mathit{V}.$$

  (17)
• Observed potential differences can be constrained with the likelihood function

  $$-2\ln \left(p\left({\mathit{Y}}_{\mathit{T}}\right|\mathit{X})\right)\propto {\left({\mathit{Y}}_{\mathit{T}}-\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left({\mathit{Y}}_{\mathit{T}}-\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}\right),$$

  (18)

  where ${\mathit{Y}}_{\mathit{T}}$ is a $h_c\times 1$ vector that has the potential difference of an observation between two points in each element, ${\mathbf{\Sigma }}_{\mathit{T}}$ is the $h_c\times h_c$ covariance matrix of the observations and

  $${\mathit{T}}_{\mathit{X}}={\mathit{T}}_{i\notin \mathcal{V},j\in \mathcal{B}}\mathrm{diag}\left({\mathit{K}}_{i\in \mathcal{B}}\right),$$

  (19)

  $${\mathit{T}}_{\overline{\mathit{X}}}={\mathit{T}}_{i\notin \mathcal{V},j\in \mathcal{N}}\mathrm{diag}\left({\mathit{K}}_{i\in \mathcal{N}}\right)\mathit{V}.$$

  (20)

Applying Bayes’ rule with each of the likelihood functions, and expanding and dropping all terms which are not functions of $\mathit{X}$ gives the posterior in the form

$$-2\ln \left(p\left(\mathit{X}\right|\mathit{y})\right)\propto {\mathit{X}}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{X}-{\mathit{X}}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{m}-{\mathit{m}}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{X}+{\mathit{X}}^{\top }{\mathit{O}}^{\top }{\mathit{S}}^{-1}\mathit{O}\mathit{X}-{\mathit{y}}^{\top }{\mathit{S}}^{-1}\mathit{O}\mathit{X}-{\mathit{X}}^{\top }{\mathit{O}}^{\top }{\mathit{S}}^{-1}\mathit{y},$$

(21)

where

$$\mathit{O}=\left[\begin{array}{c}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\end{array}\right],$$

(22)

$${\mathit{S}}^{-1}=\left[\begin{array}{cccc}{\mathbf{\Sigma }}_{\mathit{C}}^{-1}& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{W}}^{-1}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right],$$

(23)

$$\mathit{y}=\left[\begin{array}{c}\mathbf{0}\\ \mathbf{0}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right].$$

(24)

Combining like factors gives

$$-2\ln \left(p\left(\mathit{X}\right|\mathit{y})\right)\propto {\mathit{X}}^{\top }\left({\mathbf{\Sigma }}^{-1}+{\mathit{O}}^{\top }{\mathit{S}}^{-1}\mathit{O}\right)\mathit{X}-{\mathit{X}}^{\top }\left({\mathbf{\Sigma }}^{-1}\mathit{m}+{\mathit{O}}^{\top }{\mathit{S}}^{-1}\mathit{y}\right)-\left({\mathit{m}}^{\top }{\mathbf{\Sigma }}^{-1}+{\mathit{y}}^{\top }{\mathit{S}}^{-1}\mathit{O}\right)\mathit{X}.$$

(25)

Completing the square gives

$$-2\ln \left(p\left(\mathit{X}\right|\mathit{y})\right)\propto {\left(\mathit{X}-\langle \mathit{X}\rangle \right)}^{\top }{\mathbf{\Sigma }}_{\mathit{p}}^{-1}\left(\mathit{X}-\langle \mathit{X}\rangle \right),$$

(26)

where the mean flow rates and variance matrix are given by

$$\langle \mathit{X}\rangle ={\mathbf{\Sigma }}_{\mathit{p}}\left({\mathbf{\Sigma }}^{-1}\mathit{m}+{\mathit{O}}^{\top }{\mathit{S}}^{-1}\mathit{y}\right),$$

(27)

$${\mathbf{\Sigma }}_{\mathit{p}}={\left({\mathbf{\Sigma }}^{-1}+{\mathit{O}}^{\top }{\mathit{S}}^{-1}\mathit{O}\right)}^{-1}.$$

(28)

Using the Woodbury matrix identity [37] to find the posterior covariance gives

$${\mathbf{\Sigma }}_{\mathit{p}}=\mathbf{\Sigma }-\mathbf{\Sigma }{\mathit{O}}^{\top }{\left(\mathit{S}+\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }\right)}^{-1}\mathit{O}\mathbf{\Sigma }.$$

(29)

The following algebra is needed to find a form which does not require an inversion of a zero matrix arising from the delta functions. Right multiplying the inverse posterior covariance by $\mathbf{\Sigma }{\mathit{O}}^{\top }$ gives

$${\mathbf{\Sigma }}_{\mathit{P}}^{-1}\mathbf{\Sigma }{\mathit{O}}^{\top }={\mathit{O}}^{\top }+{\mathit{O}}^{\top }{\mathit{S}}^{-1}\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }={\mathit{O}}^{\top }{\mathit{S}}^{-1}\left(\mathit{S}+\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }\right),$$

(30)

then left multiplying with the posterior covariance

$$\mathbf{\Sigma }{\mathit{O}}^{\top }={\mathbf{\Sigma }}_{\mathit{p}}{\mathit{O}}^{\top }{\mathit{S}}^{-1}\left(\mathit{S}+\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }\right).$$

(31)

Extracting ${\mathbf{\Sigma }}_{\mathit{p}}{\mathit{O}}^{\top }{\mathit{S}}^{-1}$ by right multiplying by ${\left(\mathit{S}+\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }\right)}^{-1}$ gives

$$\mathbf{\Sigma }{\mathit{O}}^{\top }{\left(\mathit{S}+\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }\right)}^{-1}={\mathbf{\Sigma }}_{\mathit{p}}{\mathit{O}}^{\top }{\mathit{S}}^{-1}.$$

(32)

The posterior mean flow rates can then be found from Equation (27) by substituting Equation (29) and Equation (32) to give

$$\langle \mathit{X}\rangle =\mathit{m}+\mathbf{\Sigma }{\mathit{O}}^{\top }{\left(\mathit{S}+\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }\right)}^{-1}\left(\mathit{y}-\mathit{O}\mathit{m}\right).$$

(33)

3. MaxEnt Analysis with Soft Constraints Implemented in the Prior

3.1. Formulation

The maximum entropy method is defined by the following algorithm [6,9]: (i) define a probability measure over the uncertainties of interest; (ii) construct a relative entropy function; (iii) define a prior probability function and constraints; (iv) maximise the entropy subject to the constraints and prior, to infer the probability distribution which describes the system; and, if desired, (v) extract statistical moments of quantities of interest. Soft MaxEnt constraints have previously been suggested by the authors [24,25] but have not been formally derived. To implement soft constraints, we define a pdf which expresses the uncertainty in the system defined over a reduced parameter set $\mathit{X}$, consisting of a basis set of n flow rates selected from Ψ and also the parameter observations ${\mathit{Y}}_{\mathit{F}}$ and ${\mathit{Y}}_{\mathit{T}}$. The indices of $\mathit{X}$ in Ψ are again given by the set $\mathcal{B}$. Again for closure, at least $N-1$ basis flow rates must be chosen but up to $N+M$ can be chosen. The joint probability is defined to be:

$$p\left(\mathit{X}\right)d\mathit{X}=\mathrm{Prob}(\mathit{X}\le {\mathsf{{\rm Y}}}_{\mathit{X}}\le \mathit{X}+d\mathit{X},{\mathit{Y}}_{\mathit{F}}\le {\mathsf{{\rm Y}}}_{{\mathit{Y}}_{\mathit{F}}}\le {\mathit{Y}}_{\mathit{F}}+d{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}}\le {\mathsf{{\rm Y}}}_{{\mathit{Y}}_{\mathit{T}}}\le {\mathit{Y}}_{\mathit{T}}+d{\mathit{Y}}_{\mathit{T}}),$$

(34)

where ${\mathsf{{\rm Y}}}_{\mathit{X}}$, ${\mathsf{{\rm Y}}}_{{\mathit{Y}}_{\mathit{F}}}$ and ${\mathsf{{\rm Y}}}_{{\mathit{Y}}_{\mathit{T}}}$ are the vectors of the random variables for $\mathit{X}$, ${\mathit{Y}}_{\mathit{F}}$ and ${\mathit{Y}}_{\mathit{T}}$, respectively. We also assume that each of the flow rate and potential difference constraints are applied as soft constraints, but this does not restrict this method to only use soft constraints, and strict constraints defined by expectations can still be applied. This choice of pdf gives the following relative entropy or negative Kullback–Leibler function [21], over the space of uncertainties used in this formulation:

$$\mathfrak{H}=-{\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})\ln \frac{p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})}{q(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})}d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}},$$

(35)

where $n_o=N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}+h_c$, the number of data observations, $q(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})$ is the prior pdf, and $l_i$ and $u_i$ are the lower and upper bounds of the $i\mathrm{th}$ flow rate. The relative entropy is then maximised subject to the constraints on the system. The following constraints are always required:

• Normalisation of probability:

  $$1={\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}.$$

  (36)
• Kirchhoff’s first law, for the conservation of flow rates at each internal node, here imposed in the mean:

  $$\mathbf{0}=\left({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\mathit{X}p\left(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}}\right)d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}\right).$$

  (37)
• Kirchhoff’s second law, which requires the potential difference to vanish around each enclosed loop, again imposed in the mean:

  $$\mathbf{0}=\left({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\mathit{X}p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}\right).$$

  (38)

We also allow for any of the following constraints:

• A set of specified inflow/outflow and internal flow rate constraints:

  $$\mathbf{0}={\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}.$$

  (39)
• Potential difference constraints between pairs of nodes:

  $$\mathbf{0}={\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}.$$

  (40)

After identifying the constraints, the entropy (35) is then maximised subject to Equations (36)–(38) and whichever of Equations (39) and (40) apply. Applying the calculus of variations, we form the Lagrangian:

$$\begin{array}{cc}\hfill \mathfrak{L}=& -{\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})\ln \frac{p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})}{q(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})}d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}-\widehat{\kappa }\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}-1\right)\hfill \\ & -\mathbf{\alpha }\left(\left({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\mathit{X}p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}\right)\right)\hfill \\ & -\mathbf{\beta }\left(\left({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\mathit{X}p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}\right)\right)\hfill \\ & -\mathbf{\lambda }\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}\right)\hfill \\ & -\mathbf{\eta }\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}\right),\hfill \end{array}$$

(41)

where $\widehat{\kappa }$, (scalar) $\mathbf{\alpha }$, $\mathbf{\beta }$, $\mathbf{\lambda }$ and $\mathbf{\eta }$ (row vectors) are the Lagrange multipliers for the normalisation, Kirchhoff’s first and second laws, flow rates and the head loss constraints, respectively. The variation of $\mathfrak{L}$ is given by $\delta \mathfrak{L}=0$. Extremizing Equation (41) by taking the functional derivative with respect to $p\left(\mathit{X}\right)$ and combining integrals gives:

$$\begin{array}{c}\hfill \delta \mathfrak{L}=0={\int }_{l_1}^{u_1}\cdots {\int }_{l_{n+n_o}}^{u_{n+n_o}}[-\ln \frac{p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})}{q(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})}-\kappa -\mathbf{\alpha }({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}})\mathit{X}-\mathbf{\beta }({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}})\mathit{X}\\ \hfill -\mathbf{\lambda }\left(({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}})\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)-\mathbf{\eta }\left(({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}})\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)]d\mathit{X},\end{array}$$

(42)

where $\kappa =\widehat{\kappa }+1.$ Rearrangement gives the following solution for $p\left(\mathit{X}\right)$ (the Boltzmann distribution):

$$p^{*}(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})=q(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})e^{-\kappa -\mathbf{\alpha }({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}})\mathit{X}-\mathbf{\beta }({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}})\mathit{X}-\mathbf{\lambda }\left(({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}})\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)-\mathbf{\eta }\left(({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}})\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)}.$$

(43)

This can be solved, in conjunction with the constraints (36)–(40), to give $p^{*}(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})$ and the Lagrange multipliers κ, $\mathbf{\alpha }$, $\mathbf{\beta }$, $\mathbf{\lambda }$ and $\mathbf{\eta }$.

3.2. Solution and Comparison to Bayesian Solution

In the MaxEnt method, we choose the prior pdf as the multidimensional Gaussian

$$q(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})\propto \exp \left(-\frac{1}{2}{\left(\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]-\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]\right)}^{\top }\left[\begin{array}{ccc}{\mathbf{\Sigma }}^{-1}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right]\left(\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]-\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]\right)\right),$$

(44)

where m ($n\times 1$ vector) and Σ ($n\times n$ matrix) are the mean and covariance of the prior flow rates within the entropy function, ${\mathit{m}}_{\mathit{F}}$ ($N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}\times 1$ vector) and ${\mathit{m}}_{\mathit{T}}$ ($h_c\times 1$ vector) are the values of the observations of the flow rate and potential differences, respectively, and ${\mathbf{\Sigma }}_{\mathit{F}}$ ($N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}\times N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}$ matrix) and ${\mathbf{\Sigma }}_{\mathit{F}}$ ($h_c\times h_c$ matrix) are their respective covariances. The resulting MaxEnt pdf with normalisation, Kirchhoff’s first and second law, potential difference and flow rate constraints is proportional to

$$\begin{array}{c}\hfill \ln \left(p^{*}(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})\right)\propto -\frac{1}{2}\left({\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]}^{\top }\left[\begin{array}{ccc}{\mathbf{\Sigma }}^{-1}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right]\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]-{\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]}^{\top }\left[\begin{array}{ccc}{\mathbf{\Sigma }}^{-1}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right]\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]\right.\\ \hfill \left.-{\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]}^{\top }\left[\begin{array}{ccc}{\mathbf{\Sigma }}^{-1}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right]\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]\right)-{\mathbf{\gamma }}^{\top }\left(\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]\right),\end{array}$$

(45)

where $\mathbf{\gamma }={\left[\begin{array}{cccc}\mathbf{\alpha }& \mathbf{\beta }& \mathbf{\lambda }& \mathbf{\eta }\end{array}\right]}^{\top }$. Combining terms of the same order and assuming the covariance is symmetric and positive definite

$$\begin{array}{c}\hfill \ln \left(p^{*}(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})\right)\propto -\frac{1}{2}{\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]}^{\top }\left[\begin{array}{ccc}{\mathbf{\Sigma }}^{-1}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right]\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]-\left({\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]}^{\top }\left[\begin{array}{ccc}{\mathbf{\Sigma }}^{-1}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right]\right.\\ \hfill \left.+{\mathbf{\gamma }}^{\top }\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]\right)\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]\end{array}$$

(46)

and completing the square

$$\begin{array}{c}\hfill \ln \left(p^{*}(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})\right)\propto -\frac{1}{2}{\left(\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]-\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]+\left[\begin{array}{ccc}\mathbf{\Sigma }& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}\end{array}\right]{\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]}^{\top }\mathbf{\gamma }\right)}^{\top }\\ \hfill \times {\mathbf{\Sigma }}^{-1}\left(\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]-\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]+\left[\begin{array}{ccc}\mathbf{\Sigma }& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}\end{array}\right]{\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]}^{\top }\mathbf{\gamma }\right).\end{array}$$

(47)

The above form allows the mean to be obtained as

$$\left[\begin{array}{c}\langle \mathit{X}\rangle \\ \langle {\mathit{Y}}_{\mathit{F}}\rangle \\ \langle {\mathit{Y}}_{\mathit{T}}\rangle \end{array}\right]={\int }_{-\infty }^{\infty }\cdots {\int }_{-\infty }^{\infty }\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}=\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]-\left[\begin{array}{ccc}\mathbf{\Sigma }& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}\end{array}\right]{\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]}^{\top }\mathbf{\gamma }.$$

(48)

Using the constraint equations, the Lagrange multipliers can be found from

$$\mathbf{\gamma }={\left(\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]\left[\begin{array}{ccc}\mathbf{\Sigma }& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}\end{array}\right]{\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]}^{\top }\right)}^{-1}\left(\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]\right).$$

(49)

Substituting Equation (49) into Equation (48) gives

$$\begin{array}{c}\hfill \left[\begin{array}{c}\langle \mathit{X}\rangle \\ \langle {\mathit{Y}}_{\mathit{F}}\rangle \\ \langle {\mathit{Y}}_{\mathit{T}}\rangle \end{array}\right]={\int }_{-\infty }^{\infty }\cdots {\int }_{-\infty }^{\infty }\left[\begin{array}{c}\mathit{X}\\ {\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right]p(\mathit{X},{\mathit{Y}}_{\mathit{F}},{\mathit{Y}}_{\mathit{T}})d\mathit{X}d{\mathit{Y}}_{\mathit{F}}d{\mathit{Y}}_{\mathit{T}}=\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]-\left[\begin{array}{ccc}\mathbf{\Sigma }& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}\end{array}\right]{\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]}^{\top }\\ \hfill \times {\left(\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]\left[\begin{array}{ccc}\mathbf{\Sigma }& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}\end{array}\right]{\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]}^{\top }\right)}^{-1}\left(\left[\begin{array}{ccc}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}& \mathbf{0}& \mathbf{0}\\ {\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}& -\mathit{I}& \mathbf{0}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}& \mathbf{0}& -\mathit{I}\end{array}\right]\left[\begin{array}{c}\mathit{m}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]\right).\end{array}$$

(50)

Extracting the posterior means then gives

$$\langle \mathit{X}\rangle =\mathit{m}+\mathbf{\Sigma }{\mathit{O}}^{\top }{\left(\left[\begin{array}{ccc}\mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{F}}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}\end{array}\right]+\mathit{O}\mathbf{\Sigma }{\mathit{O}}^{\top }\right)}^{-1}\left(\left[\begin{array}{c}\mathbf{0}\\ {\mathit{m}}_{\mathit{F}}\\ {\mathit{m}}_{\mathit{T}}\end{array}\right]-\mathit{O}\mathit{m}\right).$$

(51)

Applying the limit to ${\mathbf{\Sigma }}_{\mathit{C}}$ and ${\mathbf{\Sigma }}_{\mathit{W}}$ gives Equation (33). In consequence, the MaxEnt formulation with soft prior constraints Section 3.1 and the Bayesian formulation Section 2 give the same mean flow rate prediction (33), but with different covariance matrices.

4. MaxEnt Analysis with Soft Probabilistic Constraints

4.1. Formulation

The MaxEnt method can also incorporate soft constraints using a probabilistic representation of the observed data. To implement this, we define a pdf that expresses the uncertainty in the system defined over a reduced parameter set $\mathit{X}$, again consisting of a basis set of n flow rates selected from Ψ. The indices of $\mathit{X}$ in Ψ are again given by the set $\mathcal{B}$, while their complement is given by the set $\mathcal{N}$. Again, at least $N-1$ basis flow rates must be chosen but up to $N+M$ can be chosen. The joint probability is defined to be:

$$p\left(\mathit{X}\right)d\mathit{X}=\mathrm{Prob}(\mathit{X}\le {\mathsf{{\rm Y}}}_{\mathit{X}}\le \mathit{X}+d\mathit{X}),$$

(52)

where ${\mathsf{{\rm Y}}}_{\mathit{X}}$ is the vectors of the random variables for $\mathit{X}$. We again assume that each of the flow rate and potential difference constraints are applied as soft constraints, but this does not restrict this method to only use soft constraints, and strict constraints can also be applied. This choice of pdf gives the following relative entropy or negative Kullback–Leibler function [21], over the space of uncertainties used in this formulation:

$$\mathfrak{H}=-{\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}p\left(\mathit{X}\right)\ln \frac{p\left(\mathit{X}\right)}{q\left(\mathit{X}\right)}d\mathit{X},$$

(53)

where $q\left(\mathit{X}\right)$ is the prior pdf, and $l_i$ and $u_i$ are the lower and upper bounds of the $i\mathrm{th}$ flow rate. The relative entropy is then maximised subject to the constraints on the system. The following constraints are always required:

• Normalisation of probability:

  $$1={\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}p\left(\mathit{X}\right)d\mathit{X}.$$

  (54)
• Kirchhoff’s first law, for the conservation of flow rates at each internal node, here imposed in the mean:

  $$\mathbf{0}=\left({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\mathit{X}p\left(\mathit{X}\right)d\mathit{X}\right).$$

  (55)
• Kirchhoff’s second law, which requires the potential difference to vanish around each enclosed loop, again imposed in the mean:

  $$\mathbf{0}=\left({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\mathit{X}p\left(\mathit{X}\right)d\mathit{X}\right).$$

  (56)

We also allow for any of the following constraints:

• A set of specified inflow/outflow and internal flow rate constraints assuming the uncertainty is described by a Gaussian distribution:

  $$\langle {\mathit{F}}_{\mathit{p}}\rangle ={\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{F}}\right)}{{\left(2\pi \right)}^{\frac{N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}}{2}}\left|{\mathbf{\Sigma }}_{\mathit{F}}\right|}\right)p\left(\mathit{X}\right)d\mathit{X}.$$

  (57)
• Potential difference constraints between pairs of nodes again assuming the uncertainty is described by a Gaussian distribution:

  $$\langle {\mathit{T}}_{\mathit{p}}\rangle ={\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{T}}\right)}{{\left(2\pi \right)}^{\frac{h_c}{2}}\left|{\mathbf{\Sigma }}_{\mathit{T}}\right|}\right)p\left(\mathit{X}\right)d\mathit{X}.$$

  (58)

After identifying the constraints, the entropy (53) is then maximised subject to Equations (54)–(56) and whichever of Equations (57) and (58) apply. Applying the calculus of variations, we form the Lagrangian:

$$\begin{array}{cc}\hfill \mathfrak{L}=& -{\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}p\left(\mathit{X}\right)\ln \frac{p\left(\mathit{X}\right)}{q\left(\mathit{X}\right)}d\mathit{X}-\widehat{\kappa }\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}p\left(\mathit{X}\right)d\mathit{X}-1\right)\hfill \\ & -\mathbf{\alpha }\left(\left({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\mathit{X}p\left(\mathit{X}\right)d\mathit{X}\right)\right)\hfill \\ & -\mathbf{\beta }\left(\left({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\right)\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\mathit{X}p\left(\mathit{X}\right)d\mathit{X}\right)\right)\hfill \\ & -\mathbf{\lambda }\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{F}}\right)}{{\left(2\pi \right)}^{\frac{N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}}{2}}\left|{\mathbf{\Sigma }}_{\mathit{F}}\right|}\right)p\left(\mathit{X}\right)d\mathit{X}-\langle {\mathit{F}}_{\mathit{p}}\rangle \right)\hfill \\ & -\mathbf{\eta }\left({\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{T}}\right)}{{\left(2\pi \right)}^{\frac{h_c}{2}}\left|{\mathbf{\Sigma }}_{\mathit{T}}\right|}\right)p\left(\mathit{X}\right)d\mathit{X}-\langle {\mathit{T}}_{\mathit{p}}\rangle \right),\hfill \end{array}$$

(59)

where $\widehat{\kappa }$, (scalar) $\mathbf{\alpha }$, $\mathbf{\beta }$, $\mathbf{\lambda }$ and $\mathbf{\eta }$ (row vectors) are the Lagrange multipliers for the normalisation, Kirchhoff’s first and second laws, flow rates and the head loss constraints, respectively. The variation of $\mathfrak{L}$ is given by $\delta \mathfrak{L}=0$. Extremizing Equation (59) by taking the functional derivative with respect to $p\left(\mathit{X}\right)$ and combining integrals gives:

$$\begin{array}{c}\hfill \delta \mathfrak{L}=0={\int }_{l_1}^{u_1}\cdots {\int }_{l_n}^{u_n}[-\ln \frac{p\left(\mathit{X}\right)}{q\left(\mathit{X}\right)}-\kappa -\mathbf{\alpha }({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}})\mathit{X}-\mathbf{\beta }({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}})\mathit{X}\\ \hfill -\mathbf{\lambda }\left(\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{F}}\right)}{{\left(2\pi \right)}^{\frac{N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}}{2}}\left|{\mathbf{\Sigma }}_{\mathit{F}}\right|}\right)\right)\\ \hfill -\mathbf{\eta }\left(\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{T}}\right)}{{\left(2\pi \right)}^{\frac{h_c}{2}}\left|{\mathbf{\Sigma }}_{\mathit{T}}\right|}\right)\right)]d\mathit{X},\end{array}$$

(60)

where $\kappa =\widehat{\kappa }+1.$ Rearrangement gives the following solution for $p\left(\mathit{X}\right)$ (the Boltzmann distribution):

$$\begin{array}{c}\hfill p^{*}\left(\mathit{X}\right)=q\left(\mathit{X}\right)\exp (-\kappa -\mathbf{\alpha }({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}})\mathit{X}-\mathbf{\beta }({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}})\mathit{X}\\ \hfill -\mathbf{\lambda }\left(\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{F}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left(\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{F}}\right)}{{\left(2\pi \right)}^{\frac{N_{\widehat{\mathrm{\Theta }}}+N_{\widehat{Q}}}{2}}\left|{\mathbf{\Sigma }}_{\mathit{F}}\right|}\right)\right)\\ \hfill -\mathbf{\eta }\left(\ln \left(\frac{\exp \left(-\frac{1}{2}{\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{Y}}_{\mathit{T}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left(\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)-{\mathit{Y}}_{\mathit{T}}\right)}{{\left(2\pi \right)}^{\frac{h_c}{2}}\left|{\mathbf{\Sigma }}_{\mathit{T}}\right|}\right)\right)).\end{array}$$

(61)

This can be solved, in conjunction with the constraints (36)–(40), to give $p^{*}\left(\mathit{X}\right)$ and the Lagrange multipliers κ, $\mathbf{\alpha }$, $\mathbf{\beta }$, $\mathbf{\lambda }$ and $\mathbf{\eta }$. As the purpose of the soft constraints is to incorporate a distribution as a constraint, we take $\mathbf{\lambda }=-\mathbf{1}$ and $\mathbf{\eta }=-\mathbf{1}$.

4.2. Solution and Comparison to Bayesian Solution

If the prior is chosen to be proportional to Equation (6), the MaxEnt probability distribution with normalisation, Kirchhoff’s first and second law, potential difference and flow rate constraints is proportional to

$$\begin{array}{c}\hfill \ln \left(p^{*}\left(\mathit{X}\right)\right)\propto -\frac{1}{2}\left({\mathit{X}}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{X}-{\mathit{X}}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{m}-{\mathit{m}}^{\top }{\mathbf{\Sigma }}^{-1}\mathit{X}\right)-\mathbf{\alpha }({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}})\mathit{X}-\mathbf{\beta }({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}})\mathit{X}\\ \hfill +\left(-\frac{1}{2}\left({\mathit{X}}^{\top }{\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{X}}^{\top }{\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}{\mathit{Y}}_{\mathit{F}}-{{\mathit{Y}}_{\mathit{F}}}^{\top }{\mathbf{\Sigma }}_F^{-1}\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)\right)\\ \hfill +\left(-\frac{1}{2}\left({\mathit{X}}^{\top }{\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}-{\mathit{X}}^{\top }{\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}{\mathit{Y}}_{\mathit{T}}-{{\mathit{Y}}_{\mathit{T}}}^{\top }{\mathbf{\Sigma }}_T^{-1}\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\mathit{X}\right)\right).\end{array}$$

(62)

Combining terms of the same order and assuming the covariance is symmetric and positive definite

$$\begin{array}{c}\hfill \ln \left(p^{*}\left(\mathit{X}\right)\right)\propto -\frac{1}{2}{\mathit{X}}^{\top }\left({\mathbf{\Sigma }}^{-1}+{\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)+{\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\right)\mathit{X}\\ \hfill +\left({\mathit{m}}^{\top }{\mathbf{\Sigma }}^{-1}-\mathbf{\alpha }({\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}})-\mathbf{\beta }({\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}})+{{\mathit{Y}}_{\mathit{F}}}^{\top }{\mathbf{\Sigma }}_{\mathit{F}}^{-1}\left({\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\right)+{{\mathit{Y}}_{\mathit{T}}}^{\top }{\mathbf{\Sigma }}_{\mathit{T}}^{-1}\left({\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\right)\right)\mathit{X},\end{array}$$

(63)

let

$$\widehat{\mathit{O}}=\left[\begin{array}{c}{\mathit{C}}_{\mathit{X}}+{\mathit{C}}_{\overline{\mathit{X}}}\\ {\mathit{W}}_{\mathit{X}}+{\mathit{W}}_{\overline{\mathit{X}}}\end{array}\right],$$

(64)

$$\tilde{\mathit{O}}=\left[\begin{array}{c}{\mathit{F}}_{\mathit{X}}+{\mathit{F}}_{\overline{\mathit{X}}}\\ {\mathit{T}}_{\mathit{X}}+{\mathit{T}}_{\overline{\mathit{X}}}\end{array}\right],$$

(65)

$${\tilde{\mathit{S}}}^{-1}=\left[\begin{array}{cc}{\mathbf{\Sigma }}_{\mathit{F}}^{-1}& \mathbf{0}\\ \mathbf{0}& {\mathbf{\Sigma }}_{\mathit{T}}^{-1}\end{array}\right],$$

(66)

$$\tilde{\mathit{y}}=\left[\begin{array}{c}{\mathit{Y}}_{\mathit{F}}\\ {\mathit{Y}}_{\mathit{T}}\end{array}\right],$$

(67)

so

$$\begin{array}{c}\hfill \ln \left(p^{*}\left(\mathit{X}\right)\right)\propto -\frac{1}{2}{\mathit{X}}^{\top }\left({\mathbf{\Sigma }}^{-1}+{\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}\tilde{\mathit{O}}\right)\mathit{X}+\left({\mathit{m}}^{\top }{\mathbf{\Sigma }}^{-1}-\left[\begin{array}{cc}\mathbf{\alpha }& \mathbf{\beta }\end{array}\right]\widehat{\mathit{O}}+{\tilde{\mathit{y}}}^{\top }{\tilde{\mathit{S}}}^{-1}\tilde{\mathit{O}}\right)\mathit{X},\end{array}$$

(68)

and completing the square

$$\ln \left(p^{*}\left(\mathit{X}\right)\right)\propto -\frac{1}{2}{\left(\mathit{X}-\langle \mathit{X}\rangle \right)}^{\top }{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}^{-1}\left(\mathit{X}-\langle \mathit{X}\rangle \right),$$

(69)

where

$$\langle \mathit{X}\rangle ={\int }_{-\infty }^{\infty }\cdots {\int }_{-\infty }^{\infty }\mathit{X}p\left(\mathit{X}\right)d\mathit{X}={\tilde{\mathbf{\Sigma }}}_{\mathit{P}}\left({\mathbf{\Sigma }}^{-1}\mathit{m}-{\widehat{\mathit{O}}}^{\top }\left[\begin{array}{c}\mathbf{\alpha }\\ \mathbf{\beta }\end{array}\right]+{\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}\tilde{\mathit{y}}\right)$$

(70)

and

$${\tilde{\mathbf{\Sigma }}}_{\mathit{P}}=\langle \mathit{X}{\mathit{X}}^{\top }\rangle -\langle \mathit{X}\rangle {\langle \mathit{X}\rangle }^{\top }={\int }_{-\infty }^{\infty }\cdots {\int }_{-\infty }^{\infty }\mathit{X}{\mathit{X}}^{\top }p\left(\mathit{X}\right)d\mathit{X}-\langle \mathit{X}\rangle {\langle \mathit{X}\rangle }^{\top }={\left({\mathbf{\Sigma }}^{-1}+{\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}\tilde{\mathit{O}}\right)}^{-1}.$$

(71)

Using the Woodbury matrix identity [37] gives the posterior covariance matrix

$${\tilde{\mathbf{\Sigma }}}_{\mathit{P}}=\mathbf{\Sigma }-\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\tilde{\mathit{O}}\mathbf{\Sigma }.$$

(72)

The following algebra follows that of the previous derivations. Right multiplying the inverse posterior covariance by $\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }$ gives

$${\tilde{\mathbf{\Sigma }}}_{\mathit{P}}^{-1}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }={\tilde{\mathit{O}}}^{\top }+{\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }={\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right).$$

(73)

Left multiplying with the posterior covariance then gives

$$\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }={\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)$$

(74)

and obtaining ${\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}$ by right multiplying by ${\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}$ gives

$$\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}={\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\tilde{\mathit{O}}}^{\top }{\tilde{\mathit{S}}}^{-1}.$$

(75)

The posterior mean flow rates can now be expressed using Equation (70) by substituting Equations (72) and (75) to give

$$\langle \mathit{X}\rangle =\left(\mathbf{\Sigma }-\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\tilde{\mathit{O}}\mathbf{\Sigma }\right){\mathbf{\Sigma }}^{-1}\mathit{m}+\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\tilde{\mathit{y}}-{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\widehat{\mathit{O}}}^{\top }\left[\begin{array}{c}\mathbf{\alpha }\\ \mathbf{\beta }\end{array}\right],$$

(76)

$$\langle \mathit{X}\rangle =\mathit{m}+\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\left(\tilde{\mathit{y}}-\tilde{\mathit{O}}\mathit{m}\right)-{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\widehat{\mathit{O}}}^{\top }\left[\begin{array}{c}\mathbf{\alpha }\\ \mathbf{\beta }\end{array}\right].$$

(77)

Using the constraint equations, the Lagrange multipliers can be found from

$$\left[\begin{array}{c}\mathbf{\alpha }\\ \mathbf{\beta }\end{array}\right]={\left(\widehat{\mathit{O}}{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\widehat{\mathit{O}}}^{\top }\right)}^{-1}\widehat{\mathit{O}}\left(\mathit{m}+\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\left(\tilde{\mathit{y}}-\tilde{\mathit{O}}\mathit{m}\right)\right).$$

(78)

Substituting Equation (78) into Equation (77) gives the posterior means

$$\langle \mathit{X}\rangle =\mathit{m}+\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\left(\tilde{\mathit{y}}-\tilde{\mathit{O}}\mathit{m}\right)-{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\widehat{\mathit{O}}}^{\top }{\left(\widehat{\mathit{O}}{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\widehat{\mathit{O}}}^{\top }\right)}^{-1}\widehat{\mathit{O}}\left(\mathit{m}+\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\left(\tilde{\mathit{y}}-\tilde{\mathit{O}}\mathit{m}\right)\right)$$

(79)

or

$$\langle \mathit{X}\rangle =\left(\mathit{m}+\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }{\left(\tilde{\mathit{S}}+\tilde{\mathit{O}}\mathbf{\Sigma }{\tilde{\mathit{O}}}^{\top }\right)}^{-1}\left(\tilde{\mathit{y}}-\tilde{\mathit{O}}\mathit{m}\right)\right)\left(\mathit{I}-{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\widehat{\mathit{O}}}^{\top }{\left(\widehat{\mathit{O}}{\tilde{\mathbf{\Sigma }}}_{\mathit{P}}{\widehat{\mathit{O}}}^{\top }\right)}^{-1}\widehat{\mathit{O}}\right).$$

(80)

As evident, the first bracketed term is of similar structure to Equations (33) and (51), although it contains parameters related to the soft probabilistic constraints. The second term gives a second-order expansion of that solution relating to the interaction between the hard moment constraints and soft probabilistic constraints. If all constraints were applied as soft probabilistic constraints, Equation (33) would be obtained. Numerical experiments suggest that the means obtained in Equation (80) are equal to the Bayesian posterior means Equation (33), in several examples considered, but the covariances are different.

5. Discussion

The MaxEnt and Bayesian methods rest on different theoretical foundations but are both able to predict flows on networks by updating the prior belief to the posterior with the inclusion of new information in the form of constraints or uncertain data. This study compares the application of Bayesian inference and the MaxEnt method for the analysis of flow networks, for the latter using soft constraints—included in the prior or imposed as probabilistic constraints—in addition to standard moment constraints. It is shown that both the Bayesian method and MaxEnt method with soft prior constraints, implemented using a multidimensional Gaussian prior pdf, infer the same mean flow rates but different covariance matrices. In the Bayesian method, the interactions between variables are applied through the likelihood function, using second or higher-order cross-terms within the posterior pdf. In contrast, the MaxEnt method incorporates interactions between variables using Lagrange multipliers, avoiding second-order correlation terms in the posterior covariance. The MaxEnt method with soft prior constraints therefore has a numerical advantage in its integrations, in that the covariance terms are avoided.

In contrast, the second MaxEnt method with probabilistic and moment constraints is shown to give a posterior mean of similar, but not identical, structure to the other two methods. Due to the mixture of constraint types, some of the interactions between variables are incorporated in the Lagrange multipliers and some are incorporated in the covariance matrix, leading to a more complicated formulation.

For both MaxEnt formulations given herein, the equivalence between the posterior means inferred by the Bayesian and MaxEnt methods is dependent on the choice of a multidimensional Gaussian prior and its parameterisation. Further research is required to classify the effect of other prior distributions on the MaxEnt and Bayesian formulations, and whether these lead to equivalences between the means or higher-order moments of the inferred posterior pdf.