Intrinsic Computation of a Monod-Wyman-Changeux Molecule

Abstract

Causal states are minimal sufficient statistics of prediction of a stochastic process, their coding cost is called statistical complexity, and the implied causal structure yields a sense of the process’ “intrinsic computation”. We discuss how statistical complexity changes with slight changes to the underlying model– in this case, a biologically-motivated dynamical model, that of a Monod-Wyman-Changeux molecule. Perturbations to kinetic rates cause statistical complexity to jump from finite to infinite. The same is not true for excess entropy, the mutual information between past and future, or for the molecule’s transfer function. We discuss the implications of this for the relationship between intrinsic and functional computation of biological sensory systems.

1. Introduction

Intrinsic computation [1] is a theory of how a dynamical system “intrinsically” computes. In short, one makes a minimal maximally predictive model (or $ϵ$-machine) of the process generated by a dynamical system. States of the $ϵ$-machine are called “causal states”, although these states are normally not causal in the sense of Ref. [2]. Certain words are forbidden, in that those words can never be seen. The words that are seen and thus accepted by the $ϵ$-machine constitute the $ϵ$-machine’s language, in a nod to the computation performable by finite and infinite automata. The “memory stored by the process”, the statistical complexity, is taken to mean the coding cost of the $ϵ$-machine’s states.

One interesting hypothesis is that the $ϵ$-machine’s structure provides a guide to the “functional” computation of the corresponding dynamical system. Functional computation–biologically-relevant computation, e.g., transformation of information with fitness consequences–might include everything from estimating past input [3,4] to predicting future input [5] to performing logical computations on input [6]. A more rigorous definition of “functional computation” remains an open problem; here, we merely list examples of quantities that can be identified with functional computations. As of yet, no link between intrinsic and functional computation has been found.

Here, we investigate the intrinsic computation and two functional computations (ligand concentration transduction and low-pass filtering) of a Monod-Wyman-Changeux (MWC) molecule, a widely-used model of a biological sensor [7,8,9]. This is the first time that the intrinsic computation of an MWC molecule–which here is limited to the $ϵ$-machine structure, the statistical complexity, and the excess entropy–has been calculated. The calculational techniques used here can be applied to study intrinsic computation of a more general class of biological sensors than previously studied.

We find that certain arbitrarily small perturbations to the underlying MWC molecule can lead to arbitrarily large perturbations in the process’ intrinsic structure but lead to arbitrarily small changes in the stated functional computations. To the author’s knowledge, this is the first such example in the literature. These results therefore suggest that causal structure and functional computation are orthogonal characterizations of a process, at least for oft-considered functional computations.

However, intrinsic computation could be taken to include several newer structure-related information-theoretic measures of a process, including excess entropy [10,11,12]. These newer measures do not suffer the same sensitivity as the $ϵ$-machine and statistical complexity, suggesting that these measures might help characterize functional computation.

Section 2 reviews the definition of $ϵ$-machine and MWC molecules. Section 3 explores how variations in kinetic rates change the molecule’s intrinsic and functional computation. Section 4 discusses future research directions for intrinsic computation.

2. Background

The subject of interest here is a continuous-time, discrete-event process. However, for reasons explained later, for characterization of statistical complexity we will consider time-binning the process and treating it as a discrete-time, discrete-event process. Hence, we are also interested in discrete-time, discrete-event processes.

First, we discuss discrete-time, discrete-event processes. We code these processes as $\dots ,x_{-1},x_0,x_1,\dots$ where $x_i$ is the i-th symbol appearing. The past $\overleftarrow{x}$ (with corresponding random variable $\overleftarrow{X}$) is taken to be $\dots ,x_{-2},x_{-1}$ while the future $\overrightarrow{x}$ (with corresponding random variable $\overrightarrow{X}$) is taken to be $x_0,x_1,x_2,\dots$.

Next, we discuss continuous-time, discrete-event processes. We code these processes as $\dots ,(x_{-1},{\tau }_{-1}),(x_0,{\tau }_0),(x_1,{\tau }_1),\dots$ where $x_i$ is the i-th symbol, appearing for a total duration ${\tau }_i$. We enforce $x_i\ne x_{i+1}$ so as to ensure a unique coding. The present is said to occur sometime during the presentation of $x_0$, and so we denote the past $\overleftarrow{(x,\tau )}$ (with corresponding random variable $\overleftarrow{(X,\mathcal{T})}$) as $\dots ,(x_{-1},{\tau }_{-1}),(x_0,{\tau }_{+})$ and the future $\overrightarrow{(x,\tau )}$ (with corresponding random variable $\overrightarrow{(X,\mathcal{T})}$) as $(x_0,{\tau }_{-}),(x_1,{\tau }_1),\dots$ where ${\tau }_{+}+{\tau }_{-}={\tau }_0$.

Section 2.1 reviews the definition of causal states, statistical complexity, the continuous-time $ϵ$-machine, and the mixed-state simplex. Section 2.2 reviews the dynamical models of Monod-Wyman-Changeux molecules used here.

We assume knowledge of information theory at the level of Ref. [13], but we briefly review definitions here. When X is a discrete random variable with probability distribution $p\left(x\right)$, then its entropy is $H\left[X\right]=-{\sum }_{x}p\left(x\right)logp\left(x\right)$; when X is a continuous random variable with probability density function $\rho \left(x\right)$, then the differential entropy is $H\left[X\right]=-\int \rho \left(x\right)log\rho \left(x\right)dx$; and when X is a mixed random variable (as is the case here), the entropy $H\left[X\right]$ is given by Ref. [14]. Entropy can be thought of as a measure of uncertainty. Conditional entropy of X conditioned on random variable Y is $H\left[X\right|Y]={\langle H\left[X\right|Y=y]\rangle }_y$, and the mutual or shared information $I[X;Y]$ between two random variables X and Y is merely $I[X;Y]=H\left[X\right]-H\left[X\right|Y]$.

2.1. Causal States $\mathcal{S}$, Statistical Complexity $C_{\mu }$, the $ϵ$-Machine, and the Mixed-State Simplex

Consider the equivalence relation ${\sim }_{ϵ}$ that clusters two semi-infinite pasts, $\overleftarrow{x}$ and ${\overleftarrow{x}}^{\prime }$, together if $Pr(\overrightarrow{X}|\overleftarrow{X}=\overleftarrow{x})=Pr(\overrightarrow{X}|\overleftarrow{X}={\overleftarrow{x}}^{\prime })$–that is, if the two pasts are equivalent from the standpoint of prediction. The corresponding clusters are causal states $\sigma$, which are realizations of the random variable for the causal states $\mathcal{S}$. The statistical complexity $C_{\mu }$ is simply their coding cost, $C_{\mu }=H\left[\mathcal{S}\right]$. In short, causal states $\mathcal{S}$ are minimal sufficient statistics of prediction; the statistical complexity $C_{\mu }=H\left[\mathcal{S}\right]$ is the coding cost of those causal states [15]; and the $ϵ$-machine is the minimal maximally predictive model constructed from those causal states [16].

The same constructions apply when considering continuous-time, discrete-event processes. In that case, the equivalence relation ${\sim }_{ϵ}$ clusters two semi-infinite pasts $\overleftarrow{(x,\tau )},{\overleftarrow{(x,\tau )}}^{\prime }$ together if $Pr\left(\overrightarrow{(X,\mathcal{T})}\right|\overleftarrow{(X,\mathcal{T})}=\overleftarrow{(x,\tau )})=Pr\left(\overrightarrow{(X,\mathcal{T})}\right|\overleftarrow{(X,\mathcal{T})}={\overleftarrow{(x,\tau )}}^{\prime })$. Again, the clusters are causal states $\sigma$, realizations of the random variable for causal states $\mathcal{S}$.

For what follows, we must define two terms: mixed state simplex, and unifilar. Consider any hidden Markov model whose hidden state at time t is a random variable; now consider the probability distribution over hidden states given observations $\overleftarrow{x}$. Each one of these conditional probability distributions is a mixed state, and it lies in the mixed state simplex, the set of all possible probability distributions over hidden states. The hidden Markov model is unifilar when, given one’s hidden state at time t and an observation at time t, one knows exactly which hidden state comes next at $t+1$. There is a connection between unifilarity and the mixed state simplex: when the hidden Markov model under study is unifilar, then the mixed states will lie at the edge of the simplex. (This is not true for nonunifilar hidden Markov models.) The causal states are just the mixed states of the minimal (potentially nonunifilar) generative model.

The causal states of discrete-time processes are usually uncountably infinite. When this is the case, then the box-counting dimension of the mixed state presentation in the mixed state simplex is nonzero. Let’s unpack this statement. Suppose that a (potentially nonunifilar) Hidden Markov model with states g generates the observed discrete-time process. Then we use $p(g|\overleftarrow{x})$ to denote the probability over hidden states in the generative model given past output. Typically, $p(g|\overleftarrow{x})$ is in the interior of the mixed state simplex–the space of probability distributions over hidden states. The box-counting dimension of the mixed state presentation is obtained by gridding the mixed state simplex by cubes of side length $ϵ$, counting the number of non-empty cubes $N_{ϵ}$ (which contain coarse-grainings of histories) [17], and then calculating the scaling of $N_{ϵ}$ with $ϵ$–so the box-counting dimension is ${lim}_{ϵ\to 0}\frac{log{N}_{ϵ}}{log\frac{1}{ϵ}}$. A cube is considered non-empty when there is at least one history that leads to a mixed state in the cube. When there are countable causal states for a discrete-time process, the box-counting dimension is 0.

The causal states inherit a dynamic, and the $ϵ$-machine of a process is the pairing of causal states together with that dynamic. For discrete-time, discrete-event processes, tractable $ϵ$-machines are merely countable unifilar Hidden Markov models [16], where unifilarity implies that the next hidden state is determined uniquely by the previous hidden state and the present emitted symbol. For continuous-time, discrete-event processes, tractable $ϵ$-machines can (for instance) take the form of joined conveyer belts [18]. Continuous-time causal states are then usually accompanied by labeled transition operators ${\mathcal{O}}^{\left(x\right)}$, and the list of labeled transition operators specifies the continuous-time $ϵ$-machine. A “tractable $ϵ$-machine” is one for which $C_{\mu }$ is finite, with the exception of the $ϵ$-machines of continuous-time periodic processes (which are tractable but which correspond to infinite $C_{\mu }$).

2.2. Monod-Wyman-Changeux Molecules

A Monod-Wyman-Changeux (MWC) molecule has two configurations, active (A) and inactive (I), and n binding sites. Each configuration can bind any number of molecules, from 0 to n. This gives a total of $2^{n+1}$ possible states. If binding sites are indistinguishable, then a simplified model can be made based on the symmetry in binding sites so that there are only $2(n+1)$ distinguishable possible states: it can be either active or inactive, with any number of binding sites occupied by ligand molecules. As our argument holds for any n, we focus on the case that $n=1$. The four states of the corresponding MWC molecule–$\{A_0,A_1,I_0,I_1\}$, standing for active/inactive ($A/I$) with either 0 or 1 ligands bound as written in the subscript–are shown in Figure 1, along with allowed transitions.

We denote the probability distribution of being in various states as

$$\overrightarrow{p}=\left(\begin{array}{c}p\left(A_0\right)\\ p\left(A_1\right)\\ p\left(I_0\right)\\ p\left(I_1\right)\end{array}\right).$$

(1)

This probability distribution evolves via the master equation

$$\frac{d\overrightarrow{p}}{dt}=M\left(c\left(t\right)\right)\overrightarrow{p}$$

(2)

where

$$M\left(c\right)=\left(\begin{array}{cccc}-(f_T+f_{A}c)& b_A& b_T& 0\\ f_{A}c& -(f_T^{\prime }+b_A)& 0& b_T^{\prime }\\ f_T& 0& -(b_T+f_{I}c)& b_I\\ 0& f_T^{\prime }& f_{I}c& -(b_T^{\prime }+b_I)\end{array}\right)$$

(3)

with $f_T,f_A,b_A,b_T,f_T^{\prime },b_T^{\prime }$ are kinetic parameters. In fixed ligand concentration c, Equation (2) is solved as

$$\overrightarrow{p}\left(t\right)=e^{M\left(c\right)t}\overrightarrow{p}\left(0\right)$$

(4)

where $\overrightarrow{p}\left(0\right)$ is the initial probability distribution over the MWC molecule’s states.

3. Results

We suppose that we are only allowed to see whether or not the MWC molecule is active or inactive, as would be true for most experimental observations of ligand-gated ion channels. This is a key constraint, as otherwise, the minimal generative model would be the minimal maximally predictive model and none of the discrepancies described here would arise. In what follows, we explore the effects of kinetic rates on intrinsic and functional computation.

Intrinsic computation as it was originally defined included the $ϵ$-machine and statistical complexity, and today includes other information measures, such as the excess entropy [10,11,12]. The number of functional computation-related quantities is unbounded, but we focus on two here due to their presence in the literature: the binding curve, or the probability of the MWC molecule being active as a function of ligand concentration; and the transfer function, or how the MWC molecule responds to sinusoidal perturbations of the ligand concentration.

Our argument will essentially be a proof by contradiction. We will start by assuming that there is some relationship between at least one aspect of intrinsic computation and at least one aspect of functional computation. If there were a relationship between these two quantities, then we should not be able to change kinetic rates so that one quantity changes by an arbitrarily small amount and the other by an arbitrarily large amount. (If so, these kinetic rates would then be of incredible importance to the process’ causal architecture, say, but of vanishingly small importance to the so-called functional computations.) We will then show in the following analysis that arbitrarily small perturbations in the kinetic rates $f_T^{\prime },b_T^{\prime }$ induce arbitrarily large perturbations to the $ϵ$-machine and statistical complexity, but induce arbitrarily small perturbations to excess entropy and the functional computations considered here. We therefore conclude that if there is a relationship between intrinsic computation and functional computation for these kinds of molecules, it will more likely come from excess entropy (or other more recently-studied information measures of time series [19]) than from statistical complexity or the $ϵ$-machine. We discuss the possibilities of finding functional computations that are sensitive to arbitrarily small increases in $f_T^{\prime },b_T^{\prime }$ in Section 4.

3.1. Intrinsic Computation

If $b_T^{\prime },f_T^{\prime }>0$, then there is no “sync word”–that is, no string of observed past symbols that uniquely determines the underlying present state of the MWC molecule. (Note again that this would not be the case if we were allowed to observe the full state, and not just whether or not the MWC molecule is active or inactive.) This has important consequences for $C_{\mu }$ and for the process’ $ϵ$-machine.

To analyze $C_{\mu }$, we move to the discrete-time domain, so as to avoid the interpretational difficulties with differential entropy [18]. By observing the process every $\Delta t$ for $\Delta t$ much smaller than any inherent time constant in the problem, the process is turned into a discrete-time process. The transition probabilities of this new process have corresponding labeled transition matrices $T^{\left(x\right)}=e^{M^{\left(x\right)}\Delta t}$, which are approximated to lowest order in $\Delta t$ by $T^{\left(x\right)}=I^{\left(x\right)}+M^{\left(x\right)}\Delta t$, where x is an emitted symbol:

$$\begin{array}{ccc}\hfill M^{\left(A\right)}\left(c\right)& =& \left(\begin{array}{cccc}-(f_T+f_{A}c)& b_A& 0& 0\\ f_{A}c& -(b_A+f_T^{\prime })& 0& 0\\ f_T& 0& 0& 0\\ 0& f_T^{\prime }& 0& 0\end{array}\right)\mathrm{and}\hfill \\ \hfill M^{\left(I\right)}\left(c\right)& =& \left(\begin{array}{cccc}0& 0& b_T& 0\\ 0& 0& 0& b_T^{\prime }\\ 0& 0& -(b_T+f_{I}c)& b_I\\ 0& 0& f_{I}c& -(b_I+b_T^{\prime })\end{array}\right)\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill I^{\left(A\right)}& =& \left(\begin{array}{cc}{I}_{2\times 2}& 0_{2\times 2}\\ 0_{2\times 2}& 0_{2\times 2}\end{array}\right)\hfill \\ \hfill I^{\left(I\right)}& =& \left(\begin{array}{cc}{0}_{2\times 2}& 0_{2\times 2}\\ 0_{2\times 2}& I_{2\times 2}\end{array}\right).\hfill \end{array}$$

The causal states correspond to the mixed states $Pr(\mathcal{S}|\overleftarrow{X}=\overleftarrow{x})$ defined by either $(a,1-a,0,0)$ or $(0,0,1-b,b)$. When $b_T^{\prime },f_T^{\prime }>0$, all a’s and b’s are allowed; there are an uncountable infinity of these causal states because there is no sync word. As such, the $ϵ$-machine is uncountable and intractable, and statistical complexity is very likely infinite. However, when $b_T^{\prime }=f_T^{\prime }=0$, then only a countable set of a’s and b’s are allowed according to the discrete-time analogue of Theorem 1 of Ref. [18]. In particular, the causal states are identified as both the present configuration (active or inactive) and the number of time steps since last configuration switch. As a result, the $ϵ$-machine is countably infinite and tractable, and statistical complexity is finite, though it increases with $log\frac{1}{\Delta t}$ [20].

This can be seen more directly by considering the mixed-state presentation’s box-counting dimension $h_0$ when the process is turned into a discrete-time process with small time resolution $\Delta t=0.01$. We coarse-grain mixed-state simplex into cubes of side-length $ϵ$, and count the number of non-empty boxes $N_{ϵ}$, as described in Ref. [17,21]. The scaling of $N_{ϵ}$ with $ϵ$ reveals the box-counting dimension $h_0$ of the mixed-state presentation, via $N_{ϵ}\sim {(1/ϵ)}^{h_0}$. Figure 2 shows the scaling of $N_{ϵ}$ with $ϵ$ for an MWC molecule with and without $f_T^{\prime }=b_T^{\prime }=0$. When $f_T^{\prime }=b_T^{\prime }=0$, $h_0=0$; when $f_T^{\prime },b_T^{\prime }>0$, the box-counting dimension $h_0>0$.

The reason for the former fact lies in Theorem 1 of Ref. [18]. When $f_T^{\prime }=b_T^{\prime }=0$, the dynamic MWC molecule of Figure 1 generates a semi-Markov process, a restricted version of the unifilar hidden semi-Markov processes analyzed in Ref. [18]. This is true even when there is more than one ligand binding site. Causal states are characterized by x, whether or not the MWC molecule is presently active, and ${\tau }_{+}$, the time since the MWC molecule last switched between activities. The now-tractable $ϵ$-machine takes the form shown in Figure 3.

In the continuous-time limit, which one can derive by considering the limit of the discrete-time process considered above with appropriate renormalization (e.g., compare Ref. [20] to Ref. [22]), all probability distributions over mixed states become probability density functions. Continuous-time statistical complexity can be defined using the entropy of mixed random variables [18,22], though differential entropy does have some troubling properties mentioned in those references. Given the analysis above, there is likely a singular limit in the continuous-time statistical complexity as the kinetic rates $b_T^{\prime },f_T^{\prime }$ tend to 0.

Not all structure-based characterizations of a process lack robustness in this way, as different structure-based metrics pick up on different kinds of structure. To show this, we now compare the statistical complexity $C_{\mu }$ and the excess entropy $\mathbf{E}=I[\overleftarrow{(X,\mathcal{T})};\overrightarrow{(X,\mathcal{T})}]$ [10,11,12] when $f_T^{\prime }=b_T^{\prime }=0$. (We calculate $\mathbf{E}$ of the continuous-time process, as the excess entropy of the discrete-time process converges to that of the continuous-time process in the $\Delta t\to 0$ limit [20].) The latter can be calculated via $\mathbf{E}=I[{\mathcal{S}}^{+};{\mathcal{S}}^{-}]$ [23,24], while $C_{\mu }=H\left[{\mathcal{S}}^{+}\right]$, and so calculation of both merely requires the joint distribution $p({\sigma }^{+},{\sigma }^{-})$. For that, we need ${\varphi }_{A/I}\left(t\right)$, the dwell time distributions of activity and inactivity. Note that emission of an A implies that one has just landed in $A_0$, and similarly, emission of an I implies that one has just landed in $I_0$. Hence, ${\varphi }_A\left(t\right)$ is the first-passage time distribution to state $I_0$ in which one starts in $A_0$; similarly, ${\varphi }_I\left(t\right)$ is the first-passage time distribution to state $A_0$ in which one starts in $I_0$. To aid with the calculation, we recall the labeled transition matrices of Equation (5) when $f_T^{\prime }=b_T^{\prime }=0$. The matrix $M^{\left(A\right)}$ includes only the transitions between various active conformations and the only transition from active to inactive, $A_0\to I_0$. Therefore, the probability of not having stayed in active states given that one started in $A_0$ after a time t is given by

$$1-{\mathsf{\Phi }}_A\left(t\right)={({\widehat{e}}_3+{\widehat{e}}_4)}^{\top }\overrightarrow{p}\left(t\right),\frac{d\overrightarrow{p}}{dt}=M^{\left(A\right)}\left(c\right)\overrightarrow{p}\left(t\right),\overrightarrow{p}\left(0\right)={\widehat{e}}_1,$$

(5)

where ${\widehat{e}}_k$ is the vector with elements ${\delta }_{i,k}$. Hence, the survival function ${\mathsf{\Phi }}_A\left(\tau \right)={\int }_{\tau }^{\infty }{\varphi }_A\left({\tau }^{\prime }\right)d{\tau }^{\prime }$, the probability that one stays in the active conformation (one of $A_0,A_1$) after time t given that one started in $A_0$, can be calculated via

$${\mathsf{\Phi }}_A\left(t\right)=1-{({\widehat{e}}_3+{\widehat{e}}_4)}^{\top }{e}^{M^{\left(A\right)}\left(c\right)t}{\widehat{e}}_1.$$

(6)

Similarly,

$${\mathsf{\Phi }}_I\left(t\right)=1-{({\widehat{e}}_1+{\widehat{e}}_2)}^{\top }{e}^{M^{\left(I\right)}\left(c\right)t}{\widehat{e}}_3.$$

(7)

After differentiation, we find that

$$\begin{array}{ccc}{\varphi }_A\left(t\right)\hfill & =\hfill & -\frac{d{\mathsf{\Phi }}_A\left(t\right)}{dt}\hfill \\ & =\hfill & {({\widehat{e}}_3+{\widehat{e}}_4)}^{\top }{M}^{\left(A\right)}\left(c\right)e^{M^{\left(A\right)}\left(c\right)t}{\widehat{e}}_1\hfill \end{array}$$

(8)

$$\begin{array}{ccc}{\varphi }_I\left(t\right)\hfill & =\hfill & -\frac{d{\mathsf{\Phi }}_I\left(t\right)}{dt}\hfill \\ & =\hfill & {({\widehat{e}}_1+{\widehat{e}}_2)}^{\top }{M}^{\left(I\right)}\left(c\right)e^{M^{\left(I\right)}\left(c\right)t}{\widehat{e}}_3.\hfill \end{array}$$

(9)

Examples of ${\varphi }_A\left(t\right)$ for various ligand concentrations c and kinetic rates are shown in Figure 4.

From Lemma 1 of Ref. [18], we find that the statistical complexity of this semi-Markov process is given by

$$C_{\mu }=H_b\left(p\left(A\right)\right)-\sum _{x\in \{A,I\}}p\left(x\right){\int }_0^{\infty }\left({\mu }_x{\mathsf{\Phi }}_x\left(\tau \right)\right)log\left({\mu }_x{\mathsf{\Phi }}_x\left(\tau \right)\right)d\tau ,$$

(10)

where $p\left(A\right)=\frac{{\mu }_I}{{\mu }_A+{\mu }_I}$, ${\mu }_x=1/{\int }_0^{\infty }{\mathsf{\Phi }}_x\left(\tau \right)d\tau$, and $H_b\left(x\right):=-xlogx-(1-x)log(1-x)$. Figure 5 shows how $C_{\mu }$ smoothly varies with changes in $f_T,b_T$ for $f_T,b_T>0$. Statistical complexity is maximized at small kinetic rates, $f_T,b_T\to 0$; when those kinetic rates are small, dwell time distributions have longer tails, and the memory required to losslessly predict increases. Interestingly, if either $f_T$ or $b_T$ is exactly 0, then the generated process emits only A or I (see Figure 1) and thus has $C_{\mu }=0$. In other words, the limits $f_T,b_T\to 0$ are singular, as were the limits $f_T^{\prime },b_T^{\prime }\to 0$.

We now wish to calculate $\mathbf{E}=I[\overleftarrow{(X,\mathcal{T})};\overrightarrow{(X,\mathcal{T})}]$ which is $\mathbf{E}=I[{\mathcal{S}}^{+};{\mathcal{S}}^{-}]$ [23], and so

$$\mathbf{E}=H\left[{\mathcal{S}}^{-}\right]-H\left[{\mathcal{S}}^{-}\right|{\mathcal{S}}^{+}].$$

(11)

As a semi-Markov process is causally reversible, we have

$$H\left[{\mathcal{S}}^{-}\right]=H\left[{\mathcal{S}}^{+}\right]=C_{\mu }$$

(12)

as given in Equation (10). Furthermore, the reverse-time causal states are the pair $(x_{-},{\tau }_{-})$ (the time to next symbol and present symbol) while the forward-time causal states are still the pair $(x_{+},{\tau }_{+})$ (the time since last symbol and present symbol) [18], so that $x_{+}=x_{-}$ almost surely, implying that $H\left[X_{-}\right|X_{+}]=0$. Hence,

$$\begin{array}{ccc}H\left[{\mathcal{S}}^{-}\right|{\mathcal{S}}^{+}]\hfill & =\hfill & H[X_{-},{\mathcal{T}}_{-}|X_{+},{\mathcal{T}}_{+}]\hfill \\ & =\hfill & H\left[X_{-}\right|X_{+},{\mathcal{T}}_{+}]+H\left[{\mathcal{T}}_{-}\right|X_{-},X_{+},{\mathcal{T}}_{+}]\hfill \\ & =\hfill & H\left[{\mathcal{T}}_{-}\right|X_0,{\mathcal{T}}_{+}]\hfill \end{array}$$

(13)

where $x_0$ is just the present symbol. We then note that

$$p\left({\tau }_{-}\right|x_0,{\tau }_{+})=\frac{{\varphi }_{x_0}({\tau }_{+}+{\tau }_{-})}{{\mathsf{\Phi }}_{x_0}\left({\tau }_{+}\right)},$$

(14)

as was derived in Ref. [22] for a continuous-time renewal process, but the same derivation holds for the semi-Markov process. It is then straightforward to show that excess entropy is

$$\mathbf{E}=H\left[X\right]+\sum _{x}p\left(x\right)\mathbf{E}\left[{\varphi }_x\left(t\right)\right]$$

(15)

where

$$\mathbf{E}\left[{\varphi }_x\left(t\right)\right]={\int }_0^{\infty }{\int }_0^{\infty }{\mu }_x{\varphi }_x(t+t^{\prime })log\frac{{\varphi }_x(t+t^{\prime })}{{\mu }_x{\mathsf{\Phi }}_x\left(t\right){\mathsf{\Phi }}_x\left(t^{\prime }\right)}dtd{t}^{\prime }.$$

(16)

Figure 5b shows how excess entropy $\mathbf{E}$ varies with $f_T,b_T$. Interestingly, $\mathbf{E}$ varies in opposition to $C_{\mu }$, attaining its lowest values at low values of $f_T$ and $b_T$. Hence, the singular limits $f_T,b_T\to 0$ that plague $C_{\mu }$ are not singular limits for $\mathbf{E}$. Nor are $f_T^{\prime },b_T^{\prime }\to 0$ singular limits for $\mathbf{E}$, as arbitrarily small values of $f_T^{\prime },b_T^{\prime }$ lead to arbitrarily small perturbations to the trajectory distribution, and thus arbitrarily small perturbations to the mutual information between past and future.

How different would the results be for $n>1$, i.e., when the number of binding sites of the MWC molecule exceeded 1? In short, we would expect the same qualitative trends and singular limits. In this more general case, we allow an active MWC molecule with k ligands bound to transition to an inactive MWC molecule with k ligands bound, and vice versa, both with rates $f_T^{\prime }$ and $b_T^{\prime }$, as for $n=1$. Meanwhile, also as for $n=1$, the active MWC molecule with no ligands bound can transition to the inactive MWC molecule with no ligands bound with rate $f_T$, and the reverse transition can occur with rate $b_T$. The observed process for fixed ligand concentration would still be semi-Markov when $f_T^{\prime }=b_T^{\prime }=0$, as was true for $n=1$. Then, decreases in $f_T,b_T$ would lead to longer dwell times in active and inactive states, thereby increasing the statistical complexity $C_{\mu }$; and the dwell time distributions would become closer to exponential, decreasing the excess entropy $\mathbf{E}$. When $f_T^{\prime },b_T^{\prime }$ become small but nonzero, all pasts are causal states, and so $C_{\mu }$ shoots to infinity, while $\mathbf{E}$ (because it is a function of trajectory distributions) barely changes.

There are some well-known examples of how arbitrarily large $ϵ$-machines can still have arbitrarily small excess entropies, e.g., the almost fair coin. Indeed, Ref. [23] defined crypticity as the difference between statistical complexity $C_{\mu }$ and excess entropy $\mathbf{E}$. The dynamical MWC molecule described above adds another such example to the literature, finding not only that a familiar process can have arbitrarily large crypticity, but that $C_{\mu }$ and $\mathbf{E}$ can be anti-correlated with respect to underlying kinetic rates, as is true for the process generated by the parametrized Simple Nonunifilar Source [25]. There are also examples in the literature of processes with uncountable $ϵ$-machines and nonzero box-counting dimensions of their mixed-state presentation, e.g., the Cantor process in Ref. [26].

However, the dynamical MWC molecule is more than just an example of a process with potentially arbitrarily large crypticity or an uncountable $ϵ$-machine; it is also an example of how arbitrarily small changes to a generative model can lead to arbitrarily large changes in the causal structure of a process. Of course, it may be obvious to those familiar with intrinsic computation that sometimes, arbitrarily small perturbations in transition probabilities of a generative model can lead to arbitrarily large perturbations in $ϵ$-machine structure. However, to the author’s knowledge, the above MWC molecule example is the first such example in the literature.

3.2. Functional Computation

Monod-Wyman-Changeux (MWC) molecules have been used to model everything from ligand-gated ion channels to gene regulation [8]. The functional computations that an MWC molecule is thought to perform include transduction of ligand concentration and low-pass filtering of input [7].

Let ${\mathrm{eig}}_0\left(M\left(c\right)\right)$ be the normalized eigenvector of eigenvalue 0 of matrix $M\left(c\right)$, normalized so that $1^{\top }{\mathrm{eig}}_0\left(M\left(c\right)\right)=1$; and let $p_{eq,A}\left(c\right)$ be the equilibrium probability of being in state A. The MWC molecule’s ability to convey the ligand concentration via its activity is a static property, relying only on how the equilibrium distribution

$$\begin{array}{ccc}{p}_{eq,A}\left(c\right)\hfill & =\hfill & p_{eq,A_0}\left(c\right)+p_{eq,A_1}\left(c\right)\hfill \\ & =\hfill & {({\widehat{e}}_1+{\widehat{e}}_2)}^{\top }{\mathrm{eig}}_0\left(M\left(c\right)\right)\hfill \end{array}$$

(17)

varies with kinetic rates. An observer that can only see whether or not the MWC molecule is active can discern, to some extent, the external ligand concentration c. Such a situation might occur, for instance, for the nicotinic acetylcholine receptors at the neuromuscular junction that transduce information about whether or not a muscle fiber should seize, based on acetylcholine concentration. Though ${\mathrm{eig}}_0\left(M\left(c\right)\right)$ in principle might be a non-smoothly varying function of $f_T^{\prime },b_T^{\prime }$, a Mathematica calculation finds that ${\mathrm{eig}}_0\left(M\left(c\right)\right)$ and thus $p_{eq,A}\left(c\right)$ varies smoothly with kinetic rates $f_T^{\prime },b_T^{\prime }$:

$${\mathrm{eig}}_0\left(M\left(c\right)\right)\propto \left(\begin{array}{c}{b}_A{b}_I{b}_T+b_A{b}_T{b}_T^{\prime }+b_A{b}_T^{\prime }{f}_{I}c+b_I{b}_T{f}_T^{\prime }\\ (b_I{b}_T{f}_A+b_T{b}_T^{\prime }{f}_A+b_T^{\prime }{f}_A{f}_{I}c+b_T^{\prime }{f}_T{f}_I)c\\ b_A{b}_I{f}_T+b_A{f}_T{b}_T^{\prime }+f_T^{\prime }{b}_I{f}_{A}c+b_I{f}_T{f}_T^{\prime }\\ (b_A{f}_T{f}_I+b_T{f}_A{f}_T^{\prime }+f_T^{\prime }{f}_I{f}_{A}c+f_T^{\prime }{f}_T{f}_I)c\end{array}\right),$$

where the normalization constant is chosen so that $1^{\top }{\mathrm{eig}}_0\left(M\left(c\right)\right)=1$. The smooth variation of $p_{eq,A}\left(c\right)$ with respect to $f_T^{\prime },b_T^{\prime }$ is depicted for a random choice of kinetic rates in Figure 6.

The MWC molecule is a low-pass filter of ligand concentration c. Suppose that $c\left(t\right)=c_0+\delta csin\omega t$, where $\delta c$ is small. Then $p_{eq,A}\left(t\right)$ will also take the form $p_{eq,A}=p_{eq,A}\left(c_0\right)+G\left(\omega \right)\delta csin\omega t+O\left(\delta c^2\right)$, where $G\left(\omega \right)$ is the transfer function. This transfer function therefore characterizes the dynamical response of the MWC molecule to fluctuations in the ligand concentration. From Equation 49 of Ref. [8], we find that the transfer function $G\left(\omega \right)$ is

$$G\left(\omega \right)={({\widehat{e}}_1+{\widehat{e}}_2)}^{\top }{(i\omega I-M_0)}^{-1}{M}_1{\mathrm{eig}}_0\left(M\left(c_0\right)\right)$$

(18)

where

$$\begin{array}{ccc}\hfill M_0& =& \left(\begin{array}{cccc}-f_T& b_A& b_T& 0\\ 0& -(f_T^{\prime }+b_A)& 0& b_T^{\prime }\\ f_T& 0& -b_T& b_I\\ 0& f_T^{\prime }& 0& -(b_T^{\prime }+b_I)\end{array}\right)\mathrm{and}\hfill \\ \hfill M_1& =& \left(\begin{array}{cccc}-f_A& 0& 0& 0\\ f_A& 0& 0& 0\\ 0& 0& -f_I& 0\\ 0& 0& f_I& 0\end{array}\right).\hfill \end{array}$$

A series expansion not shown here confirms that $G\left(\omega \right)$ varies smoothly with kinetic rates $f_T^{\prime },b_T^{\prime }$, as would be expected from the realization that all expressions in Equation (18) are smoothly varying with $f_T^{\prime },b_T^{\prime }$. To illustrate this, the magnitude of the transfer function, $\left|G\right(\omega \left)\right|$, is plotted in Figure 7 for a randomly chosen initial concentration of $c_0=1$.

Again, it is worth commenting on how these results would vary with larger n, i.e., a larger number of potential ligands bound to the MWC molecule. We consider the dynamical model for this more complex MWC molecule as specified in Section 3.1. Just as for the case when $n=1$, the eigenvector of eigenvalue 0 for this larger MWC molecule’s rate matrix is a continuous function of kinetic rates $f_T,b_T$ and $f_T^{\prime },b_T^{\prime }$; as a result, both the binding curve and the transfer function vary smoothly with these rates.

4. Discussion

Our overarching aim here was to study the link between intrinsic computation and functional computation by focusing on a popular model of a biological sensor–the Monod-Wyman-Changeux (MWC) molecule. While studying its intrinsic computation, we found interesting singular limits for $C_{\mu }$. In particular, we found that statistical complexity was infinite and that all pasts were causal states when two of the kinetic rates were nonzero, $f_T^{\prime },b_T^{\prime }\ne 0$, no matter how small $f_T^{\prime },b_T^{\prime }$; and we found that statistical complexity was zero when $f_T=b_T=0$ but nonzero and arbitrarily large for arbitrarily small $f_T,b_T>0$. While studying the MWC molecule’s functional computation and its process’ excess entropy $\mathbf{E}$ [10,11,12], we found no such singular limits with respect to these kinetic rates.

The reason for this is that the studied functional computations and excess entropy is a continuous function of trajectory distributions alone, while statistical complexity must be written in terms of a distribution of causal states, which can vary in a non-continuous manner with the trajectory distribution. As a result, statistical complexity (and the mixed state presentation’s box-counting dimension $h_0$) and the $ϵ$-machine are incredibly sensitive to a particular type of process structure, which includes but is not limited to forbidden words. On the other hand, the studied functional computations and excess entropy are smoothly varying functions of the generative model’s kinetic rates.

From the study of the MWC molecule alone, we can conclude that a restrictive definition of intrinsic computation, e.g., only causal structure, does not necessarily provide a guide to the functional computation of a dynamical system, at least for the functional computations considered here. Accordingly, high statistical complexity does not imply biological function.

It’s well worth emphasizing that the functional computations listed here are far from an exhaustive list of all possible functional computations, and so future research might uncover a functional computation that depends sensitively on the process’ causal structure. Also, even if no such functional computation is identified, the sensitivity of causal structure to certain changes in the generative model might be considered by some to be an interesting feature, and not a bug, perhaps as a case study in how limited computational resources yield innovation [1]. However, for those wishing to study functional computation only, the extreme sensitivity of statistical complexity to particular types of process structure might prove to be a bug rather than a feature.

However, even then, the $ϵ$-machine finds use. In more recent years, intrinsic computation has been expansively defined to include a study of other structure-related information-theoretic statistics of a process besides statistical complexity [21]. This list includes but is not limited to excess entropy [10,11,12] as studied here, bound information rate [27], and predictive rate-distortion functions [28,29]. The last is particularly notable here in that predictive rate-distortion includes statistical complexity and excess entropy as limiting cases. On the whole, these quantities enjoy the label “information anatomy” [19] or the more broadly-construed “informational architecture”. Most of these quantities are smoothly varying functions of the transition probabilities in the minimal generative model and thus trajectory distribution, and so are not as sensitive to the underlying structure of the process as is statistical complexity. (The statistical complexity can vary discontinuously with transition probabilities in the minimal generative model, but varies smoothly with the transition probabilities in the minimal maximally predictive model; the structure of the minimal maximally predictive model, the $ϵ$-machine, can change by an arbitrarily large amount with arbitrarily small changes to the minimal generative model.) Those that are not so sensitive to the process’ structure are often easily calculable from the process’ $ϵ$-machine [29,30]. In the future, these quantities might provide interesting statistics with which to interpret the functional computation performed by biological or social systems, e.g., as in Ref. [31].