The Emergence of Integrated Information, Complexity, and ‘Consciousness’ at Criticality

Abstract

Integrated Information Theory (IIT) posits that integrated information ($\mathsf{\Phi }$) represents the quantity of a conscious experience. Here, the generalized Ising model was used to calculate $\mathsf{\Phi }$ as a function of temperature in toy models of fully connected neural networks. A Monte–Carlo simulation was run on 159 normalized, random, positively weighted networks analogous to small five-node excitatory neural network motifs. Integrated information generated by this sample of small Ising models was measured across model parameter spaces. It was observed that integrated information, as an order parameter, underwent a phase transition at the critical point in the model. This critical point was demarcated by the peak of the generalized susceptibility (or variance in configuration due to temperature) of integrated information. At this critical point, integrated information was maximally receptive and responsive to perturbations of its own states. The results of this study provide evidence that $\mathsf{\Phi }$ can capture integrated information in an empirical dataset, and display critical behavior acting as an order parameter from the generalized Ising model.

1. Introduction

A growing body of evidence has emerged suggesting that many disparate natural, and particularly biological, phenomena reside in a critical regime of dynamics on the cusp between order and disorder [1,2,3,4,5,6,7,8,9,10,11]. More specifically, it has been shown that models tuned to criticality exhibit similar dynamics to the brain [12,13,14,15], which, has led to the emergence of the Critical Brain Hypothesis [8,10]. Systems tuned to criticality exhibit a number of useful informational properties that allow for the efficient distribution of, and susceptibility to, information [6,10,15,16,17]. For example, Marinazzo et al. demonstrated that information is maximized at the critical temperature in an Ising scheme using human connectome data and beyond criticality, a law of marginal diminishing returns is reached [15]. These ideas have been further developed to suggest more broadly that critical systems are evolutionarily advantageous in that they are more effective at reacting to their environment and ensuring their continued survival [18,19,20].

Integrated information theory (IIT) is a top-down, phenomenological approach to defining consciousness [21]. Starting from phenomenological axioms inherently true for all conscious experiences, mathematical postulates have been put forth creating a workspace for quantitative research on consciousness. The main measure proposed by IIT 3.0 is the mathematical entity called integrated conceptual information ($\mathsf{\Phi }$) (Big Phi) which generally seeks to measure ‘how much the whole is greater than the sum of its parts’ of the causal structure being studied [21]. Though other measures exist [22] which attempt to capture some form of integration or complexity, this paper uses $\mathsf{\Phi }$ as the metric quantifying conscious experience. For a comprehensive overview of the mathematical taxonomy of the possible variations in defining integrated information, see [23,24].

Integrated information is a type of complexity measure that quantifies how mechanisms in a system interact and constrain each other in emergent, irreducible ways. This allows us to measure the properties of a system that cannot be explained by independent components of that system. $\mathsf{\Phi }$ is sensitive to not just information, which in general can be maximized by deterministic systems with unique pasts and futures, but also to the distribution and integration of information. In general, this tendency to distribute and integrate information is maximized by strongly coupled systems. To have a system that is both strongly coupled and informative requires a balance between segregating forces acting to differentiate the system, in conjunction with integrating forces creating new forms of information not present within the individual components. In the Ising model, it is expected that these exact properties emerge near the critical temperature at the onset of a phase transition.

It is important to note here that extending the concept of criticality to IIT is not new. Aguilera has studied criticality and the behavior of $\mathsf{\Phi }$ in a classical Ising model with a constant temperature [25]. Furthermore, Kim and Lee have noted that criticality and $\mathsf{\Phi }$ are correlated in a Kuramoto model along with EEG [26]. In their study coupling strength was used as the order parameter to achieve criticality whereas here, a random set of connectivity matrices were used, and the temperature was varied as the critical parameter. Because temperature was used as the critical parameter, we could test the critical behavior of $\mathsf{\Phi }$ independent of network connectivity.

The generalized Ising model simulates brain dynamics through its ability to exhibit phase transitions and critical points and is the simplest model associated with empirical pairwise correlation data [18,27]. Historically, the 2D classical Ising model exhibits a phase transition at a critical temperature $T_c$, a global scaling parameter of the model. This model, originally intended to describe the ferromagnetic properties of materials, was soon extended to different systems with binary interactions. Recently, it has been shown to exhibit similar dynamics to that of the brain, giving rise to the Critical Brain Hypothesis [2,3,4,5,6,7,8,9,10,12,14,16,28,29,30,31]. Fraiman et al. have shown that the concept of criticality extends to classical Ising motifs with $N=200$ nodes while modelling the brain [12].

This paper aims to demonstrate the critical properties of $\mathsf{\Phi }$ extend to different network connectivities with dynamics governed by the generalized Ising model. The generalized Ising model has been shown to simulate the statistical behavior of the brain [12,13,14,15]. By generating $\mathsf{\Phi }$ for random networks, a methodology is outlined that can be applied with patient tractographies to create a novel workspace for IIT.

To this end, the Ising model was simulated and criticality was obtained on 159 randomly generated, positive weighted N = 5 nodes networks to explore the combinatoric space of these neural network motifs. Each unique network displayed its own idiosyncratic phase transition as measured across a variety of its thermodynamic variables. $\mathsf{\Phi }$ and its susceptibility ${\chi }_{\mathsf{\Phi }}$ were measured across the parameter space of these models, fitting in naturally among the other thermodynamic variables such as energy (E), magnetization (M), specific heat ($C_v$) or magnetic susceptibility ($\chi$). Simulations swept across the model’s only free parameter, the temperature of the surrounding heat bath. As the temperature was varied from low to high, increasing energetic fluctuations became more probable. In many cases, as this parameter was varied, the organizational structure of the system dramatically changed, exhibiting a phase transition. Phase transitions or generalizations of such are at the heart of many of the most interesting complex systems such as genetic networks [32] societal organizations, financial markets [33,34], or swarming behaviors [35,36]. The critical points where these transitions are located are demarcated by the ‘critical temperature’ ($T_c$) of the generalized Ising model.

In Equation (1) an overview of the strategy employed in this paper is summarized. Ising simulations were run given a connectivity matrix and temperature ($J_{ij}$, T) with the outputs: magnetization, magnetic susceptibility, energy, specific heat, the current spin configuration of the system, and, a Markovian transition probability matrix (TPM). These parameters were then used to calculate the criticality of the system along with integrated information and the susceptibility of integrated information. All parameters were calculated as a function of temperature [M, $\chi$, E, $C_v$, TPM, s, $\mathsf{\Phi }$, ${\chi }_{\mathsf{\Phi }}$](T).

$$\begin{array}{cc}\hfill f_{\mathrm{Ising}}(J_{ij},T)& \to \left[M,\chi ,E,C_v,\mathrm{TPM},\overrightarrow{\mathbf{s}}\right]\left(T\right)\hfill \\ \hfill f_{\mathsf{\Phi }}(\mathrm{TPM},\overrightarrow{\mathbf{s}})\left(T\right)& \to \left[\mathsf{\Phi },{\chi }_{\mathsf{\Phi }}\right]\left(T\right).\hfill \end{array}$$

(1)

2. Materials and Methods

2.1. Random Networks

159 fully connected networks of five nodes with random edge weights uniformly sampled between 0 and 1 were generated and the principal diagonal was set to 0. The networks were then normalized such that their strongest weight was always unity. These random networks were saved as connectivity matrices ($J_{ij}$) and fed into Monte–Carlo Metropolis Ising simulations. These random networks were designed to explore the combinatoric space of ’ferromagnetic’ (positively weighted $J_{ij}$) fully connected neural network motifs each of which was constrained to 5 nodes.

This class of network was chosen as they are computationally tractable and represent brain connectivity as obtained through tractography measures. The calculation of integrated information scales super-exponentially in the number of nodes growing on $\mathcal{O}\left(n{53}^n\right)$ [37]. By choosing networks of 5 nodes, the computational intractability was overcome. Following the methodology of Marinazzo et al. and Abeyasinghe et al., the $J_{ij}$ can be obtained through tractography measures by counting the number of fibers connecting the $i^{th}$ and $j^{th}$ regions [15,31]. This results in the primary diagonal always being 0 (since one region has no fibers connecting it to itself). We further restrict ourselves to ferromagnetic connectivity matrices which correspond to the assumption of having only excitatory interactions. Although most parcellations of the brain are larger than 5 × 5, this treatment can be extended to network-level dynamics.

2.2. Ising Model

The Ising model is a simple way to simulate many-body interactions between binary elements. The simplified Ising model takes the form of Equation (2), where it is important to note in the classical model, the $(i,j)$ indices correspond to only neighboring lattice sites and in the general model, this restriction is relaxed. The equations do not change from model to model, instead, it is only a change in the collective interaction between lattice sites.

$$\begin{array}{c}\hfill H=-\sum _{i,j}{s}_i{s}_j{J}_{ij}.\end{array}$$

(2)

In this equation, $J_{ij}$ corresponds to the weights of nodal connections, $s_i=\pm 1$ representing the binary state of a magnetic ’spin’ site, and, H is the Hamiltonian or microscopic (dependent on spin configuration) energy.

In the most generic form, the Ising model also contains a magnetic field term. This term was not included in this study as we followed the protocol of Fraiman et al. and Abeyasinghe et al. for simulating spontaneous brain activity [12,31]. Present studies in our Lab are investigating the introduction of an external magnetic field to simulate an external stimulation.

When applying the Ising model to the brain, $J_{ij}$ is the number of fibers as determined through tractography measures for each region of interest, and each region of interest’s dynamics are represented as the spin state $s_i$.

Equation (2) was used alongside nature’s tendency to minimize energy to form the transition probability matrices for each of the 159 random networks. To equilibrate the model, a Monte–Carlo Metropolis algorithm and corresponding update rules were applied [38]. Through each iteration, a random element in the model was chosen and allowed the possibility for a ‘spin-flip’. A spin-flip will occur if the energy of the system after the flip is favorable (decreases). If the energy increases, the Boltzmann factor ($k_B=1$ in this study) was used to assign the probability that the ’spin-flip’ occurs.

By using the Monte–Carlo Metropolis algorithm, the calculation of the system’s partition function was not needed, as only a ratio of probabilities was required. Equation (3) describes the probability that was assigned to the i-th ’spin-flip’ location.

$$\begin{array}{c}\hfill \begin{array}{ccc}\hfill \mathrm{Prob}\left(s_i\to -s_i\right)& =exp(-\frac{\mathsf{\Delta }E}{k_{B}T})\hfill & \hfill \mathrm{if}\mathsf{\Delta }E>0\\ & =1\hfill & \hfill \mathrm{if}\mathsf{\Delta }E\le 0.\end{array}\end{array}$$

(3)

Following the update rules of Equation (3), a ’state-by-node’ TPM was constructed. The temperature in the model affects the rate at which ‘unfavorable’ spin-flips occur; increasing the temperature increases the noise/randomness of the model’s dynamics. Within each time step, all spins could flip once, updating simultaneously for the next step until the process was repeated for the desired number of time steps. Once the system had had enough time to equilibrate past its transient initial state, observables in the model were measured repeatedly and accumulated to generate the equilibrium expectation values.

Throughout the study, the simulation was run for an initial 500 time points to allow for thermalization. Upon equilibrating, the simulation was run for 2000 iterations. Thermodynamic properties and transition probabilities were calculated on each iteration. A logarithmically scaled temperature distribution was sampled 200 times between 0.1 and 4, with the simulation rerun for each temperature.

2.3. Summary Statistics

The summary statistics for observables $\mathcal{O}$ measured in this experiment are defined below. The thermodynamic observables (M,$\chi$) are stated for completeness.

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill M=& \frac{1}{N}|\sum _{i=1}^N{s}_i|\hfill \end{array}\end{array}$$

(4)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \chi =& \frac{\langle M^2\rangle -{\langle M\rangle }^2}{T}\hfill \end{array}\end{array}$$

(5)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \langle {\mathcal{O}}_t\rangle =& \frac{1}{N_t}\sum _i^{N_t}{\mathcal{O}}_i\hfill \end{array}\end{array}$$

(6)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\chi }_{\mathcal{O}}=& \langle {\mathcal{O}}^2\rangle -{\langle \mathcal{O}\rangle }^2={\sigma }_t^2\left(\mathcal{O}\right)\hfill \end{array}\end{array}$$

(7)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\sigma }_J^2\left(\mathcal{O}\right)=& {\langle {\langle \mathcal{O}\rangle }^2\rangle }_J-{\langle \langle \mathcal{O}\rangle \rangle }_J^2,\hfill \end{array}\end{array}$$

(8)

where $\langle {\mathcal{O}}_t\rangle$ is the expectation value of an observable across each iteration t. ${\chi }_{\mathcal{O}}$ is the generalized susceptibility [19,39,40] and ${\sigma }_J^2\left(\mathcal{O}\right)$ is the variance of an observable across all networks. All these summary statistics are calculated throughout the simulation.

2.4. Phi

Integrated Information ($\mathsf{\Phi }$) was calculated (using the PyPhi Python toolbox [37]) in the 5-node Ising model for 2000 iterations after the model reached a steady-state, which was assumed to be achieved after 500 iterations. To calculate $\mathsf{\Phi }$ a TPM must be supplied as well as the configuration of the system at that time-step ($s_t$). Calculating the TPM required the calculation of probabilities from any configuration to any other by iterating Equation (3) across all spin sites that needed to flip for the transition to occur, then taking their product. This led to a state-by-node TPM, which was the input parameter for PyPhi. These transition probabilities specified and constrained the past, present, and, future states of the system. In the framework of IIT 3.0 and PyPhi, ‘the TPM completely specifies the system’ [37], once the TPM has been realized, $\mathsf{\Phi }$ is readily determined.

Integrated information is defined through the effective information $\left(\varphi \right)$ of the minimum information partition (MIP). This is the partition made to the system that results in the minimum effective information. $\mathsf{\Phi }$ is then a function of the total system X, the state of the system x and the partition P. The basic schematic of calculating $\mathsf{\Phi }$ is outlined below and more detail is available in [21,23,24,25,26].

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathsf{\Phi }\{X;x\}=& \varphi \{X;x,\mathrm{MIP}(x\left)\right\}\hfill \end{array}\end{array}$$

(9)

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathrm{MIP}\left(x\right)=& min\left[\varphi \right(X;x,P\left)\right].\hfill \end{array}\end{array}$$

(10)

3. Results

Results indicated that the integrated conceptual information generated using the generalized Ising model, much like the classical variable magnetization, underwent a phase transition at the critical temperature. This was detected by locating the peaks of its susceptibility curves as a function of temperature [41], indicating that the integrated information structure of simple neural networks behaves critically, exhibiting maximal susceptibility to perturbations and allowing for a form of consciousness that balances coherence and continuity with information and variance. These results fit into the scheme in which the nature of evolution and the adaptive advantage of critical systems are understood in the context of a universe undergoing a cascade of phase transitions [42,43]. In statistical mechanics, phase transitions are typically thought to occur in the large N limit. However, Fraiman et al. have shown that the Ising model behaves critically for N = 200 nodes. In the attached supplementary material (Appendix A), it is demonstrated that when simulating the generalized Ising model for N = 525,100,250 nodes the same critical behavior is displayed [12].

159 Ising simulations were run on fully connected randomly weighted networks with N = 5 nodes. Summary statistics were calculated for simulated variables as a function of the fitting parameter T: magnetization M, integrated information $\mathsf{\Phi }$, the magnetic susceptibility $\chi$, the generalized susceptibility of integrated information ${\chi }_{\mathsf{\Phi }}$, along with the variances across all random network samples ${\sigma }_J^2\left(\mathcal{O}\right)$.

Figure 1 displays the summary statistics for the order and susceptibility parameters of the random networks. Magnetization M, energy E, and $\mathsf{\Phi }$ are plotted in the left-most column. Susceptibility $\chi$, specific heat $C_v$, and the susceptibility of $\mathsf{\Phi }$, ${\chi }_{\mathsf{\Phi }}$ are plotted in the right-most column. By averaging these variables across all random networks, a strong parallel between the behavior of magnetization M and integrated information $\mathsf{\Phi }$ can be seen. Near the onset of criticality (generally approximated by the peak of the magnetic susceptibility curve [40]) integrated information, much like the magnetization in the Ising model, undergoes a phase transition denoted by the peak in the susceptibility of $\mathsf{\Phi }$ (${\chi }_{\mathsf{\Phi }}$). The regime where the fluctuations of integrated information are maximized suggests a transition point for integrated information as an order parameter, demonstrating that the phenomenon of criticality extends into the behavior of integrated information in the Ising model.

In Figure 2 the variance of magnetization (${\sigma }_J^2\left(M\right)$), energy (${\sigma }_J^2\left(E\right)$), and integrated information (${\sigma }_J^2\left(\mathsf{\Phi }\right)$) are plotted in the left-most column. The variance of magnetic susceptibility (${\sigma }_J^2\left(\chi \right)$), specific heat (${\sigma }_J^2\left(C_v\right)$), and the susceptibility of integrated information (${\sigma }_J^2\left({\chi }_{\mathsf{\Phi }}\right)$) are plotted in the right-most column. All variances were taken across the different random network connectivities. The variances ${\sigma }_J^2$ illustrate how the connectivity choice affected the order parameters across random networks, whereas the susceptibilities quantify the mean fluctuations of each order parameter averaged across all random networks. These summary statistics give insights into the behavior of simple fully connected neural networks. ${\sigma }_J^2\left(\mathsf{\Phi }\right)$ shows to distinct regions, the first at low temperatures corresponds to a steep fall off and the second at higher temperatures is a linear decrease. These results highlight the regions where changes in the structural connectivity of the model had the most influence on the generation of integrated information. At lower temperatures, there is a much greater effect on $\mathsf{\Phi }$ from different connectivities than at higher temperatures. On the other hand, ${\sigma }_J^2\left({\chi }_{\mathsf{\Phi }}\right)$ displays the same trend as ${\sigma }_J^2\left(\chi \right)$ where near the critical temperature there is an increase in variance suggesting multiple network connectivities exhibit this critical behavior. As the networks became increasingly irreducible at criticality, each component contributed more to the conscious experience. While the magnetization of the model near criticality was maximally sensitive to changes in the structural connectivity, whereas integrated information instead decays linearly.

4. Discussion

4.1. Phase Transitions IIT

To investigate the properties of integrated information formulated by IIT 3.0, the Ising model was employed to act as a proxy to the brain. The results show that integrated information, as with magnetization tends to be maximally susceptible at the critical temperature (Figure 1). The ‘susceptibility’ of $\mathsf{\Phi }\left(T\right)$, (${\chi }_{\mathsf{\Phi }}\left(T\right)$) has a distinct peak at criticality, typically the marker for a second-order phase transition with the classical 2D Ising model. This indicates that integrated information can be considered a macroscopic order parameter that undergoes a phase transition at the critical temperature. The present study was limited to implementing the Ising model on fully connected graphs using the Monte–Carlo Metropolis algorithm. In the future as more efficient algorithms for calculating $\mathsf{\Phi }$ emerge (or as a compromise accurate correlates of $\mathsf{\Phi }$), combined with Monte–Carlo and network renormalization group methods [16,44,45,46,47,48,49], the exploration of larger networks of different classes (e.g., sparse, modular hierarchical, small-world, fractal) could lead to the identification of a rich taxonomy of phases of integrated information.

4.2. Evolution Complexity

The exploration of integrated information in the context of critical systems undergoing phase transitions motivates several new questions regarding the relationship between evolution, complexity, and consciousness. In the work done by Joshi et al. and Albantakis et al. on the complexity and the evolution of neural models and integrated information, it was shown that fitness can correlate strongly with $\mathsf{\Phi }$ when the system is constrained in size and/or resources [50,51]. While it is not always true that a system will evolve to generate high $\mathsf{\Phi }$ under more liberal constraints (e.g., infinite resources), it does seem to be that there may be some evolutionary advantage for having high $\mathsf{\Phi }$. Because $\mathsf{\Phi }$ essentially measures the emergence of higher-order concepts within a system, it is not surprising that systems capable of generating higher-order concepts are also capable of representing and reacting to a diverse set of states. Therefore, for resource-limited systems, having an efficient means to represent internal and external states may automatically give rise to high $\mathsf{\Phi }$ or ‘consciousness’.

4.3. Utility of Criticality

Critical systems have diverging correlation lengths, undergo critical slowing down (i.e., integration in space and time), and simultaneously exhibit distinct and segregated structures at all scales (i.e., scale-invariance). They are generally found in regimes of systems undergoing a transition between phases (e.g., magnetized vs. non-magnetized in the Ising model, or synchrony vs. asynchrony in the Kuramoto model [52,53,54,55,56]). In contrast to sub-critical regimes which can become completely uniform due to their strong coupling (high integration, low differentiation), and super-critical regimes which can become completely noise-driven (low integration, high differentiation), critical systems sit at the cusp of integration and differentiation; generating non-negligible $\mathsf{\Phi }$ that is maximally susceptible to the perturbations of its environment and its own state. Our results indicate that while sub-critical regimes can generate high $\mathsf{\Phi }$, the variations in $\mathsf{\Phi }$ are negligible. Only near the critical point does $\mathsf{\Phi }$ have both large values and large fluctuations indicating that the critical point of the system is maximally receptive and responsive to its states. It is in this state of maximal susceptibility where concepts become maximally irreducible—${\chi }_{\mathsf{\Phi }}\left(T\right)$ increases to a peak. At this point of criticality, the ‘conscious experience’ as defined by IIT 3.0 is the most ’conscious’; criticality exhibited by the neural network motif leads to the ‘best’ conscious experience [21].

4.4. Future Work

Both the magnitude and susceptibility of $\mathsf{\Phi }$ in the Ising model (and in general) are extremely sensitive to the connectivity of the underlying system. So far, simulations have been run on static networks, but in general, one can allow the network to evolve. Future work regarding how the networks arise could explore different evolutionary algorithms and dynamical rules in combination with the analysis from IIT. This would assess the role of evolution and the environment in generating $\mathsf{\Phi }$ and its underlying critical structure. Exploring the behavior of $\mathsf{\Phi }$ in different classes of phase transitions would allow further development of the ideas behind the Critical Brain Hypothesis, and would combine the fields of neuroscience, complexity science, material science, and statistical mechanics in order to understand the brain. Furthermore, the methodology of extracting $\mathsf{\Phi }$ from the generalized Ising model can be applied using $J_{ij}$’s extracted from human cortical data, allowing $\mathsf{\Phi }$ to be extracted for conscious individuals.

5. Conclusions

Ultimately, this study is best framed in the context of the emerging complexity of our world [43]. The brain is one of the most complex objects ever studied and the theory of it acting critically is gaining credence. New research into critical systems has shown that criticality may be useful for learning [9], and for optimizing information processing [10,18,19], both of which are tasks the brain is constantly performing. Phase transitions and criticality are gaining more relevance, and the evidence in this paper demonstrates that by defining consciousness with IIT and using the Ising model as a substrate, ‘consciousness’ undergoes a phase transition at criticality in the investigated neural network motifs. This, when combined with evidence that the brain may be critical, suggests that ‘consciousness’ may simply arise out of the tendency of the brain to self-organize towards criticality. The Critical Brain Hypothesis and IIT appear to go hand-in-hand, paving the way for more studies concerning the interdependencies between the two models in both patient and simulated data.