Anatomy of a Spin: The Information-Theoretic Structure of Classical Spin Systems

Abstract

Collective organization in matter plays a significant role in its expressed physical properties. Typically, it is detected via an order parameter, appropriately defined for each given system’s observed emergent patterns. Recent developments in information theory, however, suggest quantifying collective organization in a system- and phenomenon-agnostic way: decomposing the system’s thermodynamic entropy density into a localized entropy, that is solely contained in the dynamics at a single location, and a bound entropy, that is stored in space as domains, clusters, excitations, or other emergent structures. As a concrete demonstration, we compute this decomposition and related quantities explicitly for the nearest-neighbor Ising model on the 1D chain, on the Bethe lattice with coordination number $k=3$, and on the 2D square lattice, illustrating its generality and the functional insights it gives near and away from phase transitions. In particular, we consider the roles that different spin motifs play (in cluster bulk, cluster edges, and the like) and how these affect the dependencies between spins.

1. Introduction

Collective behavior underlies a vast array of fascinating phenomena, many of critical importance to contemporary science and technology. Unusual material properties—such as superconductivity, metal-insulator transitions, and heavy fermions—have been attributed to collective behavior arising from the compounded interaction of system components [1]. Collective behavior is by no means limited to complex materials, however: the behavior of financial markets [2], epileptic seizures [3] and consciousness [4], and animal flocking behavior [5] are all also seen as examples. We now appreciate that it appears most prominently via phase transitions ([6] and references therein).

Operationally, collective behavior is detected and quantified using correlations or, more recently, by mutual information [7], estimated either locally via pairwise component interactions or globally. Exploring spin systems familiar from statistical mechanics, here we argue that these diagnostics can be substantially refined to become more incisive tools for quantifying collective behavior. This is part of the larger endeavor of discovering ways to automatically detect collective behavior and the emergence of organization [8].

Along these lines, much effort has been invested to explore the information-theoretic properties of the Ising model of statistical mechanics. Both Shaw [9] and Arnold [10] studied information in 1D spin-strips within the 2D Ising model. Feldman and Crutchfield [11,12,13] explored several generalizations of the excess entropy, a well known mutual information measure for time series, in 1D and 2D Ising models as a function of coupling strength, showing that they were sensitive to spin patterns and can be used as a generalized order parameter. In one of the more thoroughgoing studies to date, Lau and Grassberger [14] computed the excess entropy using adjacent rings of spins to probe criticality in the 2D cylindrical Ising model. Barnett et al. [6] tracked information flows in a kinetic Ising system using both the pairwise and global mutual information (generalized as the total correlation [15], it appears below) and the transfer entropy [16]. Abdallah and Plumbley [17] employed the dual total correlation [18,19] on small, random spin glasses. Despite their successful application, the measures used to monitor collective phenomena in these studies were motivated information theoretically rather than physically, and so their thermodynamic relevance and structural interpretation have remained unclear.

To remedy this, we decompose the thermodynamic entropy density for spin systems into a set of information measures, some already familiar in information theory [20], complex systems [21], and elsewhere [17]. We seek to illustrate how these different measures from information theory may be applied to spin systems. For example, one measure that falls out naturally—the bound entropy—is that part of the thermodynamic entropy accounting for entropy shared between spins. This is in contrast to monitoring collective behavior as the difference between the thermodynamic entropy and the entropy of a hypothetical distribution over uncorrelated spins—the total correlation mentioned above. In this way, the entropy-density decomposition provides a physical basis for informational measures of collective behavior, in particular showing how the measures lead to interpretations of emergent structure in system configurations. Paralleling the work in [22,23] with respect to decomposition of an information bit and stochastic equilibria, “anatomy” refers to dissecting something typically taken as a unitary whole—here, spin thermodynamic entropy density—to reveal its inner components.

Collective behavior can manifest itself in many ways. Perhaps the most obvious is synchronization [24]: schooling fish, flocking birds, and pulsing fireflies. It is these sorts of behaviors that maximize the total correlation. There are, however, other forms of collective behavior. Consider an error-correcting parity bit where a set of n random bits is augmented with an $n+1^{\mathrm{th}}$ bit that is the sum mod 2 of the others. In this case, any proper subset of the $n+1$ bits consists of bits that are statistically independent, but any n of them can uniquely identify the last. It is this sort of collective behavior that maximizes the dual total correlation.

To make this argument and illustrate its consequences, Section 2 first defines notation and lays the groundwork for our thermodynamic entropy-density decomposition. Section 3 then derives the decomposition. Though the decomposition is given for regular spin lattices, in principle it is meaningful for any system with well-defined thermodynamic entropy. Section 4 outlines the computational methods for estimating the resulting quantities, using them to interpret emergent organization in the nearest-neighbor ferromagnetic Ising model in one dimension (Section 4.1), in the Bethe lattice with coordination number $k=3$ (Section 4.2), and in the two-dimensional square Ising lattice (Section 4.3). Having established the different types of entropy in spin systems and connections to their underlying physical structures, we conclude by suggesting applications and future directions.

2. Spin Entropies

We write $\sigma$ to denote a configuration of spins on a lattice $\mathcal{L}$, ${\sigma }_i$ for the particular spin state at lattice site $i\in \mathcal{L}$, and ${\sigma }_{\setminus i}$ the collection of all spin states in a configuration excluding ${\sigma }_i$ at site i. The random variable over all possible spin configurations is denoted $\mathit{\sigma }$, a particular spin variable is ${\mathit{\sigma }}_i$, and all spin variables but the one at site i is ${\mathit{\sigma }}_{\setminus i}$. As a shorthand, we write $\sigma \in \mathit{\sigma }$ and similar to mean indexing into the event space of the global configuration random variable $\mathit{\sigma }$. We study the ferromagnetic spin-$1/2$ Ising model with nearest-neighbor interactions in the thermodynamic limit whose Hamiltonian is given by:

$$\begin{array}{c}\hfill \mathcal{H}\left(\sigma \right)=-J\sum _{\langle i,j\rangle }{\sigma }_i{\sigma }_j,\end{array}$$

(1)

where $\langle i,j\rangle$ denotes all pairs $(i,j)$ such that the sites i and j are directly connected in $\mathcal{L}$ and the interaction strength J is positive. We assume the system is in equilibrium and isolated. So, the probability of configuration $\sigma$ occurring is given by the Boltzmann distribution:

$$\begin{array}{c}\hfill p\left(\sigma \right)=\frac{1}{Z}{e}^{-\mathcal{H}\left(\sigma \right)/k_{\mathrm{B}}T},\end{array}$$

(2)

where Z is the partition function.

The Boltzmann entropy of a statistical mechanical system assumes the constituent degrees of freedom (here spins) are uncorrelated. To determine it, we use the isolated spin entropy:

$$\begin{array}{c}\hfill \mathrm{H}\left[{\mathit{\sigma }}_0\right]=-p\left(\uparrow \right){\log }_{2}p\left(\uparrow \right)-p\left(\downarrow \right){\log }_{2}p\left(\downarrow \right),\end{array}$$

where $p\left(\uparrow \right)=(1+m)/2$ is the probability of a spin being up, $p\left(\downarrow \right)=(1-m)/2$ is the probability of a spin being down, and $m=(\#\uparrow -\#\downarrow )/\left|\mathcal{L}\right|$ is average magnetization in a configuration. The site index 0 was chosen arbitrarily and represents any single spin in the lattice. The system’s Boltzmann entropy is then the extensive quantity $H_{\mathrm{B}}=\left|\mathcal{L}\right|·\mathrm{H}\left[{\mathit{\sigma }}_0\right]$.

As Jaynes [25] emphasizes, though correct for an ideal gas, $H_{\mathrm{B}}$ is not the empirically correct system entropy if a system develops internal correlations. In our case, if spins are correlated across the lattice, one must consider entire configurations, leading to the Gibbs entropy:

$$\begin{array}{c}\hfill \mathrm{H}\left[\mathit{\sigma }\right]=-\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right){\log }_{2}p\left(\sigma \right)\end{array}$$

(3)

and the thermodynamic entropy density h, the entropy per spin when the entire lattice is considered:

$$\begin{array}{c}\hfill h=\frac{\mathrm{H}\left[\mathit{\sigma }\right]}{\left|\mathcal{L}\right|}.\end{array}$$

(4)

Note that we dropped the factor $k_{\mathrm{B}}$ and used the base-2 logarithm. Our use of the letter h as opposed to s is meant to reflect this multiplicative constant difference. It is easy to see that $\mathrm{H}\left[\mathit{\sigma }\right]\le H_{\mathrm{B}}$. Thus, the Boltzmann entropy is typically an overestimate of the true thermodynamic entropy.

More to the point, in the thermodynamic limit the Boltzmann entropy $H_{\mathrm{B}}$ and Gibbs entropy $\mathrm{H}\left[\mathit{\sigma }\right]$ differ substantially. As Jaynes shows [25], the difference directly measures the effect of internal correlations on total energy and other thermodynamic state variables. Moreover, the difference does not vanish in the thermodynamic limit, rather it increases proportionally to system size $\left|\mathcal{L}\right|$.

Before leaving the differences between entropy definitions, it is important to note that Boltzmann’s more familiar definition—$S=k_{\mathrm{B}}\ln W$—via the number W of microstates associated with a given thermodynamic macrostate is consistent with Gibbs’ definition [25]. This follows from Shannon’s deep insight on the asymptotic equipartition property that $2^{h\left|\mathcal{L}\right|}$ measures the volume of typical microstates: the set of almost-equiprobable configurations that are simultaneously most numerous and capture the bulk of the probability—those that are typically realized (see Section 21 of [26] , Chapter 3 of [7] , and [27,28]). Thus, Boltzmann’s W should be interpreted as the size (phase space volume) of the typical set associated with a particular thermodynamic macrostate. Given this agreement, we focus on Gibbs’ approach. Though, as we now note, the contrast between Gibbs’ entropy $\mathrm{H}\left[\mathit{\sigma }\right]$ and Boltzmann’s $H_{\mathrm{B}}$ leads directly to our larger goals of interpretable decompositions of the thermodynamic entropy density.

3. Decomposing a Spin’s Thermodynamic Entropy

Since we are particularly interested here in monitoring the appearance of internal correlations, the difference between the Boltzmann and Gibbs entropies suggests itself as our first measure of a system’s internal organization. If each spin were independent, $\mathrm{H}\left[{\mathit{\sigma }}_0\right]=h$. For example, this is true at $T=\infty$ for the Ising model and for site percolation models [29]. If there are correlations between spins, however, then $h<\mathrm{H}\left[{\mathit{\sigma }}_0\right]$, as just noted for the extensive quantities. Their difference is the total correlation density [22]:

$$\begin{array}{cc}\hfill \rho & =\mathrm{H}\left[{\mathit{\sigma }}_0\right]-h\hfill \\ & =\frac{\mathrm{T}\left[\mathit{\sigma }\right]}{\left|\mathcal{L}\right|},\hfill \end{array}$$

(5)

$\mathrm{T}\left[\mathit{\sigma }\right]=\mathcal{D}\left(p\left(\sigma \right)\left|\right|{\prod }_{i=1}^{\left|\mathcal{L}\right|}p\left({\sigma }_i\right)\right)$ here is the Kullback–Leibler divergence $\mathcal{D}(·||·)$ [7] between the distribution over entire configurations and the product of isolated-spin marginal distributions. Now called the total correlation [15], this measure of internal correlation was wholly anticipated by Gibbs, as Jaynes notes (Equation (4) of [25]). Since $\rho$ vanishes only when each spin is independent, it can be considered a measure of pattern or structure in a spin system. More specifically, it measures how constrained the spin distribution is and so is maximized when the spins are strongly correlated and act in concert. While this is certainly reasonable as an operational definition, our next step decomposes both h and $\rho$ into more nuanced and incisive quantities.

Continuing, we recast the Gibbs thermodynamic entropy $\mathrm{H}\left[\mathit{\sigma }\right]$ as the sum of two nonnegative terms; starting from the configuration entropy in its standard statistical form Equation (3) and manipulating it into two terms:

$$\begin{array}{cc}\hfill \mathrm{H}\left[\mathit{\sigma }\right]& =-\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right){\log }_2\left[p\left(\sigma \right)\frac{{\displaystyle \prod _{i=1}^{\left|\mathcal{L}\right|}p\left({\sigma }_i|{\sigma }_{\setminus i}\right)}}{{\displaystyle \prod _{i=1}^{\left|\mathcal{L}\right|}p\left({\sigma }_i|{\sigma }_{\setminus i}\right)}}\right]\hfill \\ & =\mathrm{R}\left[\mathit{\sigma }\right]+\mathrm{B}\left[\mathit{\sigma }\right],\hfill \end{array}$$

(6)

where $\mathrm{R}\left[\mathit{\sigma }\right]$ and $\mathrm{B}\left[\mathit{\sigma }\right]$, known as the residual entropy and dual total correlation [17,20,22], are the following measures:

$$\begin{array}{cc}\hfill \mathrm{R}\left[\mathit{\sigma }\right]& =\sum _{i=1}^{\left|\mathcal{L}\right|}\left[-\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right){\log }_{2}p\left({\sigma }_i|{\sigma }_{\setminus i}\right)\right]\hfill \\ & =\sum _{i=1}^{\left|\mathcal{L}\right|}\mathrm{H}\left[{\mathit{\sigma }}_i|{\mathit{\sigma }}_{\setminus i}\right],\hfill \end{array}$$

(7)

and,

$$\begin{array}{cc}\hfill \mathrm{B}\left[\mathit{\sigma }\right]& =-\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right){\log }_2\frac{p\left(\sigma \right)}{{\displaystyle \prod _{i=0}^{\left|\mathcal{L}\right|}p\left({\sigma }_i|{\sigma }_{\setminus i}\right)}}\hfill \\ & =\mathrm{H}\left[\mathit{\sigma }\right]-\sum _{i=0}^{\left|\mathcal{L}\right|}\mathrm{H}\left[{\mathit{\sigma }}_i|{\mathit{\sigma }}_{\setminus i}\right].\hfill \end{array}$$

(8)

Note that both $\mathrm{R}\left[\mathit{\sigma }\right]$ and $\mathrm{B}\left[\mathit{\sigma }\right]$ are nonnegative and bound from above by $\mathrm{H}\left[\mathit{\sigma }\right]$. The dual total correlation is also zero if and only if the spins are mutually independent, and so it too is a measure of collective behavior. In this case, however, it measures the actual information content of the spin distribution and is maximized when subsets of spins are independent, but collectively can determine one another. Spatial densities are denoted in lower case: $h=\mathrm{H}\left[\mathit{\sigma }\right]/\left|\mathcal{L}\right|$, $r=\mathrm{R}\left[\mathit{\sigma }\right]/\left|\mathcal{L}\right|$, and $b=\mathrm{B}\left[\mathit{\sigma }\right]/\left|\mathcal{L}\right|$. We consider their thermodynamic limit, where $\left|\mathcal{L}\right|\to \infty$.

Similarly, we can begin with the total correlation of spins $\mathrm{T}\left[\mathit{\sigma }\right]$ and break it into two terms:

$$\begin{array}{cc}\hfill \mathrm{T}\left[\mathit{\sigma }\right]& =\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right){\log }_2\frac{p\left(\sigma \right)}{{\displaystyle \prod _{i=1}^{\left|\mathcal{L}\right|}p\left({\sigma }_i\right)}}\hfill \end{array}$$

(9)

$$\begin{array}{c}=\mathrm{B}\left[\mathit{\sigma }\right]+\mathrm{Q}\left[\mathit{\sigma }\right],\hfill \end{array}$$

(10)

where $\mathrm{B}\left[\mathit{\sigma }\right]$ is as above. Though perhaps not clear here, $\mathrm{B}\left[\mathit{\sigma }\right]$ is naturally a component of $\mathrm{T}\left[\mathit{\sigma }\right]$; see [22] Figure 7c,d. $\mathrm{Q}\left[\mathit{\sigma }\right]$ is the enigmatic information [22]:

$$\begin{array}{cc}\hfill \mathrm{Q}\left[\mathit{\sigma }\right]& =\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right){\log }_2\frac{p{\left(\sigma \right)}^2}{{\displaystyle \prod _{i=1}^{\left|\mathcal{L}\right|}p\left({\sigma }_i\right)\prod _{j=1}^{\left|\mathcal{L}\right|}p\left({\sigma }_j|{\sigma }_{\setminus j}\right)}}\hfill \\ & =\mathrm{T}\left[\mathit{\sigma }\right]-\mathrm{B}\left[\mathit{\sigma }\right],\hfill \end{array}$$

(11)

and again, we consider the spatial density $q=\mathrm{Q}\left[\mathit{\sigma }\right]/\left|\mathcal{L}\right|$ in the thermodynamic limit.

It can be difficult to initially intuit the difference between $\mathrm{T}\left[\mathit{\sigma }\right]$ and $\mathrm{B}\left[\mathit{\sigma }\right]$. The key is that they both measure dependencies among spins, but against different reference ensembles. $\mathrm{T}\left[\mathit{\sigma }\right]$ is the difference between the Boltzmann and Gibbs entropies (Equation (5)) and, therefore, quantifies the spin distribution’s deviation from a hypothetical distribution in which each spin is independent from the others. Since $\mathrm{R}\left[\mathit{\sigma }\right]$ is the amount of actual independent entropy in the spin distribution, $\mathrm{B}\left[\mathit{\sigma }\right]$ quantifies the amount of spin–spin dependency in configurations not in reference to a hypothetical independent-spin distribution, but rather to the actually realized amount of independence. This difference is pictorially represented in Figure 1.

Let us now interpret the physics captured by these entropy-density components. First, $\mathrm{H}\left[{\mathit{\sigma }}_0\right]$ quantifies the entropy per spin ignoring any dependencies (e.g., correlations) between spins. The thermodynamic entropy density h, however, is the entropy per spin including dependencies. Continuing in this way, from Equation (7) r is the entropy per spin remaining after these dependencies have been factored out, meaning it is the average amount of independent entropy—the entropy per spin when we know the state of the other spins in the lattice. With this logic, b is that portion of the thermodynamic entropy density coming from dependencies and is, therefore, the average amount of dependent entropy per spin. As we shall see, in comparison to $\rho$, b provides a complementary, quantitatively different, and more physically grounded approach to quantifying dependencies among spins.

This completes, in effect, our decomposition of the isolated-spin entropy $\mathrm{H}\left[{\mathit{\sigma }}_0\right]$, shown schematically in Figure 2. To take stock, let us back up a bit. Recall that $\rho$ is the difference between $\mathrm{H}\left[{\mathit{\sigma }}_0\right]$, which ignores the dependencies between the spins, and the thermodynamic entropy density h. That is, $\rho$ is the difference between the isolated-spin entropy and how much entropy (h) there actually is. b is the difference between the thermodynamic entropy h and the amount of independent entropy r there is. In addition, b is also part of $\rho$ [22], leaving q which therefore quantifies potential dependencies (explained shortly) not present in the thermodynamic entropy density h. To mathematically reframe these results, we showed that the Boltzmann thermodynamic entropy naturally decomposes into a complete set of irreducible atoms in the algebra of information measures [30].

Let us explore the consequences of this decomposition. Although r and b are components of the thermodynamic entropy density h—itself a measure of the irreducible per-spin randomness—individually they have very different meanings that give insight into the interplay of spin thermalization and spin ordering. The first, r, is the average amount of unshared entropy, quantifying randomness remaining in a single spin given the exact configuration of the rest of the lattice. For this reason we refer to it as the localized entropy. In the limit of high and low temperature, r’s behavior is intuitive. At low temperature, spins in the lattice are entirely aligned, and so there is no randomness in individual sites given the global alignment. At high temperatures, each spin is independent, and so there is full randomness in a site, even knowing its surroundings.

The other quantity, b, is the entropy remaining in a spin when the localized entropy is deducted, and so it captures the thermodynamic entropy shared between a spin and the rest of the lattice. We refer to it as the bound entropy. Its limits are similarly intuitive. At low temperatures there is no randomness to share, and so $b=0$. Similarly, at high temperatures all randomness is localized, and so no entropy can be shared and bound entropy also vanishes. At intermediate temperatures, though spins are in flux, a spin’s context does at least in part influence its alignment and so $b>0$. We refer to the temperature at which b is maximized by $T_b$.

Finally, we consider the enigmatic information density q. As a component of $\rho$ it also reflects dependencies, but in a way different from b. The bound entropy captures dependencies that are also part of the thermodynamic entropy, but q captures those that are not. As we shall see in Section 5, this means it is sensitive to the interior of clusters. At low temperatures, i.e., temperatures below $T_c$, we expect q to be small, since $\mathrm{H}\left[{\mathit{\sigma }}_0\right]$ is small. At high temperatures, there are no dependencies between spins, and so we also expect q to be small. At intermediate temperatures, just like b, we expect q to take larger values, but values that reflect the particular kinds of patterning in spin configurations.

The nonzero values of $\rho$, b, and q are all driven by Ising spin clustering. Consider for the moment randomly permuting the sites in the lattice, resulting in a site percolation model with site occupancy probability given by a simple function of the magnetization of the original Ising lattice. Then, the spin orientation at any particular site is now statistically independent of its neighbors, and so $\rho =b=q=0$, and therefore $\mathrm{H}\left[{\mathit{\sigma }}_0\right]=h=r$. There are, however, clusters in this permuted lattice—in fact, there may exist a giant cluster if the magnetization is such that the probability of site occupation is above the percolation threshold. These clusters are driven purely by the lattice topology. We therefore conclude that $\rho$, b, and q are each sensitive to differing aspects of how the pairwise Ising Hamiltonian affects the shape and distribution of spin clusters.

Finally, we note that all these information measures exhibit discontinuities in their derivative with respect to temperature at $T_c$. This behavior is not surprising, however. Each of them is a method of probing the spin statistics of the lattice. The magnetization—the mean of ${\sigma }_0$—also exhibits such a discontinuity, and so it is not surprising that more nuanced measures such as those considered here have a similar discontinuity.

To summarize, the disorder generated at each site is measured by the Gibbs thermodynamic entropy density h. One portion (b), participates in spatial organization and the other portion (r) does not. Thus, b is key to monitoring the emergence of spatial structures during pattern formation dynamics.

4. Results

To explore what the entropy components reveal, we calculate them for the ferromagnetic spin-$1/2$ Ising model on one- (1D) and two-dimensional (2D) square lattices and on the Bethe lattice. These results demonstrate that the preceding qualitative explanations of the information measure behaviors are correct, but the results also raise subtle questions regarding the details of exactly what the measures capture.

As part of this, we report analytical results for the 1D chain and the Bethe lattice and show that they match results from simulation. We use analytic results whenever possible for the 2D lattice. For example, we report analytical results for h, $\mathrm{H}\left[{\mathit{\sigma }}_0\right]$, and $\rho$. Then, only r needs to be estimated from simulation and that result is then combined with analytic quantities to compute b and q. To provide a comparison, estimating b purely from simulation requires accurate h and r. In this, we make use of the global Markovian property of spin lattices [31]. To obtain r we need only condition a spin on all its nearest neighbors—spins directly coupled via the Hamiltonian of Equation (1). For h estimated purely from simulation, we use the method provided in [32]. In both cases, the entropy estimator ${\widehat{H}}_2$ found in [33] was used to decrease the number of configuration samples while still providing convergence to known analytic results. Overall, when comparing and when analytic results are available, simulation results match to within standard error bars smaller than plot line widths. All simulation results were obtained using the Wolff algorithm [34]. In summary, in the reported results for those few quantities that required estimated or partial estimation from simulations, sufficient care was taken that the statistical errors fall far below the level needed to support the main conclusions.

4.1. 1D Ising Model

Simple Ising models have been broadly adapted to understand simple collective phenomena in fields ranging from surface physics [35,36] and biophysics [37,38] to economics [39]. The Ising model on a one-dimensional lattice ($\mathcal{L}=\mathbb{Z}$) can be fully analyzed in closed form. Unfortunately, its emergent patterns are rather constrained; for example, it does not exhibit a phase transition [40]. However, the exact solutions provide an important benchmark and so it is a necessary candidate with which to start and provides a base from which to generalize [41,42,43,44]. Analytic results were computed using a transfer matrix approach combined with the aforementioned Markovian property. For comparison, simulations were performed on a lattice of size $N=1024$. Note that the results presented are analytical solutions, and simulations were performed only to double check our calculations.

Independent of possessing a phase transition or not, the argument that b maximizes between temperature extremes still holds and Figure 3 verifies this. The bound entropy b reaches a maximal value at $T_b\approx 1.0117$ J/k${}_{\mathrm{B}}$ and this is not (and cannot be) associated with a phase transition. At lower temperatures, spin-spin shared entropy b is the dominant contributor to the Gibbs thermodynamic entropy density $h=r+b$. At higher temperatures, the localized entropy r is the dominant contributor. Similarly, at low temperatures q is the dominant contributor to $\rho$, and b is its dominant contributor at high temperatures. Section 5 below isolates why b peaks where it does and to which spin-configuration features each measure is sensitive. For now, let us continue with system informational phenomenology.

4.2. Bethe Lattice Ising Model

What role does the underlying lattice topology play in determining the balance between local randomness and spatially shared information that equilibrium configurations achieve? Spins on the Bethe lattice, in which $\mathcal{L}$ is an infinite Cayley tree with coordination number k, form an ideal candidate since the absence of loops makes it possible to compute quantities analytically. Moreover, spin configurations exhibit a phase transition at critical temperature [41]:

$$\begin{array}{c}\hfill T_c=2J{\left[k_{\mathrm{B}}\ln \left(\frac{k}{k-2}\right)\right]}^{-1}.\end{array}$$

(12)

We analytically calculated the entropy decomposition for a Bethe lattice with coordination number k, the details of which appear in Appendix A. Simulation results, matching the analytic to high precision, were also estimated numerically on a 1000 node random 3-regular graph [45], which has no boundary and is locally Bethe lattice-like.

Figure 4 presents the results for a Bethe lattice with coordination number $k=3$, though other ks behave similarly. Interestingly, all information measures have a discontinuity in their first derivatives, and this happens at the phase transition at $T_c$. Furthermore, the bound entropy b is maximized there: $T_b=T_c\approx 1.8205$J/k${}_{\mathrm{B}}$. Opposite the 1D lattice, at small T the dominant contributor to Gibbs entropy h is the local randomness r. Thus, not only does the change in lattice topology induce a phase transition, but it also inverts the informational components’ contributions to thermodynamic entropy density. This, in turn, indicates a rather different underlying mechanism that supports entropy production in the low temperature regime. In the 1D case entropy is spatially extended and configurations of spins are dominated by large clusters. In the Bethe lattice deviations from uniformity come in the form of isolated spins differing from their surroundings. In short, except at high temperatures, Bethe lattice spin configurations are simply more random than those in 1D at corresponding temperatures and corresponding entropy densities. Also, unlike 1D, the spin-spin shared entropy b is the dominant contributor to $\rho$ at both low and high temperatures.

4.3. 2D Ising Model

Unlike its 1D sibling, the nearest-neighbor ferromagnetic Ising model in two dimensions ($\mathcal{L}={\mathbb{Z}}^2$) has a phase transition at a finite critical temperature $T_c=2/\ln \left(1+\sqrt{2}\right)\approx 2.2692$ J/k${}_{\mathrm{B}}$. In the 2D case, although Onsager’s solution let us calculate $\mathrm{H}\left[{\sigma }_0\right]$, h, and $\rho$ [41], we do not have an analytic form for r and so the curves for r, b, and q in Figure 5 partly rely on estimates from simulation, as explained above. The simulations were conducted on a $128\times 128$ lattice with periodic boundary conditions and quantities averaged over 200,000 configuration updates. It is important to note that the finite system with periodic boundary conditions is ergodic and so, in principle, $\mathrm{H}\left[{\sigma }_0\right]$ is 1 for all temperatures. However, our integration time is several orders of magnitude shorter than the expected magnetic spin-flip time due to thermal fluctuations, and so the simulations are both qualitatively and quantitatively similar to the thermodynamic limit and closely match analytic results.

Again, the resulting standard error bars were smaller than the plotted line widths.

Phase transition aside, the behaviors of h, r, and b, seen in Figure 5, are qualitatively similar to those in the 1D lattice. However, unlike 1D, at low temperatures r is the dominant contributor to h, similar to the Bethe lattice below its transition. Also, paralleling the Bethe lattice, b is the dominant contributor to $\rho$ at both low and high temperatures. Like the 1D system, bound entropy is maximized within the disordered phase at $T_b\approx 2.675$ J/k${}_{\mathrm{B}}$, above $T_c$. Thus, the measures show that the 1D system over its whole temperature range is informationally analogous to the 2D system for $T>T_c$. Let us now turn to what configurational structures lead to the overall behaviors of these measures and how they contribute to the Gibbs entropy density h.

5. Discussion

At first, it is striking that $T_b$ is not generically identical to $T_c$. To understand why, we need to investigate to which spin-configuration motifs the various measures are sensitive. To accomplish this, we take a local approach. Each of the measures examined above is an average density, which is important when discussing the lattice as a whole and its bulk thermodynamic properties. However, each spin configuration motif contributes differently to this average. For example, an up spin surrounded by down spins contributes to the measures differently than an up spin surrounded by other up spins.

5.1. Motif Entropies

To quantify this, we appeal to spatially- and temporally-local forms of the averaged densities considered so far. This is possible on square lattices $\mathcal{L}={\mathbb{Z}}^d$, due to the existence of conditional forms of the entropy density; see [46] Theorem 2.9. The conditional forms for each of the component quantities are configuration-weighted averages over quantities of the form ${\log }_2\left(•\right)$ [22]. For example, the thermodynamic entropy density of Equation (4) can be shown to be [46]:

$$\begin{array}{cc}\hfill h& =\mathrm{H}\left[{\mathit{\sigma }}_i|\overleftarrow{{\mathit{\sigma }}_i}\right]\hfill \\ & =-\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right){\log }_{2}p\left({\sigma }_i|\overleftarrow{{\sigma }_i}\right)\hfill \end{array}$$

(13)

$$\begin{array}{c}=\phantom{-}\sum _{\sigma \in \mathit{\sigma }}p\left(\sigma \right)h_i\left(\sigma \right)\hfill \end{array}$$

(14)

where $h_i\left(\sigma \right)=-{\log }_{2}p\left({\sigma }_i|\overleftarrow{{\sigma }_i}\right)$ and $\overleftarrow{{\sigma }_i}$ is the set of spins whose indices are “lexicographically” less than i. $h_i\left(\sigma \right)$ is a spatially local or pointwise measure and quantifies how much the individual spin ${\sigma }_i$ contributes to the global entropy. We can more directly understand how local motifs contribute to the average entropy by plotting the local measures at each site within a given lattice configuration. This approach roughly parallels that in, e.g., [47], but here we employ different information measures and do not assume directionality, since we average the pointwise values resulting from each of the possible orientations centered at the given spin.

In the following, we define a cluster to be a maximal set of contiguous spins oriented identically. The edge of a cluster is that set of spins in a cluster with at least one neighbor not in the cluster. The remaining spins in the cluster are its bulk. An isolated spin is a cluster consisting of a single spin.

Figure 6 and Figure 7 show the results of the motif entropy analysis for 1D and 2D lattices. In all cases, the spatial average of the displayed quantities corresponds (in the thermodynamic limit) with the values reported in Figure 3 and Figure 5.

There are several immediate similarities that allow for structural interpretation. For example, motif q is positive (red) within the bulk of a cluster and negative (blue) on its edges. Highlighting opposite features, motif r is negligible within the bulk, but positive along edges, particularly corners and isolated spins. The motif bound entropy b, however, is more nuanced in its behavior. Considering Figure 6, at lower temperatures it is sensitive to cluster edges more so than cluster bulk. At higher temperatures, however, this relationship flips and it is more sensitive to the bulk. For all temperatures, it is negative for isolated spins—clusters of size 1.

In contrast, we see that at not-too-high temperatures motif b is largely sensitive to cluster boundaries. This leads one to speculate that, as in [6], b’s maximization in the disordered phase is due to complex interactions between cluster sizes and their surface area. This also sheds light on why b does in fact peak at $T_c$ on the Bethe lattice. On the square lattice, on the one hand, cluster boundaries have a tendency to smooth due to the presence of correlations flowing along short loops in the lattice topology that tend to align spins on a boundary. On the other, the Bethe lattice has no such loops, and so there is no pressure to reduce surface area. This suggests that b generically peaks at $T_c$ for systems where boundaries are not constrained by system energetics and that the energetic smoothing of cluster boundaries drives b to peak in the more “textured” disordered phase.

Interestingly, for general lattices such as the Bethe lattice, but especially for random graph topologies, local (conditional) forms of the thermodynamic entropy density (analogs of Equation (14)) and other motif measures are unknown and may simply not exist. While techniques, such as that found in [48], exist for estimating thermodynamic entropy density for a ferromagnetic Ising model on an arbitrary graph, the interpretation of a thermodynamic entropy density in such systems is problematic as each spin site may have differing connectivity and therefore play differing roles in the overall lattice dynamics. Therefore, while global averages may exist for arbitrary topologies and may even be tractably estimated, their structural meaning is vastly more challenging. This meaning is critical to understanding thermodynamic and statistical mechanical properties of such irregular systems and, more generally, correlations and information measures on network dynamical systems. If local information estimates, such as those used for the motif entropy analyses above, existed for such systems then studies can be undertaken that determine which nodes in a lattice or network contribute most to collective behavior.

5.2. Actively Storing Information versus Communicating It

Assuming Glauber dynamics for the nearest-neighbor 2D Ising system, Barnett et al. [6] computed a global transfer entropy [16]—the average information communicated between the entire lattice and a single spin. They found that it peaks within the disordered phase at a temperature $T_{{\mathcal{T}}_{gl}}\approx 2.354$ J/k${}_{\mathrm{B}}$. Although there are serious concerns with how the transfer entropy conflates two-spin (dyadic) and multispin (polyadic) dependencies [49], those concerns are not relevant in these Ising systems due to their known, dyadic Hamiltonian.

To compare our measures with theirs, we examine the active storage of randomly generated information in spatial patterns, as measured by the ratio of $b/h$—how much of the localized randomness (h) is converted by a system into spin-spin (spatially) shared information (b). Figure 8 plots the ratio as a function of temperature for the 2D Ising system: the ratio peaks in the disordered phase and, surprisingly, it appears to do so at $T_{{\mathcal{T}}_{gl}}$. This implies that there is a strong connection between b (storing thermal fluctuations as spatial correlation) and the potential for information communication in a system. Hopefully, the result suggests that Barnett et al.’s $T_{{\mathcal{T}}_{gl}}$ is not a result specific only to their choice of Glauber dynamics, but rather an intrinsic property of the 2D Ising model. It remains to be seen why spatially shared information is maximized within the disordered phase of the 2D Ising system, though.

6. Conclusions

As noted at the beginning, even the earliest debates over entropy’s statistical foundation turned on contrasting its thermodynamic components—Boltzmann’s isolated-spin entropy versus the Gibbs global entropy. From a modern perspective their difference, well known to Gibbs, is generalized mutual information that measures the degree of individual-component independence, now called the total correlation. From this, we showed that the Gibbs thermodynamic entropy density naturally decomposes further into two functionally distinct informational components: one quantifying independence among constituent spins and the other, dependence. The one quantifying dependence, the bound entropy b, captures collective behavior by expressing how much of the thermodynamic entropy density is locally shared. We then demonstrated the behavior of the bound entropy and related quantities for the nearest-neighbor ferromagnetic Ising model on a variety of lattices. We found that it tends to a maximum at intermediate temperatures, though not always at the magnetic phase transition. Our analyses support our earlier hypothesis [50] that q, as the dominant component of the persistent mutual information [51], should generically be maximized at critical points. Though not detailed here, this observation holds in three dimensions and in simulations of the Potts model [52] with the number of states $s=3$ on lattices with 1, 2, and 3 dimensions.

This brief phenomenological study of thermodynamic entropy components served to give a physical grounding for information measures and what they reveal in spin systems on various lattice topologies. The results suggest many avenues of potential research. One topic to explore is the behavior of b in the Potts model, which switches from exhibiting a second-order phase transition in the magnetization to a first-order transition when the number of spin states exceeds four. Another setting of particular interest is to study the behavior of b in a frustrated system, such as the antiferromagnetic Ising model on a triangular lattice, paralleling the work in [53]. Finally, as alluded to above, although beyond the present scope, a next step is to consider informational measures for spins on arbitrary graphs, with the goal of providing insight into the roles that different nodes play in information processing and storage in complex dynamical networks.