3.1. Overview of the Multimodal Experiments
This section provides a comprehensive description of the experimental results obtained with six modalities, using the 128-eletrode EEG array placed over the scalp of the participants. Details of measured EEG signals, their analytic amplitudes and phases, as well as the derived pragmatic information, are given.
Figure 4 illustrates the analytic amplitude (
AA(t)), signal amplitude (
SA(t)), and analytic frequency (
IF(t)) over a 3.5 s observation period, for frequency band (Theta), for Participant 7. Specifically,
Figure 4a,b display the
AA(t) and
SA(t) signals for all 128 electrodes, while
Figure 4c,d show the average 〈
AA(t)〉 and 〈
SA(t)〉 signals calculated across the EEG array, together with the shaded upper and lower limits, respectively. Over the ~2000 ms to 2300 ms time window, we observe significant peaks in
AA(t) and 〈
AA(t)〉, indicating that the power of the signal, and therefore its energy, have increased in that time window.
Figure 4e displays the analytic frequency
IF(t) for the 128 EEG electrodes. The curves show many spikes, which indicate rapid changes in the analytic frequency at various time instances. For our future discussions, it is important to point out that the
IF(t) values are consistently low during time interval ~2000 ms and 2300 ms, showing that the signals maintain a rather constant phase during this period, leading to only small variations in the analytic frequency. This means that the signals are highly synchronized during this time window, exactly where
AA(t) had a significant peak, indicating that the power of the signal, and therefore the energy, increased at that time of high synchronization. This observation has important implications for the behavior of the pragmatic information index, as elaborated later in
Figure 5.
Figure 5 provides a graphical illustration of the evaluation of the pragmatic information index, H
e(t), as described above. In order to characterize where and when knowledge and meanings are created to conduct an intentional action, the pragmatic information index, H
e(t), was evaluated during the 3.5 s response period depicted in
Figure 5. We can appreciate the significant changes in pragmatic information, H
e(t), as a result of computing the ratio
where D
e(t) and
AA2(t) show relevant features shaping H
e(t) dynamics. We display two versions of PI, namely H
e(t)
1 and H
e(t)
2, where D
e(t) is based on amplitude and phase, respectively [
33].
We produced similar plots for all participants, for all modalities, for all stimuli, in every frequency band of interest, as shown in
Table 1. From this first qualitative analysis and based on Freeman’s findings [
45], we expect that the most relevant frequency bands with potential impact on the creation of knowledge and meaning are High Gamma (35–48 Hz), and Alpha (8–12 Hz), which is proposed to be acting as a gating band. This is in line with the proposal of Freeman and Kozma and others, suggesting a Gamma–Alpha and Gamma–Theta link [
51,
52,
53,
54].
Figure 6 illustrates the EEG in subplots (a)–(b), and an example of the distribution of the pragmatic information index across various brain areas, during a 3.5 s experimental period. The PI plot corresponds to participant 7 (P7), in the modality WORDS, on stimulus 9 (S9), in the Theta band. This example illustrates that the creation of knowledge and meaning is manifested in different combinations of brain areas and frequency bands. These computations for H
e(t) use a specific implementation of the PI evaluation, where we have performed the evaluation in time windows of length Δt as H
e(t − Δt).
3.2. Evaluation of Pragmatic Information Variables and Parameters
Next, we introduce a set of variables to characterize the dynamics of H
e(t) during the time period necessary to process a particular stimulus [
37], which are given as follows:
NPS: the total number of peaks per unit time (s);
TBP: the time spent between peaks, describing the quiet periods;
TOP: the time describing the duration of peaks, measuring intensive periods;
QPT: the total quiet processing time;
IPT: the total intensive processing time.
The number of peaks will be described by the following aggregate quantity
, where the indices stand for stimulus (
s), participant (
p), modality (
m), frequency band (
b)
, and electrode or channel (
e), where p, m, b, e, and s can take values as shown in
Table 2.
Table 2 shows that there are different numbers of stimuli in different experiments, as specified by
in
Table 3.
Integrating over all channels,
Nch = 128, we obtain the total number of peaks in an experiment with the following parameters: stimulus (
s), participant (
p), modality (
m), and frequency band (
b):
From
, NPS can be derived as the number of peaks per second, for given stimulus (
s), participant (
p), modality (
m), and frequency band (
b), based on the total duration:
Considering all the combinations of frequency bands (b) and modalities (m), there are 36 possibilities; we can evaluate the mean values over all stimuli,
as follows:
From
we can derive (a) the number of peaks/second for participant
p, in modality
m for all frequency bands as
, and (b) the number of peaks/second for participant
p, in frequency band
b for all modalities as,
as follows:
Similarly, we compute intervals of confidence based on the standard deviation,
for the values of
, as well as intervals of confidence based on the standard deviation,
for the values of
, as follows:
Similar formulas can be derived over brain areas, when the number of electrodes over the specific area are used. For example, the following expressions are valid for the prefrontal region PF:
Considering all areas, we write:
Following the approach described above, formulas can be derived for the other statistical quantities described earlier, such as TBP, TOP, QPT, and IPT. When needed, the corresponding formulas will be provided in the section of analysis.
When characterizing He(t), the following specifications are used:
A threshold signal value is determined heuristically; a specific threshold is set at 0.1 in these studies.
Time of peak (TOP) for each significant peak is defined as the duration of the signal continuously above such threshold.
Time between peaks (TBP) is defined as the duration of the signal continuously below the threshold.
The selection of a minimum TBP, in this study ≤11 ms, where two consecutive peaks should be taken and joined as one peak.
Isolated peaks that are too short, here shorter than ≤50 ms, are rejected.
We start by setting a threshold value for the pragmatic information index [
29]. Most of the time PI has sub-threshold values, but at some moments it may cross to super-threshold values. Such crossing events are used as markers for the postulated emergence of meaning related to the specific sensory stimuli in the context of the participants’ physical and cognitive state, and their intentions and desired actions. These are precisely the moments that may be considered to consume the most energy. Details of the energy consumption during the cognitive cycle require further studies and are outside the scope of this work; see, e.g., [
29].
Figure 7 shows an example with a PI threshold set at 0.1, for frequency band (a) Alpha (b = 2) and (b) High Gamma (b = 6); these examples correspond to the fourth stimulus (s = 4) in the WORDS modality (m = 2). We observe by visual inspection 10 peaks in the case of the Alpha band, and 11 peaks for the H-Gamma band. Using the compact index notation, this is expressed as
and
= 11. Several peaks may be merged when counting their numbers if they are very close to each other, or rejected when they are individually very short. Details of the practical implementations are given in the next section.
3.4. Results Obtained with TBP and TOP
Another set of relevant variables is described by TBP, which is the time between peaks, and it is a measure of quiet processing. We also introduce results for TOP, which is the time of peaks, and it corresponds to time periods with intensive processing; see
Figure 10 for illustrations.
TOP and TBP are used to describe the intensive and the quiet processing periods, respectively. At this time, no systematic hypothesis tests were performed regarding TOP and TBP values across modalities. Nonetheless, several observations are useful. For example, when analyzing the cumulative distributions for TOP, as shown in
Figure 11, for each modality, in the Alpha (a) and H-Gamma (b) frequency band for all participants, we observe that the TOP values for the modalities of SENT and WORDS seem to behave differently from other modalities, and appear to be greater than the values for the modalities of MED and IMG for the Alpha band. Regarding the H-Gamma frequency band, the values for the modalities of MED and IMG seem to be significantly lower only in relation to the SENT modality, while the modality WORDS behaves more similarly to the modalities of MM and VDO.
The cumulative distribution of the Alpha frequency band (a) may indicate a bimodal or possibly multimodal distribution, which could be due to the fact that some stimuli have longer duration than others for different participants in a particular modality, meaning that some participants may be slower to resolve a task than others. This bimodal or multimodal behavior seems to be absent for most of the modalities in the H-Gamma frequency band. The bimodal or multimodal behavior observed in the Alpha for TOP values is also observed in each frequency band for most modalities, and this could also be associated with the behavior of different brain regions. The study of the behavior of different regions will be the objective of future studies. In general, the mean TOP for each participant shows similar values between the Alpha (c) and the H-Gamma (d) frequency bands, with slightly more variability in the H-Gamma frequency band.
The cumulative distributions for TBP are shown in the lower row (e–h) in
Figure 11. Qualitative observations indicate that the distributions of TBP in all modalities for all participants seem to exhibit smaller values for H-Gamma (f) than for Alpha (e), with the exception of the modality MED. In general, several of the mean TBP values are higher for the Alpha (g) than for the H-Gamma (h) frequency band, except for some exceptions like participants 19 and 20, for example. Note that the H-Gamma frequency band displays more variability.
Finally, in
Figure 12 we summarize the above analysis by showing the mean TOP (a,b) and TBP (c,d) for each modality, for all participants, in the Alpha (a,c) and H-Gamma (b,d) frequency bands. We can observe that, for TOP, the values are smaller for the modalities of MED and IMG, while SENT and WORDS display the largest values for both the Alpha and the H-Gamma frequency bands. For TBP, we find a more diverse range of values across modalities. IMG displays large values for both frequency bands. However, MED, and SENT and VDO, have the lowest values for the Alpha frequency band, and it is MM and SENT that display the smallest values for the H-Gamma frequency band. Again, these qualitative visual observations must be evaluated using systematic hypothesis tests in the future.
Regarding the mean NPS, some interesting observations can be made when contrasting the Alpha and H-Gamma frequency bands, as shown in
Figure 13 and
Table 4.
Table 4 uses data from [
37], but the modalities have been regrouped, to emphasize the significance of the observations on low NPS values in IMG, MEG, and MM modalities.
In the Alpha frequency band, we can clearly identify the two (2) minimum mean NPS values associated with the modalities IMG and MM, while for the H-Gamma frequency band, the two (2) minimum values are identified for the modalities MED and IMG. The common minimum of the two frequency bands is found in the modality IMG. In the IMG task, the participants are less in control of the process of knowledge creation, until the moment of sudden insight, the “aha” moment, when they identify the meaning of the earlier hidden image. This, it seems to us, would require less total intensive processing time, which is a combination of the mean TOP and the mean NPS, as previously explained in
Figure 10, if we add all the TOP values (red) of each peak above the threshold.
We expected that the TOP values for the modality MED would be smaller than for the rest of the modalities, since meditation practices are known to reduce information processing in the brain, and therefore the creation of knowledge and meaning would also be reduced, more likely leading to less NPS and shorter TOP. What is interesting, however, is that we observe a similar behavior for IMG, which suggests that between the sudden discoveries of meaningful images, there is little processing of semantic information, which coincides with longer periods of less intensive processing, as seen in TBP values for both the Alpha and H-Gamma frequency bands.
Based on the parameters (mean and standard deviation) used to compute the confidence intervals displayed in
Table 4, we performed a set of unequal variance
t-tests of hypothesis, where H0: μ1 = μ2 was tested for means between all modalities in the Alpha and H-Gamma bands. The results are displayed in
Table 5. We also tested for means between the Alpha and H-Gamma bands for each modality. It is important to note that we accept H0 for
p-values greater than or equal 0.05, while we reject it for values smaller than 0.05. An accept is labeled as {0} and a reject as {1}, and a {0*} indicates a just accepted with a value marginally greater than 0.05, like, for example, 0.06.
From
Table 5, it is clear that the modality MED shows a significant number of rejects (1) and just accept (0*), particularly for the H-Gamma band, when compared with the rest of the modalities. For example, we observe that the NPS means for the modality WORDS show the same mean, statistically speaking, when compared to the rest of the modalities in both bands, except for modality MED in the H-Gamma band.
When taking the results of the Alpha and H-Gamma bands together, we reject H0 when comparing the mean NPS values of modality MED with the rest of the modalities, except for the modality VDO where H0 is accepted (0) and MM is just accepted (0*) when compared with MED. In the case of modality WORDS, H0 is accepted (0) for all but modality MED. For modality MM, H0 is accepted (0) when compared with most modalities and only rejected (1) for modalities SENT and VDO in the Alpha band. Additionally, H0 is rejected when contrasting the modalities of IMG with SENT and VDO for the Alpha band.
In H-Gamma, all other combinations of means hypothesis testing return a value of 0, accepting that means between modalities are equal, except when contrasted with modality MED.
When we test H0: μ1 = μ2 for a comparison between the Alpha and H-Gamma bands, we accept H0 for p-values greater than or equal 0.05, otherwise H0 is rejected. It is important to note that we rejected H0 for mean values of NPS for all modalities except for modality MED.
In order to have an idea of the percentage of time that the brain is engaged in intensive processing, as well as the percentage of time that it is operating in quiet processing mode, we computed a pair of variables, as follows.
Let us define (1)
as the duration of stimuli s, for modality m, for participant p, in frequency band b; (2)
as the total intensive processing time in stimuli s, for modality m, for participant p, in frequency band b; and (3)
as the number of peaks per stimuli s, for modality m, for participant p, for a particular frequency band b*. It follows that:
where
is the percentage of intensive processing time in one stimulus, for modality m, participant p in a particular frequency band b*.
We then compute a vector where each particular becomes an element of this vector, for modality m, in a particular frequency band b*, and we use the values in this vector to compute and plot the cumulative probability distributions of for each modality m.
Similarly, we compute the percentage of quiet processing time for each modality m, for a particular frequency band b*, as vector to produce further statistical analysis.
In
Figure 14a,b,e,f we show the empirical cumulative probability distributions for both
and
for b* = [Alpha, H-Gamma], for each modality m. As expected, we can observe almost a mirror image of the cumulative probability distributions
(a, b) and
(e, f) for all modalities in both bands.
We also observe that the values for modality MED and IMG are smaller than the values for the modality of VDO and this modality in turn shows values smaller than modalities SENT and WORDS for both bands. The modality MED behaves like MM in the Alpha frequency band (a), while the modality of MED follows a closer behavior to the modality of IMG in the H-Gamma (b). It is also clear that the modalities are more clustered in Alpha than H-Gamma, making H-Gamma a better band to discriminate between modalities.
We also observe in
Figure 14c,d,g for the
values for b* = [L-Beta, H-Beta, L-Gamma] how clearly the cumulative probability distribution shows the difference between modalities, where the values for MED < IMG < MM < WORDS < VDO < SENT are smaller, in that order, with some small variability for L-Beta. These frequency bands meet the expectation that the modality MED would show the lowest processing times, since participants are expected to be in a calmer and quieter information-processing state of mind.
In
Figure 15, we show the mean and errors for the values of the
, for all modalities and frequency bands, and identify an order from the least to the most intensive processing time modalities, as follows: MED, {MM, IMG}, VDO, {SENT, WORDS}.
Here we identify the smallest and largest PIPT values per modality and per band as displayed in
Table 6.
From
Table 6, we observe how MED displays the lowest values of PIPT for most bands, apart from Alpha. This could be because in MED, Alpha is the most dominant frequency and less entropic band for most participants, as shown in [
36]. Note that modalities MED, IMG, and MM show the smallest PIPT values for all frequency bands, in contrast with the modalities of WORDS, SENT, and VDO, presumably indicating that the first group is less time and energy consuming out of all modalities.
Finally, we present in
Figure 16 the big picture about TBP, TOP, and NPS and their relationship to one another for all participants, for all modalities, in the Alpha and H-Gamma frequency bands. As important as they might be in giving us a fuller picture of brain dynamics, we leave the study of other frequency bands for future research since (a) we have still to gain more insight into the Alpha–H-Gamma linkage and (b) because it would make this paper exceed the acceptable length.
As we can appreciate in
Figure 16a,d, generally speaking, the mean values of TOP drop in H-Gamma when compared to those in Alpha for all modalities except for VDO, while the values for NPS increase significantly for all modalities (except for MED showing a slight increase) in H-Gamma when compared to the Alpha frequency band.
On the other hand, when we observe the mean values of TBP (b, e) they show a diverse behavior between modalities. Modality MED shows a significant increase and clusters together with modality IMG for H-Gamma when compared to the Alpha frequency band. However, modalities SENT, MM, VDO, and WORDS show a large decrease in H-Gamma when compared to the Alpha frequency band.
In
Figure 16c,f we note an inverse relationship between the mean TOP and the mean TBP values, with a larger variability in Alpha, where we also see three (3) clusters of modalities as follows: {MED, IMG}, {MM, WORDS, VDO} and {SENT}.
Based on the above, we constructed
Table 7 in order to have a concise picture of the behavior of the different modalities in the Alpha and H-Gamma frequency bands in terms of mean TOP (MTOP), mean TBP (MTBP), and NPS. This allows us to qualitatively characterize these variables, from the smallest to the largest values, that influence the percentage of intensive and quite processing, PIPT, and PQPT respectively.
Taken together, the Alpha and H-Gamma frequency bands show MTOP values displaying an increasing order from the modalities of MED and IMG (1st), VDO (2nd), MM (3rd), WORDS (4th), and SENT (5th). However, when we look at the MTBP values, the different modalities behave in a different manner when contrasting Alpha and H-Gamma, as follows: (a) SENT (1st), VDO (2nd), MED (3rd), MM (4th), WORDS (5th), and IMG (6th) for Alpha, and (b) SENT (1st), MM (2nd), VDO (3rd), WORDS (4th), MED (5th), and IMG (6th) for H-Gamma.
Note that modalities MM and MED show the larger changes in mean TBP values when contrasting their values for the Alpha and H-Gamma frequency bands.
For the NPS, the order is as follows: (a) IMG, MM, WORDS, MED, VDO, and SENT for the Alpha frequency band; and (b) MED, IMG, VDO, MM, SENT, and WORDS for the H-Gamma frequency band. Note that both frequency bands show very distinct orders.
We observe a general tendency of inverse correlation between the variables MTBP and NPS, except for some exceptions in the H-Gamma band, particularly for modality WORDS.
These variables describe very particular aspects of brain dynamics related to velocity of processing per stimulus and the mean intensive processing time per stimulus that can be derived from in Equation (21), which also can be estimated as the mean TOP multiplied by the mean NP per stimulus for all participants, for all modalities, for each band. This analysis could be extended to differentiate between experienced meditators and participants with little or no meditation experience. This issue is not addressed in the present study.
Note that this analysis is based on mean values. In the future, this study should be complemented following the results obtained in
Table 5, and further statistical analysis, where a test is performed for H0: μ1 < μ2 or H0: μ1 > μ2, for NPS and other variables.