Unraveling the Dynamics of Mental and Visuospatial Workload in Virtual Reality Environments

Abstract

Mental workload, visuospatial processes and autonomic nervous system (ANS) activity are highly intertwined phenomena crucial for achieving optimal performance and improved mental health. Virtual reality (VR) serves as an effective tool for creating variety of controlled environments to better probe these features. This study investigates the relationship between mental and visuospatial workload, physiological arousal, and performance during a high-demand task in a VR environment. We utilized a modified version of the popular computer game TETRIS as the task, involving 25 participants, and employed a physiological computing VR headset that simultaneously records multimodal physiological data. Our findings indicate a broadband increase in EEG power just prior to a helper event, followed by a spike of visuospatial engagement (parietal alpha and beta 0-1-3 s) occurring concurrently with a decrease in mental workload (frontal theta 2–4 s), and subsequent decreases in visuospatial engagement (parietal theta at 14 s) and physiological arousal (HRV at 20 s). Regression analysis indicated that the subjective relief and helpfulness of the helper intervention was primarily driven by a decrease in physiological arousal and an increase in visuospatial engagement. These findings highlight the importance of multimodal physiological recording in rich environments, such as real world scenarios and VR, to understand the interplay between the various physiological responses involved in mental and visuospatial workload.

1. Introduction

Mental and visuospatial workloads are among the key drivers in human cognitive processes [1,2,3,4]. Studies in this field have particularly focused on the prefrontal and parietal cortices. The prefrontal cortex plays important roles in executive functions, such as the capacity for self-control and short and long-term planning, which are widely regarded as among the most crucial aspects of the human mind [5,6]. The parietal cortex, on the other hand, is regarded to be critically important in visuospatial information processing and working memory, and in the control of behavioral responses [7,8,9]. Interconnected systems of visuospatial working memory and attention are also critical for human interaction with rich stimulus environments. These systems are mediated by the frontoparietal network, though the degree to which they are dissociable is debated, with both working memory and attention is associated with activity in both the prefrontal cortex and the posterior parietal [10]. In addition, the ANS and the interaction between its two parts, the sympathetic and parasympathetic systems, are involved in top-down and bottom-up directions of mental workload [11,12]. Maintaining a healthy equilibrium in the operation of these systems can help people achieve optimal performance, better mental health, and improved well-being [13,14,15].

Studies focused on detecting and quantifying mental workload have increasingly used physiological measures, which have various advantages over using only self-reported measures, as the latter are not optimal in continuous assessment and are prone to retrospective reconstruction of the perceived mental workload [16]. Common physiological measures of workload and engagement include electroencephalography (EEG) recordings from the parietal and prefrontal regions of the brain [17,18] and measures of cardiac activity reflecting the state of ANS [19]. Although conflicting reports exist (for a recent review, see [20]), most studied EEG indices include increased frontal theta [21] and beta [22], increased parietal beta [23] and theta [24], and decreased parietal alpha [25,26] band powers. As a complementary modality to EEG, functional Near-Infrared Spectroscopy (fNIRS) studies also have shown similar findings, with increased prefrontal activation during elevated mental workload [27]. Cardiac indices for increased mental workload and physiological arousal include increased heart rate (HR) and decreased heart rate variability (HRV) [28,29]. As discussed in previous research, while these measures provide valuable insights on their own, using a multimodal approach can provide more robust representations of mental workload, as each signal modality represents different facets of mental workload [30,31].

While measuring physiological markers is critical in mental workload studies, the experimental environment and behavioral response elicited in participants are equally important. In behavioural sciences, researchers have studied virtual reality (VR) experiment environments as both an exciting focus of study [32,33] and a tool for designing realistic experiments [34,35]. VR is a valuable platform for studying participants’ behaviors as it is easy to create any number of environments. These environments can be designed to be safe, controllable, replicable, and accessible to participants from a variety of backgrounds and with a variety of needs [36,37]. Most importantly, VR is capable of efficiently generating realistic new environments that are a powerful tool for psychological researchers [38,39,40]. There have been important advancements in using VR as an experimental platform to study mental workload. Gupta et al. [41] tracked multiple signal modalities (EEG, HRV and Electrodermal Activity (EDA)) and subjective questionnaires to investigate the trust level towards an auditory virtual agent under different mental workload levels. In another study, Zhang et al. [42] used multimodal physiological data (EEG, EDA, Electromyography (EMG) and skin temperature) in a VR-based driving system to track the mental workload of users with Autism Spectrum Disorder, aiming to build a system that facilitates learning by adjusting task difficulty. Utilizing real-time EEG data in a VR environment, Dey et al. [43] presented an adaptive training system that can adjust task difficulty to an optimal challenging level to facilitate learning. The reader is further referred to the recent review by Hadjiaros et al. [44]. While these recent studies provide valuable insights in terms of mental workload corresponding to the prefrontal cortex activity, little attention has been given to visuospatial processing in the literature. Furthermore, the temporal dynamics of the central and autonomic nervous system activity in the context of mental workload in VR environments requires further research.

The study presented here aims to explore the relationship between visual-spatial cognitive demands, prefrontal cognitive workload, and the autonomic nervous system during a high-demand task in virtual reality. The popular computer game TETRIS was modified and used as the task in the experiment, which involved 25 participants. We used Galea [45], a physiological computing VR headset that simultaneously records physiological data (EEG, EDA, EMG, and PPG). We study the dynamics of the autonomic and central nervous systems, with a focus on the activity of visuospatial workload in the parietal region and the balance between the sympathetic and parasympathetic responses. To better probe these temporal characteristics, we have added a new feature to the classic game of TETRIS called “helper event” that introduces a swift change in task difficulty. Once the stack of pieces reaches 60% of the height of the playing field, a ball-shaped helper piece appears. It can be played like any other tetromino piece, and when placed, it clears the four rows of squares underneath it and sets the game level to 5, slowing down the speed at which the pieces fall. The findings of the study highlight the importance of cognitive and visuospatial workload in subjective experience and tension relief, as well as the significance of maintaining optimum mental workload for maximizing performance and balanced autonomic nervous system activity. This exploratory study contributes to the growing understanding of the relationship between cognitive load, visuospatial engagement, the autonomic nervous system, and performance.

The study hypothesizes a decrease in cognitive load, visuospatial engagement, and physiological arousal following the helper event, which suddenly makes the game easier. The study also hypothesizes that the physiological signals examined would be correlated with subjective questionnaire responses about the impact of the helper event.

Overall, this study aims to explore the relationship between the central and autonomic nervous system phenomena and cognitive state, particularly in the context of the helper clear event in a highly demanding scenario.

2. Methods

2.1. Participants

A total of 34 participants (15 male, 16 female, three non-binary; ages 18 to 44) participated in this study. Inclusion criteria were as follows: normal or corrected-to-normal vision; no known history of epileptic seizures, migraines, or vestibular dysfunction; no known history of claustrophobic events when using VR headsets; ability to set and follow instructions for 30 min. Participants had varying levels of familiarity with VR and the game TETRIS. The data was collected from 34 total participants, but 5 subjects were excluded based on performance criteria, 3 were excluded on the basis of channel data quality and 1 subject was excluded based on outlier timing. Ultimately, 25 subjects were available for data analysis.

2.2. Experimental Protocol

The experiment involved playing a modified version of the computer game TETRIS^TM in VR. The game implementation adhered to the original source code and was implemented in Unity3D (2020.3.11f1; Unity Software, Inc., San Fransico, CA, USA). The TETRIS game involves tetrominoes, which are geometric shapes made of four squares linked edge to edge in various arrangements, that descend one at a time vertically from the top of the computer screen. While a tetromino piece falls, the player can manipulate the piece (moving it left, right and rotating). The goal of TETRIS is to arrange the pieces so that complete horizontal rows are formed. When this is done, the squares in that row vanish, and players receive points.

In this experiment, participants used the provided controller to manipulate the pieces. The game also incremented in level every 10 s of the game. Each level would have a set speed at which the tetromino pieces fall, and this speed would increase logarithmically with the level of the game. This was done to produce an increase of difficulty throughout the game.

There were two sessions of the experiment: intervention and control. Both times, the TETRIS game followed the general rules described above. The intervention session had one additional helper condition. When the stack of pieces reach 60% of the height of the playing field, a ball shaped helper condition appears. This helper piece can be played like any other tetromino piece, and once it is placed, it immediately clears the four rows of squares underneath it and sets the game level to 5, decreasing the speed at which the pieces fall. To reduce possible confusion, researchers explained to each participant before beginning the intervention session that there would be a new piece appearing at some point during the game, that this place can be manipulated like any other tetromino, and that this piece will clear four rows once it is placed. The control condition did not include the helper intervention and had no other variations from the regular rules of TETRIS.

Each participant completed both the intervention and control conditions. The order in which each participant completed the intervention and control conditions was randomized to minimize the effects of novelty from the controllers and test environment on cognitive load from the experimental conditions.

2.3. Procedure

For both intervention and control sessions, the experiment had the following procedure as outlined in Figure 1, right diagram. First, participants filled out two pre-experiment questionnaires. Second, participants were met with a headset fitting and signal impedance check. Third, a 50-s calibration occurred to remove motion artifacts in subsequent data using Neuropype (see Data Preprocessing). Fourth, there was a second, 30-s-long calibration with eyes closed, during which baseline alpha signals were collected. Fifth, after calibration, participants played a tutorial of the TETRIS game for 30 s. This was done to help reduce the effects that new controllers and unfamiliarity with the game may have on mental workload. Sixth, participants played a game of TETRIS with either the intervention or the control condition. Seventh, after the game, participants were asked to fill out a post-experiment questionnaire. Lastly, participants were debriefed on the experiment and could provide feedback if needed.

2.4. Apparatus

The Galea headset was used to present VR content and to collect physiological data [45]. The VR content was presented on the Valve Index Headset (Valve Corporation, Bellevue, WA, USA). Participants used the Valvue Index Controllers. Electroencephalography (EEG) was collected with a sampling rate of 250 Hz from dry electrodes placed according to the international 10–20 system at Fp1, Fp2, Fz, CPz, POz, PO3, PO4, Oz, O1, and O2. Headset fitting sessions were conducted prior to the start of the experiment to ensure that impedance from EEG electrodes were under 10 uVrms. Photoplethysmography (PPG) was collected using a particle and proximity sensor on the forehead

The Galea headset generates a stream of physiological measurements. Simultaneously, Unity3D creates a stream on which event markers from game events are written. In particular, the moments of interest were (1) start and end of calibration, (2) start and end of tutorial, (3) start and end of game, (4) when stack of tetromino pieces reached 60% of height of playing field. Additional event markers were sent during the intervention condition when the helper piece appeared at the top of the screen and when the helper piece cleared rows. The streams from Galea and Unity3D were merged using Lab Streaming Layer (LSL) [46].

3. Data Preprocessing

3.1. EEG

Collected EEG data were preprocessed by Neuropype (Syntrogi Inc., La Jolla, CA, USA). The stream generated by the Galea headset was inputted to the Neuropype pipeline with deterministic timing option to ensure replicability of the offline data stream. Then, a 1–35 Hz finite impulse response (FIR) bandpass filter was applied. For artifact removal, data from the 50-s calibration was taken as a baseline, and artifacts were removed with a noise amplitude threshold of 4 standard deviations and a sliding window of 1 s. This threshold was determined for our settings by visually inspecting the resultant timeseries for noise, such as muscle artifacts, eye blinks and eye movements. Resultant time series was then exported for further analysis. Additionally, artifact regression node was utilized for EEG data with heart rate pulsation artifacts. Through this node, PPG channels were labeled as artifact channels, as they had similar signal waveforms with pulsation artifacts, and the heart beat artifacts were subtracted using recursive least squares method.

3.2. PPG

Collected PPG data were preprocessed using “peakdet”, a Python-based peak detection toolbox [47]. First, a 0.7–2.0 Hz bandpass filter was applied to the PPG channels, and an aggressive automatic peak detection was run on the data. Then, the detected peaks were manually inspected, so that faulty peaks are deselected and undetected peaks are selected.

4. Measurements

4.1. EEG

For each subject and for each of the target electrodes, power was calculated for each of the Alpha (8–13 Hz), Beta (15–30 Hz), and Theta (4–8 Hz) frequency bands for each second using 75% overlapping 4 s windows. The MATLAB ‘bandpower’ function was used to calculate the average power in the target frequency range (R2022a, The MathWorks Inc., Natick, MA, USA).

4.2. PPG

From the pre-processed PPG data, beats per minute (BPM) was calculated via a simple average for each second using 90% overlapping 10-s sliding windows. Heart rate variability (HRV) was measured using Root Mean Square of Successive Differences (RMSSD), again for each second using 90% overlapping 10-s sliding windows. RMSSD was utilized to account for the relatively short time window [48].

4.3. Baseline

The last s seconds of data from the beginning of the game to the first helper event (stack reached 60% of the height of the playing field) was taken, where s is the smallest number of available seconds across all subjects. From this data, baseline measurements of the means and standard deviation were taken for both power and PPG data for each subject and each electrode.

5. Analysis

5.1. Normalization

Given that power values are bounded (at 0) the alpha power data at any given second was highly non-normal, with a rightward skew. As such a log transform was performed to normalize the data. Thus for each data point for each subject the log of the alpha power was taken. This allows us to use statistics that assume normally distributed data for subsequent analysis.

$$x_{norm}=log\left(x\right)$$

(1)

5.2. Standardization

Given that the channel signal for each electrode/subject, as well as the corresponding power signals, vary based on factors beyond differences in underlying neural activity (e.g., signal impedance) the power signal was standardized for each electrode for each subject. Such standardization are often called “z-scores”.

$$z={\displaystyle \frac{(x-{\overline{x}}_{base})}{s_{base}}}$$

(2)

The standardized data was used for subsequent analysis. This effectively weights the contribution of each electrode for each subject’s power signal equally, preventing large variations due to signal quality across subjects/electrodes from distorting the group level data.

5.3. Time-Series Analysis

Power and PPG data were calculated, then examined over time at the group level. Specifically, once all subjects’ data were standardized and normalized, the data were centered over the game event of the helper piece dropping and clearing rows. This was done by simply aligning all time series for each helper event. Then, data were collapsed across electrodes (CAE) per region by simple averages of prefrontal and midline frontal (Fp1, Fp2, Fz), and parieto-occipital (PO3, PO4, POz) electrodes. Subsequent analysis was done for each second within the analysis window starting at the helper clear event to the end of the game, which varied for each subject. For each second in the analysis window, a group level one sample t-test was done to detect deviations from the baseline. A p-value cutoff of 0.05 was used to determine a significant event.

5.4. Linear Regression Models

A variety of regression models were examined in this work, both to predict the behavioral performance improvement associated with the helper clear event, as well as to predict questionnaire responses regarding the helper clear event collected after gameplay. Performance, in our setting, was operationalized as the gameplay time in a given block, namely baseline and post helper timeframes.

For each investigated question, linear models were iteratively created and tested. Possible predictors were selected from the significant group level deviations observed in the time series analysis. Of the group level deviations from baseline activity the deviation with the most significant (lowest p-value) correlation between the subject’s EEG/PPG data and the variable to be predicted was used as the first predictor variable for a linear model. Then the next most significant correlation between subject EEG/PPG data and the questionnaire was added to the model. This iteratively continued until the p-value of the corresponding linear model began to increase as additional predictors were added. This iterative approach, rather than an exploratory approach searching a broad combination of predictor variables, both provides a simple means of model comparison as well as helps control type II statistical error.

6. Results

6.1. Performance Improvement

Baseline performance did not deviate significantly from normality in either session (session 1: p = 0.8604; session 2: p = 0.7518), and no significant difference was observed in baseline performance between sessions (p = 0.5975). For the session containing post-helper-clear, performance did not deviate from normality (session 1: p = 0.1749). In contrast, post-helper-clear event performance did significantly deviate from normality when no helper occurred (session 2: p = 0.0056). Although there was no helper piece appearing, the event was marked anyways, using the criteria explained in Experimental Protocol. This deviation may reflect a low bound of possible time played scores. A rank sum test identified a significant difference in post-helper-clear event performance across groups (p = 2.1982 × 10⁻⁶), with better performance occurring in the group that received the helper clear.

6.2. Time-Series Analysis

This section presents the time series analysis results for both EEG power and PPG data.

Table 1. Table showing the statistically significant events around the helper condition (t = 0 s) for frontal Theta power (FTP), parietal Theta power (PTP), frontal alpha power (FAP), parietal alpha power (PAP), frontal Beta power (FBP), and parietal Beta power (PBP) based on EEG power spectrum collapsed across electrodes (CAE). We also report the significant events of PPG changes including beats per minute (-BPM) and root-mean square of successive differences (RMSSD).

FTP-CAE									FAP-CAE					FBP-CAE
Time of event	−7	2	3	4	16	35	50		−4	4	51	52		−4	−3	0
Estimated Mean	−0.2277	−0.3511	−0.5505	−0.4547	−0.3108	−0.246	−0.3552		0.4499	−0.2924	−0.2932	−0.3696		0.517	0.4408	0.3164
p-value	0.029	0.0108	1.4708 × 10−6	1.8957 × 10−4	0.0379	0.0375	0.0424		0.0332	0.0314	0.0422	0.0031		0.0181	0.0065	0.044
PTP-CAE										PAP-CAE
Time of event	−7	1	5	6	9	14	16	38		0	1	36	46
Estimated Mean	−0.3068	−0.3854	−0.594	−0.5093	−0.4159	−0.3427	−0.3812	−0.2841		−0.3995	−0.3977	0.3977	0.4674
p-value	0.0447	0.0056	3.8388 × 10−4	0.0061	0.0076	0.0465	0.0402	0.0325		0.0157	0.0162	0.0198	0.0496
PBP-CAE							PPG-BPM					PPG-RMSSD
Time of event	−3	−2	1	2	3		−5	−1	0	7		−2	12	13	14	20
Estimated Mean	0.4917	0.4533	0.2963	0.4025	0.4699		−0.3912	−0.5192	−0.5427	−0.4698		0.5353	0.4936	0.5047	0.5202	0.5547
p-value	0.0146	0.0487	0.0203	0.0317	0.0191		0.0221	0.0276	0.0235	0.0232		0.025	0.0411	0.0402	0.0476	0.0447

highlights the significant events identified for each data stream and corresponding p-values. For all data streams, the time series analysis are graphically depicted in Figure 2 with the x-axis representing time and the y-axis representing normalized and standardized group data. Band power results are shown in a collection of prefrontal and midline frontal [Fp1 Fp2 Fz] and parieto-occipital [PO3 PO4 POz] electrodes. As described in the methods section, CAE power data in the corresponding region of interest is reported with respect to a baseline. Time series analysis for the PPG data is also shown below for both heart rate as well as heart rate variability measured in terms of RMSSD. Positive values mean increases in the corresponding measure relative to baseline and negative values indicate decreases relative to the baseline measurement. The dark blue line represents the estimated mean value, flanked by the the upper (yellow) and lower (light blue) bounds of a 95% confidence interval of the estimated mean. Significant events, as measured by single sample t-tests, are shown as vertical dotted lines in Figure 2 and reported in detail in the corresponding sections in Table 1.

Key events are summarized here, and detailed discussions can be found in the Discussion. The effects of the helper event (at t = 0) span multiple data streams over 20 s. Prior to the helper event, there was a notable increase in broadband EEG power, specifically in the frontal and parietal beta at 3 s, and in the frontal alpha at 4 s before the event, along with a positive trend across other power signals. Additionally, distinct changes were observed in parietal and frontal EEG activity, including a parietal beta increase from 1–3 s, a parietal alpha decrease from 0–1 s, and a frontal theta decrease from 2–4 s. Further post-event observations included a decrease in parietal theta at 14 s, an increase in HRV from 12–14 and at 20 s, and a decrease in heart rate at 7 s.

6.3. Predicting Performance Improvement

This section shows the results of a linear regression analysis predicting the magnitude of performance improvement using the significant deviations from baseline power and PPG activity that were observed in the time series analysis described above. A significant model ($F_{df=21}$ = 9.25; p = 0.00121) was obtained, with significant parietal predictors of performance improvement including parietal Beta activity at 3 s and parietal Theta activity at 5 s. The model has a high degree of predictive validity with an $R^2$ = 0.457, and an adjusted $R^2$ = 0.407.

$$\begin{array}{c}\widehat{y}={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2\end{array}$$

(3)

$$\begin{array}{c}{x}_1\equiv \mathrm{Parietal}\mathrm{Beta}\mathrm{at}3\mathrm{s}\end{array}$$

(4)

$$\begin{array}{c}{x}_2\equiv \mathrm{Parietal}\mathrm{Theta}\mathrm{at}5\mathrm{s}\end{array}$$

(5)

Table 2. Performance improvement model coefficients.

	Coefficient Estimate	p Value
${\beta }_0$	158.12	1.7066 × 10−8
${\beta }_1$	−49.843	0.0029947
${\beta }_2$	58.048	0.0045931

shows the individual contributions of the estimated components.

Here we see that both parietal Beta and Theta power were significantly predictive of the performance improvement induced by the helper clear event, with the directionality of the effect of each predictor opposed. Low parietal Beta power and high parietal Theta power predicts the largest performance improvement. This directionality is inversely related to the overall effect of the deviations from baseline observed in both measures.

Figure 3 shows the observed performance improvement on the x-axis, and the predicted performance improvement, as calculated by our linear model, on the y-axis.

We conducted a post-experiment questionnaire to gather feedback from participants about the effectiveness of the helper condition in the game “the ball shaped piece in a game”, specifically its impact on the helper and helper clear event. We selected a subset of questions from the questionnaire and scored the responses using a standardized system of integers. We then created linear models based on a standardized equation to investigate the relationship between different variables and gain insights into the impact of the ball shaped piece on participants’ gameplay experience. These questions were: (1) Was the ball shaped piece helpful in the game? If yes, how helpful was it?; (2) Was the effect of the ball shaped piece relieving? If yes, how relieving was it?; (3) Did you notice that the game became slower immediately after the ball cleared some rows? If not, how much longer did it take for you to realize that the game was now slower?; (4) After [the] ball cleared some rows and slowed down the game, did you feel a tension relief in your body? If yes, to what degree did you feel this?

Questionnaire responses were scored, converting the selected options to integers. For questions 1, 2, & 4 responses were scored as: “Very” = 4, “Moderately” = 3, “Fairly” = 2, “Slightly” = 1, “Not at all” = 0. Question 3 was scored as: “I did not realize” = 4, “Within a couple moves” = 3, “Within one move” = 2, “Almost immediately” = 1, “Immediately” = 0.

For each investigated question, linear models based on the equation below were iteratively created and tested.

The results of the most statistically significant of the observed models are reported here for each of the four investigated questions. Figure 4 displays the observed and estimated questionnaire responses, as calculated by these linear models. Questions 1, 2 and 4 each had significant models, as measured both by the significance of the F-statistic corresponding to the linear model as well as the significance of the correlation coefficient between the estimated values using the linear model and the observed values. As a note, in the cases where the selected linear model used only a single predictor, the statistical significance of both the linear model and the correlation coefficient are equivalent. In the cases where multiple predictors are used the linear regression statistical results indicate the degree to which one or more of the predictors has a significant linear relationship with the questionnaire response. The correlation results are a more direct measure of the predictive validity of the model, indicating the extent and significance of the relationship between the predicted and observed values. The $R^2$ value indicates the percentage of variance in the observed data explained by the linear model of the predictor variables.

6.3.1. Question 1: Was the Ball Shaped Piece Helpful in the Game? If Yes, How Helpful Was It?

Our linear regression variables were $x_1\equiv$ PPG HRV at 20 s, and $x_2\equiv$ Parietal Beta at 3 s.

Table 3. Individual contributions of the estimated components for question 1, 2, 3, and 4.

	Coeff. est.	SE	tstat	p Value	Coeff. est.	SE	tstat	p Value
Q1					Q 2
${\beta }_0$	1.6765	0.25031	6.6977	9.8599 × 10−7	2.5109	0.20266	12.389	2.1483 × 10−11
${\beta }_1$	0.62915	0.28374	2.2173	0.037241	−0.48211	0.22578	−2.1353	0.044115
${\beta }_2$	0.43876	0.22425	1.9566	0.063211	0.40724	0.20894	1.9491	0.06415
Q 3					Q 4
${\beta }_0$	1.6421	0.3560	4.6124	0.000012	1.6765	0.2503	6.6977	9.86 × 10−7
${\beta }_1$	−0.7636	0.43776	−1.7461	0.0941	0.62915	0.28374	2.2173	0.0372
${\beta }_2$	-	-	-	-	0.43876	0.2243	1.9566	0.0632

shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (3.65) was significant with a p-value (df = 22) = 0.0426. There was furthermore a significant correlation ($p=0.011;R^2=0.2494$) between the predicted and observed questionnaire response values.

Our findings indicate that the parietal Beta activity just following the helper clear event and the heart rate variability 20 s following the helper clear event are significantly related to subjective reports of how helpful the helper was. The correlation results indicate that 25% of the variance in subject questionnaire responses to question 1 is explained by a linear combination of the PPG HRV data 20 s after the helper clear event and parietal Beta activity 3 s after the helper clear event. While the individual predictor variables were not significant when individually tested (see table), the significance of the overall model indicates a significant effect. This may indicate a degree of correlation between the predictor variables, though a statistically significant correlation was not observed ($p>0.05$). Given that this model’s performance exceeded the performance of a model with only a single predictor variable, both terms likely contribute. The significant intercept term indicates, as expected, that the mean questionnaire responses differ from 0.

6.3.2. Question 2: “Was the Effect of the Ball Shaped Piece Relieving? If Yes, How Relieving Was It?”

In this case, our regression variables were $x_1\equiv$ Parietal Theta at 14 s, and $x_2\equiv$ Parietal Beta at 2 s. Table 3 below shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (5.39) was significant with a p-value (df = 22) = 0.0124. There was furthermore a significant correlation ($p=0.0027;R^2=0.329$).

Our findings indicate that the parietal Beta activity just following the helper, and parietal Theta 14 s following the helper, is significantly related to subjective reports of the relief experienced following the helper clear event. The correlation results indicate that 33% of the variance in the subject questionnaire responses can be explained by this linear model.

6.3.3. Question 3: Did You Notice That the Game Became Slower Immediately after the Ball Cleared Some Rows? If Not, How Much Longer Did It Take for You to Realize That the Game Was Now Slower?

Our regression variable in this case is $x_1\equiv$ Parietal Theta at 9 s. Table 3 shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (3.05) was not significant with a p-value (df = 22) = 0.0941. There was furthermore no significant correlation (p = 0.0953) between the predicted and observed questionnaire response values.

The intercept term, as expected, was significant. This indicates that the average questionnaire responses were greater than zero, though the insignificant finding of the linear model as a whole indicates there is no linear relationship between the investigated predictor variables and this particular questionnaire response.

6.3.4. Question 4: After the Ball Cleared Some Rows and Slowed Down the Game, Did You Feel a Tension Relief in Your Body? If Yes, to What Degree Did You Feel This?

The regression model variables were $x_1\equiv$ Frontal Beta at −3 s, and $x_2\equiv$ Parietal Beta at 3 s. Table 3 shows the individual contributions of the estimated components. The results of the linear model as a whole, as determined by the F-statistic vs. constant model (5.36) was significant with a p-value (df = 22) = 0.0127. There was furthermore a significant correlation ($p=0.0028;R^2=0.3276$) between the predicted and observed questionnaire response values.

Our findings indicate that the frontal Beta activity just prior to the helper clear and parietal Beta activity just following are significantly related to subjective reports regarding the degree of tension relief subjects report following the helper clear event. The correlation results indicate that 33% of the variance in the subject questionnaire responses is explained by a linear combination of the Frontal Beta activity just prior and Parietal Beta just following the helper clear event.

7. Discussion

The present study investigated the impact of a helper event in VR Tetris gameplay on various cognitive and physiological measures. Following a helper intervention that cleared tiles, we observed group level decreases in mental and visuospatial workload (as measured by frontal and parietal oscillatory activity) and in physiological arousal (as measured by heart rate and heart rate variability). Furthermore, a significant benefit to performance was observed due to the helper clear event. Subsequent predictive analysis, however, indicates that performance improvement was not predicted by the significant decrease in frontal Theta, but rather by parietal activity commonly associated with visuospatial processes [49,50]. Although they were not specifically directed, processes such as visuospatial working memory and attention were constantly challenged during gameplay due to the nature of Tetris [51,52]. The present findings are consistent with a model of dissociable but interacting functions of the prefrontal and posterior parietal cortices. Our results suggest that performance in VR Tetris, a spatial task in a rich visual environment, is driven by parietal activity associated with visuospatial processing, rather than frontal activity. This dissociation may be due to the visuospatially rich quality of the task load [24]. This framework, and these results, highlights the importance of multiunit EEG recordings that cover a broad area of the scalp, thus networks of brain regions, when investigating VR environments using EEG. Frontal electrodes alone are not sufficient to capture the activity of the frontoparietal network that drives performance improvement in this task, and parietal electrodes may miss significant deviations in cognitive load.

7.1. Temporal Dynamics of Helper Response

The differences from baseline across data streams were largely focused on here in terms of those that contributed to substantial linear models of subjective questionnaire responses regarding the helper event, as well as those that provide specific insight in relation to our stated hypotheses.

The first event of note is an increase of both frontal and parietal beta power from baseline 3 s prior to the helper clear event, corresponding for most subjects to the time after the helper notification but before the clear event itself. This corresponds to an elevation across power signals. While significant deviations were observed in Frontal Alpha and Beta as well as Parietal Beta, there was a positive trend among all other power signals. This broadband increase in power from baseline may indicate a generalized increase in engagement, mental load, and fatigue [53], corresponding to an increase in mental and visuospatial workload [22,23] (frontal and parietal beta). This may reflect the high degree of task difficulty that subjects faced prior to the helper event occurring. The frontal beta at −3 s was a selected predictor variable with a positive coefficient in the linear model of questionnaire responses related to the subjective sensation of tension relief from the body, indicating the larger the positive deviation from baseline the more tension relief subjects reported. The tension relief may actually reflect the return to baseline rather than the elevated cognitive load itself. While this result differs from our hypothesis of a decrease in beta power following the helper clear event, the general pattern of a return to baseline from an elevated cognitive load preceding the helper clear event is consistent with the model underlying the hypothesis. In particular that the helper clear event itself reduces cognitive load.

The second event of note is a sustained increase in parietal beta from baseline 1–3 s following the helper clear. At 0–1 s there is additionally a significant decrease from parietal alpha. This is consistent with a spike in visual-spatial engagement following the helper clear [23,25,26]. This increase in engagement may reflect the subject’s re-evaluation of the game state. Subject’s parietal beta power in this period is one of the selected predictive variables for linear models for all three of significant linear models reported, regarding how helpful and relieving the helper was, as well as the degree to which subjects reported tension leaving their body. The subjective reporting of how helpful/relieving the helper was is significantly and robustly (across question wording) related to degree to which visuospatial engagement was devoted to re-evaluating the game state post helper clear event. At the same time, as predicted, was a large decrease in frontal theta power indicating a decrease in cognitive load [21]. While the decrease in frontal theta was significant, subjective reports of how helpful/relieving the helper intervention was were more driven by the broadband increase in power prior to intervention, as well as parietal activity and physiological arousal after the helper event occurred. While the directionality of the cognitive load finding is in line with our hypothesized result, that subjective reports of the helpfulness was driven more by parietal activity was a deviation from our hypothesis.

The third event of note is a decrease in parietal theta power at 14 s and a subsequent increase in heart rate variability 20 s following the helper clear event. There were additional significant increases from baseline in heart rate variability 12–14 s following the helper clear and a significant decrease in heart rate 7 s following the helper clear, implying a change in the ANS balance in favor of parasympathetic tone [28,29]. The decrease in parietal theta (a selected predictor in the linear model of how relieving the ball was) may indicate a corresponding decrease in sustained visuospatial engagement [24] in return reflecting a more relaxed approach to gameplay. This is consistent with the hypothesized and observed decrease in physiological arousal following the helper clear event as evidenced by the heart rate data. Furthermore the heart rate variability at 20 s was a selected predictor variable with a positive coefficient in the linear model of the question relating to the helpfulness of the helper clear. The effects of the helper clear event observed here spans multiple data streams across a time interval of over 20 s. The data streams include frontal and parietal EEG data as well as PPG heart rate variability data that in turn predict 25–33% of the variance observed in subjective questionnaire responses about how helpful or relieving the helper event was.

7.2. Limitations

Although present exploratory study provides insight into the dynamics of mental and visuospatial workload in VR environments, several limitations should be noted. Despite randomizing the order of intervention and control conditions and providing a tutorial session, individual variations in prior in experience with VR and Tetris might have still influenced the results. Future investigations could benefit from a more controlled selection process to address these individual differences more effectively. Additionally, although our task included a rich visuospatial environment, future studies could exploit the full, ambulatory potential of VR systems by utilizing mobile brain-body imaging. Furthermore, while we used a broad range of electrodes to cover the frontoparietal network, central electrodes near sensorimotor cortex might have been affected due to hand movements. It is previously reported that event-related desynchronization (ERD) may be observed in parietal electrodes due to volume conduction [24]. These effects could be isolated in future studies by utilizing source localization methods and increased number of electrodes [54]. Moreover, although fluctuations in EEG power bands and HR and HRV values have been extensively used and validated as a means to measure workload, further analysis methods such as event-related spectral perturbations (ERSPs), functional connectivity analyses, and emotion classification systems using spatio-temporal image encodings [55] could provide deeper insights. Additionally, incorporating physiological modalities such as EDA, EOG, EMG and skin temperature could be utilized to better probe the effects of mental workload. Finally, although Tetris was chosen for its well-defined cognitive and visuospatial demands, future research should explore a range of tasks with varying cognitive requirements to assess the generalizability of our results. Investigating different types of VR activities and real-world scenarios could also provide a more comprehensive understanding of how mental workload and visuospatial engagement varies across diverse contexts.

7.3. Conclusions

In this study we observed a broadband increase in EEG power just prior to the helper event followed by a spike of visuospatial engagement (parietal alpha and beta 0-1-3 s) that occurred concurrently with decrease in mental workload (frontal theta 2–4 s), in turn followed by a subsequent decrease in visuospatial engagement (parietal theta at 14 s) and decrease in physiological arousal (HRV at 20 s). While as expected there was a decrease in cognitive load in response the helper clear event, subjective relief and the subjective reports about how helpful the event was were more driven by physiological arousal and the parietal oscillatory response to the helper, commonly regarded as a proxy for visuospatial processes. Furthermore, our findings suggest the plausibility of the modulation of ANS balance using help interventions in VR tasks. By using online task load modifications similar to our helper piece, it may be possible to optimize workload in favor of overall mental health and well-being.