Next Article in Journal
UBUMonitor: An Open-Source Desktop Application for Visual E-Learning Analysis with Moodle
Next Article in Special Issue
Camera Animation for Immersive Light Field Imaging
Previous Article in Journal
Two-Layer Bus-Independent Instruction Set Architecture for Securing Long Protocol Data Units in Automotive Open System Architecture-Based Automotive Electronic Control Units
Previous Article in Special Issue
Modeling the Level of User Frustration for the Impaired Telemeeting Service Using User Frustration Susceptibility Index (UFSI)
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Electronics
Volume 11
Issue 6
10.3390/electronics11060953
electronics-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Beyond Perceptual Thresholds and Personal Preference: Towards Novel Research Questions and Methodologies of Quality of Experience Studies on Light Field Visualization

by
Peter A. Kara
1,2,*,†,
Roopak R. Tamboli
3,†,
Edris Shafiee
1,
Maria G. Martini
1,†,
Aniko Simon
4 and
Mary Guindy
5,6
1
Wireless Multimedia and Networking Research Group, Department of Computer Science, School of Computer Science and Mathematics, Faculty of Science, Engineering and Computing, Kingston University, Penrhyn Road Campus, Kingston upon Thames, London KT1 2EE, UK
2
Laboratory of Multimedia Networks and Services, Department of Networked Systems and Services, Faculty of Electrical Engineering and Informatics, Budapest University of Technology and Economics, 1111 Budapest, Hungary
3
Department of Electrical Engineering, Indian Institute of Technology Hyderabad, Sangareddy 502284, Telangana, India
4
Sigma Technology, 1093 Budapest, Hungary
5
Holografika, 1192 Budapest, Hungary
6
Faculty of Information Technology and Bionics, Pazmany Peter Catholic University, 1083 Budapest, Hungary
*
Author to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Electronics 2022, 11(6), 953; https://doi.org/10.3390/electronics11060953
Submission received: 28 February 2022 / Revised: 15 March 2022 / Accepted: 16 March 2022 / Published: 19 March 2022
(This article belongs to the Special Issue Immersive Quality of Experience Management and Evaluation)
Browse Figure
Versions Notes

Abstract

:
With the upcoming emergence of the IEEE P3333.1.4 standard, the first ever standardization document on the subjective quality assessment of light field visualization, we are approaching a major milestone of light field Quality of Experience (QoE). The research efforts leading up to this point have answered the fundamental questions regarding perceptual thresholds and personal preference, and best practices regarding research methodologies have been formed. However, in order to introduce successful and efficient light field systems and services, research questions beyond perceptual thresholds and personal preference must be addressed, along with the associated methodologies. In this paper, we propose a comprehensive set of novel research questions and methodologies regarding light field QoE. Our work also provides a detailed discussion of related factors, particularly those that extend to test participants and thus may affect the results of subjective studies.

1. Introduction

Light field visualization enables a 3D sensation without the need of additional viewing equipment. It is thus more natural than the majority of most state-of-the-art 3D technologies, and seamlessly enables arbitrary numbers of simultaneous viewers. Many use case contexts are already envisioned (e.g., industrial, medical, educational, commercial, cinematic, etc.), and general-purpose and dedicated systems are being developed. Researchers are working tirelessly to ensure that the Quality of Experience (QoE) of light field displays and their use cases will not disappoint the viewers and users.
Over the recent years, numerous works on light field visualization have been published. Adhikarla et al. [1] performed a preference test regarding multiple levels of degradation, Bakir et al. [2] examined how view-synthesis-based light field compression affects the perceived quality; Battisti et al. [3] investigated the artefacts that may appear on views synthesized by depth-image-based rendering (DIBR); Carballeira et al. [4] studied the suitable levels of view density; Palma et al. [5] and Perra et al. [6] subjectively assessed different codecs and bitrates; Paudyal et al. [7] addressed the perceivable results of wavelet-based watermarking; Recio et al. [8] particularly focused on view transition; Shi et al. [9] evaluated the performance of the conventional objective quality metrics; and Wen et al. [10] tested encrypted light field contents. These scientific efforts have a couple of things in common. First of all, they are all valuable pieces of the literature and assist the advancement of the field. However, another important commonality is that the experiments did not include the usage of actual light field displays; the subjective tests were carried out on conventional 2D displays.
Due to the limited availability of light field displays, the number of works that involve them is notably lower in comparison. The existing studies predominantly focus on the fundamentals of light field QoE. These are either measurements of perceptual threshold levels (e.g., what level of degradation is perceivable by the test participant, what extents of distortion are still considered adequate for the investigated use case, etc.) or personal preference tests (i.e., if two representations are both degraded, which representation is more tolerable). These are, of course, the building blocks of future systems and services of light field technology, as they provide the necessary insight regarding user experience. For example, perceptual thresholds enable perceptual coding, which means that if two representations cannot be distinguished, then the one that requires fewer resources should be used. Moreover, the upcoming IEEE P3333.1.4 Standard for the Quality Assessment of Light Field Imaging (https://standards.ieee.org/ieee/3333.1.4/7724/ accessed on 28 February 2022) is also centered around such scope. However, while these works do answer fundamental questions regarding light field QoE, there are many other—sometimes significantly more complex—phenomena that need to be investigated prior to the emergence of light field technology onto the consumer market. Additionally, addressing such phenomena shall bring forth the emergence of new, more advanced standards as well.
In this paper, we provide a detailed proposal of novel research questions and methodologies related to the QoE of light field visualization. Our proposal extends to practical considerations and specific hypotheses. We also discuss topics that are not necessarily particular research questions on their own, but may highly influence the experiments.
The remainder of this paper is structured as follows: Section 2 provides a brief analysis of the works published so far, addressing the most relevant factors of QoE. The novel research questions and the related methodologies are proposed in Section 3 and Section 4, respectively. Section 5 discusses topics and issues that may significantly affect the investigation of the proposed research questions. The paper is concluded in Section 6.

2. Analysis of the State-of-the-Art Scientific Literature

In this section, the scientific literature on the QoE studies of light field visualization is reviewed. As stated earlier, some studies address light-field-related research questions on displays that do not qualify as light field displays. For example, certain scientific efforts focused on the perceptual evaluation of light field content using stereoscopic displays. These include the study performed by Shi et al. [9], where content was presented to test participants on a 55” stereoscopic display. The authors used the same approach to evaluate objective metrics in other works [11,12,13]. Conventional 2D displays are frequently used as well, such as in the works of Adhikarla et al. [1], Shan et al. [14] and Viola et al. [15].
The scope of the analysis is focused on research efforts where human observers and real light field displays are involved. A summary of the state-of-the-art scientific works is provided in Table 1.

2.1. Research Question

As stated earlier, the majority of research questions are centered around perceptual thresholds and personal preference. Ahar et al. [18] and Kara et al. [26] addressed light field reconstruction; Cserkaszky et al. studied the perceivable effects of interpolation techniques [19] and the viability of a novel light field format [20,21]; Darukumalli et al. investigated the Region of Interest (RoI) [22] and different zoom levels [23] of light field content; Dricot et al. [24] evaluated compression artefacts; Kara et al. [27,28,29,30,31,33] and Kovacs et al. [37,38,39] carried out subjective tests regarding spatial and angular resolutions; the works of Kara et al. also cover the Field of View (FOV) [25], viewing conditions [34], 3D viewing sensation [35] and preferred viewing distance [36]; and Tamboli et al. examined the effects of spatial distortion [40,41] and view synthesis [42], and assessed the personal preference regarding content orientation [43] as well. The work of Kara et al. [32] on dynamic adaptive streaming is also evaluated through personal preference. In other works, Adhikarla et al. [16,17] studied interaction and performance, and Zhang et al. [44] interviewed test participants regarding a novel telepresence system.

2.2. Test Variables

The test variables of a subjective test are parameters that differ between test conditions. Fewer variables enable more focused comparisons, while more variables are sometimes necessary for more complex and more realistic studies. In most cases, a single test variable is appropriate for research. Determined by the research questions, the variables in the literature are spatial resolution [28], angular resolution [19,20,21,26,27,29,31], both spatial and angular resolutions [30,32,33,34], spatial distortion [18], viewing distance [35,36], FOV [25], symbol size [37,38,39], zoom level [22,23], content alignment [23], compression and distortion parameters [24,40,41], the number of dropped views [42], content orientation [43] and user interface attributes [16,17].

2.3. Assessment Methodologies

2.3.1. Evaluation Task

In these works, test participants had to compare degraded stimuli to the reference quality [23,24,28], compare degraded stimuli to each other [19,20,21,22,30,32,33,34], assess stimuli on their own [18,22,23,25,26,27,29,31,40,41,42], determine symbol orientation [37,39], distinguish visual grating [38], report personal willingness (i.e., willingness-to-pay and willingness-to-use) [25], rate the 3D experience [35], select the perceptually-supported and personally-preferred viewing distance [36], set the preferred content orientation [43] and carry out and evaluate interactive tasks [16,17,44].

2.3.2. Rating Scale

The vast majority of the tests followed ITU recommendations—such as the P.910 (ITU Rec. P.910: Subjective video quality assessment methods for multimedia applications) and the BT.500 (ITU Rec. BT.500: Methodologies for the subjective assessment of the quality of television images)—for subjective score collection. Note that these standards are, of course, not specific for light field visualization. Absolute Category Rating (ACR) tests were performed on 1-step 5-point [21,22,23], 0.25-step 5-point [40,41,42] and 1-step 10-point [25,26,27,31] variations; the 5-point Degradation Category Rating (DCR) was used [18,23,24,28,34]; stimuli were compared via 3-point [20,21], 5-point [32] and 7-point [19,21,22,30,33,35] symmetrical scales; binary scales were utilized for acceptance [21], willingness [25] and content discrimination [38]; and a 25-point quasi-continuous scale [29] was used, as well as the NASA Task Load Index (TLX) questionnaire (https://humansystems.arc.nasa.gov/groups/TLX accessed on 28 February 2022) and the User Experience Questionnaire (UEQ) (https://www.ueq-online.org accessed on 28 February 2022) [16,17].

2.4. Viewing Conditions

2.4.1. Viewing Distance

For smaller displays, the viewing distance during the experiments varied between 50 cm [16,17] and 80 cm [39]. The only exception was the assessment of the FOV [25], where it was up to 5 m. For larger displays, the most common viewing distance was 4.6 m [19,20,21,22,23,26,27,28,29,30,31,32,33,34,43] but, depending on the research question, greater distances—up to 7.5 m—were frequent as well [18,20,21,24,28,29,37,38]. In the early works of Tamboli et al. [40,41,42], a close distance of 2.44 m was chosen for accurate evaluation on the large display. For the assessment of the small telepresence system of Zhang et al. [44], the distance ranged between 1.2 m and 3.6 m. Regarding experiments that addressed viewing distance directly, the possible options were either every 0.5 m between 4.5 m and 7.5 m (large display) [35], every 0.25 m between 0.25 m and 8 m (small display) [36] or every 0.25 m between 4 m and 8 m (large display) [36].

2.4.2. Observer Motion and Viewing Angle

Observer motion varies a lot in the scientific literature. In many cases, test participants were instructed to remain—either standing or sitting—in a fixed position during the experiment [16,17,18,22,23,24,29,30,37,39,40,41,42]. Since the displays selected for the tests supported horizontal parallax, sideways motion was included in the experimental setup, typically in the ranges of 1 m [19,26,27,28,31,32,33,34] and 4 m [20,21]. In other tests, either a radius was defined for arbitrary movement (e.g., 3.6 m [44] or 5 m [25]) or movement was completely unconstrained, limited only by the FOV itself [38,43]. When the viewing distance was addressed [35,36], sideways movement was excluded. As for viewing angle, the default approach was to use the center view. Only a couple of studies deviated from this practice [29,40,41,42], which all define fixed seats for the test participants.

2.4.3. Test Environment

In every single case, the test environment was a laboratory setting, isolated from external audiovisual distractions. Lighting conditions did not vary much from the best practice of providing a relatively dark environment, completely shielded from natural sources of light. The most common values of lighting conditions were around 25 lx if the display brightness was approximately 1500 cd/m2. A high portion of the experiments used the HoloVizio C80 light field cinema system (http://holografika.com/c80-glasses-free-3d-cinema accessed on 28 February 2022) [19,20,21,22,23,24,26,27,28,29,30,31,32,33,34,35,36,37,38,43], but the HoloVizio 80WLT (https://holografika.com/80wlt accessed on 28 February 2022) [25,36,39], the HoloVizio 721RC [40,41,42]—which was superseded by the 722RC (https://holografika.com/722rc accessed on 28 February 2022) [18]—and different prototypes [16,17,44] were used as well. All of these systems are horizontal-only parallax (HPO) light field displays.

2.5. Test Participant Demographics

Subjective test series most commonly include 20 test participants [22,23,25,26,27,28,31,32,40,41,42]. Other test participant numbers in the scientific literature are 6 [44], 10 [43], 12 [16,17,20], 16 [24], 18 [33,34,35], 21 [19], 22 [29,30,36], 24 [21], 40 [18] and 53 [39]. Test participants are either naïve individuals (i.e., not experts of light field technology) or experts, both screened for vision capabilities. Individuals belonging to the latter category are significantly more difficult to recruit for obvious reasons. The average age of test participants is typically between 25 and 30, yet the range of age distribution may vary a lot, particularly in the upper regions. The youngest test participants were around 20 in almost all of the experiments, yet, for example, the oldest were 30 [44], 36 [16], 42 [34], 50 [22], 58 [30] and 65 [36] in the different test series. Regarding gender distribution, it is a common aim to have nearly the same number of male and female test participants; however, in practice, it is rather common that there are more male than female test participants in a subjective study on light field visualization.

3. Novel Research Questions

In this section, we propose novel research questions, the investigation of which shall greatly contribute to the understanding of light field QoE, aid the development of future standards, and serve the advancement of the emergence and deployment of light field systems and services.
The distribution of the state-of-the-art scientific literature and the completely open research questions—which are not addressed by research yet—are shown on Figure 1. The published works are cited by the same numbers as in Table 1, and generally as throughout the entire paper. A great portion of the relevant research efforts focused on visualization-related research questions. System-related research questions are ample as well, but the majority addresses the Key Performance Indicators (KPIs) of light field visualization [45], and a few used dedicated systems (i.e., prototypes). Works on viewer-related research questions investigated viewing conditions, interaction and user interface.
Specific research questions may relate to two or three categories simultaneously. For example, the works of Cserkaszky et al. address interpolation techniques [19] and light field formats [20,21] based on the variation of KPIs, and Adhikarla et al. [16,17] studied interaction on a novel dedicated system. Thus far, no work investigated the combination of viewer-related and visualization-related research questions, and evidently this is applicable to the combination of the three categories as well.

3.1. Immersion

Immersion is a process through which individuals may forge a deeper connection with the perceived content, otherwise known as Cognitive Absorption (CA), as defined by Agarwal et al. [46]. This enables viewers to understand the content better and enhance their perception of realism. Immersion may be achieved through a variety of different factors that could impact how immersed a viewer is in the content. According to the best knowledge of the authors, up to this point in time, no subjective tests have been carried out to measure the level of immersion for light field visualization.
Subjective tests should address factors of immersion that may be specific to light field technology. Hypotheses on the potential improvement of immersion should be tested as well. Furthermore, conventional methods and metrics of immersion assessment should be utilized, and research should aim to develop newer, more fitting solutions. One particular way of measuring immersion is through questionnaires completed by test participants. Task completion time and physiological measurements may also provide information regarding the level of immersion.
Light field content itself is a key supporting factor of immersion. Volino et al. [47] discussed two main issues that could impact immersive content. According to the views of the authors, the quality of content representation and compression are essential in building an immersive bridge between the content and the viewer.

3.2. Interaction

An example of the interaction between test participants and light field systems is the modification of the representation of the visualized content. It enables test participants to change the orientation and zoom level of the represented objects in the scene, and other forms of content alteration are possible as well, such as panning the entire scene. However, interaction covers a broad range of activities, from playing 3D games to creating static and dynamic contents. Most of these activities are very well defined for the potential future use cases of light field technology.
While task completion time may provide information regarding the level of immersion, it is much more relevant to studies focused on interaction. Evidently, questionnaires may address multiple components and factors of the global user experience. It is vital to the success of interactive use cases to study the link between visualization quality and user satisfaction.
Additionally, there are, in fact, use cases for multi-user scenarios, such as the so-called “split-screen” gaming. However, unlike in the 2D counterpart where per-user screen resolution is sacrificed to enable multiple views, users may be angularly separated. This also provides the added benefit that a specific game view can only be observed by a given player—highly supporting competitive gaming. Regarding gaming in general, considerations regarding “playability” should be thoroughly addressed. This includes end-to-end delay, where one end is the input from the user, and the other end is the perception of change in the system. The investigation of delay is particularly important due to the potentially high rendering times of light field technology.
There may be interaction between users as well. This is especially applicable if users engage with the same system locally. However, such form of interaction leads to inter-user effects, which are detailed later in the section.

3.3. Human–Computer Interface

Human–Computer Interface (HCI) studies are similar to interaction studies in the sense that the test participant interactively engages with the light field visualization system. The main difference is that while interaction studies consider the entire pipeline of interaction, HCI studies particularly focus on the specific interface through which test participants provide input. Furthermore, interaction studies may completely minimize the role and scientific relevance of the interface within the research. For example, in the work of Tamboli et al. [43], an interactive use case is addressed, but there is no emphasis whatsoever on the HCI.
Thus far, Adhikarla et al. [16,17] carried out subjective tests on the HCI of light field displays. During the experiment, the test participants were instructed to touch a red square within the visualized light field. However, in order to address the primary practical properties of the HCI, two interfaces were evaluated. One was a “3D mode”, which was the light field interface, and the other was a “2D mode”, which was technically still light field, yet the plane of visualization was very close to the plane of the surface of the display. Performance was evaluated via task completion times, cognitive workload and perceived user experience.
There are many types of HCI solutions that are to be investigated, and the number of potential hypotheses is quite generous as well. Since light field displays offer glasses-free 3D visualization to multiple viewers simultaneously, it is perfectly valid to think of the viewers as users—specifically in novel interactive use cases—and thus to consider multi-user engagement. Regarding both single-user and multi-user contexts, the appropriate parametrization of 3D control interfaces (similar to what was used in the work of Adhikarla et al.) should be investigated, such as perceivable depth, resolution values and spatial separation. Finally, it should be mentioned that certain purpose-specific, dedicated light field displays are expected to have specially designed user interfaces to accommodate the tasks assigned to their use cases, and they should be optimized through exhaustive QoE research.

3.4. Viewing Conditions

Novel research questions arise regarding viewing conditions for light field displays with respect to other types of 3D displays, including glasses-based and autostereoscopic displays. A significant difference between autostereoscopic and light field displays is that the latter provides a continuous perspective over the FOV, while the other relies on so-called “sweet spots” and it is common that the small-FOV content repeats over the FOV of the display periodically.
In the case of light field displays, the viewing position and motion path of the test participant are crucial to perceived quality. The viewing position can be described by viewing angle and viewing distance. The latter is particularly important, since apart from angular resolution, it is the other major factor that determines perceived ray density, which evidently correlates with 3D experience. In the scientific literature [35,36,48], the first steps have already been taken, but they are just the beginning of a long journey.
The motion of test participants is a highly relevant research topic, since the continuous change in perspective may provide a compensation for insufficient technical properties, such as resolution [34]. This should be taken into consideration when designing systems for use cases where viewers are typically moving. QoE research should separately study the most relevant use cases, such as digital signage.
Another vital aspect of the viewing conditions of light field systems is the level of environmental illumination, commonly referred to as lighting conditions. Light field displays are designed with the purpose of enduring the lighting conditions that conventional 2D displays face in their usage contexts. The works of the scientific literature present light field visualization under the favorable conditions of laboratories (i.e., no external light sources and controllable illumination). QoE studies should aim to consider the possible lighting conditions of practical use cases and test the limits of light field displays by intentionally exposing them to bright environments.

3.5. Inter-User Effects

One of the greatest strengths of light field displays is the fact that multiple observers may view the 3D visuals simultaneously. Many of the use case contexts are envisioned in a way that this vital attribute shall provide a significant added value compared to other 3D technologies. As a matter of fact, such systems and services aim to be analogous to the conventional 2D counterparts. Let us just take for example the exhibition of cultural heritage, where many visitors view the exhibited entity at the same time. The difference in digital presentation is that light field visualization enables a 3D perception of the entity, while 2D systems require content rotation and animation to achieve a fraction of the visual experience.
However, the viewing company itself may contribute to the global experience of the individual. In fact, there are numerous ways that the presence of other viewers or users may influence the viewer or user of a use case. In the scientific literature, no research has addressed the inter-user effects of light field visualization. The vast majority of the experimental setups of the subjective tests were designed for a single test participant. Of course, from a methodological point of view, this is a perfectly valid choice. After all, excluding the option of having simultaneously present test participants enables a more focused, more controlled experiment, and prevents certain biases. For example, in an early work on the cinematic experience [29], test participants were assigned to six viewing positions (i.e., six seats). Yet, at a time, only one of these seats was occupied by a test participant.
The investigation of inter-user effects should extend across multiple research topics. First of all, test participants affect each others task performance, the fact of which has been studied in countless contexts over half a century. For example, the publication of Murray et al. [49] highlights this effect in the case of rowing in virtual reality. What is more important is that the results of quality assessment tasks may be directly affected as well—even without the approach of the classic study of Asch [50] on independence and conformity. Therefore, this consideration on its own already opens up methodological questions of subjective data collection. Naturally, the presence of others may also result in temporary visual occlusions (i.e., the view of one may be partially blocked by another). Furthermore, the more test participants are present, the more the observer motion of one is constrained; this is particularly applicable to the observation of viewing perspective transitions, especially when combined with potential occlusions. While many of such studies should be carried out in a general context, later research should specifically address use cases.

3.6. Cognitive Bias

Subjective studies are easily affected by cognitive bias. They may fundamentally shape the quality ratings, and thus, may be the source of numerous research questions and hypotheses. The better we understand cognitive bias, the easier it is to prevent or to handle distorted QoE research results.
There are many forms of cognitive bias that may apply to light field visualization. For example, in the case of confirmation bias [51], a test participant has a preconception about the quality, and seeks visual confirmation about the prior idea. Evidently, this may greatly affect the focus, the attention of the test participant. However, the memory of visual experience may be influenced as well, via the misinformation effect [52], during which new pieces of information may overwrite older ones—sometimes incorrectly. When the use cases of light field displays are addressed, the framing effect [53] may be relevant as well, which affects perception through the context itself. Additionally, since the vast majority of naïve test participants have never seen any light field system prior to the experiment—or has never even heard of the technology—the initial visual experience may result in the effect of anchoring [54], and introducing light field technology to such individuals may induce the labeling effect, the influence of which has already been studied for Ultra-High Definition (UHD) [55] and High Dynamic Range (HDR) [56] displays.

3.7. Under-Represented Observers

A common practice in QoE studies is that test participants are screened for normal vision. The literature on light field QoE is no stranger to this; in fact, all the published works so far follow the standard screening procedures, namely the Snellen chart for visual acuity, the Ishihara plates for color vision and the butterfly test for 3D vision. It does make sense to screen test participants for experiments on perceptual thresholds and for modeling; however, once light field systems and services emerge, they are meant to be used by everyone.
Individuals with imperfect vision are currently under-represented in light field research. It is vital that scientific knowledge is gained on their perception of light field visualization in order to account for their presence in future use case contexts. Furthermore, beyond the core purpose of inclusion, specific systems and solutions can be developed to assist those with imperfect vision.
Visual acuity is highly relevant to spatial resolution. Insufficient spatial resolution may lead to blurred visuals. According to findings so far [28], screened test participants typically tolerate spatial resolution reduction well, yet this may differ for those without the same level of visual acuity. Color vision is fundamentally important to depth and angular resolution, as the perceived hue may affect depth perception [57]. Finally, the perceived smoothness of the parallax effect may greatly vary between test participants screened for 3D vision and those who would not qualify as test participants in the works of the current literature.

3.8. Perceptual Fatigue

Thus far, no subjective test has been carried out to address the perceptual fatigue that may come with the observation of light field displays. QoE studies should model the exhaustion of test participants, which is particularly important to the aforementioned under-represented observers. Furthermore, comparative studies should investigate the difference between light field visualization and other 3D technologies in terms of perceptual fatigue.

3.9. Technological Comparisons

Up to this point in time, subjective tests investigated light field visualization on its own, without direct comparison to other visualization technologies. Technological comparisons could quantify the enhanced effectiveness and efficiency of light field use case contexts. A straightforward example is digital signage, where most of its commercial utilization aims to gain and hold attention. In an experiment, contents of digital signage could be displayed on a light field display and a high-end yet technologically more conventional display, and simple eye tracking could differentiate the achieved levels of visual attention.
Another important comparison could be task performance. First and foremost, it could serve as a basis to test the efficiency of light field HCI solutions, and the obtained results could support further improvements of interfaces. Secondly, such 3D interfaces are not de facto “better” in terms of performance. Direct technological comparisons could enable exhaustive analyses of the use cases, and, for instance, if the results suggest that a conventional 2D interface is more adequate for the given context, then a hybrid solution could be optimal.

3.10. Dedicated Systems

At the time of writing this paper, most light field displays are universal; they are general-purpose glasses-free 3D displays. There are already examples of dedicated systems as well, such as LightBee [44] or the telepresence system of Holografika [58], both serving communication purposes. It is expected that numerous dedicated light field display systems will emerge in the upcoming years. It is imperative that such devices are thoroughly tested in terms of QoE, particularly in light of the designed usage contexts. Such research efforts would primarily cover viewing distance, optimized resolution values and HCI solutions, but other considerations and research questions could apply as well, such as inter-user effects.

3.11. High Dynamic Range Visualization

Although light field and HDR displays are already available, their combination is still a matter of ongoing research. HDR technology shall be a valuable addition to light field visualization, as on its own, it contributes to a better representation of reality [59].
In 2009, Lumsdaine and Georgiev [60] introduced the focused plenoptic camera with a lenslet array. The latter acts as an “imaging system focused on the focal plane of the main camera lens”. This method allows the captured light field to have a higher spatial resolution compared to what may be captured by conventional plenoptic cameras. While this method produces HDR light fields with acceptable parallax, a decrease in the angular density is evident. As an extension to the focused plenoptic cameras, two techniques were suggested by Georgiev et al. [61] for rich image capture. This is achieved via multiplexing the captured plenoptic function by interleaving the filters either at the mircrolens array or at the main camera lens. Regarding videos, the work of Schedl et al. [62] addresses the motion blur problem occurring in the case of camera movements within the capture process. Wang et al. [63] designed a hybrid imaging system, which consists of a light field camera and a digital single-lens reflex camera with high resolution. Additionally, two attempts were made to generate HDR light field images via the usage of various plenoptic cameras with multiple exposures [64,65].
These scientific efforts contribute to the capture of HDR light field contents; however, appropriate display systems will be necessary to visualize them. Regarding research question, QoE studies should compare the performance of the solutions mentioned above. After carrying out the necessary subjective tests on the perceptual thresholds of brightness, contrast and color, their combinations with spatial resolution and angular resolution should be addressed; hypotheses can be built on the interdependence between the extended ranges of HDR and light field resolution values. As for use cases, studying medical visualization would be particularly beneficial, since the improved contrast could advance diagnostics accuracy.

3.12. Super Resolution

One of the greatest long-term goals of light field display development is to achieve super resolution. Normally, if an observer is within a recommended viewing distance [48], then a given point on the surface of the screen of the light field display may address the two pupils of the individual with two distinct light rays. In the case of super resolution, such a point may address a single pupil with two distinct light rays.
The most important benefit of achieving super resolution is that observers may focus within light field visualization. A notable limitation of the state-of-the-art light field technology is that no matter the perceived depth, the eyes of the observers always focus on the plane of the screen. With super resolution, one may focus on the different spatial portions of the visualized content.
At the time of writing this paper, the ray density required for super resolution has not yet been implemented in a light field display. Once it is achieved, QoE research should address the perceptual effects that arise from ray density variations around the threshold of super resolution. For example, what impact would the gradual appearance or disappearance of in-content depth differentiation have on the Human Visual System (HVS)? How would super resolution affect the perception of the other KPIs of light field visualization? Would super resolution enhance task performance on a statistically significant level? These are all important questions, and many other hypotheses are to be tested once super resolution becomes a practical reality.

3.13. Full-Parallax Imaging

The other grand milestone of light field system development is the creation of full-parallax (FP) displays. Such technology is somewhat meant to deliver the “ultimate” glasses-free visualization—particularly with a high ray density—since the correct parallax effect may be provided along both the horizontal and the vertical axis. However, while progress is indeed notable [66,67], similarly to super resolution, the emergence of FP displays is yet to come.
An array of research questions could compare the performance and subjective quality of HPO and FP light field displays. Evidently, one of the most relevant KPIs would be angular resolution, the investigation of which is highly valuable to light field QoE research, since scientific knowledge on the perceived smoothness of the vertical parallax is currently lacking. Furthermore, it is technically possible to create light fields that have different horizontal and vertical angular resolutions. A greater ray density along the horizontal axis is much more relevant, since the human eyes are horizontally separated and the majority of use-case-related motion paths have dominant horizontal components. Finally, it should be intensely studied whether it makes any practical sense to deploy FP displays to certain use cases. For example, in the case of a light field cinema, the vertical disposition of the eyes (i.e., through the up and down movement of the head) may be negligible. However, one must not forget that the introduction of novel technologies may fundamentally reinvent—or realign—the existing concepts of specific use cases and usage contexts.

4. Proposed Methodologies

In this section, we propose research methodologies that are relevant to the in-depth investigation of the novel research questions.

4.1. Physiological Measurements

Recent studies on measuring the perception of test participants through physiological signals have provided rich insight into relevant features. Such experimental approaches should be extended to studies on light field visualization. The work of Engelke et al. [68] discusses physiological measurement methods which may be grouped in three categories: (i) cognitive interaction techniques via the central nervous system (CNS) to measure brain activity such as electroencephalography (EEG) and near-infrared spectroscopy (NIRS); (ii) nervous system response through measurements associated to the sympathetic division of the autonomic nervous system (ANS) and measuring electrocardiography (ECG), electrodermal activity (EDA) and pupil diameter states to indicate the level of the test participant’s cognitive state; and (iii) eye tracking, eye blinking rate and pupil dilation may also give valuable insight into cognitive state, engagement level and fatigue state [69].
While these methods were used in preliminary studies to predict the sense of presence in immersive audiovisual communications on conventional 2D displays [70], no such studies exist for light field visualization. Therefore, research methodologies should adopt the usage of physiological measurements, particularly to assess the saliency of light field contents.

4.2. Viewing Conditions

Viewing conditions, already on their own, form research questions and hypotheses that should be investigated and tested. However, they are also highly relevant to the experimental setups of QoE studies when they are not the center of attention. This is applicable to the viewing angle, viewing distance, observer motion and lighting conditions. When addressing specific use cases of light field visualization, primarily the settings of viewing conditions simulate the context. Therefore, QoE studies should aim to progress towards realistic parameters of viewing conditions in order to properly explore the use cases. It is important to highlight that these conditions must be satisfied simultaneously in order to achieve the desired test validity. Yet, there should be a priority between the different viewing conditions during experimental progress. The higher priority should be enjoyed by viewing distance and observer motion, due to their relevance already exclaimed by the scientific literature [34,35,36,48]. Once the visual phenomena are sufficiently studied, then QoE research should proceed to repeat most of the major experiments under different lighting conditions. It should be noted that the perception of the investigated effects may vary when the environmental illumination changes. For example, it is possible that the angular compensation that originates from observer motion acts differently when external light sources (e.g., direct sunlight) reach the screen of the light field display.

4.3. Rating Methodologies

The QoE community has many standards and best practices regarding rating methodologies. However, not every option is valid in the context of light field visualization. The most apparent issue rises around the simultaneous stimuli method. This method means that two representations of a given content are presented to a test participant. This can be ACR assessment (e.g., rate the two stimuli on the same scale), DCR assessment (e.g., compare the impaired stimulus to the reference stimulus) or a paired comparison (e.g., compare the two impaired stimuli). The most common form of simultaneous stimuli is a side-by-side presentation. However, due to the angular nature of light field displays, the two representations would be seen from different perspectives, and hence their comparison is not valid. Of course, it is possible to construct visual contents that are symmetrical along the horizontal axis, but this is not practical and definitely not applicable to most use cases. Therefore, only sequential, temporally separated tests are advised.

5. Discussion

In this section, we discuss considerations regarding the QoE studies of light field visualization. The discussion is centered around test participants; after all, they provide the subjective data that are the primary output of such scientific efforts.

5.1. Subjective Test Duration and Test Participant Fatigue

As mentioned earlier in the paper, a QoE study on light field visualization quality typically involves one or two test variables. Yet, in order to progress towards the testing of more complex, realistic use cases, the number of test variables may significantly rise. The main issue with this is that exhaustive testing addresses every single combination, which may result in greatly extended test duration. Let us suppose that an experiment considers five test variables. If each variable has four possible values, then the number of test conditions in an exhaustive test is 1024, which is simply unmanageable in a subjective study. If the number of possible values is only two, then there are 32 test conditions. However, the variables are rather limited in this case. Still, a reasonable study can be designed with these parameters. Let us also assume that every stimulus is only shown once and that an ACR scale is used to collect the quality ratings. If there are only three source contents, then there are 96 stimuli. If static light field images are evaluated and each is shown for 20 s, then the duration of visualization is already over half an hour. However, this needs to be extended with an evaluation period for each stimulus (let us assume 10 s) and a separation screen between the stimuli (let us assume 5 s). This means that the total duration of the test in this example is almost an hour.
An hour, in experimental practice, can be tolerated, particularly if breaks are included. Yet, this hypothetical duration was calculated for a test with severely limited test variable values, which may not be compatible with many research questions. One solution to contain the test duration within reasonable limits is to abandon the approach of exhaustive testing. In such a case, not every variable combination forms a test condition. The number of source stimuli should not be reduced, as this increases the content-dependency of the results. Separation screens cannot be left out, as the direct transition between stimuli may greatly bias judgment. The reductions in observation and assessment times also lead to the loss of rating accuracy. Another option for the containment of total duration is that a test participant rates only a portion of the available test stimuli. However, this is only applicable to certain methodologies and it is generally less advisable than abandoning the approach of exhaustive testing.
The greatest risk of having long subjective tests is the potential fatigue of test participants. At the time of writing this paper, the scientific community is not aware of any technology-specific fatigue factor of light field visualization. However, even if the observation or usage of light field displays does not additionally exhaust test participants, fatigue may still occur if the test is too long. Again, it needs to be added that subjective studies have not addressed perceptual fatigue in the context of light field visualization.

5.2. Physical and Mental Conditions of Test Participants

As the outcome of a subjective test depends on the participants’ ability to visualize and interpret the content, while conducting subjective tests it is crucial to consider the physical and mental conditions of test participants. Since the presented contents are of 3D nature, one of the key aspects of participant selection criteria should be their ability to perceive 3D content. Individuals with a stereo blindness condition would not be able to perceive stereoscopic depth; hence, preliminary testing of participants via the Random-dot Butterfly test is required—unless the research question focuses on how the under-represented observers perceive light field visualization.
Mental conditions and the compliance of test participants may be evaluated through consistency checks, either by repeating some test conditions for each subject and checking the consistency of the associated scores or, as proposed by Pezzulli et al. [71], analyzing the relationship between scores of the different test participants and content in complete tests (i.e., where the same set of test participants scores all the processed contents under test).
The conditions of test participants after subjective tests should also be monitored for possible visual discomfort, possibly resulting in dizziness or sickness, which can occur after any type of 3D visualization: virtual stereoscopic stimuli, such as in 3D displays, may cause improper vergence and accommodative responses, which can thereby result in visual discomfort [72,73]. Although it has been hypothesized that light field displays do not cause visual discomfort and nausea as other forms of 3D visualization, in particular in the case of high angular resolution, more relevant studies are needed.

5.3. Training and Screening of Test Participants

Training the viewing subjects before subjective tests on specific light field content is important, as well as testing stereo blindness with classical tests. In particular, many test subjects have not experienced the visualization of light field content before the tests and the design of an appropriate training session is important to avoid the scores of the first tests being affected. As indicated by earlier research [27], insufficient training to the novel visual phenomena may result in inconsistent or invalid ratings. The case of low angular resolution is particularly relevant, since the degradation suffered by the smoothness of the parallax effect is completely lacking from our daily perception of the natural 3D world. Appropriately designed training sessions can also limit the cognitive bias effects discussed among the novel research questions.

5.4. Feasibility and Complexity of Novel Subjective Tests

The most straightforward research questions are viewing conditions, inter-user effects, cognitive bias and under-represented observers. Viewing conditions increase the complexity of the research only if objective measurements of disparity are involved [35]. Of course, certain experimental setups may be difficult to implement if, for instance, the research environment is not suitable for investigating specific parameters (i.e., if the location of the subjective study is limited in space and higher viewing distances are thus not supported). Addressing inter-user effects requires light field displays that provide larger, wider FOVs. This may not be relevant if the test participants are seated. The vast majority of cognitive-bias-related research questions do not increase complexity at all. For instance, it can be induced by particular descriptors of quality (i.e., labels). Investigating the perception and task performance of under-represented observers only makes research less straightforward in the sense that certain test participants—e.g., those with a specific type of colorblindness—may be difficult to recruit. This may prolong the total time period dedicated to carrying out the subjective tests, or may reduce the total number of test participants, which is counterproductive to statistical analysis and thus, weakens the findings.
Subjective studies that involve physiological measurements are de facto more complex, due to the additional devices that collect data from the test participants. This is particularly applicable to tests on immersion. If immersion is assessed by self-reported ratings, then the subjective test is rather straightforward. However, if immersion is measured, for instance, via EEG or EDA, then the complexity of the physiological measurement notably contributes to the global complexity of the subjective study. Similar statements can be made about the measurement of test participant fatigue.
System-related research questions are considerably more challenging to address in terms of implementation. If the study investigates direct technical comparisons, then two devices must be located in the same research environment. This not only poses additional requirements towards the research environment, but it also means that both devices should be available simultaneously. Provisioning dedicated displays can be particularly tricky, since at the time of writing this paper, all of them are prototypes. Research questions on interaction and HCI can be more straightforward if the research question does not necessitate special hardware. For example, using conventional controls—as in the work of Tamboli et al. [43]—does typically not increase the complexity of the research.
Evidently, the least feasible research questions currently are related to HDR visualization, super resolution and FP imaging. As no such systems are available for subjective studies, one shall have to wait for the development of the first prototypes.

6. Conclusions

In this paper, we proposed novel research questions and methodologies for studying the QoE of light field visualization, and discussed the relevant factors of such experiments. As more research institutions have started gaining access to real light field displays, it is expected that the collective output of the scientific community will steadily increase in the following years. Such novel studies shall advance the best practices and the related research methodologies, ultimately paving the way for new standards. Results regarding complex issues have the potential to significantly benefit both users and manufacturers, by enabling a user-centric yet cost-efficient perspective.

Author Contributions

Conceptualization, P.A.K. and M.G.M; methodology, P.A.K., R.R.T. and M.G.M.; validation, R.R.T.; investigation, P.A.K. and R.R.T.; writing—original draft preparation, P.A.K., R.R.T., E.S., M.G.M. and M.G.; writing—review and editing, A.S.; supervision, P.A.K. and A.S.; project administration, P.A.K.; funding acquisition, M.G.M. All authors have read and agreed to the published version of the manuscript.

Funding

This open access publication was funded by Kingston University, London, UK.

Institutional Review Board Statement

The authors did not conduct any experiment with test participants.

Informed Consent Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Adhikarla, V.K.; Vinkler, M.; Sumin, D.; Mantiuk, R.K.; Myszkowski, K.; Seidel, H.P.; Didyk, P. Towards a quality metric for dense light fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 21–26 July 2017; pp. 58–67. [Google Scholar]
  2. Bakir, N.; Fezza, S.A.; Hamidouche, W.; Samrouth, K.; Déforges, O. Subjective Evaluation of Light Field Image Compression Methods based on View Synthesis. In Proceedings of the 27th European Signal Processing Conference (EUSIPCO), Coruña, Spain, 2–6 September 2019; pp. 1–5. [Google Scholar]
  3. Battisti, F.; Bosc, E.; Carli, M.; Le Callet, P.; Perugia, S. Objective image quality assessment of 3D synthesized views. Signal Process. Image Commun. 2015, 30, 78–88. [Google Scholar] [CrossRef]
  4. Carballeira, P.; Gutiérrez, J.; Morán, F.; Cabrera, J.; García, N. Subjective evaluation of super multiview video in consumer 3D displays. In Proceedings of the Seventh International Workshop on Quality of Multimedia Experience (QoMEX), Costa Navarino, Messinia, 26–29 May 2015; pp. 1–6. [Google Scholar]
  5. Palma, E.; Battisti, F.; Carli, M.; Astola, P.; Tabus, I. Subjective Quality Evaluation of Light Field Data Under Coding Distortions. In Proceedings of the 28th European Signal Processing Conference (EUSIPCO), Dublin, Ireland, 23–27 August 2021; pp. 526–530. [Google Scholar]
  6. Perra, C. Assessing the quality of experience in viewing rendered decompressed light fields. Multimed. Tools Appl. 2018, 77, 21771–21790. [Google Scholar] [CrossRef]
  7. Paudyal, P.; Battisti, F.; Neri, A.; Carli, M. A study of the impact of light fields watermarking on the perceived quality of the refocused data. In Proceedings of the 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video (3DTV-CON), Lisbon, Portugal, 8–10 July 2015; pp. 1–4. [Google Scholar]
  8. Recio, R.; Carballeira, P.; Gutiérrez, J.; García, N. Subjective assessment of super multiview video with coding artifacts. IEEE Signal Process. Lett. 2017, 24, 868–871. [Google Scholar] [CrossRef] [Green Version]
  9. Shi, L.; Zhao, S.; Zhou, W.; Chen, Z. Perceptual evaluation of light field image. In Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 41–45. [Google Scholar]
  10. Wen, W.; Wei, K.; Fang, Y.; Zhang, Y. Visual Quality Assessment for Perceptually Encrypted Light Field Images. IEEE Trans. Circuits Syst. Video Technol. 2020, 31, 2522–2534. [Google Scholar] [CrossRef]
  11. Shi, L.; Zhou, W.; Chen, Z.; Zhang, J. No-reference light field image quality assessment based on spatial-angular measurement. IEEE Trans. Circuits Syst. Video Technol. 2019, 30, 4114–4128. [Google Scholar] [CrossRef] [Green Version]
  12. Luo, Z.; Zhou, W.; Shi, L.; Chen, Z. No-reference light field image quality assessment based on micro-lens image. In Proceedings of the 2019 Picture Coding Symposium (PCS), Ningbo, China, 12–15 November 2019; pp. 1–5. [Google Scholar]
  13. Zhou, W.; Shi, L.; Chen, Z.; Zhang, J. Tensor oriented no-reference light field image quality assessment. IEEE Trans. Image Process. 2020, 29, 4070–4084. [Google Scholar] [CrossRef]
  14. Shan, L.; An, P.; Liu, D.; Ma, R. Subjective evaluation of light field images for quality assessment database. In International Forum on Digital TV and Wireless Multimedia Communications; Springer: Berlin, Germany, 2017; pp. 267–276. [Google Scholar]
  15. Viola, I.; Ebrahimi, T. VALID: Visual quality assessment for light field images dataset. In Proceedings of the 2018 Tenth International Conference on Quality of Multimedia Experience (QoMEX), Sardinia, Italy, 29 May–1 June 2018; pp. 1–3. [Google Scholar]
  16. Adhikarla, V.K.; Jakus, G.; Sodnik, J. Design and evaluation of freehand gesture interaction for light field display. In International Conference on Human-Computer Interaction; Springer: Berlin, Germany, 2015; pp. 54–65. [Google Scholar]
  17. Adhikarla, V.K.; Sodnik, J.; Szolgay, P.; Jakus, G. Exploring direct 3D interaction for full horizontal parallax light field displays using leap motion controller. Sensors 2015, 15, 8642–8663. [Google Scholar] [CrossRef]
  18. Ahar, A.; Chlipala, M.; Birnbaum, T.; Zaperty, W.; Symeonidou, A.; Kozacki, T.; Kujawinska, M.; Schelkens, P. Suitability analysis of holographic vs light field and 2D displays for subjective quality assessment of Fourier holograms. Opt. Express 2020, 28, 37069–37091. [Google Scholar] [CrossRef]
  19. Cserkaszky, A.; Barsi, A.; Kara, P.A.; Martini, M.G. To interpolate or not to interpolate: Subjective assessment of interpolation performance on a light field display. In Proceedings of the IEEE International Conference on Multimedia & Expo (ICME) Workshops, Hong Kong, China, 10–14 July 2017; pp. 55–60. [Google Scholar]
  20. Cserkaszky, A.; Kara, P.A.; Barsi, A.; Martini, M.G. Expert evaluation of a novel light-field visualization format. In Proceedings of the 3DTV-Conference: The True Vision-Capture, Transmission and Display of 3D Video, Helsinki, Finland, 3–5 June 2018; pp. 1–4. [Google Scholar]
  21. Cserkaszky, A.; Kara, P.A.; Tamboli, R.R.; Barsi, A.; Martini, M.G.; Bokor, L.; Balogh, T. Angularly continuous light-field format: Concept, implementation, and evaluation. J. Soc. Inf. Disp. 2019, 27, 442–461. [Google Scholar] [CrossRef]
  22. Darukumalli, S.; Kara, P.A.; Barsi, A.; Martini, M.G.; Balogh, T.; Chehaibi, A. Performance comparison of subjective assessment methodologies for Light Field Displays. In Proceedings of the International Symposium on Signal Processing and Information Technology, Limassol, Cyprus, 12–14 December 2016; pp. 28–33. [Google Scholar]
  23. Darukumalli, S.; Kara, P.A.; Barsi, A.; Martini, M.G.; Balogh, T. Subjective quality assessment of zooming levels and image reconstructions based on region of interest for light field displays. In Proceedings of the International Conference on 3D Imaging (IC3D), Liege, Belgium, 13–14 December 2016; pp. 1–6. [Google Scholar]
  24. Dricot, A.; Jung, J.; Cagnazzo, M.; Pesquet, B.; Dufaux, F.; Kovács, P.T.; Adhikarla, V.K. Subjective evaluation of Super Multi-View compressed contents on high-end light-field 3D displays. Signal Process. Image Commun. 2015, 39, 369–385. [Google Scholar] [CrossRef]
  25. Kara, P.A.; Kovács, P.T.; Martini, M.G.; Barsi, A.; Lackner, K.; Balogh, T. From a different point of view: How the field of view of light field displays affects the willingness to pay and to use. In Proceedings of the 8th International Conference on Quality of Multimedia Experience (QoMEX), Lisbon, Portugal, 6–8 June 2016. [Google Scholar]
  26. Kara, P.A.; Kovacs, P.T.; Vagharshakyan, S.; Martini, M.G.; Barsi, A.; Balogh, T.; Chuchvara, A.; Chehaibi, A. The effect of light field reconstruction and angular resolution reduction on the quality of experience. In Proceedings of the 12th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), Naples, Italy, 28 November–1 December 2016; pp. 781–786. [Google Scholar]
  27. Kara, P.A.; Martini, M.G.; Kovács, P.T.; Imre, S.; Barsi, A.; Lackner, K.; Balogh, T. Perceived quality of angular resolution for light field displays and the validity of subjective assessment. In Proceedings of the International Conference on 3D Imaging (IC3D), Liege, Belgium, 13–14 December 2016; pp. 1–7. [Google Scholar]
  28. Kara, P.A.; Kovacs, P.T.; Martini, M.G.; Barsi, A.; Lackner, K.; Balogh, T. Viva la resolution: The perceivable differences between image resolutions for light field displays. In Proceedings of the 5th ISCA/DEGA Workshop on Perceptual Quality of Systems (PQS), Berlin, Germany, 29–31 August 2016; pp. 107–111. [Google Scholar]
  29. Kara, P.A.; Cserkaszky, A.; Darukumalli, S.; Barsi, A.; Martini, M.G. On the edge of the seat: Reduced angular resolution of a light field cinema with fixed observer positions. In Proceedings of the Ninth International Conference on Quality of Multimedia Experience (QoMEX), Erfurt, Germany, 29 May–2 June 2017; pp. 1–6. [Google Scholar]
  30. Kara, P.A.; Cserkaszky, A.; Barsi, A.; Papp, T.; Martini, M.G.; Bokor, L. The interdependence of spatial and angular resolution in the quality of experience of light field visualization. In Proceedings of the International Conference on 3D Immersion (IC3D), Brussels, Belgium, 11–12 December 2017; pp. 1–8. [Google Scholar]
  31. Kara, P.A.; Kovács, P.T.; Vagharshakyan, S.; Martini, M.G.; Imre, S.; Barsi, A.; Lackner, K.; Balogh, T. Perceptual quality of reconstructed medical images on projection-based light field displays. In eHealth 360∘; Springer: Berlin/Heidelberg, Germany, 2017; pp. 476–483. [Google Scholar]
  32. Kara, P.A.; Cserkaszky, A.; Martini, M.G.; Barsi, A.; Bokor, L.; Balogh, T. Evaluation of the concept of dynamic adaptive streaming of light field video. IEEE Trans. Broadcast. 2018, 64, 407–421. [Google Scholar] [CrossRef]
  33. Kara, P.A.; Tamboli, R.R.; Cserkaszky, A.; Martini, M.G.; Barsi, A.; Bokor, L. The perceived quality of light-field video services. In Applications of Digital Image Processing XLI; International Society for Optics and Photonics: Bellingham, WA, USA, 2018; Volume 10752, pp. 138–147. [Google Scholar]
  34. Kara, P.A.; Tamboli, R.R.; Cserkaszky, A.; Martini, M.G.; Barsi, A.; Bokor, L. The viewing conditions of light-field video for subjective quality assessment. In Proceedings of the International Conference on 3D Immersion (IC3D), Brussels, Belgium, 5 December 2018; pp. 1–8. [Google Scholar]
  35. Kara, P.A.; Tamboli, R.R.; Cserkaszky, A.; Barsi, A.; Simon, A.; Kusz, A.; Bokor, L.; Martini, M.G. Objective and subjective assessment of binocular disparity for projection-based light field displays. In Proceedings of the International Conference on 3D Immersion (IC3D), Brussels, Belgium, 11–12 December 2019; pp. 1–8. [Google Scholar]
  36. Kara, P.A.; Guindy, M.; Balogh, T.; Simon, A. The perceptually-supported and the subjectively-preferred viewing distance of projection-based light field displays. In Proceedings of the International Conference on 3D Immersion (IC3D), online, 8 December 2021; pp. 1–8. [Google Scholar]
  37. Kovács, P.T.; Lackner, K.; Barsi, A.; Balázs, Á.; Boev, A.; Bregović, R.; Gotchev, A. Measurement of perceived spatial resolution in 3D light-field displays. In Proceedings of the International Conference on Image Processing, Paris, France, 27–30 October 2014; pp. 768–772. [Google Scholar]
  38. Kovács, P.T.; Boev, A.; Bregović, R.; Gotchev, A. Quality measurements of 3D light-field displays. In Proceedings of the Eighth International Workshop on Video Processing and Quality Metrics for Consumer Electronics, Chandler, AZ, USA, 30–31 January 2014; pp. 1–6. [Google Scholar]
  39. Kovács, P.T.; Bregović, R.; Boev, A.; Barsi, A.; Gotchev, A. Quantifying Spatial and Angular Resolution of Light-Field 3-D Displays. IEEE J. Sel. Top. Signal Process. 2017, 11, 1213–1222. [Google Scholar] [CrossRef]
  40. Tamboli, R.; Vupparaboina, K.K.; Ready, J.; Jana, S.; Channappayya, S. A subjective evaluation of true 3D images. In Proceedings of the International Conference on 3D Imaging (IC3D), Liege, Belgium, 9–10 December 2014; pp. 1–8. [Google Scholar]
  41. Tamboli, R.R.; Appina, B.; Channappayya, S.; Jana, S. Super-multiview content with high angular resolution: 3D quality assessment on horizontal-parallax lightfield display. Signal Process. Image Commun. 2016, 47, 42–55. [Google Scholar] [CrossRef]
  42. Tamboli, R.R.; Appina, B.; Channappayya, S.S.; Jana, S. Achieving high angular resolution via view synthesis: Quality assessment of 3D content on super multiview lightfield display. In Proceedings of the International Conference on 3D Immersion (IC3D), Brussels, Belgium, 11–12 December 2017; pp. 1–8. [Google Scholar]
  43. Tamboli, R.R.; Kara, P.A.; Cserkaszky, A.; Barsi, A.; Martini, M.G.; Jana, S. Canonical 3D object orientation for interactive light-field visualization. In Applications of Digital Image Processing XLI; International Society for Optics and Photonics: Bellingham, WA, USA, 2018; Volume 10752, pp. 77–83. [Google Scholar]
  44. Zhang, X.; Braley, S.; Rubens, C.; Merritt, T.; Vertegaal, R. LightBee: A self-levitating light field display for hologrammatic telepresence. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Scotland, UK, 4–9 May 2019; pp. 1–10. [Google Scholar]
  45. Kara, P.A.; Tamboli, R.R.; Doronin, O.; Cserkaszky, A.; Barsi, A.; Nagy, Z.; Martini, M.G.; Simon, A. The key performance indicators of projection-based light field visualization. J. Inf. Disp. 2019, 20, 81–93. [Google Scholar] [CrossRef] [Green Version]
  46. Agarwal, R.; Karahanna, E. Time flies when you’re having fun: Cognitive absorption and beliefs about information technology usage. MIS Q. 2000, 665–694. [Google Scholar] [CrossRef]
  47. Volino, M.; Mustafa, A.; Guillemaut, J.Y.; Hilton, A. Light Field Video for Immersive Content Production. In Real VR–Immersive Digital Reality; Springer: Berlin/Heidelberg, Germany, 2020; pp. 33–64. [Google Scholar]
  48. Kara, P.A.; Barsi, A.; Tamboli, R.R.; Guindy, M.; Martini, M.G.; Balogh, T.; Simon, A. Recommendations on the viewing distance of light field displays. In Digital Optical Technologies 2021; International Society for Optics and Photonics: Bellingham, WA, USA, 2021; Volume 11788. [Google Scholar]
  49. Murray, E.G.; Neumann, D.L.; Moffitt, R.L.; Thomas, P.R. The effects of the presence of others during a rowing exercise in a virtual reality environment. Psychol. Sport Exerc. 2016, 22, 328–336. [Google Scholar] [CrossRef]
  50. Asch, S.E. Studies of independence and conformity: I. A minority of one against a unanimous majority. Psychol. Monogr. Gen. Appl. 1956, 70, 1. [Google Scholar] [CrossRef] [Green Version]
  51. Nickerson, R.S. Confirmation bias: A ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 1998, 2, 175–220. [Google Scholar] [CrossRef]
  52. Loftus, E.F.; Hoffman, H.G. Misinformation and memory: The creation of new memories. J. Exp. Psychol. 1989, 118, 100. [Google Scholar] [CrossRef]
  53. Tversky, A.; Kahneman, D. The framing of decisions and the psychology of choice. In Behavioral Decision Making; Springer: Berlin/Heidelberg, Germany, 1985; pp. 25–41. [Google Scholar]
  54. Sherif, M.; Taub, D.; Hovland, C.I. Assimilation and contrast effects of anchoring stimuli on judgments. J. Exp. Psychol. 1958, 55, 150. [Google Scholar] [CrossRef] [Green Version]
  55. Kara, P.A.; Robitza, W.; Pinter, N.; Martini, M.G.; Raake, A.; Simon, A. Comparison of HD and UHD video quality with and without the influence of the labeling effect. Qual. User Exp. 2019, 4, 1–29. [Google Scholar] [CrossRef] [Green Version]
  56. Kara, P.A.; Cserkaszky, A.; Martini, M.G.; Bokor, L.; Simon, A. The effect of labeling on the perceived quality of HDR video transmission. Cogn. Technol. Work. 2020, 22, 585–601. [Google Scholar] [CrossRef] [Green Version]
  57. Oyama, T.; Yamamura, T. The effect of hue and brightness on the depth perception in normal and color-blind subjects. Psychologia 1960, 3, 191–194. [Google Scholar] [CrossRef]
  58. Cserkaszky, A.; Barsi, A.; Nagy, Z.; Puhr, G.; Balogh, T.; Kara, P.A. Real-time light-field 3D telepresence. In Proceedings of the 7th European Workshop on Visual Information Processing (EUVIP), Tampere, Finland, 26–28 November 2018; pp. 1–5. [Google Scholar]
  59. Reinhard, E.; Heidrich, W.; Debevec, P.; Pattanaik, S.; Ward, G.; Myszkowski, K. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting; Morgan Kaufmann: Burlington, MA, USA, 2010. [Google Scholar]
  60. Lumsdaine, A.; Georgiev, T. The focused plenoptic camera. In Proceedings of the IEEE International Conference on Computational Photography (ICCP), San Francisco, CA USA, 16–17 April 2009; pp. 1–8. [Google Scholar]
  61. Georgiev, T.; Lumsdaine, A.; Chunev, G. Using focused plenoptic cameras for rich image capture. IEEE Comput. Graph. Appl. 2010, 31, 62–73. [Google Scholar] [CrossRef] [PubMed]
  62. Schedl, D.C.; Birklbauer, C.; Bimber, O. Coded exposure HDR light-field video recording. In Computer Graphics Forum; Wiley Online Library: New York, NY, USA, 2014; Volume 33, pp. 33–42. [Google Scholar]
  63. Wang, X.; Li, L.; Hou, G. High-resolution light field reconstruction using a hybrid imaging system. Appl. Opt. 2016, 55, 2580–2593. [Google Scholar] [CrossRef] [PubMed]
  64. Li, C.; Zhang, X. High dynamic range and all-focus image from light field. In Proceedings of the 2015 IEEE 7th International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE Conference on Robotics, Automation and Mechatronics (RAM), Siem Reap, Cambodia, 15–17 July 2015; pp. 7–12. [Google Scholar]
  65. Le Pendu, M.; Guillemot, C.; Smolic, A. High dynamic range light fields via weighted low rank approximation. In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP), Athens, Greece, 7–10 October 2018; pp. 1728–1732. [Google Scholar]
  66. Watanabe, H.; Okaichi, N.; Omura, T.; Kano, M.; Sasaki, H.; Kawakita, M. Aktina Vision: Full-parallax three-dimensional display with 100 million light rays. Sci. Rep. 2019, 9, 1–9. [Google Scholar]
  67. Gao, X.; Sang, X.; Xing, S.; Yu, X.; Yan, B.; Liu, B.; Wang, P. Full-parallax 3D light field display with uniform view density along the horizontal and vertical direction. Opt. Commun. 2020, 467, 125765. [Google Scholar] [CrossRef]
  68. Engelke, U.; Darcy, D.P.; Mulliken, G.H.; Bosse, S.; Martini, M.G.; Arndt, S.; Antons, J.N.; Chan, K.Y.; Ramzan, N.; Brunnström, K. Psychophysiology-based QoE assessment: A survey. IEEE J. Sel. Top. Signal Process. 2016, 11, 6–21. [Google Scholar] [CrossRef] [Green Version]
  69. Bang, J.W.; Heo, H.; Choi, J.S.; Park, K.R. Assessment of eye fatigue caused by 3D displays based on multimodal measurements. Sensors 2014, 14, 16467–16485. [Google Scholar] [CrossRef] [Green Version]
  70. Perrin, A.F.; Řeřábek, M.; Ebrahimi, T. Towards prediction of sense of presence in immersive audiovisual communications. Electron. Imaging 2016, 2016, 1–8. [Google Scholar] [CrossRef] [Green Version]
  71. Pezzulli, S.; Martini, M.G.; Barman, N. Estimation of Quality Scores from Subjective Tests-Beyond Subjects’ MOS. IEEE Trans. Multimed. 2020, 23, 2505–2519. [Google Scholar] [CrossRef]
  72. Solimini, A.G. Are there side effects to watching 3D movies? A prospective crossover observational study on visually induced motion sickness. PloS ONE 2013, 8, e56160. [Google Scholar] [CrossRef] [PubMed]
  73. Kim, J.; Oh, H.; Kim, W.; Choi, S.; Son, W.; Lee, S. A deep motion sickness predictor induced by visual stimuli in virtual reality. IEEE Trans. Neural Netw. Learn. Syst. 2020, 33, 554–566. [Google Scholar] [CrossRef]
Electronics 11 00953 g001 550
Figure 1. Distribution of the state-of-the-art scientific literature and open research questions.
Figure 1. Distribution of the state-of-the-art scientific literature and open research questions.
Electronics 11 00953 g001
Table
Table 1. Summary of the related work.
Table 1. Summary of the related work.
PublicationContentTest VariableRating ScaleViewing Dist.MovementDisplay
Adhikarla et al. [16,17]interactiveHCI modesNASA TLX, UEQ50 cmnoneprototype
Ahar et al. [18]imagespatial distortion5-pt. DCR5 mnone722RC
Cserkaszky et al. [19]imageangular res., interpolation7-pt. PC4.6 msidewaysC80
Cserkaszky et al. [20]imageangular res., light field format3-pt. PC4.6–6.5 mboth directionsC80
Cserkaszky et al. [21]imageangular res., light field formatbin., 5-pt. ACR, 7-pt. PC4.6–6.5 mboth directionsC80
Darukumalli et al. [22]imagezoom level5-pt. ACR, 7-pt. PC4.6 mnoneC80
Darukumalli et al. [23]imagezoom level, content alignment5-pt. ACR, 5-pt. DCR4.6 mnoneC80
Dricot et al. [24]videocompression5-pt. DCR6 mnoneC80
Kara et al. [25]imageFOV10-pt. ACRup to 5 mboth directions80WLT
Kara et al. [26]imageangular res., reconstruction10-pt. ACR4.6 msidewaysC80
Kara et al. [27]imageangular res.10-pt. ACR4.6 msidewaysC80
Kara et al. [28]imagespatial res.5-pt. DCR4.6–6.6 mboth directionsC80
Kara et al. [29]imageangular res.bin., 25-pt. QC4.6–5.6 mnoneC80
Kara et al. [30]imageangular res., spatial res.7-pt. PC4.6 msidewaysC80
Kara et al. [31]imageangular res., reconstruction10-pt. ACR4.6 msidewaysC80
Kara et al. [32]videoangular res., spatial res.5-pt. PC4.6 msidewaysC80
Kara et al. [33]videoangular res., spatial res.5-pt. DCR, 7-pt. PC4.6 msidewaysC80
Kara et al. [34]videoangular res., spatial res.5-pt. DCR4.6 msidewaysC80
Kara et al. [35]imageviewing distance7-pt. PC4.5–7.5 mnoneC80
Kara et al. [36]imageviewing distancen/a0.25–8 mback and forth80WLT, C80
Kovacs et al. [37]imagesymbol sizen/a5 mnoneC80
Kovacs et al. [38]imagegrating densitybin.5 mboth directionsC80
Kovacs et al. [39]imagesymbol sizen/a80 cmnone80WLT
Tamboli et al. [40,41]imagespatial distortion5-pt. ACR2.44 mnone721RC
Tamboli et al. [42]imageangular distortion5-pt. ACR2.44 mnone721RC
Tamboli et al. [43]interactivecontent orientationn/a4.6 msidewaysC80
Zhang et al. [44]live videon/an/a1.2–3.6 mboth directionsprototype
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Kara, P.A.; Tamboli, R.R.; Shafiee, E.; Martini, M.G.; Simon, A.; Guindy, M. Beyond Perceptual Thresholds and Personal Preference: Towards Novel Research Questions and Methodologies of Quality of Experience Studies on Light Field Visualization. Electronics 2022, 11, 953. https://doi.org/10.3390/electronics11060953

AMA Style

Kara PA, Tamboli RR, Shafiee E, Martini MG, Simon A, Guindy M. Beyond Perceptual Thresholds and Personal Preference: Towards Novel Research Questions and Methodologies of Quality of Experience Studies on Light Field Visualization. Electronics. 2022; 11(6):953. https://doi.org/10.3390/electronics11060953

Chicago/Turabian Style

Kara, Peter A., Roopak R. Tamboli, Edris Shafiee, Maria G. Martini, Aniko Simon, and Mary Guindy. 2022. "Beyond Perceptual Thresholds and Personal Preference: Towards Novel Research Questions and Methodologies of Quality of Experience Studies on Light Field Visualization" Electronics 11, no. 6: 953. https://doi.org/10.3390/electronics11060953

APA Style

Kara, P. A., Tamboli, R. R., Shafiee, E., Martini, M. G., Simon, A., & Guindy, M. (2022). Beyond Perceptual Thresholds and Personal Preference: Towards Novel Research Questions and Methodologies of Quality of Experience Studies on Light Field Visualization. Electronics, 11(6), 953. https://doi.org/10.3390/electronics11060953

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop