Experience Design, Virtual Reality and Media Hybridization for the Digital Communication Inside Museums

Abstract

Experience design, both in real and in virtual museums, is very complex to be planned, even more when digital contents are juxtaposed to real collections. Researchers in this field, curators, creatives and software developers must work together in order to evolve towards a more efficient interconnection among visitors, collections and digital applications. This paper deals with such an interconnection, providing a theoretical background and practical guidelines, on the basis of museum studies and of the author’s research experience in this domain, supported by the results of surveys carried out on European museums visitors dealing with digital technologies. Media hybridization has a long and evolving tradition and it can contribute to transmit culture in engaging way to the public, respecting the scientific plausibility of contents. The choice of narrative structures and styles, as well as of interaction paradigms and technologies, is deeply conditioned by a series of factors that will be examined in detail. A general “direction” is needed to arouse in the visitor a feeling of confidence and trust, expectation and discovery that makes him/her feel at the center of an emotional and creative experience, of a progressive appropriation of meaning. Various typologies of experiences will be discussed and compared.

1. Introduction

Today more and more museums are embracing the digital challenge, but there is not yet a real and shared awareness of how to make virtual and real contents interact and how to design the best experience for the public, aiming at cultural transmission.

Despite the great contribution that multimedia, virtual reality and mixed reality can offer to the museum sector, many curators still seem rather reluctant to benefit from the full potential of these languages, especially in permanent outfitting. They often capture those aspects that are more compatible with the traditional didactic approach, remaining suspicious especially towards the emotional components of narration. To tell a story, in fact, means to formulate a hypothesis which implies the creation of a possible “reconstruction”, related to both the physical aspect of the cultural heritage and its contexts, the human activities, behaviors, thoughts, values. A story is a bridge between reality and imagination [1]. Usually museum curators prefer to be “neutral” regarding the artifacts, and they avoid to tell or to suggest the visitors anything else beyond the pure evidence in the form of captions or descriptions.

On the other hand, despite the professionalisms working in the digital domain are rapidly growing and greatly specializing their skills, digital creators working for museums are not always aware about the design of an appropriate user experience for the museum’s communication needs, aiming at a harmonious use of technology with respect to the contents, visit paths and target. Thus, the exhibited collections remain lacking in legibility, contextualization and pertinent narration.

The purpose this work aims at is to propose best practices and guidelines on “experience design” for museums, focusing on different typologies of experiences, connected to different technologies, interactive or not, immersive or not, and different “scenarios” of use. It is addressed to those working on the experience design of museum visitors, curators and creative, as a survey of existing approaches and lessons learnt for each one, that can lead to an advancement in this filed. The contribution of this research comes from the author’s academic background as art historian and expert in preservation of cultural heritage and from her long research experience, since 1998, at the Institute of Cognitive Science and Technologies and at the Institute of Technologies Applied to the Cultural Heritage, both part of the National Research Council in Italy, where she has being dealing with cultural heritage documentation and communication using digital technologies. Several virtual museums, multimedia and virtual reality projects has been conceived and realized by the author, together with her research group, experimenting new narrative forms and interaction paradigms, and they have been addressed to diverse audiences of real museums and cultural sites, in Italy and in Europe. From many years this experience has being accompanied and supported by user experience evaluations on thousands of visitors, to test the effective impact of such projects. The continuous outcomes are extremely useful to have a better understanding of the public, of the benefits and the limits of the proposed solutions, and therefore they constantly help to increase awareness and to improve our research. On this base some priority objectives of the “experience design” intended for the museum’s public are here defined and proposed, and they constitute the “leitmotiv” of this paper:

• improvement of the integration between virtual and real contents of the museum, in order to make them synergic and harmonious with the space, the paths, the visitors’ needs;
• hybridization of media, bringing together different paradigms and languages of “representation” (virtual reality, theater, cinema, sound and light design, mixed and augmented reality, holography, game), in order to involve and guide the visitor through a vivid and incisive narrative cultural experience.

Embodiment, emotional involvement, multisensory solicitations are essential elements in the creative and intuitive processes of human nature and in the creation of an experience [2]. They are directly connected to the emotional intelligence of humans [3] and are decisive in encouraging a contact between our sensitivity and the profound essence of the cultural heritage [4].

Narration is a central theme because it implies a series of choices that are not only creative and technological but also “ethical” and cultural, inherent the respect of the correctness and of the scientific plausibility of contents, while using agile and multiform tools such as the digital ones.

Never enough attention has been devoted to narrative forms dedicated to museum heritage (especially in the field of human sciences). Research has always looked elsewhere: digitization and documentation, measurement, classification, optimization methods, archiving and standardization, visualization and rendering, interaction metaphors, and technological development. Computer scientists and knowledge theorists have devoted themselves to these challenging topics, aiming at new advancements, in fact scientific publications in the domain of digital cultural heritage deal mostly with such subjects. Narration, instead, is deeply rooted in human societies and traditions, since their origin. Thus, it has been taken for granted and considered as subsidiary, or as a not urgent issue, something more pertinent the curators’ side rather than the digital domain side. Similarly, the results of surveys carried out on the public of museums have not been sufficiently considered by the curators, nor by the digital contents developers, in order to improve the museums’ cultural offer. Most of them are simple customer satisfaction investigations, instead of deeper analyses on usability and educative impact; therefore, results have remained inside museums, unpublished in conferences.

The effectiveness of digital storytelling is conditioned by a multiplicity of factors, first of all the context in which the visitor is living his/her cultural experience: in presence or in absence of the real artifacts, staying or walking, alone or collectively, and the type of technology he/she uses.

The design and development of digital contents must also be accompanied by surveys on museum visitors, to understand the attractiveness, usability and educational potential of the digital applications proposed, in relation to the visit paths. Also, for these investigations, carefully structured methodological protocols are needed, whose criteria will be briefly presented.

In this perspective, collaboration between museum staff, researchers, creative groups and experts in user experience design is crucial, considering the public, and not the technology, as the main recipient of our creative and educational efforts.

Structure of the Paper

Section 1of the paper is a brief overview of this study, its purpose and significance in the light of the state of the art of research and curatorial practices.

The structure of the paper is also described.

Section 2 introduces the main topics around which the discussion will develop successively: the concept of “user experience” and “experience design”, especially in relation to cultural contexts, b) the challenges that museums are facing today to re-define their role and becoming “territories” of new sensory and narrative experiences and interaction with their audience. Curatorial practices are evolving towards a user-centered approach in communication, whose strategies necessarily differ in the case of real, totally virtual and hybrid museums. These latter are the subject of a deeper discussion, because here the integration between real and virtual contents, even in mixed reality experiences, poses specific problems, but also enhanced opportunities of social and cultural growth. Therefore, a focus on the concept of mixed and augmented reality aims to clarify their meaning, to let the readers understand the next dissertation. The section concludes with a definition of Virtual Reality that is the starting point of the reasoning regarding media hybridization, that will be the main topic of Section 3. In fact, in the field of digital cultural heritage virtual reality environments have represented the origin of a long research and evolution, integrating little by little paradigms, techniques, languages, motivational strategies coming from the other media. The reasons of such a contamination are connected to a progressive awareness of the importance of narration in cultural communication, and they are examined in a concise chronicle of the main steps of conceptual and technological evolution, basing on the author’s direct experience made at the National Research Council, Institute of Technologies Applied to Cultural Heritage (CNR ITABC). Therefore, Section 4 deals with the concept of narration in museums, that people demonstrate to appreciate, according to the survey carried out in various museums in the last ten years. Reasons, techniques, ethical criteria, limits and opportunities of digital storytelling are examined for guided or interactive experiences, in physical and in virtual spaces.

The subject of Section 5 is mixed reality (already defined in Section 2): How and where to use it, how to design experiences in mobility, which technologies are now available and how to choose them in relation to the context.

Table 1. The different types of experiences that will be discussed in the paper, their definitions and characteristics.

Type of Experience	Application	Definition	Characteristics	Case studies and User Experience Evaluation (UX)
Exclusively digital	Desktop Virtual reality (VR)	Artificial 3D interactive environment simulating a real environment. Visualized in real time on a screen/projection	-Not immersive. -No relation with surrounding exterior context. -Single user. -Inclusion. -Embodiment. -Free or guided exploration.	1. Scrovegni Chapel Virtual Museum, 2003, Wiegand multimedia room, Eremitani Museum, Padua, CNR ITABC. https://www.cnr.it/it/focus/doc/59/musealising-the-virtual-the-virt-paper-pubblicato-nei-proceedings-della-conferenza-mondiale-vsmm-2002-creative-and-digital-culture-gyeongjou-corea-del-sud-premiato-con-il-best-paper-award-for-the-cultural-merit 2. Building Virtual Rome, two months virtual archaeology exhibition with several VR systems, 2005, Trajan Market, Rome; CNR ITABC, Luiss Univ., Federculture, Filas. UX on the general public: http://eprints.gla.ac.uk/143553/
	VR Multi-user domain	3D interactive environment where many users meet and interact in the same VR space, sharing objectives	-Same as VR. -Avatars. -Cooperation. -Recalling a game. -Potential enhanced learning.	1. Virtual Museum of the ancient Via Flaminia, 2008, Roman National Museum—Diocletian’s Baths, Rome, CNR ITABC. http://www.v-must.net/virtual-museums/vm/virtual-museum-ancient-flaminia-2008.
	VR and Natural interaction (NI)	3D environment where the user interacts using only mid-air body gestures	-Same as VR. -Sensor to capture movements. -No markers to be worn. -Enhanced embodiment. -Recalling a game. -More immediate to use, less precise input than a pointer. -The user like a performer. -Shared experience.	1. Tiber Valley virtual museum, 2014, Etruscan National Museum of Villa Giulia, Rome, CNR ITABC. UX on 117 visitors, 270 students of secondary schools, 96 students of primary schools. https://www.researchgate.net/publication/311363190_AN_INTEGRATED_METHODOLOGICAL_APPROACH_TO_EVALUATE_VIRTUAL_MUSEUMS_IN_REAL_MUSEUM_CONTEXTS 2. Etruscanning, 2013, Vatican Museums, Rome, CNR ITABC. UX on 50 visitors at Vatican Museum and 60 visitors at Archeovirtual editions in 2011-2012: https://www.academia.edu/8863496/The_Etruscanning_project_Gesture-based_interaction_and_user_experience_in_the_virtual_reconstruction_of_the_Regolini-_Galassi_tomb 3. Livia’s Villa Reloaded, 2013, Roman National Museum, Diocletian’s Baths, CNR ITABC. UX in 2015 on 125 visitors: https://www.academia.edu/31048207/LIVIAS_VILLA_RELOADED_VIRTUAL_MUSEUM_USER_EXPERIENCE_EVALUATION
	Virtual ecosystems and Artificial life (AL)	3D interactive environment with different elements and animated beings enter in relation.	-VR space dynamic and alive. -Complex behaviors. -Pre-determined behaviors in case of animations in loop. -Unpredictable dynamics in case of AL simulations.	E-Sparks, 2005, in Building Virtual Rome, Trajan Markets, Rome, Plancton Art Studio. UX: http://eprints.gla.ac.uk/143553/
	Immersive VR—Head mounted display (HMD)	HMD are equipped with a pair of displays, put on the eyes, on which stereoscopic virtual contents are displayed.	-Same as VR but immersive -360° vision. -User isolated, without perception of the surrounding real world and persons. -Great sense of presence in the virtual world.	A Night in the Forum, 2018, game, Reveal project, CNR ITABC, Imperial Fora Museum, VRTron, Sony. www.revealvr.eu
Hybrid (overlapping and comparison real/digital)	Mixed reality (MR) Holographic showcase	Pepper’s Ghost effect produced in a museum showcase including the real object	-Not immersive. -Illusion of reality. -Virtual contents on real object. -Object legibility and contextualization. -Integration with the museum’s space. -Emotional narration, dramatization. -Shared experience.	The box of stories, CEMEC project, 2017-2019, Hungarian National Museum Budapest, Allard Pierson Museum Amsterdam, Byzantine and Chrstian Museum Athens, LVRMuseum Bonn, Art and History Museum Bruxelles, Maker Faire Rome; CNR ITABC and E.V.O.CA srl. https://cemec-eu.net UX in each museum, on more than 600 visitors in total. https://www.researchgate.net/publication/325477342_THE_BOX_OF_STORIES_USER_EXPERIENCE_EVALUATION_OF_AN_INNOVATIVE_HOLOGRAPHIC_SHOWCASE_TO_COMMUNICATE_WITH_MUSEUM_OBJECTS
	Multimedia in digital frames close to real artifacts	Audio-visual contents explaining the exhibited objects, playing on screens or projections.	-Integrated in the collection. -Simple and short contents compatible with the visitor’s flow. -Collective experience. -Accessibility easier to be implemented.	Keys2Rome exhibition, 2014-2015, Imperial Fora Museum, Rome, CNR ITABC and V-Must partners. UX evaluations on 360 general visitors and on 147 students: https://www.researchgate.net/publication/300413239_Evaluation_of_the_educational_potentials_-_interactive_technologies_applied_to_Cultural_Heritage
	Live art	Live performers directly enter inside the details of a digitized artworks becoming parts of it and dramatizing it.	-Digitized artwork re-projected in the public space and brought to life by performances of actors. -Actions and stories dealing with the artwork subject. -Site-specific installation dialoguing with the surrounding space. -Process of decomposition and re-composition of the work.	School of Athens, 2019, Fara in Sabina (RI), in the context of Invisible Cities event; CNR ITABC and Teatro Potlach.
	Immersive MR See through AR display	Transparent Head Mounted Displays (HMD), superimposing the digital on a transparent display which allow to see through the reality	-Overlap between reality and virtual contents. -Useful for the user to perceive and decode the space while walking through it. -Single user experience. -Technological performance to be improved.	Art Glass https://www.art-glass.it/en/ Arkaevision Art, Tomb of the Diver in Paestum, 2019, Digital Comedia and CNR ITABC.
	Immersive Augmented Reality (AR)	Head Mounted Displays (HMD), visualization of virtual reconstruction aligned with the reality (position/orientation).	-Useful for the user to perceive and decode the space while walking through it but stopping in a position. -In the viewer the user does not see through the reality but its corresponding VR reconstruction. -Single user experience.	Caracalla in 4D, 2018, Caralla Baths archaeological park, Rome CNR, Coopculture, E.V.O.CA. srl. UX evaluation: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=2ahUKEwjM1o2ex6XlAhWPzqQKHT5mCs0QFjACegQIBBAC&url=http%3A%2F%2Fmediageo.it%2Fojs%2Findex.php%2Farcheomatica%2Farticle%2Fdownload%2F1549%2F1404&usg=AOvVaw0pb7HRdqrz6wTX487Czgf6
	AR on tablet and mobile devices	Virtual contents overlap and align on the reality. The tablet serves as a “window on the virtual world”.	-Not immersive. -Useful for the user to perceive and decode the space while walking through it. -The user is aware of the physical space around him. -Shared experience, social relations.	V-Must Keys2Rome, 2014-2015, Fraunhofer, CNR ITABC an V-Must Partners. UX on about 360 visitors: https://www.researchgate.net/publication/300077444_A_contextualized_educational_museum_experience_connecting_objects_places_and_themes_through_mobile_virtual_museums

shows a schema of the different types of experiences that will be discussed in the paper, their definitions and characteristics.

Section 6 is just a short summary of the concepts exposed in the previous parts, in the form of a list of questions that designers should ask themselves before starting a communication project in a real, virtual or hybrid museum.

In Section 7 the topic is evaluating the user experience inside museums, in order to collect feedback and improve criticism about accessibility, usability, attractiveness, educational impact of the proposed solutions. These evaluations are fundamental to improve curators’ awareness about what to do and what not to do, to verify if their expectations have a correspondence with the reality, and which are the reactions of the visitors. Strategies, methods and tools for this kind of surveys are shortly introduced, basing on the experimentations carried out in the past years, within European and national projects.

Final conclusions are in Section 8: they summarize what emerge from the different sections, coming to a coherent and enhanced interpretation.

2. Experience Design for Virtual Museums

2.1. What Is the Experience Design

An “experience” can be defined as the direct knowledge of concepts or events, personally acquired through the senses or through the perception of inner impulses of consciousness [5]. This knowledge is obtained through observation, participation, interaction with a specific sphere of the reality, in a given period of time. The experience depends on one’s individual ability to capture data through senses, to process, store and internalize them on the basis of factors conditioned by genetics, neurological relationships, and psychological variables [6]. Therefore, the experience does not have the same value in all individuals, because the observation and/or the interaction with environments, people and objects are complex and not predictable processes, influenced by a series of conditions and variables, both objective and personal. The more the experience involves the emotional sphere, the more intimate and personal it is.

An experience can easily translate in learning. It refers not only to the acquisition of new executive capacities, knowledge and skills, but also of attitudes, values, meanings and habits; it therefore concerns the development of the whole personality.

The American educator Edgar Dale (1900–1985) [7] summed up the results of his studies in the “Cone of Experience” schema (Figure 1). It shows that the quality of information we store does not depend only on the threshold of our attention, but also on the channel we activate to receive it. He distinguishes between passive and active involvement. Experiments he carried out demonstrate that a longer memorization is favored by multi-sensory experiences, active participation and interaction.

Consequently, the User Experience (UX) indicates what a person perceive, feels and behave when using a product, a system or a service, in both a real and in a digital domain. It also includes personal perceptions on aspects such as utility, simplicity, usability and efficiency of the system [8,9,10,11]. Definitively, the term User Experience is applied to the psychological and emotional manifestations of the user, deriving from:

• interaction with the technology and interfaces in general;
• technical and practical characteristics of the latter (ergonomics, shape, practicality and visibility);
• context in which it is presented.

According to ISO 9241-210 [12,13] the user experience includes all the user’s reactions that occur before, during and after the use. It thus embraces also anticipation, desire of interaction, critical elaboration, memory and re-interpretation of such a use, projection of this experience on the processes of identity construction.

The user experience is therefore a dimension of design that focuses on users, on their characteristics, their needs, and on the context of use, as theorized by the American psychologist and engineer Donald Norman in mid-nineties [14]. Norman formulated the Seven Stages of Actions: (a) Forming the goal; (b) Forming the intention; (c) Specifying an action; (d) Executing the action; (e) Perceiving the state of the world; (f) Interpreting the state of the world; (g) Evaluating the outcome. This schema is helpful to draw down 4 main principles for a good experience design:

• Visibility—By looking, the user can tell the state of the device and the alternatives for action;
• Good Conceptual Model—The designer provides a good conceptual model for the user, with
• consistency in the presentation of operations and results;
• Good mappings—It is possible to determine the relationships between actions and results;
• Feedback—The user receives full and continuous feedback about the results of the actions.

Beside, fundamental factors that must be considered when designing an experience are: (a) content typology and structure, (b) expected duration of the experience, (c) space and place of the experience, (d) condition of use. Before being involved in an experience we always pose ourselves some questions or pre-conditions: How long will it last? Where will it happen? What am I required to do? Is it of interest for me? Am I in the physical and mental conditions to deal with it? These factors emerge as fundamental also from the feedback collected in museums, observing and querying thousands of visitors interacting with collections and digital technologies.

The great development of digital technology has opened new dimensions of life and social encounters, “augmenting” the experience of the present world but also the knowledge and the contact with dimensions far in space and time: in the web, using social media, playing a game, performing actions in a 3D virtual environment, being involved in a site-specific multimedia installation, etc., we act every day in the digital world in a multimodal way, alone or with others. The impact on our culture, preferences and behaviors is enormous.

This condition has favored the growth of interest in the discipline of User Experience Design and the awareness of the importance of digital design, to be pursued according to an interdisciplinary approach, especially when dealing with heritage valorization and promotion, education and cultural transmission addressed to the public.

2.2. Hybridization between Virtual and Real Museums

According to the most recent definition by ICOM (e.g., from 2007) [15], “A museum is a non-profit, permanent institution in the service of society and its development, open to the public, which acquires, conserves, researches, communicates and exhibits the tangible and intangible heritage of humanity and its environment for the purposes of education, study and enjoyment”.

The ultimate goal of the virtual elaboration of the Cultural Heritage is the multiplication of its communicative potentialities and the ability, by the audience of the users, to better understand the cultural message it conveys. Its translation into “virtual information” therefore recapitalizes it, beyond its pure physical consistency, disseminates and spreads the message it conveys [16]. Digital technology therefore facilitates the visitors’ mental process of abstraction and imagination, making legible and recognizable what is not always easy to identify, “decode” and contextualize.

A virtual museum (VM), as defined by V-Must.net and WiMM Virtual Multimodal Museum EU networks [17,18], is “a digital entity that draws on the characteristics of a museum, in order to complement, enhance or augment the museum experience through personalization, interactivity and richness of content. Like as a physical museum, the virtual museum is also conceived to be open and accessed by the general public; moreover curatorial practices have to guarantee the scientific validation of content and the experience quality”.

Digital technologies provide new opportunities for the curators and the public, through the conception and experimentation of hybrid collections, integrating physical artifacts with digital ones, using virtual and mixed reality and storytelling strategies for communication purposes.

Moreover, users actively participate in novel ways through their narratives, before the visit, during the on-site experience, and after, mainly connecting to social media. Thus, the virtual museum opens up compelling opportunities for inclusion and accessibility, and potentially extends the encounters between museums and audience in permanent scenarios for learning, inspiration and creativity.

Therefore, advancement in the field of museology is mandatory, also at theoretical level. In fact, on one hand, the potential of digital technology translates into great professional opportunities for creative industries in terms of storytelling, visualization, media hybridization, and interaction. On the other hand, it requires theoretical and methodological awareness on the part of the institutions responsible for preservation and valorization, together with an attitude of openness towards the inclusion of new interdisciplinary teams within the curatorial practice, aiming at improving and innovating communication.

Many visitor studies highlight how the interactive experiences where the user is guided within a story are more motivating for public and more prolonged, in comparison with traditional visits among collections organized according to a taxonomical approach [19,20].

Narration, interaction, circulation of social cultural pattern are the elements around which hybridization of media and of languages grows in constant evolution.

Many factors contribute to a good experience inside museums. Beyond the already mentioned harmonious integration between real and virtual contents, other conditions should be considered: The accessibility, the comprehensibility and credibility of the stories, the communicative rhythm, the harmonic consistency of word, image and sound in the narrative performance; the setting up and the lighting; the general usability, the balance between interactive and guided access to the contents, the duration of narrative units and their dissemination along the way; the soundscape (whose management is often a thorny issue in museums); the multiple interpretative levels, the users’ profiling (a great development is still needed in this sector); the revelation of lives, people, behaviors beyond the physical exhibited objects to create an empathy, an identification with contexts and persons aiming at a better comprehension of their past.

All this contributes to the creation of a “direction”, arousing in the visitor a feeling of comfort, expectation and discovery that makes him feel at the center of a living experience of progressive appropriation of the meaning.

Certainly, when planning an experience design, some strategies can differ and must be adapted in case the user is interacting inside an exclusively digital environment/virtual museum or he/she is visiting a real place (museum, cultural site) enriched by digital contents.

Of course, the general principles of experience design introduced by Norman are valid in both cases.

2.2.1. Experiencing an Exclusively Digital Virtual Museum

In this case the digital experience is independent from the physical cultural site the virtual museum refers to (the place is no longer existing or accessible, or the virtual museum gather artifacts and contents coming from different places, creating an ideal virtual collection). In this case the virtual museum is self-explaining, the user can be sitting in front of a workstation (at home, at school, etc.). Contents are not distributed along a physical path but they are accessible from a unique station that can be anywhere. If such a station is not located in a public space (where people should alternate), conditions and duration of the experience are completely free. Authors can decide to address their virtual museum to the general public or to a specific target; contents and interactivity can even be adapted to a specific user profile that can be selected and started, as it happens in several games. The access is solipsistic but multiplayer dynamics are also a challenge. The structure of such a kind of virtual museum can include several levels of interaction, technologies, as well, can be more experimental, as maintenance is a personal responsibility of the user who is aware of his initial technological choice.

2.2.2. Experiencing a Hybrid Museum

In the case of digital contents/installations included in a real museum and distributed along the visit paths, constraints are more restrictive and challenges different.

The digital experience needs to be well integrated and harmonized with the physical one, included in the museum’s collection and space. Unfortunately, this integration appears still weak in most part of the museums. Multimedia and virtual contents are often confined to a secluded space, or isolated in a “frame” (a screen, a projection, an immersive viewer, a tablet). Interacting with these devices can be more or less engaging for the public, but in many cases, it is an accessory experience. In fact, the organization of the collections and the communicative approach along the paths remain very traditional. The consequence is that the collections on display still suffer from a lack of contextualization, of sensorial immersion, of stories that can foster a deeper understanding and throw a bridge between past and present.

The museum/cultural site is a public space attended by a community of people interacting each other, with artifacts, influenced by environmental and social occurrences. Crossing the several rooms is often a hard effort, the time of permanence is influenced by a series of external factors not only/always depending on the effective interest towards the contents. The public is usually very heterogeneous, different in provenience, age, cultural background, technological alphabetization. User profiling is not yet very advanced in museums. Even if it is a topic of great relevance and interest among the scientific community [21,22], successful applications are still few, because the profiling criteria are usually very simplistic (male/female, nationality, adult/child...). Similarly, accessibility criteria aiming at the overcoming of cultural, cognitive and psycho-sensory barriers for disabled visitors, still needs a better attention and experimentation, despite specific guidelines have been drawn and diffused by Ministerial Decree, ICOM commissions [23,24], and research laboratories.

In such conditions it is not simple to create contents and experiences able to satisfy all the public and the curators as well. It is very important to study the public of the museum in advance and try to calibrate the cultural offer in order to encounter most of the expectations and needs. Digital installations cannot only introduce the museum’s main subject or contextualize/narrate specific artifacts: they can be used also to guide and distribute the public along the paths. In fact, if properly positioned and well designed, they can influence, or even re-create the map of the museum, the attractive power of the museum’s contents, thus modeling their visibility and the best conditions of fruition, according to the expectations of curators.

Usually, in a museum it is preferable to choose communicative forms and technological solutions able to involve a community of visitors at the same time.

Even in presence of an interactive digital installation with one active user leading the system, it is necessary to let people to alternate easily without restarting the system; beside the surface of visualization of digital contents should be quite large to be shared with the other visitors, and a wide space all around the installation should be created, where passive visitors can stand up (or preferable sit down to rest), enjoying the contents and collectively participating to the experience. Additional attributions of social and cultural values can arise from visitors’ interaction (Figure 2).

It is recommended to keep multimedia and digital contents as short and incisive as possible, when they are located in presence of the artifacts, along the path, avoiding too long time of permanence in front of one single installation. In this case it is better to work with beautiful visualizations (big projections when possible, holograms bringing the illusion of reality in museums). Narrations must be short, they have to catch the meaning and the essence of the cultural message, avoiding useless descriptions repeating what is already evident when looking at the objects.

Essential audio-visual representations, or simple dramatizations with personages performing actions, can be in some cases preferable, to give an immediate evidence of the function and context of use of an object.

Mixed and augmented reality solutions are recommended so to integrate real and virtual contents [25]. For instance, in Figure 3 a Byzantine original candleholder is exhibited in a holographic showcase. Around it a simple virtual scene appears through a hologram. The original object becomes part of the representation, the identity of its owner is also revealed (a high official of the Byzantine administration). The representation is necessarily in real scale, given the presence of the physical object (Figure 3).

Multimedia and digital contents can make the space not only more meaningful by adding new interpretative levels, but even more dynamic. They can contribute to the creation of a polyphonic score, interrupting the monotony of the arrangement of the objects.

For instance large projections can be used to show the original context from which the object originates (a church, an altar, the inner cell of a temple, a tomb, a palace), or dynamics of populations/animals interesting huge areas (as in the case of migrations), or to emphasize details of images and figures, putting in evidence the expression of their feelings. Touch tables are an easy interface to browse through contents (maps, collections), or to select and query details of images, even to show networks of connections among a set of objects. Small digital frames inside the showcases can visualize information related to the objects nearby (without audio), like as their virtual restoration, the executive techniques, the moment of their discovery [26]. The object’s sphere of life, use, story and “sensory” dimension can be revealed through the magic of holographic showcases, whose measures depend essentially on the dimension of the objects to be put inside; a big holographic showcase is functional to a powerful “dramaturgy”, given the relaxing positioning of the group of visitors in front of it; a small one favors a single user intimate visualization and contact with the object, as if something secret, transgressive or playful could be discovered inspecting it (Figure 4).

Single TV screens, as they are commonly used in museums, hung on the wall and aligned with the real objects should be avoided if the kind of communication is aesthetic, evocative, narrative or dramaturgic. They can be functional to deliver simple and short instructions, information or descriptions, but, usually at perceptive level, they do not contribute to break the tedious spatial grammar of objects’ repetition along the path. In a museum a visitor lives an “augmented” experience that, from a perceptive, cognitive and emotional point of view, brings him to an uncommon dimension of spiritual “elevation”. On the contrary, a screen, (considered as pure object/device and independently from the content), is immediately put in relation with our domestic environment. The weak emotional impact of standard screens, as they are often presented in museums, depends also on their limited dimension, on the absence of sounds or on the low volume generally set. Soundscape, on the contrary, instills energy and it is fundamental to catch the user and bring him/her inside a live experience. However, TV screens are not neglected by museums curators because their technology is robust and familiar, it is also the simplest one to maintain. On the other hand, it obliges authors to normalize and standardize the content, and considerably limits the visitor’s interaction, when not connected to a PC.

Of course, a video installation composed by many TV screens, assembled according to a specific conception and design, is a completely different case, potentially beautiful.

It is important that virtual contents do not dominate over the real collections. Visibility, clearness of exhibition and evidence of the concept to be delivered, harmony, relaxing conditions of use, but also aesthetic and cognitive solicitation, need to be well balanced.

More complex and longer digital experiences are possible in dedicated and secluded spaces favoring concentration. In this case virtual reality installations with deeper storytelling, introducing game dynamics, experimental paradigms of interaction, are possible and welcome. Here visitors can live a sophisticated experience, to prepare themselves before starting the visit, or to go deeper in contents after the visit, once they experienced the collection (Figure 5) [27,28].

In such cases the full control of the environmental setup is fundamental: visibility and accessibility, lights, acoustics, presence of seats, order in the visitors’ flow. A special care should be dedicated also to the physical outfitting and design of the installation, in order to create a “warm” and welcoming environment. For instance, some “iconic” elements characterizing the main cultural concept led by the digital installation can be replied in their material aspect and positioned in the space.

Last but not least, technologies used inside museums must be robust and sustainable along the time, because their maintenance is one of the most critical aspects, still today.

2.2.3. Experiencing a Virtual Exhibition

A third typology of context is an entirely virtual exhibition where the visitor moves along a path, with different multimedia and virtual installations distributed in the different areas, developing various aspects of the same theme. This case is something in between the previous case studies: There is not a physical collection to relate with, but there is a physical space to be crossed by a multitude of visitors, as in a real museum. The space, the lighting etc., are exclusively designed around the digital contents, even if real objects can be included, like as 3D printed replicas of original artifacts, to create mixed reality experiences.

Between September and November 2005, a pioneer virtual archaeology exhibition, Building Virtual Rome, was organized at Trajan Markets in Rome by CNR ITABC, the Museum of Imperial For a, Luiss University, Federculture and Filas. In this important and attractive archaeological site in Rome a visiting path was conceived to host exclusively digital installations, videos, multimedia, different typologies of desktop VR systems (in stereoscopy, with aptic devices, artificial life VR environments) not directly connected with the surrounding archaeological artifacts but dealing with the general theme of ancient Rome, whose “vision” was virtually brought to life. At that stage, some technologies were still at the beginning of their evolution and were developed only at research level. This event was an occasion to present them to a wide audience and to observe the reaction. Visitors were interviewed at the beginning of their experience about their expectations; then they were observed and interviewed during the visit of each installation in order to understand their approach, the interaction level, their understanding; finally they were asked to give their opinion at the end of the whole visiting path, about usability, attractiveness, involvement, learning.

What emerged from this survey is that the embodiment and the diverse difficulties to use different devices and softwares depend on many factors and that “communicating” the virtual is not a technological issue, but an epistemological question: technology cannot be fully effective without integrating the addressees’ points of view and needs. This result seems to be still very actual if compared with the several other User Experience evaluations carried out in the successive years in many real and virtual museums.

2.2.4. Mixed Reality: Focus on Holographic Showcases

The intent of creating a ‘dramatic dimension’ around the museum’s object and pushing the user’s sensory involvement within its stories, has convinced researchers and creatives of CNR ITABC to apply an ancient representation technique born with XVI century photography and diffused in theatres in the XIX century. This technique is the Pepper Ghost effect (so called from the English professor that brought it to fame, John Henry Pepper). It has been brought inside the museum’s showcases, in presence of the artifacts, in order to produce an effect of mixed reality (Figure 3 and Figure 6).

The basic idea of a hologram is an apparition of something that seems tangible, in an empty space. An illusion of reality, of something that does not exist but appears as if it was just in front of our eyes. Taking advantage of the available digital technologies in the current market, the challenge is to realize a new form of scenography and dramaturgy around the museum’s objects, with a simple technology, compatible with the museum’s preservation and maintenance needs [25].

Why holograms should be used to tell the story of an object, rather than a common 2D screen, a traditional projection or even a mobile device with a multimedia? It is not only to surprise and produce an astonishing reaction in the visitors. Through the hologram we change the traditional paradigm: if we include the real artifact inside the holographic showcase, the attention still remains focused on it. It stays as the real protagonist of the story. Not its virtual replica, but the original itself, is the center of our attention along the time of the whole experience: virtual animations, fragments of stories originate from its real figures and details.

We can mount and dismount, virtually restore, move, and transform the object, overlapping virtual projection on the real artifact, and playing with lights.

Through holograms we do not immerse ourselves in virtual reality using particular viewers and managing controllers, but it is the virtual reality that enters our space.

Due to this integration, the museum’s object becomes alive, enforced and not weakened by multimedia. This is not a marginal difference, because it greatly influences perception, engagement and comprehension, favoring a deeper relation between the visitor and the observed original object.

2.3. Virtual Reality

Before facing the next part of the paper, it is important to define exactly what we mean for “Virtual Reality”.

Virtual Reality is an artificially created environment that should give a human being the impression of interacting in a real environment [29]. The simulated environment is mostly digital, calculated on the computer. This ambitious and complex research started, at least at a theoretical level, in 60s of the last century [30,31], and reached the first significant and concrete results at the beginning of the 90s, although a novel, “The invention of Morel”, prefigured since 1940s [32].

Humans perceive the environment in which they live through their five sensory channels: sight, hearing, touch, smell and taste, so a virtually simulated environment should present such a condition [33]. We are currently still far from this goal. Technological progress and results achieved so far are limited mainly to the visual channel (also because sight is our most developed sense) and, secondly, to the auditory one. Since virtual space is a simulated space, it must necessarily reproduce a three dimensional world, a perspective that gives the impression of depth planes, chiaroscuro, distribution of light and shadows giving volume to flat surfaces [34]. Therefore, the accuracy and credibility of such visual representations becomes crucial, both in terms of content and of computer performance, which must be able to calculate fluently the changes of the point of view determined by the user’s movements inside the scenario.

The observer’s movement is, in fact, another fundamental component of virtual reality environments. The visitor is included in this space, he/she moves through it and explores by observing it from different points of view, he/she can analyze and interpret contents, opening the view on ever new perspectives in real time. On the contrary, a painting, a drawing, or a fixed 3D rendering are unchanging, and can be possessed in a single instant.

The ability to create virtual environments and render them to the computer has made enormous and rapid progress in these decades. In addition to the increased computing power, due to the evolution of graphics chips and technological miniaturization, the software algorithms for optimization and computer graphics techniques have greatly improved.

Similarly, we are able to digitally simulate sounds, with a good fidelity and to locate their source in a three-dimensional virtual scenario. This is made possible by stereo audio reproduction, Dolby surround, binaural recording and diffusion systems (in headphones), which maintain the 360° spherical directional characteristics of the acoustic perception.

Regarding the other sensory channels, the technology is far behind. Smell and taste are less significant in recreating the illusion of reality, some experimentation has been started in these fields too. The solutions implemented so far for the smell are based on the release of some essences placed in special containers already installed on the computer, but the combination possibilities are limited. Someone hypothesizes that, in the future, sensory perceptions can be obtained by directly stimulating particular brain areas [35,36]. However, this is a remote perspective: beyond technological problems, the knowledge of the brain mechanisms of perceptions is still very far from this objective.

Research on haptic interfaces and tactile sensation is in progress [37,38] and more advanced. These devices, gloves and special suits, are able to exert a force return on the user and, therefore, to provide kinesthetic and tactile stimuli, simulating the sensation of contact with the elements of the virtual environment [39]. Products are still experimental and live inside laboratories, their costs are extremely high and therefore they are not very accessible in the culture sector. All those body perceptions would make the virtual world “solid”, which otherwise remains confined to incorporeal and intangible images.

Perception through the body (proprioception) is in fact our most important channel after sight. The sense of “embodiment” [40], that is the “incorporation” in the real and virtual world, determines a sense of presence, involvement and motivation to interact within the environment, by measuring our actions with the typology and the strength of the reactions we get back. Virtual reality technologies are, in fact, progressing towards immersive devices (head mounted displays, etc.) that give the user the impression of being totally immersed and present in the virtual context (at least from a visual point of view), making him/her lose the perception (and the exact awareness) of the real space in which he/she finds in effect with his/her body. So, what is real and what is virtual? The distinction of living in a real or in a virtual space gradually fades, as technology, sensors, computers and viewers will be able to carry us, with all our senses, into the simulated worlds.

Mixed/augmented reality will be the experiential dimension of our common life in the future, probably used not only to “dream” and imagine, but even to perform everyday tasks.

Differently from mixed/augmented reality [41], virtual reality excludes from its simulation the vision and the experimentation of the external reality, it becomes totalizing when accessed in immersive devices (Figure 7).

Undoubtedly the use of immersive display at school, in museums and in cultural institutions has enormous potential. However it must be calibrated with respect to a series of other relevant factors, related to the design of the user experience, like as the visit routes, the usability, the sustainability, the degree of contact to be reserved with the real space, the level of collective involvement, the social relations, the psycho-physical tolerability of such technologies that are enabling but also possible cause of sickness and undesired effects.

3. Media Hybridization

Media hybridization is a phenomenon that today affects all the creative arts, communication and sciences themselves: it is a vehicle for innovation and progress. Settings and metaphors recalling video-game influence cinema. In virtual reality paradigms, techniques and expressive modes enter not only from games but also from theater, cinema and mixed/augmented reality [26].

Media hybridization strategies allow curators and creatives to conceive new methods of narration and representation of museum contents. Research perspectives in this field are very interesting, aiming at a deeper cultural transmission, engaging and not strenuous for the public, especially for young generations that, usually, are the most reluctant towards museums.

3.1. Chronicle of Media Hybridization in Brief

Since the end of the Nineties, with the spread of consumer-end computers with good graphic performances, the development of virtual reality (VR) environments dedicated to cultural heritage began to represent a widespread challenge for researchers and a new great opportunity for cultural institutions. In that pioneering phases the objectives were focused on: (a) Visualization (3D graphics in real time of increasing quality), (b) interaction with the virtual space, possibility to query 3D models and open contents, (c) interaction interfaces [42]. Virtual reality was perceived as a digital environment capable of involving the visitor in an enhanced learning experience, thanks to the possibility to freely explore, in an unconditional way, the three-dimensional space, following personal curiosity, and thus building a customized path of motivation and knowledge.

For researchers working on virtual reality environments dedicated to cultural heritage, the world of video games has always been an essential reference, despite the fact that the video game industry was, and still is, vastly more advanced and structured, aimed at specific targets and projected towards a powerful international market. Since the early 2000s, VR dedicated to cultural sites and landscapes have thus borrowed not only technologies but also motivational strategies from games. For instance, the ability of the avatar to amplify the embodiment, i.e., the sense of presence of the user in the virtual space, was immediately clear [40].

However, until the mid-2000s at least, virtual scenarios appeared desolated and silent, focused on the description of architectures or lands. The user, through his/her avatar (therefore with a third-person camera) or moving with a first-person camera, walked through empty spaces where life seemed to be disappeared. This was due to the initial stage of development of these simulated environments, both at technological level (limit in the computational and real-time rendering capabilities) and in terms of awareness of how to use, in such environments, the narrative paradigms in a convincing way. Most part of researchers, in fact, were not screen-players or narrators, and they were focused on theoretical formalization of VR, or on algorithms implementation aiming at visualization.

An obvious consequence of this condition was that VR environments lacked a narrative dimension, or it was still primordial.

However, the strength of such applications, dedicated to cultural sites and developed by “scientists”, was the reliability of the 3D models that were proposed, resulting from accurate measurements, surveys, specific and in-depth interpretative studies supported by historical or iconographic sources, leading to virtual reconstructions. Indeed, the center of interest was the scientific reliability of data, represented at different scales and, possibly, structured on multiple exploration/semantic levels, and the integration of tools of analysis in the interactive space.

This is an essential element, in line with the mission of cultural institutions, consisting in building and transmitting knowledge and education. However, it is not a sufficient characteristic to transmit culture to the public in an engaging way and, above all, to convey a complete cultural message. Aspects such as symbolic and intangible value, function and use of a spaces, social behaviors, lifestyle (which justify the architectonic shape, or the appearance of a landscape), are closely interrelated to the heritage meaning. They must be expressed through sophisticated forms of representation, content structuring and storytelling [43].

In the past centuries iconographies representing, or reconstructing, cultural sites usually included many narrative elements, like personages, chariots, animals, attending their daily activities. Let’s think, for instance, to architect and archaeologist Luigi Canina (1785–1856) that, in the Neoclassical period, imagined and visualized the ancient Appia Antica, realizing the first open air museum dedicated to this roman consular road, or to architect and engraver Giovan Battista Piranesi (1720–1778) (Figure 8): In their drawings and engraving the documentation of the way of living in a place was considered an important part of the representation, expressing the decline, or the vitality, or the charm of the place.

In the research domain, the awareness of the importance of these aspects, in the creation of virtual reality environments dedicated to cultural sites, has gradually matured over the years. At the beginning there was a diffused diffidence, especially from the curators’ side, towards the inclusion of characters and personages in virtual reconstructions dedicated to cultural heritage because they recalled games, that were perceived as entertainment and not learning experiences.

A further step of evolution was moving from the single user experience inside a VR environment, to multi-users domains in which a community of “players” was able to interact simultaneously meeting in the 3D space and sharing objectives and strategies [44], meeting also other characters and opening narratives. Multi-player virtual environments, also born in the game domain, have therefore been adapted to the objectives of cultural heritage communication (Figure 9).

Our Virtual Heritage Laboratory, at the CNR ITABC, is one of the protagonists of this research since its pioneering phase. It has been contributing from many years to the theorization and creation of VR environments dedicated to the study and exploitation of cultural heritage, carrying out a continuous experimentation. Some of them are conceived to be enjoyed online, others specifically designed for the public of museums, other ones oriented to the community of scholars [45]. Starting from the same dataset, the communication and technical strategies are very different for each target.

Evolution has been constant. The acquisition of a deeper awareness and maturity has progressively led researchers to review some of the initial assumptions and to gradually contaminate languages, making different media converge, thus renewing the communication paradigms. Virtual reality began to incorporate forms and techniques deriving not only from games, but also from theatre, cinema and performing arts [46].

In 2011, thanks to a series of sensors released by the video game market, such as Microsoft Kinect and Leap Motion, new experimentations began on body movements tracking systems (gesture-based interaction). This modality of interaction inside virtual environments is not mediated by traditional devices like as mouse, joystick, etc., resulting more immediate and natural.

The use of “natural interaction” strongly influences both the level of embodiment in the virtual space, and therefore the perceptive impact that the user gets, both the interface design, the way of structuring and hierarchizing contents, of choosing and selecting them (the input is not as precise and fast as the mouse click), of manipulating objects. The experience is configured as a progression of bodily as well as mental acts (Figure 10). This condition affects the narrative language itself, the duration of the various expressive units: everything must be usable, coherent and harmonious.

Numerous surveys have been carried out to evaluate the user experience related to such VR applications [47], as in the case of the Virtual Museum of the Tiber Valley, permanently accessible in the Etruscan National Museum of Villa Giulia in Rome, of the Etruscanning installation, hosted by the Vatican Museum and presented to Acheovirtual editions in 2011–2012 in Paestum, of Livia’s Villa Reloaded in the Roman National Museum of Diocletian’s Baths, in Rome. For each project hundreds of visitors were observed and interviewed, in the respective venues, taking into account both active and passive users. What we found is that interacting with a VR application through body gestures, in the center of a “performative” space, immediately generates in the visitor the impression of being involved in a playful situation, unusual inside a museum. Therefore, the experience is started and lived with enthusiasm, in most cases. This is particularly evident for those visitors under-20s and over 60s, while the intermediate age group shows a greater self-control that can lead to shyness and inhibition [48].

The environmental conditions in which the installation is located are fundamental in favoring sensorial immersion and in determining the duration of attention and concentration on the contents.

Instead, a critical aspect consists in the gestural grammar that cannot be taken for granted and must be explained: In fact, not all the actions can be translated into natural gestures, some of them must be “encoded” in symbolic gestures meaning other actions. Furthermore, body language is not unique and shared by all cultures [49].

At the same time, a greater awareness of the importance of narrative forms and techniques has grown in the transmission of cultural content to the public. Surveys conducted on museums visitors show how interactive digital technologies, especially if new and unusual, represent an undoubted attraction. However, equally clearly, they highlight that this attraction has an ephemeral duration, if it is not supported by convincing contents and expressive forms.

Interaction inside museums, to be effective, must be intuitive and immediate. Sometimes interaction with digital contents becomes a barrier for the public, or difficult to manage by the museum staff, and therefore counter-productive. The museum environment poses a fundamental problem that designers of digital applications cannot ignore: unlike games, that have a very motivated and selected target, the museum’s public is heterogeneous and varied, often of medium-high age and not accustomed to technology. It is not easy to realize a digital installation that, despite having traits of innovation, satisfies everyone, in terms of form, language and interaction methods.

Visit a museum is a complex and articulated experience, which places the visitor in front of continuous choices of route, pause times, use of communication equipment (panels with long or short texts, multimedia etc.); all these choices are made quickly and intuitively, and they are conditioned by a plurality of factors: The environmental conditions, the presence of a flow of people occupying a space, the time available for the visit, the attractiveness of the contents on display, the tastes and interests of the visitor, his physical condition, being alone or together. It then becomes very important that the digital application is able to immediately attract the visitor’s attention and curiosity through the set-up of the space and of its equipment, the technologies, the impact of interfaces and layout, the sound, the rhythm.

Immediately after, however, this attention must be consolidated and transformed into lively interest, into emotion, participation, understanding, critical elaboration of the content [50]. In this condition it is possible to develop dynamics of participation and cultural exchange between the visitors, whose consequences are often unexpected and highly effective.

It is now clear that the visitor likes to be guided, especially when he/she is not an expert on the contents and, if left alone, he/she would not know what to choose [16,26]. Then, the assumptions from which we started at the end of 90s-beginning of 2000s are definitely attenuated: the ambition is no longer to leave the user complete free to explore within virtual reality environments, without mediation and without guidance, as a presumed condition for encouraging motivation and learning. On the contrary, if the user is accompanied and motivated within a story (even if interactive and not linear), the permanence inside the virtual world will be more fruitful in terms of comprehension, and more prolonged. A partial degree of freedom in the exploration confirms to be certainly useful, as the interaction generally makes the user feel active protagonists and participant.

Making a virtual reality environment narrative means making it alive, populating it with events and characters.

Today, almost all creatives in the field of cultural heritage seem to agree in theory, but the concrete results are often partial. The attention is still very focused on the representation of the built forms, on the beauty of the materials, the rendering quality, the lighting effects, and less on the representation of the living elements.

Probably this may also depend on the ambition for a presumed scientific objectivity of the proposed reconstructive model, which is intended to be kept pure and free from representations, considered as anecdotal or accessory. Another reason may be that many of the 3D modelers are architects and archaeologists whose technical skills are more concerned with the virtual reconstruction of tangible evidence of what emerges from the ground, related to an ancient site. Furthermore, populating a virtual reality environment of events and characters in action requires considerable work and cost.

An advanced virtual reality environment must be thought of as an ecosystem characterized by multiple relationships, dynamics, levels of visualization and interpretation, which include large-scale holistic representations and detailed visions; an environment that can also be explored diachronically: (a) Present, observed heritage, (b) interpreted heritage, (c) reconstructed heritage in its various phases of life of the past or even of the future, (d) imagined or dreamed heritage, mental maps, mindscapes [44]. This ecosystem cannot exclude life. In this “performative” space actions and processes are more important than epilogue.

It is clear that a similar complexity of content must be conveyed to the general public through a guided experience that develops by successive degrees of complexity.

Inside the virtual environment, the different steps of the story must guide and bring the visitor to explore places, following a specific purpose and for a certain time. The story can be expressed by a voice over, or by characters or real actors included in the virtual space, speaking objects or animals (explicit narration). Alternatively, events happening in the virtual space directly make the story: the user becomes spectator or is involved in facts, that offer him/her the “key” to understand the context (implicit narration) [51].

When narration, in all its forms, invades the original paradigm of virtual reality, media hybridization is the natural consequence. Narration implies a direction, even if in interactive environments. Videogames, for example, are characterized by complex levels of interaction, often so hard to require the user to face specific stages of learning to become able to manage the system before playing. However, direction, in games remains, the first issue because the experience design is centered on a story.

From some years it has become possible to include real actors in interactive 3D environments, in place of 3D characters, thanks to virtual set and chroma-key techniques, managed by common editing software and real time graphic engines. This fact immediately influences the visual grammar: recitation, feeling expression, framing techniques, photography, combination of camera angles, camera movements are adopted from theatre or cinema and adapted for the context of virtual reality, thus resulting something new and “metaphysical” (Figure 11).

The representation of realistic personages through virtual characters can be more complex to realize (3D modeling, rigging, animations, behaviors) in comparison to real actors. Virtual characters are still preferred when the interaction between them and the user is very complex or when their movement and actions are not completely predetermined, for instance if they evolve in some way and are controlled by artificial intelligence algorithms. This is a typical condition in video games. Also, in the crowd simulation 3D characters are still the best choice. On the contrary real actors are preferred when narrative cinematographic or theatrical paradigms enter in the VR environments, when the expression of the personages feeling aims at raising emotion in the spectator. Actors are usually associated to specific performing techniques or camera movements and parameters (different angles, field of view, depth of field).

3.2. Focus on a Novel Kind of Media Hybridization: “Live Art”

In summer 2019 Teatro Potlach in collaboration with CNR ITABC have created a site specific installation dedicated to Raffaello Sanzio’s School of Athens. This famous fresco has been painted by the artist between 1509 and 1511 in the Signature Room, in the Apostolic Palace, on a figurative program dictated by the Pope himself: to celebrate on the walls the supreme Platonic ideas of the Thruth, of the Good and of the Beautiful. Raffaello painted a huge architectonic structure and put inside the philosophers, divided in groups, with Plato and Aristotle in central position. The site specific installation that has been derived, has been presented in Fara Sabina, Italy, at “Invisible Cities” (https://www.teatropotlach.org/citta-invisibili). This is a show conceived by Teatro Potlach, combining urban scenography, video-projection mapping and performative arts, that the public enjoyed walking following a path across the village. The School of Athens scene, sometimes disassembled, was projected on big and small walls of palaces and on floors. Live performers directly entered inside the details of Raffaello’s painting, becoming part of it and at the same time elements that broke the logic, and gave three-dimensionality. They told texts based on the theories of the same philosophers depicted, presented in a simple and metaphoric way. The aim of this installation is to connect the audience with the world of art, of human mind and knowledge, through their participation to the process of decomposition and re-composition of the work and with the mediation of live performers, in order to reawake their curiosity towards the original painting.

This site specific installation is an original example of media hybridization, conceived with the double purpose of explaining the masterpiece and to create a new live art (Figure 12).

4. Narration

Narrating means formulating a reconstructive hypothesis, both formal (of an architecture, a city, a landscape) and a life story. Historical information (often fragmented and incomplete) must be integrated with a fictional, but plausible, context. A reconstruction is not “absolutely certain” (except in a few lucky cases) but it must be “possible”.

In communicating contents to the public, storytelling has an enormously greater impact than description. The latter, in fact, is not able to generate motivation, unless its recipients are already aware of the specific topic. The description adopts an impersonal style, devoid of conflict and emotional energy, devoid of personal reference. It remains anchored to the past, the rhythm is flat and regular.

On the contrary, in narration, in evocation and in dramatization the objects become occasional points in which the story coagulates, creating an emotional involvement and an expectation in the visitors [52], that throw a bridge between past and present.

A story includes multiple points of view, it develops through an alternation of rhythms, and draws a different temporal space dimension in which the visitor is transported. Surveys on the public unequivocally reveal that narration, intended as a coherent harmony of texts, recitation style, acting, music and sounds, colors, visual mood, camera movements, rhythm, is what, more than any other element, is capable to arouse emotions, motivation and interest in the visitor, transforming the experience into a journey and therefore into potential learning [53].

Although there is an increasingly widespread awareness of the important role of emotions in learning processes [3], and of the effectiveness of narrative in museums, the narration still appears, surprisingly, to be the weakest link.

Visualization plays a leading role in curatorial practice, but it follows an analytical, typological and descriptive approach rather than narrative; it is often accompanied by descriptive texts. Rarely visualization is intended as visual storytelling whose grammars and rules are very precise and come from the domain of cinema, theater and games, but that, of course, can evolve in VR environments. For example, the style of camera movements will be designed to arouse a certain emotion (sacredness, frenzy, war, danger, etc.). In these cases, the union between word, images, sounds and interaction, is crucial and pursued in a very sophisticated way [31].

4.1. Making a Virtual Environment an Alive Ecosystem

To make a virtual reality environment a vital ecosystem, there are two types of strategies that can be followed.

The first one is to create and attribute cycles of predefined and looped animations to the various dynamic elements of the scenario. For example, a cycle repeated in a loop will cause a certain character to move continuously over the territory from point A to point B, as established by the designer of the scenario. In this way the user finds himself/herself interacting in a 3D animated “picture”, able to explore it. Of course, it can also interrupt the animation loop by interfering with one of these animated elements, to make it tell a story. All this translates into a fictional world that gives the impression of vitality and with a great narrative impact. However, whenever the user enters the scenario, he/she will always find the same situation and he/she can only repeat similar actions. Once experienced the first time, this is a predictable scenario.

The second strategy that can be followed is Artificial Intelligence (AI) or even of Artificial Life (AL). This perspective, already partially existing, opens up to the future. A living creature is characterized by the ability to self-organize and self-reproduce, aiming at its survival [54].

Artificial societies and autonomous agents develop behaviors that are self-determined in relation to a specific situation that occurs in the virtual world or based on random effects. Agents can learn to interact each other, or with humans. In any case they act in an evolutionary perspective. They generate a succession of events and unpredictable actions that every time can be different. Furthermore, each agent can have a specific identity, described by some attributes and properties, these attributes allow it to develop specific behaviors and relationships related to its identity, as for example being more attracted or rejected by certain places or events. The big question then is: what are the metaphors for relating human users, artificial societies and cultural contents? What codes can we share? What signs? How can we create and share meanings? Such a perspective can be interesting to create advanced simulations: content learning would take place through participation in dynamic processes rather than through static definitions.

4.2. Plausibility and Reliability of the Story

In case a virtual environment aims at analyzing an artifact, its topography and topology, its state of preservation, its restoration and so on, a 3D model with high resolution and high accuracy will be needed, with all the specialized information perfectly mapped on the volumes. Interactive tools of measurements, illumination can help the analytic approach with the structures and its elements. Obviously if a virtual reconstruction of the past is included in such a virtual reality application, the associated contents will be closely connected to the ongoing scientific investigation. Most probably the virtual reconstruction will serve to help scholars to simulate and test interpretative hypothesis on that specific case study, or to interact with metadata associated to the 3D elements and related to the sources and references that have been used to produce such a reconstruction.

This kind of VR application can be addressed to users provided not only of a preliminary knowledge of the cultural contents, but also confident with complex digital interfaces.

Differently, if the virtual reality environment is addressed to the general public, narration will take precedence over analysis, digitization techniques will be calibrated according to the level of detail and accuracy required by the effective experience that the visitor will be able to live in the virtual domain. This experience is not only conceptual but also sensorial and emotional. Preserving the detail and realism of the 3D representation and the quality of the information deriving from the data acquired on the field can be important to maintain continuity and consequentiality between knowledge and communication and to enhance the sense of presence within the virtual world [20]. A fundamental issue to transmit is the historical and social background of such a context, its dimension of life, the way it was attended and used, the process of its transformation through the time. For this reason, virtual reconstructions addressed to the public are encouraged to show not only what is absolutely certain, on the base of remains and evidences acquired on the field. In order to give a better understanding, they usually reconstruct also what is hypothetical, not certain but possible, on the base of historical and iconographic sources, contextual and typological comparisons. The virtual representation goes in some way beyond the purpose of analyzing that specific artifact and its materiality, and aims to tell a piece of story.

Avoiding interpretations is not a neutral choice, it does not preserve the objectivity of information: if a visitor is left alone and without supports facilitating an understanding of the context (which means offering an interpretation and a “reconstruction”, even if hypothetical), he/she will be free to fall into arbitrary, false and erroneous deductions. The omission does not produce positive effects, but a greater damage for the public.

In this process, certain and detailed contents relating to the specific object are combined, in a coherent manner, with plausible contents, deriving from a broader knowledge of the historical and cultural background from which the object comes from, and which often constitutes the “frame” of the story. Such a reconstruction represents the maximum level of approach to the truth, in the light of the present phase of the studies. The credibility of a reconstruction is to be considered valid until a new discovery demonstrates its erroneousness, by contrasting it with new conflicting evidence [16].

On the other hand, there are many criteria codified by the scientific community, to explain the reliability of a reconstruction [55].

Graphic expedients can be used to distinguish elements reconstructed with different levels of reliability: (a) certain, (b) probable on the base of remains, (c) probable on the base of historical sources or descriptions, d) evocative that usually means reconstructed in absence of specific sources, only on the basis of typological comparisons and similarities), [56]. Their graphic appearance can be characterized with a different style, for instance by a dominant color, or rendered in wireframe. Even metadata can be associated to them and queried about the interpretative sources that have been used, to let the user access specific information about the interpretative process followed (Figure 13) [44].

Likewise, various narrative tricks can be used to suggest the relevance of the various parts of the story to the established truth.

Indeed, virtual reality and narration can affect our imaginative capacity, fostering a deeper understanding and bridging the gap between past and present. In such type of experiences all our faculties are engaged: Perceptive-motor and symbolic ones, logics together with emotions.

The best choice in a museum could be to create a double version of the virtual reconstruction, dedicated to different audiences: One conceived for storytelling and the other one conceived to let scholars or researchers access to the interpretative sources, contextualized in the 3D space as interactive metadata. This latter version could be published also on the Internet, as web VR, available for the whole scientific community, a possible basis for further works.

Under these conditions the user becomes able of carrying out critical elaborations on the proposed themes, entering into a creative dimension, enriching the memory and improving the understanding through parallels and comparisons coming from his previous experience.

In order to implement such a communicative approach, it is necessary to reconfigure the professionalisms and competences active inside the museum institutes, alongside the traditional curatorial teams, placing side by side archaeologists, art historians, architects and conservatives, and writers and scriptwriters, directors, set designers, graphic designers, lighting experts, musicians, user experience design experts. In this way, the museum would become not only a place to look at objects but also where to listen and participate in their stories.

5. Mixed Reality

When the experience requires a co-presence in the real site and in its virtual reconstruction, and the two spaces are aligned in a constant correspondence of points of view, then conceptually the experience is of mixed reality (MR).

The possibility for the user to switch from the real world to the virtual one, and vice versa, is punctual and systematic, (since the two spaces are coincident and comparable). Real and virtual communicate conceptually and “physically”, becoming synergistic components of our present time and space.

Milgram and Kishino define “mixed reality” as the context in which real and virtual elements are mixed on the basis of the predominant element it can be divided into:

• Augmented reality (AR), which superimposes, on the predominant real perception, some artificial digital elements;
• Augmented Virtuality (AV), which consists of the reproduction of real elements above a predominant virtual context (for example the use of one’s real hands interacting within a virtual world).

Although this definition of mixed reality, intended as a continuum [41], is not the only one, in our opinion it presents the most solid theoretical basis and it is the least subject to equivocal interpretations.

In a MR experience the user can use a tablet or an immersive helmet to switch from one reality to another. There are of course more or less comfortable and “usable” solutions, depending on the context of use, but the intrinsic concept does not change.

Milgram distinguishes between three modes of using MR, depending on the technology used. The first two modes involve the use of displays and, depending on whether or not they allow the reality to be seen through, they are defined “See-through AR Display” or “Monitor-based AR Display”.

In the system “See-through AR display”, a semi-transparent display allows to directly view the surrounding real world. The Pepper’s Ghost effect, which we talked about in relation to the holographic showcases, falls into this category.

Instead, in the system “Monitor-based AR display”, the reality is not actually “sighted”, but it is displayed on a monitor thanks to a real time video capture made by the camera mounted of the used device. This is then overlaid with virtual contents by means of dedicated apps.

The visual correspondence between the two contexts (real and virtual) is generally guaranteed by tracking tools (markers, geometric recognition algorithms) and/or spatial positioning (GPS, beacons, network) that allow the App to match the geometries of the real space with those ones of the virtual space.

The third mode, defined “Projector-based AR/Spatial AR”, allows to superimpose virtual contents directly on the real world, projecting them on volumes (usually architectures, facades, objects), without the use of viewers that come between the user and the physical world. Virtual contents such as lights, images or videos are projected and overlapped to real surfaces by means of projection mapping techniques (Figure 14).

In terms of effectiveness, in a MR experience, the coherence of the overlap between real and virtual contents, the responsiveness of the system, i.e., the absence of latency, and the quality of the vision are crucial conditions in terms of understanding and “credibility”.

From a technological point of view, tracking the exact correspondence between real and virtual spaces, while the user is moving, becomes a very complex task. If the virtual content consists of simple written texts or generic 2D images, the need for a perfect collimation is, of course, less stringent. However, if the virtual content consists in a contextual 3D reconstruction, the problem becomes harder to be solved, at the present stage of technological evolution. “Monitor-based AR display” systems can bring to a more efficient solution rather than “See-through AR displays”, because these latter present a difficulty related to the parallax effect (parallax in the seen-through reality cannot coincide to parallax of a 3D reconstruction, that definitively is rendered on a 2D frame). However, technology is still at the beginning of its evolution in this field.

Transparent Head Mounted Displays (HMD), superimposing the virtual on a transparent display which allow to see through the reality, are not yet satisfactory in terms of technical performance and therefore of perceptual enhancement of the context. This category includes devices such as Microsoft hololens (https://www.microsoft.com/en-us/hololens) or wearable technology produced by Artglass (https://www.art-glass.it/). In these cases, virtual images may be not well overlapped on the real vision, they could be visible on a partial surface of the lens, moreover the rendering of lights, shadows and colors is unrealistic. If these problems will be resolved in the near future, these devices will probably be destined to be successful, given their portability, immediacy and ease of use.

From a cognitive perspective, MR is very useful to help the user to perceive and decode the space while walking through it, to identify its elements, relationships and meanings. The design of MR applications should continuously “excite” the comparison between real and virtual space, past and present, in a diachronic and multilevel vision. Reality thus becomes stratified, in continuous potential transformation, a “canvas” on which forms and meanings can be built or rediscovered; a series of possible and potential representative, interpretative, imaginative levels can emerge. Cities and villages can be re-lived in the light of a new narrative, for instance mapping 3D ancient shops and ancient crafts on the streets, or under the arcades of a square, or contextualizing old movie scenes in today’s space.

This potential can be expressed both in the artistic field and in the valorization of a cultural heritage according to a scientific approach, more adhering to the original meaning of the heritage itself.

However, in which contexts and under what conditions is the use of tablets preferable to the use of immersive head mounted displays (HMD), in mixed reality experiences?

When using tablet or smartphones the monitor serves as a “window on the world” (WoW). HMD, instead, are equipped with a pair of displays, put on the eyes, on which stereoscopic virtual contents are displayed.

An important difference is the level of collective and sharing of the experience. The immersive viewer isolates the user from the visiting group; thus, all group members should use the viewer to have a common conversation base.

The use of the tablet instead favors a more shared experience between the visiting group, and can more easily provoke discussions and exchange of opinions.

When the visitor is moving through a cultural site, along the path of visit of an archaeological site, across a monument or the rooms of a house-museum, the discovery of real historical space is the predominant element of the experience. It is therefore clear that, in these cases, it is preferable to use technologies and devices that are “light” and easy to use, that leave the public the sense of presence and the awareness of the surrounding environment. In these cases, tablets, or any devices of the WoW type, can represent a more natural solution than immersive viewers like HMD, that would be an obstacle during the journey. The latter, however, can be used from pre-determined observation points, specially marked along the route: here visitors are invited to stop and wear the immersive viewer [57]. From these points they will be able to observe how the place could appear in the past, looking around at 360° panoramic views, possibly accessing short descriptive, narrative or even dramatized fragments.

This condition requires the creation of micro-experiences of mixed reality, to be lived with immediacy. Any narrative must be conceived in the form of “pills”, significant but concise. Interaction should be limited to the exploration of the 360° space.

Moving away from the pre-determined observation point, the visitor will take off the immersive viewer, until the successive observation point.

Tracking, therefore the alignment between the physical and the virtual space is, both in the case of tablet and in the case of HMD, an extremely useful condition that makes the experience more immediate and allows the public to remain constantly oriented. Alternatively, it is possible to manually synchronize the views by reference points, but this could easily generate fatigue, confusion and frustration in some users.

Furthermore, it must be considered that not all users who wear HMD react positively to motion tracking (it also depends on the technical and perceptive strategies adopted in its implementation) and for some of them it could even cause loss of balance and vertigo.

6. Questions before Starting the Experience Design

What has been discussed until now makes some fundamental issues emerge, that can help designers to adopt the right strategies for the purpose of an efficient cultural transmission, since the very first stage of conception of a new project. It may be necessary to carry out also a preliminary survey on the visit condition of the cultural site.

Public and Accessibility

• Which is the relation between the cultural heritage and its public? How is its accessibility?
• Who are the visitors? Where are they from? Are they occasional or frequent attenders of the place? Do they visit the place alone or in group or with families, following guided tours?
• How many visitors are attending the place at the same time, and how long does their experience last, on average?

Issues the Project Aims at

• Does the project aim to modify or increment the target?
• Which are the urgent issues to be solved by the project? To increase the accessibility? To contextualize the whole visit path? To create just a general introduction to the collection or to operate on specific objects? To give evidence to objects that are not noticed?

Condition of Fruition

• Where multimedia and VR should be accessed? Along the visit paths or in a dedicated room? While accessing digital contents visitors are standing, or sitting or walking?
• Is it preferable that the user access digital contents passively or actively?

Relation between Real and Virtual Contents

• Is it useful to create mixed reality environment to foster the comparison between past and present from the same point of observation?

Communicative Style and Interaction

• What kind of style is preferable in communication? Descriptive? Narrative? Emotional? Dramatic? This depends also on the environmental conditions (for instance darkness favors emotional engagement), on local people’s attitude, cultural background and age.
• Which complexity in interaction is recommended?

Sustainability

• Aiming at sustainability, is it preferable that visitors use their own devices to access digital contents or that the museum delivers dedicated devices, at the forefront of technology?
• Is the scalability of the project, in terms of technological solutions and access modalities, an issue to be pursued?
• Last but not least: how many economic resources are available for the realization of the project and its sustainability?

7. User Experience Evaluation in Museums

User experience evaluations on real and virtual museums are planned because of the necessity to conduct test-beds and user experience surveys to have feedbacks and improve criticisms in relation to the interconnection among visitors characteristics, museum’s collections, digital contents aiming at communication, context of use.

Such investigations have a fundamental importance in order to understand how to shape and improve the characteristics of the cultural experience: the accessibility and usability of the system and clearness of the interface, the attractiveness, the role and the kind of storytelling, the adopted language, the clarity and credibility of contents, the emotional involvement, the interaction modes, the rhythms of narration and the time of use, the dynamics among the users (passive, active, alone, in group….), the efficiency of the integration between real and virtual contents, the visitor’s ability to memorize and critically elaborate cultural contents, definitively the design of the whole experience in the “performative space”.

There is not a long nor systematic tradition of cognitive studies, methodologies for user profiling and test-bed analyses in the humanities and computer sciences domains [58]. Several and different strategies are employed, given the diversity and specificity of concrete case studies. Approaches and tools can change, based on purpose, target, context of use. Nevertheless, some common issues have been highlighted since the past years, foreseeing the possibility to draft efficient methodological strategies and criteria.

CNR ITABC, with the support of national and international partners, has progressively developed a methodological approach for the user experience evaluation, focusing its attention on the pedagogical aspects and cognitive affordances [59]. The massive work carried out under the V-Must network [60] and other European projects as Etruscanning [61], CEMEC [https://cemec-eu.net], or single project as the Virtual Museum of the Tiber Valley [62], or exhibitions like Digital Heritage Expo and Archeovirtual [63], have allowed to test several evaluation tools and theoretical procedures. Major concerns regard how people interact with multimedia systems, in general which are the social and psychological repercussions, what are the main benefits in terms of memorability and utility of contents, which is the level of learnability and the overall satisfaction.

An apparatus of best practices has drawn down from the several experiences. For sure the multi-partitioned analysis, made of (a) observation, (b) questionnaire and (c) driven-scenario (task to be solved), proved to be very functional and efficient as evaluative method. Their combination is indeed useful to verify the sincerity of the users’ responses, their awareness and the emotional condition with which they approach the experience, their ability use the system and learn from it.

We have come up with solid considerations: The relevance of the combination of quantitative and qualitative tools; the importance of observations as preliminary means to study the public’s behaviors and categorize them; the distinction between active and passive users to better understand the fruition’s dynamics; the importance of collecting demographic and personal data of users (provenience, age, job, familiarity with technological devices), in order to associate specific feedback to typologies of users.

Data emerging from these surveys should be considered as very precious not only by researchers but also by museum directors, tourist guides, technical staff, so as to become more aware and prepared to face the criticalities and the extraordinary opportunities that the union between the real and the museum virtual can offer.

8. Conclusions

The museum, real or virtual, should be able to exhibit the artwork favoring the perception of its aesthetic value, but also, it should be able to support the visitors in their attempt to understand the cultural context in which the artwork originated. Therefore, one of the museum’s main functions is education, making understandable the general message that the artist wanted to transmit and, at the same time, triggering in the public a process of motivation, involvement, critical interpretation and elaboration of the meaning.

The reconstruction of contexts that are no longer (or not yet) materially accessible today, of beliefs, values, behaviors and symbolic dimensions that are “beyond” the object’s physical consistency, can take the visitor into the middle of a powerful cognitive experience. How can this be done?

Undoubtedly, multimedia and virtual reality technologies are the best means to convey such contents because they can support and shape our imaginative capacity and they become scenarios of different simulations. They engage our mind in sensory-motor processes (the most natural and immediate to be applied by human beings because they have evolved over millions of years) and interpretative faculties.

Social communication paradigms are evolving rapidly, people are increasingly accustomed to spectacular audio-visual media. Expectations and awareness of the public attending museums are evolving too.

Experience design, both in real and in virtual museums, is very complex to be planned, and even more in hybrid museums where digital contents are juxtaposed to real collections. It must take into consideration a wide audience, heterogeneous in terms of culture, provenience, age, gender and technological literacy.

When designing a multimedia and VR system, we need to have clearly in mind how a user gets in contact with it; which are the receptive canals that he/she activates in his/her mind; how the interaction takes place; and what are the expected benefits derived from such interaction in terms of satisfaction, sensory and emotional involvement, learning.

User experience designers, contents creators and software developers must work together in order to evolve towards a more efficient interconnection among visitors, collections, digital applications, context of use.

This interconnection is the subject the paper deals with, aiming at providing practical suggestions and guidelines useful for researchers in this field.

A new convergence of sciences, arts, technologies, media, an efficient fusion between tangible and virtual reality are now possible in the communication of cultural heritage. Different paradigms and techniques coming from theatrical and cinematographic grammars, performing arts and games can be integrated to shape more powerful users’ experiences and transform cultural “spaces” in “places” where they can live a feeling of intimate enjoyment and self-identification.

Rarely a virtual reconstruction is something “absolutely certain”, but it must be credible and “possible”. Thus, it implies “ethical” choices, in the respect of the credibility, reliability and the cultural identity of the context to be communicated. In such a perspective, beside narrative contextualizations, other levels of information should be integrated in virtual communication: historical, iconographic and bibliographic sources used for interpretation and in the story building process.

The choice of communicative approaches, narrative structures, as well as interaction interfaces and technologies, depend largely on some discriminating factors: (a) Audience characteristics; (b) presence or absence of real artifacts; (c) predisposition to isolation or social interaction among visitors; (d) time of potential use; (e) environmental conditions, (f) physical performance of the visitor while accessing digital contents (movement along the visit path, movement in a limited area, standing, sitting); (g) level of immersion that the system offers.

Thus, physical and virtual access become part of the same process of knowledge, sympathetic in conveying the material and immaterial value of the cultural heritage: they strengthen and complement each other, according to a holistic and multifunctional approach.

The translation of the cultural heritage into virtual heritage allows us to activate new interactions, the expansion and diversification of the audience, the involvement of the young generations, producing a quantitative and qualitative evolution in cultural transmission. Cultural participation means capacity, competitiveness, exchange and multiplication of cultural perspectives and, consequently, social and economic growth.