Digital Ghost: Application of Holographic Projection on Protection of Guangdong Puppet Show ^†

Abstract

Guangdong puppetry, a 700-year-old intangible cultural heritage, faces declining popularity due to economic influences and evolving youth interests. The shrinking audience and a lack of successors to inherit this tradition further compound the issue. This research aims to revitalize the puppet show by preserving its traditional roots, exploring new possibilities, and engaging the younger generation. Divided into three stages—preliminary investigation, role modeling, and holographic projection device design—the study utilizes field research, literature review, questionnaires, and experiments. Information is gathered through venue visits, which is followed by questionnaire distribution and 3D model creation based on survey results. The final stage involves designing and testing a holographic projection booth.

1. Introduction

The Guangdong puppet show is an essential aspect of Cantonese culture. Cantonese culture is a regional culture that is primarily found in southern China. It is currently available in Cantonese-speaking locations such as the Greater Bay Area of Guangdong, Hong Kong, and Macao (Figure 1). Due to societal progress and generational shifts, the Guangdong puppet performance is now an intangible cultural heritage rather than part of the mainstream culture of young people in Guangdong. The show is protected as an intangible cultural heritage by the government. Ordinary young people in the Guangdong–Hong Kong–Macao Greater Bay Area nowadays find it difficult to attend the Guangdong puppet show, let alone experience the cultural genes in this intangible cultural asset. The rapid advancement of information technology has created new opportunities for Guangdong Puppet show researchers and practitioners. Holographic projection is a very new type of media technology. Holographic projection technology in the digital age bridges the gap between reality and virtual reality by presenting floating pictures in actual space, resulting in a novel visual viewing experience, and this suspension is similar to the techniques employed in Guangdong puppet shows.

The recent findings from a study highlight a sustained interest among local administrations in providing ongoing support and guidance for significant intangible cultural heritage projects [1]. Furthermore, a separate study has delved into establishing a quantitative dimension for the preservation of Chinese cultural heritage [2]. Another research initiative aims to enhance the 3D design workflow by employing wearable motion capture (Mocap) systems for virtual humans [3]. In addition, a study has explored the application of true-color holograms for the conservation of museums and ancient Greek artifacts [4]. Lastly, there is a research endeavor focused on experimenting with both immersive and non-immersive virtual reality to craft a distinctive fusion of sound and sensory experiences for cultural heritage [5]. Consequently, the objective of this research is to examine the possibilities in design and production, along with the technological approach, for combining holographic projection technology with the essential qualities of Guangdong puppet performances.

2. Materials and Methods

Three key technologies have been employed in this study: 3D modeling, non-realistic modeling, and holographic projection. Among these, 3D modeling is the most commonly used to design and digitize the characters in the Guangdong puppet show. Non-realistic modeling technology is used to boost job productivity. Rather than pursuing the most realistic picture possible, we require rapid prototyping and iteration in early prototype design. Holographic projection is the principal technology used in this investigation. It depicts non-realistic 3D models that are used to examine the interaction of actual and virtual space. A study studied the impact of tourist satisfaction and brand loyalty on brand equity by introducing mixed reality technology to cultural heritage protection to improve tourist user experience [6]. A similar study is being conducted in which they evaluate the use of new interactive technologies to improve customer experience and client loyalty [7].

2.1. Three-Dimensional (3D) Modeling

Three-dimensional (3D) modeling is an essential element of digital modeling technology, and Autodesk’s 3D software is the go-to option for producers. The bundle includes two notable programs, 3DMax and MAYA, which are widely used for creating models. 3DMax is the preferred choice for architectural and landscape modeling and boasts sophisticated tools that cater specifically to those fields. On the other hand, MAYA is a top-rated platform for character modeling, leveraging powerful features to bring virtual characters to life with precision and detail (Figure 2). This versatile suite of software supports producers across diverse industries, allowing them to create sophisticated digital models for architecture, landscapes, and character design with ease and confidence.

Scholars have increasingly embraced 3D modeling tools for cultural conservation studies in recent years. With the aid of AR and LiDAR technology, a team of researchers successfully created a highly detailed 3D model of the Lantern Festival using nothing but a smartphone [8]. In a separate study, the literature surrounding the preservation of intangible cultural assets through 3D technologies was categorized into 12 distinct groups, allowing for a more comprehensive analysis of the topic [9]. Furthermore, another group of researchers utilized cutting-edge UAV imagery and 3D laser scanning technology to digitally preserve the cultural landscape heritage of Mount Lushan with remarkable precision and accuracy [10].

The advantages of 3D modeling over traditional manual modeling are indisputable. Being carried out digitally, 3D modeling is strongly associated with the digital environment, and the mathematical model can be displayed on various devices using computer vision technology. As a result, 3D modeling is one of the primary technologies that connect reality and virtuality, offering immense possibilities. The digital world transcends time and location, which makes Guangdong’s puppet performance in the digital environment more comprehensive than the conventional form. This work uses three-dimensional modeling technology to bridge virtuality and reality, portraying the Guangdong puppet performance model in virtual space in the most sci-fi way.

2.2. Non-Realistic Modeling

With the help of non-realistic 3D modeling, designers can produce unique and inventive objects without being constrained by the constraints of realistic modeling techniques. Unlike traditional 3D modeling techniques that strive to produce an accurate portrayal of real-world objects, non-realistic modeling methods apply artistic freedom to create unique and remarkable designs. A study conducted over a decade ago presented an algorithm for 3D collage generation using non-realistic modeling technology. This algorithm was found to be exceptionally useful in handling complex 3D processing tasks [11]. Another study used this technology to create interactive exhibition pieces that showcase the splendor of a monastery [12]. These examples demonstrate the immense potential of non-realistic 3D modeling to push the boundaries of creativity and innovation.

Non-realistic modeling is an approach that offers three significant advantages over realistic modeling methods: high efficiency, small model size, and rapid speed. With non-realistic modeling, designers can achieve greater work efficiency than with realistic modeling methods. Unlike realistic modeling, which requires designers to spend a lot of time on modeling tasks to achieve expressive model details, non-realistic modeling provides block modeling or facet modeling technology, making it easier for designers to finish the prototype design of the model or even the subsequent enhanced design. Even if the existing common non-realistic modeling methods, such as low-surface modeling, are not suitable, designers can use other techniques to complete the task. Additionally, non-realistic models have the advantage of being smaller in volume than realistic models. With these benefits, non-realistic modeling is a highly efficient and effective approach for designers to complete their tasks. Non-realistic 3D models unquestionably have several advantages over realistic ones. They have a lower mesh count, which means they have fewer materials and shading. As a result, they require less bandwidth, making them far easier to transport and display in the virtual world. The second advantage of non-realistic models is their production speed. They can be designed much more quickly in digital space. Traditional 3D modeling often produces two sets of models: a high model and a low model. By applying the low-details model to the texture produced by the high-details model, a faster visual representation is produced. A designer can omit this step and complete the model considerably more quickly, though, by beginning with a low-plane model. Moreover, most low-detail models are as good as high-detail models in future interface design, which makes them the best option for prototyping.

2.3. Holographic Projection

In contrast to Pepper’s ghost illusion technology, holographic projection technology stands as a well-established and advanced industry technology, boasting distinct advantages in this research. This technology encompasses three main categories: gas projection, air projection, and reflection projection. In the specific context of this study, holographic film served as the imaging medium, and glass acted as the projection surface. Three crucial characteristics distinguish holographic projection technology: its ability to offer a 360° vision, adjust the size of the projection, and interchange content seamlessly. The utilization of holographic projection technology introduces unprecedented flexibility in stage placement compared to traditional Cantonese puppet shows. Remarkably, this flexibility is achieved without the necessity for a separate performance and display stage, highlighting the efficiency and innovation inherent in holographic projection techniques (Figure 3).

Comparing holographic projection booths to conventional stages reveals several benefits. First of all, they offer better efficiency, more storage capacity, and simple display content switching. Conventional stages, on the other hand, offer little room for performers. Second, holographic projection technology is more adaptable than traditional stages, which are limited by the size of the performance area. This makes it a better option. Lastly, if a power source is available, 360-degree performances can be hosted by holographic projection booths. All things suggested, holographic projection technology offers a better and more adaptable stage for putting on shows. Holographic projection technology boasts the unique advantage of a 360-degree display ability, which is a significant departure from traditional Cantonese puppet shows. Unlike the conventional setup, where one side of the stand remains hidden, our holographic projection booth empowers the audience to relish the show from multiple perspectives. As a critical element of digital media, holographic projection technology seamlessly bridges the gap between the real world and virtual space. This breakthrough has the potential to revolutionize Guangdong puppet performances and even pave the way for them to enter the metaverse.

Over the past couple of years, the use of holographic projection technology has increased significantly especially in the museum industry. For instance, a study found that museums using X-reality technologies, such as ARVRXR, create an immersive experience where visitors can interact with virtual agents as if they were real [13]. Similarly, a research team explored how to preserve Kunqu Opera in China through the mediating role of authenticity on experience quality and satisfaction [14]. Another team designed an IntARSI system for the New Museum of Civilization in Rome (MuCIV), offering visitors a multi-sensory and enhanced experience [15]. Moreover, a study found that improving perceived authenticity and tourist interaction could enhance the nostalgic experience of vintage hotels. Self-congruity acted as a mediator for this interaction, while digital technology acted as a moderator, reducing it [7].

3. Results

This research is organized into three stages: preliminary investigation, role modeling, and holographic projection device design. We utilized various research methods, including field investigation, literature research, questionnaire surveys, experiments, and others, to ensure comprehensive results. Our research team visited the existing puppetry venues in Guangdong to collect first-hand information, providing a strong foundation for our study. Based on the preliminary results, we designed and distributed questionnaires to gather more data. We then used the results of the questionnaire survey to design a 3D model, taking into account various factors. Finally, we designed the holographic projection booth, and the 3D model was imported for testing. We are confident that the methodology used in this research will provide valuable insights into the design and development of holographic projection devices.

3.1. Questionnaire

This study’s preliminary investigation consisted mostly of three tasks: field investigation, literature research, and a questionnaire survey. The goal is to collect data from many sources and then compare it to obtain results for design studies. On 6 November 2021, the research team traveled to the Guangzhou Province Puppet Aat Theater to conduct field research and gather first-hand knowledge by attending Guangdong puppet shows. After returning to the lab, we looked through numerous books to learn more about Guangdong puppet shows. Based on the main data from the field survey and the secondary data from the literature study, we created a questionnaire for a general young population and distributed it via the Internet. A total of 161 questionnaires were gathered, of which 161 were legitimate. According to the results of the questionnaire survey, 38.51% of people were very willing to “learn about the intangible cultural heritage of puppet shows”, and their readiness to “try a new way of puppet show experience” was comparable, except for 6 people who selected “very unwilling”. The percentage of people who selected “willing, general, unwilling” was around 25%. When asked, “What do you think are the limitations of puppetry?” 87.58% chose “single form of expression”, and when asked, “Should traditional elements be retained in the shape of a puppet show?” (47.2%), the most common answer was “keep a little”. When asked if they agreed to the combination of puppet performance and holographic projection, 36.65% responded “very willing”.

3.2. The 3D Modeling

The 3D design segment encompasses three primary facets: 3D model appearance design, 3D model function design, and 3D model animation design. Initially, researchers embarked on crafting 3D models of puppet characters, primarily humanoid in form, drawing upon preliminary research findings. The model’s skin is intricately designed following the completion of the mesh design for the model surface. The skin drawing phase focuses on creating the material sphere and mapping the UV topology. Employing a technique known as non-realistic modeling, three-dimensional models were generated in this study, significantly simplifying the mesh complexity and expediting the modeling process (Figure 4).

Considering the puppet-like motion integral to the character, this aspect was factored in from the model’s inception. The decision to model the characters after imitation robots was deliberate, incorporating joints akin to puppets into the humanoid form. To streamline the design process, the head area of the 3D character was specially treated, with facial features covered by large VR glasses, facilitating a more efficient design process.

Upon completing the visual design, the animation design phase commenced, involving three key pieces of work. Initially, the researchers designed the skeleton for the 3D character model using a full-body inverse kinematics (IK) solver, encompassing primary skeletons such as the spine, left arm, left hand, right arm, right hand, left leg, left toe, right leg, and right toe. Subsequently, components and controllers of the 3D character model’s skeleton were adjusted accordingly, aligning joints and mesh and configuring the controller for seamless integration with subsequent animations. The 3D model skeleton was then bound by the 3D model, allowing the team to animate the puppet show in accordance with its specific requirements.

This comprehensive approach to 3D design not only involved a meticulous consideration of appearance but also extended to the functional and animated aspects, ensuring a holistic and integrated process. Through the strategic incorporation of non-realistic modeling and the deliberate choice of puppet-like features, the researchers aimed to optimize the efficiency and flexibility of their 3D design workflow, ultimately tailoring the puppet show to their unique creative and technical specifications.

3.3. Hologram Pyramid Booth

The design of a hologram projection booth involves two crucial components: the design of the holographic projection booth itself and the production and integration of holographic projection materials (Figure 5). The process commenced with the transformation of a pre-made miniature into a pyramid, utilizing PVC transparent film as the display light source and an Android phone as the image source. The creation of a holographic projection sample involved the application of Adobe After Effects 2020 non-linear audio and video editing tools. To construct a holographic projection test prototype, a holographic projection video, an Android phone, and the actual holographic projection booth were employed. Through a series of rigorous tests, the research team meticulously identified the dimensions, proportions, and other essential parameters of the 3D model. Additionally, we determined the animation time required for an initial performance of the Guangdong puppet show, amounting to approximately 60 to 300 s (Figure 6).

The subsequent step involved the construction of a life-sized (1:1) holographic projection booth, utilizing materials procured from the Internet. This booth comprised three primary components: a truss, a bracket, and a projection pyramid. The truss, crucial for stability, portability, display convenience, and adaptability, was crafted from a lightweight yet durable aluminum alloy. Four straight-angle iron bars constituted the stand, ensuring robust support for the structure. The projection pyramid, consisting of four glass pieces, each affixed with a holographic projection film, housed a 24-inch LED display. During the projection direction selection process, the team opted for top projection, effectively concealing the projection surface and facilitating seamless projection.

The meticulous engineering and selection of materials ensured that the holographic projection booth met the specific requirements for stability, portability, and display quality. The truss’s aluminum alloy construction enhanced its portability while maintaining structural integrity. The strategic use of holographic projection film on the glass pieces of the pyramid contributed to an optimal display, aligning with the project’s objectives. The research team’s detailed examination of dimensions, proportions, and animation time underscored a commitment to precision in crafting a holographic projection booth tailored to the unique demands of the Guangdong puppet show. Ultimately, this multifaceted approach resulted in a functional and purpose-built hologram projection booth that was ready to showcase the immersive fusion of technology and traditional puppetry (Figure 7).

4. Discussion

4.1. Holographic Projection Technology to Protect Intangible Cultural Heritage

Holographic projection technology, a well-established innovation, has found a novel application in the preservation of intangible cultural assets, which are exemplified by its integration into the realm of the Guangdong puppet show. The evolution of this technology has brought about three significant advancements that are reshaping the traditional puppet show experience. Firstly, the holographic pyramid booth emerges as a transformative force capable of catapulting the traditional Guangdong puppet show into the digital era, especially within the evolving landscape of the metaverse. This transition represents a seamless blend of heritage and modernity, offering audiences a dynamic and engaging spectacle. Secondly, holographic projection transcends the temporal constraints associated with traditional puppet theater. Audiences can now relish the Guangdong puppet performance from the comfort of their homes, breaking free from the limitations of fixed showtimes. For those seeking an immersive encounter, the option to attend live performances in theaters remains open, providing a flexible and inclusive experience. Lastly, the holographic projection booth introduces a remarkable aspect of 360° transparency, presenting a visually diverse and distinctive form compared to the conventional Guangdong puppet show. This transparency poses new challenges, necessitating the creation of innovative 360-degree scene performance scripts and the incorporation of transparent visual content to convey intricate plot details. Consequently, practitioners and enthusiasts of the Guangdong puppet show are presented with a wealth of new content to explore and innovative challenges to overcome, aligning with the evolving landscape of the metaverse. In essence, holographic projection technology not only preserves cultural heritage but also propels it into the future, offering an array of possibilities for both creators and audiences. The marriage of tradition and cutting-edge technology not only ensures the longevity of the Guangdong puppet show but also opens up new avenues for creativity, interaction, and engagement in the ever-evolving metaverse era.

4.2. Research Deficiencies and Future Improvement

This study exhibits three notable limitations that warrant attention for future improvements. The first limitation pertains to character creation, where the conservative 3D modeling method was employed when conclusive research results were unavailable. This conservative approach led to a role model that deviated from the conventional modeling of Guangdong puppet shows. Addressing this flaw could involve leveraging advanced technologies like 3D scanning and motion capture in future studies to enhance the fidelity and authenticity of character creation. The second limitation revolves around the size of the stage within the holographic projection booth. The current dimensions constrain the inclusion of multiple roles or intricate tale performances, which is primarily due to the utilization of a 24-inch LED display as imaging equipment. A potential remedy involves expanding the image size, perhaps by incorporating an ultra-short focal projector, accommodating more complex and nuanced performances. The third limitation concerns the impact of lighting on the holographic projection booth’s performance. The mainstream LED or LCD screens used in the current setup exhibit a relatively low brightness count, approximately 300 nits. This limitation hampers the booth’s visibility during daylight hours, potentially rendering it invisible in bright environments. To overcome this challenge, adopting a laser projector with a brightness exceeding 3000 Lm could significantly enhance visibility in various lighting conditions. Moreover, the study suggests that integrating interactive technology has the potential to elevate user engagement within the holographic projection booth. This recommendation implies that future research could explore interactive features to enhance the overall user experience, fostering greater interaction and immersion. In conclusion, while the current study contributes valuable insights, addressing these identified limitations in character creation, stage size, and lighting conditions could substantially enhance the holographic projection booth’s effectiveness, realism, and adaptability for diverse settings and performances. Embracing cutting-edge technologies and refining technical specifications are key steps toward overcoming these challenges and advancing the field of holographic puppetry.