Real-Time Projection-Based Augmented Reality System for Dynamic Objects in the Performing Arts

Abstract

This paper describes the case study of applying projection-based augmented reality, especially for dynamic objects in live performing shows, such as plays, dancing, or musicals. Our study aims to project imagery correctly inside the silhouettes of flexible objects, in other words, live actors or the surface of actor’s costumes; the silhouette transforms its own shape frequently. To realize this work, we implemented a special projection system based on the real-time masking technique, that is to say real-time projection-based augmented reality system for dynamic objects in performing arts. We installed the sets on a stage for live performance, and rehearsed particular scenes of a musical. In live performance, using projection-based augmented reality technology enhances technical and theatrical aspects which were not possible with existing video projection techniques. The projected images on the surfaces of actor’s costume could not only express the particular scene of a performance more effectively, but also lead the audience to an extraordinary visual experience.

1. Introduction

In contemporary performing arts, we can find various attempts at furnishing the audience with different visual experience by video projection. This approach is one of ongoing various attempts to create new performance paradigm by the convergence of digital technology and analog sensibility which we can feel in the classical performance genre [1]. The ordinary video projection in performing shows has a purpose similar to matt-painting effects usually employed on the film, where background video can be composited with the actors’ actions. It is primarily used to describe the background of the stage together with sets and props made elaborately in order that the audience can be immersed in the stage as an artificial place.

Recently, in various performing shows, more advanced projection-based augmented reality (AR) enhances visual aspects which were not possible with existing video projection techniques. While the projected area in previous video projection is limited to a rectangular region like a screen, projection-based AR can project contents such as images or videos onto the unconstrained nonplanar forms. This technique offers the experience of visual immersion to the audience by representing the sets of stage as a virtual environment or virtual world rather than simply showing the background of a stage on rectangular planes [2–4].

We performed a research for applying projection-based AR in a particular scene of Korean original musical show, named “Turandot”, which was produced by “Daegu International Musical Festival” in Korea. The purpose of this research was to develop an effective performance technique fused with digital technology. In particular, we focused on applying projection-based AR on the dynamic objects with flexible shapes, precisely projecting images on the inside silhouette of dynamic objects. The existing techniques stated above have constraint that only static objects can be used. Among the elements of performance, there exist dynamic objects continuously moving as well as static objects. The dynamic elements influence the narrative of the performance rather than the static ones. The dynamic objects we consider in this paper are continuously moving actors and actors’ costumes; the costume in the genre of performing arts is one of the important elements of performance, describing the actor’s character. Therefore, this paper proposes an advanced performing technique, which we call a real-time projection-based augmented reality system for dynamic objects in the performing arts, to provide the audience with distinctive visual experience through our research. The texture, patterns, and images on the surface of a moving actor’s costume can be changed arbitrarily during the live performance, and it can represent the character of specific actor more effectively with our system.

This paper is described as follows:

Firstly, we examine previous related studies regarding to projection-based AR briefly and define the costume which is a dynamic object with flexible shape in performing arts. Secondly, we explain technical approach and the configuration of our system for performing arts, in addition, we present its advantages. Thirdly, we illustrate the process about how our system can be applied to the performance, the result, and the limitations of our work. Finally, we also discuss the lessons learned from this research and the direction of future work.

2. Related Works and Motivation

2.1. Projection-Based Augmented Reality

Projection-based AR is described as a video projection technique, which can extend and reinforce visual data by throwing images on the surface of 3D objects or space; this belongs to Spatial Augmented Reality in a broad sense [5,6]. Using projection-based AR, it is easy to implement graphical representation that ordinary lighting techniques cannot express. Unlike general lighting technique, the technique can project high-definition image or video, and change the object shape visually with the flow of time. Therefore, it can show visual images dynamically. This combination of imagery and real-object allows the audiences to recognize visually extended space. In addition, unlike conventional highly equipment-dependent AR [7] which is restricting audience’s body, this technique can advantage to improve their immersion. These advantages are requirements for better audiences’ experience in the performance [8,9]. In media art field, the projection-based AR is called Projection Mapping, which covers a smaller field than the Projection-based AR.

2.2. Projection Mapping on Moving Objects

Projection mapping is a technique which causes an optical illusion by analyzing three-dimensional object, projecting images, and then precisely aligning them. This technique is being widely used in various fields such as façade of building, plastic objet, and also in the performing art field. In these cases, it usually projects the images onto fixed objects by using manual alignment between the objects and projected images. On the other hand, there has been much recent research trying to perform projection onto dynamic objects with automatic alignment, but these researches are applicable only to limited shapes and movement of objects, requiring huge computation for image alignment [10–12]. This causes latency until the image is aligned with 3D objects. The more latency, the slower the movement of projected image following the movement of object, consequently, it generates a visual error, which reduces the immersion of audiences [13,14].

2.3. Feature of Costumes in Our System

When projection mapping is applied to performing art, its purpose is augmenting the exterior of object or the interior environments in the theater, such as a set and objects in the stage. Unfortunately, we were unable to apply the existing method due to the characteristics of costume as follows.

Flexible shape

“Flexible shape” is a shape like actor’s costume. The existing techniques can be applied only to the surface of 3D solid objects, such as polyhedrons, cylinders, or spheres. In general, the shape of stage objects may be complex or atypical, but they are not flexible. However, flexible objects have silhouettes which are difficult to predict, and their shapes change frequently. Besides, if part of the costume is covered by occluding objects, the existing method cannot be applied.

Dynamic object

Similarly, when projection mapping is applied to dynamic object whose position is moving or shape is changing, this work also causes difficult problems. For example, when the actor performs simple movements like moving from side to side or lifting arms, it is difficult to apply projection mapping even in this easy case. To solve these problems, there have been various attempts; such as, showing the effect like projection mapping by merging certain pre-set motions with video contents through many exercises [15], or synchronizing pre-programmed robot’s movement with video contents [16,17]. However, it is evident that the actor cannot perform the given motion according to the scenario exactly robot during live performance.

3. Real-Time Projection Mapping System

The problems discussed in Chapter 2 are factors degrading performance quality by distracting the attention of the audience. Thus, we implemented Real-time Projection Mapping System using a masking technique, so that the images are automatically projected and aligned onto the dynamic object and actor’s costume with flexible shape in real-time. The process of the system is as follows: (1) Tracking varying actor’s silhouette; (2) Creating synthetic images by masking the silhouette and then compositing with video; (3) Aligning generated images with actor’s costume on stage through the projection.

3.1. Hardware Configuration

Figure 1 abstracts the hardware configuration of real-time projection mapping system. The main apparatuses of this system are composed of infrared (IR) camera equipment including IR light for the extraction of actor’s silhouette, computing device with graphics processing unit (GPU) for image processing, and a high-lumens projector for projecting the images onto the surface of the actor’s costume.

There have been previous attempts using depth camera, like Kinect from Microsoft, to extract dynamic object silhouette with flexible shape [18,19]. This equipment can easily extract the masking image, with the help of the depth data in space. To produce a high-quality result, the center lines of the projector lamp and IR camera lens have to be set up such that they are parallel and as close as possible. However, in the case of the depth-camera, its effective-range is too short to be used in a big theater. To solve the problem, we used an IR camera device with a zoom function in order to capture the actor in a large space as well. Capturing only one particular area reflecting IR light, the IR camera can effectively separate the background and the actor precisely.

3.2. Projection Mapping Based on Real-Time Masking

The purpose of the real-time masking technique is to project the images onto the certain part or the rest of the scene without delay. For instance, when we would like to augment an actor or his costume surface using projection mapping, we can select only his silhouette area excluding any other part, then project with exact alignment. While this masking technique needs less computation (as compared with 3D object tracking such as [12]), it is able to show a relatively high quality result. In other words, it can respond rapidly to the actor’s movement and align precisely. We perform the process illustrated Figure 2 to generate the masking images of the silhouette from IR camera and to merge the images with video contents made in advance.

First, an actor is captured at certain position by IR camera, with IR light projected over a wide range in the backside of the stage. Then, the masking image is generated from the silhouette by using binarization, morphology, and Gaussian blurring of the captured images [20,21]. Finally, the masking images are combined with the frames of video contents. This process is presented in Figure 2. These steps of the process are performed on GPU, which enables parallel operation and computational speed-up. As a result, it can reduce time latency a lot in image processing.

3.3. Precise Calibration of Mapping

In order to handle unexpected problems directly on the stage, we made real-time mapping application with several adjustment functions. This can be used for a quick and effective set up before the performance. This application is designed to deal with various problems happening under physical environment, lighting conditions, or projector positions, in the theater. In addition, it includes various functions which can directly control the parameters of image processing in real-time. The functions can be grouped into three categories.

• Mask adjustment: adjusting brightness value of IR camera according to the lighting condition and modifying the parameter for morphology and blurring to extract accurate and natural actor’s silhouette.

• Projection transform: adjusting the size and the location of projected image to minimize the errors of mapping in the case that it is hard to select optimal projector position because of the theater conditions.

• ROI area exclusion: excluding the unnecessary area from the projection when the projection mapping is applied to the actor in live performance.

4. Implementation and Result

We selected a theater, and installed stage set, stage objects, and the real-time projection mapping system, as illustrated in Figure 3.

Then, particular musical scenes applying our system were rehearsed. In this scene, two characters appeared, and one’s dress was augmented to look like a ghost decorated by our system. Although the actress kept moving frequently, the images were projected onto and combined with her dress accurately in real-time, as shown in Figures 4 and 5. The actress did not have any difficulty in performing with our system, since she did not need to act differently and nothing needed to be changed from the original movement.

Since the theater is wide (about 27 m in width and 1200 seats), the result of the projection on the actress’ costume is different from each audience member’s viewpoint. If the ghost actress’ role is located at the center of the stage and the projection mapping is performed by using one projector, the audiences who sit at the left or right side can see only half of her silhouette. To solve this problem, the projection mapping area was expanded by installing two projectors on the left and right sides and overlapping the directions of projectors toward her, illustrated in Figure 3. This method allowed the audience to see a more preferable projected imagery than using only one projector, wherever they sat in the theater, illustrated in Figure 6.

To find out how fast our system can respond to the actor’s movement, we checked the required time to align processed images with the actor’s silhouette, illustrated in Figure 7. A series of images represent the test motions for lifting arm in controlled experiment environment for precise measurement.

Initially, while the actor keeps standing at one position and spreads his arms out horizontally, we precisely calibrated the imagery on the actor’s costume. This calibration performs image transformations through adjusting the size and the location of the projection, and then accurate silhouettes are generated through morphology and blurring operations. Then, a particular scene was rehearsed according to the planned direction while recording with a camera in front of the stage. Finally, the recorded video was analyzed. In the recording, we selected a portion with the largest actor’s movement. When the actor stopped moving, we counted the frames until the images were aligned with the actor’s costume.

With this method, the result presented that the images are aligned exactly after five frames; which means that approximately 0.167 s latency occurred. If we calculate the aligning time for all the frames in the scene, our system shows lower average latency. Our system was tested on a personal computer (PC) with Intel core i7-3770 3.50GHz CPU, NVIDIA GTX680 GPU, 16GB RAM (Customized PC); Point Grey FIREFLY^® MV IR camera (Point Grey Research, Richmond, Canada); and EPSON EB-G5950 high-lumen projector (Seiko Epson Corporation, Suwa, Japan).

5. Limitation

The limitation of this result is that the experimentation of our system is optimized to express the ghost in the scene of the musical. If the actor performs any extreme movements, such as running, jumping, or swinging arms, the visual errors may be more exposed to the audience due to larger latency. In addition, when the actor moves out of the projection area, the images cannot be projected onto her silhouette.

The other limitation appears when multiple projectors are applied. As described above, two projectors were used to cover the wide audience view in the performance. In this case, it is a problem to handle the seam in the overlapping nonplanar surface. For now, we used a video content which does not make a severe artifact in the seam region. As a future research, the overlapping regions should be connected seamlessly and made into one complete image to provide a better visual experience for the audience.

Nevertheless, the experts in the musical field gave the positive response that our research can express spectacular scenes regardless of the limitations explained above.

6. Conclusions and Future Work

In this paper, we presented a projection-based augmented reality technique for a dynamic object for performing arts, which can be used effectively by merging performance costume and digital technology. In addition, we equipped sets for the performance in the theater, and showed the result of our system. Our research started in order to provide interesting visual experience to the audience. Our system augments dynamic object with flexible shape, which is the actor’s costume surface, and can change the images according to the narrative of a play in real-time. A high-quality mapping result is obtained from the application that corrects errors generated in the process of projection mapping on stage.

In the future, our plan is to improve the performance of our system further to be performed robustly as well as the performances of multiple genres or various scenes. In order to improve our system further, in the viewpoint of reducing the aligning latency, we are going to optimize the GPU based algorithm and we are currently researching the technique that generates the composed image beforehand at the estimated positions by anticipating an actor’s future movement from the previous frames. Furthermore, in order to show the audience seamless mapping results in the wide theatre environment, we will try to solve the issue about handling the overlapping region due to multiple projectors.