Design of an Interactive Spatial Augmented Reality System for Stage Performance Based on UWB Positioning and Wireless Triggering Technology

Abstract

In this research, the authors designed an interactive spatial augmented reality system for stage performance based on the technologies of UWB positioning and Bluetooth® triggering. The position of the actor is obtained through the antenna tag carried by the actor and the signal base station placed on the stage. Special effects can be triggered through the Bluetooth® module according to the actor, and rendered at the relevant location on the screen, which has higher concealment. The system has a higher degree of freedom in practical applications, which can present an interactive spatial augmented reality effect, and therefore provide new possibilities for the application of spatial augmented reality in the stage performance.

1. Introduction

With the continuous development of spatial augmented reality technology, various fields have begun to be affected by tremendous changes in technology. In the field of stage performance, the narrative form of augmented reality technology is an effective means of enhancing the expressive power of stage narrative. For example, real-time rendering of stage images by computer to achieve stage augmented reality [1], and projection-based augmented reality technology is also applied in actual performances [2]. The Disney™ theme park enhances the sense of space in the real world and the sense of reality of 3D objects, by projecting images onto the surface of real space, or actual objects, through projection-based augmented reality technology, and even achieves effective interaction between visitors and devices during the narrative process [3]. In order to achieve a better result of virtual-real fusion, the positioning of virtual elements on the stage is also an important point. In the existing augmented reality stage system, it is common to use the Microsoft™ Kinect sensor (Redmond, WA, USA), Xsens (ENSCHEDE, The Netherlands), etc. to capture the actions of the actors on the stage, and transmit their position information [4,5,6]. Kinect’s detecting range is within 4.5 meters however, so the scope of the actor’s activity may exceed the capture range of Kinect in large-scale performances, which results in the loss of location information. At the same time Kinect placement may also interrupt the view of the audience.

In the previous work of the authors, a positioning system based on infrared light emitting diode (LED) tracking was designed [7,8]. However, as the module is usually hidden in the stage props, there is a limitation for performances with fewer props. In this paper, UWB positioning technology was utilized to resolve this issue, in which the Ultra-Wideband (UWB) tag could be easily hidden without props. The signal base station is arranged in the scene, and the position information is obtained through the signal sending between the antenna tag (with the STM32 single chip as the main control chip), and the signal base station. Next, the data is converted into 2D coordinates by a certain algorithm, which corresponds to the projection position of the antenna tag in the screen. The actors can trigger virtual effects dynamically and autonomously through a trigger module hidden in the costume. The system can present more precise positioning functions of the special effects in the stage performance. The higher applicability can also make the live performance more flexible and vivid. It could bring a more immersive visual stage experience to the audience.

2. System Design and Implementation

2.1. System Design

It is necessary to arrange 3–4 UWB base stations on the stage. Based on the characteristics of the 2D screen display on most stages, 2D positioning can meet the needs of most performances; the location of the base station is shown in Figure 1. The antenna tag used by the system, which is about 51 × 15 × 36 mm in size, was worn by the actor or actress during the performance. Such small size allows that it could be hidden in the actor’s general costume items, positioning the actor or actress in real-time, and then the Unity3D-based program could get the coordinates. According to the coordinates, Unity3D forms an invisible point as a special effect position on the screen. Based on the progress of his/her own performance, the actor/actress grasps the opportunity, then sends a command to the Unity3D program through the trigger switch hidden in the clothing. As a result, the corresponding virtual special effect can be rendered at the coordinate point position of the screen (Figure 2).

2.2. Tracking and Fusing

Based on the UWB technology, the tracking system consists of two modules: The Mini3UWB (Mini3UWB, Mini3Plus and DW1000 are developed by YCHIOT, a company in Wenzhou, Zhejiang, China) signal transmitting module (Tag) and the Mini3Plus fixed position sensor (base station). The tag consists of a ceramic antenna, an RF circuit, the DW1000 peripheral circuit, a lithium battery charge and discharge circuit and the clock circuit. The communication rate is 110 kbit/s, 6.8 Mbit/s, and the working frequency is 3.5–6.5 GHz. There is a built-in 250 mAh lithium battery, and its normal working time is more than four hours, and the effective communication distance is over 150 meters. The communication distance is greater than 100 meters in the case of being blocked by cloth.

The system is based on TWR (Two-Way Ranging), TOF (Time of Flight) positioning technology and three-sided ranging algorithm in order to obtain the coordinate position of the tag in the screen’s 2D plane. The signal flight time between the tag and the base station is obtained by the transceiving response frame gap between the base station and the tag, while the calculation process is shown as Figure 3, but when there are multiple base stations, the calculation process is shown as Figure 4 and Equations (2)–(4), and the used positioning frame follows the IEEE 802.15.4 protocol.

$${\widehat{\mathrm{T}}}_{\mathrm{fab}}=\frac{{\widehat{\mathrm{R}}}_{\mathrm{a}}{\widehat{\mathrm{R}}}_{\mathrm{b}}-{\widehat{\mathrm{D}}}_{\mathrm{a}}{\widehat{\mathrm{D}}}_{\mathrm{b}}}{{\widehat{\mathrm{R}}}_{\mathrm{a}}+{\widehat{\mathrm{D}}}_{\mathrm{a}}+{\widehat{\mathrm{R}}}_{\mathrm{b}}+{\widehat{\mathrm{D}}}_{\mathrm{b}}}$$

(1)

$$T_{propA}=\frac{T_{round1A}\times T_{round2A}-T_{reply1A}\times T_{reply2A}}{T_{round1A}+T_{round2A}+T_{reply1A}+T_{reply2A}}$$

(2)

$$T_{propB}=\frac{T_{round1B}\times T_{round2B}-T_{reply1B}\times T_{reply2B}}{T_{round1B}+T_{round2B}+T_{reply1B}+T_{reply2B}}$$

(3)

$$T_{propC}=\frac{T_{round1C}\times T_{round2C}-T_{reply1C}\times T_{reply2C}}{T_{round1C}+T_{round2C}+T_{reply1C}+T_{reply2C}}$$

(4)

The signal flight distance can be calculated by using the flight time between the tag and the base station from the formula:

S = C × [(T_a2 − T_a1) − (T_b2 − T_b1)]/2

(5)

Wherein, C represents the speed of light, and T_a1 is the time at which the pulse signal of the request property transmitted by the tag arrives at the base station, and T_a2 is the time by which the response signal of the base station is received at the time point of its own time stamp, and T_b is the same. After obtaining the distance between each base station and the tag, the X and Y values of the position of the tag can be derived by three-side ranging combined with the coordinate X\Y value set by each base station (as shown in Figure 5).

$$\{\begin{array}{c}\sqrt{{\left(X-X_a\right)}^2+{\left(Y-Y_a\right)}^2}=d_a\\ \sqrt{{\left(X-X_b\right)}^2+{\left(Y-Y_b\right)}^2}=d_b\\ \sqrt{{\left(X-X_c\right)}^2+{\left(Y-Y_c\right)}^2}=d_c\end{array}$$

(6)

$$\left(\begin{array}{c}\mathrm{X}\\ \mathrm{Y}\end{array}\right)={\left(\begin{array}{c}2\left(X_a-X_c\right)\\ 2\left(X_b-X_c\right)\end{array}\begin{array}{c}2\left(Y_a-Y_c\right)\\ 2\left(Y_b-Y_c\right)\end{array}\right)}^{-1}\left(\begin{array}{c}{X}_a^2-X_c^2+Y_a^2-Y_c^2+d_c^2-d_a^2\\ X_a^2-X_c^2+Y_b^2-Y_c^2+d_c^2-d_b^2\end{array}\right)$$

(7)

After one has obtained the (X, Y) value of the actor carrying the tag, it will be matched to the size of the screen. So in the installation step, it is necessary to measure the actual distance D1 of the base station to the base station with the set coordinates of (0, 0), being the actual width D2 of the stage display screen. (D1/D2) * display width resolution to obtain the label in the corresponding position on the screen. The error of the system ranging is about 10 cm, and the positioning error is about 15 cm (increased slightly as the occlusion situation). In actually, this error has almost no negative impact on the match between the effect and the position of the actor on the stage of the actual performance of 15 meters wide.

One of the base stations was connected to the computer running the show program by USB serial port. The serial port baud rate is 9600 bps. The position information is sent to the program by the positioning module at a rate of 15 times per second and fine-tuned in the show program to ensure that the virtual special effects generation point is in the correct position. Meanwhile, since the special effects are presented on the screen of stage, when the actor/actress is far from the background, the special effects coverage needs to be appropriately expanded, so that the special effects and actor/actress form a consistent visual effect in the eyes of the audiences, avoiding excessive visual dislocation (as shown in Figure 6 and Figure 7).

2.3. Interaction

The module that triggers the special effects during the performance of the actor is mainly composed of a Bluetooth® signal transmitting module, a power supply, a module switch and a small switch. Among them, the main control adopts the STC90C516RD+ single-chip microcomputer, which is responsible for transmitting and receiving functions of wireless serial port data. The Bluetooth® adopts a full-duplex broadcast mode of 2.4 GHz broad-spectrum wireless band transparent transmission, with the communication distance greater than or equal to 30 meters.

The whole hardware module includes a USB automatic download module, power module, single-chip microcomputer minimum system (reset circuit, clock circuit, button circuit), triggering module, wireless Bluetooth® HC-31 serial port module, RS232 serial port download module (Figure 8). The Bluetooth® signal transmitting module and the power supply are packaged in a rectangular body of about 10 × 5 × 3 cm, and are controlled to be turned off by a self-locking switch control. In the single-chip microcomputer minimum system, four switch output ports are pre-set. Users can set up multiple trigger modules depending on the specific situation. For example, when there are many complex visual effects in a virtual scene, the actor can reasonably trigger the special effects according to the plot arrangement. The interactive module is small and can be hidden in the inner garment of the actor. The small switch is exported from the Bluetooth® module and extends from the inside of the actor’s outer garment to the position where the actor’s hand or wrist can be blocked by the clothes, and easily triggered. The Bluetooth® trigger module sends a status command to the receiving device continuously, and the actor modifies the status command by triggering the switch. In the show program created by Unity3D, the background receives state changes and executes virtual special effects rendering commands.

3. Results and Application

The system error is less than 10 cm, and it could result in rather smooth effects on a 15-meters-wide stage, and it could work steadily while the battery lasts. Usually a show will last for about two hours, so an external battery box was used to provide about five hours’ power supply. The cost of the whole system is about $597 which is a small cost for a show. The system can be reused in the play by constantly changing performance scenes. At the same time, it can save rehearsal time for actors to cooperate with screen effects. The system has been used in a series of traditional Chinese plays. In the performance of the Chinese traditional repertoire "Tian Xian Pei", the authors adopted the three-base station positioning mode and placed the base station behind the stage curtain (Figure 9), avoiding the impact of visual effects to the greatest extent. The antenna tag is hidden in the actress’s hair to ensure that the signal is less interfered with (Figure 10), and it maintains good stability during the actor’s movements. There are also some other programs that have been verified by us, such as tying the antenna tag to the actor’s arm. The trigger switch is fixed to the actor’s thumb and the fingertips of their forefinger, allowing the actor/actress to easily trigger the action (Figure 11). On the other hand, the occlusion of the costumes and the exposure frequency of the hand side can hide the module to the greatest extent.

Figure 12 shows the particle smoke effect triggered by the system at the actress’s position. The antenna tag in the actress’s hair cooperates with the base station to generate real-time position information, and then the fine-tuning of the control points in the program focuses the position on the actor’s hand range, accurately tracking the actors to make a step, turn and other actions after triggering the virtual special effects. When applying, the position of the base station and the antenna tag should be as far away as possible from the metal, so as not to affect the signal transmission and reception. In the actual test, it is found that the higher the installation position of the base station, the better the transmission and reception of the signal (but avoiding the height of the tag).

After the actual performance, the authors collected feedback from eight performers (from testing to acting, all performers who have direct contact with the system) and a few audience members. It is widely believed by the actress who triggered the special effects that the system was very flexible. Unlike traditional performances, in which the actress needed to adjust her performance with the fixed effects of the screen and rehearse stage animations with props again and again. In this system, she has more freedom to play, and it is easier to integrate into the plot and more conducive to the performance. As the mistakes caused by the props problem are correspondingly reduced, it is easier to show the complete stage effect with the new system. In terms of viewing experience, most viewers thought highly of the new version, and the special effects are more realistic than traditional one. Some viewers believe that the traditional operas that apply the new system retain the expressive power of traditional opera, while the addition of new technologies makes the visual perception more shocking, and has a positive significance in the spread of traditional art.

4. Conclusions and Discussion

The set of the stage-augmented reality performance system designed by the authors realized the functions of remote triggering, real-time tracking and virtual-real fusion. UWB technology provides high accuracy in positioning and low power consumption, which is more suitable for large stage performances. In practical applications, it is necessary to pay attention to the inconvenience caused by some influencing factors. For example, the system is highly affected by metal. When setting up the base station, it needs to be away from large-area metal. The base station can be set in the form of a suspension, and the antenna tag also needs to avoid contact with metal jewelry.

Compared with the previous spatial augmented reality interactive system in stage application, this system has the characteristics of a high degree of freedom and high concealment, which allows the performers to be free from technical constraints during the performance, and focus on the stage performance, while the high concealment of the module brings a higher sense of immersion to the spatial augmented reality, and improves the quality of the stage performance. It also has received high evaluation in the practical application of some Chinese dramas. The system not only provides a better viewing experience for the audience, but also offers more possibilities for the combination of rising technology and traditional art.