Robotic Choreography Inspired by the Method of Human Dance Creation

Abstract

In general, human dance is created by the imagination and innovativeness of human dancers, which in turn provides an inspiration for robotic choreography generation. This paper proposes a novel mechanism for a humanoid robot to create good choreography autonomously with the imagination of human dance. Such a mechanism combines innovativeness with the characteristic preservation of human dance, and enables a humanoid robot to present the characteristics of “imitation, memory, imagination, process and combination”. The proposed mechanism has been implemented on a real humanoid robot, NAO, to verify its feasibility and performance. Experimental results are presented to demonstrate good performance of the proposed mechanism.

1. Introduction

Social robots are emerging with various forms such as assistants, pets and servants, which show social intelligence in human style, based on deep models of human cognition and social competence [1,2]. These robots have played an important role in our society, e.g., serving the elderly people as companions at their homes. Recently, dance has been applied to social robots for exploring human–robot interaction, and has attracted much research attention in the field of robotic dance [3] and dancing robots [4,5].

Generally, robotic dance is an activity of artificial intelligence which uses robots as carriers and uses dance as art content. In our taxonomy [3], the creation of robotic choreography is an important research category for robotic dance, and it emphasizes that the robot should develop its autonomous ability based on its embodied environment or human/machine aesthetic consciousness (or cognition). In this category, several popular methods are proposed based on the different research purposes (see Section 2). However, the autonomous creation of robotic choreography, which draws lessons from the mature experiences of human dance creation, is still inadequate and has been rarely explored so far. Therefore, inspired by the method of human dance creation, this paper proposes a novel mechanism for a humanoid robot NAO to create good choreography autonomously with the imagination of human dance. Basically, this paper aims to develop the autonomous and humanoid behaviors of robots by the creation of robotic choreograhpy.

Additionally, good robotic choreography should have three features: (1) the characteristic preservation of human dance; (2) the innovativeness of human dance; and (3) accordance with human aesthetics [3]. However, because of the existence of the contradiction between the preservation of dance’s characteristics and innovativeness [3], the existing achievements rarely create good robotic choreography. The reason is as follows: The characteristics preservation of human dance requires invariance and is contradicted with the fact that the innovativeness requires variability. Thus, this paper also considers how to combine the innovativeness and the characteristics preservation of human dance in robotic choreography.

The rest of the paper is organized as follows. Section 2 introduces the previous works related to robotic choreography creation. Section 3 explains our mechanism for autonomously creating robotic choreography on a simulated and a real humanoid robot, including the characteristics of human dance creation, a framework and dance spaceexpressive and memory; imagination elements, poses, motions. Then, the simulation and the real experimental results are given in Section 4 to show the feasibility and performance of the proposed mechanism. Section 5 evaluates and analyses experimental results to validate the proposed mechanism. Section 6 presents some open-ended questions based on this research work. Finally, a brief conclusion and future work are presented in Section 7.

2. Related Work

In the creation of robotic choreography, most popular methods include random generation [6], mapping rule [7,8], chaos dynamics [9], interactive reinforcement learning [10], evolutionary computation [11,12,13,14,15,16], and the Markov chain model [17,18]. This section will briefly review these creation methods.

Shinozaki et al. [6] generated robotic dances with randomization by concatenating a sequence of dance units from a dance unit database. Although the method was simple and scalable, the same dance motions appeared repeatedly in the produced robotic dances because the dance unit database only stored 60 dance units. All the dance units were designed by a motion editor based on the representative hip-hop motions, and they were fixed in the dance unit database. Therefore, the method can preserve the characteristics of hip-hop, but not innovative ability.

Oliveira et al. [7] and de Sousa Junior et al. [8] deployed mapping rules to create robotic choreography, which maps the environmental events (features) to some fixed robotic dance motions, and produces the autonomous robotic dance with the continuous stimuli of its embodied environment. The mapping rules can be customized and robotic dance can be generated in real time. Due to the predefined dance motions and few mapping rules, the original characteristics can be kept very well, but the repeated appearance of the predefined dance motions reduces the innovativeness of robotic dance. Additionally, many important environmental factors were not used to build rules, such as human activities, etc.

Aucouturier et al. [9] used a network of FitzHugh-Nagumo neurons to enable a vehicle-like robot (MIURO) to generate chaotic itinerant behaviour of dance among low-dimensional local attractors. It made robotic dance have the chaotic property, and brought the novel and unpredicted robotic dance to us. However, the vehicle-like robot only performed its dance behaviour by its path and motion features (such as acceleration, deceleration, turn, etc.) on a 2D plane. Its dance performance was far worse than the performance of a humanoid robot.

Meng et al. [10] used the subjectively aesthetic evaluations of humans as the rewards of robotic dance motions, and encouraged their robot to create its dance choreography by using the human-preferred dance motions. The method showed its intelligence in a certain extent, and made robotic dances have certain innovativeness. However, a large number of the interactive activities between humans and robots put heavy burdens on people. Due to the use of predefined dance motions, the created dances had very limited innovativeness.

In general, the robotic choreography created by evolutionary computation can be divided into three types: Traditional Evolutionary Computation (TEC), Interactive Evolutionary Computation (IEC) and Semi-interactive Evolutionary Computation (SIEC). TEC uses the common fitness function, and IEC uses human subjective aesthetic evaluation as its fitness function to control respectively each evolutionary direction of robotic dance motions. By machine learning, SIEC makes a robot have aesthetic evaluation ability similar to humans, i.e. the autonomous evolution of robotic dance poses.

Eaton [16] proposed an approach for the synthesis of humanoid robot dance with TEC, and built its fitness function, which referred to “Performance Competence Evaluation Measure” [19], based on the sum of all the movement values over all the joints and the time the robot remained standing. Vircikova et al. [11,12,13,14] introduced the interactive genetic algorithm as the specific implementation of IEC into robotic choreography, and constructed a multi-robot system to evolve the aesthetic dance motions for a humanoid robot. After the final dance motions were acquired by several generations of evolution, they were concatenated somehow to be a robotic dance in TEC and IEC. As two different types, TEC and IEC can create robotic dance motions, and have the certain intelligence. However, they cannot keep the characteristics of dance. Considering the subjective feature of the aesthetic of dance, it still remains to be proven if the common fitness function in TEC can be consistent with human dance experts with regards to the ability of the aesthetic evaluation of dance. Additionally, a large number of human–robot interactions in IEC put heavy burdens on people too.

In Reference [15], an approach named “Semi-interactive Evolutionary Computation” (SIEC) was proposed by embedding a machine learning method in the Interactive Evolutionary Computation, and autonomous aesthetic evaluations are conducted in accordance with human aesthetic. A SIEC-based model for autonomous robotic choreography creation is used for seeking autonomously robotic dance poses that are matching with human aesthetics. The contradiction between the subjective characteristics of dance aesthetics and a large number of human-robot interactions has been reconciled in a certain extent.

Using musical beats and emotions as stimulus sources, Xia et al. [17] proposed a Markov chain model for robotic choreography to choose a more adaptable and beautiful dance motion gradually and to construct a robotic dance ultimately. The method not only can generate robotic dance autonomously under the premise of keeping the dancing characteristics, but also can synchronize robotic dance motions to musical beats and emotions, so the acceptance degree of humans is higher to other robotic dances. Unfortunately, the method had limited innovativeness, and was required to build a transition probability matrix and emotional labels for robotic dance motions. It became a heavy burden to predefine a large number of robotic dance motions.

Xia et al. designed a kind of robotic dance motion formalization as the basis to realize the Markov chain model [17]. The formalization firstly grouped the joints of the Nao robot into four categories (c∈{Head, LArm, RArm, Legs}), and used K_c = <V_c,1, V_c,2, ……, V_c,|c|> to define a keyframe that is associated with category c. A motion primitive was defined as M_c(β)=<K_c,1, βD₁, K_c,2, ……, K_c,F−1, βD_F−1, K_c,F>, where F is the number of keyframes in M_c. D is the minimum time that it takes to move (interpolate) from one keyframe K_c,f to the next keyframe K_c,f+1, and β is a parameter of synchronizing the motion primitive to music [17]. A schedule of motion primitives belonging to category c were defined as S_c = <K_c, D, M_c,1, I₁, M_c,2, ……, I_p−1, M_c,p>, where p is the total number of motion primitives in S_c; D is the time to interpolate from K_c to the first keyframe of M_c,1; and I_m is the time to interpolate from the last keyframe of M_c,m to the first keyframe of M_c,m+1 [17].

Finally, the four independent schedules of motion primitives (S_Head, S_LArm, S_RArm, S_Legs) were generated respectively according to the beats and emotions of the inputted music, and the Nao robot can execute these four schedules simultaneously to produce robotic dance for its whole body. Although the formalization adapted to the robots with the independent joints can be actuated simultaneously, it did not consider the gracefulness and goodliness of the combination of these motion primitives which come from the four schedules respectively at some time. Therefore, the Markov chain model based on the formalization does not consider keeping the characteristics of dance too.

By extracting musical features from audio file and estimating dance movements through Hidden Markov Model, Manfre et al. [18] proposed an automatic system for humanoid dance creation. The method used the Viterbi algorithm to select suitable dance movements, and it was essentially a solution of determining the sequence of hidden states (dance movements) from observation sequences (musical feature). Since a static set of dance movements was used, the method kept the dancing characteristics and had a limited innovativeness. Furthermore, the computation of two matrixes (transition matrix, emission matrix) brought substantial overhead, so it became a heavy burden to predefine a large number of robotic dance motions.

For the two key features of good robotic choreography—the preservation of dance’s characteristics requires invariance, and the innovativeness requires variability—there exists a contradiction between the two [3]. Without considering how to combine the two, the existing methods of robotic dance creation have poor performance. Inspired by the imagination used in human dance creation, this paper proposes a mechanism of creating robotic choreography autonomously by a humanoid robot with imagination. More specifically, the characteristics preservation and the innovativeness of human dance are combined in this mechanism. A framework and an imagination method are designed in this mechanism so that a humanoid robot can generate new innovative dance elements, poses and motion autonomously so that robotic dances can be created by concatenating these motions.

3. The Mechanism of Autonomous Robotic Choreography Creation

3.1. Human Dance Creation

Traditionally, human dance creation is a creative art activity based on human imagination subjectively and consciously, which reflects the essential attribute and internal regularity of human society. In this creative process, many striking features appear, such as imitation, memory, imagination, combination, choice, novelty, etc. [20].

Moreover, human dance creation is also a kind of psychological activity in which the creator emerges a unique meaning and new action images and rhythms. It includes not only the action of imagination (such as action’s learning, creation, combination, change, etc.), but also the dance nature of imagination (such as dance’s experience, creation, performance, appreciation, teaching, etc.) [21]. By using such imaginations, human choreographers can create various and colourful dance works freely.

Based on imagination, a human dancer could use the following law to create a dance work [20]: After s/he learns some basic elements of a dance species, such as hand shape, hand gesture, feet shape, feet gesture, step, etc., s/he can imagine new dance elements spontaneously based on the basic elements; then s/he can innovate dance poses based on the dance elements so that dance motions can be designed by concatenating several dance poses; finally, s/he will be able to create a dance work very well by assembling several dance motions with some meaningful methods.

3.2. Framework

Inspired by the dance creation law described in the previous subsection, a humanoid robot can create its dance choreography autonomously by using imitation, memory, imagination, process and combination. Therefore, we propose a framework for autonomous robotic choreography creation with imagination, as shown in Figure 1.

In our framework, all of the concepts are expressed and defined formally in “Dance Expressive Space”. It includes not only the related concepts of traditional dancology (such as dance element, dance pose, dance motion, dance work, etc.), but also some concepts of robotic body (such as joint, limb, etc.). Furthermore, “Dance Memory Space” is used for storing all the objects that are generated in the procedure of creating robotic choreography, such as the imitated dance element, the imagined dance element, the imagined dance pose, the created robotic dance, etc.

As can be seen in Figure 1, a humanoid robot shall master the basic dance elements firstly by imitating a human dance expert, and then by extending new dance elements by its imagination. The new dance elements are the basis of imagining new dance poses and motions further. A robotic dance consists of a sequence of dance motions, and a dance motion consists of a sequence of dance poses, each of which in turn consist of the dance elements generated by all the limbs of the whole body. Finally, the robot can concatenate the dance motions to create dance choreography autonomously, which is accomplished in “The Module of Creating dance”. The method of balance detection and compensation ensures that valid (balanced) dance poses, dance motions and dance choreographies are imagined effectively. It should be noted that the module of balance detection and compensation has been investigated before, and more details about it can be found in Reference [15].

3.3. Dance Space: Expressive and Memory

3.3.1. Dance Expressive Space

To describe the dance activity of a humanoid robot, a set of formal symbolic representations are utilized to express the intrinsic factors of robotic dance, namely “the dance expressive space of humanoid robot”. It only involves the formal expressive form of robotic dance, which is equivalent to the concept of schema, and does not involve the instances of robotic dance.

The dance expressive space of a humanoid robot is indicated as HRDES = {J, D, L, C, DE, DP, DM, RD}, which is shown in

Table 1. The dance expressive space of humanoid robot.

Name	Meaning	Expressive Form	Remarks
J	The set of Joints	J = {J1, J2, …, J|J|}	Ji indicates the ith joint of humanoid robot.
D	The set of Domains on Joints	D = {D1, D2, …, D|J|}	Di indicates the domain of the ith joint.
L	The set of Limbs	L = {L1, L2, …, L|L|}	Li indicates the ith limb of humanoid robot.
C	The set of constraints	C = {C1, C2, …, C|L|}	Ci indicates the constraint on the ith limb &Ci ∈ {0,1}.
DE	Dance Element	DEi = (Vi,1, Vi,2, …, Vi,|Li|)	DEi indicates the dance element vector on Li, and Vi,t is a value of the tth joint of Li.
DP	Dance Pose	DP = (DE1, DE2, …, DE|L|)	The dance pose vector DP that contains the dance elements on all the limbs of humanoid robot.
DM	Dance Motion	DM = (DP1, DP2, …, DPY)	The dance motion vector DM is a sequence of DP that has a length of Y.
RD	Robotic Dance	RD = (DM1, DM2, …, DMZ)	The robotic dance vector RD is a sequence of DM that has a length of Z.

. All the joints of a humanoid robot are formed as a set of joints, namely J that is expressed as J = {J₁, J₂, …, J_|J|}, and |J| is the total number of joints in J. Each joint J_i has its domain (latitude) of freedom D_i [lowerbound, upperbound], and all these domains on the joints in J constitute the set of domains D that is expressed as D = {D₁, D₂, …, D_|J|}. The limb refers to a logical whole that consists of several joints connected physiologically together, which is expressed as L = {L₁, L₂, …, L_|L|}, where |L| is the total number of limbs on the humanoid robot. Each limb L_i = {J_i,1, J_i,2, …, J_i,|Li|} is a proper subset of J, and is not an empty set.

Additionally, L must satisfy the following requirements—L₁∪L₂∪…∪L_|L| = J and L_i ∩ L_j= ø (i ≠ j). The former one requires that L must contain all of joints in J, and the latter shows that one joint cannot appear in the different limbs. In order to indicate which limb will keep the characteristics of dance, a set of constraints on limbs is introduced. It is expressed as C = {C₁, C₂, …, C_|L|}, and C_i ∈ {0,1} shows if there is a constraint on the ith limb (i = 1, 2, …, |L|). If C_i = 1, the corresponding limb is restricted to extend new dance elements by using the following method of imagination, and can only use the imitated dance elements to take part in the creation of robotic choreography. Otherwise, if C_i= 0, the limb can use the method of imagination to extend. Note, though, that there is no common approach to determine L and C because the different dance species has its distinctive characteristics of motion style. Therefore, human dance experts shall determine manually L and C according to the current dance species (discussed in Section 6).

Each dance element DE_i = (V_i,1, V_i,2, …, V_i,|Li|) can be viewed as an instance on the ith limb L_i, and V_i,t is a value of the tth joint of L_i. When the dance elements that come from all the limbs on the humanoid robot are combined together, a whole dance pose is generated and it is indicated as DP = (DE₁, DE₂, …, DE_|L|). A dance motion is DM = (DP₁, DP₂, …, DP_Y) where Y is the number of the sequence of dance poses in DM. Finally, a robotic dance is viewed as a sequence of dance motions, and is defined as RD = (DM₁, DM₂, …, DM_Z) where Z is the length of the sequence.

Furthermore, our dance expressive space of humanoid robots can be used to create robotic choreography on different kinds of humanoid robot, given that the joints of the robot have independence and can be actuated simultaneously.

3.3.2. Dance Memory Space

Memory is the psychological process of accumulating and preserving the individual experience in the mind, and it is also the process of encoding, storing and extracting the information in the brain. As important characteristics of human dance creation, the function of memory is introduced in our framework for autonomous robotic choreography creation with imagination, which exists in the form of “Dance Memory Space”.

In our framework, “Dance Memory Space” is used for instantiating the “Dance Expressive Space”. Thus, any object which is generated in the procedure of creating robotic choreography will be stored in the “Dance Memory Space”. These objects include the imitated dance element, the imagined dance element, the imagined dance pose, the created robotic dance, etc. In a certain condition or time, these objects can be resumed, so memory plays its role. Furthermore, “Dance Memory Space” can be realized by a database or a file.

3.4. Imagination: Elements, Poses, Motions

We used the method of imagination for developing the autonomous ability of a humanoid robot in order to reconcile the contradiction between the innovativeness of robotic dance and the preservation of dance’s characteristics. The method includes the imagination on three different parts: dance elements, dance poses and dance motions. The imagination of new dance elements is the core of the method. The imagined dance elements are the source of new dance poses that are in turn the sources of new dance motions. Then, several dance motions can be concatenated somehow to be a robotic dance. Therefore, how to imagine new dance elements is a key step.

3.4.1. Imaging Dance Elements

Considering any dance element belongs to a limb, the method of imagining new dance elements on a limb L_i is given as the following. In the process of imagination, we need to judge if an imagined dance element is new, i.e. a unique object in the “Dance Memory Space”, in which a dance element (DE_i) is defined as a vector, and its n components form n-dimensional space R. Therefore, a dance element can be viewed as a dot in R.

Firstly, we need to set a neighborhood d_i on the limb L_i. The neighborhood d_i is a parameter to balance the number of imagined dance elements and the differentiation between dance elements, and it takes a positive integer from [1, MAX_i]. Suppose D_min [lowerbound, upperbound] is the minimum domain among all the domains that correspond to all the joints of limb L_i, MAX_i can be computed by Equation (1):

$$MA{X}_i=[(D_{\min }.upperbound-D_{\min }.lowerbound+1)/2]$$

(1)

The bigger the neighborhood d_i, the higher the differentiation between dance elements, and the smaller the number of dance element in R, and vice versa. We shall select the appropriate value of d_i according to actual needs. All the dots in the area of the neighborhood d_i are very similar to each other, and they are viewed as a unique dance element. When the spatial distance between two different dots in R is more than 2d_i away or equals to 2d_i, we view them as two different and distinguishable dance elements. As a popular distance measure between two dots in space, Euclidean Distance is appropriate obviously to measure the similarity (or difference) between two dance elements.

In the following formulas, DE_i,u and DE_i,v indicate two dance elements in the same limb L_i and Equation (2) is used for computing the similarity S(DE_i,u, DE_i,v) between DE_i,u and DE_i,v, and Equation (3) is used for judging if DE_i,u and DE_i,v are the same dance elements.

$$S(D{E}_{i,u},D{E}_{i,v})=EuclideanDis\tan ce(D{E}_{i,u},D{E}_{i,v});$$

(2)

$$Same(D{E}_{i,u},D{E}_{i,v})=\{\begin{array}{l}\hspace{0.17em}True,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}if\hspace{0.17em}S(D{E}_{i,u},D{E}_{i,v})<2d_i;\\ False,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}if\hspace{0.17em}S(D{E}_{i,u},D{E}_{i,v})\ge 2d_i;\end{array}$$

(3)

Assuming W is the space that is formed by all of the distinguishable dance elements on a limb, it is a new dance element if there does not exist any dance element in W that is same as the current imagined dance element, and shall be joined in W. A saturation function is introduced to judge if W is saturated when any imagined dance elements cannot be joined into W. Therefore, two key judgments are required. One is used for judging if there exists the same dance element in W, which is illustrated in Equation (4). The other is used as the end condition of judging the saturation on W, which is shown in Equation (5).

Assuming the current imagined dance element DE_i,t has n components, and there exists m different and distinguishable dance elements {DE_i,1, DE_i,2, …, DE_i,m} in W. The label of Repeat is used for indicating if there exists the same dance element in W as the current imagined dance element.

$$Repeat=\{\begin{array}{l}\hspace{0.17em}True,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\exists D{E}_{i,k}\hspace{0.17em}(k=1,2,\mathrm{......},m):Same(D{E}_{i,k},D{E}_{i,t})=True;\\ \hspace{0.17em}False,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\forall D{E}_{i,k}\hspace{0.17em}(k=1,2,\mathrm{......},m):Same(D{E}_{i,k},D{E}_{i,t})=False;\end{array}$$

(4)

To judge saturation on W, it is necessary to project all the dots in W to each axis of W. Due to the vector of the dance element that has n components, there exists n axes A_k (k = 1, 2, …, n). Assuming the projective points of any two dance element vectors (DE_i,u and DE_i,v) are adjacent on an axis A_k, which are recorded as P_k,u and P_k,v respectively, and the distance between these two projective points is recorded as |P_k,u − P_k,v|, finally the label of Full is used for judging if W is saturated or not.

$$Full=\{\begin{array}{l}\hspace{0.17em}True,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\exists A_k\hspace{0.17em}(k=1,2,\mathrm{......},n):\left|P_{k,u}-P_{k,v}\right|<2d_i;\\ False,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\forall A_k\hspace{0.17em}(k=1,2,\mathrm{......},n)\hspace{0.17em}:\left|P_{k,u}-P_{k,v}\right|\ge 2d_i;\end{array}$$

(5)

The prerequisite of imagining new dance elements depends on the permission of constraint C_i on the corresponding limb L_i. Only if C_i = 0, the limb L_i is permitted to imagine new dance elements on the limb of the humanoid robot. According to the characteristics of the imitated dance, all the constraints are determined manually by human dance experts (discussed in Section 6).

Considering the imagination of new dance elements is built on the basis of the imitated dance elements and a limb of a humanoid robot that consists of many joints, it will generate a new dance element that keeps the characteristics of dance when we change a value of single joint J_i (under the range of its domain D_i) on an imitated dance element. Obviously, the generated new dance elements move away from or do not have the characteristics of dance when we change two to all values of joints on an imitated dance element simultaneously. The schematic diagram of the method of imagining new dance elements is shown in Figure 2, and the algorithm of imagining new dance elements on a single joint of a limb is illustrated in Algorithm 1.

Considering a special case, when an imitated dance element is symmetrical based on the body centre line, several new dance elements will be generated by the symmetrical or anti-symmetrical imagination, which are based on the imitated dance element. To deal with the special case, some related concepts (such as mirror symmetrical limb, symmetrical joint pair, anti-symmetrical joint pair, etc.) are introduced. If each joint of limb L_i, which is allowed to imagine new dance elements, has one and only one symmetrical joint (also included in the limb) based on the body centre line, the limb L_i is defined as a mirror symmetrical limb. For example, a mirror symmetrical limb named “Upper-body” is shown in Figure 3.

Algorithm 1 The algorithm of imagining new dance element on single joint

1. For (select each limb Li from L in turn)               //Li = {Ji,1, Ji,2, …, Ji,|Li|} 2.  If Ci = 0 Then 3.   For (select each imitated dance element DEi on Li in turn)    //DEi = (Vi,1, Vi,2, …, Vi,|Li|) 4.    {If not repeat (DEi) Then the imitated dance element DEi is joined into W 5.     For (select each joint Ji,k from Li in turn)          //k∈[1,|Li|] 6.      {For (select each value V’ from the domain Dk [lowerbound, upperbound]        that corresponds to the joint Ji,k, and the step length is 2di) 7.        { Imagine a new dance element DEi’(Vi,1, Vi,2, …, Vi,k−1,V’, Vi,k+1,…, Vi,|Li|) on Li 8.         If not repeat (DEi’) Then the imagined dance element DEi’ is joined into W 9.         } 10.      } 11.     If full Then finish the imagination on the limb Li, and return to the outer loop to select the next limb 12.     }

In a mirror symmetrical limb, a joint (J_i,k) and its mirror symmetrical joint (J_i,k’) form a joint pair <J_i,k, J_i,k’>. By judging whether the two joints take the same values in a symmetrical body pose, the joint pair can be divided into two types: symmetrical joint pair, anti-symmetrical joint pair. In a symmetrical joint pair, two joints take the same values for showing a symmetrical effect, such as <LshoulderPitch, RshoulderPitch> (see in Figure 3), etc. In an anti-symmetrical joint pair, two joints take the different values (both have the same absolute values) for showing a symmetrical effect, such as <LshoulderRoll, RshoulderRoll> (see in Figure 3), etc.

In a mirror symmetrical limb, if an imitated dance element is mirror symmetrical based on the body center line, each joint pair (symmetrical/antisymmetrical joint pair) can change synchronously on the values of the two joints, and then imagine new and symmetrical/antisymmetrical dance elements on the limb. The algorithm of imagining new and symmetrical/antisymmetrical dance elements on double joints of a mirror symmetrical limb is illustrated in Algorithm 2. The differences between the two focus on the Line 8 and 9. For example, in the symmetrical imagination algorithm, V’ is assigned as the value of J_i,k’ when the joint pair <J_i,k, J_i,k’> is symmetrical; in the antisymmetrical imagination algorithm, −V’ is assigned as the value of J_i,k’ when the joint pair <J_i,k, J_i,k’> is symmetrical.

When the procedure of imagining new dance elements is finished, to each limb L_i that has no restriction (C_i = 0), a corresponding set of the imagined dance elements is built; to each limb L_i that has restriction (C_i = 1), it cannot imagine new dance elements and only a set of the imitated dance elements is kept. These limbs and their dance element set are the basis for dance pose synthesis.

Algorithm 2 The algorithm of imagining new and symmetrical /antisymmetric dance element on double joints

1. For (select each limb Li from L in turn)                 //Li = {Ji,1, Ji,2, …, Ji,|Li|} 2. If Li is mirror symmetrical limb Then 3.  {For (select each imitated dance element DEi on Li in turn)      //DEi = (Vi,1, Vi,2, …, Vi,|Li|) 4.   If DEi is symmetrical dance element Then                      //DEi = (Vi,1,Vi,1’,…,Vi,k−1, Vi,k−1’, Vi,k, Vi,k’,…,Vi,|Li|/2, Vi,|Li|/2’) 5.    {If not repeat(DEi) Then the imitated dance element DEi is joined into W; 6.     For (select each joint pair <Ji,k, Ji,k’> on Li in turn)     //k∈[1,|Li|/2] 7.      {For (select each value V’ from the domain Dk[lowerbound, upperbound] that corresponds            to the joint Ji,k, and the step length is 2di) 8.        { If joint pair <Ji,k, Ji,k’> is symmetrical Then                   V’/−V’ is assigned as the value of Ji,k’; 9.        If joint pair <Ji,k, Ji,k’> is antisymmetrical Then                   −V’/V’ is assigned as the value of Ji,k’; 10.       Imagine a new dance element DEi’(Vi,1,Vi,1’,…,Vi,k−1,Vi,k−1’, V’, V’ or –V’,…,Vi,|Li|/2,                Vi,|Li|/2’) on Li; 11.       If not repeat(DEi’) Then                  the imagined dance element DEi’ is joined into W; 12.       } 13.      } 14.  If full Then finish the imagination on the limb Li, and return to the outer          loop to select the next limb; 15.    } 16.  }

3.4.2. Imaging Dance Poses and Dance Motions

Dance element, as the smallest unit of dance posture, is attached to the limb, so it appears as a local dance gesture. By assembling several dance elements that come from all limbs, dance pose appears as a global dance gesture, which is shown on the whole body. Furthermore, dance motion appears to have spatial-temporal continuity by assembling several dance poses to be a sequence.

Based on the dance element sets of all limbs, dance poses can be imagined. It is obvious that the generated universal set of dance poses is a Cartesian product of these dance element sets. According to some kind of purpose, a subset of dance poses can be built from the universal set of dance poses, so it is needed to determine a method of choosing a dance pose.

Similarly, a dance motion can be imagined based on the selected set of dance poses. Moreover, it is a sequence of dance poses, and its length (i.e., Y) indicates the number of dance poses in the sequence. Therefore, the generated universal set of dance motions is a Cartesian product of Y sets of dance poses. According to some kind of purpose, a subset of dance motions can be built from the universal set of dance motions, so it is still needed to determine how to choose. Furthermore, dance motion has spatio-temporal continuity and is always implemented in a time span. It presents some dancing figures by the change of dance poses in a space. In order to ensure a smooth transition between the two adjacent dance poses in a dance motion, various methods of interpolation can be used, which can bring different aesthetic effects.

Additionally, randomization is a popular method in the creation of robotic choreography [6], it can bring uncertainty and unpredictability to the selection of objects (such as dance motion, etc.). Meanwhile, there exists the stochastic uncertainty in the procedure of art creation, and the unpredictability and variability are the internal factors of innovativeness of art [22,23]. Thus, the random method is consistent with the requirement of dance innovativeness [3].

Therefore, the random method is used in our mechanism. Dance poses and dance motions are imagined randomly. For example, an imagined dance pose randomly assembles the dance elements that come from all limbs on the humanoid robot; and a series of dance poses, which are selected randomly, is constructed to be an imagined dance motion.

3.4.3. Concatenating Dance Motions

Based on the imagined set of dance motions, several dance motions can be concatenated to be a robotic dance, which is a sequence of dance motions. In general, to express a certain intention or present a particular dancing figure, some dance motions should be chosen in some way. Because the random method brings uncertainty and unpredictability, which is consistent with the requirement of the innovativeness of dance [3], the method is used for choosing dance motions in our mechanism.

Furthermore, there exists a transition between the two adjacent dance motions in the procedure of concatenation, and it can be implemented by interpolating values between the last dance pose of the previous dance motion and the first dance pose of the next dance motion. Similar to dance motions, the concatenation of dance motions has spatio-temporal continuity too. Therefore, the various methods of interpolation bring the different aesthetic effects, such as the change of rhythm, the variation of spatial extension, the change of speed, etc.

Undoubtedly, the imitated dance elements have kept the dance characteristics very well. The method of imagination creates new dance elements on the permitted limbs, and can keep the characteristics of dance by using the above algorithm. A new dance pose can be generated by assembling these dance elements that come from all limbs on a humanoid robot, and it must keep the dance characteristics too. Considering any dance motion is a sequence of dance poses, new dance motions can be produced easily while keeping the dance characteristics. Consequently, our mechanism should be able to not only create robotic dance autonomously by humanoid robot itself, but also simultaneously keep the characteristics and the innovativeness of dance.

4. Experimental Results

Our experimental facilities include a personal computer (operating system: Windows 10, CPU: Intel Core i5-3337U, RAM: 4 GB) and a biped humanoid robot NAO (Aldebaran Robotics). In our experiments, all the software of computation and simulation (including: Visual C++ 6.0, Webots 7.4.1, and Choreograph 1.14.5) run on the personal computer, and the NAO robot is regarded as a final platform of displaying the created robotic choreography.

Tibetan dance is a typical Chinese folk dance, and has its distinctive characteristics and style. The most notable characteristics of Tibetan Tap are the regular quivers on the knees and the regular step of the feet driven by the quivers. We select Tibetan Tap for a humanoid robot NAO to imitate and practice. A text file is used for implementing “Dance Memory Space”, and it stores all the instantiated objects of “Dance Expressive Space”. Visual C++ is used for implementing the method of imagination (Section 3), and Webots is used for showing the effect of the dance objects (dance element, dance pose, dance motion, and robotic dance) on a simulated NAO robot. Finally, choreography is used for mapping the robotic choreography, which is created and shown on Webots, to a real NAO robot.

Firstly, according to the selected dance species and humanoid robot, “Dance Expressive Space” has been instantiated. The Nao robot has 26 joints and their corresponding latitudes, which can be acquired easily from its technical manuals, so the set of J and D can be determined. Considering the features of Tibetan Tap and the limitations on the physical structure of the robot, we divide J into four limbs L = {L₁, L₂, L₃, L₄}, and then assign the suitable constraint values C = {C₁, C₂, C₃, C₄} for these limbs. The details about the set of limbs (L) and constraints (C) are shown in

Table 2. The details of dividing the limbs on the NAO robot.

Name	Nickname	Constraint	The Set of Joints Contained in The Limb
L1	Head	C1 = 0	{HeadYaw, HeadPitch}
L2	Upperbody	C2 = 0	{LShoulderPitch, LShoulderRoll, LElbowYaw, LElbowRoll, LWristYaw, RShoulderPitch, RShoulderRoll, RElbowYaw, RElbowRoll, RWristYaw}
L3	Hand	C3 = 1	{LHand, RHand}
L4	Lowerbody	C4 = 1	{LHipYawPitch, LHipRoll, LHipPitch, LKneePitch, LAnklePitch, LAnkleRoll, RHipYawPitch, RHipRoll, RHipPitch, RKneePitch, RAnklePitch, RAnkleRoll }

.

Since lower body plays an important role in keeping balance and reflecting the features of Tibetan Tap, the constraint on the limb L₄ (Lowerbody) of the robot is assigned to 1 (C₄ = 1), so the method of imagination will not work on limb L₄. In the Tibetan Tap, the human dancer always maintain his hand in a naturally relaxed state, and does not need to vary. Therefore, the constraint on the limb L₃ (Hand) of the robot is assigned to 1 (C₃ = 1), and the limb L₃ does not take part in imagining. Moreover, due to the special effect of dance representation [24], the limb L₁ (Head) and L₂ (Upperbody) have more flexibility on performance than L₃ and L₄, so both C₁ and C₂ take the value 0. Therefore, the robot can imagine new dance elements on the limb L₁ and L₂ based on the imitated dance elements.

Secondly, we implemented the imitation method of Reference [25], and the basic dance elements of each limb are imitated, respectively. In our experiment, seven basic dance elements on the Head limb are imitated and these are illustrated in Figure 4. Figure 5 shows 13 basic dance elements on the Upperbody limb are imitated. Additionally, seven basic dance elements on the Lowerbody limb that can represent the quiver and step characteristics of Tibetan Tap are imitated (as shown in Figure 6), and the only dance element on the Hand limb is fixed. Furthermore, to acquire a larger number of imagined dance elements on limb L₁ and L₂, the neighborhoods d₁ and d₂ take the value of 1.

After imagining autonomously by the above method, the robot has imagined 1103 new dance elements on the Head limb, and 26,462 new dance elements on the Upperbody limb: 19,670 dance elements are imagined by Algorithm 1; 3782 dance elements are imagined by Algorithm 2 (symmetrical); and 3010 dance elements are imagined by Algorithm 2 (antisymmetrical). Figure 7, Figure 8, Figure 9 and Figure 10 show respectively some of them. So the robot has a total of 1103 (Head) × 26,462 (Upperbody) × 1 (Hand) × 7 (Lowerbody) = 204,313,102 possible new dance poses.

Next, to generate a new dance pose, the robot randomly selected an imagined dance element on every limb, and assembled these dance elements together. By balance detection, valid (balanced) dance poses were imagined and stored in “Dance Memory Space”, and they served as the basis of imagining dance motions further. As shown in Figure 11, some of new dance poses are illustrated. When a sufficient number (e.g., 1000) of dance poses were ready, the imagination of dance motion started. After the number of the concatenated dance poses (the length of a dance motion, takes value in [3,20]) was determined randomly, a dance motion was imagined by concatenating the dance poses, which were selected randomly from “Dance Memory Space”.

To imagine a valid (balanced) dance motion, the linear interpolation, balance detection, and balance compensation were used between any two adjacent dance poses in a dance motion. As shown in Figure 12, a new dance motion is illustrated. Similarly, the number of the concatenated dance motions (takes value from References [3,20]) in a robotic dance was determined randomly by the robot itself, and the concatenated dance motions were selected randomly too. In any two adjacent dance motions, the linear interpolation, balance detection, and balance compensation were used between the last dance pose of the previous dance motion and the first dance pose of the next dance motion. Finally, a robotic choreography was created autonomously, as shown in Figure 13.

To perform on a real NAO robot, the robotic choreography, which is created by our mechanism, shall be mapped from the simulation environment (“Webots”) to the control software of NAO robot (“Choregraphe”). However, “Webots” uses radian as the measurement unit of joint value, and “Choregraphe” uses degree yet. After the measurement unit of joint value is converted by the formula (degree value = radian value*180/π), the created robotic choreography in “Webots” can be transformed to the equivalent form in “Choregraphe”, which is described by a motion file. Then, the example dance motion (see Figure 12) is performed on a real NAO robot (shown as Figure 14), and Figure 15 shows the example robotic choreography (see Figure 13), which is created by our mechanism on a real NAO robot.

5. Evaluation and Analysis

To verify if the method of imagination is effective, 10t human dance experts were invited to evaluate the innovativeness and preservation of he dance’s characteristics on the imagined dance elements, poses, motions, and robotic dances. The quality evaluation standard of these dance objects they used is shown in

Table 3. The quality evaluation standard of dance objects (tibetan tap).

Dance Object	Preservation of Tibetan Tap’s Characteristics	Innovation
Dance element (Head)	Head swings harmoniously with body’s movement, and head presents the status of natural relaxation.	The amplitude, range and direction of head swing have changed reasonably.
Dance element (Upperbody)	Upper body stays relaxed and leans forward slightly; and arms swing leisurely around (beside) crotch.	The amplitude, range and direction of arm swing have changed reasonably.
Dance pose	Dance pose is steady, vigorous, enthusiastic, free and easy; upper body stays relaxed and leans forward slightly; and the overall shape fits in with the carefree and leisurely personality trait of Tibetan folk dance.	There are innovation and breakthrough in the aspects of pose fluidity and shape variability.
Dance motion	Knees are loose and elastic, and steps are flexible. Dance poses are concatenated reasonably to form a dance motion, in which upper body movement coordinates rigorously with lower body movement, and it has a certain expression. The simple, free, swift and nimble Tibetan characteristics have been expressed sharply.	There are different embodiments in the variety of motion, rhythm, velocity and power.
Robotic dance	In robotic dance, the moving of gravity center, which is formed by regular quiver and regular step, drives the loose upper limb movement. Robotic dance has clear movement rhythm of quiver, and has holistic and agglomerate kinesthesia. The Tibetan characteristics of honesty and depth have been expressed sharply.	There are natural and smooth cohesions between two adjacent dance motions. Furthermore, there are some innovations in the aspects of cadenced rhythm, interlaced step, and varied motion combinations.

. None of them had any experience with robots. However, they have the rich experiences of stage performance and teaching, and each has a high ability of aesthetic appreciation and evaluation of Chinese folk dance. Notably, each human dance expert completed his/her own evaluation independently, and each dance object (a dance element, pose, motion, or robotic dance) was assessed discretely (i.e., a dance pose first, then a second dance pose etc.).

In the procedure of evaluation, we randomly chose 30 imagined dance elements on Head limb, 50 imagined dance elements on Upper-body limb, 50 imagined dance poses, 10 imagined dance motions, and 10 robotic dances respectively. Each evaluator was told to grade an object (a dance element, pose, motion, or robotic dance) on the innovativeness and preservation of dance’s characteristics respectively with a scale of 1 to 10. The value of 1 represents the worst and 10 represents the best. After the values of evaluation on the same object that come from all of the evaluators were acquired, a highest value and a lowest value were removed firstly (to avoid the adverse effects caused by the extreme data); and then the rest of the values took the average and so did all the objects. Furthermore, to understand these evaluation results, we introduced the quality scales of these dance objects: excellent (9.00–10 points), good (7.50–8.99 points), general (6.00–7.49 points) and bad (0–5.99 points) [15].

Figure 16, Figure 17, Figure 18, Figure 19 and Figure 20 show the results of evaluation on the imagined dance elements, poses, motions, and robotic dance respectively. The x-axis is the serial number of the evaluated object (a dance element, pose, motion, or robotic dance), and the y-axis is the value of evaluation on the innovativeness and the preservation of dance’s characteristics. After calculation, we have:

• The average score of the 30 imagined dance elements for the Head limb is 7.30 points on keeping the characteristics of dance, and 7.55 points for innovativeness.
• The average score of the 50 imagined dance elements for the Upperbody limb is 7.72 points on keeping the dance characteristics, and 7.39 points for innovativeness.
• The average score of the 50 imagined dance poses is 8.01 points for keeping the characteristics of dance, and their average score is 7.27 points for innovativeness.
• The average score of the 10 imagined dance motions is 7.93 points for keeping the characteristics of dance, and their average score is 8.01 points for innovativeness.
• The average score of the 10 created robotic dances is 7.44 points for keeping the characteristics of dance, and their average score is 7.54 points for innovativeness.

Table 4. The synthesis statistics results of dance objects’ quality evaluation.

Dance Object	Preservation of Tibetan Tap’s Characteristics (Average Score/Level)	Innovation (Average Score/Level)	Preservation of Tibetan Tap’s Characteristics (Standard Deviation)	Innovation (Standard Deviation)
Dance element (Head)	7.30/general	7.55/good	1.21	1.32
Dance element (Upperbody)	7.72/good	7.39/general	1.21	1.41
Dance pose	8.01/good	7.27/general	1.07	0.82
Dance motion	7.93/good	8.01/good	1.18	0.78
Robotic dance	7.44/general	7.54/good	0.31	0.27

shows the synthesis statistics results of these dance objects’ quality evaluation, including average score, level, and standard deviation. Notably, because all of these evaluated dance objects were chosen (sampled) randomly from their corresponding total space, we calculated the standard deviation of the samples (maintaining two digits after the decimal point). These results show that our mechanism could create the innovative dance elements, poses, motions, and robotic dances with keeping the characteristics of Chinese Tibetan Tap, and could reconcile effectively the contradiction between the innovation of robotic dance and the preservation of dance characteristics. These human experts have satisfied the robotic dances created by our mechanism.

6. Discussion

The experimental results showed that the proposed mechanism could create robotic choreography autonomously by a humanoid robot. The produced robotic dance is the combination of the innovativeness and the characteristics preservation of human dance. Based on the experimental results, some open-ended questions of the proposed mechanism will be discussed in terms of three aspects below.

6.1. Robotic Dance Formalization

Instead of a robotic dance motion formalization aiming to realize the synchronization for musical beats and emotions in Reference [17], we aim to accomplish the autonomous creation of robotic choreography keeping the characteristics of dance by the humanoid robot itself, so our formalized expression of robotic dance on the humanoid robot is different from it.

Table 5. The equivalence and difference on concepts between our “dance expressive space” and [17].

Dance Expressive Space in This Paper	Dance Motions Formalization in [17]	Remarks
Joint (Ji)	Joint	Equivalence
Value of joint (V)	Value of joint (V)	Equivalence
Domain (Di)		Difference
Limb (Li)	Category of joints (c)	Equivalence
Constraint (Ci)		Difference
Dance element (DEi)	Keyframe (Kc)	Equivalence
Dance pose (DPi)		Difference
Dance motion (DMi)	Motion primitive (Mc)	Equivalence
	Schedule of motion primitives (Sc)	Difference
Robotic dance (RD)	The assembly on all the schedule of motion primitives {SHead, SLArm, SRArm, SLegs} )	Equivalence

shows the equivalence and difference on concepts between them.

Essentially, our “Dance Expressive Space” considers the whole body of the humanoid robot on the concepts that are from “dance pose” to “robotic dance” in Table 5, and the dance motions formalization in Reference [17] only considers it in the last concept of Table V. Meanwhile, due to the absence of support on the concept of constraint and dance pose, the method of imagination proposed in this paper cannot be built on the dance motions formalization in Reference [17]. Additionally, the robot produced a robotic dance by executing four schedules of motion primitives {S_Head, S_LArm, S_RArm, S_Legs} simultaneously [17], and the method does not consider keeping the characteristics of dance. Ultimately, it is clear that our “Dance Expressive Space” is necessary for supporting the whole mechanism and framework in this paper.

6.2. The Imagined Dance Elements

Obviously, the number of possible combinations on the values of the domains of joints in a limb is huge. For example, the limb L₁ (Head) has two joints (HeadYaw, HeadPitch) in our experiment, and their domains are [−209, 209] and [−67, 51] (the unit of data is radians, and the data are magnified 100 times to facilitate our computation) respectively. So there are 418 × 118 = 49,324 possible combinations for the values of the domains of joints in the limb. However, the number of the imagined dance elements on the limb is 1103, which are inspired by seven imitated dance elements on the same limb. Notably, although the set composed of the imagined dance elements is a proper subset of all possible value combinations of the joint domains in the same limb, the former one is a proper subset that selects elements and keeps the characteristics of dance from the latter one.

The experimental results show that the number of the imagined dance elements on each limb plays a decisive role in the number of the imagined dance poses, motions, and works. Therefore, we explore these parameters that influence the number of the imagined dance elements, including the number of joints in a limb (|L_i|), each joint’s domain in a limb (D_k), the number of the imitated dance elements on a limb, and the neighbourhood (d_i) on a limb L_i. Obviously, the number of imagined dance elements on a limb is proportional to the number of joints and each joint’s domain in the limb, but is inversely proportional to its neighbourhood.

However, there is a quadratic function between the number of imagined dance elements and the number of imitated dance elements. At the beginning, the number of the imagined dance elements will increase with the growth on the number of imitated dance elements. After a critical point is reached, the number of imagined dance elements will decrease with the growth on the number of the imitated dance elements. In a few cases, the robot imitated all possible dance elements on a limb, and the number of the imagined dance elements was 0.

6.3. The Limb and Its Constraint

Every dance species has its own characteristics, and this is the essence in which a dance species differs from the others. The dance’s characteristics belong to the category of subjective aesthetics, and it is difficult to be defined by formal symbols. Therefore, according to the characteristics of dance, the division of the limbs and the determination of the constraints’ values are needed to be completed manually, and the work can be done easily by the observation and background knowledge of the dance. However, there is no common method so far and the problem remains to be settled in the future.

Additionally, if the dance’s characteristics can be defined by formal symbols, an interesting research direction is how to make a robot learn to divide the limbs and determine the constraints’ values autonomously. This will help improve the effect of our mechanism and develop the autonomous dance-creating ability of the robot further. Also, the machine learning algorithms can be used to further improve the performance after enough training samples are collected.

7. Conclusions

Inspired by the methods of human dance creation, this paper presents a novel mechanism for a humanoid robot to create robotic choreography autonomously. The proposed mechanism combines the innovativeness and imagination of human dance. The main contributions include the construction of a dance expressive space, the new framework for autonomous creation of robotic dance choreography, the mechanism for producing new dance elements (poses and motions), as well as the generation of robotic dance movements. The experiments were conducted on a real NAO robot and results were presented to verify the feasibility and performance of the proposed mechanism.

For future work, we will extend the method of imagining new dance elements from a single joint to more joints while keeping the characteristics of human dance. The complicated problem of balance will be considered further, and more experiments will be conducted on our NAO humanoid robot. We also aim to improve the method of concatenating dance motions by considering the similarity between the two adjacent dance poses.