1. Introduction
The production of film and television is a technology that introduces new ideas and concepts to audiences through professional photography and visual processing techniques, incorporating more artistic elements. Film and television production usually includes film shooting, sound processing, and special effects. Artificial art mainly comes from script selection, the actors’ acting, and directorial guidance [
1,
2,
3]. There have been many explorations on film and television production. Williams and Don (2015) pointed out that there are many differences between American film and television production and Chinese film and television production; in China, technologies of film and television production are weaker, and it is difficult to incorporate new technologies into films [
4]. Globally, the mode of film and television production is often a “Franchise”, which contains not only the story itself, but also its sequels, anecdotes, and prequels to continuously expand and extend the past and present of this story to form a more complete system [
5]. The “Universe” mode of Marvel applies the common inter-serial serialization mode to films [
6]. Such a novel theme and form has attracted the attention of many audiences. With the rapid development of the Internet industry and the rapid growth of artificial intelligence, Virtual Reality (VR), and big data industries, forms of film and television works are integrated with the new technologies. This has become a global development trend of the film and television industry, which requires China to value the content and form innovation in the film and television production process [
7]. Therefore, studying the form and production modes of film and television production, ensuring the diversified development of the film and television industry, and satisfying the audiences’ pursuit of art has become a scientific problem in this field that needs to be solved urgently.
VR technology uses computer simulation to generate a virtual world in a three-dimensional space, providing audiences with a simulation of vision and other senses, making the audiences feel as if they are immersed in the environment, and enabling the audiences to observe things in three-dimensional space instantly without restrictions [
8,
9,
10]. The Internet of Things (IoT) is an information carrier based on the Internet and traditional telecommunication networks, which allows all common physical objects that can be independently addressed to form an interconnected network [
11]. A survey has shown that the application of VR technology in films has significantly improved the audiences’ entertainment and experience. The public has a positive attitude towards the application of new technologies in the film and television industry. There have been many researchers on the utilization of new technologies in film and television production. Shafer et al. (2018) used VR and IoT technologies to improve the efficiency, reduce costs, and save resources in film and television production [
12]. Song and Wook (2020) found that the virtual character interaction system based on the IoT technology had a higher rate of action recognition, generating a strong sense of immersion among audiences, which could realize the real-time capture and imitation of character actions [
13]. Therefore, VR and IoT technology will play an essential role in film and television production; then, how to apply them in film and television production to establish new production methods and production models is of great significance for the development of China’s film and television industry.
Therefore, based on VR and IoT technology, by using S3 Studio Max software, Photoshop software, somatosensory interaction sensors, and voice input, a smart network platform between users, devices, and films is built. Based on Van Gogh’s paintings and the above-mentioned technology, the film Van Gogh in Dream was produced, which realizes the visual, auditory, and interactive experiences of the audience and truly enriches the form of film and television production. The results will provide a theoretical basis for the application of artificial intelligence technology in film production and production mode.
2. Materials and Methods
2.1. Film and Television Work Creation Goals and Plot Design
Van Gogh in Dream is the product to be created. First, ten paintings of Van Gogh were processed (Arles City Langlois Bridge, Aniere Seine Bridge, Van Gogh’s House in Arles, White House at Night, Arles’ Bedroom, Auer’s Church, Rhone Starry Sky Over The River, Night Cafe: Outdoor, Night Cafe: Indoor) in three dimensions based on artificial intelligence technology in VR and Internet of Things. Then the processed paintings were seamlessly stitched into a European town with oil painting texture, a set of long lenses were used to display Van Gogh’s oil paintings one by one, and the corresponding body sense was added to realize the interactive effect through the transmission of sound, and the prosperous life of Europe in the 18th century was created. It is hoped that in this way, the audience can experience the town where Van Gogh lived, and use the technology of VR and the Internet of Things to appreciate Van Gogh’s famous paintings.
In the plot of Van Gogh in Dream, the audience explores freely in this small town, and through continuous exploration, although they cannot see Van Gogh himself, they have a complete understanding of Van Gogh’s life. Although they do not find Van Gogh himself, in the process, they find the spirit of Van Gogh’s devotion to art.
2.2. Research Methods for Interactive Film and Television Production
To effectively analyze the design and application of VR and IoT in the art of film and television, many pieces of literature on the design and application of art, VR, and IoT were collected in various ways, as well as the film and television artworks utilizing VR technology. Then, the collected data and cases were summarized, the relevant research results refined, and a general idea and research system of VR and IoT technology in film and television arts was formed.
The overall design of the questionnaire survey is as follows. The film
Van Gogh in Dream was compared with the traditional films
Loving Vincent,
Vincent Van Gogh:
Painted with Words, and
Van Gogh in Dream. The evaluations and scores of traditional films were all from Douban, a popular film commenting website. The questionnaire was designed from four dimensions: the viewing effect, knowledge acquisition, historical understanding, and impression evaluation. Each dimension involved five closely related topics. Furthermore, 10 suggestions for improvement were designed, with a total of 30 questions. Each film had a questionnaire. To facilitate the statistics and understanding, the answers to the five questions were all in a consistent manner. First, the reliability and validity of the results of the questionnaire were analyzed. On this basis, the corresponding proportions of the results of five questions in different dimensions were summarized and calculated for analysis and discussion. The specific evaluation system is shown in
Table 1.
The suggestions for improvement were mainly for the film made, and the specific results and contents were discussed in the discussion section. Subjects and methods of questionnaire survey: The survey subjects were mainly from the Internet. The important plots of different films were intercepted, and the VR content was added; each film clip was kept at about 20 min and published on the Internet to users who were willing to participate in the survey. The VR glasses were distributed to the participants and film-viewing guides were provided. Audiences took turns to watch different clips. Finally, the survey participants were invited to fill in an online questionnaire survey. Each clip has its questionnaire, but the questions for each clip were the same, of which 250 questionnaires had been distributed online and 220 valid questionnaires have been received, accounting for 88%.
2.3. Interactive Design at the Film Planning Stage
2.3.1. Formal Interactive Design Application
The goal of the scene design of the film and television works is: more than ten pieces of Van Gogh’s famous paintings are made three-dimensional by three-dimensional modeling, and then they are spliced into a European town with oil painting texture and presented in the form of VR.
Figure 1 is a scene plan of
Van Gogh in Dream, in which the scene pays special attention to its perspective relationship when it is made three-dimensional, so that it can achieve the effect of being substantially the same as the original painting when viewed from the head-up angle. The characters, which are also the characters in Van Gogh’s works, have another function to unify the proportion of the scene. Because the size ratio of each painting is different, they must be stitched together in an appropriate ratio when they are composed of the scene, so the size of the characters in each painting needs to be integrated with the entire scene.
The plot structure of VR films no longer follows the traditional linear narrative model; audiences can choose different perspectives, or plot development trends, to intervene in the process of story generation to a certain extent and change the process of event occurrence. The work Van Gogh in Dream establishes many emotional nodes, so the audience can click on the work on the screen according to the guidance of actions and sounds. At the end of the narration, a selection screen pops up asking the audience whether to continue to learn more information. If “yes” is selected, the background of the original drawing is displayed, and the user is allowed to choose to continue or listen again; if “no”, the audience continues to follow the mainline. These interactive main lines and auxiliary lines need to be carefully designed, arranged, and have greater freedom when they are created.
In the work Van Gogh in Dream, not only the plot structure and character creation should be noticed, but also the production workload should be estimated. Therefore, when graduation work is designed: first, it is necessary to test whether the original work can be three-dimensional, whether the style after becoming three-dimensional is loyal to the original painting, whether VR can output image sequences to test the distance between scenes and other factors, and estimate the production cycle. At the same time, try not to use too many words of transparency and overlap, and reasonably reduce the number of polygons to ensure the smooth operation of the program.
2.3.2. Interactive Design Applications on Content
To highly restore the authenticity of history, a special technical team is built in the early stage, which includes professionals in history, art, and the humanities to provide professional imaging services for the works. Different from the other VR films, Van Gogh in Dream is based on real historical stories. Therefore, when recreating the historical content, it is necessary to restore the historical content to the maximum. So, relevant Western painting researchers are specially interviewed, and after many discussions, the final plan is designed jointly.
Shown in
Figure 2 is a bird’s eye view of the work
Van Gogh in Dream. The use of VR art to interpret Western painting art is a process of artistic deconstruction and reconstruction. Specifically, in the process of transforming a two-dimensional painting into a three-dimensional scene, it is necessary to grasp the overall color tone, and the style of the strokes should be as close as possible to the original style. At the same time, the perspective adjustment must be combined with the characteristics of VR art itself. In the work
Van Gogh in Dream, the features of Van Gogh’s strokes are extracted and repainted in Photoshop to restore the strokes in a three-dimensional space. Furthermore, some creative elements can be added, such as in this starry night, only the sky part is extracted as the third scene in the sky, and at the same time, camphor trees are placed in the first scene. Any of Van Gogh’s painting on the scene can be interactively clicked to enter the interpretation of a painting, and as long as the artistic style is consistent, the content of the scene can be combined at will, making the VR space richer and more layered.
2.4. Interactive Design Application Based on Somatosensory
The main somatosensory interaction design used in
Van Gogh in Dream is painting introduction and character interaction. The first scene is
The First Step and Langlois Bridge in Arles. When the audience walks in
The First Step, a picture of people’s inquiries appears, mainly to let the audience understand the background of the painting and Van Gogh’s personal story. Then, the audience is guided by two people and the train to a turning point, where there are people from the
White House at Night passing by, who the audience can also communicate with. They continue to pass
Arles’ Bedroom, at this time a person comes out of the house, and after a conversation, he leads the audience forward, and leaves after
Aver’s Church. At this point, the audience can find people to communicate with and finally communicate with the characters in
The Starry Sky on the Rhone River. In the same way, the audience is led to the
Night Cafe: Outdoor and the
Night Cafe: Indoor, as shown in
Figure 3. The experiencers explore the night cafe and communicate with them, and finally fully understand Van Gogh’s life.
2.4.1. Interactive Design Bases on Screen
In the work
Van Gogh in Dream, the interaction based on the interactive screen is mainly realized, so that the screen can be changed according to the movement of the audience, and can be more sensitive to the movement of the audience up, down, left, and right. As shown in
Figure 4, if the audience watches through mobile devices, such as mobile phones and tablets, they can not only change the screen through the movement of the body but also change the viewing angle by touching the screen.
2.4.2. Interactive Design Based on Gestures
In terms of gestures, the design goal is to select each picture by selecting the handle. When the selection is completed, the historical background of the picture and the description of Van Gogh’s life status is displayed, then a window pops up asking the audience if they need further explanation of the artistry and composition knowledge of the work. If the viewers click “Yes”, they enter the explanation screen of the painting, and if the viewers click “No”, the next drawing window will be closed.
2.5. Interactive Design Applications Based on the Sound
In terms of language, the main interaction is the character dialogue through which the story can be understood, and then the attention of the audience can be aroused and watch direction can be guided through voice. For example, in
Van Gogh’s Home in Arles, a woman walks out of Van Gogh’s bedroom, and when the audience yells at her, she stops to observe the direction of the audience and waits a few seconds for the audience to respond, then begins to explain the historical background of Arles’ Bedroom. As shown in
Figure 5, in the direct dialogue between the audience and the characters in the painting, this function needs to be used in conjunction with artificial intelligence, and hence it is a huge system, and this function may be tried in the future.
In terms of sound, the sound of the crowd, the train, the wind, and walking, attracts the attention of the audience and increases the immersion of the atmosphere. For example, at the scene of the Anil Seine Bridge, a train painted in the center of the bridge in the original painting slowly arrives. To have the audiences see the picture at the most appropriate time, the sound of the whistle of the train gradually passes from low to high to the ears of the audience when the train first enters the picture. When listeners hear the whistle, they naturally look for the source of the sound to see the train passing the bridge. Furthermore, the train runs at medium speed, and when the audience is tested, 90% of them can see the scene where the train passes.
2.6. Construction of Experimental Hardware and Software Equipment and IoT Platform
According to the process of film design, it is necessary to utilize some means to improve the quality of the images input into the VR devices. Therefore, the interactive design mainly adopts three-dimensional modeling. With the help of S3 Studio Max software, through the input of the collected paintings of Van Gogh, specific three-dimensional models are automatically generated, and the details are corrected professionally. Interactive design of the content utilizes Photoshop CS6 software to correct the color of the constructed three-dimensional models, thereby ensuring the quality of the images. The application of interactive design based on somatosensory aims to increase the interaction with audiences, which depends on the IoT for platform establishment. Among them, the screen-based interactive design uses infrared sensing and somatosensory recognition functions in the display of the screen, including devices such as mobile phones, tablet computers, and televisions, allowing audiences to watch different scenes in real-time by moving the screen or touching and dragging. Gesture-based interactivity adds an operating handle, in addition to the screen. The audiences can control the screen through the handle. The sound-based interactivity design uses Twirling 720, a panoramic sound recording device of Twirling. The product is characterized by high degree of integration and is pocket size. In terms of software, a panoramic sound plug-in called Work is utilized, which can seamlessly dock with video editing software. A major feature of this software is that it displays the position of the sound source in the video by means of punctuation. It provides an intuitive spatial positioning and assigns independent audio tracks so that the sound source can be located while mixing the audio. Through the use of VR technology in content, with the help of somatosensory interactive sensors and voice recording, a smart IoT platform between audiences, devices, and films is built.
2.7. Data Analysis and Processing
The questionnaire survey is analyzed by the following methods. Cronbach’s α coefficient is adopted for the reliability analysis. When the coefficient is between 0.7 and 0.8, the questionnaire results have a high degree of credibility; when the coefficient is between 0.65 and 0.7, the reliability is within the acceptable range; when the coefficient is between 0.6 and 0.65, the questionnaire survey results are not credible. The Ratio Statistic Test (RST) method is utilized for the validity analysis, whose judgment criteria include the Redundancy Degree (RD) and the Sensitivity Degree (SD). The RD represents the independence and redundancy of each indicator. When RD is ≤0.5, the indicator is valid. The smaller the RD value is, the higher the validity is. The SD represents the adaptability of different evaluation systems on evaluation indicators. When SD is ≤5, the indicator is valid. Matlab 7.0 software is utilized to test its consistency. Among them, CI is the consistency indicator, which indicates the range of population parameters estimated according to a certain probability, and CI can be used to estimate population parameters. The smaller the range of this value, the better the reliability of estimating population parameters with sample indicator. CR is the consistency ratio, which must be less than 0.1, so that the judgment matrix can meet the requirement of consistency test. RI is the average random consistency indicator, which is calculated to reduce the error caused by multiple CI. All questionnaires are analyzed using Student’s T-test and Levene test. The figures are drawn with the Origin 2019 software.
4. Discussion
The opinions on film and television production are mainly collected through questionnaire surveys, which include interactive design in form, content, somatosensory, screen sound and gesture. The aim is to improve the viewer’s film-viewing experience. After analysis and sorting, the following problems are found. (1) The interface design: according to the evaluation feedback, because the size of the interactive screen is too large, the image at the border cannot be seen, causing visual distortion, making audiences feel uncomfortable, and even destroying the immersion experience. (2) Resolution of images: since traditional films are shot with professional cameras, while the VR film for the test has poor collection quality of original images, the images are therefore blurred, and the experience is reduced. (3) The display of text: the subtitles of traditional videos are all at the lower center position of the screen, and the font size is uniform. However, to make the film interactive, the texts appear mostly in the middle, and the font and font size are quite different from the traditional films, which leads to a reduction in the viewing experience. Later, improvements in these areas will be made, to continuously improve the production level of technology.
Compared with traditional films, the advantages of films produced by VR and IoT technologies are more obvious [
14]. VR technology promotes the transformation of the traditional film industry, which allows audiences to feel the charm of films more intuitively, and opens up a new field for people to explore films [
15]. At present, most VR films are short video clips; the longer films are mainly documentaries, and VR dramas are mostly traditional propaganda films. So far, there has not been a reasonable and effective exploration of artificial intelligence film production. The current VR works are just some attempts; the films are mostly panoramic videos, and the produced interactive VR films are far from enough [
16]. Artificial intelligence is a social revolution driven by technology. Everyone will gain different life experiences through VR and IoT [
17]. Although the application of VR and IoT technologies in the field of films has the particularity of using audiovisual languages, any kind of innovation will inevitably have limitations [
18].
To realize the application of this technology in film and television production, the following issues should also be valued: (1) high cost: in terms of viewing, although some basic VR glasses are used in mobile phones, such VR glasses can only satisfy the curiosity of people who have never seen VR images. Once curiosity is satisfied, problems caused by low-end devices also appear, such as clarity, dizziness, incorrect interactions, and other problems. To get a better experience, the more professional headsets can be bought for a minimum price of more than 2000 yuan, which is relatively high for ordinary families. Therefore, at the audience level, there is a threshold to watch films through VR [
19]. In terms of production, it takes a lot of money to produce a major panoramic video, and the resolution must be at least 4K or more to get a clear picture, and professional 4KVR cameras also require tens of thousands of yuan [
20]. (2) Vicious circle: due to the certain thresholds in the interactive experience mode of VR films, the audience of VR films is relatively small, resulting in investment costs that are difficult for film producers and film producers to recover [
21]. The recovering costs cannot invest more in VR films, leading to a reduction in movie production, and the reduction in VR movie production leads to a lack of audience, which is a vicious circle [
22]. To break this cycle, first, a clear realization idea should be explored, and then corresponding films according to the needs of the audience should be made, and finally, the audience’s viewing threshold should be lowered to achieve the effect of universal participation [
23].
5. Conclusions
Based on VR and IoT technology, a VR film and television production system is built with the help of S3 Studio Max and Photoshop software; meanwhile, a smart interactive IoT system between users, devices, and film and television works is built through somatosensory interaction sensors and Twirling720 [
24,
25]. By using this production model of film and television production, the film
Van Gogh in Dream is produced, which has a higher technical level. Compared with traditional film and television works, it has brought different film-watching experiences to audiences in terms of viewing effect, knowledge acquisition, historical understanding, and impression evaluation. The facts prove that this artificial intelligence VR and IoT-based film and television production model has huge advantages, which provides new research ideas for film and television production [
26]. However, because artificial intelligence interaction technology is still in the initial stage of development, there are few reference materials, and it is difficult to perform a comprehensive and in-depth study of VR interaction based on the existing data. Therefore, there are the following shortcomings: (1) The proposed interaction form only stays in the sense of body and sound, and other interactive forms need to be further explored. (2) Professional institutions are not invited to review the film and television works produced. Therefore, these two aspects will be improved to truly enhance the level of film and television production and apply it to the market.