Scaling-Up Social Learning in Small Groups with Robot Supported Collaborative Learning (RSCL): Effects of Learners’ Prior Experience in the Case Study of Planning Poker with the Robot NAO

Abstract

The COVID-19 pandemic accelerated the shift to online learning in higher education, making the need to rethink pedagogical approaches to on-campus education stronger than ever before. While online learning offers a wide range of benefits for both teachers and students, social, face-to-face interaction provides a strong rationale for in-person education. This paper proposes a novel approach to enhancing in-person social learning experiences through robot-supported collaborative learning facilitated by a social robot NAO in the case study “Planning poker with NAO”, which is a collaborative game-based activity facilitated by the robot. The case study was designed to provide a scalable social learning experience in small groups of students. The evaluation results from a study with 46 university students who played Planning poker with NAO explore different quality aspects of the learning experience. The results indicate that students valued both the social learning experience and the robotic facilitator. The analysis revealed some significant differences among students with and without prior experience in interacting with NAO, suggesting directions for future research related to novelty and familiarisation effects as well as to scaling up of social learning in small groups by applying social robots.

1. Introduction

Building on the achievements of computer-supported and online education movements, which can be traced back to the 1980s, the COVID-19 pandemic has accelerated the global shift to online learning in higher education and beyond. Prior to the pandemic, many universities had already deployed or started to deploy online formats based on a variety of models for online, open, flexible, and technology-enhanced higher education [1]. With online education becoming mainstream rather than a trend, the need to rethink pedagogical approaches and pedagogical roles for in-person, on-campus education has become prevalent [2,3,4,5]. While online learning allows to make higher education accessible to a wider range of students, social interaction in the physical space is as important as ever. Not surprisingly, many educational institutions have embraced the opportunity to combine on-campus and online teaching and learning to provide more flexible models, such as hybrid and HyFlex, in the post-pandemic times [2].

1.1. Background

While online learning in higher education offers a wide range of benefits for both teachers and students, there is a strong rationale for in-person teaching and learning on campus. The rationale for in-person education has been seen especially in enabling direct social interaction, hands-on and collaborative learning as well as sustained motivation through presence and communication in a physical academic environment [3]. The qualitative research study by [4] asked the following question: “Has COVID-19 killed the campus?”. The results from interviews with faculty and students in multiple countries showed that both higher-education teachers and learners perceive social interaction in the physical space to be the core of the campus experience [5]. The shift from face-to-face to fully online education during the pandemic phase was often perceived as an unpleasant experience, mainly due to the absence of direct social interaction and the resulting physical and social distancing [5]. The literature review by [5] revealed the experience of emergency remote teaching (ERT), which forced the shift to online learning and teaching (OLT) due to the well-known emergency situation, can be gradually and meaningfully integrated with methods of in-person interaction in physical spaces in the post-pandemic times.

Challenges experienced by teachers and learners during the pandemic spurred many discussions about the future of on-campus education [4,5]. The study by [5] asked the following question: “Now that campus-based university teachers have experienced the unplanned and forced version of OLT, how can this experience help bridge the gap between online and in-person teaching in the following years?” The expert interviews elicited some plausible predictions and expectations, such as “the use of technology will be broadly considered in all programs as a strategic and essential aspect”, “teachers are organisers of events in which students meet with a purpose”, in a ”technology-rich learning environment, the teacher acts as tutor, organiser of the learning process, curator of the learning resources, motivator and project manager of students’ learning” [5]. Moreover, the interviewed experts emphasised the importance of using active learning methodologies such as problem-solving and game-based learning [5].

The gradual shift from lectures for large groups of students to collaborative activities in small groups started before the pandemic and has been seen as an important catalyst for the transformation of the role of higher-education teachers. The teacher‘s role has been transformed from a content provider to a group facilitator, as advocated by educational approaches such as flipped-classroom and collaborative learning in groups [6,7,8]. Active, participatory learning in small groups has been shown to be more effective, both in terms of content retention and engagement, while increasing the demand for attention of teachers to the individual needs of students [8]. The role of the teacher as a group facilitator has focused on the encouragement of engagement, discussion, and participation of all the group members [8]. However, the new role of a facilitator has been also challenging, especially for teachers aiming to provide individualised attention and support to large groups of students. Therefore, methods for scaling up small group facilitation have gained importance in academic discussions [8]. The OECD report “Digital Education Outlook 2021: Pushing the frontiers with AI, blockchain and robots” identified social robots as one of the three technologies that will transform education systems [9]. The case study presented in this paper shows how social robots can be used to scale up small-group facilitation in the context of in-person teaching and learning.

1.2. Rationale and Research Questions

The key insights from the pandemic can be summarised as the need to meaningfully integrate technologies in both online and in-person learning while embracing an ongoing digital transformation of higher education. From the perspective of this broader context of digital transformations of higher education, this paper describes a novel approach to enhancing and scaling up in-person social learning experiences through Robot-Supported Collaborative Learning (RSCL). Specifically, this paper describes the design, implementation, and evaluation of a collaborative game-based learning activity facilitated by the social robot NAO. The collaborative activity “Planning poker with NAO” was designed to provide a scalable in-person learning activity in small groups of students. In the case study, the activity was facilitated by the robot NAO, which was applied as an embodied technology to provide an engaging social learning experience for university students in the course on agile project management. As outlined in the OECD 2021 report, social robots can be assigned different educational roles, such as tutor, teacher, or peer, to support social interaction and learning in an efficient and nonjudgemental way [9].

The broader rationale for the case study “Planning poker with NAO” is threefold and encompasses the need (a) to provide scalable collaborative learning experiences in small groups for a larger number of groups; (b) to prepare university students for robotic interactions in future workplaces, which will be on the rise due to technological innovations in humanoid robotics (HR) and artificial intelligence (AI); and (c) to explore how emerging embodied technologies such as social robots can be used on the campus to enhance in-person, social learning. The case study presented in this paper explores the potential of applying a social humanoid robot as a facilitator of a collaborative learning activity by providing a structured and scalable template for a learning activity, which can be offered repeatedly for a larger number of small groups of students, while at the same time enhancing the potential of collaborative learning in small groups, such as increased engagement, teamwork, and communication. The focus of this study was on exploring the quality of the social learning experience of students playing planning poker in small groups while interacting with the robotic facilitator. Moreover, the study aimed to explore whether previous experience in interacting with the robot NAO had any influence on the quality of the learning experience. Hence, the key research questions of the study were: (1) What was the quality of the social learning experience in a group of students playing planning poker? (2) What was the quality of the interaction with the social robot NAO during the game of planning poker? (3) To what extent did the previous experience with the robot influence the quality of the learning experience?

1.3. Planning Poker

Planning poker is a game-based technique for effort estimation in agile teams. It is recommended to practice planning poker with the help of a facilitator, who guides the process and ensures that the method is used properly and effectively. This technique leverages game-based approaches to collaboration in small groups to empower all group members to make decisions and reach a consensus about how much work should be estimated for a given task in a project [9]. Several research studies have shown that planning poker in agile teams can provide more reliable estimates compared with single expert estimations [10,11,12]. The core of planning poker is a group discussion aimed at justifying the individual choices of group members and reaching a consensus as a team. Sharing perspectives and giving all team members a chance to speak in planning poker helps to identify aspects of project tasks that individual estimators could overlook and, hence, provide more accurate and realistic estimations [12]. Therefore, planning poker as a multi-perspective team-based estimation method has been widely used. For example, the 2021 State of Agile Report showed that approx. 58% of surveyed teams applied planning poker [13].

The game-based learning activity “Planning poker with NAO” presented in this paper was designed to provide a larger number of small teams of five to six students with an engaging and interactive social learning experience on campus. Compared with larger groups, which can have some critical drawbacks, such as limited participation, reduced engagement, and limited opportunities for expression and feedback, interaction in small groups tends to promote active learning through the engagement and contributions of all team members. The social robot NAO as a facilitator of the collaborative learning activity is applied to provide teams of students with a similar structure of an exercise, at the same time exploiting the potential of learning in small groups, such as active engagement, participation in group discussions, and contributing to reaching a consensus. The case study “Planning poker with NAO” was driven by the interest in understanding how scalable in-person, collaborative learning experience in small groups can be supported by a social robot and how students perceive the quality of the learning experience enhanced by the robot.

1.4. Related Work

Collaborative learning may be defined as a coordinated, synchronous activity, which results from a continued attempt to construct and maintain a shared understanding of a problem [14]. As elaborated by [15], collaborative learning can be analysed from a variety of perspectives, such as types of interactions, characteristics of a situation, mechanisms for learning, and effects of collaboration. Some of the key mechanisms enabling collaborative learning include (i) equality of interaction partners, who have a common goal and can perform the same actions while working together; (ii) interactivity of peers, who influence the cognitive processes of interaction partners; (iii) synchronicity of communication; (iv) sharing work in the group rather than splitting work and working on sub-tasks individually; and (v) possibility to negotiate, argue, and justify standpoints [15]. From this perspective, the game-based activity of planning poker can be classified as a collaborative learning activity, in which sharing a common goal, working on the same tasks, engaging in synchronous communication, negotiating, justifying, and influencing one another‘s standpoints with the aim of reaching a consensus are the core of the game play.

In general, collaboration has been considered beneficial for learning since it allows bringing together different perspectives, complementary abilities and skills to joint activities aimed at reaching a shared goal and/or solving a shared problem [16,17]. Peer interaction is an important element of collaboration as it provides opportunities for learners to support one another through information sharing and communication [15,16,17]. Moreover, collaborative learning activities may have a positive effect on the sense of self-efficacy and self-regulation [15,16]. Collaboration in the sense of integration and synthesis of ideas and expertise of all group members [16] is a crucial part of planning poker in which group members share perspectives and discuss diverse standpoints in order to reach a consensus.

Robot-Supported Collaborative Learning (RSCL) was proposed as a term by [8] to describe an approach in which a robot supports a collaborative learning activity. Compared with technologies used in Computer-Supported Collaborative Learning (CSCL), social robots are applied not only to mediate interactions but also to act as interaction partners [18]. In RSCL, the teacher and the robot have a shared presence in the classroom [18]. From this perspective, RSCL differs from CSCL in that a social robot is used as a co-present, embodied technology with a social role [18]. Furthermore, research studies, such as [19], showed that robot-supported social interaction in a small group enhances the quality of social presence. Applying a social robot as a facilitator of collaborative learning in small groups can be also helpful for enhancing learners’ attention and motivation [20]. A wide range of approaches used in CSCL to increase the probability of collaboration, such as carefully designing interactions, roles, rules, methods of scaffolding, and monitoring [15], can be applied and/or adapted to designing RSCL.

While applying social robots to support collaborative learning in small groups in higher education settings is still a novel approach, previous studies have explored some important aspects and highlighted the potential of applying social robots such as NAO to facilitate collaborative activities in small groups of students. For example, the study by [8] reported on learning activities in small groups facilitated by the NAO in an undergraduate university course on human-computer interaction. The goal of the study was to compare three conditions of instruction: a human facilitator, a robot facilitator, and facilitation with tablets. In this study, groups of four students performed a group activity, followed the instructions of a facilitator, discussed the material, and answered questions about the interaction. The study revealed a number of benefits and drawbacks of using a robotic facilitator. The benefits of the robotic facilitator included time management as an important element in facilitating group activities, accuracy and focus, which increased the efficiency of repeated activities, and the objectivity of the nonjudgmental aspect of a robotic facilitator compared with a human facilitator [8]. The key drawbacks of the robotic facilitator were related to the robot’s technical limitations in communication and responsiveness [8]. The authors concluded that Robot-Supported Collaborative Learning (RSCL) is a promising novel paradigm for higher education and that social robots can be used as efficient, objective, social facilitators that can support small-group activities in the classroom [8].

The study by [21] explored the application of the NAO robot as a facilitator of brainstorming activities in groups of students. The authors conducted a between-subjects experiment with 54 university students, in which brainstorming facilitated by the NAO robot was compared with human facilitation. The study explored whether the robot condition influenced productivity. The results showed that there was no effect on the productivity of the brainstorming activity facilitated by a social robot compared with a human facilitator [21]. Since robotic facilitation did not result in generating more or fewer ideas, the authors concluded that a robot facilitator can be effectively applied to support brainstorming activities, especially in cases when a human facilitator is unavailable or undesirable [21]. The study by [22] used the NAO robot as a teachable, adaptive robot in higher education settings and investigated whether robotic encouragement of group discussions can enhance social engagement and improve learning. The NAO robot was applied as a novice learner and an active member of a team. The robot was designed to influence group dynamics by encouraging and balancing social engagement in pairs of adult participants. In a study with 68 participants, pairs of learners were encouraged to take on the teaching role. The study showed that the robot’s adaptive encouragement resulted in more dialogues between pairs of learners and increased both task and social engagement in the group [22].

Nevertheless, possible drawbacks of applying a social robot as a facilitator of collaborative learning activities need to be taken into consideration. These include repetitive and predictive behaviour, which may have a negative impact on interest and motivation [23]. Further drawbacks include issues related to the current technical limitations of social robots, such as limited speech recognition capabilities, and issues related to the limited adaptability of social robots to situations and learners [8]. Moreover, as indicated by a number of studies, positive effects reported by participants interacting with social robots may result from the experience of novelty, which may fade away with time [8].

1.5. Contribution

As pointed out by a number of researchers, there are still relatively few studies on human–robot interaction in collaborative learning situations [19]. There is also little research about collaborative learning activities in small groups supported by social, anthropomorphic robots, especially in the complex context of a real classroom environment [18]. This paper aims to fill some of these research gaps by focusing on applying a social robot to facilitate a collaborative learning activity in small groups of students in a real classroom environment. The contributions of this paper include the following:

• A game-based, robot-supported application “Planning poker with NAO” for collaborative learning of agile estimation in small groups, using a combined macro and micro script [24] based on the model of self-regulated learning by Zimmerman [25], which can be used to enable a scalable in-person, robot-supported collaborative learning experience for a larger number of small groups;
• A shortened 20-item scale with high internal consistency, which can be used for the evaluation of the quality of social learning experiences supported by social robots;
• Quantitative and qualitative results from the evaluation with 46 adult participants showing that planning poker with the robot NAO as a robot-supported collaborative learning activity in a team can provide university students with an enjoyable and, at the same time, effective social learning experience that supports understanding and learning through play in a team.
• Comparison of the learning experience during a robot-supported collaborative learning activity of students with and without prior experience in interacting with a robot.

Following this introduction (Section 1), the next sections of this paper present materials and methods used to design, implement, and evaluate the study (Section 2), quantitative and qualitative evaluation results related to the key three research questions (Section 3), and a discussion with recommendations for future research (Section 4).

2. Materials and Methods

2.1. Scenario-Based Design

The design of the application “Planning poker with NAO” was guided by the methodology of scenario-based design (SBD), which is a design method focused on analysing current challenges, designing a problem-solving scenario, and evaluating the solution [26]. The SBD method applied to designing the application “Planning poker with NAO” included three main stages: (A) analysis of current practices and challenges; (B) design of an integrated activity, information, and interaction scenario with a narrative script; and (C) evaluation as part of iterative prototyping (Figure 1). The scenarios developed at different stages of the iterative process of prototyping and testing were used as blueprints for the programming of the robot.

2.1.1. Analysis

The analysis phase focused on analysing current practices and typical challenges in learning/teaching agile estimation in university courses. The analysis resulted in a problem scenario that included the analysis of stakeholders, claims about the current practices and a list of challenges, such as (i) the tendency to link estimation of effort to hours and days of work instead of rough and relative estimation of effort of tasks compared with each other, (ii) students’ lack of previous experience in estimating effort of project tasks, and (iii) beginner understanding of agile practices due to little previous exposure and practical experience. Furthermore, the problem scenario included a description of typical teaching practice of agile estimation and different methods applied to playing planning poker in practice, such as (i) the diversity of approaches in formulating items to be estimated (e.g., features, user stories, and tasks), and (ii) the diversity of scales used in agile estimation, e.g., linear scale (0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10), Fibonacci scale (0, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89), and modified Fibonacci scale (0, 1, 2, 3, 5, 8, 13, 21, 40, 80, 100).

2.1.2. Design

The design phase focused on the design of an integrated activity and information and interaction scenario based on the practices and challenges outlined in the problem scenario. The integrated scenario was designed as a novel approach to providing a scalable collaborative learning experience for a larger number of small groups of students. The integrated activity and information and interaction scenario was created as a narrative script of a connected flow of activities, information, and interactions, including actions and responses of both robotic and human interaction partners.

The narrative script was designed as an integrated macro- and micro-script and included both the description of the orchestration of the planning poker activity on the macro level and detailed information about the enactment of the activity, such as prompts and encouragements, on the micro level [24]. The script was based on the cyclical model of self-regulated learning (SRL) and divided into three main phases: introduction, play, and reflection, which correspond to the forethought, performance, and self-reflection phases in the SRL model by Zimmerman [25]. On the macro level, the script includes a step-by-step description of a sequence of events during the play of planning poker with the NAO robot, starting with a welcome and ending with a reflection and farewell. On the micro level, the script includes an introduction to planning poker with information about the principles of agile estimation and an explanation of how planning poker will be played in a small group, promoting simple warm-up exercises to make sure that students understand how to use planning poker cards and the modified Fibonacci scale, explanation and prompts related to how to interact with the robot (e.g., using speech and foot bumpers as tactile sensors), explanation of all the tasks to be estimated during the play, and a task-by-task prompts and encouragements for collaboration in the group during shared estimation of each task.

2.1.3. Evaluation

The evaluation phase focused on the evaluation of different prototypes of the planning poker activity, which were developed iteratively in a number of stages of scenario development and refinement. With the NAO robot taking the role of the facilitator of the collaborative activity of planning poker, most of the prototyping was focused on ensuring smooth dialogue and interaction. Earlier tests with the NAO robot showed that reliable pronunciation and a moderate speaking rate are crucial for effective interaction with students [27,28]. Therefore, dialogue and interaction design were gradually improved based on feedback from these iterative phases:

• The first iteration focused on creating a first prototype based on the draft of the problem scenario from the analysis stage, which was derived from hands-on experiences without the possibility to rely on any prior research in the application of a robotic facilitator of planning poker. In order to evaluate the first prototype, a pretest with a small group of three students was conducted to obtain early feedback related to the overall design of the game and the specific aspects such as the robot’s speech quality and the comprehensiveness of the explanation. The evaluation used a Likert-scale-based grading system for different rating criteria, such as comprehensibility of language and information and the robot’s speech quality. Based on the results from this first pretest, the prototype was iterated further.
• The second iteration focused on improving the speech quality, the explanation of the planning poker, and the explanation of interaction during agile estimation in the small group. The speaking rate of the robot was decreased and some ambiguities in the explanations were addressed. The information part was better tuned in to students without prior experience in planning poker and agile estimation. The improved version was tested with 15 undergraduate students.
• The third iteration was based on the evaluation results from the previous test and incorporated a number of changes, such as (i) an improved dialog design with more questions and presentation of information in smaller chunks; (ii) further improvement of the explanation by shortening the texts spoken by the robot; (iii) reducing the number of tasks to be estimated from eight to six; (iv) reformulating the tasks to be estimated by adding more information about each task; and (v) introducing more variation in the responses of the robot to make the interaction less predictable and more exciting. The new version was tested with 14 graduate students. Following SBD by [24], the improved design elaborated in more detail how students will perform estimation in teams and how the robot will support group collaboration.
• The fourth iteration resulted in the final version of the game “Planning poker with NAO”, which was applied in the study presented in this paper. The narrative script was transformed into the robot application using the Choregraphe software for NAO (Version 2.8.6) and additional Python code for the creation of a PDF document with the results of each estimation.

2.2. Planning Poker with NAO

The final version of “Planning poker with NAO” is composed of five main parts, i.e., “Introduction”, “Explanation”, “Planning Poker”, “Create protocol” and “Ending”. After the start of the application, the NAO robot introduces students to the planning poker, agile estimation, and interaction with the robot by speech and foot bumpers. The actual play of planning poker begins with the robot presenting all tasks to be estimated to the team of students. In the final version, there are five items (tasks) to be estimated, which are also defined in more detail to ensure a better joint understanding of each item by the group. For example, the item/task “Take out paper waste” from an earlier iteration was reformulated as “Take out paper waste. Here is some information about this task: You are on the fifth floor, so you have to go downstairs to bring the paper waste to the bin on the ground floor. There are five bins with paper waste altogether.”

The NAO robot reads the first task (or the first item in the backlog, which is a list of tasks) and encourages students to perform the estimation by choosing one of the planning poker cards. Next, students reveal the card value to other players by turning the card face up; the robot encourages students to discuss and justify their choices with the aim of reaching a consensus. Following the approach proposed by [22], the robot is programmed to provide students with encouraging utterances to enhance group discussions and reach a consensus. For example, “Ok, so now justify your estimation and discuss which number you will assign to this task” is used as an encouraging utterance by the NAO robot. Furthermore, for the collaboration to be successful, groups must have a structure for how activities are conducted and coordinated [16]. This is supported by the script that helps to structure the collaborative learning process by specifying roles and defining a sequence of activities and interactions [17]. The approach applied in the design of the planning poker with NAO makes use of scripted interaction to provide a clear structure for the activity. The script includes group-based activities that are used to structure the flow of interactions and help students engage in building a shared understanding of relative estimation in agile teams. Once the consensus in the team has been reached, one of the team members reports the final result to NAO and the robot saves the value in the PDF report. This process is repeated until all tasks are estimated and a consensus is reached. Finally, the robot wraps up the activity, asks students to reflect on what they have learned, and reviews key learnings (Figure 2).

The final version of the game “Planning poker with NAO” was published in an Open Source repository on GitHub (cf. Supplementary Materials).

2.3. Study Settings

The RSCL activity of “Planning poker with NAO” was applied and evaluated in on-campus classes with students enrolled in the course on agile project management for the undergraduate study program on digital business. The course on agile project management consists of weekly seminars taught by the principal instructor and practice-based classes taught by an assistant instructor, who works as an agile coach in companies. Students in the agile project management course learn about agile project methods such as Scrum, Kanban, Scaled Agile, and Lean-Agile through short lectures, interactive learning materials, and hands-on activities both online and in class. Students work in agile teams on selected projects. Planning poker is introduced as one of the methods of agile estimation next to other methods, such as story points and T-shirt sizing.

2.4. Study Participants

Forty-eight students participated in planning poker sessions facilitated by the NAO robot on two different days. All participants were students in the undergraduate study program on digital business. The participants were enrolled in the course on agile project management in the third semester of their studies. Students could choose and book a time slot of 20 min per planning poker session using a scheduler in the learning management system Moodle. Each time slot of 20 min was available for max. 6 participants. Students did not have to reserve a spot in a predefined team; most teams were created ad hoc by booking a time slot. Planning poker activity was part of the coursework and students received bonus points for participation. The concluding evaluation comprised an anonymous online survey and a follow-up discussion with each group.

Altogether 46 participants (96%), 17 females (37%) and 29 males (63%), filled in the online survey as part of the evaluation. The majority of the participants were between 18 and 24 years old (n 32, 69.57%). The participants had a predominantly Western European background (n 28, 61%). Forty-one % of participants interacted with the NAO robot before and 87% played planning poker before. These results can be explained in this way: both the NAO robot and planning poker were introduced in the course earlier during the semester. Since attendance in the course was not obligatory, different students had different experiences with both the robot and planning poker.

Although students who participated in the planning poker session with NAO obtained bonus points, most participants stated that the key reason why they participated in planning poker was that they were curious about the robot (n 30, 65%). Only nine students (19.57%) stated they participated because of the grade/bonus points.

2.5. Materials

The materials used in the study included the NAO robot, printed lists of tasks to be estimated, and one set of planning poker cards per person. The cards used in the game represented the modified Fibonacci scale. The materials are shown in Figure 3.

2.6. Procedure

Planning poker with NAO was played by a total of ten teams, each composed of five to six students. The participants in each planning poker session were students, the principal instructor, the programmer of the robot, and NAO itself. The role of the robot was to introduce planning poker and to facilitate game play and collaboration in the group. The role of the students was to act as an agile team and estimate effort for each item/task using planning poker cards and justifying their choice during group discussions. The role of the instructor was to welcome students, set the scene for each session, provide any assistance needed by students on demand, and administer the online evaluation survey. The role of the programmer was to explain to students how to interact with NAO, monitor the flow of the game in the software, and assist with technical issues.

2.7. Research Questions

The focus of the study was exploring to what extent a social robot NAO in its role as a facilitator or a game-based learning activity can enhance the quality of an in-person, collaborative social learning experience. The key research questions of the study were:

• RQ1: What was the quality of the social learning experience in a group of students playing planning poker?
• RQ2: What was the quality of the interaction with the robot NAO during planning poker?
• RQ3: To what extent did the previous experience with the NAO robot influence the quality of the robot-supported collaborative learning experience?

2.8. Research Instruments

The online survey included a range of questions related to the learning experience during the game “Planning poker with NAO“. The first part of the survey included items related to the sociodemographics of the sample, i.e., gender, age, and cultural background, and a question related to the previous experience—i.e., Have you ever interacted with the NAO robot before? (Yes/No).

2.8.1. Shortened Learning Experience Scale

The primary research scale applied in the survey was the shortened learning experience scale, which was derived from the scale proposed by [29]. The learning experience scale by [29] is a scale for measuring factors that affect the learning experience when playing serious games and is considered an adequate instrument for capturing the quality of the learning experience of a game-based activity of planning poker. Planning poker fulfills a similar goal as serious games, as it is applied for non-entertainment purposes and attempts to create an instructionally sound and relevant learning experience [30]. The scale by [29] is based on an extensive literature review and includes items related to (a) technology acceptance, such as perceived usefulness, perceived ease of use, and perceived usability; (b) games, such as perceived sound and visual effects’ adequacy, perceived narratives’ adequacy, immersion, enjoyment, perceived realism; and (c) learning, such as perceived goal’s clarity, perceived adequacy of the learning material, perceived adequacy of feedback, perceived competence, motivation, perceived relevance to personal interests, and perceived knowledge improvement.

For the purpose of the study, the original learning experience scale by [29] was shortened and adapted to the RSCL scenario. For example, items related to measuring the perceived realism of virtual objects and perceived adequacy of storytelling, which are relevant for video games, were considered as not adequate for the RSCL scenario and removed. The shortening of the learning experience scale also aimed at reducing some well-known problems of lengthy surveys, such as respondent fatigue, decreased response rates, increased likelihood of missing data, and data quality, which have all been well-documented in the literature, e.g., [31]. The original scale by [29] included 76 items, while the shortened scale comprised 20 items. All items were rated on a Likert-type scale from 1 = “strongly disagree” to 5 = “strongly agree”, which corresponds to [29].

The inclusion of negatively formulated items in surveys is important for both more nuanced insights into the experience of study participants and an increased scale validity [32]. Therefore, attention was paid to selecting both positive and negative items when creating the shortened version of the scale, which now includes four negatively formulated items (items 4, 6, 13, 20). Further adaptations to the original scale included changes to the wording of selected items, which were adjusted to fit the RSCL scenario of planning poker with NAO.

Table 1. Shortened learning experience scale with 20 items applied in the evaluation study.

Factor	Item	Statement
Immersion	1	I was deeply concentrated while playing planning poker in my team.
	2	I forgot about the time passing while playing planning poker in my team.
Enjoyment	3	Playing planning poker in a team was fun.
	4	I felt bored while playing planning poker in a team.
	5	I really enjoyed playing planning poker in a team.
	6	I felt frustrated while playing planning poker in a team.
Perceived usefulness	7	Planning poker in a team is an easier way to learn than usual teaching.
	8	Planning poker in a team makes learning more interesting.
Perceived knowledge improvement	9	I feel that the game planning poker in a team increased my knowledge.
	10	I understood the idea of agile estimation by playing planning poker in a team.
	11	I will definitely try to apply what I learned by playing planning poker.
Perceived ease of use	12	Planning poker in a team was easy to learn.
	13	Planning poker in a team was unnecessarily complex.
Perceived competence	14	I felt competent when playing planning poker in my team.
	15	I felt successful when playing planning poker in my team.
	16	I felt a sense of control when playing planning poker in my team.
Motivation	17	Playing planning poker in a team motivated me to learn.
	18	Playing planning poker in a team stimulated my curiosity.
Perceived relevance	19	Planning poker was relevant to my interests.
	20	Planning poker was not relevant to me because I already knew most of it.

presents the shortened learning experience scale.

2.8.2. Interaction Experience with NAO

In addition to the shortened learning experience scale, which focuses on the overall learning experience of planning poker in a small group of students, the online evaluation survey included further items and questions, which were designed to capture the interaction experience of students with the NAO robot as the facilitator of planning poker. Eight items (21–28) were designed and rated by study participants on a Likert-type scale from 1 = ”strongly disagree” to 5 = ”strongly agree” (

Table 2. Survey items related to the robot-supported learning experience applied in the evaluation study.

Item (No.)	Item (Statement)
21	The interaction with NAO was helpful to understand how to play planning poker.
22	The interaction with NAO during planning poker was interesting.
23	The interaction with NAO during planning poker was exciting.
24	The interaction with NAO during planning poker was motivating.
25	The interaction with NAO during planning poker was frustrating.
26	The interaction with NAO was disappointing.
27	I feel confident in agile estimation after playing planning poker with NAO.
28	I would like to play planning poker with NAO again.

). In addition to these eight items, an open question was asked at the end of the survey: “What are your recommendations for improving the planning poker in a team with a robot facilitator like NAO? Please share your thoughts and ideas!” With the help of this question, some qualitative information related to interactions with the robot was captured.

Most of the items presented in Table 2 were already applied in a previous study with the NAO robot as the facilitator of a daily scrum meeting described in [27].

3. Results

The analysis of the data was conducted using IBM SPSS Statistics 29. The methods included both descriptive and inferential statistics. The results are summarised below in relation to the three key research questions of the study.

3.1. RQ1: What Was the Quality of the Social Learning Experience in a Group of Students Playing Planning Poker?

In order to answer the first research question, descriptive statistics were computed for all 20 items of the shortened learning experience scale. In the second step, the four negatively formulated items were recoded and the internal consistency of the scale computed using reliability statistics. The measured consistency of the new scale was high with α = 0.897 (Cronbach’s Alpha) and α = 0.904 (Cronbach’s Alpha on Standardised Items). The total score of the scale was M = 3.325 (Min. 1.891; Max. 4.152). The item statistics are summarised in

Table 3. Item statistics of the shortened Learning Experience Scale, n = 46. Rating on a five-point scale: “Strongly Agree” (5), “Agree” (4); “Neutral” (3); “Disagree” (2); ”Strongly Disagree” (1).

Item	Mean	Std. Deviation
1. I was deeply concentrated while playing planning poker in my team.	3.57	0.981
2. I forgot about time passing while playing planning poker in my team.	3.43	1.186
3. Playing planning poker in a team was fun.	3.87	1.024
4. I felt bored while playing planning poker in a team. *	1.93	1.063
5. I really enjoyed playing planning poker in a team.	3.48	0.960
6. I felt frustrated while playing planning poker in a team. *	1.89	1.159
7. Planning poker in a team is an easier way to learn than usual teaching.	3.78	0.964
8. Planning poker in a team makes learning more interesting.	4.02	0.774
9. I feel that the game planning poker in a team increased my knowledge.	3.46	0.936
10. I understood agile estimation by playing planning poker in a team.	3.91	1.092
11. I will definitely try to apply what I learned by playing planning poker.	3.70	1.072
12. Planning poker in a team was easy to learn.	4.15	1.095
13. Planning poker in a team was unnecessarily complex. *	1.91	1.071
14. I felt competent when playing planning poker in my team.	3.41	1.066
15. I felt successful when playing planning poker in my team.	3.65	1.016
16. I felt a sense of control when playing planning poker in my team.	3.46	0.959
17. Playing planning poker in a team motivated me to learn.	3.39	1.085
18. Playing planning poker in a team stimulated my curiosity.	3.28	1.129
19. Planning poker was relevant to my interests.	3.37	1.040
20. Planning poker was not relevant to me because I already knew most of it. *	2.83	1.371

* Negatively formulated items.

below.

The results show that the study participants rated their social learning experience during planning poker in the small group in a positive way. All values of positively formulated items were above the ”neutral” score (3). The highest values were reached for item 12 “Planning poker in a team was easy to learn” (M = 4.15) and item 8 “Planning poker in a team makes learning more interesting” (M = 4.02). Next items with high ratings were item 10 “I understood agile estimation by playing planning poker in a team” (M = 3.91), item 3 “Playing planning poker in a team was fun” (M = 3.87), item 8 “Planning poker in a team is an easier way to learn than usual teaching” (M = 3.78), and item 11 “I will definitely try to apply what I learned by playing planning poker” (M = 3.70). Items with the lowest ratings were negatively formulated items, i.e., item 6 “I felt frustrated while playing planning poker in a team” (M = 1.89), item 13 “Planning poker in a team was unnecessarily complex” (M = 1.91), and item 4 “I felt bored while playing planning poker in a team” (M = 1.93). These results indicate that the social learning experience of planning poker in a group was interesting, enjoyable, and valuable for attaining learning outcomes, especially understanding agile estimation and attaining willingness to apply what was learned in the future. However, some results also indicated that the design of planning poker as a collaborative learning activity could be further improved, especially when it comes to stimulating curiosity (item 18, M = 3.28) and enhancing motivation to learn (item 17, M = 3.39). Both items were only slightly above the neutral score (3), indicating a possible direction for further research and development.

3.2. RQ2: What Was the Quality of the Interaction with the Social Robot NAO during the Game of Planning Poker?

In order to answer the second research question, descriptive statistics were computed for all eight items related to the quality of interaction with the NAO robot during the play of planning poker. In the second step, the two negatively formulated items were recoded and the internal consistency of the scale computed using reliability statistics. The measured consistency of the new scale was acceptable with α = 0.796 (Cronbach’s Alpha) and α = 0.784 (Cronbach’s Alpha on Standardised Items). The total score of the scale was M = 3.973 (Min. 3.326; Max. 4.283). The item statistics are summarised in

Table 4. Item statistics related to the quality of interaction with the robot NAO, n = 46. Rating on a five-point scale: “Strongly Agree” (5), “Agree” (4); “Neutral” (3); “Disagree” (2); ”Strongly Disagree” (1).

Item	Mean	Std. Deviation
21. The interaction with NAO was helpful to understand planning poker.	4.07	0.975
22. The interaction with NAO during planning poker was interesting.	4.28	0.655
23. The interaction with NAO during planning poker was exciting.	4.04	0.842
24. The interaction with NAO during planning poker was motivating.	3.63	1.123
25. The interaction with NAO during planning poker was frustrating. *	1.87	1.108
26. The interaction with NAO was disappointing. *	1.83	1.039
27. I feel confident in agile estimation after playing planning poker with NAO.	4.13	0.653
28. I would like to play planning poker with NAO again.	3.33	1.076

* Negatively formulated items.

below.

The results show that the study participants rated the quality of the interaction experience with NAO in a positive way. All values of positively formulated items were above the neutral score (3). The highest values were reached for item 22 “The interaction with NAO during planning poker was interesting” (M = 4.28), item 27 “I feel confident in agile estimation after playing planning poker with NAO” (M = 4.13), item 21 ”The interaction with NAO was helpful to understand planning poker” (M = 4.07), and item 23 “The interaction with NAO during planning poker was exciting” (M = 4.03). The items with the lowest ratings were negatively formulated items, i.e., item 25 “The interaction with NAO during planning poker was frustrating” (M = 1.87), and item 26 “The interaction with NAO was disappointing” (M = 1.83). These results indicate that students experienced the interaction with the robotic facilitator not only as interesting and exciting but also as helpful for understanding planning poker and gaining confidence in agile estimation. However, some results also indicate that the design of the interaction with the robot could be further improved, especially when it comes to enhancing motivation (item 24, M = 3.63) and overcoming the novelty effect (item 28, M = 3.33).

The analysis of the qualitative data from the open question, “What are your recommendations for improving the planning poker in a team with a robot facilitator like NAO? Please share your thoughts and ideas!” revealed some further important directions for improvement in the design of human–robot interaction. Although many participants confirmed that they fully enjoyed interacting with NAO (e.g., “It was really fun” or “I had a lot of fun”), some students criticised lengthy information chunks provided by the robot (e.g., “At some points he talks for too long” or “Make it short and clear”), and issues related to speech recognition (e.g.,“Other than the speech recognition the experience was actually really pleasant!” or “Use a better voice detector”). Some students also wished for more interaction with NAO for the enhancement of collaboration in the group, e.g., “More interaction and evaluation tips from NAO”.

3.3. RQ3: To What Extent Did the Previous Experience with the NAO Robot Influence the Quality of the Robot-Supported Collaborative Experience?

In order to answer the third question, descriptive statistics were computed for previous experience with NAO, which was a nominal variable with two categories: yes/no. Out of 46 respondents, 19 participants (41%) reported having previous experience with NAO, while 27 participants (59%) reported having no previous experience. Next, the Shapiro–Wilk test was used to assess the normality of data distribution in the shortened learning experience scale and in the NAO interaction experience scale. Since dependent variables were not normally distributed, Mann–Whitney U Test was used as a non-parametric alternative to ANOVA. The results of the Mann–Whitney U Test showed that there were only two significant differences between the two groups, i.e., students “with prior experience with NAO” and students “without prior experience with NAO”. The first statistically significant difference was related to the level of confidence in agile estimation (item 27 ”I feel confident in agile estimation after playing planning poker with NAO”), which was statistically significantly higher among students without prior experience with NAO compared with students with prior experience (U = 172, p = 0.035). The comparison of means showed that students “without prior experience with NAO” (n 27) felt more confident (M = 4.30) compared with students “with prior experience with NAO” (n 19, M = 3.89). The second statistically significant difference between the two groups was related to item 4 “I felt bored while playing planning poker in a team”, which was statistically significantly higher among students with prior experience with NAO compared with students without prior experience (U = 173, p = 0.047). The mean values for students without prior experience (n 27) were lower (M = 1.74) compared with the group of students with prior experience (n 19, M = 2.21).

These two statistically significant results can be interpreted in the context of the novelty experience. It is possible that students who already had some prior experience with the NAO robot, possibly from the previous iterations, already gained some knowledge about planning poker (confidence, item 27) and knew what to expect from the NAO robot (feeling bored, item 4), hence the novelty effect was reduced. Students with prior experience could have felt less confident as possible gain in knowledge was lower compared with students without prior experience; at the same time students with prior experience felt more bored compared with students for whom the robot was completely new and who might have experienced the activity as more exciting.

The comparison of mean values for all 28 items from the survey provided some further evidence. While 18 out of 22 (81%) positively formulated items related to the quality of the learning experience both in the group and with NAO had higher mean values among students without prior experience with NAO; it was the reverse for all 6 negatively formulated items. The exceptions among these 22 positively formulated items were 4 items for which the mean values were higher in the group of students with prior experience with NAO—i.e., (1) item 8 “Planning poker in a team makes learning more interesting”, (2) item 9 “I feel that the game planning poker in a team increased my knowledge”, (3) item 18 “Playing planning poker in a team stimulated my curiosity”, and (4) item 21 “The interaction with NAO was helpful to understand planning poker”. Figure 4 shows the mean plots for these four variables.

These results may be interpreted in the following way: students who already interacted with NAO before felt that the learning experience was slightly less interesting and exciting (reduced novelty effect), while they rated their learning outcomes higher than students without the prior experience (increased familiarity effect). Although these differences were not statistically significant, they point to some trends, which could be explored in further studies.

4. Discussion

This paper proposed a novel approach to enhancing scalable social learning experiences in a larger number of small of students by applying a social robot to facilitate a game-based, collaborative learning activity. The collaborative learning activity of “Planning Poker with NAO“ was designed and deployed with the aim of increasing interaction, communication, and negotiation in small groups. The broader rationale for the case study “Planning poker with NAO” presented in this paper was derived from the need to provide students with engaging social learning experiences on campus in view of the shift to online learning in post-pandemic times and to create opportunities for students to experience social robots, which are on the rise both as educational technologies and as technologies used at the workplace [9]. The case study “Planning poker with NAO” explored the quality of the learning experiences in a robot-supported collaborative learning activity facilitated by robot NAO.

4.1. Summary

The evaluation study with 46 undergraduate students examined the quality of the social learning experience in groups and the quality of the interaction experience with the NAO robot. Furthermore, differences in the quality of learning and interaction experience were explored by comparing results for students with and without prior experience in interacting with the NAO robot. The results showed that students experienced the RSCL activity of planning poker in a positive way, both on the affective and cognitive level. First, the analysis revealed that all participants, independent of their previous experience with the robot, enjoyed the robot-supported collaborative learning experience. Second, although there were only two statistically significant differences among students with and without prior experience with NAO, i.e., students without experience with NAO (n 27) felt more confident in agile estimation, while students with NAO experience (n 19) felt more bored during the interaction with NAO, there were some further interesting, but not significant, differences in the mean values of different variables. Some of these differences were related to higher mean values among students without prior experience with the robot for the variables related to affective aspects such as “exciting”, which may be contributed to the novelty effect of a first-time interaction with a new technology. Third, the group of students with prior NAO experience showed higher mean values related to cognitive outcomes such as“increased my knowledge”, “stimulated my curiosity”, and “was helpful to understand planning poker”. Although these differences were not statistically significant, they indicated directions for further research, possibly in longitudinal studies and with larger samples. It could be hypothesised that once the novelty experience related to the application of robots as a new technology fades away over time, the attention of learners can shift from the technology to the actual process and content of collaborative learning. Previous studies have already addressed possible benefits from the familiarisation effect and provided empirical evidence for the decrease in novelty experience due to the increase in the familiarisation effect over time [33].

Overall, from the perspective of the human facilitation of collaborative learning activities in small groups, the case study showed that a robot-supported collaborative learning activity like “Planning poker with NAO” offers useful support for a human teacher aiming to provide a large number of students with an engaging learning experience in small groups. Since the same standardised script can be repeatedly executed by the social robot without any fatigue or decreased focus, the potential of applying social robots can be seen in enhancing scalability, while benefiting a large number of students from the interaction experience in small groups. The application of social robots may help to exploit the potential of learning in small groups, such as increased engagement, collaboration, communication, and accountability. Furthermore, with a social robot orchestrating a collaborative learning activity, there is more time for the human facilitator to provide individualised or personalised support and feedback to the group of learners. The resulting co-facilitation approach of a robotic and human facilitator can benefit students through access to both standardised (robot) and personalised (human) support.

Considering the positive results of the study presented in this paper, combined with the positive results of previous studies in similar areas, it is possible to envisage robot-supported collaborative learning (RSCL) as “a new paradigm for higher education, in which the group facilitation can be supported by social robots” [8]. It is important to emphasise that applying a robotic facilitator should fulfill a supportive role for the human teacher. While social robots may be helpful in providing a standardised structure, human facilitators are needed to provide personalised support, especially in relation to understanding, and responding to emotional and social issues [8,9].

4.2. Limitations and Future Research

The limitations of this study presented in this paper include most notably a small sample size and a cross-sectional design. Therefore the study results must not be generalised. Next, the assessment of the quality of the learning experience and the quality of interaction with the social robot NAO as a facilitator of the planning poker activity was based primarily on the quantitative data. In order to get a deeper understanding of the possible effects of applying a social robot as a facilitator of collaborative learning activities, future studies should include more qualitative data (e.g., through in-depth interviews), design longer interventions (longitudinal studies), and aim for larger samples. Furthermore, the evaluation study presented in this paper did not employ an experimental design and did not compare different conditions, such as human and robotic facilitators. Future studies should assess the benefits and drawbacks of social robots compared with other types of human and nonhuman facilitators. Finally, another limitation of the study was the lack of pre- and post-testing, which could have been valuable for capturing knowledge and skills of students before and after playing the planning poker with NAO. Future studies should address these limitations. Further possible directions for future research include taking the diversity of participants and of facilitators’ tasks into consideration when designing the facilitation of collaborative learning with social robots. Furthermore, different styles of facilitation could be explored in future studies.