A Survey of Behavioral Models for Social Robots

Abstract

The cooperation between humans and robots is becoming increasingly important in our society. Consequently, there is a growing interest in the development of models that can enhance and enrich the interaction between humans and robots. A key challenge in the Human-Robot Interaction (HRI) field is to provide robots with cognitive and affective capabilities, by developing architectures that let them establish empathetic relationships with users. Over the last several years, multiple models were proposed to face this open-challenge. This work provides a survey of the most relevant attempts/works. In details, it offers an overview of the architectures present in literature focusing on three specific aspects of HRI: the development of adaptive behavioral models, the design of cognitive architectures, and the ability to establish empathy with the user. The research was conducted within two databases: Scopus and Web of Science. Accurate exclusion criteria were applied to screen the 4916 articles found. At the end, 56 articles were selected. For each work, an evaluation of the model is made. Pros and cons of each work are detailed by analyzing the aspects that can be improved to establish an enjoyable interaction between robots and users.

1. Introduction

Social Robotics is commonly defined as the research field dedicated to the socially skillful robots [1]. The main ability of social robots is to establish a natural interaction with humans. The Human-Robot Interaction (HRI) field of study tries to shape the interactions between one or more humans and one or more robots. Over the latest several years, there is an increasing interest in HRI due to the increasing usage of robots not only in industrial fields, but also in other areas as schools [2], homes [3], hospitals [4], and rehabilitation centers [5].

Consequently, in the near future, robots will concretely share environments with human beings to actively collaborate with them in specific daily tasks. The presence of a robot, in fact, could be a useful support during the management of daily activities [6,7], the promotion of social inclusion [8,9], and the suggestion of healthy activities [10,11]. Particularly, recent literature findings underline that robots could help users in their daily life by bringing them objects that they need ( i.e., a bottle of water, a specific drug ) [12], which helps them in dressing tasks [13,14] or in getting in contact with their families or authorities in dangerous situations [9]. An easy and continuous connection with other people (i.e., relatives, friends, or doctors), could promote social inclusion of people with disabilities or elderly people and increase the quality of their life [15]. Therefore, in this context, there is a growing necessity for developing behavioral models for social robots to have a high quality interaction and level of acceptability in providing useful and efficient services [16,17]. Remarkably, how people accept, perceive, interact, and cooperate with this intelligent machine in their life is still somewhat unknown. However, researchers with different backgrounds are trying to meet this challenge [18].

First, to achieve fluent and effective human-like communication, robots must seamlessly integrate the necessary social behaviors for a given situation using a large number of patterned behaviors that people employ to achieve particular communicative goals. Furthermore, robots should be endowed with the capability to understand feelings, intentions, and beliefs of the user, which are not only directly expressed by the user, but that are also shaped by bodily cues (i.e., gaze, posture, facial expressions) and vocal cues (i.e., vocal tones and expressions) [19]. The non-verbal immediacy, which characterizes communications between humans, should be conveyed in Human-Robot Interaction (HRI). Moreover, the ability to replicate human non-verbal immediacy in artificial agents is twofold. On one side, it allows the detection of emotional and cognitive state of the user, which is useful to develop proactive robots. On the other side, it allows us to shape the behavior of the robot in order to encode behavior capabilities in the interaction as those of humans. The latter case leads to the possibility to automatically generate new robotic behaviors that the robot learns directly from the user.

The first attempts to solve this challenge have been performed by developing intelligent systems able to detect user’s emotions [20] and by identifying the key factors that should be adjusted to make the interaction smoother (i.e., interpersonal distance, mental state, user’s feedback, and user’s profile) [21]. More advanced steps should be performed so that robots are endowed with cognitive and affective capabilities that could provide them with tools to establish empathetic relationships with users and to gain social cognitive mechanisms that are necessary to be perceived as a teammate [22,23].

Second, it is important to remark that the robot’s ability to establish empathic relationships has a key role in HRI since it indicates the degree of perceived bodily and psychological closeness between people. Over the last several years, researchers put a lot of effort in understanding how psychology and cognitive neuroscience could be integrated in the design process of artificial cognitive architectures to achieve this target. The field of brain-inspired technologies has become a hot topic in the last several years.

In this context, this paper aims to analyze the current state of the art of behavioral models to find barriers and limitations to provide guidelines for future research studies in this area. Particularly, two databases (namely Scopus and Web of Science) were analyzed to retrieve papers linked with cognitive robotics architecture and model robot empathy, affordance, facial expression, cultural adaptation, and the social robot. In effect, this survey expresses this growing interest and the need to support the research studies in this field and to organize the large amount of work, which is loosely related to the topic underling the scientific challenges. The main issues of this area is related to the fact that several models are too often described from a theoretical point of view without being tested on a real robot and the ones that are tested on a real robot are often tried in a single environment with people belonging to a specific culture [24]. Specifically, researchers are working on the development of cognitive architectures approaching a fully cognitive state, embedding mechanisms of perception, adaptation, and motivation [25]. Particularly, from the analysis of the state of the art, the papers of this survey are grouped according to three main application areas: cognitive architectures, behavioral adaptation, and empathy.

• Cognitive architectures—This term refers to research works where both abstract models of cognition and software instantiations of such models, employed in the field of artificial intelligence, are described [26]. Cognitive architectures have the fundamental role to enable artificial intelligence in robotic agents, in order to exhibit intelligent behaviors.
• Behavioral adaptation—Behavioral adaptation is defined as “learning new tasks and to adapt to changes in environmental conditions, or to failures in sensors and/or actuators” [27]. Thus, the papers included in this group describe robot’s social abilities enhanced by the robot’s capability of adapting its behavior to the user’s need and habits [28].
• Empathy—Empathy is defined as “The act of perceiving, understanding, experiencing, and responding to the emotional state and ideas of another person” [29]. In human-human relationships, this term explains the capacity to take the role of the other to adopt alternative perspectives [28]. Works clustered in this category present a particular emphasis on the attempts to reproduce this ability in robotic agents to establish an empathetic connection with the user, which improves Human-Robot Interaction (HRI). Empathy is a sub-category of the behavioral adaptation. However, we decide to make separate categories to be aligned with some recent papers [17,30,31].

In this review, several models and architectures used in social robots are presented to evaluate how these attempts fare in achieving an efficient robot-human interaction. A comparison with works presenting experimentation to demonstrate the persuasiveness of robots is also provided to highlight limitations and future trends. In details, the paper is organized as follows: in Section 2, the research methodology for the review is explained. In Section 3 and Section 4, the results and the discussions regarding the papers are shown. In Section 5, a summary of the review and its conclusions are presented.

2. Materials and Methods

This section presents the methodology used in the paper to select the most appropriate recent developments as published in the literature, covering the topics of behavioral models for robots.

Study Selection Procedures

This paper reviews empirical studies published between 2010 and 2018 since most of the advances in this area have occurred within that timeframe. A bibliography was developed upon research in Scopus and Web of Science electronic databases. Reference lists of included articles and significant review papers were examined to include other relevant studies. The search queries contained the following terms and were summarized in

Table 1. List of keywords used in this review work.

Research Keywords
(cult*) AND (adapt*) AND (behavio*) AND (model* OR system*) AND (robot*)
(cult*) AND (adapt*) AND (cognitive) AND (model* OR architecture*) AND (robot*)
affordance* AND (behavio*) AND (adapt*) AND (robot*)
affordance* AND (cognitive) AND (model* OR architecture*) AND (robot*)
fac* AND expression AND cognitive AND (model* OR architecture*) AND (robot* )
fac* AND expression AND (behavio*) AND (model* OR system*) AND (robot*)
cognitive AND robot* AND architecture*
learning AND assistive AND robot*
affective AND robot* AND behavio*
empathy AND social AND robot*

.

Application of these search keys provided a total of 4916 hits with 1520 hits in Web of Science in the field “Topic” and 3396 hits in Scopus in “Article title, abstract, keywords” fields.

After deletion of duplicates, the titles and abstracts retrieved by the electronic search were read first, to identify articles deserving a full review. Papers about users’ emotion recognition and about emotions as unique input for HRI were excluded. Additionally, papers not written in English were excluded. A total of 1297 works was selected at this stage.

Then, a full-text assessment was carried out. The reading process led to the exclusion of 1241 papers that were out of topic, papers focusing only on definitions and taxonomy, papers for missing the model’s evaluation, and papers focusing more on users’ perception about robot’s abilities without behavioral adaptation.

The final list of papers includes 56 studies, which satisfy all the following selection criteria: (i) employment of cognitive architectures and/or behavioral models, (ii) explanation of cognitive architectures and/or behavioral models, (iii) research focus on robotic agent’s capabilities; (iv) behavioral adaptation according to different strategies, and (v) analysis conducted on social or assistive robots. The studies’ selection process is shown in Figure 1.

3. Results

3.1. Application Overview

The interest toward behavioral architectures has grown, as shown in Figure 2. Particularly, of the fully evaluated papers, 7 papers (12.5%) were published before 2014 and 49 papers (87.5%) were published within the past five years.

The selected papers can be divided into two big groups: the works describing cognitive architectures, behavioral adaptation models, and empathy models from a conceptual point of view (seven papers as summarized in

Table 2. Summary of recently published works on behavioral models without an experimentation loop.

Ref.	Title and Year	Aim	Robot	Social Cues	Models	Area
[32]	Persuasive AI Technologies for Healthcare Systems (2016)	Development of an IoT toolbox toward AI-based persuasive technologies for healthcare systems	NAO	Gaze behavior, facial behavior, speech cues	A system that contains: a sensor network architecture used to observe the environment and provide real-time access to the data for interpretation a block containing context-aware applications (object detection, activity recognition) a prediction block of future states	Cognitive architectures
[23]	Enabling robotic social intelligence by engineering human social-cognitive mechanisms (2017)	Suggestions and overviews on social cognitive mechanisms	-	-	A model that takes into consideration social-cognitive mechanisms to facilitate the design of robots	Cognitive architectures
[33]	An Architecture for Emotional and Context-Aware Associative Learning for Robot Companions (2015)	Theoretical architectural model based on the brain’s fear learning system	-	Environmental cues	Theoretical architectural model that uses artificial neural networks (ANNs) representing sensory thalamus, sensory cortex, amygdala, and orbitofrontal cortex.	Cognitive architectures
[34]	From learning to new goal generation in a bio-inspired robotic setup (2016)	Imitation of the neural plasticity, the property of the cerebral cortex supporting learning	NAO	Eye behavior	A model called Intentional Distributed Robotic Architecture (IDRA) that takes inspiration from the amygdala-thalamo-cortical circuit in the brain at its functional level	Cognitive architectures
[35]	Human-aware interaction: A memory-inspired artificial cognitive architecture (2017)	Human-aware cognitive architecture to support HRI	Care-O-Bot 4	Eyes and vocal behaviors	Partially observable Markov decision process (POMDP) model	Cognitive architectures
[31]	Artificial Empathy: An Interdisciplinary Investigation (2015)	Overview the research field aimed at building emotional and empathic robots focusing on its main characteristics and ongoing transformations	-	Gestures and posture cues	-	Empathy
[17]	How can artificial empathy follow the developmental pathway of natural empathy? (2015)	A conceptual model of artificial empathy is proposed and discussed with respect to several existing studies	Emotional communication robot, WAMOEBA - WE humanoid robot	-	Conceptual model of artificial empathy	Empathy
[36]	A Cognitive Control Architecture for the Perception–Action Cycle in Robots and Agents (2013)	Visual perception, recognition, attention, cognitive control, value attribution, decision-making, affordances, and action can be melded together in a coherent manner in a cognitive control architecture of the perception–action cycle for visually guided reaching and grasping of objects by a robot or an agent.	-	-	Model composed of four modules: object localization and recognition, cognitive control, decision-making, value attribution affordances, motion, and planning	Cognitive architectures
[37]	A computational model of perception and action for cognitive robotics (2011)	A novel computational cognitive model that allows for direct interaction between perception and action as well as for cognitive control, demonstrated by task-related attentional influences.	-	-	HiTEC architecture composed of a task level, a feature level, and a sensory motor level	Cognitive architectures
[38]	Interaction of culture-based learning and cooperative co-evolution and its application to automatic behavior-based system design (2010)	A bio-inspired hybridization of reinforcement learning, cooperative co-evolution, and a cultural-inspired memetic algorithm for the automatic development of behavior-based agents		-	Reinforcement learning for structure learning that finds the organization of behavior modules during the agent’s lifetime	Behavioral adaptation
[39]	Affective Facial Expression Processing via Simulation: A Probabilistic Model (2014)	Simulation Theory and neuroscience findings on Mirror-Neuron is used as the basis for a novel computational model, as a way to handle affective facial expressions.	-	postural and vocal cues	Simulation Theory and neuroscience findings on Mirror-Neuron Systems as the basis for a novel computational model,	Cognitive architectures
[40]	Multi-robot behavior adaptation to local and global communication atmosphere in humans-robots interaction (2014)	A multi-robot behavior adaptation mechanism based on cooperative–neutral–competitive fuzzy-Q learning is developed in a robot	-	Eyes cues	Cooperative–neutral–competitive fuzzy-Q learning	Behavioral adaptation
[41]	An Ontology Design Pattern for supporting behavior arbitration in cognitive agents (2017)	An Ontology Design Pattern for the definition of situation-driven behavior selection and arbitration models for cognitive agents.	MARIO robot-	affordances cues	Affordance Ontology Design Pattern (ODP)	Cognitive architectures
[42]	Context-Aware Cloud Robotics for Material Handling in Cognitive Industrial Internet of Things (2018)	In this paper, a cognitive industrial entity called context-aware cloud robotics (CACR) for advanced material handling is introduced and analyzed.	-	-	Energy-efficient and cost-saving algorithms for material handling	Cognitive architectures

) and the works presenting experimental studies (twenty-three papers summarized in

Table 3. Summary of recently published works on behavioral models with an experimentation loop.

Ref.	Title and Year	Aim	Robot	Social Cues	Participants	Models	Area
[30]	Compassion, empathy, and sympathy expression features in affective robotics (2017)	Comparison between expression features of compassion, sympathy, and empathy in British English and Polish that need to be tuned in social robots to enable them to operate successfully	Generic assistive robots	Linguistic cues, facial cues, movement cues, and physiological features	British English-speaking participants (mean age 23.2 years, 21 females) and 29 Polish-speaking subjects (mean age 25.6 years, 26 females)	Creating culture-specific emotion (compassion, sympathy, and empathy) models	Empathy
[43]	Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors (2010)	Comparison of a text-based interface with a character persuading user	I-Cat	Gaze and posture cues	24 middle age adults age 45 to 65 years old	The model is based on persuasion and uses Big five questionnaire on personality	Empathy
[16]	Inclusion of service robots in the daily lives of frail older users: A step-by-step definition procedure on users′ requirements (2018)	Definition of metrics to build an empathic robot that helps elderly people	Pepper	-	42 participants (elderly people)	The model is based on an emotion recognition algorithm	Empathy
[44]	Promoting interactions between humans and robots using robotic emotional behavior (2015)	Emotion-based assistive behavior for a socially assistive robot	Brian	Gestures and emotional state cues	34 subjects aged 17 to 68 years old	Markov model that uses a human affective state classifier and a non-verbal interaction and states analysis (NISA)	Empathy
[45]	A personalized and platform-independent behavior control system for social robots in therapy: development and applications (2018)	A behavior control system for social robots in therapies is presented with a focus on personalization and platform-independence	NAO Pepper	-	Children and elderly people	Model based on a Body Action Coding System (BACS)	Behavioral adaptation
[46]	Adaptive artificial companions learning from users’ feedback (2017)	Capacity of an intelligent system to learn and adapt its behavior/actions	EMOX	Current user(s) attributes and current environmental attribute cues	Children, teenagers, and adults	Markov Decision Processes (MDPs) model	Behavioral adaptation
[47]	Why robots should be social: Enhancing machine learning through a social human-robot interaction (2015)	How additional social cues can improve learning performance	Robot head mounted on an articulated robot arm	Gaze behavior	41 healthy and young participants with an average age of 24 years old.	The interaction model of the robot is based on language games	Behavioral adaptation
[25]	The affective loop: A tool for autonomous and adaptive emotional human-robot interaction (2015)	Affective model for social robotics	NAO	-	Children age 5 to 7 years old.	Plutchik emotional model; uses Fuzzy rules and learning by demonstration	Empathy
[48]	Puffy: A Mobile Inflatable Interactive Companion for Children with Neurodevelopmental Disorder (2017)	Robot’s ability in gestures’ interpretation	Puffy	Visual, auditory, and Tactile cues	19 children with an average age of 6 years old	Interactional Spatial Model which considers: interpersonal distance - relative (child-robot) bodily orientations - child’s and robot’s movements in space - child’s emotional state - child’s eye contact - robot’s emotional state	Behavioral adaptation
[49]	Social intelligence for a robot engaging people in cognitive training activities (2012)	Definition of user state and task performance, adjusting robot’s behavior	Brian 2.0	-	10 healthy and young participants from 20 to 35 years old.	A hierarchical reinforcement learning approach is used to create a model that allow the robot to learn appropriate assistive behaviors based on the structure of the activity	Behavioral adaptation
[50]	Empathizing with emotional robot based on cognition reappraisal (2017)	Continuous cognitive emotional regulation model for robot	-	-	10 subjects	Hidden Markov Model; uses cognitive reappraisal strategy	Behavioral adaptation
[51]	Interpretation of interaction demanding of a user based on nonverbal behavior in a domestic environment (2017)	Model for decision making on a user’s non-verbal interaction demand	MIRob	Eyes behavior	Eight subjects from 25 to 58 years old.	A model to decide when to interact with the user. It observes movements and behavior of the patient put them through a module called Interaction Demanding Pose Identifier. The data obtained are fed into the Fuzzy Interaction Decision Making Module in order to interpret the degree of interaction demanding of the user.	Behavioral adaptation
[52]	Understanding human intention by connecting perception and action learning in artificial agents (2017)	System inspired by humans’ psychological and neurological phenomena	-	Eyes behavior	-	Generic model; uses supervised multiple timescale recurrent neural networks	Cognitive architectures
[53]	Improving human-robot interaction based on joint attention (2017)	A novel cognitive architecture for a computational model of the limbic system is proposed, inspired by human brain activity, which improves interactions between a humanoid robot and preschool children	Robotis Bioloid	Eyes, auditory, and sensory cues	16 pre-school children from 4 to 6 years old.	Dynamic neural fields (DNFs) model; used with reinforcement and unsupervised learning-based adaptation processes	Cognitive architecture
[54]	A Robot Learns the Facial Expressions Recognition and Face/Non-face Discrimination Through an Imitation Game (2014)	A robotic system that can learn online is shown to recognize facial expressions without having a teaching signal associating a facial expression	-	Gestures, gaze direction, vocalization cues	20 persons	Theoretical model for online learning of facial expression recognition	Cognitive architecture
[55]	Development of first social referencing skills: Emotional interaction as a way to regulate robot behavior (2014)	Studying how emotional interactions with a social partner can bootstrap increasingly complex behaviors such as social referencing	-	Facial expression cues	20 persons	Model that uses the child’s affective states and adapts its affective and social behavior in response to the affective states of the child	Behavioral adaptation
[24]	May I help you? (2015)	Model of adaptive behaviors to pedestrians’ intentions	Robovie	Body and facial expression cues	The participants were visitors of the shopping mall where the robot was placed.	State transition model; used with an Intention-Estimation Algorithm	Behavioral adaptation
[56]	A framework for the design of person following behaviors for social mobile robots (2012)	Framework for people detection, state estimation, and trajectories generation for an interactive social behavior	Kompai	Body and facial expression cues	-	The model combines perception, decision, and action and uses fuzzy logic and SLAM algorithms	Behavioral adaptation
[57]	People-aware navigation for goal-oriented behavior involving a human partner (2011)	Person-aware navigation system	-	Body and facial expression cues	8 healthy and young subjects (7 male and 1 female) with an average age of 20.8 years.	The model weights trajectories of a robot’s navigation system for autonomous movement using Algorithms in navigation-stack of ROS	Behavioral adaptation
[58]	Promoting interactions between humans and robots using robotic emotional behavior (2015)	Map of human psychological traits to make robots emotive and sociable	NAO	Body and facial expression cues	10 subjects	Emotional Intelligence (EI) model; uses Meyer Briggs theory (MBT) and fuzzy logic	Behavioral adaptation
[59]	Moving toward an intelligent interactive social engagement framework for information gathering (2017)	To develop an integrated robotic framework including a novel architecture and an interactive user interface to gather information	NAO, Milo, and Alice	Body and facial expression cues	186 children from 8 to 12 years old	Interactive Social Engagement Architecture (ISEA) is designed to integrate behavior-based robotics, human behavior models, cognitive architectures, and expert user input to increase social engagement between a human and system	Cognitive architectures
[60]	Moving toward an intelligent interactive social engagement framework for information gathering (2018)	Software architecture allowing a robot to socially interact with human beings, sharing with them some basilar cognitive mechanisms	NAO	non- verbal cues such as social signals	Children	Hidden Markov Model (HMM) used as a reasoning approach The model encoded social interaction and uses natural and intuitive communication channels, both to interpret the human behavioral and to transfer knowledge to the human	Cognitive architectures
[61]	Persuasive robotic assistant for health self-management of older adults: Design and evaluation of social behaviors (2010)	To implement models obtained from biological systems to humanoid robots	Robothespian	Body and facial expression cues	237 subjects of different age, gender, and education	Model that uses a facial action coding system	Behavioral adaptation
[62]	A Logic-Based Computational Framework for Inferring Cognitive Affordances (2018)	A framework that reasons about affordances in a more general manner than described in the existing literature	-	Eye, gaze body orientation; verbal request	The robot is tested in different scenarios with the following domains as, for example, at an elder care facility	A DS theory framework often interpreted as a generalization of the Bayesian framework	Cognitive architectures
[63]	The Role of Functional Affordances in Socializing Robots (2015)	A paper where affordances of objects were used as a starting point for the socialization of robots	Jenny Robot	gesture, gaze, head movements, vocal features, posture, proxemics, and touch and affordances cues	Jenny is implemented for the tea-making task under various scenarios	OWL-DL model	Cognitive architectures
[64]	Intelligent emotion and behavior based on topological consciousness and adaptive resonance theory in a companion robot (2016)	An artificial topological consciousness that uses a synthetic neurotransmitter and motivation, including a logically inspired emotion system is proposed	CONBE robot	Gaze cues	Experiments done for the object recognition and face recognition	Behavioral model for recognition of objects and faces	Cognitive architectures
[65]	Practical aspects of deploying Robotherapy systems (2018)	Our robotic system helps therapists in sessions of cognitive stimulation. Without	NAO	Orientation of the gaze	Robot tested with different kind of cultures	A new module capable of communicating implemented with a behavioral architecture BICA	Behavioral adaptation
[66]	Staged Development of Robot Skills: Behavior Formation, Affordance Learning and Imitation with Motions (2015)	Realization of an integrated developmental system where the structures emerging from the sensorimotor experience of an inter- acting real robot are used as the sole building blocks of the subsequent stages that generate increasingly more complex cognitive capabilities	7 DOF Motoman robot arm	Motionese cues robot	The robot performed 64 swipe action executions towards a graspable object that is placed in a reachable random position	Model that includes three stages: discovering behavior primitives, learning to detect affordances, learning to predict effects	Behavioral adaptation
[67]	Personality affected robotic emotional model with associative memory for human-robot interaction (2016)	This paper discusses human psychological phenomena during communication from the point of view of internal and external factors, such as perception, memory, and emotional	Iphonod robot	Facial, gesture, voice cues	The experimental part is divided into two parts; first, the processing of multi-modal information into emotional information in the emotion model is simulated. Next, based on multi-modal information and emotional information, which came from the first one, the association process will be performed to determine the robot behaviors.	Model for object, facial and gesture, voice, and biometric recognition	Behavioral adaptation
[68]	Multimodal emotional state recognition using sequence-dependent deep hierarchical features (2015)	This model uses a hierarchical feature representation to deal with spontaneous emotions, and learns how to integrate multiple modalities for non-verbal emotion recognition, which makes it suitable for use in an HRI scenario	-	Facial expressions	Three experiments are executed and evaluated. The first one uses information of the face expression to determine emotional estates. The second one extracts information from the body motion, composed by arms, torso, and head movements, and the third one uses both types of information.	Multichannel Convolutional Neural Network (MCCNN) to extract hierarchical features	Behavioural adaptation
[69]	Artificial cognition for social human–robot interaction: An implementation (2015)	This article is an attempt to characterize these challenges and to exhibit a set of key decisional issues that need to be addressed for a cognitive robot to successfully share space and tasks with a human	Manipulator	Verbal communication, gestures, and social gaze	A scenario involving multi-modal, interactive grounding: the humans can refer to invisible or ambiguous objects that the robots anchor to physical objects through multi-modal interactions with the user. A second task that the robot has to achieve is cleaning a table	Model that has collaborative cognitive skills: geometric reasoning and situation assessment based on perspective-taking and affordance analysis; acquisition and representation of knowledge models for multiple agents (humans and robots, with their specificities); natural and multi-modal dialogue; human- aware task planning; human–robot joint task achievement.	Cognitive architectures
[70]	Collaborative Autonomy between High-level Behaviours and Human Operators for Remote Manipulation Tasks using Different Humanoid Robots (2017)	This article discusses the technical challenges that two teams face and overcame during a DARPA completion to allow the human operators to interact with a robotic system with a higher level of abstraction and share control authority with it	THORMANG “Johnny” and Atlas “Florian”	Affordances Visual cues	The two robots have to complete the two tasks of opening a door and a valve	Model for a humanoid robot that is composed by a remote manipulation control approach, a high-level behavior control approach, an overarching principle, and a collaborative autonomy, which brings together the remote manipulation and high-level control approaches	Behavioral adaptation
[71]	Probabilistic Movement Primitives for Coordination of Multiple Human-Robot Collaborative Tasks (2017)	This paper proposes an interaction learning method for collaborative and assistive robots based on movement primitives	Dual arm manipulator	Social-cognitive cues	A robot co-worker must recognize the intention of the human to decide some actions: if it should hand over a screwdriver or hold the box or coordinate the location of the handover of a bottle with respect to the location of the hand of the human.	Imitation learning to construct a mixture model of human-robot interaction primitives. This probabilistic model allows the assistive trajectory of the robot to be inferred from human observations	Cognitive architectures
[2]	Robots in Education and Care of Children with Developmental Disabilities: A Study on Acceptance by Experienced and Future Professionals (2017)	A study on the acceptance of robots by experienced practitioners and university students in psychology and education sciences is presented	NAO	Verbal cues	Demonstration of the capabilities of the robot in front of participants that had to filled a questionnaire at the end according to robot’s behavior	The aim is to examine the factors, through the Unified Theory of Acceptance and Use of Technology (UTAUT) model	Cognitive architectures
[72]	Optimized Assistive Human–Robot Interaction Using Reinforcement Learning (2016)	The proposed HRI system assists the human operator to perform a given task with minimum workload demands and optimizes the overall human–robot system performance.	PR2	-	The robot has to draw a prescribed trajectory	Reinforcement Learning LQR Method	Behavioral adaptation
[73]	The Impact of Social Robotics on L2 Learners’ Anxiety and Attitude in English Vocabulary Acquisition (2015)	This study aimed to examine the effect of robot assisted language learning (RALL) on the anxiety level and attitude in English vocabulary acquisition amongst Iranian EFL junior high school students	NAO		Iranian EFL junior high school students. Forty-six female students, who were beginners at the age of 12, participated in this study.	FLCAS questionnaire that evaluates anxiety	Cognitive Architectures
[74]	The multi-modal interface of Robot-Era multi-robot services tailored for the elderly (2018)	ROBOTERA project has the objective of implementing easy and acceptable service robotic system for the elderly.	Three Robot: Coro, Doro, Oro	Gaze, vocal, facial cues	Elderly people	Architectures based on emotion recognitions and object’s recognition	Cognitive Architectures
[75]	Human–Robot Facial Expression Reciprocal Interaction Platform: Case Studies on Children with Autism (2018)	In this research, a robotic platform has been developed for reciprocal interaction consisting of two main phases, namely as Non-structured and Structured interaction modes	Mina Robot	Facial expressions	Children	The model is composed of two modules: Non-structured and Structured interaction modes. In the Non-structured interaction mode, a vision system recognizes the facial expressions of the user through a fuzzy clustering method. In the Structured interaction mode, a set of imitation scenarios with eight different posed facial behaviors were designed for the robot	Cognitive Architectures
[76]	Adaptive Robotic Tutors that Support Self-Regulated Learning: A Longer-Term Investigation with Primary School Children (2018)	This paper explores how personalized tutoring by a robot, achieved using an open learner model (OLM), promotes self-regulated learning (SRL) processes and how this can impact learning and SRL skills compared to personalized domain support alone	NAO	Gaze	Children	Using an open learner model (OLM) to learn SRL processes	Behavioral adaptation
[77]	DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self (2018)	This paper introduces a cognitive architecture for a humanoid robot to engage in a proactive, mixed-initiative explo- ration and manipulation of its environment, where the initiative can originate from both humans and robots.	i-Cub	Gaze	Picking objects	The framework, based on a biologically grounded theory of the brain and mind, integrates a reactive interaction engine, a number of state-of-the-art perceptual and motor learning algorithms, as well as planning abilities and an autobiographical memory	Cognitive Architectures
[78]	Using a Humanoid Robot to Elicit Body Awareness and Appropriate Physical Interaction in Children with Autism (2015)	A human–robot interac- tion study, focusing on tactile aspects of interaction, in which children with autism interacted with the child-like humanoid robot KASPAR	KASPAR	Tactile, gaze cues	Children	Model for the behavior analysis	Cognitive Architectures

). Additionally, the papers can be grouped subsequently on the basis of the three areas described in the introduction and Figure 3 shows that most of the papers included in this review focus on behavioral adaptation strategies (41.07%) together with cognitive architectures (46.43%), and empathy (15.22%).

3.2. Data Abstraction

Data were abstracted from each selected article, as reported in

Table 4. Challenges and opportunity.

Keywords	Barriers/Limitations	Challenges and Opportunities	Research Topics
Sensors Technology	Multimodal sensors [32]	A multisensory system should be implemented in the model of a robot to create an improved architecture	Development of a multisensory system that could be used to detect different social cues at the same time (i.e., vocal, facial, and gaze cues).
	Reliable and usable sensor technology [32]	Sensors should be designed to be reliable and acceptable in a real-life situation to reduce the time-to-market	Design new sensors that can be used for a long time by the robot without being damaged. Design and test sensors to be resistant to possible impacts that the robot can have during its work. Design sensors resistant to external agents (i.e., water)
Perception	Real-time learning [28,84]	Real-time learning should be developed to adapt the behavior of the robot, according to the changing needs of the user	Develop a robot capable of adapting in real time to the changing needs of users. Analyze “social cues” that the robot should have according to the person it is approaching (i.e., children, aged people).
	Emotional state transitions [44]	Research on the emotional state module should be done more deeply	Investigate a larger variety of emotional states including the emotion transitions.
	Improving object detections [52]	Different approaches in the area of object detection should be investigated to obtain a strong model of the robot	Development of real-time object detection and recognition using geometrical model or simple CNN, deep and sophisticated CNNs
	Learning from the user [46,71]	The robot should be able to learn from the user in order to accomplish complex tasks	A model based on learning from demonstration methods could be very useful to let the robot achieve better and more complex skills.
	Affordances [41,62,63]	Affordances are important elements to be analyzed in the process of the implementation of a behavioral model for a cognitive robot	Develop a behavioral model that includes action selection of cognitive agents, following the notion of affordance. Create a network that covers some areas as a personal sphere (e.g., people information), life events (e.g., information about memories, scheduling, plans, etc.), environment sphere (e.g., information about rooms, furniture, objects, etc.), the health sphere (e.g., living patterns, health patterns, vital signs, etc.), and the emotional sphere (e.g., emotions, sentiments, opinions, etc.). Investigate affordance strategies related to deformable objects.
Experimental	Experimental session [35,61,85]	The model should be implemented on a real robot and tested to evaluate the proposed artificial cognitive architecture in dynamical environments	Test of the model of a robot in dynamics environments (i.e., outdoor and indoor, in crowded or not crowded places) and with people from different ages.
Architecture Design	Brain-inspired architecture [34]	Research in robot’s behavioral model should be conceived with a multidisciplinary approach to be able to adapt to the user’s needs	Implement a multidisciplinary approach to create better models for social factors (i.e., engineering, neuroscience, and psychology).
	Modular and flexible architecture [19]	Robot should adapt to different context and different preferences, which could change over time. Therefore, the architecture of a robot should be modular and flexible.	Develop modular and adaptable model of a robot to operate in different contexts (schools, hospitals, and industries). Implement a cloud architecture to offload intensive tasks to the cloud, to access a vast amount of data, and to share knowledge and new skills.
	Behavioral consistency, predictability, and repeatability [48]	These requirements should be investigated to obtain a complex model	Integrating those fields in the model of a robot could be obtained by creating an algorithm that represents the episodic memory. This could bring the creation of an artificial intelligent agent that can act more independently.
	Standardization	Having a high level of interoperability	Investigate the system interoperability to create a bridge among the different components of the system.
Ethical, legal, and social	Ethical and social aspects [19]	Ethical implications also should be investigated when creating a new model for a robot	Investigate the ethical and social implications of designing social robots with advanced HRI abilities.
	Legal aspect	No rules can be found in the legal field of social robotics	Investigate the regulation for social robots to overcome the problem that social robots cannot operate in environments with people without a supervision of an operator.
	Cultural adaption [24,40,65]	The robot should be able to adjust parameters for different cultures	Test new robots with different cultures to obtain a generic model for a robot. Many models work well when tested with people born in the country where they were developed.

The tables give the main purpose of each work, the robot used, the extracted features, and a short description of the implemented model/algorithms. The last column reports the area to which they belong (cognitive architectures, behavioral adaptation criterion, and empathy). In addition, for those papers which describe an experimental protocol, the number and the type of participants involved in the experimental session are also reported. The objective of the abstraction is to provide an overview of the papers included in this survey and to facilitate their comparison.

3.3. Theoretical Works on the Development of Robotics Behavioral Models

In this section, published works on theoretical studies of robotic models are described. Occurrences of theoretical studies belong to three areas (Table 2).

3.3.1. Concepts for the Cognitive Application Area

Human cognitive systems are often adopted as an inspiration to develop a cognitive architecture for robots. In the last several years, in fact, assistive and companion robots have accomplished advanced social proficiency whenever they were equipped with cognitive architectures. Relevant examples of this trend are listed below in this section.

Reference [23] described cognitive architectures citing the Learning Intelligent Distribution Agent, Soar, and the Adaptive Control of Thought-Rationale architecture with the aim to provide a set of commitments useful to develop intelligent machines. In this work are presented the Theory of Mind (ToM) and the “perceptual-motor simulation routines,” which are two of the fundamental theories of social cognition. Particularly, the ToM would represent the inherent human ability in attributed mental states to other social agents. That is possible through the application of theoretical inference mechanisms on cues gathered by people during social interactions (e.g., facial expression could be used in order to probabilistically determine the person’s emotional state). On the other hand, the paradigm of “perceptual-motor simulation routines” state that people would be able to understand others’ mental state by the use of simulation mechanisms, which would help the subject attribute a mental state to his\her interlocutor. The authors suggested their approach, Engineering Human Social Cognition (EHSC), which incorporates social signal processing mechanisms to allow a more natural HRI and focus on verbal and non-verbal cues to support interaction. Social signal processing is able to interpret social cues and then individual mental states. The authors underlined that modelling recommendations have centered primarily on the perceptual, motor, and cognitive modelling of a robotic system that spans disciplinary perspectives. This is the area that will require extensive work in the future. As such, the next steps in this area must include both research and modelling efforts that assess the issues and challenges of integrating the proposed types of models and formalisms. That effort can aid in the development of an integrated and working system based on these recommendations. These recommendations, if instantiated, would provide some basic perceptual, motor, and cognitive abilities, but future efforts should address whether these would also support more complex forms of social interaction. Such a capability would permit an artificial system to better express or perceive emotions while interacting and communicating with humans in even more complex social scenarios that would require shared decision-making and problem-solving.

Among cognitive architectures to be implemented into social robots to improve HRI, Pieters et al. [35] presented a work with the aim to develop a human-aware cognitive architecture. This system is conceived to provide robots with the ability to understand the human state, physical and affective, and then to interact in a suitable manner. Starting from cognitive models, the authors organized the architecture by considering a cognitive model that represents how memory is organized: a declarative memory for semantic and episodic facts, and procedural memory. According to this organization, the robot’s tasks are encoded as a sequence of actions and events, thanks to a symbolic task planner, with the aim to verify if, and in what way, the task has already been executed. In Reference [77], the authors proposed an architecture that drives the robot behavior to acquire language capabilities, execute goal-oriented behavior, and express a verbal narrative of its own experience in the world.

To provide robots with believable social responses and to have a more natural interaction, a theoretical model was developed by Reference [33]. The proposed architecture is human brain-inspired and it is structured into four principal modules, which encompasses anatomic structures and cognitive functions, such as the sensory system, the amygdala system, the hippocampal system, and the working memory. This brain-inspired system provides robots with emotional memory, which is fundamental to be able to learn and adapt to dynamic environments. In particular, the authors focus on artificial emotional memory, which lets robots remember emotions, associate them with stimuli, and react in an appropriate way if unpleasant stimuli occur. External stimuli are pre-processed by the sensory system, which is composed of the sensory cortex and thalamus. Prediction and association between stimuli and emotions are conducted via the amygdala system that provides emotional feedback to the hippocampal system.

Lastly, another brain-inspired architecture was developed in Reference [34]. It focuses on the autonomous development of new goals in robotic agents. Starting from neural plasticity, the Intentional Distributed Robotic Architecture (IDRA) is an attempt to simulate a brain circuit composed of the amygdala, the thalamus, and the cortex. The cortex is responsible for receiving signals from sensory organs. The thalamus develops new motivations in mammals, while the amygdala manages the generation of somatosensory responses. Elementary units, called Deliberative Modules (DM), enable a learning process that lets the robot learn and improve its skills during the execution of a task. This process is known as Intentional Distributed Robotic Architecture (IDRA). Working memory (acting as the cerebral cortex) and goal generator (acting as the thalamus) modules compose each DM. Amygdala is represented by instincts modules. Experiments were made to verify the ability of a NAO robot (https://www.softbankrobotics.com/emea/en/robots/nao/find-out-more-about-nao. Retrieved July 2018) in learning to distinguish particular object shapes and in exploring in an autonomous way and learning new movements. Sensing and actuation as main activities required for learning and cognitive development were tested: NAO was able to learn new shapes taking sensorial inputs and to compose new behaviors, which are consistent with these goals. The authors underlined their choice to opt for directly using the high-level representation of the neural function, even though a system that uses neural coding as basic representation could be integrated into IDRA. NAO is often used to implement cognitive architectures. It was used in Reference [79] to evaluate a robot-assisted therapy for children with autism and intellectual disability (the same was done in Reference [78] with a robot named Kaspar) and in Reference [73] to examine the effect of robot-assisted language learning (RALL) on the anxiety level and attitude in English vocabulary acquisition among Iranian EFL junior high school students.

Another important element that should be considered in the field of behavioral models is the mechanism of affordances. The concept of affordance refers to the relationship between human perceivers and aspects of their environment. Being able to infer affordances is central to common sense reasoning, tool use, and creative problem solving in artificial agents.

Cutsuridis et al. [36] created a cognitive control architecture of the perception–action cycle for visually guided reaching and grasping of objects by a robot or an agent melded perception, recognition, attention, cognitive control, value attribution, decision-making, affordances, and action. The suggested visual apparatus allows the robot/agent to recognize both the object’s shape and location, extract affordances, and formulate motor plans for reaching and grasping.

Haazebroek et al. [37] presented HiTEC, a novel computational (cognitive) model that allows for direct interaction between perception and action as well as for cognitive control, demonstrated by task-related attentional influences. In their model, the notion of affordance is effectively realized by allowing for automatic translation of perceptual object features (e.g., object shape) to action by means of overlap with anticipated action effect features (e.g., hand shape). Reference [39] proposed a Simulation Theory and neuroscience findings on Mirror-Neuron Systems as the basis for a novel computational model, as a way to handle affective facial expressions. The model is based on a probabilistic mapping of observations from multiple identities onto a single fixed identity (‘internal transcoding of external stimuli’), and then onto a latent space (‘phenomenological response’). Asprino et al. [41] presented in this paper an Ontology Design Pattern for the definition of situation-driven behavior selection and arbitration models for cognitive agents. The proposed pattern relies on the descriptions and situations ontology pattern, combined with a frame-based representation scheme. Inspired by the affordance theory and behavior-based robotics principles, their reference model enables the definition of weighted relationships, or affordances, between situations (representing agent’s perception of the environmental and social context) and agent’s functional and behavioral abilities. These weighted links serve as a basis for supporting runtime task selection and arbitration policies, to dynamically and contextually select agent’s behavior.

Lastly, a different use of a cognitive industrial entity called context-aware cloud robotics (CACR) is used for advanced material handling. Compared with the one-time on-demand delivery, CACR is characterized by two features: (1) context-aware services and (2) effective load balancing. The CACR case study is performed to highlight its energy-efficient and cost-saving material handling capabilities.

3.3.2. Concepts for the Empathy Area

Empathy is becoming an important field of social robotics and several behavioral models take this aspect into consideration.

Reference [31] showed how different models based on emotions were created to build empathetic and emotional robots. The main cues used in these models are movements, gestures, and postures. In another paper, the same authors explored different dimensions of artificial empathy and revealed different empathy models: a conceptual model of artificial empathy that was structured on the developmental axis of self-other cognition, statistical models based on battery level or temperature, and a four-dimension empathy model were presented and described. The cues used in this article were unimodal and multimodal communication cues as opposed to the previous one that used movements.

Reference [80] discussed a conceptual model of artificial empathy with respect to several existing studies. This model is based on affective developmental robotics, which provide more authentic artificial empathy based on the concept of cognitive developmental robotics. The authors showed how the model worked using two different robots: an emotional communication robot called WAMOEBA and a humanoid robot called WE.

3.3.3. Concepts for Behavioral Adaptation Area

Designing an intelligent agent is a difficult task because the designer must see the problem from the agent’s viewpoint, considering all its sensors, actuators, and computation systems. Farahmand et al. [38] introduced a bio-inspired hybridization of reinforcement learning, cooperative co-evolution, and a cultural-inspired memetic algorithm for the automatic development of behavior-based agents. Reinforcement learning is responsible for the individual-level adaptation. Cooperative co-evolution performs at the population level and provides basic decision-making modules for the reinforcement-learning procedure. The culture-based memetic algorithm, which is a new computational interpretation of the meme metaphor, increases the lifetime performance of agents by sharing learning experiences between all agents in the society. To accelerate the learning process, the authors introduced a cultural-based method based on their new interpretation of the meme metaphor. Their proposed memetic algorithm is a mechanism for sharing learned structures among agents in society and lifetime performance of the agent, which is quite important for real-world applications, increases considerably when the memetic algorithm is in action.

3.4. Experimental Works on the Development and Implementation of the Behavioral Model

In this section, published works on behavioral models with the experimental loop are shown and are divided into sub-categories, according to the application area (Table 3).

3.4.1. Experimental Works for Cognitive Architectures

Concerning cognitive architectures, Reference [53] proposed a cognitive framework inspired by the human limbic system to improve HRI between humanoid robots and children during a game session. The robot’s emotional activity was modelled with computational modules representing amygdala, hippocampus, hypothalamus, and basal ganglia and used to suggest users’ optimal game actions. The results showed that this cognitive architecture provided an efficient mechanism for representing cognitive activity in humanoid robots. The children’s attention level was higher when compared to those of a game session without the use of the robot.

Reference [59] aimed to use an Interactive Social Engagement Architecture (ISEA) and an interactive user interface to gather information from children. The authors tested the developed architecture with an NAO robot and two other humanoids with 186 children. The ISEA is able to integrate and combine human behavior models, behavior-based robotics, cognitive architectures, and expert user input to improve social HRI. Eight modules compose the framework presented: knowledge, user input, sensor processing, perceptual, memory, behavior generation, behavior arbitration, and behavior execution modules. The knowledge module models human behaviors, while the perceptual module manages external sensor data from the environment, and processes and interprets data, sending results to the memory module. The behavioral generation module calculates which behavior and communication strategies must be used and sends data to the behavioral generation module. Novel emergent behaviors can be obtained by combining newly generated behaviors with the stored behaviors in memory modules. Every time that behavior is displayed, the robot’s internal state is updated to keep track of the new data storage. Preliminary results showed that children seemed to find it more comfortable to establish an engagement with a robot, rather than with humans, in sharing information about their bullying experiences at school. Although this research is only midway through the grant award period, the developments and results are promising. Moreover, the authors said that slow and steady progress is occurring with the development of this Integrated Robotic Toolkit, but there is still significant and ongoing work to be explored with this approach.

Reference [52] proposed an intention understanding system that consists of perception and action modules. It is an object-augmented model, composed of two neural network models able to integrate perception and action information to allow the robot to better predict the user’s intention. The model was tested in a cafeteria with customers and clerks. The action module was able to understand the human intention and associate a meaning to predict an object related to that action. The combination of these modules resulted in an improved human intention detection.

As explained in the theory section of the cognitive area, affordances are important elements for building a behavioral model for social robots. Those ones encode relationships between actions, objects, and effects and play an important role in basic cognitive capabilities such as prediction and planning [62], which also developed a computational framework based on the Dempster-Shafer (DS) theory for inferring cognitive affordances. They explained that this, much richer level of affordance representation is needed to allow artificial agents to be adaptable to novel open-world scenarios. Reference [63] also underlined the fact that affordances play an important role on basic cognitive capabilities such as prediction and planning. The authors said that the problem of learning affordances is a key step toward understanding the world properties and developing social skills.

Reference [69] also proposed a model that has collaborative cognitive skills such as geometric reasoning and situation assessment based on perspective-taking and affordance analysis. Another important element to be taken into consideration in the implementation of a behavioral model are facial expressions. Those ones are often based on an inner model that is related to the emotional state and are not only based on categorical choice. Chumkamon et al. [64] proposed a framework that focuses on three main topics including the relation between facial expressions and emotions. The first point of their model is the organization of the behavior including inside-state emotion regarding the consciousness-based architecture. The second one presents a method whereby the robot can have empathy toward its human user’s expressions of emotion. The last point shows the method that enables the robot to select a facial expression in response to the human user, which provides instant human-like ‘emotion’ and is based on emotional intelligence (EI) that uses a biologically inspired topological online method to express, for example, encouragement or being delighted. Another application of facial expressions in a cognitive architecture is shown in Reference [39] and in Reference [75]. Reference [55] proposed a robotic system that could learn online to recognize facial expressions without having a teaching signal associated with a facial expression. Reference [74] also created a system composed of three robots that helped elderly people during their daily works such as reminding them of taking drugs or bringing them the objects that they desired and analyzing their facial expressions to recognize them.

Lastly, learning from demonstration is used in Reference [71]. The authors proposed a learning method for collaborative and assistive robots based on movement primitives. The method allows for both action recognition and human-robot movement coordination.

3.4.2. Experimental Works on Empathy

When a social robot interacts with human users, empathy represents one of the key factors to increase natural HRI. Emotional models are fundamental for social abilities to reach empathy with users.

Reference [81], for example, evaluated and compared the emotion recognition algorithm in two different robots (NAO and Pepper) and created metrics to evaluate the empathy of these social robots.

Reference [44] developed emotion-based assistive behavior to be implemented in social assistive robots. According to the user’s state, the model is able to provide abilities to the robot to show appropriate emotions, which elicits suitable actions in humans. The robot’s environmental and internal information plus user affective state represent the inputs for the Brian robot (Brownsell, Alex (29 May 2013). “Confused.com overhauls brand in search of ‘expert’ positioning”. Marketing Magazine. http://www.marketingmagazine.co.uk/article/1183890/confusedcom-overhauls-brand-search-expert-positioning. Retrieved July 2018) to alter its emotional state according to the well-being of a participant and to the assistant in executing tasks. In this work, the robot emotional module is employed not to provoke emotional feelings, but rather in terms of assistive tasks that the robot should perform to satisfy the user’s well-being.

The experiments show the potential of integrating the proposed online updating Markov chain module into a socially assistive robot to obtain compliance from individuals to engage in activities. Using robotic behavior that focuses on the well-being of the person could be beneficial to the person’s health. Moving to a fully encompassing target user group is needed to test the overall robot in its intended assistive applications.

Reference [43] implemented an experiment with the I-Cat robot (http://www.hitech-projects.com/icat/. Retrieved July 2018), which aims to provide a computer-based assistant that could persuade and guide elderly people to behave in a healthy way. Previous works demonstrated that combining the robot’s empathy with the user’s state contributed to a better appreciation of a personal assistant [82]. I-Cat features an emotional model that makes it able to smile and express sadness. Authors implemented natural cues such as understanding, listening, and looking, to perform different roles for the robot (educator, buddy, and motivator). The analysis was conducted by considering participants’ personalities. The percentage of the total time that participants talked, laughed, and looked at the robot, and how many times the participants said “goodbye,” as a sign of interpretation of the robot as a social entity. The aim of the work was to establish behaviors for an electronic personal assistant with a high level of dialogue, emotions, and social competencies. The findings showed that natural cues used by I-Cat provoked more empathy and social involvement with users. When non-social cues were used, users perceived the robot as less trustworthy and less persuasive, while avoiding its suggestions.

During experiments, the physical characters were found to be more trustworthy but less empathetic than the virtual character, which was not expected. This negative outcome on empathy might be due to specific constraints of the iCat: it makes a relatively high amount of noise when it moves, and the head and body movements may not be fluent enough. Another technical constraint was the (occasional) appearance of errors in the movements and speech, such as skipping choices of the multiple-choice questions. Furthermore, it may be that the three-character roles did not capture important advantages of a physical character that can act in the real environment. For instance, more positive outcomes might show up with a character that helps to attend to a medicine box with a specific location in the house, compared to a virtual character that is not a real actor in the house.

Reference [25] provided their contribution to social pervasive robotics by proposing an affective model for social robots, empathizing the concept of empathy. Behavioral adaptation according to users’ needs and preferences resulted in preliminary tests that achieved a better social inclusion in a learning scenario. The first part of the model, called “Affective loop,” was a module for the perception of humans, characterized by body-based emotion recognition that can recognize human emotions. According to the perception for human module’s outputs, the internal state of the robot changed, which generates a complex emotional spectrum using a psycho-evolutionary theory of emotions. The user was able to visualize the robot’s internal state and adjust some system parameters for the duration and intensity of each emotion. The user’s interest in interaction was then monitored by the visual system: when it decreased, the robot changed its behavior to socially involve the user and selected its emotion according to the user’s state. Affective behaviors were also adapted to the goal of interaction in a cooperative task between the robot and users.

Lastly, a comparison between two different cultures was made in Reference [30]. They made, in fact, a comparison between expression features of compassion, sympathy, and empathy in British English and Polish using emotion models that had sensory cues as inputs.

3.4.3. Experimental Works on Behavioral Adaptation

An attempt to develop robots to be emotive and sociable like humans, showing a capability to adapt behavior in a social manner, is presented in Reference [58]. Starting from the Meyer-Briggs Theory on human personality, the authors mapped human psychological traits to develop an artificial emotional intelligence controller for the NAO robot. The proposed model was modelled as a biological system, and as a structure of emotionally driven and social behavior represented by three fuzzy logic blocks. Three variables were used as system input: “trigger event” that incites different psychological reactions, “behavior profiler” that models event-driven behavior to fit profiles of individuals whose behavior needs to be modelled, and “behavior booster/inhibitor” that augments or decreases the affective expressiveness. Social behavior attributes were implemented in the NAO robot controller according to this model. The robot interacted with young researchers, recognizing calls and gestures, and locating people in the environment, and showing personality traits of joy, sociability, and temperament. The model considers personality traits, social factors, and external/internal stimuli as human psychology does when interacting with others. In Reference [76], Nao was also used to assist children in developing self-regulated learning (SRL) skills. Combining the knowledge about personality traits discovered with Meyer-Briggs Theory and validated by Reference [58] and experimental measurements of affective reactions from a live model performed by an actor, Reference [61] developed a cognitive model of human psychological behavior. This model includes personality types and human temperaments to be implemented into the Robothespian humanoid robot. The authors tuned the block scheme developed in Reference [58] according to measurements from an actor performing as a behavioral live model. Different affective behaviors were played to create affective reactions to be added to the previous model.

Studies on proxemics, speed, and velocity provided unique suggestions to improve HRI, especially in behavior adaptation according to the user’s movements and position. In Reference [56], the authors investigated a robot’s trajectories and speed when it follows a user in a real domestic environment to provide a comfortable social interactive behavior. The authors presented a framework for people detection, state estimation, and trajectory generation that can regulate robotic behavior. To select the appropriate behavior, the robot used the state of the user and his/her localization as input, considering movements and the context. Trajectories and velocity were considered in Reference [57], with a robot moving with a social partner toward the same goal. The authors developed and tested a person-aware navigation system modifying a trajectory planner. The criterion to change the planner was the distance between the robot and the user, according to which the robot’s behavior adapted its velocity and trajectory to reach the goal, but remained close to the user at the same time. The approach described in this paper is limited because it only considers distance to the goal while ignoring the available free space. This model could be augmented to consider free-space features, such as free space in front of each social agent, distances to walls, and distances to other obstacles, to be more informed.

A similar work is presented by Reference [51] with a model to interpret the user’s behavior and inclination toward interaction with an assistant robot. The robot was able to determine the user’s behavior through body movements and extraction of posture features. According to its interpretation, the robot decided if it should move closer or should wait for a better inclination from the user to interact. The major benefit of this model is that it does not use verbal instruction from the user, which allows the robot to assess the suitability of starting a conversation by using posture and movement analysis.

Behavioral adaptation according to users’ preferences and feedback on robot’s actions is presented in Reference [46]. Two learning algorithms were applied to an internally developed adaptive robot, known as the EMOX (EMOtioneXchange) robot. After having identified the user’s profile, the robot proposed a personalized activity, while assisting and interacting with the user after the activity selection. The user’s feedback after each activity was traced, letting the robot have a memory about the user’s preferences to aid in suggesting a more appreciated activity later. The robot’s architecture has observations of user behavior, feedback, and environment to use as input. The robot’s actions are the system output, which are determined through knowledge rules as interaction traces, users’ profiles, decision process, and learning from the feedback process. The results showed that, even if the interaction modality, with hand gestures, was found difficult, most participants found the robot behavior adaptable and pertinent to their preferences.

Reference [49] presented a novel control architecture for the internally developed Brian 2.0 robot. The aim was to adapt the robot’s behaviors according to the user state, which is a social motivator and assists if needed. To be effectively integrated into society, robots should be provided with social intelligence to interact with humans. This architecture promoted the robot’s abilities to support and motivate users during a game memory session to stimulate humans cognitively. Encouragement and assistance were provided through a modular learning architecture that determined the user’s state and performances, which modified the robot’s behavior according to these inputs, recorded through sensors, cameras, and modules. The combination of the robot’s emotional state module and intelligence layer led establishment of the current robot’s assistive action related to the user’s state and adapts the robot’s behavior to the interactive scenario, using non-verbal modalities of communication.

Reference [24] investigated a robot’s behavior by proposing a model that adapted to the visitor’s intention. In a shopping mall, a humanoid robot was tested during approaching and interaction tasks. The robot was provided with two interaction strategies depending on users’ behaviors: when visitors showed uncertain intentions, the “proactively waiting” strategy was used and the robot went toward them. The “collaboratively initiating” strategy, instead, was used when visitors’ willingness to interact was seen and the robot started a conversation and moved closer to them. To reach a more natural context in interacting with robots, Reference [47] presented an experiment with a social robot learning to perform word-meaning associations. The authors hypothesized that a different human attitude in approaching the robot could be obtained. The robot’s design had the aim to evoke a strong social response from humans. The social cues used influenced the tutoring of the human teacher and his behavior. An HRI interaction was measured through a language game, during which the learner assimilated a lexicon and associated meanings. Based on the teacher’s feedback, the learner modified the word-meaning association. It could be considered as a sort of behavioral adaptation, applied in a different context that could improve the robot’s social abilities. Through users’ facial tracking, the robot was able to address participants during the interaction, which emphasized the social involvement. Additional multi-modal social cues (gaze and verbal statement) to express its learning preference was used by the robot, which modulates the interaction and positively influences it. Reference [48] developed a spatial relationship model that considers interpersonal distance, body orientations, emotional state, and movements. On the basis of these inputs, the robot decides how to proceed, which sets its voice and moves toward the user or not. As the robot comes close to the child, entering the “personal” distance zone, the current status of the user is re-evaluated to adapt better to the robot’s actions. Children with cognitive disabilities interacted with the robot, executing free and structured game sessions. Robot tactile sensors led us to understand tangible interaction, as an expression of touch-interaction through physical contact with the children. Depending on the touch-contact typology, the robot was able to select an appropriate behavior using multimodal emotional expressions. The robot’s behavior can be adapted depending on the user’s emotion, seen as an emotional stimulus for the robot’s cognitive architecture. Reference [50] proposed a cognitive-emotional interactive model for interactive and communication tasks between young users and a robot. During the interaction, the emotional robot acted its emotions using facial expression, movements, and gesture as a consequence of the user’s emotion, according to the Hidden Markov Model. The use of this model allowed the robot to regulate emotions as humans do, which provides a better interaction. The model starts from the hypothesis that robots might know a human’s cognitive process, in order to understand human’s behaviors. To do that, an object-functional role perspective method allowed robots to understand humans’ behaviors: objects are interpreted as object-functional roles and role interactions. An activity is interpreted as an integration of object role interactions, so the robot is able to predict and understand a human activity. Because this model is only involved in emotional intensity attenuation, the continuous prediction of spontaneous affect still needs to be improved in the future, and the authors are considering expanding the experimental sample size and seeking more effective evaluation approaches for affective computing. Reference [54] also proposed a model that used a child’s affective states and adapted its affective and social behavior in response to the affective states of the child.

In Reference [83], the authors also try to adapt robots’ behavior to human emotional intention and an information-driven multi-robot behavior adaptation mechanism is proposed for human–robot interaction (HRI). In the mechanism, the optimal policy of behavior is selected by information-driven fuzzy friend-Q learning (IDFFQ), and facial expression with identification information are used to understand human emotional intention. It aims to make robots become capable of understanding and adapting their behaviors to human emotional intention, in such a way that HRI runs smoothly. The importance of facial expressions for the implementation of social robots is shown in the other two works. Reference [67] created a model for object, facial, gesture, voice, and biometric recognition and Reference [68] used a Multi-channel Convolutional Neural Network (MCCNN) to extract emotions from facial expressions.

Affordances are also used to implement a behavioral model that can adapt to users’ needs. Another important aspect related to behavioral models is the cultural adaptation of the robot. Reference [40] proposed a multi-robot behavior adaptation mechanism based on cooperative-neutral-competitive fuzzy Q learning for coordinating local communication atmospheres in human-robots interaction. The Fuzzy Q learning is an approach that fuses fuzzy logic with the discrete Q-learning method and the authors called communication atmospheres significant information were introduced for human-robot interactions. This approach was tested with people from different countries and with different backgrounds to overcome the problem of cultural adaptation. Reference [65] also proposed a robotic system that helps therapists in sessions of cognitive stimulation. Without taking into account aspects such as the patient’s perception of the robot, or the impact of the cultural environment, the application of these systems may be doomed to failure. The authors showed pieces of evidence of how the cultural adaptation of the robots has been considered decisive in their success.

Inspired by infant development, Reference [66] proposed a three-staged developmental framework for an anthropomorphic robot manipulator. In the first stage, the robot is initialized with a basic reach-and-enclose-on-contact movement capability and discovers a set of behavior primitives by exploring its movement parameter space. In the next stage, the robot exercises the discovered behaviors on different objects and learns the caused effects. This effectively builds a library of affordances and associated predictors. In the third stage, the learned structures and predictors are used to bootstrap complex imitation and action learning with the help of a cooperative tutor. Reference [70] developed an innovative approach that allows one or more human operators to share control authority with a high-level behavior controller on the basis of previous work on operator-centric manipulation control at the level of affordances. In their work, the affordances of the object template can be requested from the Object Template Server (OTS) and can be executed so that the robot performs the required arm motions to achieve the manipulation task.

Lastly, Reinforcement Learning techniques were also used to create an HRI system (robot that assists the human operator) to perform a given task with minimum workload demands and optimizes the overall human–robot system performance [72].

4. Discussion

The aim of this work is to analyze the state of the art and, thus, to provide a list of hints regarding cognitive architectures, behavioral adaptation, and empathy. Future research efforts should lead to overcoming the limitation of the current state of the art, as summarized in Table 4.

It shows several areas that have to be analyzed in future works as sensors technology, perception, architecture design, and the presence/lack of an experimental phase.

Moreover, ethical, legal, and social aspects should be taken into consideration to build an efficient behavioral model for future robots.

4.1. Sensors Technology

A crucial aspect in HRI is how robots manage to understand intentions and emotions of the users using social cues (i.e., posture and body movements, facial expression, head and gaze orientation, and voice quality). Sensors play a fundamental role because they are used to detect these cues, which are then processed in the robot model. An issue related to sensors is the data acquisition that can face delays, so an effort for the future could be to design sensors that are reliable and usable in real-life situations. Moreover, the robot should have a multisensory to acquire different types of signals [32]. To achieve this goal, microphones, 2D and 3D vision sensors, thermal cameras, leap motion, Myo, and face-trackers could be collected to create a system that gives a complete sensor coverage to the robot. Each device could cover a different area. Microphones could be used for speech, Myo to collect IMU and EMG data, face-trackers to find the head pose, gaze, and FACS, vision sensors to acquire point cloud data, thermal cameras to detect objects in dark environments, and a leap motion to track and estimate the position of the hand.

4.2. Perception and Learning from the User

A second ability that should be deeply analyzed is the area of perception. The main problem of perception is to have a reliable real-time sensing and learning system, as expressed in Reference [84]. In this paper, the authors show that people’s preferences and knowledge change over time and a good system should be capable of adapting in real time to these changes and should be able to learn from the user. To achieve the latter competence, advanced learning-based methods [86] should be used to satisfy user needs, while increasing the performance of the robot [87]. Additionally, future works should detect and handle the emotion transition since humans change their emotions steadily [44]. This topic is one of the main issues of HRI because the robot must have a real time data acquisition to handle the emotional transitions and this is quite difficult to gain due to delays during the data acquisition.

Another limitation of this area is the disconnection between perception and actions [52] and this problem should be overcome to have a reactive robot. Reference [28] underlined this concept, saying that changes in the user profile should be promptly detected to adapt behavior automatically.

An effort that could be interesting to study more deeply in the future is the possibility for the robot to achieve complex skills learned from the user (i.e., for cleaning a table). Reference [46] proposed that the robot should be able to change the internal model on the basis of what it learns from human beings, using the learning from the demonstration approach.

Moreover, it is worth underlining the importance to increase the number of non-verbal input parameters considered in the analysis, to make the robot more compliant and adaptable to the user’s state and preferences [51].

4.3. Architecture Design

Concerning the architecture design, more modular and more flexible architectures should be created, in the future: robots should be able to, for example, autonomously react to an unplanned event and a complete risk analysis procedure should be performed in the design phase to correctly handle the unplanned situations [19,24].

Behavioral consistency, predictability, and repeatability should be investigated since they are fundamental requirements in the design of socially assistive robots in different contexts, such as for children with autism as underlined in Reference [48]. Addressing them requires an accurate case analysis, grounded on the current practice and on extensive experimentation. A possible approach could be the use of learning from demonstration to teach the robot some skills to achieve a better performance [87].

Moreover, a multidisciplinary approach should be pursued to design and develop reliable and acceptable behavioral models [34]. Psychology, biology, and physiology, among others, are areas of expertise, which should be part of the development process since they can help improve the HRI experience [88]. Innovative behavioral models for assistive robots could be developed by taking inspiration from the studies of human social models or from the study of a specific anatomical apparatus. In this manner, future research will not only consider the physical safety (of the robot and the human beings) but also psychological, anatomical, and social spheres of humans [48].

It is important to underline that the robot might be appropriate not only in the context for which it was originally conceived (i.e., private home, hospital, and residential facility) but also for people with different levels of residual abilities. In other words, the robot should be able to adapt to the variability and different cultural and social contexts [89].

Lastly, a model of a robot should have a cloud architecture to have the ability to offload intensive tasks to the cloud, to access a vast amount of data, to access shared knowledge, and not to lose information in case of connection problems.

4.4. Experimental Phase

The advantage to test robotic solutions with real users is absolutely remarkable, as underlined by the comparison between papers with or without the experimental sessions. Particularly, some works [33,35,61] remark on the importance of testing the proposed model as a fundamental step for future research. The main issue is that, in some scenarios, robots that completed a task during the simulation phase, do not succeed in the experimental phase. This is the reason why testing the behavior of the robot in a real environment is of extreme importance in order to find good parameters that work for the experimental phase. In addition, the architecture should be validated using physical robots that interact with users in dynamic environments (i.e., schools, industries, and hospitals).

Lastly, another limitation that can be found in several papers is the absence of a database (i.e., physical forces or emotional states of the user). A database could be useful to collect information achieved from the experimental phase and could be useful not only for the researchers that are working on the project, but also for other researchers that want to use the data for their own projects.

4.5. Ethical, Legal, and Social Aspects

Future research effort should include the ethical implications of designing robots that interact with people and the data that should be acquired and correctly stored to guarantee privacy [19].

Regarding legal and social aspects, Reference [24] underlined that the robot should be able to adjust its parameters for different cultures that have different needs, in order to be able to satisfy real user requests. Lastly, the birth of a regulation for social robots, like the one created for drones during recent years, could be an important step in order to have the possibility to use a robot in crowded environments [90].

5. Conclusions

This paper focuses on behavioral adaptation, cognitive architectures, and the establishment of empathy between social robots and users. The current state-of-the-art of existing systems used in this field is presented to identify the pros and cons of each work with the aim to provide recommendations for future improvements.

To establish a set of benchmarks to define an HRI similar to human-human interaction is an enormous challenge because of the complexity of non-verbal phenomena in social interactions. Its interpretation needs the support of psychological processes and neural mechanisms. The topic of the behavioral model is huge, and several factors contribute to making robots more accepted, perceived as friends, and empathetic with users. A common limitation of the works presented is that, often, authors focused on a particular aspect of HRI emphasizing a communication strategy or a particular behavior as a reaction to the user’s action. Since it is not easy to include behavioral adaptation techniques, cognitive architectures, persuasive communication strategies, and empathy in a unique solution, researchers are often limited to organize experimental studies, which include only some of these factors. This, unfortunately, provides useful information only to a limited part of persuasive robotics. To maintain the importance of each contribution, it is fundamental to include, in a whole vision, all the suggestions provided by each work. Although many improvements remain to be accomplished, the already satisfying results from the authors have achieved an optimum starting point to develop a better solution using knowledge of human cognitive and psychological structures.