Next Article in Journal
Job Insecurity and Employee Engagement: A Moderated Dual Path Model
Next Article in Special Issue
Self-Perception of the Digital Competence of Educators during the COVID-19 Pandemic: A Cross-Analysis of Different Educational Stages
Previous Article in Journal
The Mainstreaming of NBS in the SECAP of San Donà di Piave: The LIFE Master Adapt Methodology
Previous Article in Special Issue
Internet and People with Intellectual Disability: A Bibliometric Analysis
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Integration of Sustainable Development Goals in Educational Robotics: A Teacher Education Experience

by
Despoina Schina
*,
Vanessa Esteve-González
,
Mireia Usart
,
José-Luis Lázaro-Cantabrana
and
Mercè Gisbert
Department of Pedagogy, Applied Research in Education and Technology (ARGET), Universitat Rovira i Virgili, 43003 Tarragona, Spain
*
Author to whom correspondence should be addressed.
Sustainability 2020, 12(23), 10085; https://doi.org/10.3390/su122310085
Submission received: 13 November 2020 / Revised: 29 November 2020 / Accepted: 1 December 2020 / Published: 3 December 2020
(This article belongs to the Special Issue ICT and Sustainable Education)

Abstract

:
Education is of key importance to achieve the Sustainable Development Goals (SGDs), particularly now that the COVID-19 pandemic threatens the progress made toward a more inclusive, peaceful, fair, and prosperous society. In this research study, students of the bachelor’s degree in Pedagogy receive a specialized training that addresses the SDGs through Information and Communication Technologies (ICT) referring to computers and digital technologies. In particular, the training addresses the SDGs through Educational Robotics (ER). The aim of the training course is twofold: first, to reinforce the students’ Teacher Digital Competence (TDC), defined as teachers’ ICT competence and didactic and methodological skills; and second, to train them in the integration of SDGs in ER educational projects. This paper presents a case study in which qualitative and quantitative methods are implemented. The objectives of the study are to investigate the students’ level of TDC and examine their ability to integrate SDGs in ER projects. The research instruments employed are the COMDID-A questionnaire, a lesson plan template, and a lesson plan evaluation rubric. Among the most prominent findings, it is shown that the training enabled the students to work on sustainability in a cross-curricular manner and successfully develop ER projects centered on different SDGs. These conclusions will encourage teacher education institutions to integrate SDGs in their curriculum.

1. Introduction

The COVID-19 pandemic is menacing the future of our society and people’s well-being, making the accomplishment of the Sustainable Development Goals (SDGs) more challenging than ever. The pandemic is affecting the achievement of all the SDGs, as they do not seem to be a primary concern anymore [1], even though they were considered as an urgent call for action before the outbreak of the pandemic [2]. The pandemic has caused a global economic recession; an increasing number of people will probably suffer from poverty and inequalities [3]. The pandemic has also affected the field of education; it is having a negative impact on the learning outcomes and the social and behavioral development of most of the world’s student population, placing vulnerable and disadvantaged students at risk of educational exclusion, and widening existing educational inequalities [4]. The educational inequalities are becoming more and more accentuated, as the developing countries do not have the means to support students’ education through remote learning, as the developed ones.
In the current literature, technology is seen as a tool to solve challenges provoked by the COVID-19 pandemic in several areas, including education [5]. To restrain the effect of the pandemic to the student population and address the increasing educational inequalities, the digitalization of education should be promoted. The possibilities for digitalization should be maximized to make content, technological resources, and services accessible to everyone [6]. The possibilities for digitalization refer both to the technological infrastructure and to the development of people’s digital competences through appropriate teaching methodologies. The educational institutions should be equipped with resources to recover the learning lost during the pandemic [7]. It is advisable to provide teachers and students with home-based equipment, carry out teacher training, develop massive online education, and support academic research in online education [8]. Apart from the technological infrastructure, the improvement of the teachers’ digital competence, through professional development, is decisive in the process of the digitalization of education [9]. Successful distance learning depends on teachers’ and parents’ computer skills [4]. A higher level of digital competence would enable educators to contribute to a sustainable digitalization of the field of education and promote inclusiveness, providing equal opportunities to their students.
The present research investigates Teacher Digital Competence (TDC) in an Educational Robotics (ER) teacher training course from the point of view of sustainable development, linking TDC to SDG4. SDG4 points out the importance of ensuring inclusive and equitable quality education and promotes lifelong learning opportunities for all [2]. The link of TDC to SDG4 and the integration of the SDGs in education will be further explained in the following two subsections.

1.1. Teacher Digital Competence

The COVID-19 pandemic brought to the surface several technology- and education-related challenges. The teachers’ job is undergoing tremendous changes [10]: teaching and assessment duties require the development of a set of competences that are completely different from the traditional ones, and teachers need to adapt to the new standards and modalities of teaching [11]. The role of the teacher during the COVID-19 pandemic is of crucial importance, as teachers can contribute to a sustainable digitalization of the education sector and provide equal opportunities to their students, addressing SDG4, on inclusiveness and equity of education. However, to achieve that, teachers need to have a good level of both citizen and teacher digital competences, including knowledge regarding digital content creation, assessment, data security, privacy, collaborating, sharing, and fixing problems. In times of crisis, in particular, it is of crucial importance that online educators have both pedagogical and technical competences [12].
Before the COVID-19 pandemic, the European education systems had been progressively mapping the necessary competences in terms of what a teacher should know and be able to do, and this effort has led to the development of teacher competence frameworks [13]. A teacher competence framework is a collection of statements about what teachers as professionals should know, understand, and be able to do, and which values and attitudes they should have [13]. EU countries have developed various teacher competence frameworks through which teachers’ level of digital competence is evaluated. In the case of Spain, a framework was created for the evaluation of TDC which suggests the evaluation of teachers’ ICT skills in the following areas [14]: 1. Didactic, curricular, and methodological; 2. Planning, organization, and management of digital technological spaces and resources; 3. Relational, ethics, and security; and 4. Personal and professional.
According to [15] (p. 18), “the algorithmic and programming thinking are basic digital competences for the development of the new world economic order in terms of work, social relations and sustainability”. Algorithmic and programming thinking are part of the TDC, and teachers should acquire these skills through teacher education. This is also reflected in the DigComp 2.0 framework [16], which addresses the current demand for citizens to have a better understanding of programming and coding. In DigComp 2.0, programming refers to planning and developing a sequence of understandable instructions for a computing system, a competence that all citizens should acquire. Based on the above, the present research study reports on an ER teacher training course designed to enable future teachers to develop their algorithmic and programming thinking as part of their teacher digital competence, in relation to the SDGs.

1.2. The SDGs in Higher Education and Teacher Training

The 2030 Agenda for Sustainable Development, adopted by all United Nations Member States in 2015, provides a shared blueprint for peace and prosperity for people and the planet, now and into the future [2]. The SDGs recognize that ending poverty and other deprivations must comport with strategies that improve health and education, reduce inequality, and prompt economic growth—while confronting climate change and protecting the natural habitat [2]. Education is central to the achievement of the SDGs: there is an exclusive-to-education sustainable goal (SDG4), and, more importantly, education is linked to all the SDGs in one way or another.
Higher education plays an important role in the achievement of SDGs [17]. Different authors point out that higher education should educate students in sustainability by addressing the SDGs in all university programs [18,19]. Apart from the curriculum, [20] supports that sustainability should become an integral part of higher education policies, and should be reflected in pedagogies, practices, and activities. The integration of SDGs in higher education teaching is a matter of concern for current research and is currently being examined in different countries [21,22]. Apart from higher education, several studies examine the SDGs being integrated in teacher education and initial teacher training. Initial teacher training in sustainability is of crucial importance, as future teachers will not be able to provide sustainable development education to their students unless they have themselves the necessary competences related to sustainability [23]. An increasing number of actions have taken place in initial teacher training in sustainability over the latest years. In the research of [24], future primary school teachers were introduced to SDG teaching, in the Didactics of Matter and Energy university course. In addition, in [23], future teachers participated in a cross-disciplinary workshop on Sustainable Food by developing a research project on SDG12, (sustainable food consumption). Current research places educators at the center of SDG research by investigating future primary school teachers’ knowledge of all SDGs, their implications, and responsibility [25], by evaluating the educational model application for future educators’ development of sustainable competences [26], and by studying secondary school teachers’ attitudes toward sustainability [27]. Given the above-mentioned research in teacher education and initial teacher training, research in this area is growing both in terms of quantity and quality. This is opposed to the results of the literature review [28] back in 2017, which considered research in teacher education as small-scale and theoretically weak.

1.3. Technology and Educational Robotics to Address the SDGs

Technology is identified as an important tool for the implementation of SDGs across several fields, including education and teacher training. Technology can influence many of the important components of the global system that need to be addressed to achieve the SDGs set in the UNESCO 2030 Agenda [29]. In our study, technology and, more concretely, Educational Robotics (ER), is viewed as a means to promote the SDGs in the field of education and teacher training. Our study is in line with the views that ER can have a positive impact on reducing the gender gap in the field of ICT, support innovations to improve the industry, and help to develop sustainable cities and societies [30].
Even though limited, there is previous research in the field of ER education and ER teacher training from the perspective of sustainable development [15,30,31]. For instance, students of the Faculty of Education at the University of Seville, in Spain, were trained in ER and computational thinking (CT) in order to reduce the gender digital divide, addressing SDG4 [31]. In [30], ER activities were carried out with students at high risk of Early School Leaving (ESL) in the form of after-school, extracurricular activities designed to reduce the risks of ESL. Aiming to promote the interest of students with a high degree of apathy and demotivation, and with difficulties to continue their studies, the research of [15] made use of Computational Thinking (CT) and ER with students with such profiles. As there is limited research in ER teacher training from the perspective of sustainable development, we decided to implement an ER course addressing SDGs and examine how it reinforces participants’ TDC and instructs them on the integration of SDGs in education.

2. Methods

This research has a qualitative approach—in particular, it has a case study design. The results of this explanatory case study may provide insights into other, similar situations and cases, and therefore assist their interpretation [32]. Both qualitative and quantitative instruments and data analysis methods are implemented. The research instruments employed are presented in Section 2.4. In the following subsections, information on the research context, sample, and training content will be provided.

2.1. Context and Sample

The research took place at the department of Pedagogy of the University of Rovira i Virgili, in the framework of the university course entitled “Design of Educational Resources and Technological Environments for Training”, during the first semester of the academic year 2019–2020. The course is designed for third-year students of Pedagogy, and its purpose is to provide students with tools to design educational resources, collaborative work environments, and virtual teaching-learning environments and apply them correctly to the teaching process. The graduates of Pedagogy may pursue a career in a variety of fields; they may work in pedagogical materials design, in the area of special educational needs, and in non-formal educational institutions. However, the bachelor’s degree in Pedagogy does not enable them to teach in formal education. Graduates of Pedagogy can also work in the business field, in the organization and management of innovation projects and training programs, in the training of trainers, in the management of social services, centers, and public institutions of education, culture, and social welfare, and in educational administration (inspection, management of educational centers, quality assessment services or agencies, educational research-innovation centers).
The sample of our study consisted of 21 university students. The average age of the students was M = 22.14 (SD = 2.65), ranging from 19 to 27 years old. In the sample there were two male students, and the rest of the students were female. Almost half of the students had some teaching experience in non-formal educational contexts as extracurricular activities, private classes, support tuition, and leisure time instruction. Examining our sample closely, six students had previous experience with Educational Robotics (ER) resources. However, all students reported that they would be interested in learning how to apply ER resources in educational contexts.

2.2. Training Description

The ER training took place in the second module of the course; the objective of the second module is to get students familiarized with advanced tools, environments, and strategies. The second module includes instruction in advanced technology for creating new learning scenarios, collaborative projects in geolocation, augmented reality, ER, and programming, design, and use of mobile applications. The ER training consisted of four 4-h sessions which took place from mid-November to mid-December 2019. The aim of the training was twofold: first, to reinforce students’ digital skills by familiarizing them with different ER resources, programming interfaces, and basic programming concepts; and second, to enable them to integrate ER resources in educational contexts by exploring the potential of ER resources as learning tools across disciplines.
In each session, the students were introduced to a new programming or ER resource, and the training took place in a lab format, rather than a lecture. The first session of the ER training was a hands-on introduction to basic programming concepts through Scratch visual programming interface [33]. After the students’ familiarization with Scratch interface and the completion of several programming tasks of graded difficulty, both individually and in pairs, the potential of programming for the students’ acquisition of skills as creative thinking, computational thinking, and collaboration skills was discussed. Teaching methodologies employed in teaching with Scratch, as exploration and experimentation, project-based learning and learning through storytelling, were presented in class together with unplugged, tinkering making, and remixing activities [34]. The second session of the ER training started by building on the students’ prior knowledge of programming with Scratch, by setting a series of more advanced programming tasks for the students to complete individually or in pairs. After completing the advanced Scratch activities, the students were presented to an ER resource, the Edison robot [35]. The students experimented with the Edison visual programming interfaces and discovered in groups its functionalities and potential use in the field of education. Throughout this session, the students completed several activities in groups (e.g., car race, tracking lines, disco party, maze, etc.). In the third session, the Blue-bot resource [36] was presented to the class; the students had the chance to experiment with the robotic toy and its functionalities by carrying out tasks in groups or pairs. They also explored the Blue-bot’s potential use across different areas of the curriculum by being presented to online educational materials and resources developed by the researchers. To be more precise, the teacher/researcher provided the students with an example of an interdisciplinary Blue-bot project; the project aimed at encouraging the scientific-technological vocation, addressing issues such as stereotypes and the role of women scientists throughout history, promoting gender equality, and reinforcing children’s digital competences, knowledge of science, problem-solving, and creative thinking skills. The university students, after having the chance to experiment with this interdisciplinary project, were instructed on the SDGs and were assigned to create a Blue-bot interdisciplinary project on SDGs, in groups, for different educational contexts (e.g., informal education, special needs, etc.). They had to develop an SDGs lesson plan and the accompanying teaching materials for the following class. The students were given topics related to the SDGs; the first group was asked to raise the awareness of lower primary school students to poverty and hunger; the second group had to promote social, financial, and political inclusion in a youth center; the third group had to make students with special needs aware of marine pollution; the fourth group had to explain to kindergarten pupils the importance of protecting the fauna; and the fifth group of students had to instruct primary school pupils on good practices for saving drinking water. The third session was concluded with an expert’s talk on STEM and STEAM Education, skills, activities, and methodologies. In the fourth session of the ER training, the groups presented their Blue-bot SDGs projects in class; each group of students explained the content of their lesson plan and demonstrated the respective teaching materials. The students had the chance to experiment with the teaching materials created by their fellow students and evaluated their work by completing an evaluation rubric developed by the researchers. Upon completion of the presentations and evaluations, the students were introduced to ER and programming applications and experimented with them. They explored the blocks, created animations, and, finally, discussed with their fellow students the positive aspects of the applications and their use.
The researchers drew attention to the pedagogical approaches followed in the ER training sessions, as the training itself could be a model for the students on how to address, as future educators, ER with their own students. The training activities were designed in accordance with the constructivist learning approach. The students were immersed in a constructionist learning student-centered environment [37], meaning that they played an active role in constructing their own knowledge and skills. The teacher played a guidance role; only when the students faced problems did the teacher help them to find a solution. In addition, the training was grounded in the idea of “learning by design”, which is central in the constructivist pedagogy; for instance, the students learned about robotic toys and their application in educational contexts by designing their own SDGs project. The SDGs project was based on the ADDIE model, which consists of five phases: analysis, design, development, implementation, and evaluation [38]. The students analyzed the educational context to which the project was destined, designed and developed the teaching materials, implemented the project during the presentation class with the help of their fellow students, and were given time to self-evaluate their work. The ER training was grounded in a project-based learning approach. This approach encouraged the collaborative hands-on design of ER interdisciplinary projects and the completion of programming challenges throughout all four training sessions. To summarize, the training was developed based on the following principles: (1) To provide students with both technical and pedagogical instruction to enhance their digital competence; (2) To enable students to make an interdisciplinary use of ER, across different areas of studies and different SDGs; (3) To adopt a project-based learning approach, thereby enabling the students to assume responsibility for their own learning; (4) To encourage an open reflection on the projects developed and the pedagogical approaches applied in class.

2.3. Research Questions

  • RQ1: How do students of Pedagogy at the University of Rovira i Virgili perceive the level of their TDC upon the ER training completion?
  • RQ2: To what extent does a module in ER contribute to the development of students’ abilities to design educational projects according to the Sustainable Development Goals (SDG)?

2.4. Research Instruments

The following research instruments were employed in the research study: the COMDID-A questionnaire (see Section 2.4.1), with a template for the students’ lesson plans (see Section 2.4.2), and an evaluation rubric (see Section 2.4.3). The data collection procedure complied with the General Data Protection Regulation—EU (GDPR). The students agreed to participate in this research. To maintain respondents’ anonymity in the COMDID-A questionnaire and ensure data protection, students’ names were deleted from the database and changed to “student n” identifiers.

2.4.1. COMDID-A Questionnaire

In order to measure the self-perception of TDC among pre-service teachers, COMDID-A was used after the ER training was completed. The self-evaluation questionnaire of TDC [14] measures teachers’ perceptions of their own TDC, divided into four factors or dimensions extracted from the same study and based on International and European standards (1. Didactic, curricular, and methodological aspects; 2. Planning, organization, and management of digital technological resources and spaces; 3. Relational aspects, ethics, and security; 4. Personal and professional aspects). The final version includes 22 Likert-scale items (1: totally agree; 10: totally disagree), distributed into the 4 mentioned dimensions. Example of item D3.2: “I promote the access and use of digital technologies by all students with the intention of compensating for inequalities”. The dimensional structure and internal reliability of the COMDID-A was measured using CFA and Cronbach’s alpha (α) coefficient [39], respectively. The goodness of fit of the model was first determined with absolute measures. The total internal reliability of COMDID-A in our sample is good; α = 0.96. Likewise, the reliability of each of the dimensions in our sample is good: D1, α = 0.92; D2, α = 0.78; D3, α = 0.84; D4, α = 0.87.

2.4.2. Students’ Lesson Plans—Template

The students created their project lesson plans based on a template provided by the researchers. The template included the following information: project title, project duration, students’ age, teaching materials required, learning objectives, interdisciplinary project goals, project description, project evaluation, and an annex for Appendix A, photos, etc.

2.4.3. Evaluation Rubric

An evaluation rubric with 8 Likert items was used to assess the following aspects of the lesson plans on a scale of 0 (inadequate) to 3 (very good): sustainable development content, learning objectives, interdisciplinarity, project description, and teaching material design. In the evaluation rubric there was one item referring to sustainable development (C1), two items referring to learning objectives (OA1 and OA2), one referring to interdisciplinarity (I3), three to lesson plan description (DPS1, DPS2, and DPS3), and one to teaching material design (MD1).

3. Results

The results are presented in the following Section 3.1 and Section 3.2, and answer the first (RQ1) and second research question (RQ2), respectively.

3.1. Students’ TDC Analysis

Examining students’ perceptions of their TDC, it was observed that after the completion of the ER training, their TDC was 79.42% (SD = 10.64). This means that the students in our sample consider themselves as digitally competent. However, D1 (Didactic, curricular, and methodological aspects) and D2 (2. Planning, organization, and management of digital technological resources and spaces) are the dimensions in which they are less confident, while in D3 (Relational aspects, ethics, and security) and D4 (Personal and professional aspects) students seem to be more confident (Table 1).
Furthermore, when students were asked whether the university course and the Blue-bot project itself had an impact on their TDC, they reported that both the course and the Blue-bot project were beneficial for the development of their TDC (Subject-TDC = 3.38/4 and Blue-bot-TDC = 3.23/4).

3.2. Sustainable Development Goals in ER Projects

In this section, the evaluation of the Sustainable Development Goals lesson plans developed by the groups of students of Pedagogy is presented. The lesson plans are summarized in Table 2, while a more detailed description is provided in Appendix A. In Figure A1, Figure A2 and Figure A3, the Blue-bot mats of the lesson plans are presented.
The students’ lesson plans are evaluated according to the criteria set on the evaluation rubric (see Section 2.4.3.), using the 360° evaluation method to add objectivity and validity to the assessment. In the 360° evaluation method, three different perspectives are taken into consideration: the students’, the peers’, and the teachers’ perspective. The final score is based on the average of self-evaluation, peer, and teacher evaluation; however, if there is an important deviation between the 360° final score and the teacher evaluation, the final score needs to be recalculated. To be more precise, if the deviation surpasses 10% but remains under 20%, the final score derives from the average of the teacher evaluation and the 360° final score. However, if the deviation exceeds 20%, the final score is based only on the teacher evaluation.
Studying the final evaluation of the lesson plans, it was found that all five groups of students completed the task successfully, with an average score of 7.85 out of 10 (SD = 1.00). Group 1 obtained a lower score compared to the other groups (6.18, SD = 0.6). In the case of the evaluation of group 1, the deviation between the 360° final score and the teacher evaluation exceeded the index of 10% rate but remained under 20%; therefore, the final score was recalculated based on the average of the 360° final score and the teacher evaluation. Examining more closely the evaluation of the lesson plans, there are some aspects in which the students did very well and some others that require improvement (see Table 3). According to the self-evaluation, the groups of students considered that they did very well in particular in the lesson plan description (DPS1 = 3.00, SD = 0), teaching material design (MD1 = 2.80, SD = 0.45), and SDG content (C1 = 2.80, SD = 0.45). In the case of the peer evaluation, the evaluation of the students is still high, though not as high as in the self-evaluation. According to the peer-evaluation, the weakest aspect of the lesson plan was related to the learning objectives (OA2 = 1.95, SD = 0.93). According to the teacher evaluation, the areas that need improvement are the lesson plan description (DPS1 = 1.96, SD = 0.46) and interdisciplinarity (I = 2.08, SD = 0.30), while the aspect of the lesson plan that the teachers considered really good is the Teaching Material Design (MD1 = 2.64, SD = 0.33). Interestingly, if the items’ average is calculated per evaluation perspective, self-evaluation is the highest (2.75, SD = 0.14) and teacher evaluation is the lowest (2.19, SD = 0.22), while peer evaluation is in the middle (2.23, SD = 0.14).
Regarding the evaluation of item C1, that is related to the sustainable development content of the lesson plans, it was observed that all five lesson plans successfully addressed the SDGs. They were rated at 8.15 out of 10 (SD = 0.91). In Table 4, the evaluation of the content of the lesson plans regarding their compliance with the SDGs is presented from the three different evaluation perspectives: self-evaluation, peer-evaluation, and teacher evaluation. As seen in Table 4, the groups of students considered that their lesson plans complied with the SDGs’ content. However, group 1 evaluated the content of the lesson plan with 2 out of 3, meaning that they considered that their lesson plan was linked to sustainable development, but they should have explained that more clearly. The average of the self-evaluation scores is 2.80 (SD = 0.45) and the average of peer-evaluation is 2.35 (SD = 0.49), while the average of teacher evaluation is 2.40 (SD = 0.35), slightly higher than the peer evaluation. Examining closely the content of the lesson plans and their relation to the SDGs (see Table 4), group 4 received the highest teacher evaluation, which coincided with the self-evaluation; however, group 4 also received the lowest peer evaluation. Interestingly, in the case of group 1, the self-evaluation was slightly lower than the teacher evaluation. As observed in both tables (Table 3 and Table 4), self-evaluation, peer-evaluation, and teacher evaluation do not always coincide in the evaluation of the sustainable development content of the lesson plans.

4. Discussion

The COVID-19 pandemic could be an opportunity for transitioning to a more sustainable system of supply and production [40], for urging the practice of digital sustainability [5], and for promoting a more sustainable education. This could be the right moment to make reforms in the education system. Students should develop skills such as problem-solving, critical thinking, and, most importantly, adaptability to survive any crisis [12]. The pandemic could be an opportunity to improve students’ digital competence and help them to become productive learners and responsible citizens in the virtual world [41]. However, not only students, but also educators in formal and non-formal education, should develop their skills together with their digital competence and TDC to facilitate the process of the digitalization of education.
Based on the above, our research contributes to the transition to a more sustainable education, as it supports the TDC development of students of Pedagogy through an ER teacher training course and studies their ability to integrate SDGs in ER projects. Given the fact there is little or no teaching of programming and robotics in schools in Spain [31], our research attempts to address this deficiency by providing future educators with training in ER through a course at the University of Rovira i Virgili in Spain from the perspective of sustainable development.
In this study, we draw the conclusions based on the first research question (RQ1), “How do students of Pedagogy at the University of Rovira i Virgili perceive the level of their TDC upon the ER training completion?”. The training provided students of Pedagogy with the chance to practice their teacher digital skills and to get familiarized with different ER resources, programming interfaces, and basic programming concepts. After completing the training, future educators’ digital competence was measured, and it was found that they perceive themselves as digitally competent. In line with our findings is the research of [42], who found out that future teachers’ TDC improved after attending an ER intervention as part of a university course. In our research, future educators perceive themselves as more competent in Dimension 3 of the COMDID-A questionnaire (Relational aspects, ethics, and security) and Dimension 4 (Personal and professional aspects). Even though our results are promising, they cannot be generalized, as students’ positive perceptions of their TDC could be due to other factors, not exclusively related to the course.
Regarding the second research question (RQ2), “To what extent does a module in ER contribute to the development of students’ abilities to design educational projects according to the Sustainable Development Goals (SDG)?”, we drew a number of different conclusions. First, upon completion of the training, students of Pedagogy were able to use ER resources in order to create interdisciplinary projects for different educational contexts, from the perspective of different SDGs (SDG1,6,10,14,15), regarding poverty, water consumption, inequalities, marine sources, and the protection of forests. The training adopted a holistic approach [43], enabling future teachers to work on sustainability in a cross-curricular manner in their university projects and in their future teaching. Our results are in line with the results of [23], who found that project-oriented learning supports the development of future teachers’ competencies in sustainability. In our study, the projects created by the students received a positive evaluation. The students were able to develop a lesson plan integrating ER with SDGs. The objectives were clearly set in all lesson plans, while different evaluation methods were chosen by each group of university students to assess the learning outcomes. The evaluation methods included evaluation activities, direct observation, and reflection activities. All lesson plans set indicators for assessment apart from group 1. The projects created by the future educators received a positive evaluation by the teachers, peers, and the future educators themselves (see Section 3.2. However, self-evaluation, peer-evaluation, and teacher evaluation do not always coincide in the evaluation of the lesson plans. Despite this, we consider that the 360° evaluation is very enriching in our context, as students are encouraged to reflect on their own work and their peers’, practicing in this way their critical thinking and reasoning skills. Interestingly, self-evaluation tends to be higher than peer-evaluation and teacher evaluation in most cases. Regarding the evaluation of the sustainable development content of the lesson plans, students of Pedagogy successfully addressed the SDGs in their lesson plans (see Section 3.2). In item C1, self-evaluation is higher than both peer-evaluation and teacher evaluation, except for the case of group 1, whose self-evaluation was slightly lower than the teacher evaluation.
To further investigate students’ level of TDC after a training course, future research in ER integration in SDGs should set a pre and post TDC evaluation tool, apart from the self-perception tool employed in this study. The use of this evaluation tool before and after the training course will observe whether important changes in students’ TDC occurred throughout the training and will improve the reliability and generalizability of the results. Further study is also suggested if indeed future educators perceive themselves as more competent in Dimension 3 (Relational aspects, ethics, and security) and Dimension 4 (Personal and professional aspects) and investigate why this occurs. Based on the literature reviewed and on the results of this case study, we believe that SDGs could be integrated in teaching and learning through Educational Robotics (ER). Given the limited research in the field of ER education and ER teacher training from the perspective of sustainable development, we recommend that further research take place in this field, as technology can facilitate the integration of SDGs into the learning and teaching process. The present study could be further extended by conducting additional sessions, introducing participants to a wider variety of resources, and asking from the participants to develop supplementary projects related to SDGs in pairs or individually. Finally, we believe that future research should contribute to the transition to a more sustainable education. Efforts should be made to reduce the digital divide and provide students and teachers with the chance to improve their digital skills and TDC, respectively, addressing SDG4.

Author Contributions

All authors contributed extensively to the work presented in this paper and approved the final manuscript. V.E.-G., M.U., and D.S. contributed to the study design and methodology. D.S. analyzed and interpreted data. V.E.-G., M.U., J.-L.L.-C., and D.S. contributed to the design of the data collection instruments and D.S. wrote the manuscript as part of her PhD thesis, with significant input from the rest of the authors. V.E.-G., M.U., J.-L.L.-C., and M.G. reviewed, edited, and approved the final manuscript. M.G. was in charge of the supervision, project administration, and funding acquisition. All authors have read and agreed to the published version of the manuscript.

Funding

This project has received funding from the European Union’s Horizon 2020 research and innovation program under the Marie Skłodowska-Curie grant agreement no. 713679. This paper has been possible with the support of the Universitat Rovira i Virgili (URV) and Banco Santander. This project has also received funding from Observ@comdid: grant agreement no. RTI2018-096815-B-I00 and ACEDIM: grant agreement no. 2017ARMIF00031.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A. Lesson Plan Description

Lesson Plan 1: The aim of the first group’s lesson plan is to make primary school pupils aware of poverty, hunger, and economic differences among countries, related to SDG1. The lesson starts with an open discussion as a warm-up activity. Then, the element of storytelling is introduced; stories of children living in developing countries are narrated by the teacher. Based on these stories, the pupils later guide the Blue-bot robotic toy around the Blue-bot mat (see Figure A1) to the item each child mostly needs. The activity on the Blue-bot mat functions itself as an evaluation activity of students’ comprehension skills and critical thinking. However, this is not clearly explained in the lesson plan description, while the activity does not set any evaluation indicators in the lesson plan.
Figure A1. Blue-bot mats of lesson plans 1 and 2.
Figure A1. Blue-bot mats of lesson plans 1 and 2.
Sustainability 12 10085 g0a1
Lesson Plan 2: The aim of the second group’s lesson plan is to promote the social, economic, and political inclusion of all peoples, regardless of age, gender, disability, race, ethnicity, origin, religion, or economic situation, among teenagers in a youth center. This lesson plan is associated with SDG10, as it clearly sets the following learning objectives: practicing critical thinking skills toward situations of inequality in society and detecting situations of inequality in all areas (age, sex, disability, ethnicity, origin, religion, economic situation, etc.). The lesson starts with an introduction to fundamental human rights. Then, the students use the Blue-bot robotic toy to go through a maze with obstacles (see Figure A1). When the teenagers encounter an obstacle, they discuss the socio-economic barrier it represents (gender identity, racism, political ideology, etc.). The lesson continues with a writing activity; the teenagers write about a relevant situation they have experienced, and the moderator of the activity reads the stories to the rest of the group without disclosing the writers’ identity. The lesson concludes with an evaluation activity in which the teenagers watch short videos and identify situations of inequality. This activity, together with the moderator’s observation reports, contributes to the students’ evaluation. The evaluation activity is directly linked to the lesson plan’s learning objectives.
Lesson Plan 3: The objective of the third lesson plan is to raise awareness among special education primary school students about marine pollution, associated with the SDG14. The lesson is designed to take place as an extracurricular activity, and for this reason the university students planned a card game and designed a Blue-bot mat, like a board game (see Figure A2). The participants are divided in two teams. Each team rolls a colorful dice and then programs the robot to move to the nearest square of the designated dice color. After programming the robot to reach the correct square, the team gets a card of the designated dice color and answers a question related to marine pollution. The activity moderator evaluates students’ performance in robot programming and knowledge of marine pollution through observation.
Figure A2. Blue-bot mats of lesson plans 3 and 4.
Figure A2. Blue-bot mats of lesson plans 3 and 4.
Sustainability 12 10085 g0a2
Lesson Plan 4: The lesson plan of the fourth group aims at making kindergarten pupils aware of the importance of protecting the forests, associated with SDG15. This lesson aims at enhancing pupils’ knowledge of the natural environment, developing an understanding of environmental issues, and forming an environmental ethic of their own to reduce their ecological footprint. The lesson is divided into three parts. In the first part, the moderator reads a forest-related tale to the pupils using puppets. The Blue-bot is used in the second part of the lesson in which the students guide the Blue-bot around the mat (see Figure A2) to collect items that pollute the forest (e.g., glass bottles, cans, etc.). Then, a reflection activity on the learning process take place. The moderator of the activity evaluates if the objectives of the lesson were met through an evaluation rubric.
Lesson Plan 5: The lesson plan of the fifth group of university students aims at making 1st-year primary school pupils aware of the importance of saving water, associated with SDG6. The university students approach the objectives of the lesson by creating a board game; they use Blue-bot as the game piece to move around the Blue-bot mat and reach the ending point. (see Figure A3). Depending on the color of the square, the pupils need to perform a different challenge related to water consumption (e.g., mime, draw, answer a question etc.) or move a square back. Upon completion of the activity, an open discussion will follow on eco-friendly habits and the importance of water. The moderator evaluates the students’ performance and knowledge by completing an observation report for each pupil assessing their teamwork skills, initiative, interest, motivation, and interdisciplinary knowledge (e.g., environmental knowledge, language skills, technological skills, etc.).
Figure A3. Blue-bot mats of lesson 5.
Figure A3. Blue-bot mats of lesson 5.
Sustainability 12 10085 g0a3

References

  1. Khetrapal, S.; Bhatia, R. Impact of COVID-19 pandemic on health system & Sustainable development goal 3. Indian J. Med. Res. 2020, 151, 395–399. [Google Scholar] [PubMed]
  2. United Nations Organization. Transforming Our World: The 2030 Agenda for Sustainable Development; UN: New York, NY, USA, 2015; Available online: https://sustainabledevelopment.un.org/content/documents/21252030%20Agenda%20for%20Sustainable%20Development%20web.pdf (accessed on 12 November 2020).
  3. Nguyen, L.H. Risk of COVID-19 among front-line health-care workers and the general community: A prospective cohort study. Lancet Public Health 2020, 5, 361. [Google Scholar] [CrossRef]
  4. United Nations Organization. The Sustainable Development Goals Report; UN: New York, NY, USA, 2020; Available online: https://unstats.un.org/sdgs/report/2020/ (accessed on 12 November 2020).
  5. Pan, S.L.; Zhang, S. From fighting COVID-19 pandemic to tackling sustainable development goals: An opportunity for responsible information systems research. Int. J. Inf. Manag. 2020, 55. [Google Scholar] [CrossRef] [PubMed]
  6. Fundación, T. A Manifesto for a New Digital Deal; Fundación Telefónica: Madrid, Spain; Barcelona, Spain, 2020; Available online: https://www.telefonica.com/digital-manifesto/ (accessed on 12 November 2020).
  7. Burgess, S.; Sievertsen, H.H. Schools, skills, and learning: The impact of COVID-19 on education. VoxEu. Org. 2020. Available online: https://voxeu.org/article/impact-covid-19-education (accessed on 12 November 2020).
  8. Zhang, W.; Wang, Y.; Yang, L.; Wang, C. Suspending classes without stopping learning: China’s education emergency management policy in the COVID-19 outbreak. J. Risk Financ. Manag. 2020, 13, 55. [Google Scholar] [CrossRef] [Green Version]
  9. Kalaš, I.; Laval, E.; Laurillard, D. ICT in Primary Education: Analytical Survey, Volume 1: Exploring the Origins, Settings and Initiatives; UNESCO Institute for Information Technology in Education: Moscow, Russian, 2012. [Google Scholar]
  10. Flores-Lueg, C.B.; Roig-Vila, R. Competencia digital docente: Una cuestión clave para la educación del siglo XXI. In Instructional Strategies in Teacher Training; Gómez-Galán, J., López-Meneses, E., Molina, L., Eds.; UMET Press: San Juan, Puerto Rico, 2016; pp. 87–98. [Google Scholar] [CrossRef] [Green Version]
  11. Khan, R.A.; Jawaid, M. Technology Enhanced Assessment (TEA) in COVID 19 Pandemic. Pak. J. Med. Sci. 2020, 36, S108–S110. [Google Scholar] [CrossRef]
  12. Dhawan, S. Online learning: A panacea in the time of COVID-19 Crisis. J. Ed. Tech. Syst. 2020, 49, 5–22. [Google Scholar] [CrossRef]
  13. European Commission/EACEA/Eurydice. Digital Education at School in Europe. Eurydice Report; Publications Office of the European Union: Luxembourg, 2019; Available online: https://op.europa.eu/en/publication-detail/-/publication/d7834ad0-ddac-11e9-9c4e-01aa75ed71a1/language-en (accessed on 12 November 2020).
  14. Lázaro, J.L.; Gisbert, M. Elaboración de una rúbrica para evaluar la competencia digital del docente. Univ. Tarrac. 2015, 1, 30–47. [Google Scholar] [CrossRef] [Green Version]
  15. Díaz-Lauzurica, B.; Moreno-Salinas, D. Computational thinking and robotics: A teaching experience in compulsory secondary education with students with high degree of apathy and demotivation. Sustainability 2019, 11, 5109. [Google Scholar] [CrossRef] [Green Version]
  16. Redecker, C. European Framework for the Digital Competence of Educators: DigCompEdu; Punie, Y., Ed.; Publications Office of the European Union: Luxembourg, 2017. [Google Scholar] [CrossRef]
  17. Owens, T.L. Higher education in the sustainable development goals framework. Eur. J. Educ. 2017, 52, 414–420. [Google Scholar] [CrossRef]
  18. Leal Filho, W.; Shiel, C.; Paço, A.; Mifsud, M.; Ávila, L.V.; Brandli, L.L.; Molthan-Hill, P.; Pace, P.; Azeiteiro, U.M.; Vargas, V.R.; et al. Sustainable Development Goals and sustainability teaching at universities: Falling behind or getting ahead of the pack? J. Clean. Prod. 2019, 232, 285–294. [Google Scholar] [CrossRef] [Green Version]
  19. SDSN Australia/Pacific. Getting Started with SDGs in Universities: A Guide for Universities, Higher Education Institutions, and the Academic Sector; Australia, New Zealand and Pacific Edition Sustainable Development Solutions Network: Melbourne, Australia, 2017; Available online: http://ap-unsdsn.org/wp-content/uploads/University-SDG-Guide_web.pdf (accessed on 12 November 2020).
  20. Evans, N. Teacher Education and education for sustainability. In Learning to Embed Sustainability in Teacher Education. SpringerBriefs in Education; Springer: Singapore, 2019; pp. 7–21. [Google Scholar] [CrossRef]
  21. Albareda-Tiana, S.; Vidal-Raméntol, S.; Fernández-Morilla, M. Implementing the sustainable development goals at University level. Int. J. Sustain. High Educ. 2018, 19, 473–497. [Google Scholar] [CrossRef]
  22. Denby, L.; Rickards, S. An approach to embedding sustainability into undergraduate curriculum: Macquarie university, australia case study. In Teaching Education for Sustainable Development at University Level. World Sustainability Series; Leal Filho, W., Pace, P., Eds.; Springer: Cham, Switzerland, 2016; pp. 9–33. [Google Scholar] [CrossRef]
  23. Albareda-Tiana, S.; Vidal-Raméntol, S.; Pujol-Valls, M.; Fernández-Morilla, M. Holistic approaches to develop sustainability and research competencies in pre-service teacher training. Sustainability 2018, 10, 3698. [Google Scholar] [CrossRef] [Green Version]
  24. Zamora-Polo, F.; Sánchez-Martín, J. Teaching for a better world. sustainability and sustainable development goals in the construction of a change-maker university. Sustainability 2019, 11, 4224. [Google Scholar] [CrossRef] [Green Version]
  25. Vernia-Carrasco, A.; Pastor-Fuentes, V.; López-Navarro, M. Preliminary Study on the Awareness of the SDGs in Future Primary School Teachers; Editorial Universitat Politècnica de València: Valencia, Spain, 2020. [Google Scholar] [CrossRef]
  26. Vega-Marcote, P.; Varela-Losada, M.; Álvarez-Suárez, P. Evaluation of an educational model based on the development of sustainable competencies in basic teacher training in Spain. Sustainability 2015, 7, 2603–2622. [Google Scholar] [CrossRef] [Green Version]
  27. Agirreazkuenaga, L. Embedding sustainable development goals in education. teachers’ perspective about education for sustainability in the basque autonomous community. Sustainability 2019, 11, 1496. [Google Scholar] [CrossRef] [Green Version]
  28. Evans, N.; Stevenson, R.B.; Lasen, M.; Ferreira, J.; Davis, J. Approaches to embedding sustainability in teacher education: A synthesis of the literature. Teach. Teach. Educ. 2017, 63, 405–417. [Google Scholar] [CrossRef] [Green Version]
  29. Walsh, P.P.; Murphy, E.; Horan, D. The role of science, technology and innovation in the UN 2030 agenda. Technol. Forecast. Soc. Change 2020, 154. [Google Scholar] [CrossRef]
  30. Daniela, L.; Strods, R. Educational Robotics for Reducing Early School Leaving from the Perspective of Sustainable Education. In Smart Learning with Educational Robotics; Daniela, L., Ed.; Springer: Cham, Switzerland, 2019; pp. 43–61. [Google Scholar] [CrossRef]
  31. Román-Graván, P.; Hervás-Gómez, C.; Martín-Padilla, A.H.; Fernández-Márquez, E. Perceptions about the use of educational robotics in the initial training of future teachers: A Study on steam sustainability among female teachers. Sustainability 2020, 12, 4154. [Google Scholar] [CrossRef]
  32. Cohen, L.; Manion, L.; Morrison, K.R. Research Methods in Education; Routledge: New York, NY, USA, 2007; ISBN 0-203-02905-4. [Google Scholar]
  33. Lifelong Kindergarten Group at the MIT Media Lab, Scratch. Available online: https://scratch.mit.edu/ (accessed on 12 November 2020).
  34. Kotsopoulos, D.; Floyd, L.; Khan, S.; Namukasa, I.K.; Somanath, S.; Weber, J.; Yiu, C. A pedagogical framework for computational thinking. Digit. Exp. Math. Educ. 2017, 3, 154–171. [Google Scholar] [CrossRef]
  35. Microbric Pty Ltd, Edison. Available online: https://meetedison.com/ (accessed on 12 November 2020).
  36. TTS International Schools, Bluebot. Available online: https://www.tts-international.com/blue-bot-bluetooth-programmable-floor-robot/1015269.html (accessed on 12 November 2020).
  37. Papert, S. Mindstorms: Children, Computers, and Powerful Ideas; Basic Books: New York, NY, USA, 1993; ISBN 0465046746. [Google Scholar]
  38. Allen, W.C. Overview and evolution of the ADDIE training system. Adv. Dev. Hum. Resour. 2006, 8, 430–441. [Google Scholar] [CrossRef]
  39. Dunn, T.J.; Baguley, T.; Brunsden, V. From alpha to omega: A practical solution to the pervasive problem of internal consistency estimation. Br. J. Psychol. 2014, 105, 399–412. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  40. Sarkis, J.; Cohen, M.J.; Dewick, P.; Schröder, P. A brave new world: Lessons from the COVID-19 pandemic for transitioning to sustainable supply and production. Resour. Conserv. Recycl. 2020, 159. [Google Scholar] [CrossRef] [PubMed]
  41. Zhao, Y. Tofu Is Not Cheese: Rethinking Education Amid the COVID-19 Pandemic. ECNU Rev. Ed. 2020, 3, 89–203. [Google Scholar] [CrossRef]
  42. Esteve-Mon, F.M.; Adell-Segura, J.; Llopis Nebot, M.A.; Valdeolivas Novella, G.; Pacheco Aparicio, J. The development of computational thinking in student teachers through an intervention with educational robotics. J. Inf. Tech. Ed. Innov. Pract. 2019, 18, 139–152. [Google Scholar] [CrossRef] [Green Version]
  43. UNESCO. Education for Sustainable Development Goals. Learning Objectives; UNESCO: Paris, France, 2017; Available online: http://unesdoc.unesco.org/images/0024/002474/247444e.pdf (accessed on 12 November 2020).
Table 1. TDC results.
Table 1. TDC results.
n = 13D1D2D3D4Subject TDCBlue-Bot TDC
Mean75.61574.00084.76983.3083.3853.231
Std. Deviation13.40710.7399.6108.8170.5060.439
Min.58.00056.00066.00070.0003.0003.000
Max.100.00094.00096.00095.0004.0004.000
Table 2. Summary of the SDGs in Students’ Lesson Plans.
Table 2. Summary of the SDGs in Students’ Lesson Plans.
Link to SDGsSDGs in the Lesson ContentSDGs in Evaluation Activities
Lesson plan 1SDG1—End poverty in all its forms everywhere.The lesson plan aims at making primary school pupils aware of poverty and hunger in developing countries.The Blue-bot activity functions as an evaluation activity of students’ comprehension and critical thinking skills regarding the living standards in developing countries.
Lesson plan 2SDG10—Reduce inequality within and among countries.This lesson introduces teenagers to the fundamental human rights and promotes the discussion of potential socio-economic barriers as gender identity, racism, political ideology, etc.The students’ evaluation is accomplished through an activity in which the students watch short videos and identify situations of inequality.
Lesson plan 3SDG14—Conserve and sustainably use the oceans, seas, and marine resources for sustainable development.This lesson raises awareness among special education primary school students to marine pollution.The students’ performance in robot programming and knowledge in marine pollution is evaluated through observation.
Lesson plan 5SDG15—Protect, restore, and promote the sustainable use of terrestrial ecosystems, sustainably manage forests, combat desertification, halt and reverse land degradation, and halt biodiversity loss.This lesson makes kindergarten pupils conscious of the importance of protecting the forests.The pupils are evaluated through a reflection activity on forest pollution.
Lesson plan 5SDG6—Ensure the availability and sustainable management of water and sanitation for all.The lesson raises awareness among primary school pupils on saving water by implementing a board game that includes different challenges related to water consumption.The students’ skills and knowledge are evaluated through an observation report.
Table 3. Lesson Plan Evaluation per evaluation perspective.
Table 3. Lesson Plan Evaluation per evaluation perspective.
EvaluationSDG ContentLearning ObjectivesInterdisciplinarityLesson Plan DescriptionTeaching Material Design
C1OA1OA2I1DPS1DPS2DPS3MD1
Self2.80
(SD = 0.45)
2.60
(SD = 0.55)
2.80
(SD = 0.45)
2.60
(SD = 0.55)
3.00
(SD = 0)
2.80
(SD = 0.45)
2.60
(SD = 0.55)
2.80
(SD = 0.45)
Peer2.35
(SD = 0.49)
2.35
(SD = 0.38)
1.95
(SD = 0.93)
2.15
(SD = 0.42)
2.35
(SD = 0.29)
2.15
(SD = 0.60)
2.30
(SD = 0.45)
2.25
(SD = 0.59)
Teacher2.40
(SD = 0.35)
2.20
(SD = 0.37)
2.12
(SD = 0.33)
2.08
(SD = 0.30)
1.96
(SD = 0.46)
2.08
(SD = 0.33)
2.04
(SD = 0.68)
2.64
(SD = 0.33)
Mean2.52
(SD = 0.25)
2.38
(SD = 0.20)
2.29
(SD = 0.45)
2.28
(SD = 0.28)
2.44
(SD = 0.53)
2.34
(SD = 0.40)
2.31
(SD = 0.28)
2.56
(SD = 0.28)
Table 4. SDG content evaluation per evaluation perspective.
Table 4. SDG content evaluation per evaluation perspective.
Self-EvaluationPeer-EvaluationTeacher Evaluation
Group 12.002.00 (SD = 0.00)2.20 (SD = 0.45)
Group 23.003.00 (SD = 0.00)2.40 (SD = 0.55)
Group 33.002.50 (SD = 0.58)2.20 (SD = 0.45)
Group 43.001.75 (SD = 1.26)3.00 (SD = 0.00)
Group 53.002.50 (SD = 1.00)2.20 (SD = 0.45)
Average2.80 (SD = 0.45)2.35 (SD = 0.49)2.40 (SD = 0.35)
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Schina, D.; Esteve-González, V.; Usart, M.; Lázaro-Cantabrana, J.-L.; Gisbert, M. The Integration of Sustainable Development Goals in Educational Robotics: A Teacher Education Experience. Sustainability 2020, 12, 10085. https://doi.org/10.3390/su122310085

AMA Style

Schina D, Esteve-González V, Usart M, Lázaro-Cantabrana J-L, Gisbert M. The Integration of Sustainable Development Goals in Educational Robotics: A Teacher Education Experience. Sustainability. 2020; 12(23):10085. https://doi.org/10.3390/su122310085

Chicago/Turabian Style

Schina, Despoina, Vanessa Esteve-González, Mireia Usart, José-Luis Lázaro-Cantabrana, and Mercè Gisbert. 2020. "The Integration of Sustainable Development Goals in Educational Robotics: A Teacher Education Experience" Sustainability 12, no. 23: 10085. https://doi.org/10.3390/su122310085

APA Style

Schina, D., Esteve-González, V., Usart, M., Lázaro-Cantabrana, J. -L., & Gisbert, M. (2020). The Integration of Sustainable Development Goals in Educational Robotics: A Teacher Education Experience. Sustainability, 12(23), 10085. https://doi.org/10.3390/su122310085

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop