Next Article in Journal
Conceptualizing the Moderating Role of CEO Power and Ownership Concentration in the Relationship between Audit Committee and Firm Performance: Empirical Evidence from Pakistan
Next Article in Special Issue
The Effects of European Recommendations on the Validation of Lifelong Learning: A Quality Assurance Model for VET in Spain
Previous Article in Journal
Data-Driven Analysis on Inter-City Commuting Decisions in Germany
Previous Article in Special Issue
Active Learning as a Beyond-the-Classroom Strategy to Improve University Students’ Career Adaptability
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Systematic Review

Sustainable Learning Environment by Mobile-Assisted Language Learning Methods on the Improvement of Productive and Receptive Foreign Language Skills: A Comparative Study for Asian Universities

by
Mohsen Mortazavi
1,*,
Mahyuddin K. M. Nasution
2,*,
Foad Abdolahzadeh
3,
Mojtaba Behroozi
3 and
Afshin Davarpanah
4,*
1
Department of Computer Education and Instructional Technologies, Eastern Mediterranean University (EMU), Famagusta 99628, Cyprus
2
Fasilkom-TI, Information Technology Department, Universitas Sumatera Utara, Medan 20222, Indonesia
3
Department of Arabic Language and Literature, University of Zabol, Zabol 9861615881, Iran
4
Department of Mathematics, Aberystwyth University, Aberystwyth, Ceredigion SY23 3BZ, UK
*
Authors to whom correspondence should be addressed.
Sustainability 2021, 13(11), 6328; https://doi.org/10.3390/su13116328
Submission received: 24 April 2021 / Revised: 17 May 2021 / Accepted: 25 May 2021 / Published: 2 June 2021

Abstract

:
In this comparative study, we investigated different influential factors of mobile-assisted language learning (MALL) for improving productive and receptive language learning skills. To do this, 100 scientific research papers were selected from the top and high-quality journals based on the implications of MALL in higher education language learning. Eight papers were selected according to the specific criteria to categorize the results based on language skills interpretation and technological concepts. Therefore, after thoroughly understanding proposed methodologies and comparing them properly, underlying assumptions about this phenomenon are elaborated, and holistic and sustainable solutions are provided to address this idea. The results of this analysis indicated that Mobile devices are being utilized extensively in developing countries, with vocabulary being the primary language learning area assisted by technology and yielding satisfactory results. According to the findings of this comparative study, LINE and WhatsApp would be of interest among university students to improve receptively (listening and reading) and productivity (writing and speaking), respectively. Moreover, the technology acceptance model (TAM) would be a prominent option for teachers to adapt their current and future educational programs instead of only in-person teaching to improve students’ learning quality.

1. Introduction

Language education requires efficient foreign language practices due to individuals’ urgent need in their travels to other countries. It has enabled them to adapt to fast-changing and competitive societies [1,2,3]. There is a stunning range of research for academic and educational purposes on learning English. Language education is categorized into four principal steps. These steps are described briefly as communicative competence, linguistic proficiencies, cross-cultural experiences, and multiple literacies [1,2,3,4,5,6]. Communicative competence is defined as the keen competition among individuals with divergent views and accents to communicate in their relations. It is also described as the fluency of other languages [7,8]. Jeong proposed a qualitative study on the profound impact of multimedia-assisted language instruction on improving the communicative competence property of foreign language learners. Multimedia-assisted language learning materials can help students be more motivated and learn how to enhance their perceptional understanding of foreign language learning. He qualitatively demonstrated the effectiveness of multimedia-assisted language learning materials incorporated with instructional strategies to improve their communicative competence in language skills. According to the analyzed results from the provided survey in EFL students’ in one of the Korean universities, as the multimedia-assisted language learning materials were entertaining and more motivated, it can help them improve their communicative competence of foreign language. Therefore, it is recommended by them as an effective tool to have more practical and efficient ways of learning a foreign language [9]. Valeeva et al. developed a theoretical framework method to study the critical impact of different factors on the foreign language communicative competence. They used theoretical methods contained in publication surveys on the various issues that impacted communicative competence and experience of innovative pedagogical frameworks. It is incorporated with empirical methods that consisted of participant observation, a questionnaire, and teaching staff activity in learning issues. According to their model, it is concluded that personal skills and characteristics would be of essential on the providing of perfect conditions in leaning environments to improve communicative competence [10]. Linguistic proficiencies are considered the individuals’ ability to speak skills or perform them [11,12]. Cross-cultural experiences are described as the utilization of culture in terms of language instructions as an outcome [13,14]. It has objectified the classification of knowledge, skills, and the means of process making. Multiple literacies express the cultural or communicative tasks’ understandings of the other languages [15,16].
Learning a foreign language is one of life’s necessities, primarily used as a lingua franca for peoples’ communication with different nationalities in business. Foreign language education (henceforth; FLE) is an area significantly influenced by technological advancements. Nowadays, traditional devices like desktop computers have been replaced with electrical gadgets such as smartphones, equipped with numerous applications. They have developed language learning skills such as grammar, reading, writing, listening, and speaking [16,17,18]. In this paper, a systematic review was based on application of mobile-assisted language learning impact among higher education students to improve productive and receptive language skills. It is revealed that WhatsApp and LINE would be practical tools for improving productive and receptive skills, respectively.
The pervasive force of using mobile phones to obtain a disseminating source of information quickly has rescued students from monotonous class activities with their colorful applications [18,19,20]. Therefore, students could generate most of their innovations using e-learning devices like mobile phones [21]. For example, students can use different websites to use hypertext, video, mp3 files and find papers and other required information [22]. It can also help individuals be more successful in learning processes like learning a foreign language and preparing themselves for international language exams such as TOEFL, IELTS, and IGCSE [23,24,25]. Regarding the development of modern technology, traditional societies have been changed to “mobile societies.” It means that the changes occurring in these societies are a result of the advent of modern technology that can persuade people to change their lifestyle. Hence, people can utilize mobile technologies in a wide range of life aspects such as teaching in virtual classes and distant learning [26,27].
Integration of mobile devices with educational purposes would be a useful tool to provide language learning facilities via several types of equipment such as YouTube, Instagram, etc. [28,29]. Mobile-Assisted Language Learning (MALL) methods can support learning by offering a unique time and place-dependent language learning experience. MALL can provide convenient, real-time, and contextual learning opportunities for students to be more adaptable to the academic environment [30,31,32]. It no longer matters if they are present or absent from the classroom [33]. Indeed, one of the most frequently cited benefits of mobile technology includes its potential applicability to produce an authentic environment. It provides the opportunity to be exposed to real objects and daily-life situations [34,35,36,37]. Mobile devices are considered useful technologies to spread vocabulary learning instead of computer-assisted learning tools regarding their potential applicability for learning purposes [38,39].
Seliaman and Al-Turk evaluated mobile devices’ advantages in foreign language learning by university students in Saudi Arabia. They concluded that since the students were using mobile technology for accessing the Internet, lecture materials, and sharing information, they are useful tools for learning purposes. According to the ComScore report, comparing the number of desktop computers and mobile device ownership between 2007 and 2015, a strong upward trend was observed in mobile device ownership from 400 to 1900 million. The shift has also led to smartphones’ development with more advanced options with appropriate browsers, screens, and applications that provide more compatible platforms for internet use through these devices [40]. Basoglu and Akdemir compared traditional vocabulary learning via mobile phone applications and programs by 60 Turkish undergraduate students. The results indicated that the second approach improved students’ vocabulary learning more than the first approach [41]. Chang and Hsu developed a system for using PDAs that involved an instant translation/annotation/multi-user mode in supporting university students in an intensive reading course. According to the results, most of the students agreed that the system was beneficial and easy to use. However, the students who worked in groups had a higher level of comprehension than those who read individually [42].
Recently, Sung et al. conducted a comprehensive meta-analysis to evaluate mobile devices’ effectiveness for language learning that included 44 articles written and published over the past 20 years. The results indicated that both adults and school children benefited from the MALL and handheld devices such as smartphones were more effective than laptops. The mobile devices were more functional outdoors rather than in restricted environments (e.g., a classroom) as utilization of mobile devices for vocabulary or mixed language skills is more effective than using them in learning discrete skills such as listening [43].
It is noteworthy that the positive effect of technology on language learning is usually taken for granted. For example, El-Mouelhy et al. found that there is no significant difference between the traditional (i.e., paper-based) and technological (i.e., tablet PC-based) approaches in the English reading class. In their study, the electronic reader system developed to assist elementary school students’ EFL reading comprehension was more adequate for high-achieving rather than low-achieving students [44].
In this paper, we aimed to investigate different influential factors of mobile-assisted language learning in foreign language learning among university students. Productive and receptive language learning skills are essential in our systematic review, as they can provide a significant improvement in the students’ skills. Moreover, it can increase their motivation, satisfaction, and self-confidence due to distance learning, and it might be an excellent choice to address their weaknesses after classroom teaching.

2. Purpose and Research Questions

The present study aimed to investigate mobile-assisted Language learning on foreign language learning skills (productive and receptive skills) for higher educational purposes. The following research questions are considered in our paper to be more concise about the analysis discussed mobile-assisted language learning’s efficacy:
-
What are the useful features of mobile-assisted language learning methods in foreign language learning for productive (speaking and writing) and receptive (listening and reading) skills?
-
Do technology acceptance models (TAM) play a vitally essential role for prospective language learning teachers to adopt their educational purposes?
-
Why do students choose each type of application in their future and current targets?
-
What are the differences between the university students’ preferences and perceptions on the selection of mobile-assisted language learning methods?
-
How can the students improve their communicational skills by using MALL?

3. Methodology

To provide a systematic review based on the selected papers and provide coherency between them, we first chose 100 papers from high-quality journals. The top selected papers from 2012–2020 were chosen to provide a reliable comparative analysis among different crucial factors and valuable tools to improve learning foreign language skills. The selection of these papers was based on the higher education concepts among Asian universities. In this part, we set our first criteria to choose papers based on the mobile-assisted language learning (MALL) method for learning foreign languages, especially for productive and receptive language skills. Therefore, 68 papers were selected at this stage.
Next, as review papers did not sufficiently describe the primary purpose of MALL methods for a particular community, 29 papers were rejected. This left 39 papers, and criteria have been considered to choose more relevant papers that can illustrate more efficiently the concept of MALL. These criteria included EFL learners, mobile apps, vocabulary learning, and mobile devices. At this stage, eight papers were selected according to MALL use in Asian universities (see Figure 1). Therefore, foreign language learning methods via various applicable techniques and tools such as electronic gadget devices like mobile devices, mobile applications, etc. were considered. This theory was based on the investigations of Mills et al. to implement a comparative analysis to determine the best method of learning a foreign language [45]. The following criteria would be implemented to select the papers that can be compared together. In this regard, the learning foreign language skills focus was concentrated on higher education students that have dealt with more electronic gadgets and mobile applications through their training.
Selected papers are listed as follows:
  • The impact of mobiles on language learning on English Foreign Language (EFL) university students [46].
  • Foreign language vocabulary learning with mobile technologies [23].
  • Learning ESL vocabulary with smartphones [47].
  • Effects of English spelling learning experience through a mobile LINE APP for college students [48].
  • Prospective English teachers’ ownership and usage of mobile devices as M-learning tools [49].
  • A study of EFL college students’ acceptance of mobile learning [50].
  • Utilization of WhatsApp on mobile learning for YDS foreign language exam held in Turkey [51].
  • Use learning resources as a self-directed technique by Chinese postgraduate EFL learners [52].

Description of the Selected Papers

Mobile devices are considered efficient tools which can be used for language learning. They can help humans improve their productive and receptive language skills by the use of mobile applications. Muhammed investigated the impact of mobile technology on English language learning among EFL university students. A small group of students (N = 20 EFL students) with different English levels and backgrounds from Sulaimani University in Iraq were selected. To perform the qualitative analysis in this study, a focus group discussion (interview) instead of a questionnaire was implemented to virtually eliminate the required time for analyzing the student’s notes. The students appointed to a variety of smartphone applications for English learning which covered the following language areas: English Language Skills (including receptive and productive skills applications), applications that integrated language skills and systems (such as chain of thought), International Test Applications (e.g., TOEFL), and the English language skills applications which were significantly related to reading and listening skills. All the participants used mobile devices to support their learning and understanding in language lectures. All participants reported that smartphones were used as their first priority in foreign language learning issues as some applications have been specifically developed for smartphones. On the other hand, due to the compatibility and portability of smartphones, they can be used as a pedagogical instrument; “mobile learning” or “m-learning.” Thus, they encouraged students to learn English even outside the classroom environment. The results showed that 99% of the participants used mobile phones to develop their English skills using PDF books, radio programs, spell checking, and grammar software. They indicated that the significant influence of smartphones on students’ English language learning among EFL students at Sulaimani University in Iraq [46].
Agca and Özdemir explored multimedia content integrated into language learning materials using 2D barcode technology for vocabulary learning. The participants were 40 students in the age range of 18–21 majoring in ELT at Gazi University who were randomly assigned to control and experimental groups. A determined printed coursebook containing 84 new words were presented to the students using 2D barcodes placed on the book pages in the experimental group for two weeks. The primary purpose of this course book is to increase the required time needed for learning new vocabularies by reducing the physical distance between Microsoft Tags and the course book. Other students were interviewed about their perception of this new experience, applying this new technology in general conditions, and the Microsoft Tag. The participants in both groups were pre-tested and post-tested on an achievement test to measure their vocabulary understanding level after this test. The results indicated that the mobile-assisted language learning environment led to having significant vocabulary development regarding their experimental group tests. The participants believed that the mobile learning environment was valuable and applicable because they could see the word definition and its visual representation (image-based vocabularies). It has caused to improve the student’s curiosity throughout a funny exercise included Microsoft Tag. It has easy access to information, no need to enter data into the device, and usability in any place where the students reported the advantages reported by the students [23].
Wu explored the effectiveness of smartphones in English vocabulary learning in a study which considered 852 English words accompanied by spelling, pronunciation, graphic representation, Chinese equivalent, and the related syntactic information developed as a JAVA application software (Word Learning). The study was conducted for 50 ESL college participants in the age range of 20–23 in Jiujiang University in China assigned to different experimental and control groups. The students were randomly divided into two groups of 25 students. Although the experimental group used the mentioned application via their smartphones, the control group studied their course book. A pre-test and post-test were administered at the beginning and the end of the study to evaluate students’ vocabulary development. Furthermore, they can use Chinese-English dictionaries and consult with the researcher about unknown items too. The test asked students to write down the Chinese equivalents of 100 words selected from the total of 6674 words in the book’s glossary. The results showed that the experimental group had better scores compared with the control group. The reason for these high scores corresponded to the following issues. As the experimental group had explicit and direct access to the word learning application in their smartphones, they were enabled to enhance all the features for their vocabulary learning by visual perceptions. Moreover, they can improve their visual understanding with the actual context of vocabulary learning that can be used in their future activity through watching videos and computer games. Learning new and unknown words can give the students the chance to be more curious about finding the meaning of unknown words in sample tests that has caused to have higher scores. Finally, as the students can use the smartphone application for a longer time (through the whole semester), they enabled to administer longer accumulated time to learn these 852 words which was more inconvenient for control group as they can’t use course book in every situation. It was concluded that visual media could significantly help students learn a foreign language precisely as far as vocabulary is concerned due to the feasibility and accessibility of smartphones in every situations [47].
Shih et al. adopted a blended teaching approach to examine the influence of incorporating ubiquitous learning into an English spelling course. The participants were 29 Chinese college students who received spelling suggestions via the LINE App on their smartphones for six weeks. These teaching classes focus on improving spelling contexts such as syllables, consonants, short vowels, and distinguishing stress. LINE App was selected because of its focus on vocabulary learning, especially spelling. First, four words were put in the LINE application on Mondays, Wednesdays, and Thursdays as an audio file. Students were required to submit their answers in Chinese through their dialogue window. Then, their teacher checked their pronunciations and gave them marks. The difficulty level of the words was increased each week by the teacher, and the students were notified about their performance and their improper mistakes in their pronunciations. Moreover, the students can use a dictionary to check their pronunciations too. The study had a pre-and post-assessment design. Before and after the intervention, the comparison results showed significant differences in three spelling areas: vowel learning, scramble, and syllable learning, apart from the overall improved performance. The participants’ perception about this learning experience was also surveyed, and the results indicated their satisfaction and increased confidence in learning spelling without any fear of new vocabulary learning by listening to vowels, consonants, and syllables. Furthermore, due to their teacher’s instruction, they can resubmit or reattempt to check their pronunciations without fear and more confidence. Therefore, it can be concluded that the LINE application would not be restricted to a limited number of places and learning environments; the students can use their smartphones in each situation.
Oz investigated whether prospective ELT teachers (fourth-year student teachers) at a large state university in Ankara (Turkey) own a mobile device, how they currently utilize mobile technology for learning, how they intend to use it in the teaching profession. The main obstacles of this paper corresponded to mobile phones adopting and incorporating in language learning eras. In this regard, an invitation email was sent to 75 students via email to complete the survey, and 98 students were invited to complete a paper-based questionnaire. Finally, 144 (75.7% male and 24.3% female) English teachers (in the age range of 21–29) participated as volunteers in the study. This study had two different parts. First, a survey was conducted to distinguish the age and gender as the characterization of the teachers. Then, 21 self-report questions were administered as a comprehensive questionnaire concentrating on the teachers’ perspective about their current ownership, usage, and purchase intent of mobile devices and the obstacles impeding the future use of mobile devices for teaching purposes. Most participants preferred to answer the questions in paper-based questionnaire form, while the others preferred to answer the question online. The findings indicated that most teachers already owned a mobile phone or a tablet with connectivity to the internet. Some of them were also intended to buy other devices such as an iPad and e-book reader shortly. Essential functions of mobile phones including sending SMS, taking photos, making phone calls, recording voice messages and searching on the Internet were the most frequent tasks done by the students and teachers. As far as English language learning was concerned, 88.19% of the participants used online dictionaries, while a lower percentage (63%) used their devices for social networking, and only 22.91% used language learning games. The results also indicated that most student-teachers intended to use mobile devices for teaching and learning activities, including course apps. Using mobile devices (e.g., developing course materials), communicating with their peers and colleagues outside the classroom (68.06%), and developing tests. Additionally, the high diversity of applications and administration factors were the biggest impediments to adopting and using these devices for language learning purposes. These findings have suggested that although almost all the student-teachers commonly own mobile devices and frequently use them, they are not yet well-prepared to use such modern technologies for research and educational purposes and are not aware of the many functions of their mobile devices [49].
Chung et al. investigated using mobile technology use for learning English by 84 tertiary levels EFL university students (68 males and 16 females) majoring in engineering who participated in the research and answered a 20-item questionnaire based on Roger’s Innovation Diffusion Theory and Davis’ Technology Acceptance Model [53,54]. The primary purpose of this study was to implement mobile learning in developing countries to identify the environmental factors that affect mobile technology acceptance as a useful tool for learning a foreign language. In this questionnaire, the questions were contained four items in each of the mobile phone’s convenience, self-efficacy, usefulness, intentional utilization, and compatibility. The authors used a 5-point Likert scale to range their decisions from strongly disagree (1) to strongly agree (5) to indicate and organize their statements. Then, to analyze and provide a descriptive statistical comparison, the statistical software SPSS 17 (IBM Company, New York, NY, USA) was used in their study. According to the findings of this study, mobile technology played a significant role in promoting the students’ vocabulary learning and saving more time in learning vocabulary. This corresponds to the impact of mobile devices on the student’s behavioral intentions to use them in their learning. Moreover, cultural issues would be a critical point in administering mobile learning acceptance for daily learning activities instead of the actual learning environment. Regression analysis was performed to provide an appropriate relationship between the variables and their effectiveness on each variable. According to this analysis, it was illustrated that perceived helpfulness, self-capability, perceived readiness to use, and compatibility accounted for 71% of the variance explained in behavioral objectives to use mobile technology in English vocabulary learning [50].
WhatsApp is considered an influential and applicable software used for distance learning, especially in higher education training. It can help students and trainers handle various tasks such as graphical tasks, video conferencing, and audio typesetting. Learning foreign language skills by administering this application would be of interest and importance for language institutions and teachers to improve students’ levels and assign more different tasks offline. Saritepeci et al. proposed a comprehensive study to utilize WhatsApp to learn foreign language skills and educational contexts remotely, especially for YDS’s national academic language exam in Turkey. They conducted their investigations on 29 participants (55.20% male and 44.80% female) with different backgrounds and personal features by the semi-structured interview form. The form consisted of two different sections. In the first part, there are some demographic questions about education level, age, gender, and duration time for using WhatsApp. On the other hand, the second part contained six evaluating questions about the use of WhatsApp in FLA preparations. The age range of the participants is 26–30 (34.5%) and 36–40 (31%). 58.6% of the participants had a master’s degree, 37.9% had a bachelor’s degree, and 3.4% had a Ph.D. degree. The participants used WhatsApp in various periods (44.80% for less than six months, 41.40% for 6 to 12 months, and 13.80% for 12 months). After collecting the data from questionnaires, a content analysis method was used to have a preliminary coding for the revised statements (punctuation, grammatical errors, and fluency). Then, 38 codes were defined for the preliminary questions to be analyzed. They concluded that WhatsApp’s use in learning foreign language skills has indicated that this application could provide instructor-learner interactions, active learning, motivation and satisfaction, and continuous learning. As this exam is structurally based on the reading and grammatical issues, listening and speaking in this application is not hypothesized enough; however, it can be covered by this application for four foreign language learning elements [51].
Zhang and Perez-Paredes investigated the utilization of mobile devices such as Mobile-Assisted Language learning (MALL) on EFL learners for postgraduate Chinese students. They used a mixed-method analysis by combining an online questionnaire (quantitative analysis) and a set of interviews (qualitative analysis) with learning English. The questionnaire consisted of 11 closed-ended and one open-ended (enabled students to propose their viewpoints freely) questions. Moreover, seven close-ended questions based on the demographical information about students were included in the questionnaire. To do this, 95 postgraduate Chinese students (28.4% male and 71.6% female) were chosen from four universities with different backgrounds and learning prospects to participate in a questionnaire. However, eight of these students were preferred to do an interview instead. The percent of students who had passed international language tests is about 5.3%. According to their findings, postgraduate students have indicated that mobile English learning resources (MELR) would be a good option for passing their exams, significantly improving English vocabulary development. These resources contained mobile dictionary applications that are considered the most appropriate supplementary resources. As a result, the duration of using MELR was preferred in a short time (less than one hour or between 1–2 h at least). Therefore, MALL can provide different useful aspects such as short intervals for learning vocabularies. In this regard, vocabulary learning applications and mobile dictionaries would be of interest for postgraduate students and considered the two essentially robust solutions for their learning improvement [52].

4. Results and Discussion

Here, we consider the following criteria to provide a comparative analysis among different learning foreign language skills. The main reasons why students use smartphones for learning English is mainly reported as strengthening their perception about language experiences and their portability, flexibility, and compatibility. It can persuade them to support their learning during lectures and raise their motivation to develop their language skills, especially listening and reading, to activate their schemata of speaking with others. As it is evident, learning vocabularies by images could exercise a dominant influence on the vocabulary memorization and cause to bear them in mind, to familiarize individuals with different aspects of an unknown expression without any sense of monotonous for individuals. One of the chief aims of learning English by mobile phones is related to the vast majority of innovative and funny applications that gave individuals the chance to surpass their intelligence; to serve the purpose of being organized in their responsibilities. We divided the comparisons into two different sections to be more concise about the provided analysis.

4.1. Language Skills Interpretation

First, we consider productive and receptive skills as a separate category to find which methods would be more suitable for each type of skill. Shih has focused on utilizing smartphone Apps (LINE app) to ascertain the perception and effectiveness of college students’ performance on English spelling improvement. Smartphone Apps could help enhance listening skills such as syllables, phonemes, and vowels. According to the student’s opinions, the LINE app would be an excellent choice to improve receptive skills such as listening and reading. In this application, teachers can introduce the number of selected words daily from simple words in the first sessions, and then it can be more difficult words in the next following days. Therefore, students can read and check the spelling of the provided vocabularies in dictionaries to burn them in mind more concisely [48]. As a result, by practicing daily or on a routine program, students can improve their receptive skills individually by spending less time. In Saritepeci et al.’s paper, WhatsApp is introduced as a useful tool for productive skills such as speaking and writing that can be used in online exam preparation. They concluded that teachers could assign several short writing tasks like short essays for students and simultaneously check their reports. It can help them strengthen their productive skills by using and learning different new vocabularies that provide a large word domain for their future study plans. Thereby, WhatsApp is considered a beneficial tool for improving productive skills as teachers and students can adapt it for international language exam preparation. Moreover, this application can reinforce the conducted activities in classrooms. According to their findings, WhatsApp can provide motivation, satisfactory, and continuous learning for language learners that is another positive point of this application for productive skills enhancement among university students [51]. Consequently, it is concluded that LINE and WhatsApp would be two useful applications for receptive and productive language skills, respectively. This is schematically depicted in Figure 2.
Chung et al. reported similar research describing more advantages of using mobile devices among students, significantly influencing vocabulary learning. Hence, such factors include self-efficacy, behavior intention use, and usefulness, considered as the main steps of this investigation due to the consideration of compatibility, portability, and ubiquity of mobile devices on learning English. Investigating the two groups of males and female teachers in mobile devices enhances learning situations. As clarified in the paper, most teachers used mobile devices and their applications in learning regarding social networking with others and language learning by utilizing game applications to encourage students to improve their perceptions and skills concerning current circumstances [50]. In Chung et al.’s paper, students’ intentional behavior on accepting technology models for learning purposes was considered. It can provide a theoretical frame to investigate the usefulness of mobile devices in vocabulary learning. According to the findings of these two studies on the utilization of smartphone applications in foreign language learning, it can be concluded that the technology acceptance model (TAM) would be a prominent option for teachers to adapt their current and future educational program instead of only in-person teaching. This can improve the quality of student’s learning as they might not be attended in some sessions or have some understanding problem due to the high number of students in the classroom, and there is no additional time for all of them to participate in group discussions. On the other hand, according to the students’ perceptions and their targets for their current and plans, they agreed that technology acceptance models would be vital for them to provide a targeting plan in their studying preparations.

4.2. Technology Concepts

Mobile English learning resources (e.g., vocabulary learning applications and mobile dictionaries) would significantly improve productive language skills. In Zhang et al.’s paper, MALL is a supplementary method for learning approaches. Therefore, MALL can provide different useful aspects such as short intervals for learning vocabularies. In this regard, vocabulary learning applications and mobile dictionaries would be of interest for postgraduate students and considered the two essentially robust solutions for their learning improvement [52]. On the other hand, Oz investigated prospective English teachers and their insights on mobile devices’ purchases for learning purposes. Teachers’ intention to use mobile devices to develop material courses and tests for university students is critical for this study. The literature also shows that most researchers tend to address one language skill or evaluate a specific application’s influence than overall language proficiency or achievement [42,49,54,55,56]. It has also confirmed that vocabulary learning is the essential computer-assisted language area. The reason for extensive attention to vocabulary may be that vocabulary is a vital aspect of language learning, to the extent that according to Williams [57]. Concerning the fifth and sixth studies, whereas the former one had a more extensive scope, addressing several research problems (ownership of mobile device, its current and future use for language learning, and identifying the barriers in its use of language learning), the latter one only dealt with the ownership of mobile devices by university students. Because these two studies investigated mobile devices’ ownership and mobile technology acceptance, no specific applications were examined. Consequently, it can be concluded that MALL methods would be of interest, especially for academic educational purposes.
The multimedia content integrated into language learning materials to improve vocabulary learning and students’ perception using 2D barcode technology has been investigated by Wu as the mixed method. Wu divided participants into two groups, experimental and control groups, to analyze the results more precisely. As can be seen from his results, the experimental group students had memorized more words than the opponents; that is to say that environmentally friendly circumstances played a significant role in learning foreign languages [47]. It can be found that mobile learning environment circumstances (e.g., innovative and funny applications) enhance vocabulary learning. On the other hand, smartphones’ and Microsoft Tag’s effect on enhancing students’ vocabulary acquisition were considered. It is implied that technology administration in vocabulary learning would be of interest; however, as it is not accessible by all students, they might be ignored in the academic environment [47].
The eight investigational research papers’ features are descriptively summarized in Table 1. It concentrates on learning English language aspects by utilizing validated references to facilitate the comparative analysis to provide holistic and sustainable solutions [58]. There are contrary opinions about mobile phones, explained by each article’s context, respectively; that is to say that many participants widely mention the importance of mobile devices, especially their powerful applications on nurturing the vocabularies with colorful techniques like images innovative apps. Thereby, it is evident that mobile device apps play a substantial role in fostering a deeper understanding of new experiences and views of new vocabularies and enable users to excel in their tasks and responsibilities more than other participants. The acceptance of mobile technology and ownership of mobile devices as two main factors affecting the integration and use of mobile devices in education is specifically investigated among students to whether they might be useful or not to grasp the importance of each application.

5. Conclusions

Mobile-assisted language learning (MALL) is considered an effective method for improving productive and receptive language skills, especially in higher education users in non-native countries. Thereby, it is necessary to choose homogeneous and relevant papers to discussed more the language skills and technical concepts. According to MALL methods for higher education, the selected papers were chosen from among 100 research papers by focusing on productive and receptive language learning. The findings specifically proved mobile learning effectiveness for vocabulary learning by incorporating audio-visual elements, facilitating students’ comprehension and retention. In other words, in line with more communicative approaches to foreign language teaching and learning, more integrative methodologies should be addressed where the language is viewed holistically, and mobile technology is integrated into foreign language learning as a whole, not merely applied to a single language skill. Despite these studies’ satisfactory results, the acceptance of mobile technology and developing adequate knowledge of these devices’ educational functions appeared to be areas that still require much consideration. The main conclusions of this comparative study are as follows:
-
LINE and WhatsApp would be two useful applications for receptive and productive language skills, respectively.
-
The technology acceptance model (TAM) would be a prominent option for teachers to adapt their current and future educational programs instead of only in-person teaching to improve students’ learning quality.
-
MALL methods would be of interest, especially for academic educational purposes, to improve vocabulary learning.

Author Contributions

M.M.; Writing—original draft, Resources, Formal analysis, Investigation, Methodology. M.K.M.N.; Writing—review & editing. F.A.; Writing—review & editing. M.B.; Writing—review & editing. A.D.; Writing—review & editing, Visualization, Supervision. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

All the data are available in the paper.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

MALLMobile-Assisted Language Learning
TAMTechnology Acceptance Model
FLEForeign Language Education
IELTSInternational English Language Testing System
TOEFLTest of English as a Foreign Language
IGCSEInternational General Certificate of Secondary Education
ICTInformation and Communications Technology
PDAPersonal Digital Assistant
EFLEnglish as a Foreign Language
YDSLanguage Proficiency Test administered in Turkey
2D2 Dimensional
ELTEnglish Language Teaching

References

  1. Byram, M.; Holmes, P.; Savvides, N. Intercultural communicative competence in foreign language education: Questions of theory, practice and research. Lang. Learn. J. 2013, 41, 251–253. [Google Scholar] [CrossRef] [Green Version]
  2. Barnard, R.; Richards, J.C.; Rodgers, T.S. Approaches and Methods in Language Teaching. TESOL Q. 2002. [Google Scholar] [CrossRef]
  3. Granger, S. Learner Corpora in Foreign Language Education. Lang. Technol. 2017. [Google Scholar] [CrossRef]
  4. Liaw, M.L.; Johnson, R.J. E-mail writing as a cross-cultural learning experience. System 2001, 29, 235–251. [Google Scholar] [CrossRef]
  5. Le Hir, M.-P. Remapping the Foreign Language Curriculum: An Approach through Multiple Literacies (review). Composite 2007. [Google Scholar] [CrossRef]
  6. Davarpanah, A.; Shirmohammadi, R.; Mirshekari, B.; Aslani, A. Analysis of hydraulic fracturing techniques: Hybrid fuzzy approaches. Arab. J. Geosci. 2019, 12, 402. [Google Scholar] [CrossRef]
  7. Mohamed, M.N.A.; Ngadiran, N.M.; Samad, N.A.; Powzi, N.F.A. English language learning beyond the borders: Constructing e-collaborative learning between students of different regions. Univ. J. Educ. Res. 2020, 8, 108–113. [Google Scholar] [CrossRef]
  8. Chaisuriya, A.; Shin, S.Y. Examining english test practicality among different stakeholders in Thailand. J. Asia TEFL 2019. [Google Scholar] [CrossRef]
  9. Jeong, K.O. Developing efl learners’ communicative competence through multimedia-assisted language learning. J. Theor. Appl. Inf. Technol. 2018, 96, 1367–1376. [Google Scholar]
  10. Valeeva, R.A.; Baykova, O.V.; Kusainov, A.K. Foreign language professional communicative competence as a component of the academic science teacher’s professional competence. Int. J. Environ. Sci. Educ. 2016, 11, 173–183. [Google Scholar] [CrossRef]
  11. Peets, K.F.; Yim, O.; Bialystok, E. Language proficiency, reading comprehension and home literacy in bilingual children: The impact of context. Int. J. Biling. Educ. Biling. 2019. [Google Scholar] [CrossRef]
  12. Presbitero, A. Foreign language skill, anxiety, cultural intelligence and individual task performance in global virtual teams: A cognitive perspective. J. Int. Manag. 2020, 26, 100729. [Google Scholar] [CrossRef]
  13. Loh, J.M.I.; Thorsteinsson, E.B.; Loi, N.M. Workplace incivility and work outcomes: Cross-cultural comparison between Australian and Singaporean employees. Asia Pac. J. Hum. Resour. 2019, 59, 305–329. [Google Scholar] [CrossRef]
  14. Mogias, A.; Boubonari, T.; Realdon, G.; Previati, M.; Mokos, M.; Koulouri, P.; Cheimonopoulou, M.T. Evaluating ocean literacy of elementary school students: Preliminary results of a cross-cultural study in the Mediterranean Region. Front. Mar. Sci. 2019. [Google Scholar] [CrossRef]
  15. Brownell, C.J. Inventing expert in English language arts: A case study of critical literacies in a third grade classroom. J. Early Child. Lit. 2020. [Google Scholar] [CrossRef]
  16. Zhang, Z.; Heydon, R.; Li, W.; Malins, P. Literacies and identities in transnational education: A case study of literacy curricula in a Canadian transnational education programme in China. Curric. J. 2020. [Google Scholar] [CrossRef] [Green Version]
  17. Macías, D.F. Classroom management in Foreign Language education: An exploratory review. Profile Issues Teach. Prof. Dev. 2018. [Google Scholar] [CrossRef]
  18. Mortazavi, M.; Hocanın, F.T.; Davarpanah, A. Application of quantitative computer-based analysis for student’s learning tendency on the efficient utilization of mobile phones during lecture hours. Sustainability 2020, 12, 8345. [Google Scholar] [CrossRef]
  19. Üstün Aksoy, Y.; Dimililer, Ç. Teacher opinions on usage of mobile learning in pre-school foreign language learning. Eurasia J. Math. Sci. Technol. Educ. 2017, 13, 5405–5412. [Google Scholar] [CrossRef]
  20. Dashtestani, R. Moving bravely towards mobile learning: Iranian students’ use of mobile devices for learning English as a foreign language. Comput. Assist. Lang. Learn. 2016. [Google Scholar] [CrossRef]
  21. Cavus, N.; Ibrahim, D. Learning English using children’s stories in mobile devices. Br. J. Educ. Technol. 2017, 48, 625–641. [Google Scholar] [CrossRef]
  22. Da Silva, A.C.; Freire, F.M.P.; Mourão, V.H.M.; De Oliveira Da Cruz, M.D.; Da Rocha, H.V. Portability and usability of open educational resources on mobile devices: A study in the context of Brazilian educational portals and android-based devices. In Proceedings of the 10th International Conference on Mobile Learning, Madrid, Spain, 28 February–2 March 2014. [Google Scholar]
  23. Agca, R.K.; Özdemir, S. Foreign Language Vocabulary Learning with Mobile Technologies. Procedia Soc. Behav. Sci. 2013, 83, 781–785. [Google Scholar] [CrossRef] [Green Version]
  24. Prinsloo, D.J. Electronic Dictionaries viewed from South Africa. HERMES J. Lang. Commun. Bus. 2017. [Google Scholar] [CrossRef] [Green Version]
  25. Huang, R.T.; Hsiao, C.H.; Tang, T.W.; Lien, T.C. Exploring the moderating role of perceived flexibility advantages in mobile learning continuance intention (MLCI). Int. Rev. Res. Open Distance Learn. 2014. [Google Scholar] [CrossRef]
  26. Ally, M.; Prieto-Blázquez, J. What is the future of mobile learning in education? Int. J. Educ. Technol. High. Educ. 2014, 11, 142–151. [Google Scholar] [CrossRef] [Green Version]
  27. Barbour, M.K.; Grzebyk, T.Q.; Eye, J. Any time, any place, any pace-really? Examining mobile learning in a virtual school environment. Turk. Online J. Distance Educ. 2014. [Google Scholar] [CrossRef] [Green Version]
  28. Andujar, A.; Salaberri-Ramiro, M.S.; Martínez, M.S.C. Integrating Flipped Foreign Language Learning through Mobile Devices: Technology Acceptance and Flipped Learning Experience. Sustainability 2020, 12, 1110. [Google Scholar] [CrossRef] [Green Version]
  29. Abdullah Al-Garawi, A.Y. Investigating the Use of Instagram as a MALL Tool: The Case of Al-Imam University EFL Students. Arab World Engl. J. 2019. [Google Scholar] [CrossRef] [Green Version]
  30. Gonulal, T. The use of instagram as a mobile-assisted language learning tool. Contemp. Educ. Technol. 2019. [Google Scholar] [CrossRef] [Green Version]
  31. Lee, S.M. A systematic review of context-aware technology use in foreign language learning. Comput. Assist. Lang. Learn. 2019. [Google Scholar] [CrossRef]
  32. Sarré, C.; Grosbois, M.; Brudermann, C. Fostering accuracy in L2 writing: Impact of different types of corrective feedback in an experimental blended learning EFL course. Comput. Assist. Lang. Learn. 2019. [Google Scholar] [CrossRef]
  33. Kukulska-Hulme, A.; Shield, L. An overview of mobile assisted language learning: From content delivery to supported collaboration and interaction. ReCALL 2008. [Google Scholar] [CrossRef] [Green Version]
  34. Al Azri, R.H.; Al-Rashdi, M.H. The Effect Of Using Authentic Materials In Teaching. Int. J. Sci. Technol. Res. 2014, 47, 2422–2843. [Google Scholar]
  35. Herrington, J.; Kervin, L. Authentic learning supported by technology: Ten suggestions and cases of integration in classrooms. EMI. Educ. Media Int. 2007. [Google Scholar] [CrossRef] [Green Version]
  36. Shadiev, R.; Hwang, W.Y.; Huang, Y.M.; Liu, T.Y. Cognitive Diffusion Model: Facilitating EFL Learning in an Authentic Environment. IEEE Trans. Learn. Technol. 2017, 10, 168–181. [Google Scholar] [CrossRef]
  37. Abu-Al-Aish, A.; Love, S. Factors influencing students’ acceptance of m-learning: An investigation in higher education. Int. Rev. Res. Open Distance Learn. 2013. [Google Scholar] [CrossRef]
  38. Kennedy, C.; Levy, M. L’italiano al telefonino: Using SMS to support beginners’ language learning. ReCALL 2008. [Google Scholar] [CrossRef] [Green Version]
  39. Zhang, H.; Song, W.; Burston, J. Reexamining the effectiveness of vocabulary learning via mobile phones. Turkish Online J. Educ. Technol. 2011, 10, 203–214. [Google Scholar]
  40. Seliaman, M.E.; Al-Turki, M. Mobile Learning Adoption in Saudi Arabia. World Acad. Sci. Eng. Technol. 2012. [Google Scholar] [CrossRef]
  41. Başoǧlu, E.B.; Akdemir, Ö. A comparison of undergraduate students’ English vocabulary learning: Using mobile phones and flash cards. Turk. Online J. Educ. Technol. 2010, 9, 1–7. [Google Scholar]
  42. Chang, C.K.; Hsu, C.K. A mobile-assisted synchronously collaborative translation-annotation system for english as a foreign language (EFL) reading comprehension. Comput. Assist. Lang. Learn. 2011. [Google Scholar] [CrossRef]
  43. Sung, Y.T.; Chang, K.E.; Yang, J.M. How effective are mobile devices for language learning? A meta-analysis. Educ. Res. Rev. 2015, 16, 68–84. [Google Scholar] [CrossRef]
  44. El-Mouelhy, I.; Poon, I.H.C.; Hui, A.N.N.; Sue-Chan, C. Does a Creative Learning Medium Matter? Impact of Low Cost Android Tablets on Elementary Students’ English Comprehension, Perceived Performance and Memory Retention. Creat. Educ. 2013. [Google Scholar] [CrossRef] [Green Version]
  45. Mills, M.; Van De Bunt, G.G.; De Bruijn, J. Comparative research: Persistent problems and promising solutions. Int. Sociol. 2006. [Google Scholar] [CrossRef] [Green Version]
  46. Muhammed, A.A. The Impact of Mobiles on Language Learning on the Part of English Foreign Language (EFL) University Students. Procedia Soc. Behav. Sci. 2014, 136, 104–108. [Google Scholar] [CrossRef] [Green Version]
  47. Wu, Q. Learning ESL Vocabulary with Smartphones. Procedia Soc. Behav. Sci. 2014, 143, 302–307. [Google Scholar] [CrossRef] [Green Version]
  48. Shih, R.-C.; Lee, C.; Cheng, T.-F. Effects of English Spelling Learning Experience through a Mobile LINE APP for College Students. Procedia Soc. Behav. Sci. 2015, 174, 2634–2638. [Google Scholar] [CrossRef] [Green Version]
  49. Oz, H. Prospective English Teachers’ Ownership and Usage of Mobile Devices as M-learning Tools. Procedia Soc. Behav. Sci. 2014. [Google Scholar] [CrossRef] [Green Version]
  50. Chung, H.-H.; Chen, S.-C.; Kuo, M.-H. A Study of EFL College Students’ Acceptance of Mobile Learning. Procedia Soc. Behav. Sci. 2015. [Google Scholar] [CrossRef] [Green Version]
  51. Saritepeci, M.; Duran, A.; Ermiş, U.F. A new trend in preparing for foreign language exam (YDS) in Turkey: Case of WhatsApp in mobile learning. Educ. Inf. Technol. 2019. [Google Scholar] [CrossRef]
  52. Zhang, D.; Pérez-Paredes, P. Chinese postgraduate EFL learners’ self-directed use of mobile English learning resources. Comput. Assist. Lang. Learn. 2019. [Google Scholar] [CrossRef]
  53. Davis, F.D. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q. Manag. Inf. Syst. 1989. [Google Scholar] [CrossRef] [Green Version]
  54. Mishra, V.M. Diffusion of Innovations, 4th ed.; Free Press: New York, NY, USA, 1996. [Google Scholar]
  55. Steel, C. Fitting learning into life: Language students’ perspectives on benefits of using mobile apps. In Proceedings of the ASCILITE 2012—Annual conference of the Australian Society for Computers in Tertiary Education, Wellington, New Zealand, 23 February 2012. [Google Scholar]
  56. Kuo, Y.C.; Chu, H.C.; Huang, C.H. A learning style-based grouping collaborative learning approach to improve EFL students’ performance in english courses. Educ. Technol. Soc. 2015, 18, 284–298. [Google Scholar]
  57. Belenkova, N.; Kruse, I. Determining Descriptors of Professional Foreign Language Competence for Law-setting Specialists. Procedia Soc. Behav. Sci. 2017, 237, 1398–1404. [Google Scholar] [CrossRef]
  58. Williams, H. Linguistics in language teaching. In The Linguistic Structure of Modern English; John Benjamins Publishing: Amsterdam, The Netherlands, 2010; ISBN 978-9-02728-824-0. [Google Scholar]
Figure 1. Selection procedure of the papers for comparing in this study.
Figure 1. Selection procedure of the papers for comparing in this study.
Sustainability 13 06328 g001
Figure 2. The schematic diagram for the comparison of LINE and WhatsApp in receptive and productive language learning skills.
Figure 2. The schematic diagram for the comparison of LINE and WhatsApp in receptive and productive language learning skills.
Sustainability 13 06328 g002
Table 1. Analysis of eight divergent papers of learning the English language by mobile devices.
Table 1. Analysis of eight divergent papers of learning the English language by mobile devices.
PaperContextParticipantsArticle ScopePurpose ToolsResearch TypeResult (Keypoints)
1IraqUniversity EFL students
(N = 20)
The influence of mobile technology on learning a foreign languageSmartphone applicationsQualitative (Focus group)
-
Significant improvement in productive (Writing and Speaking especially vocabulary use) and receptive (Listening and Reading) skills.
-
Compatibility and feasibility utilization of smartphone applications
2TurkeyFresh and junior university students
(N = 40)
The effect of the multimedia content integrated into language learning materials to improve vocabulary learning and students’ perception2D barcode technologyMixed method
-
Mobile Learning Environment circumstances (e.g., innovative and funny applications) to enhance vocabulary learnings.
3ChinaESL sophomore college students
(N = 50)
The effect of smartphones on learning English vocabularySmartphones and Microsoft TagPre-test and post-test
-
Enhancement of student’s vocabulary acquisition.
-
Smartphones help to improve vocabulary building.
4Taiwancollege students
(N = 29)
The influence of mobile learning using LINE APP on college students’ spellingSmartphone APPs (LINE APP)Pre-assessment, a Post-assessment, and a survey questionnaire
-
Utilization of smartphone Apps to ascertain the perception and effectiveness of college students’ performance on English spelling improvement.
-
Smartphone Apps could help to enhance listening skills such as syllables, phonemes, and vowels.
5TurkeyUniversity students and teachers
(N = 144)
English language-teaching teachers and their enthusiasm and insights to administer smartphones in their teaching environment are an M-learning method.M-learningQuantitative (Questionnaire)
-
Prospective English teachers and their insights on the purchase of mobile devices for learning purposes.
-
Teachers’ intention to implement mobile devices to develop material courses and tests for university students.
6TaiwanJunior university students
(N = 84)
Behavioral intentions of students to accept technology devices as a convenient tool for educational purposes.Mobile APPsQuantitative (Questionnaire)
-
Students’ intentional behavior on the acceptance of technology models in learning purposes.
-
Provide a theoretical frame to investigate the usefulness of mobile devices in vocabulary learning.
7TurkeyUniversity Student
(N = 29)
Application of WhatsApp in educational contexts for foreign language exam (YDS)Mobile Apps (WhatsApp)Qualitative (Semi-structured interview)
-
Reinforce the conducted activities in classrooms by the utilization of WhatsApp
-
WhatsApp can provide motivation, satisfactory, and continuous learning for language learners.
8ChinaPostgraduate EFL university students
(N = 95)
Mobile English learning resources impact the self-efficacy of language learning Mobile-Assisted Language learningMixed-Method (semi-structured interview and questionnaire)
-
Mobile English learning resources (e.g., vocabulary learning applications and mobile dictionaries) would significantly improve language production skills.
-
MALL is known as a supplementary method for learning approaches.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Mortazavi, M.; Nasution, M.K.M.; Abdolahzadeh, F.; Behroozi, M.; Davarpanah, A. Sustainable Learning Environment by Mobile-Assisted Language Learning Methods on the Improvement of Productive and Receptive Foreign Language Skills: A Comparative Study for Asian Universities. Sustainability 2021, 13, 6328. https://doi.org/10.3390/su13116328

AMA Style

Mortazavi M, Nasution MKM, Abdolahzadeh F, Behroozi M, Davarpanah A. Sustainable Learning Environment by Mobile-Assisted Language Learning Methods on the Improvement of Productive and Receptive Foreign Language Skills: A Comparative Study for Asian Universities. Sustainability. 2021; 13(11):6328. https://doi.org/10.3390/su13116328

Chicago/Turabian Style

Mortazavi, Mohsen, Mahyuddin K. M. Nasution, Foad Abdolahzadeh, Mojtaba Behroozi, and Afshin Davarpanah. 2021. "Sustainable Learning Environment by Mobile-Assisted Language Learning Methods on the Improvement of Productive and Receptive Foreign Language Skills: A Comparative Study for Asian Universities" Sustainability 13, no. 11: 6328. https://doi.org/10.3390/su13116328

APA Style

Mortazavi, M., Nasution, M. K. M., Abdolahzadeh, F., Behroozi, M., & Davarpanah, A. (2021). Sustainable Learning Environment by Mobile-Assisted Language Learning Methods on the Improvement of Productive and Receptive Foreign Language Skills: A Comparative Study for Asian Universities. Sustainability, 13(11), 6328. https://doi.org/10.3390/su13116328

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop