A Low-Cost Interactive Writing Board for Primary Education using Distinct Augmented Reality Markers

Abstract

Educational institutions demand cost-effective and simple-to-use augmented reality systems. ARToolKit, an open-source computer tracking library for the creation of augmented reality applications that overlay virtual imagery on the real world, is such a system. It uses a simple camera and black-and-white markers printed on paper. However, due to inter-marker confusion, if the marker distinctions are not ensured, the markers are often miss-recognized. This paper presents an ARToolKit-based Interactive Writing Board (IWB) with a simple mechanism for designing confusion-free marker libraries. The board is used for teaching single characters of Arabic/Urdu to primary level students. It uses a simple ARToolKit marker for the recognition of each character. After marker recognition, the IWB displays the corresponding image, helping students with character understanding and pronunciation. Experimental results reveal that the system improves students’ motivation and learning skills.

1. Introduction

Technology-enhanced learning systems are rapidly growing with amazing improvements in classrooms and educational environments. These systems are significantly improving students’ motivation and learning skills. However, the use of these systems is not so common, due to either their high cost or their complexity in use. Therefore, cost-effective and easy-to-use technology-enhanced systems are needed in educational environments.

ARToolKit is an open-source and simple tool used in various augmented reality (AR) applications [1]. ARToolKit has no additional costs as it uses printed images detected by a video camera for pose estimation and/or human-computer-interactions. However, if the marker distinctions are not ensured, the markers are often miss-recognized due to inter-marker confusion. Typically, a single Latin character is added in ARToolKit markers as its inside pattern [2]. In order to make it more user-friendly the pattern can also be written in other languages like Arabic, Urdu, Pashtu, or Persian. However, marker distinction becomes more difficult when the characters have a similar structure. Therefore, the user needs to follow some mechanism during marker designing or library management phase to avoid possible inter-marker confusion.

Ullah et al. [3] used ARToolKit, which is an easy solution at a low cost. However, their board only has the capability of writing characters with no graphical images. Furthermore, their board has different modules, each with a different marker. For example, one marker is used for writing, one is used for cleaning the board, and similarly one for the saving option and one for the opening option. To memorize all these markers for different tasks is difficult for primary level students. In addition, even if students can remember the markers, this has no relation to their learning and education.

In this paper we present an easy-to-use interactive writing board (IWB) for teaching Urdu and Arabic characters in primary education. The board uses ARToolKit markers for interaction. Each marker has a single character of the Urdu or Arabic language. First, we ensure the marker distinction by classifying the markers into different classes. The classification algorithm uses the K-means algorithm which classifies the markers based on their degree of similarities with other markers. Then, we use each class of markers as a single ARToolKit library for the IWB. With the detection of each character, the system displays a relevant image that helps students to understand the character and its pronunciation.

The system was evaluated by teachers and students in August 2019. Three groups of users including group 1 (30 students, 4 to 7 years in age), group 2 (30 students, 8 to 12 years in age), and group 3 (20 teachers, 22 to 60 years in age) were randomly selected. The evaluation was carried out in the primary-level educational institutes of Malakand district, Khyber Pakhtunkhwa, Pakistan. The evaluation results revealed that the system improved students’ motivation and learning skills. In addition, the system was compared with a previous IWB [3], and was found to have better performance, was easier to use, and was more attractive for the students.

The rest of this paper is organized as follows. Section 2 defines the basic terms used in this paper and Section 3 covers the related work. Section 4 describes the classification method and Section 5 presents the proposed writing board. Section 6 presents the experimental results. Finally, Section 7 concludes the paper with possible future improvements.

2. Preliminaries and Definitions

In this section, we present definitions of the basics terms including augmented reality, ARToolKit, and the Arabic and Urdu alphabet sets.

2.1. Augmented Reality

Augmented reality (AR) augments virtual objects over the real-world environment, in order to improve the user perception of the environment [4,5,6]. AR is widely used in various fields including teaching and learning [7,8,9], robot navigation [10], tour sharing [11], video conferences [12], medical, entertainment, military, robotics, manufacturing, maintenance, and repairing [13].

2.2. Fiducial Markers and ARToolKit

Different toolkits such as ARTag [14], ARToolKit and ARToolKit Plus [15] are used for marker tracking. Fiducial markers are printed images, recognized via video camera, and used for human-computer interaction or information visualization [16]. Fiducial markers are used in various AR applications, such as robot navigation [10], 3D modeling [17], and education and learning aids [8,9,18]. The markers are used for accurate pose estimations [19]. Figure 1 shows a screenshot of virtual solid teapots overlaying the detected ARToolKit markers.

2.3. Arabic and Urdu Alphabets

The Arabic language alphabet set contains 29 different characters as shown in Figure 2. The Arabic language and many other languages like Urdu, Pashtu, and Persian are written from right to left and have similar writing rules. Language processing and text recognition of the Arabic language have several challenges and therefore has remained an active research area [20,21].

The use of Arabic characters in ARToolKit markers instead of Latin characters or other graphical objects will make the AR application more user-friendly for students. However, Arabic characters are mostly similar in shapes, thereby causing more inter-marker confusion in the ARToolKit. Therefore, it is desirable to find a similarity among the markers having a pattern of Arabic characters and classify them into different groups. The Urdu alphabet set contains 37 characters as shown in Figure 3 which contains all the Arabic characters with some more additional characters.

3. Related Work

Various studies have been conducted in the field of marker-based AR on the quality and applications of fiducial markers. This section covers the literature review on fiducial markers and their quality. In addition, the ARToolKit library and the problem of inter-marker confusion are also discussed.

3.1. Inter-Marker Confusion in ARToolKit Library

Marker-based tracking is a popular method in AR where predefined markers are placed in the real scene, from which their position can be calculated [5]. ARToolKit [1] uses these markers to interact with virtual objects in a real world environment [22]. The pattern inside the ARToolKit markers typically causes inter-marker confusion [2]. Badeche and Benmohammed proposed a marker classification mechanism to reduce inter-marker confusion [2]. They tested 26 different markers, each with a single Latin alphabet in its interior white region. They performed an experiment for each marker using ARToolKit, computed the degree of similarity with each of the 26 markers. Finally, they applied the K-means algorithm on the stored experimental results and classified these markers into seven different classes. They also repeated the experiments for different values of k. For minimal inter-marker confusion, they recommended the use of characters of different classes and especially advised the use of characters from smaller classes [2]. The classification is useful, however it is not applicable if someone uses markers that have a pattern rather than a single Latin character.

Khan et al. [23] studied 11 factors that affect ARToolKit marker recognition. They conducted detailed experiments by varying these factors. They identified the optimal values for these factors to be used for robust marker recognition. These factors include (1) black to white ratio, (2) sharp and de-noised edges, (3) the complexity of the marker, (4) intensity of the light, (5) the pose of marker to camera and light source, (6) smoothness of marker surface, (7) movement of the marker, (8) frame rate, (9) marker segmentation, (10) resolution of the camera, and (11) quality of the printer used.

The automation of the marker design process is gaining the attention of researchers in marker-based AR. Layered marker generation tool (LMGT) [24] is a tool used for layered marker creation. Similarly, the black to white ratio [25] and the noise in the marker [26] also affect marker recognition. To ensure the distinction of all existing markers in the library, is a challenging task. Recently, Khan et al. [27] introduced a tool called FMO (fiducial marker optimizer) for designing a high-quality marker. FMO [27] also recommends marker classification for ensuring marker distinctions. Image correlation [28] based classification also has a similar recommendation for using a marker from a smaller size class after marker classifications.

3.2. Interactive Writing Boards and Technology-Enhanced Learning

Technology-enhanced learning is significantly improving the teaching and learning methodologies because it allows students to access, extend, change, and exchange concepts, ideas, and knowledge in a multi-modal fashion [29]. Interactive writing boards (IWB) play a vital role in enhancing student interest and their learning skills [3]. Vergara et al. [30] proposed an active methodology to increase motivation for secondary school students in mathematics courses. The method [30] recommends a small group (four students) for effective learning. Hartono et al. [31] presented a game-like 2D cartoon art for teaching mathematics to primary level students.

Horii and Miyajima presented an ARToolKit-based education system for visualizing and teaching mechanical drawing [32]. They used cubic ARToolKit markers where the user can watch the objects from different directions by rotating the cubic marker. Their system is efficient and cost-effective. However it is limited to mechanical drawing.

Recently, Liang et al. [33] proposed an intelligent interactive system for teaching. Figure 4 shows the overall pipeline of this system. This system [33] uses smart watches and smart phones to capture the hand gestures of the users (students) and respond in real-time to teachers, thus increasing student interaction an motivation. However, the use of smart watches by all students is expensive in terms of costs. Similarly, $Create@School$ [34] is another system that gives a technical resource to the users (students/teachers) in the classrooms. It improves the users’s development skills and prepares them for programming and game design. The system improves engagement between students and teachers in the classroom.

4. Marker Classification and Library Management

Our writing board (Section 5) is based on ARToolKit markers. However, there exists confusion among the Arabic/Urdu characters. Therefore, we need to organize the markers as a lesson plan into different groups such that to minimize the inter-marker confusion. The teacher will set the library according to the lesson plan before starting the lesson. In this section, we describe the marker classification strategy.

The method described in [27,28] is followed for marker classification with a new variation of MSE. The algorithm for classification of markers into k classes is given in Algorithm 1:

Algorithm 1 MarkersClassification(MarkersLibrary[n], K).

1: $N1\leftarrow \mathrm{CountMarkers}\left(\mathrm{MarkersLibrary}\right[\mathrm{n}\left]\right)$ 2: for$i\leftarrow 1toN1$do       (i) $key.marker\leftarrow \mathrm{MarkersLibrary}\left[\mathrm{a}\right]$       (ii) for $j\leftarrow itoN$ do             (*) $marker2\leftarrow \mathrm{MarkersLibrary}\left[\mathrm{j}\right]$             (*) $err\leftarrow \mathrm{MSE}(\mathrm{key}.\mathrm{marker},\mathrm{marker}2)$             (*) $ResultsMatrix\left[i\right]\left[j\right]\leftarrow err$             (*) $ResultsMatrix\left[j\right]\left[i\right]\leftarrow err$          end inner for loop end outer for loop 3: $List.of.Classes\leftarrow \mathrm{K}-\mathrm{means}(\mathrm{ResultsMatrix},\mathrm{K})$ 4: $end$

This algorithm takes all the markers from the library and the k value (i.e., the number of classes), as input. In the initial step, the algorithm counts the total number of markers in the ARToolKit library. For example for Arabic character $N=29$, whereas for Urdu $N=37$. In the second step, all the markers along with their numeric values for the similarity between any two markers are added to a N × N Table i.e., the ResultsMatrix (see example in

Table 1. A 3 × 3 Results matrix, containing the degree of similarities among the Haa1, Jeem, and Haa2 (see Figure 8) pattern on the ARToolKit marker.

(a) Correlation				(b) MSE
Marker	Haa2	Jeem	Haa1	Marker	Haa2	Jeem	Haa1
Haa2	1.000	0.8116	0.8275	Haa2	0.0000	0.0773	0.0711
Jeem	0.8116	1.000	0.8653	Jeem	0.0773	0.0000	0.0551
Haa1	0.8275	0.8653	1.000	Haa1	0.0711	0.0551	0.0000

). Here the similarity is computed by using either correlation or MSE between the two markers. A lower value for MSE indicates a smaller error and a higher similarity. In the correlation, the higher value indicates the two markers are more similar. In the next step (third step), the markers are grouped into the given number (k), distinct groups.

For similarity computation, two methods were used. One was image correlation (2D correlation coefficient) which was previously used in our marker designing tool [27] for calculating the similarity between the two markers. The second method was a mean-squared error (MSE) between the two markers. For MSE each marker is converted to $uint8$ class.

5. The Writing Board and Marker Libraries

We classified the markers into seven different classes using the MSE method. We used this classification and design six ARToolKit libraries in such a way that none of the two markers from the same class lay in the same library. Figure 5 shows the classification with the corresponding libraries.

Each single marker was then used for writing a single Arabic/Urdu character on the IWB. Figure 6 shows the overall working mechanism of the proposed system. A user (student/teacher) brings a marker in front of the camera. The marker recognition system detects a marker and then recognizes it by comparing it with all existing markers in the library. The ID of the recognized marker is sent to the image selection system. The image selection system picks the corresponding image from the images library and sends it to the display module (i.e., IWB). The white board is a display screen which displays a single character with recognition of the corresponding marker. A virtual character that starts with this character is also displayed on the board.

A teacher sets the ARToolKit library and asks the students to bring different markers in front of the camera. The students pick a marker, bring it to the front of the camera, and see the resultant image on the 2D board. The board also has a 3D mode where the character is displayed on the 2D screen while the corresponding image is displayed in the background as a 3D scene. Figure 7 shows a screenshot of teaching/learning the Urdu character (alif) with the proposed IWB.

6. Experimental Results

This section describes the experimental results in two categories. Section 6.1 presents the marker classification results, whereas Section 6.2 presents the results from the user study. Furthermore, a comparison with a previous IWB [3] is also given.

6.1. Classification Results

The correlation method for similarity checking has been tested in [26] and FMO [27], which shows that the correlation method has a semantic relationship with the ARToolKit matching algorithm. Our first experiment was the illustration of our classification method using three different markers shown in Figure 8. Table 1 shows the similarity computed for these markers. If we apply K-means with $K=2$, in both cases (correlation and MSE) the Jeem and Haa1 are in class 1 and Haa2 is in class 2.

In the second experiment, a simple experiment was performed to show the significance of marker library management. In this experiment, three markers’ library were used as shown in Figure 8. We executed the ARToolKit and moved the marker with character Jeem (Figure 8 (Left)). We noted true detections (marker was detected as Jeem) and false detections (marker was detected as Haa1). The true detections amounted to 1750 whereas there were 1380 false detections, which means a huge rate of false detection (see Figure 9).

The third experiment was the classification of all 29 characters of the Arabic language, for which we designed 29 ARToolKit markers each containing an isolated character as a pattern (see Figure 10).

Table 2. A 29 × 29 Results matrix, containing the degree of similarities among the characters of the Arabic language used as a pattern on the ARToolKit marker, calculated via binary correlation.

Marker	Alif	Baa	Taa	Saa	Jeem	.	.	.	Yaa
Alif	1.0000	0.4684	0.4571	0.4332	0.1983	.	.	.	0.3050
Baa	0.4684	1.0000	0.8253	0.7825	0.3609	.	.	.	0.3760
Taa	0.4571	0.8253	1.0000	0.9519	0.4199		.	.	0.4343
Saa	0.4332	0.7825	0.9519	1.0000	0.3889	.	.	.	0.4175
Jeem	0.1983	0.3609	0.4199	0.3889	1.0000		.	.	0.2926
.	.	.	.	.	.	.	.	.	.
.	.	.	.	.	.	.	.	.	.
.	.	.	.	.	.	.	.	.	.
Yaa	0.3050	0.3760	0.4343	0.4175	0.2926	.	.	.	1.0000

shows the $29\times 29$ matrix containing the similarities among different characters using the binary correlation method. We set the value of K and apply the K-means algorithm to classify these 29 markers. We repeated the experiments with different values of K, some of them are given here.

For K = 7, we got the seven classes shown in Figure 11. Both methods give different classifications. However, based on the structure of the ARToolKit markers and its detection mechanism, we recommend MSE for marker classification.

Similarly, for $K=13$, we got 13 classes as shown in Figure 12. We recommend this classification for ARToolKit as it provides enough library size and with minimal inter-marker confusion. In addition, the value of K can be changed to get a different number of classes as desired in our AR application. Here again both methods have different results, but still the classes are matching. Either method can be used to classify a marker library and eliminate possible confusion in the markers library.

6.2. Results Concerned with the Writing Board and Students’ Learning

We conducted a similar subjective study like that of Ullah et al. [3] using 30 primary level students (Group 1: 22 male, and 8 female) with an age range from 4 to 7 years (for easy questions i.e., Q1–Q6, which compare our system with an existing system [3]). We also conducted further analysis on another group of students (Group 2: 25 male, and 5 female) with an age range from 8 to 12 years. In addition, we also conducted the subjective analysis with 20 teachers (Group 3) who participated in all experiments. The participants (students and teachers) were randomly selected for the experiments. We noted the opinions of the users and compared our results with Ullah et al. [3]. We asked the following questions from each user.

Q. 1:

Which of the two writing boards is easy to use?

Q. 2:

Which of the two writing boards is more interesting?

Q. 3:

Which of the two writing boards do you like more?

Q. 4:

Which of the two writing boards is better for group study?

Q. 5:

Which of the two writing boards looks similar to a traditional board (physical board)?

Q. 6:

Which of the two writing boards makes you more tired?

For further analysis (i.e.,

Table 3. Analysis of user fatigue and students’ motivation using our system.

Concerned Opinion	Users	Total	Lowest	Low	Medium	High	Highest
Users fatigue	Teachers	20	0	7	13	0	0
Users fatigue	Students	30	5	20	5	0	0
Students motivation	Teachers	20	0	0	1	7	12
Students motivation	Students	30	0	2	1	5	22

and

Table 4. Results from the user study, for the additional questions/statements.

Statement/Question	Users	Strongly	Agree	Neutral	Disagree	Strongly
		Agree				Disagree
The cost is affordable	Teachers	14	5	1	0	0
	Students	16	12	0	2	0
The board is easy-to-use, but	Teachers	17	3	0	0	0
installation is difficult	Students	10	20	0	0	0
The students will enjoy	Teachers	15	3	1	1	0
while using the board	Students	28	2	0	0	0
The board is better for small	Teachers	5	13	0	2	0
group only ($\le 6$ students)	Students	0	25	4	1	0
The board is better for quiz	Teachers	5	13	0	2	0
rather than lesson teaching	Students	0	25	4	1	0
The board can not completely	Teachers	20	0	0	0	0
replace the traditional boards	Students	20	9	0	1	0
We will need an IT Expert/	Teachers	1	2	0	2	15
special IT teacher for this board	Students	5	9	3	3	10

), we needed mature students. Therefore, group 1 was replaced by group 2 and group 3 remained unchanged. Figure 13 shows students interacting with the proposed board.

Figure 14 shows the results collected from users (group 1 and group 3) after the experiments whereas Figure 15 shows the results collected from users (group 2 and group 3) on user fatigue and student motivations. Similarly, Table 3 and Table 4 show the corresponding numerical results of the user study. The results show that our system had a significant improvement over Ullah et al. [3]. The results further reveal that our system gave less fatigue and motivated students by increasing their interest in the classroom.

7. Discussion and Conclusions

In this paper we presented a simple and easy-to-use interactive writing board. The board used simple ARToolKit markers that had a single Arabic or Urdu character as a pattern. We contributed in two aspects. Our first contribution was from the computer graphics point of view, i.e., ARToolKit library management to reduce inter-marker confusion. We found a high false-positive rate (recognizing one marker as another one) in ARToolKit markers. Therefore, we used a simple yet effective mechanism to avoid inter-marker confusion. The mechanism computed the degree of similarity between two characters in a semantical similar manner as used in the ARToolKit library. Though there were four different positions of Arabic characters including first, middle, last, and isolated [20,21,35]. Our work was only concerned with Arabic characters in their isolated positions. There are 29 characters in the Arabic language as shown in Figure 2. We designed 29 ARToolKit markers, each having an isolated Arabic character in its interior white region (see Figure 10). The same was repeated for 37 Urdu characters (Figure 3).

We used two different methods for computing similarity between two markers. One was image correlation (2D correlation coefficient) which was previously used in a marker designing tool [27]. The second method was mean-squared error (MSE) between the two markers. Then the K-means algorithm was used to classify these markers based on the computed degree of similarities. The markers if selected from different classes will have minimal inter-marker confusion. The maker that was in a small class size with a larger number of classes (k), was the most distinguishable. Inspired by this observation, we classified the markers into seven different classes. Based on this classification, we created different ARToolKit libraries used for writing on an interactive writing board (IWB). This classification solved the issue of the inter-marker confusion.

The second main contribution is in the field of education in the form of the IWB, i.e., a computer application for education. We designed an easy-to-use interactive writing board for teaching isolated Arabic/Urdu characters using ARToolKit markers. The IWB displayed a character/image after recognition of the ARToolKit marker. The displayed character/image is selected such that its name starts with the character on the recognized marker, which helps students to learn the character and its pronunciation.

In the experimental study, we first showed our classification of the markers. Then, we evaluated the proposed IWB via students’ and teachers’ opinions. We also gathered users opinions to compare the current board with a previous one [3]. Experimental results revealed that the IWB improved student interest and was relatively easy to use. Furthermore, users preferred to our current board instead of the previous one [3]. The evaluation results were collected from a particular sample of participants in the Malakand district of Khyber Pakhtunkhwa, Pakistan. Changing the participants (from the same place or other) may give different results.

Limitations: There are three main limitations of the current work. (1) The board uses a single camera, therefore it is applicable to small groups only ($\le 6$ students). One solution is to use multiple cameras, however, it may increase the cost. (2) We grouped the distinct characters in one class which contradicts the actual teaching methodologies where similar characters are placed together. (3) The current system is only used for teaching single characters in their isolated forms.

Future work: For future work, we are planning to use the concept of layered markers [24] for concatenation of these characters, which will be used for Urdu, Arabic, and Pashtu characters. We also plan to use the concept of layered markers for interaction with different 3D objects on the board.