Potentials and Challenges in Students’ Meaning-Making via Sign Systems

Abstract

The relationship between sign systems and the meaning potentials and affordances of multimodal technologies has received increasing attention in research on digital technology use in education. Students constantly adhere to and engage with semiotic shifts in sign systems when they work with digital technologies for learning purposes. This study explores students’ use of digital technologies in Swedish schools. We trace the way semiotic activity systems and cognitive processes are transformed and realized when students engage with shifts in sign systems into various meaning-making strategies. Methodologically, the study is based on a data set of video recordings, interviews, and observations of classroom practice in three primary schools. An analysis that draws on quantitative ethnography was applied to process and analyse the data. The main findings revealed that sign systems prompted by the technologies and the social space compete to some extent for students’ attention, and that technology design is monotonously rendering lower levels of mediation. These findings show that various sign system prompts need to be balanced and streamlined to support students in their meaning-making. This article conveys the importance of understanding sign systems, as they are the most common resources for technology-assisted learning, and change the prerequisites for meaning-making.

1. Introduction

Digital technologies are commonly used for teaching and learning purposes in schools. Although many students are digitally proficient and use digital technologies on a daily basis, they are nevertheless challenged when encountering technologies aimed to assist learning activities in the classroom, because different hardware and software configurations change the conditions for meaning-making [1]. When different technological arrangements are applied to support learning, it also increases the complexity [2,3]. If we do not recognize the potentials and challenges posed by the shifts in various technologies into composite interfaces across different media [4], the meaning potentials and affordances of various digital hardware and software that support students’ cognitive processes will remain unknown. The ability to sift and critically analyse detailed information conveyed by digital technologies are important 21st-century skills, especially in situations characterized by interactivity and shifts across technologies [5,6]. These skills are often developed with the interpretation of and engagement with signs [7,8], in which different configurations of digital technologies and actors facilitate various meaning-making potentials [9,10].

Research on multimodality has shown that the relationships between signs across users and digital technologies contribute to a semiotic multiplicity in change, in which the signs’ different roles for meaning-making become important units of semiotic analysis [7,11,12]. When using digital technologies, students face a host of sign-based human and technological prompts [13,14]. Their understanding is affected by the technological hardware and software configurations into composite interfaces, the sign’s meaning potential, and the types of learning situations, which impact their individual interpretive efforts and engagement [15,16]. A student’s interpretation of human and technological signs is an ongoing internalization process and can be made visible by what Bezemer and Kress [1] refer to as transformative engagement, which activates the externalization of signs through cognitive processes of actions and sign-making.

A burgeoning research field has targeted the way semiotic signs shape social practices in digital technology use [9,17,18]. However, most of this research has studied signs and their relationships with various meaning-making activities. Gaps still exist in the knowledge of how shifts between multimodal signs on composite interfaces and the social space affect students’ various cognitive processes in multimodal meaning-making. The purpose of this paper is thus to explore the way students in primary school use today’s most common digital technologies, such as laptops, smartphones, and tablets, in the classroom by tracing how cognitive processes are transformed and realized through their engagement with the signs into various meaning-making strategies. Two research questions guided this work: (1) How do semiotic shifts of signs in technologies and the social space affect meaning-making activities? (2) What cognitive processes are evoked in students’ engagement with the shift of signs on composite interfaces that affect meaning-making outcomes?

1.1. Background and Related Work

In today’s schools, students must constantly attend to changes in shifts in resources (e.g., semiotic shifts across signs [19])—in digital technologies and in social space. New hardware and software are regularly implemented, and new sign-based methods are introduced to the classroom. Thus, signs are considered multifaceted shapers of learning conditions [10] that, together with physical, virtual, and social learning affordances, transform the potentials of learning activities due to the communicative potentials that are or are not taken up [20,21].

Various potentials and challenges can arise when students engage with these ongoing changes. Bateman [22] argued that changes in multimodal artefacts could lead to interpretation issues and variations that might challenge organizing meanings, which is also noted in a recent study on film by Tseng et al. [23]. Shifts in signs are contextual and therefore have different potentials that demand various levels of engagement [10,16]. Shifts in signs also affect a student’s cognitive processes of internalization and externalization that are transacted through signs [24]. Adami [25] described the different roles signs can take and the complexity in engaging with interactive signs in technologies. She put forth that signs visualized on an interface need to be interpreted, but also acted upon. This relates to the tools and signs’ role for mediated activity in human development. Vygotsky [24] claimed the tools’ role for the development of practical intelligence and action as a predecessor to the use of and production of signs. However, the association of these tools and signs to cognitive processes becomes uncertain when interacting with digital interfaces, because the tools and signs are blurred with notions of symbolic signs. A sign can be visualized and interpreted as a “tool” with a function for action. Simultaneously, a sign is visualized with abstract connotations for developing thinking, and it must be interpreted based on those connotations. This means that the tools’ and signs’ roles merge through symbolic signs in physically or virtually manifested representations [26,27] and relate to what Vygotsky [24] called “higher psychological function, or higher behaviour as referring to the combination of tool and sign in psychological activity” where the “internalization of culturally produced sign systems brings about behavioural transformations and forms the bridge between early and later forms of individual development” (p. 7).

The semiotic shifts impact the interpretation of symbolic signs and have several implications. First, the meanings of the signs are intermixed in digital technologies and social space with dual unfoldment, as manifested in representations for enactments (e.g., a function, [17,25]) and as “material stuff” for use in meaning-making production (e.g., signs such as colours, images, size, etc., [7,8]). This means that the originally arranged distinction between tool (physical) and sign (symbolic) is now relevant for the interpretation of symbolic signs’ different meanings. Second, because signs have dual meanings, the interpretation of signs becomes complex. Different signs are internalized and externalized depending on the way they facilitate action and sign-making. The user in turn reshapes the prompt into a new sign that becomes available for interpretation, creating a circularity that sets the semiotic systems in a state of flux [1]. Third, technical activation (e.g., interactive signs) has several dimensions that are parallel to the human activation of signs in actions and sign-making activities. For instance, combinations of various digital hardware and software in composite interfaces shift the sign and signifier–signified relationships [4]. These shifts are also a matter of how direct symbolism [24] is activated in students’ awareness and physically and virtually manifested in the technological design relative different symbolic sign systems.

The full capacity of digital technologies and the meaning produced in the social space will risk being unexploited if the implications of semiotic shifts in signs are unattended in learning settings. In turn, this may cause monotonous, one-dimensional, and hence inefficient, meaning-making activities and outcomes. Recent efforts have been made to encompass the semiotic shifts in signs and their complex relationship with the expansion of meaning-making related to digital technologies. Ravelli and van Leeuwen [4] discussed this in terms of a shift where “the medium itself may change the message” (p. 291), something that must be regarded as having considerable semiotic implications. For instance, different digital hardware prompt software sign systems in various ways. Different hardware also needs to be interpreted based on their symbolic signs. From this perspective, digital hardware technologies are considered semiotic [28]. The semiotic shifts entailed by various digital media have also been discussed by Jewitt [5,29] and Kress and Adami [30]. The digital hardware component, such as the operational system, provides a more nuanced picture of the interaction [17]. However, including hardware in the analysis has often proved challenging, which is why previous research has favoured studies on the use of software. This has led to a situation in which we have limited knowledge of the ways digital hardware can contribute to various semiotic effects.

Zhao and Zappavigna [3] analysed the shift between digital hardware and software technologies configured into composite interfaces in social practices of making selfie pictures. They focused on the way the use of a selfie stick, size of the phone, or the camera lenses of certain software interacted to create prerequisites for various signs to be conveyed. Similarly, Poulsen [17] analysed different versions of Instagram apps and noted that the variation in the semiotic representation of functions influenced interpretations of the function, resulting in different social procedures of searching. Zhao and van Leeuwen [31] and Kvåle [32] have also noted that the “material stuff” of technologies shifts between software templates where the variously available signs have been integrated into different forms of communicated meanings. Specifically, the interchangeable role of the symbolic signs’ material realizations (e.g., as tangible) was explored by Djonov and van Leeuwen [33] in their work on texture.

Although the activation of signs in digital technologies and social space has been much recognized in different forms of meaning-making, the way the semiotic shifts of signs become relative to interpretation and engagement efforts and how cognitive processes of internalization and externalization are combined and connected are still rather unexplored. In sum, by studying semiotic shifts of signs, we can obtain insights in the ways different technology uses are related to specific setups of signs that can potentially inform the development of more multimodal technology-assisted learning activities. To identify and trace the links between students’ cognitive processes and signs, the multimodal layer (ML) framework [34,35,36] was applied as a context for analysis to examine the way meaning-making based on symbolic sign systems is carried out and developed. The ML will be presented in the next section.

1.2. Multimodal Layers: Semiotic Technology and Sign Systems

In this section, we will outline an ML framework that draws on insights from semiotic technology research to enable the analysis of semiotic shifts in sign systems. The notion of sign systems is central to the framework, and it is defined as “signs combined through formal/social rules (that become) meaningful in certain combinations” [37] (p. 257). Sign systems are comprised of signs that have meaning potentials, which expand their role in semiotic shifts and possibilities for the cognitive process of actions and sign-making. When studying meaning-making, students’ agency and choice of sign systems are important [1] because “meaning is choice from a system” [38] (p. 147).

Multimodal epistemologies address a need to access the many details that come into play between actors and digital technologies. This makes the definition of certain components important [13,39] in aspirations to apprehend the learning outcome as one central tenet for multimodal research, seizing various technological effects [40]. The focus is on multimodal components’ relationships and the resourcefulness of learning environments [1] that go beyond the appreciation of single entities in uniting the problematic separation of resources from their uses and their users [39]. The ML framework enables tracing the connections between semiotic technologies and the inherent meaning-making potential of social spaces by relating the semiotic work of interpretation of symbolic sign systems to the outcome as modes of representation. By studying meaning-making activities via digital composite interfaces, actions that are understudied within semiotic research are addressed [41]. The technological properties in hardware–software combinations also become visible. By focusing on certain components, this framework can be used to synthesize how human and technological sign systems interact to produce different outcomes. The ML framework consists of five components: activities, modes of representation, digital technologies, technologies’ functional properties, and technologies’ semiotic properties.

1.2.1. Activities

Activities do not stand free from societal embracement, they are framed by macro-level principles and imbued with micro-level principles. Jewitt [13] proposed that on a macro-level, activities are formed by the socially and contemporary accepted knowledge available in a particular context at a specific time that shapes different discourses and subjects. On a micro-level, activities are related to the meaning framed and innate in the social relations between teachers and students within a specific setup of subjects and resources [20]. Bezemer and Kress [1] argued that the intentional design that is renegotiated in the communicational context of classroom learning is also framed in a student’s micro-context. Then, learning design is related to the teacher’s perception of the meaning-making outcome manifested by the students. The teacher’s role as learning designer is essential, incorporating macro- and micro-perspectives, to tailor learning activities by balancing various epistemological undertakings, materials, and technologies between students’ needs and the curricular demands [16]. These macro- and micro-relationships make classroom activities enormously complex and multifaceted. Any contextual factor, either human or technological, can act as a prompt and attract an individual’s interest [41]. The reciprocity among the ML prompts that briefly manifest into various learning design models are of specific interest for tracing activities.

1.2.2. Modes of Representation

The reciprocity among prompts generated in a learning setting is important to understand from a sign system viewpoint. When the sign system prompts are perceived and interpreted by the actors, they are internalized, and eventually externalized based on the prompts’ meaning potentials and affordances into new sign systems that establish new prompts [7]. Semiotically, these instances relate to the process of signification [22], where the outcome forms unified messages called “modes of representation” [42] and become identifiable for analysis as a meaning-making outcome in a learning situation [1]. The students perceive sign systems by their social, cultural, and historical means of representation [43], and they are guided by their experiences, interests, and attention [15,16], which alter the sign systems to become the most apt signifier for the newly signified to communicate meaning [7,10,41]. The process of signification in a technology-assisted learning situation entails a multiplicity of students’ agentive interpretations of the sign systems’ meaning potentials [8,13,44], which, through cognitive processing, realizes the actions and sign-making (i.e., sign-making “is meaning-making and learning is the result of these processes” [7], (p. 178)). The conventions laden in the sign systems render various possibilities for meaning-making to be conveyed where images are understood to communicate meaning by their affordance of spatiality and simultaneity, while linguistic and auditory modalities afford temporal and sequential messages and so on [45]. By identifying and tracing the prompts’ connection to the outcome in modes of representation through ML mapping, a synthesis can be made of the reshaping process that relates the original sign system to the newly produced sign systems as an outcome of meaning-making.

1.2.3. Digital Technology

Digital technology is a layer aiming to identify how digital hardware and software are combined into composite interfaces. The students’ uses depend on the composite interfaces’ technical and temporal/spatial affordances such as mobility, capacity, size, shape, and compatibility to support meaning-making. Both hardware and software technologies have significant and compound semiotic roles in meaning-making activities by intermixed physical and virtual properties that transcend symbolic sign systems’ meaning potentials in the technological design. Besides, hardware and system software render the application software in a uniquely different way [2,3,17]. The merging role of tools and signs [26] in multiple symbolic sign systems on composite (hardware–software) interfaces is emphasized by employing this layer.

1.2.4. Technologies’ Functional Properties

The layer of functional properties defines the relationship between an individual’s actions and the use of the symbolic sign systems in the shape of functions that emphasize merged role of tools and signs. The physical and virtual functional properties interconnect the technical capacities of the functions with the symbolic representation of the function [17,46]. They enable observation of the way functionality works in the user’s process of signifying the multimodal design in interpretation of composite interfaces and how the representational and technical functions become teleological for people’s social actions [cf. 17]. Functional properties work as sign system prompts and trigger a student’s reshaping process. This layer is used to identify and synthesize how technologies’ different properties are used in actions to shape meaning.

Wartofsky [27,43] made an interesting proposal to understand humans’ intentionality engraved in representations and the way they become functional through actions, taxonomized into primary, secondary, and tertiary mediating levels. This idea helps to explain how meaning-making is mediated via both the tool and sign through symbolic sign systems and how the activation of functional properties in actions transcends into the use of the technologies’ semiotic properties in sign-making activities in composite interfaces. The primary level of mediation is linked to the historical modes of praxis [27]. It is a lower form of cognition and actions to deliver work using the functional properties. The primary mediating level is related to the secondary mediating level that concerns actions with functional properties in endeavours to arrange the substance for production in work [47,48]. Wartofsky [27] declared that primary and secondary mediating levels are online and real-time externalized actions. Simultaneously, in engaging with composite interfaces, these actions involve the students’ internalization processes of interpreting the potential meaning of symbolic sign systems that evoke higher psychological functioning [24]. The online feedback loops relate to the offline loop: the tertiary level [27,43,47]. The tertiary mediating level is linked to actions with the intention of conveying meaning through a function by also drawing on the sign systems that are available via the technologies’ semiotic properties (the “material stuff” [7]). These actions move beyond the goals of obtaining a direct work-related performance by attending to the affordances of the sign system to convey abstract meaning. Thus, the functional properties have a complex role in the development of practical skills and abstract intelligence through the symbolic sign systems, which work as both tool and sign in human awareness [24]. This layer is used to identify and map various forms of cognitive processes in students’ different actions and sign-making activities with technological sign systems.

1.2.5. Technologies’ Semiotic Properties

When a function is mediated through the tertiary level, it is coupled to the user’s wish to convey abstract forms of meaning [27,47] through the technology’s semiotic properties [36]. The semiotic properties explicate the “material stuff”—the symbolic sign systems technologically available through various functional properties with meaning potentials and affordances for combination, to convey abstract meaning into modes of representation in sign-making. Both functional and semiotic properties make it possible to identify and trace the way cognitive processes of actions and sign-making are coupled and transacted through the shifts in semiotic sign systems. It allows for synthesis of what constitutes the meaning-making performance into new sign systems [36]. The new sign systems are assessed as the outcome of meaning-making via digital technology use [49], where, for instance, one-dimensional modes of representation would reflect a minimal advantage-taking of the properties’ affordances in the technological design and marginal learning outcomes through the technologies.

MLs are used for multimodal mapping and synthesis of the connections between human and technological processes based on symbolic sign systems. The analysis always begins with the notice of a sign system prompt that is observed from the technology, the users, or social activities. An example of the ML connections is shown in Figure 1.

2. Materials and Methods

This section presents an empirical study that draws on the ML framework to map and analyze semiotic shifts in symbolic sign systems. With an analysis that is conducted in multiple layers, we were able to provide a detailed description of how students interpret sign systems and how sign systems affect meaning-making strategies. The study is based on video recordings and observations in classroom and interviews with the students. All data were collected during different periods in the autumn of 2020 in three Swedish schools.

2.1. Sampling Procedures and Participants

To obtain in-depth data and enable an ML mapping, we applied a purposive sampling method [50] of school classes. The most imperative sampling criterion was that the setting had to be characterized by the occurrence of commonly used digital hardware devices in schools (laptops, smartphones, and tablets) in combination with multiple types of digital software for learning purposes. Three different-sized schools from three different regions were selected. The schools had technology-dense practices (one-to-one) where digital technologies were used in daily learning activities. Fifth-grade students were chosen as participants because they are considered digitally confident [51]. With the help of teachers, one student from each school was selected to participate (two girls and one boy).

In addition to the direct setting, the sampling also included data representing different subjects, different learning design methods, and characteristics of assignments where students actively used digital technologies. From an ML perspective, these dimensions of classroom practices entail an analysis of sign systems in activities.

Table 1. Participant demographics.

Participant	Gender	Hardware Technology	School Setting	# of Students in Class	Subjects
Student 1	Girl	Laptop	Rural area, <500	21 students	Swedish, Social science, Mathematics
Student 2	Boy	Tablet	City area, >500	28 students	Swedish, Social science
Student 3	Girl	Smartphone	Rural area, <200	25 students	Science, Mathematics

presents an overview of the sampling.

2.2. Data Collection

As mentioned, a combination of video recordings, observations, and interviews [40,52,53] were applied to map multimodal layer information. Video recordings can reveal fine-grained multimodal aspects because events can be captured visually and auditorily [52,53]. Examples include the digital technologies’ sign systems, the user–technology interactions with sign systems, and the outcomes of those exchanges into new sign systems through actions and sign-making activities in the social space. In the present study, two cameras were used in every lesson—one focused on the students’ activities in the whole classroom context, and one with a close-up perspective of the digital technologies and the student’s activities on and around the screen. The total duration of video recordings of the three students was approximately 350 min (5.8 h). During the video recordings, structured observation notes were made on a simple observation schedule to enable easy and fast notes (Supplementary File S1). The purpose of the observation notes was to map additional ML and user data that could supplement the video data for pairing and strengthen the researchers’ interpretation and understanding of the video data in the subsequent analysis.

Following the video recordings, semi-structured video elicitation interviews (VEIs) were conducted with the three students. These interviews were based on selected video sequences and a question guide grounded in the five MLs (Supplementary File S2). Video sequences were selected by considering how different events portrayed various aspects of the MLs to exemplify the interview questions to enable a request about each layer. The VEI approach aimed to prompt a dialogue between the researcher and the students around video sequences. This had the advantage of aiding students in recalling the way the events unfolded and prompting discussion about the use of digital technologies in relation to the meaning-making activities associated with sign systems [53]. The VEIs were conducted individually in the students’ school environments and recorded with sound-recording equipment. They varied in time and lasted approximately 165 min (2.7 h) for all three students.

2.3. Data Processing

As is common in multimodal research, the study is based on a deductive approach to the data [53], in which the ML framework fits well with an in-depth and streamlined mapping strategy. More specifically, the strategy of “foregrounding” and “backgrounding” information was applied to obtain comprehensive multimodal information in the data [44,54]. Foregrounding and backgrounding mean that certain multimodal features were foregrounded, and others were backgrounded within a specific contextual framing. This strategy was used to condense the large amounts of detailed data. By adopting strategies of foregrounding, the informants’ awareness and focus became visible in their actions and sign-making through, for instance, observing their direction of gaze and gestures, oral and auditory utterances, or writing activities.

In addition, the study follows a quantitative ethnographic (QE; see [55,56]) methodology grounded in epistemic frame theory. This theory states that any community of practice creates a structure that constructs epistemological frames through, for instance, different knowledge and actions [57]. With QE methodology, the epistemological frame’s components are focused through the ML and ML mapping by visualizing important connections among them rather than taking only single entities into account. QE strategies are beneficial for processing and analysing details in data sets (see [56]). In practice, data processing from a QE perspective meant that the qualitatively collected data were transferred into text and had to be transcribed and segmented as well as coded as part of the systematic and quantitative paring procedure of all the data sets. These steps are outlined in the following sections.

2.4. Transcription and Segmentation

To enable pairing of data from the different data sources to conduct an analysis of its components’ connections, the information was transcribed from its original media into text. Video recordings were transformed into text using the Transana software. Annotations were made by describing foregrounded sign systems identified in the video recordings using the MLs as a context. This was carried out in the text editor in the transcript window in Transana [58]. For instance, the annotations could concern which functions the students interacted with on the composite interface. Regarding interview data, the information was transcribed verbatim, first into Swedish using a word-processing program and later into English with the help of translation software. The transcripts and translations were compiled into one document and were organized according to the units (the students), where all the data sources (video recordings, interviews and observation notes) that belonged to the same unit were gathered together (see Supplementary File S3). The document was manually checked afterward to correct any missing or incorrect information, as the coding program is sensitive to spelling errors, doublets, etc. when operating on the data. Observation notes were reorganized into plain text and transferred to a Word document.

After all of the data had been transferred to a spreadsheet, we conducted three types of qualitative segmentation to organize the text data and prepare it for coding and analysis (

Table 2. Descriptive statistics of data sets.

Technology User	#Unique Datasets	#Data Lines of Each Dataset	#Coded Lines (680)
Student 1	Video Interview Observation	77	259
		104
		78
Student 2	Video Interview Observation	86	204
		97
		21
Student 3	Video Interview Observation	120	217
		65
		32

and

Table 3. Example of the segmentation (full version in Supplementary File S3).

	Stanza Window Making the Connections across the Segmented Data
	Conversation in Epistemic Network Analysis					First Segmentation
	The Third Delimiter-Based Segmentation			The Second Source-Based Segmentation
Unit	Subjects	Individual/ Collaborative	Production/ Consumption	Data Type	Metadata (Utterances)	Raw Data Lines
Student 1 (259 rows)	Swedish	Individual	Production	Video recording	1	The student navigates with the mouse pad while she waits.
Student 1	Swedish	Individual	Production	Video recording	2	The student types passcode on Laptop’s keyboard in the password line and presses enter tangent to confirm action.
Student 1	Swedish	Collaborative	Consumption	Observation	1	Gaze is towards what is displayed on the monitor and towards others during the discussion and the board.
Student 1	Swedish	Individual	Production	Observation	2	The student often uses the mouse pad and clicks on hyperlinks using the cursor to retrieve a Google document.
Student 1	Geography	Individual	Production	Interview	1	I cannot even run with that on my laptop.
Student 1	Science	Individual	Consumption	Interview	2	And in the book, it’s just colors.
Student 2 (204 rows)	Etc.
Student 3 (217 rows)	Etc.

). The first segmentation included separating the text data into individual lines (680 sentences) based on how they framed and described the sign systems related to the MLs. The second segmentation entailed a source-based segmentation by adding metadata to the lines, which were referred to as utterances related to the temporal aspect of how the MLs were addressed during the data collection (e.g., during the course of the interview), and data types with spatial changes in ML frequency. This metadata provided important information for subsequent analysis. The third was a delimiter-based segmentation, which entailed identifying activities related to the subjects and individual/collaborative and production/consumption activities, which determined the different MLs by variances in code occurrences. These three types of segmentation prepared the data set for subsequent analysis because the process provided a spreadsheet with added contextual information (see Table 3 and Supplementary File S3, [59,60]).

2.5. Code Construction and Coding

After segmentation, we conducted a quantitative content analysis (QCA) [61] to identify and define codes (variables) and refine the detailed information via keywords (values) related to the ML as part of a more elaborated mapping procedure. In defining the data through QCA, hypothesis modelling is a deductive procedure guiding the code construction [61]. A hypothesis is fundamental for conceptual identification so that further conclusions can be logically derived from data [62]. Our hypothesis was that students’ meaning-making via technologies (the dependent variables) would vary and connect variously to the defined ML categories (the independent variables).

Code construction involved iterative processes of combining top-down (theory-driven) and bottom-up (data-driven) moderation in screening the text data based on the first segmentation [55]. In the top-down moderation, the ML categories were used in constructing grouped codes that created overarching principles through which to understand the data initially. In the bottom-up moderation, the data were screened multiple times to refine the ML categories into detailed codes by adding a large number of keywords to attain quantitative and comprehensive information on how various technological properties and meaning-making activities intersect based on sign systems. Thirty-six codes were defined (

Table 4. Number of refined codes for each grouped code.

Grouped Codes	# of Refined Codes (36)
Technologies	6
Technologies’ functional properties	13
Technologies’ semiotic properties	1
Modes of representation	5
Activities	11

and Supplementary File S4).

The program nCoder [63] offered an efficient and secure procedure to map the occurrence of the ML details in the text data quantitatively through the 36 codes. With machine-learning techniques, nCoder enables the assessment of inter-rate reliability (IRR) and the validity of the codes. By including two human raters (H1/H2) and an automated classifier (AC), randomly selected lines (first segmentation) for each code were examined. In cases when the IRRs were below the thresholds (kappa > 0.9, rho < 0.05), the researchers used social moderation [55] to discuss the codes in relation to different keywords for qualitative refinement and entailed try-outs with new test sets in nCoder. These processes were manually conducted by notetaking of content keywords (instead of using the “update classifier” function in nCoder) to re-run test sets, which resulted in refining codes. Thirty-eight percent of the total number of coded lines (N = 680) were tested in the last three test sets. Thus, the agreements between H1 and AC, H1 and H2, and H2 and AC, respectively, were measured on all codes with a Cohen’s kappa of >0.9. The codebook offers more information on the statistics, codes, keywords, and examples (Supplementary File S4).

2.6. Analysis with Epistemic Network Analysis

Epistemic network analysis (ENA; [59]) was also used to map the MLs by visualizing their positions and connections. To use the ENA, the coded data were exported from nCoder to ENA Web Tool v1.7.0 [64]. The ENA could then operationalize the data into the different ML connections and display how the MLs varied in frequency across the three students into network models. Modelling with ENA is done by setting a stanza window that allows algorithms to operate mechanically in the data and compute connections between coded lines (first segmentation), aggregated based on the conversation setting (second and third segmentations) for each unit of analysis (the students, described in, e.g., [59], see Table 3). The segmentation was a vital part of the content analysis and ML derivation because the stanza window setting is used to determine how the codes in the data set are mechanically addressed based on that. The moving stanza window was set over the whole conversation and infinite stanza. The stanza window of size 4 also enabled ENA to seizure the frequency of code co-occurrences within the closer temporal/spatial context (in the text data). However, the decision on how to segment and which settings to model data in ENA must be based on qualitative and contextual estimations [60].

3. Result and Analysis

In this section, we present the findings on how primary school students use digital technologies in the classroom. To do so, we traced how cognitive processes are transformed and realized through students’ interpretation of the shifts in symbolic sign systems into different meaning-making strategies. ML mapping in ENA can help identify the semiotic shifts between human and technological sign systems and allow the potentials and challenges for students’ interpretation to be synthesized. In the first part, we focus on how the semiotic shifts varied the semiotic activity system among the three students. The second part outlines the semiotic shifts in the sign systems of the composite interfaces in relation to the transactions among the students’ cognitive processes.

The three students’ interactions with digital technologies are visualized in the ENA projection spaces (Figure 2, Figure 3, Figure 4, Figure 5, Figure 6 and Figure 7), in which each colour represents a student (Student 1 in red colour—laptop use, student 2 in blue colour—tablet use, and student 3 in purple colour—smartphone use). The codes related to the groups and refined MLs were connected via network graphs. Two signals were used in the analysis: (1) the position of the ML codes and (2) the weighted connection (the line thickness) in each network accounting for the relative frequency of co-occurrence between codes.

3.1. Sign Systems Variously Realizing the Semiotic Activity System

The composite interfaces prompted certain sign systems, which, together with the other sign systems prompted from the classrooms’ social space, formed a momentary pattern of reciprocity that realized specific semiotic activity systems. In this section, these sign systems’ prompts, related to the MLs activities and technologies for each student, are identified and analysed through mean network graphs (Figure 2, Figure 3 and Figure 4). The composite interfaces were configured with software and coded based on how they were modally expounded in the meaning-making activities (e.g., Software.Visual, expounding images and moving images), whereas the activities were focused on a range of events such as individual work, collaboration, and subjects.

As showed in Figure 2, student 1 was an active part in establishing the activity system in which the technological prompts in the composite interface with the laptop were rather backgrounded in a range of classroom activities (strong ties between the individual, collaborative, production, and consumption nodes). The weaker connection to technological sign systems limited the technological impact on the activities (e.g., weak ties to the node Software.Visual— “images” and “moving images”). This finding is different from those of student 2’s tablet use, as illustrated in Figure 3. The tablet was mostly used in individual production and consumption activities that foregrounded linguistically prompted sign systems (indicated by the ties between the nodes Activities.Individual, Consumption, Production, and Software.Linguistic). The main activities noted in the observations—“writing” and “reading”—were conveyed individually, in that the social space did not stimulate/prompt sign systems (e.g., inactive collaborative node) to support multimodal meaning-making. In student 3’s use of the smartphone (Figure 4), the network suggests a balance between the human and technological sign system prompts. The networks indicated that the sign systems prompted by the composite interfaces were integrated in both individual consumption (the Activities.Individual and consumption nodes) such as “reading” (node Software.Linguistic) from the device, and shared assignments and discussions (Activities.Collaborative/Production).

These findings put forward a synthesis of possibilities and challenges arising in students’ engagement with and interpretations of the semiotic shifts between human and technological sign systems. These shifts were explicated by modelling the ML’s activities and technologies’ configurations in composite interfaces into various technology-assisted learning activities. The sign systems demand the actors’ attention and interest in selecting and internalizing the sign systems for interpretation that are most apt for different communicational and meaning-making purposes [7], which seem to compete with each other. For instance, related to the interface configured with the tablet, the individually planned activities limited student 2’s agentive selection and internalization of the sign systems in meaning-making to constructing the social space as well as the use of the technologies. The technological sign systems (only linguistic activation) were backgrounded in the writing. The activities became a matter of single-handedly identifying which sign systems to act upon, which differed from student 3’s smartphone use. The smartphone configuration with rapid shifts among sign systems seemed to provide more transient engagement in meaning-making through the volatile modes of “speech” in “discussions”, as noted in the video recordings. It seems that the prompts in the social space and the technologies’ “material stuff” (e.g., text) were equally engaged. Either the technological sign systems are limited by the social space or vice versa, as seen in student 1’s use of the laptop, in which the social space “took over” the semiotic shifts’ capacities. If the activity system does not activate the technological properties, the technologies will limitedly enhance multimodal teaching and learning activities, which then restrict the students’ sense of how technologies can support their meaning-making.

3.2. Realizing Meaning-Making through the Technologies’ Properties—The Semiotic Shifts

The refined codes modelled in Figure 5, Figure 6 and Figure 7 provide additional information on the symbolic sign systems’ semiotic shifts with a specific focus on mapping the composite interfaces’ prompts. These sign systems are interpreted and transacted variously through the students’ cognitive processes of their actions and sign-making, as visible in various modes of representation (into new sign systems). Note that the software configurations of the respective composite interfaces presented in the previous section were not included in Figure 5, Figure 6 and Figure 7 because they would have complicated the visualizations.

This analysis focuses on the layers of functional properties including the representations of functions (RoF in the networks) in the technological design, the actions (e.g., Actions.Primary.Level in the networks), and the modes of representation (MoR in the networks). We explore which symbolic sign systems were foregrounded and how they were used, given attention to, and produced by the students on the composite interfaces. We must describe several codes to help explain the figures. For instance, RoF.Virtual is related to symbolic sign systems on the interface that represent the phenomena in the real world but virtually conveyed, while RoF.Physical is related to symbolic sign systems that are physically manifested on the composite interfaces, such as the screen’s size and the keyboard features. MoR.Linguistic is related to the students’ explication of symbolic sign systems in language-based productions (text and speech), while actions were grouped into different levels based on Wartofsky’s [27] taxonomy.

The analysis of semiotic shifts focused on the composite interfaces’ inherent meaning-making potentials for meaning-making [9] from four aspects: (1) interpretation preferences; (2) shifts between the physical and virtual; (3) shifts in the switch between different technologies and operating systems; and (4) shifts in the students’ cognitive processes through transformative actions.

3.2.1. Interpretation Preferences

Student 1 used the laptop configuration (Figure 5) in which symbolic sign systems shaped as virtual representations of functions (with icons, symbols, and text) seemed to attract attention and interpretive efforts to “buttons” (Figure 8), which means that actions to “push” and “press” were conveyed to complete and organize undertakings effectively at a primary and secondary level. Student 1 said: “Additionally, then there is the window tab there—press it, or they, eeee… Press the cross symbol there. It could have been a little bigger maybe… because if I use a mouse and so, then, when I want to click on the cross, I get outside and so, so I click on that tab window instead”.

The same use pattern was noted in the video recordings of student 2’s and 3’s use of the tablet and smartphone (Figure 6 and Figure 7), in which “tapping” became the mode of action through “touch” [65] on the screens’ virtual “button” features at a primary level.

It seems that the functional properties prompted in constant semiotic shifts across the different composite interfaces worked similarly, with most of the interpretation preferences seeming to culminate into operative actions and “easy” manoeuvring, associated with the visualization of the “tool” regardless of activity orientation and task design.

3.2.2. Shifts between the Physical and Virtual and Technical

Some possibilities and challenges were observed related to the symbolic sign system prompts that occurred in the same manner across the composite interfaces, regardless of user preferences. One is related to the trade-off between the function’s representation and the technical function. An example was given in an interview with a student: “Mm, yes, it says differently on the boxes and then, when you access it, there is one that you can click on”. The interpretation of how the functional properties were represented (as a box) symbolically transacts at a lower level of actions (to click) by using the practical intelligence in acting on the “tool” through activating the hardware’s technical capacities to access a virtual space.

Another one is, from a semiotic perspective, that the symbolic RoF is constantly manifested into physical and virtual visualizations across composite interfaces and accessories (Figure 8). The fluctuations also occur in symbolic signs when they can be interpreted as a physical object or a symbol. These shifts affect engagement associated with symbolic interpretations. When mediated via the students’ representation, their previous experiences put the sense of real up for interpretation. For instance, student 1 said: “The [laptop keyboard] is larger [than the tablet], and it’s like that—that is, to tap the [tablet]—and then I think it’s nice to click on real tangents. So, it’s more that the [keys] are close to each other [on the tablet] and, occasionally, there is a place, the real tangent [on the laptop] where you can make capital letters and so on. Additionally, I think it is very noisy because when you type A or 1, that is left [on the keyboard]; then, occasionally you press it, and it is very noisy”.

This example shows that the virtual and physical keys across composite interfaces are both regarded as having affordances as “physical” objects and conveying the action of typing via the first visualization, which encounter the students in the shape of a “tool” in the interpretation of these functions’ symbolic signs. Yet, the physical keys were preferred, related to the sense of directly manipulating the keyboard’s capacities rather than managing virtual keys by tapping on the touch screen.

3.2.3. Semiotic Shifts When Switching among Technologies and Operating Systems

Switches among different technologies and operating systems shift the symbolic sign systems to prompt the same functions by different visualizations, e.g., shapes, colours, and forms, affecting the users’ interpretation of these functions. In the interview, student 2 explained how he negotiated the size and shape of the virtual keyboards when switching between hardware technologies in relation to the primary action of typing text: “However, tablets are a little bigger [compared to smartphones] so you have to, like… [the student held up their arms as if they were holding a tablet in enlarged horizontal mode and typing with their thumbs on the keyboard]”. This also occurred in the video recordings where, for instance, the “add a document” button shifted between blue, grey, and orange, often with different images or signs (a document or a plus sign; see Figure 9) across different software. Another example of technological activation occurred when one student combined the action of placing a label with the interpretation of the semiotic shifts in interactive functions accompanying the cursor: “The student tried to assign the names of the countries in the right place by using the mouse-over function, which highlights different spots”.

3.2.4. The Technological Processes Rendered Shifts in Students’ Transformative Actions

Some challenges occurred when the students interpreted the multiple prompted symbolic sign systems, which brings to the fore the complexity of how the students transacted their cognitive processes through the technological properties. The networks (Figure 5, Figure 6 and Figure 7) demonstrate that the functional and the semiotic properties on the tertiary mediating level expounding higher levels of actions and sign-making are rather backgrounded. This finding might indicate that the technological sign systems’ meaning potentials and affordances were not utilized extensively. The three students seemed to externalize linguistic and multimodal modes of representation such as “writing”, “gaze”, and “images” in which the technological sign systems indicated by the MoR-nodes’ detachment was also rather backgrounded. Instead, the outcomes in modes of representation might be more related to the engagement with sign system prompts existing in the social space (student 1 and 3).

The ties in the networks to the primary and secondary online levels of mediation imply that the action level is evoked by how a functions’ virtual (RoF.Virtual/RoF.Virtual.Buttons) visualizations are realized. For example, students usually use “click”, “log in”, “turn off”, or “access” on the primary level of mediation to carry out their actions through the functions. Sometimes, the mediation transpires to the secondary level of action (Students 1 and 2), observed as “sharing” or “adding”. This shows that the functional properties acting as “tools” were more likely mediating students’ cognitive processes. The offline tertiary mediating level, i.e., engaging the “material stuff” in sign systems to support higher-level meaning-making was not realized by the combination of available functional and semiotic properties.

4. Discussion

In this paper, we have explored how students’ meaning-making was transformed through their interpretations of and engagement with the symbolic sign system prompts available in the social space and through composite interfaces.

One initial conclusion is that the challenges for students’ meaning-making via technology use lie beyond the individual students and technologies, as seen through both the differences and similarities in these investigations. We observed this in the similar actions students transacted through differently configured interfaces and in their mediation through the same technological design features. This finding supports an understanding of the symbolic sign systems’ important role in meaning-making. The sign systems are also relevant to understand the communicative division between technologies and users in constructing the social space. The complex relationship among signs needs to be addressed in educational research as sign systems can offer different affordances and meaning-making potentials [9,12,44,45]. The social space’s design and configuration, the technologies, as well as the students’ interpretation efforts and engagement, are equal partakers that shape multifaceted and complex semiotic shifts and different prerequisites for meaning-making via technologies and sign systems [1,12,14,15,16]. Using a fine-grained multimodal lens, we will discuss the findings in relation to potentials and challenges for student’s meaning-making via technology use in the upcoming sections.

4.1. Interpretation of Sign Systems Realizing the Semiotic Activity System

As different symbolic sign systems became intermixed in technologies and social space, the semiotic shifts brought some challenges for the students’ interpretation and engagement. Either the sign systems prompted in the activities backgrounded the multimodal affordances of the composite interfaces in the students’ meaning-making (student 1), or the composite interfaces seemed to render individual actions with the technologies’ functional properties monotonous (student 2). These might limit the dimensions of the activities and constrain advanced meaning-making with the available technological sign systems (e.g., by activating only linguistic sign systems). Greater reciprocity was observed between human and technological prompts in the smartphone use, which balanced the semiotic activity system. From a semiotic technology perspective, the mobility and characteristics afforded by smartphones might be one factor promoting this kind of balance, as other studies have noted [30]. In addition, certain consumption activities seem to evoke the semiotic properties of the composite interface material stuff becoming integrated with social prompts in meaning-making, engaging a higher level of learning activities through collaborative discussions. Some questions of importance to discuss include how the different configurations of teachers’ and students’ meaning-making when using physical, virtual, and social learning affordances transform the potential learning activities, as well as how cohesive environments are created and compel to balance productive learning and teaching processes through different signs [21].

To facilitate students’ exploitation of the multimodal sign systems available in social spaces and composite interfaces, it is vital to recognize the power of symbolic sign system prompts. Thus, it is important to give students a fair chance to attend to these sign systems multimodally based on their different affordances and reshape them into productive meaning-making outcomes to enhance technology-assisted learning [1,14,16]. However, streamlined technology integration, activity planning, and advanced interpretation skills are needed to sift sign systems by backgrounding and foregrounding them to appraise what sign systems are more apt to realize meaning, regardless of the prompters’ origin [10,15,44]. Regarding the potential of symbolic sign systems for evoking higher psychological functioning [24], one question remains as to how different interpretations strategies and selections preferences can be promoted to handle these shifts in sign systems in a digitalized learning environment.

4.2. Cognitive Processes Evoked by Technological Properties—Some Potentials and Challenges

A focus on symbolic sign systems makes it possible to identify several potentials and challenges in technology-assisted learning environments, which are associated with the technologies’ inherent meaning potentials and their demands on students to channel the purposes of the sign systems [25].

Several implications are specifically noteworthy regarding how students come to engage with and interpret the sign systems of functional properties when performing different actions. The affordances of the functional properties are at the intersection between the visualized representations of the functions’ (the technological design) meaning potentials and technical capacities (cf. [17]). As noted in the findings, regardless of the interface configuration, the sign systems of the composite interface seemed to favour mechanical meaning-making performances by activating mostly lower-level actions in students’ cognitive processing. Related to this finding, we reflect on the following:

• The functions’ meaning potentials beyond the sense of the “tool” where not expanded when students engaged in reading off the representation [26]. The students forwarded their meaning-making through actions with virtual buttons.
• The representations were first-hand operationalizations of the users’ previous knowledge regarding what the function can technically do, rather than being a gateway for them to utilize sign systems that could help them develop abstract thinking skills, such as conveying learning evidence through the technologies’ semiotic properties.

Thus, the basic function of the “tool” to promote work mostly signified the symbolic sign systems across composite interfaces, properties, users, and activities. The sign systems meaning-potentials seemed to channel the students’ interpretation efforts and engagement along with functions’ potentials for actions [25], which backgrounded the sign systems’ different purposes and the affordances of different properties to evoke a sense of the material stuff [7]. The technological design then seems to background the higher psychological function and higher forms of cognition that are already active when interpreting the symbolic sign systems [24] and prevent more advanced forms of meaning-making to emerge. Another possible explanation for this is that semiotic shifts imposed by human and technological activation increase the complexity of the visualisations, which demand different interpreting efforts and engagement in analysing the different purposes of the symbolic sign systems. When complexity of symbolic sign systems increased, the first basic visualization prompted the students to activate the cognitive processes of practical intelligence and skills in managing the functions’ technical capacities.

An additional potential and challenge associated with the semiotic shifts is related to a different aspect of the “real”, which becomes blurred and induces a strategy for the user, in which the “physical” sense of the symbolic sign systems become superior. We have three reflections. (1) The “physical tool” is fundamentally unified with the performances achieved by the human body through actions and therefore is more easily evoked [27]. (2) According to Kress [7], “The movement of visual entities on the screen as an effect of a touch suggests control through action and more specifically is suggestive of physical handling and acting on the objects […] the perceived gap between virtual and real handling of objects is narrowed” (p. 188). (3) The physical/manual and semiotic/conceptual relate to how sign systems materialize in ways that make them “physically” manifested, which detaches their symbolic potentials in human awareness.

5. Conclusions and Implications

Combinations of different prompts from the composite interfaces and the social space set symbolic sign systems in flux. The prompts have unstable perceptions because they shift in sign–signifier–signified relationships and provoke a signification process involving multiple interpretation and engagement efforts. The quite monotonous use at lower levels of actions in the findings reflects a need to meticulously evaluate the technological sign systems meaning potentials, and how to evoke and promote students’ interpretation and engagement efforts in mediating shifts in symbolic sign systems’ meanings through means of their representations [27]. The action-oriented use promoted by the technological sign systems prevented the direct visualizations from transcending into exploitation of the “material stuff” and mirrored issues with direct symbolism [24]. The lower-level uses are then an effect of the semiotic shifts, and technology and task design manifested through the technologies’ semiotic regimes (cf. [66]). This suggests that if higher cognitive forms are already in motion, they could, from a bottom-up user-perspective, promote both actions and sign-making through the different potentials of the symbolic sign systems related to their technological properties. Simultaneously, the lower level of actions can be considered as preceding the development of higher-complexity forms of meaning-making [24]. However, they are often not recognized as a part of mediating learning with technologies in school, which need to be addressed as this study suggests.

The complexity of semiotic shifts among symbolic sign systems and the potentials and challenges evoked for students’ interpretation when interacting with composite interfaces are also important for educators to consider. Questions remain as to how different “interpretation efforts” associated with semiotic shifts can be understood as a vital contemporary 21st-century skill. Actors select sign systems based on their aptness for meaning-making [1] and not because one instance outcompeted the other. As symbolic sign systems can provide many opportunities for productive meaning-making, it is important for teachers and students to account for them in incorporating technologies to design and plan multimodal technology-assisted teaching and learning activities. Insufficiently streamlined semiotic technologies in adoption may illustrate why students interpret the visualizations of buttons and so on as means for executing individual work, rather than as leading to actions on the semiotic properties to promote sign-making through various modes of representation that can feed into and promote a technology-assisted social learning space. If more of the latter occurs, then different configurations of technologies and meaning-making activities can serve as models to support the advancement of meaning-making. Moreover, because the activities or learning design reveal the meaning-making through how different elements interact with each other and the users as part of a semiotic process, learning designs with more sign system cohesion can help identify what fluid communicative potentials are or are not taken up, and how variations impact the processes of balancing learning and teaching differently. Such insights can guide educators in selecting and implementing technologies in schools and in using strategies to promote active learning environments.

A fine-grained lens such as the ML framework helps with mapping and synthesizing detailed information about interactions among users, technologies, and social spaces, as well as with identifying certain potentials and challenges by addressing semiotic shifts in symbolic sign systems. The findings presented here are not generalizable, but they provide detailed and in-depth insights into students’ meaning-making via technology use, which can inform future ML explorations with e.g., learning analytics. A semiotic technology approach can be applied more generally in EdTech research because sign systems are the most common features encountered by students when interacting with everyday technologies.