Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Understanding technology use through multimodal layers: a research review

Understanding technology use through multimodal layers: a research review The current issue and full text archive of this journal is available on Emerald Insight at: https://www.emerald.com/insight/2056-4880.htm Understanding Understanding technology use the use of technology through multimodal layers: a research review Karoline Schnaider, Limin Gu and Oscar Rantatalo Department of Education, Umea University, Umea, Sweden Received 28 February 2020 Revised 18 June 2020 Abstract 21 August 2020 Accepted 23 August 2020 Purpose – The purpose of this study is to examine the use of digital technologies by teachers and students in teaching and learning from a multimodal layer perspective. Design/methodology/approach – The article reviews 64 studies on technology use. A content analysis based on the theoretical concepts of “multimodal layers” was used to synthesise previous research. Findings – The findings indicate that the use of technology in classroom practices by teachers and students is multifaceted and that transitions exist between technologies and sign-systems and are differently related to sign-making activities and thus constitute different uses. Between layers, traces can be made that connect the use of technology to differences in sign-making activities. Practical implications – A multimodal layer perspective on technology use is fruitful to understand what happens at the intersection of technology and human activities in school practices. Moreover, more attention to multimodal layers can inform future effective technology usage and design. Originality/value – The review offers comprehensive insights on how previous research has studied technology using multimodal layers as an analytical lens. Keywords Technology, Software, Education, Multimodal, Hardware, Layer Paper type Literature review Introduction Integration of digital technology into education has become a natural part of the school’s everyday work, which calls for a more comprehensive understanding of the relationship between their nature, the use and their users (Bezemer and Kress, 2016). Recent studies have focussed on technologies’ visual features related to sign-systems (i.e. semiotic resources/modalities), which link technologies, and technologies and individuals in use (Jewitt, 2017). However, so far, many studies have not been considering various components of the technologies, the combinations of these components in use, and the transitions between them that affect learning in different ways (Ravelli and van Leeuwen, 2018). There is also some confusion in research and school practice in terms of how meaning-making actually happens when using technology in teaching and learning, especially how to trace the connections between technologies, their properties and the outcomes (Kress et al., 2014). Jewitt (2002) acknowledged how technologies and sign-systems interrelate and vary in usage from the notion of transitions between mediums. She found that the visual medium of the hardware screen, combined with software CD-ROM, resulted in the formerly written text being transformed into a visual multimodal representation, traced in individuals’ sign-making. Sign-making is an ongoing process that, e.g. occurs when signs are prompted from the external world, interpreted by the individual and The International Journal of © Karoline Schnaider, Limin Gu and Oscar Rantatalo. Published by Emerald Publishing Limited. This Information and Learning Technology article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non- pp. 375-387 Emerald Publishing Limited commercial purposes), subject to full attribution to the original publication and authors. The full terms of 2056-4880 this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode DOI 10.1108/IJILT-02-2020-0020 transformed either into an inward meaning or outward materialization of new IJILT representation (Kress, 2010). 37,5 Similarly, Adami (2010) emphasized transitions and traces related to the functional and semiotic properties of smartphones. Sign-systems such as labels, keys in different colours and screens of different sizes were activated in meaning-making by touch and oral representations. More recently, these tenets were brought up by Jewitt et al. (2020), who found that there is a transition between technologies that is needed for shifting from visual to other sensory modalities. Furthermore, Adami (2014) explored transitions and traces in relation to software and found that the functional properties of software, such as anchors, make an interactive site for actions of accessing and transferring texts. Moreover, Vigild Poulsen and Kv ale (2018) studied Instagram and the transitions and traces between the use of functional properties of the icons to highlight parts of pictures with semiotic resources such as colours to convey meaning in visual modes of representation. The notions of transitions and traces appear central to understand technology use in sign- making activities. To obtain a more comprehensive image on such matters, the purpose of this research review is to map and synthesise how previous studies have investigated the use of technology in school settings from a multimodal layer perspective, with a special focus on the variations and connections in usage between technologies, technologies’ properties and human’s sign-making. A multimodal layer approach is developed in this paper for that purpose, based on a combination of the social semiotic multimodal approach proposed by, e.g. Bezemer and Kress (2016), Kress (2010) and Wartofsky’s (1973) artefact taxonomy, consisting of five main elements: technologies, technologies’ functional and semiotic properties, and modes of representations made by individuals in activities. The research question is, From a multimodal layer perspective, how has previous research explored teachers’ and students’ use of technologies in sign-making activities in K-12 schools? Theoretical outline – multimodal layers In this section, the approach of multimodal layers is presented, focussing on the five main elements. Technologies The use of technologies is related to artifacts, humans’ perception and interpretation of representations. Wartofsky (1973) argued that perception is a highly evolved human trait that not only refers to biological senses and projections on the retina but is also mediated by means of our representation, the artifacts we create, which refer to humans activities to engrave sign-systems in physical and symbolic form. When perceived and interpreted according to our socially and culturally shaped knowledge, they will evoke new representations and actions (Kress, 2010; Wartofsky, 1973). This is relevant to understand digital technologies as artifacts in this way, especially since previous research tends to separate the physical from the symbolic, i.e. software is often seen as not being physically manipulated in use, and hardware as a physical rather than a semiotic artifact. The separation of the semiotic features from the physical hardware have neglected the intimate interrelations between hardware and software technologies without acknowledging that various combinations of hardware and software prompt sign-systems differently and affect what is visualized. This has led to an underestimation of the inherent sign-systems of different artifacts that facilitate human needs through multimodal communication (Ravelli and van Leeuwen, 2018). Stressing the semiotic features of technology use offers an analytical lens to understand how the combined use of hardware and software in relation to sign- systems may contribute to shaping meaning (Djonov and van Leeuwen, 2013). Technologies’ functional properties Understanding Artifacts have different functions in use (Innis, 2013; Scheele, 2006). Wartofsky (1973) explained the use of the relationships between representations engraved in artifacts and their purposes and technology functions in use – taxonomized into primary, secondary and tertiary levels, here with focus on sign-systems (Innis, 2013; Vigild Poulsen, 2018). The functions are not fixed but in constant flux – transitions are made between the different levels related to the users’ activities with different technologies and their combinations that render affordances (Kaptelinin and Nardi, 2009). Wartofsky’s proposal of functional properties is significant to understand humans’ meaning-making when using multirepresentational digital technologies. The primary functional property is related to sign-systems that convey modes of practice and are linked to production and use (Cole, 1996; Wartofsky, 1973). This kind of representations evokes actions and work but is not immediately connected to patterns of thinking, and is hard to relate to more abstract forms of meaning-making such as learning (Adami, 2014; Wartofsky, 1973). The secondary function, as it preserves or transmits the mode of action by its sign-based properties, provides models and systems to organize production and work (Wartofsky, 1973), for instance, a template for organizing documentation in a specific way (Alnervik, 2018), or a PowerPoint for arranging text by the use of bullet points (Zhao and van Leeuwen, 2014). The tertiary function is more implicitly related to modes of action (Wartofsky, 1973). Cole (2019) asserted that the tertiary function is separated from its direct representational form and has an abstract meaning. For instance, it can be related to the online activity of using the colours inherent in the function of the technology’s toolbar, while attending to the offline intention of conveying abstract meaning through the resources hue or saturation (Kress, 2010). Technologies’ semiotic properties From the social semiotic perspective, it is important to address the relationships between technologies’ functions, the semiotic resources used and humans’ representations (Vigild Poulsen, 2018). The semiotic properties are something “in-between” that mediate the use of technologies on the tertiary functional level (Wartofsky, 1973) through making the semiotic resources available for semiotic production that goes beyond the designed functional properties. For example, using the semiotic properties of zooming to negotiate certain content in speech and writing. Semiotic properties are therefore the link that additionally enables observing the traces and transitions between technologies’ functions and the outcome in sign- making activities (Kress et al., 2014). Modes of representation Sign-systems engraved in physical and symbolic artefacts are the precondition for internal sign-making and development of cognitive processing and learning (Bezemer and Kress, 2016). This is a circular process that can be referred to as the ongoing human activities of internalization and externalization (Vygotskij, 1978) – signs are prompted by technologies and humans, internalized into mental processing and externalized into new signs. This process transforms the resources available, and the transformations are viewed as signs of learning (Kress, 2010). Representations made by individuals have traces related to the external sign-systems that were considered visually prominent and of interest in the use of technologies in own sign-making activities (Kress et al., 2014), in connection with the notions of foregrounding/backgrounding of information (van Leeuwen, 2005). Activities Activities relate to a context consisting of actors, social action and semiotic work (Kress, 2010). Contexts have a specific set of features and means of relating entities (e.g. technologies) that are distinct to that activity site (Norris and Jones, 2005). Kress and van Leeuwen (2001) IJILT related the means to the concept of discourse, which is the knowledge and entities developed 37,5 in a specific social context that is appropriate for and maintained by the actors in that environment. Activities in learning environments are enclosed by a subject as an example of a discourse (Jewitt, 2009) and instructional or learning design principles (Selander and Kress, 2017). Method To gain a comprehensive understanding of technology use in sign-making activities, this paper investigates how previous researchers have studied technology use in schools in relation to multimodal layers. The study design followed recommendations proposed by Gough et al. (2012): establishing the target of the research area and questions, creating a data log, formulating inclusion and exclusion criteria, conducting pilot searches, developing the search strategy and identifying relevant literature. These actions were followed by text searching, which in turn entailed the construction of refined selection criteria, as well as screening, assessment and selection of the appropriate articles. The selected research articles were coded and analysed with the multimodal layer framework. Search strategies The articles were retrieved using educationally oriented databases: Scopus, Eric and Academic Search Elite. The first cycle of searches was executed at the end of May 2019 through the Scopus database, rendering 352 articles, and the second cycle was carried out at the beginning of June, covering 622 articles from the Eric and Academic Search Elite databases. The main search terms used were “digital tools” and “digital resources”, as these were regarded as the most frequently and interchangeably addressed in research on digital technologies in education. Because these terms are commonly and widely used and refer to both hardware and software, they enabled collection of a large quantity of potential articles. Moreover, because of the breadth of studies addressing various aspects of “use” of technologies in teaching and learning, a decision was taken to exclude this term in the search string. As a result, the final search string was: ((digital AND tools OR digital AND resource) AND teacher AND student AND NOT “higher education” AND NOT university). Selection of articles To conduct a valid selection of articles, and to reach a consensus amongst the researchers, an iterative process was undertaken to refine the selection criteria. Initially, general inclusion and exclusion criteria were defined as connected to the purpose of the systematic review on peer-reviewed empirical articles in English, published from 2010 to 2019. From the two searches, 974 articles were added to Mendeley, and, as duplicates were found, the articles were reduced to 796. From the general selection criteria and by making judgements on the quality and relevance of the found studies, the process started with assessing the first 100 articles. The three authors independently read the abstracts and followed a two-step process, starting with the first 43 articles: (1) individually labelled them with “include”, “exclude” or “maybe” in an Excel document; (2) after discussion, a joint decision on the selection criteria was taken. The remaining 57 articles were independently assessed following the same procedure. This resulted in a joint decision on inclusion and exclusion of the articles displayed in Table 1. The remaining 696 articles’ abstracts were divided equally between the authors for inclusion or exclusion assessment based on the final selection criteria using the program Understanding Criteria Include Exclude the use of Digital technologies Digital hardware and software Analogue technologies technology Teacher and/or Teachers’ and in-service teachers’ use in Articles on digital literacy, computer and student usage teaching activities in schools connected software engineering, competence to their school practice courses/periods development and general training, and Student usage in learning activities technology adoption/adaptation Teachers’ and students’ joint use Education All subjects Libraries, leisure time, other work-related practices Research conducted All school forms between kindergarten The purpose is to investigate technology in education (K-12) and school year 12 use in K-12 schools with exclusion of higher education. Other contexts, such as from purely technical aspects, teacher surveys, within healthcare issues or from a context that only reflects use in private life/leisure time Articles Scholarly peer-reviewed, full-text Systematic reviews, reviews, shorter empirical studies and book chapters, articles from forums, columns and Table 1. qualitative and quantitative magazines Final selection criteria Rayyan. General exclusion categories were adapted during a second round, and articles in languages other than English, duplicate articles and shorter articles from forums, columns and journals that no longer exist were removed. This process resulted in 391 articles. To enhance reliability in the coding process, one of the authors selected five articles as good and less good examples of the selection criteria. After discussing these examples, a delimitation was made on articles according to the relevance to “digital tools” and “digital resources”, which resulted in 105 articles. The three authors divided the remaining 105 articles for full- text reading with the aim to review their correspondence with all the inclusion criteria, which resulted in 64 articles being included in this paper. Data analysis A simple version of content analysis (Silverman, 2006) was adopted for data coding in the program NVivo 12. The first step of the coding process used the main categories from the multimodal layers and paired them for analysis (visible in Tables 2–8): technologies (divided into categories of hardware and software), technologies’ functional properties (primary, secondary, tertiary), technologies’ semiotic properties (modalities/semiotic resources), modes of representation (modalities/semiotic resources) and activities. Combinations of hardware and software are important examples of technologies used in meaning-making, why the initial division into different hardware was the first categorization, under which the other layers were organized. All layers where examined from the principles of foregrounding and backgrounding of information (van Leeuwen, 2005), i.e. information explicitly and more frequently described was deemed foregrounded and coded, while other information less salient, deemed more backgrounded and overlooked. Results and analysis This section presents findings from the multimodal layers and the numbers of examples. Students’ use Rich data on students’ use of PCs (see table 2) related to mostly visual and linguistic software, the traces of primary functions of, for example, displaying and recording, to the use of the IJILT Multimodal layers Data n Examples 37,5 Technologies Personal computers: 29 PCs: Desktop computers, laptops, unnamed computers –hardware Technologies – software Visual: 30 Visual: iMovie, Photoshop, TED talks, Google Hangouts, YouTube, Pinterest Linguistic: 23 Linguistic: Wordle, PowerPoint, Wikis, blogs Unnamed/other: 15 Unnamed/other: web pages Multiplicity of modes : Multiplicity of modes: VoiceThread, Glogster, StoryKit Auditory: 8 Auditory: GarageBand, Audacity, podcasts Technologies’ Primary function: 15 Primary: record, scan, display on screen, mouse/ functional properties voiceover, keyboard, insert, import, transfer Secondary function: 13 Secondary: paste, edit/cut, hyperlink, navigate, bookmarks, pinning, hotkeys, zoom function Tertiary function: 5 Tertiary: manipulate visual display (toolbar), effects (speeding up/slowing down), typographic style Technologies’ semiotic 25 Colour, background, font, typographic elements, properties graphics, space, layout, shape Modes of representation Visual: 30 Visual: images, moving images, animations, photographs, graphs Linguistic: 22 Linguistic: word clouds, writing Multiplicity of modes : Multiplicity of modes: combination of auditory- 18 linguistic-visual-gestures Auditory: 16 Auditory: dialogue, music, sound effects Gestures: 4 Gestures: gesticulating, indicating with hands Activities Collaboration: 20 Collaboration: sharing, joint production STEM: 19 STEM: mathematical content and rules, science Literacy: 18 Literacy: writing/reading literature Individual work: 11 Individual work: managing tasks and assignments separately Note(s): Refers to number of mentioned examples in the articles (i.e. not number of articles) Software where more than one modality is foregrounded in use Table 2. Students’ use of PCs Humans’ representations where people clearly place emphasis on the use of more than one modality secondary functions to organize such actions, by editing and cutting videos and sound, and navigating content by using hotkeys was observed. These actions are used with the tertiary function of adding special effects in video making and manipulating visual components using semiotic properties such as colours, background, etcetera to convey specific meaning in mainly visual and linguistic modalities (Wartofsky, 1973). The affordances of the technologies’ sign-systems – spatiality and simultaneity (visual), kineikonic with combination of different modes such as sound in recording and sequencing (linguistic) – can therefore give clues to individuals’ semiotic actions and production in the same modalities (Kress, 2010; Kress et al., 2014). Unlike PCs, smartphones (see table 3) evoke other kinds of use and actualize transitions between modes of representation and technologies. Incorporated in mostly collaborative activities, visual and unnamed software such as web pages were observed in the use of primary functions, such as navigating and scrolling on the touchscreen to access information. Often, auditory modes of representation were generated in use, indicating that smartphones are not yet fully incorporated in other sign-making activities, which need to be manifested in time and space (e.g. linguistic) (Kress, 2010). Because it is mostly feasible to trace actions but not meanings from primary functions (Wartofsky, 1973), smartphones are considered in volatile involvement in speech, prior to engaging in other activities, not least shown in the absence of tertiary functional properties. Understanding Multimodal layers Data n Examples the use of Technologies – hardware Smartphones: 5 iPhones technology Technologies – software Visual: 2 Visual: AR animations Multiplicity of Multiplicity of modes: snapchat modes: 1 Unnamed/other: 2 Unnamed/other: Wikipedia, digital text Technologies’ functional Primary function: 10 Primary: scan, navigate/scroll, storage, camera, properties display, keypad, touchscreen Secondary: 2 Secondary: organize notes, photos Technologies’ semiotic –– properties Modes of representation Auditory: 6 Auditory: dialogue Linguistic: 3 Linguistic: reading Gaze: 3 Gaze: directing attention Visual: 2 Visual: animations, videos Touch: 1 Touch: interacting with system using fingers Activities Collaboration: 3 Collaboration: pair and group work discussion Table 3. Literacy: 2 Literacy: reading Students’ use of STEM: 1 STEM: science simulations smartphones In students’ collaborative use of IWBs (see table 4), yet other traces could be observed, and transitions were visible between properties and modes of representation. Unnamed software was used with the primary capabilities of the interactive board to write and draw, organized through the secondary function of hiding and revealing content. The tertiary function was used to layer typographic semiotic properties to convey meaning, such as that of graphic structure and volatile auditory modalities including speech and gestures, into multimodal compositions (Kress, 2010; Wartofsky, 1973). The transition and traces found in the use of tablets (see table 5) are similar to those found for PCs. A combination of visual and linguistic software was used, mainly for their primary functions such as displaying, and recording, followed by the secondary function of organizing modes of action such as taking photos, inserting and zooming. However, when the Multimodal layers Data n Examples Technologies – hardware IWBs: 4 IWBs: SMARTboard, Camtasia Technologies – software Unnamed: 7 Documents, notebooks, websites Technologies’ functional Primary function: 2 Primary: writing/drawing function, recording properties Secondary Secondary: manipulating objects, hiding and revealing function: 2 content Tertiary function: 2 Tertiary: drawing on board, linking and annotating photos/text, displaying and communicating Technologies’ semiotic 3 Typographic elements properties Modes of representation Gestures: 4 Gestures: pointing, dragging objects Auditory: 4 Auditory: talking Visual: 3 Visual: drawings, diagrams, images Linguistic: 3 Linguistic: writing, annotating words Multiplicity of Multiplicity of modes: combination of auditory-visual- modes: 1 gestures Table 4. Activities STEM: 2 STEM: mathematical and science content Students’ use of Literacy: 2 Literacy: annotating/discussing/producing text interactive Collaboration: 2 Collaboration: discussions whiteboards (IWBs) IJILT Multimodal layers Data n Examples 37,5 Technologies – hardware Tablets: 12 Tablets: iPads Technologies – software Visual: 7 Visual: Fotobabble, iMovie, GlogsterEdu, video Linguistic: 6 Linguistic: storybookmaker, worksheets Multiplicity of Multiplicity of modes: Poll, Padlet modes: 2 Auditory: 1 Auditory: podcasts Technologies’ functional Primary function: 5 Primary: dragging pointer, visualizing, recording properties Secondary Secondary: taking photos/inserting into organizer, function: 4 editing/shaping videos, zoom Tertiary function: 2 Tertiary: recording videos on narratives, tracking alphabetic inscriptions on touchscreen Technologies’ semiotic 3 Font, colour, typographic elements properties Modes of representation Linguistic: 6 Linguistic: words, writing Visual: 6 Visual: moving images, photos, drawings Auditory: 5 Auditory: dialogues, discussions Multiplicity of Multiplicity of modes: combination of auditory-linguistic- modes: 4 visual-gaze-touch Gaze: 2 Gaze: directing attention Touch: 2 Touch: touching the screen Activities STEM: 5 STEM: science Collaboration: 4 Collaboration: shared production Literacy: 3 Literacy: creating e-books Table 5. Students’ use of tablets Individual: 3 Individual: managing tasks and assignments separately visual, linguistic and auditory modes of representation come about in combined activities in which the recording function is used as a tertiary function in writing activities, the semiotic properties are used to alter fonts in text or engage with typographic elements to combine with the recordings into multimodal ensembles (Kress, 2010; Wartofsky, 1973). Teachers’ use Teachers’ use of PCs (see table 6) is mainly in instructional design, using their primary function to display and record to create and disseminate content to students (Selander and Kress, 2017). Because it is mostly feasible to trace actions but not meaning from primary functions (Adami, 2014), connections between software technologies and the modes of representation in the same modalities might provide insights into meaning created. Teachers’ use of IWBs (see table 7) is similar to students’ use and demonstrated by unifying the actors in learning design activities (Selander and Kress, 2017). The chain of layers enables traces from visual software used with the primary displaying function to organizing the visual properties by the secondary function to, for example, hide and reveal and illuminate important parts. By the tertiary spotlight function, combinations were made with certain semiotic properties, such as colours, to signify visual meaning that was collaboratively negotiated in volatile auditory modes of representation such as speech (Kress, 2010; Wartofsky, 1973). Teachers’ use of tablets (see table 8) is limited in previous research. Undertaken in mainly instructional design, auditory modes of representation complemented by gaze and gestures were used in engagement with the primary and secondary functions to display content to students and organize further learning actions by using, for example, the zoom function. When using the tertiary functions of making recordings in the design of learning material, multimodal affordances of the technologies were combined with similar modes of representation into multimodal compositions (Kress, 2010; Wartofsky, 1973). Understanding Multimodal layers Data n Examples the use of Technologies – hardware 5 PCs: Desktop computers, laptops, unnamed computers technology Technologies – software Multiplicity of Multiplicity of modes: Poll, VoiceThread, blogs modes: 9 Visual: 9 Visual: iMovie, YouTube Linguistic: 7 Linguistic: Socrative, PowerPoint, WhatsApp Unnamed: 7 Unnamed: Internet Auditory: 1 Auditory: podcasts Technologies’ functional Primary function: 3 Primary: display, drag-and-drop, cursor, record properties Technologies’ semiotic properties Modes of representation Visual: 6 Visual: images, graphs Linguistic: 6 Linguistic: text/communication Multiplicity of Multiplicity of modes: combination of auditory-linguistic- modes: 4 visual-gestures Auditory: 3 Auditory: speech, oral tutoring Activities Instructional Instructional design: presentation/demonstration/ design: 3 development of teaching content Table 6. Learning design: 1 Learning design: teacher and student shared design Teachers’ use of PCs Multimodal layers Data n Examples Technologies – hardware IWBs: 6 IWBs: SMARTboard, Camtasia Technologies – software Visual: 5 Visual: digital photos, videos, YouTube, Flipchart Unnamed: 4 Unnamed: DOC files, notebooks, websites, polls Technologies’ functional Primary function: 4 Primary: displaying content, handwriting recognition, properties screen captures Secondary function: 4 Secondary: linking to documents, flipping back to recover material, storing/embedding in files, hiding and revealing, layering to manipulate content, drag- and-drop to focus on salient features of representations Tertiary function: 1 Tertiary: spotlight function to highlight important parts of contents Technologies’ semiotic 8 Highlighting, colour, shading, size properties Modes of representation Auditory: 5 Auditory: discuss, talk Visual: 3 Visual: images, graphs Multiplicity of modes: Multiplicity of modes: combination of auditory- 3 linguistic-visual-gestures Linguistic: 1 Linguistic: writing on screen Gestures: 1 Gestures: pointing (at screen) Table 7. Activities Learning design: 5 Learning design: teacher and student shared design Teachers’ use of Instructional design: Instructional design: presentation/demonstration/ interactive 1 development of teaching content whiteboards (IWBs) Discussion Research on digital technologies in education that illuminates connections between digital hardware and software use is scarce. However, the research compiled in this review indicates that PCs and visual software are frequently employed, in favour of student use. Research on other technologies such as IWBs, tablets and smartphones is limited. Teachers’ use of smartphones has been absent. Another finding is that different technologies are usually IJILT Multimodal layers Data n Examples 37,5 Technologies – hardware Tablets: 6 Tablets: iPads Technologies – software Visual: 2 Visual: Explain Everything, SPARK, Vue Multiplicity of Multiplicity of modes: Skitch modes: 1 Technologies’ functional Primary function: 1 Primary: displaying content properties Secondary Secondary: using fingers to zoom in on content function: 1 Tertiary function: 1 Tertiary: recording written work, making videos Technologies’ semiotic 3 Highlighting, colour, underlining properties Modes of representation Auditory: 3 Auditory: conversations, speech, talk Multiplicity of Multiplicity of modes: combination of auditory-linguistic- modes: 2 visual-gestures-gaze Gaze: 2 Gaze: looking at technologies Gestures: 2 Gestures: pointing, nodding Visual: 1 Visual: moving images Activities Instructional Instructional design: presentation/demonstration/ design: 4 development of teaching content Table 8. Teachers’ use of tablets Learning design: 2 Learning design: teacher and student shared design incorporated into the same activities, which is why differences or similarities between activities and the other layers are difficult to distinguish, important to investigate further. In line with Jewitt (2009) on the visual affordances of technologies, the findings indicate a strong focus on visual, closely accompanied by linguistic modes of representation. An overall tendency is oriented towards multimodality. Gaze and gestures, and sensory modality of touch is depicted in relation to students’ use of IWBs, tablets and smartphones combined with visual, linguistic and multimodal software (Jewitt et al., 2020). Below, the notions of transitions and traces in relation to the multimodal layers will be stressed in discussing the findings. Traces According to Kress et al. (2014), if influencing factors are defined, proposed here by the framework of multimodal layers, traces between the technologies and their properties and humans’ modes of representation and activities can be illuminated to understand the outcome of use. Examples of traces can be found when the links between the layers are intact. For example, when a hardware is combined with a specific software, the primary functions used in work, and secondary functions used to organize the work, provide possibilities for the tertiary functions to add meaning to the actions. Links can be made between the sign-systems used in the functional properties and human interpretation of them; actions that will be additionally enhanced by the connection to semiotic properties. The choices made amongst all sign-systems can be related to the making of abstract meaning (Cole, 2019) and are relative to the affordances of the properties and their meaning potentials and how they are ultimately drawn upon. Yet, the traces give clues to which of all the multiple features are regarded as visually salient, choices that are integrated into mental processing and renewed and communicated outwardly in sign-making activities (Kress, 2010; Kress et al., 2014). The circular process of internalization and externalization of interpreting and transforming sign- systems constitutes learning in connection with technology use (Bezemer and Kress, 2016; Vygotskij, 1978). In this study, these kinds of traces are observed. For instance, in students’ use of PCs and tablets, properties are demonstrated to come about differently in the combination with visual and linguistic software, which is unlike the use of other technologies. Understanding Traces are found between the primary recording function and the secondary editing function, the use of as well as the tertiary function of manipulating various aspects of visual display by colour technology and typographic elements as examples of semiotic properties, which are linked to visual or linguistic modes of representation. That entails that, first, technologies and sign-systems handled in technology use do impinge on what meaning is subsequently formed, and their affordances and constraints impact on modal choices made by the individual (Kress, 2010; Kress et al., 2014). Second, the choices made by the individual incorporating several traits from the original input will be a new form meaning made. Hence, although it is not possible here to conduct a detailed analysis, it is important to understand how functional properties are used for action and how they merge with the semiotic properties, as well as how the sign- systems therefore are selected and used to highlight specific aspects of meaning – in particular, if certain functional properties can be paired with specific semiotic properties, and how they relate to meaning created in a specific context. When one layer is left out, the links between the layers are broken and makes traces harder to observe. It is clear that if only primary functional properties are used, the semiotic properties are left out, and difficulties in relating technology use to sign-making activities arise (Adami, 2014). Hence, the online actions associated with the use of properties cannot be related to the offline activities of meaning-making (Wartofsky, 1973). In summary, traces made between all layers, can give comprehensive clues to how technologies relate to the construction of meaning through the sign-systems, a recognition important to acknowledge in learning settings. Transitions The notion of transitions refers to the process of negotiation and selection of sign-systems inherent in all technologies and used in sign-making activities. Transitions provide insight on how sign-systems move between the multimodal layers in use (Kaptelinin and Nardi, 2009). Technologies and their properties are therefore backgrounded or foregrounded in use (van Leeuwen, 2005), which entails various attentions paid to their affordances and meaning potentials, which in turn affect meaning-making (Ravelli and van Leeuwen, 2018). Transitions are implicitly recognized by previous researchers as a common phenomenon but have not yet been explored in-depth. This review shows that transitions exist in relation to both modes of representation and properties, and they move between technologies and users. Examples are students’ use of visual software, where the tertiary function to manipulate the visual display by semiotic properties such as colours and layout was used by teachers to organize content on a secondary level. In relation to hardware, the affordances of the screen to display content related to the primary function in the use of PCs and the tertiary function in the use of IWBs. More specifically, this means that the offline shape and size of the screen related to the online displaying function in hardware, or the online highlighting functions in software, to convey offline meaning in the use of colours and layout (Wartofsky, 1973) were all foregrounded sign- systems and conveyers of meaning in the use of certain technologies, and overlooked in the use of others. These findings contribute to an understanding of the complexity of sign- making in the use of technologies and why technologies can support diverse actions and sign- making activities in some cases and for some users but not others. Conclusion and implications Mapping technology use by focussing on the multimodal layers can provide comprehensive insights into traces – how different sign-systems are engraved in combinations of technologies, drawn upon variously by individuals, and therefore lead to variations in actions and sign-making activities. The multimodal layers approach can also contribute to IJILT clues regarding what role transitions play in representational signs between technologies, 37,5 levels of functions and properties, and how such shifts can be understood when traces are linked to individuals’ choices of resources to make meaning. The importance of combining technologies’ inherent properties to the patterns of human action and thinking will help us to understand how technology use relates to sign-making activities (Jewitt, 2017). The shifts in functional and semiotic properties between technologies need to be addressed when selecting appropriate technologies (Vigild Poulsen, 2018). The properties are multifaceted, and people make different choices regarding foregrounding or backgrounding information (van Leeuwen, 2005), therefore contributing to variations in meaning-making by the users. This also means that hardware and software technologies are sectioned in relation to the multimodal layers and to the user’s need to represent meaning. Evidence has also been provided that the primary functional level is more commonly used in teaching and learning. Wartofsky (1973) and Adami (2014) pointed out that the solitary use of primary functions could be problematic because it promotes actions rather than meaning- making. Hence, it is important to consider the role of primary functions in a learning situation, as they have slight relations to the other layers and poor connections to sign-making activities. Thus, teaching and learning undertaken through technology use from mainly a primary function is troublesome, why further explorations are needed on how transitions between technologies’ functional properties can give clues to other kinds of multiple use. References Adami, E. (2010), “A social semiotic analysis of mobile devices: interrelations of technology and social habitus”, in Pachler, N., Bachmair, B. and Cook, J. (Eds), Mobile Learning, Springer, Boston, MA, pp. 185-204. Adami, E. (2014), “What’s in a click? A social semiotic framework for the multimodal analysis of website interactivity”, Visual Communication, Vol. 14 No. 2, pp. 133-153. Alnervik, K. (2018), “Systematic documentation: structures and tools in practice of communicative documentation”, Contemporary Issues in Early Childhood, Vol. 19 No. 1, pp. 72-84. Bezemer, J. and Kress, G. (2016), Multimodality, Learning and Communication a Social Semiotic Frame, Routledge, London. Cole, M. (1996), Cultural Psychology. A Once and Future Discipline, Harvard university press, Cambridge, Mass. Cole, M. (2019), “Re-covering the idea of a tertiary artifact”, in Edwards, A., Fleer, M. and Bøttcher, L. (Eds), Cultural-Historical Approaches to Studying Learning and Development, Perspectives in Cultural-Historical Research, Springer, Singapore, Vol. 6, pp. 303-323. Djonov, E. and van Leeuwen, T. (2013), “Multimodality and software”, in Chapelle, C.A. (Ed.), The Encyclopedia of Applied Linguistics, Wiley-Blackwell, Wiley, Chichester, pp. 1-5. Gough, D., Oliver, S. and Thomas, J. (2012), An Introduction to Systematic Reviews, SAGE, Los Angeles, Ca. Innis, R.E. (2013), “Semiotics of technology”, in Olsen, J.K.B., Pedersen, S.A. and Hendricks, V.F. (Eds), A Companion to the Philosophy of Technology, Blackwell, West Sussex, pp. 206-212. Jewitt, C. (2002), “The move from page to screen: the multimodal reshaping of school English”, Journal of Visual Communication, Vol. 1 No. 2, pp. 171-196. Jewitt, C. (2009), Technology, Liceracy and Learning. A Multimodal Approach, Routledge, Oxon. Jewitt, C. (Ed.) (2017), Routledge Handbook of Multimodal Analysis, Routledge, London. Jewitt, C., Price, S., Mackley, K., Giannoutsou, N. and Atkinson, D. (2020), Interdisciplinary Insights for Digital Touch Communication, Springer, Cham. Kaptelinin, V. and Nardi, B.A. (2009), Acting with Technology. Activity Theory and Interaction Design, Understanding The MIT Press, Cambridge, Mass. the use of Kress, G. (2010), Multimodality. A Social Semiotic Approach to Contemporary Communication, technology Routledge, London. Kress, G. and van Leeuwen, T. (2001), Multimodal Discourse: The Modes and Media of Contemporary Communication, Arnold, London. Kress, G., Jewitt, C., Ogborn, J. and Tsatsarelis, C. (2014), Multimodal Teaching and Learning, The Rhetorics of the Science Classroom Bloomsbury, London. Norris, S. and Jones, R.H. (2005), Discourse in Action. Introducing Mediated Discourse Analysis, Routledge, Oxon. Ravelli, L.J. and van Leeuwen, T. (2018), “Modality in the digital age”, Visual Communication, Vol. 17 No. 3, pp. 277-297. Scheele, M. (2006), “Function and use of technical artefacts: social conditions of function ascription”, Studies in History and Philosophy of Science, Vol. 37 No. 1, pp. 23-36. Selander, S. and Kress, G. (2017), Design F€or L€ arande. Ett Multimodalt Perspektiv, Studentlitteratur, Lund. Silverman, D. (2006), Interpreting Qualitative Data: Methods for Analyzing Talk, Text and Interaction, Sage, London. van Leeuwen, T. (2005), Introducing Social Semiotics, Routledge, London. Vigild Poulsen, S. (2018), “Becoming a semiotic technology – a historical study of Instagram’s tools for making and sharing photos and videos”, Internet Histories, Vol. 2 Nos 1-2, pp. 121-139. Vigild Poulsen, S. and Kv ale, G. (2018), “Studying social media as semiotic technology: a social semiotic multimodal framework”, Social Semiotics, Vol. 28 No. 5, pp. 700-717. Vygotskij, L. (1978), Mind in Society: The Development of Higher Psycological Processes, in Cole, M., John-Steiner, V., Scribner, S. and Souberman, E. (Eds), Harvard University Press, Cambridge, Mass. Wartofsky, M.W. (1973), Perception, Representation, and the Forms of Action: Toward an Historical Epistemology, Springer, Dordrecht. Zhao, S. and van Leeuwen, T. (2014), “Understanding semiotic technology in university classrooms: a social semiotic approach to PowerPoint-assisted cultural studies lectures”, Classroom Discourse, Vol. 5 No. 1, pp. 71-90. Corresponding author Karoline Schnaider can be contacted at: karoline.schnaider@umu.se For instructions on how to order reprints of this article, please visit our website: www.emeraldgrouppublishing.com/licensing/reprints.htm Or contact us for further details: permissions@emeraldinsight.com http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png The International Journal of Information and Learning Technology Emerald Publishing

Understanding technology use through multimodal layers: a research review

Loading next page...
 
/lp/emerald-publishing/understanding-technology-use-through-multimodal-layers-a-research-ISiiqgvFzp
Publisher
Emerald Publishing
Copyright
© Karoline Schnaider, Limin Gu and Oscar Rantatalo
ISSN
2056-4880
DOI
10.1108/ijilt-02-2020-0020
Publisher site
See Article on Publisher Site

Abstract

The current issue and full text archive of this journal is available on Emerald Insight at: https://www.emerald.com/insight/2056-4880.htm Understanding Understanding technology use the use of technology through multimodal layers: a research review Karoline Schnaider, Limin Gu and Oscar Rantatalo Department of Education, Umea University, Umea, Sweden Received 28 February 2020 Revised 18 June 2020 Abstract 21 August 2020 Accepted 23 August 2020 Purpose – The purpose of this study is to examine the use of digital technologies by teachers and students in teaching and learning from a multimodal layer perspective. Design/methodology/approach – The article reviews 64 studies on technology use. A content analysis based on the theoretical concepts of “multimodal layers” was used to synthesise previous research. Findings – The findings indicate that the use of technology in classroom practices by teachers and students is multifaceted and that transitions exist between technologies and sign-systems and are differently related to sign-making activities and thus constitute different uses. Between layers, traces can be made that connect the use of technology to differences in sign-making activities. Practical implications – A multimodal layer perspective on technology use is fruitful to understand what happens at the intersection of technology and human activities in school practices. Moreover, more attention to multimodal layers can inform future effective technology usage and design. Originality/value – The review offers comprehensive insights on how previous research has studied technology using multimodal layers as an analytical lens. Keywords Technology, Software, Education, Multimodal, Hardware, Layer Paper type Literature review Introduction Integration of digital technology into education has become a natural part of the school’s everyday work, which calls for a more comprehensive understanding of the relationship between their nature, the use and their users (Bezemer and Kress, 2016). Recent studies have focussed on technologies’ visual features related to sign-systems (i.e. semiotic resources/modalities), which link technologies, and technologies and individuals in use (Jewitt, 2017). However, so far, many studies have not been considering various components of the technologies, the combinations of these components in use, and the transitions between them that affect learning in different ways (Ravelli and van Leeuwen, 2018). There is also some confusion in research and school practice in terms of how meaning-making actually happens when using technology in teaching and learning, especially how to trace the connections between technologies, their properties and the outcomes (Kress et al., 2014). Jewitt (2002) acknowledged how technologies and sign-systems interrelate and vary in usage from the notion of transitions between mediums. She found that the visual medium of the hardware screen, combined with software CD-ROM, resulted in the formerly written text being transformed into a visual multimodal representation, traced in individuals’ sign-making. Sign-making is an ongoing process that, e.g. occurs when signs are prompted from the external world, interpreted by the individual and The International Journal of © Karoline Schnaider, Limin Gu and Oscar Rantatalo. Published by Emerald Publishing Limited. This Information and Learning Technology article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non- pp. 375-387 Emerald Publishing Limited commercial purposes), subject to full attribution to the original publication and authors. The full terms of 2056-4880 this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode DOI 10.1108/IJILT-02-2020-0020 transformed either into an inward meaning or outward materialization of new IJILT representation (Kress, 2010). 37,5 Similarly, Adami (2010) emphasized transitions and traces related to the functional and semiotic properties of smartphones. Sign-systems such as labels, keys in different colours and screens of different sizes were activated in meaning-making by touch and oral representations. More recently, these tenets were brought up by Jewitt et al. (2020), who found that there is a transition between technologies that is needed for shifting from visual to other sensory modalities. Furthermore, Adami (2014) explored transitions and traces in relation to software and found that the functional properties of software, such as anchors, make an interactive site for actions of accessing and transferring texts. Moreover, Vigild Poulsen and Kv ale (2018) studied Instagram and the transitions and traces between the use of functional properties of the icons to highlight parts of pictures with semiotic resources such as colours to convey meaning in visual modes of representation. The notions of transitions and traces appear central to understand technology use in sign- making activities. To obtain a more comprehensive image on such matters, the purpose of this research review is to map and synthesise how previous studies have investigated the use of technology in school settings from a multimodal layer perspective, with a special focus on the variations and connections in usage between technologies, technologies’ properties and human’s sign-making. A multimodal layer approach is developed in this paper for that purpose, based on a combination of the social semiotic multimodal approach proposed by, e.g. Bezemer and Kress (2016), Kress (2010) and Wartofsky’s (1973) artefact taxonomy, consisting of five main elements: technologies, technologies’ functional and semiotic properties, and modes of representations made by individuals in activities. The research question is, From a multimodal layer perspective, how has previous research explored teachers’ and students’ use of technologies in sign-making activities in K-12 schools? Theoretical outline – multimodal layers In this section, the approach of multimodal layers is presented, focussing on the five main elements. Technologies The use of technologies is related to artifacts, humans’ perception and interpretation of representations. Wartofsky (1973) argued that perception is a highly evolved human trait that not only refers to biological senses and projections on the retina but is also mediated by means of our representation, the artifacts we create, which refer to humans activities to engrave sign-systems in physical and symbolic form. When perceived and interpreted according to our socially and culturally shaped knowledge, they will evoke new representations and actions (Kress, 2010; Wartofsky, 1973). This is relevant to understand digital technologies as artifacts in this way, especially since previous research tends to separate the physical from the symbolic, i.e. software is often seen as not being physically manipulated in use, and hardware as a physical rather than a semiotic artifact. The separation of the semiotic features from the physical hardware have neglected the intimate interrelations between hardware and software technologies without acknowledging that various combinations of hardware and software prompt sign-systems differently and affect what is visualized. This has led to an underestimation of the inherent sign-systems of different artifacts that facilitate human needs through multimodal communication (Ravelli and van Leeuwen, 2018). Stressing the semiotic features of technology use offers an analytical lens to understand how the combined use of hardware and software in relation to sign- systems may contribute to shaping meaning (Djonov and van Leeuwen, 2013). Technologies’ functional properties Understanding Artifacts have different functions in use (Innis, 2013; Scheele, 2006). Wartofsky (1973) explained the use of the relationships between representations engraved in artifacts and their purposes and technology functions in use – taxonomized into primary, secondary and tertiary levels, here with focus on sign-systems (Innis, 2013; Vigild Poulsen, 2018). The functions are not fixed but in constant flux – transitions are made between the different levels related to the users’ activities with different technologies and their combinations that render affordances (Kaptelinin and Nardi, 2009). Wartofsky’s proposal of functional properties is significant to understand humans’ meaning-making when using multirepresentational digital technologies. The primary functional property is related to sign-systems that convey modes of practice and are linked to production and use (Cole, 1996; Wartofsky, 1973). This kind of representations evokes actions and work but is not immediately connected to patterns of thinking, and is hard to relate to more abstract forms of meaning-making such as learning (Adami, 2014; Wartofsky, 1973). The secondary function, as it preserves or transmits the mode of action by its sign-based properties, provides models and systems to organize production and work (Wartofsky, 1973), for instance, a template for organizing documentation in a specific way (Alnervik, 2018), or a PowerPoint for arranging text by the use of bullet points (Zhao and van Leeuwen, 2014). The tertiary function is more implicitly related to modes of action (Wartofsky, 1973). Cole (2019) asserted that the tertiary function is separated from its direct representational form and has an abstract meaning. For instance, it can be related to the online activity of using the colours inherent in the function of the technology’s toolbar, while attending to the offline intention of conveying abstract meaning through the resources hue or saturation (Kress, 2010). Technologies’ semiotic properties From the social semiotic perspective, it is important to address the relationships between technologies’ functions, the semiotic resources used and humans’ representations (Vigild Poulsen, 2018). The semiotic properties are something “in-between” that mediate the use of technologies on the tertiary functional level (Wartofsky, 1973) through making the semiotic resources available for semiotic production that goes beyond the designed functional properties. For example, using the semiotic properties of zooming to negotiate certain content in speech and writing. Semiotic properties are therefore the link that additionally enables observing the traces and transitions between technologies’ functions and the outcome in sign- making activities (Kress et al., 2014). Modes of representation Sign-systems engraved in physical and symbolic artefacts are the precondition for internal sign-making and development of cognitive processing and learning (Bezemer and Kress, 2016). This is a circular process that can be referred to as the ongoing human activities of internalization and externalization (Vygotskij, 1978) – signs are prompted by technologies and humans, internalized into mental processing and externalized into new signs. This process transforms the resources available, and the transformations are viewed as signs of learning (Kress, 2010). Representations made by individuals have traces related to the external sign-systems that were considered visually prominent and of interest in the use of technologies in own sign-making activities (Kress et al., 2014), in connection with the notions of foregrounding/backgrounding of information (van Leeuwen, 2005). Activities Activities relate to a context consisting of actors, social action and semiotic work (Kress, 2010). Contexts have a specific set of features and means of relating entities (e.g. technologies) that are distinct to that activity site (Norris and Jones, 2005). Kress and van Leeuwen (2001) IJILT related the means to the concept of discourse, which is the knowledge and entities developed 37,5 in a specific social context that is appropriate for and maintained by the actors in that environment. Activities in learning environments are enclosed by a subject as an example of a discourse (Jewitt, 2009) and instructional or learning design principles (Selander and Kress, 2017). Method To gain a comprehensive understanding of technology use in sign-making activities, this paper investigates how previous researchers have studied technology use in schools in relation to multimodal layers. The study design followed recommendations proposed by Gough et al. (2012): establishing the target of the research area and questions, creating a data log, formulating inclusion and exclusion criteria, conducting pilot searches, developing the search strategy and identifying relevant literature. These actions were followed by text searching, which in turn entailed the construction of refined selection criteria, as well as screening, assessment and selection of the appropriate articles. The selected research articles were coded and analysed with the multimodal layer framework. Search strategies The articles were retrieved using educationally oriented databases: Scopus, Eric and Academic Search Elite. The first cycle of searches was executed at the end of May 2019 through the Scopus database, rendering 352 articles, and the second cycle was carried out at the beginning of June, covering 622 articles from the Eric and Academic Search Elite databases. The main search terms used were “digital tools” and “digital resources”, as these were regarded as the most frequently and interchangeably addressed in research on digital technologies in education. Because these terms are commonly and widely used and refer to both hardware and software, they enabled collection of a large quantity of potential articles. Moreover, because of the breadth of studies addressing various aspects of “use” of technologies in teaching and learning, a decision was taken to exclude this term in the search string. As a result, the final search string was: ((digital AND tools OR digital AND resource) AND teacher AND student AND NOT “higher education” AND NOT university). Selection of articles To conduct a valid selection of articles, and to reach a consensus amongst the researchers, an iterative process was undertaken to refine the selection criteria. Initially, general inclusion and exclusion criteria were defined as connected to the purpose of the systematic review on peer-reviewed empirical articles in English, published from 2010 to 2019. From the two searches, 974 articles were added to Mendeley, and, as duplicates were found, the articles were reduced to 796. From the general selection criteria and by making judgements on the quality and relevance of the found studies, the process started with assessing the first 100 articles. The three authors independently read the abstracts and followed a two-step process, starting with the first 43 articles: (1) individually labelled them with “include”, “exclude” or “maybe” in an Excel document; (2) after discussion, a joint decision on the selection criteria was taken. The remaining 57 articles were independently assessed following the same procedure. This resulted in a joint decision on inclusion and exclusion of the articles displayed in Table 1. The remaining 696 articles’ abstracts were divided equally between the authors for inclusion or exclusion assessment based on the final selection criteria using the program Understanding Criteria Include Exclude the use of Digital technologies Digital hardware and software Analogue technologies technology Teacher and/or Teachers’ and in-service teachers’ use in Articles on digital literacy, computer and student usage teaching activities in schools connected software engineering, competence to their school practice courses/periods development and general training, and Student usage in learning activities technology adoption/adaptation Teachers’ and students’ joint use Education All subjects Libraries, leisure time, other work-related practices Research conducted All school forms between kindergarten The purpose is to investigate technology in education (K-12) and school year 12 use in K-12 schools with exclusion of higher education. Other contexts, such as from purely technical aspects, teacher surveys, within healthcare issues or from a context that only reflects use in private life/leisure time Articles Scholarly peer-reviewed, full-text Systematic reviews, reviews, shorter empirical studies and book chapters, articles from forums, columns and Table 1. qualitative and quantitative magazines Final selection criteria Rayyan. General exclusion categories were adapted during a second round, and articles in languages other than English, duplicate articles and shorter articles from forums, columns and journals that no longer exist were removed. This process resulted in 391 articles. To enhance reliability in the coding process, one of the authors selected five articles as good and less good examples of the selection criteria. After discussing these examples, a delimitation was made on articles according to the relevance to “digital tools” and “digital resources”, which resulted in 105 articles. The three authors divided the remaining 105 articles for full- text reading with the aim to review their correspondence with all the inclusion criteria, which resulted in 64 articles being included in this paper. Data analysis A simple version of content analysis (Silverman, 2006) was adopted for data coding in the program NVivo 12. The first step of the coding process used the main categories from the multimodal layers and paired them for analysis (visible in Tables 2–8): technologies (divided into categories of hardware and software), technologies’ functional properties (primary, secondary, tertiary), technologies’ semiotic properties (modalities/semiotic resources), modes of representation (modalities/semiotic resources) and activities. Combinations of hardware and software are important examples of technologies used in meaning-making, why the initial division into different hardware was the first categorization, under which the other layers were organized. All layers where examined from the principles of foregrounding and backgrounding of information (van Leeuwen, 2005), i.e. information explicitly and more frequently described was deemed foregrounded and coded, while other information less salient, deemed more backgrounded and overlooked. Results and analysis This section presents findings from the multimodal layers and the numbers of examples. Students’ use Rich data on students’ use of PCs (see table 2) related to mostly visual and linguistic software, the traces of primary functions of, for example, displaying and recording, to the use of the IJILT Multimodal layers Data n Examples 37,5 Technologies Personal computers: 29 PCs: Desktop computers, laptops, unnamed computers –hardware Technologies – software Visual: 30 Visual: iMovie, Photoshop, TED talks, Google Hangouts, YouTube, Pinterest Linguistic: 23 Linguistic: Wordle, PowerPoint, Wikis, blogs Unnamed/other: 15 Unnamed/other: web pages Multiplicity of modes : Multiplicity of modes: VoiceThread, Glogster, StoryKit Auditory: 8 Auditory: GarageBand, Audacity, podcasts Technologies’ Primary function: 15 Primary: record, scan, display on screen, mouse/ functional properties voiceover, keyboard, insert, import, transfer Secondary function: 13 Secondary: paste, edit/cut, hyperlink, navigate, bookmarks, pinning, hotkeys, zoom function Tertiary function: 5 Tertiary: manipulate visual display (toolbar), effects (speeding up/slowing down), typographic style Technologies’ semiotic 25 Colour, background, font, typographic elements, properties graphics, space, layout, shape Modes of representation Visual: 30 Visual: images, moving images, animations, photographs, graphs Linguistic: 22 Linguistic: word clouds, writing Multiplicity of modes : Multiplicity of modes: combination of auditory- 18 linguistic-visual-gestures Auditory: 16 Auditory: dialogue, music, sound effects Gestures: 4 Gestures: gesticulating, indicating with hands Activities Collaboration: 20 Collaboration: sharing, joint production STEM: 19 STEM: mathematical content and rules, science Literacy: 18 Literacy: writing/reading literature Individual work: 11 Individual work: managing tasks and assignments separately Note(s): Refers to number of mentioned examples in the articles (i.e. not number of articles) Software where more than one modality is foregrounded in use Table 2. Students’ use of PCs Humans’ representations where people clearly place emphasis on the use of more than one modality secondary functions to organize such actions, by editing and cutting videos and sound, and navigating content by using hotkeys was observed. These actions are used with the tertiary function of adding special effects in video making and manipulating visual components using semiotic properties such as colours, background, etcetera to convey specific meaning in mainly visual and linguistic modalities (Wartofsky, 1973). The affordances of the technologies’ sign-systems – spatiality and simultaneity (visual), kineikonic with combination of different modes such as sound in recording and sequencing (linguistic) – can therefore give clues to individuals’ semiotic actions and production in the same modalities (Kress, 2010; Kress et al., 2014). Unlike PCs, smartphones (see table 3) evoke other kinds of use and actualize transitions between modes of representation and technologies. Incorporated in mostly collaborative activities, visual and unnamed software such as web pages were observed in the use of primary functions, such as navigating and scrolling on the touchscreen to access information. Often, auditory modes of representation were generated in use, indicating that smartphones are not yet fully incorporated in other sign-making activities, which need to be manifested in time and space (e.g. linguistic) (Kress, 2010). Because it is mostly feasible to trace actions but not meanings from primary functions (Wartofsky, 1973), smartphones are considered in volatile involvement in speech, prior to engaging in other activities, not least shown in the absence of tertiary functional properties. Understanding Multimodal layers Data n Examples the use of Technologies – hardware Smartphones: 5 iPhones technology Technologies – software Visual: 2 Visual: AR animations Multiplicity of Multiplicity of modes: snapchat modes: 1 Unnamed/other: 2 Unnamed/other: Wikipedia, digital text Technologies’ functional Primary function: 10 Primary: scan, navigate/scroll, storage, camera, properties display, keypad, touchscreen Secondary: 2 Secondary: organize notes, photos Technologies’ semiotic –– properties Modes of representation Auditory: 6 Auditory: dialogue Linguistic: 3 Linguistic: reading Gaze: 3 Gaze: directing attention Visual: 2 Visual: animations, videos Touch: 1 Touch: interacting with system using fingers Activities Collaboration: 3 Collaboration: pair and group work discussion Table 3. Literacy: 2 Literacy: reading Students’ use of STEM: 1 STEM: science simulations smartphones In students’ collaborative use of IWBs (see table 4), yet other traces could be observed, and transitions were visible between properties and modes of representation. Unnamed software was used with the primary capabilities of the interactive board to write and draw, organized through the secondary function of hiding and revealing content. The tertiary function was used to layer typographic semiotic properties to convey meaning, such as that of graphic structure and volatile auditory modalities including speech and gestures, into multimodal compositions (Kress, 2010; Wartofsky, 1973). The transition and traces found in the use of tablets (see table 5) are similar to those found for PCs. A combination of visual and linguistic software was used, mainly for their primary functions such as displaying, and recording, followed by the secondary function of organizing modes of action such as taking photos, inserting and zooming. However, when the Multimodal layers Data n Examples Technologies – hardware IWBs: 4 IWBs: SMARTboard, Camtasia Technologies – software Unnamed: 7 Documents, notebooks, websites Technologies’ functional Primary function: 2 Primary: writing/drawing function, recording properties Secondary Secondary: manipulating objects, hiding and revealing function: 2 content Tertiary function: 2 Tertiary: drawing on board, linking and annotating photos/text, displaying and communicating Technologies’ semiotic 3 Typographic elements properties Modes of representation Gestures: 4 Gestures: pointing, dragging objects Auditory: 4 Auditory: talking Visual: 3 Visual: drawings, diagrams, images Linguistic: 3 Linguistic: writing, annotating words Multiplicity of Multiplicity of modes: combination of auditory-visual- modes: 1 gestures Table 4. Activities STEM: 2 STEM: mathematical and science content Students’ use of Literacy: 2 Literacy: annotating/discussing/producing text interactive Collaboration: 2 Collaboration: discussions whiteboards (IWBs) IJILT Multimodal layers Data n Examples 37,5 Technologies – hardware Tablets: 12 Tablets: iPads Technologies – software Visual: 7 Visual: Fotobabble, iMovie, GlogsterEdu, video Linguistic: 6 Linguistic: storybookmaker, worksheets Multiplicity of Multiplicity of modes: Poll, Padlet modes: 2 Auditory: 1 Auditory: podcasts Technologies’ functional Primary function: 5 Primary: dragging pointer, visualizing, recording properties Secondary Secondary: taking photos/inserting into organizer, function: 4 editing/shaping videos, zoom Tertiary function: 2 Tertiary: recording videos on narratives, tracking alphabetic inscriptions on touchscreen Technologies’ semiotic 3 Font, colour, typographic elements properties Modes of representation Linguistic: 6 Linguistic: words, writing Visual: 6 Visual: moving images, photos, drawings Auditory: 5 Auditory: dialogues, discussions Multiplicity of Multiplicity of modes: combination of auditory-linguistic- modes: 4 visual-gaze-touch Gaze: 2 Gaze: directing attention Touch: 2 Touch: touching the screen Activities STEM: 5 STEM: science Collaboration: 4 Collaboration: shared production Literacy: 3 Literacy: creating e-books Table 5. Students’ use of tablets Individual: 3 Individual: managing tasks and assignments separately visual, linguistic and auditory modes of representation come about in combined activities in which the recording function is used as a tertiary function in writing activities, the semiotic properties are used to alter fonts in text or engage with typographic elements to combine with the recordings into multimodal ensembles (Kress, 2010; Wartofsky, 1973). Teachers’ use Teachers’ use of PCs (see table 6) is mainly in instructional design, using their primary function to display and record to create and disseminate content to students (Selander and Kress, 2017). Because it is mostly feasible to trace actions but not meaning from primary functions (Adami, 2014), connections between software technologies and the modes of representation in the same modalities might provide insights into meaning created. Teachers’ use of IWBs (see table 7) is similar to students’ use and demonstrated by unifying the actors in learning design activities (Selander and Kress, 2017). The chain of layers enables traces from visual software used with the primary displaying function to organizing the visual properties by the secondary function to, for example, hide and reveal and illuminate important parts. By the tertiary spotlight function, combinations were made with certain semiotic properties, such as colours, to signify visual meaning that was collaboratively negotiated in volatile auditory modes of representation such as speech (Kress, 2010; Wartofsky, 1973). Teachers’ use of tablets (see table 8) is limited in previous research. Undertaken in mainly instructional design, auditory modes of representation complemented by gaze and gestures were used in engagement with the primary and secondary functions to display content to students and organize further learning actions by using, for example, the zoom function. When using the tertiary functions of making recordings in the design of learning material, multimodal affordances of the technologies were combined with similar modes of representation into multimodal compositions (Kress, 2010; Wartofsky, 1973). Understanding Multimodal layers Data n Examples the use of Technologies – hardware 5 PCs: Desktop computers, laptops, unnamed computers technology Technologies – software Multiplicity of Multiplicity of modes: Poll, VoiceThread, blogs modes: 9 Visual: 9 Visual: iMovie, YouTube Linguistic: 7 Linguistic: Socrative, PowerPoint, WhatsApp Unnamed: 7 Unnamed: Internet Auditory: 1 Auditory: podcasts Technologies’ functional Primary function: 3 Primary: display, drag-and-drop, cursor, record properties Technologies’ semiotic properties Modes of representation Visual: 6 Visual: images, graphs Linguistic: 6 Linguistic: text/communication Multiplicity of Multiplicity of modes: combination of auditory-linguistic- modes: 4 visual-gestures Auditory: 3 Auditory: speech, oral tutoring Activities Instructional Instructional design: presentation/demonstration/ design: 3 development of teaching content Table 6. Learning design: 1 Learning design: teacher and student shared design Teachers’ use of PCs Multimodal layers Data n Examples Technologies – hardware IWBs: 6 IWBs: SMARTboard, Camtasia Technologies – software Visual: 5 Visual: digital photos, videos, YouTube, Flipchart Unnamed: 4 Unnamed: DOC files, notebooks, websites, polls Technologies’ functional Primary function: 4 Primary: displaying content, handwriting recognition, properties screen captures Secondary function: 4 Secondary: linking to documents, flipping back to recover material, storing/embedding in files, hiding and revealing, layering to manipulate content, drag- and-drop to focus on salient features of representations Tertiary function: 1 Tertiary: spotlight function to highlight important parts of contents Technologies’ semiotic 8 Highlighting, colour, shading, size properties Modes of representation Auditory: 5 Auditory: discuss, talk Visual: 3 Visual: images, graphs Multiplicity of modes: Multiplicity of modes: combination of auditory- 3 linguistic-visual-gestures Linguistic: 1 Linguistic: writing on screen Gestures: 1 Gestures: pointing (at screen) Table 7. Activities Learning design: 5 Learning design: teacher and student shared design Teachers’ use of Instructional design: Instructional design: presentation/demonstration/ interactive 1 development of teaching content whiteboards (IWBs) Discussion Research on digital technologies in education that illuminates connections between digital hardware and software use is scarce. However, the research compiled in this review indicates that PCs and visual software are frequently employed, in favour of student use. Research on other technologies such as IWBs, tablets and smartphones is limited. Teachers’ use of smartphones has been absent. Another finding is that different technologies are usually IJILT Multimodal layers Data n Examples 37,5 Technologies – hardware Tablets: 6 Tablets: iPads Technologies – software Visual: 2 Visual: Explain Everything, SPARK, Vue Multiplicity of Multiplicity of modes: Skitch modes: 1 Technologies’ functional Primary function: 1 Primary: displaying content properties Secondary Secondary: using fingers to zoom in on content function: 1 Tertiary function: 1 Tertiary: recording written work, making videos Technologies’ semiotic 3 Highlighting, colour, underlining properties Modes of representation Auditory: 3 Auditory: conversations, speech, talk Multiplicity of Multiplicity of modes: combination of auditory-linguistic- modes: 2 visual-gestures-gaze Gaze: 2 Gaze: looking at technologies Gestures: 2 Gestures: pointing, nodding Visual: 1 Visual: moving images Activities Instructional Instructional design: presentation/demonstration/ design: 4 development of teaching content Table 8. Teachers’ use of tablets Learning design: 2 Learning design: teacher and student shared design incorporated into the same activities, which is why differences or similarities between activities and the other layers are difficult to distinguish, important to investigate further. In line with Jewitt (2009) on the visual affordances of technologies, the findings indicate a strong focus on visual, closely accompanied by linguistic modes of representation. An overall tendency is oriented towards multimodality. Gaze and gestures, and sensory modality of touch is depicted in relation to students’ use of IWBs, tablets and smartphones combined with visual, linguistic and multimodal software (Jewitt et al., 2020). Below, the notions of transitions and traces in relation to the multimodal layers will be stressed in discussing the findings. Traces According to Kress et al. (2014), if influencing factors are defined, proposed here by the framework of multimodal layers, traces between the technologies and their properties and humans’ modes of representation and activities can be illuminated to understand the outcome of use. Examples of traces can be found when the links between the layers are intact. For example, when a hardware is combined with a specific software, the primary functions used in work, and secondary functions used to organize the work, provide possibilities for the tertiary functions to add meaning to the actions. Links can be made between the sign-systems used in the functional properties and human interpretation of them; actions that will be additionally enhanced by the connection to semiotic properties. The choices made amongst all sign-systems can be related to the making of abstract meaning (Cole, 2019) and are relative to the affordances of the properties and their meaning potentials and how they are ultimately drawn upon. Yet, the traces give clues to which of all the multiple features are regarded as visually salient, choices that are integrated into mental processing and renewed and communicated outwardly in sign-making activities (Kress, 2010; Kress et al., 2014). The circular process of internalization and externalization of interpreting and transforming sign- systems constitutes learning in connection with technology use (Bezemer and Kress, 2016; Vygotskij, 1978). In this study, these kinds of traces are observed. For instance, in students’ use of PCs and tablets, properties are demonstrated to come about differently in the combination with visual and linguistic software, which is unlike the use of other technologies. Understanding Traces are found between the primary recording function and the secondary editing function, the use of as well as the tertiary function of manipulating various aspects of visual display by colour technology and typographic elements as examples of semiotic properties, which are linked to visual or linguistic modes of representation. That entails that, first, technologies and sign-systems handled in technology use do impinge on what meaning is subsequently formed, and their affordances and constraints impact on modal choices made by the individual (Kress, 2010; Kress et al., 2014). Second, the choices made by the individual incorporating several traits from the original input will be a new form meaning made. Hence, although it is not possible here to conduct a detailed analysis, it is important to understand how functional properties are used for action and how they merge with the semiotic properties, as well as how the sign- systems therefore are selected and used to highlight specific aspects of meaning – in particular, if certain functional properties can be paired with specific semiotic properties, and how they relate to meaning created in a specific context. When one layer is left out, the links between the layers are broken and makes traces harder to observe. It is clear that if only primary functional properties are used, the semiotic properties are left out, and difficulties in relating technology use to sign-making activities arise (Adami, 2014). Hence, the online actions associated with the use of properties cannot be related to the offline activities of meaning-making (Wartofsky, 1973). In summary, traces made between all layers, can give comprehensive clues to how technologies relate to the construction of meaning through the sign-systems, a recognition important to acknowledge in learning settings. Transitions The notion of transitions refers to the process of negotiation and selection of sign-systems inherent in all technologies and used in sign-making activities. Transitions provide insight on how sign-systems move between the multimodal layers in use (Kaptelinin and Nardi, 2009). Technologies and their properties are therefore backgrounded or foregrounded in use (van Leeuwen, 2005), which entails various attentions paid to their affordances and meaning potentials, which in turn affect meaning-making (Ravelli and van Leeuwen, 2018). Transitions are implicitly recognized by previous researchers as a common phenomenon but have not yet been explored in-depth. This review shows that transitions exist in relation to both modes of representation and properties, and they move between technologies and users. Examples are students’ use of visual software, where the tertiary function to manipulate the visual display by semiotic properties such as colours and layout was used by teachers to organize content on a secondary level. In relation to hardware, the affordances of the screen to display content related to the primary function in the use of PCs and the tertiary function in the use of IWBs. More specifically, this means that the offline shape and size of the screen related to the online displaying function in hardware, or the online highlighting functions in software, to convey offline meaning in the use of colours and layout (Wartofsky, 1973) were all foregrounded sign- systems and conveyers of meaning in the use of certain technologies, and overlooked in the use of others. These findings contribute to an understanding of the complexity of sign- making in the use of technologies and why technologies can support diverse actions and sign- making activities in some cases and for some users but not others. Conclusion and implications Mapping technology use by focussing on the multimodal layers can provide comprehensive insights into traces – how different sign-systems are engraved in combinations of technologies, drawn upon variously by individuals, and therefore lead to variations in actions and sign-making activities. The multimodal layers approach can also contribute to IJILT clues regarding what role transitions play in representational signs between technologies, 37,5 levels of functions and properties, and how such shifts can be understood when traces are linked to individuals’ choices of resources to make meaning. The importance of combining technologies’ inherent properties to the patterns of human action and thinking will help us to understand how technology use relates to sign-making activities (Jewitt, 2017). The shifts in functional and semiotic properties between technologies need to be addressed when selecting appropriate technologies (Vigild Poulsen, 2018). The properties are multifaceted, and people make different choices regarding foregrounding or backgrounding information (van Leeuwen, 2005), therefore contributing to variations in meaning-making by the users. This also means that hardware and software technologies are sectioned in relation to the multimodal layers and to the user’s need to represent meaning. Evidence has also been provided that the primary functional level is more commonly used in teaching and learning. Wartofsky (1973) and Adami (2014) pointed out that the solitary use of primary functions could be problematic because it promotes actions rather than meaning- making. Hence, it is important to consider the role of primary functions in a learning situation, as they have slight relations to the other layers and poor connections to sign-making activities. Thus, teaching and learning undertaken through technology use from mainly a primary function is troublesome, why further explorations are needed on how transitions between technologies’ functional properties can give clues to other kinds of multiple use. References Adami, E. (2010), “A social semiotic analysis of mobile devices: interrelations of technology and social habitus”, in Pachler, N., Bachmair, B. and Cook, J. (Eds), Mobile Learning, Springer, Boston, MA, pp. 185-204. Adami, E. (2014), “What’s in a click? A social semiotic framework for the multimodal analysis of website interactivity”, Visual Communication, Vol. 14 No. 2, pp. 133-153. Alnervik, K. (2018), “Systematic documentation: structures and tools in practice of communicative documentation”, Contemporary Issues in Early Childhood, Vol. 19 No. 1, pp. 72-84. Bezemer, J. and Kress, G. (2016), Multimodality, Learning and Communication a Social Semiotic Frame, Routledge, London. Cole, M. (1996), Cultural Psychology. A Once and Future Discipline, Harvard university press, Cambridge, Mass. Cole, M. (2019), “Re-covering the idea of a tertiary artifact”, in Edwards, A., Fleer, M. and Bøttcher, L. (Eds), Cultural-Historical Approaches to Studying Learning and Development, Perspectives in Cultural-Historical Research, Springer, Singapore, Vol. 6, pp. 303-323. Djonov, E. and van Leeuwen, T. (2013), “Multimodality and software”, in Chapelle, C.A. (Ed.), The Encyclopedia of Applied Linguistics, Wiley-Blackwell, Wiley, Chichester, pp. 1-5. Gough, D., Oliver, S. and Thomas, J. (2012), An Introduction to Systematic Reviews, SAGE, Los Angeles, Ca. Innis, R.E. (2013), “Semiotics of technology”, in Olsen, J.K.B., Pedersen, S.A. and Hendricks, V.F. (Eds), A Companion to the Philosophy of Technology, Blackwell, West Sussex, pp. 206-212. Jewitt, C. (2002), “The move from page to screen: the multimodal reshaping of school English”, Journal of Visual Communication, Vol. 1 No. 2, pp. 171-196. Jewitt, C. (2009), Technology, Liceracy and Learning. A Multimodal Approach, Routledge, Oxon. Jewitt, C. (Ed.) (2017), Routledge Handbook of Multimodal Analysis, Routledge, London. Jewitt, C., Price, S., Mackley, K., Giannoutsou, N. and Atkinson, D. (2020), Interdisciplinary Insights for Digital Touch Communication, Springer, Cham. Kaptelinin, V. and Nardi, B.A. (2009), Acting with Technology. Activity Theory and Interaction Design, Understanding The MIT Press, Cambridge, Mass. the use of Kress, G. (2010), Multimodality. A Social Semiotic Approach to Contemporary Communication, technology Routledge, London. Kress, G. and van Leeuwen, T. (2001), Multimodal Discourse: The Modes and Media of Contemporary Communication, Arnold, London. Kress, G., Jewitt, C., Ogborn, J. and Tsatsarelis, C. (2014), Multimodal Teaching and Learning, The Rhetorics of the Science Classroom Bloomsbury, London. Norris, S. and Jones, R.H. (2005), Discourse in Action. Introducing Mediated Discourse Analysis, Routledge, Oxon. Ravelli, L.J. and van Leeuwen, T. (2018), “Modality in the digital age”, Visual Communication, Vol. 17 No. 3, pp. 277-297. Scheele, M. (2006), “Function and use of technical artefacts: social conditions of function ascription”, Studies in History and Philosophy of Science, Vol. 37 No. 1, pp. 23-36. Selander, S. and Kress, G. (2017), Design F€or L€ arande. Ett Multimodalt Perspektiv, Studentlitteratur, Lund. Silverman, D. (2006), Interpreting Qualitative Data: Methods for Analyzing Talk, Text and Interaction, Sage, London. van Leeuwen, T. (2005), Introducing Social Semiotics, Routledge, London. Vigild Poulsen, S. (2018), “Becoming a semiotic technology – a historical study of Instagram’s tools for making and sharing photos and videos”, Internet Histories, Vol. 2 Nos 1-2, pp. 121-139. Vigild Poulsen, S. and Kv ale, G. (2018), “Studying social media as semiotic technology: a social semiotic multimodal framework”, Social Semiotics, Vol. 28 No. 5, pp. 700-717. Vygotskij, L. (1978), Mind in Society: The Development of Higher Psycological Processes, in Cole, M., John-Steiner, V., Scribner, S. and Souberman, E. (Eds), Harvard University Press, Cambridge, Mass. Wartofsky, M.W. (1973), Perception, Representation, and the Forms of Action: Toward an Historical Epistemology, Springer, Dordrecht. Zhao, S. and van Leeuwen, T. (2014), “Understanding semiotic technology in university classrooms: a social semiotic approach to PowerPoint-assisted cultural studies lectures”, Classroom Discourse, Vol. 5 No. 1, pp. 71-90. Corresponding author Karoline Schnaider can be contacted at: karoline.schnaider@umu.se For instructions on how to order reprints of this article, please visit our website: www.emeraldgrouppublishing.com/licensing/reprints.htm Or contact us for further details: permissions@emeraldinsight.com

Journal

The International Journal of Information and Learning TechnologyEmerald Publishing

Published: Nov 9, 2020

Keywords: Technology; Software; Education; Multimodal; Hardware; Layer

References