1. Introduction

Mobility, mixed reality & learning

Multimodality, multiliteracies, movement and mixed media are some of the themes we take up in this article in a discussion of the changing character of digital media education. We address matters of production-based media arts in higher education that involve movement or kinetic competencies and spatial, or proprioceptive, awareness. We argue

for a view of multiliteracies that extends beyond the interfaces and screen-based media of the Web, online environments and games. New media art is one of the areas that is often lacking in publications of multimodal multiliteracies in general, and in media studies and new media education in particular. New media art crosses art, design, informatics, performance and media. Typically it is covered as part of electronic art (e.g. Paul 2003), or more recently in informatics perspectives on electronic art (e.g. Fishwick 2006). These publications most often refer to research, technology and aesthetic elements, and less commonly to learning or notions of multiliteracies and competencies that are already widely circulated (Cope & Kalantzis 2000, Lankshear & Knobel 2006).

We argue that media education, especially in order to prepare higher education graduates for work in the culture industries (Lash & Lurry 2007), may benefit from collaborative learning and reflection that includes production-based inquiry into digital tools, materials, mediation and expression. This inquiry is very much to do with exploratory and innovative expression and popular cultural contexts of creation and use that extend from mobile phone messaging to artistic constructions. We support this claim through reference to a collaborative student and research project that involves the design, development and analysis of a mixed reality installation and performance piece called Tapet, meaning Wallpaper in Norwegian.

In this article, we adopt an essayistic form that refers to this one production-based educational and research project and its performative outcome. This is a work that results from collaboration in making new media art, and one that moves between choreography, media and informatics. The work accentuates the expressive and the explorative; it also encompasses a synthesis of analysis and production. We argue that developing and exhibiting such a work may help further both our digital literacies and our understanding of multimodal literacies (Kress & van Leeuwen 2001, Jewitt & Kress 2003) and their role in learning about wider cultural production (Buckingham 2003).

Many of the publications on multimodality still remain within a strongly systemic functional linguistic frame, rich as this may be (Kress & van Leeuwen 2001; O’Halloran 2004; Machin 2007). Publications are preoccupied with developing systematic patterns of analysis of discourse in which a variety of modes may co-occur (Royce 2007). Much of the research into multimodality and learning that draws on this tradition (Jewitt & Kress 2003, Jewitt 2006), however, remains outside the practices involved in the production of multimodal discourse in specifically digital domains.1

Come on in…

Elsewhere we have written about the design of Tapet in terms of the intersections between choreography, media and informatics in the frame of mixed reality arts that blend the digital and the materialities of the physical here and now (Morrison et al. 2008). The opening image (above) has been chosen to signify a number of key issues related to a

‘stretched’ notion of multiliteracies via production-based learning in the field of mixed reality arts. The image captures only one context evocative of creative engagement in learning and researching new media arts. The image is taken from the setting up of the installation work. The main, large human figure has projected onto it a small dancer in an orange dress. Immediately, the notion of a mixed reality space is demonstrated. This is where digital technologies with content developed by student-researcher-designers are being explored. The computer equipment on the plastic green grass covering the floor suggests the context of setting up an installation. Here the pre-recorded sequences of the dancer, to be projected 4 by 3 metres onto the gallery wall, are instead playfully shrunken to miniature dimensions on the leg of one of the project participants.

Lured in by the occasional forest sounds reaching out into the public university library setting where the installation Tapet was exhibited in 2007, audience members enter a space, suggested in this opening image, of play, representational variety and a toggling between a digitally mediated, pre-recorded figure and human participant in their immediate material space. This is shown in the various photographs below from this particular event and exhibition.

In Tapet, participants are prompted to place themselves at several spots marked in orange on the fake grass that covers the gallery floor. The dancer in the forest in front of them, depending on the combinations and linkages generated computationally, may then line up with them, dance, suggest they move, or move away to a dance sequence of her own. In this way, the work mixes mediations in real-time. A camera also records the participants’ actions and projects them into the screened forest, so that a mediated mirroring is conveyed. However, a short time delay is scripted into this mirroring so that the embodied, watching and moving participant not only observes themself in a series of captured/recorded dance sequences, but also moves to anticipate their own mirroring. In this way, the work blurs the connections between the ‘real’, immediate world and that which is generated digitally, but, also, comes into being by way of participants’ own actions.

Mixed reality mediation & a mix of competenices

As Buckingham (2007) argues, media educators can respond to the challenges and potentials of digital media by applying existing concepts and practices to new objects of study, but also by extending these. Yet teachers, he argues, may also engage with the creative potential of ‘new’ media and the implications they may offer pedagogically. Importantly, he further argues that working with digital media in contexts of learning offers us opportunities to develop and question modes of participation and their relations to wider media culture. In this chapter, we address each of these three suggestions as to how we might take part as teachers and students, designers, artists and researchers in our emerging and – in a Vygotskian frame – developmental approaches to mediated meaning-making (Wertsch 1991).

Engaging in production-based inquiry that crosses and merges elements of media and technology (e.g. Jewitt 2006) in an art-making context highlights the need for a range of competencies. These are ones that are widely implicated in electronically designed and mediated environments in daily work, learning and leisure. Digital and physical scenography, mediated dance and the potential for participation via sensors, projections and live and stored media demand that we engage with issues in moving media. Such engagement may also point to the role of movement inside the prepared work and through a kinetics that is afforded to audiences themselves. We argue that this is to extend the notion of multiliteracies from articulations on computer screens to settings where students and audiences engage with mixed reality design, environments and participation. To do so is to also take the notions of multimodality (Kress & van Leeuwen 2001) into digital domains (Morrison in press). These are arenas where extended multiliteracies includes dynamic, kinetic, spatial and performative modes of mediation and expression.

2. On the mixed reality work Tapet

Photograph of a media researcher (Synne Skjulstad) takinga picture on her phone of the dancer projected into the forest.This is the environment into which the participants see themselvesprojected. Note the mixed reality pun of the power cable at floorlevel.

From collaborative composition…

The installation-performance work Tapet was developed as part of a longer process of pedagogical and expressive experimentation with students in media, choreography and informatics (Skjulstad, Morrison & Aaberge 2002; Morrison 2003a; Skjulstad & Morrison 2005). Tapet was framed within a socio-cultural framework to technology-enhanced learning. This framework accentuates relationships between mediation, tools, technologies and contexts of production and reception that are geared towards participants’ meaning-making in learning events and environments. The socio-cultural approach we adopted was also influenced by multimodal discourse studies and new media analyses. Briefly, Tapet invites participants into a space of potential modes of engaging with a dancer who is projected into a screenspace and with whom they may also appear as mediated performers. In Tapet, mediational affordances for mediated meaning-making were designed and embedded in the work. The aim was to offer audiences a means for finding these themselves and becoming implicated in their interplay through movement.

Tapet was first developed as part of a research project entitled MULTIMO that included graduate students and researchers in collaborative and production-based inquiry into multimodal digital discourse. Tapet was devised on the basis of earlier collaboration in a graduate course for students of New Media Education (NEWME) with final year undergraduates in choreography. This collaboration resulted in four dance performances under the title Extended 2 that reflected the stretching of both mediated sceno-graphy and technology enhanced performances in four works (Sem 2006). In one of these works, Proximal, live performance and digital projection were alternately present in a reflexive play between the projections and the choreography, as light and as canvas (Sem 2006: 55, online).3 The visually driven, exploratory design behind Proximal created a non-representational, multimodal, unstable and shifting work that explored and challenged notions of ‘presence’. This joint production established a strong co-operative practice between the choreographer, Inger-Reidunn Olsen, and the media student, Idunn Sem. They developed this further in the research and production-driven, project-based inquiry that was central to the design and development of Tapet. In this latter work, conceptual inquiries introduced in the work Proximal were continued and expanded with Martin Havnør, a Master’s student in informatics.

… to public participation

Tapet was first devised and built in a large space originally designed as a TV studio, now used as a mixed media space for experiments involving digital technologies. As part of the marking of the 10th anniversary of the ITU at its annual conference in 2007 we were invited to include Tapet in the programme of events in an exhibition called Interfaces. For us this required repositioning an installation work that had been developed in an experimental lab setting to the very public space of Gallery Svedrup at the University of Oslo’s

main library. This space is accessible to students, staff and the general public. This setting adjacent to the main conference one also offered educators, students and researchers a shared space for engaging with our propositions through their own embodied interaction.

Encompassing an array of literacies

As the selected images we have included illustrate, Tapet mixes landscape wallpaper, artificial grass and occasional forest sounds with mediated dance performances and scripted logic. The ability to ‘read and write’ such multimodal texts is confronted with digitally driven non-linearity and co-composition, location and movement as input and content, and augmented space and time. Deciphering and also producing these diverse modes of representation demands an array of literacies. These are entailed in the composition of the work, especially its collaboration. They are also present in how people engage with the work, and what it asks of them in terms of making sense of its dynamics and learning how to move with the (absent) choreographer-dancer.

We argue that Tapet is an artefact that inhabits or embodies both practice and theory. It is itself a form of reflection and a site for reflection. It functions as an heuristic for investigating conceptual border crossings by way of collaborative experimental design and performativity. This was achieved through student interdisciplinary collaboration in design, implementation and trialling in the guided lab setting at two interconnected levels. First, the work was conceived of as a site that would involve participants in multimodal engagements with several layers or states of their own embodied performance. Second, Tapet aimed to include participants in exploring multiliteracies that involve their own mediated movement and meta-understanding of the activities involved in shifting between various modes of engagement between media and mediation. Tapet was geared towards designing and incorporating these modes as affordances for participation through which audiences could explore notions and concepts of mixed reality mediatised performance, and experience their own performance via an unfolding dynamic of mixed media.

3. Multiliteracies extended

New media arts & literacies

Few of the publications on multiliteracies refer to such an unfolding of collaborative design and development that is common to the composition practices and experiential participation central to many new media arts (e.g. Fuller 2005, Grau 2007). Publications on multiliteracies are usually situated in school level activity; more recently, these writings also address informal contexts of learning (Sefton Green 2006). Emphasis is given to digital literacies that include students’ own learning through production. Students’ devel

opment of critical analytical acumen about these productions is also central (Buckingham 2006). Importantly, however, such productive constructions of mediated texts and expressions are seen to include competencies located in students’ uses of digital tools and modes of expression that software may engender (Sefton-Green 2005) and that are located in popular cultural uses. Where studies do address students’ digital multiliteracies, these are typically not concerned with higher education contexts that include media studies. Further, where the realisation of such digital multiliteracies involves students’ own learning via production in and out of school (e.g. Sefton-Greene 2005, Burn & Parker 2001), focus on media in new media arts is absent, as opposed to, for example, digital video production. Rarely do we see substantial reference to writings from ‘new media studies’ itself; even less often do we find reference to domains of electronic art and technology enhanced performance.

In higher educational contexts, media studies has traditionally been concerned with analysis and not production. The partnering of these two is pedagogically challenging, but also potentially fruitful. At a wider level, collaborative production-based inquiry linked with critical analysis may involve educators and students in iterative and reflexive activities of making and reflecting. These activities often entail ongoing shifts, ones that are at times uncomfortable and require patience, trust and curiosity, all oriented towards supporting students’ understanding of digital design and critique. Here a medley of domains, content knowledge and competencies may be explored and developed through collaboration. Such interconnections have the potential to lead to a richer mix of multimodal media and mediation.

Experimental laboratory contexts and related research clusters in higher education settings may provide a context for situating exploratory and innovative courses for media studies, as has been the case in, for example, research and educational collaboration on dance and technology (e.g. Schiphorst 2005). Such activities and linkages to a wider community of practice are an essential ingredient for effectively and innovatively engaging with the expanding areas of exhibition and experience design within the culture industries. Media studies education, we argue, has much to offer these domains in joining its students with art and informatics students in production-based learning in new media arts.

Embodied interaction

In universities media education largely continues to cover an approach to media studies that emphasises critique. Although this approach now includes the study of computer games and other participatory media, it does not yet very substantially address the rapidly growing field of embodied interaction (Dourish 2001) that moves us out of our chairs, off our laptops and away from our remote controls into spaces of mixed media. Nor is the role of media in these environments highlighted much. Yet, mixed reality arts

that are rich in media are flourishing in exhibition and museum design and in theme parks. These are some of the domains where experiences of shared meaning making between the expressive and the expository may occur. Such public contexts for the enactment of multiliteracies may include media that are actively and pervasively a part of wider, popular culture.

A recent notable commercial product that highlights the kinetic and proprioceptive in its embodied use is the Wii from Nintendo. This wand-like multi-device takes movement and shared engagement from amusement arcades to the ‘living room’. As a result, our embodied interactions are also enacted in domestic spaces.4 Perhaps most pervasively today, the iPhone from Apple, with its touch-based interface that includes dynamic interface elements (Skjulstad & Morrison 2005), takes our kinetic senses and spatial competencies into new commercial and communicative domains. This mobile phone is a multimodal artefact. It symbolises and enacts a kinetics that may become more prominent in other domains of popular culture and mediated expression involving tangible interfaces. Haptic, kinetic and mobile media too may be taken up as part of an extended multiliteracy in technology enhanced learning.

Multiple influences

There is a need for media education – and that in higher education settings – to further explore digital literacies in relation to the experience and knowledge students bring from their own popular cultural contexts of use and experimentation (Buckingham 1998). A body of work now exists in school-level media education that examines students’ expressive productions, for example in animation (Burn & Parker 2001), in relation to the concept of re-mixing of multimodal resources (Erstad et al. 2007) and the re-editing of video (Burn & Durran 2006). Higher education production-based courses in new media, that include the arts, will also need to link this knowledge to critical and analytical approaches so as to build practices that help students move from the phenomenon of the newly mediated in emerging expressions and devices to their close critique of them.

Linking production with analysis in production-based learning is one means of stretching traditional academic literacies into an electronic and dynamic frame (Sullivan & Porter 1997, Hawisher & Selfe 1999, Wysocki et al. 2004). Following the work of Ulmer (e.g. 1998), this has been referred to as ‘electracies’ (Morrison 2003b). Media Studies is a potentially rich site for building both competencies and their critiques within and across media types and modes of mediated discourse. In a project related to the one we present here on Tapet, undergraduate media students in South Africa, some with limited access to digital production tools and commercial media consumption, used a specially designed digital video editing environment to remix a video sequence. To this they were able to add their own voice-overs and learn about a specific genre of film at the same time (Morrison et al. 2005, Deacon et al. in press 2008). This specially designed pedagogy included a set

of supporting lectures and tutorials on critical terminology and analysis. This type of experimental space inside media studies and digitally mediated pedagogies is often achieved through collaboration with technology-enhanced learning specialists and other partners. Examples of such work can also be found inside academic composition and communication programmes and their online rhetorical accounts of such innovations (e.g. Reid 2003, Gillette 2005). In terms of the further development of multiliteracies in higher education, stronger connections might be made between programmes in digital rhetoric and composition and media studies in which production with critique features.

Competencies on the move

When collaborating on production-based learning in electronic art it is necessary to take up matters of materials and the design of screen based media. In installation arts settings the intersections between interfaces and our performing bodies are involved as part of a shift from being spectators to that of active participation (Courchesne 2002). The design and realisation of these relations suggest that we might extend the notion of digital literacies to movement that is beyond the mouse or joystick and is realised through embodied interaction where performers bodies, including those of our own projected selves, move together. This may be done playfully and artistically, and sometimes as ‘bodies without bodies’ (Melrose 2006).

Working experimentally with multiliteracies in higher educational contexts also heightens the relations between such features and functions. There is a need for co-design on the part of students who are not simply using off-the-shelf tools for blogging or structuring databases in parallel to known patterns, but are themselves designing and shaping environments for mediated enagagement. These are environments that are primarily communicative. In our view, they need to be situated within a Communication Design perspective (Morrison in press 2008) in which the sociocultural semiotics of mediated meaning-making are central. The demands here are of various intersecting competencies, many of which can only be developed through processes of shared production and meaning-making that is digitally mediated. This also applies to mixed reality settings, ones where choreography and movement may meet interactive art practice (Schiller 2005).

In the sections below we unpack this in a reflection on developing Tapet. We discuss some of the challenges to our existing competencies and practices. We suggest ways in which these may be considered as part of a notion of multiliteracies that extended from print and computer screen arenas to those of mixed reality settings. These we suggest are settings that are likely to become more closely connected with personal and popular cultural media use. New media and mixed reality arts may be only a part of wider screen media and mixed reality mediation. However, they contain several elements that apply more broadly to mobile media, to moving with media, and, importantly, to how we engage bodily with technologies (e.g. Zuninga Shaw & Lewis 2006).

4. Aspects of production-based inquiry

Two participants, one moving, the other watching andwaiting, engage with the dancer before them.

Contexts for design

In Tapet, we adopted a mixed media space rather than the live stage mode of the previous collaboration Proximal. Compositionally, we did this to be able to toy with the interplay between the interactive and generative aspects of multimediational, mixed reality expression. We were further interested in moving away from a presentational performance mode to one that would enable, and hopefully motivate, performativity on the part of an audience. We have used the term multimodal performativity (Morrison et al. in press) to encompass the intersections of dance, digital media and audience enactment.

Four states

In terms of the border crossing of modes of enactment and engagement identified in our explorative inquiry, Tapet can be unfolded as states, though not necessarily sequential ones. We labelled these as: Embodied, Mediated, Interactive and Generative (Morrison et al. in press). As these terms have been applied variously elsewhere, we now only explain their provenance concerning this work and some of the implications for reconsidering multiliteracies in digital environments that extend beyond the boundaries of the desktop computer screen.

In the physical site of Tapet, four spots are marked on the grass. A small camera in the ceiling reads activity at these positions. The dancer, projected onto the screen, ‘identifies’

the participant’s location and a video from the database is cued to suit that position. This cuing is varied for the particular spot, thereby adding a sense of novelty or difference. At times the dancer moves energetically and with emphasis, sometimes almost still, at times facing the participants, sometimes turned away.

The Embodied state refers to the participants’ engagement in a mixed reality space where a variety of media (print, sound, projection) provide the potential for the participant’s movement to situate a sense of their own body as a part of realising the work. The reflexivity involved in the movements between the ‘real’ and the ‘simulated’ are also apparent in the state labelled Mediated. Here it is important that the participant responds to a performer who is clearly not physically present, but whose mediated movements are dynamically enacted in the unfolding present. This is an unfolding that reveals alternatives. How the participant engages in this unfolding is what is covered by the state labelled Interactive. Movement does not just happen; the participant depends on prompts and guidance. The Generative state is used to cover the variety and multiple combinations of both the videos delivered by the system and the potential for the participant’s own movements which with the time delay are overlaid onto the screenspace, allowing for a further reflexive loop of their own performance.

In each of these intersecting, and not necessarily sequential states, the participants are active partners in mediated performance. They may also move with and adjust their own sense of learning how to move in this mode of mediated interaction. To call the work an installation, suggesting it is put into place and left alone representationally, belies an ongoing ‘installation’ of the intersection of the activities and movement between system, mediatised performer and live and mediatised participant.

5. Tapet as heuristic

Two contexts in one setting

The adaptive digital design of Tapet is emphasised not only in its generative uses, but also in it being recustomisable for different settings and events. At the ITU anniversary event the installation was located in two contexts – a public educational conference space, a gallery setting – to bring forth both intended and less intended engagement and enactment. In the lab, conference and gallery, Tapet may be viewed as a means for investigating homogeneity, linearity, complexity and longevity from within an informatics view on system interaction (Havnør 2006).

From a humanities perspective, Tapet may be viewed as a device through which to explore cultural, digital and expressive shifts and resultant theoretical challenges and border crossings by embodying them in a mixed reality work (Sem 2006). The production-based, collaborative and process-oriented experiments presented here, from Proximal to the redesign of Tapet, have been scaffolding-like inquiries into expressive shifts. These

were shifts from ‘… temporal sequential logic of spoken and written language to a spatial-simultaneous logic of the visual (Kress 1999: 68), ‘from narrative to display’ (Kress 1999: 82), and ‘… from the object of representation to the emergent situation, the performative current, and the materialisation of technology itself’ (Birringer 2003: online).

Most explicitly, the reflexive, playful embodiment of theory in these iterations was informed by the dialectic of looking through and looking at mediating artefacts, expressions and texts. Here we refer to the notions of immediacy and hypermediacy proposed by Bolter and Grusin (1999). These terms refer to a shift beyond representation, and its boundaries in mediating the ‘real’ or that which is apparent (immediacy), to one of looking at the medium itself as a meta-reflexive entity. This is a critical component, we argue, of extending notions of multiliteracies in higher education.

Inside the forest. Looking at the projection screen.The dancer (left, Inger-Reidunn Olsen) is generated from withinthe computer system; the image of the media student designer (right, facingcamera, Idunn Sem) is projected into the forest as she stands facingthe screen where the dancer dances. In the foreground, one of thedesigner-researchers from MULTIMO records the media student designerusing her mobile phone.

Accentuating the mediated

We aimed to embody such immediacy and hypermediacy in our collaborative, production-based inquiries in aspects of both the design and implementation of the work. In other words, as we blended digital media with ‘modes’ of live performance in the multimedia performance Proximal, and with the physical environment and system logic in

Tapet, the post-structuralist and cross-disciplinary view of looking through and looking at mediating artefacts, expressions and texts surfaced. The double logic of immediacy and hypermediacy became creative concepts for us to adapt, invert, alter and refine in our expression via and as a mixed reality work. This was made possible by way of practices of multimodal discourse in action (Norris & Jones 2005) in a mixed reality arts arena and by post-production reflection upon practice (Sem 2006: 109 online). In a recursive series of theory embodiment and improvisation – that is of creating, experiencing and reflecting – our knowledge about these conceptual artefacts was challenged and broadened.5

On a reflexive level, the making of Tapet proved to be a means to developing a better understanding of the dynamic between conceptual knowledge and practice. Practice may be understood as an accompanying line of inquiry to the interpretive and critical analysis of traditional learning and research designs of the humanities (Sem 2006 online). Engaging with the making and reflection of new media arts can help reveal that humanities-based approaches to media studies themselves may be extended from studies of texts as products to ones where understanding of critical terms and analysis may be acquired via a developmental process that includes text construction, generation and critique.

6. Production-based digital media education & literacies

Three participants, each in movement, demonstrate thatthe work allows for multiple participants and multiple experienceswithin one interactive environment. To the right, two other membersof the audience watch this activity and photograph it. There aremany multiliteracies around kinetic, visual and spatial media atplay here.

Practice with theory

There is a growing awareness in both the development of digital media studies and disciplines concerned with digital literacies of how ‘reading’ digital media in some senses is rather like ‘writing’ or ‘constructing’. In a sense, the object of analysis is not a fixed and stable entity (e.g. non-linear hyper-textual newspapers or narratives, multiple distributed and displayed videos in a non-frontal performance-space, or varieties of generative art). Even the appropriateness of the term ‘media’ is questioned in some digital communication contexts. Penny (2004), for example, suggests artefact as a more suitable term for real time computation. The boundaries between critical analysis and practical production, and between practice and theory, are increasingly blurred (Buckingham et al. 1995: 10).

Confronted with rapidly evolving and diverse digital media forms with potential for synthesising modes and of co-construction (Liestøl & Morrison 2001, Fagerjord 2003), digital literacies and digital media studies and education may be said to have interlinked grounds for a more production-based approach and common concerns around production-based methods. The need to explore alternative, multimodal and hypertextual ways of communicating and evaluating media practice and reflection upon practice is an issue shared by both digital literacies and digital media education. This is not only a matter of conveying, in the case of Tapet, multimodal spatio-temporal expressions through writing. It is also a matter of doing so through embodied participation, photographic documentation and multimodal presentations that may all be connected to project-based inquiry as part of extended and creative, multiliteracies.

In summary, ‘new’ media arts challenge us to enter into the co-composition of works, potentially at several levels. These may include their design and shaping – artistically and technically – and their realisation through performative participation. These are often works, or environments, that are unstable and without fixed representational forms. As with earlier hyperfictional works, they are spaces for the realisation of multiple textual patternings. It is this multiplicity inherent to multimodal digital texts that students may learn to understand, create and analyse. Each of these aspects may be interlinked in a holistic approach to teaching and learning about digital multimodal creative expression. For media students, knowledge of the diversity and malleability of digital media through programming, and qualities of tangible computing more generally may be drawn together in mixed reality works. The co-construction of such works offers students a potentially rich experience of work together, productively and analytically.

Shared grounds for production-based digital media education

In order to better understand the interplay between media technology and media texts, it is important to recognize that ‘passive’, mediated knowledge of media creation and distribution may be expanded by knowledge that can only be fully developed through the expe

rience of production itself (Buckingham et al. 1995). Through recursive translation between different modes of engagement with media – creation, experience and reflection – the interplay between media technology and media texts materialises. By a recursive series of creation, experience and reflection on experience, the distinctiveness of the context in which the media-text was created, and the sameness – the affordances and constraints – of the applied media, may project the particular media-text (i.e. the object of the traditional critical analysis) (Sem 2006: 112, online). Such recursive translation between different modes of engagement with media is present in vocational media training and may be very apparent in interrogative and exploratory contexts of media improvisations. These too may be understood in the developmental view offered by a Vygotskian framework.

The creation and distribution of digital media may differ significantly from how former media have been created and distributed – and continue to evolve and diverge at a rapid pace. Hence, first-hand, recursive and production-based, rather than second-hand, mediated and passive knowledge of, creation and distribution might be particularly crucial for both the acquisition of digital literacies and in the study of digital media. This is where secondary mediated knowledge such as theoretical adaptation from neighbouring disciplines or the potential of a general semiotic approach may reach a boundary. Production-based methods inform studies of digital media with emerging and also distinct vocabularies and conceptual knowledge of novel digital media forms. Particularly important in digital media studies, such knowledge might level the widely acknowledged distance and risk of misrepresentation between objects and concepts (Liestøl 2003). Delay in the formalisation of concepts and analytical vocabularies may stymie our framing of digital literacies in media education. Although digital, multimodal communication might be said to be part of a broader break with tradition, it may compel us to question the precedence of language-based theories of communication and meaning. This, in turn, calls for conditional responses to challenges of visualisation such as production-based learning and research methods that may inform both studies of digital media and digital literacy by way of artifacts that inhabit or embody critical inquiry. Such a means of exploring and questioning new media arts both creatively and critically may help students – in media and informatics, as well as young choreographers, digital artists and designers – to meet the challenges of working in and analysing the digital in culture industries more broadly, both in their university level studies and for future work.

Interplays of conceptual knowledge and practice

Dynamic interaction between media practice and media theory in media education may be conceptualised as a dialectic process of translation between language modes, between ‘writing’ and ‘reading’, media-making and media-analysis, in which conceptual knowledge is acquired through a process of theoretical reflection upon practice (Buckingham et

al. 1995). Emphasising methods of the humanities and sciences, interactions between practice and theory may be perceived as translations between ‘synthesis’ and ‘analysis’. This allows us to acknowledge the implicit concepts (and theories) of practical construction, as well as the latent constructive potential of theory (Liestøl 2001, 2003).

A closely related way of conceptualising the dynamic of practice and theory is to consider it a ‘two-way shuttle of insights between theorising and experimenting’ (Liestøl et al. 2003). In our view, this emphasises the innovative and explorative potential of production-based method. Part of the dynamic between theory and practice in courses with collaboration such as the one we report on here, is that new competencies emerge and need to be conceptualised as they have the potential to raise new issues and concepts, or that known ones need to be identified or labelled. Situated within perspectives of embodied cultural theory, the notions of translation may be stretched towards embodiment. In Bolter’s view, theory or critique may speak through the artefacts themselves (2003: 28).

The settings of the production-based inquiry presented above may not be the optimal ones for production-based digital literacy. We believe that production-based digital literacy may benefit from collaborative, explicit exploratory and innovative contexts in which the practicing student or researcher are encouraged and enabled to enact recursive and process-oriented translations between media-making and analysing. Given our limited resources, we have not yet been able to follow through on the views and mediated meaning-making of participants taking part in this work. What would be potentially interesting would be to see how media students might develop their own formulations of the interplay of media, mediation and participation, and perhaps take these up and on into new works of their own shaping. In this way, reflection on and in action from design research (Schön 1987) could be highlighted in new media education.

Concerning the ‘turn’ to contextual approaches to interpretation, Stanley Fish asked ‘Is there a text in this class?’ (Fish 1980), arguing that meaning inheres in readers’ active engagement with texts. In the context of engaging with dynamic media in contexts of embodied interaction, (and not simply phenomenologically as if being in this mixed reality was itself enough), we might ask, more plurally, ‘What multimodal expressions and multiliteracies are there in this environment?’ The challenge to us as educators, students and researchers is to also address the means by which we work interpretively in the many layers of designing, implementing and participating in these environments.

Both production-based digital media literacies, education and studies may benefit from collaborative and explicitly exploratory and innovative contexts in which the practicing student or researcher is encouraged and enabled to enact recursively in process-oriented translations between media-making and analysing. We fully agree with Buckingham (2007: 112) who argues that ‘Education about the media should be seen as an indispensable prerequisite for education with and through the media…. We need to equip students to understand and to critique these media.’

7. Performative moves

Mixed reality on the move. Photograph of designer-researcherSynne Skjulstad dancing with the figure in the forest. The shortexposure leaves traces of her striped shirt that appear blendedinto the background. The projected forest scene is actually in frontof her.

Stretching multiliteracies

As mentioned earlier, ‘new’ media arts are now a major contributor to the global creative and culture industries. They inform and influence gaming, exhibition and interaction design, and the emerging domain of experience design where affective elements are presented as affordances for interacting with installation spaces or environments. Mixed reality arts allow students and audiences alike to import and explore their own understandings of digital technologies and their spatial and communicative qualities and constraints. With so much attention given to the Web and to the garnering of procedural knowledge in many of the task-based approaches to learning environments, engaging students in the kinetic and conceptual experiences of mixed reality environments may widen their sense of composition from digitally mediated writing and picturing to connections between creative arts and tangible computing. Digital multiliteracies may be said to be on the move.

Towards designing for performativtiy

Concerning performativity, how we move with dynamic screens and their mediational kinesis will need to be studied and interpreted by students and critics alike. So too will we

need to investigate, by acts of creative composition, just how to design for performativity. There needs to be space also for students of media, computer science and the arts to also move from game spaces, such as The World of Warcraft, to differently shaped conceptual spaces, often ‘offscreen’, but still in the public domain, in galleries, museums and the mixed realities of urban architectures, design and advertising (see Morrison & Skjulstad 2007; Morrison & Skjulstad forthcoming).

Mixed reality media artworks available at art venues and conferences remind us that the environments and the affordances we embed and yet leave under-determined to allow for mediated participation can vary greatly. Flipping and turning between medium and mediation is what we increasingly experience and what students may need to learn to produce in anticipation of careers in the creative sector. These activities, situated in a developmental and production-based approach to technology-enhanced learning, may realise an extended multimodality that crosses boundaries between media, art, performance and informatics. This multimodality may be explored and enacted through our designs for movement. It may also be realised by way of our embodied interactions that include kinetic and proprioceptive aspects of mediated meaning-making.

Media education – outside of art and design schools at higher education level – needs to consider more broadly the cultural contexts of media use, and to assist students in developing knowledge about it from both production and analysis. This too demands that we rethink the notion and arguments for access to hardware, software and knowledge of digital media that has at times been somewhat functionalist as students learn new tools. For graduate media and informatics students, and others involved in learning to be choreographers or interaction designers, however, ‘Access needs to be seen not merely in terms of access to technology or to technical skills, but also to cultural forms of expression and communication …’ (Buckingham 2007: 115, italics in original). Access to expressive and communicative cultural forms may be realised through production-based education and learning to critique ‘new’ media that stretch notions of multiliteracies along with their unfolding creative practices and critical analysis.


This article is a direct result of collaboration on the MULTIMO project at InterMedia, University of Oslo, funded by the KIM programme, Research Council of Norway. For details see: Reflections on new media arts and their genesis related to Tapet is a part of an EU Culture 2000 funded project into mixed reality arts to which InterMedia contributes (GRIG: Guild for Reality Integrators and Generators). This article is an outcome of ongoing work into studying multiliteracies in multiple sites in the Transactions Research Group, Faculty of Education, University of Oslo, to which we also belong. Our thanks to the Faculty for their support of the work and to the Department of Media and

Communication for making the innovation space of the NEWME master’s course in new media education possible. We are grateful to the choreographer Inger-Reidun Olsen for her input to Tapet and to Martin Havnør for additional programming and set-up for the ITU event. Taken together, this work is part of the Communication Design Research Group at InterMedia; this publication is also an outcome of the Digital Design Research strategy at the University of Oslo. Critical comments from the reviewers, Synne Skjulstad and the Communication Design Group at InterMedia have contributed greatly to the final text.

E. & Kendrick, M. R. (Eds.) Eloquent Images. Cambridge: The MIT Press. 19–36.


Auslander, P. (1997). ‘Ontology vs. history. Making distinctions between the live and mediatised’. 1997 Performance Studies Conference. At: http://webcast.

Birringer, J. (2003). ‘Dance, the body and the Internet’. Design and Performance Lab. Brunel University: London. At: html

Bolter, J. & Grusin, A. (1999). Remediation: Understanding New Media. Cambridge: The MIT Press.

Bolter, J. (2003). ‘Critical theory and the challenges of new media’. In Hocks, M.

Brown, C. 2006. ‘Learning to dance with Angelfish: choreographic encounters between virtuality and reality’. In Broadhurst, S. & Machon, J. (Eds). Performance and Technology: Practices of Verbal Embodiment and Interactivity. Palgrave Macmillan: Basingstoke. 85–99.

Buckingham, D. (1998). ‘Introduction: fantasies of empowerment? Radical pedagogy and popular culture’. In Buckingham, D. (Ed.). Teaching Popular Culture: Beyond Radical Pedagogy. UCL Press: London. 1–17.

Buckingham, D. (2003). Media Education. Literacy, Learning and Contemporary Culture. Cambridge: Polity Press.

Buckingham, D. (2006). ‘Defining digital literacy. What do young people need to know about digital media?’ In Digital kompetanse. Vol. 1, No. 4. 263–276.

Buckingham, D. (2007). ‘Media education goes digital: an introduction’. Learning, Media and Technology. Vol. 32, No. 2. 111–119.

Buckingham, D., Graham, J. & Sefton-Green, J. (1995). Making Media: Practical Production in Media Education. London: The English and Media Centre.

Burn, A. & Parker, D. (2001). ‘Making your mark: digital inscription, animation and a new visual semiotic’. Education, Communication and Information. Vol. 1, No. 2. 155–179.

Burn, A. & Durran, J. (2006). ‘Digital anatomies: analysis as production in media education’. In Buckingham, D. & Willett, R. (Eds). Digital Generations: Children, Young People, and New Media. London: Lawrence Erlbaum. 273–293.

Cope, B. & Kalantzis, M. (2000). (Eds). Multiliteracies: Literacy Learning and the Design of Social Futures. London: Rout-ledge.

Courchesne, L. (2002). ‘The construction of experience: turning spectators into visitors’. In Resier, M. & Zapp, A. (Eds). New Screen Media. BFI: London. 256–267.

Deacon, A., Morrison, A. & Stadler, J. (in press 2008). ‘Multimodal production and semiotic resources for learning about film narrative’. In Morrison, A. (Ed.) Inside Multimodal Composition. Hampton Press: Cresskill NJ.

Dourish, P. (2001). Where the Action Is. Cambridge: The MIT Press.

ESG (Extended Stage Group). (2004). ‘Interactive generative stage and dynamic costumes for Andre Werner’s “Marlowe: The Jew of Malta”.’ In Leopoldseder, H., Schopf, C. & Stocker, G. (Eds). Cyberarts 2004. Ars Electronica/Hatje Cantz: Linz. 130–132.

Fagerjord, A. (2003). Rhetorical Convergence. Earlier Media Influence on Web. Media Form. Oslo: Unipub forlag.

Fish, S. (1980). Is There a Text in This Class?: The Authority of Interpretive Communities. Cambridge: Harvard University Press.

Fuller, M. (2005). Media Ecologies: Materialist Energies in Art and Technoculture. The MIT Press: Cambridge.

Fishwick, P. (2006). (Ed.). Aesthetic Computing. The MIT Press: Cambridge.

Gillette, D. (2005).‘Lumiere ghosting and the new media classroom’. Kairos. Vol. 9, No 1. Accessed 1 February 2008:

Grau, W. 2007. (Ed.). MediaArtHistories. The MIT Press: Cambridge.

Havnør, M. (2006). A Conceptual Framework for Development Iterations of Digital, Interactive Art; System Response Patterns and Multi-Narrative Temporal Data Based Presentation. Unpublished Master’s thesis. Oslo: Department of Informatics, University of Oslo.

Hawisher, G. & Selfe. C. (1999). (Eds). Passions, Pedagogies and 21st Century Technologies. Logan: Utah State University Press.

Jewitt, C. & Kress, G. (2003). Multimodal Literacy. New York: Peter Lang.

Jewitt, C. (2006). Technology, Literacy and Learning: a multimodal approach. London: Routledge.

Knobel, M. & Lankshear C. (2007). ‘Sampling “the new” in new literacies’. In Knobel, M. & Lankshear, C. (Eds). A New Literacies Sampler. New York: Peter Lang. 1–24.

Kress, G. (1999). ‘English’ at the crossroads: rethinking curricula of communication in the context of the turn to the visual’. In Hawisher, G. and Selfe, C. (Eds.). Passions, Pedagogies and 21st Century Technologies. Logan: Utah State University Press. 66–88.

Kress, G. & van Leeuwen, T. (2001). Multimodal Discourse: the Modes and Media of Contemporary Communication. London: Arnold.

Lankshear, C. & Knobel, M. (2006). ‘Digital Literacy and Digital Literacies: Policy, Pedagogy and Research Considerations for Education’. In Digital kompetanse. Vol. 1, No.1. 12–24.

Lash, S. & Lury, C. (2007). Global Culture Industry. The Mediation of Things. Polity Press: Cambridge.

Lemke, J. (2005). ‘Place, pace, and meaning: multimedia chronotopes’. In Norris, S. & Jones, R. (Eds). Discourse in Action: Introducing Mediated Discourse Analysis. Routledge: London. 110–122.

Liestøl, G. (2001). ‘Research into the development of digital media as an interdisciplinary field based in the sciences and the humanities’. In Liestøl, G. & Morrison, A. (Eds). Tverrfaglighet og digitale medier. (Crossdisciplinarity and Digital Media). Oslo: UniPub/Inter-Media. 41–55.

Liestøl. G., Morrison, A. & Rasmussen, T. (Eds.). (2003). Digital Media Revisited. Cambridge: The MIT Press.

Liestøl, G. (2003). ‘“Gameplay”: From synthesis to analysis (and vice versa): topics of conceptualisation and construction in digital media’. In Liestøl, G., Morrison, A. & Rasmussen, T. (Eds.). (2003). Digital Media Revisited. Cambridge: The MIT Press. 389–413.

Machin, D. (2007). Introduction to Multimodal Analysis. New York: OUP.

Melrose, S. (2006). ‘Bodies without bodies’. In Broadhurst, S. & Machon, J. (Eds). Performance and Technology: Practices of Verbal Embodiment and Interactivity. Palgrave Macmillan: Basingstoke. 1–17.

Morrison, A. (2003a). ‘Dancing with postcolonial theory: digital scenography and the performance of local culture’. Norsk medietidsskrift. Vol. 10, No. 2. 28–56. Available: asp?I=26&C=20&D=2

Morrison, A. (2003b). ‘From oracy to electracies: hypernarrative, place and multimodal discourses in learning’ in Lie-stol, G., Morrison, A. & Rasmussen, T. (Eds). Digital Media Revisited. The MIT Press: Cambridge. 115–154.

Morrison, A. (in press). ‘Views from the inside out’. In Morrison A. (Ed.). Inside Multimodal Composition. Hampton Press: Cresskill.

Morrison, A., Deacon, A. & Stadler, J. (2005). ‘Designs for learning about film genre’. Proceedings of In-The-Making. First Nordic Design Conference, Copen-hagen. Available:

Morrison, A., Sem, I. & Havnør, M. (in press). ‘Behind the wallpaper: performativity in mixed reality arts’. In Morrison, A. (Ed.) Inside Multimodal Composition. Creskill: Hampton Press.

Morrison, A. & Skjulstad, S. (2007). ‘Talking cleanly about convergence’. In Storsul, T. & Stuedahl, D. (Eds). The Ambi-valence of Convergence. Nordicom: Gothenburg. 217–335.

Morrison, A. & Skjulstad, S. (forthcoming). Literacy – Technology – Cultural Techniques. How Does Communication Technology Mediate Culture? In Gentikow, B., Skogseth, E. & Østerud, S. (Eds.). Literacy and Mediated Cultures. Creskill NJ: Hampton Press.

O’Halloran, Kay. (2004). (Ed.). Multimdoal Discourse Analysis. London: Continuum.

Paul, C. (2003). ‘Public Cultural Production Art(software)’. In Stocker, G. & Schöpf, C. (Eds). CODE: The Language of Our Time. Osterfildern-Ruit: Hatje Cantz Verlag. 129–135.

Penny, S. (2004). ‘Towards an aesthetics of behaviour’, an abridged version of the paper given at ISEA2004 Tallinn Conference entitled ‘After Interdisciplinarity’.

Reid, A. (2003). ‘New media’s long history and global future: the Uniplanet project’ Kairos. Vol. 8, No. 1. Accessed 1 February 2008: http://kairos.technorhetoric. net/8.1/index.html

Royce, T. (2007). ‘Intersemiotic complementarity: a framework for multimodal discourse analysis’. In Royce, T. & Bowcher, W. 2006. (Eds). New Directions in the Analysis of Multimodal Discourse. London: Routledge. 63–109.

Schiller, G. (2005). ‘Awakening the “dynamic index of the body”: linking movement awareness methods with interactive art practice’. International Journal of Performance Arts and Digital Media. Vol. 1, No. 3. 179–188.

Schön, D. (1987). Educating the Reflective Practitioner: Toward a New Design for Teaching and Learning in the Professions. San Francisco: Jossey-Bass.

Sefton-Green, J. (2005). ‘Timelines, timeframes and special effects: soft and creative media production’. Education, Communication & Information. Vol. 5, No. 1. 99–110.

Sefton-Greene, J. (2006). New Spaces for Learning: Developing the Ecology of Out-of-School Education. Hawke Research Institute Working Paper Series. No. 35. Accessed 1 February 2008.

Sem, I. (2006). Practice-based Method. Exploring Digital Media through the Dynamics of Practice, Theory, and Collaborative, Multimedia Performance. Unpublished Master’s thesis. Department of Media and Communication. University of Oslo. At:

Schiphorst, T. (2005). ‘Soft, softer, softly: whispering between the lines’. In Brouwer, J., Fauconnier, S., Mulder, A. & Nigten, A. (Eds.). aRt&D: Research and Development in Art. V2_Publishing/NAI Publishers: Rotterdam 166–176.

Skjulstad, S. & Morrison, A. (2005). ‘Movement in the interface.’ Computers and Composition. Vol. 22, No. 4. 413–433.

Skjulstad, S., Morrison, A. & Aaberge, A. (2002). ‘Researching performance, performing research: dance, multimedia and learning’. In Morrison, A. (Ed.) Researching ICTs in Context. Inter-Media/UniPub: Oslo. 211–248. Available:

Sullivan, P. & Porter, J. (1997). Opening Spaces: Writing Technologies and Critical Research Practices. Ablex: Greenwich CN.

van Leeuwen, T. (1999). Speech, Music, Sound. London: Macmillan Press.

Ulmer, G. (1998). ‘Foreword/forward (into electracy)’. In Taylor, T. & Ward, E. (Eds) Literacy Theory in the Age of the Internet. New York: Columbia University Press. ix–xiii.

Wertsch, J. (1991). Voices of the Mind: A Sociocultural Approach to Mediated Action. Cambridge MA: Harvard University Press.

Wysocki, A., Johnson-Eilola, J., Self, C. & Sirc, G. (2004). Writing New Media. Logan: Utah State University Press.

Zuniga Shaw, N. & Lewis, M. (2006). ‘Inflecting particles: locating generative indexes for performance in the interstices of dance and computer science’. Performance Research. Vol. 11, No. 2. 75–86.