In-Between. Intermedial understanding and analysis of children’s literature
Exemplified by the digital story NORD (2018)
- Side: 1-15
- DOI: https://doi.org/10.18261/issn.2000-7493-3031-01-03
- Publisert på Idunn: 2021-04-22
- Publisert: 2021-04-22
- Creative Commons (CC BY-NC 4.0)
Interactions between image, written text and sound are especially prominent in children’s literature, just as children’s literature tends to be embodied in a variety of media. Accordingly, it is highly relevant to use theories of intermediality that focus on relations and interactions between different forms of expression and media, and theories focusing on the relations and interactions between texts and readers that are prompted by these conditions. However, intermedial theories have been only sparingly applied in research into children’s literature and even less as an analysis strategy. Still, in the age of digitalization the intermedial understanding and analysis of children’s literature deserve greater attention because digital literary texts for children often employ a combination of writing, pictures and sound involving film, animation and interactivity. This article aims to present an understanding of children’s literature from the perspective of intermedial theory and method, and to offer a model for intermedial analysis. Drawing on Elleström’s theory of intermediality combined with concepts relating to literary materiality and videogames, the model focuses on four intermedial and cross-aesthetic analytical levels: material, sensory, spatio-temporal and semiotic. The model will be exemplified and employed in an intermedial analysis of the born digital story NORD by Hübbe, Meisler and Olsen (2018). NORD is a coming-of-age story in the fantasy genre, as well as being a contribution to the eco-critical tradition. This will be exemplified in the article, although the main focus will be on the way NORD is positioned intermedially in-between forms of expression, content and media.Keywords: Intermediality, materiality, digital literature for children, creation of meaning, touch interaction, interface, interactivity
Opening chapter 1 in NORD (Figure 1), the reader is met with the nerve-shattering sound of rain. The heavy rain is visible as drops on the screen, and three figures are discernible in the background in a foggy, mountainous and inaccessible landscape. At the top, a line of white dots and a compass needle can be seen. By moving your fingers or the cursor on the screen, you open the chapter and go through transparent layers into a harsh landscape with constant rain accompanied by deep thunder, which continues throughout the chapter. Your attention is caught by some moving stones which, when touched, produce a box with text read aloud by a female voice. When you approach the three figures, they turn out to be the three norns from Norse mythology. They look directly at the reader, and are dressed in quite modern hiking gear for goddesses of fate (Figure 2). Further on in the chapter, the sound includes piercing bird cries, low-pitched dirges, and sequences of instrumental music. So, in the reading of NORD, the meaning-making process is closely tied to the use of touch, hearing and sight, and the reading becomes a multisensory process and experience.
The aim of this detailed description of the opening of NORD is to give an example of the intermedial interplay operating in children’s literature. In principle, all literature and art can be described using the intermedial approach (Elleström 2010, 2014 and Bruhn 2010, 2012, 2016); but children’s literature in particular, with its close connection to orality and its extensive combination of words and images in e.g. picturebooks, invites this approach. The intermedial approach’s focus on coexistence and simultaneity between various forms of expression can also be applied to literature in paper books/codex. However, as it is especially prominent in digital children’s literature, the case chosen for this article is the born digital young adult story NORD by Camilla Hübbe, Rasmus Meisler and Roar Skal Olsen (2018), which can be read via computer or tablet. NORD is about a 14-year-old girl called Nord who sets out to find her mother, who has disappeared. Along the way she meets the squirrel boy Ratatosk, the three norns Urd, Skuld and Verdante, and the hair-raising monster Nidhug, which gnaws the roots of the Tree of Life deep down in Nifelheim. Gradually Nord realizes that a new Ragnorok is brewing. NORD is narrated through text, picture, sound and simple animations, and the reader progresses through the story by finding and touching interactive stones that make text boxes appear. The text is read aloud, but this function can be disabled, just as other languages can be enabled (the story is available in Danish, Norwegian, Swedish, Icelandic, Faroese and English).
There Is No Written Children’s Literature
Kristin Hallberg’s concept iconotext (Hallberg, 1982), and W.J.T. Mitchell’s concept imagetext (Mitchell 1994), have contributed important insight into picturebook research (e.g. Nikolajeva & Scott 2000, Christensen 2003 and Druker 2008). They both emphasize that picture and text are mutually associated, equivalent, and not to be separated in the reception process. In other words, picture and text should be seen as a whole. This mutual interchange and simultaneity between various forms of expression and art is also central in intermediality theories. This theory complex will be further elaborated later on, but the overriding notion is that literature and other art forms are complex and in constant exchange with each other. Mitchell’s books and articles consistently express this condition, and in recent years he has developed his theory to include not only the relation between text and image, but also the complexity and interdependence of all forms of art and expression. His point is that art forms are composite, and that all media are mixed and diverse: “My claim is that the image text is the convergence point of semiotics, the theory of signs, and aesthetics, the theory of the senses. It is the place where the eye and the ear encounter the logical, analogical, and cognitive relations that give rise to meaning in the first place.” (Mitchell, 2015, pp. 47).
Mitchell emphasizes the mutual involvement of writing, image and sound in a text, as well as the verbal, visual, auditive and tactile aspects of the complex sensorial address. He characterizes these relations as a “murmur of discourse and language” (Mitchell, 2015, pp. 39), and these murmurs and mumblings are reflected in relations, syntheses and ruptures (Mitchell, 2015). However, the concept of imagetext neglects sound, speech, gesture and spatiality semantically, a criticism of which Mitchell is aware (Mitchell, 2015, pp. 44). As an alternative, I apply the concept of intermedial literature to literary texts which combine writing, picture, sound and interaction in a simultaneous perception (Henkel, 2017). Mitchell’s contribution inspires research into intermediality and leads to an emphasis on the sensory approach to media and art works as well as on a higher degree of equality between different forms of expression.
Following the acknowledgement of the equivalence of picture and writing, research into picturebooks has gained an early intermedial understanding which may increase in relevance as the fluctuation between literature and other media increases in digital literature for children. For example, Mitchell has made it programmatically clear in his article “There Are No Visual Media” (Mitchell, 2005) that all media are complex because they all involve a mix of sensory, perceptual and semiotic elements (Mitchell, 2005, pp. 260). In the context of children’s literature, this premise may inspire an intermedial recognition: there is no such thing as written children’s literature. First, the black word on white paper will in itself be a written picture by virtue of typography, font size, layout, etc., creating a relation between different visual forms of expression. Secondly, this written picture will always be physically imbedded, e.g. in a codex or a tablet computer, which contributes to the creation of meaning. Thirdly, literature in general and children’s literature in particular operate through an interaction between writing, sound, music, still images, moving images, and speech, as we see in NORD.
Theoretical and methodological framework
The premise for this article is that children’s literature should be seen in an intermedial culture, and that it is interesting to consider how literature for children and young adults develops in a dialogue with the changes in the media landscape. Such a development involving intersections between technology, art and culture can be viewed historically, the pivotal example being the invention of the printing press in the fifteenth century, which was a defining event for the book medium. Currently, this development is taking place in an exchange with other technical media such as computers, tablets and smartphones, which have released literature from its preferred medium until now (the paper book) and eliminated the natural connection between the book and literature. Literature in an intermedial culture involves various media, thereby gaining increased opportunities for combining writing with other forms of expression such as picture and sound, and ways of inviting touch interaction with the text.
The notion that texts are a compound of various forms of expression and art is not new. After all, the premise for young children’s access to literature is that writing and pictures interact, and that children encounter literature audibly when adults read aloud to them. In contemporary intermedial culture, picturebooks are attracting more older readers owing to the growth of the graphic novel; and digital development has facilitated and simplified access to audiobooks with mobile devices and streaming services.
The intermediality perspective has been highlighted earlier in children’s literature research (e.g. Rhedin 1992, Druker 2008, White 2011, Bjørlo 2018) and has received renewed accentuation in recent years due to the many possibilities that come with digitalization (Henkel 2017, Kurwinkel 2018 and three anthologies edited by Tønnessen 2014, Kümmerling-Meibauer & Surmatz 2011, and Weinkauf, Dettmar, Möbius & Tomkowiak 2014). Nordic Journal of ChildLit Aesthetics excels at having an extensive collection of articles thematizing intermedial conditions related to digital literature for children and children’s reading of this literature (Schwebs 2014, Turrión 2014, Al-Yaqout & Nikolajeva 2015, Henkel 2015, Søyland & Gulliksen 2019, Guanio-Uluru 2019 and Hagen 2019). In light of the pronounced presence of intermedial conditions in the literature for children and young adults, it is, however, remarkable that this approach is rarely applied throughout as a strategy of analysis and, when applied, then only on a limited text corpus.1E.g. Druker 2008 with regard to modernist picturebooks, White with regard to the photo-lyrical picturebook 2011, and Bjørlo 2018 with regard to poetry picturebooks and art picturebooks.
When the concept of intermediality has been applied, it has often been through an understanding of the adaptation of a narrative content between media, typically from book to film, theatre or sculpture (e.g. in Kümmerling-Meibauer & Surmatz 2011 and Weinkauf et al. 2014), but without the application of intermediality theories that could otherwise contribute with conceptual and analytical conclusions regarding the complex and composite nature of literary artefacts per se. Only in picturebook research is the depth of the intermediality perspective present, implicitly and explicitly. Christensen (2003) and Druker (2008), among others, include Mitchell’s theory complex, which also forms the basis for other researchers into intermediality (e.g. Elleström 2010, Jørgensen, Paldam & Steffensen 2012, Rippl 2015, and Bruhn 2016).
The present article is based primarily on theories developed by Lars Elleström and Jørgen Bruhn, who have provided theory, models and operationalizations in relation to the theoretical, analytical and didactical (Bruhn) potential for offering cross-aesthetic levels of analysis which can be applied to other texts. Elleström develops a refined understanding of media and a cross-dimensional, general way to understand the modalities of media (Elleström 2010 and 2014), thereby transcending traditional dichotomies such as text versus image. He stresses that all media are characterized by four basic and interacting modalities: 1) material modality, 2) sensorial modality, 3) spatio-temporal modality, and 4) semiotic modality. His approach is hermeneutic (e.g. Elleström 2010: 37 and 2014: 24), and he accentuates the perception process and emphasizes that meaning is ascribed via sensory perception of media. Bruhn is inspired by Elleström, and also underlines that combination and heterogeneity are a priori assumptions for all texts (Bruhn, 2010); he establishes a theoretical frame and applies intermediality as a concept and literary method (Bruhn, 2016). His essential argument is that intermediality concerns not only the relation between forms of expression in literary texts, but also the literary characters’ sensory impulses and intermedial position in the world (Bruhn, 2016). Thus, his focus is on intermediality in terms of form and content, which is of interest in realistic young adult fiction reflecting life with and between media, but also in fantasy narratives like NORD, as I will expand upon below.
In conclusion, Elleström and Bruhn point out that forms of expression, media and sensory perceptions are never isolated and pure, even though attempts can be made to distinguish between them analytically; and that this perspective expresses the conditions for human understanding of and interaction with the world and for literature in general. This approach is applied to a more specific understanding of cross-aesthetic aspects and intermedial relations, with a focus on the relationship between writing and other forms of expression and art in children’s literature, as exemplified in NORD. So, the intermedial analysis is cross-aesthetic because synchronic analyses can identify sensory appeals and the simultaneity between forms of expression, with Elleström’s four modalities as its point of departure. He registers the four modalities at a general level, and is not occupied with text analyses. In order to probe deeper into the understanding and analysis of text, points from materiality and videogame research are included. Thus, the article serves two goals: to demonstrate through the intermedial and synchronic analysis levels how various art forms and media meet in the creation of meaning and are made available for the reader’s perception, and to offer the four intermedial levels as an analysis approach which can be generally applied to (digital) literature for children.
1. The material creation of meaning
At the material level, it is important to focus on the media in which a literary text is realized, i.e. whether the literature is accessed via a paper book, computer, tablet or smartphone. However, the materiality perspective is not only about objects, but also about the significance of the object (the media) for the reader’s reception (Elleström, 2010, pp. 37 and 2014, pp. 24), and the wider scope of the media in relation to culture and technology, as pointed out by literary materiality research in recent years (e.g. McGann 1991, Hayles 2002, 2004 and 2008). In the material creation of meaning, the focus is placed on two aspects of the literary text: the physical interface of the media, its manifestation and tactile communication with the reader, and the reader’s experience and exchange with this object. Therefore, the object in question can be digital because its materialization is not always concrete. In this context, the interesting point concerns the text’s way of being-in-the-world. Accordingly, the way the reader meets NORD matters, whether it is via a tablet, a computer with a mouse and keyboard, or with the “gamer keys”, i.e. WASD. If NORD is read via a tablet, the story is materially characterized by the touch-dependent interface, with the fingers quite literally opening the literary universe. If NORD is read via a computer, the use of WASD can make the experience just as intuitive, if the reader is familiar with video games. Since NORD is realized through simultaneity between writing, picture, sound and interactive aspects, its reception will vary from one reader to the next.
At the material level, the focus is thus on the relation between artefact and media, i.e. in this case between NORD and the computer or tablet, and the way the literary text is embodied. A central point for literary materialists such as N. Katherine Hayles is that literary criticism has been concerned with the “brain” of literature at the cost of its “body” and ways of materializing, and she develops the understanding of the “embodiment” of literature (Hayles, 2002, pp. 31, 32, 197). In NORD the fantasy universe of the story is digitally embodied in various ways. It is a story in which a magical world featuring norns, the Völva in various shapes and other creatures from Norse mythology is combined with a realistic universe in a flat in Copenhagen and on the island of Møn, and with Sejr’s dystopian universe in Iceland. The inner material creation of meaning in NORD arises in the interplay between writing, picture, sound compositions, the voice of the reader, and animations. In tandem with the interactivity, this interplay addresses the reader in a coming-of-age story which is about identity, adversity, evil, trust, love, temptations and destiny, while voicing harsh criticism of the exploitation of the planet’s resources.
In the interface, the reader moves through transparent layers and different environments or scenes, e.g. corridors in the underworld or photorealistic scenes like the cottage on Møn (Figure 3). To start reading a chapter, the reader must touch a play button (Figure 4, 5 and 6) and then use a compass needle at the top of the interface, where small circles indicate the page in the chapter. The transparent layers combine illustrations, photography and animations. The sound combines reading with popular music tracks, sound compositions and sound effects. The interactivity occurs when the reader touches stones to generate text. From the literary materiality perspective, NORD presents a dynamic and synchronous sum of different material articulations. In addition, the digital version is complemented by an illustrated novel (Hübbe & Meisler, 2017) and an audio book, thus being materially embodied in various forms. In this light, its embodiment can be seen and understood as “a literary work as something that is to be located not only in books, but between media” (Andersen, 2015, pp. 132) and can be characterized as a “cluster work” (Mygind, 2019, pp. 12).
2. The sensory creation of meaning
From an intermediality perspective, forms of expression and media are perceived as far more than representations, because they affect experience and sensory perception and are haptically imbedded. They stimulate and challenge the body’s interaction with the world. The understanding of textual physicality affects the understanding of the sensory appeal of the texts, which is the focal point at this second level. The sensory exchange with NORD in the first chapter was described above through the distressingly heavy rain. Similarly, throughout the story Nord is afflicted with a severe headache causing unpleasant piercing sounds and blue-purplish colors which affect the reader’s senses, and might even give him a headache too. In this manner, many of the symbol-laden leitmotifs of the story, such as the bees and the eagle, appear in different ways in writing, picture and sound. The bees and their strong connection to the norn’s “threads of fate”, which in NORD are replaced by flowers in a greenhouse, also carry an important message in the eco-critical perspective in the story. Their presence has already been made symbolically clear in the letter of O in the title that is shaped as an orange hexagon (Figure 1), referring to the form and color of beehives.
In the sensory creation of meaning in NORD, the vestibular sense is challenged several times. On one occasion, the reader shares Sejr’s point of view in a car accident when the wheel and the car turn around and create confusion, with the reader seeing himself, i.e. Sejr, in the rearview mirror (Figure 4). On another occasion, the vestibular sense is challenged as Nord sails and later swims in a turbulent sea with large waves (Figure 5), making it difficult for the reader to catch the interactive stones and move the story forward. Here the reading process becomes even more multisensorial, with many senses in play to convey the various points of view, moods and feelings in the story.
Thus, the sensory creation of meaning in the intermedial analysis indicates how literature is sensed. In most cases, people refer to the senses of sight, hearing, touch and taste. However, Elleström adds what he calls “sense-data” (Elleström, 2010, pp. 20), which arises from objects, phenomena and events and invokes a complex perception, for example by letting the reader hear the words or imagine the pictures. The interactive stones in NORD are an example of this complex perception: they are visually and substantially a part of NORD, they have symbolic value and fit into the harsh landscape of the norns; at the same time they are meta-fictional elements outside the story and are vital for the plot through the reader’s touch. Hence, they exemplify the “polyaesthetic” (Engberg, 2013) simultaneous appeal to the senses of touch, sight and hearing, which is characteristic of contemporary media aesthetics. They also contribute to the significance of the multisensory interface for the reading experience.
3. The spatio-temporal creation of meaning
The spatio-temporal level (Elleström, 2010) is about the complex constitution of time and space through width, height, depth and time, which in the intermedial perspective is created in the interplay between writing, picture, sound and interactivity. In NORD the reader moves through the transparent layers deep into the fictional space by using his fingers or a cursor. There are many horizontal and vertical movements in the narrative space: the reader moves with Nord through endless staircases, over bridges, across vast mountainous landscapes, or down Mimer’s throat (Figure 6), through corridors and tunnels in the underworld, and, finally, upwards into the sky, where she enters a new world on the seed for a new Tree of Life which was hidden in the amber pendant she was given by the Völva, indicating new hope and a new age. Several times she literally falls into water or down a cliff (Figure 7), and these falls in space are closely connected to the temptations she meets, e.g. anthropomorphized sexuality in the shape of the ram Greyback, a shapeshifter which has visual references to Iggy Pop. Thus, the movements in time and space have an impact – concrete and symbolic – as they physically reflect Nord’s mental development while being a connective link between the three worlds of the story.2Like, in another fantasy, a “mindscape”, to use a term taken from Nikolajeva (1988).
The spatio-temporal creation of space and spatialization is often supported by a change in point of view in the pictures, which change in sequences. The point of view in the written text is generally that of a third-person narrator, but it is complemented by the visual point of view. This is often Nord’s view, and the reader often finds himself at her eye level, e.g. in the above-mentioned chapter with lapping waves or, with a tilt of the view, at the concert where the reader is literally pushed forward among the audience (Figure 8). This point of view resembles the first-person shooter movements of video games, but in NORD this inner point of view alternates between different persons. For instance, the reader often borrows the point of view and the hand of one of the norns, causing a bodily shift from the reader to the interface. On other occasions, the point of view is that of the antagonist, Sejr, as in the car accident described above (Figure 4). Thus, the reader is invited to identify with the various persons through different points of view and movements in space.
The horizontal and vertical movements, like the change of perspective and the first-person perspective, contribute to the creation of what video game theorists call the phenomenological space, which allows bodily and sensorimotor events and the illusion of being present in the fictional room (Keogh 2018, Crick 2011, Walther & Larsen 2019). This experience of being a necessary actor in the room is created in NORD by means of changes of perspective as well as tactility and sound, with sound compositions and background noises from machines, trains, cars or munching sounds in the underworld creating space, atmosphere and intensity (Have & Pedersen, 2015), just as the interactivity can potentially create the immersion known from video games. NORD incorporates several game-aesthetic elements, even though the experience of space and freedom of movement is limited, and becomes a meta-fictive comment on the format. This is the case, for example, when Nord has to choose between three doors and the freedom of choice is only apparent, just as the interactive stones give the illusion of various movements and reading paths, but must be followed in a certain order. NORD plays with media references and with the reader, and tests written and unwritten conventions for children’s literature.
4. The collective semiotic creation of meaning
Elleström sees the semiotic creation of meaning or modality as being decisive, given that man is a sign-interpreting being (Elleström, 2010, pp. 22). One of the first semiotic signs the reader must decode and interpret in NORD is the interactive stones which set in motion the text and the reading, just as the above-mentioned play button is a semiotic sign the reader may be familiar with from other media. Before it appears at the start of each chapter, successively flashing keyboard letters and a flashing cursor button on a black background invite the reader to interact. The written text gives the location of the chapter, e.g. “Sejr. Eastern Island” and “Nord. The road to Hel, between worlds”. This introduction to each chapter is the only occurrence of text as the single form of expression on an interface toned down to black and white. Even so, these chapter pages are semiotically essential, requiring that the reader use his knowledge from video games and digital media in the decoding. This also applies to the consistent layout, with dots marking the page in each chapter and the compass needle aiding the reader to find his way round the universe and find the right way. It marks the navigational space and is one more video game-aesthetic element and a consistent meta-fictional feature, with the two elements pointing out the created nature of the story. The many different semiotic elements are not only formally but also highly substantially significant. The compass needle, for example, indicates that as the reader must find his way in the story and help Nord in her quest in the fictional world, he is challenged to make similar considerations about the direction of his own life.
As pointed out by Bruhn, intermediality is not only about forms of expression but also about content, including the presence of media at the level of the narrative (Bruhn, 2016). From this perspective it is interesting that what acts as a catalyst for Nord’s passage from the realistic world through the symbol-laden fog into the fantastic world is a technical medium, her mobile phone, stolen by Ratatosk to lure her into the fog and the magical parallel world (Nikolajeva, 1988). As a fantasy, NORD relies on Norse mythology, while including young people’s digital life with and between media. Another example of technical media and their influence on both the literary characters’ sensory impulses and those of the reader is the radio in the cottage (Figure 3). It is touch-interactive and plays a weather forecast announcing an approaching storm. However, the reader may not realize this, so this narrative lever is vitiated by a certain arbitrariness. In general, NORD is characterized by an assembly structure in which writing, images (both illustrated and photographic), reading, sound compositions, music, animation and video games “murmur” (Mitchell, 2015, pp. 39) together. Using Hayles’ concept, NORD can, as intermedial and digital literature, be described as a “hopeful monster” (Hayles, 2008, pp. 4), which negotiates and experiments with different art forms even though the semiotic creation of meaning – written and oral – is dominant.
Intermedial understanding and model for analysis of literature
Intermediality focuses on the way literature can be characterized as something which is situated in between forms of expression, content and media, entailing a relational and complex perception and reception. It indicates the conditions for man’s understanding of and exchange with the world, while saying something general about literature and something specific about digital children’s literature, which uses different forms of expression, art and interactivity. On the basis of its fluctuation between a mythological, realistic and science-fictional level and a fusion of written, visual and auditive spaces, NORD has been chosen as an example of an artefact in which content and forms of expression meet and create new forms.
With inspiration mainly from Elleström and Bruhn combined with knowledge from research on literary materiality and video games, the article crystallizes four cross-aesthetic analytical levels with the aim of emphasizing the reader’s aesthetic exchange with and perception of the text. The first level is the material creation of meaning, maintaining that texts are never pure and stable transitions but always embedded in a medium and a context, thus having their own “body language”. In the case of NORD, written text, picture, photo, animation, sound, music and various forms of interaction propel the story. Thus, NORD’s body is characterized by a complex textuality which holds perspectives for the reading culture, inviting the readers to employ their senses before everything else. This constitutes the basis for the second cross-aesthetic level of the analysis. At the sensory level, the analysis focuses on how the reader creates meaning in the meeting with the text through the senses of sight, hearing and touch. In NORD, visual and auditive sense appeals are of great importance as the writing in small boxes is read aloud by a mature female voice, suggestive of the literary character, the Völva, while background noises create depth, atmosphere and space. The reader meets another and quite concrete, sensory aspect in the form of haptic interaction with either computer or tablet, perceiving NORD through fingers and applying the vestibular sense. This leads on to the third level in the form of the spatio-temporal creation of meaning, where the focus is on the way the story is created in time and space. Through transparent layers the reader in NORD moves around in the space of the story, with movements high and low and deep into the underworld reflecting Nord’s mental development. At this level, interesting shifts in perspective between points of view in writing and images and between different characters occur. Altogether, the semiotic creation of meaning emerges through signs which are related to different art forms like literature, visual arts and music, and with different sense appeals like auditive and haptic impressions. The very potential of the intermedial and cross-aesthetic approach lies in the analysis of the relationship between the art forms and the way they support and/or challenge each other in the creation of meaning. Accordingly, our understanding of literature is being refined when the hierarchy of senses is challenged and when visual, auditory and tactile sense appeal are all weighted in the reading and interpretation process. In conclusion, the intermedial understanding of children’s literature as a “murmur” – to use Mitchell’s term – between art forms and media is exemplified in NORD since meaning is created in-between in the sense of coexistence, simultaneity and mutual exchange of expression, content and media.