Re-conceptualising picturebook theory in the digital age
Contemporary picturebook theory has produced a wide range of concepts and terms for the analysis of the various aspects of conventional picturebooks. However, as picturebooks are rapidly entering the digital age, there is an urgent need to keep picturebook theory up to date. The multimodal nature of picturebooks, that so far has predominantly implied a combination of the verbal and the visual modes, is expanding to include auditory, tactile, and performative dimensions. This article explores the ways that digital picturebooks, or apps, demand not only new approaches, but also new terminology to describe features characteristic of the new medium, as well as conventional picturebook features acquiring a new significance in digital picturebooks. These include materiality, paratexts, page layout and performance modes. Special attention is paid to the ways app developers employ the spatio-temporal affordances of digital visual texts. The article also investigates the various levels and types of user participation, in a range from merely swiping between screens to co-creating the narrative. The predominantly theoretical argument of the article is illustrated by a selection of digital picturebooks, where relevant in comparison to printed versions.Keywords: picturebook apps, picturebook theory, digital literacy, children's media, interactivity, multimodality
During the past decade picturebooks not only have become a legitimate object of academic pursuit, but also generated a whole theoretical field, producing a set of concepts and terms. However, as picturebooks are rapidly entering the digital age, there is an urgent need to keep picturebook theory up to date. The multimodal nature of picturebooks, that so far has predominantly implied a combination of the verbal and the visual modes, is expanding to include auditory, tactile, and performative dimensions. Today, digital picturebooks are often the first literature young children engage with. While the artistic and educational value of digital picturebooks as opposed to printed picturebooks has been questioned, they doubtless deserve serious academic study. Much attention has been paid to children's digital literacy (e.g. Dresang and Kortla 2009; Mackey 2011); less so to digital picturebooks themselves, apart from educational advice (e.g. Bird 2011). However, if we are to discuss digital picturebooks academically, we need a new meta-language, just as a new meta-language emerged within children's literature studies when picturebooks became a prominent area of scholarship. Recent scholarly investigations of digital picturebooks successfully bring together several disciplines, including game studies, hypertext theory, and narrative theory, to develop new approaches to the new phenomenon (Schwebs 2014; Stichnothe 2014; Turrión 2014). Yet this so far limited research has been inconsistent in terminology, beginning with the object of study itself. There is no agreement among scholars as to whether these texts should be called digital picturebooks, electronic picturebooks, e-picturebooks; or should the label pertaining to a particular kind of multimedial digital text, a picturebook app(lication), be expanded to cover the whole range: from existing printed picturebooks merely scanned and displayed on a device screen to a fully animated narrative that only with some reservations can be called a book. At the moment it is impossible to predict which label will become established. We have therefore decided, for the purpose of this article, to refer to printed picturebooks as books and to digital picturebooks as (picturebook) apps, well aware that not all multimedial digital texts are strictly speaking apps.
Further, while we can borrow concepts from the established picturebook theory, including its central concepts of iconotext, materiality, and sequentiality, some characteristic features of digital picturebooks need definitions and labels, before an avalanche of publications creates a terminological chaos. In her previous research, Ghada Al-Yaqout (2011, 2012) has coined a number of terms to describe some phenomena that occurred in early digital picturebooks, such as the icon, the opening screen, the virtual library and the sub-icon (the latter two usually found in picturebook series). In this article, we intend to develop the terminology more systematically and explore some of these medium-specific features, as well as printed picturebook features that work differently in digital medium and therefore need new terms. We examine the many shades of apps available at the moment, exemplifying our discussion with a selection of titles representative of the range, without aspiring to be exhaustive. Thus, text analysis is not the primary objective of this research. Moreover, the purpose is not to compare books and apps, and evaluate printed picturebooks and picturebook apps; neither are we interested in the process of remediation as such, although we will explore the implications of transferring a printed picturebook into a digital form. Our focus is on what concepts and analytical tools from existing picturebook theory are fruitful to employ when examining digital picturebooks; which of these need adjusting to the new medium; which are irrelevant or redundant; and not least which new terms are necessary to address new features unknown from printed books.
THE (IM)MATERIALITY OF APPS
The material existence of books and apps is radically different. Printed picturebooks are purchased from a bookshop (brick-and-mortar or virtual) or borrowed from a library or a friend. Apps are available from a digital store. Locating an app is not yet straightforward. To find a particular app you frequently need to know what you are looking for. Some virtual libraries, or platforms, allow you to search by category, which can be anything: children's books, kids’ books, picturebooks, and picture books. Platforms such as MeBooks have already made a selection of children's picturebooks that makes it easier to browse, similar to a children's department in a conventional or a virtual bookstore, or a library. The availability is, however, limited to the selection provided by the platform. Initially it was not possible to browse an app before you bought it; some platforms now allow you to test a sample before making a purchase. Some publishers have their own app stores. You buy the latest version, and your software periodically prompts you to update; therefore there is no way of ever retaining a first version, something that with printed picturebooks is of value, as comparisons between editions can occasionally be illuminating. Early updates only contained technical changes. Recent updates contain new features that may add to the aesthetical experience of the narrative, similarly to new book editions. Yet since there is no way to save the earlier digital version the comparison between versions is so far hardly possible. It is conceivable that in the future, updates may change covers and develop new features.
With few exceptions, such as movable books, the only action expected from a printed picturebook reader is page turning. With apps there is a wide spectrum of levels and modes of interaction, and, as the aforementioned publications emphasise, some apps afford and encourage more interactivity than others. Alluding to Marc Prensky (2001), Stichnothe (2014) makes a distinction between “digital immigrants,” that is, texts transferred from print to screen, and “digital natives,” apps developed directly for digital devices that fully utilise the medium. These binaries are not particularly useful since they do not reflect the wide variety of digital texts. Turrión (2014) suggests a range of false—simulated—real participation, which is helpful, but not exhaustive. Let us therefore examine what different existing apps expect their users to do.
The least imaginative way of employing the medium is a printed picturebook straightforwardly digitalised, for instance, scanned, doublespread by doublespread, and delivered on a device. The only difference in interaction with such a text is that instead of turning pages we swipe between screens. The swiping can be accompanied by the sound of pages turning and occasionally by music. Sometimes, but far from always, these apps retain paratextual elements of books, such as front and back covers, endpapers, the title page, or the copyright page. As known from picturebook theory, paratexts are an important part of the picturebook as a whole, and omitting paratexts in a digital version interferes with the aesthetic experience of the narrative. The doublespread layout is often the same, including the gutter and the position of the text. Let us call this type of app non-performed (Turrión does not offer a term for it in her categorisation). They typically do not have any menus with function buttons. Most apps available in MeBooks belong to this type.
Goodnight Moon app is also a non-performed text, with original doublespreads and a gutter, but some interactive features are added, and both traditional and app-specific paratexts are present. The opening screen shows the book's front cover and two other volumes, which works like a page in a catalogue or possibly like the back cover of a printed book, on which related books are advertised. Music and tapping add two sensory elements to our experience: auditory and tactile. Thus, as already pointed out, the inherent multimodality of the picturebook is amplified.
Some digitalised texts are performed, that is, read out loud and accompanied by music. There is no option to read yourself, but you can turn off the music. This type is similar to an audiobook, where the recording is accompanied by a printed book and where you look at pictures and listen to text. Users have less control over their experience than in the non-performed version, because voice and music are imposed on them, although they still need to swipe the screens.
In the very first app launched in 2010 together with the iPad, The Cat in the Hat, some elementary possibilities of the digital medium were employed, such as zoom—a function for obvious reasons not available in a printed book. Zooming in and out is imposed on users, whether they read alone or are being read to. In such apps, the gutter of the original has been appropriately eliminated and the space of the virtual spread is fully utilised. Thus, the potential of the spatiality of the digital medium is realised. However, the user has no control over zooming or other movements. The text appears after a movement, and the original position of text and image is not retained. Dear Zoo is an interesting example of a successfully remediated text: whereas in the book you open flaps, in the app you tap on the crates that open to reveal various animals; the new medium is creatively employed.
At the other end of the spectrum, digital texts are so far removed from the printed picturebook that they come closer to an animated film, for instance The Fantastic Flying Books of Mr. Morris Lessmore, that indeed started as animation before it was transmediated into other formats (see Schwebs 2014). However, unlike conventional animated films they contain swiping, hotspots, and various performance modes. The boundary between book and animation become blurred; but as frequently happens, it is in the border area that interesting experimental, hybrid texts emerge.
Whilst recent picturebook studies emphasise the materiality of the book, an app makes materiality intriguingly ambiguous. The device on which the app is delivered is unequivocally material, while the app itself is purely virtual. A feature that an app cannot emulate is the spine. Spines are not essential for picturebooks and sometimes get lost in paperback editions. However, as Al-Yaqout (2012) shows, spines can be an important element supporting recognisability and continuity. In bookshops and libraries, books are displayed with spines outwards, providing ideas about the size and colour of the book, the font of the title and the author's name; occasionally a series or publisher's logo.
While what we otherwise meet first in a book is the cover, the first paratextual constituent in an app is the icon. The icon is the closest feature to a book front cover, albeit in an abridged version. Therefore, just as we judge the book by front and back covers, and by flaps if there are any, we judge an app by the icon. The icon appears on the homescreen, accompanied by the shortened or corrupted title. Icons are of the same size and format, square with rounded corners, as other icons on your device. Size and format are important material picturebook features, and the standardised icon format, like the standardised screen, limits the variety and diversity of picturebooks expressed through these material features. Some icons are single images, without verbal text. Interestingly, these icons display “iconic” characters that are supposed to be recognised by the recipients (The Cat in the Hat, Elmer, Lorax, Miffy). In series books, this single image is accompanied by a smaller sub-icon at the bottom left corner, to signify that the larger icon belongs to a set. For instance, the Lorax icon has a sub-icon of The Cat in the Hat, while Elmer's Special Day has the icon of Elmer the Elephant to indicate that the character is the same. Thus the sub-icon signals the continuity of author, series or character, just as logos or other paratexts do in printed books.
Other icons are cropped images of the cover, showing the title, but not the author. They thus ignore an essential part of the paratext. The Heart and the Bottle icon shows the title, but since the image is cropped, you cannot easily guess that it is a bottle, and the heart is not visible at all. The icon disturbs the text/image interplay of the original cover. The protagonist, not mentioned in the title, but an important element of the book cover, is cropped too. The icon, thus, does not have the same cognitive and emotional appeal as the book cover. The icon of Goodnight Moon, on the contrary, introduces a character absent on the book cover. The absence creates uncertainty and evokes curiosity. Naturally, one can argue that an icon is not really a part of an app; therefore, it does not matter how it interacts with the user. It could hypothetically be a simple button with the title underneath. Yet if we, with printed picturebooks, take various paratexts into consideration, app icons are indeed integral to them.
In addition to icons, apps have opening screens, another feature both similar to and different from book covers. The opening screen of The Heart and the Bottle is similar to the book cover, with the addition of the “click here to begin” button that may feel distracting, especially as it utilises the significant red colour connected with the heart. Goodnight Moon moves on from the opening screen to another screen corresponding to the front endpaper and the title page of the book. The mock endpaper is interactive: three icons allow users to personalise the app with their photo and name. Interesting as it is, this feature merely emulates a similar element in a book where users can write their names; it is a question of delivery, not function; the only true new element is that they can post the page to Facebook.
In The Heart and the Bottle the opening screen is followed by yet another screen that offers basic instructions. The third screen, the title screen, provides further tips. The title screen is almost identical with the printed title page; only the mutual position of the characters is different. However, touching the image makes the girl move to the left, stopping at the position corresponding to the title page. This is a significant movement: backwards rather than forwards in the narrative. In the book, it is merely suggested, while the app actually realises it, with the user's assistance.
The back matter of apps can contains back endpapers, but they frequently omit back covers. They may have tabs that work similarly to epitextual blurbs on back covers or flaps, providing information about the author, the publisher, or other apps by the same author or in the same series.
Once again, as we know from picturebook theory, page layout is an important aesthetic feature affecting our perception. Publishers of printed books frequently ignore page layout in reprints and anthologies. Picturebook app developers are in addition restrained by the medium, that is, the determined size and format of the screen. Although the device can be rotated to emulate portrait and landscape format, it cannot reflect the vast variety of sizes or formats of printed books. The fixed area of the screen does not allow utilising landscape format to emphasise a broad panoramic view in a printed doublespread. The portrait format with facing verso and recto can be emulated on the screen, with or without the gutter. Scanned picturebooks frequently retain the gutter. Others use full screen, either portrait or landscape, to correspond to a page rather than a doublespread, which does not allow the important balance between verso and recto. Similarly, the separation of text and image on the facing pages, that may be significant for the pace of the narrative, cannot be recreated in an app, except in scanned texts discussed above. Multi-panel doublespreads are sometimes presented on separate screens. This eliminates the gutter, but disturbs the original page layout. The position of the verbal text is frequently the same as in the book; however, many apps have a zoom function that moves between text and image, sometimes without the user's control.
In printed books, borders and frames effectively distance the reader from the fictional space, while borderless spreads invite us to step inside. Although the visual point of view firmly positions us as spectators in front of the theatrically staged setting, the absence of borders makes the “curtain” between the stage and the viewer almost non-existent. In apps, there are several frames or borders between the user and the narrative space: visible cover edges in scanned books, various backgrounds and the black borders of the device itself. The protective case, especially if brightly coloured, creates an additional detaching frame. Thus, an app is always more detached from the user than a book. It is not necessarily a disadvantage, but should be noted. Picturebooks employ a variation of page layout to emphasise the rhythm of the narrative, including the alternation of framed and non-framed images, multiple panels, bleeding, and wordless spreads. All these elements are significant for the aesthetic experience of the narrative. The uniformity of the screen in a digital narrative sets limitations to this experience. In contrast, the digital medium affords a whole new range of spatio-temporal features, which are, however, not always utilised. Clever apps compensate for the loss of printed book layout by exploiting the affordance of the medium to convey spatiality, movement and rhythm.
Let us, for instance, consider how The Heart and the Bottle app utilises the possibilities of the medium in screen layout, attempting to go beyond the statement that the user's experience is different (cf. Schons 2011). The book is square, while the app uses the available format of the screen. The book spreads are highly imaginative with the layout, alternating between full surface, sequence of panels, interplay between verso and recto, prominent pageturners and implicit movement. The app replicates the images as such, adding various levels of interactivity, with a “hint” button to indicate required actions. It is possible to navigate to the next screen without performing the actions or discovering the hidden images, but a substantial part of the narrative will be lost.
The app fully realises the spatio-temporal potential of the medium. The panels of the book are divided between screens, which creates the sense of duration. Broader panoramic views become visible by moving the screen back and forth, which imitates additional eye movement on the printed spread. This feature produces a particularly poignant effect on the screen with the empty chair. If we touch or try to swipe, dark green colour invades the screen, our tactile experience resulting in the bleak mood of the image. The girl's body posture changes automatically, from standing to kneeling. The time it takes to finger-paint the screen signifies the flow of time while the girl contemplates the empty chair.
In some screens where the movement is automatic, for instance, the girl running from left to right into the next screen, inviting swipe: a screenswiper, corresponding to a pageturner. These actions are not supported by words and thus fill in a narrative ellipsis. Some actions, while provoking movement, do not add to the narrative, such as sending the man's hat flying and landing on the girl's head, switching on the lighthouse, making the stars sing, and later in the story assist the character in her attempts to smash the bottle. Turrión (2014) would perhaps call such participation false, although it certainly encourages users to engage with the visual narrative, which otherwise would remain static. The game-like element where the user is invited to draw is not a simple distraction, since the drawing appears on the wall in the next screen. The user shares embodied engagement with the girl, stimulating empathy.
Yet on most screens, tapping truly makes users co-creators; Turrión's real participation. To make the “curiosities of the world,” mentioned by the verbal text, appear, the user must tap on the image of the book that the grandfather is holding. On the prompt: “Feeling unsure, the girl thought the best thing was to put her heart in a safe place,” unlike the book, that shows completed actions, the app needs the user's intervention to tap on the X-ray machine to hear the heart beat and move the black square over the girl's chest to reveal the heart. The text/image counterpoint only works through the user replicating the character's action, which effectively invites empathy. Similarly, the user's participation makes the girl grow up, which also causes the memory of the grandfather to disappear. The verbal prompt “forgot” is more subtle still: on touch, the images fade, and as the girl leaves the screen we are left with an empty white surface. The resolution of the narrative again demands actions: help the little girl to get the heart from the bottle, place the protagonist in the chair and bring back her memories. We propose to call such actions, resulting in movement of characters or objects, tracing, something that book readers may do as they explore the images.
The Heart and the Bottle app thus cleverly emulates the intricate layout of the book, further adapting it to the medium and adding interactive elements that, unlike the meaningless shaking and jumping of various static elements in many apps, contribute to the narrative and encourage the user not only to explore, but also to become a co-creator. Indeed, the flow of narrative only emerges if the user interacts with the images. The word/image counterpoint, essential in any picturebook, requires the user's action, providing them with stronger agency. Tapping, touching and tracing become embodied additions to reading and viewing that enhances the user's affective engagement. In summary, while it is pointless to argue whether printed or digital picturebooks are generally better, it is obvious that some apps are more successful than others in making the most of the medium, just as some printed picturebooks are more stimulating and artistically challenging than others.
The most radical additional feature of apps is performativity. With books we seldom theorise the distinction between readers reading on their own or being read to; we seldom consider the option of the book being read by different voices. Although audiobooks do add a performance aspect, it has not been emphasised in picturebook theory as a significant dimension of the reader's interaction with the narrative. In an app, performance is decisive. Typically apps have three performance modes, offering function buttons on the menu screen: Autoplay, Read to Me and Read Myself. In Autoplay and Read to Me, the text is narrated with words highlighted while they are read. The difference between Autoplay and Read to Me is that the latter demands swiping between screens while Autoplay moves from screen to screen automatically. Thus, the Read to Me mode allows the user more agency, to stay on the screen and explore it further by tapping on various objects, causing the words to appear on the screen, be read aloud, or both. These options enhance the educational value of apps. Some apps also include elements of games. In Miffy at School the autovoice prompts the user to trace Miffy's walk to school, build towers with blocks, paint a picture, or do math together with the protagonist. These actions interrupt the flow of the narrative, while the results of the actions do not affect it (Turrión's false participation).
The recorded narrator's voice is predetermined, which creates some worries. Some apps are narrated in childish voices; some have prominent accents, which can be irritating. In The Cat in the Hat and The Lorax, the narrator imitates the voices of the characters. More important, the imposed narrative voice can govern interpretation. For instance, the verbal text in the printed Goodnight Moon is recited by an ambiguous voice, either the little bunny's, the adult's in the rocking chair, or, less likely, an invisible narrator. The adult audiovoice in the Autoplay and Read to Me modes precludes the interpretation of the verbal text as belonging to the child/bunny. Some recent apps offer a recording mode where either a parent or the child can record their own voice and replay it afterwards. There are also apps with only Read to Me mode. In addition, some apps have gaming functions, such as a matching game in Dear Zoo. When a game is offered in the end, it does not interrupt the narrative, but if inserted inside the narrative it may become disruptive (cf. Stichnothe 2014). Occasionally a Read and Play mode combines the two activities. Some apps offer “Rewards,” that is, scores and tips for interactivity. Yet while these simple tasks, involving tapping on various objects to make them move, shake, fall or burst, may seem a game, they do exactly what we encourage children to do with picturebooks: look carefully and explore details. Most apps have music that can be switched off; the default is music on.
Although the Read Myself mode may seem closest to a book, in actual fact, as we have shown in our discussion of The Heart and the Bottle, it can offer a high degree of interactivity. In Cinderella, developed by Nosy Crow, a separate dialogue goes on in addition to the written narrative at the bottom of the screen. The speech balloons appear when you tap on the characters, each new tap producing a new speech. On each screen users may discover the robin who provides additional comments on the narrative. The main character prompts actions such as putting logs on the fire or collecting plates to the shelf. On the sisters’ orders users fetch various items of clothing as they prepare for the ball. On another screen, users need to find a pumpkin, three mice and a wheelbarrow to assist the fairy godmother, completing the tasks before moving on to the next screen, co-creating the story by these actions, slightly different on each re-reading. The narrative can still be enjoyed without interaction, but the dialogue and the various actions enhance the experience. Cinderella makes good use of the possibility of the medium. However, another app from the same publisher, Jack and the Beanstalk, has possibly passed the boundary of a picturebook and entered the realm of gaming, since the user's attention is focused more on accomplishing tasks than on the narrative itself. The question of exactly where the boundary goes is similar to the recurrent discussion about the boundary between books, particularly washable, squeezable or chewable baby books, and toys.
SO WHAT IS NEW?
We have identified a number of concepts and terms that can be useful in discussing digital picturebook aesthetics, some of which, but far from all, have been already used in digital picturebook research mentioned above. We have attempted to cover all essential formal aspects, derived from picturebook theory. Some elements found in printed picturebooks do not translate into their digital equivalents, others acquire a different meaning, while some, that have become prevalent in apps, have so far been irrelevant in picturebook scholarship. Two app-specific aspects are of particular interest. Firstly, clever apps utilise movement, including zoom, to emphasise the spatio-temporal aspect of the narrative. More important, increasing degree of interactivity leads into imaginative co-creation rather than merely making things jump, squeak, or shake on a screen.
Brown Margaret Wise, and Clement Hurd. Goodnight, Moon. London: HarperCollins, 2011.
Campbell, Rod. Dear Zoo. London: Macmillan, 2010.
Jeffers, Oliver. The Heart and the Bottle. London: HarperCollins, 2010.
The Cat in the Hat, by Dr. Seuss. Version 2.6.1. © 2014. Oceanhouse Media.
Cinderella. Version 1.1.6. © 2013. Nosy Crow.
Dear Zoo by Rod Campbell. Version 1.0. © 2011. Pan Macmillan.
Elmer's Special Day, by David McKee. Version 2.5 © 2013. Oceanhouse Media.
The Fantastic Flying Books of Mr. Morris Lessmore, by William Joyce Version 1.4. © 2012. Moonbot Studios.
Goodnight, Moon, by Margaret Wise Brown & Clement Hurd. Version 1.4. © 2013. Loud Crow Interactive.
The Heart and the Bottle, by Oliver Jeffers. Version 1.0. © 2011. Penguin.
Jack and the Beanstalk. Version 1.0.1. © 2014. Nosy Crow.
The Lorax, by Dr. Seuss. Version 2.6.1. © 2014. Oceanhouse Media.
Miffy at School, by Dick Bruna. Version 2.0. © 2013. Sanoma Media Netherlands.