Mobile applications as semiotic technologies: a multimodal social semiotic approach to interactivity & picture book apps

Date: Monday 11th of May 2015, 5:00-6:30 pm

Venue: London Knowledge Lab, 23-29 Emerald Street, WC1N 3QS

Dr Sumin Zhao

Chancellor’s Post-doctoral Research Fellow, University of Technology, Sydney

Visiting Research Associate, UCL Institute of Education, London

In this talk, I explore the “myth” of interactivity surrounding mobile applications, using picture book apps as a case.  I shall start by problematizing the ways in which interactivity has been discussed in various literature (e.g. Chiong, 2012; Friedlander, 2013; Hutchison, 2012; Moms with Apps, 2012; Verenikina, 2010; Simpson, et al, 2013). I will then propose a multimodal social semiotic (Halliday, 1978; Kress, 2010; Van Leeuwen, 2005) framework for exploring interactivity in touch-based environment, which involves the analysis of multiple sets of multimodal relations. Here, I am particularly interested in exploring the potential of a Vygotskyian perspective in helping us to unpack the relation between gesture and meaning making in touch-based semiotic technologies (Zhao, et al 2014; Van Leeuwen, & Djonov, 2013). Next, to illustrate the potential of this approach for literacy education, I will focus on a detailed contrastive analysis (Zhao & Unsworth, in progress) of the book and the app version of the Heart and the Bottle (Oliver Jeffers 2009, 2010). In the final part of the talk, I will bring the three threads together and discuss the ways in which a single semiotic artefact can be analyzed across a series of interconnected semiotic practices. While the talk is based largely on the analysis from the first stage of my fellowship project, I would like to leave the conclusion open. In doing so, I hope to invite reflection and debate on the nature of multimodal semiotic research, in particular, the relation between description and interpretation.

The Visual and Multimodal Research Forum is a student run platform for academic discussion on multimodality and open to all postgraduate research students and other researchers at UCL Institute of Education and elsewhere. Participation is free. Please contact Sophia Diamantopoulou, Forum Coordinator, at sdiamantopoulou@ioe.ac.uk for any queries.