Editorial – Seeing Into Things: Eye Tracking the Moving Image – Sean Redmond & Craig Batty

Seeing into Things

We chose Seeing into Things: Eye Tracking the Moving Image as the title of this special edition to foreground the importance of reaching beyond – and beneath – the surface of the screen and the worlds that it creates and envisions. Through the empirical data that eye tracking affords us we are able to evidence and account for the depth in perception and sensibility that accompanies or anchors viewing. Seeing into Things is also recognition of the layers – or epidermi – of technological vision: depth cues, focal length, camera movement, and the delicious qualities of mise-en-scene all invite or demand that the image is looked into. There is much to observe across the textures and texturality of the screen. Eye tracking technology sees into the eyes of the viewer who peers – pierces – into the immersive world of the screen, factual or fictional. We see beauty in this alignment between the eye tracker, the viewer, and the screen. As this special edition finds, Seeing into Things is an enriching and intoxicating way of (re)discovering the complexities of viewing the moving image.

The poetry in and of seeing is not simply experiential, but connected to neurological, anatomical and cognitive processes. It is also connected to culture, discourse and ideology, where seeing into things is always gendered, classed and raced, amongst other encultured practices and modes of being in the world. Seeing into Things enables us to see into ourselves and into the complex and sometimes messy relationships between biology and culture, the human and technology, and between eye, brain, body and ear.

This last carnal conjunction is essential to the work being undertaken in this special edition because Seeing into Things is also meant to critically draw attention to the ocularcentric way through which the world is presently imagined to be experienced. Our position here is not to support this insightful supremacy, but rather to offer challenges and counter-points to it. When we see into things in this special edition it is with the need to recognise the centrality of hearing to seeing; of touching to viewing; and of the incorporation of the full human sensorium as it is taken up and in, and extends itself towards, the screen worlds that move and affect it. As Vivian Sobchack (2000) observes:

As “lived bodies” (to use a phenomenological term that insists on “the” objective body as always also lived subjectively as “my” body, diacritically invested and active in making sense and meaning in and of the world), our vision is always already “fleshed out”–and even at the movies it is “in-formed” and given meaning by our other sensory means of access to the world: our capacity not only to hear, but also to touch, to smell, to taste, and always to proprioceptively feel our dimension and movement in the world. In sum, the film experience is meaningful not to the side of my body, but because of my body.

The eye, brain, body and ear conjunction is also recognition that in order to understand viewing processes, one needs to incorporate different academic disciplines and approaches; from the vision sciences, neuroscience and linguistics; from ethnography and anthropology; and from the arteries and veins of creative practice, to the orbital concerns of the phenomenological. To do our work properly, then, Seeing into Things requires the eyes, brains, bodies and ears of scientists, anatomists, anthropologists, musicologists, filmmakers, screenwriters, and screen theorists, amongst others. It is this exciting arts-science nexus that this special edition draws uniquely from and is built upon, offering a foundational intervention into the way one makes critical and creative sense of viewers’ engagement with the moving image.

But what are the origins of this interdisciplinary approach? From where did the impetus for Seeing into Things come? Let us now return to the origins of the formation of the research group that drives many of the articles in this edition. Let us set the cinematic mood for some groundbreaking eye tracking research.

In the Mood for Eye Tracking Research

After a screening of the film In the Mood for Love (Kar-wai, 2000), Sean Redmond, a film and television scholar, mentioned to neuroscientist and anatomist Jodi Sita, how its rich colour scheme, expressionistic lighting and meandering narrative had fascinated and affected him. He suggested that he was sure his eyes were focusing on these visual elements, as they were being foregrounded, but also that they ‘wandered’ about the screen, choosing to look at motifs, characters and textures at their own volition, and where ‘mood’ took them. Sean contended that his eyes, or the way he viewed film, were both under the command of the film’s narrative and aesthetics, but were also free to discover the opulent fictive world for themselves. He suggested that viewing is an embodied experience.

Jodi responded quite directly: how do you know this? What evidence do you have? She continued that perception and comprehension are cognitive processes, and that what the eyes attend to in any viewing context can be measured objectively and understood through eye tracking, as well as other physiological technologies such as the measurement of pupil dilation. In that moment an arts-science debate was ignited, and an idea for an empirically driven eye tracking the moving image research group was born. They were now in the mood for some landmark empirical research of the moving image.

Jodi and Sean set up the Melbourne-based Eye Tracking and the Moving Image Research group at the end of 2012. They had two central goals in bringing the group together: one, they wanted to utilise eye tracking technology more centrally in the analysis and examination of the moving image; and two, they wanted to draw together scholars and practitioners from the Sciences, and the (Creative) Arts and Humanities so that different modes of enquiry, and theoretical and methodological apparatus, were placed in the same analytical arena (see Jodi’s account of the group’s formation in this edition). It was felt that having a room full of filmmakers, artists, film and cultural theorists, screenwriters, visual ethnographers, vision scientists and neuroscientists would generate new and exciting conversations and deliberations about how viewers engage with the moving image. To employ a games analogy, Jodi and Sean felt it was as if we had we all pinned our tails to different parts of the donkey, but that through opening our eyes together, we would all finally get to see and comprehend its full and glorious anatomy.

Their desire was to build upon existing research that drew disparate disciplines together, extending the type of work being conducted in arts-science research centres such as the Department of Psychology, Neuroscience and Behaviour’s NeuroArts Lab at McMaster University, Hamilton, Ontario. The formation of the group created a strong commitment to inter-disciplinary and cross-institutional relationships, and to what was considered a necessary dialogue between different disciplines united by a shared desire: to investigate vision regimes in relation to the affecting power and beauty of the moving image.

The utilisation of eye tracking technology was thus not born out of a technological determinism, but as a tool to bridge and fuse different approaches and methodologies in order that new findings, new knowledge, and new ways of understanding seeing and sensing images could emerge. This approach drew upon work by scholars who had already ‘crossed the line’, so-to-speak, including the work of Uri Hasson, Ohad Landesman, Barbara Knappmeyer, Ignacio Vallines, Nava Rubin, and David J. Heeger; who had already introduced to the field the idea of neurocinematics, the neuroscience of film, and the ‘inter-subject correlation analysis (ISC) … used to assess similarities in the spatiotemporal responses across viewers’ brains during movie watching’ (2008: 1). But you may ask: what is eye tracking?

But what is Eye Tracking?

Eye tracking enables us to empirically measure what viewers look at when watching screen-based media. The technology allows us to gather data from all platforms, interfaces and portals through which the moving image is distributed and consumed, including the television set, the cinema screen, the computer, and mobile devices such as smartphones and tablets. It also enables us to enter different types of environment to record viewing patterns, including the home and public spaces, such as the mall or the commute to work.  Analysis of viewers’ engagement with the moving image includes assessing where they look; interpreting why and how they look within determined visual fields or Areas of Interest (AOIs); and exploring what they feel or experience when they look. One can employ eye trackers to analyse viewer engagement with elements such as narrative, cinematography, editing, aesthetics, sound design and score, and characterization – elements that feature in many of the articles in this special edition. To do so, however, requires not only recognition and understanding of the languages employed in telling moving image stories, but also engagement with the science of the eye and the physiological and cardio-vascular transformations that take place when screen content is being viewed. To this end, a range of supportive investigative and methodological tools is also often employed, including the measurement of pupil dilation and the monitoring of heart and breathing rates.

Eye trackers work by shining infrared light onto the eye, which is then reflected back and captured by a sensor. The way we view images involves rapid eye movements that alter between points of fixation, in which the visual system gathers information and quickly moves between fixations called saccades. The sensor allows these eye movements (fixations) to be tracked, and specialist software then visualises these movements in the form of heat maps, swarms and gaze plot graphs. Statistical data can be extracted from these visualisations, and an interpretative framework can also be employed. For example, heat maps show effectively the weighting of all the viewing that occurred in a given scene, and gaze plots show the location of the fixations as well as the sequence in which they were made. To draw conclusions from this data, an area of interest analysis can be performed in which the number of times viewers visited specific objects or areas can be computed. By then analysing the amount of processing time spent in these areas, researchers are able to consider things such as the number of return visits made, building a picture of what was concentrated on. As can be seen from the articles in this special edition, analysis of this data draws us into open, and sometimes competing, exchanges about what has been discovered and why.

Double Dialogue

The articles in this edition are engaged in what we would like to define as a double dialogue. Each of the articles stands in their own right as discrete research, and yet they are also engaged in reflective and reflexive commentary. This dialoguing happens both within articles (see, for example, Redmond et al.) and also across articles (see, for example, Dyer and Pink, who draw upon the work of Batty et al.), to explore the possibilities and limitations of eye tracking research. The conversations that emerge enable the arts-science nexus to gather its power, since the different approaches to the text and their findings are foregrounded, drawn into syncretic union, or else are openly contested (see, for example, Brown and Smith’s engagement with each other’s work).

One can read this special edition, then, as the literal embodiment of the grounded work that takes place in a shared, respectful and mutually supportive interdisciplinary working environment. The virtues of the double dialogue approach to a special edition such as this are many, but most importantly one can see the value of the research on its own terms, and see how it has grown out of a dynamic research environment. We are able to witness directly how contributors have worked with and for each other, and how they are able to accommodate and enrich each other’s understandings of the texts under investigation. By seeing into things in this way, powerful research stories emerge.

The Stories of Seeing into Things

We have chosen to present the articles in this edition in a way that tells a research story, where conversations emerge and narrative arcs progress within and across the work presented. We have ordered them in a way that creates a narrative pattern; one can see ideas and themes introduced in one article picked up and developed in another. The story is also one that moves across screen media, from film to television and from features to serials. The special edition opens with a master shot of the field and closes with a tying up of the narrative threads that have been presented throughout the special edition. That is not to say, as previously noted, that each article does not stand as discrete research, but to recognise the beautiful truth of bringing overlapping and communicative research stories together like this.

The stories of Seeing into Things are also about the research environment that has been cultivated through the work of the Eye Tracking and the Moving Image Research group, and in the process of putting this special edition together. New international research relationships have been fashioned; and new friends have been made. We find in the inter-disciplinary stories of this special edition a range of content, styles and approaches in a deliberate attempt to engage readers (other researchers and practitioners) in recognising the power of crossing the research line.

Adrian G. Dyer (a vision scientist) and Sarah Pink (a visual anthropologist and ethnographer) open the edition with a critical, holistic overview of eye tracking research in relation to the screen. In Movement, Attention and Movies: the Possibilities and Limitations of Eye Tracking? Dyer and Pink suggest that film narrative and the conditions of viewing have a significant influence on gaze relations and subjectivity, but that there is yet limited work on the complexities and variables of such connections and alignments. Drawing upon their own research fields in vision science and anthropology and ethnography, Dyer and Pink demonstrate the value and importance of inter-disciplinary scholarship to understanding the poetics and politics of viewing the moving image. To make their observations they draw upon research carried out by Craig Batty, Claire Perkins and Jodi Sita, whose article naturally follows in this edition.

In How We Came To Eye Tracking Animation: A Cross-Disciplinary Approach to Researching the Moving Image, Batty, Perkins and Sita draw upon their pilot study of eye tracking a time-lapse montage sequence from the film Up (a study that also included Dyer). In their article, they outline how the ‘research journey’ of their project took shape, and they discuss how each of them came to the study from their individual disciplines: screenwriting, screen studies and neuroscience. They suggest that their own discipline backgrounds initially influenced and shaped both their research methodology and also the analysis of the research findings. However, they then point towards the layering of these approaches, as a way to fully discover how the montage scene under analysis can be best understood. This inter-disciplinary approach is fully taken up in the next article.

In Sound and Sight: An Exploratory Look at Saving Private Ryan through the Eye Tracking Lens, Jenny Robinson, Jane Stadler and Andrea Rassell place emphasis on the connection between looking and hearing; or, seeing and sounding. By focusing on sonic aesthetics that, arguably, direct viewer attention as much as any other film aesthetic, they use a sound-on, sound-off methodology to test their hypothesis. The resulting discussion will be as useful to film practitioners as it is to screen and eye tracking scholars.

The question of utility, or practice, is taken up in Jan Louis Kruger, Agnieszka Szarkowska and Izabela Krejtz’s, Subtitles on the Moving Image: An Overview of Eye Tracking Studies. They look towards new cognitive research horizons in the field of audiovisual translation (AVT). Seeing limitations and weaknesses in the current eye tracking research being conducted on subtitling, they argue that attention needs to be directed to the actual processing of verbal information. Drawing upon data gathered from numerous eye tracking studies, they contend that it demonstrates the way shot changes, language and subtitles impact upon cognitive processes, and how this has implications for subtitling and captioning.

Drawing on her doctoral research, Tessa Dwyer also explores subtitling but in relation to the BBC television series Sherlock. Dwyer’s fascinating article focuses upon its use of post-production (though scripted) free-floating text.  In From Subtitles to SMS: Eye-Tracking, Texting and Sherlock, Dwyer offers an in-depth analysis of viewer engagement with the show, exploring notions of reading vs. viewing, and attraction vs. distraction. Dwyer also draws upon ideas raised by Sean Redmond, Jodi Sita and Kim Vincs in their article, Our Sherlockian Eyes: the Surveillance of Vision.

In this article, Redmond, Sita and Vincs offer us a unique interior dialogue as they each read the eye tracking data gathered through their own discipline filters while also dialoging with each other’s approaches. Each author sees the hands of direction, misdirection, movement, surveillance and relationality in the scene under analysis, with agreement that vision is never simply cognitive or anatomical but multi-modal and haptic. They employ eye tracking data in ways that recognise the phenomenological embedded in the viewing experience, and which can be ‘extracted’ from what are normally seen or interpreted as qualitative findings.

The final two articles in this special edition then engage in a different type of dialogue or debate. In William Brown’s Politicizing Eye-tracking Studies of Film, he draws upon the (short) history of eye tracking and the moving image research, and specifically the work of Tim J. Smith, to demonstrate its theoretical and applied limitations. While Brown sees great value in eye tracking research he draws our attention to its obviousness in terms of telling us what we may already know. Nonetheless, Brown also outlines where the research may or should go and supplies instructive illustrations to help us chart new courses and terrains.

In what is also a critical commentary on the articles contained in this special edition, Tim J. Smith responds to Brown’s article, pointing to what he sees are misconceptions. In Read, Watch, Listen: A Commentary on Eye Tracking and Moving Images, Smith reflects on his own ground-breaking work as he also summarises and problematizes the articles in this edition. Working from a position as a cognitive psychologist and from within a version of neo-formalist film criticism, Smith’s position on eye tracking is persuasive if caroled.

Combined, the articles in this special edition reflect on the past, present and future of eye tracking and the moving image research and include critiques of the very nature of research itself. The case study material that the articles draw from is predominately from mainstream film and television texts but these are explored through new vectors. Unlike Smith’s work, the authors extend their utilization of eye tracking data to consider the cultural, the ethnographic and anthropological, the ideological and the phenomenological, albeit within the house of film and television aesthetics and genre. We hope that other researchers will draw inspiration and insight from the studies undertaken in this edition.

Future Research Directions

As indicated, the Eye Tracking and the Moving Image Research group features practitioner-academics who are interested in how research can be both carried out and disseminated through creative practice. As two of the articles in this edition signal, there is interest and expertise in sound design and scoring, and in screenwriting. Both of these aspects, which speak to the broader gamut of film and television-making practices, have found natural positions within the research undertaken to date, and also feature in two forthcoming book chapters authored by the group’s members. What is of special interest to us in the near future is how we might use these practices to further develop research methods and research outputs. For example, rather than relying on pre-existing moving image texts, what if we were to make our own? How might we use specific practices – sound, screenwriting – in order to influence the eye tracking experiments that we conduct?

One idea is for the group to make one or more short films in order to test patterns of viewer engagement, where narrative and aesthetics are controlled by the researchers, thus becoming a creative practice research variable. Another idea is to analyse eye tracking data alongside aspects such as the score and the screenplay, in order to make original connections between the source text – intentionality – and its reception. We should also consider how the scientific data provided by the research – heat maps, gaze plots, etc. – might be used as the basis of a creative work in and of itself, such as an artwork or another moving image text. Andrea Rassell, Sean Redmond, Jodi Sita and Darrin Verhagen are currently engaged in public projection and installation projects that use the colours, spirals and vortexes of eye tracking data within thematised artworks.

The make-up of the group has also resulted in some interest in how viewers engage spatially and environmentally with moving image texts, posing questions such as: does the viewing environment alter gaze patterns? How might room set-up and screen size change where and for how long people look at a defined area of interest? In this way, the group might seek to add ethnographic methods to the studies that take place, allowing us to add another set of research variables that could produce interesting and original results. Depending on the context, this type of research would also be of use to the screen industry – distributors, cinema groups, screen manufacturers, interior designers, etc.

We are mindful, nonetheless, where others might take eye tracking research. It is already being used in the commercial moving making industry and one of the worries is that it will become a device to reduce production costs as filmmakers use the data to literally paint the screen by numbers. Film and television are artforms, they beautify the world and they enrich our lives. All members of the eye tracking and moving image research group want to employ eye tracking technology to get to know and understand this beauty, and to fully comprehend what the viewer sees, hears and feels when they watch Fred Astaire dance, Ryan Gosling seduce, or Sherlock deduce and detect.

References

Hasson, Uri, Landesman, Ohad, Knappmeyer, Barbara, Vallines, Ignacio, Rubin, Nava, and Heeger, David J. 2008. Neurocinematics: The neuroscience of film. Projections, 2(1), 1-26

Sobchack, Vivian.2000. What My Fingers Knew: The Cinesthetic Subject, or Vision in the Flesh, Senses of Cinema, Issue 5, available at: http://sensesofcinema.com/2000/conference-special-effects-special-affects/fingers/ (accessed 19th January, 2015).