• Refractory: a Journal of Entertainment Media
    • Past Issues
  • About
  • Editorial Board
  • Submissions
  • Contact

Category Archives: Television

Volume 26

Contents

  1. “Children should play with dead things”: transforming Frankenstein in Tim Burton’s Frankenweenie –  Erin Hawley
  2. “You gave me no choice”: A queer reading of Mordred’s journey to villainy and struggle for identity in BBC’s Merlin  –  Joseph Brennan
  3. Days of YouTube-ing Days of Heaven: Participatory Culture and the Fan Trailer  –  Kyle R. McDaniel
  4. When a Good Girl Goes to War: Claire Adams Mackinnon and Her Service During World War I – Heather L. Robinson 
  5. ‘Rock‘n’roll’s evil doll’: the Female Popular Music Genre of Barbie Rock  –  Rock Chugg
  6. Morality, Mortality and Materialism: an Art Historian Watches Mad Men – Catherine Wilkins
  7. Playing At Work  –  Samuel Tobin
  8. 1970s Disaster Films: The Star In Jeopardy – Nathan Smith

 

 

Read, Watch, Listen: A commentary on eye tracking and moving images – Tim J. Smith

Abstract

Eye tracking is a research tool that has great potential for advancing our understanding of how we watch movies. Questions such as how differences in the movie influences where we look and how individual differences between viewers alters what we see can be operationalised and empirically tested using a variety of eye tracking measures. This special issue collects together an inspiring interdisciplinary range of opinions on what eye tracking can (and cannot) bring to film and television studies and practice. In this article I will reflect on each of these contributions with specific focus on three aspects: how subtitling and digital effects can reinvigorate visual attention, how audio can guide and alter our visual experience of film, and how methodological, theoretical and statistical considerations are paramount when trying to derive conclusions from eye tracking data.

 

Introduction

I have been obsessed with how people watch movies since I was a child. All you have to do is turn and look at an audience member’s face at the movies or at home in front of the TV to see the power the medium holds over them. We sit enraptured, transfixed and immersed in the sensory patterns of light and sound projected back at us from the screen. As our physical activity diminishes our mental activity takes over. We piece together minimal audiovisual cues to perceive rich otherworldly spaces, believable characters and complex narratives that engage us mentally and move us emotionally. As I progressed through my education in Cognitive Science and Psychology I was struck by how little science understood about cinema and the mechanisms filmmakers used to create this powerful experience.[i] Reading the film literature, listening to filmmakers discuss their craft and excavating gems of their craft knowledge I started to realise that film was a medium ripe for psychological investigation. The empirical study of film would further our understanding of how films work and how we experience them but it would also serve as a test bed for investigating complex aspects of real-world cognition that were often considered beyond the realms of experimentation. As I (Smith, Levin & Cutting, 2010) and others (Anderson, 2006) have argued elsewhere, film evolved to “piggy back” normal cognitive development and use basic cognitive tendencies such as attentional preferences, theory of mind, empathy and narrative structuring of memory to make the perception of film as enjoyable and effortless as possible. By investigating film cognition we can, in turn advance our understanding of general cognition. But to do so we need to step outside of traditional disciplinary boundaries concerning the study of film and approach the topic from an interdisciplinary perspective. This special issue represents a highly commendable attempt to do just that.

By bringing together psychologists, film theorists, philosophers, vision scientists, neuroscientists and screenwriters this special issue (and the Melbourne research group that most contributors belong to) provides a unique perspective on film viewing. The authors included in this special issue share my passion for understanding the relationship between viewers and film but this interest manifests in very different ways depending on their perspectives (see Redmond, Sita, and Vincs, this issue; for a similar personal journey into eye tracking as that presented above). By focussing on viewer eye movements the articles in this special issue provide readers from a range of disciplines a way into the eye tracking investigation of film viewing. Eye tracking (as comprehensively introduced and discussed by Dyer and Pink, this issue) is a powerful tool for quantifying a viewer’s experience of a film, comparing viewing behaviour across different viewing conditions and groups as well as testing hypotheses about how certain cinematic techniques impact where we look. But, as is rightly highlighted by several of the authors in this special issue eye tracking is not a panacea for all questions about film spectatorship.

Like all experimental techniques it can only measure a limited range of psychological states and behaviours and the data it produces does not say anything in and of itself. Data requires interpretation. Interpretation can take many forms[ii] but if conclusions are to be drawn about how the data relates to psychological states of the viewer this interpretation must be based on theories of psychology and ideally confirmed using secondary/supporting measures. For example, the affective experience of a movie is a critical aspect which cognitive approaches to film are often wrongly accused of ignoring. Although, cognitive approaches to film often focus on how we comprehend narratives (Magliano and Zacks, 2011), attend to the image (Smith, 2013) or follow formal patterns within a film (Cutting, DeLong and Nothelfer, 2010) several cognitivists have focussed in depth on emotional aspects (see the work of Carl Plantinga, Torben Grodal or Murray Smith). Eye tracking is the perfect tool for investigating the impact of immediate audiovisual information on visual attention but it is less suitable for measuring viewer affect. Psychophysiological measures such as heart rate and skin conductance, neuroimaging methods such as fMRI or EEG, or even self-report ratings may be better for capturing a viewer’s emotional responses to a film as has been demonstrated by several research teams (Suckfull, 2000; Raz et al, 2014). Unless the emotional state of the viewer changed where they looked or how quickly they moved their eyes the eye tracker may not detect any differences between two viewers with different emotional states.[iii]

As such, a researcher interested in studying the emotional impact of a film should either choose a different measurement technique or combine eye tracking with another more suitable technique (Dyer and Pink, this issue). This does not mean that eye tracking is unsuitable for studying the cinematic experience. It simply means that you should always choose the right tool for the job and often this means combining multiple tools that are strong in different ways. As Murray Smith (the current President of the Society for the Cognitive Study of the Moving Images; SCSMI) has argued, a fully rounded investigation of the cinematic experience requires “triangulation” through the combination of multiple perspectives including psychological, neuroscientific and phenomenological/philosophical theory and methods (Smith, 2011) – an approach taken proudly across this special issue.

For the remainder of my commentary I would like to focus on certain themes that struck me as most personally relevant and interesting when reading the other articles in this special issue. This is by no means an exhaustive list of the themes raised by the other articles or even an assessment of the importance of the particular themes I chose to select. There are many other interesting observations made in the articles I do not focus on below but given my perspective as a cognitive scientist and current interests I decided to focus my commentary on these specific themes rather than make a comprehensive review of the special issues or tackle topics I am unqualified to comment on. Also, I wanted to take the opportunity to dispel some common misconceptions about eye tracking (see the section ‘Listening to the data’) and empirical methods in general.

Reading an image

One area of film cognition that has received considerable empirical investigation is subtitling. As Kruger, Szarkowska and Krejtz (this issue) so comprehensively review, they and I believe eye tracking is the perfect tool for investigating how we watch subtitled films. The presentation of subtitles divides the film viewing experience into a dual- task: reading and watching. Given that the media was originally designed to communicate critical information through two channels, the image and soundtrack introducing text as a third channel of communication places extra demands on the viewer’s visual system. However, for most competent readers serially shifting attention between these two tasks does not lead to difficulties in comprehension (Kruger, Szarkowska and Krejtz, this issue). Immediately following the presentation of the subtitles gaze will shift to the beginning of the text, saccade across the text and return to the centre of interest within a couple of seconds. Gaze heatmaps comparing the same scenes with and without subtitles (Kruger, Szarkowska and Krejtz, this issue; Fig. 3) show that the areas of the image fixated are very similar (ignoring the area of the screen occupied by the subtitles themselves) and rather than distracting from the visual content the presence of subtitles seems to actually condense the gaze behaviour on the areas of central interest in an image, e.g. faces and the centre of the image. This illustrates the redundancy of a lot of the visual information presented in films and the fact that under non-subtitle conditions viewers rarely explore the periphery of the image (Smith, 2013).

My colleague Anna Vilaró and I recently demonstrated this similarity in an eye tracking study in which the gaze behaviour of viewers was compared across versions of an animated film, Disney’s Bolt (Howard & Williams, 2008) either in the original English audio condition, a Spanish language version with English subtitles, an English language version with Spanish subtitles and a Spanish language version without subtitles (Vilaró, & Smith, 2011). Given that our participants were English speakers who did not know Spanish these conditions allowed us to investigate both where they looked under the different audio and subtitle conditions but also what they comprehended. Using cued recall tests of memory for verbal and visual content we found no significant differences in recall for either types of content across the viewing conditions except for verbal recall in the Spanish-only condition (not surprisingly given that our English participants couldn’t understand the Spanish dialogue). Analysis of the gaze behaviour showed clear evidence of subtitle reading, even in the Spanish subtitle condition (see Figure 1) but no differences in the degree to which peripheral objects were explored. This indicates that even when participants are watching film sequences without subtitles and know that their memory will be tested for the visual content their gaze still remains focussed on central features of a traditionally composed film. This supports arguments for subtitling movies over dubbing as, whilst placing greater demands on viewer gaze and a heightened cognitive load there is no evidence that subtitling leads to poorer comprehension.

Figure 1: Figure from Vilaró & Smith (2011) showing the gaze behaviour of multiple viewers directed to own language subtitles (A) and foreign language/uninterpretable subtitles (B).

Figure 1: Figure from Vilaró & Smith (2011) showing the gaze behaviour of multiple viewers directed to own language subtitles (A) and foreign language/uninterpretable subtitles (B).

The high degree of attentional synchrony (Smith and Mital, 2013) observed in the above experiment and during most film sequences indicates that all visual features in the image and areas of semantic significance (e.g. social information and objects relevant to the narrative) tend to point to the same part of the image (Mital, Smith, Hill and Henderson, 2011). Only when areas of the image are placed in conflict through image composition (e.g. depth of field, lighting, colour or motion contrast) or staging (e.g. multiple actors) does attentional synchrony break down and viewer gaze divide between multiple locations. Such shots are relatively rare in mainstream Hollywood cinema or TV (Salt, 2009; Smith, 2013) and when used the depicted action tends to be highly choreographed so attention shifts between the multiple centres of image in a predictable fashion (Smith, 2012). If such choreographing of action is not used the viewer can quickly exhaust the information in the image and start craving either new action or a cut to a new shot.

Hochberg and Brooks (1978) referred to this as the visual momentum of the image: the pace at which visual information is acquired. This momentum is directly observable in the saccadic behaviour during an images presentation with frequent short duration fixations at the beginning of a scene’s presentation interspersed by large amplitude saccades (known as the ambient phase of viewing; Velichovsky, Dornhoefer, Pannasch and Unema, 2000) and less frequent, longer duration fixations separated by smaller amplitude saccades as the presentation duration increases (known as the focal phase of viewing; Velichovsky et al., 2000). I have recently demonstrated the same pattern of fixations during viewing of dynamic scenes (Smith and Mital, 2013) and shown how this pattern gives rise to more central fixations at shot onset and greater exploration of the image and decreased attentional synchrony as the shot duration increases (Mital, Smith, Hill and Henderson, 2011). Interestingly, the introduction of subtitles to a movie may have the unintended consequence of sustaining visual momentum throughout a shot. The viewer is less likely to exhaust the information in the image because their eyes are busy saccading across the text to acquire the information that would otherwise be presented in parallel to the image via the soundtrack. This increased saccadic activity may increase the cognitive load experienced by viewers of subtitled films and change their affective experience, producing greater arousal and an increased sense of pace.

For some filmmakers and producers of dynamic visual media, increasing the visual momentum of an image sequence may be desirable as it maintains interest and attention on the screen (e.g. Michael Bay’s use of rapidly edited extreme Close-Ups and intense camera movements in the Transformer movies). In this modern age of multiple screens fighting for our attention when we are consuming moving images (e.g. mobile phones and computer screens in our living rooms and even, sadly increasingly at the cinema) if the designers of this media are to ensure that our visual attention is focussed on their screen over the other competing screens they need to design the visual display in a way that makes comprehension impossible without visual attention. Feature Films and Television dramas often rely heavily on dialogue for narrative communication and the information communicated through the image may be of secondary narrative importance to the dialogue so viewers can generally follow the story just by listening to the film rather than watching it. If producers of dynamic visual media are to draw visual attention back to the screen and away from secondary devices they need to increase the ratio of visual to verbal information. A simple way of accomplishing this is to present the critical audio information through subtitling. The more visually attentive mode of viewing afforded by watching subtitled film and TV may partly explain the growing interest in foreign TV series (at least in the UK) such as the popularity of Nordic Noir series such as The Bridge (2011) and The Killing (2007).

Another way of drawing attention back to the screen is to constantly “refresh” the visual content of the image by either increasing the editing rate or creatively using digital composition.[iv] The latter technique is wonderfully exploited by Sherlock (2010) as discussed brilliantly by Dwyer (this issue). Sherlock contemporised the detective techniques of Sherlock Holmes and John Watson by incorporating modern technologies such as the Internet and mobile phones and simultaneously updated the visual narrative techniques used to portray this information by using digital composition to playfully superimpose this information onto the photographic image. In a similar way to how the sudden appearance of traditional subtitles involuntarily captures visual attention and draws our eyes down to the start of the text, the digital inserts used in Sherlock overtly capture our eyes and encourage reading within the viewing of the image.

If Dwyer (this issue) had eyetracked viewers watching these excerpts she would have likely observed this interesting shifting between phases of reading and dynamic scene perception. Given that the appearance of the digital inserts produce sudden visual transients and are highly incongruous with the visual features of the background scene they are likely to involuntarily attract attention (Mital, Smith, Hill & Henderson, 2012). As such, they can be creatively used to reinvigorate the pace of viewing and strategically direct visual attention to parts of the image away from the screen centre. Traditionally, the same content may have been presented either verbally as narration, heavy handed dialogue exposition (e.g. “Oh my! I have just received a text message stating….”) or as a slow and laboured cut to close-up of the actual mobile phone so we can read it from the perspective of the character. Neither takes full advantage of the communicative potential of the whole screen space or our ability to rapidly attend to and comprehend visual information and audio information in parallel.

Such intermixing of text, digital inserts and filmed footage is common in advertisements, music videos, and documentaries (see Figure 2) but is still surprisingly rare in mainstream Western film and TV. Short-form audiovisual messages have recently experienced a massive increase in popularity due to the internet and direct streaming to smartphones and mobile devices. To maximise their communicative potential and increase their likelihood of being “shared” these videos use all audiovisual tricks available to them. Text, animations, digital effects, audio and classic filmed footage all mix together on the screen, packing every frame with as much info as possible (Figure 2), essentially maximising the visual momentum of each video and maintaining interest for as long as possible.[v] Such videos are so effective at grabbing attention and delivering satisfying/entertaining/informative experiences in a short period of time that they often compete directly with TV and film for our attention. Once we click play, the audiovisual bombardment ensures that our attention remains latched on to the second screen (i.e., the tablet or smartphone) for its duration and away from the primary screen, i.e., the TV set. Whilst distressing for producers of TV and Film who wish our experience of their material to be undistracted, the ease with which we pick up a handheld device and seek other stimulation in parallel to the primary experience may indicate that the primary material does not require our full attention for us to follow what is going on. As attention has a natural ebb-and-flow (Cutting, DeLong and Nothelfer, 2010) and “There is no such thing as voluntary attention sustained for more than a few seconds at a time” (p. 421; James, 1890) if modern producers of Film and TV want to maintain a high level of audience attention and ensure it is directed to the screen they must either rely on viewer self-discipline to inhibit distraction, reward attention to the screen with rich and nuanced visual information (as fans of “slow cinema” would argue of films like those of Bela Tarr) or utilise the full range of postproduction effects to keep visual interest high and maintained on the image, as Sherlock so masterfully demonstrates.

Figure 2: Gaze Heatmaps of participants’ free-viewing a trailer for Lego Indiana Jones computer game (left column) and the Video Republic documentary (right column). Notice how both make copious use of text within the image, as intertitles and as extra sources of information in the image (such as the head-up display in A3). Data and images were taken from the Dynamic Images and Eye Movement project (DIEM; Mital, Smith, Hill & Henderson, 2010). Videos can be found here (http://vimeo.com/6628451) and here (http://vimeo.com/2883321).

Figure 2: Gaze Heatmaps of participants’ free-viewing a trailer for Lego Indiana Jones computer game (left column) and the Video Republic documentary (right column). Notice how both make copious use of text within the image, as intertitles and as extra sources of information in the image (such as the head-up display in A3). Data and images were taken from the Dynamic Images and Eye Movement project (DIEM; Mital, Smith, Hill & Henderson, 2010). Videos can be found here (http://vimeo.com/6628451) and here (http://vimeo.com/2883321).

A number of modern filmmakers are beginning to experiment with the language of visual storytelling by questioning our assumptions of how we perceive moving images. Forefront in this movement are Ang Lee and Andy and Lana Wachowski. In Ang Lee’s Hulk (2003), Lee worked very closely with editor Tim Squyers to use non-linear digital editing and after effects to break apart the traditional frame and shot boundaries and create an approximation of a comic book style within film. This chaotic unpredictable style polarised viewers and was partly blamed for the film’s poor reception. However, it cannot be argued that this experiment was wholly unsuccessful. Several sequences within the film used multiple frames, split screens, and digital transformation of images to increase the amount of centres of interest on the screen and, as a consequence increase pace of viewing and the arousal experienced by viewers. In the sequence depicted below (Figure 3) two parallel scenes depicting Hulk’s escape from a containment chamber (A1) and this action being watched from a control room by General Ross (B1) were presented simultaneously by presenting elements of both scenes on the screen at the same time. Instead of using a point of view (POV) shot to show Ross looking off screen (known as the glance shot; Branigan, 1984) followed by a cut to what he was looking at (the object shot) both shots were combined into one image (F1 and F2) with the latter shot sliding into from behind Ross’ head (E2). These digital inserts float within the frame, often gliding behind objects or suddenly enlarging to fill the screen (A2-B2). Such visual activity and use of shots-within-shots makes viewer gaze highly active (notice how the gaze heatmap is rarely clustered in one place; Figure 3). Note that this method of embedding a POV object shot within a glance shot is similar to Sherlock’s method of displaying text messages as both the glance, i.e., Watson looking at his phone, and the object, i.e., the message, are shown in one image. Both uses take full advantage of our ability to rapidly switch from watching action to reading text without having to wait for a cut to give us the information.

Figure 3: Gaze heatmap of eight participants watching a series of shots and digital inserts from Hulk (Ang Lee, 2003). Full heatmap video is available at http://youtu.be/tErdurgN8Yg.

Figure 3: Gaze heatmap of eight participants watching a series of shots and digital inserts from Hulk (Ang Lee, 2003). Full heatmap video is available at http://youtu.be/tErdurgN8Yg.

Similar techniques have been used Andy and Lana Wachowski’s films including most audaciously in Speed Racer (2008). Interestingly, both sets of filmmakers seem to intuitively understand that packing an image with as much visual and textual information as possible can lead to viewer fatigue and so they limit such intense periods to only a few minutes and separate them with more traditionally composed sequences (typically shot/reverse-shot dialogue sequences). These filmmakers have also demonstrated similar respect for viewer attention and the difficulty in actively locating and encoding visual information in a complex visual composition in their more recent 3D movies. Ang Lee’s Life of Pi (2012) uses the visual volume created by stereoscopic presentation to its full potential. Characters inhabit layers within the volume as foreground and background objects fluidly slide around each other within this space. The lessons Lee and his editor Tim Squyers learned on Hulk (2003) clearly informed the decisions they made when tackling their first 3D film and allowed them to avoid some of the issues most 3D films experience such as eye strain, sudden unexpected shifts in depth and an inability to ensure viewers are attending to the part of the image easiest to fuse across the two eye images (Banks, Read, Allison & Watt, 2012).

Watching Audio

I now turn to another topic featured in this special issue, the influence of audio on gaze (Robinson, Stadler and Rassell, this issue). Film and TV are inherently multimodal. Both media have always existed as a combination of visual and audio information. Even early silent film was almost always presented with either live musical accompaniment or a narrator. As such, the relative lack of empirical investigation into how the combination of audio and visual input influences how we perceive movies and, specifically how we attend to them is surprising. Robinson, Stadler and Rassell (this issue) have attempted to address this omission by comparing eye movements for participants either watching the original version of the Omaha beach sequence from Steven Spielberg’s Saving Private Ryan (1998) or the same sequence with the sound removed. This film sequence is a great choice for investigating AV influences on viewer experience as the intensity of the action, the hand-held cinematography and the immersive soundscape all work together to create a disorientating embodied experience for the viewer. The authors could have approached this question by simply showing a set of participants the sequence with audio and qualitatively describing the gaze behaviour at interesting AV moments during the sequence. Such description of the data would have served as inspiration for further investigation but in itself can’t say anything about the causal contribution of audio to this behaviour as there would be nothing to compare the behaviour to. Thankfully, the authors avoided this problem by choosing to manipulate the audio.

In order to identify the causal contribution of any factor you need to design an experiment in which that factor (known as the Independent Variable) is either removed or manipulated and the significant impact of this manipulation on the behaviour of interest (known as the Dependent Variable) is tested using appropriate inferential statistics. I commend Robinson, Stadler and Rassell’s experimental design as they present such an manipulation and are therefore able to produce data that will allow them to test their hypotheses about the causal impact of audio on viewer gaze behaviour. Several other papers in this special issue (Redmond, Sita and Vincs; Batty, Perkins and Sita) discuss gaze data (typically in the form of scanpaths or heatmaps) from one viewing condition without quantifying its difference to another viewing condition. As such, they are only able to describe the gaze data, not use it to test hypotheses. There is always a temptation to attribute too much meaning to a gaze heatmap (I too am guilty of this; Smith, 2013) due to their seeming intuitive nature (i.e., they looked here and not there) but, as in all psychological measures they are only as good as the experimental design within which there are employed.[vi]

Qualitative interpretation of individual fixation locations, scanpaths or group heatmaps are useful for informing initial interpretation of which visual details are most likely to make it into later visual processing (e.g. perception, encoding and long term memory representations) but care has to be taken in falsely assuming that fixation equals awareness (Smith, Lamont and Henderson, 2012). Also, the visual form of gaze heatmaps vary widely depending on how many participants contribute to the heatmap, which parameters you choose to generate the heatmaps and which oculomotor measures the heatmap represent (Holmqvist, et al., 2011). For example, I have demonstrated that unlike during reading visual encoding during scene perception requires over 150ms during each fixation (Rayner, Smith, Malcolm and Henderson, 2009). This means that if fixations with durations less than 150ms are included in a heatmap it may suggest parts of the image have been processed which in actual fact were fixated too briefly to be processed adequately. Similarly, heatmaps representing fixation duration instead of just fixation location have been shown to be a better representation of visual processing (Henderson, 2003). Heatmaps have an immediate allure but care has to be taken about imposing too much meaning on them especially when the gaze and the image are changing over time (see Smith and Mital, 2013; and Sawahata et al, 2008 for further discussion). As eye tracking hardware becomes more available to researchers from across a range of disciplines we need to work harder to ensure that it is not used inappropriately and that the conclusions that are drawn from eye tracking data are theoretically and statistically motivated (see Rayner, 1998; and Holmqvist et al, 2013 for clear guidance on how to conduct sound eye tracking studies).

Given that Robinson, Stadler and Rassell (this issue) manipulated the critical factor, i.e., the presence of audio the question now is whether their study tells us anything new about the AV influences on gaze during film viewing. To examine the influence of audio they chose two traditional methods for expressing the gaze data: area of interest (AOI) analysis and dispersal. By using nine static (relative to the screen) AOIs they were able to quantify how much time the gaze spent in each AOI and utilise this measure to work out how distributed gaze was across all AOIs. Using these measures they reported a trend towards greater dispersal in the mute condition compared to the audio condition and a small number of significant differences in the amount of time spent in some regions across the audio conditions.

However, the conclusions we can draw from these findings are seriously hindered by the low sample size (only four participants were tested, meaning that any statistical test is unlikely to reveal significant differences) and the static AOIs that did not move with the image content. By locking the AOIs to static screen coordinates their AOI measures express the deviation of gaze relative to these coordinates, not to the image content. This approach can be informative for quantifying gaze exploration away from the screen centre (Mital, Smith, Hill and Henderson, 2011) but in order to draw conclusions about what was being fixated the gaze needs to be quantified relative to dynamic AOIs that track objects of interest on the screen (see Smith an Mital, 2013). For example, their question about whether we fixate a speaker’s mouth more in scenes where the clarity of the speech is difficult due to background noise (i.e., their “Indistinct Dialogue” scene) has previously been investigated in studies that have manipulated the presence of audio (Võ, Smith, Mital and Henderson, 2012) or the level of background noise (Buchan, Paré and Munhall, 2007) and measured gaze to dynamic mouth regions. As Robinson, Stadler and Rassell correctly predicted, lip reading increases as speech becomes less distinct or the listener’s linguistic competence in the spoken language decreases (see Võ et al, 2012 for review).

Similarly, by measuring gaze dispersal using a limited number of static AOIs they are losing considerable nuance in the gaze data and have to resort to qualitative description of unintuitive bar charts (figure 4). There exist several methods for quantifying gaze dispersal (see Smith and Mital, 2013, for review) and even open-source tools for calculating this measure and comparing dispersal across groups (Le Meur and Baccino, 2013). Some methods are as easy, if not easier to calculate than the static AOIs used in the present study. For example, the Euclidean distance between the screen centre and the x/y gaze coordinates at each frame of the movie provides a rough measure of how spread out the gaze is from the screen centre (typically the default viewing location; Mital et al, 2011) and a similar calculation can be performed between the gaze position of all participants within a viewing condition to get a measure of group dispersal.

Using such measures, Coutrot and colleagues (2012) showed that gaze dispersal is greater when you remove audio from dialogue film sequences and they have also observed shorter amplitude saccades and marginally shorter fixation durations. Although, I have recently shown that a non-dialogue sequence from Sergei Eisenstein’s Alexander Nevsky (1938) does not show significant differences in eye movement metrics when the accompanying music is removed (Smith, 2014). This difference in findings points towards interesting differences in the impact diegetic (within the depicted scene, e.g. dialogue) and non-diegetic (outside of the depicted scene, e.g. the musical score) may have on gaze guidance. It also highlights how some cinematic features may have a greater impact on other aspects of a viewer’s experience than those measureable by eye tracking such as physiological markers of arousal and emotional states. This is also the conclusion that Robinson, Stadler and Rassell come to.    

Listening to the Data (aka, What is Eye Tracking Good For?)

The methodological concerns I have raised in the previous section lead nicely to the article by William Brown, entitled There’s no I in Eye Tracking: How useful is Eye Tracking to Film Studies (this issue). I have known William Brown for several years through our attendance of the Society for Cognitive Studies of the Moving Image (SCSMI) annual conference and I have a deep respect for his philosophical approach to film and his ability to incorporate empirical findings from the cognitive neurosciences, including some references to my own work into his theories. Therefore, it comes somewhat as a surprise that his article openly attacks the application of eye tracking to film studies. However, I welcome Brown’s criticisms as it provides me with an opportunity to address some general assumptions about the scientific investigation of film and hopefully suggest future directions in which eye tracking research can avoid falling into some of the pitfalls Brown identifies.

Brown’s main criticisms of current eye tracking research are: 1) eye tracking studies neglect “marginal” viewers or marginal ways of watching movies; 2) studies so far have neglected “marginal” films; 3) they only provide “truisms”, i.e., already known facts; and 4) they have an implicit political agenda to argue that the only “true” way to study film is a scientific approach and the “best” way to make a film is to ensure homogeneity of viewer experience. I will address these criticisms in turn but before I do so I would like to state that a lot of Brown’s arguments could generally be recast as an argument against science in general and are built upon a misunderstanding of how scientific studies should be conducted and what they mean.

To respond to Brown’s first criticism that eye tracking “has up until now been limited somewhat by its emphasis on statistical significance – or, put simply, by its emphasis on telling us what most viewers look at when they watch films” (Brown, this issue; 1), I first have to subdivide the criticism into ‘the search for significance’ and ‘attentional synchrony’, i.e., how similar gaze is across viewers (Smith and Mital, 2013). Brown tells an anecdote about a Dutch film scholar who’s data had to be excluded from an eye tracking study because they did not look where the experimenter wanted them to look. I wholeheartedly agree with Brown that this sounds like a bad study as data should never be excluded for subjective reasons such as not supporting the hypothesis, i.e., looking as predicted. However, exclusion due to statistical reasons is valid if the research question being tested relates to how representative the behaviour of a small set of participants (known as the sample) are to the overall population. To explain when such a decision is valid and to respond to Brown’s criticism about only ‘searching for significance’ I will first need to provide a brief overview of how empirical eye tracking studies are designed and why significance testing is important.

For example, if we were interested in the impact sound had on the probability of fixating an actor’s mouth (e.g., Robinson, Stadler and Rassell, this issue) we would need to compare the gaze behaviour of a sample of participants who watch a sequence with the sound turned on to a sample who watched it with the sound turned off. By comparing the behaviour between these two groups using inferential statistics we are testing the likelihood that these two viewing conditions would differ in a population of all viewers given the variation within and between these two groups. In actual fact we do this by performing the opposite test: testing the probability that that the two groups belong to a single statistically indistinguishable group. This is known as the null hypothesis. By showing that there is less than a 5% chance that the null hypothesis is true we can conclude that there is a statistically significant chance that another sample of participants presented with the same two viewing conditions would show similar differences in viewing behaviour.

In order to test whether our two viewing conditions belong to one or two distributions we need to be able to express this distribution. This is typically done by identifying the mean score for each participant on the dependent variable of interest, in this case the probability of fixating a dynamic mouth AOI then calculating the mean for this measure across all participants within a group and their variation in scores (known as the standard deviation). Most natural measures produce a distribution of scores looking somewhat like a bell curve (known as the normal distribution) with most observations near the centre of the distribution and an ever decreasing number of observations as you move away from this central score. Each observation (in our case, participants) can be expressed relative to this distribution by subtracting the mean of the distribution from its score and dividing by the standard deviation. This converts a raw score into a normalized or z-score. Roughly ninety-five percent of all observations will fall within two standard deviations of the mean for normally distributed data. This means that observations with a z-score greater than two are highly unrepresentative of that distribution and may be considered outliers.

However, being unrepresentative of the group mean is insufficient motivation to exclude a participant. The outlier still belongs to the group distribution and should be included unless there is a supporting reason for exclusion such as measurement error, e.g. poor calibration of the eye tracker. If an extreme outlier is not excluded it can often have a disproportionate impact on the group mean and make statistical comparison of groups difficult. However, if this is the case it suggests that the sample size is too small and not representative of the overall population. Correct choice of sample size given an estimate of the predicted effect size combined with minimising measurement error should mean that subjective decisions do not have to be made about who’s data is “right” and who should be included or excluded.

Brown also believes that eye tracking research has so far marginalised viewers who have atypical ways of watching film, such as film scholars either by not studying them or treating them as statistical outliers and excluding them from analyses. However, I would argue that the only way to know if their way of watching a film is atypical is to first map out the distribution of how viewers typically watch films. If a viewer attended more to the screen edge than the majority of other viewers in a random sample of the population (as was the case with Brown’s film scholar colleague) this should show up as a large z-score when their gaze data is expressed relative to the group on a suitable measure such as Euclidean distance from the screen centre. Similarly, a non-native speaker of English may have appeared as an outlier in terms of how much time they spent looking at the speaker’s mouth in Robinson, Stadler and Rassell’s (this issue) study. Such idiosyncrasies may be of interest to researchers and there are statistical methods for expressing emergent groupings within the data (e.g. cluster analysis) or seeing whether group membership predicts behaviour (e.g. regression). These approaches may have not previously been applied to questions of film viewing but this is simply due to the immaturity of the field and the limited availability of the equipment or expertise to conduct such studies.

In my own recent work I have shown how viewing task influences how we watch unedited video clips (Smith and Mital, 2013), how infants watch TV (Wass and Smith, in press), how infant gaze differs to adult gaze (Smith, Dekker, Mital, Saez De Urabain and Karmiloff-Smith, in prep) and even how film scholars attend to and remember a short film compared to non-expert film viewers (Smith and Smith, in prep). Such group viewing differences are of great interest to me and I hope these studies illustrate how eye tracking has a lot to offer to such research questions if the right statistics and experimental designs are employed.

Brown’s second main criticism is that the field of eye tracking neglects “marginal” films. I agree that the majority of films that have so far been used in eye tracking studies could be considered mainstream. For example, the film/TV clips used in this special issue include Sherlock (2010), Up (2009) and Saving Private Ryan (1998). However, this limit is simply a sign of how few eye tracking studies of moving images there have been. All research areas take time to fully explore the range of possible research questions within that area.

I have always employed a range of films from diverse film traditions, cultures, and languages. My first published eye tracking study (Smith and Henderson, 2008) used film clips from Citizen Kane (1941), Dogville (2003), October (1928), Requiem for a Dream (2000), Dancer in the Dark (2000), Koyaanisqatsi (1982) and Blade Runner (1982). Several of these films may be considered “marginal” relative to the mainstream. If I have chosen to focus most of my analyses on mainstream Hollywood cinema this is only because they were the most suitable exemplars of the phenomena I was investigating such as continuity editing and its creation of a universal pattern of viewing (Smith, 2006; 2012). This interest is not because, as Brown argues, I have a hidden political agenda or an implicit belief that this style of filmmaking is the “right” way to make films. I am interested in this style because it is the dominant style and, as a cognitive scientist I wish to use film as a way of understanding how most people process audiovisual dynamic scenes.

Hollywood film stands as a wonderfully rich example of what filmmakers think “fits” human cognition. By testing filmmaker intuitions and seeing what impact particular compositional decisions have on viewer eye movements and behavioural responses I hope to gain greater insight into how audiovisual perception operates in non-mediated situations (Smith, Levin and Cutting, 2012). But, just as a neuropsychologist can learn about typical brain function by studying patients with pathologies such as lesions and strokes, I can also learn about how we perceive a “typical” film by studying how we watch experimental or innovative films. My previous work is testament to this interest (Smith, 2006; 2012a; 2012b; 2014; Smith & Henderson, 2008) and I hope to continue finding intriguing films to study and further my understanding of film cognition.

One practical reason why eye tracking studies rarely use foreign language films is the presence of subtitles. As has been comprehensively demonstrated by other authors in this special issue (Kruger, Szarkowska and Krejtz, this issue) and earlier in this article, the sudden appearance of text on the screen, even if it is incomprehensible leads to differences in eye movement behaviour. This invalidates the use of eye tracking as a way to measure how the filmmaker intended to shape viewer attention and perception. The alternatives would be to either use silent film (an approach I employed with October; Smith and Henderson, 2008), remove the audio (which changes gaze behaviour and awareness of editing; Smith & Martin-Portugues Santacreau, under review) or use dubbing (which can bias the gaze down to the poorly synched lips; Smith, Batten, and Bedford, 2014). None of these options are ideal for investigating foreign language sound film and until there is a suitable methodological solution this will restrict eye tracking studies to experimental films in a participant’s native language.

Finally, I would like to counter Brown’s assertion that eye tracking investigations of film have so far only generated “truisms”. I admit that there is often a temptation to reduce empirical findings to simplified take-home messages that only seem to confirm previous intuitions such as a bias of gaze towards the screen centre, towards speaking faces, moving objects or subtitles. However, I would argue that such messages fail to appreciate the nuance in the data. Empirical data correctly measured and analysed can provide subtle insights into a phenomenon that subjective introspection could never supply.

For example, film editors believe that an impression of continuous action can be created across a cut by overlapping somewhere between two (Anderson, 1996) and four frames (Dmytryk, 1986) of the action. However, psychological investigations of time perception revealed that our judgements of duration depend on how attention is allocated during the estimated period (Zakay and Block, 1996) and will vary depending on whether our eyes remain still or saccade during the period (Yarrow et al, 2001). In my thesis (Smith, 2006) I used simplified film stimuli to investigate the role that visual attention played in estimation of temporal continuity across a cut and found that participants experienced an overlap of 58.44ms as continuous when an unexpected cut occurred during fixation and an omission of 43.63ms as continuous when they performed a saccade in response to the cut. As different cuts may result in different degrees of overt (i.e., eye movements) and covert attentional shifts these empirical findings both support editor intuitions that temporal continuity varies between cuts (Dmytryk, 1986) whilst also explaining the factors that are important in influencing time perception at a level of precision not possible through introspection.

Reflecting on our own experience of a film suffers from the fact that it relies on our own senses and cognitive abilities to identify, interpret and express what we experience. I may feel that my experience of a dialogue sequence from Antichrist (2010) differs radically from a similar sequence from Secrets & Lies (1996) but I would be unable to attribute these differences to different aspects of the two scenes without quantifying both the cinematic features and my responses to them. Without isolating individual features I cannot know their causal contribution to my experience. Was it the rapid camera movements in Antichrist, the temporally incongruous editing, the emotionally extreme dialogue or the combination of these features that made me feel so unsettled whilst watching the scene? If one is not interested in understanding the causal contributions of each cinematic decision to an audience member’s response then one may be content with informed introspection and not find empirical hypothesis testing the right method. I make no judgement about the validity of either approach as long as each researcher understands the limits of their approach.

Introspection utilises the imprecise measurement tool that is the human brain and is therefore subject to distortion, human bias and an inability to extrapolate the subjective experience of one person to another. Empirical hypothesis testing also has its limitations: research questions have to be clearly formulated so that hypotheses can be stated in a way that allows them to be statistically tested using appropriate observable and reliable measurements. A failure at any of these stages can invalidate the conclusions that can be drawn from the data. For example, an eye tracker may be poorly calibrated resulting in an inaccurate record of where somebody was looking or it could be used to test an ill formed hypothesis such as how a particular film sequence caused attentional synchrony without having another film sequence to compare the gaze data to. Each approach has its strength and weaknesses and no single approach should be considered “better” than any other, just as no film should be considered “better” than any other film.

Conclusion

The articles collected here constitute the first attempt to bring together interdisciplinary perspectives on the application of eye tracking to film studies. I fully commend the intention of this special issue and hope that it encourages future researchers to conduct further studies using these methods to investigate research questions and film experiences we have not even conceived of. However, given that the recent release of low-cost eye tracking peripherals such as the EyeTribe[vii] tracker and the Tobii EyeX[viii] has moved eye tracking from a niche and highly expensive research tool to an accessible option for researchers in a range of disciplines, I need to take this opportunity to issue a word of warning. As I have outlined in this article, eye tracking is like any other research tool in that it is only useful if used correctly, its limitations are respected, its data is interpreted through the appropriate application of statistics and conclusions are only drawn that are based on the data in combination with a sound theoretical base. Eye tracking is not the “saviour” of film studies , nor is science the only “valid” way to investigate somebody’s experience of a film. Hopefully, the articles in this special issue and the ideas I have put forward here suggest how eye tracking can function within an interdisciplinary approach to film analysis that furthers our appreciation of film in previously unfathomed ways.

 

Acknowledgements

Thanks to Rachael Bedford, Sean Redmond and Craig Batty for comments on earlier drafts of this article. Thank you to John Henderson, Parag Mital and Robin Hill for help in gathering and visualising the eye movement data used in the Figures presented here. Their work was part of the DIEM Leverhulme Trust funded project (https://thediemproject.wordpress.com/). The author, Tim Smith is funded by EPSRC (EP/K012428/1), Leverhulme Trust (PLP-2013-028) and BIAL Foundation grant (224/12).

 

References

Anderson, Joseph. 1996. The Reality of Illusion: An Ecological Approach to Cognitive Film Theory. Southern. Illinois University Press.

Batty, Craig, Claire Perkins and Jodi Sita. 2015. “How We Came To Eye Tracking Animation: A Cross-Disciplinary Approach to Researching the Moving Image”, Refractory: a Journal of Entertainment Media, 25.

Banks, Martin S., Jenny R. Read, Robert S. Allison and Simon J. Watt. 2012. “Stereoscopy and the human visual system.” SMPTE Mot. Imag. J., 121 (4), 24-43

Bradley, Margaret M., Laura Miccoli, Miguel A. Escrig and Peter J. Lang. 2008. “The pupil as a measure of emotional arousal and autonomic activation.” Psychophysiology, 45(4), 602-607.

Branigan, Edward R. 1984. Point of View in the Cinema: A Theory of Narration and Subjectivity in Classical Film. Berlin: Mouton.

Brown, William. 2015. “There’s no I in Eye Tacking: How Useful is Eye Tracking to Film Studies?”, Refractory: a Journal of Entertainment Media, 25.

Buchan, Julie N., Martin Paré and Kevin G. Munhall. 2007. “Spatial statistics of gaze fixations during dynamic face processing.” Social Neuroscience, 2, 1–13.

Coutrot, Antoine, Nathalie Guyader, Gelu Ionesc and Alice Caplier. 2012. “Influence of Soundtrack on Eye Movements During Video Exploration”, Journal of Eye Movement Research 5, no. 4.2: 1-10.

Cutting, James. E., Jordan E. DeLong and Christine E. Nothelfer. 2010. “Attention and the evolution of Hollywood film.” Psychological Science, 21, 440-447.

Dwyer, Tessa. 2015. “From Subtitles to SMS: Eye Tracking, Texting and Sherlock”, Refractory: a Journal of Entertainment Media, 25.

Dyer, Adrian. G and Sarah Pink. 2015. “Movement, attention and movies: the possibilities and limitations of eye tracking?”, Refractory: a Journal of Entertainment Media, 25.

Dmytryk, Edward. 1986. On Filmmaking. London, UK: Focal Press.

Henderson, John. M., 2003. “Human gaze control during real-world scene perception.” Trends in Cognitive Sciences, 7, 498-504.

Hochberg, Julian and Virginia Brooks. 1978). “Film Cutting and Visual Momentum”. In John W. Senders, Dennis F. Fisher and Richard A. Monty (Eds.), Eye Movements and the Higher Psychological Functions (pp. 293-317). Hillsdale, NJ: Lawrence Erlbaum.

Holmqvist, Kenneth, Marcus Nyström, Richard Andersson, Richard Dewhurst, Halszka Jarodzka and Joost van de Weijer. 2011. Eye Tracking: A comprehensive guide to methods and measures. Oxford, UK: OUP Press.

James, William. 1890. The principles of psychology (Vol.1). New York: Holt

Kruger, Jan Louis, Agnieszka Szarkowska and Izabela Krejtz. 2015. “Subtitles on the Moving Image: An Overview of Eye Tracking Studies”, Refractory: a Journal of Entertainment Media, 25.

Le Meur, Olivier and Baccino, Thierry. 2013. “Methods for comparing scanpaths and saliency maps: strengths and weaknesses.” Behavior research methods, 45(1), 251-266.

Magliano, Joseph P. and Jeffrey M. Zacks. 2011. “The Impact of Continuity Editing in Narrative Film on Event Segmentation.” Cognitive Science, 35(8), 1-29.

Mital, Parag K., Tim J. Smith, Robin Hill. and John M. Henderson. 2011. “Clustering of gaze during dynamic scene viewing is predicted by motion.” Cognitive Computation, 3(1), 5-24

Rayner, Keith. 1998. “Eye movements in reading and information processing: 20 years of research”. Psychological Bulletin, 124(3), 372-422.

Rayner, Keith, Tim J. Smith, George Malcolm and John M. Henderson, J.M. 2009. “Eye movements and visual encoding during scene perception.” Psychological Science, 20, 6-10.

Raz, Gal, Yael Jacob, Tal Gonen, Yonatan Winetraub, Tamar Flash, Eyal Soreq and Talma Hendler. 2014. “Cry for her or cry with her: context-dependent dissociation of two modes of cinematic empathy reflected in network cohesion dynamics.” Social cognitive and affective neuroscience, 9(1), 30-38.

Redmond, Sean, Jodi Sita and Kim Vincs. 2015. “Our Sherlockian Eyes: the Surveillance of Vision”, Refractory: a Journal of Entertainment Media, 25.

Robinson, Jennifer, Jane Stadler and Andrea Rassell. 2015. “Sound and Sight: An Exploratory Look at Saving Private Ryan through the Eye-tracking Lens”, Refractory: a Journal of Entertainment Media, 25.

Salt, Barry. 2009. Film Style and Technology: History and Analysis (Vol. 3rd). Totton, Hampshire, UK: Starword.

Sawahata, Yasuhito, Rajiv Khosla, Kazuteru Komine, Nobuyuki Hiruma, Takayuki Itou, Seiji Watanabe, Yuji Suzuki, Yumiko Hara and Nobuo Issiki. 2008. “Determining comprehension and quality of TV programs using eye-gaze tracking.” Pattern Recognition, 41(5), 1610-1626.

Smith, Murray. 2011. “Triangulating Aesthetic Experience”, paper presented at the annual Society for Cognitive Studies of the Moving Image conference, Budapest, June 8–11, 201

Smith, Tim J. 2006. An Attentional Theory of Continuity Editing. Ph.D., University of Edinburgh, Edinburgh, UK.

Smith, Tim J. 2012a. “The Attentional Theory of Cinematic Continuity”, Projections: The Journal for Movies and the Mind. 6(1), 1-27.

Smith, Tim J. 2012b. “Extending AToCC: a reply,” Projections: The Journal for Movies and the Mind. 6(1), 71-78

Smith, Tim J. 2013. “Watching you watch movies: Using eye tracking to inform cognitive film theory.” In A. P. Shimamura (Ed.), Psychocinematics: Exploring Cognition at the Movies. New York: Oxford University Press. pages 165-191

Smith, Tim J. 2014. “Audiovisual correspondences in Sergei Eisenstein’s Alexander Nevsky: a case study in viewer attention”. Cognitive Media Theory (AFI Film Reader), Eds. P. Taberham & T. Nannicelli.

Smith, Tim J., Jonathan Batten and Rachael Bedford. 2014. “Implicit detection of asynchronous audiovisual speech by eye movements.” Journal of Vision,14(10), 440-440.

Smith, Tim J., Dekker, T., Mital, Parag K., Saez De Urabain, I. R. & Karmiloff-Smith, A., In Prep. “Watch like mother: Motion and faces make infant gaze indistinguishable from adult gaze during Tot TV.”

Smith, Tim J. and John M. Henderson. 2008. “Edit Blindness: The relationship between attention and global change blindness in dynamic scenes”. Journal of Eye Movement Research, 2(2):6, 1-17.

Smith Tim J., Peter Lamont and John M. Henderson. 2012. “The penny drops: Change blindness at fixation.” Perception 41(4) 489 – 492

Smith, Tim J., Daniel Levin and James E. Cutting. 2012. “A Window on Reality: Perceiving Edited Moving Images.” Current Directions in Psychological Science. 21: 101-106

Smith, Tim J. and Parag K. Mital. 2013. “Attentional synchrony and the influence of viewing task on gaze behaviour in static and dynamic scenes”. Journal of Vision 13(8): 16.

Smith, Tim J. and Janet Y. Martin-Portugues Santacreu. Under Review. “Match-Action: The role of motion and audio in limiting awareness of global change blindness in film.”

Smith, Tim. J. and Murray Smith. In Prep. “The impact of expertise on eye movements during film viewing.”

Suckfull, Monika. 2000. “Film Analysis and Psychophysiology Effects of Moments of Impact and Protagonists”. Media Psychology, 2(3), 269-301.

Vilaro, Anna and Tim J. Smith. 2011. “Subtitle reading effects on visual and verbal information processing in films.” Published abstract In Perception. ECVP abstract supplement, 40. (p. 153). European Conference on Visual Perception. Toulousse, France.

Velichkovsky, Boris M., Sascha M. Dornhoefer, Sebastian Pannasch and Pieter J. A. Unema. 2001. “Visual fixations and level of attentional processing”. In Andrew T. Duhowski (Ed.), Proceedings of the International Conference Eye Tracking Research & Applications, Palm Beach Gardens, FL, November 6-8, ACM Press.

Wass, Sam V. and Tim J. Smith. In Press. “Visual motherese? Signal-to-noise ratios in toddler-directed television,” Developmental Science

Yarrow, Kielan, Patrick Haggard, Ron Heal, Peter Brown and John C. Rothwell. 2001. “Illusory perceptions of space and time preserve cross-saccadic perceptual continuity”. Nature, 414.

Zakay, Dan and Richard A. Block. 1996. Role of Attention in Time Estimation Processes. Time, Internal Clocks, and Movement. Elsevier Science.

 

Notes

[ii] An alternative take on eye tracking data is to divorce the data itself from psychological interpretation. Instead of viewing a gaze point as an index of where a viewer’s overt attention is focussed and a record of the visual input most likely to be encoded into the viewer’s long-term experience of the media, researchers can instead take a qualitative, or even aesthetic approach to the data. The gaze point becomes a trace of some aspect of the viewer’s engagement with the film. The patterns of gaze, its movements across the screen and the coordination/disagreement between viewers can inform qualitative interpretation without recourse to visual cognition. Such an approach is evident in several of the articles in this special issue (including Redmond, Sita, and Vincs, this issue; Batty, Perkins, and Sita, this issue). This approach can be interesting and important for stimulating hypotheses about how such patterns of viewing have come about and may be a satisfying endpoint for some disciplinary approaches to film. However, if researchers are interested in testing these hypotheses further empirical manipulation of the factors that are believed to be important and statistical testing would be required. During such investigation current theories about what eye movements are and how they relate to cognition must also be respected.

[iii] Although, one promising area of research is the use of pupil diameter changes as an index of arousal (Bradley, Miccoli, Escrig and Lang, 2008).

[iv] This technique has been used for decades by producers of TV advertisements and by some “pop” serials such as Hollyoaks in the UK (Thanks for Craig Batty for this observation).

[v] This trend in increasing pace and visual complexity of film is confirmed by statistical analyses of film corpora over time (Cutting, DeLong and Nothelfer, 2010) and has resulted in a backlash and increasing interest in “slow cinema”.

[vi] Other authors in this special issue may argue that taking a critical approach to gaze heatmaps without recourse to psychology allows them to embed eye tracking within their existing theoretical framework (such as hermeneutics). However, I would warn that eye tracking data is simply a record of how a relatively arbitrary piece of machinery (the eye tracking hardware) and associated software decided to represent the centre of a viewer’s gaze. There are numerous parameters that can be tweaked to massively alter how such gaze traces and heatmaps appear. Without understanding the psychology and the physiology of the human eye a researcher cannot know how to set these parameters, how much to trust the equipment they are using, or the data it is recording and as a consequence may over attribute interpretation to a representation that is not reliable.

[vii] https://theeyetribe.com/ (accessed 13/12/14). The EyeTribe tracker is $99 and is as spatially and temporally accurate (up to 60Hz sampling rate) as some science-grade trackers.

[viii] http://www.tobii.com/eye-experience/ (accessed 13/12/14). The Tobii EyeX tracker is $139, samples at 30Hz and is as spatially accurate as the EyeTribe although the EyeX does not give you as much access to the raw gaze data (e.g., pupil size and binocular gaze coordinates) as the EyeTribe.

 

Bio

Dr Tim J. Smith is a senior lecturer in the Department of Psychological Sciences at Birkbeck, University of London. He applies empirical Cognitive Psychology methods including eye tracking to questions of Film Cognition and has published extensively on the subject both in Psychology and Film journals.

 

Our Sherlockian Eyes: the Surveillance of Vision – Sean Redmond, Jodi Sita and Kim Vincs

Abstract

For this inter-disciplinary article, we undertook a pilot case study that eye-tracked the ‘Holmes Saves Mrs. Hudson’ sequence from the episode, A Scandal in Belgravia (Sherlock, BBC, 2012). This small-scale empirical study involved a total of 13 participants (3 males and 10 females, mean age was: 27 years), comprised of a mixture of academics and undergraduate students at La Trobe University in Melbourne, Australia. The article examines its findings through a range of threaded frames – neuroscience, forensics, surveillance, haptics, memory, performance-movement, and relationality – and uniquely draws upon the interests of the authors to set the examination in context. The article is both a reading of Sherlock and a dialogue between its authors. We discover that the codes and conventions of Sherlock have a direct impact on where viewers look but we also discover eyes emerging in the periphery of the frame, and we account for these ways of seeing in different ways.

My Sherlockian Eyes

Sean Redmond

I have always been fascinated, perhaps even obsessed, with my eyes. I have often felt them looking into things, as if they had their own embodied consciousness that I was entirely, simultaneously, conscious of. It was as if we, my eyes and I, saw the world separately and together, possessing a double vision, one set within the meaty windows of my sockets, and the other looking outside, grasping the world with a replete hapticity, sending shivers across my pupils and retinas as they did so.

I have found myself trying to catch my eyes out, to second guess their movements, their sightlines, and their interests. I must be a sight for sore eyes on the rush hour train, wrestling with what I will allow my eyes to see. I often try to resist my conforming eyes, to make them look towards the cultural periphery, to the aesthetic margins, and to the haphazard shards of broken, refracted light on oily windows that few others see as they go about their busy, and sometimes dreary lives. I have a deep yearning to see my eyes politicised, to turn them completely into organs of touch (Marks, 2000), and to feel them wander freely across the intricate layers of the film and television screen. I want Sherlockian eyes.

I have held a rather romantic notion about my viewing eyes, and the eyes of some viewers: that they sometimes wander freely across the spaces, objects, lights, colours, bodies, movements and sounds of the diegetic world they are presented with. Narrative action may be centre frame, and all the elements of the mise en scène may be attempting to draw one’s eyes to this interaction, but I will catch myself looking to the far left of the screen, to hold my sight on an obscure pattern on a wall, or to search for the origins of a distant minor or insignificant sound just off-screen. I want to see inside and outside the narrative simultaneously. I imagine my eyes as Sherlock-like, searching for narrative clues, new plot developments, and for the sensuous expression of character, mood and feeling. But I also see them loosing or freeing themselves; my eyes (unconsciously) float within all the elements of filmic or televisual material as they happen on the screen.

I see in Sherlock’s eyes this double vision: the ability to have foresight, to see into the margins of things, and to be consciously aware of the vision within and all around him. As Sherlock sees into the finest grain of things, so do my eyes and I. My Sherlockian eyes are forensic, haptic, self-processing and are blessed with twenty twenty vision – they have the power to see into all things clearly. Sherlock mirrors or rather embodies the very qualities of the cinema machine (Metz, 1982), and of the surveillance regimes (Foucault, 1977), that emerged at the time the first Sherlock Holmes books were written (1887-1927). Sherlock is a text that already embodies the eye tracking experience.

But is this so, or just a fictive longing? What evidence do we have that our eyes do what we say they do? What evidence do we have that viewers possess a double vision? This romantic, phenomenological notion of the viewing, carnal, haptic eyes, then, we wanted to test, to explore, to see in action and interaction…

The Science of the Sherlockian Eye

Jodi Sita

I can often be found staring off into space, deep in thought, looking at nothing in particular. If I were being eye tracked it would look like I was staring at something. Where people look and, more particularly, why they are looking there, are questions that fascinate me and make me think about the phenomenon of blank stares. Human beings have fascinating eyes that, because they are housed independently, with their own localized environments, need to and can move about quite a lot. A tiny spot on the back of the eye, the retina, houses the receptors for high visual acuity, and this spot must be directed at the object we want to see, in order for us to see it clearly, and see its fine detail. People tend to move their eyes to aspects of a scene which are interesting or useful. The visual system in the brain directs these movements; they are not random. Bottom–up control processes (see; Itti & Koch, 2000) help direct some shifts in visual gaze and involve features that are thought to attract attention due to their ability to be noticeable. These include salient features such as luminosity, colour and movement. More importantly, in the human gaze we know that top-down processes are at play when viewing complex or meaningful scenes; our eyes employ feature selection which are based on our understanding of the scene and our internal expectations about where important things are or are likely to occur (Torralba et al., 2006; Birmingham et al., 2008, and Vincent et al., 2009).

What I am curious to learn more about is how our viewing behavior is shaped by what we are doing, by how we are interacting with the world and how our brains are responding to that and shaping that encounter. Thus, my involvement in this work comes from these curiosities, however, it also stems from my own forensic tendencies. It develops from my own need to ask the Sherlockian questions about viewers viewing Sherlock.

My early research had me investigate a branch of forensic science; handwriting and signature examination. At the time the area had a lot of practitioners, many quite experienced and successful, and there had been a substantial amount written about it, yet very little objective evidence for the fields’ claims had been produced. What the field needed were studies which produced hard evidence to support or dispute its original claims. My work was part of a large and ongoing body of work, where the field’s ideas and claims are tested objectively, and whose results can be used as evidence to support existing notions or derive new ones. It was within this area of research that I started using eye tracking, yet it has also led me to want to bring eye tracking to this moving-image field; where the focus of the eye tracker, shining like an objective lens over some of the theories of the area, can help bring to it some other method for its practitioners to use – to examine how viewers watch and are involved with what they are watching.

The Optometry of Sherlock

Kim Vincs

The science of the gaze—of how eyes fixate or fail to fixate—has always been of great interest to me, firstly in my original career as an optometrist, and more latterly as a choreographer and then a transmedia dance artist. As an optometrist, I was less concerned with where people looked than with whether they could look, and with the accuracy and resolution of the sensory information they received and interpreted when they did look. What do I mean by this? In considering whether people could look, I am referring to whether they were able to accurately fixate the static and moving targets they wanted and needed to. Fixating, that is, aiming one’s eyes at a static or moving target, is a function of attention, and integrated as action by the sensory and muscular systems of the eye and brain. There are many pathological conditions that interfere with the capacity to fixate a static visual stimulus quickly, accurately and efficiently. As an optometrist, I was primarily concerned with detecting these conditions and referring patients who had them for appropriate treatment. I was, in a very real sense, perfectly happy to allow my patients to decide for themselves what to fixate on. My job was simply to ensure that, should they wish to, they would be capable of locating and tracking something. This willingness to allow dissociation between capacity and will, between ability and decision, is something I consider foundational to the ways in which I have pursued my subsequent research into creative practices. I have never, as a choreographer or an interactive / transmedia artist, wished to dictate to people where they should look or what they should perceive. I consider my job to be to place appropriate objects / events / movements within a context in which they can be perceived should people so choose.

This outlook has had some specific implications for my art practice. As a choreographer, I have never thought to ask what someone watches when they observe a dancer moving. Cognitive psychologist Kate Stevens’ seminal work on eye movements in dance has demonstrated a classic novice/expert shift in the way that observers view dance. As with many other fields of expertise such as airline pilots, and driving instructors, experts make significantly fewer saccades, that is, changes in fixation, watching a dance performance than do novices, where experts are people with professional experience in dance and novices are people with no particular prior experience of the artform (Stevens et al., 2010). The implication of these results is that experts do not need to change fixation as many times as novices because they are able, to some extent, to predict where the dancing body will move. In essence, they understand what they are looking for, and are therefore able to maximize the efficiency of their fixation choices.

What Steven’s work does tell us is which movement features most attract fixation when watching a dancing body. My own work in motion capture analysis of dance movement provides me with a theory about why this might be a difficult thing to measure. Dance, at the movement level, comprises movement of some 33 major joints, each of which may make movements of entirely different velocity, acceleration and magnitude to achieve an overall aesthetic effect. The dancing body essentially has no ‘centre of focus’ that can be interpolated from movement data such as the speed, momentum or even position of specific body parts, because the semantics of dance movement are only meaningful in relation to the composition across the body. As I have argued previously, (Vincs, 2014) the semantic significance of a movement bears no relationship to its metrics, such as amplitude or speed. In some aesthetic contexts, tiny movements of the fingers may be essential to the meaning and feeling tone of the movement. In others, such as large virtuosic or acrobatic movement forms, hand gestures may contribute relatively little to a movement’s significance.

Dance grammars are aesthetically and culturally, rather than anatomically determined. I think that this fact has also contributed to my attraction to the notion of Sherlockian eyes. As a choreographer, I am always a detective, seeking potential significances in movement rather than predetermined ones. I value the opportunity to go looking for the dancing body, browsing, shuffling, wandering through the multiple and complex joint actions that comprise a single ‘step,’ looking for something of newness and emotional value rather than assuming I know what it is and where I will find it. Yet I am always aware that my aesthetic search is underpinned by a neurosensory apparatus that is primed to respond selectively to human movement (Hagendoorn, 2004, Vincs, 2009). I am therefore armed, at least potentially, with an inherent ‘grammar’ that is defined by the morphology and physical capacity of the human body, and I am curious as to what predilections and biases my visual sensory system imposes on my seemingly adventurous gaze.

What now follows is an exploration of our different approaches to the eye tracking data that we generated. Jodi is first off, outlining our empirical method and undertaking a close reading of the preliminary results. Jodi shows how the results begin to tell us that the viewers’ gaze patterns and fixations are closely clustered together, and she situates these findings in relation to the science of the eye, the importance of the face in human communication, and to the visual and narrative codes and conventions of Sherlock. Sean then explores the results in terms of haptic visuality and the surveillance gaze, drawing upon phenomenology and the discourses of conspiracy to argue that the vision in Sherlock is marked by touch, texture, and control. Kim examines the results in terms of movement and relationality, examining the eye tracking data in terms of the way it supports and confirms the necessary nature of vision in seeing into moving things. Kim shows that even though there is a high degree of direction in terms of where viewers are being asked to look, visual perception allows or enable the eyes to wander. Finally, we conclude our article together, drawing together our voices to offer an interdisciplinary way forward.

Eye Tracking Sherlock (the objective viewing): Methods and Preliminary Results

Jodi Sita

We undertook a pilot case study that eye-tracked the ‘Holmes Saves Mrs. Hudson’ sequence from the episode, A Scandal in Belgravia (Sherlock, BBC, 2012). This small-scale empirical study involved a total of 13 participants (3 males and 10 females, mean age was: 27 years), comprised of a mixture of academics and undergraduate students at La Trobe University in Melbourne, Australia.

A Tobii X-120 remote eye tracker (Tobii Technology, Stockholm, Sweden) was used to record participants eye movements which has an accuracy of 0.5ºof visual angle and allows a moderate amount of free head movement (30 x 22 x 30 cm at 70 cm (Width x Height x Depth)). This data collection technique uses reflected infrared light from the eye to determine participants viewer gaze positioning and allows for natural head movements and natural human responses to screened material. The eye tracker was connected to a PC running an Intel ® core ™ i7 CPU ‘Cool Master’ hard-drive. The eye tracker used Tobii Studio 2.3.2 professional edition software for the presentation of the movie scene stimuli and recording eye movements. The eye tracker was set up on a desk, situated below a Dell PC monitor (1680×1050), which was utilised by the participants to view the Sherlock sequence. Participants were seated on a sturdy chair between 55-65cm away from the eye tracker and between 65-75cm from the viewing screen. A second screen (Dell; 1920×1080) was utilised by the researchers to view, in real time, the eye movements of the participants as they were being tracked and calibrated, although all computer analyses and statistics reported here was based on stored data.

Participants were recruited via posters advertising the study at La Trobe University, with ethics approval (Ethics approval number: FHEC13/101). Participants were required to be at least 18 years of age to be considered eligible. People whom expressed interest in taking part in the study were contacted via email to attend a single recording session. In preliminary tests participants were introduced to the study and screened for exclusion criteria such as taking medications (e.g. benzodiazepines) that may potentially affect their eye movement, known neurological conditions, disorders or injuries that could potentially affect their eye movement. All participants were screened for normal or corrected to normal near visual acuity of N8 or better on the Designs for Vision near sighted visual acuity test, and with a pen-torch eye movement excursion test to screen for symmetrical movement of the eyes. Participants who were ametropic were allowed to wear their glasses to watch the stimuli.

Prior to eye movement data being collected, the eye tracker was configured for each participant using a 9-point on screen calibration test within the Tobii Studio recording software. Participants were told only that they would watch short segments from a variety of films. Recording sessions typically lasted between 15-25 minutes, and each participant was tested individually.

First, we found that our viewers’ eyes were strongly drawn to follow movement and directional cues and signs. This included camera and character movement. In the opening scene, where Mrs. Hudson’s fingers scrape along the wall, followed by Sherlock’s fingers retracing her steps (03-010 seconds), we see all viewers making strings of successive fixations – each following these finger movements (see Figures 1 and 2). The sound of these fingers scraping along the wall was heavily amplified, and fully sychronised, and we suggest, then, that sound was also an aesthetic device being employed to direct where viewer’s looked. These results confirm previous findings where camera movement, sound, character behavior, and editing patterns are seen to inform gaze patterns and fixations (see Smith, forthcoming, Smith and Mital, 2011).

In one brief shot in the middle of this scene, we cut to a close-up of Mrs. Hudson’s face, full of anguish. All the subjects discern this face in amongst the movement and chaos of the surrounding action, as seen by their fixating to its features. Her face is captured in the center of the screen, making it central to the scene’s visualisation. However, the face is known to be a strong attractor of what the eye attends to (Treuting (2006)), and with it being such an important narrative component in this scene, would have been a strong attentional cue.

Figure 1: Finger drag: 2 subjects

Figure 1: Finger drag: 2 subjects

Figure 2: Finger drag: 13 subjects

Figure 2: Finger drag: 13 subjects

Second, we observed an alignment in vision with regards to where Sherlock was looking. This sight co-proximation is referred to as ‘joint attention,’ in which what one attends to seems to shift automatically to where another is looking (Birmingham et al, 2009). Interestingly, this is a common misdirection trick used by magicians (Kuhn, et al, 2009).

In particular, we observed that Sherlock’s’ point of view in the scene very often produced a close proximity in viewers focus and attention (participant’s looked where Sherlock looked, and with the same overall gaze patterning, see from 1.05 to 1.14). This also supports the findings reported by other film scholars using eye tracking methods, such as Rassell et al. (forthcoming, 2015) who found that a character’s point of view and subjective experiences have an influence on where  viewers look.

The trends for this short sequence support the idea that Sherlock is a character-driven drama in which his vision is not only foregrounded but given omnipotent and omniscient power. Thus, viewers are not only being positioned to observe from his authorial position but to trust where he looks and what he discovers there. There are recognizable genre codes and conventions also in play, structuring the looking patterns we have observed. This is a detective-thriller series that repeats a series of camera and editing motifs that become familiar to audiences (Neale, 1990).

Sherlock Image 3

Figure 3: A heap map showing the hot spots where viewer’s gazed; red indicating longer dwelling time

Figure 4: The gaze plots showing the sequence of looks that viewers made over Sherlock’s face

Figure 4: The gaze plots showing the sequence of looks that viewers made over Sherlock’s face

Third, we found that viewers focused heavily on the characters faces, both in scenes with dialogue and those without. In scenes where Sherlock was clearly putting together the evidence, viewers focused heavily on his eyes, dwelling there for almost the entire shot. (Figures 3 and 4).

Viewers fixated back and forth between the eyes, face, and mouth of the central characters. These viewing patterns are characteristic of the movements made in facial and emotional recognition (Ekman & Friesen, 1971; Hernandez, et al, 2009) and show some indication that viewers were paying attention to the different character’s in the scene, working out the role of each character and what their intentions and emotions might be. These patterns of eye movements suggest that viewers are engaging with the scene as they would in a normal face-to-face encounter, using eye movements to verify who people are and what they are feeling. It is interesting to note that people who are not able to perform these socially informative tasks, such as those with the disorder schizophrenia, and with some traumatic brain injuries, do not show the same eye movement behaviors (Watt & Douglas, 2006; Loughland et al, 2002; Williams,et al, 1999).

Our viewers clearly followed narrative cues in line with the dialogue exchanges, looking back and forth between the character’s interpersonal relays (Figures 5, 6 and 7). These results are similar to those of Treuting (2006), who eye-tracked 14 participants viewing short clips from such films as Shawshank Redemption (1994) and Harry Potter and the Philosopher’s Stone (2001). Treuting found that gaze clusters emerged in and around the central character’s faces involved in dialogue and moments of heightened drama (see also, Redmond, Sita, 2013).

Sherlock-5.-Single-viewer-charecter-alighnment

Figure 5: Single viewer character alignment

Figure 6: 6 subjects and the relay of looks on eyes, mouths and faces.

Figure 6: 6 subjects and the relay of looks on eyes, mouths and faces.

Figure 7: Final scene, last 12 seconds, searching for information: 13 subjects (most of the fixations are falling over the faces of the 2 central character’s in dialogue)

Figure 7: Final scene, last 12 seconds, searching for information: 13 subjects (most of the fixations are falling over the faces of the 2 central character’s in dialogue)

Fourth, we saw evidence that viewers searched for narrative information and cues: this included fixating on aspects of the background wall before Sherlock first enters the scene (from 0.33-0.36 seconds), then moving between the image of a smile seen on the wall and Holmes’ face, spending time ‘reading’ the shop window signs and the note on the front door (Figure 8) as Watson arrives at the scene (from 2.22 to 2.37). One can understand such scanning as influenced by the meticulous work of the mise en scène: where all the elements have been carefully placed to enact this type of searching for narrative cues (see Smith, forthcoming).

Clip-8

Figure 8: Searching for narrative information

Finally, albeit in relation to our last point, we observed that certain viewers looked at more elements of the mise en scène (Figure 9, shows gaze patterns for 4 of the 13 viewers), including the interior lights, the computer, and furniture, even as the more dramatic moments of the scene were taking place.

These findings were interesting but not totally unexpected; we would hope that not all people viewing the same scene would watch it in the same way (this is something that is discussed further below). Insights like this allow us to see that even though there are some aspects that are strong attention grabbers, such as faces and movement within a scene, other aspects can captivate and draw attention away from those areas of interest. For example, the scene shown below (Figure 9) involves a particularly emotive exchange between two key characters, Mrs. Hudson and Dr. Watson. The fact that 4 of the viewers were attending elsewhere helps us to see these aspects of interest outside the main narrative at play. Why certain viewers look to the margins of the screen, to the more ‘insignificant’ elements of the mise en scène remains of great interest. One possibility would be that these viewers were not fully engaged with the exchange between the characters, and their attention therefore drifted to other elements in the scene. Another possibility is that these scenic elements drew particular interest because of their pattern, colour, etc. Further testing is needed to begin to tease out whether this response is scene-dependent, or a characteristic of these particular observers.

Figure 9: A slightly different patterning – 4 subjects – and wider viewing

Figure 9: A slightly different patterning – 4 subjects – and wider viewing

It should be noted, nonetheless, that these observations come from only a very small sample (13 participants to date), which will be increased, and which still needs to undergo further data analysis and interpretation.

In summary, what have we see in these results so far? Evidence of the eyes being held to attention by narrative cues, by camera and character movement, faces, dialogue, point of view and performance. These were elements to be expected and add to the growing body of eye tracking evidence that supports much of current film and television theory, particularly those working in the cognitive tradition such as David Bordwell (2007) and Noël Carroll (1996). The results equally support the results of other studies into; narrative centered visual texts (see Batty et al, forthcoming, 2015); and how sound and movement affect gaze patterns (Rassell et al, forthcoming, 2015) Further, they speak to the way viewers are pulled seamlessly into the diegetic worlds they believe and invest in.

In this article we would now like to apply two different theoretical filters to the results just summarized; the first will be an examination of the gaze, by Sean, and the second will be an examination of the physiological and perceptual processes of the eye in relation to movement, by Kim. Both filters will attempt to make deeper sense of the results from the traditions in which the scholars operate from. Following this analysis, a summary conclusion will draw their approaches together to make further inter-disciplinary sense of Sherlock’s eyes.

Sherlock’s Gaze

Sean Redmond

The concept of the gaze has a long and contentious history in Film Studies if much less so in the study of television. In fact, John Ellis has suggested that the domestic context in which television viewing has historically taken place, with a host of likely distractions, and in a context of constant programme flow and segmentation, produces a glance aesthetic whereby the image isn’t looked into deeply or for a sustained period of time (1982: 138). Sherlock, of course, contests this idea since the programme is heavily built around the details of forensic gazing.

Most notably, the idea of the gaze has been employed in psychoanalytical film theory to make the argument that the cinema looking apparatus is patriarchal and heterosexual, and viewers are positioned as ‘male’ subjects through which masculine identifications emerge (Mulvey, 1975, 1989). Its main male characters, and male writers and directors of course control the vision regime in Sherlock, although its objects of focus are rarely to-be-looked-at female characters.

Critical race theory, by contrast, has employed the concept of the gaze to demonstrate how the racial Other is fixed in inferior, marginal and fetishized subject positions (Hall, 2001). Sherlock can be read as a post-colonial text enacting a present England that centres whiteness and ‘invisibly’ marginalizes the Other from its panopticon empowered centre (Cuningham, 2004). The Other in Sherlock of course extends to those who sit outside the bourgeois social centre; there are particular class dimensions to the way crime is surveyed and defined (Jann, 1990).

In terms of surveillance discourse, film has been read as a vision machine set within a invasive visual culture that promotes:

the normalizing gaze, a surveillance that makes it possible to qualify, to classify and to punish. It establishes over individuals a visibility through which one differentiates and judges them (Foucault, 1977: 25).

Sherlock can be read as a text that carries out this normalizing gaze, defining the parameters of law and order and the way the criminal can be discovered, classified, and ultimately disciplined. That is not to say that the visual excesses of the programme do not at times undermine its simple binaries. To the contrary, Sherlock is constantly troubled by its own dominant discourses particularly through the way Sherlock is also a maverick outsider.

Finally, film phenomenology has made use of the gaze to demonstrate how looking and seeing is always embodied, experiential, and depending on the text, haptic and synesthetic – where ‘the eyes themselves function like organs of touch’ (Marks, 2010, 162). Sherlock creates the conditions of both embodied presence and haptic visuality through the way the gaze is employed to see deeply into things, while the programmes textural mise en scène ‘demands’ to be attended to.

What I would now like to do is analyse two particular aspects of the way the gaze can be understood in Sherlock, relating my reading back to the eye tracking results that we have, and to eye tracking technology itself. First, I will explore Sherlock through its forensic gazing and the way this creates the particular conditions for the way viewers become locked into particular viewing patters and relations. Second, I will explore Sherlock through its haptic elements whereby the viewer is understood to gaze at and touch (things) simultaneously.

The Forensic Gaze

In Sherlock one can argue that camera movement and position are motivated by the following factors. First, to reveal narrative information such as a new location, or setting; character relations and their relative physical proximity; time, and temporal detail; and moments of revelation where a new angle or focus reveals something previously hidden or a new ‘enigma’ emerges. Second, as a dramatic device: the camera is re-positioned to signal and cue moments of narrative development, crisis, reaction, and activation. Third, there are repeated and recognised televisual conventions of the programme: one can locate and expect certain camera motifs to function in Sherlock, such as the way we enter Sherlock’s mind’s eye to see what he is unearthing in microscopic close-up. Finally, camera movement and position signals certain emotional states and modes of feeling. The cut to a close-up, for example, a moment of affecting intensity, such as is the case with the fingers being scraped along the wall in the scene under analysis in this article.

When one takes these Sherlockian codes and conventions into consideration one can make better sense of the eye tracking results that we have gotten. The eyes of the viewer seem to be relentlessly led and directed. Viewers familiar with the programme’s codes can be expected to have expectations of its visual tapestry, and to make predictions about where to look (see Rassell et al. forthcoming). This would explain both the way that viewers seem closely aligned with the looking operations of the scene (figures 1-7), and the way that viewers scan shots for narrative information (figures 8-10).

However, I also think there is something more telling to discover here – one around a consistent forensic looking regime where the text and the viewer align. This is the ‘double vision’ we refer to in my introduction to this article. Viewers come to embody the gazing powers that Sherlock possess and look at the diegetic world through his eyes even where no direct or imagined point of view is in operation. Viewers experience their very own form of social surveillance becoming detectives and snoopers in the process. Sherlock, then, can be read as a text of and for paranoid surveillance, fuelled by the constant search for facts, omissions, falsehoods, and half-truths. At a more general cultural level, trust is at issue here in what is perceived to be an age of ‘faithless’ activity and widespread corruption, where politicians are regarded to be as corrupt as the criminals they covertly support. Sherlockians ultimately become part of this age of conspiracy (Knight, 2002).

As does, in a very real sense, eye tracking technology and the data it produces. Sherlock is his very own eye tracker – he creates his own heat maps and relays and through this inbuilt biotechnology he sees into everything. Eye tracking technology is Sherlocklian and the data it produces allows us to see into everything the viewer sees. Or, at least mostly…

The Haptic Gaze

Laura U Marks (2010) has written that haptic visuality is a more intimate form of looking, where the eyes, ‘move over the surface of its object rather than plunge into illusionist depth, not to distinguish form so much as to discern texture” (162). For Marks, film and video may be “thought of as impressionable and conductive, like skin.” (2000: xi-xii) and this sensory materiality is heightened by it containing:

Grainy, unclear images; sensuous imagery that evokes memory of the senses (i.e. water, nature); the depiction of characters in acute states of sensory activity (smelling, sniffing, tasting, etc.); close-to-the-body camera positions and panning across the surface of objects; changes in focus, under- and overexposure, decaying film and video imagery; optical printing; scratching on the emulsion; densely textured, effects and formats such as Pixelvision… and alternating between film/video.  (Totaro, 2002)

The gaze found in Sherlock is very often a haptic one. The programme’s entire mise en scène evokes the activity and memory of sensation. Lights, objects, clothes, furniture, exteriors are given deep and layered textures. Sherlockian environments are populated with objects and qualities that are themselves sensory driven (poison, oil, tactile fabrics, beads of sweat, cigarette smoke, wet soil). The camera very often dwells on these, picks them out, and tracks and pans over them in close and proximate detail. Sherlock of course is a master of haptic visuality – his eyes touches the things that he observes or that he conjures up in his imagination. In many respects, then, the viewer is also invited to see Sherlock through a haptic lens.

If one was to return to the eye tracking results on the scene what we might be observing is not just an alignment in vision, and the search for narrative information, but eyes that have been turned into organs of touch and deep sensual appreciation. For example, in image 1 and 2 viewers are not just following the fingers that scrape along the wall but touching (with) them, and in touching them feeling them as if it is their fingers suffering this pain. In figures 5-7 viewers are not just following the relay of looks between the two characters but ‘touching’ faces, eyes and mouths. In figures 8-10, viewers are not just searching for narrative information and clues but are actively seeing into the textures, lights, objects, items that populate those scenes. The heat maps that eye tracking technology can generate may be more apt than we imagine since the suggestion of temperature, of body-heat, may well give truth to the embodied and carnal nature of vision. This is one of the limitations of eye tracking technology, however, since it cannot tell us what people are feeling when watching a film or television text.

I would like to make one final observation about the eye that searches the mise en scène for narrative information or clues, as in figures 8-10. This is a point about the privacy or individualism of watching a screen text so co-dependent it is on personal memory, biography, and the contexts one finds one self-viewing something. After the viewing of the scene one of our subjects remarked that they had actually spent much of the time trying to figure out whom the actor was playing Dr Watson. Any number of ‘personal’ factors might get in the way of the looking regime of the text and for why we might scan a particular text.

Roland Barthes (1981) has usefully employed the concept of the punctum (a Latin word derived from the Greek word for trauma) to viewing photographs. He argues that the still image inspires an intensely private meaning, one in which an affecting ‘partial object’ emerges from its centre to ‘prick’ or ‘wound’ the viewer. The punctum is personal and as soon as it emerges it holds the viewer’s gaze. Although Barthes is singularly writing about the photograph I think the idea of the punctum can be applied to the moving image text, to Sherlock. Although a dynamic media, television and film still settle on images and representations that reach out into the private realm of the viewer; and the viewer still finds their memories, traumas, life events activated and mobilised in the fictive worlds constructed. The wandering eyes of figures 8-10 are caught in their own biographical exploration, looking for objects that may ultimately wound them, their haptic eyes. In Sherlock we are not just being positioned as objects and subjects of surveillance but as carnal beings.

The Eyes of Sherlock; the Eyes of the Viewer

Kim Vincs

Coming, as I do, from the perspective of a dancer and choreographer, I read these eye tracking results not exactly as touch, but as a search for relationality. Erin Manning, in her seminal work on the philosophy of movement (Manning, 2009) emphasizes the incipiency of movement—movement as something in the act of becoming something, of reaching towards something or someone—over its positionality. That is to say, movement, in Manning’s terms, is always a process of relationality or reaching towards the world, rather than simply a series of coordinates on a grid. Bodies, or, more precisely “bodies-in-the-making,” are a means of thought rather than simply of action because they define, by the possibilities embedded in the moment of pre-articulation, a relationship or set of relationships within the world (Manning, 2009: 78).

Our eye movement data defines very clearly the relationships between the protagonists. In figures 1 and 2, ‘finger drag,’ the fixations map the pathway of Mrs. Hudson’s fingers along the wall. The relationships between Mrs. Hudson’s fingers and the texture of the wall are, in fact, the only potential human relationships within the scene. These relationships are not ‘positional’— that is to say, fingers defining points on the wall—but, by virtue of their spatial distribution, an articulation of the trajectory of Mrs. Hudson’s hand in relation to the wall.

Figures 3—6 reveal fixation patterns that are concentrated around the face, and in particular the eyes. These fixations address the origin of relationality in the scenes within the eyes, as if the eyes are understood to reveal the incipient thought of the characters. In Manning’s terms, incipiency the moment in which a movement is in a state of ‘pre-acceleration,’ organizing and mobilizing itself, yet still capable of any number of actual outcomes, is the most potent aspect of a movement. The predominance of fixations on the eyes and face, while perfectly interpretable in as simply a biological reflex designed to respond to and recognize human faces, also speaks of the process by which relationality is thought into being.

Figures 7—10, with their fixations distributed between human-to-human gaze and gesture, (Figure 7) non-human scenic elements that lend cognitive elements or ‘clues,’ (Figure 8) and poetic elements such as the look and texture of surrounding objects in Figures 8 (curtain texture) and 10 (lens flare), articulate an expanded notion of relationality in which human and non-human elements are implicated within a web of actions.

For me, these results suggest a mutability between human and non-human elements that is reminiscent of Manning’s understanding of relationality as something that can have both interpersonal and person-world dimensions, and also points to the kind of ‘Sherlockian body’ I seek as a dance artist. A purely ‘narrative’ approach to the filmic bodies analysed here would suggest that only human factors would feature in the gaze analysis. Similarly, a biologically driven notion of human movement perception as pre-wired to detect human shapes and actions would not predict migrations to and from the bodies to surrounding objects.

I read these results as revealing a semantic ambiguity that can form the basis for a Sherlockian search for relationality that is produced and constructed by the viewer as much as it is dictated by the film-maker. These scenes offer relatively few visual details from which to construct such a relationality, and this, no doubt, is indicative of a film-making style designed to direct the eye, and hence the mind, to very specifically and deliberately arranged narrative events. However, despite the seeming prescriptiveness of these images, they offer the viewer an opportunity, to construct relational scenarios across like and non-like (human and non-human) scenic elements. They therefore demonstrate at least the potential for a Sherlockian approach to the visual perception of movement.

Conclusion

So, what have we seen with your eyes? On the one hand, we have demonstrated how Sherlock’s narrative and mise en scène pulls the viewer into taking certain (emotive, forensic, relational) viewing positions. Sherlock is tightly bound by a number of codes and conventions and its palette and composition are highly constructed, creating spaces and interactions that focus our eyes. On the other hand, we have looked at the way haptics and relationality open up the possibility for better understanding the synaesthetic and organic/inorganic ways through which the vision of movement takes place. A televisual text such as Sherlock intends to marshal our viewing experience but as we have also seen, eyes wander; they search, their own movement and the poetics of movement opening up textual encounters not always pre-determined by the scenes deliberate operations. Our eyes escape themselves. Finally, we have noted how memory and biography can impact on where and why we look, or why we might look away. One of our respondents searched Watson’s face in the hope of remembering the actor’s name. The brutal violence metered out on Mrs. Holmes may be looked at (felt) differently by someone who has themselves undergone such misery. This is ultimately about our-being-in-the-world, and about how we make sense of beings-in-the-fictive world. Vision is never disembodied but full of the drink, food and love of life itself.

Vision needs to be understood as something that can only be fully understood through combining different theoretical and methodological frameworks. What we have found in this article, and in the broader work of the Eye Tracking the Moving Image Research group – see the introduction to this special edition for more on this group – is that it is in the conversations and deliberations, provocations and discussions between vision scientists, neuroscientists, anatomists, choreographers, film makers, ethnographers, screen theorists, and screen writers where insights are best made, conclusions thickened, and arguments enriched and extended. What we have found in this article is that Sherlock exists at the eye of the ArtsScience nexus, and it is at this nexus where the authors would like to situate their work.

The language of film and television constantly creates these spaces of vision and for seeing to take place, whether this be; the embodied point-of-view shot which allows us to become the character; aerial cinematography that brings wide open exteriors and bejeweled cityscapes into view; the furtive camera that glimpses into dark corners, allowing us to happen on to what is supposedly hidden; or the interiorized gaze that expressionistically captures the nightmare visions of the lost, the hunted, and the alien. With a close-up shot one can trace the undulating valleys of emotion on a character’s face and feel their affecting eyes reaching out and into yours. In Sherlock, the very force of his vision, mobilized through special effects and the power of digital photography, enables his/our eyes to create mathematical formulations out of thin air and to re-visit crime scenes as if one is witnessing, experiencing it all over again. Sherlock establishes the sense that all vision is embodied and personally engineered, and to the wider conceit that television is an artform that gives a miraculous, omnipotent and omnipresent vision to its ever watchful viewers.

 

References

Barthes, Roland. 1981. Camera lucida: Reflections on photography. Macmillan.

Batty, Craig, Dyer, Adrian, Perkins, Claire and Sita, Jodi. 2015. “Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative.” In Making Sense of Cinema, edited by CarrieLynn D. Reinhard and Christopher Olson, New York: Bloomsbury, forthcoming.

Birmingham, Elina, Bischof, Walter. F., and Kingstone, Alan. 2008. “Gaze selection in complex social scene.” Visual Cognition 16 (2–3): 341–355

Bordwell, David. 2007. Poetics of Cinema. London: Routledge.

Carroll, Noël. 1996. Theorizing The Moving Image, Cambridge, Cambridge University Press.

Cuningham, Henry. 1994. “Sherlock Holmes and the Case of Race”. The Journal of Popular Culture, 28: 113–125. doi: 10.1111/j.0022-3840.1994.2802_113.x

Ellis, John. 1982. Visible Fictions, London: Routledge.

Ekman, Paul and Friesen, Wallace, V. 1971. “Constants across cultures in the face and emotion.” Journal of Personality and Social Psychology Vol 17 (2): 124-129.

Foucault, Michel. 1977. Discipline and Punish: the Birth of the Prison, New York: Random House.

Itti, Laurent and Koch, Christof. 2000. “A saliency-based search mechanism for overt and covert shifts of visual attention.” Vision Research 40 (10–12): 1489–1506.

Jann, Rosemary. 1990. “Sherlock Holmes codes the social body”. ELH: 685-708.

Knight, Peter. (Ed.). 2002. Conspiracy nation: The politics of paranoia in postwar America. New York: New York University Press.

Kuhn, Gustav, Tatler, Ben, and Cole, Geoff. 2009. “You look where I look! Effect of gaze cues on overt and covert attention misdirection.” Visual Cognition 17 (6/7): 925-944.

Hagendoorn, Ivar. 2004. “Some speculative hypotheses about the nature and perception of dance and choreography”. Journal of consciousness studies, 11, 79 – 110.

Hall, Stuart. 2001. :The Spectacle of the Other”. Discourse theory and practice: A reader, Routledge: 324-344.

Hernandez, Nadia, Metzger, Aude, Magné, Rémy, Bonnet-Brilhault, Frédérique, Roux, Sylvie, Barthelemy, Catherine, and Martineau, Joëlle. 2009. “Exploration of core features of a human face by healthy and autistic adults analyzed by visual scanning.” Neuropsychologia 47(4): 1004-1012.

Loughland, Carmel, M., Williams, Leanne. M., and Gordon, Evian. 2002. “Visual scanpaths to positive and negative facial emotions in an outpatient schizophrenia sample.” Schizophrenia Research, 55: 159–170.

Manning, Erin. 2009. Relationscapes: movement, art, philosophy. Kindle edition Cambridge, Mass: MIT Press.

Marks, Laura U. 2000. The Skin of the Film: Intercultural Cinema, Embodiment and the Senses, Duke University Press.

Metz, Christian. 1982. The imaginary signifier: Psychoanalysis and the cinema. Indiana University Press.

Mulvey, Laura. 1975. “Visual Pleasure and Narrative Cinema”. Screen, 16(3), 6-18.

Mulvey, Laura. 1989. Visual and Other Pleasures. London: Macmillan.

Neale, Steve. “Questions of Genre”. Screen: 31(1) (1990): 45-65.

Rassell, Andrea, Redmond, Sean, Robinson, Jenny, Stadler, Jane, Verhagen, Darrin and Pink, Sarah. 2015. “Seeing, Sensing Sound: Eye Tracking Soundscapes in Saving Private Ryan and Monsters, Inc.”. In Making Sense of Cinema, edited by CarrieLynn D. Reinhard and Christopher Olson, New York: Bloomsbury, forthcoming.

Redmond, Sean and Sita, Jodi. 2013. “What eye tracking tells us about the way we watch films”, 5th December, The Conversation. Accessed 5th September 2014.

Smith, Tim, J. 2013. “Watching You Watch Movies: Using Eye Tracking to Inform Cognitive Film Theory.” In Psychocinematics: Exploring Cognition at the Movies, edited by Arthur P. Shimamura, Oxford University Press.

Smith, Tim. J. and Mital, Parag. K. 2011. “Watching the world go by: Attentional prioritization of social motion during dynamic scene viewing.” [conference abstract]. Journal of Vision, 11(11): 478.

Stevens, Catherine, Winskel, Heather, Howell, Claire, Vidal, Lyne, Latimer, Cyril, and Milne-Home, Josephine. 2010. “Perceiving Dance: Schematic Expectations Guide Experts’ Scanning of a Contemporary Dance Film. Journal of Dance Medicine & Science, 14(1), 19 – 25.

Torralba, Antonio, Oliva, Aude., Castelhano, Monica. S., Henderson, John, M. 2006. “Contextual guidance of eye movements and attention in real-world scenes: The role of global features in object search.” Psychological Review, 113 (4): 766–786.

Totaro, Donato. 2002. “Deleuzian Film Analysis: The Skin of the Film”. Off-Screen,  (accessed August 1st, 2011).

Treuting, Jennifer. 2006. “Eye tracking and cinema: A study of film theory and visual perception.” Society of Motion Picture and Television Engineers, 115(1): 31-40.

Vincent, Benjamin.T., Baddeley, Roland, Correani, Alessia, Troscianko, Tom, and Leonards, Ute. 2009. “Do we look at lights? Using mixture modelling to distinguish between low- and high-level factors in natural image viewing”. Visual Cognition, 17 (6–7), 856–879.

Vincs, Kim, and Barbour, Kim. 2014. “Snapshots of Complexity: using motion capture and principal component analysis to reconceptualise dance”. Digital Creativity, 25(1) 62-78.

Vincs, Kim, Schubert, Emery, and Stevens, Catherine. 2009. “Measuring responses to dance: is there a ‘grammar’ of dance?” Proceedings of the World Dance Alliance Global Summit, Brisbane, July 14-16, 2008.

Watts, Amber, J., and Douglas, Jacinta, M. 2006. “Interpreting facial expression and communication competence following severe traumatic brain injury”. Aphasiology, 20, 707–722.

Williams, Leanne, M., Loughland, Carmel, M., Gordon, Evian. and Davidson, Dean. 1999. “Visual scanpaths in schizophrenia: Is there a deficit in face recognition?”. Schizophrenia Research, 40: 189–199.

 

Bios

Sean Redmond is an Associate Professor in Media and Communication at Deakin University. He has research interests in film and television aesthetics, film and television genre, film authorship, film sound, stardom and celebrity, and film phenomenology. He has published nine books including The Cinema of Takeshi Kitano: Flowering Blood (Columbia, 2013), and Celebrity and the Media (Palgrave, 2014), and with Su Holmes he edits the journal Celebrity Studies. Sean Redmond and Jodi Sita set up the Eye Tracking the Moving Image Research group in 2011.

Jodi Sita is an academic working within the areas of neuroscience and anatomy with expertise in eye tracking research. She has extensive experience, with multiple project types using eye tracking technologies and other biophysical data. Her current research uses eye tracking to study viewers gaze patterns while watching moving images; to examine expertise in Australian Rules Football League coaches and players and to examine the signature forgery process.

Professor Kim Vincs is Director and founder of Deakin Motion.Lab, at Deakin University. Kim integrates scientific and artistic approaches through research. She is currently working on a three-year project, supported by the Australian Research Council’s Discovery program. Her collaborations with mathematicians, biomechanists and cognitive psychologists span Deakin and the Universities of Sydney, Western Sydney and New South Wales.

From Subtitles to SMS: Eye Tracking, Texting and Sherlock – Tessa Dwyer

Abstract

As we progress into the digital age, text is experiencing a resurgence and reshaping as blogging, tweeting and phone messaging establish new textual forms and frameworks. At the same time, an intrusive layer of text, obviously added in post, has started to feature on mainstream screen media – from the running subtitles of TV news broadcasts to the creative portrayals of mobile phone texting on film and TV dramas. In this paper, I examine the free-floating text used in BBC series Sherlock (2010–). While commentators laud this series for the novel way it integrates text into its narrative, aesthetic and characterisation, it requires eye tracking to unpack the cognitive implications involved. Through recourse to eye tracking data on image and textual processing, I revisit distinctions between reading and viewing, attraction and distraction, while addressing a range of issues relating to eye bias, media access and multimodal redundancy effects.

Figure 1

Figure 1: Press conference in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Introduction

BBC’s Sherlock (2010–) has received considerable acclaim for its creative deployment of text to convey thought processes and, most notably, to depict mobile phone messaging. Receiving high-profile write-ups in The Wall Street Journal (Dodes, 2013) and Wired UK, this innovative representational strategy has been hailed an incisive reflection of our current “transhuman” reality and “a core element of the series’ identity” (McMillan 2014).[1] In the following discussion, I deploy eye tracking data to develop an alternate perspective on this phenomenon. While Sherlock’s on-screen text directly engages with the emerging modalities of digital and online technologies, it also borrows from more conventional textual tools like subtitling and captioning or SDH (subtitling for the deaf and hard-of-hearing). Most emphatically, the presence of floating text in Sherlock challenges the presumption that screen media is made to be viewed, not read. To explore this challenge in detail, I bring Sherlock’s inventive titling into contact with eye tracking research on subtitle processing, using insights from audiovisual translation (AVT) studies to investigate the complexities involved in processing dynamic text on moving-image screens. Bridging screen and translation studies via eye tracking, I consider recent on-screen text developments in relation to issues of media access and linguistic diversity, noting the gaps or blind spots that regularly infiltrate research frameworks. Discussion focuses on ‘A Study in Pink’ – the first episode of Sherlock’s initial season – which producer Sue Vertue explains was actually “written and shot last, and so could make the best use of onscreen text as additional script and plot points” (qtd in McMillan, 2014).

Texting Sherlock

Figure 2

Figure 2: Watson reads a text message in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

The phenomenon under investigation in this article is by no means easy to define. Already it has inspired neologisms, word mashes and acronyms including TELOP (television optical projection), ‘impact captioning’ (Sasamoto, 2014), ‘decotitles’ (Kofoed, 2011), ‘beyond screen text messaging’ (Zhang 2014) and ‘authorial titling’ (Pérez González, 2012). While slight differences in meaning separate such terms from one another, the on-screen text in Sherlock fits all. Hence, in this discussion, I alternate between them and often default to more general terms like ‘titling’ and ‘on-screen text’ for their wide applicability across viewing devices and subject matter. This approach preserves the terminological ambiguity that attaches to this phenomenon instead of seeking to solve it, finding it symptomatic of the rapid rate of technological development with which it engages. Whatever term is decided upon today could well be obsolete tomorrow. Additionally, as Rick Altman (2004: 16) notes in his ‘crisis historiography’ of silent and early sound film, the “apparently innocuous process of naming is actually one of culture’s most powerful forms of appropriation.” He argues that in the context of new technologies and the representational codes they engender, terminological variance and confusion signals an identity crisis “reflected in every aspect of the new technology’s socially defined existence” (19).

According to the write-ups, phone messaging is the hero of BBC’s updated and rebooted Sherlock adaptation. Almost all the press garnered around Sherlock’s on-screen text links this strategy to mobile phone ‘texting’ or SMS (short messaging service). Reporting on “the storytelling challenges of a world filled with unglamorous smartphones, texting and social media”, The Wall Street Journal’s Rachel Dodes (2013) credits Sherlock with solving this dilemma and establishing a new convention for depicting texting on the big screen, creatively capturing “the real world’s digital transformation of everyday life.” For Mariel Calloway (2013), “Sherlock is honest about the role of technology and social media in daily life and daily thought… the seamless way that text messages and internet searches integrate into our lives.” Wired’s Graeme McMillan (2014) ups the ante, naming Sherlock a “new take” on “television drama as a whole” due precisely to its on-screen texting technique that sets it apart from other “tech-savvy shows out there”. McMillan continues, that “as with so many aspects of Sherlock, there’s an element of misdirection going on here, with the fun, eye-catching slickness of the visualization distracting from a deeper commentary the show is making about its characters relationship with technology – and, by extension, our own relationship with it, as well.”

As this flurry of media attention makes clear, praise for Sherlock’s on-screen text or texting firmly anchors this strategy to technology and its newly evolving forms, most notably the iPhone or smartphone. Appearing consistently throughout the series’ three seasons to date, on-screen text in Sherlock occurs in a plain, uniform white sans-serif font that appears unadorned over the screen image, obviously added during post-production. This text is superimposed, pure and simple, relying on neither text bubbles nor coloured boxes nor sender ID’s to formally separate it from the rest of the image area. As Michele Tepper (2011) eloquently notes, by utilising text in this way, Sherlock “is capturing the viewer’s screen as part of the narrative itself”:

It’s a remarkably elegant solution from director Paul McGuigan. And it works because we, the viewing audience, have been trained to understand it by the last several years of service-driven, multi-platform, multi-screen applications. Last week’s iCloud announcement is just the latest iteration of what can happen when your data is in the cloud and can be accessed by a wide range of smart-enough devices. Your VOIP phone can show caller ID on your TV; your iPod can talk to both your car and your sneakers; Twitter is equally accessible via SMS or a desktop application. It doesn’t matter where or what the screen is, as long as it’s connected to a network device. … In this technological environment, the visual conceit that Sherlock’s text message could migrate from John Watson’s screen to ours makes complete and utter sense.

Unlike on-screen text in Glee (Fox, 2009–), for instance (see Fig. 3), that is used only occasionally in episodes like ‘Feud’ (Season 4, Ep 16, March 14, 2013), Sherlock flaunts its on-screen text as signature. Its consistently interesting textual play helps to give the series cohesion. Yet, just as it aids in characterisation, helps to progress the narrative, and binds the series as a whole, it also, necessarily, remains at somewhat of a remove, as an overtly post-production effect.

Figure 3

Figure 3: Ryder chats online in ‘Feud’, Glee (2013), Episode 16, Season 4.

While Tepper (2011) explains how Sherlock’s “disembodied” (Banks, 2014) texting ‘makes sense’ in the age of cross-platform devices and online clouds, this argument falters when the on-screen text in question is less overtly technological. The extradiegetic nature of this on-screen text – so obviously a ‘post’ effect – is brought to the fore when it is used to render thoughts and emotions rather than technological interfacing. In ‘A Study in Pink’, a large proportion of the text that pops up intermittently on-screen functions to represent Sherlock’s interiority, not his Internet prowess. In concert with camera angles and “microscopic close-ups”, it elucidates Sherlock’s forensic “mind’s eye” (Redmond, Sita and Vincs, this issue), highlighting clues and literally spelling out their significance (see Figs. 4 and 5). The fact that these human-coded moments of titling have received far less attention in the press than those that more directly index new technologies is fascinating in itself, revealing the degree to which praise for Sherlock’s on-screen text is invested in ideas of newness and technological innovation – underlined by the predilection for neologisms.

Figure 4

Figures 4: Sherlock examines the pink lady’s ring in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Figure 5

Figures 5: Sherlock examines the pink lady’s ring in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Of course, even when not attached to smartphones or data retrieval, Sherlock’s deployment of on-screen text remains fresh, creative and playful and still signals perceptual shifts resulting from technological transformation. Even when representing Sherlock’s thoughts, text flashes on screen manage to recall the excesses of the digital, when email, Facebook and Twitter ensconce us in streams of endlessly circulating words, and textual pop-ups are ubiquitous. Nevertheless, the blinkered way in which Sherlock’s on-screen text is repeatedly framed as, above all, a means of representing mobile phone texting functions to conceal some of its links to older, more conventional forms of titling and textual intervention, from silent-era intertitles to expository titles to subtitles. By relentlessly emphasising its newness, much discussion of Sherlock’s on-screen text overlooks links to a host of related past and present practices. Moreover, Sherlock’s textual play actually invites a rethinking of these older, ongoing text-on-screen devices.

Reading, Watching, Listening

As Szarkowska and Kruger (this issue) explain, research into subtitle processing builds upon earlier eye tracking studies on the reading of static, printed text. They proceed to detail differences between subtitle and ‘regular’ reading, in relation to factors like presentation speed, information redundancy, and sensory competition between different multimodal channels. Here, I focus on differences between saccadic or scanning movements and fixations, in order to compare data across the screen and translation fields. During ‘regular’ reading (of static texts) average saccades last 20 to 50 milliseconds (ms) while fixations range between 100 and 500ms, averaging 200 to 300ms (Rayner, 1998). Referencing pioneering studies into subtitle processing by Géry d’Ydewalle and associates, Szarkowska et al. (2013: 155) note that “when reading film subtitles, as opposed to print, viewers tend to make more regressions” and fixations tend to be shorter. Regressions occur when the eye returns to material that has already been read, and Rayner (1998: 393) finds that slower readers (of static text) make more regressions than faster readers. A study by d’Ydewalle and de Bruycker (2007: 202) found “the percentage of regressions in reading subtitles was globally, among children and adults, much higher than in normal text reading.” They also report that mean fixation durations in the subtitles was shorter, at 178 ms (for adults) and explain that subtitle regressions (where the eye travels back across words already read) can be partly explained by the “considerable information redundancy” that occurs when “[s]ubtitle, soundtrack (including the voice and additional information such as intonation, background noise, etc.), and image all provide partially overlapping information, eliciting back and forth shifts with the image and more regressive eye-movements” (202).

What happens to saccades and fixations when image processing is brought into the mix? When looking at static images, average fixations last 330 ms (Rayner, 1998). This figure is slightly longer than average fixations during regular reading and longer again than average subtitle fixations. Szarkowska and Kruger (this issue) note that “reading requires many successive fixations to extract information whereas looking at a scene requires fewer, but longer fixations” that tend to be more exploratory or ambient in nature, taking in a greater area of focus. In relation to moving-images, Smith (2013: 168) finds that viewers take in roughly 3.8% of the total screen area during an average length shot. Peripheral processing is at play but “is mostly reserved for selecting future saccade targets, tracking moving targets, and extracting gist about scene category, layout and vague object information”. In thinking about these differences in regular reading behaviour, screen viewing, and subtitle processing, it is noticeable that with subtitles, distinctions between fixations and saccades are less clear-cut. While saccades last between 20 and 50ms, Smith (2013: 169) notes that the smallest amount of time taken to perform a saccadic eye movement (taking into account saccadic reaction time) is 100-130ms. Recalling d’Ydewalle and de Bruycker’s (2007: 202) finding that fixations during subtitle processing last around 178ms, it would seem that subtitle conditions blur the boundaries somewhat between saccades and fixations, scanning and reading.

Interestingly, studies have also shown that the processing of two-line subtitles involves more regular word-by-word reading than for one-liners (D’Ydewalle and de Bruycker, 2007: 199). D’Ydewalle and de Bruycker (2007: 199) report, for instance, that more words are skipped and more regressions occur for one-line subtitles than for two-line subtitles. Two-line subtitles result in a larger proportion of time being spent in the subtitle area, and occasion more back-and-forth shifts between the subtitles and the remaining image area (201). This finding suggests that the processing of one-line subtitles differs considerably from regular reading behaviour. D’Ydewalle and de Bruycker (2007: 202) surmise that the distinct way in which one-line subtitles are processed relates to a redundancy effect caused by the multimodal nature of screen media. Noting how one-line subtitles often convey short exclamations and outcries, they suggest that a “standard one-line subtitle generally does not provide much more information than what can already be extracted from the picture and the auditory message.” They conclude that one-line subtitles occasion “less reading” than two-line subtitles (202). Extrapolating further, I posit that the routine overlapping of information that occurs in subtitled screen media blurs lines between reading and watching. One-line subtitles are ‘read’ irregularly and partly blind – that is, they are regularly skipped and processed through saccadic eye movements rather than fixations.

This suggestion is supported by data on subtitle skipping. Szarkowska and Kruger (this issue) find that longer subtitles containing frequently used words are easier and quicker to process than shorter subtitles containing low-frequency words. Hence, they conclude that cognitive load relates more to word familiarity than quantity, something that is overlooked in many professional subtitling guidelines. This finding indicates that high-frequency words are processed ‘differently’ in subtitling than in static text, in a manner more akin to visual recognition or scanning than reading. Szarkowska and Kruger find that high-frequency words in subtitles are often skipped. Hence, as with one-line subtitles, high-frequency words are, to a degree, processed blind, possibly through shape recognition and mapping more than durational focus. In relation to other types of on-screen text, such as the short, free-floating type that characterises Sherlock, it seems entirely possible that this innovative mode of titling may just challenge distinctions between text and image processing. While commentators laud this series for the way it integrates on-screen text into its narrative, style and characterisation, eye tracking is required to unpack the cognitive implications of Sherlock’s text/image morph.

The Pink Lady

Figure 6

Figure 6: Letters scratched into the floor in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

Sherlock producer Vertue refers to the pink lady scene in ‘A Study in Pink’ as particularly noteworthy for its “text all around the screen”, referring to it as the “best use” of on-screen text in the series (qtd in McMillan, 2014). In this scene, a dead woman dressed in pink lies face first on the floor of a derelict building into which she has painstakingly etched a word or series of letters (‘Rache’) with her fingernails. As Sherlock investigates the crime scene, forensics officer Anderson interrupts to explain that ‘Rache’ is the German word for ‘revenge’. The German-to-English translation pops up on screen (see Fig. 6), and this time Sherlock sees it too. This superimposed text, so obviously laid over the image, oversteps its surface positioning to enter Sherlock’s diegetic space, and we next view it backwards, from Sherlock’s point of view, not ours (see Fig. 7). After an exasperated eye roll that signals his disregard for Anderson, Sherlock dismisses this textual intervention and we watch it swirl into oblivion. Here, on-screen text is at once both inside and outside the narrative, diegetic and extra-diegetic, informative and affecting. In this way it self-reflexively draws attention to the show’s narrative framing, demonstrating its complexity as distinct diegetic levels merge.

Figure 7

Figure 7: Sherlock sees on-screen text in ‘A Study in Pink’, Sherlock (2010), Episode 1, Season 1.

For Carol O’Sullivan (2011), when on-screen text affords this type of play between the diegetic and extra-diegetic it functions as an “extreme anti-naturalistic device” (166) that she unpacks via Gérard Genette’s notion of narrative metalepsis (164). Detailing numerous examples of humourous, formally transgressive diegetic subtitles, such as those found in Annie Hall (Woody Allen, 1977) (Fig. 8), O’Sullivan points to their metatextual function, referring to them as “metasubtitles” (166) that implicitly comment on the limits and nature of subtitling itself. When Sherlock’s on-screen titles oscillate between character and viewer point-of-view shots, they too become metatextual, demonstrating, in Genette’s terms, “the importance of the boundary they tax their ingenuity to overstep in defiance of verisimilitude – a boundary that is precisely the narrating (or the performance) itself: a shifting but sacred frontier between two worlds, the world in which one tells, the world of which one tells” (qtd in O’Sullivan 2011: 165). Moreover, for O’Sullivan, “all subtitles are metatextual” (166) necessarily foregrounding their own act of mediation and interpretation. Specifically linking such ideas to Sherlock, Luis Perez Gonzalez (2012: 18) notes how “the series creators incorporate titles that draw attention to the material apparatus of filmic production”, thereby creating an complex alienation-attraction effect “that shapes audience engagement by commenting upon the diegetic action and disrupting conventional forms of semiotic representation, making viewers consciously work as co-creators of media content.”

Figure 8

Figure 8: Subtitled thoughts in the balcony scene, Annie Hall (1977).

Eye Bias

One finding from subtitle eye tracking research particularly pertinent to Sherlock is the notion that on-screen text causes eye bias. This was established in various studies conducted by d’Ydewalle and associates, which found that subtitle processing is largely automatic and obligatory. D’Ydewalle and de Bruycker (2007: 196) state:

Paying attention to the subtitle at its presentation onset is more or less obligatory and is unaffected by major contextual factors such as the availability of the soundtrack, knowledge of the foreign language in the soundtrack, and important episodic characteristics of actions in the movie: Switching attention from the visual image to “reading” the subtitles happens effortlessly and almost automatically (196).

This point is confirmed by Bisson et al. (2014: 399) who report that participants read subtitles even in ‘reversed’ conditions – that is, when subtitles are rendered in an unfamiliar language and the screen audio is fully comprehensible (in the viewers’ first language) (413). Again, in intralingual or same-language subtitling – when titles replicate the language spoken on screen –hearing audiences still divert to the subtitle area (413). These findings indicate that viewers track subtitles irrespective of language or accessibility requirements. In fact, the tracking of subtitles overrides function. As Bisson et al. (413) surmise, “the dynamic nature of the subtitles, i.e., the appearance and disappearance of the subtitles on the screen, coupled with the fact that the subtitles contained words was enough to generate reading behavior”.

Szarkowska and Kruger (this issue) reach a similar conclusion, explaining eye bias towards subtitles in terms of both bottom-up and top-down impulses. When subtitles or other forms of text flash up on screen, they affect a change to the scene that automatically pulls our eyes. The appearance and disappearance of text on screen is registered in terms of motion contrast, which according to Smith (2013: 176), is the “critical component predicting gaze behavior”, attaching to small movements as well as big. Additionally, we are drawn to words on screen because we identify them as a ready source of relevant information, as found in Batty et al. (forthcoming). Analysing a dialogue-free montage sequence from animated feature Up (Pete Docter, 2009), Batty et al. found that on-screen text in the form of signage replicates in miniature how ‘classical’ montage functions as a condensed form of storytelling aiming for enhanced communication and exposition. They suggest that montage offers a rhetorical amplification of an implicit intertitle, thereby alluding to the historical roots of text on screen while underlining its narrative as well as visual salience. One frame from the montage sequence focuses in close-up on a basket containing picnic items and airline tickets (see Fig. 9). Eye tracking tests conducted on twelve participants indicates a high degree of attentional synchrony in relation to the text elements of the airline ticket on which Ellie’s name is printed. Here, text provides a highly expedient visual clue as to the narrative significance of the scene and viewers are drawn to it precisely for its intertitle-like, expository function, highlighting the top-down impulse also at play in the eye bias caused by on-screen text.

Figure 9

Figure 9: Heat map showing collective gaze weightings during the montage sequence in Up (2009).

In this image from Up, printed text appears in the centre of the frame and, as Smith (2013: 178) elucidates, eyes are instinctively drawn towards frame centre, a finding backed up by much subtitle research (see Skarkowska and Kruger, this issue). However, eye tracking results on Sherlock conducted by Redmond, Sita and Vincs (this issue) indicate that viewers also scan static text when it is not in the centre of the frame. In an establishing shot of 221B Baker Street from the first episode of Sherlock’s second season, ‘A Scandal in Belgravia’, viewers track static text that borders the frame across its top and right hand sides, again searching for information (See Fig. 10). Hence, the eye-pull exerted by text is noticeable even in the absence of movement, contrast and central framing. In part, viewers are attracted to text simply because it is text – identified as an efficient communication mode that facilitates speedy comprehension (see Lavaur, 2011: 457).

Figure 10

Figure 10: Single viewer gaze path for ‘A Scandal in Belgravia’, Sherlock (2012), Episode 1, Season 2.

Distraction/Attraction

What do these eye tracking results across screen and translation studies tell us about Sherlock’s innovative use of on-screen text and texting? Based on the notion that text on screen draws the eye in at least dual ways, due to both its dynamic/contrastive nature and its communicative expediency, we can surmise that for Sherlock viewers, on-screen text is highly visible and more than likely to be in that 3.8% of the screen on which they will focus at any one point in time (see Smith, 2013: 168). The marked eye bias caused by text on screen is further accentuated in Sherlock by the freshness of its textual flashes, especially for English-speaking audiences given the language hierarchies of global screen media (see Acland 2012, UNESCO 2013). The small percentage of foreign-language media imported into most English-speaking markets tends to result in a lack of familiarity with subtitling beyond niche audience segments. For those unfamiliar with subtitling or captioning, on-screen text appears particularly novel. Additionally, as explored, floating TELOPs in Sherlock attract attention due to the complex functions they fulfil, providing narrative and character clues as well as textual and stylistic cohesion. As Tepper (2011) points out, in the first episode of the series, viewers are introduced to Sherlock’s character via text, before seeing him on screen. “When he texts the word ‘Wrong!’ to DI Lestrade and all the reporters at Lestrade’s press conference,” notes Tepper, “the technological savvy and the imperiousness of tone tell you most of what you need to know about the character.”

There seems no doubt that on-screen text in Sherlock attracts eye movement, and that it therefore distracts from other parts of the image. One question then that immediately presents itself is why Sherlock’s textual distractions are tolerated – even celebrated – to a far greater extent than other, more conventional or routine forms of titling like subtitles and captions. While Sherlock’s on-screen text is praised as innovative and incisive, interlingual subtitling and SDH are criticised by detractors for the way in which they supposedly force viewers to read rather than watch, effectively transforming film into “a kind of high-class comic book with sound effects” (Canby, 1983).[2] Certainly, differences in scale affect such attitudes and the quantitative variance between post-subtitles (produced for distribution only) and authorial or diegetic titling (as seen in Sherlock) is pronounced.[3] However, eye tracking research on subtitle processing indicates that, on the whole, viewers easily accommodate the increased cognitive load it presents. Although attentional splitting occurs, leading to an increase in back-and-forth shifts between the subtitles and the rest of the image area (Skarkowska and Kruger, this issue), viewers acclimatise by making shorter fixations than in regular reading and by skipping high-frequency words and subtitles while still managing to register meaning (see d’Ydewalle and de Bruycker, 2007: 199). In this way, subtitle processing reveals many differences to reading of static text, and approximates techniques of visual scanning. Bearing these findings in mind, I propose it is more accurate to see subtitling as transforming reading into viewing and text into image, rather than vice versa.

Situating Sherlock in relation to a range of related TELOP practices across diverse TV genres (such as game shows, panel shows, news broadcasting and dramas) Ryoko Sasamoto (2014: 7) notes that the additional processing effort caused by on-screen text is offset by its editorial function.[4] TELOPs are often deployed by TV producers to guide interpretation and ensure comprehension by selecting and highlighting information deemed most relevant. This suggestion is backed up by research by Rei Matsukawa et al. (2009), which found that the information redundancy effect caused by TELOPs facilitates understanding of TV news. For Sasamoto (2014: 7), ‘impact captioning’ highlights salient information in much the same way as voice intonation or contrastive stress. It acts as a “written prop on screen” enabling “TV producers to achieve their communicative aims… in a highly economical manner” (8). Focusing on Sherlock specifically, Sasamoto suggests that its captioning provides “a route for viewers into complex narratives” (9). Moreover, as Szarkowska and Kruger (this issue) note, in static reading conditions, “longer fixations typically reflect higher cognitive load.” Consequently, the shorter fixations that characterise subtitle viewing supports the contention that on-screen text processing is eased by its expedient, editorial function and by redundancy effects resulting from its multimodality.

Switched On

Another way in which Sherlock’s text and titling innovations extend beyond mobile phone usage was exemplified in July 2013 by a promotional campaign that promised viewers a ‘sneak peak’ at a yet-to-be-released episode title, requiring them to find and piece together a series of clues. In true Sherlockian style, the clues were well hidden, only visible to viewers if they switched on closed-captioning or SDH available for deaf and hard-of-hearing audiences. With this device turned on, viewers encountered intralingual captioning along the bottom of their screen and additionally, individually boxed letters that appeared top left (see Figs. 11 and 12). Viewers needed to gather all these single letter clues in order to deduce the episode title: ‘His Last Vow’. According to the ‘I Heart Subtitles’ blog (July 16, 2013), in doing so, Sherlock once again displayed its ability to “think outside the box and consider all…options”. It also cemented its commitment to on-screen text in various guises, and effectively gave voice to an audience segment typically disregarded in screen commentary and analysis. Through this highly unusual, cryptic campaign, Sherlock alerted viewers to more overtly functional forms of titling, and intimated points of connection between language, textual intervention and access.

Figure 11

Figures 11: Boxed letter clues (top left of frame) that appeared when closed captioning was switched on, during a re-run of ‘A Scandal in Belgravia’, Sherlock (2012), Episode 1, Season 2.

Figure 12

Figures 12: Boxed letter clues (top left of frame) that appeared when closed captioning was switched on, during a re-run of ‘A Scandal in Belgravia’, Sherlock (2012), Episode 1, Season 2.

Conclusion

On-screen text invites a rethinking of the visual, expanding its borders and blurring its definitional clarity. Eye tracking research demonstrates that moving text on screens is processed differently to static text, affected by a range of factors issuing from its multimodal complexity. Sherlock subtly signals such issues through its playful, irreverent deployment of text, which enables viewers to directly access Sherlock’s thoughts and understand his reasoning, while also distancing them, asking them to marvel at his ‘millennial’ technological prowess (Stein and Busse, 2012: 11) while remaining self-consciously aware of his complex narrative framing as it flips inside out, inviting audiences to watch themselves watching. Such diegetic transgression is yet to be mapped through eye tracking, intimating a profitable direction for future studies. To date, data on text and image processing demonstrates how on-screen text attracts eye movement and hence, it can be inferred that it distracts from other parts of the image area. Yet, despite rendering more of the image effectively ‘invisible’, text in the form of TELOPs are increasingly prevalent in news broadcasts, current affairs panel shows (when audience text messages are displayed) and, most notably, in Asian TV genres where they are now a “standard editorial prop” featured in many dramas and game shows (Sasamoto, 2014: 1). In order to take up the challenge presented by such emerging modes of screen address, research needs to move beyond surface assessments of the attraction/distraction nexus. It is the very attraction to TELOP distraction that Sherlock – via eye tracking – brings to the fore.

 

References

Acland, Charles. 2012. “From International Blockbusters to National Hits: Analysis of the 2010 UIS Survey on Feature Film Statistics.” UIS Information Bulletin 8: 1-24. UNESCO Institute for Statistics.

Altman, Rick. 2004. Silent Film Sound. New York: Columbia University Press.

Banks, David. 2012. “Sherlock: A Perspective on Technology and Story Telling.” Cyborgology, January 25. Accessed October 9, 2014.

Batty, Craig, Adrian Dyer, Claire Perkins and Jodi Sita (forthcoming). “Seeing Animated Worlds: Eye Tracking and the Spectator’s Experience of Narrative.” In Making Sense of Cinema: Empirical Studies into Film Spectators and Spectatorship, edited by Carrie Lynn D. Reinhard and Christopher J. Olson. London and New York: Bloomsbury.

Bennet, Alannah. 2014. “From Sherlock to House of Cards: Who’s Figured Out How to Translate Texting to Film.” Bustle, August 18. Entertainment. Accessed October 9. http://www.bustle.com/articles/36115-from-sherlock-to-house-of-cards-whos-figured-out-how-to-translate-texting-to-film/image/36115.

Biedenharn, Isabella. 2014. “A Brief Visual History of On-Screen Text Messages in Movies and TV.” Flavorwire, April 24. Accessed October 13.

Bisson, Marie-Jos´ee, Walter J. B. Van Heuven, Kathy Conklin And Richard J. Tunney. 2014. “Processing of native and foreign language subtitles in films: An eye tracking study.” Applied Psycholinguistics 35: 399–418. Accessed October 13, 2014. doi: 10.1017/S0142716412000434.

Calloway, Mariel. 2013. “The Game is On(line): BBC’s ‘Sherlock’ in the Age of Social Media” Mariel Calloway, March 8. Accessed October 14, 2014.

Canby, Vincent. 1983. “A Rebel Lion Breaks Out.” New York Times, March 27, 21.

Dodes, Rachel. 2013. “From Talkies to Texties.” Wall Street Journal, April 4, Arts and Entertainment Section. Accessed October 13, 2014.

d’Ydewalle, Géry and Wim De Bruycker, 2007. “Eye movements of children and adults while reading television subtitles.” European Psychologist 12 (3): 196-205.

Kofoed, D. T. 2011. “Decotitles, the Animated Discourse of Fox’s Recent Anglophonic Internationalism.” Reconstruction 11 (1). Accessed October 5, 2012.

Lavaur, Jean-Marc and Dominic Bairstow. 2011. “Languages on the screen: Is film comprehension related to the viewers’ fluency level and to the language in the subtitles?” International Journal of Psychology 46 (6): 455-462. doi: 10.1080/00207594.2011.565343.

McMillan, Graeme. 2014. “Sherlock’s Text Messages Reveal Our Transhumanism” Wired UK, February 3. Accessed October 14.

Matsukawa, Rei, Yosuke Miyata and Shuichi Ueda. 2009. “Information Redundancy Effect on Watching TV News: Analysis of Eye Tracking Data and Examination of the Contents.” Literary and Information Science 62: 193-205.

O’Sullivan, Carol. 2011. Translating Popular Film. Basingstoke and New York: Palgrave Macmillan.

Pérez González, Luis. 2013. “Co-Creational Subtitling in the Digital Media: Transformative and Authorial Practices.” International Journal of Cultural Studies 16 (1): 3-21. Accessed September 25, 2014. doi: 10.1177/1367877912459145.

Rayner, K. 1998. “Eye Movements in Reading and Information Processing: 20 Years of Research.” Psychological Bulletin 124: 372-422.

Redmond, Sean, Jodi Sita and Kim Vincs. 2015. “Our Sherlockian Eyes: The Surveillance of Vision” Refractory: a Journal of Entertainment Media, 25.

Romero-Fresco, Pablo. 2013. “Accessible filmmaking: Joining the dots between audiovisual translation, accessibility and filmmaking.” JoSTrans: The Journal of Specialised Translation 20: 201-23. Accessed September 20, 2014.

Sasamoto, Ryoko. 2014. “Impact caption as a highlighting device: Attempts at viewer manipulation on TV.” Discourse, Context and Media 6: 1-10. Accessed September 18 (Article in Press). doi: 10.1016/j.dcm.2014.03.003.

Schrodt, Paul. 2013. “This is How to Shoot Text Messaging” Esquire, February 4. The Culture Blog. Accessed October 13, 2014.

Smith, Tim J. 2013. “Watching You Watch Movies: Using Eye Tracking to Inform

Cognitive Film Theory” in Psychocinematics: Exploring Cognition at the Movies, edited by Arthur P. Shimamura, 165-91. Oxford and New York: Oxford University Press. Accessed October 7, 2014. doi: http://dx.doi.org/10.1093/acprof:oso/9780199862139.001.0001.

Stein, Louise Ellen and Kristina Busse. 2012. “Introduction: The Literary, Televisual and Digital Adventures of the Beloved Detective.” In Sherlock and Transmedia Fandom: Essays on the BBC Series, edited by Louise Ellen Stein and Kristina Busse, 9-24. Jefferson: McFarland and Company.

Szarkowska, Agnieszka et. al. 2013. “Harnessing the Potential of Eye-Tracking for Media Accessibility.” in Translation Studies and Eye-Tracking Analysis, edited by Sambor Grucza, Monika Płużyczka and Justyna Zając, 153-83. Frankfurt am Mein: Peter Lang.

Szarkowska, Agnieszka and Jan Louis Kruger. 2015. “Subtitles on the Moving Image: An Overview of Eye Tracking Studies.” Refractory: a Journal of Entertainment Media, 25.

Tepper, Michele. 2011. “The Case of the Travelling Text Message.” Interactions Everywhere, June 14. Accessed October 14, 2014.

UNESCO. 2013. “Feature Film Diversity”, UIS Fact Sheet 24, May. Accessed October 3, 2014.

Zhang, Sarah. 2014. “How Hollywood Figured Out A Way To Make Texting In Movies Look Less Dumb.” Gizmodo, August 18. Accessed August 19.

Zhou, Tony. 2014. “A Brief Look at Texting and the Internet in Film”. Video Essay, Every Frame a Painting, August 15. Accessed August 19.

 

List of Figures

 

 

Notes

[1] While some commentators point out that Sherlock was by no means the first to depict text messaging in this way – as floating text on screen – it is this series more than any other that has brought this phenomenon into the limelight. Other notable uses of on-screen text to depict mobile phone messaging occur in films All About Lily Chou-Chou (Iwai, 2001), Disconnect (Rubin, 2013), The Fault in Our Stars (Boone, 2014), LOL (Azuelos, 2012), Non-Stop (Collet-Serra, 2014), Wall Street: Money Never Sleeps (Stone, 2010), and in TV series Glee (Fox, 2009–), House of Cards (Netflix, 2013–), Hollyoaks (Channel 4, 1995–), Married Single Other (ITV, 2010) and Slide (Fox8, 2011). For discussion of some ‘early adopters’, see Biendenharn 2014.

 

Notes

[2] Notably, in this New York Times piece, Canby (1983) actually defends subtitling against this charge, and advocates for subtitling over dubbing.

[3] On distinctions between post-subtitling and pre-subtitling (including diegetic subtitling), see O’Sullivan (2011).

[4] According to Sasamoto (2014: 1), “the use of OCT [Open Caption Telop] as an aid for enhanced viewing experience originated in Japan in 1990.”

 

Bio

Dr Tessa Dwyer teaches Screen Studies at the University of Melbourne, specialising in language politics and issues of screen translation. Her publications have appeared in journals such as The Velvet Light Trap, The Translator and The South Atlantic Quarterly and in a range of anthologies including B is for Bad Cinema (2014), Words, Images and Performances in Translation (2012) and the forthcoming Locating the Voice in Film (2016), Contemporary Publics (2016) and the Routledge Handbook of Audiovisual Translation (2017). In 2008, she co-edited a special issue of Refractory on split screens. She is a member of the ETMI research group and is currently writing a book on error and screen translation.

Subtitles on the Moving Image: an Overview of Eye Tracking Studies – Jan Louis Kruger, Agnieszka Szarkowska and Izabela Krejtz

Abstract

This article provides an overview of eye tracking studies on subtitling (also known as captioning), and makes recommendations for future cognitive research in the field of audiovisual translation (AVT). We find that most studies in the field that have been conducted to date fail to address the actual processing of verbal information contained in subtitles, and rather focus on the impact of subtitles on viewing behaviour. We also show how eye tracking can be utilised to measure not only the reading of subtitles, but also the impact of stylistic elements such as language usage and technical issues such as the presence of subtitles during shot changes on the cognitive processing of the audiovisual text as a whole. We support our overview with empirical evidence from various eye tracking studies conducted on a number of languages, language combinations, viewing contexts as well as different types of viewers/readers, such as hearing, hard of hearing and Deaf people.

Introduction

The reading of printed text has received substantial attention from scholars since the 1970s (for an overview of the first two decades see Rayner et al. 1998). Many of these studies, conducted from a psycholinguistic angle, made use of eye tracking. As a result, a large body of knowledge exists on the eye movements during reading of people with varying levels of reading skills and language proficiency, with a range of ages, different first languages and cultural backgrounds, and in different contexts. Studies on subtitle reading, however, have not achieved the same level of scientific rigour largely for practical reasons: subtitles are not static for more than a few seconds at a time; they compete for visual attention with a moving image; and they compete for overall cognitive resources with verbal and non-verbal sounds. This article will identify some of the gaps in current research in the field, and also illustrate how some of these gaps can be bridged.

Studying the reading of subtitles is significantly different from studying the reading of static text. In the first place, as far as eye tracking software is concerned, the subtitles appear on a moving image as image rather than text, which renders traditional text-based reading statistics and software all but useless. This also makes the collection of data for reading research on subtitles a painstakingly slow process involving substantial manual inspections and coding. Secondly, the fact that subtitles appear against the background of the moving image means that they are always in competition with this image, which renders the reading process fundamentally different from the reading process of static texts: on the one hand because the reading of subtitles compete with the processing of the image, sometimes resulting in interrupted reading, but on the other hand the limited time the subtitles are on screen means that readers have less time to reread or regress to study difficult words or to check information. Either way, studying this reading process, and the cognitive processing that takes place during the reading, is much more complicated than in the case of static texts where we know that the reader is mainly focussing on the words before her/him without additional auditory and visual information to process.

While the viewing of subtitles has been the object of many eye tracking studies in recent years, with increasing frequency (see, for example Bisson et al. 2012; d’Ydewalle and Gielen 1992; d’Ydewalle and De Bruycker 2007; Ghia 2012; Krejtz et al. 2013; Kruger 2013; Kruger et al. 2013; Kruger and Steyn 2014; Perego et al. 2010; Rajendran et al. 2013; Specker 2008; Szarkowska et al. 2011; Winke et al. 2013), the study of the reading of subtitles remains a largely uncharted territory with many research avenues still to be explored. Those studies that do venture to measure more than just attention to the subtitle area, seldom do this for extended texts.

In this article we provide an overview of studies on how subtitles change the way viewers process audiovisual material, and also of studies on the unique characteristics of the subtitle reading process. Taking an analysis of the differences between reading printed (static) text and subtitles as point of departure, we examine a number of aspects typical of the way subtitle text is processed in reading. We also look at the impact of the dynamic nature of the text and the competition with other sources of information on the reading process (including scene perception, changes in the viewing process, shifts between subtitles and image, visual saliency of text, faces, and movement, and cognitive load), as well as discussing studies on the impact of graphic elements on subtitle reading (e.g. number of lines, and text chunking), and studies that attempt to measure the subtitle reading process in more detail.

We start off with a discussion of the way in which watching an audiovisual text with subtitles alters viewing behaviour as well as of the complexities of studying subtitles due to the dynamic nature of the image it has as a backdrop. Here we focus on the fleeting nature of the subtitle text, the competition between reading the subtitles and scanning the image, and the interaction between different sources of information. We further discuss internal factors that impact on subtitle processing, like the language and culture of the audience, the language of the subtitles, the degree of access the audience has to sound, and other internal factors, before turning to external factors related to the nature of the audiovisual text and the presentation of the subtitles. Finally, we provide an overview of studies attempting to measure the processing of subtitles as well as findings from two studies that approach the processing of subtitles

The dynamic nature of the subtitle reading process

Reading subtitles differs substantially from reading printed text in a number of respects. As opposed to “static text on a stable background”, the viewer of subtitled audiovisual material is confronted with “fleeting text on a dynamic background” (Kruger and Steyn 2014, 105). In consequence, viewers not only need to process and integrate information from different communication channels (verbal visual, non-verbal visual, verbal auditory, non-verbal auditory, see Gottlieb 1998), but they also have no control over the presentation speed (see Kruger and Steyn 2014; Szarkowska et al. forthcoming). As a consequence, unlike in the reading of static texts, the pace of reading is in part dictated by the text rather than the reader – by the time the text is available to be read – and there is much less time for the reader to regress to an earlier part of a sentence or phrase, and no opportunity to return to previous sentences. Reading traditionally takes place in a limited window which the reader is acutely aware will disappear in a few seconds. Even though there are exceptions to the level of control a viewer has, for example in the case of DVD and PVR as well as other electronic media where the viewer can rewind and forward at will, the typical viewing of subtitles for most audiovisual products happens continuously and without pauses just as when watching live television.

Regressions, which form an important consideration in the reading of static text, take on a different aspect in the context of the knowledge (the viewer has) that dwelling too much on any part of a subtitle may make it difficult to finish reading the subtitle before it disappears. Any subtitle is on screen for between one and six seconds, and the viewer also has to simultaneously process all the other auditory (in the case of hearing audiences) and visual cues. In other words, unlike when reading printed text, reading becomes only one of the cognitive processes the viewer has to juggle in order to understand the audiovisual text as a whole. Some regressions are in fact triggered by the change of the image in shot changes (and to a much lesser extent scene changes) when the text stays on across these boundaries, which means that the viewer sometimes returns to the beginning of the subtitle to check whether it is a new subtitle, and sometimes even re-reads the subtitle. For example, in a recent study, Krejtz et al. (2013) established that participants tend not to re-read subtitles after a shot change or cut. But their data also revealed that a proportion of the participants did return their gaze to the beginning of the subtitle after such a change (see also De Linde and Kay, 1999). What this means for the study of subtitle reading is that these momentary returns (even if only for checking) result in a class of regressions that is not in fact a regression to re-read a word or section, but rather a false initiation of reading for what some viewers initially perceive to be a new sentence.

On the positive side, the fact that subtitles are embedded on a moving image and are accompanied by a soundtrack (in the case of hearing audiences) facilitates the processing of language in context. Unfortunately, this context also introduces competition for attention and cognitive resources. For the Deaf and hard of hearing audience, attention has to be divided between reading the subtitles and processing the scene, extracting information from facial expressions, lip movements and gestures, and matching or checking this against the information obtained in the subtitles. For the hearing audience who makes use of subtitles for support or to provide access to foreign language dialogue, attention is likewise divided between subtitles and the visual scene, and just as the Deaf and hard of hearing audiences have the added demand on their cognitive resources of having to match what they read with what they get from non-verbal signs and lip movements, the hearing audience matches what they read with what they hear, checking for correspondence of information and interpreting intonation, tenor and other non-verbal elements of speech.

What stands beyond doubt is that the appearance of subtitles changes the viewing process. In 2000, Jensema et al. famously stated that “the addition of captions to a video resulted in major changes in eye movement patterns, with the viewing process becoming primarily a reading process” (2000a, 275). Having examined the eye movements of six subjects watching video clips with and without subtitles, they found that the onset of a subtitle triggers a change in the eye movement pattern: when a subtitle appears, viewers move their gaze from whatever they were watching in order to follow the subtitle. In a more wide-scale study it was concluded by d’Ydewalle and de Bruycker (2007,196) that “paying attention to the subtitle at its presentation onset is more or less obligatory and is unaffected by major contextual factors such as the availability of the soundtrack, knowledge of the foreign language in the soundtrack, and important episodic characteristics of actions in the movie: Switching attention from the visual image to “reading” the subtitles happens effortlessly and almost automatically”.

Subtitles therefore appear to be the cause of eye movement bias similar to faces (see Hershler & Hochstein, 2005; Langton, Law, Burton, & Schweinberger, 2008; Yarbus, 1967), the centre of the screen, contrast and movement. In other words, subtitles attract the gaze at least in part because of the fact that the eye is drawn to the words on screen just as the eye is drawn to movement and other elements. Eyes are drawn to subtitles not only because the text is identified as a source of meaningful information (in other words a top-down impulse as the viewer consciously consults the subtitles to obtain relevant information), but also because of the change to the scene that the appearance of a subtitle causes (in other words a bottom-up impulse, automatically drawing the eyes to what has changed on the screen).

As in most other contexts, the degree to which viewers will process the subtitles (i.e. read them rather than merely look at them when they appear and then look away) will be determined by the extent to which they need the subtitles to follow the dialogue or to obtain information on relevant sounds. In studying visual attention to subtitles it therefore remains a priority to measure the degree of processing, something that has not been done in more than a handful of studies, and something to which we will return later in the article.

Viewers usually attend to the image on the screen, but when subtitles appear, it only takes a few frames for most viewers to move their gaze to read the subtitles. The fact that people tend to move their gaze to subtitles the moment they appear on the screen is illustrated in Figures 1 and 2.

Figure. 1 Heat maps of three consecutive film stills – Polish news programme Fakty (TVN) with intralingual subtitles.

Figure. 1 Heat maps of three consecutive film stills – Polish news programme Fakty (TVN) with intralingual subtitles.

Figure 2. Heat maps of two consecutive film stills – Polish news programme Wiadomości (TVP1) with intralingual subtitles

Figure 2. Heat maps of two consecutive film stills – Polish news programme Wiadomości (TVP1) with intralingual subtitles

Likewise, when the gaze of a group of viewers watching an audiovisual text without subtitles is compared to that of a similar group watching the same text with subtitles, the split in attention is immediately visible as the second group reads the subtitles and attends less to the image, as can be seen in Figure 3.

Figure 3. Heat maps of the same scene seen without subtitles and with subtitles – recording of an academic lecture.

Figure 3. Heat maps of the same scene seen without subtitles and with subtitles – recording of an academic lecture.

Viewer-internal factors that impact on subtitle processing

The degree to which the subtitles are processed is far from straightforward. In a study performed at a South African university in the context of Sesotho students looking at a recorded lecture with subtitles in their first language and audio in English (their language of instruction), students were found to avoid looking at the subtitles (see Kruger, Hefer and Matthew, 2013b). Sesotho students in a different group who saw the same lecture with English subtitles processed the subtitles to a much larger extent. This contrast is illustrated in the focus maps in Figures 4.

4-Kruger

Figure 4. Focus maps of Sesotho students looking at a lecture in intralingual English subtitles (left) and another group looking at the same lecture with interlingual Sesotho subtitles (right) – recording of an academic lecture.

The difference in eye movement behaviour between the conditions is also evident when considering the number of subtitles skipped. Participants in the above study who saw the video with Sesotho subtitles skipped an average of around 50% of the Sesotho subtitles (median at around 58%), whereas participants who saw the video with English subtitles only skipped an average of around 20% of the English subtitles (with a median of around 8%) (see Kruger, Hefer & Matthew, 2014).

This example does not, however, represent the conventional use of subtitles where viewers would rely on the subtitles to gain access to a text from which they would have been excluded without the subtitles. It does serve to illustrate that subtitle reading is not unproblematic and that more research is needed on the nature of processing in different contexts by different audiences. For example, in a study in Poland, interlingual subtitles (English to Polish) were skipped slightly less often by hearing viewers compared to intralingual subtitles (Polish to Polish), possibly because hearing viewers didn’t need them to follow the plot (see Szarkowska et al., forthcoming).

Another important finding from eye tracking studies on the subtitle process relates to how viewers typically go about reading a subtitle. Jensema et al. (2000) found that in subtitled videos, “there appears to be a general tendency to start by looking at the middle of the screen and then moving the gaze to the beginning of a caption within a fraction of a second. Viewers read the caption and then glance at the video action after they finish reading” (2000, 284). This pattern is indeed often found, as illustrated in the sequence of frames from a short video from our study in Figure 5.

Figure 5. Sequence of typical subtitle reading – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

Figure 5. Sequence of typical subtitle reading – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

Some viewers, however, do not read so smoothly and tend to shift their gaze between the image and the subtitles, as demonstrated in Figure 6. The gaze shifts between the image and the subtitle, also referred to in literature as ‘deflections’ (de Linde and Kay 1999) or ‘back-and-forth shifts’ (d’Ydewalle and De Bruycker (2007), can be regarded as an indication of the smoothness of the subtitle reading process: the fewer the gaze shifts, the more fluent the reading and vice versa.

Figure 6. Scanpath of frequent gaze shifting between text and image – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

Figure 6. Scanpath of frequent gaze shifting between text and image – a recording of Polish news programme Fakty (TVN) with intralingual subtitles.

An important factor that influences subtitle reading patterns is the nature of the audience. In Figure 7 an interesting difference is shown between the way a Deaf and a hard of hearing viewer watched a subtitled video. The Deaf viewer moved her gaze from the centre of the screen to read the subtitle and then, after having read the subtitle, returned the gaze to the centre of the screen. In contrast, the hard of hearing viewer made constant comparisons between the subtitles and the image, possibly relying on residual hearing and trying to support the subtitle reading process with lip-reading. Such a result was reported by Szarkowska et al. (2011), who found differences in the number of gaze shifts between the subtitles and the image in the verbatim subtitles condition, particularly discernible (and statistically significant) in the hard of hearing group (when compared to the hearing and Deaf groups).

Figure 7. Scanpaths of Deaf and hard of hearing viewers. Left: Gaze plot illustrating the viewing pattern of a Deaf participant watching a clip with verbatim subtitles.  Right: Gaze plot illustrating the viewing pattern of a hard of hearing participant watching a clip with verbatim subtitles.

Figure 7. Scanpaths of Deaf and hard of hearing viewers. Left: Gaze plot illustrating the viewing pattern of a Deaf participant watching a clip with verbatim subtitles. Right: Gaze plot illustrating the viewing pattern of a hard of hearing participant watching a clip with verbatim subtitles.

These provisional qualitative indications of differences between eye movements of users with different profiles require more in-depth quantitative investigation and the subsequent section will provide a few steps in this direction.

As mentioned above, subtitle reading patterns largely depend on the type of viewers. Fluent readers have been found to have no difficulty following subtitles. Diao et al. (2007), for example, found a direct correlation between the impact of subtitles on learning and the academic and literacy levels of participants. Similarly, given that “hearing status and literacy tend to covary” (Burnham et al. 2008, 392), some previous studies found important differences in the way hearing and hearing-impaired people watch subtitled programmes. Robson (2004, 21) notes that “regardless of their intelligence, if English is their second language (after sign language), they [i.e. Deaf people] cannot be expected to have the same comprehension levels as hearing people who grew up exposed to English”. This is indeed confirmed by Szarkowska et al. (forthcoming) who report that Deaf and hard of hearing viewers in their study made more fixations on subtitles and that their dwell time on the subtitles was longer compared to hearing viewers. This result may indicate a larger effort needed to process subtitled content and more difficulty in extracting information (see Holmqvist et al. 2011, 387-388). This, in turn, may stem from the fact that for some Deaf people the language in the subtitles is not their mother tongue (their L1 being sign language). At the same time, for hearing-impaired viewers, subtitles provide an important source of information on the words spoken in the audiovisual text as well as other information contained in the audio track, which in itself explains the fact that they would spend more time looking at the subtitles.

Viewer-external factors that impact on subtitle processing

The ‘smoothness’ of the subtitle reading process depends on a number of factors, including the nature of the audiovisual material as well as technical and graphical aspects of subtitles themselves. At a general level, genre has an impact on both the role of subtitles in the total viewing experience, and on the way viewers process the subtitles. For example, d’Ydewalle and Van Rensbergen (1989) found that children in Grade 2 paid less attention to subtitles if a film involved a lot of action (see d’Ydewalle & Bruycker 2007 for a discussion). The reasons for this could simply be that action film tends to have less dialogue in the first place, but secondly and more significantly, the pace of the visual editing and the use of special effects creates a stronger visual element which shifts the balance of content towards the action (visual content) and away from dialogue (soundtrack and therefore subtitles). This, however, is an area that has to be investigated empirically. At a more specific level, technical characteristics of an audiovisual text such as film editing have an impact on the processing of subtitles.

1 Film editing

Film editing has a strong influence on the way people read subtitles, even beyond the difference in editing pace as a result of genre (for example, action and experimental films could typically be said to have a higher editing pace than dramas and documentaries). In terms of audience perception, viewers have been found to be unaware of standard film editing techniques (such as continuity editing) and are thus able to perceive film as a continuous whole in spite of numerous cuts – the phenomenon termed “edit blindness” (Smith & Henderson, 2008, 2). With more erratic and fast-paced editing, it stands to reason that the cognitive demands will increase as viewers have to work harder to sustain the illusion of a continuous whole.

When subtitles clash with editing such as cuts (i.e. if subtitles stay on screen over a shot or scene change), conventional wisdom as passed on by generations of subtitling guides (see Díaz Cintas & Remael 2007, ITC Guidance on Standards for Subtitling 1999) suggests that the viewer will assume that the subtitle has changed with the image and as a consequence they will re-read it (see above). However, Krejtz et al. (2013) reported that subtitles displayed over shot changes are more likely to cause perceptual confusion by making viewers shift their gaze between the subtitle and the rest of the image more frequently than subtitles which do not cross film cuts (cf. de Linde and Kay 1999). As such, the cognitive load is bound to increase.

2 Text chunking and line segmentation

Another conventional wisdom, perpetuated in subtitling guidelines and standards, is that poor line segmentation will result in less efficient processing (see Díaz Cintas & Remael 2007, Karamitroglou 1998). In other words, subtitles should be chunked per line and between subtitles in terms of self-contained semantic units. The line of dialogue: “He told me that he would meet me at the red mailbox” should therefore be segmented in something like the following ways:

He told me he would meet me
at the red mailbox.

Or

He told me
he would meet me at the red mailbox.

Neither of the following segmentations would be optimal because the prepositional phrase ‘at the red mailbox’ and the verb phrase ‘he would meet me’, respectively, are split, which is considered an error:

He told me he would meet me at the
red mailbox

He told me he
would meet me at the red mailbox.

However, Perego et al. (2010) found that poor line segmentation in two-line subtitles did not affect subtitle comprehension negatively. They also investigated 28 subtitles viewed by 16 participants using a threshold line between the subtitle region and the upper part of the screen, or main film zone, but did not find a statistically significant difference between the well-segmented and ill-segmented subtitles in terms of fixation counts, total fixation time, or number of shifts between subtitle region and upper area. The only statistically significant difference they found was between the mean fixation duration within the subtitle area between the two conditions, with the mean fixation duration in the ill-segmented subtitles being on average 12ms longer than in the well-segmented subtitles. Although the authors downplay the importance of this difference on the grounds that the difference is so small, it does seem to indicate at least a slightly higher cognitive load when the subtitles are ill-segmented. The small number of subtitles and participants, however, make it difficult to generalize from their results, again a result of the fact that it is difficult to extract reading statistics for subtitles unless the reading behaviour can be quantified over longer audiovisual texts.

In a study conducted a few years later, Rajendran et al. (2013) found that “chunking improves the viewing experience by reducing the amount of time spent on reading subtitles” (2013, 5). This study compared conditions different from those investigated in the previous study, excluding the ill-segmented condition of Perego et al. (2010), and focused mostly on live subtitling with respeaking. In the earlier study, which focused on pre-recorded subtitling, the subtitles in the two conditions were essentially still part of one sense unit that appeared as one two-line subtitle. In the later study, the conditions were chunked by phrase (similar to the well-segmented condition of the earlier study but with phrases appearing one by one on one line), no segmentation (where the subtitle area was filled with as much text as possible with no attempt at segmentation), word by word (where words appeared one by one) and chunked by sentence (where the sentences showed up one by one). Regardless of the fact that this later study therefore essentially investigated different conditions, they did find that the most disruptive condition was where the subtitle appeared word by word – eliciting more gaze points (defined less strictly than in fixation algorithms used by commercial eye trackers) and more “saccadic crossovers” or switches between image and subtitle area. However, in this study by Rajendran et al. (2013), the videos were extremely short (under a minute), and the sound was muted, hampering the ecological validity of the material, and once again making the findings less suitable to generalization.

Although both these studies have limitations in terms of generalizability, they both provide some indication that segmentation has an impact on subtitle processing. Future studies will nonetheless have to investigate this aspect over longer videos to determine whether the graphical appearance, and particularly the segmentation of subtitles, has a detrimental effect on subtitle processing in terms of cognitive load and effectiveness.

3 Language

The language of subtitles has received considerable attention from psycholinguists in the context of subtitle reading. D’Ydewalle and de Bruycker (2007) examined eye movement behaviour of people reading standard interlingual subtitles (with the audio track in a foreign language and subtitles in their native language) and reversed subtitles (with the audio in their mother tongue and subtitles in a foreign language). They found more regular reading patterns in the standard interlingual subtitling condition, with the reversed subtitling condition having more subtitles skipped, fewer fixations per subtitle, etc. (see also d’Ydewalle and de Bruycker 2003 and Pavakanun 1993). This is an interesting finding in itself, as it is the reversed subtitling that has been found to be particularly conducive to foreign language learning (see Díaz Cintas and Fernández Cruz 2008, and Vanderplank 1988).

Szarkowska et al. (forthcoming) examined differences in reading patterns of intralingual (Polish to Polish) and interlingual (English to Polish) subtitles among a group of Deaf, hard of hearing and hearing viewers. They found no differences in reading for the Deaf and hard of hearing audiences, but hearing people made significantly more fixations to subtitles when watching English clips with interlingual Polish subtitles than Polish clips with intralingual Polish subtitles. This confirms that the hearing viewers processed the subtitles to a significantly lower degree when they were redundant, as in the case of intralingual transcriptions of the soundtrack. What would be interesting to investigate in this context is those instances when the hearing audience did in fact read the subtitles, to determine to what extent and under what circumstances the redundant written information is used by viewers to support their auditory intake of information.

In a study on the influence of translation strategies on subtitle reading, Ghia (2012) investigated the differences in the processing of literal vs. non-literal translations into Italian of an English film clip (6 minutes) when watched by Italian EFL learners. According to Ghia, just as subtitle format, layout, and segmentation have the potential to affect visual and perceptual dynamics, the relationship translation establishes with the original text means that “subtitle translation is also likely to influence the perception of the audiovisual product and viewers’ general reading patterns” (2012,175). Ghia particularly wanted to investigate the processing of different translation strategies in the presence of sound and image with the subtitles. In her study she found that the non-literal translations (where the target text diverged from the source text) resulted in more deflections between text and image. This is similar to the findings of Rajendran et al. (2013) in terms of less fluent graphics in word-by-word subtitles.

As can be seen from the above, the aspect of language processing in the context of subtitled audiovisual texts has received some attention, but has not to date been approached in any comprehensive manner. In particular, there is a need for more psycholinguistic studies to determine how subtitle reading differs from the reading of static text, and how this knowledge can be applied to the practice of subtitling.

Measuring subtitle processing

1 Attention distribution and presentation speed

In the study by Jensema et al. (2000), subjects spent on average 84% of the time looking at subtitles, 14% at the video picture and 2% outside of the frame. The study represents an important early attempt to identify reading patterns in subtitle reading, but it has considerable limitations. The study had only six participants, three deaf and three hearing, and the video clips were extremely short (around 11 seconds each), presented with English subtitles (in upper case) without sound. The fact that there was no soundtrack therefore impacted on the time spent on the subtitles. In Perego et al’s study (2010), the ratio is reported as 67% on the subtitle area and 33% on the image. In this study there were 41 Italian participants who watched a 15-minute clip with Hungarian soundtrack and subtitles in Italian. As in the previous study, the audience therefore had to rely heavily on the subtitles in order to follow the dialogue. Kruger et al. (2014), in the context of intralingual subtitles in a Psychology lecture in English, found a ratio of 43% on subtitles, 43% on the speaker and slides and 14% on the rest of the screen. When the same lecture was subtitled into Sotho, the ratio changed to 20% on the subtitles, 66% on the speaker and slides, and 14% on the rest of the screen. This wide range is an indication of the difference in the distribution of visual attention in different contexts with different language combinations, different levels of redundancy of information, and differences in audiences.

In order to account for “the audiovisual nature of subtitled programmes”, Romero-Fresco (in press) puts forward the notion of ‘viewing speed’ – as opposed to reading speed and subtitling speed – which he defines as “the speed at which a given viewer watches a piece of audiovisual material, which in the case of subtitling includes accessing the subtitle, the accompanying images and the sound, if available”. The perception of subtitled programmes is therefore a result of not only the subtitle reading patterns, but also the visual elements of the film. Based on the analysis of over seventy-one thousand subtitles created in the course of the Digital Television for All project, Romero Fresco provides the following data on the viewing speed, reflecting the proportion of time spent by viewers looking at subtitles and at the images, proportional to the subtitle presentation rates (see Table 1).

Viewing speed Time on subtitles Time on images
120wpm ±40% ±60%
150wpm ±50% ±50%
180wpm ±60%-70% ±40%-30%
200wpm ±80% ±20%

Table 1. Viewing speed and distribution of gaze between subtitles and images (Romero-Fresco) 

Jensema et al. also suggested that the subtitle presentation rate may have an influence on the time spent reading subtitles vs. watching the rest of the image: “higher captioning speed results in more time spent reading captions on a video segment” (2000, 275). This was later confirmed by Szarkowska et al. (2011), who found that viewers spent more time on verbatim subtitles displayed at higher presentation rates compared to edited subtitles displayed with low reading speed, as illustrated by Figure 8.

Figure 8. Fixation-count based heatmaps illustrating changes in attention allocation of hearing and Deaf viewers watching videos subtitled at different rates.

Figure 8. Fixation-count based heatmaps illustrating changes in attention allocation of hearing and Deaf viewers watching videos subtitled at different rates.

2 Mean fixation duration

Irwin (2004, 94) states that “fixation location corresponds to the spatial locus of cognitive processing and that fixation or gaze duration corresponds to the duration of cognitive processing of the material located at fixation”. Within the same activity (e.g. reading), longer mean fixation durations could therefore be said to reflect more cognitive processing and higher cognitive load. One would therefore expect viewers to have longer fixations when the subject matter is more difficult, or when the language is more specialized. Across activities, however, comparisons of fixation duration is less meaningful as reading elicits more shorter fixations than scene perception or visual scanning simply because of the nature of the activities. It is therefore essential in eye tracking studies of subtitle reading to distinguish between the actual subtitles when they are on screen, the rest of the screen, and the subtitle area when there is no text (between successive subtitles).

The difference between reading and scene perception is illustrated in Figure 9, demonstrating that fixations on the image tend to be longer (indicated here by a bigger circle) than those on subtitles (which indicates more focused viewing), and more exploratory in nature (see the distinction between focal and ambient fixations in Velichkovsky et al. 2005).

Figure 9. Differences in fixation durations between the image and subtitle text – from Polish TV series Londyńczycy.

Figure 9. Differences in fixation durations between the image and subtitle text – from Polish TV series Londyńczycy.

Rayner (1984) indicated the impact of different tasks on mean fixation durations, as reflected in Table 2 below:

Task Mean fixation duration (ms) Mean saccade size (degrees)
Silent reading 225 2 (about 8 letters)
Oral reading 275 1.5 (about 6 letters)
Visual search 275 3
Scene perception 330 4
Music reading 375 1
Typing 400 1 (about 4 letters)

 Table 2. Approximate Mean Fixation Duration and Saccade Length in Reading, Visual Search, Scene Perception, Music Reading, and Typing[1]

In subtitling, silent reading is accompanied by simultaneous processing of the same information in the soundtrack (in the same or another language) as well as of other sounds and visual signs (for a hearing audience, that is – for a Deaf audience, it would be text and visual signs). The difference in mean fixation duration in these different tasks therefore reflects the difference in cognitive load. In silent reading of static text, there is no external competition for cognitive resources. When reading out loud, the speaker/reader inevitably monitor his/her own reading, introducing additional cognitive load. As the nature of the sign becomes more abstract, the load, and the fixation duration increases, and in the case of typing, different processing, production and checking activities are performed simultaneously, resulting in even higher cognitive load. This is inevitably an oversimplification of cognitive load, and indeed the nature of information acquisition between reading successive groups of letters (words) in a linear fashion is significantly different from that of scanning a visual scene for cues.

Undoubtedly, subtitle reading imposes different cognitive demands, and these demands are also very much dependent on the audience. In an extensive study on the differences in subtitle reading between Deaf, hard of hearing and hearing participants, we found a high degree of variation in mean fixation duration between the groups, and also a difference between the mean fixation duration in the Deaf and the hard of hearing groups between subtitles presented at 12 characters per second and 15 characters per second (see Szarkowska et al. forthcoming).

  12 characters per second 15 characters per second
Deaf 241.93 ms 232.82 ms
Hard of hearing 218.51 ms 214.78 ms
Hearing 186.66 ms 186.58 ms

Table 3. Differences in reading subtitles presented at different rates

Statistical analyses performed on the three groups with mean fixation duration as a dependent variable and groups and speed as categorical factors produced a statistically significant main effect, further confirmed by subsequent t-tests that yielded statistically significant differences in mean fixation duration for both subtitling speeds between all three groups. The difference within the Deaf and hard of hearing groups was also significant between 12cps and 15cps. What this suggests is that reading speed has a more pronounced effect on Deaf and hard of hearing viewers than on hearing ones.

3 Subtitle reading

As indicated at the outset, one of the biggest hurdles in studying the processing of subtitles is the fact that the subtitles appear as image on image rather than text on image as far as eye tracking analysis software is concerned. Whereas reading statistics software can therefore automatically mark words as areas of interest in static texts, and then calculate number of regressions, refixations, saccade length, fixation duration and count as related to the specific words, this process has to be done manually for subtitles. The fact that it is virtually impossible to create similar areas of interest on the subtitle words that are embedded in the image over large numbers of subtitles makes it very difficult to obtain reliable eye tracking results on subtitles as text. This explains the predominance of measures such as fixation count and fixation duration as well as shifts between subtitle area and image in eye tracking studies on subtitle processing. As a result, many of these studies do not distinguish directly between looking at the subtitle area and reading the subtitles, and, “they tend to define crude areas of interest (AOIs), such as the entire subtitle area, which means that eye movement data are also collected for the subtitle area when there are no subtitles on screen, which further skews the data” (Kruger and Steyn, 2014, 109).

Although a handful of studies come closer to studying subtitle reading by going beyond the study of fixation counts, mean fixation duration, and shifts between subtitle area and image area, most studies tend to focus on amount of attention rather than nature of attention. Briefly, the exceptions can be identified in the following studies: Specker (2008) looks at consecutive fixations; Perego et al. (2010) add the path length (sum of saccade lengths in pixels) to the more conventional measures; Rajendran et al. (2013) add the proportion of gaze points; Ghia (2012) looks at fixations on specific words as well as regressions; Bisson et al. (2012) look at the number of subtitles skipped, and proportion of successive fixations (number of successive fixations divided by total number of fixations); and in one of the most comprehensive studies on the subject of subtitle processing, d’Ydewalle and De Bruycker (2007) look at attention allocation (percentage of skipped subtitles, latency time, and percentage of time spent in the subtitle area), fixations (number, duration, and word-fixation probability), and saccades (saccade amplitude, percentage of regressive eye movements, and number of back-and-forth shifts between visual image and subtitle).

In a recent study, Kruger and Steyn (2014) provide a reading index for dynamic texts (RIDT) designed specifically to measure the degree of reading that takes place when subtitled material is viewed. This index is explained as “a product of the number of unique fixations per standard word in any given subtitle by each individual viewer and the average forward saccade length of the viewer on this subtitle per length of the standard word in the text as a whole” (2014, 110). Taking the location and start time of successive fixations within the subtitle area when a subtitle is present as the point of departure, the number of unique fixations (i.e. excluding refixations, and fixations following a regression) is determined, as well as the average length of forward saccades in the subtitle. This information gives an indication of the meaningful processing of the words in the subtitle when the number of fixations per word, as well as the length of saccades as ratio of the length of the average word in the audiovisual text are calculated. Essentially, the formula quantifies the reading of a particular subtitle by a particular participant by measuring the eye movement during subtitle reading against what is known about eye movements during reading and perceptual span.

In a little more detail, the formula can be written as follows for video v, with participant p viewing subtitle s”:

10

(Kruger and Steyn, 2014, 110).

This index was validated by performing a comparison of the manual inspection of the reading of 145 subtitles by 17 participants, and makes it possible to study the reading of subtitles over extended texts. In their study, Kruger and Steyn (2014) use the index to determine the relationship between subtitle reading and performance in an academic context, finding a significant positive correlation between the degree to which participants read the subtitles and their performance in a test written after watching subtitled lectures. The RIDT therefore presents a robust index of the degree to which subtitles are processed over extended texts, and could add significant value to psycholinguistic studies on subtitles. Using the index, previous claims that subtitles have a positive or negative impact on comprehension, vocabulary acquisition, language learning or other dependent variables, can be correlated with whether or not viewers actually read the subtitles, and to what extent the subtitles were read.

Conclusion

From this overview of studies investigating the processing of subtitles on the moving image it should be clear that much still needs to be done to gain a better understanding of the impact of various independent variables on subtitle processing. The complexity of the multimodal text, and in particular the competition between different sources of information, means that a subtitled audiovisual text is a substantially altered product from a cognitive perspective. Much progress has been made in coming to grips with the way different viewers behave when looking at subtitled audiovisual texts, but there are still more questions than answers – relating, for instance, to differences in how people process subtitled content on various devices (cf. the HBBTV4ALL project). The use of physiological measures like eye tracking and EEG (see Kruger et al. 2014) in combination with subjective measures like post-report questionnaires is, however, continually bringing us closer to understanding the impact of audiovisual translation like subtitling on the experience and processing of audiovisual texts.

 

Acknowledgements

This study was partially supported by research grant No. IP2011 053471 “Subtitling for the deaf and hard of hearing on digital television” from the Polish Ministry of Science and Higher Education for the years 2011–2014.

 

References

Bisson, Marie-Josée, Walter Van Heuven, Kathy Conklin, and Richard Tunney. 2014. “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics 35(2):399-418.

Burnham, Denis, Leigh Greg, Noble William, Jones Caroline, Tyler Michael, Grebennikov Leonid and Alex Varley. 2008. Parameters in television captioning for deaf and hard-of-hearing adults: effects of caption rate versus text reduction on comprehension. Journal of Deaf Studies and Deaf Education 13 (3):391-404.

de Linde, Zoé and Neil Kay. 1999. The Semiotics of Subtitling. Manchester: St. Jerome.

Diao, Y., Chandler, P., Sweller, J. 2007. The effect of written text on comprehension of spoken English as a foreign language. The American Journal of Psychology 120(2): 237-261.

Díaz Cintas, Jorge and Marco Fernandez Cruz. (2008) “Using subtitled video materials for foreign language instruction”. In The Didactics of Audiovisual Translation edited by Jorge Díaz Cintas, 201-214. Amsterdam/Philadelphia: John Benjamins.

Díaz Cintas, Jorge and Aline Remael. 2007. Audiovisual Translation: Subtitling. Manchester: St. Jerome.

d’Ydewalle, Géry and Wim De Bruycker. 2003. Reading native and foreign language television subtitles in children and adults. In The mind’s eyes: Cognitive and applied aspects of eye movement research, edited by J. Hyönä, R. Radach and H. Deubel, 444-461. New York: Springer-Verlag.

d’Ydewalle, Géry and Wim De Bruycker. 2007. “Eye Movements of Children and Adults while Reading Television Subtitles.” European Psychologist 12:196–205.

d’Ydewalle, Géry and Ingrid Gielen. 1992. “Attention Allocation with Overlapping Sound, Image, and Text.” In Eye Movements and Visual Cognition: Scene Perception and Reading, edited by Keith Rayner, 415–427. New York: Springer-Verlag.

d’Ydewalle, Géry, Johan Van Rensbergen, and Joris Pollet. 1987. Reading a message when the same message is available auditorily in another language: The case of subtitling. In Eye Movements: From Physiology to Cognition edited by J.K O’Reagan and A. Lévy Schoen, 313-321. Amsterdam: Elsevier Science Publishers B.V. (North-Holland).

Ghia, Elisa. 2012. “The Impact of Translation Strategies on Subtitle Reading.” In Eye Tracking in Audiovisual Translation, edited by Elisa Perego, 155–182. Roma: Aracne Editrice.

Gottlieb, Henrik. 1998. Subtitling. In Routledge Encyclopaedia of Translation Studies, edited by Mona Baker, 244-248. London & New York: Routledge.

Hershler, Orit and Shaul Hochstein. 2005. At first sight: a high-level pop out effect for faces. Vision Research, 45, 1707–1724.

Holmqvist, Kenneth et al. 2011. Eyetracking. A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press.

Irwin, David E. 2004. Fixation location and fixation duration as indices of cognitive processing. In J.M. Henderson & F. Ferreira (Eds.), The interface of language, vision, and action: Eye movements and the visual world, 105-133. New York, NY: Psychology Press.

ITC Guidance on Standards for Subtitling. Online at: http://www.ofcom.org.uk/static/archive/itc/itc_publications/codes_guidance/standards_for_subtitling/subtitling_1.asp.html

Jensema, Carl. 2000. Eye movement patterns of captioned TV viewers. American Annals of the Deaf vo. 145, no. 3, 275-285.

Karamitroglou, Fotios. 1998. A Proposed Set of Subtitling Standards in Europe. Translation Journal 2(2). http://translationjournal.net/journal/04stndrd.htm

Krejtz, Izabela, Agnieszka Szarkowska, and Krzysztof Krejtz. 2013. “The Effects of Shot Changes on Eye Movements in Subtitling.” Journal of Eye Movement Research 6 (5): 1–12.

Kruger, Jan-Louis and Faans Steyn. 2014. “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly 49 (1): 105–120.

Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew. 2013a. “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts.” Proceedings of Eye Tracking South Africa 29-31 August 2013, Cape Town.

Kruger, Jan-Louis, Esté Hefer, and Gordon Matthew. 2013b. The impact of subtitles on academic performance at tertiary level. Paper presented at the Linguistics Society of Southern Africa annual conference in Stellenbosch, June, 2013.

Kruger, Jan-Louis. 2013. “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load.” Journal for Language Teaching 47(1):29–53.

Kruger, Jan-Louis, Hefer, Esté, and Gordon Matthew. 2014. Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2. Journal of Eye Movement Research 7(5):4, 1–15.

Langton, Stephen R.H., Anna S. Law, Burton, A. Mike and Stefan R. Schweinberger. 2008. Attention capture by faces. Cognition, 107:330-342.

Pavakanun, Ubowanna. 1992. Incidental acquisition of foreign language through subtitled television programs as a function of similarity with native language and as a function of presentation mode. Unpublished doctoral thesis, Leuven, Belgium, University of Leuven.

Perego, Elisa, Fabio Del Missier, Marco Porta and Mauro Mosconi. 2010. “The Cognitive Effectiveness of Subtitle Processing.” Media Psychology 13(3):243–272.

Rajendran, Dhevi, Andrew Duchowski, Pilar Orero, Juan Martínez, and Pablo Romero-Fresco. 2013. “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination.” Perspectives: Studies in Translatology 21(1):5–31.

Rayner, Keith. 1984. Visual selection in reading, picture perception, and visual search: A tutorial review. In Attention and performance edited by H. Bouma and D. Bouhwhuis, vol. 10. Hillsdale, NJ: Erlbaum.

Rayner, Keith 1998. “Eye movements in reading and information processing: Twenty years of research.” Psychological Bulletin, 124:372–422.

Robson, Gary D. 2004. The closed captioning handbook. Amsterdam: Elsevier.

Romero Fresco, Pablo (in press) The Reception of Subtitles for the Deaf and Hard of Hearing in Europe. Peter Lang.

Smith, Tim, and John M. Henderson. 2008. Edit Blindness: The relationship between attention and global change blindness in dynamic scenes. Journal of Eye Movement Research 2(2), 6:1-17.

Specker, Elizabeth, A. 2008. L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. Unpublished PhD thesis. University of Arizona.

Szarkowska, Agnieszka, Krejtz, Izabela, and Łukasz Dutka. (forthcoming) The effects of subtitle presentation rate, text editing and type of subtitling on the comprehension and reading patterns of subtitles among deaf, hard of hearing and hearing viewers. To appear in: Across Languages and Cultures 2016, vol. 2.

Szarkowska, Agnieszka, Krejtz, Izabela, Kłyszejko, Zuzanna and Anna Wieczorek. 2011. “Verbatim, standard, or edited? Reading patterns of different captioning styles among deaf, hard of hearing, and hearing viewers”. American Annals of the Deaf 156 (4):363-378.

Vanderplank, Robert. 1988 “The value of teletext sub-titles in language learning”. ELT Journal 42(4):272-81.

Velichkovsky, Boris M., Joos, Markus, Helmert, Jens R., and Sebastian Pannasch. 2005. Two Visual Systems and Their Eye Movements: Evidence from Static and Dynamic Scene Perception. InCogSci 2005: Proceedings of the XXVII Conference of the Cognitive Science Society, 2283–2288.

Winke, Paula, Susan Gass, and Tetyana Syderenko. 2013. “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study.” The Modern Language Journal 97 (1):254–275.

Yarbus, Alfred L. 1967. Eye movements and vision. New York, NY: Plenum Press.

 

Notes

[1] Values are taken from a number of sources and vary depending on a number of factors (see Rayner, 1984)

 

Bios

Jan-Louis Kruger is director of translation and interpreting in the Department of Linguistics at Macquarie University in Sydney, Australia.  He holds a PhD in English on the translation of narrative point of view. His main research interests include studies on the reception and cognitive processing of audiovisual translation products including aspects such as cognitive load, comprehension, attention allocation, and psychological immersion.

Agnieszka Szarkowska, PhD, is Assistant Professor in the Institute of Applied Linguistics at the University of Warsaw, Poland. She is the founder and head of the Audiovisual Translation Lab, a research group working on media accessibility. Her main research interests lies in audiovisual translation, especially subtitling for the deaf and the hard of hearing and audio description.

Izabela Krejtz, PhD, is Assistant Professor at University of Social Sciences and Humanities, Warsaw. She is a co-founder of Eyetracking Research Center at USSH. Her research interests include neurocognitive and educational psychology. Her applied work focuses on pro-positive trainings of attention control, eye tracking studies in perception of audiovisual material and emotions regulation.

Volume 24, 2014

Themed Issue: Intermediations

Edited by Kevin Fisher and Holly Randell-Moon

Contents:

1. Editorial Introduction — Kevin Fisher and Holly Randell-Moon

2. Animating Ephemeral Surfaces: Transparency, Translucency and Disney’s World of Color  — Kirsten Moana Thompson

3. Vertical Framing: Authenticity and New Aesthetic Practice in Online Videos — Miriam Ross

4. Attached To My Devices: Across Individual, Collective and Panspectric Worlds — John Farnsworth

5. The Ecstatic Gestalt in Werner Herzog’s Cave of Forgotten Dreams — Kevin Fisher

6. Intermediality and Interventions: Applying Intermediality Frameworks to Reality Television and Microblogs — Rosemary Overell

7. ‘God Hates Fangs’: Gay Rights As Transmedia Story in True Blood — Holly Randell-Moon

8. We are the Borg (in a good way): Mapping The Development Of New Kinds Of Being And Knowing Through Inter- and Trans-Mediality — Anne Cranny Francis

We are the Borg (in a good way): Mapping The Development Of New Kinds Of Being And Knowing Through Inter- and Trans-Mediality — Anne Cranny Francis

Abstract: Digital technologies have enabled new ways of communicating and relating to others and this has fundamental consequences for being and for meaning. In this paper I map the development of concepts of intermediality and transmediality that are used to describe textual practice and audience engagement in order to explore these changes to communication practice. At the same time I explore the new kinds of audience engagement enabled by this technology, which includes active participation in the reconstruction of older narratives in new media and the potential this affords for new meanings. It also includes the dissemination of stories, old and new, across multiple platforms by both makers and audiences, who themselves become makers, and the proliferation of stories and meanings this enables. Finally I consider the possibilities for co-creation—my hardware, your software (or vice-versa)—which can enable new forms of sharing and mutual knowledge-formation.

Sherlock (BBC, 2010-- )

Sherlock (BBC, 2010– )

1. On thinking about inter- and trans-

The research for this paper led me through a range of ideas and arguments about the meanings of intermediation and transmediation, as well as their relationship to intertextuality (for example, Bakhtin 1984; Jenkins 2006; Herzogenrath 2012; Stein and Busse 2012; Phillips 2012). It led me to think about a multiplicity of texts that are all inter in some way—either intertextually related texts and the kinds of meanings they make or intermediated narratives that tell their story across a range of media and platforms—and about texts, producers and audiences that are most definitely trans—deploying a range of media and platforms to create a composite and complex world, engage with that world, and generate new meanings. This textual multiplicity in the contemporary media environment in turn raised questions about what has caused or generated these differing ways of telling a story and what is the significance of these different modes of story-telling: whether this reflects simply a change in technology (if that is ever truly simple) or if that change has consequences that move far beyond the material technologies involved—the material artefacts and related communication practices—to our ways of thinking and of being in the world.

My argument is that digital technologies have enabled new ways of communicating and relating to others and that this has fundamental consequences for being and for meaning. Further, we are only just starting to realise the possibilities and potential offered by this technology for new forms of relationship, knowledge creation and sharing. I work through these possibilities by reference to a range of texts that were suggested by my research and which recur in discussions of these new modes of story-telling and text production. My interest is not only in digital texts themselves, but also in the new forms of engagement they offer to readers, viewers and listeners to become active producers or makers of meaning alongside the creators of the work. This engagement includes our participation in the reconstruction of older narratives in new media and the potential this affords for new meanings; the dissemination of stories, old and new, across multiple platforms by both makers and audiences, who themselves become makers, and the proliferation of stories and meanings this enables; and finally the possibilities for co-creation—my hardware, your software (or vice-versa)—which can enable new forms of sharing and mutual knowledge-formation.

This exploration of shared storytelling and textual production occurs through my engagement with the theory used by media and cultural analysts to understand transformations in creativity, knowledge-formation and being. This work includes the concepts of intermediation, which explores the possibilities opened up by new media and focuses on the textual practices that enable new forms of audience engagement, and transmediation, which also explores the effect of new technologies on meaning-making but shifts its focus from textual practice to audience response. This is a subtle shift as both concepts essentially study the same phenomena (including both textual practice and audience responses), but it mirrors what Henry Jenkins called the development of ‘convergence culture’: “the flow of content across multiple media platforms, the cooperation between multiple media industries, and the migratory behavior of media audiences who will go almost anywhere in search of the kinds of entertainment experiences they want” (2006, 2). As I will go on to argue, this convergence, this sharing of and linking via new media technologies, has the potential to transform our experience of the world and, along with that, our formation of knowledge and fundamental understandings of being.

2. The Consulting Detective and The Doctor

My first thought when beginning this paper was to use the BBC (British Broadcasting Corporation) version of Sherlock (2010-) as my example of intermediation. One of the things that attracted me to this text was that it re-tells Sir Arthur Conan Doyle’s original stories in such a fresh and engaging way, not only through the revised characterisations of its principals (Holmes, Watson, Lestrade, Moriarty, Mycroft) and the rapid editing and visual layering of the mise-en-scène that creates 21st century London as the technological and social successor to Conan Doyle’s 19th century industrial London, but also by the re-framing of familiar narratives to make them directly relevant to contemporary British society. For example, The Hound of the Baskervilles (1901) is re-written by Mark Gatiss as “The Hounds of Baskerville” (2012), a story about experiments with nerve agents and genetic mutation at a United Kingdom military base. The story focuses around a local man, Henry Knight who, as a child, saw his father torn apart by a giant hound on Dartmoor, near the Baskerville military establishment. Fear of the hound is produced not, as in the original story, by phosphorescence painted onto a large dog (though the local innkeepers have a large dog that they used to spread the ‘giant hound’ story to tourists), but by a hallucinogenic drug that is released into the air by nerve pads buried in a certain part of the nearby moors. We eventually discover that Knight’s father was killed accidentally when he wandered into the test area for these nerve pads. Under the influence of the air-borne toxin, Knight tripped and hit his head on a rock while attempting to run away from Baskerville scientist, Robert Frankland, who was wearing a gas mask and so appeared monstrous. The young Henry Knight witnessed his father’s accidental death but under the influence of the nerve toxin transformed the memory into the story of the giant hound, suggested to him by the initials H.O.U.N.D. on Franklin’s jumper.

Gatiss’s story uses elements of Conan Doyle’s original but reworks them into a contemporary story about the development of chemical and biological weapons and their production within an environment of secrecy that puts citizens’ lives at risk. The main characters (Sherlock Holmes [Benedict Cumberbatch], Dr Watson [Martin Freeman], Mycroft Holmes [Mark Gatiss] and Inspector Lestrade [Rupert Graves]) are also developed further in this story, including exploration of Sherlock’s ambiguous sexuality and his relationship with Watson, which is mapped explicitly onto the gay relationship of the local innkeepers. It is an engaging tale for the Conan Doyle enthusiast as it preserves the central motif of the narrative—the ghostly hound—but finds a way of re-presenting it that changes the story from one about evil aristocrats (the original Baskerville and his ruthless treatment of the local peasants) and modern greed (a villainous descendent of the original attempting to kill the successor to the title so that he inherits the family fortune) to one about weapons of mass destruction and government secrecy. It also presents a different ‘take’ on the sexuality of Holmes (also explored in the recent films directed by Guy Ritchie and starring Robert Downey Jr. as Holmes and Jude Law as Watson [2009, 2011]), opening up the possibility that he is either gay or bisexual whereas Conan Doyle presents Holmes as relatively asexual.[1] This re-working of the story and its characters constitutes the text as more than a period adaptation of Conan Doyle’s story, set in the late Victorian period with Holmes and Watson inhabiting the world of brougham cabs and steam trains. So is this an example of intertextuality or intermediality, with the literary creation of Conan Doyle cast as another text or medium that incorporates audience engagement with the story?

Perhaps the most obvious answer here is that this re-casting of the Holmes story is an example of intermediality, defined in an early essay by Dick Higgins as generated by “the desire to fuse two or more existing media” (1966). Berndt Herzogenrath notes, however, that Higgins saw intermediality not as the final text but as “‘the uncharted land that lies between’ … different media” (2012, loc. 129-142).[2] The intermediality generated by the Sherlock re-visioning of The Hound of the Baskervilles enables the presentation of different meanings (about weapons production and secrecy) while maintaining the bones of the original narrative (about the abuse of power and the production of fear). Herzogenrath notes that in Image-Music-Text (1977) Roland Barthes related intermediality to interdisciplinarity, which occurs:

… when the solidarity of the old disciplines breaks down—perhaps even violently, via the jolts of fashion—in the interests of a new object and a new language neither of which has a place in the field of the sciences that were to be brought peacefully together, this unease in classification being precisely the point from which it is possible to diagnose a certain mutation. (loc. 129)

This disciplinary transformation might seem a heavy burden to place on Sherlock, however it is certainly the case that this production of The Hound of the Baskervilles in a different medium tells different stories and interrogates different aspects of everyday life (military activity, government control, sexual identity) from Conan Doyle’s original. Moreover, as discussed further below, Mark Gatiss’s revision of The Hound of the Baskervilles might be seen as Bakhtin’s heteroglossia in practice with Gatiss’ story constituting another voice/telling that reiterates some original narrative elements whilst adding some and transforming others.

Jeremy Brett as/in Sherlock Holmes (1984-94)

Jeremy Brett as/in Sherlock Holmes (1984-94)

From a contemporary perspective the transfer from literary text to television may not seem a case of disciplinary violence, however, some time ago it did. When television was younger and literature was a canonical art form, the production of a literary work as a television program led inevitably to discussions of what was ‘lost’ by the transfer to such an ‘impoverished’ medium. It is only far more recently that we have understood that an intermediated work is offering something new and different, unconstrained by the disciplinary shackles of the past. This realisation enables Sherlock to be written as a contemporary series, while retaining characteristics of its Victorian predecessor—as distinct, for example, from the older BBC series, The Adventures of Sherlock Holmes starring Jeremy Brett (1984-1994) that retained the Victorian setting for the stories. This successful relocation of the narrative for Sherlock depends on viewers being able to read across media platforms without the disciplinary blinkers of an earlier time; they no longer consider the narrative confined to a particular space/time as defined by the originary text. Instead, as regular consumers of postmodern pastiche, they adjust their reading practice for the complex network of intertextual references and narrative transpositions that constitutes this contemporary Sherlock.

This is more than simply a change in forms of entertainment or the emergence of new technologies. This radical unhooking of the narrative from its original space/time and the ability to read the stories for a different age, with different values and different concerns, is characteristic of the specificity and locatedness (sometimes read as relativism) of postmodernity. The postmodern producer appreciates the origin of textual forms and practices and is able to re-mediate them in order to make new meanings for a new time. Similarly, the postmodern consumer is able to appreciate the multiplicity of (textual) voices that constitute their world, and is not constrained to one major or canonical form of textual address as the bearer of cultural value. This is a reflexive consumer who maps networks of meaning extending beyond the confines of a specific text and its world; the viewer of The Matrix (The Wachowski Brothers, 1999) who knows to ‘follow the white rabbit’ to a looking-glass world that is our own world, and yet is not.

One of the means by which this reflexive writing and viewing practice has been understood is through the concept of intertextuality—used to describe the practice of referencing from one text to another via a character, icon, event or interaction, along with the meanings associated with that reference. Based on the work of Mikhail Bakhtin who saw every text as the premise for and related to every other text, via the heteroglossia (different voices) that constitute(s) our world, intertextuality is a way of mapping the complexity of communication practices and the meanings they convey, along with the impossibility of exerting total control over the meanings associated with a particular utterance (1984, 278). Intertextuality is about meaning and its constant deferral (in Derrida’s terms) not just the appearance of story elements in different texts. So intermediality acknowledges the use of different media or platforms to convey a specific narrative while intertextuality is a way of exploring the meanings constructed.

One way of mapping the possible meanings generated by viewer engagement with (intermediated) texts—including their constant deferral of meaning—is through the notion of genre, since this is the way that we typically classify texts in order to render them accessible. In a sense genre imposes order on the chaotic heteroglossia of our world so that it does not become an incomprehensible Babel in which each individual is isolated by a wholly idiosyncratic reading/viewing/meaning-making practice. Not only does genre identify the conventions or characteristics shared by the texts that we recognise as similar and so enable us to trace their history, it also identifies the kinds of issues commonly addressed by those texts. Science fiction, for example, commonly addresses the relationship between human beings and their technology, how technology influences our lives and even the fundamental nature of human being. This is evident in science fiction works such as Blade Runner (Ridley Scott, 1982) and The Matrix (1999), both of which explore how we deploy technology and what this tells us about ourselves. And this exploration of identity and technology has its roots in what is commonly regarded the first science fiction text, Mary Shelley’s Frankenstein ([1818] 1982), written at the height of the first Industrial Revolution in western societies, when steam power had transformed work practices and social relationships, obliterating older forms of labour and the classes who performed it and reconstructing society into new classes. This industrial context may not be explicit in every reference to Frankenstein but it echoes through portrayals of the angry, sad and abandoned creature and his deluded creator, who become the robots/androids of today and us, their sometimes deluded or unaware creators and users.

Sherlock and Moriarty

Moriarty and Sherlock

One of the striking features of Sherlock is its stylistic similarity to Doctor Who, generated by the visual aesthetic, costuming, editing, and the enigmatic and manic main character, Sherlock/The Doctor and his mirror self, Moriarty/The Master. This might seem unsurprising given that the same creative team is responsible for both programs; writers, Stephen Moffat and Mark Gatiss devised the idea for Sherlock on the train to Wales to work on Doctor Who (which is produced in Cardiff). However, that fact does not explain the resulting program and its success. A generic analysis of the two series is suggestive, showing that both science fiction (Doctor Who) and detective fiction (Sherlock) have their story-telling roots in Gothic fiction, which was preoccupied with questions about being, the nature of the real, the nature of good and evil, and the dual (good/evil) nature of humanity. In science fiction those concerns are directed to an exploration of our relationship with new technologies, as discussed above.

The Doctor and The Master

The Doctor and The Master

Detective fiction focuses on the nature of knowing, personified in the detective, beginning with Edgar Allan Poe’s brilliant investigator, C. Auguste Dupin in stories the author described as “tales of ratiocination” (2010). Dupin employs a version of the scientific method (involving observation and analysis) leavened with imagination, which enables him to look beyond the obvious. Conan Doyle’s Sherlock Holmes is even more scientific in his practice, but with the same disdain for conventional ways of thinking. This deployment of scientific method in order to solve social (rather than scientific) problems focuses attention on the process of knowledge formation (how we know and understand our world and each other) and its role in our understanding of morality (whether good and evil are easily identified) and of being (whether human beings are simply good or evil). The contemporary BBC Sherlock continues this tradition of the scientific detective informed by an eccentric imagination that enables him to step outside conventionalised patterns of thought and assumption.

Intertextually, Doctor Who and Sherlock share the Gothic preoccupation with interrogating the nature of being and of knowledge, which is evident in some shared generic conventions and preoccupations, though each also has other specific interests—technology (science fiction), the social construction of good and evil (detective fiction). The value of intertextuality is that it enables us to see how these texts are constituted by the kinds of meanings they are making. It allows us to understand why two genres that we now consider quite different can have shared ontological and epistemological preoccupations, because of a common generic ancestor.

Like intertextuality, intermediality is about textual practice. We saw above that the interdisciplinarity that was generated by the postmodern recognition of diversity and difference (and hence the rejection of certainty, grand narratives and canonical textuality) enabled the production of a Sherlock that is not a period drama but a contemporary construct, telling stories of today’s world. At the same time, as the brief intertextual study of genre shows, it also deploys a conventional detective with an eccentric mix of scientific method and artistic creativity whose ‘ratiocination’ at times leads him to find villainy not in evil individuals, but in the government and its representatives. Intermediality is useful for mapping that kind of practice, where a narrative devised in one medium is transposed into another where it deploys meanings enabled by its original production, but also produces new and different meanings that are generated via this transposition.

3. Spirituality and stained glass

The stained glass windows in Christian churches deploy a similar practice, taking stories from one medium (the Biblical word of God) and realising them in another medium (coloured glass). Interestingly the windows feature a complex iconography that would appeal to the modern gamer, with icons emblematic of values and ideas that cluster around the central theme and its story arc but open up depths of spiritual meaning. One reading of these windows is that they told these stories for illiterate peasants who had no access to written versions of biblical tales. Roger Homan notes: “The great transept window at Canterbury known as the Biblia Pauperium (poor person’s bible), for example, depends upon an extensive visual vocabulary of symbols and an awareness of the supposed theological links between the biblical scenes featured in adjacent panels” (2005). In this way the windows acted as a point of meditation for the viewer, recalling the story and its religious significance. Homan notes also that many scholars believe that preachers used the windows as a reference point in sermons, especially those delivered in the vernacular of the uneducated. They could literally point to the visual representation of the story and explain their exegesis, so that later viewings of the window would recall not only the details of the story but its religious significance.

In his study, Religious Art in France XIII Century (1913) Émile Mâle begins by noting:

To the Middle Ages art was didactic. All that it was necessary that men should know—the history of the world from the creation, the dogmas of religion, the examples of the saints, the hierarchy of the virtues, the range of the sciences, arts and crafts—all these were taught them by the windows of the church or by the statues in the porch. (vii)

Mâle goes on to explain that this art is not easily decipherable to the modern viewer who may mistake elements of the works as purely figurative, bringing a momentary pleasure to the eye. By contrast: “In mediæval art every form clothes a thought; one could say that thought works within the material and animates it” (viii). Roger Homan adds to this an appreciation of the role of the material used in the art-work:

But there are properties of coloured glass that are of deeply spiritual significance and have been recognized by, for example, Pseudo-Dionysius in the first century and Bishop Grosseteste in the thirteenth. We view not an image but the light beyond which it mediates for us. The image owes its life to that ultimate light. This sense is much keener than it is in respect of the reflection of light upon opaque surfaces. The stained glass image is therefore like an ikon: we are not to look at it but through it. (2005)

If we regard the stained glass window as an intermediated presentation of religious and spiritual concepts and stories, then Homan’s analysis leads us directly to the point of intermediation—the light generated by the glass, which is as critical to the meanings of the windows as the images and icons created. Homan speaks of the role of the stained glass as being “to sedate light”: “A stained glass window slows us down; it inclines us to proceed reverently and lower our voices” (2005). The sensory effect of the coloured light produced by the windows is to remove viewers from the everyday world, locating them in an otherworldly space in which to contemplate religious mysteries and spiritual truths. This is surely the essence of the intermedial experience, not a translation from one art form to another, but a transformation of being and knowing generated by the (sensory) engagement of the viewer. Again note that although intermediality does address the effect on viewers of a particular form of text, its focus is on textual practice rather than audience interaction. Which is to say, the concept of intermediality tends to address primarily the ways in which the text positions the viewer, rather than the multiple active engagements of viewers.

4.   Boba Fett, children’s television and transmediality

The term that seems to best capture the active engagement of audiences or consumers of contemporary texts is transmediality. Henry Jenkins popularised this term in his influential study, Convergence Culture: Where Old and New Media Collide, first published in 2006. Writing about the Matrix phenomenon that had recently developed through the Wachowskis’ interrelated films, games and online comics, Jenkins identifies the work as transmedia storytelling as follows:

A transmedia story unfolds across multiple media platforms, with each new text making a distinct and valuable contribution to the whole. In the ideal form of transmedia storytelling, each medium does what it does best—so that a story might be introduced in a film, expanded through television, novels, and comics; its world might be explored through game play or experienced as an amusement park attraction. Each franchise entry needs to be self-contained so you don’t need to have seen the film to enjoy the game, and vice-versa. (loc. 1974)

This directly confronts older canonical notions of the text as a bounded entity, with the roles of the reader, viewer or listener being to unlock the meaning of that text. Instead it acknowledges the active role of the consumer (who moves between these different media) in creating story and generating meaning that is implicit in the notion of intertextuality. However, this is a different consumer from the medieval worshipper, and the key to that difference is the accessibility of a range of media.

Some thirty years ago, as a creative consultant to a network television producer of children’s programming, my job was to construct the world of a particular television program. Like Lucas’s enormously influential Star Wars series it was set in a different space—a set of planets orbiting a small star, each with their own names and characteristics. I no longer remember the details of the exercise but the project report was about forty pages long, and detailed everything a child might want to know about living on that planet. The aim of the exercise was to create a world that all the separate sequences of the program—games, stories, cartoons, write-in quizzes, the club—could refer back to, so that the show maintained a basic coherence. We wanted our viewers to feel at home in that universe, to feel a sense of engagement and belonging.

Lucasfilm led the way with this kind of world-formation by marketing a series of products that not only capitalised on viewers’ responses to the films, but also provided them with the tools to repeat and enhance that experience imaginatively. And, as Jenkins noted in Convergence Culture, Lucas did not simply endlessly repeat the story of the movie: “When Star Wars went to games, those games didn’t just enact film events; they showed what life would be like for a Jedi trainee or bounty hunter” (2006, loc. 2172). Later in the same chapter Jenkins notes that Lucas found that the value of developing toys based on secondary characters was that they might take on a life of their own: “Boba Fett eventually became the protagonist of his own novels and games and played a much larger role in the later films” (loc. 2273).

Again we might argue that this has happened before, with stories based on earlier texts that expand their imaginary world, including some based on Conan Doyle’s Sherlock Holmes stories: for example, Nicholas Meyer’s novel Seven-Per-Cent Solution ([1974] 1993) presents a back-story to Holmes’ addiction to cocaine (the novel was made into a film of the same name in 1976). What is new, however, is both the number of different media to which consumers have access and the degree to which they can engage with those media. Jenkins quotes Janet Murray’s assessment of the ‘“encyclopedic capacity’ of digital media, which she thinks will lead to new narrative forms as audiences seek information beyond the limits of the individual story” (2006, loc. 2283). Jenkins goes on to argue that, unlike some critics, he does not see this as leading to the death of narrative: “Rather, we are seeing the emergence of new story structures, which create complexity by expanding the range of narrative possibility rather than pursuing a single path with a beginning, middle, and end” (loc. 2323). Of course, it is crucial to know who is developing these new stories and how they relate to the original text.

If we use the example of the Matrix franchise, the whole massive narrative edifice stayed effectively in the control of the Wachowskis. For some viewers it was too complex to try to follow its development and they found the films increasingly difficult to understand, whilst the more dedicated fans were unhappy with the Wachowskis’ attempts to explain every aspect of their narrative, as Jenkins documents (2006, loc. 2436-2446). A fine line exists between the authorial control required to maintain the integrity of the narrative and the dictation of detail that closes down the engagement of the audience. Andrea Phillips discusses this in her practical introduction, A Creator’s Guide to Transmedia Storytelling (2012). She argues “the most effective tool is to actually create a small piece of your world and give it to your audience to play with” (41).

Phillips’ description of transmediality is subtly different from that of Jenkins, perhaps because of their different roles (Jenkins as critic and theorist, Phillips as maker). In her role as storyteller Phillips is concerned not to shut out the audience, so describes her world-building in a way that prioritises audience engagement. In Chapter 8, “Writing for Transmedia Is Different” Phillips notes that “we’ll be concentrating mainly on the requirements of telling a single, highly fragmented story across multiple platforms, and most particularly across digital platforms—you might call it social media storytelling as much as transmedia. That’s because this is where the methods of traditional single-platform or flat narratives become inadequate” (74-75). She goes on to explain this distinction in terms of the strategies used to enable the world of the narrative to be expanded by the audience: “Transmedia storytelling is an exercise in open-ended storytelling, boundless where a traditional single-medium story is finite” (75). Phillips explains that the storyteller should suggest to the audience that the world of the narrative includes more stories than the one that they have been given (75).

As noted earlier, one of the great successes of Star Wars is that its narrative is not confined to a specific set of incidents, rather the narrative contains the seeds of many other stories, featuring characters such as Boba Fett whose role in the core narrative is relatively minor but has the potential for new storytelling and world-building. By contrast, die-hard Matrix fans were disappointed when the Wachowskis attempted to lock down the meanings of the trilogy to a specific story by resolving the mystery, leaving little scope for imaginative retellings by fans. Instead Phillips notes the value of deliberately leaving loose ends that might become the source of new stories, which directly contradicts conventional advice given to writers. Though she also notes that these narrative possibilities have to be executed judiciously so that you do not “accidentally create narrative expectations that never achieve any kind of payoff” (76). Hence her earlier point about the importance of a clear story arc: “It is especially important in transmedia to have a plot that goes from beginning to end before you launch” (57). Another strategy to enhance narrative openness is “to create story elements in one medium that have their payoffs in another medium” (78), such as a game based on a film. All of this has to be achieved in relation to the basic premise with which she opens the study: “every single element of a transmedia story has to be fulfilling a narrative purpose, without exception” (40-41). And as she notes the aim of transmedia storytelling, as well as the marketers who use it, is engagement: “Transmedia storytelling can provide more engagement and more potential points of sale for any given story, and when it’s done well, each piece can effectively become a promotional tool pointing toward every other piece of the whole” (39). Every strategy used by the storyteller, therefore, should be about giving the audience “things to do, not just things to consume” (117).

Phillips’ Guide addresses textual practice directly in relation to audience or consumer engagement, though Phillips also stresses the need for a critical understanding of textuality (63). This engagement is the both the reason for transmedia production (to sell products, to tell a story) and the result of audience access to multiple media. As Phillips reiterates in her book, this engagement, and the textual openness that enables it, makes transmedia storytelling different from earlier forms of media narratives and audience-media relationships.

The Matrix (1999)

The Matrix (1999)

5. The joy of discovery and the fossilised dolphin

I return here to Jenkins’ crucial insight in Convergence Culture, that this different form of storytelling, described so well by Phillips, and common to the popular culture that preoccupies most children, signifies a new way of being and knowing:

Our workplaces have become more collaborative; our political process has become more decentered; we are living more and more within knowledge cultures based on collective intelligence. Our schools are not teaching what it means to live and work in such knowledge communities but popular culture may be doing so. (2006, loc. 2477)

For Jenkins this makes literacy training for children essential so that they can “develop the skills needed to become full participants in their culture” (loc. 5295), as Phillips argued when she stressed the need to be critical. The joy of transmedia engagement is that of discovery, of finding a way to contribute to the meanings of a text through your own creativity so that your stories are woven into that ever-expanding composite text. As Jenkins notes, however, this is more than a solitary venture. It is about being able to collaborate with others and to contribute to a collective venture without feeling a loss of individual achievement.

Digital technology has enabled this kind of sharing on an extraordinary scale—whether through kids playing games online with others across the globe, researchers collaborating on a project across cities, countries or continents or fans world-wide expanding a beloved narrative. It is also evident in the ways that older media such as radio and television use online resources to expand their research, engage their audiences, and incorporate audience responses and knowledge into their broadcast formats. Museums and libraries too are sharing resources and inviting visitors to become part of the knowledge-production for the institution. For example, by checking the digitisation of older manuscripts and newspapers for verisimilitude. On the one hand, this reflects economic necessity and the poor resourcing of many public institutions. On the other hand, it creates a wholly different, expanded knowledge base for the library, an enhanced level of engagement for visitors. Effectively, this visitor/user involvement changes the nature of the library from that of a central authority giving access to knowledge to a collaborative, creative, knowledge-building project. In December 2013 the British Library released an archive of over 1,000,000 images onto Flickr Commons for free use and reproduction. Dan Colman reported in Open Culture (2013):

The librarians behind the project freely admit that they don’t exactly have a great handle on the images in the collection. They know what books the images come from. (For example, the image above comes from Historia de las Indias de Nueva-España y islas de Tierra Firme, 1867.) But they don’t know much about the particulars of each visual. And so they’re turning to crowdsourcing for answers. In fairly short order, the Library plans to release tools that will let willing participants gather information and deepen our understanding of everything in the Flickr Commons collection.

Many other libraries and art galleries around the world have released part of their archives to open access and at the same time invite visitors to join them in becoming producers of knowledge.

Recently the Smithsonian Museum in Washington D.C. announced Smithsonian X 3D, a web portal that enables visitors to use the museum’s 3D scans of artefacts to build their own models using 3D printers. Günter Waibel, Director of the Digitization Program Office, explains:

These projects indicate that this new technology has the potential not only to support the Smithsonian mission, but to transform museum core functions. Researchers working in the field may not come back with specimens, but with 3D data documenting a site or a find. Curators and educators can use 3D data as the scaffolding to tell stories or send students on a quest of discovery. Conservators can benchmark today’s condition state of a collection item against a past state—a deviation analysis of 3D data will tell them exactly what changes have occurred. All of these uses cases are accessible through the Beta Smithsonian X 3D Explorer, as well as videos documenting the project. For many of the 3D models, raw data can be downloaded to support further inquiry and 3D printing.

And he concludes:

With only 1% of collections on display in Smithsonian museum galleries, digitization affords the opportunity to bring the remaining 99% of the collection into the virtual light. All of these digital assets become the infrastructure which will allow not just the Smithsonian, but the world at large to tell new stories about the familiar, as well as the unfamiliar, treasures in these collections.

This venture confirms many of Jenkins’ earlier predictions about how digital technologies will change our ways of producing knowledge. One of the artefacts currently available is the fossilised skull of an unknown species of dolphin, found in rocks that are 6-7 million years old. The Smithsonian X 3D website now supplies the software and instructions to print your own 3D copy of the skull. Even though this will not be the original skull, the value of a tactile engagement with the reproduction should not be underestimated. As a number of recent studies have argued (see Classen 2005, 2012; Howes 2005; Chatterjee 2008; Candlin 2010; Cranny-Francis 2013) tactile contact, indeed all kinds of sensory engagement, generate bodily responses that in turn produce new ways of knowing and understanding an object and our relationship to it. By sharing these knowledges, we learn more about not only the objects, but also ourselves.

6. Conclusion

The terms intertextuality, intermediality and transmediality map the development of new communication technologies through the twentieth and into the twenty-first century. They all effectively interrogate older canonical notions of textuality and of reading, as closed practices controlled by the author. Intertextuality was used to argue that texts have never been closed but part of an infinite conversation to which all texts contribute, and that each textual reading adds another voice to the conversation. Intermediality reflected the beginnings of popular access to multiple media, enabling users to explore the ways in a particular narrative or text may be transposed from one medium to another, expanding or enhancing the original story or idea. Transmediality is an articulation of convergence culture, whereby audiences are able easily to traverse and correlate a range of media in order to explore a complex and growing narrative or argument. The difference between intermediality and transmediality is not simply quantitative, however, it reflects a new way of understanding our relationship to texts, knowledge, and each other. It reflects, as Jenkins notes, the development of a collective knowledge culture in which collaboration is a key component of thinking and being. Further, the materials and practices that new technologies are making available, which incorporate bodily knowledges into this collaborative production of knowledge, presage new kinds of understanding and self-knowledge. As both Jenkins and Phillips argue above, the element required to leaven this heady mix is critical awareness—of the texts we produce and the meanings we make.

 

References

Bakhtin, Mikhail. 1984. The Dialogic Imagination: Four Essays. Translated by Michael Holquist and Caryl Emerson. Austin, TX: University of Texas Press.

Barthes, Roland. 1977. Image, Music, Text: Essays Selected and Translated by Stephen Heath. Translated by Stephen Heath. London: Fontana.

Candlin, Fiona. 2010. Art, Museums and Touch. Manchester: Manchester University Press.

Chatterjee, Helen, ed. 2008. Touch in Museums: Policy and Practice in Object Handling. Oxford and New York: Berg.

Classen, Constance. 2012. The Deepest Sense: a Cultural History of Touch. Chicago: University of Illinois Press.

Classen, Constance, ed. 2005. The Book of Touch. Oxford: Berg.

Colman, Dan. 2013. “The British Library Puts 1,000,000 Images into the Public Domain, Making Them Free to Reuse & Remix.” Open Culture, December 1. Accessed January 23, 2014. http://www.openculture.com/2013/12/british-library-puts-1000000-images-into-public-domain.html.

Conan Doyle, Sir Arthur. 1981. “The Hound of the Baskervilles.” In The Penguin Complete Sherlock Holmes, 669-768. Harmondsworth: Penguin.

Cranny-Francis, Anne. 2013. Technology and Touch: the Biopolitics of Emerging Technologies. London: Palgrave.

Herzogenrath, Berndt, ed. 2012. Travels in Intermedia(lity): reblurring the boundaries. Kindle edition. Hanover, NH: Darmouth College Press.

Higgins, Dick. 1966. “Synaesthesia and Intersenses: Intermedia”, originally published in Something Else,Newsletter1, No.1 (Something Else Press). Accessed May 19, 2014. http://www.artesonoro.net/artesonoroglobal/intermedia.html.

Homan, Roger. 2005. “Who Looks on Glass? The Spiritual Significance of Stained Glass.” The Social Affairs Unit, August 3. Accessed January 23, 2014. http://www.socialaffairsunit.org.uk/blog/archives/000536.php.

Homan, Roger. 2006. The Art of the Sublime: Principles of Christian Art and Architecture. Farnham: Ashgate.

Howes, David, ed. 2005. Empire of the Senses: the Sensual Culture Reader. Oxford: Berg.

Jenkins, Henry. 2006. Convergence Culture: Where Old and New Media Collide. Kindle edition. New York and London: New York University Press.

Lavigne, Carlen. 2012. “The Noble Bachelor and the Crooked Man: Subtext and Sexuality in the BBC’s Sherlock” in Sherlock Holmes for the 21st Century: Essays on New Adaptations.Kindle edition, edited by Lynette Porter, 13-23. London: McFarland & Company.

Mâle, Émile. 1913. Religious Art in France XIII Century: A Study in Mediaeval Iconography and Its Sources of Inspiration. Kindle edition. London: Dent.

Meyer, Nicholas. (1974) 1993. The Seven-Per-Cent Solution. New York and London: W.W. Norton.

Phillips, Andrea. 2012. A Creator’s Guide to Transmedia Storytelling: How to Captivate and Engage Audiences Across Multiple Platforms.Kindle edition. New York: McGraw-Hill.

Poe, Edgar Allan. 2010. The Dupin Mysteries with The Gold Bug. London: Capuchin Classics.

Shelley, Mary. (1818) 1982. Frankenstein or The Modern Prometheus, edited by Maurice Hindle. Harmondsworth: Penguin.

Stein, Louisa Ellen, and Kristina Busse, eds. 2012. Sherlock and Transmedia Fandom: Essays on the BBC Series. Kindle edition. London: McFarland & Company.

Waibel, Günter. “About Smithsonian X 3D.” Smithsonian X 3D. Accessed January 23, 2014. http://3d.si.edu/about.

 

Filmography:

Cox, Michael. 1984-1994. The Adventures of Sherlock Holmes.London: BBC.

Doctor Who. 2005 -. Wales, UK: BBC; Canada: CBC.

Gatiss, Mark, and Moffat, Steven. 2010-. Sherlock. London: BBC.

Gatiss, Mark, and Moffat, Steven. “The Hounds of Baskerville.” Sherlock, series 2, episode 2. Original airdate 8 January 2012. London: BBC.

Lucas, George. 1977-2005. Star Wars, Episode I-VI. USA: Lucasfilm.

Ritchie,Guy. 2009. Sherlock Holmes. USA: Warner Bros.

Ritchie,Guy. 2011. Sherlock Holmes: A Game of Shadows. USA: Warner Bros.

Ross, Herbert. 1976. The Seven-Per-Cent Solution. USA: Herbert Ross Productions, Universal Pictures.

Scott, Ridley. 1982. Blade Runner.USA: Ladd Company, Shaw Bros, Warner Bros.

The Wachowski Brothers. 1999. The Matrix. USA: Warner Bros.

 

Notes:

[1]Steven Moffat has been reported as saying that he sees Sherlock as asexual. However, the iconography used with Sherlock and the way in which his relationships with Watson and Moriarty (among others) are presented allow for the many fan readings of him as gay or bisexual—as Carlen Lavigne argues (2012).

[2]References to Kindle books are given as locations, unless the book also provides page numbers.

Bio: Anne Cranny-Francis is Professor of Cultural Studies at the University of Technology Sydney. Her recent work includes ARC funded projects on the sense of touch and its deployment by new technologies, described in Technology and Touch: the Biopolitics of Emerging Technologies (Palgrave, 2013), and on ex-patriot Australian writer, Jack Lindsay.

 

‘God Hates Fangs’: Gay Rights As Transmedia Story in True Blood — Holly Randell-Moon

Abstract: In this paper I examine the television program True Blood’s allusions to gay liberation in terms of the biopolitical and neoliberal implications of consuming civil rights as a transmedia story. In the program, vampires have ‘outed’ themselves to the population at large and in conjunction with the invention of synthetic blood (Tru Blood) are able to publicly participate in social and economic activities without harming humans. Home Box Office’s (HBO) use of Tru Blood to market the show is premised on the commodification of a (vampire) rights based movement across a range of different story-telling mediums. On the one hand, this means that the program is drawing attention to the biopolitical function of rights discourse by suggesting that it is the management of particular kinds of life, through particular kinds of consumption, which remains valuable to the dominant political and economic order. On the other hand, the mapping of vampirism onto civil rights also functions to legitimise a political discourse wherein the purported social ‘harm’ of granting minority groups equal rights can be mitigated by market forces and the cultivation of a constituency whose political power is linked to their ability to consume. The consumption of the True Blood story by fans thereby enacts principles of biopolitical management and containment of civil rights groups through HBO’s and fans’ willingness to enact play-political consumption and performance of rights in a transmediated public sphere.

rm1The television series True Blood (HBO, 2008-2014), based on The Southern Vampire Mysteries novels by Charlaine Harris, features a number of allusions to gay liberation and lesbian, gay, bisexual, transgender and intersex (LGBTI) politics in its depiction of ‘vampire rights’. In the fictional town of Bon Temps, in Louisiana, United States, where True Blood is set, vampires have ‘outed’ themselves to the population at large and in conjunction with the invention of synthetic blood (Tru Blood) are able to publicly participate in social and economic activities without harming humans. The production of Tru Blood as a commodity enables individual and collective groups of vampires to advocate for the civil and political rights enjoyed by humans. In the vampires’ attempts to become part of ‘mainstream culture’, there are several references to gay liberation. These include the American Vampire League, whose activism and media interventions mirror that of groups such as the Human Rights Campaign, the use of the phrase ‘coming out of the coffin’ to describe the increasing numbers of vampires publicly acknowledging their existence to humans, and the prejudice directed at vampires by humans, particularly by those with conservative or evangelical Christian beliefs. This specific cultural, political and religious milieu for vampire rights is telegraphed in the opening title sequence by a brief shot of a church sign, which reads, “God Hates Fangs”. Amongst the ostensibly non-fictional images of Southern quotidian life—swamps, road kill, baptisms, church choirs, bar brawls—it is the only indication in the sequence of the program’s focus on the supernatural.

The diegetic plausibility of the vampire liberation movement is aided by various transmedia paraphernalia simultaneously operating outside of and in relation to events in the show’s narrative. This includes the availability of Tru Blood beverages and merchandise, Facebook and social media material for the advocacy groups featured within the show and partnerships between Home Box Office (HBO—the channel that broadcasts True Blood) and advertising companies, such as Geico insurance, to produce fictional campaigns targeted explicitly towards vampire consumers but implicitly, True Blood fans. In this extension of the program’s narrative of vampire rights to other types of media and forms of consumption, True Blood is exemplary of the new practices of transmedia storytelling championed by Henry Jenkins. He defines transmedia as

a process where integral elements of a fiction get dispersed systematically across multiple delivery channels for the purpose of creating a unified and coordinated entertainment experience. Ideally, each medium makes its own unique contribution to the unfolding of the story. (Jenkins 2011; original emphases)

For Jenkins, this type of storytelling enables and builds on audience participation in the meaning-making process of media texts (2006). This mode of storytelling is also closely associated with viral marketing, which utilises “pre-existing social networks like websites and YouTube in order to increase franchise or brand awareness” (Ndalianis 2012, 164). Transmedia forms of storytelling, like those employed for True Blood, can be quite complex and multi-faceted, involving the extension of a text across not only different types of media but also different geographical locations and consumer activities. In her excellent book, The Horror Sensorium (2012), Angela Ndalianis details transmedia stories and campaigns involving scavenger hunts, political rallies, social media tourism and urban graffiti that centre on the production of an embodied fan relationship with media texts. She argues that the transmedia stories deployed for texts such as The Dark Knight (Christopher Nolan, 2008), Lost (Walt Disney Studios Home Entertainment, 2004-2010)and True Blood “address the fiction/reality interplay by mitigating their stories more invasively into the social sphere” (165). They do this by encouraging fans and consumers to become ‘actors’ in a transmedia performance of a ‘living’ narrative (166). This performance produces a kind of meta-affect because fans “extract cerebral and sensory pleasure participating in and contributing to a highly crafted fictional world that’s in the process of unveiling itself” (169). An example of this type of meta-affective performance occurred in early 2009, in Auckland, New Zealand, when a series of wooden posters advertising True Blood were installed along public streets. Featuring information about True Blood’s airdate (the series was premiering on New Zealand television at this time), the posters had “In case of vampire” written across the top and “Snap here” at the bottom presented alongside flat wooden stakes. Potential fans and viewers of True Blood were invited to participate as performers in the program’s narrative by exercising vigilance and protection from the newly outed vampires by snapping off a wooden stake and carrying the physical textual detritus into their everyday lives.

trubloodbotWhat structures this kind of performance and participation by fans is the story and narrative used to extend a text via transmediation. In this paper I want to examine the execution of True Blood’s transmedia storytelling through a narrative of vampire rights that alludes to civil rights debates around gay liberation. I want to focus on the specifically transmedia dimensions of this narrative and how this particular media form interpellates viewers into a biopolitical and neoliberal mode of consuming civil rights. The program’s use of Tru Blood, both intra- and extra-textually, is premised on the commodification of a rights based movement across a range of different story-telling mediums. On the one hand, this means that the program is drawing attention to the biopolitical function of rights discourse by suggesting that it is the management of particular kinds of life, through particular kinds of consumption, that remains valuable to the dominant political and economic order. On the other hand, the mapping of vampirism onto civil rights also functions to legitimise a political discourse wherein the purported social ‘harm’ of granting minority groups equal rights can be mitigated by market forces and the cultivation of a constituency whose political power is linked to their ability to consume. Fans’ affective investment in vampire rights is then managed via consumption in a transmedia format that mirrors biopolitical strategies of management and containment of minority groups through civil rights discourse.

“No darlin’, we’re white, he’s dead”: Vampires and biopolitics

In her essay “Technologies of Monstrosity”, Judith Halberstam argues that “[a]ttempts to consume … vampirism within one interpretive model inevitably produce vampirism. They reproduce, in other words, the very model they claim to have discovered” (1993, 334). For this reason, in her analysis of Bram Stoker’s Dracula she argues that the central figure is “not simply a monster, but a technology of monstrosity” (334). Representations of monstrosity in texts like Dracula function not so much to reify particular characteristics of monstrosity (be it sexual immorality or corporeal difference) but to produce and disseminate particular discourses constituted as monstrous. So if we take a particular representation of vampires to signify for example, minority rights, we are also at the same time producing an understanding of what minority rights mean in popular and political culture.

Given that monstrosity is typically construed as a threat to human life, textual portrayals of monstrosity are also concerned with the management of that threat and the balancing of the value of human life with the containment of monstrosity. The development and application of various governmental strategies designed foster the life and health of citizens is defined by Michel Foucault as biopower (1991b, 263). In order to maximise the economic productivity of the state, governments and state institutions have “to qualify, measure, appraise, and hierarchize … the living in the domain of value and utility” (1991b, 266). One way to organise social practices around ‘value and utility’ is to encourage citizens to invest in a racialised and heteronormative construction of the family as the site through which life can be fostered or neglected (1991a, 99). As the management of the economic and social life of the polity comes to pivot on heterosexual familial reproduction, non-heterosexual or non-normative sexualities can be positioned in biopolitical terms as threats to the ‘health’ and productive order of a society. In her essay “Tracking the Vampire” Sue-Ellen Case explains:

From the heterosexist perspective, the sexual practice that produced babies was associated with giving life, or practicing a life-giving sexuality, and the living was established as the category of the natural. Thus, the right to life was a slogan not only for the unborn, but for those whose sexual practices could produce them. In contrast, homosexual sex was mandated as sterile—an unlive practice that was consequently unnatural, or queer, and, as that which was unlive, without the right to life. Queer sexual practice, then, impels one out of the generational production of what has been called “life” and historically, and ultimately out of the category of the living. (1991, 4)

In a biopolitical paradigm, subjects deemed unable to contribute productively to the life of a society can be excluded from the rights and protections offered by that society. This exclusion is then overlain with a naturalising discourse, which works to justify the asymmetries of legal and social recognition as simply part of the ‘natural order of things’. This is why Case sees a link between the cultural discourses used to frame both vampirism and homosexuality. In a dominant heteronormative order that conflates a particular kind of social and political life with life itself, both vampirism and homosexuality become aligned with death or unlife.

rm3The representation of the various kinds of harm vampire rights pose to humans in True Blood then seems an apposite metaphor for the biopolitical exclusion of LGBTI people from certain state-based rights. As a number of scholars have pointed out, True Blood’s treatment of vampiresis characteristic of a wider shift in textual portrayals of vampires “from the right to exile … to the right to citizenship in the postcolonial United States” (Hudson 2013, 663). Bernard Beck sees “[t]he plain message of today’s vampire lore” as evidence “that we are becoming less fearful and hostile, more curious and sympathetic to those we insist on defining as strangers” (2011, 92). This narrative shift from exclusion to inclusion in representations of vampiric difference is reflective of a broader social and political consensus around managing minority groups through integration rather than expulsion from a neoliberal economic order. Deborah Mutch notes that the narrative framework for the acceptance of vampires in book series such as Twilight and The Southern Vampire Mysteries are premised on “accepting human definitions of nation and race which are then superceded by globalised trade” (2011, 75).

While the supernatural genre has the ability to, as Dale Hudson puts it, “decolonize our familiar habits of thinking”, particularly with respect to cinematic and televisual “political realism” (2013, 662), textual portrayals of supernatural creatures nevertheless tend to incorporate dominant biopolitical conceptions of human life as the normative narrative bedrock against which other kinds of lives or living is measured. Hudson points out that in True Blood, vampirism is constituted as species difference through reference to characters as ‘vampire Bill’, whereas human characters are not described as ‘white Jason’ or ‘black Tara’ within the diegesis of the show (666). Where vampirism is discursively positioned as bodily distinct from human-ness, the nation on which this embodiment is placed remains invisible. True Blood’s representation of First Nations peoples and their interaction with vampires (those old enough to have arrived in North America during colonisation) is limited enough to suggest an erasure of colonialism as significant to the historical formation of the United States. As Hudson notes, “Indigenous nations appear only in the realm of the supernatural in True Blood” (669). For Hudson, the program’s use of the supernatural allows an imagining of “the New South as a space inhabited by multiple species on multiple planes of reality” (664), which invites consideration of “the right to rights” (685). My interest in this paper is how True Blood’s portrayal of “the right to rights” is linked to the public management and presentation of rights-based groups via transmedia texts, which are dependent on public forms of consumption and fan activity.

“You are not our equals. We will eat you. After we eat your children”: Vampire rights

In True Blood’s narrative conflicts around vampire rights, there are several allusions to civil rights and equality movements. The series has been received predominantly as a commentary on gay liberation. A New York Post article, for example, contends that “the fictional vampires’ quest for the same rights and social acceptance enjoyed by” humans “has become synonymous with the very real fight for gay rights” (Shen 2009). The author of the novels on which the show is based also seems to encourage this association (see Solomon 2010). As with the gay rights movement, vampires’ attempts to achieve equality are perceived by their opponents as a threat to the social and cultural stability of the polity they inhabit. However, the crucial difference between vampires and LGBTI peoples is that the alleged ‘harm’ posed to society by granting the latter civil rights is symbolic and imagined whereas vampires, within the diegesis of the show, do perpetrate considerable violence. In this vein, a reviewer of the show opined, “[t]hese vamps are assholes, not oppressed minorities. They deserve to be hated. If these murderous, evil creatures are figures for gay people, then they are figures for the religious right’s worst nightmare of what gay people are” (Newitz 2008). The program’s creator, Alan Ball, also avers with this reasoning “because the vampires on our show are, for the most part, vicious murderers and predators, and I’m gay myself, so I don’t really want to say, ‘Hey, gays and lesbians are basically viciously amoral murderers’” (Grigoriadis 2010).

outdoor-advertising-aimed-at-vampiresThe question of whether rights should be reserved only for those who are morally deserving is addressed in an interesting way by the American Vampire League (AVL) within the show. In the first episode (“Strange Love”, 1.1), the AVL spokesperson, Nan Flanagan (in an interview with Bill Maher) refutes assertions that vampires perpetrate large-scale murder and assault against humans (for lack of documented evidence) and counters that humans themselves are responsible for slavery and genocide. Later on in the series, another vampire Russell Edgington uses this same logic—humans have caused irreparable damage to the environment and the species they share it with—to reach a very different conclusion regarding vampire-human relations. For Edgington, vampires are right to insist on their superiority to and difference from humans. He broadcasts these views on a live news program and after deboning the anchor, proclaims to the human audience, “You are not our equals. We will eat you. After we eat your children” (“Everything is Broken”, 3.9). Human anti-vampire bigotry meanwhile stems from a corporeal vulnerability to vampires’ biological requirement for human blood. In its extreme form, anti-vampire prejudice manifests as a speciest right to survival exercised by vigilante groups such as the one seen in Season Five. This group of men don Barack Obama masks as they inflict violence and in some cases, death, upon vampires and other supernatural beings. This group mentions and appears to be linked to the ‘Keep American Human’ movement, which has its own website and promotional material. This doubly imbricated right to ‘America’ and to life is framed by anti-vampire humans as exclusive. One of the vigilante characters complains, “it’s some sort of crime now being a regular old human” (“In the Beginning”, 5.7) as if the uniqueness of being human cannot be co-extensive with the existence of other species.

Vampire prejudice thus goes beyond the simple fear of death or bodily harm and involves a speciest condemnation of vampire existence that is often inflected with a moral discourse. When the show begins, vampires have achieved a limited degree of civil equality such as the right to marry (in certain states in the US and if the unions are heterosexual) and are protected by anti-discrimination laws (businesses cannot refuse to serve vampires as customers), which are reluctantly enforced by police. There are also a series of moral and social codes, centred primarily on sexuality, that police vampire and human interactions. Humans who engage with or are thought to engage in sexual relations with vampires are derisively referred to as “fang-bangers”. The central character Sookie Stackhouse is often judged negatively in terms of her moral standing and character for her relationship with the vampire Bill Compton. The first season features a violent expression of this chauvinism in the form of a serial killer with a pathological hatred of women who sleep with vampires.

The corporeal vulnerability of humans to vampire attack is balanced by the portrayal of vampire blood as producing hallucinatory and amphetamine-like effects when consumed by humans. Vampire blood or V-juice is a highly sought-after but illegal commodity associated with the vampire bar scene and fang-bangers, which may allude to subcultural forms of clubbing and recreational drug use. In Season One, a lonely vampire named Eddie claims that he can only express and act on his homosexual orientation by trading his blood for sexual favours with human men (in particular Sookie’s co-worker and friend, Lafayette Reynolds). In an inversion of the life-giving connotations of heterosexual sex, one scene in the first season shows Sookie’s brother Jason and his girlfriend consume V-juice and make love whilst Eddie is tied up and tortured in the basement below them. Here it is an undead subject whose blood provides the impetus and facilitation of heterosexual sex.

The moral repugnance at the tarnishing of human life and sexuality bought about by vampire-human contact is aligned with most (although not all) forms of Christianity in True Blood. The second season features an evangelical group called the Fellowship of the Sun that promotes “pro-livin’ values” (Home Box Office 2012) and warns the human polity about the dangers of vampire rights and the “the wing nuts on the left” who advocate for them (“The Fourth Man in the Fire”, 1.8). In a television interview, the pastor of the church, Reverend Steve Newlin, explains that vampire rights threaten “the rights of our sons and daughters to go to school without fear of molestation by a bloodthirsty predator in the playground or in the classroom” (“The Fourth Man in the Fire”, 1.8). One of the advertisements produced by the Fellowship of the Sun, not featured in the show but distributed online and in poster form in some cities, depicts a young blonde boy with the caption, “To them he’s just a midnight snack” (Ndalianis 2012, 178).

The figure of the child here is important as Ben Davies and Jana Funke note, “the teleology of straight time is projected onto the sex act, which displaces its own meaning, significance or indeed non-significance for the production of the future” (2011, 6). In this way, the future viability of a heterosexual society is linked to the purity and protection of children. In a video press release for the advertising campaign, the elder Reverend Theodore Newlin passionately declares, “our children are our most precious resource, our lifeblood” (the video appears on YouTube under the category ‘Nonprofits & Activism’). On the Fellowship’s website, homosexuality is listed alongside vampirism as a social danger: “It’s nothing new for teenagers and young adults to flock to the newest trend, and it’s hardly uncommon for these fashion choices to be self-destructive, like smoking, drugs, tattoos or homosexuality. But the latest fad—a soulless eternity of drinking blood—can’t be undone with a laser treatment or rehab. Vampirism is forever” (Home Box Office 2012). While some organisations and US Republican presidential candidates view homosexuality as a choice or temporary lifestyle that can be cured or corrected, what makes vampirism especially pernicious for the Fellowship is that it cannot be erased or overcome, it’s “forever”. In another television interview, the younger Reverend Newlin says, “the vampires as a group have cheated death. And when death has no meaning, then life has no meaning. And when life has no meaning, it is very, very easy to kill” (“Nothing but the Blood”, 2.1).

Anti-vampire sentiment is not an opposition to the merits or otherwise of particular vampire rights, rather the opposition stems from the consequence that these rights serve to entrench vampire presence in civil and social spaces. It is precisely because vampirism constitutes a permanent state of being that the necessity of repealing vampire rights takes on an apocalyptic sense of urgency. Such rhetoric alludes to and perhaps parodies anti-gay rights activism, particularly the National Organisation for Marriage’s (NOM) Proposition 8 “gathering storm” commercials which featured activists and citizens expressing concern about marriage equality backgrounded by blue screens depicting severe lightening storms and flooding. Here the public recognition of difference is conflated with disaster. In the type of advocacy employed by the Fellowship of the Sun, and NOM, the out-group’s very existence seems to imperil a safe and normal social and political order.

Where NOM’s advocacy and rhetoric is left open to debate and parody in the marketplace of democratic political suasion, the Fellowship is clearly set up as an object of ridicule within True Blood. First Newlin (in Season Two) and then his wife Sarah (in Season Six) are positioned as villains whose attempts to instigate genocidal war against vampires figure as obstructions and then climatic battles against which Sookie and friends must contend. Hudson argues that “Steve’s punishment is to be ‘made’ vampire, presumably unleashing his latent desires for Jason” and he “becomes a self-defined ‘gay vampire American’” (2013, 672). Such a transformation is presented humorously as a revelation of the character’s moral and political hypocrisy because his hatred of vampires is ostensibly linked to a self-hatred of his orientation. The reading of groups such as the Fellowship as opposed to progressive social and political causes is reflected in scholarly and popular reception of the show. For example, J. M. Tyree explains the premise of True Blood by noting, “The resistance movement to vampire rights is formed out of the ideological dregs of fundamentalist Christianity” (2009, 32). An online recapper describes the vigilante Keep America Human group as “a bungling bunch of bigoted idiots who spew thinly veiled Fox News talking points like ‘lamestream media’” (Berkshire 2012).By framing the Fellowship and Keep America Human’s advocacy against vampires as villainous, True Blood can be seen as participating in progressive representations of civil rights wherein “proclaiming a future in which the current resistance to gay marriage will seem backward” allows those subjects who already accept civil rights to be “projected forward in time” (Davies and Funke 2011, 6).

True Blood’s vampire rights narrative enables the production and facilitation of a set of transmedia texts framed around advocacy. As various groups within the show vie for political, cultural, economic and species preservation, this sets up an affective biopolitical participation wherein fans and reviewers debate the merits of civil rights, equality and state protection. A positive reading of this biopolitical transmedia engagement with the show is that a popular political consensus around inclusion and integration encourages fans to view the contribution of violence and essentialised forms of prejudice to political debate in negative terms—whether in the form of the Fellowship’s moral inflection to humans’ right to life or vampires’ reduction of human ontological existence to food. In the next section of the paper, I want to unpack the implications of how this fan engagement with the biopolitics of vampire rights is achieved through transmedia storytelling as a specifically commodified activity.

“There’s no such thing as bad; or time for that matter”: Vampires and neoliberalism

Aside from some obvious corporeal differences—fast movement, sharp orthodontics, sartorial preference for dark, binding clothing—vampires in True Blood attempt, for the most part, to fit into the social and cultural environment around them. In an interview for The New York Times Harris explains that her vampires “are more sympathetic” than previous sanguisuge incarnations. Of Dracula she says: “He had disgusting personal habits. He had the three wives; he crawled up the sides of the buildings; he had the sharp teeth and fingernails. Mine are at least trying to look like everyone else, but it’s not working out too well for them” (Solomon 2010). While earlier representations of vampires tended to exacerbate their monstrosity as difference, in Harris’ novels and its televisual counterpart, monstrosity is framed around the problem with assimilation to a human-centred social and political order. This integration is premised on the presence of a biotechnological industry, economic infrastructure and political consensus enabling them to do so.

The AVL is able to advocate for the public acceptance of vampires, on the basis that they do not pose a threat to humans, because of the development of the synthetic Tru Blood replacement for human blood. Originally developed by a Japanese biomedical company as a solution for human blood loss and transfusions, an accidental side effect is that the product can provide sustenance to vampires. Thus while the show centres around the politics of integration, the fulcrum for this integration is the successful branding and marketing of Tru Blood as “a globally transported commodity” (Mutch 2011, 81). The second vampire we see in True Blood is shown purchasing the beverage from a 7-Eleven style convenience store. In this opening scene of the first episode, two bored white teenagers eagerly approach the store clerk, fashioned in dark clothing, piercings and long black hair, to inquire about the possibility of scoring V-juice. The clerk indulges the potential V customers, menacing them with intimations of violence, before abruptly revealing his status as human, to the delight of the male teenager and relieved anger of his female counterpart. A burly gentleman in military garb and a cap adorned with a Confederate flag comes forward to express his displeasure with the ruse. After the male teen excoriates the customer by saying, “fuck you Billy Bob”, ‘Billy Bob’ reveals his fangs and responds, “Fuck me. I’ll fuck you boy. I’ll fuck ya’ and then I’ll eat ya’” (“Strange Love”, 1.1). The vampire’s interactions with both the clerk and the young couple subvert generic expectations, from the characters within the show as well as the audience, of the vampire as reclusive and gothic. Hudson reads this scene as evoking “the lingering embers of ‘lost cause’ for white-male-human privilege” where “the privileged position of the white-male-human in the Old South might be restored only in supernatural terms in the New South” (2013, 672). Now a vampire, the Southern white Confederate man can still expect his purchasing power and public presence to proceed without humiliation or impediment.

The development and dissemination of Tru Blood for public consumption creates new forms of human and vampire interaction, which diverse sets of stakeholders attempt to negotiate and regulate in different ways. The AVL attempts to gain political enfranchisement through a Vampire Rights Amendment (VRA) while other supernatural species, such as werewolves, wait cautiously to see how vampires are treated before likewise revealing themselves publicly (Hudson 2013, 665). The means through which a pharmaceutical product propels the development of vampire rights reinforces Halberstam’s point that Gothic monstrosity is always “an aggregate of race, class, and gender” (1993, 334). In order to participate as good biopolitical citizens, vampires must have the capital to access Tru Blood as well as the legal protection to purchase and consume the product in a discrimination free environment. The fake commercials for Tru Blood, released on YouTube, attempt to help this economic and political process along by portraying Tru Blood consumption as alternatively cool and sexy or folksy and non-threatening. For example, in one commercial, three young white men approach a bar and place their orders in quick succession:

I’ll take that vodka with the really cool ad campaign.

Ridiculously expensive imported beer with a name I can’t pronounce.

I’ll have one of those exotic cocktails.

Their requests are interrupted by a conventionally attractive white woman who orders Tru Blood and then carries it to her wan date, languishing in the shadows of the bar. The men stare at the Tru Blood customer in astonishment and awe. The ad ends with the tagline, “Tru Blood, because you don’t need a pulse to make hearts race”:

The commercial has no branding for True Blood or HBO and is a self-contained transmedia text—the Tru Blood logo shown at the end even has small legalise advising potential consumers, “Synthetic blood products contain varied cellular content than actual blood. Please consult a Tru Blood Cellular Specialist for specific nutritional information”. True Blood fans are addressed as both consumers of the show and of the fictional Tru Blood beverage. These fans are positioned as savvy and media literate cognisors in a way that disarms the purpose of both the True Blood text and the Tru Blood advertisement to establish a blatantly commercial relationship with fans through a postmodern knowingness of alcohol marketing. The intended affective response here, as per Ndalianis, is to generate meta-pleasure in recognising the text’s transmedia connection to the show (in the absence of specific show branding) amidst the generic conventions of alcohol commercials.

Another commercial features a group of mostly white men camping and enjoying beer around a fire. We then see the group through a point of view shot from the darkness in a way that appears to show a predator sneaking up on them. In a reverse shot, a vampire emerges behind one of the men and snarls. The men are startled and then begin to laugh as they welcome the vampire as a recognised friend. “You boys got something for me to drink?” the vampire chuckles as his friends hand him a Tru Blood.

These commercials generate a convivial affective connection to the show anchored through transmedia commodity relations that mirror the internal commodity relations between characters in True Blood. The success of Sookie and Bill’s relationship for example, is implicated in the proliferation of cheap pharmaceutical substitutes. After a passionate bout of lovemaking and bloodletting, Bill tenderly instructs Sookie to take vitamin B-12 tablets to compensate for and replenish her blood loss. Coming out of the coffin is also made more consequential for some vampires due to their social media proficiency. Hudson notes that, “Unlike Jessica today, whose ‘babyvamp’ blog  is part of the series’ multiplatform format” Bill “could not interact with a human society that knew him to be a vampire” (2013, 665). Here the internal narrative of the show permits a younger character to be expanded into its transmedia storytelling in a way that would seem implausible and inauthentic to Bill’s character (at least before he is recruited as an AVL figurehead in Season Three). These video blogs, which are performed by the actors in character, also function to link consumption practices to vampire integration. One vlog has the vampire Pam dispense fashion advice to Jessica and her ‘audience’ about where humans should shop to avoid wearing silver (a metal that enkindles vampire flesh in True Blood). Extra-textually, the real brands that Pam lists off as acceptable for human-vampire contact also confirm to True Blood viewers which consumption practices will identify them as fans of the show (below).

Where once vampires could be seen to attest to “the consequences of over-consumption” (Halberstam 1993, 342), the vampires in True Blood reflect a different set of economic and biopolitical concerns. Writing for Newsweek Jennie Yabroff posits that the current crop of vampire films and televisions shows are permeated by “vampires who have enough self-control to resist the lure of human blood, reflecting, perhaps, the conservative direction the culture has taken” (2008). The popularity of vampires who are able to exercise self-control is politically conservative insomuch as it reflects a neoliberal focus on improving and maximising the capacities of the self. In such an economic climate, Stephen Ball writes that workers are encouraged “to think about themselves as individuals who calculate about themselves, ‘add value’ to themselves, improve their productivity, live an existence of calculation” (2001, 223). That this neoliberal calculation and control could be construed as vampiric speaks to cultural shifts in assessing social and economic success. In his book The Culture of the New Capitalism, Richard Sennett writes that workers who flourish in the contemporary business climate are “oriented to the short term, focused on potential ability, willing to abandon past experience”. This type of employee “is—to put a kindly face on the matter—an unusual sort of human being” (2006, 5). While this continual need to improve, calculate and enhance oneself and one’s resources can prove taxing to a living human, vampires have the physical capabilities as well as an endless amount of time to adapt to and thrive in volatile neoliberal economic conditions.

Vampires who are able to successfully pursue their business and political endeavours recognise the strategic value of performance. Despite her exhortations that vampires can ‘mainstream’ through the consumption of Tru Blood, the AVL’s Nan Flanagan presents herself quite differently to humans in comparison with her fellow vampires. In the episode, “Everything is Broken” (3.9), Russell Edgington kills a human on live television and Nan is revealed watching the event unfold mid-snack on a female human. When Bill is invited by Nan to appear at the AVL-sponsored Festival of Tolerance (“Let’s Get Out of Here”,4.9), he queries the political efficacy of only having three vampires present at the event, “it’s like having a civil rights protest without any black people”. In response, Nan scolds him, “They’re called African Americans and maybe those protests wouldn’t have turned into the blood baths they became if they hadn’t been there, ever consider that?” This cynical and racist understanding of minority groups as responsible for the institutional and social violence inflicted on them is an instrumentalised version of strategic essentialism (see Spivak 1987). The disjunction between Nan’s private ‘life’ and the AVL’s public management of vampire behaviour and comportment draws attention to the ways identity politics bargains on the securing of certain rights at the expense of the lived, or undead, complexity of the identities being politicised.

The shifting between rights discourse in Nan and Bill’s conversation, from the African-American Civil Rights Movement to vampire rights, is indicative of True Blood’s dual treatment of historical inequality as a topic that is both serious and linked to a post-industrial commodification of identity politics. The program typically presents critical views of the US’ racist history through the character of Tara. She is sceptical of Bill’s intentions when they first meet because he admits that his family owned slaves (“The First Taste”, 1.2) and complains, “People think just cause we got vampires out in the open now race isn’t an issue no more” (Hudson 2013, 674). Later Tara is ‘outed’ as a vampire to a former high school classmate who patronisingly affirms her identities by saying, “now you’re a member of two minorities!” (“Somebody That I Used to Know”, 5.8). The politics of being ‘out’ as a vampire are also refracted through allusions to racial segregation. Where Eddie and Steve Newlin’s status as vampires allows them to act on their sexual attraction to men (albeit in different and limited ways), other vampires do not have “built-in privileges of masculine whiteness” (672). For Tara, her body reads as both vampire and African-American, Bill meanwhile is discursively positioned as simply ‘vampire Bill’. As Arlene Fowler explains to her child (upon seeing Bill), “No darlin’, we’re white, he’s dead” (“Sparks Fly Out”, 1.5), whiteness and race are embodied by the living first and non-white bodies second. While the AVL stakes an authoritative claim to what constitutes ‘good’ vampire behaviour, vampires must negotiate their public presence among humans along normatively defined lines of race, gender and sexuality.

These intersections of vampire rights and human-centred identity politics are dramatised in transmedia texts which portray vampires’ attempts to police themselves according to competing sets of claims about ‘good’ and ‘bad’ vampire behaviour. In one of her vlogs, Jessica politely advises Tara to avoid saying phrases like “it sucks” now that she is a vampire, for fear of alarming her audience and the public at large (see vlog below).

The ways in which vampires in True Blood are portrayed “both as a threat and as a fully paid up part of civilian life” (Matthews 2011, 200) exemplifies a biopolitical order which depends on the self-policing and disciplining of subjects according to social norms so that excessive external coercion by the state is not required (Foucault 1977). In this sense, True Blood is the culmination of a representational trajectory of vampires as ostensible outsiders to ciphers for sensible consumption, civic pride and business ethics. In an AVL sponsored Public Service Announcement entitled “Accept the Truth” (below), various vampires describe themselves as ordinary “Americans”, for example, “I’m a short-order cook in New York City, I’m cold to the human touch”, and “I run a horse ranch in Northern Montana, sunlight turns me to ash”.

These dramatic declarations of nationality read as humorous precisely because audiences are used to seeing vampires as obviously different from and suspicious of human life. The extension of the True Blood narrative primarily through these media texts, which simultaneously exhort and parody ‘good’ performances of citizenship and consumption, interpellates fans into a transmedia public sphere along the same lines, through HBO-approved forms of consumption. In the final section of the paper, I want to unpack the distinctions and comingling of political-play as consumption and activism in terms of the role of transmedia storytelling and marketing in disciplining the use of public space.

“But please remember I can rip your throat out if I need to”: Vampires and political-play consumption

I have argued so far that True Blood’s vampire trope conjoins civil rights with consumption and civic pride based on a neoliberal performance and management of the self. The program’s focus on the performance of vampirism enabled by a state protected mode of consumption is carried over into fans’ engagement with the show through officially sanctioned forms of consumption. The program’s production and broadcast through the premium HBO cable channel enables a much more explicit and liberal portrayal of sex and violence than traditional broadcast television, and this is undoubtedly a significant reason the show was pitched to and commissioned by HBO. The positioning of the show as both risqué and compatible with a politically progressive demographic is used in marketing material for the show.

For example, one HBO commercial (above), advertising the Season Two DVD box set, has a white family unwrapping Christmas presents from a young woman, presumably their daughter. In response to her Grandma’s query, “What’s this honey?”, the woman gives a quick recap of the season culminating in this description, “and the whole town has a huge orgy. Merry Christmas Grandma, I love you so much”. The commercial’s tagline is “The perfect gift for almost everybody” . The marketing of True Blood’s sexually explicit and graphically violent content as different to or in opposition to the ‘safe’ television programming that your grandmother enjoys sits at odds with the class and cultural capital required to actually consume the show. This includes access to premium cable or at least reliable broadband Internet to download or view the program as well as the supplementary web material that accompanies the program and is designed to satiate audience interest in between episodes and seasons. Whatever form of risk or subversion the vampires in True Blood present to the existing textual order of vampirism is incorporated into an already safely established mode of television production and consumption.

As Ndalianis points out, the goal of an effective transmedia campaign and story is to make audiences “forget that they’re a marketing strategy devised to sell a product” (2012, 166). Fans are encouraged to immerse themselves “in an emerging narrative that isn’t fixed or pre-staged but which they perform a key role in unraveling” (189) and “the participant is invited to literally play and become part of a performance as if it’s real” (172; original emphases). The unfolding of transmedia participation in ‘real-time’ is precisely how the constructed nature of the story is obfuscated. While fans can unravel or make sense of a transmedia story in diverse ways, the underlying narrative which structures the assemblage of transmedia texts is nevertheless necessarily fixed or pre-staged in order to generate an economy of performance that will move the story along.

The framing of transmedia stories around questions of rights, survival or torture can legitimate biopolitical performances through the commodification of fan activity. For instance, Ndalianis describes an aspect of The Dark Knight campaign, which “included phoning a security guard and trying to convince him to save someone being tortured” (168). In this scenario, fans can ‘create’ their own story based on their conversations with the ‘security guard’ but the narrative economy of bargaining over torture still remains intact. An interesting feature of the transmedia campaigns analysed by Ndalianis are the attempts to import ‘real’ protest into the fictional political campaigns devised for Harvey Dent, the protagonist/antagonist in The Dark Knight,and True Blood’s AVL. In the former, Dent’s campaign website was overlain with graffiti that painted his image with clown make up, signifying the Joker’s growing ‘invasion’ of the movie’s promotion (186). In the latter, AVL ads promoting the VRA were covered over, after their initial ‘clean’ public presentation, with anti-vampire slurs such as ‘Killers’ (179). The more consumers interacted with the campaigns, the more oppositional dissent was introduced into their advertising. This ‘dissent’ then becomes an entertaining spectacle, in which fans can participate, that drives the unfolding transmedia narrative as a story about biopolitical conflict; i.e. what are the democratic limits to expelling the Joker and criminals from Gotham City and vampires from public space in True Blood respectively.

In Simulacra and Simulation, Jean Baudrillard argues that the “impossibility of rediscovering an absolute level of the real is of the same order as the impossibility of staging illusion” (2006, 19). To illustrate this point he talks about the impossibility of staging a ‘fake’ bank robbery and assumes that “the network of artificial signs will become inextricably mixed up with real elements” (20). It is impossible therefore, to stage something that remains “close to the ‘truth,’ in order to test the reaction of the apparatus to a perfect simulation” (20). I would argue however that successful transmedia campaigns illustrate the degree to which the simulacra of political and juridical order is routinely accomplished by corporate and commercial interests and even accommodated by municipal councils and local governments. These transmedia activities seem to be premised on an expectation and acceptance that political campaigns which ostensibly aim to address crime and inequality will inevitably meet public backlash or violent acts of civil disobedience. Contestation over rights and public space are a normalised feature of transmedia campaigns.

Presumably this is entertaining in the context of a performance for a fictional text, albeit one that requires performance in the non-fictional social and political realm of everyday life, but we might compare this transmediation of political contestation with the everyday disciplining of activism in the public sphere. For example, in 2012, pro-Israel advertisements placed in New York subways by the American Freedom Defense Initiative were defaced with words such as “Racist” and “Hate Speech” and activists such as Egyptian-American writer Mona Eltahawy were arrested for spray-painting over them (Holpuch 2012). Here the spectacle of the invasion and countering of advocate discourse is swiftly disciplined by police and security forces, who acted to protect the purchase of advertising space by the American Freedom Defense Initiative. In New Zealand, 2007 saw a series of anti-terror raids resulting in heavy fines, long court proceedings and jail time for anarchist and Māori activists. Among the evidence used to surveil and arrest the defendants were recorded conversations detailing an apparently jocular suggestion that former US President George W. Bush could be assassinated on his next visit to New Zealand by launching a bus at his person (see Operation 8 [Abi King-Jones and Errol Wright, 2011]). Vijay Devadas (2008) provides a thorough examination of the events by situating them within the convergence of government and private security agendas during the ‘war on terror’. I note here that in distinction to transmedia campaigns that compel play-performance of public safety and order issues, parodic suggestions in the execution of advocacy by marginalised communities exacerbate rather than diminish their biopolitical position as threat.

Of course the difference between these ‘real’ events and transmedia storytelling is that the latter involves “a cognitive and sensory satisfaction that relishes in the performativity and playfulness of the text” (Ndalianis 2012, 183). The playfulness and enjoyment of transmedia fan participation seems to occur by virtue of the lack of substantive social and political consequences to transmedia performances. Where Baudrillard might see such performances as testing the authoritative apparatus of juridical and state institutions in such a way as to restate the latter’s epistemological authority to delineate ‘real’ from ‘fake’ civic activity, I would argue that transmedia activity, provided it is authorised by corporate and municipal bodies, does not test ‘the apparatus’ of a juridical and institutional order so much as it ‘simulates’ this order safely and with a positive affective disposition protected by officially authorised forms of consumption.

Ndalianis’ work maps out a framework of analysis, which takes into account the embodied, affective and urban social participation of transmedia storytelling as a significant dimension of fan activity. Given that transmedia storytelling involves the cultivation of activity and participation in the public sphere and urban environment, by connecting private acts of consumption to a theatre of public brand performance, it would be productive to extend Ndalianis’ analytic framework to an investigation of the types of affective relations emerging between fans, the public sphere, media texts, corporate industry and processes of social and political inclusion and exclusion. Does transmedia storytelling encourage a positive affective relation to biopolitical performance so long as this performance is confined to the ‘fictional’ realm? Do media scholars need to account for the consequences of transmedia ‘play’ such as the mass-shooting which took place in an Aurora, Colorado, cinema during a screening of the film The Dark Knight Rises by a young man impersonating a character from the Batman textual archive? How might we compare the increasing surveillance of political advocacy and creative protest with the seeming acquiescence of municipal and city councils to permit corporate branding to invade civil and public spaces for transmedia storytelling campaigns? Notwithstanding the possibility for resistance or divergence on the part of fans with the ‘intended’ transmedia story, the type of narrative used to anchor transmedia campaigns nevertheless frames and orients fan relations to texts through modes of consumer engagement that are legitimated by corporate, state and municipal institutions. Although my focus here has been on the ways in which transmedia consumer engagement legitimises biopolitical modes of performance and debate around civil rights, it may prove fruitful to investigate other types of relations that emerge from embedding fans into state institutions and discourses via transmedia storytelling.

Conclusion: “That’s the sickest shit I’ve ever seen … and I watch Dance Moms!”

In this paper, I have examined how biopolitical imperatives and constraints around vampire integration in True Blood are mediated through transmedia forms of storytelling and marketing. The transmediation of vampire rights involves fan immersion in discursive and representational practices which (re)produce vampirism as an allusion to gay liberation and LGBTI politics. The program’s use of Tru Blood, both intra- and extra-textually, is premised on the commodification of identity politics but also attests to the permeation and popularisation of a rights-based consensus for minority groups. In a positive reading of the program’s allusions to gay rights, True Blood’s transmedia storytelling appears to evince an inclusive textual and representational landscape for LGBTI politics. At the same time, the program draws attention to the biopolitical function of rights discourse by suggesting that it is the management of particular kinds of life, through particular kinds of consumption, that remains valuable to the dominant political and economic order rather than the identities these rights are attached to. In this sense, the mapping of vampirism onto civil rights also functions to legitimise a political discourse that measures some rights against others in terms of the strategic economic and social benefits such rights grant to the polity or fan community as a whole. This weighing up and measuring of rights in terms of who deserves social and political life, and what ‘life’ can be ‘good’ for the community, is surely more monstrous than anything True Blood’s vampires are capable of.

 

References

Ball, Stephen. 2001. “Performativities and fabrications in the education economy.” In The Performing School: Managing teaching and learning in a performance culture, ed. Denis Gleeson and Chris Husbands, 210-226. London: Routledge.

Baudrillard, Jean. 2006. Simulacra and Simulation. Translated by Sheila Faria Glaser. Ann Arbor: University of Michigan Press.

Beck, Bernard. 2011. “Fearless Vampire Kissers: Bloodsuckers We Love in Twilight, True Blood and Others.” Multicultural Perspectives 13 (2): 90-92.

Berkshire, Geoff. 2012. “‘True Blood’ recap: Roman’s fate revealed ‘In the Beginning’.”

HitFix, July 23. Accessed April 27, 2014. http://www.hitfix.com/monkeys-as-critics/true-blood-recap-romans-fate-revealed-in-the-beginning.

Case, Sue-Ellen. 1991. “Tracking the Vampire.” Differences: A Journal of Feminist Cultural Studies 3(2): 1-20.

Davies, Ben, and Jana Funke. 2011. “Introduction: Sexual Temporalities.” In Sex, Gender and Time in Fiction and Culture, edited by Ben Davies and Jana Funke, 1-16. New York: Palgrave Macmillan.

Devadas, Vijay. 2008. “15 October 2007, Aotearoa: Race, terror and sovereignty.” Sites 5(1): 124-151.

Foucault, Michel. 1977. Discipline and punish: the birth of the prison. New York: Pantheon Books.

Foucault, Michel. 1991a. “Governmentality.” In The Foucault Effect: Studies in Governmentality, edited by Graham Burchell, Colin Gordon, and Peter Miller, 87–104. Chicago: University of Chicago Press.

Foucault, Michel. 1991b. “Right of Death and Power over Life.” In The Foucault Reader: An Introduction to Foucault’s Thought, edited by Paul Rabinow, 258–272. New York: Penguin Books.

Grigoriadis, Vanessa. 2010. “The Joy of Vampire Sex: The Schlocky, Sensual Secrets Behind the Success.” Rolling Stone, September 21112: 54-59.

Halberstam, Judith. 1993. “Technologies of Monstrosity: Bram Stoker’s Dracula.” Victorian Studies 36(3): 333-352.

Home Box Office. 2012. Fellowship of the Sun. Accessed January 1, 2012. fellowshipofthesun.org. [site archived here: http://archive.today/9Nr9]

Holpuch, Amanda. 2012. “Activist Mona Eltahawy released after arrest in New York subway protest.” The Guardian, September 26. Accessed April 26, 2014. http://www.theguardian.com/world/2012/sep/26/mona-eltahawy-released-new-york-subway.

Hudson, Dale. 2013. “‘Of Course There Are Werewolves and Vampires’: True Blood and the Right to Rights for Other Species.” American Quarterly 65 (3): 661-687.

Jenkins, Henry. 2006. Fans, Bloggers, and Gamers: Exploring Participatory Culture. New York: New York University Press.

Jenkins, Henry. 2011. “Transmedia 202: Further Reflections.” Confessions of an Aca-Fan, August 1. Accessed April 28, 2014. http://henryjenkins.org/2011/08/defining_transmedia_further_re.html.

Matthews, Nicole. 2011. “Noughties Reading.” In The New Politics of Leisure and Pleasure, edited by Peter Bramham and Stephen Wagg, 195-210. New York: Palgrave Macmillan.

Mutch, Deborah. 2011. “Coming Out of the Coffin: The Vampire and Transnationalism in the Twilight and Sookie Stackhouse Series.” Critical Survey 23(2): 75-90.

Newitz, Annalee. 2008. “Let’s Face It: ‘True Blood’ Hates Gay People.” io9, November 1. Accessed April 28, 2014. http://io9.com/5071755/lets-face-it-true-blood-hates-gay-people.

Ndalianis, Angela. 2012. The Horror Sensorium: Media and the Senses. Jefferson: McFarland Publishing.

Sennett, Richard. 2006. The Culture of the New Capitalism. London: Yale University Press.

Shen, Maxine. 2009. “Flesh & ‘Blood’.” New York Post, June 23. Accessed April 28, 2014. http://nypost.com/2009/06/23/flesh-blood/.

Solomon, Deborah. 2010. “Once Bitten: Questions for Charlaine Harris.” The New York Times, April 30. Accessed April 28, 2014. http://www.nytimes.com/2010/05/02/magazine/02fob-Q4-t.html?_r=0.

Spivak, Gayatri Chakravorty. 1987. In Other Worlds: Essays in Cultural Politics. New York: Methuen.

Tyree, J. M. 2009. “Warm-Blooded: True Blood and Let the Right One In.” Film Quarterly 63(2): 31-37.

Yabroff, Jennie. 2008. “A Bit Long in the Tooth.” Newsweek, December 15. 152(24).

 

Filmography

Ball, Alan. True Blood. 2008-2014. USA: HBO.

King-Jones, Abi and Errol Wright. Operation 8. 2011. NZ: www.cutcutcut.com.

Lieber, Jeffrey, Abrams, J. J., and Damon Lindelof. Lost.2004-2010. USA: Walt Disney Studios Home Entertainment.

Nolan, Christopher. 2008. The Dark Knight. USA: Warner Home Video.

 

Notes

[1] My thanks to the anonymous referee for their thoughtful comments and suggestions for improving the paper’s analytical focus. I am also grateful to Kevin Fisher for sharing his insights on Baudrillard and transmedia during the writing of this paper and to Katharine Legun for her help with improving the clarity and coherency of the paper. An early version of this paper was published in the magazine Cherrie. The original version of the paper can be found here: http://gaynewsnetwork.com.au/feature/vamps-and-queers-5136.html

 

Bio: Holly Randell-Moon is a Lecturer in Communication and Media Studies at the University of Otago, New Zealand. Her publications on popular culture, gender, and sexuality have appeared in the edited book collections Common Sense: Intelligence as Presented on Popular Television (2008) and Television Aesthetics and Style (2013) and the journal Feminist Media Studies. She has also published on race, religion, and secularism in the journals Critical Race and Whiteness Studies, borderlands and Social Semiotics and in the edited book collections Religion, Spirituality and the Social Sciences (2008) and Mediating Faiths (2010).

 

Intermediality and Interventions: Applying Intermediality Frameworks to Reality Television and Microblogs — Rosemary Overell

Abstract: This article explores the usefulness of ‘intermediality’ approaches for understanding contemporary reality television. Through a case study of Intervention, it is proposed that intermedial frameworks illuminate reality television’s function as a “dream of presence”. The article focuses particularly on intermedial manifestations of the television program on the microblogging platform, tumblr. Building on studies of intermediality within cinema and visual cultural studies, this article highlights the liminal, affective and processual elements that arise from the intermedial movement of content. It does this through an application of ideas from non-representational theory as a means for expanding intermediality beyond the cinematic. This article suggests that liminality, affect and process are, in turn, presented in Intervention, and emphasised in the intermedial presentation of Intervention ‘screencaps’ on fan-made tumblr microblogs.

Figure 1. An example of the black and white titles used in Intervention to elaborate on the addict’s back story (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Tammi and Daniel” (A&E Network 2006b).

Figure 1. An example of the black and white titles used in Intervention to elaborate on the addict’s back story (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Tammi and Daniel” (A&E Network 2006b).

Introduction

This article is an intervention into contemporary accounts of intermediality. Building on recent work in cinema and visual culture studies (Barker 2009; Bennett 2007; Pethő 2010; Pethő 2011; Rajewsky 2005),I propose that intermediality functions as a useful framework for understanding how contemporary popular cultural production works as what Derrida calls the “dream of presence” (1978) –a representational manifestation of a striving for significance and meaning in response to more than representational experiences. Here, I integrate work in non-representational theory (NRT) to elaborate on the more than representational potential of intermedial products.

Intermediality’s concern with the between-ness of media signifiers highlights the liminal, affective and processual aspects that arise via the movement of content between media forms. These, in turn, destabilise the apparent coherence and ‘purity’ of mediated representation and the presumption of media specific modes of spectatorship. This complements NRT approaches, which argue for a privileging of the more than representational dimensions of the cultural landscape. In particular, Mitch Rose’s (2006) elaboration on Derrida’s “dreams of presence” as an inclination and striving – but necessarily also failing – towards an ossified, accountable understanding of culture enlivens intermedial approaches. In this article, I apply the framework of intermediality to the reality television programme Intervention (A&E Network 1999 – 2013) and its surrounding fan blogs.

Intermediality

Intermediality rejects media specificity. It moves away from understanding media as discrete technologies of representation and, in particular, rejects assumptions that particular types of representation are bound to particular types of media (for example moving images with cinema). Furthermore, it differs from transmedia frameworks, which focus on how textual content and significance change when moved from one media form to another. While transmedia approaches acknowledge the mobility of media content, they still present media forms as discrete signifying systems, which produce coherent meaning – either across or within formats. Intermediality troubles this assumption by focusing on how media products come into being via the dynamic inter-relation between media. That is, intermediality is characterised by medial transposition. The process of the transposition of media content, and of one medium into another, also points to the instability of signification. Ágnes Pethő (2011) posits intermediality as constitutive of new mediated experiences in between media forms where media – and their associated significations – are radically dislocated and displaced. This is partly because, via this movement, traces of the originary medial form and content are incorporated into the new medial form and content. Pethő notes the possibilities constituted via intermedial relations and the potential of an intermedial perspective to highlight the multiple mediated relations that produce our comprehension of media content.

Pethő emphasises the affective elements integral to intermediality through her rejection of Kristevan intertextuality.[1] According to Pethő, intermediality is more than textual. It is also more than representational. It is about the myriad of experiences, which are beyond articulation and cognition via standard signifying frameworks such as language: “Intermediality … is not something one ‘deciphers’, it is something one perceives or senses” (2011, 68). Pethő thereby emphasizes the sensuous aspects of embodied spectatorship of cinematic products. She argues for a phenomenological approach to intermediality as a means of accounting for the pre-cognitive experience of cinema. Her understanding, then, is that the experience of intermediality is affective. Here, affect is defined in Massumian (1992; 2002; [1987] 2007) terms as a “prepersonal intensity” ([1987] 2007, xvi) beyond representational frameworks and descriptors such as emotions. The instability of a coherent cinematic mediality is highlighted precisely through intermediality’s concern with the liminal spaces constituted in the moments when media content and representations move between media. This, again, echoes Massumi who argues that affect can only occur in terms of process, passage and interaction between subjects, things and spaces.[2] Massumi posits process as primary to “every formation” (1992, 194) and notes that processuality works as “sites of passage that gather up movement and send it back translated” (ibid.). As Rossiter (2003) points out in his work on processual media theory, this movement between media formations is important, partly because such a dynamic changes the media’s form, content and reception, but also because, by understanding contemporary media as processual, space is made for understanding media as socially, and culturally contingent.

Figure 2. An example of the black and white titles used in Intervention to present ‘facts’ about addiction and health problems (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Linda” (A&E Network 2009a).

Figure 2. An example of the black and white titles used in Intervention to present ‘facts’ about addiction and health problems (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Linda” (A&E Network 2009a).

Intermediality as a “dream of presence”

I propose that we can understand intermediality in terms of non-representational theory (NRT) – a conceptual framework associated with cultural geography. NRT’s remit is to “enliven” the humanities through a privileging of affective experiences over representational—particularly textual—analyses (Thrift 2008; Anderson and Harrison 2010).

A key site for NRT analyses is “cultural landscapes”(Rose 2002; 2006). Rose broadens the geographical implications of “landscape” to include theoretical engagements with cultural practices and objects that attempt to “capture” such manifestations “into sense … into something that can be … imagined in a mental tableau” (2006, 537). Rose argues that the representational concerns of much cultural analysis ossify the vitality of cultural processes. Instead, he presents an NRT approach to culture as one that collapses subject and object into co-emergent becomings (538). In this way his approach is resonant with Pethő’s emphasis on intermediality as constituted through phenomenological correlations of the types of embodiment implied by different media forms.

Rose’s (2006) approach to cultural landscape further complements intermedial approaches through his understanding of cultural analysis as a ‘dream of presence’ instead of a ‘thingified’ culture. Rose claims that we are drawn to stable understandings of culture that mitigate the messy experience of engaging with the world. Rose dubs these inclinations – both in their manifestations as coherent, signifying systems and the process of striving towards these coherencies – dreams of presence. Here, he draws on Derrida (1978), who describes the “dream” of the “philosophical man” [sic] for “full presence, the reassuring foundation” (292). In Derridean terms, this is an impossible possibility because of the instability of language systems.

Nonetheless, as Rose posits, we are inclined to the certitude of teleologies, grand narratives and coherent signification. Rose provocatively suggests that cultural theory is itself a manifestation of this dream. Like other NRT theorists, Rose calls for the humanities to account for the “more than representational” (2006, 345). One means for doing this is a foregrounding of the process of these inclinations towards the “performance of closure and encirclement” (2006, 345) that goes with cultural analysis, but also permeates our everyday encounters with cultural products and processes. Despite the impossibility of stabilizing culture into a coherent landscape, we repeat this performance as a way of being in the world which Rose dubs “affective cabling that connects self and word” (ibid.). The privileging of the strivings towards meaning undercut claims to representational meaning.[3]

In terms of intermediality, Rajewsky gestures towards a similarly impossible—though desired state, arguing that the apparent material specificity of originary media is illusory. That is, intermedial products foreground the surface claims of media as inscribing a discrete materiality within their modes of representation. She notes that this foregrounding produces a particular ‘as if’ experience for the intermedial consumer: “the book reads as if it is a film”.[4] Here, we see the inclination towards coherent significations. Medial approaches are a “dream of presence” and intermedial approaches operate similarly to Rose’s work – in destabilizing the assumption of medial solidity. Pethő echoes a similar idea in her discussion of the viewer’s embodied response to cinema – the desire and striving to stabilise and decipher the intermedial experience. The work of the spectator in fostering what Rose dubs a “dream of presence” in fact draws attention to the intermediality of cinematic experience and the impossibility of a coherently signifying, cinematic product.

To summarise, intermediality is characterised by liminality, affect and processuality. These three characteristics highlight the significatory instability of media content, as well as the power of media themselves and of intermedial experiences to push beyond the representational systems that characterize discrete, medial, approaches. In turn, I propose that intermedial approaches are enriched through recourse to NRT frameworks that privilege the in-betweenness within all “dreams of presence”.

Figure 3. An example of how the production mechanisms of the program are shown on Intervention (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Linda” (A&E Network 2009a).

Figure 3. An example of how the production mechanisms of the program are shown on Intervention (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Linda” (A&E Network 2009a).

Case study: Intervention

Intervention was a reality television program that aired on the American cable channel A&E Network (A&E) between 2005 and 2013. The show was demonstrative of A&E’s shift from “Arts and Entertainment” programming – made up of imported British dramas and documentaries on opera, theatre and cinema – to a focus on reality television.[5] The premise of Intervention is the representation of an addicted subject, who undergoes a “Johnson Institute” – so called “ambush” – intervention for their addiction and is encouraged to go into rehabilitation.[6] The addict is contextualised through to-camera commentary from the addict and their family and friends. The intervention is overseen by a professional interventionist, and the program usually closes with a “catch up” clip, generally of the recovering addict in a rehabilitation clinic.

Intervention is formulaic. Certain formal elements are consistent across ten of the thirteen seasons. The stylistic changes in the final seasons were minimal, indicating that the show had found its formula. Common elements included the theme song, advertising break bookends which incorporated sirens over a blurred, fast moving image of flashing lights, the closing theme of The Davenports’ song about recovery “5 steps” (2000), the use of black-background-white-text inter-titles to inform viewers of personal information on the addict subject and ‘facts’ about addiction (Figures 1 and 2), the narrative of introducing the addict and showing their substance use and relaying the addict’s (often traumatic) personal history, the intervention, and then finally rehabilitation. Two of the interventionists appeared in all thirteen seasons, Jeff Van Vonderen and Candy Finnigan. Both Van Vonderen and Finnigan are themselves former addicts. Like other A&E reality programs, and reality television broadly, Intervention used a documentary aesthetic. Camera work, particularly in the initial section of each episode, where the addict is tracked in their everyday life, was handheld. The production staff was regularly revealed – particularly when interventions did not go to plan – talking to the addict and their family from either behind or in front of the camera. Camera operators and production equipment were often shown in mirrors or by second cameras (Figure 3). In the final few seasons, the intervention “scene” began with an overhead establishing shot that revealed the family and interventionist as well as the production staff, lights and cameras. This cinema vérité style was crucial to building the program’s generic classification as reality television,[7] and claim to an authentic ‘documentary’ portrayal of substance use. This aesthetic also positioned it apart from slicker programs with similar themes, such as Celebrity Rehab with Dr. Drew (VH1 2008 – 2012) and Addicted (TLC 2010). The hand-held aesthetic of Intervention only lifts in the final scenes at rehab – where steady shots of doctors, counselors and the recovering addict are used.

Figure 4. One of the liminal spaces common to Intervention, a hallway (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Tammi and Daniel” (A&E Network 2006b).

Figure 4. One of the liminal spaces common to Intervention, a hallway (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Tammi and Daniel” (A&E Network 2006b).

Intervention is transmediated in numerous ways. Its content has crossed primarily from television to Internet platforms. Two popular memes have been created by viewers from the television episodes. The first shows a clip of inhalant addict Allison (A&E Network 2008) “huffing” computer duster and mumbling “I’m walking on sunshine” mashed up with the K. C. and the Sunshine Band song “Walking on Sunshine”.[8] The disconnect between Allison’s obvious desperation and the song’s upbeat lyric serves as the “punch line” for the meme. The second Intervention meme shows Rocky (A&E Network 2010e), a cocaine addict, emitting a high-pitched cry during his intervention. The meme became a viral video dubbed “The Best Cry Ever” and was remixed into clips of people expressing disappointment at ostensibly trivial matters.[9] These memes appropriate content from Intervention and re-signify it through incorporating it with apparently unrelated content from popular culture. There are also numerous other viewer-made YouTube mash ups of Intervention, including facebook fan pages and discussion boards focused on the program. Like many other reality television programs, Intervention has prompted various tumblr pages where users can share, like and upload primarily visual and video material. tumblr has a handful of blogs focused on Intervention where users screencap (still frame) and gif (moving frame) particular moments from the show and sometimes provide commentary on the program.

A number of academic articles have discussed Intervention. None, however, have looked at the intermediality of the program, nor its transmedia engagement by audiences. Instead, work on Intervention has critiqued the ethics and efficacy of the program from a health-science perspective (Kosovski and Smith 2011); the commodification of drug addiction and Intervention’s position within the wider genre of transformation reality television (Oriekose 2013); and the over-representation of white addicts as ‘wasted’ white citizens on the program (Daniels 2012).[10]

Intervention and intermediality

I propose that intermediality is a useful method for understanding Intervention, particularly “tumbld” interventions which emphasise liminality, processuality and affect. I argue that the intermedial characteristics of tumbld Intervention demonstrate the “dreams of presence” that Rose discusses via Derrida.

Figure 5. One of the liminal spaces common to Intervention, a streetscape (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Trent” (A&E Network 2007).

Figure 5. One of the liminal spaces common to Intervention, a streetscape (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Trent” (A&E Network 2007).

Intervention, the television program, focuses on liminal spaces, experiences and subjects. These representations of in-betweeness gesture towards televisual media’s desire to overcome liminality in favor of neat teleology. However, the repetition and foregrounding of such spaces simultaneously highlight the inability of media to cordon off experience into coherent, self-contained narrative forms. The settings for all but the final section of Intervention (in rehab) are usually interim sites. We encounter squats, cheap hotel rooms, brothels, conurban housing estates, fast food outlets and, most of all, streetscapes (Figures 4 – 6). The chief ‘interim’ space common to every episode is the hotel conference room where the intervention occurs. The “beige blandness” (Daniels, 110) of the conference room signals liminality in a most banal way. It is striking how similar these rooms are despite being scattered across North America (Figures 8 – 10). This sameness indicates the purpose of such rooms – as inoffensive sites through which hundreds of professionals move every year. Further, the conference room works as a waiting room for each episode’s ‘cast’. Prior to the intervention, we see the family and friends waiting for the addict and anticipating the confrontation. During the intervention, the interventionist, family and friends wait to see if the addict will move on from the liminal site of the conference room to rehab. The repeated trope in the later seasons of zooming out to show the mechanics of the show’s production prior to the intervention further highlight the temporary function of the conference room. The wide shot gives an impression of urgent assembly of production equipment which will then be hastily dismantled so that the room can be used again by conference delegates.

Figure 6.One of the liminal spaces common to Intervention, an image of a fast-food restaurant sign, presumably taken from a car (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Salina and Troy” (A&E Network 2006a).

Figure 6. One of the liminal spaces common to Intervention, an image of a fast-food restaurant sign, presumably taken from a car (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Salina and Troy” (A&E Network 2006a).

Furthermore, the overarching discourse of Intervention is that of “moving on” from one’s addiction. As Daniels discusses, Intervention’s narrative hinges on the transformation of the addict from the negatively framed substance (ab)user to a healthy citizen-subject.[11] This discourse requires a focus on the addict’s life as a series of transitional experiences. The instability of addiction is emphasised in each episode’s opening scenes where we see addicts scrounging for money, usually via illicit activities, and the apparently infinite risks of scoring and then consuming their substance. As with the waiting space of the hotel conference room, these scenes position the addict’s experience as liminal. The addict is almost always on the move – their experiences are framed as transient. This is most evident in the repeated trope of the addict resisting the format of the show itself. Usually this manifests in the addict storming out of a to-camera scene, removing their microphone and running away into the street. The addicts – by warrant of their addiction – occupy a marginal subjectivity. This is compounded by the regularity of the inclusion of addicts who signify marginality within dominant discourse. Intervention, though white-dominated, as Daniels observes, offers a parade of working-class people, abuse victims, prostitutes, non-heterosexual subjects and the mentally ill. The addicts also embody an interim subjectivity – relentlessly striving for, and dreaming of, a stable, present self via the use of drugs or alcohol, while simultaneously demonstrating the impossibility of this position through a heightened instability in the form of chasing their substance and waiting for a connect.

Figure 7. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Trent” (A&E Network 2007).

Figure 7. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Trent” (A&E Network 2007).

The liminality of Intervention – and by implication the televisual medium – is highlighted when screencaps and gif images of the program are presented on tumblr. “Ordinary Interventions” (ordinary intervention, 2012 -) tumbls screencaps from Intervention without narrative linearity. Tumblr is a microblogging platform, launched in 2007, which is mostly used to post images. Users can ‘follow’ others’ blog posts and access them through a generic ‘dashboard’ which displays blog posts chronologically. That is, all the posts from bloggers one follows are collated onto the dashboard. Tumblr also follows an “infinite scroll” format. The screen refreshes with more posts each time the user appears to reach the end of a feed. Provided the user follows a number of bloggers, the dashboard is constantly offering new images, which may have little to do with the other images displayed.

If one follows a cute puppy blog and a feminist comic blog, for example, seemingly unrelated images from both will appear sequentially – as the blogger posts – producing a discordant series of images. Users can also personalize their individual blog interface through the application of ‘themes’ which determine font, layout etc. With “Ordinary Interventions”, images are displayed two across and rarely are images from the same episode placed alongside each other. The screencaps are of varying sizes, determined by the format of the video file from which the image was taken. They are hashtagged with the name of the addict, addiction as well as #intervention and #aetv. However, and unlike most tumblr interfaces, the hashtags are not readily viewable to browsers, though they can be searched on the blog’s search function or by clicking the ‘+’ button, though this is not readily apparent. Importantly, “Ordinary Interventions” does not include images of drugs, alcohol or substance use. That is, the blogger focuses on the ordinary, liminal moments of the addict’s everyday life – the interim moments between the spectacularised (through the television program in extreme close ups and repetition) instances of substance use. “Ordinary Interventions” provides no information about the blog’s author, no text or re-blogged images.

Figure 8. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Tammi and Daniel” (A&E Network 2006a).

Figure 8. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Tammi and Daniel” (A&E Network 2006a).

The apparently random collation of screencaps on “Ordinary Interventions” produces liminality through its un-anchoring of ‘moments’ from Intervention in a non-linear format. The ‘capping’ of people: crying and laughing, ordinary household scenes, urban landscapes, old photographs, direct-to-cameras against a blue background and the banality of the hotel conference rooms, lack narrative coherence. This is not a “remix” like the Allison and Rocky memes where elements from Intervention are incorporated into a new narrative for comic effect. Instead, “Ordinary Interventions’” presentation of Intervention appears meaningless. In fact, tumblr users without familiarity with the program would most likely find ‘Ordinary Interventions’ impenetrable. The very act of tumbling indiscriminate screencaps from Intervention radically dislocates the transformational, solidifying narrative of the program as televisual media. There is no rehabilitation or resolution, simply an endless scroll through interim moments. Instead, Intervention mobilises affect as crucial to the flow of each episode. The representation of affective responses to addiction, both from the addict and their loved ones, generates sympathy in viewers and a desire for the protagonist to “move beyond” the liminality of addiction to the apparent solidity of rehabilitation. Recall Massumi’s and NRT theorists’ understanding of affect as more than articulated, representable and cognized emotion. Instead it refers to the intensities one experiences through the movement from “one … state … to another” (Massumi 2007, xvi), hence the program’s liminality. Thus, while the affect gestured towards in the program may provoke an emotional response in viewers, affect is not synonymous with emotion. However, the subject’s containment of affect within language – through the 12-step process of ‘admitting’ a problem and containing affective responses within a pathologised discourse of “emotional trauma” – gestures yet again towards the inclination to pin down the instability of everyday life experiences.

 Figure 9. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Nichole” (A&E Network 2012).

Figure 9. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Nichole” (A&E Network 2012).

Affect is most evident in the liminal moments of the program. In fact, Rocky’s “Best Cry Ever” clearly demonstrates the more-than-ness of the experience of addiction and, accordingly, how affect is mobilized in Intervention. His cry is wordless, high-pitched and completely embodied. He shudders as he emits it from deep in his body. This moment, of what Dolar (2007) might call “voice” demonstrates the inadequacy of language for representing embodied experience.[12] Further, Rocky’s cry demonstrates affect’s own liminality. The moment occurs in the interim space of the intervention in a hotel conference room. Moreover, as Dolar writes, screams demonstrate the pre-cognitive “penultimate stage” (69) prior to the subject’s interpellation within representational structures.[13] Alongside affective moments, such as Rocky’s scream during the intervention, the program dwells on addicts’ repeated insistence that they are “without words” and that their subjectivity – bound to their addiction – is beyond representation. For example, addicts say: “wow I can’t even talk right now” (A&E Network 2010d), “it’s like having an orgasm; you can’t just describe it” (A&E Network 2010e), and “it’s kind of hard … that feeling you get” (A&E Network 2011).

Figure 10. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 - ). The episode from which this is screencapped is “Greg” (A&E Network 2009b).

Figure 10. The hotel room, the site of the intervention, is ubiquitous and similar across episodes of Intervention (ordinary intervention, 2012 – ). The episode from which this is screencapped is “Greg” (A&E Network 2009b).

The intermedial presentation of Intervention’s representations of affective intensity on tumblr generates an affective response from tumblr users. Such responses can be understood as yet another variation of the “dream of presence”. The affective engagement with the program – generated through tumbling and commenting – foregrounds the impossibility of understanding either the program content, or the medium, as singular, discrete and, most importantly, representational.

“Fuck Yes Intervention” (Fuck Yes Intervention, 2011 – ) is a tumblr, which presents screencaps alongside gifs from Intervention. Unlike “Ordinary Interventions”, the interface offers clear hashtags and information about the blogger, who is a drug addict.[14] “Fuck Yes Intervention” also encourages interaction from tumblr users. Followers of the blog can comment, ask the blogger questions and request that particular episodes be screencapped or giffed. “Fuck Yes Intervention” presents affective moments from the program that then generates discussion of viewers’ affective responses to Intervention. For example, responding to a screencap of crystal-meth addict, Cristy (A&E Network 2006c), radhabits writes: “i cried the first time i watched this episode, and every time I watch intervention” [sic] (Fuck Yes Intervention, 2011a). Another Cristy post (Fuck Yes Intervention, 2011b) – this time showing Cristy’s experience of being high as she dances around her bedroom – prompted unintelligible responses from other users, unable to represent linguistically “what they just watched”:

dynomitemedley: uhhhhhh

luxuryintherough: what. the. fuck. did. I. just. watch???? luxuryintherough

zombiecupcake: I DON’T KNOW (ibid.).

Rather than the apparently straightforward governmental meaning of Intervention as a lesson in how not to be a good, healthy, citizen, the users of “Fuck Yes Intervention” present multiple, affective responses to the screencaps and gifs on the blog.

“Ordinary Interventions” also focuses on affective moments from Intervention. There are numerous images of addicts, family-members and friends in the liminal space of embodied intensity. Furthermore, the tumblr’s screencaps of the inter-titles and subtitles common to every episode undercut the apparent authority and hegemony of textuality not only in the program, but more broadly, in representational understandings of reality television. Intervention uses inter-titles to elaborate on the addict’s back-story as well as to present facts about addiction. Subtitles are used when the speech onscreen is inaudible. The haphazard insertion of text-based screencaps beside images of Intervention’s overwrought subjects produces a dislocation of language and image, which draws attention to the inadequacy of representational structures for expressing embodied experience. More importantly, the blog’s bricolage of images and text foregrounds the illusory claim of reality television forms to signifying authority. The affect – isolated in the screencaps of “Ordinary Intervention” – overwhelms pretentions to representational solidity. Writing on captioned visual art, Bennett indicates that rather than the printed words operating as a Barthesian anchor for the image they “run relentlessly unable to flow through the normal communication channels … register[ing] as pure intensity: affect characterised by … lack of attachment, disarticulated from motives” (442). “Ordinary Interventions” enacts this affective disarticulation in sequences, which themselves are sense-less. For example, an image of a woman laughing to camera is followed by an inter-title stating: “Coley has been collecting burl for the past three months. He has never sold any” (Figure 10). Next is a screencap of a woman petting a puppy on a mattress, then a middle-aged woman sitting in front of giant rosary beads, and so on. As Pethő points out in relation to cinema, the potential of intermediality to break up and dislocate signifying systems through a foregrounding of the disjunction between speech and image not only rejects a fixity of meaning for the content represented, but also the media form ‘doing’ the representing (61).

Figure 11.“Ordinary Interventions” offers a bricolage of disarticulated screencaps from Intervention (ordinary intervention, 2012 - ).

Figure 11.“Ordinary Interventions” offers a bricolage of disarticulated screencaps from Intervention (ordinary intervention, 2012 – ).

The presentation of the processual nature of addiction in Intervention resonates with intermedial understandings of media forms as mutable, changing and striving towards coherence, though never reaching it. The premise of the program, as discussed above, is a transformative ‘journey’ from addiction to rehabilitation. Using pop-psychological platitudes common to therapy culture, interventionists Finnigan and Van Vonderen encourage the addicts’ families to “process” the intervention and “work through” their own neuroses (often via a the “Betty Ford Family Program”) as a means of overcoming the co-dependency of their relationship with the addict. For example, in season 8, Van Vonderen tells the family before the intervention: “recovery is not a sudden landing. It’s a long journey” (A&E Network 2010a). In a later episode, Finnigan tells crack addict Vinnie’s family that addiction is “a progressive, terminal disease … nothing changes, if nothing changes” (A&E Network 2010b).

Despite its emphasis on process, Intervention rarely offers a straightforward transformation of either the addict or their loved ones. By frustrating deterministic narratives of self-improvement the show highlights both the desire for wholeness and progress, as well as its futility. This is clearly demonstrated in the closing inter-titles which regularly inform viewers that the addict “left treatment” before the proscribed ninety days, has “relapsed”, or simply refused to continue treatment and be “rehabilitated”. Over thirteen seasons, viewers, addicts and their families and friends repeat a desire for wholeness and wellbeing, which can never be achieved. The viewer who tunes in weekly finds pleasure in both the desire for narrative neatness, coupled with knowledge of the trajectory’s likely failure.

It is in its intermedial form on tumblr, presentations of Intervention demonstrate the processuality of intermedial transposition. We can understand the screencaps of “Ordinary Interventions” as foregrounding process, but like the content of the program itself, repeatedly failing. “Ordinary Interventions” presents a radical asynchronicity in its infinitely scrolling, random presentation of Intervention screencaps. It repeatedly interpellates users to “make sense” of the images while always undermining this through the random order in which the images appear. “Ordinary Intervention” does use tags, which could ostensibly sort the images. However, these tags are haphazard. Addicts have multiple addictions or their name or addiction is incorrectly tagged, and images of non-addict family members will often also be tagged with the substance name. Users remain stymied in their search for narrative clarity. This failure of comprehension highlights the processual nature of cognition and foregrounds our initial affective responses as unable to “fit” the signifying structures assumed by media forms.

The processual reconfiguation of Intervention via “Ordinary Interventions” is also reflexive. The blog draws attention to how affect is produced within the medium of television via its re-presentation on tumblr. It is therefore intermedial in the sense that it is concerned with the emergence of both Intervention the program and reality television more broadly. Rather than being presented with a one-way – or even dialogic – communication from media to consumer, “Ordinary Interventions” in the asynchronicity of its mediality is a porous “dream of presence”.[15] As “Ordinary Interventions” lacks clear narrative conventions, it fails to communicate a coherent explication of addiction. Instead, its interface generates a repeated experience of striving for and inclining towards present-ness. Even more than the repeated failures depicted in its televisual form, the transposition of Intervention through “Ordinary Interventions” will never offer resolution. When the medium migrates it is “reconfigured” (Bennett, 448) by its presentation in a new medium’s interface. The screencaps of addicts, their loved ones and the interventionists – for lack of a dialogic reverse-shot – seep into one another to potentially produce multiple, contradictory, affective responses which make any clear articulation in language difficult. The apparently random sequences of images resist the sequential reading to which televisual content lays claim.

Conclusion

In closing, I wish to briefly describe one of the most recent screencaps posted on “Ordinary Interventions” (Figure 11). It shows a black screen with a white blur. The blogger has clearly screencapped the interim moment between the affective enaction of the addict’s ‘process’ to intervention and the insertion of an explanatory title card. The hashtags are general (#substanceabuse; #addiction #intervention etc.) – they do not indicate the episode. The one unique, though perhaps predictably inadequate, hashtag is #blackscreen. The screencaps around it show a tattooed woman in profile, a hand reaching for an hors d’ouevre, and a close up of a woman crying. This image: #blackscreen, perhaps best embodies the intermediality of Intervention as transposed through tumblr. It is a representation of liminality, a captured microsecond in-between narration. Its position in relation to the affect-steeped screencaps around it foregrounds affect and refuses clear signification. Its lack of communication, despite gesturing towards textuality (it is obviously a fade from text to black), demonstrates processuality.

Figure 12.Recent posts on “Ordinary Interventions” (ordinary intervention, 2012 - ).

Figure 12.Recent posts on “Ordinary Interventions” (ordinary intervention, 2012 – ).

This discussion of intermediality in relation to reality television and microblogging demonstrates the usefulness of intermedial frameworks for analysing and destabilizing popular cultural texts and media. Further, I want to suggest that the application of intermedial theory to Intervention’s televisual and tumbld manifestations could fruitfully be expanded through reference to NRT understandings of Derridean “dreams of presence”.

 

 

 

 

References

A&E Network. 2013a. “All Shows” A&E. Accessed December 10, 2013. http://www.aetv.com/allshows.jsp.

A&E Network. 2013b. “Intervention: About the Show” A&E. Accessed December 10, 2013. http://www.aetv.com/intervention/about/.

A&E Network. 2013c. “Intervention: Meet the Interventionists” A&E. Accessed December 10, 2013. http://www.aetv.com/intervention/meet-interventionists.

Anderson, Ben and Paul Harrison. eds, 2010. Taking Place: Non-Representational Theories and Geography. Farnham: Ashgate.

Barker, Jennifer M. 2009. The Tactile Eye: Touch and the Cinematic Experience. Berkeley, CA: University of California Press.

Bennett, Jill. 2007. “Aesthetics of Intermediality.” Art History 30 (3): 432-450.

Tony Bennett and others, eds, 1981. Popular Televisions and Film. London: BFI Publishing in association with the Open University Press,

Berlant, Lauren. 2011. Cruel Optimism. Durham, NC: Duke University Press.

Daniels, Jessie. 2012. “Intervention: Reality TV, Whiteness, and Narratives of Addiction.” Advances in Medical Sociology 14: 103-125.

The Davenports. 2000. “5 Steps.” Speaking of the Davenports. New York, NY: Mother West.

Derrida, Jacques. 1978. Writing and Difference. Chicago: Chicago University Press.

Don. 2013. “Walking on Sunshine” Know Your Meme. Accessed December 10, 2013. http://knowyourmeme.com/memes/walking-on-sunshine.

Dolar, Mladen. 2007. A Voice and Nothing More. Cambridge, MA: The MIT Press.

Fuck Yes Intervention. 2011 – . “Fuck Yes Intervention,” tumblr. Accessed December 10, 2013. http://fuckyesintervention.tumblr.com.

Fuck Yes Intervention. 25 August 2011a. “Cristy Intervention Vodka Bed,” tumblr. Accessed: December 10, 2013. http://fuckyesintervention.tumblr.com/post/9384802554/cristy-intervention-vodka-bed.

Fuck Yes Intervention. 31 August 2011b. “Cristy,” tumblr. Accessed: December 10, 2013. http://fuckyesintervention.tumblr.com/post/9640210736/cristy.

Fuck Yes Intervention. 27 May 2013. “Submissions?”, tumblr. Accessed: December 10, 2013. http://fuckyesintervention.tumblr.com/post/51492204728/submissions.

Idah, Oriekose. 2013. “Viewer-Patient Confidentiality: Commodification of Illness in Contemporary U.S. Medical Reality TV.” Intersect 60 (2). http://ojs.stanford.edu/ojs/index.php/intersect/article/view/569. Accessed November 7, 2013.

Kosovski, Jason R. and Douglas C. Smith. 2011. “Everybody Hurts: Addiction, Drama, and the Family in the Reality Television Show Intervention.” Substance Use and Misuse 46: 852-858.

Kristeva, Julia. 1986. “Word dialogue, and the novel.” In The Kristeva reader, edited by Toril Moi, 79-106. New York, NY: Columbia University Press.

Massumi, Brian. 1992. A User’s Guide to Capitalism and Schizophrenia: Deviations from Deleuze and Guattari. Cambridge, MA: MIT Press.

Massumi, Brian. (1987) 2007. “Pleasures of Philosophy.” In A Thousand Plateaus: Capitalism and Schizophrenia, Gilles Deleuze and Félix Guatarri. Translated by Brian Massumi, ix-xix. Minneapolis, MN: University of Minnesota Press.

Massumi, Brian. 2002. Parables for the Virtual: Movement, Affect, Sensation. Durham, NC: Duke University Press.

Ordinary intervention. 2012 – . “Ordinary Interventions.” tumblr. Accessed December 10, 2013. http://ordinaryintervention.tumblr.com.

Paech, Joachim. 2000. “ESF ‘Changing Media in Changing Europe’.” Paper presented at Artwork – Text – Medium: Steps en Route to Intermediality, Paris, May 26 – 28.

Pethő.“Ágnes. 2010. Intermediality in Film: A Historiography of Methodologies.” Film and Media Studies 2: 39-72.

Pethő, Ágnes. 2011. “Reading the Intermedial: Abysmal Mediality and Trans-Figuration in the Cinema.” In Cinema and Intermediality: The Passion for the In-Between, 55-94. Newcastle upon Tyne: Cambridge Scholars Publishing.

Rajewsky, Irina O. 2005. “Intermediality, Intertextuality and Remediation: A Literary Perspective on Intermediality.” History and Theory of the Arts, Literature and Technologies 6: 43-64.

Rose, Mitch. 2002. “Landscapes and Labyrinths.” Geoforum 33: 455-467.

Rose, Mitch. 2006. “Gathering ‘dreams of presence’: a project for the cultural landscape.” Environment and Planning D: Society and Space 24: 537-554.

Rossiter, Ned. 2003. “Processual Media Theory.” symploke 11.1-2: 104-131.

Steez. 2010. “Best Cry Ever.” Know Your Meme. Accessed: December 10, 2013. http://knowyourmeme.com/memes/best-cry-ever.

Thrift, Nigel. 2008. Non-Representational Theory: Space|Politics|Affect. Milton Park: Routledge.

 

Moving Image Works Cited

A&E Network. January 8, 2006a. “Salina and Troy”. Intervention. Season 2, episode 5.

A&E Network. July 23rd 2006b. “Tammi and Daniel”. Intervention. Season 2, episode 14.

A&E Network. August 20, 2006c. “Cristy”. Intervention. Season 2, episode 18.

A&E Network. April 20 2007. “Trent”. Season 3, episode 5.

A&E Network. August 18, 2008. “Allison”. Intervention. Season 5, episode 9.

A&E Network. November 23rd 2009a. “Linda”. Intervention. Season 8, episode 1.

A&E Network. December 7, 2009b. “Greg”. Intervention. Season 8, episode 2.

A&E Network. January 4, 2010a. “Sarah”. Intervention. Season 8, episode 6.

A&E Network. January 18, 2010b. “Vinnie”. Intervention. Season 8, episode 8.

A&E Network. April 5, 2010c. “Rocky”. Intervention. Season 8, episode 14.

A&E Network. April 12, 2010d. “Ashley”. Intervention. Season 8, episode 15.

A&E Network. August 23, 2010e. “Ryan / Jason”. Intervention. Season 9, episode 9.

A&E Network. January 3, 2011. “Erin”. Intervention. Season 10, episode 4.

A&E Network. August 13, 2012. “Nichole”. Intervention. Season 13, episode 1.

Comedy Central. April 28, 2010. “Crippled Summer”. South Park. Season 14, episode 7.

Phillips, Todd. The Hangover Part III. 2013.

TLC. Addicted. 2010.

VH1. Celebrity Rehab with Dr. Drew. 2008-2012.

Notes

[1] Rajewsky also discusses the difference between Kristeva’s framework of intertextuality and intermediality (Rajewsky 2005, 48). See also, Julia Kristeva (1986).

[2] Other scholars have looked at the potential for an affective becoming through the application of an intermedial methodology to understanding cinematic texts. Barker, for example, suggests that the interaction between viewer and film is one of emergence, rather than consumption. Bennett deviates from studies of cinematic intermediality in her work on visual arts that incorporate or gesture towards the moving image. However, she is also emphatic that intermediality is processual, liminal and more-than representational.

[3] See also, Lauren Berlant, Cruel Optimism (2011). Berlant argues that it is the desire and striving for an end, which we know will always already fail, which is seductive.

[4] Pethő also discusses this: “we can always tell for example when a piece of prose writing or poetry in literature is unfolding like ‘moving images’, we recognize the characteristics of cinematic ‘framing’ or ‘montage’ whenever it is reflected in any other medium” (2010, 66).

[5] Examples of recent and current reality programs on A&E include Hoarders, Duck Dynasty, Rodeo Girls and Storage Wars (A&E Network 2013a).

[6] Says the website’s copy: “Each episode follows addicts through their daily life and the devastation their dependency has brought to their family and friends. Upon reaching the brink, their loved ones stage a surprise intervention conducted by one of four specialists” (A&E Network 2013b).

[7] The ‘recovered’ status of Van Vonderen and Finnigan also built the program’s claim to authenticity. Both interventionists regularly mentioned their own “struggles with addiction” in episodes and the profiles of Van Vonderen and Finnigan on Intervention’s website foreground this aspect of their history (A&E Network 2013c).

[8] A summary of the variations on the Allison meme are available on Know Your Meme (Don 2013). The “Walking on Sunshine” motif gained further status when it was included in an episode of South Park (Comedy Central 2010).

[9] A summary of the variations on the Rocky meme are available on Know Your Meme (Steez 2010).

[10] Daniels argues that white addicts in Intervention are moralized. Firstly – as addicted – they are framed as ‘failures’ in terms of their performance of normative whiteness. If the addict accepts the offer of rehabilitation, however, they are positioned as redeemed and “deserving” (2012, 114).

[11] This is also reflected in the lyrics to the program’s closing theme song. The lyrics include: “No reprimand / Deliberate, demand / With your two feet at hand / Get back / This train’s a comin’ down the track / Five steps you’re over” (The Davenports 2000). These lyrics presumably gesture towards the “Five Major Steps to Intervention”.

[12] Rocky’s cry is also reproduced in The Hangover 3 where the emphasis lies on the affective aurality of his cry (Phillips 2013).

[13] Here, and throughout the book, Dolar draws on Lacanian psychonanalysis.

[14] The blogger writes: “i don’t know if i have ever mentioned this on here but i am also an addict” [sic] (Fuck Yes Intervention 2013).

[15] See Bennett: intermedial gesture is a “tension … that it expresses … the experience of ‘being in’ an interaction – rather than … articulated communication” (2007, 441).

 

Bio

Rosemary Overell completed a PhD on extreme metal music at the University of Melbourne, Australia. After teaching in Melbourne, she moved to the University of Otago, New Zealand, to take up a lecturer position in the Department of Media, Film and Communication. Currently, she is researching how nikkeijin (Japanese-Brazilians working in Japan) relate to Japanese national space through their experiences in extreme metal music. Her book, Affective Intensities in Extreme Music Scenes: Cases from Australia and Japan was recently published by Palgrave Macmillan.

Editorial: Intermediations — Kevin Fisher & Holly Randell-Moon

This special issue developed out the Intermediations symposium held at the University of Otago on May 31, 2013,[1] and on the invitation of keynote speaker and Refractory Editor, Angela Ndalianis. Presenters at this symposium who have contributed essays here include Kirsten Moana Thomson (the other keynote speaker), John Farnsworth, Kevin Fisher, and Miriam Ross. Topics at the symposium ranged across the terrain of intermedia and transmedia theory, provoking new lines of inquiry on both fronts, and drawing into question the complex relationships between the two emerging paradigms. It is from the extended conversations during and following the symposium that the issue expanded to include essays by Anne Cranny Francis, Rosemary Overell, and Holly Randell-Moon. Some of these essays directly engage the intermedia/transmedia relationship. Kirsten Moana Thompson explores the affinities between animation and more ephemeral forms of theatrical exhibition at Disney theme parks in terms of the sensual dimensions of colour. Rosemary Overell considers the affective intermedial dimensions of the reception and blogging practices surrounding the rehab-based reality TV show Intervention (A&E Network, 2005-2013). Anne Cranny Francis analyses the development of the Sherlock Holmes story world within the convergence culture of transmedia.

Other essays, while working more decisively on one side of the inter/trans spectrum, challenge or expand upon existing approaches in ways that suggest new dialogues. Miriam Ross’s essay investigates sociotechnical debates around vertical framing that issue from the convergence of video and cell phone technologies, and explores their implications within her own media practices. John Farnsworth combines psychological theories of ‘attachment’ with affect studies to suggest how mobile devices simultaneously augment and substitute for social relations. Kevin Fisher describes how the use of 3D imagery in the documentary Cave of Forgotten Dreams (Werner Herzog, 2010) stages the intermedial encounter between a human and pre-human consciousness. Holly Randell-Moon analyses how allusions to civil rights advocacy and debate in True Blood (HBO, 2008-2014) work in the service of the biopolitical management of difference under the aegis of transmedia consumer participation. Together, the essays constitute a critical inquiry into the emergence of inter- and transmedia in the disciplines of media, cultural and film studies and how these terms both illustrate and re-ignite sociotechnical forces and debates in digital media and convergence culture. In the following section, we offer a brief genealogy of inter/trans media analysis, focusing specifically on the terms’ phenomenological and ideological valences in scholarly reception and utility.

In between and among: a brief tracing of inter/trans media analysis

Over the past two decades academic discussions of intermediality and transmediation have undergone a parallel development within the context of what Henry Jenkins describes as digital convergence culture. However, the exponents of each have, with few exceptions, tended to talk past one another. This is paradoxical insofar as the phenomena they respectively describe are often intertwined in the media examples they differently engage. While transmedia analysis has been primarily concerned with the distribution of narrative across media platforms, intermedial analysis has interrogated the internal singularity and ‘specificity’ of those same medialities. The experience of transmediation involves the participation of interpretive communities in the co-creation of stories and the enactment of story worlds. By contrast, intermedial experience unfolds within the heterogeneous spaces generated along the various intersections of medial forms and traces within a given medium.

The subject of transmedia combines the active viewer of cultural studies and the social media user within an expanded understanding of narrative as an irreducible component of human experience, cognition and social activity. This anthropological notion of homo narrativus is shared by the academic methods of transmedia analysis as well as creative methods of transmedia storytelling co-emergent with commercial practices such as viral marketing. Scholarly interest in this ‘new’ form of storytelling can be traced to Alvin Toffler’s development of the term ‘prosumer’, coined to describe a shift in audience and consumer activity that was more self-directed, individualised and selective than the traditional mass media model of consumption and production (1980). Following on from this work, Axel Bruns (2008) and Henry Jenkins (2006) have explored how the ‘produser’ repositions the production and communication flows of media content from media companies and creators to the consumer/user. As Jenkins explains, “Reading across the media sustains a depth of experience that motivates consumption” (2003) and “A good transmedia franchise attracts a wider audience by pitching the content differently in the different media” (2003). Audiences can read media texts with an awareness of their transmedial dimensions or they can consume different media forms in isolation whilst still being interpellated into a broader transmedia story. Jenkins’ development of transmedia is thus an attempt to capture the new specificities of medial engagement that have emerged from digital convergence and new media formats. He identifies a number of transmedia modes of communication which include: transmedia storytelling, transmedia branding, transmedia performance, transmedia ritual, transmedia play, transmedia activism, and transmedia spectacle (2011). [2]

There are two important implications to be drawn from this type of cross-media communication. The first is that transmedia forms of communication require an explicit appreciation of the intertextual (though not necessarily intermedial) elements of storytelling on the part of media producers. The second is that this type of media storytelling and communication recognises the social character of narrative and textual construction. Writing about transmedia fan activity, Jenkins speaks of “a new kind of cultural power emerging as fans bond together within larger communities, pool their information, shape each other’s opinions, and develop a greater self-consciousness about their shared agendas and common interests” (2007, 362-363). Kaarina Nikunen also suggests that fan activities reveal “the institutional and technological spaces of shaping the pleasures of media” which also “possibly reshape […] audience practises more widely” (2007, 111). What this type of media engagement does is shift political and ideological discussion of audiences’ (pleasurable and social) involvement in meaning making from the passive/active consumer debate to questions of the audience’s role in the economy of media production and consumption.

It is this seeming incorporation of fan and audience desire into the narratives of media production that has generated scepticism about the extent to which produsage challenges or subverts existing media structures. S. Elizabeth Bird for example, points out that “True produsers are a reality, but they are not the norm, and can often seem to be so in thrall to big media and technological ‘coolness’ that they accept the disciplining of their creative activities” (2011, 512). Indeed, the end goal of transmedia branding according to social media marketer Rick Liebling is “creating an environment that is so authentic and compelling that when consumers do generate their own content that utilizes your brand, they do so in a way that is in line with your existing messaging” (2011; emphasis in original). For this reason, fan activity as a form of produsage qua consumer action (or more idealistically, resistance) may also be understood as “a form of market-sanctioned cultural experimentation through which the market rejuvenates itself” (Holt, as cited in Kline 2009, 32). The critical distance between a marketing approach to transmedia activity and a more scholarly one is the extent to which audience activity can instantiate resistance or subversion to existing media and communication hierarchies. Indeed, such concerns as they relate to media’s enmeshment in other political institutions specifically inform Randell-Moon’s essay in this issue.

One of the more salient critiques of transmedia analysis is that medial specificity is subsumed within the overall importance of the story, even if as Jenkins argues, transmedia storytelling relies neither on the continuity nor homogeneity of its narrative. Still, according to Jenkins, “Most discussions of transmedia place a high emphasis on continuity—assuming that transmedia requires a high level of coordination and creative control and that all of the pieces have to cohere into a consistent narrative or world” (2011). For Ndalianis it is the “holes” within transmedia stories that create opportunities for audience co-creation and performance, and that these types of co-creation are among the most successful examples of transmedia campaigns (2012, 174). Yet, even with its emphasis on the cross-media processes of audience engagement, transmedia still implies a substrate of medial relations where there is an experiential sameness across platforms. As Bernd Herzogenrath notes, the transmedial version of intermediality “is built on the concept that there are formal structures (such as narrative structures) that are not specific to one medium but can be found (perhaps differently instantiated) in different media” (2012, 4). Consequently, transmedia analysis “has the problem that ‘media specificity’ cannot be conceptualized within it” (4). By contrast, the issue of media specificity takes centre stage in Francesco Casetti’s analysis of the “relocation of cinema” as medial form beyond its traditional substrate (2011), which also animates Ross’s examination of the convergence of video and mobile telephony in this issue. This centrifugal thrust of intermedial analysis against the internal coherence and specificity of medialities within what Rosalind Krauss terms “the post medium condition” (1999) provides a counterpoint to the centripetal force of narrative implied in Jenkins’ convergence culture.

In this issue, Cranny-Francis traces the term intermediality back to Roland Barthes, where he appeals to the interdisciplinarity required by new cultural objects that defy prevailing codes and classifications. She argues that intertextuality, in Mikhail Bakhtin’s sense of “heteroglossia”, provides the methodological link between intermedial and transmedial analysis. Transmedia storytelling is, in important ways, an inherently intermedial phenomenon because it depends on and generates engagement with media texts as multiple and heterogeneous. The forms of reading and engagement across transmedia stories, as outlined by Jenkins, have similarities with intermedia defined by Herzogenrath as “between the between” (2012, 2) in the sense that “we can only refer to media using other media” (3). In relation to what Cranny-Francis describes as a process of endless intertextual deferral, Herzongenrath observes: “Individual media do not exist in isolation, to be suddenly taken into intermedial relations. Intermediality is rather the ontological condition sine qua non, which is always before ‘pure’ and specific media, which have to be extracted from the arch-intermediality” (4). The intermedial thus constitutes “the quicksand out of which specific media emerge” as well as “the various interconnections” made possible between the audience and different types of media (3).

Other contemporary theorists, such as Ágnes Pethő (2011) and Joachim Paech (2011), insist that intermediality is altogether distinct from intertextuality, which reproduces the privileging of narrative characteristic of transmedia, conflating relations between stories with intersections between medialities. Pethő, for example, describes intermedial experience as extra-narrative, extra-representational, and a-signifying. Hence, “it cannot be read” (2011, 67). Rather, as an encounter with the ‘in-between’ generated along the interstices of different medial forms and traces, intermediality makes itself felt on the prereflective level of embodied sensation. Hence, for Pethő, and contributors Moana-Thomson as well as Fisher, intermediality is an irreducibly phenomenological experience. Other essays, such as those by Farnsworth and Overell draw upon affect studies and non-representational theory to approach the embodied aspects of intermediality that escape both medium-specific and hermeneutic containment of media texts. For example, Farnsworth explores the affective and psychoanalytical dimensions of attachment as a constituent feature of embodiment and sociality that become augmented or constrained through mobile technologies.

However, the emphasis of the intermedial on embodiment and affect over interpretation has also informed some strains of transmedia theory, in particular Ndalianis’ work on transmedia horror as predicated on affective participation in a particular “sensorium” (2012). The focus of intermedial analysis on the heterogeneous spaces and experiences between medialities also complements the methodological and historiographical projects of media archaeology (Elsaesser 2005, 2009; Huhtamo and Parikka 2011; Parikka 2012) and remediation (Bolter and Grusin 1999). Paech, for example, echoes Jay David Bolter and Richard Grusin’s logic of remediation by arguing that film has always been intermedial, though its experience as such becomes more pronounced or “hypermediated” during historical periods characterised by intensified sociotechnical change (1999). At this moment in time, renewed interest in medial co-creation is heightened by the shifting economies of convergence culture and the post-medium environment, in whose context the paradigms of intermedial and transmedial analysis will continue to be subject to the same exchanges and mutations as the medialities they describe. Such mutations occur, we would argue, as intermediations between audience, text, screen and body as a constitutive feature of medial meaning and sensation.

In this issue, we offer some intermediations on the changing dynamics of mediality in relation to embodiment, media specificity, and audience participation in and performance of textuality. We hope you enjoy reading the essays.

 

References

Bird, Elizabeth S. 2011. “Are We All Produsers Now? Convergence and Media Audience Practices.” Cultural Studies 25 (4-5): 502-516.

Bolter, Jay David and Richard Grusin. 1999. Remediation: Understanding New Media.Cambridge: MIT Press.

Bruns, Alex. 2008. Blogs, Wikipedia, Second Life and Beyond: From Production to Produsage. New York: Peter Lang Publishing, Inc.

Casetti, Francesco. 2011. “Back to the Motherland: The Film Theatre in the Postmedia Age.” Screen 52 (1): 1-12.

Elsaesser, Thomas. 2005. “The New Film History as Media Archaeology.” Cinemas 14 (2-3 Spring): 75-117.

Elsaesser, Thomas. 2009. “Archaeologies of Interactivity: Early Cinema, Narrative and Spectatorship.” In Film 1900: Technology, Perception, Culture, edited by Klaus Kreimeier and Annemone Ligensa, 9-22. Indianapolis: Indiana University Press.

Herzogenrath, Bernd. 2012. “Travels in Intermedia[lity]: An Introduction.” In Travels in Intermedia[lity]: ReBlurring the Boundaries, edited by Bernd Herzogenrath: 1-14. Lebanon, NH: Dartmouth College Press.

Huhtamo, Erkki and Jussi Parikka, editors. 2011. Media Archaeology: Approaches, Applications, and Implications. Berkeley: University of California Press.

Jenkins, Henry. 2003. “Transmedia Storytelling.” Technology Review, January 15. Accessed June 28, 2014. http://www.technologyreview.com/news/401760/transmedia-storytelling/.

Jenkins, Henry. 2006. Fans, Bloggers, and Gamers: Exploring Participatory Culture. New York: New York University Press.

Jenkins, Henry. 2007. “Afterword: the future of fandom.” In Fandom: Identities and Communities in a Mediated World, edited by Jonathan Gray, Cornel Sandvoss and C. Lee Harrington, 357-364. New York: New York University Press.

Jenkins, Henry. 2011. “Transmedia 202: Further Reflections.” Confessions of an Aca-Fan, August 1. Accessed April 28, 2014. http://henryjenkins.org/2011/08/defining_transmedia_further_re.html.

Kline, Stephen. 2009. “Ronald’s New Dance: A Case Study of Corporate Rebranding in the Age of Integrated Communication.” In The Advertising Handbook (3rd edition), edited by Helen Powell, Jonathan Hardy, Sarah Hawkin and Iain MacRury, 24-33. London: Routledge.

Krauss, Rosalind. 1999. “A Voyage on the North Sea”: Art in the Age of the Post-Medium Condition. New York: Thames & Hudson.

Liebling, Rick. 2011. “Intermedia—The Next Phase in Consumer Engagement.” How Soon is Now?: Culture in a 24/7 World, September 11. Accessed April 28, 2014. http://www.rickliebling.com/2011/09/11/intermedia-the-next-phase-in-consumer-engagement/.

Ndalianis, Angela. 2012. The Horror Sensorium: Media and the Senses. Jefferson: McFarland Publishing.

Nikunen, Kaarina. 2007. “The Intermedial Practises of Fandom.” Nordicom Review 28 (2): 111-128.

Paech, Joachim. 2011. “The Intermediality of Film.” Acta Univ. Sapientiae, Film and Media Studies 4: 7-21. Accessed May 7, 2014. http://www.acta.sapientia.ro/acta-film/C4/Film4-1.pdf.

Parikka, Jussi. 2012. What is Media Archaeology? Cambridge: Polity Press.

Pethő, Ágnes. 2011. Cinema and Intermediality: The Passion for the In-Between. Newcastle upon Tyne: Cambridge Scholars Publishing.

Toffler, Alvin. 1980. The Third Wave. New York: Bantam Books.

 

Filmography

Ball, Alan. True Blood. 2008-2014. USA: HBO.

Herzog, Werner. 2010. Cave of Forgotten Dreams. USA: Sundance Selects.

Mettler, Sam. Intervention.2005-2013. USA: A&E Network.

 

Notes

[1] The “Intermediations” Symposium was organised by Catherine Fowler and Paul Ramaeker in conjunction with the Screen Cultures Research Group and the Department of Media, Film and Communication at the University of Otago.

[2] Of these types of transmedia communication, transmedia storytelling and branding appear to have captured scholarly and popular interest above the other significant and no less interesting forms of transmedia identified by Jenkins.

« Previous Page
Next Page »

Archives

Subscribe to Blog via Email

Categories

Tags

blockbuster born digital cinephilia computer games cultural heritage Dorian Gray Dracula Dr Jekyll and Mr Hyde eye-tracking fan culture fans film flâneur Frankenstein Games gender gothic Grand Guignol Hollywood horror horror films horror television horror TV intermedia intermediality Manovich monsters music narrative Penny Dreadful reality tv Sherlock spectator spectators spectatorship supernatural Television transmedia uncanny vampires Vanessa Ives victorian videogames vision werewolf

Meta

  • Log in
  • Entries RSS
  • Comments RSS
  • WordPress.org

Tags

blockbuster born digital cinephilia computer games cultural heritage Dorian Gray Dracula Dr Jekyll and Mr Hyde eye-tracking fan culture fans film flâneur Frankenstein Games gender gothic Grand Guignol Hollywood horror horror films horror television horror TV intermedia intermediality Manovich monsters music narrative Penny Dreadful reality tv Sherlock spectator spectators spectatorship supernatural Television transmedia uncanny vampires Vanessa Ives victorian videogames vision werewolf

CyberChimps WordPress Themes

CyberChimps ©2021