Subtitles on the movie image: an overview of eye tracking studies

Authors

DOI:

https://doi.org/10.5007/2175-7968.2020v40n2p377

Abstract

This article provides an overview of eye tracking studies on subtitling (also known as captioning), and makes recommendations for future cognitive research in the field of audiovisual translation (AVT). We find that most studies in the field that have been conducted to date fail to address the actual processing of verbal information contained in subtitles, and rather focus on the impact of subtitles on viewing behaviour. We also show how eye tracking can be utilized to measure not only the reading of subtitles, but also the impact of stylistic elements such as language usage and technical issues such as the presence of subtitles during shot changes on the cognitive processing of the audiovisual text as a whole. We support our overview with empirical evidence from various eye tracking studies conducted on a number of languages, language combinations, viewing contexts as well as different types of viewers/readers, such as hearing, hard of hearing and Deaf people.

Author Biographies

Jan-Louis Kruger, Universidade de Macquarie, Sidney,

My research uses eye tracking experiments combined with psychometric instruments and performance measures to investigate the processing of language in multimodal contexts. I am particularly interested in the processing of text (reading) when it has to compete for cognitive resources with auditory and visual information (such as reading subtitles in film). My current work looks at the impact of such redundant sources of information on the reading of subtitles at different presentation rates and in the presence of different languages.

Szarkowska Agniezka, Universidade de Varsóvia, Varsóvia,

Agnieszka is a researcher, academic teacher, ex-translator, translator trainer, and media accessibility consultant. She is the head of AVT Lab, one of the first research groups on audiovisual translation. Her research projects include eye tracking studies on subtitling, voice-over, audio description, multilingualism in subtitling for the deaf and the hard of hearing, respeaking, and modern art for all.

Izabela Krejtz, SWPS Universidade de Ciências Sociais e Humanidades, Varsóvia,

Dr. Izabela Krejtz is an associate professor of Interdisciplinary Center for Applied Cognitive Studies at University of Social Sciences and Humanities Computer Science.

She received her MA (1998) from Warsaw University, and doctorate (2002) from Institute of Psychology, Polish Academy of Sciences, both in Psychology. She is a specialist in eye-tracking methodology and cognitive psychopathology. Her research interests circle around the field of eye tracking and positive psychology focusing on factors influencing daily well-being. Her applied work focuses on pro-positive trainings of attention control and eye-tracking studies in perception of subtitles by deaf and hard-of-hearing. She is a cofounder of Eye tracking Research Center at University of Social Sciences and Humanities Computer Science.

She is a project leader of an innovative online pro-positive training aimed at attentional bias modification among clinically depressed. She was a leader of a grant from Foundation for Polish Science related to predictors of daily well-being and stress.

Sebastião Braga-Junior, Universidade Federal do Ceará, Fortaleza, Ceará,

Tem experiência na área de Letras, com ênfase em Línguas Estrangeiras Modernas e Tradução. Licenciado em Letras Inglês pela Universidade Estadual do Ceará. Mestrando em Estudos da Tradução na Universidade Federal do Cerará. Autor do Livro "Diário Póstumo de Charlotte" e "O brilho da Maldade". Atende pelo pseudônimo de Jairo Sarfati. É escritor, professor, tradutor e revisor.

References

Bisson, Marie-Josée, et al. “Processing of Native and Foreign Language Subtitles in Films: An Eye Tracking Study.” Applied Psycholinguistics. v. 35, núm. 2, (2014):399-418.

Burnham, Denis, et al. “Parameters in television captioning for deaf and hard-of-hearing adults: effects of caption rate versus text reduction on comprehension.” Journal of Deaf Studies and Deaf Education. v. 13, núm. 3, (2008):391-404.

de Linde, Zoé; Kay, Neil. The Semiotics of Subtitling. Manchester: St. Jerome, 1999.

Diao, Y. et al. “The effect of written text on comprehension of spoken English as a foreign language.” The American Journal of Psychology. v. 120, núm. 2, (2007): 237-261.

Díaz Cintas, Jorge; Fernandez Cruz, Marco. “Using subtitled video materials for foreign language instruction”. In: Díaz Cintas, Jorge. (Ed.). The Didactics of Audiovisual Translation. Amsterdam/Philadelphia: John Benjamins, 2008, p. 201-214.

Díaz Cintas, Jorge; Remael, Aline. Audiovisual Translation: Subtitling. Manchester: St. Jerome, 2007.

d’Ydewalle, Géry; De Bruycker, Wim (a). “Reading native and foreign language television subtitles in children and adults.” In: Hyönä, J.; Radach, R.; Deubel, H. (Ed.). The mind’s eyes: Cognitive and applied aspects of eye movement research. New York: Springer-Verlag, 2003, p. 444-461.

d’Ydewalle, Géry; De Bruycker, Wim (b). “Eye Movements of Children and Adults while Reading Television Subtitles”. European Psychologist. v. 12, (2007):196–205.

d’Ydewalle, Géry; Gielen, Ingrid. “Attention Allocation with Overlapping Sound, Image, and Text.” In: Rayner, K. (Ed.). Eye Movements and Visual Cognition: Scene Perception and Reading. New York: Springer-Verlag, 1994, p. 415–427.

d’Ydewalle, Géry, et al. “Reading a message when the same message is available auditorily in another language: The case of subtitling.” In: O’Reagan, J. K.; Lévy Schoen, A. (Eds.). Eye Movements: From Physiology to Cognition. Amsterdam: Elsevier Science Publishers B.V, 1987, p. 313-321.

Ghia, Elisa. 2012. “The Impact of Translation Strategies on Subtitle Reading.” In: Perego, Elisa. (Ed.) Eye Tracking in Audiovisual Translation. Roma: Aracne Editrice, 2012, p. 155–182.

Gottlieb, Henrik. “Subtitling”. In: Baker, Mona. (Ed.). Routledge Encyclopaedia of Translation Studies. London & New York: Routledge, 1998, p. 244-248.

Hershler, Orit; Hochstein, Shaul. “At first sight: a high-level pop out effect for faces”. Vision Research. v. 45, (2005):1707–1724.

Holmqvist, Kenneth et al. Eyetracking. A Comprehensive Guide to Methods and Measures. Oxford: Oxford University Press, 2011.

Irwin, David E. “Fixation location and fixation duration as indices of cognitive processing”. In: Henderson, J.M.; Ferreira, F. (Eds.). The interface of language, vision, and action: Eye movements and the visual world. New York, NY: Psychology Press, 2004, p. 105-133.

ITC Guidance on Standards for Subtitling. Disponível em: http://www.ofcom.org. uk/static/archive/itc/itc_publications/codes_guidance/standards_for_subtitling/ subtitling_1.asp.html.

Jensema, Carl. “Eye movement patterns of captioned TV viewers”. American Annals of the Deaf. v. 145, núm. 3, (2000):275-285.

Karamitroglou, Fotios. “A Proposed Set of Subtitling Standards in Europe”. Translation Journal. v. 2, núm 2, 1999. Disponível em: http:// translationjournal.net/journal/04stndrd.htm.

Krejtz, Izabela, et al. “The Effects of Shot Changes on Eye Movements in Subtitling”. Journal of Eye Movement Research. v. 6, núm, 5, (2013): 1–12.

Kruger, Jan-Louis; Steyn, Faans. “Subtitles and Eye Tracking: Reading and Performance.” Reading Research Quarterly. v. 49, núm. 1, (2014): 105–120.

Kruger, Jan-Louis, et al (a). “Measuring the Impact of Subtitles on Cognitive Load: Eye Tracking and Dynamic Audiovisual Texts”. Proceedings of Eye Tracking South Africa. 29-31 August 2013, Cape Town, 2013.

Kruger, Jan-Louis, et al (b). “The impact of subtitles on academic performance at tertiary level”. Paper presented at the Linguistics Society of Southern Africa annual conference in Stellenbosch, June, 2013.

Kruger, Jan-Louis. “Subtitles in the Classroom: Balancing the Benefits of Dual Coding with the Cost of Increased Cognitive Load”. Journal for Language Teaching. v. 47, núm. 1, (2013):29–53.

Kruger, Jan-Louis, et al (c). “Attention distribution and cognitive load in a subtitled academic lecture: L1 vs. L2”. Journal of Eye Movement Research. v. 7, núm. 5, (2014): 1–15.

Langton, Stephen R.H., et al. “Attention capture by faces”. Cognition. v. 107, (2008):330-342.

Pavakanun, Ubowanna. Incidental acquisition of foreign language through subtitled television programs as a function of similarity with native language and as a function of presentation mode. 1992. Tese de doutorado. Universidade de Leuven, Bélgica.

Perego, Elisa, et al. “The Cognitive Effectiveness of Subtitle Processing”. Media Psychology. v. 13, núm. 3, (2010):243–272.

Rajendran, Dhevi, et al. “Effects of Text Chunking on Subtitling: A Quantitative and Qualitative Examination”. Perspectives: Studies in Translatology. v. 21, núm. 1, (2013):5–31.

Rayner, Keith (a). “Visual selection in reading, picture perception, and visual search: A tutorial review”. In: Bouma, H; Bouhwhuis, D. (Ed.). Attention and performance. vol. 10. Hillsdale, NJ: Erlbaum, 1994.

Rayner, Keith (b). “Eye movements in reading and information processing: Twenty years of research”. Psychological Bulletin. v. 124, (1998):372–422.

Robson, Gary D. The closed captioning handbook. Amsterdam: Elsevier, 2004.

Romero Fresco, Pablo. The Reception of Subtitles for the Deaf and Hard of Hearing in Europe. Berna: Peter Lang. (in press).

Smith, Tim; Henderson, John M. “Edit Blindness: The relationship between attention and global change blindness in dynamic scenes”. Journal of Eye Movement Research. v. 2, núm. 2, (2008):1-17.

Specker, Elizabeth, A. L1/L2 Eye Movement Reading of Closed Captioning: A Multimodal Analysis of Multimodal Use. 2008. Unpublished PhD thesis. Universidade do Arizona.

Szarkowska, Agnieszka, et al. “The effects of subtitle presentation rate, text editing and type of subtitling on the comprehension and reading patterns of subtitles among deaf, hard of hearing and hearing viewers”. Across Languages and Cultures. vol. 2, 2016. (forthcoming).

Szarkowska, Agnieszka, et al. “Verbatim, standard, or edited? Reading patterns of different captioning styles among deaf, hard of hearing, and hearing viewers”. American Annals of the Deaf. v. 156, núm. 4, (2011):363-378.

Vanderplank, Robert. “The value of teletext sub-titles in language learning”. ELT Journal. v. 42, núm. 4, (1988):272-81.

Velichkovsky, Boris M., et al. “Two Visual Systems and Their Eye Movements: Evidence from Static and Dynamic Scene Perception”. InCogSci 2005: Proceedings of the XXVII Conference of the Cognitive Science Society. (2005):2283–2288.

Winke, Paula, et al. “Factors Influencing the Use of Captions by Foreign Language Learners: An Eye Tracking Study”. The Modern Language Journal. v. 97, núm. 1, (2013):254–275.

Yarbus, Alfred L. Eye movements and vision. New York: Plenum Press, 1957.

Downloads

Published

2020-05-19

How to Cite

Kruger, J.-L., Agniezka, S., Krejtz, I., & Braga-Junior, S. (2020). Subtitles on the movie image: an overview of eye tracking studies. Cadernos De Tradução, 40(2), 377–409. https://doi.org/10.5007/2175-7968.2020v40n2p377

Issue

Vol. 40 No. 2 (2020): Edição Regular

Section

Translated Articles

License

Copyright Notice

Authors hold the copyright and grant the journal the right for their articles' first publication, being their works simultaneously licensed under the Creative Commons Attribution License (CC BY), which allows the sharing of such works with its authorship acknowledged and its initial publication in this journal.

Authors are allowed to enter into separate additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or as a book chapter, with an acknowledgment of its initial publication in this journal).

 
Licença Creative Commons