How well can state-of-the-art machine translation systems render a 16th-century Chinese novel?
DOI:
https://doi.org/10.5007/2175-7968.2025.e108394Palavras-chave:
machine translation, Chinese-Portuguese literary translation, Journey to the West, human evaluation, culture-specific itemResumo
This study evaluates the performance of state-of-the-art machine translation systems in rendering Journey to the West, a culturally rich 16th-century Chinese novel, into Portuguese. Employing a mixed-methods approach, we compare translations produced by DeepSeek-V3, GPT-4o, DeepL Pro, and NovelTrans-J against a published human translation. Quantitative assessments conducted by an expert evaluator examine accuracy, fluency, stylistic elegance, cultural appropriateness, and overall translation quality at both the sentence and chunk levels. The results reveal that three MT systems (DeepSeek-V3, GPT-4o, and NovelTrans-J) produce translations of comparable or superior quality to the human translation. Among them, NovelTrans-J consistently outperforms all other participants, particularly in terms of cultural appropriateness. In contrast, DeepL Pro demonstrates significantly weaker performance across all evaluated dimensions. To complement the quantitative analysis, a qualitative investigation focuses on the rendering of culture-specific items (CSIs). NovelTrans-J exhibits outstanding performance, producing the fewest mistranslations and uniquely providing explanatory notes that facilitate reader comprehension. DeepSeek-V3 and GPT-4o also handle CSIs competently, though with less consistency, while DeepL Pro struggles considerably, showing a high rate of CSI mistranslations and generally low quality. Interestingly, the human translation also contains notable CSI-related errors, particularly in cases involving semantically opaque expressions, an area in which all participants encounter significant difficulty. These findings underscore the growing potential of MT systems to handle complex, culturally rich literary texts, although certain challenges, such as the translation of semantically opaque expressions, remain significant obstacles. We hope this study provides an updated perspective on the current capabilities of MT and offers practical insights to guide the development of future systems that can more accurately capture and transmit the distinctive cultural nuances embedded in literary works.
Referências
Akhbardeh, F., Arkhangorodsky, A., Biesialska, M., Bojar, O., Chatterjee, R., Chaudhary, V., Costa-jussa, M. R., España-Bonet, C., Fan, A., Federmann, C., Freitag, M., Graham, Y., Grundkiewicz, R., Haddow, B., Harter, L., Heafield, K., Homan, C., Huck, M., Amponsah-Kaakyire, K., Kasai, J., Khashabi, D., Knight, K., Kocmi, T., Koehn, P., Lourie, N., Monz, C., Morishita, M., Nagata, M., Nagesh, A., Nakazawa, T., Negri, M., Pal, S., Tapo, A. A., Turchi, M., Vydrin, V., & Zampieri, M. (2021). Findings of the 2021 Conference on Machine Translation (WMT21). In L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussa, C. Federmann, M. Fishel, A. Fraser, M. Freitag, Y. Graham, R. Grundkiewicz, P. Guzman, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, T. Kocmi, A. Martins, M. Morishita & C. Monz (Eds.), Proceedings of the Sixth Conference on Machine Translation (pp. 1–88). Association for Computational Linguistics. https://aclanthology.org/2021.wmt-1.1/
Amenador, K. B., & Wang, Z. (2023). The Image of China as a Destination for Tourist: Translation Strategies of Culture-Specific Items in the Chinese-English Food Menus. SAGE Open, 13(3), 1–14. https://doi.org/10.1177/21582440231196656
Barrault, L., Bojar, O., Costa-jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., Monz, C., Müller, M., Pal, S., Post, M., & Zampieri, M. (2019). Findings of the 2019 Conference on Machine Translation (WMT19). In O. Bojar, R. Chatterjee, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, A. Martins, C. Monz, M. Negri, A. Névéol, M. Neves, M. Post, M. Turchi & K. Verspoor (Eds.), Proceedings of the Fourth Conference on Machine Translation – Volume 2: Shared Task Papers, Day 1 (pp. 1–61). Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-5301
Barrault, L., Biesialska, M., Bojar, O., Costa-jussà, M. R., Federmann, C., Graham, Y., Grundkiewicz, R., Haddow, B., Huck, M., Joanis, E., Kocmi, T., Koehn, P., Lo, C.-k., Ljubešić, N., Monz, C., Morishita, M., Nagata, M., Nakazawa, T., Pal, S., Post, M., & Zampieri, M. (2020). Findings of the 2020 Conference on Machine Translation (WMT20). In L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussà, C. Federmann, M. Fishel, A. Fraser, Y. Graham, P. Guzman, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, A. Martins, M. Morishita, C. Monz, M. Nagata, T. Nakazawa & M. Negri (Eds.), Proceedings of the Fifth Conference on Machine Translation (pp. 1–55). Association for Computational Linguistics. https://aclanthology.org/2020.wmt-1.1/
Bojar, O., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huang, S., Huck, M., Koehn, P., Liu, Q., Logacheva, V., Monz, C., Negri, M., Post, M., Rubino, R., Specia, L., & Turchi, M. (2017). Findings of the 2017 Conference on Machine Translation (WMT17). In O. Bojar, C. Buck, R. Chatterjee, C. Federmann, Y. Graham, B. Haddow, M. Huck, A. J. Yepes, P. Koehn & J. Kreutzer (Eds.), Proceedings of the Second Conference on Machine Translation (pp. 169–214). Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-4717
Bojar, O., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., & Monz, C. (2018). Findings of the 2018 Conference on Machine Translation (WMT18). In O. Bojar, R. Chatterjee, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, C. Monz, M. Negri, A. Névéol, M. Neves, M. Post, L. Specia, M. Turchi & K. Verspoor (Eds.), Proceedings of the Third Conference on Machine Translation: Shared Task Papers (pp. 272–303). Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-6401
Castilho, S., Doherty, S., Gaspari, F., & Moorkens, J. (2018). Approaches to Human and Machine Translation Quality Assessment. In J. Moorkens, S. Castilho, F. Gaspari & S. Doherty (Eds.), Translation Quality Assessment: From Principles to Practice (pp. 9-38). Springer International Publishing. https://doi.org/10.1007/978-3-319-91241-7_2
Colina, S. (2008). Translation Quality Evaluation: Empirical Evidence for a Functionalist Approach. The Translator, 14(1), 97–134. https://doi.org/10.1080/13556509.2008.10799251
Colina, S. (2009). Further Evidence for a Functionalist Approach to Translation Quality Evaluation. Target, 21(2), 235–264. https://doi.org/10.1075/target.21.2.02col
Colina, S. (2015). Fundamentals of Translation. Cambridge University Press.
Davies, E. E. (2003). A Goblin or a Dirty Nose? The Treatment of Culture-Specific References in Translations of the Harry Potter Books. The Translator, 9(1), 65–100. https://doi.org/10.1080/13556509.2003.10799146
Díaz Cintas, J., & Remael, A. (2007). Audiovisual Translation: Subtitling. Routledge. https://doi.org/10.4324/9781315759678
Drugan, J. (2013). Quality in Professional Translation: Assessment and Improvement. Bloomsbury.
Efron, B. (1992). Bootstrap Methods: Another Look at the Jackknife. In S. Kotz & N. L. Johnson (Eds.), Breakthroughs in Statistics: Methodology and Distribution (pp. 569-593). Springer. https://doi.org/10.1007/978-1-4612-4380-9_41
Franco Aixelá, J. F. (1996). Culture-Specific Items in Translation. In Á. Román & M. C.-Á. Vidal (Eds.), Translation, power, subversion (pp. 52-78). Multilingual Matters.
Gottlieb, H. (2009). Subtitling Against the Current: Danish Concepts, English Minds. In J. Díaz Cintas (Ed.), New Trends in Audiovisual Translation (pp. 21-43). Multilingual Matters. https://doi.org/10.21832/9781847691552-004
Graham, Y., Baldwin, T., Moffat, A., & Zobel, J. (2013). Continuous Measurement Scales in Human Evaluation of Machine Translation. In A. Pareja-Lora, M. Liakata & S. Dipper (Eds.), Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse (pp. 33–41). Association for Computational Linguistics. https://aclanthology.org/W13-2305/
Hague, D., Alan, M., & Zheng, W. (2011). Surveying Translation Quality Assessment: a Specification Approach. The Interpreter and Translator Trainer, 5(2), 243–267. https://doi.org/gr5qdq
House, J. (1997). Translation Quality Assessment: a Model Revisited. Narr.
House, J. (2001). Translation Quality Assessment: Linguistic Description versus Social Evaluation. Meta, 46(2), 243–257. https://doi.org/10.7202/003141ar
House, J. (2014). Translation Quality Assessment: Past and Present. Routledge. https://doi.org/10.4324/9781315752839
Katan, D. (2009). Translator Training and Intercultural Competence. In S. Cavagnoli, E. Di Giovanni & R. Merlini (Eds.), La Ricerca nella Comunicazione Interlinguistica. Modelli Teorici e Metodologici (pp. 282–301). Franco Angeli.
Kocmi, T., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Gowda, T., Graham, Y., Grundkiewicz, R., Haddow, B., Knowles, R., Koehn, P., Monz, C., Morishita, M., Nagata, M., Nakazawa, T., Novák, M., Popel, M., & Popović, M. (2022). Findings of the 2022 Conference on Machine Translation (WMT22). In P. Koehn, L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussà, C. Federmann, M. Fishel, A. Fraser, M. Freitag, Y. Graham, R. Grundkiewicz, P. Guzman, B. Haddow, M. Huck, A. Jimeno Yepes, T. Kocmi, A. Martins, M. Morishita, C. Monz, M. Nagata, T. Nakazawa, M. Negri, A. Névéol, M. Neves, M. Popel, M. Turchi & M. Zampieri (Eds.), Proceedings of the Seventh Conference on Machine Translation (WMT) (pp. 1–45). Association for Computational Linguistics. https://aclanthology.org/2022.wmt-1.1/
Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Koehn, P., Marie, B., Monz, C., Morishita, M., Murray, K., Nagata, M., Nakazawa, T., Popel, M., Popović, M., Shmatova, M., & Suzuki, J. (2023). Findings of the 2023 Conference on Machine Translation (WMT23): LLMs Are Here but Not Quite There Yet. In P. Koehn, B. Haddow, T. Kocmi & C. Monz (Eds.), Proceedings of the Eighth Conference on Machine Translation (pp. 1–42). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.1
Kocmi, T., Avramidis, E., Bawden, R., Bojar, O., Dvorkovich, A., Federmann, C., Fishel, M., Freitag, M., Gowda, T., Grundkiewicz, R., Haddow, B., Karpinska, M., Koehn, P., Marie, B., Monz, C., Murray, K., Nagata, M., Popel, M., Popović, M., Shmatova, M., Steingrímsson, S., & Zouhar, V. (2024). Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet. In B. Haddow, T. Kocmi, P. Koehn & C. Monz (Eds.), Proceedings of the Ninth Conference on Machine Translation (pp. 1–46). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.wmt-1.1
Lauscher, S. (2000). Translation Quality Assessment: Where Can Theory and Practice Meet? The Translator, 6(2), 149–168. https://doi.org/10.1080/13556509.2000.10799063
Leppihalme, R. (2011). Realia. In Y. Gambier & L. van Doorslaer (Eds.), Handbook of Translation Studies (pp. 126–130). John Benjamins Publishing Company. https://doi.org/10.1075/hts.2.rea1
Liu, Y., Yao, Y., Zhan, R., Lin, Y., & Wong, D. F. (2024). NovelTrans: System for WMT24 Discourse-Level Literary Translation. In B. Haddow, T. Kocmi, P. Koehn & C. Monz (Eds.), Proceedings of the Ninth Conference on Machine Translation (pp. 980–986). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.wmt-1.98
Lommel, A. (2018). Metrics for Translation Quality Assessment: A Case for Standardising Error Typologies. In J. Moorkens, S. Castilho, F. Gaspari & S. Doherty (Eds.), Translation Quality Assessment: From Principles to Practice (pp. 109–127). Springer International Publishing. https://doi.org/10.1007/978-3-319-91241-7_6
Marco, J. (2007). The Terminology of Translation: Epistemological, Conceptual and Intercultural Problems and Their Social Consequences. Target, 19(2), 255-269. https://doi.org/10.1075/target.19.2.06mar
Marco, J. (2019). The Translation of Food-Related Culture-Specific Items in the Valencian Corpus of Translated Literature (COVALT) Corpus: A Study of Techniques and Factors. Perspectives, 27(1), 20–41. https://doi.org/10.1080/0907676X.2018.1449228
Munday, J., Pinto, S. R., & Blakesley, J. (2022). Introducing Translation Studies: Theories and Applications. Routledge. https://doi.org/10.4324/9780429352461
Newmark, P. (1988). A Textbook of Translation. Prentice Hall.
Olk, H. M. (2013). Cultural References in Translation: A Framework for Quantitative Translation Analysis. Perspectives, 21(3), 344–357. https://doi.org/10.1080/0907676X.2011.646279
Pedersen, J. (2005). How is Culture Rendered in Subtitles? In H. Gerzymisch-Arbogast & S. Nauert (Ed.), Marie Curie Euroconferences MuTra: Challenges of Multidimensional Translation (pp. 2–6). MuTra.
Ranzato, I. (2013). The Translation of Cultural References in the Italian Dubbing of Television Series [Doctoral thesis]. Imperial College London.
Rei, R., Treviso, M., Guerreiro, N. M., Zerva, C., Farinha, A. C., Maroti, C., C. de Souza, J. G., Glushkova, T., Alves, D., Coheur, L., Lavie, A., & Martins, A. F. T. (2022). CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task. In P. Koehn, L. Barrault, O. Bojar, F. Bougares, R. Chatterjee, M. R. Costa-jussà, C. Federmann, M. Fishel, A. Fraser, M. Freitag, Y. Graham, R. Grundkiewicz, P. Guzman, B. Haddow, M. Huck, A. Jimeno Yepes, T. Kocmi, A. Martins, M. Morishita, C. Monz, M. Nagata, T. Nakazawa, M. Negri, A. Névéol, M. Neves, M. Popel, M. Turchi & M. Zampieri (Eds.), Proceedings of the Seventh Conference on Machine Translation (WMT) (pp. 634–645). Association for Computational Linguistics. https://aclanthology.org/2022.wmt-1.60/
Reiss, K., & Vermeer, H. J. (2014). Towards a General Theory of Translational Action: Skopos Theory Explained (C. Nord, Trans.). Routledge. https://doi.org/10.4324/9781315759715
Taivalkoski-Shilov, K. (2019). Ethical Issues Regarding Machine(-assisted) Translation of Literary Texts. Perspectives, 27(5), 689–703. https://doi.org/10.1080/0907676X.2018.1520907
Toral, A., & Way, A. (2015). Machine-assisted Translation of Literary Text: a Case Study. Translation Spaces, 4(2), 240–267. https://doi.org/10.1075/ts.4.2.04tor
Toral, A., & Way, A. (2018). What Level of Quality Can Neural Machine Translation Attain on Literary Text? In J. Moorkens, S. Castilho, F. Gaspari & S. Doherty (Eds.), Translation Quality Assessment: From Principles to Practice (pp. 263–287). Springer International Publishing. https://doi.org/10.1007/978-3-319-91241-7_12
Venuti, L. (2017). The Translator's Invisibility: A History of Translation. Routledge. https://doi.org/10.4324/9781315098746
Vinay, J.-P., & Darbelnet, J. (2000). A Methodology for Translation (J. C. Sager & M.-J. Hamel, Trans.). In L. Venuti (Ed.), The Translation Studies Reader (pp. 84–93). Routledge.
Wang, L., Tu, Z., Gu, Y., Liu, S., Yu, D., Ma, Q., Lyu, C., Zhou, L., Liu, C.-H., Ma, Y., Chen, W., Graham, Y., Webber, B., Koehn, P., Way, A., Yuan, Y., & Shi, S. (2023). Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs. In P. Koehn, B. Haddow, T. Kocmi & C. Monz (Eds.), Proceedings of the Eighth Conference on Machine Translation (pp. 55–67). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.wmt-1.3
Wang, L., Liu, S., Lyu, C., Jiao, W., Wang, X., Xu, J., Tu, Z., Gu, Y., Chen, W., & Wu, M. (2024). Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation. In B. Haddow, T. Kocmi, P. Koehn & C. Monz (Eds.), Proceedings of the Ninth Conference on Machine Translation (pp. 699–700). Association for Computational Linguistics. https://doi.org/10.18653/v1/2024.wmt-1.58
Way, A. (2012). David Bellos (ed): Is that a fish in your ear: translation and the meaning of everything. Machine Translation, 26(3), 255–269. https://doi.org/10.1007/s10590-012-9129-x
Williams, M. (2004). Translation Quality Assessment: An Argumentation-Centred Approach. University of Ottawa Press.
Wu, C-E. (2024). Jornada ao Oeste (I. Aldebrand, Trans.). [Kindle edition]. Amazon Kindle.
Downloads
Publicado
Como Citar
Edição
Seção
Licença
Copyright (c) 2025 Cadernos de Tradução
Este trabalho está licenciado sob uma licença Creative Commons Attribution 4.0 International License.
Autores têm autorização para assumir contratos adicionais separadamente, para distribuição não exclusiva da versão do trabalho publicada nesta revista (ex.: publicar em repositório institucional ou como capítulo de livro, com reconhecimento de autoria e publicação inicial nesta revista).
