Evaluation of the Annif automatic indexing system in information science journal articles
DOI:
https://doi.org/10.5007/1518-2924.2026.e109644Keywords:
Annif, Machine learning, Automatic indexing, Digital repositories, Brazilian Thesaurus of Information ScienceAbstract
Objective: This study evaluates the performance of the Annif automatic indexing tool in assigning descriptors to scientific articles within the field of Information Science.
Methods: The research adopted an empirical and exploratory approach. It used a corpus composed of 60 articles written in Portuguese and previously indexed manually with the Brazilian Thesaurus of Information Science (TBCI). The corpus was divided into two subsets: 30 articles were used for training and 30 for testing. Annif was configured with the MLLM backend and compared with the terms assigned through intellectual indexing using the metrics of Precision, Recall, F1-score, and Consistency. A qualitative analysis of the generated terms was also conducted.
Results: The system demonstrated moderate performance, achieving a Precision of 49.52%, Recall of 50.1%, F1-score of 49.23%, and Consistency of 31.63%.
Conclusions: It is concluded that Annif is a viable tool for supporting semi-automatic indexing workflows, provided that its suggestions are validated by human indexers.
Downloads
References
AHMED, M. Automatic indexing for agriculture: designing a framework by deploying Agrovoc, Agris and Annif. Journal of Information and Knowledge, [s.l.], v. 60, n. 2, April 2023, p.85-95. DOI:10.17821/srels/2023/v60i2/170966. Disponível em: https://www.srels.org/index.php/sjim/article/view/170966. Acesso em: 01 nov. 2025.
AHMED, M.; MUKHOPADHYAY, M; MUKHOPADHYAY, P. Automated knowledge organization: AI/ML-based subject indexing system for libraries. Journal of Library & Information Technology, [s.l.], v. 43, n. 1, p. 45-54, January 2023. DOI:10.14429/djlit.43.01.18619. Disponível em: https://www.proquest.com/openview/706bd22421815bd8de889f92ef99d612/1?pq-origsite=gscholar&cbl=2028807. Acesso em: 19 out. 2025.
BRITO, J. C. B.; MARTINS, D. L. Framework genérico para geração automática de assuntos e indexação em repositório digital. Perspectiva em Ciência da Informação, Belo Horizonte, v. 28, Fluxo Contínuo, 2023a. DOI: 10.1590/1981-5344/46629. Disponível em: https://www.scielo.br/j/pci/a/cnCL89BRvwqnQyGwJ69rXGk/?lang=pt. Acesso em: 01 nov. 2025.
BRITO, J. C. B.; MARTINS, D. L. Geração automática de metadados: estudo de caso utilizando a técnica de indexação automática estatística com a ferramenta Annif. Revista caribeña de las ciências sociales, Miami, v. 12, n. 4, p. 1980-1996. 2023b. DOI: 10.55905/rcssv12n4-029. Disponível em: https://revistacaribena.com/ojs/index.php/rccs/article/view/2995. Acesso em: 01 nov. 2025.
DERMENTZI, M.; BRYANT, M.; ROVIGO, F.; GARCÍA-GONZÁLEZ, H. Multilingual automated subject indexing: a comparative study of LLMs vs alternative approaches in the context of the EHRI project. Hal, [s. l.], 2025. Disponível em: https://hal.science/hal-05142136v1. Acesso em: 19 out. 2025.
GIL-LEIVA, I.; ALONSO-ARROYO, A. Keywords given by authors of scientific articles in database descriptors. Journal of the American Society for Information Science and Technology, [s.l.], v. 58, n. 8, p. 1175-1187, 2007. DOI: 10.1002/asi.20595. Disponível em: https://onlinelibrary.wiley.com/doi/10.1002/asi.20595. Acesso em: 01 nov. 2025.
GOLUB, K.; SOERGEL, D.; BUCHANAN, G.; TUDHOPE, D.; HIOM, D.; LYKKE, M. A framework for evaluating automatic indexing or classification in the context of retrieval. J. Assn. Inf. Sci. Tec., [s.l.], n. 67, p. 3-16, 2016. DOI: 10.1002/asi.23600. Disponível em: https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.23600. Acesso em: 01 nov. 2025.
GOLUB, K.; SUOMINEN, O.; MOHAMMED, A. T.; AAGAARD, H.; OSTERMAN, O. Automated Dewey Decimal Classification of swedish library metadata using Annif software. Journal of Documentation, [s.l.], v. 80, n. 5, p. 1057-1079, 2024. DOI: 10.1108/JD-01-2022-0026. Disponível em: https://www.emerald.com/jd/article/80/5/1057/1236186/Automated-Dewey-Decimal-Classification-of-Swedish. Acesso em: 01 nov. 2025.
HAHN, J. Semi-automated methods for BIBFRAME work entity description. Cataloging & Classification Quarterly, [s.l.], v. 59, n. 8, p. 853-867, 2021. DOI: 10.1080/01639374.2021.2014011. Disponível em: https://www.tandfonline.com/doi/full/10.1080/01639374.2021.2014011. Acesso em: 01 nov. 2025.
INKINEN, J. (ed.). Backends. NatLibFi / Annif, 30 jun. 2025. Disponível em: https://github.com/NatLibFi/Annif/wiki/Backends. Acesso em: 13 out. 2025.
KERKETTA, S.; MUKHOPADHYAY, P. Automated class numbers prediction for books: an AI/ML based approach using Annif. In: INTERNATIONAL CONFERENCE ON MARCHING BEYOND THE LIBRARIES (ICMBL): Leadership, Creativity, and Innovation, 2024, Bhubaneswar, India. Proceedings [...]. [s.l.]: Atlantis Press, 2025. DOI: 10.2991/978-94-6463-712-0_12. Disponível em: https://www.atlantis-press.com/proceedings/icmbl-24/126011042. Acesso em: 01 nov. 2025.
KIVIMÄKI, V.; HAUTAKANGAS, S.; JÄRVELÄ, H.; MALTUSCH, P. Artificial intelligence enhanced course content labelling: conceptual model and testing with university teachers. EPiC Series in Computing, [s.l.], v. 105, p. 314-325, 2025. DOI: 10.29007/j4q6. Disponível em: https://easychair.org/publications/paper/vGfT. Acesso em: 01 nov. 2025.
LANCASTER, F. W. Indexação e resumo: teoria e prática. Brasília, DF: Briquet de lemos, 2004.
LAPPALAINEN, M.; HULKKONEN, J.; INKINEN, J.; KALLIO, A.; KOSKELA, M.; LEHTINEN, M.; SJÖBERG, M.; SUOMINEN, O.; YETUKURI, L. Automaattisen sisällönkuvailun ohjelmiston rakentaminen: case Annif. Signum, [s.l.], vol. 53, n. 4, p. 14–20, 2021. DOI: 10.25033/sig.113612. Disponível em: https://journal.fi/signum/article/view/113612. Acesso em: 01 nov. 2025.
LEHTONEN, T.; PIUKKULA, J. Automaattinen asiasanoitus radio, ja television, ohjelmatietokanta ritvassa. Informaatiotutkimus, [s.l.], v. 1, n. 39, p. 27-45, 2020. DOI: 10.23978/inf.88107. Disponível em: https://journal.fi/inf/article/view/88107. Acesso em: 01 nov. 2025.
MEDELYAN, O. Human-competitive automatic topic indexing. 2009. 214f. Tese (Doutorado) – The University of Waikato, Hamilton, New Zealand, 2009. Disponível em: https://www.researchgate.net/publication/40660732. Acesso em: 19 out. 2025.
MITRA, R.; MUKHOPADHYAY, P. Machine learning applications in digital humanities: designing a semi-automated subject indexing system for a low-resource domain. Journal of Library & Information Technology, [s.l.], v. 43, n. 4, p. 219-225, July 2023. DOI:10.14429/djlit.43.04.19227. Disponível em: https://www.researchgate.net/publication/380804005_Machine_Learning_Applications_in_Digital_Humanities_Designing_a_Semi-automated_Subject_Indexing_System_for_a_Low-resource_Domain. Acesso em: 19 out. 2025.
POLEY, C.; UHLMANN, S.; BUSSE, F.; JACOBS, J-H.; KÄHLER, M.; NAGELSCHMIDT, M.; SCHUMACHER, M. Automatic subject cataloguing at the German National Library. Liber Quarterly, [s.l.], v. 35, p. 1-29, 2025. DOI: 10.53377/lq.19422. Disponível em: https://liberquarterly.eu/article/view/19422. Acesso em: 01 nov. 2025.
SILVA, B. F. de M.; CORRÊA, R. F. Aplicação da folksonomia assistida na construção de corpus de referência em ciência da informação. Em Questão, Porto Alegre, v. 26, n. 2, p. 413-436, maio/ago. 2020. DOI: 10.19132/1808-5245262.413-436. Disponível em: https://seer.ufrgs.br/EmQuestao/article/view/90123. Acesso em: 19 out. 2025.
SILVA, B. F. de M.; CORRÊA, R. F. O processo de construção do corpus de referência em ciência da informação. Encontros Bibli: revista eletrônica de biblioteconomia e ciência da informação, Florianópolis, v. 24, n. 56, p. 01-27, set./dez., 2019. DOI: 10.5007/1518-2924.2019.e65166. Disponível em: https://periodicos.ufsc.br/index.php/eb/article/view/1518-2924.2019.e65166. Acesso em: 01 nov. 2025.
SILVA, S. R. de B.; CORRÊA, R. F.; GIL-LEIVA, I. Avaliação direta e conjunta de Sistemas de Indexação Automática por Atribuição. Informação & Sociedade: Est., João Pessoa, v. 30, n. 4, p. 1-27, 2020. DOI: 10.22478/ufpb.1809-4783.2020v30n4.57259. Disponível em: https://www.researchgate.net/publication/348702009. Acesso em: 19 out. 2025.
SUOMINEN, O. Annif: DIY automated subject indexing using multiple algorithms. LIBER Quarterly, [s.l.], v. 29, n. 1, 2019. DOI: 10.18352/lq.10285. Disponível em: https://liberquarterly.eu/article/view/10732. Acesso em: 01 nov. 2025.
SUOMINEN, O.; INKINEN, J.; LEHTINEN, M. Annif and Finto AI: developing and implementing automated subject indexing. JLIS.it, [s.l.], v. 13, n. 1, jan., 2022. DOI: 10.4403/jlis.it-12740. Disponível em: https://www.medra.org/servlet/MREngine?hdl=10.4403/jlis.it-12740. Acesso em: 01 nov. 2025.
SUOMINEN, O.; INKINEN, J.; LEHTINEN, M. Annif at SemEval-2025 task 5: traditional XMTC augmented by LLMs. In: International Workshop on Semantic Evaluation (SemEval-2025), 19., 31 July - 1 August 2025, Vienna, Austria. Proceedings [...]. Vienna, Austria: Association for Computational Linguistics, 2025. p. 2424-2431. DOI: 10.48550/arXiv.2504.19675. Disponível em: https://arxiv.org/abs/2504.19675. Acesso em: 01 nov. 2025.
SUOMINEN, O.; KOSKENNIEMI, I. Annif analyzer shootout: comparing text lemmatization methods for automated subject indexing. Journal Code4lib, [s.l.], n. 54, 2022. Disponível em: https://journal.code4lib.org/articles/16719. Acesso em: 19 out. 2025.
TOEPFER, M.; SEIFERT, C. Fusion architectures for automatic subject indexing under concept drift: analysis and empirical results on short texts. International Journal on Digital Libraries, [s.l.], v. 21, n. 2, p. 169-189, 2020. DOI: 10.1007/s00799-018-0240-3. Disponível em: https://link.springer.com/article/10.1007/s00799-018-0240-3. Acesso em: 01 nov. 2025.
Downloads
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Remi Correia Lapa, Renato Fernandes Correa
This work is licensed under a Creative Commons Attribution 4.0 International License.
The author must guarantee that:
- there is full consensus among all the coauthors in approving the final version of the document and its submission for publication.
- the work is original, and when the work and/or words from other people were used, they were properly acknowledged.
Plagiarism in all of its forms constitutes an unethical publication behavior and is unacceptable. Encontros Bibli has the right to use software or any other method of plagiarism detection.
All manuscripts submitted to Encontros Bibli go through plagiarism and self-plagiarism identification. Plagiarism identified during the evaluation process will result in the filing of the submission. In case plagiarism is identified in a manuscript published in the journal, the Editor-in-Chief will conduct a preliminary investigation and, if necessary, will make a retraction.
This journal, following the recommendations of the Open Source movement, provides full open access to its content. By doing this, the authors keep all of their rights allowing Encontros Bibli to publish and make its articles available to the whole community.
Encontros Bibli content is licensed under a Creative Commons Attribution 4.0 International License.
Any user has the right to:
- Share - copy, download, print or redistribute the material in any medium or format.
- Adapt - remix, transform and build upon the material for any purpose, even commercially.
According to the following terms:
- Attribution - You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions - You may not apply legal terms or technological measures that legally restrict others from doing anything that the license permits.
As obras deste periódico estão licenciadas com 