Evaluation of Visual Metaphors of School Coexistence Using Generative Artificial Intelligence and Expert Judgment
DOI:
https://doi.org/10.17398/1695-288X.25.2.225Keywords:
Artificial Intelligence, Educational Assessment, Interrater Reliability, Scoring Rubrics, Visual Metaphor, School Culture, Test ReliabilityAbstract
This study compares agreement and reliability between human judges and GAI models when evaluating visual metaphors of school coexistence, understood as graphic representations that translate institutional experiences into symbolic scenes (a ranch with corrals, a market with stalls, and countries/faculties). The design was comparative, with repeated measures across three metaphors and the transcribed verbal explanations provided by the students who produced them in focus groups. Six human experts participated (18 ratings) and five LLMs (ChatGPT-5, ChatGPT-5 Pro, ChatGPT-o3, DeepSeek, Gemini 2.5), with five runs per model (75 ratings). The rubric included a global judgment (1-3), binary indicators, and 10 Likert items (1-5). Ordinal alpha, Cohen kappa, Fleiss kappa, Krippendorff alpha, and a p x r G-study with a D-study for human-GAI mixtures were estimated. The input was not identical across models: ChatGPT variants worked with image and description, whereas DeepSeek and Gemini evaluated the narrative description and transcript. Therefore, the results are interpreted as a baseline situated in the versions and conditions analyzed, not as a definitive conclusion about all current or future multimodal systems. Findings showed that GAI produced more words than human experts, although length was treated as a descriptive indicator rather than a measure of quality or depth. The Coexistence Index was higher for humans, especially in Metaphor 2 (Delta=+0.91). The D-study indicated that 1-2 human experts plus around 10 GAI runs reach G >= 0.95. ChatGPT-5 Pro was the model closest to human consensus. The study provides a reference point for future pedagogical analysis of student productions supported by GAI.
Downloads
References
AERA, APA, & NCME (2014). Standards for educational and psychological testing. American Educational Research Association.
Arif, T., Asthana, S., & Collins-Thompson, K. (2024). Generation and assessment of multiple-choice questions from video transcripts using large language models. Proceedings of the 11th ACM Conference on Learning at Scale (L@S ’24), 530–534. https://doi.org/10.1145/3657604.3664714
Artsi, Y., Sorin, V., Konen, E., Glicksberg, B. S., Nadkarni, G., & Klang, E. (2024). Large language models for generating medical examinations: Systematic review. BMC Medical Education, 24, 354. https://doi.org/10.1186/s12909-024-05239-y
Atasoy, A., & Moslemi Nezhad Arani, S. (2025). ChatGPT: A reliable assistant for the evaluation of students’ written texts? Education and Information Technologies. https://doi.org/10.1007/s10639-025-13553-1
Bouziane, K., & Bouziane, A. (2024). AI versus human effectiveness in essay evaluation. Discover Education, 3, 201. https://doi.org/10.1007/s44217-024-00320-6
Brennan, R. L. (2001). Generalizability theory. Springer. https://link.springer.com/book/10.1007/978-1-4757-3456-0
Bui, N. M., & Barrot, J. (2025). Using generative artificial intelligence as an automated essay scoring tool: A comparative study. Innovation in Language Learning and Teaching. https://doi.org/10.1080/17501229.2025.2521003
Cleveland, W. S. (1993). Visualizing data. Hobart Press.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256
Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches (5th ed.). SAGE. https://us.sagepub.com/en-us/nam/research-design/book246125
Escobar-Pérez, J., & Cuervo-Martínez, A. (2008). Validez de contenido y juicio de expertos: Una aproximación a su utilización. Avances en Medición, 6, 27–36. https://www.researchgate.net/publication/302438451_Validez_de_contenido_y_juicio_de_expertos_Una_aproximacion_a_su_utilizacion
Fierro-Evans, C., & Carbajal-Padilla, P. (2019). Convivencia escolar: una revisión del concepto. Psicoperspectivas, 18(1), 1–14. https://doi.org/10.5027/psicoperspectivas-vol18-issue1-
Fierro-Evans, M. C. (2013). Convivencia inclusiva y democrática: una perspectiva para gestionar la seguridad escolar. Sinéctica: Revista Electrónica de Educación, (40), 1–18. http://www.scielo.org.mx/scielo.php?script=sci_arttext&pid=S1665-109X2013000100005&nrm=iso
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
Gadermann, A. M., Guhn, M., & Zumbo, B. D. (2012). Estimating ordinal reliability for Likert-type and ordinal item response data. Practical Assessment, Research, and Evaluation, 17(1). https://doi.org/10.7275/n560-j767
Giddens, A. (1998). La constitución de la sociedad: bases para la teoría de la estructuración (2da reimpr.). Amorrortu Editores.
Gemini Team Google. (2025). Multi-task performance of LLMs: A Google perspective. Google Research Papers, 12(4), 76–89.
Hirmas, C. y Carranza, G. (2009). Matriz de indicadores sobre convivencia de-mocrática y cultura de paz en la escuela. En III Jornadas de Cooperación Ibe-roamericana sobre Educación para la Paz, la Convivencia Democrática y los Derechos Humanos (pp. 56-136). Salesianos Impresores.
Hunter, J. D. (2007). Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3), 90–95. https://doi.org/10.1109/MCSE.2007.55
Kiryakova, G. (2025). ChatGPT as a supportive tool for creating assessment resources. TEM Journal, 14(2), 1014–1023. https://doi.org/10.18421/TEM142-04
Kıyak, Y. S., & Emekli, E. (2024). ChatGPT prompts for generating multiple-choice questions in medical education and evidence on their validity: A literature review. Postgraduate Medical Journal, 100(1189), 858–865. https://doi.org/10.1093/postmj/qgae065
Kosmyna, N., et al. (2023). Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task (arXiv:2506.08872). https://arxiv.org/abs/2506.08872
Krippendorff, K. (2004). Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research, 30(3), 411–433. https://doi.org/10.1093/hcr/30.3.411
Laupichler, M. C., Rother, J. F., Grunwald Kadow, I. C., Ahmadi, S., & Raupach, T. (2024). Large language models in medical education: Comparing ChatGPT- to human-generated exam questions. Academic Medicine, 99(5), 508–512. https://doi.org/10.1097/ACM.0000000000005626
Law, A. K. K., So, J., Lui, C. T., Choi, Y. F., Cheung, K. H., Hung, K. K.-C., & Graham, C. A. (2025). AI versus human-generated multiple-choice questions for medical education: A cohort study in a high-stakes examination. BMC Medical Education, 25, 208. https://doi.org/10.1186/s12909-025-06796-6
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
Liao, Z., Liu, X., Qin, W., Li, Q., Wang, Q., Wan, P., Zhang, D., Zeng, L., & Feng, P. (2025). HumanAesExpert: Advancing a multi-modality foundation model for human image aesthetic assessment (arXiv:2503.23907). https://arxiv.org/abs/2503.23907
Martínez-Arias, R. (1995). Psicometría: Teoría de los tests psicológicos y educativos. Síntesis.
May, T. A., Fan, Y. K., Stone, G. E., Koskey, K. L. K., Sondergeld, C. J., Folger, T. D., Archer, J. N., Provinzano, K., & Johnson, C. C. (2025). An effectiveness study of generative AI tools used to develop multiple-choice test items. Education Sciences, 15(2), 144. https://doi.org/10.3390/educsci15020144
McKinney, W. (2010). Data structures for statistical computing in Python. In S. van der Walt & J. Millman (Eds.), Proceedings of the 9th Python in Science Conference (pp. 51–56). https://doi.org/10.25080/Majora-92bf1922-00a
Mistry, N. P., Saeed, H., Rafique, S., Le, T., Obaid, H., & Adams, S. J. (2024). Large language models as tools to generate radiology board-style multiple-choice questions. Academic Radiology, 31(9), 3872–3878. https://doi.org/10.1016/j.acra.2024.06.046
Muñiz, J., & Fonseca-Pedrero, E. (2019). Diez pasos para la construcción de un test. Psicothema, 31(1), 7–16. https://doi.org/10.7334/psicothema2018.291
Nasr, N. R., Tu, C.-H., Sujo-Montes, L., Yen, C.-J., et al. (2025, abril). Generative artificial intelligence impact on critical thinking in higher education: A comprehensive bibliometric analysis and systematic review. Preprint. ResearchGate. https://doi.org/10.13140/RG.2.2.18979.16168
OpenAI. (2025). Introducing GPT-5. https://openai.com/index/introducing-gpt-5
Art of Problem Solving. (2025). AMC historical results. https://wiki.artofproblemsolving.com/wiki/index.php?title=AMC_historical_results
Rodríguez-Figueroa, H. M. (2019). La convivencia escolar desde la perspectiva sociocultural [tesis doctoral]. Universidad Autónoma de Aguascalientes.
Rodríguez-Figueroa, H. M. (2021). Convivencia escolar: Revisión del concepto a partir de dos estudios de caso. Sinéctica, (57), e1272. https://doi.org/10.31391/S2007-7033(2021)0057-003
Scaria, N., Chenna, S. D., & Subramani, D. (2024). How good are modern LLMs in generating relevant and high-quality questions at different Bloom’s skill levels for Indian high school social science curriculum? Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024). https://aclanthology.org/2024.bea-1.1
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage.
Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101. https://doi.org/10.1093/ije/dyq191
Tufte, E. R. (2001). The visual display of quantitative information (2nd ed.). Graphics Press. https://www.edwardtufte.com/book/the-visual-display-of-quantitative-information
Usher, M. (2025). Generative AI vs. instructor vs. peer assessments: A comparison of grading and feedback in higher education. Assessment & Evaluation in Higher Education. https://doi.org/10.1080/02602938.2025.2487495
Wilkinson, L. (2005). The grammar of graphics (2nd ed.). Springer. https://link.springer.com/book/10.1007/0-387-28695-0
Weiss, E. (2015). Más allá de la socialización y de la sociabilidad: jóvenes y ba-chillerato en México. Educ. Pesqui., 41, número especial, pp. 1257-1272. https://doi.org/10.1590/S1517-9702201508144889
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Karla Karina Ruiz-Mendoza, Luis Horacio Pedroza-Zúñiga, Héctor Manuel Rodriguez-Figueroa
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish in this journal accept the following conditions:
Authors retain copyright over their works and grant the journal the right of first publication. Articles are published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which allows third parties to share, copy, distribute, publicly communicate, adapt, transform, and reuse the work in any medium or format, including for commercial purposes, provided that authorship is properly acknowledged, the original source is cited, a link to the license is included, and any changes made are indicated. Note: This license applies to articles published from Vol. 25, No. 2, 2026 onwards.
Authors may enter into separate and additional contractual arrangements for the non-exclusive distribution of the published version of the article —for example, its deposit in an institutional repository or its subsequent inclusion in a book—, provided that it is clearly stated that the work was first published in this journal.
Authors are permitted and encouraged to deposit and disseminate their work on the Internet, for example, in institutional repositories, institutional websites, or personal websites before, during, and after the review and publication process, as this may foster scholarly exchange, increase the visibility of the work, and enable broader and faster dissemination of the published research.
