Artificial Intelligence Generative, Perceived Effort, and Critical Thinking: Evidence from an Experimental Study with ChatGPT in Higher Education
DOI:
https://doi.org/10.17398/1695-288X.25.2.201Keywords:
Artificial intelligence, ChatGPT, Illusion of Explanatory Depth, Critical thinking, Higher educationAbstract
The emergence of Artificial Intelligence (AI) has had a profound impact on diverse sectors, including education. While the integration of technology into classrooms has been a constant for decades, both educators and students face new challenges due to the advent of NLP systems. Educators deal with analyzing and evaluating the originality of an essay. With the application of AI, students can answer theoretical questions and solve complex exercises, but grapple with the challenge of collaborating and engaging with a tool that relies on natural language instructions but can also provide outright incorrect responses. Our study aims to assess the effectiveness of ChatGPT in solving a case study with quantitative questions. We seek to compare the performance of students who use the tool with those who do not, from formulating instructions to validating or assessing the results. The data collected through pre-, and post-test questionnaires reveal consistently better results in students who did not use ChatGPT. However, the expectation of result accuracy and confidence in the outcomes obtained are similar in both groups. We suggest this to be interpreted as an effect that we call "Blind to Fraud" describing the situation in which students appear unaware of the likelihood of inaccuracy when using an AI-based NLP tool, which shows overconfidence in the results obtained. This can significantly impact learning outcomes and should be counterbalanced through critical thinking skills in education.
Downloads
References
Aiyappa, R., An, J., Kwak, H., & Ahn, Y. Y. (2023). Can we trust the evaluation on ChatGPT? arXiv preprint arXiv:2303.12767. https://doi.org/10.48550/arXiv.2303.12767
Arif, T. B., Munaf, U., & Ul-Haque, I. (2023). The future of medical education and research: Is ChatGPT a blessing or blight in disguise? Medical Education Online, 28(1): 2181052. https://doi.org/10.1080/10872981.2023.2181052
Chromik, M., Eiband, M., Buchner, F., Krüger, A., & Butz, A. (2021). I Think I Get Your Point, AI! The Illusion of Explanatory Depth in Explainable AI. In 26th International Conference on Intelligent User Interfaces (IUI '21). Association for Computing Machinery, NY, USA, 307–317. https://doi.org/10.1145/3397481.3450644
Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International. https://doi.org/10.1080/14703297.2023.2195846
Fauzi, F., Tuhuteru, L., Sampe, F., Ausat, A. M. A., & Hatta, H. R. (2023). Analysing the role of ChatGPT in improving student productivity in higher education. Journal on Education, 5(4), 14886-14891. https://doi.org/10.31004/joe.v5i4.2563
Firat, M. (2023). How ChatGPT Can Transform Autodidactic Experiences and Open Education? https://doi.org/10.31219/osf.io/9ge8m
Fuchs, K. (2023). Exploring the opportunities and challenges of NLP models in higher education: is ChatGPT a blessing or a curse? In Frontiers in Education, 8, 1166682. Frontiers. https://www.frontiersin.org/articles/10.3389/feduc.2023.1166682/full
Gerlich, M. (2025). AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking. Societies, 15(1), 6. https://doi.org/10.3390/soc15010006
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Medical Education, 9(1), e45312. https://doi.org/10.3389/feduc.2023.1166682
Harasim, L. (2000). Shift happens: Online education as a new paradigm in learning. The Internet and Higher Education, 3(1-2), 41-61. https://doi.org/10.1016/S1096-7516(00)00032-4
Irfan, M., Murray, L., & Ali, S. (2023). Insights into Student Perceptions: Investigating Artificial Intelligence (AI) Tool Usability in Irish Higher Education at the University of Limerick. Global Digital & Print Media Review, VI(II), 48-63. http://dx.doi.org/10.31703/gdpmr.2023(VI-II).05
Irfan, M., & Murray, L. (2023). Empowering Critical skills of Teaching and Learning with Artificial Intelligence (AI) tools. University of Limerick. https://doi.org/10.34961/researchrepository-ul.22788506.v1
Joyner, D., (2023). ChatGPT in Education: Partner or Pariah? XRDS 29, 3 (Spring 2023), 48–51. https://doi.org/10.1145/3589651
Lanzing, M. (2019). “Strongly recommended” revisiting decisional privacy to judge hypernudging in self-tracking technologies. Philosophy & Technology, 32(3), 549-568. https://doi.org/10.1007/s13347-018-0316-4
Leontiev, N., (1978). Activity, consciousness and personality. Prentice-Hall.
Liesa-Orus, M., Latorre-Cosculluela, C., Vázquez-Toledo, S., & Sierra-Sánchez, V. (2020). The technological challenge facing higher education professors: Perceptions of ICT tools for developing 21st century skills. Sustainability, 12(13), 5339. http://dx.doi.org/10.3390/su12135339
Lo, C. K. (2023). What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature. Education Sciences, 13(4), 410. MDPI AG. http://dx.doi.org/10.3390/educsci13040410
Mhlanga, D. (2023). Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning. http://dx.doi.org/10.2139/SSRN.4354422
Moss, G, Jewitt, C, Levacic, R, Armstrong, V, Cardini, A. & Castle, F. (2007). The interactive whiteboards, pedagogy, and pupil performance evaluation: an evaluation of the Schools Whiteboard Expansion (SWE). Project London Challenge. https://www.researchgate.net/publication/267239102_The_Interactive_Whiteboards_Pedagogy_and_Pupil_Performance_Evaluation_An_Evaluation_of_the_Schools_Whiteboard_Expansion_SWE_Project_London_Challenge
Nikolaos, K., Tzafea, O., & Thanos, T. (2019). University students’skills in using the internet for educational purposes and the digital divide. European Journal of Open Education and E-Learning Studies, 4(1), 60-72. http://www.oapub.org/edu
Nisar, S., & Aslam, M. S. (2023). Is ChatGPT a Good Tool for T&CM Students in Studying Pharmacology? https://dx.doi.org/10.2139/ssrn.4324310
O'Connor, P. & McAndrew, S. (2021). Enhancing Learning Experiences with AI-Powered Chatbots in Higher Education. Journal of Educational Technology, 48(3), 167-185.
Odden, T. O. B., Marin, A., & Rudolph, J. L. (2021). How has Science Education changed over the last 100 years? An analysis using natural language processing. Sci. Educ. 105, 653–680. doi: 10.1002/sce.21623
Raja, R., & Nagasubramani, P. C. (2018). Impact of modern technology in education. Journal of Applied and Advanced Research, 3(1), 33-35. https://doi.org/10.21839/jaar.2018.v3iS1.165
Ravšelj, D., Keržič, D., Tomaževič, N., Umek, L., Brezovar, N., Iahad, N. A., ... & Aristovnik, A. (2025). Higher education students’ perceptions of ChatGPT: A global study of early reactions. PLoS One, 20(2), e0315011
Rawlins, P., & Kehrwald, B. (2010). Education technology in teacher education: Overcoming challenges, realizing opportunities. In Cases on digital technologies in higher education: Issues and challenges (50-63). IGI Global. https://doi.org/10.4018/978-1-61520-869-2.ch004
Rospigliosi, P. (2023) Artificial intelligence in teaching and learning: What questions should we ask of ChatGPT? Interactive Learning Environments, 31(1), 1–3. https://doi.org/10.1080/10494820.2023.2180191
Rozenblit, L. & Keil, F. (2002). The misunderstood limits of folk science: An illusion of explanatory depth. Cognitive science, 26(5), 521-562. https://doi.org/10.1207/s15516709cog2605_1
Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.9
Smith, J. (2022). Personalized Learning through AI in Higher Education. International Journal of Educational Technology, 39(2), 79-94.
Sok, S., & Heng, K. (2024). Opportunities, challenges, and strategies for using ChatGPT in higher education: A literature review. Journal of Digital Educational Technology, 4(1), ep2401. https://doi.org/10.30935/jdet/13152
Tajik, E. & Tajik, F. (2023, April). A Comprehensive Examination of the Potential Application of ChatGPT In Higher Education Institutions. TechRxiv. https://doi.org/10.36227/techrxiv.22589497
Tate, T., Doroudi, S., Ritchie, D. & Xu, Y. (2023). Educational Research and AI-Generated Writing: Confronting the Coming Tsunami. https://doi.org/10.35542/osf.io/4mec3
Thurzo, A., Strunga, M., Urban, R., Surovkova, J., & Afrashtehfar, K. I. (2023). Impact of artificial intelligence on dental education: a review and guide for curriculum update. Education Sciences, 13(2), 150. https://doi.org/10.3390/educsci13020150
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10(1), 15. https://doi.org/10.1186/s40561-023-00237-x
Verenikina, I. (2010). Vygotsky in twenty-first-century research. In EdMedia+ innovate learning (pp. 16-25). Association for the Advancement of Computing in Education (AACE). https://ro.uow.edu.au/cgi/viewcontent.cgi?article=2337&context=edupapers
Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes. Harvard University Press. https://doi.org/10.2307/j.ctvjf9vz4
Westin, T., Wiklund, M., Mozelius, P., & Norberg, L. (2015). Regression From Game-Oriented to Traditional School. Journal of Educational Technology Systems, 43(4), 349–370. https://doi.org/10.1177/004723951558
Yang, D., Oh, E. S., & Wang, Y. (2020). Hybrid physical education teaching and curriculum design based on a voice interactive artificial intelligence educational robot. Sustainability, 12(19), 8000. https://doi.org/10.3390/su12198000
Yeung, K., (2017) ‘Hypernudge’: Big Data as a mode of regulation by design, Information, Communication & Society, 20:1, 118-136, DOI: 10.1080/1369118X.2016.1186713
Zhu, C., Sun, M., Luo, J., Li, T., & Wang, M. (2023). How to harness the potential of ChatGPT in education? Knowledge Management & eLearning, 15(2), 133–152. https://doi.org/10.34105/j.kmel.2023.15.008
Downloads
Published
Versions
- 2026-07-21 (2)
- 2026-07-20 (1)
Issue
Section
License
Copyright (c) 2026 Emilio Velasco-Bartolome, Sofía Ruiz-Campo
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish in this journal accept the following conditions:
Authors retain copyright over their works and grant the journal the right of first publication. Articles are published under the Creative Commons Attribution 4.0 International License (CC BY 4.0), which allows third parties to share, copy, distribute, publicly communicate, adapt, transform, and reuse the work in any medium or format, including for commercial purposes, provided that authorship is properly acknowledged, the original source is cited, a link to the license is included, and any changes made are indicated. Note: This license applies to articles published from Vol. 25, No. 2, 2026 onwards.
Authors may enter into separate and additional contractual arrangements for the non-exclusive distribution of the published version of the article —for example, its deposit in an institutional repository or its subsequent inclusion in a book—, provided that it is clearly stated that the work was first published in this journal.
Authors are permitted and encouraged to deposit and disseminate their work on the Internet, for example, in institutional repositories, institutional websites, or personal websites before, during, and after the review and publication process, as this may foster scholarly exchange, increase the visibility of the work, and enable broader and faster dissemination of the published research.
