Assessment of the LLM-based Chatbots on Student Engagement and Learning Outcomes in Afghanistan
DOI:
https://doi.org/10.59490/dgo.2025.956Keywords:
Conversational AI, online deliberation, online learning, GPT-4, LLMs, AI, ethnographic studies, learning outcomes, women, AfghanistanAbstract
The integration of Generative AI (GenAI) technologies, such as ChatGPT, into online education is accelerating; however, their effectiveness in under-resourced contexts remains insufficiently studied. This paper investigates the impact of a Large Language Model (LLM)-based conversational agent on student engagement and learning outcomes in Afghanistan, where access to formal education—particularly for women—is severely restricted or banned. We conducted an experimental study involving 80 undergraduate computer science students (40 male, 40 female) in Afghanistan, randomly assigned to control and treatment groups. All participants attended identical 50-minute online lectures followed by 40-minute post-lecture discussions moderated by a human instructor, and completed a follow-up self-report questionnaire. The treatment group additionally engaged in AI-facilitated discussions using a GPT-4-based chatbot during post-lecture discussion. Analysis of discussion logs and post-intervention surveys revealed that the treatment group demonstrated significantly higher participation rates, with more posts and replies, during post-lecture discussion and reported greater confidence in their understanding of the course material. These findings highlight the potential of LLM-based chatbots to enhance interactive learning and foster educational inclusion, particularly for marginalized populations in low-resource environments.
Downloads
References
Hart, C. M., Friedmann, E., & Hill, M. (2018). Online course-taking and student outcomes in California community colleges. Education Finance and Policy, 13(1), 42–71.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023). Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9), 1–35. https://doi.org/10.1145/3505244
Salmon, A. K., & Barrera, M. X. (2021). Intentional questioning to promote thinking and learning. Thinking Skills and Creativity, 40. https://doi.org/10.1016/j.tsc.2021.100822
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. (2023). GPT-4 technical report. arXiv. [link]
Yan, C., Ren, R., Meng, M. H., Wan, L., Ooi, T. Y., & Bai, G. (2024). Exploring ChatGPT app ecosystem: Distribution, deployment, and security. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (pp. 1370–1382).
Anand, P. (2023, March 16). Khan Academy creates GPT-4 based helper Khanmigo marking formal entry of AI into education. Dataquest.
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2024). Generative AI can harm learning. Research Paper, The Wharton School. https://doi.org/10.2139/ssrn.4895486
Haqbeen, J., Sahab, S., Ito, T., & Rizzi, P. (2021). Using decision support system to enable crowd identify neighborhood issues and its solutions for policy makers: An online experiment at Kabul municipal level. Sustainability, 13(10), 5453. https://doi.org/10.3390/su13105453
Szopiński, T., & Bachnik, K. (2022). Student evaluation of online learning during the COVID-19 pandemic. Technological Forecasting and Social Change, 174, 121195.
Haqbeen, J., Sahab, S., & Ito, T. (2023). A digital initiative to address girls' education challenges in collaboration with NPO in post-2021 Afghanistan. IIAI Letters on Information and Interdisciplinary Research, 4(1).
Lyons, E. (2023). Household and student survey-based evidence on girls' secondary education in Afghanistan. Social Sciences & Humanities Open, 8(1), 100533.
Shayan, Z. (2015). Gender inequality in education in Afghanistan: Access and barriers. Open Journal of Philosophy, 5, 277–284.
Singh, A. K., & Meena, M. K. (2022). Challenges of virtual classroom during COVID-19 pandemic: An empirical analysis of Indian higher education. International Journal of Evaluation and Research in Education, 11(1), 207–212.
Haleem, A., Javaid, M., Qadri, M. A., & Suman, R. S. (2022). Understanding the role of digital technologies in education: A review. Sustainable Operations and Computers, 3, 275–285.
Sorensen, C. (2015). An examination of the relationship between online class size and instructor performance. Journal of Educators Online, 12(1), 140–159.
Russell, V., & Curtis, W. (2013). Comparing a large- and small-scale online language course: An examination of teacher and learner perceptions. The Internet and Higher Education, 16, 1–13.
Espasa, A., & Meneses, J. (2010). Analyzing feedback processes in an online teaching and learning environment: An exploratory study. Higher Education, 59(3), 277–292.
Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251–19257.
Chandra, S., & Palvia, S. (2021). Online education next wave: Peer to peer learning. Journal of Information Technology Case & Application Research, 23(3), 157–172.
Kasneci, E., Sessler, K., Winkel, K., Schmid, U., & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, 102274.
Ito, T., Hadfi, R., & Suzuki, S. (2022). An agent that facilitates crowd discussion. Group Decision and Negotiation, 31, 621–647.
Hadfi, R., Okuhara, S., Haqbeen, J., et al. (2023). Conversational agents enhance women's contribution in online debates. Scientific Reports, 13, 14534.
Sahab, S., Haqbeen, J., & Ito, T. (2024a). Conversational AI as a facilitator improves participant engagement and problem-solving in online discussion. IEICE Transactions, E107-D(4).
Hadfi, R., et al. (2021). Argumentative conversational agents for online discussions. Journal of Systems Science and Systems Engineering, 30, 450–464.
Tanprasert, T., Fels, S., Sinnamon, L., & Yoon, D. (2024). Debate chatbots to facilitate critical thinking on YouTube: Social identity and conversational style make a difference. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA.
Sahab, S., Haqbeen, J., Hadfi, R., et al. (2024b). E-contact facilitated by conversational agents reduces interethnic prejudice and anxiety in Afghanistan. Communications Psychology, 2, 22.
Hyper-Democracy. (2025). Hyper-Democracy Platform. [link]
Discourse. (2025). Discourse Platform. [link]
Ito, T., Dong, Y., Haqbeen, J., Matsuo, T., & Sahab, S. (2024). Multiple AI agents that support crowd discussion. In Proceedings of the 12th ACM Collective Intelligence Conference, Northeastern University, Boston, MA.
Sullivan, G. M., & Artino, A. R., Jr. (2013). Analyzing and interpreting data from likert-type scales. Journal of Graduate Medical Education, 5(4), 541–542. PubMed. https://doi.org/10.4300/JGME-5-4-18
Tunstall, L., et al. (2022). Efficient few-shot learning without prompts. arXiv preprint, arXiv:2209.11055.
Tofade, T., Elsner, J., & Haines, S. T. (2013). Best practice strategies for effective use of questions as a teaching tool. American Journal of Pharmaceutical Education, 77(7), 155.
Schmitt-Beck, R. (2015). Bandwagon effect. In The International Encyclopedia of Political Communication (pp. 1–5).
Gao, L. (2016). Motives for posting and liking on social media: Comparing differences between Chinese and European users (Master’s thesis). University of Twente. [link]
Schreiner, M., Fischer, T., & Riedl, R. (2021). Impact of content characteristics and emotion on behavioral engagement in social media: Literature review and research agenda. Electronic Commerce Research, 21, 329–345.
Pence, P. L. (2022). Student satisfaction and self-confidence in learning with virtual simulations. Teaching and Learning in Nursing, 17(1), 31–35.
Irasutoya. (2025). 2025 年. [link]
Downloads
Published
How to Cite
Conference Proceedings Volume
Section
License
Copyright (c) 2025 Jawad Ahmad Haqbeen, Sofia Sahab, Takayuki Ito
This work is licensed under a Creative Commons Attribution 4.0 International License.
