Christiaan Hamaker Prize: The Effect of Artificial Intelligence Chatbot Use on Intrinsic Learning Motivation

Welcome to the winning propaedeutic thesis of the 2023/2024 academic year: A new era of artificial intelligence has ushered in many technological interventions that can be used in educational contexts. Among them are artificial intelligence chatbots (AICs), which have been shown to be frequently used among students. Thus, this paper examines the effect of AIC use on intrinsic learning motivation (ILM), an important learning experience factor. It was expected that AIC use would have a positive effect on ILM, and this hypothesis was supported by all four studies included. The positive effect was maintained even when careful manipulation eliminated confounds and in courses unrelated to students’ major. These findings have important practical implications, as it has been found that students’ ILM tends to decrease over the duration of the school year.

Introduction

Artificial intelligence (AI) has ushered in a new era of progress across various disciplines, including education (Adiguzel et al., 2023). The release and rapid adoption of AI chatbots (AICs), software programmes that understand human language and can provide a human-like response, has sparked reactions ranging from optimistic to fearful (Tseng & Warschauer, 2023; Ventoniemi, n.d.). The widespread use of AICs among learners highlights the need for thorough research – according to one study, every fourth student uses AICs frequently, and almost a half uses AICs at least occasionally (von Garrel & Mayer, 2023). Given this extensive use, it is crucial to understand the possible effects AICs may have on educational experience. Certainly, one of the most important factors in educational experience is intrinsic learning motivation (ILM), the spontaneous exploration and curiosity related to studying, since it has been shown to be related to achievement of learning outcomes, academic performance, and learning satisfaction (Chang & Chang, 2012; Education, n.d.; Maqbool et al., 2020; Zhu et al., 2022). According to Ryan and Deci’s (2000) self-determination theory, intrinsic motivation is the inherent tendency to seek challenges, extend one’s skills, and explore. The theory identifies three universal, innate psychological needs that enhance intrinsic motivation when satisfied. These three needs are: autonomy (the feeling of control over one’s own behaviour and goals), competence (gaining mastery of tasks), and relatedness (sense of connectedness and belonging) (Cherry, 2022). The self-determination theory can be applied to educational context to explain ILM (Education, n.d.), as has been shown by Wang et al. (2019), who explored the relationships between secondary school students’ class experiences and ILM. They found a positive relationship between each of the three physiological needs proposed by the self-determination theory and ILM. Furthermore, Annamalai et al. (2023) conducted a study on students’ motivation to learn English language with the help of AICs. They concluded that AIC use supports autonomy, competence, and relatedness. Hence, ILM is a type of intrinsic motivation connected to learning. According to selfdetermination theory, intrinsic motivation arises form satisfaction of three basic psychological needs: autonomy, competence, and relatedness. Annamalai et al. (2023) showed that AIC use supports autonomy, competence, and relatedness, while Wang et al. (2019) demonstrated that there is a positive relationship between students’ perceived autonomy, competence, and relatedness and their ILM. Therefore, it is plausible that AIC use could affect ILM. Since different research studies use different AICs, this literature review defines AIC as a software programme that includes a feature where students receive at least some information through conversation with an AIC. Thus, this paper aims to address the effect of AIC use on ILM. Based on the self-determination theory and prior research, it is expected that AIC use will have a positive effect on ILM, which signifies that ILM will be higher in AIC use conditions than traditional teaching methods conditions that do not include AICs. To explore the research question, it will firstly be shown if there is a positive correlation between AIC use and ILM. To promote causal inference, an experimental study will be introduced showing if AIC use can increase ILM. Since previous two studies did not manipulate AIC use clearly enough, thirdly, a study with clear AIC use manipulation that eliminates some confounds will be presented. Lastly, it will be demonstrated if AIC use can have a positive effect on ILM even when the topic studied does not fall within the student’s chosen major. 

Empirical Evidence

Gao et al. (2023) studied the relationship between AIC use and ILM. Their sample consisted of 376 Chinese international business students in Malaysia (aged between 21 and 25 years). Firstly, their AIC use was assessed with the ICAP Technology Scale (ICAP-TS; Antonietti et al., 2023), which is a 12-item scale. Responses on each item range from 0 to 4 and can be summed. Higher total scores indicate greater AIC use. Secondly, ILM was measured with the Self-Directed Learning Readiness Scale for Nursing Education (SDLRSNE; Fisher & King, 2010), which consists of four items across three subscales. Responses on each item range from 1 to 5 and can be summed. Higher total scores indicate higher ILM. Results showed that those who scored higher on ICAP-TS also tended to score higher on SDLRSNE. Thus, it was concluded there is a positive association between AIC use and ILM.

The study by Gao et al. (2023) suggests a potential link between AIC use and ILM, but as a correlational study, it cannot establish causality. Since AIC use was only measured, but not manipulated, the actual effect could be reverse – students with greater ILM might use AICs more frequently as a means of active learning to answer their academic inquiries, as was shown in Lai et al. (2023). To determine whether AIC use increases ILM, an experimental study is needed where AIC use is manipulated by incorporating a control condition that employs traditional learning methods.

Beketov et al. (2023) explored the effect of AIC use on ILM. Their sample consisted of 246 Russian national medicine students (mean age 21 years). Participants were randomly assigned to one of two conditions: the AIC use condition, which covered various topics from medical education curriculum via an online module that included various interactive materials, simulations of practical situations and an AIC for personalized feedback; or the traditional training condition which covered the materials through lectures and teaching materials. Both conditions’ trainings lasted for 8 weeks. ILM was assessed using the Test of Motivation for Educational Activity (TMEA; Beketov et al., 2023). TMEA is a questionnaire that consists of 20 items, from which each can be assessed on a 5-point scale. The ratings were then averaged and the higher the mean, the higher the ILM. The results showed that the AIC use condition had higher mean scores on TMEA than the traditional training condition. Hence, it was concluded there is a positive effect of AIC use on ILM.

Even though the conclusion of Beketov et al. (2023) does suggest a causal relationship between AIC use and ILM, there are confounding factors in the manipulation of AIC use in both studies discussed. Gao et al. (2023), in addition to conducting a correlational study, also displayed a prominent notice in their questionnaire, alerting students that the information they receive form AICs may not always be accurate. Such notice could lead students to underreport their AIC use, as they may avoid admitting to using an unreliable tool, potentially biasing the correlation found (Brennan et al., 2021). Additionally, in Beketov et al. (2023), the AIC was only a part of the online module used in the experimental condition, which also included supplemental resources, tests, and other materials.

Thus, AIC use was not the only difference between the experimental and control conditions. Therefore, it is possible that some other feature of the online module, rather than AIC use specifically, had an effect on ILM. Hence, to establish valid causal interpretations, it is essential to introduce studies where AIC use was carefully manipulated.

Lee et al. (2022) conducted a study in which AIC use was clearly manipulated. Their sample consisted of two classes of Taiwanese national healthcare freshmen students (38 participants). Each class underwent an after-class review of an infectious disease course by filling out a worksheet, but the manner in which reviews were performed differed. The classes were randomly assigned to one of two conditions: the AIC use condition, where students reviewed their knowledge using an app in which they could ask AIC questions about infectious diseases; while the traditional review condition engaged in an interactive review with a teacher. For both conditions, two review sessions were carried out in the span of 2 weeks. Pre-review and post-review ILM measures were taken using the motivation aspect of the Game Flow Experience and Motivation Questionnaire (GFEMQ; Wang & Chen, 2010), which includes six items and each of them can be rated on a 5-point scale. The ratings were then averaged and the higher the mean, the higher the ILM. The results showed that the AIC use condition had higher mean scores on the motivation aspect of GFEMO post-review than the traditional review condition. Thus, it was concluded there is a positive effect of AIC use on ILM.

Since Lee et al. (2022) implemented clearer AIC use manipulation, their study provides further indication that a causal link between AIC use and ILM might exist. However, all studies discussed thus far have investigated the effect of AIC use on ILM in connection to students’ major only. Since Ganasih et al. (2023) showed that motivation plays an important part in choosing a major, it can be assumed that students likely already had some baseline ILM for the topics studied, thus making it easier to increase ILM with AIC use. According to self-determination theory (Ryan & Deci, 2000), if a causal link between AIC use and ILM does indeed exist, it should also be evident when AIC interventions focus on topics unrelated to student’s major, as needs of autonomy, competence, and relatedness can be satisfied when learning any topic. Thus, the next study discussed was carried out on a sample of students taking a course outside their major.

Li et al. (2023) carried out a study on the effect of AIC use on ILM in a course unrelated to the students’ major. The sample consisted of two classes of Chinese national chemistry major students enrolled in a modern education course (81 participants, mean age 20 years). Their task was to complete a series of small projects concerning animation in Microsoft PowerPoint with the usage of an online learning module, which included animation videos and learning tasks. Each class was randomly assigned to one of two conditions: the AIC use condition, where the online learning module also included an AIC, which provided personal and instant guidance; or the traditional online module condition, where AIC was not included. The intervention lasted for 3 weeks. Pre-intervention and post-intervention measures of ILM were taken using the intrinsic motivation dimension of the Learning Motivation questionnaire (LM; Hwang & Chen, 2016), which contains three items and each is rated on a 5-point scale. The ratings are then summed to obtain a total; the higher the total score, the higher the ILM. The results showed that the AIC use condition had higher total scores on LM post-intervention than the traditional online module condition. Hence, it was concluded there is a positive effect of AIC use on ILM. 

Conclusions and Discussion

The studies discussed in this paper demonstrate a positive effect of AIC use on ILM, using various research designs (correlational, experimental, quasi-experimental) across samples of fairly diverse nationalities. Thus, it can be concluded that AIC use does have a positive effect on ILM, when AIC use conditions are compared to traditional teaching methods without AIC use conditions. Such findings are aligned with the self-determination theory (Ryan & Deci, 2000) and prior research, as Annamalai et al. (2023) showed that AIC use supports autonomy, competence, and relatedness, the three psychological needs posited by the self-determination theory that need to be satisfied to facilitate intrinsic motivation (Ryan & Deci, 2000); and Wang et al. (2019) demonstrated a positive relationship between these three psychological needs and ILM. Found effect can be explained by the unique ways in which the three psychological needs can be satisfied via AIC use: autonomy because AICs provide activity selection, flexible time and duration of access to information; competence because AICs provide personalised support in improving knowledge and skills; relatedness because AICs can mimic human-like interactions, giving learners a sense of connection and involvement (Annamalai et al., 2023). Thus, AICs supply a ubiquitous learning environment, that extends beyond classroom time and is most importantly personalized to learner’s own needs, which can hardly be provided by traditional teaching methods. However, it is a limitation that studies by Lee et al. (2022) and Li et al. (2023), which manipulated AIC use most clearly, were quasi-experimental. Some causal conclusions can still be drawn from them, as they have methodologically accounted for possible obscuring factors: they both inspected pre-intervention ILM in addition to post-intervention ILM and used some statistical methods, which can sometimes act as a substitute for randomization (Friedrich & Friede, 2020). However, it remains true that a stronger causal link could be established with a true experimental study with randomization of participants. Additionally, all studies discussed had very similar samples concerning age, as all participants were university students. Thus, future research should also focus on other age groups, especially younger students, since technology can be distracting and they get distracted more easily (Attia et al., 2017; Hoyer et al., 2021); and older adults who were shown to experience more anxiety when using technology (Robbins et al., 2023), and anxiety is negatively associated with ILM (Hancock, 2010). Thus, AIC use could be distracting and increase anxiety in some cases, and thus, may potentially have a negative effect on ILM in different age samples.

Moreover, there are possible alternative explanations for the observed effect. One such explanation is that the novelty of the intervention, rather than AIC use itself, could cause the effect. Since AICs are a relatively new technology that has been popularised only recently, students have had few formal experiences using AICs for educational purposes (Lee et al., 2022). Thus, their curiosity in exploration of the new study technique could account for the observed positive effect on ILM. A similar limitation was recognized in the study by Rodrigues et al., (2022) on the effect of gamification on learning, another topic where new technology was also used in educational purposes. When researchers carried out a longitudinal study, they found that initially there was some novelty effect, as gamification’s effect seemed to decrease after the first few weeks; however, after a longer period of time, familiarization effect has been established, as the impact increased again without further intervention. Thus, it is suggested that future research on the effect of AIC use on ILM should adopt a longitudinal approach and collect ILM measurements at regular intervals over an extended time period.

Additionally, it could be argued that none of the studies were conducted in entirely real-life scenarios where students receive grades based on examinations at the end of the course. Grades play into a long-standing debate concerning self-determination theory (Ryan & Deci, 2000). Its proponents argue that external motivators (such as certain rewards), can have a negative effect on intrinsic motivation (Deci, 1971), while some argue that the negative effect occurs only under certain conditions (Cameron & Pierce, 1996). Similarly, some consider grades as external motivators that reduce ILM by decreasing autonomy (Pulfrey, 2013). Thus, it could be argued that there is a possibility that the positive effect of AIC use on ILM would not be observed in a real-world classroom scenario with grading, as grades could act as external motivators that could decrease ILM. Moreover, grades may not only act as external motivators, but may also increase anxiety (Chamberlin, 2023), which is negatively associated with ILM (Hancock, 2010). Thus, to determine the full scope of the effect of AIC use on ILM, future research should be conducted in settings where knowledge obtained with AIC use is graded.

In conclusion, this paper shows that AIC use could have a positive effect on ILM. Such findings are of practical importance because ILM has been shown to decrease over the duration of the school year (Wild et al., 2024). Since ILM has been shown to be related to achievement of learning outcomes, academic performance and learning satisfaction (Chang & Chang, 2012; Maqbool et al., 2020; Zhu et al., 2022), it is crucial to explore interventions that could be used to enhance it. Since a significant proportion of students is already using AICs frequently (von Garrel & Mayer, 2023), incorporating AICs into course materials should be considered to leverage their positive effect on ILM.

Reference list

• Adiguzel, T., Kaya, M. H., & Cansu, F. K.. (2023). Revolutionizing education with AI: Exploring the transformative potential of ChatGPT. Contemporary Educational Technology, 70(3), 429. https://doi.org/10.30935/cedtech/13152
• Alfredo, R., Echeverria, V., Jin, Y., Yan, L., Swiecki, Z., Gašević, D., & MartinezMaldonado, R. (2024). Human-centred learning analytics and AI in education: A systematic literature review. Computers and Education: Artificial Intelligence, 6, 100215. https://doi.org/10.1016/j.caeai.2024.100215
• Annamalai, N., Elmagzoub Eltahir, M., Zyoud, S. H., Soundrarajan, D., Zakarneh, B., & Al Salhi, N. R. (2023). Exploring English language learning via chabot: A case study from a self determination theory perspective. Computers and Education: Artificial Intelligence, 5, 100148. https://doi.org/10.1016/j.caeai.2023.100148
• Antonietti, C., Schmitz, M. L., Consoli, T., Cattaneo, A., Gonon, P., & Petko, D. (2023). Development and validation of the ICAP Technology Scale to measure how teachers integrate technology into learning activities. Computers & Education, 192, 104648. https://doi.org/10.1016/j.compedu.2022.104648
• Attia, N. A., Baig, L., Marzouk, Y. I., & Khan, A. (2017). The potential effect of technology and distractions on undergraduate students’ concentration. Pakistan journal of medical sciences, 33(4), 860–865. https://doi.org/10.12669/pjms.334.12560
• Beketov, V., Lebedeva, M., & Taranova, M. (2024). The use of artificial intelligence in teaching medical students to increase motivation and reduce anxiety during academic practice. Current Psychology, 43, 14367–14377. https://doi.org/10.1007/s12144-023- 05471-7 12
• Brennan, J., Bannick, M., Kassebaum, N., Wilner, L., Thomson, A., Aravkin, A., & Zheng, P. (2021). Analysis and methods to mitigate effects of under-reporting in count data. arXiv. https://doi.org/10.48550/arXiv.2109.12247
• Cameron, J., & Pierce, W. D. (1996). The debate about rewards and intrinsic motivation: Protests and accusations do not alter the results. Review of Educational Research, 66(1), 39–51. https://doi.org/10.2307/1170725
• Chamberlin, K., Yasué, M., & Chiang, I. C. A. (2023). The impact of grades on student motivation. Active Learning in Higher Education, 24(2), 109–124. https://doi.org/10.1177/1469787418819728
• Chang, I. Y., & Chang, W. Y. (2012). The effect of student learning motivation on learning satisfaction. International Journal of Organizational Innovation, 4, 281–305. https://www.ijoi-online.org/attachments/article/29/Vol4Num3.pdf#page=281
• Cherry, K. (2022, November 8). What is self-determination theory?: How self-determination influences motivation. Verywell Mind. https://www.verywellmind.com/what-is-selfdetermination-theory-2795387
• Deci, E. L. (1971). The effects of externally mediated rewards on intrinsic motivation. Journal of Personality and Social Psychology, 18(1), 105–115. https://doi.org/10.1037/h0030644 Education (n.d.). Self-Determination Theory. https://selfdeterminationtheory.org/topics/application-education/
• Fisher, M. J., & King, J. (2010). The self-directed learning readiness scale for nursing education revisited: A confirmatory factor analysis. Nurse Education Today, 30(1), 44–48. https://doi.org/10.1016/j.nedt.2009.05.020 13
• Friedrich, S., & Friede, T. (2020). Causal inference methods for small non-randomized studies: Methods and an application in COVID-19. Contemporary Clinical Trials, 99, 106213. https://doi.org/10.1016/j.cct.2020.106213
• Gao, Z., Cheah, J. H., Lim, X. J., & Luo, X. (2024). Enhancing academic performance of business students using generative AI: An interactive-constructive-active-passive (ICAP) self-determination perspective. The International Journal of Management Education, 22(2), 100958. https://doi.org/10.1016/j.ijme.2024.100958
• Hancock, D. R. (2001). Effects of test anxiety and evaluative threat on students’ achievement and motivation. The Journal of Educational Research, 94(5), 284–290. https://doi.org/10.1080/00220670109598764
• Hoyer, R. S., Elshafei, H., Hemmerlin, J., Bouet, R., & Bidet-Caulet, A. (2021). Why are children so distractible? Development of attention and motor control from childhood to adulthood. Child Development, 92(4), 716–737. https://doi.org/10.1111/cdev.13561
• Hwang, G. J., & Chen, C. H. (2016). Influences of an inquiry-based ubiquitous gaming design on students’ learning achievements, motivation, behavioral patterns, and tendency towards critical thinking and problem solving. British Journal of Educational Technology, 48(4), 950–971. https://doi.org/10.1111/bjet.12464
• Lai, C. Y., Cheung, K. Y., & Chan, C. S. (2023). Exploring the role of intrinsic motivation in ChatGPT adoption to support active learning: An extension of the technology acceptance model. Computers and Education: Artificial Intelligence, 5, 100178. https://doi.org/10.1016/j.caeai.2023.100178
• Lee, Y. F., Hwang, G. J., & Chen, P. Y. (2022). Impacts of an AI-based chabot on college students’ after-class review, academic performance, self-efficacy, learning attitude, and motivation. Educational Technology Research and Development, 70, 1843–1865. https://doi.org/10.1007/s11423-022-10142-8 14
• Lestari Ganasih, R., Rama, R., & Aswin, R. (2023, July 8). The influence of motivation, perceptions and attitudes on student decisions in choosing a major (study of management students, Faculty of Economics and Business, University of Riau) [Paper presentation]. The Third International Conference on Innovations in Social Sciences Education and Engineering, Bandug, Indonesia.
• Li, H. (2023). Effects of a ChatGPT-based flipped learning guiding approach on learners’ courseware project performances and perceptions. Australasian Journal of Educational Technology, 39(5), 40–58. https://doi.org/10.14742/ajet.8923
• Maqbool, S., Malik Muhamed Ismail, A., & Maqbool, S. (2020). A correlational study between learning motivation and satisfaction in online courses. Humanities & Social Sciences Reviews, 8(1), 716–724. https://doi.org/10.18510/hssr.2020.8186
• Pulfrey, C., Darnon, C., & Butera, F. (2013). Autonomy and task performance: Explaining the impact of grades on intrinsic motivation. Journal of Educational Psychology, 105(1), 39–57. https://doi.org/10.1037/a0029376
• Robbins, R., DiClemente, R. J., Baig, N., Johnson, A., Chou, A., & Van den Bulck, J. (2023). Digital communications technology use and feelings of anxiety, depression, and loneliness among older adults during the COVID-19 pandemic. Journal of Applied Gerontology, 42(9), 1911–1920. https://doi.org/10.1177/07334648231169086
• Rodrigues, L., Pereira, F. D., Toda, A. M., Palomino, P. T., Pessoa, M., Carvalho, L. S. G., Fernandes, D., Oliveira, E. H. T., Cristea, A. I., & Isotani, S. (2022). Gamification suffers from the novelty effect but benefits from the familiarization effect: Findings from a longitudinal study. International Journal of Educational Technology in Higher Education, 19(13). https://doi.org/10.1186/s41239-021-00314-6 15
• Ryan, R. M., & Deci, E. L. (2000). Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American Psychologist, 55(1), 68–78. https://doi.org/10.1037/0003-066X.55.1.68
• Tseng, W., & Warschauer, M. (2023). AI-writing tools in education: if you can’t beat them, join them. Journal of China Computer-Assisted Language Learning, 3(2), 258–262. https://doi.org/10.1515/jccall-2023-0008
• Ventoniemi, J. (n.d.). What is an AI chatbot? Here’s everything you need to know. Giosg. https://www.giosg.com/blog/what-is-ai-chatbot von Garrel, J., & Mayer, J. (2023). Artificial Intelligence in studies—use of ChatGPT and AIbased tools among students in Germany. Humanities and Social Sciences Communications, 10, 799. https://doi.org/10.1057/s41599-023-02304-7
• Wang, C. K. J., Liu, W. C., Kee, Y. H., & Chian, L. K. (2019). Competence, autonomy, and relatedness in the classroom: Understanding students’ motivational processes using the self-determination theory. Heliyon, 5(7), e01983. https://doi.org/10.1016/j.heliyon.2019.e01983
• Wang, L., & Chen, M. (2010). The effects of game strategy and preference‐matching on flow experience and programming performance in game‐based learning. Innovations in Education and Teaching International, 47(1), 39–52. https://doi.org/10.1080/14703290903525838
• Wild, S., Rahn, S., & Meyer, T. (2024). Factors mitigating the decline of motivation during the first academic year: A latent change score analysis. Motivation and Emotion, 48, 36–50. https://doi.org/10.1007/s11031-023-10050-1
• Xia, Q., Chiu, T. K. F., Sing Chai, C., & Xie, K. (2023). The mediating effects of needs satisfaction on the relationships between prior knowledge and self-regulated learning 16 through artificial intelligence chatbot. British Journal of Educational Technology, 54(4), 967–986. https://doi.org/10.1111/bjet.13305
• Zhu, Y., Xu, S., Wang, W., Zhang, L., Liu, D., Liu, Z., & Xu, Y. (2022). The impact of online and offline learning motivation on learning performance: The mediating role of positive academic emotion. Education and Information Technologies, 27(7), 8921– 8938. https://doi.org/10.1007/s10639-022-10961-5