1 Introduction

Business leaders around the world continue to digitally transform their enterprises to optimise processes to meet changing market requirements. Alongside the proliferation of A.I. technologies in the workplace, institutions of higher learning are also experiencing an unprecedented push to integrate digital technologies such as A.I. into the education ecosystem. A.I. in education (AIED) has the propensity to enrich teaching and learning in higher education by personalizing students’ learning courses, automating assessment tasks or providing 24/7 access to learning resources (Karandish, 2021). According to estimates by the A.I. Market in the US Education Report, the AIED market will grow at a CAGR of 47.77% during the period 2018–2022 (Report, 2018).

A popular application of AIED is deploying chatbots as conversational agents to engage students by supporting their learning both within and beyond the classroom. For instance, chatbot prototypes can be integrated with e-learning platforms, using NLP to interact with students by interpreting their queries, and drawing up relevant information from the knowledge base module to assist students in their revision (Clarizia et al., 2018). Especially in the digital age, as educational chatbots can be seamlessly integrated into various social platforms and easily accessed on mobile devices, educators find it to be an effective tool for engagement and sustaining students’ interest (Topal et al., 2021).

Chatbots are also a viable solution to bridge the communication gap between students and instructors for distance learning (Tamayo et al., 2020) and effective when deployed appropriately by educators in areas such as teaching and learning foreign languages (Kim et al., 2019; Nghi et al., 2019). They are also suitable pedagogical tools to equip students with the necessary knowledge and skills to better understand emerging A.I. technologies and how they are applied beyond the classroom in areas such as customer support, customer communication or sales & marketing.

As educators, we are cognisant of the importance of the effects of motivation and student engagement on learning outcomes. Navarro et al. (2020) likens motivation to a resource that can be tapped onto as a ‘natural source of learning’ and to spur the achievement of learning objectives. Similarly, Monteiro et al. (2015) consider motivation to be the central factor in terms of performance and engagement of students.

In this paper, we report experiences made during the implementation of an experiential chatbot workshop integrated into an introductory undergraduate management course on ‘MGMT240—Doing Business with A.I.’ at Singapore Management University (SMU) that provides non-IT students with an opportunity to build a chatbot prototype using the ‘Dialogflow’ program. One basic premise is the assumption that such a novel, learning outcomes-related, hands-on chatbot workshop is motivational and engages learners provided its pedagogical approach is effective.

The research questions for this study are:

  • RQ1 To what extent do students find the experiential chatbot workshop motivational and engaging?

  • RQ2 What are possible relationships between the experiential chatbot workshop, students’ intrinsic motivation and engagement as well as desired (learning) outcomes such as students’ satisfaction and acquisition of chatbot-related competencies.

2 Theoretical framework

2.1 Dimensions of AIED

Experiential learning refers to a process that is designed for learners to acquire knowledge and skills by “doing”. Having critically reviewed the teaching experience from past terms, the instructors of the course ‘Doing Business with A.I.’ concluded that incorporating hands on elements (ie. activities that enable students to apply what’s being taught) into the course design qua a workshop component creates a more valuable learning experience for students.

In view of the nascent space of AIED and technology mediated learning (TML) at course level, the design of the experiential chatbot workshop was influenced by learning theories such as Kolb’s Experiential Learning Model and Gagne’s Nine Events of Learning (Menkhoff & Teo, 2021). One important intent was to create a novel and interesting learning experience for students to enhance student engagement by deploying an attractive, activity-based instructional method.

A key learning objective of MGMT240 is to enable students to understand and explain the workings and applications of A.I. in various business functions:

“Explain how private and public sector organizations engaged in customer service management, finance, marketing, supply chain management and manufacturing use machine learning, deep learning, neural networks, image analysis etc. to potentially become more competitive and ‘effective’ in real- time”.

The intended learning outcomes of the experiential chatbot workshop can be categorized into three main pillars of competencies as illustrated in Fig. 1.

Fig. 1
figure 1

Intended Learning Outcomes of Experiential Chatbot Workshop

Full size image

One key educational tool to support the course objectives is the deployment of a 3 h long Chatbot workshop aimed at enabling students to demonstrate a basic level of understanding of Natural Language Processing (NLP), the AI technology that facilitates interaction between natural human language and computers, as exemplified by conversational agents. Aligned with the popularization of chatbot usage beyond the classroom (Deloitte Digital, 2018; Han, 2021), skills related to creating chatbots, or even understanding how chatbots are developed, are deemed as competencies valued in the workplace (Brynjolfsson & McAfee, 2017; Davenport & Ronanki, 2018; Zawacki-Richter et al., 2019). Hence, the experiential workshop provides students with an opportunity to acquire skills to build a chatbot prototype using the ‘Dialogflow’ (a Google-owned developer of human–computer interaction technologies) program during the hands-on activity (see Fig. 2). Experience with using Dialogflow forms the basis of NLP-related knowledge and familiarity, making it easier for students to pick up or utilize other chatbot platforms or programs in the future. The workshop and the experiential learning activity were designed to impart students with other relevant skills such as conversation design and user-centric design, important competencies that are transferrable to other situational contexts beyond the course.

Fig. 2
figure 2

Lesson Plan for Experiential Chatbot Workshop

Full size image

Part I: Introduction (Warm-Up)

Duration: 30–45 min

• Introducing chatbots

• Types of chatbots

• Introduction to Dialogflow, NLP

• Dialogflow demonstration

• (using freshman orientation as case study)

Content covered:

• Types of chatbots

• How to select chatbots Technology involved (NLP) Dialogflow functions

• Conversation design

• User requirements, user-centric design, needs assessment

Teaching Objectives:

• Overview on chatbot options and technology powering its capabilities

• Demonstrate real-life application of chatbots using case studies

Part II: Class Activities

Duration: 45 min

Activity 1 (25 min)

• Create conversational agent on Dialogflow

• Test chatbot prototype

• Volunteers to present

• In-class discussion

Activity 2 (20 min)

• Discussion: Implementing chatbot in the workplace

Content covered:

• Design of conversation flow on chatbot

• Step-by-step process of creating chatbot on Dialogflow (coding not required for workshop)

• User-centric design

Teaching Objectives:

••• “Hands on” segment on Dialogflow to practice conversation design

• Create a complete set of (‘Intents’, ‘Entities’, ‘Response, ‘Fallback’)

• Students to test their chatbot agent

• Critical thinking:

• application of chatbots in different

• contexts (beyond example given in class)

2.2 Motivation

Motivation has been identified as a crucial factor central to student engagement and performance (Monteiro et al., 2015). Hence it is studied as a key variable in this research study. We consider motivation at the process stage of Winkler & Söllner’s input-process-output model (2018) and argue that one’s motivation influences the learning experience during a practical workshop designed to acquire a particular A.I.-related skill such as NLP.

A well-researched theoretical framework for studying motivation is the Self-Determination Theory (SDT) by Deci and Ryan (2008). Intrinsic motivation refers to doing things “for their own sake” or acting as the task is perceived to be inherently interesting or pleasant to the individual. Intrinsic motivation is associated with positive learning outcomes in formal education such as school performance and achievements (Augustyniak et al., 2016). SDT posits that humans have proactive tendencies that are manifested in “learning, mastery and connection with others” (Ryan & Deci, 2020). In an educational setting, basic psychological needs for autonomy, competence and relatedness need to be met for effective learning to occur.

In our chatbot workshop, we tried to create such as a needs-supportive setting to create and harness this motivational resource, e.g. by enabling students to feel confident that they can achieve the workshop goals and their own goals based on meaningful assignments, step by step explanations, learning experiences, attention check questions and constant (positive) feedback as exemplified in Table 1.

Table 1 Motivation and Engagement Approach of Our Chatbot Workshop (Examples)
Full size table

As part of our data collection efforts, we used The Intrinsic Motivation Inventory (IMI) as it is derived from the Self-Determination Theory. As a valid and reliable measurement instrument, IMI has been widely used to measure intrinsic motivation (Augustyniak et al., 2016; Heindl, 2020; Leng et al., 2010; Monteiro et al., 2015; Navarro et al., 2020). The IMI is a suitable scale which we incorporated into our survey instrument as its items consists of simple, short sentences which are easily comprehensible by undergraduate students. Besides a few items which were modified to do justice to the context of the experiential chatbot workshop, we used five of the most relevant sub-scales of intrinsic motivation, namely interest/enjoyment, perceived competence, effort/importance, pressure/tension and value/usefulness. The sub-scales of perceived choice and relatedness were excluded (Ryan & Deci, 2020; Navarro et al., 2020) because the experiential chatbot was a compulsory component of the course syllabus which all students had to attend (i.e. they had no choice). Similarly, relatedness was perceived to have a minimal role as the students mostly interacted with the workshop instructor over a short duration via a Zoom meeting.

2.3 Engagement

Student engagement through worthwhile tasks (and interaction with others) is closely linked to learning outcomes, ie. it is an “integral component” for learning effectiveness according to Mandernach (2015). Higher levels of engagement often translate into better learning outcomes, as engaged students who devote physical and psychological energy to their assignments are considered to be ‘good’ learners (Handelsman et al., 2005). Engagement research across different educational levels has shown that a learner’s level of engagement can vary due to intrinsic factors, or “learner variables” such as intrinsic motivation, and extrinsic factors that influence the learning process such as the involvement and quality of instructors, the use of relevant concepts, challenging and competency-enhancing assignments, support, positive feedback etc. (Handelsman et al., 2005; Jung & Lee, 2018).

During the experiential chatbot workshop, we tried to foster a safe and open learning environment for collaborative learning. Learners could ask questions, interact with the instructor or peers to respond to any topic-related questions or comments at any point during the workshop. Similarly, students were encouraged to volunteer and present their work to the class to share constructive feedback in the spirit of mutual, engaged learning. The workshop also included attention checkers for students to indicate their stage of progress during the guided demonstration, and they could make use of both the audio/video and text (chat box) function to share their input or reach out to the instructor for help.

To effectively examine and measure student engagement (a multidimensional construct), it is important to choose an instrument that is suitable for the respective “unique” (Mandernach, 2015) research context. We used the Student Course Engagement Questionnaire (SCEQ) as the scale of reference because the 23-item measure is robust and easy to administer. The items capture four important dimensions of engagement, namely (i) skills engagement, (ii) participation/interaction, (iii) emotional engagement, and (iv) performance engagement (Handelsman et al., 2005; Nasir et al., 2020). Some items from the SCEQ were modified slightly to fit the context of the Chatbot workshop. The SCEQ provides instructors with deeper insights into student engagement and how it affects learning beyond the information that can be observed from student’s in-class responses during the workshop and making hindsight assessments and inferences based on their grades (Handelsman et al., 2005).

Figure 3 illustrates our conceptual model which we intend to further corroborate during a future quantitative study with a bigger sample size. Its development is based on the learning theories and TML models introduced earlier aimed at examining the possible correlations between the various model variables depicting possible effects of the hands-on chatbot workshop on students’ engagement and motivation as drivers of acquiring A.I.-related competencies such as understanding and applying NLP.

Fig. 3
figure 3

Model Variables: Towards Understanding the Impact of the Experiential Chatbot Workshop on Student Engagement, Intrinsic Motivation and Learning Outcomes

Full size image

2.4 Hypotheses

The following hypotheses were derived from the conceptual model as illustrated in Fig. 3:

  • H1: Students with a high level of intrinsic motivation will be more engaged than students with a low level of intrinsic motivation.

  • H2: Students with a low level of pre-workshop (functionality-related) proficiency will be more motivated than students with a high level of pre-workshop (functionality-related) proficiency.

  • H3: Students with a high level of engagement as result of the chatbot workshop experience will have a higher level of chatbot-related competencies than students who have a low level of engagement.

  • H4: Students with a high level of engagement will be more satisfied with the workshop than students who have a low level of engagement.

  • H5: Students with a high level of intrinsic motivation will be more satisfied with the Chatbot workshop than students who have a low level of intrinsic motivation.

  • H6: Students with a high level of intrinsic motivation will report a higher level of chatbot-related competencies than students who have a low level of intrinsic motivation.

  • H7: Students with a high level of pre-workshop (functionality-related) proficiency will have a higher level of post-workshop competencies than students with a low level of pre-workshop (functionality-related) proficiency.

  • H8: Students with a low level of pre-workshop (functionality-related) proficiency will be more engaged than students with a high level of pre-workshop (functionality-related) proficiency.

  • H9: Students with a low level of pre-workshop (functionality-related) proficiency will be more satisfied with the chatbot workshop than students with a high level of pre-workshop (functionality-related) proficiency.

  • H10: Students with a high level of alignment between their expectations and the actual workshop experience will be more satisfied with the Chatbot-related learning approach than students with a low level of alignment.

3 Method

3.1 Sample

An online survey was created and released at course level to all participants of the experiential chatbot workshop. In educational research, convenience sampling of an entire class is a common sampling type. Of 44 students enrolled in the business elective ‘MGMT240 Doing Business with A.I.’, a total of 43 students responded to the post-workshop survey with a response rate of 97.7%. The response of one student was disregarded because it was incomplete. All respondents in this study were from the Lee Kong Chian School of Business (LKCSB). 53.5% of the survey participants were female (n = 23), and 46.5% of the respondents were male (n = 20). Third and fourth year university students made up more than 50% (n = 26) of the sample vis-à-vis 17 first and second year students (Table 2).

Table 2 Sample Distribution of Student Demographics
Full size table

While the small sample size limits the generalizability of the study (Tipton et al., 2017), obtaining more data from other (future) classes was not feasible due to time constraints as the course is only offered once a year and the unavailability of the chatbot trainer. Looking at the generalizability challenge from a pragmatic point of view, we would argue that our results are useful for other instructors interested in experimenting with NLP-powered bots in class and that our findings are also applicable to different non-ICT disciplines such as marketing. Our survey helps to increase the contextual understanding of engaging undergraduates with ‘smart’ conversational agents. Therefore, one could argue that our “small” sample research study has ‘good’ generalizability as suggested by Etz and Arroyo (2015).

3.2 Measures

Individual learner-related measures include proficiency and expectations.

Proficiency: The item scale was adapted from Dreyfus’ Model of Skill Acquisition (2004) to measure proficiency towards the functionality of chatbots by comparing the change in respondent’s self-assessed level of proficiency before and after attending the experiential chatbot workshop. Response options ranged from (0) “No Knowledge” to (5) “Expert”.

Alignment with learner’s expectation: To assess the alignment of learner’s expectations with the actual workshop-related learning objectives we used a 5-item measure. Sample items include ‘The workshop helped me to recognize that soft skills such as user-centric conversation flow design related to AI-powered chatbots are important competencies for the workplace of the future’, and ‘The chatbot workshop adequately fulfilled my learning expectations regarding the use and adoption of chatbots as smart solutions in business and beyond.’ Response options ranged from (1) “Strongly disagree” to (5) “Strongly Agree” for each of the items. The alpha reliability in this study is 0.844.

Process-related variables include intrinsic motivation and engagement.

Intrinsic Motivation: The 30-item measure was adapted from the Intrinsic Motivation Inventory Questionnaire (Ryan & Deci, 2020; Leng et al., 2010; Monteiro et al., 2015; Augustyniak et al., 2016; Heindl, 2020; Navarro et al., 2020). It features five sub-scales: interest/enjoyment, perceived competence, effort/importance, pressure/tension and value/usefulness. Sample items include ‘I believe I am pretty good at the hands-on exercise of building a basic chatbot on Dialogflow’ (for perceived competence), and ‘I think that participating in this workshop is useful for gaining critical competencies that will be important for future work opportunities’ (for value/usefulness). Response options ranged from (1) “Not true at all” to (5) “Very true” for each of the items. The alpha reliability in this study is 0.861.

Engagement: The 23-item measure was adapted from the Student Course Engagement Questionnaire (Handelsman et al., 2005) to assess behaviors and attitudes indicative of engagement. Sample items include ‘Asking questions when I do not understand the instructor’ and ‘Participating actively in small group setting during activities’. Response options ranged from (1) “Not at all characteristic of me” to (5) “Very characteristic of me” for each of the items. The alpha reliability in this study is 0.952.

Outcome-related measures include learner’s chatbot-related competencies (post-workshop) and workshop satisfaction.

Chatbot-related competencies: To assess to what extent the chatbot workshop helped students to acquire related key competencies (outcome level), we constructed a 20-item measure. Sample items include ‘Explain why an effective chatbot requires NLP and other Deep Learning techniques to understand the underlying intent of human language’ and ‘Draw a simple representation of a conversational chatbot workflow’. Response options ranged from (0) “No Knowledge” to (5) “Expert” for each item. The alpha reliability in this study is 0.959.

Workshop satisfaction: The item measure is included as a general indicator of respondent’s level of satisfaction toward the experiential chatbot workshop in general. Response options ranged from (1) “Strongly disagree” to (5) “Strongly Agree”.

4 Data analysis

Correlation analysis using Pearson correlation coefficient was applied to determine the degree to which the model variables are associated with each other. For each model variable, we calculated the average of all question responses on a 5-point scale. For example, there are 30 survey questions for the "Intrinsic Motivation" model variable where all questions are on a 5-point scale. For each survey participant, we calculated the average of the 30 5-point scale values, and the resulting average is the "Intrinsic Motivation" model variable value used for correlation analysis. Additionally, since the questions were categorized by the different groups of traits, the Cronbach’s alpha provides additional insights into the internal consistency of the results for these specific groups. It would help consider the scale of reliability of the results found over a set of items rather than just being unidimensional. Further explanations are provided in footnotes 1–4.

5 Results

According to the survey results (see Fig. 4), 90.7% (n = 39) of all respondents were pleased with the experiential learning chatbot workshop of which 60.5% (n = 26) were satisfied while 30.2% (n = 13) reported high satisfaction scores. None of the students felt dissatisfied with the experiential learning workshop. As shown in Table 3, the mean workshop satisfaction score is 4.2 (mode and median are both 4.0) suggesting that most of the workshop participants exhibited positive sentiments towards the workshop.

Fig. 4
figure 4

Students’ Satisfaction, Average Engagement, Average Motivation Scores, Average Chatbot Competencies, and Average Achievement of Learning Outcomes

Full size image
Table 3 Distribution of Satisfaction Scores
Full size table

As illustrated in Fig. 4, 81.4Footnote 1% (n = 35) of the chatbot workshop participants surveyed reported to be engaged of which 18.6% (n = 8) were highly engaged. As shown in Table 4, the mean of the average engagement score is 3.7 (mode is 4 and median is 3.8), indicating that the experiential learning approach of the chatbot workshop might be an effective method for engaging students.

Table 4 Distribution of Engagement Scores
Full size table

Figure 4 shows that altogether 83.8% of the surveyed students (n = 38) who attended the experiential learning process reported to be motivatedFootnote 2 (including 2 highly motivated students). As shown in Table 5, The mean of the average motivation score is 3.5 (mode is 3.7 and median is 3.5) suggesting that students’ average motivation is moderately high. One conclusion which can be drawn from the data is that the chosen chatbot teaching and learning approach is indeed motivational. The core aspect of the experiential learning approach is to “learn by doing”, and it is reasonable to deduce that students feel intrinsically motivated when they are empowered and enabled to put into practice the skills and techniques taught during the activity segment of the workshop.

Table 5 Distribution of Motivation Scores
Full size table

Most students reported moderate to high levels of competenciesFootnote 3 after the participation in the experiential learning workshop. 48.8% (n = 21) of the participants categorized their newly acquired chatbot-related competencies as ‘professional working proficiency’ while 32.5% (n = 14) rated it as ‘full professional proficiency’. Two (4.7%) students assessed themselves and their chatbot-related competencies (post-workshop) as / at ‘expert’ level (Table 6).

Table 6 Distribution of Average Competencies Scores
Full size table

The mean chatbot-related competency score is 2.9 due to the spread in the students’ reported proficiency levels. Due to the bi-modal distribution in our sample (Fig. 5), the mode scores are 2.7 and 3.0 respectively. The median proficiency score is 3.0 (moderate). The spread with regards to the proficiency levels suggests that students attending the same workshop might have different learning needs and preferences. As we observed during the workshop, some students require more time to replicate the steps demonstrated by instructors, while other students are following at the same pace or are ahead of their peers. Another factor that might affect the learning process and outcome (proficiency level) include the student’s individual cognitive load capacity. Novices unfamiliar with the bot subject matter might require more time to understand and keep pace with the workshop flow.

Fig. 5
figure 5

Distribution of Average Competencies Scores

Full size image

As shown in Fig. 4, almost all students surveyed (97.7%, n = 42) felt that the experiential chatbot workshop had met their learning outcomes.Footnote 4 As shown in Table 7, the mean of the average scores for learning outcome items is 4.2 (both mode and median scores are 4.0). This positive result suggests that most students who participated in the experiential chatbot workshop found the hands-on session useful and effective, ie. it helped them to achieve the intended learning outcomes (Table 8).

Table 7 Distribution of Average Scores for Achieving Learning Outcomes
Full size table
Table 8 Items to Measure Achievement of Learning Outcomes
Full size table

6 Hypotheses and interpretations

6.1 Intrinsic motivation and engagement

  • H1: Students with a high level of intrinsic motivation will be more engaged than students who have a low level of intrinsic motivation

The r-score for intrinsic motivation and engagement is 0.471. The findings suggest that at chatbot workshop level, there is moderate support for the hypothesis that students with a higher score for motivation tend to be more engaged during the workshop compared to students with low motivation. In general, H1 is supported by the correlational analysis, ie. students who score lower in motivation are likely to experience less engagement during the experiential learning workshop. This result is aligned with the literature on student motivation and engagement. From an educational perspective, the challenge remains to keep students motivated and engaged throughout the workshop (and the course of learning). This calls for a continuous review of the course curriculum, a flexible workshop design and student-centric teaching method to optimise students’ motivational levels and their actual engagement.

6.2 Pre-Chatbot workshop proficiency and motivation

  • H2: Students with a low level of pre-workshop (functionality-related) proficiency will be more motivated than students with a high level of pre-workshop (functionality-related) proficiency

As the r-score for pre-workshop (functionality-related) proficiency and high motivation is 0.005, H2 is not supported by the findings. In other words, there is seemingly no correlation between the student’s pre-workshop (functionality-related) proficiency and his/her intrinsic motivation related to the workshop. While this finding is puzzling given the hype about the need for ICT-related upskilling, the data suggest that it is important for educators to know more specifics about students’ prior knowledge about the subject matter in order to adjust the teaching and learning approach.

6.3 Student engagement vis-à-vis chatbot-related competencies and satisfaction (learning outcome level)

  • H3: Students with a high level of engagement will have a higher level of chatbot-related competencies than students who have a low level of engagement

The r-score for student engagement and acquisition of chatbot-related competencies is 0.427 which provides moderate support for H3. The strength of the correlation between engagement and chat-related competencies is moderate. This suggests that there is a relationship between engaged students and the level of skills which they had acquired from the workshop. To further interpret this finding, it is important to note that the workshop design exposes participants to the topic of chatbots at an introductory level only. Most participating students are from a non-computing background. Thus, given the scope and limited time frame of the chatbot workshop, it is not possible that students become fully competent on the topic at hand. As such, the above value suggests that students did in fact acquire a certain level of skills but certainly not mastery skills in line with the pedagogical objectives. Whether the workshop ought to be increased in length and depth needs to be discussed.

6.4 Student engagement and workshop satisfaction

  • H4: Students with a high level of engagement will be more satisfied with the workshop than students who have a low level of engagement

The r-score for engagement and workshop satisfaction at the learning outcome level is 0.298, i.e. there is no strong support for H4. One reason could be the fact that the chatbot workshop was conducted fully online due to the Covid 19 situation. Post-workshop student interviews revealed that those students who reported low satisfaction desired a face-to-face or physical mode of learning. Students may have experienced Zoom fatigue due to hours of exposure to visual cues as nearly all lessons took place online due to the pandemic. Furthermore, the “hands-on” nature of the workshop usually requires the teaching staff to offer help with troubleshooting, which could have been done more efficiently and effectively within a face-to-face learning context. Lastly, some students reported that they wished to have a fully working prototype at the end of the session. However, this was not part of the chatbot workshop design. Thus, it is important that instructors and students’ clearly spell out their mutual expectations (eg. in terms of desired learning outcomes) prior to the workshop.

6.5 Intrinsic motivation, Chatbot workshop-related satisfaction and competencies

  • H5: Students with a high level of intrinsic motivation will be more satisfied with the Chatbot workshop than students who have a low level of intrinsic motivation

  • H6: Students with a high level of intrinsic motivation will report a higher level of chatbot-related competencies than students who have a low level of intrinsic motivation

The r-score for high intrinsic motivation and high level of workshop satisfaction is 0.590 which provides moderate support for H5. The r-score for motivation and post-workshop competencies is 0.196. At the current stage of this study, this finding is not conclusive enough to support H6. While the results highlight the importance of intrinsic motivation with regard to being satisfied with the bot-related teaching and learning approach, they also underscore the educational importance of (i) priming learners at the beginning of the course so that they appreciate the power and use cases of smart chatbots not only in education but also in business and society, and (ii) experimenting with different ways to keep students motivated and making them competent throughout the course in order to optimize their capacity to attain the desired learning outcomes.

6.6 Pre-workshop proficiency, chatbot workshop-induced competencies, engagement and satisfaction

  • H7: Students with a high level of pre-workshop (functionality-related) proficiency will have a higher level of post-workshop competencies than students with a low level of pre-workshop (functionality-related) proficiency

The r-score measuring pre-workshop (functionality-related) proficiency and post-workshop competencies is 0.363, which shows weak support for H7. The results suggests that students with prior knowledge and experience in chatbots might not necessarily have a substantially higher advantage compared to their peers who lack the knowledge and experience.

  • H8: Students with a low level of pre-workshop (functionality-related) proficiency will be more engaged as a result of the chatbot workshop than students with a high level of pre-workshop (functionality-related) proficiency.

The r-score for pre-workshop (functionality-related) proficiency and student engagement during the workshop is 0.050. There is no support for H8.

  • H9: Students with a low level of pre-workshop (functionality-related) proficiency will be more satisfied with the chatbot workshop than students with a high level of pre-workshop (functionality-related) proficiency

The r-score for pre-workshop (functionality-related) proficiency and workshop satisfaction is -0.107. This points to a negative correlation between student’s prior knowledge and experience on the one hand and workshop-related satisfaction on the other hand. Within the current data sample, the results do not support H9.

With regard to H7, H8 and H9, it is worthwhile to ask deeper questions about students’ prior knowledge in chatbots. While accurate and appropriate prior knowledge can aid their learning, inaccurate and inappropriate prior knowledge can hinder their learning during the workshop. For those students that indicated “Yes, I have prior knowledge and experience in chatbots”, a pre-workshop assessment (e.g. quiz) can help to evaluate whether their knowledge is accurate and appropriate.

6.7 Alignment of Chatbot-related learning expectations and students’ workshop satisfaction

  • H10: Students with a high level of alignment will be more satisfied with the Chatbot workshop than students who have a low level of alignment.

The r-score for alignment of learning expectation and workshop satisfaction is 0.572. The results show moderately high support for H10. This suggests that managing the student’s learning expectation could be an important aspect for instructors to consider in designing and facilitating the experiential learning workshop (Table 9).

Table 9 Summary of Correlational Analysis (n = 43)
Full size table

7 Discussion

According to the survey results, 90.7% of all respondents were satisfied with the experiential learning chatbot workshop. Altogether 83.8% of the surveyed students who attended the experiential learning process reported to be motivated while 81.4% of the chatbot workshop participants felt engaged. Most students reported moderate to high levels of competencies (81.3%) after the participation in the experiential learning workshop. Figure 6 shows the distribution of students’ responses to the survey question where the students were asked to rate their chatbot-related proficiency before and after the workshop. A t-statistic was calculated to be -13.3 with 41 degrees of freedom and a significance level of 0.05. The p-value was 8.99E-17, which is less than the significance level, indicating that the difference between the groups is statistically significant at the 0.05 level. This indicates that according to the students’ assessment of their own proficiency, there is an improvement in their chatbot-related proficiency.

Fig. 6
figure 6

Distribution of Students’ Chatbot-Related Proficiency

Full size image

Almost all students surveyed (97.7%, n = 42) felt that the experiential chatbot workshop had met the expected learning outcomes. Overall, the empirical study results underscore the pedagogical usefulness of conducting an experiential Chatbot workshop and that it positively influences students’ intrinsic motivation and engagement levels so that they can acquire basic AI competencies with special reference to NLP (Table 10).

Table 10 Scales and Sample Items (Examples)
Full size table

The correlation analysis suggests that students with a high level of intrinsic motivation are gaining more from a hands-on Chatbot workshop than those learners who have a low level of motivation (H1). Students with a high level of engagement in relation to the chatbot workshop experience reported a higher level of chatbot-related competencies than learners with a low level of engagement (H3). Students with a high level of intrinsic motivation turned out to be more satisfied with the Chatbot workshop than those with a low level of intrinsic motivation (H5). Students whose expectations match the way the Chatbot workshop is conducted seem to be more satisfied with the Chatbot workshop than students with a low level of alignment (H10).

One caveat for interpreting our study is the small sample size and the fact that 100% of the respondents were enrolled in the same school (LKCSB) which may limit the model generalization. Despite belonging to different course levels, year of study and subject specialization (course major), the sample variation is limited which in turn may negatively affect the transferability of the results. However, as explained earlier, we believe that our results are useful for other instructors interested in experimenting with NLP-powered bots in class and that our findings are also applicable to non-ICT disciplines such as marketing (Etz & Arroyo, 2015).

Key strengths of our study include: (i) the innovative application of a hands-on chatbot workshop in an introductory AI for business course and initial evidence that such a practicum helps learners to acquire basic A.I./NLP skills; (ii) the evaluative survey design of the study and sharing of lessons learned; (iii) the multi-disciplinarity of the research team, and (iv) the data-driven recommendations for other instructors who wish to experiment with bots in teaching and learning.

Our study adds to the discourse about competency development related to A.I. (Benner, 2004; Tamayo et al., 2020) by sharing evaluative lessons learnt during the roll-out of an experiential chatbot workshop as an engagement component in a basic (elective) A.I. course.

While our empirical study results support the findings of previous research studies on student engagement and proficiency development (Augustyniak et al., 2016; Heindl, 2020; Mandernach, 2015; Monteiro et al., 2015; Navarro et al., 2020); the challenge remains to fully understand the drivers of low motivational and engagement levels amongst some of the students surveyed and what can be done to change that.

One possible reason could be the novelty of the experiential learning approach and the fact that many if not most of the students had no prior experience with Google’s Dialogflow platform, incl. the type of hands-on learning activity practiced during the virtual MGMT240 workshop segment which was delivered via Zoom. To what extent the perceived lack of learner autonomy hampers the autonomous motivation (Gagne, 2003) of our students has yet to be examined. Other possible causes are best explored in a qualitative context via a future focus group discussion. Acquiring more knowledge about student’s prior motivation and bot-related knowledge depth is arguably very important to further increase the effectiveness of our chatbot workshop approach.

8 Conclusion, limitations, and future research direction

In this paper we presented results of an ongoing exploratory, empirical study of students enrolled in an elective course “Doing Business with A.I.” at the Lee Kong Chian School of Business (LKCSB), Singapore Management University, to examine the pedagogical usefulness of conducting an experiential Chatbot workshop that intends to get students motivated and engaged with regards to the acquisition of basic AI competencies, namely NLP.

The empirical study results suggest that a hands-on teaching and learning approach in form of an experiential Chatbot workshop is indeed instrumental in motivating and engaging students to acquire basic bot competencies. To summarise some of the key survey results: 90.7% of all respondents were satisfied with the experiential learning chatbot workshop; 83.8% of the surveyed workshop participants reported to be motivated while 81.4% of the students turned out to be engaged. Most students reported moderate to high levels of competencies (81.3%) after the participation in the experiential learning workshop. Almost all learners who took part in the survey (97.7%) felt that the experiential chatbot workshop had met the expected learning outcomes.

Overall, our empirical study results underscore the pedagogical usefulness of conducting an experiential Chatbot workshop and ascertain the argument that it positively influences students’ motivation and engagement levels so that they acquire basic AI competencies such as NLP. Educators who wish to replicate our hands-on chatbot approach are advised to consider the following recommendations:

  • It is important to know whether and how much the participants do know about the subject matter / workshop topic (bots), including prior experiences and actual competencies (NLP) before the start of the course unit (a pre-workshop quiz can help to ascertain that).

  • Students must be made fully aware before or at the beginning of the workshop why chatbots are increasingly used in ‘real’ business contexts and beyond and how they are powered up by AI techniques such as NLP.

  • Case analyses of high-performance organizations who managed to create novel value by deploying chatbots are useful in supporting students’ understanding of the bot subject matter.

  • Ensure high motivation and engagement levels by allowing for mistakes and providing specific, positive feedback on completed assignments.

  • Create challenging and meaningful learning experiences that are both enriching and competency-enhancing.

Limitations include constraints caused by the Covid-19 situation which led to several changes in the way the experiential chatbot workshop was conducted. Instead of physically attending the session in a classroom with instructors, all students had to participate online via home-based learning / Zoom. This implies that some of the proven instructional strategies to impart chatbot-related know how and know why to students such as small group-based collaborative learning projects and interaction over assigned tasks had to be omitted due to the distance learning arrangement. We were unable to identify to what extent these factors may have affected the overall learning effectiveness.

The small sample size and the fact that 100% of the respondents were enrolled in the same school (LKCSB) do limit the model generalization. Despite belonging to different course levels, year of study and subject specialization (course major), the sample variation may be limited and insufficient to fully understand the implications of the results.

In terms of future research, it would be meaningful to expand the scope of this study to compare responses from a larger sample size of students to conduct model-specific regression analyses and solid hypotheses testing.