Assessed by Machines: Development of a TAM-Based Tool to Measure AI-based Assessment Acceptance Among Students

In recent years, the use of more and more technology in education has been a trend. The shift of traditional learning procedures into more online and tech-ish approaches has contributed to a context that can favor integrating Artificial-Intelligence-based or algorithm-based assessment of learning. Even more, with the current acceleration because of the COVID-19 pandemic, more and more learning processes are becoming online and are incorporating technologies related to automatize assessment or help instructors in the process. While we are in an initial stage of that integration, it is the moment to reflect on the students' perceptions of being assessed by a non-conscious software entity like a machine learning model or any other artificial intelligence application. As a result of the paper, we present a TAM-based model and a ready-to-use instrument based on five aspects concerning understanding technology adoption like the AI-based assessment on education. These aspects are perceived usefulness, perceived ease of use, attitude towards use, behavioral intention, and actual use. The paper's outcomes can be relevant to the research community since there is a lack of this kind of proposal in the literature.


I. Introduction
W ILL be your next teacher a neural network model? What would be your feelings related to being evaluated by a non-human entity in such a human process like learning? In the last years, we have been experiencing a rising in integrating Artificial Intelligence (AI) applications in our daily lives [1]- [3].
People are beginning to be aware that they may interact with AI applications when interacting with mobile devices, computers, or in general, in technology-surrounded environments. What is less known, or the people are less aware, is that AI applications surround us even in more traditional contexts, even in those that typically were pure humanbased and did not involve any intelligent software stakeholder [4]- [6]. Many previously pure human-based contexts are now intermediated by intelligent systems or capable decision-making software that affect how we experience our lives or interact with other people.
As commented in the literature [7], [8], one of the fields in which we can observe the introduction of more sophisticated software programs is education. As [7] outlines, in recent years, we have experienced the inclusion of virtual teaching assistants in classrooms (for example, Jill Watson, deployed at the Georgia Institute of Technology), technology to study and improve students' learning performance, AR/VR technologies, teaching robots, etc. The trend of including more technology in different educational environments is even more significant today, during the current global COVID-19 pandemic. Since the beginning of the pandemic, many people have used virtual learning and teaching and different approaches to improve those online experiences. By using video-meeting resources and online educational platforms, many teachers have solved part of the issue; they have given their lectures and provided learning materials.
However, what happens with other fundamental aspects of the learning processes like the assessment? Are there intelligent systems being used to evaluate students learning? Are the students OK with being potentially evaluated by software artifacts based on AI algorithms or any other similar approach? According to our previous research [8]- [10], there is a lack of studies on this topic, especially in studying acceptance among students and teachers.
Based on the literature gap detected and related to our previous research, this paper presents a novel TAM-based tool to measure students' acceptance of being assessed by intelligent software artifacts (AI-based ones). The purpose of this tool is to serve as the foundation to develop further empiric studies that could enrich our understanding of how people (and students) interact with intelligent software, in particular in contexts like education, and even more when the AIbased software is in a power position like in the role of an evaluator.
The paper includes the following sections: Section I introduces the problem and some relevant research. Section II presents the theoretical background of the paper. Section III describes the TAMbased model we propose to study the AI-based assessment acceptance among students, while section IV presents the specific instrument we propose to research that question. Section V discusses the outcomes of the paper and its future implications. Finally, Section VI presents some conclusions.

II. Theoretical Background
The possibilities entailed by using AI-based systems in the educational field constitute a topic of growing attention among the scientific community, based on the fast-development of these technologies and the vivid social debate regarding the risks ethical considerations on the use of AIs [8].
This way, along the last decade, we can observe an increasing number of initiatives that explore the different uses of these technologies to support the teaching-learning process that can be classified into three groups: use of AIs to analyze human behavior, use of AIs as didactic tools and use of AIs as assessment tools.
In the first group, we can find investigations that use AIs as tools to perform complex statistical analysis in data-driven approaches with large groups of data to detect behavioral patterns of teachers and students susceptible to intervention. The initiatives in this group are mainly focused on students' elements such as their learning strategies [11], [12], although we can find some examples of its use on investigations with teachers [13].
On the other hand, the second group is focused on the didactic potential of AIs and examines its application in the creation of virtual persons to help students during their learning [14], the development of virtual learning environments [15] or even the use of social robots [16].
Finally, the third group is interested in the possibilities derived from using AI-based systems on the students' assessment, which constitutes one of the students' primary concerns and a key competence of the teachers [17].
The study of the use of AIs for evaluation is still a very young field in an early stage of development in which most research is concentrated in the last decade, mainly in North America [8].
Even so, we can find exciting initiatives in this area developed in the context of higher education. These investigations can be grouped into three categories: • Use of AIs for the assessment of behaviors: These investigations are focused on the analysis of student behavior at different circumstances of the teaching-learning process.
This way, we can find initiatives that evaluate through AIs the interactions of students in online courses to predict their academic performance [18], [19] or their behavior during activities to personalize their learning experience or develop adaptive processes [20], [21].
• Use of AIs for the assessment of feelings: Experiences that apply AI-based tools to determine the emotions experienced by students during the development of different educational processes The most common application of these technologies is the analysis of students' satisfaction in online courses [22], [23], although we can find research that uses the analysis of feelings with other objectives such as predicting the success of MOOCs [24].
• Use of AIs for the assessment of academic performance: In this group of investigations, the most numerous of the three, the AI-based tools are used to automate the correction process of both exams (multiple choice or short answers) [25], [26] and other types of student productions [27].
The application of these tools is intended to lighten teachers' workload and improve the fairness of their grading [28].
Despite this growing presence of AIs in the educational field and the interest among the scientific community and organizations in developing this practice, the study of the adoption of these technologies among the educational agents constitutes an underexplored research area.
However, we can find examples of research focused on this topic in other fields, such as the development of automated vehicles, banking, or e-commerce [29]- [31] that indicate the critical role played by factors such as perceived usefulness, confidence, or social pressure in the decision to use AI-based tools.
The educational agents' willingness is a fundamental element to guarantee the success of any technological innovation process. Therefore, the development of adoption models to analyze the factors that condition such adoption can provide essential information for developing of new initiatives in this field.

III. Model Development
As a starting point for developing the model, we reference the technology acceptance model (TAM) proposed by Davis [32]. This theory understands the process of adoption of a specific technology based on five factors: • Perceived usefulness (PU): The subject's perception of the effect that the tool's use has on their work performance.
• Perceived ease of use (PEU): The person's assessment of the degree of effort required to use the system. TAM's main advantages are its parsimony and flexibility, which allows the application of the model to a wide variety of contexts and technologies [33].
In the educational field, enhanced by the fast technological development, TAM based models expanded with constructs from other theories have been widely applied to analyze the adoption process of technologies such as mobile devices [34], LMS [35], or AR [36] both between students [37] and teachers [38].
However, since the use of AI for the assessment is still in a very initial stage of development, the development of TAM-based models for the analysis of these technologies' adoption process among the educational agents constitutes an unexplored line of research.
Following this line of development, this model includes the six hypotheses proposed by Davis [32] applied to the object of study: H1 PU is positively related to the intention to participate in the AI-driven assessment activities of the students.
H2 PU is positively related to the attitude towards the participation in AI-driven assessment activities of the students.
H3 PEU is positively related to the attitude towards the participation in AI-driven assessment activities of the students.
H4 PEU is positively related to the students' usefulness in the implementation of AI technologies for their assessment.
H5 AU is positively related to the intention to participate in AIdriven assessment activities of the students.
H6 BI is positively related to the participation in the AI-driven assessment activities of the students.
One of the most common practices during the process of adapting TAM to a new field of technology is the expansion of the original proposal and the inclusion of constructs from other theories pertinent for the object of study, with the intention to increase the percentage of variance explained by the model [39].
Following this line of reasoning, we decided to expand TAM with three additional constructs to measure the influence of environmental pressure, the feeling of distrust towards AIs, and the natural opposition to the individual's changes.
Firstly, to measure the effect of the individual's perception of the social discourse on the use of AIs for evaluation on their adoption of this technology, this model includes subjective norm (SN).
This construct refers to the perceived social pressure towards the performance of a particular behavior, the belief that a substantial group or individual approves or disapproves a given action [40], in this case, the use of AIs in the evaluation of the students.
Since its formulation in the theory of reasoned action (TRA) [41] and the theory of planned behavior (TPB) [42] this construct has been widely incorporated in the design of technology adoption models such as TAM2 [43], TAM3 [44] and expanded TAM models [45].
In the educational field, we can find a wide variety of studies that incorporate SN to measure the social influence on the technology adoption of the students [46] confirming its effect on the behavior of the subjects through its influence on other TAM constructs such us PU and BI.
Given the heated debate surrounding the use of AIs in education, and the lack of information and experience of the students regarding this practice, SN should have an important influence on both, the perception of the individual of AIs and their disposition towards its use. Therefore, the model includes the following two hypotheses for SN based on the proposals by Venkatesh and Davis [43] and Venkatesh and Bala [44] : H7 SN is positively related to the intention to participate in AIdriven assessment activities of the students.
H8 SN is positively related to the usefulness perceived by the students in the implementation of AI technologies for their assessment.
One of the main problems when dealing with the implementation of the AI-based systems is the lack of user confidence in both the handling of the user's personal information and the resolution of complex problems [47]. This issue has been acquiring greater importance due to the proliferation and development of these systems and the interest of organizations and administrations in their incorporation [48] .
Trust is a variable often included in technology adoption studies, especially in contexts such as online banking [49], e-commerce [50] or eGovernment [51] defined as the willingness of the individual to rely on the other part [52], establishing its relationship with other variables from TAM like PU or BI.
In the educational field, the effect of this dimension is still little explored, although we can find some examples of its application in research on the adoption of technologies such as cloud computing [53].
In this case, trust (TR) plays a special role since, although they may be perceived as agents, AIs have no conscience or morals [54], thus in the context of this study, users will perceive AIs to be useful and be more inclined to be evaluated by them if they trust both the teachers that use them and the AIs themselves. In consequence, for this research, we propose the following hypotheses based on the works of [49]- [51]: H9 TR is positively related to the intention to participate in the AIdriven assessment activities of the students.
H10 TR is positively related to the attitude towards the participation in AI-driven assessment activities of the students.
H11 TR is positively related to the usefulness perceived by the students in the implementation of AI technologies for their assessment.
Incorporating the use of AI in education implies a series of changes in the teaching-learning process, including the attribution of one of the most important tasks of teachers, as is the assessment, to a technological tool.
This change can cause a feeling of anxiety among the students in the face of the expectation of not being evaluated by a human being, which may generate a negative predisposition towards participation in educational activities with this modality of evaluation. In order to measure this effect, the model includes the construct of resistance to change (RC).
RC was born in the field of organizational sciences to analyze the organizational elements that generate an attitude of resistance in the individual, although this concept was also applied to study factors related to the individual [55], [56].
In this proposal, RC is modeled after the construct developed by Guo et al [57] and defined as the opposition of the individual to the rupture of the status quo produced by the use of AIs for their evaluation. This way, RC would be an inhibiting factor that will negatively affect the students' technology adoption [58].
In the educational field, RC is recently being incorporated in expanded TAM models to study students' adoption of ICTs with promising results [59] supporting is the effect in the three main dimensions of TAM; namely, PU, AT, and BI. Considering the importance of the changes that the implementation of this technology entails for an area that constitutes one of the students' main concerns, RC is expected to affect the aforementioned three constructs. Therefore, the model is completed (Fig. 2) with the following three hypotheses: H12 RC is positively related to the intention to participate in AIdriven assessment activities of the students.
H13 RC is positively related to the attitude towards the participation in the AI-driven assessment activities of the students.
H14 RC is positively related to the usefulness perceived by the students in the implementation of AI technologies for their assessment.

IV. Instrument
The instrument proposed to measure the seven constructs of our model is divided into three sections. Firstly, taking into account that the use of these technologies is still in the early stages of development, it is very likely that students will not have a clear idea of what constitutes the use of AIs for assessment, the instrument to perform the data gathering process includes a brief introduction text explaining the fundamentals of this topic.
The second section is dedicated to gathering the participants' identification data regarding four variables that may affect the adoption process, namely, gender, age, experience with AIs, and the branch of knowledge.
The third section is composed of 30 Likert-type items adapted from previous works to measure the object of study. Firstly, the items to assess the constructs from the original TAM were adapted from the proposals by Davis [32] and Venkatesh and Bala [44] (table I):   TABLE I. Items to Measure the Tam Constructs   Item  Text AU1 Using AIs to support the assessment of the students is a good idea.

AU2
Using AI-based tools to assess the academic achievement of the students is a smart choice.

AU3
I like the idea of using AIs for the assessment of the students.

AU4
I have positive feelings about implementing AI-based tools for the assessment of the students.

BI1
I would participate in AI-based assessment processes.

BI2
I plan to participate in AI-based assessment processes often.

BI3
I hope to participate in AI-based assessment processes in the future.

PEU1
My interaction with AI-based tools is clear and understandable. Secondly, to measure the new constructs added to the model, we took as reference different expanded TAM models (Table II). People who are important to me think AIs should be used to assess the students.

SN2
My classmates think teachers should use AIs to assess their students.

SN3
In the universities, teachers are expected to use AI-based tools to assess the academic achievement of the students.

SN4
People who influence my behavior think I should participate in AI-based assessment processes often.

SN5
People who are important to me think I should participate in AIbased assessment processes often.

RC1
I would like AIs to change the way assessment is carried out.

RC2
I want AIs to change teacher-student interaction during the assessment.

RC3
I would like AIs to change the way I am assessed.

RC4
I agree with the changes entailed by the use of AI for the assessment.
TR2 I tend to trust AIs.

TR3
I trust AIs even though I have little knowledge of them.

TR4
AIs can provide an accurate assessment of the students.

TR5
AIs can provide a reliable assessment of the students.

TR6
AIs can provide a convenient assessment of the students.
In the case of SN, the five items for this dimension were adapted from the works of Venkatesh and Bala [44] and Taylor and Todd [60], [61] to measure both the general sources of SN and the specific sources related to the educational environment of the student.
The RC items were developed based on the proposals by Guo et al. [57] and Sánchez-Prieto et al. [59] that refer both to elements related to the teaching-learning process and with the interaction between teacher and student.
Finally, the items to measure the trust of the students in AI and their capability to assess them were adapted from the versions of Gefen et al. [52].

V. Discussion and Conclusion
As we have shown, AI-based evaluation and the integration of algorithmic solutions in the learning process have been a trend in recent years. It is increasing its relevancy at a rapid pace. The shift of traditional learning procedures into more online and techish approaches contributes to algorithmic systems' implantation. Even more, with the current acceleration because of the COVID-19 pandemic, more and more learning processes are becoming online and are incorporating technologies like those described relative to assessment. The opportunity of going online is a powerful temptation in the current situation or, in some cases, is the only way to continue learning. Related to that, the use of new assessment technologies based on AI or any other algorithmic approaches correlates perfectly with that trend of going online. Since most of the interaction is online and produces a digital footprint, we can think that we can use, easily, automated ways of evaluating the interaction. That is partially true. It is right in the sense that many digital footprints can be analyzed automatically. We can indeed make simple analytics in almost any digital data. We can probably even introduce improvements in the education process that can favor students and teachers, fostering their performance, opportunities, and engagement [62]. What is not exact is that we can introduce a relevant change like AI-based or algorithmicbased assessment of learning without paying much attention to some different aspects. Before going forward, we must evaluate the real need to use this kind of procedure, the level of detail required for the analysis, the interaction of humans in the process, the ethical issues related, or, as underlies in this paper, the perception of being evaluated (or helped in evaluation) by a non-conscious software entity that enables decisions or takes them in our behalf or while considering our human performance.
Furthermore, considering most of the existing algorithms and systems cannot grasp the human processes' entropy and their multiple dimensions and subtle details completely, leading to harmful results for the humans involved. For these reasons, there are needed studies that research the human perceptions of these procedures driven by algorithms.
This article presents a theoretical model of technology adoption that aims to explain students' adoption using AI-based tools for their assessment. The model is based on TAM and expanded with the most relevant constructs in adopting this technology: perceived usefulness, perceived ease of use, attitude towards the use, behavioral intention, trust, resistance to change, and subjective norm.
With the inclusion of these variables, the model proposed can address both the effect of the perceived advantages and disadvantages and the factors related to the trust issues and the reluctance of the subjects to be assessed by these tools. The resulting model includes 14 relational assumptions that determine the effects of the different variables.
To measure these constructs and aim to guarantee the proposal's parsimony, we have designed a questionnaire composed of 30 Likerttype items adapted from previous theories, which allows for the empirical testing of the model and its expansion and modification.
This way, this model constitutes a first step in the study of the adoption of AI-based assessment among students, which opens the door to further investigations. Firstly, although the items are adapted from previous investigations and have been exhaustively tested in other adoption models with good results, the instrument could be subjected to a content validation process to confirm the items' quality. In this line, a pilot study is currently in development to test the proposal's statistical validity in an empirical setting. Finally, this model is also susceptible to further modification, including its expansion with additional constructs from other theories that may increase the understanding of the adoption process of these technologies.

José Carlos Sánchez-Prieto
José Carlos Sánchez-Prieto received a bachelor's degree in Pedagogy and a master's degree in ICT applied in education from the University of Salamanca (Spain) where he also presented his PhD on the adoption of mobile technologies among teachers within the Programme on Education in the Knowledge Society. Currently he is an associate professor at the Faculty of Education of said university. His area of research is the assessment of attitudes among in-service and pre-service teachers and students. He is the author of several articles on the subject published in prestigious international journals. You can contact him at: josecarlos.sp@usal.es. He is the co-author of more than 80 publications, serves as a reviewer for many high-respected journals and conferences, and is Associate Editor of the Journal of Global Information Management (JGIM) and the Journal of Information Technology Research (JITR). His research activity conducted him to receive the Research Award "Scientific Computer Society of Spain and BBVA Foundation" as one of the best young researchers in Computer Science in Spain in 2019. As a software engineer and developer, he is involved in many different technical communities and Open Source initiatives. Contact him at https://juancb.es/ or juan.cruz@ibm.com

Roberto Therón
Roberto Therón Sánchez is currently the Manager of the VisUSAL Group (within the Recognized Research Group GRIAL), University of Salamanca, Salamanca, Spain, which focusses on the combination of approaches from computer science, statistics, graphic design, and information visualization to obtain an adequate understanding of complex datasets. He received the Diploma degree in computer science from the University of Salamanca, the B.S. degree from the University of A Coruña, the B.S. degree in communication studies and the B.A. degree in humanities from the University of Salamanca, and the Ph.D. degree from the Research Group Robotics, University of Salamanca. His Ph.D. thesis was on parallel calculation of the configuration space for redundant robots. He has authored more than 100 articles in international journals and conferences. In recent years, he has been involved in developing advanced visualization tools for multidimensional data, such as genetics or paleoclimate data. In the field of visual analytics, he develops productive collaborations with groups and institutions internationally recognized as the Laboratory of Climate Sciences and the Environment, France, or the Austrian Academy of Sciences, Austria. He was the recipient of the Extraordinary Doctoral Award for his Ph.D. thesis. Contact him at theron@usal.es.  (51 Q1). For more detailed information on the publications, these are the public links to the profiles in Google Scholar (http://goo.gl/sDwrr0), Publons (https://bit.ly/2u2FN5l) and ORCID (http:// orcid.org/0000-0001-9987-5584).