Assessment of Matrix Multiplication Learning with a Rule-Based Analytical Model – A Bayesian Network Representation

This study explored an alternative assessment procedure to examine learning trajectories of matrix multiplication. It took rule-based analytical and cognitive task analysis methods specifically to break down operation rules for a given matrix multiplication. Based on the analysis results, a hierarchical Bayesian network, an assessment model, comprising of 2 layers of explanatory variables-Matrix Multiplication, Performance and Semantic Explanations; and one layer of evidential variables containing 9 evidential variables-was developed. With the simulating data, 9 students’ Performance and Semantic Explanation evidences were recorded. The results indicated that the hierarchical Bayesian assessment effectively traced and recorded students’ learning trajectories; and assessed students’ learning dynamically and diagnostically.

referred to as a scalar such as 8, 2.58, 3 and π etc.A vector, algebraically, is a set of ordered n-tuples of real numbers written in either a row vector such as v'= [3,7,12,15,21] or column vector such as: . A matrix is m-by-n dimensional array of real numbers.The individual items in a matrix are called its elements or entries.The matrix multiplications are the multiplication operations.The multiplication relations include the operations between a scalar and a vector, a scalar and a matrix, two vectors, a vector and a matrix, and two matrices.The individual items in a matrix are called its elements or entries such as: 3 2 5 1 1 4 2 6 2 9 7 3 8 2 1 7 2 3 4 3 . The matrix can generally be written as

The Rules of Matrix Multiplications
There are a set of rules in matrix multiplications used to guide the learners to solve the problems.A vector is designated by black fond lower case u, v or other lowercase letters; a matrix is designated by an uppercase letter A, B or other uppercase letters.
2) cA= 3) In the vector multiplication, the column dimension of the first vector should be equal to the row dimension of the second vector.In other words, the multiplication between two vectors should consist of one row vector v' and one column vector u.
4) The product of two vectors v' and u is defined as 1 1 ' j j a v u =  .5) The product of two vectors u and v' is defined as 6) v'u≠uv'.v'u is a scalar.uv' is a matrix which can be designated by A ik .The row dimension of the matrix A is determined by the row dimension of the u; the column dimension of the matrix A is determined by the column dimension of the v'.In order to further understand this rule, I use two examples to show the results of v'u and uv'.
The products can be illustrated with examples.
Assuming there are vectors w'=[1, 2, 3], and matrix The multiplication between two matrices requires that the column dimension of the first matrix be equal to the row dimension of the second matrix.
There are matrices X and Y. the product of these two matrices Z is defined as , the product of these two matrices: 9) The positions of vectors/matrices in composite matrix multiplications cannot be changed.
For example:

Relevant Knowledge Supporting Matrix Multiplication
Matrix multiplication learning is at the advanced level of research study design and data analysis.It is assumed that the learners have all mathematics, geometrics and statistics knowledge to support the learning task.All pieces of knowledge are potential problem solving rules and conditions.There is weak association between the relevant knowledge and completion of matrix multiplication.
1) The learners are able to understand, R n which is the set of all ordered n-tuples of real numbers written as row or column vectors.Thus a vector v in R n is of the form .
2) The learners understand the operations of addition, subtraction and multiplication of scalars.
3) The learners understand positive numbers and negative numbers.
4) The learners are able to understand and do calculations of sum of products.
All these above consist of the supplements of the rules of matrix multiplications.At the advanced learning level, amount of the knowledge and problem solving skills to be prepared to solve matrix multiplication is unknown.
The mistakes and non-proficiency can receive the evidence from a specific study based on a specific learner group.

Rule-Space Model and Rule-Based Analytical Model
6.1 Rule-Space Model Tatsuoka (1983) developed the rule-space model, which was the combination of cognitive task analysis and psychometric measurement models (Snow & Lohman, 1989).The rule space was explored by using cognitive task analysis (Clark & Estes, 1996) to obtain a set of attributes which represent a set of specific skills or competencies (Gierl, Leighton, & Hunka, 2000).A typical example is fraction operation where Q matrix was applied to (Tatsuoka, 2009).The Q matrix provides a two-dimension structure to highlight a set of rules being written as elements in the matrix to represent a given underlying cognitive structure.The elements consist of the skills/attribute set.Each cognitive task (/test item) requires some or all of elements within the Q matrix.Mathematically the J × K Q-matrix can be defined with the following descriptions.
Q jk = 1 or 0 depending on whether or not attribute k is required by task j.
In the algebra multiplication learning task, it assumes that there are i students.
X ij = 1 or 0 depending on whether or not student i performs task j correctly a ik = 1 or 0 depending on whether or not student i possesses attribute k.
Thus, the Q-matrix forms the basis of the cognitive diagnostic assessment structure.
Here, an arbitrary Q-matrix is shown for the illustrated purpose.
In this two-dimension table there are 5 learners and they face 5 attribute elements to solve fraction problem, for example.We can see that learner 1 has attribute 3 and 4, learner 2 has attribute 1, 2 and 5; learner 3 only has attribute 1 and 5; learner 4 has attribute 3, 4 and 5. Lastly learner 5 has all of 5 attributes.
Q-matrix has two properties: a) the proficiency status is dichotomous, 0 and 1; and the rule space is closed.Stated differently, there are only two statuses for each learner to complete the fraction cognitive task: success and fail; the cognitive task is well-structured and all of the rules are well-defined.Thus, the rule space is closed.Such ideal problem structure is possible in the elementary level of algebra operation and learning.However, at advanced level of algebra, operation such as matrix multiplication is not appropriate.Multi-statuses for the cognitive task completion are required for both learning strategies and diagnostic assessment.Further, the rule space is not closed.Beyond the rules of matrix multiplication, there are some potential knowledge and problem solving skills regarded as cognitive attributes, candidates must possess.Thus, an alternative model-rule-based analytical model is necessary to be developed.

Rule-Based Analytical Model
Rule-based analytical model is a mixed model from rule space model and cognitive task analysis (Clark, Feldon, van Merriënboer, Yates, & Early, 2007;Graco, 2012).How to establish a cognitive model in a diagnostic assessment is dominated by the characteristics of the cognitive task itself.It is cognitive task specific.When we solve fundamental algebra problems such as fraction problem, a closed rule space model with dichotomous attributes is sufficient.When a cognitive task is completely ill-structured such as learning in medical emergency environment, cognitive task analysis approach is strongly suggested (Zhang & Lu, 2014b).The operation in matrix multiplication requires both a set of rules and cognitive task analysis.In other words, the problem space of attributes is opened rather than closed because the rules cannot be exhausted and relevant problem solving knowledge and skills may become attribute elements with the analysis of learners' evidence of matrix multiplication, especially those mistake evidence.

Research Questions
The critical issue in this study is how to establish an effect diagnostic assessment procedure for learners in matrix multiplications.This effective assessment model can provide both sufficient learning strategies, and diagnostic assessment information when they are challenged by some problem ran across in the operation of matrix multiplications.Thus, the following research question can be addressed: 1) How the rules of matrix multiplication can be used to develop rule based analytical model in the diagnostic assessment?
2) How the cogitative task analysis can be used in setting up a cognitive diagnostic model?
3) How are the learners' knowledge and problem solving skills in matrix multiplication assessed sufficiently and reported diagnostically?

Methodology
The research methodology includes both theoretical framework, data-driven model analytical process.The theoretical framework describes the rationale how to develop an assessment model which reflects both learning and assessment process.The data-driven model analytical process depict the step-by-step process in the assessment model development

Rule-Based Model and Cognitive Task Analysis
There are 8 rules for the matrix multiplication.As the guide to develop evidential variable, the set of learner's rules can be referred to building the diagnostic assessment framework.Briefly, these rules and relevant knowledge are summarized here again: 1) cv: the multiplication of scalar c and vector v.
2) cA: the multiplication of scalar c and matrix A.
3) v'u or uv': the production of two vectors.4) v'u≠uv'.v'u is a scalar.uv' is a matrix 5) For w, w', two vector and matrix Q, w'Q≠Qw.

6) For the product
, the dimensions should be effectively defined.
In addition, some relevant knowledge should be proficient for the learner, such as concept and problem solving skills of the sum of products: 7) The positions of vectors/matrices in composite matrix multiplications cannot be changed.

Diagnostic Assessment Model and Structured Representation
The matrix multiplication is an advanced operation in mathematics computation.Even there are several rules being recognized, the all sample space of the problem solving cannot be exhausted.It is better to take an example to illustrate the development of the diagnostic assessment model and further represent the assessment structure.
Supposed there are two matrices 33 R and 32 S .The product is mn T and the learner is asked to get the solution to the matrix multiplication.
Generally we have . Now we have two specific matrices: , and now the learning task is to gain The rule and knowledge to complete this matrix multiplication at the moment include: 1) To be able to determine the definition, applying to Rule 4 and 8 of Section 4.
2) To be able to know that the sum of product of ij r and jk s is the element ik t of new matrix.
3) To be able to know 32 mn T T = 4) To be able to perform the product of each pair ij r and jk s , such as 11 11 s r × .5) To be able to perform the sum of the product ik t for each ij r and jk s such as: 6) To be able to explain the subscripts for both r and s Based on these knowledge and cognitive trajectory analysis, we attempt to establish a hierarchical cognitive process model.The model consists of two cognitive aspects: performance and semantics explanations.Under each aspect following the cognitive trajectory, we can establish the model which is hierarchical.The model is conceptually represented via BayesiaLab 6.02 (Conrady & Jouffe, 2015)

Updating Probabilities of Explanatory Variable with Random Evidence
There are nine evidential variables which can be used to update Matrix Multiplication diagnostic assessment model.A randomly sampling method is applied to test evidence states from zero success to nine success evidence observations.The success status means that a learner receives a positive score of the given variable-either performance or semantic explanation.
Table 1 presents the relations of instantiated evidential variable and explanatory variables.Matrix Multiplication indicates the general level of a matrix multiplication problem.The minimum value for Matrix Multiplication is 0.37, Performance is 0.1074 and Semantic Explanation is 0.0148.The maximum value for Matrix Multiplication is 0.8747, Performance is 0.9296 and Semantic Explanation is 0.9904.The minimum and maximum values indicate the range of each variable.For example, the range of Matrix Multiplication is 0.8747-0.3713=0.5034.Further, we observe the increase of the values of each variable with the evidences increase.We expect a robust trend of mono increase for the learning curve.We observe that evidence numbers equal to and greater than 7 while the updated values of each column keep a mono increase.Thus, it seems it is plausible we say when the learner masters 7 and 7 + score points, the learner's progress becomes robust.Of course, such a conclusion is model specific.Different cognitive and assessment models will have different cognitive trajectory patterns.

Students' Proficiency in Matrix Multiplication Problem Solving
There is a simulated data set assuming it consists of 9 individual learners.Sometimes, there is an unbalance in the learning tasks of performance and semantic explanation.There are at least two reasons for such a phenomena: a) the learners' individual experience bias and b) the property of the learning tasks.Some learning task is easier to perform and challenging to explain semantically, and some learning task is easier to explain, but it is very difficult to perform.

Conclusions and Discussion
This study examined the processes and procedures of developing a diagnostic assessment model with Bayesian network representation for matrix multiplication learning.The diagnostic assessment model received a conceptual and quantitative representation with the Bayesian network model.The study presented a step-by-step process to display how to develop an effective diagnostic assessment model of matrix multiplication by using rule-based analytical model and the techniques of cognitive task analysis.The rules of matrix multiplication and relevant results of cognitive task analysis were integrated into the assessment element representation in the diagnostic assessment model with Bayesian network.The model had three exploratory variables which represent the learners' proficiency in solving matrix multiplication problems.The proficiency was represented in two aspects: performance and semantic explanations.The learners should be able to know matrix multiplication rules and apply these rules in their own practice in solving matrix multiplication problems.Further the learners were able to explain the rationale of matrix multiplication rules and the relations of these rules.There were 9 evidential variables by which learner's score evidence could be updated through the evidential variables to upper variables in the Bayesian network assessment model-Performance and Semantic Explanation, and then to Matrix Multiplication.Statistically, this hierarchical diagnostic assessment model was non-linear and provided an effective assessment tool to measure and assess learners' performance and semantic explanation of Matrix Multiplication.
The diagnostic assessment model successfully assessed learners' progress, and it also effectively differentiated cognitive aspects into performance and semantic explanations.The non-linear hierarchical assessment model could report the mastery proficiency at macro cognitive level in Matrix Multiplication; and further the mastery proficiency at sub-cognitive levels in both Performance and Semantic Explanation.Sometimes both sub-cognitive proficiencies presented unbalance status even though the macro cognitive level did not indicate much difference in two or three different cases.In study, Performance indicated a very high level, and Semantic Explanation scored very low, or vice versa.
The hierarchical diagnostic assessment model for Matrix Multiplication also effectively differentiates proficiencies among different learners, and meantime, the model provides dynamic diagnostic information.It allows the learners to understand what and where they perform unskillfully.
Lastly, the hierarchical diagnostic assessment model for Matrix Multiplication indicates a robust learning progress trajectory which can be described as learning progress curve.From Table 1, we conclude that the learners should master at least 7 knowledge elements based on the rule-based analysis, thus, the learning becomes more robust.The development process of the hierarchical diagnostic assessment model for Matrix Multiplication can be transferred to other academic learning assessment domains.

Limitations
The hierarchical diagnostic assessment model for Matrix Multiplication was simulated with 9 learners' data.The conclusions were limitedly generalized to different sample groups.The model was also cognitive task structure specific.The exact models for different cognitive tasks are not expected.However, the matrix multiplication, as an example, presents an effective model for statistical and research method researchers to continue to explore valid learning and assessment model.

Figure 4 .
Figure 4. Joint probabilities of matrix multiplication, performance and semantic explanation

Figure 5 .
Figure 5. Matrix multiplication assessment model with updated evidences

Perfo nce on the of rij&sjk nce on the Sum sjk Product To Find Each t
because Bayesian network model can be multi-representations.At this level, it is only a hierarchally conceptual representation of the diagnostic assessment model of Matrix Multiplication.

Table 1 .
Updated probabilities of random evidence variables Table 2 records a simulation records that reflect 9 students' performance.The patterns of learners' scores are very interesting.Subjects 1, 2 and 3 show that their Performance scores very well, but not do well in Semantic Explanation.Subjects 4, 5 and 6's scores on Matrix Multiplication indicate that they score on Semantic Explanation very well, but do not do well in Performance.Subjects 7, 8 and 9 present balance patterns.Subject 7 receives balance achievement among both Performance and Semantic Explanation at lower level; Subject 8 receives balance score in both Performance and Semantic Explanation with intermedia scores; and subject 9 receives highest score in both Performance and Semantic Explanation.