How Well Do Teachers Predict Students’ Actions in Solving an Ill-Defined Problem in STEM Education: A Solution Using Sequential Pattern Mining

Predicting students’ line of actions helps educators give adequate guidance to students, but this remains a challenge in science, technology, engineering, and mathematics (STEM) education. Given this, there is a scarcity of related research that will help improve teachers’ prediction capabilities on students’ line of actions when tackling ill-defined problems (IDPs), as well as how emerging data mining techniques could contribute to such prediction. The present study aims to fill the gap by measuring the quality of teachers’ predictions (labeled expert prediction), where 43 elementary teachers predict students’ step-by-step actions when solving an IDP through the light path task (LPT), and then comparing its quality with that of machine prediction, executed via sequential pattern mining techniques. Data on students’ lines of action were collected from 501 5th- and 6th-grade students, aged 11–12. The results showed the significantly lower accuracy of expert prediction compared to machine prediction, which highlights the advantages of using data mining in predicting students’ actions and shows its possible application as a recommendation system to provide adaptive guidance in future STEM education.


I. INTRODUCTION
Teachers require a good grasp of what students already know and their corresponding actions to design more effective teaching methods [1], [2]. Ensuring accurate predictions remains difficult, however, because teachers would need to identify, interpret, and respond to the knowledge level of dozens of students, as well as have sufficient awareness of classroom-related events [3], [4]. In science, technology, engineering, and mathematics (STEM) education, particularly, students are required to solve well-defined and ill-defined problems (IDPs), with the former having a clearly defined goal/routine/solution, while the latter has multiple possible goals/routines/solutions [5], [6]. This setup makes understanding students' thinking and possible actions during problem-solving processes even more problematic [7].
The associate editor coordinating the review of this manuscript and approving it for publication was Haruna Chiroma .
To discover information that would be difficult or impossible to analyze manually, data mining techniques have been widely adopted in many fields [8]. In education, they are used to aid teachers' understanding of students' learning as well as recommend instructional materials and teaching methods [9]- [12]. However, related literature on how data mining techniques support teachers' prediction of students' actions or provisions on adaptive guidance is scarce. To fill that gap, the present study developed a machine prediction model wherein a STEM problem, labeled light path task (LPT), was designed to collect students' step-by-step actions in ill-defined problem (IDP)-solving and their level of completion. The data were then further analyzed using sequential pattern mining (SPM). On the other hand, expert prediction (or a teacher's prediction) regarding students' progressive actions can be compared with that of machine prediction, and, thus, allows comparative performance to be evaluated.
The present study aims to highlight the usefulness of machine prediction as a teaching guide and contribute to the development and implementation of data mining technology in the field of education by answering the research question (RQ): How accurate are expert prediction (teachers' prediction) and machine prediction regarding students' step-by-step actions in ill-defined problem-solving?

II. THEORETICAL BACKGROUND A. KNOWING STUDENTS' THINKING AND ACTIONS
High familiarity with students and their capacities is considered necessary in guaranteeing teaching effectiveness, as pointed out by many veteran psychologists and educators in their educational practices [13]. However, the long history of related research has also implicitly shown how difficult it is for teachers to predict their students' actions in problemsolving, especially with IDPs in STEM.
Educators and educational psychologists, such as John Dewey, Jean Piaget, and Lev Vygotsky, suggested how learners' prior experiences were fundamental to build continuous learning, as supported by the empirical research of several others [1], [2]. However, different individuals have varying intellectual levels, mental schemas, and learning experiences, which makes it challenging to know the learning status of each student fully. In STEM education, for example, students are frequently asked to explore IDPs-problems without a clearly defined goal, a single routine, or an absolute solution [5]-which makes teachers' understanding of individual thinking even more problematic. Many studies have indicated the complexity of evaluating students' thinking on an individual level [3]- [4], [7]; hence, it is crucial to evaluate the extent of teachers' knowledge of their students' thought processes and actions and further tackle the challenges faced by teachers in understanding their students.

B. WELL-DEFINED AND ILL-DEFINED PROBLEMS (IDPs) IN STEM
Most problems discussed in STEM education do not stop with absolute solutions and would require nonlinear and more complex solving methods. Much effort has been extended to teach students how to solve both well-defined problems and IDPs. Well-defined problems, such as ''What is the ratio of the volume of oxygen to that of hydrogen when water is electrolyzed?'' or ''What is the distance a ball travels in 1 s after free release?'', are not new to STEM teachers as these have clear goals for which formulated methods can be used as solutions. However, questions that lack a clear goal, specific investigation boundaries, and solving methods are referred to as IDPs, which are becoming more common in STEM education. Lynch et al. [5] defined IDPs as problems with no definitive answer and heavily dependent upon their conception and relevant concepts at hand, giving field-specific questions related to swallows as examples, such as ''What is the airspeed velocity of an unladen swallow?'' (physics) and ''Design a residential building with housing for swallows'' (architecture).
Le et al. [7] proposed that problems could be seen at the continuum between well-defined problems and IDPs. To better categorize a problem, the researchers divided questions into five levels based on the solution's complexity, strategic diversity, and ease of verification regarding solution correctness. This categorization provided a framework for a better understanding of IDPs in STEM education. Furthermore, STEM problems have ill-defined attributes to some extent. Because of the elastic nature of an IDP, educators have frequently used two strategies, namely expert review and peer review/collaboration, to give feedback to learners who have tackled the problem [5]. Sensibaugh et al. [14] used ill-defined biochemistry cases for students' group learning by holding small group discussions online facilitated by an instructor or a teaching assistant via students' asynchronous discussion boards. The instructor or teaching assistant in the study also monitored the discussion boards and guided students' problem-solving. Shared knowledge from an expert and a small group of peers was also notably used in addition to computer-based instruction and evaluation. Furthermore, these studies revealed that information sharing with and learning from others are the main concepts for instruction in tackling IDPs.

C. SPATIAL ABILITY IN STEM
In an increasingly competitive technological society, STEM professions require spatial ability as a vital part of their work, ranging from electric circuitry design and mechanical engineering to aeronautics, among others. Tasks requiring spatial ability that STEM professionals encounter are mostly IDPs. Therefore, students' spatial abilities are vital in STEM-related learning, such as physics, chemistry, biology, engineering, technical aptitude, and design, as well as geography, arts, and sports. In K-12 STEM education, solving problems using spatial cognition and reasoning is regarded as an essential part of a curriculum [6].
Previous studies have shown how spatial ability influenced a STEM student's capacity to solve problems. A large-scale study [15] supported how students' spatial ability affected their advanced learning in STEM-a finding also advocated by other psychologists. Another study [16] using the five spatial skill-related tests developed by the Office of Naval Research [17] found that although there were existing instruments, there were no spatial tests that satisfied the following requirements: (a) of an ill-defined nature that elementary students could solve in a limited time, (b) allowed observations on how students continue the task of problem-solving, and (c) enabled measurement of students' task performance. To address this void in the literature, the researchers designed a two-dimensional light path problem that satisfied the requirements above, in which students can have different approaches to reach one goal. Using the students' gathered data could help examine how well teachers know about students' thinking and actions in IDP-solving. VOLUME 8, 2020

D. ANALYZING SEQUENTIAL PATTERNS OF STUDENTS' PROBLEM-SOLVING
Given the number of students in a classroom and the complexity of instructional content, it is difficult to identify, interpret, and respond to students' thinking on an individual level and provide adaptive guidance accordingly. In particular, the more ill-defined attributes there are, the more complex students' actions can be. Moreover, when the scope of the problem expands and the solving methods diversify, the standard of operations becomes nonexistent. Thus, the rapid development of computer technology has been considered an opportunity to support teachers in the aspect of knowing students and predicting their actions.
With this, Wang [18] observed that when students were allowed to share and access information with peers, their levels of self-regulated learning and e-learning effectiveness were enhanced, and this was supported by several other studies [14], [19]. Thus, sharing or using collective knowledge was recognized as a promising approach for helping students solve IDPs. For this, many techniques in the computer science field have been adopted to organize collective knowledge for use in instruction.
Data mining techniques, especially clustering, classification, regression, association rule mining, and SPM, have been widely used in the industrial, medical, and business fields [8], [20] and have now received more attention in the educational field [9]- [12]. Le et al. [7] stressed that educational data mining techniques have the potential for use in IDP instruction. Using the experiment of Fournier-Viger et al. [21] as an example, learners operated RomanTutor, a tutoring system, to simulate how astronauts use a robotic arm in an international space station. Data mining techniques, including SPM, were used to support and guide learners to tackle IDPs, and their study's results supported the potential of using machine prediction in educational assistance.
SPM is designed to reveal patterns in a sequence by measuring the frequency of how sets of items appear in a given data set (e.g., analyzing a data set of students' procedures while solving a STEM problem). For instance, using the ''A → B → C'' sequence in solving problems may be more frequent in a set compared to using the ''C → B → A'' sequence, and so on, depending on the number of patterns that emerge. Given the complexity of studying various patterns, several SPM algorithms have been developed for different contexts. Febrer-Hernández and Hernández-Palancar [22] presented and introduced a major a priori algorithm that addressed the variance in contexts, including Generalized Sequential Pattern (GSP), which is more commonly used, and Sequential Pattern Discovery using Equivalence classes (SPADE), as well as Constraint-based Apriori Algorithm for Mining Long Sequences (CAMLS), among others. Fournier et al. [23] reported how a wide selection of open-source software could be used for SPM purposes and identified another variation of a typical SPM technique that included mining frequent partial orders, which could complement SPM. Considering the data set size, objective, and availability, the researchers selected Weka ver. 3.8 [24], which uses the GSP algorithm as its mining tool.

E. APPLICATIONS OF SEQUENTIAL PATTERN MINING IN STEM
Wang et al. [25] used lag sequential analysis [26] to explore university students' collaborative learning behaviors in different learning environments and successfully identified and demonstrated that students exhibit different behavior patterns in diverse learning environments. Kucuk and Sisman [27] also used lag sequential pattern techniques to reveal differences between teachers' and students' behavioral patterns in one-to-one robotics instruction processes. On the other hand, Kinnebrew et al. [28] used SPM to detect students' learning behavior patterns in a computer-based learning environment wherein a STEM topic (i.e., global climate change) was selected for grade 8 students to explore. This was done through a piecewise linear segmentation algorithm with the differential sequence mining technique. After coding students' behaviors into categories (i.e., reading, editing, querying, explaining, quizzing, etc.), the researchers identified productive and unproductive learning behaviors by comparing those of less and more successful students.
Identifying differences in behavior within the spectrum of less and more successful students continued in a study done by Chang et al. [29], wherein a web-based collaborative simulation was developed to allow students to collaboratively solve a problem for which they needed to apply kinematics principles and tune variables. After gathering the simulation results, the researchers carried out a lag sequential analysis to explore the students' activities. Their results revealed that less and more successful students observed different sequential patterns as ''the discussion of 'monitoring & reflecting' was also linked to 'exploring & understanding,' which was further linked to the problem-solving activity 'browsing problem''' [29, p. 230]. From this, they concluded that successful students tended to apply analytical reasoning strategies. Meanwhile, Sung and Kelley [12] revealed students' problem-solving patterns by analyzing the sequential patterns of iterative design processes when working on a STEM learning activity of designing a doggie door alarm. They then identified significant sequential patterns and a problem-solving pathway model that includes six design process elements. All these studies showed how data regarding students' behaviors in STEM activities can be extracted with adequate SPM techniques.
In applying such techniques in STEM education, Perera et al. [30] utilized SPM to reveal patterns in less and more successful students' performances using students' online group work data to facilitate their learning of software development. Their study indicated that the SPM of stronger groups was promising for providing advice to their peers' learning. Chiu and Lin [31] used an innovative approach to explore students' concept map construction by developing a concept-mapping platform that could trace and record students' mapping activities. In this study, students' mapping sequences were analyzed, and sequential patterns in the stepby-step processes of less and more successful students were compared. Their results showed that students who had superior performance on concept map construction demonstrated similar sequences, while no similarities were found among students who had inferior construction performance. These studies support the present study's claim that the sequential patterns of more successful students could be identified and have the potential to provide useful information for instruction.
The present study's researchers noted that previous studies were more focused on discovering students' behavioral patterns or measuring the effect of using SPM in learning. Studies rarely investigated empirically and compared how well teachers and machines predict students' actions when solving IDPs in STEM-related activities. It was also just as rare to find a study that centered on how such identified patterns can be used to predict individuals' next actions. Therefore, the present study may not only shed light on the extent of expert and machine predictions but also enhance the applicability of using machine recommendations to guide students in solving IDPs adaptively.

A. PARTICIPANTS
In the present study, the researchers asked 501 grade 5 and 6 students (aged 11-12) from 7 elementary schools in Taiwan to solve the two-dimensional LPT designed specifically by the researchers. After excluding those who did not have any correct answers (5 students) and those who had only 1 correct answer (35 students), the data of only 461 students were used. The criterion was set because only when students had more than two correct answers did their sequential patterns become useful for SPM. To measure machine prediction, the data were divided into test and training sets (further explained in the Data Analysis section). Stratified sampling of 10.4% (48 students) of each score level was used for the test set, while the remaining 89.6% (413 students) was for the training set. Then, the researchers adopted the Pareto principle, also represented as the 80/20 rule or the law of the vital few [32], and used the answers of the approximate top 20% of more successful students to establish the sequential pattern rule set. To be specific, 82 (19.9%) of the 413 students correctly answered 6 or 7 light paths, with a total of 532 light paths included in the training set. The performance of machine prediction, which was based on the sequential pattern rule set established from the training set, was then measured.
To evaluate expert prediction performance, the researchers chose the sample of teachers based on their educational and professional backgrounds. Of 43 elementary teachers, 18 took up master's studies in science education in a major university, while the remaining 25 were in-service teachers who had experiences in teaching STEM-related courses in an elementary school setting (14 of whom had master's degrees and 11 with bachelor's degrees for varying majors). These teachers have prior knowledge in teaching and understanding the interdisciplinary nature of problem-solving. However, terms, such as well-defined and IDPs, have not been emphasized in their preparation programs. Hence, the researchers designed a task that will allow teachers to make predictions without interference from their knowledge of IDP-solving. In the experiment, these teachers were asked to predict students' step-by-step actions based on real data in the test set. However, the test set's size was too large and beyond the capacity of human experts to complete the prediction. Thus, the present study randomly selected 30% from the above test set (n = 16 students) and used these students' answers and sequences as test contents for the teachers to predict. Our pilot test found that teachers needed about 40 min to complete the prediction task, which can be considered too long for a student exercise. As a result, the test contents were divided into two equivalent parts (Forms A and B) from which teachers can select randomly, reducing the test time to 20 min. Forms A and B were used by 19 and 24 teachers, respectively.

B. LIGHT PATH TASK (LPT)
The researchers designed a two-dimensional LPT that shows a laser light beam target experiment (Fig. 1). This 20 min long LPT included the following instructions: ''There are many ways to use and guide a laser light beam to reach point B from point A. You can only use the minimum number of mirrors to complete each light path. Given this, please draw as many light paths as you can.'' All seven possible light paths are shown in Fig. 2. It is worth noting that this LPT's design emphasis was to create an IDP task with multiple solutions that have similar cognitive loads so that students' problem-solving sequences mainly reflect their preferences and not the relative difficulties of each solution. This will yield meaningful sequential patterns and produce more reliable results. With many research design restrictions, this light task is more likely set as a research instrument than an instructional instrument. However, the task can still be used as the students' exploration activity where they can apply their STEM knowledge (e.g., light travels in a straight line and reflection of light), spatial ability, and/or skills (e.g., planning and causal relationship) in the elementary school level.

C. INSTRUMENTS FOR EXPERT PREDICTION AND MACHINE PREDICTION
For collecting data from expert prediction, the present study developed a program (Fig. 3) for teachers to input their predictions, which were randomly selected from the test set, based on students' actions. For each prediction, teachers were asked to give their best and second-best predictions regarding students' possible corresponding actions. Moreover, the current study designed a machine prediction model (Fig. 4) that is composed of the following elements: a training set, a test set (which has been introduced in the Participants section), the SPM algorithm for creating the rule set, and the machine prediction algorithm to produce machine recommendations. The following two sections explain the study's machine prediction model.

D. SEQUENTIAL PATTERN MINING ALGORITHM AND RULE SET
To discover students' sequential patterns as they work on the LPT, the researchers coded students' sequence of drawing correct light paths and the total number of correct paths. Students' records were then analyzed with SPM and used to create the rule set. To establish the rule set of sequential patterns, the present study used Weka ver. 3.8 as a mining tool. Furthermore, all calculations were executed using a computer that has a 1.8 GHz central processing unit (CPU). Using the GSP algorithm, Weka first discovered the frequent  sequences and then filtered those with a user-selected minimum support level (the default value is at 10% but can be changed depending on the purpose of its use). Considering the large diversity of the students' sequences of drawing paths, the researchers set the minimum support level at 5%. The rule set in the present study showed a total of 462 frequent sequential patterns, which represented 532 paths from the 82 students who were more successful in the task and had 6-7 correct answers. In the 462 frequent sequential patterns, the numbers of each length of the sequence from 1 to 5 were 7, 42, 206, 205, and 2, respectively. A sample of the sequential patterns in the rule set is shown in Fig. 5. It can be noted that the sequential patterns with lengths 6 and 7 did not reach the minimum support threshold of 5%, which called for the need to establish a machine prediction algorithm that can address this problem and continue predictions for longer sequences.

E. MACHINE PREDICTION ALGORITHM AND MACHINE RECOMMENDATIONS
The machine prediction algorithm was presented as follows: Algorithm Machine Prediction Algorithm 1. Input 2. A student's light path record is in the Test Set.
3. All rules are in the Sequential Rule Set. 4. Output 5. Predictions on this student's next action: 6.
1. The best prediction and 2. The second-best prediction 7. Begin 8. INPUT a student's record in the Test Set: SequenceOfStudentA 9.
if TestSequence matches rules in Rule Set that has the length of (LENGTH (TestSequence + 1)) then 12.
COMPARE above frequencies of all matched sequential rules 13.
REPORT the highest frequency as the best prediction and the second highest as the second-best prediction a 14.
else if // when two more predictions were needed from a lower sequential length 15.
FIND sequential rules in Rule Set with the next short length and started with RIGHT(TestSequence, i − 1) 16.
COMPARE above frequencies of all matched sequential rules 17.
REPORT the highest frequency as the best prediction and the second highest as the second-best prediction a 18.
else if // when only one more prediction was needed from lower sequential length 19.
REPORT the best prediction obtained from rules of longer sequential length and the second-best prediction from shorter sequential length a 20.
end for 22. End a In the case of a third roundup, an arbitrary one was selected.
The machine prediction algorithm in the present study used a student's light path record (line 2 of the algorithm), which is read from the test set (as shown in Fig. 4), and all sequential pattern rules (line 3), which are read from the rule set of sequential patterns, to output two predictions for his or her next action, including one best prediction and one second-best prediction (lines 4-6).
For example, the algorithm fetched a student's record sequence from the test set that has a path sequence of ''(C) → (F) → (B)'' (SequenceOfStudentA; line 8). Based on this student's record, which has a sequence length of 3, only two sets of predictions would be made for this evaluation. The first set was to make predictions based on the first known ''(C),'' and the second set was to make predictions based on the currently known ''(C) → (F).'' Here, we used the second set of predictions for the explanation, to which i = 2 and the variable of TestSequence equaled ''(C) → (F)'' (lines 9-10). Afterward, the algorithm was to produce two predictions, one best and one second-best, for each set of predictions (lines 11-21). These two predictions would be compared with the student's actual drawing path, ''(B),'' to judge the quality of this machine recommendation model (Fig. 4) with the evaluation methods explained in the Data Analysis section. Continuing this example, the length of the current known TestSequence was 2. The algorithm's task, then, is to predict the student's third act. To fulfill this, the algorithm sought all three-series sequential patterns in the rule set starting with ''(C) → (F)'' (line 11), which were based on numbers 80-84 in Fig. 5. After comparing the five candidates' frequencies, the best prediction (G) and the second-best prediction (B), with the highest frequencies of 10 and 8, respectively, made up the final output (lines [12][13]. In this case, the student's drawing after ''(C) → (F)'' was ''(B).'' These results indicated that the machine prediction algorithm missed the answer with its first prediction but got it correctly with its second-best prediction.
As indicated by Fournier-Viger et al. [33], sequential pattern rules in the rule set may not be sufficient for many reasons and, thus, become unable to produce predictions. As such, the present study used a partially ordered approach by scanning the sequential pattern rules with the next shorter length in the rule set. For example, if our algorithm was unable to find the six-series rules to give two predictions for the TestSequence ''(A) → (B) → (C) → (D) → (E),'' the algorithm would neglect the leftmost item and treat it as ''(B) → (C) → (D) → (E)'' only. Consequently, the fiveseries rules in the rule set would be used to make further predictions (lines [14][15][16][17]. When only one prediction is lacking from the previous output (lines [11][12][13], the second-best prediction would be produced (lines [18][19] with the same partially ordered approach as explained above.

F. DATA ANALYSIS
Mean of precision (MP) and mean reciprocal ranking (MRR) were used as evaluation metrics of the prediction quality. Precision, which is defined as the ratio of correct VOLUME 8, 2020 predictions (which is a true positive (tp)) to the total predictions (which include true and false predictions (tp + fp)), is a well-established metric that has been widely used in many studies [34].
Precision(i) (1) where N refers to the number of predictions, and Precision(i) is the precision of the i-th prediction. For example, if a student's sequence is path A and then path B, and the expert or machine correctly predicts the second path, which is path B, then this prediction's precision is regarded as 1 (tp is 1, and fp is 0). Meanwhile, if the prediction is path C, which is an incorrect prediction, then the precision value would be 0. MP would be the mean of precisions of tries. MRR is also frequently used in evaluating the performance of recommendation systems [35], and, here, it measured the reciprocal of the highest-ranked correct answers.
where N is the number of predictions, rank(i) is the rank of the correct prediction of the i-th prediction. The prediction for each student's next drawing was set only to two in the present study, considering human capacity. Taking the previous student's case as an example (path A then path B), if the expert or machine errs in its first attempt and only predicts correctly in the second, then a 0.5 value is gained (0.5 = 0/1 + 1/2). MRR was calculated from all attempts. The researchers further applied the Mann-Whitney U test to compare the performance of expert and machine predictions. Finally, the MP's and MRR's effect sizes will be calculated [36].

IV. RESULTS AND DISCUSSION
The total number of predictions was 1,099 made by 43 teachers (Table 1). For each prediction of students' step-by-step actions, teachers gave two guesses, namely best and secondbest predictions, simultaneously. The result showed that of all expert predictions, there were 162 correct best predictions, 238 correct second-best predictions, and 699 incorrect predictions. The performance of expert predictions was expressed by the MP and MRR values. The MP of expert predictions was 14.8% (standard deviation (sd) = 6.7%), and the MRR was .256 (sd =.074). On the other hand, the machine predicted students' step-by-step actions 149 times, of which there were 68 correct best predictions, 42 correct secondbest predictions, and 39 incorrect predictions. The performance of machine prediction was as follows: MP = 45.6%, MRR = .597. These values were unlike those of expert prediction in which each value was obtained from many experts' predictions; hence, there were no sd values presented in Table 1.
The results of the two prediction approaches were analyzed with the Mann-Whitney U test, and the results of comparing the predictions' precisions with the reciprocal rankings are shown in Table 2. It revealed that there were statistically significant differences between the expert and machine predictions, both in terms of precision (z = −9. 124, p < .05) and reciprocal ranking (z = −9.718, p < .05). Moreover, the statistical tests' results indicated that the SPM-based machine prediction outperformed expert prediction with a value close to the medium effect sizes (es) for precision (es = .26) and reciprocal ranking (es = .28). The results shown in Tables 1 and 2 have demonstrated that machine prediction surpassed expert prediction both in MP and MRR and showed significant differences with the Mann-Whitney U Test. To closely observe all teacher's (expert) predictions, the predictions were further demonstrated individually and contrasted with machine prediction in Fig. 6, which shows that most teachers had an MP of around 10%-20% (mean = 14.8%, Table 1). This indicates that when a teacher is faced with a student who is working on an IDP like the LPT, the teacher could only have a 10%-20% possibility to predict the student's next action correctly. Most of the teachers have an MRR between 0.2 and 0.3 (mean =.256; Table 1).
In the context of the present study, the teacher gave both best and second-best predictions for each student's step-bystep actions. When a teacher has an MRR value of 1, it means that every student's next action was correctly guessed in the expert's best prediction. For the value of 0.5, the student's next action was correctly guessed in the second-best prediction on average. However, if it is 0, then none of the guesses were correct. Given this, the MRR results in the present study mostly ranged from 0.2 to 0.3 only, indicating that teachers were barely able to predict the students' next thinking despite being given two opportunities to make predictions. The low MP and MRR values also indicated the teachers' poor ability to predict students' actions in STEM education. As mentioned by Barnhart and Van Es [3], it was difficult for teachers to see the details of students' thinking and actions during their learning activity. With the increasing presence of IDPs in STEM education, which has raised the teaching difficulty level, the present study's results and methods can help address problems commonly encountered by teachers.
To look into more detail about how well each teacher's prediction compares to machine prediction, this study illustrated every teacher's prediction performance in terms of MP and MRR in contrast with that of machine prediction as shown in Fig. 6. Interestingly, 1 of the 43 teachers had exceptionally high prediction performance (no. of teachers = 24; MP = 38%; MRR = 0.54), which indicated that some teachers had a better understanding of their students' actions than their colleagues. Further exploration of the reasons behind this was not done, but this phenomenon can be a topic for future research to improve teachers' performance. Two horizontal lines were shown in Fig. 6 to represent machine prediction performance (MP = 45.6%; MRR =.597; see data in Table 1), which is seen as much higher among all cases of expert prediction. Adding this to the statistical results reported earlier, the performance of machine prediction using SPM to analyze students' collective experiences demonstrated its power and potential to predict students' actions. With this unfavorable result regarding the teachers' prediction performance, some might suspect that this could result from the low preparedness of teachers. However, that cause is unlikely as all elementary teachers in the present study have at least four-year college degrees (with some having master's degrees) and an elementary school teaching certificate. As elementary teacher qualifications and student performance in international assessments in Taiwan have been examined and considered as high standard [37], [38], it was reasonable to infer that the lack of competency in predicting students' thinking was a common and challenging subject that deserves attention worldwide. Considering the results from Table 1, Table 2, and Fig. 6, the present study had not only answered the RQ by displaying the superiority of machine prediction but also revealed the noteworthy phenomenon wherein machine prediction outperformed all predictions made by teachers during the experiment.
These empirical results echoed Le et al. [7], who stated that data mining techniques were promising for instructing higher-level IDPs. Moreover, in addition to existing studies that used data mining techniques to increase knowledge and enhance students' STEM learning [9]- [12], [18], [21], [25], [27], [29], [31], the present study extended knowledge in the field by demonstrating machine prediction's capacity. All ''next actions'' recommended by the machine prediction model were based on the collective knowledge of more successful students, which revealed that the Pareto principle [32] could be used to support the predictions.
Predictions based on the collective knowledge of more successful students made in this machine prediction model can be used in guiding those less successful students throughout their problem-solving processes when tackling an IDP. More specifically, for example, many previous studies have reported that students' problem-solving processes in STEM education all have sequential patterns [12], [29], [39]. These have allowed the researchers to claim that by incorporating the present study's machine prediction model, students' problem-solving learning, which was mentioned in the studies above (e.g., ''Design a Doggie Door Alarm'' activity [12]) and other STEM problem-solving activities (e.g., ''Interactive Problem Solving Questions'' [40], ''Interactive Simulations for Science and Math'' [41]), can be supported by machine-generated recommendations based on SPM. However, the present study has not emphasized and considered the time complexity of the algorithm because the empirical tests have shown that the bottleneck of the machine prediction model lies in establishing a sequential pattern rule set from a matrix with a sequential length of 7 and 501 participating students that can be accomplished in less than a second. In the most exemplary learning activities in K-12 STEM education [12], [40], [41], IDPs have similar sequential lengths and their rule sets can be established from the same number of students as that of the present study. As such, this enabled the researchers to assert that the present study is vital and applicable for guiding students who are learning about IDPs in STEM education. However, it should also be noted that there are many other machine learning techniques [22], [23], [26] that can be utilized to enhance executive performance when needed.
From the perspectives of pedagogical theories, high machine prediction accuracy has implications in improving teaching and learning. Its capability to detect the differences between sequential patterns while solving IDPs between students and their more successful counterparts enabled the researchers to achieve what was emphasized in pedagogical theories-to ascertain what students already know and then teach them accordingly [42]. In addition, the implications that VOLUME 8, 2020 gave timely and adaptive guidance based on collective knowledge constructed from more successful IDP solvers aligned with Vygotsky's theory of the zone of proximate development and scaffolding [43, p. 86], which highlighted the importance of accurately providing the level of potential development zone-a set of skills or knowledge that a student cannot do by himself or herself without guidance. These studies have pointed out that the machine prediction model developed in the present study was not only capable of predicting students' next actions in solving IDPs but also has the potential to transform into a recommendation system compatible with major pedagogical theories. This can substantially change the traditional approach of teaching IDP-solving, which has the disadvantage of relying heavily on human interactions [5], making it time-consuming, ineffective, inefficient, and highly unreliable.

V. CONCLUSION
Philosophical and psychological views have identified the importance of knowing students' thinking and behaviors for successful subsequent instruction. The present study's results revealed that expert prediction on students' progress in solving an IDP performed significantly weaker than machine prediction, which means that the prediction of students' thinking and actions were frequently beyond the teachers' capacities. To conduct machine prediction, the researchers used SPM aided with a partial sequential algorithm, which raised the mean of precisions (MP) and the mean of reciprocal ranking (MRR) to 45.6% and.597, respectively, and reached statistical significance. The results concluded that machine prediction outperformed expert prediction in terms of knowing students' next actions and could be a useful approach to enhance the instruction quality in STEM education. This shows how the present study's methodology and results can be extended and applied in various problem-solving activities in STEM. However, some limitations should be considered regarding the present study and its results. First, although teachers involved were not outliers, caution is needed when making inferences to other regions. A bigger population sample for future studies should be helpful. Second, although the machine prediction model used can be extended in predicting more complicated well-defined problems or IDPs and has the potential for further application to higher school levels, the generalization should be scrutinized. Third, although the researchers have found that both MP and MRR of machine prediction have reached relatively higher levels than any teacher's prediction in this study, they have also noted that a disadvantage of the GSP algorithm is that it makes multiple database passes and generates a large set of candidate sequences. As explained previously, while the SPM-based prediction method in this study was able to meet most needs in supporting students' learning activities of solving IDPs in K-12 STEM education, the executive time performance for different complexities was not measured.
As such, further studies should explore the distribution and types of IDPs that would exceed the capacity of this method or focus on fine-tuning data mining and machine prediction using revised and improved algorithms, such as those used in other studies [25], [27], [44], or the combined machine technique [45] to elevate computational and prediction performance.
The rapid development of data mining techniques creates the advantage of continuously providing new possibilities for assisting teachers in discovering students' learning behaviors and needs and guiding students in making timely and adequate dynamic decisions [46]. Although the research subjects of the present study and many other studies [12] were elementary students and elementary-level contents [29], [39], it can be inferred that similar sequential patterns of more and less successful students can also be expected for higher school levels based on the findings of previous studies. Therefore, there is a high likelihood that this machine prediction approach can be extended and used in higher school levels to enhance the teaching and learning qualities of future STEM education.
For further research, a study to compare educators' teaching methods with or without machine prediction support will establish a technologically supported environment that may better arm teachers with information that can help inform their instructional decisions in STEM education. The research findings will shed light on how machine prediction can be implemented in STEM classrooms. In addition, the expansion of the current machine prediction model into a machine recommendation model may also be a topic for further studies to provide adaptive and immediate guidance on online problem-solving activities. The researchers believe that these will prove useful in online STEM education development. Finally, further studies that explore different degrees of IDPs and/or complexities of teaching content that affect MP and expert prediction performance are needed to bolster the results of the present study. The future findings of such studies will add to the gaps in knowledge in this particular field and will be vital for the further implementation of machine prediction and recommendation to enhance student learning in STEM education. She served as a science teacher with junior and senior high schools. She is currently a Professor of science education with the National Taipei University of Education, Taipei City. She has published articles and instructional materials such as books and K-12 science-specific courseware, which explore science curriculum development, instructional design, and technology-enhanced learning.
Dr. Lu was the chairperson of Taiwan's Association of Science Education, a research group devoted to promoting the quality of science education research, in 2018 and 2019.