Linear Support Vector Machines for Prediction of Student Performance in School-Based Education

Educational Data Mining (EDM) is a rich research ﬁeld in computer science. Tools and techniques in EDM are useful to predict student performance which gives practitioners useful insights to develop appropriate intervention strategies to improve pass rates and increase retention. The performance of the state-of-the-art machine learning classiﬁers is very much dependent on the task at hand. Investigating support vector machines has been used extensively in classiﬁcation problems; however, the extant of literature shows a gap in the application of linear support vector machines as a predictor of student performance. The aim of this study was to compare the performance of linear support vector machines with the performance of the state-of-the-art classical machine learning algorithms in order to determine the algorithm that would improve prediction of student performance. In this quantitative study, an experimental research design was used. Experiments were set up using feature selection on a publicly available dataset of 1000 alpha-numeric student records. Linear support vector machines benchmarked with ten categorical machine learning algorithms showed superior performance in predicting student performance. The results of this research showed that features like race, gender, and lunch inﬂuence performance in mathematics whilst access to lunch was the primary factor which inﬂuences reading and writing performance.


Introduction
Predicting the factors impacting student performance early in the academic programme can assist in combating the high dropout rate experienced by higher education institutions. Predicting factors can guide lecturers adjusting their lessons in order to assist students who are at risk of failing. Several studies have been conducted on using machine learning algorithms for early prediction of student performance. ese studies fall into the field of Educational Data Mining (EDM) which provides great value to educational institutions [1]. EDM is an emerging field in data mining and lies in the intersection between education, computer science, and statistics [1]. Not only is Educational Data Mining limited to education but also it covers fields like transportation, sports, marketing, and sales [2]. ere are several data mining tools used in EDM to analyse and predict student performance to the benefit of educational institutions. ese interventions improved pass rates, curbed dropout rates, and increased retention rates [3].
ere are several tools that are used in EDM. Data manipulation and feature engineering tools include Microsoft Excel, EDM Workbench, Python and Jupyter notebook, and Structured Query Language (SQL). No one tool can be used for EDM as different tools suit different tasks [4].
A wide range of classification algorithms can be used to predict processes and performance, namely, random forest, support vector machines, AdaBoost, decision tree, Naïve Bayes, and K-nearest neighbour [3]. Kumar et al. [2] used EDM effectively in research to improve retention rates by predicting slow learners' in a high school class and providing them with intervention to improve. ey found Naïve Bayes, Multilayer Perceptron, SMO, J48, and REPtree to be the most widely used techniques in EDM research. is paper will explore other techniques in the field of EDM.
is research paper will attempt to answer the following research question in the field of EDM: What are the strong impacting factors for school based learners' performance in reading, writing, and mathematics? e paper is organized as follows: In the next section, a review of related literature is presented, Section 3 introduces the materials and methods used in this research, followed by Section 4 that provides the applied data mining algorithms on the selected dataset and presentation of experimental results and, finally, the paper ends with the section that concludes the work.

Related Works
Acharya and Sinha in [5] showed, in a study of students majoring in computer science, the best results for early prediction was obtained with the decision tree class of algorithms. In this study, the training consisted of 309 instances whilst the testing comprised 104 instances. e study found that caste and religion played a vital role in student performance in India. Other strong factors related to academic performance were the family size and family income.
In another study by Koutina and Kermanidis in [6], early prediction of postgraduate master's students was useful to assist tutors. Data on a total of 117 instances were collected from three courses, namely, Advanced Language Technology, Computer Networks, and Information Systems Management. Naïve Bayes and 1-NN achieved the best prediction results compared to well-known classification algorithms. e results show that, in small datasets, Naïve Bayes and 1-NN can perform better than decision trees [6]. Support for better performance on Naïve Bayes on smaller datasets was also shown in a study by Belachew and Gobena in [7]. e study found that the size of the dataset and the imbalance and distribution of class values are the main challenges in work obtaining better accuracies. In this study, predictors of student performance were the presence in class and in-term performance. It was also found that student's occupation, the type of degree, and their possession of another master's degree were not good indicators for early prediction of student performance.
In a study by Kotsiantis et al. [8] on the performance of distance learning, students showed that Naïve Bayes was able to produce more than satisfactory accuracy when compared to other state-of-the-art algorithms. Two experiments were conducted using datasets of 354 and 28 instances. e results showed that 28 instances were too few and the number of instances needed for better accuracy was 70 instances.
El Aissaoui et al. [9] implemented an adaptive e-learning system to take into account learning styles so convenient content can be provided to enhance learning.
is study automatically detected learning styles using machine learning. K-means clustering was used to extract learning styles from login sequences. ereafter, using a supervised learning algorithm, namely, Naïve Bayes, the learning style for a new learner could be predicted. A real dataset extracted from the e-learning system's log file was used in the experiments. In a study, Xu et al. [10] used machine learning for tracking student performance in a degree programme. Students' future performance in degree programmes based on ongoing information could be predicted. e following predictors were cited in the study, namely, backgrounds of the students, the courses that were selected by students, the information provided by the courses to predict future performance, and the progress made by the students. is study asserts that predicting students' performance in degree programmes is not a one-off task but it is a progressive process of tracking and updating. ey developed a novel algorithm to predict students' performance based on a bilayered structure comprising a base predictor layer and an ensemble predictor layer. Base predictors are trained using offline data (past performance) and ensemble predictors are trained using online student data (current performance). e dataset had 1169 anonymous undergraduate students. e data contained student's precollege information (high school GPA and SAT), courses that students take each quarter, the course credits, and the obtained grades. e study concluded that the proposed method achieved superior performance over benchmarked approaches. Mao et al. in [11] used Bayesian Knowledge Tracing and Long Short-Term Memory (LSTM) to predict posttest scores and learning gains prediction. Two training datasets namely, Pyrenees and Cordillera were employed. e findings showed that the learning environment can predict students' performance and learning gains.
Cui et al. in [12] investigated student performance prediction by assessing the possibility of a common set of student activity variables that predict performance, the machine learning classifiers that perform well across different courses model that could be used to predict performance based on LMS data. ere were 18 features extracted from data processing on the LMS activity. e recursive feature elimination (RFE) algorithm was employed to select the most important features. e features were ranked and the top 5 features were selected. In the experiment, 8 classical machine learning algorithms were used, namely, logistic regression (LR), Naïve Bayes (NB), neural network (NN), support vector machine (SVM), decision tree (DT), knearest neighbours (kNN), random forest (RF), Gradient boosting machine (GBM), and 1 ensemble classifier, namely, ensemble model (EM). e results showed that the usability of the LMS could predict student performance and can provide feedback to support students and higher education institutions.
Gray and Perkins in [13] showed that students "At Risk" can be identified as early as week three of the semester with approximately 95% accuracy. e dataset consisted of 4970 students, 32 features, and 5 classes, namely, pass, supplementary, repeat year, fail, and repeat semester. Zohair in [14] explored identifying key features using a small dataset using clustering algorithms. Records of 50 students were broken up into 2 small datasets. e classifiers K-nearest neighbour (KNN), Naïve Bayes (NB), LDA, support vector machines (SVM), and MLP-ANN were used for the training. A comparative analysis showed which algorithm was superior when dealing with prediction using small datasets. e study concluded that support vector machines produced the most acceptable results on small datasets.
Linear support vector machine (LSVM) is a variant of support vector machine (SVM) which is one of the most popular supervised machine learning methods. It has been proven in the literature that LSVM can guarantee global optimization for regression or classification problems in small-to-large datasets [15,16] and deals with predictive binary classification, that is, the assignment of class labels to unlabeled data [17]. Based on the benefits LSVM proffers, this paper adopted LSVM for the statistical analysis of student performance. e aim of this study is to compare the performance of linear support vector machines with state-of-the-art classical machine learning algorithms in order to determine the algorithm that would improve prediction of student performance.

Materials and Methods
e experiments were conducted on a computer running Windows 10 operating system with the configuration of Intel ® Core ™ i5-8250U CPU @ 1.60 GHz (8 CPUs), 1.8 GHz, 8 GB RAM memory, and 500 gigabytes hard disk drive. e dataset used in this study was selected from a set of categorical data obtained from a public domain referred to as the Kaggle data and available online (https://www. kaggle.com/spscientist/students-performance-in-exams/ activity). e database consists of 1000 high school learner records based on student performance in mathematics, reading, and writing. No preprocessing of the data was required. e features of the dataset include the following: (1) gender, (2) race/ethnicity, (3) parental level of education, (4) access to lunch, (5) test preparation, (6) mathematics score, (7) reading score, and (8) writing score.
A machine learning approach was adopted to establish the contributing factors that impact student performance. e training dataset comprised of all 1000 student data and thereafter the testing dataset comprised of 1000 student data for the Kaggle data. In the training stage, the classification rules were adopted. e testing stage is used to test the accuracy of the classification rules [5]. All results are provided for the testing of the dataset. e machine learning library in MATLAB R2020a was used to implement algorithms and obtain statistical results. Figure 1 shows the architecture of the student performance testing used in this study.

Classification.
A popular machine learning algorithm, the linear support vector machine (LSVM) classifier, was used as a supervised learning model to analyse data for classification. SVM is one of the most actively developed classifications and regression methodologies in data mining and machine learning. It provides salient properties such as the margin maximization and nonlinear classification via kernel tricks and has proven to be effective in many realworld applications [18].
According to Foody and Mathur in [19] a linear support vector machine is often able to classify a dataset to a higher accuracy than conventional statistical classifiers. Based on the advantage that the linear support vector machine classifier algorithm provides, we used the algorithm to classify and predict factors that impact student performance [20,21]. e linear SVM classifier was benchmarked with ten other algorithms such as coarse decision tree, medium decision tree, fine decision tree, logistic regression, Gaussian Naive Bayes, Kernel Naive Bayes, quadratic SVM, cubic SVM, fine Gaussian SVM, and medium Gaussian SVM.

Performance Measures.
e standard performance metrics, namely, accuracy, total misclassification cost (TMC), prediction speed (PS), training time (TT), and area under Receiver Operating Characteristics (ROC) were calculated in the experimental comparison of classifiers [22]. A parallel coordinate plot was used to visualise the multivariate data. A parallel coordinate plot maps each row in the data table as a line, or profile, and allows for the visualisation of data points across many dimensions.

Results and Discussion
e LSVM was used to analyse the performance by extracting useful knowledge from the student dataset. e usefulness of the algorithm was to interpret relationships among variables and to determine factors that affect student performances in their mathematics, reading, and writing score. Table 1 presents the frequency and percentage of all variables for a student population of n � 1000. e summary data from Table 1 was trained, using linear SVM 5-fold cross-validation, a preprocessing operation for classification. e trained data was analysed using linear SVM (LSVM) and benchmarked with fine decision tree (FDT), coarse decision tree (CDT), medium decision tree (MDT), logistic regression (LR), Gaussian Naive Bayes (GNB), Kernel Naive Bayes (KNB), quadratic SVM (QSVM), cubic SVM (CSVM), and fine Gaussian SVM (FGSVM) and medium Gaussian SVM (MGSVM). Table 2 shows the results for all the benchmarked algorithms for the area under ROC curve (AUC), accuracy, total misclassification cost (TMC), prediction speed (PS), and training time (TT). Table 2 shows that LSVM has the highest accuracy of 90.1% and appositely the lowest total misclassification cost of 99. According to time metrics, LSVM was outperformed by other classifiers. GNB boasts the best performance for prediction speed while CDT has the fastest training time.

Input
Output Feature selection Classification Figure 1: e architecture of the student performance testing model using learning algorithms. Figure 2 illustrates the parallel coordinate plot with five dimensions represented by N � 5 (race/ethnicity, parental level of education, lunch, test preparation, and math score) such that every vertical line, that is, factors impacting the mathematics score, appears exactly once. e standard deviation is 65% shown as  Mathematical Problems in Engineering point 0 on the parallel coordinate plot. e standard deviation from 0 to 2 gives scores greater than or equal to 65% while the standard deviation between −1 and 0 gives scores greater than or equal to 52% but less than 65%; furthermore, the standard deviation between −4 and −1 gives scores between 0% and 49%. In Figure 2, the topmost plot for mathematics score shows that a standard deviation greater than 0 represents scores greater than or equal to 65% and enables us to predict factors that impacted student performance for the mathematics score. e five-dimension vertical lines predict that the high level of education of parents does not necessarily impact student performance as a student with a parent with some college or high school degree performed better than a student whose parent has an associate's or master's degree. e parallel coordinate plot in Figure 2 shows that another important factor that affected student performance in math was the type of lunch. It was observed that students with a standard lunch performed better than students with a free or reduced form of lunch. It was also observed that the test preparation did not affect student performance for a student who scored more than 65% in math. e bottom row in Figure 2 with a standard deviation of less than −1 shows a math score lower than 50%. It was identified that a student with free/reduced lunch had a bad performance in math and it was further observed that students that fall under race group c and are females also had a poor performance in math while the level of parents' education was not an important factor. Figure 3, the topmost plot for reading score shows a standard deviation greater than 0 represents reading scores greater than or equal to 65%, which enables us to predict factors that impact student performance for reading score.

Strong Predictors for Reading. In
In Figure 3, the five-dimension vertical lines predict that the high level of education of parents still does not impact student performance as a student with a parent with some college or high school degree performed better than a student whose parent has an associate's or master's degree. However, contrary to the observation of the math score, the reading observation gave a different result. Females in group C race/ethnicity performed better in reading than math; furthermore, the parent level of education is not an important factor in student performance. e bottom row in Figure 3 with a standard deviation of less than −1 shows a reading score lower than 50%. It was identified that the free/reduced lunch factor has a significant impact in affecting the performance of students. Furthermore, females in group E race/ethnicity recorded the lowest reading scores. Figure 4 shows the parallel coordinates with five dimensions represented by N � 5 vertical lines impacting the writing score.

Strong Predictors for Writing.
In Figure 4, the topmost plot for writing score further reiterates that parental level of education does not impact student performance as a student with a parent with some college or high school degree performed better than a student whose parent has an associate or masters degree. Females in group C race/ethnicity performed better in writing than math; furthermore, the parent level of education is not an important factor in student performance. e bottom row in Figure 4 with a standard deviation of less than −1 shows a reading score lower than 50%. It was identified that the free/reduced lunch factor has a significant impact on affecting the writing score of students. Furthermore, females in group E race/ethnicity recorded the lowest reading scores.   show the diagnostic ability of binary classifiers. e AUC provides a trade-off between true positives and false positives [3]. Table 3 shows the summary of results of AUC values for the linear SVM versus the ten benchmarked algorithms. Figure 5 shows the ROC curve for the linear SVM compared to the ten benchmarked algorithms used in this study. e results depicted in Figure 5 show that linear SVM outperformed other algorithms iterating the accuracy of the linear SVM in classifying and predicting factors that impact student performance.

Conclusions
e results show that, in predicting students' performance, linear support vector machines showed superior performance when applied to student data. e algorithm predicted that the parental level of education does not influence students' performance whilst impactors such as race, gender, and lunch have a bearing on student performance. In future work, we will consider using an ensemble of classical machine learning algorithms to boost accuracy in the prediction of student performance.

Conflicts of Interest
e authors declare no conflicts of interest.