Instrument and evaluation of students’ involvement in learning Mathematics in 21-century skills at senior high schools of Riau province

the involvement in learning is a determining factor for students' success in learning in an

students' involvement; learning mathematics; senior high school

Scan Me:
Student engagement is the most crucial factor in learning activities because it is the main factor in improving student learning outcomes in class. This study aims to develop instruments and evaluate student involvement in learning mathematics to achieve maximum learning outcomes at school. The population in this study was all high school students in Riau province. The sample is 150 high school students selected by multiple-stage random sampling based on high, medium, and low student achievement levels. Data analysis in this study was carried out four times: (1) data analysis based on the results of expert assessment of the instrument with the Aiken formula; (2) small-scale validity data analysis with first-order CFA; (3) large-scale data analysis with second-order CFA analysis; (4) descriptive analysis to see the categorization of student learning engagement in Riau province whether in very good, good, not good and very not good. The analysis using the Aiken formula shows that 24 out of 27 items developed are valid. The first-order CFA analysis shows that the 24 items distributed are valid. The second-order analysis shows that the construct validity of the learning involvement variable is valid and reliable. The analysis of learning involvement results shows that students' involvement in learning mathematics is in the not-good category. The Riau provincial government needs to maximize student learning engagement which can be done by regulating students' effective study hours at school and home. Collaboration between parents and schools must be improved so parents can control student learning outside of school. the involvement in learning is a determining factor for students' success in learning in an era of global challenges and success in mastering information technology (Bonilla-Molina, 2020).
Indonesia's mathematical ability internationally is still at a Low International Benchmark level. Indonesian Mathematics skills, on average, only recognize basic facts and cannot yet communicate, relate various topics, let alone apply complex and abstract concepts in mathematics (Mullis et al., 2020). Based on data released by the OECD in December 2019, the average score of Mathematics students in Indonesia is 379, and Indonesia is ranked 72 out of 78 countries (OECD, 2018). These results show that Indonesia's international mathematics skills could be better because, in terms of ranking, Indonesia still needs to catch up to neighboring countries such as Malaysia, Thailand, and Singapore. The low math ability of Indonesian students is the intense interest in learning, understanding, reading, and seriousness in learning mathematics (Korpershoek et al., 2015;Yüksel, 2014). One way to improve students' mathematical abilities is to increase students' involvement in learning mathematics with a variety of relevant learning resources.
Based on these problems, the development of instruments for evaluating students' involvement in learning mathematics in the era of global challenges and the factors influencing them is urgently needed to manage students' involvement in learning mathematics. The research novelty is developed by combining the research by some previous researchers (Jay et al., 2018;Li et al., 2021;Pan et al., 2018;Takeuchi, 2018) who recommended four indicators to describe students' involvement in learning mathematics: agentic engagement, behavioral engagement, emotional engagement, and cognitive engagement. Experts suggest two indicators to complete this research: an outdoor learning experience and engaging nearby sources. The instrument that will be developed can be a valuable resource for education stakeholders in Riau province to improve the Mathematics abilities of high school students in Riau province. Improving the involvement of learning mathematics is an essential thing that needs to be done by the education element. With valid and reliable instruments both in content and construction, there are areas for improvement in the learning involvement mathematics in Riau province. Even operational recommendations can be proposed to improve students' involvement in mathematics.
The involvement in learning mathematics is part of increasing or developing students' academic competence in learning at school, which significantly influences learning achievement (Daumiller et al., 2020). Student engagement is a determining factor and an essential concern for student performance in learning mathematics (Wang & Zhang, 2020). High motivation in learning can increase engagement resulting in high self-efficacy to succeed in learning Mathematics (Alemayehu & Chen, 2021). Student learning involvement can shape a student's success beliefs because learning involvement brings students into maximum activity to provide students with provisions for successful learning (Adeshola & Agoyi, 2022). Learning involvement makes it easier for students to master complex math material because there is a maximum effort to solve each problem through various strategies (Moon & Ke, 2020). Maximum learning involvement outside of school hours can control students' learning to the fullest and what needs to be done optimally (Khalid et al., 2020). The involvement of students in learning mathematics is a non-academic aspect that can improve student achievement in learning mathematics. Maximizing students' mathematics learning engagement is the best practice to enhance students' mathematics learning performance in class (Watt et al., 2017). Learning involvement or engagement of mathematics the best strategy to maximize students skill in learning the mathematics.

RESEARCH METHOD
This research is development research proposed by Borg and Gall (1983), which consists of 10 research stages which are further simplified into four stages according to research needs that is, (1) initial investigation, design stage, (2) instrument validation, (3) trial small and large scale, evaluation, revision, and (4) Instrument implementation of the instrument by using it to measure on a large sample. The investigative stage is the exploratory stage which is carried out by interviewing school principals and mathematics teachers, and students and finding constructs by exploring theories about the involvement of students in mathematics learning and the factors that influence it. The design stage is designing an instrument to measure/evaluate the involvement in learning mathematics at high schools in Riau province. The validation stage is through FDG activities with Evaluation Experts, Measurement Experts, and learning experts, where at this stage, the feasibility of the instrument is assessed by experts. The instrument's testing, evaluating, and revising stages are the instrument's testing stages on students in a small number of 150 students. The results of the trials will be evaluated and revised. The implementation or dissemination stage was carried out to high school students throughout Riau province. The sampling technique is cluster random sampling taken with a target sample of 438 Students from three districts. Each district is taken three schools and 48 to 50 students randomly. The instruments used in this study were questionnaires and interview guidelines. The instrument was developed based on the results of theoretical exploration or theoretical studies of research variables through various international books and journals. The indicators of the variables form the basis for developing items or instrument items that will be used as tools to obtain accurate information. Six indicators are used to get information, agentic engagement (AE), behavioral engagement (BE), emotional engagement (EE), and cognitive engagement (CE), outdoor learning experience (OLE), engaging nearby sources (ENS). Data analysis is an analysis of the validity and reliability of the instrument using EFA (Exploratory Factor Analysis), CFA (Confirmatory Factor Analysis), and Construct Reliability. Further data analysis uses descriptive statistics to evaluate student learning involvement by comparing whether there are differences in the measurement results with the criteria that have been developed.

Findings
The first analysis is an analysis of the instrument's validity developed through an assessment process carried out by experts (lecturers in the field of evaluation, statistics, or educational measurement). The assessment results are then analyzed to determine whether the instrument developed is in the valid category. The results of the analysis can be seen in Table 1. From the analysis results, it can be concluded that there are three items in the low category, namely OLE4, OLE6, and ENS4 items. These three items should not be considered for data collection because they can produce invalid information. The next analysis is a first-order analysis of CFA based on 150 student respondents who were distributed. From the analysis results, it can be seen empirically whether the 27 items are valid and reliable. Before item 27 is checked with CFA, it is necessary to analyze the adequacy of the sample through Kaiser-Meyer-Olkin Measure.  Based on the results of the analysis in Table 2, it can be concluded that the respondents used were 150 respondents who were already represented for analysis using CFA. From analysis is obtained value of the Kaiser-Meyer-Olkin Measure of Sampling Adequacy of 0.817. Table 3 shows the KMO value.  Table 3 describes that 24 items from six indicators have a loading factor is more than 0.3. These results showed that twenty-four items are valid and could be used to get information on student involvement in mathematics.

Reliability
Instrument reliability uses Cronbach's Alpha coefficient. The reliability value is good if Cronbach's Alpha coefficient is more than 0.7. The results of the analysis are shown in Table 4. Based on Table 4, it can be concluded that the instrument developed has a Cronbach's Alpha coefficient that meets the requirements so that the instrument can be used to evaluate students' learning involvement in mathematics material in junior high schools.

The Large-scale Trial
Large-scale trials were conducted to see the validity and reliability of the constructs obtained from the factors or components to be evaluated. The constructs of indicators obtained from the factors will be seen for their validity and reliability. This is done to prove that the constructs found through FGDs and literature review are valid and reliable. Validity and reliability were analyzed with CFA using Lisrel 8.80 software. The analysis results can be seen in Table 5,  Table 6, and Table 7.  Based on Table 5, eight fit model indicators with acceptable criteria are obtained. Chisquare is 4.34 is less than 10 (2 x df). The P-value is more than 0.05. RSMEA is less than 0.08, GFI is more than 0.90, NFI is more than 0.90, CFI is more than 0.90, IFI is more than 0.90, and NNFI is more than 0.90. These results indicate that the data obtained is feasible with the model used. Furthermore, the construct validity of the six indicators can be carried out. Table  6 describes the results of the construct validity analysis.  Table 7 illustrates the construct reliability coefficient obtained. Based on Table 6, the loading factor value is more than 0.3. The results of the analysis in Table 7 explain that the constructs of the six indicators have an acceptable validity value and can be used to evaluate the learning involvement in learning mathematics in junior high schools in Riau province. The reliability coefficient from Table 6 shows that the instrument for students' involvement in learning mathematics can be used with appropriate procedures to evaluate the involvement in learning mathematics for high school students in Riau province. Based on Table 7, it can be concluded that the construct reliability results are greater than 0.7.

Evaluation Results
After every procedure to develop instrument have finished and instrument are stated valid and reliable based on content and construct, evaluation on the mathematics learning involve- ment can be implemented. Figure 1 and Table 8 show the evaluation results of students mathematics learning involvement.  Table 8 shows the evaluation result of students' involvement in learning mathematics in Riau province. The results show 2,05% in the very good category, 36.07% in the good category, 60.07% in the not good category (still bad), and 1.60% in the very not good category (terrible). This result is described in Figure 1.  Table 8 explain, there is 2.01% students has very good criteria of mathematics learning involvement, 36.07% students has good criteria of mathematics learning involvement, 60.27% students has not good criteria of mathematics learning involvement, and 1.07% students has very not good criteria of mathematics learning involvement. Overall, students mathematics learning involvement in not good criteria with score 56.99.

Discussion
The instrument of the mathematics learning involvement has been valid and reliable by analyzing content and konstruk method. Valid and reliable instruments can provide information with a high degree of accuracy (Clifton, 2020;Setiawan et al., 2019). The quality of the instrument determines the success of the data collection process (Hadi et al., 2022;Amir & Risnawati, 2015). Quality instruments can determine policies to positively impact the work environment (Shrotryia & Dhanda, 2019). Valid and reliable instruments are a strong basis in determining the success of an educational program that runs for a certain time (Bagozzi et al., 1991;Taylor-Ritzler et al., 2013). The great instrument will provide comprehensive information about the advantages or disadvantages of educational programs (Hadi et al., 2019;Nicolella et al., 2018). Instruments that have met the validity and reliability criteria will answer information about ongoing programs (Ács et al., 2020;Bray et al., 2020). The instrument validated in terms of content and constructs provides meaningful information to stakeholders about the program being led.
The development of mathematics learning involvement instruments to maximize a measurement process at junior high schools. The instrument-preparing activity begins from the phase of determining the goal compiling test specifications, constructing initial item patterns, reviewing items, conducting initial testing of items, field testing items, determining statistics from item scores, designing and conducting reliability and validity tests and stages the last is to develop guidelines for administration, scoring, interpretation of test scores. These stages are carried out systematically so that an instrument is obtained that is used to measure students' intellectual character. Instruments that are developed with the proper procedures or rules will produce instruments that can measure the objectives of an instrument (Andrian et al., 2018;Setiawan et al., 2019). The stages are arranged coherently and ultimately can produce valid and reliable instrument items so that measurements can be carried out accurately (Andrian, 2019;Hadi et al., 2022).
The results show that the instrument's content developed is valid and reliable, following the advice of experts and practitioners so that this instrument can provide the best information in making decisions about the mathematics learning involvement of senior high schools of Riau province. Valid and reliable instruments based on expert judgment, practitioners, and field trials will give maximum results in measurement activities (Wahyuni et al., 2020;Wright & Craig, 2011). Quality instruments are instruments that have been validated with appropriate procedures through clear stages and can provide the information needed for policymakers (Shrotryia & Dhanda, 2019). The best information only get from the best instrument have validated by experienced validators from various aspects of knowledge (Terwee et al., 2018). Accurate information will be guaranteed by valid and reliable instruments that are validated through strict procedures (Risnawati et al., 2019). The instrument is one of the most crucial roles in research (Hadi et al., 2019) Instruments are used to measure data, so the quality of an instrument is needed to get the accuracy of good data. A good quality instrument will be able to obtain data that accurately describes the trait of the research subject.
Evaluation results show the students' mathematics learning involvement was still in the not good category. The results indicated that many indicators or aspects of the student's mathematics learning involvement are needed to improve so the government goal according to vision can be achieved. Low students' involvement in learning is affected by the low involvement of the parent, teachers, education policymakers (Panaoura, 2021). Students learning involvement of mathematics is basic to understand and master mathematics material on the high level (Igbo