A Rasch Analysis of Item Quality of the Chemical Literacy Assessment for Investigating Student’s Chemical Literacy on Chemical Rate Concepts

Assessment is a topic that continues to be developed in science education research. Assessment evaluates not only students' cognitive abilities but also their thinking skills. Therefore, in this study, an assessment that could measure students' chemical literacy was developed. Chemical literacy is a thinking skill that students must develop as part of their chemistry learning. The goal of this study was to assess item' quality, as well as student’ chemical literacy on the concept of chemical rate. The Rasch model was employed to analyze the data in this study. The results of this study depict that the developed assessment had sufficient reliability and validity to be used to assess students' chemical literacy. Furthermore, the analysis of the students’ responses to the items revealed that many students did not understand or were unaware of the context presented. These findings suggest that students' chemical literacy in the material for the reaction rate is still lacking and needs to be improved. As a result, the teacher's role in assisting students in improving their chemical literacy through chemistry learning is critical.


Introduction
Assessment is one of the topics that are frequently discussed in science education research. The assessment is a process in which teachers and students work together to produce a result that will be used for feedback and the improvement of future learning (Black & Wiliam, 1998). Because students' understanding of concepts changes over time, assessment is always discussed. Therefore, in order to diagnose students, an assessment that can provide information about students' level of understanding, whether they already understand, misunderstand, or even lack conceptions, must be developed (Johnstone et al., 1977). Diagnostic tests are typically undertaken in various formats, the most common of which is multiple-choice. This is because multiple-choice questions can provide a quick analysis and a reliable score (Caleon & Subramaniam, 2010;Mcclary & Bretz, 2012). Assessment to diagnose students' cognitive understanding has made a lot of progress. However, the assessment can also be designed to assess students' thinking skills (Sadler & Zeidler, 2009).
Currently, only a few assessments can be used to analyze the thinking skills required by students in the 21 st century. One of the thinking skills that students need to have is chemical literacy, which is a part of scientific literacy. Similar to scientific literacy, chemical literacy must also be improved by learning chemistry (Cigdemoglu et al., 2017). There have been many studies on chemical literacy as knowledge and skills that must be present in learning chemistry. However, research on how literacy is taught and measured is considered new (Celik, 2014). Thus, it is necessary to develop an assessment to evaluate students' chemical literacy (Shwartz et al., 2006).
In this study, this assessment was developed to analyze students' chemical literacy, by following the dimensions of chemical literacy by Shwartz et al. (2006), including chemical content knowledge, chemistry in context, high-order learning skills, and affective aspects. The assessment was developed with multiple-choice questions, which could give simple and quick analysis, be objective, and be used to analyze students' conceptions. Only three dimensions were used in this assessment: chemical content knowledge, chemistry in context, and high-order learning skills. The affective aspect was not included because it differed from the others, necessitating the use of other assessments to evaluate it. Moreover, tools for analyzing student test results are required to obtain good test analysis results. The Rasch model is one of the tools that can be used to analyze the student's response. This model is useful since it can analyze students' responses after taking the test based on the difficulty level of the item as well as the level of students' ability (Englehard, 2013). The Rasch model is also distinct in that the data used are calibrated to take the form of interval data, the relationship between the difficulty of the item, and the students' ability to calculate the Rasch model independently (Bond & Fox, 2012;Boone et al., 2014). Furthermore, many people believe that Rasch analysis is a suitable approach to measurement in the field of social science (Bond & Fox, 2012;Boone et al., 2014;Liu, 2010).
The assessment developed in this study was limited to the chemical rate concepts. This is due to the fact that the chemical rate is one of the concepts that is both difficult to teach and challenging for students to grasp (Cakmakci et al., 2006). Besides, the chemical rate is frequently misunderstood (Bain & Towns, 2016;Chairam et al., 2015). Therefore, it is essential to conduct a proper assessment of the students' understanding of this chemical rate concept. The study aims to assess the item' quality as well as to determine the students' chemical literacy on the chemical rate concept.

Literature Review Chemical Literacy
Science literacy is one of the primary goals of science education and is an essential component of science education worldwide (American Association for the Advancement of Science [AAAS], 1993;Coll & Taylor, 2009;DeBoer, 2000;National Research Council [NRC], 1996). Scientific literacy is essential for students to become well-informed about how to perceive science issues in order to make decisions in a variety of subjects that affect their daily lives (Dijk, 2011;Tal & Dierking, 2014). The PISA 2015 framework refers to the importance of understanding not only scientific ideas and theories but also general methods associated with scientific research and how this allows science to improve (De Jong, 2012). Someone with scientific literacy should understand scientific knowledge, science application, cognitive skills, the ability to solve problems using scientific knowledge, the nature of science, the ethics of scientists, as well as their relevance to culture, society, and technology (Barlia, 2016;Holbrook & Rannikmae, 2007;Liu, 2009;National Research Council [NRC], 2007;Norris & Phillips, 2003;Preczewski et al., 2009). Science literacy has been recognized as important in our current civilization since it has been twisted with science and technology. Furthermore, science literacy includes the following components: (1) understanding natural science concepts, (2) understanding and critically judging scientific content, and (3) preparing members of society to face real-life scientific and technological contexts (Miller, 1983;Norris & Phillips, 2003;Shwartz et al., 2006). Because scientific literacy is a broad concept, teaching any specific subject in science learning should provide students with the opportunity to practice to be scientifically literate. Teaching chemistry develops students' chemistry literacy, as well as their scientific literacy in general (Shwartz et al., 2006).
Chemistry is an important branch of science. Chemistry topics typically include learning materials that are relevant to many disciplines, such as health, geography, physics, environment, and economics (Brown et al., 2000). Understanding chemistry is also very significant for most people because chemicals and chemical products are related to daily lives (Gilbert & Treagust, 2009). In addition, chemical literacy entails knowledge of the nature of matter, chemical reactions, laws, and chemical theory, as well as general chemical applications in everyday life (Barnea et al., 2010). Someone who is chemically literate is someone who can apply knowledge of chemistry in everyday situations (Tsaparlis, 2000). Thus, understanding chemistry will enable people to participate in public debates and solve problems in their daily lives. Chemical literacy involves knowledge of chemistry as well as the necessary skills for a chemistry-based understanding of social and scientific issues. Chemical literacy consists of three components: (1) key concepts in general chemistry, such as elements, symbols, processes, and models; (2) understanding of what expert chemists in academia and industry do; and (3) social context, or the ability to place chemistry in a real-world context Holman & Hunt, 2002;Shwartz et al., 2013). Moreover, chemical literacy encompasses an affective component, which refers to an interest in chemistry-related topics (Shwartz et al., 2006). Because chemical literacy is an important aspect or skill for students when learning chemistry, it is necessary to assess their chemical literacy. The purpose of this assessment is to determine the level of chemical literacy among students. As a result, teachers will receive feedback on how to improve chemistry learning, which will help students improve their chemical literacy. Chemistry exams used to assess students' chemical literacy must include dealing with given information in a chemistry problem, as well as students' knowledge of how to use chemistry and skills to include information from an everyday context (Witte & Beers, 2003). In these assessments, the context can be an industrial process, an environmental issue, or any other issues from school, home, or science in laboratories.

Rasch Model
Rasch model is a measurement method that is part of a group called item response theory (IRT) and is frequently used to analyze data in educational research (Khine, 2020). Rasch model analysis provides an alternative method for investigating the psychometric properties of measures and addressing response bias (Bradley et al., 2015). The analysis approach presented by this model can be applied to promote the test item as an essential tool in assessment for learning (Suminto & Widhiarsho, 2015). The data are analyzed using WINSTEPS (Linacre, 2011) to estimate students and item measures. For the dichotomous data in the Rasch model, the probability that a student will answer an item correctly is determined by the variation in the level of student's performance and item's difficulty (Bond & Fox, 2012). The Rasch model produces an interval scale with units of ''log-odds," which is also known as logits.
Rasch analysis measures item difficulty as an independent unit. Item difficulty is measured on the same interval scale as students' ability, allowing item targeting to be estimated. The Rasch model determines the opportunity that a student will correctly respond to an item based on the difference between the student's overall performance level and the difficulty of the item. This is measured using the following equation: where Pni is the probability that student n of overall performance level Bn answers correctly to item i with a difficulty of Di (Bond & Fox, 2012;Liu & Boone, 2006). How well the items predict student responses based on ability and difficulty measures are delivered by fit statistics (Linacre, 2011). Fit statistics are used for both item difficulty and students' ability measures to the Rasch model and a powerful tool for deciding item performing for a given target population. Items that give unexpected student responses and students who give unexpected answers can be identified by fit statistics.
In addition to the fit statistic, the option probability curve depicts the likelihood that the student will select each answer option presented in the Rasch analysis. It is critical to examine the curve for each option in multiple-choice items because the shape of the curve provides information about the students' overall understanding. The response options reveal information about students' comprehension of the context. Analyzing the option probability curve for each answer choice thus provides more qualitative information about the student's understanding of each context on the item (Herrmann-abell & Deboer, 2016).

Methodology
This study employed a mixed-methods research design, combining quantitative and qualitative research (Creswell, 2006). This study began with the development of item questions based on the chemical literacy dimensions, which were then randomly tested on students. The results were analyzed using the Rasch model to determine the feasibility of the developed assessment as well as to analyze students' performance.
Item Development Shwartz et al. (2006) developed this assessment based on the dimensions of chemical literacy. The first dimension is understanding chemical content. Students who can explain chemical processes and energy changes at the macroscopic and microscopic levels using chemical language can also perform scientific investigations based on their knowledge in this dimension. The items in the content knowledge dimension consist of chemical reactions related to the graph of the reaction rate and the effect of concentration on the reaction rate. Furthermore, the second dimension is chemistry in context. Students can use their chemistry knowledge to explain everyday phenomena and make effective decisions in social situations involving chemistry. Moreover, the items in the dimension of chemistry in context include the examples of chemical phenomena in everyday life. The third dimension is higher-order learning skills. Students can ask questions, investigate information, and weigh the benefits and drawbacks of a phenomenon related to their chemical knowledge. The items created were derived from three dimensions and the questions were designed using the chemical rate concept. The items on the high-order learning skills dimension cover the examples of phenomena in nature and industry, as well as their roles in the human body related to chemical rates.
Also, the items developed were validated by high school chemistry teachers, chemistry education lecturers, and educational evaluation experts. This assessment covered 14 multiple-choice items that had been validated by nine experts. The validation was performed to determine the validity content of the items. The most common method for measuring content validity is the Item-level CVI (I-CVI), while the other method to measure content validity is Scalelevel CVI (S-CVI), which can be determined using Universal Agreement among expert (S-CVI/UA) or the Average CVI (S-CVI/Ave) (Lynn, 1986). The I-CVI is considered excellent if its value is 0.78. The S-CVI with a value between 0.80 and 0.90 is acceptable. The computation analysis yielded an average I-CVI of 0.98; S-CVI/UA of 0.85; and S-CVI/Ave of 0.98. These results indicate that the constructed items have excellent and acceptable content validity.
These questions were used to assess students' chemical literacy as well as their understanding of the chemical rate concept. These items were classified into four items of chemical content knowledge, four items of chemistry in context dimension, and six items of high-order learning skills. The assessment was then administered to students and analyzed using the Rasch model. Table 1 depicts the distribution of items.

Participants
The participants in this study were 277 students of 11 th grade from three different schools taken with random sampling. The schools are located in Surakarta, Central Java, Indonesia. The 11 th -grade students were chosen because the chemical rate concept was given in this grade.

Data Collection
Participants in this study were chosen randomly. The students participating in this study received previous learning chemical rate concept as well. This test was given immediately after participants had learned the chemical rate concept taught by each teacher at the representative school.

Data Analysis
The data were analyzed using the Winstep software (Linacre, 2006). The analysis results were used to compare students' responses and the items' difficulty level. The easier item had the lower logit value, while the more difficult item had higher logit. This is also portrayed on the items' Wright map. The easier items are at the bottom, while the more difficult items are at the top. The Rasch model would be used to quantify the item quality, the item and person reliability, item and person separation, and Alpha Cronbach value would be presented, as well as the item validity in the form of the infit value and the outfit MNSQ, Pt-measure correlation value, and unidimensionality. Besides, students' chemical literacy was qualitatively presented on probability items. The students' choice indicated their understanding of the context item.

The Quality of the Items of Chemical Literacy Assessment
Reliability This assessment's reliability was divided into the person and item reliability, as well as Cronbach's Alpha value. This assessment was also used to test the separation. Table 2 shows the reliability of this assessment.  Table 2 presents that the Cronbach Alpha value was 0.85, indicating that the developed items were reliable. An assessment resulting a Cronbach alpha value of 0.7 indicates acceptable level of realibility, while an alpha value of 0.8 is good and 0.9 is very good (Bond & Fox, 2012). The person's reliability was 0.79, signifying that the participants in this study were diverse and reliable. Meanwhile, the item reliability was 0.99, suggesting that this item was highly trustworthy. Moreover, separation of person was considered quite good because the value was close to two, and separation of item was very good because the value was greater than three. A good separation value is >2 for the person and >3 for the items (Bond & Fox, 2012;Duncan et al., 2003).

Validity
Validation is the process of seeking empirical evidence from the assessment developed (Stevens et al., 2010). The item fit statistic demonstrates the Rasch model validation. The fit statistical items for this assessment are presented in Table  3.  Table 3 demonstrates that the MNSQ values, both infit and outfit, mostly ranged between 0.5 and 1.5, meaning that the developed items can measure the expected indicators. MNSQ infit and outfit values of 0.5<MNSQ<1.5 are eligible (Bond & Fox, 2012). The Pt-measure correlation was greater than 0.3, noting that these items could discriminate against students' abilities (Li et al., 2016;Linacre, 2011). Furthermore, all items had a standard error less than 0.5, indicating that the items developed were accurate. Analysis of the items revealed unidimensional properties based on the value of the raw variance explained as well as the value of the unexplained variance. The raw variance value that shows good construct validity is >40% and the eigenvalue for each contrast is <2 (Linacre, 2011). The raw variance value explained by this assessment was 44% and the unexplained variance value for each contrast on each contrast had an eigenvalue less than 2 (1.2 to 1.9), indicating that the items being developed had unidimensional properties, or they have a good performance to measure. The developed measure items were classified as easy, with negative logit, medium that close to 0, and difficult, with logit values greater than 0. This demonstrates that the items were evenly distributed, as visible on the Wright map in Figure 1.

Figure 1. Wright Map of Item and Person
The questions ranged from easy to difficult, as illustrated in Figure 1. The easy items are listed at the bottom, and the difficult items are listed above. Similarly, the students' abilities are itemized on the left, where students with low abilities are listed at the bottom and those with high abilities are listed at the top. Based on these findings, the items produced could represent the students' abilities ranging from low to high.

Students' Chemical Literacy on Chemical Rate
The option probability of each item answered by the students revealed their responses. In this assessment, the options selected by students for each item could be analyzed as students' understandings of the chemical rate concept. The responses provided an overview of whether students perceived the concepts and contexts or whether a concept was incorrect.
The researcher selected three sample items, including item 8 (the most difficult), item 11 (the medium), and item 9 (the easiest). Items 8 and 9 were from the chemistry in context dimension, while item 11 was from the high-order learning skills dimension. Figure 2 shows the option probability for item 8.

Figure 2. Option Probability of Item 8
Item 8 was the most difficult for students to complete. This item presents the avocado experiment, in which the avocado is first left at room temperature. Following that, the avocado is given an additional treatment by spraying ethylene on its surface. Students were asked to figure out why the selected conditions cause the avocado to ripen faster. Option C is the correct answer; the addition of ethylene from the outside increases the ethylene concentration on the surface of the fruit produced by the avocado, causing the fruit to ripen faster.
Students with a positive logit value are those who can work on the items being tested. Students with negative logit values include those having lower ability to work on the items being tested. Figure 2 shows that students with low understanding, those with negative logit scores, guessed a lot. This is because students who did not understand the concept were more likely to select options A, B, D, and E. Students having a high understanding with logits between -1 and -2 were more likely to choose option D. Option D was, in fact, the most popular among students. This option contains information that the addition of ethylene is for catalysts that can accelerate fruit ripening. However, this assumption is incorrect because the fruit produces its own ethylene during the ripening process, and the addition of ethylene concentration accelerates fruit ripening. As a result, many students answered item 8 incorrectly.
Further, the following is the analysis of the medium item of item 11, and the option probability of the item is presented in Figure 3. Item 11 contains a question about the causes of acid rain, and the correct answer is option D. Students with a low level of understanding and a negative logit score guessed a lot on this item. It was indicated by the numerous probability options A, B, C, and E. Option D was chosen by more students with a higher level of understanding and a positive logit value. Option D states that acid rain is caused by human activities that increase the levels of gases such as SO2 and SO3. Students, on the other hand, preferred option C, mentioning an increase in CO2 gas and chlorine gas. However, CO2 gas and chlorine gas are more precisely the cause of ozone layer depletion. Meanwhile, the rising levels of SO2 and SO3 gases contribute significantly to acid rain. These findings indicate that many students were unaware of acid rain. The majority of them were mindful of one of the factors influencing the chemical rate, namely the addition of concentration, but they were clueless of such context.
The next is the analysis result of item 9, the easiest item. The option probability of this item is presented in Figure 4.  Figure 4 shows that nearly all students selected the correct answer, option C. This item posed a question about a comparison of two cooking circumstances for vegetables. The first condition is cutting the vegetables into large pieces, and the second condition is cutting the vegetables into small pieces. The students were asked to select the condition that causes vegetables to cook faster. Students with a lack of comprehension answered this item incorrectly. A logit value of -2 indicated that the D option was frequently chosen. Option D states that if more water is added, both conditions will trigger the vegetables to cook faster, whereas Option C states that the second condition, slicing vegetables into small pieces, is the reason. Small pieces of vegetables have larger surface areas and this makes the vegetables cook quicker. Students having the understanding that adding water or increasing volume can accelerate the reaction rate provided incorrect answers because, in this context, this condition can not make vegetables cook quicker.

Discussion
Based on the analysis, the developed assessment met the criteria in terms of reliability and validity aspects. The items had a good Cronbach alpha value, large separation, and good construct validity, as well as were quite thorough and able to discriminate against abilities. The validity was depicted from the MNSQ, Pt-measure correlation, standard error, and unidimensionality value. The MNSQ value, Pt-measure correlation, and unidimensionality value indicate that the constructed items have a good quality and can be used for measurement according to the specified indicators (Bond & Fox, 2012;Linacre, 2011). The Wright map results show a scattering of items ranging from easy to difficult. Furthermore, the population was slightly higher than the item difficulty average. The average difficulty item was 0.00, while the average student ability was 0.16, indicating that the students' ability was slightly higher than the average item difficulty level. This means that the students' level of understanding was nearly equal to the average item difficulty level. In addition, the selected respondents had varying abilities, as evidenced by the person's reliability of 0.79. A person's reliability close to or greater than 0.8 indicates that the respondent had good reliability and used a variety of abilities (Xie et al., 2014). Respondents' various abilities could also be used to evaluate the instrument's performance.
Some items, however, had lower MNSQ outfit values. This could be due to inconsistencies in student responses as well as a large number of students guessing (Boone et al., 2014). The data showed that unexplained raw variance value was 14%, meaning that 14% of the students were unpredictable in answering questions. Students' unpredictable responses reduced the instruments' fit. Nonetheless, the developed assessment was effective in measuring students' chemical literacy on the chemical rate concept.
Moreover, after being tested for this assessment, students' perceptions were found to be diverse. In item number 8, students did not understand the context and they responded that ethylene was a catalyst. This is because students understood catalysts to be materials that can speed up reactions (Kingir & Geban, 2012;Yalçınkaya et al., 2012). In this case, ethylene is added to increase the concentration of ethylene produced by the fruit itself, allowing the fruit to ripen more quickly. This means that students did not understand the context and neither did they maximize chemistry in context. However, many students correctly answered item 9, which is a simple question. This item assessed the chemistry dimension in context as well. This suggests that people could grasp the concept that expanding the surface area of a reactant increases particle collision (Yalçınkaya et al., 2012). The process of cooking vegetables is sped up by cutting them into small pieces. Moreover, the exemplified cases are very close to the students' everyday lives. This is beneficial to students because they can observe events in their daily lives that are closely related to chemistry. A person with good chemical literacy skills will always apply his knowledge in everyday life (Holman & Hunt, 2002).
In addition, many students have misconceptions about or do not comprehend the context presented in item 11. Some students were unaware of the cause of acid rain. Despite their incorrect answers, students understood the concept of a linear relationship between concentration and reaction rate (Cakmakci et al., 2006;Turanyi & Toth, 2013). Based on the analysis, it was discovered that the students were not well literate in chemistry, as evidenced by their incorrect answers to the context presented. Previous research disclosed that many students had difficulty in understanding chemical concepts to everyday life and making decisions on social problems related to chemical concepts (Broman & Parchmann, 2014;Cigdemoglu & Geban, 2015;Parchmann, et al., 2006;Yapicioglu & Aycan, 2018). In the context of chemistry, researchers agree on the importance of associating chemical concepts with the phenomena in everyday life to make abstract chemical concepts more concrete (Pabuccu & Erduran, 2016;Sevian et al., 2018). Putting students in contextual situations provides a clear picture of the application of the previously learned knowledge and improves skills relating knowledge to the relevance of everyday life (Broman, et al., 2018). Therefore, chemistry education must address everyday issues, technology, and social and environmental issues. This enables students to understand how chemistry relates to daily life and to apply it in decision making; in other words, students who are literate in chemistry (Shwartz et al., 2013;Tsaparlis, 2000). The student's inability to relate classroom learning to real-life situation stems from the lack of opportunities to learn to connect concepts and phenomena in everyday life (Wu, 2003). Teachers must facilitate students to develop their chemical literacy through chemistry learning. As a result, learning chemistry helps students understand not only the concept of chemistry, but also the role of chemistry in everyday life, technology, and social life.

Conclusion
Chemical literacy is a critical thinking skill that students must develop as part of their chemistry education. Based on the findings, the developed chemical literacy assessment has high reliability and validity for measuring students' chemical literacy. After being tested on students, it was discovered that many students' perceptions were incorrect. Besides, many students were unaware of the contexts in everyday life. These results indicate that the students' chemical literacy needs to be improved. Experts in chemistry education agree that learning chemistry must foster scientific literacy in general, and chemical literacy in particular (Dori et al., 2008;Talanquer & Sevian, 2013). The chemistry learning provided by the teacher must be capable of assisting students in developing their chemical literacy. Chemical literacy is also critical for teachers to enhance because of its impact on social life, the economy, and decision making (Avargil et al., 2013;Dori et al., 2018). Teachers can continue to provide cases relating to phenomena and items found in everyday life in the future (Kubiatko, 2015). This is necessary for students' chemical literacy to improve.

Recommendations
The researcher made several recommendations based on the findings. The assessment being developed was still limited to the reaction rate material, so future research with other materials in the chemistry subject can be developed. Those who are interested in this topic can combine chemical literacy dimensions with other chemistry subjects. However, when selecting material, future researchers must consider whether the material's characteristics are compatible or not with the chemical literacy dimensions to effectively measure students' chemical literacy.

Limitations
Although this research was carried out following scientific procedures, several limitations remain. The material chosen was limited to the reaction rate, and 14 items were created for each chemical literacy dimension. There were four items in the chemical content knowledge section, four items in the chemistry in context section, and six items in the highorder learning skills section.