VALIDITY AND RELIABILITY OF EDITORIAL PLAN PRODUCT ASSESSMENT RUBRIC IN DIGITAL PUBLIC RELATIONS COURSE

This study aims to determine the validity and to estimate the reliability of the editorial plan product assessment rubric in the Digital Public Relations course. The focus of the research is the content validity and reliability estimation of the product assessment rubric: editorial plan. The Content Validity Ratio (CVR) and Content Validity Index (CVI) from the assessment rubric and V Aiken were calculated using Ms. Excel. The Calculation of content validity by five experts was valid. The Interclass Correlation Coefficients (ICC) were used to estimate the reliability of the scoring rubric and were calculated using SPSS 24.0 software. The rubric was tested on 3 raters who assessed 30 students each. The results of the calculation of the reliability estimate show a good level of reliability. This validity and reliability estimation of the rubric is expected to be able to support an objective assessment, especially on editorial planning products in the Digital Public Relations course.

There are at least three types of instruments based on the type of learning outcomes measured: cognitive assessment instruments, affective assessment instruments, and psychomotor assessment instruments (Budiastuti, 2014).Cognitive instruments assess' one's thinking ability; affective instruments assess ones' attitude towards certain matters; and psychomotor instruments assess one's specific skills.The form of each assessment instrument also varies.Cognitive assessment instruments can be in the form of formal, informal, or non-test test instruments.The affective assessment instrument is usually in the form of a non-test.In addition, there are at least three performance assessment instruments, products, and portfolios to measure psychomotor (Saepuzaman et al., 2021).
Digital Public Relations is one of the practical courses that promote project-based learning.
This course is part of a curriculum based on the 2016 Indonesian National Work Competency Standard (SKKNI) for Public Relations.One of the course outcomes is that students can create and execute digital public relations plans.This course outcome is supported by the material of preparing an editorial plan for a public relations campaign.The campaign itself plays as a public relations strategy to support an institution or corporation's branding.The learning process consists of several explanations of the subject, discussion of related cases, and a project.In this project, students are asked to find a potential client independently, to research on potential client's branding and public relations-related problems, to offer a digital public relations campaign plan based an editorial plan, and finally to execute and evaluate their campaigns using available tools.In order to conduct an ideal assessment for this course, valid and reliable instruments are needed.
Studies on instrument development, specifically focusing on its validity and reliability have been conducted by many.Most of the instruments being discussed measure cognitive outcomes (Bashooir & Supahar, 2018;Ramadhan et al., 2019;Retnawati, 2018), while others discuss affective outcomes (Saepuzaman et al., 2021).There were also focusing on psychomotor outcomes, specifically in performance, as seen in industry (Syahrul, 2010) and sport (Currell & Jeukendrup, 2008).However, discussion on the validity and reliability of instruments to assess psychomotor outcomes in the form of products is still limited.For this reason, this research focuses on the content validity and reliability of the editorial plan product assessment instrument in the Digital Public Relations course.

Methods
This research is a part of the development of a product assessment instrument, specifically an editorial plan for the Digital Public Relations course.This psychomotor assessment instrument, in the form of a product assessment, was developed to measure the ability of undergraduate students in Bachelor of Applied English in making editorial plans, which are part of the Digital Public Relations course.This is a practical course in which one of the course outcomes is that students are able to create and execute digital public relations campaign plans.One of the materials in it is to make an editorial plan.In general, indicators of competency achievement are determined by the student's ability to: 1. determine the key message(s) to be delivered; 2. determine the digital media to be used; 3. determine the type of content to be created and, if necessary, determine its completeness (e.g., caption); and 4. create a schedule for uploading content, including delegating the persons in charge.
One of the criteria for a good instrument is to have a high level of validity and reliability.
Instruments that have met the validity and reliability standards can be used in the measurement stage (Bashooir & Supahar, 2018).Meanwhile, there are also studies on the validity and reliability of instruments assessing psychomotor outcomes, specifically on performance (Currell & Jeukendrup, 2008;Syahrul, 2010).However, discussion on validity and reliability of psychomotor outcomes focusing on product assessment is still limited.For this reason, this research was conducted to determine the validity and to estimate the reliability of the editorial plan product assessment instrument of the Digital Public Relations course.

Research instruments
This editorial plan product assessment is measured using an assessment rubric.Thus, the object of the study is the assessment rubric for editorial planning products made by students taking a Digital Public Relations course.The product assessment rubric that has been developed was then tested on the assignment of making an editorial plan in the Digital Public Relations course.To determine the instruments validity, five experts were involved.The experts in both pedagogy and public relations were chosen using a convenient sampling technique (Taherdoost, 2016).They were contacted, given the developed product assessment instruments, and asked to give scores on subject matter, construct, and language/culture on the content validity rubric.The reliability of the instrument was then estimated through a trial.The trial was conducted in a vocational study programme at a higher vocational education institution in Indonesia in May 2022.Using a convenient sampling technique (Taherdoost, 2016), a total of 30 students taking a Digital Public Relations course and 3 raters were involved in this trial process.The students were given an assignment to make an editorial product, and then the 3 raters assessed the students' work independently using the developed assessment rubric.

Data analysis
The validity and reliability of an instrument can be seen in various ways (Bazvand & Ahmadi, 2020;Leung, 2015;Retnawati, 2018).Validity can be categorised into criterion validity, content validity, and construct validity.This study focuses on content validity and construct validity.The construct validity and reliability of the editorial plan product assessment rubric from the Digital Public Relations course were investigated.

Content validity
Content validity can be done through focus group discussions (FGD), where the determination is made by experts (Wasidi & Mardapi, 2016).There are at least two methods that can determine the validity of an instrument, namely the Lawshe method (Lawshe, 1975) and the Aiken method (Aiken, 1985).The Lawshe method uses a minimum of five (5) panels consisting of experts and practitioners in the field being measured, where each panel expresses their opinion regarding the instrument developed.The number of experts who were asked to provide expert judgement in this study were five, who were experts in the fields of teaching and public relations.Rating scales that can be used include: "essential", "useful but not essential", and "not necessary" (Bashooir & Supahar, 2018;Ramadhan et al., 2019;Wasidi & Mardapi, 2016).
To determine the validity of this study we uses the Content Validity Ratio (CVR) as the content validity of the items and the Content Validity Index (CVI) to see the validity of the content of the test.CVI is the average of all items' CVR (Hendryadi, 2017).Both were conducted because CVI reporting should be accompanied by a report on the range of CVR values from the selected items (Polit & Beck, 2010).In addition to the Lawshe method, the Aiken formula (Aiken, 1985) was also used to calculate content validity.The number of rating categories used in the Aiken formula is between two and seven.This study uses five rating categories and involves five raters.Index V of an item can be categorized based on the values that can be seen in Table 1.Very valid Saepuzaman et al., (2021) Aiken's V formula was used to calculate the content validity coefficient.The calculation of this coefficient is based on the results of an expert judgement of n people on an item in terms of the extent to which the item represents the construct being measured (Hendryadi, 2017).The assessment was carried out by assigning a number between 1, which represents "irrelevant", and five which represents "very relevant".

Reliability estimation
This study uses Interclass Correlation Coefficients (ICC) to estimate the reliability of the assessment rubric.The method of estimating the reliability between raters was calculated using the correlation coefficient between classes (Widhiarso & Suhapti, 2015).ICC shows the comparison between the variation caused by the measured attribute and the overall measurement variation.ICC is the reliability used for rating results from observations of several raters (Bashooir & Supahar, 2018).The number of raters involved in this study was 3. The calculation and analysis of ICC in this study were conducted using SPSS 24.0 software.From the measurement results, ICC reliability estimation was used to figure out how reliable the assessment rubric instrument was.

Content validity of the assessment rubric
The assessment rubric validity was analysed using Lawshe's content validity.The CVR value must be greater than 0.99 so that the item can be declared as valid.This applies to the validity of using 5 SME (Lawshe, 1975).Table 2 shows the results of figuring out the CVR value based on the content of the evaluation rubric.Table 2 shows the perfect scores given by experts on all items used on the developed assessment rubric.The overall aspects of subject matter, construction, and language/culture received a score of five.This then made the CVR value obtained 1 and the instrument could be declared as valid (Lawshe, 1975).The CVI value obtained from the average CVR is 1.When the CVR value is more than 0.99, it means that all of the items have been declared valid (Lawshe, 1975) and deserve more research.

Aspect
Table 3 shows the scores from the evaluation rubric that were based on the validity of Aiken's V. Table 3 shows the average calculation result of Aiken's V with 5 raters is 0.93.By looking at the criteria for content validity in Table 1, we can say that the editorial plan product assessment rubric instrument is very valid and can be used again.

Estimation of the assessment rubric's reliability
In the pilot phase, students were assigned to make an editorial plan.The results of student work or products were then assessed by each of the 3 raters using the assessment rubric.The scores given by the rater on student products were then analysed to determine the estimated reliability value by looking at the ICC value.The numbers shown represent the comparison between the variation caused by the measured attribute and the overall measurement variation.The results of the correlation analysis between ICC classes using SPSS 24.0 software can be seen in Table 4.The results of the ICC analysis seen in Table 4 show that the average agreement between raters is 0.896, with a consistency of 0.963.The ICC value indicates a very good level of instrument reliability because it is in the range of 0.75-1.00(Cicchetti, 1994).

Conclusion
The observation sheet of an editorial plan product assessment rubric in the Digital Public Relations course is declared valid based on its contents.The product assessment rubric is also declared reliable.Based on the results of the validity and reliability tests, it can be said that the editorial plan product assessment rubric used in the Digital Public Relations course meets the content validity and reliability requirements.
Regarding the development of good instruments, further researchers can consider that the more experts involved in the development of instruments, the better the quality of the instruments being developed.In addition, during the trial process, it is also necessary to consider choosing respondents who represent students with different abilities, either low, medium, or high.This is necessary in order to get a better estimate of reliability.

Table 1 .
V index category.