Structural Validity and Reliability of the Dutch Older Patient in Acute Care Survey (OPACS), Measuring Nurses Attitude towards Older Patients

Background: In clinical practice, nurses’ attitudes regarding older patients are important in relation to quality of care. The Older People in Acute Care Survey (OPACS) is an instrument measuring hospital nurses attitudes regarding older patients and is validated in Australia and the USA. The OPACS is translated in Dutch language and content validity of this translation is previously assessed, presenting questionable results. Measurement instruments, however, cannot be “validated” based on content validity evidence alone. Judgmental evidence and statistical analysis should be combined to fully evaluate content domain definition and representation and guide further development. Objective: Assess structural validity and reliability to fully evaluate the OPACS for use in the Netherlands, complementing previous conducted content validity results. Design: Cross-sectional. Setting: Three general hospitals in the Netherlands. Participants: 201 registered nurses. Methods: Confirmatory factor analysis was used to assess the structural validity. Reliability was assessed with Cronbach’s alpha. Results: OPACS Section A (measuring practice experiences) demonstrated to have acceptable structural validityand good reliability outcomes after exclusion of two items (model fit: x2 [df=537]=8475.40, p <0.001, CFI=0.96, TLI=0.96, RMSEA=0.21; Cronbach’s alpha=0.82). Section B (measuring general opinion) demonstrated to have inadequate structural validity outcomes (model fit: x2 [df=1127]=9200.29, p<0.001, CFI=0.68, TLI=0.67, RMSEA=0.15). None of the items contributed significant to the factor and therefore no further analysis could be performed (range p(>|z|)= 0.551 -0 .788). Conclusion: Even though structural validity for section A was acceptable, content validity scores of a majority of items in this subscale were low, resulting in questionable use of this subscale for the Dutch context. The findings of this study, in relation to the earlier findings regarding content validity, justify the conclusion that use of the Dutch OPACS in clinical practice and research is not recommended. Given these findings, future research should pursue the development or (cross-cultural) validation of other instruments measuring hospital nurses attitudes towards older patients for the Dutch cultural context. Furthermore, this study demonstrated the influence of cultural differences on measurement instruments and the need for rigorous research before using a measurement instrument in a new culture or context.


Introduction
Worldwide, people are aging [1]. This demographic change results in an increase of older people admitted in hospitals. A growing number of registered nurses will encounter older patients in their daily work and a positive attitude is often promoted [2,3]. Healthcare professionals need to understand current attitudes regarding older patients when workforce strategies for promoting positive attitudes are to be implemented [4,5].
The Older Patient in Acute Care Survey (OPACS), developed in Australia, measures hospital nurses practice experiences and general opinion regarding older patients which are considered aspects of attitude [6,7]. The OPACS was developed using focus groups with 16 nurses discussing their experience of caring for older patients in the acute care setting. The final OPACS consisted of 86 items related to 13 different aspects influencing the nursing care of older patients. Verbal statements regarding these 13 aspects are scored on a 5 point Likert scale ranging from strongly disagree -strongly agree [6]. The OPACS has been translated and content validity is assessed in the USA. Results demonstrated excellent content validity scores [8]. Furthermore, structural validity and reliability outcomes for the American OPACS proved to be good [7]. After translation towards the Dutch language, however, a majority of items were considered "not to be relevant" by experts, resulting in low content validity scores even though translation was considered good [9]. Why the content was not considered relevant for the Netherlands, and whether or not this is reflected in the construct is unknown, making it difficult to adjust the OPACS to the Dutch context.
In literature, the concept of content validity has been controversial since its inception and it is described that although content validity is a fundamental requirement of all assessment instruments, measures cannot be considered "valid" based on content validity evidence alone [10]. Both judgmental and statistical analysis provide important information regarding content-and construct validity and both approaches have their limitations. Therefore it is recommended to use both types of analysis to fully evaluate content domain definition and representation [11][12][13]. With assessment of the structural validity and reliability of the Dutch OPACS, content validity results of a previous study will be complemented, resulting in a full evaluation of the OPACS content and use for the Dutch cultural context.
The aim of this study is therefore evaluating the structural validity and reliability of the Dutch OPACS, complementing previous study results.

Design
This study followed a multicenter cross-sectional design.

Setting and subjects
Data of nurses from the Netherlands were derived over a six-month period. Registered nurses working in three general hospitals located in the middle of the Netherlands were recruited and included after informed consent was obtained. Nurses were invited to participate through e-mail from their ward manager, flyers, and a message on the online hospital communication boards. Nurses completed the Dutch OPACS and several questions regarding their socio-demographic characteristics online. The study was approved by the medical review board of the University Medical Center Utrecht, the Netherlands (METC protocol number: 14-345/C).

Measurement
The OPACS is originally developed in Australia [6]. It consists of two scales; section A measuring practical experience (36 items) and section B measuring general opinions towards older patient's needs (50 items) on a 5 point Likert scale. The Australian OPACS demonstrated good face validity and high reliability scores (Kappa 0.76). The United States version of the OPACS showed a high Scale-Content Validity Index/universal agreement (S-CVI/ua) score (S-CVI/ua=0.92) [8] and good structural validity and excellent reliability scores (Cronbach's alpha=0.93) [7]. The American OPACS was translated and validated on content in the Netherlands, demonstrating positive translation but alarming content validation results (S-CVI/average 0.62) with major differences in rating of relevance between experts (S-CVI/ua=0.13). The same Dutch OPACS was used for data collection in the present study to assess the structural validity.

Analysis
Confirmatory factor analysis (CFA) was used to assess the structural validity of the Dutch OPACS. The aim of CFA is to test a hypothesized factor structure or model and assess its fit to the data. Relations of indicators (observed variables) to factors (latent variables) as well as the correlations among the latter are tested in the measurement model [14].
First, missing values were assessed to determine whether list-wise deletion could be used. Then CFA was performed for Dutch OPACS section A and section B by testing several models using Lavaan: an R package for structural equation modeling [15]. Evaluation of each model was based on considering a variety of fit measures: the X² minimum fit function test; the Comparative Fit Index (CFI); the Tucker Lewis Index (TLI) and the Root Mean Square Error of Approximation (RMSEA). Values of >0.95 for the CFI/TLI indicate a good fitting model. The RMSEA should be <0.06 indicating a good fitting model [14,16]. All analysis are performed using R [17].

Results
Of the participating sample, only complete cases were included in this study (73.6%). The socio-demographic characteristics of respondents with missing values were not significantly different from complete cases (all p>0.05). Socio-demographic characteristics are presented in (Table 1).

Validity and reliability of the Dutch OPACS section A (Practice experiences)
In

Validity and reliability of the Dutch OPACS section B (General opinion)
The same CFA model was used to assess the Dutch OPACS section B: general opinion. The unidimensional model for the Dutch OPACS section B (Model 1) did not fit the data well (x² [df=1127]=9200.29, p<0.001, CFI=0.68, TLI=.67, RMSEA=0.15). When looking at the items separately to assess which items should be excluded to improve the model fit to the data, it appeared that none of them contributed significant to the factor (Table 3) and therefore no further analysis could be performed (range p(>|z|)=0.551 -0.788).

Discussion
This study assessed the structural validity and reliability of the Dutch OPACS measuring practice experiences and general opinion of hospital nurses regarding older patients.
The items of section A: practice experiences, demonstrated to measure one construct. Only two items did not contribute to the construct and were therefore excluded. None of the items in OPACS section B, contributed significant to the factor general opinion, meaning that none of the items measured the construct solely making it impossible to include good items and exclude bad items using statistics.
In a previous study by van Schelven et al. [9], low content validity scores for 20 (58.8%) of the 34 items were presented for subscale A: practice experiences. By assessment of items with low content validity, several cultural reasons were found explaining the low rating by experts. First, 26 items (72.2%) mentioned a difference in care giving between old and young patients with only 5 (19.2%) of these items considered relevant. Focusing on the difference in care giving between old and young patients undermines the Dutch vision that care should be adjusted to the need of the individual patient (the same basic principle for old and young) which is taught in education and in clinical practice in the Netherlands. For example the item: "I ask older patients if they have pain more often than I ask younger patients", with total agreement reflecting a positive attitude, is considered not to be relevant because nurses should assess pain three times a day in every hospitalized patient regardless their age according to Dutch quality systems. Second, the relation between several items and "positive or negative attitude" was unclear for experts resulting in a questionable scoring system. For example, the question: "I am more likely to speak in simple language to an older patient than to a younger patient" with total agreement reflecting a positive attitude. However, language used by nurses should always be adjusted to the individual patient, and not be based on age alone because this can lead to a feeling of stereotyping by the older patient influencing the perceived quality of care [18]. Therefore, not agreeing on this item can also be explained as "good attitude" by nurses respecting the older patient and approach him/her as an adult. This makes it disputable what "good attitude" is in relation to the item as presented by the OPACS. Before this subscale can be used in clinical practice in the Netherlands, items should be reexamined, discussed and adjusted by experts on content. The number and form of adjustments needed is so rigorous that this will result in a new instrument which means that 1) it will not be comparable with the OPACS-US or any other existing instrument making cross-national comparisons impossible and 2) might not be worth the effort with other existing instruments possibly more suited to the Dutch culture in its origin. The study by van Schelven et al. [9] also presented low content validity scores for 60% of the items in section B: general opinion for comparible reasons as with section A. Results from the content validity study by van Schelven [9], combined with this study suggest that the translated version of the OPACS in the Netherlands should not be used.
This study demonstrated that assessment of both content-and structural validity are necessary to fully comprehend the validity of an instrument in a particular culture [10]. Content validity is considered a fundamental requirement [10]. Our results support this, demonstrated by the Dutch OPACS section A which would be assessed valid if only structural validity and reliability scores would have been conducted.
Too often, instruments are tested only using quantitative tests to assess validity and reliability when used in different cultures, settings and groups. Our studies demonstrate that validity and reliability of instruments can differ substantial between countries emphasizing the importance for rigorous cross-cultural validation before an instrument should be used in clinical practice in different cultures and in research. Researchers should therefore always assess content validity and describe possible (cultural) differences on item and scale level, as this influences the results (and interpretation of results) of the study conducted.
Some considerations regarding this study should be discussed. Missing data were not imputed and cases were excluded (even though missing values were completely at random) to maximize the validity of the item selection during the item reduction process. This is considered acceptable as no differences were found in characteristics between full cases and cases having missing values and performance of analysis was not affected by sample size. Furthermore, considering the response rate and sample size of nurses from the different centers, the representativeness (having an convenience sample) can be questioned and selection bias could have led to an overestimation of effect as nurses with interest in older patients are more likely to participate. However, this is not considered a problem as the primary focus was on structural validation of the Dutch OPACS and not an exploration of attitudes of Dutch hospital nurses. Furthermore, no sample size problems were indicated in analysis of the data. Third, OPACS section A proved unidimensional. However, whether the same construct is measured in the United States as in the Netherlands is not assessed in this study. Measurement invariance between items should always be assessed before comparisons between countries can be performed [19]. With regards to the Dutch OPACS, it is likely that the Dutch subscale measures a different construct taking content validity results into account.

Conclusion
In conclusion, in clinical practice, identifying attitude problems is an important step to improve the quality of care for older patients [4]. However, it is important that valid and reliable instruments are used to do so. The results from this study cannot justify the use of the Dutch OPACS in clinical practice and/or research. Even though section A (measuring practice experiences) demonstrated to have good structural validity results, items measuring practice experiences are considered unclear in interpretation and scoring and therefore not ready for use in the Netherlands. Section B (measuring general opinion) also demonstrated not to be applicable for use in the Netherlands as a result of low structural validity and reliability. Although section A might have some pointers for developing a new instrument, it might not worth the effort having other instruments potentially more suited to the Dutch culture in its origin.