Factorial Structure and Measurement Invariance of the Acceptance and Action Questionnaire-Stigma (AAQ-S) in Spain

The objective of the present study was to validate and adapt the Acceptance and Action Questionnaire-Stigma (AAQ-S) to the Spanish context. Method: The study included the participation of 1212 subjects, with an average age of 17.12 years old. Results: The confirmatory factorial analysis revealed a number of adequate fit indices for the new version of the scale χ2/df = 3.24; Comparative Fit Index = 0.96; Incremental Fit Index = 0.96; Root Mean Square Error of Approximation = 0.060; Standardized Root Mean Square Residual = 0.035, in which the factorial structures displayed gender invariance. The two factors comprise the scale both exhibited high internal consistency (+0.90) and temporal stability. Conclusion: The Spanish version of the AAQ-S proved to be a robust and adequate psychometric instrument. In this sense, future lines of research focused on determining the role of psychological flexibility in stigma and the processes of change at the base of interventions could benefit substantially from the use of AAQ-S.


Introduction
In recent years, there has been increasing interest in the study of stigma and the psychological processes with the objective of advancing in the design of strategies of intervention that end with harmful consequences for those who suffer it. Stigma is a psychological process which is universal, complex and multidimensional [1,2]. It is the combination of social attitudes that affect, either directly or indirectly, the person of focus, promoting loneliness and social isolation [3,4]. Therefore, not only must mental health patients endure the symptomatology and problems derived from their conditions, they must also deal with treatment from others, which is often based on prejudices and stigmatizing beliefs. This social rejection can be observed in various aspects of life, such as access to employment, housing, and specific services, and certainly in the development of meaningful interpersonal relationships. Moreover, such exclusion also deteriorates the self-image of these individuals, further contributing to their isolation [5,6].
Studies have also shown that people who manifest a high level of stigma towards individuals with mental disorders express similar feelings towards other groups, such as ethnic minorities or those promoting sexual diversity [7]. It is therefore important to research and also bear in mind the common aspects associated with stigmatizing processes [8].
observed that these scales displayed good internal consistency and correlated with other measures of psychological inflexibility and stigma (construct validity). In addition, AAQ-S is more strongly correlated with stigma-related measures (i.e., the Interpersonal Reactivity Index; IRI [27]; the Bogardus Social Distance Scale, SDS [28]; or the Scale of Ethnocultural Empathy, SEE; [29]).
The objective of the present study was to adapt and validate the AAQ-S of Levin et al. [26] to adolescence in the Spanish context. It was decided to work with the adolescent population for many reasons: firstly, because of the high prevalence rate of mental disorders in young people (it is estimated that in the last year, two million young people have suffered some kind of psychopathology [30]) which often carry stigma as a collateral problem that delays the search for help [31,32]; Secondly, because the instruments for measuring stigma in the adult population do not show appropriate psychometric properties for recommending their use in the adolescent population [33], and thirdly, because of the enormous number of studies that demand the need for intervention since young people are found to have important stigmatizing beliefs towards those who suffer mental health problems, such as being dangerous, unpredictable or incapable of assuming job responsibilities [34,35].
Thus, in order to provide the international community with an effective, valid and reliable tool, an exploratory factor analysis and a confirmatory factor analysis were carried out to test and confirm the factor structure of the scale. Subsequently, the reliability of the instrument was analysed using Cronbach alpha. Likewise, an invariance analysis was performed in order to test if the factorial structure of the questionnaire is understood in a similar way regardless of gender and sex, and a temporal stability analysis was also performed to determine whether the questionnaire is understood in a similar way by the same population despite the passage of time. Therefore, the hypothesis of the present study is that the confirmatory factorial analysis (CFA) of the proposed tool (AAQS-S) would offer adequate fit indices for a model with two correlated factors and said model would exhibit gender invariance and display adequate temporal stability.

Participants
The participants in this study were 1212 teenagers (646 men and 566 women). They were between the ages of 15 and 19 (M = 17.12; SD = 1.30). This sample was used for a confirmatory factorial analysis and belonged to three randomly selected youth associations and educational institutions from 10th grade to 12th grade in the Autonomous Region of Andalusia.
For the exploratory factorial analysis, an independent sample of 304 adolescents took part (154 male and 150 female), who were between the ages of 15 and 17 (M = 15.92; SD = 0.68). For the temporal stability analysis, an independent sample of 64 adolescents took part (34 men and 30 women), who were between the ages of 15 and 17 (M = 15.73; SD = 0.72).

Measures
The Acceptance and Action Questionnaire-Stigma (AAQ-S; [26]). For the purpose of measuring these factors, the Acceptance and Action Questionnaire-Stigma was validated and adapted. This questionnaire is comprised of 21 items divided between two factors: psychological flexibility (11 items) and psychological inflexibility (10 items which use reverse scoring). The subjects had to indicate their response according to a Likert scale from 1 ("Never true") to 7 ("Always true"). It should be noted that the elements on the subscale of psychological inflexibility were classified using reverse scoring, meaning that the low scores on this subscale indicate high flexibility with stigmatizing thoughts.

Procedure
The back-translation method [36] was chosen to adapt and validate the Acceptance and Action Questionnaire-Stigma. This strategy consists of two steps. Firstly, a group of translators with more than 10 years of experience and training in psychology carried out a direct translation of the questionnaire to Spanish. Next, a group of translators translated the items proposed in the first translation back to their original language. The degree to which they coincided with the original version was judged according to the goodness of fit [37]. Once the final Spanish version was obtained, a group of four psychologists with more than 10 years of research experience assessed the items proposed to determine whether the items were valid for teenagers.
Having obtained the definitive questionnaire, various high school and associations in the Autonomous Region of Andalusia were contacted. They were also informed of the goal of the research and asked to offer their collaboration. As the subjects were minors, they were required to obtain parental authorization to participate in the study. Prior to being administered, the scale was tested on a small group of people to ensure that all the items were understood correctly. The scale was applied with the insistence that responses would be anonymous and that there were no true or false answers, and that participants were only asked to respond honestly. The estimated time provided to complete the questionnaire was about 10 min.
This study was carried out in accordance with the recommendations of the American Psychology Association. The entire process was conducted in accordance with the Declaration of Helsinki. All the participants also gave oral informed consent. Ethics approval was obtained from the Research Ethics Committee of the University of Almeria, Spain (Ref. UALBIO 2019/014).

Data Analysis
In order to determine the validity and reliability of the Acceptance and Action Questionnaire-Stigma scale, their own psychometric properties were analyzed. Firstly, a descriptive statistical analysis and heterotrait-monotrait (HTMT), to assess the validity discriminant between the two factors, the score having to be less than 0.85 [38], were conducted; furthermore, the reliability of the tool was tested by means of an internal consistency analysis (Cronbach's alpha) and a temporal stability analysis (intraclass correlation coefficient, ICC) in order to provide evidence of the stability of the content of reports across time. Secondly, an exploratory factorial analysis (EFA) was carried out, followed by a confirmatory factorial analysis (CFA), for the purpose of testing the factorial structure of both the two-factor model and the higher-order model. Finally, a multigroup analysis was performed to examine any gender and age invariance present in the models. The statistical software used for data analysis was SPSS 24.0 (IBM, Armonk, NY, USA) and AMOS 19.0 (IBM, Armonk, NY, USA).
The sample was shown not to have a normal multivariate distribution since the Mardia coefficient proved high (86.56), the method opted for was the maximum likelihood estimation, in conjunction with a bootstrapping procedure for the CFA [39]. The estimators were not affected and can therefore be considered robust despite lack of normality [40]. With the aim of accepting or rejecting the tested model, a set of fit indices were taken into consideration: χ 2 /df, CFI (Comparative Fit Index), IFI (Incremental Fit Index), TLI (Tucker Lewis Index), NFI (Normed Fit Index), PNFI (Parsimonious Normed Fit Index), RMSEA (Root Mean Square Error of Approximation) and its 90% confidence interval (CI), and SRMR (Standardized Root Mean Square Residual). Given that χ 2 is highly sensitive to the sample size [41], χ 2 /df was utilized and any values below 5 were considered acceptable [42]. The incremental indices (CFI, TLI, NFI and IFI) reveal a good fit with values equal to or greater than 0.95 [43], while the error indices (RMSEA and SRMR) are considered acceptable with a value equal to or less than 0.08 [44]. Finally, the PNFI reveals a good fit with values equal to or greater than 0.70.

Analysis of Discriminant Validity, Bivariate Correlations, Descriptive Statistics and Reliability
The proportion of HTMT in the correlations between latent factors (Table 1) was 0.49, suggesting the existence of discriminant validity. Moreover, Table 1 shows the existing positive correlation between both factors, demonstrating the clear reciprocity between both factors. It must be noted that the elements on the subscale of psychological inflexibility were classified using reverse scoring, so the highest scores indicate more flexibility with stigmatizing thoughts. In addition, the average score was higher for psychological inflexibility than psychological flexibility. With the aim of obtaining evidence of the scale's reliability, an internal consistency analysis was conducted using Cronbach's alpha test. The scores proved satisfactory, with 0.97 for psychological flexibility and 0.95 for psychological inflexibility. Table 2 displays the correlations between each item and the total score of the scale, which was within a general range between 0.70 and 0.83. These results support maintaining all the items, considering the item-test correlation is higher at the cut-off point established at 0.30 [45]. The total Cronbach's alpha was 0.82. Furthermore, the exploratory factorial analysis proved the existence of two factors, revealing a saturation factor that ranges between 0.71 and 0.87 for psychological flexibility and between 0.72 and 0.84 for psychological inflexibility.

Confirmatory Factorial Analysis
The fit indices of the tested model ( Figure 1

Gender Invariance Analysis
A multigroup analysis was conducted to verify whether the factorial structure of the model exhibited gender and age invariance. Precisely as shown in Tables 3 and 4, no significant differences were found between Model 1 (model of unconstrained) and Models 2 (model of measurement weight) and 3 (model of structural covariances). However, the results did reveal significant differences between Model 1 and 4 (model of measurement residuals).
Factorial invariance tests were performed sequentially and hierarchically. First, the configural invariance (Model 1) was established, consisting of analyzing the level of adjustment achieved by imposing only the same factorial structure. This level of adjustment would indicate to what extent

Gender Invariance Analysis
A multigroup analysis was conducted to verify whether the factorial structure of the model exhibited gender and age invariance. Precisely as shown in Tables 3 and 4, no significant differences were found between Model 1 (model of unconstrained) and Models 2 (model of measurement weight) and 3 (model of structural covariances). However, the results did reveal significant differences between Model 1 and 4 (model of measurement residuals).  Factorial invariance tests were performed sequentially and hierarchically. First, the configural invariance (Model 1) was established, consisting of analyzing the level of adjustment achieved by imposing only the same factorial structure. This level of adjustment would indicate to what extent the dimensional model, in its configuration, is stable or invariant in both groups, as well as playing the role of baseline of the adjustment. We then proceeded to impose new restrictions on the model. Secondly, it was imposed that the estimated factorial weights be identical for men and women constituting a test of the metrical or factorial invariance (Model 2). Thirdly, to the above restrictions it was added that the variances and covariances of the factors have identical values in their estimation for men and women (Model 3). Fourth, the equality of the error variances was attributed to analyze the reliability or invariance of the items (Model 4).
The absence of significant differences between Models 1 and 2 constitutes a minimum criterion for accepting that the structure of the model exhibits gender invariance [46].
As for the higher-order model, no significant differences were found between Model 1 (model of unconstraints), Model 2 (model of measurement weights) and Model 3 (model of structural weights). The results showed significant differences between Models 1 and 4 (model of structural covariances), Models 5 (model of structural residuals) and 6 (model of measurement residuals). These results also support the presence of gender and age invariance in the higher-order model.

Temporal Stability Analysis
As for the temporal stability analysis, the intraclass correlation coefficients (CCI) were calculated, along with their confidence intervals (CI), providing a score of 0.87 (CI = 0.85-0.91) for psychological flexibility and 0.88 (CI = 0.84-0.90) for psychological inflexibility.
Regarding the higher-order factor called acceptance and action, it obtained a score of 0.96.

Discussion
The objective of the present study was to validate and adapt the AAQ-S of Levin, et al. [26] to Spanish (see, Appendix A) by following a translation and adaptation process in order to ultimately analyze its psychometric properties in a study on teenagers. The analysis of these psychometric properties made it possible to confirm that it is indeed an instrument that reveals evidence of validity and reliability for obtaining a measurement of psychological flexibility in relation to stigma towards mental disorders, displaying positive correlations between the various factors that comprise the questionnaire. This instrument would be of great use to future studies dealing with assessment and intervention, and, in any case, would be more predictive than the existing general measurement of psychological flexibility (AAQ and AAQ-II).
The results of the AFC revealed rather adequate fit indices, providing positive correlations between the two factors that comprise the scale. In fact, psychological flexibility and inflexibility subscales refer to different aspects of a single construct but they do not necessarily represent distinct poles on a single dimension. For example, low psychological inflexibility does not always indicate high psychological flexibility and vice versa. These results are coherent with other similar studies (i.e., [26]). On the other hand, the results of the internal consistency test showed Cronbach's alpha values which were greater than 0.85 on each of the two subscales (0.97 and 0.95 for psychological flexibility and inflexibility, respectively). As for the higher-order model, the values obtained were greater than 0.95. Thus, these data are consistent with the research of Wagnild and Young [47] and Vigário et al. [48].
Regarding temporal stability, acceptable values were obtained which were over 0.80 [49] on each of the two subscales and in the higher-order model. Furthermore, the stability proved invariant with respect to gender and age, which would enable it to be utilized in the future to conduct comparisons of psychological flexibility in relation to gender and age. We believe, therefore, that these results support the robustness of the scale, with the version adapted to Spanish successfully replicating the original theoretical structure. In short, this scale will prove useful as a relatively brief tool that is also easy to administer.
Taking into account the fact that this tool has shown itself to be more sensitive than the AAQ-II, correlating more strongly with other stigma measurements (e.g., [23,25]), it could be of great relevance when quantifying the impact of interventions focused on psychological flexibility towards stigmatizing thoughts.
Although the results of the present student provide psychometric support for the Spanish version of the AAQ-S, there are also some limitations that should be taken into account. Firstly, the present sample was a convenience sample and therefore, this pattern of results should be examined with other population groups that may vary by age, geographical location and other variables, given that validation of the instrument is an ongoing process. Second, future research could attempt to determine the predictive validity of the instrument through comparison with studies using similar scales (e.g., The Stigmatizing Attitudes Believability Scale [15]), as well as attempt to identify and determine psychological and behavioural predictors of prosocial behaviour, personal well-being, etc.

Conclusions
Based on the results of the present study, it can thus be concluded that the Spanish version of the AAQ-S constitutes a measure with adequate reliability and validity properties in relation to stigmatizing thoughts. Given that this population is acquiring and settling its beliefs and is therefore permeable to change, it is highly desirable to design specific interventions aimed at these stages of the life cycle [3]. The tool provided would be of great help in assessing and quantifying the magnitude of changes following anti-stigma interventions. It therefore follows that, in view to future studies, this questionnaire constitutes an invaluable tool when attempting to clarify the role of psychological flexibility in stigma, as well as the mechanisms behind the processes of change that are supposedly in action during anti-stigma interventions. Acknowledgments: Spanish Ministry of Science, Innovation and Universities.

Conflicts of Interest:
The authors declare no conflict of interest. Table A1. This questionnaire was validated in Spanish.

Appendix A
1 Mis presuposiciones y prejuicios afectan a la forma en que interactúo con personas de diferentes orígenes.
3 Dejo de hacer cosas que son importantes para mí cuando se trata de alguien que no me gusta.