Complexity, Pseudoscience and Rorschach Performance Assessment System

Objective: Complexity is one of the most important components of the Rorschach Performance Assessment System, R-PAS (the equivalent of general intelligence for IQ test). However Complexity is problematic on conceptual grounds and the objective of this research is to assess the scientificity of this notion. Method: The Belgian adult non-patient reference sample of the RCS was used in this study. (A) Data were analyzed according to R-PAS procedures, (B) protocols were randomly rescored and analyses were repeated; (C) analyses were compared. Results: Complexity main results were replicated with both real and random protocols. In addition, the Complexity of real and random protocols are highly correlated (r=0.82). Conclusions: Complexity is a pseudoscientific notion and the use of the R-PAS in clinical practice is questionable. The nature of the problem of R is discussed, and means to address it are proposed. *Corresponding author: Patrick Fontan, Ipsé Laboratory, Office C110 Paris Ouest University, 200 Avenue de la République, 92000 Nanterre, France, Tel: +33 1 40 97 72 00; E-mail: fontan.patrick@gmail.com Received: November 21, 2016; Accepted: December 16, 2016; Published: December 23, 2016 Citation: Fontan P, Andronikof A (2016) Complexity, Pseudoscience and Rorschach Performance Assessment System. Clin Exp Psychol 3: 145. doi: 10.4172/2471-2701.1000145 Copyright: © 2016 Fontan P, et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.


Introduction
A few years after the Rorschach test was translated in English in 1942 [1]. Lee Cronbach noticed that response frequency is problematic in Rorschach data [2]. Indeed, most Rorschach interpretations are based on counts of response characteristics, whereas the total number of Rorschach responses is variable. For example the sum of human contents responses SumH is thought to assess interest in others [3,4] but it is highly correlated with the total number of responses R given to the test (r=0.60). Consequently, a higher value of SumH might simply be the result of a higher productivity (the tendency to give many responses) rather than a reliable measure of interest in other people. This issue -the problem of R -is relatively pervasive in Rorschach data and research and it has been much discussed in the literature [2,[5][6][7][8][9][10][11][12].
In the Rorschach Comprehensive System (CS) [3] the problem of R is not addressed in a systematic manner. Percentage scores are simply divided by R (XA%, WDA%, X-%, X+%, Xu%, EGO, Isolate), ratio scores compare two mutually exclusive class of responses (Lambda=F/ (R-F), Afr=R8910/(R-R8910)), the cutoff values for some scores depend on R (Human Contents, pureH), while the problem of R is not addressed for most RCS scores. Though John Exner considered the problem of R as a "ghost" [5], factor analytic studies have shown that the first dimension of Rorschach data (i.e., the most important one) is essentially defined by R, suggesting that the problem of R is substantial and should be addressed more systematically, which is precisely one of the main purpose of the newly developed Rorschach Performance Assessment System (R-PAS) [13]. The first dimension of the Rorschach has gone under different names: Productivity, R factor, Task-Engagement, Rorschach 1 st factor and Complexity. In this paper we will refer to it as "Complexity" as it is the current designation of this dimension.
Improving on the CS, the R-PAS addresses the problem of R by two additional means: R-Optimized Administration and Complexity Adjusted scores. (a) Actually, the R-Optimized procedure (push for two responses per card, pull out after four) limits response range so as to avoid very short protocols (which are likely to be invalid) and very long protocols (which produces many outliers in Rorschach variables). Though the R-Optimized procedure limits the variability of the number of responses, R remains a variable which is correlated with many R-PAS scores. (b) Complexity is an original index of the R-PAS. It corresponds to the sum of all contents but single animal, all determinants but pure Form responses, ordinarily developed responses, two times synthesis between details responses and three times synthesis responses in whole or space locations [13]. This index has been developed to closely estimate the Rorschach first factor, the correlation being r=0.95 [13]. In the R-PAS it is possible to adjust all test scores for the level of Complexity of a protocol so as to minimize extraneous variance associated with R. In addition, Complexity is "one of the most important components" [13] of the R-PAS interpretation (like g on IQ test or Welsh's A on the MMPI-2).
In spite of the importance of Complexity for the Rorschach and the problem of R, this dimension is problematic on conceptual grounds. Indeed, it is not clear whether this dimension is a psychological construct or a statistical artifact. As stated by Gregory Meyer in his factor analysis of the Rorschach.
This factor is difficult to conceptualize as a single psychological continuum because it is defined by such diverse variables. It may be considered an articulation factor that reflects a tendency to see and describe many qualities of the inkblots. However, more simply, Factor 1 may just reflect the fact that the number of times one responds to the Rorschach determines to a large degree the number of determinants that will be found across all determinant categories. As R increases so does the general frequency of all determinants and as R decreases so does the general frequency of all determinants. If this is the case, Factor 1 can be considered a statistical artifact of Rorschach administration procedures [10].
To our knowledge, this specific problematic (does Complexity corresponds to a psychological construct or a statistical artifact?) has never been addressed in the literature. If Complexity was indeed artificial, our understanding of the Rorschach (or more precisely of its dimensional structure) should be revised in substantial ways, given the importance of this dimension. The objective of this research is to study this question using randomly rescored Rorschach protocols.
Complexity could be artificial because it might represent "a factor that reflects the somewhat tautological position that frequent responding leads to increased scoring across all determinants categories" [10]. Tautologies are a circular form of reasoning in which the conclusion is equivalent to the premise. An important property of tautologies is that they are always true whatever the data [14]. A consequence of this property is that tautologies are meaningless or "devoid of content" [15]. Indeed, since the truthfulness of tautologies do not depend on data, they cannot have any empirical meaning.
If Complexity was tautological, we hypothesize that (a) it should be possible to replicate results concerning Complexity using randomly rescored Rorschach protocols (true whatever the data). In addition, (b) if Complexity did not depend on data, the Complexity of Rorschach protocols should be highly correlated to the Complexity of those same protocols which have been randomly rescored (no empirical meaning).

Sample size
According to Guadagnoli and Velicer, if "components possess four or more variables with loadings above .60, the pattern may be interpreted whatever the sample size used" [16]. R-PAS manual reports correlations above 0.60 with Complexity for the 5 individually coded variables: number of responses R (0.79), multiple determinant response Blend (0.82), synthesis responses Sy (0.83), human movement responses M (0.68) and human contents SumH (0.70) (All protocolsnorm sample) [13].

Participants
The Belgian adult non-patient Rorschach CS reference sample was used in this study. Detailed information concerning recruitment procedures, examiners, administration and scoring procedures is available in the original article [17]. The Belgian database was used primarily because it is available in CHESSSS (an open-source software for the Rorschach CS) [18], which allowed us to generate random protocols. The original database included 100 participants and 98 were used for this study (one protocol included more than 50 responses and could not be scored in CHESSSS, and one other protocol was found to be duplicated).
Participants were recruited by graduate psychology students who administered the RCS under the supervision of senior staff members of the Clinical Psychology Service of the University of Liège. All protocols were administered and scored according to the RCS [19] and none has less than 14 responses. Concerning interratter reliability, 25 protocols were double-scored and all kappa coefficients were above 0.75 (Table  1), which is considered excellent [20].

Randomly rescored protocols
The database used in this study includes 98 participants and 2365 responses which are all coded for the following scoring categories (a) Location and Developmental Quality, (b) Determinants, (c) Form Quality, (d) Pair, (e) Contents, (f) Popular, (g) Z score and (h) Special Scores. Randomly rescored protocols were generated as follow: (1) The first protocol (according to participants IDs) is selected from the 98 available, (2) The scoring of the protocol responses is deleted, (3) For the first response, a Location and Developmental Quality is randomly selected among the 2365 available responses in the Belgian database, then this procedure is repeated for the other scoring categories (Determinants, Form Quality, etc.), (4) Once the first response is scored for all scoring categories, the protocol second response is randomly rescored in the same manner and so on until every responses of the selected protocol are randomly rescored, (5) These steps are repeated for the second protocol and so on until every protocols in the database are randomly rescored. These protocols will simply be called "random protocols" in the rest of this paper.
This procedure is based on a Bayesian approach in which the probability of occurrence of the different RCS codes is not inferred according to a statistical law (uniform or normal distributions), but rather according to the actual distribution of these codes observed in the Belgian sample.

Analysis
A principal component analysis (PCA) was conducted on the counts of individually coded variables (i.e., W, D, F, P etc.) of the real protocols of the Belgian sample. However, some variables of the contents category have been regrouped into broader categories so as to simplify the results (e.g. Human Contents (Table 2). In accordance with the R-PAS manual, the first unrotated component was extracted [13], and component scores were saved. The same analysis was performed on the random protocols. There is confusion in the R-PAS between factor analysis and PCA. Indeed, the first dimension of the Rorschach is called "Rorschach first factor", while it is actually the result of a PCA, which makes it the "Rorschach 1 st Component". Finally, the correlations between the Complexity and the Rorschach 1 st component of real and random protocols were computed.

Results
The first unrotated component of real Rorschach protocols is essentially defined by the total number of responses (R is loaded at 0.95 by this component), and most Rorschach variables are significantly and substantially correlated with this component ( Table 2). The correlation between Complexity and the Rorschach 1st component of real protocols is r=0.95 (Table 3). The R-PAS manual reports a correlation of r=0.95 between these two variables.
Note: 25 cases of the Belgian sample (25%) were scored independently by two judges   The first unrotated component of random Rorschach protocols is essentially defined by the total number of responses (R is loaded at 0.99 by this component), and most Rorschach variables are significantly and substantially correlated with this component ( Table 2). The correlation between Complexity and the Rorschach 1st component of random protocols is r=0.97 (Table 3).
The overall pattern of component loadings are very similar for real and random protocols with the exception of 4 variables: W, SumV, F, P ( Table 2). The first component of real and random Rorschach protocols are essentially equivalent (r=0.95) and the Complexity (which is an estimate of the Rorschach 1 st component) of real and random protocols are highly correlated (r=0.82).

Complexity is a pseudo-scientific notion
The first unrotated component of randomly rescored Rorschach protocols is essentially defined by the total number of responses data, but rather that the problem of R should be addressed prior to any analysis.
We strongly recommend controlling Rorschach variables for R directly. It can be done by three different means which are equivalent in their principles : (a) computing the residuals of the regression of Rorschach variables using R as the predictor (b) computing the semipartial correlation between Rorschach variables (controlled for R) and external validity criterion and (c) computing partial correlations between Rorschach variables (controlled for R). In all cases, the correlation between the total number of responses R and other Rorschach variables will be exactly equal to zero, which completely solves the problem of R. This approach might be problematic because many Rorschach variables are not normally distributed (positively skewed), and therefore parametric statistics would not be appropriate. In this case, transformations might be used in order to get more normal distributions (rank order, square root, etc.). One interesting advantage of this approach is that R would not be considered as a confusing factor anymore, but as a potentially informative variable of the Rorschach which could be studied in itself (in relation to different psychiatric diagnosis, examiner expertise, clinical settings to give few examples). and is correlated with most Rorschach variables, just as it is the case with real data. In addition, the Complexity of real and randomly rescored protocols are highly correlated (r=0.82). Results indicate that Complexity should be considered as a statistical artifact rather than a psychological construct. Indeed, since the number of responses of a Rorschach protocol remains constant whatever the scoring, Complexity -the most important dimension of the R-PAS, the equivalent of g for IQ test -hardly depends on the scoring of Rorschach responses. Complexity scores of accurate Rorschach protocols are very similar to those of randomly rescored protocols which mean that this variable does not reflect psychological characteristics of a person.
Evidences concerning Complexity present the two main characteristics of tautologies: they are true whatever the data and they have no empirical meaning. These conclusions have important scientific implications: as tautologies cannot be refuted by any possible event, they are actually pseudoscientific notions according to Karl Popper falsifiability criterion [21]. Therefore, "one of the most important components" of the R-PAS interpretation [13], is actually pseudoscientific.

The nature of the problem of R
Correlations between the total number of responses given to the Rorschach R, and Rorschach scores have been frequently reported, suggesting that the problem of R was of an empirical nature. The results of this study tend to infirm this notion as it is possible to find similar results with random Rorschach protocols. Since Complexity seems to be tautological, its truthfulness might not rely on evidences but rather on definitions, in this case on the Rorschach scoring system. For example, in most Rorschach systems every response is scored for locations. There are three codes for locations: Whole (W), Detail (D) and unusual detail (Dd) which are mutually exclusive, consequently R=W+D+Dd by definition, and whatever the scoring accuracy (true whatever the data). This point (i.e., multicolinearity) explains the tautological nature of Complexity. By definition, R is also equal to the sum of Developmental Qualities, the sum of Form Qualities, the sum of the primary determinants and the sum of the primary contents. Prior to the analysis, we know that R is directly or indirectly mathematically related to other Rorschach variables and after the analysis we learn that R is more or less correlated to most Rorschach variables.

Tall people have long bones…
The problem of R can somehow be compared to an anatomical issue. Computing correlations between R and other Rorschach variables is similar to computing correlations between the height of a person and the length of his bones. The correlations are expected to be very high, but if morphology has to be studied, height should be controlled in a way or another as a factor analysis showing that tall people have long bones would not be very helpful to study morphological variability. Ideally, height should be made independent from morphological issues which can be done statistically.

Addressing the problem of R
The results of the present study and their consequences demonstrate how pervasive the problem of R is in Rorschach data. It appears clearly that this issue cannot be considered as a "ghost", and that it needs to be addressed systematically on statistical grounds. However, from this point of view, factor analysis and principal component analysis of the Rorschach led to the apparently misleading idea that the most important dimension of the test is essentially related to response frequency. These points do not mean that factor analysis is not applicable to Rorschach