Construction and Clinical Validation of a Questionnaire-based Risk Score to Identify Patients Suffering from Immunodeficiency or Systemic Autoimmunity

This work was carried out in collaboration between all authors. Authors HHP and SG designed the questionnaire. Authors SG, JH, HHP, KG, KW, BG and WG collected clinical data. The ISAQ database was established by author SR. Statistical data analysis was performed by authors WV and AN. Authors HHP and WV wrote the manuscript and author BG gave critical input to the manuscript. All authors read and approved the final manuscript. ABSTRACT patients and normal adults. collected Methodology: Two experienced clinical immunologists selected 17 informative topics including frequency and duration of infections, previous surgery on immunological organs, recent polytrauma, vaccination history, allergies, co-morbidities, use of antibiotics and immunomodulating agents. The ensuing questions were differently weighted based on published evidence and assumed relevance for immunological dysfunction. The questionnaire was distributed to 539 cases suffering from chronic immunodeficiency (n=322) or autoimmunity (n=217) and to 1020 healthy controls. An ISAQ score was calculated for each participant based on the sum of the weighted items. Results: The median ISAQ-score was 39.5 for cases and 30.4 for controls. 93.7% of all controls scored lower than the median of the cases, which resulted in an AUC of the ROC curve of 0.838 (95%CI: 0.817-0.859). Items with highest predictive value were frequency and duration of infections, use of antibiotics, corticosteroids, immunosuppressants, sinus-mucosa resection, chronic disease of kidney, lungs, gut and hematopoietic system. Although the results indicate that neither the item selection nor the weighting of single items were optimal, several sub-scores with reduced item number could not substitute for the whole ISAQ. Conclusion: The ISAQ can discriminate between healthy individuals and persons suffering from immune-based disease. The ISAQ performs better for chronic immunodeficiency than for chronic rheumatic diseases; the highest values were obtained for common variable immunodeficiency (CVID) and systemic vasculitides. Future studies are needed to refine item selection and validate the score for predicting immune dysfunction.


INTRODUCTION
There exists a large body of knowledge about development, structure, organization and function of the human immune system. The interplay of three main components is essential for its proper function: i. the protecting body surfaces, ii. the innate immune system, which is composed of phagocytes, dendritic cells and NK cells; the innate system senses biological danger by recognizing pathogen associated molecular structures (PAMS) via pattern recognition receptors (PRR, e.g. toll-like receptors), and iii. the adaptive immune system represented by T and B cells, exerting the role of a back-up defence system against viral and bacterial infections in case the innate system has been overwhelmed by pathogens [1]. Inherited or acquired disturbances of the immune system translate into clinical symptoms whereas as a normally functioning immune system goes asymptomatic [2]. With the exception of a few disease-related immune function scores [3][4][5], there has been to our knowledge, little attempt to translate symptoms of immune dysfunction into comprehensive questions suitable for a self-assessment questionnaire. Here we present a novel tool for selfassessing the functionality of the immune system. The selected questions address topics based on a long-standing clinical experience of two co-authors (HHP, SG) working for up to 30 years in an immunodeficiency and rheumatology outpatient clinic. The questionnaire was designed to test for the integrity of body surfaces, surgically removed immune organs, lifestyle impacts and chronic use of antibiotics or immuno-suppressive drugs. In addition, the individuals were asked for frequency and duration of infectious episodes, indicator infections, vaccination history and co-morbidities. To test this novel immune system assessment questionnaire (ISAQ) for its ability to identify subjects with increased risk of immune-based diseases, we applied the ISAQ to several cohorts of patients with an established diagnosis of chronic immunodeficiency or autoimmunity (n=539) as well as to different cohorts of healthy controls (n=1020). We observed a distinct difference of the ISAQ scores between cases and controls, indicating that the new tool may be suited to identify individuals at risk for immune-based diseases, particularly immunodeficiency syndromes.

Recruitment of Patients and Controls
All cases and controls were adults registered in 4 age groups: ≤30, 31 to 45, 46 to 60 and >60 years. They gave informed oral consent to respond in pseudonymized or anonymized form to the ISAQ questionnaire, which was approved by the Ethics Committee of Freiburg University Medical Centre (UMC) (Project Nr. 174/08). With the exception of 79 Sjögren's, 16 SLE and 33 ANCA-positive vasculitis patients who were enrolled during patients' organization reunions, all cases were recruited from the outpatient clinic of the Dept. of Rheumatology and Clinical Immunology of the Freiburg UMC. All patients had an established diagnosis; they were not newly diagnosed but rather under long-term treatment. For patients recruited at the three patients' organization reunions, the diagnoses were selfreported and not verified at our outpatient clinic.

Healthy controls
1020 apparently healthy individuals were recruited at different sites and occasions throughout Germany. The first recruitment was organized in May 2005 during the 11 th Health Days of the Dr.ing.h.c.F.Porsche AG, Stuttgart for the employees of the car manufacturer company (P1-workers, n=363). Subsequently, the occupational physicians of the Porsche company (J.H. and K.G.) recruited among employees another 94 controls (P2-workers) who were seen for minor injuries and a third group (P3-workers n=31) who consulted them for common cold infections [6]. Two sampling events were organized in Freiburg at a "Fit-forlife" Fare in 2011 (n=145) and at a University Science Day in 2006 (n=94). Other recruitment sessions were organized during the Annual Meeting of the German Society for Immunology 2007 in Kiel (Kiel DGfI n=33) and a junior immunologist workshop 2009 in Jena (Jena DGfI n=54) as well as at Freiburg UMC (UMC students n=182; UMC employees n=25). Data collection was performed on paper or via an online questionnaire and entered into the ISAQ database hosted at the UMC Freiburg.

Common variable immunodeficiency (CVID)
Common variable immunodeficiency (CVID) was the largest and best-characterized cohort of patients enrolled in the study (n= 145). They have been followed between 2005 and 2010 in our CVID outpatient clinic and were classified according to circulating B cell phenotypes [7], distinguishing four subsets of patients (Freiburg classification): All patients suffered from recurrent respiratory or gastrointestinal infections and had low serum immunoglobulin levels of at least two isotypes. In addition, some patients presented with autoimmune phenomena, lymphoproliferation, granulomas and secondary malignancies. CVID Type I exhibits reduced numbers of circulating switched memory B (smB) cells and can be subdivided into a more severe form Ia characterized by an increased proportion of CD21 low B cells, frequent splenomegaly and autoimmune phenomena and form Ib with normal CD21 low B cells numbers. CVID type II patients have normal numbers of smB cells and usually exhibit a less severe clinical phenotype. The 4 th group of patients is unclassifiable by the Freiburg classification protocol due to the lack of circulating B cells (<1%). Among the 145 CVID patients were also included two patients with defined monogenic defects described for CVID (Baff-R, and TACI deficiency), four with hyper-IgM syndrome (type 1), two Good syndrome patients and one X-linked agammaglobulinemia patient (M.Bruton). All patients were under regular intravenous (IVIg) or subcutaneous (SCIg) immunoglobulin replacement therapy. In a pseudonymized subset of 122 CVID patients composed of 46 type Ia, 46 type Ib, 13 type II and 17 unclassifiable patients the ISAQ was separately analysed and compared among CVID subtypes.

Selective IgA, IgM or IgG subclass deficiencies (C/ScD) (n=33)
These patients presented with increased respiratory tract infections and reduced serum level of one or two IgG subclasses and were grouped together as antibody deficiencies without need of IVIg substitution. The group comprised also a patients with selective IgA deficiency (n=4) or low IgM serum concentrations (n=3).

Diverse, so far unclassified immunodeficiencies (DuID)
Diverse, so far unclassified immunodeficiencies (DuID) (n=80) which did not fit into the CVID and C/ScD groups but consulted our outpatient clinic for frequent infections of diverse nature, mostly respiratory tract infections. None of them received IVIg substitution therapy.

HIV patients
HIV patients (n=64) were recruited from our HIV outpatient clinic; all received highly active anti-retroviral therapy but no IVIg replacement therapy.

Rheumatoid arthritis (RA)
Rheumatoid arthritis (RA) patients (n=43) were recruited from our rheumatology outpatient clinic. They were treated according to current protocols including low-dose corticosteroids, immunosuppressants, and in a minority of the cases also biologicals (Anti-TNFalpha).

Systemic lupus erythematosus (SLE) patients
Systemic lupus erythematosus (SLE) patients (n=45) were either seen in our SLE outpatients clinic (n=28) or recruited during a patient organization reunion (n=17). Most of them (>90%) had stable disease under low dose steroids and varying degrees of immunosuppressants (hydroxychloroquine, azathiporpine or mycophenolate mofetil).

Sjögren's syndrome patients
Sjögren's syndrome patients (n=87) were mainly (n=79) enrolled during a National Sjögren's Day organized by the German Sjögrens' patient organization in Freiburg 2008. The remaining 8 patients were recruited from our Rheumatology outpatient clinic. Treatment modalities were mainly symptomatic (artificial tears, saliva spray and Salagen®); a minority (<10%) received low-dose steroids and immunosuppressants.

Vasculitis patients
Vasculitis patients (n=42) were mainly (n=33) recruited during an Annual Vasculitis Day 2011 organized by a vasculitis patients' organization and the Vasculitis Center of Lübeck/Bramstedt (Head: Prof. W. Gross). The remaining 9 cases were seen at the Vasculitis outpatient clinic of the UMC Freiburg. The majority (>90%) of the patients suffered from ANCA positive vasculitides and were on regular low-dose corticosteroids plus immunosuppressant.

Item Selection and Composition of the Immune System Assessment
Questionnaire (ISAQ)

Process of item selection
The questionnaire for self-assessing the immune system asks for items thought to depend on or influence the function and integrity of the immune system such as frequency and duration of infections, indicator infections (e.g. septicemia, bronchiectasis, recurrent bronchopneumonia, tuberculosis, salmonellosis, HIV, chronic viral hepatitis, meningitis a.o.), vaccination history and co-morbidities. In addition we designed questions to address the integrity of the four body surfaces (skin, respiratory tract, gastrointestinal tract, urogenital tract), recorded surgically removed immune organs, lifestyle impacts, chronic or recurrent use of antibiotics, immunosuppressants, radiation exposure and use of immunostimulants.

Construction of the ISAQ score
For the binary items, a negative response got always the weight 0.5, whereas a positive response received the weight 4 or 2, respectively, reflecting whether we regarded the items as more or less important. Items on ordinal or categorical scales were always reduced to a binary item or a trichotomous item. In the latter case the weights 0.5, 2 and 4 were used. All weights are shown in Table 2. The ISAQ score of an individual represents the sum of the weighted items.

Statistics
As a first test for the potential value of the new tool, we compared the distribution of the single items as well as of the overall score between the patients (termed "cases") and the controls. To evaluate the value of the single items we considered the prevalence, positive and negative predictive values, sensitivity, specificity, and the odds ratio. The odds ratio is supplemented with a p-value of testing the null hypothesis of no association. For nondichotomous items we considered the categorization used in the weighting process. If the latter resulted in a trichotomous item, we considered both possible dichotomizations. For some of the categorizations we considered additional dichotomizations. To compare the value of the different items the positive predictive value is plotted against the prevalence. For items with a similar prevalence those with the higher positive predictive value are more useful for inclusion into the score. On the other side, for items with similar positive predictive value, those with the higher prevalence are more useful, as they allow identifying a higher number of patients at risk. For some items (antirheumatic drugs, immunosuppressants, cytostatic drugs, immunoglobulin) and some patient groups (RA, CVID) a positive response to the item is more or less a precondition for the diagnosis or it reflects a typical treatment for the patient group. In these instances we omitted the corresponding patient groups from the predictive value calculation. Information on these items and the corresponding subgroups are given in Table 3. The item on HIV was excluded from all analyses, as only one subject outside of the HIV patient group reported a positive test result. To evaluate the value of the ISAQ score, as well as the selected sub-scores, receiver operating characteristic (ROC) curves and corresponding area under the curve (AUC) values were considered. To take the above-mentioned relationship between certain items and patient groups into account, these items did not count in computing the scores for patients from the specific subgroups. The HIV item was not used in computing scores, and neither was the item corticosteroid use, since cortisone is a typical treatment in many patient groups suffering from autoimmunity. To evaluate the value for specific case groups, we also performed analyses comparing the individual case groups with all controls.
For all quantities considered (prevalence, predictive values, sensitivity, specificity, odds ratios and AUCs) we obtained age and gender adjusted values by stratification and averaging over the strata specific results. We used eight strata defined by gender and age with cut points 30, 45, and 60 years, and used weighted averages weighting each stratum with the fraction of subjects of the whole German population in the corresponding group, assuming an overall age range from 18 to 80. If there were strata with no variation of an item such that one predictive value could not be computed, we coarsened the age stratification to two groups based on the cut point 45 years. If this was not sufficient, we adjusted only for age, but not for gender, using two age groups. For the odds ratio we performed these calculations on the log odds scale. For the log odds ratios and the AUC the standard error of the adjusted values were computed by combining the standard errors from each stratum appropriately. If the standard error was undefined in a stratum, we used again coarsened strata. The standard error of the adjusted log odds ratio was used to compute a p-value using the Wald test principle. Confidence intervals for the AUC were computed as +/-1.96 times the standard error.
To evaluate the potential value of the ISAQ when using an optimal weighting, we used a logistic regression model including all (dichotomized) items and entering the ISAQ-score together with a five-fold cross validation. All computations were performed using Stata 12.1. [8].

Subject Characteristics
Overall 1020 controls and 539 cases were included in our analysis. The size of the different control and case populations, the distribution of gender and age in each population as well as for all controls and cases are shown in Supplementary Table 2. Females constitute 2/3 of all cases , but only 1/3 of the controls. Young patients were overrepresented in the controls (36%) compared to the cases (14%). The substantial difference in age-and gender distribution between cases and controls reflected the different recruitment strategies for the two groups. We took this into account by adjusting all measures considered for age and gender, and found out that this has little impact on the results. Smoking was slightly more prevalent among cases than controls (16.1% versus 13.5%). Body mass index (BMI) distribution in cases and controls was comparable with a tendency of more subjects with BMI <18.5 among cases (7.5%) compared to the controls (3.2%).

Overall Value of the ISAQ
The distribution of the ISAQ scores in cases and controls (Fig. 1, left) reveals that most control individuals have a score below 35, whereas the majority of cases score 35 or higher.
Actually, the ISAQ median in the cases is 39.5 and that in controls is 30.4. As 93.7% of all controls score lower than the median of the cases, this discrepancy resulted in an AUC of the ROC) curve of 0.838 (95%CI: 0.817-0.859) (Fig. 1, right).

Validation of Items
The diagnostic value of each individual item is summarized in Table 3 and illustrated in Fig. 2 as scatter plot of the positive predictive value against the prevalence of each item. There were minor differences between age and gender adjusted and unadjusted values ( Table 3, see comparison of adjusted and unadjusted odds ratios).
For some items there was a distinct discrepancy between the original decision to assign the item a low or a high weight and the actual positive predictive value relative to the prevalence. Thus chronic kidney disease, antibiotic use, diarrhoea, and inflammatory rheumatic disease got originally a low weight although they turned out to be among the items with a high predictive value compared to items with similar prevalence. Similarly, duration of infections of 3 weeks seems to have already a similar value than duration of more than 3 weeks. Items with much lower predictive value than anticipated were splenectomy, M. tuberculosis infection, chronic hepatitis, vaccination complications, septic arthritis and in particular, radiation therapy. The latter was actually more frequently reported by controls compared to cases. Fig. 2 illustrates the general difficulty to find items with a high positive predictive value and a relevant prevalence. From this perspective, chronic systemic vasculitis, a frequency of 4 or more infections per year, duration of infections of 3 or more weeks, inflammatory rheumatic disease, gastrointestinal and lower respiratory tract infections are the most promising candidates.
Since for several items the a priori weighting in the ISAQ-score seems suboptimal, an optimal weighting based on a logistic regression approach was investigated. This led to an increase in the AUC of 0.06.

The Value of ISAQ for Specific Patient Groups
In further analyses the entire control group was compared with eight major patient subgroups as well as the two larger categories of autoimmune diseases (SLE/RA/Sjoegren/vasculitides) and immunodeficiencies (CVID, C/ScD, DuID) seperately. The ISAQ score performed best for CVID and chronic vasculitis patients, and worst for rheumatic diseases and HIV patients. The score seems to better characterize individuals with immunodeficiencies than with autoimmune diseases. Within the clinically bestcharacterised group of 122 CVID patients there was a trend towards a better discrimination in the most severe CVID subset Ia (Table 4 above). However, when stratifying by age or gender no substantial differences in the AUC values could be observed (lower part of Table 4).

The Value of ISAQ Subscores
As mentioned in section 3.1., the ISAQ consists of different parts, which allow to define subscores for the topics #5 (injuries to the immune system), #12 (localisation of infections), #14 (indicator infections), and #15 (co-morbidities) of the ISAQ (see Table 1). Moreover, some of the single items provide quantitative information, in particular the questions on the frequency and duration of infections (topic 11). In Fig. 3(above) we assess the value of the four different sub-scores and these two single items of the ISAQ. All sub-scores perform distinctly worse than the overall score, but with very similar AUC values in the magnitude of 0.65 to 0.70. On the other side, both single items on the frequency and duration of infections obtain already an AUC of nearly 0.7.

DISCUSSION
To our knowledge the ISAQ study is the first attempt to translate modern immunological knowledge and long-standing clinical experience in immune-based diseases into a questionnaire for self-reporting of signs, symptoms, anamnestic features and lifestyle factors. The underlying hypothesis was to construct with this information a score that would be able to identify individuals at risk for immune-based diseases, preferentially immunodeficiency. The selected items have not been validated previously, but were solely selected on the basis of long-standing experience in clinical immunology and rheumatology of two co-authors (HHP, SG). Cohorts of established systemic autoimmune disease and immunodeficiency syndromes served as internal disease controls. The latter ones were grouped into patients with hypogammaglobulinemia of at least two isotypes (e.g. agammaglobulinemia, CVID) requiring intravenous immunoglobulin (IVIg) substitution, Ig class or subclass deficiency (C/ScD) without need of IVig substitution and diverse, unclassified cases (DuID) with increased susceptibility of infections, normal serum Ig levels and no IVIg treatment. Our investigation revealed that the ISAQ-score is to some degree able to discriminate between healthy subjects and patients suffering from chronic immunodeficiency and/or autoimmune diseases. Healthy subjects tend to have rather rarely high values of the ISAQ-score (and some of them may actually suffer from undiagnosed immune-based diseases) whereas in patients with diagnosed disease a substantial fraction scored rather low, suggesting the ISAQ has a high specificity, but only a moderate sensitivity to detect immune dysfunction. Taken into consideration the enormous complexity of the immune system with its multiple interactions, back up loops and memory functions involving barrier defence mechanisms as well as innate and adaptive immune compartments, a low sensitivity of the ISAQ was to be expected. Thus, a patient with impaired specific antibody formation (such as in CVID) may still be sufficiently protected by proper barrier and innate immune functions as long as the individual is not repeatedly exposed to a high load of virulent pathogens. Conversely impaired barrier and innate immune functions may go along with normal antibody levels not preventing proneness to infections in the "diverse unclassified immunodeficiencies" (DuID) cohort with normal serum Ig levels and in the ANCA associated "chronic vasculitides" with their highly vulnerable mucous membranes and an ANCA-activated neutrophile compartment. The relative high specificity of the ISAQ for immune-based diseases in general might be explained with the remarkable contribution to the total score of some single items such as frequency and duration of infections and antibiotic use. On the other side an unexpected poor predictive contribution was seen for previous injuries to the immune system such as radiation therapy, appendectomy, tonsillectomy and splenectomy, not justifying an increased weighting factor. A possible reason for this may be due to the fact that our question on "radiotherapy" ("Wurden Sie in den letzten beiden Jahren bestrahlt?" "Did you receive irradiation during the last two years?") did not specify ionising radiotherapy and therefore might have been misinterpreted for any type of irradiation. A similar misinterpretation of the question on resection of lymphoid tissue is less likely, therefore the low predictive value of previous appendectomy, tonsillectomy and splenectomy is probably real whereas surgical resection of sinus mucosa gave a good predictive value (Fig.2). Single items with unexpectedly high predictive values were chronic disease of kidney, lungs, hematopoietic system and inflammatory bowl disease without, however, reaching that of chronic systemic vasculitis (Fig. 2, Table 3). Taken together the single item analysis allows us to rank the various questions with respect to their predictive value and diagnostic strength.
Although the ISAQ performs rather uniformly over many different subjects, age and gender groups it seems to be better targeted for immunodeficiency syndromes (notably CVID) when compared to autoimmune diseases such as SLE, RA and Sjögren's. This tendency becomes even more relevant when keeping in mind the high single item contribution of immunosuppressive drugs, which are usually given in autoimmune diseases and not in immunodeficient patients.
We could not find evidence for any subsection of the ISAQ alone being able to substitute for the whole IASQ. However, some single items of the ISAQ, in particular those on frequency and duration of infections, antibiotic use and the above mentioned chronic renal, pulmonary and haematological co-morbidities, provide already alone valuable information. This may suggest the application of a short form of the ISAQ with a few questions if the full ISAQ cannot be applied due to limited resources.
A basic limitation of the study is the retrospective nature of our data and the lack of a blinded cohort of patients. The ISAQ was applied in patients with a known diagnosis, and the response of the patients to some items might have been different prior to the diagnosis. We tried to take this into account by different weighting factors and by not counting some items in some patient groups, where the item reflected either a precondition or a typical treatment (e.g. IVIg therapy in CVID, immunosuppressive treatment in RA). In any case, the ISAQ requires further validation, ideally in prospective studies, where subjects with a high risk according to the ISAQ are clinically examined and followed up for several years for immunebased diseases and incident infections.
A further limitation of our study is the substantial difference in age-and gender distribution between cases and controls, reflecting the different recruitment strategies for the two groups.
We took this into account by adjusting all measures considered for age and gender, and actually we found out that this has little impact on the results.
The ISAQ in its present form includes several items of limited value. Moreover, the weighting of items seems to be suboptimal and the formulation of some questions needs to be more succinct. A logistic regression model using all items suggests potential for optimizing the weighting factors. The construction and validation of an optimized version of the ISAQ and its scoring logic should be part of further prospective studies or of retrospective studies with careful selection of cases and population-based controls. To the best of our knowledge a similar validation of questions related to the functioning of the immune system have not been undertaken in previous studies. In the field of rheumatic diseases self-reporting questionnaires have been proposed and validated [9,10]. Moreover, recently the RABBIT risk score for serious infections applicable in RA prior to the switch from conventional disease modifying anti-rheumatic drugs (DMARDs) to anti-TNFa blocking agents has been validated and proven to be useful [11,12]. However, this score included information on previous and actual treatment and is hence not a self-reporting, diagnostic tool for a general assessment of the immune system. An improved ISAQ score, on the other hand, may serve as a valuable tool to stratify patients in clinical studies involving immunomodulatory or immunosuppressive treatments.
It remains an open question why the ISAQ failed to identify a substantial fraction of the cases. This may reflect subgroups of patients with no observable signs (e.g. remission), other signs than covered by the ISAQ or patients unable or unwilling to respond to items of the ISAQ. Also this should be a topic of future research.

CONCLUSION
A novel self-reporting questionnaire (ISAQ) was designed to assess individuals for the functionality of their immune system. The ISAQ derived score was probed in healthy adults and in patients with various forms immunodeficiency and chronic autoimmune disease. A marked difference of the ISAQ scores between cases and controls was evident, suggesting that this tool has potential to identify individuals at risk for immune-based diseases, particularly with immunodeficiency phenotypes or systemic vasculitis. However, as items with high and low predictive power were identified, the ISAQ in its present form is suboptimal and requires refinement preferentially in the context of longitudinal studies.

CONSENT
All patients and controls were orally informed about the scientific purpose of the ISAQ study and about their free decision to take part or not in the study. They were instructed about the anonymous character of the study allowing via an individual 6-digit ID code only the study participant and the study team to access the calculated ISAQ score. All patients and controls gave oral consent to the study principles and were freely filling out the ISAQ form.

ETHICAL APPROVAL
All authors hereby declare that the Ethics Committee of Freiburg University Medical Centre (UMC) approved the ISAQ project (Project Nr. 174/08) and that it was conducted in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki.