Methods: Materials

An instrument with 29 items resulted from a total of 13 focus groups with eligible patients (with neurogenic bladder due to SCI or SB, and who use intermittent catheters) or their caregivers, from around the country, plus iterative focused discussions with a core of four clinicians with expertise in this domain (as described in Tractenberg et al. 2017).[16] As described below, a national sample of people with neurogenic bladder who manage their bladders with intermittent catheterization were recruited from patient populations of SCI (N = 336) and SB (N = 178), plus caregivers of people with SB (N = 66). Participants were recruited to complete the instrument online, using SurveyMonkey. Each item was presented as a query about whether the respondent had experienced it during the past year (yes/no), with three additional required responses: average frequency (0–365); average severity (usually not at all severe; usually somewhat severe; usually severe; always very severe); and average impact on, or importance in, daily life (rarely affects my actions or decisions to go about my daily life; sometimes affects my actions or decisions to go about my daily life; usually affects my actions or decisions to go about my daily life; always affects my actions or decisions to go about my daily life). The analyses described below were focused on the endorsement (yes/no) of items as sources of validity evidence for the instrument and its makeup. Once preliminary validity evidence is established, we plan to revisit the frequency, severity, and impact ratings to develop and refine a scoring algorithm in a future article.

Methods: Validation

Our analyses were planned to describe the validity of the USQNB-IC using classical psychometric methods.[21] The COSMIN report identifies four key psychometric or measurement domains on which a health related PRO should be characterized: Reliability (minimization of measurement error; internal consistency or interrelatedness of the items; and maximization of variability that is due to “true” difference between levels of the symptoms across patients), validity (content, reflection of the construct to be measured; face, recognizability of the contents as representing the construct to be measured; structural, the extent to which the instrument captures recognizable dimensions of the construct to be measured; and criterion, association with a gold standard), cross-cultural validity, and the interpretability of scoring. Cross-cultural validity would not be relevant if any of the other measurement properties are not met, so we did not address this type of validity in this study. Similarly, responsiveness, defined by the COSMIN criteria as the sensitivity of an instrument to change in the construct of interest, was also not evaluated in this study because like cross-cultural validity, that would be premature.

The development of this instrument was characterized by a focus on face and content validity from both the clinician and the patient perspectives (Fig 1, above), thus this report augments that content and face validity evidence. Because there is, by their origins, no diagnostic “gold standard” for what these signs and symptoms represent, our evidence on convergent and divergent validity was classified as supporting both “reliability” and “criterion validity”, as described in the next section.

Importantly, the COSMIN requirements for “structural validity”, which they defined as dimensionality, are not consistent with either the approach to patient-centered patient reported outcomes that we followed, nor with the early stage of our instrument development. Specifically, for formal structural dimensionality to be a feature of a PRO at this early stage, it would need to have been developed following procedures outlined by Bollen (1989), with a measurement model in mind.[15,21] Instead, our instrument was developed (as noted) with no measurement model in mind, but instead with the specific intention of capturing the patient experience with high fidelity. Thus, the dimensionality on which we can report relates to the aspects of urinary signs and symptoms, and not to specific causal or response-generating features of the items on the instrument. We estimated internal consistency of the items with Cronbach’s alpha, which is a function of the correlation of each item with the total score that takes the total score variability into account, and provides an estimate of the shared covariance of items on the instrument–but does not relate to the instrument’s dimensionality or response-generating mechanism.[22,23]

In this report we describe new evidence of face and content validity as well as estimating internal consistency (COSMIN defined “reliability”) with Cronbach’s alpha, dimensionality (COSMIN defined “construct validity”) with exploratory common factor analysis, modeling the covariance of items; and inferred causal model with Bayesian networks, modeling the shared Shannon Information. Given the early stage of this instrument, we demonstrate both construct and criterion validity (which COSMIN defined as including divergent and convergent validity) with distributions of scores (total and item endorsements) for the target population of individuals with neurogenic bladder who manage their bladders with intermittent catheterization, and also with data from individuals with neurogenic bladder who do not have a history of UTIs, individuals with chronic mobility impairments (neither SCI nor SB) and without neurogenic bladder, and those with no mobility impairment, no neurogenic bladder, and no history of UTIs.

Table 1 below recapitulates the COSMIN measurement properties and describes the methods by which these were analyzed in this article.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. COSMIN²⁰ criteria and how they were assessed in this study.

https://doi.org/10.1371/journal.pone.0197568.t001

Methods: Target patient population participants and recruitment

A nationally representative sample of individuals with neurogenic bladder due to either SCI or SB was recruited to complete the survey. We also obtained responses from individuals caring for people with SCI or SB. These samples represent the target populations as well as convergent validity data, and are described below.

Inclusion Criteria: Inclusion criteria for participants in the national sample with SCI: 1) age≥18 years; 2) SCI at least 1-year duration; 3) neurogenic bladder, as reported by respondents; 4) utilizing intermittent catheterization for bladder management; 5) and a history of 2 or more UTIs in the past year. Inclusion criteria for participants in the national sample with SB were: 1) age≥ 18 years; 2) neurogenic bladder, as reported by respondents; 3) utilization of intermittent catheterization for bladder management; 4) and a history of 2 or more UTIs in the past year. The inclusion criterion for participants in the national sample who were caring for a dependent with SB was only that they were a caregiver for an individual with SB or SCI.

Our consumer partner, United Spinal Association, assisted in recruiting nationally-representative samples of adults with SCI (N = 336), adults with SB (SB; N = 179) caregivers of children with SB (CG; N = 66).

Methods: Divergent validity participant recruitment

Individuals with neurogenic bladder who did not self-report a history of UTIs (however the respondent defined this), individuals with chronic mobility impairments (neither SCI nor SB) and without neurogenic bladder, and individuals with no mobility impairment, no neurogenic bladder, and no history of UTIs were recruited largely from an inner city rehabilitation hospital to complete the same instrument.

Methods: Data analysis

Statistical analyses were carried out using SPSS v. 24[24] for descriptive statistics and to estimate the variance explained by the exploratory factor analysis; exploratory factor analysis with tetrachoric correlations was carried out in MPlus v. 8 (Statsoft, Los Angeles, CA). BayesiaLab v. 6 (Bayesia S.A.S., Laval, France) was used for the Bayesian network modeling: unsupervised structural learning (searching for the overall data structure) using a Maximum Weight Spanning Tree (MWST) learning algorithm to constrain the structure so that only one “parent” per variable is identified. Minimum Description Length (MDL–information, not probability) was used for scoring because linearity in relationships cannot be safely assumed for these items. The resulting tree was visualized in automatic layout for interpretability (features discussed in Ch. 7 of Conrady & Jouffe, 2015).[25] The data analyzed in this manuscript are available as Supporting information files.

Results: Descriptive statistics, SCI

Complete and valid (i.e., from eligible respondents) responses to the USQNB-IC were obtained from 336 individuals with spinal cord injury (SCI). On average, respondents with neurogenic bladder were 46.7 years old (sd: 13.64y; range 18–75) and had been living with their injury (and neurogenic bladder) for an average of 14.7 years (sd: 12.5; range 1–62 years). Of this sample, 45.5% were women and 30.1% reported that they reuse catheters; we did not obtain the level of injury but participants did report whether their injury was complete (39.9%), incomplete (53.9%), or unknown (6.8%).

Results: Descriptive statistics, SB

Complete and valid responses to the USQNB-IC were obtained from 179 individuals with spina bifida (SB). On average, respondents were 35.3 years old (sd: 11.7 y; range 18–65). Of this sample, 72% were women and 42.8% reported that they reuse catheters.

Results: Descriptive statistics, caregivers of patients

The most difficult group to recruit were caregivers of individuals with SB or SCI. Complete and valid responses to the USQNB-IC were obtained from 66 individuals who are currently caring for an individual with SB or SCI. On average, the individual being cared for was 12.4 years old (sd: 9.4 y; range 1–44). Of those being cared for in this sample, 55% were female and 6% reported that they reuse catheters.

Results: Descriptive statistics, three control groups

The total number of control respondents was 160, with 53.75% female overall (36% females with NB, no UTI; 43% females with mobility impairment, without NB, no UTI; and 75% female no mobility impairment, without NB, no UTI). The average age was 55.7 years (SD = 16.8) years (49.5 years (SD = 15.3) for 47 participants with NB, no UTI; 60.3 years (SD = 15.7) for 49 participants with mobility impairment, without NB, no UTI; and 56.6 years (SD = 17.4) for 64 participants with no mobility impairment, without NB, no UTI). Of these 160 individuals, 16 use catheters.

Results: Validity evidence, reliability

The approach to validation was derived from Mokkink et al. (2010)[20], as discussed above; content and face validity are conferred by the process by which the instrument was created (Fig 1).[16] Table 2 summarizes the COSMIN criteria and our results following the methods outlined in Table 1.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. COSMIN²⁰ reliability evidence: Patient groups.

https://doi.org/10.1371/journal.pone.0197568.t002

Results: Exploratory factor model and inferred causal network

Principal axis factoring (PAF) with oblimin rotation extracted eight factors from the tetrachoric correlation matrix of endorsement rates of the overall sample (N = 581; data not shown). This model fit the data well (root mean square of the approximation, RMSEA = 0.024, 95% CI: 0.016–0.031; confirmatory factor index, CFI = 0.985; Tucker’s factor index, TFI = 0.971; standardized root mean square of the residuals 0.037). The 8-factor model fit significantly better than the 7 (chi square 52.8 with 22 degrees of freedom, p = 0.002) and all larger models up to 8 fit significantly better than models with fewer factors (data not shown but all p<0.001); however, we estimated that just 35.6% of the total variance in this system was explained by the 8- factor model using the PAF method with oblimin rotation based on the Pearson Correlation matrix in SPSS. Oblimin was used because of the high likelihood of correlated factors (given the origins of the items within focus group sessions guided by a single common script); many of the eight extracted factors were correlated with p<0.05; these data are not shown because although the 8 factor model fit well according to the fit indices, a) an 8 factor model did not have much interpretability; b) the estimated variance explained was inadequate (35.6%)–yet this was greater than for any smaller model; and c) we did not hypothesize a meaningful measurement model given the time frame of our questions and the origins of these items (i.e., without a measurement model in mind).

The sub-samples were too small (apart from SCI) to re-run this extraction procedure with interpretable results; however it was not hypothesized that collapsing over diagnostic group was the cause of the limited explanatory power of the modeling. Given the origins of these items (i.e., not following Bollen 1989/no measurement model in mind) and the fact that all queries were formulated for “the past year”, the lack of a coherent causal measurement model that explaining a majority of the variability among the three neurogenic bladder response groups was not surprising.

We next used inferred Bayesian networks (Bayesialab v. 6, Bayesia S.A.S., Laval, France) to discover evidence of underlying structure for these items. This method is not parametric, does not use or require inferences, and uses information instead of probability for estimating associations. The results are, therefore, associative and not specifically causal. This method allowed us to identify possible clusterings of symptoms, without sample sizes or parameter estimation precision being the issue that they can be in an inference-, or estimation-, oriented method like PAF. Like the PAF analysis above, the Bayesian network was inferred based on the yes/no endorsement of all items for the full sample using unsupervised learning and the maximum spanning tree option. This method was used to uncover basic structure, but limit the associations that appeared to a single-neighbor (i.e., a Markov Blanket approach). This method is similar to the employment of rotation in factor analysis solutions for “simple structure”.

We discovered a network with a complex structure from the full sample of 581. The network is represented in Fig 2 below.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Bayesian network of USQNB-IC items, full sample.

https://doi.org/10.1371/journal.pone.0197568.g002

The network shown in Fig 2 highlights the complexity of the lived experience of urinary signs and symptoms in neurogenic bladder. Importantly, as anticipated (see comments above), the structure does not show a single causal or originating factor for all or even some of the symptoms; this has both treatment (clinician) and management (patient) implications, as well as highlighting critical characteristics of the patient’s experience for clinical research and clinical trials of future interventions for UTI. For example, “incontinence” and “frequency of spasms” are directly or indirectly related to the majority of symptoms; therefore, if interventions address these two symptoms, they may have the most distributed effects. By contrast, the associations of “generally not feeling well” are more limited (i.e., only associated with fatigue, fever, headaches, etc.), and therefore, an intervention that addressed “generally not feeling well” might have much more limited effects than one addressing incontinence. At this preliminary stage of validation we asked whether each item had been experienced within one year. Our follow-up research questions will include whether each item was experienced within any one-week period, because that is the putative timeframe for use of the instrument in research or treatment. Because of the confounding effects of the timeframe of the questions, and because the measurement model structure (e.g., path weights, factor makeup) was not our purpose with this model, we did not estimate fit, factors, or strengths of associations between any of the items.

That this network was readily interpretable is additional evidence that the USQ-NB-IC has sufficient structural validity for continuing the evaluation of its measurement properties. The interpretability of this network also contributes to the reliability evidence for the instrument.

The instrument demonstrates reliability and construct validity as discussed in the preceding section. Content and face validity are conferred by the process by which the instrument was created (Fig 1). Additional content and face validity evidence arises from an examination of endorsement rates across the three groups of nationally-representative patients with neurogenic bladder in our sample. Table 3 provides the presenting the endorsement rates (number of respondents identifying having experienced the symptom within the past year/total number of respondents in that group) by persons with neurogenic bladder for each symptom on the USQNB-IC in the preceding 12 months.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Item endorsement (in percent) by group.

https://doi.org/10.1371/journal.pone.0197568.t003

Endorsement rates are similar for individuals with SCI & SB, whereas endorsement by item was slightly different for the caregivers. The endorsement rates for each item on the instrument were at least 10% in every group, supporting a claim of face validity across these participants since all items are perceptible to these three patient groups.

Table 3 shows that the endorsement rates for the items are generally similar across groups, except for rigor (far more frequently endorsed by SCI). We did not test for significant differences across groups due to a) a need for correction for 29 comparisons (reducing the likelihood that any would remain significant) and b) our assumption that the focus groups and focus group facilitators, who were consumers themselves (as described previously[16]), were able to capture the lived experience of neurogenic bladder across etiology.

Four of the five most highly endorsed items were the same in each group: Darker urine; cloudy urine; bad smelling urine; and generally not feeling well. Fatigue was reported in the SB group with greater frequency than the other two groups. The fifth most frequently endorsed item was increased incontinence for SCI; increased leakage for SB and fever for CG.

The similarities of endorsement rates across groups offers content and face validity evidence, together with convergent validity and some reliability evidence. It also suggests that neurogenic bladder—which was the defining feature of the focus groups from which the symptoms were derived–does have a set of “signs and symptoms” that neurogenic bladder patients may experience. The focus groups convened neurogenic bladder patients from diverse etiologies and their input created a set of signs and symptoms that the national sample of neurogenic bladder respondents recognized. The similarities suggest the instrument is relevant/resonant for neurogenic bladder respondents who manage their bladders with intermittent catheters, also contributing evidence of face and content validity.

Table 4 presents the endorsement rates across our control groups. These groups were recruited (as noted above) to evaluate the prevalence of these symptoms for other groups (not SCI or SB) with mobility impairments without neuropathic bladder, those with neuropathic bladder but who do not have frequent UTIs, and finally from those without mobility impairment, neuropathic bladder, or frequent UTIs. Importantly, in Table 4, endorsement of items was not simply interpretable as “yes/no” because for some of these participants the items had no possibility of endorsement. That is, for divergent validity participants, catheterization was very rare (just 16 of 160) whereas all participants in the USQNB-IC groups did use intermittent catheterization. Thus, the endorsement rates in Table 4 are given together with the number of respondents in any given group for whom the item was possible to endorse.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Item endorsement in percentage, by group, with the number of respondents endorsing each item in parentheses.

https://doi.org/10.1371/journal.pone.0197568.t004

In Table 4, the endorsement rates for “increased catheterization” and “burning on catheterization” are low, but the number of individuals responding to the item is also quite low. This is highlighted because these endorsement rates (%s) cannot be directly compared to the rates from Table 3, as the values in Table 4 for these items, at least, are based on dramatically smaller samples.

The last row in the table shows that the average number of endorsed items was 5.84 of 29 for the controls overall; this is less than half of the average for the patient groups (Table 3). The highest average number of endorsed items was in the group without mobility impairments and without neurogenic bladder (7.0, SD = 4.45). The lowest average number of endorsed items (4.55, SD = 4.27) was for those with mobility impairments who did not have neurogenic bladder.

There was agreement among the four most-frequently endorsed items among the control groups shown in Table 4. Increased urgency was among the top five most endorsed items in all three groups, while fatigue was in the top five for those with mobility impairment/no neurogenic bladder and those with neurogenic bladder but no UTIs; generally not feeling well and muscle aches were in the top five for both those with neurogenic bladder but no UTIs and those with neither UTIs, neurogenic bladder, nor mobility impairment. None of the endorsement rates for these items was as high as it was for any of the neurogenic bladder patient groups; the most striking difference between these control group endorsement rates and those from our national sample is for those with neurogenic bladder who have not experienced any (many) UTIs. Their rate of endorsement is the second lowest of the three control groups (with those who have neither mobility impairment nor neurogenic bladder having the highest average endorsement rate).

A final aspect of validity was explored by studying whether respondents tended to attribute the symptoms to a UTI. This feature of the instrument was challenging to conceptualize, not only because the frame of reference for endorsement was the previous year, but also because attribution, by the patient, of a symptom “to a UTI” could arise from a wide variety of ultimately unverifiable rationales. For example, a respondent could attribute a symptom to a UTI for any (one or more) of the following reasons:

1. Their clinician called it a UTI (whether or not this was confirmed);
2. Their clinician treated it with/prescribed antibiotics (whether or not this was consistent with clinical practice guidelines, which are noted to be based more on expert consensus than evidence)[26];
3. They self-diagnosed a UTI;
4. Previous experience with that or a similar symptom was clinically- or self- diagnosed as a UTI;
5. The symptom represented a deviation from normal and was attributed to UTI.

Because it is (and was) impossible to verify whether any experience was actually a UTI, we instead present results based on respondents’ assertions that the symptom was never associated with a UTI. This would therefore exclude all of the above possible associations with, or attributions to, a UTI. Therefore, Table 5 presents the proportions of respondents, collapsed over our national patients-with-neurogenic bladder samples (n = 581) and control (n = 160) groups. In each case, the proportions of those who endorsed each item and also never attributed it to a UTI are given.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Proportions of respondents endorsing a USQ-NB item and not attributing it to a UTI.

https://doi.org/10.1371/journal.pone.0197568.t005

Table 5 shows that the controls attributed just one of the USQNB-IC items to UTI in any appreciable numbers: “burning on passing urine”. “Blood in urine”, “cloudy urine”, and “abdominal pain” were the items with the lowest proportion of controls attributing the symptom to a UTI. By contrast, the majority of the patient groups attributed 16 of these 29 symptoms to UTIs–these are indicated with an asterisk (*). Because of how participants in both groups were recruited, it is supportive of face and content validity claims that most patients with experience with UTIs do attribute most items to having a UTI while participants who were recruited specifically because they have limited experience with UTIs do not attribute these symptoms to having a UTI. Also, members of our focus groups for developing the instrument originally were from the patient group, not from the divergent validity “control” groups. Thus the higher likelihood of attributing these symptoms to having a UTI in our target patient group also supports the reliability and validity of these items.

Taken together, these results suggest that our instrument possesses important validity evidence according to the COSMIN criteria. Specifically, the results present several lines and sources of evidence of face, content, criterion, convergent, and divergent validity for these items; reliability evidence also arises from the results considered all together.