What is new?
- •
This study is differentiating between the general selection bias and the additional response bias arising from the use of Web-based data collection that can be assumed to have an indirect effect on study results, as different individuals are attracted to take part.
- •
Concerning selection bias, we found that our sample is overall better educated, more often living in a partnership, more often female, and older than the general population.
- •
Concerning the additional response bias, we can confirm that participants using the Internet are younger, better educated, and more often male.
- •
However, the method of data collection had no direct effect on the results of various self-report instruments after adjusting for the three characteristics defining the response bias.
Web-based questionnaires and participation in epidemiologic studies through remote data entry by using the Internet has been described as an attractive alternative to more traditional methods with paper-and-pencil questionnaires or interviews [1], [2], [3]. Web-based data collection is not only faster and requires less field logistics, thereby reducing financial demands, it has also been claimed that data quality and response rates to single problematic items may increase because of the use of immediate data checking devices and interactive missing item reminders. This is especially relevant for self-administered questionnaires. Additionally, the use of a “pleading” style for e-mail invitations is reported to increase response rates for Web-based surveys [4]. There are reports of high data quality from Web-based or electronic questionnaires in various fields of epidemiologic research such as anthropometric questionnaires [2], quality-of-life measures [5], usage of oral contraceptives [6], assessment of alcohol use [7], smoking cessation [8], and diet history [9].
Still these advantages are shadowed by some remaining critical issues. It is doubtful that, at least for self-selected participation, strictly Web-based studies and registries are able to attract the same people as paper-and-pencil questionnaires distributed by ordinary postal mail. The resulting response bias, because of the Web-based data collection, must be seen as an additional bias to the more general selection bias and can be suspected to have at least an indirect effect on the study outcome. But, there are also concerns about a direct effect: Klovning et al. [10] found that their Web-based survey yielded not only an age-biased sample but that the participants reported a higher illness severity compared with others in a paper-based survey. Although the first finding implicates that Web-based data collection may not be adequate for studies aiming to include older participants, due to the presumably lower Internet-access rate in older people, the second finding is even more problematic. If Web-based surveys have a substantial direct effect on study outcome, this must lead to the conclusion that results from Web-based questionnaires are not comparable to traditional postal surveys [11].
In this study, we used the data of the German Weight Control Registry (GWCR). The objective of the GWCR is to identify and investigate the characteristics of individuals who have succeeded at long-term weight loss. During the ongoing process of establishing the GWCR, we follow a mixed-mode approach: Participation in the registry is possible via different ways for all persons fulfilling a weight loss maintenance criterion [12]. After signing and returning a mailed informed consent, the participants can choose whether they prefer to fill out the questionnaires in a mailed paper-and-pencil version or directly by using the Internet.
In addition, we investigated an independent sample of individuals from a representative population-based survey who met all criteria necessary to be eligible for the GWCR. This setup gives us the chance to investigate an important aspect of the current discussion about Web-based questionnaires: we can isolate the additional response bias resulting from the mode of participation from a more general selection bias resulting from the self-selection of participants. Analyzing differences between participants of the GWCR and individuals from the general population meeting inclusion criteria for the GWCR allows for assessing the selection bias and consequently the examination of the generalizability of our results. Study nonparticipants met all inclusion criteria but were either not reached or not convinced to take part—no matter how data collection was carried out. Study participants who preferred to use the paper-and-pencil version are the potential nonresponders of a pure Web-based study. We are able to compare participants within the same study under exactly the same circumstances, only differing with respect to the fact that one group had access to the Web and was willing to use it, whereas the other either had no Web access or, because of unspecified reasons, was not willing to use it to send their personal data. Furthermore, we are also able to investigate a potential direct effect of the type of data collection on the outcome measures not resulting from the response bias: The main outcome measures of the GWCR are weight loss and various self-report scales assessing eating behavior, diet, physical activity, psychopathology, and social network. By comparing these measures between the two groups defined by the mode of data collection, we do not have to rely on comparable results of other studies but can directly investigate if the method of participation has an additional direct effect on the results of the self-report scales independent of a possibly biased response rate.
Hence, the goal of our study was 1) to assess the common selection bias resulting from the self-selection of participants for the GWCR, 2) to analyze the potential response bias because of Web-based data collection, and 3) to investigate any independent effect of the method of data collection on the self-report scales and outcome measures after controlling for response bias.