Response Rates to Questionnaire-based Studies in the Contemporary Dental Literature: A Systematic Review

Objectives: This systematic review aimed to investigate what is a reasonable response rate for dental questionnaire-based studies in recent literature and to assess the factors that affect the response rates. Methods: We used MEDLINE/PubMed to search the dental literature of 2021 (January-October). Two reviewers independently assessed studies eligibility and extracted date using standardized electronic extraction form. Results: One hundred and seventy-two studies were eligible, of theme a total of 149 response rates were reported from 133 studies, whereas the remaining 39 studies were excluded as they did not report response rates. The median response rate across the included studies was 77% (mean=70.8%). We found significant negative correlation between the response rate and the actual number of distributed questionnaires (sample size) (r = -0.4127; P<0.001). We also found an association between the response rate and the area of distribution, e.g., national or international (P= 0.0012). However, a wide variation was observed in the quality of information reported within this review and we did not find clear evidence of association between the response rate and other variables such as questionnaire piloting, number of questions in the questionnaire and the journal impact factor. Conclusions: The findings of this systematic review confirm the association between the response rate and the sample size, where the response rate increases when the sample size less than 300 participants. In addition, a higher response rate could be achieved when the study conducted within the same institution (e.g., university). Significance: Questionnaire-based research can provide answers to several questions that could not be answered by other types of research related to the field of dentistry, dental health practitioners and students‟ attitudes and behaviours and more. Questionnaire-based publications can effectively contribute to dental research; thus, dental journals should consider development of a minimum set of guidelines in the reporting of questionnaire-based manuscripts.

Thi s v e r sio n is b ei n g m a d e a v ail a bl e in a c c o r d a n c e wit h p u blis h e r p olici e s. S e e h t t p://o r c a . cf. a c. u k/ p olici e s. h t ml fo r u s a g e p olici e s. Co py ri g h t a n d m o r al ri g h t s fo r p u blic a tio n s m a d e a v ail a bl e in ORCA a r e r e t ai n e d by t h e c o py ri g h t h ol d e r s .

Introduction
Questionnaire-based research are widely published within the dental literature. This methodology can offer an objective methods of collecting information about people's knowledge, beliefs, attitudes, and behaviour [1]. Questionnaires can be used as the sole research instrument e.g., cross-sectional surveys or within clinical trials and epidemiological studies [2]. However, concern exists in relation to what represents a "reasonable" response ratei.e., being of sufficient magnitude to reliably answer the questions posed, while excluding non-responder bias. Therefor a questionnaire should aim to obtain as representative a range of responses as possible, and thereby provide reliable and valid answers to the research questions posed. [3].
A brief review of the dental literature reveals a wide range of response ratesfrom as low as 7% on the assessment of the knowledge and attitudes of western Australian dental health practitioners towards identifying and reporting child abuse [4] to as high as 100% on the assessment of the diagnostic skills of general dentists in different types of orthodontic malocclusions [5]. Thus, questionnaire response rates become one of the "most controversial issues" which may affect the journal acceptance and publication of the questionnaire-based researches [6].
In 1997, Tan and Burke conducted a review that included sample of 77 articles from four dental journals during the period 1989-1992. Their aim was to investigate the range and factors affecting response rates for mailed distributed questionnaire-based studies [7]. The authors found that 64% is the average response rate for the investigated questionnaire-based studies. Additionally, they suggested that questionnaire subject, incentives offered and length of questionnaire may influence response rates [7]. However, since then, research methods, computer technology, internet access, statistics and editorial policies have changed the practice of data collection.
Nowadays, electronic survey become widely used among researchers. It minimises the processing costs because it automates the transformation of raw data into electronic form, and combines the advantages of interviews (e.g., complex branching) with those of paper-form surveys (e.g., standardization, anonymity) [8]. In 1984, Kraut used a computer network in international corporation to conduct research on work behaviours and attitudes [9]. Then in 1986, a study aimed to conduct an experimental study by using both electronic and paper mail sample survey. The authors found that more respondents returned the paper survey (75%) than the electronic survey (67%). However, they reported less completion mistakes and faster returned response for the electronic surveys [8]. In literature, there is a conflict on whether email or paper-form questionnaires result in higher response rates. Some papers reported higher response rate for electronic questionnaires [10], others reported higher response rates for mail questionnaires, [8,11], while others reported comparable response rates [12,13].
Furthermore, email propagation or social media can be utilized to ensure reaching the most significant number of the targeted population. Hence, a higher response rate for the questionnaire-based study and a minimum non-responder bias might be obtained. However, participant recruitment through social media platforms or email propagation might be a challenging process, and the response rate cannot always be measured. A recent descriptive WhatsApp messenger-based cross-sectional survey study found that social media platforms are reliable and could be used for disseminating information as well as a research tool among medical students and healthcare professionals. The study also found that more than two-thirds of medical students and healthcare professionals routinely use social media, although a 21% response rate was reported [14].
It has been claimed that use of mixed-mode approach will enhance the survey response rates or if mailed surveys combined with e-mail follow-up [15] . In a study comparing different methods of administration, response rates close to 60% were achieved by mixed-mode contacts [12]. This approach, combining both mailed and e-mailed survey instruments with an Internet-based response mechanism, also is an approach to help reduce the problem of coverage error in administration of surveys [12]. In comparison, Dillman et al. suggested that mixed-mode distribution of questionnaires may increase response rates with respondents choosing their preferred method of response [16]. Similarly, it was reported that 41% of electronic questionnaire respondents would not complete a telephone interview on the survey, confirming the potential of mixed-mode distribution to reduce non-response bias [17].
Moreover, a response rate of 72% was reported when a mixed-mode method was used, noticing that this technique "improved representativeness of the sample without biasing other results" [18].
Asch et al. reported that questionnaires distributed to medical professionals yield low response rates (mean response rate among medical doctors was 54% and it is ≈60% among mail surveys published in medical journals) [19]. This is confirmed in a systematic review by Cook et al. in 2009 (median response rate of 60%) on healthcare professionals" response rates which was significantly lower than the estimate for the prior 10 year period. Authors also highlights the importance of non-response analysis and indicated that sending reminders and conducting studies on less than 1000 population will increase the response rate [20].
The research question for this study is to determine "what is considered to be a "reasonable" response rate to merit publication of a questionnaire-based study in the contemporary dental literature". Hence, we conducted a systematic review to investigate what is a reasonable response rate for dental questionnaire-based studies in recent literature and to assess the factors that affect the response rates.

Materials and methods
The Preferred Reporting Items for Systematic Review and Meta-Analysis (PRISMA) guidelines for all procedures and reporting [21].

Information sources and search strategy
i) Electronic database: In February 2022, an electronic search of the online MEDLINE/PubMed 2018present database using Ovid-dc2 interface was performed using the MeSH terms related to questionnaire and dental literature. ii) Manual searching: To ensure a manageable number of articles, the search was restricted to articles published in the January-October 2021 period inclusive. At this stage, of the 782 citations, 556 articles were excluded after screening the title/abstract because they were not questionnaire-based studies or the date of publication was not within our study frame-time. When publication dates were unclear (e.g., Winter, Spring, etc.), journal websites were reviewed. Then, the remaining 226 articles were screened by 2 independent reviewers (K.A.) and (S.O"D.-K.) to determine whether eligibility criteria were met. Following this, 54 articles were excluded and a total of 172 studies were eligible for inclusion in this review, of these 39 studies did not report information regarding to the response rate, therefore these studies were excluded ( Figure 1).

Selection process and eligibility criteria
Inclusion and exclusion criteria were set prior to the selection of articles for this systematic review. Inclusion criteria included: papers published between January and October 2021 (even if available online earlier), methodology sections reporting questionnaires as tools of investigation and subsequently reported on in the results section. In addition, the questionnaire in the study must be directed to the dental schools" faculties/academics, dental health practitioners (dentists, hygienist, therapists, etc) or dental students. Studies were excluded if: the reported questionnaire was used to collect participant baseline characteristics as part of a larger study (e.g., randomised controlled trial) or if the questionnaire was not subsequently reported on in the article. Did not report response rates (n=39)

Data collection process and quality assessment
An electronic standardized data extraction form was used to extract the data from the eligible studies. Two investigators (K.A.) and (S.O"D.-K.) extracted data for all included studies. We extracted data on the article title, reference details (ISSN/Doi), study ID, country(ies), discipline, reported response rate, sample size, area of distribution (e.g., national or international), actual number of questionnaires distributed, participant population and sampling frame, method of distribution (electronic or paper questionnaire), provision of stamped-addressed envelope for return of postal questionnaires, ethical approval, questionnaire piloting, follow-up, incentives, questionnaire reproduced in published article, number of questions in questionnaire, reference to response rate in discussion section, validity testing of questionnaire and the impact factor. In addition, a third investigator (C.D.L.) resolved any inconsistencies between the two investigators about included articles and data extraction.

Statistical analysis
While this project was mainly exploratory in nature, some potential associations of interest were investigated for statistical significance. Data distribution was considered to facilitate appropriate use of parametric/ non-parametric tests. We assessed the distribution of response rate and as it was not normally distributed, we used non-parametric tests for almost all reported results in this review. A Spearman"s correlation was performed to assess the relationship between the response rates and the following numerical variables (actual number of distributed questionnaires, number of questions in questionnaire and journals impact factor). Furthermore, Kruskal-Wallis test was used to assess the pattern of response rate across the following categorical variables (area of distribution, geographical distribution, method of distribution, piloting, and follow up). A statistically significant P value was based on a threshold of <0.05 and all analyses were using Stata/MP software (version 16).

Results
A total of 172 articles were eligible to be included in this review. Of these, 14 articles had more than one population (e.g., dentists and dental assistants) with different response rate distribution. Thirty-nine articles did not report information regarding to the response rate; therefore, these studies were excluded. Eventually, a total of 133 articles with 149 reported response rates were included. The characteristics of included studies are presented in Table 1 according to the response rate.  Table 2 shows the distribution of response rates in the sample. Ninety-two of the reported response rates (61.7%) were greater than 70%. Thirty-seven reported response rates (24.8%)

Actual number of questionnaires distributed
Of the 149 reported response rates, 140 reported information about the number of questionnaires distributed to participants (93.9%). Nineteen studies reported questionnaires distributed to ≤50 participants (13.5), 24 (17.1%) reported questionnaires distributed to participants between 51-100, 40 (28.5%) reported questionnaires distributed to 101-300 participants, 37 (26.4%) reported questionnaires distributed to 301-1000 participants and 20 (14.2%) reported questionnaires distributed to >1000 participants (Table 1). We also found a statistically significant moderate negative correlation between the number of questionnaires distributed and response rate (Spearman's Correlation coefficient = -0.4127; P<0.001) suggesting that when the questionnaire was distributed to a smaller sample population, a higher response rate was achieved.

Area of distribution
A total of 55 (36.9%) studies were conducted within the same institution as the study investigators, 81 studies (54.3%) were conducted nationally, and 13 (8.7%) were international studies. Table 3 shows the pattern of response rate according to the area of distribution. Significant differences in response rates based on the area of distribution were found, with higher response rates among studies that conducted within the institute (Kruskal-Wallis test P value =0.0012). However, it should be noted that few international studies were included in our review (n=13).

Geographical distribution
The 149 reported response rate were categorised into six geographical area namely, Africa, Asia, Europe, North America, South America and other. Table 4 illustrates the pattern of response rate in regard to the geographical distribution. There were significant differences in response rate across the geographical distribution (Kruskal-Wallis test P value = 0.067).
Approximately, two-third of studies in Asia had response rates ≥ 70%, whereas about half of European and north American studies had response rates ≥ 70%.

Method of distribution
The majority of the included studies reported electronic/ internet distribution of the questionnaires (82, 55%). Forty-five studies (30%) reported paper distribution whereas two studies (1.5%) reported mixed-mode distribution. No information was reported in the remaining 20 studies (13%). Table 4 shows the pattern of response rate according to the method of distribution. There is weak evidence of differences between the method of distribution and the response rate (Kruskal-Wallis test P value = 0.0489).

Provision of a stamped-addressed envelope for return of postal questionnaires
Only three studies out of the 45 (6.6%) that used paper distribution of the questionnaire reported provision of a stamped-addressed envelope for return.

Ethical approval
Of the 149 reported response rate, 127 (85.2%) reported ethical approval being sought, 20 studies (13.4%) did not report seeking ethical approval and two studies (1.3%) reported that ethical approval was not sought or being advised that ethical approval was not required.

Piloting
Of the 149 reported response rates, 64 (43%) had been piloted, while no information on piloting was reported in the remaining studies (85 , 57%) ( Table 5). There was no statistical significance differences in the response rates across these groups (Kruskal-Wallis test P value = 0.0877).

Follow-up
Of the 149 reported response rate, 38 studies (25.5%) reported follow-up was performed.

Incentives
One-hundred and forty-two studies (95.3%) did not report any incentives. Five studies (3.3%) reported that no incentives had been offered. Two studies (1.3%) reported incentives had been offered.

Questionnaire reproduced
Sixty-four out of the 133 included articles (48%) reproduced the original questionnaire within the published article. The remaining 69 articles (52%) did not reproduce their questionnaire. Table 6 shows the number of questions included in the questionnaires and Figure 2 illustrates the association between number of questions and the response rate. It is clear from the scatter plot that there is no correlation between number of questions and response rate (Spearman's correlation coefficient: 0.1148; P = 0.2604).

Reference to response rate in discussion section
Of the 149 reported response rate, 53 studies (35.5%) commented on the response rate achieved in their discussion or conclusion section. Ninety-six studies (64.4%) did not comment on the response rate.

Validity testing of questionnaire
Fifty-nine studies (39%) reported that the questionnaire had been validated or was based on a previously-validated questionnaire. Three studies (2%) did not validate the used questionnaire and no information on validation was included in the remaining 87 studies (58%) out of the 149.

Impact factor
The range of journal impact factors was from 0.406 to 6.99. The scatter plot ( Figure 3) shows no association between impact factor and response rate (Spearman's correlation coefficient: -0.0939; P = 0.2548).

Discussion
This systematic review included 133 questionnaire-based articles with a total of 149 reported response rates. The median of response rate across the dental literature was 77% (mean=70.8%). In 1997, Tan and Burke reviewed 77 articles that were published between 1988-1992, the response rates to "questionnaires mailed to dentists" had an average of 64% [7]. Thus, based on the information included in this systematic review, the response rate of the questionnaire-based dental studies is higher than what it was reported 25 years ago.
Furthermore, this review revealed that studies conducted within the institute (e.g., university) had higher response rates than the studies that conducted nationally or internationally.
Moreover, we found that about two-third of studies in Asia had response rates ≥ 70%, whereas about half of the European and north American studies had response rates ≥ 70%.
However, we found weak evidence that the method of distribution (e.g., electronic) could affect the response rate, and comparable response rates could be achieved with any method of distribution as previously reported [12,13]. Thus, this finding does not support using of electronic questionnaire [10] or the mixed-mode method [15,16,18]to increase the response rate.
Beside the many advantages of electronic distribution of the questionnaire-base studies (e.g., automates the transformation of raw data into electronic form), concern has been expressed in relation to potential reduced response rates associated with this method. Yun et al. stated that "e-mail research raises many ethical concerns because unsolicited e-mail invades a person's private space" [18]. Likewise, Dillman et al. also noted that junk email risks reducing response rates [16]. Comparable to older studies [22,23], this systematic review have a median response rate for electronic distribution (72.8%) which was lower than that of paper distribution (84.3%).
More than two-third of the studies that reported ≥70% response rate had distributed their questionnaire to participants of 300 or less. This indicates that a higher response rate could be achieved when the questionnaire is distributed to a smaller population. Similarly, Cook et al.
suggested that conducting studies on less than 1000 population would increase the response rate [20]. Cook et al. also suggested that sending reminder will increase the response rate [20] which is in keeping with the original recommendations of Dillman et al. [16]. However, we found that studies that reported sending follow-up reminder had lower response rates than the studies that did not report any information on sending reminder. Therefore, our results could not support the importance of sending reminder to increase the response rate as 63.7% of articles did not report any information on follow-up.
While piloting is deemed essential and a recognised means of increasing response rate [16], the difference in response rates did not reach a significant level between the included studies reporting piloting (43%) and those which did not report any information (57%). A wide variation was also noted in the quality of information reported within the systematic review sample of publications. Thirteen percent of articles did not report seeking ethical approval for their questionnaire, 58% did not report whether or not their questionnaire had been validated, and 56% did not report whether their questionnaire reproducible or not (i.e., can be used again). Additionally, we found no relationship between the number of questions in questionnaire and the response rate. In addition, 96 articles (64.4%) did not comment or mention the achieved response rate in the discussion section of their study and 142 studies (95.3%) did not report offering any incentives.
Questionnaire-based studies and the response rate achieved are controversial. There is no evidence base for specific thresholds for response rates among these studies [6]. However, conducting good research practice by following the core principle based on Dillman's work [16] which have achieved high levels of acceptance within the scientific community and considered as fundamental to high quality surveys [6]. Briefly, Dillman et al. recommended that: (1) more work and scientific rigour goes into the planning and execution of a questionnaire survey, the more likely the results are to be valid, (2) the quality and clarity of the survey's covering letter will improve response rates, (3) the questions set should be simple, short, specific without being too specific, unambiguous and should avoid bias, (4) the questionnaire should be piloted amongst colleagues, potential users of the information and also amongst the population to be surveyed, (5) the questionnaire recipients should be randomly selected, representative of the population as a whole (to avoid selection bias and ensure that results are 'generalisable') and of sufficient size, and (6) if one mailing achieves a poor response rate, then repeat the mailing twice and try to identify how non-responders differ (demographically) from responders [16].
Systematic reviews offer advantages such as efficiency, integration of information from a diverse range of related articles, while reducing bias and increasing reliability and accuracy of recommendations because of its formalised and thorough method of investigation [24]. We conducted a comprehensive search strategy and the investigation was exploratory in nature; some potential associations of interest were investigated for statistical significance. Due to the non-normal distribution of data, non-parametric tests were selected. These tests are valuable; however, they are less sensitive than parametric tests [25]. In addition, our review was limited to the English language studies that published between January and October 2021 which could considered as limitations of this systematic review.

Conclusion
The findings of this systematic review confirm that high response rate is associated with smaller sample size and conducting the study within the same institute. However, we noted considerable variation exists amongst response rates and the reporting of other information.
Questionnaire-based publications can effectively contribute to dental research; thus, dental journals should consider development of a minimum set of guidelines in the reporting of questionnaire-based manuscripts.