An irretrievably biased sample

Sir, we would like to comment on the recent survey on an interesting topic by Vasant et al. published in this journal.¹ In this study, a survey was distributed through 'messaging apps and dental social media channels'. It is unclear whether the intention was to contact the whole population of dental professionals but it seems clear that many would not have been aware of this survey. Readers are not informed about which apps and social media channels were used. The authors report an impressive response of 3,309 participants from a population of dental professionals of 113,439. Nevertheless, a simple sample size calculation suggests that a sample of 383 would have given 95% confidence that estimates are within 5% of the true value. This would, of course, assume that the sample was randomly selected from a sample frame which represents all members of the population. What we had in this study was a much bigger sample size but self-selected and representing only 2.55% of the population. Therefore, we have no idea how the other 97.45% of the population would have responded. Those who did respond may have done so for their own reasons, for example, only because they had a story to tell. The sample is, therefore, irretrievably biased. Its size is irrelevant if not randomly selected from the whole population. It follows that statistical analysis of such data is meaningless, if intended to provide information about the population as a whole, and conclusions are unreliable.

Although the authors rightly report the limitations of a self-selected sample, it could be argued that publication encourages bad survey practice. We would urge authors and reviewers to take into consideration the principles of good survey practice set out in our article in this journal.² This sets out the four potential areas of survey error, coverage, sampling, measurement and response which were first described by Groves et al.³

The first Key Point in the Vasant et al. article is 'Demonstrates the use of an online survey tool'. The Survey Monkey web-based platform, used in this survey, is undoubtably useful and convenient to use. Nevertheless, the way it is used, and the planning of a survey, is crucial to produce dependable results which can inform decision making.

Ronuk Vasant, Dominic O'Hooley and André Haigh respond: We thank Dr A. Shelley and Professor Horner for their interest in our recent publication and welcome their comments. We accept their criticisms regarding sampling under non-pandemic circumstances. The BDJ Editor-in-Chief was fully aware of the background whilst the manuscript was being considered for publication. However, at the time, we felt it inappropriate to include a historical narrative of the survey within the publication. We welcome the opportunity to do so now.

We are private practitioners with no formal background in epidemiology. It is probably an unusual declaration for the authors of any published paper to make, however from the outset we never felt that we were the right people to be doing this sort of work. We immediately recognised that there were institutions and individuals with tremendous expertise and resources such as Public Health England, NHS England and Universities. Some have statutory roles in protecting the public, such as the CQC and the GDC. We therefore hoped and assumed that there would be co-ordinated efforts between at least some of these bodies to rapidly gather data during this exceptional period. History has sadly revealed there were no such efforts.

As we noted in our paper, the prevailing view at the very outset of the pandemic was that dental staff were at greater risk of exposure to SARS-CoV-2. However this belief lacked any evidence base. A basic tenet of science is to test assumptions. Since the entire UK population was immunologically naïve, it occurred to us that this early period represented a unique situation.

Face-to-face dentistry had almost totally ceased on 25 March. As we became increasingly aware that those most capable of capturing this important data were failing to do so, the timing became increasingly crucial. RV was aware that the BDA were in frequent communication with NHS England and PHE and therefore communicated the idea on 22 March 2020 to the BDA PEC Chair with the suggestion that the idea be passed on and taken forward. The Chair grasped the value in the idea. The BDA naturally had many other important work streams happening simultaneously given the enormous professional, legal and financial challenges affecting the entire profession and public at the time. We heard nothing more from the BDA or any other institutions.

We were faced with a decision - either let the moment pass knowing that this specific period of history would never be repeated, or at least attempt it ourselves. We chose to jump with both feet into 'Roosevelt's arena' and do our best.

At the time, there were very few PCR tests available in the UK and no lateral flow tests. We were mindful that delaying things even further would obviously degrade the data as we were relying on people's memories about the timing of their own symptoms. Due to the prevailing circumstances - and a total lack of external resources, or any regulatory or institutional interest - there was no prospect of using more sophisticated sampling method. As private practitioners, we certainly were not in a position to request a list of contact details of all registrants from the GDC from which we could create an idealised sample. Even had we been afforded this opportunity, given the unique circumstances of a pandemic, the risk of bias remained. As Dr Shelley and Professor Horner highlight, those motivated to respond '…may have done so [responded] for their own reasons, for example, only because they had a story to tell'. Due to the limitations we faced, the increasing time pressures, and with no pandemic surveys to reference, we opted for what we thought most effective in engaging the maximum number of people in the shortest time frame — 'snowball' sampling.⁴

From the outset the intention was to upload the data onto a pre-print server quickly. We did so on 30 April 2020. It was hoped that epidemiologists may be able to combine our survey data with other data sets to further understanding of SARS-CoV-2 transmission. It was never intended that, unaided, our survey would provide definitive answers, and we knew that it would be imperfect.² However, it was hoped that when combined with other data, it might eventually provide useful insights for future pandemic plans. It took a further two years before we came to accept there would be no other publications on the effectiveness of pre-pandemic infection control measures in UK dental professionals. Despite the study limitations, we felt a responsibility to publish our findings: to allow others to critique and arrive at their own conclusions as to the effectiveness of the pre-pandemic infection control measures. We acknowledged that over 3,000 dental professionals had made the effort and given their time so that a picture of the earliest phases of the pandemic may be better understood by future pandemic planners.

We are all in favour of high-quality research. We accept that Dr Shelley and Professor Horner may feel the study sufficiently flawed that it adds nothing to the knowledge gap, and encourages bad survey practice under non-pandemic circumstances. The question remains as to why public institutions were either disinterested or unable to react quickly to gather this sort of important information at the start of a pandemic. It ought not to be the case that the only data set of COVID symptoms of UK dental professionals from this unique, initial period (pre-vaccination, pre-pandemic infection control measures and immunologically naïve population) is from a group of amateurs. It is our view there ought to have been, as part of any effective pandemic plan, an 'oven-ready' research project, with trained personnel, established methodology (including sampling), funding and ethical approval. Once the novel pandemic appeared, it should have been a matter of pulling the trigger on the study. We accept for some, this may be the only recognisably scientifically valid conclusion from the publication.