Introduction

The British Ophthalmological Surveillance Unit (BOSU), established in 1997, aims to provide a methodological framework for the systematic investigation of the incidence and clinical features of rare eye conditions of public health or scientific importance1 that will lead to improvement in prevention, treatment and service planning.

The BOSU runs an active surveillance scheme in the UK and Ireland through which research groups in ophthalmology and related fields can ascertain cases on a nationwide basis. The surveillance scheme involves all consultant or associate specialist ophthalmologists with clinical autonomy in the UK who form the reporting base. At the end of every month, they receive a report card (Figure 1). Included with the card are case definitions of all conditions currently the subject of surveillance. Respondents indicate either how many cases of each disorder they have seen or confirm that they have no new cases to report. Individual investigators are notified of all positive case reports by the BOSU. The investigator then contacts the reporting ophthalmologist directly using a questionnaire to collect information about the reported case. This includes the patient's eligibility for inclusion in the study and confirmation of the diagnosis. Also by collecting unique identifiers, duplicate reports of the same case can be excluded (Figure 2).

Figure 1
figure 1

A sample report card.

Figure 2
figure 2

The BOSU process.

The benefits of operating such a surveillance system extend to both researchers and the participating ophthalmologists. Researchers are able to ascertain cases using a well-established methodology shown to be more effective than comparable methods.2,3 The burden on the ophthalmological community is lessened by removing requests from numerous sources for reporting cases of uncommon disorders.4 BOSU's supplementary activities include providing advice and assistance to research workers on the design of their studies and increasing awareness within the medical profession of the conditions studied. In other specialties, both in the UK and abroad, nationwide surveillance schemes have been successfully used in the study of rare conditions resulting in important changes in clinical practice.1 The structure of the National Health Service allows total population surveillance for ophthalmological conditions since all people with rare eye disease should come under the care of an ophthalmologist. However, nationwide rare disease surveillance is new to ophthalmology and has not previously been evaluated.

Against this background, we report the results of participation rates, opinions of ophthalmologists, study applications and research outputs during the first 3 years of the BOSU to investigate whether this method of case ascertainment is appropriate and productive in an ophthalmological setting.

Methods

This paper reports participation rates, case notification patterns, a postal survey of ophthalmologists' opinions of the scheme, and an audit of study applications and research outputs.

Participation rates

Patterns and trends in card return rates were examined. The 4-month ‘moving mean’, ie the mean card return rate for each consecutive 4 months, was repeatedly calculated and used to describe the trend in overall card return rates. Annual card return rates for each health region were used to illustrate the size and pattern of geographical variations in reporting behaviour.

Respondents were divided into four equal-sized groups (quartiles) according to their response rates to assess whether some ophthalmologists only returned their cards when they had a case to report. For each group, the proportion of all cards returned that included an identified true case was calculated. Differences between groups in the proportion of cases were compared using logistic regression with adjustment for clustering, where each participant's returned cards were regarded as a cluster.

Opinions of ophthalmologists

In July 2000, we carried out a postal questionnaire survey of all ophthalmologists on the reporting database at that time (n=856). The questionnaires requested no identifying information to encourage unprejudiced responses. This anonymity precluded reminders to non-responders. Awareness of the survey was raised through an article in the BOSU newsletter, as well as direct written communication with all participants.

The questionnaire sought respondents' opinions about

  • the perceived usefulness of the scheme and the study protocol cards,

  • satisfaction with the levels of feedback from the unit and from research groups,

  • barriers to participation.

The questionnaire listed four previously identified potential barriers to participation:

  • having to wait until the end of each month before reporting identified cases,

  • receiving one card, but working at more than one hospital site,

  • having to remember the reported patient's identity,

  • not having return postage paid.

Respondents were asked to grade each of these potential barriers as either making it harder to participate, easier to participate or making no difference. Respondents were also asked to indicate the proportion of cases they believed they had reported to the surveillance unit for each of the completed surveillance studies. Levels of ascertainment were categorised as all cases, more than half, less than half, or no cases seen. A mean estimated proportion was calculated using category midpoints (100%, 75%, 25%).

Study applications and research outputs

Applications to the unit to undertake new studies were categorised by subspecialty. Reasons as to why submissions did not complete the application process or were not accepted were reviewed. Research outputs of accepted applications were measured in terms of journal publications and conference presentations.

Results

Participation rates

Between July 1997 and June 2000, the 4-month mean card return rate improved from 58% to 71%, with the most notable increases occurring during the first 18 months (Figure 3). Despite some shifts within the positions of the regions, overall there were consistent differences with a spread of 19% between the maximum and minimum card return rates (Table 1). In addition, there was an association between higher individual card return rates and an increased likelihood that a returned card reported a case (P=0.003) (Table 2).

Figure 3
figure 3

Trend in mean card return rates, July 97–June 00.

Table 1 Mean annual card return rates by region
Table 2 Proportion of cards returned reporting a case by responding quartile

Survey of ophthalmologists

A total of 582 (68%) ophthalmologists returned their questionnaires. There were no significant differences between the respondents and non-respondents by grade of appointment (P=0.67) or type of hospital (P=0.48). However, there was an over-representation of ophthalmologists who had reported a case, with 65% of respondents having reported at least one case to the BOSU compared with 39% of non-responders (P<0.0001).

The perceived usefulness of the scheme

Forty-five per cent of respondents reported that they considered the BOSU as very useful in assisting the study of rare eye conditions, 50% described it as quite useful and 5% as not at all useful. In all, 39% found the study protocols sent out at the start of each study to be very useful, 55% quite useful and 6% not at all useful.

Satisfaction with the levels of feedback from the unit and from research groups

The majority of respondents felt feedback was sufficient, both from the BOSU (75%) and the research study groups (64%). However, 34% and 23% of respondents wished to receive more information from the research study groups and the BOSU respectively.

Barriers to participation

Figure 4 shows that having to remember the patient's identity until the researchers questionnaire was received and not having return postage paid were considered to be barriers for participation for a large proportion of respondents.

Figure 4
figure 4

Potential barriers to participation.

In all, 67% of respondents had not found it difficult to identify which patient they had notified and to whom the questionnaire referred, 32% experienced some difficulty and 1% were unable to identify the patient. Reasons for experiencing difficulty in identifying the patient are detailed in Figure 5 with almost half being due to not keeping a record of the patients' identity (eg hospital number) after notification of the case to the BOSU.

Figure 5
figure 5

Reasons for experiencing difficulty in identifying reported cases.

Self-reported estimates of completeness of reporting

A total of 74% of respondents were confident that they had reported all cases they were aware of having seen. An estimate of the level of case ascertainment for each study using category midpoints provides a range between 75% for new visual impairment in unilateral amblyopes and 94% for sympathetic ophthalmia (Table 3).

Table 3 Proportion of cases reported by survey respondents

Study applications and research outputs

Table 4 summarises the number of applications by subspecialty and identifiable research outputs, by December 2000.

Table 4 Study applications and research outputs

Discussion

This evaluation of the first 3 years' activities indicates that the BOSU has fulfilled its main objectives and is broadly acceptable to reporting ophthalmologists. Participation rates have increased significantly over time, although more slowly over the past 12 months. Despite this rise in participation, card return rates are less than those achieved by similar national schemes in paediatrics5,6 but equivalent to other specialties.7 Although not directly linked to ascertainment rates, response rates are straightforward to measure and are the most common method for assessing the potential for selection bias.8 Whilst no differences in response rates between ophthalmologists by type of hospital and principal appointment (NHS consultants and other grades) were noted, a variation in regional response rates was evident. Despite this and the positive correlation between card return rates and true case reporting, it remains difficult to estimate the overall effects on ascertainment. This is partly because the magnitude and type of error, whether systematic or random, also depends on any geographical trends in the distribution of the condition studied. However, biased ascertainment is more likely when response rates are low rather than high.

Furthermore, the increased rate of case reporting associated with higher participation also illustrates how achieving high response rates benefits case ascertainment. This association emphasises the value of using an active surveillance system, where case reports are routinely requested and respondents return the report card whether or not a case of interest has been identified to confirm that no cases have been seen. Where compliance is better, it is likely that higher proportions of cases are being identified and reported, which is consistent with comparative trials of surveillance methodologies.2,3

Survey of participating ophthalmologists

The postal survey was anonymised to encourage unprejudiced responses.9 Anonymity should make respondents more comfortable with responding honestly and reduce the tendency for responders to provide answers they perceive to be acceptable (acquiescing).9 However, some may also feel more comfortable not responding, since their non-response will go undetected.8 There were no opportunities to issue reminders or replacement questionnaires, which would have increased the response rate, but the response rate achieved was comparable with similar surveys.8 As there was an over-representation of case reporters amongst those who participated in the survey, there may have been an over-stating of positive sentiments. However, it is known that those with extreme opinions, both positive and negative, are likely to be more motivated to reply than those with no strong opinions.9

A majority of UK ophthalmologists believed that the BOSU was useful in assisting with the investigation of rare eye diseases. A need for more frequent and detailed feedback, especially from the researchers, was observed. The importance of timely feedback to all those who contribute and others who need to know has long been identified as a key component of surveillance.10 One possible reason for the delay in providing feedback is the time required to collect data in these types of study, ranging between 18 months and 3 years in most studies undertaken through the BOSU. In response, the BOSU will now require researchers to report more information about their studies in the BOSU newsletters, without prejudicing the process of peer review. There have been journal papers from some studies and conference presentations from all completed studies. Further papers from studies will appear in due course. The BOSU audits the dissemination of findings and encourages all research groups to publish promptly.

Most characteristics of the BOSU system were broadly acceptable to respondents, with the two exceptions of firstly not being able to report a patient identifier to BOSU at the time of notifying a case, and secondly difficulties experienced in subsequently identifying reported cases. Both do seem to be contributing to questionnaires not being completed and information being lost. Whilst the potential completeness of reporting would be improved if the BOSU collected patient identifiers, this is not feasible as individual researchers hold the ethical permission for the collection of such data.

Case ascertainment

Self-reported levels of case ascertainment indicated a good level of participation amongst those who responded to the survey. However, as with all similar reporting systems, complete case ascertainment through the BOSU is not possible and estimates of ascertainment for each study should be made. The best method to achieve this is to use an alternative independent source of case ascertainment, and apply capture–recapture analysis.11 Unfortunately, for studies completed to date, these sources have not been available. In these circumstances, one approach to investigating ascertainment levels may be a comparison of the rate at which a selected sample of ‘good reporters’ sees new cases per head of population, with the overall rate from the study to provide an estimate of ascertainment. The level of ascertainment could then be estimated as

In this context, good reporters are identified by the principal researcher based upon his/her perception of individual responders' understanding of, and compliance with, the aims and objectives of the study. Attributable populations would be estimated from the Office of National Statistics' population estimates for either health authority districts or regions depending upon how the ‘good’ reporter defined the population they served. The mean, median, 75th percentile and 25th percentile of the distribution of attributable populations then provide a range for calculating the denominator population. The size of the denominator population from which these cases are reported is adjusted to include the populations served by ophthalmologists who did not see any cases. This would be calculated from the proportion of ophthalmologists who did not report a case but had comparable card return rates on the BOSU database. When this method was used for the analysis of data from the study of incidence and causes of loss of vision in the better eye amongst unilateral amblyopes, it provided ascertainment estimates between 64% and 76%. This was similar to the self-reported levels of ascertainment in the postal survey, suggesting that this may provide a useful method for estimating ascertainment levels.

The BOSU is, to our knowledge, the only nationwide active surveillance system for the study of rare eye diseases in the world. This evaluation of its first 3 years indicates that this is an appropriate and effective method for the ascertainment of individuals with such conditions. It is important that both the unit and researchers continue to monitor response rates and examine the influences of bias in samples collected through the BOSU scheme to ensure the methodological rigour and epidemiological integrity of surveillance studies. It is encouraging that the positive perceptions of the BOSU activities outweigh problematic issues. Active surveillance has consistently been demonstrated to be a powerful epidemiological tool in the study of rare diseases.1 It is of note that the majority of UK ophthalmologists already recognise this and support the BOSU and its objectives.