Assessing the readership of the UK National Database of Telemedicine

Introduction The National Database of Telemedicine (NDTM) is a Web site containing information resources on UK telemedicine projects and related work. Its purpose is to provide a source of information to anyone researching the field or proposing a trial or a larger scale implementation of Telemedicine. The NDTM Web site was launched on 27th October 1998 and publicised extensively. Before and after its launch, the Web site has been kept up to date with information about new projects and modifications to the details kept about existing ones. It is clearly important for such a resource to keep track of who is reading it. We may need to tailor information to a particular audience, or bring the Web site to the attention of people who may not yet have found it. This paper describes a mechanism for tracking usage of the Web site that provides more information than conventional approaches. Methods For technical reasons associated with the way the Internet works, the information that can be gathered about visitors to a page is severely limited. Counters can only tell you how many people have accessed a particular page - they do not tell you anything about those people. It is possible to find the Internet address of the visitor's computer and to use the domain name service (DNS) to translate this into a host name (e.g. www.dis.port.ac.uk). This provides some information about the visitor's origins. However obtaining information such as the visitor's name or email address is not possible. There are other factors that affect the ability to collect information, including how the user has configured their browser. However, by seeking the co-operation of the browser, it is possible to glean more information. The paper describes how we did this. Results We have tracked usage of the Web site. The peak of visits was immediately after the launch but since then we have averaged about 120 visits per week. Where identifiable, the majority of these are from within the UK (though we have had visits from at least 40 countries). Over 200 web pages have links to NDTM and that number is rising. Discussion The information obtained from this has proved very useful in developing the Web site. For example, the link that is most frequently followed to NDTM is from the NHS Executive IM&T Web site, indicating that our Web site has a large proportion of its readership within the UK health community. We have also been able to analyse the keywords that people use to find us using the Internet search engines, and revised the content of our pages to make our site easier to find. In the future, we may also use the information to target content at particular types of reader or to help in assessing where on the Internet to set up mirrors of the Web site.


Introduction Historical background to the NDTM website
Between April and September 1996 the Research and Development Division of the United Kingdom (UK) Department of Health (RDD) commissioned a survey of telemedicine activity in the UK [1].The objective of the survey was to provide information that could help with the formulation of policy in this area and to identify the barriers to progress.A further survey of telecare applications was conducted between May and October 1997 [2].
Due to the increased interest in telemedicine and telecare and in response to several requests for information received from Trusts and Health Authorities, RDD decided that the best way to disseminate the information it had gathered would be to create a Website on the Internet.It was felt that for this to be of most use it should contain not only a list of telemedicine and telecare projects and responsible officers but also a list of the suppliers involved and reference to other sources of information on telemedicine.Also it was felt that to have continued value the information needed to be updated as new projects were conducted.
The website is known as the UK National Database of Telemedicine (NDTM) [3].It contains information resources on UK telemedicine projects and related work [4].Its purpose is to provide a background source of information to anyone researching the field or proposing a trial or a larger scale implementation of telemedicine.The NDTM website was launched on 27 th October 1998 and publicised extensively by electronic means such as email and newsgroup messages.Before and after its launch, the website has been kept up to date with information about new projects and modifications to the details kept about existing ones.The database currently (July 1999) contains information about 140 projects and companies.
NDTM is hosted and maintained on behalf of the Department of Health by the University of Portsmouth, with the support of the British Library (Telemedicine Information Service).A three-year contract for continued maintenance and updating of NDTM commenced in May 1999.

Assessing the readership of the website
For an online resource like NDTM, it is important that we keep track of who is reading it.Reasons for this include: • we may need to tailor information to a particular audience • we may need to bring the website to the attention of people who would be interested in its contents but may not yet have found it • there would be no point in expending resources on keeping it up to date if nobody was reading it • our contract with the British Library and the Department of Health requires that we report regularly on usage of the website While it is possible to do a variety of surveys in different forms to determine the demographics of NDTM's readership, in this paper we concentrate exclusively on technical mechanisms based on the website itself.We describe the mechanism we use for tracking usage of the website and show that it provides more information than comparable approaches.We also discuss the information we have collected and draw some conclusions from that data.

Alternative means of measuring readership
Various website authors have adopted a variety of different means to find out who is reading their online material.The most common approaches are: 1. "Guestbooks", in which visitors can record comments about the site.In our experience, visitors infrequently use these, and the visitors who do use them are often unrepresentative of the majority.
2. "Feedback forms" as part of the website.For reasons similar to those of guestbooks, we don't believe this produces information representative of the majority of readers.
3. "Page counters" provided by various Internet services.These provide a raw measure of how many times a particular page has been accessed, but provide no other information.
4. "Access logs" maintained by the web server.These record each web page access and the Internet address from where it was accessed.This allows analysis of how many times each page was accessed and some analysis of origin, but it is not possible to discern from where the reader was referred to the website.A major problem is that the log can quickly consume large amounts of disk space, making it difficult to perform long term analysis.
A more detailed discussion of these approaches, and why they were judged to be unsuitable for NDTM, can be found in a forthcoming paper [5].

The approach we adopted
The limitations in ways of measuring readership are because of the way the Internet works.When a web page is requested, the request does not identify the reader, merely the computer that they are using (strictly its Internet address).Using the domain name service (DNS), it is usually possible to translate the address into a host name (e.g.www.dis.port.ac.uk).This provides some information about the visitor's origins.There are other factors that affect the ability to collect information, including how the user has configured their browser.However, by seeking the co-operation of the browser, it is possible to glean more information.
We subscribed to a free Internet service (based in the Netherlands) called Extreme Tracking [6].This keeps track of visitors to a particular web page (in our case, the NDTM home page) and gathers some basic statistics about them.The big advantage of using this mechanism is that we do not have to take up large amounts of our own disk capacity in storing access logs and other data.The service works because we incorporate into our page an image that has to be fetched from the Extreme Tracking website by the reader's browser.It is the operation of fetching this image that is logged rather than an actual access to NDTM.To collect further information about the reader, we embedded some JavaScript code provided by Extreme Tracking into the NDTM home page.It gathers information available to the visitor's web browser and sends it to Extreme Tracking.This includes how the visitor got to the NDTM site (if they were referred there from elsewhere) as well as some information about their browser itself and the operating system it runs on.
We have observed that the data that Extreme Tracking can collect is subject to certain limitations: • It is possible in a web browser to turn off the automatic loading of images.If loading images is turned off, Extreme Tracking will know nothing about the visit.We assume, however, that the vast majority of visitors do load images automatically.
• Not all browsers support JavaScript and, in those that do, it is possible to turn it off.Under either of those circumstances, Extreme Tracking would not record any additional information about the visitor to the site other than their Internet address.Extreme Tracking estimates that only 3.6% of recorded visits to our site do not have JavaScript enabled.
• The count of visits can be affected by how much disk and memory space the visitor allocates for their cache of recently read pages.If the Extreme Tracking image is not retained in the cache when the home page is accessed again, it will be fetched again and will cause a second visit to be recorded.With a small or non-existent cache, it is possible that some visitors would be counted more than once during the same visit, but we don't believe this would be a significantly large number.
• Extreme Tracking attempts to distinguish between the first visit and a repeated visit to the site.We believe it does this on the basis of visits from the same Internet address.However, many Internet Service Providers (ISPs) maintain a pool of addresses which they allocate dynamically on demand when users (typically) dial-in to the service.It is likely therefore the same user who accesses NDTM on several separate occasions through a dialup service will be regarded as several distinct users since each access session will likely have a different Internet address.By the same token, it is also possible that different users will appear to have the same Internet address.The results presented below are therefore qualified by the above.

Quantity of visits
We enabled the tracking on the website several months before its launch.It was therefore possible to observe the effect the launch publicity had on the number of visits to the site.As expected because of the publicity, the graph peaked shortly after the launch.Indeed, we had over 400 visitors in the four days between the launch on 27 th October 1998 and the end of that month.Since then it has settled down to around 200 visits a week, though the number appears to be declining slowly.
It is interesting to note that about 30 people a week were visiting the site before it was officially launched.It is clear that publicity is not always necessary for a website to be found.
We estimate that the total number of visits to the website is at least 9000 (since we started counting) and that there have been around 5000 distinct visitors.

What do we know about the visitors?
About 34% of the recorded visitors to the NDTM website are from sites within the ".uk" Internet domain; it is therefore reasonable to assume they are located in the UK.This proportion has been slowly falling (it was 40% in January 1999) and at the same time the proportion of visits from locations Extreme Tracking is unable to identify has been increasing (from 12% to 22%).We suspect this is because ISPs and some other large users do not always register their pools of Internet addresses with the domain name service.
The domains ".net" and ".com" account for 16% and 12% respectively of our readers.Many of these addresses tend to represent international ISPs such as AOL or CompuServe, making it impossible to tell whether a user is in the USA, UK or elsewhere in the world.
As well as the UK and the USA, NDTM has had visitors from many other countries, though no single one accounts for more than 2% of visits.Australia, Canada and Germany are the countries from which most visits have been made.
We have statistics on what time of day people visit the site.Most visits (62%) are during UK normal business hours (i.e.9am to 5pm), with afternoons slightly busier than mornings (22.4% between 9am and 12noon, 24.9% between 2pm and 4pm).About 8.5% of visits are between midnight and 7am.Access is very much a weekday activity -only about 10% of visits are at weekends.
We know a little about the technical capabilities of our visitors.We can tell that Microsoft is winning the web browser war.52% of our visitors use Internet Explorer, while 46% use Netscape.Back in January, Netscape was in the lead 54% to 45%.We can also see how the market for operating systems has changed since earlier this year.The proportion of our readers using each of the major operating systems is shown in Table 1.

How do visitors find the website?
From data collected by Extreme Tracking we can tell by what means many of the visitors to the site find it.They come in four ways: via links from another website, as the result of an Internet search engine, via an email message or via a USENET news message.

Website links
The most common way to find NDTM is via a link from a page on another website.This accounts for 55% of tracked visits.Extreme Tracking has identified 288 web pages that at some time during the lifetime of the project appear to have contained links to NDTM.Spot checks on these have shown that some do not appear to exist any more, some no longer contain links to NDTM, and some do not appear to have anything to do with telemedicine or NDTM.Also, of the pages that do contain valid links to NDTM, by inspection of their URLs we can see that some of these are obviously duplicates, indicating mirrored websites or where a website has been moved to a different server.Nevertheless, we have identified 30 distinct websites that contain links to NDTM.
The data reveals that by far the most frequently followed link to NDTM is the one on the National Health Service (NHS) Information Authority website [7].That page alone accounts for 35% of the total number of referrers via websites.The next (with 15% of referrers) is the NDTM website itself, as Extreme Tracking counts visitors who move to our home page after entering the site at another of its pages.2.8% of references come via a link from the web site of the Radiological Society of North America [8].No other website accounts for more than 2% of tracked references.However, NDTM is (or has been) referred to by a number of other prominent telemedicine websites, including the Telemedicine Information Exchange (TIE) [9], Telemedicine Today magazine [10], and the Royal Society of Medicine's telemedicine forum [11], although they have not (yet) produced significant numbers of referrals.

Search engines
Although finding it via another website is still the most popular means of locating NDTM, the use of search engines is rapidly catching up.Back in January 1999, only 8% of our referrals came via search engines.In July 1999 that figure was up to 39%.
Yahoo [12] is much the most popular, producing 80% of successful search engine referrals to NDTM.Behind Yahoo, Excite [13] produces 9% of referrals while Alta Vista [14] produces about 3%.A total of 16 different search engines have found NDTM at least once and we will be working to increase that number.
It took us some time and effort to get NDTM registered with Yahoo.We were keen to get registered with it since it is considered one of the major Internet resources of this type, as our results bear out.We submitted the site to them on four separate occasions between October 1998 and January 1999 before they acknowledged it.
The most common keyword used to find NDTM via a search engine is, unsurprisingly, "telemedicine" (26% of successful searches).Other significant keywords used in more than 3% of queries were "UK" (11%), "health" (8%) and "telecare" (4%).Periodically we test the search engines to see which of them would turn up a link to NDTM given certain search strings.That aspect of our work will be the subject of a separate paper.
It should be emphasised that the references being counted here are those where the reader has specifically chosen to visit NDTM, and therefore regarding these visits as being the result of "successful" searches is reasonable.Presumably, NDTM has also cropped up in a number of other searches where the reader has chosen not to visit our site.These will not have been counted by Extreme Tracking.In addition, of course it is not possible (at least by the means we currently employ) to discern information about people who might want to read NDTM but who, by choosing inappropriate keywords or the wrong search engine, never find it.
Other references About 6% of references to NDTM came from email or newsgroup messages.These are presumed to be people who have received messages notifying them of NDTM's address and who invoke their web browser from within their email or news program to look at the site.References from email and newsgroup messages have tailed off over the course of the project.The last recorded reference via a newsgroup message was on 18 th November 1998 and there have been only 17 via email since 7 th January 1999.This indicates that our initial publicity, sent out via these means, probably caused most of these visits.Since then, people have tended to find the site by other means.
A note on the data A word of caution though.Tracked visits comprise only 54% of the total number of visits to the site.The remaining 46% we can speculate are by people who: • access us by one of the above means, but have JavaScript turned off in their browser, so defeating the efforts of Extreme Tracking • people who move on from the home page to another page before the JavaScript to record their visit has finished executing • have a link to NDTM in their "bookmarks" file or as an Internet shortcut • type our address directly into their web browser, perhaps after reading an email or newsgroup message referring to NDTM, or hearing about it by word of mouth

Discussion
The information we have obtained by tracking readership of NDTM has proved to be of considerable interest.The number of people who have apparently visited the site has amazed us.The figure of 5000 is by an order of magnitude bigger than the estimated size of the UK telemedicine community (its intended audience), and by no means do we believe all of that community has accessed it yet.Clearly a number of visitors to the site may have got there as a result of idle curiosity or by mistake, but many people have found it a useful resource.

Conclusions drawn
From our observations we have drawn the following conclusions: 1. Most readers are from the United Kingdom.This is based on the preponderance of visits from Internet addresses in the ".uk" domain, together with the observation that the majority of visits are during UK working hours.
2. The fact that the majority of visits are during UK working hours also suggests that most people are accessing NDTM as part of their jobs rather than as a spare time activity.This is encouraging since it indicates the seriousness with which people regard telemedicine.
3. The website continues to reach new people.Although some of the first time visits may be accounted for by repeat visits by people using a different Internet address, this cannot reasonably account for all of them.The number of new visitors to the site has remained high despite the initial publicity having worn off.
4. People find the contents of NDTM to be of value.The substantial number of repeat visits recorded supports this, and, for the reasons described above, that number is likely to be an underestimate of the actual number.People would not return to the site unless they found it useful to them.
5. Many of the people who access NDTM are within the UK health community.This is indicated by the high incidence of visits to the site that come via the NHS Information Authority website.
6.A potential reader looking for telemedicine websites would be very likely to find NDTM.This is supported by two pieces of evidence: a) the number of links to NDTM on other prominent websites, and b) the increasing number of references to our site emanating from Internet search engines 7. Data gathered by Extreme Tracking shows that Microsoft operating systems dominate the market represented by NDTM's readers.Also, Internet Explorer is gradually becoming more popular than Netscape in the web browsing market.

Future work
The information we have gathered has, however, not enabled us to answer all the questions we have about our readership.Some, together with ways in which the answers might be obtained, are listed below.
1. We anticipate that our readership might be broadly divided into two categories: those who are users or potential users of telemedicine systems, and those who are developers or suppliers of equipment that the former might use.What proportion are in each category?More detailed analysis of the Internet domains from which NDTM is accessed might reveal the answer.We might assume that readers in the ".nhs.uk"(UK National Health Service sites) domain are by and large in the former category, while those in the ".co.uk" and ".ltd.uk"(UK commercial) domains are in the latter category.
2. How many of NDTM's readers are actively involved in telemedicine and how many are currently only thinking about or planning it?By matching the Internet addresses or domains of visitors against those stored in the database itself, we should be able to get some measure of how many of our readers are associated with projects or companies we already know about.
3. How many more readers of NDTM can we stimulate by publicising its existence more?
We have in mind to do a series of electronic mailshots at periodic intervals in order to observe their effect on the number of visits to the site.The information we have collected to date allows us to identify a base population against which to measure any change.
4. How many potential readers of NDTM fail to find the website?We plan to regularly test the efficacy of the Internet search engines by giving them possible search strings and seeing whether they suggest NDTM in response.The results can be used to suggest steps to take to improve efficacy in the future, e.g. by registering NDTM with new search engines or by improving the quality of keywords associated with our pages.We are particularly interested in ensuring that NDTM's contents will be found by healthcarespecific search engines and other online specialist library facilities.

Summary
We are continuing to develop the NDTM website in ways that will improve its efficiency in providing information to readers and gathering new information about telemedicine projects and companies that it does not yet know about.During those developments, the information we gather about our readership allows us to take their particular requirements into account when assessing our priorities.Conventional technologies such as guestbooks and counters do not provide as much information as the mechanism that we have chosen to use.Our results show that much useful information can be collected by this means.

Figure 1 Figure 1 -
Figure1shows, month by month, the average number of visitors who have visited the site each week.(N.B.Due to technical difficulties, no data was collected between 24 th November 1998 and 7 th January 1999, so we have no December data point and the November and January figures are each based on only 24 days' figures.)

Table 1 -
Proportion of visitors using each operating system