Welcome to the Journal of Medical Internet Research

Background: The Internet offers a great amount of health related websites, but concern has been raised about their reliability. Several subjective evaluation criteria and websites rating systems have been proposed as a help for the Internet users to distinguish among web resources with different quality, but their efficacy has not been proven. Objective: To evaluate the agreement of a subset of Internet rating systems editorial boards regarding their evaluations of a sample of pediatric websites. To evaluate certain websites characteristics as possible quality indicators for pediatric websites. Methods: Comparative survey of the results of systematic evaluations of the contents and formal aspects of a sample of pediatric websites, with the number of daily visits to those websites, the time since their last update, the impact factor of their authors or editors, and the number of websites linked to them. Results: 363 websites were compiled from eight rating systems. Only 25 were indexed and evaluated by at least two rating systems. This subset included more updated and more linked websites. There was no correlation among the results of the evaluation of these 25 websites by the rating systems. The number of inbound links to the websites significantly correlated with their updating frequency (p<.001), with the number of daily visits (p=.005), and with the results of their evaluation by the largest rating system, HealthAtoZ (p<.001). The websites updating frequency also significantly correlated with the results of the websites evaluation by HealthAtoZ, both about their contents (p=.001) and their total values (p<.05). The number of daily visits significantly correlated (p<.05) with the results of the evaluations by Medical Matrix. Conclusions: Some websites characteristics as the number of daily visits, their updating frequency and, overall, the number of websites linked to them, correlate with their evaluation by some of the largest rating systems on the Internet, what means that certain indexes obtained from the usage analysis of pediatric websites could be used as quality indicators. On the other hand, the citation analysis on the Web by the quantification of inbound links to medical websites could be an objective and feasible tool in rating great amounts of websites. (J Med Internet Res 1999;1(1):e1) doi:10.2196/jmir.1.1.e1

Welcome to the "Journal of Medical Internet Research" -JMIR, the first international scientific peer-reviewed journal on all aspects of research, information and communication in healthcare using Internet and Intranet-related technologies.
Why does the world need the JMIR? The Internet -and more specifically, the World-Wide-Web -has an impact on many areas of medicine -broadly we can divide them into "clinical information and telemedicine", "medical education and information exchange" and "consumer health informatics": • First, Internet protocols are used for clinical information and communication. In the future, Internet technology will be the platform for many telemedical applications.
• Second, the Internet revolutionizes the gathering, access and dissemination of non-clinical information in medicine: Bibliographic and factual databases are now world-wide accessible via graphical user interfaces, epidemiological and public health information can be gathered using the Internet, and increasingly the Internet is used for interactive medical education applications.
• Third, the Internet plays an important role for consumer health education, health promotion and teleprevention. (As an aside, it should be emphasized that "health education" on the Internet goes beyond the traditional model of health education, where a medical professional teaches the patient: On the Internet, much "health education" is done "consumer-to-consumer" by means of patient self support groups organizing in cyberspace. These patient-to-patient interchanges are becoming an important part of healthcare and are redefining the traditional model of preventive medicine and health promotion).
All these aspects of "cybermedicine" have implications for consumer empowerment and evidence-based medicine: The Internet (or Intranets) enables health professionals to access clinical data just in time, it allows health professionals to access the evidence on the efficacy of available interventions, and finally, it empowers consumers to actively take part in the decision making process.
Clearly, the medical use of the Internet presents enormous opportunities and challenges. The need for research and rapid publication of the findings is obvious. Research in this area should go beyond mere development and provision of technical solutions; it should also address social and human factors, and evaluate the impact of the Internet on society and health care, and public health.
JMIR wishes to publish papers that help physicians and consumers to maximize the use of the Internet. We invite researchers to evaluate the effectiveness and efficiency of Internet communications in health care. We encourage publishers of Internet health information to apply rigorous research methods (such as randomization of users) to evaluate different methods and determinants of communication effectiveness. We invite researchers to compare the effectiveness of communication and information on the Internet with other (traditional) methods of communication. We call for papers that describe and evaluate the effects of the Internet on the patient-physician relationship and the impact on public health. We invite papers that describe the use of the Internet for evidence-based medicine, for example work that demonstrates the development and dissemination of clinical guidelines using the Internet. We wish to receive papers on ethical and legal problems, as well as cross-border and cross-cultural issues, affecting medicine on the Internet, and papers describing possible solutions to the problem of equity of information access. We would like to receive systematic studies examining the quality of medical information available in various online venues. We encourage thought regarding methods of evaluation, quality assessment and improvement of Internet information. We would like to receive proposals for standards in the field of medical publishing on the Internet, including self-regulation issues, policies and guidelines to provide reliable healthcare information. We encourage researchers to experiment with online questionnaires and other data collection experiments such as medical surveys and psychological tests. We would like to publish innovative approaches to use the Internet for healthcare research, examples might include clinical studies, drug reaction reporting and surveillance systems. We would like to publish comments and papers on electronic medical publishing and the use of the Internet for traditional scholarly publishing. We welcome descriptions of websites with innovative content or form, but authors should always make attempts to evaluate the impact of their work, for example trying to determine basic information about user demographics and traffic, where appropriate.

Introduction
After the early enthusiasm generated by the potential use of the Internet in Medicine [1,2,3], concern has been raised about the quality of the resources available on the Internet compared to more academic media. It is technically very easy to publish on the Internet [4]. The lack of a review process of the documents on the Net, and the power of this media in transmitting the data has the risk of misinforming both lay people [5,6,7] and health care professionals [8]. However, only a few studies have tried to measure this risk of misinformation [9,10,11]. Nothing yet is known about the users' ability to discriminate between low and high quality resources.
Several initiatives have been proposed which could be applied at different levels to improve the average quality of medical websites. For instance, we could apply certain basic methods for the websites to be correctly designed. In this sense, some academic organizations have proposed a set of basic information that every medical web site should provide about the author and sources of the web site contents, their potential conflicts of interest and funding, and the currency of the information [12]. But many of the available medical websites have been created without any quality control by a third party. How can Internet health care visitors distinguish between such different resources?
Internet users can find health and medical related websites in several ways. World Wide Web search engines (e.g., AltaVista, Excite, Infoseek and many others) provide the users with a list of websites that match a given topic, with the results ordered by syntactic similarity with the query [13]. Unfortunately, the quality of contents is not guaranteed.
On the other hand, certain websites indexes and review services, such as Medical Matrix (http://www.medmatrix.org/) and HealthAtoZ (http://www.HealthAtoZ.com/), offer systematic evaluations of medical resources on the Web [14], as a post publication editorial process. These rating systems could be an useful tool for guiding the visitors of medical websites [12]. However, authors who have reviewed these Internet resources, point out the variability of their evaluation criteria and their doubtful efficacy [14].
The quality of a given medical article on the Internet could be measured by the users opinion about it, for example by counting the number of times it is retrieved [15]. However, this idea has been criticized because it would replace the scientific peer review process with the opinion of the Internet users, whatever their qualification [6].
Despite the differences between the printed medical information and the Internet, several evaluation tools from the former could be useful if applied on the "Net." Similarly to printed medical journals, medical documents on the Internet could be ranked by their citation analysis [15,16], but no methods have been proven for use with medical websites. When an article is quoted in a paper, certain agreement among the authors may be supposed. Similarly, when a webmaster makes a link from his web site to another, certain credibility is given to the latter. In fact, the International Committee of Medical Journals Editors recommend caution when a link is made from a peer reviewed journal site to other sites [17]. If linking on the web can be equivalent to quoting in printed medical articles, a citation analysis on the web could be performed by the quantification of the links to a given medical web site.
The ideal method for assessing the quality of medical websites should provide a means of rating great amounts of medical web resources while respecting the World Wide Web peculiarities, such as its multimedia capabilities and changing contents. At the same time, it should at least be as reliable as systematic reviews of those resources by editorial boards. In summary, it should be a method born in the Internet but with the efficacy of those used in the printed media.
In this study, we evaluated the reliability of four websites characteristics as medical websites quality indicators. The four characteristis used: their authors' impact factor, their grade of updating, their daily visits and inbound links. The evaluations of a sample of pediatric websites by a number of Internet rating systems was the gold standard with which these websites characteristics were compared.

Methods
During March 1998 a subset of websites rating systems were compiled. From these, we selected a sample of websites that were studied during the first week of April 1998.
Eight web rating systems, whose evaluations were offered as figures, were compiled from previous studies [13,14] (Table 1). One half of the selected rating systems gave the results of their evaluations by means of graphic analog scales, and the other half by numeric scales. Every web site evaluated by these rating systems that provided information about child health, whether for lay people or health professionals, was included in the study. Some of these rating systems (e.g., Lycos Top 5%) provides the visitors with a search tool by keyword. In these cases, the websites were selected using the keywords "Pediatrics", "Infancy", "Child health", and "Child Care." For the remaining rating systems, the pediatric websites were compiled manually. Those websites not accessible twice during the study period were excluded.
Only three rating systems (Medical Matrix, Physician Choice, and Six Senses) gave information about their editorial boards. Most of their members were physicians. Two of the web rating systems only gave a global result of their websites evaluation (Medical Matrix and Magellan), while the rest (HealthAtoZ, Argus Clearinghouse, Lycos Top 5%, Sympatico Health, Physician Choice, and Six Senses) gave a result for each considered criterion. Content was a common criterion to all the eight ranking systems. Therefore, the results of the evaluation of each web site were divided in two categories, content and non-content (design) aspects. In order to make comparisons, the results of the evaluations of the websites supplied by each rating system were transformed to a one hundred scale. When provided, the daily visits registered by the websites visits counters were recorded. In some websites the date from which the counter was started was not available. Thus, their webmasters were asked for this information by electronic mail, and it was included in the statistical study if provided before the end of the observation period, 15th April 1998.
The websites authors and editors' names were searched in 1997 MEDLINE [18], and their articles were registered. Their impact factors of the journals wherein they were published were obtained by using the 1996 Science Citation Index (Institute for Scientific Information, Philadelphia, PA). The impact factor of a given web site author was the sum of the impact factors of his or her articles. For institutional websites only the name of the web editor was considered.
When provided, the time since the last update was also recorded.
Finally, by means of the Web search engine Infoseek [19], we calculated how many websites on the Internet linked to each web site of our sample. The searching strategy by syntax of this engine allows to know the websites that are linked to a given web site [20]. As a web site may be linked not only from external websites but also from websites of its own organization, we only considered external links. Although other search engines such as AltaVista, Excite or HotBot offer similar searching options, we chose Infoseek because it provided the results of the queries grouped by web site, which makes the exclusion of the internal links easier.
Comparison of means was performed by Mann-Whitney U test, and correlation analysis by means of Spearman's correlation coefficient ( r S ). P values equal or less than .05 were considered significant. All computations were made with SPSS for Windows 7.0 (SPSS Inc., Chicago, IL) statistical package.

Results
After excluding 93 non-accessible websites, a total of 363 pediatric websites were compiled. On average, the websites of our sample received links from 470 other sites on the Internet (range, 0 to 3574). In 48% of the websites, information on their last update was given. On average, they had been updated 47.5 weeks before (range, 0 to 395). Only 10% of the websites had a visit counter, and the average daily visits were 470 (range, 1.2 to 3145). Seven visit counters did not distinguish among different visitors, that is, they registered any visit to their websites. In 137 websites (38%) the editor/author's name was given, but only 60 of them had published at least one article since January 1997 in the journals included in MEDLINE database. Their average impact factor was 2.14. Figure 1. Weeks since the last update for the total of the sample, n=363, and for the websites evaluated at least by two rating systems, n=25 (median, 25 th and 75 th percentiles) Figure 2. Number of inbound links to websites for the total of the sample, n=363, and for the websites evaluated at least by two rating systems, n=25 (median, 25 th and 75 th percentiles) Table 4. Top 50 pediatric web sites of the sample (N= 363) by the number of their inbound links. The weeks since the last update, the number of daily visits to the web sites and their editor/author's impact factor are also provided. In parenthesis, the place that each web site would obtain if ranked by the two latter criteria. In italics, those web sites indexed at least by two rating systems. Missing values are due to the lack of visits counter, editor's name, or information about the last update, for many web sites Only 25 websites of the sample were indexed and evaluated at least by two rating systems, and none by the eight. This subset of websites showed significantly better results of the evaluation of their contents and design by HealthAtoZ, and higher grade of updating ( Figure 1) and higher number of inbound links ( Figure 2). When the evaluations of these 25 websites by the different rating systems were compared, no significant correlations were found. Changes regarding the average impact factor of the authors of the websites or the number of daily visits could not be demonstrated in this subset of websites.
Some interesting correlations between the results of the evaluations of the websites and the rest of study variables were found. The number of links received by the websites significantly correlated with their daily visits and with the time since the last update ( Table 2). The number of inbound links also correlated with the results of the websites evaluation by HealthAtoZ ( Table 3).
The number of daily visits significantly correlated with the results of the websites evaluation made by Medical Matrix, and the grade of updating significantly correlated with the results of the contents and designs evaluation made by HealthAtoZ (Table 3).
Finally, no correlation was demonstrated between the average impact factor of the websites authors and the other variables.
The top fifty pediatric websites of the sample are shown in Table  4, ordered by the number of their inbound links according to the Infoseek indexing engine. More than a half of the 25 websites indexed by at least two rating systems may be found among these top fifty websites.

Discussion
In this study, certain websites characteristics that depend on the users' preferences have been compared with evaluations of pediatric resources on the Web by third parties. Although rating systems have been previously criticized because their editorial boards frequently do not employ uniform criteria [13], we have considered them as the standard method because it somewhat represents a post-publication review process.
Some aspects of our method are open to discussion. Firstly, the reliability of the data regarding the daily visits and the updating frequency depends on the accuracy of the information that the websites editors offer in their sites. In this sense, we considered the grade of updating of the websites by the dates of their last changes. Clearly these changes could involve very different aspects and in different grades, and not necessarily provide more current contents. However, we believe that it could demonstrate the editor's efforts in maintaining or increasing the interest of his web site for the visitors.
The results regarding the number of daily visits to the websites must be considered with caution when comparing one web site to another, because some visit counters were set to register every visit, instead of every distinct visitor. Nevertheless, both can be considered usage indexes of a given web site.
On the other hand, quantification of links to the websites clearly depends on the power of the search engine we employ. By no means our results show the total number of links to the websites in our sample. In fact, a previous article states that it would be necessary to combine the databases from at least five large search engines to cover the most of the web [21].
Although all bibliometric indexes have limitations [22,23], we employed the impact factor as a measure of the webmasters' publishing capacity because it is a classical indicator of the quality of biomedical articles. Recently, it has been suggested that every medical web site should be evaluated following some basic criteria [24]. One of the more accepted criteria is that the authorship must be clearly stated, as a basic means for assessing the reliability of the web site contents. However, we could not demonstrate that the more highly evaluated, the most updated, or the most linked or visited pediatric websites, had the authors with the highest publishing capacity measured by their impact factor. In other words, some web quality standards do not correlate with classical quality standards from the printed media such as the impact factor of a given author's articles.
We could not find statically significant correlations among the evaluations of the websites by the different rating systems. This is probably due to the small size of the subset of websites indexed and evaluated by all the systems, and their different evaluation criteria. However, some interesting data were found when we considered the correlations among the four websites characteristics and the evaluations. We found that the best websites for HealthAtoZ, the largest analyzed rating system, were the most updated and the most linked ones. On the other hand, the most valuable websites for Medical Matrix, the second rating system by size, were the most visited ones. In any case, both the number of daily visits and the time since the last update highly correlated with the number of inbound links. The lack of correlation among the four variables and the evaluations by the other rating systems could be due to their little contribution to our sample.
Many efforts to establish quality criteria will have limited efficacy due to the dynamic behaviour of the Internet as a publishing medium. In fact, a recent article demonstrates the lack of consensus among the editorial boards of a large sample of evaluation and rating systems regarding the evaluation criteria they employ. The same authors pointed out that "... it may be difficult or even inappropriate to develop a static tool or system for assessing health related websites." [25] Therefore, the question could be to provide context to this issue. That is, to know how good a given medical web site is in comparison with the rest of medical websites. A democratic and feasible method for reaching this objective could be let the Internet community say which medical websites are the best ones, that is, which they usually visit or which they usually recommend by linking to them. Moreover, we believe that the fact that these usage indexes correlate with the evaluations by third parties, qualifies them as quality markers.
Eysenbach and Diepgen [16] have recently proposed that an ideal quality control system for medical resources on the Internet should take in account the users opinion, and not only their evaluation by a third party, that is, a "downstream filtering" and not only an "upstream filtering" approach. More interestingly, our study demonstrates certain agreement among both approaches in identifying high quality resources.
LaPorte et al [15] proposed an electronic publishing system in which the impact of a given resource on the Internet could be measured by counting how many times the document was retrieved or quoted. The introduction of the citation analysis of the medical resources on the web as a method to assess their quality has been recently proposed [16]. On the other hand, a very promising software system is being developed by Kleinberg [26,27]. This system would provide the users with a way of knowing the very best of the web on a given topic in a faster and more complete way than commercial human compiled directories. This system is based in the identification of two subsets of websites when a query on a given topic is made, those websites containing a lot of information about the topic (authoritative websites) and those which contain large amounts of links to the former (hub sites). Our work demonstrates that those authoritative websites, that is the more linked ones, are indeed the best ones regarding the evaluation of its contents and design by the editorial boards of some large web rating systems.
The citation analysis of biomedical journals has been a classic tool in assessing their relative quality [28]. Similarly, medical web resources could be ranked by a "webcite index" [16], which is not yet defined. Linking in the World Wide Web could be equivalent to quoting in printed publications, and its quantification could be useful for measuring the relative quality of medical websites. Some indexes could be created to make more rational comparisons among websites with different sizes. For example, in the same way that the calculation of the impact factor of a given medical journal takes into account the number of articles published by that journal yearly, the size of a given domain could be considered to obtain some indexes that would express more accurately the grade of linkage of a medical web site. Moreover, Platform for Internet Content Selection (PICS) [29], an infrastructure that could be applied as a filtering system of the medical information on the Net [16], could incorporate these indexes as one of the meta-data assigned to every medical document as electronic labels. Then, these electronic labels could be checked automatically by an user's browser, bypassing those documents with a "webcite index" not high enough. A problem could be how to avoid false "self-labelling" by dishonest webmasters. In any case, more work is needed to give answers to these and other technical questions on the emerging field of Webometrics [30].
An evaluation system based on this quantification would bring advantages and risks. Rankings could be generated very quickly and in an objective way, because the Internet community by itself would evaluate great amounts of medical websites. However, this evaluation process would be made a posteriori, and the potential harmful effects of the diffusion of documents without enough quality could not be avoided. Therefore, this method could not replace previous editorial effort that warrants a minimal quality for each resource.
Our work demonstrates that the visitors of pediatric websites and the editors of websites on the "Net," so called webmasters, show certain maturity when they have to identify the pediatric resources with high quality. We believe that the key point is how to augment the proportion of these resources. An important issue could be to establish a citation style not only for articles from peer reviewed electronic journals [31], but also for any medical document on the Net. The prestige that citation in a printed journal represents will stimulate high quality publishing on the Internet, and web site editors will employ enough review processes to obtain the necessary quality. A web site's ranking system based on the citation analysis on the web by the quantification of links would be an additional incentive. The more valuable resources will attract the Internet users' visits and the webmasters' links, and very likely the best funding and financial supports.
In summary, although the Internet provides a very different publishing medium, traditional means borrowed from printed journals could also be used with this electronic media for achieving minimal levels of quality. These include certain peer review processes, that enhance the rigor of the documents submitted for publication taking in account the peculiarities of this media, and linking analysis as a measure of the citation on the World Wide Web.

Introduction
The rapid growth and constant change in the number of health care related web sites make the evaluation of the quality of the web sites a difficult but beneficial task. The Internet is "a medium in which anyone with a computer can serve simultaneously as author, editor, and publisher and can fill any or all of these roles anonymously if he or she so choose. In such an environment, novices and savvy Internet users alike can have trouble distinguishing the wheat from the chaff, the useful from the harmful" [1]. In a systematic search by means of two search engines (Yahoo and Excite) for parent-oriented web pages relating to home management of feverish children, [2] the investigators of this study, compared the web site information with the guidelines to parents for managing fever at home supplied by a printed book. The investigators found among 41 web pages retrieved and reviewed: 28 web pages gave a specific temperature above which a child is feverish, 26 pages indicated the optimal site for taking temperature, 38 pages recommended non-drug measures, and 36 pages gave some indication of when a doctor should be called. Only four web pages adhered closely to the main recommendations in the guidelines. The investigators concluded from these observations only a few web sites provided complete and accurate information for this common and widely discussed condition. According to McClung, [3] 48 out of 60 major medical institution web sites checked had inaccurate information about the treatment of childhood diarrhea. While it is virtually impossible (and probably undesirable) to control the content of web pages, it is certainly useful to have some measure of the quality of the information provided.
One possibility is to establish an official rating system based on standard criteria. In the survey mentioned above, [2] the author also suggested an urgent need to check public oriented health care information on the Internet for accuracy, completeness, and consistency. Many attempts have been made, and core standards that can help to achieve these goals have been developed. The most widely accepted suggestion is adapting the five traditional print evaluation criteria: accuracy, authority, objectivity, currency and coverage, to web resources [4,5,6].
However, "many Internet users object strongly to any 'official' attempts to regulate information", though few want to see inaccurate information appearing! In addition, "the Web's interactive format means criteria used for paper-based journals may not be valid for web-based information." [7]. Jadad points out that the "Net's very nature makes this difficult, if not impossible". After an investigation to identify instruments used to rate web sites providing health information on the Internet, Jadad concluded, "many incompletely developed instruments to evaluate health information exist on the Internet. It is unclear, however, whether they should exist in the first place, whether they measure what they claim to measure, or whether they lead to more good than harm" [8]. At this point it is very difficult to reach or develop a standard that every user of the Internet could observe.
It has been suggested that Web sites can be evaluated in a similar way to traditional print media. When we evaluate a textbook or a journal, we not only assess the authors, content, and structure, but also more objectively, measure the impact of the publication on its readers. Citation analysis, the practice of counting citations to determine the scholarly impact of a work, is a method long used by librarians as an important tool of collection development. With bibliometrics the impact of a journal is evaluated by the frequency that it was cited during a certain period.
One major instrument to evaluate scientific journals is Journal Citation Reports (JCR) [9]. JCR is published by the Institute for Scientific Information and includes several citation-based measures of journal impact for the journals that they review. Librarians and researchers can utilize JCR to see how many times and how quickly articles published in certain journals are cited. There is also a measure of effectiveness, the impact factor, which normalizes the citations received by the selected journals and looks only at the previous two years of publication.
Though there is no similar tool available to evaluate the impact of a WWW page, it is comparatively easy to determine which pages are cited ("linked to") by the compilers of other pages. We also found a study conducted on the WWW pages of selected fine art libraries [10]. By analyzing the linked sources on art library Web pages, Neth's study found a set of twenty commonly cited WWW sites out of thousands of cited links. As we investigate health science related web sites, we also find some well-established sites already use this method successfully. For example the compilers for the Hardin-Meta of the University of Iowa look at many sites in each field and chose the lists that are most frequently cited by people in the field. This analysis provides a rudimentary form of peer evaluation. They call it a "list of lists" [11]. Another example, in a paper on the quality management of medical information on the Internet by Eysenbach, the author presented some indirect quality indicators, among them is "Web citation". A "webcite index," analogous to the Science Citation Index, could be compiled from the absolute number of hyperlinks to a certain website or new hyperlinks established over a period of time [17]. The author has developed a website network (http://webcite.net) contributing and practicing this methodology.
In this paper, we analysis the pages linked to in the "other links" sections of the web pages of a selection of the top 25 US medical schools. On the assumption that a Web Master will only cite or link to pages he/she thinks are authoritative. We examine the links made from these pages and obtain a listing of the most cited pages. This affords a new approach to evaluate web sites by using the principles of citation analysis. (1). Sample selection:

Methods
The selection of the "key sites" used to count the most frequently cited web sites is very important. For our approach, we used the listing of "the top 25 medical schools in the United States" as published by U.S. News and World Report [12]. Next we identified their primary health information WWW site. Normally this was the home page for the medical school library. Among these 25 medical schools, the web pages for seven of the medical schools were eliminated due to technical limitation of the URL checking software and the variations of the Web sites.
We finally examined the web pages of nineteen of the top twenty-five US medical schools. The top 25 are listed in Table  1 with those eliminated from this study indicated by an asterisk (*).  (2). Ranking the web sites by the cited frequency The next step was to examine the links made from these pages. This was achieved by using a software program "Checkweb" [13], which checks the links of the selected web page and reports which ones have moved, or cannot be located or connected to. The second step is to clean up this list of and eliminate the orphans (Status 404 -no longer existing and Status 301 and 302 -moved), and the "noise items". Noise items are "noise" from the host web page such as "go home" or links to other sections on the same site. This ensures that the final list is only to active links to external URLs.  Table  2.

Results and Analysis
The three levels of URLs were counted and the results are shown in Table 3, Table 4 and Table 5.
The frequency of links is very concentrated in several TLDs, notably .edu, .com, and .gov and. org. These accounted for 88.61% of the Links. Table 3 the most highly cited TLDs (greater than 600 times) are .edu, .com, .gov, and .org. These TLD's are all registered in the United States. Other less cited US TLDs are .net, .us and .mil. The United States related web pages account for almost 90% of the URLs cited. This was not unexpected because the source samples are U.S. medical schools and the Internet is highly developed in this country. Among the US TLDs, those from four years colleges and universities, those entitled to use the .edu suffix, are cited most frequently and therefore are considered the most important. The .edu suffix accounts for almost one third of all links.

As shown in
Other countries whose TLDs are frequently cited are United Kingdom (uk), Switzerland (ch), Canada (ca), Germany (de), Australia (au), Sweden (se) and Netherlands (nl). This distribution is very similar to the results of 30 nations ranked by the citations per paper from 1992 to 1996 by Institute of Scientific Information (ISI) published in the Science Watch [14]. In this study the top ten nations were Switzerland, United States, Netherlands, Sweden, Denmark, United Kingdom, Belgium, Finland, Canada, and Germany. It seems that in some degree our results may also represent the developmental level of medical information publishing and research in the world. However, the focus of this paper is not placed on the comparison of these two lists.  (2). Distribution of the First Level Domains.
One of the goals of this study was to identify the web sites cited most frequently by US academic health sciences libraries. Table  3 shows that a total of 1731 web sites were cited by (linked to) these 19 institutional home pages.
According to the Bradford 's Law of Scatter: [15] "if scientific journals are arranged in order of decreasing productivity of articles on a given subject, they may be divided into a nucleus of periodicals more particularly devoted to the subject and several groups or zones containing the same number of articles as the nucleus, when the numbers of periodicals in the nucleus and succeeding zones will be as 1:n:n2 ". In our study, we list the web sites in order of decreasing frequency of citation, and as Bradford has done in his original paper we divide the total cited times of the web sites into 3 equal sections. The first section is the top 78 web sites (as shown in details in Table 3) 33.69% of total cited times, the second section is from rank No. 79 to No.530, nearly another 33% of total cited times, the last section is No. 531 to No. 1731. So the numbers of these web sites with almost equal cited frequency is 78:452:1201, close to 1:4:42. Thus by application of this law in the web sites citation analysis, we can take the first section (78 web sites) as a core section of these 1731 web sites.
(3). Distribution of the Whole Domain Name Web sites: Most of the web sites listed in the whole domain name table (Table 4) are already listed in the earlier tables. This is because most "other links" are directed to the first level domains of URLs. Only URLs with asterisks (*) in this table have more details.
In fact, most of the whole URLs list were already been identified in the "First Level Domain" (Table 3), as most of the whole URLs are also represented in the "first level domain". A few links found are to pages deeper into the site and give us information as to why a site was selected for a link. For example, many, though not all of, visitors to CDC want to look up the Morbidity and Mortality Weekly Report (MMWR) and many visitors to NIH want information on grant and fellowship programs, both pages thus often get direct links in addition to a more general link to the CDC or NIH sites.  Combining the results of the "first level of domains" analyses and the whole URLs analyses, we replaced some "first level of domain" with the whole URL expansion if it existed. From this analysis a guide to the most cited health sciences related web sites was determined. We hope this list might serve as a more complete listing of the core web sites on health care.
To further represent these health-related core web sites clearly, we classified these core web sites respectively by their main utility, original sites into 6 clusters (Table 5).

Conclusions
Among the URLs cited by the selected academic medical institutions, almost 90% of the Top Level Domains (TLDs) are from the United States. Less than 10% come from the United Kingdom, Switzerland, Canada, Germany, Australia and the Netherlands. The number of remaining TLDs is less than 2%.
The first level domains are distributed according to Bradford's Law. There is a nucleus that contains the 78 most highly cited health sciences related web sites. These core web sites represented a broad field of information needs.

Discussion
The identification of a core section of health-related web sites with bibliometrics method gives librarians and information scientists another approach to evaluate the web sites. While "core lists" of printed publications have their drawbacks, they are useful guides to help librarians and users to select publications. Similarly, lists of commonly linked-to WWW pages can provide suggestions as to important health-related sites and assist home-page compilers in selecting suitable and reliable links. It would be desirable to examine the home pages of all U.S. medical school libraries and to compare these results to those from the pages produced by medical school libraries in other English-speaking countries such as Canada, the United Kingdom and Australia.

Introduction
This article results from a diploma thesis in media sciences written at the Institut für Journalistik und Kommunikationsforschung, Hochschule für Musik und Theater Hannover, Germany.
In the last years, the number of Internet users has exploded. As the Internet becomes a greater factor in both the fields of media and of economy, new users get access, new research and consulting requirements and new markets arise, and all this in a breath-taking speed.
In the same way the number of Internet users has exploded, the costs in the public health sector have also increased during this time in Germany. This resulted in cost saving efforts by those responsible people in politics and administration. These economy measures have impacted in particular the established physicians. The German practice owners are faced with an unknown economic situation: competitive conditions resulting from empty cash-boxes and a competition for patients and budgets force the physicians to think and act like businessmen. Doubts arise concerning the age-old tradition of the advertising prohibition. Now advertising is recognized as an important necessity. In Germany, physicians advertising on the WWW has only recently become possible [1,2] and is still being discussed and debated by lawyers and judges as well as by the members of the medical organizations defending the age-old professional ethics prohibiting advertising [3,4,5].
Various reasons can be listed for these changes: First, there are juridical questions: the advertising prohibition breaks some rules of the German constitution [6,7,8,9,10,11,12] and the European convention on human rights [13]. Second, the increasing Europeanizing and globalization of economies and markets demands an adaptation of German rules to those of other countries in order to equalize competitive conditions among physicians [4]. Finally, the advertising prohibition obstructs competition between physicians in Germany, too, a fact which collides with the concept of a market economy [14,15].
This study also investigated the relationship of physicians to the information technology, their knowledge, use of computers and especially of the Internet. Prior research from other institutions showed that the use of computers and of the Internet among established physicians depends on the age of the physician. Compared to the "general" German Internet user (most of them are between 20 and 29 years old) [16,17], medical users are older with a median age of 45 years [18,19,20]. Another important consideration is the medical specialty of the physicians using the Internet: It has been shown that physicians from different specialties show a different affinity for the use of computers. This can be concluded from the number of practices delivering their quarterly cost overview electronically. Orthopedists, urologists, general practitioners, oto-laryngologists and gynecologists were highest, whereas radiologists, neurologists, psychotherapeutists, anaesthesists, and pathologists do not use computers for this purpose very often [21]. Male physicians use the Internet much more often than female doctors.
Additionally German physicians from large and partnership practices tend to be more frequent medical online users [19,22].
A number of hypotheses were been concluded from the theoretical background information:

Physicians with an "open-minded" attitude towards IT and
Internet attach a higher importance to the own website than those who are less "open-minded." 2. Physicians who are familiar with IT and Internet attach a greater importance to their own website than those who are less familiar with IT and Internet do. 3. Younger physicians and physicians from younger practices are more open-minded towards a breakdown of the advertising prohibition than older physicians and physicians from older practices. 4. For physicians from younger practices, the motive of "winning new patients" is more important than for physicians from older practices. 5. Physicians from partnership practices and group practices attach a higher importance to advertising on the WWW than physicians from small and single practices. 6. Male physicians are much more likely than female physicians to have their own practice websites. 7. Physicians participate in professional medical politics and organizations are more "open-minded" towards a breakdown of the advertising prohibition than those who participate less. This introduction page indicated hyperlinks to a support page for those who had never filled out a WWW questionnaire before (http://unics.rrzn.uni-hannover.de/schuh/hilfe4.html) as well as to a "more information" page informing the interested reader about framework, goals and hypotheses of the study (http://unics. rrzn.uni-hannover.de/schuh/claudia.html). All pages contain a mail-to link that enabled the user to send an email to the author. After filling out the questionnaire and sending it to the author, a "thank you" page appears.

Materials and Methods
In order to guarantee anonymity to the respondents, the completed questionnaires were sent back to the author via a U.S. remailer provided by Perseus to all users of their software.

Results
194 physicians participated, and 159 valid questionnaires were included in the analysis and furnished the following results, represented by the corresponding tables: To a large extent, the age (Table 2) and the sex (Table 1) distribution in the examined group corresponded to the expectations from the general information on medical Internet users.
The distribution of the specialties in the examined group (Table  3) did not surprise us after the figures presented in the theoretical section of this paper.
The high percentage of physicians engaged in professional organizations (Table 14) was remarkable in the examined group.
More than 20 percent of the respondents held a position in a professional organization. The median time in practice of a little more than ten years (Table 6) with a substantially smaller modus of five years permits the assumption that the number of physicians who present their practice on the WWW will further grow in the future.
Due to the economic conditions for the operation of a medical practice, as expected especially young practices count on this new method of advertising. This can also be concluded from the fact that many of the websites of the respondents had only been online for less than one year (Table 12).
However, winning new patients is a less important motive for running a website than retaining existing patients by offering a new or better service to them.

Discussion
The hypotheses could only partly be confirmed by the results of the survey: Physicians of different specialties deal with their own website differently. The specialty of a physician could be considered an indicator to how and how intensively he uses the advertising method of a website. It can be assumed that apart from the ITand Internet open-mindedness, there are other decisive factors that were not examined in this study.
The Internet familiarity of the physicians (Table 10) is responsible for the importance they attach to advertisement on the web and particularly to the own website. This hypothesis could be confirmed and suggests at the same time that the WWW will gain in importance as an advertising medium for established physicians. More than a third of the respondents had used the Internet for less than two years (Table  9); with the explosive growth of the Internet the number of new physician users will increase and with the growth, develop new websites.
Surprisingly, the attitude towards the advertising prohibition results apparently less from economic considerations than from age-conditioned opinions. The key factor is the age of the physician, not the age of the practice, although naturally young physicians tend to be associated with younger practices. Not surprisingly, the tendency to try winning new patients by means of advertisement and a pronounced economic consciousness are stronger among owners of younger practices.
The size of a medical practice (Table 5) did not influence the attitude of the physicians towards their own website. However, the type of practice in which a physician works played a crucial role in this context (Table 4).

Conclusion
In conclusion, it appears that the importance of the Internet for the recruitment of new patients is still small, but it will continue to expand in the future. This is indicated the results presented in this study, by the general development of the Internet, by the European direction of markets and juridical systems, and by the examples from other countries. The few website operators among the established physicians are doing the pioneering work in Cyberspace. It is hoped for them -the respondents of this survey -that their pioneer spirit will be rewarded.

Introduction
The most important function of physicians is to help their patients make decisions among competing options of therapeutic interventions." [R.F. Brubaker [1]] As a librarian providing access to information for both patients and the physicians and other health care professionals who work with them, I share with many in the health care and health information fields a deep concern regarding the quality of information on the Internet. More to the point, I am concerned with the quality of information identified and used by patients, their physicians and other health care providers. Information on the Internet exists primarily, without regard to the traditional boundaries of time and place, for which there is limited access, and also of language and information-seeking behaviors. A search on the Web for a medical condition retrieves information from around the world. So while the marketing director of my institution is primarily interested in information that will attract and retain patients in our own clinics, in a sense we also serve patients and the medical staff of our sister institutions around the world.
Ten years ago gophers and CD-ROMs dramatically changed the way we thought about access to information. Today the World Wide Web looks and acts nothing like any medium we have known before. The coincidence of several social, economic and technological changes has led to a unique point in information history, in which there co-exist a greatly increased demand for information and a greatly increased interest on the part of many diverse institutions to fulfill what they perceive to be that need. The resulting increase in medical information, and specifically that available readily and electronically on the Web, is driven by two groups of factors, which can be termed "pull," for those which create a demand for information, and "push," in which providers of information are actively seeking to find users.

Pull Factors
Pull factors reflect changes in American society which have created a growing, seemingly limitless demand for more access to medical information. The first factor is demographic: an increase in the population of the country, and a shift in age distribution of that population. Overall the population in the United States is predicted to continue to increase, from 151 million in 1950 to 249 million in 1990 with a projected 394 million in 2050 [2,3]. This is due both to birth rates, which continue to more than replace losses by death in our population, and by immigration, particularly from countries in which birth rates are traditionally high-a trend which continues even after arrival in the U.S. Because of political changes, equivalent statistics are not available for all of Europe, but in Germany figures show a different situation: 68.4 million in 1950, 80 million in 1990, with a projected decline to 57.4 million in 2050 [4]. However, in both countries the percentage of those past working age is increasing at a rate higher than that of the overall population, and lifespans are continuing to increase as well. Those 65 and over comprised 8.1% of the U.S. population in 1950, 12.5% in 1990, and are projected to be 18% (70 million) in 2050 [5]. For Germany, the comparable percentages are 14.9%, 21% at present and 37% by 2050 [6,7,8]. Edward Schneider, Dean of the School of Gerontology at the University of Southern California, comments: "The issue that will most affect the quality of life for tomorrow's older population is their future health requirement." [9] The increase in number and proportion of this older population, which is the heaviest user of the medical system, creates two subsidiary pull factors. The first is an increased need for information on which to base medical decisions, and the second is increased discretionary time on the part of this population-although not on the part of health care providers-in which to look for this information.
A second pull factor is a continuing increase in general educational levels attained and literacy rates in the American population. More students graduate from secondary school, the basic level of education funded by all state governments, each year [10]. Consumer book sales are rising at 5% each year [11], with the official literacy rate being close to 98% [12]. One might argue with all of these figures that the general level represented by each of these measures is not as high as could be desired, but compared to any time in our past history, the educational level can be seen only as higher, and on the rise. This creates an increase in the general ability of the population to read and act on information, an increased confidence in doing so, and increased sophistication in evaluating the information and using it to make decisions about medical care.
Increased comfort in dealing with new technologies, particularly those that are computer-based, is a third factor. Half of all American homes now have access to the Internet [13], and the elderly are increasingly willing to try high-tech versions of traditional behaviors [14]. In a recent survey, 67% declared their willingness to "try something new" in computer use [15]. Web use is tripling annually [16]. Nearly all public libraries in the United States now provide both access to and instruction in the use of the Web; it is no longer true that one must invest significant time and financial resources in acquiring and mastering hardware and software. In addition to access in home and libraries, workplace access is growing. One in three American workplace computers had Internet access in 1998 [17]; 30 percent of Americans who categorize at least half of their Internet use as "personal" report that this activity takes place at work [16].
The last thirty years have witnessed a dramatic change in consumerism in the U.S. The publication in 1973 of a slim book, Our Bodies, Our Selves, by a group of feminists in Boston, the Boston Women's Health Book Collective, was one of the first signs that consumers were no longer willing to let physicians dictate medical care. This book, which has just appeared in a new edition for 1999, advocates that women make informed decisions about their own health care, and over the past 25 years has made a great contribution to the information behind those decisions. Also in the early 70's, Americans learned that their government had been involved in an appalling experiment during the 1940s: a group of Black men had been infected with syphilis without their knowledge, in order to study the effects of the disease in a controlled setting. This led to a sense of distrust, particularly by our large minority population groups, of the government as a source of reliable medical advice and help in decision-making, and an element of skepticism being introduced to the public's respect for the medical professions. The film Lorenzo's Oil in 1993 chronicled one family's quest for a treatment for their son's life-threatening illness, and is viewed by some as a model of consumer activism focused on a specific condition. The hospice movement, patterned on a model of end-of-life care that originated in England, is another indication that patients and their families demand to be involved in both medical decision making and actual care. Yet another indication is the increased interest in non-traditional medical care; in the last few years the National Institutes of Health established the Office of Alternative Medicine (now the National Center for Complementary and Alternative Medicine) to fund and promote research in alternative and complementary medicine. Some within the medical establishment view this as a waste of precious resources or worse, but many such therapies are welcomed by patients. The number of Medline searches performed by directly accessing the database at the National Library of Medicine increased from 7 million in 1996 to 120 million in 1997, when free public access was opened; the new searches are attributed primarily to non-physicians [18]. Finally, in late March of 1999 a new site, www.health-mart.net, was announced; this site, under a grant from two agencies of the U.S. government, will attempt to determine the feasibility of providing pricing and outcomes information on 400 diagnoses for essentially all U.S. hospitals to consumers. All of these are indicators of another pull factor: an increase in health care consumerism.
Several dramatic changes within the health care system, perhaps amounting to a revolution, also create a demand for information. In the 1960s, at a time when many were without medical insurance, the national government established two programs to assure access to medical care: Medicare, which provides coverage for services by physicians and hospital stays, and Medicaid, which insures the poor for these same services. Shortly afterwards, health maintenance organizations, which had started in California in the 1940s in very limited numbers, began to seem a viable alternative to fee-for-service care coupled with traditional medical insurance; in HMOs, groups of physicians provide all medical care for a set monthly fee. These organizations were attractive to both employers, who pay most of our medical insurance costs, and patients, who under traditional insurance policies had been responsible for whatever insurance did not cover, because the costs of care were now fixed annually, and known in advance. Now, however, there are concerns that too many of health care providers are primarily concerned with controlling costs rather than caring for patients-that the incentive is to NOT provide medical care-and that patients are not free to choose the best medical care they can identify.
Led by Medicare, under which the government establishes reimbursement rates for every procedure and reimburses physicians and hospitals by diagnosis rather than according to the care an individual patient needs, and by HMOs, which reward physicians for controlling costs, our medical system has dramatically changed. This has affected the care received by those requiring hospitalization. To compare countries in 1998 in Germany the average length of hospitalization for all causes was 11 days [19], whereas the University of Michigan hospitals, which as a tertiary care facility, care for the sickest patients, have reduced the average length of stay to fewer than 6 days in 1998 [unpublished data provided by the Budget Office, University of Michigan Health Care System]. Cost-cutting measures have provoked strong reactions among patients and legislators. The federal government recently passed legislation requiring that women who give birth must be allowed to remain in the hospital for at least 24 hours after delivery. In addition, consumers have successfully fought changes that would have made mastectomy an outpatient procedure. In the United States cataract extraction has been performed only as an outpatient procedure for nearly 15 years, except in rare cases in which the patient is so ill as to require hospitalization for another reason. By contrast, many German hospitals still keep patients in the hospital 3-5 days.
The positive aspect of this change has been a shift in emphasis from treating illness to maintaining and improving health; patients are increasingly viewed as partners with primary responsibility for maintaining their own health. Patient education programs, particularly those emphasizing prevention of illness, consume a new and increasing percentage of the expenditures of all medical care providers. Negatives contributing to the same "pull" on medical information include the fact that fewer nurses are caring for more patients who are sicker for briefer periods of time in the hospital. Figures as high as one nurse for 20 patients are not uncommon [20]. The traditional educational function of nurses, explaining details of care and the medical reasons for that care, is rare today. The cumulative effect of all of these changes is an increased need for information on maintaining and improving health, and on recovering from illness or coping with chronic or terminal disease. Health care providers as individuals are less likely to be the primary source of this information.
Changes in technology are also part of the "pull", enabling consumers of medical information to demand and use the information. Compared to 20 years ago, when personal computers were in their infancy, today's machines are fast and easy to use, requiring much less technical skill on the part of the user. They are cheaper; recently there have been advertisements for "e-tower" machines for less than $500. Even cheaper: "Free PC", a recent development in the States; this is an unsubtle marketing effort, in which free computer, software, and Internet access are provided in exchange for a heavy advertising content every time one accesses the Web. One can also use the Web without actually needing a computer at all: WebTV, now available for less than $100, uses simple equipment and the user's own television monitor and pre-existing telephone lines to access email and the Web. New adaptive technologies are announced almost daily; these enable those with limited visual and manipulative abilities to access the Web with increasing ease. All of the above create a demand, or pull, for information.

Push Factors
By contrast, the "push" factors are those initiated by information providers, specifically those on the Web, to meet this demand or create demand where none existed before. Who are they, and what do they want? 1. Governmental Agencies (which have historically had a public-information mission.) The National Library of Medicine and Medline, as well as many other databases, both health and science or technology-related, are prime examples familiar to Americans. Increasingly, websites from the federal government are being used to carry out the government's traditional role of making information freely available. This is simply an extension of a long-established practice in which the public funds its own access to information; the Web makes it quicker and easier in many ways, although the corresponding decrease in printed documents available in what are known as "depository libraries," and available at no charge or nearly so, is of concern to many. The government also provides most funding for clinical and biomedical research in the United States, and is thus also interested in attracting subjects for clinical trials, as well as making the results of research available to patients, medical personnel and other researchers. 2. Medical Health Care Providers who are trying to attract and retain consumers by providing information. This category may include university medical centers, large independent hospitals, and even solo or group medical practices. Additional goals for the investment of time and money in creating and maintaining sophisticated websites include enhancing the image and reputation of the institution, providing information for referring physicians, recruiting research subjects and patients, and attracting students and research associates for the biomedical teaching and research activities of the institution. In the increasingly competitive world of managed care, where the continuing existence of a hospital may depend on its ability to meet the needs of a single large employer such as Ford or Kellogg, an effective Web presence is not a trivial matter. 3. Marketers of Medical Care or Products. This category includes a range of providers, but they are distinguished from the above category by their approach, which involves advertising a highly competitive or perhaps not-yet-accepted therapy, or aggressive marketing of a product. One example of this within the eye care field is LASIK, a laser procedure that reduces or eliminates the need for spectacles or contact lenses. It was only recently approved in the U.S., and is not covered by most medical insurance, so is not subject to price controls. When this procedure was approved in Canada, before the United States, Canadian laser centers marketed this new procedure to the U.S. audience. Another example is ozone therapy for the blinding eye diseases known as retinitis pigmentosa. This very controversial procedure, which costs $15,000 and must be repeated at intervals, is offered primarily in Cuba; the scientific basis for the procedure is unknown. The international nature of the Web allows marketing of these procedures to patients who may be desperate, or merely wealthy. This category may also include commercial operations; perhaps those marketing night vision goggles or other low vision aids, using direct sales approaches aimed primarily at patients as consumers. Commercial information products can also be included here. 4. Libraries. Sometimes discounted simply because they are such a fixture in the U.S. and Europe, libraries, continue to "push" information, in print and audio-visual formats, and increasingly through the Web. Librarians' traditional roles in acquiring, preserving, organizing and disseminating information are changing; their roles in teaching information retrieval and evaluation skills are being enhanced. Librarians may call themselves "cybrarians", but do not fear losing their jobs. 5. Organizations and Support Groups devoted to education and research on diseases and conditions. These vary in their sophistication and legitimacy, but have an obvious presence and appeal on the Web as consumers of health care look for sources of information they can understand and use, and others with whom they can share similar experiences. Individual resources may have the same apparent status as sources when a list of sites is retrieved by one of the search engines.

Quality vs. Quantity
There are many analogies to dramatize the difficulties faced by health information consumers in retrieving reliable, understandable information in reasonable quantities. One is that using the Internet to look for information is similar to using a firehose to take a drink of water: the virtual flood of information, unsorted, unedited, is of unknown validity and utility. Another is a quote from Molly Mettler, senior vice president of Healthwise, Inc., a non-profit consumer organization, that searching on the Web is "...like hunting for wild mushrooms. If you know what you're doing and you've got a trusted guide, you can find a real treasure. But you run the chance of picking something toxic." [21].
Not only is the Web unwieldy, but consumers may not be in the best position to understand and evaluate the information they find. A study published in JAMA in February of 1999 described the results of using the Short Test of Functional Health Literacy in an adult population in three U.S. cities. Both Spanish and English speakers were tested in their native language. Approximately 34% of the English speakers, and 54% of the Spanish speakers, were rated "inadequate" or "marginal" users of health information; only 1.6% could correctly interpret instructions on preparing for an upper GI system procedure, while 11.5% were able to correctly interpret instructions on taking medications [22].
We assume that this "medical literacy" is not a problem with professional health care providers, but they face their own challenges. A 1986 study in the Annals of Internal Medicine claimed that if one read two articles per day in the biomedical literature, in one year the reader would be 55 centuries behind on that one year's production [23]. A recent Medline search for asthma-related articles retrieved over 47,000 citations, while an Infoseek search on the Web found over 154,000 sites. Sophisticated medical professionals are often unsophisticated consumers of medical information. The number of competing sources of information makes identifying, evaluating and using new sources of information an increasingly difficult task. One study showed a decline in the knowledge level of general practitioners between five and ten years after the completion of training [24]; this may be a reflection of the gradual loss of previously-learned knowledge, and advances in medical knowledge, coupled with lack of skills for continued learning.

Strategies
What strategies can be suggested to deal with the real needs of consumers, the "pull" factors described above; the "push" of the needs of producers of information on the Internet; and the associated problems of retrieving, understanding, evaluating and using that information? Several possibilities are suggested in other papers on this issue. Possibilities include: A recent study of incoming college students showed that students who cited more sources in their papers, and believed material on the Internet should be viewed with skepticism, had higher grade point averages when compared to their peers [25]. 4. Insisting on a clearer understanding of the difference between information technologies and knowledge management strategies. 5. Encouraging and rewarding adherence to Web codes of conduct, such as HON. Easy-to-navigate and reliable sites will be of increasing importance. HON reinforces the importance of principles that are easy to articulate but may nevertheless be overlooked in the preparation of sites. 6. Making a commitment to providing accurate, high-quality links from our own sites seems obvious but is not always acted upon. Individual personal referrals of consumers and patients to good sites are equally important, and are a close corollary to good links.