Unmet Needs of Primary Care Patients in Using the Internet for Health-related Activities (e19)

Background: Psychoeducational programs are increasingly being delivered over the Internet. We created an Internet-based, cognitive therapy, self-help program to be used as a stand-alone intervention for mild-to-moderate depression, or as an adjunct to traditional services for more severe depression. Objective: To evaluate the effectiveness of a web-based intervention program to reduce depression in a randomized, controlled trial Methods: In a private, nonprofit health maintenance organization, we mailed recruitment brochures to two populations: depressed adults (n = 6994) who received traditional medical services for depression, and an age/gender matched sample of nondepressed adults (n = 6996). Participants consenting to the study were randomized to either the experimental Web site (n = 144), or a no-access control group (n = 155). Participants in both groups were free to obtain nonexperimental, usual care services for their depression. All participants completed an on-line version of the Center for Epidemiological Studies Depression Scale (CES-D) at enrollment and at 4-, 8-, 16and 32-weeks after enrollment. Mean intake scores were in the severely depressed range. 74% of participants completed at least one follow-up assessment. Unfortunately, most intervention participants accessed the Internet site infrequently. Results: We failed to find an effect for the Internet program across the entire sample. However, post-hoc, exploratory analyses revealed a modest effect among persons reporting low levels of depression at intake. Conclusions: The negative results might have resulted from infrequent patient use of the Internet site, or a more seriously depressed sample than the intervention was intended to help. Future studies should focus on recruiting persons with mild to moderate levels of depression, and on increasing participant use of the Internet site. (J Med Internet Res 2002;4(3):e14)   doi:10.2196/jmir.4.3.e14

Implementation and Integration of Regional Health Care Data Networks in the Hellenic National Health Service (e20) Petros  Critical, evidence-based evaluation of the effectiveness of information and communication technology should be one important component of eHealth [1]. In clinical medicine, randomized controlled trials (RCTs) are considered the "gold-standard" to assess the effectiveness of a treatment: next to systematic reviews, a well-conducted randomized trial can provide the strongest evidence for (or against) the effectiveness of an intervention. In a RCT, eligible patients are typically randomly assigned to receive either the new intervention or a control, for example, standard care or a placebo. Patients are then followed over a period of time and outcomes in both arms are measured and compared. Can we evaluate the effectiveness of a single health website in a similar way?
In this issue of the Journal, researchers at the Kaiser Permanente Center for Health Research present results of a study to develop and evaluate a web-based psychoeducational program designed to reduce depression [2]. This study represents one of the first randomized trials of a web-based mental health intervention ever conducted. A recent systematic review on comparative studies evaluating Internet-interventions stated a dearth of evidence in the field [3], and only one other recently published RCT known to us has evaluated a mental health intervention [4]. The results of the study published in this issue of the Journal are unexpectedly negative: the trial suggests that Internet-delivered mental health intervention may have no or limited treatment effects.
Two hundred and ninety-nine patients were randomly selected to receive either the online depression program in addition to usual care, or usual care alone. Previous studies evaluating traditional bibliotherapy (the treatment of depression with educational material) suggest that bibliotherapy is an effective intervention. Despite the fact that the Internet offers various possibilities to improve traditional educational and bibliotherapy material, the trial disappointingly did not result in any measurable effect in the overall sample. Although in an exploratory data analysis researchers found a small benefit for those participants who entered the study with lower levels of depression, these results need to be confirmed by subsequent studies. "We are at the very threshold of this burgeoning field, and we know very little about the circumstances and processes that will optimize the delivery and acceptance of these interventions", write Greg Clarke and colleagues. "There is no accumulated clinical lore about how to best provide Internet services; we are blazing this trail as we progress." The study of Clarke and colleagues illustrates the possibilities and challenges of evaluating the impact of a health website on health outcomes in a web-based RCT.
Among the appealing factors is the fact that hundreds or thousands of people per day may frequent a health website, which gives plenty of sample size. Patients from all over the world can be recruited, enhancing the external validity (generalizability) of the results. No costly face-to-face interactions such as clinical examinations may be necessary if psychological outcomes are measured through self-administered electronic questionnaires. The administration of the intervention or the control, respectively, data collection and outcome measurement can be completely automatized. Not only does this make web-based RCTs very cheap, it minimizes biases, at least those introduced by human observers. Often an investigator's earliest opportunity to interact with the research subjects is when he opens the database to analyze the data.
On the other hand, there are significant challenges inherent in web-based randomized trials that don't exist in clinical trials studying drugs. Some of them are similar to challenges in educational interventions or surgical trials: For example, the trial cannot be conducted in a double-blind fashion as the patient always knows what intervention he receives.
But there are perhaps even more serious challenges. First, there is a considerable risk that the control group becomes "contaminated" by accessing a similar intervention from somewhere else on the web. This is particularly true if the intervention is "giving information" or an educational program that -in a similar way -can be easily found somewhere else on the web. Interventions such as smoking cessation programs can be easily found and used elsewhere on the web, threatening the ability to detect differences between the groups. Institutional review boards may require investigators to describe their intervention in detail before patients consent to participate. Participants who are randomized into the control group may be disappointed that they are not getting the intervention and may search the web for a comparable intervention, using it without the knowledge of the investigator. To minimize this bias the only option is to reduce the amount of information about the intervention given to participants, which may be ethically problematic. One should also ask participants in the control group whether they used similar interventions elsewhere on the web, or even monitor their use of other websites directly by using client-side proxy software.
Apart from the problem of ensuring that the control group actually stays a control group, we are facing the opposite problem in the intervention group: How do we assure that the intervention group is actually using the intervention? The relative ease of enrolling participants to a web-based trial seems to come at the cost of a high probability to lose them again -as many as about half of the patients [3; 4] may be lost to follow-up. In an intention-to-treat analysis such high drop-out rates greatly affect the ability to detect small differences between the groups. Even those who fill in the follow-up questionnaire (i.e. not dropped-out) may not actually have used the intervention, emphasizing the importance of asking about the frequency of use (or measuring it directly through log-files) and conducting a dose-response analysis. One may also have to think about employing novel techniques to reduce drop-outs. For example, preceding the actual trial one may employ a run-in period, where users are required to return to the website several times prior to enrollment and randomization. Only returning users will eventually be randomized into the intervention or control group.
The bias introduced by these issues is typically a "bias towards the null", i.e. through these methodological difficulties a trial may fail to show a small effect of an intervention, through the "noise" introduced. Despite these issues and despite alternative possibilities to evaluate a website (surveys, log-file analysis, before-after trials, and interrupted time series) the RCT remains the gold standard and we are eagerly looking forward to see more of these trials.

Introduction
Many persons with clinical depression or subdiagnostic symptoms of depression obtain traditional medication and/or psychotherapy services [1,2]. However,many of them, as many as 50% or more, do not obtain such traditional treatments [3]. The reasons for this seeming disconnect between disorder and treatment are only partially understood, but include barriers to care such as inadequate insurance coverage and limits on treatment [4].
Contributing to this pattern of undertreatment, some patients may elect not to seek professional help, at least initially. Persons who do not seek traditional treatments for depression may use less-intensive, self-help materials, such as books, pamphlets, videotapes, or computer programs, collectively called bibliotherapy [5]. For persons with subdiagnostic symptoms of

Beyond Bibliotherapy
Depression self-help books and videotapes, while valuable, have several drawbacks. Health providers may find it difficult to accurately monitor a given patient's progress and use of bibliotherapy materials. In addition, these materials are static; that is, they are not tailored to an individual patient's particular problems. Books or pamphlets also presume a minimum reading level for use. Another limitation of printed material is the inability to employ recorded sound or video vignettes. These multimedia features may make self-help more accessible to members with lower reading skills, may engage all patients to a greater degree, and may prove more effective at conveying self-help skills [5].
We have recently developed an Internet-based, depression self-help program that addresses many of these drawbacks [11]. We call this site unattended, in that live personnel do not staff it. Instead, it provides self-guided, interactive tutorials to help users acquire antidepression skills such as cognitive restructuring [12]. In this respect, it is more like an interactive booklet than an analog of live therapy. This unattended aspect of the program distinguishes it from Internet sites where mental health professionals conduct person-to-person psychotherapy or counseling via e-mail exchanges with patients.
To the best of our knowledge, this program was unique and was not duplicated by any other depression resource on the Internet at the time of this study. While many other mental health and depression Internet sites provided information (such as the Depression Awareness, Recognition and Treatment site maintained by the National Institute of Mental Health, http:/ /www.nimh.nih.gov/publicat/depressionmenu.cfm),all of those that we found were limited to explanations of causes and treatments. None of these informational sites offered training in behavior change skills. Research from related behavioral fields such as drug prevention suggest that information-only interventions are inadequate to bring about positive change, and that skills-training interventions are needed to achieve detectable and enduring improvement [13,14]. Our experimental Internet program offered both information and direct training in self-help, cognitive restructuring skills.
In contrast to the informational Internet sites, Marks [15] summarizes computerized programs that provide treatments for depression and other mental health problems. Several of these programs have interactive and multimedia features in common with this Internet program. However, none of these programs were available on the Internet at the time of this study, which means they were not available to a whole community in the same manner as an Internet site.
Another related but distinct category of studies has reported successful results obtained with Internet-based, mental health interventions targeting problems such as eating disorder risk factors [16,17], weight loss [18], and panic disorder [19]. These studies have in common the use of the Internet to deliver at least some components of the experimental intervention. However, at least a portion of each of these interventions includes person-based counseling or psychotherapy offered either face-to-face or through the Internet (similar in many ways to therapy delivered via telephone). In contrast, the current study presents the results of a randomized trial of an Internet depression intervention that relies exclusively on a psychoeducational tutorial that does not employ any monitoring or live interaction between participants and clinical staff. To the best of our knowledge, at the time of this study there were no other unattended but interactive interventions for depression on the Web. Since that time, there has been a report of a similar self-guided depression skills training program on the Internet [20], but to date it has not been evaluated in a randomized, controlled trial.
This study is also distinct from other randomized trials of Internet health programs because of its effectiveness design [21,22]. That is, we evaluated the Internet program in the context of a variety of other usual care health services, which were provided to the participants outside the research protocol. The comparison condition was usual care in a health maintenance organization (HMO), rather than an enforced no-treatment comparison condition.
We hypothesized that, compared to persons randomized to a no-access control group, persons who were randomized to have access to the Internet depression site would demonstrate greater reduction in self-reported depression symptoms.

Subjects and Recruitment
The sampling frame was approximately 430000 members enrolled in the Kaiser Permanente Northwest (KPNW) health maintenance organization, in northwest Oregon and southwest Washington. The study research center is located within the HMO, and is scientifically autonomous, self-governed, and committed to conducting public-domain research. The Human Subjects Committee for the HMO approved all study procedures.
The study recruitment pool was generated in several cohorts over a 7-month period in 1999. We employed the HMO's electronic medical record (EMR) systems to identify 2 groups of potential participants (see Figure 1). The first group was adults (n = 6994 over 7 cohorts) who received traditional HMO medical services (medication, psychotherapy) in the previous 30 days in association with a recorded diagnosis of depression in the EMR. We called this first group depressed cases, although we do not have any independent research confirmation of their clinical mood diagnosis such as with a structured diagnostic interview. The second group was nondepressed adults (n = 6996 over 7 cohorts) who did not receive any depression-related HMO services and did not have an EMR diagnosis of depression, but who were age-and gender-matched to the first group. All potential participants were mailed an identical study recruitment brochure, which contained the following statements: We are inviting you to participate in a new Internet-based program for people who may be feeling sad or depressed. We are evaluating whether this new service is helpful. We are mailing this brochure to thousands of HMO members. We do not know which members will benefit from this new Internet service, so we are mailing this letter to many people, only some of whom may be feeling depressed.
The brochure explained more details of the study, and provided the research Internet address. No further contacts were made; it was left to the initiative of invited individuals to take the first step and visit the study Internet site.
At the study home page (the starting point for the study and the intervention), invited members were asked to enter their HMO health plan number, birth day, and birth month. If these did not match the stored data for invited members, the individual was not permitted to proceed. Of a total of 13990 brochures mailed, a total of 526 participants entered confirmed health plan numbers at the study home page. This represented an initial engagement rate of 3.8% of all invited members. However, because not all invited members had Internet access, it is useful to estimate the engagement rate among those with the ability to participate. An internal survey conducted by the HMO health plan at the end of 1999 found that approximately 62.6% (n = 745/1190) of HMO members in this region had Internet access either at home or work or both. Therefore, these 526 participants may also be viewed as an initial engagement rate of 6.0% out of 8758 invited HMO members (62.6% of 13990) estimated to have Internet access.
The 526 participants entering the study site were asked to read the on-line consent form, and to indicate their consent by selecting an I agree to take part button. Participants were next asked to complete the on-line assessment battery (described below). Following this, 299 members (56.8%) agreed to be randomized to conditions. Randomization was accomplished immediately by a random-assignment algorithm encoded into the Web site programming: 144 participants were granted access to the experimental Internet site (the intervention condition), and 155 were not granted access (the usual care control condition). Seventy-six members of the nondepressed recruitment group (n = 6996) were enrolled in the study (1.1% of those invited) and 223 of the depressed recruitment group (n = 6994) were enrolled in the study (3.2% of those invited). Compared to HMO members who were mailed a brochure but who were not randomized, those in the randomized sample were more likely to be female (76% of the randomized sample vs 70% of the nonrandomized sample, P= .03) but did not differ in age.
Following the on-line consent and data collection (see below), subjects randomized to the Internet depression program were immediately linked to the home page for the intervention program. Subjects randomized to the control condition were immediately linked to the Kaiser Permanente Online home page, a Web site maintained by the health plan for HMO members only [23]. At the Kaiser Permanente Online site, participants could obtain noninteractive information about health concerns, including depression. They could also send a question to an HMO advice nurse or pharmacist, or request an appointment at local medical centers. In both conditions, subjects were free to continue or initiate any other traditional health care services for depression or other psychiatric problems, and/or access Kaiser Permanente Online or any other Internet health resource.

Assessment Battery
While enrolling in the study, participants were asked to provide information on their age, gender, marital status, ethnicity, and educational attainment. Table 1 presents these characteristics by experimental condition. Participants completed an on-line version of the Center for Epidemiological Studies Depression Scale (CES-D) [24] a self-report measure of the frequency of 20 depressive symptoms over the past week using a 5-point Likert scale.
Reviews of computerized depression assessment methods support their reliability, validity, and equivalence to paper versions, with patients generally more truthful in their answers and often preferring computerized methods for assessing sensitive areas such as suicide and depression [25]. For example, computerized and paper-and-pencil versions of the CES-D correlate at a very high level [26].
Subjects in both conditions were sent e-mail reminders to return to the study Web site at 4-, 8-, 16-, and 32-weeks post-randomization, to complete on-line CES-D follow-up questionnaires. If subjects failed to respond to two e-mail reminders for any single follow-up point, study staff attempted to reach them by telephone. Participants were sent a $5 e-mail gift certificate redeemable with an Internet merchant (Amazon.com) for each completed assessment, for a possible total of $25.

Intervention
The Internet intervention was a self-paced, skills training program focusing on the acquisition and use of cognitive restructuring techniques [12,27]. A non-research version of this program, identical to the research program but without the consent and assessment, was made available for public viewing after the trial [11]. Figure 2 shows a screenshot of the program; further screenshots are available in the multimedia appendix at the end of this paper.
Much of the Internet site content was adapted from group CBT psychotherapy manuals [28,29] that have been successfully employed in several face-to-face randomized trials [30][31][32][33][34]. Theskills training program was organized much like a brief booklet, with 7 chapters. Each chapter presented a new skill or component technique, and offered interactive examples and practice opportunities.
A representative section of the program is the Thought Helper, made available to participants after they had been taught the basics of identifying negative thoughts and generating positive counter-thoughts. Participants typed their negative or irrational thought into a text box, then clicked on a search button. The Web-server computer searched a predefined list of 200 negative thoughts for ones that best matched the negative thought submitted by the participant, and returned a screen with the most likely matches. Participants selected the returned negative thought that they felt was closest to their original. The program then returned a list of 2 to 5 possible positive counter-thoughts relevant to that belief. The participant was then encouraged to create a personalized positive counter-thought, using relevant portions of the provided examples as they saw fit, and enter it into the Web site for storage.
Other interactive aspects of the Web site included an automatically-scored CES-D questionnaire embedded into the first and last sessions (as well as the follow-up assessments); feedback on improvement across time; and cartoons that taught the basics of cognitive restructuring, with participants clicking on cartoon character dialog balloons to identify, for example, irrational beliefs and positive counter-thoughts. We elected not to incorporate live video or audio in this initial version of the program because we wanted to make this program available to all participants including those with slow connections to the Internet. However, we are preparing newer versions of this program (to be tested in subsequent randomized trials) which have live video vignettes and exercises, to take advantage of the increasing availability of high-speed, high-bandwidth Internet connections, such as cable modems.
Participants randomized to the intervention arm of the study were able to sign-in and use the Web site whenever they wanted, for the entire duration of the study. The modal number of sessions was 1 (41.0% of users), the median was 2 sessions, the mean was 2.6 (SD = 3.5), and the range was 1 to 20 sessions. Overall, this suggested a low level of use of the depression skills training site.

Health Care Utilization
Computerized data systems provided encounter data for HMO inpatient and outpatient services, prescriptions, lab tests, emergency room visits, etc. For this trial, participant use of non-HMO health care services was not assessed.

Analytical Plan
All analyses were intent-to-treat. We examined continuous CES-D scores using random effect regression analyses, modeling an unstructured covariance matrix, with slope and intercept as random effects [35,36]. We estimated both linear and quadratic effects for our data, as these models best fit the data. The linear trend indicates the direction and rate of change, while the quadratic trend indicates whether the rate of change increased or decreased at some point during the observation period. However, only the significance for linear trends is reported, as this effect is relevant for hypothesis testing.

Sample Size
Determining the needed sample for this trial was difficult, given the lack of prior Internet randomized trials in mental health. However, extrapolating from reviews of the relevant mental health bibliotherapy literature [5], we identified an upper limit effect size (ES) of .55 for the current study. However, all participants in this trial were also free to obtain traditional usual care depression treatments, which might result in fewer differences between study conditions. Therefore, we selected a more conservative ES of .30 to .35 as more appropriate for this trial. For a sample calculation, we assumed a 2-sided alpha = 0.05, power = .80, and a standard deviation of 8.5 for the CES-D (the main dependent variable), derived from the depressed adult sample originally used to validate this scale [37]. This resulted in a needed sample of between 256 and 350 for the specified ES range (. 30-.35). We selected the midpoint sample size of 300 as our goal.

Comparability of Conditions
The experimental and control groups did not differ with respect to recruitment group, gender, age, or baseline CES-D score (Table 1).

Depression
We found no differences between the control and experimental conditions on self-reported depression (CES-D) over the study period, indicating a lack of a treatment effect. Table 2 presents the depression scale scores across assessment points for the total sample (uppermost rows). Table 2 also presents depression scale results for several subgroups. These analyses were exploratory, and thus were not powered to detect effects within the relatively small subsamples. Subgroups included (a) subjects recruited from among depression cases (n = 223); (b) subjects recruited from among nondepressed controls (n = 76); (c) subjects with high baseline depression scores (CES-D < 20) (n = 236); (d) subjects with low baseline depression scores (CES-D < 20) (n = 63); (e) female subjects (n = 226); (f) male subjects (n = 73); (g) subjects age 45 or older (n = 144); and (h) subjects younger than age 45 (n = 155). There were no significant intervention effects across the entire study duration for any of these subgroups. However, among subjects who reported lower baseline depression scores (Figure 3), experimental participants were significantly less depressed than the relevant control subjects at the 16-

Dose-adjusted Effects
Dose analyses were conducted for the participants randomized to the Web site intervention. We predicted CES-D scores from the total number of sign-ins to the Web site (our measure of dose). The random effects regression analyses did not show a significant effect for dose (F = 0.95, P= .33 for sign-in-by-time effect).

Health Care Services
In the 12 months following randomization, we found no differences in the use of health care services or psychoactive medications, including mental health outpatient visits ( We did find a significant difference in venlafaxine dispensings (2.8% for treatment cases vs 8.5%; chi-square1 = 4.4, P< .05), but given the number of comparisons this is probably not meaningful. P value for the interaction term of sex x treatment group x time (test of whether the effect of treatment on CES-D score change differed by sex) P value for the interaction term of age group x treatment group x time (test of whether the effect of treatment on CES-D score change differed by age group)

Discussion
This intervention improves on currently-available traditional bibliotherapy in several respects such as multimedia capabilities, automated scoring of questionnaires, and remote accessibility. Despite these advantages, or perhaps even because of these differences, we were unable to detect a main intervention effect for the Internet program. We conducted a series of exploratory analyses among subgroups of participants, in an attempt to better understand why we were unable to find positive effects in the overall sample. However, we also failed to find positive intervention effects among most of the subgroups, although there was a small benefit for those participants who entered the study with lower levels of depression. Because of this possible subgroup benefit, we believe this intervention requires careful examination in further studies.
There are several possible reasons for this overall lack of effect. Because they are newly developed, the Internet intervention materials themselves may have content shortcomings. That is, curative elements that are present in the hard copy bibliotherapy books and pamphlets [6] may be missing or inappropriately delivered in this Internet program. Despite these specific issues, the medium itself should improve the accessibility of bibliotherapy for a subgroup of patients likely to benefit from guided self-care. We are examining the intervention materials and improving the content for 2 subsequent trials (see below).
A related issue is whether an unattended, psychoeducational approach to delivering behavioral or lifestyle interventions via the Internet is less potent than conducting person-directed counseling through the same media. The previously conducted studies of Internet-based interventions for eating disorders [16,17], weight loss [18], and panic disorder [19] employed counselors or therapists who composed e-mail or bulletin board (chat) responses to participant questions and problems. It may be that the general absence of benefit seen in this study is due to the lack of the caring, supportive therapeutic alliance that is an integral part of in vivo psychotherapy. However, this doesn't explain how traditional bibliotherapy materials such as books achieve significant effects [5], since these materials also lack the personal relationship that is characteristic of live therapy.
Another likely reason for the lack of intervention main effects is that the recruitment procedures attracted a more seriously depressed sample than intended, resulting in participants who may have been too depressed to have benefited from such a low intensity, self-help program. Some evidence supports this explanation. Radloff [37] suggests a "serious depression" cutoff score of 16 or greater on the CES-D, the depression scale employed in this study. She also notes that only 5% of adults in a representative community sample have CES-D scores 28 or greater. This suggests that our sample, with a mean baseline CES-D of approximately 31, would be among the most depressed persons in a normal population. Further evidence comes from the exploratory analyses for persons with low baseline CES-D scores; experimental participants were significantly less depressed than the relevant control subjects at the 16-and 32-week follow-up points. However, given the large number of exploratory analyses in various subgroups, these findings must be viewed as tentative and hypothesis generating. We are preparing a subsequent randomized trial to test this finding in a more systematic manner, by specifically recruiting a much larger sample of participants with low-grade, subdiagnostic depression.
Yet another explanation for the lack of main effects assumes that both the sample and the intervention content were appropriate but that participants did not return to use the Internet site frequently enough to obtain full benefit. This would be analogous to persons who attend only one or two psychotherapy sessions. However, the dose analyses failed to demonstrate that outcome was associated with frequency of use of the Internet site, even when baseline depression severity was controlled. Nonetheless, to address this issue we have just completed enrolling 259 participants in yet another randomized trial similar to this study, but employing several low-intensity methods (e-mail messages, live telephone calls, postcards) to remind subjects randomized to the Internet program to return to the site, complete more of the content, and potentially obtain more benefit.
This study was limited in several ways that may have contributed to the absence of effects. Most importantly, attrition at follow-up was quite high, with follow-up rates at any one point ranging from 53% to 66% (although three quarters of the participants completed at least one follow-up assessment). While the lost participants were similar to retained participants on almost all baseline characteristics (except for of slightly lower depression scores), this high attrition rate reduces our confidence in the observed results. Nonetheless, given that we relied primarily on e-mail assessment reminders and the assessment was conducted via an Internet-administered questionnaire, these retention/attrition rates may represent what can be expected of this new medium where the norm is surfing(rapidly jumping from one Internet site to another).
Another major limitation of this study was our reliance on a single, self-report measure of depression. In our previous depression randomized trials [30] we have often employed research psychiatric interviews to ascertain diagnoses. However, this methodology is not amenable to the Internet medium at present. We could have conducted research interviews by telephone or in-person, but did not because of cost and effort limitations in this preliminary study. In addition, we were somewhat concerned that the nonspecific, semi-therapeutic impact of several hours of research diagnostic interviewing over the life of the study would potentially swamp the small therapeutic benefit expected from the Internet intervention. This is not an altogether speculative concern, but has been considered as a possible reason for null findings in another, much larger randomized trial in the alcoholism field [38].
All these issues remind us that we are at the very threshold of this burgeoning field, and that we know very little about the circumstances and processes that will optimize the delivery and acceptance of these interventions. We are in an unusual situation relative to the development and testing of in vivo psychotherapies. By the time controlled research studies were mounted to test traditional, person-to-person psychotherapy, therapists had been delivering these services in non-research settings for decades and knew a great deal about the process and parameters of service delivery. In contrast, studies of the use of the Internet to deliver mental health interventions are being conducted before widespread, non-research service delivery has occurred via this medium. Therefore, there is no accumulated clinical lore about how to best provide Internet services; we are blazing this trail as we progress. This study, and the others planned and underway, will help address some of these issues.

Introduction
The increased use of the Internet by ordinary people is changing the way health care providers and the general population search for and retrieve medical information, which, in turn, modifies user-provider interaction and health care delivery. Studies have evaluated the use of the Internet in different medical conditions to assess its impact on patients' knowledge and well-being [1][2][3][4].
The role of patients in the management and control of type 1 diabetes mellitus, a chronic disease, is well established. Several studies have evaluated different Internet-based solutions for diabetes care [5][6][7][8]. However, we are not aware of any study assessing Internet accessibility and use among patients with type 1 diabetes mellitus. This data might be important for assessing the potential impact of Internet-based solutions for diabetes care in different settings. Therefore, we conducted this study to assess the accessibility and use of Internet resources in a population of patients with type 1 diabetes mellitus.

Methods
Between October 2000 and March 2001 all patients with type 1 diabetes mellitus who attended the Diabetes Clinic of the Hospital de Sabadell, Sabadell, Spain, were asked to answer a structured questionnaire about education level, Internet accessibility and frequency of use, and access to health-related Web sites. Ownership and use of mobile phones was also evaluated. Demographic and clinical data regarding the level of metabolic control and the associated morbidity complications were retrieved from clinical records. Type 1 diabetes mellitus was diagnosed according to criteria published elsewhere [9].

Statistical analysis
Discrete and continuous variables were compared using the Pearson chi-square test and the Student t test respectively. Logistic regression analysis was performed using the stepwise method to predict use of the Internet and access to health-related Web sites. Differences between variables were considered significant when P value was less than .05. All analyses were performed using SPSS 11.0 software.
Internet accessibility was defined as the possibility of access to the Internet either at home, school, work, or other places. Internet users were defined as those accessing the Internet at least once a month. Access to health-related Web sites was defined as ever having accessed a health-related Web site. Mean HbA 1c (glycosylated hemoglobin) level was defined as the mean of all HbA 1c levels obtained during the 12 months before the interview. Presence of severe hypoglycemia was defined as any episodes of hypoglycemia requiring external help during the 12 months before the interview. Intensified diabetes treatment was defined according to usual criteria [10].

Results
Of 302 patients with type 1 diabetes mellitus who attended the Diabetes Clinic at least once a year and who kept appointments for visits, 255 had an appointment during the study period. Of these, 11 failed to keep the appointment. Therefore, a total of 244 patients were interviewed. Patients not interviewed (n = 58) were older (41.5 [15.6]  As seen in Table 1, compared with Internet nonusers, Internet users were younger, were more frequently men, were of higher education level, had diabetes of shorter duration, and had a lower degree of complications. In a logistic regression analysis after introducing all significant variables included in Table 1, only educational level, age, and gender predicted Internet use. Internet users owned personal computers (87.6% vs 41.4%; P < . 001) and mobile phones (87.6% vs 41.3%; P < . 001) more frequently than Internet nonusers.
Among Internet users only 44 (49.4%) had ever accessed a health-related Web site. Table 2 shows the characteristics of Internet users who had ever accessed a health-related Web site and Internet users who had never accessed a health-related Web site. As shown, Internet users who had ever accessed a health-related Web site had a higher level of education, presented severe hypoglycemia more frequently, were more likely to have access to the Internet at home, and were more likely to own personal computers. Level of education, severe hypoglycemia, and Internet access at home remained significant after logistic regression analysis of significant variables. Severe hypoglycemia * Data are mean +/-SD, mean (SD), n/n, or n (%) † Secondary school refers to the obligatory education between 12 and 16 in Spain; afterwards people can opt for high school which allows for University education (Spanish: Bachillerato Superior [16][17][18] or for other options ‡ Grouping primary and secondary school

Discussion
To our knowledge, this study is the first to evaluate the use of new communication technologies in patients with type 1 diabetes mellitus in the real world. Although the purpose of the study was to evaluate its use in a nonselected population, patients evaluated were significantly different from those not interviewed. Clinical characteristics of the latter group suggest that they may be less prone to use new technologies. Therefore, the present study may have overestimated the use of these technologies.
Internet utilization and demographic characteristics of Internet users compare well with national data about Internet use [11] and are lower than results obtained from other European countries with the exception of France and Ireland [12]. Internet users were of higher education level, younger, and predominantly men.
Patients looking for health information were, as expected, of higher education level and they were more likely to have access to the Internet at home. Additionally, they presented serious hypoglycemia more frequently. However, the low number of patients with this condition casts doubt on the significance of this result. No differences were found in metabolic control between Internet users and nonusers or between Internet users who had ever accessed a health-related Web site and Internet users who had never accessed a health-related Web site, contrary to other studies where Internet use has been associated with a better health profile [3].
The rate of use of the Internet for health purposes is in accordance with other studies [2,3]. Internet users who access health information on the Web are around 50% of all Internet users [1][2][3][4], a figure which can be considered low for chronic diseases in which patient's self-management is highly encouraged. Reasons for not accessing health information on the Web were not addressed in our study. Results from other studies suggest that lack of training in information technology is felt to be one of the main determinants for not retrieving medical information from the Web [1], an obstacle which can be easily overcome [13,14]. Other issues which can affect the patient's willingness to use the Internet as a health tool are related to the anxiety and stress derived from having different sources of information [2], lack of time [15], and poor readability [16]. Alternatively, quality of information, one of the major concerns of health professionals [16,17], although important for those who actually search for health information on the Web [15], does not seem to worry those who do not seek for it [1]. Another aspect to be considered is the lack of a specifically-designed, professionally-moderated Web page, which is felt by patients to be a reassuring tool [18] and might increase the rate of health-related Internet use. However, in the best case, this specific product would have been used by 59% of our patients with type 1 diabetes mellitus. Based on the profile of younger, male Internet users with a shorter duration of diabetes (Table 1), this percentage would have been lower in patients with type 2 diabetes mellitus, because (a) patients with type 2 diabetes mellitus are older (because type 2 diabetes mellitus usually starts in people older than 40 years) and (b) it seems that age is one of the determinants of Internet use.
The rate of ownership and use of mobile phones in our study is high. Use of wireless technology in health care has been evaluated mainly as a telemedicine tool [19,20] and patient-oriented tools are still under development [21]. We are not aware of any study exploring the role of Short Messages Systems as reminders or as empowerment tools. There is a need to explore the role of present and future mobile-phone technologies in health care delivery. However, these technologies may not yet be powerful enough to support the tools needed for delivery of health care.
In summary, although the advent of the Internet will probably change the way in which health care is delivered, at present its impact, according to our study, might be partially jeopardized by the rate of access to and utilization of the Internet for health-related purposes. Further studies are warranted to evaluate the needs and worries of patients to better address patient-oriented Internet-based solutions for type 1 diabetes mellitus. Because of their high rate of ownership and use, mobile phones show promise as a tool in health care communication technologies.

Conflicts of Interest
None declared.

Introduction
The Internet continues to evolve as an increasingly-important source of health information for millions. Based on a national survey in March 2002, an estimated 73 million Americans have used the Internet for health information, with approximately 6 million Americans going online for health advice each day [1,2].
With an estimated 100000 health related Web sites, the Internet has changed the way that Americans access health information [3]. Patients use the Internet to investigate many health-related topics commonly encountered by primary care providers [4]. It is quite likely that 1 hour of Internet searching by an intelligent patient on a reputable Web site can give the patient information about his or her condition that the physician is not aware of [5].
Previous reports indicate that patients feel that information on the Internet is "better than" information from their doctor [4]. In fact, patients with lower self-rated health (ie, sicker patients) are the most likely to talk to health care providers about the information they found on the Internet [6]. Thus, this revolution in health care information has great potential to affect the way that patients interact with their physicians.
Furthermore, the pace of eHealth development has meant that more and more traditionally offline health-related activities can now be done online. For example, in addition to researching their medical conditions and seeking second opinions, patients can now access information about the quality of care in hospitals (http://www.healthgrades.com/ [7], order prescription drugs online (http://www.drugstore.com/),schedule an appointment with their doctor, participate in thousands of medical discussion groups (http://groups.yahoo.com/), and seekmedical advice from experts in various fields [8]. A select few patients have begun to use e-mail to communicate with their physician [9] and surveys suggest that many more desire the convenience of electronic mail with their provider [10].
This revolution in health care information has great potential to affect the way that patients interact with their physicians. Though many studies have examined the information available on the Internet, both in terms of patient's experiences and the quality of the information, little work has been done to evaluate the use of the Internet for other health-related activities, such as finding information about the quality of care that a hospital provides (http://www.leapfroggroup.org/hospital.htm) [3,[11][12][13][14][15]. What eHealth activities do primary care patients most desire? This question has great relevance to health care providers as it is likely that patients use the Internet, at least in part, to make up for deficiencies in the health care system. Most notable among these deficiencies are physicians' lack of adherence to guidelines [16,17], the chasm between care that is received and that which is possible [18], and the 31% of 6722 adults in a recent national survey by the Commonwealth Fund who did not have a great deal of confidence in their doctor [18].
Although access to many Internet-based health care activities is still limited, the potential for patient-centered applications is broad. We conducted the present study to understand, among primary care patients, the interest in and experience with using the Internet for a variety of health-related activities including electronic communication with health care providers, use of electronic records, ordering medications, and assessing the quality of care. We have defined the difference between what people are doing and what people are interested in doing on the Internet as the opportunity gap. Identifying this opportunity gap is important for future development of Internet health-related activities, as consumer demand is felt to be one of the most influential drivers of Internet health-related activities (eHealth) over the next 5 years [19].

Methods
We recruited a convenience sample of 4 community-based primary care practices from Providence County, Rhode Island. Physicians in each practice were affiliated with the Brown University teaching-hospital network. One of the practices was a state-supported, suburban public-health clinic serving low-income individuals, while the other 3 were suburban, primary care practices. The practices had an average of 3 full-time physicians on staff. The demographic profile for Providence County includes: 14.6% over the age of 65, 13.4% Hispanic or Latino, 6.5% Black or African American, and 15.5% of people living below the federal poverty level [20].
A research assistant approached 355 consecutive adult outpatients from June 1, 2001 to August 15, 2001, to complete a self-administered questionnaire. Subjects were paid $20 to complete the survey. The Institutional Review Board of The Miriam Hospital approved the protocol.

Measures
In order to identify online health-related activities, 20 subjects were recruited for 2 focus groups by e-mailing notices to employees of The Miriam Hospital and Rhode Island Hospital and placing posters in public areas in both hospitals. Focus group participants were asked to identify health-related activities that they currently performed on the Internet. Health-related activities that were noted more than once or that the investigators felt were emerging Internet capabilities were added to the Internet Interest and Experience Survey below.

Internet Interest and Experience Survey (IIES)
Questionnaire items were created to measure the interest of subjects in using the Internet for each of 14 potential activities, such as to "find information about a specific disease or medical condition" and to "find out what questions you should ask your doctors when you see them?" All 14 items were asked of each subject, regardless of whether or not they currently had access to the Internet. The 14 items are in the Appendix.

Sample Descriptors
Brief screening questions for age, gender, educational attainment, health insurance status, perceived health, and race and ethnicity were adapted from the year-2000 Behavioral Risk Factor Surveillance System (BRFSS) [21]. Internet use was assessed using questions adapted from the Pew Internet and American Life Project [2]. A single item was used to measure perceived quality of care. The item was used previously in the 2000 Behavioral Risk Factor Surveillance System [21]. The item asked patients to rate the quality of "all your health care" on a scale from 1 to 5 where 1 is the "worst health care possible" and 5 is the "best health care possible."

Data Analysis
Many of the Internet Interest and Experience Survey (IIES) items had a bimodal distribution, so IIES items were categorized into more interested(4, 5 = 1) and less interested(1, 2, 3 = 0). First, we compared patients' interest in using the Internet for health-related activities by whether or not they had access to the Internet. Second, among patients with access to the Internet, we subtracted the percentage of patients who had used the Internet for each health-related activity from the percentage who expressed an interest in using the Internet for each health-related activity. We defined this difference in percentage as the opportunity gap, the percentage of patients whose needs for using the Internet for health-related activities may not be being met.
All data analyses were carried out using SPSS for Windows, version 10.0.5. Chi-square tests and analysis of variance were used to examine differences in categorical data among variables with 2 categories (eg, age and gender), and more than 2 categories (education level only), respectively. Due to the presence of some missing data, some analyses include fewer than 300 individuals. No variable included in our analysis was missing for more than 2% of the sample.

Results
The survey was completed by 300 patients, for a response rate of 84.5%. Approximately two-thirds (63.7%) reported having Internet access at home, work, school, family or friend's home, or at a library. The mean patient age was 45.2 years (range: 18-75 years), 83.0% (n = 249) were female, 21.3% (n = 64) had completed at least 4 years of college, and 9.7% (n = 29) had no health insurance. Table 1 shows the bivariate associations between Internet access and background characteristics. Internet access was greater among subjects who were younger, who had more formal education, and who had better self-reported health. Internet access was not related to gender, race, health insurance status, or perceived quality of care.  Table 2 shows the self-reported interest in and experience of subjects with the use of the Internet for each specific health-related activity, limited to those with access to the Internet. Percentages in Table 2 reflect individuals who were more interested in using the Internet for each health-related activity; the health-related activities are listed in order of descending opportunity gap. Among all subjects, interest was greatest in using the Internet to: (1) find information about a disease (61.7%), (2) find information about a medication (55.4%), and (3) find out what questions they should be asking during doctor visits (48.0%) (data not shown). Patients with Internet access were more interested, compared to those without access, in all of the Internet health-related activities asked in the questionnaire (data not shown).
Subjects with access to the Internet most frequently reported using it to: (1) find information about a disease (67.0%), (2) find information about a medication (53.4%), and (3) to help modify their lifestyle (such as quitting smoking) (46.6%). The greatest opportunity gap existed for using the Internet to: (1) find out if their health care provider is giving them the tests and treatments they need (39.8%), (2) schedule an appointment with their doctor (36.6%), and (3) find out how the quality of care their doctor providers compares to other doctors (35.9%). For example, in Table 2, 57% of patients with Internet access expressed an interest in using the Internet to find out if their health care provider is giving them the tests and treatments they need, compared to only 17.3% who reported ever doing this on the Internet. The smallest opportunity gap (6.0%) existed for using the Internet to assist in lifestyle modifications, where 52.6% of subjects reported an interest in this online activity and 46.6% said they had already done this on the Internet. Table 2. Self-reported interest and experience of primary care patients with the use of the Internet for specific health-related activities among patients with Internet access (n = 191) -percentages reflect individuals who were more interested in using the Internet for each health-related activity Interest= responding 4 or 5 on a scale where 1 was "not at all interested" and 5 was "extremely interested" in using the Internet for the specific purpose. Experience= responded "yes" that subject had used the Internet for the specific purpose. § Opportunity gap = percentage of patients having interest in Internet( †) minus percentage of patients having experience with Internet( ‡).

Discussion
The Internet is changing the doctor-patient relationship as it provides patients with the potential to make better health decisions via easy access to vast amounts of health information. In the present study, we attempted to investigate the interest in and experience with using the Internet for a variety of health-related activities among a group of primary care patients in Rhode Island. The main findings were that great gaps exist between the health-related activities patients are currently doing on the Internet and the activities they would like to be doing.
For 4 of the 14 Internet health-related activities in the questionnaire, there was an opportunity gap (see definition in Data Analysis section of Methods) of greater than 30%. Three of the 4 health-related activities with the largest opportunity gap were related either directly or indirectly to health care quality, including using the Internet to: (1) "find out if your health care provider was giving you all of the tests or treatments that you are due to have?", (2) find out how the quality of care your doctor provides compares to other doctors?", and (3) "find out what questions you should ask your doctors when you see them?" Another area of opportunity focused on the ability of the Internet to perform administrative functions, including appointment scheduling and creating an online chart. Increasing transparency in quality of care has become a major policy issue, as highlighted in the recent Institute of Medicine report, Crossing the Quality Chasm [18].
Based on our survey, the Internet is meeting the needs of primary care patients for information about diseases and medications, because for each of these activities the majority of patients were interested and the opportunity gap was small. With regards to e-mail, fewer patients in our survey with Internet access were interested in e-mailing their health care providers than noted by Sittig and colleagues (32% vs 65%, respectively) [11]. This difference is most likely due to response bias in the study by Sitttig and colleagues, which reported data on a survey e-mailed to registered users of http://www.webmd.com/, which had aresponse rate of only 15.9%.
Web sites designed to enable patients to learn about the quality of care they are receiving and the questions they should be asking during doctor visits currently do exist, but are relatively uncommon. Two sites in particular, http://www. doctorquality.com/http://www.healthgrades.com/,allow patients to rate health care providers and facilities, but currently have limited functionality due to the lack of high-quality data upon which to base information. On the other hand, many Web sites have suggested questions that patients should ask their doctor (eg, http://www.niddk.nih.gov/health/diabetes/pubs/bldsgr/ bldsgr.htmand http://www.americanheart.org/presenter. jhtml?identifier=4678).Patients may not know these sites or may not perceive them to be useful. Future studies will be necessary to understand how best to either (1) develop Internet resources to address these unmet needs or (2) make the Web sites that meet these needs more available or known to online health seekers.
Another area of potential development focused on the ability of the Internet to perform administrative functions, including appointment scheduling and creating an online chart. Although some patients may have used e-mail with their provider to perform administrative functions [9], use of this technology is not widespread and is not well integrated into the triage and traditional workflow of clinical care. New programming standards, including Extensible Markup Language (XML), that increase the flexibility of the Internet for use as a data warehouse may accelerate the development of these applications [22]. Electronic medication-refill requests integrated with physician decision support systems could potentially reduce errors and be cost-effective [23]. Thus, this Internet-based administration of health care may be desirable to both health care consumers and payers.
If the history of the Internet has been a teacher, then the findings in our survey have great implications for health care providers, insurers, and hospitals. Content on the Internet has generally tracked consumer demand because the two most common ways that Web sites are funded is either by advertising revenue or by subscriptions. Our study suggests that patients want to start to use the Internet not only as a source of information about conditions and medications, but also as a way to inform their health care decisions. Half of our respondents were interested in using the Internet to find out about the quality of care their doctor provides and several Web sites are currently being developed specifically for this purpose (eg, http://www. doctorquality.com/www/, http://www.leapfroggroup.org/ ).Should history prove correct, that the Internet evolves to deliver what consumers desire, our data suggest that consumers may have the greatest potential as a driver of improving health care quality.
The study, however, has several noteworthy limitations: • First, our survey did not measure every current health-related activity available on the Internet, because we based questions on the responses of our focus group participants and on the knowledge of the investigators. For example, support groups and health risk appraisal sites (eg, http://www.realage.com/, http://www. yourcancerrisk.harvard.edu/, http://chess.chsra.wisc.edu/ Chess/)are common on the Internet [24][25][26][27][28], but our survey did not measure the use or interest in these online health-related activities.
• Second, our measurement of the opportunity gap for online health-related activities was intuitive but somewhat arbitrary. For example, we did not measure the degree to which patients' perceived needs were met by each online health-related activity. Diaz and colleagues reported, however, that health information on the Internet was generally perceived as quite useful, ranked second only to health information from a physician or nurse [4]. Also, though the opportunity gap for lifestyle modification over the Internet was small, few sites that offer personally tailored information exist [29]. From a public health perspective, delivery of interactive, tailored health information can be effective in changing patient health behaviors [30,31]. Future studies should examine this issue in more detail.
• Third, though our response rate was greater than 80%, our survey was only done in 4 primary care practices in Rhode Island, therefore it may not generalize to other populations or settings.
• Fourth, our survey relied on self-report of Internet health-related activities. Future studies should consider methods such as installing software on patients' computers to record their Internet activities [3].
Despite the study's limitations, the results of this study have important implications as the number of patients using the Internet for health-related activities continues to grow. People frequently use the Internet to gather health information; about 6 million Americans do so each day [1]. This study confirms these findings but also identifies additional activities where patients show interest in furthering their use of the Internet. As the spectrum of available health-related Internet activities expands, patients may soon use the Internet to research the quality of physicians, schedule their own appointments, and investigate the quality of care they receive. The traditional doctor-patient relationship will continue to evolve as health care on the Internet advances.
Future studies are needed to address this rapidly-evolving technology. Given that the field is quite new, valid and reliable measurements need to be developed. The majority of the data on Internet use is collected using self-reported surveys, as in the present study, yet little is known regarding the validity or reliability of such survey data on Internet use [1]. Without these improved methodologic approaches, the science of the field will move slowly, as studies of doctor-patient communication have been hampered by over-reliance on survey data, rather than more-valid and more-reliable methods, such as videotape and audiotape methods [32]. Eysenbach and colleagues made an important step in this direction by videotaping sessions in which individuals were asked to find specific health-related information on the Internet [3]. Another approach is to incorporate technologies that track and record the sites visited and activities performed on health-related Web sites [3]. Given the proliferation of health-related activities on the Internet, these methodologic advances are greatly needed to advance research in the field.

Conflicts of Interest
The first author (CNS) is a part-owner of an SBIR (Small Business Innovative Research)-supported company (Green Lights, Inc.), which is developing a Web-based tailored-messaging system to help patients prepare for doctor visits.

Appendix 1
(Each numbered question was asked in two ways.) Please answer the following questions based on a scale of 1 to 5 where 1 is "not at all interested" and 5 is "extremely interested".  8. Find information about a specific disease or medicalcondition? 9. Find out if your health care provider was giving you all of thetests or treatments that you were due to have? 10. Find out how the quality of care your doctor provides comparesto other doctors? 11. Find out what questions you should ask your doctors when you seethem? 12. Buy medications from a pharmacy? 13. Look for information about a medication? 14. Create your own personal, online, medical chart of your pastillnesses, tests and treatments?

Introduction
It is widely recognized that utilization of information and communication technologies (ICT) can be of great benefit for the Hellenic, as well as any other, National Health System (NHS). The current penetration of information and communication technologies in the Hellenic health care sector can be summarized as: • lack of appropriate infrastructure, applications, and specialized personnel Projects such as installing health information systems, developing dedicated health portals, and implementing related training programs are expected to reduce expenses and significantly improve clinical and administrative services.
Providing interconnection capabilities to health care establishments will serve not only the Hellenic National Health System administration mechanism, but also medical care itself. The term interconnection capabilities includes the Internet connections of hospitals and administration units, the participation of general health care service-provision units in health care networks, the installation and operation of regional telematic applications, and the development of health-information sources for consumers.
For the development of health-information sources for consumers, validity and non-repudiation (that is, authentication that with high assurance can be asserted to be genuine, and that cannot subsequently be refuted) of health-related information should be ensured. The other capabilities presume the existence of a secure private network maintained by a management and technical-support mechanism to make their use effective.
By secure we mean that the network should reinforce the protection of sensitive information. The characterization private(which does not necessarily mean privately owned) encompasses quality of service, availability, and technical support.
In this paper, we present the proposed design for implementing a secure, cost-effective nationwide data-network infrastructure for the functional interconnection of health care and administration units. This infrastructure will facilitate the exchange of medical information, the exchange of administrative information, and the support of regional telematic applications (eg, telemedicine services in difficult-to-approach areas), wherever required.

Methods
Since 2001, a reformation aimed at the overall improvement of the Hellenic health care sector has been evolving. One of the main changes, directly affecting the framework examined, is the division of the country into 17 autonomous health care regions. Each region has its own administrative structure, supervises all the health care units that reside in its territory, and typically includes 6 to 10 state hospitals, 10 to 15 primary health care units, and 15 to 20 community clinics. The head department (regional health authority) is responsible for the delivery of high-quality health services to the people.
The main objectives related to implementing a secure network infrastructure for health authorities are on the one hand tied up with certain factors of the above-mentioned reformation of the Hellenic health care sector and on the other hand tied up with major advances that make use of information and communication technologies essential in such a field (eg, development of medical portals and the worldwide trend for patient-centered care) [1]. Table 1 lists the factors that lead to the need to implement this health-authorities' interconnection infrastructure. These factors may also be seen as the targets that will eventually be accomplished by the Hellenic National Health System. Creation of the appropriate infrastructure for future provision of special health services for specific population groups (eg, in-house services for older people or patients with long-lasting attendance and nursing needs). Table 1 constitutes the functional requirements for the network infrastructure and includes elements of the above-mentioned evolving reformation and of current advances in the field of data communications. Therefore, certain factors, such as: • the regional structure of the administration services By establishing end-to-end secure links among multiple sites in a public network, VPNs are the answer to the very-expensive solution of using dedicated leased lines for enterprises' and organizations' data communications [2]. Recently, the shared infrastructure across which this alternative way to build a private network operates tends to be the Internet. The Internet is an obvious choice for the realization of a VPN, because most organizations' intranets and collaborating establishments' extranets use TCP/IP (Transmission Control Protocol over Internet Protocol) technologies to exchange data and almost every organization possesses at least one Internet connection.
Other factors contributing to the selection of the Internet are the recent advances in data networking, such as the ability to transmit real-time traffic using the TCP/IP protocol suite.
The evaluation of the success or lack of success of such a solution is going to be based on the following major technological and functional criteria for the implementation of the health authorities' network infrastructure [3]: • security, one of the most important requirements when dealing with extremely-sensitive data, like health care records • quality of service (QoS), especially important in certain applications (eg, teleconsultation or telesurgery) • implementation period, which should be the shortest possible, since the Hellenic National Health System has to stay competitive with the private sector • technical support schema, a matter essential in our case, because of the lack of information technology personnel in Hellenic health care units [4] • backbone and distribution-network bandwidth (capacity for data transfer), which should be the highest possible, while keeping implementation and operating costs low • ability to provide regional telematic services, to supportamong other things -the new regional structure of the Hellenic National Health System • ability to incorporate technology improvements.
Regional Health Care Data Network Infrastructure: Main Characteristics As mentioned earlier, the proposed solution comprises use of the Internet and realization of VPN technologies.
The regional health care data networks will encompass the following technologies related to common VPN architectures [5][6][7]: • remote access VPNs, which extend the corporate network to telecommuters, mobile workers, and remote offices, enabling users to connect to their enterprises' intranets and extranets, using access methods such as dial-up, ISDN (Integrated Services Digital Network), and wireless mobile IP (Internet Protocol); this is the type of VPN service that will be used to provide access to general practitioners • intranet VPNs, which replace private wide area network (WAN) infrastructures, connecting central and remote offices within a corporate intranet with the same policies as a private network; this type of VPN service will be used to interconnect state hospitals, primary health care units, and secondary health care units to the administrative authority of their region • extranet VPNs, which extend these services beyond the organizations' limits to connect customers and partners, eg, pharmacies and suppliers.
The implementation of the Regional Health Care Data Network's infrastructure should at least initially support intranet and remote-access services based on TCP/IP-protocol-suite standards. This requirement leads to selecting one Internet service provider (ISP) per region, meeting a range of proposed criteria, such as: backbone speed and technology, technical personnel know-how, ability to complete ambitious undertakings, points of presence allocation (number and dispersion) per region and nationwide (that is, extensive network presence), cost-effectiveness, capacity of international links, capability to monitor state-of-the-art VPN solutions, commitment to upgrades and to satisfactory service-level agreements, amount of required protocols and of services offered, efficient management schema, and local customer support.
We use the term distribution network to describe the infrastructure of the ISP selected to implement the VPN solution in a specific region. Distribution network nodes are the points of presence of the ISP in that region. Related health care and administration units should be interconnected and should access the Internet through suitable telecommunication circuits terminating at the ISP's nodes. This will minimize the corresponding installation and maintenance costs of special equipment and specialized personnel. A schematic view of the proposed network infrastructure is shown in Figure 1. Another issue of significant importance is the existence of a suitable access network, ie, a local area network in the health care and administration units' premises, which is presupposed existent and not taken into account within the context of the present study.
Separating the network-infrastructure implementation project into phases should be considered (pilots taking place in technologically-advanced regions would be a reasonable approach). The early stages of such a development timetable are expected to bring out problems associated with the extent that public sector services and relevant contractors are capable of assuming charge of such large VPN solutions.
The commitment of every regional health authority to a single private-sector entity is a key element of the proposed architecture, since selecting more than one ISP would probably create complicated management-related obstacles. The obstacles would originate mainly from compatibility issues emerging when different suppliers' equipment is imposed to ensure end-to-end security, impeding uniform services provision throughout the regional network. The main advantage of the proposed schema utilizing VPN-related technologies is the rapid implementation of the national network infrastructure, focusing on reliability and security, with minimal cost. Given that many regional health care and administrative units are expected to have an active Internet connection by the time the project begins, development will be much easier if some complementary measures are taken, such as selecting one ISP per region and enforcing uniform management procedures (ie, IP addressing and domain name scheme [8]). Selecting a single entity per region for carrier and service provision gives a broad variety of connection alternatives -Ethernet in the local loop, xDSL (Digital Subscriber Line protocols), ISDN, different types of leased lines -for telecommuters and health care units, depending on the number of active users in each regional health authority. This can also be considered as another step towards reducing telecommunication costs, by enabling the inclusion of regional requirements and characteristics in the overall solution. Even more, the responsibility of maintaining network availability, suitable computer rooms, and guaranteed technical support is shifted to the ISP's specialized personnel through outsourcing.
This is a great advantage of the proposed solution, since the Hellenic health care sector lacks specialized information technology personnel [9].
Advanced data communication services, including Voice over IP (VoIP) -technology providing voice-telephony services over IP connections -and telemedicine applications are expected to considerably reduce the functional costs of the Hellenic Ministry of Health and Welfare and improve the quality of services provided. Desirable quality of service, protection of sensitive traffic or data, infrastructure availability, network administration, development of technologies, and advanced data communication services should be retained in a predefined level through commitment to applicable service-level agreements. The requirements included in the corresponding contracts should be stated by the Hellenic Ministry of Health and Welfare in the form of a global service-level agreement, in order for every regional health authority to be in a position not only to refine and adjust it, but also to maintain the appropriate mechanism for ensuring its terms. The constraint of one ISP per region simplifies such monitoring of a service-level agreement.
It must be clear from what was previously discussed that, because of the possibility of different VPNs being implemented by more than one ISP, there should be a central point per region (called a Regional Data Center) to support nationwide information interchange. Suitable configuration of the different ISPs' equipment and the use of common protocols should ensure end-to-end security. The interconnection between regional data centers and the Hellenic Ministry of Health and Welfare Central Service may be dealt with by using private lines or by building another VPN between the Hellenic Ministry of Health and Welfare Central service and the Regional Data Centers, with the alternative of using an existing backbone network in both cases. A possible network is the SYZEFXIS network, a project of the Hellenic Ministry of the Interior, Public Administration and Decentralization, aimed at developing a uniform telecommunications infrastructure and providing the gate to the trans-European network TESTA for the Hellenic Public Administration [10]. Consequently, the functional integration of regional VPNs into a national network should either be done through developing a private high-speed backbone network for the Hellenic Ministry of Health and Welfare or by using VPN technologies. The proper exploitation of such a backbone network is critical for successful overall infrastructure operation, requiring effective technical support and constant monitoring, as well as improvement of the offered networking services to meet the constantly-increasing demands.
Moreover, regional data centers should act as: (1) application service providers (ASPs) hosting necessary applications or data and (2) as concentrators (points where the data streams from many simultaneously active inputs can be combined into one shared channel in such a way that the streams can be separated after transmission) between primary and secondary health care units of their regions and the Internet, with the use of encrypted tunnels (a way to implement a secure link). An obvious advantage of this architecture is the need to develop increased security-protection systems only for this central point of Internet access per region. The corresponding schematic view is depicted in Figure 2.

Results
Based on the issues previously discussed, the successful development of a secure cost-effective nationwide-network infrastructure for the Hellenic National Health System should be achieved through selecting a proper technology for implementing each regional health authority's VPN and, in a second phase, their functional interconnection. Proper technologies include IPSec (a protocol that provides security for transmission of sensitive information over unprotected networks, such as the Internet, acting at the network layer, protecting and authenticating IP packets between participating devices) or MPLS (a packet-switching protocol). This should be combined with the gradual application of PKI (Public Key Infrastructure) -a system of public key encryption using digital certificates -or other solutions, in the form of pilot projects, ensuring strong authentication, integrity, validity, and non-repudiation.
The ongoing evolution in the way health care is delivered in Greece and corresponding conclusions from several relevant European and international investigations [11][12][13] lead us to conclude that the problem of secure interconnection of health authorities should be addressed through regional health care data networks. Therefore each region (or in some cases regions with geographic proximity) should deploy its own data network. The nationwide interconnection should be mainly considered as a functional interconnection, comprising specific services and security characteristics, and not necessarily as a physical linkage between the regional health care data networks and the central authority.
By running the project per region, we achieve independence (eg, a possible failure in a region does not necessarily imply direct effects to others) and life-cycle autonomy, without obstacles originating from time lags due to incompatibilities in penetration of information and communication technologies or from private-sector inability to provide such a solution in a region.

Discussion
Recognizing the current situation of the Hellenic health care sector, which sums up to: Since the big challenge is to connect the health care and administration units using public Internet technologies and infrastructures in a secure way, an issue that needs further study is the application of PKI technologies (note that PKI has not been deployed yet on a broad scale in such a complex environment in Europe) [14].
Certain factors should be considered for the Hellenic National Health System to successfully incorporate and exploit the infrastructure proposed in this paper: The issues mentioned above, the well-known problem of understaffing of Greek hospitals' Management Information System departments, and the considerable experience gained during relevant former projects, support the idea of engaging outsourcing mechanisms as a viable solution for a reliable and prompt beginning of the infrastructure's productive use.

Introduction
The Internet and emergent telecommunications infrastructures are transforming health care management as well as the dynamics of health care service markets. The costs of health care delivery systems, products, and services continue to rise everywhere, but performance of health care delivery is associated with institutional and ideological considerations as well as availability of financial and technological resources [1][2][3]. Health is defined here as "a state of complete physical, mental and social well-being and not merely the absence of disease or infirmity" [4]. This definition extends beyond the context of the individual to include population health and quality of life as well as community well-being [5]. In this view, the social environment is considered the source of health and social functioning as well as disease and social problems [6].
The World Health Organization evaluation of the United Kingdom and US health care systems ranks the United Kingdom 18th and the United States 37th of 191 countries rated, while the United States spends 40% more per capita on health care [7][8][9]. Economic models of competitive markets have been demonstrated to be inadequate for understanding health care sector performance [10]. Research shows for example, that life expectancy, a commonly-accepted health care performance indicator, is only indirectly correlated with gross national product through variables related to equitable wealth distribution and investment in public health services [1]. Information and telecommunications technologies are changing the configuration and modifying the definition of health care system efficiency and effectiveness. While rapid access to medical information and expert consultation represents a very significant contribution to health care services, distribution of technological resources and the dynamics of information flows in health care markets are not symmetrical, posing a particular challenge to the design of sustainable health care management systems.
The objective of this paper is to identify ideological differences in health care market infrastructures, including the Internet and telecommunications technologies, by a comparative case analysis of two large health care organizations: the British National Health Service (NHS) and the California-based Kaiser Permanente health maintenance organization.

The Internet, Telecommunications, and Health Care Systems
Growth of the Internet and of the telecommunications sector has affected management in all economic sectors including health care [11][12][13][14][15]. Convergence between enterprise intranets and the Internet contributes to development of services such as content distribution and access infrastructures directly offered on the Internet. Examples in the health care sector include e-mail services among health care professionals and patients, discussion and support groups, messaging and fax services, specialized information search, portal infrastructures, and Web hosting. Management of such services on the Internet reduces the need for specialized personnel such as Webmasters and network managers within traditional health care institutions. Emergence of software applications -ASPs (Application Service Providers) -and infrastructures on the Internet reduces software licensing as well as human resources and other administrative costs associated with local acquisition and management. Database management outsourced to Internet infrastructures reduces network complexity and overload while increasing the speed, reliability, and rigor of information searching and processing.
Examples of these health care infrastructures are apparent in the British model of social medicine at the National Health Service's NHSNet, where convergence between the system intranet and the Internet contributes to a culture of open information exchange. Commercial services on the Internet, such as Cymedix [16] and MedSeek [17] also offer integrated intranet and Internet infrastructures for health care providers, health plans, laboratories, and hospitals. Gateway information infrastructures further serve market supply-and-demand requirements at the national, institutional, professional, or consumer (search engine) levels of analysis [18]. Wireless application service providers are bringing Internet-leveraged services to health care while minimizing modification to physician behavior required for adoption of health care applications [19]. Beyond institutional and area-network boundaries, evolutionary pressures transform assumptions of relatively stable, homogeneous, and centralized systems. Such pressures for collaboration, data sharing, and access to distributed resources increase the focus on interconnection of services both within and across organizations. These services include intelligent networks, switching devices, caching services, appliance servers, storage systems, and storage-area network-management systems in computational and semantic grid architectures comprised of standard protocols, services, application programming interfaces, and software development tools enabling resource sharing. Thus both technological trends and commercial pressures foster service decomposition and distribution through networks rather than host-centric systems [14,15,20].
Medical errors often occur in the clinical communication space, and they may be associated with cognitive difficulties of processing complex information as well as limitations of human memory. Cognitive strategies for coping with such difficulties include development of selective schemata for retention of critical information [21] and use of asynchronous communication channels to manage memory overload in an interruptive communication environment [22]. The type of computational support or telecommunications infrastructure appropriate to clinical decision making may be associated with the common ground shared by human and technological agents in communication processes, as well as the context of such common ground [23]. Stable common ground shared by actors in the clinical decision-making system may be associated with asynchronous modes (prerecorded; store-and-forward) of information transmission, and active computational modeling based on preemptive information storage. For example, prefetching methods may be used to restore archived images to workstations in anticipation of needs for historical case comparisons, patient management, or clinical problem solving [24]. On the other hand, shifting ground may require isochronous (real-time) telecommunications transmission, and conversation through passive computing channels [23]. It is critical to note that real-time talk accounts for as much as 50% of clinical information transactions. This continues to be necessary in increasingly interdisciplinary and diverse medical teams despite availability of advanced information technologies for clinical decision support. Modeling clinical decision making in practice requires holistic analysis of human, software, and infrastructural components of both clinical decision making and human collaboration focused on goal achievement [25]. Cultural and ideological assumptions underlying work practice determine the dynamics of system-wide behavior [26][27][28].
Diverse models of service distribution and payment promote entry of new institutional actors into the health care system, while development of advanced telecommunications infrastructures may support either centralized health care authority or free-market dynamics driven by consumer demand [29]. Consumer participation is generally associated with system emphasis on preventive health care strategies while at the same time managerial tools are designed to control health care costs [28]. Limited health care resources may be associated with participative rationalization of health care services by systematic gathering and integration of individual and collective consumer preferences in methodological models for budgeting and other resource allocation. The efficiency of rationalization in health care markets depends upon the principles of universal access to information and services and the symmetry of information flows among consumers and health care professionals [30,31]. (Symmetry of information flows refers to a consequence of universal access to information: consumers have access to information as do health care professionals. In traditional health care organizations, consumers do not have such access and so are unable to participate in decision making.) Some research reports, for example, that consumer information with participative decision making and risk sharing may reduce demand for expensive medical procedures such as surgery [32].
The Internet and telecommunications infrastructures contribute to control mechanisms of health care management systems through network structures and transaction services. Technological advances extend the classic concepts of markets or clans and hierarchies in economics and organizational theory [33]. Clan control is expressed through norms and standards emergent in organizational behavior on the Internet as well as publication of health care performance data by professional and regulatory organizations [34]. Examples of clan control include codes of professional conduct and codes of ethics governing cyber behavior as well as norms for presentation of Web content and criteria for consumer evaluation of electronic information. On the other hand, network technologies and the Internet give rise to institutional hierarchies of control embedded within the technologies themselves. For example, the Dynamic Authorization Framework for Multiple Authorization Types (DAFMAT) provides a foundation for user-based, role-based, context-based, and emergency authorizations in health care application systems (vs operating systems) as required by HIPAA (US Health Insurance Portability and Accountability Act of 1996) Security Standards [35]. Such structures may be developed to organize enterprise roles such as those identified in organizational hierarchies. Technological control mechanisms may effectively control access to health care, and ensure social or business contract security, confidentiality, and integrity [29].
While in some cases these infrastructures may replace traditional institutional networks, proprietary networks such as WANs (wide area networks) and LANs (local area networks), intranets, and extranets [36] generally extend or complement such traditional structures and enhance federative system integration. Intranets within such traditional institutions may also serve as vehicles for clan or behavioral control processes. Furthermore, proprietary professional or institutional networks may function in parallel to the Internet to offer hierarchical control while at the same time making the system accessible to an extended community. These tightly-linked electronic hierarchies favor the distribution of high quality products and services as well as the adoption of technological innovations throughout the system, but may impede identification of new trading partners [37].
In the centralized social-medicine model, the role of technology may be fundamentally different from that in the free health care market driven by consumer demand. System performance may be related to the internal coherence of Internet-based, market-oriented governance structures and e-commerce transaction characteristics. Relevant governance structures include electronic hierarchies as well as markets [37]. Electronic hierarchies are firm-specific, generally-bilateral electronic linkages controlled by a centralized managerial system as in the model of social medicine, while electronic markets supported by the Internet and telecommunications networks foster competition among multiple buyers and sellers. In the case of behavioral control, autonomous and distributive multiagent architectures contribute to associative system integration including both human and automated actors, while centralized hierarchical control yields structural or business-process integration through federation architectures [38]. As health care systems extend beyond enterprise, institutional, and national boundaries, associative system integration tends to characterize the evolutionary growth process of functional complementation, while federative architecture emerges in the process of enterprise consolidation [39]. Extensive health care systems may exhibit varying degrees of overlap or redundancy among their associated or federated organizational components. Consistent with open systems theory, such redundancy may contribute to health care system flexibility and adaptability [40]. Table 1 summarizes models describing health care management systems within market dynamics dominated by supply or demand: (1) professional covenants, (2) national constitutions, (3) free markets, and (4) business contracts, governed respectively by professional, citizenship, consumer, and managerial values. These models are not mutually exclusive but reflect ideological diversity in health care [29].
The next sections of the paper will present a case-analysis research methodology used in this study and case analyses of the British National Health Service (NHS) and the California-based Kaiser Permanente (KP) health maintenance organization, selected to validate the national constitution and the business contract models of health care management respectively.

Comparative Case Analysis
The research methodology used for this study is comparative case analysis. Case analysis is particularly appropriate to health care services research for a number of reasons. The complexity of health care management is increasing, linking diverse subsystems in ever-larger "mega-systems." Such organizational networks, including alliances and partnerships and their underlying ideologies in the health care sector, are extremely difficult to track and measure. System structures and processes are also rapidly changing as a function of technological innovation and economic globalization, rendering model identification more difficult and variable quantification less relevant. These tendencies contribute to the usefulness of case research methods for analysis and assessment of complex health care management systems within their social, economic, and cultural contexts [41][42][43]. Table 1 presents the health care system models under consideration: the national constitution based on citizenship values, and the institutional business contract model based on managerial values. The organizations to be analyzed are selected according to their classification in these models; the NHS is an example of the national constitution model, while Kaiser Permanente illustrates the business contract model. Each model represents a distinct configuration of characteristics within an institutional and market context. The analytical strategy appropriate to this comparative study is pattern analysis, where patterns of empirical observations are compared to a predicted or theoretical pattern [42]. The model dimensions describing dominant market dynamics (supply and demand) and control processes (clans and hierarchies) are defined as the independent variables, while the characteristics of each model of health care governance specify nonequivalent dependent variables. Multi-trait observations serve to validate the patterns identified in the models. Multiple sources of evidence through a variety of data collection techniques further serve analysis of convergence among different kinds of evidence to validate a single pattern dimension through triangulation [43]. Sources of data used in this study included published research reports from various sources, Internet sites of the health care sector including the organizations under study, and Internet sites of the telecommunications industry. The Web and network structures of NHS and Kaiser Permanente were also analyzed to validate model patterns.
The holistic level of analysis is the health care system and its context [44]. The matrix of health care models in Table 1 (in [44]) guided the identification of evidence within each model to show interrelationships among model dimensions: network architecture and control, information gateways, process and structural integration, access, form of authority, centralization of hierarchy, information quality control, and certification, as well as the values underpinning the institutional health care system.

National Health Service (NHS)
The British National Health Service (NHS) was created in 1948 to provide health care to all British citizens without regard for their ability to pay for services. The population of the United Kingdom (UK) is now estimated to be nearly 60 million people [45]. The mission of the NHS is to promote the highest level of physical and mental health for all citizens by prevention of ill health, diagnosis and treatment of injury and disease, and long-term care of the chronically ill or disabled. The NHS is publicly funded, accountable to Parliament, and managed by the Department of Health [46] through 8 regional offices. It is responsible for setting, implementing, and evaluating public health policies [47]. The NHS employs nearly 1 million people and requires more than £50 billion for its annual operations. Supported by the Department of Health, local health authorities determine health needs and allocate government funding for services provided by NHS hospital trusts and primary care agencies, while special health authorities, such as the National Blood Authority, provide national services. Hospital treatment is arranged through general practitioner's referrals, except in the case of emergencies. There is also a network of NHS walk-in centers where any citizen can access services and advice. In 1999 a special health authority, the NHS Information Authority [48], was created to develop a national Internet and telecommunications infrastructure for electronic health records, an electronic library of health-related knowledge, and service delivery to all citizens.
National Service Frameworks (NSFs) [49] are being developed by the NHS to improve the quality of services and care offered to patients, to reduce the individual and collective burden of certain conditions, to reduce associated health care inequalities, and to assure uniform standards of quality care throughout the system. An important tenet of these policies for chronic disease management is the empowerment of the patient to participate in health care decision making through information, education, and partnership with health care providers [50,51].
The criteria for definition of National Service Frameworks priorities are relevance to the health care policy agenda; significance in terms of mortality, morbidity, disability, or cost; public concern; care complexity; existence of evidence base for clinical and/or managerial improvements; and need for innovation and service reconfiguration. The aims of this program are to define and implement standards and care models for priority services and care groups, and to elaborate performance measures for systematic reviews conducted by the Commission for Health Improvement [52]. The areas so far identified for framework development include cancer, coronary heart disease, renal failure, diabetes, and mental illnesses. Priority populations include children, the aging, and those suffering from long-term disabilities or conditions threatening their independence and quality of life. An important debate concerns the strategy for NHS networking, particularly the role of the Internet. The Internet would offer ubiquitous access to information necessary to link (through secure, encrypted connections) general practitioners and patients to medical records and clinical information [71,72]. The sustainability of the NHSnet network technology is also an important consideration [73]. A variety of connections link affiliated organizations to NHSnet: public switched telephone networks (PSTN) with modems and ID (identification) security cards, integrated services digital network (ISDN) connections, and fixed-link leased lines and routers. A continuing barrier to NHSnet connection is the diversity of computing systems in general practice organizations and their rate of obsolescence. This problem may be solved by the introduction of national standards for systems suppliers known as Requirements for Accreditation (RFA). However, few systems suppliers are able to conform to the required code of connectivity [74]. The NHS Information Authority offers technical advice and support concerning all aspects of telecommunications infrastructure and connection for NHS organizations including trusts, health authorities, general practices, and third-party affiliates. As the primary link between users and suppliers, NHSnet plays a critical role in maintaining network security standards.  [78]. The two strategies are very different in the organizational cultures and financial resources necessary to support them [79]. Proprietary software may be supported with outside resources while open-source software requires specialized personnel and programmers within the enterprise to adapt and maintain organization-specific systems. According to some observers, the NHS is unlikely to mobilize the resources necessary to manage open-source software.
Quality control of information and services offered through the telecommunications infrastructure of the NHS is assured by internal processes of standardization, accreditation, and data-driven tracking and assessment. The integrity of the institutional network is protected by technical audit performed by the WAN service providers and NHSnet-member sponsorship of new affiliates. Clinical practice is supported by research organizations and information sources validated through their linkage to NHS Web sites. The challenge is to evaluate care in terms of a community-based assessment rather than the services rendered to an individual patient requesting care [72,80].
The information-led market dynamics of the NHS system promote evidence-based management, governance, and clinical practice for quality monitoring and improvement throughout the system. At the foundation of the model are the integrative core processes of governance, research, and development [81][82][83][84][85] , engaging about 15% of practices in extraction of data and diffusion of benchmarking information. Although these structures are in place, some research has shown that in the case of the NHS (as in the United States), impact of published clinical outcomes data remains minimal for reasons including credibility, timeliness, awareness, and lack of accountability [89]. These institutions are accessible through the Internet to the public, patients, and health care professionals, thereby achieving broad dissemination and contributing to information symmetry among actors in NHS health care markets. The Centre for Health Information Quality [90], now part of the Help for Health Trust [91], further promotes evidence-based medicine and the production of quality health information for the involvement of consumers as active partners in health care.
Clinical governance has been defined for the NHS as a framework for the development of the organizational capabilities required for sustainable, accountable, patient-oriented, quality-assured health care service delivery integrating health care infrastructures and services delivered within the system through 3 main types of information: (1) health care guidelines, policies, and treatment programs, (2) data descriptive of the care delivered, and (3) data descriptive of the clinical governance system itself [83]. Important issues associated with this definition of clinical governance are the processes of data codification and analysis for translation into useful knowledge as well as the essential contributions of research and development to health care quality [85].
These considerations are particularly important in light of the planned transition from primary care group development in close association with health authorities to independent commissioning agency [92]. This transition will have significant implications for network infrastructures supporting the NHS in terms of information dissemination and outreach to determine population needs. Features of the commissioning process include user involvement, long-term health objectives, conciliation of regulatory requirements with population preferences, and contractor service obligations to the public. These features emphasize relationship management and marketing in contrast to a focus on transactions. Institutional structures will probably lose their boundaries to become synapses linking functions of the health care nervous system in service of individual and community needs. The Internet culture of free and universal access to health care information and services is more consistent with the explicit mission of the NHS than the proprietary, transaction-oriented culture found in many US health care organizations, including Kaiser Permanente, the large California-based health maintenance organization which is the subject of the case analysis presented in the next section of this paper.

Kaiser Permanente
The organization now known as Kaiser Permanente was founded in 1933 by Dr. Sydney Garfield as a prepaid health plan for workers on a construction project in southern California. In 1938 it was extended to a group practice prepayment plan for Grand Coulee Dam construction workers and their families. By 1945 its programs were available for public enrolment, and Kaiser Permanente is now the largest not-for-profit health maintenance organization in the United States, with more than 8 million members in 9 states and the District of Columbia managed in 7 regional structures. These structures include 3 types of organizations in close cooperation: (1) Kaiser Foundation health plans contracting with individuals and groups to provide medical services through hospitals and medical groups, (2) Kaiser Foundation hospitals owning and operating facilities and services and sponsoring educational and research activities, and (3) Permanente medical groups, partnerships, or physician corporations, fully responsible for providing or arranging regional medical services [9].
Kaiser Permanente requires subscribers to make monthly prepayments depending upon age, sex, and marital status of adult subscribers and number of children to be covered. In addition, subscribers pay fees depending upon the specific services such as office consultations received. Kaiser Permanente also offers programs to groups and businesses. Kaiser Permanente is a competitive group-practice HMO (Health Maintenance Organization) whose performance depends on individual and collective choices of affiliation in US health care markets. Physicians generally receive a salary while some portion of their remuneration may reflect service quality, or financial performance of health plans or medical groups. Plan subscribers are encouraged to choose the personal primary care physician responsible for supervising general care and referrals to special services offered within regional organizations. While the personal physician is characterized as the medical authority to prescribe care, many common services and treatments (including all nonemergency surgical procedures) require pre-certification to determine if the service requested by the patient is medically necessary, conforms to accepted medical practice, and should be covered by Kaiser Permanente. The patient, not the physician, is responsible for informing the health plan management of emergency care, initiating requests for services, and responding to service refusal.
The KP enterprise is an affiliation between two distinct organizations, the Kaiser Permanente Health Plan, an administrative and managerial entity, and the Permanente Medical Group of physicians. Telecommunications network applications and medical information technology innovations have been introduced in HMO operations in telemedicine, for example, in teledermatology, teleradiology, and retinal screening of diabetic patients [93]. Since the mid-1990s, the organization has developed strategies for elaboration of Internet applications on 3 axes: (1) provider knowledge requirements through the Permanente Knowledge Connection (PKC), (2) customer needs through KPOnline, and (3) a shared database. There are 5 major portals associated with the KP health care infrastructure of which the members' portal, KPOnline [94], and the Kaiser Permanente public site [95] are accessible directly through the Internet. Three of the portal sites are accessible only through the KP Intranet: the Permanente Knowledge Connection for clinicians, the KP employee site, and a vendor portal for e-commerce and supply-chain content [96].
The KP Care Management Institute (CMI) [97] created in 1997 is responsible for the Permanente Knowledge Connection and its database. The mission of the Care Management Institute is to create, implement, and evaluate effective and efficient care management programs based on the organization's clinical experience, research, data, and synthesis of information about the best clinical practices. Some research suggests that treatment for acute care in chronic disease populations may account for as much as 80% of hospital stays and 55% of emergency room visits. Care management programs are designed to reduce the costs associated with acute-care incidence and chronic-disease management, through preventive strategies aimed at logical groupings of member populations suffering from priority chronic diseases including diabetes, asthma, congestive heart failure, coronary artery disease, and depression. Characteristics of these diseases are high treatment cost, high incidence in the general population, and existence of a research base for medical practice [98]. Economies of scale and scope continue to motivate the choice of a national framework for care management programs: elimination of program duplication, harmonization of regional efforts, rapid diffusion of innovations across regional organizations, generation of comparable clinical and managerial data throughout the health care system, and improved ability to partner with national health care purchasers, accrediting agencies, and other national stakeholder organizations. Some of the difficulties in implementing such programs on a national scale include variance in resources available in regional organizations and lack of integration of contracted facilities outside Kaiser Permanente [98].
The Permanente Knowledge Connection is a network-based application to support shared access to Care Management Institute content regarding clinical best practices identified throughout the regional organizations. The national intranet connects national and regional databases of best practice information where regional and local offices develop procedures for validation and inclusion of information. This design integrates local authority and national needs for centralized structure. Local market areas are offered assistance to develop Web sites conforming to national KP standards. The goals of the national intranet strategy are to identify procedures for the efficient production and diffusion of information, to facilitate and fund the diffusion of information from best practice sites to other sites throughout the Permanente Knowledge Connection, to develop standards for Web information development, to provide national training for developers, to support research and development on enterprise information and communication, and to develop cost-benefit measurement tools to assess Web technology investments [99]. The information linked in the databases is accessible and searchable through the national intranet Web site thus minimizing redundancy. The Permanente Knowledge Connection offers a system for tracking continuing medical education credits, access to text books and journals, and open and closed discussion and work groups.
The major issues to be addressed in managing the Permanente Knowledge Connection are metadata search capabilities, "push" (that is, supply-driven market dynamics dominated by health care institutions) technologies for delivery of information more specifically useful to care providers, and extending access to affiliated care providers. Related to push information strategies are the goals of computerization of all KP physicians' offices with access to the Permanente Knowledge Connection. In December, 1999, 40% of KP physicians were equipped with desktop computers [93]. An important question to be resolved is the proprietary nature of practice guidelines and the extent to which such information should be freely shared with affiliated providers and the public on consumer-oriented Web sites. One approach is to consider the information nonproprietary while protecting the tools for practice implementation. Ethical issues of privacy also need to be considered, such as performance evaluation of providers' medical-information searches as continuous learning or as evidence of inadequate knowledge and training. Increasing use of the Permanente Knowledge Connection is expected to result in improved knowledge management and health care education, more productive patient-provider interactions, and improved economies of scale with minimized duplication of effort.
KPOnline, the members-only consumer Web site is a 3-tiered system that interacts with legacy systems through an intermediate object layer [93]. The main purpose of the site is to provide a service for interacting with KP members as an alternative to telephone calls and office visits. The service includes health learning materials, communication capabilities, and information about Kaiser Permanente. Members may do research on health concerns through the drug and health encyclopedias, complete a personal health assessment, or browse links to other Internet Web sites. Members may also communicate with KP staff or with other members. For example, advice nurses and pharmacists provide answers to routine questions, and discussion groups with the participation of KP staff members offer opportunities to share experience on a variety of health-related themes, including the Web site itself. Kaiser Permanente's strategy is to offer members attractive services as a foundation for more valuable interactions, contributing to perceptions of self-efficacy and for patients' responsibility in decision making regarding their own health care [100]. Monitoring discussion groups creates an opportunity to collect data concerning member needs and satisfaction. KP peer discussion moderators are trained to create an atmosphere of trust, to inform participants without directly offering medical advice, and to facilitate mutual support among members. The objective of this strategy is to avoid the legal difficulties created in offering online advice to an anonymous audience.
Development of the national intranet has proceeded from the design of local networks in 13 semiautonomous regions with independent data centers and a variety of network configurations, to integration of national operations. Each region of the Kaiser Permanente structure controlled its own information technology resources, resulting in some system duplication [93]. Various networking standards were in use at different locations, making communications between systems and networks difficult. Kaiser Permanente started building the foundation for its national information technology (IT) strategy by selecting standardized technology for all its LANs and consolidating 13 regional data centers into 2 national centers. The national network was designed to support Clinical Information System data, Internet and intranet applications, nationwide e-mail, and telemedicine/teleradiology applications. While the networking foundation of the information technology strategy was put into place, Kaiser Permanente began designing national applications, starting with its National Clinical Information System (NCIS) in the fall of 1998, with deployment to be completed by 2004. Increased network bandwidth will allow KP clinicians to offer customized care to members in person, or via telephone, the Internet, or e-mail.
Adoption of health care standards, in particular HL7 and SNOMED, is of critical importance to the National Clinical Information System strategy. Kaiser Permanente has collaborated with the College of American Pathologists to develop the SNOMED terminology, a foundational component of the evolving National Clinical Information System. Experience in the Southern California region in translating clinical practice guidelines into XML (Extensible Markup Language) has shown that a common set of metadata, for instance based on the Dublin Core [101], needs to be identified before standardization of document structure at the national level [102,103]. Kaiser Permanente has also participated in projects focused on the development of computational grids for high-speed, wide-area, data-intensive computing [104][105][106]. In the United States, integration of grid infrastructures poses significant challenges to identifying shared tools and resources as well as a business model for management and operation of a service delivery system based on shared responsibility where corporate business models, such as that of Kaiser Permanente, may be considered proprietary.
In the US market, 90% of managed care organizations are evaluated for consumer information by the National Committee for Quality Assurance (NCQA) [107] based on the same performance indicators (the Health Plan Employer Data and Information Set -HEDIS [108]). These measures include effectiveness and accessibility/availability of care; satisfaction with the experience of care, health plan stability, use and cost of services, informed health care choices, and general health plan descriptive information. Quality control in US health care markets has been assured by governmental and independent accreditation agencies, and new agencies are being introduced to address information and telecommunications technologies in health care. The US Health Insurance Portability and Accountability Act of 1996 (HIPAA) [109] has mandated regulations that govern privacy, security, and electronic transactions standards for health care information related to electronic transactions and privacy. These regulations will require major changes in how health care organizations handle all facets of information management, including reimbursement, coding, security, and patient records. In addition, the Electronic Healthcare Network Accreditation Commission (EHNAC) [110], an independent, not-for-profit accrediting body, provides independent peer evaluation of an organization's ability to perform at industry-established levels. EHNAC Security Accreditation will be appropriate for most institutions under the jurisdiction of HIPAA security regulations including clearinghouses, transaction processors, value-added networks (VANs), payers, providers and provider management organizations. However, accreditation for compliance with HIPAA security regulations will also be assured by other accrediting bodies such as the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) [111] and the National Committee for Quality Assurance.
The case of Kaiser Permanente must be considered in the wider context of managed care in the United States and California. Kaiser Permanente is a group health maintenance organization (HMO) working with exclusive, multi-specialty groups of physicians. Other types of HMOs include those that contract with independent practice associations (IPA HMOs), with other medical groups (network HMOs), as well as those that employ staff physicians (staff HMOs). Managed health care has grown rapidly in California where in 1999, 17 million people representing 52% of the state's population were enrolled in 38 HMOs [112]. Some research has shown that HMOs mark up their health care premiums less in competitive markets; however, it is not clear what effect this has on quality of health care [113].
In California, a number of structural problems related to the growth and diversity of managed health care have emerged in the health care industry: inability to select efficient medical groups due to the objective of inclusiveness, resulting lack of competition among medical groups, lack of incentives for physician loyalty or investment in the health care system, and redundant as well as contradictory rules and procedures [112]. In this consumer-led market, there is little consensus regarding the contractual definition of medical necessity, and the decision-making roles of institutional purchasers of health plans are subject to wide variation. Other problems related to market dynamics and the association between medicine and business involve the pharmaceuticals industry. Mass media marketing of prescription drugs and conflicts of interest among pharmaceutical companies, physicians, and medical schools are of particular concern [114][115][116][117][118][119][120].
Solutions to these structural problems would likely require more centralized regulation and control of health care organizations inconsistent with the US health care market ideology of free individual or collective consumer choice of physicians and service providers. On the other hand, information and telecommunications infrastructures may contribute to reduce administrative redundancy and increase transaction efficiency in ways already emergent on the Internet [121]. Some examples include data collection and management for research and quality control, standardized auditing and credentialing applications, and general administrative services. See, for example, the California Information Exchange (CALINX) [122] for health care data standardization and exchange, and HealthScope [123] for health care evaluation services focusing on health plans, hospitals, medical groups, and doctors. These services are managed under the auspices of the Pacific Business Group on Health [124], a health care purchaser coalition.
Telecommunications networks may also facilitate direct contracting of services to bypass entirely health care management organizations and to delegate decision making and control to medical groups and physicians, although this tendency will be shaped by regulation governing health care and market responses of traditional health care organizations to their virtual competitors [125,112]. Where health care regulation is determined at the state level, emergence of large buyer and provider organizations will be slowed [126], and the scalability of a large system will be hindered. The fragmented US health care environment further complicates multiple-payer, transaction-oriented systems.

Case Comparison
The foregoing case analyses of the British National Health Service and the California-based Kaiser Permanente health maintenance organization are summarized in Table 2 showing the comparison of two health care system models, the National Constitution(NHS) and Business Contract(KP), developed on dimensions guiding this study. Both organizations were founded in the 1940s to facilitate access to health care services. These health care systems, founded on very-different ideological premises, share the strategic objective of building telecommunications and Internet infrastructures to reduce health care costs and improve the quality of products and services delivered to patients. Differing market dynamics -"push" (market dynamics dominated by health care institutions) and "pull" (market dynamics dominated by consumer demand) -consistent with the National Constitution (NHS) and Business Contract (KP) models, have significant effects on the configuration of such infrastructures. In the case of the British NHS, the integrative process of governance focuses on collective stakeholders' interests and on flows of information among market actors and institutions. In the case of the California-based health maintenance organization Kaiser Permanente, the integrative process of management focuses rather on financial transactions and evaluation of their cost/benefit efficiency.
The relationship marketing and management model of the NHS places less emphasis on discrete transactions and emphasizes long-term collaboration and contracting. NHS resources are more broadly accessible on the Internet for patient and public information consistent with citizenship values as well as the principles of evidence-based medicine, universal access, and the empowerment of the system user. Such resource availability also facilitates user participation in policy debates and health care decision making as well as related clinical risk sharing. On the other hand, the KP model builds its competitive advantage on the proprietary nature of its products and services, particularly the tools and methods applied to health care delivery. Protection of core KP competencies is consistent with an ideology of managerial values and proprietorship affecting networks, information resources, and software tools. An example of the difference between the two cultures is expressed in the development of clinical and business process decision models. Such models are proprietary tools in the KP culture, while the NHS has a publicly-accessible Internet Web site presenting the flow charts of patient treatment and related business processes, and offering a free license to prospective users of the model [67].
Both health care systems are involved in the development of appropriate SNOMED and HL7 standards. At the NHS, the internal health care model has long served the national infrastructure, and offers a basis for evaluation of SNOMED and HL7 standards adopted primarily where international health care companies are present in internal markets. These national standards also serve to benchmark key areas where international standards need to be adapted to the NHS context. At Kaiser Permanente on the other hand, development of these standards provides a foundation for integrating a diverse national health care organization, making possible a national intranet.
In both cases there is emphasis on data warehousing on the Internet for clinical decision making and evaluation of health care services, while at the NHS, there is further reference to system-wide need for access to tools for data analysis, including algorithms and methods for calculation of national benchmarks to facilitate comparison of local performance on clinical indicators [83]. In the US context, data such as the Health Plan Employer Data and Information Set (HEDIS) collected by the National Committee for Quality Assurance presents standardized indicators for comparative analysis of competitive health plan performance relating health care outcomes to cost. While these tendencies appear to suggest some convergence between the two models and their associated information cultures, it is important to emphasize that such infrastructures serve processes of governance within the NHS, while in the case of Kaiser Permanente they serve competition and the proprietary tools of health care management.
Based on the comparative case analysis, the next section of the paper presents some conclusions, questions and recommendations for researchers and policy-makers in the health care sector.

Discussion
This comparative case study of two large health care organizations, the British National Health Service and the California-based Kaiser Permanente health maintenance organization has shown how telecommunications and the Internet with other information technologies contribute infrastructures for interactive, integrated, and user-oriented or community-oriented services, as well as governance and resource allocation depending upon organizational forms of control and market ideologies. Information is the foundation of future health care management systems in both cases, including evidence-based medical practice and data-driven health care governance.
The study offers evidence to validate alternative models of health care governance: the national constitution, and the enterprise business contract. This evidence also suggests important questions for health care policy makers as well as for researchers in telecommunications, organizational theory, and health care management. These questions are: • The public sector NHS and not-for-profit private sector Kaiser Permanente offer services in health care markets dominated respectively by the dynamics of supply (push) and demand (pull) respectively. An important question for future development of information strategies and telecommunications infrastructures in the two organizations is the role of the Internet relative to proprietary network structures (WANs and LANs). Some critics of the NHSnet suggest that secure, encrypted Internet connections would better serve the objectives of the system, particularly to satisfy the need for ubiquity and the integration of patient access. The information culture of NHS also seems more open than does the proprietary culture of Kaiser Permanente. How will such networks and their information cultures coevolve in the future? How will national, community, and ideological boundaries be affected in industry structures and regulation? How will international communication standards, data ownership and intellectual property influence model development? How will open-source and proprietary software serve these models?
• A very important difference between the two systems under study is the mechanism for payment; the NHS is motivated by the principle of universal health care with a single payer and service of collective, community-assessed needs; while Kaiser Permanente is founded on the ideology of individual choice and a diverse multiple-payer system. In the US health care market, some research has shown that the number of financial transactions in the sector is increasing faster than the dollar amount of such transactions, with the consequence that the industry could eventually be "strangled" by the administrative demands of the system [127]. As much as 50% percent of US HMO costs may be associated with administration, defining a priority for development of e-business models applicable to health care and technologies for tracking and automating integrated health care business processes [121,128]. In addition, cost-effectiveness analysis (CEA) related to the transaction-based health care model creates liabilities subject to costly jurisprudence, defining medical necessity and the standard of managed care [129]. The performance ranking published by the World Health Organization [7] suggests that UK health care expenditures are more effective than US expenditures in the sector. How can clinical and business process integration incorporate activity-based management and cost-based measures of performance [130]? Will integrated business processes with network architectures improve the efficiency and effectiveness of the multiple-payer system?
• The processes of diversification and deprofessionalization of health care information, products, and services in markets increasingly driven by consumers raise issues concerning quality control and the ethics of the health care system [82]. The American experience and the case of Kaiser Permanente have demonstrated some of the dysfunctions resulting from the predominance of managerial over professional values. How can managerial and professional values be better reconciled through clinical and business process integration to address efficient system performance and health care rationalization?
• Consumer participation and free health care market dynamics rely on the aggregate of individual choice in governance of health care systems with respect to development of health care models and efficiency of products and services offered by the system. How can individual consumer choices contribute to the ethic of sustainable and equitable health care? What methodologies can be applied to identify and respond to collective community health care needs rather than limiting service to patients who request care individually [131]? How can individual and system requirements be reconciled with the support of telecommunications technologies and the Internet?
• The analyses suggest diverse paths to health care performance, while trends suggest integration among provider, clinician, and patient roles as well as tools for health care communication, clinical decision support, evidence-based medical practice, and performance assessment [132][133][134]. These changes further contribute to the breakdown of boundaries among medical research, education, and practice as well as between related disciplines such as medical informatics and health care systems management [135][136][137][138][139]. What are the best strategies to ensure integration of differing health care disciplines, cultures, and ideologies? How can international telecommunications and Internet governance through the Internet Corporation for Assigned Names and Numbers (ICANN) [140] contribute to broader system integration?
• The research methodology of the World Health Report [7] has been developed to facilitate comparative analysis of diverse national health care systems, while the Health Plan Employer Data and Information Set (HEDIS) is designed by the US National Committee for Quality Assurance for evaluation of managed health care plan performance. Some researchers suggest that complexity science and system dynamics modelling would be useful to the study of complex health care systems [141]. What theoretical frameworks, as well as qualitative and quantitative methodological approaches, would contribute to definition and study of global health care system value and performance?
This qualitative comparative analysis has shown the significant extent of diversity in health care delivery systems and the critical contributions of the Internet and telecommunications technologies to health care infrastructures. Conclusions and recommendations for future inquiry will contribute to understanding of health care system evolution and formulation of public policy to manage ideologically-diverse markets in the emerging global health care economy.

Introduction
Health related web sites are now amongst the most frequently accessed sites on the internet with current estimates indicating that there are now over 100,000 sites offering health related information [1]. As a result of the wealth of information available and its apparent popularity, a number of organisations have begun to provide specific tools for searching, rating, and grading this information, while others have set up codes of conduct by which site providers can attest to their high quality services. The aim of such tools is to assist individuals to sift through the mountains of information available so as to be better able to discern valid and reliable messages from those which are misleading or inaccurate.
Recognising that European citizens are avid consumers of health related information on the internet and recognising that they are already using the types of rating system described above, the European Council at Feira on June 19-20 2000 supported an initiative within eEurope 2002 to develop a core set of Quality Criteria for Health Related Websites.
Accordingly a series of meetings was held during 2001 which drew together key players from Government departments, International Organisations, non-governmental organisations and industry, to explore current practices and experiments in this field. Some sixty invited participants from all the Member States, Norway, Switzerland, and the United States of America took part in the kick-off meeting of June 7-8, 2001: they included delegates from industrial, medical, and patient interest groups, delegates from Member States' governments, and key invited speakers from the field of health information ethics. These individuals, and many others, also took part in the web-based consultation which was open from august to November 2001 [2].
The focus of the discussions was primarily on the reliability of health related websites as a potential vehicle for health related messages, rather than on the substance and content of the health messages themselves. The specific aim was to draw up a commonly agreed set of simple quality criteria on which Member States, as well as public and private bodies, may draw in the development of quality initiatives for health related websites. These criteria should be applied in addition to relevant Community law [3].
As a result of the meetings, as well as a web-based public consultation, a core set of quality criteria was established. The criteria may be used as a basis in the development of user guides, voluntary codes of conduct, trustmarks, accreditation systems, or any other initiative adopted by relevant parties, at European, national, regional or organisational level. By using a common set of criteria as a starting point, such initiatives can develop in a focused manner across the European Union.
The objectives for the criteria were defined as follows: • The quality criteria should address issues of both supplier and user education: one document that simultaneously tells suppliers how to comply with key quality criteria and educates users as to what they ought to expect from a good health website; • The quality criteria should address both passive information-giving sites as well as sites that allow for transactions between service or information providers and users (i.e. information, products and services).

•
The quality criteria should facilitate compliance with EU Directives, other current guidelines, and technical standards relevant to this area.
It should be noted that the objective was not to develop a method for the implementation of such criteria at a European level. Although some actors in the field have called for an EU trustmark for health related websites which would operate in a way similar to the CE marking of certain goods [4,5], such initiatives are not within the ambit of the eEurope2002 action. They may, however, be considered within future eEurope action plans and other European programmes.

Textbox 1. Quality Criteria for Health Related Websites
Developed in widespread consultation with representatives of private and public eHealth websites and information providers, other industrial representatives, public officials, and representatives of government departments, international organisations, and non-governmental organisations.
These criteria should be applied in addition to relevant Community law Transparency and Honesty

•
Transparency of provider of site -including name, physical address and electronic address of the person or organisation responsible for the site (see Article 5 and 6 Directive 2000/31/EC on Electronic Commerce).
• Transparency of purpose and objective of the site • Target audience clearly defined (further detail on purpose, multiple audience could be defined at different levels).
• Transparency of all sources of funding for site (grants, sponsors, advertisers, non-profit, voluntary assistance).

Authority
• Clear statement of sources for all information provided and date of publication of source.
• Name and credentials of all human/institutional providers of information put up on the site, including dates at which credentials were received.
Privacy and data protection • Privacy and data protection policy and system for the processing of personal data, including processing invisible to users, to be clearly defined in accordance with community Data Protection legislation (Directives 95/46/EC and 2002/58/EC).

Updating of information
• Clear and regular updating of the site, with date of up-date clearly displayed for each page and/or item as relevant. Regular checking of relevance of information.

Accountability
• Accountability-user feedback, and appropriate oversight responsibility (such as a named quality compliance officer for each site).

•
Responsible partnering -all efforts should be made to ensure that partnering or linking to other websites is undertaken only with trustworthy individuals and organisations who themselves comply with relevant codes of good practice.
• Editorial policy -clear statement describing what procedure was used for selection of content.

Accessibility
• Accessibility-attention to guidelines on physical accessibility as well as general findability, searchability, readability, usability, etc.
Relevant Community Law is listed in reference 3. Terms in italics are further discussed in the Glossary of Terms.
It should also be noted that while this Communication is addressed to the Member States of the European Union and private or public bodies operating in those States, due consideration should be given to the global nature of information disseminated through websites. Accordingly bodies adopting measures to implement the criteria should be aware of the fact that their information will be accessed by many individuals of different nations and cultures. In particular, attention should be paid to the fact that the developing world is a keen consumer of health information and that culturally specific content should be clearly identifiable as such.
Textbox 1 sets out the resulting quality criteria; the ensuing text then explores some of the ways in which they may be implemented. This illustrates what could be done at national or regional level to promote high quality, accessible health related information to the European citizen. The table may easily be detached from the present text to form a simple reminder of the key Quality Criteria for Health related Websites.

Tailoring the Criteria For Different Types of Health Related Content
The criteria set out above are designed to be applicable to the development and maintenance of a health related site irrespective of the type of information or audience to whom the information is targeted. However, one essential quality criterion is that a health-related web site should state clearly what is its target audience and that care should be taken to ensure that both the style and nature of the information, and its presentation, are appropriate for the chosen audience. A number of the respondents to the consultation on the draft criteria, which was conducted between August and October 2001 via the eEurope website, identified the need to address not only site development and maintenance, but also the specific quality issues particular to health related content [6].
When tailoring the content to a chosen audience, a number of factors should be borne in mind in addition to those set out above which should govern the construction of a site. These factors may be considered under the same broad headings as the general site development criteria:

Transparency of Health Related Content
• Transparency of the health related objectives of the provider of the information, including the purpose and objective of content provision, should be clearly defined and stated.
• Where advice or information on particular conditions, lifestyles or medications is given, funding from producers of products thereby implicitly or explicitly endorsed should be transparent to the site user.

Authority of Health Related Content Providers
• Where a policy of using only accredited medical professionals to generate content is adopted, this should be clearly stated and adhered to.
• Where a mixed group of content providers is used, (medical professionals, journalists, personal testimony, etc) the category of content provider of each item should be clearly identifiable.
• Where scientific evidence is cited, the sources of such evidence should be easily identifiable to the user.
• Where a medicinal product is recommended, EU legislation on Medicinal Product advertising should be adhered to, and any documents authorised by a regulatory authority should be made available to the site user.
• Where advice is offered, the site provider should always include a reminder that internet based advice, whether personalised or not, cannot replace a face to face consultation with a healthcare practitioner.

Privacy and data protection of Health Data
• Where any personal information is collected and further processed by the site user, including data processing invisible to the users, the requirements of Directive 95/46/EC on the protection of individuals with regard to the processing of personal data and on the free movement of such data, in particular article 8 on sensitive and health data, should be carefully assessed and full compliance assured.

Updating of Health Related Information
• Where specific health related data are provided, the relevance of such content should be regularly verified.

Accountability for Health Related Content
• Where specific health related user feedback is provided by the site, particularly where personalised medical advice is offered, every effort should be made to ensure that such advice is bona fide and that advisors are suitably qualified to offer advice.

Accessibility in Health Related Content
• Where a particular type of audience is targeted (eg children), the presentation and content of information should be appropriate to the chosen target audience.
• The use of a metadata labelling system may be used to make health data more findable. Such a system may also be used in conjunction with quality criteria to give higher ranking by search engines to those sites or pages labelled as complying with defined quality criteria.
• Apply International or European standards, wherever possible, in order to facilitate notably the interoperability between different services and the cross-border provision of web based health services.

Issues for the European Community
The purpose of the eEurope 2002 action on Quality Criteria for health-related Websites was to encourage the adoption of a common set of basic quality criteria for such sites. The issue of whether and how these criteria might be implemented at European level was not within the terms of the action. The implicit assumption was that this was a matter to be addressed in Member States at national or regional level, making use of the wide range of private and not-for-profit organisations which are already operating systems for implementing quality criteria for health-related websites.
In view of the rapid increase in health-related websites in the European Union and the increase in the number of European Union citizens consulting such sites, it could be argued that there would be merit in the Community establishing its own system for implementing agreed quality criteria. Such a Community-sponsored system would however require considerable resources to set up and operate, and it is unclear that it would offer clear value added to the Member States. The Commission therefore considers that at the moment the difficulties inherent in a Community system would outweigh any possible advantages. Nevertheless, the issue of how and how effectively quality criteria are being implemented is of considerable significance at the European level. To ensure that European citizens have access to reliable health information on the Internet implies not only that there is a consensus on the necessary quality standards, but that those standards are satisfactorily implemented right across the European Union.
This does not mean that the same method of implementation should be used everywhere -indeed it must be doubtful that any particular mechanism would be appropriate in all circumstances and in all countries For example, in pharmaceuticals the Commission is considering ways of meeting the growing demand by patients to be able to access information directly about their medicines. The Commission has included proposals within the current review of EU pharmaceutical legislation, Review 2001, to take account of this growing demand. This has also been recently reinforced by the work of the High Level Group on Innovation and Provision of Medicines -G10 Medicines -which has covered this area in their reports. However, whatever system adopted, there should be clarity about the mechanisms being used in the different Member States and the extent of the involvement of the national and regional health authorities. With the forthcoming enlargement of the European Union, this requirement for transparency becomes even greater.

Simple Codes of Conduct
A number of organisations have adopted an approach similar to that described in this Communication, of setting up meetings and consultations between experts in order to establish by consensus a set of quality criteria. The eHealth Code of Ethics adopted in May 2000 by the Internet Health Coalition [7] is perhaps the best known of such `codes of conduct'.
The object of this and other similar codes is to offer a process of self-assessment by health site providers. However, a basic code of good conduct, or set of quality criteria will form the basis of all the approaches described below.
The way in which such codes are implemented varies. Where the code is adopted by an umbrella organisation, such as the Pharmaceutical Group of the European Union, then the organisation itself seeks to ensure that all members comply with the code. In other cases a code has been adopted for the purposes of in-house application only, as in the case of the American Medical Association. Although other organisations may cite the code, and claim to use it, the organisation developing the code makes no attempt to ensure that other parties are in fact implementing it.
The costs of the code of conduct approach are generally rather low, requiring only an initial outlay on meetings to draw up the code. However, the benefit of such codes can also be rather limited given the absence of effective enforcement mechanisms.
A code of conduct which addresses issues concerning the protection of personal could in itself form a Code of Conduct relevant to a specific area of practice as foreseen in

Self Applied Code of Conduct or Quality Label
A next step in the implementation of a code of conduct can be characterised as the self applied quality label. In such a case a third party organisation develops a code of conduct and allows those who undertake to abide by the terms of the code to display a label, seal or logo which certifies compliance with the code.
The oldest, and perhaps best known, of such labels is the Health on the Net Foundation (HON) [8] label whose eight point set of quality criteria is currently used by more than 3000 internet sites worldwide. A site provider wishing to use the HON label has to make a formal application and a commitment to strictly observe all the HON code principles. Compliant sites identify themselves by the HON code hyperlink (or "active") seal displayed at a prominent location. The seal is termed `active' because clicking on it links the user to the HON site. Conformity with the HON code principles is verified by the team of checkers at HON. HON cannot prevent dishonest operators from simply cutting and pasting the HON code seal onto their Web sites in a bid to enhance their credibility. It does, however, conduct random checks on subscribers to ensure they remain compliant with the HON code. By way of additional policing, the Internet community is invited to report misuse of the label.
The costs of this system of applied labelling are not very high, requiring a relatively small team to process applications for use, maintain random checks of sites displaying the label and respond to any reports of misuse. The benefits may be significant in drawing to the attention of users the importance of the criteria inherent in the label. However, the benefits must be weighed against the requirement of the users of the sites to understand the nature of the label, and perhaps more importantly, to care about its aims and objectives.

User Guidance Tools
A further application of the code of good conduct takes the form of a user guidance tool. In this case compliance with a code is demonstrated not by a label, but by a link to a guidance tool which invites the user to check for him or herself if a site and its contents comply with pre-set criteria.
A typical such tool is displayed by the site as a logo on which the user may click to reveal a series of questions with which to interrogate the site so as to assess whether the information offered is trustworthy. Such tools may be specific to a particular type of information, such as DISCERN [9] which provides a brief questionnaire through which users gain a valid and reliable way of assessing the quality of written information on treatment choices for a health problem. Other tools seek to give guidance on the trustworthiness of any health-related information. An example of this is NETSCORING [10], which uses a questionnaire of 49 criteria falling into eight categories: credibility, content, links, design, interactivity, quantitative aspects, ethics, and accessibility. Yet other tools are targeted at particular categories of internet users. For example, the QUICK [11] tool seeks to provide children with a step-by-step guide to assessing health related information on the internet.
While such tools are frequently adopted for the guidance of users by national health portals (such as National Health Service DIRECT in England and Wales), they may also be used as site development tools by authors and publishers of information since they define the standards which users are entitled to expect.
The financial costs of the user guide are low, often not extending beyond the initial development costs. However, the burden of the use of this kind of tool falls on the internet user, mostly because of the time it takes to apply, which reduces the incentive to use it.

Filtering tools
Where a guidance tool is provided by a third party to a user to apply for him or herself, a filtering tool is applied to provide a searchable database of filtered and accredited information. Such filtering tools are often based on the gateway approach to organising access to Internet. The fundamentals of this approach are that Internet resources are selected for their quality and relevance to a particular target audience. They are then reviewed and resource descriptions created, which are stored, generally with the associated metadata, and generally in a structured database. The consequence of this effort is to improve the recall, and especially the precision, of Internet searches for a particular group of users.
An example of this type of tool is found in the OMNI site (Organising Medical Networked Information) [12] which provides a gateway to evaluated, quality Internet resources in health and medicine, aimed at students, researchers, academics and practitioners in the health and medical sciences.
The costs of such a filtering tool are relatively high in that a team of trained experts must be employed to search for, abstract and classify information on the internet in order that it may be entered into the database. The benefits of such a tool, for the initiated user, are also high since it provides a valuable shortcut to individual searches of the internet using non-specific search engines.

Third Party Quality and Accreditation Labels
The most advanced, and also most costly, of the mechanisms available for implementing quality criteria for health related websites, is the third party accreditation system. A third party issues a label to certify the compliance of the site with the criteria of evaluation.
A range of implementations fall into this category, from lower cost intra-organisation bodies for quality certification, acting in a similar to the notified bodies used in CE marking, to high cost external independent assessors who perform audits and grant accreditation.
At present no third party accreditation bodies are fully operational in Europe, although two noteworthy pilots are running in MEDCERTAIN (a demonstration project of the European Union "Safer Internet Action Plan") and TNO QMIC, a pilot study of the Netherlands Organisation for Applied Scientific Research.
In the case of MEDCERTAIN [13] a series of levels of accreditation are envisaged, starting with a self-certification label in which the provider of the site uses the MEDCERTAIN metalabelling system which incorporates a machine read language to describe and evaluate health information on the Internet. These labels are then in turn used to place a given item of site correctly within a gateway system, such as the OMNI system described above. The next level envisaged by MEDCERTAIN is one in which non-medical experts personally check the site for compliance with the level 1 tagging and also against the agreed set of quality criteria. The highest level involves medical assessment of the content and a rating of the content by relevant healthcare professionals.
The QMIC [14] system, on the other hand, envisages a system similar to the ISO 9000:2000 standard. The QMIC system is based on a complex set of standards drawn up by the third party (TNO in this case) but implemented by the site provider through an internal `quality certification body' who is in turn regularly assessed by the third party organisation to ensure that it is performing its function of internal quality assurance properly. The site, once duly assessed by the internal notified body is then admitted to a portal maintained by the third party who undertakes to ensure that the sites linked into the portal are applying the internal quality assurance system with due care.

Purposes of Implementation of Quality Criteria For Health Related Websites
The general purpose of any quality initiative, whatever method of implementation is chosen, must be the protection of the consumer. However, in some cases that general purpose may be best achieved through educating the user of the service while in other cases the provider of the service will be the target of the quality initiative. In order to assist in the selection of an appropriate implementation method, the targeted purposes of the various methods are examined in more detail below

Educating Users
In their daily lives as consumers of information delivered via the traditional media, most people learn to use a wide range of assessment tools: judging the nature of the outlet providing the information (a general or specialist bookshop or a work exclusively available from the author), the look and feel of the publication as a whole (a magazine with several contributions or a one page pamphlet). In addition, most people know whom to contact for further information (librarian, bookshop assistant, publisher).
In the world of internet content, however, it is less evident what are the relevant indicators of quality. It is for this reason that quality marks and user guides have proliferated, namely to educate the consumer and to provide a recognisable "quality" label which site creators may use to promote their sites. Accordingly, for such codes to be effective it is highly important that the public are informed about the existence of the Codes through public education campaigns.

Assisting searchers
The purpose of quality marks is not, however, simply to provide access to qualified information but also to assist the citizen in coping with the torrent of information which a search on a health related subject might produce: it has been said that "trying to get information form the internet is like drinking from a fire hose. You don't even know what the source of the water is" [15]. In order to try to manage the flow from the fire hydrant into a steady stream from a tap, some organisations have developed and applied tools for rating web sites in order that they may offer pre-selected and more easily searchable sources to their consumers (see for example OMNI or MEDCERTAIN).

Educating Site Providers
The problem is not only with the torrent of information, but also with the behaviour of its purveyors. Whilst it may take considerable effort to find an outlet for unusual or extreme ideas in the traditional media, virtually anyone with a modicum of computer skills and very little money can create their own website. The objective of many of the code of conduct initiatives is therefore to educate both the providers and consumers of information about the processes and good practices that a website should be able to demonstrate.
In order to educate not only the provider, but also the consumer of information a further set of actors have developed a wide range of user assessment tools. Such tools are usually in the form of on-line check lists which ask the consumer to check off types of information as they find them: statement of aim, explicit statement of source of information, explicit date of information, etc. Some may be rather short (HON), some quite detailed (NETSCORING); some are aimed at specific markets (DISCERN -for treatment choices) and some aimed at children (QUICK) to mention but a few.

Assuring Quality
Most of the organisations publishing and administering such codes operate on a simple selflabelling processes in which the site provider undertakes to follow the code and in return displays its "trustmark" relying on spot check and vigilant users to identify those who are not complying with the given code of conduct. While this may not be as effective as a fully policed trustmark system of the type we are used to seeing as regards, for example, electronic products, it nonetheless addresses a need in a reasonably effective manner.

Conclusions
The To achieve this objective, one of the proposed actions is to promote e-health services. It also commits the Commission to monitor " actions taken by Member States to make health information as accessible as possible to citizens as well as initiatives to implement quality criteria for web sites".
In this respect, the e Europe 2005 Action Plan affirms that " it is critical that e-health content and services are developed efficiently, are available for all and health related web sites comply with established quality criteria".
With respect to the enlargement of the European Union it should also be noted that the eEurope+ Action Plan, which was adopted by the accession States to mirror the eEurope 2002 Action Plan, includes similar action on quality criteria for health related websites.
Accordingly it will also be important to monitor the activities undertaken pursuant to that Action Plan.
This Communication sets the scene for the implementation of a core set of quality criteria in Member States for health related web sites, within the context of the relevant existing Community legislation (as listed in footnote 3) and in accordance with the requirements of that legislation. The set of quality criteria is based upon a broad consensus among specialists in this field, health authorities, and prospective users. It is now to be expected that national and regional health authorities, relevant professional associations, and private medical website owners will: • implement the Quality Criteria for Health Related Websites in a manner appropriate to their website and consumers.
• develop information campaigns to educate site developers and citizens about minimum quality standards for health related websites.
• draw on the wide range of health information offered across the European Union and localise such information for the benefit of citizens (translation and cultural adaptation).
• exchange information and experience at European level about how quality standards are being implemented.
Finally, within the context of the Information Society activities and as part of the implementation of the European Union public health programme, consideration will be given to the possibilities of developing and operating a joint action, with the plans drawn up under eEurope, to improve availability to the general public on the Internet of information on health matters, and considering the possibilities for establishing a system of recognizable Community seals of approval for Internet sites.